Re: [oVirt Jenkins] ovirt-system-tests_he-basic-suite-master - Build # 1655 - Still Failing!
On Sun, Jun 28, 2020 at 6:23 AM <jenkins@jenkins.phx.ovirt.org> wrote:
Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1655/
This fails for some time now. Checked last one ^^, and it failed in: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1655/... 2020-06-27 23:05:02,253-0400 INFO ansible task start {'status': 'OK', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_task': 'ovirt.hosted_engine_setup : Check OVF_STORE volume status'} 2020-06-27 23:05:02,253-0400 DEBUG ansible on_any args TASK: ovirt.hosted_engine_setup : Check OVF_STORE volume status kwargs is_conditional:False 2020-06-27 23:05:02,254-0400 DEBUG ansible on_any args localhostTASK: ovirt.hosted_engine_setup : Check OVF_STORE volume status kwargs 2020-06-27 23:05:03,816-0400 DEBUG ansible on_any args <ansible.executor.task_result.TaskResult object at 0x7f454e3d1eb8> kwargs ... 2020-06-27 23:09:39,166-0400 DEBUG var changed: host "localhost" var "ovf_store_status" type "<class 'dict'>" value: "{ "changed": true, "failed": true, "msg": "All items completed", "results": [ { "ansible_loop_var": "item", "attempts": 12, Meaning, it ran 12 times the command: vdsm-client Volume getInfo storagepoolID=41c9fdea-b8e9-11ea-ae2a-5452c0a8c863 storagedomainID=10a69775-8fb6-437d-9e78-2ecfd77c0a45 imageID=c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9 volumeID=6b835f55-a512-4f83-9d25-f6837d8b5cb1 and never got a result with output including "Updated". Any idea? Perhaps it's just a bug in the test, and we should test something else other than "Updated"? Or perhaps there was some other problem but I failed to find it? Thanks,
Build Number: 1655 Build Status: Still Failing Triggered By: Started by timer
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1643 [Galit] Add missing repo which has collectd-write_syslog
[Sandro Bonazzola] Revert "ovirt-release: run only on fc29 nodes"
Changes for Build #1644 [Michal Skrivanek] make GLANCE failures fatal
Changes for Build #1645 [Michal Skrivanek] use // in glance URL
Changes for Build #1646 [Galit] Fix: No module named ost_utils.memoized
Changes for Build #1647 [Martin Necas] ansible: fix deploy scripts to el8
[arachmani] Add arachman@redhat.com to jenkins recipient lists
Changes for Build #1648 [Martin Necas] ansible: fix deploy scripts to el8
[Evgheni Dereveanchin] Do not run jenkins check-patch job on fc29
Changes for Build #1649 [Martin Necas] ansible: fix deploy scripts to el8
Changes for Build #1650 [Martin Necas] ansible: fix deploy scripts to el8
Changes for Build #1651 [Galit] Add centos8.2 official to templates
Changes for Build #1652 [Evgeny Slutsky] he-basic-suite-master: Add Domain_name to the host when vm_run
[Gal Ben Haim] Add gcc to fcraw
[Sandro Bonazzola] pipelines: add ovirt-dependencies
Changes for Build #1653 [Evgeny Slutsky] he-basic-suite-master: Add Domain_name to the host when vm_run
Changes for Build #1654 [Galit] Add centos 8.2 image: el8.2-base
[Evgheni Dereveanchin] do not expand values in run_oc_playbook
Changes for Build #1655 [Galit] Add centos 8.2 image: el8.2-base
----------------- Failed Tests: ----------------- No tests ran.
-- Didi
On Sun, Jun 28, 2020 at 1:23 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Sun, Jun 28, 2020 at 6:23 AM <jenkins@jenkins.phx.ovirt.org> wrote:
Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1655/
This fails for some time now. Checked last one ^^, and it failed in:
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1655/...
2020-06-27 23:05:02,253-0400 INFO ansible task start {'status': 'OK', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_task': 'ovirt.hosted_engine_setup : Check OVF_STORE volume status'} 2020-06-27 23:05:02,253-0400 DEBUG ansible on_any args TASK: ovirt.hosted_engine_setup : Check OVF_STORE volume status kwargs is_conditional:False 2020-06-27 23:05:02,254-0400 DEBUG ansible on_any args localhostTASK: ovirt.hosted_engine_setup : Check OVF_STORE volume status kwargs 2020-06-27 23:05:03,816-0400 DEBUG ansible on_any args <ansible.executor.task_result.TaskResult object at 0x7f454e3d1eb8> kwargs ...
2020-06-27 23:09:39,166-0400 DEBUG var changed: host "localhost" var "ovf_store_status" type "<class 'dict'>" value: "{ "changed": true, "failed": true, "msg": "All items completed", "results": [ { "ansible_loop_var": "item", "attempts": 12,
Meaning, it ran 12 times the command:
vdsm-client Volume getInfo storagepoolID=41c9fdea-b8e9-11ea-ae2a-5452c0a8c863 storagedomainID=10a69775-8fb6-437d-9e78-2ecfd77c0a45 imageID=c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9 volumeID=6b835f55-a512-4f83-9d25-f6837d8b5cb1
and never got a result with output including "Updated". Any idea?
I don't know what is the meaning of "Updated". When you call the vdsm API you should get a response. If you don't get any response maybe vdsm is blocked on inaccessible storage. Or maybe the OVF volume was not created yet and you get an error that the volume does not exist.
Perhaps it's just a bug in the test, and we should test something else other than "Updated"?
Or perhaps there was some other problem but I failed to find it?
Thanks,
Build Number: 1655 Build Status: Still Failing Triggered By: Started by timer
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1643 [Galit] Add missing repo which has collectd-write_syslog
[Sandro Bonazzola] Revert "ovirt-release: run only on fc29 nodes"
Changes for Build #1644 [Michal Skrivanek] make GLANCE failures fatal
Changes for Build #1645 [Michal Skrivanek] use // in glance URL
Changes for Build #1646 [Galit] Fix: No module named ost_utils.memoized
Changes for Build #1647 [Martin Necas] ansible: fix deploy scripts to el8
[arachmani] Add arachman@redhat.com to jenkins recipient lists
Changes for Build #1648 [Martin Necas] ansible: fix deploy scripts to el8
[Evgheni Dereveanchin] Do not run jenkins check-patch job on fc29
Changes for Build #1649 [Martin Necas] ansible: fix deploy scripts to el8
Changes for Build #1650 [Martin Necas] ansible: fix deploy scripts to el8
Changes for Build #1651 [Galit] Add centos8.2 official to templates
Changes for Build #1652 [Evgeny Slutsky] he-basic-suite-master: Add Domain_name to the host when vm_run
[Gal Ben Haim] Add gcc to fcraw
[Sandro Bonazzola] pipelines: add ovirt-dependencies
Changes for Build #1653 [Evgeny Slutsky] he-basic-suite-master: Add Domain_name to the host when vm_run
Changes for Build #1654 [Galit] Add centos 8.2 image: el8.2-base
[Evgheni Dereveanchin] do not expand values in run_oc_playbook
Changes for Build #1655 [Galit] Add centos 8.2 image: el8.2-base
----------------- Failed Tests: ----------------- No tests ran.
-- Didi
On Sun, Jun 28, 2020 at 1:37 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Jun 28, 2020 at 1:23 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Sun, Jun 28, 2020 at 6:23 AM <jenkins@jenkins.phx.ovirt.org> wrote:
Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1655/
This fails for some time now. Checked last one ^^, and it failed in:
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1655/...
2020-06-27 23:05:02,253-0400 INFO ansible task start {'status': 'OK', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_task': 'ovirt.hosted_engine_setup : Check OVF_STORE volume status'} 2020-06-27 23:05:02,253-0400 DEBUG ansible on_any args TASK: ovirt.hosted_engine_setup : Check OVF_STORE volume status kwargs is_conditional:False 2020-06-27 23:05:02,254-0400 DEBUG ansible on_any args localhostTASK: ovirt.hosted_engine_setup : Check OVF_STORE volume status kwargs 2020-06-27 23:05:03,816-0400 DEBUG ansible on_any args <ansible.executor.task_result.TaskResult object at 0x7f454e3d1eb8> kwargs ...
2020-06-27 23:09:39,166-0400 DEBUG var changed: host "localhost" var "ovf_store_status" type "<class 'dict'>" value: "{ "changed": true, "failed": true, "msg": "All items completed", "results": [ { "ansible_loop_var": "item", "attempts": 12,
Meaning, it ran 12 times the command:
vdsm-client Volume getInfo storagepoolID=41c9fdea-b8e9-11ea-ae2a-5452c0a8c863 storagedomainID=10a69775-8fb6-437d-9e78-2ecfd77c0a45 imageID=c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9 volumeID=6b835f55-a512-4f83-9d25-f6837d8b5cb1
and never got a result with output including "Updated". Any idea?
I don't know what is the meaning of "Updated".
Sorry, meant Updated:true. This is a snippet from above ansible-create_target_vm log: "cmd": [ "vdsm-client", "Volume", "getInfo", "storagepoolID=41c9fdea-b8e9-11ea-ae2a-5452c0a8c863", "storagedomainID=10a69775-8fb6-437d-9e78-2ecfd77c0a45", "imageID=c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9", "volumeID=6b835f55-a512-4f83-9d25-f6837d8b5cb1" ], ... "stdout_lines": [ "{", " \"apparentsize\": \"134217728\",", " \"capacity\": \"134217728\",", " \"children\": [],", " \"ctime\": \"1593313323\",", " \"description\": \"{\\\"Updated\\\":false,\\\"Last Updated\\\":null,\\\"Storage Domains\\\":[{\\\"uuid\\\":\\\"10a69775-8fb6-437d-9e78-2ecfd77c0a45\\\"}],\\\"Disk Description\\\":\\\"OVF_STORE\\\"}\",", " \"disktype\": \"OVFS\",", " \"domain\": \"10a69775-8fb6-437d-9e78-2ecfd77c0a45\",", " \"format\": \"RAW\",", " \"generation\": 0,", " \"image\": \"c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9\",", " \"lease\": {", " \"offset\": 0,", " \"owners\": [],", " \"path\": \"/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/10a69775-8fb6-437d-9e78-2ecfd77c0a45/images/c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9/6b835f55-a512-4f83-9d25-f6837d8b5cb1.lease\",", " \"version\": null", " },", " \"legality\": \"LEGAL\",", " \"mtime\": \"0\",", " \"parent\": \"00000000-0000-0000-0000-000000000000\",", " \"pool\": \"\",", " \"status\": \"OK\",", " \"truesize\": \"134217728\",", " \"type\": \"PREALLOCATED\",", " \"uuid\": \"6b835f55-a512-4f83-9d25-f6837d8b5cb1\",", " \"voltype\": \"LEAF\"", "}" ] As you can see, it includes: " \"description\": \"{\\\"Updated\\\":false, Compare with a successful run: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1632/ https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1632/... "stdout_lines": [ "{", " \"apparentsize\": \"23552\",", " \"capacity\": \"134217728\",", " \"children\": [],", " \"ctime\": \"1591843833\",", " \"description\": \"{\\\"Updated\\\":true,\\\"Size\\\":23552,\\\"Last Updated\\\":\\\"Wed Jun 10 22:53:27 EDT 2020\\\",\\\"Storage Domains\\\":[{\\\"uuid\\\":\\\"adf90c1d-52b1-4440-8523-4759498fb055\\\"}],\\\"Disk Description\\\":\\\"OVF_STORE\\\"}\",", " \"disktype\": \"OVFS\",", " \"domain\": \"adf90c1d-52b1-4440-8523-4759498fb055\",", " \"format\": \"RAW\",", " \"generation\": 0,", " \"image\": \"a9982674-a297-4a40-8649-5539b675cb86\",", " \"lease\": {", " \"offset\": 0,", " \"owners\": [],", " \"path\": \"/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/adf90c1d-52b1-4440-8523-4759498fb055/images/a9982674-a297-4a40-8649-5539b675cb86/47fca7a4-f5bc-4873-8660-430449004b8f.lease\",", " \"version\": null", " },", " \"legality\": \"LEGAL\",", " \"mtime\": \"0\",", " \"parent\": \"00000000-0000-0000-0000-000000000000\",", " \"pool\": \"\",", " \"status\": \"OK\",", " \"truesize\": \"24576\",", " \"type\": \"PREALLOCATED\",", " \"uuid\": \"47fca7a4-f5bc-4873-8660-430449004b8f\",", " \"voltype\": \"LEAF\"", "}" ]
When you call the vdsm API you should get a response. If you don't get any response maybe vdsm is blocked on inaccessible storage.
I do get a response, but it never "Finishes", or at least according to the login we have for testing this - we require both 'status=OK' and 'Updated:true'.
Or maybe the OVF volume was not created yet and you get an error that the volume does not exist.
It does exist. We also get e.g.: status:OK. What is the meaning of: Updated: False, Last Updated: null? Is it wrong for checking these?
Perhaps it's just a bug in the test, and we should test something else other than "Updated"?
Or perhaps there was some other problem but I failed to find it?
Thanks,
Build Number: 1655 Build Status: Still Failing Triggered By: Started by timer
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1643 [Galit] Add missing repo which has collectd-write_syslog
[Sandro Bonazzola] Revert "ovirt-release: run only on fc29 nodes"
Changes for Build #1644 [Michal Skrivanek] make GLANCE failures fatal
Changes for Build #1645 [Michal Skrivanek] use // in glance URL
Changes for Build #1646 [Galit] Fix: No module named ost_utils.memoized
Changes for Build #1647 [Martin Necas] ansible: fix deploy scripts to el8
[arachmani] Add arachman@redhat.com to jenkins recipient lists
Changes for Build #1648 [Martin Necas] ansible: fix deploy scripts to el8
[Evgheni Dereveanchin] Do not run jenkins check-patch job on fc29
Changes for Build #1649 [Martin Necas] ansible: fix deploy scripts to el8
Changes for Build #1650 [Martin Necas] ansible: fix deploy scripts to el8
Changes for Build #1651 [Galit] Add centos8.2 official to templates
Changes for Build #1652 [Evgeny Slutsky] he-basic-suite-master: Add Domain_name to the host when vm_run
[Gal Ben Haim] Add gcc to fcraw
[Sandro Bonazzola] pipelines: add ovirt-dependencies
Changes for Build #1653 [Evgeny Slutsky] he-basic-suite-master: Add Domain_name to the host when vm_run
Changes for Build #1654 [Galit] Add centos 8.2 image: el8.2-base
[Evgheni Dereveanchin] do not expand values in run_oc_playbook
Changes for Build #1655 [Galit] Add centos 8.2 image: el8.2-base
----------------- Failed Tests: ----------------- No tests ran.
-- Didi
-- Didi
On Sun, Jun 28, 2020 at 2:35 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Sun, Jun 28, 2020 at 1:37 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Jun 28, 2020 at 1:23 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Sun, Jun 28, 2020 at 6:23 AM <jenkins@jenkins.phx.ovirt.org> wrote:
Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1655/
This fails for some time now. Checked last one ^^, and it failed in:
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1655/...
2020-06-27 23:05:02,253-0400 INFO ansible task start {'status': 'OK', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_task': 'ovirt.hosted_engine_setup : Check OVF_STORE volume status'} 2020-06-27 23:05:02,253-0400 DEBUG ansible on_any args TASK: ovirt.hosted_engine_setup : Check OVF_STORE volume status kwargs is_conditional:False 2020-06-27 23:05:02,254-0400 DEBUG ansible on_any args localhostTASK: ovirt.hosted_engine_setup : Check OVF_STORE volume status kwargs 2020-06-27 23:05:03,816-0400 DEBUG ansible on_any args <ansible.executor.task_result.TaskResult object at 0x7f454e3d1eb8> kwargs ...
2020-06-27 23:09:39,166-0400 DEBUG var changed: host "localhost" var "ovf_store_status" type "<class 'dict'>" value: "{ "changed": true, "failed": true, "msg": "All items completed", "results": [ { "ansible_loop_var": "item", "attempts": 12,
Meaning, it ran 12 times the command:
vdsm-client Volume getInfo storagepoolID=41c9fdea-b8e9-11ea-ae2a-5452c0a8c863 storagedomainID=10a69775-8fb6-437d-9e78-2ecfd77c0a45 imageID=c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9 volumeID=6b835f55-a512-4f83-9d25-f6837d8b5cb1
and never got a result with output including "Updated". Any idea?
I don't know what is the meaning of "Updated".
Sorry, meant Updated:true. This is a snippet from above ansible-create_target_vm log:
"cmd": [ "vdsm-client", "Volume", "getInfo", "storagepoolID=41c9fdea-b8e9-11ea-ae2a-5452c0a8c863", "storagedomainID=10a69775-8fb6-437d-9e78-2ecfd77c0a45", "imageID=c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9", "volumeID=6b835f55-a512-4f83-9d25-f6837d8b5cb1" ], ...
"stdout_lines": [ "{", " \"apparentsize\": \"134217728\",", " \"capacity\": \"134217728\",", " \"children\": [],", " \"ctime\": \"1593313323\",", " \"description\": \"{\\\"Updated\\\":false,\\\"Last Updated\\\":null,\\\"Storage Domains\\\":[{\\\"uuid\\\":\\\"10a69775-8fb6-437d-9e78-2ecfd77c0a45\\\"}],\\\"Disk Description\\\":\\\"OVF_STORE\\\"}\",", " \"disktype\": \"OVFS\",", " \"domain\": \"10a69775-8fb6-437d-9e78-2ecfd77c0a45\",", " \"format\": \"RAW\",", " \"generation\": 0,", " \"image\": \"c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9\",", " \"lease\": {", " \"offset\": 0,", " \"owners\": [],", " \"path\": \"/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/10a69775-8fb6-437d-9e78-2ecfd77c0a45/images/c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9/6b835f55-a512-4f83-9d25-f6837d8b5cb1.lease\",", " \"version\": null", " },", " \"legality\": \"LEGAL\",", " \"mtime\": \"0\",", " \"parent\": \"00000000-0000-0000-0000-000000000000\",", " \"pool\": \"\",", " \"status\": \"OK\",", " \"truesize\": \"134217728\",", " \"type\": \"PREALLOCATED\",", " \"uuid\": \"6b835f55-a512-4f83-9d25-f6837d8b5cb1\",", " \"voltype\": \"LEAF\"", "}" ]
As you can see, it includes: " \"description\": \"{\\\"Updated\\\":false,
Compare with a successful run:
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1632/
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1632/...
"stdout_lines": [ "{", " \"apparentsize\": \"23552\",", " \"capacity\": \"134217728\",", " \"children\": [],", " \"ctime\": \"1591843833\",", " \"description\": \"{\\\"Updated\\\":true,\\\"Size\\\":23552,\\\"Last Updated\\\":\\\"Wed Jun 10 22:53:27 EDT 2020\\\",\\\"Storage Domains\\\":[{\\\"uuid\\\":\\\"adf90c1d-52b1-4440-8523-4759498fb055\\\"}],\\\"Disk Description\\\":\\\"OVF_STORE\\\"}\",", " \"disktype\": \"OVFS\",", " \"domain\": \"adf90c1d-52b1-4440-8523-4759498fb055\",", " \"format\": \"RAW\",", " \"generation\": 0,", " \"image\": \"a9982674-a297-4a40-8649-5539b675cb86\",", " \"lease\": {", " \"offset\": 0,", " \"owners\": [],", " \"path\": \"/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/adf90c1d-52b1-4440-8523-4759498fb055/images/a9982674-a297-4a40-8649-5539b675cb86/47fca7a4-f5bc-4873-8660-430449004b8f.lease\",", " \"version\": null", " },", " \"legality\": \"LEGAL\",", " \"mtime\": \"0\",", " \"parent\": \"00000000-0000-0000-0000-000000000000\",", " \"pool\": \"\",", " \"status\": \"OK\",", " \"truesize\": \"24576\",", " \"type\": \"PREALLOCATED\",", " \"uuid\": \"47fca7a4-f5bc-4873-8660-430449004b8f\",", " \"voltype\": \"LEAF\"", "}" ]
When you call the vdsm API you should get a response. If you don't get any response maybe vdsm is blocked on inaccessible storage.
I do get a response, but it never "Finishes", or at least according to the login we have for testing this - we require both 'status=OK' and 'Updated:true'.
Or maybe the OVF volume was not created yet and you get an error that the volume does not exist.
It does exist. We also get e.g.: status:OK.
What is the meaning of: Updated: False, Last Updated: null?
Is it wrong for checking these?
I don't know, this info is generated by engine. Vdsm does not care about the content of the description field. This is also not a public API, so if you check this you depend on engine private implementation details. Generally using HTTP APIs in vdsm for reading OVF data is abuse of private APIs that were added for engine and should be used only by engine. We are missing a real API for passing configuration updates from hosted engine to hosted engine agent running on the host. Instead we read OVF_STORE volume or engine configuration volume to extract data. Eyal, did we change anything related to updating OVF_STORE recently?
Perhaps it's just a bug in the test, and we should test something else other than "Updated"?
Or perhaps there was some other problem but I failed to find it?
Thanks,
Build Number: 1655 Build Status: Still Failing Triggered By: Started by timer
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1643 [Galit] Add missing repo which has collectd-write_syslog
[Sandro Bonazzola] Revert "ovirt-release: run only on fc29 nodes"
Changes for Build #1644 [Michal Skrivanek] make GLANCE failures fatal
Changes for Build #1645 [Michal Skrivanek] use // in glance URL
Changes for Build #1646 [Galit] Fix: No module named ost_utils.memoized
Changes for Build #1647 [Martin Necas] ansible: fix deploy scripts to el8
[arachmani] Add arachman@redhat.com to jenkins recipient lists
Changes for Build #1648 [Martin Necas] ansible: fix deploy scripts to el8
[Evgheni Dereveanchin] Do not run jenkins check-patch job on fc29
Changes for Build #1649 [Martin Necas] ansible: fix deploy scripts to el8
Changes for Build #1650 [Martin Necas] ansible: fix deploy scripts to el8
Changes for Build #1651 [Galit] Add centos8.2 official to templates
Changes for Build #1652 [Evgeny Slutsky] he-basic-suite-master: Add Domain_name to the host when vm_run
[Gal Ben Haim] Add gcc to fcraw
[Sandro Bonazzola] pipelines: add ovirt-dependencies
Changes for Build #1653 [Evgeny Slutsky] he-basic-suite-master: Add Domain_name to the host when vm_run
Changes for Build #1654 [Galit] Add centos 8.2 image: el8.2-base
[Evgheni Dereveanchin] do not expand values in run_oc_playbook
Changes for Build #1655 [Galit] Add centos 8.2 image: el8.2-base
----------------- Failed Tests: ----------------- No tests ran.
-- Didi
-- Didi
On Sun, Jun 28, 2020 at 2:51 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Jun 28, 2020 at 2:35 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Sun, Jun 28, 2020 at 1:37 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Jun 28, 2020 at 1:23 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Sun, Jun 28, 2020 at 6:23 AM <jenkins@jenkins.phx.ovirt.org> wrote:
Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1655/
This fails for some time now. Checked last one ^^, and it failed in:
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1655/...
2020-06-27 23:05:02,253-0400 INFO ansible task start {'status': 'OK', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_task': 'ovirt.hosted_engine_setup : Check OVF_STORE volume status'} 2020-06-27 23:05:02,253-0400 DEBUG ansible on_any args TASK: ovirt.hosted_engine_setup : Check OVF_STORE volume status kwargs is_conditional:False 2020-06-27 23:05:02,254-0400 DEBUG ansible on_any args localhostTASK: ovirt.hosted_engine_setup : Check OVF_STORE volume status kwargs 2020-06-27 23:05:03,816-0400 DEBUG ansible on_any args <ansible.executor.task_result.TaskResult object at 0x7f454e3d1eb8> kwargs ...
2020-06-27 23:09:39,166-0400 DEBUG var changed: host "localhost" var "ovf_store_status" type "<class 'dict'>" value: "{ "changed": true, "failed": true, "msg": "All items completed", "results": [ { "ansible_loop_var": "item", "attempts": 12,
Meaning, it ran 12 times the command:
vdsm-client Volume getInfo storagepoolID=41c9fdea-b8e9-11ea-ae2a-5452c0a8c863 storagedomainID=10a69775-8fb6-437d-9e78-2ecfd77c0a45 imageID=c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9 volumeID=6b835f55-a512-4f83-9d25-f6837d8b5cb1
and never got a result with output including "Updated". Any idea?
I don't know what is the meaning of "Updated".
Sorry, meant Updated:true. This is a snippet from above ansible-create_target_vm log:
"cmd": [ "vdsm-client", "Volume", "getInfo", "storagepoolID=41c9fdea-b8e9-11ea-ae2a-5452c0a8c863", "storagedomainID=10a69775-8fb6-437d-9e78-2ecfd77c0a45", "imageID=c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9", "volumeID=6b835f55-a512-4f83-9d25-f6837d8b5cb1" ], ...
"stdout_lines": [ "{", " \"apparentsize\": \"134217728\",", " \"capacity\": \"134217728\",", " \"children\": [],", " \"ctime\": \"1593313323\",", " \"description\": \"{\\\"Updated\\\":false,\\\"Last Updated\\\":null,\\\"Storage Domains\\\":[{\\\"uuid\\\":\\\"10a69775-8fb6-437d-9e78-2ecfd77c0a45\\\"}],\\\"Disk Description\\\":\\\"OVF_STORE\\\"}\",", " \"disktype\": \"OVFS\",", " \"domain\": \"10a69775-8fb6-437d-9e78-2ecfd77c0a45\",", " \"format\": \"RAW\",", " \"generation\": 0,", " \"image\": \"c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9\",", " \"lease\": {", " \"offset\": 0,", " \"owners\": [],", " \"path\": \"/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/10a69775-8fb6-437d-9e78-2ecfd77c0a45/images/c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9/6b835f55-a512-4f83-9d25-f6837d8b5cb1.lease\",", " \"version\": null", " },", " \"legality\": \"LEGAL\",", " \"mtime\": \"0\",", " \"parent\": \"00000000-0000-0000-0000-000000000000\",", " \"pool\": \"\",", " \"status\": \"OK\",", " \"truesize\": \"134217728\",", " \"type\": \"PREALLOCATED\",", " \"uuid\": \"6b835f55-a512-4f83-9d25-f6837d8b5cb1\",", " \"voltype\": \"LEAF\"", "}" ]
As you can see, it includes: " \"description\": \"{\\\"Updated\\\":false,
Compare with a successful run:
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1632/
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1632/...
"stdout_lines": [ "{", " \"apparentsize\": \"23552\",", " \"capacity\": \"134217728\",", " \"children\": [],", " \"ctime\": \"1591843833\",", " \"description\": \"{\\\"Updated\\\":true,\\\"Size\\\":23552,\\\"Last Updated\\\":\\\"Wed Jun 10 22:53:27 EDT 2020\\\",\\\"Storage Domains\\\":[{\\\"uuid\\\":\\\"adf90c1d-52b1-4440-8523-4759498fb055\\\"}],\\\"Disk Description\\\":\\\"OVF_STORE\\\"}\",", " \"disktype\": \"OVFS\",", " \"domain\": \"adf90c1d-52b1-4440-8523-4759498fb055\",", " \"format\": \"RAW\",", " \"generation\": 0,", " \"image\": \"a9982674-a297-4a40-8649-5539b675cb86\",", " \"lease\": {", " \"offset\": 0,", " \"owners\": [],", " \"path\": \"/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/adf90c1d-52b1-4440-8523-4759498fb055/images/a9982674-a297-4a40-8649-5539b675cb86/47fca7a4-f5bc-4873-8660-430449004b8f.lease\",", " \"version\": null", " },", " \"legality\": \"LEGAL\",", " \"mtime\": \"0\",", " \"parent\": \"00000000-0000-0000-0000-000000000000\",", " \"pool\": \"\",", " \"status\": \"OK\",", " \"truesize\": \"24576\",", " \"type\": \"PREALLOCATED\",", " \"uuid\": \"47fca7a4-f5bc-4873-8660-430449004b8f\",", " \"voltype\": \"LEAF\"", "}" ]
When you call the vdsm API you should get a response. If you don't get any response maybe vdsm is blocked on inaccessible storage.
I do get a response, but it never "Finishes", or at least according to the login we have for testing this - we require both 'status=OK' and 'Updated:true'.
Or maybe the OVF volume was not created yet and you get an error that the volume does not exist.
It does exist. We also get e.g.: status:OK.
What is the meaning of: Updated: False, Last Updated: null?
Is it wrong for checking these?
I don't know, this info is generated by engine. Vdsm does not care about the content of the description field.
For reference, it was added here: https://gerrit.ovirt.org/88650
This is also not a public API, so if you check this you depend on engine private implementation details.
Generally using HTTP APIs in vdsm for reading OVF data is abuse of private APIs that were added for engine and should be used only by engine. We are missing a real API for passing configuration updates from hosted engine to hosted engine agent running on the host. Instead we read OVF_STORE volume or engine configuration volume to extract data.
Eyal, did we change anything related to updating OVF_STORE recently?
Perhaps it's just a bug in the test, and we should test something else other than "Updated"?
Or perhaps there was some other problem but I failed to find it?
Thanks,
Build Number: 1655 Build Status: Still Failing Triggered By: Started by timer
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1643 [Galit] Add missing repo which has collectd-write_syslog
[Sandro Bonazzola] Revert "ovirt-release: run only on fc29 nodes"
Changes for Build #1644 [Michal Skrivanek] make GLANCE failures fatal
Changes for Build #1645 [Michal Skrivanek] use // in glance URL
Changes for Build #1646 [Galit] Fix: No module named ost_utils.memoized
Changes for Build #1647 [Martin Necas] ansible: fix deploy scripts to el8
[arachmani] Add arachman@redhat.com to jenkins recipient lists
Changes for Build #1648 [Martin Necas] ansible: fix deploy scripts to el8
[Evgheni Dereveanchin] Do not run jenkins check-patch job on fc29
Changes for Build #1649 [Martin Necas] ansible: fix deploy scripts to el8
Changes for Build #1650 [Martin Necas] ansible: fix deploy scripts to el8
Changes for Build #1651 [Galit] Add centos8.2 official to templates
Changes for Build #1652 [Evgeny Slutsky] he-basic-suite-master: Add Domain_name to the host when vm_run
[Gal Ben Haim] Add gcc to fcraw
[Sandro Bonazzola] pipelines: add ovirt-dependencies
Changes for Build #1653 [Evgeny Slutsky] he-basic-suite-master: Add Domain_name to the host when vm_run
Changes for Build #1654 [Galit] Add centos 8.2 image: el8.2-base
[Evgheni Dereveanchin] do not expand values in run_oc_playbook
Changes for Build #1655 [Galit] Add centos 8.2 image: el8.2-base
----------------- Failed Tests: ----------------- No tests ran.
-- Didi
-- Didi
-- Didi
hit into the same issue https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/10162/... On Sun, Jun 28, 2020 at 3:01 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Sun, Jun 28, 2020 at 2:51 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Jun 28, 2020 at 2:35 PM Yedidyah Bar David <didi@redhat.com>
On Sun, Jun 28, 2020 at 1:37 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Jun 28, 2020 at 1:23 PM Yedidyah Bar David <didi@redhat.com>
wrote:
On Sun, Jun 28, 2020 at 6:23 AM <jenkins@jenkins.phx.ovirt.org>
wrote:
Project:
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/
Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1655/
This fails for some time now. Checked last one ^^, and it failed in:
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1655/...
2020-06-27 23:05:02,253-0400 INFO ansible task start {'status':
'OK',
'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_task': 'ovirt.hosted_engine_setup : Check OVF_STORE volume status'} 2020-06-27 23:05:02,253-0400 DEBUG ansible on_any args TASK: ovirt.hosted_engine_setup : Check OVF_STORE volume status kwargs is_conditional:False 2020-06-27 23:05:02,254-0400 DEBUG ansible on_any args localhostTASK: ovirt.hosted_engine_setup : Check OVF_STORE volume status kwargs 2020-06-27 23:05:03,816-0400 DEBUG ansible on_any args <ansible.executor.task_result.TaskResult object at 0x7f454e3d1eb8> kwargs ...
2020-06-27 23:09:39,166-0400 DEBUG var changed: host "localhost" var "ovf_store_status" type "<class 'dict'>" value: "{ "changed": true, "failed": true, "msg": "All items completed", "results": [ { "ansible_loop_var": "item", "attempts": 12,
Meaning, it ran 12 times the command:
vdsm-client Volume getInfo storagepoolID=41c9fdea-b8e9-11ea-ae2a-5452c0a8c863 storagedomainID=10a69775-8fb6-437d-9e78-2ecfd77c0a45 imageID=c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9 volumeID=6b835f55-a512-4f83-9d25-f6837d8b5cb1
and never got a result with output including "Updated". Any idea?
I don't know what is the meaning of "Updated".
Sorry, meant Updated:true. This is a snippet from above ansible-create_target_vm log:
"cmd": [ "vdsm-client", "Volume", "getInfo", "storagepoolID=41c9fdea-b8e9-11ea-ae2a-5452c0a8c863", "storagedomainID=10a69775-8fb6-437d-9e78-2ecfd77c0a45", "imageID=c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9", "volumeID=6b835f55-a512-4f83-9d25-f6837d8b5cb1" ], ...
"stdout_lines": [ "{", " \"apparentsize\": \"134217728\",", " \"capacity\": \"134217728\",", " \"children\": [],", " \"ctime\": \"1593313323\",", " \"description\": \"{\\\"Updated\\\":false,\\\"Last Updated\\\":null,\\\"Storage
Domains\\\":[{\\\"uuid\\\":\\\"10a69775-8fb6-437d-9e78-2ecfd77c0a45\\\"}],\\\"Disk
Description\\\":\\\"OVF_STORE\\\"}\",", " \"disktype\": \"OVFS\",", " \"domain\": \"10a69775-8fb6-437d-9e78-2ecfd77c0a45\",", " \"format\": \"RAW\",", " \"generation\": 0,", " \"image\": \"c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9\",", " \"lease\": {", " \"offset\": 0,", " \"owners\": [],", " \"path\":
\"/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/10a69775-8fb6-437d-9e78-2ecfd77c0a45/images/c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9/6b835f55-a512-4f83-9d25-f6837d8b5cb1.lease\",",
" \"version\": null", " },", " \"legality\": \"LEGAL\",", " \"mtime\": \"0\",", " \"parent\":
\"00000000-0000-0000-0000-000000000000\",",
" \"pool\": \"\",", " \"status\": \"OK\",", " \"truesize\": \"134217728\",", " \"type\": \"PREALLOCATED\",", " \"uuid\":
\"6b835f55-a512-4f83-9d25-f6837d8b5cb1\",",
" \"voltype\": \"LEAF\"", "}" ]
As you can see, it includes: " \"description\":
\"{\\\"Updated\\\":false,
Compare with a successful run:
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1632/
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1632/...
"stdout_lines": [ "{", " \"apparentsize\": \"23552\",", " \"capacity\": \"134217728\",", " \"children\": [],", " \"ctime\": \"1591843833\",", " \"description\": \"{\\\"Updated\\\":true,\\\"Size\\\":23552,\\\"Last Updated\\\":\\\"Wed Jun 10 22:53:27 EDT 2020\\\",\\\"Storage
Domains\\\":[{\\\"uuid\\\":\\\"adf90c1d-52b1-4440-8523-4759498fb055\\\"}],\\\"Disk
Description\\\":\\\"OVF_STORE\\\"}\",", " \"disktype\": \"OVFS\",", " \"domain\": \"adf90c1d-52b1-4440-8523-4759498fb055\",", " \"format\": \"RAW\",", " \"generation\": 0,", " \"image\": \"a9982674-a297-4a40-8649-5539b675cb86\",", " \"lease\": {", " \"offset\": 0,", " \"owners\": [],", " \"path\":
\"/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/adf90c1d-52b1-4440-8523-4759498fb055/images/a9982674-a297-4a40-8649-5539b675cb86/47fca7a4-f5bc-4873-8660-430449004b8f.lease\",",
" \"version\": null", " },", " \"legality\": \"LEGAL\",", " \"mtime\": \"0\",", " \"parent\":
\"00000000-0000-0000-0000-000000000000\",",
" \"pool\": \"\",", " \"status\": \"OK\",", " \"truesize\": \"24576\",", " \"type\": \"PREALLOCATED\",", " \"uuid\":
\"47fca7a4-f5bc-4873-8660-430449004b8f\",",
" \"voltype\": \"LEAF\"", "}" ]
When you call the vdsm API you should get a response. If you don't
get
any response maybe vdsm is blocked on inaccessible storage.
I do get a response, but it never "Finishes", or at least according to
login we have for testing this - we require both 'status=OK' and 'Updated:true'.
Or maybe the OVF volume was not created yet and you get an error that the volume does not exist.
It does exist. We also get e.g.: status:OK.
What is the meaning of: Updated: False, Last Updated: null?
Is it wrong for checking these?
I don't know, this info is generated by engine. Vdsm does not care about
wrote: the the
content of the description field.
For reference, it was added here:
https://gerrit.ovirt.org/88650
This is also not a public API, so if you check this you depend on engine
implementation details.
Generally using HTTP APIs in vdsm for reading OVF data is abuse of
private private
APIs that were added for engine and should be used only by engine. We are missing a real API for passing configuration updates from hosted engine to hosted engine agent running on the host. Instead we read OVF_STORE volume or engine configuration volume to extract data.
Eyal, did we change anything related to updating OVF_STORE recently?
Perhaps it's just a bug in the test, and we should test something else other than "Updated"?
Or perhaps there was some other problem but I failed to find it?
Thanks,
Build Number: 1655 Build Status: Still Failing Triggered By: Started by timer
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1643 [Galit] Add missing repo which has collectd-write_syslog
[Sandro Bonazzola] Revert "ovirt-release: run only on fc29 nodes"
Changes for Build #1644 [Michal Skrivanek] make GLANCE failures fatal
Changes for Build #1645 [Michal Skrivanek] use // in glance URL
Changes for Build #1646 [Galit] Fix: No module named ost_utils.memoized
Changes for Build #1647 [Martin Necas] ansible: fix deploy scripts to el8
[arachmani] Add arachman@redhat.com to jenkins recipient lists
Changes for Build #1648 [Martin Necas] ansible: fix deploy scripts to el8
[Evgheni Dereveanchin] Do not run jenkins check-patch job on fc29
Changes for Build #1649 [Martin Necas] ansible: fix deploy scripts to el8
Changes for Build #1650 [Martin Necas] ansible: fix deploy scripts to el8
Changes for Build #1651 [Galit] Add centos8.2 official to templates
Changes for Build #1652 [Evgeny Slutsky] he-basic-suite-master: Add Domain_name to the host when vm_run
[Gal Ben Haim] Add gcc to fcraw
[Sandro Bonazzola] pipelines: add ovirt-dependencies
Changes for Build #1653 [Evgeny Slutsky] he-basic-suite-master: Add Domain_name to the host when vm_run
Changes for Build #1654 [Galit] Add centos 8.2 image: el8.2-base
[Evgheni Dereveanchin] do not expand values in run_oc_playbook
Changes for Build #1655 [Galit] Add centos 8.2 image: el8.2-base
----------------- Failed Tests: ----------------- No tests ran.
-- Didi
-- Didi
-- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/WJW2GDRUERSXLD...
participants (3)
-
Artem Hrechanychenko -
Nir Soffer -
Yedidyah Bar David