[ovirt-devel] Re: [oVirt Jenkins] ovirt-system-tests_he-basic-suite-master - Build # 1655 - Still Failing!

On Sun, Jun 28, 2020 at 2:35 PM Yedidyah Bar David <didi(a)redhat.com&gt; wrote: > > On Sun, Jun 28, 2020 at 1:37 PM Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > > > > On Sun, Jun 28, 2020 at 1:23 PM Yedidyah Bar David <didi(a)redhat.com&gt; wrote: > > > > > > On Sun, Jun 28, 2020 at 6:23 AM <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: > > > > > > > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ > > > > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1655/ > > > > > > This fails for some time now. Checked last one ^^, and it failed in: > > > > > > https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/16... > > > > > > 2020-06-27 23:05:02,253-0400 INFO ansible task start {'status': 'OK', > > > 'ansible_type': 'task', 'ansible_playbook': > > > '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', > > > 'ansible_task': 'ovirt.hosted_engine_setup : Check OVF_STORE volume > > > status'} > > > 2020-06-27 23:05:02,253-0400 DEBUG ansible on_any args TASK: > > > ovirt.hosted_engine_setup : Check OVF_STORE volume status kwargs > > > is_conditional:False > > > 2020-06-27 23:05:02,254-0400 DEBUG ansible on_any args localhostTASK: > > > ovirt.hosted_engine_setup : Check OVF_STORE volume status kwargs > > > 2020-06-27 23:05:03,816-0400 DEBUG ansible on_any args > > > <ansible.executor.task_result.TaskResult object at 0x7f454e3d1eb8> > > > kwargs > > > ... > > > > > > 2020-06-27 23:09:39,166-0400 DEBUG var changed: host "localhost" var > > > "ovf_store_status" type "<class 'dict'>" value: "{ > > > "changed": true, > > > "failed": true, > > > "msg": "All items completed", > > > "results": [ > > > { > > > "ansible_loop_var": "item", > > > "attempts": 12, > > > > > > Meaning, it ran 12 times the command: > > > > > > vdsm-client Volume getInfo > > > storagepoolID=41c9fdea-b8e9-11ea-ae2a-5452c0a8c863 > > > storagedomainID=10a69775-8fb6-437d-9e78-2ecfd77c0a45 > > > imageID=c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9 > > > volumeID=6b835f55-a512-4f83-9d25-f6837d8b5cb1 > > > > > > and never got a result with output including "Updated". Any idea? > > > > I don't know what is the meaning of "Updated". > > Sorry, meant Updated:true. This is a snippet from above > ansible-create_target_vm log: > > "cmd": [ > "vdsm-client", > "Volume", > "getInfo", > "storagepoolID=41c9fdea-b8e9-11ea-ae2a-5452c0a8c863", > "storagedomainID=10a69775-8fb6-437d-9e78-2ecfd77c0a45", > "imageID=c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9", > "volumeID=6b835f55-a512-4f83-9d25-f6837d8b5cb1" > ], > ... > > "stdout_lines": [ > "{", > " \"apparentsize\": \"134217728\",", > " \"capacity\": \"134217728\",", > " \"children\": [],", > " \"ctime\": \"1593313323\",", > " \"description\": > \"{\\\"Updated\\\":false,\\\"Last Updated\\\":null,\\\"Storage > Domains\\\":[{\\\"uuid\\\":\\\"10a69775-8fb6-437d-9e78-2ecfd77c0a45\\\"}],\\\"Disk > Description\\\":\\\"OVF_STORE\\\"}\",", > " \"disktype\": \"OVFS\",", > " \"domain\": \"10a69775-8fb6-437d-9e78-2ecfd77c0a45\",", > " \"format\": \"RAW\",", > " \"generation\": 0,", > " \"image\": \"c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9\",", > " \"lease\": {", > " \"offset\": 0,", > " \"owners\": [],", > " \"path\": > \"/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/10a69775-8fb6-437d-9e78-2ecfd77c0a45/images/c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9/6b835f55-a512-4f83-9d25-f6837d8b5cb1.lease\",", > " \"version\": null", > " },", > " \"legality\": \"LEGAL\",", > " \"mtime\": \"0\",", > " \"parent\": \"00000000-0000-0000-0000-000000000000\",", > " \"pool\": \"\",", > " \"status\": \"OK\",", > " \"truesize\": \"134217728\",", > " \"type\": \"PREALLOCATED\",", > " \"uuid\": \"6b835f55-a512-4f83-9d25-f6837d8b5cb1\",", > " \"voltype\": \"LEAF\"", > "}" > ] > > As you can see, it includes: " \"description\": \"{\\\"Updated\\\":false, > > Compare with a successful run: > > https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1632/ > > https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/16... > > "stdout_lines": [ > "{", > " \"apparentsize\": \"23552\",", > " \"capacity\": \"134217728\",", > " \"children\": [],", > " \"ctime\": \"1591843833\",", > " \"description\": > \"{\\\"Updated\\\":true,\\\"Size\\\":23552,\\\"Last > Updated\\\":\\\"Wed Jun 10 22:53:27 EDT 2020\\\",\\\"Storage > Domains\\\":[{\\\"uuid\\\":\\\"adf90c1d-52b1-4440-8523-4759498fb055\\\"}],\\\"Disk > Description\\\":\\\"OVF_STORE\\\"}\",", > " \"disktype\": \"OVFS\",", > " \"domain\": \"adf90c1d-52b1-4440-8523-4759498fb055\",", > " \"format\": \"RAW\",", > " \"generation\": 0,", > " \"image\": \"a9982674-a297-4a40-8649-5539b675cb86\",", > " \"lease\": {", > " \"offset\": 0,", > " \"owners\": [],", > " \"path\": > \"/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/adf90c1d-52b1-4440-8523-4759498fb055/images/a9982674-a297-4a40-8649-5539b675cb86/47fca7a4-f5bc-4873-8660-430449004b8f.lease\",", > " \"version\": null", > " },", > " \"legality\": \"LEGAL\",", > " \"mtime\": \"0\",", > " \"parent\": \"00000000-0000-0000-0000-000000000000\",", > " \"pool\": \"\",", > " \"status\": \"OK\",", > " \"truesize\": \"24576\",", > " \"type\": \"PREALLOCATED\",", > " \"uuid\": \"47fca7a4-f5bc-4873-8660-430449004b8f\",", > " \"voltype\": \"LEAF\"", > "}" > ] > > > > > When you call the vdsm API you should get a response. If you don't get > > any response maybe vdsm is blocked on inaccessible storage. > > I do get a response, but it never "Finishes", or at least according to the > login we have for testing this - we require both 'status=OK' and 'Updated:true'. > > > > > Or maybe the OVF volume was not created yet and you get an error > > that the volume does not exist. > > It does exist. We also get e.g.: status:OK. > > What is the meaning of: Updated: False, Last Updated: null? > > Is it wrong for checking these? I don't know, this info is generated by engine. Vdsm does not care about the content of the description field.

This is also not a public API, so if you check this you depend on engine private implementation details. Generally using HTTP APIs in vdsm for reading OVF data is abuse of private APIs that were added for engine and should be used only by engine. We are missing a real API for passing configuration updates from hosted engine to hosted engine agent running on the host. Instead we read OVF_STORE volume or engine configuration volume to extract data. Eyal, did we change anything related to updating OVF_STORE recently? > > > Perhaps it's just a bug in the test, and we should test something else > > > other than "Updated"? > > > > > > Or perhaps there was some other problem but I failed to find it? > > > > > > Thanks, > > > > > > > Build Number: 1655 > > > > Build Status: Still Failing > > > > Triggered By: Started by timer > > > > > > > > ------------------------------------- > > > > Changes Since Last Success: > > > > ------------------------------------- > > > > Changes for Build #1643 > > > > [Galit] Add missing repo which has collectd-write_syslog > > > > > > > > [Sandro Bonazzola] Revert "ovirt-release: run only on fc29 nodes" > > > > > > > > > > > > Changes for Build #1644 > > > > [Michal Skrivanek] make GLANCE failures fatal > > > > > > > > > > > > Changes for Build #1645 > > > > [Michal Skrivanek] use // in glance URL > > > > > > > > > > > > Changes for Build #1646 > > > > [Galit] Fix: No module named ost_utils.memoized > > > > > > > > > > > > Changes for Build #1647 > > > > [Martin Necas] ansible: fix deploy scripts to el8 > > > > > > > > [arachmani] Add arachman(a)redhat.com to jenkins recipient lists > > > > > > > > > > > > Changes for Build #1648 > > > > [Martin Necas] ansible: fix deploy scripts to el8 > > > > > > > > [Evgheni Dereveanchin] Do not run jenkins check-patch job on fc29 > > > > > > > > > > > > Changes for Build #1649 > > > > [Martin Necas] ansible: fix deploy scripts to el8 > > > > > > > > > > > > Changes for Build #1650 > > > > [Martin Necas] ansible: fix deploy scripts to el8 > > > > > > > > > > > > Changes for Build #1651 > > > > [Galit] Add centos8.2 official to templates > > > > > > > > > > > > Changes for Build #1652 > > > > [Evgeny Slutsky] he-basic-suite-master: Add Domain_name to the host when vm_run > > > > > > > > [Gal Ben Haim] Add gcc to fcraw > > > > > > > > [Sandro Bonazzola] pipelines: add ovirt-dependencies > > > > > > > > > > > > Changes for Build #1653 > > > > [Evgeny Slutsky] he-basic-suite-master: Add Domain_name to the host when vm_run > > > > > > > > > > > > Changes for Build #1654 > > > > [Galit] Add centos 8.2 image: el8.2-base > > > > > > > > [Evgheni Dereveanchin] do not expand values in run_oc_playbook > > > > > > > > > > > > Changes for Build #1655 > > > > [Galit] Add centos 8.2 image: el8.2-base > > > > > > > > > > > > > > > > > > > > ----------------- > > > > Failed Tests: > > > > ----------------- > > > > No tests ran. > > > > > > > > > > > > -- > > > Didi > > > > > > > > -- > Didi >