[ovirt-devel] Re: [oVirt Jenkins] ovirt-system-tests_he-basic-suite-master - Build # 1655 - Still Failing!

On Sun, Jun 28, 2020 at 1:37 PM Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > > On Sun, Jun 28, 2020 at 1:23 PM Yedidyah Bar David <didi(a)redhat.com&gt; wrote: > > > > On Sun, Jun 28, 2020 at 6:23 AM <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: > > > > > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ > > > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1655/ > > > > This fails for some time now. Checked last one ^^, and it failed in: > > > > https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/16... > > > > 2020-06-27 23:05:02,253-0400 INFO ansible task start {'status': 'OK', > > 'ansible_type': 'task', 'ansible_playbook': > > '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', > > 'ansible_task': 'ovirt.hosted_engine_setup : Check OVF_STORE volume > > status'} > > 2020-06-27 23:05:02,253-0400 DEBUG ansible on_any args TASK: > > ovirt.hosted_engine_setup : Check OVF_STORE volume status kwargs > > is_conditional:False > > 2020-06-27 23:05:02,254-0400 DEBUG ansible on_any args localhostTASK: > > ovirt.hosted_engine_setup : Check OVF_STORE volume status kwargs > > 2020-06-27 23:05:03,816-0400 DEBUG ansible on_any args > > <ansible.executor.task_result.TaskResult object at 0x7f454e3d1eb8> > > kwargs > > ... > > > > 2020-06-27 23:09:39,166-0400 DEBUG var changed: host "localhost" var > > "ovf_store_status" type "<class 'dict'>" value: "{ > > "changed": true, > > "failed": true, > > "msg": "All items completed", > > "results": [ > > { > > "ansible_loop_var": "item", > > "attempts": 12, > > > > Meaning, it ran 12 times the command: > > > > vdsm-client Volume getInfo > > storagepoolID=41c9fdea-b8e9-11ea-ae2a-5452c0a8c863 > > storagedomainID=10a69775-8fb6-437d-9e78-2ecfd77c0a45 > > imageID=c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9 > > volumeID=6b835f55-a512-4f83-9d25-f6837d8b5cb1 > > > > and never got a result with output including "Updated". Any idea? > > I don't know what is the meaning of "Updated". Sorry, meant Updated:true. This is a snippet from above ansible-create_target_vm log: "cmd": [ "vdsm-client", "Volume", "getInfo", "storagepoolID=41c9fdea-b8e9-11ea-ae2a-5452c0a8c863", "storagedomainID=10a69775-8fb6-437d-9e78-2ecfd77c0a45", "imageID=c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9", "volumeID=6b835f55-a512-4f83-9d25-f6837d8b5cb1" ], ... "stdout_lines": [ "{", " \"apparentsize\": \"134217728\",", " \"capacity\": \"134217728\",", " \"children\": [],", " \"ctime\": \"1593313323\",", " \"description\": \"{\\\"Updated\\\":false,\\\"Last Updated\\\":null,\\\"Storage Domains\\\":[{\\\"uuid\\\":\\\"10a69775-8fb6-437d-9e78-2ecfd77c0a45\\\"}],\\\"Disk Description\\\":\\\"OVF_STORE\\\"}\",", " \"disktype\": \"OVFS\",", " \"domain\": \"10a69775-8fb6-437d-9e78-2ecfd77c0a45\",", " \"format\": \"RAW\",", " \"generation\": 0,", " \"image\": \"c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9\",", " \"lease\": {", " \"offset\": 0,", " \"owners\": [],", " \"path\": \"/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/10a69775-8fb6-437d-9e78-2ecfd77c0a45/images/c2ad2065-1c8b-4ec1-afdd-f7cefc708cf9/6b835f55-a512-4f83-9d25-f6837d8b5cb1.lease\",", " \"version\": null", " },", " \"legality\": \"LEGAL\",", " \"mtime\": \"0\",", " \"parent\": \"00000000-0000-0000-0000-000000000000\",", " \"pool\": \"\",", " \"status\": \"OK\",", " \"truesize\": \"134217728\",", " \"type\": \"PREALLOCATED\",", " \"uuid\": \"6b835f55-a512-4f83-9d25-f6837d8b5cb1\",", " \"voltype\": \"LEAF\"", "}" ] As you can see, it includes: " \"description\": \"{\\\"Updated\\\":false, Compare with a successful run: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1632/ https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/16... "stdout_lines": [ "{", " \"apparentsize\": \"23552\",", " \"capacity\": \"134217728\",", " \"children\": [],", " \"ctime\": \"1591843833\",", " \"description\": \"{\\\"Updated\\\":true,\\\"Size\\\":23552,\\\"Last Updated\\\":\\\"Wed Jun 10 22:53:27 EDT 2020\\\",\\\"Storage Domains\\\":[{\\\"uuid\\\":\\\"adf90c1d-52b1-4440-8523-4759498fb055\\\"}],\\\"Disk Description\\\":\\\"OVF_STORE\\\"}\",", " \"disktype\": \"OVFS\",", " \"domain\": \"adf90c1d-52b1-4440-8523-4759498fb055\",", " \"format\": \"RAW\",", " \"generation\": 0,", " \"image\": \"a9982674-a297-4a40-8649-5539b675cb86\",", " \"lease\": {", " \"offset\": 0,", " \"owners\": [],", " \"path\": \"/rhev/data-center/mnt/lago-he-basic-suite-master-storage:_exports_nfs__he/adf90c1d-52b1-4440-8523-4759498fb055/images/a9982674-a297-4a40-8649-5539b675cb86/47fca7a4-f5bc-4873-8660-430449004b8f.lease\",", " \"version\": null", " },", " \"legality\": \"LEGAL\",", " \"mtime\": \"0\",", " \"parent\": \"00000000-0000-0000-0000-000000000000\",", " \"pool\": \"\",", " \"status\": \"OK\",", " \"truesize\": \"24576\",", " \"type\": \"PREALLOCATED\",", " \"uuid\": \"47fca7a4-f5bc-4873-8660-430449004b8f\",", " \"voltype\": \"LEAF\"", "}" ] > > When you call the vdsm API you should get a response. If you don't get > any response maybe vdsm is blocked on inaccessible storage. I do get a response, but it never "Finishes", or at least according to the login we have for testing this - we require both 'status=OK' and 'Updated:true'. > > Or maybe the OVF volume was not created yet and you get an error > that the volume does not exist. It does exist. We also get e.g.: status:OK. What is the meaning of: Updated: False, Last Updated: null? Is it wrong for checking these?

> > Perhaps it's just a bug in the test, and we should test something else > > other than "Updated"? > > > > Or perhaps there was some other problem but I failed to find it? > > > > Thanks, > > > > > Build Number: 1655 > > > Build Status: Still Failing > > > Triggered By: Started by timer > > > > > > ------------------------------------- > > > Changes Since Last Success: > > > ------------------------------------- > > > Changes for Build #1643 > > > [Galit] Add missing repo which has collectd-write_syslog > > > > > > [Sandro Bonazzola] Revert "ovirt-release: run only on fc29 nodes" > > > > > > > > > Changes for Build #1644 > > > [Michal Skrivanek] make GLANCE failures fatal > > > > > > > > > Changes for Build #1645 > > > [Michal Skrivanek] use // in glance URL > > > > > > > > > Changes for Build #1646 > > > [Galit] Fix: No module named ost_utils.memoized > > > > > > > > > Changes for Build #1647 > > > [Martin Necas] ansible: fix deploy scripts to el8 > > > > > > [arachmani] Add arachman(a)redhat.com to jenkins recipient lists > > > > > > > > > Changes for Build #1648 > > > [Martin Necas] ansible: fix deploy scripts to el8 > > > > > > [Evgheni Dereveanchin] Do not run jenkins check-patch job on fc29 > > > > > > > > > Changes for Build #1649 > > > [Martin Necas] ansible: fix deploy scripts to el8 > > > > > > > > > Changes for Build #1650 > > > [Martin Necas] ansible: fix deploy scripts to el8 > > > > > > > > > Changes for Build #1651 > > > [Galit] Add centos8.2 official to templates > > > > > > > > > Changes for Build #1652 > > > [Evgeny Slutsky] he-basic-suite-master: Add Domain_name to the host when vm_run > > > > > > [Gal Ben Haim] Add gcc to fcraw > > > > > > [Sandro Bonazzola] pipelines: add ovirt-dependencies > > > > > > > > > Changes for Build #1653 > > > [Evgeny Slutsky] he-basic-suite-master: Add Domain_name to the host when vm_run > > > > > > > > > Changes for Build #1654 > > > [Galit] Add centos 8.2 image: el8.2-base > > > > > > [Evgheni Dereveanchin] do not expand values in run_oc_playbook > > > > > > > > > Changes for Build #1655 > > > [Galit] Add centos 8.2 image: el8.2-base > > > > > > > > > > > > > > > ----------------- > > > Failed Tests: > > > ----------------- > > > No tests ran. > > > > > > > > -- > > Didi > > > -- Didi