[ovirt-devel] Re: [CQ ovirt master] [ovirt-engine] - not passing for 10 days

On 13 Nov 2018, at 12:20, Dominik Holler <dholler(a)redhat.com&gt; wrote: On Tue, 13 Nov 2018 11:56:37 +0100 Martin Perina <mperina(a)redhat.com&gt; wrote: On Tue, Nov 13, 2018 at 11:02 AM Dafna Ron <dron(a)redhat.com&gt; wrote: Martin? can you please look at the patch that Dominik sent? We need to resolve this as we have not had an engine build for the last 11 days Yesterday I've merged Dominik's revert patch https://gerrit.ovirt.org/95377 which should switch cluster level back to 4.2. Below mentioned change https://gerrit.ovirt.org/95310 is relevant only to cluster level 4.3, am I right Michal? The build mentioned https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_cha... is from yesterday. Are we sure that it was executed only after #95377 was merged? I'd like to see the results from latest https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_cha... but unfortunately it already waits more than an hour for available hosts ... https://gerrit.ovirt.org/#/c/95283/ results in http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_... which is used in https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... results in run_vms succeeding. The next merged change https://gerrit.ovirt.org/#/c/95310/ results in http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_... which is used in https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... results in run_vms failing with 2018-11-12 17:35:10,109-05 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-1) [6930b632-5593-4481-bf2a-a1d8b14a583a] Running command: RunVmOnceCommand internal: false. Entities affected : ID: d10aa133-b9b6-455d-8137-ab822d1c1971 Type: VMAction group RUN_VM with role type USER 2018-11-12 17:35:10,113-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-1) [6930b632-5593-4481-bf2a-a1d8b14a583a] method: getVmManager, params: [d10aa133-b9b6-455d-8137-ab822d1c1971], timeElapsed: 4ms 2018-11-12 17:35:10,128-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-1) [6930b632-5593-4481-bf2a-a1d8b14a583a] method: getAllForClusterWithStatus, params: [2ca9ccd8-61f0-470c-ba3f-07766202f260, Up], timeElapsed: 7ms 2018-11-12 17:35:10,129-05 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-1) [6930b632-5593-4481-bf2a-a1d8b14a583a] Candidate host 'lago-basic-suite-master-host-1' ('282860ab-8873-4702-a2be-100a6da111af') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU-Level' (correlation id: 6930b632-5593-4481-bf2a-a1d8b14a583a) 2018-11-12 17:35:10,129-05 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-1) [6930b632-5593-4481-bf2a-a1d8b14a583a] Candidate host 'lago-basic-suite-master-host-0' ('c48eca36-ea98-46b2-8473-f184833e68a8') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU-Level' (correlation id: 6930b632-5593-4481-bf2a-a1d8b14a583a) 2018-11-12 17:35:10,130-05 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (default task-1) [6930b632-5593-4481-bf2a-a1d8b14a583a] Can't find VDS to run the VM 'd10aa133-b9b6-455d-8137-ab822d1c1971' on, so this VM will not be run. in https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... Is this helpful for you? actually, there ire two issues 1) cluster is still 4.3 even after Martin’s revert.

it just goes ahead and sets the cluster cpu to whatever the host reported regardless if it is valid or not. Steven, please fix that (line 96 in backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/HandleVdsCpuFlagsOrClusterChangedCommand.java). It needs to pass the validation or we need some other solution. 3) regardless, we should make 4.3 work too , I tried to play with it a bit in https://gerrit.ovirt.org/#/c/95407/, let’s see… Thanks, michal On Mon, Nov 12, 2018 at 3:58 PM Dominik Holler <dholler(a)redhat.com&gt; wrote: On Mon, 12 Nov 2018 13:45:54 +0100 Martin Perina <mperina(a)redhat.com&gt; wrote: On Mon, Nov 12, 2018 at 12:58 PM Dominik Holler <dholler(a)redhat.com&gt; wrote: On Mon, 12 Nov 2018 12:29:17 +0100 Martin Perina <mperina(a)redhat.com&gt; wrote: On Mon, Nov 12, 2018 at 12:20 PM Dafna Ron <dron(a)redhat.com&gt; wrote: There are currently two issues failing ovirt-engine on CQ ovirt master: 1. edit vm pool is causing failure in different tests. it has a patch *waiting to be merged*: https://gerrit.ovirt.org/#/c/95354/ Merged 2. we have a failure in upgrade suite as well to run vm but this seems to be related to the tests as well: 2018-11-12 05:41:07,831-05 WARN [org.ovirt.engine.core.bll.validator.VirtIoRngValidator] (default task-1) [] Random number source URANDOM is not supported in cluster 'test-cluster' compatibility version 4.0. here is the full error from the upgrade suite failure in run vm: https://pastebin.com/XLHtWGGx Here is the latest failure: https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_cha... I will try to take a look later today I have the idea that this might be related to https://gerrit.ovirt.org/#/c/95377/ , and I check in https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... , but I have to stop now, if not solved I can go on later today. OK, both CI and above manual OST job went fine, so I've just merged the revert patch. I will take a look at it later in detail, we should really be testing 4.3 on master and not 4.2 Ack. Now https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_cha... is failing on File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/004_basic_sanity.py", line 698, in run_vms api.vms.get(VM0_NAME).start(start_params) status: 400 reason: Bad Request 2018-11-12 10:06:30,722-05 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-3) [b8d11cb0-5be9-4b7e-b45a-c95fa1f18681] Candidate host 'lago-basic-suite-master-host-1' ('dbfe1b0c-f940-4dba-8fb1-0cfe5ca7ddfc') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU-Level' (correlation id: b8d11cb0-5be9-4b7e-b45a-c95fa1f18681) 2018-11-12 10:06:30,722-05 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-3) [b8d11cb0-5be9-4b7e-b45a-c95fa1f18681] Candidate host 'lago-basic-suite-master-host-0' ('e83a63ca-381e-40db-acb2-65a3e7953e11') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU-Level' (correlation id: b8d11cb0-5be9-4b7e-b45a-c95fa1f18681) 2018-11-12 10:06:30,723-05 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (default task-3) [b8d11cb0-5be9-4b7e-b45a-c95fa1f18681] Can't find VDS to run the VM '57a66eff-8cbf-4643-b045-43d4dda80c66' on, so this VM will not be run. Is this related to https://gerrit.ovirt.org/#/c/95310/ ? Thanks, Dafna On Mon, Nov 12, 2018 at 9:23 AM Dominik Holler < dholler(a)redhat.com&gt; wrote: On Sun, 11 Nov 2018 19:04:40 +0200 Dan Kenigsberg <danken(a)redhat.com&gt; wrote: On Sun, Nov 11, 2018 at 5:27 PM Eyal Edri <eedri(a)redhat.com&gt; wrote: On Sun, Nov 11, 2018 at 5:24 PM Eyal Edri <eedri(a)redhat.com&gt; wrote: On Sun, Nov 11, 2018 at 5:20 PM Dan Kenigsberg < danken(a)redhat.com&gt; wrote: On Sun, Nov 11, 2018 at 4:36 PM Ehud Yonasi < eyonasi(a)redhat.com&gt; wrote: Hey, I've seen that CQ Master is not passing ovirt-engine for 10 days and fails on test suite called restore_vm0_networking here's a snap error regarding it: https://pastebin.com/7msEYqKT Link to a sample job with the error: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11113/artif... I cannot follow this link because I'm 4 minutes too late jenkins.ovirt.org uses an invalid security certificate. The certificate expired on November 11, 2018, 5:13:25 PM GMT+2. The current time is November 11, 2018, 5:17 PM. Yes, we're looking into that issue now. Fixed, you should be able to access it now. OST fails during restore_vm0_networking in line 101 of 004_basic_sanity.py while comparing vm_service.get().status == state It seems that instead of reporting back the VM status, Engine set garbage "The response content type 'text/html; charset=iso-8859-1' isn't the expected XML" The relevant line in https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11113/arti... seems to be 192.168.201.1 - - [11/Nov/2018:04:27:43 -0500] "GET /ovirt-engine/api/vms/26088164-d1a0-4254-a377-5d3c242c8105 HTTP/1.1" 503 299 and I guess the 503 error message is sent in HTML instead of XML. If I run manually https://gerrit.ovirt.org/#/c/95354/ with latest build of engine-master http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_... basic suite seems to be happy: https://jenkins.ovirt.org/view/oVirt system tests/job/ovirt-system-tests_manual/3484/ I do not know what could cause that, and engine.log does not mention it. But it seems like a problem in engine API hence +Martin Perina and +Ondra Machacek . Can some1 have a look at it and help to resolve the issue? _______________________________________________ Infra mailing list -- infra(a)ovirt.org To unsubscribe send an email to infra-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/ZQAYWTLZJKG... _______________________________________________ Devel mailing list -- devel(a)ovirt.org To unsubscribe send an email to devel-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/R5LOJH73XCL... -- Eyal edri MANAGER RHV/CNV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA TRIED. TESTED. TRUSTED. phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ) -- Eyal edri MANAGER RHV/CNV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA TRIED. TESTED. TRUSTED. phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ) _______________________________________________ Devel mailing list -- devel(a)ovirt.org To unsubscribe send an email to devel-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/DA6Q5RE5JO3...