Re: [ovirt-devel] [CQ ovirt master] [ovirt-engine] - not passing for 10 days

On 14 Nov 2018, at 10:50, Dominik Holler <dholler(a)redhat.com&gt; wrote: On Wed, 14 Nov 2018 09:27:39 +0100 Dominik Holler <dholler(a)redhat.com&gt; wrote: > On Tue, 13 Nov 2018 13:01:09 +0100 > Martin Perina <mperina(a)redhat.com&gt; wrote: > >> On Tue, Nov 13, 2018 at 12:49 PM Michal Skrivanek <mskrivan(a)redhat.com&gt; >> wrote: >> >>> >>> >>> On 13 Nov 2018, at 12:20, Dominik Holler <dholler(a)redhat.com&gt; wrote: >>> >>> On Tue, 13 Nov 2018 11:56:37 +0100 >>> Martin Perina <mperina(a)redhat.com&gt; wrote: >>> >>> On Tue, Nov 13, 2018 at 11:02 AM Dafna Ron <dron(a)redhat.com&gt; wrote: >>> >>> Martin? can you please look at the patch that Dominik sent? >>> We need to resolve this as we have not had an engine build for the last 11 >>> days >>> >>> >>> Yesterday I've merged Dominik's revert patch >>> https://gerrit.ovirt.org/95377 >>> which should switch cluster level back to 4.2. Below mentioned change >>> https://gerrit.ovirt.org/95310 is relevant only to cluster level 4.3, am I >>> right Michal? >>> >>> The build mentioned >>> >>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_cha... >>> is from yesterday. Are we sure that it was executed only after #95377 was >>> merged? I'd like to see the results from latest >>> >>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_cha... >>> but unfortunately it already waits more than an hour for available hosts >>> ... >>> >>> >>> >>> >>> >>> https://gerrit.ovirt.org/#/c/95283/ results in >>> >>> http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_... >>> which is used in >>> >>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... >>> results in run_vms succeeding. >>> >>> The next merged change >>> https://gerrit.ovirt.org/#/c/95310/ results in >>> >>> http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_... >>> which is used in >>> >>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... >>> results in run_vms failing with >>> 2018-11-12 17:35:10,109-05 INFO >>> [org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-1) >>> [6930b632-5593-4481-bf2a-a1d8b14a583a] Running command: RunVmOnceCommand >>> internal: false. Entities affected : ID: >>> d10aa133-b9b6-455d-8137-ab822d1c1971 Type: VMAction group RUN_VM with role >>> type USER >>> 2018-11-12 17:35:10,113-05 DEBUG >>> [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] >>> (default task-1) [6930b632-5593-4481-bf2a-a1d8b14a583a] method: >>> getVmManager, params: [d10aa133-b9b6-455d-8137-ab822d1c1971], timeElapsed: >>> 4ms >>> 2018-11-12 17:35:10,128-05 DEBUG >>> [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] >>> (default task-1) [6930b632-5593-4481-bf2a-a1d8b14a583a] method: >>> getAllForClusterWithStatus, params: [2ca9ccd8-61f0-470c-ba3f-07766202f260, >>> Up], timeElapsed: 7ms >>> 2018-11-12 17:35:10,129-05 INFO >>> [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-1) >>> [6930b632-5593-4481-bf2a-a1d8b14a583a] Candidate host >>> 'lago-basic-suite-master-host-1' ('282860ab-8873-4702-a2be-100a6da111af') >>> was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU-Level' >>> (correlation id: 6930b632-5593-4481-bf2a-a1d8b14a583a) >>> 2018-11-12 17:35:10,129-05 INFO >>> [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-1) >>> [6930b632-5593-4481-bf2a-a1d8b14a583a] Candidate host >>> 'lago-basic-suite-master-host-0' ('c48eca36-ea98-46b2-8473-f184833e68a8') >>> was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU-Level' >>> (correlation id: 6930b632-5593-4481-bf2a-a1d8b14a583a) >>> 2018-11-12 17:35:10,130-05 ERROR [org.ovirt.engine.core.bll.RunVmCommand] >>> (default task-1) [6930b632-5593-4481-bf2a-a1d8b14a583a] Can't find VDS to >>> run the VM 'd10aa133-b9b6-455d-8137-ab822d1c1971' on, so this VM will not >>> be run. >>> in >>> >>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... >>> >>> Is this helpful for you? >>> >>> >>> >>> actually, there ire two issues >>> 1) cluster is still 4.3 even after Martin’s revert. >>> >> >> https://gerrit.ovirt.org/#/c/95409/ should align cluster level with dc level >> > > This change aligns the cluster level, but > https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... > consuming build result from > https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_cha... > looks like that this does not solve the issue: > File "/home/jenkins/workspace/ovirt-system-tests_manual/ovirt-system-tests/basic-suite-master/test-scenarios/004_basic_sanity.py", line 698, in run_vms > api.vms.get(VM0_NAME).start(start_params) > File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 31193, in start > headers={"Correlation-Id":correlation_id} > File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request > persistent_auth=self.__persistent_auth > File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request > persistent_auth) > File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request > raise errors.RequestError(response_code, response_reason, response_body) > RequestError: > status: 400 > reason: Bad Request > > engine.log: > 2018-11-14 03:10:36,802-05 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-3) [99e282ea-577a-4dab-857b-285b1df5e6f6] Candidate host 'lago-basic-suite-master-host-0' ('4dbfb937-ac4b-4cef-8ae3-124944829add') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU-Level' (correlation id: 99e282ea-577a-4dab-857b-285b1df5e6f6) > 2018-11-14 03:10:36,802-05 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-3) [99e282ea-577a-4dab-857b-285b1df5e6f6] Candidate host 'lago-basic-suite-master-host-1' ('731e5055-706e-4310-a062-045e32ffbfeb') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU-Level' (correlation id: 99e282ea-577a-4dab-857b-285b1df5e6f6) > 2018-11-14 03:10:36,802-05 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (default task-3) [99e282ea-577a-4dab-857b-285b1df5e6f6] Can't find VDS to run the VM 'dc1e1e92-1e5c-415e-8ac2-b919017adf40' on, so this VM will not be run. > > https://gerrit.ovirt.org/#/c/95283/ results in http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_... which is used in https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... results in run_vms succeeding. The next merged change https://gerrit.ovirt.org/#/c/95310/ results in http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_... which is used in https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... results in run_vms failing with File "/home/jenkins/workspace/ovirt-system-tests_manual/ovirt-system-tests/basic-suite-master/test-scenarios/004_basic_sanity.py", line 698, in run_vms api.vms.get(VM0_NAME).start(start_params) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 31193, in start headers={"Correlation-Id":correlation_id} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 400 reason: Bad Request So even if the Cluster Level should be 4.2 now, still https://gerrit.ovirt.org/#/c/95310/ seems influence the behavior.

>> 2) the patch is wrong too, as in HandleVdsCpuFlagsOrClusterChangedCommand >>> it just goes ahead and sets the cluster cpu to whatever the host reported >>> regardless if it is valid or not. Steven, please fix that (line 96 in >>> backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/HandleVdsCpuFlagsOrClusterChangedCommand.java). >>> It needs to pass the validation or we need some other solution. >>> 3) regardless, we should make 4.3 work too , I tried to play with it a bit >>> in https://gerrit.ovirt.org/#/c/95407/, let’s see… >>> >>> Thanks, >>> michal >>> >>> >>> >>> On Mon, Nov 12, 2018 at 3:58 PM Dominik Holler <dholler(a)redhat.com&gt; wrote: >>> >>> On Mon, 12 Nov 2018 13:45:54 +0100 >>> Martin Perina <mperina(a)redhat.com&gt; wrote: >>> >>> On Mon, Nov 12, 2018 at 12:58 PM Dominik Holler <dholler(a)redhat.com&gt; >>> >>> wrote: >>> >>> >>> On Mon, 12 Nov 2018 12:29:17 +0100 >>> Martin Perina <mperina(a)redhat.com&gt; wrote: >>> >>> On Mon, Nov 12, 2018 at 12:20 PM Dafna Ron <dron(a)redhat.com&gt; wrote: >>> >>> There are currently two issues failing ovirt-engine on CQ ovirt >>> >>> master: >>> >>> >>> 1. edit vm pool is causing failure in different tests. it has a >>> >>> patch >>> >>> *waiting >>> >>> to be merged*: https://gerrit.ovirt.org/#/c/95354/ >>> >>> >>> Merged >>> >>> >>> 2. we have a failure in upgrade suite as well to run vm but this >>> >>> seems >>> >>> to >>> >>> be related to the tests as well: >>> 2018-11-12 05:41:07,831-05 WARN >>> [org.ovirt.engine.core.bll.validator.VirtIoRngValidator] >>> >>> (default >>> >>> task-1) >>> >>> [] Random number source URANDOM is not supported in cluster >>> >>> 'test-cluster' >>> >>> compatibility version 4.0. >>> >>> here is the full error from the upgrade suite failure in run vm: >>> https://pastebin.com/XLHtWGGx >>> >>> Here is the latest failure: >>> >>> >>> >>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_cha... >>> >>> >>> >>> I will try to take a look later today >>> >>> >>> I have the idea that this might be related to >>> https://gerrit.ovirt.org/#/c/95377/ , and I check in >>> >>> >>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... >>> >>> >>> , but I have to stop now, if not solved I can go on later today. >>> >>> >>> OK, both CI and above manual OST job went fine, so I've just merged the >>> revert patch. I will take a look at it later in detail, we should >>> >>> really be >>> >>> testing 4.3 on master and not 4.2 >>> >>> >>> Ack. >>> >>> Now >>> >>> >>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_cha... >>> is failing on >>> File >>> >>> "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/004_basic_sanity.py", >>> line 698, in run_vms >>> api.vms.get(VM0_NAME).start(start_params) >>> status: 400 >>> reason: Bad Request >>> >>> 2018-11-12 10:06:30,722-05 INFO >>> [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-3) >>> [b8d11cb0-5be9-4b7e-b45a-c95fa1f18681] Candidate host >>> 'lago-basic-suite-master-host-1' ('dbfe1b0c-f940-4dba-8fb1-0cfe5ca7ddfc') >>> was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU-Level' >>> (correlation id: b8d11cb0-5be9-4b7e-b45a-c95fa1f18681) >>> 2018-11-12 10:06:30,722-05 INFO >>> [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-3) >>> [b8d11cb0-5be9-4b7e-b45a-c95fa1f18681] Candidate host >>> 'lago-basic-suite-master-host-0' ('e83a63ca-381e-40db-acb2-65a3e7953e11') >>> was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU-Level' >>> (correlation id: b8d11cb0-5be9-4b7e-b45a-c95fa1f18681) >>> 2018-11-12 10:06:30,723-05 ERROR [org.ovirt.engine.core.bll.RunVmCommand] >>> (default task-3) [b8d11cb0-5be9-4b7e-b45a-c95fa1f18681] Can't find VDS to >>> run the VM '57a66eff-8cbf-4643-b045-43d4dda80c66' on, so this VM will not >>> be run. >>> >>> Is this related to >>> https://gerrit.ovirt.org/#/c/95310/ >>> ? >>> >>> >>> >>> >>> >>> >>> Thanks, >>> Dafna >>> >>> >>> >>> >>> On Mon, Nov 12, 2018 at 9:23 AM Dominik Holler < >>> >>> dholler(a)redhat.com&gt; >>> >>> wrote: >>> >>> >>> On Sun, 11 Nov 2018 19:04:40 +0200 >>> Dan Kenigsberg <danken(a)redhat.com&gt; wrote: >>> >>> On Sun, Nov 11, 2018 at 5:27 PM Eyal Edri <eedri(a)redhat.com&gt; >>> >>> wrote: >>> >>> >>> >>> >>> On Sun, Nov 11, 2018 at 5:24 PM Eyal Edri <eedri(a)redhat.com&gt; >>> >>> >>> wrote: >>> >>> >>> >>> >>> On Sun, Nov 11, 2018 at 5:20 PM Dan Kenigsberg < >>> >>> danken(a)redhat.com&gt; >>> >>> wrote: >>> >>> >>> On Sun, Nov 11, 2018 at 4:36 PM Ehud Yonasi < >>> >>> eyonasi(a)redhat.com&gt; >>> >>> >>> wrote: >>> >>> >>> Hey, >>> I've seen that CQ Master is not passing ovirt-engine for >>> >>> 10 >>> >>> days >>> >>> and fails on test suite called restore_vm0_networking >>> >>> here's a snap error regarding it: >>> >>> https://pastebin.com/7msEYqKT >>> >>> Link to a sample job with the error: >>> >>> >>> >>> >>> >>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11113/artif... >>> >>> >>> >>> I cannot follow this link because I'm 4 minutes too late >>> >>> jenkins.ovirt.org uses an invalid security certificate. >>> >>> The >>> >>> certificate expired on November 11, 2018, 5:13:25 PM >>> >>> GMT+2. The >>> >>> current time is November 11, 2018, 5:17 PM. >>> >>> >>> >>> Yes, we're looking into that issue now. >>> >>> >>> >>> Fixed, you should be able to access it now. >>> >>> >>> OST fails during restore_vm0_networking in line 101 of >>> 004_basic_sanity.py while comparing >>> vm_service.get().status == state >>> >>> It seems that instead of reporting back the VM status, Engine >>> >>> set >>> >>> garbage >>> >>> "The response content type 'text/html; charset=iso-8859-1' >>> >>> isn't the >>> >>> expected XML" >>> >>> >>> The relevant line in >>> >>> >>> >>> >>> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11113/arti... >>> >>> seems to be >>> 192.168.201.1 - - [11/Nov/2018:04:27:43 -0500] "GET >>> /ovirt-engine/api/vms/26088164-d1a0-4254-a377-5d3c242c8105 >>> >>> HTTP/1.1" >>> >>> 503 299 >>> >>> and I guess the 503 error message is sent in HTML instead of XML. >>> >>> If I run manually >>> https://gerrit.ovirt.org/#/c/95354/ >>> with latest build of engine-master >>> >>> >>> >>> >>> http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_... >>> >>> basic suite seems to be happy: >>> https://jenkins.ovirt.org/view/oVirt system >>> tests/job/ovirt-system-tests_manual/3484/ >>> >>> >>> I do not know what could cause that, and engine.log does not >>> >>> mention >>> >>> it. But it seems like a problem in engine API hence +Martin >>> >>> Perina >>> >>> and >>> >>> +Ondra Machacek . >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> Can some1 have a look at it and help to resolve the >>> >>> issue? >>> >>> >>> >>> _______________________________________________ >>> Infra mailing list -- infra(a)ovirt.org >>> To unsubscribe send an email to infra-leave(a)ovirt.org >>> Privacy Statement: >>> >>> https://www.ovirt.org/site/privacy-policy/ >>> >>> oVirt Code of Conduct: >>> >>> https://www.ovirt.org/community/about/community-guidelines/ >>> >>> List Archives: >>> >>> >>> >>> >>> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/ZQAYWTLZJKG... >>> >>> >>> _______________________________________________ >>> Devel mailing list -- devel(a)ovirt.org >>> To unsubscribe send an email to devel-leave(a)ovirt.org >>> Privacy Statement: >>> >>> https://www.ovirt.org/site/privacy-policy/ >>> >>> oVirt Code of Conduct: >>> >>> https://www.ovirt.org/community/about/community-guidelines/ >>> >>> List Archives: >>> >>> >>> >>> >>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/R5LOJH73XCL... >>> >>> >>> >>> >>> >>> -- >>> >>> Eyal edri >>> >>> >>> MANAGER >>> >>> RHV/CNV DevOps >>> >>> EMEA VIRTUALIZATION R&D >>> >>> >>> Red Hat EMEA >>> >>> TRIED. TESTED. TRUSTED. >>> phone: +972-9-7692018 >>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>> >>> >>> >>> >>> -- >>> >>> Eyal edri >>> >>> >>> MANAGER >>> >>> RHV/CNV DevOps >>> >>> EMEA VIRTUALIZATION R&D >>> >>> >>> Red Hat EMEA >>> >>> TRIED. TESTED. TRUSTED. >>> phone: +972-9-7692018 >>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>> >>> _______________________________________________ >>> Devel mailing list -- devel(a)ovirt.org >>> To unsubscribe send an email to devel-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: >>> >>> https://www.ovirt.org/community/about/community-guidelines/ >>> >>> List Archives: >>> >>> >>> >>> >>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/DA6Q5RE5JO3... >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >> >