[ovirt-devel] Re: [CQ ovirt master] [ovirt-engine] - not passing for 10 days

On Wed, Nov 14, 2018 at 5:07 PM Dan Kenigsberg <danken(a)redhat.com&gt; wrote: > > On Wed, Nov 14, 2018 at 12:42 PM Dominik Holler <dholler(a)redhat.com&gt; wrote: > > > > On Wed, 14 Nov 2018 11:24:10 +0100 > > Michal Skrivanek <mskrivan(a)redhat.com&gt; wrote: > > > > > > On 14 Nov 2018, at 10:50, Dominik Holler <dholler(a)redhat.com&gt; wrote: > > > > > > > > On Wed, 14 Nov 2018 09:27:39 +0100 > > > > Dominik Holler <dholler(a)redhat.com&gt; wrote: > > > > > > > >> On Tue, 13 Nov 2018 13:01:09 +0100 > > > >> Martin Perina <mperina(a)redhat.com&gt; wrote: > > > >> > > > >>> On Tue, Nov 13, 2018 at 12:49 PM Michal Skrivanek < mskrivan(a)redhat.com&gt; > > > >>> wrote: > > > >>> > > > >>>> > > > >>>> > > > >>>> On 13 Nov 2018, at 12:20, Dominik Holler <dholler(a)redhat.com&gt; wrote: > > > >>>> > > > >>>> On Tue, 13 Nov 2018 11:56:37 +0100 > > > >>>> Martin Perina <mperina(a)redhat.com&gt; wrote: > > > >>>> > > > >>>> On Tue, Nov 13, 2018 at 11:02 AM Dafna Ron <dron(a)redhat.com&gt; wrote: > > > >>>> > > > >>>> Martin? can you please look at the patch that Dominik sent? > > > >>>> We need to resolve this as we have not had an engine build for the last 11 > > > >>>> days > > > >>>> > > > >>>> > > > >>>> Yesterday I've merged Dominik's revert patch > > > >>>> https://gerrit.ovirt.org/95377 > > > >>>> which should switch cluster level back to 4.2. Below mentioned change > > > >>>> https://gerrit.ovirt.org/95310 is relevant only to cluster level 4.3, am I > > > >>>> right Michal? > > > >>>> > > > >>>> The build mentioned > > > >>>> > > > >>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_cha... > > > >>>> is from yesterday. Are we sure that it was executed only after #95377 was > > > >>>> merged? I'd like to see the results from latest > > > >>>> > > > >>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_cha... > > > >>>> but unfortunately it already waits more than an hour for available hosts > > > >>>> ... > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> https://gerrit.ovirt.org/#/c/95283/ results in > > > >>>> > > > >>>> http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_... > > > >>>> which is used in > > > >>>> > > > >>>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... > > > >>>> results in run_vms succeeding. > > > >>>> > > > >>>> The next merged change > > > >>>> https://gerrit.ovirt.org/#/c/95310/ results in > > > >>>> > > > >>>> http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_... > > > >>>> which is used in > > > >>>> > > > >>>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... > > > >>>> results in run_vms failing with > > > >>>> 2018-11-12 17:35:10,109-05 INFO > > > >>>> [org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-1) > > > >>>> [6930b632-5593-4481-bf2a-a1d8b14a583a] Running command: RunVmOnceCommand > > > >>>> internal: false. Entities affected : ID: > > > >>>> d10aa133-b9b6-455d-8137-ab822d1c1971 Type: VMAction group RUN_VM with role > > > >>>> type USER > > > >>>> 2018-11-12 17:35:10,113-05 DEBUG > > > >>>> [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] > > > >>>> (default task-1) [6930b632-5593-4481-bf2a-a1d8b14a583a] method: > > > >>>> getVmManager, params: [d10aa133-b9b6-455d-8137-ab822d1c1971], timeElapsed: > > > >>>> 4ms > > > >>>> 2018-11-12 17:35:10,128-05 DEBUG > > > >>>> [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] > > > >>>> (default task-1) [6930b632-5593-4481-bf2a-a1d8b14a583a] method: > > > >>>> getAllForClusterWithStatus, params: [2ca9ccd8-61f0-470c-ba3f-07766202f260, > > > >>>> Up], timeElapsed: 7ms > > > >>>> 2018-11-12 17:35:10,129-05 INFO > > > >>>> [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-1) > > > >>>> [6930b632-5593-4481-bf2a-a1d8b14a583a] Candidate host > > > >>>> 'lago-basic-suite-master-host-1' ('282860ab-8873-4702-a2be-100a6da111af') > > > >>>> was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU-Level' > > > >>>> (correlation id: 6930b632-5593-4481-bf2a-a1d8b14a583a) > > > >>>> 2018-11-12 17:35:10,129-05 INFO > > > >>>> [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-1) > > > >>>> [6930b632-5593-4481-bf2a-a1d8b14a583a] Candidate host > > > >>>> 'lago-basic-suite-master-host-0' ('c48eca36-ea98-46b2-8473-f184833e68a8') > > > >>>> was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU-Level' > > > >>>> (correlation id: 6930b632-5593-4481-bf2a-a1d8b14a583a) > > > >>>> 2018-11-12 17:35:10,130-05 ERROR [org.ovirt.engine.core.bll.RunVmCommand] > > > >>>> (default task-1) [6930b632-5593-4481-bf2a-a1d8b14a583a] Can't find VDS to > > > >>>> run the VM 'd10aa133-b9b6-455d-8137-ab822d1c1971' on, so this VM will not > > > >>>> be run. > > > >>>> in > > > >>>> > > > >>>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... > > > >>>> > > > >>>> Is this helpful for you? > > > >>>> > > > >>>> > > > >>>> > > > >>>> actually, there ire two issues > > > >>>> 1) cluster is still 4.3 even after Martin’s revert. > > > >>>> > > > >>> > > > >>> https://gerrit.ovirt.org/#/c/95409/ should align cluster level with dc level > > > >>> > > > >> > > > >> This change aligns the cluster level, but > > > >> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... > > > >> consuming build result from > > > >> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_cha... > > > >> looks like that this does not solve the issue: > > > >> File "/home/jenkins/workspace/ovirt-system-tests_manual/ovirt-system-tests/basic-suite-master/test-scenarios/004_basic_sanity.py", line 698, in run_vms > > > >> api.vms.get(VM0_NAME).start(start_params) > > > >> File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 31193, in start > > > >> headers={"Correlation-Id":correlation_id} > > > >> File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request > > > >> persistent_auth=self.__persistent_auth > > > >> File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request > > > >> persistent_auth) > > > >> File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request > > > >> raise errors.RequestError(response_code, response_reason, response_body) > > > >> RequestError: > > > >> status: 400 > > > >> reason: Bad Request > > > >> > > > >> engine.log: > > > >> 2018-11-14 03:10:36,802-05 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-3) [99e282ea-577a-4dab-857b-285b1df5e6f6] Candidate host 'lago-basic-suite-master-host-0' ('4dbfb937-ac4b-4cef-8ae3-124944829add') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU-Level' (correlation id: 99e282ea-577a-4dab-857b-285b1df5e6f6) > > > >> 2018-11-14 03:10:36,802-05 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-3) [99e282ea-577a-4dab-857b-285b1df5e6f6] Candidate host 'lago-basic-suite-master-host-1' ('731e5055-706e-4310-a062-045e32ffbfeb') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU-Level' (correlation id: 99e282ea-577a-4dab-857b-285b1df5e6f6) > > > >> 2018-11-14 03:10:36,802-05 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (default task-3) [99e282ea-577a-4dab-857b-285b1df5e6f6] Can't find VDS to run the VM 'dc1e1e92-1e5c-415e-8ac2-b919017adf40' on, so this VM will not be run. > > > >> > > > >> > > > > > > > > > > > > https://gerrit.ovirt.org/#/c/95283/ results in > > > > http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_... > > > > which is used in > > > > https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... > > > > results in run_vms succeeding. > > > > > > > > The next merged change > > > > https://gerrit.ovirt.org/#/c/95310/ results in > > > > http://jenkins.ovirt.org/job/ovirt-engine_master_build-artifacts-el7-x86_... > > > > which is used in > > > > https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... > > > > results in run_vms failing with > > > > File "/home/jenkins/workspace/ovirt-system-tests_manual/ovirt-system-tests/basic-suite-master/test-scenarios/004_basic_sanity.py", line 698, in run_vms > > > > api.vms.get(VM0_NAME).start(start_params) > > > > File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 31193, in start > > > > headers={"Correlation-Id":correlation_id} > > > > File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request > > > > persistent_auth=self.__persistent_auth > > > > File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request > > > > persistent_auth) > > > > File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request > > > > raise errors.RequestError(response_code, response_reason, response_body) > > > > RequestError: > > > > status: 400 > > > > reason: Bad Request > > > > > > > > > > > > So even if the Cluster Level should be 4.2 now, > > > > still https://gerrit.ovirt.org/#/c/95310/ seems influence the behavior. > > > > > > I really do not see how it can affect 4.2. > > > > Me neither. > > > > > Are you sure the cluster is really 4.2? Sadly it’s not being logged at all > > > > screenshot from local execution https://imgur.com/a/yiWBw3c > > > > > But if it really seem to matter (and since it needs a fix anyway for 4.3) feel free to revert it of course > > > > > > > I will post a revert change and check if this changes the behavior. > > Dominik, thanks for the research and for Martin's and your > reverts/fixes. Finally Engine passes OST > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11153/ > and QE can expect a build tomorrow, after 2 weeks of droughts. unfortunately, the drought continues.

Barrak tells me that something is broken in the nightly cron job copying the the tested repo onto the master-snapshot one.

+Edri: please make it a priority to have it fixed.