2018-09-04 10:13 GMT+02:00 Ehud Yonasi <eyonasi(a)redhat.com&gt;: > Hey, > > I see failures on hosted-engine and when i was looking at the > build-artifacts jobs of he-setup: > > http://jenkins.ovirt.org/job/ovirt-hosted-engine-setup_master_build-artif... > > I can see a lot of triggering of old patches, so as far as i understand > you restored that repository. > > Is the error below is the root cause of that recovery? > Would appreciate a quick answer because we need to merge a patch to fix > CQ failures because of that. > > Thanks in advance! > Not sure if the restoing caused the mentioned traceback. We got an accidental deletion of the last year of development on hosted engine setup master branch and we had to restore it causing a few hundreds of jenkins job to be triggered. > > Ehud Yonasi. > > > ---------- Forwarded message --------- > From: Yaniv Kaul <ykaul(a)redhat.com&gt; > Date: Tue, Sep 4, 2018 at 10:03 AM > Subject: Re: [rhev-devel] Error Message in HE master suite: "Operation > Failed". Fault detail is "[Cannot add Host. You are using an unmanaged > hosted engine VM .... > To: Galit Rosenthal <grosenth(a)redhat.com&gt; > Cc: rhev-devel <rhev-devel(a)redhat.com&gt; > > > Well, looking at engine.log[1], there's NPE around: > ERROR [org.ovirt.engine.core.vdsbroker.monitoring.FullListAdapter] > (EE-ManagedThreadFactory-engineScheduled-Thread-74) [57b53db9] Failed > during parsing configuration of VM b9c1bf34-c42f-4999-97d4-8d4895826abd > (<domain type='kvm' id='1'> > <name>HostedEngine</name> > ... > </domain> > ), error is: {}: java.lang.NullPointerException > at > org.ovirt.engine.core.vdsbroker.libvirt.VmConverter.extractCoreGeneralInfo(VmConverter.java:47) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.libvirt.VmConverter.convert(VmConverter.java:36) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.monitoring.FullListAdapter.extractCoreInfo(FullListAdapter.java:86) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.monitoring.FullListAdapter.lambda$dumpXmls$0(FullListAdapter.java:54) > [vdsbroker.jar:] > at > java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) > [rt.jar:1.8.0_181] > at > java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) > [rt.jar:1.8.0_181] > > > > Y. > [1] > https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... > > On Tue, Sep 4, 2018 at 9:21 AM, Galit Rosenthal <grosenth(a)redhat.com&gt; > wrote: > >> >> Hi, >> >> This is an example of a he basic suite master that fails on : >> >> Error Message: >> >> *00:26:36* Error while running thread >> >> *00:26:36* Traceback (most recent call last): >> >> *00:26:36* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue >> >> *00:26:36* queue.put({'return': func()}) >> >> *00:26:36* File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-master/ovirt-system-tests/he-basic-suite-master/test-scenarios/002_bootstrap.py", line 379, in _add_he_host_4 >> >> *00:26:36* deploy_hosted_engine=True, >> >> *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 11530, in add >> >> *00:26:36* return self._internal_add(host, headers, query, wait) >> >> *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add >> >> *00:26:36* return future.wait() if wait else future >> >> *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait >> >> *00:26:36* return self._code(response) >> >> *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback >> >> *00:26:36* self._check_fault(response) >> >> *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault >> >> *00:26:36* self._raise_error(response, body) >> >> *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error >> >> *00:26:36* raise error >> >> *00:26:36* Error: Fault reason is "Operation Failed". Fault detail is "[Cannot add Host. You are using an unmanaged hosted engine VM. Please add the first storage domain in order to start the hosted engine import process.]". HTTP response code is 409. >> >> >> >> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... >> >> Who can help me with this issue? >> >> Thanks, >> Galit >> >> -- >> >> GALIT ROSENTHAL >> >> SOFTWARE ENGINEER >> >> Red Hat >> >> <https://www.redhat.com/> >> >> galit(a)gmail.com T: 972-9-7692230 >> <https://red.ht/sig> >> > > -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=70...

2018-09-04 10:33 GMT+02:00 Ehud Yonasi <eyonasi(a)redhat.com&gt;: > Hey Sandro, > I see that CQ Master is trying to find the artifacts from that project: > > http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_chan... > > and fails cause it fails to build. > > I've sent a patch now to fix a package name that hopefully will fix it. > https://gerrit.ovirt.org/94144 > Fix is not correct. python2-pytest provides pytest. If the package is missing within OST, it's a OST bug. > > > > On Tue, Sep 4, 2018 at 11:20 AM Sandro Bonazzola <sbonazzo(a)redhat.com&gt; > wrote: > >> 2018-09-04 10:13 GMT+02:00 Ehud Yonasi <eyonasi(a)redhat.com&gt;: >> >>> Hey, >>> >>> I see failures on hosted-engine and when i was looking at the >>> build-artifacts jobs of he-setup: >>> >>> http://jenkins.ovirt.org/job/ovirt-hosted-engine-setup_master_build-artif... >>> >>> I can see a lot of triggering of old patches, so as far as i understand >>> you restored that repository. >>> >>> Is the error below is the root cause of that recovery? >>> Would appreciate a quick answer because we need to merge a patch to fix >>> CQ failures because of that. >>> >>> Thanks in advance! >>> >> >> Not sure if the restoing caused the mentioned traceback. >> We got an accidental deletion of the last year of development on hosted >> engine setup master branch and we had to restore it causing a few hundreds >> of jenkins job to be triggered. >> >> >> >> >> >>> >>> Ehud Yonasi. >>> >>> >>> ---------- Forwarded message --------- >>> From: Yaniv Kaul <ykaul(a)redhat.com&gt; >>> Date: Tue, Sep 4, 2018 at 10:03 AM >>> Subject: Re: [rhev-devel] Error Message in HE master suite: "Operation >>> Failed". Fault detail is "[Cannot add Host. You are using an unmanaged >>> hosted engine VM .... >>> To: Galit Rosenthal <grosenth(a)redhat.com&gt; >>> Cc: rhev-devel <rhev-devel(a)redhat.com&gt; >>> >>> >>> Well, looking at engine.log[1], there's NPE around: >>> ERROR [org.ovirt.engine.core.vdsbroker.monitoring.FullListAdapter] >>> (EE-ManagedThreadFactory-engineScheduled-Thread-74) [57b53db9] Failed >>> during parsing configuration of VM b9c1bf34-c42f-4999-97d4-8d4895826abd >>> (<domain type='kvm' id='1'> >>> <name>HostedEngine</name> >>> ... >>> </domain> >>> ), error is: {}: java.lang.NullPointerException >>> at >>> org.ovirt.engine.core.vdsbroker.libvirt.VmConverter.extractCoreGeneralInfo(VmConverter.java:47) >>> [vdsbroker.jar:] >>> at >>> org.ovirt.engine.core.vdsbroker.libvirt.VmConverter.convert(VmConverter.java:36) >>> [vdsbroker.jar:] >>> at >>> org.ovirt.engine.core.vdsbroker.monitoring.FullListAdapter.extractCoreInfo(FullListAdapter.java:86) >>> [vdsbroker.jar:] >>> at >>> org.ovirt.engine.core.vdsbroker.monitoring.FullListAdapter.lambda$dumpXmls$0(FullListAdapter.java:54) >>> [vdsbroker.jar:] >>> at >>> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) >>> [rt.jar:1.8.0_181] >>> at >>> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) >>> [rt.jar:1.8.0_181] >>> >>> >>> >>> Y. >>> [1] >>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... >>> >>> On Tue, Sep 4, 2018 at 9:21 AM, Galit Rosenthal <grosenth(a)redhat.com&gt; >>> wrote: >>> >>>> >>>> Hi, >>>> >>>> This is an example of a he basic suite master that fails on : >>>> >>>> Error Message: >>>> >>>> *00:26:36* Error while running thread >>>> >>>> *00:26:36* Traceback (most recent call last): >>>> >>>> *00:26:36* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue >>>> >>>> *00:26:36* queue.put({'return': func()}) >>>> >>>> *00:26:36* File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-master/ovirt-system-tests/he-basic-suite-master/test-scenarios/002_bootstrap.py", line 379, in _add_he_host_4 >>>> >>>> *00:26:36* deploy_hosted_engine=True, >>>> >>>> *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 11530, in add >>>> >>>> *00:26:36* return self._internal_add(host, headers, query, wait) >>>> >>>> *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add >>>> >>>> *00:26:36* return future.wait() if wait else future >>>> >>>> *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait >>>> >>>> *00:26:36* return self._code(response) >>>> >>>> *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback >>>> >>>> *00:26:36* self._check_fault(response) >>>> >>>> *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault >>>> >>>> *00:26:36* self._raise_error(response, body) >>>> >>>> *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error >>>> >>>> *00:26:36* raise error >>>> >>>> *00:26:36* Error: Fault reason is "Operation Failed". Fault detail is "[Cannot add Host. You are using an unmanaged hosted engine VM. Please add the first storage domain in order to start the hosted engine import process.]". HTTP response code is 409. >>>> >>>> >>>> >>>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... >>>> >>>> Who can help me with this issue? >>>> >>>> Thanks, >>>> Galit >>>> >>>> -- >>>> >>>> GALIT ROSENTHAL >>>> >>>> SOFTWARE ENGINEER >>>> >>>> Red Hat >>>> >>>> <https://www.redhat.com/> >>>> >>>> galit(a)gmail.com T: 972-9-7692230 >>>> <https://red.ht/sig> >>>> >>> >>> >> >> >> -- >> >> SANDRO BONAZZOLA >> >> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV >> >> Red Hat EMEA <https://www.redhat.com/> >> >> sbonazzo(a)redhat.com >> <https://red.ht/sig> >> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=70... >> > -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=70...

2018-09-04 10:48 GMT+02:00 Ehud Yonasi <eyonasi(a)redhat.com&gt;: > I can see that under the build-art job for fc28 it's fails under: > > *14:04:08* + yum-builddep ovirt-hosted-engine-setup.spec > > yum-builddep is used only if dnf is missing. The slave building this package is missing dnf within its chroot. Did something change in how slaves are provisioned? I'm adding dnf to the required packages.

> *14:04:08* *14:04:08* Yum-utils package has been deprecated, use dnf instead.*14:04:08* See 'man yum2dnf' for more information.*14:04:08* *14:04:08* *14:04:09* Getting requirements for ovirt-hosted-engine-setup.spec*14:04:09* --> Already installed : gettext-0.19.8.1-14.fc28.x86_64*14:04:09* --> Already installed : python2-devel-2.7.15-2.fc28.x86_64*14:04:09* Error: No Package found for pytest > > > I don't think that's related to OST. > > Link to job example: > http://jenkins.ovirt.org/job/ovirt-hosted-engine-setup_maste > r_build-artifacts-fc28-x86_64/250/console > > On Tue, Sep 4, 2018 at 11:40 AM Sandro Bonazzola <sbonazzo(a)redhat.com&gt; > wrote: > >> 2018-09-04 10:33 GMT+02:00 Ehud Yonasi <eyonasi(a)redhat.com&gt;: >> >>> Hey Sandro, >>> I see that CQ Master is trying to find the artifacts from that project: >>> http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovir >>> t-master_change-queue-tester/9583/console >>> >>> and fails cause it fails to build. >>> >>> I've sent a patch now to fix a package name that hopefully will fix it. >>> https://gerrit.ovirt.org/94144 >>> >> >> Fix is not correct. python2-pytest provides pytest. >> If the package is missing within OST, it's a OST bug. >> >> >> >> >>> >>> >>> >>> On Tue, Sep 4, 2018 at 11:20 AM Sandro Bonazzola <sbonazzo(a)redhat.com&gt; >>> wrote: >>> >>>> 2018-09-04 10:13 GMT+02:00 Ehud Yonasi <eyonasi(a)redhat.com&gt;: >>>> >>>>> Hey, >>>>> >>>>> I see failures on hosted-engine and when i was looking at the >>>>> build-artifacts jobs of he-setup: >>>>> http://jenkins.ovirt.org/job/ovirt-hosted-engine-setup_maste >>>>> r_build-artifacts-fc28-x86_64/ >>>>> >>>>> I can see a lot of triggering of old patches, so as far as i >>>>> understand you restored that repository. >>>>> >>>>> Is the error below is the root cause of that recovery? >>>>> Would appreciate a quick answer because we need to merge a patch to >>>>> fix CQ failures because of that. >>>>> >>>>> Thanks in advance! >>>>> >>>> >>>> Not sure if the restoing caused the mentioned traceback. >>>> We got an accidental deletion of the last year of development on >>>> hosted engine setup master branch and we had to restore it causing a few >>>> hundreds of jenkins job to be triggered. >>>> >>>> >>>> >>>> >>>> >>>>> >>>>> Ehud Yonasi. >>>>> >>>>> >>>>> ---------- Forwarded message --------- >>>>> From: Yaniv Kaul <ykaul(a)redhat.com&gt; >>>>> Date: Tue, Sep 4, 2018 at 10:03 AM >>>>> Subject: Re: [rhev-devel] Error Message in HE master suite: >>>>> "Operation Failed". Fault detail is "[Cannot add Host. You are using an >>>>> unmanaged hosted engine VM .... >>>>> To: Galit Rosenthal <grosenth(a)redhat.com&gt; >>>>> Cc: rhev-devel <rhev-devel(a)redhat.com&gt; >>>>> >>>>> >>>>> Well, looking at engine.log[1], there's NPE around: >>>>> ERROR [org.ovirt.engine.core.vdsbroker.monitoring.FullListAdapter] >>>>> (EE-ManagedThreadFactory-engineScheduled-Thread-74) [57b53db9] >>>>> Failed during parsing configuration of VM b9c1bf34-c42f-4999-97d4-8d4895826abd >>>>> (<domain type='kvm' id='1'> >>>>> <name>HostedEngine</name> >>>>> ... >>>>> </domain> >>>>> ), error is: {}: java.lang.NullPointerException >>>>> at org.ovirt.engine.core.vdsbroker.libvirt.VmConverter.extractC >>>>> oreGeneralInfo(VmConverter.java:47) [vdsbroker.jar:] >>>>> at org.ovirt.engine.core.vdsbroker.libvirt.VmConverter.convert(VmConverter.java:36) >>>>> [vdsbroker.jar:] >>>>> at org.ovirt.engine.core.vdsbroker.monitoring.FullListAdapter.e >>>>> xtractCoreInfo(FullListAdapter.java:86) [vdsbroker.jar:] >>>>> at org.ovirt.engine.core.vdsbroker.monitoring.FullListAdapter. >>>>> lambda$dumpXmls$0(FullListAdapter.java:54) [vdsbroker.jar:] >>>>> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) >>>>> [rt.jar:1.8.0_181] >>>>> at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) >>>>> [rt.jar:1.8.0_181] >>>>> >>>>> >>>>> >>>>> Y. >>>>> [1] https://jenkins.ovirt.org/view/oVirt%20system%20tests/jo >>>>> b/ovirt-system-tests_he-basic-suite-master/761/artifact/ >>>>> exported-artifacts/test_logs/he-basic-suite-master/post- >>>>> 002_bootstrap.py/lago-he-basic-suite-master-engine/_ >>>>> var_log/ovirt-engine/engine.log >>>>> >>>>> On Tue, Sep 4, 2018 at 9:21 AM, Galit Rosenthal <grosenth(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>> This is an example of a he basic suite master that fails on : >>>>>> >>>>>> Error Message: >>>>>> >>>>>> *00:26:36* Error while running thread >>>>>> >>>>>> *00:26:36* Traceback (most recent call last): >>>>>> >>>>>> *00:26:36* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue >>>>>> >>>>>> *00:26:36* queue.put({'return': func()}) >>>>>> >>>>>> *00:26:36* File "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-master/ovirt-system-tests/he-basic-suite-master/test-scenarios/002_bootstrap.py", line 379, in _add_he_host_4 >>>>>> >>>>>> *00:26:36* deploy_hosted_engine=True, >>>>>> >>>>>> *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 11530, in add >>>>>> >>>>>> *00:26:36* return self._internal_add(host, headers, query, wait) >>>>>> >>>>>> *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add >>>>>> >>>>>> *00:26:36* return future.wait() if wait else future >>>>>> >>>>>> *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait >>>>>> >>>>>> *00:26:36* return self._code(response) >>>>>> >>>>>> *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback >>>>>> >>>>>> *00:26:36* self._check_fault(response) >>>>>> >>>>>> *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault >>>>>> >>>>>> *00:26:36* self._raise_error(response, body) >>>>>> >>>>>> *00:26:36* File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error >>>>>> >>>>>> *00:26:36* raise error >>>>>> >>>>>> *00:26:36* Error: Fault reason is "Operation Failed". Fault detail is "[Cannot add Host. You are using an unmanaged hosted engine VM. Please add the first storage domain in order to start the hosted engine import process.]". HTTP response code is 409. >>>>>> >>>>>> >>>>>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ >>>>>> ovirt-system-tests_he-basic-suite-master/761/consoleFull >>>>>> >>>>>> Who can help me with this issue? >>>>>> >>>>>> Thanks, >>>>>> Galit >>>>>> >>>>>> -- >>>>>> >>>>>> GALIT ROSENTHAL >>>>>> >>>>>> SOFTWARE ENGINEER >>>>>> >>>>>> Red Hat >>>>>> >>>>>> <https://www.redhat.com/> >>>>>> >>>>>> galit(a)gmail.com T: 972-9-7692230 >>>>>> <https://red.ht/sig> >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> SANDRO BONAZZOLA >>>> >>>> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV >>>> >>>> Red Hat EMEA <https://www.redhat.com/> >>>> >>>> sbonazzo(a)redhat.com >>>> <https://red.ht/sig> >>>> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=70... >>>> >>> >> >> >> -- >> >> SANDRO BONAZZOLA >> >> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV >> >> Red Hat EMEA <https://www.redhat.com/> >> >> sbonazzo(a)redhat.com >> <https://red.ht/sig> >> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=70... >> > -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=70...