will trigger the job again On Sun, Jul 1, 2018 at 3:27 PM oVirt Jenkins <jenkins(a)ovirt.org&gt; wrote: > Change 92610,3 (ovirt-engine) is probably the reason behind recent system > test > failures in the "ovirt-4.2" change queue and needs to be fixed. > > This change had been removed from the testing queue. Artifacts build from > this > change will not be released until it is fixed. > > For further details about the change see: > https://gerrit.ovirt.org/#/c/92610/3 > > For failed test results see: > http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2534/ > _______________________________________________ > Infra mailing list -- infra(a)ovirt.org > To unsubscribe send an email to infra-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MVIJ6JKUKZS... >

Due to a bug in the retention policy, some packages were deleted from tested repo. We're rollbacking from nightly snapshot for all versions, and will update asap. On Sun, Jul 1, 2018 at 4:37 PM Ehud Yonasi <eyonasi(a)redhat.com&gt; wrote: > failed on installing packages. restoring tested repo > > > On Sun, Jul 1, 2018 at 4:34 PM Ehud Yonasi <eyonasi(a)redhat.com&gt; wrote: > >> will trigger the job again >> >> >> On Sun, Jul 1, 2018 at 3:27 PM oVirt Jenkins <jenkins(a)ovirt.org&gt; wrote: >> >>> Change 92610,3 (ovirt-engine) is probably the reason behind recent >>> system test >>> failures in the "ovirt-4.2" change queue and needs to be fixed. >>> >>> This change had been removed from the testing queue. Artifacts build >>> from this >>> change will not be released until it is fixed. >>> >>> For further details about the change see: >>> https://gerrit.ovirt.org/#/c/92610/3 >>> >>> For failed test results see: >>> http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2534/ >>> _______________________________________________ >>> Infra mailing list -- infra(a)ovirt.org >>> To unsubscribe send an email to infra-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MVIJ6JKUKZS... >>> >> _______________________________________________ Infra mailing list -- infra(a)ovirt.org To unsubscribe send an email to infra-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/FOCZJSME4AY...

Please prepare an RCA how we hit again a bug in the script if it was verified before deployment. On Sun, Jul 1, 2018 at 4:40 PM Ehud Yonasi <eyonasi(a)redhat.com&gt; wrote: > Due to a bug in the retention policy, some packages were deleted from > tested repo. > We're rollbacking from nightly snapshot for all versions, and will update > asap. > > On Sun, Jul 1, 2018 at 4:37 PM Ehud Yonasi <eyonasi(a)redhat.com&gt; wrote: > >> failed on installing packages. restoring tested repo >> >> >> On Sun, Jul 1, 2018 at 4:34 PM Ehud Yonasi <eyonasi(a)redhat.com&gt; wrote: >> >>> will trigger the job again >>> >>> >>> On Sun, Jul 1, 2018 at 3:27 PM oVirt Jenkins <jenkins(a)ovirt.org&gt; wrote: >>> >>>> Change 92610,3 (ovirt-engine) is probably the reason behind recent >>>> system test >>>> failures in the "ovirt-4.2" change queue and needs to be fixed. >>>> >>>> This change had been removed from the testing queue. Artifacts build >>>> from this >>>> change will not be released until it is fixed. >>>> >>>> For further details about the change see: >>>> https://gerrit.ovirt.org/#/c/92610/3 >>>> >>>> For failed test results see: >>>> http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2534/ >>>> _______________________________________________ >>>> Infra mailing list -- infra(a)ovirt.org >>>> To unsubscribe send an email to infra-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: >>>> https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: >>>> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MVIJ6JKUKZS... >>>> >>> _______________________________________________ > Infra mailing list -- infra(a)ovirt.org > To unsubscribe send an email to infra-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/FOCZJSME4AY... > -- Eyal edri MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ) _______________________________________________ Infra mailing list -- infra(a)ovirt.org To unsubscribe send an email to infra-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/3CGDFQDQMSW...

Seems that the root cause is the fact that CQ tester is using stdci_runner now and stdci_runner is not taking 'extra_sources' into account.

Meanwhile, I've reverted the change so we can re-examine stdci_runner.groovy https://gerrit.ovirt.org/#/c/92714/ On Sun, Jul 1, 2018 at 5:59 PM Daniel Belenky <dbelenky(a)redhat.com&gt; wrote: > Hi, > > There's something that seems odd to me. @Barak Korren > <bkorren(a)redhat.com&gt; @Gal Ben Haim <gbenhaim(a)redhat.com&gt; would > appreciate you to have a look as well. > > According to Alona, the patch that introduced the error is: > https://gerrit.ovirt.org/#/c/92009/ > > It was *tested* (and *deployed*) in https://jenkins.ovirt.org/ > job/ovirt-master_change-queue-tester/8372/ > > But, from yum.log on engine's machine, it seems that an older version of > ovirt-engine was used (not the version that was built with the patch that > introduced the error - hence why it was deployed without failing the tests) > > From https://jenkins.ovirt.org/job/ovirt-master_change- > queue-tester/8372/artifact/exported-artifacts/basic-suit. > el7.x86_64/test_logs/basic-suite-master/post-099_aaa- > ldap.py/lago-basic-suite-master-engine/_var_log/yum.log > > Jun 27 09:47:30 Installed: ovirt-engine-4.3.0-0.0.master.20180627094550.gita6b3aa4.el7.noarch > > Where Alona's patch built ovirt-engine-4.3.0-0.0.master.20180627130008.gita3072d3.el7.noarch.rpm > > As you can see here in engine's log, *a6b3aa4* is older than *a3072d3* > 11aeec678f2 webadmin: Add 'Multi Queues Enabled' to vm dialog > 920bf6a75a1 packaging: Add python-netaddr requirement > *a3072d3*ceba engine: Add "MultiQueuesEnabled" to ovf reader and writer > 4893678a272 engine: Update libvirtVmXml to consider > vmBase.multiQueuesEnabled attribute > 072bad8f4d3 engine: Introduce algorithm for calculating how many queues > asign per vnic > c4dd484c4ee engine: Add multiQueuesEnabled to VmBase > *a6b3aa4*5d18 packaging: setup: Return value 0 when using --help > > > And now, with Alona's patch that supposed to fix the issue (verified via > manual OST), it seems that again, the CQ run is not using Alona's version > of engine but an older one. > > The patch that fixes the issue is https://gerrit.ovirt.org/#/c/92656/ > which was tested here https://jenkins.ovirt.org/job/ > ovirt-master_change-queue-tester/8511/ (but again, the engine was > actually installed from an older version that doesn't include the fix - > hence it failed). > > > Could someone have a look too? If it's indeed the case, it's a severe bug > form our side that we have to address ASAP. > > On Sun, Jul 1, 2018 at 5:15 PM Eyal Edri <eedri(a)redhat.com&gt; wrote: > >> Please prepare an RCA how we hit again a bug in the script if it was >> verified before deployment. >> >> On Sun, Jul 1, 2018 at 4:40 PM Ehud Yonasi <eyonasi(a)redhat.com&gt; wrote: >> >>> Due to a bug in the retention policy, some packages were deleted from >>> tested repo. >>> We're rollbacking from nightly snapshot for all versions, and will >>> update asap. >>> >>> On Sun, Jul 1, 2018 at 4:37 PM Ehud Yonasi <eyonasi(a)redhat.com&gt; wrote: >>> >>>> failed on installing packages. restoring tested repo >>>> >>>> >>>> On Sun, Jul 1, 2018 at 4:34 PM Ehud Yonasi <eyonasi(a)redhat.com&gt; wrote: >>>> >>>>> will trigger the job again >>>>> >>>>> >>>>> On Sun, Jul 1, 2018 at 3:27 PM oVirt Jenkins <jenkins(a)ovirt.org&gt; >>>>> wrote: >>>>> >>>>>> Change 92610,3 (ovirt-engine) is probably the reason behind recent >>>>>> system test >>>>>> failures in the "ovirt-4.2" change queue and needs to be fixed. >>>>>> >>>>>> This change had been removed from the testing queue. Artifacts build >>>>>> from this >>>>>> change will not be released until it is fixed. >>>>>> >>>>>> For further details about the change see: >>>>>> https://gerrit.ovirt.org/#/c/92610/3 >>>>>> >>>>>> For failed test results see: >>>>>> http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2534/ >>>>>> _______________________________________________ >>>>>> Infra mailing list -- infra(a)ovirt.org >>>>>> To unsubscribe send an email to infra-leave(a)ovirt.org >>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>> oVirt Code of Conduct: https://www.ovirt.org/ >>>>>> community/about/community-guidelines/ >>>>>> List Archives: https://lists.ovirt.org/ >>>>>> archives/list/infra(a)ovirt.org/message/MVIJ6JKUKZSW3XKJTAJNIHPLVIVGWB >>>>>> N2/ >>>>>> >>>>> _______________________________________________ >>> Infra mailing list -- infra(a)ovirt.org >>> To unsubscribe send an email to infra-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community- >>> guidelines/ >>> List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ >>> message/FOCZJSME4AYMZEDMC4GZ63UNVJ5WRSYT/ >>> >> >> >> -- >> >> Eyal edri >> >> >> MANAGER >> >> RHV DevOps >> >> EMEA VIRTUALIZATION R&D >> >> >> Red Hat EMEA <https://www.redhat.com/> >> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >> <https://redhat.com/trusted> >> phone: +972-9-7692018 >> irc: eedri (on #tlv #rhev-dev #rhev-integ) >> _______________________________________________ >> Infra mailing list -- infra(a)ovirt.org >> To unsubscribe send an email to infra-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community- >> guidelines/ >> List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ >> message/3CGDFQDQMSWAWXUBNPAH4VXBSYGVWCYE/ >> > > > -- > > DANIEL BELENKY > > RHV DEVOPS > -- DANIEL BELENKY RHV DEVOPS

For some reason, after merging the revert patch, JJB deploy didn't update the master change-queue tester job so it still used my patch. We need to check why.

This should finally run tests with Alona's fix and hopefully, if there's no another bug, deploy the batch.

On Sun, Jul 1, 2018 at 8:24 PM Barak Korren <bkorren(a)redhat.com&gt; wrote: > > > On 1 July 2018 at 18:32, Daniel Belenky <dbelenky(a)redhat.com&gt; wrote: > >> Seems that the root cause is the fact that CQ tester is using >> stdci_runner now and stdci_runner is not taking 'extra_sources' into >> account. >> > > I guess I should've been more careful in the review, and you should be > more careful refactoring! > > I was wondering today how come I don't remember seeing the mirror > management features from CQ make their way into STDCI. Need to make sure we > don't "accidentally" drop those too. > > >> >> Meanwhile, I've reverted the change so we can re-examine >> stdci_runner.groovy >> >> https://gerrit.ovirt.org/#/c/92714/ >> >> On Sun, Jul 1, 2018 at 5:59 PM Daniel Belenky <dbelenky(a)redhat.com&gt; >> wrote: >> >>> Hi, >>> >>> There's something that seems odd to me. @Barak Korren >>> <bkorren(a)redhat.com&gt; @Gal Ben Haim <gbenhaim(a)redhat.com&gt; would >>> appreciate you to have a look as well. >>> >>> According to Alona, the patch that introduced the error is: >>> https://gerrit.ovirt.org/#/c/92009/ >>> >>> It was *tested* (and *deployed*) in https://jenkins.ovirt.org/ >>> job/ovirt-master_change-queue-tester/8372/ >>> >>> But, from yum.log on engine's machine, it seems that an older version >>> of ovirt-engine was used (not the version that was built with the patch >>> that introduced the error - hence why it was deployed without failing the >>> tests) >>> >>> From https://jenkins.ovirt.org/job/ovirt-master_change- >>> queue-tester/8372/artifact/exported-artifacts/basic-suit. >>> el7.x86_64/test_logs/basic-suite-master/post-099_aaa- >>> ldap.py/lago-basic-suite-master-engine/_var_log/yum.log >>> >>> Jun 27 09:47:30 Installed: ovirt-engine-4.3.0-0.0.master.20180627094550.gita6b3aa4.el7.noarch >>> >>> Where Alona's patch built ovirt-engine-4.3.0-0.0.master.20180627130008.gita3072d3.el7.noarch.rpm >>> >>> As you can see here in engine's log, *a6b3aa4* is older than *a3072d3* >>> 11aeec678f2 webadmin: Add 'Multi Queues Enabled' to vm dialog >>> 920bf6a75a1 packaging: Add python-netaddr requirement >>> *a3072d3*ceba engine: Add "MultiQueuesEnabled" to ovf reader and writer >>> 4893678a272 engine: Update libvirtVmXml to consider >>> vmBase.multiQueuesEnabled attribute >>> 072bad8f4d3 engine: Introduce algorithm for calculating how many queues >>> asign per vnic >>> c4dd484c4ee engine: Add multiQueuesEnabled to VmBase >>> *a6b3aa4*5d18 packaging: setup: Return value 0 when using --help >>> >>> >>> And now, with Alona's patch that supposed to fix the issue (verified >>> via manual OST), it seems that again, the CQ run is not using Alona's >>> version of engine but an older one. >>> >>> The patch that fixes the issue is https://gerrit.ovirt.org/#/c/92656/ >>> which was tested here https://jenkins.ovirt.org/job/ >>> ovirt-master_change-queue-tester/8511/ (but again, the engine was >>> actually installed from an older version that doesn't include the fix - >>> hence it failed). >>> >>> >>> Could someone have a look too? If it's indeed the case, it's a severe >>> bug form our side that we have to address ASAP. >>> >>> On Sun, Jul 1, 2018 at 5:15 PM Eyal Edri <eedri(a)redhat.com&gt; wrote: >>> >>>> Please prepare an RCA how we hit again a bug in the script if it was >>>> verified before deployment. >>>> >>>> On Sun, Jul 1, 2018 at 4:40 PM Ehud Yonasi <eyonasi(a)redhat.com&gt; wrote: >>>> >>>>> Due to a bug in the retention policy, some packages were deleted from >>>>> tested repo. >>>>> We're rollbacking from nightly snapshot for all versions, and will >>>>> update asap. >>>>> >>>>> On Sun, Jul 1, 2018 at 4:37 PM Ehud Yonasi <eyonasi(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> failed on installing packages. restoring tested repo >>>>>> >>>>>> >>>>>> On Sun, Jul 1, 2018 at 4:34 PM Ehud Yonasi <eyonasi(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>>> will trigger the job again >>>>>>> >>>>>>> >>>>>>> On Sun, Jul 1, 2018 at 3:27 PM oVirt Jenkins <jenkins(a)ovirt.org&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> Change 92610,3 (ovirt-engine) is probably the reason behind recent >>>>>>>> system test >>>>>>>> failures in the "ovirt-4.2" change queue and needs to be fixed. >>>>>>>> >>>>>>>> This change had been removed from the testing queue. Artifacts >>>>>>>> build from this >>>>>>>> change will not be released until it is fixed. >>>>>>>> >>>>>>>> For further details about the change see: >>>>>>>> https://gerrit.ovirt.org/#/c/92610/3 >>>>>>>> >>>>>>>> For failed test results see: >>>>>>>> http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2534/ >>>>>>>> _______________________________________________ >>>>>>>> Infra mailing list -- infra(a)ovirt.org >>>>>>>> To unsubscribe send an email to infra-leave(a)ovirt.org >>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>> oVirt Code of Conduct: https://www.ovirt.org/ >>>>>>>> community/about/community-guidelines/ >>>>>>>> List Archives: https://lists.ovirt.org/ >>>>>>>> archives/list/infra(a)ovirt.org/message/ >>>>>>>> MVIJ6JKUKZSW3XKJTAJNIHPLVIVGWBN2/ >>>>>>>> >>>>>>> _______________________________________________ >>>>> Infra mailing list -- infra(a)ovirt.org >>>>> To unsubscribe send an email to infra-leave(a)ovirt.org >>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>> oVirt Code of Conduct: https://www.ovirt.org/ >>>>> community/about/community-guidelines/ >>>>> List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ >>>>> message/FOCZJSME4AYMZEDMC4GZ63UNVJ5WRSYT/ >>>>> >>>> >>>> >>>> -- >>>> >>>> Eyal edri >>>> >>>> >>>> MANAGER >>>> >>>> RHV DevOps >>>> >>>> EMEA VIRTUALIZATION R&D >>>> >>>> >>>> Red Hat EMEA <https://www.redhat.com/> >>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>> <https://redhat.com/trusted> >>>> phone: +972-9-7692018 >>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>> _______________________________________________ >>>> Infra mailing list -- infra(a)ovirt.org >>>> To unsubscribe send an email to infra-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: https://www.ovirt.org/ >>>> community/about/community-guidelines/ >>>> List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ >>>> message/3CGDFQDQMSWAWXUBNPAH4VXBSYGVWCYE/ >>>> >>> >>> >>> -- >>> >>> DANIEL BELENKY >>> >>> RHV DEVOPS >>> >> >> >> -- >> >> DANIEL BELENKY >> >> RHV DEVOPS >> > > > > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted > -- DANIEL BELENKY RHV DEVOPS