[CQ]: 92610,3 (ovirt-engine) failed "ovirt-4.2" system tests
Change 92610,3 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/92610/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2534/
will trigger the job again On Sun, Jul 1, 2018 at 3:27 PM oVirt Jenkins <jenkins@ovirt.org> wrote:
Change 92610,3 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed.
This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed.
For further details about the change see: https://gerrit.ovirt.org/#/c/92610/3
For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2534/ _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MVIJ6JKUKZSW3X...
failed on installing packages. restoring tested repo On Sun, Jul 1, 2018 at 4:34 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
will trigger the job again
On Sun, Jul 1, 2018 at 3:27 PM oVirt Jenkins <jenkins@ovirt.org> wrote:
Change 92610,3 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed.
This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed.
For further details about the change see: https://gerrit.ovirt.org/#/c/92610/3
For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2534/ _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MVIJ6JKUKZSW3X...
Due to a bug in the retention policy, some packages were deleted from tested repo. We're rollbacking from nightly snapshot for all versions, and will update asap. On Sun, Jul 1, 2018 at 4:37 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
failed on installing packages. restoring tested repo
On Sun, Jul 1, 2018 at 4:34 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
will trigger the job again
On Sun, Jul 1, 2018 at 3:27 PM oVirt Jenkins <jenkins@ovirt.org> wrote:
Change 92610,3 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed.
This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed.
For further details about the change see: https://gerrit.ovirt.org/#/c/92610/3
For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2534/ _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MVIJ6JKUKZSW3X...
Restore from master snapshot is done now. I've re-built https://jenkins.ovirt.org/job/ovirt-system-tests_manual/2878/ which should verify Alona's fix. Once things will clear out with this issue, I'll try to figure out how one 'bad' change affected all projects. On Sun, Jul 1, 2018 at 4:40 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
Due to a bug in the retention policy, some packages were deleted from tested repo. We're rollbacking from nightly snapshot for all versions, and will update asap.
On Sun, Jul 1, 2018 at 4:37 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
failed on installing packages. restoring tested repo
On Sun, Jul 1, 2018 at 4:34 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
will trigger the job again
On Sun, Jul 1, 2018 at 3:27 PM oVirt Jenkins <jenkins@ovirt.org> wrote:
Change 92610,3 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed.
This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed.
For further details about the change see: https://gerrit.ovirt.org/#/c/92610/3
For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2534/ _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MVIJ6JKUKZSW3X...
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/FOCZJSME4AYMZE...
-- DANIEL BELENKY RHV DEVOPS
Please prepare an RCA how we hit again a bug in the script if it was verified before deployment. On Sun, Jul 1, 2018 at 4:40 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
Due to a bug in the retention policy, some packages were deleted from tested repo. We're rollbacking from nightly snapshot for all versions, and will update asap.
On Sun, Jul 1, 2018 at 4:37 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
failed on installing packages. restoring tested repo
On Sun, Jul 1, 2018 at 4:34 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
will trigger the job again
On Sun, Jul 1, 2018 at 3:27 PM oVirt Jenkins <jenkins@ovirt.org> wrote:
Change 92610,3 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed.
This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed.
For further details about the change see: https://gerrit.ovirt.org/#/c/92610/3
For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2534/ _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MVIJ6JKUKZSW3X...
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/FOCZJSME4AYMZE...
-- Eyal edri MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
Hi, There's something that seems odd to me. @Barak Korren <bkorren@redhat.com> @Gal Ben Haim <gbenhaim@redhat.com> would appreciate you to have a look as well. According to Alona, the patch that introduced the error is: https://gerrit.ovirt.org/#/c/92009/ It was *tested* (and *deployed*) in https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8372/ But, from yum.log on engine's machine, it seems that an older version of ovirt-engine was used (not the version that was built with the patch that introduced the error - hence why it was deployed without failing the tests) From https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8372/artifact... Jun 27 09:47:30 Installed: ovirt-engine-4.3.0-0.0.master.20180627094550.gita6b3aa4.el7.noarch Where Alona's patch built ovirt-engine-4.3.0-0.0.master.20180627130008.gita3072d3.el7.noarch.rpm As you can see here in engine's log, *a6b3aa4* is older than *a3072d3* 11aeec678f2 webadmin: Add 'Multi Queues Enabled' to vm dialog 920bf6a75a1 packaging: Add python-netaddr requirement *a3072d3*ceba engine: Add "MultiQueuesEnabled" to ovf reader and writer 4893678a272 engine: Update libvirtVmXml to consider vmBase.multiQueuesEnabled attribute 072bad8f4d3 engine: Introduce algorithm for calculating how many queues asign per vnic c4dd484c4ee engine: Add multiQueuesEnabled to VmBase *a6b3aa4*5d18 packaging: setup: Return value 0 when using --help And now, with Alona's patch that supposed to fix the issue (verified via manual OST), it seems that again, the CQ run is not using Alona's version of engine but an older one. The patch that fixes the issue is https://gerrit.ovirt.org/#/c/92656/ which was tested here https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8511/ (but again, the engine was actually installed from an older version that doesn't include the fix - hence it failed). Could someone have a look too? If it's indeed the case, it's a severe bug form our side that we have to address ASAP. On Sun, Jul 1, 2018 at 5:15 PM Eyal Edri <eedri@redhat.com> wrote:
Please prepare an RCA how we hit again a bug in the script if it was verified before deployment.
On Sun, Jul 1, 2018 at 4:40 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
Due to a bug in the retention policy, some packages were deleted from tested repo. We're rollbacking from nightly snapshot for all versions, and will update asap.
On Sun, Jul 1, 2018 at 4:37 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
failed on installing packages. restoring tested repo
On Sun, Jul 1, 2018 at 4:34 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
will trigger the job again
On Sun, Jul 1, 2018 at 3:27 PM oVirt Jenkins <jenkins@ovirt.org> wrote:
Change 92610,3 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed.
This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed.
For further details about the change see: https://gerrit.ovirt.org/#/c/92610/3
For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2534/ _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MVIJ6JKUKZSW3X...
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/FOCZJSME4AYMZE...
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ) _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/3CGDFQDQMSWAWX...
-- DANIEL BELENKY RHV DEVOPS
Seems that the root cause is the fact that CQ tester is using stdci_runner now and stdci_runner is not taking 'extra_sources' into account. Meanwhile, I've reverted the change so we can re-examine stdci_runner.groovy https://gerrit.ovirt.org/#/c/92714/ On Sun, Jul 1, 2018 at 5:59 PM Daniel Belenky <dbelenky@redhat.com> wrote:
Hi,
There's something that seems odd to me. @Barak Korren <bkorren@redhat.com> @Gal Ben Haim <gbenhaim@redhat.com> would appreciate you to have a look as well.
According to Alona, the patch that introduced the error is: https://gerrit.ovirt.org/#/c/92009/
It was *tested* (and *deployed*) in https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8372/
But, from yum.log on engine's machine, it seems that an older version of ovirt-engine was used (not the version that was built with the patch that introduced the error - hence why it was deployed without failing the tests)
From https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8372/artifact...
Jun 27 09:47:30 Installed: ovirt-engine-4.3.0-0.0.master.20180627094550.gita6b3aa4.el7.noarch
Where Alona's patch built ovirt-engine-4.3.0-0.0.master.20180627130008.gita3072d3.el7.noarch.rpm
As you can see here in engine's log, *a6b3aa4* is older than *a3072d3* 11aeec678f2 webadmin: Add 'Multi Queues Enabled' to vm dialog 920bf6a75a1 packaging: Add python-netaddr requirement *a3072d3*ceba engine: Add "MultiQueuesEnabled" to ovf reader and writer 4893678a272 engine: Update libvirtVmXml to consider vmBase.multiQueuesEnabled attribute 072bad8f4d3 engine: Introduce algorithm for calculating how many queues asign per vnic c4dd484c4ee engine: Add multiQueuesEnabled to VmBase *a6b3aa4*5d18 packaging: setup: Return value 0 when using --help
And now, with Alona's patch that supposed to fix the issue (verified via manual OST), it seems that again, the CQ run is not using Alona's version of engine but an older one.
The patch that fixes the issue is https://gerrit.ovirt.org/#/c/92656/ which was tested here https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8511/ (but again, the engine was actually installed from an older version that doesn't include the fix - hence it failed).
Could someone have a look too? If it's indeed the case, it's a severe bug form our side that we have to address ASAP.
On Sun, Jul 1, 2018 at 5:15 PM Eyal Edri <eedri@redhat.com> wrote:
Please prepare an RCA how we hit again a bug in the script if it was verified before deployment.
On Sun, Jul 1, 2018 at 4:40 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
Due to a bug in the retention policy, some packages were deleted from tested repo. We're rollbacking from nightly snapshot for all versions, and will update asap.
On Sun, Jul 1, 2018 at 4:37 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
failed on installing packages. restoring tested repo
On Sun, Jul 1, 2018 at 4:34 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
will trigger the job again
On Sun, Jul 1, 2018 at 3:27 PM oVirt Jenkins <jenkins@ovirt.org> wrote:
Change 92610,3 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-4.2" change queue and needs to be fixed.
This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed.
For further details about the change see: https://gerrit.ovirt.org/#/c/92610/3
For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2534/ _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MVIJ6JKUKZSW3X...
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/FOCZJSME4AYMZE...
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ) _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/3CGDFQDQMSWAWX...
--
DANIEL BELENKY
RHV DEVOPS
-- DANIEL BELENKY RHV DEVOPS
On 1 July 2018 at 18:32, Daniel Belenky <dbelenky@redhat.com> wrote:
Seems that the root cause is the fact that CQ tester is using stdci_runner now and stdci_runner is not taking 'extra_sources' into account.
I guess I should've been more careful in the review, and you should be more careful refactoring! I was wondering today how come I don't remember seeing the mirror management features from CQ make their way into STDCI. Need to make sure we don't "accidentally" drop those too.
Meanwhile, I've reverted the change so we can re-examine stdci_runner.groovy
https://gerrit.ovirt.org/#/c/92714/
On Sun, Jul 1, 2018 at 5:59 PM Daniel Belenky <dbelenky@redhat.com> wrote:
Hi,
There's something that seems odd to me. @Barak Korren <bkorren@redhat.com> @Gal Ben Haim <gbenhaim@redhat.com> would appreciate you to have a look as well.
According to Alona, the patch that introduced the error is: https://gerrit.ovirt.org/#/c/92009/
It was *tested* (and *deployed*) in https://jenkins.ovirt.org/ job/ovirt-master_change-queue-tester/8372/
But, from yum.log on engine's machine, it seems that an older version of ovirt-engine was used (not the version that was built with the patch that introduced the error - hence why it was deployed without failing the tests)
From https://jenkins.ovirt.org/job/ovirt-master_change- queue-tester/8372/artifact/exported-artifacts/basic-suit. el7.x86_64/test_logs/basic-suite-master/post-099_aaa- ldap.py/lago-basic-suite-master-engine/_var_log/yum.log
Jun 27 09:47:30 Installed: ovirt-engine-4.3.0-0.0.master.20180627094550.gita6b3aa4.el7.noarch
Where Alona's patch built ovirt-engine-4.3.0-0.0.master.20180627130008.gita3072d3.el7.noarch.rpm
As you can see here in engine's log, *a6b3aa4* is older than *a3072d3* 11aeec678f2 webadmin: Add 'Multi Queues Enabled' to vm dialog 920bf6a75a1 packaging: Add python-netaddr requirement *a3072d3*ceba engine: Add "MultiQueuesEnabled" to ovf reader and writer 4893678a272 engine: Update libvirtVmXml to consider vmBase.multiQueuesEnabled attribute 072bad8f4d3 engine: Introduce algorithm for calculating how many queues asign per vnic c4dd484c4ee engine: Add multiQueuesEnabled to VmBase *a6b3aa4*5d18 packaging: setup: Return value 0 when using --help
And now, with Alona's patch that supposed to fix the issue (verified via manual OST), it seems that again, the CQ run is not using Alona's version of engine but an older one.
The patch that fixes the issue is https://gerrit.ovirt.org/#/c/92656/ which was tested here https://jenkins.ovirt.org/job/ ovirt-master_change-queue-tester/8511/ (but again, the engine was actually installed from an older version that doesn't include the fix - hence it failed).
Could someone have a look too? If it's indeed the case, it's a severe bug form our side that we have to address ASAP.
On Sun, Jul 1, 2018 at 5:15 PM Eyal Edri <eedri@redhat.com> wrote:
Please prepare an RCA how we hit again a bug in the script if it was verified before deployment.
On Sun, Jul 1, 2018 at 4:40 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
Due to a bug in the retention policy, some packages were deleted from tested repo. We're rollbacking from nightly snapshot for all versions, and will update asap.
On Sun, Jul 1, 2018 at 4:37 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
failed on installing packages. restoring tested repo
On Sun, Jul 1, 2018 at 4:34 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
will trigger the job again
On Sun, Jul 1, 2018 at 3:27 PM oVirt Jenkins <jenkins@ovirt.org> wrote:
> Change 92610,3 (ovirt-engine) is probably the reason behind recent > system test > failures in the "ovirt-4.2" change queue and needs to be fixed. > > This change had been removed from the testing queue. Artifacts build > from this > change will not be released until it is fixed. > > For further details about the change see: > https://gerrit.ovirt.org/#/c/92610/3 > > For failed test results see: > http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2534/ > _______________________________________________ > Infra mailing list -- infra@ovirt.org > To unsubscribe send an email to infra-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/ > community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/ > archives/list/infra@ovirt.org/message/MVIJ6JKUKZSW3XKJTAJNIHPLVIVGWB > N2/ > _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community- guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ message/FOCZJSME4AYMZEDMC4GZ63UNVJ5WRSYT/
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ) _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community- guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ message/3CGDFQDQMSWAWXUBNPAH4VXBSYGVWCYE/
--
DANIEL BELENKY
RHV DEVOPS
--
DANIEL BELENKY
RHV DEVOPS
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
For some reason, after merging the revert patch, JJB deploy didn't update the master change-queue tester job so it still used my patch. We need to check why. Meanwhile, I've modified the tester manually so it will use master HEAD now. This should finally run tests with Alona's fix and hopefully, if there's no another bug, deploy the batch. On Sun, Jul 1, 2018 at 8:24 PM Barak Korren <bkorren@redhat.com> wrote:
On 1 July 2018 at 18:32, Daniel Belenky <dbelenky@redhat.com> wrote:
Seems that the root cause is the fact that CQ tester is using stdci_runner now and stdci_runner is not taking 'extra_sources' into account.
I guess I should've been more careful in the review, and you should be more careful refactoring!
I was wondering today how come I don't remember seeing the mirror management features from CQ make their way into STDCI. Need to make sure we don't "accidentally" drop those too.
Meanwhile, I've reverted the change so we can re-examine stdci_runner.groovy
https://gerrit.ovirt.org/#/c/92714/
On Sun, Jul 1, 2018 at 5:59 PM Daniel Belenky <dbelenky@redhat.com> wrote:
Hi,
There's something that seems odd to me. @Barak Korren <bkorren@redhat.com> @Gal Ben Haim <gbenhaim@redhat.com> would appreciate you to have a look as well.
According to Alona, the patch that introduced the error is: https://gerrit.ovirt.org/#/c/92009/
It was *tested* (and *deployed*) in https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8372/
But, from yum.log on engine's machine, it seems that an older version of ovirt-engine was used (not the version that was built with the patch that introduced the error - hence why it was deployed without failing the tests)
From https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8372/artifact...
Jun 27 09:47:30 Installed: ovirt-engine-4.3.0-0.0.master.20180627094550.gita6b3aa4.el7.noarch
Where Alona's patch built ovirt-engine-4.3.0-0.0.master.20180627130008.gita3072d3.el7.noarch.rpm
As you can see here in engine's log, *a6b3aa4* is older than *a3072d3* 11aeec678f2 webadmin: Add 'Multi Queues Enabled' to vm dialog 920bf6a75a1 packaging: Add python-netaddr requirement *a3072d3*ceba engine: Add "MultiQueuesEnabled" to ovf reader and writer 4893678a272 engine: Update libvirtVmXml to consider vmBase.multiQueuesEnabled attribute 072bad8f4d3 engine: Introduce algorithm for calculating how many queues asign per vnic c4dd484c4ee engine: Add multiQueuesEnabled to VmBase *a6b3aa4*5d18 packaging: setup: Return value 0 when using --help
And now, with Alona's patch that supposed to fix the issue (verified via manual OST), it seems that again, the CQ run is not using Alona's version of engine but an older one.
The patch that fixes the issue is https://gerrit.ovirt.org/#/c/92656/ which was tested here https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/8511/ (but again, the engine was actually installed from an older version that doesn't include the fix - hence it failed).
Could someone have a look too? If it's indeed the case, it's a severe bug form our side that we have to address ASAP.
On Sun, Jul 1, 2018 at 5:15 PM Eyal Edri <eedri@redhat.com> wrote:
Please prepare an RCA how we hit again a bug in the script if it was verified before deployment.
On Sun, Jul 1, 2018 at 4:40 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
Due to a bug in the retention policy, some packages were deleted from tested repo. We're rollbacking from nightly snapshot for all versions, and will update asap.
On Sun, Jul 1, 2018 at 4:37 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
failed on installing packages. restoring tested repo
On Sun, Jul 1, 2018 at 4:34 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
> will trigger the job again > > > On Sun, Jul 1, 2018 at 3:27 PM oVirt Jenkins <jenkins@ovirt.org> > wrote: > >> Change 92610,3 (ovirt-engine) is probably the reason behind recent >> system test >> failures in the "ovirt-4.2" change queue and needs to be fixed. >> >> This change had been removed from the testing queue. Artifacts >> build from this >> change will not be released until it is fixed. >> >> For further details about the change see: >> https://gerrit.ovirt.org/#/c/92610/3 >> >> For failed test results see: >> http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2534/ >> _______________________________________________ >> Infra mailing list -- infra@ovirt.org >> To unsubscribe send an email to infra-leave@ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MVIJ6JKUKZSW3X... >> > _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/FOCZJSME4AYMZE...
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ) _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/3CGDFQDQMSWAWX...
--
DANIEL BELENKY
RHV DEVOPS
--
DANIEL BELENKY
RHV DEVOPS
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
-- DANIEL BELENKY RHV DEVOPS
On 2 July 2018 at 08:57, Daniel Belenky <dbelenky@redhat.com> wrote:
For some reason, after merging the revert patch, JJB deploy didn't update the master change-queue tester job so it still used my patch. We need to check why.
Its a pipeline groovy change, its not managed by JJB unless you changed pipeline-loader itself... If someone changed the refspec that the job loads manually, this change does not show up in the Git diff and therefore JJB does not update the job... Meanwhile, I've modified the tester manually so it will use master HEAD now.
This should finally run tests with Alona's fix and hopefully, if there's no another bug, deploy the batch.
Please make sure to re-add any changes from any projects that got dropped in the meantime if not later changes got added.
On Sun, Jul 1, 2018 at 8:24 PM Barak Korren <bkorren@redhat.com> wrote:
On 1 July 2018 at 18:32, Daniel Belenky <dbelenky@redhat.com> wrote:
Seems that the root cause is the fact that CQ tester is using stdci_runner now and stdci_runner is not taking 'extra_sources' into account.
I guess I should've been more careful in the review, and you should be more careful refactoring!
I was wondering today how come I don't remember seeing the mirror management features from CQ make their way into STDCI. Need to make sure we don't "accidentally" drop those too.
Meanwhile, I've reverted the change so we can re-examine stdci_runner.groovy
https://gerrit.ovirt.org/#/c/92714/
On Sun, Jul 1, 2018 at 5:59 PM Daniel Belenky <dbelenky@redhat.com> wrote:
Hi,
There's something that seems odd to me. @Barak Korren <bkorren@redhat.com> @Gal Ben Haim <gbenhaim@redhat.com> would appreciate you to have a look as well.
According to Alona, the patch that introduced the error is: https://gerrit.ovirt.org/#/c/92009/
It was *tested* (and *deployed*) in https://jenkins.ovirt.org/ job/ovirt-master_change-queue-tester/8372/
But, from yum.log on engine's machine, it seems that an older version of ovirt-engine was used (not the version that was built with the patch that introduced the error - hence why it was deployed without failing the tests)
From https://jenkins.ovirt.org/job/ovirt-master_change- queue-tester/8372/artifact/exported-artifacts/basic-suit. el7.x86_64/test_logs/basic-suite-master/post-099_aaa- ldap.py/lago-basic-suite-master-engine/_var_log/yum.log
Jun 27 09:47:30 Installed: ovirt-engine-4.3.0-0.0.master.20180627094550.gita6b3aa4.el7.noarch
Where Alona's patch built ovirt-engine-4.3.0-0.0.master.20180627130008.gita3072d3.el7.noarch.rpm
As you can see here in engine's log, *a6b3aa4* is older than *a3072d3* 11aeec678f2 webadmin: Add 'Multi Queues Enabled' to vm dialog 920bf6a75a1 packaging: Add python-netaddr requirement *a3072d3*ceba engine: Add "MultiQueuesEnabled" to ovf reader and writer 4893678a272 engine: Update libvirtVmXml to consider vmBase.multiQueuesEnabled attribute 072bad8f4d3 engine: Introduce algorithm for calculating how many queues asign per vnic c4dd484c4ee engine: Add multiQueuesEnabled to VmBase *a6b3aa4*5d18 packaging: setup: Return value 0 when using --help
And now, with Alona's patch that supposed to fix the issue (verified via manual OST), it seems that again, the CQ run is not using Alona's version of engine but an older one.
The patch that fixes the issue is https://gerrit.ovirt.org/#/c/92656/ which was tested here https://jenkins.ovirt.org/job/ ovirt-master_change-queue-tester/8511/ (but again, the engine was actually installed from an older version that doesn't include the fix - hence it failed).
Could someone have a look too? If it's indeed the case, it's a severe bug form our side that we have to address ASAP.
On Sun, Jul 1, 2018 at 5:15 PM Eyal Edri <eedri@redhat.com> wrote:
Please prepare an RCA how we hit again a bug in the script if it was verified before deployment.
On Sun, Jul 1, 2018 at 4:40 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
Due to a bug in the retention policy, some packages were deleted from tested repo. We're rollbacking from nightly snapshot for all versions, and will update asap.
On Sun, Jul 1, 2018 at 4:37 PM Ehud Yonasi <eyonasi@redhat.com> wrote:
> failed on installing packages. restoring tested repo > > > On Sun, Jul 1, 2018 at 4:34 PM Ehud Yonasi <eyonasi@redhat.com> > wrote: > >> will trigger the job again >> >> >> On Sun, Jul 1, 2018 at 3:27 PM oVirt Jenkins <jenkins@ovirt.org> >> wrote: >> >>> Change 92610,3 (ovirt-engine) is probably the reason behind recent >>> system test >>> failures in the "ovirt-4.2" change queue and needs to be fixed. >>> >>> This change had been removed from the testing queue. Artifacts >>> build from this >>> change will not be released until it is fixed. >>> >>> For further details about the change see: >>> https://gerrit.ovirt.org/#/c/92610/3 >>> >>> For failed test results see: >>> http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2534/ >>> _______________________________________________ >>> Infra mailing list -- infra@ovirt.org >>> To unsubscribe send an email to infra-leave@ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: https://www.ovirt.org/ >>> community/about/community-guidelines/ >>> List Archives: https://lists.ovirt.org/ >>> archives/list/infra@ovirt.org/message/ >>> MVIJ6JKUKZSW3XKJTAJNIHPLVIVGWBN2/ >>> >> _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/ community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ message/FOCZJSME4AYMZEDMC4GZ63UNVJ5WRSYT/
--
Eyal edri
MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ) _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/ community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ message/3CGDFQDQMSWAWXUBNPAH4VXBSYGVWCYE/
--
DANIEL BELENKY
RHV DEVOPS
--
DANIEL BELENKY
RHV DEVOPS
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
--
DANIEL BELENKY
RHV DEVOPS
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
participants (5)
-
Barak Korren -
Daniel Belenky -
Ehud Yonasi -
Eyal Edri -
oVirt Jenkins