[oVirt Jenkins] ovirt-engine_3.6_upgrade-from-3.6_el6_merged - Build # 735 - Failure!
------=_Part_21_1797059276.1452102315861 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Project: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/ Build: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/73... Build Number: 735 Build Status: Failure Triggered By: Triggered by Gerrit: https://gerrit.ovirt.org/51331 ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #735 [Maor Lipchuk] core: Refactor call back of CloneSingleCinderDisk ----------------- Failed Tests: ----------------- No tests ran. ------=_Part_21_1797059276.1452102315861--
------=_Part_23_1798454720.1452103944599 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Project: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/ Build: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/73... Build Number: 736 Build Status: Still Failing Triggered By: Triggered by Gerrit: https://gerrit.ovirt.org/51332 ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #735 [Maor Lipchuk] core: Refactor call back of CloneSingleCinderDisk Changes for Build #736 [Maor Lipchuk] core: CoCo revert tasks of clone Cinder disks. ----------------- Failed Tests: ----------------- No tests ran. ------=_Part_23_1798454720.1452103944599--
------=_Part_25_1060954259.1452105387033 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Project: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/ Build: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/73... Build Number: 737 Build Status: Still Failing Triggered By: Triggered by Gerrit: https://gerrit.ovirt.org/51333 ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #735 [Maor Lipchuk] core: Refactor call back of CloneSingleCinderDisk Changes for Build #736 [Maor Lipchuk] core: CoCo revert tasks of clone Cinder disks. Changes for Build #737 [Maor Lipchuk] core: Add callback for create VM from Template with Cinder. ----------------- Failed Tests: ----------------- No tests ran. ------=_Part_25_1060954259.1452105387033--
------=_Part_27_1585539555.1452106947893 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Project: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/ Build: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/73... Build Number: 738 Build Status: Still Failing Triggered By: Triggered by Gerrit: https://gerrit.ovirt.org/51334 ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #735 [Maor Lipchuk] core: Refactor call back of CloneSingleCinderDisk Changes for Build #736 [Maor Lipchuk] core: CoCo revert tasks of clone Cinder disks. Changes for Build #737 [Maor Lipchuk] core: Add callback for create VM from Template with Cinder. Changes for Build #738 [Maor Lipchuk] core: Call concurrent execution callback on add vm command. ----------------- Failed Tests: ----------------- No tests ran. ------=_Part_27_1585539555.1452106947893--
Project: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/ Build: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/73... Build Number: 739 Build Status: Still Failing Triggered By: Triggered by Gerrit: https://gerrit.ovirt.org/51335 ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #735 [Maor Lipchuk] core: Refactor call back of CloneSingleCinderDisk Changes for Build #736 [Maor Lipchuk] core: CoCo revert tasks of clone Cinder disks. Changes for Build #737 [Maor Lipchuk] core: Add callback for create VM from Template with Cinder. Changes for Build #738 [Maor Lipchuk] core: Call concurrent execution callback on add vm command. Changes for Build #739 [Maor Lipchuk] core: CoCo, prevent using removeFromHirerchy on callback. ----------------- Failed Tests: ----------------- No tests ran.
------=_Part_31_957555218.1452109847620 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Project: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/ Build: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/74... Build Number: 740 Build Status: Still Failing Triggered By: Triggered by Gerrit: https://gerrit.ovirt.org/51336 ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #735 [Maor Lipchuk] core: Refactor call back of CloneSingleCinderDisk Changes for Build #736 [Maor Lipchuk] core: CoCo revert tasks of clone Cinder disks. Changes for Build #737 [Maor Lipchuk] core: Add callback for create VM from Template with Cinder. Changes for Build #738 [Maor Lipchuk] core: Call concurrent execution callback on add vm command. Changes for Build #739 [Maor Lipchuk] core: CoCo, prevent using removeFromHirerchy on callback. Changes for Build #740 [Maor Lipchuk] core: Lock VM should not be done on revert tasks ----------------- Failed Tests: ----------------- No tests ran. ------=_Part_31_957555218.1452109847620--
------=_Part_33_2115989441.1452111503427 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Project: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/ Build: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/74... Build Number: 741 Build Status: Still Failing Triggered By: Triggered by Gerrit: https://gerrit.ovirt.org/51401 ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #735 [Maor Lipchuk] core: Refactor call back of CloneSingleCinderDisk Changes for Build #736 [Maor Lipchuk] core: CoCo revert tasks of clone Cinder disks. Changes for Build #737 [Maor Lipchuk] core: Add callback for create VM from Template with Cinder. Changes for Build #738 [Maor Lipchuk] core: Call concurrent execution callback on add vm command. Changes for Build #739 [Maor Lipchuk] core: CoCo, prevent using removeFromHirerchy on callback. Changes for Build #740 [Maor Lipchuk] core: Lock VM should not be done on revert tasks Changes for Build #741 [Maor Lipchuk] core: Filter out child Cinder commands on end action. ----------------- Failed Tests: ----------------- No tests ran. ------=_Part_33_2115989441.1452111503427--
Not the foggiest. But this job seems to be failing intermittently regardless of those patches. On Wed, Jan 6, 2016 at 10:18 PM, <jenkins@ovirt.org> wrote:
Project: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/ Build: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/74... Build Number: 741 Build Status: Still Failing Triggered By: Triggered by Gerrit: https://gerrit.ovirt.org/51401
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #735 [Maor Lipchuk] core: Refactor call back of CloneSingleCinderDisk
Changes for Build #736 [Maor Lipchuk] core: CoCo revert tasks of clone Cinder disks.
Changes for Build #737 [Maor Lipchuk] core: Add callback for create VM from Template with Cinder.
Changes for Build #738 [Maor Lipchuk] core: Call concurrent execution callback on add vm command.
Changes for Build #739 [Maor Lipchuk] core: CoCo, prevent using removeFromHirerchy on callback.
Changes for Build #740 [Maor Lipchuk] core: Lock VM should not be done on revert tasks
Changes for Build #741 [Maor Lipchuk] core: Filter out child Cinder commands on end action.
----------------- Failed Tests: ----------------- No tests ran.
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Cleanup issue? Sandro/Simone - can you look? topi.plugins.ovirt_engine_remove.base.core.misc misc._setup:90 Could not detect a completed product setup 2016-01-06 20:18:05 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:219 DIALOG:SEND Please use the cleanup utility only after a setup or after an upgrade from an older installation. 2016-01-06 20:18:05 DEBUG otopi.context context._executeMethod:156 method exception Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/otopi/context.py", line 146, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-remove/base/core/misc.py", line 102, in _setup _('Could not detect product setup') RuntimeError: Could not detect product setup 2016-01-06 20:18:05 ERROR otopi.context context._executeMethod:165 Failed to execute stage 'Environment setup': Could not detect product setup 2016-01-06 20:18:05 DEBUG otopi.context context.dumpEnvironment:500 ENVIRONMENT DUMP - BEGIN 2016-01-06 20:18:05 DEBUG otopi.context context.dumpEnvironment:510 ENV BASE/error=bool:'True' 2016-01-06 20:18:05 DEBUG otopi.context context.dumpEnvironment:510 ENV BASE/exceptionInfo=list:'[(<type 'exceptions.RuntimeError'>, RuntimeError('Could not detect product setup',), <traceback object at 0x2eabdd0>)]' 2016-01-06 20:18:05 DEBUG otopi.context context.dumpEnvironment:510 ENV BASE/exitCode=list:'[{'priority': 90001, 'code': 0}, {'priority': 5000, 'code': 11}]' 2016-01-06 20:18:05 DEBUG otopi.context context.dumpEnvironment:510 ENV OVESET On Thu, Jan 7, 2016 at 6:37 PM, Allon Mureinik <amureini@redhat.com> wrote:
Not the foggiest. But this job seems to be failing intermittently regardless of those patches.
On Wed, Jan 6, 2016 at 10:18 PM, <jenkins@ovirt.org> wrote:
Project: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/ Build: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/74... Build Number: 741 Build Status: Still Failing Triggered By: Triggered by Gerrit: https://gerrit.ovirt.org/51401
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #735 [Maor Lipchuk] core: Refactor call back of CloneSingleCinderDisk
Changes for Build #736 [Maor Lipchuk] core: CoCo revert tasks of clone Cinder disks.
Changes for Build #737 [Maor Lipchuk] core: Add callback for create VM from Template with Cinder.
Changes for Build #738 [Maor Lipchuk] core: Call concurrent execution callback on add vm command.
Changes for Build #739 [Maor Lipchuk] core: CoCo, prevent using removeFromHirerchy on callback.
Changes for Build #740 [Maor Lipchuk] core: Lock VM should not be done on revert tasks
Changes for Build #741 [Maor Lipchuk] core: Filter out child Cinder commands on end action.
----------------- Failed Tests: ----------------- No tests ran.
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
On Fri, Jan 8, 2016 at 4:23 PM, Eyal Edri <eedri@redhat.com> wrote:
Cleanup issue? Sandro/Simone - can you look?
topi.plugins.ovirt_engine_remove.base.core.misc misc._setup:90 Could not detect a completed product setup 2016-01-06 20:18:05 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:219 DIALOG:SEND Please use the cleanup utility only after a setup or after an upgrade from an older installation. 2016-01-06 20:18:05 DEBUG otopi.context context._executeMethod:156 method exception Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/otopi/context.py", line 146, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-remove/base/core/misc.py", line 102, in _setup _('Could not detect product setup')
This is just what it says - cleanup doing nothing because it didn't find a setup. Most likely due to a failed setup in a previous job. Should be ignored. Note that in 3.6, we can easily identify if this was the reason for a failure and not something else, based on the exit code, see [1]. Other exit codes are defined as EXIT_CODE_* constants, currently around [2] and [3]. We might add more in the future, including in other places (such as dwh/reports). [1] https://bugzilla.redhat.com/show_bug.cgi?id=1079726 [2] https://gerrit.ovirt.org/gitweb?p=otopi.git;a=blob;f=src/otopi/constants.py;... [3] https://gerrit.ovirt.org/gitweb?p=ovirt-engine.git;a=blob;f=packaging/setup/... Some other comments:
On Thu, Jan 7, 2016 at 6:37 PM, Allon Mureinik <amureini@redhat.com> wrote:
Not the foggiest. But this job seems to be failing intermittently regardless of those patches.
A later run of this job 748 succeeded. The specific failure was likely due to rpmdb corruption: Rpmdb checksum is invalid: dCDPT(pkg checksums): ovirt-engine.noarch 0:3.6.3-0.0.master.20160105173810.git7421257.el6 - u IMO we should try harder to: 1. Uniquely mark the point at which a job decided that it failed, inside the output, so that it's easy to find, and easy to ignore unrelated error messages. 2. Once we do that, add some of the preceding text directly to the message sent by mail and added in gerrit.
On Wed, Jan 6, 2016 at 10:18 PM, <jenkins@ovirt.org> wrote:
Project: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/ Build: http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/74... Build Number: 741 Build Status: Still Failing Triggered By: Triggered by Gerrit: https://gerrit.ovirt.org/51401
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #735 [Maor Lipchuk] core: Refactor call back of CloneSingleCinderDisk
Changes for Build #736 [Maor Lipchuk] core: CoCo revert tasks of clone Cinder disks.
Changes for Build #737 [Maor Lipchuk] core: Add callback for create VM from Template with Cinder.
Changes for Build #738 [Maor Lipchuk] core: Call concurrent execution callback on add vm command.
Changes for Build #739 [Maor Lipchuk] core: CoCo, prevent using removeFromHirerchy on callback.
Changes for Build #740 [Maor Lipchuk] core: Lock VM should not be done on revert tasks
Changes for Build #741 [Maor Lipchuk] core: Filter out child Cinder commands on end action.
----------------- Failed Tests: ----------------- No tests ran.
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Didi
On Sun, Jan 10, 2016 at 9:26 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Fri, Jan 8, 2016 at 4:23 PM, Eyal Edri <eedri@redhat.com> wrote:
Cleanup issue? Sandro/Simone - can you look?
topi.plugins.ovirt_engine_remove.base.core.misc misc._setup:90 Could not detect a completed product setup 2016-01-06 20:18:05 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:219 DIALOG:SEND Please use the cleanup utility only after a setup or after an upgrade from an older installation. 2016-01-06 20:18:05 DEBUG otopi.context context._executeMethod:156 method exception Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/otopi/context.py", line 146, in _executeMethod method['method']() File
"/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-remove/base/core/misc.py",
line 102, in _setup _('Could not detect product setup')
This is just what it says - cleanup doing nothing because it didn't find a setup. Most likely due to a failed setup in a previous job. Should be ignored. Note that in 3.6, we can easily identify if this was the reason for a failure and not something else, based on the exit code, see [1]. Other exit codes are defined as EXIT_CODE_* constants, currently around [2] and [3]. We might add more in the future, including in other places (such as dwh/reports).
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1079726 [2] https://gerrit.ovirt.org/gitweb?p=otopi.git;a=blob;f=src/otopi/constants.py;... [3] https://gerrit.ovirt.org/gitweb?p=ovirt-engine.git;a=blob;f=packaging/setup/...
Some other comments:
On Thu, Jan 7, 2016 at 6:37 PM, Allon Mureinik <amureini@redhat.com> wrote:
Not the foggiest. But this job seems to be failing intermittently regardless of those patches.
A later run of this job 748 succeeded.
The specific failure was likely due to rpmdb corruption:
Rpmdb checksum is invalid: dCDPT(pkg checksums): ovirt-engine.noarch 0:3.6.3-0.0.master.20160105173810.git7421257.el6 - u
IMO we should try harder to: 1. Uniquely mark the point at which a job decided that it failed, inside the output, so that it's easy to find, and easy to ignore unrelated error messages. 2. Once we do that, add some of the preceding text directly to the message sent by mail and added in gerrit.
The RPMDB corruption is a known bug still open on rpm in Bugzilla. It happens because we install/remove numerous times the same packages without reinstalling the VM, For now we have a script that will rebuild the rpmdb each run, but going forward the job will move to Lago and run on an isolated new vm on each run. E.
On Wed, Jan 6, 2016 at 10:18 PM, <jenkins@ovirt.org> wrote:
Project:
http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/
Build:
http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/74...
Build Number: 741 Build Status: Still Failing Triggered By: Triggered by Gerrit: https://gerrit.ovirt.org/51401
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #735 [Maor Lipchuk] core: Refactor call back of CloneSingleCinderDisk
Changes for Build #736 [Maor Lipchuk] core: CoCo revert tasks of clone Cinder disks.
Changes for Build #737 [Maor Lipchuk] core: Add callback for create VM from Template with Cinder.
Changes for Build #738 [Maor Lipchuk] core: Call concurrent execution callback on add vm command.
Changes for Build #739 [Maor Lipchuk] core: CoCo, prevent using removeFromHirerchy on callback.
Changes for Build #740 [Maor Lipchuk] core: Lock VM should not be done on revert tasks
Changes for Build #741 [Maor Lipchuk] core: Filter out child Cinder commands on end action.
----------------- Failed Tests: ----------------- No tests ran.
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Eyal Edri Associate Manager EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
-- Didi
-- Eyal Edri Associate Manager EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
participants (4)
-
Allon Mureinik -
Eyal Edri -
jenkins＠ovirt.org -
Yedidyah Bar David