On Sun, Jan 10, 2016 at 9:26 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Fri, Jan 8, 2016 at 4:23 PM, Eyal Edri <eedri@redhat.com> wrote:
> Cleanup issue? Sandro/Simone - can you look?
>
> topi.plugins.ovirt_engine_remove.base.core.misc misc._setup:90 Could not
> detect a completed product setup
> 2016-01-06 20:18:05 DEBUG otopi.plugins.otopi.dialog.human
> dialog.__logString:219 DIALOG:SEND                 Please use the cleanup
> utility only after a setup or after an upgrade from an older installation.
> 2016-01-06 20:18:05 DEBUG otopi.context context._executeMethod:156 method
> exception
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/site-packages/otopi/context.py", line 146, in
> _executeMethod
>     method['method']()
>   File
> "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-remove/base/core/misc.py",
> line 102, in _setup
>     _('Could not detect product setup')

This is just what it says - cleanup doing nothing because it didn't
find a setup.
Most likely due to a failed setup in a previous job.
Should be ignored.
Note that in 3.6, we can easily identify if this was the reason for a failure
and not something else, based on the exit code, see [1].
Other exit codes are defined as EXIT_CODE_* constants, currently
around [2] and [3].
We might add more in the future, including in other places (such as
dwh/reports).

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1079726
[2] https://gerrit.ovirt.org/gitweb?p=otopi.git;a=blob;f=src/otopi/constants.py;hb=HEAD#l84
[3] https://gerrit.ovirt.org/gitweb?p=ovirt-engine.git;a=blob;f=packaging/setup/ovirt_engine_setup/constants.py;hb=HEAD#l267

Some other comments:

> On Thu, Jan 7, 2016 at 6:37 PM, Allon Mureinik <amureini@redhat.com> wrote:
>>
>> Not the foggiest.
>> But this job seems to be failing intermittently regardless of those
>> patches.

A later run of this job 748 succeeded.

The specific failure was likely due to rpmdb corruption:

Rpmdb checksum is invalid: dCDPT(pkg checksums): ovirt-engine.noarch
0:3.6.3-0.0.master.20160105173810.git7421257.el6 - u

IMO we should try harder to:
1. Uniquely mark the point at which a job decided that it failed,
inside the output, so
that it's easy to find, and easy to ignore unrelated error messages.
2. Once we do that, add some of the preceding text directly to the
message sent by mail and added in gerrit.


The RPMDB corruption is a known bug still open on rpm in Bugzilla.
It happens because we install/remove numerous times the same packages without reinstalling the VM,
For now we have a script that will rebuild the rpmdb each run, but going forward the job will move to
Lago and run on an isolated new vm on each run.

E.
 

>>
>> On Wed, Jan 6, 2016 at 10:18 PM, <jenkins@ovirt.org> wrote:
>>>
>>> Project:
>>> http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/
>>> Build:
>>> http://jenkins.ovirt.org/job/ovirt-engine_3.6_upgrade-from-3.6_el6_merged/741/
>>> Build Number: 741
>>> Build Status:  Still Failing
>>> Triggered By: Triggered by Gerrit: https://gerrit.ovirt.org/51401
>>>
>>> -------------------------------------
>>> Changes Since Last Success:
>>> -------------------------------------
>>> Changes for Build #735
>>> [Maor Lipchuk] core: Refactor call back of CloneSingleCinderDisk
>>>
>>>
>>> Changes for Build #736
>>> [Maor Lipchuk] core: CoCo revert tasks of clone Cinder disks.
>>>
>>>
>>> Changes for Build #737
>>> [Maor Lipchuk] core: Add callback for create VM from Template with
>>> Cinder.
>>>
>>>
>>> Changes for Build #738
>>> [Maor Lipchuk] core: Call concurrent execution callback on add vm
>>> command.
>>>
>>>
>>> Changes for Build #739
>>> [Maor Lipchuk] core: CoCo, prevent using removeFromHirerchy on callback.
>>>
>>>
>>> Changes for Build #740
>>> [Maor Lipchuk] core: Lock VM should not be done on revert tasks
>>>
>>>
>>> Changes for Build #741
>>> [Maor Lipchuk] core: Filter out child Cinder commands on end action.
>>>
>>>
>>>
>>>
>>> -----------------
>>> Failed Tests:
>>> -----------------
>>> No tests ran.
>>>
>>>
>>> _______________________________________________
>>> Infra mailing list
>>> Infra@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/infra
>>>
>>
>>
>> _______________________________________________
>> Infra mailing list
>> Infra@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/infra
>>
>
>
>
> --
> Eyal Edri
> Associate Manager
> EMEA ENG Virtualization R&D
> Red Hat Israel
>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>
> _______________________________________________
> Infra mailing list
> Infra@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>



--
Didi



--
Eyal Edri
Associate Manager
EMEA ENG Virtualization R&D
Red Hat Israel

phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)