[ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ upgrade suites failed! ] [ 15/10/17 ]

Sandro Bonazzola sbonazzo at redhat.com
Mon Oct 16 08:45:03 UTC 2017 

2017-10-16 10:30 GMT+02:00 Yedidyah Bar David <didi at redhat.com>:

> On Mon, Oct 16, 2017 at 11:01 AM, Miroslava Voglova <mvoglova at redhat.com>
> wrote:
>
>> On Mon, Oct 16, 2017 at 9:44 AM, Martin Perina <mperina at redhat.com>
>> wrote:
>>
>>>
>>>
>>> On Mon, Oct 16, 2017 at 9:38 AM, Yedidyah Bar David <didi at redhat.com>
>>> wrote:
>>>
>>>> On Mon, Oct 16, 2017 at 10:34 AM, Miroslava Voglova <
>>>> mvoglova at redhat.com> wrote:
>>>>
>>>>> Fix on review https://gerrit.ovirt.org/#/c/82799/
>>>>>
>>>>
>>>> That's indeed a related patch, but not sure how it solves current
>>>> failure.
>>>>
>>>
>>> ​Let's copy generate-pgpass.sh to packaging/setup/dbutils a​nd source it
>>> from this location for taskcleaner.sh and unlock_entity.sh
>>>
>>
>>
>> After offline discussion merging https://gerrit.ovirt.org/82800 and then
>> https://gerrit.ovirt.org/#/c/82799/ will fix the issue. Both patches are
>> needed.
>>
>
> So this means:
>
> Merge https://gerrit.ovirt.org/82800 . We might want to open a
> real 4.1 bug for this.
>

Agreed, let's open a real 4.1.7 bug to track this and get proper
verification by QE.


>
> Build 4.1.7 (or 4.1.8?) with it .
>

Let's point to 4.1.7


>
> Patch 4.2 engine-setup to require tools-4.1.7.
>

well, tools >= 4.1.7 :-)




>
> Adding Sandro.
>
>
>>
>>
>>>
>>>
>>>>
>>>>>
>>>>> On Mon, Oct 16, 2017 at 9:32 AM, Yaniv Kaul <ykaul at redhat.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 16, 2017 at 10:24 AM, Yedidyah Bar David <didi at redhat.com
>>>>>> > wrote:
>>>>>>
>>>>>>> On Mon, Oct 16, 2017 at 10:21 AM, Yedidyah Bar David <
>>>>>>> didi at redhat.com> wrote:
>>>>>>>
>>>>>>>> On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky <
>>>>>>>> dbelenky at redhat.com> wrote:
>>>>>>>>
>>>>>>>>> can someone address this issue? every patch to *ovirt-engine* that
>>>>>>>>> is based on top of this patch is failing OST and* won't deploy to
>>>>>>>>> the tested repo*.
>>>>>>>>>
>>>>>>>>> On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky <
>>>>>>>>> dbelenky at redhat.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>> The following tests are failing both of the upgrade suites in OST
>>>>>>>>>> (upgrade_from_release and upgrade_from_prevrelease).
>>>>>>>>>>
>>>>>>>>>> *Link to console:* ovirt-master_change-queue-tester/3146/console
>>>>>>>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/console>
>>>>>>>>>> *Link to test logs:*
>>>>>>>>>>     - upgrade-from-release-suit-master-el7
>>>>>>>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7>
>>>>>>>>>>     - upgrade-from-prevrelease-suit-master-el7
>>>>>>>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-prevrelease-suit-master-el7>
>>>>>>>>>> *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5
>>>>>>>>>> *Please note that every patch that is based on top of the patch
>>>>>>>>>> above was not deployed to the tested repo.*
>>>>>>>>>>
>>>>>>>>>> *Error snippet from engine setup log:*
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> Please add a direct link next time, if possible. This is it:
>>>>>>>>
>>>>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste
>>>>>>>> r/3146/artifact/exported-artifacts/upgrade-from-release-suit
>>>>>>>> -master-el7/test_logs/upgrade-from-release-suite-master/post
>>>>>>>> -001_upgrade_engine.py/lago-upgrade-from-release-suite-maste
>>>>>>>> r-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-2017
>>>>>>>> 1013222617-73f0df.log
>>>>>>>>
>>>>>>>> And a bit above the snippet below, there is:
>>>>>>>>
>>>>>>>> 2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_set
>>>>>>>> up.ovirt_engine.upgrade.asynctasks plugin.execute:926
>>>>>>>> execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh',
>>>>>>>> '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log',
>>>>>>>> '-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q',
>>>>>>>> '-r', '-Z') stderr:
>>>>>>>>
>>>>>>>> /usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory
>>>>>>>>
>>>>>>>>
>>>>>>>> 2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception
>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>   File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod
>>>>>>>>>>     method['method']()
>>>>>>>>>>   File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies
>>>>>>>>>>     self._clearZombies()
>>>>>>>>>>   File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies
>>>>>>>>>>     'Failed to clear zombie commands. '
>>>>>>>>>> RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem
>>>>>>>>>> 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>> With [1], taskcleaner.sh sources generate-pgpass.sh .
>>>>>>>>
>>>>>>>> generate-pgpass.sh is in ovirt-engine-tools, which in upgrade
>>>>>>>> flows, is not
>>>>>>>> yet upgraded (at the point of above failure).
>>>>>>>>
>>>>>>>> generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using a
>>>>>>>> path
>>>>>>>> relative to "$0". In master it does not, but we now upgrade and it
>>>>>>>> does.
>>>>>>>>
>>>>>>>> This, in principle, is the core of the bug:
>>>>>>>>
>>>>>>>> A file, such as generate-pgpass.sh, that's supposed to be sourced
>>>>>>>> from some other files, should not by itself source other files
>>>>>>>> that are relative to "$0", because it can't know what "$0" is - it's
>>>>>>>> the path of the script sourcing it, not of itself.
>>>>>>>>
>>>>>>>> It seems like luckily we were not affected by this in 4.1, because
>>>>>>>> all of the files that sourced generate-pgpass.sh were together with
>>>>>>>> it in the same directory. But with [1], taskcleaner does too now,
>>>>>>>> and is in a different directory.
>>>>>>>>
>>>>>>>> Not sure what's the best solution:
>>>>>>>>
>>>>>>>> - revert [1] (and introduce it later on, in 4.3)
>>>>>>>>
>>>>>>>> - patch 4.1's generate-pgpass.sh and require the fixed 4.1 version
>>>>>>>> in 4.2 setup
>>>>>>>>
>>>>>>>
>>>> E.g.:
>>>>
>>>> https://gerrit.ovirt.org/82800
>>>>
>>>>
>>>>>
>>>>>>>> - Somehow trick everything to work together? Not sure. Seems like
>>>>>>>> you can't set $0.
>>>>>>>>
>>>>>>>> [1] https://gerrit.ovirt.org/82511
>>>>>>>>
>>>>>>>
>>>>>>> The same bug exists with unlock_entity:
>>>>>>>
>>>>>>> https://gerrit.ovirt.org/82615
>>>>>>>
>>>>>>> So we should probably revert both.
>>>>>>>
>>>>>>
>>>>>> I tend to agree.
>>>>>> Doesn't mean we cannot fix this for 4.2, but let's revert for the
>>>>>> meantime.
>>>>>> Y.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>>
>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> DANIEL BELENKY
>>>>>>>>>>
>>>>>>>>>> RHV DEVOPS
>>>>>>>>>>
>>>>>>>>>> EMEA VIRTUALIZATION R&D
>>>>>>>>>> <https://red.ht/sig>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> DANIEL BELENKY
>>>>>>>>>
>>>>>>>>> RHV DEVOPS
>>>>>>>>>
>>>>>>>>> EMEA VIRTUALIZATION R&D
>>>>>>>>> <https://red.ht/sig>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Devel mailing list
>>>>>>>>> Devel at ovirt.org
>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Didi
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Didi
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Devel mailing list
>>>>>>> Devel at ovirt.org
>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Devel mailing list
>>>>>> Devel at ovirt.org
>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Didi
>>>>
>>>> _______________________________________________
>>>> Devel mailing list
>>>> Devel at ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>
>>>
>>>
>>
>
>
> --
> Didi
>



-- 

SANDRO BONAZZOLA

ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D

Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
<http://www.teraplan.it/redhat-osd-2017/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20171016/16698b31/attachment-0001.html>

More information about the Devel mailing list