[ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ upgrade suites failed! ] [ 15/10/17 ]

[Yedidyah Bar David]

On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky <dbelenky at redhat.com> wrote:

> can someone address this issue? every patch to *ovirt-engine* that is
> based on top of this patch is failing OST and* won't deploy to the tested
> repo*.
>
> On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky <dbelenky at redhat.com>
> wrote:
>
>> Hi all,
>> The following tests are failing both of the upgrade suites in OST
>> (upgrade_from_release and upgrade_from_prevrelease).
>>
>> *Link to console:* ovirt-master_change-queue-tester/3146/console
>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/console>
>> *Link to test logs:*
>>     - upgrade-from-release-suit-master-el7
>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7>
>>     - upgrade-from-prevrelease-suit-master-el7
>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-prevrelease-suit-master-el7>
>> *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5
>> *Please note that every patch that is based on top of the patch above was
>> not deployed to the tested repo.*
>>
>> *Error snippet from engine setup log:*
>>
>
Please add a direct link next time, if possible. This is it:

http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_upgrade_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log

And a bit above the snippet below, there is:

2017-10-13 22:26:24,274-0400 DEBUG
otopi.plugins.ovirt_engine_setup.ovirt_engine.upgrade.asynctasks
plugin.execute:926 execute-output:
('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', '-l',
'/var/log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log',
'-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q',
'-r', '-Z') stderr:

/usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3:
/usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file
or directory


2017-10-13 22:26:24,274-0400 DEBUG otopi.context
context._executeMethod:143 method exception
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod
>>     method['method']()
>>   File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies
>>     self._clearZombies()
>>   File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies
>>     'Failed to clear zombie commands. '
>> RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem
>> 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem
>>
>>
With [1], taskcleaner.sh sources generate-pgpass.sh .

generate-pgpass.sh is in ovirt-engine-tools, which in upgrade flows, is not
yet upgraded (at the point of above failure).

generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using a path
relative to "$0". In master it does not, but we now upgrade and it does.

This, in principle, is the core of the bug:

A file, such as generate-pgpass.sh, that's supposed to be sourced
from some other files, should not by itself source other files
that are relative to "$0", because it can't know what "$0" is - it's
the path of the script sourcing it, not of itself.

It seems like luckily we were not affected by this in 4.1, because
all of the files that sourced generate-pgpass.sh were together with
it in the same directory. But with [1], taskcleaner does too now,
and is in a different directory.

Not sure what's the best solution:

- revert [1] (and introduce it later on, in 4.3)

- patch 4.1's generate-pgpass.sh and require the fixed 4.1 version
in 4.2 setup

- Somehow trick everything to work together? Not sure. Seems like
you can't set $0.

[1] https://gerrit.ovirt.org/82511

Regards,


> --
>>
>> DANIEL BELENKY
>>
>> RHV DEVOPS
>>
>> EMEA VIRTUALIZATION R&D
>> <https://red.ht/sig>
>>
>
>
>
> --
>
> DANIEL BELENKY
>
> RHV DEVOPS
>
> EMEA VIRTUALIZATION R&D
> <https://red.ht/sig>
>
> _______________________________________________
> Devel mailing list
> Devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>



-- 
Didi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20171016/6e8c663c/attachment.html>