[ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ upgrade suites failed! ] [ 15/10/17 ]

Tue Oct 17 12:23:30 UTC 2017

On Tue, Oct 17, 2017 at 1:17 PM, Daniel Belenky <dbelenky at redhat.com> wrote:

> Hey, I see that both of the patches that were supposed to fix the issue (
> https://gerrit.ovirt.org/82800 and https://gerrit.ovirt.org/#/c/82799/)
> were merged, but the issue remains.
>

Are we sure that when testing upgrade from 4.1 to master, we are using
4.1.7.XXX from latest tested as a base version? Because it we will try to
upgrade from any previous version, the upgrade

will always fail. I'm asking about it, because we wasn't able to find a
way how to force manual OST to use latest tested as base version, it always
use latest released 4.1.6. That's why Miroslava had to verify that manually

>
> On Mon, Oct 16, 2017 at 11:45 AM, Sandro Bonazzola <sbonazzo at redhat.com>
> wrote:
>
>>
>>
>> 2017-10-16 10:30 GMT+02:00 Yedidyah Bar David <didi at redhat.com>:
>>
>>> On Mon, Oct 16, 2017 at 11:01 AM, Miroslava Voglova <mvoglova at redhat.com
>>> > wrote:
>>>
>>>> On Mon, Oct 16, 2017 at 9:44 AM, Martin Perina <mperina at redhat.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 16, 2017 at 9:38 AM, Yedidyah Bar David <didi at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> On Mon, Oct 16, 2017 at 10:34 AM, Miroslava Voglova <
>>>>>> mvoglova at redhat.com> wrote:
>>>>>>
>>>>>>> Fix on review https://gerrit.ovirt.org/#/c/82799/
>>>>>>>
>>>>>>
>>>>>> That's indeed a related patch, but not sure how it solves current
>>>>>> failure.
>>>>>>
>>>>>
>>>>> Let's copy generate-pgpass.sh to packaging/setup/dbutils and source
>>>>> it from this location for taskcleaner.sh and unlock_entity.sh
>>>>>
>>>>
>>>>
>>>> After offline discussion merging https://gerrit.ovirt.org/82800 and
>>>> then https://gerrit.ovirt.org/#/c/82799/ will fix the issue. Both
>>>> patches are needed.
>>>>
>>>
>>> So this means:
>>>
>>> Merge https://gerrit.ovirt.org/82800 . We might want to open a
>>> real 4.1 bug for this.
>>>
>>
>> Agreed, let's open a real 4.1.7 bug to track this and get proper
>> verification by QE.
>>
>>
>>>
>>> Build 4.1.7 (or 4.1.8?) with it .
>>>
>>
>> Let's point to 4.1.7
>>
>>
>>>
>>> Patch 4.2 engine-setup to require tools-4.1.7.
>>>
>>
>> well, tools >= 4.1.7 :-)
>>
>>
>>
>>
>>>
>>> Adding Sandro.
>>>
>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Mon, Oct 16, 2017 at 9:32 AM, Yaniv Kaul <ykaul at redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Oct 16, 2017 at 10:24 AM, Yedidyah Bar David <
>>>>>>>> didi at redhat.com> wrote:
>>>>>>>>
>>>>>>>>> On Mon, Oct 16, 2017 at 10:21 AM, Yedidyah Bar David <
>>>>>>>>> didi at redhat.com> wrote:
>>>>>>>>>
>>>>>>>>>> On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky <
>>>>>>>>>> dbelenky at redhat.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> can someone address this issue? every patch to *ovirt-engine* that
>>>>>>>>>>> is based on top of this patch is failing OST and* won't deploy
>>>>>>>>>>> to the tested repo*.
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky <
>>>>>>>>>>> dbelenky at redhat.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>> The following tests are failing both of the upgrade suites in
>>>>>>>>>>>> OST (upgrade_from_release and upgrade_from_prevrelease).
>>>>>>>>>>>>
>>>>>>>>>>>> *Link to console:* ovirt-master_change-q
>>>>>>>>>>>> ueue-tester/3146/console
>>>>>>>>>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/console>
>>>>>>>>>>>> *Link to test logs:*
>>>>>>>>>>>>     - upgrade-from-release-suit-master-el7
>>>>>>>>>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7>
>>>>>>>>>>>>     - upgrade-from-prevrelease-suit-master-el7
>>>>>>>>>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-prevrelease-suit-master-el7>
>>>>>>>>>>>> *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5
>>>>>>>>>>>> *Please note that every patch that is based on top of the patch
>>>>>>>>>>>> above was not deployed to the tested repo.*
>>>>>>>>>>>>
>>>>>>>>>>>> *Error snippet from engine setup log:*
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Please add a direct link next time, if possible. This is it:
>>>>>>>>>>
>>>>>>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste
>>>>>>>>>> r/3146/artifact/exported-artifacts/upgrade-from-release-suit
>>>>>>>>>> -master-el7/test_logs/upgrade-from-release-suite-master/post
>>>>>>>>>> -001_upgrade_engine.py/lago-upgrade-from-release-suite-maste
>>>>>>>>>> r-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-2017
>>>>>>>>>> 1013222617-73f0df.log
>>>>>>>>>>
>>>>>>>>>> And a bit above the snippet below, there is:
>>>>>>>>>>
>>>>>>>>>> 2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_set
>>>>>>>>>> up.ovirt_engine.upgrade.asynctasks plugin.execute:926
>>>>>>>>>> execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh',
>>>>>>>>>> '-l', '/var/log/ovirt-engine/setup/o
>>>>>>>>>> virt-engine-setup-20171013222617-73f0df.log', '-u', 'engine',
>>>>>>>>>> '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q', '-r', '-Z') stderr:
>>>>>>>>>>
>>>>>>>>>> /usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception
>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>   File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod
>>>>>>>>>>>>     method['method']()
>>>>>>>>>>>>   File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies
>>>>>>>>>>>>     self._clearZombies()
>>>>>>>>>>>>   File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies
>>>>>>>>>>>>     'Failed to clear zombie commands. '
>>>>>>>>>>>> RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem
>>>>>>>>>>>> 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>> With [1], taskcleaner.sh sources generate-pgpass.sh .
>>>>>>>>>>
>>>>>>>>>> generate-pgpass.sh is in ovirt-engine-tools, which in upgrade
>>>>>>>>>> flows, is not
>>>>>>>>>> yet upgraded (at the point of above failure).
>>>>>>>>>>
>>>>>>>>>> generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using
>>>>>>>>>> a path
>>>>>>>>>> relative to "$0". In master it does not, but we now upgrade and
>>>>>>>>>> it does.
>>>>>>>>>>
>>>>>>>>>> This, in principle, is the core of the bug:
>>>>>>>>>>
>>>>>>>>>> A file, such as generate-pgpass.sh, that's supposed to be sourced
>>>>>>>>>> from some other files, should not by itself source other files
>>>>>>>>>> that are relative to "$0", because it can't know what "$0" is -
>>>>>>>>>> it's
>>>>>>>>>> the path of the script sourcing it, not of itself.
>>>>>>>>>>
>>>>>>>>>> It seems like luckily we were not affected by this in 4.1, because
>>>>>>>>>> all of the files that sourced generate-pgpass.sh were together
>>>>>>>>>> with
>>>>>>>>>> it in the same directory. But with [1], taskcleaner does too now,
>>>>>>>>>> and is in a different directory.
>>>>>>>>>>
>>>>>>>>>> Not sure what's the best solution:
>>>>>>>>>>
>>>>>>>>>> - revert [1] (and introduce it later on, in 4.3)
>>>>>>>>>>
>>>>>>>>>> - patch 4.1's generate-pgpass.sh and require the fixed 4.1 version
>>>>>>>>>> in 4.2 setup
>>>>>>>>>>
>>>>>>>>>
>>>>>> E.g.:
>>>>>>
>>>>>> https://gerrit.ovirt.org/82800
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>>>> - Somehow trick everything to work together? Not sure. Seems like
>>>>>>>>>> you can't set $0.
>>>>>>>>>>
>>>>>>>>>> [1] https://gerrit.ovirt.org/82511
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The same bug exists with unlock_entity:
>>>>>>>>>
>>>>>>>>> https://gerrit.ovirt.org/82615
>>>>>>>>>
>>>>>>>>> So we should probably revert both.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I tend to agree.
>>>>>>>> Doesn't mean we cannot fix this for 4.2, but let's revert for the
>>>>>>>> meantime.
>>>>>>>> Y.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> DANIEL BELENKY
>>>>>>>>>>>>
>>>>>>>>>>>> RHV DEVOPS
>>>>>>>>>>>>
>>>>>>>>>>>> EMEA VIRTUALIZATION R&D
>>>>>>>>>>>> <https://red.ht/sig>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> DANIEL BELENKY
>>>>>>>>>>>
>>>>>>>>>>> RHV DEVOPS
>>>>>>>>>>>
>>>>>>>>>>> EMEA VIRTUALIZATION R&D
>>>>>>>>>>> <https://red.ht/sig>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Devel mailing list
>>>>>>>>>>> Devel at ovirt.org
>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Didi
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Didi
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Devel mailing list
>>>>>>>>> Devel at ovirt.org
>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Devel mailing list
>>>>>>>> Devel at ovirt.org
>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Didi
>>>>>>
>>>>>> _______________________________________________
>>>>>> Devel mailing list
>>>>>> Devel at ovirt.org
>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Didi
>>>
>>
>>
>>
>> --
>>
>> SANDRO BONAZZOLA
>>
>> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
>>
>> Red Hat EMEA <https://www.redhat.com/>
>> <https://red.ht/sig>
>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>> <http://www.teraplan.it/redhat-osd-2017/>
>>
>
>
>
> --
>
> DANIEL BELENKY
>
> RHV DEVOPS
>
> EMEA VIRTUALIZATION R&D
> <https://red.ht/sig>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20171017/e5a82a77/attachment-0001.html>