[ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ upgrade suites failed! ] [ 15/10/17 ]

Yedidyah Bar David didi at redhat.com
Tue Oct 17 12:40:16 UTC 2017 

On Tue, Oct 17, 2017 at 3:23 PM, Martin Perina <mperina at redhat.com> wrote:
>
>
>
> On Tue, Oct 17, 2017 at 1:17 PM, Daniel Belenky <dbelenky at redhat.com> wrote:
>>
>> Hey, I see that both of the patches that were supposed to fix the issue (https://gerrit.ovirt.org/82800 and https://gerrit.ovirt.org/#/c/82799/) were merged, but the issue remains.
>
>
> Are we sure that when testing upgrade from 4.1 to master, we are using 4.1.7.XXX from latest tested as a base version? Because it we will try to upgrade from any previous version, the upgrade
>
> will always fail. I'm asking about it, because we wasn't able to find a way how to force manual OST to use latest tested as base version, it always use latest released 4.1.6. That's why Miroslava had to verify that manually

This build upgraded 4.0 -> 4.2.

Daniel will disable it.

I also pushed a patch to prevent it: https://gerrit.ovirt.org/82860 .

>>
>>
>> On Mon, Oct 16, 2017 at 11:45 AM, Sandro Bonazzola <sbonazzo at redhat.com> wrote:
>>>
>>>
>>>
>>> 2017-10-16 10:30 GMT+02:00 Yedidyah Bar David <didi at redhat.com>:
>>>>
>>>> On Mon, Oct 16, 2017 at 11:01 AM, Miroslava Voglova <mvoglova at redhat.com> wrote:
>>>>>
>>>>> On Mon, Oct 16, 2017 at 9:44 AM, Martin Perina <mperina at redhat.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 16, 2017 at 9:38 AM, Yedidyah Bar David <didi at redhat.com> wrote:
>>>>>>>
>>>>>>> On Mon, Oct 16, 2017 at 10:34 AM, Miroslava Voglova <mvoglova at redhat.com> wrote:
>>>>>>>>
>>>>>>>> Fix on review https://gerrit.ovirt.org/#/c/82799/
>>>>>>>
>>>>>>>
>>>>>>> That's indeed a related patch, but not sure how it solves current failure.
>>>>>>
>>>>>>
>>>>>> Let's copy generate-pgpass.sh to packaging/setup/dbutils and source it from this location for taskcleaner.sh and unlock_entity.sh
>>>>>
>>>>>
>>>>>
>>>>> After offline discussion merging https://gerrit.ovirt.org/82800 and then https://gerrit.ovirt.org/#/c/82799/ will fix the issue. Both patches are needed.
>>>>
>>>>
>>>> So this means:
>>>>
>>>> Merge https://gerrit.ovirt.org/82800 . We might want to open a
>>>> real 4.1 bug for this.
>>>
>>>
>>> Agreed, let's open a real 4.1.7 bug to track this and get proper verification by QE.
>>>
>>>>
>>>>
>>>> Build 4.1.7 (or 4.1.8?) with it .
>>>
>>>
>>> Let's point to 4.1.7
>>>
>>>>
>>>>
>>>> Patch 4.2 engine-setup to require tools-4.1.7.
>>>
>>>
>>> well, tools >= 4.1.7 :-)
>>>
>>>
>>>
>>>>
>>>>
>>>> Adding Sandro.
>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Oct 16, 2017 at 9:32 AM, Yaniv Kaul <ykaul at redhat.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Oct 16, 2017 at 10:24 AM, Yedidyah Bar David <didi at redhat.com> wrote:
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 16, 2017 at 10:21 AM, Yedidyah Bar David <didi at redhat.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky <dbelenky at redhat.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> can someone address this issue? every patch to ovirt-engine that is based on top of this patch is failing OST and won't deploy to the tested repo.
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky <dbelenky at redhat.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>> The following tests are failing both of the upgrade suites in OST (upgrade_from_release and upgrade_from_prevrelease).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Link to console: ovirt-master_change-queue-tester/3146/console
>>>>>>>>>>>>> Link to test logs:
>>>>>>>>>>>>>     - upgrade-from-release-suit-master-el7
>>>>>>>>>>>>>     - upgrade-from-prevrelease-suit-master-el7
>>>>>>>>>>>>> Suspected patch: https://gerrit.ovirt.org/#/c/82615/5
>>>>>>>>>>>>> Please note that every patch that is based on top of the patch above was not deployed to the tested repo.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Error snippet from engine setup log:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Please add a direct link next time, if possible. This is it:
>>>>>>>>>>>
>>>>>>>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_upgrade_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log
>>>>>>>>>>>
>>>>>>>>>>> And a bit above the snippet below, there is:
>>>>>>>>>>>
>>>>>>>>>>> 2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.upgrade.asynctasks plugin.execute:926 execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log', '-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q', '-r', '-Z') stderr:
>>>>>>>>>>>
>>>>>>>>>>> /usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory
>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception
>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>   File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod
>>>>>>>>>>>>>     method['method']()
>>>>>>>>>>>>>   File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies
>>>>>>>>>>>>>     self._clearZombies()
>>>>>>>>>>>>>   File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies
>>>>>>>>>>>>>     'Failed to clear zombie commands. '
>>>>>>>>>>>>> RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem
>>>>>>>>>>>>> 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> With [1], taskcleaner.sh sources generate-pgpass.sh .
>>>>>>>>>>>
>>>>>>>>>>> generate-pgpass.sh is in ovirt-engine-tools, which in upgrade flows, is not
>>>>>>>>>>> yet upgraded (at the point of above failure).
>>>>>>>>>>>
>>>>>>>>>>> generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using a path
>>>>>>>>>>> relative to "$0". In master it does not, but we now upgrade and it does.
>>>>>>>>>>>
>>>>>>>>>>> This, in principle, is the core of the bug:
>>>>>>>>>>>
>>>>>>>>>>> A file, such as generate-pgpass.sh, that's supposed to be sourced
>>>>>>>>>>> from some other files, should not by itself source other files
>>>>>>>>>>> that are relative to "$0", because it can't know what "$0" is - it's
>>>>>>>>>>> the path of the script sourcing it, not of itself.
>>>>>>>>>>>
>>>>>>>>>>> It seems like luckily we were not affected by this in 4.1, because
>>>>>>>>>>> all of the files that sourced generate-pgpass.sh were together with
>>>>>>>>>>> it in the same directory. But with [1], taskcleaner does too now,
>>>>>>>>>>> and is in a different directory.
>>>>>>>>>>>
>>>>>>>>>>> Not sure what's the best solution:
>>>>>>>>>>>
>>>>>>>>>>> - revert [1] (and introduce it later on, in 4.3)
>>>>>>>>>>>
>>>>>>>>>>> - patch 4.1's generate-pgpass.sh and require the fixed 4.1 version
>>>>>>>>>>> in 4.2 setup
>>>>>>>
>>>>>>>
>>>>>>> E.g.:
>>>>>>>
>>>>>>> https://gerrit.ovirt.org/82800
>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> - Somehow trick everything to work together? Not sure. Seems like
>>>>>>>>>>> you can't set $0.
>>>>>>>>>>>
>>>>>>>>>>> [1] https://gerrit.ovirt.org/82511
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The same bug exists with unlock_entity:
>>>>>>>>>>
>>>>>>>>>> https://gerrit.ovirt.org/82615
>>>>>>>>>>
>>>>>>>>>> So we should probably revert both.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I tend to agree.
>>>>>>>>> Doesn't mean we cannot fix this for 4.2, but let's revert for the meantime.
>>>>>>>>> Y.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> DANIEL BELENKY
>>>>>>>>>>>>>
>>>>>>>>>>>>> RHV DEVOPS
>>>>>>>>>>>>>
>>>>>>>>>>>>> EMEA VIRTUALIZATION R&D
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> DANIEL BELENKY
>>>>>>>>>>>>
>>>>>>>>>>>> RHV DEVOPS
>>>>>>>>>>>>
>>>>>>>>>>>> EMEA VIRTUALIZATION R&D
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Devel mailing list
>>>>>>>>>>>> Devel at ovirt.org
>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Didi
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Didi
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Devel mailing list
>>>>>>>>>> Devel at ovirt.org
>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Devel mailing list
>>>>>>>>> Devel at ovirt.org
>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Didi
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Devel mailing list
>>>>>>> Devel at ovirt.org
>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Didi
>>>
>>>
>>>
>>>
>>> --
>>>
>>> SANDRO BONAZZOLA
>>>
>>> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
>>>
>>> Red Hat EMEA
>>>
>>> TRIED. TESTED. TRUSTED.
>>>
>>
>>
>>
>> --
>>
>> DANIEL BELENKY
>>
>> RHV DEVOPS
>>
>> EMEA VIRTUALIZATION R&D
>
>



-- 
Didi

More information about the Devel mailing list