Re: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ upgrade suites failed! ] [ 15/10/17 ]

On Tue, Oct 17, 2017 at 1:17 PM, Daniel Belenky <dbelenky(a)redhat.com&gt; wrote: > > Hey, I see that both of the patches that were supposed to fix the issue (https://gerrit.ovirt.org/82800 and https://gerrit.ovirt.org/#/c/82799/) were merged, but the issue remains. Are we sure that when testing upgrade from 4.1 to master, we are using 4.1.7.XXX from latest tested as a base version? Because it we will try to upgrade from any previous version, the upgrade will always fail. I'm asking about it, because we wasn't able to find a way how to force manual OST to use latest tested as base version, it always use latest released 4.1.6. That's why Miroslava had to verify that manually

> > > On Mon, Oct 16, 2017 at 11:45 AM, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; wrote: >> >> >> >> 2017-10-16 10:30 GMT+02:00 Yedidyah Bar David <didi(a)redhat.com&gt;: >>> >>> On Mon, Oct 16, 2017 at 11:01 AM, Miroslava Voglova <mvoglova(a)redhat.com&gt; wrote: >>>> >>>> On Mon, Oct 16, 2017 at 9:44 AM, Martin Perina <mperina(a)redhat.com&gt; wrote: >>>>> >>>>> >>>>> >>>>> On Mon, Oct 16, 2017 at 9:38 AM, Yedidyah Bar David <didi(a)redhat.com&gt; wrote: >>>>>> >>>>>> On Mon, Oct 16, 2017 at 10:34 AM, Miroslava Voglova <mvoglova(a)redhat.com&gt; wrote: >>>>>>> >>>>>>> Fix on review https://gerrit.ovirt.org/#/c/82799/ >>>>>> >>>>>> >>>>>> That's indeed a related patch, but not sure how it solves current failure. >>>>> >>>>> >>>>> Let's copy generate-pgpass.sh to packaging/setup/dbutils and source it from this location for taskcleaner.sh and unlock_entity.sh >>>> >>>> >>>> >>>> After offline discussion merging https://gerrit.ovirt.org/82800 and then https://gerrit.ovirt.org/#/c/82799/ will fix the issue. Both patches are needed. >>> >>> >>> So this means: >>> >>> Merge https://gerrit.ovirt.org/82800 . We might want to open a >>> real 4.1 bug for this. >> >> >> Agreed, let's open a real 4.1.7 bug to track this and get proper verification by QE. >> >>> >>> >>> Build 4.1.7 (or 4.1.8?) with it . >> >> >> Let's point to 4.1.7 >> >>> >>> >>> Patch 4.2 engine-setup to require tools-4.1.7. >> >> >> well, tools >= 4.1.7 :-) >> >> >> >>> >>> >>> Adding Sandro. >>> >>>> >>>> >>>>> >>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Oct 16, 2017 at 9:32 AM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Oct 16, 2017 at 10:24 AM, Yedidyah Bar David <didi(a)redhat.com&gt; wrote: >>>>>>>>> >>>>>>>>> On Mon, Oct 16, 2017 at 10:21 AM, Yedidyah Bar David <didi(a)redhat.com&gt; wrote: >>>>>>>>>> >>>>>>>>>> On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky <dbelenky(a)redhat.com&gt; wrote: >>>>>>>>>>> >>>>>>>>>>> can someone address this issue? every patch to ovirt-engine that is based on top of this patch is failing OST and won't deploy to the tested repo. >>>>>>>>>>> >>>>>>>>>>> On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky <dbelenky(a)redhat.com&gt; wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi all, >>>>>>>>>>>> The following tests are failing both of the upgrade suites in OST (upgrade_from_release and upgrade_from_prevrelease). >>>>>>>>>>>> >>>>>>>>>>>> Link to console: ovirt-master_change-queue-tester/3146/console >>>>>>>>>>>> Link to test logs: >>>>>>>>>>>> - upgrade-from-release-suit-master-el7 >>>>>>>>>>>> - upgrade-from-prevrelease-suit-master-el7 >>>>>>>>>>>> Suspected patch: https://gerrit.ovirt.org/#/c/82615/5 >>>>>>>>>>>> Please note that every patch that is based on top of the patch above was not deployed to the tested repo. >>>>>>>>>>>> >>>>>>>>>>>> Error snippet from engine setup log: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Please add a direct link next time, if possible. This is it: >>>>>>>>>> >>>>>>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifa... >>>>>>>>>> >>>>>>>>>> And a bit above the snippet below, there is: >>>>>>>>>> >>>>>>>>>> 2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.upgrade.asynctasks plugin.execute:926 execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log', '-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q', '-r', '-Z') stderr: >>>>>>>>>> >>>>>>>>>> /usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory >>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception >>>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>>> File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod >>>>>>>>>>>> method['method']() >>>>>>>>>>>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies >>>>>>>>>>>> self._clearZombies() >>>>>>>>>>>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies >>>>>>>>>>>> 'Failed to clear zombie commands. ' >>>>>>>>>>>> RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem >>>>>>>>>>>> 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> With [1], taskcleaner.sh sources generate-pgpass.sh . >>>>>>>>>> >>>>>>>>>> generate-pgpass.sh is in ovirt-engine-tools, which in upgrade flows, is not >>>>>>>>>> yet upgraded (at the point of above failure). >>>>>>>>>> >>>>>>>>>> generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using a path >>>>>>>>>> relative to "$0". In master it does not, but we now upgrade and it does. >>>>>>>>>> >>>>>>>>>> This, in principle, is the core of the bug: >>>>>>>>>> >>>>>>>>>> A file, such as generate-pgpass.sh, that's supposed to be sourced >>>>>>>>>> from some other files, should not by itself source other files >>>>>>>>>> that are relative to "$0", because it can't know what "$0" is - it's >>>>>>>>>> the path of the script sourcing it, not of itself. >>>>>>>>>> >>>>>>>>>> It seems like luckily we were not affected by this in 4.1, because >>>>>>>>>> all of the files that sourced generate-pgpass.sh were together with >>>>>>>>>> it in the same directory. But with [1], taskcleaner does too now, >>>>>>>>>> and is in a different directory. >>>>>>>>>> >>>>>>>>>> Not sure what's the best solution: >>>>>>>>>> >>>>>>>>>> - revert [1] (and introduce it later on, in 4.3) >>>>>>>>>> >>>>>>>>>> - patch 4.1's generate-pgpass.sh and require the fixed 4.1 version >>>>>>>>>> in 4.2 setup >>>>>> >>>>>> >>>>>> E.g.: >>>>>> >>>>>> https://gerrit.ovirt.org/82800 >>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> - Somehow trick everything to work together? Not sure. Seems like >>>>>>>>>> you can't set $0. >>>>>>>>>> >>>>>>>>>> [1] https://gerrit.ovirt.org/82511 >>>>>>>>> >>>>>>>>> >>>>>>>>> The same bug exists with unlock_entity: >>>>>>>>> >>>>>>>>> https://gerrit.ovirt.org/82615 >>>>>>>>> >>>>>>>>> So we should probably revert both. >>>>>>>> >>>>>>>> >>>>>>>> I tend to agree. >>>>>>>> Doesn't mean we cannot fix this for 4.2, but let's revert for the meantime. >>>>>>>> Y. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> DANIEL BELENKY >>>>>>>>>>>> >>>>>>>>>>>> RHV DEVOPS >>>>>>>>>>>> >>>>>>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> DANIEL BELENKY >>>>>>>>>>> >>>>>>>>>>> RHV DEVOPS >>>>>>>>>>> >>>>>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Devel mailing list >>>>>>>>>>> Devel(a)ovirt.org >>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Didi >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Didi >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Devel mailing list >>>>>>>>> Devel(a)ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Devel mailing list >>>>>>>> Devel(a)ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Didi >>>>>> >>>>>> _______________________________________________ >>>>>> Devel mailing list >>>>>> Devel(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>> >>>>> >>>> >>> >>> >>> >>> -- >>> Didi >> >> >> >> >> -- >> >> SANDRO BONAZZOLA >> >> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D >> >> Red Hat EMEA >> >> TRIED. TESTED. TRUSTED. >> > > > > -- > > DANIEL BELENKY > > RHV DEVOPS > > EMEA VIRTUALIZATION R&D