Re: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ upgrade suites failed! ] [ 15/10/17 ]

2017-10-16 10:30 GMT+02:00 Yedidyah Bar David <didi(a)redhat.com&gt;: > On Mon, Oct 16, 2017 at 11:01 AM, Miroslava Voglova <mvoglova(a)redhat.com&gt; > wrote: > >> On Mon, Oct 16, 2017 at 9:44 AM, Martin Perina <mperina(a)redhat.com&gt; >> wrote: >> >>> >>> >>> On Mon, Oct 16, 2017 at 9:38 AM, Yedidyah Bar David <didi(a)redhat.com&gt; >>> wrote: >>> >>>> On Mon, Oct 16, 2017 at 10:34 AM, Miroslava Voglova < >>>> mvoglova(a)redhat.com&gt; wrote: >>>> >>>>> Fix on review https://gerrit.ovirt.org/#/c/82799/ >>>>> >>>> >>>> That's indeed a related patch, but not sure how it solves current >>>> failure. >>>> >>> >>> ​Let's copy generate-pgpass.sh to packaging/setup/dbutils a​nd source >>> it from this location for taskcleaner.sh and unlock_entity.sh >>> >> >> >> After offline discussion merging https://gerrit.ovirt.org/82800 and >> then https://gerrit.ovirt.org/#/c/82799/ will fix the issue. Both >> patches are needed. >> > > So this means: > > Merge https://gerrit.ovirt.org/82800 . We might want to open a > real 4.1 bug for this. > Agreed, let's open a real 4.1.7 bug to track this and get proper verification by QE. > > Build 4.1.7 (or 4.1.8?) with it . > Let's point to 4.1.7 > > Patch 4.2 engine-setup to require tools-4.1.7. > well, tools >= 4.1.7 :-) > > Adding Sandro. > > >> >> >>> >>> >>>> >>>>> >>>>> On Mon, Oct 16, 2017 at 9:32 AM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: >>>>> >>>>>> >>>>>> >>>>>> On Mon, Oct 16, 2017 at 10:24 AM, Yedidyah Bar David < >>>>>> didi(a)redhat.com&gt; wrote: >>>>>> >>>>>>> On Mon, Oct 16, 2017 at 10:21 AM, Yedidyah Bar David < >>>>>>> didi(a)redhat.com&gt; wrote: >>>>>>> >>>>>>>> On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky < >>>>>>>> dbelenky(a)redhat.com&gt; wrote: >>>>>>>> >>>>>>>>> can someone address this issue? every patch to *ovirt-engine* that >>>>>>>>> is based on top of this patch is failing OST and* won't deploy >>>>>>>>> to the tested repo*. >>>>>>>>> >>>>>>>>> On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky < >>>>>>>>> dbelenky(a)redhat.com&gt; wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> The following tests are failing both of the upgrade suites in >>>>>>>>>> OST (upgrade_from_release and upgrade_from_prevrelease). >>>>>>>>>> >>>>>>>>>> *Link to console:* ovirt-master_change-queue-tester/3146/console >>>>>>>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/consol... >>>>>>>>>> *Link to test logs:* >>>>>>>>>> - upgrade-from-release-suit-master-el7 >>>>>>>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifa... >>>>>>>>>> - upgrade-from-prevrelease-suit-master-el7 >>>>>>>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifa... >>>>>>>>>> *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5 >>>>>>>>>> *Please note that every patch that is based on top of the patch >>>>>>>>>> above was not deployed to the tested repo.* >>>>>>>>>> >>>>>>>>>> *Error snippet from engine setup log:* >>>>>>>>>> >>>>>>>>> >>>>>>>> Please add a direct link next time, if possible. This is it: >>>>>>>> >>>>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste >>>>>>>> r/3146/artifact/exported-artifacts/upgrade-from-release-suit >>>>>>>> -master-el7/test_logs/upgrade-from-release-suite-master/post >>>>>>>> -001_upgrade_engine.py/lago-upgrade-from-release-suite-maste >>>>>>>> r-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-2017 >>>>>>>> 1013222617-73f0df.log >>>>>>>> >>>>>>>> And a bit above the snippet below, there is: >>>>>>>> >>>>>>>> 2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_set >>>>>>>> up.ovirt_engine.upgrade.asynctasks plugin.execute:926 >>>>>>>> execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', >>>>>>>> '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log', >>>>>>>> '-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q', >>>>>>>> '-r', '-Z') stderr: >>>>>>>> >>>>>>>> /usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory >>>>>>>> >>>>>>>> >>>>>>>> 2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception >>>>>>>>>> Traceback (most recent call last): >>>>>>>>>> File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod >>>>>>>>>> method['method']() >>>>>>>>>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies >>>>>>>>>> self._clearZombies() >>>>>>>>>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies >>>>>>>>>> 'Failed to clear zombie commands. ' >>>>>>>>>> RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem >>>>>>>>>> 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem >>>>>>>>>> >>>>>>>>>> >>>>>>>> With [1], taskcleaner.sh sources generate-pgpass.sh . >>>>>>>> >>>>>>>> generate-pgpass.sh is in ovirt-engine-tools, which in upgrade >>>>>>>> flows, is not >>>>>>>> yet upgraded (at the point of above failure). >>>>>>>> >>>>>>>> generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using >>>>>>>> a path >>>>>>>> relative to "$0". In master it does not, but we now upgrade and it >>>>>>>> does. >>>>>>>> >>>>>>>> This, in principle, is the core of the bug: >>>>>>>> >>>>>>>> A file, such as generate-pgpass.sh, that's supposed to be sourced >>>>>>>> from some other files, should not by itself source other files >>>>>>>> that are relative to "$0", because it can't know what "$0" is - >>>>>>>> it's >>>>>>>> the path of the script sourcing it, not of itself. >>>>>>>> >>>>>>>> It seems like luckily we were not affected by this in 4.1, because >>>>>>>> all of the files that sourced generate-pgpass.sh were together with >>>>>>>> it in the same directory. But with [1], taskcleaner does too now, >>>>>>>> and is in a different directory. >>>>>>>> >>>>>>>> Not sure what's the best solution: >>>>>>>> >>>>>>>> - revert [1] (and introduce it later on, in 4.3) >>>>>>>> >>>>>>>> - patch 4.1's generate-pgpass.sh and require the fixed 4.1 version >>>>>>>> in 4.2 setup >>>>>>>> >>>>>>> >>>> E.g.: >>>> >>>> https://gerrit.ovirt.org/82800 >>>> >>>> >>>>> >>>>>>>> - Somehow trick everything to work together? Not sure. Seems like >>>>>>>> you can't set $0. >>>>>>>> >>>>>>>> [1] https://gerrit.ovirt.org/82511 >>>>>>>> >>>>>>> >>>>>>> The same bug exists with unlock_entity: >>>>>>> >>>>>>> https://gerrit.ovirt.org/82615 >>>>>>> >>>>>>> So we should probably revert both. >>>>>>> >>>>>> >>>>>> I tend to agree. >>>>>> Doesn't mean we cannot fix this for 4.2, but let's revert for the >>>>>> meantime. >>>>>> Y. >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> >>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> DANIEL BELENKY >>>>>>>>>> >>>>>>>>>> RHV DEVOPS >>>>>>>>>> >>>>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>>>> <https://red.ht/sig> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> DANIEL BELENKY >>>>>>>>> >>>>>>>>> RHV DEVOPS >>>>>>>>> >>>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>>> <https://red.ht/sig> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Devel mailing list >>>>>>>>> Devel(a)ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Didi >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Didi >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Devel mailing list >>>>>>> Devel(a)ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Devel mailing list >>>>>> Devel(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Didi >>>> >>>> _______________________________________________ >>>> Devel mailing list >>>> Devel(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>> >>> >> > > > -- > Didi > -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> <http://www.teraplan.it/redhat-osd-2017/>