Re: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ upgrade suites failed! ] [ 15/10/17 ]

On Mon, Oct 16, 2017 at 9:44 AM, Martin Perina <mperina(a)redhat.com&gt; wrote: > > > On Mon, Oct 16, 2017 at 9:38 AM, Yedidyah Bar David <didi(a)redhat.com&gt; > wrote: > >> On Mon, Oct 16, 2017 at 10:34 AM, Miroslava Voglova <mvoglova(a)redhat.com >> > wrote: >> >>> Fix on review https://gerrit.ovirt.org/#/c/82799/ >>> >> >> That's indeed a related patch, but not sure how it solves current >> failure. >> > > ​Let's copy generate-pgpass.sh to packaging/setup/dbutils a​nd source it > from this location for taskcleaner.sh and unlock_entity.sh > After offline discussion merging https://gerrit.ovirt.org/82800 and then https://gerrit.ovirt.org/#/c/82799/ will fix the issue. Both patches are needed.

> > >> >>> >>> On Mon, Oct 16, 2017 at 9:32 AM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: >>> >>>> >>>> >>>> On Mon, Oct 16, 2017 at 10:24 AM, Yedidyah Bar David <didi(a)redhat.com&gt; >>>> wrote: >>>> >>>>> On Mon, Oct 16, 2017 at 10:21 AM, Yedidyah Bar David <didi(a)redhat.com >>>>> > wrote: >>>>> >>>>>> On Mon, Oct 16, 2017 at 9:28 AM, Daniel Belenky <dbelenky(a)redhat.com >>>>>> > wrote: >>>>>> >>>>>>> can someone address this issue? every patch to *ovirt-engine* that >>>>>>> is based on top of this patch is failing OST and* won't deploy to >>>>>>> the tested repo*. >>>>>>> >>>>>>> On Sun, Oct 15, 2017 at 9:33 AM, Daniel Belenky < >>>>>>> dbelenky(a)redhat.com&gt; wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> The following tests are failing both of the upgrade suites in OST >>>>>>>> (upgrade_from_release and upgrade_from_prevrelease). >>>>>>>> >>>>>>>> *Link to console:* ovirt-master_change-queue-tester/3146/console >>>>>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/consol... >>>>>>>> *Link to test logs:* >>>>>>>> - upgrade-from-release-suit-master-el7 >>>>>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifa... >>>>>>>> - upgrade-from-prevrelease-suit-master-el7 >>>>>>>> <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3146/artifa... >>>>>>>> *Suspected patch:* https://gerrit.ovirt.org/#/c/82615/5 >>>>>>>> *Please note that every patch that is based on top of the patch >>>>>>>> above was not deployed to the tested repo.* >>>>>>>> >>>>>>>> *Error snippet from engine setup log:* >>>>>>>> >>>>>>> >>>>>> Please add a direct link next time, if possible. This is it: >>>>>> >>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste >>>>>> r/3146/artifact/exported-artifacts/upgrade-from-release-suit >>>>>> -master-el7/test_logs/upgrade-from-release-suite-master/post >>>>>> -001_upgrade_engine.py/lago-upgrade-from-release-suite-maste >>>>>> r-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-2017 >>>>>> 1013222617-73f0df.log >>>>>> >>>>>> And a bit above the snippet below, there is: >>>>>> >>>>>> 2017-10-13 22:26:24,274-0400 DEBUG otopi.plugins.ovirt_engine_set >>>>>> up.ovirt_engine.upgrade.asynctasks plugin.execute:926 >>>>>> execute-output: ('/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh', >>>>>> '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20171013222617-73f0df.log', >>>>>> '-u', 'engine', '-s', 'localhost', '-p', '5432', '-d', 'engine', '-q', >>>>>> '-r', '-Z') stderr: >>>>>> >>>>>> /usr/share/ovirt-engine/bin/generate-pgpass.sh: line 3: /usr/share/ovirt-engine/setup/dbutils/engine-prolog.sh: No such file or directory >>>>>> >>>>>> >>>>>> 2017-10-13 22:26:24,274-0400 DEBUG otopi.context context._executeMethod:143 method exception >>>>>>>> Traceback (most recent call last): >>>>>>>> File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod >>>>>>>> method['method']() >>>>>>>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 470, in _validateZombies >>>>>>>> self._clearZombies() >>>>>>>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 135, in _clearZombies >>>>>>>> 'Failed to clear zombie commands. ' >>>>>>>> RuntimeError: Failed to clear zombie commands. Please access support in attempt to resolve the problem >>>>>>>> 2017-10-13 22:26:24,275-0400 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem >>>>>>>> >>>>>>>> >>>>>> With [1], taskcleaner.sh sources generate-pgpass.sh . >>>>>> >>>>>> generate-pgpass.sh is in ovirt-engine-tools, which in upgrade flows, >>>>>> is not >>>>>> yet upgraded (at the point of above failure). >>>>>> >>>>>> generate-pgpass.sh in 4.1 used to source engine-prolog.sh , using a >>>>>> path >>>>>> relative to "$0". In master it does not, but we now upgrade and it >>>>>> does. >>>>>> >>>>>> This, in principle, is the core of the bug: >>>>>> >>>>>> A file, such as generate-pgpass.sh, that's supposed to be sourced >>>>>> from some other files, should not by itself source other files >>>>>> that are relative to "$0", because it can't know what "$0" is - it's >>>>>> the path of the script sourcing it, not of itself. >>>>>> >>>>>> It seems like luckily we were not affected by this in 4.1, because >>>>>> all of the files that sourced generate-pgpass.sh were together with >>>>>> it in the same directory. But with [1], taskcleaner does too now, >>>>>> and is in a different directory. >>>>>> >>>>>> Not sure what's the best solution: >>>>>> >>>>>> - revert [1] (and introduce it later on, in 4.3) >>>>>> >>>>>> - patch 4.1's generate-pgpass.sh and require the fixed 4.1 version >>>>>> in 4.2 setup >>>>>> >>>>> >> E.g.: >> >> https://gerrit.ovirt.org/82800 >> >> >>> >>>>>> - Somehow trick everything to work together? Not sure. Seems like >>>>>> you can't set $0. >>>>>> >>>>>> [1] https://gerrit.ovirt.org/82511 >>>>>> >>>>> >>>>> The same bug exists with unlock_entity: >>>>> >>>>> https://gerrit.ovirt.org/82615 >>>>> >>>>> So we should probably revert both. >>>>> >>>> >>>> I tend to agree. >>>> Doesn't mean we cannot fix this for 4.2, but let's revert for the >>>> meantime. >>>> Y. >>>> >>>> >>>>> >>>>> >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> >>>>>>> -- >>>>>>>> >>>>>>>> DANIEL BELENKY >>>>>>>> >>>>>>>> RHV DEVOPS >>>>>>>> >>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>> <https://red.ht/sig> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> DANIEL BELENKY >>>>>>> >>>>>>> RHV DEVOPS >>>>>>> >>>>>>> EMEA VIRTUALIZATION R&D >>>>>>> <https://red.ht/sig> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Devel mailing list >>>>>>> Devel(a)ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Didi >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Didi >>>>> >>>>> _______________________________________________ >>>>> Devel mailing list >>>>> Devel(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Devel mailing list >>>> Devel(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>> >>> >> >> >> -- >> Didi >> >> _______________________________________________ >> Devel mailing list >> Devel(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel >> > >