[ovirt-devel] Re: package versioning problem causing failure on CQ ovirt-master on all projects

On Wed, 6 Feb 2019 at 11:57, Simone Tiraboschi <stirabos(a)redhat.com&gt; wrote: > > > On Wed, Feb 6, 2019 at 10:44 AM Barak Korren <bkorren(a)redhat.com&gt; wrote: > >> >> >> On Wed, 6 Feb 2019 at 11:34, Simone Tiraboschi <stirabos(a)redhat.com&gt; >> wrote: >> >>> >>> >>> On Wed, Feb 6, 2019 at 10:23 AM Barak Korren <bkorren(a)redhat.com&gt; >>> wrote: >>> >>>> >>>> >>>> On Wed, 6 Feb 2019 at 11:15, Simone Tiraboschi <stirabos(a)redhat.com&gt; >>>> wrote: >>>> >>>>> >>>>> >>>>> On Wed, Feb 6, 2019 at 10:00 AM Dan Kenigsberg <danken(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> On Wed, Feb 6, 2019 at 10:54 AM Simone Tiraboschi < >>>>>> stirabos(a)redhat.com&gt; wrote: >>>>>> > >>>>>> > >>>>>> > >>>>>> > On Wed, Feb 6, 2019 at 9:45 AM Dan Kenigsberg <danken(a)redhat.com&gt; >>>>>> wrote: >>>>>> >> >>>>>> >> On Wed, Feb 6, 2019 at 10:16 AM Simone Tiraboschi < >>>>>> stirabos(a)redhat.com&gt; wrote: >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > On Tue, Feb 5, 2019 at 7:07 PM Dafna Ron <dron(a)redhat.com&gt; >>>>>> wrote: >>>>>> >> >> >>>>>> >> >> Hi, >>>>>> >> >> >>>>>> >> >> Please note that ovirt-ansible-hosted-engine-setup has a >>>>>> versioning problem with the package and is causing bootstrap to fail for >>>>>> upgrade suite [1] >>>>>> >> >> >>>>>> >> >> This is effecting all projects, its been reported to the >>>>>> developers and should be fixed as soon as possible. >>>>>> >> >> >>>>>> >> >> you can view CQ status here: >>>>>> >> >> >>>>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_cha... >>>>>> >> >> >>>>>> >> >> [1] http://pastebin.test.redhat.com/708086 >>>>>> >> >>>>>> >> It is unfair to refer to an internal pastebin here. It is also not >>>>>> >> very sensible, as it is quite short. >>>>>> >> >>>>>> >> 2019-02-05 11:23:51,390-0500 ERROR >>>>>> >> otopi.plugins.otopi.packagers.yumpackager yumpackager.error:85 Yum >>>>>> >> >>>>>> [u'ovirt-hosted-engine-setup-2.3.5-0.0.master.20190205110929.gitfdbc215.el7.noarch >>>>>> >> requires ovirt-ansible-hosted-engine-setup >= 1.0.10'] >>>>>> >> 2019-02-05 11:23:51,390-0500 DEBUG otopi.context >>>>>> >> context._executeMethod:142 method exception >>>>>> >> Traceback (most recent call last): >>>>>> >> File "/tmp/ovirt-6fV8LBWX5i/pythonlib/otopi/context.py", line >>>>>> 132, >>>>>> >> in _executeMethod >>>>>> >> method['method']() >>>>>> >> File >>>>>> "/tmp/ovirt-6fV8LBWX5i/otopi-plugins/otopi/packagers/yumpackager.py", >>>>>> >> line 248, in _packages >>>>>> >> self.processTransaction() >>>>>> >> File >>>>>> "/tmp/ovirt-6fV8LBWX5i/otopi-plugins/otopi/packagers/yumpackager.py", >>>>>> >> line 262, in processTransaction >>>>>> >> if self._miniyum.buildTransaction(): >>>>>> >> File "/tmp/ovirt-6fV8LBWX5i/pythonlib/otopi/miniyum.py", line >>>>>> 920, >>>>>> >> in buildTransaction >>>>>> >> raise yum.Errors.YumBaseError(msg) >>>>>> >> YumBaseError: >>>>>> [u'ovirt-hosted-engine-setup-2.3.5-0.0.master.20190205110929.gitfdbc215.el7.noarch >>>>>> >> requires ovirt-ansible-hosted-engine-setup >= 1.0.10'] >>>>>> >> 2019-02-05 11:23:51,391-0500 ERROR otopi.context >>>>>> >> context._executeMethod:151 Failed to execute stage 'Package >>>>>> >> installation': >>>>>> [u'ovirt-hosted-engine-setup-2.3.5-0.0.master.20190205110929.gitfdbc215.el7.noarch >>>>>> >> requires ovirt-ansible-hosted-engine-setup >= 1.0.10'] >>>>>> >> 2019-02-05 11:23:51,413-0500 DEBUG >>>>>> >> otopi.plugins.otopi.debug.debug_failure.debug_failure >>>>>> >> debug_failure._notification:100 tcp connections: >>>>>> >> >>>>>> >> >> >>>>>> >> > >>>>>> >> > The issue is that on github we already have >>>>>> >> > VERSION="1.0.10" >>>>>> >> > as we can see in >>>>>> >> > >>>>>> https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/blob/master/bu... >>>>>> >> > >>>>>> >> > And this has been bumped before the commit that now is reported >>>>>> as broken. >>>>>> >> > >>>>>> >> > CI instead is still building the package as 1.0.9 ignoring the >>>>>> commit that bumped the version. >>>>>> >> > Honestly I don't know how I can fix it if the version value is >>>>>> already the desired one in the source code. >>>>>> >> >>>>>> >> I don't see your ovirt-ansible-hosted-engine-setup-1.0.10, only >>>>>> >> >>>>>> https://plain.resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7/noarc... >>>>>> >> Not even under "tested": >>>>>> >> >>>>>> https://plain.resources.ovirt.org/repos/ovirt/tested/master/rpm/el7/noarc... >>>>>> >> >>>>>> >> Simone, can you doublecheck that its artifacts have been built and >>>>>> >> have been accepted by the change queue? >>>>>> > >>>>>> > >>>>>> > It has been built here once as 1.0.10: >>>>>> > >>>>>> https://jenkins.ovirt.org/job/oVirt_ovirt-ansible-hosted-engine-setup_sta... >>>>>> > >>>>>> > then on the next commit, CI started building it again as 1.0.9 >>>>>> although in the source code we have 1.0.10 and so this issue. >>>>>> >>>>>> I don't understand the issue yet (that's not surprising as I do not >>>>>> know what is that "ghpush" job). Which CI job has built the wrong >>>>>> version? can you share its logs? who owns it? >>>>>> >>>>> >>>>> In the git log I see: >>>>> commit b5a6c1db135d81d75f3330160e7ef4a84c97fd60 (HEAD -> master, >>>>> upstream/master, origin/master, origin/HEAD, nolog) >>>>> Author: Simone Tiraboschi <stirabos(a)redhat.com&gt; >>>>> Date: Tue Feb 5 10:56:58 2019 +0100 >>>>> >>>>> Avoid using no_log when we have to pass back values to otopi >>>>> >>>>> commit 96974fad1ee6aee33f8183e49240f8a2a7a617d4 >>>>> Author: Simone Tiraboschi <stirabos(a)redhat.com&gt; >>>>> Date: Thu Jan 31 16:39:58 2019 +0100 >>>>> >>>>> use dynamic inclusion to avoid tag inheritance >>>>> >>>>> commit 4a9a23fb8e88acba5af4febed43d9e4b02e7a2c5 >>>>> Author: Simone Tiraboschi <stirabos(a)redhat.com&gt; >>>>> Date: Thu Jan 31 15:15:04 2019 +0100 >>>>> >>>>> Force facts gathering on partial executions >>>>> >>>>> commit 7428b54a5ba8458379b1a27d116f9504bb830e69 >>>>> Author: Simone Tiraboschi <stirabos(a)redhat.com&gt; >>>>> Date: Wed Jan 30 17:01:01 2019 +0100 >>>>> >>>>> Use static imports and tags >>>>> >>>>> Fixes >>>>> https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/issues/20 >>>>> Reuires >>>>> https://github.com/oVirt/ovirt-ansible-engine-setup/pull/39 >>>>> >>>>> >>>>> >>>>> Version has been bumped to 1.0.10 on commit >>>>> 7428b54a5ba8458379b1a27d116f9504bb830e69 since it introduces a backward >>>>> incompatible change and we need to track it. >>>>> >>>>> 7428b54a5ba8458379b1a27d116f9504bb830e69 failed CI tests due to an >>>>> issue on a different package found yesterday. >>>>> >>>>> So 7428b54a5ba8458379b1a27d116f9504bb830e69 got ignored and now CI is >>>>> building from commit b5a6c1db135d81d75f3330160e7ef4a84c97fd60 (the last >>>>> one) rebased on something before 7428b54a5ba8458379b1a27d116f9504bb830e69 >>>>> which is not what we have in git so now, after >>>>> b5a6c1db135d81d75f3330160e7ef4a84c97fd60 (last commit) the package builds >>>>> in CI as 1.0.9 although in the code we have 1.0.10 and so the issue. >>>>> >>>>> >>>> We never ignore commits, certainly not merged ones... >>>> >>>> We can fall back to older builds on system test failures and throw >>>> away newer build if we suspect they cause the failure, if which case the >>>> builds need to be resubmitted, but this logic happens at the build leve not >>>> the commit level, there is no commit reordering or dropping anywhere. >>>> >>>> >>> I'd double check the CI code for that: >>> >>> on git side, in build.sh at commit >>> b5a6c1db135d81d75f3330160e7ef4a84c97fd60 we have >>> VERSION="1.0.10" >>> as you can see in >>> >>> https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/blob/b5a6c1db1... >>> >>> but, as you can see in >>> >>> https://jenkins.ovirt.org/job/oVirt_ovirt-ansible-hosted-engine-setup_sta... >>> CI builds it as 1.0.9 ignoring commit >>> 7428b54a5ba8458379b1a27d116f9504bb830e69 >>> that has been merged before b5a6c1db135d81d75f3330160e7ef4a84c97fd60 >>> >>> >> I'm not sure if build 138 ran before or after >> 7428b54a5ba8458379b1a27d116f9504bb830e69 was merged. >> since you merged the PR already, we cannot check what will happen if we >> rerun the job now. >> >> I suspect you're simply looking at the output of a job that stated >> running before 7428b54a5ba8458379b1a27d116f9504bb830e69 was merged >> > > On master branch on > https://github.com/oVirt/ovirt-ansible-hosted-engine-setup I have: > > b5a6c1d (HEAD -> master, upstream/master, origin/master, origin/HEAD, > nolog) Avoid using no_log when we have to pass back values to otopi > 96974fa use dynamic inclusion to avoid tag inheritance > 4a9a23f Force facts gathering on partial executions > 7428b54 Use static imports and tags > 9267118 Fix rhbz1654697 (#108) > 5e928a1 build: post ovirt-ansible-hosted-engine-setup-1.0.8 > 8607d90 (tag: 1.0.8) build: ovirt-ansible-hosted-engine-setup-1.0.8 > > In the console output of > https://jenkins.ovirt.org/job/oVirt_ovirt-ansible-hosted-engine-setup_sta... > > I see > > *10:59:42* 2019-02-05 09:59:42,448:INFO:__main__:File 'automation/upstream_sources.yaml' cannot be opened*10:59:42* 2019-02-05 09:59:42,449:INFO:__main__:Executing command: 'git --git-dir=/home/jenkins/workspace/oVirt_ovirt-ansible-hosted-engine-setup_standard-check-pr/ovirt-ansible-hosted-engine-setup/.git --work-tree=/home/jenkins/workspace/oVirt_ovirt-ansible-hosted-engine-setup_standard-check-pr/ovirt-ansible-hosted-engine-setup reset --hard'*10:59:42* 2019-02-05 09:59:42,472:DEBUG:__main__:Git exited with status: 0*10:59:42* 2019-02-05 09:59:42,472:DEBUG:__main__: ---- stderr ----*10:59:42* 2019-02-05 09:59:42,472:DEBUG:__main__: ---- stdout ----*10:59:42* 2019-02-05 09:59:42,472:DEBUG:__main__: HEAD is now at af00824 Merge ceca0a852d94bd5af7226b15d15c9e3ff917cd5c into 9267118f456de46b9059e76b63bc85c18fcab5dd > > > Now, ceca0a852d94bd5af7226b15d15c9e3ff917cd5c is the last commit on my > develop branch on > https://github.com/tiraboschi/ovirt-ansible-hosted-engine-setup > > > commit ceca0a852d94bd5af7226b15d15c9e3ff917cd5c (origin/nolog) > Author: Simone Tiraboschi <stirabos(a)redhat.com&gt; > Date: Tue Feb 5 10:56:58 2019 +0100 > > Avoid using no_log when we have to pass back values to otopi > > > which corresponds to b5a6c1d on master on > https://github.com/oVirt/ovirt-ansible-hosted-engine-setup > > so "Avoid using no_log when we have to pass back values to otopi" got > tested over 9267118 - Fix rhbz1654697 (#108) > skipping > 96974fa use dynamic inclusion to avoid tag inheritance > 4a9a23f Force facts gathering on partial executions > 7428b54 Use static imports and tags > > in the middle. > > The version bump change is in 7428b54 > and so it builds as 1.0.9 and not as 1.0.10 as expected. > > This is a matter of timing - as I already said - I suspect the build you're looking at happened before b5a6c1d was merged. In any case the build you're seeing ot the `check-pr` job has nothing to do with what gets sent into system testing and checked with OST. The build that gets checked is the one that is generated by the `on-ghpush` job and is always built from merged commits in the branch itself. You can see the last successful build is with the right version: https://jenkins.ovirt.org/job/oVirt_ovirt-ansible-hosted-engine-setup_sta...

> > >> >> >>> >>>> -- >>>> Barak Korren >>>> RHV DevOps team , RHCE, RHCi >>>> Red Hat EMEA >>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>>> >>> >> >> -- >> Barak Korren >> RHV DevOps team , RHCE, RHCi >> Red Hat EMEA >> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >> > -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted