[ovirt-devel] Re: package versioning problem causing failure on CQ ovirt-master on all projects

On Wed, 6 Feb 2019 at 11:34, Simone Tiraboschi <stirabos(a)redhat.com&gt; wrote: > > > On Wed, Feb 6, 2019 at 10:23 AM Barak Korren <bkorren(a)redhat.com&gt; wrote: > >> >> >> On Wed, 6 Feb 2019 at 11:15, Simone Tiraboschi <stirabos(a)redhat.com&gt; >> wrote: >> >>> >>> >>> On Wed, Feb 6, 2019 at 10:00 AM Dan Kenigsberg <danken(a)redhat.com&gt; >>> wrote: >>> >>>> On Wed, Feb 6, 2019 at 10:54 AM Simone Tiraboschi <stirabos(a)redhat.com&gt; >>>> wrote: >>>> > >>>> > >>>> > >>>> > On Wed, Feb 6, 2019 at 9:45 AM Dan Kenigsberg <danken(a)redhat.com&gt; >>>> wrote: >>>> >> >>>> >> On Wed, Feb 6, 2019 at 10:16 AM Simone Tiraboschi < >>>> stirabos(a)redhat.com&gt; wrote: >>>> >> > >>>> >> > >>>> >> > >>>> >> > On Tue, Feb 5, 2019 at 7:07 PM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>> >> >> >>>> >> >> Hi, >>>> >> >> >>>> >> >> Please note that ovirt-ansible-hosted-engine-setup has a >>>> versioning problem with the package and is causing bootstrap to fail for >>>> upgrade suite [1] >>>> >> >> >>>> >> >> This is effecting all projects, its been reported to the >>>> developers and should be fixed as soon as possible. >>>> >> >> >>>> >> >> you can view CQ status here: >>>> >> >> >>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_cha... >>>> >> >> >>>> >> >> [1] http://pastebin.test.redhat.com/708086 >>>> >> >>>> >> It is unfair to refer to an internal pastebin here. It is also not >>>> >> very sensible, as it is quite short. >>>> >> >>>> >> 2019-02-05 11:23:51,390-0500 ERROR >>>> >> otopi.plugins.otopi.packagers.yumpackager yumpackager.error:85 Yum >>>> >> >>>> [u'ovirt-hosted-engine-setup-2.3.5-0.0.master.20190205110929.gitfdbc215.el7.noarch >>>> >> requires ovirt-ansible-hosted-engine-setup >= 1.0.10'] >>>> >> 2019-02-05 11:23:51,390-0500 DEBUG otopi.context >>>> >> context._executeMethod:142 method exception >>>> >> Traceback (most recent call last): >>>> >> File "/tmp/ovirt-6fV8LBWX5i/pythonlib/otopi/context.py", line 132, >>>> >> in _executeMethod >>>> >> method['method']() >>>> >> File >>>> "/tmp/ovirt-6fV8LBWX5i/otopi-plugins/otopi/packagers/yumpackager.py", >>>> >> line 248, in _packages >>>> >> self.processTransaction() >>>> >> File >>>> "/tmp/ovirt-6fV8LBWX5i/otopi-plugins/otopi/packagers/yumpackager.py", >>>> >> line 262, in processTransaction >>>> >> if self._miniyum.buildTransaction(): >>>> >> File "/tmp/ovirt-6fV8LBWX5i/pythonlib/otopi/miniyum.py", line 920, >>>> >> in buildTransaction >>>> >> raise yum.Errors.YumBaseError(msg) >>>> >> YumBaseError: >>>> [u'ovirt-hosted-engine-setup-2.3.5-0.0.master.20190205110929.gitfdbc215.el7.noarch >>>> >> requires ovirt-ansible-hosted-engine-setup >= 1.0.10'] >>>> >> 2019-02-05 11:23:51,391-0500 ERROR otopi.context >>>> >> context._executeMethod:151 Failed to execute stage 'Package >>>> >> installation': >>>> [u'ovirt-hosted-engine-setup-2.3.5-0.0.master.20190205110929.gitfdbc215.el7.noarch >>>> >> requires ovirt-ansible-hosted-engine-setup >= 1.0.10'] >>>> >> 2019-02-05 11:23:51,413-0500 DEBUG >>>> >> otopi.plugins.otopi.debug.debug_failure.debug_failure >>>> >> debug_failure._notification:100 tcp connections: >>>> >> >>>> >> >> >>>> >> > >>>> >> > The issue is that on github we already have >>>> >> > VERSION="1.0.10" >>>> >> > as we can see in >>>> >> > >>>> https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/blob/master/bu... >>>> >> > >>>> >> > And this has been bumped before the commit that now is reported >>>> as broken. >>>> >> > >>>> >> > CI instead is still building the package as 1.0.9 ignoring the >>>> commit that bumped the version. >>>> >> > Honestly I don't know how I can fix it if the version value is >>>> already the desired one in the source code. >>>> >> >>>> >> I don't see your ovirt-ansible-hosted-engine-setup-1.0.10, only >>>> >> >>>> https://plain.resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7/noarc... >>>> >> Not even under "tested": >>>> >> >>>> https://plain.resources.ovirt.org/repos/ovirt/tested/master/rpm/el7/noarc... >>>> >> >>>> >> Simone, can you doublecheck that its artifacts have been built and >>>> >> have been accepted by the change queue? >>>> > >>>> > >>>> > It has been built here once as 1.0.10: >>>> > >>>> https://jenkins.ovirt.org/job/oVirt_ovirt-ansible-hosted-engine-setup_sta... >>>> > >>>> > then on the next commit, CI started building it again as 1.0.9 >>>> although in the source code we have 1.0.10 and so this issue. >>>> >>>> I don't understand the issue yet (that's not surprising as I do not >>>> know what is that "ghpush" job). Which CI job has built the wrong >>>> version? can you share its logs? who owns it? >>>> >>> >>> In the git log I see: >>> commit b5a6c1db135d81d75f3330160e7ef4a84c97fd60 (HEAD -> master, >>> upstream/master, origin/master, origin/HEAD, nolog) >>> Author: Simone Tiraboschi <stirabos(a)redhat.com&gt; >>> Date: Tue Feb 5 10:56:58 2019 +0100 >>> >>> Avoid using no_log when we have to pass back values to otopi >>> >>> commit 96974fad1ee6aee33f8183e49240f8a2a7a617d4 >>> Author: Simone Tiraboschi <stirabos(a)redhat.com&gt; >>> Date: Thu Jan 31 16:39:58 2019 +0100 >>> >>> use dynamic inclusion to avoid tag inheritance >>> >>> commit 4a9a23fb8e88acba5af4febed43d9e4b02e7a2c5 >>> Author: Simone Tiraboschi <stirabos(a)redhat.com&gt; >>> Date: Thu Jan 31 15:15:04 2019 +0100 >>> >>> Force facts gathering on partial executions >>> >>> commit 7428b54a5ba8458379b1a27d116f9504bb830e69 >>> Author: Simone Tiraboschi <stirabos(a)redhat.com&gt; >>> Date: Wed Jan 30 17:01:01 2019 +0100 >>> >>> Use static imports and tags >>> >>> Fixes >>> https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/issues/20 >>> Reuires https://github.com/oVirt/ovirt-ansible-engine-setup/pull/39 >>> >>> >>> >>> Version has been bumped to 1.0.10 on commit >>> 7428b54a5ba8458379b1a27d116f9504bb830e69 since it introduces a backward >>> incompatible change and we need to track it. >>> >>> 7428b54a5ba8458379b1a27d116f9504bb830e69 failed CI tests due to an >>> issue on a different package found yesterday. >>> >>> So 7428b54a5ba8458379b1a27d116f9504bb830e69 got ignored and now CI is >>> building from commit b5a6c1db135d81d75f3330160e7ef4a84c97fd60 (the last >>> one) rebased on something before 7428b54a5ba8458379b1a27d116f9504bb830e69 >>> which is not what we have in git so now, after >>> b5a6c1db135d81d75f3330160e7ef4a84c97fd60 (last commit) the package builds >>> in CI as 1.0.9 although in the code we have 1.0.10 and so the issue. >>> >>> >> We never ignore commits, certainly not merged ones... >> >> We can fall back to older builds on system test failures and throw away >> newer build if we suspect they cause the failure, if which case the builds >> need to be resubmitted, but this logic happens at the build leve not the >> commit level, there is no commit reordering or dropping anywhere. >> >> > I'd double check the CI code for that: > > on git side, in build.sh at commit > b5a6c1db135d81d75f3330160e7ef4a84c97fd60 we have > VERSION="1.0.10" > as you can see in > > https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/blob/b5a6c1db1... > > but, as you can see in > > https://jenkins.ovirt.org/job/oVirt_ovirt-ansible-hosted-engine-setup_sta... > CI builds it as 1.0.9 ignoring commit > 7428b54a5ba8458379b1a27d116f9504bb830e69 > that has been merged before b5a6c1db135d81d75f3330160e7ef4a84c97fd60 > > I'm not sure if build 138 ran before or after 7428b54a5ba8458379b1a27d116f9504bb830e69 was merged. since you merged the PR already, we cannot check what will happen if we rerun the job now. I suspect you're simply looking at the output of a job that stated running before 7428b54a5ba8458379b1a27d116f9504bb830e69 was merged

> >> -- >> Barak Korren >> RHV DevOps team , RHCE, RHCi >> Red Hat EMEA >> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >> > -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted