Hi, We are failing to deploy hosts in upgrade suites on both master and 4.2 for project ovirt-vmconsole. it seems we are missing packages for selinux-policy. Root cause identified by CQ as: https://gerrit.ovirt.org/#/c/97704/ - spec: clean up and reorganize can you please take a look at this issue? Sandro requested a bug so I opened one: https://bugzilla.redhat.com/show_bug.cgi?id=1677630

Error: 019-02-14 12:11:42,063-0500 ERROR otopi.plugins.otopi.packagers.yumpackager yumpackager.error:85 Yum [u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy >= 3.13.1-229.el7_6.9', u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy-base >= 3.13.1-229.el7_6.9'] 2019-02-14 12:11:42,063-0500 DEBUG otopi.context context._executeMethod:142 method exception Traceback (most recent call last): File "/tmp/ovirt-8JzESBo7eU/pythonlib/otopi/context.py", line 132, in _executeMethod method['method']() File "/tmp/ovirt-8JzESBo7eU/otopi-plugins/otopi/packagers/yumpackager.py", line 248, in _packages self.processTransaction() File "/tmp/ovirt-8JzESBo7eU/otopi-plugins/otopi/packagers/yumpackager.py", line 262, in processTransaction if self._miniyum.buildTransaction(): File "/tmp/ovirt-8JzESBo7eU/pythonlib/otopi/miniyum.py", line 920, in buildTransaction raise yum.Errors.YumBaseError(msg) YumBaseError: [u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy >= 3.13.1-229.el7_6.9', u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy-base >= 3.13.1-229.el7_6.9'] 2019-02-14 12:11:42,064-0500 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Package installation': [u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy >= 3.13.1-229.el7_6.9', u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy-base >= 3.13.1-229.el7_6.9'] Thanks, Dafna

On Fri, Feb 15, 2019 at 2:39 PM Francesco Romani <fromani(a)redhat.com <mailto:fromani@redhat.com>> wrote: On 2/15/19 1:40 PM, Dafna Ron wrote: Hi, > Hi, > > We are failing to deploy hosts in upgrade suites on both master > and 4.2 for project ovirt-vmconsole. > > it seems we are missing packages for selinux-policy. > > Root cause identified by CQ as: > https://gerrit.ovirt.org/#/c/97704/ - spec: clean up and reorganize > > can you please take a look at this issue? > > Sandro requested a bug so I opened one: > https://bugzilla.redhat.com/show_bug.cgi?id=1677630 Yep, I replied https://bugzilla.redhat.com/show_bug.cgi?id=1677630#c2 more discussion follows: > > Error: > > 019-02-14 12:11:42,063-0500 ERROR otopi.plugins.otopi.packagers.yumpackager yumpackager.error:85 Yum [u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy >= 3.13.1-229.el7_6.9', u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy-base >= 3.13.1-229.el7_6.9'] > 2019-02-14 12:11:42,063-0500 DEBUG otopi.context context._executeMethod:142 method exception > Traceback (most recent call last): > File "/tmp/ovirt-8JzESBo7eU/pythonlib/otopi/context.py", line 132, in _executeMethod > method['method']() > File "/tmp/ovirt-8JzESBo7eU/otopi-plugins/otopi/packagers/yumpackager.py", line 248, in _packages > self.processTransaction() > File "/tmp/ovirt-8JzESBo7eU/otopi-plugins/otopi/packagers/yumpackager.py", line 262, in processTransaction > if self._miniyum.buildTransaction(): > File "/tmp/ovirt-8JzESBo7eU/pythonlib/otopi/miniyum.py", line 920, in buildTransaction > raise yum.Errors.YumBaseError(msg) > YumBaseError: [u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy >= 3.13.1-229.el7_6.9', u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy-base >= 3.13.1-229.el7_6.9'] > 2019-02-14 12:11:42,064-0500 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Package installation': [u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy >= 3.13.1-229.el7_6.9', u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy-base >= 3.13.1-229.el7_6.9'] > Thanks, > Dafna > It seems to me this is happening in CentOS. So: The patchhttps://gerrit.ovirt.org/#/c/97704/ *wants* to use this spec file macro %{?selinux_requires} This macro automatically set the right dependency for the platform on which the package is being built. From the error above, we can see that the host on which the package, built from master, is going to be installed does *not* have that right package. However, on a test box of mine: 1005 15:07:43 root@kenji:~ $ cat /etc/redhat-release CentOS Linux release 7.6.1810 (Core) 1006 15:07:49 root@kenji:~ $ rpm -qa | grep selinux-policy selinux-policy-3.13.1-229.el7_6.9.noarch selinux-policy-targeted-3.13.1-229.el7_6.9.noarch selinux-policy-devel-3.13.1-229.el7_6.6.noarch 1010 15:08:50 root@kenji:~ $ rpm -q --provides selinux-policy config(selinux-policy) = 3.13.1-229.el7_6.9 selinux-policy = 3.13.1-229.el7_6.9 1011 15:08:52 root@kenji:~ $ rpm -q --provides selinux-policy-targeted config(selinux-policy-targeted) = 3.13.1-229.el7_6.9 selinux-policy-base = 3.13.1-229.el7_6.9 selinux-policy-targeted = 3.13.1-229.el7_6.9 so it seems that the package was built on up-to-date host, while is being installed in a host outdated. Not sure I understand that. we are running on an isolated environment which is running 7.6 and the package we have available in the centos repo is: selinux-policy-0:3.13.1-229.el7_6.9.noarch when i force the download of the package (i.e I tell lago to grab that package on deploy of vms) then the package is available and downloaded. So I am not sure what you mean about the package running on an outdated host?

On Fri, Feb 15, 2019 at 2:54 PM Francesco Romani <fromani(a)redhat.com <mailto:fromani@redhat.com>> wrote: On 2/15/19 3:49 PM, Dafna Ron wrote: > > > On Fri, Feb 15, 2019 at 2:39 PM Francesco Romani > <fromani(a)redhat.com <mailto:fromani@redhat.com>> wrote: > > On 2/15/19 1:40 PM, Dafna Ron wrote: > > Hi, > >> Hi, >> >> We are failing to deploy hosts in upgrade suites on both >> master and 4.2 for project ovirt-vmconsole. >> >> it seems we are missing packages for selinux-policy. >> >> Root cause identified by CQ as: >> https://gerrit.ovirt.org/#/c/97704/ - spec: clean up and >> reorganize >> >> can you please take a look at this issue? >> >> Sandro requested a bug so I opened one: >> https://bugzilla.redhat.com/show_bug.cgi?id=1677630 > > > Yep, I replied > https://bugzilla.redhat.com/show_bug.cgi?id=1677630#c2 > > > more discussion follows: > > >> >> Error: >> >> 019-02-14 12:11:42,063-0500 ERROR otopi.plugins.otopi.packagers.yumpackager yumpackager.error:85 Yum [u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy >= 3.13.1-229.el7_6.9', u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy-base >= 3.13.1-229.el7_6.9'] >> 2019-02-14 12:11:42,063-0500 DEBUG otopi.context context._executeMethod:142 method exception >> Traceback (most recent call last): >> File "/tmp/ovirt-8JzESBo7eU/pythonlib/otopi/context.py", line 132, in _executeMethod >> method['method']() >> File "/tmp/ovirt-8JzESBo7eU/otopi-plugins/otopi/packagers/yumpackager.py", line 248, in _packages >> self.processTransaction() >> File "/tmp/ovirt-8JzESBo7eU/otopi-plugins/otopi/packagers/yumpackager.py", line 262, in processTransaction >> if self._miniyum.buildTransaction(): >> File "/tmp/ovirt-8JzESBo7eU/pythonlib/otopi/miniyum.py", line 920, in buildTransaction >> raise yum.Errors.YumBaseError(msg) >> YumBaseError: [u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy >= 3.13.1-229.el7_6.9', u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy-base >= 3.13.1-229.el7_6.9'] >> 2019-02-14 12:11:42,064-0500 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Package installation': [u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy >= 3.13.1-229.el7_6.9', u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy-base >= 3.13.1-229.el7_6.9'] >> Thanks, >> Dafna >> > > It seems to me this is happening in CentOS. So: > > The patchhttps://gerrit.ovirt.org/#/c/97704/ *wants* to use this spec file macro > %{?selinux_requires} > > This macro automatically set the right dependency for the platform on which the package is being built. > > From the error above, we can see that the host on which the package, built from master, is going to be installed does *not* have that right package. > > However, on a test box of mine: > > 1005 15:07:43 root@kenji:~ $ cat /etc/redhat-release > CentOS Linux release 7.6.1810 (Core) > 1006 15:07:49 root@kenji:~ $ rpm -qa | grep selinux-policy > selinux-policy-3.13.1-229.el7_6.9.noarch > selinux-policy-targeted-3.13.1-229.el7_6.9.noarch > selinux-policy-devel-3.13.1-229.el7_6.6.noarch > > 1010 15:08:50 root@kenji:~ $ rpm -q --provides selinux-policy > config(selinux-policy) = 3.13.1-229.el7_6.9 > selinux-policy = 3.13.1-229.el7_6.9 > 1011 15:08:52 root@kenji:~ $ rpm -q --provides selinux-policy-targeted > config(selinux-policy-targeted) = 3.13.1-229.el7_6.9 > selinux-policy-base = 3.13.1-229.el7_6.9 > selinux-policy-targeted = 3.13.1-229.el7_6.9 > > so it seems that the package was built on up-to-date host, while is being installed in a host outdated. > > > Not sure I understand that. we are running on an isolated > environment which is running 7.6 and the package we have > available in the centos repo is: > selinux-policy-0:3.13.1-229.el7_6.9.noarch > when i force the download of the package (i.e I tell lago to grab > that package on deploy of vms) then the package is available and > downloaded. > So I am not sure what you mean about the package running on an > outdated host? I mean that the package is available, so the dependency could be fullfilled (e.g ovirt-vmconsole is not depending on bogus, unreleased package). If the dependency is not being fullfilled, it's an issue of the specific host on which the test fails. It should install cleanly on an up-to-date RHEL/CentOS 7.6 host. But as I was saying, the host is not related to our CI runs as they run in mock in a clean environment (each run is cleaned and re-installed) The vms are created and destroyed on each run and packages are downloaded based on the project spec you can also see that when I manually add the package list that its grabbed with no problem: https://gerrit.ovirt.org/#/c/97785/ Another thing that points that this is only an issue on your project is that its not effecting any other project other then yours. when there is a package issue on our servers/lago/OST it would effect all projects. This is not a CI issue although if you insist it is, I can fix it in CI - however, this would be allowing a bug in the project to be merged.

There is nothing wrong on CI. I explained that our automation cannot be the cause of this package disappearing as we do not control the package hierarchy and/or dependencies and simply download the packages that are available based on the configuration of the projects. Since the package is available for download from mirrors and the only project effected is ovirt-vmconsole it means that the cause of this issue is in vmconsole project. CQ identified this patch as the root cause: spec: https://gerrit.ovirt.org/#/c/97704/ - clean up and reorganize After discussing the issue with Sandro who suggested that this is a bug and should be addressed by the project and not fixed in CI, I alerted you that the project is currently failing on a regression which started after the merge of your bug. However, as you continue to claim this is a CI issue and I feel this is going no where, I am adding the managers of RHV to approve fixing this issue in CI by forcing the selinux package to download ( https://gerrit.ovirt.org/#/c/97785/) RHV managers, since there is a release on Monday and if fixed in CI may cause a respin (if issue is reported by QE), I would like someone to confirm for me that you would like this issue to be fixed in the CI system by merging the patch above (97785) Thanks, Dafna On Fri, Feb 15, 2019 at 3:39 PM Francesco Romani <fromani(a)redhat.com&gt; wrote: > On 2/15/19 4:05 PM, Dafna Ron wrote: > > > > On Fri, Feb 15, 2019 at 2:54 PM Francesco Romani <fromani(a)redhat.com&gt; > wrote: > >> On 2/15/19 3:49 PM, Dafna Ron wrote: >> >> >> >> On Fri, Feb 15, 2019 at 2:39 PM Francesco Romani <fromani(a)redhat.com&gt; >> wrote: >> >>> On 2/15/19 1:40 PM, Dafna Ron wrote: >>> >>> Hi, >>> >>> Hi, >>> >>> We are failing to deploy hosts in upgrade suites on both master and 4.2 >>> for project ovirt-vmconsole. >>> >>> it seems we are missing packages for selinux-policy. >>> >>> Root cause identified by CQ as: https://gerrit.ovirt.org/#/c/97704/ - >>> spec: clean up and reorganize >>> >>> can you please take a look at this issue? >>> >>> Sandro requested a bug so I opened one: >>> https://bugzilla.redhat.com/show_bug.cgi?id=1677630 >>> >>> >>> Yep, I replied https://bugzilla.redhat.com/show_bug.cgi?id=1677630#c2 >>> >>> >>> more discussion follows: >>> >>> >>> >>> Error: >>> >>> 019-02-14 12:11:42,063-0500 ERROR otopi.plugins.otopi.packagers.yumpackager yumpackager.error:85 Yum [u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy >= 3.13.1-229.el7_6.9', u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy-base >= 3.13.1-229.el7_6.9'] >>> 2019-02-14 12:11:42,063-0500 DEBUG otopi.context context._executeMethod:142 method exception >>> Traceback (most recent call last): >>> File "/tmp/ovirt-8JzESBo7eU/pythonlib/otopi/context.py", line 132, in _executeMethod >>> method['method']() >>> File "/tmp/ovirt-8JzESBo7eU/otopi-plugins/otopi/packagers/yumpackager.py", line 248, in _packages >>> self.processTransaction() >>> File "/tmp/ovirt-8JzESBo7eU/otopi-plugins/otopi/packagers/yumpackager.py", line 262, in processTransaction >>> if self._miniyum.buildTransaction(): >>> File "/tmp/ovirt-8JzESBo7eU/pythonlib/otopi/miniyum.py", line 920, in buildTransaction >>> raise yum.Errors.YumBaseError(msg) >>> YumBaseError: [u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy >= 3.13.1-229.el7_6.9', u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy-base >= 3.13.1-229.el7_6.9'] >>> 2019-02-14 12:11:42,064-0500 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Package installation': [u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy >= 3.13.1-229.el7_6.9', u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy-base >= 3.13.1-229.el7_6.9'] >>> >>> Thanks, >>> Dafna >>> >>> >>> It seems to me this is happening in CentOS. So: >>> >>> The patch https://gerrit.ovirt.org/#/c/97704/ *wants* to use this spec file macro >>> %{?selinux_requires} >>> >>> This macro automatically set the right dependency for the platform on which the package is being built. >>> >>> From the error above, we can see that the host on which the package, built from master, is going to be installed does *not* have that right package. >>> >>> However, on a test box of mine: >>> >>> 1005 15:07:43 root@kenji:~ $ cat /etc/redhat-release >>> CentOS Linux release 7.6.1810 (Core) >>> 1006 15:07:49 root@kenji:~ $ rpm -qa | grep selinux-policy >>> selinux-policy-3.13.1-229.el7_6.9.noarch >>> selinux-policy-targeted-3.13.1-229.el7_6.9.noarch >>> selinux-policy-devel-3.13.1-229.el7_6.6.noarch >>> >>> 1010 15:08:50 root@kenji:~ $ rpm -q --provides selinux-policy >>> config(selinux-policy) = 3.13.1-229.el7_6.9 >>> selinux-policy = 3.13.1-229.el7_6.9 >>> 1011 15:08:52 root@kenji:~ $ rpm -q --provides selinux-policy-targeted >>> config(selinux-policy-targeted) = 3.13.1-229.el7_6.9 >>> selinux-policy-base = 3.13.1-229.el7_6.9 >>> selinux-policy-targeted = 3.13.1-229.el7_6.9 >>> >>> so it seems that the package was built on up-to-date host, while is being installed in a host outdated. >>> >>> >> Not sure I understand that. we are running on an isolated environment >> which is running 7.6 and the package we have available in the centos repo >> is: selinux-policy-0:3.13.1-229.el7_6.9.noarch >> when i force the download of the package (i.e I tell lago to grab that >> package on deploy of vms) then the package is available and downloaded. >> So I am not sure what you mean about the package running on an outdated >> host? >> >> >> I mean that the package is available, so the dependency could be >> fullfilled (e.g ovirt-vmconsole is not depending on bogus, unreleased >> package). >> >> If the dependency is not being fullfilled, it's an issue of the specific >> host on which the test fails. >> >> It should install cleanly on an up-to-date RHEL/CentOS 7.6 host. >> > > But as I was saying, the host is not related to our CI runs as they run > in mock in a clean environment (each run is cleaned and re-installed) > The vms are created and destroyed on each run and packages are downloaded > based on the project spec > you can also see that when I manually add the package list that its > grabbed with no problem: https://gerrit.ovirt.org/#/c/97785/ > Another thing that points that this is only an issue on your project is > that its not effecting any other project other then yours. when there is a > package issue on our servers/lago/OST it would effect all projects. > This is not a CI issue although if you insist it is, I can fix it in CI - > however, this would be allowing a bug in the project to be merged. > > > I'm not insisting, I'm trying to explain. > > ovirt-vmconsole 1.0.7 *is* unique (afaik) because it is the only package > which is using the aforementioned selinux_requires magic macro. > > The magic macro ensures that the package being builts depend on the > selinux-policy version >= the version used on the building host. > > We can see the macro itself from my fc29 laptop: > > > %_selinux_policy_version 3.14.2-48.fc29 > > > # %selinux_requires > %selinux_requires \ > Requires: selinux-policy >= %{_selinux_policy_version} \ > BuildRequires: git \ > BuildRequires: pkgconfig(systemd) \ > BuildRequires: selinux-policy \ > BuildRequires: selinux-policy-devel \ > Requires(post): selinux-policy-base >= %{_selinux_policy_version} \ > Requires(post): libselinux-utils \ > Requires(post): policycoreutils \ > %if 0%{?fedora} || 0%{?rhel} > 7\ > Requires(post): policycoreutils-python-utils \ > %else \ > Requires(post): policycoreutils-python \ > %endif \ > %{nil} > > > Now, if we crosscheck the packages in the build > https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=845144 > > > 1067 16:34:41 fromani@musashi2 /tmp $ wget -v > http://download.eng.bos.redhat.com/brewroot/vol/rhel-7/packages/ovirt-vmc... > --2019-02-15 16:34:44-- > http://download.eng.bos.redhat.com/brewroot/vol/rhel-7/packages/ovirt-vmc... > Resolving download.eng.bos.redhat.com (download.eng.bos.redhat.com)... > 10.19.43.4 > Connecting to download.eng.bos.redhat.com (download.eng.bos.redhat.com)|10.19.43.4|:80... > connected. > HTTP request sent, awaiting response... 200 OK > Length: 35160 (34K) [application/x-rpm] > Saving to: ‘ovirt-vmconsole-1.0.7-1.el7ev.noarch.rpm’ > > ovirt-vmconsole-1.0.7-1.el7ev.noarch.rpm > 100%[==========================================================================================================================================================================>] > 34.34K 108KB/s in 0.3s > > 2019-02-15 16:34:45 (108 KB/s) - > ‘ovirt-vmconsole-1.0.7-1.el7ev.noarch.rpm’ saved [35160/35160] > > 1068 16:34:45 fromani@musashi2 /tmp $ rpm -qpR > ovirt-vmconsole-1.0.7-1.el7ev.noarch.rpm > /bin/sh > /bin/sh > /bin/sh > /bin/sh > libselinux-utils > libselinux-utils > libselinux-utils > policycoreutils > policycoreutils > policycoreutils-python > policycoreutils-python > python > python(abi) = 2.7 > rpmlib(CompressedFileNames) <= 3.0.4-1 > rpmlib(FileDigests) <= 4.6.0-1 > rpmlib(PartialHardlinkSets) <= 4.0.4-1 > rpmlib(PayloadFilesHavePrefix) <= 4.0-1 > selinux-policy >= 3.13.1-229.el7_6.9 > selinux-policy-base >= 3.13.1-229.el7_6.9 > shadow-utils > rpmlib(PayloadIsXz) <= 5.2-1 > > and a package built locally on my fc29 laptop: > > 1076 16:36:38 fromani@musashi2 ~/rpmbuild/RPMS/noarch $ rpm -qpR > ovirt-vmconsole-1.0.7-2.fc29.noarch.rpm > /bin/sh > /bin/sh > /bin/sh > /bin/sh > libselinux-utils > libselinux-utils > libselinux-utils > policycoreutils > policycoreutils > policycoreutils-python > policycoreutils-python > policycoreutils-python-utils > python > python(abi) = 2.7 > rpmlib(CompressedFileNames) <= 3.0.4-1 > rpmlib(FileDigests) <= 4.6.0-1 > rpmlib(PartialHardlinkSets) <= 4.0.4-1 > rpmlib(PayloadFilesHavePrefix) <= 4.0-1 > rpmlib(PayloadIsXz) <= 5.2-1 > selinux-policy >= 3.14.2-48.fc29 > selinux-policy-base >= 3.14.2-48.fc29 > shadow-utils > > We can see all the deps seems right - and surely it depends on > selinux-policy > > > I can't say what's wrong on CI or what's the best fix, but the deps looks > fine in the vmconsole package. > > > -- > Francesco Romani > Senior SW Eng., Virtualization R&D > Red Hat > IRC: fromani github: @fromanirh > >