There is nothing wrong on CI.

I explained that our automation cannot be the cause of this package disappearing as we do not control the package hierarchy and/or dependencies and simply download the packages that are available based on the configuration of the projects.

Since the package is available for download from mirrors and the only project effected is ovirt-vmconsole it means that the cause of this issue is in vmconsole project.

CQ identified this patch as the root cause: spec: https://gerrit.ovirt.org/#/c/97704/ - clean up and reorganize

After discussing the issue with Sandro who suggested that this is a bug and should be addressed by the project and not fixed in CI, I alerted you that the project is currently failing on a regression which started after the merge of your bug.

However, as you continue to claim this is a CI issue and I feel this is going no where, I am adding the managers of RHV to approve fixing this issue in CI by forcing the selinux package to download (https://gerrit.ovirt.org/#/c/97785/)

RHV managers, since there is a release on Monday and if fixed in CI may cause a respin (if issue is reported by QE),  I would like someone to confirm for me that you would like this issue to be fixed in the CI system by merging the patch above (97785)

Thanks,

Dafna

On Fri, Feb 15, 2019 at 3:39 PM Francesco Romani <fromani@redhat.com> wrote:

On 2/15/19 4:05 PM, Dafna Ron wrote:

On Fri, Feb 15, 2019 at 2:54 PM Francesco Romani <fromani@redhat.com> wrote:

On 2/15/19 3:49 PM, Dafna Ron wrote:

On Fri, Feb 15, 2019 at 2:39 PM Francesco Romani <fromani@redhat.com> wrote:

On 2/15/19 1:40 PM, Dafna Ron wrote:

Hi,

Hi,

We are failing to deploy hosts in upgrade suites on both master and 4.2 for project ovirt-vmconsole.

it seems we are missing packages for selinux-policy.

Root cause identified by CQ as: https://gerrit.ovirt.org/#/c/97704/ - spec: clean up and reorganize

can you please take a look at this issue?

Sandro requested a bug so I opened one:

https://bugzilla.redhat.com/show_bug.cgi?id=1677630

Yep, I replied https://bugzilla.redhat.com/show_bug.cgi?id=1677630#c2

more discussion follows:

Error:

019-02-14 12:11:42,063-0500 ERROR otopi.plugins.otopi.packagers.yumpackager yumpackager.error:85 Yum [u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy >= 3.13.1-229.el7_6.9', u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy-base >= 3.13.1-229.el7_6.9']
2019-02-14 12:11:42,063-0500 DEBUG otopi.context context._executeMethod:142 method exception
Traceback (most recent call last):
  File "/tmp/ovirt-8JzESBo7eU/pythonlib/otopi/context.py", line 132, in _executeMethod
    method['method']()
  File "/tmp/ovirt-8JzESBo7eU/otopi-plugins/otopi/packagers/yumpackager.py", line 248, in _packages
    self.processTransaction()
  File "/tmp/ovirt-8JzESBo7eU/otopi-plugins/otopi/packagers/yumpackager.py", line 262, in processTransaction
    if self._miniyum.buildTransaction():
  File "/tmp/ovirt-8JzESBo7eU/pythonlib/otopi/miniyum.py", line 920, in buildTransaction
    raise yum.Errors.YumBaseError(msg)
YumBaseError: [u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy >= 3.13.1-229.el7_6.9', u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy-base >= 3.13.1-229.el7_6.9']
2019-02-14 12:11:42,064-0500 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Package installation': [u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy >= 3.13.1-229.el7_6.9', u'ovirt-vmconsole-1.0.6-3.el7.noarch requires selinux-policy-base >= 3.13.1-229.el7_6.9']

Thanks,

Dafna

It seems to me this is happening in CentOS. So:

The patch https://gerrit.ovirt.org/#/c/97704/ *wants* to use this spec file macro
%{?selinux_requires}

This macro automatically set the right dependency for the platform on which the package is being built.

From the error above, we can see that the host on which the package, built from master, is going to be installed does *not* have that right package.

However, on a test box of mine:

1005 15:07:43 root@kenji:~ $ cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core) 
1006 15:07:49 root@kenji:~ $ rpm -qa | grep selinux-policy
selinux-policy-3.13.1-229.el7_6.9.noarch
selinux-policy-targeted-3.13.1-229.el7_6.9.noarch
selinux-policy-devel-3.13.1-229.el7_6.6.noarch

1010 15:08:50 root@kenji:~ $ rpm -q --provides selinux-policy
config(selinux-policy) = 3.13.1-229.el7_6.9
selinux-policy = 3.13.1-229.el7_6.9
1011 15:08:52 root@kenji:~ $ rpm -q --provides selinux-policy-targeted
config(selinux-policy-targeted) = 3.13.1-229.el7_6.9
selinux-policy-base = 3.13.1-229.el7_6.9
selinux-policy-targeted = 3.13.1-229.el7_6.9

so it seems that the package was built on up-to-date host, while is being installed in a host outdated.

Not sure I understand that. we are running on an isolated environment which is running 7.6 and the package we have available in the centos repo is: selinux-policy-0:3.13.1-229.el7_6.9.noarch

when i force the download of the package (i.e I tell lago to grab that package on deploy of vms) then the package is available and downloaded.

So I am not sure what you mean about the package running on an outdated host?

I mean that the package is available, so the dependency could be fullfilled (e.g ovirt-vmconsole is not depending on bogus, unreleased package).

If the dependency is not being fullfilled, it's an issue of the specific host on which the test fails.

It should install cleanly on an up-to-date RHEL/CentOS 7.6 host.

But as I was saying, the host is not related to our CI runs as they run in mock in a clean environment (each run is cleaned and re-installed)

The vms are created and destroyed on each run and packages are downloaded based on the project spec

you can also see that when I manually add the package list that its grabbed with no problem: https://gerrit.ovirt.org/#/c/97785/

Another thing that points that this is only an issue on your project is that its not effecting any other project other then yours. when there is a package issue on our servers/lago/OST it would effect all projects.

This is not a CI issue although if you insist it is, I can fix it in CI - however, this would be allowing a bug in the project to be merged.

I'm not insisting, I'm trying to explain.

ovirt-vmconsole 1.0.7 *is* unique (afaik) because it is the only package which is using the aforementioned selinux_requires magic macro.

The magic macro ensures that the package being builts depend on the selinux-policy version >= the version used on the building host.

We can see the macro itself from my fc29 laptop:

%_selinux_policy_version 3.14.2-48.fc29


# %selinux_requires
%selinux_requires \
Requires: selinux-policy >= %{_selinux_policy_version} \
BuildRequires: git \
BuildRequires: pkgconfig(systemd) \
BuildRequires: selinux-policy \
BuildRequires: selinux-policy-devel \
Requires(post): selinux-policy-base >= %{_selinux_policy_version} \
Requires(post): libselinux-utils \
Requires(post): policycoreutils \
%if 0%{?fedora} || 0%{?rhel} > 7\
Requires(post): policycoreutils-python-utils \
%else \
Requires(post): policycoreutils-python \
%endif \
%{nil}

Now, if we crosscheck the packages in the build https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=845144

1067 16:34:41 fromani@musashi2 /tmp $ wget -v http://download.eng.bos.redhat.com/brewroot/vol/rhel-7/packages/ovirt-vmconsole/1.0.7/1.el7ev/noarch/ovirt-vmconsole-1.0.7-1.el7ev.noarch.rpm
--2019-02-15 16:34:44--  http://download.eng.bos.redhat.com/brewroot/vol/rhel-7/packages/ovirt-vmconsole/1.0.7/1.el7ev/noarch/ovirt-vmconsole-1.0.7-1.el7ev.noarch.rpm
Resolving download.eng.bos.redhat.com (download.eng.bos.redhat.com)... 10.19.43.4
Connecting to download.eng.bos.redhat.com (download.eng.bos.redhat.com)|10.19.43.4|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 35160 (34K) [application/x-rpm]
Saving to: ‘ovirt-vmconsole-1.0.7-1.el7ev.noarch.rpm’

ovirt-vmconsole-1.0.7-1.el7ev.noarch.rpm                               100%[==========================================================================================================================================================================>]  34.34K   108KB/s    in 0.3s   

2019-02-15 16:34:45 (108 KB/s) - ‘ovirt-vmconsole-1.0.7-1.el7ev.noarch.rpm’ saved [35160/35160]

1068 16:34:45 fromani@musashi2 /tmp $ rpm -qpR ovirt-vmconsole-1.0.7-1.el7ev.noarch.rpm
/bin/sh
/bin/sh
/bin/sh
/bin/sh
libselinux-utils
libselinux-utils
libselinux-utils
policycoreutils
policycoreutils
policycoreutils-python
policycoreutils-python
python
python(abi) = 2.7
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PartialHardlinkSets) <= 4.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
selinux-policy >= 3.13.1-229.el7_6.9
selinux-policy-base >= 3.13.1-229.el7_6.9
shadow-utils
rpmlib(PayloadIsXz) <= 5.2-1

and a package built locally on my fc29 laptop:

1076 16:36:38 fromani@musashi2 ~/rpmbuild/RPMS/noarch $ rpm -qpR ovirt-vmconsole-1.0.7-2.fc29.noarch.rpm
/bin/sh
/bin/sh
/bin/sh
/bin/sh
libselinux-utils
libselinux-utils
libselinux-utils
policycoreutils
policycoreutils
policycoreutils-python
policycoreutils-python
policycoreutils-python-utils
python
python(abi) = 2.7
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PartialHardlinkSets) <= 4.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(PayloadIsXz) <= 5.2-1
selinux-policy >= 3.14.2-48.fc29
selinux-policy-base >= 3.14.2-48.fc29
shadow-utils

We can see all the deps seems right - and surely it depends on selinux-policy

I can't say what's wrong on CI or what's the best fix, but the deps looks fine in the vmconsole package.

-- 
Francesco Romani
Senior SW Eng., Virtualization R&D
Red Hat
IRC: fromani github: @fromanirh