On Wed, Oct 9, 2019 at 11:56 PM Nir Soffer <nsoffer@redhat.com> wrote:

I'm trying to run imageio tests on el8 mock, and the tests fail early when trying to create storage

for testing:
[userstorage] INFO    Creating filesystem /var/tmp/imageio-storage/file-512-ext4-mount
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
[userstorage] INFO    Creating file /var/tmp/imageio-storage/file-512-ext4-mount/file
[userstorage] INFO    Creating backing file /var/tmp/imageio-storage/file-512-xfs-backing
[userstorage] INFO    Creating loop device /var/tmp/imageio-storage/file-512-xfs-loop
[userstorage] INFO    Creating filesystem /var/tmp/imageio-storage/file-512-xfs-mount
mount: /var/tmp/imageio-storage/file-512-xfs-mount: wrong fs type, bad option, bad superblock on /dev/loop4, missing codepage or helper program, or other error.
Traceback (most recent call last):
  File "/usr/local/bin/userstorage", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/userstorage/__main__.py", line 42, in main
    create(cfg)
  File "/usr/local/lib/python3.6/site-packages/userstorage/__main__.py", line 52, in create
    b.create()
  File "/usr/local/lib/python3.6/site-packages/userstorage/file.py", line 47, in create
    self._mount.create()
  File "/usr/local/lib/python3.6/site-packages/userstorage/mount.py", line 53, in create
    self._mount_loop()
  File "/usr/local/lib/python3.6/site-packages/userstorage/mount.py", line 94, in _mount_loop
    ["sudo", "mount", "-t", self.fstype, self._loop.path, self.path])
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', 'mount', '-t', 'xfs', '/var/tmp/imageio-storage/file-512-xfs-loop', '/var/tmp/imageio-storage/file-512-xfs-mount']' returned non-zero exit status 32.
https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/1593//artifact/check-patch.el8.ppc64le/mock_logs/script/stdout_stderr.log

Same code runs fine in Travis:

https://travis-ci.org/nirs/ovirt-imageio/jobs/595794863

And also locally on Fedora 29:

$ ../jenkins/mock_configs/mock_runner.sh -C ../jenkins/mock_configs -p el8

...

## Wed Oct 9 23:37:22 IDT 2019 Finished env: el8:epel-8-x86_64
## took 85 seconds
## rc = 0

My guess is that we run el8 jobs on el7 hosts with old kernels

(Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.)

Here is info from failed builds:

DEBUG buildroot.py:503: kernel version == 3.10.0-693.11.6.el7.ppc64le
https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/1593/artifact/check-patch.el8.ppc64le/mock_logs/init/root.log

DEBUG buildroot.py:503: kernel version == 3.10.0-957.12.1.el7.x86_64
https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/1593/artifact/check-patch.el8.x86_64/mock_logs/init/root.log

and successful builds:

DEBUG buildroot.py:503: kernel version == 5.1.18-200.fc29.x86_64

My laptop

Runtime kernel version: 4.15.0-1032-gcp

https://travis-ci.org/nirs/ovirt-imageio/jobs/595794863

This issue will affects vdsm, using similar code to create storage for testing.

For vdsm the issue is more tricky, since it requires same host/distro:

49 - tests-py37:
50 runtime-requirements:
51 host-distro: same

So I think we need:

- make slaves with newer kernel (fc29? fc30?) with the el8 mock env

- add el8 slaves for running el8 mock with "host-distro: same"