You may be running out of loop devices, if that's the case, you need to manually mknod them, see https://github.com/oVirt/ovirt-node-ng-image/blob/56a2797b5ef84bd56ab95fdb0cbfb908b4bc8ec1/automation/build-artifacts.sh#L24

On Thursday, October 10, 2019, Nir Soffer <nsoffer@redhat.com> wrote:
On Wed, Oct 9, 2019 at 11:56 PM Nir Soffer <nsoffer@redhat.com> wrote:
I'm trying to run imageio tests on el8 mock, and the tests fail early when trying to create storage
for testing:
[userstorage] INFO    Creating filesystem /var/tmp/imageio-storage/file-512-ext4-mount
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
[userstorage] INFO    Creating file /var/tmp/imageio-storage/file-512-ext4-mount/file
[userstorage] INFO    Creating backing file /var/tmp/imageio-storage/file-512-xfs-backing
[userstorage] INFO    Creating loop device /var/tmp/imageio-storage/file-512-xfs-loop
[userstorage] INFO    Creating filesystem /var/tmp/imageio-storage/file-512-xfs-mount
mount: /var/tmp/imageio-storage/file-512-xfs-mount: wrong fs type, bad option, bad superblock on /dev/loop4, missing codepage or helper program, or other error.
Traceback (most recent call last):
  File "/usr/local/bin/userstorage", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/userstorage/__main__.py", line 42, in main
    create(cfg)
  File "/usr/local/lib/python3.6/site-packages/userstorage/__main__.py", line 52, in create
    b.create()
  File "/usr/local/lib/python3.6/site-packages/userstorage/file.py", line 47, in create
    self._mount.create()
  File "/usr/local/lib/python3.6/site-packages/userstorage/mount.py", line 53, in create
    self._mount_loop()
  File "/usr/local/lib/python3.6/site-packages/userstorage/mount.py", line 94, in _mount_loop
    ["sudo", "mount", "-t", self.fstype, self._loop.path, self.path])
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', 'mount', '-t', 'xfs', '/var/tmp/imageio-storage/file-512-xfs-loop', '/var/tmp/imageio-storage/file-512-xfs-mount']' returned non-zero exit status 32.


Same code runs fine in Travis:


And also locally on Fedora 29:
$ ../jenkins/mock_configs/mock_runner.sh -C ../jenkins/mock_configs -p el8
...
## Wed Oct  9 23:37:22 IDT 2019 Finished env: el8:epel-8-x86_64
##      took 85 seconds
##      rc = 0


My guess is that we run el8 jobs on el7 hosts with old kernels
(Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.)

Here is info from failed builds:

DEBUG buildroot.py:503: kernel version == 3.10.0-693.11.6.el7.ppc64le
https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/1593/artifact/check-patch.el8.ppc64le/mock_logs/init/root.log

DEBUG buildroot.py:503:  kernel version == 3.10.0-957.12.1.el7.x86_64
https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/1593/artifact/check-patch.el8.x86_64/mock_logs/init/root.log
 
and successful builds:

DEBUG buildroot.py:503:  kernel version == 5.1.18-200.fc29.x86_64
My laptop

Runtime kernel version: 4.15.0-1032-gcp
https://travis-ci.org/nirs/ovirt-imageio/jobs/595794863


This issue will affects vdsm, using similar code to create storage for testing.

For vdsm the issue is more tricky, since it requires same host/distro:

 49         - tests-py37:
 50             runtime-requirements:
 51               host-distro: same

So I think we need:
- make slaves with newer kernel (fc29? fc30?) with the el8 mock env
- add el8 slaves for running el8 mock with "host-distro: same"

If we don't have a solution we need to disable el8 tests, or continue testing without
storage, which will skip about 200 tests.

Nir