You may be running out of loop devices, if that's the case, you need to manually mknod them, see https://github.com/oVirt/ovirt-node-ng-image/blob/56a2797b5ef84bd56ab95fd... On Thursday, October 10, 2019, Nir Soffer <nsoffer(a)redhat.com <mailto:nsoffer@redhat.com>> wrote: On Wed, Oct 9, 2019 at 11:56 PM Nir Soffer <nsoffer(a)redhat.com <mailto:nsoffer@redhat.com>> wrote: I'm trying to run imageio tests on el8 mock, and the tests fail early when trying to create storage for testing: [userstorage] INFO Creating filesystem /var/tmp/imageio-storage/file-512-ext4-mount Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features. [userstorage] INFO Creating file /var/tmp/imageio-storage/file-512-ext4-mount/file [userstorage] INFO Creating backing file /var/tmp/imageio-storage/file-512-xfs-backing [userstorage] INFO Creating loop device /var/tmp/imageio-storage/file-512-xfs-loop [userstorage] INFO Creating filesystem /var/tmp/imageio-storage/file-512-xfs-mount mount: /var/tmp/imageio-storage/file-512-xfs-mount: wrong fs type, bad option, bad superblock on /dev/loop4, missing codepage or helper program, or other error. Traceback (most recent call last): File "/usr/local/bin/userstorage", line 10, in <module> sys.exit(main()) File "/usr/local/lib/python3.6/site-packages/userstorage/__main__.py", line 42, in main create(cfg) File "/usr/local/lib/python3.6/site-packages/userstorage/__main__.py", line 52, in create b.create() File "/usr/local/lib/python3.6/site-packages/userstorage/file.py", line 47, in create self._mount.create() File "/usr/local/lib/python3.6/site-packages/userstorage/mount.py", line 53, in create self._mount_loop() File "/usr/local/lib/python3.6/site-packages/userstorage/mount.py", line 94, in _mount_loop ["sudo", "mount", "-t", self.fstype, self._loop.path, self.path]) File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['sudo', 'mount', '-t', 'xfs', '/var/tmp/imageio-storage/file-512-xfs-loop', '/var/tmp/imageio-storage/file-512-xfs-mount']' returned non-zero exit status 32. https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/1593//ar... <https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/1593//ar... Same code runs fine in Travis: https://travis-ci.org/nirs/ovirt-imageio/jobs/595794863 <https://travis-ci.org/nirs/ovirt-imageio/jobs/595794863> And also locally on Fedora 29: $ ../jenkins/mock_configs/mock_runner.sh -C ../jenkins/mock_configs -p el8 ... ## Wed Oct  9 23:37:22 IDT 2019 Finished env: el8:epel-8-x86_64 ##      took 85 seconds ##      rc = 0 My guess is that we run el8 jobs on el7 hosts with old kernels (Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.) Here is info from failed builds: DEBUG buildroot.py:503: kernel version == 3.10.0-693.11.6.el7.ppc64le https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/1593/art... <https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/1593/art... DEBUG buildroot.py:503:  kernel version == 3.10.0-957.12.1.el7.x86_64 https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/1593/art... <https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/1593/art... and successful builds: DEBUG buildroot.py:503:  kernel version == 5.1.18-200.fc29.x86_64 My laptop Runtime kernel version: 4.15.0-1032-gcp https://travis-ci.org/nirs/ovirt-imageio/jobs/595794863 <https://travis-ci.org/nirs/ovirt-imageio/jobs/595794863> This issue will affects vdsm, using similar code to create storage for testing. For vdsm the issue is more tricky, since it requires same host/distro:  49   - tests-py37:  50             runtime-requirements:  51               host-distro: same So I think we need: - make slaves with newer kernel (fc29? fc30?) with the el8 mock env - add el8 slaves for running el8 mock with "host-distro: same"

If we don't have a solution we need to disable el8 tests, or continue testing without storage, which will skip about 200 tests. Nir

EL8 builds fail to mount loop device - kernel too old? ------------------------------------------------------ Key: OVIRT-2813 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2813 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Nir Soffer Assignee: infra I'm trying to run imageio tests on el8 mock, and the tests fail early when trying to create storage for testing: [userstorage] INFO Creating filesystem /var/tmp/imageio-storage/file-512-ext4-mount Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features. [userstorage] INFO Creating file /var/tmp/imageio-storage/file-512-ext4-mount/file [userstorage] INFO Creating backing file /var/tmp/imageio-storage/file-512-xfs-backing [userstorage] INFO Creating loop device /var/tmp/imageio-storage/file-512-xfs-loop [userstorage] INFO Creating filesystem /var/tmp/imageio-storage/file-512-xfs-mount mount: /var/tmp/imageio-storage/file-512-xfs-mount: wrong fs type, bad option, bad superblock on /dev/loop4, missing codepage or helper program, or other error. Traceback (most recent call last): File "/usr/local/bin/userstorage", line 10, in <module> sys.exit(main()) File "/usr/local/lib/python3.6/site-packages/userstorage/__main__.py", line 42, in main create(cfg) File "/usr/local/lib/python3.6/site-packages/userstorage/__main__.py", line 52, in create b.create() File "/usr/local/lib/python3.6/site-packages/userstorage/file.py", line 47, in create self._mount.create() File "/usr/local/lib/python3.6/site-packages/userstorage/mount.py", line 53, in create self._mount_loop() File "/usr/local/lib/python3.6/site-packages/userstorage/mount.py", line 94, in _mount_loop ["sudo", "mount", "-t", self.fstype, self._loop.path, self.path]) File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['sudo', 'mount', '-t', 'xfs', '/var/tmp/imageio-storage/file-512-xfs-loop', '/var/tmp/imageio-storage/file-512-xfs-mount']' returned non-zero exit status 32. https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/1593//ar... Same code runs fine in Travis: https://travis-ci.org/nirs/ovirt-imageio/jobs/595794863 And also locally on Fedora 29: $ ../jenkins/mock_configs/mock_runner.sh -C ../jenkins/mock_configs -p el8 ... ## Wed Oct 9 23:37:22 IDT 2019 Finished env: el8:epel-8-x86_64 ## took 85 seconds ## rc = 0 My guess is that we run el8 jobs on el7 hosts with old kernels (Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.) This issue will affects vdsm, using similar code to create storage for testing. For vdsm the issue is more tricky, since it requires same host/distro: 49 - tests-py37: 50 runtime-requirements: 51 host-distro: same So I think we need: - make slaves with newer kernel (fc29? fc30?) with the el8 mock env - add el8 slaves for running el8 mock with "host-distro: same" If we don't have a solution we need to disable el8 tests, or continue testing without storage, which will skip about 200 tests. Nir