Re: Tested failing because of missing loop devices

Hi all, We see too many failures of tests using loop devices. Is it possible that we run tests concurrently on the same slave, using all the available loop devices, or maybe creating races between different tests? It seems that we need new decorator for disabling tests on the CI slaves, since this environment is too fragile. Here are some failures: 01:10:33 ====================================================================== 01:10:33 ERROR: testLoopMount (mountTests.MountTests) 01:10:33 ---------------------------------------------------------------------- 01:10:33 Traceback (most recent call last): 01:10:33 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/mountTests.py", line 128, in testLoopMount 01:10:33 m.mount(mntOpts="loop") 01:10:33 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/vdsm/storage/mount.py", line 225, in mount 01:10:33 return self._runcmd(cmd, timeout) 01:10:33 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/vdsm/storage/mount.py", line 241, in _runcmd 01:10:33 raise MountError(rc, ";".join((out, err))) 01:10:33 MountError: (32, ';mount: /tmp/tmpZuJRNk: failed to setup loop device: No such file or directory\n') 01:10:33 -------------------- >> begin captured logging << -------------------- 01:10:33 Storage.Misc.excCmd: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/mkfs.ext2 -F /tmp/tmpZuJRNk (cwd None) 01:10:33 Storage.Misc.excCmd: DEBUG: SUCCESS: <err> = 'mke2fs 1.42.13 (17-May-2015)\n'; <rc> = 0 01:10:33 Storage.Misc.excCmd: DEBUG: /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -o loop /tmp/tmpZuJRNk /var/tmp/tmpJO52Xj (cwd None) 01:10:33 --------------------- >> end captured logging << --------------------- 01:10:33 01:10:33 ====================================================================== 01:10:33 ERROR: testSymlinkMount (mountTests.MountTests) 01:10:33 ---------------------------------------------------------------------- 01:10:33 Traceback (most recent call last): 01:10:33 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/mountTests.py", line 150, in testSymlinkMount 01:10:33 m.mount(mntOpts="loop") 01:10:33 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/vdsm/storage/mount.py", line 225, in mount 01:10:33 return self._runcmd(cmd, timeout) 01:10:33 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/vdsm/storage/mount.py", line 241, in _runcmd 01:10:33 raise MountError(rc, ";".join((out, err))) 01:10:33 MountError: (32, ';mount: /var/tmp/tmp1UQFPz/backing.img: failed to setup loop device: No such file or directory\n') 01:10:33 -------------------- >> begin captured logging << -------------------- 01:10:33 Storage.Misc.excCmd: DEBUG: /usr/bin/taskset --cpu-list 0-1 /sbin/mkfs.ext2 -F /var/tmp/tmp1UQFPz/backing.img (cwd None) 01:10:33 Storage.Misc.excCmd: DEBUG: SUCCESS: <err> = 'mke2fs 1.42.13 (17-May-2015)\n'; <rc> = 0 01:10:33 Storage.Misc.excCmd: DEBUG: /usr/bin/taskset --cpu-list 0-1 /usr/bin/mount -o loop /var/tmp/tmp1UQFPz/link_to_image /var/tmp/tmp1UQFPz/mountpoint (cwd None) 01:10:33 --------------------- >> end captured logging << --------------------- 01:10:33 01:10:33 ====================================================================== 01:10:33 ERROR: test_getDevicePartedInfo (parted_utils_tests.PartedUtilsTests) 01:10:33 ---------------------------------------------------------------------- 01:10:33 Traceback (most recent call last): 01:10:33 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testValidation.py", line 97, in wrapper 01:10:33 return f(*args, **kwargs) 01:10:33 File "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/parted_utils_tests.py", line 61, in setUp 01:10:33 self.assertEquals(rc, 0) 01:10:33 AssertionError: 1 != 0 01:10:33 -------------------- >> begin captured logging << -------------------- 01:10:33 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 dd if=/dev/zero of=/tmp/tmpasV8TD bs=100M count=1 (cwd None) 01:10:33 root: DEBUG: SUCCESS: <err> = '1+0 records in\n1+0 records out\n104857600 bytes (105 MB) copied, 0.368498 s, 285 MB/s\n'; <rc> = 0 01:10:33 root: DEBUG: /usr/bin/taskset --cpu-list 0-1 losetup -f --show /tmp/tmpasV8TD (cwd None) 01:10:33 root: DEBUG: FAILED: <err> = 'losetup: /tmp/tmpasV8TD: failed to set up loop device: No such file or directory\n'; <rc> = 1 01:10:33 --------------------- >> end captured logging << --------------------- Nir