[VDSM] loop device tests fail again on jenkins
We have random failures of loop device tests on jenkins (see example bellow). Barak commented that losetup -f /path does not work sometimes in the CI but I don't recall what it the alternative way to get a loop device. We need a reliable way to create a loop device for vdsm tests. What is the recommended way to do this? Until we have a reliable solution I'm going to mark this test as broken on ovirt CI: https://gerrit.ovirt.org/#/c/86241/ Please report if you see other tests fail with this error - typically: Error: Command ['losetup', '--find', '--show', '/tmp/tmp17Wqri/file'] failed with rc=1 out='' err='losetup: /tmp/tmp17Wqri/file: failed to set up loop device: No such file or directory\n' http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/20862/consol... *00:05:48.766* ======================================================================*00:05:48.766* ERROR: test_attach_detach_manually (loopback_test.TestDevice)*00:05:48.767* ----------------------------------------------------------------------*00:05:48.767* Traceback (most recent call last):*00:05:48.767* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/testValidation.py", line 191, in wrapper*00:05:48.768* return f(*args, **kwargs)*00:05:48.768* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/loopback_test.py", line 56, in test_attach_detach_manually*00:05:48.768* device.attach()*00:05:48.768* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/loopback.py", line 56, in attach*00:05:48.769* raise cmdutils.Error(cmd, rc, out, err)*00:05:48.769* Error: Command ['losetup', '--find', '--show', '/tmp/tmp17Wqri/file'] failed with rc=1 out='' err='losetup: /tmp/tmp17Wqri/file: failed to set up loop device: No such file or directory\n'*00:05:48.770* -------------------- >> begin captured logging << --------------------*00:05:48.770* 2018-01-11 13:42:45,676 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 losetup --find --show /tmp/tmp17Wqri/file (cwd None) (commands:65)*00:05:48.771* 2018-01-11 13:42:45,733 DEBUG (MainThread) [root] FAILED: <err> = 'losetup: /tmp/tmp17Wqri/file: failed to set up loop device: No such file or directory\n'; <rc> = 1 (commands:86)*00:05:48.772* --------------------- >> end captured logging << ---------------------
I thought I explained this before. Writing to losetup from inside mock gets you a device outside mock that is invisible inside. The way to solve this is to run mknod a few times to get the needed device files. You can see examples of this in the ovirt-node build scripts. Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/ בתאריך 11 בינו׳ 2018 04:35 PM, "Nir Soffer" <nsoffer@redhat.com> כתב:
We have random failures of loop device tests on jenkins (see example bellow).
Barak commented that losetup -f /path does not work sometimes in the CI but I don't recall what it the alternative way to get a loop device.
We need a reliable way to create a loop device for vdsm tests. What is the recommended way to do this?
Until we have a reliable solution I'm going to mark this test as broken on ovirt CI: https://gerrit.ovirt.org/#/c/86241/
Please report if you see other tests fail with this error - typically:
Error: Command ['losetup', '--find', '--show', '/tmp/tmp17Wqri/file'] failed with rc=1 out='' err='losetup: /tmp/tmp17Wqri/file: failed to set up loop device: No such file or directory\n'
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7- x86_64/20862/consoleFull
*00:05:48.766* ======================================================================*00:05:48.766* ERROR: test_attach_detach_manually (loopback_test.TestDevice)*00:05:48.767* ----------------------------------------------------------------------*00:05:48.767* Traceback (most recent call last):*00:05:48.767* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/testValidation.py", line 191, in wrapper*00:05:48.768* return f(*args, **kwargs)*00:05:48.768* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/loopback_test.py", line 56, in test_attach_detach_manually*00:05:48.768* device.attach()*00:05:48.768* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/loopback.py", line 56, in attach*00:05:48.769* raise cmdutils.Error(cmd, rc, out, err)*00:05:48.769* Error: Command ['losetup', '--find', '--show', '/tmp/tmp17Wqri/file'] failed with rc=1 out='' err='losetup: /tmp/tmp17Wqri/file: failed to set up loop device: No such file or directory\n'*00:05:48.770* -------------------- >> begin captured logging << --------------------*00:05:48.770* 2018-01-11 13:42:45,676 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 losetup --find --show /tmp/tmp17Wqri/file (cwd None) (commands:65)*00:05:48.771* 2018-01-11 13:42:45,733 DEBUG (MainThread) [root] FAILED: <err> = 'losetup: /tmp/tmp17Wqri/file: failed to set up loop device: No such file or directory\n'; <rc> = 1 (commands:86)*00:05:48.772* --------------------- >> end captured logging << ---------------------
On Thu, Jan 11, 2018 at 8:31 PM Barak Korren <bkorren@redhat.com> wrote:
I thought I explained this before.
Writing to losetup from inside mock gets you a device outside mock that is invisible inside.
How the tests work 99.9% of the time?
The way to solve this is to run mknod a few times to get the needed device files.
You can see examples of this in the ovirt-node build scripts.
"git grep mknod" in ovirt-node source does not show anything.
Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
בתאריך 11 בינו׳ 2018 04:35 PM, "Nir Soffer" <nsoffer@redhat.com> כתב:
We have random failures of loop device tests on jenkins (see example
bellow).
Barak commented that losetup -f /path does not work sometimes in the CI but I don't recall what it the alternative way to get a loop device.
We need a reliable way to create a loop device for vdsm tests. What is the recommended way to do this?
Until we have a reliable solution I'm going to mark this test as broken on ovirt CI: https://gerrit.ovirt.org/#/c/86241/
Please report if you see other tests fail with this error - typically:
Error: Command ['losetup', '--find', '--show', '/tmp/tmp17Wqri/file'] failed with rc=1 out='' err='losetup: /tmp/tmp17Wqri/file: failed to set up loop device: No such file or directory\n'
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/20862/consol...
*00:05:48.766* ======================================================================*00:05:48.766* ERROR: test_attach_detach_manually (loopback_test.TestDevice)*00:05:48.767* ----------------------------------------------------------------------*00:05:48.767* Traceback (most recent call last):*00:05:48.767* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/testValidation.py", line 191, in wrapper*00:05:48.768* return f(*args, **kwargs)*00:05:48.768* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/loopback_test.py", line 56, in test_attach_detach_manually*00:05:48.768* device.attach()*00:05:48.768* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/loopback.py", line 56, in attach*00:05:48.769* raise cmdutils.Error(cmd, rc, out, err)*00:05:48.769* Error: Command ['losetup', '--find', '--show', '/tmp/tmp17Wqri/file'] failed with rc=1 out='' err='losetup: /tmp/tmp17Wqri/file: failed to set up loop device: No such file or directory\n'*00:05:48.770* -------------------- >> begin captured logging << --------------------*00:05:48.770* 2018-01-11 13:42:45,676 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 losetup --find --show /tmp/tmp17Wqri/file (cwd None) (commands:65)*00:05:48.771* 2018-01-11 13:42:45,733 DEBUG (MainThread) [root] FAILED: <err> = 'losetup: /tmp/tmp17Wqri/file: failed to set up loop device: No such file or directory\n'; <rc> = 1 (commands:86)*00:05:48.772* --------------------- >> end captured logging << ---------------------
from ovirt-node-ng: build-artifacts.sh: seq 0 9 | xargs -I {} mknod /dev/loop{} b 7 {} || : On Thu, Jan 11, 2018 at 8:38 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Thu, Jan 11, 2018 at 8:31 PM Barak Korren <bkorren@redhat.com> wrote:
I thought I explained this before.
Writing to losetup from inside mock gets you a device outside mock that is invisible inside.
How the tests work 99.9% of the time?
The way to solve this is to run mknod a few times to get the needed device files.
You can see examples of this in the ovirt-node build scripts.
"git grep mknod" in ovirt-node source does not show anything.
Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
בתאריך 11 בינו׳ 2018 04:35 PM, "Nir Soffer" <nsoffer@redhat.com> כתב:
We have random failures of loop device tests on jenkins (see example
bellow).
Barak commented that losetup -f /path does not work sometimes in the CI but I don't recall what it the alternative way to get a loop device.
We need a reliable way to create a loop device for vdsm tests. What is the recommended way to do this?
Until we have a reliable solution I'm going to mark this test as broken on ovirt CI: https://gerrit.ovirt.org/#/c/86241/
Please report if you see other tests fail with this error - typically:
Error: Command ['losetup', '--find', '--show', '/tmp/tmp17Wqri/file'] failed with rc=1 out='' err='losetup: /tmp/tmp17Wqri/file: failed to set up loop device: No such file or directory\n'
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7- x86_64/20862/consoleFull
*00:05:48.766* ======================================================================*00:05:48.766* ERROR: test_attach_detach_manually (loopback_test.TestDevice)*00:05:48.767* ----------------------------------------------------------------------*00:05:48.767* Traceback (most recent call last):*00:05:48.767* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/testValidation.py", line 191, in wrapper*00:05:48.768* return f(*args, **kwargs)*00:05:48.768* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/loopback_test.py", line 56, in test_attach_detach_manually*00:05:48.768* device.attach()*00:05:48.768* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/loopback.py", line 56, in attach*00:05:48.769* raise cmdutils.Error(cmd, rc, out, err)*00:05:48.769* Error: Command ['losetup', '--find', '--show', '/tmp/tmp17Wqri/file'] failed with rc=1 out='' err='losetup: /tmp/tmp17Wqri/file: failed to set up loop device: No such file or directory\n'*00:05:48.770* -------------------- >> begin captured logging << --------------------*00:05:48.770* 2018-01-11 13:42:45,676 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 losetup --find --show /tmp/tmp17Wqri/file (cwd None) (commands:65)*00:05:48.771* 2018-01-11 13:42:45,733 DEBUG (MainThread) [root] FAILED: <err> = 'losetup: /tmp/tmp17Wqri/file: failed to set up loop device: No such file or directory\n'; <rc> = 1 (commands:86)*00:05:48.772* --------------------- >> end captured logging << ---------------------
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Thu, Jan 11, 2018 at 9:08 PM Yuval Turgeman <yuvalt@redhat.com> wrote:
from ovirt-node-ng:
build-artifacts.sh: seq 0 9 | xargs -I {} mknod /dev/loop{} b 7 {} || :
Thanks, will try that.
On Thu, Jan 11, 2018 at 8:38 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Thu, Jan 11, 2018 at 8:31 PM Barak Korren <bkorren@redhat.com> wrote:
I thought I explained this before.
Writing to losetup from inside mock gets you a device outside mock that is invisible inside.
How the tests work 99.9% of the time?
The way to solve this is to run mknod a few times to get the needed device files.
You can see examples of this in the ovirt-node build scripts.
"git grep mknod" in ovirt-node source does not show anything.
Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
בתאריך 11 בינו׳ 2018 04:35 PM, "Nir Soffer" <nsoffer@redhat.com> כתב:
We have random failures of loop device tests on jenkins (see example
bellow).
Barak commented that losetup -f /path does not work sometimes in the CI but I don't recall what it the alternative way to get a loop device.
We need a reliable way to create a loop device for vdsm tests. What is the recommended way to do this?
Until we have a reliable solution I'm going to mark this test as broken on ovirt CI: https://gerrit.ovirt.org/#/c/86241/
Please report if you see other tests fail with this error - typically:
Error: Command ['losetup', '--find', '--show', '/tmp/tmp17Wqri/file'] failed with rc=1 out='' err='losetup: /tmp/tmp17Wqri/file: failed to set up loop device: No such file or directory\n'
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/20862/consol...
*00:05:48.766* ======================================================================*00:05:48.766* ERROR: test_attach_detach_manually (loopback_test.TestDevice)*00:05:48.767* ----------------------------------------------------------------------*00:05:48.767* Traceback (most recent call last):*00:05:48.767* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/testValidation.py", line 191, in wrapper*00:05:48.768* return f(*args, **kwargs)*00:05:48.768* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/loopback_test.py", line 56, in test_attach_detach_manually*00:05:48.768* device.attach()*00:05:48.768* File "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/loopback.py", line 56, in attach*00:05:48.769* raise cmdutils.Error(cmd, rc, out, err)*00:05:48.769* Error: Command ['losetup', '--find', '--show', '/tmp/tmp17Wqri/file'] failed with rc=1 out='' err='losetup: /tmp/tmp17Wqri/file: failed to set up loop device: No such file or directory\n'*00:05:48.770* -------------------- >> begin captured logging << --------------------*00:05:48.770* 2018-01-11 13:42:45,676 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-1 losetup --find --show /tmp/tmp17Wqri/file (cwd None) (commands:65)*00:05:48.771* 2018-01-11 13:42:45,733 DEBUG (MainThread) [root] FAILED: <err> = 'losetup: /tmp/tmp17Wqri/file: failed to set up loop device: No such file or directory\n'; <rc> = 1 (commands:86)*00:05:48.772* --------------------- >> end captured logging << ---------------------
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
participants (3)
-
Barak Korren -
Nir Soffer -
Yuval Turgeman