Vdsm: Failing test_no_match
Hi, test_no_match fails on CI most of the time (but not always) in https://gerrit.ovirt.org/95518: _____________________________ test_no_match[qcow2] _____________________________ 11:30:16 11:30:16 tmpdir = local('/var/tmp/vdsm/test_no_match_qcow2_0'), image_format = 'qcow2' 11:30:16 11:30:16 def test_no_match(tmpdir, image_format): 11:30:16 path = str(tmpdir.join('test.' + image_format)) 11:30:16 op = qemuimg.create(path, '1m', image_format) 11:30:16 op.run() 11:30:16 qemuio.write_pattern(path, image_format, pattern=2) 11:30:16 with pytest.raises(qemuio.VerificationError): 11:30:16 > qemuio.verify_pattern(path, image_format, pattern=4) 11:30:16 11:30:16 storage/qemuio_test.py:59: 11:30:16 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 11:30:16 11:30:16 path = '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2', format = 'qcow2' 11:30:16 offset = 512, len = 1024, pattern = 4 11:30:16 11:30:16 def verify_pattern(path, format, offset=512, len=1024, pattern=5): 11:30:16 read_cmd = 'read -P %d -s 0 -l %d %d %d' % (pattern, len, offset, len) 11:30:16 cmd = ['qemu-io', '-f', format, '-c', read_cmd, path] 11:30:16 rc, out, err = commands.execCmd(cmd, raw=True) 11:30:16 if rc != 0 or err != b"": 11:30:16 > raise cmdutils.Error(cmd, rc, out, err) 11:30:16 E Error: Command ['qemu-io', '-f', 'qcow2', '-c', 'read -P 4 -s 0 -l 1024 512 1024', '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2'] failed with rc=1 out='Pattern verification failed at offset 512, 1024 bytes\nread 1024/1024 bytes at offset 512\n1 KiB, 1 ops; 0.0002 sec (3.756 MiB/sec and 3846.1538 ops/sec)\n' err='' 11:30:16 11:30:16 storage/qemuio.py:50: Error (Similarly for raw.) You can see the complete test run log here (or in other CI runs of the patch): https://jenkins.ovirt.org/job/vdsm_master_check-patch-fc28-x86_64/2040/conso... It fails on both Fedora and CentOS. It may or may not be related to the fact that QEMU 2.11 is used in the failed runs. Any idea what could be wrong? Thanks, Milan
On Wed, Nov 21, 2018, 17:46 Milan Zamazal <mzamazal@redhat.com wrote:
Hi, test_no_match fails on CI most of the time (but not always) in https://gerrit.ovirt.org/95518:
_____________________________ test_no_match[qcow2] _____________________________ 11:30:16 11:30:16 tmpdir = local('/var/tmp/vdsm/test_no_match_qcow2_0'), image_format = 'qcow2' 11:30:16 11:30:16 def test_no_match(tmpdir, image_format): 11:30:16 path = str(tmpdir.join('test.' + image_format)) 11:30:16 op = qemuimg.create(path, '1m', image_format) 11:30:16 op.run() 11:30:16 qemuio.write_pattern(path, image_format, pattern=2) 11:30:16 with pytest.raises(qemuio.VerificationError): 11:30:16 > qemuio.verify_pattern(path, image_format, pattern=4) 11:30:16 11:30:16 storage/qemuio_test.py:59: 11:30:16 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 11:30:16 11:30:16 path = '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2', format = 'qcow2' 11:30:16 offset = 512, len = 1024, pattern = 4 11:30:16 11:30:16 def verify_pattern(path, format, offset=512, len=1024, pattern=5): 11:30:16 read_cmd = 'read -P %d -s 0 -l %d %d %d' % (pattern, len, offset, len) 11:30:16 cmd = ['qemu-io', '-f', format, '-c', read_cmd, path] 11:30:16 rc, out, err = commands.execCmd(cmd, raw=True) 11:30:16 if rc != 0 or err != b"": 11:30:16 > raise cmdutils.Error(cmd, rc, out, err) 11:30:16 E Error: Command ['qemu-io', '-f', 'qcow2', '-c', 'read -P 4 -s 0 -l 1024 512 1024', '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2'] failed with rc=1 out='Pattern verification failed at offset 512, 1024 bytes\nread 1024/1024 bytes at offset 512\n1 KiB, 1 ops; 0.0002 sec (3.756 MiB/sec and 3846.1538 ops/sec)\n' err='' 11:30:16 11:30:16 storage/qemuio.py:50: Error
(Similarly for raw.)
You can see the complete test run log here (or in other CI runs of the patch):
https://jenkins.ovirt.org/job/vdsm_master_check-patch-fc28-x86_64/2040/conso...
It fails on both Fedora and CentOS. It may or may not be related to the fact that QEMU 2.11 is used in the failed runs.
Any idea what could be wrong?
Yes. Qemu-io was fixed lately to fail when pattern does not match, but our wrapper still expects the old behaviour (return 0, log warning). Are you sure you run 2.11 and not 2.12? We will fix this soon. Nir
On 21 Nov 2018, at 21:10, Nir Soffer <nsoffer@redhat.com> wrote:
On Wed, Nov 21, 2018, 17:46 Milan Zamazal <mzamazal@redhat.com <mailto:mzamazal@redhat.com> wrote: Hi, test_no_match fails on CI most of the time (but not always) in https://gerrit.ovirt.org/95518 <https://gerrit.ovirt.org/95518>:
_____________________________ test_no_match[qcow2] _____________________________ 11:30:16 11:30:16 tmpdir = local('/var/tmp/vdsm/test_no_match_qcow2_0'), image_format = 'qcow2' 11:30:16 11:30:16 def test_no_match(tmpdir, image_format): 11:30:16 path = str(tmpdir.join('test.' + image_format)) 11:30:16 op = qemuimg.create(path, '1m', image_format) 11:30:16 op.run() 11:30:16 qemuio.write_pattern(path, image_format, pattern=2) 11:30:16 with pytest.raises(qemuio.VerificationError): 11:30:16 > qemuio.verify_pattern(path, image_format, pattern=4) 11:30:16 11:30:16 storage/qemuio_test.py:59: 11:30:16 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 11:30:16 11:30:16 path = '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2', format = 'qcow2' 11:30:16 offset = 512, len = 1024, pattern = 4 11:30:16 11:30:16 def verify_pattern(path, format, offset=512, len=1024, pattern=5): 11:30:16 read_cmd = 'read -P %d -s 0 -l %d %d %d' % (pattern, len, offset, len) 11:30:16 cmd = ['qemu-io', '-f', format, '-c', read_cmd, path] 11:30:16 rc, out, err = commands.execCmd(cmd, raw=True) 11:30:16 if rc != 0 or err != b"": 11:30:16 > raise cmdutils.Error(cmd, rc, out, err) 11:30:16 E Error: Command ['qemu-io', '-f', 'qcow2', '-c', 'read -P 4 -s 0 -l 1024 512 1024', '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2'] failed with rc=1 out='Pattern verification failed at offset 512, 1024 bytes\nread 1024/1024 bytes at offset 512\n1 KiB, 1 ops; 0.0002 sec (3.756 MiB/sec and 3846.1538 ops/sec)\n' err='' 11:30:16 11:30:16 storage/qemuio.py:50: Error
(Similarly for raw.)
You can see the complete test run log here (or in other CI runs of the patch): https://jenkins.ovirt.org/job/vdsm_master_check-patch-fc28-x86_64/2040/conso... <https://jenkins.ovirt.org/job/vdsm_master_check-patch-fc28-x86_64/2040/consoleFull>
It fails on both Fedora and CentOS. It may or may not be related to the fact that QEMU 2.11 is used in the failed runs.
Any idea what could be wrong?
Yes. Qemu-io was fixed lately to fail when pattern does not match, but our wrapper still expects the old behaviour (return 0, log warning).
Are you sure you run 2.11 and not 2.12?
we’re testing 2.12 in vdsm.spec, but the logs show that the actual run is done still on 2.11. but who knows what it does for real….
We will fix this soon.
Nir
Nir Soffer <nsoffer@redhat.com> writes:
On Wed, Nov 21, 2018, 17:46 Milan Zamazal <mzamazal@redhat.com wrote:
Hi, test_no_match fails on CI most of the time (but not always) in https://gerrit.ovirt.org/95518:
_____________________________ test_no_match[qcow2] _____________________________ 11:30:16 11:30:16 tmpdir = local('/var/tmp/vdsm/test_no_match_qcow2_0'), image_format = 'qcow2' 11:30:16 11:30:16 def test_no_match(tmpdir, image_format): 11:30:16 path = str(tmpdir.join('test.' + image_format)) 11:30:16 op = qemuimg.create(path, '1m', image_format) 11:30:16 op.run() 11:30:16 qemuio.write_pattern(path, image_format, pattern=2) 11:30:16 with pytest.raises(qemuio.VerificationError): 11:30:16 > qemuio.verify_pattern(path, image_format, pattern=4) 11:30:16 11:30:16 storage/qemuio_test.py:59: 11:30:16 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 11:30:16 11:30:16 path = '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2', format = 'qcow2' 11:30:16 offset = 512, len = 1024, pattern = 4 11:30:16 11:30:16 def verify_pattern(path, format, offset=512, len=1024, pattern=5): 11:30:16 read_cmd = 'read -P %d -s 0 -l %d %d %d' % (pattern, len, offset, len) 11:30:16 cmd = ['qemu-io', '-f', format, '-c', read_cmd, path] 11:30:16 rc, out, err = commands.execCmd(cmd, raw=True) 11:30:16 if rc != 0 or err != b"": 11:30:16 > raise cmdutils.Error(cmd, rc, out, err) 11:30:16 E Error: Command ['qemu-io', '-f', 'qcow2', '-c', 'read -P 4 -s 0 -l 1024 512 1024', '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2'] failed with rc=1 out='Pattern verification failed at offset 512, 1024 bytes\nread 1024/1024 bytes at offset 512\n1 KiB, 1 ops; 0.0002 sec (3.756 MiB/sec and 3846.1538 ops/sec)\n' err='' 11:30:16 11:30:16 storage/qemuio.py:50: Error
(Similarly for raw.)
You can see the complete test run log here (or in other CI runs of the patch):
https://jenkins.ovirt.org/job/vdsm_master_check-patch-fc28-x86_64/2040/conso...
It fails on both Fedora and CentOS. It may or may not be related to the fact that QEMU 2.11 is used in the failed runs.
Any idea what could be wrong?
Yes. Qemu-io was fixed lately to fail when pattern does not match, but our wrapper still expects the old behaviour (return 0, log warning).
I see, thank you for explanation.
Are you sure you run 2.11 and not 2.12?
Actually not. Looking into the CI log once more, I can see it reports initially installed QEMU version before additional repos are added. There are no reports on QEMU versions or upgrades afterwards but that may be just silence of some automation script. Since new QEMU version would be expected with the added repos and it would explain the test failure, let's assume it's indeed a newer QEMU.
We will fix this soon.
OK, thank you. We can disable the test temporarily in our patches updating repos & requirements and re-enable it before merge or later, depending on availability of your fix.
On Thu, Nov 22, 2018 at 2:08 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Nir Soffer <nsoffer@redhat.com> writes:
On Wed, Nov 21, 2018, 17:46 Milan Zamazal <mzamazal@redhat.com wrote:
Hi, test_no_match fails on CI most of the time (but not always) in https://gerrit.ovirt.org/95518:
_____________________________ test_no_match[qcow2] _____________________________ 11:30:16 11:30:16 tmpdir = local('/var/tmp/vdsm/test_no_match_qcow2_0'), image_format = 'qcow2' 11:30:16 11:30:16 def test_no_match(tmpdir, image_format): 11:30:16 path = str(tmpdir.join('test.' + image_format)) 11:30:16 op = qemuimg.create(path, '1m', image_format) 11:30:16 op.run() 11:30:16 qemuio.write_pattern(path, image_format, pattern=2) 11:30:16 with pytest.raises(qemuio.VerificationError): 11:30:16 > qemuio.verify_pattern(path, image_format, pattern=4) 11:30:16 11:30:16 storage/qemuio_test.py:59: 11:30:16 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 11:30:16 11:30:16 path = '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2', format = 'qcow2' 11:30:16 offset = 512, len = 1024, pattern = 4 11:30:16 11:30:16 def verify_pattern(path, format, offset=512, len=1024, pattern=5): 11:30:16 read_cmd = 'read -P %d -s 0 -l %d %d %d' % (pattern, len, offset, len) 11:30:16 cmd = ['qemu-io', '-f', format, '-c', read_cmd, path] 11:30:16 rc, out, err = commands.execCmd(cmd, raw=True) 11:30:16 if rc != 0 or err != b"": 11:30:16 > raise cmdutils.Error(cmd, rc, out, err) 11:30:16 E Error: Command ['qemu-io', '-f', 'qcow2', '-c', 'read -P 4 -s 0 -l 1024 512 1024', '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2'] failed with rc=1 out='Pattern verification failed at offset 512, 1024 bytes\nread 1024/1024 bytes at offset 512\n1 KiB, 1 ops; 0.0002 sec (3.756 MiB/sec and 3846.1538 ops/sec)\n' err='' 11:30:16 11:30:16 storage/qemuio.py:50: Error
(Similarly for raw.)
You can see the complete test run log here (or in other CI runs of the patch):
https://jenkins.ovirt.org/job/vdsm_master_check-patch-fc28-x86_64/2040/conso...
It fails on both Fedora and CentOS. It may or may not be related to the fact that QEMU 2.11 is used in the failed runs.
Any idea what could be wrong?
Yes. Qemu-io was fixed lately to fail when pattern does not match, but our wrapper still expects the old behaviour (return 0, log warning).
I see, thank you for explanation.
Are you sure you run 2.11 and not 2.12?
Actually not. Looking into the CI log once more, I can see it reports initially installed QEMU version before additional repos are added. There are no reports on QEMU versions or upgrades afterwards but that may be just silence of some automation script. Since new QEMU version would be expected with the added repos and it would explain the test failure, let's assume it's indeed a newer QEMU.
We will fix this soon.
OK, thank you. We can disable the test temporarily in our patches updating repos & requirements and re-enable it before merge or later, depending on availability of your fix.
I cannot reproduce the error on Fedora 28 (qemu-img-2.12.0-0.5.rc1.fc28.x86_64) but I hope this change will fix your setup. Can you confirm that this solves the issue? Nir
On Sun, Nov 25, 2018 at 11:07 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Thu, Nov 22, 2018 at 2:08 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Nir Soffer <nsoffer@redhat.com> writes:
On Wed, Nov 21, 2018, 17:46 Milan Zamazal <mzamazal@redhat.com wrote:
Hi, test_no_match fails on CI most of the time (but not always) in https://gerrit.ovirt.org/95518:
_____________________________ test_no_match[qcow2] _____________________________ 11:30:16 11:30:16 tmpdir = local('/var/tmp/vdsm/test_no_match_qcow2_0'), image_format = 'qcow2' 11:30:16 11:30:16 def test_no_match(tmpdir, image_format): 11:30:16 path = str(tmpdir.join('test.' + image_format)) 11:30:16 op = qemuimg.create(path, '1m', image_format) 11:30:16 op.run() 11:30:16 qemuio.write_pattern(path, image_format, pattern=2) 11:30:16 with pytest.raises(qemuio.VerificationError): 11:30:16 > qemuio.verify_pattern(path, image_format, pattern=4) 11:30:16 11:30:16 storage/qemuio_test.py:59: 11:30:16 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 11:30:16 11:30:16 path = '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2', format = 'qcow2' 11:30:16 offset = 512, len = 1024, pattern = 4 11:30:16 11:30:16 def verify_pattern(path, format, offset=512, len=1024, pattern=5): 11:30:16 read_cmd = 'read -P %d -s 0 -l %d %d %d' % (pattern, len, offset, len) 11:30:16 cmd = ['qemu-io', '-f', format, '-c', read_cmd, path] 11:30:16 rc, out, err = commands.execCmd(cmd, raw=True) 11:30:16 if rc != 0 or err != b"": 11:30:16 > raise cmdutils.Error(cmd, rc, out, err) 11:30:16 E Error: Command ['qemu-io', '-f', 'qcow2', '-c', 'read -P 4 -s 0 -l 1024 512 1024', '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2'] failed with rc=1 out='Pattern verification failed at offset 512, 1024 bytes\nread 1024/1024 bytes at offset 512\n1 KiB, 1 ops; 0.0002 sec (3.756 MiB/sec and 3846.1538 ops/sec)\n' err='' 11:30:16 11:30:16 storage/qemuio.py:50: Error
(Similarly for raw.)
You can see the complete test run log here (or in other CI runs of the patch):
https://jenkins.ovirt.org/job/vdsm_master_check-patch-fc28-x86_64/2040/conso...
It fails on both Fedora and CentOS. It may or may not be related to
the
fact that QEMU 2.11 is used in the failed runs.
Any idea what could be wrong?
Yes. Qemu-io was fixed lately to fail when pattern does not match, but our wrapper still expects the old behaviour (return 0, log warning).
I see, thank you for explanation.
Are you sure you run 2.11 and not 2.12?
Actually not. Looking into the CI log once more, I can see it reports initially installed QEMU version before additional repos are added. There are no reports on QEMU versions or upgrades afterwards but that may be just silence of some automation script. Since new QEMU version would be expected with the added repos and it would explain the test failure, let's assume it's indeed a newer QEMU.
We will fix this soon.
OK, thank you. We can disable the test temporarily in our patches updating repos & requirements and re-enable it before merge or later, depending on availability of your fix.
I cannot reproduce the error on Fedora 28 (qemu-img-2.12.0-0.5.rc1.fc28.x86_64) but I hope this change will fix your setup.
Can you confirm that this solves the issue?
CI failed once with "target is busy" error in mount tests. This was reported few weeks ago by Germnao. I posted https://gerrit.ovirt.org/c/95719/ If you see these errors again, you can rebase on this patch to check if it solve this issue. Nir
Nir Soffer <nsoffer@redhat.com> writes:
On Thu, Nov 22, 2018 at 2:08 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Nir Soffer <nsoffer@redhat.com> writes:
On Wed, Nov 21, 2018, 17:46 Milan Zamazal <mzamazal@redhat.com wrote:
Hi, test_no_match fails on CI most of the time (but not always) in https://gerrit.ovirt.org/95518:
_____________________________ test_no_match[qcow2] _____________________________ 11:30:16 11:30:16 tmpdir = local('/var/tmp/vdsm/test_no_match_qcow2_0'), image_format = 'qcow2' 11:30:16 11:30:16 def test_no_match(tmpdir, image_format): 11:30:16 path = str(tmpdir.join('test.' + image_format)) 11:30:16 op = qemuimg.create(path, '1m', image_format) 11:30:16 op.run() 11:30:16 qemuio.write_pattern(path, image_format, pattern=2) 11:30:16 with pytest.raises(qemuio.VerificationError): 11:30:16 > qemuio.verify_pattern(path, image_format, pattern=4) 11:30:16 11:30:16 storage/qemuio_test.py:59: 11:30:16 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 11:30:16 11:30:16 path = '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2', format = 'qcow2' 11:30:16 offset = 512, len = 1024, pattern = 4 11:30:16 11:30:16 def verify_pattern(path, format, offset=512, len=1024, pattern=5): 11:30:16 read_cmd = 'read -P %d -s 0 -l %d %d %d' % (pattern, len, offset, len) 11:30:16 cmd = ['qemu-io', '-f', format, '-c', read_cmd, path] 11:30:16 rc, out, err = commands.execCmd(cmd, raw=True) 11:30:16 if rc != 0 or err != b"": 11:30:16 > raise cmdutils.Error(cmd, rc, out, err) 11:30:16 E Error: Command ['qemu-io', '-f', 'qcow2', '-c', 'read -P 4 -s 0 -l 1024 512 1024', '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2'] failed with rc=1 out='Pattern verification failed at offset 512, 1024 bytes\nread 1024/1024 bytes at offset 512\n1 KiB, 1 ops; 0.0002 sec (3.756 MiB/sec and 3846.1538 ops/sec)\n' err='' 11:30:16 11:30:16 storage/qemuio.py:50: Error
(Similarly for raw.)
You can see the complete test run log here (or in other CI runs of the patch):
https://jenkins.ovirt.org/job/vdsm_master_check-patch-fc28-x86_64/2040/conso...
It fails on both Fedora and CentOS. It may or may not be related to the fact that QEMU 2.11 is used in the failed runs.
Any idea what could be wrong?
Yes. Qemu-io was fixed lately to fail when pattern does not match, but our wrapper still expects the old behaviour (return 0, log warning).
I see, thank you for explanation.
Are you sure you run 2.11 and not 2.12?
Actually not. Looking into the CI log once more, I can see it reports initially installed QEMU version before additional repos are added. There are no reports on QEMU versions or upgrades afterwards but that may be just silence of some automation script. Since new QEMU version would be expected with the added repos and it would explain the test failure, let's assume it's indeed a newer QEMU.
We will fix this soon.
OK, thank you. We can disable the test temporarily in our patches updating repos & requirements and re-enable it before merge or later, depending on availability of your fix.
I cannot reproduce the error on Fedora 28 (qemu-img-2.12.0-0.5.rc1.fc28.x86_64) ^^^^^^^^^^^^^^ This looks suspicious. Where do you get it from? Perhaps from the old virt-preview repo? Please note there is a copr virt-preview repo now, see https://fedoraproject.org/wiki/Virtualization_Preview_Repository and https://copr.fedorainfracloud.org/coprs/g/virtmaint-sig/virt-preview/, which should be up-to-date and contain QEMU 3.1.
but I hope this change will fix your setup.
Which change?
Can you confirm that this solves the issue?
Nir
On Mon, Nov 26, 2018 at 12:27 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Nir Soffer <nsoffer@redhat.com> writes:
On Thu, Nov 22, 2018 at 2:08 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Nir Soffer <nsoffer@redhat.com> writes:
On Wed, Nov 21, 2018, 17:46 Milan Zamazal <mzamazal@redhat.com wrote:
Hi, test_no_match fails on CI most of the time (but not always) in https://gerrit.ovirt.org/95518:
_____________________________ test_no_match[qcow2] _____________________________ 11:30:16 11:30:16 tmpdir = local('/var/tmp/vdsm/test_no_match_qcow2_0'), image_format = 'qcow2' 11:30:16 11:30:16 def test_no_match(tmpdir, image_format): 11:30:16 path = str(tmpdir.join('test.' + image_format)) 11:30:16 op = qemuimg.create(path, '1m', image_format) 11:30:16 op.run() 11:30:16 qemuio.write_pattern(path, image_format, pattern=2) 11:30:16 with pytest.raises(qemuio.VerificationError): 11:30:16 > qemuio.verify_pattern(path, image_format, pattern=4) 11:30:16 11:30:16 storage/qemuio_test.py:59: 11:30:16 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 11:30:16 11:30:16 path = '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2', format = 'qcow2' 11:30:16 offset = 512, len = 1024, pattern = 4 11:30:16 11:30:16 def verify_pattern(path, format, offset=512, len=1024, pattern=5): 11:30:16 read_cmd = 'read -P %d -s 0 -l %d %d %d' % (pattern, len, offset, len) 11:30:16 cmd = ['qemu-io', '-f', format, '-c', read_cmd, path] 11:30:16 rc, out, err = commands.execCmd(cmd, raw=True) 11:30:16 if rc != 0 or err != b"": 11:30:16 > raise cmdutils.Error(cmd, rc, out, err) 11:30:16 E Error: Command ['qemu-io', '-f', 'qcow2', '-c', 'read -P 4 -s 0 -l 1024 512 1024', '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2'] failed with rc=1 out='Pattern verification failed at offset 512, 1024 bytes\nread 1024/1024 bytes at offset 512\n1 KiB, 1 ops; 0.0002 sec (3.756 MiB/sec and 3846.1538 ops/sec)\n' err='' 11:30:16 11:30:16 storage/qemuio.py:50: Error
(Similarly for raw.)
You can see the complete test run log here (or in other CI runs of the patch):
It fails on both Fedora and CentOS. It may or may not be related to
https://jenkins.ovirt.org/job/vdsm_master_check-patch-fc28-x86_64/2040/conso... the
fact that QEMU 2.11 is used in the failed runs.
Any idea what could be wrong?
Yes. Qemu-io was fixed lately to fail when pattern does not match, but our wrapper still expects the old behaviour (return 0, log warning).
I see, thank you for explanation.
Are you sure you run 2.11 and not 2.12?
Actually not. Looking into the CI log once more, I can see it reports initially installed QEMU version before additional repos are added. There are no reports on QEMU versions or upgrades afterwards but that may be just silence of some automation script. Since new QEMU version would be expected with the added repos and it would explain the test failure, let's assume it's indeed a newer QEMU.
We will fix this soon.
OK, thank you. We can disable the test temporarily in our patches updating repos & requirements and re-enable it before merge or later, depending on availability of your fix.
I cannot reproduce the error on Fedora 28 (qemu-img-2.12.0-0.5.rc1.fc28.x86_64) ^^^^^^^^^^^^^^ This looks suspicious. Where do you get it from? Perhaps from the old virt-preview repo? Please note there is a copr virt-preview repo now, see https://fedoraproject.org/wiki/Virtualization_Preview_Repository and https://copr.fedorainfracloud.org/coprs/g/virtmaint-sig/virt-preview/, which should be up-to-date and contain QEMU 3.1.
I'm using the virt-preview repos enabled by this: http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm If the repo was changed, updates to this rpm should have modified my system to include the new repo. Sandro, do we need to change ovirt-release-master.rpm?
but I hope this change will fix your setup.
Which change?
Looks like you found it, but anyway: https://gerrit.ovirt.org/c/95718/ Nir
Nir Soffer <nsoffer@redhat.com> writes:
On Mon, Nov 26, 2018 at 12:27 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Nir Soffer <nsoffer@redhat.com> writes:
On Thu, Nov 22, 2018 at 2:08 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Nir Soffer <nsoffer@redhat.com> writes:
On Wed, Nov 21, 2018, 17:46 Milan Zamazal <mzamazal@redhat.com wrote:
Hi, test_no_match fails on CI most of the time (but not always) in https://gerrit.ovirt.org/95518:
_____________________________ test_no_match[qcow2] _____________________________ 11:30:16 11:30:16 tmpdir = local('/var/tmp/vdsm/test_no_match_qcow2_0'), image_format = 'qcow2' 11:30:16 11:30:16 def test_no_match(tmpdir, image_format): 11:30:16 path = str(tmpdir.join('test.' + image_format)) 11:30:16 op = qemuimg.create(path, '1m', image_format) 11:30:16 op.run() 11:30:16 qemuio.write_pattern(path, image_format, pattern=2) 11:30:16 with pytest.raises(qemuio.VerificationError): 11:30:16 > qemuio.verify_pattern(path, image_format, pattern=4) 11:30:16 11:30:16 storage/qemuio_test.py:59: 11:30:16 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 11:30:16 11:30:16 path = '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2', format = 'qcow2' 11:30:16 offset = 512, len = 1024, pattern = 4 11:30:16 11:30:16 def verify_pattern(path, format, offset=512, len=1024, pattern=5): 11:30:16 read_cmd = 'read -P %d -s 0 -l %d %d %d' % (pattern, len, offset, len) 11:30:16 cmd = ['qemu-io', '-f', format, '-c', read_cmd, path] 11:30:16 rc, out, err = commands.execCmd(cmd, raw=True) 11:30:16 if rc != 0 or err != b"": 11:30:16 > raise cmdutils.Error(cmd, rc, out, err) 11:30:16 E Error: Command ['qemu-io', '-f', 'qcow2', '-c', 'read -P 4 -s 0 -l 1024 512 1024', '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2'] failed with rc=1 out='Pattern verification failed at offset 512, 1024 bytes\nread 1024/1024 bytes at offset 512\n1 KiB, 1 ops; 0.0002 sec (3.756 MiB/sec and 3846.1538 ops/sec)\n' err='' 11:30:16 11:30:16 storage/qemuio.py:50: Error
(Similarly for raw.)
You can see the complete test run log here (or in other CI runs of the patch):
It fails on both Fedora and CentOS. It may or may not be related to
https://jenkins.ovirt.org/job/vdsm_master_check-patch-fc28-x86_64/2040/conso... the
fact that QEMU 2.11 is used in the failed runs.
Any idea what could be wrong?
Yes. Qemu-io was fixed lately to fail when pattern does not match, but our wrapper still expects the old behaviour (return 0, log warning).
I see, thank you for explanation.
Are you sure you run 2.11 and not 2.12?
Actually not. Looking into the CI log once more, I can see it reports initially installed QEMU version before additional repos are added. There are no reports on QEMU versions or upgrades afterwards but that may be just silence of some automation script. Since new QEMU version would be expected with the added repos and it would explain the test failure, let's assume it's indeed a newer QEMU.
We will fix this soon.
OK, thank you. We can disable the test temporarily in our patches updating repos & requirements and re-enable it before merge or later, depending on availability of your fix.
I cannot reproduce the error on Fedora 28 (qemu-img-2.12.0-0.5.rc1.fc28.x86_64) ^^^^^^^^^^^^^^ This looks suspicious. Where do you get it from? Perhaps from the old virt-preview repo? Please note there is a copr virt-preview repo now, see https://fedoraproject.org/wiki/Virtualization_Preview_Repository and https://copr.fedorainfracloud.org/coprs/g/virtmaint-sig/virt-preview/, which should be up-to-date and contain QEMU 3.1.
I'm using the virt-preview repos enabled by this: http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
If the repo was changed, updates to this rpm should have modified my system to include the new repo.
I see. Sandro and I have already updated ovirt-release last week and I can see the repo in https://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc28/noarch/ovirt-.... Do you have the latest ovirt-release-master.rpm?
Sandro, do we need to change ovirt-release-master.rpm?
but I hope this change will fix your setup.
Which change?
Looks like you found it, but anyway: https://gerrit.ovirt.org/c/95718/
Sorry, I responded before reading gerrit notifications. Michal, would you rebase your patch on the fix above to test it?
On Mon, Nov 26, 2018 at 2:25 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Nir Soffer <nsoffer@redhat.com> writes:
On Mon, Nov 26, 2018 at 12:27 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Nir Soffer <nsoffer@redhat.com> writes:
On Thu, Nov 22, 2018 at 2:08 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Nir Soffer <nsoffer@redhat.com> writes:
On Wed, Nov 21, 2018, 17:46 Milan Zamazal <mzamazal@redhat.com wrote:
> Hi, test_no_match fails on CI most of the time (but not always) in > https://gerrit.ovirt.org/95518: > > _____________________________ test_no_match[qcow2] > _____________________________ > 11:30:16 > 11:30:16 tmpdir = local('/var/tmp/vdsm/test_no_match_qcow2_0'), > image_format = 'qcow2' > 11:30:16 > 11:30:16 def test_no_match(tmpdir, image_format): > 11:30:16 path = str(tmpdir.join('test.' + image_format)) > 11:30:16 op = qemuimg.create(path, '1m', image_format) > 11:30:16 op.run() > 11:30:16 qemuio.write_pattern(path, image_format, pattern=2) > 11:30:16 with pytest.raises(qemuio.VerificationError): > 11:30:16 > qemuio.verify_pattern(path, image_format, pattern=4) > 11:30:16 > 11:30:16 storage/qemuio_test.py:59: > 11:30:16 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ _ _ > 11:30:16 > 11:30:16 path = '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2', format > = 'qcow2' > 11:30:16 offset = 512, len = 1024, pattern = 4 > 11:30:16 > 11:30:16 def verify_pattern(path, format, offset=512, len=1024, > pattern=5): > 11:30:16 read_cmd = 'read -P %d -s 0 -l %d %d %d' % (pattern, > len, offset, len) > 11:30:16 cmd = ['qemu-io', '-f', format, '-c', read_cmd, path] > 11:30:16 rc, out, err = commands.execCmd(cmd, raw=True) > 11:30:16 if rc != 0 or err != b"": > 11:30:16 > raise cmdutils.Error(cmd, rc, out, err) > 11:30:16 E Error: Command ['qemu-io', '-f', 'qcow2', '-c', > 'read -P 4 -s 0 -l 1024 512 1024', > '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2'] failed with rc=1 > out='Pattern verification failed at offset 512, 1024 bytes\nread 1024/1024 > bytes at offset 512\n1 KiB, 1 ops; 0.0002 sec (3.756 MiB/sec and 3846.1538 > ops/sec)\n' err='' > 11:30:16 > 11:30:16 storage/qemuio.py:50: Error > > (Similarly for raw.) > > You can see the complete test run log here (or in other CI runs of the > patch): > >
https://jenkins.ovirt.org/job/vdsm_master_check-patch-fc28-x86_64/2040/conso...
> > It fails on both Fedora and CentOS. It may or may not be related to the > fact that QEMU 2.11 is used in the failed runs. > > Any idea what could be wrong?
Yes. Qemu-io was fixed lately to fail when pattern does not match, but our wrapper still expects the old behaviour (return 0, log warning).
I see, thank you for explanation.
Are you sure you run 2.11 and not 2.12?
Actually not. Looking into the CI log once more, I can see it reports initially installed QEMU version before additional repos are added. There are no reports on QEMU versions or upgrades afterwards but that may be just silence of some automation script. Since new QEMU version would be expected with the added repos and it would explain the test failure, let's assume it's indeed a newer QEMU.
We will fix this soon.
OK, thank you. We can disable the test temporarily in our patches updating repos & requirements and re-enable it before merge or later, depending on availability of your fix.
I cannot reproduce the error on Fedora 28 (qemu-img-2.12.0-0.5.rc1.fc28.x86_64) ^^^^^^^^^^^^^^ This looks suspicious. Where do you get it from? Perhaps from the old virt-preview repo? Please note there is a copr virt-preview repo now, see https://fedoraproject.org/wiki/Virtualization_Preview_Repository and https://copr.fedorainfracloud.org/coprs/g/virtmaint-sig/virt-preview/, which should be up-to-date and contain QEMU 3.1.
I'm using the virt-preview repos enabled by this: http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
If the repo was changed, updates to this rpm should have modified my system to include the new repo.
I see. Sandro and I have already updated ovirt-release last week and I can see the repo in
https://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc28/noarch/ovirt-... . Do you have the latest ovirt-release-master.rpm?
I had when I sent the mail yesterday, but now we have new version from Nov 25 including the correct repo.
Sandro, do we need to change ovirt-release-master.rpm?
but I hope this change will fix your setup.
Which change?
Looks like you found it, but anyway: https://gerrit.ovirt.org/c/95718/
Sorry, I responded before reading gerrit notifications.
Michal, would you rebase your patch on the fix above to test it?
On 26 Nov 2018, at 13:24, Milan Zamazal <mzamazal@redhat.com> wrote:
Nir Soffer <nsoffer@redhat.com> writes:
On Mon, Nov 26, 2018 at 12:27 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Nir Soffer <nsoffer@redhat.com> writes:
On Thu, Nov 22, 2018 at 2:08 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Nir Soffer <nsoffer@redhat.com> writes:
On Wed, Nov 21, 2018, 17:46 Milan Zamazal <mzamazal@redhat.com wrote:
> Hi, test_no_match fails on CI most of the time (but not always) in > https://gerrit.ovirt.org/95518: > > _____________________________ test_no_match[qcow2] > _____________________________ > 11:30:16 > 11:30:16 tmpdir = local('/var/tmp/vdsm/test_no_match_qcow2_0'), > image_format = 'qcow2' > 11:30:16 > 11:30:16 def test_no_match(tmpdir, image_format): > 11:30:16 path = str(tmpdir.join('test.' + image_format)) > 11:30:16 op = qemuimg.create(path, '1m', image_format) > 11:30:16 op.run() > 11:30:16 qemuio.write_pattern(path, image_format, pattern=2) > 11:30:16 with pytest.raises(qemuio.VerificationError): > 11:30:16 > qemuio.verify_pattern(path, image_format, pattern=4) > 11:30:16 > 11:30:16 storage/qemuio_test.py:59: > 11:30:16 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ _ _ > 11:30:16 > 11:30:16 path = '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2', format > = 'qcow2' > 11:30:16 offset = 512, len = 1024, pattern = 4 > 11:30:16 > 11:30:16 def verify_pattern(path, format, offset=512, len=1024, > pattern=5): > 11:30:16 read_cmd = 'read -P %d -s 0 -l %d %d %d' % (pattern, > len, offset, len) > 11:30:16 cmd = ['qemu-io', '-f', format, '-c', read_cmd, path] > 11:30:16 rc, out, err = commands.execCmd(cmd, raw=True) > 11:30:16 if rc != 0 or err != b"": > 11:30:16 > raise cmdutils.Error(cmd, rc, out, err) > 11:30:16 E Error: Command ['qemu-io', '-f', 'qcow2', '-c', > 'read -P 4 -s 0 -l 1024 512 1024', > '/var/tmp/vdsm/test_no_match_qcow2_0/test.qcow2'] failed with rc=1 > out='Pattern verification failed at offset 512, 1024 bytes\nread 1024/1024 > bytes at offset 512\n1 KiB, 1 ops; 0.0002 sec (3.756 MiB/sec and 3846.1538 > ops/sec)\n' err='' > 11:30:16 > 11:30:16 storage/qemuio.py:50: Error > > (Similarly for raw.) > > You can see the complete test run log here (or in other CI runs of the > patch): > >
> > It fails on both Fedora and CentOS. It may or may not be related to
https://jenkins.ovirt.org/job/vdsm_master_check-patch-fc28-x86_64/2040/conso... the
> fact that QEMU 2.11 is used in the failed runs. > > Any idea what could be wrong?
Yes. Qemu-io was fixed lately to fail when pattern does not match, but our wrapper still expects the old behaviour (return 0, log warning).
I see, thank you for explanation.
Are you sure you run 2.11 and not 2.12?
Actually not. Looking into the CI log once more, I can see it reports initially installed QEMU version before additional repos are added. There are no reports on QEMU versions or upgrades afterwards but that may be just silence of some automation script. Since new QEMU version would be expected with the added repos and it would explain the test failure, let's assume it's indeed a newer QEMU.
We will fix this soon.
OK, thank you. We can disable the test temporarily in our patches updating repos & requirements and re-enable it before merge or later, depending on availability of your fix.
I cannot reproduce the error on Fedora 28 (qemu-img-2.12.0-0.5.rc1.fc28.x86_64) ^^^^^^^^^^^^^^ This looks suspicious. Where do you get it from? Perhaps from the old virt-preview repo? Please note there is a copr virt-preview repo now, see https://fedoraproject.org/wiki/Virtualization_Preview_Repository and https://copr.fedorainfracloud.org/coprs/g/virtmaint-sig/virt-preview/, which should be up-to-date and contain QEMU 3.1.
I'm using the virt-preview repos enabled by this: http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
If the repo was changed, updates to this rpm should have modified my system to include the new repo.
I see. Sandro and I have already updated ovirt-release last week and I can see the repo in https://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/fc28/noarch/ovirt-.... Do you have the latest ovirt-release-master.rpm?
Sandro, do we need to change ovirt-release-master.rpm?
but I hope this change will fix your setup.
Which change?
Looks like you found it, but anyway: https://gerrit.ovirt.org/c/95718/
Sorry, I responded before reading gerrit notifications.
Michal, would you rebase your patch on the fix above to test it?
yeah, works fine!
participants (3)
-
Michal Skrivanek -
Milan Zamazal -
Nir Soffer