On 17/11/16 01:11 +0200, Nir Soffer wrote:
I got very consistent failures with this patch, which should not be related in any way to the failing code: https://gerrit.ovirt.org/66109
After adding logs in qemuinfo.Operation, the problem disappear: https://gerrit.ovirt.org/66943
This confirms my suspicion of a timing issue. I think it's inherently racy to rely on progress == 100.0 when job.status == 'DONE' because QemuImgOperation can read partial progress output. When this happens, the output is discarded and an update is attempted during the next read. tldr; I think the solution is to drop the asserts about progress for all tests other than progress-specific tests since progress is advisory anyhow.
On Wed, Nov 16, 2016 at 11:33 PM, Nir Soffer <nsoffer@redhat.com> wrote:
This test pass usually, Adam, can you check this?
21:15:52 ====================================================================== 21:15:52 FAIL: test_intra_domain_copy('block', 'cow', 'cow') (storage_sdm_copy_data_test.TestCopyDataDIV) 21:15:52 ---------------------------------------------------------------------- 21:15:52 Traceback (most recent call last): 21:15:52 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/testlib.py", line 133, in wrapper 21:15:52 return f(self, *args) 21:15:52 File "/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/storage_sdm_copy_data_test.py", line 138, in test_intra_domain_copy 21:15:52 self.assertEqual(100.0, job.progress) 21:15:52 AssertionError: 100.0 != 0.0 21:15:52 -------------------- >> begin captured logging << -------------------- 21:15:52 2016-11-16 21:14:26,096 DEBUG (MainThread) [storage.PersistentDict] Created a persistent dict with VGTagMetadataRW backend (persistent:194) 21:15:52 2016-11-16 21:14:26,096 DEBUG (MainThread) [storage.PersistentDict] read lines (VGTagMetadataRW)=[] (persistent:236) 21:15:52 2016-11-16 21:14:26,097 DEBUG (MainThread) [storage.PersistentDict] Empty metadata (persistent:254) 21:15:52 2016-11-16 21:14:26,097 DEBUG (MainThread) [storage.PersistentDict] Starting transaction (persistent:169) 21:15:52 2016-11-16 21:14:26,097 DEBUG (MainThread) [storage.PersistentDict] Flushing changes (persistent:175) 21:15:52 2016-11-16 21:14:26,097 DEBUG (MainThread) [storage.PersistentDict] about to write lines (VGTagMetadataRW)=['CLASS=Data', 'POOL_UUID=fb6edad8-51ab-4aad-9090-a833efd6a1e2', 'SDUUID=beb999a9-a90b-4c49-9d4c-7b21b3adb164', 'VERSION=3', '_SHA_CKSUM=63ec9e7bc3d23a7da8ef5e318fdd65203ac7c3fe'] (persistent:307) 21:15:52 2016-11-16 21:14:26,098 DEBUG (MainThread) [storage.Metadata.VGTagMetadataRW] Updating metadata adding=MDT__SHA_CKSUM=63ec9e7bc3d23a7da8ef5e318fdd65203ac7c3fe, MDT_VERSION=3, MDT_POOL_UUID=fb6edad8-51ab-4aad-9090-a833efd6a1e2, MDT_CLASS=Data, MDT_SDUUID=beb999a9-a90b-4c49-9d4c-7b21b3adb164 removing= (blockSD:296) 21:15:52 2016-11-16 21:14:26,098 DEBUG (MainThread) [storage.PersistentDict] Finished transaction (persistent:177) 21:15:52 2016-11-16 21:14:26,101 WARNING (MainThread) [storage.StorageDomainManifest] Could not find mapping for lv beb999a9-a90b-4c49-9d4c-7b21b3adb164/1b8476de-236b-4c8e-aea0-134b213b7cb7 (blockSD:786) 21:15:52 2016-11-16 21:14:26,101 DEBUG (MainThread) [storage.StorageDomainManifest] Found freeSlot 4 in VG beb999a9-a90b-4c49-9d4c-7b21b3adb164 (blockSD:759) 21:15:52 2016-11-16 21:14:26,103 DEBUG (MainThread) [storage.VolumeManifest] Creating symlink from /var/tmp/tmpKpf5ys/dev/beb999a9-a90b-4c49-9d4c-7b21b3adb164/1b8476de-236b-4c8e-aea0-134b213b7cb7 to /var/tmp/tmpKpf5ys/mnt/blockSD/beb999a9-a90b-4c49-9d4c-7b21b3adb164/images/c406fa29-72c8-4a26-9601-2bd5e0b1cbd0/1b8476de-236b-4c8e-aea0-134b213b7cb7 (blockVolume:147) 21:15:52 2016-11-16 21:14:26,103 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 /usr/bin/qemu-img create -f qcow2 -o compat=0.10 /var/tmp/tmpKpf5ys/mnt/blockSD/beb999a9-a90b-4c49-9d4c-7b21b3adb164/images/c406fa29-72c8-4a26-9601-2bd5e0b1cbd0/1b8476de-236b-4c8e-aea0-134b213b7cb7 1048576 (cwd None) (commands:69) 21:15:52 2016-11-16 21:14:26,144 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) 21:15:52 2016-11-16 21:14:26,148 WARNING (MainThread) [storage.StorageDomainManifest] Could not find mapping for lv beb999a9-a90b-4c49-9d4c-7b21b3adb164/0e7c49ae-6c7c-4526-89c2-17b923c70dfa (blockSD:786) 21:15:52 2016-11-16 21:14:26,148 DEBUG (MainThread) [storage.StorageDomainManifest] Found freeSlot 5 in VG beb999a9-a90b-4c49-9d4c-7b21b3adb164 (blockSD:759) 21:15:52 2016-11-16 21:14:26,149 DEBUG (MainThread) [storage.VolumeManifest] Creating symlink from /var/tmp/tmpKpf5ys/dev/beb999a9-a90b-4c49-9d4c-7b21b3adb164/0e7c49ae-6c7c-4526-89c2-17b923c70dfa to /var/tmp/tmpKpf5ys/mnt/blockSD/beb999a9-a90b-4c49-9d4c-7b21b3adb164/images/939d015f-e57e-4bf1-9013-b648fae347ae/0e7c49ae-6c7c-4526-89c2-17b923c70dfa (blockVolume:147) 21:15:52 2016-11-16 21:14:26,150 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 /usr/bin/qemu-img create -f qcow2 -o compat=0.10 /var/tmp/tmpKpf5ys/mnt/blockSD/beb999a9-a90b-4c49-9d4c-7b21b3adb164/images/939d015f-e57e-4bf1-9013-b648fae347ae/0e7c49ae-6c7c-4526-89c2-17b923c70dfa 1048576 (cwd None) (commands:69) 21:15:52 2016-11-16 21:14:26,195 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) 21:15:52 2016-11-16 21:14:26,196 DEBUG (MainThread) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-3 /usr/bin/dd iflag=direct skip=4 bs=512 if=/var/tmp/tmpKpf5ys/dev/beb999a9-a90b-4c49-9d4c-7b21b3adb164/metadata count=1 (cwd None) (commands:69) 21:15:52 2016-11-16 21:14:26,207 DEBUG (MainThread) [storage.Misc.excCmd] SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes copied, 0.000394548 s, 1.3 MB/s\n'; <rc> = 0 (commands:93) 21:15:52 2016-11-16 21:14:26,208 DEBUG (MainThread) [storage.Misc] err: ['1+0 records in', '1+0 records out', '512 bytes copied, 0.000394548 s, 1.3 MB/s'], size: 512 (misc:138) 21:15:52 2016-11-16 21:14:26,209 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 qemu-io -f qcow2 -c 'write -P 240 0 1024' /var/tmp/tmpKpf5ys/mnt/blockSD/beb999a9-a90b-4c49-9d4c-7b21b3adb164/images/c406fa29-72c8-4a26-9601-2bd5e0b1cbd0/1b8476de-236b-4c8e-aea0-134b213b7cb7 (cwd None) (commands:69) 21:15:52 2016-11-16 21:14:26,257 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) 21:15:52 2016-11-16 21:14:26,260 DEBUG (MainThread) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-3 /usr/bin/dd iflag=direct skip=5 bs=512 if=/var/tmp/tmpKpf5ys/dev/beb999a9-a90b-4c49-9d4c-7b21b3adb164/metadata count=1 (cwd None) (commands:69) 21:15:52 2016-11-16 21:14:26,274 DEBUG (MainThread) [storage.Misc.excCmd] SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes copied, 0.000359356 s, 1.4 MB/s\n'; <rc> = 0 (commands:93) 21:15:52 2016-11-16 21:14:26,274 DEBUG (MainThread) [storage.Misc] err: ['1+0 records in', '1+0 records out', '512 bytes copied, 0.000359356 s, 1.4 MB/s'], size: 512 (misc:138) 21:15:52 2016-11-16 21:14:26,275 DEBUG (MainThread) [root] /usr/bin/taskset --cpu-list 0-3 qemu-io -f qcow2 -c 'read -P 240 -s 0 -l 1024 0 1024' /var/tmp/tmpKpf5ys/mnt/blockSD/beb999a9-a90b-4c49-9d4c-7b21b3adb164/images/939d015f-e57e-4bf1-9013-b648fae347ae/0e7c49ae-6c7c-4526-89c2-17b923c70dfa (cwd None) (commands:69) 21:15:52 2016-11-16 21:14:26,316 DEBUG (MainThread) [root] SUCCESS: <err> = ''; <rc> = 0 (commands:93) 21:15:52 2016-11-16 21:14:26,318 INFO (MainThread) [root] Running job 'f8a60ab8-2ba9-473d-bf46-82080c283137'... (jobs:179) 21:15:52 2016-11-16 21:14:26,320 INFO (MainThread) [storage.VolumeManifest] Volume: preparing volume beb999a9-a90b-4c49-9d4c-7b21b3adb164/1b8476de-236b-4c8e-aea0-134b213b7cb7 (volume:462) 21:15:52 2016-11-16 21:14:26,321 DEBUG (MainThread) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-3 /usr/bin/dd iflag=direct skip=4 bs=512 if=/var/tmp/tmpKpf5ys/dev/beb999a9-a90b-4c49-9d4c-7b21b3adb164/metadata count=1 (cwd None) (commands:69) 21:15:52 2016-11-16 21:14:26,335 DEBUG (MainThread) [storage.Misc.excCmd] SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes copied, 0.000302993 s, 1.7 MB/s\n'; <rc> = 0 (commands:93) 21:15:52 2016-11-16 21:14:26,336 DEBUG (MainThread) [storage.Misc] err: ['1+0 records in', '1+0 records out', '512 bytes copied, 0.000302993 s, 1.7 MB/s'], size: 512 (misc:138) 21:15:52 2016-11-16 21:14:26,338 DEBUG (MainThread) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-3 /usr/bin/dd iflag=direct skip=4 bs=512 if=/var/tmp/tmpKpf5ys/dev/beb999a9-a90b-4c49-9d4c-7b21b3adb164/metadata count=1 (cwd None) (commands:69) 21:15:52 2016-11-16 21:14:26,351 DEBUG (MainThread) [storage.Misc.excCmd] SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes copied, 0.000368094 s, 1.4 MB/s\n'; <rc> = 0 (commands:93) 21:15:52 2016-11-16 21:14:26,352 DEBUG (MainThread) [storage.Misc] err: ['1+0 records in', '1+0 records out', '512 bytes copied, 0.000368094 s, 1.4 MB/s'], size: 512 (misc:138) 21:15:52 2016-11-16 21:14:26,354 INFO (MainThread) [storage.VolumeManifest] Volume: preparing volume beb999a9-a90b-4c49-9d4c-7b21b3adb164/0e7c49ae-6c7c-4526-89c2-17b923c70dfa (volume:462) 21:15:52 2016-11-16 21:14:26,355 DEBUG (MainThread) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-3 /usr/bin/dd iflag=direct skip=5 bs=512 if=/var/tmp/tmpKpf5ys/dev/beb999a9-a90b-4c49-9d4c-7b21b3adb164/metadata count=1 (cwd None) (commands:69) 21:15:52 2016-11-16 21:14:26,368 DEBUG (MainThread) [storage.Misc.excCmd] SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes copied, 0.000382558 s, 1.3 MB/s\n'; <rc> = 0 (commands:93) 21:15:52 2016-11-16 21:14:26,369 DEBUG (MainThread) [storage.Misc] err: ['1+0 records in', '1+0 records out', '512 bytes copied, 0.000382558 s, 1.3 MB/s'], size: 512 (misc:138) 21:15:52 2016-11-16 21:14:26,370 DEBUG (MainThread) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-3 /usr/bin/dd iflag=direct skip=5 bs=512 if=/var/tmp/tmpKpf5ys/dev/beb999a9-a90b-4c49-9d4c-7b21b3adb164/metadata count=1 (cwd None) (commands:69) 21:15:52 2016-11-16 21:14:26,388 DEBUG (MainThread) [storage.Misc.excCmd] SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes copied, 0.000326875 s, 1.6 MB/s\n'; <rc> = 0 (commands:93) 21:15:52 2016-11-16 21:14:26,389 DEBUG (MainThread) [storage.Misc] err: ['1+0 records in', '1+0 records out', '512 bytes copied, 0.000326875 s, 1.6 MB/s'], size: 512 (misc:138) 21:15:52 2016-11-16 21:14:26,391 DEBUG (MainThread) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-3 /usr/bin/dd iflag=direct skip=5 bs=512 if=/var/tmp/tmpKpf5ys/dev/beb999a9-a90b-4c49-9d4c-7b21b3adb164/metadata count=1 (cwd None) (commands:69) 21:15:52 2016-11-16 21:14:26,404 DEBUG (MainThread) [storage.Misc.excCmd] SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes copied, 0.000335536 s, 1.5 MB/s\n'; <rc> = 0 (commands:93) 21:15:52 2016-11-16 21:14:26,405 DEBUG (MainThread) [storage.Misc] err: ['1+0 records in', '1+0 records out', '512 bytes copied, 0.000335536 s, 1.5 MB/s'], size: 512 (misc:138) 21:15:52 2016-11-16 21:14:26,406 DEBUG (MainThread) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-3 /usr/bin/dd iflag=direct skip=4 bs=512 if=/var/tmp/tmpKpf5ys/dev/beb999a9-a90b-4c49-9d4c-7b21b3adb164/metadata count=1 (cwd None) (commands:69) 21:15:52 2016-11-16 21:14:26,420 DEBUG (MainThread) [storage.Misc.excCmd] SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes copied, 0.000292733 s, 1.7 MB/s\n'; <rc> = 0 (commands:93) 21:15:52 2016-11-16 21:14:26,421 DEBUG (MainThread) [storage.Misc] err: ['1+0 records in', '1+0 records out', '512 bytes copied, 0.000292733 s, 1.7 MB/s'], size: 512 (misc:138) 21:15:52 2016-11-16 21:14:26,422 DEBUG (MainThread) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-3 /usr/bin/dd iflag=direct skip=4 bs=512 if=/var/tmp/tmpKpf5ys/dev/beb999a9-a90b-4c49-9d4c-7b21b3adb164/metadata count=1 (cwd None) (commands:69) 21:15:52 2016-11-16 21:14:26,436 DEBUG (MainThread) [storage.Misc.excCmd] SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes copied, 0.000359596 s, 1.4 MB/s\n'; <rc> = 0 (commands:93) 21:15:52 2016-11-16 21:14:26,437 DEBUG (MainThread) [storage.Misc] err: ['1+0 records in', '1+0 records out', '512 bytes copied, 0.000359596 s, 1.4 MB/s'], size: 512 (misc:138) 21:15:52 2016-11-16 21:14:26,438 DEBUG (MainThread) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-3 /usr/bin/dd iflag=direct skip=4 bs=512 if=/var/tmp/tmpKpf5ys/dev/beb999a9-a90b-4c49-9d4c-7b21b3adb164/metadata count=1 (cwd None) (commands:69) 21:15:52 2016-11-16 21:14:26,451 DEBUG (MainThread) [storage.Misc.excCmd] SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes copied, 0.000322085 s, 1.6 MB/s\n'; <rc> = 0 (commands:93) 21:15:52 2016-11-16 21:14:26,452 DEBUG (MainThread) [storage.Misc] err: ['1+0 records in', '1+0 records out', '512 bytes copied, 0.000322085 s, 1.6 MB/s'], size: 512 (misc:138) 21:15:52 2016-11-16 21:14:26,454 DEBUG (MainThread) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-3 /usr/bin/dd iflag=direct skip=5 bs=512 if=/var/tmp/tmpKpf5ys/dev/beb999a9-a90b-4c49-9d4c-7b21b3adb164/metadata count=1 (cwd None) (commands:69) 21:15:52 2016-11-16 21:14:26,467 DEBUG (MainThread) [storage.Misc.excCmd] SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes copied, 0.00033405 s, 1.5 MB/s\n'; <rc> = 0 (commands:93) 21:15:52 2016-11-16 21:14:26,468 DEBUG (MainThread) [storage.Misc] err: ['1+0 records in', '1+0 records out', '512 bytes copied, 0.00033405 s, 1.5 MB/s'], size: 512 (misc:138) 21:15:52 2016-11-16 21:14:26,469 DEBUG (MainThread) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-3 /usr/bin/dd iflag=direct skip=5 bs=512 if=/var/tmp/tmpKpf5ys/dev/beb999a9-a90b-4c49-9d4c-7b21b3adb164/metadata count=1 (cwd None) (commands:69) 21:16:08 2016-11-16 21:14:26,481 DEBUG (MainThread) [storage.Misc.excCmd] SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes copied, 0.00030909 s, 1.7 MB/s\n'; <rc> = 0 (commands:93) 21:16:08 2016-11-16 21:14:26,482 DEBUG (MainThread) [storage.Misc] err: ['1+0 records in', '1+0 records out', '512 bytes copied, 0.00030909 s, 1.7 MB/s'], size: 512 (misc:138) 21:16:08 2016-11-16 21:14:26,483 INFO (MainThread) [storage.VolumeManifest] sdUUID=beb999a9-a90b-4c49-9d4c-7b21b3adb164 imgUUID=939d015f-e57e-4bf1-9013-b648fae347ae volUUID = 0e7c49ae-6c7c-4526-89c2-17b923c70dfa legality = ILLEGAL (volume:368) 21:16:08 2016-11-16 21:14:26,484 DEBUG (MainThread) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-3 /usr/bin/dd iflag=direct skip=5 bs=512 if=/var/tmp/tmpKpf5ys/dev/beb999a9-a90b-4c49-9d4c-7b21b3adb164/metadata count=1 (cwd None) (commands:69) 21:16:08 2016-11-16 21:14:26,497 DEBUG (MainThread) [storage.Misc.excCmd] SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes copied, 0.000307797 s, 1.7 MB/s\n'; <rc> = 0 (commands:93) 21:16:08 2016-11-16 21:14:26,498 DEBUG (MainThread) [storage.Misc] err: ['1+0 records in', '1+0 records out', '512 bytes copied, 0.000307797 s, 1.7 MB/s'], size: 512 (misc:138) 21:16:08 2016-11-16 21:14:26,502 DEBUG (MainThread) [QemuImg] /usr/bin/taskset --cpu-list 0-3 /usr/bin/nice -n 19 /usr/bin/ionice -c 3 /usr/bin/qemu-img convert -p -t none -T none -f qcow2 /var/tmp/tmpKpf5ys/mnt/blockSD/beb999a9-a90b-4c49-9d4c-7b21b3adb164/images/c406fa29-72c8-4a26-9601-2bd5e0b1cbd0/1b8476de-236b-4c8e-aea0-134b213b7cb7 -O qcow2 -o compat=0.10 /var/tmp/tmpKpf5ys/mnt/blockSD/beb999a9-a90b-4c49-9d4c-7b21b3adb164/images/939d015f-e57e-4bf1-9013-b648fae347ae/0e7c49ae-6c7c-4526-89c2-17b923c70dfa (cwd None) (qemuimg:247) 21:16:08 2016-11-16 21:14:26,805 DEBUG (MainThread) [storage.Misc.excCmd] /usr/bin/taskset --cpu-list 0-3 /usr/bin/dd iflag=direct skip=5 bs=512 if=/var/tmp/tmpKpf5ys/dev/beb999a9-a90b-4c49-9d4c-7b21b3adb164/metadata count=1 (cwd None) (commands:69) 21:16:08 2016-11-16 21:14:26,820 DEBUG (MainThread) [storage.Misc.excCmd] SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes copied, 0.000380116 s, 1.3 MB/s\n'; <rc> = 0 (commands:93) 21:16:08 2016-11-16 21:14:26,821 DEBUG (MainThread) [storage.Misc] err: ['1+0 records in', '1+0 records out', '512 bytes copied, 0.000380116 s, 1.3 MB/s'], size: 512 (misc:138) 21:16:08 2016-11-16 21:14:26,823 INFO (MainThread) [storage.VolumeManifest] Tearing down volume beb999a9-a90b-4c49-9d4c-7b21b3adb164/0e7c49ae-6c7c-4526-89c2-17b923c70dfa justme True (blockVolume:386) 21:16:08 2016-11-16 21:14:26,824 INFO (MainThread) [storage.VolumeManifest] Tearing down volume beb999a9-a90b-4c49-9d4c-7b21b3adb164/1b8476de-236b-4c8e-aea0-134b213b7cb7 justme True (blockVolume:386) 21:16:08 2016-11-16 21:14:26,824 INFO (MainThread) [root] Job 'f8a60ab8-2ba9-473d-bf46-82080c283137' completed (jobs:203) 21:16:08 2016-11-16 21:14:26,825 INFO (MainThread) [root] Job 'f8a60ab8-2ba9-473d-bf46-82080c283137' will be deleted in 3600 seconds (jobs:245) 21:16:08 --------------------- >> end captured logging << ---------------------
-- Adam Litke