oVirt 4.2.7 export VM seems stuck at end of sparse disk
Hello, it happened two times, always with VMs composed by more than 1 disk. Now I have a VM with 9 disks and a total of about 370Gb: 1 x 90Gb 5 x 50Gb 3 x 10Gb I export vm to an export domain and it started 2 hours and 10 minutes ago at 15:46. At beginning the write rate on export domain was 120MB/s (in line with I/O capabilities of storage subsystem). It seems 5 disks completed ok, while 4 don't complete, even if it seems to be some reading activity: with command iotop -d 3 -k -o -P I get this on hypervisor where qemu-img convert command is executing Total DISK READ : 5712.75 K/s | Total DISK WRITE : 746.58 K/s Actual DISK READ: 6537.11 K/s | Actual DISK WRITE: 6.89 K/s PID PRIO USER DISK READ> DISK WRITE SWAPIN IO COMMAND 21238 idle vdsm 1344.18 K/s 183.12 K/s 0.00 % 0.15 % qemu-img convert -p -t non~1d23-4829-8350-f81fa16ea8b0 21454 idle vdsm 1344.18 K/s 183.12 K/s 0.00 % 0.06 % qemu-img convert -p -t non~443d-4563-8879-3dbd711b6936 21455 idle vdsm 1344.18 K/s 189.68 K/s 0.00 % 0.08 % qemu-img convert -p -t non~b67d-4acd-9bb5-485bc3ffdf2e 21548 idle vdsm 1344.18 K/s 190.67 K/s 0.00 % 0.06 % qemu-img convert -p -t non~7fc On export share [root@xfer ~]# ll -td /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/* drwxr-xr-x 2 36 36 4096 Jan 21 17:21 /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/ac33a8fe-f10e-4cbb-b121-2956f9925ade drwxr-xr-x 2 36 36 4096 Jan 21 17:06 /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/20c07082-d5ed-4804-867d-1f3f7c202b0e drwxr-xr-x 2 36 36 4096 Jan 21 17:02 /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/f8b1c6bc-de13-4416-9fe6-d7c26fb082b1 drwxr-xr-x 2 36 36 4096 Jan 21 17:02 /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/1b5313ef-0ee5-4f9a-b2a6-baa153ff0984 drwxr-xr-x 2 36 36 4096 Jan 21 17:02 /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/3f21c566-bc00-4dab-acbb-db9a3a1d76fa drwxr-xr-x 2 36 36 4096 Jan 21 15:47 /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/ed3a721e-fd1b-4fea-ad41-7ca56e94890e drwxr-xr-x 2 36 36 4096 Jan 21 15:46 /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/213f9ce9-6c34-4911-820e-4d1a96ba1791 drwxr-xr-x 2 36 36 4096 Jan 21 15:46 /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/28fef57c-4254-4469-9f3f-0d1f114d4f78 drwxr-xr-x 2 36 36 4096 Jan 21 15:46 /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/ea4fa5b1-a93a-4cb9-9552-e8ec67c5ff75 [root@xfer ~]# du -sh /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/* 11G /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/1b5313ef-0ee5-4f9a-b2a6-baa153ff0984 51G /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/20c07082-d5ed-4804-867d-1f3f7c202b0e 51G /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/213f9ce9-6c34-4911-820e-4d1a96ba1791 51G /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/28fef57c-4254-4469-9f3f-0d1f114d4f78 11G /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/3f21c566-bc00-4dab-acbb-db9a3a1d76fa 85G /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/ac33a8fe-f10e-4cbb-b121-2956f9925ade 51G /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/ea4fa5b1-a93a-4cb9-9552-e8ec67c5ff75 51G /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/ed3a721e-fd1b-4fea-ad41-7ca56e94890e 11G /export/ovirt/a6a289ea-f160-4d35-b3aa-e59d171b4633/images/f8b1c6bc-de13-4416-9fe6-d7c26fb082b1 [root@xfer ~]# It seems more than 30 minutes no more progress.... I see no errors in vdsm.log of hypervisor and in engine.log I keep this message every 10 seconds: 2019-01-21 18:08:19,060+01 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-10) [a1c0e39f-1924-4ae8-8b48-30bcce669278] Command 'ExportVm' (id: 'dc0a58bd-e8ca-42a9-8de0-dd3ccabe0512') waiting on child command id: '29d40db8-a0b9-4408-9e62-1abf3bccca65' type:'CopyImageGroup' to complete All the disks except the "apparently" not completed one (the 90Gb disk) are preallocated. Any other hint on what to check? Thanks, Gianluca
On Mon, Jan 21, 2019 at 6:10 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
Hello, it happened two times, always with VMs composed by more than 1 disk. Now I have a VM with 9 disks and a total of about 370Gb: 1 x 90Gb 5 x 50Gb 3 x 10Gb
I export vm to an export domain and it started 2 hours and 10 minutes ago at 15:46. At beginning the write rate on export domain was 120MB/s (in line with I/O capabilities of storage subsystem). It seems 5 disks completed ok, while 4 don't complete, even if it seems to be some reading activity:
with command iotop -d 3 -k -o -P I get this on hypervisor where qemu-img convert command is executing
Total DISK READ : 5712.75 K/s | Total DISK WRITE : 746.58 K/s Actual DISK READ: 6537.11 K/s | Actual DISK WRITE: 6.89 K/s PID PRIO USER DISK READ> DISK WRITE SWAPIN IO COMMAND
21238 idle vdsm 1344.18 K/s 183.12 K/s 0.00 % 0.15 % qemu-img convert -p -t non~1d23-4829-8350-f81fa16ea8b0 21454 idle vdsm 1344.18 K/s 183.12 K/s 0.00 % 0.06 % qemu-img convert -p -t non~443d-4563-8879-3dbd711b6936 21455 idle vdsm 1344.18 K/s 189.68 K/s 0.00 % 0.08 % qemu-img convert -p -t non~b67d-4acd-9bb5-485bc3ffdf2e 21548 idle vdsm 1344.18 K/s 190.67 K/s 0.00 % 0.06 % qemu-img convert -p -t non~7fc
It seems it was only a matter of waiting. Completed after 4 hours and 47 minutes: Vm raclab1 was exported successfully to xfer 21/01/1920:33:22 Probably depending on I/O priority set by default as "idle" for the "qemu-img convert" command Any fair/supported way to manually change I/O priority, just in case? Gianluca
participants (1)
-
Gianluca Cecchi