Tasks stuck waiting on another after failed storage migration (yet not visible on SPM)
Hello, I'm running oVirt version 4.3.9.4-1.el7. After a failed live storage migration a VM got stuck with snapshot. Checking the engine logs I can see that the snapshot removal task is waiting for Merge to complete and vice versa. 2020-05-26 18:34:04,826+02 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [90f428b0-9c4e-4ac0-8de6-1103fc13da9e] Command 'RemoveSnapshotSingleDiskLive' (id: '60ce36c1-bf74-40a9-9fb0-7fcf7eb95f40') waiting on child command id: 'f7d1de7b-9e87-47ba-9ba0-ee04301ba3b1' type:'Merge' to complete 2020-05-26 18:34:04,827+02 INFO [org.ovirt.engine.core.bll.MergeCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [90f428b0-9c4e-4ac0-8de6-1103fc13da9e] Waiting on merge command to complete (jobId = f694590a-1577-4dce-bf0c-3a8d74adf341) 2020-05-26 18:34:04,845+02 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [90f428b0-9c4e-4ac0-8de6-1103fc13da9e] Command 'RemoveSnapshot' (id: '47c9a847-5b4b-4256-9264-a760acde8275') waiting on child command id: '60ce36c1-bf74-40a9-9fb0-7fcf7eb95f40' type:'RemoveSnapshotSingleDiskLive' to complete 2020-05-26 18:34:14,277+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmJobsMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-96) [] VM Job [f694590a-1577-4dce-bf0c-3a8d74adf341]: In progress (no change) I cannot see any runnig tasks on the SPM (vdsm-client Host getAllTasksInfo). I also cannot find the task ID in any of the other node's logs. I already tried restarting the Engine (didn't help). To start I'm puzzled as to where this task is queueing? Any Ideas on how I could resolve this? Thank you. Regards, David
Live merge (snapshot removal) is running on the host where the VM is running, you can look for the job id (f694590a-1577-4dce-bf0c-3a8d74adf341) on the relevant host On Wed, May 27, 2020 at 9:02 AM David Sekne <david.sekne@gmail.com> wrote:
Hello,
I'm running oVirt version 4.3.9.4-1.el7.
After a failed live storage migration a VM got stuck with snapshot. Checking the engine logs I can see that the snapshot removal task is waiting for Merge to complete and vice versa.
2020-05-26 18:34:04,826+02 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [90f428b0-9c4e-4ac0-8de6-1103fc13da9e] Command 'RemoveSnapshotSingleDiskLive' (id: '60ce36c1-bf74-40a9-9fb0-7fcf7eb95f40') waiting on child command id: 'f7d1de7b-9e87-47ba-9ba0-ee04301ba3b1' type:'Merge' to complete 2020-05-26 18:34:04,827+02 INFO [org.ovirt.engine.core.bll.MergeCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [90f428b0-9c4e-4ac0-8de6-1103fc13da9e] Waiting on merge command to complete (jobId = f694590a-1577-4dce-bf0c-3a8d74adf341) 2020-05-26 18:34:04,845+02 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [90f428b0-9c4e-4ac0-8de6-1103fc13da9e] Command 'RemoveSnapshot' (id: '47c9a847-5b4b-4256-9264-a760acde8275') waiting on child command id: '60ce36c1-bf74-40a9-9fb0-7fcf7eb95f40' type:'RemoveSnapshotSingleDiskLive' to complete 2020-05-26 18:34:14,277+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmJobsMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-96) [] VM Job [f694590a-1577-4dce-bf0c-3a8d74adf341]: In progress (no change)
I cannot see any runnig tasks on the SPM (vdsm-client Host getAllTasksInfo). I also cannot find the task ID in any of the other node's logs.
I already tried restarting the Engine (didn't help).
To start I'm puzzled as to where this task is queueing?
Any Ideas on how I could resolve this?
Thank you. Regards, David
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/VJBI3SMVXTPSGG...
Hello, Thank you for the reply. Unfortunately I cant see the task on any on the hosts: vdsm-client Task getInfo taskID=f694590a-1577-4dce-bf0c-3a8d74adf341 vdsm-client: Command Task.getInfo with args {'taskID': 'f694590a-1577-4dce-bf0c-3a8d74adf341'} failed: (code=401, message=Task id unknown: (u'f694590a-1577-4dce-bf0c-3a8d74adf341',)) I can see it starting in VDSM log on the host runnig the VM: /var/log/vdsm/vdsm.log.2:2020-05-26 12:15:09,349+0200 INFO (jsonrpc/6) [virt.vm] (vmId='e113ff18-5687-4e03-8a27-b12c82ad6d6b') Starting merge with jobUUID=u'f694590a-1577-4dce-bf0c-3a8d74adf341', original chain=a78c7505-a949-43f3-b3d0-9d17bdb41af5 < aabf3788-8e47-4f8b-84ad-a7eb311659fa (top), disk='sda', base='sda[1]', top=None, bandwidth=0, flags=12 (vm:5945) Also running vdsm-client Host getAllTasks I don't see any runnig tasks (on any host). Am I missing something? Regards,
You can't see it because it is not a task, tasks only run on SPM, It is a VM job and the data about it is stored in the VM's XML, it's also stored in the vm_jobs table. You can see the status of the job in libvirt with `virsh blockjob <vm_name> sda --info` (if it's still running) On Wed, May 27, 2020 at 2:03 PM David Sekne <david.sekne@gmail.com> wrote:
Hello,
Thank you for the reply.
Unfortunately I cant see the task on any on the hosts:
vdsm-client Task getInfo taskID=f694590a-1577-4dce-bf0c-3a8d74adf341 vdsm-client: Command Task.getInfo with args {'taskID': 'f694590a-1577-4dce-bf0c-3a8d74adf341'} failed: (code=401, message=Task id unknown: (u'f694590a-1577-4dce-bf0c-3a8d74adf341',))
I can see it starting in VDSM log on the host runnig the VM:
/var/log/vdsm/vdsm.log.2:2020-05-26 12:15:09,349+0200 INFO (jsonrpc/6) [virt.vm] (vmId='e113ff18-5687-4e03-8a27-b12c82ad6d6b') Starting merge with jobUUID=u'f694590a-1577-4dce-bf0c-3a8d74adf341', original chain=a78c7505-a949-43f3-b3d0-9d17bdb41af5 < aabf3788-8e47-4f8b-84ad-a7eb311659fa (top), disk='sda', base='sda[1]', top=None, bandwidth=0, flags=12 (vm:5945)
Also running vdsm-client Host getAllTasks I don't see any runnig tasks (on any host).
Am I missing something?
Regards,
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/VBTD3HLXPK7F7M...
Hello, Running virsh blockjob<vm_name> sda --info a couple of times it shows 99 or 100 %. Looks like it is stuck / flapping for some reason. Active Block Commit: [ 99 %] Active Block Commit: [100 %] What would be the best approach to resolve this? I see taskcleaner.sh can be used in cases like these? Regards,
Can you share the VM's xml? Can be obtained with `virsh -r dumpxml <vm_name>` Is the VM overloaded? I suspect it has trouble converging taskcleaner only cleans up the database, I don't think it will help here
Hello, Yes, no problem. XML is attached (I ommited the hostname and IP). Server is quite big (8 CPU / 32 Gb RAM / 1 Tb disk) yet not overloaded. We have multiple servers with the same specs with no issues. Regards, On Wed, May 27, 2020 at 2:28 PM Benny Zlotnik <bzlotnik@redhat.com> wrote:
Can you share the VM's xml? Can be obtained with `virsh -r dumpxml <vm_name>` Is the VM overloaded? I suspect it has trouble converging
taskcleaner only cleans up the database, I don't think it will help here
Sorry, by overloaded I meant in terms of I/O, because this is an active layer merge, the active layer (aabf3788-8e47-4f8b-84ad-a7eb311659fa) is merged into the base image (a78c7505-a949-43f3-b3d0-9d17bdb41af5), before the VM switches to use it as the active layer. So if there is constantly additional data written to the current active layer, vdsm may have trouble finishing the synchronization On Wed, May 27, 2020 at 4:55 PM David Sekne <david.sekne@gmail.com> wrote:
Hello,
Yes, no problem. XML is attached (I ommited the hostname and IP).
Server is quite big (8 CPU / 32 Gb RAM / 1 Tb disk) yet not overloaded. We have multiple servers with the same specs with no issues.
Regards,
On Wed, May 27, 2020 at 2:28 PM Benny Zlotnik <bzlotnik@redhat.com> wrote:
Can you share the VM's xml? Can be obtained with `virsh -r dumpxml <vm_name>` Is the VM overloaded? I suspect it has trouble converging
taskcleaner only cleans up the database, I don't think it will help here
Hello, Not sure IO could be the case. The underlying storage itself is brand new (nvme) connected with FC and is barely at 10 % capacity with low IOPS and practically zero latency. There are no IO limitations on the LUN itself. I would also be able to see any IO problems on the other VMs (none in this case). I'm out of ideas on what to do for the time how to stop / complete the task. Any suggestion welcome. Shutting down the VM in this state means that It probably wont start back (snapshots with disks in illegal state). Worst case I plan to restore this VM from a backup tonight. Regards, On Wed, May 27, 2020 at 4:39 PM Benny Zlotnik <bzlotnik@redhat.com> wrote:
Sorry, by overloaded I meant in terms of I/O, because this is an active layer merge, the active layer (aabf3788-8e47-4f8b-84ad-a7eb311659fa) is merged into the base image (a78c7505-a949-43f3-b3d0-9d17bdb41af5), before the VM switches to use it as the active layer. So if there is constantly additional data written to the current active layer, vdsm may have trouble finishing the synchronization
On Wed, May 27, 2020 at 4:55 PM David Sekne <david.sekne@gmail.com> wrote:
Hello,
Yes, no problem. XML is attached (I ommited the hostname and IP).
Server is quite big (8 CPU / 32 Gb RAM / 1 Tb disk) yet not overloaded.
We have multiple servers with the same specs with no issues.
Regards,
On Wed, May 27, 2020 at 2:28 PM Benny Zlotnik <bzlotnik@redhat.com>
wrote:
Can you share the VM's xml? Can be obtained with `virsh -r dumpxml <vm_name>` Is the VM overloaded? I suspect it has trouble converging
taskcleaner only cleans up the database, I don't think it will help here
I used to have a similar issue and when I live migrated (from 1 host to another) it automatically completed. Best Regards, Strahil Nikolov На 27 май 2020 г. 17:39:36 GMT+03:00, Benny Zlotnik <bzlotnik@redhat.com> написа:
Sorry, by overloaded I meant in terms of I/O, because this is an active layer merge, the active layer (aabf3788-8e47-4f8b-84ad-a7eb311659fa) is merged into the base image (a78c7505-a949-43f3-b3d0-9d17bdb41af5), before the VM switches to use it as the active layer. So if there is constantly additional data written to the current active layer, vdsm may have trouble finishing the synchronization
On Wed, May 27, 2020 at 4:55 PM David Sekne <david.sekne@gmail.com> wrote:
Hello,
Yes, no problem. XML is attached (I ommited the hostname and IP).
Server is quite big (8 CPU / 32 Gb RAM / 1 Tb disk) yet not
overloaded. We have multiple servers with the same specs with no issues.
Regards,
On Wed, May 27, 2020 at 2:28 PM Benny Zlotnik <bzlotnik@redhat.com>
wrote:
Can you share the VM's xml? Can be obtained with `virsh -r dumpxml <vm_name>` Is the VM overloaded? I suspect it has trouble converging
taskcleaner only cleans up the database, I don't think it will help
here
Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/HX4QZDIKXH7ETW...
Hello, I tried the live migrate as well and it didn't help (it failed). The VM disks were in a illegal state so I ended up restoring the VM from backup (It was least complex solution for my case). Thank you both for the help. Regards, On Thu, May 28, 2020 at 5:01 PM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
I used to have a similar issue and when I live migrated (from 1 host to another) it automatically completed.
Best Regards, Strahil Nikolov
На 27 май 2020 г. 17:39:36 GMT+03:00, Benny Zlotnik <bzlotnik@redhat.com> написа:
Sorry, by overloaded I meant in terms of I/O, because this is an active layer merge, the active layer (aabf3788-8e47-4f8b-84ad-a7eb311659fa) is merged into the base image (a78c7505-a949-43f3-b3d0-9d17bdb41af5), before the VM switches to use it as the active layer. So if there is constantly additional data written to the current active layer, vdsm may have trouble finishing the synchronization
On Wed, May 27, 2020 at 4:55 PM David Sekne <david.sekne@gmail.com> wrote:
Hello,
Yes, no problem. XML is attached (I ommited the hostname and IP).
Server is quite big (8 CPU / 32 Gb RAM / 1 Tb disk) yet not
overloaded. We have multiple servers with the same specs with no issues.
Regards,
On Wed, May 27, 2020 at 2:28 PM Benny Zlotnik <bzlotnik@redhat.com>
wrote:
Can you share the VM's xml? Can be obtained with `virsh -r dumpxml <vm_name>` Is the VM overloaded? I suspect it has trouble converging
taskcleaner only cleans up the database, I don't think it will help
here
Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/HX4QZDIKXH7ETW...
Sorry for the late reply, but you may have hit this bug[1], I forgot about it. The bug happens when you live migrate a VM in post-copy mode, vdsm stops monitoring the VM's jobs. The root cause is an issue in libvirt, so it depends on which libvirt version you have [1] https://bugzilla.redhat.com/show_bug.cgi?id=1774230 On Fri, May 29, 2020 at 3:54 PM David Sekne <david.sekne@gmail.com> wrote:
Hello,
I tried the live migrate as well and it didn't help (it failed).
The VM disks were in a illegal state so I ended up restoring the VM from backup (It was least complex solution for my case).
Thank you both for the help.
Regards,
On Thu, May 28, 2020 at 5:01 PM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
I used to have a similar issue and when I live migrated (from 1 host to another) it automatically completed.
Best Regards, Strahil Nikolov
На 27 май 2020 г. 17:39:36 GMT+03:00, Benny Zlotnik <bzlotnik@redhat.com> написа:
Sorry, by overloaded I meant in terms of I/O, because this is an active layer merge, the active layer (aabf3788-8e47-4f8b-84ad-a7eb311659fa) is merged into the base image (a78c7505-a949-43f3-b3d0-9d17bdb41af5), before the VM switches to use it as the active layer. So if there is constantly additional data written to the current active layer, vdsm may have trouble finishing the synchronization
On Wed, May 27, 2020 at 4:55 PM David Sekne <david.sekne@gmail.com> wrote:
Hello,
Yes, no problem. XML is attached (I ommited the hostname and IP).
Server is quite big (8 CPU / 32 Gb RAM / 1 Tb disk) yet not
overloaded. We have multiple servers with the same specs with no issues.
Regards,
On Wed, May 27, 2020 at 2:28 PM Benny Zlotnik <bzlotnik@redhat.com>
wrote:
Can you share the VM's xml? Can be obtained with `virsh -r dumpxml <vm_name>` Is the VM overloaded? I suspect it has trouble converging
taskcleaner only cleans up the database, I don't think it will help
here
Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/HX4QZDIKXH7ETW...
Hello, It looks like this was the problem indeed. I have the migration policy set to post copy (thought this was relevant only to VM migration and not disk migration) and had libvirt-4.5.0-23.el7_7.6.x86_64 on the problematic hosts. Restarting the VDSM after the migration indeed resolved the issue. This issue only appeared during disk move for me. I have updated all of the hosts since (libvirt-4.5.0-33.el7_8.1.x86_64) and have not noticed the issue since. Thank you again. Regards, On Mon, Jun 1, 2020 at 6:53 PM Benny Zlotnik <bzlotnik@redhat.com> wrote:
Sorry for the late reply, but you may have hit this bug[1], I forgot about it. The bug happens when you live migrate a VM in post-copy mode, vdsm stops monitoring the VM's jobs. The root cause is an issue in libvirt, so it depends on which libvirt version you have
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1774230
On Fri, May 29, 2020 at 3:54 PM David Sekne <david.sekne@gmail.com> wrote:
Hello,
I tried the live migrate as well and it didn't help (it failed).
The VM disks were in a illegal state so I ended up restoring the VM from
backup (It was least complex solution for my case).
Thank you both for the help.
Regards,
On Thu, May 28, 2020 at 5:01 PM Strahil Nikolov <hunter86_bg@yahoo.com>
wrote:
I used to have a similar issue and when I live migrated (from 1 host
to another) it automatically completed.
Best Regards, Strahil Nikolov
На 27 май 2020 г. 17:39:36 GMT+03:00, Benny Zlotnik <
bzlotnik@redhat.com> написа:
Sorry, by overloaded I meant in terms of I/O, because this is an active layer merge, the active layer (aabf3788-8e47-4f8b-84ad-a7eb311659fa) is merged into the base image (a78c7505-a949-43f3-b3d0-9d17bdb41af5), before the VM switches to use it as the active layer. So if there is constantly additional data written to the current active layer, vdsm may have trouble finishing the synchronization
On Wed, May 27, 2020 at 4:55 PM David Sekne <david.sekne@gmail.com> wrote:
Hello,
Yes, no problem. XML is attached (I ommited the hostname and IP).
Server is quite big (8 CPU / 32 Gb RAM / 1 Tb disk) yet not
overloaded. We have multiple servers with the same specs with no issues.
Regards,
On Wed, May 27, 2020 at 2:28 PM Benny Zlotnik <bzlotnik@redhat.com>
wrote:
Can you share the VM's xml? Can be obtained with `virsh -r dumpxml <vm_name>` Is the VM overloaded? I suspect it has trouble converging
taskcleaner only cleans up the database, I don't think it will help
here
Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/HX4QZDIKXH7ETW...
participants (3)
-
Benny Zlotnik -
David Sekne -
Strahil Nikolov