8:58 p.m.
Hello,
I'm running oVirt version 4.3.9.4-1.el7.
After a failed live storage migration a VM got stuck with snapshot.
Checking the engine logs I can see that the snapshot removal task is
waiting for Merge to complete and vice versa.
2020-05-26 18:34:04,826+02 INFO
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
(EE-ManagedThreadFactory-engineScheduled-Thread-70)
[90f428b0-9c4e-4ac0-8de6-1103fc13da9e] Command
'RemoveSnapshotSingleDiskLive' (id:
'60ce36c1-bf74-40a9-9fb0-7fcf7eb95f40')
waiting on child command id: 'f7d1de7b-9e87-47ba-9ba0-ee04301ba3b1'
type:'Merge' to complete
2020-05-26 18:34:04,827+02 INFO
[org.ovirt.engine.core.bll.MergeCommandCallback]
(EE-ManagedThreadFactory-engineScheduled-Thread-70)
[90f428b0-9c4e-4ac0-8de6-1103fc13da9e] Waiting on merge command to complete
(jobId = f694590a-1577-4dce-bf0c-3a8d74adf341)
2020-05-26 18:34:04,845+02 INFO
[org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback]
(EE-ManagedThreadFactory-engineScheduled-Thread-70)
[90f428b0-9c4e-4ac0-8de6-1103fc13da9e] Command 'RemoveSnapshot' (id:
'47c9a847-5b4b-4256-9264-a760acde8275') waiting on child command id:
'60ce36c1-bf74-40a9-9fb0-7fcf7eb95f40'
type:'RemoveSnapshotSingleDiskLive'
to complete
2020-05-26 18:34:14,277+02 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.VmJobsMonitoring]
(EE-ManagedThreadFactory-engineScheduled-Thread-96) [] VM Job
[f694590a-1577-4dce-bf0c-3a8d74adf341]: In progress (no change)
I cannot see any runnig tasks on the SPM (vdsm-client Host
getAllTasksInfo). I also cannot find the task ID in any of the other node's
logs.
I already tried restarting the Engine (didn't help).
To start I'm puzzled as to where this task is queueing?
Any Ideas on how I could resolve this?
Thank you.
Regards,
David
10:52 a.m.
New subject: Tasks stuck waiting on another after failed storage migration (yet not visible on SPM)
Live merge (snapshot removal) is running on the host where the VM is
running, you can look for the job id
(f694590a-1577-4dce-bf0c-3a8d74adf341) on the relevant host
On Wed, May 27, 2020 at 9:02 AM David Sekne <david.sekne(a)gmail.com> wrote:
Hello,
I'm running oVirt version 4.3.9.4-1.el7.
After a failed live storage migration a VM got stuck with snapshot. Checking the engine
logs I can see that the snapshot removal task is waiting for Merge to complete and vice
versa.
2020-05-26 18:34:04,826+02 INFO
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback]
(EE-ManagedThreadFactory-engineScheduled-Thread-70) [90f428b0-9c4e-4ac0-8de6-1103fc13da9e]
Command 'RemoveSnapshotSingleDiskLive' (id:
'60ce36c1-bf74-40a9-9fb0-7fcf7eb95f40') waiting on child command id:
'f7d1de7b-9e87-47ba-9ba0-ee04301ba3b1' type:'Merge' to complete
2020-05-26 18:34:04,827+02 INFO [org.ovirt.engine.core.bll.MergeCommandCallback]
(EE-ManagedThreadFactory-engineScheduled-Thread-70) [90f428b0-9c4e-4ac0-8de6-1103fc13da9e]
Waiting on merge command to complete (jobId = f694590a-1577-4dce-bf0c-3a8d74adf341)
2020-05-26 18:34:04,845+02 INFO
[org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback]
(EE-ManagedThreadFactory-engineScheduled-Thread-70) [90f428b0-9c4e-4ac0-8de6-1103fc13da9e]
Command 'RemoveSnapshot' (id: '47c9a847-5b4b-4256-9264-a760acde8275')
waiting on child command id: '60ce36c1-bf74-40a9-9fb0-7fcf7eb95f40'
type:'RemoveSnapshotSingleDiskLive' to complete
2020-05-26 18:34:14,277+02 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.VmJobsMonitoring]
(EE-ManagedThreadFactory-engineScheduled-Thread-96) [] VM Job
[f694590a-1577-4dce-bf0c-3a8d74adf341]: In progress (no change)
I cannot see any runnig tasks on the SPM (vdsm-client Host getAllTasksInfo). I also
cannot find the task ID in any of the other node's logs.
I already tried restarting the Engine (didn't help).
To start I'm puzzled as to where this task is queueing?
Any Ideas on how I could resolve this?
Thank you.
Regards,
David
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/VJBI3SMVXTP...
2:01 p.m.
New subject: Tasks stuck waiting on another after failed storage migration (yet not visible on SPM)
Hello,
Thank you for the reply.
Unfortunately I cant see the task on any on the hosts:
vdsm-client Task getInfo taskID=f694590a-1577-4dce-bf0c-3a8d74adf341
vdsm-client: Command Task.getInfo with args {'taskID':
'f694590a-1577-4dce-bf0c-3a8d74adf341'} failed:
(code=401, message=Task id unknown: (u'f694590a-1577-4dce-bf0c-3a8d74adf341',))
I can see it starting in VDSM log on the host runnig the VM:
/var/log/vdsm/vdsm.log.2:2020-05-26 12:15:09,349+0200 INFO (jsonrpc/6) [virt.vm]
(vmId='e113ff18-5687-4e03-8a27-b12c82ad6d6b') Starting merge with
jobUUID=u'f694590a-1577-4dce-bf0c-3a8d74adf341', original
chain=a78c7505-a949-43f3-b3d0-9d17bdb41af5 < aabf3788-8e47-4f8b-84ad-a7eb311659fa
(top), disk='sda', base='sda[1]', top=None, bandwidth=0, flags=12
(vm:5945)
Also running vdsm-client Host getAllTasks I don't see any runnig tasks (on any host).
Am I missing something?
Regards,
2:23 p.m.
New subject: Tasks stuck waiting on another after failed storage migration (yet not visible on SPM)
You can't see it because it is not a task, tasks only run on SPM, It
is a VM job and the data about it is stored in the VM's XML, it's also
stored in the vm_jobs table.
You can see the status of the job in libvirt with `virsh blockjob
<vm_name> sda --info` (if it's still running)
On Wed, May 27, 2020 at 2:03 PM David Sekne <david.sekne(a)gmail.com> wrote:
Hello,
Thank you for the reply.
Unfortunately I cant see the task on any on the hosts:
vdsm-client Task getInfo taskID=f694590a-1577-4dce-bf0c-3a8d74adf341
vdsm-client: Command Task.getInfo with args {'taskID':
'f694590a-1577-4dce-bf0c-3a8d74adf341'} failed:
(code=401, message=Task id unknown: (u'f694590a-1577-4dce-bf0c-3a8d74adf341',))
I can see it starting in VDSM log on the host runnig the VM:
/var/log/vdsm/vdsm.log.2:2020-05-26 12:15:09,349+0200 INFO (jsonrpc/6) [virt.vm]
(vmId='e113ff18-5687-4e03-8a27-b12c82ad6d6b') Starting merge with
jobUUID=u'f694590a-1577-4dce-bf0c-3a8d74adf341', original
chain=a78c7505-a949-43f3-b3d0-9d17bdb41af5 < aabf3788-8e47-4f8b-84ad-a7eb311659fa
(top), disk='sda', base='sda[1]', top=None, bandwidth=0, flags=12
(vm:5945)
Also running vdsm-client Host getAllTasks I don't see any runnig tasks (on any
host).
Am I missing something?
Regards,
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/VBTD3HLXPK7...
2:56 p.m.
New subject: Tasks stuck waiting on another after failed storage migration (yet not visible on SPM)
Hello,
Running virsh blockjob<vm_name> sda --info a couple of times it shows 99 or 100 %.
Looks like it is stuck / flapping for some reason.
Active Block Commit: [ 99 %]
Active Block Commit: [100 %]
What would be the best approach to resolve this?
I see taskcleaner.sh can be used in cases like these?
Regards,
3:28 p.m.
New subject: Tasks stuck waiting on another after failed storage migration (yet not visible on SPM)
Can you share the VM's xml?
Can be obtained with `virsh -r dumpxml <vm_name>`
Is the VM overloaded? I suspect it has trouble converging
taskcleaner only cleans up the database, I don't think it will help here
4:55 p.m.
New subject: Tasks stuck waiting on another after failed storage migration (yet not visible on SPM)
Hello,
Yes, no problem. XML is attached (I ommited the hostname and IP).
Server is quite big (8 CPU / 32 Gb RAM / 1 Tb disk) yet not overloaded. We
have multiple servers with the same specs with no issues.
Regards,
On Wed, May 27, 2020 at 2:28 PM Benny Zlotnik <bzlotnik(a)redhat.com> wrote:
Can you share the VM's xml?
Can be obtained with `virsh -r dumpxml <vm_name>`
Is the VM overloaded? I suspect it has trouble converging
taskcleaner only cleans up the database, I don't think it will help here
5:39 p.m.
New subject: Tasks stuck waiting on another after failed storage migration (yet not visible on SPM)
Sorry, by overloaded I meant in terms of I/O, because this is an
active layer merge, the active layer
(aabf3788-8e47-4f8b-84ad-a7eb311659fa) is merged into the base image
(a78c7505-a949-43f3-b3d0-9d17bdb41af5), before the VM switches to use
it as the active layer. So if there is constantly additional data
written to the current active layer, vdsm may have trouble finishing
the synchronization
On Wed, May 27, 2020 at 4:55 PM David Sekne <david.sekne(a)gmail.com> wrote:
Hello,
Yes, no problem. XML is attached (I ommited the hostname and IP).
Server is quite big (8 CPU / 32 Gb RAM / 1 Tb disk) yet not overloaded. We have multiple
servers with the same specs with no issues.
Regards,
On Wed, May 27, 2020 at 2:28 PM Benny Zlotnik <bzlotnik(a)redhat.com> wrote:
>
> Can you share the VM's xml?
> Can be obtained with `virsh -r dumpxml <vm_name>`
> Is the VM overloaded? I suspect it has trouble converging
>
> taskcleaner only cleans up the database, I don't think it will help here
>
9:33 a.m.
New subject: Tasks stuck waiting on another after failed storage migration (yet not visible on SPM)
Hello,
Not sure IO could be the case. The underlying storage itself is brand new
(nvme) connected with FC and is barely at 10 % capacity with low IOPS and
practically zero latency. There are no IO limitations on the LUN itself. I
would also be able to see any IO problems on the other VMs (none in this
case).
I'm out of ideas on what to do for the time how to stop / complete the
task. Any suggestion welcome.
Shutting down the VM in this state means that It probably wont start back
(snapshots with disks in illegal state). Worst case I plan to restore this
VM from a backup tonight.
Regards,
On Wed, May 27, 2020 at 4:39 PM Benny Zlotnik <bzlotnik(a)redhat.com> wrote:
Sorry, by overloaded I meant in terms of I/O, because this is an
active layer merge, the active layer
(aabf3788-8e47-4f8b-84ad-a7eb311659fa) is merged into the base image
(a78c7505-a949-43f3-b3d0-9d17bdb41af5), before the VM switches to use
it as the active layer. So if there is constantly additional data
written to the current active layer, vdsm may have trouble finishing
the synchronization
On Wed, May 27, 2020 at 4:55 PM David Sekne <david.sekne(a)gmail.com> wrote:
>
> Hello,
>
> Yes, no problem. XML is attached (I ommited the hostname and IP).
>
> Server is quite big (8 CPU / 32 Gb RAM / 1 Tb disk) yet not overloaded.
We have multiple servers with the same specs with no issues.
>
> Regards,
>
> On Wed, May 27, 2020 at 2:28 PM Benny Zlotnik <bzlotnik(a)redhat.com>
wrote:
>>
>> Can you share the VM's xml?
>> Can be obtained with `virsh -r dumpxml <vm_name>`
>> Is the VM overloaded? I suspect it has trouble converging
>>
>> taskcleaner only cleans up the database, I don't think it will help here
>>
6:01 p.m.
New subject: Tasks stuck waiting on another after failed storage migration (yet not visible on SPM)
I used to have a similar issue and when I live migrated (from 1 host to another) it
automatically completed.
Best Regards,
Strahil Nikolov
На 27 май 2020 г. 17:39:36 GMT+03:00, Benny Zlotnik <bzlotnik(a)redhat.com> написа:
>Sorry, by overloaded I meant in terms of I/O, because this is an
>active layer merge, the active layer
>(aabf3788-8e47-4f8b-84ad-a7eb311659fa) is merged into the base image
>(a78c7505-a949-43f3-b3d0-9d17bdb41af5), before the VM switches to use
>it as the active layer. So if there is constantly additional data
>written to the current active layer, vdsm may have trouble finishing
>the synchronization
>
>
>On Wed, May 27, 2020 at 4:55 PM David Sekne <david.sekne(a)gmail.com>
>wrote:
>>
>> Hello,
>>
>> Yes, no problem. XML is attached (I ommited the hostname and IP).
>>
>> Server is quite big (8 CPU / 32 Gb RAM / 1 Tb disk) yet not
>overloaded. We have multiple servers with the same specs with no
>issues.
>>
>> Regards,
>>
>> On Wed, May 27, 2020 at 2:28 PM Benny Zlotnik <bzlotnik(a)redhat.com>
>wrote:
>>>
>>> Can you share the VM's xml?
>>> Can be obtained with `virsh -r dumpxml <vm_name>`
>>> Is the VM overloaded? I suspect it has trouble converging
>>>
>>> taskcleaner only cleans up the database, I don't think it will help
>here
>>>
>_______________________________________________
>Users mailing list -- users(a)ovirt.org
>To unsubscribe send an email to users-leave(a)ovirt.org
>Privacy Statement: https://www.ovirt.org/privacy-policy.html
>oVirt Code of Conduct:
>https://www.ovirt.org/community/about/community-guidelines/
>List Archives:
>https://lists.ovirt.org/archives/list/users@ovirt.org/message/HX4QZDIKXH7ETWPDNI3SKZ535WHBXE2V/
3:54 p.m.
New subject: Tasks stuck waiting on another after failed storage migration (yet not visible on SPM)
Hello,
I tried the live migrate as well and it didn't help (it failed).
The VM disks were in a illegal state so I ended up restoring the VM from
backup (It was least complex solution for my case).
Thank you both for the help.
Regards,
On Thu, May 28, 2020 at 5:01 PM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
I used to have a similar issue and when I live migrated (from 1
host to
another) it automatically completed.
Best Regards,
Strahil Nikolov
На 27 май 2020 г. 17:39:36 GMT+03:00, Benny Zlotnik <bzlotnik(a)redhat.com>
написа:
>Sorry, by overloaded I meant in terms of I/O, because this is an
>active layer merge, the active layer
>(aabf3788-8e47-4f8b-84ad-a7eb311659fa) is merged into the base image
>(a78c7505-a949-43f3-b3d0-9d17bdb41af5), before the VM switches to use
>it as the active layer. So if there is constantly additional data
>written to the current active layer, vdsm may have trouble finishing
>the synchronization
>
>
>On Wed, May 27, 2020 at 4:55 PM David Sekne <david.sekne(a)gmail.com>
>wrote:
>>
>> Hello,
>>
>> Yes, no problem. XML is attached (I ommited the hostname and IP).
>>
>> Server is quite big (8 CPU / 32 Gb RAM / 1 Tb disk) yet not
>overloaded. We have multiple servers with the same specs with no
>issues.
>>
>> Regards,
>>
>> On Wed, May 27, 2020 at 2:28 PM Benny Zlotnik <bzlotnik(a)redhat.com>
>wrote:
>>>
>>> Can you share the VM's xml?
>>> Can be obtained with `virsh -r dumpxml <vm_name>`
>>> Is the VM overloaded? I suspect it has trouble converging
>>>
>>> taskcleaner only cleans up the database, I don't think it will help
>here
>>>
>_______________________________________________
>Users mailing list -- users(a)ovirt.org
>To unsubscribe send an email to users-leave(a)ovirt.org
>Privacy Statement: https://www.ovirt.org/privacy-policy.html
>oVirt Code of Conduct:
>https://www.ovirt.org/community/about/community-guidelines/
>List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/HX4QZDIKXH7...
7:53 p.m.
New subject: Tasks stuck waiting on another after failed storage migration (yet not visible on SPM)
Sorry for the late reply, but you may have hit this bug[1], I forgot about it.
The bug happens when you live migrate a VM in post-copy mode, vdsm
stops monitoring the VM's jobs.
The root cause is an issue in libvirt, so it depends on which libvirt
version you have
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1774230
On Fri, May 29, 2020 at 3:54 PM David Sekne <david.sekne(a)gmail.com> wrote:
Hello,
I tried the live migrate as well and it didn't help (it failed).
The VM disks were in a illegal state so I ended up restoring the VM from backup (It was
least complex solution for my case).
Thank you both for the help.
Regards,
On Thu, May 28, 2020 at 5:01 PM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
>
> I used to have a similar issue and when I live migrated (from 1 host to another)
it automatically completed.
>
> Best Regards,
> Strahil Nikolov
>
> На 27 май 2020 г. 17:39:36 GMT+03:00, Benny Zlotnik <bzlotnik(a)redhat.com>
написа:
> >Sorry, by overloaded I meant in terms of I/O, because this is an
> >active layer merge, the active layer
> >(aabf3788-8e47-4f8b-84ad-a7eb311659fa) is merged into the base image
> >(a78c7505-a949-43f3-b3d0-9d17bdb41af5), before the VM switches to use
> >it as the active layer. So if there is constantly additional data
> >written to the current active layer, vdsm may have trouble finishing
> >the synchronization
> >
> >
> >On Wed, May 27, 2020 at 4:55 PM David Sekne <david.sekne(a)gmail.com>
> >wrote:
> >>
> >> Hello,
> >>
> >> Yes, no problem. XML is attached (I ommited the hostname and IP).
> >>
> >> Server is quite big (8 CPU / 32 Gb RAM / 1 Tb disk) yet not
> >overloaded. We have multiple servers with the same specs with no
> >issues.
> >>
> >> Regards,
> >>
> >> On Wed, May 27, 2020 at 2:28 PM Benny Zlotnik <bzlotnik(a)redhat.com>
> >wrote:
> >>>
> >>> Can you share the VM's xml?
> >>> Can be obtained with `virsh -r dumpxml <vm_name>`
> >>> Is the VM overloaded? I suspect it has trouble converging
> >>>
> >>> taskcleaner only cleans up the database, I don't think it will help
> >here
> >>>
> >_______________________________________________
> >Users mailing list -- users(a)ovirt.org
> >To unsubscribe send an email to users-leave(a)ovirt.org
> >Privacy Statement: https://www.ovirt.org/privacy-policy.html
> >oVirt Code of Conduct:
> >https://www.ovirt.org/community/about/community-guidelines/
> >List Archives:
>
>https://lists.ovirt.org/archives/list/users@ovirt.org/message/HX4QZDIKXH7ETWPDNI3SKZ535WHBXE2V/
1:12 p.m.
New subject: Tasks stuck waiting on another after failed storage migration (yet not visible on SPM)
Hello,
It looks like this was the problem indeed.
I have the migration policy set to post copy (thought this was relevant
only to VM migration and not disk migration) and had
libvirt-4.5.0-23.el7_7.6.x86_64 on the problematic hosts. Restarting the
VDSM after the migration indeed resolved the issue.
This issue only appeared during disk move for me.
I have updated all of the hosts since (libvirt-4.5.0-33.el7_8.1.x86_64) and
have not noticed the issue since.
Thank you again.
Regards,
On Mon, Jun 1, 2020 at 6:53 PM Benny Zlotnik <bzlotnik(a)redhat.com> wrote:
Sorry for the late reply, but you may have hit this bug[1], I forgot
about
it.
The bug happens when you live migrate a VM in post-copy mode, vdsm
stops monitoring the VM's jobs.
The root cause is an issue in libvirt, so it depends on which libvirt
version you have
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1774230
On Fri, May 29, 2020 at 3:54 PM David Sekne <david.sekne(a)gmail.com> wrote:
>
> Hello,
>
> I tried the live migrate as well and it didn't help (it failed).
>
> The VM disks were in a illegal state so I ended up restoring the VM from
backup (It was least complex solution for my case).
>
> Thank you both for the help.
>
> Regards,
>
> On Thu, May 28, 2020 at 5:01 PM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
>>
>> I used to have a similar issue and when I live migrated (from 1 host
to another) it automatically completed.
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> На 27 май 2020 г. 17:39:36 GMT+03:00, Benny Zlotnik <
bzlotnik(a)redhat.com> написа:
>> >Sorry, by overloaded I meant in terms of I/O, because this is an
>> >active layer merge, the active layer
>> >(aabf3788-8e47-4f8b-84ad-a7eb311659fa) is merged into the base image
>> >(a78c7505-a949-43f3-b3d0-9d17bdb41af5), before the VM switches to use
>> >it as the active layer. So if there is constantly additional data
>> >written to the current active layer, vdsm may have trouble finishing
>> >the synchronization
>> >
>> >
>> >On Wed, May 27, 2020 at 4:55 PM David Sekne <david.sekne(a)gmail.com>
>> >wrote:
>> >>
>> >> Hello,
>> >>
>> >> Yes, no problem. XML is attached (I ommited the hostname and IP).
>> >>
>> >> Server is quite big (8 CPU / 32 Gb RAM / 1 Tb disk) yet not
>> >overloaded. We have multiple servers with the same specs with no
>> >issues.
>> >>
>> >> Regards,
>> >>
>> >> On Wed, May 27, 2020 at 2:28 PM Benny Zlotnik
<bzlotnik(a)redhat.com>
>> >wrote:
>> >>>
>> >>> Can you share the VM's xml?
>> >>> Can be obtained with `virsh -r dumpxml <vm_name>`
>> >>> Is the VM overloaded? I suspect it has trouble converging
>> >>>
>> >>> taskcleaner only cleans up the database, I don't think it will
help
>> >here
>> >>>
>> >_______________________________________________
>> >Users mailing list -- users(a)ovirt.org
>> >To unsubscribe send an email to users-leave(a)ovirt.org
>> >Privacy Statement: https://www.ovirt.org/privacy-policy.html
>> >oVirt Code of Conduct:
>> >https://www.ovirt.org/community/about/community-guidelines/
>> >List Archives:
>> >
https://lists.ovirt.org/archives/list/users@ovirt.org/message/HX4QZDIKXH7...
1633
days inactive
1643
days old
12 comments
3 participants
participants (3)
-
Benny Zlotnik -
David Sekne -
Strahil Nikolov