10:49 p.m.
Sorry just hit reply....
I'm seeing these errors in the logs which look related to the problem:
2016-09-07 06:46:35,123 ERROR
[org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
(DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method
'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
exception 'null', the callback is marked for end method retries
2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.CommandsFactory]
(DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command
'LiveMigrateDisk': null
2016-09-07 06:46:45,185 ERROR
[org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
(DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method
'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
exception 'null', the callback is marked for end method retries
On 5 September 2016 at 06:46, Nir Soffer <nsoffer(a)redhat.com> wrote:
Hi Maton,
Please reply to the list, not to me directly.
Ala, can you look at this? is this a known issue?
Thanks,
Nir
On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett <matonb(a)ltresources.co.uk>
wrote:
> Log files as requested
>
> https://ufile.io/4fc35 vdsm log
> https://ufile.io/e9836 engine 03-Sep
> https://ufile.io/15f37 engine 04-Sep
>
> vdsm log stops on the 01-Sep...
>
> Couple of entries from the event log:
>
> Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage
> Migration' deletion for VM 'lv01' has been completed.
> Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage
> Migration' deletion for VM 'lv01' was initiated by SYSTEM
>
> And the related tasks
>
> Removing Snapshot Auto-generated for Live Storage Migration of VM lv01
> Sep 3, 2016 6:46:44 PM N/A 29f45ca9
> Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 6:46:44 PM
> Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 PM
>
> Finalizing Sep 3, 2016 7:31:06 PM N/A
>
>
>
> On 4 September 2016 at 14:27, Nir Soffer <nsoffer(a)redhat.com> wrote:
>>
>> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <matonb(a)ltresources.co.uk
>
>> wrote:
>>>
>>> How do I fix / kill a hung vdsm task?
>>>
>>> It seems to have completed the task but is stuck finalising.
>>>
>>> Removing Snapshot Auto-generated for Live Storage Migration
>>> Validating
>>> Executing
>>> (hour glass) Finalizing
>>>
>>> Task has been 'stuck' finalising for over 13 hours
>>
>>
>> Can you share engine and vdsm logs since the time the merge was started?
>>
>> Nir
>
>
10:13 p.m.
New subject: Hung task finalizing live migration
Any suggestions ?
THe task has been hung for 5 days now, I can't start the machine or destroy
it.
On 7 September 2016 at 06:49, Maton, Brett <matonb(a)ltresources.co.uk> wrote:
Sorry just hit reply....
I'm seeing these errors in the logs which look related to the problem:
2016-09-07 06:46:35,123 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
(DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method
'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
exception 'null', the callback is marked for end method retries
2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.CommandsFactory]
(DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command
'LiveMigrateDisk': null
2016-09-07 06:46:45,185 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
(DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method
'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
exception 'null', the callback is marked for end method retries
On 5 September 2016 at 06:46, Nir Soffer <nsoffer(a)redhat.com> wrote:
> Hi Maton,
>
> Please reply to the list, not to me directly.
>
> Ala, can you look at this? is this a known issue?
>
> Thanks,
> Nir
>
> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett <matonb(a)ltresources.co.uk>
> wrote:
> > Log files as requested
> >
> > https://ufile.io/4fc35 vdsm log
> > https://ufile.io/e9836 engine 03-Sep
> > https://ufile.io/15f37 engine 04-Sep
> >
> > vdsm log stops on the 01-Sep...
> >
> > Couple of entries from the event log:
> >
> > Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage
> > Migration' deletion for VM 'lv01' has been completed.
> > Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage
> > Migration' deletion for VM 'lv01' was initiated by SYSTEM
> >
> > And the related tasks
> >
> > Removing Snapshot Auto-generated for Live Storage Migration of VM lv01
> > Sep 3, 2016 6:46:44 PM N/A 29f45ca9
> > Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 6:46:44 PM
> > Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 PM
> >
> > Finalizing Sep 3, 2016 7:31:06 PM N/A
> >
> >
> >
> > On 4 September 2016 at 14:27, Nir Soffer <nsoffer(a)redhat.com> wrote:
> >>
> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <
> matonb(a)ltresources.co.uk>
> >> wrote:
> >>>
> >>> How do I fix / kill a hung vdsm task?
> >>>
> >>> It seems to have completed the task but is stuck finalising.
> >>>
> >>> Removing Snapshot Auto-generated for Live Storage Migration
> >>> Validating
> >>> Executing
> >>> (hour glass) Finalizing
> >>>
> >>> Task has been 'stuck' finalising for over 13 hours
> >>
> >>
> >> Can you share engine and vdsm logs since the time the merge was
> started?
> >>
> >> Nir
> >
> >
>
5:25 a.m.
New subject: Hung task finalizing live migration
Thanks for the pointer Mikhail, however I don't get any tasks listed with
that command:
vdsClient -s 0 getAllTasksStatuses
/usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlrpc.
since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli
from vdsm import utils, vdscli, constants
{'status': {'message': 'OK', 'code': 0},
'allTasksStatus': {}}
On 8 September 2016 at 09:51, Краснобаев Михаил <milo1(a)ya.ru> wrote:
Hi,
There is a way to cancel a running task - look here
http://lists.ovirt.org/pipermail/users/2014-November/028946.html
I was able to stop snapshot deletion this way.
Best, Mikhail.
08.09.2016, 08:14, "Maton, Brett" <matonb(a)ltresources.co.uk>:
Any suggestions ?
THe task has been hung for 5 days now, I can't start the machine or
destroy it.
On 7 September 2016 at 06:49, Maton, Brett <matonb(a)ltresources.co.uk>
wrote:
Sorry just hit reply....
I'm seeing these errors in the logs which look related to the problem:
2016-09-07 06:46:35,123 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
(DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method
'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
exception 'null', the callback is marked for end method retries
2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.CommandsFactory]
(DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command
'LiveMigrateDisk': null
2016-09-07 06:46:45,185 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
(DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method
'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
exception 'null', the callback is marked for end method retries
On 5 September 2016 at 06:46, Nir Soffer <nsoffer(a)redhat.com> wrote:
Hi Maton,
Please reply to the list, not to me directly.
Ala, can you look at this? is this a known issue?
Thanks,
Nir
On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett <matonb(a)ltresources.co.uk>
wrote:
> Log files as requested
>
> https://ufile.io/4fc35 vdsm log
> https://ufile.io/e9836 engine 03-Sep
> https://ufile.io/15f37 engine 04-Sep
>
> vdsm log stops on the 01-Sep...
>
> Couple of entries from the event log:
>
> Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage
> Migration' deletion for VM 'lv01' has been completed.
> Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage
> Migration' deletion for VM 'lv01' was initiated by SYSTEM
>
> And the related tasks
>
> Removing Snapshot Auto-generated for Live Storage Migration of VM lv01
> Sep 3, 2016 6:46:44 PM N/A 29f45ca9
> Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 6:46:44 PM
> Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 PM
>
> Finalizing Sep 3, 2016 7:31:06 PM N/A
>
>
>
> On 4 September 2016 at 14:27, Nir Soffer <nsoffer(a)redhat.com> wrote:
>>
>> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <matonb(a)ltresources.co.uk
>
>> wrote:
>>>
>>> How do I fix / kill a hung vdsm task?
>>>
>>> It seems to have completed the task but is stuck finalising.
>>>
>>> Removing Snapshot Auto-generated for Live Storage Migration
>>> Validating
>>> Executing
>>> (hour glass) Finalizing
>>>
>>> Task has been 'stuck' finalising for over 13 hours
>>
>>
>> Can you share engine and vdsm logs since the time the merge was started?
>>
>> Nir
>
>
,
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
С уважением, Краснобаев Михаил.
7:06 a.m.
New subject: Hung task finalizing live migration
Anyone know how to fix this broken task ?
It's persisted through a reboot of all hosts and the engine, something
needs deleting from the database to clear the task and release the locked
disk
On 8 September 2016 at 13:25, Maton, Brett <matonb(a)ltresources.co.uk> wrote:
Thanks for the pointer Mikhail, however I don't get any tasks
listed with
that command:
vdsClient -s 0 getAllTasksStatuses
/usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlrpc.
since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli
from vdsm import utils, vdscli, constants
{'status': {'message': 'OK', 'code': 0},
'allTasksStatus': {}}
On 8 September 2016 at 09:51, Краснобаев Михаил <milo1(a)ya.ru> wrote:
> Hi,
>
> There is a way to cancel a running task - look here
> http://lists.ovirt.org/pipermail/users/2014-November/028946.html
> I was able to stop snapshot deletion this way.
>
> Best, Mikhail.
>
> 08.09.2016, 08:14, "Maton, Brett" <matonb(a)ltresources.co.uk>:
>
> Any suggestions ?
>
> THe task has been hung for 5 days now, I can't start the machine or
> destroy it.
>
>
> On 7 September 2016 at 06:49, Maton, Brett <matonb(a)ltresources.co.uk>
> wrote:
>
> Sorry just hit reply....
>
> I'm seeing these errors in the logs which look related to the problem:
>
>
> 2016-09-07 06:46:35,123 ERROR
[org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
> (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method
> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
> exception 'null', the callback is marked for end method retries
> 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.CommandsFactory]
> (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command
> 'LiveMigrateDisk': null
> 2016-09-07 06:46:45,185 ERROR
[org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
> (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method
> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with
> exception 'null', the callback is marked for end method retries
>
> On 5 September 2016 at 06:46, Nir Soffer <nsoffer(a)redhat.com> wrote:
>
> Hi Maton,
>
> Please reply to the list, not to me directly.
>
> Ala, can you look at this? is this a known issue?
>
> Thanks,
> Nir
>
> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett <matonb(a)ltresources.co.uk>
> wrote:
> > Log files as requested
> >
> > https://ufile.io/4fc35 vdsm log
> > https://ufile.io/e9836 engine 03-Sep
> > https://ufile.io/15f37 engine 04-Sep
> >
> > vdsm log stops on the 01-Sep...
> >
> > Couple of entries from the event log:
> >
> > Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage
> > Migration' deletion for VM 'lv01' has been completed.
> > Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage
> > Migration' deletion for VM 'lv01' was initiated by SYSTEM
> >
> > And the related tasks
> >
> > Removing Snapshot Auto-generated for Live Storage Migration of VM lv01
> > Sep 3, 2016 6:46:44 PM N/A 29f45ca9
> > Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 6:46:44 PM
> > Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 PM
> >
> > Finalizing Sep 3, 2016 7:31:06 PM N/A
> >
> >
> >
> > On 4 September 2016 at 14:27, Nir Soffer <nsoffer(a)redhat.com> wrote:
> >>
> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <
> matonb(a)ltresources.co.uk>
> >> wrote:
> >>>
> >>> How do I fix / kill a hung vdsm task?
> >>>
> >>> It seems to have completed the task but is stuck finalising.
> >>>
> >>> Removing Snapshot Auto-generated for Live Storage Migration
> >>> Validating
> >>> Executing
> >>> (hour glass) Finalizing
> >>>
> >>> Task has been 'stuck' finalising for over 13 hours
> >>
> >>
> >> Can you share engine and vdsm logs since the time the merge was
> started?
> >>
> >> Nir
> >
> >
>
> ,
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
> --
> С уважением, Краснобаев Михаил.
>
>
>
7:39 a.m.
New subject: Hung task finalizing live migration
--Apple-Mail=_BE696610-4EC6-4158-8D67-989CBCD17D96
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
Hi Maton,
I have seen tasks in a weird state on my cluster also. I've had a vm get =
"stuck" during a migration where it says "migrating to" in the web
GUI, =
but it has finished migrating hours ago... If I click "Cancel Migraton" =
the gui tells me that it is not migrating, but I can't do any action on =
the vm because I am then told that the vm can't be acted upon while it =
is migrating. I also try to kill the task, but there are none listed
What has worked for me has been to put my hosted-engine in global =
maintenance mode, then ssh into the hosted engine and run the =
"engine-setup" command. I am not saying the is the best course of =
action, but when the engine comes back online the task is cleared.
Cheers,
Gervais
On Sep 10, 2016, at 11:06 AM, Maton, Brett
<matonb(a)ltresources.co.uk> =
wrote:
=20
Anyone know how to fix this broken task ?
=20
It's persisted through a reboot of all hosts and the engine, something =
needs
deleting from the database to clear the task and release the =
locked disk
=20
On 8 September 2016 at 13:25, Maton, Brett <matonb(a)ltresources.co.uk =
<mailto:matonb@ltresources.co.uk>> wrote:
Thanks for the pointer Mikhail, however I don't get any tasks
listed =
with that command:
=20
vdsClient -s 0 getAllTasksStatuses
=20
/usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses =
xmlrpc. since
ovirt 3.6 xmlrpc is deprecated, please use =
vdsm.jsonrpcvdscli
from vdsm import utils, vdscli, constants
=20
{'status': {'message': 'OK', 'code': 0},
'allTasksStatus': {}}
=20
=20
On 8 September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=
=D0=B0=D0=B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB <milo1(a)ya.ru =
<mailto:milo1@ya.ru>> wrote:
Hi,
=20
There is a way to cancel a running task - look here =
http://lists.ovirt.org/pipermail/users/2014-November/028946.html =
<http://lists.ovirt.org/pipermail/users/2014-November/028946.html>
I was able to stop snapshot deletion this way.
=20
Best, Mikhail.
=20
08.09.2016, 08:14, "Maton, Brett" <matonb(a)ltresources.co.uk =
<mailto:matonb@ltresources.co.uk>>:
> Any suggestions ?
>=20
> THe task has been hung for 5 days now, I can't start the machine or =
destroy it.
>=20
>=20
> On 7 September 2016 at 06:49, Maton, Brett <matonb(a)ltresources.co.uk =
<mailto:matonb@ltresources.co.uk>> wrote:
> Sorry just hit reply....
>=20
> I'm seeing these errors in the logs which look related to the =
problem:
>=20
>=20
> 2016-09-07 06:46:35,123 ERROR =
[org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] =
(DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method =
'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with =
exception 'null', the callback is marked for end method retries
> 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.Com =
<http://org.ovirt.engine.core.bll.com/>mandsFactory] =
(DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command =
'LiveMigrateDisk': null
> 2016-09-07 06:46:45,185 ERROR =
[org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] =
(DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method =
'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with =
exception 'null', the callback is marked for end method retries
>=20
> On 5 September 2016 at 06:46, Nir Soffer <nsoffer(a)redhat.com =
<mailto:nsoffer@redhat.com>> wrote:
> Hi Maton,
>=20
> Please reply to the list, not to me directly.
>=20
> Ala, can you look at this? is this a known issue?
>=20
> Thanks,
> Nir
>=20
> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett =
<matonb(a)ltresources.co.uk
<mailto:matonb@ltresources.co.uk>> wrote:
> > Log files as requested
> >
> > https://ufile.io/4fc35 <https://ufile.io/4fc35> vdsm log
> > https://ufile.io/e9836 <https://ufile.io/e9836> engine 03-Sep
> > https://ufile.io/15f37 <https://ufile.io/15f37> engine 04-Sep
> >
> > vdsm log stops on the 01-Sep...
> >
> > Couple of entries from the event log:
> >
> > Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage
> > Migration' deletion for VM 'lv01' has been completed.
> > Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage
> > Migration' deletion for VM 'lv01' was initiated by SYSTEM
> >
> > And the related tasks
> >
> > Removing Snapshot Auto-generated for Live Storage Migration of VM =
lv01
> > Sep 3, 2016 6:46:44 PM N/A 29f45ca9
> > Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 =
6:46:44 PM
> > Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016
7:31:06 =
PM
> >
> > Finalizing Sep 3, 2016 7:31:06 PM N/A
> >
> >
> >
> > On 4 September 2016 at 14:27, Nir Soffer <nsoffer(a)redhat.com =
<mailto:nsoffer@redhat.com>> wrote:
> >>
> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett =
<matonb(a)ltresources.co.uk <mailto:matonb@ltresources.co.uk>>
> >> wrote:
> >>>
> >>> How do I fix / kill a hung vdsm task?
> >>>
> >>> It seems to have completed the task but is stuck finalising.
> >>>
> >>> Removing Snapshot Auto-generated for Live Storage Migration
> >>> Validating
> >>> Executing
> >>> (hour glass) Finalizing
> >>>
> >>> Task has been 'stuck' finalising for over 13 hours
> >>
> >>
> >> Can you share engine and vdsm logs since the time the merge was =
started?
> >>
> >> Nir
> >
> >
> ,
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>=20
=20
--=20
=D0=A1 =D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=B5=D0=BC, =
=D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 =
=D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB.
=20
=20
=20
=20
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_BE696610-4EC6-4158-8D67-989CBCD17D96
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D"">Hi Maton,<div class=3D""><br
class=3D""></div><div class=3D"">I=
have seen tasks in a weird state on my cluster also. I've had a vm get =
"stuck" during a migration where it says "migrating to" in the web
GUI, =
but it has finished migrating hours ago... If I click "Cancel Migraton" =
the gui tells me that it is not migrating, but I can't do any action on =
the vm because I am then told that the vm can't be acted upon while it =
is migrating. I also try to kill the task, but there are none listed<br =
class=3D""><div class=3D""><br =
class=3D"webkit-block-placeholder"></div><div
class=3D"">What has worked =
for me has been to put my hosted-engine in global maintenance mode, then =
ssh into the hosted engine and run the "engine-setup" command. I am not =
saying the is the best course of action, but when the engine comes back =
online the task is cleared.</div><div class=3D"">
<div id=3D"signature" class=3D""><br
class=3D"">Cheers,<br =
class=3D"">Gervais<br class=3D""><br
class=3D""><br class=3D""></div>
</div>
<br class=3D""><div><blockquote type=3D"cite"
class=3D""><div =
class=3D"">On Sep 10, 2016, at 11:06 AM, Maton, Brett <<a =
href=3D"mailto:matonb@ltresources.co.uk" =
class=3D"">matonb(a)ltresources.co.uk</a>&gt;
wrote:</div><br =
class=3D"Apple-interchange-newline"><div class=3D""><div
dir=3D"ltr" =
class=3D""><div class=3D"">Anyone know how to fix this broken
task ?<br =
class=3D""><br class=3D""></div><div
class=3D"">It's persisted through a =
reboot of all hosts and the engine, something needs deleting from the =
database to clear the task and release the locked disk<br =
class=3D""></div></div><div
class=3D"gmail_extra"><br class=3D""><div =
class=3D"gmail_quote">On 8 September 2016 at 13:25, Maton, Brett <span =
dir=3D"ltr" class=3D""><<a
href=3D"mailto:matonb@ltresources.co.uk" =
target=3D"_blank"
class=3D"">matonb(a)ltresources.co.uk</a>&gt;</span> =
wrote:<br class=3D""><blockquote class=3D"gmail_quote"
style=3D"margin:0 =
0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div
dir=3D"ltr" =
class=3D"">Thanks for the pointer Mikhail, however I don't get any tasks
=
listed with that command:<br class=3D""><br
class=3D"">vdsClient -s 0 =
getAllTasksStatuses<br class=3D""><br =
class=3D"">/usr/share/vdsm/vdsClient.py:<wbr class=3D"">33:
=
DeprecationWarning: vdscli uses xmlrpc. since ovirt 3.6 xmlrpc is =
deprecated, please use vdsm.jsonrpcvdscli<br class=3D""> from
vdsm =
import utils, vdscli, constants<br class=3D""><br
class=3D"">{'status': =
{'message': 'OK', 'code': 0}, 'allTasksStatus': {}}<br
class=3D""><br =
class=3D""></div><div class=3D"HOEnZb"><div
class=3D"h5"><div =
class=3D"gmail_extra"><br class=3D""><div
class=3D"gmail_quote">On 8 =
September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=
=B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB <span dir=3D"ltr" =
class=3D""><<a href=3D"mailto:milo1@ya.ru"
target=3D"_blank" =
class=3D"">milo1(a)ya.ru</a>&gt;</span> wrote:<br
class=3D""><blockquote =
class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc =
solid;padding-left:1ex"><div class=3D"">Hi,</div><div =
class=3D""> </div><div class=3D"">There is a
way to cancel a =
running task - look here <a =
href=3D"http://lists.ovirt.org/pipermail/users/2014-November/028946.... =
target=3D"_blank" class=3D"">http://lists.ovirt.org/piperma<wbr
=
class=3D"">il/users/2014-November/028946.<wbr =
class=3D"">html</a></div><div class=3D"">I was
able to stop snapshot =
deletion this way.</div><div
class=3D""> </div><div class=3D"">Best, =
Mikhail.</div><div class=3D""> </div><div
class=3D"">08.09.2016, =
08:14, "Maton, Brett" <<a
href=3D"mailto:matonb@ltresources.co.uk" =
target=3D"_blank" =
class=3D"">matonb@ltresources.co.uk</a>>:</div><blockquote
=
type=3D"cite" class=3D""><div class=3D""><div
class=3D""><div =
class=3D""><div class=3D""><div
class=3D"">Any suggestions ?<br =
class=3D""><br class=3D""></div>THe task has been hung
for 5 days now, I =
can't start the machine or destroy it.<br class=3D""><br =
class=3D""></div></div><div class=3D""><br
class=3D""><div class=3D"">On =
7 September 2016 at 06:49, Maton, Brett <span class=3D""><<a =
href=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" =
class=3D"">matonb(a)ltresources.co.uk</a>&gt;</span>
wrote:<br =
class=3D""><blockquote style=3D"margin:0 0 0 0.8ex;border-left:1px
#ccc =
solid;padding-left:1ex" class=3D""><div
class=3D""><div class=3D""><div =
class=3D"">Sorry just hit reply....<br class=3D""><br
class=3D""></div>I'm=
seeing these errors in the logs which look related to the problem:<br =
class=3D""><br class=3D""><br
class=3D""><span class=3D"">2016-09-07 =
06</span>:46:35,123 ERROR [org.ovirt.engine.core.bll.tas<wbr =
class=3D"">ks.CommandCallbacksPoller] (DefaultQuartzScheduler6) =
[19c58c0d] Failed invoking callback end method 'onFailed' for command =
'<span class=3D"">07608003</span>-ca05-4e2e-b917-85ce5<wbr =
class=3D"">25c011b' with exception 'null', the callback is marked
for =
end method retries<br class=3D""><span
class=3D"">2016-09-07 =
06</span>:46:45,184 ERROR [<a =
href=3D"http://org.ovirt.engine.core.bll.com/" =
class=3D"">org.ovirt.engine.core.bll.Com</a><wbr
class=3D"">mandsFactory] =
(DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command =
'LiveMigrateDisk': null<br class=3D""><span
class=3D"">2016-09-07 =
06</span>:46:45,185 ERROR [org.ovirt.engine.core.bll.tas<wbr =
class=3D"">ks.CommandCallbacksPoller] (DefaultQuartzScheduler7) =
[19c58c0d] Failed invoking callback end method 'onFailed' for command =
'<span class=3D"">07608003</span>-ca05-4e2e-b917-85ce5<wbr =
class=3D"">25c011b' with exception 'null', the callback is marked
for =
end method retries</div><div class=3D""><div
class=3D""><div =
class=3D""><div class=3D""><div
class=3D""><div class=3D""><br =
class=3D""><div class=3D"">On 5 September 2016 at 06:46, Nir
Soffer =
<span class=3D""><<a
href=3D"mailto:nsoffer@redhat.com" =
target=3D"_blank"
class=3D"">nsoffer(a)redhat.com</a>&gt;</span> wrote:<br =
class=3D""><blockquote style=3D"margin:0px 0px 0px
0.8ex;border-left:1px =
solid #cccccc;padding-left:1ex" class=3D"">Hi Maton,<br
class=3D""> <br =
class=3D""> Please reply to the list, not to me directly.<br
class=3D""> =
<br class=3D""> Ala, can you look at this? is this a known issue?<br =
class=3D""> <br class=3D""> Thanks,<br
class=3D""> Nir<br class=3D""><div =
class=3D""><div class=3D""><br class=3D""> On
Mon, Sep 5, 2016 at 8:43 =
AM, Maton, Brett <<a href=3D"mailto:matonb@ltresources.co.uk" =
target=3D"_blank"
class=3D"">matonb(a)ltresources.co.uk</a>&gt; wrote:<br =
class=3D""> > Log files as requested<br class=3D"">
><br class=3D"">=
> <a href=3D"https://ufile.io/4fc35" target=3D"_blank" =
class=3D"">https://ufile.io/4fc35</a> vdsm log<br
class=3D""> > <a =
href=3D"https://ufile.io/e9836" target=3D"_blank" =
class=3D"">https://ufile.io/e9836</a> engine 03-Sep<br
class=3D""> > =
<a href=3D"https://ufile.io/15f37" target=3D"_blank" =
class=3D"">https://ufile.io/15f37</a> engine 04-Sep<br
class=3D""> =
><br class=3D""> > vdsm log stops on the 01-Sep...<br
class=3D""> =
><br class=3D""> > Couple of entries from the event
log:<br =
class=3D""> ><br class=3D""> > Sep 3, 2016
7:31:07 PM =
Snapshot 'Auto-generated for Live Storage<br class=3D""> >
Migration' =
deletion for VM 'lv01' has been completed.<br class=3D""> >
Sep 3, =
2016 6:46:46 PM Snapshot 'Auto-generated for Live =
Storage<br class=3D""> > Migration' deletion for VM
'lv01' was =
initiated by SYSTEM<br class=3D""> ><br class=3D"">
> And the =
related tasks<br class=3D""> ><br class=3D"">
> Removing Snapshot =
Auto-generated for Live Storage Migration of VM lv01<br class=3D"">
> =
Sep 3, 2016 6:46:44 PM N/A
=
29f45ca9<br class=3D""> > Validating Sep 3,
2016 6:46:44 =
PM until Sep 3, 2016 6:46:44 PM<br
class=3D""> =
> Executing Sep 3, 2016 6:46:44 PM =
until Sep 3, 2016 7:31:06 PM<br class=3D"">
><br =
class=3D""> > Finalizing Sep 3, 2016 7:31:06
PM =
N/A<br class=3D""> ><br
class=3D""> ><br =
class=3D""> ><br class=3D""> > On 4 September
2016 at 14:27, Nir =
Soffer <<a href=3D"mailto:nsoffer@redhat.com"
target=3D"_blank" =
class=3D"">nsoffer(a)redhat.com</a>&gt; wrote:<br
class=3D""> >><br =
class=3D""> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett =
<<a href=3D"mailto:matonb@ltresources.co.uk"
target=3D"_blank" =
class=3D"">matonb(a)ltresources.co.uk</a>&gt;<br
class=3D""> >> =
wrote:<br class=3D""> >>><br
class=3D""> >>> How do I =
fix / kill a hung vdsm task?<br class=3D""> >>><br
class=3D""> =
>>> It seems to have completed the task but is stuck =
finalising.<br class=3D""> >>><br
class=3D""> >>> =
Removing Snapshot Auto-generated for Live Storage Migration<br
class=3D"">=
>>> Validating<br class=3D"">
>>> Executing<br =
class=3D""> >>> (hour glass) Finalizing<br
class=3D""> =
>>><br class=3D""> >>> Task has
been 'stuck' =
finalising for over 13 hours<br class=3D""> >><br
class=3D""> =
>><br class=3D""> >> Can you share engine and
vdsm logs =
since the time the merge was started?<br class=3D"">
>><br =
class=3D""> >> Nir<br class=3D"">
><br class=3D""> =
></div></div></blockquote></div></div></div></div></div></div></div></d=
iv></blockquote></div></div></div></div>,<p =
class=3D"">______________________________<wbr =
class=3D"">_________________<br class=3D"">Users mailing
list<br =
class=3D""><a href=3D"mailto:Users@ovirt.org"
target=3D"_blank" =
class=3D"">Users(a)ovirt.org</a><br class=3D""><a =
href=3D"http://lists.ovirt.org/mailman/listinfo/users"
target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman<wbr =
class=3D"">/listinfo/users</a></p></blockquote><span
class=3D""><font =
color=3D"#888888" class=3D""><div
class=3D""> </div><div =
class=3D""> </div><div
class=3D"">-- </div><div class=3D"">=D0=A1=
=D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=B5=D0=BC, =
=D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 =
=D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB.</div><div =
class=3D""> </div><div =
class=3D""> </div></font></span></blockquote></div><br
=
class=3D""></div>
</div></div></blockquote></div><br
class=3D""></div>
_______________________________________________<br class=3D"">Users =
mailing list<br class=3D""><a href=3D"mailto:Users@ovirt.org"
=
class=3D"">Users(a)ovirt.org</a><br =
class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br =
class=3D""></div></blockquote></div><br
class=3D""></div></body></html>=
--Apple-Mail=_BE696610-4EC6-4158-8D67-989CBCD17D96--
7:40 a.m.
New subject: Hung task finalizing live migration
Thanks Gervais I'll give that a go
On 10 September 2016 at 15:39, Gervais de Montbrun <gervais(a)demontbrun.com>
wrote:
Hi Maton,
I have seen tasks in a weird state on my cluster also. I've had a vm get
"stuck" during a migration where it says "migrating to" in the web
GUI, but
it has finished migrating hours ago... If I click "Cancel Migraton" the gui
tells me that it is not migrating, but I can't do any action on the vm
because I am then told that the vm can't be acted upon while it is
migrating. I also try to kill the task, but there are none listed
What has worked for me has been to put my hosted-engine in global
maintenance mode, then ssh into the hosted engine and run the
"engine-setup" command. I am not saying the is the best course of action,
but when the engine comes back online the task is cleared.
Cheers,
Gervais
On Sep 10, 2016, at 11:06 AM, Maton, Brett <matonb(a)ltresources.co.uk>
wrote:
Anyone know how to fix this broken task ?
It's persisted through a reboot of all hosts and the engine, something
needs deleting from the database to clear the task and release the locked
disk
On 8 September 2016 at 13:25, Maton, Brett <matonb(a)ltresources.co.uk>
wrote:
> Thanks for the pointer Mikhail, however I don't get any tasks listed with
> that command:
>
> vdsClient -s 0 getAllTasksStatuses
>
> /usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlrpc.
> since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli
> from vdsm import utils, vdscli, constants
>
> {'status': {'message': 'OK', 'code': 0},
'allTasksStatus': {}}
>
>
> On 8 September 2016 at 09:51, Краснобаев Михаил <milo1(a)ya.ru> wrote:
>
>> Hi,
>>
>> There is a way to cancel a running task - look here
>> http://lists.ovirt.org/pipermail/users/2014-November/028946.html
>> I was able to stop snapshot deletion this way.
>>
>> Best, Mikhail.
>>
>> 08.09.2016, 08:14, "Maton, Brett" <matonb(a)ltresources.co.uk>:
>>
>> Any suggestions ?
>>
>> THe task has been hung for 5 days now, I can't start the machine or
>> destroy it.
>>
>>
>> On 7 September 2016 at 06:49, Maton, Brett <matonb(a)ltresources.co.uk>
>> wrote:
>>
>> Sorry just hit reply....
>>
>> I'm seeing these errors in the logs which look related to the problem:
>>
>>
>> 2016-09-07 06:46:35,123 ERROR
[org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
>> (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method
>> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b'
with
>> exception 'null', the callback is marked for end method retries
>> 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.Com
>> <http://org.ovirt.engine.core.bll.com/>mandsFactory]
>> (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command
>> 'LiveMigrateDisk': null
>> 2016-09-07 06:46:45,185 ERROR
[org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
>> (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method
>> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b'
with
>> exception 'null', the callback is marked for end method retries
>>
>> On 5 September 2016 at 06:46, Nir Soffer <nsoffer(a)redhat.com> wrote:
>>
>> Hi Maton,
>>
>> Please reply to the list, not to me directly.
>>
>> Ala, can you look at this? is this a known issue?
>>
>> Thanks,
>> Nir
>>
>> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett <matonb(a)ltresources.co.uk>
>> wrote:
>> > Log files as requested
>> >
>> > https://ufile.io/4fc35 vdsm log
>> > https://ufile.io/e9836 engine 03-Sep
>> > https://ufile.io/15f37 engine 04-Sep
>> >
>> > vdsm log stops on the 01-Sep...
>> >
>> > Couple of entries from the event log:
>> >
>> > Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage
>> > Migration' deletion for VM 'lv01' has been completed.
>> > Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage
>> > Migration' deletion for VM 'lv01' was initiated by SYSTEM
>> >
>> > And the related tasks
>> >
>> > Removing Snapshot Auto-generated for Live Storage Migration of VM lv01
>> > Sep 3, 2016 6:46:44 PM N/A 29f45ca9
>> > Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 6:46:44 PM
>> > Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 PM
>> >
>> > Finalizing Sep 3, 2016 7:31:06 PM N/A
>> >
>> >
>> >
>> > On 4 September 2016 at 14:27, Nir Soffer <nsoffer(a)redhat.com> wrote:
>> >>
>> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <
>> matonb(a)ltresources.co.uk>
>> >> wrote:
>> >>>
>> >>> How do I fix / kill a hung vdsm task?
>> >>>
>> >>> It seems to have completed the task but is stuck finalising.
>> >>>
>> >>> Removing Snapshot Auto-generated for Live Storage Migration
>> >>> Validating
>> >>> Executing
>> >>> (hour glass) Finalizing
>> >>>
>> >>> Task has been 'stuck' finalising for over 13 hours
>> >>
>> >>
>> >> Can you share engine and vdsm logs since the time the merge was
>> started?
>> >>
>> >> Nir
>> >
>> >
>>
>> ,
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>>
>> --
>> С уважением, Краснобаев Михаил.
>>
>>
>>
>
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
8:05 a.m.
New subject: Hung task finalizing live migration
Way-hey! finally the task has gone and I can do 'stuff' with that VM again.
Thanks Gervais, you're a star
On 10 September 2016 at 15:40, Maton, Brett <matonb(a)ltresources.co.uk>
wrote:
Thanks Gervais I'll give that a go
On 10 September 2016 at 15:39, Gervais de Montbrun <gervais(a)demontbrun.com
> wrote:
> Hi Maton,
>
> I have seen tasks in a weird state on my cluster also. I've had a vm get
> "stuck" during a migration where it says "migrating to" in the
web GUI, but
> it has finished migrating hours ago... If I click "Cancel Migraton" the
gui
> tells me that it is not migrating, but I can't do any action on the vm
> because I am then told that the vm can't be acted upon while it is
> migrating. I also try to kill the task, but there are none listed
>
> What has worked for me has been to put my hosted-engine in global
> maintenance mode, then ssh into the hosted engine and run the
> "engine-setup" command. I am not saying the is the best course of action,
> but when the engine comes back online the task is cleared.
>
> Cheers,
> Gervais
>
>
>
> On Sep 10, 2016, at 11:06 AM, Maton, Brett <matonb(a)ltresources.co.uk>
> wrote:
>
> Anyone know how to fix this broken task ?
>
> It's persisted through a reboot of all hosts and the engine, something
> needs deleting from the database to clear the task and release the locked
> disk
>
> On 8 September 2016 at 13:25, Maton, Brett <matonb(a)ltresources.co.uk>
> wrote:
>
>> Thanks for the pointer Mikhail, however I don't get any tasks listed
>> with that command:
>>
>> vdsClient -s 0 getAllTasksStatuses
>>
>> /usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses
>> xmlrpc. since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli
>> from vdsm import utils, vdscli, constants
>>
>> {'status': {'message': 'OK', 'code': 0},
'allTasksStatus': {}}
>>
>>
>> On 8 September 2016 at 09:51, Краснобаев Михаил <milo1(a)ya.ru> wrote:
>>
>>> Hi,
>>>
>>> There is a way to cancel a running task - look here
>>> http://lists.ovirt.org/pipermail/users/2014-November/028946.html
>>> I was able to stop snapshot deletion this way.
>>>
>>> Best, Mikhail.
>>>
>>> 08.09.2016, 08:14, "Maton, Brett"
<matonb(a)ltresources.co.uk>:
>>>
>>> Any suggestions ?
>>>
>>> THe task has been hung for 5 days now, I can't start the machine or
>>> destroy it.
>>>
>>>
>>> On 7 September 2016 at 06:49, Maton, Brett <matonb(a)ltresources.co.uk>
>>> wrote:
>>>
>>> Sorry just hit reply....
>>>
>>> I'm seeing these errors in the logs which look related to the problem:
>>>
>>>
>>> 2016-09-07 06:46:35,123 ERROR
[org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
>>> (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method
>>> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b'
with
>>> exception 'null', the callback is marked for end method retries
>>> 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.Com
>>> <http://org.ovirt.engine.core.bll.com/>mandsFactory]
>>> (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command
>>> 'LiveMigrateDisk': null
>>> 2016-09-07 06:46:45,185 ERROR
[org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
>>> (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method
>>> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b'
with
>>> exception 'null', the callback is marked for end method retries
>>>
>>> On 5 September 2016 at 06:46, Nir Soffer <nsoffer(a)redhat.com> wrote:
>>>
>>> Hi Maton,
>>>
>>> Please reply to the list, not to me directly.
>>>
>>> Ala, can you look at this? is this a known issue?
>>>
>>> Thanks,
>>> Nir
>>>
>>> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett
<matonb(a)ltresources.co.uk>
>>> wrote:
>>> > Log files as requested
>>> >
>>> > https://ufile.io/4fc35 vdsm log
>>> > https://ufile.io/e9836 engine 03-Sep
>>> > https://ufile.io/15f37 engine 04-Sep
>>> >
>>> > vdsm log stops on the 01-Sep...
>>> >
>>> > Couple of entries from the event log:
>>> >
>>> > Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage
>>> > Migration' deletion for VM 'lv01' has been completed.
>>> > Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage
>>> > Migration' deletion for VM 'lv01' was initiated by SYSTEM
>>> >
>>> > And the related tasks
>>> >
>>> > Removing Snapshot Auto-generated for Live Storage Migration of VM lv01
>>> > Sep 3, 2016 6:46:44 PM N/A 29f45ca9
>>> > Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 6:46:44
>>> PM
>>> > Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 PM
>>> >
>>> > Finalizing Sep 3, 2016 7:31:06 PM N/A
>>> >
>>> >
>>> >
>>> > On 4 September 2016 at 14:27, Nir Soffer <nsoffer(a)redhat.com>
wrote:
>>> >>
>>> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <
>>> matonb(a)ltresources.co.uk>
>>> >> wrote:
>>> >>>
>>> >>> How do I fix / kill a hung vdsm task?
>>> >>>
>>> >>> It seems to have completed the task but is stuck finalising.
>>> >>>
>>> >>> Removing Snapshot Auto-generated for Live Storage Migration
>>> >>> Validating
>>> >>> Executing
>>> >>> (hour glass) Finalizing
>>> >>>
>>> >>> Task has been 'stuck' finalising for over 13 hours
>>> >>
>>> >>
>>> >> Can you share engine and vdsm logs since the time the merge was
>>> started?
>>> >>
>>> >> Nir
>>> >
>>> >
>>>
>>> ,
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>>
>>> --
>>> С уважением, Краснобаев Михаил.
>>>
>>>
>>>
>>
>>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
8:09 a.m.
New subject: Hung task finalizing live migration
--Apple-Mail=_EF692F18-F11E-46F1-8425-9C5284E5B3E2
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
YAY!! Glad it worked for you.
:-)
Cheers,
Gervais
On Sep 10, 2016, at 12:05 PM, Maton, Brett
<matonb(a)ltresources.co.uk> =
wrote:
=20
Way-hey! finally the task has gone and I can do 'stuff' with that VM =
again.
=20
Thanks Gervais, you're a star
=20
On 10 September 2016 at 15:40, Maton, Brett <matonb(a)ltresources.co.uk =
<mailto:matonb@ltresources.co.uk>> wrote:
Thanks Gervais I'll give that a go
=20
On 10 September 2016 at 15:39, Gervais de Montbrun =
<gervais(a)demontbrun.com
<mailto:gervais@demontbrun.com>> wrote:
Hi Maton,
=20
I have seen tasks in a weird state on my cluster also. I've had a vm =
get
"stuck" during a migration where it says "migrating to" in the web =
GUI, but it has finished migrating hours ago... If I click "Cancel =
Migraton" the gui tells me that it is not migrating, but I can't do any =
action on the vm because I am then told that the vm can't be acted upon =
while it is migrating. I also try to kill the task, but there are none =
listed
=20
What has worked for me has been to put my hosted-engine in global =
maintenance
mode, then ssh into the hosted engine and run the =
"engine-setup" command. I am not saying the is the best course of =
action, but when the engine comes back online the task is cleared.
=20
Cheers,
Gervais
=20
=20
=20
> On Sep 10, 2016, at 11:06 AM, Maton, Brett <matonb(a)ltresources.co.uk =
<mailto:matonb@ltresources.co.uk>> wrote:
>=20
> Anyone know how to fix this broken task ?
>=20
> It's persisted through a reboot of all hosts and the engine, =
something
needs deleting from the database to clear the task and release =
the locked disk
>=20
> On 8 September 2016 at 13:25, Maton, Brett <matonb(a)ltresources.co.uk =
<mailto:matonb@ltresources.co.uk>> wrote:
> Thanks for the pointer Mikhail, however I don't get any tasks
listed =
with that command:
>=20
> vdsClient -s 0 getAllTasksStatuses
>=20
> /usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses =
xmlrpc.
since ovirt 3.6 xmlrpc is deprecated, please use =
vdsm.jsonrpcvdscli
> from vdsm import utils, vdscli, constants
>=20
> {'status': {'message': 'OK', 'code': 0},
'allTasksStatus': {}}
>=20
>=20
> On 8 September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=
=D0=B0=D0=B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB <milo1(a)ya.ru =
<mailto:milo1@ya.ru>> wrote:
> Hi,
> =20
> There is a way to cancel a running task - look here =
http://lists.ovirt.org/pipermail/users/2014-November/028946.html =
<http://lists.ovirt.org/pipermail/users/2014-November/028946.html>
> I was able to stop snapshot deletion this way.
> =20
> Best, Mikhail.
> =20
> 08.09.2016, 08:14, "Maton, Brett" <matonb(a)ltresources.co.uk =
<mailto:matonb@ltresources.co.uk>>:
>> Any suggestions ?
>>=20
>> THe task has been hung for 5 days now, I can't start the machine or =
destroy it.
>>=20
>>=20
>> On 7 September 2016 at 06:49, Maton, Brett <matonb(a)ltresources.co.uk =
<mailto:matonb@ltresources.co.uk>> wrote:
>> Sorry just hit reply....
>>=20
>> I'm seeing these errors in the logs which look related to the =
problem:
>>=20
>>=20
>> 2016-09-07 06:46:35,123 ERROR =
[org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] =
(DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method =
'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with =
exception 'null', the callback is marked for end method retries
>> 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.Com
=
<http://org.ovirt.engine.core.bll.com/>mandsFactory] =
(DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command =
'LiveMigrateDisk': null
>> 2016-09-07 06:46:45,185 ERROR =
[org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] =
(DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method =
'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with =
exception 'null', the callback is marked for end method retries
>>=20
>> On 5 September 2016 at 06:46, Nir Soffer <nsoffer(a)redhat.com =
<mailto:nsoffer@redhat.com>> wrote:
>> Hi Maton,
>>=20
>> Please reply to the list, not to me directly.
>>=20
>> Ala, can you look at this? is this a known issue?
>>=20
>> Thanks,
>> Nir
>>=20
>> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett =
<matonb(a)ltresources.co.uk
<mailto:matonb@ltresources.co.uk>> wrote:
>> > Log files as requested
>> >
>> > https://ufile.io/4fc35 <https://ufile.io/4fc35> vdsm log
>> > https://ufile.io/e9836 <https://ufile.io/e9836> engine 03-Sep
>> > https://ufile.io/15f37 <https://ufile.io/15f37> engine 04-Sep
>> >
>> > vdsm log stops on the 01-Sep...
>> >
>> > Couple of entries from the event log:
>> >
>> > Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live =
Storage
>> > Migration' deletion for VM 'lv01' has been
completed.
>> > Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live =
Storage
>> > Migration' deletion for VM 'lv01' was
initiated by SYSTEM
>> >
>> > And the related tasks
>> >
>> > Removing Snapshot Auto-generated for Live Storage Migration of VM =
lv01
>> > Sep 3, 2016 6:46:44 PM N/A 29f45ca9
>> > Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 =
6:46:44
PM
>> > Executing Sep 3, 2016 6:46:44 PM until Sep 3,
2016 =
7:31:06 PM
>> >
>> > Finalizing Sep 3, 2016 7:31:06 PM N/A
>> >
>> >
>> >
>> > On 4 September 2016 at 14:27, Nir Soffer <nsoffer(a)redhat.com =
<mailto:nsoffer@redhat.com>> wrote:
>> >>
>> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett =
<matonb(a)ltresources.co.uk <mailto:matonb@ltresources.co.uk>>
>> >> wrote:
>> >>>
>> >>> How do I fix / kill a hung vdsm task?
>> >>>
>> >>> It seems to have completed the task but is stuck finalising.
>> >>>
>> >>> Removing Snapshot Auto-generated for Live Storage Migration
>> >>> Validating
>> >>> Executing
>> >>> (hour glass) Finalizing
>> >>>
>> >>> Task has been 'stuck' finalising for over 13 hours
>> >>
>> >>
>> >> Can you share engine and vdsm logs since the time the merge was =
started?
>> >>
>> >> Nir
>> >
>> >
>> ,
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>=20
> =20
> --=20
> =D0=A1 =D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=B5=D0=BC, =
=D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 =
=D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB.
> =20
> =20
>=20
>=20
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>
=20
=20
=20
--Apple-Mail=_EF692F18-F11E-46F1-8425-9C5284E5B3E2
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D"">YAY!! Glad it worked for you.<div
class=3D""><br =
class=3D""></div><div class=3D"">:-)<br
class=3D""><div class=3D"">
<div id=3D"signature" class=3D""><br
class=3D"">Cheers,<br =
class=3D"">Gervais<br class=3D""><br
class=3D""><br class=3D""></div>
</div>
<br class=3D""><div><blockquote type=3D"cite"
class=3D""><div =
class=3D"">On Sep 10, 2016, at 12:05 PM, Maton, Brett <<a =
href=3D"mailto:matonb@ltresources.co.uk" =
class=3D"">matonb(a)ltresources.co.uk</a>&gt;
wrote:</div><br =
class=3D"Apple-interchange-newline"><div class=3D""><div
dir=3D"ltr" =
class=3D""><div class=3D"">Way-hey! finally the task has gone
and I can =
do 'stuff' with that VM again.<br class=3D""><br
class=3D""></div>Thanks =
Gervais, you're a star<br class=3D""></div><div
class=3D"gmail_extra"><br =
class=3D""><div class=3D"gmail_quote">On 10 September 2016 at
15:40, =
Maton, Brett <span dir=3D"ltr" class=3D""><<a =
href=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" =
class=3D"">matonb(a)ltresources.co.uk</a>&gt;</span>
wrote:<br =
class=3D""><blockquote class=3D"gmail_quote"
style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr" =
class=3D"">Thanks Gervais I'll give that a go<br
class=3D""></div><div =
class=3D"HOEnZb"><div class=3D"h5"><div
class=3D"gmail_extra"><br =
class=3D""><div class=3D"gmail_quote">On 10 September 2016 at
15:39, =
Gervais de Montbrun <span dir=3D"ltr" class=3D""><<a
=
href=3D"mailto:gervais@demontbrun.com" target=3D"_blank" =
class=3D"">gervais(a)demontbrun.com</a>&gt;</span> wrote:<br
=
class=3D""><blockquote class=3D"gmail_quote"
style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div =
style=3D"word-wrap:break-word" class=3D"">Hi Maton,<div
class=3D""><br =
class=3D""></div><div class=3D"">I have seen tasks in a
weird state on =
my cluster also. I've had a vm get "stuck" during a migration where it =
says "migrating to" in the web GUI, but it has finished migrating hours =
ago... If I click "Cancel Migraton" the gui tells me that it is not =
migrating, but I can't do any action on the vm because I am then told =
that the vm can't be acted upon while it is migrating. I also try to =
kill the task, but there are none listed<br class=3D""><div
class=3D""><br=
class=3D""></div><div class=3D"">What has worked for
me has been to put =
my hosted-engine in global maintenance mode, then ssh into the hosted =
engine and run the "engine-setup" command. I am not saying the is the =
best course of action, but when the engine comes back online the task is =
cleared.</div><div class=3D"">
<div class=3D""><br class=3D"">Cheers,<br
class=3D"">Gervais<br =
class=3D""><br class=3D""><br
class=3D""></div>
</div><div class=3D""><div class=3D"">
<br class=3D""><div class=3D""><blockquote
type=3D"cite" class=3D""><div =
class=3D"">On Sep 10, 2016, at 11:06 AM, Maton, Brett <<a =
href=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" =
class=3D"">matonb(a)ltresources.co.uk</a>&gt;
wrote:</div><br =
class=3D""><div class=3D""><div dir=3D"ltr"
class=3D""><div =
class=3D"">Anyone know how to fix this broken task ?<br
class=3D""><br =
class=3D""></div><div class=3D"">It's persisted
through a reboot of all =
hosts and the engine, something needs deleting from the database to =
clear the task and release the locked disk<br
class=3D""></div></div><div =
class=3D"gmail_extra"><br class=3D""><div
class=3D"gmail_quote">On 8 =
September 2016 at 13:25, Maton, Brett <span dir=3D"ltr"
class=3D""><<a =
href=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" =
class=3D"">matonb(a)ltresources.co.uk</a>&gt;</span>
wrote:<br =
class=3D""><blockquote class=3D"gmail_quote"
style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr" =
class=3D"">Thanks for the pointer Mikhail, however I don't get any tasks
=
listed with that command:<br class=3D""><br
class=3D"">vdsClient -s 0 =
getAllTasksStatuses<br class=3D""><br =
class=3D"">/usr/share/vdsm/vdsClient.py:3<wbr class=3D"">3:
=
DeprecationWarning: vdscli uses xmlrpc. since ovirt 3.6 xmlrpc is =
deprecated, please use vdsm.jsonrpcvdscli<br class=3D""> from
vdsm =
import utils, vdscli, constants<br class=3D""><br
class=3D"">{'status': =
{'message': 'OK', 'code': 0}, 'allTasksStatus': {}}<br
class=3D""><br =
class=3D""></div><div class=3D""><div
class=3D""><div =
class=3D"gmail_extra"><br class=3D""><div
class=3D"gmail_quote">On 8 =
September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=
=B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB <span dir=3D"ltr" =
class=3D""><<a href=3D"mailto:milo1@ya.ru"
target=3D"_blank" =
class=3D"">milo1(a)ya.ru</a>&gt;</span> wrote:<br
class=3D""><blockquote =
class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc =
solid;padding-left:1ex"><div class=3D"">Hi,</div><div =
class=3D""> </div><div class=3D"">There is a
way to cancel a =
running task - look here <a =
href=3D"http://lists.ovirt.org/pipermail/users/2014-November/028946.... =
target=3D"_blank" class=3D"">http://lists.ovirt.org/piperma<wbr
=
class=3D"">il/users/2014-November/028946.<wbr =
class=3D"">html</a></div><div class=3D"">I was
able to stop snapshot =
deletion this way.</div><div
class=3D""> </div><div class=3D"">Best, =
Mikhail.</div><div class=3D""> </div><div
class=3D"">08.09.2016, =
08:14, "Maton, Brett" <<a
href=3D"mailto:matonb@ltresources.co.uk" =
target=3D"_blank" =
class=3D"">matonb@ltresources.co.uk</a>>:</div><blockquote
=
type=3D"cite" class=3D""><div class=3D""><div
class=3D""><div =
class=3D""><div class=3D""><div
class=3D"">Any suggestions ?<br =
class=3D""><br class=3D""></div>THe task has been hung
for 5 days now, I =
can't start the machine or destroy it.<br class=3D""><br =
class=3D""></div></div><div class=3D""><br
class=3D""><div class=3D"">On =
7 September 2016 at 06:49, Maton, Brett <span class=3D""><<a =
href=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" =
class=3D"">matonb(a)ltresources.co.uk</a>&gt;</span>
wrote:<br =
class=3D""><blockquote style=3D"margin:0 0 0 0.8ex;border-left:1px
#ccc =
solid;padding-left:1ex" class=3D""><div
class=3D""><div class=3D""><div =
class=3D"">Sorry just hit reply....<br class=3D""><br
class=3D""></div>I'm=
seeing these errors in the logs which look related to the problem:<br =
class=3D""><br class=3D""><br
class=3D""><span class=3D"">2016-09-07 =
06</span>:46:35,123 ERROR [org.ovirt.engine.core.bll.tas<wbr =
class=3D"">ks.CommandCallbacksPoller] (DefaultQuartzScheduler6) =
[19c58c0d] Failed invoking callback end method 'onFailed' for command =
'<span class=3D"">07608003</span>-ca05-4e2e-b917-85ce5<wbr =
class=3D"">25c011b' with exception 'null', the callback is marked
for =
end method retries<br class=3D""><span
class=3D"">2016-09-07 =
06</span>:46:45,184 ERROR [<a =
href=3D"http://org.ovirt.engine.core.bll.com/" target=3D"_blank" =
class=3D"">org.ovirt.engine.core.bll.Com</a><wbr
class=3D"">mandsFactory] =
(DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command =
'LiveMigrateDisk': null<br class=3D""><span
class=3D"">2016-09-07 =
06</span>:46:45,185 ERROR [org.ovirt.engine.core.bll.tas<wbr =
class=3D"">ks.CommandCallbacksPoller] (DefaultQuartzScheduler7) =
[19c58c0d] Failed invoking callback end method 'onFailed' for command =
'<span class=3D"">07608003</span>-ca05-4e2e-b917-85ce5<wbr =
class=3D"">25c011b' with exception 'null', the callback is marked
for =
end method retries</div><div class=3D""><div
class=3D""><div =
class=3D""><div class=3D""><div
class=3D""><div class=3D""><br =
class=3D""><div class=3D"">On 5 September 2016 at 06:46, Nir
Soffer =
<span class=3D""><<a
href=3D"mailto:nsoffer@redhat.com" =
target=3D"_blank"
class=3D"">nsoffer(a)redhat.com</a>&gt;</span> wrote:<br =
class=3D""><blockquote style=3D"margin:0px 0px 0px
0.8ex;border-left:1px =
solid #cccccc;padding-left:1ex" class=3D"">Hi Maton,<br
class=3D""> <br =
class=3D""> Please reply to the list, not to me directly.<br
class=3D""> =
<br class=3D""> Ala, can you look at this? is this a known issue?<br =
class=3D""> <br class=3D""> Thanks,<br
class=3D""> Nir<br class=3D""><div =
class=3D""><div class=3D""><br class=3D""> On
Mon, Sep 5, 2016 at 8:43 =
AM, Maton, Brett <<a href=3D"mailto:matonb@ltresources.co.uk" =
target=3D"_blank"
class=3D"">matonb(a)ltresources.co.uk</a>&gt; wrote:<br =
class=3D""> > Log files as requested<br class=3D"">
><br class=3D"">=
> <a href=3D"https://ufile.io/4fc35" target=3D"_blank" =
class=3D"">https://ufile.io/4fc35</a> vdsm log<br
class=3D""> > <a =
href=3D"https://ufile.io/e9836" target=3D"_blank" =
class=3D"">https://ufile.io/e9836</a> engine 03-Sep<br
class=3D""> > =
<a href=3D"https://ufile.io/15f37" target=3D"_blank" =
class=3D"">https://ufile.io/15f37</a> engine 04-Sep<br
class=3D""> =
><br class=3D""> > vdsm log stops on the 01-Sep...<br
class=3D""> =
><br class=3D""> > Couple of entries from the event
log:<br =
class=3D""> ><br class=3D""> > Sep 3, 2016
7:31:07 PM =
Snapshot 'Auto-generated for Live Storage<br class=3D""> >
Migration' =
deletion for VM 'lv01' has been completed.<br class=3D""> >
Sep 3, =
2016 6:46:46 PM Snapshot 'Auto-generated for Live =
Storage<br class=3D""> > Migration' deletion for VM
'lv01' was =
initiated by SYSTEM<br class=3D""> ><br class=3D"">
> And the =
related tasks<br class=3D""> ><br class=3D"">
> Removing Snapshot =
Auto-generated for Live Storage Migration of VM lv01<br class=3D"">
> =
Sep 3, 2016 6:46:44 PM N/A
=
29f45ca9<br class=3D""> > Validating Sep 3,
2016 6:46:44 =
PM until Sep 3, 2016 6:46:44 PM<br
class=3D""> =
> Executing Sep 3, 2016 6:46:44 PM =
until Sep 3, 2016 7:31:06 PM<br class=3D"">
><br =
class=3D""> > Finalizing Sep 3, 2016 7:31:06
PM =
N/A<br class=3D""> ><br
class=3D""> ><br =
class=3D""> ><br class=3D""> > On 4 September
2016 at 14:27, Nir =
Soffer <<a href=3D"mailto:nsoffer@redhat.com"
target=3D"_blank" =
class=3D"">nsoffer(a)redhat.com</a>&gt; wrote:<br
class=3D""> >><br =
class=3D""> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett =
<<a href=3D"mailto:matonb@ltresources.co.uk"
target=3D"_blank" =
class=3D"">matonb(a)ltresources.co.uk</a>&gt;<br
class=3D""> >> =
wrote:<br class=3D""> >>><br
class=3D""> >>> How do I =
fix / kill a hung vdsm task?<br class=3D""> >>><br
class=3D""> =
>>> It seems to have completed the task but is stuck =
finalising.<br class=3D""> >>><br
class=3D""> >>> =
Removing Snapshot Auto-generated for Live Storage Migration<br
class=3D"">=
>>> Validating<br class=3D"">
>>> Executing<br =
class=3D""> >>> (hour glass) Finalizing<br
class=3D""> =
>>><br class=3D""> >>> Task has
been 'stuck' =
finalising for over 13 hours<br class=3D""> >><br
class=3D""> =
>><br class=3D""> >> Can you share engine and
vdsm logs =
since the time the merge was started?<br class=3D"">
>><br =
class=3D""> >> Nir<br class=3D"">
><br class=3D""> =
></div></div></blockquote></div></div></div></div></div></div></div></d=
iv></blockquote></div></div></div></div>,<p =
class=3D"">______________________________<wbr =
class=3D"">_________________<br class=3D"">Users mailing
list<br =
class=3D""><a href=3D"mailto:Users@ovirt.org"
target=3D"_blank" =
class=3D"">Users(a)ovirt.org</a><br class=3D""><a =
href=3D"http://lists.ovirt.org/mailman/listinfo/users"
target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman<wbr =
class=3D"">/listinfo/users</a></p></blockquote><span
class=3D""><font =
color=3D"#888888" class=3D""><div
class=3D""> </div><div =
class=3D""> </div><div
class=3D"">-- </div><div class=3D"">=D0=A1=
=D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=B5=D0=BC, =
=D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 =
=D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB.</div><div =
class=3D""> </div><div =
class=3D""> </div></font></span></blockquote></div><br
=
class=3D""></div>
</div></div></blockquote></div><br
class=3D""></div>
______________________________<wbr class=3D"">_________________<br =
class=3D"">Users mailing list<br class=3D""><a =
href=3D"mailto:Users@ovirt.org" target=3D"_blank" =
class=3D"">Users(a)ovirt.org</a><br class=3D""><a =
href=3D"http://lists.ovirt.org/mailman/listinfo/users"
target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman<wbr =
class=3D"">/listinfo/users</a><br
class=3D""></div></blockquote></div><br =
class=3D""></div></div></div></div></blockquote></div><br
=
class=3D""></div>
</div></div></blockquote></div><br
class=3D""></div>
</div></blockquote></div><br
class=3D""></div></body></html>=
--Apple-Mail=_EF692F18-F11E-46F1-8425-9C5284E5B3E2--
5:46 a.m.
New subject: Hung task finalizing live migration
------=_Part_1725676_1350899779.1481204814198
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
I had a similar situation where I was attempting and failing to delete a ci=
nder disk and this fixed it. I'm using 4.0.5-5=20
Regards,=20
Logan=20
----- On Sep 10, 2016, at 9:39 AM, Gervais de Montbrun <gervais(a)demontbrun.=
com> wrote:=20
| Hi Maton,
| I have seen tasks in a weird state on my cluster also. I've had a vm get =
"stuck"
| during a migration where it says "migrating to" in the web GUI, but it ha=
s
| finished migrating hours ago... If I click "Cancel Migraton" the gui tell=
s me
| that it is not migrating, but I can't do any action on the vm because I a=
m then
| told that the vm can't be acted upon while it is migrating. I also try to=
kill
| the task, but there are none listed
| What has worked for me has been to put my hosted-engine in global mainten=
ance
| mode, then ssh into the hosted engine and run the "engine-setup" command.=
I am
| not saying the is the best course of action, but when the engine comes ba=
ck
| online the task is cleared.
| Cheers,
| Gervais
|| On Sep 10, 2016, at 11:06 AM, Maton, Brett < matonb(a)ltresources.co.uk > =
wrote:
|| Anyone know how to fix this broken task ?
|| It's persisted through a reboot of all hosts and the engine, something n=
eeds
|| deleting from the database to clear the task and release the locked disk
|| On 8 September 2016 at 13:25, Maton, Brett < matonb(a)ltresources.co.uk > =
wrote:
||| Thanks for the pointer Mikhail, however I don't get any tasks listed wi=
th that
||| command:
||| vdsClient -s 0 getAllTasksStatuses
||| /usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlrpc=
. since
||| ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli
||| from vdsm import utils, vdscli, constants
||| {'status': {'message': 'OK', 'code': 0},
'allTasksStatus': {}}
||| On 8 September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=
=B1=D0=B0=D0=B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB < milo1(a)ya.ru > =
wrote:
|||| Hi,
|||| There is a way to cancel a running task - look here
|||| http://lists.ovirt.org/pipermail/users/2014-November/028946.html
|||| I was able to stop snapshot deletion this way.
|||| Best, Mikhail.
|||| 08.09.2016, 08:14, "Maton, Brett" < matonb(a)ltresources.co.uk >:
||||| Any suggestions ?
||||| THe task has been hung for 5 days now, I can't start the machine or d=
estroy it.
||||| On 7 September 2016 at 06:49, Maton, Brett < matonb(a)ltresources.co.uk=
wrote:
|||||| Sorry just hit reply....
|||||| I'm seeing these errors in the logs which look related to the proble=
m:
|||||| 2016-09-07 06 :46:35,123 ERROR
|||||| [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
|||||| (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end me=
thod
|||||| 'onFailed' for command ' 07608003 -ca05-4e2e-b917-85ce525c011b'
with=
exception
|||||| 'null', the callback is marked for end method retries
|||||| 2016-09-07 06 :46:45,184 ERROR [ org.ovirt.engine.core.bll.Com mands=
Factory]
|||||| (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of com=
mand
|||||| 'LiveMigrateDisk': null
|||||| 2016-09-07 06 :46:45,185 ERROR
|||||| [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
|||||| (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end me=
thod
|||||| 'onFailed' for command ' 07608003 -ca05-4e2e-b917-85ce525c011b'
with=
exception
|||||| 'null', the callback is marked for end method retries
|||||| On 5 September 2016 at 06:46, Nir Soffer < nsoffer(a)redhat.com > wrot=
e:
||||||| Hi Maton,
||||||| Please reply to the list, not to me directly.
||||||| Ala, can you look at this? is this a known issue?
||||||| Thanks,
||||||| Nir
||||||| On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett < matonb(a)ltresources.c=
o.uk wrote:
||||||| > Log files as requested
||||||| > https://ufile.io/4fc35 vdsm log
||||||| > https://ufile.io/e9836 engine 03-Sep
||||||| > https://ufile.io/15f37 engine 04-Sep
||||||| > vdsm log stops on the 01-Sep...
||||||| > Couple of entries from the event log:
||||||| > Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage
||||||| > Migration' deletion for VM 'lv01' has been completed.
||||||| > Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage
||||||| > Migration' deletion for VM 'lv01' was initiated by SYSTEM
||||||| > And the related tasks
||||||| > Removing Snapshot Auto-generated for Live Storage Migration of VM=
lv01
||||||| > Sep 3, 2016 6:46:44 PM N/A 29f45ca9
||||||| > Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 6:46:44 PM
||||||| > Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 PM
||||||| > Finalizing Sep 3, 2016 7:31:06 PM N/A
||||||| > On 4 September 2016 at 14:27, Nir Soffer < nsoffer(a)redhat.com > w=
rote:
||||||| >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett < matonb@ltresourc=
es.co.uk >
||||||| >> wrote:
||||||| >>> How do I fix / kill a hung vdsm task?
||||||| >>> It seems to have completed the task but is stuck finalising.
||||||| >>> Removing Snapshot Auto-generated for Live Storage Migration
||||||| >>> Validating
||||||| >>> Executing
||||||| >>> (hour glass) Finalizing
||||||| >>> Task has been 'stuck' finalising for over 13 hours
||||||| >> Can you share engine and vdsm logs since the time the merge was =
started?
||||||| >> Nir
||||| ,
||||| _______________________________________________
||||| Users mailing list
||||| Users(a)ovirt.org
||||| http://lists.ovirt.org/mailman/listinfo/users
|||| --
|||| =D0=A1 =D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=B5=D0=BC, =D0=9A=
=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 =D0=9C=D0=B8=D1=85=
=D0=B0=D0=B8=D0=BB.
|| _______________________________________________
|| Users mailing list
|| Users(a)ovirt.org
|| http://lists.ovirt.org/mailman/listinfo/users
| _______________________________________________
| Users mailing list
| Users(a)ovirt.org
| http://lists.ovirt.org/mailman/listinfo/users
------=_Part_1725676_1350899779.1481204814198
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable
<html><body><div style=3D"font-family: Arial; font-size: 12pt; color:
#0000=
00"><div>I had a similar situation where I was attempting and failing to
de=
lete a cinder disk and this fixed it. I'm using
4.0.5-5<br></div><div=
<br></div><div
data-marker=3D"__SIG_PRE__">Regards,<br>Logan</div><br><spa=
n id=3D"zwchr" data-marker=3D"__DIVIDER__">----- On Sep 10,
2016, at 9:39 A=
M, Gervais de Montbrun &lt;gervais(a)demontbrun.com&gt;
wrote:<br></span><div=
data-marker=3D"__QUOTED_TEXT__"><blockquote style=3D"border-left:
2px soli=
d #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: n=
ormal; font-style: normal; text-decoration: none; font-family: Helvetica,Ar=
ial,sans-serif; font-size: 12pt;" data-mce-style=3D"border-left: 2px solid =
#1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: nor=
mal; font-style: normal; text-decoration: none; font-family: Helvetica,Aria=
l,sans-serif; font-size: 12pt;">Hi Maton,<div class=3D""><br
class=3D""></d=
iv><div class=3D"">I have seen tasks in a weird state on my cluster
also. I=
've had a vm get "stuck" during a migration where it says "migrating
to" in=
the web GUI, but it has finished migrating hours ago... If I click "Cancel=
Migraton" the gui tells me that it is not migrating, but I can't do any ac=
tion on the vm because I am then told that the vm can't be acted upon while=
it is migrating. I also try to kill the task, but there are none listed<br=
class=3D""><div class=3D""><br
class=3D"webkit-block-placeholder"></div><d=
iv class=3D"">What has worked for me has been to put my hosted-engine in gl=
obal maintenance mode, then ssh into the hosted engine and run the "engine-=
setup" command. I am not saying the is the best course of action, but when =
the engine comes back online the task is cleared.</div><div
class=3D"">
<div id=3D"signature" class=3D""><br
class=3D"">Cheers,<br class=3D"">Gerva=
is<br class=3D""><br class=3D""><br
class=3D""></div>
</div>
<br class=3D""><div><blockquote class=3D""><div
class=3D"">On Sep 10, 2016,=
at 11:06 AM, Maton, Brett <<a
href=3D"mailto:matonb@ltresources.co.uk" =
class=3D""
target=3D"_blank">matonb(a)ltresources.co.uk</a>&gt;
wrote:</div><=
br class=3D"Apple-interchange-newline"><div
class=3D""><div dir=3D"ltr" cla=
ss=3D""><div class=3D"">Anyone know how to fix this broken
task ?<br class=
=3D""><br class=3D""></div><div
class=3D"">It's persisted through a reboot =
of all hosts and the engine, something needs deleting from the database to =
clear the task and release the locked disk<br
class=3D""></div></div><div c=
lass=3D"gmail_extra"><br class=3D""><div
class=3D"gmail_quote">On 8 Septemb=
er 2016 at 13:25, Maton, Brett <span dir=3D"ltr"
class=3D""><<a href=3D"=
mailto:matonb@ltresources.co.uk" target=3D"_blank"
class=3D"">matonb@ltreso=
urces.co.uk</a>></span> wrote:<br
class=3D""><blockquote class=3D"gmail_=
quote" style=3D"margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-le=
ft: 1ex;" data-mce-style=3D"margin: 0 0 0 .8ex; border-left: 1px #ccc solid=
; padding-left: 1ex;"><div dir=3D"ltr" class=3D"">Thanks
for the pointer Mi=
khail, however I don't get any tasks listed with that command:<br class=3D"=
"><br class=3D"">vdsClient -s 0 getAllTasksStatuses<br
class=3D""><br class=
=3D"">/usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlr=
pc. since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli<br =
class=3D""> from vdsm import utils, vdscli, constants<br
class=3D""><=
br class=3D"">{'status': {'message': 'OK',
'code': 0}, 'allTasksStatus': {}=
}<br class=3D""><br class=3D""></div><div
class=3D"HOEnZb"><div class=3D"h5=
"><div class=3D"gmail_extra"><br
class=3D""><div class=3D"gmail_quote">On 8=
September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=
=D0=B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB <span dir=3D"ltr" class=
=3D""><<a href=3D"mailto:milo1@ya.ru"
target=3D"_blank" class=3D"">milo1=
@ya.ru</a>></span> wrote:<br class=3D""><blockquote
class=3D"gmail_quote=
" style=3D"margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1=
ex;" data-mce-style=3D"margin: 0 0 0 .8ex; border-left: 1px #ccc solid; pad=
ding-left: 1ex;"><div class=3D"">Hi,</div><div
class=3D""> </div><div =
class=3D"">There is a way to cancel a running task - look
here =
<a href=3D"http://lists.ovirt.org/pipermail/users/2014-November/028946....
" target=3D"_blank"
class=3D"">http://lists.ovirt.org/pipermail/users/2014-=
November/028946.html</a></div><div class=3D"">I was able to
stop snapshot d=
eletion this way.</div><div class=3D""> </div><div
class=3D"">Best, Mi=
khail.</div><div class=3D""> </div><div
class=3D"">08.09.2016, 08:14, =
"Maton, Brett" <<a href=3D"mailto:matonb@ltresources.co.uk"
target=3D"_b=
lank"
class=3D"">matonb@ltresources.co.uk</a>>:</div><blockquote
class=
=3D""><div class=3D""><div class=3D""><div
class=3D""><div class=3D""><div =
class=3D"">Any suggestions ?<br class=3D""><br
class=3D""></div>THe task ha=
s been hung for 5 days now, I can't start the machine or destroy it.<br cla=
ss=3D""><br class=3D""></div></div><div
class=3D""><br class=3D""><div clas=
s=3D"">On 7 September 2016 at 06:49, Maton, Brett <span
class=3D""><<a h=
ref=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank"
class=3D"">matonb=
@ltresources.co.uk</a>></span> wrote:<br
class=3D""><blockquote style=3D=
"margin: 0 0 0 0.8ex; border-left: 1px #ccc solid; padding-left: 1ex;" clas=
s=3D"" data-mce-style=3D"margin: 0 0 0 0.8ex; border-left: 1px #ccc solid;
=
padding-left: 1ex;"><div class=3D""><div
class=3D""><div class=3D"">Sorry j=
ust hit reply....<br class=3D""><br
class=3D""></div>I'm seeing these error=
s in the logs which look related to the problem:<br class=3D""><br
class=3D=
""><br class=3D""><span class=3D"">2016-09-07
06</span>:46:35,123 ERROR [or=
g.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzSchedul=
er6) [19c58c0d] Failed invoking callback end method 'onFailed' for command =
'<span
class=3D"">07608003</span>-ca05-4e2e-b917-85ce525c011b' with
excepti=
on 'null', the callback is marked for end method retries<br
class=3D""><spa=
n class=3D"">2016-09-07 06</span>:46:45,184 ERROR [<a
href=3D"http://org.ov=
irt.engine.core.bll.com/" class=3D""
target=3D"_blank">org.ovirt.engine.cor=
e.bll.Com</a>mandsFactory] (DefaultQuartzScheduler7) [19c58c0d] Error in in=
vocating CTOR of command 'LiveMigrateDisk': null<br
class=3D""><span class=
=3D"">2016-09-07 06</span>:46:45,185 ERROR
[org.ovirt.engine.core.bll.tasks=
.CommandCallbacksPoller] (DefaultQuartzScheduler7) [19c58c0d] Failed invoki=
ng callback end method 'onFailed' for command '<span
class=3D"">07608003</s=
pan>-ca05-4e2e-b917-85ce525c011b' with exception 'null', the callback is
ma=
rked for end method retries</div><div class=3D""><div
class=3D""><div class=
=3D""><div class=3D""><div class=3D""><div
class=3D""><br class=3D""><div c=
lass=3D"">On 5 September 2016 at 06:46, Nir Soffer <span
class=3D""><<a =
href=3D"mailto:nsoffer@redhat.com" target=3D"_blank"
class=3D"">nsoffer@red=
hat.com</a>></span> wrote:<br class=3D""><blockquote
style=3D"margin: 0p=
x 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;" class=
=3D"" data-mce-style=3D"margin: 0px 0px 0px 0.8ex; border-left: 1px solid
#=
cccccc; padding-left: 1ex;">Hi Maton,<br class=3D""> <br
class=3D""> Please=
reply to the list, not to me directly.<br class=3D""> <br
class=3D""> Ala,=
can you look at this? is this a known issue?<br class=3D""> <br
class=3D""=
Thanks,<br class=3D""> Nir<br
class=3D""><div class=3D""><div class=3D"">=
<br class=3D""> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett
<<a href=3D=
"mailto:matonb@ltresources.co.uk" target=3D"_blank"
class=3D"">matonb@ltres=
ources.co.uk</a>> wrote:<br class=3D""> > Log files as
requested<br c=
lass=3D""> ><br class=3D""> > <a
href=3D"https://ufile.io/4fc35" targ=
et=3D"_blank" class=3D"">https://ufile.io/4fc35</a> vdsm
log<br class=3D"">=
> <a href=3D"https://ufile.io/e9836" target=3D"_blank"
class=3D"">https=
://ufile.io/e9836</a> engine 03-Sep<br class=3D""> > <a
href=3D"https://=
ufile.io/15f37" target=3D"_blank"
class=3D"">https://ufile.io/15f37</a> eng=
ine 04-Sep<br class=3D""> ><br class=3D""> >
vdsm log stops on the 01=
-Sep...<br class=3D""> ><br class=3D""> >
Couple of entries from the =
event log:<br class=3D""> ><br class=3D""> >
Sep 3, 2016 7:31:07 PM&n=
bsp; Snapshot 'Auto-generated for Live Storage<br
class=3D""> > M=
igration' deletion for VM 'lv01' has been completed.<br
class=3D""> > Se=
p 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage=
<br class=3D""> > Migration' deletion for VM 'lv01' was
initiated by SYS=
TEM<br class=3D""> ><br class=3D""> > And
the related tasks<br class=
=3D""> ><br class=3D""> > Removing Snapshot
Auto-generated for Live S=
torage Migration of VM lv01<br class=3D""> > Sep 3, 2016 6:46:44
PM =
; N/A 29f45ca9<br
class=3D""> > Valida=
ting Sep 3, 2016 6:46:44 PM until
Se=
p 3, 2016 6:46:44 PM<br class=3D""> > Executing
Sep 3, 2016=
6:46:44 PM until Sep 3, 2016 7:31:06 PM<br
class=
=3D""> ><br class=3D""> > Finalizing
Sep 3, 2016 7:31:06=
PM N/A<br class=3D"">
><br class=3D""> ><=
br class=3D""> ><br class=3D""> > On 4
September 2016 at 14:27, Nir S=
offer <<a href=3D"mailto:nsoffer@redhat.com"
target=3D"_blank" class=3D"=
">nsoffer(a)redhat.com</a>&gt; wrote:<br class=3D"">
>><br class=3D""> =
>> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <<a
href=3D"mailt=
o:matonb@ltresources.co.uk" target=3D"_blank"
class=3D"">matonb@ltresources=
.co.uk</a>><br class=3D""> >> wrote:<br
class=3D""> >>><b=
r class=3D""> >>> How do I fix / kill a hung vdsm
task?<br class=
=3D""> >>><br class=3D"">
>>> It seems to have completed =
the task but is stuck finalising.<br class=3D"">
>>><br class=3D""=
>>> Removing Snapshot Auto-generated for Live
Storage Migration<=
br class=3D""> >>>
Validating<br class=3D""> >>> Executin=
g<br class=3D""> >>> (hour glass) Finalizing<br
class=3D""> >&g=
t;><br class=3D""> >>> Task has been
'stuck' finalising for ove=
r 13 hours<br class=3D""> >><br class=3D"">
>><br class=3D""> &=
gt;> Can you share engine and vdsm logs since the time the merge was sta=
rted?<br class=3D""> >><br class=3D"">
>> Nir<br class=3D""> &g=
t;<br class=3D"">
></div></div></blockquote></div></div></div></div></di=
v></div></div></div></blockquote></div></div></div></div>,<p
class=3D"">___=
____________________________________________<br class=3D"">Users mailing
li=
st<br class=3D""><a href=3D"mailto:Users@ovirt.org"
target=3D"_blank" class=
=3D"">Users(a)ovirt.org</a><br class=3D""><a
href=3D"http://lists.ovirt.org/m=
ailman/listinfo/users" target=3D"_blank"
class=3D"">http://lists.ovirt.org/=
mailman/listinfo/users</a></p></blockquote><span
class=3D""><span color=3D"=
#888888" data-mce-style=3D"color: #888888;" style=3D"color:
#888888;"><div =
class=3D""> </div><div
class=3D""> </div><div
class=3D"">-- =
</div><div class=3D"">=D0=A1
=D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=
=B5=D0=BC, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 =D0=
=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB.</div><div
class=3D""> </div><div cl=
ass=3D""> </div></span></span></blockquote></div><br
class=3D""></div>
</div></div></blockquote></div><br
class=3D""></div>
_______________________________________________<br class=3D"">Users
mailing=
list<br class=3D""><a href=3D"mailto:Users@ovirt.org"
class=3D"" target=3D=
"_blank">Users(a)ovirt.org</a><br
class=3D"">http://lists.ovirt.org/mailman/l=
istinfo/users<br
class=3D""></div></blockquote></div><br
class=3D""></div><=
br>_______________________________________________<br>Users mailing list<br=
Users@ovirt.org<br>http://lists.ovirt.org/mailman/listinfo/users<br></bloc=
kquote></div></div></body></html>
------=_Part_1725676_1350899779.1481204814198--
2696
days inactive
2788
days old
9 comments
4 participants
participants (4)
-
Gervais de Montbrun -
Logan Kuhn -
Maton, Brett -
Краснобаев Михаил