9:39 a.m.
--Apple-Mail=_BE696610-4EC6-4158-8D67-989CBCD17D96
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
Hi Maton,
I have seen tasks in a weird state on my cluster also. I've had a vm get =
"stuck" during a migration where it says "migrating to" in the web
GUI, =
but it has finished migrating hours ago... If I click "Cancel Migraton" =
the gui tells me that it is not migrating, but I can't do any action on =
the vm because I am then told that the vm can't be acted upon while it =
is migrating. I also try to kill the task, but there are none listed
What has worked for me has been to put my hosted-engine in global =
maintenance mode, then ssh into the hosted engine and run the =
"engine-setup" command. I am not saying the is the best course of =
action, but when the engine comes back online the task is cleared.
Cheers,
Gervais
On Sep 10, 2016, at 11:06 AM, Maton, Brett
<matonb(a)ltresources.co.uk> =
wrote:
=20
Anyone know how to fix this broken task ?
=20
It's persisted through a reboot of all hosts and the engine, something =
needs
deleting from the database to clear the task and release the =
locked disk
=20
On 8 September 2016 at 13:25, Maton, Brett <matonb(a)ltresources.co.uk =
<mailto:matonb@ltresources.co.uk>> wrote:
Thanks for the pointer Mikhail, however I don't get any tasks
listed =
with that command:
=20
vdsClient -s 0 getAllTasksStatuses
=20
/usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses =
xmlrpc. since
ovirt 3.6 xmlrpc is deprecated, please use =
vdsm.jsonrpcvdscli
from vdsm import utils, vdscli, constants
=20
{'status': {'message': 'OK', 'code': 0},
'allTasksStatus': {}}
=20
=20
On 8 September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=
=D0=B0=D0=B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB <milo1(a)ya.ru =
<mailto:milo1@ya.ru>> wrote:
Hi,
=20
There is a way to cancel a running task - look here =
http://lists.ovirt.org/pipermail/users/2014-November/028946.html =
<http://lists.ovirt.org/pipermail/users/2014-November/028946.html>
I was able to stop snapshot deletion this way.
=20
Best, Mikhail.
=20
08.09.2016, 08:14, "Maton, Brett" <matonb(a)ltresources.co.uk =
<mailto:matonb@ltresources.co.uk>>:
> Any suggestions ?
>=20
> THe task has been hung for 5 days now, I can't start the machine or =
destroy it.
>=20
>=20
> On 7 September 2016 at 06:49, Maton, Brett <matonb(a)ltresources.co.uk =
<mailto:matonb@ltresources.co.uk>> wrote:
> Sorry just hit reply....
>=20
> I'm seeing these errors in the logs which look related to the =
problem:
>=20
>=20
> 2016-09-07 06:46:35,123 ERROR =
[org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] =
(DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method =
'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with =
exception 'null', the callback is marked for end method retries
> 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.Com =
<http://org.ovirt.engine.core.bll.com/>mandsFactory] =
(DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command =
'LiveMigrateDisk': null
> 2016-09-07 06:46:45,185 ERROR =
[org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] =
(DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method =
'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with =
exception 'null', the callback is marked for end method retries
>=20
> On 5 September 2016 at 06:46, Nir Soffer <nsoffer(a)redhat.com =
<mailto:nsoffer@redhat.com>> wrote:
> Hi Maton,
>=20
> Please reply to the list, not to me directly.
>=20
> Ala, can you look at this? is this a known issue?
>=20
> Thanks,
> Nir
>=20
> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett =
<matonb(a)ltresources.co.uk
<mailto:matonb@ltresources.co.uk>> wrote:
> > Log files as requested
> >
> > https://ufile.io/4fc35 <https://ufile.io/4fc35> vdsm log
> > https://ufile.io/e9836 <https://ufile.io/e9836> engine 03-Sep
> > https://ufile.io/15f37 <https://ufile.io/15f37> engine 04-Sep
> >
> > vdsm log stops on the 01-Sep...
> >
> > Couple of entries from the event log:
> >
> > Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage
> > Migration' deletion for VM 'lv01' has been completed.
> > Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage
> > Migration' deletion for VM 'lv01' was initiated by SYSTEM
> >
> > And the related tasks
> >
> > Removing Snapshot Auto-generated for Live Storage Migration of VM =
lv01
> > Sep 3, 2016 6:46:44 PM N/A 29f45ca9
> > Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 =
6:46:44 PM
> > Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016
7:31:06 =
PM
> >
> > Finalizing Sep 3, 2016 7:31:06 PM N/A
> >
> >
> >
> > On 4 September 2016 at 14:27, Nir Soffer <nsoffer(a)redhat.com =
<mailto:nsoffer@redhat.com>> wrote:
> >>
> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett =
<matonb(a)ltresources.co.uk <mailto:matonb@ltresources.co.uk>>
> >> wrote:
> >>>
> >>> How do I fix / kill a hung vdsm task?
> >>>
> >>> It seems to have completed the task but is stuck finalising.
> >>>
> >>> Removing Snapshot Auto-generated for Live Storage Migration
> >>> Validating
> >>> Executing
> >>> (hour glass) Finalizing
> >>>
> >>> Task has been 'stuck' finalising for over 13 hours
> >>
> >>
> >> Can you share engine and vdsm logs since the time the merge was =
started?
> >>
> >> Nir
> >
> >
> ,
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>=20
=20
--=20
=D0=A1 =D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=B5=D0=BC, =
=D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 =
=D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB.
=20
=20
=20
=20
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_BE696610-4EC6-4158-8D67-989CBCD17D96
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D"">Hi Maton,<div class=3D""><br
class=3D""></div><div class=3D"">I=
have seen tasks in a weird state on my cluster also. I've had a vm get =
"stuck" during a migration where it says "migrating to" in the web
GUI, =
but it has finished migrating hours ago... If I click "Cancel Migraton" =
the gui tells me that it is not migrating, but I can't do any action on =
the vm because I am then told that the vm can't be acted upon while it =
is migrating. I also try to kill the task, but there are none listed<br =
class=3D""><div class=3D""><br =
class=3D"webkit-block-placeholder"></div><div
class=3D"">What has worked =
for me has been to put my hosted-engine in global maintenance mode, then =
ssh into the hosted engine and run the "engine-setup" command. I am not =
saying the is the best course of action, but when the engine comes back =
online the task is cleared.</div><div class=3D"">
<div id=3D"signature" class=3D""><br
class=3D"">Cheers,<br =
class=3D"">Gervais<br class=3D""><br
class=3D""><br class=3D""></div>
</div>
<br class=3D""><div><blockquote type=3D"cite"
class=3D""><div =
class=3D"">On Sep 10, 2016, at 11:06 AM, Maton, Brett <<a =
href=3D"mailto:matonb@ltresources.co.uk" =
class=3D"">matonb(a)ltresources.co.uk</a>&gt;
wrote:</div><br =
class=3D"Apple-interchange-newline"><div class=3D""><div
dir=3D"ltr" =
class=3D""><div class=3D"">Anyone know how to fix this broken
task ?<br =
class=3D""><br class=3D""></div><div
class=3D"">It's persisted through a =
reboot of all hosts and the engine, something needs deleting from the =
database to clear the task and release the locked disk<br =
class=3D""></div></div><div
class=3D"gmail_extra"><br class=3D""><div =
class=3D"gmail_quote">On 8 September 2016 at 13:25, Maton, Brett <span =
dir=3D"ltr" class=3D""><<a
href=3D"mailto:matonb@ltresources.co.uk" =
target=3D"_blank"
class=3D"">matonb(a)ltresources.co.uk</a>&gt;</span> =
wrote:<br class=3D""><blockquote class=3D"gmail_quote"
style=3D"margin:0 =
0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div
dir=3D"ltr" =
class=3D"">Thanks for the pointer Mikhail, however I don't get any tasks
=
listed with that command:<br class=3D""><br
class=3D"">vdsClient -s 0 =
getAllTasksStatuses<br class=3D""><br =
class=3D"">/usr/share/vdsm/vdsClient.py:<wbr class=3D"">33:
=
DeprecationWarning: vdscli uses xmlrpc. since ovirt 3.6 xmlrpc is =
deprecated, please use vdsm.jsonrpcvdscli<br class=3D""> from
vdsm =
import utils, vdscli, constants<br class=3D""><br
class=3D"">{'status': =
{'message': 'OK', 'code': 0}, 'allTasksStatus': {}}<br
class=3D""><br =
class=3D""></div><div class=3D"HOEnZb"><div
class=3D"h5"><div =
class=3D"gmail_extra"><br class=3D""><div
class=3D"gmail_quote">On 8 =
September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=
=B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB <span dir=3D"ltr" =
class=3D""><<a href=3D"mailto:milo1@ya.ru"
target=3D"_blank" =
class=3D"">milo1(a)ya.ru</a>&gt;</span> wrote:<br
class=3D""><blockquote =
class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc =
solid;padding-left:1ex"><div class=3D"">Hi,</div><div =
class=3D""> </div><div class=3D"">There is a
way to cancel a =
running task - look here <a =
href=3D"http://lists.ovirt.org/pipermail/users/2014-November/028946.... =
target=3D"_blank" class=3D"">http://lists.ovirt.org/piperma<wbr
=
class=3D"">il/users/2014-November/028946.<wbr =
class=3D"">html</a></div><div class=3D"">I was
able to stop snapshot =
deletion this way.</div><div
class=3D""> </div><div class=3D"">Best, =
Mikhail.</div><div class=3D""> </div><div
class=3D"">08.09.2016, =
08:14, "Maton, Brett" <<a
href=3D"mailto:matonb@ltresources.co.uk" =
target=3D"_blank" =
class=3D"">matonb@ltresources.co.uk</a>>:</div><blockquote
=
type=3D"cite" class=3D""><div class=3D""><div
class=3D""><div =
class=3D""><div class=3D""><div
class=3D"">Any suggestions ?<br =
class=3D""><br class=3D""></div>THe task has been hung
for 5 days now, I =
can't start the machine or destroy it.<br class=3D""><br =
class=3D""></div></div><div class=3D""><br
class=3D""><div class=3D"">On =
7 September 2016 at 06:49, Maton, Brett <span class=3D""><<a =
href=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" =
class=3D"">matonb(a)ltresources.co.uk</a>&gt;</span>
wrote:<br =
class=3D""><blockquote style=3D"margin:0 0 0 0.8ex;border-left:1px
#ccc =
solid;padding-left:1ex" class=3D""><div
class=3D""><div class=3D""><div =
class=3D"">Sorry just hit reply....<br class=3D""><br
class=3D""></div>I'm=
seeing these errors in the logs which look related to the problem:<br =
class=3D""><br class=3D""><br
class=3D""><span class=3D"">2016-09-07 =
06</span>:46:35,123 ERROR [org.ovirt.engine.core.bll.tas<wbr =
class=3D"">ks.CommandCallbacksPoller] (DefaultQuartzScheduler6) =
[19c58c0d] Failed invoking callback end method 'onFailed' for command =
'<span class=3D"">07608003</span>-ca05-4e2e-b917-85ce5<wbr =
class=3D"">25c011b' with exception 'null', the callback is marked
for =
end method retries<br class=3D""><span
class=3D"">2016-09-07 =
06</span>:46:45,184 ERROR [<a =
href=3D"http://org.ovirt.engine.core.bll.com/" =
class=3D"">org.ovirt.engine.core.bll.Com</a><wbr
class=3D"">mandsFactory] =
(DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command =
'LiveMigrateDisk': null<br class=3D""><span
class=3D"">2016-09-07 =
06</span>:46:45,185 ERROR [org.ovirt.engine.core.bll.tas<wbr =
class=3D"">ks.CommandCallbacksPoller] (DefaultQuartzScheduler7) =
[19c58c0d] Failed invoking callback end method 'onFailed' for command =
'<span class=3D"">07608003</span>-ca05-4e2e-b917-85ce5<wbr =
class=3D"">25c011b' with exception 'null', the callback is marked
for =
end method retries</div><div class=3D""><div
class=3D""><div =
class=3D""><div class=3D""><div
class=3D""><div class=3D""><br =
class=3D""><div class=3D"">On 5 September 2016 at 06:46, Nir
Soffer =
<span class=3D""><<a
href=3D"mailto:nsoffer@redhat.com" =
target=3D"_blank"
class=3D"">nsoffer(a)redhat.com</a>&gt;</span> wrote:<br =
class=3D""><blockquote style=3D"margin:0px 0px 0px
0.8ex;border-left:1px =
solid #cccccc;padding-left:1ex" class=3D"">Hi Maton,<br
class=3D""> <br =
class=3D""> Please reply to the list, not to me directly.<br
class=3D""> =
<br class=3D""> Ala, can you look at this? is this a known issue?<br =
class=3D""> <br class=3D""> Thanks,<br
class=3D""> Nir<br class=3D""><div =
class=3D""><div class=3D""><br class=3D""> On
Mon, Sep 5, 2016 at 8:43 =
AM, Maton, Brett <<a href=3D"mailto:matonb@ltresources.co.uk" =
target=3D"_blank"
class=3D"">matonb(a)ltresources.co.uk</a>&gt; wrote:<br =
class=3D""> > Log files as requested<br class=3D"">
><br class=3D"">=
> <a href=3D"https://ufile.io/4fc35" target=3D"_blank" =
class=3D"">https://ufile.io/4fc35</a> vdsm log<br
class=3D""> > <a =
href=3D"https://ufile.io/e9836" target=3D"_blank" =
class=3D"">https://ufile.io/e9836</a> engine 03-Sep<br
class=3D""> > =
<a href=3D"https://ufile.io/15f37" target=3D"_blank" =
class=3D"">https://ufile.io/15f37</a> engine 04-Sep<br
class=3D""> =
><br class=3D""> > vdsm log stops on the 01-Sep...<br
class=3D""> =
><br class=3D""> > Couple of entries from the event
log:<br =
class=3D""> ><br class=3D""> > Sep 3, 2016
7:31:07 PM =
Snapshot 'Auto-generated for Live Storage<br class=3D""> >
Migration' =
deletion for VM 'lv01' has been completed.<br class=3D""> >
Sep 3, =
2016 6:46:46 PM Snapshot 'Auto-generated for Live =
Storage<br class=3D""> > Migration' deletion for VM
'lv01' was =
initiated by SYSTEM<br class=3D""> ><br class=3D"">
> And the =
related tasks<br class=3D""> ><br class=3D"">
> Removing Snapshot =
Auto-generated for Live Storage Migration of VM lv01<br class=3D"">
> =
Sep 3, 2016 6:46:44 PM N/A
=
29f45ca9<br class=3D""> > Validating Sep 3,
2016 6:46:44 =
PM until Sep 3, 2016 6:46:44 PM<br
class=3D""> =
> Executing Sep 3, 2016 6:46:44 PM =
until Sep 3, 2016 7:31:06 PM<br class=3D"">
><br =
class=3D""> > Finalizing Sep 3, 2016 7:31:06
PM =
N/A<br class=3D""> ><br
class=3D""> ><br =
class=3D""> ><br class=3D""> > On 4 September
2016 at 14:27, Nir =
Soffer <<a href=3D"mailto:nsoffer@redhat.com"
target=3D"_blank" =
class=3D"">nsoffer(a)redhat.com</a>&gt; wrote:<br
class=3D""> >><br =
class=3D""> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett =
<<a href=3D"mailto:matonb@ltresources.co.uk"
target=3D"_blank" =
class=3D"">matonb(a)ltresources.co.uk</a>&gt;<br
class=3D""> >> =
wrote:<br class=3D""> >>><br
class=3D""> >>> How do I =
fix / kill a hung vdsm task?<br class=3D""> >>><br
class=3D""> =
>>> It seems to have completed the task but is stuck =
finalising.<br class=3D""> >>><br
class=3D""> >>> =
Removing Snapshot Auto-generated for Live Storage Migration<br
class=3D"">=
>>> Validating<br class=3D"">
>>> Executing<br =
class=3D""> >>> (hour glass) Finalizing<br
class=3D""> =
>>><br class=3D""> >>> Task has
been 'stuck' =
finalising for over 13 hours<br class=3D""> >><br
class=3D""> =
>><br class=3D""> >> Can you share engine and
vdsm logs =
since the time the merge was started?<br class=3D"">
>><br =
class=3D""> >> Nir<br class=3D"">
><br class=3D""> =
></div></div></blockquote></div></div></div></div></div></div></div></d=
iv></blockquote></div></div></div></div>,<p =
class=3D"">______________________________<wbr =
class=3D"">_________________<br class=3D"">Users mailing
list<br =
class=3D""><a href=3D"mailto:Users@ovirt.org"
target=3D"_blank" =
class=3D"">Users(a)ovirt.org</a><br class=3D""><a =
href=3D"http://lists.ovirt.org/mailman/listinfo/users"
target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman<wbr =
class=3D"">/listinfo/users</a></p></blockquote><span
class=3D""><font =
color=3D"#888888" class=3D""><div
class=3D""> </div><div =
class=3D""> </div><div
class=3D"">-- </div><div class=3D"">=D0=A1=
=D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=B5=D0=BC, =
=D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 =
=D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB.</div><div =
class=3D""> </div><div =
class=3D""> </div></font></span></blockquote></div><br
=
class=3D""></div>
</div></div></blockquote></div><br
class=3D""></div>
_______________________________________________<br class=3D"">Users =
mailing list<br class=3D""><a href=3D"mailto:Users@ovirt.org"
=
class=3D"">Users(a)ovirt.org</a><br =
class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br =
class=3D""></div></blockquote></div><br
class=3D""></div></body></html>=
--Apple-Mail=_BE696610-4EC6-4158-8D67-989CBCD17D96--