On Sep 10, 2016, at 12:05 PM, Maton, Brett <matonb@ltresources.co.uk> = wrote: =20 Way-hey! finally the task has gone and I can do 'stuff' with that VM = again. =20 Thanks Gervais, you're a star =20 On 10 September 2016 at 15:40, Maton, Brett <matonb@ltresources.co.uk = <mailto:matonb@ltresources.co.uk>> wrote: Thanks Gervais I'll give that a go =20 On 10 September 2016 at 15:39, Gervais de Montbrun = <gervais@demontbrun.com <mailto:gervais@demontbrun.com>> wrote: Hi Maton, =20 I have seen tasks in a weird state on my cluster also. I've had a vm = get "stuck" during a migration where it says "migrating to" in the web = GUI, but it has finished migrating hours ago... If I click "Cancel = Migraton" the gui tells me that it is not migrating, but I can't do any = action on the vm because I am then told that the vm can't be acted upon = while it is migrating. I also try to kill the task, but there are none =
=20 What has worked for me has been to put my hosted-engine in global =
=20 Cheers, Gervais =20 =20 =20
On Sep 10, 2016, at 11:06 AM, Maton, Brett <matonb@ltresources.co.uk = <mailto:matonb@ltresources.co.uk>> wrote: =20 Anyone know how to fix this broken task ? =20 It's persisted through a reboot of all hosts and the engine, = something needs deleting from the database to clear the task and release =
=20 On 8 September 2016 at 13:25, Maton, Brett <matonb@ltresources.co.uk = <mailto:matonb@ltresources.co.uk>> wrote: Thanks for the pointer Mikhail, however I don't get any tasks listed = with that command: =20 vdsClient -s 0 getAllTasksStatuses =20 /usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses = xmlrpc. since ovirt 3.6 xmlrpc is deprecated, please use = vdsm.jsonrpcvdscli from vdsm import utils, vdscli, constants =20 {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}} =20 =20 On 8 September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1= =D0=B0=D0=B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB <milo1@ya.ru = <mailto:milo1@ya.ru>> wrote: Hi, =20 There is a way to cancel a running task - look here = http://lists.ovirt.org/pipermail/users/2014-November/028946.html = <http://lists.ovirt.org/pipermail/users/2014-November/028946.html> I was able to stop snapshot deletion this way. =20 Best, Mikhail. =20 08.09.2016, 08:14, "Maton, Brett" <matonb@ltresources.co.uk = <mailto:matonb@ltresources.co.uk>>:
Any suggestions ? =20 THe task has been hung for 5 days now, I can't start the machine or = destroy it. =20 =20 On 7 September 2016 at 06:49, Maton, Brett <matonb@ltresources.co.uk = <mailto:matonb@ltresources.co.uk>> wrote: Sorry just hit reply.... =20 I'm seeing these errors in the logs which look related to the =
--Apple-Mail=_EF692F18-F11E-46F1-8425-9C5284E5B3E2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 YAY!! Glad it worked for you. :-) Cheers, Gervais listed maintenance mode, then ssh into the hosted engine and run the = "engine-setup" command. I am not saying the is the best course of = action, but when the engine comes back online the task is cleared. the locked disk problem:
=20 =20 2016-09-07 06:46:35,123 ERROR = [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] = (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method = 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with = exception 'null', the callback is marked for end method retries 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.Com = <http://org.ovirt.engine.core.bll.com/>mandsFactory] = (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command = 'LiveMigrateDisk': null 2016-09-07 06:46:45,185 ERROR = [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] = (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method = 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with = exception 'null', the callback is marked for end method retries =20 On 5 September 2016 at 06:46, Nir Soffer <nsoffer@redhat.com = <mailto:nsoffer@redhat.com>> wrote: Hi Maton, =20 Please reply to the list, not to me directly. =20 Ala, can you look at this? is this a known issue? =20 Thanks, Nir =20 On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett = <matonb@ltresources.co.uk <mailto:matonb@ltresources.co.uk>> wrote:
Log files as requested
https://ufile.io/4fc35 <https://ufile.io/4fc35> vdsm log https://ufile.io/e9836 <https://ufile.io/e9836> engine 03-Sep https://ufile.io/15f37 <https://ufile.io/15f37> engine 04-Sep
vdsm log stops on the 01-Sep...
Couple of entries from the event log:
Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live = Storage Migration' deletion for VM 'lv01' has been completed. Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live = Storage Migration' deletion for VM 'lv01' was initiated by SYSTEM
And the related tasks
Removing Snapshot Auto-generated for Live Storage Migration of VM = lv01 Sep 3, 2016 6:46:44 PM N/A 29f45ca9 Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 = 6:46:44 PM Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 = 7:31:06 PM
Finalizing Sep 3, 2016 7:31:06 PM N/A
On 4 September 2016 at 14:27, Nir Soffer <nsoffer@redhat.com = <mailto:nsoffer@redhat.com>> wrote:
On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett =
<matonb@ltresources.co.uk <mailto:matonb@ltresources.co.uk>>
wrote:
How do I fix / kill a hung vdsm task?
It seems to have completed the task but is stuck finalising.
Removing Snapshot Auto-generated for Live Storage Migration Validating Executing (hour glass) Finalizing
Task has been 'stuck' finalising for over 13 hours
Can you share engine and vdsm logs since the time the merge was = started?
Nir
, _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users = <http://lists.ovirt.org/mailman/listinfo/users>=20 =20 --=20 =D0=A1 =D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=B5=D0=BC, = =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 = =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB. =20 =20 =20 =20
Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users = <http://lists.ovirt.org/mailman/listinfo/users> =20 =20 =20
--Apple-Mail=_EF692F18-F11E-46F1-8425-9C5284E5B3E2 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D"">YAY!! Glad it worked for you.<div class=3D""><br = class=3D""></div><div class=3D"">:-)<br class=3D""><div class=3D""> <div id=3D"signature" class=3D""><br class=3D"">Cheers,<br = class=3D"">Gervais<br class=3D""><br class=3D""><br class=3D""></div> </div> <br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On Sep 10, 2016, at 12:05 PM, Maton, Brett <<a = href=3D"mailto:matonb@ltresources.co.uk" = class=3D"">matonb@ltresources.co.uk</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><div dir=3D"ltr" = class=3D""><div class=3D"">Way-hey! finally the task has gone and I can = do 'stuff' with that VM again.<br class=3D""><br class=3D""></div>Thanks = Gervais, you're a star<br class=3D""></div><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">On 10 September 2016 at 15:40, = Maton, Brett <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" = class=3D"">matonb@ltresources.co.uk</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr" = class=3D"">Thanks Gervais I'll give that a go<br class=3D""></div><div = class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">On 10 September 2016 at 15:39, = Gervais de Montbrun <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:gervais@demontbrun.com" target=3D"_blank" = class=3D"">gervais@demontbrun.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div = style=3D"word-wrap:break-word" class=3D"">Hi Maton,<div class=3D""><br = class=3D""></div><div class=3D"">I have seen tasks in a weird state on = my cluster also. I've had a vm get "stuck" during a migration where it = says "migrating to" in the web GUI, but it has finished migrating hours = ago... If I click "Cancel Migraton" the gui tells me that it is not = migrating, but I can't do any action on the vm because I am then told = that the vm can't be acted upon while it is migrating. I also try to = kill the task, but there are none listed<br class=3D""><div class=3D""><br= class=3D""></div><div class=3D"">What has worked for me has been to put = my hosted-engine in global maintenance mode, then ssh into the hosted = engine and run the "engine-setup" command. I am not saying the is the = best course of action, but when the engine comes back online the task is = cleared.</div><div class=3D""> <div class=3D""><br class=3D"">Cheers,<br class=3D"">Gervais<br = class=3D""><br class=3D""><br class=3D""></div> </div><div class=3D""><div class=3D""> <br class=3D""><div class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On Sep 10, 2016, at 11:06 AM, Maton, Brett <<a = href=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" = class=3D"">matonb@ltresources.co.uk</a>> wrote:</div><br = class=3D""><div class=3D""><div dir=3D"ltr" class=3D""><div = class=3D"">Anyone know how to fix this broken task ?<br class=3D""><br = class=3D""></div><div class=3D"">It's persisted through a reboot of all = hosts and the engine, something needs deleting from the database to = clear the task and release the locked disk<br class=3D""></div></div><div = class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">On 8 = September 2016 at 13:25, Maton, Brett <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" = class=3D"">matonb@ltresources.co.uk</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr" = class=3D"">Thanks for the pointer Mikhail, however I don't get any tasks = listed with that command:<br class=3D""><br class=3D"">vdsClient -s 0 = getAllTasksStatuses<br class=3D""><br = class=3D"">/usr/share/vdsm/vdsClient.py:3<wbr class=3D"">3: = DeprecationWarning: vdscli uses xmlrpc. since ovirt 3.6 xmlrpc is = deprecated, please use vdsm.jsonrpcvdscli<br class=3D""> from vdsm = import utils, vdscli, constants<br class=3D""><br class=3D"">{'status': = {'message': 'OK', 'code': 0}, 'allTasksStatus': {}}<br class=3D""><br = class=3D""></div><div class=3D""><div class=3D""><div = class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">On 8 = September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0= =B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB <span dir=3D"ltr" = class=3D""><<a href=3D"mailto:milo1@ya.ru" target=3D"_blank" = class=3D"">milo1@ya.ru</a>></span> wrote:<br class=3D""><blockquote = class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc = solid;padding-left:1ex"><div class=3D"">Hi,</div><div = class=3D""> </div><div class=3D"">There is a way to cancel a = running task - look here <a = href=3D"http://lists.ovirt.org/pipermail/users/2014-November/028946.html" = target=3D"_blank" class=3D"">http://lists.ovirt.org/piperma<wbr = class=3D"">il/users/2014-November/028946.<wbr = class=3D"">html</a></div><div class=3D"">I was able to stop snapshot = deletion this way.</div><div class=3D""> </div><div class=3D"">Best, = Mikhail.</div><div class=3D""> </div><div class=3D"">08.09.2016, = 08:14, "Maton, Brett" <<a href=3D"mailto:matonb@ltresources.co.uk" = target=3D"_blank" = class=3D"">matonb@ltresources.co.uk</a>>:</div><blockquote = type=3D"cite" class=3D""><div class=3D""><div class=3D""><div = class=3D""><div class=3D""><div class=3D"">Any suggestions ?<br = class=3D""><br class=3D""></div>THe task has been hung for 5 days now, I = can't start the machine or destroy it.<br class=3D""><br = class=3D""></div></div><div class=3D""><br class=3D""><div class=3D"">On = 7 September 2016 at 06:49, Maton, Brett <span class=3D""><<a = href=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" = class=3D"">matonb@ltresources.co.uk</a>></span> wrote:<br = class=3D""><blockquote style=3D"margin:0 0 0 0.8ex;border-left:1px #ccc = solid;padding-left:1ex" class=3D""><div class=3D""><div class=3D""><div = class=3D"">Sorry just hit reply....<br class=3D""><br class=3D""></div>I'm= seeing these errors in the logs which look related to the problem:<br = class=3D""><br class=3D""><br class=3D""><span class=3D"">2016-09-07 = 06</span>:46:35,123 ERROR [org.ovirt.engine.core.bll.tas<wbr = class=3D"">ks.CommandCallbacksPoller] (DefaultQuartzScheduler6) = [19c58c0d] Failed invoking callback end method 'onFailed' for command = '<span class=3D"">07608003</span>-ca05-4e2e-b917-85ce5<wbr = class=3D"">25c011b' with exception 'null', the callback is marked for = end method retries<br class=3D""><span class=3D"">2016-09-07 = 06</span>:46:45,184 ERROR [<a = href=3D"http://org.ovirt.engine.core.bll.com/" target=3D"_blank" = class=3D"">org.ovirt.engine.core.bll.Com</a><wbr class=3D"">mandsFactory] = (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command = 'LiveMigrateDisk': null<br class=3D""><span class=3D"">2016-09-07 = 06</span>:46:45,185 ERROR [org.ovirt.engine.core.bll.tas<wbr = class=3D"">ks.CommandCallbacksPoller] (DefaultQuartzScheduler7) = [19c58c0d] Failed invoking callback end method 'onFailed' for command = '<span class=3D"">07608003</span>-ca05-4e2e-b917-85ce5<wbr = class=3D"">25c011b' with exception 'null', the callback is marked for = end method retries</div><div class=3D""><div class=3D""><div = class=3D""><div class=3D""><div class=3D""><div class=3D""><br = class=3D""><div class=3D"">On 5 September 2016 at 06:46, Nir Soffer = <span class=3D""><<a href=3D"mailto:nsoffer@redhat.com" = target=3D"_blank" class=3D"">nsoffer@redhat.com</a>></span> wrote:<br = class=3D""><blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px = solid #cccccc;padding-left:1ex" class=3D"">Hi Maton,<br class=3D""> <br = class=3D""> Please reply to the list, not to me directly.<br class=3D""> = <br class=3D""> Ala, can you look at this? is this a known issue?<br = class=3D""> <br class=3D""> Thanks,<br class=3D""> Nir<br class=3D""><div = class=3D""><div class=3D""><br class=3D""> On Mon, Sep 5, 2016 at 8:43 = AM, Maton, Brett <<a href=3D"mailto:matonb@ltresources.co.uk" = target=3D"_blank" class=3D"">matonb@ltresources.co.uk</a>> wrote:<br = class=3D""> > Log files as requested<br class=3D""> ><br class=3D"">= > <a href=3D"https://ufile.io/4fc35" target=3D"_blank" = class=3D"">https://ufile.io/4fc35</a> vdsm log<br class=3D""> > <a = href=3D"https://ufile.io/e9836" target=3D"_blank" = class=3D"">https://ufile.io/e9836</a> engine 03-Sep<br class=3D""> > = <a href=3D"https://ufile.io/15f37" target=3D"_blank" = class=3D"">https://ufile.io/15f37</a> engine 04-Sep<br class=3D""> = ><br class=3D""> > vdsm log stops on the 01-Sep...<br class=3D""> = ><br class=3D""> > Couple of entries from the event log:<br = class=3D""> ><br class=3D""> > Sep 3, 2016 7:31:07 PM = Snapshot 'Auto-generated for Live Storage<br class=3D""> > Migration' = deletion for VM 'lv01' has been completed.<br class=3D""> > Sep 3, = 2016 6:46:46 PM Snapshot 'Auto-generated for Live = Storage<br class=3D""> > Migration' deletion for VM 'lv01' was = initiated by SYSTEM<br class=3D""> ><br class=3D""> > And the = related tasks<br class=3D""> ><br class=3D""> > Removing Snapshot = Auto-generated for Live Storage Migration of VM lv01<br class=3D""> > = Sep 3, 2016 6:46:44 PM N/A = 29f45ca9<br class=3D""> > Validating Sep 3, 2016 6:46:44 = PM until Sep 3, 2016 6:46:44 PM<br class=3D""> = > Executing Sep 3, 2016 6:46:44 PM = until Sep 3, 2016 7:31:06 PM<br class=3D""> ><br = class=3D""> > Finalizing Sep 3, 2016 7:31:06 PM = N/A<br class=3D""> ><br class=3D""> ><br = class=3D""> ><br class=3D""> > On 4 September 2016 at 14:27, Nir = Soffer <<a href=3D"mailto:nsoffer@redhat.com" target=3D"_blank" = class=3D"">nsoffer@redhat.com</a>> wrote:<br class=3D""> >><br = class=3D""> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett = <<a href=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" = class=3D"">matonb@ltresources.co.uk</a>><br class=3D""> >> = wrote:<br class=3D""> >>><br class=3D""> >>> How do I = fix / kill a hung vdsm task?<br class=3D""> >>><br class=3D""> = >>> It seems to have completed the task but is stuck = finalising.<br class=3D""> >>><br class=3D""> >>> = Removing Snapshot Auto-generated for Live Storage Migration<br class=3D"">= >>> Validating<br class=3D""> >>> Executing<br = class=3D""> >>> (hour glass) Finalizing<br class=3D""> = >>><br class=3D""> >>> Task has been 'stuck' = finalising for over 13 hours<br class=3D""> >><br class=3D""> = >><br class=3D""> >> Can you share engine and vdsm logs = since the time the merge was started?<br class=3D""> >><br = class=3D""> >> Nir<br class=3D""> ><br class=3D""> = ></div></div></blockquote></div></div></div></div></div></div></div></d= iv></blockquote></div></div></div></div>,<p = class=3D"">______________________________<wbr = class=3D"">_________________<br class=3D"">Users mailing list<br = class=3D""><a href=3D"mailto:Users@ovirt.org" target=3D"_blank" = class=3D"">Users@ovirt.org</a><br class=3D""><a = href=3D"http://lists.ovirt.org/mailman/listinfo/users" target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/users</a></p></blockquote><span class=3D""><font = color=3D"#888888" class=3D""><div class=3D""> </div><div = class=3D""> </div><div class=3D"">-- </div><div class=3D"">=D0=A1= =D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=B5=D0=BC, = =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 = =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB.</div><div = class=3D""> </div><div = class=3D""> </div></font></span></blockquote></div><br = class=3D""></div> </div></div></blockquote></div><br class=3D""></div> ______________________________<wbr class=3D"">_________________<br = class=3D"">Users mailing list<br class=3D""><a = href=3D"mailto:Users@ovirt.org" target=3D"_blank" = class=3D"">Users@ovirt.org</a><br class=3D""><a = href=3D"http://lists.ovirt.org/mailman/listinfo/users" target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/users</a><br class=3D""></div></blockquote></div><br = class=3D""></div></div></div></div></blockquote></div><br = class=3D""></div> </div></div></blockquote></div><br class=3D""></div> </div></blockquote></div><br class=3D""></div></body></html>= --Apple-Mail=_EF692F18-F11E-46F1-8425-9C5284E5B3E2--