Re: [ovirt-users] Hung task finalizing live migration

8 Dec 2016

      ------=_Part_1725676_1350899779.1481204814198
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

I had a similar situation where I was attempting and failing to delete a ci=
nder disk and this fixed it. I'm using 4.0.5-5=20

Regards,=20
Logan=20

----- On Sep 10, 2016, at 9:39 AM, Gervais de Montbrun <gervais@demontbrun.=
com> wrote:=20

| Hi Maton,

| I have seen tasks in a weird state on my cluster also. I've had a vm get =
"stuck"
| during a migration where it says "migrating to" in the web GUI, but it ha=
s
| finished migrating hours ago... If I click "Cancel Migraton" the gui tell=
s me
| that it is not migrating, but I can't do any action on the vm because I a=
m then
| told that the vm can't be acted upon while it is migrating. I also try to=
 kill
| the task, but there are none listed

| What has worked for me has been to put my hosted-engine in global mainten=
ance
| mode, then ssh into the hosted engine and run the "engine-setup" command.=
 I am
| not saying the is the best course of action, but when the engine comes ba=
ck
| online the task is cleared.

| Cheers,
| Gervais

|| On Sep 10, 2016, at 11:06 AM, Maton, Brett < matonb@ltresources.co.uk > =
wrote:

|| Anyone know how to fix this broken task ?

|| It's persisted through a reboot of all hosts and the engine, something n=
eeds
|| deleting from the database to clear the task and release the locked disk

|| On 8 September 2016 at 13:25, Maton, Brett < matonb@ltresources.co.uk > =
wrote:

||| Thanks for the pointer Mikhail, however I don't get any tasks listed wi=
th that
||| command:

||| vdsClient -s 0 getAllTasksStatuses

||| /usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlrpc=
. since
||| ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli
||| from vdsm import utils, vdscli, constants

||| {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}}

||| On 8 September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=
=B1=D0=B0=D0=B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB < milo1@ya.ru > =
wrote:

|||| Hi,
|||| There is a way to cancel a running task - look here
|||| http://lists.ovirt.org/pipermail/users/2014-November/028946.html
|||| I was able to stop snapshot deletion this way.
|||| Best, Mikhail.
|||| 08.09.2016, 08:14, "Maton, Brett" < matonb@ltresources.co.uk >:

||||| Any suggestions ?

||||| THe task has been hung for 5 days now, I can't start the machine or d=
estroy it.

||||| On 7 September 2016 at 06:49, Maton, Brett < matonb@ltresources.co.uk=
...
wrote:
...
<br></div><div data-marker=3D"__SIG_PRE__">Regards,<br>Logan</div><br><spa=
n id=3D"zwchr" data-marker=3D"__DIVIDER__">----- On Sep 10, 2016, at 9:39 A=
M, Gervais de Montbrun <gervais@demontbrun.com> wrote:<br></span><div=
 data-marker=3D"__QUOTED_TEXT__"><blockquote style=3D"border-left: 2px soli=
d #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: n=
ormal; font-style: normal; text-decoration: none; font-family: Helvetica,Ar=
ial,sans-serif; font-size: 12pt;" data-mce-style=3D"border-left: 2px solid =
#1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: nor=
mal; font-style: normal; text-decoration: none; font-family: Helvetica,Aria=
l,sans-serif; font-size: 12pt;">Hi Maton,<div class=3D""><br class=3D""></d=
iv><div class=3D"">I have seen tasks in a weird state on my cluster also. I=
've had a vm get "stuck" during a migration where it says "migrating to" in=
...
Thanks,<br class=3D""> Nir<br class=3D""><div class=3D""><div class=3D"">=
<br class=3D""> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett <<a href=3D=
"mailto:matonb@ltresources.co.uk" target=3D"_blank" class=3D"">matonb@ltres=
ources.co.uk</a>> wrote:<br class=3D""> > Log files as requested<br c=
lass=3D""> ><br class=3D""> > <a href=3D"https://ufile.io/4fc35" targ=
et=3D"_blank" class=3D"">https://ufile.io/4fc35</a> vdsm log<br class=3D"">=
 > <a href=3D"https://ufile.io/e9836" target=3D"_blank" class=3D"">https=
://ufile.io/e9836</a> engine 03-Sep<br class=3D""> > <a href=3D"https://=
ufile.io/15f37" target=3D"_blank" class=3D"">https://ufile.io/15f37</a> eng=
ine 04-Sep<br class=3D""> ><br class=3D""> > vdsm log stops on the 01=
-Sep...<br class=3D""> ><br class=3D""> > Couple of entries from the =
event log:<br class=3D""> ><br class=3D""> > Sep 3, 2016 7:31:07 PM&n=
bsp;   Snapshot 'Auto-generated for Live Storage<br class=3D""> > M=
igration' deletion for VM 'lv01' has been completed.<br class=3D""> > Se=
...
>>> Removing Snapshot Auto-generated for Live Storage Migration<=
br class=3D""> >>> Validating<br class=3D""> >>> Executin=
g<br class=3D""> >>> (hour glass) Finalizing<br class=3D""> >&g=
t;><br class=3D""> >>> Task has been 'stuck' finalising for ove=
r 13 hours<br class=3D""> >><br class=3D""> >><br class=3D""> &=
gt;> Can you share engine and vdsm logs since the time the merge was sta=
rted?<br class=3D""> >><br class=3D""> >> Nir<br class=3D""> &g=
t;<br class=3D""> ></div></div></blockquote></div></div></div></div></di=
v></div></div></div></blockquote></div></div></div></div>,<p class=3D"">___=
____________________________________________<br class=3D"">Users mailing li=
st<br class=3D""><a href=3D"mailto:Users@ovirt.org" target=3D"_blank" class=
=3D"">Users@ovirt.org</a><br class=3D""><a href=3D"http://lists.ovirt.org/m=
ailman/listinfo/users" target=3D"_blank" class=3D"">http://lists.ovirt.org/=
mailman/listinfo/users</a></p></blockquote><span class=3D""><span color=3D"=
#888888" data-mce-style=3D"color: #888888;" style=3D"color: #888888;"><div =
class=3D""> </div><div class=3D""> </div><div class=3D"">-- =
</div><div class=3D"">=D0=A1 =D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=
=B5=D0=BC, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 =D0=
=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB.</div><div class=3D""> </div><div cl=
ass=3D""> </div></span></span></blockquote></div><br class=3D""></div>
</div></div></blockquote></div><br class=3D""></div>
_______________________________________________<br class=3D"">Users mailing=
|||||| Sorry just hit reply....

|||||| I'm seeing these errors in the logs which look related to the proble=
m:

|||||| 2016-09-07 06 :46:35,123 ERROR
|||||| [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
|||||| (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end me=
thod
|||||| 'onFailed' for command ' 07608003 -ca05-4e2e-b917-85ce525c011b' with=
 exception
|||||| 'null', the callback is marked for end method retries
|||||| 2016-09-07 06 :46:45,184 ERROR [ org.ovirt.engine.core.bll.Com mands=
Factory]
|||||| (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of com=
mand
|||||| 'LiveMigrateDisk': null
|||||| 2016-09-07 06 :46:45,185 ERROR
|||||| [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
|||||| (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end me=
thod
|||||| 'onFailed' for command ' 07608003 -ca05-4e2e-b917-85ce525c011b' with=
 exception
|||||| 'null', the callback is marked for end method retries

|||||| On 5 September 2016 at 06:46, Nir Soffer < nsoffer@redhat.com > wrot=
e:

||||||| Hi Maton,

||||||| Please reply to the list, not to me directly.

||||||| Ala, can you look at this? is this a known issue?

||||||| Thanks,
||||||| Nir

||||||| On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett < matonb@ltresources.c=
o.uk > wrote:
||||||| > Log files as requested

||||||| > https://ufile.io/4fc35 vdsm log
||||||| > https://ufile.io/e9836 engine 03-Sep
||||||| > https://ufile.io/15f37 engine 04-Sep

||||||| > vdsm log stops on the 01-Sep...

||||||| > Couple of entries from the event log:

||||||| > Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage
||||||| > Migration' deletion for VM 'lv01' has been completed.
||||||| > Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage
||||||| > Migration' deletion for VM 'lv01' was initiated by SYSTEM

||||||| > And the related tasks

||||||| > Removing Snapshot Auto-generated for Live Storage Migration of VM=
 lv01
||||||| > Sep 3, 2016 6:46:44 PM N/A 29f45ca9
||||||| > Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 6:46:44 PM
||||||| > Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 PM

||||||| > Finalizing Sep 3, 2016 7:31:06 PM N/A

||||||| > On 4 September 2016 at 14:27, Nir Soffer < nsoffer@redhat.com > w=
rote:

||||||| >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett < matonb@ltresourc=
es.co.uk >
||||||| >> wrote:

||||||| >>> How do I fix / kill a hung vdsm task?

||||||| >>> It seems to have completed the task but is stuck finalising.

||||||| >>> Removing Snapshot Auto-generated for Live Storage Migration
||||||| >>> Validating
||||||| >>> Executing
||||||| >>> (hour glass) Finalizing

||||||| >>> Task has been 'stuck' finalising for over 13 hours

||||||| >> Can you share engine and vdsm logs since the time the merge was =
started?

||||||| >> Nir

||||| ,

||||| _______________________________________________
||||| Users mailing list
||||| Users@ovirt.org
||||| http://lists.ovirt.org/mailman/listinfo/users
|||| --
|||| =D0=A1 =D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=B5=D0=BC, =D0=9A=
=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 =D0=9C=D0=B8=D1=85=
=D0=B0=D0=B8=D0=BB.

|| _______________________________________________
|| Users mailing list
|| Users@ovirt.org
|| http://lists.ovirt.org/mailman/listinfo/users

| _______________________________________________
| Users mailing list
| Users@ovirt.org
| http://lists.ovirt.org/mailman/listinfo/users

------=_Part_1725676_1350899779.1481204814198
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><body><div style=3D"font-family: Arial; font-size: 12pt; color: #0000=
00"><div>I had a similar situation where I was attempting and failing to de=
lete a cinder disk and this fixed it.  I'm using 4.0.5-5<br></div><div=
 the web GUI, but it has finished migrating hours ago... If I click "Cancel=
 Migraton" the gui tells me that it is not migrating, but I can't do any ac=
tion on the vm because I am then told that the vm can't be acted upon while=
 it is migrating. I also try to kill the task, but there are none listed<br=
 class=3D""><div class=3D""><br class=3D"webkit-block-placeholder"></div><d=
iv class=3D"">What has worked for me has been to put my hosted-engine in gl=
obal maintenance mode, then ssh into the hosted engine and run the "engine-=
setup" command. I am not saying the is the best course of action, but when =
the engine comes back online the task is cleared.</div><div class=3D"">
<div id=3D"signature" class=3D""><br class=3D"">Cheers,<br class=3D"">Gerva=
is<br class=3D""><br class=3D""><br class=3D""></div>

</div>
<br class=3D""><div><blockquote class=3D""><div class=3D"">On Sep 10, 2016,=
 at 11:06 AM, Maton, Brett <<a href=3D"mailto:matonb@ltresources.co.uk" =
class=3D"" target=3D"_blank">matonb@ltresources.co.uk</a>> wrote:</div><=
br class=3D"Apple-interchange-newline"><div class=3D""><div dir=3D"ltr" cla=
ss=3D""><div class=3D"">Anyone know how to fix this broken task ?<br class=
=3D""><br class=3D""></div><div class=3D"">It's persisted through a reboot =
of all hosts and the engine, something needs deleting from the database to =
clear the task and release the locked disk<br class=3D""></div></div><div c=
lass=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">On 8 Septemb=
er 2016 at 13:25, Maton, Brett <span dir=3D"ltr" class=3D""><<a href=3D"=
mailto:matonb@ltresources.co.uk" target=3D"_blank" class=3D"">matonb@ltreso=
urces.co.uk</a>></span> wrote:<br class=3D""><blockquote class=3D"gmail_=
quote" style=3D"margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-le=
ft: 1ex;" data-mce-style=3D"margin: 0 0 0 .8ex; border-left: 1px #ccc solid=
; padding-left: 1ex;"><div dir=3D"ltr" class=3D"">Thanks for the pointer Mi=
khail, however I don't get any tasks listed with that command:<br class=3D"=
"><br class=3D"">vdsClient -s 0 getAllTasksStatuses<br class=3D""><br class=
=3D"">/usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlr=
pc. since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli<br =
class=3D"">  from vdsm import utils, vdscli, constants<br class=3D""><=
br class=3D"">{'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}=
}<br class=3D""><br class=3D""></div><div class=3D"HOEnZb"><div class=3D"h5=
"><div class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">On 8=
 September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=
=D0=B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB <span dir=3D"ltr" class=
=3D""><<a href=3D"mailto:milo1@ya.ru" target=3D"_blank" class=3D"">milo1=
@ya.ru</a>></span> wrote:<br class=3D""><blockquote class=3D"gmail_quote=
" style=3D"margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1=
ex;" data-mce-style=3D"margin: 0 0 0 .8ex; border-left: 1px #ccc solid; pad=
ding-left: 1ex;"><div class=3D"">Hi,</div><div class=3D""> </div><div =
class=3D"">There is a way to cancel a running task  -  look here =
<a href=3D"http://lists.ovirt.org/pipermail/users/2014-November/028946.html=
" target=3D"_blank" class=3D"">http://lists.ovirt.org/pipermail/users/2014-=
November/028946.html</a></div><div class=3D"">I was able to stop snapshot d=
eletion this way.</div><div class=3D""> </div><div class=3D"">Best, Mi=
khail.</div><div class=3D""> </div><div class=3D"">08.09.2016, 08:14, =
"Maton, Brett" <<a href=3D"mailto:matonb@ltresources.co.uk" target=3D"_b=
lank" class=3D"">matonb@ltresources.co.uk</a>>:</div><blockquote class=
=3D""><div class=3D""><div class=3D""><div class=3D""><div class=3D""><div =
class=3D"">Any suggestions ?<br class=3D""><br class=3D""></div>THe task ha=
s been hung for 5 days now, I can't start the machine or destroy it.<br cla=
ss=3D""><br class=3D""></div></div><div class=3D""><br class=3D""><div clas=
s=3D"">On 7 September 2016 at 06:49, Maton, Brett <span class=3D""><<a h=
ref=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" class=3D"">matonb=
@ltresources.co.uk</a>></span> wrote:<br class=3D""><blockquote style=3D=
"margin: 0 0 0 0.8ex; border-left: 1px #ccc solid; padding-left: 1ex;" clas=
s=3D"" data-mce-style=3D"margin: 0 0 0 0.8ex; border-left: 1px #ccc solid; =
padding-left: 1ex;"><div class=3D""><div class=3D""><div class=3D"">Sorry j=
ust hit reply....<br class=3D""><br class=3D""></div>I'm seeing these error=
s in the logs which look related to the problem:<br class=3D""><br class=3D=
""><br class=3D""><span class=3D"">2016-09-07 06</span>:46:35,123 ERROR [or=
g.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzSchedul=
er6) [19c58c0d] Failed invoking callback end method 'onFailed' for command =
'<span class=3D"">07608003</span>-ca05-4e2e-b917-85ce525c011b' with excepti=
on 'null', the callback is marked for end method retries<br class=3D""><spa=
n class=3D"">2016-09-07 06</span>:46:45,184 ERROR [<a href=3D"http://org.ov=
irt.engine.core.bll.com/" class=3D"" target=3D"_blank">org.ovirt.engine.cor=
e.bll.Com</a>mandsFactory] (DefaultQuartzScheduler7) [19c58c0d] Error in in=
vocating CTOR of command 'LiveMigrateDisk': null<br class=3D""><span class=
=3D"">2016-09-07 06</span>:46:45,185 ERROR [org.ovirt.engine.core.bll.tasks=
.CommandCallbacksPoller] (DefaultQuartzScheduler7) [19c58c0d] Failed invoki=
ng callback end method 'onFailed' for command '<span class=3D"">07608003</s=
pan>-ca05-4e2e-b917-85ce525c011b' with exception 'null', the callback is ma=
rked for end method retries</div><div class=3D""><div class=3D""><div class=
=3D""><div class=3D""><div class=3D""><div class=3D""><br class=3D""><div c=
lass=3D"">On 5 September 2016 at 06:46, Nir Soffer <span class=3D""><<a =
href=3D"mailto:nsoffer@redhat.com" target=3D"_blank" class=3D"">nsoffer@red=
hat.com</a>></span> wrote:<br class=3D""><blockquote style=3D"margin: 0p=
x 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;" class=
=3D"" data-mce-style=3D"margin: 0px 0px 0px 0.8ex; border-left: 1px solid #=
cccccc; padding-left: 1ex;">Hi Maton,<br class=3D""> <br class=3D""> Please=
 reply to the list, not to me directly.<br class=3D""> <br class=3D""> Ala,=
 can you look at this? is this a known issue?<br class=3D""> <br class=3D""=
p 3, 2016 6:46:46 PM    Snapshot 'Auto-generated for Live Storage=
<br class=3D""> > Migration' deletion for VM 'lv01' was initiated by SYS=
TEM<br class=3D""> ><br class=3D""> > And the related tasks<br class=
=3D""> ><br class=3D""> > Removing Snapshot Auto-generated for Live S=
torage Migration of VM lv01<br class=3D""> > Sep 3, 2016 6:46:44 PM =
;       N/A    29f45ca9<br class=3D""> > Valida=
ting    Sep 3, 2016 6:46:44 PM    until    Se=
p 3, 2016 6:46:44 PM<br class=3D""> > Executing    Sep 3, 2016=
 6:46:44 PM    until    Sep 3, 2016 7:31:06 PM<br class=
=3D""> ><br class=3D""> > Finalizing    Sep 3, 2016 7:31:06=
 PM        N/A<br class=3D""> ><br class=3D""> ><=
br class=3D""> ><br class=3D""> > On 4 September 2016 at 14:27, Nir S=
offer <<a href=3D"mailto:nsoffer@redhat.com" target=3D"_blank" class=3D"=
">nsoffer@redhat.com</a>> wrote:<br class=3D""> >><br class=3D""> =
>> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <<a href=3D"mailt=
o:matonb@ltresources.co.uk" target=3D"_blank" class=3D"">matonb@ltresources=
.co.uk</a>><br class=3D""> >> wrote:<br class=3D""> >>><b=
r class=3D""> >>> How do I fix / kill a hung vdsm task?<br class=
=3D""> >>><br class=3D""> >>> It seems to have completed =
the task but is stuck finalising.<br class=3D""> >>><br class=3D""=
 list<br class=3D""><a href=3D"mailto:Users@ovirt.org" class=3D"" target=3D=
"_blank">Users@ovirt.org</a><br class=3D"">http://lists.ovirt.org/mailman/l=
istinfo/users<br class=3D""></div></blockquote></div><br class=3D""></div><=
br>_______________________________________________<br>Users mailing list<br=
...
Users@ovirt.org<br>http://lists.ovirt.org/mailman/listinfo/users<br></bloc=
kquote></div></div></body></html>
------=_Part_1725676_1350899779.1481204814198--

Re: [ovirt-users] Hung task finalizing live migration

Logan Kuhn