7:46 a.m.
------=_Part_1725676_1350899779.1481204814198
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
I had a similar situation where I was attempting and failing to delete a ci=
nder disk and this fixed it. I'm using 4.0.5-5=20
Regards,=20
Logan=20
----- On Sep 10, 2016, at 9:39 AM, Gervais de Montbrun <gervais(a)demontbrun.=
com> wrote:=20
| Hi Maton,
| I have seen tasks in a weird state on my cluster also. I've had a vm get =
"stuck"
| during a migration where it says "migrating to" in the web GUI, but it ha=
s
| finished migrating hours ago... If I click "Cancel Migraton" the gui tell=
s me
| that it is not migrating, but I can't do any action on the vm because I a=
m then
| told that the vm can't be acted upon while it is migrating. I also try to=
kill
| the task, but there are none listed
| What has worked for me has been to put my hosted-engine in global mainten=
ance
| mode, then ssh into the hosted engine and run the "engine-setup" command.=
I am
| not saying the is the best course of action, but when the engine comes ba=
ck
| online the task is cleared.
| Cheers,
| Gervais
|| On Sep 10, 2016, at 11:06 AM, Maton, Brett < matonb(a)ltresources.co.uk > =
wrote:
|| Anyone know how to fix this broken task ?
|| It's persisted through a reboot of all hosts and the engine, something n=
eeds
|| deleting from the database to clear the task and release the locked disk
|| On 8 September 2016 at 13:25, Maton, Brett < matonb(a)ltresources.co.uk > =
wrote:
||| Thanks for the pointer Mikhail, however I don't get any tasks listed wi=
th that
||| command:
||| vdsClient -s 0 getAllTasksStatuses
||| /usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlrpc=
. since
||| ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli
||| from vdsm import utils, vdscli, constants
||| {'status': {'message': 'OK', 'code': 0},
'allTasksStatus': {}}
||| On 8 September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=
=B1=D0=B0=D0=B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB < milo1(a)ya.ru > =
wrote:
|||| Hi,
|||| There is a way to cancel a running task - look here
|||| http://lists.ovirt.org/pipermail/users/2014-November/028946.html
|||| I was able to stop snapshot deletion this way.
|||| Best, Mikhail.
|||| 08.09.2016, 08:14, "Maton, Brett" < matonb(a)ltresources.co.uk >:
||||| Any suggestions ?
||||| THe task has been hung for 5 days now, I can't start the machine or d=
estroy it.
||||| On 7 September 2016 at 06:49, Maton, Brett < matonb(a)ltresources.co.uk=
wrote:
|||||| Sorry just hit reply....
|||||| I'm seeing these errors in the logs which look related to the proble=
m:
|||||| 2016-09-07 06 :46:35,123 ERROR
|||||| [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
|||||| (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end me=
thod
|||||| 'onFailed' for command ' 07608003 -ca05-4e2e-b917-85ce525c011b'
with=
exception
|||||| 'null', the callback is marked for end method retries
|||||| 2016-09-07 06 :46:45,184 ERROR [ org.ovirt.engine.core.bll.Com mands=
Factory]
|||||| (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of com=
mand
|||||| 'LiveMigrateDisk': null
|||||| 2016-09-07 06 :46:45,185 ERROR
|||||| [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller]
|||||| (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end me=
thod
|||||| 'onFailed' for command ' 07608003 -ca05-4e2e-b917-85ce525c011b'
with=
exception
|||||| 'null', the callback is marked for end method retries
|||||| On 5 September 2016 at 06:46, Nir Soffer < nsoffer(a)redhat.com > wrot=
e:
||||||| Hi Maton,
||||||| Please reply to the list, not to me directly.
||||||| Ala, can you look at this? is this a known issue?
||||||| Thanks,
||||||| Nir
||||||| On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett < matonb(a)ltresources.c=
o.uk wrote:
||||||| > Log files as requested
||||||| > https://ufile.io/4fc35 vdsm log
||||||| > https://ufile.io/e9836 engine 03-Sep
||||||| > https://ufile.io/15f37 engine 04-Sep
||||||| > vdsm log stops on the 01-Sep...
||||||| > Couple of entries from the event log:
||||||| > Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage
||||||| > Migration' deletion for VM 'lv01' has been completed.
||||||| > Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage
||||||| > Migration' deletion for VM 'lv01' was initiated by SYSTEM
||||||| > And the related tasks
||||||| > Removing Snapshot Auto-generated for Live Storage Migration of VM=
lv01
||||||| > Sep 3, 2016 6:46:44 PM N/A 29f45ca9
||||||| > Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 6:46:44 PM
||||||| > Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 PM
||||||| > Finalizing Sep 3, 2016 7:31:06 PM N/A
||||||| > On 4 September 2016 at 14:27, Nir Soffer < nsoffer(a)redhat.com > w=
rote:
||||||| >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett < matonb@ltresourc=
es.co.uk >
||||||| >> wrote:
||||||| >>> How do I fix / kill a hung vdsm task?
||||||| >>> It seems to have completed the task but is stuck finalising.
||||||| >>> Removing Snapshot Auto-generated for Live Storage Migration
||||||| >>> Validating
||||||| >>> Executing
||||||| >>> (hour glass) Finalizing
||||||| >>> Task has been 'stuck' finalising for over 13 hours
||||||| >> Can you share engine and vdsm logs since the time the merge was =
started?
||||||| >> Nir
||||| ,
||||| _______________________________________________
||||| Users mailing list
||||| Users(a)ovirt.org
||||| http://lists.ovirt.org/mailman/listinfo/users
|||| --
|||| =D0=A1 =D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=B5=D0=BC, =D0=9A=
=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 =D0=9C=D0=B8=D1=85=
=D0=B0=D0=B8=D0=BB.
|| _______________________________________________
|| Users mailing list
|| Users(a)ovirt.org
|| http://lists.ovirt.org/mailman/listinfo/users
| _______________________________________________
| Users mailing list
| Users(a)ovirt.org
| http://lists.ovirt.org/mailman/listinfo/users
------=_Part_1725676_1350899779.1481204814198
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable
<html><body><div style=3D"font-family: Arial; font-size: 12pt; color:
#0000=
00"><div>I had a similar situation where I was attempting and failing to
de=
lete a cinder disk and this fixed it. I'm using
4.0.5-5<br></div><div=
<br></div><div
data-marker=3D"__SIG_PRE__">Regards,<br>Logan</div><br><spa=
n id=3D"zwchr" data-marker=3D"__DIVIDER__">----- On Sep 10,
2016, at 9:39 A=
M, Gervais de Montbrun &lt;gervais(a)demontbrun.com&gt;
wrote:<br></span><div=
data-marker=3D"__QUOTED_TEXT__"><blockquote style=3D"border-left:
2px soli=
d #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: n=
ormal; font-style: normal; text-decoration: none; font-family: Helvetica,Ar=
ial,sans-serif; font-size: 12pt;" data-mce-style=3D"border-left: 2px solid =
#1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: nor=
mal; font-style: normal; text-decoration: none; font-family: Helvetica,Aria=
l,sans-serif; font-size: 12pt;">Hi Maton,<div class=3D""><br
class=3D""></d=
iv><div class=3D"">I have seen tasks in a weird state on my cluster
also. I=
've had a vm get "stuck" during a migration where it says "migrating
to" in=
the web GUI, but it has finished migrating hours ago... If I click "Cancel=
Migraton" the gui tells me that it is not migrating, but I can't do any ac=
tion on the vm because I am then told that the vm can't be acted upon while=
it is migrating. I also try to kill the task, but there are none listed<br=
class=3D""><div class=3D""><br
class=3D"webkit-block-placeholder"></div><d=
iv class=3D"">What has worked for me has been to put my hosted-engine in gl=
obal maintenance mode, then ssh into the hosted engine and run the "engine-=
setup" command. I am not saying the is the best course of action, but when =
the engine comes back online the task is cleared.</div><div
class=3D"">
<div id=3D"signature" class=3D""><br
class=3D"">Cheers,<br class=3D"">Gerva=
is<br class=3D""><br class=3D""><br
class=3D""></div>
</div>
<br class=3D""><div><blockquote class=3D""><div
class=3D"">On Sep 10, 2016,=
at 11:06 AM, Maton, Brett <<a
href=3D"mailto:matonb@ltresources.co.uk" =
class=3D""
target=3D"_blank">matonb(a)ltresources.co.uk</a>&gt;
wrote:</div><=
br class=3D"Apple-interchange-newline"><div
class=3D""><div dir=3D"ltr" cla=
ss=3D""><div class=3D"">Anyone know how to fix this broken
task ?<br class=
=3D""><br class=3D""></div><div
class=3D"">It's persisted through a reboot =
of all hosts and the engine, something needs deleting from the database to =
clear the task and release the locked disk<br
class=3D""></div></div><div c=
lass=3D"gmail_extra"><br class=3D""><div
class=3D"gmail_quote">On 8 Septemb=
er 2016 at 13:25, Maton, Brett <span dir=3D"ltr"
class=3D""><<a href=3D"=
mailto:matonb@ltresources.co.uk" target=3D"_blank"
class=3D"">matonb@ltreso=
urces.co.uk</a>></span> wrote:<br
class=3D""><blockquote class=3D"gmail_=
quote" style=3D"margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-le=
ft: 1ex;" data-mce-style=3D"margin: 0 0 0 .8ex; border-left: 1px #ccc solid=
; padding-left: 1ex;"><div dir=3D"ltr" class=3D"">Thanks
for the pointer Mi=
khail, however I don't get any tasks listed with that command:<br class=3D"=
"><br class=3D"">vdsClient -s 0 getAllTasksStatuses<br
class=3D""><br class=
=3D"">/usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlr=
pc. since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli<br =
class=3D""> from vdsm import utils, vdscli, constants<br
class=3D""><=
br class=3D"">{'status': {'message': 'OK',
'code': 0}, 'allTasksStatus': {}=
}<br class=3D""><br class=3D""></div><div
class=3D"HOEnZb"><div class=3D"h5=
"><div class=3D"gmail_extra"><br
class=3D""><div class=3D"gmail_quote">On 8=
September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=
=D0=B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB <span dir=3D"ltr" class=
=3D""><<a href=3D"mailto:milo1@ya.ru"
target=3D"_blank" class=3D"">milo1=
@ya.ru</a>></span> wrote:<br class=3D""><blockquote
class=3D"gmail_quote=
" style=3D"margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1=
ex;" data-mce-style=3D"margin: 0 0 0 .8ex; border-left: 1px #ccc solid; pad=
ding-left: 1ex;"><div class=3D"">Hi,</div><div
class=3D""> </div><div =
class=3D"">There is a way to cancel a running task - look
here =
<a href=3D"http://lists.ovirt.org/pipermail/users/2014-November/028946....
" target=3D"_blank"
class=3D"">http://lists.ovirt.org/pipermail/users/2014-=
November/028946.html</a></div><div class=3D"">I was able to
stop snapshot d=
eletion this way.</div><div class=3D""> </div><div
class=3D"">Best, Mi=
khail.</div><div class=3D""> </div><div
class=3D"">08.09.2016, 08:14, =
"Maton, Brett" <<a href=3D"mailto:matonb@ltresources.co.uk"
target=3D"_b=
lank"
class=3D"">matonb@ltresources.co.uk</a>>:</div><blockquote
class=
=3D""><div class=3D""><div class=3D""><div
class=3D""><div class=3D""><div =
class=3D"">Any suggestions ?<br class=3D""><br
class=3D""></div>THe task ha=
s been hung for 5 days now, I can't start the machine or destroy it.<br cla=
ss=3D""><br class=3D""></div></div><div
class=3D""><br class=3D""><div clas=
s=3D"">On 7 September 2016 at 06:49, Maton, Brett <span
class=3D""><<a h=
ref=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank"
class=3D"">matonb=
@ltresources.co.uk</a>></span> wrote:<br
class=3D""><blockquote style=3D=
"margin: 0 0 0 0.8ex; border-left: 1px #ccc solid; padding-left: 1ex;" clas=
s=3D"" data-mce-style=3D"margin: 0 0 0 0.8ex; border-left: 1px #ccc solid;
=
padding-left: 1ex;"><div class=3D""><div
class=3D""><div class=3D"">Sorry j=
ust hit reply....<br class=3D""><br
class=3D""></div>I'm seeing these error=
s in the logs which look related to the problem:<br class=3D""><br
class=3D=
""><br class=3D""><span class=3D"">2016-09-07
06</span>:46:35,123 ERROR [or=
g.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzSchedul=
er6) [19c58c0d] Failed invoking callback end method 'onFailed' for command =
'<span
class=3D"">07608003</span>-ca05-4e2e-b917-85ce525c011b' with
excepti=
on 'null', the callback is marked for end method retries<br
class=3D""><spa=
n class=3D"">2016-09-07 06</span>:46:45,184 ERROR [<a
href=3D"http://org.ov=
irt.engine.core.bll.com/" class=3D""
target=3D"_blank">org.ovirt.engine.cor=
e.bll.Com</a>mandsFactory] (DefaultQuartzScheduler7) [19c58c0d] Error in in=
vocating CTOR of command 'LiveMigrateDisk': null<br
class=3D""><span class=
=3D"">2016-09-07 06</span>:46:45,185 ERROR
[org.ovirt.engine.core.bll.tasks=
.CommandCallbacksPoller] (DefaultQuartzScheduler7) [19c58c0d] Failed invoki=
ng callback end method 'onFailed' for command '<span
class=3D"">07608003</s=
pan>-ca05-4e2e-b917-85ce525c011b' with exception 'null', the callback is
ma=
rked for end method retries</div><div class=3D""><div
class=3D""><div class=
=3D""><div class=3D""><div class=3D""><div
class=3D""><br class=3D""><div c=
lass=3D"">On 5 September 2016 at 06:46, Nir Soffer <span
class=3D""><<a =
href=3D"mailto:nsoffer@redhat.com" target=3D"_blank"
class=3D"">nsoffer@red=
hat.com</a>></span> wrote:<br class=3D""><blockquote
style=3D"margin: 0p=
x 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;" class=
=3D"" data-mce-style=3D"margin: 0px 0px 0px 0.8ex; border-left: 1px solid
#=
cccccc; padding-left: 1ex;">Hi Maton,<br class=3D""> <br
class=3D""> Please=
reply to the list, not to me directly.<br class=3D""> <br
class=3D""> Ala,=
can you look at this? is this a known issue?<br class=3D""> <br
class=3D""=
Thanks,<br class=3D""> Nir<br
class=3D""><div class=3D""><div class=3D"">=
<br class=3D""> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett
<<a href=3D=
"mailto:matonb@ltresources.co.uk" target=3D"_blank"
class=3D"">matonb@ltres=
ources.co.uk</a>> wrote:<br class=3D""> > Log files as
requested<br c=
lass=3D""> ><br class=3D""> > <a
href=3D"https://ufile.io/4fc35" targ=
et=3D"_blank" class=3D"">https://ufile.io/4fc35</a> vdsm
log<br class=3D"">=
> <a href=3D"https://ufile.io/e9836" target=3D"_blank"
class=3D"">https=
://ufile.io/e9836</a> engine 03-Sep<br class=3D""> > <a
href=3D"https://=
ufile.io/15f37" target=3D"_blank"
class=3D"">https://ufile.io/15f37</a> eng=
ine 04-Sep<br class=3D""> ><br class=3D""> >
vdsm log stops on the 01=
-Sep...<br class=3D""> ><br class=3D""> >
Couple of entries from the =
event log:<br class=3D""> ><br class=3D""> >
Sep 3, 2016 7:31:07 PM&n=
bsp; Snapshot 'Auto-generated for Live Storage<br
class=3D""> > M=
igration' deletion for VM 'lv01' has been completed.<br
class=3D""> > Se=
p 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage=
<br class=3D""> > Migration' deletion for VM 'lv01' was
initiated by SYS=
TEM<br class=3D""> ><br class=3D""> > And
the related tasks<br class=
=3D""> ><br class=3D""> > Removing Snapshot
Auto-generated for Live S=
torage Migration of VM lv01<br class=3D""> > Sep 3, 2016 6:46:44
PM =
; N/A 29f45ca9<br
class=3D""> > Valida=
ting Sep 3, 2016 6:46:44 PM until
Se=
p 3, 2016 6:46:44 PM<br class=3D""> > Executing
Sep 3, 2016=
6:46:44 PM until Sep 3, 2016 7:31:06 PM<br
class=
=3D""> ><br class=3D""> > Finalizing
Sep 3, 2016 7:31:06=
PM N/A<br class=3D"">
><br class=3D""> ><=
br class=3D""> ><br class=3D""> > On 4
September 2016 at 14:27, Nir S=
offer <<a href=3D"mailto:nsoffer@redhat.com"
target=3D"_blank" class=3D"=
">nsoffer(a)redhat.com</a>&gt; wrote:<br class=3D"">
>><br class=3D""> =
>> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <<a
href=3D"mailt=
o:matonb@ltresources.co.uk" target=3D"_blank"
class=3D"">matonb@ltresources=
.co.uk</a>><br class=3D""> >> wrote:<br
class=3D""> >>><b=
r class=3D""> >>> How do I fix / kill a hung vdsm
task?<br class=
=3D""> >>><br class=3D"">
>>> It seems to have completed =
the task but is stuck finalising.<br class=3D"">
>>><br class=3D""=
>>> Removing Snapshot Auto-generated for Live
Storage Migration<=
br class=3D""> >>>
Validating<br class=3D""> >>> Executin=
g<br class=3D""> >>> (hour glass) Finalizing<br
class=3D""> >&g=
t;><br class=3D""> >>> Task has been
'stuck' finalising for ove=
r 13 hours<br class=3D""> >><br class=3D"">
>><br class=3D""> &=
gt;> Can you share engine and vdsm logs since the time the merge was sta=
rted?<br class=3D""> >><br class=3D"">
>> Nir<br class=3D""> &g=
t;<br class=3D"">
></div></div></blockquote></div></div></div></div></di=
v></div></div></div></blockquote></div></div></div></div>,<p
class=3D"">___=
____________________________________________<br class=3D"">Users mailing
li=
st<br class=3D""><a href=3D"mailto:Users@ovirt.org"
target=3D"_blank" class=
=3D"">Users(a)ovirt.org</a><br class=3D""><a
href=3D"http://lists.ovirt.org/m=
ailman/listinfo/users" target=3D"_blank"
class=3D"">http://lists.ovirt.org/=
mailman/listinfo/users</a></p></blockquote><span
class=3D""><span color=3D"=
#888888" data-mce-style=3D"color: #888888;" style=3D"color:
#888888;"><div =
class=3D""> </div><div
class=3D""> </div><div
class=3D"">-- =
</div><div class=3D"">=D0=A1
=D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=
=B5=D0=BC, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 =D0=
=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB.</div><div
class=3D""> </div><div cl=
ass=3D""> </div></span></span></blockquote></div><br
class=3D""></div>
</div></div></blockquote></div><br
class=3D""></div>
_______________________________________________<br class=3D"">Users
mailing=
list<br class=3D""><a href=3D"mailto:Users@ovirt.org"
class=3D"" target=3D=
"_blank">Users(a)ovirt.org</a><br
class=3D"">http://lists.ovirt.org/mailman/l=
istinfo/users<br
class=3D""></div></blockquote></div><br
class=3D""></div><=
br>_______________________________________________<br>Users mailing list<br=
Users@ovirt.org<br>http://lists.ovirt.org/mailman/listinfo/users<br></bloc=
kquote></div></div></body></html>
------=_Part_1725676_1350899779.1481204814198--