Re: [ovirt-users] Hung task finalizing live migration
Sorry just hit reply.... I'm seeing these errors in the logs which look related to the problem: 2016-09-07 06:46:35,123 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with exception 'null', the callback is marked for end method retries 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.CommandsFactory] (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command 'LiveMigrateDisk': null 2016-09-07 06:46:45,185 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with exception 'null', the callback is marked for end method retries On 5 September 2016 at 06:46, Nir Soffer <nsoffer@redhat.com> wrote:
Hi Maton,
Please reply to the list, not to me directly.
Ala, can you look at this? is this a known issue?
Thanks, Nir
On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett <matonb@ltresources.co.uk> wrote:
Log files as requested
https://ufile.io/4fc35 vdsm log https://ufile.io/e9836 engine 03-Sep https://ufile.io/15f37 engine 04-Sep
vdsm log stops on the 01-Sep...
Couple of entries from the event log:
Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage Migration' deletion for VM 'lv01' has been completed. Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage Migration' deletion for VM 'lv01' was initiated by SYSTEM
And the related tasks
Removing Snapshot Auto-generated for Live Storage Migration of VM lv01 Sep 3, 2016 6:46:44 PM N/A 29f45ca9 Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 6:46:44 PM Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 PM
Finalizing Sep 3, 2016 7:31:06 PM N/A
On 4 September 2016 at 14:27, Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <matonb@ltresources.co.uk
wrote:
How do I fix / kill a hung vdsm task?
It seems to have completed the task but is stuck finalising.
Removing Snapshot Auto-generated for Live Storage Migration Validating Executing (hour glass) Finalizing
Task has been 'stuck' finalising for over 13 hours
Can you share engine and vdsm logs since the time the merge was started?
Nir
Any suggestions ? THe task has been hung for 5 days now, I can't start the machine or destroy it. On 7 September 2016 at 06:49, Maton, Brett <matonb@ltresources.co.uk> wrote:
Sorry just hit reply....
I'm seeing these errors in the logs which look related to the problem:
2016-09-07 06:46:35,123 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with exception 'null', the callback is marked for end method retries 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.CommandsFactory] (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command 'LiveMigrateDisk': null 2016-09-07 06:46:45,185 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with exception 'null', the callback is marked for end method retries
On 5 September 2016 at 06:46, Nir Soffer <nsoffer@redhat.com> wrote:
Hi Maton,
Please reply to the list, not to me directly.
Ala, can you look at this? is this a known issue?
Thanks, Nir
On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett <matonb@ltresources.co.uk> wrote:
Log files as requested
https://ufile.io/4fc35 vdsm log https://ufile.io/e9836 engine 03-Sep https://ufile.io/15f37 engine 04-Sep
vdsm log stops on the 01-Sep...
Couple of entries from the event log:
Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage Migration' deletion for VM 'lv01' has been completed. Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage Migration' deletion for VM 'lv01' was initiated by SYSTEM
And the related tasks
Removing Snapshot Auto-generated for Live Storage Migration of VM lv01 Sep 3, 2016 6:46:44 PM N/A 29f45ca9 Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 6:46:44 PM Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 PM
Finalizing Sep 3, 2016 7:31:06 PM N/A
On 4 September 2016 at 14:27, Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <
matonb@ltresources.co.uk>
wrote:
How do I fix / kill a hung vdsm task?
It seems to have completed the task but is stuck finalising.
Removing Snapshot Auto-generated for Live Storage Migration Validating Executing (hour glass) Finalizing
Task has been 'stuck' finalising for over 13 hours
Can you share engine and vdsm logs since the time the merge was started?
Nir
Thanks for the pointer Mikhail, however I don't get any tasks listed with that command: vdsClient -s 0 getAllTasksStatuses /usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlrpc. since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli from vdsm import utils, vdscli, constants {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}} On 8 September 2016 at 09:51, Краснобаев Михаил <milo1@ya.ru> wrote:
Hi,
There is a way to cancel a running task - look here http://lists.ovirt.org/pipermail/users/2014-November/028946.html I was able to stop snapshot deletion this way.
Best, Mikhail.
08.09.2016, 08:14, "Maton, Brett" <matonb@ltresources.co.uk>:
Any suggestions ?
THe task has been hung for 5 days now, I can't start the machine or destroy it.
On 7 September 2016 at 06:49, Maton, Brett <matonb@ltresources.co.uk> wrote:
Sorry just hit reply....
I'm seeing these errors in the logs which look related to the problem:
2016-09-07 06:46:35,123 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with exception 'null', the callback is marked for end method retries 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.CommandsFactory] (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command 'LiveMigrateDisk': null 2016-09-07 06:46:45,185 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with exception 'null', the callback is marked for end method retries
On 5 September 2016 at 06:46, Nir Soffer <nsoffer@redhat.com> wrote:
Hi Maton,
Please reply to the list, not to me directly.
Ala, can you look at this? is this a known issue?
Thanks, Nir
On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett <matonb@ltresources.co.uk> wrote:
Log files as requested
https://ufile.io/4fc35 vdsm log https://ufile.io/e9836 engine 03-Sep https://ufile.io/15f37 engine 04-Sep
vdsm log stops on the 01-Sep...
Couple of entries from the event log:
Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage Migration' deletion for VM 'lv01' has been completed. Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage Migration' deletion for VM 'lv01' was initiated by SYSTEM
And the related tasks
Removing Snapshot Auto-generated for Live Storage Migration of VM lv01 Sep 3, 2016 6:46:44 PM N/A 29f45ca9 Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 6:46:44 PM Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 PM
Finalizing Sep 3, 2016 7:31:06 PM N/A
On 4 September 2016 at 14:27, Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <matonb@ltresources.co.uk
wrote:
How do I fix / kill a hung vdsm task?
It seems to have completed the task but is stuck finalising.
Removing Snapshot Auto-generated for Live Storage Migration Validating Executing (hour glass) Finalizing
Task has been 'stuck' finalising for over 13 hours
Can you share engine and vdsm logs since the time the merge was started?
Nir
,
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- С уважением, Краснобаев Михаил.
Anyone know how to fix this broken task ? It's persisted through a reboot of all hosts and the engine, something needs deleting from the database to clear the task and release the locked disk On 8 September 2016 at 13:25, Maton, Brett <matonb@ltresources.co.uk> wrote:
Thanks for the pointer Mikhail, however I don't get any tasks listed with that command:
vdsClient -s 0 getAllTasksStatuses
/usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlrpc. since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli from vdsm import utils, vdscli, constants
{'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}}
On 8 September 2016 at 09:51, Краснобаев Михаил <milo1@ya.ru> wrote:
Hi,
There is a way to cancel a running task - look here http://lists.ovirt.org/pipermail/users/2014-November/028946.html I was able to stop snapshot deletion this way.
Best, Mikhail.
08.09.2016, 08:14, "Maton, Brett" <matonb@ltresources.co.uk>:
Any suggestions ?
THe task has been hung for 5 days now, I can't start the machine or destroy it.
On 7 September 2016 at 06:49, Maton, Brett <matonb@ltresources.co.uk> wrote:
Sorry just hit reply....
I'm seeing these errors in the logs which look related to the problem:
2016-09-07 06:46:35,123 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with exception 'null', the callback is marked for end method retries 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.CommandsFactory] (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command 'LiveMigrateDisk': null 2016-09-07 06:46:45,185 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with exception 'null', the callback is marked for end method retries
On 5 September 2016 at 06:46, Nir Soffer <nsoffer@redhat.com> wrote:
Hi Maton,
Please reply to the list, not to me directly.
Ala, can you look at this? is this a known issue?
Thanks, Nir
On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett <matonb@ltresources.co.uk> wrote:
Log files as requested
https://ufile.io/4fc35 vdsm log https://ufile.io/e9836 engine 03-Sep https://ufile.io/15f37 engine 04-Sep
vdsm log stops on the 01-Sep...
Couple of entries from the event log:
Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage Migration' deletion for VM 'lv01' has been completed. Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage Migration' deletion for VM 'lv01' was initiated by SYSTEM
And the related tasks
Removing Snapshot Auto-generated for Live Storage Migration of VM lv01 Sep 3, 2016 6:46:44 PM N/A 29f45ca9 Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 6:46:44 PM Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 PM
Finalizing Sep 3, 2016 7:31:06 PM N/A
On 4 September 2016 at 14:27, Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <
matonb@ltresources.co.uk>
wrote:
How do I fix / kill a hung vdsm task?
It seems to have completed the task but is stuck finalising.
Removing Snapshot Auto-generated for Live Storage Migration Validating Executing (hour glass) Finalizing
Task has been 'stuck' finalising for over 13 hours
Can you share engine and vdsm logs since the time the merge was started?
Nir
,
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- С уважением, Краснобаев Михаил.
On Sep 10, 2016, at 11:06 AM, Maton, Brett <matonb@ltresources.co.uk> = wrote: =20 Anyone know how to fix this broken task ? =20 It's persisted through a reboot of all hosts and the engine, something = needs deleting from the database to clear the task and release the = locked disk =20 On 8 September 2016 at 13:25, Maton, Brett <matonb@ltresources.co.uk = <mailto:matonb@ltresources.co.uk>> wrote: Thanks for the pointer Mikhail, however I don't get any tasks listed = with that command: =20 vdsClient -s 0 getAllTasksStatuses =20 /usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses = xmlrpc. since ovirt 3.6 xmlrpc is deprecated, please use = vdsm.jsonrpcvdscli from vdsm import utils, vdscli, constants =20 {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}} =20 =20 On 8 September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1= =D0=B0=D0=B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB <milo1@ya.ru = <mailto:milo1@ya.ru>> wrote: Hi, =20 There is a way to cancel a running task - look here = http://lists.ovirt.org/pipermail/users/2014-November/028946.html = <http://lists.ovirt.org/pipermail/users/2014-November/028946.html> I was able to stop snapshot deletion this way. =20 Best, Mikhail. =20 08.09.2016, 08:14, "Maton, Brett" <matonb@ltresources.co.uk = <mailto:matonb@ltresources.co.uk>>:
Any suggestions ? =20 THe task has been hung for 5 days now, I can't start the machine or = destroy it. =20 =20 On 7 September 2016 at 06:49, Maton, Brett <matonb@ltresources.co.uk = <mailto:matonb@ltresources.co.uk>> wrote: Sorry just hit reply.... =20 I'm seeing these errors in the logs which look related to the =
--Apple-Mail=_BE696610-4EC6-4158-8D67-989CBCD17D96 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Maton, I have seen tasks in a weird state on my cluster also. I've had a vm get = "stuck" during a migration where it says "migrating to" in the web GUI, = but it has finished migrating hours ago... If I click "Cancel Migraton" = the gui tells me that it is not migrating, but I can't do any action on = the vm because I am then told that the vm can't be acted upon while it = is migrating. I also try to kill the task, but there are none listed What has worked for me has been to put my hosted-engine in global = maintenance mode, then ssh into the hosted engine and run the = "engine-setup" command. I am not saying the is the best course of = action, but when the engine comes back online the task is cleared. Cheers, Gervais problem:
=20 =20 2016-09-07 06:46:35,123 ERROR = [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] = (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method = 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with = exception 'null', the callback is marked for end method retries 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.Com = <http://org.ovirt.engine.core.bll.com/>mandsFactory] = (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command = 'LiveMigrateDisk': null 2016-09-07 06:46:45,185 ERROR = [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] = (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method = 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with = exception 'null', the callback is marked for end method retries =20 On 5 September 2016 at 06:46, Nir Soffer <nsoffer@redhat.com = <mailto:nsoffer@redhat.com>> wrote: Hi Maton, =20 Please reply to the list, not to me directly. =20 Ala, can you look at this? is this a known issue? =20 Thanks, Nir =20 On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett = <matonb@ltresources.co.uk <mailto:matonb@ltresources.co.uk>> wrote:
Log files as requested
https://ufile.io/4fc35 <https://ufile.io/4fc35> vdsm log https://ufile.io/e9836 <https://ufile.io/e9836> engine 03-Sep https://ufile.io/15f37 <https://ufile.io/15f37> engine 04-Sep
vdsm log stops on the 01-Sep...
Couple of entries from the event log:
Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage Migration' deletion for VM 'lv01' has been completed. Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage Migration' deletion for VM 'lv01' was initiated by SYSTEM
And the related tasks
Removing Snapshot Auto-generated for Live Storage Migration of VM = lv01 Sep 3, 2016 6:46:44 PM N/A 29f45ca9 Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 = 6:46:44 PM Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 = PM
Finalizing Sep 3, 2016 7:31:06 PM N/A
On 4 September 2016 at 14:27, Nir Soffer <nsoffer@redhat.com = <mailto:nsoffer@redhat.com>> wrote:
On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett =
<matonb@ltresources.co.uk <mailto:matonb@ltresources.co.uk>>
wrote:
How do I fix / kill a hung vdsm task?
It seems to have completed the task but is stuck finalising.
Removing Snapshot Auto-generated for Live Storage Migration Validating Executing (hour glass) Finalizing
Task has been 'stuck' finalising for over 13 hours
Can you share engine and vdsm logs since the time the merge was = started?
Nir
, _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users = <http://lists.ovirt.org/mailman/listinfo/users>=20 =20 --=20 =D0=A1 =D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=B5=D0=BC, = =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 = =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB. =20 =20 =20 =20
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_BE696610-4EC6-4158-8D67-989CBCD17D96 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D"">Hi Maton,<div class=3D""><br class=3D""></div><div class=3D"">I= have seen tasks in a weird state on my cluster also. I've had a vm get = "stuck" during a migration where it says "migrating to" in the web GUI, = but it has finished migrating hours ago... If I click "Cancel Migraton" = the gui tells me that it is not migrating, but I can't do any action on = the vm because I am then told that the vm can't be acted upon while it = is migrating. I also try to kill the task, but there are none listed<br = class=3D""><div class=3D""><br = class=3D"webkit-block-placeholder"></div><div class=3D"">What has worked = for me has been to put my hosted-engine in global maintenance mode, then = ssh into the hosted engine and run the "engine-setup" command. I am not = saying the is the best course of action, but when the engine comes back = online the task is cleared.</div><div class=3D""> <div id=3D"signature" class=3D""><br class=3D"">Cheers,<br = class=3D"">Gervais<br class=3D""><br class=3D""><br class=3D""></div> </div> <br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On Sep 10, 2016, at 11:06 AM, Maton, Brett <<a = href=3D"mailto:matonb@ltresources.co.uk" = class=3D"">matonb@ltresources.co.uk</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><div dir=3D"ltr" = class=3D""><div class=3D"">Anyone know how to fix this broken task ?<br = class=3D""><br class=3D""></div><div class=3D"">It's persisted through a = reboot of all hosts and the engine, something needs deleting from the = database to clear the task and release the locked disk<br = class=3D""></div></div><div class=3D"gmail_extra"><br class=3D""><div = class=3D"gmail_quote">On 8 September 2016 at 13:25, Maton, Brett <span = dir=3D"ltr" class=3D""><<a href=3D"mailto:matonb@ltresources.co.uk" = target=3D"_blank" class=3D"">matonb@ltresources.co.uk</a>></span> = wrote:<br class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 = 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr" = class=3D"">Thanks for the pointer Mikhail, however I don't get any tasks = listed with that command:<br class=3D""><br class=3D"">vdsClient -s 0 = getAllTasksStatuses<br class=3D""><br = class=3D"">/usr/share/vdsm/vdsClient.py:<wbr class=3D"">33: = DeprecationWarning: vdscli uses xmlrpc. since ovirt 3.6 xmlrpc is = deprecated, please use vdsm.jsonrpcvdscli<br class=3D""> from vdsm = import utils, vdscli, constants<br class=3D""><br class=3D"">{'status': = {'message': 'OK', 'code': 0}, 'allTasksStatus': {}}<br class=3D""><br = class=3D""></div><div class=3D"HOEnZb"><div class=3D"h5"><div = class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">On 8 = September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0= =B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB <span dir=3D"ltr" = class=3D""><<a href=3D"mailto:milo1@ya.ru" target=3D"_blank" = class=3D"">milo1@ya.ru</a>></span> wrote:<br class=3D""><blockquote = class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc = solid;padding-left:1ex"><div class=3D"">Hi,</div><div = class=3D""> </div><div class=3D"">There is a way to cancel a = running task - look here <a = href=3D"http://lists.ovirt.org/pipermail/users/2014-November/028946.html" = target=3D"_blank" class=3D"">http://lists.ovirt.org/piperma<wbr = class=3D"">il/users/2014-November/028946.<wbr = class=3D"">html</a></div><div class=3D"">I was able to stop snapshot = deletion this way.</div><div class=3D""> </div><div class=3D"">Best, = Mikhail.</div><div class=3D""> </div><div class=3D"">08.09.2016, = 08:14, "Maton, Brett" <<a href=3D"mailto:matonb@ltresources.co.uk" = target=3D"_blank" = class=3D"">matonb@ltresources.co.uk</a>>:</div><blockquote = type=3D"cite" class=3D""><div class=3D""><div class=3D""><div = class=3D""><div class=3D""><div class=3D"">Any suggestions ?<br = class=3D""><br class=3D""></div>THe task has been hung for 5 days now, I = can't start the machine or destroy it.<br class=3D""><br = class=3D""></div></div><div class=3D""><br class=3D""><div class=3D"">On = 7 September 2016 at 06:49, Maton, Brett <span class=3D""><<a = href=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" = class=3D"">matonb@ltresources.co.uk</a>></span> wrote:<br = class=3D""><blockquote style=3D"margin:0 0 0 0.8ex;border-left:1px #ccc = solid;padding-left:1ex" class=3D""><div class=3D""><div class=3D""><div = class=3D"">Sorry just hit reply....<br class=3D""><br class=3D""></div>I'm= seeing these errors in the logs which look related to the problem:<br = class=3D""><br class=3D""><br class=3D""><span class=3D"">2016-09-07 = 06</span>:46:35,123 ERROR [org.ovirt.engine.core.bll.tas<wbr = class=3D"">ks.CommandCallbacksPoller] (DefaultQuartzScheduler6) = [19c58c0d] Failed invoking callback end method 'onFailed' for command = '<span class=3D"">07608003</span>-ca05-4e2e-b917-85ce5<wbr = class=3D"">25c011b' with exception 'null', the callback is marked for = end method retries<br class=3D""><span class=3D"">2016-09-07 = 06</span>:46:45,184 ERROR [<a = href=3D"http://org.ovirt.engine.core.bll.com/" = class=3D"">org.ovirt.engine.core.bll.Com</a><wbr class=3D"">mandsFactory] = (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command = 'LiveMigrateDisk': null<br class=3D""><span class=3D"">2016-09-07 = 06</span>:46:45,185 ERROR [org.ovirt.engine.core.bll.tas<wbr = class=3D"">ks.CommandCallbacksPoller] (DefaultQuartzScheduler7) = [19c58c0d] Failed invoking callback end method 'onFailed' for command = '<span class=3D"">07608003</span>-ca05-4e2e-b917-85ce5<wbr = class=3D"">25c011b' with exception 'null', the callback is marked for = end method retries</div><div class=3D""><div class=3D""><div = class=3D""><div class=3D""><div class=3D""><div class=3D""><br = class=3D""><div class=3D"">On 5 September 2016 at 06:46, Nir Soffer = <span class=3D""><<a href=3D"mailto:nsoffer@redhat.com" = target=3D"_blank" class=3D"">nsoffer@redhat.com</a>></span> wrote:<br = class=3D""><blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px = solid #cccccc;padding-left:1ex" class=3D"">Hi Maton,<br class=3D""> <br = class=3D""> Please reply to the list, not to me directly.<br class=3D""> = <br class=3D""> Ala, can you look at this? is this a known issue?<br = class=3D""> <br class=3D""> Thanks,<br class=3D""> Nir<br class=3D""><div = class=3D""><div class=3D""><br class=3D""> On Mon, Sep 5, 2016 at 8:43 = AM, Maton, Brett <<a href=3D"mailto:matonb@ltresources.co.uk" = target=3D"_blank" class=3D"">matonb@ltresources.co.uk</a>> wrote:<br = class=3D""> > Log files as requested<br class=3D""> ><br class=3D"">= > <a href=3D"https://ufile.io/4fc35" target=3D"_blank" = class=3D"">https://ufile.io/4fc35</a> vdsm log<br class=3D""> > <a = href=3D"https://ufile.io/e9836" target=3D"_blank" = class=3D"">https://ufile.io/e9836</a> engine 03-Sep<br class=3D""> > = <a href=3D"https://ufile.io/15f37" target=3D"_blank" = class=3D"">https://ufile.io/15f37</a> engine 04-Sep<br class=3D""> = ><br class=3D""> > vdsm log stops on the 01-Sep...<br class=3D""> = ><br class=3D""> > Couple of entries from the event log:<br = class=3D""> ><br class=3D""> > Sep 3, 2016 7:31:07 PM = Snapshot 'Auto-generated for Live Storage<br class=3D""> > Migration' = deletion for VM 'lv01' has been completed.<br class=3D""> > Sep 3, = 2016 6:46:46 PM Snapshot 'Auto-generated for Live = Storage<br class=3D""> > Migration' deletion for VM 'lv01' was = initiated by SYSTEM<br class=3D""> ><br class=3D""> > And the = related tasks<br class=3D""> ><br class=3D""> > Removing Snapshot = Auto-generated for Live Storage Migration of VM lv01<br class=3D""> > = Sep 3, 2016 6:46:44 PM N/A = 29f45ca9<br class=3D""> > Validating Sep 3, 2016 6:46:44 = PM until Sep 3, 2016 6:46:44 PM<br class=3D""> = > Executing Sep 3, 2016 6:46:44 PM = until Sep 3, 2016 7:31:06 PM<br class=3D""> ><br = class=3D""> > Finalizing Sep 3, 2016 7:31:06 PM = N/A<br class=3D""> ><br class=3D""> ><br = class=3D""> ><br class=3D""> > On 4 September 2016 at 14:27, Nir = Soffer <<a href=3D"mailto:nsoffer@redhat.com" target=3D"_blank" = class=3D"">nsoffer@redhat.com</a>> wrote:<br class=3D""> >><br = class=3D""> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett = <<a href=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" = class=3D"">matonb@ltresources.co.uk</a>><br class=3D""> >> = wrote:<br class=3D""> >>><br class=3D""> >>> How do I = fix / kill a hung vdsm task?<br class=3D""> >>><br class=3D""> = >>> It seems to have completed the task but is stuck = finalising.<br class=3D""> >>><br class=3D""> >>> = Removing Snapshot Auto-generated for Live Storage Migration<br class=3D"">= >>> Validating<br class=3D""> >>> Executing<br = class=3D""> >>> (hour glass) Finalizing<br class=3D""> = >>><br class=3D""> >>> Task has been 'stuck' = finalising for over 13 hours<br class=3D""> >><br class=3D""> = >><br class=3D""> >> Can you share engine and vdsm logs = since the time the merge was started?<br class=3D""> >><br = class=3D""> >> Nir<br class=3D""> ><br class=3D""> = ></div></div></blockquote></div></div></div></div></div></div></div></d= iv></blockquote></div></div></div></div>,<p = class=3D"">______________________________<wbr = class=3D"">_________________<br class=3D"">Users mailing list<br = class=3D""><a href=3D"mailto:Users@ovirt.org" target=3D"_blank" = class=3D"">Users@ovirt.org</a><br class=3D""><a = href=3D"http://lists.ovirt.org/mailman/listinfo/users" target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/users</a></p></blockquote><span class=3D""><font = color=3D"#888888" class=3D""><div class=3D""> </div><div = class=3D""> </div><div class=3D"">-- </div><div class=3D"">=D0=A1= =D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=B5=D0=BC, = =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 = =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB.</div><div = class=3D""> </div><div = class=3D""> </div></font></span></blockquote></div><br = class=3D""></div> </div></div></blockquote></div><br class=3D""></div> _______________________________________________<br class=3D"">Users = mailing list<br class=3D""><a href=3D"mailto:Users@ovirt.org" = class=3D"">Users@ovirt.org</a><br = class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br = class=3D""></div></blockquote></div><br class=3D""></div></body></html>= --Apple-Mail=_BE696610-4EC6-4158-8D67-989CBCD17D96--
Thanks Gervais I'll give that a go On 10 September 2016 at 15:39, Gervais de Montbrun <gervais@demontbrun.com> wrote:
Hi Maton,
I have seen tasks in a weird state on my cluster also. I've had a vm get "stuck" during a migration where it says "migrating to" in the web GUI, but it has finished migrating hours ago... If I click "Cancel Migraton" the gui tells me that it is not migrating, but I can't do any action on the vm because I am then told that the vm can't be acted upon while it is migrating. I also try to kill the task, but there are none listed
What has worked for me has been to put my hosted-engine in global maintenance mode, then ssh into the hosted engine and run the "engine-setup" command. I am not saying the is the best course of action, but when the engine comes back online the task is cleared.
Cheers, Gervais
On Sep 10, 2016, at 11:06 AM, Maton, Brett <matonb@ltresources.co.uk> wrote:
Anyone know how to fix this broken task ?
It's persisted through a reboot of all hosts and the engine, something needs deleting from the database to clear the task and release the locked disk
On 8 September 2016 at 13:25, Maton, Brett <matonb@ltresources.co.uk> wrote:
Thanks for the pointer Mikhail, however I don't get any tasks listed with that command:
vdsClient -s 0 getAllTasksStatuses
/usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlrpc. since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli from vdsm import utils, vdscli, constants
{'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}}
On 8 September 2016 at 09:51, Краснобаев Михаил <milo1@ya.ru> wrote:
Hi,
There is a way to cancel a running task - look here http://lists.ovirt.org/pipermail/users/2014-November/028946.html I was able to stop snapshot deletion this way.
Best, Mikhail.
08.09.2016, 08:14, "Maton, Brett" <matonb@ltresources.co.uk>:
Any suggestions ?
THe task has been hung for 5 days now, I can't start the machine or destroy it.
On 7 September 2016 at 06:49, Maton, Brett <matonb@ltresources.co.uk> wrote:
Sorry just hit reply....
I'm seeing these errors in the logs which look related to the problem:
2016-09-07 06:46:35,123 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with exception 'null', the callback is marked for end method retries 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.Com <http://org.ovirt.engine.core.bll.com/>mandsFactory] (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command 'LiveMigrateDisk': null 2016-09-07 06:46:45,185 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with exception 'null', the callback is marked for end method retries
On 5 September 2016 at 06:46, Nir Soffer <nsoffer@redhat.com> wrote:
Hi Maton,
Please reply to the list, not to me directly.
Ala, can you look at this? is this a known issue?
Thanks, Nir
On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett <matonb@ltresources.co.uk> wrote:
Log files as requested
https://ufile.io/4fc35 vdsm log https://ufile.io/e9836 engine 03-Sep https://ufile.io/15f37 engine 04-Sep
vdsm log stops on the 01-Sep...
Couple of entries from the event log:
Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage Migration' deletion for VM 'lv01' has been completed. Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage Migration' deletion for VM 'lv01' was initiated by SYSTEM
And the related tasks
Removing Snapshot Auto-generated for Live Storage Migration of VM lv01 Sep 3, 2016 6:46:44 PM N/A 29f45ca9 Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 6:46:44 PM Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 PM
Finalizing Sep 3, 2016 7:31:06 PM N/A
On 4 September 2016 at 14:27, Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <
matonb@ltresources.co.uk>
wrote:
How do I fix / kill a hung vdsm task?
It seems to have completed the task but is stuck finalising.
Removing Snapshot Auto-generated for Live Storage Migration Validating Executing (hour glass) Finalizing
Task has been 'stuck' finalising for over 13 hours
Can you share engine and vdsm logs since the time the merge was started?
Nir
,
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- С уважением, Краснобаев Михаил.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Way-hey! finally the task has gone and I can do 'stuff' with that VM again. Thanks Gervais, you're a star On 10 September 2016 at 15:40, Maton, Brett <matonb@ltresources.co.uk> wrote:
Thanks Gervais I'll give that a go
On 10 September 2016 at 15:39, Gervais de Montbrun <gervais@demontbrun.com
wrote:
Hi Maton,
I have seen tasks in a weird state on my cluster also. I've had a vm get "stuck" during a migration where it says "migrating to" in the web GUI, but it has finished migrating hours ago... If I click "Cancel Migraton" the gui tells me that it is not migrating, but I can't do any action on the vm because I am then told that the vm can't be acted upon while it is migrating. I also try to kill the task, but there are none listed
What has worked for me has been to put my hosted-engine in global maintenance mode, then ssh into the hosted engine and run the "engine-setup" command. I am not saying the is the best course of action, but when the engine comes back online the task is cleared.
Cheers, Gervais
On Sep 10, 2016, at 11:06 AM, Maton, Brett <matonb@ltresources.co.uk> wrote:
Anyone know how to fix this broken task ?
It's persisted through a reboot of all hosts and the engine, something needs deleting from the database to clear the task and release the locked disk
On 8 September 2016 at 13:25, Maton, Brett <matonb@ltresources.co.uk> wrote:
Thanks for the pointer Mikhail, however I don't get any tasks listed with that command:
vdsClient -s 0 getAllTasksStatuses
/usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlrpc. since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli from vdsm import utils, vdscli, constants
{'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}}
On 8 September 2016 at 09:51, Краснобаев Михаил <milo1@ya.ru> wrote:
Hi,
There is a way to cancel a running task - look here http://lists.ovirt.org/pipermail/users/2014-November/028946.html I was able to stop snapshot deletion this way.
Best, Mikhail.
08.09.2016, 08:14, "Maton, Brett" <matonb@ltresources.co.uk>:
Any suggestions ?
THe task has been hung for 5 days now, I can't start the machine or destroy it.
On 7 September 2016 at 06:49, Maton, Brett <matonb@ltresources.co.uk> wrote:
Sorry just hit reply....
I'm seeing these errors in the logs which look related to the problem:
2016-09-07 06:46:35,123 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with exception 'null', the callback is marked for end method retries 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.Com <http://org.ovirt.engine.core.bll.com/>mandsFactory] (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command 'LiveMigrateDisk': null 2016-09-07 06:46:45,185 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with exception 'null', the callback is marked for end method retries
On 5 September 2016 at 06:46, Nir Soffer <nsoffer@redhat.com> wrote:
Hi Maton,
Please reply to the list, not to me directly.
Ala, can you look at this? is this a known issue?
Thanks, Nir
On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett <matonb@ltresources.co.uk> wrote:
Log files as requested
https://ufile.io/4fc35 vdsm log https://ufile.io/e9836 engine 03-Sep https://ufile.io/15f37 engine 04-Sep
vdsm log stops on the 01-Sep...
Couple of entries from the event log:
Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage Migration' deletion for VM 'lv01' has been completed. Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage Migration' deletion for VM 'lv01' was initiated by SYSTEM
And the related tasks
Removing Snapshot Auto-generated for Live Storage Migration of VM lv01 Sep 3, 2016 6:46:44 PM N/A 29f45ca9 Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 6:46:44 PM Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 PM
Finalizing Sep 3, 2016 7:31:06 PM N/A
On 4 September 2016 at 14:27, Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <
matonb@ltresources.co.uk>
wrote: > > How do I fix / kill a hung vdsm task? > > It seems to have completed the task but is stuck finalising. > > Removing Snapshot Auto-generated for Live Storage Migration > Validating > Executing > (hour glass) Finalizing > > Task has been 'stuck' finalising for over 13 hours
Can you share engine and vdsm logs since the time the merge was started?
Nir
,
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- С уважением, Краснобаев Михаил.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Sep 10, 2016, at 12:05 PM, Maton, Brett <matonb@ltresources.co.uk> = wrote: =20 Way-hey! finally the task has gone and I can do 'stuff' with that VM = again. =20 Thanks Gervais, you're a star =20 On 10 September 2016 at 15:40, Maton, Brett <matonb@ltresources.co.uk = <mailto:matonb@ltresources.co.uk>> wrote: Thanks Gervais I'll give that a go =20 On 10 September 2016 at 15:39, Gervais de Montbrun = <gervais@demontbrun.com <mailto:gervais@demontbrun.com>> wrote: Hi Maton, =20 I have seen tasks in a weird state on my cluster also. I've had a vm = get "stuck" during a migration where it says "migrating to" in the web = GUI, but it has finished migrating hours ago... If I click "Cancel = Migraton" the gui tells me that it is not migrating, but I can't do any = action on the vm because I am then told that the vm can't be acted upon = while it is migrating. I also try to kill the task, but there are none =
=20 What has worked for me has been to put my hosted-engine in global =
=20 Cheers, Gervais =20 =20 =20
On Sep 10, 2016, at 11:06 AM, Maton, Brett <matonb@ltresources.co.uk = <mailto:matonb@ltresources.co.uk>> wrote: =20 Anyone know how to fix this broken task ? =20 It's persisted through a reboot of all hosts and the engine, = something needs deleting from the database to clear the task and release =
=20 On 8 September 2016 at 13:25, Maton, Brett <matonb@ltresources.co.uk = <mailto:matonb@ltresources.co.uk>> wrote: Thanks for the pointer Mikhail, however I don't get any tasks listed = with that command: =20 vdsClient -s 0 getAllTasksStatuses =20 /usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses = xmlrpc. since ovirt 3.6 xmlrpc is deprecated, please use = vdsm.jsonrpcvdscli from vdsm import utils, vdscli, constants =20 {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}} =20 =20 On 8 September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1= =D0=B0=D0=B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB <milo1@ya.ru = <mailto:milo1@ya.ru>> wrote: Hi, =20 There is a way to cancel a running task - look here = http://lists.ovirt.org/pipermail/users/2014-November/028946.html = <http://lists.ovirt.org/pipermail/users/2014-November/028946.html> I was able to stop snapshot deletion this way. =20 Best, Mikhail. =20 08.09.2016, 08:14, "Maton, Brett" <matonb@ltresources.co.uk = <mailto:matonb@ltresources.co.uk>>:
Any suggestions ? =20 THe task has been hung for 5 days now, I can't start the machine or = destroy it. =20 =20 On 7 September 2016 at 06:49, Maton, Brett <matonb@ltresources.co.uk = <mailto:matonb@ltresources.co.uk>> wrote: Sorry just hit reply.... =20 I'm seeing these errors in the logs which look related to the =
--Apple-Mail=_EF692F18-F11E-46F1-8425-9C5284E5B3E2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 YAY!! Glad it worked for you. :-) Cheers, Gervais listed maintenance mode, then ssh into the hosted engine and run the = "engine-setup" command. I am not saying the is the best course of = action, but when the engine comes back online the task is cleared. the locked disk problem:
=20 =20 2016-09-07 06:46:35,123 ERROR = [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] = (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method = 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with = exception 'null', the callback is marked for end method retries 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.Com = <http://org.ovirt.engine.core.bll.com/>mandsFactory] = (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command = 'LiveMigrateDisk': null 2016-09-07 06:46:45,185 ERROR = [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] = (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method = 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with = exception 'null', the callback is marked for end method retries =20 On 5 September 2016 at 06:46, Nir Soffer <nsoffer@redhat.com = <mailto:nsoffer@redhat.com>> wrote: Hi Maton, =20 Please reply to the list, not to me directly. =20 Ala, can you look at this? is this a known issue? =20 Thanks, Nir =20 On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett = <matonb@ltresources.co.uk <mailto:matonb@ltresources.co.uk>> wrote:
Log files as requested
https://ufile.io/4fc35 <https://ufile.io/4fc35> vdsm log https://ufile.io/e9836 <https://ufile.io/e9836> engine 03-Sep https://ufile.io/15f37 <https://ufile.io/15f37> engine 04-Sep
vdsm log stops on the 01-Sep...
Couple of entries from the event log:
Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live = Storage Migration' deletion for VM 'lv01' has been completed. Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live = Storage Migration' deletion for VM 'lv01' was initiated by SYSTEM
And the related tasks
Removing Snapshot Auto-generated for Live Storage Migration of VM = lv01 Sep 3, 2016 6:46:44 PM N/A 29f45ca9 Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 = 6:46:44 PM Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 = 7:31:06 PM
Finalizing Sep 3, 2016 7:31:06 PM N/A
On 4 September 2016 at 14:27, Nir Soffer <nsoffer@redhat.com = <mailto:nsoffer@redhat.com>> wrote:
On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett =
<matonb@ltresources.co.uk <mailto:matonb@ltresources.co.uk>>
wrote:
How do I fix / kill a hung vdsm task?
It seems to have completed the task but is stuck finalising.
Removing Snapshot Auto-generated for Live Storage Migration Validating Executing (hour glass) Finalizing
Task has been 'stuck' finalising for over 13 hours
Can you share engine and vdsm logs since the time the merge was = started?
Nir
, _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users = <http://lists.ovirt.org/mailman/listinfo/users>=20 =20 --=20 =D0=A1 =D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=B5=D0=BC, = =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 = =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB. =20 =20 =20 =20
Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users = <http://lists.ovirt.org/mailman/listinfo/users> =20 =20 =20
--Apple-Mail=_EF692F18-F11E-46F1-8425-9C5284E5B3E2 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D"">YAY!! Glad it worked for you.<div class=3D""><br = class=3D""></div><div class=3D"">:-)<br class=3D""><div class=3D""> <div id=3D"signature" class=3D""><br class=3D"">Cheers,<br = class=3D"">Gervais<br class=3D""><br class=3D""><br class=3D""></div> </div> <br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On Sep 10, 2016, at 12:05 PM, Maton, Brett <<a = href=3D"mailto:matonb@ltresources.co.uk" = class=3D"">matonb@ltresources.co.uk</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><div dir=3D"ltr" = class=3D""><div class=3D"">Way-hey! finally the task has gone and I can = do 'stuff' with that VM again.<br class=3D""><br class=3D""></div>Thanks = Gervais, you're a star<br class=3D""></div><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">On 10 September 2016 at 15:40, = Maton, Brett <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" = class=3D"">matonb@ltresources.co.uk</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr" = class=3D"">Thanks Gervais I'll give that a go<br class=3D""></div><div = class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">On 10 September 2016 at 15:39, = Gervais de Montbrun <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:gervais@demontbrun.com" target=3D"_blank" = class=3D"">gervais@demontbrun.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div = style=3D"word-wrap:break-word" class=3D"">Hi Maton,<div class=3D""><br = class=3D""></div><div class=3D"">I have seen tasks in a weird state on = my cluster also. I've had a vm get "stuck" during a migration where it = says "migrating to" in the web GUI, but it has finished migrating hours = ago... If I click "Cancel Migraton" the gui tells me that it is not = migrating, but I can't do any action on the vm because I am then told = that the vm can't be acted upon while it is migrating. I also try to = kill the task, but there are none listed<br class=3D""><div class=3D""><br= class=3D""></div><div class=3D"">What has worked for me has been to put = my hosted-engine in global maintenance mode, then ssh into the hosted = engine and run the "engine-setup" command. I am not saying the is the = best course of action, but when the engine comes back online the task is = cleared.</div><div class=3D""> <div class=3D""><br class=3D"">Cheers,<br class=3D"">Gervais<br = class=3D""><br class=3D""><br class=3D""></div> </div><div class=3D""><div class=3D""> <br class=3D""><div class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On Sep 10, 2016, at 11:06 AM, Maton, Brett <<a = href=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" = class=3D"">matonb@ltresources.co.uk</a>> wrote:</div><br = class=3D""><div class=3D""><div dir=3D"ltr" class=3D""><div = class=3D"">Anyone know how to fix this broken task ?<br class=3D""><br = class=3D""></div><div class=3D"">It's persisted through a reboot of all = hosts and the engine, something needs deleting from the database to = clear the task and release the locked disk<br class=3D""></div></div><div = class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">On 8 = September 2016 at 13:25, Maton, Brett <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" = class=3D"">matonb@ltresources.co.uk</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr" = class=3D"">Thanks for the pointer Mikhail, however I don't get any tasks = listed with that command:<br class=3D""><br class=3D"">vdsClient -s 0 = getAllTasksStatuses<br class=3D""><br = class=3D"">/usr/share/vdsm/vdsClient.py:3<wbr class=3D"">3: = DeprecationWarning: vdscli uses xmlrpc. since ovirt 3.6 xmlrpc is = deprecated, please use vdsm.jsonrpcvdscli<br class=3D""> from vdsm = import utils, vdscli, constants<br class=3D""><br class=3D"">{'status': = {'message': 'OK', 'code': 0}, 'allTasksStatus': {}}<br class=3D""><br = class=3D""></div><div class=3D""><div class=3D""><div = class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">On 8 = September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0= =B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB <span dir=3D"ltr" = class=3D""><<a href=3D"mailto:milo1@ya.ru" target=3D"_blank" = class=3D"">milo1@ya.ru</a>></span> wrote:<br class=3D""><blockquote = class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc = solid;padding-left:1ex"><div class=3D"">Hi,</div><div = class=3D""> </div><div class=3D"">There is a way to cancel a = running task - look here <a = href=3D"http://lists.ovirt.org/pipermail/users/2014-November/028946.html" = target=3D"_blank" class=3D"">http://lists.ovirt.org/piperma<wbr = class=3D"">il/users/2014-November/028946.<wbr = class=3D"">html</a></div><div class=3D"">I was able to stop snapshot = deletion this way.</div><div class=3D""> </div><div class=3D"">Best, = Mikhail.</div><div class=3D""> </div><div class=3D"">08.09.2016, = 08:14, "Maton, Brett" <<a href=3D"mailto:matonb@ltresources.co.uk" = target=3D"_blank" = class=3D"">matonb@ltresources.co.uk</a>>:</div><blockquote = type=3D"cite" class=3D""><div class=3D""><div class=3D""><div = class=3D""><div class=3D""><div class=3D"">Any suggestions ?<br = class=3D""><br class=3D""></div>THe task has been hung for 5 days now, I = can't start the machine or destroy it.<br class=3D""><br = class=3D""></div></div><div class=3D""><br class=3D""><div class=3D"">On = 7 September 2016 at 06:49, Maton, Brett <span class=3D""><<a = href=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" = class=3D"">matonb@ltresources.co.uk</a>></span> wrote:<br = class=3D""><blockquote style=3D"margin:0 0 0 0.8ex;border-left:1px #ccc = solid;padding-left:1ex" class=3D""><div class=3D""><div class=3D""><div = class=3D"">Sorry just hit reply....<br class=3D""><br class=3D""></div>I'm= seeing these errors in the logs which look related to the problem:<br = class=3D""><br class=3D""><br class=3D""><span class=3D"">2016-09-07 = 06</span>:46:35,123 ERROR [org.ovirt.engine.core.bll.tas<wbr = class=3D"">ks.CommandCallbacksPoller] (DefaultQuartzScheduler6) = [19c58c0d] Failed invoking callback end method 'onFailed' for command = '<span class=3D"">07608003</span>-ca05-4e2e-b917-85ce5<wbr = class=3D"">25c011b' with exception 'null', the callback is marked for = end method retries<br class=3D""><span class=3D"">2016-09-07 = 06</span>:46:45,184 ERROR [<a = href=3D"http://org.ovirt.engine.core.bll.com/" target=3D"_blank" = class=3D"">org.ovirt.engine.core.bll.Com</a><wbr class=3D"">mandsFactory] = (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command = 'LiveMigrateDisk': null<br class=3D""><span class=3D"">2016-09-07 = 06</span>:46:45,185 ERROR [org.ovirt.engine.core.bll.tas<wbr = class=3D"">ks.CommandCallbacksPoller] (DefaultQuartzScheduler7) = [19c58c0d] Failed invoking callback end method 'onFailed' for command = '<span class=3D"">07608003</span>-ca05-4e2e-b917-85ce5<wbr = class=3D"">25c011b' with exception 'null', the callback is marked for = end method retries</div><div class=3D""><div class=3D""><div = class=3D""><div class=3D""><div class=3D""><div class=3D""><br = class=3D""><div class=3D"">On 5 September 2016 at 06:46, Nir Soffer = <span class=3D""><<a href=3D"mailto:nsoffer@redhat.com" = target=3D"_blank" class=3D"">nsoffer@redhat.com</a>></span> wrote:<br = class=3D""><blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px = solid #cccccc;padding-left:1ex" class=3D"">Hi Maton,<br class=3D""> <br = class=3D""> Please reply to the list, not to me directly.<br class=3D""> = <br class=3D""> Ala, can you look at this? is this a known issue?<br = class=3D""> <br class=3D""> Thanks,<br class=3D""> Nir<br class=3D""><div = class=3D""><div class=3D""><br class=3D""> On Mon, Sep 5, 2016 at 8:43 = AM, Maton, Brett <<a href=3D"mailto:matonb@ltresources.co.uk" = target=3D"_blank" class=3D"">matonb@ltresources.co.uk</a>> wrote:<br = class=3D""> > Log files as requested<br class=3D""> ><br class=3D"">= > <a href=3D"https://ufile.io/4fc35" target=3D"_blank" = class=3D"">https://ufile.io/4fc35</a> vdsm log<br class=3D""> > <a = href=3D"https://ufile.io/e9836" target=3D"_blank" = class=3D"">https://ufile.io/e9836</a> engine 03-Sep<br class=3D""> > = <a href=3D"https://ufile.io/15f37" target=3D"_blank" = class=3D"">https://ufile.io/15f37</a> engine 04-Sep<br class=3D""> = ><br class=3D""> > vdsm log stops on the 01-Sep...<br class=3D""> = ><br class=3D""> > Couple of entries from the event log:<br = class=3D""> ><br class=3D""> > Sep 3, 2016 7:31:07 PM = Snapshot 'Auto-generated for Live Storage<br class=3D""> > Migration' = deletion for VM 'lv01' has been completed.<br class=3D""> > Sep 3, = 2016 6:46:46 PM Snapshot 'Auto-generated for Live = Storage<br class=3D""> > Migration' deletion for VM 'lv01' was = initiated by SYSTEM<br class=3D""> ><br class=3D""> > And the = related tasks<br class=3D""> ><br class=3D""> > Removing Snapshot = Auto-generated for Live Storage Migration of VM lv01<br class=3D""> > = Sep 3, 2016 6:46:44 PM N/A = 29f45ca9<br class=3D""> > Validating Sep 3, 2016 6:46:44 = PM until Sep 3, 2016 6:46:44 PM<br class=3D""> = > Executing Sep 3, 2016 6:46:44 PM = until Sep 3, 2016 7:31:06 PM<br class=3D""> ><br = class=3D""> > Finalizing Sep 3, 2016 7:31:06 PM = N/A<br class=3D""> ><br class=3D""> ><br = class=3D""> ><br class=3D""> > On 4 September 2016 at 14:27, Nir = Soffer <<a href=3D"mailto:nsoffer@redhat.com" target=3D"_blank" = class=3D"">nsoffer@redhat.com</a>> wrote:<br class=3D""> >><br = class=3D""> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett = <<a href=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" = class=3D"">matonb@ltresources.co.uk</a>><br class=3D""> >> = wrote:<br class=3D""> >>><br class=3D""> >>> How do I = fix / kill a hung vdsm task?<br class=3D""> >>><br class=3D""> = >>> It seems to have completed the task but is stuck = finalising.<br class=3D""> >>><br class=3D""> >>> = Removing Snapshot Auto-generated for Live Storage Migration<br class=3D"">= >>> Validating<br class=3D""> >>> Executing<br = class=3D""> >>> (hour glass) Finalizing<br class=3D""> = >>><br class=3D""> >>> Task has been 'stuck' = finalising for over 13 hours<br class=3D""> >><br class=3D""> = >><br class=3D""> >> Can you share engine and vdsm logs = since the time the merge was started?<br class=3D""> >><br = class=3D""> >> Nir<br class=3D""> ><br class=3D""> = ></div></div></blockquote></div></div></div></div></div></div></div></d= iv></blockquote></div></div></div></div>,<p = class=3D"">______________________________<wbr = class=3D"">_________________<br class=3D"">Users mailing list<br = class=3D""><a href=3D"mailto:Users@ovirt.org" target=3D"_blank" = class=3D"">Users@ovirt.org</a><br class=3D""><a = href=3D"http://lists.ovirt.org/mailman/listinfo/users" target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/users</a></p></blockquote><span class=3D""><font = color=3D"#888888" class=3D""><div class=3D""> </div><div = class=3D""> </div><div class=3D"">-- </div><div class=3D"">=D0=A1= =D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=B5=D0=BC, = =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 = =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB.</div><div = class=3D""> </div><div = class=3D""> </div></font></span></blockquote></div><br = class=3D""></div> </div></div></blockquote></div><br class=3D""></div> ______________________________<wbr class=3D"">_________________<br = class=3D"">Users mailing list<br class=3D""><a = href=3D"mailto:Users@ovirt.org" target=3D"_blank" = class=3D"">Users@ovirt.org</a><br class=3D""><a = href=3D"http://lists.ovirt.org/mailman/listinfo/users" target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/users</a><br class=3D""></div></blockquote></div><br = class=3D""></div></div></div></div></blockquote></div><br = class=3D""></div> </div></div></blockquote></div><br class=3D""></div> </div></blockquote></div><br class=3D""></div></body></html>= --Apple-Mail=_EF692F18-F11E-46F1-8425-9C5284E5B3E2--
------=_Part_1725676_1350899779.1481204814198 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable I had a similar situation where I was attempting and failing to delete a ci= nder disk and this fixed it. I'm using 4.0.5-5=20 Regards,=20 Logan=20 ----- On Sep 10, 2016, at 9:39 AM, Gervais de Montbrun <gervais@demontbrun.= com> wrote:=20 | Hi Maton, | I have seen tasks in a weird state on my cluster also. I've had a vm get = "stuck" | during a migration where it says "migrating to" in the web GUI, but it ha= s | finished migrating hours ago... If I click "Cancel Migraton" the gui tell= s me | that it is not migrating, but I can't do any action on the vm because I a= m then | told that the vm can't be acted upon while it is migrating. I also try to= kill | the task, but there are none listed | What has worked for me has been to put my hosted-engine in global mainten= ance | mode, then ssh into the hosted engine and run the "engine-setup" command.= I am | not saying the is the best course of action, but when the engine comes ba= ck | online the task is cleared. | Cheers, | Gervais || On Sep 10, 2016, at 11:06 AM, Maton, Brett < matonb@ltresources.co.uk > = wrote: || Anyone know how to fix this broken task ? || It's persisted through a reboot of all hosts and the engine, something n= eeds || deleting from the database to clear the task and release the locked disk || On 8 September 2016 at 13:25, Maton, Brett < matonb@ltresources.co.uk > = wrote: ||| Thanks for the pointer Mikhail, however I don't get any tasks listed wi= th that ||| command: ||| vdsClient -s 0 getAllTasksStatuses ||| /usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlrpc= . since ||| ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli ||| from vdsm import utils, vdscli, constants ||| {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}} ||| On 8 September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0= =B1=D0=B0=D0=B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB < milo1@ya.ru > = wrote: |||| Hi, |||| There is a way to cancel a running task - look here |||| http://lists.ovirt.org/pipermail/users/2014-November/028946.html |||| I was able to stop snapshot deletion this way. |||| Best, Mikhail. |||| 08.09.2016, 08:14, "Maton, Brett" < matonb@ltresources.co.uk >: ||||| Any suggestions ? ||||| THe task has been hung for 5 days now, I can't start the machine or d= estroy it. ||||| On 7 September 2016 at 06:49, Maton, Brett < matonb@ltresources.co.uk=
wrote:
<br></div><div data-marker=3D"__SIG_PRE__">Regards,<br>Logan</div><br><spa= n id=3D"zwchr" data-marker=3D"__DIVIDER__">----- On Sep 10, 2016, at 9:39 A= M, Gervais de Montbrun <gervais@demontbrun.com> wrote:<br></span><div= data-marker=3D"__QUOTED_TEXT__"><blockquote style=3D"border-left: 2px soli= d #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: n= ormal; font-style: normal; text-decoration: none; font-family: Helvetica,Ar= ial,sans-serif; font-size: 12pt;" data-mce-style=3D"border-left: 2px solid = #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: nor= mal; font-style: normal; text-decoration: none; font-family: Helvetica,Aria= l,sans-serif; font-size: 12pt;">Hi Maton,<div class=3D""><br class=3D""></d= iv><div class=3D"">I have seen tasks in a weird state on my cluster also. I= 've had a vm get "stuck" during a migration where it says "migrating to" in=
Thanks,<br class=3D""> Nir<br class=3D""><div class=3D""><div class=3D"">= <br class=3D""> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett <<a href=3D= "mailto:matonb@ltresources.co.uk" target=3D"_blank" class=3D"">matonb@ltres= ources.co.uk</a>> wrote:<br class=3D""> > Log files as requested<br c= lass=3D""> ><br class=3D""> > <a href=3D"https://ufile.io/4fc35" targ= et=3D"_blank" class=3D"">https://ufile.io/4fc35</a> vdsm log<br class=3D"">= > <a href=3D"https://ufile.io/e9836" target=3D"_blank" class=3D"">https= ://ufile.io/e9836</a> engine 03-Sep<br class=3D""> > <a href=3D"https://= ufile.io/15f37" target=3D"_blank" class=3D"">https://ufile.io/15f37</a> eng= ine 04-Sep<br class=3D""> ><br class=3D""> > vdsm log stops on the 01= -Sep...<br class=3D""> ><br class=3D""> > Couple of entries from the = event log:<br class=3D""> ><br class=3D""> > Sep 3, 2016 7:31:07 PM&n= bsp; Snapshot 'Auto-generated for Live Storage<br class=3D""> > M= igration' deletion for VM 'lv01' has been completed.<br class=3D""> > Se=
>>> Removing Snapshot Auto-generated for Live Storage Migration<= br class=3D""> >>> Validating<br class=3D""> >>> Executin= g<br class=3D""> >>> (hour glass) Finalizing<br class=3D""> >&g= t;><br class=3D""> >>> Task has been 'stuck' finalising for ove= r 13 hours<br class=3D""> >><br class=3D""> >><br class=3D""> &= gt;> Can you share engine and vdsm logs since the time the merge was sta= rted?<br class=3D""> >><br class=3D""> >> Nir<br class=3D""> &g= t;<br class=3D""> ></div></div></blockquote></div></div></div></div></di= v></div></div></div></blockquote></div></div></div></div>,<p class=3D"">___= ____________________________________________<br class=3D"">Users mailing li= st<br class=3D""><a href=3D"mailto:Users@ovirt.org" target=3D"_blank" class= =3D"">Users@ovirt.org</a><br class=3D""><a href=3D"http://lists.ovirt.org/m= ailman/listinfo/users" target=3D"_blank" class=3D"">http://lists.ovirt.org/= mailman/listinfo/users</a></p></blockquote><span class=3D""><span color=3D"= #888888" data-mce-style=3D"color: #888888;" style=3D"color: #888888;"><div = class=3D""> </div><div class=3D""> </div><div class=3D"">-- = </div><div class=3D"">=D0=A1 =D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0= =B5=D0=BC, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 =D0= =9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB.</div><div class=3D""> </div><div cl= ass=3D""> </div></span></span></blockquote></div><br class=3D""></div> </div></div></blockquote></div><br class=3D""></div> _______________________________________________<br class=3D"">Users mailing=
|||||| Sorry just hit reply.... |||||| I'm seeing these errors in the logs which look related to the proble= m: |||||| 2016-09-07 06 :46:35,123 ERROR |||||| [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] |||||| (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end me= thod |||||| 'onFailed' for command ' 07608003 -ca05-4e2e-b917-85ce525c011b' with= exception |||||| 'null', the callback is marked for end method retries |||||| 2016-09-07 06 :46:45,184 ERROR [ org.ovirt.engine.core.bll.Com mands= Factory] |||||| (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of com= mand |||||| 'LiveMigrateDisk': null |||||| 2016-09-07 06 :46:45,185 ERROR |||||| [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] |||||| (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end me= thod |||||| 'onFailed' for command ' 07608003 -ca05-4e2e-b917-85ce525c011b' with= exception |||||| 'null', the callback is marked for end method retries |||||| On 5 September 2016 at 06:46, Nir Soffer < nsoffer@redhat.com > wrot= e: ||||||| Hi Maton, ||||||| Please reply to the list, not to me directly. ||||||| Ala, can you look at this? is this a known issue? ||||||| Thanks, ||||||| Nir ||||||| On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett < matonb@ltresources.c= o.uk > wrote: ||||||| > Log files as requested ||||||| > https://ufile.io/4fc35 vdsm log ||||||| > https://ufile.io/e9836 engine 03-Sep ||||||| > https://ufile.io/15f37 engine 04-Sep ||||||| > vdsm log stops on the 01-Sep... ||||||| > Couple of entries from the event log: ||||||| > Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage ||||||| > Migration' deletion for VM 'lv01' has been completed. ||||||| > Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage ||||||| > Migration' deletion for VM 'lv01' was initiated by SYSTEM ||||||| > And the related tasks ||||||| > Removing Snapshot Auto-generated for Live Storage Migration of VM= lv01 ||||||| > Sep 3, 2016 6:46:44 PM N/A 29f45ca9 ||||||| > Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 6:46:44 PM ||||||| > Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 PM ||||||| > Finalizing Sep 3, 2016 7:31:06 PM N/A ||||||| > On 4 September 2016 at 14:27, Nir Soffer < nsoffer@redhat.com > w= rote: ||||||| >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett < matonb@ltresourc= es.co.uk > ||||||| >> wrote: ||||||| >>> How do I fix / kill a hung vdsm task? ||||||| >>> It seems to have completed the task but is stuck finalising. ||||||| >>> Removing Snapshot Auto-generated for Live Storage Migration ||||||| >>> Validating ||||||| >>> Executing ||||||| >>> (hour glass) Finalizing ||||||| >>> Task has been 'stuck' finalising for over 13 hours ||||||| >> Can you share engine and vdsm logs since the time the merge was = started? ||||||| >> Nir ||||| , ||||| _______________________________________________ ||||| Users mailing list ||||| Users@ovirt.org ||||| http://lists.ovirt.org/mailman/listinfo/users |||| -- |||| =D0=A1 =D1=83=D0=B2=D0=B0=D0=B6=D0=B5=D0=BD=D0=B8=D0=B5=D0=BC, =D0=9A= =D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0=D0=B5=D0=B2 =D0=9C=D0=B8=D1=85= =D0=B0=D0=B8=D0=BB. || _______________________________________________ || Users mailing list || Users@ovirt.org || http://lists.ovirt.org/mailman/listinfo/users | _______________________________________________ | Users mailing list | Users@ovirt.org | http://lists.ovirt.org/mailman/listinfo/users ------=_Part_1725676_1350899779.1481204814198 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: Arial; font-size: 12pt; color: #0000= 00"><div>I had a similar situation where I was attempting and failing to de= lete a cinder disk and this fixed it. I'm using 4.0.5-5<br></div><div= the web GUI, but it has finished migrating hours ago... If I click "Cancel= Migraton" the gui tells me that it is not migrating, but I can't do any ac= tion on the vm because I am then told that the vm can't be acted upon while= it is migrating. I also try to kill the task, but there are none listed<br= class=3D""><div class=3D""><br class=3D"webkit-block-placeholder"></div><d= iv class=3D"">What has worked for me has been to put my hosted-engine in gl= obal maintenance mode, then ssh into the hosted engine and run the "engine-= setup" command. I am not saying the is the best course of action, but when = the engine comes back online the task is cleared.</div><div class=3D""> <div id=3D"signature" class=3D""><br class=3D"">Cheers,<br class=3D"">Gerva= is<br class=3D""><br class=3D""><br class=3D""></div> </div> <br class=3D""><div><blockquote class=3D""><div class=3D"">On Sep 10, 2016,= at 11:06 AM, Maton, Brett <<a href=3D"mailto:matonb@ltresources.co.uk" = class=3D"" target=3D"_blank">matonb@ltresources.co.uk</a>> wrote:</div><= br class=3D"Apple-interchange-newline"><div class=3D""><div dir=3D"ltr" cla= ss=3D""><div class=3D"">Anyone know how to fix this broken task ?<br class= =3D""><br class=3D""></div><div class=3D"">It's persisted through a reboot = of all hosts and the engine, something needs deleting from the database to = clear the task and release the locked disk<br class=3D""></div></div><div c= lass=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">On 8 Septemb= er 2016 at 13:25, Maton, Brett <span dir=3D"ltr" class=3D""><<a href=3D"= mailto:matonb@ltresources.co.uk" target=3D"_blank" class=3D"">matonb@ltreso= urces.co.uk</a>></span> wrote:<br class=3D""><blockquote class=3D"gmail_= quote" style=3D"margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-le= ft: 1ex;" data-mce-style=3D"margin: 0 0 0 .8ex; border-left: 1px #ccc solid= ; padding-left: 1ex;"><div dir=3D"ltr" class=3D"">Thanks for the pointer Mi= khail, however I don't get any tasks listed with that command:<br class=3D"= "><br class=3D"">vdsClient -s 0 getAllTasksStatuses<br class=3D""><br class= =3D"">/usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses xmlr= pc. since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli<br = class=3D""> from vdsm import utils, vdscli, constants<br class=3D""><= br class=3D"">{'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}= }<br class=3D""><br class=3D""></div><div class=3D"HOEnZb"><div class=3D"h5= "><div class=3D"gmail_extra"><br class=3D""><div class=3D"gmail_quote">On 8= September 2016 at 09:51, =D0=9A=D1=80=D0=B0=D1=81=D0=BD=D0=BE=D0=B1=D0=B0= =D0=B5=D0=B2 =D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB <span dir=3D"ltr" class= =3D""><<a href=3D"mailto:milo1@ya.ru" target=3D"_blank" class=3D"">milo1= @ya.ru</a>></span> wrote:<br class=3D""><blockquote class=3D"gmail_quote= " style=3D"margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1= ex;" data-mce-style=3D"margin: 0 0 0 .8ex; border-left: 1px #ccc solid; pad= ding-left: 1ex;"><div class=3D"">Hi,</div><div class=3D""> </div><div = class=3D"">There is a way to cancel a running task - look here = <a href=3D"http://lists.ovirt.org/pipermail/users/2014-November/028946.html= " target=3D"_blank" class=3D"">http://lists.ovirt.org/pipermail/users/2014-= November/028946.html</a></div><div class=3D"">I was able to stop snapshot d= eletion this way.</div><div class=3D""> </div><div class=3D"">Best, Mi= khail.</div><div class=3D""> </div><div class=3D"">08.09.2016, 08:14, = "Maton, Brett" <<a href=3D"mailto:matonb@ltresources.co.uk" target=3D"_b= lank" class=3D"">matonb@ltresources.co.uk</a>>:</div><blockquote class= =3D""><div class=3D""><div class=3D""><div class=3D""><div class=3D""><div = class=3D"">Any suggestions ?<br class=3D""><br class=3D""></div>THe task ha= s been hung for 5 days now, I can't start the machine or destroy it.<br cla= ss=3D""><br class=3D""></div></div><div class=3D""><br class=3D""><div clas= s=3D"">On 7 September 2016 at 06:49, Maton, Brett <span class=3D""><<a h= ref=3D"mailto:matonb@ltresources.co.uk" target=3D"_blank" class=3D"">matonb= @ltresources.co.uk</a>></span> wrote:<br class=3D""><blockquote style=3D= "margin: 0 0 0 0.8ex; border-left: 1px #ccc solid; padding-left: 1ex;" clas= s=3D"" data-mce-style=3D"margin: 0 0 0 0.8ex; border-left: 1px #ccc solid; = padding-left: 1ex;"><div class=3D""><div class=3D""><div class=3D"">Sorry j= ust hit reply....<br class=3D""><br class=3D""></div>I'm seeing these error= s in the logs which look related to the problem:<br class=3D""><br class=3D= ""><br class=3D""><span class=3D"">2016-09-07 06</span>:46:35,123 ERROR [or= g.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzSchedul= er6) [19c58c0d] Failed invoking callback end method 'onFailed' for command = '<span class=3D"">07608003</span>-ca05-4e2e-b917-85ce525c011b' with excepti= on 'null', the callback is marked for end method retries<br class=3D""><spa= n class=3D"">2016-09-07 06</span>:46:45,184 ERROR [<a href=3D"http://org.ov= irt.engine.core.bll.com/" class=3D"" target=3D"_blank">org.ovirt.engine.cor= e.bll.Com</a>mandsFactory] (DefaultQuartzScheduler7) [19c58c0d] Error in in= vocating CTOR of command 'LiveMigrateDisk': null<br class=3D""><span class= =3D"">2016-09-07 06</span>:46:45,185 ERROR [org.ovirt.engine.core.bll.tasks= .CommandCallbacksPoller] (DefaultQuartzScheduler7) [19c58c0d] Failed invoki= ng callback end method 'onFailed' for command '<span class=3D"">07608003</s= pan>-ca05-4e2e-b917-85ce525c011b' with exception 'null', the callback is ma= rked for end method retries</div><div class=3D""><div class=3D""><div class= =3D""><div class=3D""><div class=3D""><div class=3D""><br class=3D""><div c= lass=3D"">On 5 September 2016 at 06:46, Nir Soffer <span class=3D""><<a = href=3D"mailto:nsoffer@redhat.com" target=3D"_blank" class=3D"">nsoffer@red= hat.com</a>></span> wrote:<br class=3D""><blockquote style=3D"margin: 0p= x 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;" class= =3D"" data-mce-style=3D"margin: 0px 0px 0px 0.8ex; border-left: 1px solid #= cccccc; padding-left: 1ex;">Hi Maton,<br class=3D""> <br class=3D""> Please= reply to the list, not to me directly.<br class=3D""> <br class=3D""> Ala,= can you look at this? is this a known issue?<br class=3D""> <br class=3D""= p 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage= <br class=3D""> > Migration' deletion for VM 'lv01' was initiated by SYS= TEM<br class=3D""> ><br class=3D""> > And the related tasks<br class= =3D""> ><br class=3D""> > Removing Snapshot Auto-generated for Live S= torage Migration of VM lv01<br class=3D""> > Sep 3, 2016 6:46:44 PM = ; N/A 29f45ca9<br class=3D""> > Valida= ting Sep 3, 2016 6:46:44 PM until Se= p 3, 2016 6:46:44 PM<br class=3D""> > Executing Sep 3, 2016= 6:46:44 PM until Sep 3, 2016 7:31:06 PM<br class= =3D""> ><br class=3D""> > Finalizing Sep 3, 2016 7:31:06= PM N/A<br class=3D""> ><br class=3D""> ><= br class=3D""> ><br class=3D""> > On 4 September 2016 at 14:27, Nir S= offer <<a href=3D"mailto:nsoffer@redhat.com" target=3D"_blank" class=3D"= ">nsoffer@redhat.com</a>> wrote:<br class=3D""> >><br class=3D""> = >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett <<a href=3D"mailt= o:matonb@ltresources.co.uk" target=3D"_blank" class=3D"">matonb@ltresources= .co.uk</a>><br class=3D""> >> wrote:<br class=3D""> >>><b= r class=3D""> >>> How do I fix / kill a hung vdsm task?<br class= =3D""> >>><br class=3D""> >>> It seems to have completed = the task but is stuck finalising.<br class=3D""> >>><br class=3D""= list<br class=3D""><a href=3D"mailto:Users@ovirt.org" class=3D"" target=3D= "_blank">Users@ovirt.org</a><br class=3D"">http://lists.ovirt.org/mailman/l= istinfo/users<br class=3D""></div></blockquote></div><br class=3D""></div><= br>_______________________________________________<br>Users mailing list<br=
Users@ovirt.org<br>http://lists.ovirt.org/mailman/listinfo/users<br></bloc= kquote></div></div></body></html> ------=_Part_1725676_1350899779.1481204814198--
participants (4)
-
Gervais de Montbrun -
Logan Kuhn -
Maton, Brett -
Краснобаев Михаил