Re: [ovirt-users] Hung task finalizing live migration

Thanks Gervais I'll give that a go On 10 September 2016 at 15:39, Gervais de Montbrun <gervais(a)demontbrun.com > wrote: > Hi Maton, > > I have seen tasks in a weird state on my cluster also. I've had a vm get > "stuck" during a migration where it says "migrating to" in the web GUI, but > it has finished migrating hours ago... If I click "Cancel Migraton" the gui > tells me that it is not migrating, but I can't do any action on the vm > because I am then told that the vm can't be acted upon while it is > migrating. I also try to kill the task, but there are none listed > > What has worked for me has been to put my hosted-engine in global > maintenance mode, then ssh into the hosted engine and run the > "engine-setup" command. I am not saying the is the best course of action, > but when the engine comes back online the task is cleared. > > Cheers, > Gervais > > > > On Sep 10, 2016, at 11:06 AM, Maton, Brett <matonb(a)ltresources.co.uk&gt; > wrote: > > Anyone know how to fix this broken task ? > > It's persisted through a reboot of all hosts and the engine, something > needs deleting from the database to clear the task and release the locked > disk > > On 8 September 2016 at 13:25, Maton, Brett <matonb(a)ltresources.co.uk&gt; > wrote: > >> Thanks for the pointer Mikhail, however I don't get any tasks listed >> with that command: >> >> vdsClient -s 0 getAllTasksStatuses >> >> /usr/share/vdsm/vdsClient.py:33: DeprecationWarning: vdscli uses >> xmlrpc. since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli >> from vdsm import utils, vdscli, constants >> >> {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}} >> >> >> On 8 September 2016 at 09:51, Краснобаев Михаил <milo1(a)ya.ru&gt; wrote: >> >>> Hi, >>> >>> There is a way to cancel a running task - look here >>> http://lists.ovirt.org/pipermail/users/2014-November/028946.html >>> I was able to stop snapshot deletion this way. >>> >>> Best, Mikhail. >>> >>> 08.09.2016, 08:14, "Maton, Brett" <matonb(a)ltresources.co.uk&gt;: >>> >>> Any suggestions ? >>> >>> THe task has been hung for 5 days now, I can't start the machine or >>> destroy it. >>> >>> >>> On 7 September 2016 at 06:49, Maton, Brett <matonb(a)ltresources.co.uk&gt; >>> wrote: >>> >>> Sorry just hit reply.... >>> >>> I'm seeing these errors in the logs which look related to the problem: >>> >>> >>> 2016-09-07 06:46:35,123 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] >>> (DefaultQuartzScheduler6) [19c58c0d] Failed invoking callback end method >>> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with >>> exception 'null', the callback is marked for end method retries >>> 2016-09-07 06:46:45,184 ERROR [org.ovirt.engine.core.bll.Com >>> <http://org.ovirt.engine.core.bll.com/>mandsFactory] >>> (DefaultQuartzScheduler7) [19c58c0d] Error in invocating CTOR of command >>> 'LiveMigrateDisk': null >>> 2016-09-07 06:46:45,185 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] >>> (DefaultQuartzScheduler7) [19c58c0d] Failed invoking callback end method >>> 'onFailed' for command '07608003-ca05-4e2e-b917-85ce525c011b' with >>> exception 'null', the callback is marked for end method retries >>> >>> On 5 September 2016 at 06:46, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >>> >>> Hi Maton, >>> >>> Please reply to the list, not to me directly. >>> >>> Ala, can you look at this? is this a known issue? >>> >>> Thanks, >>> Nir >>> >>> On Mon, Sep 5, 2016 at 8:43 AM, Maton, Brett <matonb(a)ltresources.co.uk&gt; >>> wrote: >>> > Log files as requested >>> > >>> > https://ufile.io/4fc35 vdsm log >>> > https://ufile.io/e9836 engine 03-Sep >>> > https://ufile.io/15f37 engine 04-Sep >>> > >>> > vdsm log stops on the 01-Sep... >>> > >>> > Couple of entries from the event log: >>> > >>> > Sep 3, 2016 7:31:07 PM Snapshot 'Auto-generated for Live Storage >>> > Migration' deletion for VM 'lv01' has been completed. >>> > Sep 3, 2016 6:46:46 PM Snapshot 'Auto-generated for Live Storage >>> > Migration' deletion for VM 'lv01' was initiated by SYSTEM >>> > >>> > And the related tasks >>> > >>> > Removing Snapshot Auto-generated for Live Storage Migration of VM lv01 >>> > Sep 3, 2016 6:46:44 PM N/A 29f45ca9 >>> > Validating Sep 3, 2016 6:46:44 PM until Sep 3, 2016 6:46:44 >>> PM >>> > Executing Sep 3, 2016 6:46:44 PM until Sep 3, 2016 7:31:06 PM >>> > >>> > Finalizing Sep 3, 2016 7:31:06 PM N/A >>> > >>> > >>> > >>> > On 4 September 2016 at 14:27, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >>> >> >>> >> On Sun, Sep 4, 2016 at 12:40 PM, Maton, Brett < >>> matonb(a)ltresources.co.uk&gt; >>> >> wrote: >>> >>> >>> >>> How do I fix / kill a hung vdsm task? >>> >>> >>> >>> It seems to have completed the task but is stuck finalising. >>> >>> >>> >>> Removing Snapshot Auto-generated for Live Storage Migration >>> >>> Validating >>> >>> Executing >>> >>> (hour glass) Finalizing >>> >>> >>> >>> Task has been 'stuck' finalising for over 13 hours >>> >> >>> >> >>> >> Can you share engine and vdsm logs since the time the merge was >>> started? >>> >> >>> >> Nir >>> > >>> > >>> >>> , >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >>> >>> -- >>> С уважением, Краснобаев Михаил. >>> >>> >>> >> >> > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > >