How to kill hanged task listed in engine but not in vdsm-client?
Hello, I'm struggling with an hanged task. I can see it from RestAPI (and UI) as: <job href="/ovirt-engine/api/jobs/9326a95e-4686-4205-bbac-b552beae4d29" id="9326a95e-4686-4205-bbac-b552beae4d29"> <actions> <link href="/ovirt-engine/api/jobs/9326a95e-4686-4205-bbac-b552beae4d29/clear" rel="clear"/> <link href="/ovirt-engine/api/jobs/9326a95e-4686-4205-bbac-b552beae4d29/end" rel="end"/> </actions> <description>Adding Disk</description> <link href="/ovirt-engine/api/jobs/9326a95e-4686-4205-bbac-b552beae4d29/steps" rel="steps"/> <auto_cleared>true</auto_cleared> <external>false</external> <last_updated>2018-11-23T18:40:59.466+01:00</last_updated> <start_time>2018-11-23T18:06:18.512+01:00</start_time> <status>started</status> <owner href="/ovirt-engine/api/users/6f4573b6-0cf1-4cbb-ac03-bf8e126b394c" id="6f4573b6-0cf1-4cbb-ac03-bf8e126b394c"/> </job> The job is hanged since user has killed fallocate process on the host. However, I can't kill task from the hosts itself since vdsm-client returns zero task on any host: # vdsm-client Host getAllTasksInfo {} How can I deal with it? I already tried to restart ovirt-engine.service but task is still there and prevent me to put the involved storage domain into maintenance. I also tried the "clear" method for service "job" via RestAPI but it fails since it's not an external job. Thank you, -- Stefano Stagnaro Prisma Telecom Testing S.r.l. Via Petrocchi, 4 20127 Milano – Italy Tel. 02 26113507 int 339 e-mail: stefanos@prismatelecomtesting.com skype: stefano.stagnaro
On Mon, Nov 26, 2018 at 1:39 PM Stefano Stagnaro < stefanos@prismatelecomtesting.com> wrote: [snip]
The job is hanged since user has killed fallocate process on the host. However, I can't kill task from the hosts itself since vdsm-client returns zero task on any host:
# vdsm-client Host getAllTasksInfo {}
How can I deal with it? I already tried to restart ovirt-engine.service but task is still there and prevent me to put the involved storage domain into maintenance.
I also tried the "clear" method for service "job" via RestAPI but it fails since it's not an external job.
Thank you, -- Stefano Stagnaro
I think in these cases you have to use taskcleaner.sh Last year I had similar problems with hung tasks and I was pointed to taskcleaner.sh from Fred: https://lists.ovirt.org/pipermail/users/2017-November/085194.html but it was not so user friendly in its syntax During engine-setup there are cleaning stages when you see [ INFO ] Cleaning async tasks and compensations [ INFO ] Unlocking existing entities So one chance could be to run engine-setup --offline (to prevent updates) and let it do the clean stage Putting environment into global maintenance before in case of self hosted engine There are also detailed docs on rh portal, such as https://access.redhat.com/solutions/500263 but they need subscription for access.... and for pure oVirt I only found this: https://www.ovirt.org/develop/developer-guide/db-issues/helperutilities/ It would be nice to have a free documentation entry point for oVirt users facing this kind of problems, in my opinion. HIH, Gianluca
On Mon, Nov 26, 2018 at 1:39 PM Stefano Stagnaro < stefanos(a)prismatelecomtesting.com> wrote: [snip]
I think in these cases you have to use taskcleaner.sh Last year I had similar problems with hung tasks and I was pointed to taskcleaner.sh from Fred: https://lists.ovirt.org/pipermail/users/2017-November/085194.html
even taskcleaner.sh is not showing any tasks: [root@engine ~]# /usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh -v select exists (select * from information_schema.tables where table_schema = 'public' and table_name = 'command_entities'); t SELECT task_id,task_type,status,started_at,result,action_type as command_type,command_id,step_id,storage_pool_id as DC FROM GetAllFromasync_tasks();
but it was not so user friendly in its syntax During engine-setup there are cleaning stages when you see
[ INFO ] Cleaning async tasks and compensations [ INFO ] Unlocking existing entities
So one chance could be to run engine-setup --offline (to prevent updates) and let it do the clean stage Putting environment into global maintenance before in case of self hosted engine
I will try it as last resort.
There are also detailed docs on rh portal, such as https://access.redhat.com/solutions/500263 Nothing there rather then, again, vdsm-client and taskcleaner.sh
but they need subscription for access.... and for pure oVirt I only found this: https://www.ovirt.org/develop/developer-guide/db-issues/helperutilities/
It would be nice to have a free documentation entry point for oVirt users facing this kind of problems, in my opinion. I totally agree.
HIH, Gianluca
Grazie, Stefano.
participants (3)
-
Gianluca Cecchi -
Stefano Stagnaro -
stefanos＠prismatelecomtesting.com