[ovirt-users] Re: SPM and Task error ...

In my lab I would restart the vdsm.service , but as you are in prod - you should not to.It could cause the node get fenced and all the running VMs will be killed. Have you manually migrated all VMs to the other host ?If yes, then you can try this risky approach. Otherwise I'm out of ideas. Maybe a restart of the HostedEngine/Engine will help... Best Regards,Strahil Nikolov В петък, 26 юли 2019 г., 13:28:13 ч. Гринуич+3, Enrico <enrico.becchetti(a)pg.infn.it&gt; написа:    Dear all, I try this: # /usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh -v -t fdcf4d1b-82fe-49a6-b233-323ebe568f8e select exists (select * from information_schema.tables where table_schema = 'public' and table_name = 'command_entities');  t  This will remove the given Task!!! Caution, this operation should be used with care. Please contact support prior to running this command Are you sure you want to proceed? [y/n] y SELECT Deleteasync_tasks('fdcf4d1b-82fe-49a6-b233-323ebe568f8e');                  0 # /usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh -v -R select exists (select * from information_schema.tables where table_schema = 'public' and table_name = 'command_entities');  t  This will remove all async_tasks table content!!! Caution, this operation should be used with care. Please contact support prior to running this command Are you sure you want to proceed? [y/n] y TRUNCATE TABLE async_tasks cascade; TRUNCATE TABLE but after these commands I see the same messages inside engine.log: 2019-07-26 12:25:19,727+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-26 12:25:19,727+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-26 12:25:19,779+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-26 12:25:19,779+02 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] BaseAsyncTask::logEndTaskFailure: Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: 2019-07-26 12:25:19,779+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-26 12:25:19,780+02 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, SPMClearTaskVDSCommand(SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 753de6fe 2019-07-26 12:25:19,781+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management,HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: e5dc020 2019-07-26 12:25:19,786+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned 2019-07-26 12:25:19,786+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] BaseAsyncTask::onTaskEndSuccess: Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2019-07-26 12:25:19,786+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-26 12:25:19,787+02 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, SPMClearTaskVDSCommand(SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 40383d58 2019-07-26 12:25:19,788+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management,HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 758d79b 2019-07-26 12:25:19,793+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned 2019-07-26 12:25:19,793+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-26 12:25:19,793+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-26 12:25:19,794+02 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, SPMClearTaskVDSCommand(SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 1dc8cb98 2019-07-26 12:25:19,795+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management,HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 7df444c1 2019-07-26 12:25:19,800+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned it's seems to be task id   fdcf4d1b-82fe-49a6-b233-323ebe568f8e  is still running. Any other ideas ? Thank you so much. Best Regards Enrico Il 26/07/19 09:39, Strahil Nikolov ha scritto: Have you tried to remove your task via the taskcleaner.sh You can find some details on https://www.ovirt.org/develop/developer-guide/db-issues/helperutilitie... I should admit , that I have never used that. Best Regards, Strahil Nikolov В петък, 26 юли 2019 г., 09:03:10 ч. Гринуич+3, Enrico <enrico.becchetti(a)pg.infn.it&gt; написа: Il 25/07/19 22:27, Strahil Nikolov ha scritto: successful. Why would you do that ? Just put the host in maintenance and the system will evacuate all VMs and another host will be promoted as SPM .  Once the host is in maintenance - you can proceed with your tasks. P.S: Don't forget to remove the mainteance mode once you are over. Best Regards, Strahil Nikolov Hi Strahil, I did not tell you that this is the first thing I tried but it didn't work because SPM can't be activated in another host, then I tried to manually move SPM without success. Any ideas ? Thanks a lot ! Best Regards Enrico -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it _______________________________________________________________________ -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it _______________________________________________________________________