[ovirt-users] Re: SPM and Task error ...

In my lab I would restart the vdsm.service , but as you are in prod - you should not to. It could cause the node get fenced and all the running VMs will be killed. Have you manually migrated all VMs to the other host ? If yes, then you can try this risky approach. Otherwise I'm out of ideas. Maybe a restart of the HostedEngine/Engine will help...

Best Regards, Strahil Nikolov В петък, 26 юли 2019 г., 13:28:13 ч. Гринуич+3, Enrico <enrico.becchetti(a)pg.infn.it&gt; написа: Dear all, I try this: # /usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh -v -t fdcf4d1b-82fe-49a6-b233-323ebe568f8e select exists (select * from information_schema.tables where table_schema = 'public' and table_name = 'command_entities');  t  This will remove the given Task!!! Caution, this operation should be used with care. Please contact support prior to running this command Are you sure you want to proceed? [y/n] y SELECT Deleteasync_tasks('fdcf4d1b-82fe-49a6-b233-323ebe568f8e');                  0 # /usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh -v -R select exists (select * from information_schema.tables where table_schema = 'public' and table_name = 'command_entities');  t  This will remove all async_tasks table content!!! Caution, this operation should be used with care. Please contact support prior to running this command Are you sure you want to proceed? [y/n] y TRUNCATE TABLE async_tasks cascade; TRUNCATE TABLE but after these commands I see the same messages inside engine.log: 2019-07-26 12:25:19,727+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-26 12:25:19,727+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-26 12:25:19,779+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-26 12:25:19,779+02 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] BaseAsyncTask::logEndTaskFailure: Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: 2019-07-26 12:25:19,779+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-26 12:25:19,780+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 753de6fe 2019-07-26 12:25:19,781+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: e5dc020 2019-07-26 12:25:19,786+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned 2019-07-26 12:25:19,786+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] BaseAsyncTask::onTaskEndSuccess: Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2019-07-26 12:25:19,786+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-26 12:25:19,787+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 40383d58 2019-07-26 12:25:19,788+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 758d79b 2019-07-26 12:25:19,793+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned 2019-07-26 12:25:19,793+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-26 12:25:19,793+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-26 12:25:19,794+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 1dc8cb98 2019-07-26 12:25:19,795+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 7df444c1 2019-07-26 12:25:19,800+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned it's seems to be task id fdcf4d1b-82fe-49a6-b233-323ebe568f8e  is still running. Any other ideas ? Thank you so much. Best Regards Enrico Il 26/07/19 09:39, Strahil Nikolov ha scritto: Have you tried to remove your task via the taskcleaner.sh You can find some details on https://www.ovirt.org/develop/developer-guide/db-issues/helperutilities.html I should admit , that I have never used that. Best Regards, Strahil Nikolov В петък, 26 юли 2019 г., 09:03:10 ч. Гринуич+3, Enrico <enrico.becchetti(a)pg.infn.it&gt; <mailto:enrico.becchetti@pg.infn.it> написа: Il 25/07/19 22:27, Strahil Nikolov ha scritto: > Hi all, >my ovirt cluster has got 3 Hypervisors runnig Centos 7.5.1804 vdsm is 4.20.39.1-1.el7, >ovirt engine is 4.2.4.5-1.el7, the storage systems are HP MSA P2000 and 2050 (fibre channel). >I need to stop one of the hypervisors for maintenance but this system is the storage pool manager. >For this reason I decided to manually activate SPM in one of the other nodes but this operation is not successful. Why would you do that ? Just put the host in maintenance and the system will evacuate all VMs and another host will be promoted as SPM . Once the host is in maintenance - you can proceed with your tasks. P.S: Don't forget to remove the mainteance mode once you are over. Best Regards, Strahil Nikolov Hi Strahil, I did not tell you that this is the first thing I tried but it didn't work because SPM can't be activated in another host, then I tried to manually move SPM without success. Any ideas ? Thanks a lot ! Best Regards Enrico -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it _______________________________________________________________________ -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it _______________________________________________________________________