In my lab I would restart the vdsm.service , but as you are in prod - you should not to.

It could cause the node get fenced and all the running VMs will be killed.

Have you manually migrated all VMs to the other host ?

If yes, then you can try this risky approach. Otherwise I'm out of ideas.

Maybe a restart of the HostedEngine/Engine will help...

Best Regards,

Strahil Nikolov

В петък, 26 юли 2019 г., 13:28:13 ч. Гринуич+3, Enrico <enrico.becchetti@pg.infn.it> написа:

   Dear all,

I try this:

# /usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh -v -t fdcf4d1b-82fe-49a6-b233-323ebe568f8e
select exists (select * from information_schema.tables where table_schema = 'public' and table_name = 'command_entities');
 t
 This will remove the given Task!!!
Caution, this operation should be used with care. Please contact support prior to running this command
Are you sure you want to proceed? [y/n]
y
SELECT Deleteasync_tasks('fdcf4d1b-82fe-49a6-b233-323ebe568f8e');
                 0

# /usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh -v -R
select exists (select * from information_schema.tables where table_schema = 'public' and table_name = 'command_entities');
 t
 This will remove all async_tasks table content!!!
Caution, this operation should be used with care. Please contact support prior to running this command
Are you sure you want to proceed? [y/n]
y
TRUNCATE TABLE async_tasks cascade;
TRUNCATE TABLE

but after these commands I see the same messages inside engine.log:

2019-07-26 12:25:19,727+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis.
2019-07-26 12:25:19,727+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis.
2019-07-26 12:25:19,779+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis.
2019-07-26 12:25:19,779+02 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] BaseAsyncTask::logEndTaskFailure: Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure:
2019-07-26 12:25:19,779+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e'
2019-07-26 12:25:19,780+02 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 753de6fe
2019-07-26 12:25:19,781+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: e5dc020
2019-07-26 12:25:19,786+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned
2019-07-26 12:25:19,786+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] BaseAsyncTask::onTaskEndSuccess: Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully.
2019-07-26 12:25:19,786+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e'
2019-07-26 12:25:19,787+02 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 40383d58
2019-07-26 12:25:19,788+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 758d79b
2019-07-26 12:25:19,793+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned
2019-07-26 12:25:19,793+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis.
2019-07-26 12:25:19,793+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e'
2019-07-26 12:25:19,794+02 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 1dc8cb98
2019-07-26 12:25:19,795+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 7df444c1
2019-07-26 12:25:19,800+02 INFO  [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned

it's seems to be task id   fdcf4d1b-82fe-49a6-b233-323ebe568f8e  is still running.

Any other ideas ?

Thank you so much.
Best Regards
Enrico

Il 26/07/19 09:39, Strahil Nikolov ha scritto:

Have you tried to remove your task via the taskcleaner.sh

You can find some details on https://www.ovirt.org/develop/developer-guide/db-issues/helperutilities.html

I should admit , that I have never used that.

Best Regards,

Strahil Nikolov

В петък, 26 юли 2019 г., 09:03:10 ч. Гринуич+3, Enrico <enrico.becchetti@pg.infn.it> написа:

Il 25/07/19 22:27, Strahil Nikolov ha scritto:

> Hi all,

>my ovirt cluster has got 3 Hypervisors runnig Centos 7.5.1804 vdsm is 4.20.39.1-1.el7,
>ovirt engine is 4.2.4.5-1.el7, the storage systems are HP MSA P2000 and 2050 (fibre channel).

>I need to stop one of the hypervisors for maintenance but this system is the storage pool manager.

>For this reason I decided to manually activate SPM in one of the other nodes but this operation is not
successful.

Why would you do that ?

Just put the host in maintenance and the system will evacuate all VMs and another host will be promoted as SPM . 

Once the host is in maintenance - you can proceed with your tasks.

P.S: Don't forget to remove the mainteance mode once you are over.

Best Regards,

Strahil Nikolov

Hi Strahil,
I did not tell you that this is the first thing I tried but it didn't work because SPM can't be activated in another host,
then I tried to manually move SPM without success.
Any ideas ?
Thanks a lot !

Best Regards

Enrico

-- 
_______________________________________________________________________

Enrico Becchetti                    Servizio di Calcolo e Reti

Istituto Nazionale di Fisica Nucleare - Sezione di Perugia
Via Pascoli,c/o Dipartimento di Fisica  06123 Perugia (ITALY)
Phone:+39 075 5852777             Mail: Enrico.Becchetti<at>pg.infn.it 
_______________________________________________________________________

-- 
_______________________________________________________________________

Enrico Becchetti                    Servizio di Calcolo e Reti

Istituto Nazionale di Fisica Nucleare - Sezione di Perugia
Via Pascoli,c/o Dipartimento di Fisica  06123 Perugia (ITALY)
Phone:+39 075 5852777             Mail: Enrico.Becchetti<at>pg.infn.it 
_______________________________________________________________________

-- 
_______________________________________________________________________

Enrico Becchetti                    Servizio di Calcolo e Reti

Istituto Nazionale di Fisica Nucleare - Sezione di Perugia
Via Pascoli,c/o Dipartimento di Fisica  06123 Perugia (ITALY)
Phone:+39 075 5852777             Mail: Enrico.Becchetti<at>pg.infn.it 
_______________________________________________________________________