Do you have vdsm logs?

Is this task still running?

On Thu, Jul 25, 2019 at 5:00 PM Enrico <enrico.becchetti(a)pg.infn.it <mailto:enrico.becchetti@pg.infn.it>> wrote:  Hi all, my ovirt cluster has got 3 Hypervisors runnig Centos 7.5.1804 vdsm is 4.20.39.1-1.el7, ovirt engine is 4.2.4.5-1.el7, the storage systems are HP MSA P2000 and 2050 (fibre channel). I need to stop one of the hypervisors for maintenance but this system is the storage pool manager. For this reason I decided to manually activate SPM in one of the other nodes but this operation is not successful. In the ovirt engine (engine.log) the error is this: 2019-07-25 12:39:16,744+02 INFO [org.ovirt.engine.core.bll.storage.pool.ForceSelectSPMCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] Running command: ForceSelectSPMCommand internal: false. Entities affected :  ID: 81c9bd3c-ae0a-467f-bf7f-63ab30cd8d9e Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2019-07-25 12:39:16,745+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, SpmStopOnIrsVDSCommand( SpmStopOnIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false'}), log id: 37bf4639 2019-07-25 12:39:16,747+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, ResetIrsVDSCommand( ResetIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', vdsId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', ignoreStopFailed='false'}), log id: 2522686f 2019-07-25 12:39:16,749+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, SpmStopVDSCommand(HostName = infn-vm05.management, SpmStopVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7'}), log id: 1810fd8b 2019-07-25 12:39:16,758+02 *ERROR* [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] SpmStopVDSCommand::Not stopping SPM on vds 'infn-vm05.management', pool id '18d57688-6ed4-43b8-bd7c-0665b55950b7' as there are uncleared tasks 'Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e', status 'running'' 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, SpmStopVDSCommand, log id: 1810fd8b 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, ResetIrsVDSCommand, log id: 2522686f 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, SpmStopOnIrsVDSCommand, log id: 37bf4639 2019-07-25 12:39:16,760+02 *ERROR* [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] EVENT_ID: USER_FORCE_SELECTED_SPM_STOP_FAILED(4,096), Failed to force select infn-vm07.management as the SPM due to a failure to stop the current SPM. while in the hypervisor (SPM) vdsm.log: 2019-07-25 12:39:16,744+02 INFO [org.ovirt.engine.core.bll.storage.pool.ForceSelectSPMCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] Running command: ForceSelectSPMCommand internal: false. Entities affected :  ID: 81c9bd3c-ae0a-467f-bf7f-63ab30cd8d9e Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2019-07-25 12:39:16,745+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, SpmStopOnIrsVDSCommand( SpmStopOnIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false'}), log id: 37bf4639 2019-07-25 12:39:16,747+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, ResetIrsVDSCommand( ResetIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', vdsId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', ignoreStopFailed='false'}), log id: 2522686f 2019-07-25 12:39:16,749+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, SpmStopVDSCommand(HostName = infn-vm05.management, SpmStopVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7'}), log id: 1810fd8b 2019-07-25 12:39:16,758+02 *ERROR* [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] SpmStopVDSCommand::Not stopping SPM on vds 'infn-vm05.management', pool id '18d57688-6ed4-43b8-bd7c-0665b55950b7' as there are uncleared tasks 'Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e', status 'running'' 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, SpmStopVDSCommand, log id: 1810fd8b 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, ResetIrsVDSCommand, log id: 2522686f 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, SpmStopOnIrsVDSCommand, log id: 37bf4639 2019-07-25 12:39:16,760+02 *ERROR* [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] EVENT_ID: USER_FORCE_SELECTED_SPM_STOP_FAILED(4,096), Failed to force select infn-vm07.management as the SPM due to a failure to stop the current SPM. 2019-07-25 12:39:18,660+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-25 12:39:18,660+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-25 12:39:18,750+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-25 12:39:18,750+02 *ERROR* [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] BaseAsyncTask::logEndTaskFailure: Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: 2019-07-25 12:39:18,750+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-25 12:39:18,751+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 34ae2b2f 2019-07-25 12:39:18,752+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: d3a78ad 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, HSMClearTaskVDSCommand, log id: d3a78ad 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, SPMClearTaskVDSCommand, log id: 34ae2b2f 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] BaseAsyncTask::onTaskEndSuccess: Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-25 12:39:18,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 42de0c2b 2019-07-25 12:39:18,759+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 4895c79c 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, HSMClearTaskVDSCommand, log id: 4895c79c 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, SPMClearTaskVDSCommand, log id: 42de0c2b 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Cleaning zombie tasks: Clearing async task 'Unknown' that started at 'Fri May 03 14:48:50 CEST 2019' 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-25 12:39:18,765+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: da77af2 2019-07-25 12:39:18,766+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 530694fb 2019-07-25 12:39:18,771+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, HSMClearTaskVDSCommand, log id: 530694fb 2019-07-25 12:39:18,771+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, SPMClearTaskVDSCommand, log id: da77af2 2019-07-25 12:39:18,771+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned there is some relation between this error and a task that has remained hanging, from SPM server: # vdsm-client Task  getInfo taskID=fdcf4d1b-82fe-49a6-b233-323ebe568f8e {     "verb": "prepareMerge",     "id": "fdcf4d1b-82fe-49a6-b233-323ebe568f8e" } # vdsm-client Task  getStatus taskID=fdcf4d1b-82fe-49a6-b233-323ebe568f8e {     "message": "running job 1 of 1",     "code": 0,     "taskID": "fdcf4d1b-82fe-49a6-b233-323ebe568f8e",     "taskResult": "",     "taskState": "running" } How can I solve this problem ? Thanks a lot for your help !! Best Regards Enrico -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it <http://pg.infn.it> _______________________________________________________________________ _______________________________________________ Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave(a)ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KQBFVUCEFAA...

can you grep vdsm logs to see if it is actually running?

you can use vdsm-client Task stop taskID=... and then vdsm-clien Task clear taskID=... but if it's actually running it can leave the system in an undesired state, so be sure to check On Thu, Jul 25, 2019 at 5:58 PM Enrico <enrico.becchetti(a)pg.infn.it <mailto:enrico.becchetti@pg.infn.it>> wrote: Il 25/07/19 16:45, Benny Zlotnik ha scritto: > Do you have vdsm logs? I'M not sure because this task is very old > Is this task still running? I made this : # vdsm-client Task  getStatus taskID=fdcf4d1b-82fe-49a6-b233-323ebe568f8e {     "message": "running job 1 of 1",     "code": 0,     "taskID": "fdcf4d1b-82fe-49a6-b233-323ebe568f8e",     "taskResult": "",     "taskState": "running" }  are there any other tools to check it ? Thanks Enrico > > On Thu, Jul 25, 2019 at 5:00 PM Enrico > <enrico.becchetti(a)pg.infn.it > <mailto:enrico.becchetti@pg.infn.it>> wrote: > >  Hi all, > my ovirt cluster has got 3 Hypervisors runnig Centos 7.5.1804 > vdsm is 4.20.39.1-1.el7, > ovirt engine is 4.2.4.5-1.el7, the storage systems are HP MSA > P2000 and 2050 (fibre channel). > > I need to stop one of the hypervisors for maintenance but > this system is the storage pool manager. > > For this reason I decided to manually activate SPM in one of > the other nodes but this operation is not > successful. > > In the ovirt engine (engine.log) the error is this: > > 2019-07-25 12:39:16,744+02 INFO > [org.ovirt.engine.core.bll.storage.pool.ForceSelectSPMCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > Running command: ForceSelectSPMCommand internal: false. > Entities affected :  ID: 81c9bd3c-ae0a-467f-bf7f-63ab30cd8d9e > Type: VDSAction group MANIPULATE_HOST with role type ADMIN > 2019-07-25 12:39:16,745+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > START, SpmStopOnIrsVDSCommand( > SpmStopOnIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', > ignoreFailoverLimit='false'}), log id: 37bf4639 > 2019-07-25 12:39:16,747+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > START, ResetIrsVDSCommand( > ResetIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', > ignoreFailoverLimit='false', > vdsId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', > ignoreStopFailed='false'}), log id: 2522686f > 2019-07-25 12:39:16,749+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > START, SpmStopVDSCommand(HostName = infn-vm05.management, > SpmStopVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', > storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7'}), log > id: 1810fd8b > 2019-07-25 12:39:16,758+02 *ERROR* > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > SpmStopVDSCommand::Not stopping SPM on vds > 'infn-vm05.management', pool id > '18d57688-6ed4-43b8-bd7c-0665b55950b7' as there are uncleared > tasks 'Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e', status > 'running'' > 2019-07-25 12:39:16,758+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > FINISH, SpmStopVDSCommand, log id: 1810fd8b > 2019-07-25 12:39:16,758+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > FINISH, ResetIrsVDSCommand, log id: 2522686f > 2019-07-25 12:39:16,758+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > FINISH, SpmStopOnIrsVDSCommand, log id: 37bf4639 > 2019-07-25 12:39:16,760+02 *ERROR* > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > EVENT_ID: USER_FORCE_SELECTED_SPM_STOP_FAILED(4,096), Failed > to force select infn-vm07.management as the SPM due to a > failure to stop the current SPM. > > while in the hypervisor (SPM) vdsm.log: > > 2019-07-25 12:39:16,744+02 INFO > [org.ovirt.engine.core.bll.storage.pool.ForceSelectSPMCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > Running command: ForceSelectSPMCommand internal: false. > Entities affected :  ID: 81c9bd3c-ae0a-467f-bf7f-63ab30cd8d9e > Type: VDSAction group MANIPULATE_HOST with role type ADMIN > 2019-07-25 12:39:16,745+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > START, SpmStopOnIrsVDSCommand( > SpmStopOnIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', > ignoreFailoverLimit='false'}), log id: 37bf4639 > 2019-07-25 12:39:16,747+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > START, ResetIrsVDSCommand( > ResetIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', > ignoreFailoverLimit='false', > vdsId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', > ignoreStopFailed='false'}), log id: 2522686f > 2019-07-25 12:39:16,749+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > START, SpmStopVDSCommand(HostName = infn-vm05.management, > SpmStopVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', > storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7'}), log > id: 1810fd8b > 2019-07-25 12:39:16,758+02 *ERROR* > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > SpmStopVDSCommand::Not stopping SPM on vds > 'infn-vm05.management', pool id > '18d57688-6ed4-43b8-bd7c-0665b55950b7' as there are uncleared > tasks 'Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e', status > 'running'' > 2019-07-25 12:39:16,758+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > FINISH, SpmStopVDSCommand, log id: 1810fd8b > 2019-07-25 12:39:16,758+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > FINISH, ResetIrsVDSCommand, log id: 2522686f > 2019-07-25 12:39:16,758+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > FINISH, SpmStopOnIrsVDSCommand, log id: 37bf4639 > 2019-07-25 12:39:16,760+02 *ERROR* > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > EVENT_ID: USER_FORCE_SELECTED_SPM_STOP_FAILED(4,096), Failed > to force select infn-vm07.management as the SPM due to a > failure to stop the current SPM. > 2019-07-25 12:39:18,660+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task > id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed > pre-polling period time and should be polled. Pre-polling > period is 60000 millis. > 2019-07-25 12:39:18,660+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task > id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed > pre-polling period time and should be polled. Pre-polling > period is 60000 millis. > 2019-07-25 12:39:18,750+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task > id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed > pre-polling period time and should be polled. Pre-polling > period is 60000 millis. > 2019-07-25 12:39:18,750+02 *ERROR* > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > BaseAsyncTask::logEndTaskFailure: Task > 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command > 'Unknown', Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') > ended with failure: > 2019-07-25 12:39:18,750+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > SPMAsyncTask::ClearAsyncTask: Attempting to clear task > 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' > 2019-07-25 12:39:18,751+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, > SPMClearTaskVDSCommand( > SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', > ignoreFailoverLimit='false', > taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 34ae2b2f > 2019-07-25 12:39:18,752+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, > HSMClearTaskVDSCommand(HostName = infn-vm05.management, > HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', > taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: d3a78ad > 2019-07-25 12:39:18,757+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > FINISH, HSMClearTaskVDSCommand, log id: d3a78ad > 2019-07-25 12:39:18,757+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > FINISH, SPMClearTaskVDSCommand, log id: 34ae2b2f > 2019-07-25 12:39:18,757+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > SPMAsyncTask::ClearAsyncTask: At time of attempt to clear > task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code > was 'TaskStateError' and message was 'Operation is not > allowed in this task state: ("can't clean in state > running",)'. Task will not be cleaned > 2019-07-25 12:39:18,757+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > BaseAsyncTask::onTaskEndSuccess: Task > 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command > 'Unknown', Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') > ended successfully. > 2019-07-25 12:39:18,757+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > SPMAsyncTask::ClearAsyncTask: Attempting to clear task > 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' > 2019-07-25 12:39:18,758+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, > SPMClearTaskVDSCommand( > SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', > ignoreFailoverLimit='false', > taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 42de0c2b > 2019-07-25 12:39:18,759+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, > HSMClearTaskVDSCommand(HostName = infn-vm05.management, > HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', > taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 4895c79c > 2019-07-25 12:39:18,764+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > FINISH, HSMClearTaskVDSCommand, log id: 4895c79c > 2019-07-25 12:39:18,764+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > FINISH, SPMClearTaskVDSCommand, log id: 42de0c2b > 2019-07-25 12:39:18,764+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > SPMAsyncTask::ClearAsyncTask: At time of attempt to clear > task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code > was 'TaskStateError' and message was 'Operation is not > allowed in this task state: ("can't clean in state > running",)'. Task will not be cleaned > 2019-07-25 12:39:18,764+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task > id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed > pre-polling period time and should be polled. Pre-polling > period is 60000 millis. > 2019-07-25 12:39:18,764+02 INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > Cleaning zombie tasks: Clearing async task 'Unknown' that > started at 'Fri May 03 14:48:50 CEST 2019' > 2019-07-25 12:39:18,764+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > SPMAsyncTask::ClearAsyncTask: Attempting to clear task > 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' > 2019-07-25 12:39:18,765+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, > SPMClearTaskVDSCommand( > SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', > ignoreFailoverLimit='false', > taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: da77af2 > 2019-07-25 12:39:18,766+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, > HSMClearTaskVDSCommand(HostName = infn-vm05.management, > HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', > taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 530694fb > 2019-07-25 12:39:18,771+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > FINISH, HSMClearTaskVDSCommand, log id: 530694fb > 2019-07-25 12:39:18,771+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > FINISH, SPMClearTaskVDSCommand, log id: da77af2 > 2019-07-25 12:39:18,771+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > SPMAsyncTask::ClearAsyncTask: At time of attempt to clear > task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code > was 'TaskStateError' and message was 'Operation is not > allowed in this task state: ("can't clean in state > running",)'. Task will not be cleaned > > there is some relation between this error and a task that has > remained hanging, from SPM server: > > # vdsm-client Task  getInfo > taskID=fdcf4d1b-82fe-49a6-b233-323ebe568f8e > { >     "verb": "prepareMerge", >     "id": "fdcf4d1b-82fe-49a6-b233-323ebe568f8e" > } > > # vdsm-client Task  getStatus > taskID=fdcf4d1b-82fe-49a6-b233-323ebe568f8e > { >     "message": "running job 1 of 1", >     "code": 0, >     "taskID": "fdcf4d1b-82fe-49a6-b233-323ebe568f8e", >     "taskResult": "", >     "taskState": "running" > } > > How can I solve this problem ? > > Thanks a lot for your help !! > Best Regards > Enrico > > -- > _______________________________________________________________________ > > Enrico Becchetti Servizio di Calcolo e Reti > > Istituto Nazionale di Fisica Nucleare - Sezione di Perugia > Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) > Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it <http://pg.infn.it> > _______________________________________________________________________ > > _______________________________________________ > Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> > To unsubscribe send an email to users-leave(a)ovirt.org > <mailto:users-leave@ovirt.org> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/KQBFVUCEFAA... > -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it <http://pg.infn.it> _______________________________________________________________________

> Hi all, >my ovirt cluster has got 3 Hypervisors runnig Centos 7.5.1804 vdsm is 4.20.39.1-1.el7, >ovirt engine is 4.2.4.5-1.el7, the storage systems are HP MSA P2000 and 2050 (fibre channel). >I need to stop one of the hypervisors for maintenance but this system is the storage pool manager. >For this reason I decided to manually activate SPM in one of the other nodes but this operation is not successful. Why would you do that ? Just put the host in maintenance and the system will evacuate all VMs and another host will be promoted as SPM . Once the host is in maintenance - you can proceed with your tasks. P.S: Don't forget to remove the mainteance mode once you are over. Best Regards, Strahil Nikolov

Have you tried to remove your task via the taskcleaner.sh You can find some details on https://www.ovirt.org/develop/developer-guide/db-issues/helperutilities.html I should admit , that I have never used that. Best Regards, Strahil Nikolov В петък, 26 юли 2019 г., 09:03:10 ч. Гринуич+3, Enrico <enrico.becchetti(a)pg.infn.it&gt; написа: Il 25/07/19 22:27, Strahil Nikolov ha scritto: > Hi all, >my ovirt cluster has got 3 Hypervisors runnig Centos 7.5.1804 vdsm is 4.20.39.1-1.el7, >ovirt engine is 4.2.4.5-1.el7, the storage systems are HP MSA P2000 and 2050 (fibre channel). >I need to stop one of the hypervisors for maintenance but this system is the storage pool manager. >For this reason I decided to manually activate SPM in one of the other nodes but this operation is not successful. Why would you do that ? Just put the host in maintenance and the system will evacuate all VMs and another host will be promoted as SPM . Once the host is in maintenance - you can proceed with your tasks. P.S: Don't forget to remove the mainteance mode once you are over. Best Regards, Strahil Nikolov Hi Strahil, I did not tell you that this is the first thing I tried but it didn't work because SPM can't be activated in another host, then I tried to manually move SPM without success. Any ideas ? Thanks a lot ! Best Regards Enrico -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it _______________________________________________________________________

taskcleaner.sh only clears tasks form the engine database. did you check your engine logs to see if this task is running?

it's a task that is executed during a snapshot merge (removal of a snapshot), do you have any running snapshot removals?

if not you can stop and clear the task using vdsm-client On Fri, Jul 26, 2019 at 1:31 PM Enrico <enrico.becchetti(a)pg.infn.it <mailto:enrico.becchetti@pg.infn.it>> wrote: Dear all, I try this: # /usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh -v -t fdcf4d1b-82fe-49a6-b233-323ebe568f8e select exists (select * from information_schema.tables where table_schema = 'public' and table_name = 'command_entities');  t  This will remove the given Task!!! Caution, this operation should be used with care. Please contact support prior to running this command Are you sure you want to proceed? [y/n] y SELECT Deleteasync_tasks('fdcf4d1b-82fe-49a6-b233-323ebe568f8e');                  0 # /usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh -v -R select exists (select * from information_schema.tables where table_schema = 'public' and table_name = 'command_entities');  t  This will remove all async_tasks table content!!! Caution, this operation should be used with care. Please contact support prior to running this command Are you sure you want to proceed? [y/n] y TRUNCATE TABLE async_tasks cascade; TRUNCATE TABLE but after these commands I see the same messages inside engine.log: 2019-07-26 12:25:19,727+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-26 12:25:19,727+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-26 12:25:19,779+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-26 12:25:19,779+02 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] BaseAsyncTask::logEndTaskFailure: Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: 2019-07-26 12:25:19,779+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-26 12:25:19,780+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 753de6fe 2019-07-26 12:25:19,781+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: e5dc020 2019-07-26 12:25:19,786+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned 2019-07-26 12:25:19,786+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] BaseAsyncTask::onTaskEndSuccess: Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2019-07-26 12:25:19,786+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-26 12:25:19,787+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 40383d58 2019-07-26 12:25:19,788+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 758d79b 2019-07-26 12:25:19,793+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned 2019-07-26 12:25:19,793+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-26 12:25:19,793+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-26 12:25:19,794+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 1dc8cb98 2019-07-26 12:25:19,795+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 7df444c1 2019-07-26 12:25:19,800+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned it's seems to be task id fdcf4d1b-82fe-49a6-b233-323ebe568f8e  is still running. Any other ideas ? Thank you so much. Best Regards Enrico Il 26/07/19 09:39, Strahil Nikolov ha scritto: > Have you tried to remove your task via the taskcleaner.sh > You can find some details on > https://www.ovirt.org/develop/developer-guide/db-issues/helperutilities.html > > I should admit , that I have never used that. > > Best Regards, > Strahil Nikolov > > В петък, 26 юли 2019 г., 09:03:10 ч. Гринуич+3, Enrico > <enrico.becchetti(a)pg.infn.it&gt; > <mailto:enrico.becchetti@pg.infn.it> написа: > > > Il 25/07/19 22:27, Strahil Nikolov ha scritto: > > Hi all, > >my ovirt cluster has got 3 Hypervisors runnig Centos 7.5.1804 > vdsm is 4.20.39.1-1.el7, > >ovirt engine is 4.2.4.5-1.el7, the storage systems are HP MSA > P2000 and 2050 (fibre channel). > > >I need to stop one of the hypervisors for maintenance but this > system is the storage pool manager. > > >For this reason I decided to manually activate SPM in one of the > other nodes but this operation is not > successful. > > Why would you do that ? > Just put the host in maintenance and the system will evacuate all > VMs and another host will be promoted as SPM . > Once the host is in maintenance - you can proceed with your tasks. > > P.S: Don't forget to remove the mainteance mode once you are over. > > Best Regards, > Strahil Nikolov > Hi Strahil, > I did not tell you that this is the first thing I tried but it > didn't work because SPM can't be activated in another host, > then I tried to manually move SPM without success. > Any ideas ? > Thanks a lot ! > > Best Regards > > Enrico > > -- > _______________________________________________________________________ > > Enrico Becchetti Servizio di Calcolo e Reti > > Istituto Nazionale di Fisica Nucleare - Sezione di Perugia > Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) > Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it <http://pg.infn.it> > _______________________________________________________________________ -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it <http://pg.infn.it> _______________________________________________________________________ _______________________________________________ Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave(a)ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/VQJCGFXAA37...

In my lab I would restart the vdsm.service , but as you are in prod - you should not to. It could cause the node get fenced and all the running VMs will be killed. Have you manually migrated all VMs to the other host ? If yes, then you can try this risky approach. Otherwise I'm out of ideas. Maybe a restart of the HostedEngine/Engine will help...

Best Regards, Strahil Nikolov В петък, 26 юли 2019 г., 13:28:13 ч. Гринуич+3, Enrico <enrico.becchetti(a)pg.infn.it&gt; написа: Dear all, I try this: # /usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh -v -t fdcf4d1b-82fe-49a6-b233-323ebe568f8e select exists (select * from information_schema.tables where table_schema = 'public' and table_name = 'command_entities');  t  This will remove the given Task!!! Caution, this operation should be used with care. Please contact support prior to running this command Are you sure you want to proceed? [y/n] y SELECT Deleteasync_tasks('fdcf4d1b-82fe-49a6-b233-323ebe568f8e');                  0 # /usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh -v -R select exists (select * from information_schema.tables where table_schema = 'public' and table_name = 'command_entities');  t  This will remove all async_tasks table content!!! Caution, this operation should be used with care. Please contact support prior to running this command Are you sure you want to proceed? [y/n] y TRUNCATE TABLE async_tasks cascade; TRUNCATE TABLE but after these commands I see the same messages inside engine.log: 2019-07-26 12:25:19,727+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-26 12:25:19,727+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-26 12:25:19,779+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-26 12:25:19,779+02 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] BaseAsyncTask::logEndTaskFailure: Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: 2019-07-26 12:25:19,779+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-26 12:25:19,780+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 753de6fe 2019-07-26 12:25:19,781+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: e5dc020 2019-07-26 12:25:19,786+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned 2019-07-26 12:25:19,786+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] BaseAsyncTask::onTaskEndSuccess: Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2019-07-26 12:25:19,786+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-26 12:25:19,787+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 40383d58 2019-07-26 12:25:19,788+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 758d79b 2019-07-26 12:25:19,793+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned 2019-07-26 12:25:19,793+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-26 12:25:19,793+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-26 12:25:19,794+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 1dc8cb98 2019-07-26 12:25:19,795+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 7df444c1 2019-07-26 12:25:19,800+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-77) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned it's seems to be task id fdcf4d1b-82fe-49a6-b233-323ebe568f8e  is still running. Any other ideas ? Thank you so much. Best Regards Enrico Il 26/07/19 09:39, Strahil Nikolov ha scritto: Have you tried to remove your task via the taskcleaner.sh You can find some details on https://www.ovirt.org/develop/developer-guide/db-issues/helperutilities.html I should admit , that I have never used that. Best Regards, Strahil Nikolov В петък, 26 юли 2019 г., 09:03:10 ч. Гринуич+3, Enrico <enrico.becchetti(a)pg.infn.it&gt; <mailto:enrico.becchetti@pg.infn.it> написа: Il 25/07/19 22:27, Strahil Nikolov ha scritto: > Hi all, >my ovirt cluster has got 3 Hypervisors runnig Centos 7.5.1804 vdsm is 4.20.39.1-1.el7, >ovirt engine is 4.2.4.5-1.el7, the storage systems are HP MSA P2000 and 2050 (fibre channel). >I need to stop one of the hypervisors for maintenance but this system is the storage pool manager. >For this reason I decided to manually activate SPM in one of the other nodes but this operation is not successful. Why would you do that ? Just put the host in maintenance and the system will evacuate all VMs and another host will be promoted as SPM . Once the host is in maintenance - you can proceed with your tasks. P.S: Don't forget to remove the mainteance mode once you are over. Best Regards, Strahil Nikolov Hi Strahil, I did not tell you that this is the first thing I tried but it didn't work because SPM can't be activated in another host, then I tried to manually move SPM without success. Any ideas ? Thanks a lot ! Best Regards Enrico -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it _______________________________________________________________________ -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it _______________________________________________________________________