[ovirt-users] Re: SPM and Task error ...

can you grep vdsm logs to see if it is actually running?

you can use vdsm-client Task stop taskID=... and then vdsm-clien Task clear taskID=... but if it's actually running it can leave the system in an undesired state, so be sure to check On Thu, Jul 25, 2019 at 5:58 PM Enrico <enrico.becchetti(a)pg.infn.it <mailto:enrico.becchetti@pg.infn.it>> wrote: Il 25/07/19 16:45, Benny Zlotnik ha scritto: > Do you have vdsm logs? I'M not sure because this task is very old > Is this task still running? I made this : # vdsm-client Task  getStatus taskID=fdcf4d1b-82fe-49a6-b233-323ebe568f8e {     "message": "running job 1 of 1",     "code": 0,     "taskID": "fdcf4d1b-82fe-49a6-b233-323ebe568f8e",     "taskResult": "",     "taskState": "running" }  are there any other tools to check it ? Thanks Enrico > > On Thu, Jul 25, 2019 at 5:00 PM Enrico > <enrico.becchetti(a)pg.infn.it > <mailto:enrico.becchetti@pg.infn.it>> wrote: > >  Hi all, > my ovirt cluster has got 3 Hypervisors runnig Centos 7.5.1804 > vdsm is 4.20.39.1-1.el7, > ovirt engine is 4.2.4.5-1.el7, the storage systems are HP MSA > P2000 and 2050 (fibre channel). > > I need to stop one of the hypervisors for maintenance but > this system is the storage pool manager. > > For this reason I decided to manually activate SPM in one of > the other nodes but this operation is not > successful. > > In the ovirt engine (engine.log) the error is this: > > 2019-07-25 12:39:16,744+02 INFO > [org.ovirt.engine.core.bll.storage.pool.ForceSelectSPMCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > Running command: ForceSelectSPMCommand internal: false. > Entities affected :  ID: 81c9bd3c-ae0a-467f-bf7f-63ab30cd8d9e > Type: VDSAction group MANIPULATE_HOST with role type ADMIN > 2019-07-25 12:39:16,745+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > START, SpmStopOnIrsVDSCommand( > SpmStopOnIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', > ignoreFailoverLimit='false'}), log id: 37bf4639 > 2019-07-25 12:39:16,747+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > START, ResetIrsVDSCommand( > ResetIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', > ignoreFailoverLimit='false', > vdsId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', > ignoreStopFailed='false'}), log id: 2522686f > 2019-07-25 12:39:16,749+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > START, SpmStopVDSCommand(HostName = infn-vm05.management, > SpmStopVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', > storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7'}), log > id: 1810fd8b > 2019-07-25 12:39:16,758+02 *ERROR* > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > SpmStopVDSCommand::Not stopping SPM on vds > 'infn-vm05.management', pool id > '18d57688-6ed4-43b8-bd7c-0665b55950b7' as there are uncleared > tasks 'Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e', status > 'running'' > 2019-07-25 12:39:16,758+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > FINISH, SpmStopVDSCommand, log id: 1810fd8b > 2019-07-25 12:39:16,758+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > FINISH, ResetIrsVDSCommand, log id: 2522686f > 2019-07-25 12:39:16,758+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > FINISH, SpmStopOnIrsVDSCommand, log id: 37bf4639 > 2019-07-25 12:39:16,760+02 *ERROR* > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > EVENT_ID: USER_FORCE_SELECTED_SPM_STOP_FAILED(4,096), Failed > to force select infn-vm07.management as the SPM due to a > failure to stop the current SPM. > > while in the hypervisor (SPM) vdsm.log: > > 2019-07-25 12:39:16,744+02 INFO > [org.ovirt.engine.core.bll.storage.pool.ForceSelectSPMCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > Running command: ForceSelectSPMCommand internal: false. > Entities affected :  ID: 81c9bd3c-ae0a-467f-bf7f-63ab30cd8d9e > Type: VDSAction group MANIPULATE_HOST with role type ADMIN > 2019-07-25 12:39:16,745+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > START, SpmStopOnIrsVDSCommand( > SpmStopOnIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', > ignoreFailoverLimit='false'}), log id: 37bf4639 > 2019-07-25 12:39:16,747+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > START, ResetIrsVDSCommand( > ResetIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', > ignoreFailoverLimit='false', > vdsId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', > ignoreStopFailed='false'}), log id: 2522686f > 2019-07-25 12:39:16,749+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > START, SpmStopVDSCommand(HostName = infn-vm05.management, > SpmStopVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', > storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7'}), log > id: 1810fd8b > 2019-07-25 12:39:16,758+02 *ERROR* > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > SpmStopVDSCommand::Not stopping SPM on vds > 'infn-vm05.management', pool id > '18d57688-6ed4-43b8-bd7c-0665b55950b7' as there are uncleared > tasks 'Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e', status > 'running'' > 2019-07-25 12:39:16,758+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > FINISH, SpmStopVDSCommand, log id: 1810fd8b > 2019-07-25 12:39:16,758+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > FINISH, ResetIrsVDSCommand, log id: 2522686f > 2019-07-25 12:39:16,758+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > FINISH, SpmStopOnIrsVDSCommand, log id: 37bf4639 > 2019-07-25 12:39:16,760+02 *ERROR* > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] > EVENT_ID: USER_FORCE_SELECTED_SPM_STOP_FAILED(4,096), Failed > to force select infn-vm07.management as the SPM due to a > failure to stop the current SPM. > 2019-07-25 12:39:18,660+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task > id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed > pre-polling period time and should be polled. Pre-polling > period is 60000 millis. > 2019-07-25 12:39:18,660+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task > id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed > pre-polling period time and should be polled. Pre-polling > period is 60000 millis. > 2019-07-25 12:39:18,750+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task > id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed > pre-polling period time and should be polled. Pre-polling > period is 60000 millis. > 2019-07-25 12:39:18,750+02 *ERROR* > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > BaseAsyncTask::logEndTaskFailure: Task > 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command > 'Unknown', Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') > ended with failure: > 2019-07-25 12:39:18,750+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > SPMAsyncTask::ClearAsyncTask: Attempting to clear task > 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' > 2019-07-25 12:39:18,751+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, > SPMClearTaskVDSCommand( > SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', > ignoreFailoverLimit='false', > taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 34ae2b2f > 2019-07-25 12:39:18,752+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, > HSMClearTaskVDSCommand(HostName = infn-vm05.management, > HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', > taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: d3a78ad > 2019-07-25 12:39:18,757+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > FINISH, HSMClearTaskVDSCommand, log id: d3a78ad > 2019-07-25 12:39:18,757+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > FINISH, SPMClearTaskVDSCommand, log id: 34ae2b2f > 2019-07-25 12:39:18,757+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > SPMAsyncTask::ClearAsyncTask: At time of attempt to clear > task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code > was 'TaskStateError' and message was 'Operation is not > allowed in this task state: ("can't clean in state > running",)'. Task will not be cleaned > 2019-07-25 12:39:18,757+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > BaseAsyncTask::onTaskEndSuccess: Task > 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command > 'Unknown', Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') > ended successfully. > 2019-07-25 12:39:18,757+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > SPMAsyncTask::ClearAsyncTask: Attempting to clear task > 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' > 2019-07-25 12:39:18,758+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, > SPMClearTaskVDSCommand( > SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', > ignoreFailoverLimit='false', > taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 42de0c2b > 2019-07-25 12:39:18,759+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, > HSMClearTaskVDSCommand(HostName = infn-vm05.management, > HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', > taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 4895c79c > 2019-07-25 12:39:18,764+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > FINISH, HSMClearTaskVDSCommand, log id: 4895c79c > 2019-07-25 12:39:18,764+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > FINISH, SPMClearTaskVDSCommand, log id: 42de0c2b > 2019-07-25 12:39:18,764+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > SPMAsyncTask::ClearAsyncTask: At time of attempt to clear > task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code > was 'TaskStateError' and message was 'Operation is not > allowed in this task state: ("can't clean in state > running",)'. Task will not be cleaned > 2019-07-25 12:39:18,764+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task > id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed > pre-polling period time and should be polled. Pre-polling > period is 60000 millis. > 2019-07-25 12:39:18,764+02 INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > Cleaning zombie tasks: Clearing async task 'Unknown' that > started at 'Fri May 03 14:48:50 CEST 2019' > 2019-07-25 12:39:18,764+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > SPMAsyncTask::ClearAsyncTask: Attempting to clear task > 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' > 2019-07-25 12:39:18,765+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, > SPMClearTaskVDSCommand( > SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', > ignoreFailoverLimit='false', > taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: da77af2 > 2019-07-25 12:39:18,766+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, > HSMClearTaskVDSCommand(HostName = infn-vm05.management, > HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', > taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 530694fb > 2019-07-25 12:39:18,771+02 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > FINISH, HSMClearTaskVDSCommand, log id: 530694fb > 2019-07-25 12:39:18,771+02 INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > FINISH, SPMClearTaskVDSCommand, log id: da77af2 > 2019-07-25 12:39:18,771+02 INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] > SPMAsyncTask::ClearAsyncTask: At time of attempt to clear > task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code > was 'TaskStateError' and message was 'Operation is not > allowed in this task state: ("can't clean in state > running",)'. Task will not be cleaned > > there is some relation between this error and a task that has > remained hanging, from SPM server: > > # vdsm-client Task  getInfo > taskID=fdcf4d1b-82fe-49a6-b233-323ebe568f8e > { >     "verb": "prepareMerge", >     "id": "fdcf4d1b-82fe-49a6-b233-323ebe568f8e" > } > > # vdsm-client Task  getStatus > taskID=fdcf4d1b-82fe-49a6-b233-323ebe568f8e > { >     "message": "running job 1 of 1", >     "code": 0, >     "taskID": "fdcf4d1b-82fe-49a6-b233-323ebe568f8e", >     "taskResult": "", >     "taskState": "running" > } > > How can I solve this problem ? > > Thanks a lot for your help !! > Best Regards > Enrico > > -- > _______________________________________________________________________ > > Enrico Becchetti Servizio di Calcolo e Reti > > Istituto Nazionale di Fisica Nucleare - Sezione di Perugia > Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) > Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it <http://pg.infn.it> > _______________________________________________________________________ > > _______________________________________________ > Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> > To unsubscribe send an email to users-leave(a)ovirt.org > <mailto:users-leave@ovirt.org> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/KQBFVUCEFAA... > -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it <http://pg.infn.it> _______________________________________________________________________