[ovirt-users] Re: cannot activate host

On Fri, Aug 10, 2018 at 11:30 AM, Jiří Sléžka <jiri.slezka(a)slu.cz <mailto:jiri.slezka@slu.cz>> wrote: Hi, On 08/09/2018 11:25 PM, Raz Tamir wrote: > You can try using vdsm-client to check if there are running tasks on the > host using this command: > >     vdsm-client Host getAllTasksInfo thanks for support, I tried this on affected host [root@blade01 ~]# vdsm-client Host getAllTasksInfo vdsm-client: Command Host.getAllTasksInfo with args {} failed: (code=654, message=Not SPM: ()) so I moved into SPM host (blade03 at this time) [root@blade03 ~]# vdsm-client Host getAllTasksInfo {} I tried also [root@blade01 ~]# vdsm-client Host getAllTasks {} [root@blade01 ~]# vdsm-client Host getJobs {} [root@blade03 ~]# vdsm-client Host getAllTasks {} [root@blade03 ~]# vdsm-client Host getJobs {} It looks like there are no tasks/jobs running on hosts. It looks like something is stucked in engine db. At the beginning of this issue, there were some race condition where blade01 had two running tasks/jobs which cannot be finished (network configuration and activating of host). I deleted this jobs manually in db via SELECT * FROM job ORDER BY start_time DESC; SELECT DeleteJob('...job_id...'); SELECT DeleteJob('...job_id...'); Maybe it was not wise... but what can I do now to solve this? :-) Yes it can be bad to mess with the DB. My best suugestion at the momet is to try and re-add this host.

  Cheers, Jiri > > If there are running tasks, you can stop them: > >     vdsm-client Task stop taskID=xxx-yyy > > If there are finished tasks, you can clear them: > >     vdsm-client Task clear taskID=xxx-yyy > > > On Thu, Aug 9, 2018, 16:15 Jiří Sléžka <jiri.slezka(a)slu.cz <mailto:jiri.slezka@slu.cz> > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>> wrote: > >     Hello, > >     still no luck with solving this issue. > >     I cannot even remove this host. > >     engine log is now spammed with this messages > >     2018-08-09 15:03:08,410+02 INFO >     [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >     (EE-ManagedThreadFactory-engine-Thread-1057) >     [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait >     lock >     'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', >     sharedLocks=''}' >     2018-08-09 15:03:08,446+02 INFO >     [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >     (EE-ManagedThreadFactory-engine-Thread-1057) >     [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait >     lock >     'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', >     sharedLocks=''}' >     2018-08-09 15:03:08,451+02 INFO >     [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >     (EE-ManagedThreadFactory-engine-Thread-1057) >     [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait >     lock >     'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', >     sharedLocks=''}' >     2018-08-09 15:03:08,486+02 INFO >     [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >     (EE-ManagedThreadFactory-engine-Thread-1057) >     [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait >     lock >     'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', >     sharedLocks=''}' > >     btw. oVirt 4.2.5.2-1.el7/CentOS7 > >     Any tips how to resolve this and at least remove this host from ovirt? > >     Cheers, Jiri > > >     On 08/06/2018 12:52 PM, Gobinda Das wrote: >     > Can you please post vdsm log which is inside /var/log/vdsm/vdsm.log ? >     > >     > On Mon, Aug 6, 2018 at 3:51 PM, Jiří Sléžka <jiri.slezka(a)slu.cz <mailto:jiri.slezka@slu.cz> >     <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> >     > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz> <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>>> wrote: >     > >     >     Hi, >     > >     >     no one can help? >     > >     >     I still cannot activate this host - error is "Cannot activate >     Host. >     >     Related operation is currently in progress. Please try again >     later." >     > >     >     I believe relevant log entrieas are >     > >     >     2018-08-06 12:15:50,398+02 INFO >     >     [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) >     >     [9387077a-8276-4a3f-a087-584a10a09b08] Failed to Acquire Lock >     to object >     >    >      'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]', >     >     sharedLocks=''}' >     >     2018-08-06 12:15:50,398+02 WARN >     >     [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) >     >     [9387077a-8276-4a3f-a087-584a10a09b08] Validation of action >     >     'ActivateVds' failed for user ***my_username***. Reasons: >     >    >      VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED >     > >     > >     >     when I tried to "reinstall host" I got entry in Events "Failed >     to update >     >     Host blade01" and correlation id. Relevant log entries are >     > >     > >     >     cat /var/log/ovirt-engine/engine.log | grep >     >     d2b332da-9050-419c-8188-8fdda5e9d807 >     > >     >     2018-08-06 12:08:57,451+02 INFO >     >     [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default >     >     task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Running command: >     >     InstallVdsCommand internal: false. Entities affected :  ID: >     >     786646cd-c9ef-49a8-8aea-56e858dcf202 Type: VDSAction group >     >     EDIT_HOST_CONFIGURATION with role type ADMIN >     >     2018-08-06 12:08:57,476+02 INFO >     >     [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] >     >     (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] >     Failed to >     >     Acquire Lock to object >     >    >      'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]', >     >     sharedLocks=''}' >     >     2018-08-06 12:08:57,476+02 WARN >     >     [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] >     >     (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] >     Validation of >     >     action 'InstallVdsInternal' failed for user ***my_username***. >     Reasons: >     >     ACTION_TYPE_FAILED_OBJECT_LOCKED >     >     2018-08-06 12:08:57,479+02 ERROR >     >     [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default >     >     task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] >     Installation/upgrade of >     >     Host '786646cd-c9ef-49a8-8aea-56e858dcf202', 'blade01' failed: >     Cannot >     >     ${action} ${type}. Related operation is currently in progress. >     Please >     >     try again later. >     >     2018-08-06 12:08:57,482+02 INFO >     >     [org.ovirt.engine.core.bll.CommandCompensator] (default task-425) >     >     [d2b332da-9050-419c-8188-8fdda5e9d807] Command >     >     [id=b74adfbb-c043-4fa9-a163-0a6ea7fba9dc]: Compensating >     >     DELETED_OR_UPDATED_ENTITY of >     >     org.ovirt.engine.core.common.businessentities.VdsStatic; snapshot: >     >     id=786646cd-c9ef-49a8-8aea-56e858dcf202. >     >     2018-08-06 12:08:57,486+02 ERROR >     >     [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default >     >     task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Transaction >     rolled-back >     >     for command >     'org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand'. >     >     2018-08-06 12:08:57,491+02 ERROR >     >    >      [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >     >     (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] >     EVENT_ID: >     >     USER_FAILED_UPDATE_VDS(105), Failed to update Host blade01 (User: >     >     ***my_username***). >     > >     >     How can I unlock this entries? >     > >     >     Thanks in advance, >     > >     >     Jiri Slezka >     > >     > >     > >     >     On 08/03/2018 02:07 PM, Jiří Sléžka wrote: >     >     > Hi, >     >     > >     >     > I have one interesting issue. I was asked by colleague to >     repair his >     >     > cluster which is part of our oVirt installation. He tried to >     >     upgrade his >     >     > host to gain 4.2 compatibility level but it fails in some >     cases. Never >     >     > mind, I checked it, repaired, installed upgrades and >     activate his >     >     hosts >     >     > with one exception. >     >     > >     >     > One of hosts was stucked in finishing two tasks. One was >     changing >     >     > network configuration, the second was activating of this host. I >     >     have to >     >     > manually clear this tasks in database by >     >     > >     >     > SELECT * FROM job ORDER BY start_time DESC; >     >     > SELECT DeleteJob('...job_id...'); >     >     > SELECT DeleteJob('...job_id...'); >     >     > >     >     > Now this tasks are gone but still cannot activate this host. >     >     > >     >     > >     >     > Error while executing action: >     >     > >     >     > blade01: >     >     > >     >     >     Cannot activate Host. Related operation is currently in >     progress. >     >     > Please try again later. >     >     > >     >     > Host is updatet (latest 4.2 repo), rebooted, looks fine... >     >     > >     >     > In tasks in UI are no tasks found (the same in db)... >     >     > >     >     > What can I do now? >     >     > >     >     > Cheers, Jiri >     >     > >     >     > >     >     > >     >     > >     >     > _______________________________________________ >     >     > Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> >     <mailto:users@ovirt.org <mailto:users@ovirt.org>> <mailto:users@ovirt.org <mailto:users@ovirt.org> >     <mailto:users@ovirt.org <mailto:users@ovirt.org>>> >     >     > To unsubscribe send an email to users-leave(a)ovirt.org <mailto:users-leave@ovirt.org> >     <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> >     >     <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>>> >     >     > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> >     >     <https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/>> >     >     > oVirt Code of Conduct: >     >     https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> >     >     <https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/>> >     >     > List Archives: >     >    >      https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/ <https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEES... >     >    >      <https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/ <https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEES... >     >     > >     > >     > >     > >     >     _______________________________________________ >     >     Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>> >     <mailto:users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>>> >     >     To unsubscribe send an email to users-leave(a)ovirt.org <mailto:users-leave@ovirt.org> >     <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> >     >     <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>>> >     >     Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> >     >     <https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/>> >     >     oVirt Code of Conduct: >     >     https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> >     >     <https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/>> >     >     List Archives: >     >    >      https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/ <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37... >     >    >      <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/ <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37... >     > >     > >     > >     > >     > -- >     > Thanks, >     > Gobinda > > >     _______________________________________________ >     Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>> >     To unsubscribe send an email to users-leave(a)ovirt.org <mailto:users-leave@ovirt.org> >     <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> >     Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> >     oVirt Code of Conduct: >     https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> >     List Archives: >     https://lists.ovirt.org/archives/list/users@ovirt.org/message/BP7SG5BQKZIVR75A6NSCUMGQQBXYL3FJ/ <https://lists.ovirt.org/archives/list/users@ovirt.org/message/BP7SG5BQKZI... > -- Raz Tamir Manager, RHV QE