[ovirt-users] Re: cannot activate host

Well, issue was solved by installing latest async update 4.2.5... Probably helped this part of upgrade process... ... [ INFO ] Cleaning async tasks and compensations [ INFO ] Unlocking existing entities

... Cheers, Jiri On 08/10/2018 11:03 AM, Jiří Sléžka wrote: > On 08/10/2018 10:52 AM, Raz Tamir wrote: >> >> >> On Fri, Aug 10, 2018 at 11:30 AM, Jiří Sléžka <jiri.slezka(a)slu.cz >> <mailto:jiri.slezka@slu.cz>> wrote: >> >> Hi, >> >> On 08/09/2018 11:25 PM, Raz Tamir wrote: >> > You can try using vdsm-client to check if there are running tasks on the >> > host using this command: >> > >> > vdsm-client Host getAllTasksInfo >> >> thanks for support, I tried this on affected host >> >> [root@blade01 ~]# vdsm-client Host getAllTasksInfo >> vdsm-client: Command Host.getAllTasksInfo with args {} failed: >> (code=654, message=Not SPM: ()) >> >> so I moved into SPM host (blade03 at this time) >> >> [root@blade03 ~]# vdsm-client Host getAllTasksInfo >> {} >> >> I tried also >> >> [root@blade01 ~]# vdsm-client Host getAllTasks >> {} >> [root@blade01 ~]# vdsm-client Host getJobs >> {} >> >> [root@blade03 ~]# vdsm-client Host getAllTasks >> {} >> [root@blade03 ~]# vdsm-client Host getJobs >> {} >> >> It looks like there are no tasks/jobs running on hosts. It looks like >> something is stucked in engine db. >> >> At the beginning of this issue, there were some race condition where >> blade01 had two running tasks/jobs which cannot be finished (network >> configuration and activating of host). I deleted this jobs manually in >> db via >> >> SELECT * FROM job ORDER BY start_time DESC; >> SELECT DeleteJob('...job_id...'); >> SELECT DeleteJob('...job_id...'); >> >> Maybe it was not wise... but what can I do now to solve this? :-) >> >> Yes it can be bad to mess with the DB. >> My best suugestion at the momet is to try and re-add this host. > > I would like to but I cannot even remove this host (Cannot remove Host. > Related operation is currently in progress. Please try again later.) :-( > But maybe I can try to add it as brand new one? > > But I am curious what checks do manager and why we cannot see the > tasks/jobs/locks/anything in db... > > Cheers, Jiri > > > >> >> >> >> Cheers, Jiri >> >> >> > >> > If there are running tasks, you can stop them: >> > >> > vdsm-client Task stop taskID=xxx-yyy >> > >> > If there are finished tasks, you can clear them: >> > >> > vdsm-client Task clear taskID=xxx-yyy >> > >> > >> > On Thu, Aug 9, 2018, 16:15 Jiří Sléžka <jiri.slezka(a)slu.cz <mailto:jiri.slezka@slu.cz> >> > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>> wrote: >> > >> > Hello, >> > >> > still no luck with solving this issue. >> > >> > I cannot even remove this host. >> > >> > engine log is now spammed with this messages >> > >> > 2018-08-09 15:03:08,410+02 INFO >> > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >> > (EE-ManagedThreadFactory-engine-Thread-1057) >> > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock >> and wait >> > lock >> > >> 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', >> > sharedLocks=''}' >> > 2018-08-09 15:03:08,446+02 INFO >> > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >> > (EE-ManagedThreadFactory-engine-Thread-1057) >> > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock >> and wait >> > lock >> > >> 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', >> > sharedLocks=''}' >> > 2018-08-09 15:03:08,451+02 INFO >> > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >> > (EE-ManagedThreadFactory-engine-Thread-1057) >> > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock >> and wait >> > lock >> > >> 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', >> > sharedLocks=''}' >> > 2018-08-09 15:03:08,486+02 INFO >> > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] >> > (EE-ManagedThreadFactory-engine-Thread-1057) >> > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock >> and wait >> > lock >> > >> 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', >> > sharedLocks=''}' >> > >> > btw. oVirt 4.2.5.2-1.el7/CentOS7 >> > >> > Any tips how to resolve this and at least remove this host >> from ovirt? >> > >> > Cheers, Jiri >> > >> > >> > On 08/06/2018 12:52 PM, Gobinda Das wrote: >> > > Can you please post vdsm log which is inside >> /var/log/vdsm/vdsm.log ? >> > > >> > > On Mon, Aug 6, 2018 at 3:51 PM, Jiří Sléžka >> <jiri.slezka(a)slu.cz <mailto:jiri.slezka@slu.cz> >> > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> >> > > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz> >> <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>>> wrote: >> > > >> > > Hi, >> > > >> > > no one can help? >> > > >> > > I still cannot activate this host - error is "Cannot >> activate >> > Host. >> > > Related operation is currently in progress. Please try again >> > later." >> > > >> > > I believe relevant log entrieas are >> > > >> > > 2018-08-06 12:15:50,398+02 INFO >> > > [org.ovirt.engine.core.bll.ActivateVdsCommand] (default >> task-426) >> > > [9387077a-8276-4a3f-a087-584a10a09b08] Failed to Acquire >> Lock >> > to object >> > > >> > >> 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]', >> > > sharedLocks=''}' >> > > 2018-08-06 12:15:50,398+02 WARN >> > > [org.ovirt.engine.core.bll.ActivateVdsCommand] (default >> task-426) >> > > [9387077a-8276-4a3f-a087-584a10a09b08] Validation of action >> > > 'ActivateVds' failed for user ***my_username***. Reasons: >> > > >> > >> VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED >> > > >> > > >> > > when I tried to "reinstall host" I got entry in Events >> "Failed >> > to update >> > > Host blade01" and correlation id. Relevant log entries are >> > > >> > > >> > > cat /var/log/ovirt-engine/engine.log | grep >> > > d2b332da-9050-419c-8188-8fdda5e9d807 >> > > >> > > 2018-08-06 12:08:57,451+02 INFO >> > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] >> (default >> > > task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Running >> command: >> > > InstallVdsCommand internal: false. Entities affected : ID: >> > > 786646cd-c9ef-49a8-8aea-56e858dcf202 Type: VDSAction group >> > > EDIT_HOST_CONFIGURATION with role type ADMIN >> > > 2018-08-06 12:08:57,476+02 INFO >> > > >> [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] >> > > (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] >> > Failed to >> > > Acquire Lock to object >> > > >> > >> 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]', >> > > sharedLocks=''}' >> > > 2018-08-06 12:08:57,476+02 WARN >> > > >> [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] >> > > (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] >> > Validation of >> > > action 'InstallVdsInternal' failed for user >> ***my_username***. >> > Reasons: >> > > ACTION_TYPE_FAILED_OBJECT_LOCKED >> > > 2018-08-06 12:08:57,479+02 ERROR >> > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] >> (default >> > > task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] >> > Installation/upgrade of >> > > Host '786646cd-c9ef-49a8-8aea-56e858dcf202', 'blade01' >> failed: >> > Cannot >> > > ${action} ${type}. Related operation is currently in >> progress. >> > Please >> > > try again later. >> > > 2018-08-06 12:08:57,482+02 INFO >> > > [org.ovirt.engine.core.bll.CommandCompensator] (default >> task-425) >> > > [d2b332da-9050-419c-8188-8fdda5e9d807] Command >> > > [id=b74adfbb-c043-4fa9-a163-0a6ea7fba9dc]: Compensating >> > > DELETED_OR_UPDATED_ENTITY of >> > > org.ovirt.engine.core.common.businessentities.VdsStatic; >> snapshot: >> > > id=786646cd-c9ef-49a8-8aea-56e858dcf202. >> > > 2018-08-06 12:08:57,486+02 ERROR >> > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] >> (default >> > > task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Transaction >> > rolled-back >> > > for command >> > 'org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand'. >> > > 2018-08-06 12:08:57,491+02 ERROR >> > > >> > >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> > > (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] >> > EVENT_ID: >> > > USER_FAILED_UPDATE_VDS(105), Failed to update Host >> blade01 (User: >> > > ***my_username***). >> > > >> > > How can I unlock this entries? >> > > >> > > Thanks in advance, >> > > >> > > Jiri Slezka >> > > >> > > >> > > >> > > On 08/03/2018 02:07 PM, Jiří Sléžka wrote: >> > > > Hi, >> > > > >> > > > I have one interesting issue. I was asked by colleague to >> > repair his >> > > > cluster which is part of our oVirt installation. He >> tried to >> > > upgrade his >> > > > host to gain 4.2 compatibility level but it fails in some >> > cases. Never >> > > > mind, I checked it, repaired, installed upgrades and >> > activate his >> > > hosts >> > > > with one exception. >> > > > >> > > > One of hosts was stucked in finishing two tasks. One was >> > changing >> > > > network configuration, the second was activating of >> this host. I >> > > have to >> > > > manually clear this tasks in database by >> > > > >> > > > SELECT * FROM job ORDER BY start_time DESC; >> > > > SELECT DeleteJob('...job_id...'); >> > > > SELECT DeleteJob('...job_id...'); >> > > > >> > > > Now this tasks are gone but still cannot activate this >> host. >> > > > >> > > > >> > > > Error while executing action: >> > > > >> > > > blade01: >> > > > >> > > > Cannot activate Host. Related operation is >> currently in >> > progress. >> > > > Please try again later. >> > > > >> > > > Host is updatet (latest 4.2 repo), rebooted, looks fine... >> > > > >> > > > In tasks in UI are no tasks found (the same in db)... >> > > > >> > > > What can I do now? >> > > > >> > > > Cheers, Jiri >> > > > >> > > > >> > > > >> > > > >> > > > _______________________________________________ >> > > > Users mailing list -- users(a)ovirt.org >> <mailto:users@ovirt.org> >> > <mailto:users@ovirt.org <mailto:users@ovirt.org>> >> <mailto:users@ovirt.org <mailto:users@ovirt.org> >> > <mailto:users@ovirt.org <mailto:users@ovirt.org>>> >> > > > To unsubscribe send an email to users-leave(a)ovirt.org <mailto:users-leave@ovirt.org> >> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> >> > > <mailto:users-leave@ovirt.org >> <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org >> <mailto:users-leave@ovirt.org>>> >> > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> <https://www.ovirt.org/site/privacy-policy/> >> > > <https://www.ovirt.org/site/privacy-policy/ >> <https://www.ovirt.org/site/privacy-policy/>> >> > > > oVirt Code of Conduct: >> > > https://www.ovirt.org/community/about/community-guidelines/ >> <https://www.ovirt.org/community/about/community-guidelines/> >> > > < https://www.ovirt.org/community/about/community-guidelines/ >> <https://www.ovirt.org/community/about/community-guidelines/>> >> > > > List Archives: >> > > >> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEES... >> < https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEES... > >> > > >> > < https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEES... >> < https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEES... >> >> > > > >> > > >> > > >> > > >> > > _______________________________________________ >> > > Users mailing list -- users(a)ovirt.org <mailto: users(a)ovirt.org&gt; <mailto:users@ovirt.org >> <mailto:users@ovirt.org>> >> > <mailto:users@ovirt.org <mailto:users@ovirt.org> >> <mailto:users@ovirt.org <mailto:users@ovirt.org>>> >> > > To unsubscribe send an email to users-leave(a)ovirt.org <mailto:users-leave@ovirt.org> >> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> >> > > <mailto:users-leave@ovirt.org >> <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org >> <mailto:users-leave@ovirt.org>>> >> > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> <https://www.ovirt.org/site/privacy-policy/> >> > > <https://www.ovirt.org/site/privacy-policy/ >> <https://www.ovirt.org/site/privacy-policy/>> >> > > oVirt Code of Conduct: >> > > https://www.ovirt.org/community/about/community-guidelines/ >> <https://www.ovirt.org/community/about/community-guidelines/> >> > > < https://www.ovirt.org/community/about/community-guidelines/ >> <https://www.ovirt.org/community/about/community-guidelines/>> >> > > List Archives: >> > > >> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37... >> < https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37... > >> > > >> > < https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37... >> < https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37... >> >> > > >> > > >> > > >> > > >> > > -- >> > > Thanks, >> > > Gobinda >> > >> > >> > _______________________________________________ >> > Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org >> <mailto:users@ovirt.org>> >> > To unsubscribe send an email to users-leave(a)ovirt.org <mailto:users-leave@ovirt.org> >> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> >> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> <https://www.ovirt.org/site/privacy-policy/> >> > oVirt Code of Conduct: >> > https://www.ovirt.org/community/about/community-guidelines/ >> <https://www.ovirt.org/community/about/community-guidelines/> >> > List Archives: >> > >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/BP7SG5BQKZI... >> < https://lists.ovirt.org/archives/list/users@ovirt.org/message/BP7SG5BQKZI... > >> > >> >> >> >> >> >> -- >> >> >> Raz Tamir >> Manager, RHV QE > > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/HGS5KF22YQD... > _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/OOTD2GXB7XX...