Hi, I have one interesting issue. I was asked by colleague to repair his cluster which is part of our oVirt installation. He tried to upgrade his host to gain 4.2 compatibility level but it fails in some cases. Never mind, I checked it, repaired, installed upgrades and activate his hosts with one exception. One of hosts was stucked in finishing two tasks. One was changing network configuration, the second was activating of this host. I have to manually clear this tasks in database by SELECT * FROM job ORDER BY start_time DESC; SELECT DeleteJob('...job_id...'); SELECT DeleteJob('...job_id...'); Now this tasks are gone but still cannot activate this host. Error while executing action: blade01: Cannot activate Host. Related operation is currently in progress. Please try again later. Host is updatet (latest 4.2 repo), rebooted, looks fine... In tasks in UI are no tasks found (the same in db)... What can I do now? Cheers, Jiri
Hi, no one can help? I still cannot activate this host - error is "Cannot activate Host. Related operation is currently in progress. Please try again later." I believe relevant log entrieas are 2018-08-06 12:15:50,398+02 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) [9387077a-8276-4a3f-a087-584a10a09b08] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]', sharedLocks=''}' 2018-08-06 12:15:50,398+02 WARN [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) [9387077a-8276-4a3f-a087-584a10a09b08] Validation of action 'ActivateVds' failed for user ***my_username***. Reasons: VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED when I tried to "reinstall host" I got entry in Events "Failed to update Host blade01" and correlation id. Relevant log entries are cat /var/log/ovirt-engine/engine.log | grep d2b332da-9050-419c-8188-8fdda5e9d807 2018-08-06 12:08:57,451+02 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Running command: InstallVdsCommand internal: false. Entities affected : ID: 786646cd-c9ef-49a8-8aea-56e858dcf202 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN 2018-08-06 12:08:57,476+02 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]', sharedLocks=''}' 2018-08-06 12:08:57,476+02 WARN [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Validation of action 'InstallVdsInternal' failed for user ***my_username***. Reasons: ACTION_TYPE_FAILED_OBJECT_LOCKED 2018-08-06 12:08:57,479+02 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Installation/upgrade of Host '786646cd-c9ef-49a8-8aea-56e858dcf202', 'blade01' failed: Cannot ${action} ${type}. Related operation is currently in progress. Please try again later. 2018-08-06 12:08:57,482+02 INFO [org.ovirt.engine.core.bll.CommandCompensator] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Command [id=b74adfbb-c043-4fa9-a163-0a6ea7fba9dc]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.VdsStatic; snapshot: id=786646cd-c9ef-49a8-8aea-56e858dcf202. 2018-08-06 12:08:57,486+02 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Transaction rolled-back for command 'org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand'. 2018-08-06 12:08:57,491+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] EVENT_ID: USER_FAILED_UPDATE_VDS(105), Failed to update Host blade01 (User: ***my_username***). How can I unlock this entries? Thanks in advance, Jiri Slezka On 08/03/2018 02:07 PM, Jiří Sléžka wrote:
Hi,
I have one interesting issue. I was asked by colleague to repair his cluster which is part of our oVirt installation. He tried to upgrade his host to gain 4.2 compatibility level but it fails in some cases. Never mind, I checked it, repaired, installed upgrades and activate his hosts with one exception.
One of hosts was stucked in finishing two tasks. One was changing network configuration, the second was activating of this host. I have to manually clear this tasks in database by
SELECT * FROM job ORDER BY start_time DESC; SELECT DeleteJob('...job_id...'); SELECT DeleteJob('...job_id...');
Now this tasks are gone but still cannot activate this host.
Error while executing action:
blade01:
Cannot activate Host. Related operation is currently in progress. Please try again later.
Host is updatet (latest 4.2 repo), rebooted, looks fine...
In tasks in UI are no tasks found (the same in db)...
What can I do now?
Cheers, Jiri
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCD...
Can you please post vdsm log which is inside /var/log/vdsm/vdsm.log ? On Mon, Aug 6, 2018 at 3:51 PM, Jiří Sléžka <jiri.slezka@slu.cz> wrote:
Hi,
no one can help?
I still cannot activate this host - error is "Cannot activate Host. Related operation is currently in progress. Please try again later."
I believe relevant log entrieas are
2018-08-06 12:15:50,398+02 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) [9387077a-8276-4a3f-a087-584a10a09b08] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]', sharedLocks=''}' 2018-08-06 12:15:50,398+02 WARN [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) [9387077a-8276-4a3f-a087-584a10a09b08] Validation of action 'ActivateVds' failed for user ***my_username***. Reasons: VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED
when I tried to "reinstall host" I got entry in Events "Failed to update Host blade01" and correlation id. Relevant log entries are
cat /var/log/ovirt-engine/engine.log | grep d2b332da-9050-419c-8188-8fdda5e9d807
2018-08-06 12:08:57,451+02 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Running command: InstallVdsCommand internal: false. Entities affected : ID: 786646cd-c9ef-49a8-8aea-56e858dcf202 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN 2018-08-06 12:08:57,476+02 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]', sharedLocks=''}' 2018-08-06 12:08:57,476+02 WARN [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Validation of action 'InstallVdsInternal' failed for user ***my_username***. Reasons: ACTION_TYPE_FAILED_OBJECT_LOCKED 2018-08-06 12:08:57,479+02 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Installation/upgrade of Host '786646cd-c9ef-49a8-8aea-56e858dcf202', 'blade01' failed: Cannot ${action} ${type}. Related operation is currently in progress. Please try again later. 2018-08-06 12:08:57,482+02 INFO [org.ovirt.engine.core.bll.CommandCompensator] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Command [id=b74adfbb-c043-4fa9-a163-0a6ea7fba9dc]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.VdsStatic; snapshot: id=786646cd-c9ef-49a8-8aea-56e858dcf202. 2018-08-06 12:08:57,486+02 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Transaction rolled-back for command 'org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand'. 2018-08-06 12:08:57,491+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] EVENT_ID: USER_FAILED_UPDATE_VDS(105), Failed to update Host blade01 (User: ***my_username***).
How can I unlock this entries?
Thanks in advance,
Jiri Slezka
On 08/03/2018 02:07 PM, Jiří Sléžka wrote:
Hi,
I have one interesting issue. I was asked by colleague to repair his cluster which is part of our oVirt installation. He tried to upgrade his host to gain 4.2 compatibility level but it fails in some cases. Never mind, I checked it, repaired, installed upgrades and activate his hosts with one exception.
One of hosts was stucked in finishing two tasks. One was changing network configuration, the second was activating of this host. I have to manually clear this tasks in database by
SELECT * FROM job ORDER BY start_time DESC; SELECT DeleteJob('...job_id...'); SELECT DeleteJob('...job_id...');
Now this tasks are gone but still cannot activate this host.
Error while executing action:
blade01:
Cannot activate Host. Related operation is currently in progress. Please try again later.
Host is updatet (latest 4.2 repo), rebooted, looks fine...
In tasks in UI are no tasks found (the same in db)...
What can I do now?
Cheers, Jiri
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community- guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/ message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community- guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/ message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/
-- Thanks, Gobinda
Hello, On 08/06/2018 12:52 PM, Gobinda Das wrote:
Can you please post vdsm log which is inside /var/log/vdsm/vdsm.log ?
I suppose you are interested in vdsm log of affected host blade01 It looks like nothing special is there... 2018-08-06 12:08:37,736+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, tas k_id=e2c4f6da-8179-4022-a204-9e79f83223d2 (api:46) 2018-08-06 12:08:37,737+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={} from=internal, task _id=e2c4f6da-8179-4022-a204-9e79f83223d2 (api:52) 2018-08-06 12:08:37,737+0200 INFO (periodic/0) [vdsm.api] START multipath_health() from=internal, task_i d=c5a1406e-1afd-458e-a97d-4942f86b7be0 (api:46) 2018-08-06 12:08:37,737+0200 INFO (periodic/0) [vdsm.api] FINISH multipath_health return={} from=interna l, task_id=c5a1406e-1afd-458e-a97d-4942f86b7be0 (api:52) 2018-08-06 12:08:37,841+0200 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,43898 (api:46) 2018-08-06 12:08:37,842+0200 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'messag e': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,43898 (api:52) 2018-08-06 12:08:37,842+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succee ded in 0.00 seconds (__init__:573) 2018-08-06 12:08:52,761+0200 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, tas k_id=6a0a0ecd-941c-4f21-99b2-ea8bd140df0e (api:46) 2018-08-06 12:08:52,761+0200 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task _id=6a0a0ecd-941c-4f21-99b2-ea8bd140df0e (api:52) 2018-08-06 12:08:52,761+0200 INFO (periodic/3) [vdsm.api] START multipath_health() from=internal, task_i d=2243d5f5-2ca8-434d-953d-33778318d79d (api:46) 2018-08-06 12:08:52,761+0200 INFO (periodic/3) [vdsm.api] FINISH multipath_health return={} from=interna l, task_id=2243d5f5-2ca8-434d-953d-33778318d79d (api:52) 2018-08-06 12:08:52,852+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,43898 (api:46) 2018-08-06 12:08:52,853+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'messag e': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,43898 (api:52) 2018-08-06 12:08:52,853+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succee ded in 0.00 seconds (__init__:573) 2018-08-06 12:09:07,784+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, tas k_id=05bc82c8-bca4-4ee3-81a1-d9f3f0b142e9 (api:46) 2018-08-06 12:09:07,785+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats return={} from=internal, task _id=05bc82c8-bca4-4ee3-81a1-d9f3f0b142e9 (api:52) 2018-08-06 12:09:07,785+0200 INFO (periodic/0) [vdsm.api] START multipath_health() from=internal, task_i d=602cf720-762e-4f05-9ef7-3b9741377136 (api:46) 2018-08-06 12:09:07,785+0200 INFO (periodic/0) [vdsm.api] FINISH multipath_health return={} from=interna l, task_id=602cf720-762e-4f05-9ef7-3b9741377136 (api:52) 2018-08-06 12:09:07,871+0200 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,43898 (api:46) 2018-08-06 12:09:07,872+0200 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'messag e': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,43898 (api:52) 2018-08-06 12:09:07,872+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succee ded in 0.00 seconds (__init__:573) Cheers, Jiri
On Mon, Aug 6, 2018 at 3:51 PM, Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote:
Hi,
no one can help?
I still cannot activate this host - error is "Cannot activate Host. Related operation is currently in progress. Please try again later."
I believe relevant log entrieas are
2018-08-06 12:15:50,398+02 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) [9387077a-8276-4a3f-a087-584a10a09b08] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]', sharedLocks=''}' 2018-08-06 12:15:50,398+02 WARN [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) [9387077a-8276-4a3f-a087-584a10a09b08] Validation of action 'ActivateVds' failed for user ***my_username***. Reasons: VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED
when I tried to "reinstall host" I got entry in Events "Failed to update Host blade01" and correlation id. Relevant log entries are
cat /var/log/ovirt-engine/engine.log | grep d2b332da-9050-419c-8188-8fdda5e9d807
2018-08-06 12:08:57,451+02 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Running command: InstallVdsCommand internal: false. Entities affected : ID: 786646cd-c9ef-49a8-8aea-56e858dcf202 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN 2018-08-06 12:08:57,476+02 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]', sharedLocks=''}' 2018-08-06 12:08:57,476+02 WARN [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Validation of action 'InstallVdsInternal' failed for user ***my_username***. Reasons: ACTION_TYPE_FAILED_OBJECT_LOCKED 2018-08-06 12:08:57,479+02 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Installation/upgrade of Host '786646cd-c9ef-49a8-8aea-56e858dcf202', 'blade01' failed: Cannot ${action} ${type}. Related operation is currently in progress. Please try again later. 2018-08-06 12:08:57,482+02 INFO [org.ovirt.engine.core.bll.CommandCompensator] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Command [id=b74adfbb-c043-4fa9-a163-0a6ea7fba9dc]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.VdsStatic; snapshot: id=786646cd-c9ef-49a8-8aea-56e858dcf202. 2018-08-06 12:08:57,486+02 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Transaction rolled-back for command 'org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand'. 2018-08-06 12:08:57,491+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] EVENT_ID: USER_FAILED_UPDATE_VDS(105), Failed to update Host blade01 (User: ***my_username***).
How can I unlock this entries?
Thanks in advance,
Jiri Slezka
On 08/03/2018 02:07 PM, Jiří Sléžka wrote: > Hi, > > I have one interesting issue. I was asked by colleague to repair his > cluster which is part of our oVirt installation. He tried to upgrade his > host to gain 4.2 compatibility level but it fails in some cases. Never > mind, I checked it, repaired, installed upgrades and activate his hosts > with one exception. > > One of hosts was stucked in finishing two tasks. One was changing > network configuration, the second was activating of this host. I have to > manually clear this tasks in database by > > SELECT * FROM job ORDER BY start_time DESC; > SELECT DeleteJob('...job_id...'); > SELECT DeleteJob('...job_id...'); > > Now this tasks are gone but still cannot activate this host. > > > Error while executing action: > > blade01: > > Cannot activate Host. Related operation is currently in progress. > Please try again later. > > Host is updatet (latest 4.2 repo), rebooted, looks fine... > > In tasks in UI are no tasks found (the same in db)... > > What can I do now? > > Cheers, Jiri > > > > > _______________________________________________ > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCD... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/> >
_______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEI... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/>
-- Thanks, Gobinda
Hello, still no luck with solving this issue. I cannot even remove this host. engine log is now spammed with this messages 2018-08-09 15:03:08,410+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', sharedLocks=''}' 2018-08-09 15:03:08,446+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', sharedLocks=''}' 2018-08-09 15:03:08,451+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', sharedLocks=''}' 2018-08-09 15:03:08,486+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', sharedLocks=''}' btw. oVirt 4.2.5.2-1.el7/CentOS7 Any tips how to resolve this and at least remove this host from ovirt? Cheers, Jiri On 08/06/2018 12:52 PM, Gobinda Das wrote:
Can you please post vdsm log which is inside /var/log/vdsm/vdsm.log ?
On Mon, Aug 6, 2018 at 3:51 PM, Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote:
Hi,
no one can help?
I still cannot activate this host - error is "Cannot activate Host. Related operation is currently in progress. Please try again later."
I believe relevant log entrieas are
2018-08-06 12:15:50,398+02 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) [9387077a-8276-4a3f-a087-584a10a09b08] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]', sharedLocks=''}' 2018-08-06 12:15:50,398+02 WARN [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) [9387077a-8276-4a3f-a087-584a10a09b08] Validation of action 'ActivateVds' failed for user ***my_username***. Reasons: VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED
when I tried to "reinstall host" I got entry in Events "Failed to update Host blade01" and correlation id. Relevant log entries are
cat /var/log/ovirt-engine/engine.log | grep d2b332da-9050-419c-8188-8fdda5e9d807
2018-08-06 12:08:57,451+02 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Running command: InstallVdsCommand internal: false. Entities affected : ID: 786646cd-c9ef-49a8-8aea-56e858dcf202 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN 2018-08-06 12:08:57,476+02 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]', sharedLocks=''}' 2018-08-06 12:08:57,476+02 WARN [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Validation of action 'InstallVdsInternal' failed for user ***my_username***. Reasons: ACTION_TYPE_FAILED_OBJECT_LOCKED 2018-08-06 12:08:57,479+02 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Installation/upgrade of Host '786646cd-c9ef-49a8-8aea-56e858dcf202', 'blade01' failed: Cannot ${action} ${type}. Related operation is currently in progress. Please try again later. 2018-08-06 12:08:57,482+02 INFO [org.ovirt.engine.core.bll.CommandCompensator] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Command [id=b74adfbb-c043-4fa9-a163-0a6ea7fba9dc]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.VdsStatic; snapshot: id=786646cd-c9ef-49a8-8aea-56e858dcf202. 2018-08-06 12:08:57,486+02 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Transaction rolled-back for command 'org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand'. 2018-08-06 12:08:57,491+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] EVENT_ID: USER_FAILED_UPDATE_VDS(105), Failed to update Host blade01 (User: ***my_username***).
How can I unlock this entries?
Thanks in advance,
Jiri Slezka
On 08/03/2018 02:07 PM, Jiří Sléžka wrote: > Hi, > > I have one interesting issue. I was asked by colleague to repair his > cluster which is part of our oVirt installation. He tried to upgrade his > host to gain 4.2 compatibility level but it fails in some cases. Never > mind, I checked it, repaired, installed upgrades and activate his hosts > with one exception. > > One of hosts was stucked in finishing two tasks. One was changing > network configuration, the second was activating of this host. I have to > manually clear this tasks in database by > > SELECT * FROM job ORDER BY start_time DESC; > SELECT DeleteJob('...job_id...'); > SELECT DeleteJob('...job_id...'); > > Now this tasks are gone but still cannot activate this host. > > > Error while executing action: > > blade01: > > Cannot activate Host. Related operation is currently in progress. > Please try again later. > > Host is updatet (latest 4.2 repo), rebooted, looks fine... > > In tasks in UI are no tasks found (the same in db)... > > What can I do now? > > Cheers, Jiri > > > > > _______________________________________________ > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCD... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/> >
_______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEI... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/>
-- Thanks, Gobinda
@Arik Hadas <ahadas@redhat.com> any ideas? On Thu, Aug 9, 2018 at 9:16 AM Jiří Sléžka <jiri.slezka@slu.cz> wrote:
Hello,
still no luck with solving this issue.
I cannot even remove this host.
engine log is now spammed with this messages
2018-08-09 15:03:08,410+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait lock
'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', sharedLocks=''}' 2018-08-09 15:03:08,446+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait lock
'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', sharedLocks=''}' 2018-08-09 15:03:08,451+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait lock
'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', sharedLocks=''}' 2018-08-09 15:03:08,486+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait lock
'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', sharedLocks=''}'
btw. oVirt 4.2.5.2-1.el7/CentOS7
Any tips how to resolve this and at least remove this host from ovirt?
Cheers, Jiri
On 08/06/2018 12:52 PM, Gobinda Das wrote:
Can you please post vdsm log which is inside /var/log/vdsm/vdsm.log ?
On Mon, Aug 6, 2018 at 3:51 PM, Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote:
Hi,
no one can help?
I still cannot activate this host - error is "Cannot activate Host. Related operation is currently in progress. Please try again later."
I believe relevant log entrieas are
2018-08-06 12:15:50,398+02 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) [9387077a-8276-4a3f-a087-584a10a09b08] Failed to Acquire Lock to object
'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]',
sharedLocks=''}' 2018-08-06 12:15:50,398+02 WARN [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) [9387077a-8276-4a3f-a087-584a10a09b08] Validation of action 'ActivateVds' failed for user ***my_username***. Reasons:
VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED
when I tried to "reinstall host" I got entry in Events "Failed to
update
Host blade01" and correlation id. Relevant log entries are
cat /var/log/ovirt-engine/engine.log | grep d2b332da-9050-419c-8188-8fdda5e9d807
2018-08-06 12:08:57,451+02 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Running command: InstallVdsCommand internal: false. Entities affected : ID: 786646cd-c9ef-49a8-8aea-56e858dcf202 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN 2018-08-06 12:08:57,476+02 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Failed to Acquire Lock to object
'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]',
sharedLocks=''}' 2018-08-06 12:08:57,476+02 WARN [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Validation
of
action 'InstallVdsInternal' failed for user ***my_username***.
Reasons:
ACTION_TYPE_FAILED_OBJECT_LOCKED 2018-08-06 12:08:57,479+02 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807]
Installation/upgrade of
Host '786646cd-c9ef-49a8-8aea-56e858dcf202', 'blade01' failed: Cannot ${action} ${type}. Related operation is currently in progress. Please try again later. 2018-08-06 12:08:57,482+02 INFO [org.ovirt.engine.core.bll.CommandCompensator] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Command [id=b74adfbb-c043-4fa9-a163-0a6ea7fba9dc]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.VdsStatic; snapshot: id=786646cd-c9ef-49a8-8aea-56e858dcf202. 2018-08-06 12:08:57,486+02 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Transaction
rolled-back
for command 'org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand'. 2018-08-06 12:08:57,491+02 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] EVENT_ID: USER_FAILED_UPDATE_VDS(105), Failed to update Host blade01 (User: ***my_username***).
How can I unlock this entries?
Thanks in advance,
Jiri Slezka
On 08/03/2018 02:07 PM, Jiří Sléžka wrote: > Hi, > > I have one interesting issue. I was asked by colleague to repair
his
> cluster which is part of our oVirt installation. He tried to upgrade his > host to gain 4.2 compatibility level but it fails in some cases.
Never
> mind, I checked it, repaired, installed upgrades and activate his hosts > with one exception. > > One of hosts was stucked in finishing two tasks. One was changing > network configuration, the second was activating of this host. I have to > manually clear this tasks in database by > > SELECT * FROM job ORDER BY start_time DESC; > SELECT DeleteJob('...job_id...'); > SELECT DeleteJob('...job_id...'); > > Now this tasks are gone but still cannot activate this host. > > > Error while executing action: > > blade01: > > Cannot activate Host. Related operation is currently in
progress.
> Please try again later. > > Host is updatet (latest 4.2 repo), rebooted, looks fine... > > In tasks in UI are no tasks found (the same in db)... > > What can I do now? > > Cheers, Jiri > > > > > _______________________________________________ > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> > List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCD...
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCD...
>
_______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEI...
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEI...
-- Thanks, Gobinda
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/BP7SG5BQKZIVR7...
-- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme@redhat.com IRC: gshereme <https://red.ht/sig>
You can try using vdsm-client to check if there are running tasks on the host using this command: vdsm-client Host getAllTasksInfo If there are running tasks, you can stop them: vdsm-client Task stop taskID=xxx-yyy If there are finished tasks, you can clear them: vdsm-client Task clear taskID=xxx-yyy On Thu, Aug 9, 2018, 16:15 Jiří Sléžka <jiri.slezka@slu.cz> wrote:
Hello,
still no luck with solving this issue.
I cannot even remove this host.
engine log is now spammed with this messages
2018-08-09 15:03:08,410+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait lock
'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', sharedLocks=''}' 2018-08-09 15:03:08,446+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait lock
'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', sharedLocks=''}' 2018-08-09 15:03:08,451+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait lock
'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', sharedLocks=''}' 2018-08-09 15:03:08,486+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait lock
'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', sharedLocks=''}'
btw. oVirt 4.2.5.2-1.el7/CentOS7
Any tips how to resolve this and at least remove this host from ovirt?
Cheers, Jiri
On 08/06/2018 12:52 PM, Gobinda Das wrote:
Can you please post vdsm log which is inside /var/log/vdsm/vdsm.log ?
On Mon, Aug 6, 2018 at 3:51 PM, Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote:
Hi,
no one can help?
I still cannot activate this host - error is "Cannot activate Host. Related operation is currently in progress. Please try again later."
I believe relevant log entrieas are
2018-08-06 12:15:50,398+02 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) [9387077a-8276-4a3f-a087-584a10a09b08] Failed to Acquire Lock to object
'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]',
sharedLocks=''}' 2018-08-06 12:15:50,398+02 WARN [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) [9387077a-8276-4a3f-a087-584a10a09b08] Validation of action 'ActivateVds' failed for user ***my_username***. Reasons:
VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED
when I tried to "reinstall host" I got entry in Events "Failed to
update
Host blade01" and correlation id. Relevant log entries are
cat /var/log/ovirt-engine/engine.log | grep d2b332da-9050-419c-8188-8fdda5e9d807
2018-08-06 12:08:57,451+02 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Running command: InstallVdsCommand internal: false. Entities affected : ID: 786646cd-c9ef-49a8-8aea-56e858dcf202 Type: VDSAction group EDIT_HOST_CONFIGURATION with role type ADMIN 2018-08-06 12:08:57,476+02 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Failed to Acquire Lock to object
'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]',
sharedLocks=''}' 2018-08-06 12:08:57,476+02 WARN [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Validation
of
action 'InstallVdsInternal' failed for user ***my_username***.
Reasons:
ACTION_TYPE_FAILED_OBJECT_LOCKED 2018-08-06 12:08:57,479+02 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807]
Installation/upgrade of
Host '786646cd-c9ef-49a8-8aea-56e858dcf202', 'blade01' failed: Cannot ${action} ${type}. Related operation is currently in progress. Please try again later. 2018-08-06 12:08:57,482+02 INFO [org.ovirt.engine.core.bll.CommandCompensator] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Command [id=b74adfbb-c043-4fa9-a163-0a6ea7fba9dc]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.VdsStatic; snapshot: id=786646cd-c9ef-49a8-8aea-56e858dcf202. 2018-08-06 12:08:57,486+02 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Transaction
rolled-back
for command 'org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand'. 2018-08-06 12:08:57,491+02 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] EVENT_ID: USER_FAILED_UPDATE_VDS(105), Failed to update Host blade01 (User: ***my_username***).
How can I unlock this entries?
Thanks in advance,
Jiri Slezka
On 08/03/2018 02:07 PM, Jiří Sléžka wrote: > Hi, > > I have one interesting issue. I was asked by colleague to repair
his
> cluster which is part of our oVirt installation. He tried to upgrade his > host to gain 4.2 compatibility level but it fails in some cases.
Never
> mind, I checked it, repaired, installed upgrades and activate his hosts > with one exception. > > One of hosts was stucked in finishing two tasks. One was changing > network configuration, the second was activating of this host. I have to > manually clear this tasks in database by > > SELECT * FROM job ORDER BY start_time DESC; > SELECT DeleteJob('...job_id...'); > SELECT DeleteJob('...job_id...'); > > Now this tasks are gone but still cannot activate this host. > > > Error while executing action: > > blade01: > > Cannot activate Host. Related operation is currently in
progress.
> Please try again later. > > Host is updatet (latest 4.2 repo), rebooted, looks fine... > > In tasks in UI are no tasks found (the same in db)... > > What can I do now? > > Cheers, Jiri > > > > > _______________________________________________ > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> > List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCD...
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCD...
>
_______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEI...
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEI...
-- Thanks, Gobinda
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/BP7SG5BQKZIVR7...
Hi, On 08/09/2018 11:25 PM, Raz Tamir wrote:
You can try using vdsm-client to check if there are running tasks on the host using this command:
vdsm-client Host getAllTasksInfo
thanks for support, I tried this on affected host [root@blade01 ~]# vdsm-client Host getAllTasksInfo vdsm-client: Command Host.getAllTasksInfo with args {} failed: (code=654, message=Not SPM: ()) so I moved into SPM host (blade03 at this time) [root@blade03 ~]# vdsm-client Host getAllTasksInfo {} I tried also [root@blade01 ~]# vdsm-client Host getAllTasks {} [root@blade01 ~]# vdsm-client Host getJobs {} [root@blade03 ~]# vdsm-client Host getAllTasks {} [root@blade03 ~]# vdsm-client Host getJobs {} It looks like there are no tasks/jobs running on hosts. It looks like something is stucked in engine db. At the beginning of this issue, there were some race condition where blade01 had two running tasks/jobs which cannot be finished (network configuration and activating of host). I deleted this jobs manually in db via SELECT * FROM job ORDER BY start_time DESC; SELECT DeleteJob('...job_id...'); SELECT DeleteJob('...job_id...'); Maybe it was not wise... but what can I do now to solve this? :-) Cheers, Jiri
If there are running tasks, you can stop them:
vdsm-client Task stop taskID=xxx-yyy
If there are finished tasks, you can clear them:
vdsm-client Task clear taskID=xxx-yyy
On Thu, Aug 9, 2018, 16:15 Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote:
Hello,
still no luck with solving this issue.
I cannot even remove this host.
engine log is now spammed with this messages
2018-08-09 15:03:08,410+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', sharedLocks=''}' 2018-08-09 15:03:08,446+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', sharedLocks=''}' 2018-08-09 15:03:08,451+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', sharedLocks=''}' 2018-08-09 15:03:08,486+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', sharedLocks=''}'
btw. oVirt 4.2.5.2-1.el7/CentOS7
Any tips how to resolve this and at least remove this host from ovirt?
Cheers, Jiri
On 08/06/2018 12:52 PM, Gobinda Das wrote: > Can you please post vdsm log which is inside /var/log/vdsm/vdsm.log ? > > On Mon, Aug 6, 2018 at 3:51 PM, Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz> > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>> wrote: > > Hi, > > no one can help? > > I still cannot activate this host - error is "Cannot activate Host. > Related operation is currently in progress. Please try again later." > > I believe relevant log entrieas are > > 2018-08-06 12:15:50,398+02 INFO > [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) > [9387077a-8276-4a3f-a087-584a10a09b08] Failed to Acquire Lock to object > 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]', > sharedLocks=''}' > 2018-08-06 12:15:50,398+02 WARN > [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) > [9387077a-8276-4a3f-a087-584a10a09b08] Validation of action > 'ActivateVds' failed for user ***my_username***. Reasons: > VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED > > > when I tried to "reinstall host" I got entry in Events "Failed to update > Host blade01" and correlation id. Relevant log entries are > > > cat /var/log/ovirt-engine/engine.log | grep > d2b332da-9050-419c-8188-8fdda5e9d807 > > 2018-08-06 12:08:57,451+02 INFO > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default > task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Running command: > InstallVdsCommand internal: false. Entities affected : ID: > 786646cd-c9ef-49a8-8aea-56e858dcf202 Type: VDSAction group > EDIT_HOST_CONFIGURATION with role type ADMIN > 2018-08-06 12:08:57,476+02 INFO > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] > (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Failed to > Acquire Lock to object > 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]', > sharedLocks=''}' > 2018-08-06 12:08:57,476+02 WARN > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] > (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Validation of > action 'InstallVdsInternal' failed for user ***my_username***. Reasons: > ACTION_TYPE_FAILED_OBJECT_LOCKED > 2018-08-06 12:08:57,479+02 ERROR > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default > task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Installation/upgrade of > Host '786646cd-c9ef-49a8-8aea-56e858dcf202', 'blade01' failed: Cannot > ${action} ${type}. Related operation is currently in progress. Please > try again later. > 2018-08-06 12:08:57,482+02 INFO > [org.ovirt.engine.core.bll.CommandCompensator] (default task-425) > [d2b332da-9050-419c-8188-8fdda5e9d807] Command > [id=b74adfbb-c043-4fa9-a163-0a6ea7fba9dc]: Compensating > DELETED_OR_UPDATED_ENTITY of > org.ovirt.engine.core.common.businessentities.VdsStatic; snapshot: > id=786646cd-c9ef-49a8-8aea-56e858dcf202. > 2018-08-06 12:08:57,486+02 ERROR > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default > task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Transaction rolled-back > for command 'org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand'. > 2018-08-06 12:08:57,491+02 ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] EVENT_ID: > USER_FAILED_UPDATE_VDS(105), Failed to update Host blade01 (User: > ***my_username***). > > How can I unlock this entries? > > Thanks in advance, > > Jiri Slezka > > > > On 08/03/2018 02:07 PM, Jiří Sléžka wrote: > > Hi, > > > > I have one interesting issue. I was asked by colleague to repair his > > cluster which is part of our oVirt installation. He tried to > upgrade his > > host to gain 4.2 compatibility level but it fails in some cases. Never > > mind, I checked it, repaired, installed upgrades and activate his > hosts > > with one exception. > > > > One of hosts was stucked in finishing two tasks. One was changing > > network configuration, the second was activating of this host. I > have to > > manually clear this tasks in database by > > > > SELECT * FROM job ORDER BY start_time DESC; > > SELECT DeleteJob('...job_id...'); > > SELECT DeleteJob('...job_id...'); > > > > Now this tasks are gone but still cannot activate this host. > > > > > > Error while executing action: > > > > blade01: > > > > Cannot activate Host. Related operation is currently in progress. > > Please try again later. > > > > Host is updatet (latest 4.2 repo), rebooted, looks fine... > > > > In tasks in UI are no tasks found (the same in db)... > > > > What can I do now? > > > > Cheers, Jiri > > > > > > > > > > _______________________________________________ > > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>> > > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > <https://www.ovirt.org/site/privacy-policy/> > > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > <https://www.ovirt.org/community/about/community-guidelines/> > > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCD... > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/> > > > > > > _______________________________________________ > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>> > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > <https://www.ovirt.org/site/privacy-policy/> > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > <https://www.ovirt.org/community/about/community-guidelines/> > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEI... > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/> > > > > > -- > Thanks, > Gobinda
_______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/BP7SG5BQKZIVR7...
On Fri, Aug 10, 2018 at 11:30 AM, Jiří Sléžka <jiri.slezka@slu.cz> wrote:
Hi,
On 08/09/2018 11:25 PM, Raz Tamir wrote:
You can try using vdsm-client to check if there are running tasks on the host using this command:
vdsm-client Host getAllTasksInfo
thanks for support, I tried this on affected host
[root@blade01 ~]# vdsm-client Host getAllTasksInfo vdsm-client: Command Host.getAllTasksInfo with args {} failed: (code=654, message=Not SPM: ())
so I moved into SPM host (blade03 at this time)
[root@blade03 ~]# vdsm-client Host getAllTasksInfo {}
I tried also
[root@blade01 ~]# vdsm-client Host getAllTasks {} [root@blade01 ~]# vdsm-client Host getJobs {}
[root@blade03 ~]# vdsm-client Host getAllTasks {} [root@blade03 ~]# vdsm-client Host getJobs {}
It looks like there are no tasks/jobs running on hosts. It looks like something is stucked in engine db.
At the beginning of this issue, there were some race condition where blade01 had two running tasks/jobs which cannot be finished (network configuration and activating of host). I deleted this jobs manually in db via
SELECT * FROM job ORDER BY start_time DESC; SELECT DeleteJob('...job_id...'); SELECT DeleteJob('...job_id...');
Maybe it was not wise... but what can I do now to solve this? :-)
Yes it can be bad to mess with the DB. My best suugestion at the momet is to try and re-add this host.
Cheers, Jiri
If there are running tasks, you can stop them:
vdsm-client Task stop taskID=xxx-yyy
If there are finished tasks, you can clear them:
vdsm-client Task clear taskID=xxx-yyy
On Thu, Aug 9, 2018, 16:15 Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote:
Hello,
still no luck with solving this issue.
I cannot even remove this host.
engine log is now spammed with this messages
2018-08-09 15:03:08,410+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and
wait
lock 'HostEngineLock:{exclusiveLocks='[786646cd-
c9ef-49a8-8aea-56e858dcf202=VDS_INIT]',
sharedLocks=''}' 2018-08-09 15:03:08,446+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and
wait
lock 'HostEngineLock:{exclusiveLocks='[786646cd-
c9ef-49a8-8aea-56e858dcf202=VDS_INIT]',
sharedLocks=''}' 2018-08-09 15:03:08,451+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and
wait
lock 'HostEngineLock:{exclusiveLocks='[786646cd-
c9ef-49a8-8aea-56e858dcf202=VDS_INIT]',
sharedLocks=''}' 2018-08-09 15:03:08,486+02 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engine-Thread-1057) [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and
wait
lock 'HostEngineLock:{exclusiveLocks='[786646cd-
c9ef-49a8-8aea-56e858dcf202=VDS_INIT]',
sharedLocks=''}'
btw. oVirt 4.2.5.2-1.el7/CentOS7
Any tips how to resolve this and at least remove this host from
ovirt?
Cheers, Jiri
On 08/06/2018 12:52 PM, Gobinda Das wrote: > Can you please post vdsm log which is inside
/var/log/vdsm/vdsm.log ?
> > On Mon, Aug 6, 2018 at 3:51 PM, Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz> > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>> wrote: > > Hi, > > no one can help? > > I still cannot activate this host - error is "Cannot activate Host. > Related operation is currently in progress. Please try again later." > > I believe relevant log entrieas are > > 2018-08-06 12:15:50,398+02 INFO > [org.ovirt.engine.core.bll.ActivateVdsCommand] (default
task-426)
> [9387077a-8276-4a3f-a087-584a10a09b08] Failed to Acquire Lock to object > 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-
56e858dcf202=VDS]',
> sharedLocks=''}' > 2018-08-06 12:15:50,398+02 WARN > [org.ovirt.engine.core.bll.ActivateVdsCommand] (default
task-426)
> [9387077a-8276-4a3f-a087-584a10a09b08] Validation of action > 'ActivateVds' failed for user ***my_username***. Reasons: > VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_
OBJECT_LOCKED
> > > when I tried to "reinstall host" I got entry in Events "Failed to update > Host blade01" and correlation id. Relevant log entries are > > > cat /var/log/ovirt-engine/engine.log | grep > d2b332da-9050-419c-8188-8fdda5e9d807 > > 2018-08-06 12:08:57,451+02 INFO > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand]
(default
> task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Running
command:
> InstallVdsCommand internal: false. Entities affected : ID: > 786646cd-c9ef-49a8-8aea-56e858dcf202 Type: VDSAction group > EDIT_HOST_CONFIGURATION with role type ADMIN > 2018-08-06 12:08:57,476+02 INFO > [org.ovirt.engine.core.bll.hostdeploy.
InstallVdsInternalCommand]
> (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Failed to > Acquire Lock to object > 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-
56e858dcf202=VDS]',
> sharedLocks=''}' > 2018-08-06 12:08:57,476+02 WARN > [org.ovirt.engine.core.bll.hostdeploy.
InstallVdsInternalCommand]
> (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Validation of > action 'InstallVdsInternal' failed for user ***my_username***. Reasons: > ACTION_TYPE_FAILED_OBJECT_LOCKED > 2018-08-06 12:08:57,479+02 ERROR > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand]
(default
> task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Installation/upgrade of > Host '786646cd-c9ef-49a8-8aea-56e858dcf202', 'blade01' failed: Cannot > ${action} ${type}. Related operation is currently in progress. Please > try again later. > 2018-08-06 12:08:57,482+02 INFO > [org.ovirt.engine.core.bll.CommandCompensator] (default
task-425)
> [d2b332da-9050-419c-8188-8fdda5e9d807] Command > [id=b74adfbb-c043-4fa9-a163-0a6ea7fba9dc]: Compensating > DELETED_OR_UPDATED_ENTITY of > org.ovirt.engine.core.common.businessentities.VdsStatic;
snapshot:
> id=786646cd-c9ef-49a8-8aea-56e858dcf202. > 2018-08-06 12:08:57,486+02 ERROR > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand]
(default
> task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Transaction rolled-back > for command 'org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand'. > 2018-08-06 12:08:57,491+02 ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.
AuditLogDirector]
> (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] EVENT_ID: > USER_FAILED_UPDATE_VDS(105), Failed to update Host blade01
(User:
> ***my_username***). > > How can I unlock this entries? > > Thanks in advance, > > Jiri Slezka > > > > On 08/03/2018 02:07 PM, Jiří Sléžka wrote: > > Hi, > > > > I have one interesting issue. I was asked by colleague to repair his > > cluster which is part of our oVirt installation. He tried to > upgrade his > > host to gain 4.2 compatibility level but it fails in some cases. Never > > mind, I checked it, repaired, installed upgrades and activate his > hosts > > with one exception. > > > > One of hosts was stucked in finishing two tasks. One was changing > > network configuration, the second was activating of this
host. I
> have to > > manually clear this tasks in database by > > > > SELECT * FROM job ORDER BY start_time DESC; > > SELECT DeleteJob('...job_id...'); > > SELECT DeleteJob('...job_id...'); > > > > Now this tasks are gone but still cannot activate this host. > > > > > > Error while executing action: > > > > blade01: > > > > Cannot activate Host. Related operation is currently in progress. > > Please try again later. > > > > Host is updatet (latest 4.2 repo), rebooted, looks fine... > > > > In tasks in UI are no tasks found (the same in db)... > > > > What can I do now? > > > > Cheers, Jiri > > > > > > > > > > _______________________________________________ > > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>> > > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> > > Privacy Statement: https://www.ovirt.org/site/
privacy-policy/
> <https://www.ovirt.org/site/privacy-policy/> > > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > <https://www.ovirt.org/community/about/community-guidelines/> > > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/
C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/
> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/
C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/>
> > > > > > _______________________________________________ > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>> > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > <https://www.ovirt.org/site/privacy-policy/> > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > <https://www.ovirt.org/community/about/community-guidelines/> > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/
6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/
> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/
6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/>
> > > > > -- > Thanks, > Gobinda
_______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/
BP7SG5BQKZIVR75A6NSCUMGQQBXYL3FJ/
-- Raz Tamir Manager, RHV QE
On 08/10/2018 10:52 AM, Raz Tamir wrote:
On Fri, Aug 10, 2018 at 11:30 AM, Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote:
Hi,
On 08/09/2018 11:25 PM, Raz Tamir wrote: > You can try using vdsm-client to check if there are running tasks on the > host using this command: > > vdsm-client Host getAllTasksInfo
thanks for support, I tried this on affected host
[root@blade01 ~]# vdsm-client Host getAllTasksInfo vdsm-client: Command Host.getAllTasksInfo with args {} failed: (code=654, message=Not SPM: ())
so I moved into SPM host (blade03 at this time)
[root@blade03 ~]# vdsm-client Host getAllTasksInfo {}
I tried also
[root@blade01 ~]# vdsm-client Host getAllTasks {} [root@blade01 ~]# vdsm-client Host getJobs {}
[root@blade03 ~]# vdsm-client Host getAllTasks {} [root@blade03 ~]# vdsm-client Host getJobs {}
It looks like there are no tasks/jobs running on hosts. It looks like something is stucked in engine db.
At the beginning of this issue, there were some race condition where blade01 had two running tasks/jobs which cannot be finished (network configuration and activating of host). I deleted this jobs manually in db via
SELECT * FROM job ORDER BY start_time DESC; SELECT DeleteJob('...job_id...'); SELECT DeleteJob('...job_id...');
Maybe it was not wise... but what can I do now to solve this? :-)
Yes it can be bad to mess with the DB. My best suugestion at the momet is to try and re-add this host.
I would like to but I cannot even remove this host (Cannot remove Host. Related operation is currently in progress. Please try again later.) :-( But maybe I can try to add it as brand new one? But I am curious what checks do manager and why we cannot see the tasks/jobs/locks/anything in db... Cheers, Jiri
Cheers, Jiri
> > If there are running tasks, you can stop them: > > vdsm-client Task stop taskID=xxx-yyy > > If there are finished tasks, you can clear them: > > vdsm-client Task clear taskID=xxx-yyy > > > On Thu, Aug 9, 2018, 16:15 Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz> > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>> wrote: > > Hello, > > still no luck with solving this issue. > > I cannot even remove this host. > > engine log is now spammed with this messages > > 2018-08-09 15:03:08,410+02 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engine-Thread-1057) > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait > lock > 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', > sharedLocks=''}' > 2018-08-09 15:03:08,446+02 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engine-Thread-1057) > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait > lock > 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', > sharedLocks=''}' > 2018-08-09 15:03:08,451+02 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engine-Thread-1057) > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait > lock > 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', > sharedLocks=''}' > 2018-08-09 15:03:08,486+02 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engine-Thread-1057) > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait > lock > 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', > sharedLocks=''}' > > btw. oVirt 4.2.5.2-1.el7/CentOS7 > > Any tips how to resolve this and at least remove this host from ovirt? > > Cheers, Jiri > > > On 08/06/2018 12:52 PM, Gobinda Das wrote: > > Can you please post vdsm log which is inside /var/log/vdsm/vdsm.log ? > > > > On Mon, Aug 6, 2018 at 3:51 PM, Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz> > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> > > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz> <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>>> wrote: > > > > Hi, > > > > no one can help? > > > > I still cannot activate this host - error is "Cannot activate > Host. > > Related operation is currently in progress. Please try again > later." > > > > I believe relevant log entrieas are > > > > 2018-08-06 12:15:50,398+02 INFO > > [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) > > [9387077a-8276-4a3f-a087-584a10a09b08] Failed to Acquire Lock > to object > > > 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]', > > sharedLocks=''}' > > 2018-08-06 12:15:50,398+02 WARN > > [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) > > [9387077a-8276-4a3f-a087-584a10a09b08] Validation of action > > 'ActivateVds' failed for user ***my_username***. Reasons: > > > VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED > > > > > > when I tried to "reinstall host" I got entry in Events "Failed > to update > > Host blade01" and correlation id. Relevant log entries are > > > > > > cat /var/log/ovirt-engine/engine.log | grep > > d2b332da-9050-419c-8188-8fdda5e9d807 > > > > 2018-08-06 12:08:57,451+02 INFO > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default > > task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Running command: > > InstallVdsCommand internal: false. Entities affected : ID: > > 786646cd-c9ef-49a8-8aea-56e858dcf202 Type: VDSAction group > > EDIT_HOST_CONFIGURATION with role type ADMIN > > 2018-08-06 12:08:57,476+02 INFO > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] > > (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] > Failed to > > Acquire Lock to object > > > 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]', > > sharedLocks=''}' > > 2018-08-06 12:08:57,476+02 WARN > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] > > (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] > Validation of > > action 'InstallVdsInternal' failed for user ***my_username***. > Reasons: > > ACTION_TYPE_FAILED_OBJECT_LOCKED > > 2018-08-06 12:08:57,479+02 ERROR > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default > > task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] > Installation/upgrade of > > Host '786646cd-c9ef-49a8-8aea-56e858dcf202', 'blade01' failed: > Cannot > > ${action} ${type}. Related operation is currently in progress. > Please > > try again later. > > 2018-08-06 12:08:57,482+02 INFO > > [org.ovirt.engine.core.bll.CommandCompensator] (default task-425) > > [d2b332da-9050-419c-8188-8fdda5e9d807] Command > > [id=b74adfbb-c043-4fa9-a163-0a6ea7fba9dc]: Compensating > > DELETED_OR_UPDATED_ENTITY of > > org.ovirt.engine.core.common.businessentities.VdsStatic; snapshot: > > id=786646cd-c9ef-49a8-8aea-56e858dcf202. > > 2018-08-06 12:08:57,486+02 ERROR > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default > > task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Transaction > rolled-back > > for command > 'org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand'. > > 2018-08-06 12:08:57,491+02 ERROR > > > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > > (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] > EVENT_ID: > > USER_FAILED_UPDATE_VDS(105), Failed to update Host blade01 (User: > > ***my_username***). > > > > How can I unlock this entries? > > > > Thanks in advance, > > > > Jiri Slezka > > > > > > > > On 08/03/2018 02:07 PM, Jiří Sléžka wrote: > > > Hi, > > > > > > I have one interesting issue. I was asked by colleague to > repair his > > > cluster which is part of our oVirt installation. He tried to > > upgrade his > > > host to gain 4.2 compatibility level but it fails in some > cases. Never > > > mind, I checked it, repaired, installed upgrades and > activate his > > hosts > > > with one exception. > > > > > > One of hosts was stucked in finishing two tasks. One was > changing > > > network configuration, the second was activating of this host. I > > have to > > > manually clear this tasks in database by > > > > > > SELECT * FROM job ORDER BY start_time DESC; > > > SELECT DeleteJob('...job_id...'); > > > SELECT DeleteJob('...job_id...'); > > > > > > Now this tasks are gone but still cannot activate this host. > > > > > > > > > Error while executing action: > > > > > > blade01: > > > > > > Cannot activate Host. Related operation is currently in > progress. > > > Please try again later. > > > > > > Host is updatet (latest 4.2 repo), rebooted, looks fine... > > > > > > In tasks in UI are no tasks found (the same in db)... > > > > > > What can I do now? > > > > > > Cheers, Jiri > > > > > > > > > > > > > > > _______________________________________________ > > > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> > <mailto:users@ovirt.org <mailto:users@ovirt.org>> <mailto:users@ovirt.org <mailto:users@ovirt.org> > <mailto:users@ovirt.org <mailto:users@ovirt.org>>> > > > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> > > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>>> > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> > > <https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/>> > > > oVirt Code of Conduct: > > https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> > > <https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/>> > > > List Archives: > > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCD... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/> > > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCD... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/>> > > > > > > > > > > > _______________________________________________ > > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>> > <mailto:users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>>> > > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> > > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>>> > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> > > <https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/>> > > oVirt Code of Conduct: > > https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> > > <https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/>> > > List Archives: > > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEI... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/> > > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEI... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/>> > > > > > > > > > > -- > > Thanks, > > Gobinda > > > _______________________________________________ > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>> > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/BP7SG5BQKZIVR7... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/BP7SG5BQKZIVR75A6NSCUMGQQBXYL3FJ/> >
--
Raz Tamir Manager, RHV QE
Well, issue was solved by installing latest async update 4.2.5... Probably helped this part of upgrade process... ... [ INFO ] Cleaning async tasks and compensations [ INFO ] Unlocking existing entities ... Cheers, Jiri On 08/10/2018 11:03 AM, Jiří Sléžka wrote:
On 08/10/2018 10:52 AM, Raz Tamir wrote:
On Fri, Aug 10, 2018 at 11:30 AM, Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote:
Hi,
On 08/09/2018 11:25 PM, Raz Tamir wrote: > You can try using vdsm-client to check if there are running tasks on the > host using this command: > > vdsm-client Host getAllTasksInfo
thanks for support, I tried this on affected host
[root@blade01 ~]# vdsm-client Host getAllTasksInfo vdsm-client: Command Host.getAllTasksInfo with args {} failed: (code=654, message=Not SPM: ())
so I moved into SPM host (blade03 at this time)
[root@blade03 ~]# vdsm-client Host getAllTasksInfo {}
I tried also
[root@blade01 ~]# vdsm-client Host getAllTasks {} [root@blade01 ~]# vdsm-client Host getJobs {}
[root@blade03 ~]# vdsm-client Host getAllTasks {} [root@blade03 ~]# vdsm-client Host getJobs {}
It looks like there are no tasks/jobs running on hosts. It looks like something is stucked in engine db.
At the beginning of this issue, there were some race condition where blade01 had two running tasks/jobs which cannot be finished (network configuration and activating of host). I deleted this jobs manually in db via
SELECT * FROM job ORDER BY start_time DESC; SELECT DeleteJob('...job_id...'); SELECT DeleteJob('...job_id...');
Maybe it was not wise... but what can I do now to solve this? :-)
Yes it can be bad to mess with the DB. My best suugestion at the momet is to try and re-add this host.
I would like to but I cannot even remove this host (Cannot remove Host. Related operation is currently in progress. Please try again later.) :-( But maybe I can try to add it as brand new one?
But I am curious what checks do manager and why we cannot see the tasks/jobs/locks/anything in db...
Cheers, Jiri
Cheers, Jiri
> > If there are running tasks, you can stop them: > > vdsm-client Task stop taskID=xxx-yyy > > If there are finished tasks, you can clear them: > > vdsm-client Task clear taskID=xxx-yyy > > > On Thu, Aug 9, 2018, 16:15 Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz> > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>> wrote: > > Hello, > > still no luck with solving this issue. > > I cannot even remove this host. > > engine log is now spammed with this messages > > 2018-08-09 15:03:08,410+02 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engine-Thread-1057) > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait > lock > 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', > sharedLocks=''}' > 2018-08-09 15:03:08,446+02 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engine-Thread-1057) > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait > lock > 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', > sharedLocks=''}' > 2018-08-09 15:03:08,451+02 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engine-Thread-1057) > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait > lock > 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', > sharedLocks=''}' > 2018-08-09 15:03:08,486+02 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engine-Thread-1057) > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait > lock > 'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]', > sharedLocks=''}' > > btw. oVirt 4.2.5.2-1.el7/CentOS7 > > Any tips how to resolve this and at least remove this host from ovirt? > > Cheers, Jiri > > > On 08/06/2018 12:52 PM, Gobinda Das wrote: > > Can you please post vdsm log which is inside /var/log/vdsm/vdsm.log ? > > > > On Mon, Aug 6, 2018 at 3:51 PM, Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz> > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> > > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz> <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>>> wrote: > > > > Hi, > > > > no one can help? > > > > I still cannot activate this host - error is "Cannot activate > Host. > > Related operation is currently in progress. Please try again > later." > > > > I believe relevant log entrieas are > > > > 2018-08-06 12:15:50,398+02 INFO > > [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) > > [9387077a-8276-4a3f-a087-584a10a09b08] Failed to Acquire Lock > to object > > > 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]', > > sharedLocks=''}' > > 2018-08-06 12:15:50,398+02 WARN > > [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) > > [9387077a-8276-4a3f-a087-584a10a09b08] Validation of action > > 'ActivateVds' failed for user ***my_username***. Reasons: > > > VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED > > > > > > when I tried to "reinstall host" I got entry in Events "Failed > to update > > Host blade01" and correlation id. Relevant log entries are > > > > > > cat /var/log/ovirt-engine/engine.log | grep > > d2b332da-9050-419c-8188-8fdda5e9d807 > > > > 2018-08-06 12:08:57,451+02 INFO > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default > > task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Running command: > > InstallVdsCommand internal: false. Entities affected : ID: > > 786646cd-c9ef-49a8-8aea-56e858dcf202 Type: VDSAction group > > EDIT_HOST_CONFIGURATION with role type ADMIN > > 2018-08-06 12:08:57,476+02 INFO > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] > > (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] > Failed to > > Acquire Lock to object > > > 'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]', > > sharedLocks=''}' > > 2018-08-06 12:08:57,476+02 WARN > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] > > (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] > Validation of > > action 'InstallVdsInternal' failed for user ***my_username***. > Reasons: > > ACTION_TYPE_FAILED_OBJECT_LOCKED > > 2018-08-06 12:08:57,479+02 ERROR > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default > > task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] > Installation/upgrade of > > Host '786646cd-c9ef-49a8-8aea-56e858dcf202', 'blade01' failed: > Cannot > > ${action} ${type}. Related operation is currently in progress. > Please > > try again later. > > 2018-08-06 12:08:57,482+02 INFO > > [org.ovirt.engine.core.bll.CommandCompensator] (default task-425) > > [d2b332da-9050-419c-8188-8fdda5e9d807] Command > > [id=b74adfbb-c043-4fa9-a163-0a6ea7fba9dc]: Compensating > > DELETED_OR_UPDATED_ENTITY of > > org.ovirt.engine.core.common.businessentities.VdsStatic; snapshot: > > id=786646cd-c9ef-49a8-8aea-56e858dcf202. > > 2018-08-06 12:08:57,486+02 ERROR > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default > > task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Transaction > rolled-back > > for command > 'org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand'. > > 2018-08-06 12:08:57,491+02 ERROR > > > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > > (default task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] > EVENT_ID: > > USER_FAILED_UPDATE_VDS(105), Failed to update Host blade01 (User: > > ***my_username***). > > > > How can I unlock this entries? > > > > Thanks in advance, > > > > Jiri Slezka > > > > > > > > On 08/03/2018 02:07 PM, Jiří Sléžka wrote: > > > Hi, > > > > > > I have one interesting issue. I was asked by colleague to > repair his > > > cluster which is part of our oVirt installation. He tried to > > upgrade his > > > host to gain 4.2 compatibility level but it fails in some > cases. Never > > > mind, I checked it, repaired, installed upgrades and > activate his > > hosts > > > with one exception. > > > > > > One of hosts was stucked in finishing two tasks. One was > changing > > > network configuration, the second was activating of this host. I > > have to > > > manually clear this tasks in database by > > > > > > SELECT * FROM job ORDER BY start_time DESC; > > > SELECT DeleteJob('...job_id...'); > > > SELECT DeleteJob('...job_id...'); > > > > > > Now this tasks are gone but still cannot activate this host. > > > > > > > > > Error while executing action: > > > > > > blade01: > > > > > > Cannot activate Host. Related operation is currently in > progress. > > > Please try again later. > > > > > > Host is updatet (latest 4.2 repo), rebooted, looks fine... > > > > > > In tasks in UI are no tasks found (the same in db)... > > > > > > What can I do now? > > > > > > Cheers, Jiri > > > > > > > > > > > > > > > _______________________________________________ > > > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> > <mailto:users@ovirt.org <mailto:users@ovirt.org>> <mailto:users@ovirt.org <mailto:users@ovirt.org> > <mailto:users@ovirt.org <mailto:users@ovirt.org>>> > > > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> > > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>>> > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> > > <https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/>> > > > oVirt Code of Conduct: > > https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> > > <https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/>> > > > List Archives: > > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCD... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/> > > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCD... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/>> > > > > > > > > > > > _______________________________________________ > > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>> > <mailto:users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>>> > > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> > > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>>> > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> > > <https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/>> > > oVirt Code of Conduct: > > https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> > > <https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/>> > > List Archives: > > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEI... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/> > > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEI... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/>> > > > > > > > > > > -- > > Thanks, > > Gobinda > > > _______________________________________________ > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>> > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/BP7SG5BQKZIVR7... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/BP7SG5BQKZIVR75A6NSCUMGQQBXYL3FJ/> >
--
Raz Tamir Manager, RHV QE
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/HGS5KF22YQD3XS...
On Fri, Aug 10, 2018 at 4:23 PM Jiří Sléžka <jiri.slezka@slu.cz> wrote:
Well, issue was solved by installing latest async update 4.2.5... Probably helped this part of upgrade process...
... [ INFO ] Cleaning async tasks and compensations [ INFO ] Unlocking existing entities
Right, that was an in-memory lock that blocked you so you probably had to remove the relevant command that acquired the lock from the command_entities table.
...
Cheers,
Jiri
On 08/10/2018 10:52 AM, Raz Tamir wrote:
On Fri, Aug 10, 2018 at 11:30 AM, Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote:
Hi,
On 08/09/2018 11:25 PM, Raz Tamir wrote: > You can try using vdsm-client to check if there are running tasks
on the
> host using this command: > > vdsm-client Host getAllTasksInfo
thanks for support, I tried this on affected host
[root@blade01 ~]# vdsm-client Host getAllTasksInfo vdsm-client: Command Host.getAllTasksInfo with args {} failed: (code=654, message=Not SPM: ())
so I moved into SPM host (blade03 at this time)
[root@blade03 ~]# vdsm-client Host getAllTasksInfo {}
I tried also
[root@blade01 ~]# vdsm-client Host getAllTasks {} [root@blade01 ~]# vdsm-client Host getJobs {}
[root@blade03 ~]# vdsm-client Host getAllTasks {} [root@blade03 ~]# vdsm-client Host getJobs {}
It looks like there are no tasks/jobs running on hosts. It looks
On 08/10/2018 11:03 AM, Jiří Sléžka wrote: like
something is stucked in engine db.
At the beginning of this issue, there were some race condition where blade01 had two running tasks/jobs which cannot be finished (network configuration and activating of host). I deleted this jobs manually
in
db via
SELECT * FROM job ORDER BY start_time DESC; SELECT DeleteJob('...job_id...'); SELECT DeleteJob('...job_id...');
Maybe it was not wise... but what can I do now to solve this? :-)
Yes it can be bad to mess with the DB. My best suugestion at the momet is to try and re-add this host.
I would like to but I cannot even remove this host (Cannot remove Host. Related operation is currently in progress. Please try again later.) :-( But maybe I can try to add it as brand new one?
But I am curious what checks do manager and why we cannot see the tasks/jobs/locks/anything in db...
Cheers, Jiri
Cheers, Jiri
> > If there are running tasks, you can stop them: > > vdsm-client Task stop taskID=xxx-yyy > > If there are finished tasks, you can clear them: > > vdsm-client Task clear taskID=xxx-yyy > > > On Thu, Aug 9, 2018, 16:15 Jiří Sléžka <jiri.slezka@slu.cz
<mailto:jiri.slezka@slu.cz>
> <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>> wrote: > > Hello, > > still no luck with solving this issue. > > I cannot even remove this host. > > engine log is now spammed with this messages > > 2018-08-09 15:03:08,410+02 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engine-Thread-1057) > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait > lock >
'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]',
> sharedLocks=''}' > 2018-08-09 15:03:08,446+02 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engine-Thread-1057) > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait > lock >
'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]',
> sharedLocks=''}' > 2018-08-09 15:03:08,451+02 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engine-Thread-1057) > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait > lock >
'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]',
> sharedLocks=''}' > 2018-08-09 15:03:08,486+02 INFO > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engine-Thread-1057) > [cb8aa091-70ce-419a-b45b-3ffad6f2529b] Failed to acquire lock and wait > lock >
'HostEngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS_INIT]',
> sharedLocks=''}' > > btw. oVirt 4.2.5.2-1.el7/CentOS7 > > Any tips how to resolve this and at least remove this host from ovirt? > > Cheers, Jiri > > > On 08/06/2018 12:52 PM, Gobinda Das wrote: > > Can you please post vdsm log which is inside /var/log/vdsm/vdsm.log ? > > > > On Mon, Aug 6, 2018 at 3:51 PM, Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz> > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> > > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz> <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>>> wrote: > > > > Hi, > > > > no one can help? > > > > I still cannot activate this host - error is "Cannot activate > Host. > > Related operation is currently in progress. Please try
again
> later." > > > > I believe relevant log entrieas are > > > > 2018-08-06 12:15:50,398+02 INFO > > [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) > > [9387077a-8276-4a3f-a087-584a10a09b08] Failed to Acquire Lock > to object > > >
'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]',
> > sharedLocks=''}' > > 2018-08-06 12:15:50,398+02 WARN > > [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-426) > > [9387077a-8276-4a3f-a087-584a10a09b08] Validation of
action
> > 'ActivateVds' failed for user ***my_username***.
Reasons:
> > >
VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED
> > > > > > when I tried to "reinstall host" I got entry in Events "Failed > to update > > Host blade01" and correlation id. Relevant log entries
are
> > > > > > cat /var/log/ovirt-engine/engine.log | grep > > d2b332da-9050-419c-8188-8fdda5e9d807 > > > > 2018-08-06 12:08:57,451+02 INFO > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default > > task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] Running command: > > InstallVdsCommand internal: false. Entities affected :
ID:
> > 786646cd-c9ef-49a8-8aea-56e858dcf202 Type: VDSAction
group
> > EDIT_HOST_CONFIGURATION with role type ADMIN > > 2018-08-06 12:08:57,476+02 INFO > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] > > (default task-425)
[d2b332da-9050-419c-8188-8fdda5e9d807]
> Failed to > > Acquire Lock to object > > >
'EngineLock:{exclusiveLocks='[786646cd-c9ef-49a8-8aea-56e858dcf202=VDS]',
> > sharedLocks=''}' > > 2018-08-06 12:08:57,476+02 WARN > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] > > (default task-425)
[d2b332da-9050-419c-8188-8fdda5e9d807]
> Validation of > > action 'InstallVdsInternal' failed for user ***my_username***. > Reasons: > > ACTION_TYPE_FAILED_OBJECT_LOCKED > > 2018-08-06 12:08:57,479+02 ERROR > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default > > task-425) [d2b332da-9050-419c-8188-8fdda5e9d807] > Installation/upgrade of > > Host '786646cd-c9ef-49a8-8aea-56e858dcf202', 'blade01' failed: > Cannot > > ${action} ${type}. Related operation is currently in progress. > Please > > try again later. > > 2018-08-06 12:08:57,482+02 INFO > > [org.ovirt.engine.core.bll.CommandCompensator] (default task-425) > > [d2b332da-9050-419c-8188-8fdda5e9d807] Command > > [id=b74adfbb-c043-4fa9-a163-0a6ea7fba9dc]: Compensating > > DELETED_OR_UPDATED_ENTITY of > > org.ovirt.engine.core.common.businessentities.VdsStatic; snapshot: > > id=786646cd-c9ef-49a8-8aea-56e858dcf202. > > 2018-08-06 12:08:57,486+02 ERROR > > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand] (default > > task-425) [d2b332da-9050-419c-8188-8fdda5e9d807]
Transaction
> rolled-back > > for command > 'org.ovirt.engine.core.bll.hostdeploy.InstallVdsCommand'. > > 2018-08-06 12:08:57,491+02 ERROR > > >
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> > (default task-425)
[d2b332da-9050-419c-8188-8fdda5e9d807]
> EVENT_ID: > > USER_FAILED_UPDATE_VDS(105), Failed to update Host blade01 (User: > > ***my_username***). > > > > How can I unlock this entries? > > > > Thanks in advance, > > > > Jiri Slezka > > > > > > > > On 08/03/2018 02:07 PM, Jiří Sléžka wrote: > > > Hi, > > > > > > I have one interesting issue. I was asked by
colleague to >> > repair his >> > > > cluster which is part of our oVirt installation. He >> tried to >> > > upgrade his >> > > > host to gain 4.2 compatibility level but it fails in some >> > cases. Never >> > > > mind, I checked it, repaired, installed upgrades and >> > activate his >> > > hosts >> > > > with one exception. >> > > > >> > > > One of hosts was stucked in finishing two tasks. One was >> > changing >> > > > network configuration, the second was activating of >> this host. I >> > > have to >> > > > manually clear this tasks in database by >> > > > >> > > > SELECT * FROM job ORDER BY start_time DESC; >> > > > SELECT DeleteJob('...job_id...'); >> > > > SELECT DeleteJob('...job_id...'); >> > > > >> > > > Now this tasks are gone but still cannot activate this >> host. >> > > > >> > > > >> > > > Error while executing action: >> > > > >> > > > blade01: >> > > > >> > > > Cannot activate Host. Related operation is >> currently in >> > progress. >> > > > Please try again later. >> > > > >> > > > Host is updatet (latest 4.2 repo), rebooted, looks fine... >> > > > >> > > > In tasks in UI are no tasks found (the same in db)... >> > > > >> > > > What can I do now? >> > > > >> > > > Cheers, Jiri >> > > > >> > > > >> > > > >> > > > >> > > > _______________________________________________ >> > > > Users mailing list -- users@ovirt.org >> <mailto:users@ovirt.org> >> > <mailto:users@ovirt.org <mailto:users@ovirt.org>> >> <mailto:users@ovirt.org <mailto:users@ovirt.org> >> > <mailto:users@ovirt.org <mailto:users@ovirt.org>>> >> > > > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> >> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> >> > > <mailto:users-leave@ovirt.org >> <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org >> <mailto:users-leave@ovirt.org>>> >> > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> <https://www.ovirt.org/site/privacy-policy/> >> > > <https://www.ovirt.org/site/privacy-policy/ >> <https://www.ovirt.org/site/privacy-policy/>> >> > > > oVirt Code of Conduct: >> > > https://www.ovirt.org/community/about/community-guidelines/ >> <https://www.ovirt.org/community/about/community-guidelines/> >> > > < https://www.ovirt.org/community/about/community-guidelines/ >> <https://www.ovirt.org/community/about/community-guidelines/>> >> > > > List Archives: >> > > >> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/ >> < https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/ > >> > > >> > < https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/ >> < https://lists.ovirt.org/archives/list/users@ovirt.org/message/C2TWKXGOEESXCDJUZ7IBKYDE2APFN4ZB/ >> >> > > > >> > > >> > > >> > > >> > > _______________________________________________ >> > > Users mailing list -- users@ovirt.org <mailto: users@ovirt.org> <mailto:users@ovirt.org >> <mailto:users@ovirt.org>> >> > <mailto:users@ovirt.org <mailto:users@ovirt.org> >> <mailto:users@ovirt.org <mailto:users@ovirt.org>>> >> > > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> >> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> >> > > <mailto:users-leave@ovirt.org >> <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org >> <mailto:users-leave@ovirt.org>>> >> > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> <https://www.ovirt.org/site/privacy-policy/> >> > > <https://www.ovirt.org/site/privacy-policy/ >> <https://www.ovirt.org/site/privacy-policy/>> >> > > oVirt Code of Conduct: >> > > https://www.ovirt.org/community/about/community-guidelines/ >> <https://www.ovirt.org/community/about/community-guidelines/> >> > > < https://www.ovirt.org/community/about/community-guidelines/ >> <https://www.ovirt.org/community/about/community-guidelines/>> >> > > List Archives: >> > > >> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/ >> < https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/ > >> > > >> > < https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/ >> < https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H4QYVTTV37IEIA2MTABJ67P43LQ5AGI/ >> >> > > >> > > >> > > >> > > >> > > -- >> > > Thanks, >> > > Gobinda >> > >> > >> > _______________________________________________ >> > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org >> <mailto:users@ovirt.org>> >> > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> >> > <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> >> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> <https://www.ovirt.org/site/privacy-policy/> >> > oVirt Code of Conduct: >> > https://www.ovirt.org/community/about/community-guidelines/ >> <https://www.ovirt.org/community/about/community-guidelines/> >> > List Archives: >> > >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/BP7SG5BQKZIVR75A6NSCUMGQQBXYL3FJ/ >> < https://lists.ovirt.org/archives/list/users@ovirt.org/message/BP7SG5BQKZIVR75A6NSCUMGQQBXYL3FJ/ > >> > >> >> >> >> >> >> -- >> >> >> Raz Tamir >> Manager, RHV QE > > > > > _______________________________________________ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/HGS5KF22YQD3XSNXHSQARGZHUVDXXZAQ/ >
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/OOTD2GXB7XXQVQ...
participants (5)
-
Arik Hadas -
Gobinda Das -
Greg Sheremeta -
Jiří Sléžka -
Raz Tamir