2:31 a.m.
I'm going to answer my own question here - as I figured it out about 2
minutes after sending the email, and hopefully for other users to find
in case they have a similar issue.
The last thing I tried was clearing the command_entities table with a
DELETE statement - a bit risky but I was getting desperate. It didn't
work at first but after restarting the ovirt-engine service on the
hosted engine, it suddenly started working - so it appears that this
information is not a fresh DB lookup each time and is perhaps stored in
memory until a new LockState is taken or removed (and not even re-looked
at when a lock state fails).
I wish I knew more about how the issue came about to post a proper bug
for the devs but ultimately I just think there needs to be an extra
check for 'orphaned' command_entities entries.
have you tried to first to restart your HostedEngine ?
I did at first, both the ovirt-engine service and the HostedEngine VM -
but the problem persisted between both until I actually cleared the
command_entities table and restarted again. Thanks for replying though
im just glad I got it figured out and know about it for future.
Thanks,
Joe
On 2019-11-25 9:56 AM, Joseph Goldman wrote:
Hi *,
Trying to figure out where a VM's lockstate is stored.
I have a few VM's that seem to be stuck in lock state.
I have done an ./unlock_entity.sh -t all, and a -q gives me nothing
in a lock state.
I have jumped on psql DB direct and looked through as many tables as
I can looking for a locked status, everything seems fine.
In the engine, on these VM's if I try to do any actions I get a
message similar to:
'Failed to Acquire Lock to object
'EngineLock:{exclusiveLocks='[6c2fc524-2f13-4cad-9108-347be2e88d1a=VM]',
sharedLocks=''}''
When this issue first started, I was trying to do backups with
snapshots, I got a lot of messages similar to this:
/var/log/ovirt-engine/engine.log-20191121.gz:2019-11-20
11:44:30,441+10 WARN
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(EE-ManagedThreadFactory-engineScheduled-Thread-41) [6cafc1ef] Trying
to release a shared lock for key: '6
c2fc524-2f13-4cad-9108-347be2e88d1aVM' , but lock does not exist
/var/log/ovirt-engine/engine.log-20191121.gz:2019-11-20
11:44:30,441+10 INFO
[org.ovirt.engine.core.bll.storage.disk.image.TransferImageStatusCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-41) [6cafc1ef] Lock
freed to object '
EngineLock:{exclusiveLocks='',
sharedLocks='[6c2fc524-2f13-4cad-9108-347be2e88d1a=VM,
ab5d4e18-f0c6-4472-8d2e-8c3bd1ff1a6a=DISK]'}'
I have made sure there is nothing in async_tasks, commands and the
_entities tables for both. There was some command_entities, but I took
a backup of the engine and decided to run a delete on that table
(There was about 5 stuck records there) but that hasn't helped
The affected VMs, cant delete snapshots, cant delete their disks,
cant delete the VM, cant boot the VMs - they are just stuck in the
engine.
My ultimate question is, where is the engine looking up that
lockstate as I believe it is stuck locked in error, so I can go there
and manually force a removal of the lock state and at least delete
these VM's - as ultimately I have cloned them to get the VM's back up
and running.
Can anyone point me in the right direction? Willing to pay consulting
fee if you believe you have the answer for this one.
Thanks,
Joe
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/G2ME6XMVQRY...