[ovirt-users] Re: Q: How to fix ghost "locked" status of VM

Hi, Benny, select * from command_entities where root_command_id = 'ed816f9d-e25c-4b58-8c8f-fd0393abda2f’; -> 0 rows select * from command_entities where root_command_id = '2f8b32d8-fd3c-46c9-90e9-4863d63c0530’; -> Huge page of smth, have to page out with “more”, is it possible to truncate it to meaningful value, e.g. just number of rows? I can restore this VM from backup copy, bit since it is unmanageable, I can’t even remove it. > On 8 Aug 2022, at 13:07, Benny Zlotnik <bzlotnik(a)redhat.com&gt; wrote: > > you can look up the relevant command by command_id in the > command_entities table, in your case it would be > 2f8b32d8-fd3c-46c9-90e9-4863d63c0530 and > ed816f9d-e25c-4b58-8c8f-fd0393abda2f, there might be more as the log > is trimmed, so I suggest to look it up with > select * > from command_entities > where root_command_id = '2f8b32d8-fd3c-46c9-90e9-4863d63c0530'; > > Then delete the relevant entries and restart ovirt-engine (as they > might still be present in the cache) > > But before doing that, is the command still running? Async commands > like create snapshot are failed automatically after 50 hours and I > believe it has already passed > Also, this manual operation is very intrusive and might have > unexpected consequences so make sure you have backups. > > > > > > On Mon, Aug 8, 2022 at 9:33 AM Andrei Verovski <andreil1(a)starlett.lv&gt; wrote: >> >> HI, >> >> OK, how to properly remove this lock? Right now VM is locked and is unmanageable at all in any way. >> I suppose its with some SQL commands in Postgres. >> >> Thanks. >> >> >>> On 5 Aug 2022, at 11:34, Benny Zlotnik <bzlotnik(a)redhat.com&gt; wrote: >>> >>> So based on your logs the lock you are seeing is a memory lock, >>> unlock_entity.sh can't really help with these. >>> Also, the job table is used mainly for presentation so removing an >>> entry from will not help. >>> >>> Do you have the logs from when this snapshot operation started, you >>> can use the correlation id (28353fa0-5e36-4fe8-8609-e74cd1da6d36) to >>> search? Also, do you have the vdsm logs (SPM at the time and the host >>> running the VM), same correlation id can be used for this as well >>> >>> The table that's used to coordinate this is command_entities, so in >>> theory removing the entries with this correlation id can help, but I'd >>> like to see what led to this first >>> >>> On Fri, Aug 5, 2022 at 8:37 AM Andrei Verovski <andreil1(a)starlett.lv&gt; wrote: >>>> >>>> Hi, Benny, >>>> >>>> I have sent log on your mailbox, its too big to post here on mailing list. >>>> >>>> Looks like ghost task is still running, anything else need to be removed from Postgres DB? >>>> >>>> BTW, frozen dead snapshot is in invalid state, is there any way to get rid of it? I think it's actually exists, but due to invalid state its not possible to do anything with it. >>>> >>>> >>>> —— >>>> >>>> Log file still shows zombie task: >>>> >>>> 2022-08-04 20:57:30,145+03 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-24) [28353fa0-5e36-4fe8-8609-e74cd1da6d36] Command 'CreateSnapshotForVm' (id: '2f8b32d8-fd3c-46c9-90e9-4863d63c0530') waiting on child command id: 'ed816f9d-e25c-4b58-8c8f-fd0393abda2f' type:'CreateLiveSnapshotForVm' to complete >>>> 2022-08-04 20:57:40,176+03 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-85) [28353fa0-5e36-4fe8-8609-e74cd1da6d36] Command 'CreateSnapshotForVm' (id: '2f8b32d8-fd3c-46c9-90e9-4863d63c0530') waiting on child command id: 'ed816f9d-e25c-4b58-8c8f-fd0393abda2f' type:'CreateLiveSnapshotForVm' to complete >>>> 2022-08-04 20:57:50,252+03 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-66) [28353fa0-5e36-4fe8-8609-e74cd1da6d36] Command 'CreateSnapshotForVm' (id: '2f8b32d8-fd3c-46c9-90e9-4863d63c0530') waiting on child command id: 'ed816f9d-e25c-4b58-8c8f-fd0393abda2f' type:'CreateLiveSnapshotForVm' to complete >>>> >>>> >>>>> On 4 Aug 2022, at 19:06, Benny Zlotnik <bzlotnik(a)redhat.com&gt; wrote: >>>>> >>>>> can you share the logs after restarting ovirt-engine? >>>>> >>>>> On Thu, Aug 4, 2022 at 4:58 PM Andrei Verovski <andreil1(a)starlett.lv&gt; wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> >>>>>> Creating snapshot of one of the VM vailed, and zombie tasks was killed with: >>>>>> >>>>>> su postgres >>>>>> psql -d engine -U postgres >>>>>> select * from job order by start_time desc; >>>>>> >>>>>> select DeleteJob('UUID_FROZEN_TASK_ID’); >>>>>> >>>>>> >>>>>> However, VM remains in locked state (with lock sign left-below red “DOWN” arrow in status column of web interface. >>>>>> >>>>>> I run: >>>>>> /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t all >>>>>> >>>>>> then rebooted engine VM, still no luck. Can’t do anything with that VM. >>>>>> >>>>>> Please advise how to fix. >>>>>> Thanks in advance. >>>>>> _______________________________________________ >>>>>> Users mailing list -- users(a)ovirt.org >>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/P2TVMLHC53J... >>>>> >>>> >>> >> >