[ovirt-users] Re: Q: How to fix ghost "locked" status of VM

On 8 Aug 2022, at 16:27, Benny Zlotnik <bzlotnik(a)redhat.com&gt; wrote: you can do: select command_id,root_command_id from command_entities where root_command_id = '2f8b32d8-fd3c-46c9-90e9-4863d63c0530’; On Mon, Aug 8, 2022 at 4:19 PM Andrei Verovski <andreil1(a)starlett.lv&gt; wrote: > > Hi, Benny, > > > select * from command_entities where root_command_id = 'ed816f9d-e25c-4b58-8c8f-fd0393abda2f’; > -> 0 rows > > select * from command_entities where root_command_id = '2f8b32d8-fd3c-46c9-90e9-4863d63c0530’; > -> Huge page of smth, have to page out with “more”, is it possible to truncate it to meaningful value, e.g. just number of rows? > > I can restore this VM from backup copy, bit since it is unmanageable, I can’t even remove it. > > >> On 8 Aug 2022, at 13:07, Benny Zlotnik <bzlotnik(a)redhat.com&gt; wrote: >> >> you can look up the relevant command by command_id in the >> command_entities table, in your case it would be >> 2f8b32d8-fd3c-46c9-90e9-4863d63c0530 and >> ed816f9d-e25c-4b58-8c8f-fd0393abda2f, there might be more as the log >> is trimmed, so I suggest to look it up with >> select * >> from command_entities >> where root_command_id = '2f8b32d8-fd3c-46c9-90e9-4863d63c0530'; >> >> Then delete the relevant entries and restart ovirt-engine (as they >> might still be present in the cache) >> >> But before doing that, is the command still running? Async commands >> like create snapshot are failed automatically after 50 hours and I >> believe it has already passed >> Also, this manual operation is very intrusive and might have >> unexpected consequences so make sure you have backups. >> >> >> >> >> >> On Mon, Aug 8, 2022 at 9:33 AM Andrei Verovski <andreil1(a)starlett.lv&gt; wrote: >>> >>> HI, >>> >>> OK, how to properly remove this lock? Right now VM is locked and is unmanageable at all in any way. >>> I suppose its with some SQL commands in Postgres. >>> >>> Thanks. >>> >>> >>>> On 5 Aug 2022, at 11:34, Benny Zlotnik <bzlotnik(a)redhat.com&gt; wrote: >>>> >>>> So based on your logs the lock you are seeing is a memory lock, >>>> unlock_entity.sh can't really help with these. >>>> Also, the job table is used mainly for presentation so removing an >>>> entry from will not help. >>>> >>>> Do you have the logs from when this snapshot operation started, you >>>> can use the correlation id (28353fa0-5e36-4fe8-8609-e74cd1da6d36) to >>>> search? Also, do you have the vdsm logs (SPM at the time and the host >>>> running the VM), same correlation id can be used for this as well >>>> >>>> The table that's used to coordinate this is command_entities, so in >>>> theory removing the entries with this correlation id can help, but I'd >>>> like to see what led to this first >>>> >>>> On Fri, Aug 5, 2022 at 8:37 AM Andrei Verovski <andreil1(a)starlett.lv&gt; wrote: >>>>> >>>>> Hi, Benny, >>>>> >>>>> I have sent log on your mailbox, its too big to post here on mailing list. >>>>> >>>>> Looks like ghost task is still running, anything else need to be removed from Postgres DB? >>>>> >>>>> BTW, frozen dead snapshot is in invalid state, is there any way to get rid of it? I think it's actually exists, but due to invalid state its not possible to do anything with it. >>>>> >>>>> >>>>> —— >>>>> >>>>> Log file still shows zombie task: >>>>> >>>>> 2022-08-04 20:57:30,145+03 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-24) [28353fa0-5e36-4fe8-8609-e74cd1da6d36] Command 'CreateSnapshotForVm' (id: '2f8b32d8-fd3c-46c9-90e9-4863d63c0530') waiting on child command id: 'ed816f9d-e25c-4b58-8c8f-fd0393abda2f' type:'CreateLiveSnapshotForVm' to complete >>>>> 2022-08-04 20:57:40,176+03 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-85) [28353fa0-5e36-4fe8-8609-e74cd1da6d36] Command 'CreateSnapshotForVm' (id: '2f8b32d8-fd3c-46c9-90e9-4863d63c0530') waiting on child command id: 'ed816f9d-e25c-4b58-8c8f-fd0393abda2f' type:'CreateLiveSnapshotForVm' to complete >>>>> 2022-08-04 20:57:50,252+03 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-66) [28353fa0-5e36-4fe8-8609-e74cd1da6d36] Command 'CreateSnapshotForVm' (id: '2f8b32d8-fd3c-46c9-90e9-4863d63c0530') waiting on child command id: 'ed816f9d-e25c-4b58-8c8f-fd0393abda2f' type:'CreateLiveSnapshotForVm' to complete >>>>> >>>>> >>>>>> On 4 Aug 2022, at 19:06, Benny Zlotnik <bzlotnik(a)redhat.com&gt; wrote: >>>>>> >>>>>> can you share the logs after restarting ovirt-engine? >>>>>> >>>>>> On Thu, Aug 4, 2022 at 4:58 PM Andrei Verovski <andreil1(a)starlett.lv&gt; wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> >>>>>>> Creating snapshot of one of the VM vailed, and zombie tasks was killed with: >>>>>>> >>>>>>> su postgres >>>>>>> psql -d engine -U postgres >>>>>>> select * from job order by start_time desc; >>>>>>> >>>>>>> select DeleteJob('UUID_FROZEN_TASK_ID’); >>>>>>> >>>>>>> >>>>>>> However, VM remains in locked state (with lock sign left-below red “DOWN” arrow in status column of web interface. >>>>>>> >>>>>>> I run: >>>>>>> /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t all >>>>>>> >>>>>>> then rebooted engine VM, still no luck. Can’t do anything with that VM. >>>>>>> >>>>>>> Please advise how to fix. >>>>>>> Thanks in advance. >>>>>>> _______________________________________________ >>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>>>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/P2TVMLHC53J... >>>>>> >>>>> >>>> >>> >> >