Hi Everyone, I'm waiting since 3 days for 5 x delete snapshot tasks to finish, and for some reason it seems to be stucked.For other vms snapshot removal took at most 20 mins, with havin the disks pretty much same size, and snapshots numbers. Any thoughts on how should I get this fixed ? Below, some lines from the engine.log, and it seems to show some complains regarding locks ( Failed to acquire lock and wait lock) , although I am not sure if thats the root cause: Thank you very much ! Leo 2019-10-08 09:52:48,692+03 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshotSingleDiskLive' (id: '341d9c1b-2915-48d6-a8a9-9146ab19d5f8') waiting on child command id: '329da0fd-801b-4e0d-b7c0-fbb5c2a98bb5' type:'DestroyImage' to complete 2019-10-08 09:52:48,702+03 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshotSingleDiskLive' (id: '580fa033-35fd-44f0-9979-e60e9bbf8a29') waiting on child command id: 'c00bdeb6-2e8b-4ef8-a3dc-1aaa088ae052' type:'DestroyImage' to complete 2019-10-08 09:52:49,713+03 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-50) [539ba19e-0cb5-42cf-9a23-7916ee2de4a9] Command 'RemoveSnapshotSingleDiskLive' (id: 'de747f91-ec59-4e70-9345-77e16234bfe0') waiting on child command id: '10812160-cf4c-4239-bb92-1d5a847687ee' type:'DestroyImage' to complete 2019-10-08 09:52:50,725+03 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-100) [baed2fa3-bcad-43b2-8164-480598bc72f3] Command 'RemoveSnapshotSingleDiskLive' (id: '4919b287-e980-4d34-a219-c08a169cd8f7') waiting on child command id: '5eceb6a8-f08e-42aa-8258-c907f5927e6c' type:'DestroyImage' to complete 2019-10-08 09:52:51,563+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:51,583+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:51,604+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:51,606+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:51,735+03 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-94) [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshot' (id: 'c9ab1344-ae27-4934-9358-d6a7b10a4f0a') waiting on child command id: '341d9c1b-2915-48d6-a8a9-9146ab19d5f8' type:'RemoveSnapshotSingleDiskLive' to complete 2019-10-08 09:52:52,706+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@21830b5f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@676adc3e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@385a3510, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@af24d00, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@331266f2], log id: 1a3515fe 2019-10-08 09:52:52,708+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterVDOVolumeListVDSCommand(HostName = host-dell-2.domain.int, VdsIdVDSCommandParametersBase:{hostId='f598de28-296a-46ce-8b8d-6f19a2c892e2'}), log id: 2a8c782 2019-10-08 09:52:52,745+03 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-22) [539ba19e-0cb5-42cf-9a23-7916ee2de4a9] Command 'RemoveSnapshot' (id: '28ac73f3-8b07-43d4-8b4b-1df90831ed50') waiting on child command id: 'de747f91-ec59-4e70-9345-77e16234bfe0' type:'RemoveSnapshotSingleDiskLive' to complete 2019-10-08 09:52:53,115+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 2a8c782 2019-10-08 09:52:53,116+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = host-dell-1.domain.int, GlusterVolumeAdvancedDetailsVDSParameters:{hostId='109ffe86-6498-415d-8b00-1e1303a0085b', volumeName='samsung_sata'}), log id: 6bb38072 2019-10-08 09:52:54,659+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:54,662+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:54,663+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:54,679+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:54,755+03 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-21) [4f2bedf7-b9b3-481e-936f-e180803ac1b8] Command 'RemoveSnapshot' (id: 'ceeb99c3-68be-4e0f-8dd5-3ef6509d4013') waiting on child command id: '66094952-88e3-41d9-9191-7ef8873de511' type:'RemoveSnapshotSingleDiskLive' to complete 2019-10-08 09:52:54,762+03 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-21) [825c211d-b40d-45d7-9dc7-e5c6af28a269] Command 'RemoveSnapshotSingleDiskLive' (id: '5e9f9b03-4df9-4e31-b2e1-5efaa4fcf66c') waiting on child command id: 'b3b0a892-e652-45dd-a71f-b2f77a942491' type:'DestroyImage' to complete 2019-10-08 09:52:55,276+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterVolumeAdvancedDetailsVDSCommand, return: org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails@db5a3b94, log id: 6bb38072 2019-10-08 09:52:55,288+03 INFO [org.ovirt.engine.core.bll.gluster.GlusterThinDeviceService] (DefaultQuartzScheduler10) [8921c9c] Volume samsung_sata have non-thin bricks, skipping confirmed free size calculation 2019-10-08 09:52:55,295+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:55,308+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = host-dell-3.domain.int, VdsIdVDSCommandParametersBase:{hostId='a129fd8c-33b4-4425-92f8-1fb3bd22541e'}), log id: cc40e52 2019-10-08 09:52:55,376+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@723faf2a, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@faa8525, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@17581efa, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3bc224dd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@340b8a70, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6fd3a279, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@382bceb7, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1265e1a3, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7a55a195, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@86714cc, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6e5f3d01, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@538aa190, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@484eb7d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@34754e55, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2355daef], log id: cc40e52 2019-10-08 09:52:55,378+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = host-dell-3.domain.int, VdsIdVDSCommandParametersBase:{hostId='a129fd8c-33b4-4425-92f8-1fb3bd22541e'}), log id: 25557934 2019-10-08 09:52:55,436+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@629b1e0, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@18305096, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@153f73dd], log id: 25557934 2019-10-08 09:52:55,438+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterVDOVolumeListVDSCommand(HostName = host-dell-3.domain.int, VdsIdVDSCommandParametersBase:{hostId='a129fd8c-33b4-4425-92f8-1fb3bd22541e'}), log id: 1bad2497 2019-10-08 09:52:55,740+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 1bad2497 2019-10-08 09:52:55,742+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = host-dell-1.domain.int, VdsIdVDSCommandParametersBase:{hostId='109ffe86-6498-415d-8b00-1e1303a0085b'}), log id: 504020b3 2019-10-08 09:52:55,890+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4a6e6ecd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@24a15dc6, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@56da4c59, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@74eab89, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3e6fb5ab, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@100e90cd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@14b2ae11, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@66d25ded, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@25537e71, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@41bf0aee, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@40fce6ee, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3435f629, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@a2ba628, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@337a25f9, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@14967eee, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@f8946eb, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5844abd4], log id: 504020b3 2019-10-08 09:52:55,891+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = host-dell-1.domain.int, VdsIdVDSCommandParametersBase:{hostId='109ffe86-6498-415d-8b00-1e1303a0085b'}), log id: 3af318c8 2019-10-08 09:52:56,620+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler2) [5f2fb2c6] START, GlusterTasksListVDSCommand(HostName = host-dell-1.domain.int, VdsIdVDSCommandParametersBase:{hostId='109ffe86-6498-415d-8b00-1e1303a0085b'}), log id: 4dd7bc07 2019-10-08 09:52:56,681+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@1d356076, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@584ec325, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@2529c9f3, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@73eb0c6, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@31be4687], log id: 3af318c8 2019-10-08 09:52:56,682+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterVDOVolumeListVDSCommand(HostName = host-dell-1.domain.int, VdsIdVDSCommandParametersBase:{hostId='109ffe86-6498-415d-8b00-1e1303a0085b'}), log id: 1f47fc7e 2019-10-08 09:52:56,774+03 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-32) [825c211d-b40d-45d7-9dc7-e5c6af28a269] Command 'RemoveSnapshotSingleDiskLive' (id: '1593ccd5-4e2a-4f8c-9c10-7944b95836b3') waiting on child command id: '8b017634-d965-4273-a4a4-6281a337ed94' type:'DestroyImage' to complete 2019-10-08 09:52:57,095+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 1f47fc7e 2019-10-08 09:52:57,096+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = host-dell-4.domain.int, VdsIdVDSCommandParametersBase:{hostId='5dde7a16-8fdb-4ae3-a061-75c451397d81'}), log id: 74cad834 2019-10-08 09:52:57,144+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@600eeeb, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@45491ad2, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@654daa09, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4764d5ec, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@46121c5d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4b0001c7, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4315e74c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@22bf8177, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@74999189, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3f5ca8e4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@531f17c4], log id: 74cad834 2019-10-08 09:52:57,145+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = host-dell-4.domain.int, VdsIdVDSCommandParametersBase:{hostId='5dde7a16-8fdb-4ae3-a061-75c451397d81'}), log id: df580e9 2019-10-08 09:52:57,191+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@35585d07], log id: df580e9 2019-10-08 09:52:57,192+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterVDOVolumeListVDSCommand(HostName = host-dell-4.domain.int, VdsIdVDSCommandParametersBase:{hostId='5dde7a16-8fdb-4ae3-a061-75c451397d81'}), log id: 6f19c8c0 2019-10-08 09:52:57,198+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler2) [5f2fb2c6] FINISH, GlusterTasksListVDSCommand, return: [], log id: 4dd7bc07 2019-10-08 09:52:57,432+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 6f19c8c0 2019-10-08 09:52:57,433+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = host-dell-2.domain.int, VdsIdVDSCommandParametersBase:{hostId='f598de28-296a-46ce-8b8d-6f19a2c892e2'}), log id: 449353d0 2019-10-08 09:52:57,576+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:57,594+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:57,616+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:57,619+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:57,785+03 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-90) [4f2bedf7-b9b3-481e-936f-e180803ac1b8] Command 'RemoveSnapshotSingleDiskLive' (id: '66094952-88e3-41d9-9191-7ef8873de511') waiting on child command id: '122be070-d45f-4e6b-bb61-4943a096b88b' type:'DestroyImage' to complete 2019-10-08 09:52:57,792+03 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-90) [baed2fa3-bcad-43b2-8164-480598bc72f3] Command 'RemoveSnapshot' (id: '7aafe2fe-e05b-44bc-a716-c48734a8c2de') waiting on child command id: '4919b287-e980-4d34-a219-c08a169cd8f7' type:'RemoveSnapshotSingleDiskLive' to complete 2019-10-08 09:52:57,799+03 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-90) [825c211d-b40d-45d7-9dc7-e5c6af28a269] Command 'RemoveSnapshot' (id: 'cb6e7a1d-1902-4e61-9c3d-e541cf4c6348') waiting on child command id: '5e9f9b03-4df9-4e31-b2e1-5efaa4fcf66c' type:'RemoveSnapshotSingleDiskLive' to complete 2019-10-08 09:52:58,345+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@65406163, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1207b8a9, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@33f93de1, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6d2ae59d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@b07ca22, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@246ae91b, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5e86ce42, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@643ade02, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@51d95e7d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2288576c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4a9469a1, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@78ec643f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@63dde7e9, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@30effc9, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6a3e4768], log id: 449353d0 2019-10-08 09:52:58,346+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = host-dell-2.domain.int, VdsIdVDSCommandParametersBase:{hostId='f598de28-296a-46ce-8b8d-6f19a2c892e2'}), log id: 71680b8c 2019-10-08 09:52:58,807+03 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-81) [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshotSingleDiskLive' (id: '341d9c1b-2915-48d6-a8a9-9146ab19d5f8') waiting on child command id: '329da0fd-801b-4e0d-b7c0-fbb5c2a98bb5' type:'DestroyImage' to complete 2019-10-08 09:52:58,817+03 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-81) [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshotSingleDiskLive' (id: '580fa033-35fd-44f0-9979-e60e9bbf8a29') waiting on child command id: 'c00bdeb6-2e8b-4ef8-a3dc-1aaa088ae052' type:'DestroyImage' to complete -- Best regards, Leo David
Now I'm in a worst position, after unlocking "all" entities. The tasks are not present anymore, the snapshots are not locked anymore, but these 5 vms are not able to start: 2019-10-08 12:54:25,088+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-7) [] EVENT_ID: VM_DOWN_ERROR(119), VM openshift-04-os-infra-1 is down with error. Exit message: Bad volume specification {'protocol': 'gluster', 'address': {'function': '0x0', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'slot': '0x06'}, 'serial': 'eb6331b3-ec4f-4b8e-a1cd-cb763ada9f6a', 'index': 0, 'iface': 'virtio', 'apparentsize': '29933240320', 'specParams': {'pinToIoThread': '1'}, 'cache': 'none', 'imageID': 'eb6331b3-ec4f-4b8e-a1cd-cb763ada9f6a', 'truesize': '30025396224', 'type': 'disk', 'domainID': '97ced32d-bdb9-4913-a272-8a4a83ca3d1b', 'reqsize': '0', 'format': 'cow', 'poolID': 'c604b50e-2214-11e9-b449-00163e168814', 'device': 'disk', 'path': 'ssd-samsung-evo860/97ced32d-bdb9-4913-a272-8a4a83ca3d1b/images/eb6331b3-ec4f-4b8e-a1cd-cb763ada9f6a/ *f8b86437-e54c-4728-8100-ed05ef312212'*, 'propagateErrors': 'off', 'name': 'vda', 'bootOrder': '1', 'volumeID': 'f8b86437-e54c-4728-8100-ed05ef312212', 'diskType': 'network', 'alias': 'ua-eb6331b3-ec4f-4b8e-a1cd-cb763ada9f6a', 'hosts': [{'name': '192.168.80.191', 'port': '0'}], 'discard': False}. So i have checked the img file, it seems to be present an healthy: # cd /rhev/data-center/mnt/glusterSD/192.168.80.191: _ssd-samsung-evo860/97ced32d-bdb9-4913-a272-8a4a83ca3d1b/images/eb6331b3-ec4f-4b8e-a1cd-cb763ada9f6a # qemu-img info *f8b86437-e54c-4728-8100-ed05ef312212* image: f8b86437-e54c-4728-8100-ed05ef312212 file format: qcow2 virtual size: 50G (53687091200 bytes) disk size: 28G cluster_size: 65536 backing file: 4de5eae5-ab25-4a9e-a41a-2f7c2d8b272f backing file format: qcow2 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false # qemu-img check *f8b86437-e54c-4728-8100-ed05ef312212* No errors were found on the image. 456643/819200 = 55.74% allocated, 5.36% fragmented, 0.00% compressed clusters Image end offset: 29933240320 # ls -l total 62430134 -rw-rw----. 1 vdsm kvm 33863434240 Oct 7 09:40 4de5eae5-ab25-4a9e-a41a-2f7c2d8b272f -rw-rw----. 1 vdsm kvm 1048576 Jun 27 13:10 4de5eae5-ab25-4a9e-a41a-2f7c2d8b272f.lease -rw-r--r--. 1 vdsm kvm 338 Oct 6 19:01 4de5eae5-ab25-4a9e-a41a-2f7c2d8b272f.meta -rw-rw----. 1 vdsm kvm 29933240320 Oct 6 19:00 *f8b86437-e54c-4728-8100-ed05ef312212* -rw-rw----. 1 vdsm kvm 1048576 Jul 1 22:56 f8b86437-e54c-4728-8100-ed05ef312212.lease -rw-r--r--. 1 vdsm kvm 271 Oct 6 19:01 f8b86437-e54c-4728-8100-ed05ef312212.meta Any ideea how should I proceed to have the vms to start ? I am kind of stuck in this issue... Thank you very much in advance, Leo On Tue, Oct 8, 2019 at 9:59 AM Leo David <leoalex@gmail.com> wrote:
Hi Everyone, I'm waiting since 3 days for 5 x delete snapshot tasks to finish, and for some reason it seems to be stucked.For other vms snapshot removal took at most 20 mins, with havin the disks pretty much same size, and snapshots numbers. Any thoughts on how should I get this fixed ? Below, some lines from the engine.log, and it seems to show some complains regarding locks ( Failed to acquire lock and wait lock) , although I am not sure if thats the root cause: Thank you very much !
Leo
2019-10-08 09:52:48,692+03 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshotSingleDiskLive' (id: '341d9c1b-2915-48d6-a8a9-9146ab19d5f8') waiting on child command id: '329da0fd-801b-4e0d-b7c0-fbb5c2a98bb5' type:'DestroyImage' to complete 2019-10-08 09:52:48,702+03 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshotSingleDiskLive' (id: '580fa033-35fd-44f0-9979-e60e9bbf8a29') waiting on child command id: 'c00bdeb6-2e8b-4ef8-a3dc-1aaa088ae052' type:'DestroyImage' to complete 2019-10-08 09:52:49,713+03 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-50) [539ba19e-0cb5-42cf-9a23-7916ee2de4a9] Command 'RemoveSnapshotSingleDiskLive' (id: 'de747f91-ec59-4e70-9345-77e16234bfe0') waiting on child command id: '10812160-cf4c-4239-bb92-1d5a847687ee' type:'DestroyImage' to complete 2019-10-08 09:52:50,725+03 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-100) [baed2fa3-bcad-43b2-8164-480598bc72f3] Command 'RemoveSnapshotSingleDiskLive' (id: '4919b287-e980-4d34-a219-c08a169cd8f7') waiting on child command id: '5eceb6a8-f08e-42aa-8258-c907f5927e6c' type:'DestroyImage' to complete 2019-10-08 09:52:51,563+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:51,583+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:51,604+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:51,606+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:51,735+03 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-94) [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshot' (id: 'c9ab1344-ae27-4934-9358-d6a7b10a4f0a') waiting on child command id: '341d9c1b-2915-48d6-a8a9-9146ab19d5f8' type:'RemoveSnapshotSingleDiskLive' to complete 2019-10-08 09:52:52,706+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@21830b5f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@676adc3e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@385a3510, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@af24d00, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@331266f2], log id: 1a3515fe 2019-10-08 09:52:52,708+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterVDOVolumeListVDSCommand(HostName = host-dell-2.domain.int, VdsIdVDSCommandParametersBase:{hostId='f598de28-296a-46ce-8b8d-6f19a2c892e2'}), log id: 2a8c782 2019-10-08 09:52:52,745+03 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-22) [539ba19e-0cb5-42cf-9a23-7916ee2de4a9] Command 'RemoveSnapshot' (id: '28ac73f3-8b07-43d4-8b4b-1df90831ed50') waiting on child command id: 'de747f91-ec59-4e70-9345-77e16234bfe0' type:'RemoveSnapshotSingleDiskLive' to complete 2019-10-08 09:52:53,115+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 2a8c782 2019-10-08 09:52:53,116+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = host-dell-1.domain.int, GlusterVolumeAdvancedDetailsVDSParameters:{hostId='109ffe86-6498-415d-8b00-1e1303a0085b', volumeName='samsung_sata'}), log id: 6bb38072 2019-10-08 09:52:54,659+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:54,662+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:54,663+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:54,679+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:54,755+03 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-21) [4f2bedf7-b9b3-481e-936f-e180803ac1b8] Command 'RemoveSnapshot' (id: 'ceeb99c3-68be-4e0f-8dd5-3ef6509d4013') waiting on child command id: '66094952-88e3-41d9-9191-7ef8873de511' type:'RemoveSnapshotSingleDiskLive' to complete 2019-10-08 09:52:54,762+03 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-21) [825c211d-b40d-45d7-9dc7-e5c6af28a269] Command 'RemoveSnapshotSingleDiskLive' (id: '5e9f9b03-4df9-4e31-b2e1-5efaa4fcf66c') waiting on child command id: 'b3b0a892-e652-45dd-a71f-b2f77a942491' type:'DestroyImage' to complete 2019-10-08 09:52:55,276+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterVolumeAdvancedDetailsVDSCommand, return: org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails@db5a3b94, log id: 6bb38072 2019-10-08 09:52:55,288+03 INFO [org.ovirt.engine.core.bll.gluster.GlusterThinDeviceService] (DefaultQuartzScheduler10) [8921c9c] Volume samsung_sata have non-thin bricks, skipping confirmed free size calculation 2019-10-08 09:52:55,295+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:55,308+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = host-dell-3.domain.int, VdsIdVDSCommandParametersBase:{hostId='a129fd8c-33b4-4425-92f8-1fb3bd22541e'}), log id: cc40e52 2019-10-08 09:52:55,376+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@723faf2a, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@faa8525, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@17581efa, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3bc224dd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@340b8a70, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6fd3a279, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@382bceb7, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1265e1a3, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7a55a195, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@86714cc, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6e5f3d01, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@538aa190, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@484eb7d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@34754e55, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2355daef], log id: cc40e52 2019-10-08 09:52:55,378+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = host-dell-3.domain.int, VdsIdVDSCommandParametersBase:{hostId='a129fd8c-33b4-4425-92f8-1fb3bd22541e'}), log id: 25557934 2019-10-08 09:52:55,436+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@629b1e0, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@18305096, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@153f73dd], log id: 25557934 2019-10-08 09:52:55,438+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterVDOVolumeListVDSCommand(HostName = host-dell-3.domain.int, VdsIdVDSCommandParametersBase:{hostId='a129fd8c-33b4-4425-92f8-1fb3bd22541e'}), log id: 1bad2497 2019-10-08 09:52:55,740+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 1bad2497 2019-10-08 09:52:55,742+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = host-dell-1.domain.int, VdsIdVDSCommandParametersBase:{hostId='109ffe86-6498-415d-8b00-1e1303a0085b'}), log id: 504020b3 2019-10-08 09:52:55,890+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4a6e6ecd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@24a15dc6, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@56da4c59, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@74eab89, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3e6fb5ab, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@100e90cd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@14b2ae11, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@66d25ded, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@25537e71, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@41bf0aee, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@40fce6ee, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3435f629, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@a2ba628, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@337a25f9, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@14967eee, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@f8946eb, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5844abd4], log id: 504020b3 2019-10-08 09:52:55,891+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = host-dell-1.domain.int, VdsIdVDSCommandParametersBase:{hostId='109ffe86-6498-415d-8b00-1e1303a0085b'}), log id: 3af318c8 2019-10-08 09:52:56,620+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler2) [5f2fb2c6] START, GlusterTasksListVDSCommand(HostName = host-dell-1.domain.int, VdsIdVDSCommandParametersBase:{hostId='109ffe86-6498-415d-8b00-1e1303a0085b'}), log id: 4dd7bc07 2019-10-08 09:52:56,681+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@1d356076, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@584ec325, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@2529c9f3, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@73eb0c6, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@31be4687], log id: 3af318c8 2019-10-08 09:52:56,682+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterVDOVolumeListVDSCommand(HostName = host-dell-1.domain.int, VdsIdVDSCommandParametersBase:{hostId='109ffe86-6498-415d-8b00-1e1303a0085b'}), log id: 1f47fc7e 2019-10-08 09:52:56,774+03 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-32) [825c211d-b40d-45d7-9dc7-e5c6af28a269] Command 'RemoveSnapshotSingleDiskLive' (id: '1593ccd5-4e2a-4f8c-9c10-7944b95836b3') waiting on child command id: '8b017634-d965-4273-a4a4-6281a337ed94' type:'DestroyImage' to complete 2019-10-08 09:52:57,095+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 1f47fc7e 2019-10-08 09:52:57,096+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = host-dell-4.domain.int, VdsIdVDSCommandParametersBase:{hostId='5dde7a16-8fdb-4ae3-a061-75c451397d81'}), log id: 74cad834 2019-10-08 09:52:57,144+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@600eeeb, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@45491ad2, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@654daa09, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4764d5ec, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@46121c5d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4b0001c7, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4315e74c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@22bf8177, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@74999189, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3f5ca8e4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@531f17c4], log id: 74cad834 2019-10-08 09:52:57,145+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = host-dell-4.domain.int, VdsIdVDSCommandParametersBase:{hostId='5dde7a16-8fdb-4ae3-a061-75c451397d81'}), log id: df580e9 2019-10-08 09:52:57,191+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@35585d07], log id: df580e9 2019-10-08 09:52:57,192+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterVDOVolumeListVDSCommand(HostName = host-dell-4.domain.int, VdsIdVDSCommandParametersBase:{hostId='5dde7a16-8fdb-4ae3-a061-75c451397d81'}), log id: 6f19c8c0 2019-10-08 09:52:57,198+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler2) [5f2fb2c6] FINISH, GlusterTasksListVDSCommand, return: [], log id: 4dd7bc07 2019-10-08 09:52:57,432+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 6f19c8c0 2019-10-08 09:52:57,433+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = host-dell-2.domain.int, VdsIdVDSCommandParametersBase:{hostId='f598de28-296a-46ce-8b8d-6f19a2c892e2'}), log id: 449353d0 2019-10-08 09:52:57,576+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:57,594+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:57,616+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:57,619+03 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}' 2019-10-08 09:52:57,785+03 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-90) [4f2bedf7-b9b3-481e-936f-e180803ac1b8] Command 'RemoveSnapshotSingleDiskLive' (id: '66094952-88e3-41d9-9191-7ef8873de511') waiting on child command id: '122be070-d45f-4e6b-bb61-4943a096b88b' type:'DestroyImage' to complete 2019-10-08 09:52:57,792+03 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-90) [baed2fa3-bcad-43b2-8164-480598bc72f3] Command 'RemoveSnapshot' (id: '7aafe2fe-e05b-44bc-a716-c48734a8c2de') waiting on child command id: '4919b287-e980-4d34-a219-c08a169cd8f7' type:'RemoveSnapshotSingleDiskLive' to complete 2019-10-08 09:52:57,799+03 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-90) [825c211d-b40d-45d7-9dc7-e5c6af28a269] Command 'RemoveSnapshot' (id: 'cb6e7a1d-1902-4e61-9c3d-e541cf4c6348') waiting on child command id: '5e9f9b03-4df9-4e31-b2e1-5efaa4fcf66c' type:'RemoveSnapshotSingleDiskLive' to complete 2019-10-08 09:52:58,345+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@65406163, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1207b8a9, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@33f93de1, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6d2ae59d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@b07ca22, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@246ae91b, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5e86ce42, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@643ade02, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@51d95e7d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2288576c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4a9469a1, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@78ec643f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@63dde7e9, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@30effc9, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6a3e4768], log id: 449353d0 2019-10-08 09:52:58,346+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = host-dell-2.domain.int, VdsIdVDSCommandParametersBase:{hostId='f598de28-296a-46ce-8b8d-6f19a2c892e2'}), log id: 71680b8c 2019-10-08 09:52:58,807+03 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-81) [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshotSingleDiskLive' (id: '341d9c1b-2915-48d6-a8a9-9146ab19d5f8') waiting on child command id: '329da0fd-801b-4e0d-b7c0-fbb5c2a98bb5' type:'DestroyImage' to complete 2019-10-08 09:52:58,817+03 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-81) [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshotSingleDiskLive' (id: '580fa033-35fd-44f0-9979-e60e9bbf8a29') waiting on child command id: 'c00bdeb6-2e8b-4ef8-a3dc-1aaa088ae052' type:'DestroyImage' to complete
-- Best regards, Leo David
-- Best regards, Leo David
On Tue, Oct 8, 2019 at 12:10 PM Leo David <leoalex@gmail.com> wrote:
Now I'm in a worst position, after unlocking "all" entities. The tasks are not present anymore, the snapshots are not locked anymore, but these 5 vms are not able to start:
[snip]
Any ideea how should I proceed to have the vms to start ? I am kind of stuck in this issue... Thank you very much in advance, Leo
Are you on 4.3.6? In case could it be related with the async announcement made some hours ago? Gianluca
No, its 4.2.8... Thank you, Leo On Tue, Oct 8, 2019 at 1:29 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Tue, Oct 8, 2019 at 12:10 PM Leo David <leoalex@gmail.com> wrote:
Now I'm in a worst position, after unlocking "all" entities. The tasks are not present anymore, the snapshots are not locked anymore, but these 5 vms are not able to start:
[snip]
Any ideea how should I proceed to have the vms to start ? I am kind of stuck in this issue... Thank you very much in advance, Leo
Are you on 4.3.6? In case could it be related with the async announcement made some hours ago?
Gianluca
-- Best regards, Leo David
participants (2)
-
Gianluca Cecchi -
Leo David