Now I'm in a worst position,  after unlocking "all" entities.  The tasks are not present anymore,  the snapshots are not locked anymore,  but these 5 vms are not able to start:

2019-10-08 12:54:25,088+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-7) [] EVENT_ID: VM_DOWN_ERROR(119), VM openshift-04-os-infra-1 is down with error. Exit message: Bad volume specification {'protocol': 'gluster', 'address': {'function': '0x0', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'slot': '0x06'}, 'serial': 'eb6331b3-ec4f-4b8e-a1cd-cb763ada9f6a', 'index': 0, 'iface': 'virtio', 'apparentsize': '29933240320', 'specParams': {'pinToIoThread': '1'}, 'cache': 'none', 'imageID': 'eb6331b3-ec4f-4b8e-a1cd-cb763ada9f6a', 'truesize': '30025396224', 'type': 'disk', 'domainID': '97ced32d-bdb9-4913-a272-8a4a83ca3d1b', 'reqsize': '0', 'format': 'cow', 'poolID': 'c604b50e-2214-11e9-b449-00163e168814', 'device': 'disk', 'path': 'ssd-samsung-evo860/97ced32d-bdb9-4913-a272-8a4a83ca3d1b/images/eb6331b3-ec4f-4b8e-a1cd-cb763ada9f6a/f8b86437-e54c-4728-8100-ed05ef312212', 'propagateErrors': 'off', 'name': 'vda', 'bootOrder': '1', 'volumeID': 'f8b86437-e54c-4728-8100-ed05ef312212', 'diskType': 'network', 'alias': 'ua-eb6331b3-ec4f-4b8e-a1cd-cb763ada9f6a', 'hosts': [{'name': '192.168.80.191', 'port': '0'}], 'discard': False}.

So i have checked the img file,  it seems to be present an healthy:

# cd /rhev/data-center/mnt/glusterSD/192.168.80.191:_ssd-samsung-evo860/97ced32d-bdb9-4913-a272-8a4a83ca3d1b/images/eb6331b3-ec4f-4b8e-a1cd-cb763ada9f6a

 # qemu-img info f8b86437-e54c-4728-8100-ed05ef312212
image: f8b86437-e54c-4728-8100-ed05ef312212
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 28G
cluster_size: 65536
backing file: 4de5eae5-ab25-4a9e-a41a-2f7c2d8b272f
backing file format: qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

# qemu-img check f8b86437-e54c-4728-8100-ed05ef312212
No errors were found on the image.
456643/819200 = 55.74% allocated, 5.36% fragmented, 0.00% compressed clusters
Image end offset: 29933240320

# ls -l
total 62430134
-rw-rw----. 1 vdsm kvm 33863434240 Oct  7 09:40 4de5eae5-ab25-4a9e-a41a-2f7c2d8b272f
-rw-rw----. 1 vdsm kvm     1048576 Jun 27 13:10 4de5eae5-ab25-4a9e-a41a-2f7c2d8b272f.lease
-rw-r--r--. 1 vdsm kvm         338 Oct  6 19:01 4de5eae5-ab25-4a9e-a41a-2f7c2d8b272f.meta
-rw-rw----. 1 vdsm kvm 29933240320 Oct  6 19:00 f8b86437-e54c-4728-8100-ed05ef312212
-rw-rw----. 1 vdsm kvm     1048576 Jul  1 22:56 f8b86437-e54c-4728-8100-ed05ef312212.lease
-rw-r--r--. 1 vdsm kvm         271 Oct  6 19:01 f8b86437-e54c-4728-8100-ed05ef312212.meta


Any ideea how should I proceed to have the vms to start ? I am kind of stuck in this issue...
Thank you very much in advance,
Leo



On Tue, Oct 8, 2019 at 9:59 AM Leo David <leoalex@gmail.com> wrote:
Hi Everyone,
I'm waiting since 3 days for 5 x delete snapshot tasks to finish, and for some reason it seems to be stucked.For other vms snapshot removal took at most 20 mins, with havin the disks pretty much same size,  and snapshots numbers.
Any thoughts on how should I get this fixed ?
Below,  some lines from the engine.log, and it seems to show some complains regarding locks ( Failed to acquire lock and wait lock) , although I am not sure if thats the root cause:
Thank you very much !

Leo

2019-10-08 09:52:48,692+03 INFO  [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshotSingleDiskLive' (id: '341d9c1b-2915-48d6-a8a9-9146ab19d5f8') waiting on child command id: '329da0fd-801b-4e0d-b7c0-fbb5c2a98bb5' type:'DestroyImage' to complete
2019-10-08 09:52:48,702+03 INFO  [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshotSingleDiskLive' (id: '580fa033-35fd-44f0-9979-e60e9bbf8a29') waiting on child command id: 'c00bdeb6-2e8b-4ef8-a3dc-1aaa088ae052' type:'DestroyImage' to complete
2019-10-08 09:52:49,713+03 INFO  [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-50) [539ba19e-0cb5-42cf-9a23-7916ee2de4a9] Command 'RemoveSnapshotSingleDiskLive' (id: 'de747f91-ec59-4e70-9345-77e16234bfe0') waiting on child command id: '10812160-cf4c-4239-bb92-1d5a847687ee' type:'DestroyImage' to complete
2019-10-08 09:52:50,725+03 INFO  [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-100) [baed2fa3-bcad-43b2-8164-480598bc72f3] Command 'RemoveSnapshotSingleDiskLive' (id: '4919b287-e980-4d34-a219-c08a169cd8f7') waiting on child command id: '5eceb6a8-f08e-42aa-8258-c907f5927e6c' type:'DestroyImage' to complete
2019-10-08 09:52:51,563+03 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}'
2019-10-08 09:52:51,583+03 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}'
2019-10-08 09:52:51,604+03 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}'
2019-10-08 09:52:51,606+03 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}'
2019-10-08 09:52:51,735+03 INFO  [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-94) [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshot' (id: 'c9ab1344-ae27-4934-9358-d6a7b10a4f0a') waiting on child command id: '341d9c1b-2915-48d6-a8a9-9146ab19d5f8' type:'RemoveSnapshotSingleDiskLive' to complete
2019-10-08 09:52:52,706+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@21830b5f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@676adc3e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@385a3510, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@af24d00, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@331266f2], log id: 1a3515fe
2019-10-08 09:52:52,708+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterVDOVolumeListVDSCommand(HostName = host-dell-2.domain.int, VdsIdVDSCommandParametersBase:{hostId='f598de28-296a-46ce-8b8d-6f19a2c892e2'}), log id: 2a8c782
2019-10-08 09:52:52,745+03 INFO  [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-22) [539ba19e-0cb5-42cf-9a23-7916ee2de4a9] Command 'RemoveSnapshot' (id: '28ac73f3-8b07-43d4-8b4b-1df90831ed50') waiting on child command id: 'de747f91-ec59-4e70-9345-77e16234bfe0' type:'RemoveSnapshotSingleDiskLive' to complete
2019-10-08 09:52:53,115+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 2a8c782
2019-10-08 09:52:53,116+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = host-dell-1.domain.int, GlusterVolumeAdvancedDetailsVDSParameters:{hostId='109ffe86-6498-415d-8b00-1e1303a0085b', volumeName='samsung_sata'}), log id: 6bb38072
2019-10-08 09:52:54,659+03 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}'
2019-10-08 09:52:54,662+03 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}'
2019-10-08 09:52:54,663+03 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}'
2019-10-08 09:52:54,679+03 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}'
2019-10-08 09:52:54,755+03 INFO  [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-21) [4f2bedf7-b9b3-481e-936f-e180803ac1b8] Command 'RemoveSnapshot' (id: 'ceeb99c3-68be-4e0f-8dd5-3ef6509d4013') waiting on child command id: '66094952-88e3-41d9-9191-7ef8873de511' type:'RemoveSnapshotSingleDiskLive' to complete
2019-10-08 09:52:54,762+03 INFO  [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-21) [825c211d-b40d-45d7-9dc7-e5c6af28a269] Command 'RemoveSnapshotSingleDiskLive' (id: '5e9f9b03-4df9-4e31-b2e1-5efaa4fcf66c') waiting on child command id: 'b3b0a892-e652-45dd-a71f-b2f77a942491' type:'DestroyImage' to complete
2019-10-08 09:52:55,276+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterVolumeAdvancedDetailsVDSCommand, return: org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails@db5a3b94, log id: 6bb38072
2019-10-08 09:52:55,288+03 INFO  [org.ovirt.engine.core.bll.gluster.GlusterThinDeviceService] (DefaultQuartzScheduler10) [8921c9c] Volume samsung_sata have non-thin bricks, skipping confirmed free size calculation
2019-10-08 09:52:55,295+03 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}'
2019-10-08 09:52:55,308+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = host-dell-3.domain.int, VdsIdVDSCommandParametersBase:{hostId='a129fd8c-33b4-4425-92f8-1fb3bd22541e'}), log id: cc40e52
2019-10-08 09:52:55,376+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@723faf2a, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@faa8525, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@17581efa, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3bc224dd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@340b8a70, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6fd3a279, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@382bceb7, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1265e1a3, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7a55a195, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@86714cc, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6e5f3d01, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@538aa190, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@484eb7d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@34754e55, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2355daef], log id: cc40e52
2019-10-08 09:52:55,378+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = host-dell-3.domain.int, VdsIdVDSCommandParametersBase:{hostId='a129fd8c-33b4-4425-92f8-1fb3bd22541e'}), log id: 25557934
2019-10-08 09:52:55,436+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@629b1e0, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@18305096, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@153f73dd], log id: 25557934
2019-10-08 09:52:55,438+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterVDOVolumeListVDSCommand(HostName = host-dell-3.domain.int, VdsIdVDSCommandParametersBase:{hostId='a129fd8c-33b4-4425-92f8-1fb3bd22541e'}), log id: 1bad2497
2019-10-08 09:52:55,740+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 1bad2497
2019-10-08 09:52:55,742+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = host-dell-1.domain.int, VdsIdVDSCommandParametersBase:{hostId='109ffe86-6498-415d-8b00-1e1303a0085b'}), log id: 504020b3
2019-10-08 09:52:55,890+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4a6e6ecd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@24a15dc6, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@56da4c59, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@74eab89, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3e6fb5ab, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@100e90cd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@14b2ae11, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@66d25ded, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@25537e71, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@41bf0aee, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@40fce6ee, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3435f629, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@a2ba628, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@337a25f9, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@14967eee, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@f8946eb, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5844abd4], log id: 504020b3
2019-10-08 09:52:55,891+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = host-dell-1.domain.int, VdsIdVDSCommandParametersBase:{hostId='109ffe86-6498-415d-8b00-1e1303a0085b'}), log id: 3af318c8
2019-10-08 09:52:56,620+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler2) [5f2fb2c6] START, GlusterTasksListVDSCommand(HostName = host-dell-1.domain.int, VdsIdVDSCommandParametersBase:{hostId='109ffe86-6498-415d-8b00-1e1303a0085b'}), log id: 4dd7bc07
2019-10-08 09:52:56,681+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@1d356076, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@584ec325, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@2529c9f3, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@73eb0c6, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@31be4687], log id: 3af318c8
2019-10-08 09:52:56,682+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterVDOVolumeListVDSCommand(HostName = host-dell-1.domain.int, VdsIdVDSCommandParametersBase:{hostId='109ffe86-6498-415d-8b00-1e1303a0085b'}), log id: 1f47fc7e
2019-10-08 09:52:56,774+03 INFO  [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-32) [825c211d-b40d-45d7-9dc7-e5c6af28a269] Command 'RemoveSnapshotSingleDiskLive' (id: '1593ccd5-4e2a-4f8c-9c10-7944b95836b3') waiting on child command id: '8b017634-d965-4273-a4a4-6281a337ed94' type:'DestroyImage' to complete
2019-10-08 09:52:57,095+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 1f47fc7e
2019-10-08 09:52:57,096+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = host-dell-4.domain.int, VdsIdVDSCommandParametersBase:{hostId='5dde7a16-8fdb-4ae3-a061-75c451397d81'}), log id: 74cad834
2019-10-08 09:52:57,144+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@600eeeb, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@45491ad2, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@654daa09, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4764d5ec, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@46121c5d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4b0001c7, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4315e74c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@22bf8177, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@74999189, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3f5ca8e4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@531f17c4], log id: 74cad834
2019-10-08 09:52:57,145+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = host-dell-4.domain.int, VdsIdVDSCommandParametersBase:{hostId='5dde7a16-8fdb-4ae3-a061-75c451397d81'}), log id: df580e9
2019-10-08 09:52:57,191+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@35585d07], log id: df580e9
2019-10-08 09:52:57,192+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterVDOVolumeListVDSCommand(HostName = host-dell-4.domain.int, VdsIdVDSCommandParametersBase:{hostId='5dde7a16-8fdb-4ae3-a061-75c451397d81'}), log id: 6f19c8c0
2019-10-08 09:52:57,198+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler2) [5f2fb2c6] FINISH, GlusterTasksListVDSCommand, return: [], log id: 4dd7bc07
2019-10-08 09:52:57,432+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 6f19c8c0
2019-10-08 09:52:57,433+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = host-dell-2.domain.int, VdsIdVDSCommandParametersBase:{hostId='f598de28-296a-46ce-8b8d-6f19a2c892e2'}), log id: 449353d0
2019-10-08 09:52:57,576+03 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}'
2019-10-08 09:52:57,594+03 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}'
2019-10-08 09:52:57,616+03 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}'
2019-10-08 09:52:57,619+03 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler7) [306a2296] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[c6087b9e-2214-11e9-9288-00163e168814=GLUSTER]', sharedLocks=''}'
2019-10-08 09:52:57,785+03 INFO  [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-90) [4f2bedf7-b9b3-481e-936f-e180803ac1b8] Command 'RemoveSnapshotSingleDiskLive' (id: '66094952-88e3-41d9-9191-7ef8873de511') waiting on child command id: '122be070-d45f-4e6b-bb61-4943a096b88b' type:'DestroyImage' to complete
2019-10-08 09:52:57,792+03 INFO  [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-90) [baed2fa3-bcad-43b2-8164-480598bc72f3] Command 'RemoveSnapshot' (id: '7aafe2fe-e05b-44bc-a716-c48734a8c2de') waiting on child command id: '4919b287-e980-4d34-a219-c08a169cd8f7' type:'RemoveSnapshotSingleDiskLive' to complete
2019-10-08 09:52:57,799+03 INFO  [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-90) [825c211d-b40d-45d7-9dc7-e5c6af28a269] Command 'RemoveSnapshot' (id: 'cb6e7a1d-1902-4e61-9c3d-e541cf4c6348') waiting on child command id: '5e9f9b03-4df9-4e31-b2e1-5efaa4fcf66c' type:'RemoveSnapshotSingleDiskLive' to complete
2019-10-08 09:52:58,345+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@65406163, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1207b8a9, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@33f93de1, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6d2ae59d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@b07ca22, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@246ae91b, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5e86ce42, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@643ade02, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@51d95e7d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2288576c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4a9469a1, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@78ec643f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@63dde7e9, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@30effc9, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6a3e4768], log id: 449353d0
2019-10-08 09:52:58,346+03 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler10) [8921c9c] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = host-dell-2.domain.int, VdsIdVDSCommandParametersBase:{hostId='f598de28-296a-46ce-8b8d-6f19a2c892e2'}), log id: 71680b8c
2019-10-08 09:52:58,807+03 INFO  [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-81) [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshotSingleDiskLive' (id: '341d9c1b-2915-48d6-a8a9-9146ab19d5f8') waiting on child command id: '329da0fd-801b-4e0d-b7c0-fbb5c2a98bb5' type:'DestroyImage' to complete
2019-10-08 09:52:58,817+03 INFO  [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-81) [73016a4a-bb2f-487f-91c5-cd027b278930] Command 'RemoveSnapshotSingleDiskLive' (id: '580fa033-35fd-44f0-9979-e60e9bbf8a29') waiting on child command id: 'c00bdeb6-2e8b-4ef8-a3dc-1aaa088ae052' type:'DestroyImage' to complete



--
Best regards, Leo David


--
Best regards, Leo David