hello everyone, I hope you can help me I have 3 servers with rocky linux 8.6 glusterfs in replica. Everything is working correctly, except for the snapshots. When I do a snapshot, it ends up saying "VM snapshot xxx failed". and in the section of storage and disks, the snapshot of the memory and the metada is LOCKED. Logs pasted centos-release-ceph-pacific.noarch centos-release-gluster10.noarch centos-release-nfv-common.noarch centos-release-nfv-openvswitch.noarch centos-release-opstools.noarch centos-release-ovirt45.noarch centos-release-storage-common.noarch centos-release-virt-common.noarch centos-stream-release.noarch centos-stream-repos.noarch ovirt-engine.noarch cat /var/log/ovirt-engine/engine.log | grep snapshot 2022-11-21 17:14:04,896-03 INFO [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (default task-9) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[f1acea1c-7f4f-4476-a2e2-990f01c86c4a=VM]', sharedLocks=''}' 2022-11-21 17:14:05,017-03 INFO [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-25885) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] Running command: CreateSnapshotForVmCommand internal: false. Entities affected : ID: f1acea1c-7f4f-4476-a2e2-990f01c86c4a Type: VMAction group MANIPULATE_VM_SNAPSHOTS with role type USER 2022-11-21 17:14:05,076-03 INFO [org.ovirt.engine.core.bll.snapshots.CreateSnapshotDiskCommand] (EE-ManagedThreadFactory-engine-Thread-25885) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] Running command: CreateSnapshotDiskCommand internal: true. Entities affected : ID: f1acea1c-7f4f-4476-a2e2-990f01c86c4a Type: VMAction group MANIPULATE_VM_SNAPSHOTS with role type USER 2022-11-21 17:14:05,118-03 INFO [org.ovirt.engine.core.bll.snapshots.CreateSnapshotCommand] (EE-ManagedThreadFactory-engine-Thread-25885) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] Running command: CreateSnapshotCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage 2022-11-21 17:14:05,659-03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateVolumeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25885) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] START, CreateVolumeVDSCommand( CreateVolumeVDSCommandParameters:{storagePoolId='f91fca67-aee1-4dc7-b67d-c869b6875791', ignoreFailoverLimit='false', storageDomainId='88309e23-1446-459e-b175-9e4a019cd77e', imageGroupId='4dd56401-dbda-4d13-bdb0-3d0afd7402ca', imageSizeInBytes='4605345792', volumeFormat='RAW', newImageId='9ffe2fcc-5d99-4529-8baa-1a33b48072f0', imageType='Sparse', newImageDescription='{"DiskAlias":"servervpn_snapshot_memory","DiskDescription":"Memory snapsho disk for snapshot 'testsnap' of VM 'servervpn' (VM ID: 'f1acea1c-7f4f-4476-a2e2-990f01c86c4a')"}', imageInitialSizeInBytes='0', imageId='00000000-0000-0000-0000-000000000000', sourceImageGroupId='00000000-0000-0000-0000-000000000000', shouldAddBitmaps='false', legal='true', sequenceNumber='1', bitmap='null'}), log id: 61fd9b79 2022-11-21 17:14:06,178-03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateVolumeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25885) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] START, CreateVolumeVDSCommand( CreateVolumeVDSCommandParameters:{storagePoolId='f91fca67-aee1-4dc7-b67d-c869b6875791', ignoreFailoverLimit='false', storageDomainId='88309e23-1446-459e-b175-9e4a019cd77e', imageGroupId='bb07d9d7-0048-473d-bb8d-5b141a659085', imageSizeInBytes='10240', volumeFormat='RAW', newImageId='0e10989e-e7ee-466b-96d6-831e949ed243', imageType='Preallocated', newImageDescription='{"DiskAlias":"servervpn_snapshot_metadata","DiskDescription":"Memory snapshot disk for snapshot 'testsnap' of VM 'servervpn' (VM ID: 'f1acea1c-7f4f-4476-a2e2-990f01c86c4a')"}', imageInitialSizeInBytes='0', imageId='00000000-0000-0000-0000-000000000000', sourceImageGroupId='00000000-0000-0000-0000-000000000000', shouldAddBitmaps='false', legal='true', sequenceNumber='1', bitmap='null'}), log id: 31d3ee0a 2022-11-21 17:14:12,859-03 INFO [org.ovirt.engine.core.bll.snapshots.CreateSnapshotCommand] (EE-ManagedThreadFactory-engine-Thread-25892) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] Command [id=e160791f-e4ca-4173-9628-21ee0058274d]: Updating status to 'SUCCEEDED', The command end method logic will be executed by one of its parent commands. 2022-11-21 17:14:21,956-03 INFO [org.ovirt.engine.core.bll.snapshots.CreateLiveSnapshotForVmCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-12) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] Running command: CreateLiveSnapshotForVmCommand internal: true. Entities affected : ID: f1acea1c-7f4f-4476-a2e2-990f01c86c4a Type: VMAction group MANIPULATE_VM_SNAPSHOTS with role type USER 2022-11-21 17:14:22,031-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-12) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] FINISH, GetHostJobsVDSCommand, return: {833244f8-8378-4640-b437-58ee21f80de0=HostJobInfo:{id='833244f8-8378-4640-b437-58ee21f80de0', type='virt', description='snapshot_vm', status='running', progress='null', error='null'}}, log id: 6d9165a9 2022-11-21 17:14:24,053-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-39) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] FINISH, GetHostJobsVDSCommand, return: {833244f8-8378-4640-b437-58ee21f80de0=HostJobInfo:{id='833244f8-8378-4640-b437-58ee21f80de0', type='virt', description='snapshot_vm', status='running', progress='null', error='null'}}, log id: 41362ccc 2022-11-21 17:14:28,067-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-79) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] FINISH, GetHostJobsVDSCommand, return: {833244f8-8378-4640-b437-58ee21f80de0=HostJobInfo:{id='833244f8-8378-4640-b437-58ee21f80de0', type='virt', description='snapshot_vm', status='running', progress='null', error='null'}}, log id: 7f4dba82 2022-11-21 17:14:37,794-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-56) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] FINISH, GetHostJobsVDSCommand, return: {833244f8-8378-4640-b437-58ee21f80de0=HostJobInfo:{id='833244f8-8378-4640-b437-58ee21f80de0', type='virt', description='snapshot_vm', status='running', progress='null', error='null'}}, log id: 661515b0 2022-11-21 17:14:47,808-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] FINISH, GetHostJobsVDSCommand, return: {833244f8-8378-4640-b437-58ee21f80de0=HostJobInfo:{id='833244f8-8378-4640-b437-58ee21f80de0', type='virt', description='snapshot_vm', status='running', progress='null', error='null'}}, log id: f085136 2022-11-21 17:14:57,823-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-56) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] FINISH, GetHostJobsVDSCommand, return: {833244f8-8378-4640-b437-58ee21f80de0=HostJobInfo:{id='833244f8-8378-4640-b437-58ee21f80de0', type='virt', description='snapshot_vm', status='running', progress='null', error='null'}}, log id: 5a1354bf 2022-11-21 17:15:14,726-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] FINISH, GetHostJobsVDSCommand, return: {833244f8-8378-4640-b437-58ee21f80de0=HostJobInfo:{id='833244f8-8378-4640-b437-58ee21f80de0', type='virt', description='snapshot_vm', status='running', progress='null', error='null'}}, log id: 128152b5 2022-11-21 17:15:24,741-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-12) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] FINISH, GetHostJobsVDSCommand, return: {833244f8-8378-4640-b437-58ee21f80de0=HostJobInfo:{id='833244f8-8378-4640-b437-58ee21f80de0', type='virt', description='snapshot_vm', status='failed', progress='null', error='VDSError:{code='SNAPSHOT_FAILED', message='Snapshot failed'}'}}, log id: 795a5046 2022-11-21 17:15:25,766-03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateLiveSnapshotForVmCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-83) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] Ending command 'org.ovirt.engine.core.bll.snapsho s.CreateLiveSnapshotForVmCommand' with failure. 2022-11-21 17:15:27,995-03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-9) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' with failure. <disk type='network' device='disk' snapshot='no'> 2022-11-21 17:15:28,040-03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotDiskCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-9) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotDiskCommand' with failure. 2022-11-21 17:15:28,055-03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-9) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotCommand' with failure. <disk type='network' device='disk' snapshot='no'> 2022-11-21 17:15:29,056-03 INFO [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-9) [] Lock freed to object 'EngineLock:{exclusiveLocks='[f1acea1c-7f4f-4476-a2e2-990f01c86c4a=VM]', sharedLocks=''}' 2022-11-21 17:15:29,108-03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-9) [] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_FAILURE(69), Failed to complete snapshot 'testsnap' creation for VM 'servervpn' 2022-11-21 17:14:22,031-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-12) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] FINISH, GetHostJobsVDSCommand, return: {833244f8-8378-4640-b437-58ee21f80de0=HostJobInfo:{id='833244f8-8378-4640-b437-58ee21f80de0', type='virt', description='snapshot_vm', status='running', progress='null', error='null'}}, log id: 6d9165a9 2022-11-21 17:14:24,053-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-39) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] FINISH, GetHostJobsVDSCommand, return: {833244f8-8378-4640-b437-58ee21f80de0=HostJobInfo:{id='833244f8-8378-4640-b437-58ee21f80de0', type='virt', description='snapshot_vm', status='running', progress='null', error='null'}}, log id: 41362ccc 2022-11-21 17:14:28,067-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-79) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] FINISH, GetHostJobsVDSCommand, return: {833244f8-8378-4640-b437-58ee21f80de0=HostJobInfo:{id='833244f8-8378-4640-b437-58ee21f80de0', type='virt', description='snapshot_vm', status='running', progress='null', error='null'}}, log id: 7f4dba82 2022-11-21 17:14:37,794-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-56) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] FINISH, GetHostJobsVDSCommand, return: {833244f8-8378-4640-b437-58ee21f80de0=HostJobInfo:{id='833244f8-8378-4640-b437-58ee21f80de0', type='virt', description='snapshot_vm', status='running', progress='null', error='null'}}, log id: 661515b0 2022-11-21 17:14:47,808-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] FINISH, GetHostJobsVDSCommand, return: {833244f8-8378-4640-b437-58ee21f80de0=HostJobInfo:{id='833244f8-8378-4640-b437-58ee21f80de0', type='virt', description='snapshot_vm', status='running', progress='null', error='null'}}, log id: f085136 2022-11-21 17:14:57,823-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-56) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] FINISH, GetHostJobsVDSCommand, return: {833244f8-8378-4640-b437-58ee21f80de0=HostJobInfo:{id='833244f8-8378-4640-b437-58ee21f80de0', type='virt', description='snapshot_vm', status='running', progress='null', error='null'}}, log id: 5a1354bf 2022-11-21 17:15:14,726-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] FINISH, GetHostJobsVDSCommand, return: {833244f8-8378-4640-b437-58ee21f80de0=HostJobInfo:{id='833244f8-8378-4640-b437-58ee21f80de0', type='virt', description='snapshot_vm', status='running', progress='null', error='null'}}, log id: 128152b5 2022-11-21 17:15:24,741-03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-12) [2d1eb8f9-3149-4d8c-9c0a-a6e30534263f] FINISH, GetHostJobsVDSCommand, return: {833244f8-8378-4640-b437-58ee21f80de0=HostJobInfo:{id='833244f8-8378-4640-b437-58ee21f80de0', type='virt', description='snapshot_vm', status='failed', progress='null', error='VDSError:{code='SNAPSHOT_FAILED', message='Snapshot failed'}'}}, log id: 795a5046 <driver name='qemu' error_policy='report'/> <driver name='qemu' type='raw' cache='none' error_policy='stop' io='native'/> <driver name='qemu' error_policy='report'/> <driver name='qemu' type='raw' cache='none' error_policy='stop' io='native'/> cat /var/log/vdsm/vdsm.log | grep snapshot 2022-11-21 17:14:21,966-0300 INFO (jsonrpc/7) [api.virt] START snapshot(snapDrives=[{'imageID': '36c5d550-cd53-4557-ba57-7d8cf98b77e6', 'baseVolumeID': 'c4e1c0b2-6d10-4e5a-aa37-c0354476590e', 'volumeID': '91fe2d14-145f-45ef-aaa1-36df50b90771', 'domainID': '88309e23-1446-459e-b175-9e4a019cd77e'}], snapMemory='88309e23-1446-459e-b175-9e4a019cd77e,f91fca67-aee1-4dc7-b67d-c869b6875791,4dd56401-dbda-4d13-bdb0-3d0afd7402ca,9ffe2fcc-5d99-4529-8baa-1a33b48072f0,bb07d9d7-0048-473d-bb8d-5b141a659085,0e10989e-e7ee-466b-96d6-831e949ed243', frozen=False, jobUUID='833244f8-8378-4640-b437-58ee21f80de0', timeout=30, freeze_timeout=8) from=::ffff:192.168.2.180,35082, flow_id=2d1eb8f9-3149-4d8c-9c0a-a6e30534263f, vmId=f1acea1c-7f4f-4476-a2e2-990f01c86c4a (api:48) 2022-11-21 17:14:21,966-0300 INFO (jsonrpc/7) [api.virt] FINISH snapshot return={'status': {'code': 0, 'message': 'Done'}} from=::ffff:192.168.2.180,35082, flow_id=2d1eb8f9-3149-4d8c-9c0a-a6e30534263f, vmId=f1acea1c-7f4f-4476-a2e2-990f01c86c4a (api:54) 2022-11-21 17:14:21,967-0300 INFO (snap_abort/833244f8) [virt.vm] (vmId='f1acea1c-7f4f-4476-a2e2-990f01c86c4a') Starting snapshot abort job, with check interval 60 (snapshot:680) 2022-11-21 17:14:21,967-0300 INFO (virt/833244f8) [virt.vm] (vmId='f1acea1c-7f4f-4476-a2e2-990f01c86c4a') Starting snapshot job (snapshot:120) 2022-11-21 17:14:22,025-0300 INFO (jsonrpc/1) [api.host] FINISH getJobs return={'jobs': {'833244f8-8378-4640-b437-58ee21f80de0': {'id': '833244f8-8378-4640-b437-58ee21f80de0', 'status': 'running', 'description': 'snapshot_vm', 'job_type': 'virt'}}, 'status': {'code': 0, 'message': 'Done'}} from=::ffff:192.168.2.180,35082, flow_id=2d1eb8f9-3149-4d8c-9c0a-a6e30534263f (api:54) <domainsnapshot><disks><disk name="sda" snapshot="external" type="network"><source name="vmstore/88309e23-1446-459e-b175-9e4a019cd77e/images/36c5d550-cd53-4557-ba57-7d8cf98b77e6/91fe2d14-145f-45ef-aaa1-36df50b90771" protocol="gluster" type="network"><host name="10.5.10.3" port="0" transport="tcp" /><seclabel model="dac" relabel="no" type="none" /></source></disk></disks><memory file="/rhev/data-center/mnt/glusterSD/10.5.10.3:_vmstore/88309e23-1446-459e-b175-9e4a019cd77e/images/4dd56401-dbda-4d13-bdb0-3d0afd7402ca/9ffe2fcc-5d99-4529-8baa-1a33b48072f0" snapshot="external" /></domainsnapshot> (snapshot:456) 2022-11-21 17:14:23,552-0300 INFO (virt/833244f8) [virt.vm] (vmId='f1acea1c-7f4f-4476-a2e2-990f01c86c4a') Taking a live snapshot (drives=sda,memory=True) (snapshot:491) 2022-11-21 17:14:24,048-0300 INFO (jsonrpc/6) [api.host] FINISH getJobs return={'jobs': {'833244f8-8378-4640-b437-58ee21f80de0': {'id': '833244f8-8378-4640-b437-58ee21f80de0', 'status': 'running', 'description': 'snapshot_vm', 'job_type': 'virt'}}, 'status': {'code': 0, 'message': 'Done'}} from=::ffff:192.168.2.180,35082, flow_id=2d1eb8f9-3149-4d8c-9c0a-a6e30534263f (api:54) 2022-11-21 17:14:24,087-0300 INFO (snap_abort/833244f8) [virt.vm] (vmId='f1acea1c-7f4f-4476-a2e2-990f01c86c4a') Time passed: 2.1208975352346897, out of 1800 (snapshot:687) 2022-11-21 17:14:28,061-0300 INFO (jsonrpc/0) [api.host] FINISH getJobs return={'jobs': {'833244f8-8378-4640-b437-58ee21f80de0': {'id': '833244f8-8378-4640-b437-58ee21f80de0', 'status': 'running', 'description': 'snapshot_vm', 'job_type': 'virt'}}, 'status': {'code': 0, 'message': 'Done'}} from=::ffff:192.168.2.180,35082, flow_id=2d1eb8f9-3149-4d8c-9c0a-a6e30534263f (api:54) 2022-11-21 17:14:37,788-0300 INFO (jsonrpc/3) [api.host] FINISH getJobs return={'jobs': {'833244f8-8378-4640-b437-58ee21f80de0': {'id': '833244f8-8378-4640-b437-58ee21f80de0', 'status': 'running', 'description': 'snapshot_vm', 'job_type': 'virt'}}, 'status': {'code': 0, 'message': 'Done'}} from=::ffff:192.168.2.180,35082, flow_id=2d1eb8f9-3149-4d8c-9c0a-a6e30534263f (api:54) 2022-11-21 17:14:47,802-0300 INFO (jsonrpc/0) [api.host] FINISH getJobs return={'jobs': {'833244f8-8378-4640-b437-58ee21f80de0': {'id': '833244f8-8378-4640-b437-58ee21f80de0', 'status': 'running', 'description': 'snapshot_vm', 'job_type': 'virt'}}, 'status': {'code': 0, 'message': 'Done'}} from=::ffff:192.168.2.180,35082, flow_id=2d1eb8f9-3149-4d8c-9c0a-a6e30534263f (api:54) 2022-11-21 17:14:57,818-0300 INFO (jsonrpc/0) [api.host] FINISH getJobs return={'jobs': {'833244f8-8378-4640-b437-58ee21f80de0': {'id': '833244f8-8378-4640-b437-58ee21f80de0', 'status': 'running', 'description': 'snapshot_vm', 'job_type': 'virt'}}, 'status': {'code': 0, 'message': 'Done'}} from=::ffff:192.168.2.180,35082, flow_id=2d1eb8f9-3149-4d8c-9c0a-a6e30534263f (api:54) 2022-11-21 17:15:14,716-0300 INFO (jsonrpc/5) [api.host] FINISH getJobs return={'jobs': {'833244f8-8378-4640-b437-58ee21f80de0': {'id': '833244f8-8378-4640-b437-58ee21f80de0', 'status': 'running', 'description': 'snapshot_vm', 'job_type': 'virt'}}, 'status': {'code': 0, 'message': 'Done'}} from=::ffff:192.168.2.180,35082, flow_id=2d1eb8f9-3149-4d8c-9c0a-a6e30534263f (api:54) 2022-11-21 17:15:15,423-0300 ERROR (virt/833244f8) [virt.vm] (vmId='f1acea1c-7f4f-4476-a2e2-990f01c86c4a') Unable to take snapshot (snapshot:509) File "/usr/lib/python3.6/site-packages/vdsm/virt/jobs/snapshot.py", line 493, in snapshot self._vm.run_dom_snapshot(snapxml, snap_flags) File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 6173, in run_dom_snapshot self._dom.snapshotCreateXML(snapxml, snap_flags) File "/usr/lib64/python3.6/site-packages/libvirt.py", line 3070, in snapshotCreateXML File "/usr/lib/python3.6/site-packages/vdsm/virt/jobs/snapshot.py", line 131, in _run snap.snapshot() File "/usr/lib/python3.6/site-packages/vdsm/virt/jobs/snapshot.py", line 534, in snapshot raise RuntimeError("Failed to execute snapshot, " RuntimeError: Failed to execute snapshot, considering the operation as failure File "/usr/lib/python3.6/site-packages/vdsm/virt/jobs/snapshot.py", line 141, in _run 2022-11-21 17:15:24,736-0300 INFO (jsonrpc/5) [api.host] FINISH getJobs return={'jobs': {'833244f8-8378-4640-b437-58ee21f80de0': {'id': '833244f8-8378-4640-b437-58ee21f80de0', 'status': 'failed', 'description': 'snapshot_vm', 'job_type': 'virt', 'error': {'code': 48, 'message': 'Snapshot failed'}}}, 'status': {'code': 0, 'message': 'Done'}} from=::ffff:192.168.2.180,35082, flow_id=2d1eb8f9-3149-4d8c-9c0a-a6e30534263f (api:54) 2022-11-21 17:15:27,996-0300 INFO (jsonrpc/4) [api.host] FINISH dumpxmls return={'domxmls': {'f1acea1c-7f4f-4476-a2e2-990f01c86c4a': '<domain type=\'kvm\' id=\'10\' xmlns:qemu=\'http://libvirt.org/schemas/domain/qemu/1.0\'>\n <name>servervpn</name>\n <uuid>f1acea1c-7f4f-4476-a2e2-990f01c86c4a</uuid>\n <metadata xmlns:ns0="http://ovirt.org/vm/tune/1.0" xmlns:ns1="http://ovirt.org/vm/1.0">\n <ns0:qos/>\n <ovirt-vm:vm xmlns:ovirt-vm="http://ovirt.org/vm/1.0">\n <ovirt-vm:balloonTarget type="int">4194304</ovirt-vm:balloonTarget>\n <ovirt-vm:ballooningEnabled>true</ovirt-vm:ballooningEnabled>\n <ovirt-vm:clusterVersion>4.7</ovirt-vm:clusterVersion>\n <ovirt-vm:cpuPolicy>none</ovirt-vm:cpuPolicy>\n <ovirt-vm:destroy_on_reboot type="bool">False</ovirt-vm:destroy_on_reboot>\n <ovirt-vm:guestAgentAPIVersion type="int">0</ovirt-vm:guestAgentAPIVersion>\n <ovirt-vm:launchPaused>false</ovirt-vm:launchPaused>\n <ovirt-vm:memGuaranteedSize type="int">2730</ovirt-v m:memGuaranteedSize>\n <ovirt-vm:minGuaranteedMemoryMb type="int">2730</ovirt-vm:minGuaranteedMemoryMb>\n <ovirt-vm:resumeBehavior>kill</ovirt-vm:resumeBehavior>\n <ovirt-vm:startTime type="float">1664482137.5924373</ovirt-vm:startTime>\n <ovirt-vm:device alias="ua-c360503d-42ad-4a57-aff3-8f4477634b7d" mac_address="56:6f:cf:87:00:07">\n <ovirt-vm:network>ovirtmgmt</ovirt-vm:network>\n <ovirt-vm:custom>\n <ovirt-vm:queues>2</ovirt-vm:queues>\n </ovirt-vm:custom>\n </ovirt-vm:device>\n <ovirt-vm:device devtype="disk" name="sda">\n <ovirt-vm:domainID>88309e23-1446-459e-b175-9e4a019cd77e</ovirt-vm:domainID>\n <ovirt-vm:guestName>/dev/sda</ovirt-vm:guestName>\n <ovirt-vm:imageID>36c5d550-cd53-4557-ba57-7d8cf98b77e6</ovirt-vm:imageID>\n <ovirt-vm:managed type="bool">False</ovirt-vm:managed>\n <ovirt-vm:poolID>f91fca67-aee1-4dc7-b67d-c869b6875791</ovirt-vm:poolID>\n <ovirt-vm:volumeID>c4e1c0b2-6d10- 4e5a-aa37-c0354476590e</ovirt-vm:volumeID>\n <ovirt-vm:volumeChain>\n <ovirt-vm:volumeChainNode>\n <ovirt-vm:domainID>88309e23-1446-459e-b175-9e4a019cd77e</ovirt-vm:domainID>\n <ovirt-vm:imageID>36c5d550-cd53-4557-ba57-7d8cf98b77e6</ovirt-vm:imageID>\n <ovirt-vm:leaseOffset type="int">0</ovirt-vm:leaseOffset>\n <ovirt-vm:leasePath>/rhev/data-center/mnt/glusterSD/10.5.10.3:_vmstore/88309e23-1446-459e-b175-9e4a019cd77e/images/36c5d550-cd53-4557-ba57-7d8cf98b77e6/c4e1c0b2-6d10-4e5a-aa37-c0354476590e.lease</ovirt-vm:leasePath>\n <ovirt-vm:path>vmstore/88309e23-1446-459e-b175-9e4a019cd77e/images/36c5d550-cd53-4557-ba57-7d8cf98b77e6/c4e1c0b2-6d10-4e5a-aa37-c0354476590e</ovirt-vm:path>\n <ovirt-vm:volumeID>c4e1c0b2-6d10-4e5a-aa37-c0354476590e</ovirt-vm:volumeID>\n </ovirt-vm:volumeChainNode>\n </ovirt-vm:volumeChain>\n </ovirt-vm:device>\n <ovirt-vm:device dev type="disk" name="sdc">\n <ovirt-vm:managed type="bool">False</ovirt-vm:managed>\n </ovirt-vm:device>\n</ovirt-vm:vm>\n </metadata>\n <maxMemory slots=\'16\' unit=\'KiB\'>16777216</maxMemory>\n <memory unit=\'KiB\'>4194304</memory>\n <currentMemory unit=\'KiB\'>4194304</currentMemory>\n <vcpu placement=\'static\' current=\'2\'>16</vcpu>\n <iothreads>1</iothreads>\n <cputune>\n <vcpupin vcpu=\'0\' cpuset=\'0-15\'/>\n <vcpupin vcpu=\'1\' cpuset=\'0-15\'/>\n </cputune>\n <resource>\n <partition>/machine</partition>\n </resource>\n <sysinfo type=\'smbios\'>\n <system>\n <entry name=\'manufacturer\'>oVirt</entry>\n <entry name=\'product\'>RHEL</entry>\n <entry name=\'version\'>8.6-4.el8</entry>\n <entry name=\'serial\'>ecf6f35f-0fd8-65d0-3ee7-fc3497a40353</entry>\n <entry name=\'uuid\'>f1acea1c-7f4f-4476-a2e2-990f01c86c4a</entry>\n <entry name=\'family\'>oVirt</entry>\n </system>\n </sysinfo>\n <os>\n <type arch=\'x86 _64\' machine=\'pc-q35-rhel8.6.0\'>hvm</type>\n <loader readonly=\'yes\' secure=\'no\' type=\'pflash\'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>\n <nvram template=\'/usr/share/OVMF/OVMF_VARS.fd\'>/var/lib/libvirt/qemu/nvram/f1acea1c-7f4f-4476-a2e2-990f01c86c4a.fd</nvram>\n <smbios mode=\'sysinfo\'/>\n </os>\n <features>\n <acpi/>\n <smm state=\'on\'>\n <tseg unit=\'MiB\'>24</tseg>\n </smm>\n <vmcoreinfo state=\'on\'/>\n </features>\n <cpu mode=\'custom\' match=\'exact\' check=\'full\'>\n <model fallback=\'forbid\'>Skylake-Client-noTSX-IBRS</model>\n <topology sockets=\'16\' dies=\'1\' cores=\'1\' threads=\'1\'/>\n <feature policy=\'require\' name=\'ssbd\'/>\n <feature policy=\'require\' name=\'md-clear\'/>\n <feature policy=\'disable\' name=\'mpx\'/>\n <feature policy=\'require\' name=\'hypervisor\'/>\n <numa>\n <cell id=\'0\' cpus=\'0-15\' memory=\'4194304\' unit=\'KiB\'/>\n </numa>\n </cpu>\n <clock offset=\'variable \' adjustment=\'-10801\' basis=\'utc\'>\n <timer name=\'rtc\' tickpolicy=\'catchup\'/>\n <timer name=\'pit\' tickpolicy=\'delay\'/>\n <timer name=\'hpet\' present=\'no\'/>\n </clock>\n <on_poweroff>destroy</on_poweroff>\n <on_reboot>restart</on_reboot>\n <on_crash>destroy</on_crash>\n <pm>\n <suspend-to-mem enabled=\'no\'/>\n <suspend-to-disk enabled=\'no\'/>\n </pm>\n <devices>\n <emulator>/usr/libexec/qemu-kvm</emulator>\n <disk type=\'file\' device=\'cdrom\'>\n <driver name=\'qemu\' error_policy=\'report\'/>\n <source startupPolicy=\'optional\'/>\n <target dev=\'sdc\' bus=\'sata\'/>\n <readonly/>\n <alias name=\'ua-d03c597c-4a3a-4a64-b1f1-d6beba933d8a\'/>\n <address type=\'drive\' controller=\'0\' bus=\'0\' target=\'0\' unit=\'2\'/>\n </disk>\n <disk type=\'network\' device=\'disk\' snapshot=\'no\'>\n <driver name=\'qemu\' type=\'raw\' cache=\'none\' error_policy=\'stop\' io=\'native\'/>\n <source protocol= \'gluster\' name=\'vmstore/88309e23-1446-459e-b175-9e4a019cd77e/images/36c5d550-cd53-4557-ba57-7d8cf98b77e6/c4e1c0b2-6d10-4e5a-aa37-c0354476590e\' index=\'1\'>\n <host name=\'10.5.10.3\' port=\'24007\'/>\n </source>\n <backingStore/>\n <target dev=\'sda\' bus=\'scsi\'/>\n <serial>36c5d550-cd53-4557-ba57-7d8cf98b77e6</serial>\n <boot order=\'1\'/>\n <alias name=\'ua-36c5d550-cd53-4557-ba57-7d8cf98b77e6\'/>\n <address type=\'drive\' controller=\'0\' bus=\'0\' target=\'0\' unit=\'0\'/>\n </disk>\n <controller type=\'virtio-serial\' index=\'0\' ports=\'16\'>\n <alias name=\'ua-6bc99e07-233c-4bc6-a2ff-5f0d87bec2ae\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x02\' slot=\'0x00\' function=\'0x0\'/>\n </controller>\n <controller type=\'usb\' index=\'0\' model=\'qemu-xhci\' ports=\'8\'>\n <alias name=\'ua-cd4d2205-685d-43b4-96e5-4d99df80c57d\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x04\' slot=\'0x00 \' function=\'0x0\'/>\n </controller>\n <controller type=\'scsi\' index=\'0\' model=\'virtio-scsi\'>\n <driver iothread=\'1\'/>\n <alias name=\'ua-ee4bf807-9b4c-4314-899d-1786aec5aa48\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x03\' slot=\'0x00\' function=\'0x0\'/>\n </controller>\n <controller type=\'pci\' index=\'0\' model=\'pcie-root\'>\n <alias name=\'pcie.0\'/>\n </controller>\n <controller type=\'pci\' index=\'1\' model=\'pcie-root-port\'>\n <model name=\'pcie-root-port\'/>\n <target chassis=\'1\' port=\'0x10\'/>\n <alias name=\'pci.1\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x02\' function=\'0x0\' multifunction=\'on\'/>\n </controller>\n <controller type=\'pci\' index=\'2\' model=\'pcie-root-port\'>\n <model name=\'pcie-root-port\'/>\n <target chassis=\'2\' port=\'0x11\'/>\n <alias name=\'pci.2\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\ '0x02\' function=\'0x1\'/>\n </controller>\n <controller type=\'pci\' index=\'3\' model=\'pcie-root-port\'>\n <model name=\'pcie-root-port\'/>\n <target chassis=\'3\' port=\'0x12\'/>\n <alias name=\'pci.3\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x02\' function=\'0x2\'/>\n </controller>\n <controller type=\'pci\' index=\'4\' model=\'pcie-root-port\'>\n <model name=\'pcie-root-port\'/>\n <target chassis=\'4\' port=\'0x13\'/>\n <alias name=\'pci.4\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x02\' function=\'0x3\'/>\n </controller>\n <controller type=\'pci\' index=\'5\' model=\'pcie-root-port\'>\n <model name=\'pcie-root-port\'/>\n <target chassis=\'5\' port=\'0x14\'/>\n <alias name=\'pci.5\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x02\' function=\'0x4\'/>\n </controller>\n <controller type=\'pci\' index=\'6\' model=\'pcie-root-por t\'>\n <model name=\'pcie-root-port\'/>\n <target chassis=\'6\' port=\'0x15\'/>\n <alias name=\'pci.6\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x02\' function=\'0x5\'/>\n </controller>\n <controller type=\'pci\' index=\'7\' model=\'pcie-root-port\'>\n <model name=\'pcie-root-port\'/>\n <target chassis=\'7\' port=\'0x16\'/>\n <alias name=\'pci.7\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x02\' function=\'0x6\'/>\n </controller>\n <controller type=\'pci\' index=\'8\' model=\'pcie-root-port\'>\n <model name=\'pcie-root-port\'/>\n <target chassis=\'8\' port=\'0x17\'/>\n <alias name=\'pci.8\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x02\' function=\'0x7\'/>\n </controller>\n <controller type=\'pci\' index=\'9\' model=\'pcie-root-port\'>\n <model name=\'pcie-root-port\'/>\n <target chassis=\'9\' port=\'0x18\'/>\n <alias name= \'pci.9\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x03\' function=\'0x0\' multifunction=\'on\'/>\n </controller>\n <controller type=\'pci\' index=\'10\' model=\'pcie-root-port\'>\n <model name=\'pcie-root-port\'/>\n <target chassis=\'10\' port=\'0x19\'/>\n <alias name=\'pci.10\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x03\' function=\'0x1\'/>\n </controller>\n <controller type=\'pci\' index=\'11\' model=\'pcie-root-port\'>\n <model name=\'pcie-root-port\'/>\n <target chassis=\'11\' port=\'0x1a\'/>\n <alias name=\'pci.11\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x03\' function=\'0x2\'/>\n </controller>\n <controller type=\'pci\' index=\'12\' model=\'pcie-root-port\'>\n <model name=\'pcie-root-port\'/>\n <target chassis=\'12\' port=\'0x1b\'/>\n <alias name=\'pci.12\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\' 0x03\' function=\'0x3\'/>\n </controller>\n <controller type=\'pci\' index=\'13\' model=\'pcie-root-port\'>\n <model name=\'pcie-root-port\'/>\n <target chassis=\'13\' port=\'0x1c\'/>\n <alias name=\'pci.13\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x03\' function=\'0x4\'/>\n </controller>\n <controller type=\'pci\' index=\'14\' model=\'pcie-root-port\'>\n <model name=\'pcie-root-port\'/>\n <target chassis=\'14\' port=\'0x1d\'/>\n <alias name=\'pci.14\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x03\' function=\'0x5\'/>\n </controller>\n <controller type=\'pci\' index=\'15\' model=\'pcie-root-port\'>\n <model name=\'pcie-root-port\'/>\n <target chassis=\'15\' port=\'0x1e\'/>\n <alias name=\'pci.15\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x03\' function=\'0x6\'/>\n </controller>\n <controller type=\'pci\' index=\'16\' model=\'pcie -root-port\'>\n <model name=\'pcie-root-port\'/>\n <target chassis=\'16\' port=\'0x1f\'/>\n <alias name=\'pci.16\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x03\' function=\'0x7\'/>\n </controller>\n <controller type=\'sata\' index=\'0\'>\n <alias name=\'ide\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x1f\' function=\'0x2\'/>\n </controller>\n <lease>\n <lockspace>88309e23-1446-459e-b175-9e4a019cd77e</lockspace>\n <key>f1acea1c-7f4f-4476-a2e2-990f01c86c4a</key>\n <target path=\'/rhev/data-center/mnt/glusterSD/10.5.10.3:_vmstore/88309e23-1446-459e-b175-9e4a019cd77e/dom_md/xleases\' offset=\'8388608\'/>\n </lease>\n <interface type=\'bridge\'>\n <mac address=\'56:6f:cf:87:00:07\'/>\n <source bridge=\'ovirtmgmt\'/>\n <target dev=\'vnet10\'/>\n <model type=\'virtio\'/>\n <driver name=\'vhost\' queues=\'2\'/>\n <filterref filter=\'vdsm-no-mac-spoofing \'/>\n <link state=\'up\'/>\n <mtu size=\'1500\'/>\n <alias name=\'ua-c360503d-42ad-4a57-aff3-8f4477634b7d\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x01\' slot=\'0x00\' function=\'0x0\'/>\n </interface>\n <channel type=\'unix\'>\n <source mode=\'bind\' path=\'/var/lib/libvirt/qemu/channels/f1acea1c-7f4f-4476-a2e2-990f01c86c4a.org.qemu.guest_agent.0\'/>\n <target type=\'virtio\' name=\'org.qemu.guest_agent.0\' state=\'connected\'/>\n <alias name=\'channel0\'/>\n <address type=\'virtio-serial\' controller=\'0\' bus=\'0\' port=\'1\'/>\n </channel>\n <channel type=\'spicevmc\'>\n <target type=\'virtio\' name=\'com.redhat.spice.0\' state=\'disconnected\'/>\n <alias name=\'channel1\'/>\n <address type=\'virtio-serial\' controller=\'0\' bus=\'0\' port=\'2\'/>\n </channel>\n <input type=\'tablet\' bus=\'usb\'>\n <alias name=\'input0\'/>\n <address type=\'usb\' bus=\'0\' port=\'1\'/>\n </input>\n <input type=\'mouse\' bus=\'ps2\'>\n <alias name=\'input1\'/>\n </input>\n <input type=\'keyboard\' bus=\'ps2\'>\n <alias name=\'input2\'/>\n </input>\n <graphics type=\'spice\' port=\'5921\' tlsPort=\'5922\' autoport=\'yes\' listen=\'192.168.2.177\' passwdValidTo=\'1970-01-01T00:00:01\'>\n <listen type=\'network\' address=\'192.168.2.177\' network=\'vdsm-ovirtmgmt\'/>\n <channel name=\'main\' mode=\'secure\'/>\n <channel name=\'display\' mode=\'secure\'/>\n <channel name=\'inputs\' mode=\'secure\'/>\n <channel name=\'cursor\' mode=\'secure\'/>\n <channel name=\'playback\' mode=\'secure\'/>\n <channel name=\'record\' mode=\'secure\'/>\n <channel name=\'smartcard\' mode=\'secure\'/>\n <channel name=\'usbredir\' mode=\'secure\'/>\n </graphics>\n <graphics type=\'vnc\' port=\'5923\' autoport=\'yes\' listen=\'192.168.2.177\' keymap=\'en-us\' passwdValidTo=\'1970-01-01T00:00:01\'>\n <listen type=\'network\' address=\'192.168.2.177\' network=\'vdsm-ovirtmgmt\'/>\n </graphics>\n <audio id=\'1\' type=\'none\'/>\n <video>\n <model type=\'qxl\' ram=\'65536\' vram=\'32768\' vgamem=\'16384\' heads=\'1\' primary=\'yes\'/>\n <alias name=\'ua-8ea66811-4bd4-45b4-9cc8-8acc0b761741\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x00\' slot=\'0x01\' function=\'0x0\'/>\n </video>\n <memballoon model=\'virtio\'>\n <stats period=\'5\'/>\n <alias name=\'ua-ff59fffe-2d15-47c4-8f28-0710a9859882\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x05\' slot=\'0x00\' function=\'0x0\'/>\n </memballoon>\n <rng model=\'virtio\'>\n <backend model=\'random\'>/dev/urandom</backend>\n <alias name=\'ua-c8dcc7ca-f784-40e3-84eb-a972febcb3fe\'/>\n <address type=\'pci\' domain=\'0x0000\' bus=\'0x06\' slot=\'0x00\' function=\'0x0\'/>\n </rng>\n </devices>\n <seclabel type=\'dynamic\' model=\'selinux\' relabel=\'yes\'>\n <label>system_u:syst em_r:svirt_t:s0:c14,c676</label>\n <imagelabel>system_u:object_r:svirt_image_t:s0:c14,c676</imagelabel>\n </seclabel>\n <seclabel type=\'dynamic\' model=\'dac\' relabel=\'yes\'>\n <label>+107:+107</label>\n <imagelabel>+107:+107</imagelabel>\n </seclabel>\n <qemu:capabilities>\n <qemu:add capability=\'blockdev\'/>\n <qemu:add capability=\'incremental-backup\'/>\n </qemu:capabilities>\n</domain>\n'}, 'status': {'code': 0, 'message': 'Done'}} from=::ffff:192.168.2.180,35082, flow_id=2d1eb8f9-3149-4d8c-9c0a-a6e30534263f (api:54)