We have an oVirt 4.5.6 cluster with a self-hosted engine and 92 virtual machines, and run a monthly cron job to snapshot and then clone each VM for disaster recovery. This works fine for 91 of our VMs, but recently started failing on our VM with the largest disks (two 3TB drives) with "Failed to flush the refcount block cache: No space left on device.” My reading of the error must be wrong, but it seems to be saying that storage domain "86867486-e14e-4ff4-bf7c-1b0c94816032” is filling up? This is the “Data (Master)” iSCSI domain that the VM’s disks are on, and the oVirt GUI says it has plenty of space: Size: 66559 GiB Available: 46359 GiB Used: 20200 GiB Allocated: 22589 GiB The operation fails while cloning the snapshot into a new VM. The new VM is created and the disks are successful created, but the clone fails after several hours. Neither the SPM host nor the host the operation is running on (the host the hosted-engine is running on) see any filesystems filling up from the underlying host OS. The full excerpt of the error from engine.log is below. I’d greatly appreciate any help determining what exactly is filling up and how to fix it. Many thanks, Devin PS. Please disregard my previous posts - been struggling to figure out exactly what’s going on. ——— 2026-03-19 21:40:05,508-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-71) [] BaseAsyncTask::onTaskEndSuccess: Task '8eb13f96-8a66-4559-a0e8-c7ff109410f6' (Parent Command 'AddVmFromSnapshot', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2026-03-19 21:40:05,508-04 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-71) [] Task with DB Task ID '4e0cafe5-624a-410e-a2c4-d072c47e8571' and VDSM Task ID '5a7d25ac-1ab9-4f69-85a3-9e8e081857d3' is in state Polling. End action for command c6340f4e-ddb0-4e1d-ba3a-ac82e9ee4164 will proceed when all the entity's tasks are completed. 2026-03-19 21:40:15,544-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-92) [] Failed in 'HSMGetAllTasksStatusesVDS' method 2026-03-19 21:40:15,552-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-92) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM lnxvirt06 command HSMGetAllTasksStatusesVDS failed: value=low level Image copy failed: ("Command ['/usr/bin/qemu-img', 'convert', '-p', '-t', 'none', '-T', 'none', '-f', 'qcow2', '-O', 'qcow2', '-o', 'compat=1.1', '/rhev/data-center/mnt/blockSD/86867486-e14e-4ff4-bf7c-1b0c94816032/images/211e91b0-2c72-45ad-90ca-badc63cb03f3/0a1b650f-8ab5-4eef-bee0-ea557c6eeb9f', '/rhev/data-center/mnt/blockSD/86867486-e14e-4ff4-bf7c-1b0c94816032/images/5bfb9a05-103c-4041-8f94-098f1be88c67/93bbfbab-7a62-49d4-bb27-dd939dd830d0'] failed with rc=1 out=b'' err=bytearray(b'qemu-img: error while writing at byte 3220932263936: No space left on device\\nqemu-img: Failed to flush the refcount block cache: No space left on device\\n')",) abortedcode=261 2026-03-19 21:40:15,552-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-92) [] SPMAsyncTask::PollTask: Polling task '5a7d25ac-1ab9-4f69-85a3-9e8e081857d3' (Parent Command 'AddVmFromSnapshot', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'cleanSuccess'. 2026-03-19 21:40:15,554-04 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-92) [] BaseAsyncTask::logEndTaskFailure: Task '5a7d25ac-1ab9-4f69-85a3-9e8e081857d3' (Parent Command 'AddVmFromSnapshot', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: 2026-03-19 21:40:15,557-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-92) [] CommandAsyncTask::endActionIfNecessary: All tasks of command 'c6340f4e-ddb0-4e1d-ba3a-ac82e9ee4164' has ended -> executing 'endAction' 2026-03-19 21:40:15,557-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-92) [] CommandAsyncTask::endAction: Ending action for '2' tasks (command ID: 'c6340f4e-ddb0-4e1d-ba3a-ac82e9ee4164'): calling endAction '. 2026-03-19 21:40:15,557-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'AddVmFromSnapshot', 2026-03-19 21:40:15,597-04 ERROR [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] Ending command 'org.ovirt.engine.core.bll.AddVmFromSnapshotCommand' with failure. 2026-03-19 21:40:15,612-04 ERROR [org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand' with failure. 2026-03-19 21:40:15,626-04 INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage 2026-03-19 21:40:15,669-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] START, DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='00000001-0001-0001-0001-000000000311', ignoreFailoverLimit='false', storageDomainId='86867486-e14e-4ff4-bf7c-1b0c94816032', imageGroupId='5bfb9a05-103c-4041-8f94-098f1be88c67', postZeros='false', discard='false', forceDelete='false'}), log id: 32b75a5f 2026-03-19 21:40:15,797-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS failed: Image does not exist in domain: 'image=5bfb9a05-103c-4041-8f94-098f1be88c67, domain=86867486-e14e-4ff4-bf7c-1b0c94816032' 2026-03-19 21:40:15,798-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] Command 'DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='00000001-0001-0001-0001-000000000311', ignoreFailoverLimit='false', storageDomainId='86867486-e14e-4ff4-bf7c-1b0c94816032', imageGroupId='5bfb9a05-103c-4041-8f94-098f1be88c67', postZeros='false', discard='false', forceDelete='false'})' execution failed: IRSGenericException: IRSErrorException: Image does not exist in domain: 'image=5bfb9a05-103c-4041-8f94-098f1be88c67, domain=86867486-e14e-4ff4-bf7c-1b0c94816032' 2026-03-19 21:40:15,798-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] FINISH, DeleteImageGroupVDSCommand, return: , log id: 32b75a5f 2026-03-19 21:40:15,798-04 INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] Disk '5bfb9a05-103c-4041-8f94-098f1be88c67' doesn't exist on storage domain '86867486-e14e-4ff4-bf7c-1b0c94816032', rolling forward 2026-03-19 21:40:15,874-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] Removed task 'cce0a49f-58b6-404d-8873-bd3b4d1465a4' from DataBase 2026-03-19 21:40:15,881-04 ERROR [org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand' with failure. 2026-03-19 21:40:15,887-04 INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage 2026-03-19 21:40:15,900-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] START, DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='00000001-0001-0001-0001-000000000311', ignoreFailoverLimit='false', storageDomainId='86867486-e14e-4ff4-bf7c-1b0c94816032', imageGroupId='b511da9e-e42c-4e7b-95d6-ab0262d19da1', postZeros='false', discard='false', forceDelete='false'}), log id: 48da38b0 2026-03-19 21:40:16,172-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] FINISH, DeleteImageGroupVDSCommand, return: , log id: 48da38b0 2026-03-19 21:40:16,173-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 'a06dd36e-3a2d-4e19-94d5-90fa8aa861e4' 2026-03-19 21:40:16,173-04 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] CommandMultiAsyncTasks::attachTask: Attaching task '20c9a088-6790-49d2-9b0d-9fdcf424e868' to command 'a06dd36e-3a2d-4e19-94d5-90fa8aa861e4'. 2026-03-19 21:40:16,180-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] Adding task '20c9a088-6790-49d2-9b0d-9fdcf424e868' (Parent Command 'RemoveImage', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet.. 2026-03-19 21:40:16,184-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] BaseAsyncTask::startPollingTask: Starting to poll task '20c9a088-6790-49d2-9b0d-9fdcf424e868'. 2026-03-19 21:40:16,185-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] BaseAsyncTask::startPollingTask: Starting to poll task '20c9a088-6790-49d2-9b0d-9fdcf424e868'. 2026-03-19 21:40:16,216-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-838369) [] EVENT_ID: USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM VS1_bk_20260319_133427 creation. 2026-03-19 21:40:16,216-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'AddVmFromSnapshot' completed, handling the result. 2026-03-19 21:40:16,216-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'AddVmFromSnapshot' succeeded, clearing tasks. 2026-03-19 21:40:16,216-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task '5a7d25ac-1ab9-4f69-85a3-9e8e081857d3' 2026-03-19 21:40:16,216-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='00000001-0001-0001-0001-000000000311', ignoreFailoverLimit='false', taskId='5a7d25ac-1ab9-4f69-85a3-9e8e081857d3'}), log id: 11fe9bb5 2026-03-19 21:40:16,217-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [] START, HSMClearTaskVDSCommand(HostName = lnxvirt06, HSMTaskGuidBaseVDSCommandParameters:{hostId='98e9372d-e2cb-4cfa-bae9-e90a5990193f', taskId='5a7d25ac-1ab9-4f69-85a3-9e8e081857d3'}), log id: 3e60a41d 2026-03-19 21:40:16,233-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [] FINISH, HSMClearTaskVDSCommand, return: , log id: 3e60a41d 2026-03-19 21:40:16,233-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [] FINISH, SPMClearTaskVDSCommand, return: , log id: 11fe9bb5 2026-03-19 21:40:16,235-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [] BaseAsyncTask::removeTaskFromDB: Removed task '5a7d25ac-1ab9-4f69-85a3-9e8e081857d3' from DataBase 2026-03-19 21:40:16,235-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task '8eb13f96-8a66-4559-a0e8-c7ff109410f6' 2026-03-19 21:40:16,235-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='00000001-0001-0001-0001-000000000311', ignoreFailoverLimit='false', taskId='8eb13f96-8a66-4559-a0e8-c7ff109410f6'}), log id: 53d1be35 2026-03-19 21:40:16,235-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [] START, HSMClearTaskVDSCommand(HostName = lnxvirt06, HSMTaskGuidBaseVDSCommandParameters:{hostId='98e9372d-e2cb-4cfa-bae9-e90a5990193f', taskId='8eb13f96-8a66-4559-a0e8-c7ff109410f6'}), log id: 5950a159 2026-03-19 21:40:16,242-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [] FINISH, HSMClearTaskVDSCommand, return: , log id: 5950a159 2026-03-19 21:40:16,242-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [] FINISH, SPMClearTaskVDSCommand, return: , log id: 53d1be35 2026-03-19 21:40:16,244-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [] BaseAsyncTask::removeTaskFromDB: Removed task '8eb13f96-8a66-4559-a0e8-c7ff109410f6' from DataBase 2026-03-19 21:40:16,244-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 'c6340f4e-ddb0-4e1d-ba3a-ac82e9ee4164' 2026-03-19 21:40:16,948-04 INFO [org.ovirt.engine.core.sso.servlets.OAuthRevokeServlet] (default task-112) [] User admin@internal-authz with profile [internal] successfully logged out 2026-03-19 21:40:16,957-04 INFO [org.ovirt.engine.core.bll.aaa.TerminateSessionsForTokenCommand] (default task-113) [d2601da] Running command: TerminateSessionsForTokenCommand internal: true. 2026-03-19 21:40:25,557-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-80) [] Polling and updating Async Tasks: 5 tasks, 1 tasks to poll now 2026-03-19 21:40:25,579-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-80) [] SPMAsyncTask::PollTask: Polling task '20c9a088-6790-49d2-9b0d-9fdcf424e868' (Parent Command 'RemoveImage', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'success'. 2026-03-19 21:40:25,579-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-80) [] BaseAsyncTask::onTaskEndSuccess: Task '20c9a088-6790-49d2-9b0d-9fdcf424e868' (Parent Command 'RemoveImage', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2026-03-19 21:40:25,581-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-80) [] CommandAsyncTask::endActionIfNecessary: All tasks of command 'a06dd36e-3a2d-4e19-94d5-90fa8aa861e4' has ended -> executing 'endAction' 2026-03-19 21:40:25,581-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-80) [] CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: 'a06dd36e-3a2d-4e19-94d5-90fa8aa861e4'): calling endAction '. 2026-03-19 21:40:25,581-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838373) [] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'CopyImageGroup', 2026-03-19 21:40:25,581-04 ERROR [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838373) [] [within thread]: endAction for action type CopyImageGroup threw an exception.: java.lang.NullPointerException 2026-03-19 21:40:25,581-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838373) [] CommandAsyncTask::HandleEndActionResult: endAction for action type 'RemoveImage' threw an unrecoverable RuntimeException the task will be cleared. 2026-03-19 21:40:25,584-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838373) [] BaseAsyncTask::removeTaskFromDB: Removed task '20c9a088-6790-49d2-9b0d-9fdcf424e868' from DataBase 2026-03-19 21:40:25,584-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838373) [] CommandAsyncTask::HandleEndActionRuntimeException: Removing CommandMultiAsyncTasks object for entity 'a06dd36e-3a2d-4e19-94d5-90fa8aa861e4’ ———