- Users - oVirt List Archives

Mailinglist issues
by Jean-Louis Dupond 28 May '26

28 May '26

Hi, There was some issue with the mailinglist the last 1-2 months. This has been fixed, and mails are now accepted again! Feel free to resend your email that was dropped. Thanks Jean-Louis

1 0

Failed to flush the refcount block cache: No space left on device
by Devin A. Bougie 22 Mar '26

22 Mar '26

We have an oVirt 4.5.6 cluster with a self-hosted engine and 92 virtual machines, and run a monthly cron job to snapshot and then clone each VM for disaster recovery. This works fine for 91 of our VMs, but recently started failing on our VM with the largest disks (two 3TB drives) with "Failed to flush the refcount block cache: No space left on device.” My reading of the error must be wrong, but it seems to be saying that storage domain "86867486-e14e-4ff4-bf7c-1b0c94816032” is filling up? This is the “Data (Master)” iSCSI domain that the VM’s disks are on, and the oVirt GUI says it has plenty of space: Size: 66559 GiB Available: 46359 GiB Used: 20200 GiB Allocated: 22589 GiB The operation fails while cloning the snapshot into a new VM. The new VM is created and the disks are successful created, but the clone fails after several hours. Neither the SPM host nor the host the operation is running on (the host the hosted-engine is running on) see any filesystems filling up from the underlying host OS. The full excerpt of the error from engine.log is below. I’d greatly appreciate any help determining what exactly is filling up and how to fix it. Many thanks, Devin PS. Please disregard my previous posts - been struggling to figure out exactly what’s going on. ——— 2026-03-19 21:40:05,508-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-71) [] BaseAsyncTask::onTaskEndSuccess: Task '8eb13f96-8a66-4559-a0e8-c7ff109410f6' (Parent Command 'AddVmFromSnapshot', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2026-03-19 21:40:05,508-04 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-71) [] Task with DB Task ID '4e0cafe5-624a-410e-a2c4-d072c47e8571' and VDSM Task ID '5a7d25ac-1ab9-4f69-85a3-9e8e081857d3' is in state Polling. End action for command c6340f4e-ddb0-4e1d-ba3a-ac82e9ee4164 will proceed when all the entity's tasks are completed. 2026-03-19 21:40:15,544-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-92) [] Failed in 'HSMGetAllTasksStatusesVDS' method 2026-03-19 21:40:15,552-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-92) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM lnxvirt06 command HSMGetAllTasksStatusesVDS failed: value=low level Image copy failed: ("Command ['/usr/bin/qemu-img', 'convert', '-p', '-t', 'none', '-T', 'none', '-f', 'qcow2', '-O', 'qcow2', '-o', 'compat=1.1', '/rhev/data-center/mnt/blockSD/86867486-e14e-4ff4-bf7c-1b0c94816032/images/211e91b0-2c72-45ad-90ca-badc63cb03f3/0a1b650f-8ab5-4eef-bee0-ea557c6eeb9f', '/rhev/data-center/mnt/blockSD/86867486-e14e-4ff4-bf7c-1b0c94816032/images/5bfb9a05-103c-4041-8f94-098f1be88c67/93bbfbab-7a62-49d4-bb27-dd939dd830d0'] failed with rc=1 out=b'' err=bytearray(b'qemu-img: error while writing at byte 3220932263936: No space left on device\\nqemu-img: Failed to flush the refcount block cache: No space left on device\\n')",) abortedcode=261 2026-03-19 21:40:15,552-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-92) [] SPMAsyncTask::PollTask: Polling task '5a7d25ac-1ab9-4f69-85a3-9e8e081857d3' (Parent Command 'AddVmFromSnapshot', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'cleanSuccess'. 2026-03-19 21:40:15,554-04 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-92) [] BaseAsyncTask::logEndTaskFailure: Task '5a7d25ac-1ab9-4f69-85a3-9e8e081857d3' (Parent Command 'AddVmFromSnapshot', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: 2026-03-19 21:40:15,557-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-92) [] CommandAsyncTask::endActionIfNecessary: All tasks of command 'c6340f4e-ddb0-4e1d-ba3a-ac82e9ee4164' has ended -> executing 'endAction' 2026-03-19 21:40:15,557-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-92) [] CommandAsyncTask::endAction: Ending action for '2' tasks (command ID: 'c6340f4e-ddb0-4e1d-ba3a-ac82e9ee4164'): calling endAction '. 2026-03-19 21:40:15,557-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'AddVmFromSnapshot', 2026-03-19 21:40:15,597-04 ERROR [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] Ending command 'org.ovirt.engine.core.bll.AddVmFromSnapshotCommand' with failure. 2026-03-19 21:40:15,612-04 ERROR [org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand' with failure. 2026-03-19 21:40:15,626-04 INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage 2026-03-19 21:40:15,669-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] START, DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='00000001-0001-0001-0001-000000000311', ignoreFailoverLimit='false', storageDomainId='86867486-e14e-4ff4-bf7c-1b0c94816032', imageGroupId='5bfb9a05-103c-4041-8f94-098f1be88c67', postZeros='false', discard='false', forceDelete='false'}), log id: 32b75a5f 2026-03-19 21:40:15,797-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS failed: Image does not exist in domain: 'image=5bfb9a05-103c-4041-8f94-098f1be88c67, domain=86867486-e14e-4ff4-bf7c-1b0c94816032' 2026-03-19 21:40:15,798-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] Command 'DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='00000001-0001-0001-0001-000000000311', ignoreFailoverLimit='false', storageDomainId='86867486-e14e-4ff4-bf7c-1b0c94816032', imageGroupId='5bfb9a05-103c-4041-8f94-098f1be88c67', postZeros='false', discard='false', forceDelete='false'})' execution failed: IRSGenericException: IRSErrorException: Image does not exist in domain: 'image=5bfb9a05-103c-4041-8f94-098f1be88c67, domain=86867486-e14e-4ff4-bf7c-1b0c94816032' 2026-03-19 21:40:15,798-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] FINISH, DeleteImageGroupVDSCommand, return: , log id: 32b75a5f 2026-03-19 21:40:15,798-04 INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] Disk '5bfb9a05-103c-4041-8f94-098f1be88c67' doesn't exist on storage domain '86867486-e14e-4ff4-bf7c-1b0c94816032', rolling forward 2026-03-19 21:40:15,874-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] Removed task 'cce0a49f-58b6-404d-8873-bd3b4d1465a4' from DataBase 2026-03-19 21:40:15,881-04 ERROR [org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.CopyImageGroupCommand' with failure. 2026-03-19 21:40:15,887-04 INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage 2026-03-19 21:40:15,900-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] START, DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='00000001-0001-0001-0001-000000000311', ignoreFailoverLimit='false', storageDomainId='86867486-e14e-4ff4-bf7c-1b0c94816032', imageGroupId='b511da9e-e42c-4e7b-95d6-ab0262d19da1', postZeros='false', discard='false', forceDelete='false'}), log id: 48da38b0 2026-03-19 21:40:16,172-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] FINISH, DeleteImageGroupVDSCommand, return: , log id: 48da38b0 2026-03-19 21:40:16,173-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 'a06dd36e-3a2d-4e19-94d5-90fa8aa861e4' 2026-03-19 21:40:16,173-04 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] CommandMultiAsyncTasks::attachTask: Attaching task '20c9a088-6790-49d2-9b0d-9fdcf424e868' to command 'a06dd36e-3a2d-4e19-94d5-90fa8aa861e4'. 2026-03-19 21:40:16,180-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] Adding task '20c9a088-6790-49d2-9b0d-9fdcf424e868' (Parent Command 'RemoveImage', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet.. 2026-03-19 21:40:16,184-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] BaseAsyncTask::startPollingTask: Starting to poll task '20c9a088-6790-49d2-9b0d-9fdcf424e868'. 2026-03-19 21:40:16,185-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [6492a8e8-1028-4504-9e56-9f22213f5f24] BaseAsyncTask::startPollingTask: Starting to poll task '20c9a088-6790-49d2-9b0d-9fdcf424e868'. 2026-03-19 21:40:16,216-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-838369) [] EVENT_ID: USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM VS1_bk_20260319_133427 creation. 2026-03-19 21:40:16,216-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'AddVmFromSnapshot' completed, handling the result. 2026-03-19 21:40:16,216-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'AddVmFromSnapshot' succeeded, clearing tasks. 2026-03-19 21:40:16,216-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task '5a7d25ac-1ab9-4f69-85a3-9e8e081857d3' 2026-03-19 21:40:16,216-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='00000001-0001-0001-0001-000000000311', ignoreFailoverLimit='false', taskId='5a7d25ac-1ab9-4f69-85a3-9e8e081857d3'}), log id: 11fe9bb5 2026-03-19 21:40:16,217-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [] START, HSMClearTaskVDSCommand(HostName = lnxvirt06, HSMTaskGuidBaseVDSCommandParameters:{hostId='98e9372d-e2cb-4cfa-bae9-e90a5990193f', taskId='5a7d25ac-1ab9-4f69-85a3-9e8e081857d3'}), log id: 3e60a41d 2026-03-19 21:40:16,233-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [] FINISH, HSMClearTaskVDSCommand, return: , log id: 3e60a41d 2026-03-19 21:40:16,233-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [] FINISH, SPMClearTaskVDSCommand, return: , log id: 11fe9bb5 2026-03-19 21:40:16,235-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [] BaseAsyncTask::removeTaskFromDB: Removed task '5a7d25ac-1ab9-4f69-85a3-9e8e081857d3' from DataBase 2026-03-19 21:40:16,235-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task '8eb13f96-8a66-4559-a0e8-c7ff109410f6' 2026-03-19 21:40:16,235-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='00000001-0001-0001-0001-000000000311', ignoreFailoverLimit='false', taskId='8eb13f96-8a66-4559-a0e8-c7ff109410f6'}), log id: 53d1be35 2026-03-19 21:40:16,235-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [] START, HSMClearTaskVDSCommand(HostName = lnxvirt06, HSMTaskGuidBaseVDSCommandParameters:{hostId='98e9372d-e2cb-4cfa-bae9-e90a5990193f', taskId='8eb13f96-8a66-4559-a0e8-c7ff109410f6'}), log id: 5950a159 2026-03-19 21:40:16,242-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [] FINISH, HSMClearTaskVDSCommand, return: , log id: 5950a159 2026-03-19 21:40:16,242-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-838369) [] FINISH, SPMClearTaskVDSCommand, return: , log id: 53d1be35 2026-03-19 21:40:16,244-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [] BaseAsyncTask::removeTaskFromDB: Removed task '8eb13f96-8a66-4559-a0e8-c7ff109410f6' from DataBase 2026-03-19 21:40:16,244-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838369) [] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 'c6340f4e-ddb0-4e1d-ba3a-ac82e9ee4164' 2026-03-19 21:40:16,948-04 INFO [org.ovirt.engine.core.sso.servlets.OAuthRevokeServlet] (default task-112) [] User admin@internal-authz with profile [internal] successfully logged out 2026-03-19 21:40:16,957-04 INFO [org.ovirt.engine.core.bll.aaa.TerminateSessionsForTokenCommand] (default task-113) [d2601da] Running command: TerminateSessionsForTokenCommand internal: true. 2026-03-19 21:40:25,557-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-80) [] Polling and updating Async Tasks: 5 tasks, 1 tasks to poll now 2026-03-19 21:40:25,579-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-80) [] SPMAsyncTask::PollTask: Polling task '20c9a088-6790-49d2-9b0d-9fdcf424e868' (Parent Command 'RemoveImage', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'success'. 2026-03-19 21:40:25,579-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-80) [] BaseAsyncTask::onTaskEndSuccess: Task '20c9a088-6790-49d2-9b0d-9fdcf424e868' (Parent Command 'RemoveImage', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2026-03-19 21:40:25,581-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-80) [] CommandAsyncTask::endActionIfNecessary: All tasks of command 'a06dd36e-3a2d-4e19-94d5-90fa8aa861e4' has ended -> executing 'endAction' 2026-03-19 21:40:25,581-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-80) [] CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: 'a06dd36e-3a2d-4e19-94d5-90fa8aa861e4'): calling endAction '. 2026-03-19 21:40:25,581-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838373) [] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'CopyImageGroup', 2026-03-19 21:40:25,581-04 ERROR [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838373) [] [within thread]: endAction for action type CopyImageGroup threw an exception.: java.lang.NullPointerException 2026-03-19 21:40:25,581-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838373) [] CommandAsyncTask::HandleEndActionResult: endAction for action type 'RemoveImage' threw an unrecoverable RuntimeException the task will be cleared. 2026-03-19 21:40:25,584-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838373) [] BaseAsyncTask::removeTaskFromDB: Removed task '20c9a088-6790-49d2-9b0d-9fdcf424e868' from DataBase 2026-03-19 21:40:25,584-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-838373) [] CommandAsyncTask::HandleEndActionRuntimeException: Removing CommandMultiAsyncTasks object for entity 'a06dd36e-3a2d-4e19-94d5-90fa8aa861e4’ ———

1 0

Reclaim space from iSCSI storage domain
by Devin A. Bougie 20 Mar '26

20 Mar '26

We have an oVirt 4.5.6 cluster with a self-hosted engine with an iSCSI Data (Master) domain. The oVirt UI shows that it has plenty of space: Size: 66559 GiB Available: 46359 GiB Used: 20200 GiB Allocated: 22589 GiB When trying to create a new disk, however, we receive the errors below complaining that the Volume Group has insufficient free space. How can we safely reclaim this space, ideally without shutting down ever VM that’s using this storage domain? Although “multipath -ll” shows the underlying devices on the host machines, the normal “pvs / vgs / lvs” commands don’t see them. Many thanks, Devin ——— 2026-03-20 14:39:13,038-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-31) [] Failed in 'HSMGetAllTasksStatusesVDS' method 2026-03-20 14:39:13,080-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-31) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM lnxvirt06 command HSMGetAllTasksStatusesVDS failed: value=Cannot create Logical Volume: 'cmd=[\'/sbin/lvm\', \'lvcreate\', \'--devices\', \'/dev/mapper/mpathd,/dev/mapper/mpathj,/dev/mapper/mpathm\', \'--config\', \'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 hints="none" obtain_device_list_from_udev=0 } global { prioritise_write_locks=1 wait_for_locks=1 use_lvmpolld=1 } backup { retain_min=50 retain_days=0 }\', \'--autobackup\', \'n\', \'--contiguous\', \'n\', \'--size\', \'3072000m\', \'--wipesignatures\', \'n\', \'--addtag\', \'OVIRT_VOL_INITIALIZING\', \'--name\', \'3b8c4d37-0807-4a16-8dcf-f6aee207b4e6\', \'4f1e2b92-fe8d-47ef-b0ba-a9c7501bf254\'] rc=5 out=[] err=[\' Volume group "4f1e2b92-fe8d-47ef-b0ba-a9c7501bf254" has insufficient free space (7726 extents): 24000 required.\']' abortedcode=550 2026-03-20 14:39:13,080-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-31) [] SPMAsyncTask::PollTask: Polling task '4f9a9858-290b-4ed8-9a42-0bc8d4d7aca6' (Parent Command 'AddImageFromScratch', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'cleanSuccess'. 2026-03-20 14:39:13,083-04 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-31) [] BaseAsyncTask::logEndTaskFailure: Task '4f9a9858-290b-4ed8-9a42-0bc8d4d7aca6' (Parent Command 'AddImageFromScratch', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: -- Result: 'cleanSuccess' -- Message: 'VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = value=Cannot create Logical Volume: 'cmd=[\'/sbin/lvm\', \'lvcreate\', \'--devices\', \'/dev/mapper/mpathd,/dev/mapper/mpathj,/dev/mapper/mpathm\', \'--config\', \'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 hints="none" obtain_device_list_from_udev=0 } global { prioritise_write_locks=1 wait_for_locks=1 use_lvmpolld=1 } backup { retain_min=50 retain_days=0 }\', \'--autobackup\', \'n\', \'--contiguous\', \'n\', \'--size\', \'3072000m\', \'--wipesignatures\', \'n\', \'--addtag\', \'OVIRT_VOL_INITIALIZING\', \'--name\', \'3b8c4d37-0807-4a16-8dcf-f6aee207b4e6\', \'4f1e2b92-fe8d-47ef-b0ba-a9c7501bf254\'] rc=5 out=[] err=[\' Volume group "4f1e2b92-fe8d-47ef-b0ba-a9c7501bf254" has insufficient free space (7726 extents): 24000 required.\']' abortedcode=550, code = 550', -- Exception: 'VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = value=Cannot create Logical Volume: 'cmd=[\'/sbin/lvm\', \'lvcreate\', \'--devices\', \'/dev/mapper/mpathd,/dev/mapper/mpathj,/dev/mapper/mpathm\', \'--config\', \'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 hints="none" obtain_device_list_from_udev=0 } global { prioritise_write_locks=1 wait_for_locks=1 use_lvmpolld=1 } backup { retain_min=50 retain_days=0 }\', \'--autobackup\', \'n\', \'--contiguous\', \'n\', \'--size\', \'3072000m\', \'--wipesignatures\', \'n\', \'--addtag\', \'OVIRT_VOL_INITIALIZING\', \'--name\', \'3b8c4d37-0807-4a16-8dcf-f6aee207b4e6\', \'4f1e2b92-fe8d-47ef-b0ba-a9c7501bf254\'] rc=5 out=[] err=[\' Volume group "4f1e2b92-fe8d-47ef-b0ba-a9c7501bf254" has insufficient free space (7726 extents): 24000 required.\']' abortedcode=550, code = 550’ ———

1 0

Re: Reattach gluster brick after rebuilt hypervisor
by Jeremy Tourville 20 Mar '26

20 Mar '26

I got a little further but just wanted to confirm a scenario. I realized part of my issue was due to having the volume mounted. I unmounted it and the message I get changed. cd / umount /gluster_bricks/vmstore now comment out /etc/fstab, systemctl daemon-reload and mount -a to confirm it is no longer mounted. [root@vmh /]# gluster volume create vmstore vmh.tourville.org:/gluster_bricks/vmstore volume create: vmstore: failed: The brick vmh.tourville.org:/gluster_bricks/vmstore is being created in the root partition. It is recommended that you don't use the system's root partition for storage backend. Or use 'force' at the end of the command if you want to override this behavior. ## DISK LAYOUT [root@vmh /]# vgs VG #PV #LV #SN Attr VSize VFree gluster_vg_sda 1 2 0 wz--n- <5.46t 0 rl_vmh 1 10 0 wz--n- <219.18g <43.84g [root@vmh /]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert gluster_lv_vmstore gluster_vg_sda Vwi-a-t--- 4.39t gluster_thinpool_gluster_vg_sda 48.48 gluster_thinpool_gluster_vg_sda gluster_vg_sda twi-aot--- <5.43t 39.26 0.51 home rl_vmh Vwi-aotz-- 70.00g pool00 0.09 pool00 rl_vmh twi-aotz-- <171.01g 4.36 11.17 root rl_vmh Vwi-aotz-- 20.00g pool00 22.29 swap rl_vmh -wi-ao---- 4.00g tmp rl_vmh Vwi-aotz-- 6.00g pool00 1.20 var rl_vmh Vwi-aotz-- 30.00g pool00 7.56 var_crash rl_vmh Vwi-aotz-- 15.00g pool00 0.42 var_log rl_vmh Vwi-aotz-- 16.00g pool00 0.84 var_log_audit rl_vmh Vwi-aotz-- 4.00g pool00 1.70 var_tmp rl_vmh Vwi-aotz-- 10.00g pool00 3.35 [root@vmh /]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 5.5T 0 disk └─3600605b00a2faca2233aa64a0966bb09 253:14 0 5.5T 0 mpath ├─gluster_vg_sda-gluster_thinpool_gluster_vg_sda_tmeta 253:16 0 15.8G 0 lvm │ └─gluster_vg_sda-gluster_thinpool_gluster_vg_sda-tpool 253:18 0 5.4T 0 lvm │ ├─gluster_vg_sda-gluster_thinpool_gluster_vg_sda 253:19 0 5.4T 1 lvm │ └─gluster_vg_sda-gluster_lv_vmstore 253:20 0 4.4T 0 lvm └─gluster_vg_sda-gluster_thinpool_gluster_vg_sda_tdata 253:17 0 5.4T 0 lvm └─gluster_vg_sda-gluster_thinpool_gluster_vg_sda-tpool 253:18 0 5.4T 0 lvm ├─gluster_vg_sda-gluster_thinpool_gluster_vg_sda 253:19 0 5.4T 1 lvm └─gluster_vg_sda-gluster_lv_vmstore 253:20 0 4.4T 0 lvm sdb 8:16 0 223.6G 0 disk └─sdb1 8:17 0 219.3G 0 part └─md127 9:127 0 219.2G 0 raid1 ├─rl_vmh-pool00_tmeta 253:0 0 172M 0 lvm │ └─rl_vmh-pool00-tpool 253:2 0 171G 0 lvm │ ├─rl_vmh-root 253:3 0 20G 0 lvm / │ ├─rl_vmh-pool00 253:5 0 171G 1 lvm │ ├─rl_vmh-home 253:6 0 70G 0 lvm /home │ ├─rl_vmh-var_crash 253:7 0 15G 0 lvm /var/crash │ ├─rl_vmh-tmp 253:8 0 6G 0 lvm /tmp │ ├─rl_vmh-var_tmp 253:9 0 10G 0 lvm /var/tmp │ ├─rl_vmh-var_log_audit 253:10 0 4G 0 lvm /var/log/audit │ ├─rl_vmh-var_log 253:11 0 16G 0 lvm /var/log │ └─rl_vmh-var 253:12 0 30G 0 lvm /var ├─rl_vmh-pool00_tdata 253:1 0 171G 0 lvm │ └─rl_vmh-pool00-tpool 253:2 0 171G 0 lvm │ ├─rl_vmh-root 253:3 0 20G 0 lvm / │ ├─rl_vmh-pool00 253:5 0 171G 1 lvm │ ├─rl_vmh-home 253:6 0 70G 0 lvm /home │ ├─rl_vmh-var_crash 253:7 0 15G 0 lvm /var/crash │ ├─rl_vmh-tmp 253:8 0 6G 0 lvm /tmp │ ├─rl_vmh-var_tmp 253:9 0 10G 0 lvm /var/tmp │ ├─rl_vmh-var_log_audit 253:10 0 4G 0 lvm /var/log/audit │ ├─rl_vmh-var_log 253:11 0 16G 0 lvm /var/log │ └─rl_vmh-var 253:12 0 30G 0 lvm /var └─rl_vmh-swap 253:4 0 4G 0 lvm [SWAP] sdc 8:32 0 223.6G 0 disk ├─sdc1 8:33 0 600M 0 part /boot/efi ├─sdc2 8:34 0 1M 0 part ├─sdc3 8:35 0 1G 0 part /boot └─sdc4 8:36 0 219.3G 0 part └─md127 9:127 0 219.2G 0 raid1 ├─rl_vmh-pool00_tmeta 253:0 0 172M 0 lvm │ └─rl_vmh-pool00-tpool 253:2 0 171G 0 lvm │ ├─rl_vmh-root 253:3 0 20G 0 lvm / │ ├─rl_vmh-pool00 253:5 0 171G 1 lvm │ ├─rl_vmh-home 253:6 0 70G 0 lvm /home │ ├─rl_vmh-var_crash 253:7 0 15G 0 lvm /var/crash │ ├─rl_vmh-tmp 253:8 0 6G 0 lvm /tmp │ ├─rl_vmh-var_tmp 253:9 0 10G 0 lvm /var/tmp │ ├─rl_vmh-var_log_audit 253:10 0 4G 0 lvm /var/log/audit │ ├─rl_vmh-var_log 253:11 0 16G 0 lvm /var/log │ └─rl_vmh-var 253:12 0 30G 0 lvm /var ├─rl_vmh-pool00_tdata 253:1 0 171G 0 lvm │ └─rl_vmh-pool00-tpool 253:2 0 171G 0 lvm │ ├─rl_vmh-root 253:3 0 20G 0 lvm / │ ├─rl_vmh-pool00 253:5 0 171G 1 lvm │ ├─rl_vmh-home 253:6 0 70G 0 lvm /home │ ├─rl_vmh-var_crash 253:7 0 15G 0 lvm /var/crash │ ├─rl_vmh-tmp 253:8 0 6G 0 lvm /tmp │ ├─rl_vmh-var_tmp 253:9 0 10G 0 lvm /var/tmp │ ├─rl_vmh-var_log_audit 253:10 0 4G 0 lvm /var/log/audit │ ├─rl_vmh-var_log 253:11 0 16G 0 lvm /var/log │ └─rl_vmh-var 253:12 0 30G 0 lvm /var └─rl_vmh-swap 253:4 0 4G 0 lvm [SWAP] sdd 8:48 0 596.2G 0 disk └─Hitachi_HTS547564A9E384_J21B0053G8E15R 253:13 0 596.2G 0 mpath └─Hitachi_HTS547564A9E384_J21B0053G8E15R1 253:15 0 596.2G 0 part So I think I can do a force because they are on different VGs. Can anyone confirm? Is there an unseen scenario (IE- bad practices) where forcing could cause issues?

1 0

Reattach gluster brick after rebuilt hypervisor
by Jeremy Tourville 20 Mar '26

20 Mar '26

I have a gluster volume that was created using an older version 4.x of oVirt. This was so long ago I am not sure I remember how I did it except that I used the wizard and the built in ansible job. I recently rebuilt my system that is hyperconverged. The storage raid array (hardware backed) is in my hypervisor. The engine is a stand alone external device. I am now at ver 4.5.7 of Ovirt. I want to add my gluster brick back into the setup. I have reviewed the wiki notes but they presume a new build, not my scenario. From what I read and researched, I think I need to create a distributed brick. How do you do that with just a single system when there is no replication to other hosts? Can anyone share a general procedure? ##DIAGNOSTICS [root@vmh /]# mount ... /dev/mapper/gluster_vg_sda-gluster_lv_vmstore on /gluster_bricks/vmstore type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=2048,noquota) [root@vmh /]# tail -n 1 /etc/fstab /dev/mapper/gluster_vg_sda-gluster_lv_vmstore /gluster_bricks/vmstore xfs defaults 0 0 [root@vmh /]# gluster volume create vmstore vmh.tourville.org:/gluster_bricks/vmstore volume create: vmstore: failed: The brick vmh.tourville.org:/gluster_bricks/vmstore is a mount point. Please create a sub-directory under the mount point and use that as the brick directory. Or use 'force' at the end of the command if you want to override this behavior. [root@vmh /]# gluster volume create vmstore vmh.tourville.org:/gluster_bricks/vmstore/vmstore volume create: vmstore: failed: /gluster_bricks/vmstore/vmstore is already part of a volume [root@vmh /]# tree /gluster_bricks/ /gluster_bricks/ └── vmstore └── vmstore └── 42c329c3-8eca-4748-af7a-f48aca494d72 ├── dom_md │ ├── ids │ ├── inbox │ ├── leases │ ├── metadata │ ├── outbox │ └── xleases ├── images │ ├── 019f828b-8f5d-47cf-bccd-7044402e7dd8 ... Do I need to force creation at this point? I just want to proceed cautiously at this point as I am not super confident with gluster. IE- [root@vmh /]# gluster volume create vmstore vmh.tourville.org:/gluster_bricks/vmstore force

1 0

hci setup with replica3 but one brick is not used
by Jiri Slezka 08 Mar '26

08 Mar '26

Hi, I am using distributed-replicated volume for vm's storage. There are 3 bricks (mounts on separate disks) in each of 3 hosts, so topology is 3x3 Volume Name: vms Type: Distributed-Replicate Volume ID: 52032ec6-99d4-4210-8fb8-ffbd7a1e0bf7 Status: Started Snapshot Count: 0 Number of Bricks: 3 x 3 = 9 Transport-type: tcp Bricks: Brick1: 10.0.4.11:/gluster_bricks/vms/vms Brick2: 10.0.4.13:/gluster_bricks/vms/vms Brick3: 10.0.4.12:/gluster_bricks/vms/vms Brick4: 10.0.4.11:/gluster_bricks/vms2/vms2 Brick5: 10.0.4.13:/gluster_bricks/vms2/vms2 Brick6: 10.0.4.12:/gluster_bricks/vms2/vms2 Brick7: 10.0.4.11:/gluster_bricks/vms3/vms3 Brick8: 10.0.4.12:/gluster_bricks/vms3/vms3 Brick9: 10.0.4.13:/gluster_bricks/vms3/vms3 I can see that bricks vms and vms2 are used but vms3 is almost (256MB vs hundreds of GBs) empty. In the fact I added vms3 bricks later and did not mention it because free volume space was reported (as I believed) correctly. Today I added one vm with 400GB disk and creating of disk failed because "OSError: [Errno 28] No space left on device". Now vms and vms2 bricks are full but vms volume has (should have) 914GiB of free space. I cannot see nothing interesting in glusterfs logs, glusterfs volume info and status looks good. There are errors in engine and vdsm logs but as can I see, it is just because of lack of space (vdsm side) and because image creation failed (engine side) Also as side effect, vms volume is full and disk from that new vm is locked (I would like to delete it). Any ideas what can be wrong or where to look and how to debug? Thanks in advance, Jiri

2 3

oVirt offline install
by Ethan Provensen 26 Feb '26

26 Feb '26

For the last 3-4 months, I have been trying to install oVirt on 3 bare metal servers with 1 SAN on an air-gapped network (no internet connection). I have failed with every attempt. I first determined what repositories I needed to upload to my air-gapped network to my Apache server, so the 3 bare metal servers could get the needed RPMs to install oVirt. I uploaded the following repositories from the internet: copr:copr.fedorainfracloud.org:ovirt:ovirt-master-snapshot ovirt-appliance-master-snapshot ovirt-master-centos-stream-ceph-pacific-testing ovirt-master-centos-stream-gluster10-testing ovirt-master-centos-stream-nfv-openvswitch2-testing ovirt-master-centos-stream-openstack-yoga-testing ovirt-master-centos-stream-opstools-collectd5-testing ovirt-master-centos-stream-ovirt45-testing ovirt-node-master-snapshot I first tried to install a oVirt self-hosted engine architecture on 1 server running Alma 9, but I ran into multiple dead ends with errors that I did not understand. My second attempt was to refresh the 1 Alma 9 server and install the oVirt CentOS9 node ISO instead on the 1 server. Again, I ran into multiple errors and did not understand the error messages. Recently, the oVirt Blog recommended we stopped using the oVirt CentOS9 node ISO for oVirt v4.5.7, so I went back on tried to install oVirt on Alma 9, and again I ran into multiple errors. I do not know what I am doing wrong. I have followed all the directions and Googled all the error messages I received, and still don't understand any of it. Now I am wondering if it's even possible to install oVirt on an server that doesn't have a DHCP connection to the internet. The last time I tried to create the "HostedEngineLocal" virtual machine on the server, I noticed the virtual machine creation relied on an internet connection and that is why the Hosted Engine setup failed everytime. Am I wrong? What am I doing wrong? Can someone help?

3 8

Deploy process frezzes with "Wait for the host to be up"
by Leandro Pereira 16 Feb '26

16 Feb '26

Hello everyone I'm installing ovirt 4.5 on a homelab VMWare Workstation (which I did several times weeks ago). Not so much that I'm going through technical problems. The deploy engine works in a stage called '[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Wait for the host to be up]'. I couldn't find any error messages in both the Host logs and the Engine. Try different ways to automatically create the ovirtmgmt network, as you previously created it. The fact is that at this stage the engine is still not connected to the ovirtmgmt network, and from this point the depoloy does not pass through the host cannot communicate with the Engine with the FQDN informed.

4 4

Unable to deploy hosted engine. It keeps stuck wait for the host to be up.
by dushyantk.sun＠gmail.com 15 Feb '26

15 Feb '26

I am trying to deploy hosted engine 4.5 on OL8.10. However it keeps getting stuck at step "wait for the host to be up." without any error. During this i found that vdsm service and related services went down. I am able to ssh into temp local hosted engine. But installation just stuck there. Nothing in ovirt-hosted-engine-setup logs.

1 0

Expanding blockSD volume
by Devin A. Bougie 12 Feb '26

12 Feb '26

What is the proper procedure for expanding the “blockSD” volume in our ovirt 4.5.6 cluster? We’ve started getting "No space left on device” errors when trying to clone a snapshot into a VM for backup purposes. Here’s the error: ——— VDSM lnxvirt06 command HSMGetAllTasksStatusesVDS failed: value=low level Image copy failed: ("Command ['/usr/bin/qemu-img', 'convert', '-p', '-t', 'none', '-T', 'none', '-f', 'qcow2', '-O', 'qcow2', '-o', 'compat=1.1', '/rhev/data-center/mnt/blockSD/86867486-e14e-4ff4-bf7c-1b0c94816032/images/211e91b0-2c72-45ad-90ca-badc63cb03f3/0a1b650f-8ab5-4eef-bee0-ea557c6eeb9f', '/rhev/data-center/mnt/blockSD/86867486-e14e-4ff4-bf7c-1b0c94816032/images/a2d4cef1-0342-41c0-85c2-e1b2ba6768b1/017e6d7e-4016-4108-8edc-188d8096a719'] failed with rc=1 out=b'' err=bytearray(b'qemu-img: error while writing at byte 3220937441280: No space left on device\\nqemu-img: Failed to flush the refcount block cache: No space left on device\\n')",) abortedcode=261 ——— And here’s what the host OS shows: ——— [root@lnxvirt06 ~]# df -h |grep rhev ovirtfs:/mnt/ovirtbackup 75T 36T 40T 47% /rhev/data-center/mnt/ovirtfs:_mnt_ovirtbackup /dev/mapper/86867486--e14e--4ff4--bf7c--1b0c94816032-master 975M 84K 924M 1% /rhev/data-center/mnt/blockSD/86867486-e14e-4ff4-bf7c-1b0c94816032/master ——— Please let me know if there’s any more information I can provide. Many thanks, Devin

1 0