Hello all, since we updated our Commvault to the version "service pack 26 hotfix 16", random VM backups are failing to attach disk snapshots to the backup proxy VM. We are running oVirt Version 4.4.10.6-1.el8 and the hypervisor is oVirt Node 4.4.9. These are the logs from the managerVM: ############## 2022-03-24 11:13:59,390+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (default task-1578) [0aeb5794-5c2e-44c6-8fc5-34ec0e81d16b] Disk hot-plug: <?xml version="1.0" encoding="UTF-8"?><hotplug> <devices> <disk snapshot="no" type="file" device="disk"> <target dev="sda" bus="scsi"/> <source file="/rhev/data-center/mnt/blockSD/6da85fb1-ff4b-4665-ade3-0f32f83250bc/images/e3c292e0-dfd2-4cd5-a27f-f87308a18e64/8b524896-902c-445f-a33a-65335cb75eff"> <seclabel model="dac" type="none" relabel="no"/> </source> <driver name="qemu" io="threads" type="qcow2" error_policy="stop" cache="writethrough"/> <alias name="ua-e3c292e0-dfd2-4cd5-a27f-f87308a18e64"/> <address bus="0" controller="0" unit="0" type="drive" target="0"/> <serial>e3c292e0-dfd2-4cd5-a27f-f87308a18e64</serial> </disk> </devices> <metadata xmlns:ovirt-vm="http://ovirt.org/vm/1.0"> <ovirt-vm:vm> <ovirt-vm:device devtype="disk" name="sda"> <ovirt-vm:poolID>98d981da-d010-11ea-9e4d-00163e1be424</ovirt-vm:poolID> <ovirt-vm:volumeID>8b524896-902c-445f-a33a-65335cb75eff</ovirt-vm:volumeID> <ovirt-vm:shared>transient</ovirt-vm:shared> <ovirt-vm:imageID>e3c292e0-dfd2-4cd5-a27f-f87308a18e64</ovirt-vm:imageID> <ovirt-vm:domainID>6da85fb1-ff4b-4665-ade3-0f32f83250bc</ovirt-vm:domainID> </ovirt-vm:device> </ovirt-vm:vm> </metadata> </hotplug> 2022-03-24 11:13:59,898+01 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (default task-1578) [0aeb5794-5c2e-44c6-8fc5-34ec0e81d16b] Failed in 'HotPlugDiskVDS' method 2022-03-24 11:13:59,898+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (default task-1578) [0aeb5794-5c2e-44c6-8fc5-34ec0e81d16b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand' return value 'StatusOnlyReturn [status=Status [code=45, message=Requested operation is not valid: Domain already contains a disk with that address]]' 2022-03-24 11:13:59,898+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (default task-1578) [0aeb5794-5c2e-44c6-8fc5-34ec0e81d16b] HostName = hypervisor1.domain 2022-03-24 11:13:59,898+01 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (default task-1578) [0aeb5794-5c2e-44c6-8fc5-34ec0e81d16b] Command 'HotPlugDiskVDSCommand(HostName = hypervisor1.domain, HotPlugDiskVDSParameters:{hostId='c2fd81cd-8fc8-410e-8333-a0a36b87ab2b', vmId='890c105d-333b-435e-8d8b-0b889f4f9c14', diskId='e3c292e0-dfd2-4cd5-a27f-f87308a18e64'})' execution failed: VDSGenericException: VDSErrorException: Failed to HotPlugDiskVDS, error = Requested operation is not valid: Domain already contains a disk with that address, code = 45 2022-03-24 11:13:59,898+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (default task-1578) [0aeb5794-5c2e-44c6-8fc5-34ec0e81d16b] FINISH, HotPlugDiskVDSCommand, return: , log id: 536a4414 2022-03-24 11:13:59,898+01 ERROR [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default task-1578) [0aeb5794-5c2e-44c6-8fc5-34ec0e81d16b] Command 'org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to HotPlugDiskVDS, error = Requested operation is not valid: Domain already contains a disk with that address, code = 45 (Failed with error FailedToPlugDisk and code 45) 2022-03-24 11:13:59,899+01 INFO [org.ovirt.engine.core.bll.CommandCompensator] (default task-1578) [0aeb5794-5c2e-44c6-8fc5-34ec0e81d16b] Command [id=87712152-4660-40ca-b292-397796c735d5]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.storage.DiskVmElement; snapshot: VmDeviceId:{deviceId='e3c292e0-dfd2-4cd5-a27f-f87308a18e64', vmId='890c105d-333b-435e-8d8b-0b889f4f9c14'}. 2022-03-24 11:13:59,900+01 INFO [org.ovirt.engine.core.bll.CommandCompensator] (default task-1578) [0aeb5794-5c2e-44c6-8fc5-34ec0e81d16b] Command [id=87712152-4660-40ca-b292-397796c735d5]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.VmDevice; snapshot: VmDeviceId:{deviceId='e3c292e0-dfd2-4cd5-a27f-f87308a18e64', vmId='890c105d-333b-435e-8d8b-0b889f4f9c14'}. 2022-03-24 11:13:59,910+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1578) [0aeb5794-5c2e-44c6-8fc5-34ec0e81d16b] EVENT_ID: USER_FAILED_ATTACH_DISK_TO_VM(2,017), Failed to attach Disk VM1_Disk1 to VM backupproxyVM (User: backupagent@internal-authz). 2022-03-24 11:13:59,910+01 INFO [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default task-1578) [0aeb5794-5c2e-44c6-8fc5-34ec0e81d16b] Lock freed to object 'EngineLock:{exclusiveLocks='[e3c292e0-dfd2-4cd5-a27f-f87308a18e64=DISK]', sharedLocks=''}' 2022-03-24 11:13:59,910+01 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-1578) [] Operation Failed: [Failed to hot-plug disk] ################### Here are the logs from the hypervisor: ################## 2022-03-24 11:13:59,712+0100 INFO (jsonrpc/4) [virt.vm] (vmId='890c105d-333b-435e-8d8b-0b889f4f9c14') Hotplug disk xml: <?xml version='1.0' encoding='utf-8'?> <disk device="disk" snapshot="no" type="file"> <address bus="0" controller="0" target="0" type="drive" unit="0" /> <source file="/var/lib/vdsm/transient/6da85fb1-ff4b-4665-ade3-0f32f83250bc-8b524896-902c-445f-a33a-65335cb75eff.pxhtm0_4"> <seclabel model="dac" relabel="no" type="none" /> </source> <target bus="scsi" dev="sdbzk" /> <serial>e3c292e0-dfd2-4cd5-a27f-f87308a18e64</serial> <driver cache="writethrough" error_policy="stop" io="threads" name="qemu" type="qcow2" /> <alias name="ua-e3c292e0-dfd2-4cd5-a27f-f87308a18e64" /> </disk> (vm:3851) 2022-03-24 11:13:59,721+0100 ERROR (jsonrpc/4) [virt.vm] (vmId='890c105d-333b-435e-8d8b-0b889f4f9c14') Hotplug failed (vm:3859) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 3857, in hotplugDisk self._dom.attachDevice(driveXml) File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 101, in f ret = attr(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python3.6/site-packages/libvirt.py", line 682, in attachDevice raise libvirtError('virDomainAttachDevice() failed') libvirt.libvirtError: Requested operation is not valid: Domain already contains a disk with that address ##################### Any idea what could cause these errors? Best regards, Martin