[ovirt-users] Re: Libvirt ERROR cannot access backing file after importing VM from OpenStack

Dear oVirt team, When trying to start imported VM, it fails with following message: ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-2) [] EVENT_ID: VM_DOWN_ERROR(119), VM instance-00000673 is down with error. Exit message: Cannot access backing file '/var/lib/nova/instances/_base/2f4f8c5fc11bb83bcab03f4c829ddda4da8c0bce' of storage file '/rhev/data-center/mnt/glusterSD/aws-gfs-01.awesome.lan:_gv0__he/2607c265-248c-40ad-b020-f3756454839e/images/816ac00f-ba98-4827-b5c8-42a8ba496089/8ecfcd5b-db67-4c23-9869-0e20d7553aba' (as uid:107, gid:107): No such file or directory. Platform details: Ovirt SHE Version 4.2.2.6-1.el7.centos GlusterFS, unmanaged by oVirt. VM is imported & converted from OpenStack, according to log files, successfully (one WARN, related to different MAC address): 2018-05-24 12:03:31,028+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmsNamesFromExternalProviderVDSCommand] (default task-29) [cc5931a2-1af5-4d65-b0b3-362588db9d3f] FINISH, GetVmsNamesFromExternalProviderVDSCommand, return: [VM [instance-0001f94c], VM [instance-00078f6a], VM [instance-00000814], VM [instance-0001f9ac], VM [instance-000001ff], VM [instance-0001f718], VM [instance-00000673], VM [instance-0001ecf2], VM [instance-00078d38]], log id: 7f178a5e 2018-05-24 12:48:33,722+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmsNamesFromExternalProviderVDSCommand] (default task-8) [103d56e1-7449-4853-ae50-48ee94d43d77] FINISH, GetVmsNamesFromExternalProviderVDSCommand, return: [VM [instance-0001f94c], VM [instance-00078f6a], VM [instance-00000814], VM [instance-0001f9ac], VM [instance-000001ff], VM [instance-0001f718], VM [instance-00000673], VM [instance-0001ecf2], VM [instance-00078d38]], log id: 3aa178c5 2018-05-24 12:48:47,291+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmsFullInfoFromExternalProviderVDSCommand] (default task-17) [4bf555c7-9d64-4ecc-b059-8a60a4b27bdd] START, GetVmsFullInfoFromExternalProviderVDSCommand(HostName = aws-ovhv-01, GetVmsFromExternalProviderParameters:{hostId='cbabe1e8-9e7f-4c4b-be9c-49154953564d', url='qemu+tcp://root@172.19.0.12/system', username='null', originType='KVM', namesOfVms='[instance-00000673]'}), log id: 4c445109 2018-05-24 12:48:47,318+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmsFullInfoFromExternalProviderVDSCommand] (default task-17) [4bf555c7-9d64-4ecc-b059-8a60a4b27bdd] FINISH, GetVmsFullInfoFromExternalProviderVDSCommand, return: [VM [instance-00000673]], log id: 4c445109 2018-05-24 12:49:20,466+02 INFO [org.ovirt.engine.core.bll.exportimport.ImportVmFromExternalProviderCommand] (default task-41) [14edb003-b4a0-4355-b3de-da2b68774fe3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[instance-00000673=VM_NAME, 1f0b608f-7cfc-4b27-a876-b5d8073011a1=VM]', sharedLocks=''}' 2018-05-24 12:49:20,586+02 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-653408) [14edb003-b4a0-4355-b3de-da2b68774fe3] EVENT_ID: MAC_ADDRESS_IS_EXTERNAL(925), VM instance-00000673 has MAC address(es) fa:16:3e:74:18:50, which is/are out of its MAC pool definitions. 2018-05-24 12:49:21,021+02 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-653408) [14edb003-b4a0-4355-b3de-da2b68774fe3] EVENT_ID: IMPORTEXPORT_STARTING_IMPORT_VM(1,165), Starting to import Vm instance-00000673 to Data Center AVEUNL, Cluster AWSEUOPS 2018-05-24 12:49:28,816+02 INFO [org.ovirt.engine.core.bll.exportimport.ImportVmFromExternalProviderCommand] (EE-ManagedThreadFactory-engine-Thread-653407) [] Lock freed to object 'EngineLock:{exclusiveLocks='[instance-00000673=VM_NAME, 1f0b608f-7cfc-4b27-a876-b5d8073011a1=VM]', sharedLocks=''}' 2018-05-24 12:49:28,911+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConvertVmVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [2047673e] START, ConvertVmVDSCommand(HostName = aws-ovhv-01, ConvertVmVDSParameters:{hostId='cbabe1e8-9e7f-4c4b-be9c-49154953564d', url='qemu+tcp://root@172.19.0.12/system', username='null', vmId='1f0b608f-7cfc-4b27-a876-b5d8073011a1', vmName='instance-00000673', storageDomainId='2607c265-248c-40ad-b020-f3756454839e', storagePoolId='5a5de92c-0120-0167-03cb-00000000038a', virtioIsoPath='null', compatVersion='null', Disk0='816ac00f-ba98-4827-b5c8-42a8ba496089'}), log id: 53408517 2018-05-24 12:49:29,010+02 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [2047673e] EVENT_ID: IMPORTEXPORT_STARTING_CONVERT_VM(1,193), Starting to convert Vm instance-00000673 2018-05-24 12:52:57,982+02 INFO [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-16) [df1d5f72-eb17-46e4-9946-20ca9809b54c] Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[instance-00000673=VM_NAME]', sharedLocks='[1f0b608f-7cfc-4b27-a876-b5d8073011a1=VM]'}' 2018-05-24 12:59:24,575+02 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [2047673e] EVENT_ID: IMPORTEXPORT_IMPORT_VM(1,152), Vm instance-00000673 was imported successfully to Data Center AVEUNL, Cluster AWSEUOPS Than trying to start VM fails with following messages: 2018-05-24 13:00:32,085+02 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-653729) [] EVENT_ID: USER_STARTED_VM(153), VM instance-00000673 was started by admin@internal-authz (Host: aws-ovhv-06). 2018-05-24 13:00:33,417+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-9) [] VM '1f0b608f-7cfc-4b27-a876-b5d8073011a1'(instance-00000673) moved from 'WaitForLaunch' --> 'Down' 2018-05-24 13:00:33,436+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-9) [] EVENT_ID: VM_DOWN_ERROR(119), VM instance-00000673 is down with error. Exit message: Cannot access backing file '/var/lib/nova/instances/_base/2f4f8c5fc11bb83bcab03f4c829ddda4da8c0bce' of storage file '/rhev/data-center/mnt/glusterSD/aws-gfs-01.awesome.lan:_gv0__he/2607c265-248c-40ad-b020-f3756454839e/images/816ac00f-ba98-4827-b5c8-42a8ba496089/8ecfcd5b-db67-4c23-9869-0e20d7553aba' (as uid:107, gid:107): No such file or directory. 2018-05-24 13:00:33,437+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-9) [] add VM '1f0b608f-7cfc-4b27-a876-b5d8073011a1'(instance-00000673) to rerun treatment 2018-05-24 13:00:33,455+02 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-653732) [] EVENT_ID: USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM instance-00000673 on Host aws-ovhv-06. 2018-05-24 13:00:33,460+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-653732) [] EVENT_ID: USER_FAILED_RUN_VM(54), Failed to run VM instance-00000673 (User: admin@internal-authz). Checking on the Gluster volume, directory and files exist, permissions are in order: [root@aws-ovhv-01 816ac00f-ba98-4827-b5c8-42a8ba496089] -rw-rw----. 1 vdsm kvm 14G May 24 12:59 8ecfcd5b-db67-4c23-9869-0e20d7553aba -rw-rw----. 1 vdsm kvm 1.0M May 24 12:49 8ecfcd5b-db67-4c23-9869-0e20d7553aba.lease -rw-r--r--. 1 vdsm kvm 310 May 24 12:49 8ecfcd5b-db67-4c23-9869-0e20d7553aba.meta Than I have checked image info, and noticed that backing file entry is pointing to non-existing location, which does and should not exist on oVirt hosts: [root@aws-ovhv-01 816ac00f-ba98-4827-b5c8-42a8ba496089]# qemu-img info 8ecfcd5b-db67-4c23-9869-0e20d7553aba image: 8ecfcd5b-db67-4c23-9869-0e20d7553aba file format: qcow2 virtual size: 160G (171798691840 bytes) disk size: 14G cluster_size: 65536 *backing file: /var/lib/nova/instances/_base/2f4f8c5fc11bb83bcab03f4c829ddda4da8c0bce* Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false Can somebody advise me how to fix, address this, as I am in need of importing 200+ VMs from OpenStack to oVirt?