Re: [ovirt-users] Migrate between 3.3.4 and 3.4.1

I am trying to migrate my VMs from my old host running ovirt 3.3.4 to a new setup running 3.4.1. My basic set up is: OLD vmhost1 - ovirt 3.3.4 - NFS storages NEW ovirtmgr - ovirt 3.4.1 (virt only setup) - gluster storage domains vmhost2 - cluster1 vmhost3 - cluster2 vmhost4 - cluster2 vmhost5 - cluster3 vmhost6 - cluster3 My gluster volumes are created via gluster command line and I have All hosts are running scientific Linux 6.5, and the intention is to migrate vmhost1 to new environment cluster1. I have an NFS export storage domain which I am using to migrate VMs from vmhost1. Volume Name: vol-vminf Type: Distributed-Replicate Volume ID: b0b456bb-76e9-42e7-bb95-3415db79d631 Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: vmhost3:/storage/inf/br-inf Brick2: vmhost4:/storage/inf/br-inf Brick3: vmhost5:/storage/inf/br-inf Brick4: vmhost6:/storage/inf/br-inf Options Reconfigured: storage.owner-gid: 36 storage.owner-uid: 36 server.allow-insecure: on Volume Name: vol-vmimages Type: Distribute Volume ID: 91e2cf8b-2662-4c26-b937-84b8f5b62e2b Status: Started Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: vmhost3:/storage/vmimages/br-vmimages Brick2: vmhost3:/storage/vmimages/br-vmimages Brick3: vmhost3:/storage/vmimages/br-vmimages Brick4: vmhost3:/storage/vmimages/br-vmimages Options Reconfigured: storage.owner-gid: 36 storage.owner-uid: 36 server.allow-insecure: on I have had many varying fail results which I have tried to match up with threads here, and I am now a bit stuck and would appreciate any help. I quite regularly cannot import vms, and I now cannot create a new disk for a new VM (no import). The error always seems to boild down to the following error on engine.log (the following specifically for the create of a new disk image on a gluster storage domain): 2014-05-20 08:51:21,136 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-9) [4637af09] Correlation ID: 2b0b55ab, Job ID: 1a583643-e28a-4f09-a39d-46e4fc6d20b8, Call Stack: null, Custom Event ID: -1, Message: Add-Disk operation of rhel-7_Disk1 was initiated on VM rhel-7 by peter.harris. 2014-05-20 08:51:21,137 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp--127.0.0.1-8702-9) [4637af09] BaseAsyncTask::startPollingTask: Starting to poll task 720b4d92-1425-478c-8351-4ff827b8f728. 2014-05-20 08:51:28,077 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-19) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now 2014-05-20 08:51:28,084 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-19) Failed in HSMGetAllTasksStatusesVDS method 2014-05-20 08:51:28,085 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-19) SPMAsyncTask::PollTask: Polling task 720b4d92-1425-478c-8351-4ff827b8f728 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'cleanSuccess'. 2014-05-20 08:51:28,104 ERROR [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-19) BaseAsyncTask::LogEndTaskFailure: Task 720b4d92-1425-478c-8351-4ff827b8f728 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended with failure:^M -- Result: cleanSuccess^M -- Message: VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = [Errno 2] No such file or directory: '/rhev/data-center/06930787-a091-49a3-8217-1418c5a9881e/967aec77-46d5-418b-8979-d0a86389a77b/images/7726b997-7e58-45f8-a5a6-9cb9a689a45a', code = 100,^M -- Exception: VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = [Errno 2] No such file or directory: '/rhev/data-center/06930787-a091-49a3-8217-1418c5a9881e/967aec77-46d5-418b-8979-d0a86389a77b/images/7726b997-7e58-45f8-a5a6-9cb9a689a45a', code = 100 Certainly, if I check /rhev/data-center/06930787-a091-49a3-8217-1418c5a9881e/ on the SPM server, there is no 967aec77-46d5-418b-8979-d0a86389a77b subdirectory. The only elements I have are NFS mounts. There appear to be no errors in the SPM vdsm.log for this disk ============= When I tried to import the vm (the one that I then tried to create from scratch above), I had the following errors in the SPM vdsm log: Thread-2220::DEBUG::2014-05-20 08:35:05,255::task::595::TaskManager.Task::(_updateState) Task=`15bc07b5-201f-4bba-bf5f-f79eb92c6a61`::moving from state init -> state preparing Thread-2220::INFO::2014-05-20 08:35:05,255::logUtils::44::dispatcher::(wrapper) Run and protect: deleteImage(sdUUID='615647e2-1f60-47e1-8e55-be9f7ead6f15', spUUID='06930787-a091-49a3-8217-1418c5a9881e', imgUUID='80ed133c-fd72-4d35-aae5-e1313be3cf23', postZero='false', force='false') Thread-2220::DEBUG::2014-05-20 08:35:05,255::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.80ed133c-fd72-4d35-aae5-e1313be3cf23`ReqID=`499de454-c563-4156-a3ed-13b7eb9defa6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1496' at 'deleteImage' Thread-2220::DEBUG::2014-05-20 08:35:05,255::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.80ed133c-fd72-4d35-aae5-e1313be3cf23' for lock type 'exclusive' Thread-2220::DEBUG::2014-05-20 08:35:05,255::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.80ed133c-fd72-4d35-aae5-e1313be3cf23' is free. Now locking as 'exclusive' (1 active user) Thread-2220::DEBUG::2014-05-20 08:35:05,256::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.80ed133c-fd72-4d35-aae5-e1313be3cf23`ReqID=`499de454-c563-4156-a3ed-13b7eb9defa6`::Granted request Thread-2220::DEBUG::2014-05-20 08:35:05,256::task::827::TaskManager.Task::(resourceAcquired) Task=`15bc07b5-201f-4bba-bf5f-f79eb92c6a61`::_resourcesAcquired: Storage.80ed133c-fd72-4d35-aae5-e1313be3cf23 (exclusive) Thread-2220::DEBUG::2014-05-20 08:35:05,256::task::990::TaskManager.Task::(_decref) Task=`15bc07b5-201f-4bba-bf5f-f79eb92c6a61`::ref 1 aborting False Thread-2220::DEBUG::2014-05-20 08:35:05,256::resourceManager::198::ResourceManager.Request::(__init__) ResName=`Storage.615647e2-1f60-47e1-8e55-be9f7ead6f15`ReqID=`73f79517-f13a-4e5b-999a-6f1994d2818a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1497' at 'deleteImage' Thread-2220::DEBUG::2014-05-20 08:35:05,256::resourceManager::542::ResourceManager::(registerResource) Trying to register resource 'Storage.615647e2-1f60-47e1-8e55-be9f7ead6f15' for lock type 'shared' Thread-2220::DEBUG::2014-05-20 08:35:05,257::resourceManager::601::ResourceManager::(registerResource) Resource 'Storage.615647e2-1f60-47e1-8e55-be9f7ead6f15' is free. Now locking as 'shared' (1 active user) Thread-2220::DEBUG::2014-05-20 08:35:05,257::resourceManager::238::ResourceManager.Request::(grant) ResName=`Storage.615647e2-1f60-47e1-8e55-be9f7ead6f15`ReqID=`73f79517-f13a-4e5b-999a-6f1994d2818a`::Granted request Thread-2220::DEBUG::2014-05-20 08:35:05,257::task::827::TaskManager.Task::(resourceAcquired) Task=`15bc07b5-201f-4bba-bf5f-f79eb92c6a61`::_resourcesAcquired: Storage.615647e2-1f60-47e1-8e55-be9f7ead6f15 (shared) Thread-2220::DEBUG::2014-05-20 08:35:05,257::task::990::TaskManager.Task::(_decref) Task=`15bc07b5-201f-4bba-bf5f-f79eb92c6a61`::ref 1 aborting False Thread-2220::ERROR::2014-05-20 08:35:05,266::hsm::1502::Storage.HSM::(deleteImage) Empty or not found image 80ed133c-fd72-4d35-aae5-e1313be3cf23 in SD 615647e2-1f60-47e1-8e55-be9f7ead6f15. {'1f41529a-e02e-4cd8-987c-b1ea4fcba2be': ImgsPar(imgs=('290f5cdf-b5d7-462b-958d-d41458a26bf6',), parent=None), '1748a8f0-8668-4f21-9b26-d2e3b180e35b': ImgsPar(imgs=('67fd552b-8b3d-4117-82d2-e801bb600992',), parent=None)} Thread-2220::ERROR::2014-05-20 08:35:05,266::task::866::TaskManager.Task::(_setError) Task=`15bc07b5-201f-4bba-bf5f-f79eb92c6a61`::Unexpected error When I installed/setup ovirt-engine, I did chose NFS as the file system. I am clearly doing something weird