11:37 a.m.
I am trying to migrate my VMs from my old host running ovirt 3.3.4 to a new
setup running 3.4.1. My basic set up is:
OLD
vmhost1 - ovirt 3.3.4 - NFS storages
NEW
ovirtmgr - ovirt 3.4.1 (virt only setup) - gluster storage domains
vmhost2 - cluster1
vmhost3 - cluster2
vmhost4 - cluster2
vmhost5 - cluster3
vmhost6 - cluster3
My gluster volumes are created via gluster command line and I have
All hosts are running scientific Linux 6.5, and the intention is to migrate
vmhost1 to new environment cluster1.
I have an NFS export storage domain which I am using to migrate VMs from
vmhost1.
Volume Name: vol-vminf
Type: Distributed-Replicate
Volume ID: b0b456bb-76e9-42e7-bb95-3415db79d631
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: vmhost3:/storage/inf/br-inf
Brick2: vmhost4:/storage/inf/br-inf
Brick3: vmhost5:/storage/inf/br-inf
Brick4: vmhost6:/storage/inf/br-inf
Options Reconfigured:
storage.owner-gid: 36
storage.owner-uid: 36
server.allow-insecure: on
Volume Name: vol-vmimages
Type: Distribute
Volume ID: 91e2cf8b-2662-4c26-b937-84b8f5b62e2b
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: vmhost3:/storage/vmimages/br-vmimages
Brick2: vmhost3:/storage/vmimages/br-vmimages
Brick3: vmhost3:/storage/vmimages/br-vmimages
Brick4: vmhost3:/storage/vmimages/br-vmimages
Options Reconfigured:
storage.owner-gid: 36
storage.owner-uid: 36
server.allow-insecure: on
I have had many varying fail results which I have tried to match up with
threads here, and I am now a bit stuck and would appreciate any help.
I quite regularly cannot import vms, and I now cannot create a new disk for
a new VM (no import). The error always seems to boild down to the following
error on engine.log (the following specifically for the create of a new
disk image on a gluster storage domain):
2014-05-20 08:51:21,136 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(ajp--127.0.0.1-8702-9) [4637af09] Correlation ID: 2b0b55ab, Job ID:
1a583643-e28a-4f09-a39d-46e4fc6d20b8, Call Stack: null, Custom Event ID:
-1, Message: Add-Disk operation of rhel-7_Disk1 was initiated on VM rhel-7
by peter.harris.
2014-05-20 08:51:21,137 INFO [org.ovirt.engine.core.bll.SPMAsyncTask]
(ajp--127.0.0.1-8702-9) [4637af09] BaseAsyncTask::startPollingTask:
Starting to poll task 720b4d92-1425-478c-8351-4ff827b8f728.
2014-05-20 08:51:28,077 INFO [org.ovirt.engine.core.bll.AsyncTaskManager]
(DefaultQuartzScheduler_Worker-19) Polling and updating Async Tasks: 1
tasks, 1 tasks to poll now
2014-05-20 08:51:28,084 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-19) Failed in HSMGetAllTasksStatusesVDS
method
2014-05-20 08:51:28,085 INFO [org.ovirt.engine.core.bll.SPMAsyncTask]
(DefaultQuartzScheduler_Worker-19) SPMAsyncTask::PollTask: Polling task
720b4d92-1425-478c-8351-4ff827b8f728 (Parent Command AddDisk, Parameters
Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned
status finished, result 'cleanSuccess'.
2014-05-20 08:51:28,104 ERROR [org.ovirt.engine.core.bll.SPMAsyncTask]
(DefaultQuartzScheduler_Worker-19) BaseAsyncTask::LogEndTaskFailure: Task
720b4d92-1425-478c-8351-4ff827b8f728 (Parent Command AddDisk, Parameters
Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended
with failure:^M
-- Result: cleanSuccess^M
-- Message: VDSGenericException: VDSErrorException: Failed to
HSMGetAllTasksStatusesVDS, error = [Errno 2] No such file or directory:
'/rhev/data-center/06930787-a091-49a3-8217-1418c5a9881e/967aec77-46d5-418b-8979-d0a86389a77b/images/7726b997-7e58-45f8-a5a6-9cb9a689a45a',
code = 100,^M
-- Exception: VDSGenericException: VDSErrorException: Failed to
HSMGetAllTasksStatusesVDS, error = [Errno 2] No such file or directory:
'/rhev/data-center/06930787-a091-49a3-8217-1418c5a9881e/967aec77-46d5-418b-8979-d0a86389a77b/images/7726b997-7e58-45f8-a5a6-9cb9a689a45a',
code = 100
Certainly, if I check
/rhev/data-center/06930787-a091-49a3-8217-1418c5a9881e/ on the SPM server,
there is no 967aec77-46d5-418b-8979-d0a86389a77b subdirectory. The only
elements I have are NFS mounts.
There appear to be no errors in the SPM vdsm.log for this disk
=============
When I tried to import the vm (the one that I then tried to create from
scratch above), I had the following errors in the SPM vdsm log:
Thread-2220::DEBUG::2014-05-20
08:35:05,255::task::595::TaskManager.Task::(_updateState)
Task=`15bc07b5-201f-4bba-bf5f-f79eb92c6a61`::moving from state init ->
state preparing
Thread-2220::INFO::2014-05-20
08:35:05,255::logUtils::44::dispatcher::(wrapper) Run and protect:
deleteImage(sdUUID='615647e2-1f60-47e1-8e55-be9f7ead6f15',
spUUID='06930787-a091-49a3-8217-1418c5a9881e',
imgUUID='80ed133c-fd72-4d35-aae5-e1313be3cf23', postZero='false',
force='false')
Thread-2220::DEBUG::2014-05-20
08:35:05,255::resourceManager::198::ResourceManager.Request::(__init__)
ResName=`Storage.80ed133c-fd72-4d35-aae5-e1313be3cf23`ReqID=`499de454-c563-4156-a3ed-13b7eb9defa6`::Request
was made in '/usr/share/vdsm/storage/hsm.py' line '1496' at
'deleteImage'
Thread-2220::DEBUG::2014-05-20
08:35:05,255::resourceManager::542::ResourceManager::(registerResource)
Trying to register resource 'Storage.80ed133c-fd72-4d35-aae5-e1313be3cf23'
for lock type 'exclusive'
Thread-2220::DEBUG::2014-05-20
08:35:05,255::resourceManager::601::ResourceManager::(registerResource)
Resource 'Storage.80ed133c-fd72-4d35-aae5-e1313be3cf23' is free. Now
locking as 'exclusive' (1 active user)
Thread-2220::DEBUG::2014-05-20
08:35:05,256::resourceManager::238::ResourceManager.Request::(grant)
ResName=`Storage.80ed133c-fd72-4d35-aae5-e1313be3cf23`ReqID=`499de454-c563-4156-a3ed-13b7eb9defa6`::Granted
request
Thread-2220::DEBUG::2014-05-20
08:35:05,256::task::827::TaskManager.Task::(resourceAcquired)
Task=`15bc07b5-201f-4bba-bf5f-f79eb92c6a61`::_resourcesAcquired:
Storage.80ed133c-fd72-4d35-aae5-e1313be3cf23 (exclusive)
Thread-2220::DEBUG::2014-05-20
08:35:05,256::task::990::TaskManager.Task::(_decref)
Task=`15bc07b5-201f-4bba-bf5f-f79eb92c6a61`::ref 1 aborting False
Thread-2220::DEBUG::2014-05-20
08:35:05,256::resourceManager::198::ResourceManager.Request::(__init__)
ResName=`Storage.615647e2-1f60-47e1-8e55-be9f7ead6f15`ReqID=`73f79517-f13a-4e5b-999a-6f1994d2818a`::Request
was made in '/usr/share/vdsm/storage/hsm.py' line '1497' at
'deleteImage'
Thread-2220::DEBUG::2014-05-20
08:35:05,256::resourceManager::542::ResourceManager::(registerResource)
Trying to register resource 'Storage.615647e2-1f60-47e1-8e55-be9f7ead6f15'
for lock type 'shared'
Thread-2220::DEBUG::2014-05-20
08:35:05,257::resourceManager::601::ResourceManager::(registerResource)
Resource 'Storage.615647e2-1f60-47e1-8e55-be9f7ead6f15' is free. Now
locking as 'shared' (1 active user)
Thread-2220::DEBUG::2014-05-20
08:35:05,257::resourceManager::238::ResourceManager.Request::(grant)
ResName=`Storage.615647e2-1f60-47e1-8e55-be9f7ead6f15`ReqID=`73f79517-f13a-4e5b-999a-6f1994d2818a`::Granted
request
Thread-2220::DEBUG::2014-05-20
08:35:05,257::task::827::TaskManager.Task::(resourceAcquired)
Task=`15bc07b5-201f-4bba-bf5f-f79eb92c6a61`::_resourcesAcquired:
Storage.615647e2-1f60-47e1-8e55-be9f7ead6f15 (shared)
Thread-2220::DEBUG::2014-05-20
08:35:05,257::task::990::TaskManager.Task::(_decref)
Task=`15bc07b5-201f-4bba-bf5f-f79eb92c6a61`::ref 1 aborting False
Thread-2220::ERROR::2014-05-20
08:35:05,266::hsm::1502::Storage.HSM::(deleteImage) Empty or not found
image 80ed133c-fd72-4d35-aae5-e1313be3cf23 in SD
615647e2-1f60-47e1-8e55-be9f7ead6f15.
{'1f41529a-e02e-4cd8-987c-b1ea4fcba2be':
ImgsPar(imgs=('290f5cdf-b5d7-462b-958d-d41458a26bf6',), parent=None),
'1748a8f0-8668-4f21-9b26-d2e3b180e35b':
ImgsPar(imgs=('67fd552b-8b3d-4117-82d2-e801bb600992',), parent=None)}
Thread-2220::ERROR::2014-05-20
08:35:05,266::task::866::TaskManager.Task::(_setError)
Task=`15bc07b5-201f-4bba-bf5f-f79eb92c6a61`::Unexpected error
When I installed/setup ovirt-engine, I did chose NFS as the file system.
I am clearly doing something weird