[ovirt-users] Switch from Fedora 20 to CentOS 7.1
Soeren Malchow
soeren.malchow at mcon.net
Thu May 21 16:35:02 UTC 2015
Hi,
We now created the new Cluster based on CentOS 7.1 which went fine, then we migrated 2 machines – no problem, we have Live Migration (back), Live Merge and so on, all good.
But some additional machine have problems starting on the new cluster and this happens
Grep for the Thread in vdsm.log
<— snip —>
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:21,999::vm::2264::vm.Vm::(_startUnderlyingVm) vmId=`24bd5074-64fc-4aa0-87cd-5de3dd7b50d1`::Start
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,003::vm::2268::vm.Vm::(_startUnderlyingVm) vmId=`24bd5074-64fc-4aa0-87cd-5de3dd7b50d1`::_ongoingCreations acquired
vdsm/vdsm.log:Thread-5475::INFO::2015-05-21 18:27:22,008::vm::3261::vm.Vm::(_run) vmId=`24bd5074-64fc-4aa0-87cd-5de3dd7b50d1`::VM wrapper has started
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,021::task::595::Storage.TaskManager.Task::(_updateState) Task=`2bc7fe9c-204a-4ab7-a116-f7fbba32bd34`::moving from state init -> state preparing
vdsm/vdsm.log:Thread-5475::INFO::2015-05-21 18:27:22,028::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID=u'276e9ba7-e19a-49c5-8ad7-26711934d5e4', spUUID=u'0f954891-b1cd-4f09-99ae-75d404d95f9d', imgUUID=u'eae65249-e5e8-49e7-90a0-c7385e80e6ca', volUUID=u'8791f6ec-a6ef-484d-bd5a-730b22b19250', options=None)
vdsm/vdsm.log:Thread-5475::INFO::2015-05-21 18:27:22,069::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2696552448', 'apparentsize': '2696609792'}
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,069::task::1191::Storage.TaskManager.Task::(prepare) Task=`2bc7fe9c-204a-4ab7-a116-f7fbba32bd34`::finished: {'truesize': '2696552448', 'apparentsize': '2696609792'}
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,069::task::595::Storage.TaskManager.Task::(_updateState) Task=`2bc7fe9c-204a-4ab7-a116-f7fbba32bd34`::moving from state preparing -> state finished
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,070::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,070::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,070::task::993::Storage.TaskManager.Task::(_decref) Task=`2bc7fe9c-204a-4ab7-a116-f7fbba32bd34`::ref 0 aborting False
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,071::task::595::Storage.TaskManager.Task::(_updateState) Task=`c508cf8f-9f02-43a6-a45d-2b3f1d7e66be`::moving from state init -> state preparing
vdsm/vdsm.log:Thread-5475::INFO::2015-05-21 18:27:22,071::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID=u'276e9ba7-e19a-49c5-8ad7-26711934d5e4', spUUID=u'0f954891-b1cd-4f09-99ae-75d404d95f9d', imgUUID=u'967d966c-3653-4ff6-9299-2fb5b4197c37', volUUID=u'99b085e6-6662-43ef-8ab4-40bc00e82460', options=None)
vdsm/vdsm.log:Thread-5475::INFO::2015-05-21 18:27:22,086::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '1110773760', 'apparentsize': '1110835200'}
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,087::task::1191::Storage.TaskManager.Task::(prepare) Task=`c508cf8f-9f02-43a6-a45d-2b3f1d7e66be`::finished: {'truesize': '1110773760', 'apparentsize': '1110835200'}
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,087::task::595::Storage.TaskManager.Task::(_updateState) Task=`c508cf8f-9f02-43a6-a45d-2b3f1d7e66be`::moving from state preparing -> state finished
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,087::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,088::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,088::task::993::Storage.TaskManager.Task::(_decref) Task=`c508cf8f-9f02-43a6-a45d-2b3f1d7e66be`::ref 0 aborting False
vdsm/vdsm.log:Thread-5475::INFO::2015-05-21 18:27:22,088::clientIF::335::vds::(prepareVolumePath) prepared volume path:
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,089::task::595::Storage.TaskManager.Task::(_updateState) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::moving from state init -> state preparing
vdsm/vdsm.log:Thread-5475::INFO::2015-05-21 18:27:22,089::logUtils::44::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID=u'276e9ba7-e19a-49c5-8ad7-26711934d5e4', spUUID=u'0f954891-b1cd-4f09-99ae-75d404d95f9d', imgUUID=u'eae65249-e5e8-49e7-90a0-c7385e80e6ca', leafUUID=u'8791f6ec-a6ef-484d-bd5a-730b22b19250')
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,090::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4`ReqID=`2ba5bd10-3b98-44fa-9c90-8a2ade3261dc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3226' at 'prepareImage'
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,090::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4' for lock type 'shared'
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,091::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4' is free. Now locking as 'shared' (1 active user)
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,091::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4`ReqID=`2ba5bd10-3b98-44fa-9c90-8a2ade3261dc`::Granted request
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,092::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::_resourcesAcquired: Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4 (shared)
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,092::task::993::Storage.TaskManager.Task::(_decref) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::ref 1 aborting False
vdsm/vdsm.log:Thread-5475::ERROR::2015-05-21 18:27:24,107::task::866::Storage.TaskManager.Task::(_setError) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::Unexpected error
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,108::task::885::Storage.TaskManager.Task::(_run) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::Task._run: 7ca2f743-09b1-4499-b9e4-5f640002a2bc (u'276e9ba7-e19a-49c5-8ad7-26711934d5e4', u'0f954891-b1cd-4f09-99ae-75d404d95f9d', u'eae65249-e5e8-49e7-90a0-c7385e80e6ca', u'8791f6ec-a6ef-484d-bd5a-730b22b19250') {} failed - stopping task
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,108::task::1217::Storage.TaskManager.Task::(stop) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::stopping in state preparing (force False)
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,108::task::993::Storage.TaskManager.Task::(_decref) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::ref 1 aborting True
vdsm/vdsm.log:Thread-5475::INFO::2015-05-21 18:27:24,109::task::1171::Storage.TaskManager.Task::(prepare) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::aborting: Task is aborted: 'Volume does not exist' - code 201
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,109::task::1176::Storage.TaskManager.Task::(prepare) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::Prepare: aborted: Volume does not exist
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,109::task::993::Storage.TaskManager.Task::(_decref) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::ref 0 aborting True
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,110::task::928::Storage.TaskManager.Task::(_doAbort) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::Task._doAbort: force False
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,110::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,110::task::595::Storage.TaskManager.Task::(_updateState) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::moving from state preparing -> state aborting
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,110::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::_aborting: recover policy none
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,111::task::595::Storage.TaskManager.Task::(_updateState) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::moving from state aborting -> state failed
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,111::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4': < ResourceRef 'Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4', isValid: 'True' obj: 'None'>}
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,111::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,112::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4'
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,112::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4' (0 active users)
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,112::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4' is free, finding out if anyone is waiting for it.
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,113::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4', Clearing records.
vdsm/vdsm.log:Thread-5475::ERROR::2015-05-21 18:27:24,113::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Volume does not exist: (u'8791f6ec-a6ef-484d-bd5a-730b22b19250',)", 'code': 201}}
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,114::vm::2294::vm.Vm::(_startUnderlyingVm) vmId=`24bd5074-64fc-4aa0-87cd-5de3dd7b50d1`::_ongoingCreations released
vdsm/vdsm.log:Thread-5475::ERROR::2015-05-21 18:27:24,114::vm::2331::vm.Vm::(_startUnderlyingVm) vmId=`24bd5074-64fc-4aa0-87cd-5de3dd7b50d1`::The vm start process failed
vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,117::vm::2786::vm.Vm::(setDownStatus) vmId=`24bd5074-64fc-4aa0-87cd-5de3dd7b50d1`::Changed state to Down: Bad volume specification {u'index': 0, u'iface': u'virtio', u'type': u'disk', u'format': u'cow', u'bootOrder': u'1', u'address': {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'volumeID': u'8791f6ec-a6ef-484d-bd5a-730b22b19250', 'apparentsize': '2696609792', u'imageID': u'eae65249-e5e8-49e7-90a0-c7385e80e6ca', u'specParams': {}, u'readonly': u'false', u'domainID': u'276e9ba7-e19a-49c5-8ad7-26711934d5e4', 'reqsize': '0', u'deviceId': u'eae65249-e5e8-49e7-90a0-c7385e80e6ca', 'truesize': '2696552448', u'poolID': u'0f954891-b1cd-4f09-99ae-75d404d95f9d', u'device': u'disk', u'shared': u'false', u'propagateErrors': u'off', u'optional': u'false'} (code=1)
<— snip —>
Additionally i can find this
—
Thread-5475::ERROR::2015-05-21 18:27:24,107::task::866::Storage.TaskManager.Task::(_setError) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 3235, in prepareImage
raise se.VolumeDoesNotExist(leafUUID)
VolumeDoesNotExist: Volume does not exist: (u'8791f6ec-a6ef-484d-bd5a-730b22b19250’,)
—
—
Thread-5475::ERROR::2015-05-21 18:27:24,114::vm::2331::vm.Vm::(_startUnderlyingVm) vmId=`24bd5074-64fc-4aa0-87cd-5de3dd7b50d1`::The vm start process failed
Traceback (most recent call last):
File "/usr/share/vdsm/virt/vm.py", line 2271, in _startUnderlyingVm
self._run()
File "/usr/share/vdsm/virt/vm.py", line 3266, in _run
self.preparePaths(devices[DISK_DEVICES])
File "/usr/share/vdsm/virt/vm.py", line 2353, in preparePaths
drive['path'] = self.cif.prepareVolumePath(drive, self.id)
File "/usr/share/vdsm/clientIF.py", line 277, in prepareVolumePath
raise vm.VolumeError(drive)
VolumeError: Bad volume specification {u'index': 0, u'iface': u'virtio', u'type': u'disk', u'format': u'cow', u'bootOrder': u'1', u'address': {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'volumeID': u'8791f6ec-a6ef-484d-bd5a-730b22b19250', 'apparentsize': '2696609792', u'imageID': u'eae65249-e5e8-49e7-90a0-c7385e80e6ca', u'specParams': {}, u'readonly': u'false', u'domainID': u'276e9ba7-e19a-49c5-8ad7-26711934d5e4', 'reqsize': '0', u'deviceId': u'eae65249-e5e8-49e7-90a0-c7385e80e6ca', 'truesize': '2696552448', u'poolID': u'0f954891-b1cd-4f09-99ae-75d404d95f9d', u'device': u'disk', u'shared': u'false', u'propagateErrors': u'off', u'optional': u'false'}
Thread-5475::DEBUG::2015-05-21 18:27:24,117::vm::2786::vm.Vm::(setDownStatus) vmId=`24bd5074-64fc-4aa0-87cd-5de3dd7b50d1`::Changed state to Down: Bad volume specification {u'index': 0, u'iface': u'virtio', u'type': u'disk', u'format': u'cow', u'bootOrder': u'1', u'address': {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'volumeID': u'8791f6ec-a6ef-484d-bd5a-730b22b19250', 'apparentsize': '2696609792', u'imageID': u'eae65249-e5e8-49e7-90a0-c7385e80e6ca', u'specParams': {}, u'readonly': u'false', u'domainID': u'276e9ba7-e19a-49c5-8ad7-26711934d5e4', 'reqsize': '0', u'deviceId': u'eae65249-e5e8-49e7-90a0-c7385e80e6ca', 'truesize': '2696552448', u'poolID': u'0f954891-b1cd-4f09-99ae-75d404d95f9d', u'device': u'disk', u'shared': u'false', u'propagateErrors': u'off', u'optional': u'false'} (code=1)
—
The thing is, if i move that respective VM back to the old Cluster running Fedora 20 with the libvirt from the libvirt-preview repo, then the VM start with no problem.
That volume ‘8791f6ec-a6ef-484d-bd5a-730b22b19250’ does not exist
I experienced that problem with several Vms now – OS does not matter, also i check snapshots, there are none, i also tried cloning the VM and then moving it over, no luck either.
Any ideas where to look ?
Regards
Soeren
From: <users-bounces at ovirt.org<mailto:users-bounces at ovirt.org>> on behalf of Soeren Malchow
Date: Wednesday 20 May 2015 15:42
To: "Bloemen, Jurriën", "users at ovirt.org<mailto:users at ovirt.org>"
Subject: Re: [ovirt-users] Switch from Fedora 20 to CentOS 7.1
Great, thanks, that is the plan then
From: <users-bounces at ovirt.org<mailto:users-bounces at ovirt.org>> on behalf of "Bloemen, Jurriën"
Date: Wednesday 20 May 2015 15:27
To: "users at ovirt.org<mailto:users at ovirt.org>"
Subject: Re: [ovirt-users] Switch from Fedora 20 to CentOS 7.1
Hi Soeren,
Yes! That works perfectly. Did it myself several times.
Regards,
Jurriën
On 20-05-15 14:19, Soeren Malchow wrote:
Hi Vered,
Thanks for the quick answer, ok, understood
Then i could create a new Cluster in the same datacenter with newly installed hosts and then migrate the machines through shutting them down in the old cluster and then starting them in the new cluster, only thing i loose is the live migration
Regards
Soeren
On 20/05/15 14:04, "Vered Volansky" <vered at redhat.com><mailto:vered at redhat.com> wrote:
Hi Soeren,
oVirt Clusters support one host distribution (all hosts must be of the same distribution).
If the cluster is empty at one point, you can add a host of a different distribution than the cluster occupied before.
But there can't be two type of distributions at the same time in one cluster.
Regards,
Vered
----- Original Message -----
From: "Soeren Malchow" <soeren.malchow at mcon.net><mailto:soeren.malchow at mcon.net>
To: users at ovirt.org<mailto:users at ovirt.org>
Sent: Wednesday, May 20, 2015 2:58:11 PM
Subject: [ovirt-users] Switch from Fedora 20 to CentOS 7.1
Dear all,
Would it be possible to switch from fedora 20 to centos 7.1 (as far as i
understood it has live merge support now) within one cluster, meaning
* take out one compute host
* Reinstall that compute host with Centos 7.1
* Do a hosted-engine —deploy
* Migrate VM to the CentOS 7.1 host
* Take the next fedora host and reinstall
Any experiences, recommendations or remarks on that ?
Regards
Soeren
_______________________________________________
Users mailing list
Users at ovirt.org<mailto:Users at ovirt.org>http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users at ovirt.org<mailto:Users at ovirt.org>http://lists.ovirt.org/mailman/listinfo/users
This message (including any attachments) may contain information that is privileged or confidential. If you are not the intended recipient, please notify the sender and delete this email immediately from your systems and destroy all copies of it. You may not, directly or indirectly, use, disclose, distribute, print or copy this email or any part of it if you are not the intended recipient
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20150521/0bf06f8e/attachment-0001.html>
More information about the Users
mailing list