Re: [ovirt-users] Switch from Fedora 20 to CentOS 7.1

From: "Soeren Malchow" <soeren.malchow(a)mcon.net&gt; To: "Jurriën Bloemen" <Jurrien.Bloemen(a)dmc.amcnetworks.com&gt;, users(a)ovirt.org Sent: Thursday, May 21, 2015 7:35:02 PM Subject: Re: [ovirt-users] Switch from Fedora 20 to CentOS 7.1 Hi, We now created the new Cluster based on CentOS 7.1 which went fine, then we migrated 2 machines – no problem, we have Live Migration (back), Live Merge and so on, all good. But some additional machine have problems starting on the new cluster and this happens Grep for the Thread in vdsm.log <— snip —> vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:21,999::vm::2264::vm.Vm::(_startUnderlyingVm) vmId=`24bd5074-64fc-4aa0-87cd-5de3dd7b50d1`::Start vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,003::vm::2268::vm.Vm::(_startUnderlyingVm) vmId=`24bd5074-64fc-4aa0-87cd-5de3dd7b50d1`::_ongoingCreations acquired vdsm/vdsm.log:Thread-5475::INFO::2015-05-21 18:27:22,008::vm::3261::vm.Vm::(_run) vmId=`24bd5074-64fc-4aa0-87cd-5de3dd7b50d1`::VM wrapper has started vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,021::task::595::Storage.TaskManager.Task::(_updateState) Task=`2bc7fe9c-204a-4ab7-a116-f7fbba32bd34`::moving from state init -> state preparing vdsm/vdsm.log:Thread-5475::INFO::2015-05-21 18:27:22,028::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID=u'276e9ba7-e19a-49c5-8ad7-26711934d5e4', spUUID=u'0f954891-b1cd-4f09-99ae-75d404d95f9d', imgUUID=u'eae65249-e5e8-49e7-90a0-c7385e80e6ca', volUUID=u'8791f6ec-a6ef-484d-bd5a-730b22b19250', options=None) vdsm/vdsm.log:Thread-5475::INFO::2015-05-21 18:27:22,069::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '2696552448', 'apparentsize': '2696609792'} vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,069::task::1191::Storage.TaskManager.Task::(prepare) Task=`2bc7fe9c-204a-4ab7-a116-f7fbba32bd34`::finished: {'truesize': '2696552448', 'apparentsize': '2696609792'} vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,069::task::595::Storage.TaskManager.Task::(_updateState) Task=`2bc7fe9c-204a-4ab7-a116-f7fbba32bd34`::moving from state preparing -> state finished vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,070::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,070::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,070::task::993::Storage.TaskManager.Task::(_decref) Task=`2bc7fe9c-204a-4ab7-a116-f7fbba32bd34`::ref 0 aborting False vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,071::task::595::Storage.TaskManager.Task::(_updateState) Task=`c508cf8f-9f02-43a6-a45d-2b3f1d7e66be`::moving from state init -> state preparing vdsm/vdsm.log:Thread-5475::INFO::2015-05-21 18:27:22,071::logUtils::44::dispatcher::(wrapper) Run and protect: getVolumeSize(sdUUID=u'276e9ba7-e19a-49c5-8ad7-26711934d5e4', spUUID=u'0f954891-b1cd-4f09-99ae-75d404d95f9d', imgUUID=u'967d966c-3653-4ff6-9299-2fb5b4197c37', volUUID=u'99b085e6-6662-43ef-8ab4-40bc00e82460', options=None) vdsm/vdsm.log:Thread-5475::INFO::2015-05-21 18:27:22,086::logUtils::47::dispatcher::(wrapper) Run and protect: getVolumeSize, Return response: {'truesize': '1110773760', 'apparentsize': '1110835200'} vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,087::task::1191::Storage.TaskManager.Task::(prepare) Task=`c508cf8f-9f02-43a6-a45d-2b3f1d7e66be`::finished: {'truesize': '1110773760', 'apparentsize': '1110835200'} vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,087::task::595::Storage.TaskManager.Task::(_updateState) Task=`c508cf8f-9f02-43a6-a45d-2b3f1d7e66be`::moving from state preparing -> state finished vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,087::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,088::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,088::task::993::Storage.TaskManager.Task::(_decref) Task=`c508cf8f-9f02-43a6-a45d-2b3f1d7e66be`::ref 0 aborting False vdsm/vdsm.log:Thread-5475::INFO::2015-05-21 18:27:22,088::clientIF::335::vds::(prepareVolumePath) prepared volume path: vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,089::task::595::Storage.TaskManager.Task::(_updateState) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::moving from state init -> state preparing vdsm/vdsm.log:Thread-5475::INFO::2015-05-21 18:27:22,089::logUtils::44::dispatcher::(wrapper) Run and protect: prepareImage(sdUUID=u'276e9ba7-e19a-49c5-8ad7-26711934d5e4', spUUID=u'0f954891-b1cd-4f09-99ae-75d404d95f9d', imgUUID=u'eae65249-e5e8-49e7-90a0-c7385e80e6ca', leafUUID=u'8791f6ec-a6ef-484d-bd5a-730b22b19250') vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,090::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4`ReqID=`2ba5bd10-3b98-44fa-9c90-8a2ade3261dc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '3226' at 'prepareImage' vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,090::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4' for lock type 'shared' vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,091::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4' is free. Now locking as 'shared' (1 active user) vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,091::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4`ReqID=`2ba5bd10-3b98-44fa-9c90-8a2ade3261dc`::Granted request vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,092::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::_resourcesAcquired: Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4 (shared) vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:22,092::task::993::Storage.TaskManager.Task::(_decref) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::ref 1 aborting False vdsm/vdsm.log:Thread-5475::ERROR::2015-05-21 18:27:24,107::task::866::Storage.TaskManager.Task::(_setError) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::Unexpected error vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,108::task::885::Storage.TaskManager.Task::(_run) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::Task._run: 7ca2f743-09b1-4499-b9e4-5f640002a2bc (u'276e9ba7-e19a-49c5-8ad7-26711934d5e4', u'0f954891-b1cd-4f09-99ae-75d404d95f9d', u'eae65249-e5e8-49e7-90a0-c7385e80e6ca', u'8791f6ec-a6ef-484d-bd5a-730b22b19250') {} failed - stopping task vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,108::task::1217::Storage.TaskManager.Task::(stop) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::stopping in state preparing (force False) vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,108::task::993::Storage.TaskManager.Task::(_decref) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::ref 1 aborting True vdsm/vdsm.log:Thread-5475::INFO::2015-05-21 18:27:24,109::task::1171::Storage.TaskManager.Task::(prepare) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::aborting: Task is aborted: 'Volume does not exist' - code 201 vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,109::task::1176::Storage.TaskManager.Task::(prepare) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::Prepare: aborted: Volume does not exist vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,109::task::993::Storage.TaskManager.Task::(_decref) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::ref 0 aborting True vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,110::task::928::Storage.TaskManager.Task::(_doAbort) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::Task._doAbort: force False vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,110::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,110::task::595::Storage.TaskManager.Task::(_updateState) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::moving from state preparing -> state aborting vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,110::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::_aborting: recover policy none vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,111::task::595::Storage.TaskManager.Task::(_updateState) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::moving from state aborting -> state failed vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,111::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4': < ResourceRef 'Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4', isValid: 'True' obj: 'None'>} vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,111::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,112::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4' vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,112::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4' (0 active users) vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,112::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4' is free, finding out if anyone is waiting for it. vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,113::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.276e9ba7-e19a-49c5-8ad7-26711934d5e4', Clearing records. vdsm/vdsm.log:Thread-5475::ERROR::2015-05-21 18:27:24,113::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Volume does not exist: (u'8791f6ec-a6ef-484d-bd5a-730b22b19250',)", 'code': 201}} vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,114::vm::2294::vm.Vm::(_startUnderlyingVm) vmId=`24bd5074-64fc-4aa0-87cd-5de3dd7b50d1`::_ongoingCreations released vdsm/vdsm.log:Thread-5475::ERROR::2015-05-21 18:27:24,114::vm::2331::vm.Vm::(_startUnderlyingVm) vmId=`24bd5074-64fc-4aa0-87cd-5de3dd7b50d1`::The vm start process failed vdsm/vdsm.log:Thread-5475::DEBUG::2015-05-21 18:27:24,117::vm::2786::vm.Vm::(setDownStatus) vmId=`24bd5074-64fc-4aa0-87cd-5de3dd7b50d1`::Changed state to Down: Bad volume specification {u'index': 0, u'iface': u'virtio', u'type': u'disk', u'format': u'cow', u'bootOrder': u'1', u'address': {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'volumeID': u'8791f6ec-a6ef-484d-bd5a-730b22b19250', 'apparentsize': '2696609792', u'imageID': u'eae65249-e5e8-49e7-90a0-c7385e80e6ca', u'specParams': {}, u'readonly': u'false', u'domainID': u'276e9ba7-e19a-49c5-8ad7-26711934d5e4', 'reqsize': '0', u'deviceId': u'eae65249-e5e8-49e7-90a0-c7385e80e6ca', 'truesize': '2696552448', u'poolID': u'0f954891-b1cd-4f09-99ae-75d404d95f9d', u'device': u'disk', u'shared': u'false', u'propagateErrors': u'off', u'optional': u'false'} (code=1) <— snip —> Additionally i can find this — Thread-5475::ERROR::2015-05-21 18:27:24,107::task::866::Storage.TaskManager.Task::(_setError) Task=`7ca2f743-09b1-4499-b9e4-5f640002a2bc`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3235, in prepareImage raise se.VolumeDoesNotExist(leafUUID) VolumeDoesNotExist: Volume does not exist: (u'8791f6ec-a6ef-484d-bd5a-730b22b19250’,) — — Thread-5475::ERROR::2015-05-21 18:27:24,114::vm::2331::vm.Vm::(_startUnderlyingVm) vmId=`24bd5074-64fc-4aa0-87cd-5de3dd7b50d1`::The vm start process failed Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 2271, in _startUnderlyingVm self._run() File "/usr/share/vdsm/virt/vm.py", line 3266, in _run self.preparePaths(devices[DISK_DEVICES]) File "/usr/share/vdsm/virt/vm.py", line 2353, in preparePaths drive['path'] = self.cif.prepareVolumePath(drive, self.id) File "/usr/share/vdsm/clientIF.py", line 277, in prepareVolumePath raise vm.VolumeError(drive) VolumeError: Bad volume specification {u'index': 0, u'iface': u'virtio', u'type': u'disk', u'format': u'cow', u'bootOrder': u'1', u'address': {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'volumeID': u'8791f6ec-a6ef-484d-bd5a-730b22b19250', 'apparentsize': '2696609792', u'imageID': u'eae65249-e5e8-49e7-90a0-c7385e80e6ca', u'specParams': {}, u'readonly': u'false', u'domainID': u'276e9ba7-e19a-49c5-8ad7-26711934d5e4', 'reqsize': '0', u'deviceId': u'eae65249-e5e8-49e7-90a0-c7385e80e6ca', 'truesize': '2696552448', u'poolID': u'0f954891-b1cd-4f09-99ae-75d404d95f9d', u'device': u'disk', u'shared': u'false', u'propagateErrors': u'off', u'optional': u'false'} Thread-5475::DEBUG::2015-05-21 18:27:24,117::vm::2786::vm.Vm::(setDownStatus) vmId=`24bd5074-64fc-4aa0-87cd-5de3dd7b50d1`::Changed state to Down: Bad volume specification {u'index': 0, u'iface': u'virtio', u'type': u'disk', u'format': u'cow', u'bootOrder': u'1', u'address': {u'slot': u'0x06', u'bus': u'0x00', u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, u'volumeID': u'8791f6ec-a6ef-484d-bd5a-730b22b19250', 'apparentsize': '2696609792', u'imageID': u'eae65249-e5e8-49e7-90a0-c7385e80e6ca', u'specParams': {}, u'readonly': u'false', u'domainID': u'276e9ba7-e19a-49c5-8ad7-26711934d5e4', 'reqsize': '0', u'deviceId': u'eae65249-e5e8-49e7-90a0-c7385e80e6ca', 'truesize': '2696552448', u'poolID': u'0f954891-b1cd-4f09-99ae-75d404d95f9d', u'device': u'disk', u'shared': u'false', u'propagateErrors': u'off', u'optional': u'false'} (code=1) — The thing is, if i move that respective VM back to the old Cluster running Fedora 20 with the libvirt from the libvirt-preview repo, then the VM start with no problem. That volume ‘8791f6ec-a6ef-484d-bd5a-730b22b19250’ does not exist I experienced that problem with several Vms now – OS does not matter, also i check snapshots, there are none, i also tried cloning the VM and then moving it over, no luck either. Any ideas where to look ?

Regards Soeren From: < users-bounces(a)ovirt.org > on behalf of Soeren Malchow Date: Wednesday 20 May 2015 15:42 To: "Bloemen, Jurriën", " users(a)ovirt.org " Subject: Re: [ovirt-users] Switch from Fedora 20 to CentOS 7.1 Great, thanks, that is the plan then From: < users-bounces(a)ovirt.org > on behalf of "Bloemen, Jurriën" Date: Wednesday 20 May 2015 15:27 To: " users(a)ovirt.org " Subject: Re: [ovirt-users] Switch from Fedora 20 to CentOS 7.1 Hi Soeren, Yes! That works perfectly. Did it myself several times. Regards, Jurriën On 20-05-15 14:19, Soeren Malchow wrote: Hi Vered, Thanks for the quick answer, ok, understood Then i could create a new Cluster in the same datacenter with newly installed hosts and then migrate the machines through shutting them down in the old cluster and then starting them in the new cluster, only thing i loose is the live migration Regards Soeren On 20/05/15 14:04, "Vered Volansky" <vered(a)redhat.com&gt; wrote: Hi Soeren, oVirt Clusters support one host distribution (all hosts must be of the same distribution). If the cluster is empty at one point, you can add a host of a different distribution than the cluster occupied before. But there can't be two type of distributions at the same time in one cluster. Regards, Vered ----- Original Message ----- From: "Soeren Malchow" <soeren.malchow(a)mcon.net&gt; To: users(a)ovirt.org Sent: Wednesday, May 20, 2015 2:58:11 PM Subject: [ovirt-users] Switch from Fedora 20 to CentOS 7.1 Dear all, Would it be possible to switch from fedora 20 to centos 7.1 (as far as i understood it has live merge support now) within one cluster, meaning * take out one compute host * Reinstall that compute host with Centos 7.1 * Do a hosted-engine —deploy * Migrate VM to the CentOS 7.1 host * Take the next fedora host and reinstall Any experiences, recommendations or remarks on that ? Regards Soeren _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users This message (including any attachments) may contain information that is privileged or confidential. If you are not the intended recipient, please notify the sender and delete this email immediately from your systems and destroy all copies of it. You may not, directly or indirectly, use, disclose, distribute, print or copy this email or any part of it if you are not the intended recipient _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users