7:35 p.m.
This may be happening because I changed cluster compatibility to 4.3 then
immediately after changed data center compatibility to 4.3 (before
restarting VMs after cluster compatibility change). If this is the case I
can't fix by downgrading the data center compatibility to 4.2 as it won't
allow me to do so. What can I do to fix this, any VM I restart will break
(I am leaving the others running for now, but there are some down that I
can't start).
Full error from VDSM:
2019-02-13 13:30:55,465-0400 ERROR (vm/d070ce80) [storage.TaskManager.Task]
(Task='d5c8e50a-0a6f-4fe7-be79-fd322b273a1e') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882,
in _run
return fn(*args, **kargs)
File "<string>", line 2, in prepareImage
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3198,
in prepareImage
legality = dom.produceVolume(imgUUID, volUUID).getLegality()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 818, in
produceVolume
volUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/glusterVolume.py",
line 45, in __init__
volUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 800,
in __init__
self._manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line
71, in __init__
volUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 86,
in __init__
self.validate()
File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 112,
in validate
self.validateVolumePath()
File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line
131, in validateVolumePath
raise se.VolumeDoesNotExist(self.volUUID)
VolumeDoesNotExist: Volume does not exist:
(u'2d6d5f87-ccb0-48ce-b3ac-84495bd12d32',)
2019-02-13 13:30:55,468-0400 ERROR (vm/d070ce80) [storage.Dispatcher]
FINISH prepareImage error=Volume does not exist:
(u'2d6d5f87-ccb0-48ce-b3ac-84495bd12d32',) (dispatcher:81)
2019-02-13 13:30:55,469-0400 ERROR (vm/d070ce80) [virt.vm]
(vmId='d070ce80-e0bc-489d-8ee0-47d5926d5ae2') The vm start process failed
(vm:937)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 866, in
_startUnderlyingVm
self._run()
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2749, in
_run
self._devices = self._make_devices()
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2589, in
_make_devices
disk_objs = self._perform_host_local_adjustment()
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2662, in
_perform_host_local_adjustment
self._preparePathsForDrives(disk_params)
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1011, in
_preparePathsForDrives
drive['path'] = self.cif.prepareVolumePath(drive, self.id)
File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 415, in
prepareVolumePath
raise vm.VolumeError(drive)
VolumeError: Bad volume specification {'address': {'function':
'0x0',
'bus': '0x00', 'domain': '0x0000', 'type':
'pci', 'slot': '0x06'},
'serial': 'd81a6826-dc46-44db-8de7-405d30e44d57', 'index': 0,
'iface':
'virtio', 'apparentsize': '64293699584', 'specParams': {},
'cache': 'none',
'imageID': 'd81a6826-dc46-44db-8de7-405d30e44d57', 'truesize':
'64293814272', 'type': 'disk', 'domainID':
'1f2e9989-9ab3-43d5-971d-568b8feca918', 'reqsize': '0',
'format': 'cow',
'poolID': 'a45e442e-9989-11e8-b0e4-00163e4bf18a', 'device':
'disk', 'path':
'/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/1f2e9989-9ab3-43d5-971d-568b8feca918/images/d81a6826-dc46-44db-8de7-405d30e44d57/2d6d5f87-ccb0-48ce-b3ac-84495bd12d32',
'propagateErrors': 'off', 'name': 'vda',
'bootOrder': '1', 'volumeID':
'2d6d5f87-ccb0-48ce-b3ac-84495bd12d32', 'diskType': 'file',
'alias':
'ua-d81a6826-dc46-44db-8de7-405d30e44d57', 'discard': False}
On Wed, Feb 13, 2019 at 1:19 PM Jayme <jaymef(a)gmail.com> wrote:
I may have made matters worse. So I changed to 4.3 compatible
cluster
then 4.3 compatible data center. All VMs were marked as requiring a
reboot. I restarted a couple of them and none of them will start up, they
are saying "bad volume specification". The ones running that I did not yet
restart are still running ok. I need to figure out why the VMs aren't
restarting.
Here is an example from vdsm.log
olumeError: Bad volume specification {'address': {'function':
'0x0',
'bus': '0x00', 'domain': '0x0000', 'type':
'pci', 'slot': '0x06'},
'serial': 'd81a6826-dc46-44db-8de7-405d30e44d57', 'index': 0,
'iface':
'virtio', 'apparentsize': '64293699584', 'specParams':
{}, 'cache': 'none',
'imageID': 'd81a6826-dc46-44db-8de7-405d30e44d57', 'truesize':
'64293814272', 'type': 'disk', 'domainID':
'1f2e9989-9ab3-43d5-971d-568b8feca918', 'reqsize': '0',
'format': 'cow',
'poolID': 'a45e442e-9989-11e8-b0e4-00163e4bf18a', 'device':
'disk', 'path':
'/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/1f2e9989-9ab3-43d5-971d-568b8feca918/images/d81a6826-dc46-44db-8de7-405d30e44d57/2d6d5f87-ccb0-48ce-b3ac-84495bd12d32',
'propagateErrors': 'off', 'name': 'vda',
'bootOrder': '1', 'volumeID':
'2d6d5f87-ccb0-48ce-b3ac-84495bd12d32', 'diskType': 'file',
'alias':
'ua-d81a6826-dc46-44db-8de7-405d30e44d57', 'discard': False}
On Wed, Feb 13, 2019 at 1:01 PM Jayme <jaymef(a)gmail.com> wrote:
> I think I just figured out what I was doing wrong. On edit cluster
> screen I was changing both the CPU type and cluster level 4.3. I tried it
> again by switching to the new CPU type first (leaving cluster on 4.2) then
> saving, then going back in and switching compat level to 4.3. It appears
> that you need to do this in two steps for it to work.
>
>
>
> On Wed, Feb 13, 2019 at 12:57 PM Jayme <jaymef(a)gmail.com> wrote:
>
>> Hmm interesting, I wonder how you were able to switch from SandyBridge
>> IBRS to SandyBridge IBRS SSBD. I just attempted the same in both regular
>> mode and in global maintenance mode and it won't allow me to, it says that
>> all hosts have to be in maintenance mode (screenshots attached). Are you
>> also running HCI/Gluster setup?
>>
>>
>>
>> On Wed, Feb 13, 2019 at 12:44 PM Ron Jerome <ronjero(a)gmail.com> wrote:
>>
>>> > Environment setup:
>>> >
>>> > 3 Host HCI GlusterFS setup. Identical hosts, Dell R720s w/ Intel
>>> E5-2690
>>> > CPUs
>>> >
>>> > 1 default data center (4.2 compat)
>>> > 1 default cluster (4.2 compat)
>>> >
>>> > Situation: I recently upgraded my three node HCI cluster from Ovirt
>>> 4.2 to
>>> > 4.3. I did so by first updating the engine to 4.3 then upgrading each
>>> > ovirt-node host to 4.3 and rebooting.
>>> >
>>> > Currently engine and all hosts are running 4.3 and all is working
>>> fine.
>>> >
>>> > To complete the upgrade I need to update cluster compatibility to 4.3
>>> and
>>> > then data centre to 4.3. This is where I am stuck.
>>> >
>>> > The CPU type on cluster is "Intel SandyBridge IBRS Family".
This
>>> option is
>>> > no longer available if I select 4.3 compatibility. Any other option
>>> chosen
>>> > such as SandyBridge IBRS SSBD will not allow me to switch to 4.3 as
>>> all
>>> > hosts must be in maintenance mode (which is not possible w/ self
>>> hosted
>>> > engine).
>>> >
>>> > I saw another post about this where someone else followed steps to
>>> create a
>>> > second cluster on 4.3 with new CPU type then move one host to it,
>>> start
>>> > engine on it then perform other steps to eventually get to 4.3
>>> > compatibility.
>>> >
>>>
>>> I have the exact same hardware configuration and was able to change to
>>> "SandyBridge IBRS SSBD" without creating a new cluster. How I made
that
>>> happen, I'm not so sure, but the cluster may have been in "Global
>>> Maintenance" mode when I changed it.
>>>
>>>
>>> _______________________________________________
>>> Users mailing list -- users(a)ovirt.org
>>> To unsubscribe send an email to users-leave(a)ovirt.org
>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>> oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/
>>> List Archives:
>>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/5B3TAXKO7IB...
>>>
>>