On Wed, Feb 13, 2019 at 8:06 PM Jayme <jaymef@gmail.com> wrote:


Yes, you definitively are.
Fixing files ownership on file system side is a valid workaround. 
 

On Wed, Feb 13, 2019 at 1:35 PM Jayme <jaymef@gmail.com> wrote:
This may be happening because I changed cluster compatibility to 4.3 then immediately after changed data center compatibility to 4.3 (before restarting VMs after cluster compatibility change).  If this is the case I can't fix by downgrading the data center compatibility to 4.2 as it won't allow me to do so.  What can I do to fix this, any VM I restart will break (I am leaving the others running for now, but there are some down that I can't start). 

Full error from VDSM:

2019-02-13 13:30:55,465-0400 ERROR (vm/d070ce80) [storage.TaskManager.Task] (Task='d5c8e50a-0a6f-4fe7-be79-fd322b273a1e') Unexpected error (task:875)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run
    return fn(*args, **kargs)
  File "<string>", line 2, in prepareImage
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method
    ret = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3198, in prepareImage
    legality = dom.produceVolume(imgUUID, volUUID).getLegality()
  File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 818, in produceVolume
    volUUID)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/glusterVolume.py", line 45, in __init__
    volUUID)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 800, in __init__
    self._manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line 71, in __init__
    volUUID)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 86, in __init__
    self.validate()
  File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 112, in validate
    self.validateVolumePath()
  File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line 131, in validateVolumePath
    raise se.VolumeDoesNotExist(self.volUUID)
VolumeDoesNotExist: Volume does not exist: (u'2d6d5f87-ccb0-48ce-b3ac-84495bd12d32',)
2019-02-13 13:30:55,468-0400 ERROR (vm/d070ce80) [storage.Dispatcher] FINISH prepareImage error=Volume does not exist: (u'2d6d5f87-ccb0-48ce-b3ac-84495bd12d32',) (dispatcher:81)
2019-02-13 13:30:55,469-0400 ERROR (vm/d070ce80) [virt.vm] (vmId='d070ce80-e0bc-489d-8ee0-47d5926d5ae2') The vm start process failed (vm:937)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 866, in _startUnderlyingVm
    self._run()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2749, in _run
    self._devices = self._make_devices()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2589, in _make_devices
    disk_objs = self._perform_host_local_adjustment()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2662, in _perform_host_local_adjustment
    self._preparePathsForDrives(disk_params)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1011, in _preparePathsForDrives
    drive['path'] = self.cif.prepareVolumePath(drive, self.id)
  File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 415, in prepareVolumePath
    raise vm.VolumeError(drive)
VolumeError: Bad volume specification {'address': {'function': '0x0', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'slot': '0x06'}, 'serial': 'd81a6826-dc46-44db-8de7-405d30e44d57', 'index': 0, 'iface': 'virtio', 'apparentsize': '64293699584', 'specParams': {}, 'cache': 'none', 'imageID': 'd81a6826-dc46-44db-8de7-405d30e44d57', 'truesize': '64293814272', 'type': 'disk', 'domainID': '1f2e9989-9ab3-43d5-971d-568b8feca918', 'reqsize': '0', 'format': 'cow', 'poolID': 'a45e442e-9989-11e8-b0e4-00163e4bf18a', 'device': 'disk', 'path': '/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/1f2e9989-9ab3-43d5-971d-568b8feca918/images/d81a6826-dc46-44db-8de7-405d30e44d57/2d6d5f87-ccb0-48ce-b3ac-84495bd12d32', 'propagateErrors': 'off', 'name': 'vda', 'bootOrder': '1', 'volumeID': '2d6d5f87-ccb0-48ce-b3ac-84495bd12d32', 'diskType': 'file', 'alias': 'ua-d81a6826-dc46-44db-8de7-405d30e44d57', 'discard': False}

On Wed, Feb 13, 2019 at 1:19 PM Jayme <jaymef@gmail.com> wrote:
I may have made matters worse.  So I changed to 4.3 compatible cluster then 4.3 compatible data center.  All VMs were marked as requiring a reboot.  I restarted a couple of them and none of them will start up, they are saying "bad volume specification".  The ones running that I did not yet restart are still running ok.  I need to figure out why the VMs aren't restarting. 

Here is an example from vdsm.log

olumeError: Bad volume specification {'address': {'function': '0x0', 'bus': '0x00', 'domain': '0x0000', 'type': 'pci', 'slot': '0x06'}, 'serial': 'd81a6826-dc46-44db-8de7-405d30e44d57', 'index': 0, 'iface': 'virtio', 'apparentsize': '64293699584', 'specParams': {}, 'cache': 'none', 'imageID': 'd81a6826-dc46-44db-8de7-405d30e44d57', 'truesize': '64293814272', 'type': 'disk', 'domainID': '1f2e9989-9ab3-43d5-971d-568b8feca918', 'reqsize': '0', 'format': 'cow', 'poolID': 'a45e442e-9989-11e8-b0e4-00163e4bf18a', 'device': 'disk', 'path': '/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/1f2e9989-9ab3-43d5-971d-568b8feca918/images/d81a6826-dc46-44db-8de7-405d30e44d57/2d6d5f87-ccb0-48ce-b3ac-84495bd12d32', 'propagateErrors': 'off', 'name': 'vda', 'bootOrder': '1', 'volumeID': '2d6d5f87-ccb0-48ce-b3ac-84495bd12d32', 'diskType': 'file', 'alias': 'ua-d81a6826-dc46-44db-8de7-405d30e44d57', 'discard': False}

On Wed, Feb 13, 2019 at 1:01 PM Jayme <jaymef@gmail.com> wrote:
I think I just figured out what I was doing wrong.  On edit cluster screen I was changing both the CPU type and cluster level 4.3.  I tried it again by switching to the new CPU type first (leaving cluster on 4.2) then saving, then going back in and switching compat level to 4.3.  It appears that you need to do this in two steps for it to work.



On Wed, Feb 13, 2019 at 12:57 PM Jayme <jaymef@gmail.com> wrote:
Hmm interesting, I wonder how you were able to switch from SandyBridge IBRS to SandyBridge IBRS SSBD.  I just attempted the same in both regular mode and in global maintenance mode and it won't allow me to, it says that all hosts have to be in maintenance mode (screenshots attached).   Are you also running HCI/Gluster setup?



On Wed, Feb 13, 2019 at 12:44 PM Ron Jerome <ronjero@gmail.com> wrote:
> Environment setup:
>
> 3 Host HCI GlusterFS setup.  Identical hosts, Dell R720s w/ Intel E5-2690
> CPUs
>
> 1 default data center (4.2 compat)
> 1 default cluster (4.2 compat)
>
> Situation: I recently upgraded my three node HCI cluster from Ovirt 4.2 to
> 4.3.  I did so by first updating the engine to 4.3 then upgrading each
> ovirt-node host to 4.3 and rebooting.
>
> Currently engine and all hosts are running 4.3 and all is working fine.
>
> To complete the upgrade I need to update cluster compatibility to 4.3 and
> then data centre to 4.3.  This is where I am stuck.
>
> The CPU type on cluster is "Intel SandyBridge IBRS Family".  This option is
> no longer available if I select 4.3 compatibility.  Any other option chosen
> such as SandyBridge IBRS SSBD will not allow me to switch to 4.3 as all
> hosts must be in maintenance mode (which is not possible w/ self hosted
> engine).
>
> I saw another post about this where someone else followed steps to create a
> second cluster on 4.3 with new CPU type then move one host to it, start
> engine on it then perform other steps to eventually get to 4.3
> compatibility.
>

I have the exact same hardware configuration and was able to change to "SandyBridge IBRS SSBD" without creating a new cluster.  How I made that happen, I'm not so sure, but the cluster may have been in "Global Maintenance" mode when I changed it.


_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/5B3TAXKO7IBTWRVNF2K4II472TDISO6P/
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PK7IR27DGLZRZSXVZEN66FL4O377GOHT/