6:29 a.m.
Hi,
I have a 3 node cluster with gluster (2 rhel 8.7 and one rhel 9.1).
I updated one of my nodes to rhel9 and I'm unable to start any VM that's
running on gluster volumes on the rhel9 node or move a running VM on the
rhel9 node.
However, I'm able to start a node stored on a NFS node on the rhel9 node.
According to gluster (10.3) all 3 nodes are working fine.
I don't want to render my setup useless, that's why I didn't move the 2
other nodes.
wells = rhel9 node.
In the event log I'm getting the following error:
Failed to sync storage devices from host wells
VDSM wells command GetStorageDeviceListVDS failed: Internal JSON-RPC error
Here is a part of the vdsm.log:
2023-04-03 23:25:01,242-0400 INFO (vm/69f6480f) [vds] prepared volume
path: (clientIF:506)
2023-04-03 23:25:01,243-0400 INFO (vm/69f6480f) [vdsm.api] START
prepareImage(sdUUID='a874a247-d8de-4951-86e8-99aaeda1a510',
spUUID='595abf38-2c6d-11eb-9ba9-3ca82afed888',
imgUUID='69de5e2b-dfb7-4e47-93d0-65ed4f978017',
leafUUID='e66df35e-1254-48ef-bb83-ae8957fe8651', allowIllegal=False)
from=internal, task_id=bf3e89e0-5d4e-47a1-8d86-3933c5004d25 (api:31)
2023-04-03 23:25:01,243-0400 INFO (vm/69f6480f) [vdsm.api] FINISH
prepareImage error=Unknown pool id, pool not connected:
('595abf38-2c6d-11eb-9ba9-3ca82afed888',) from=internal,
task_id=bf3e89e0-5d4e-47a1-8d86-3933c5004d25 (api:35)
2023-04-03 23:25:01,244-0400 INFO (vm/69f6480f) [storage.taskmanager.task]
(Task='bf3e89e0-5d4e-47a1-8d86-3933c5004d25') aborting: Task is aborted:
"value=Unknown pool id, pool not connected:
('595abf38-2c6d-11eb-9ba9-3ca82afed888',) abortedcode=309" (task:1165)
2023-04-03 23:25:01,244-0400 INFO (vm/69f6480f) [storage.dispatcher]
FINISH prepareImage error=Unknown pool id, pool not connected:
('595abf38-2c6d-11eb-9ba9-3ca82afed888',) (dispatcher:64)
2023-04-03 23:25:01,244-0400 ERROR (vm/69f6480f) [virt.vm]
(vmId='69f6480f-5ee0-4210-84b9-58b4dbb4e423') The vm start process failed
(vm:1001)
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/vdsm/virt/vm.py", line 928, in
_startUnderlyingVm
self._run()
File "/usr/lib/python3.9/site-packages/vdsm/virt/vm.py", line 2769, in
_run
self._devices = self._make_devices()
File "/usr/lib/python3.9/site-packages/vdsm/virt/vm.py", line 2649, in
_make_devices
disk_objs = self._perform_host_local_adjustment()
File "/usr/lib/python3.9/site-packages/vdsm/virt/vm.py", line 2687, in
_perform_host_local_adjustment
self._preparePathsForDrives(disk_params)
File "/usr/lib/python3.9/site-packages/vdsm/virt/vm.py", line 1109, in
_preparePathsForDrives
drive['path'] = self.cif.prepareVolumePath(
File "/usr/lib/python3.9/site-packages/vdsm/clientIF.py", line 418, in
prepareVolumePath
raise vm.VolumeError(drive)
vdsm.virt.vm.VolumeError: Bad volume specification {'device': 'disk',
'type': 'disk', 'diskType': 'file', 'specParams':
{}, 'alias':
'ua-69de5e2b-dfb7-4e47-93d0-65ed4f978017', 'address': {'bus':
'0',
'controller': '0', 'target': '0', 'type':
'drive', 'unit': '0'},
'domainID': 'a874a247-d8de-4951-86e8-99aaeda1a510', 'guestName':
'/dev/sdb', 'imageID': '69de5e2b-dfb7-4e47-93d0-65ed4f978017',
'poolID':
'595abf38-2c6d-11eb-9ba9-3ca82afed888', 'volumeID':
'e66df35e-1254-48ef-bb83-ae8957fe8651', 'managed': False,
'volumeChain':
[{'domainID': 'a874a247-d8de-4951-86e8-99aaeda1a510', 'imageID':
'69de5e2b-dfb7-4e47-93d0-65ed4f978017', 'leaseOffset': 0,
'leasePath':
'/rhev/data-center/mnt/glusterSD/192.168.8.11:_GlusterVolLow/a874a247-d8de-4951-86e8-99aaeda1a510/images/69de5e2b-dfb7-4e47-93d0-65ed4f978017/e66df35e-1254-48ef-bb83-ae8957fe8651.lease',
'path':
'/rhev/data-center/mnt/glusterSD/192.168.8.11:_GlusterVolLow/a874a247-d8de-4951-86e8-99aaeda1a510/images/69de5e2b-dfb7-4e47-93d0-65ed4f978017/e66df35e-1254-48ef-bb83-ae8957fe8651',
'volumeID': 'e66df35e-1254-48ef-bb83-ae8957fe8651'}], 'path':
'/rhev/data-center/mnt/glusterSD/192.168.8.11:_GlusterVolLow/a874a247-d8de-4951-86e8-99aaeda1a510/images/69de5e2b-dfb7-4e47-93d0-65ed4f978017/e66df35e-1254-48ef-bb83-ae8957fe8651',
'discard': False, 'format': 'raw', 'propagateErrors':
'off', 'cache':
'none', 'iface': 'scsi', 'name': 'sda',
'bootOrder': '1', 'serial':
'69de5e2b-dfb7-4e47-93d0-65ed4f978017', 'index': 0, 'reqsize':
'0',
'truesize': '5331688448', 'apparentsize': '42949672960'}
2023-04-03 23:25:01,244-0400 INFO (vm/69f6480f) [virt.vm]
(vmId='69f6480f-5ee0-4210-84b9-58b4dbb4e423') Changed state to Down: Bad
volume specification {'device': 'disk', 'type': 'disk',
'diskType': 'file',
'specParams': {}, 'alias':
'ua-69de5e2b-dfb7-4e47-93d0-65ed4f978017',
'address': {'bus': '0', 'controller': '0',
'target': '0', 'type': 'drive',
'unit': '0'}, 'domainID':
'a874a247-d8de-4951-86e8-99aaeda1a510',
'guestName': '/dev/sdb', 'imageID':
'69de5e2b-dfb7-4e47-93d0-65ed4f978017',
'poolID': '595abf38-2c6d-11eb-9ba9-3ca82afed888', 'volumeID':
'e66df35e-1254-48ef-bb83-ae8957fe8651', 'managed': False,
'volumeChain':
[{'domainID': 'a874a247-d8de-4951-86e8-99aaeda1a510', 'imageID':
'69de5e2b-dfb7-4e47-93d0-65ed4f978017', 'leaseOffset': 0,
'leasePath':
'/rhev/data-center/mnt/glusterSD/192.168.8.11:_GlusterVolLow/a874a247-d8de-4951-86e8-99aaeda1a510/images/69de5e2b-dfb7-4e47-93d0-65ed4f978017/e66df35e-1254-48ef-bb83-ae8957fe8651.lease',
'path':
'/rhev/data-center/mnt/glusterSD/192.168.8.11:_GlusterVolLow/a874a247-d8de-4951-86e8-99aaeda1a510/images/69de5e2b-dfb7-4e47-93d0-65ed4f978017/e66df35e-1254-48ef-bb83-ae8957fe8651',
'volumeID': 'e66df35e-1254-48ef-bb83-ae8957fe8651'}], 'path':
'/rhev/data-center/mnt/glusterSD/192.168.8.11:_GlusterVolLow/a874a247-d8de-4951-86e8-99aaeda1a510/images/69de5e2b-dfb7-4e47-93d0-65ed4f978017/e66df35e-1254-48ef-bb83-ae8957fe8651',
'discard': False, 'format': 'raw', 'propagateErrors':
'off', 'cache':
'none', 'iface': 'scsi', 'name': 'sda',
'bootOrder': '1', 'serial':
'69de5e2b-dfb7-4e47-93d0-65ed4f978017', 'index': 0, 'reqsize':
'0',
'truesize': '5331688448', 'apparentsize': '42949672960'}
(code=1) (vm:1743)
2023-04-03 23:25:01,245-0400 INFO (vm/69f6480f) [virt.vm]
(vmId='69f6480f-5ee0-4210-84b9-58b4dbb4e423') Stopping connection
(guestagent:421)
2023-04-03 23:25:01,252-0400 INFO (jsonrpc/7) [api.virt] START
destroy(gracefulAttempts=1) from=::ffff:192.168.2.24,56838,
vmId=69f6480f-5ee0-4210-84b9-58b4dbb4e423 (api:31)
2023-04-03 23:25:01,252-0400 INFO (jsonrpc/7) [virt.vm]
(vmId='69f6480f-5ee0-4210-84b9-58b4dbb4e423') Release VM resources (vm:5324)