6:18 a.m.
My cluster was originally built on 4.3, and things were working as long as
my SPM was on 4.3. I just killed off the last 4.3 host and rebuilt it as
4.4, and upgraded my cluster and DC to compatibility level 4.6.
We had cephfs mounted as a posix FS which worked fine, but oddly in 4.3 we
would end up with two mounts for the same volume. The configuration had a
comma separated list of IPs as that is how ceph was configured for
redundancy, and this is the mount that shows up on both 4.3 and 4.4 hosts
(/rhev/data-center/mnt/10.1.88.75,10.1.88.76,10.1.88.77:_vmstore/). But
the 4.3 hosts would also have a duplicate mount which had the FQDN of one
of the servers instead of the comma separated list.
In 4.4, there's only a single mount and existing VMs will start just fine,
but you can't create new disks or migrate existing disks onto the posix
storage volume. My suspicion is this is an issue with the mount parser
not liking the comma in the name of the mount from the error that I get on
the SPM host when it tries to create a volume (migration would also fail
on the volume creation task):
2021-08-31 19:34:07,767-0700 INFO (jsonrpc/6) [vdsm.api] START
createVolume(sdUUID='e8ec5645-fc1b-4d64-a145-44aa8ac5ef48',
spUUID='2948c860-9bdf-11e8-a6b3-00163e0419f0',
imgUUID='7d704b4d-1ebe-462f-b11e-b91039f43637', size='1073741824',
volFormat=5, preallocate=1, diskType='DATA',
volUUID='be6cb033-4e42-4bf5-a4a3-6ab5bf03edee',
desc='{"DiskAlias":"test","DiskDescription":""}',
srcImgUUID='00000000-0000-0000-0000-000000000000',
srcVolUUID='00000000-0000-0000-0000-000000000000', initialSize=None,
addBitmaps=False) from=::ffff:10.1.2.37,43490,
flow_id=bb137995-1ffa-429f-b6eb-5b9ca9f8dfd7, task_id=2ddfd1bc-d7e1-4a1e-877a-68e1c2a897ed
(api:48)
2021-08-31 19:34:07,767-0700 INFO (jsonrpc/6) [IOProcessClient] (Global) Starting client
(__init__:340)
2021-08-31 19:34:07,782-0700 INFO (ioprocess/3193398) [IOProcess] (Global) Starting
ioprocess (__init__:465)
2021-08-31 19:34:07,803-0700 INFO (jsonrpc/6) [vdsm.api] FINISH createVolume return=None
from=::ffff:10.1.2.37,43490, flow_id=bb137995-1ffa-429f-b6eb-5b9ca9f8dfd7,
task_id=2ddfd1bc-d7e1-4a1e-877a-68e1c2a897ed (api:54)
2021-08-31 19:34:07,844-0700 INFO (tasks/5) [storage.ThreadPool.WorkerThread] START task
2ddfd1bc-d7e1-4a1e-877a-68e1c2a897ed (cmd=<bound method Task.commit of
<vdsm.storage.task.Task object at 0x7f4894279860>>, args=None) (threadPool:146)
2021-08-31 19:34:07,869-0700 INFO (tasks/5) [storage.StorageDomain] Create placeholder
/rhev/data-center/mnt/10.1.88.75,10.1.88.76,10.1.88.77:_vmstore/e8ec5645-fc1b-4d64-a145-44aa8ac5ef48/images/7d704b4d-1ebe-462f-b11e-b91039f43637
for image's volumes (sd:1718)
2021-08-31 19:34:07,869-0700 ERROR (tasks/5) [storage.TaskManager.Task]
(Task='2ddfd1bc-d7e1-4a1e-877a-68e1c2a897ed') Unexpected error (task:877)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 884, in
_run
return fn(*args, **kargs)
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 350, in
run
return self.cmd(*self.argslist, **self.argsdict)
File "/usr/lib/python3.6/site-packages/vdsm/storage/securable.py", line 79,
in wrapper
return method(self, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sp.py", line 1945, in
createVolume
initial_size=initialSize, add_bitmaps=addBitmaps)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 1216, in
createVolume
initial_size=initial_size, add_bitmaps=add_bitmaps)
File "/usr/lib/python3.6/site-packages/vdsm/storage/volume.py", line 1174, in
create
imgPath = dom.create_image(imgUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 1721, in
create_image
"create_image_rollback", [image_dir])
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 385, in
__init__
self.params = ParamList(argslist)
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 298, in
__init__
raise ValueError("ParamsList: sep %s in %s" % (sep, i))
ValueError: ParamsList: sep , in
/rhev/data-center/mnt/10.1.88.75,10.1.88.76,10.1.88.77:_vmstore/e8ec5645-fc1b-4d64-a145-44aa8ac5ef48/images/7d704b4d-1ebe-462f-b11e-b91039f43637
2021-08-31 19:34:07,964-0700 INFO (tasks/5) [storage.ThreadPool.WorkerThread] FINISH task
2ddfd1bc-d7e1-4a1e-877a-68e1c2a897ed (threadPool:148)
This is a pretty major issue since we can no longer create new VMs. As a
workaround, I could change the mount path of the volume to only reference
a single IP, but oVirt won't let me edit the mount. I wonder if I could
manually edit in the database, then reboot the hosts one by one to make
the change take effect without having to shut down hundreds of VMs at
once?
10:52 p.m.
On Wed, Sep 1, 2021 at 6:21 PM Sketch <ovirt(a)rednsx.org> wrote:
My cluster was originally built on 4.3, and things were working as
long as
my SPM was on 4.3. I just killed off the last 4.3 host and rebuilt it as
4.4, and upgraded my cluster and DC to compatibility level 4.6.
We had cephfs mounted as a posix FS which worked fine, but oddly in 4.3 we
would end up with two mounts for the same volume. The configuration had a
comma separated list of IPs as that is how ceph was configured for
redundancy, and this is the mount that shows up on both 4.3 and 4.4 hosts
(/rhev/data-center/mnt/10.1.88.75,10.1.88.76,10.1.88.77:_vmstore/).
This was never supported.
We had this old fix that was rejected:
https://gerrit.ovirt.org/c/vdsm/+/94027
but it will not help to solve the issue with the task argument below.
But
the 4.3 hosts would also have a duplicate mount which had the FQDN of one
of the servers instead of the comma separated list.
In 4.4, there's only a single mount and existing VMs will start just fine,
but you can't create new disks or migrate existing disks onto the posix
storage volume. My suspicion is this is an issue with the mount parser
not liking the comma in the name of the mount from the error that I get on
the SPM host when it tries to create a volume (migration would also fail
on the volume creation task):
2021-08-31 19:34:07,767-0700 INFO (jsonrpc/6) [vdsm.api] START
createVolume(sdUUID='e8ec5645-fc1b-4d64-a145-44aa8ac5ef48',
spUUID='2948c860-9bdf-11e8-a6b3-00163e0419f0',
imgUUID='7d704b4d-1ebe-462f-b11e-b91039f43637', size='1073741824',
volFormat=5, preallocate=1, diskType='DATA',
volUUID='be6cb033-4e42-4bf5-a4a3-6ab5bf03edee',
desc='{"DiskAlias":"test","DiskDescription":""}',
srcImgUUID='00000000-0000-0000-0000-000000000000',
srcVolUUID='00000000-0000-0000-0000-000000000000', initialSize=None,
addBitmaps=False) from=::ffff:10.1.2.37,43490,
flow_id=bb137995-1ffa-429f-b6eb-5b9ca9f8dfd7,
task_id=2ddfd1bc-d7e1-4a1e-877a-68e1c2a897ed (api:48)
2021-08-31 19:34:07,767-0700 INFO (jsonrpc/6) [IOProcessClient] (Global)
Starting client (__init__:340)
2021-08-31 19:34:07,782-0700 INFO (ioprocess/3193398) [IOProcess]
(Global) Starting ioprocess (__init__:465)
2021-08-31 19:34:07,803-0700 INFO (jsonrpc/6) [vdsm.api] FINISH
createVolume return=None from=::ffff:10.1.2.37,43490,
flow_id=bb137995-1ffa-429f-b6eb-5b9ca9f8dfd7,
task_id=2ddfd1bc-d7e1-4a1e-877a-68e1c2a897ed (api:54)
2021-08-31 19:34:07,844-0700 INFO (tasks/5)
[storage.ThreadPool.WorkerThread] START task
2ddfd1bc-d7e1-4a1e-877a-68e1c2a897ed (cmd=<bound method Task.commit of
<vdsm.storage.task.Task object at 0x7f4894279860>>, args=None)
(threadPool:146)
2021-08-31 19:34:07,869-0700 INFO (tasks/5) [storage.StorageDomain]
Create placeholder
/rhev/data-center/mnt/10.1.88.75,10.1.88.76,10.1.88.77:_vmstore/e8ec5645-fc1b-4d64-a145-44aa8ac5ef48/images/7d704b4d-1ebe-462f-b11e-b91039f43637
for image's volumes (sd:1718)
2021-08-31 19:34:07,869-0700 ERROR (tasks/5) [storage.TaskManager.Task]
(Task='2ddfd1bc-d7e1-4a1e-877a-68e1c2a897ed') Unexpected error (task:877)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 884,
in _run
return fn(*args, **kargs)
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 350,
in run
return self.cmd(*self.argslist, **self.argsdict)
File "/usr/lib/python3.6/site-packages/vdsm/storage/securable.py", line
79, in wrapper
return method(self, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sp.py", line 1945,
in createVolume
initial_size=initialSize, add_bitmaps=addBitmaps)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 1216,
in createVolume
initial_size=initial_size, add_bitmaps=add_bitmaps)
File "/usr/lib/python3.6/site-packages/vdsm/storage/volume.py", line
1174, in create
imgPath = dom.create_image(imgUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 1721,
in create_image
"create_image_rollback", [image_dir])
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 385,
in __init__
self.params = ParamList(argslist)
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 298,
in __init__
raise ValueError("ParamsList: sep %s in %s" % (sep, i))
ValueError: ParamsList: sep , in /rhev/data-center/mnt/10.1.88.75
,10.1.88.76,10.1.88.77:
_vmstore/e8ec5645-fc1b-4d64-a145-44aa8ac5ef48/images/7d704b4d-1ebe-462f-b11e-b91039f43637
2021-08-31 19:34:07,964-0700 INFO (tasks/5)
[storage.ThreadPool.WorkerThread] FINISH task
2ddfd1bc-d7e1-4a1e-877a-68e1c2a897ed (threadPool:148)
I think the issue is the task arguments parser - these are separated by
",", and
arguments including "," breaks the parser.
This is a pretty major issue since we can no longer create new VMs.
As a
workaround, I could change the mount path of the volume to only reference
a single IP, but oVirt won't let me edit the mount. I wonder if I could
manually edit in the database, then reboot the hosts one by one to make
the change take effect without having to shut down hundreds of VMs at
once?
This should work.
Please file a bug for this, so we can consider this for the next release.
Nir
6:37 a.m.
New subject: posix storage mount path error when creating volumes
On Wed, 1 Sep 2021, Nir Soffer wrote:
This is a pretty major issue since we can no longer create new
VMs. As a
workaround, I could change the mount path of the volume to only
reference
a single IP, but oVirt won't let me edit the mount. I wonder
if I could
manually edit in the database, then reboot the hosts one by one
to make
the change take effect without having to shut down hundreds of
VMs at
once?
This should work.
I tried editing the database to point to a single IP and it works, I'm
able to mount it and create disks on it.
However, attempts to migrate any live VMs between a system with the old
mount and one with the new mount fail, presumably because the mountpoint
is now named differently. I believe doing this would effectively split my
cluster into hosts on two separate storage pools. I wonder if there's
some way to force the mount name to be the same?
10:44 a.m.
New subject: posix storage mount path error when creating volumes
We are use CephFS domains without issues from 4.3, currently on 4.4
[root@ovirt-host1 /]# ls -la /rhev/data-center/mnt/*ovirt*
'/rhev/data-center/mnt/172.16.16.2:3300,172.16.16.3:3300,172.16.16.4:3300:_ovirt__iso':
total 0
drwxrwxrwx. 3 root root 1 Sep 3 15:43 .
drwxr-xr-x. 5 vdsm kvm 182 Aug 19 14:35 ..
drwxr-xr-x. 4 vdsm kvm 2 Aug 19 14:35 def09ff0-e986-44c4-ac1c-470668ec2822
[root@ovirt-host1 /]# mount | grep iso
172.16.16.2:3300,172.16.16.3:3300,172.16.16.4:3300:/ovirt_iso on
/rhev/data-center/mnt/172.16.16.2:3300,172.16.16.3:3300,172.16.16.4:3300:_ovirt__iso type
ceph
(rw,relatime,seclabel,name=mds_ovirt_iso,secret=<hidden>,ms_mode=prefer-crc,dirstat,acl)
k
On 1 Sep 2021, at 23:52, Nir Soffer <nsoffer(a)redhat.com>
wrote:
This was never supported.
We had this old fix that was rejected:
https://gerrit.ovirt.org/c/vdsm/+/94027 <https://gerrit.ovirt.org/c/vdsm/+/94027>
986
days inactive
987
days old
3 comments
3 participants
participants (3)
-
Konstantin Shalygin -
Nir Soffer -
Sketch