8:11 p.m.
After this
(https://devconfcz2020a.sched.com/event/YOtG/ovirt-4k-teaching-an-old-dog-...)
I sure do not expect this (log below):
Actually I am trying to evaluate just how portable oVirt storage is and this case I had
prepared a USB3 HDD with VDO, which I could literally move between farms to transport
VMs.
Logical disks are typically large for simplicity within the VMs, QCOW2 and VDO assumed to
compensate for this 'lack of planning' while the allocated storage easily fits the
HDD.
Once I got beyond the initial issues, I came across this somewhat unexpected issue: VDO
storage uses 4k blocks all around, but evidently when you mount an export domain (or I
guess any domain) as POSIX, 512byte blocks are assumed somewhere and 4k blocks rejected.
I'd say that is a bug in VDSM, right?
Or is there anything in the mount options to fix this?
020-08-31 18:44:40,424+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=())
from=internal, task_id=7a293dec-85b3-4b82-92b7-4e7d03b40343 (api:48)
2020-08-31 18:44:40,425+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats
return={'9992dc21-edf2-4951-9020-7c78f1220e02': {'code': 0,
'lastCheck': '1.6', 'delay': '0.00283651',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '25d95783-44df-4fda-b642-55fe09162149': {'code': 0,
'lastCheck': '2.2', 'delay': '0.00256944',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '148d9e9e-d7b8-4220-9ee8-057da96e608c': {'code': 0,
'lastCheck': '2.2', 'delay': '0.00050217',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}} from=internal, task_id=7a293dec-85b3-4b82-92b7-4e7d03b40343 (api:54)
2020-08-31 18:44:41,121+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.ping2
succeeded in 0.00 seconds (__init__:312)
2020-08-31 18:44:41,412+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call Host.ping2
succeeded in 0.00 seconds (__init__:312)
2020-08-31 18:44:41,414+0200 INFO (jsonrpc/2) [vdsm.api] START
repoStats(domains=['9992dc21-edf2-4951-9020-7c78f1220e02']) from=::1,44782,
task_id=657c820e-e2eb-4b1d-b2fb-1772b3af6f32 (api:48)
2020-08-31 18:44:41,414+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats
return={'9992dc21-edf2-4951-9020-7c78f1220e02': {'code': 0,
'lastCheck': '2.6', 'delay': '0.00283651',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}} from=::1,44782, task_id=657c820e-e2eb-4b1d-b2fb-1772b3af6f32 (api:54)
2020-08-31 18:44:41,414+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call
Host.getStorageRepoStats succeeded in 0.00 seconds (__init__:312)
2020-08-31 18:44:41,965+0200 INFO (jsonrpc/3) [IOProcessClient] (Global) Starting client
(__init__:308)
2020-08-31 18:44:41,984+0200 INFO (ioprocess/87332) [IOProcess] (Global) Starting
ioprocess (__init__:434)
2020-08-31 18:44:42,006+0200 INFO (jsonrpc/3) [storage.StorageDomainCache] Removing
domain fe9fb0db-2743-457a-80f0-9a4edc509e9d from storage domain cache (sdc:211)
2020-08-31 18:44:42,006+0200 INFO (jsonrpc/3) [storage.StorageDomainCache] Invalidating
storage domain cache (sdc:74)
2020-08-31 18:44:42,006+0200 INFO (jsonrpc/3) [vdsm.api] FINISH connectStorageServer
return={'statuslist': [{'id':
'00000000-0000-0000-0000-000000000000', 'status': 0}]}
from=::ffff:192.168.0.87,40378, flow_id=05fe72ef-c8ac-4e03-8453-5171e5fc5f8b,
task_id=b92d4144-a9db-4423-ba6b-35934d4f9200 (api:54)
2020-08-31 18:44:42,008+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call
StoragePool.connectStorageServer succeeded in 3.11 seconds (__init__:312)
2020-08-31 18:44:42,063+0200 INFO (jsonrpc/5) [vdsm.api] START
getStorageDomainsList(spUUID='00000000-0000-0000-0000-000000000000',
domainClass=3, storageType='', remotePath='/dev/mapper/vdo1',
options=None) from=::ffff:192.168.0.87,40378,
flow_id=eff548f9-b663-45dd-b8a5-5854f9b5dde8, task_id=98ca66a5-edee-40d4-9253-6a46409241cc
(api:48)
2020-08-31 18:44:42,063+0200 INFO (jsonrpc/5) [storage.StorageDomainCache] Refreshing
storage domain cache (resize=True) (sdc:80)
2020-08-31 18:44:42,063+0200 INFO (jsonrpc/5) [storage.ISCSI] Scanning iSCSI devices
(iscsi:442)
2020-08-31 18:44:42,101+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.ping2
succeeded in 0.00 seconds (__init__:312)
2020-08-31 18:44:42,128+0200 INFO (jsonrpc/5) [storage.ISCSI] Scanning iSCSI devices:
0.06 seconds (utils:390)
2020-08-31 18:44:42,129+0200 INFO (jsonrpc/5) [storage.HBA] Scanning FC devices (hba:60)
2020-08-31 18:44:42,182+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.ping2
succeeded in 0.00 seconds (__init__:312)
2020-08-31 18:44:42,184+0200 INFO (jsonrpc/4) [vdsm.api] START
repoStats(domains=['9992dc21-edf2-4951-9020-7c78f1220e02']) from=::1,44782,
task_id=7b3f1eb8-f7e3-4467-93a1-27ef9721c90c (api:48)
2020-08-31 18:44:42,184+0200 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats
return={'9992dc21-edf2-4951-9020-7c78f1220e02': {'code': 0,
'lastCheck': '3.4', 'delay': '0.00283651',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}} from=::1,44782, task_id=7b3f1eb8-f7e3-4467-93a1-27ef9721c90c (api:54)
2020-08-31 18:44:42,184+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call
Host.getStorageRepoStats succeeded in 0.00 seconds (__init__:312)
2020-08-31 18:44:42,277+0200 INFO (jsonrpc/5) [storage.HBA] Scanning FC devices: 0.15
seconds (utils:390)
2020-08-31 18:44:42,309+0200 INFO (jsonrpc/5) [storage.Multipath] Resizing multipath
devices (multipath:104)
2020-08-31 18:44:42,312+0200 INFO (jsonrpc/5) [storage.Multipath] Resizing multipath
devices: 0.00 seconds (utils:390)
2020-08-31 18:44:42,312+0200 INFO (jsonrpc/5) [storage.StorageDomainCache] Refreshing
storage domain cache: 0.25 seconds (utils:390)
2020-08-31 18:44:42,418+0200 INFO (tmap-569/0) [IOProcessClient] (_dev_mapper_vdo1)
Starting client (__init__:308)
2020-08-31 18:44:42,445+0200 INFO (tmap-569/1) [IOProcessClient]
(glusterSD/nucvirt.mtk.hoberg.net:_data) Starting client (__init__:308)
2020-08-31 18:44:42,449+0200 INFO (ioprocess/87496) [IOProcess] (_dev_mapper_vdo1)
Starting ioprocess (__init__:434)
2020-08-31 18:44:42,457+0200 INFO (tmap-569/2) [IOProcessClient]
(glusterSD/nucvirt.mtk.hoberg.net:_vmstore) Starting client (__init__:308)
2020-08-31 18:44:42,465+0200 INFO (ioprocess/87508) [IOProcess]
(glusterSD/nucvirt.mtk.hoberg.net:_data) Starting ioprocess (__init__:434)
2020-08-31 18:44:42,478+0200 INFO (tmap-569/3) [IOProcessClient]
(glusterSD/nucvirt.mtk.hoberg.net:_engine) Starting client (__init__:308)
2020-08-31 18:44:42,486+0200 INFO (ioprocess/87516) [IOProcess]
(glusterSD/nucvirt.mtk.hoberg.net:_vmstore) Starting ioprocess (__init__:434)
2020-08-31 18:44:42,514+0200 INFO (ioprocess/87533) [IOProcess]
(glusterSD/nucvirt.mtk.hoberg.net:_engine) Starting ioprocess (__init__:434)
2020-08-31 18:44:42,522+0200 INFO (jsonrpc/5) [IOProcessClient]
(fe9fb0db-2743-457a-80f0-9a4edc509e9d) Starting client (__init__:308)
2020-08-31 18:44:42,553+0200 INFO (ioprocess/87552) [IOProcess]
(fe9fb0db-2743-457a-80f0-9a4edc509e9d) Starting ioprocess (__init__:434)
2020-08-31 18:44:42,582+0200 ERROR (jsonrpc/5) [storage.HSM] Unexpected error (hsm:2843)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2829, in
getStorageDomainsList
dom = sdCache.produce(sdUUID=sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 115, in
produce
domain.getRealDomain()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 156, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 146, in
findDomain
return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
File "/usr/lib/python3.6/site-packages/vdsm/storage/fileSD.py", line 390, in
__init__
manifest.block_size, storage_block_size)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 939, in
_validate_storage_block_size
raise se.StorageDomainBlockSizeMismatch(block_size, storage_block_size)
vdsm.storage.exception.StorageDomainBlockSizeMismatch: Block size does not match storage
block size: 'block_size=512, storage_block_size=4096'
2020-08-31 18:44:42,582+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStorageDomainsList
return={'domlist': []} from=::ffff:192.168.0.87,40378,
flow_id=eff548f9-b663-45dd-b8a5-5854f9b5dde8, task_id=98ca66a5-edee-40d4-9253-6a46409241cc
(api:54)
2020-08-31 18:44:42,582+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call
Host.getStorageDomains succeeded in 0.52 seconds (__init__:312)
2020-08-31 18:44:42,801+0200 INFO (jsonrpc/7) [vdsm.api] START
disconnectStorageServer(domType=6, spUUID='00000000-0000-0000-0000-000000000000',
conList=[{'password': '********', 'vfs_type': 'xfs',
'port': '', 'mnt_options':
'rw,relatime,seclabel,attr2,inode64,noquota,_netdev,x-systemd.device-timeout=0',
'iqn': '', 'connection': '/dev/mapper/vdo1',
'ipv6_enabled': 'false', 'id':
'b4a73d6a-344c-446b-a33d-7b665e78578d', 'user': '',
'tpgt': '1'}], options=None) from=::ffff:192.168.0.87,40378,
flow_id=6678917a-b29c-416c-aa2d-5abc1d5351a3, task_id=3d0cf36e-427d-4000-8857-a8a2af641b10
(api:48)
2020-08-31 18:44:42,801+0200 INFO (jsonrpc/7) [storage.Mount] unmounting
/rhev/data-center/mnt/_dev_mapper_vdo1 (mount:215)
2020-08-31 18:44:42,806+0200 INFO (jsonrpc/0) [vdsm.api] START
disconnectStorageServer(domType=6, spUUID='00000000-0000-0000-0000-000000000000',
conList=[{'password': '********', 'vfs_type': 'xfs',
'port': '', 'mnt_options':
'rw,relatime,seclabel,attr2,inode64,noquota,_netdev,x-systemd.device-timeout=0',
'iqn': '', 'connection': '/dev/mapper/vdo1',
'ipv6_enabled': 'false', 'id':
'b4a73d6a-344c-446b-a33d-7b665e78578d', 'user': '',
'tpgt': '1'}], options=None) from=::ffff:192.168.0.87,40378,
flow_id=bb449405-5b78-4b65-a352-e2e94a269df9, task_id=09783e0a-933c-448d-b0ca-1b5d351eb042
(api:48)
2020-08-31 18:44:42,806+0200 INFO (jsonrpc/0) [storage.Mount] unmounting
/rhev/data-center/mnt/_dev_mapper_vdo1 (mount:215)
2020-08-31 18:44:42,821+0200 ERROR (jsonrpc/0) [storage.HSM] Could not disconnect from
storageServer (hsm:2528)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2524, in
disconnectStorageServer
conObj.disconnect()
File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line
201, in disconnect
self._mount.umount(True, True)
File "/usr/lib/python3.6/site-packages/vdsm/storage/mount.py", line 217, in
umount
umount(self.fs_file, force=force, lazy=lazy, freeloop=freeloop)
File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 56, in
__call__
return callMethod()
File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 54, in
<lambda>
**kwargs)
File "<string>", line 2, in umount
File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in
_callmethod
raise convert_to_error(kind, result)
vdsm.storage.mount.MountError: Command ['/usr/bin/umount', '-f',
'-l', '/rhev/data-center/mnt/_dev_mapper_vdo1'] failed with rc=32
out=b'' err=b'umount: /rhev/data-center/mnt/_dev_mapper_vdo1: not
mounted.\n'
2020-08-31 18:44:42,821+0200 INFO (jsonrpc/0) [storage.StorageDomainCache] Refreshing
storage domain cache (resize=False) (sdc:80)
9:22 p.m.
New subject: can't mount an export domain with 4K block size (VDO) as PosixFS (xfs)
On Mon, Aug 31, 2020 at 8:17 PM <thomas(a)hoberg.net> wrote:
Vdsm learned a few tricks but is still old :-)
I sure do not expect this (log below):
Actually I am trying to evaluate just how portable oVirt storage is and this case I had
prepared a USB3 HDD with VDO,
which I could literally move between farms to transport VMs.
Logical disks are typically large for simplicity within the VMs, QCOW2 and VDO assumed to
compensate for this 'lack of planning' while the allocated storage easily fits the
HDD.
Once I got beyond the initial issues, I came across this somewhat unexpected issue: VDO
storage uses 4k blocks all around,
VDO has 512 bytes emulation mode. This is how gluster used VDO before 4.3.8.
but evidently when you mount an export domain (or I guess any domain)
as POSIX, 512byte blocks are assumed somewhere and 4k blocks rejected.
Yes, NFS and POSIX (internally NFS) storage domains do not support 4k.
With NFS we cannot detect storage domain block size, since NFS does not use
direct I/O on the server side, so we kept the old configuration, 512.
Theoretically
we can change the code to default to 4k in this case, but it was never tested.
POSIX was never tested wth 4k and since internally it is NFS, it has the same
limits. It it likely to support 4k if we create a specific storage
class for it, like
local storage.
Why do you need VDO for export domain? qcow2 keeps only the used data.
You don't have enough space on the disk for keep all VMs in qcow2 format?
I'd say that is a bug in VDSM, right?
No, this is unsupported configuration.
export domain is using safelease, which does not support 4k storage,
and since export
domain it is deprecated and safelease is unmaintained, it will never support 4k.
If you need 4k support for POSIX or NFS storage *data* domain, this is
something that
can be considered for a future version, please file RFE for this.
Nir
Or is there anything in the mount options to fix this?
020-08-31 18:44:40,424+0200 INFO (periodic/0) [vdsm.api] START repoStats(domains=())
from=internal, task_id=7a293dec-85b3-4b82-92b7-4e7d03b40343 (api:48)
2020-08-31 18:44:40,425+0200 INFO (periodic/0) [vdsm.api] FINISH repoStats
return={'9992dc21-edf2-4951-9020-7c78f1220e02': {'code': 0,
'lastCheck': '1.6', 'delay': '0.00283651',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '25d95783-44df-4fda-b642-55fe09162149': {'code': 0,
'lastCheck': '2.2', 'delay': '0.00256944',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '148d9e9e-d7b8-4220-9ee8-057da96e608c': {'code': 0,
'lastCheck': '2.2', 'delay': '0.00050217',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}} from=internal, task_id=7a293dec-85b3-4b82-92b7-4e7d03b40343 (api:54)
2020-08-31 18:44:41,121+0200 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call
Host.ping2 succeeded in 0.00 seconds (__init__:312)
2020-08-31 18:44:41,412+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call
Host.ping2 succeeded in 0.00 seconds (__init__:312)
2020-08-31 18:44:41,414+0200 INFO (jsonrpc/2) [vdsm.api] START
repoStats(domains=['9992dc21-edf2-4951-9020-7c78f1220e02']) from=::1,44782,
task_id=657c820e-e2eb-4b1d-b2fb-1772b3af6f32 (api:48)
2020-08-31 18:44:41,414+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats
return={'9992dc21-edf2-4951-9020-7c78f1220e02': {'code': 0,
'lastCheck': '2.6', 'delay': '0.00283651',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}} from=::1,44782, task_id=657c820e-e2eb-4b1d-b2fb-1772b3af6f32 (api:54)
2020-08-31 18:44:41,414+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call
Host.getStorageRepoStats succeeded in 0.00 seconds (__init__:312)
2020-08-31 18:44:41,965+0200 INFO (jsonrpc/3) [IOProcessClient] (Global) Starting client
(__init__:308)
2020-08-31 18:44:41,984+0200 INFO (ioprocess/87332) [IOProcess] (Global) Starting
ioprocess (__init__:434)
2020-08-31 18:44:42,006+0200 INFO (jsonrpc/3) [storage.StorageDomainCache] Removing
domain fe9fb0db-2743-457a-80f0-9a4edc509e9d from storage domain cache (sdc:211)
2020-08-31 18:44:42,006+0200 INFO (jsonrpc/3) [storage.StorageDomainCache] Invalidating
storage domain cache (sdc:74)
2020-08-31 18:44:42,006+0200 INFO (jsonrpc/3) [vdsm.api] FINISH connectStorageServer
return={'statuslist': [{'id':
'00000000-0000-0000-0000-000000000000', 'status': 0}]}
from=::ffff:192.168.0.87,40378, flow_id=05fe72ef-c8ac-4e03-8453-5171e5fc5f8b,
task_id=b92d4144-a9db-4423-ba6b-35934d4f9200 (api:54)
2020-08-31 18:44:42,008+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call
StoragePool.connectStorageServer succeeded in 3.11 seconds (__init__:312)
2020-08-31 18:44:42,063+0200 INFO (jsonrpc/5) [vdsm.api] START
getStorageDomainsList(spUUID='00000000-0000-0000-0000-000000000000',
domainClass=3, storageType='', remotePath='/dev/mapper/vdo1',
options=None) from=::ffff:192.168.0.87,40378,
flow_id=eff548f9-b663-45dd-b8a5-5854f9b5dde8, task_id=98ca66a5-edee-40d4-9253-6a46409241cc
(api:48)
2020-08-31 18:44:42,063+0200 INFO (jsonrpc/5) [storage.StorageDomainCache] Refreshing
storage domain cache (resize=True) (sdc:80)
2020-08-31 18:44:42,063+0200 INFO (jsonrpc/5) [storage.ISCSI] Scanning iSCSI devices
(iscsi:442)
2020-08-31 18:44:42,101+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call
Host.ping2 succeeded in 0.00 seconds (__init__:312)
2020-08-31 18:44:42,128+0200 INFO (jsonrpc/5) [storage.ISCSI] Scanning iSCSI devices:
0.06 seconds (utils:390)
2020-08-31 18:44:42,129+0200 INFO (jsonrpc/5) [storage.HBA] Scanning FC devices
(hba:60)
2020-08-31 18:44:42,182+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call
Host.ping2 succeeded in 0.00 seconds (__init__:312)
2020-08-31 18:44:42,184+0200 INFO (jsonrpc/4) [vdsm.api] START
repoStats(domains=['9992dc21-edf2-4951-9020-7c78f1220e02']) from=::1,44782,
task_id=7b3f1eb8-f7e3-4467-93a1-27ef9721c90c (api:48)
2020-08-31 18:44:42,184+0200 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats
return={'9992dc21-edf2-4951-9020-7c78f1220e02': {'code': 0,
'lastCheck': '3.4', 'delay': '0.00283651',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}} from=::1,44782, task_id=7b3f1eb8-f7e3-4467-93a1-27ef9721c90c (api:54)
2020-08-31 18:44:42,184+0200 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call
Host.getStorageRepoStats succeeded in 0.00 seconds (__init__:312)
2020-08-31 18:44:42,277+0200 INFO (jsonrpc/5) [storage.HBA] Scanning FC devices: 0.15
seconds (utils:390)
2020-08-31 18:44:42,309+0200 INFO (jsonrpc/5) [storage.Multipath] Resizing multipath
devices (multipath:104)
2020-08-31 18:44:42,312+0200 INFO (jsonrpc/5) [storage.Multipath] Resizing multipath
devices: 0.00 seconds (utils:390)
2020-08-31 18:44:42,312+0200 INFO (jsonrpc/5) [storage.StorageDomainCache] Refreshing
storage domain cache: 0.25 seconds (utils:390)
2020-08-31 18:44:42,418+0200 INFO (tmap-569/0) [IOProcessClient] (_dev_mapper_vdo1)
Starting client (__init__:308)
2020-08-31 18:44:42,445+0200 INFO (tmap-569/1) [IOProcessClient]
(glusterSD/nucvirt.mtk.hoberg.net:_data) Starting client (__init__:308)
2020-08-31 18:44:42,449+0200 INFO (ioprocess/87496) [IOProcess] (_dev_mapper_vdo1)
Starting ioprocess (__init__:434)
2020-08-31 18:44:42,457+0200 INFO (tmap-569/2) [IOProcessClient]
(glusterSD/nucvirt.mtk.hoberg.net:_vmstore) Starting client (__init__:308)
2020-08-31 18:44:42,465+0200 INFO (ioprocess/87508) [IOProcess]
(glusterSD/nucvirt.mtk.hoberg.net:_data) Starting ioprocess (__init__:434)
2020-08-31 18:44:42,478+0200 INFO (tmap-569/3) [IOProcessClient]
(glusterSD/nucvirt.mtk.hoberg.net:_engine) Starting client (__init__:308)
2020-08-31 18:44:42,486+0200 INFO (ioprocess/87516) [IOProcess]
(glusterSD/nucvirt.mtk.hoberg.net:_vmstore) Starting ioprocess (__init__:434)
2020-08-31 18:44:42,514+0200 INFO (ioprocess/87533) [IOProcess]
(glusterSD/nucvirt.mtk.hoberg.net:_engine) Starting ioprocess (__init__:434)
2020-08-31 18:44:42,522+0200 INFO (jsonrpc/5) [IOProcessClient]
(fe9fb0db-2743-457a-80f0-9a4edc509e9d) Starting client (__init__:308)
2020-08-31 18:44:42,553+0200 INFO (ioprocess/87552) [IOProcess]
(fe9fb0db-2743-457a-80f0-9a4edc509e9d) Starting ioprocess (__init__:434)
2020-08-31 18:44:42,582+0200 ERROR (jsonrpc/5) [storage.HSM] Unexpected error (hsm:2843)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2829, in
getStorageDomainsList
dom = sdCache.produce(sdUUID=sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 115, in
produce
domain.getRealDomain()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 156, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 146, in
findDomain
return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
File "/usr/lib/python3.6/site-packages/vdsm/storage/fileSD.py", line 390, in
__init__
manifest.block_size, storage_block_size)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sd.py", line 939, in
_validate_storage_block_size
raise se.StorageDomainBlockSizeMismatch(block_size, storage_block_size)
vdsm.storage.exception.StorageDomainBlockSizeMismatch: Block size does not match storage
block size: 'block_size=512, storage_block_size=4096'
2020-08-31 18:44:42,582+0200 INFO (jsonrpc/5) [vdsm.api] FINISH getStorageDomainsList
return={'domlist': []} from=::ffff:192.168.0.87,40378,
flow_id=eff548f9-b663-45dd-b8a5-5854f9b5dde8, task_id=98ca66a5-edee-40d4-9253-6a46409241cc
(api:54)
2020-08-31 18:44:42,582+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call
Host.getStorageDomains succeeded in 0.52 seconds (__init__:312)
2020-08-31 18:44:42,801+0200 INFO (jsonrpc/7) [vdsm.api] START
disconnectStorageServer(domType=6, spUUID='00000000-0000-0000-0000-000000000000',
conList=[{'password': '********', 'vfs_type': 'xfs',
'port': '', 'mnt_options':
'rw,relatime,seclabel,attr2,inode64,noquota,_netdev,x-systemd.device-timeout=0',
'iqn': '', 'connection': '/dev/mapper/vdo1',
'ipv6_enabled': 'false', 'id':
'b4a73d6a-344c-446b-a33d-7b665e78578d', 'user': '',
'tpgt': '1'}], options=None) from=::ffff:192.168.0.87,40378,
flow_id=6678917a-b29c-416c-aa2d-5abc1d5351a3, task_id=3d0cf36e-427d-4000-8857-a8a2af641b10
(api:48)
2020-08-31 18:44:42,801+0200 INFO (jsonrpc/7) [storage.Mount] unmounting
/rhev/data-center/mnt/_dev_mapper_vdo1 (mount:215)
2020-08-31 18:44:42,806+0200 INFO (jsonrpc/0) [vdsm.api] START
disconnectStorageServer(domType=6, spUUID='00000000-0000-0000-0000-000000000000',
conList=[{'password': '********', 'vfs_type': 'xfs',
'port': '', 'mnt_options':
'rw,relatime,seclabel,attr2,inode64,noquota,_netdev,x-systemd.device-timeout=0',
'iqn': '', 'connection': '/dev/mapper/vdo1',
'ipv6_enabled': 'false', 'id':
'b4a73d6a-344c-446b-a33d-7b665e78578d', 'user': '',
'tpgt': '1'}], options=None) from=::ffff:192.168.0.87,40378,
flow_id=bb449405-5b78-4b65-a352-e2e94a269df9, task_id=09783e0a-933c-448d-b0ca-1b5d351eb042
(api:48)
2020-08-31 18:44:42,806+0200 INFO (jsonrpc/0) [storage.Mount] unmounting
/rhev/data-center/mnt/_dev_mapper_vdo1 (mount:215)
2020-08-31 18:44:42,821+0200 ERROR (jsonrpc/0) [storage.HSM] Could not disconnect from
storageServer (hsm:2528)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2524, in
disconnectStorageServer
conObj.disconnect()
File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line
201, in disconnect
self._mount.umount(True, True)
File "/usr/lib/python3.6/site-packages/vdsm/storage/mount.py", line 217, in
umount
umount(self.fs_file, force=force, lazy=lazy, freeloop=freeloop)
File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 56, in
__call__
return callMethod()
File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 54, in
<lambda>
**kwargs)
File "<string>", line 2, in umount
File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in
_callmethod
raise convert_to_error(kind, result)
vdsm.storage.mount.MountError: Command ['/usr/bin/umount', '-f',
'-l', '/rhev/data-center/mnt/_dev_mapper_vdo1'] failed with rc=32
out=b'' err=b'umount: /rhev/data-center/mnt/_dev_mapper_vdo1: not
mounted.\n'
2020-08-31 18:44:42,821+0200 INFO (jsonrpc/0) [storage.StorageDomainCache] Refreshing
storage domain cache (resize=False) (sdc:80)
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZCLXQA2NIX5...
9:59 p.m.
New subject: can't mount an export domain with 4K block size (VDO) as PosixFS (xfs)
I like playing with shiny new toys and VDO is one :-)
Actually with that it shouldn't matter if whatever is put on top actually has spare
allocation in the file format or just blocks with zeros, which makes for a reusable HDD in
case I am working with imageio images that are raw.
Good to know that using NFS directly won't cost extra performance, because that's
how I got around the 4K issue.
Alas, plenty of qemu-img imports still fail with the ENOENT error on the 3nHCI gluster.
From what I am reading between the lines, oVirt would be rock solid, if somebody
didn't want to compete with Nutanix and had put Gluster underneath...
With a 25% success rate at trying to transfer VMs from 4.3 to 4.4 I am... in no position
to migrate, really.
Trying the backup domains next, but their documentation is pretty awful if non-existent.
1544
days inactive
1544
days old
2 comments
2 participants
participants (2)
-
Nir Soffer -
thomas＠hoberg.net