1:15 a.m.
Um...
Does the gluster included in version 4.2.7 improve small file I/O performance?
If that 's the case, I will test you as you say.
Finally, I want to ask one more time.
How can I use lustre?
보내는사람 : Yaniv Kaul <ykaul(a)redhat.com>
받는사람 : '김형근'<okok102928(a)fusiondata.co.kr>
참조 : Ovirt Users <users(a)ovirt.org>
보낸 날짜 : 2018-10-29 16:57:57
제목 : Re: [ovirt-users] Storage domain mount error: Lustre file system(Posix compliant
FS)
On Thu, Oct 25, 2018 at 7:25 AM <okok102928(a)fusiondata.co.kr> wrote:
Hi.
I am an ovirt user in Korea. I am working on VDI. It's a pleasure to meet you, the
ovirt specialists.
(I do not speak English well... Thank you for your understanding!)
I am testing Lustre File System in Ovirt or RH(E)V environment.
(The reason is simple: glusterfs and nfs have limit of performance, SAN Storage and
excellent Software Defined Storage are quite expensive.)
Please re-test Gluster with latest 4.2.7, where we have improved performance nicely
specifically for Gluster based storage.
Testing for file system performance was successful.
As expected, luster showed amazing performance.
However, there was an error adding luster storage to the storage domain as Posix
compliant FS.
Domain Function : Data
Storage Type : POSIX compliant FS
Host to Use : [SPM_HOSTNAME]
Name : [STORAGE_DOMAIN_NAME]
Path : 10.10.10.15@tcp:/lustre/vmstore
VFS Type : lustre
Mount Options :
The vdsm debug logs are shown below.
2018-10-25 12:46:58,963+0900 INFO (jsonrpc/2) [storage.xlease] Formatting
index for lockspace u'c0ef7ee6-1da9-4eef-9e03-387cd3a24445' (version=1)
(xlease:653)
2018-10-25 12:46:58,971+0900 DEBUG (jsonrpc/2) [root] /usr/bin/dd iflag=fullblock
of=/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases
oflag=direct,seek_bytes seek=1048576 bs=256512 count=1 conv=notrunc,nocreat,fsync (cwd
None) (commands:65)
2018-10-25 12:46:58,985+0900 DEBUG (jsonrpc/2) [root] FAILED: <err> =
"/usr/bin/dd: error writing
'/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases':
Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000943896 s,
0.0 kB/s\n"; <rc> = 1 (commands:86)
2018-10-25 12:46:58,985+0900 INFO (jsonrpc/2) [vdsm.api] FINISH
createStorageDomain error=Command ['/usr/bin/dd', 'iflag=fullblock',
u'of=/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases',
'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512',
'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1
out='[suppressed]' err="/usr/bin/dd: error writing
'/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases':
Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000943896 s,
0.0 kB/s\n" from=::ffff:192.168.161.104,52188, flow_id=794bd395,
task_id=c9847bf3-2267-483b-9099-f05a46981f7f (api:50)
2018-10-25 12:46:58,985+0900 ERROR (jsonrpc/2) [storage.TaskManager.Task]
(Task='c9847bf3-2267-483b-9099-f05a46981f7f') Unexpected error (task:875)
Traceback (most recent call last): File
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run return fn(*args, **kargs) File
"<string>", line 2, in createStorageDomain
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line
48, in method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line
2591, in createStorageDomain
storageType, domVersion)
File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py",
line 87, in create
remotePath, storageType, version)
File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py",
line 465, in _prepareMetadata
cls.format_external_leases(sdUUID, xleases_path)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line
1200, in format_external_leases
xlease.format_index(lockspace, backend)
File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py",
line 661, in format_index
index.dump(file)
File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py",
line 761, in dump
file.pwrite(INDEX_BASE, self._buf)
File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py",
line 994, in pwrite
self._run(args, data=buf[:])
File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py",
line 1011, in _run
raise cmdutils.Error(args, rc, "[suppressed]", err)
Error: Command ['/usr/bin/dd', 'iflag=fullblock',
u'of=/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases',
'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512',
'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1
out='[suppressed]' err="/usr/bin/dd: error writing
'/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases':
Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000943896 s,
0.0 kB/s\n"
2018-10-25 12:46:58,986+0900 DEBUG (jsonrpc/2) [storage.TaskManager.Task]
(Task='c9847bf3-2267-483b-9099-f05a46981f7f') Task._run:
c9847bf3-2267-483b-9099-f05a46981f7f (6, u'c0ef7ee6-1da9-4eef-9e03-387cd3a24445',
u'vmstore', u'10.10.10.15@tcp:/lustre/vmstore', 1, u'4') {} failed
- stopping task (task:894)
2018-10-25 12:46:58,986+0900 DEBUG (jsonrpc/2) [storage.TaskManager.Task]
(Task='c9847bf3-2267-483b-9099-f05a46981f7f') stopping in state failed (force
False) (task:1256)
2018-10-25 12:46:58,986+0900 DEBUG (jsonrpc/2) [storage.TaskManager.Task]
(Task='c9847bf3-2267-483b-9099-f05a46981f7f') ref 1 aborting True (task:1002)
2018-10-25 12:46:58,986+0900 INFO (jsonrpc/2) [storage.TaskManager.Task]
(Task='c9847bf3-2267-483b-9099-f05a46981f7f') aborting: Task is aborted:
u'Command [\'/usr/bin/dd\', \'iflag=fullblock\',
u\'of=/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases\',
\'oflag=direct,seek_bytes\', \'seek=1048576\', \'bs=256512\',
\'count=1\', \'conv=notrunc,nocreat,fsync\'] failed with rc=1
out=\'[suppressed]\' err="/usr/bin/dd: error writing
\'/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases\':
Invalid argument\\n1+0 records in\\n0+0 records out\\n0 bytes (0 B) copied, 0.000943896 s,
0.0 kB/s\\n"' - code 100 (task:1181)
2018-10-25 12:46:58,986+0900 DEBUG (jsonrpc/2) [storage.TaskManager.Task]
(Task='c9847bf3-2267-483b-9099-f05a46981f7f') Prepare: aborted: Command
['/usr/bin/dd', 'iflag=fullblock',
u'of=/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases',
'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512',
'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1
out='[suppressed]' err="/usr/bin/dd: error writing
'/rhev/data-center/mnt/10.10..10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases':
Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000943896 s,
0.0 kB/s\n" (task:1186)
2018-10-25 12:46:58,986+0900 DEBUG (jsonrpc/2) [storage.TaskManager.Task]
(Task='c9847bf3-2267-483b-9099-f05a46981f7f') ref 0 aborting True (task:1002)
2018-10-25 12:46:58,986+0900 DEBUG (jsonrpc/2) [storage.TaskManager.Task]
(Task='c9847bf3-2267-483b-9099-f05a46981f7f') Task._doAbort: force False
(task:937)
2018-10-25 12:46:58,986+0900 DEBUG (jsonrpc/2) [storage.ResourceManager.Owner]
Owner.cancelAll requests {} (resourceManager:947)
2018-10-25 12:46:58,986+0900 DEBUG (jsonrpc/2) [storage.TaskManager.Task]
(Task='c9847bf3-2267-483b-9099-f05a46981f7f') moving from state failed ->
state aborting (task:602)
2018-10-25 12:46:58,986+0900 DEBUG (jsonrpc/2) [storage.TaskManager.Task]
(Task='c9847bf3-2267-483b-9099-f05a46981f7f') _aborting: recover policy none
(task:557)
2018-10-25 12:46:58,987+0900 DEBUG (jsonrpc/2) [storage.TaskManager.Task]
(Task='c9847bf3-2267-483b-9099-f05a46981f7f') moving from state failed ->
state failed (task:602)
2018-10-25 12:46:58,987+0900 DEBUG (jsonrpc/2) [storage.ResourceManager.Owner]
Owner.releaseAll requests {} resources {} (resourceManager:910)
2018-10-25 12:46:58,987+0900 DEBUG (jsonrpc/2) [storage.ResourceManager.Owner]
Owner.cancelAll requests {} (resourceManager:947)
2018-10-25 12:46:58,987+0900 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH
createStorageDomain error=Command ['/usr/bin/dd', 'iflag=fullblock',
u'of=/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases',
'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512',
'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1
out='[suppressed]' err="/usr/bin/dd: error writing
'/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases':
Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000943896 s,
0.0 kB/s\n" (dispatcher:86)
Traceback (most recent call last):
File
"/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 73, in
wrapper
result = ctask.prepare(func, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line
108, in wrapper
return m(self, *a, **kw)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line
1189, in prepare
raise self.error
Error: Command ['/usr/bin/dd', 'iflag=fullblock',
u'of=/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases',
'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512',
'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1
out='[suppressed]' err="/usr/bin/dd: error writing
'/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases':
Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000943896 s,
0.0 kB/s\n"
2018-10-25 12:46:58,987+0900 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC
call StorageDomain.create failed (error 351) in 0.41 seconds (__init__:573)
2018-10-25 12:46:59,058+0900 DEBUG (jsonrpc/3) [jsonrpc.JsonRpcServer] Calling
'StoragePool.disconnectStorageServer' in bridge with {u'connectionParams':
[{u'id': u'316c5f1f-753e-42a1-8e30-4ee6f976906a', u'connection':
u'10.10.10.15@tcp:/lustre/vmstore', u'iqn': u'', u'user':
u'', u'tpgt': u'1', u'vfs_type': u'lustre',
u'password': '********', u'port': u''}],
u'storagepoolID': u'00000000-0000-0000-0000-000000000000',
u'domainType': 6} (__init__:590)
2018-10-25 12:46:59,058+0900 WARN (jsonrpc/3) [devel] Provided value
"6" not defined in StorageDomainType enum for
StoragePool.disconnectStorageServer (vdsmapi:275)
2018-10-25 12:46:59,058+0900 WARN (jsonrpc/3) [devel] Provided parameters
{u'id': u'316c5f1f-753e-42a1-8e30-4ee6f976906a', u'connection':
u'10.10.10.15@tcp:/lustre/vmstore', u'iqn': u'', u'user':
u'', u'tpgt': u'1', u'vfs_type': u'lustre',
u'password': '********', u'port': u''} do not match any of
union ConnectionRefParameters values (vdsmapi:275)
2018-10-25 12:46:59,059+0900 DEBUG (jsonrpc/3) [storage.TaskManager.Task]
(Task='3c6b249f-a47f-47f1-a647-5893b6f60b7c') moving from state preparing
-> state preparing (task:602)
2018-10-25 12:46:59,059+0900 INFO (jsonrpc/3) [vdsm.api] START
disconnectStorageServer(domType=6, spUUID=u'00000000-0000-0000-0000-000000000000',
conList=[{u'id': u'316c5f1f-753e-42a1-8e30-4ee6f976906a',
u'connection': u'10.10.10.15@tcp:/lustre/vmstore', u'iqn':
u'', u'user': u'', u'tpgt': u'1',
u'vfs_type': u'lustre', u'password': '********',
u'port': u''}], options=None) from=::ffff:192.168.161.104,52188,
flow_id=f1bf4bf8-9033-42af-9329-69960638ba0e, task_id=3c6b249f-a47f-47f1-a647-5893b6f60b7c
(api:46)
2018-10-25 12:46:59,059+0900 INFO (jsonrpc/3) [storage.Mount] unmounting
/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore (mount:212)
2018-10-25 12:46:59,094+0900 DEBUG (jsonrpc/3) [storage.Mount]
/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore unmounted: 0.03 seconds (utils:452)
1. If you use the direct flag when using the dd command, luster only works in multiples
of 4k (4096).
This is it, VDSM does not support native 4K.
Y.
Therefore, bs = 256512, which is not a multiple of 4096, causes an error.
2. Ovirt / RH(E)V An error occurs regardless of version. I have tested both in 3.6,
4.1, and 4.2 environments.
I searched hard, but I did not have a similar case. I want to ask three questions.
1. Is there a way to fix the problem in Ovirt? (Secure bypass or some configurations)
2. (Extension of Question # 1) Where does block size 256512 come from? Why is it
256512?
3. Is this a problem you need to solve in the Lustre file system? (For example, a
setting capable of direct IO in units of 512 bytes)
I need help. Thank you for your reply.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/2XVAJGYIJEZ...