Re: Storage domain mount error: Lustre file system(Posix compliant FS)

31 Oct 2018

      Um...

Does the gluster included in version 4.2.7 improve small file I/O performance?
If that 's the case, I will test you as you say.

Finally, I want to ask one more time.
How can I use lustre?

보내는사람 : Yaniv Kaul <ykaul@redhat.com>

받는사람 : '김형근'<okok102928@fusiondata.co.kr>

참조 : Ovirt Users <users@ovirt.org>

보낸 날짜 : 2018-10-29 16:57:57

제목 : Re: [ovirt-users] Storage domain mount error: Lustre file system(Posix compliant FS)

			On Thu, Oct 25, 2018 at 7:25 AM <okok102928@fusiondata.co.kr> wrote:

			Hi.

			I am an ovirt user in Korea. I am working on VDI. It's a pleasure to meet you, the ovirt specialists.

			(I do not speak English well... Thank you for your understanding!)

			I am testing Lustre File System in Ovirt or RH(E)V environment.

		(The reason is simple: glusterfs and nfs have limit of performance, SAN Storage and excellent Software Defined Storage are quite expensive.)

			Please re-test Gluster with latest 4.2.7, where we have improved performance nicely specifically for Gluster based storage.

			Testing for file system performance was successful.

			As expected, luster showed amazing performance.

			However, there was an error adding luster storage to the storage domain as Posix compliant FS.

			Domain Function : Data

			Storage Type : POSIX compliant FS

			Host to Use : [SPM_HOSTNAME]

			Name : [STORAGE_DOMAIN_NAME]

			Path : 10.10.10.15@tcp:/lustre/vmstore

			VFS Type : lustre

			Mount Options : 

			The vdsm debug logs are shown below.

			2018-10-25 12:46:58,963+0900 INFO  (jsonrpc/2) [storage.xlease] Formatting index for lockspace u'c0ef7ee6-1da9-4eef-9e03-387cd3a24445' (version=1) (xlease:653)

			2018-10-25 12:46:58,971+0900 DEBUG (jsonrpc/2) [root] /usr/bin/dd iflag=fullblock of=/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases oflag=direct,seek_bytes seek=1048576 bs=256512 count=1 conv=notrunc,nocreat,fsync (cwd None) (commands:65)

			2018-10-25 12:46:58,985+0900 DEBUG (jsonrpc/2) [root] FAILED: <err> = "/usr/bin/dd: error writing '/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases': Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000943896 s, 0.0 kB/s\n"; <rc> = 1 (commands:86) 

			2018-10-25 12:46:58,985+0900 INFO  (jsonrpc/2) [vdsm.api] FINISH createStorageDomain error=Command ['/usr/bin/dd', 'iflag=fullblock', u'of=/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases', 'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512', 'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1 out='[suppressed]' err="/usr/bin/dd: error writing '/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases': Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000943896 s, 0.0 kB/s\n" from=::ffff:192.168.161.104,52188, flow_id=794bd395, task_id=c9847bf3-2267-483b-9099-f05a46981f7f (api:50)

			2018-10-25 12:46:58,985+0900 ERROR (jsonrpc/2) [storage.TaskManager.Task] (Task='c9847bf3-2267-483b-9099-f05a46981f7f') Unexpected error (task:875)

			Traceback (most recent call last):  File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run    return fn(*args, **kargs)  File "<string>", line 2, in createStorageDomain

			  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method

			    ret = func(*args, **kwargs)

			  File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2591, in createStorageDomain

			    storageType, domVersion)

			  File "/usr/lib/python2.7/site-packages/vdsm/storage/nfsSD.py", line 87, in create

			    remotePath, storageType, version)

			  File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 465, in _prepareMetadata

			    cls.format_external_leases(sdUUID, xleases_path)

			  File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 1200, in format_external_leases

			    xlease.format_index(lockspace, backend)

			  File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 661, in format_index

			    index.dump(file)

			  File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 761, in dump

			    file.pwrite(INDEX_BASE, self._buf)

			  File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 994, in pwrite

			    self._run(args, data=buf[:])

			  File "/usr/lib/python2.7/site-packages/vdsm/storage/xlease.py", line 1011, in _run

			    raise cmdutils.Error(args, rc, "[suppressed]", err)

			Error: Command ['/usr/bin/dd', 'iflag=fullblock', u'of=/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases', 'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512', 'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1 out='[suppressed]' err="/usr/bin/dd: error writing '/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases': Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000943896 s, 0.0 kB/s\n"

			2018-10-25 12:46:58,986+0900 DEBUG (jsonrpc/2) [storage.TaskManager.Task] (Task='c9847bf3-2267-483b-9099-f05a46981f7f') Task._run: c9847bf3-2267-483b-9099-f05a46981f7f (6, u'c0ef7ee6-1da9-4eef-9e03-387cd3a24445', u'vmstore', u'10.10.10.15@tcp:/lustre/vmstore', 1, u'4') {} failed - stopping task (task:894)

			2018-10-25 12:46:58,986+0900 DEBUG (jsonrpc/2) [storage.TaskManager.Task] (Task='c9847bf3-2267-483b-9099-f05a46981f7f') stopping in state failed (force False) (task:1256)

			2018-10-25 12:46:58,986+0900 DEBUG (jsonrpc/2) [storage.TaskManager.Task] (Task='c9847bf3-2267-483b-9099-f05a46981f7f') ref 1 aborting True (task:1002)

			2018-10-25 12:46:58,986+0900 INFO  (jsonrpc/2) [storage.TaskManager.Task] (Task='c9847bf3-2267-483b-9099-f05a46981f7f') aborting: Task is aborted: u'Command [\'/usr/bin/dd\', \'iflag=fullblock\', u\'of=/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases\', \'oflag=direct,seek_bytes\', \'seek=1048576\', \'bs=256512\', \'count=1\', \'conv=notrunc,nocreat,fsync\'] failed with rc=1 out=\'[suppressed]\' err="/usr/bin/dd: error writing \'/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases\': Invalid argument\\n1+0 records in\\n0+0 records out\\n0 bytes (0 B) copied, 0.000943896 s, 0.0 kB/s\\n"' - code 100 (task:1181)

			2018-10-25 12:46:58,986+0900 DEBUG (jsonrpc/2) [storage.TaskManager.Task] (Task='c9847bf3-2267-483b-9099-f05a46981f7f') Prepare: aborted: Command ['/usr/bin/dd', 'iflag=fullblock', u'of=/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases', 'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512', 'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1 out='[suppressed]' err="/usr/bin/dd: error writing '/rhev/data-center/mnt/10.10..10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases': Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000943896 s, 0.0 kB/s\n" (task:1186)

			2018-10-25 12:46:58,986+0900 DEBUG (jsonrpc/2) [storage.TaskManager.Task] (Task='c9847bf3-2267-483b-9099-f05a46981f7f') ref 0 aborting True (task:1002)

			2018-10-25 12:46:58,986+0900 DEBUG (jsonrpc/2) [storage.TaskManager.Task] (Task='c9847bf3-2267-483b-9099-f05a46981f7f') Task._doAbort: force False (task:937)

			2018-10-25 12:46:58,986+0900 DEBUG (jsonrpc/2) [storage.ResourceManager.Owner] Owner.cancelAll requests {} (resourceManager:947)

			2018-10-25 12:46:58,986+0900 DEBUG (jsonrpc/2) [storage.TaskManager.Task] (Task='c9847bf3-2267-483b-9099-f05a46981f7f') moving from state failed -> state aborting (task:602)

			2018-10-25 12:46:58,986+0900 DEBUG (jsonrpc/2) [storage.TaskManager.Task] (Task='c9847bf3-2267-483b-9099-f05a46981f7f') _aborting: recover policy none (task:557)

			2018-10-25 12:46:58,987+0900 DEBUG (jsonrpc/2) [storage.TaskManager.Task] (Task='c9847bf3-2267-483b-9099-f05a46981f7f') moving from state failed -> state failed (task:602)

			2018-10-25 12:46:58,987+0900 DEBUG (jsonrpc/2) [storage.ResourceManager.Owner] Owner.releaseAll requests {} resources {} (resourceManager:910)

			2018-10-25 12:46:58,987+0900 DEBUG (jsonrpc/2) [storage.ResourceManager.Owner] Owner.cancelAll requests {} (resourceManager:947)

			2018-10-25 12:46:58,987+0900 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH createStorageDomain error=Command ['/usr/bin/dd', 'iflag=fullblock', u'of=/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases', 'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512', 'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1 out='[suppressed]' err="/usr/bin/dd: error writing '/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases': Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000943896 s, 0.0 kB/s\n" (dispatcher:86)

			Traceback (most recent call last):

			  File "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 73, in wrapper

			    result = ctask.prepare(func, *args, **kwargs)

			  File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper

			    return m(self, *a, **kw)

			  File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare

			    raise self.error

			Error: Command ['/usr/bin/dd', 'iflag=fullblock', u'of=/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases', 'oflag=direct,seek_bytes', 'seek=1048576', 'bs=256512', 'count=1', 'conv=notrunc,nocreat,fsync'] failed with rc=1 out='[suppressed]' err="/usr/bin/dd: error writing '/rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore/c0ef7ee6-1da9-4eef-9e03-387cd3a24445/dom_md/xleases': Invalid argument\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000943896 s, 0.0 kB/s\n"

			2018-10-25 12:46:58,987+0900 INFO  (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call StorageDomain.create failed (error 351) in 0.41 seconds (__init__:573)

			2018-10-25 12:46:59,058+0900 DEBUG (jsonrpc/3) [jsonrpc.JsonRpcServer] Calling 'StoragePool.disconnectStorageServer' in bridge with {u'connectionParams': [{u'id': u'316c5f1f-753e-42a1-8e30-4ee6f976906a', u'connection': u'10.10.10.15@tcp:/lustre/vmstore', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'lustre', u'password': '********', u'port': u''}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 6} (__init__:590)

			2018-10-25 12:46:59,058+0900 WARN  (jsonrpc/3) [devel] Provided value "6" not defined in StorageDomainType enum for StoragePool.disconnectStorageServer (vdsmapi:275)

			2018-10-25 12:46:59,058+0900 WARN  (jsonrpc/3) [devel] Provided parameters {u'id': u'316c5f1f-753e-42a1-8e30-4ee6f976906a', u'connection': u'10.10.10.15@tcp:/lustre/vmstore', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'lustre', u'password': '********', u'port': u''} do not match any of union ConnectionRefParameters values (vdsmapi:275)

			2018-10-25 12:46:59,059+0900 DEBUG (jsonrpc/3) [storage.TaskManager.Task] (Task='3c6b249f-a47f-47f1-a647-5893b6f60b7c') moving from state preparing -> state preparing (task:602)

			2018-10-25 12:46:59,059+0900 INFO  (jsonrpc/3) [vdsm.api] START disconnectStorageServer(domType=6, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'id': u'316c5f1f-753e-42a1-8e30-4ee6f976906a', u'connection': u'10.10.10.15@tcp:/lustre/vmstore', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'vfs_type': u'lustre', u'password': '********', u'port': u''}], options=None) from=::ffff:192.168.161.104,52188, flow_id=f1bf4bf8-9033-42af-9329-69960638ba0e, task_id=3c6b249f-a47f-47f1-a647-5893b6f60b7c (api:46)

			2018-10-25 12:46:59,059+0900 INFO  (jsonrpc/3) [storage.Mount] unmounting /rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore (mount:212)

			2018-10-25 12:46:59,094+0900 DEBUG (jsonrpc/3) [storage.Mount] /rhev/data-center/mnt/10.10.10.15@tcp:_lustre_vmstore unmounted: 0.03 seconds (utils:452)

		1. If you use the direct flag when using the dd command, luster only works in multiples of 4k (4096).

			This is it, VDSM does not support native 4K.

			Y.

			 Therefore, bs = 256512, which is not a multiple of 4096, causes an error.

			2. Ovirt / RH(E)V An error occurs regardless of version. I have tested both in 3.6, 4.1, and 4.2 environments.

			I searched hard, but I did not have a similar case. I want to ask three questions.

			1. Is there a way to fix the problem in Ovirt? (Secure bypass or some configurations)

			2. (Extension of Question # 1) Where does block size 256512 come from? Why is it 256512?

			3. Is this a problem you need to solve in the Lustre file system? (For example, a setting capable of direct IO in units of 512 bytes)

			I need help. Thank you for your reply.

			_______________________________________________

			Users mailing list -- users@ovirt.org

			To unsubscribe send an email to users-leave@ovirt.org

			Privacy Statement: https://www.ovirt.org/site/privacy-policy/

			oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/

		List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2XVAJGYIJEZKR4...

김형근

tags

participants (1)