Re: [ovirt-users] [Gluster-users] Hot to force glusterfs to use RDMA?

Thank you all for the nice hints. Somehow my host was not able to access the userspace RDMA, after installing: yum install -y libmlx4.x86_64 I can mount: /usr/bin/mount -t glusterfs -o backup-volfile-servers=10.10. 10.44:10.10.10.42:10.10.10.41,transport=rdma 10.10.10.44:/GluReplica /mnt 10.10.10.44:/GluReplica.rdma 3770662912 407817216 3362845696 <(336)%20284-5696> 11% /mnt Looks the rdma and gluster are working except ovirt GUI:( With MountOptions: backup-volfile-servers=10.10.10.44:10.10.10.42:10.10.10.41,transport=rdma I am not able to activate storage. ---Gluster Status ---- gluster volume status Status of volume: GluReplica Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------ ------------------ Brick 10.10.10.44:/zclei22/01/glu 49162 49163 Y 17173 Brick 10.10.10.42:/zclei21/01/glu 49156 49157 Y 17113 Brick 10.10.10.41:/zclei26/01/glu 49157 49158 Y 16404 Self-heal Daemon on localhost N/A N/A Y 16536 Self-heal Daemon on clei21.vib N/A N/A Y 17134 Self-heal Daemon on 10.10.10.44 N/A N/A Y 17329 Task Status of Volume GluReplica ------------------------------------------------------------ ------------------ There are no active volume tasks -----IB status ----- ibstat CA 'mlx4_0' CA type: MT26428 Number of ports: 1 Firmware version: 2.7.700 Hardware version: b0 Node GUID: 0x002590ffff163758 System image GUID: 0x002590ffff16375b Port 1: State: Active Physical state: LinkUp Rate: 10 Base lid: 273 LMC: 0 SM lid: 3 Capability mask: 0x02590868 Port GUID: 0x002590ffff163759 Link layer: InfiniBand Not bad for SDR switch ! :-P qperf clei22.vib ud_lat ud_bw ud_lat: latency = 23.6 us ud_bw: send_bw = 981 MB/sec recv_bw = 980 MB/sec On Fri, Mar 3, 2017 at 9:08 AM, Deepak Naidu <dnaidu(a)nvidia.com&gt; wrote: > >> As you can see from my previous email that the RDMA connection tested > with qperf. > > I think you have wrong command. Your testing *TCP & not RDMA. *Also > check if you have RDMA & IB modules loaded on your hosts. > > root@clei26 ~]# qperf clei22.vib tcp_bw tcp_lat > tcp_bw: > bw = 475 MB/sec > tcp_lat: > latency = 52.8 us > [root@clei26 ~]# > > > > *Please run below command to test RDMA* > > > > *[root@storageN2 ~]# qperf storageN1 ud_lat ud_bw* > > *ud_lat**:* > > * latency = 7.51 us* > > *ud_bw**:* > > * send_bw = 9.21 GB/sec* > > * recv_bw = 9.21 GB/sec* > > *[root@sc-sdgx-202 ~]#* > > > > Read qperf man pages for more info. > > > > * To run a TCP bandwidth and latency test: > > qperf myserver tcp_bw tcp_lat > > * To run a UDP latency test and then cause the server to terminate: > > qperf myserver udp_lat quit > > * To measure the RDMA UD latency and bandwidth: > > qperf myserver ud_lat ud_bw > > * To measure RDMA UC bi-directional bandwidth: > > qperf myserver rc_bi_bw > > * To get a range of TCP latencies with a message size from 1 to 64K > > qperf myserver -oo msg_size:1:64K:*2 -vu tcp_lat > > > > > > *Check if you have RDMA & IB modules loaded* > > > > lsmod | grep -i ib > > > > lsmod | grep -i rdma > > > > > > > > -- > > Deepak > > > > > > > > *From:* Arman Khalatyan [mailto:arm2arm@gmail.com] > *Sent:* Thursday, March 02, 2017 10:57 PM > *To:* Deepak Naidu > *Cc:* Rafi Kavungal Chundattu Parambil; gluster-users(a)gluster.org; > users; Sahina Bose > *Subject:* RE: [Gluster-users] [ovirt-users] Hot to force glusterfs to > use RDMA? > > > > Dear Deepak, thank you for the hints, which gluster are you using? > > As you can see from my previous email that the RDMA connection tested > with qperf. It is working as expected. In my case the clients are servers > as well, they are hosts for the ovirt. Disabling selinux is nor recommended > by ovirt, but i will give a try. > > > > Am 03.03.2017 7:50 vorm. schrieb "Deepak Naidu" <dnaidu(a)nvidia.com&gt;: > > I have been testing glusterfs over RDMA & below is the command I use. > Reading up the logs, it looks like your IB(InfiniBand) device is not being > initialized. I am not sure if u have an issue on the client IB or the > storage server IB. Also have you configured ur IB devices correctly. I am > using IPoIB. > > Can you check your firewall, disable selinux, I think, you might have > checked it already ? > > > > *mount -t glusterfs -o transport=rdma storageN1:/vol0 /mnt/vol0* > > > > > > · *The below error seems if you have issue starting your volume. > I had issue, when my transport was set to tcp,rdma. I had to force start my > volume. If I had set it only to tcp on the volume, the volume would start > easily.* > > > > [2017-03-02 11:49:47.829391] E [MSGID: 114022] > [client.c:2530:client_init_rpc] 0-GluReplica-client-2: failed to > initialize RPC > [2017-03-02 11:49:47.829413] E [MSGID: 101019] [xlator.c:433:xlator_init] > 0-GluReplica-client-2: Initialization of volume 'GluReplica-client-2' > failed, review your volfile again > [2017-03-02 11:49:47.829425] E [MSGID: 101066] > [graph.c:324:glusterfs_graph_init] 0-GluReplica-client-2: initializing > translator failed > [2017-03-02 11:49:47.829436] E [MSGID: 101176] > [graph.c:673:glusterfs_graph_activate] 0-graph: init failed > > > > · *The below error seems if you have issue with IB device. If > not configured properly.* > > > > [2017-03-02 11:49:47.828996] W [MSGID: 103071] > [rdma.c:4589:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event > channel creation failed [No such device] > [2017-03-02 11:49:47.829067] W [MSGID: 103055] [rdma.c:4896:init] > 0-GluReplica-client-2: Failed to initialize IB Device > [2017-03-02 11:49:47.829080] W [rpc-transport.c:354:rpc_transport_load] > 0-rpc-transport: 'rdma' initialization failed > > > > > > -- > > Deepak > > > > > > *From:* gluster-users-bounces(a)gluster.org [mailto:gluster-users-bounces@ > gluster.org] *On Behalf Of *Sahina Bose > *Sent:* Thursday, March 02, 2017 10:26 PM > *To:* Arman Khalatyan; gluster-users(a)gluster.org; Rafi Kavungal > Chundattu Parambil > *Cc:* users > *Subject:* Re: [Gluster-users] [ovirt-users] Hot to force glusterfs to > use RDMA? > > > > [Adding gluster users to help with error] > > [2017-03-02 11:49:47.828996] W [MSGID: 103071] > [rdma.c:4589:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event > channel creation failed [No such device] > > > > On Thu, Mar 2, 2017 at 5:36 PM, Arman Khalatyan <arm2arm(a)gmail.com&gt; > wrote: > > BTW RDMA is working as expected: > root@clei26 ~]# qperf clei22.vib tcp_bw tcp_lat > tcp_bw: > bw = 475 MB/sec > tcp_lat: > latency = 52.8 us > [root@clei26 ~]# > > thank you beforehand. > > Arman. > > > > On Thu, Mar 2, 2017 at 12:54 PM, Arman Khalatyan <arm2arm(a)gmail.com&gt; > wrote: > > just for reference: > gluster volume info > > Volume Name: GluReplica > Type: Replicate > Volume ID: ee686dfe-203a-4caa-a691-26353460cc48 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x (2 + 1) = 3 > Transport-type: tcp,rdma > Bricks: > Brick1: 10.10.10.44:/zclei22/01/glu > Brick2: 10.10.10.42:/zclei21/01/glu > Brick3: 10.10.10.41:/zclei26/01/glu (arbiter) > Options Reconfigured: > network.ping-timeout: 30 > server.allow-insecure: on > storage.owner-gid: 36 > storage.owner-uid: 36 > cluster.data-self-heal-algorithm: full > features.shard: on > cluster.server-quorum-type: server > cluster.quorum-type: auto > network.remote-dio: enable > cluster.eager-lock: enable > performance.stat-prefetch: off > performance.io-cache: off > performance.read-ahead: off > performance.quick-read: off > performance.readdir-ahead: on > nfs.disable: on > > > > [root@clei21 ~]# gluster volume status > Status of volume: GluReplica > Gluster process TCP Port RDMA Port Online > Pid > ------------------------------------------------------------ > ------------------ > Brick 10.10.10.44:/zclei22/01/glu 49158 49159 Y > 15870 > Brick 10.10.10.42:/zclei21/01/glu 49156 49157 Y > 17473 > Brick 10.10.10.41:/zclei26/01/glu 49153 49154 Y > 18897 > Self-heal Daemon on localhost N/A N/A Y > 17502 > Self-heal Daemon on 10.10.10.41 N/A N/A Y > 13353 > Self-heal Daemon on 10.10.10.44 N/A N/A Y > 32745 > > Task Status of Volume GluReplica > ------------------------------------------------------------ > ------------------ > There are no active volume tasks > > > > On Thu, Mar 2, 2017 at 12:52 PM, Arman Khalatyan <arm2arm(a)gmail.com&gt; > wrote: > > I am not able to mount with RDMA over cli.... > > Are there some volfile parameters needs to be tuned? > /usr/bin/mount -t glusterfs -o backup-volfile-servers=10.10.1 > 0.44:10.10.10.42:10.10.10.41,transport=rdma 10.10.10.44:/GluReplica /mnt > > [2017-03-02 11:49:47.795511] I [MSGID: 100030] [glusterfsd.c:2454:main] > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.9 > (args: /usr/sbin/glusterfs --volfile-server=10.10.10.44 > --volfile-server=10.10.10.44 --volfile-server=10.10.10.42 > --volfile-server=10.10.10.41 --volfile-server-transport=rdma > --volfile-id=/GluReplica.rdma /mnt) > [2017-03-02 11:49:47.812699] I [MSGID: 101190] > [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 1 > [2017-03-02 11:49:47.825210] I [MSGID: 101190] > [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 2 > [2017-03-02 11:49:47.828996] W [MSGID: 103071] > [rdma.c:4589:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event > channel creation failed [No such device] > [2017-03-02 11:49:47.829067] W [MSGID: 103055] [rdma.c:4896:init] > 0-GluReplica-client-2: Failed to initialize IB Device > [2017-03-02 11:49:47.829080] W [rpc-transport.c:354:rpc_transport_load] > 0-rpc-transport: 'rdma' initialization failed > [2017-03-02 11:49:47.829272] W [rpc-clnt.c:1070:rpc_clnt_connection_init] > 0-GluReplica-client-2: loading of new rpc-transport failed > [2017-03-02 11:49:47.829325] I [MSGID: 101053] > [mem-pool.c:641:mem_pool_destroy] 0-GluReplica-client-2: size=588 max=0 > total=0 > [2017-03-02 11:49:47.829371] I [MSGID: 101053] > [mem-pool.c:641:mem_pool_destroy] 0-GluReplica-client-2: size=124 max=0 > total=0 > [2017-03-02 11:49:47.829391] E [MSGID: 114022] > [client.c:2530:client_init_rpc] 0-GluReplica-client-2: failed to > initialize RPC > [2017-03-02 11:49:47.829413] E [MSGID: 101019] [xlator.c:433:xlator_init] > 0-GluReplica-client-2: Initialization of volume 'GluReplica-client-2' > failed, review your volfile again > [2017-03-02 11:49:47.829425] E [MSGID: 101066] > [graph.c:324:glusterfs_graph_init] 0-GluReplica-client-2: initializing > translator failed > [2017-03-02 11:49:47.829436] E [MSGID: 101176] > [graph.c:673:glusterfs_graph_activate] 0-graph: init failed > [2017-03-02 11:49:47.830003] W [glusterfsd.c:1327:cleanup_and_exit] > (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x3c1) [0x7f524c9dbeb1] > -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x172) [0x7f524c9d65d2] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f524c9d5b4b] ) 0-: > received signum (1), shutting down > [2017-03-02 11:49:47.830053] I [fuse-bridge.c:5794:fini] 0-fuse: > Unmounting '/mnt'. > [2017-03-02 11:49:47.831014] W [glusterfsd.c:1327:cleanup_and_exit] > (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f524b343dc5] > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f524c9d5cd5] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f524c9d5b4b] ) 0-: > received signum (15), shutting down > [2017-03-02 11:49:47.831014] W [glusterfsd.c:1327:cleanup_and_exit] > (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f524b343dc5] > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f524c9d5cd5] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f524c9d5b4b] ) 0-: > received signum (15), shutting down > > > > On Thu, Mar 2, 2017 at 12:11 PM, Sahina Bose <sabose(a)redhat.com&gt; wrote: > > You will need to pass additional mount options while creating the storage > domain (transport=rdma) > > Please let us know if this works. > > > > On Thu, Mar 2, 2017 at 2:42 PM, Arman Khalatyan <arm2arm(a)gmail.com&gt; > wrote: > > Hi, > > Are there way to force the connections over RDMA only? > > If I check host mounts I cannot see rdma mount option: > mount -l| grep gluster > 10.10.10.44:/GluReplica on /rhev/data-center/mnt/glusterSD/10.10.10.44:_GluReplica > type fuse.glusterfs (rw,relatime,user_id=0,group_i > d=0,default_permissions,allow_other,max_read=131072) > > I have glusterized 3 nodes: > > GluReplica > Volume ID: > ee686dfe-203a-4caa-a691-26353460cc48 > Volume Type: > Replicate (Arbiter) > Replica Count: > 2 + 1 > Number of Bricks: > 3 > Transport Types: > TCP, RDMA > Maximum no of snapshots: > 256 > Capacity: > 3.51 TiB total, 190.56 GiB used, 3.33 TiB free > > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > > > > > > > > > > ------------------------------ > > This email message is for the sole use of the intended recipient(s) and > may contain confidential information. Any unauthorized review, use, > disclosure or distribution is prohibited. If you are not the intended > recipient, please contact the sender by reply email and destroy all copies > of the original message. > ------------------------------ > > >