<div dir="ltr"><div><div>BTW RDMA is working as expected:<br>root@clei26 ~]# qperf clei22.vib tcp_bw tcp_lat<br>tcp_bw:<br> bw = 475 MB/sec<br>tcp_lat:<br> latency = 52.8 us<br>[root@clei26 ~]# <br><br></div>thank you beforehand.<br></div>Arman.<br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 2, 2017 at 12:54 PM, Arman Khalatyan <span dir="ltr"><<a href="mailto:arm2arm@gmail.com" target="_blank">arm2arm@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">just for reference:<br> gluster volume info <br> <br>Volume Name: GluReplica<br>Type: Replicate<br>Volume ID: ee686dfe-203a-4caa-a691-<wbr>26353460cc48<br>Status: Started<br>Snapshot Count: 0<br>Number of Bricks: 1 x (2 + 1) = 3<br>Transport-type: tcp,rdma<br>Bricks:<br>Brick1: 10.10.10.44:/zclei22/01/glu<br>Brick2: 10.10.10.42:/zclei21/01/glu<br>Brick3: 10.10.10.41:/zclei26/01/glu (arbiter)<br>Options Reconfigured:<br>network.ping-timeout: 30<br>server.allow-insecure: on<br>storage.owner-gid: 36<br>storage.owner-uid: 36<br>cluster.data-self-heal-<wbr>algorithm: full<br>features.shard: on<br>cluster.server-quorum-type: server<br>cluster.quorum-type: auto<br>network.remote-dio: enable<br>cluster.eager-lock: enable<br>performance.stat-prefetch: off<br>performance.io-cache: off<br>performance.read-ahead: off<br>performance.quick-read: off<br>performance.readdir-ahead: on<br>nfs.disable: on<br><br><br><br>[root@clei21 ~]# gluster volume status <br>Status of volume: GluReplica<br>Gluster process <wbr> TCP Port RDMA Port Online Pid<br>------------------------------<wbr>------------------------------<wbr>------------------<br>Brick 10.10.10.44:/zclei22/01/glu <wbr> 49158 49159 Y 15870<br>Brick 10.10.10.42:/zclei21/01/glu <wbr> 49156 49157 Y 17473<br>Brick 10.10.10.41:/zclei26/01/glu <wbr> 49153 49154 Y 18897<br>Self-heal Daemon on localhost N/A N/A Y 17502<br>Self-heal Daemon on 10.10.10.41 N/A N/A Y 13353<br>Self-heal Daemon on 10.10.10.44 N/A N/A Y 32745<br> <br>Task Status of Volume GluReplica<br>------------------------------<wbr>------------------------------<wbr>------------------<br>There are no active volume tasks<br><br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 2, 2017 at 12:52 PM, Arman Khalatyan <span dir="ltr"><<a href="mailto:arm2arm@gmail.com" target="_blank">arm2arm@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I am not able to mount with RDMA over cli....<br></div>Are there some volfile parameters needs to be tuned?<br>/usr/bin/mount -t glusterfs -o backup-volfile-servers=10.10.1<wbr>0.44:10.10.10.42:10.10.10.41,t<wbr>ransport=rdma 10.10.10.44:/GluReplica /mnt<br><br>[2017-03-02 11:49:47.795511] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.9 (args: /usr/sbin/glusterfs --volfile-server=10.10.10.44 --volfile-server=10.10.10.44 --volfile-server=10.10.10.42 --volfile-server=10.10.10.41 --volfile-server-transport=rdm<wbr>a --volfile-id=/GluReplica.rdma /mnt)<br>[2017-03-02 11:49:47.812699] I [MSGID: 101190] [event-epoll.c:628:event_dispa<wbr>tch_epoll_worker] 0-epoll: Started thread with index 1<br>[2017-03-02 11:49:47.825210] I [MSGID: 101190] [event-epoll.c:628:event_dispa<wbr>tch_epoll_worker] 0-epoll: Started thread with index 2<br>[2017-03-02 11:49:47.828996] W [MSGID: 103071] [rdma.c:4589:__gf_rdma_ctx_cre<wbr>ate] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]<br>[2017-03-02 11:49:47.829067] W [MSGID: 103055] [rdma.c:4896:init] 0-GluReplica-client-2: Failed to initialize IB Device<br>[2017-03-02 11:49:47.829080] W [rpc-transport.c:354:rpc_trans<wbr>port_load] 0-rpc-transport: 'rdma' initialization failed<br>[2017-03-02 11:49:47.829272] W [rpc-clnt.c:1070:rpc_clnt_conn<wbr>ection_init] 0-GluReplica-client-2: loading of new rpc-transport failed<br>[2017-03-02 11:49:47.829325] I [MSGID: 101053] [mem-pool.c:641:mem_pool_destr<wbr>oy] 0-GluReplica-client-2: size=588 max=0 total=0<br>[2017-03-02 11:49:47.829371] I [MSGID: 101053] [mem-pool.c:641:mem_pool_destr<wbr>oy] 0-GluReplica-client-2: size=124 max=0 total=0<br>[2017-03-02 11:49:47.829391] E [MSGID: 114022] [client.c:2530:client_init_rpc<wbr>] 0-GluReplica-client-2: failed to initialize RPC<br>[2017-03-02 11:49:47.829413] E [MSGID: 101019] [xlator.c:433:xlator_init] 0-GluReplica-client-2: Initialization of volume 'GluReplica-client-2' failed, review your volfile again<br>[2017-03-02 11:49:47.829425] E [MSGID: 101066] [graph.c:324:glusterfs_graph_i<wbr>nit] 0-GluReplica-client-2: initializing translator failed<br>[2017-03-02 11:49:47.829436] E [MSGID: 101176] [graph.c:673:glusterfs_graph_a<wbr>ctivate] 0-graph: init failed<br>[2017-03-02 11:49:47.830003] W [glusterfsd.c:1327:cleanup_and<wbr>_exit] (-->/usr/sbin/glusterfs(mgmt_g<wbr>etspec_cbk+0x3c1) [0x7f524c9dbeb1] -->/usr/sbin/glusterfs(gluster<wbr>fs_process_volfp+0x172) [0x7f524c9d65d2] -->/usr/sbin/glusterfs(cleanup<wbr>_and_exit+0x6b) [0x7f524c9d5b4b] ) 0-: received signum (1), shutting down<br>[2017-03-02 11:49:47.830053] I [fuse-bridge.c:5794:fini] 0-fuse: Unmounting '/mnt'.<br>[2017-03-02 11:49:47.831014] W [glusterfsd.c:1327:cleanup_and<wbr>_exit] (-->/lib64/libpthread.so.0(+0x<wbr>7dc5) [0x7f524b343dc5] -->/usr/sbin/glusterfs(gluster<wbr>fs_sigwaiter+0xe5) [0x7f524c9d5cd5] -->/usr/sbin/glusterfs(cleanup<wbr>_and_exit+0x6b) [0x7f524c9d5b4b] ) 0-: received signum (15), shutting down<br>[2017-03-02 11:49:47.831014] W [glusterfsd.c:1327:cleanup_and<wbr>_exit] (-->/lib64/libpthread.so.0(+0x<wbr>7dc5) [0x7f524b343dc5] -->/usr/sbin/glusterfs(gluster<wbr>fs_sigwaiter+0xe5) [0x7f524c9d5cd5] -->/usr/sbin/glusterfs(cleanup<wbr>_and_exit+0x6b) [0x7f524c9d5b4b] ) 0-: received signum (15), shutting down<br><br><br><br></div><div class="m_-2659542588269776057HOEnZb"><div class="m_-2659542588269776057h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 2, 2017 at 12:11 PM, Sahina Bose <span dir="ltr"><<a href="mailto:sabose@redhat.com" target="_blank">sabose@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>You will need to pass additional mount options while creating the storage domain (transport=rdma)<br><br><br></div><div>Please let us know if this works.<br></div><span style="color:rgb(231,76,60);font-family:consolas,"andale mono wt","andale mono","lucida console","lucida sans typewriter","dejavu sans mono","bitstream vera sans mono","liberation mono","nimbus mono l",monaco,"courier new",courier,monospace;font-size:10.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;line-height:24px;text-align:left;text-indent:0px;text-transform:none;white-space:pre-wrap;word-spacing:0px;display:inline;float:none;background-color:rgb(255,255,255)"></span></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_-2659542588269776057m_6563949903404183567h5">On Thu, Mar 2, 2017 at 2:42 PM, Arman Khalatyan <span dir="ltr"><<a href="mailto:arm2arm@gmail.com" target="_blank">arm2arm@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_-2659542588269776057m_6563949903404183567h5"><div dir="ltr"><div>Hi,<br></div><div>Are there way to force the connections over RDMA only?<br></div><div>If I check host mounts I cannot see rdma mount option:<br> mount -l| grep gluster<br>10.10.10.44:/GluReplica on /rhev/data-center/mnt/glusterS<wbr>D/10.10.10.44:_GluReplica type fuse.glusterfs (rw,relatime,user_id=0,group_i<wbr>d=0,default_permissions,allow_<wbr>other,max_read=131072)<br><br></div>I have glusterized 3 nodes: <br><br>GluReplica<br>Volume ID:<br>ee686dfe-203a-4caa-a691-263534<wbr>60cc48<br>Volume Type:<br>Replicate (Arbiter)<br>Replica Count:<br>2 + 1<br>Number of Bricks:<br>3<br>Transport Types:<br>TCP, RDMA<br>Maximum no of snapshots:<br>256<br>Capacity:<br>3.51 TiB total, 190.56 GiB used, 3.33 TiB free<br></div>
<br></div></div>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>