<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hi Arman,<br>
</p>
<br>
<div class="moz-cite-prefix">On 03/03/2017 12:27 PM, Arman Khalatyan
wrote:<br>
</div>
<blockquote
cite="mid:CAAqDm6Zb=DiWsrCS2Lp8vC=_Or5NFONoH-Ou83E5P6LssHXmEg@mail.gmail.com"
type="cite">
<div dir="auto">Dear Deepak, thank you for the hints, which
gluster are you using?
<div dir="auto">As you can see from my previous email that the
RDMA connection tested with qperf. It is working as expected.
In my case the clients are servers as well, they are hosts for
the ovirt. Disabling selinux is nor recommended by ovirt, but
i will give a try.</div>
</div>
</blockquote>
<br>
Gluster use IPoIB as mentioned by Deepak. So qperf with default
options may not be a good choice to test IPoIB. Because it will
fallback to any link available between the mentioned server and
client. You can force this behavior, please refer the link [1]. <br>
<br>
In addition to that, Can you please provide your gluster version,
glusterd logs and brick logs. Because since it complains about
absence of the device, mostly it could be a set up issue. Otherwise
it could have been a permission denied error, I'm not completely
ruling out the possibility of selinux preventing the creation if IB
channel. We had this issue in rhel which is fixed in 7.2 [2] .<br>
<br>
<br>
[1] :
<a class="moz-txt-link-freetext" href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/sec-Testing_an_RDMA_network_after_IPoIB_is_configured.html">https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/sec-Testing_an_RDMA_network_after_IPoIB_is_configured.html</a><br>
[2] : <a class="moz-txt-link-freetext" href="https://bugzilla.redhat.com/show_bug.cgi?id=1386620">https://bugzilla.redhat.com/show_bug.cgi?id=1386620</a><br>
<br>
Regards<br>
Rafi KC<br>
<br>
<br>
<blockquote
cite="mid:CAAqDm6Zb=DiWsrCS2Lp8vC=_Or5NFONoH-Ou83E5P6LssHXmEg@mail.gmail.com"
type="cite">
<div class="gmail_extra"><br>
<div class="gmail_quote">Am 03.03.2017 7:50 vorm. schrieb
"Deepak Naidu" <<a moz-do-not-send="true"
href="mailto:dnaidu@nvidia.com">dnaidu@nvidia.com</a>>:<br
type="attribution">
<blockquote class="quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-US">
<div class="m_8813890542634899653WordSection1">
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I
have been testing glusterfs over RDMA & below is
the command I use. Reading up the logs, it looks
like your IB(InfiniBand) device is not being
initialized. I am not sure if u have an issue on the
client IB or the storage server IB. Also have you
configured ur IB devices correctly. I am using
IPoIB.</span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Can
you check your firewall, disable selinux, I think,
you might have checked it already ?</span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p>
<p class="MsoNormal"><b><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">mount
-t glusterfs -o transport=rdma storageN1:/vol0
/mnt/vol0</span></b></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p>
<p class="m_8813890542634899653MsoListParagraph"><span
style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><b><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The
below error seems if you have issue starting your
volume. I had issue, when my transport was set to
tcp,rdma. I had to force start my volume. If I had
set it only to tcp on the volume, the volume would
start easily.</span></b></p>
<div class="quoted-text">
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="background:yellow">[2017-03-02
11:49:47.829391] E [MSGID: 114022]
[client.c:2530:client_init_<wbr>rpc]
0-GluReplica-client-2: failed to initialize RPC<br>
[2017-03-02 11:49:47.829413] E [MSGID: 101019]
[xlator.c:433:xlator_init] 0-GluReplica-client-2:
Initialization of volume 'GluReplica-client-2'
failed, review your volfile again<br>
[2017-03-02 11:49:47.829425] E [MSGID: 101066]
[graph.c:324:glusterfs_graph_<wbr>init]
0-GluReplica-client-2: initializing translator
failed<br>
[2017-03-02 11:49:47.829436] E [MSGID: 101176]
[graph.c:673:glusterfs_graph_<wbr>activate]
0-graph: init failed</span><br>
<br>
<span style="background:yellow"></span></p>
<p class="MsoNormal"><span style="background:yellow"> </span></p>
</div>
<p class="m_8813890542634899653MsoListParagraph"><span
style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span
style="font:7.0pt "Times New Roman"">
</span></span></span><b><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The
below error seems if you have issue with IB
device. If not configured properly.</span></b></p>
<div class="quoted-text">
<p class="MsoNormal"><span style="background:yellow"> </span></p>
<p class="MsoNormal"><span style="background:yellow">[2017-03-02
11:49:47.828996] W [MSGID: 103071]
[rdma.c:4589:__gf_rdma_ctx_<wbr>create]
0-rpc-transport/rdma: rdma_cm event channel
creation failed [No such device]<br>
[2017-03-02 11:49:47.829067] W [MSGID: 103055]
[rdma.c:4896:init] 0-GluReplica-client-2: Failed
to initialize IB Device<br>
[2017-03-02 11:49:47.829080] W
[rpc-transport.c:354:rpc_<wbr>transport_load]
0-rpc-transport: 'rdma' initialization failed</span><br>
<br>
<span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p>
</div>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">--</span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Deepak</span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p>
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a moz-do-not-send="true"
href="mailto:gluster-users-bounces@gluster.org"
target="_blank">gluster-users-bounces@gluster.<wbr>org</a>
[mailto:<a moz-do-not-send="true"
href="mailto:gluster-users-bounces@gluster.org"
target="_blank">gluster-users-bounces@<wbr>gluster.org</a>]
<b>On Behalf Of </b>Sahina Bose<br>
<b>Sent:</b> Thursday, March 02, 2017 10:26 PM<br>
<b>To:</b> Arman Khalatyan; <a
moz-do-not-send="true"
href="mailto:gluster-users@gluster.org"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a></a>;
Rafi Kavungal Chundattu Parambil<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [Gluster-users] [ovirt-users]
Hot to force glusterfs to use RDMA?</span></p>
<div class="elided-text">
<p class="MsoNormal"> </p>
<div>
<p class="MsoNormal">[Adding gluster users to help
with error]<br>
<br>
[2017-03-02 11:49:47.828996] W [MSGID: 103071]
[rdma.c:4589:__gf_rdma_ctx_<wbr>create]
0-rpc-transport/rdma: rdma_cm event channel
creation failed [No such device]</p>
</div>
<div>
<p class="MsoNormal"> </p>
<div>
<p class="MsoNormal">On Thu, Mar 2, 2017 at 5:36
PM, Arman Khalatyan <<a
moz-do-not-send="true"
href="mailto:arm2arm@gmail.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:arm2arm@gmail.com">arm2arm@gmail.com</a></a>>
wrote:</p>
<div>
<div>
<div>
<p class="MsoNormal"
style="margin-bottom:12.0pt">BTW RDMA is
working as expected:<br>
root@clei26 ~]# qperf clei22.vib tcp_bw
tcp_lat<br>
tcp_bw:<br>
bw = 475 MB/sec<br>
tcp_lat:<br>
latency = 52.8 us<br>
[root@clei26 ~]# </p>
</div>
<p class="MsoNormal">thank you beforehand.</p>
</div>
<p class="MsoNormal"
style="margin-bottom:12.0pt"><span
class="m_8813890542634899653hoenzb"><span
style="color:#888888">Arman.</span></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> </p>
<div>
<p class="MsoNormal">On Thu, Mar 2, 2017
at 12:54 PM, Arman Khalatyan <<a
moz-do-not-send="true"
href="mailto:arm2arm@gmail.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:arm2arm@gmail.com">arm2arm@gmail.com</a></a>>
wrote:</p>
<div>
<p class="MsoNormal"
style="margin-bottom:12.0pt">just for
reference:<br>
gluster volume info <br>
<br>
Volume Name: GluReplica<br>
Type: Replicate<br>
Volume ID: ee686dfe-203a-4caa-a691-<wbr>26353460cc48<br>
Status: Started<br>
Snapshot Count: 0<br>
Number of Bricks: 1 x (2 + 1) = 3<br>
Transport-type: tcp,rdma<br>
Bricks:<br>
Brick1: 10.10.10.44:/zclei22/01/glu<br>
Brick2: 10.10.10.42:/zclei21/01/glu<br>
Brick3: 10.10.10.41:/zclei26/01/glu
(arbiter)<br>
Options Reconfigured:<br>
network.ping-timeout: 30<br>
server.allow-insecure: on<br>
storage.owner-gid: 36<br>
storage.owner-uid: 36<br>
cluster.data-self-heal-<wbr>algorithm:
full<br>
features.shard: on<br>
cluster.server-quorum-type: server<br>
cluster.quorum-type: auto<br>
network.remote-dio: enable<br>
cluster.eager-lock: enable<br>
performance.stat-prefetch: off<br>
performance.io-cache: off<br>
performance.read-ahead: off<br>
performance.quick-read: off<br>
performance.readdir-ahead: on<br>
nfs.disable: on<br>
<br>
<br>
<br>
[root@clei21 ~]# gluster volume status
<br>
Status of volume: GluReplica<br>
Gluster process <wbr>
TCP Port RDMA Port Online Pid<br>
------------------------------<wbr>------------------------------<wbr>------------------<br>
Brick 10.10.10.44:/zclei22/01/glu <wbr>
49158 49159 Y 15870<br>
Brick 10.10.10.42:/zclei21/01/glu <wbr>
49156 49157 Y 17473<br>
Brick 10.10.10.41:/zclei26/01/glu <wbr>
49153 49154 Y 18897<br>
Self-heal Daemon on
localhost N/A
N/A Y 17502<br>
Self-heal Daemon on
10.10.10.41 N/A
N/A Y 13353<br>
Self-heal Daemon on
10.10.10.44 N/A
N/A Y 32745<br>
<br>
Task Status of Volume GluReplica<br>
------------------------------<wbr>------------------------------<wbr>------------------<br>
There are no active volume tasks</p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> </p>
<div>
<p class="MsoNormal">On Thu, Mar
2, 2017 at 12:52 PM, Arman
Khalatyan <<a
moz-do-not-send="true"
href="mailto:arm2arm@gmail.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:arm2arm@gmail.com">arm2arm@gmail.com</a></a>>
wrote:</p>
<div>
<div>
<p class="MsoNormal">I am not
able to mount with RDMA over
cli....</p>
</div>
<p class="MsoNormal"
style="margin-bottom:12.0pt">Are
there some volfile parameters
needs to be tuned?<br>
/usr/bin/mount -t glusterfs
-o
backup-volfile-servers=10.10.<wbr>10.44:10.10.10.42:10.10.10.41,<wbr>transport=rdma
10.10.10.44:/GluReplica /mnt<br>
<br>
[2017-03-02 11:49:47.795511] I
[MSGID: 100030]
[glusterfsd.c:2454:main]
0-/usr/sbin/glusterfs: Started
running /usr/sbin/glusterfs
version 3.8.9 (args:
/usr/sbin/glusterfs
--volfile-server=10.10.10.44
--volfile-server=10.10.10.44
--volfile-server=10.10.10.42
--volfile-server=10.10.10.41
--volfile-server-transport=<wbr>rdma
--volfile-id=/GluReplica.rdma
/mnt)<br>
[2017-03-02 11:49:47.812699] I
[MSGID: 101190]
[event-epoll.c:628:event_<wbr>dispatch_epoll_worker]
0-epoll: Started thread with
index 1<br>
[2017-03-02 11:49:47.825210] I
[MSGID: 101190]
[event-epoll.c:628:event_<wbr>dispatch_epoll_worker]
0-epoll: Started thread with
index 2<br>
[2017-03-02 11:49:47.828996] W
[MSGID: 103071]
[rdma.c:4589:__gf_rdma_ctx_<wbr>create]
0-rpc-transport/rdma: rdma_cm
event channel creation failed
[No such device]<br>
[2017-03-02 11:49:47.829067] W
[MSGID: 103055]
[rdma.c:4896:init]
0-GluReplica-client-2: Failed
to initialize IB Device<br>
[2017-03-02 11:49:47.829080] W
[rpc-transport.c:354:rpc_<wbr>transport_load]
0-rpc-transport: 'rdma'
initialization failed<br>
[2017-03-02 11:49:47.829272] W
[rpc-clnt.c:1070:rpc_clnt_<wbr>connection_init]
0-GluReplica-client-2: loading
of new rpc-transport failed<br>
[2017-03-02 11:49:47.829325] I
[MSGID: 101053]
[mem-pool.c:641:mem_pool_<wbr>destroy]
0-GluReplica-client-2:
size=588 max=0 total=0<br>
[2017-03-02 11:49:47.829371] I
[MSGID: 101053]
[mem-pool.c:641:mem_pool_<wbr>destroy]
0-GluReplica-client-2:
size=124 max=0 total=0<br>
[2017-03-02 11:49:47.829391] E
[MSGID: 114022]
[client.c:2530:client_init_<wbr>rpc]
0-GluReplica-client-2: failed
to initialize RPC<br>
[2017-03-02 11:49:47.829413] E
[MSGID: 101019]
[xlator.c:433:xlator_init]
0-GluReplica-client-2:
Initialization of volume
'GluReplica-client-2' failed,
review your volfile again<br>
[2017-03-02 11:49:47.829425] E
[MSGID: 101066]
[graph.c:324:glusterfs_graph_<wbr>init]
0-GluReplica-client-2:
initializing translator failed<br>
[2017-03-02 11:49:47.829436] E
[MSGID: 101176]
[graph.c:673:glusterfs_graph_<wbr>activate]
0-graph: init failed<br>
[2017-03-02 11:49:47.830003] W
[glusterfsd.c:1327:cleanup_<wbr>and_exit]
(-->/usr/sbin/glusterfs(mgmt_<wbr>getspec_cbk+0x3c1) [0x7f524c9dbeb1]
-->/usr/sbin/glusterfs(<wbr>glusterfs_process_volfp+0x172)
[0x7f524c9d65d2]
-->/usr/sbin/glusterfs(<wbr>cleanup_and_exit+0x6b)
[0x7f524c9d5b4b] ) 0-:
received signum (1), shutting
down<br>
[2017-03-02 11:49:47.830053] I
[fuse-bridge.c:5794:fini]
0-fuse: Unmounting '/mnt'.<br>
[2017-03-02 11:49:47.831014] W
[glusterfsd.c:1327:cleanup_<wbr>and_exit]
(-->/lib64/libpthread.so.0(+<wbr>0x7dc5) [0x7f524b343dc5]
-->/usr/sbin/glusterfs(<wbr>glusterfs_sigwaiter+0xe5)
[0x7f524c9d5cd5]
-->/usr/sbin/glusterfs(<wbr>cleanup_and_exit+0x6b)
[0x7f524c9d5b4b] ) 0-:
received signum (15), shutting
down<br>
[2017-03-02 11:49:47.831014] W
[glusterfsd.c:1327:cleanup_<wbr>and_exit]
(-->/lib64/libpthread.so.0(+<wbr>0x7dc5) [0x7f524b343dc5]
-->/usr/sbin/glusterfs(<wbr>glusterfs_sigwaiter+0xe5)
[0x7f524c9d5cd5]
-->/usr/sbin/glusterfs(<wbr>cleanup_and_exit+0x6b)
[0x7f524c9d5b4b] ) 0-:
received signum (15), shutting
down<br>
<br>
<br>
</p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> </p>
<div>
<p class="MsoNormal">On
Thu, Mar 2, 2017 at
12:11 PM, Sahina Bose
<<a
moz-do-not-send="true"
href="mailto:sabose@redhat.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:sabose@redhat.com">sabose@redhat.com</a></a>>
wrote:</p>
<div>
<div>
<p class="MsoNormal"
style="margin-bottom:12.0pt">You
will need to pass
additional mount
options while
creating the storage
domain
(transport=rdma)<br>
<br>
</p>
</div>
<div>
<p class="MsoNormal">Please
let us know if this
works.</p>
</div>
</div>
<div>
<p class="MsoNormal"> </p>
<div>
<div>
<div>
<p
class="MsoNormal">On
Thu, Mar 2, 2017
at 2:42 PM,
Arman Khalatyan
<<a
moz-do-not-send="true"
href="mailto:arm2arm@gmail.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:arm2arm@gmail.com">arm2arm@gmail.com</a></a>>
wrote:</p>
</div>
</div>
<blockquote
style="border:none;border-left:solid
#cccccc
1.0pt;padding:0in
0in 0in
6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p
class="MsoNormal">Hi,</p>
</div>
<div>
<p
class="MsoNormal">Are
there way to
force the
connections
over RDMA
only?</p>
</div>
<div>
<p
class="MsoNormal"
style="margin-bottom:12.0pt">If I check host mounts I cannot see rdma
mount option:<br>
mount -l|
grep gluster<br>
10.10.10.44:/GluReplica on /rhev/data-center/mnt/<wbr>glusterSD/10.10.10.44:_<wbr>GluReplica
type
fuse.glusterfs
(rw,relatime,user_id=0,group_<wbr>id=0,default_permissions,<wbr>allow_other,max_read=131072)</p>
</div>
<p
class="MsoNormal">I
have
glusterized 3
nodes: <br>
<br>
GluReplica<br>
Volume ID:<br>
ee686dfe-203a-4caa-a691-<wbr>26353460cc48<br>
Volume Type:<br>
Replicate
(Arbiter)<br>
Replica Count:<br>
2 + 1<br>
Number of
Bricks:<br>
3<br>
Transport
Types:<br>
TCP, RDMA<br>
Maximum no of
snapshots:<br>
256<br>
Capacity:<br>
3.51 TiB
total, 190.56
GiB used, 3.33
TiB free</p>
</div>
<p
class="MsoNormal"> </p>
</div>
</div>
<p class="MsoNormal"
style="margin-bottom:12.0pt">______________________________<wbr>_________________<br>
Users mailing list<br>
<a
moz-do-not-send="true"
href="mailto:Users@ovirt.org" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a></a><br>
<a
moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank"><a class="moz-txt-link-freetext" href="http://lists.ovirt.org/">http://lists.ovirt.org/</a><wbr>mailman/listinfo/users</a></p>
</blockquote>
</div>
<p class="MsoNormal"> </p>
</div>
</div>
<p class="MsoNormal"> </p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"> </p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"> </p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"> </p>
</div>
</div>
</div>
<div>
<hr>
</div>
<div>This email message is for the sole use of the
intended recipient(s) and may contain confidential
information. Any unauthorized review, use, disclosure
or distribution is prohibited. If you are not the
intended recipient, please contact the sender by reply
email and destroy all copies of the original message. </div>
<div>
<hr>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>