[ovirt-users] Hot to force glusterfs to use RDMA?

Thu Mar 2 12:06:55 UTC 2017

BTW RDMA is working as expected:
root at clei26 ~]# qperf clei22.vib  tcp_bw tcp_lat
tcp_bw:
    bw  =  475 MB/sec
tcp_lat:
    latency  =  52.8 us
[root at clei26 ~]#

thank you beforehand.
Arman.

On Thu, Mar 2, 2017 at 12:54 PM, Arman Khalatyan <arm2arm at gmail.com> wrote:

> just for reference:
>  gluster volume info
>
> Volume Name: GluReplica
> Type: Replicate
> Volume ID: ee686dfe-203a-4caa-a691-26353460cc48
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x (2 + 1) = 3
> Transport-type: tcp,rdma
> Bricks:
> Brick1: 10.10.10.44:/zclei22/01/glu
> Brick2: 10.10.10.42:/zclei21/01/glu
> Brick3: 10.10.10.41:/zclei26/01/glu (arbiter)
> Options Reconfigured:
> network.ping-timeout: 30
> server.allow-insecure: on
> storage.owner-gid: 36
> storage.owner-uid: 36
> cluster.data-self-heal-algorithm: full
> features.shard: on
> cluster.server-quorum-type: server
> cluster.quorum-type: auto
> network.remote-dio: enable
> cluster.eager-lock: enable
> performance.stat-prefetch: off
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
> performance.readdir-ahead: on
> nfs.disable: on
>
>
>
> [root at clei21 ~]# gluster volume status
> Status of volume: GluReplica
> Gluster process                             TCP Port  RDMA Port  Online
> Pid
> ------------------------------------------------------------
> ------------------
> Brick 10.10.10.44:/zclei22/01/glu           49158     49159      Y
> 15870
> Brick 10.10.10.42:/zclei21/01/glu           49156     49157      Y
> 17473
> Brick 10.10.10.41:/zclei26/01/glu           49153     49154      Y
> 18897
> Self-heal Daemon on localhost               N/A       N/A        Y
> 17502
> Self-heal Daemon on 10.10.10.41             N/A       N/A        Y
> 13353
> Self-heal Daemon on 10.10.10.44             N/A       N/A        Y
> 32745
>
> Task Status of Volume GluReplica
> ------------------------------------------------------------
> ------------------
> There are no active volume tasks
>
>
> On Thu, Mar 2, 2017 at 12:52 PM, Arman Khalatyan <arm2arm at gmail.com>
> wrote:
>
>> I am not able to mount with RDMA over cli....
>> Are there some volfile parameters needs to be tuned?
>> /usr/bin/mount  -t glusterfs  -o backup-volfile-servers=10.10.1
>> 0.44:10.10.10.42:10.10.10.41,transport=rdma 10.10.10.44:/GluReplica /mnt
>>
>> [2017-03-02 11:49:47.795511] I [MSGID: 100030] [glusterfsd.c:2454:main]
>> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.9
>> (args: /usr/sbin/glusterfs --volfile-server=10.10.10.44
>> --volfile-server=10.10.10.44 --volfile-server=10.10.10.42
>> --volfile-server=10.10.10.41 --volfile-server-transport=rdma
>> --volfile-id=/GluReplica.rdma /mnt)
>> [2017-03-02 11:49:47.812699] I [MSGID: 101190]
>> [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread
>> with index 1
>> [2017-03-02 11:49:47.825210] I [MSGID: 101190]
>> [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread
>> with index 2
>> [2017-03-02 11:49:47.828996] W [MSGID: 103071]
>> [rdma.c:4589:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event
>> channel creation failed [No such device]
>> [2017-03-02 11:49:47.829067] W [MSGID: 103055] [rdma.c:4896:init]
>> 0-GluReplica-client-2: Failed to initialize IB Device
>> [2017-03-02 11:49:47.829080] W [rpc-transport.c:354:rpc_transport_load]
>> 0-rpc-transport: 'rdma' initialization failed
>> [2017-03-02 11:49:47.829272] W [rpc-clnt.c:1070:rpc_clnt_connection_init]
>> 0-GluReplica-client-2: loading of new rpc-transport failed
>> [2017-03-02 11:49:47.829325] I [MSGID: 101053]
>> [mem-pool.c:641:mem_pool_destroy] 0-GluReplica-client-2: size=588 max=0
>> total=0
>> [2017-03-02 11:49:47.829371] I [MSGID: 101053]
>> [mem-pool.c:641:mem_pool_destroy] 0-GluReplica-client-2: size=124 max=0
>> total=0
>> [2017-03-02 11:49:47.829391] E [MSGID: 114022]
>> [client.c:2530:client_init_rpc] 0-GluReplica-client-2: failed to
>> initialize RPC
>> [2017-03-02 11:49:47.829413] E [MSGID: 101019] [xlator.c:433:xlator_init]
>> 0-GluReplica-client-2: Initialization of volume 'GluReplica-client-2'
>> failed, review your volfile again
>> [2017-03-02 11:49:47.829425] E [MSGID: 101066]
>> [graph.c:324:glusterfs_graph_init] 0-GluReplica-client-2: initializing
>> translator failed
>> [2017-03-02 11:49:47.829436] E [MSGID: 101176]
>> [graph.c:673:glusterfs_graph_activate] 0-graph: init failed
>> [2017-03-02 11:49:47.830003] W [glusterfsd.c:1327:cleanup_and_exit]
>> (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x3c1) [0x7f524c9dbeb1]
>> -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x172) [0x7f524c9d65d2]
>> -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f524c9d5b4b] ) 0-:
>> received signum (1), shutting down
>> [2017-03-02 11:49:47.830053] I [fuse-bridge.c:5794:fini] 0-fuse:
>> Unmounting '/mnt'.
>> [2017-03-02 11:49:47.831014] W [glusterfsd.c:1327:cleanup_and_exit]
>> (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f524b343dc5]
>> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f524c9d5cd5]
>> -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f524c9d5b4b] ) 0-:
>> received signum (15), shutting down
>> [2017-03-02 11:49:47.831014] W [glusterfsd.c:1327:cleanup_and_exit]
>> (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f524b343dc5]
>> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f524c9d5cd5]
>> -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7f524c9d5b4b] ) 0-:
>> received signum (15), shutting down
>>
>>
>>
>>
>> On Thu, Mar 2, 2017 at 12:11 PM, Sahina Bose <sabose at redhat.com> wrote:
>>
>>> You will need to pass additional mount options while creating the
>>> storage domain (transport=rdma)
>>>
>>>
>>> Please let us know if this works.
>>>
>>> On Thu, Mar 2, 2017 at 2:42 PM, Arman Khalatyan <arm2arm at gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> Are there way to force the connections over RDMA only?
>>>> If I check host mounts I cannot see rdma mount option:
>>>>  mount -l| grep gluster
>>>> 10.10.10.44:/GluReplica on /rhev/data-center/mnt/glusterSD/10.10.10.44:_GluReplica
>>>> type fuse.glusterfs (rw,relatime,user_id=0,group_i
>>>> d=0,default_permissions,allow_other,max_read=131072)
>>>>
>>>> I have glusterized 3 nodes:
>>>>
>>>> GluReplica
>>>> Volume ID:
>>>> ee686dfe-203a-4caa-a691-26353460cc48
>>>> Volume Type:
>>>> Replicate (Arbiter)
>>>> Replica Count:
>>>> 2 + 1
>>>> Number of Bricks:
>>>> 3
>>>> Transport Types:
>>>> TCP, RDMA
>>>> Maximum no of snapshots:
>>>> 256
>>>> Capacity:
>>>> 3.51 TiB total, 190.56 GiB used, 3.33 TiB free
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170302/a2bfae53/attachment.html>