1:12 p.m.
I am trying to setup gluster using Infiniband and mounting it in RDMA mode. I have created
a network hook so that it configures the interfaces as MTU=65520 and CONNECTED_MODE=yes. I
have a second server doing NFS with RDMA over Infiniband with some VMs on it. When I try
and transfer files to the gluster storage it is taking a while and I am seeing the message
“VDSM {hostname} command GetGlusterVolumeHealInfoVDS failed: Message timeout which can be
caused by communication issues” This is usually followed by “Host {hostname} is not
responding. It will stay in Connecting state for a grace period of 60 seconds and after
that an attempt to fence the host will be issued.” I just installed the Infiniband
hardware. The switch is a Qlogic 12200-18 with QLE7340 single port Infiniband cards in
each of the 3 servers. The error message varies on which of the 3 nodes it comes from.
Each of the 3 servers is running the opensm service.
2:20 p.m.
try to leave datagramm mode, do not change mtu. <http://mtu.it> it looks
like gluster is connected over the tcp/ipoib. you might get packets drops
with mtu> 2k. as i remember you should tune your ib switch for the mtu
size,at least on the mellanox managed switch.
if the gluster connected over rdma it should not use any of tcp settings.
Is your mount -l shows that the gluster mounted with .rdma?
btw opensm should run only once, the others are going to stay in the
standby mode.
<housemouse(a)squeakz.net> schrieb am Mo., 13. Aug. 2018, 20:13:
I am trying to setup gluster using Infiniband and mounting it in
RDMA
mode. I have created a network hook so that it configures the interfaces as
MTU=65520 and CONNECTED_MODE=yes. I have a second server doing NFS with
RDMA over Infiniband with some VMs on it. When I try and transfer files to
the gluster storage it is taking a while and I am seeing the message “VDSM
{hostname} command GetGlusterVolumeHealInfoVDS failed: Message timeout
which can be caused by communication issues” This is usually followed by
“Host {hostname} is not responding. It will stay in Connecting state for a
grace period of 60 seconds and after that an attempt to fence the host will
be issued.” I just installed the Infiniband hardware. The switch is a
Qlogic 12200-18 with QLE7340 single port Infiniband cards in each of the 3
servers. The error message varies on which of the 3 nodes it comes from.
Each of the 3 servers is running the opensm service.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KGR2CL7M34I...
3:08 p.m.
New subject: Connection issues when using gluster + infiniband + RDMA
My gluster volume is mounted with *.rdma. I cannot make changes to the switch as it
doesn’t have any management to it as it is unmanaged. I can transfer files and run VMs
from the gluster volume. I just receive these messages. I am using IBoIP. I have no
problems doing a NFS over RDMA share from a second server that is also hooked up to the
Infiniband switch.
From: Arman Khalatyan <arm2arm(a)gmail.com>
Date: Monday, August 13, 2018 at 2:21 PM
To: Matthew Hoberg <housemouse(a)squeakz.net>
Cc: users <users(a)ovirt.org>
Subject: Re: [ovirt-users] Connection issues when using gluster + infiniband + RDMA
try to leave datagramm mode, do not change mtu.<http://mtu.it> it looks like gluster
is connected over the tcp/ipoib. you might get packets drops with mtu> 2k. as i
remember you should tune your ib switch for the mtu size,at least on the mellanox managed
switch.
if the gluster connected over rdma it should not use any of tcp settings.
Is your mount -l shows that the gluster mounted with .rdma?
btw opensm should run only once, the others are going to stay in the standby mode.
<housemouse@squeakz.net<mailto:housemouse@squeakz.net>> schrieb am Mo., 13.
Aug. 2018, 20:13:
I am trying to setup gluster using Infiniband and mounting it in RDMA mode. I have created
a network hook so that it configures the interfaces as MTU=65520 and CONNECTED_MODE=yes. I
have a second server doing NFS with RDMA over Infiniband with some VMs on it. When I try
and transfer files to the gluster storage it is taking a while and I am seeing the message
“VDSM {hostname} command GetGlusterVolumeHealInfoVDS failed: Message timeout which can be
caused by communication issues” This is usually followed by “Host {hostname} is not
responding. It will stay in Connecting state for a grace period of 60 seconds and after
that an attempt to fence the host will be issued.” I just installed the Infiniband
hardware. The switch is a Qlogic 12200-18 with QLE7340 single port Infiniband cards in
each of the 3 servers. The error message varies on which of the 3 nodes it comes from.
Each of the 3 servers is running the opensm service.
_______________________________________________
Users mailing list -- users@ovirt.org<mailto:users@ovirt.org>
To unsubscribe send an email to
users-leave@ovirt.org<mailto:users-leave@ovirt.org>
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KGR2CL7M34I...
2091
days inactive
2091
days old
2 comments
3 participants
participants (3)
-
Arman Khalatyan -
housemouse＠squeakz.net -
Matthew Hoberg