12:17 a.m.
Hello,
I want to use my Infiniband Network for Storage Access, so I created a new network and
tried to assign it to a host. When I do this, I get a success message but it's
actually not assigned to my Infiniband NIC.
I'm running the latest CentOS with the latest Mellanox OFED drivers and multiple
Subnet Managers are running on the network. The NIC is a Mellanox ConnectX-2 VPI, set to
Infiniband mode.
I didnt't knew which logs to send, so when you need to look into a log, just tell me
which one you need.
I hope you can help me out.
Roman Meusch
1:08 a.m.
Hello fellow Infiniband-er! I have a 3 node cluster with Connectx-3 cards under CentosOS
and it seems to be working well. One thing I didn't realhotize is once I assigned it
to the host, I had to edit it and assign the network manually, otherwise it left it has
DHCP, it never got an address (for obvious reasons) and it stopped working. When I
originally set the servers up, I installed the infiniband support packages and I see them
in cockpit before I Actually assigned them.
1:25 a.m.
Hello, thanks for the answer!
I was missing that, I thought it would pick up the previously configured IP I set in
cockpit before I added the host!
But I also found out the main issue: it was set as a VM network. But I don't know if
that's meant to be like this or this is just a bug.
Now it's added to the host but I can't send traffic through it, not even a ping!
Have you some idea why it's still not working?
Roman
2:19 a.m.
Hello,
now everything is working.
For some reason I had to reconfigure the NIC on my engine machine.
Roman
2:55 a.m.
You were just like me, I also thought it would pick up settings from cockpit, but once I
realized you had to configure the network again when adding to the host, everything sort
of fell into line. I would make sure and monitor the networks so you can see the traffic
going through the infiniband and not the ethernet.
10:42 a.m.
Are you guys running IPoIB for Infiniband Storage? Is it true IB mode? Or Ethernet over
IB?
If using NFS on top of it, RDMA is working
with NFSoRDMA?
I’m really curious about your setup.
Thanks,
Sent from my iPhone
> On 10 Feb 2019, at 23:55, darkytoo(a)gmail.com wrote:
>
> You were just like me, I also thought it would pick up settings from cockpit, but
once I realized you had to configure the network again when adding to the host, everything
sort of fell into line. I would make sure and monitor the networks so you can see the
traffic going through the infiniband and not the ethernet.
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WC6W7KRXDEN...
7:36 p.m.
Darkytoo have you received my message.
I’m really curious about your setup using Infiniband.
Thanks,
On 14 Feb 2019, at 07:42, Vinícius Ferrão
<ferrao(a)versatushpc.com.br> wrote:
Are you guys running IPoIB for Infiniband Storage? Is it true IB mode? Or Ethernet over
IB?
If using NFS on top of it, RDMA is working
with NFSoRDMA?
I’m really curious about your setup.
Thanks,
Sent from my iPhone
> On 10 Feb 2019, at 23:55, darkytoo(a)gmail.com wrote:
>
> You were just like me, I also thought it would pick up settings from cockpit, but
once I realized you had to configure the network again when adding to the host, everything
sort of fell into line. I would make sure and monitor the networks so you can see the
traffic going through the infiniband and not the ethernet.
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WC6W7KRXDEN...
1:36 a.m.
Hi Vinicius, i'm sorry, I thought I had replied already.
I'm mostly using IPoIB, but gluster is using some over RDMA. I'm using it
a few different ways,
5 IB nodes, all have dual port ConnectX-3 cards @ 40GB
2 nodes are running windows, one port on each is used for a sync channel
for starwind Vsan to mirror ~20tb of date. The second ports are using for
iSCS sharing SMB fiile shares for other IB nodes.
3 nides are running CentOS 7 and ovirt. One of the ports on each are used
for gluster synchronzaton, gluster is set to use both IPoIB & RDMA. Then
the other ports are using for migratons, iSCSI sharing, and a backup admin
channel.
On Fri, Mar 1, 2019 at 1:36 PM Vinícius Ferrão <ferrao(a)versatushpc.com.br>
wrote:
Darkytoo have you received my message.
I’m really curious about your setup using Infiniband.
Thanks,
> On 14 Feb 2019, at 07:42, Vinícius Ferrão <ferrao(a)versatushpc.com.br>
wrote:
>
> Are you guys running IPoIB for Infiniband Storage? Is it true IB mode?
Or Ethernet over IB?
>
> If using NFS on top of it, RDMA is working
> with NFSoRDMA?
>
> I’m really curious about your setup.
>
> Thanks,
>
> Sent from my iPhone
>
>> On 10 Feb 2019, at 23:55, darkytoo(a)gmail.com wrote:
>>
>> You were just like me, I also thought it would pick up settings from
cockpit, but once I realized you had to configure the network again when
adding to the host, everything sort of fell into line. I would make sure
and monitor the networks so you can see the traffic going through the
infiniband and not the ethernet.
>> _______________________________________________
>> Users mailing list -- users(a)ovirt.org
>> To unsubscribe send an email to users-leave(a)ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WC6W7KRXDEN...
9:43 p.m.
Interesting.
I have a lot of stability issues with IPoIB in EL7. In Windows it’s works flawlessly, but
on Linux is just unstable. So I never had the courage to put an hypervisor Storage network
over IPoIB.
Are you using Mellanox OFED, or the upstream one?
The iSCSI is only through software? No RDMA involved? Aren’t the CPU being wasted in this
scenario since Connect-X3 does not offload TCP/IP connections?
Finally the IPoIB mode is connected or dataram? Have you tunned the MTU settings?
Thanks,
Sent from my iPhone
On 1 Mar 2019, at 21:36, Mike Blanchard <darkytoo(a)gmail.com>
wrote:
Hi Vinicius, i'm sorry, I thought I had replied already.
I'm mostly using IPoIB, but gluster is using some over RDMA. I'm using it a few
different ways,
5 IB nodes, all have dual port ConnectX-3 cards @ 40GB
2 nodes are running windows, one port on each is used for a sync channel for starwind
Vsan to mirror ~20tb of date. The second ports are using for iSCS sharing SMB fiile
shares for other IB nodes.
3 nides are running CentOS 7 and ovirt. One of the ports on each are used for gluster
synchronzaton, gluster is set to use both IPoIB & RDMA. Then the other ports are
using for migratons, iSCSI sharing, and a backup admin channel.
> On Fri, Mar 1, 2019 at 1:36 PM Vinícius Ferrão <ferrao(a)versatushpc.com.br>
wrote:
> Darkytoo have you received my message.
>
> I’m really curious about your setup using Infiniband.
>
> Thanks,
>
> > On 14 Feb 2019, at 07:42, Vinícius Ferrão <ferrao(a)versatushpc.com.br>
wrote:
> >
> > Are you guys running IPoIB for Infiniband Storage? Is it true IB mode? Or
Ethernet over IB?
> >
> > If using NFS on top of it, RDMA is working
> > with NFSoRDMA?
> >
> > I’m really curious about your setup.
> >
> > Thanks,
> >
> > Sent from my iPhone
> >
> >> On 10 Feb 2019, at 23:55, darkytoo(a)gmail.com wrote:
> >>
> >> You were just like me, I also thought it would pick up settings from
cockpit, but once I realized you had to configure the network again when adding to the
host, everything sort of fell into line. I would make sure and monitor the networks so
you can see the traffic going through the infiniband and not the ethernet.
> >> _______________________________________________
> >> Users mailing list -- users(a)ovirt.org
> >> To unsubscribe send an email to users-leave(a)ovirt.org
> >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> >> oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
> >> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WC6W7KRXDEN...
1874
days inactive
1894
days old
8 comments
4 participants
participants (4)
-
cbop.mail＠gmail.com -
darkytoo＠gmail.com -
Mike Blanchard
-
Vinícius Ferrão