Cannot assign Network to Mellanox Interface
Hello, I want to use my Infiniband Network for Storage Access, so I created a new network and tried to assign it to a host. When I do this, I get a success message but it's actually not assigned to my Infiniband NIC. I'm running the latest CentOS with the latest Mellanox OFED drivers and multiple Subnet Managers are running on the network. The NIC is a Mellanox ConnectX-2 VPI, set to Infiniband mode. I didnt't knew which logs to send, so when you need to look into a log, just tell me which one you need. I hope you can help me out. Roman Meusch
Hello fellow Infiniband-er! I have a 3 node cluster with Connectx-3 cards under CentosOS and it seems to be working well. One thing I didn't realhotize is once I assigned it to the host, I had to edit it and assign the network manually, otherwise it left it has DHCP, it never got an address (for obvious reasons) and it stopped working. When I originally set the servers up, I installed the infiniband support packages and I see them in cockpit before I Actually assigned them.
Hello, thanks for the answer! I was missing that, I thought it would pick up the previously configured IP I set in cockpit before I added the host! But I also found out the main issue: it was set as a VM network. But I don't know if that's meant to be like this or this is just a bug. Now it's added to the host but I can't send traffic through it, not even a ping! Have you some idea why it's still not working? Roman
Hello, now everything is working. For some reason I had to reconfigure the NIC on my engine machine. Roman
You were just like me, I also thought it would pick up settings from cockpit, but once I realized you had to configure the network again when adding to the host, everything sort of fell into line. I would make sure and monitor the networks so you can see the traffic going through the infiniband and not the ethernet.
Are you guys running IPoIB for Infiniband Storage? Is it true IB mode? Or Ethernet over IB? If using NFS on top of it, RDMA is working with NFSoRDMA? I’m really curious about your setup. Thanks, Sent from my iPhone
On 10 Feb 2019, at 23:55, darkytoo@gmail.com wrote:
You were just like me, I also thought it would pick up settings from cockpit, but once I realized you had to configure the network again when adding to the host, everything sort of fell into line. I would make sure and monitor the networks so you can see the traffic going through the infiniband and not the ethernet. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WC6W7KRXDENXVC...
Darkytoo have you received my message. I’m really curious about your setup using Infiniband. Thanks,
On 14 Feb 2019, at 07:42, Vinícius Ferrão <ferrao@versatushpc.com.br> wrote:
Are you guys running IPoIB for Infiniband Storage? Is it true IB mode? Or Ethernet over IB?
If using NFS on top of it, RDMA is working with NFSoRDMA?
I’m really curious about your setup.
Thanks,
Sent from my iPhone
On 10 Feb 2019, at 23:55, darkytoo@gmail.com wrote:
You were just like me, I also thought it would pick up settings from cockpit, but once I realized you had to configure the network again when adding to the host, everything sort of fell into line. I would make sure and monitor the networks so you can see the traffic going through the infiniband and not the ethernet. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WC6W7KRXDENXVC...
Hi Vinicius, i'm sorry, I thought I had replied already. I'm mostly using IPoIB, but gluster is using some over RDMA. I'm using it a few different ways, 5 IB nodes, all have dual port ConnectX-3 cards @ 40GB 2 nodes are running windows, one port on each is used for a sync channel for starwind Vsan to mirror ~20tb of date. The second ports are using for iSCS sharing SMB fiile shares for other IB nodes. 3 nides are running CentOS 7 and ovirt. One of the ports on each are used for gluster synchronzaton, gluster is set to use both IPoIB & RDMA. Then the other ports are using for migratons, iSCSI sharing, and a backup admin channel. On Fri, Mar 1, 2019 at 1:36 PM Vinícius Ferrão <ferrao@versatushpc.com.br> wrote:
Darkytoo have you received my message.
I’m really curious about your setup using Infiniband.
Thanks,
On 14 Feb 2019, at 07:42, Vinícius Ferrão <ferrao@versatushpc.com.br> wrote:
Are you guys running IPoIB for Infiniband Storage? Is it true IB mode? Or Ethernet over IB?
If using NFS on top of it, RDMA is working with NFSoRDMA?
I’m really curious about your setup.
Thanks,
Sent from my iPhone
On 10 Feb 2019, at 23:55, darkytoo@gmail.com wrote:
You were just like me, I also thought it would pick up settings from cockpit, but once I realized you had to configure the network again when adding to the host, everything sort of fell into line. I would make sure and monitor the networks so you can see the traffic going through the infiniband and not the ethernet. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WC6W7KRXDENXVC...
Interesting. I have a lot of stability issues with IPoIB in EL7. In Windows it’s works flawlessly, but on Linux is just unstable. So I never had the courage to put an hypervisor Storage network over IPoIB. Are you using Mellanox OFED, or the upstream one? The iSCSI is only through software? No RDMA involved? Aren’t the CPU being wasted in this scenario since Connect-X3 does not offload TCP/IP connections? Finally the IPoIB mode is connected or dataram? Have you tunned the MTU settings? Thanks, Sent from my iPhone
On 1 Mar 2019, at 21:36, Mike Blanchard <darkytoo@gmail.com> wrote:
Hi Vinicius, i'm sorry, I thought I had replied already.
I'm mostly using IPoIB, but gluster is using some over RDMA. I'm using it a few different ways, 5 IB nodes, all have dual port ConnectX-3 cards @ 40GB
2 nodes are running windows, one port on each is used for a sync channel for starwind Vsan to mirror ~20tb of date. The second ports are using for iSCS sharing SMB fiile shares for other IB nodes. 3 nides are running CentOS 7 and ovirt. One of the ports on each are used for gluster synchronzaton, gluster is set to use both IPoIB & RDMA. Then the other ports are using for migratons, iSCSI sharing, and a backup admin channel.
On Fri, Mar 1, 2019 at 1:36 PM Vinícius Ferrão <ferrao@versatushpc.com.br> wrote: Darkytoo have you received my message.
I’m really curious about your setup using Infiniband.
Thanks,
On 14 Feb 2019, at 07:42, Vinícius Ferrão <ferrao@versatushpc.com.br> wrote:
Are you guys running IPoIB for Infiniband Storage? Is it true IB mode? Or Ethernet over IB?
If using NFS on top of it, RDMA is working with NFSoRDMA?
I’m really curious about your setup.
Thanks,
Sent from my iPhone
On 10 Feb 2019, at 23:55, darkytoo@gmail.com wrote:
You were just like me, I also thought it would pick up settings from cockpit, but once I realized you had to configure the network again when adding to the host, everything sort of fell into line. I would make sure and monitor the networks so you can see the traffic going through the infiniband and not the ethernet. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WC6W7KRXDENXVC...
participants (4)
-
cbop.mail＠gmail.com -
darkytoo＠gmail.com -
Mike Blanchard
-
Vinícius Ferrão