On Thu, Jun 8, 2017 at 10:25 AM, Nelson Lameiras <nelson.lameiras@lyra-network.com> wrote:

Hello,

In our oVirt hosts, we are using DELL equallogic SAN with each server connecting to SAN via 2 physical interfaces. Since both interfaces share the same network (Equalogic limitation) we must patch the linux kernel to to allow iSCSI multipath with multiple NICs in the same subnet with sysctl :

--------------------------------------------------------------------------------------------
net.ipv4.conf.p2p1.arp_ignore=1 net.ipv4.conf.p2p1.arp_announce=2 net.ipv4.conf.p2p1.rp_filter=2 net.ipv4.conf.p2p2.arp_ignore=1 net.ipv4.conf.p2p2.arp_announce=2 net.ipv4.conf.p2p2.rp_filter=2

Hello Nelson,
nice to see another environment based on EQL iSCSI SAN.
The systctl.conf customized lines above are not what you mean with "we must patch the linux kernel", is it?
What do you mean exactly?

Sorry, by "patch" I meant "sysctl.conf configuration" indeed.
This configuration is what we use to be able to configure 2 network interfaces in the same subnet.

OK.

I'm using that configuration too.

And also with physical hosts, used as physical RDBMS servers, that connect to Dell EQL iSCSI SAN, not only for oVirt.

It is a requirement so that you are sure that the packets come back to the same interface, because of more that 1 network interface on the same lan.

That said, I noticed that oVirt creates the file "vdsm.conf" under /etc/sysctl.d, with these contents

net.ipv4.conf.default.arp_ignore = 1

net.ipv4.conf.default.arp_announce = 2

So this settings doesn't need to be overwritten by you. Do you have that file too?

In my case I left the oVirt generated file and created two more files in /etc/sysctl.d, named 50-iscsi.conf and equallogic.conf

Their contents:

- 50-iscsi.conf

net.ipv4.conf.p1p1/100.arp_announce=2

net.ipv4.conf.p1p1/100.arp_ignore=1

net.ipv4.conf.p1p1/100.rp_filter=2

net.ipv4.conf.p1p2.rp_filter=2

The reason of the first two lines is that vlan interfaces apparently didn't inherited the arp_announce and arp_ignore settings so I put them for the vlan set interface, no need for the other one (p1p2 in my case)

The reason of the second two lines is to add the third sysctl setting needed by EQL

So I think you can have a 50-iscsi.conf file only composed by:

net.ipv4.conf.p2p1.rp_filter=2

net.ipv4.conf.p2p2.rp_filter=2

- equallogic.conf

net.core.rmem_default = 262144

net.core.rmem_max = 16777216

net.core.wmem_default = 262144

net.core.wmem_max = 16777216

net.ipv4.tcp_rmem = 8192 87380 16777216

net.ipv4.tcp_wmem = 4096 65536 16777216

# Defaults were:

#net.core.rmem_default = 212992

#net.core.rmem_max = 212992

#net.core.wmem_default = 212992

#net.core.wmem_max = 212992

#net.ipv4.tcp_rmem = 4096 87380 6291456

#net.ipv4.tcp_wmem = 4096 16384 4194304

I've read the link you gave and it seems to me that you are using the same physical interface for SAN and vm network.
In my setup :
- SAN network is using two disctinct physical interfaces (p2pX) on same subnet (therefore the need for sysctl patch)
- VM network (p1pX) is using 2 other distinct physical interfaces on "trunk" mode
- Administration network (emX) is using 2 distinct bonded network interfaces

Our setup, using a total of 6 network interfaces is "simpler" than yours since each interface has a "disctinct" role.

To make multipath work with oVirt on this setup, we had to configure SAN network on system before installing oVirt (using sysctl.conf above),
- next declare 2 SAN networks on oVirt that are "dragued" to each SAN interface in host network configuration,
- and finally, in datacenter listing, chosing a datacenter and selecting "iSCSI Multipathing" tab below I can add a multipath configuration concerning both SAN oVirt networks and relevant iqn storage targets.

Hope it's clear, if not, please ask me.

It's very strange to me also since this configuration did work great on our labs. But half of production sites have the problem decribed above : second SAN interface stops working uppon host activation because vdsm forces sysctl net.ipv4.conf.p2p2.rp_filter to 1 (strict mode).

Still searching for clues as why this behaviour...

I had the further need to use both interfaces for VM network, but, also if not, I couldn't setup multipathing.

I have a requirement to have iSCSI on vlan and when setting the two interfaces on the same vlan (so also in oVirt), oVirt complained about that.

So this lets me think that if instead you have both network interfaces in access mode, without vlan, oVirt doesn't complain when you try toset the multipathing????

If this is the case, I think I could have tested (probably I didn't, I don't remember) this scenario

Both interfaces on vlan 100 but the network switch port configured with default vlan as 100, so that form OS point of view, and oVirt point of view, no vlan in place and perhaps it doesn't complain....

Gianluca