[ovirt-users] ISCSI storage with multiple nics on same subnet disabled on host activation

[Gianluca Cecchi]

On Thu, Jun 8, 2017 at 10:25 AM, Nelson Lameiras <
nelson.lameiras at lyra-network.com> wrote:

>
>> Hello,
>>
>> In our oVirt hosts, we are using DELL equallogic SAN with each server
>> connecting to SAN via 2 physical interfaces. Since both interfaces share
>> the same network (Equalogic limitation) we must patch the linux kernel to
>> to allow iSCSI multipath with multiple NICs in the same subnet with sysctl :
>>
>> ------------------------------------------------------------
>> --------------------------------
>>
>> net.ipv4.conf.p2p1.arp_ignore=1
>> net.ipv4.conf.p2p1.arp_announce=2
>> net.ipv4.conf.p2p1.rp_filter=2
>>
>> net.ipv4.conf.p2p2.arp_ignore=1
>> net.ipv4.conf.p2p2.arp_announce=2
>> net.ipv4.conf.p2p2.rp_filter=2
>>
>> Hello Nelson,
> nice to see another environment based on EQL iSCSI SAN.
> The systctl.conf customized lines above are not what you mean with "we
> must patch the linux kernel", is it?
> What do you mean exactly?
>
>
> Sorry, by "patch" I meant "sysctl.conf configuration" indeed.
> This configuration is what we use to be able to configure 2 network
> interfaces in the same subnet.
>
>

OK.
I'm using that configuration too.
And also with physical hosts, used as physical RDBMS servers,  that connect
to Dell EQL iSCSI SAN, not only for oVirt.
It is a requirement so that you are sure that the packets come back to the
same interface, because of more that 1 network interface on the same lan.
That said, I noticed that oVirt creates the file "vdsm.conf" under
/etc/sysctl.d, with these contents

net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2

So this settings doesn't need to be overwritten by you. Do you have that
file too?

In my case I left the oVirt generated file and created two more files in
/etc/sysctl.d, named 50-iscsi.conf and equallogic.conf

Their contents:

- 50-iscsi.conf

net.ipv4.conf.p1p1/100.arp_announce=2
net.ipv4.conf.p1p1/100.arp_ignore=1
#
net.ipv4.conf.p1p1/100.rp_filter=2
net.ipv4.conf.p1p2.rp_filter=2

The reason of the first two lines is that vlan interfaces apparently didn't
inherited the arp_announce and arp_ignore settings so I put them for the
vlan set interface, no need for the other one (p1p2 in my case)
The reason of the second two lines is to add the third sysctl setting
needed by EQL
So I think you can have a 50-iscsi.conf file only composed by:

net.ipv4.conf.p2p1.rp_filter=2
net.ipv4.conf.p2p2.rp_filter=2

- equallogic.conf

net.core.rmem_default = 262144
net.core.rmem_max = 16777216
net.core.wmem_default = 262144
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Defaults were:
#net.core.rmem_default = 212992
#net.core.rmem_max = 212992
#net.core.wmem_default = 212992
#net.core.wmem_max = 212992
#net.ipv4.tcp_rmem = 4096 87380 6291456
#net.ipv4.tcp_wmem = 4096 16384 4194304

> I've read the link you gave and it seems to me that you are using the same
> physical interface for SAN and vm network.
>
> In my setup :
>   - SAN network is using two disctinct physical interfaces (p2pX) on same
> subnet (therefore the need for sysctl patch)
>   - VM network (p1pX) is using 2 other distinct physical interfaces on
> "trunk" mode
>   - Administration network (emX) is using 2 distinct bonded network
> interfaces
>
> Our setup, using a total of 6 network interfaces is "simpler" than yours
> since each interface has a "disctinct" role.
>
> To make multipath work with oVirt on this setup, we had to configure SAN
> network on system before installing oVirt (using sysctl.conf above),
>   - next declare 2 SAN networks on oVirt that are "dragued" to each SAN
> interface in host network configuration,
>   - and finally, in datacenter listing, chosing a datacenter and selecting
> "iSCSI Multipathing" tab below I can add a multipath configuration
> concerning both SAN oVirt networks and relevant iqn storage targets.
>
> Hope it's clear, if not, please ask me.
>
> It's very strange to me also since this configuration did work great on
> our labs. But half of production sites have the problem decribed above :
> second SAN interface stops working uppon host activation because vdsm
> forces sysctl net.ipv4.conf.p2p2.rp_filter to 1 (strict mode).
>
> Still searching for clues as why this behaviour...
>
>
I had the further need to use both interfaces for VM network, but, also if
not, I couldn't setup multipathing.
I have a requirement to have iSCSI on vlan and when setting the two
interfaces on the same vlan (so also in oVirt), oVirt complained about that.
So this lets me think that if instead you have both network interfaces in
access mode, without vlan, oVirt doesn't complain when you try toset the
multipathing????

If this is the case, I think I could have tested (probably I didn't, I
don't remember) this scenario

Both interfaces on vlan 100 but the network switch port configured with
default vlan as 100, so that form OS point of view, and oVirt point of
view, no vlan in place and perhaps it doesn't complain....

Gianluca
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170608/fb4fa028/attachment.html>