[ovirt-users] ISCSI storage with multiple nics on same subnet disabled on host activation

Thu Jun 8 08:25:45 UTC 2017

Hi Gianluca, 

My answers below, 

cordialement, regards, 

	Nelson LAMEIRAS 
Ingénieur Systèmes et Réseaux / Systems and Networks engineer 
Tel: +33 5 32 09 09 70 
nelson.lameiras at lyra-network.com 

www.lyra-network.com | www.payzen.eu 

Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE 

From: "Gianluca Cecchi" <gianluca.cecchi at gmail.com> 
To: "Nelson Lameiras" <nelson.lameiras at lyra-network.com> 
Cc: "ovirt users" <users at ovirt.org> 
Sent: Thursday, June 8, 2017 9:36:33 AM 
Subject: Re: [ovirt-users] ISCSI storage with multiple nics on same subnet disabled on host activation 

On Wed, Jun 7, 2017 at 2:59 PM, Nelson Lameiras < nelson.lameiras at lyra-network.com > wrote: 

BQ_BEGIN

Hello, 

In our oVirt hosts, we are using DELL equallogic SAN with each server connecting to SAN via 2 physical interfaces. Since both interfaces share the same network (Equalogic limitation) we must patch the linux kernel to to allow iSCSI multipath with multiple NICs in the same subnet with sysctl : 

-------------------------------------------------------------------------------------------- 
net.ipv4.conf.p2p1.arp_ignore=1
net.ipv4.conf.p2p1.arp_announce=2
net.ipv4.conf.p2p1.rp_filter=2

net.ipv4.conf.p2p2.arp_ignore=1
net.ipv4.conf.p2p2.arp_announce=2
net.ipv4.conf.p2p2.rp_filter=2 

BQ_END

BQ_BEGIN

Hello Nelson, 
nice to see another environment based on EQL iSCSI SAN. 
The systctl.conf customized lines above are not what you mean with "we must patch the linux kernel", is it? 
What do you mean exactly? 

BQ_END

Sorry, by "patch" I meant "sysctl.conf configuration" indeed. 
This configuration is what we use to be able to configure 2 network interfaces in the same subnet. 

BQ_BEGIN

-------------------------------------------------------------------------------------------- 

BQ_END

BQ_BEGIN

This works great in most setups, but for a strange reason, on some of our setups, the sysctl configuration is updated by VDSM when activating a host and the second interface stops working immeadiatly : 
-------------------------------------------------------------------------------------------- 
vdsm.log 
2017-06-07 11:51:51,063+0200 INFO  (jsonrpc/5) [storage.ISCSI] Setting strict mode rp_filter for device 'p2p2'. (iscsi:602)
2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not connect to storageServer (hsm:2392)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect
    iscsi.addIscsiNode(self._iface, self._target, self._cred)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, in addIscsiNode
    iscsiadm.node_login( iface.name , target.address, target.iqn)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 337, in node_login
    raise IscsiNodeError(rc, out, err) 

In my environment I don't see these kinds of problems. 
I'm using plain CentOS 7.3 hosts as hypervisors; is this your case too or are you using ovirt-ng nodes? 

BQ_END

Plain CentOS 7.3 hosts also. Built from a Minimal centos and adding oVirt repositories. 

BQ_BEGIN

-------------------------------------------------------------------------------------------- 

"strict mode" is enforced for second interface, and it no longuer works... 
Which means - at least - that there is no redundancy in case of hardware faillure and this is not acceptable for our production needs. 

What is really strange is that we have another "twin" site on another geographic region with simillar hardware configuration and same oVirt installation, and this problem does not happen. 

What can be the root cause of this behaviour? How can I correct it? 

BQ_END

BQ_BEGIN

Actually it is very strange to me to understand how you don't have problems on the other side. 
My workaround to get multipath with EQL in oVirt was the one explained here with screenshots: 
http://lists.ovirt.org/pipermail/users/2017-March/080992.html 
It has been working now for about 3 months without problems. We have several Oracle RDBMS server VMs on it, from 2virtual threads and 8Gb of ram to 16 threads (2-4-2) and 64Gb of ram 
How did you configure multipath instead? As far as I know oVirt will not let you do it. 
The only way would be to use bonding but that is not certified/supported by Dell. 
HIH sharing experiences, 
Gianluca 

BQ_END

I've read the link you gave and it seems to me that you are using the same physical interface for SAN and vm network. 
In my setup : 
- SAN network is using two disctinct physical interfaces (p2pX) on same subnet (therefore the need for sysctl patch) 
- VM network (p1pX) is using 2 other distinct physical interfaces on "trunk" mode 
- Administration network (emX) is using 2 distinct bonded network interfaces 

Our setup, using a total of 6 network interfaces is "simpler" than yours since each interface has a "disctinct" role. 

To make multipath work with oVirt on this setup, we had to configure SAN network on system before installing oVirt (using sysctl.conf above), 
- next declare 2 SAN networks on oVirt that are "dragued" to each SAN interface in host network configuration, 
- and finally, in datacenter listing, chosing a datacenter and selecting "iSCSI Multipathing" tab below I can add a multipath configuration concerning both SAN oVirt networks and relevant iqn storage targets. 

Hope it's clear, if not, please ask me. 

It's very strange to me also since this configuration did work great on our labs. But half of production sites have the problem decribed above : second SAN interface stops working uppon host activation because vdsm forces sysctl net.ipv4.conf.p2p2.rp_filter to 1 (strict mode). 

Still searching for clues as why this behaviour... 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170608/089b4b99/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: element-signature_logo_lyra_115x94.jpg
Type: image/jpeg
Size: 3846 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170608/089b4b99/attachment-0005.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: element-signature_logo_YouTube_32x28.jpg
Type: image/jpeg
Size: 1604 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170608/089b4b99/attachment-0006.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: element-signature_logo_LinkedIn_41x28.jpg
Type: image/jpeg
Size: 1635 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170608/089b4b99/attachment-0007.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: element-signature_logo_Twitter_42x28.jpg
Type: image/jpeg
Size: 1629 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170608/089b4b99/attachment-0008.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: element-signature_payzen_61x28.jpg
Type: image/jpeg
Size: 1864 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170608/089b4b99/attachment-0009.jpg>