<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Thu, Jun 8, 2017 at 10:25 AM, Nelson Lameiras <span dir="ltr"><<a href="mailto:nelson.lameiras@lyra-network.com" target="_blank">nelson.lameiras@lyra-network.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><div dir="ltr"><div class="gmail_extra"><span class="gmail-"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><div><br>Hello,<br></div><br><div>In our oVirt hosts, we are using DELL equallogic SAN with each server connecting to SAN via 2 physical interfaces. Since both interfaces share the same network (Equalogic limitation) we must patch the linux kernel to to allow iSCSI multipath with multiple NICs in the same subnet with sysctl :<br></div><br><div>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--<br></div><div><pre>net.ipv4.conf.p2p1.arp_ignore=<wbr>1
net.ipv4.conf.p2p1.arp_<wbr>announce=2
net.ipv4.conf.p2p1.rp_filter=2
net.ipv4.conf.p2p2.arp_ignore=<wbr>1
net.ipv4.conf.p2p2.arp_<wbr>announce=2
net.ipv4.conf.p2p2.rp_filter=2<br><br></pre></div></div></blockquote><blockquote><div>Hello Nelson,</div><div>nice to see another environment based on EQL iSCSI SAN.</div><div>The systctl.conf customized lines above are not what you mean with "we must patch the linux kernel", is it?</div><div>What do you mean exactly?</div></blockquote></div><div class="gmail_quote"><br></div></span><div class="gmail_quote">Sorry, by "patch" I meant "sysctl.conf configuration" indeed. <br></div><div class="gmail_quote">This configuration is what we use to be able to configure 2 network interfaces in the same subnet.<span class="gmail-"><br><div> <br></div></span></div></div></div></div></blockquote><div><br></div><div>OK.</div><div>I'm using that configuration too.</div><div>And also with physical hosts, used as physical RDBMS servers, that connect to Dell EQL iSCSI SAN, not only for oVirt.</div><div>It is a requirement so that you are sure that the packets come back to the same interface, because of more that 1 network interface on the same lan.</div><div>That said, I noticed that oVirt creates the file "vdsm.conf" under /etc/sysctl.d, with these contents</div><div> </div><div><div>net.ipv4.conf.default.arp_ignore = 1</div><div>net.ipv4.conf.default.arp_announce = 2</div></div><div><br></div><div>So this settings doesn't need to be overwritten by you. Do you have that file too?</div><div><br></div><div>In my case I left the oVirt generated file and created two more files in /etc/sysctl.d, named 50-iscsi.conf and equallogic.conf </div><div><br></div><div>Their contents:</div><div><br></div><div>- 50-iscsi.conf</div><div><br></div><div><div>net.ipv4.conf.p1p1/100.arp_announce=2</div><div>net.ipv4.conf.p1p1/100.arp_ignore=1<br></div><div>#</div><div>net.ipv4.conf.p1p1/100.rp_filter=2</div><div>net.ipv4.conf.p1p2.rp_filter=2<br></div><div><br></div><div>The reason of the first two lines is that vlan interfaces apparently didn't inherited the arp_announce and arp_ignore settings so I put them for the vlan set interface, no need for the other one (p1p2 in my case)</div><div>The reason of the second two lines is to add the third sysctl setting needed by EQL</div><div>So I think you can have a 50-iscsi.conf file only composed by:</div><div><br></div><div><div>net.ipv4.conf.p2p1.rp_filter=2</div><div>net.ipv4.conf.p2p2.rp_filter=2</div></div><div><br></div><div>- equallogic.conf</div><div><br></div><div><div>net.core.rmem_default = 262144</div><div>net.core.rmem_max = 16777216</div><div>net.core.wmem_default = 262144</div><div>net.core.wmem_max = 16777216</div><div>net.ipv4.tcp_rmem = 8192 87380 16777216</div><div>net.ipv4.tcp_wmem = 4096 65536 16777216</div><div><br></div><div># Defaults were:</div><div>#net.core.rmem_default = 212992</div><div>#net.core.rmem_max = 212992</div><div>#net.core.wmem_default = 212992</div><div>#net.core.wmem_max = 212992</div><div>#net.ipv4.tcp_rmem = 4096<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>87380<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>6291456</div><div>#net.ipv4.tcp_wmem = 4096<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>16384<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>4194304</div></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="gmail-"><blockquote><div><span style="font-size:12pt">I've read the link you gave and it seems to me that you are using the same physical interface for SAN and vm network.</span><br></div></blockquote></span><div>In my setup : </div><div> - SAN network is using two disctinct physical interfaces (p2pX) on same subnet (therefore the need for sysctl patch)<br></div><div> - VM network (p1pX) is using 2 other distinct physical interfaces on "trunk" mode<br></div><div> - Administration network (emX) is using 2 distinct bonded network interfaces <br></div><div><br></div><div>Our setup, using a total of 6 network interfaces is "simpler" than yours since each interface has a "disctinct" role.<br></div><div><br></div><div>To make multipath work with oVirt on this setup, we had to configure SAN network on system before installing oVirt (using sysctl.conf above), </div><div> - next declare 2 SAN networks on oVirt that are "dragued" to each SAN interface in host network configuration, </div><div> - and finally, in datacenter listing, chosing a datacenter and selecting "iSCSI Multipathing" tab below I can add a multipath configuration concerning both SAN oVirt networks and relevant iqn storage targets.<br></div><div><br></div><div>Hope it's clear, if not, please ask me.<br></div><div><br></div><div>It's very strange to me also since this configuration did work great on our labs. But half of production sites have the problem decribed above : second SAN interface stops working uppon host activation because vdsm forces sysctl net.ipv4.conf.p2p2.rp_filter to 1 (strict mode).<br></div><div><br></div><div>Still searching for clues as why this behaviour...<br></div></div></div></div><br></div></div></div></blockquote></div><br></div><div class="gmail_extra">I had the further need to use both interfaces for VM network, but, also if not, I couldn't setup multipathing.</div><div class="gmail_extra">I have a requirement to have iSCSI on vlan and when setting the two interfaces on the same vlan (so also in oVirt), oVirt complained about that.</div><div class="gmail_extra">So this lets me think that if instead you have both network interfaces in access mode, without vlan, oVirt doesn't complain when you try toset the multipathing????</div><div class="gmail_extra"><br></div><div class="gmail_extra">If this is the case, I think I could have tested (probably I didn't, I don't remember) this scenario</div><div class="gmail_extra"><br></div><div class="gmail_extra">Both interfaces on vlan 100 but the network switch port configured with default vlan as 100, so that form OS point of view, and oVirt point of view, no vlan in place and perhaps it doesn't complain....</div><div class="gmail_extra"><br></div><div class="gmail_extra">Gianluca</div></div>