10:25 a.m.
------=_Part_57900106_686621279.1496910345349
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Hi Gianluca,=20
My answers below,=20
cordialement, regards,=20
=09Nelson LAMEIRAS=20
Ing=C3=A9nieur Syst=C3=A8mes et R=C3=A9seaux / Systems and Networks enginee=
r=20
Tel: +33 5 32 09 09 70=20
nelson.lameiras(a)lyra-network.com=20
www.lyra-network.com | www.payzen.eu=20
=09
=09
=09
Lyra Network, 109 rue de l'innovation, 31670 Lab=C3=A8ge, FRANCE=20
From: "Gianluca Cecchi" <gianluca.cecchi(a)gmail.com>=20
To: "Nelson Lameiras" <nelson.lameiras(a)lyra-network.com>=20
Cc: "ovirt users" <users(a)ovirt.org>=20
Sent: Thursday, June 8, 2017 9:36:33 AM=20
Subject: Re: [ovirt-users] ISCSI storage with multiple nics on same subnet =
disabled on host activation=20
On Wed, Jun 7, 2017 at 2:59 PM, Nelson Lameiras < nelson.lameiras@lyra-netw=
ork.com > wrote:=20
BQ_BEGIN
Hello,=20
In our oVirt hosts, we are using DELL equallogic SAN with each server conne=
cting to SAN via 2 physical interfaces. Since both interfaces share the sam=
e network (Equalogic limitation) we must patch the linux kernel to to allow=
iSCSI multipath with multiple NICs in the same subnet with sysctl :=20
---------------------------------------------------------------------------=
-----------------=20
net.ipv4.conf.p2p1.arp_ignore=3D1
net.ipv4.conf.p2p1.arp_announce=3D2
net.ipv4.conf.p2p1.rp_filter=3D2
net.ipv4.conf.p2p2.arp_ignore=3D1
net.ipv4.conf.p2p2.arp_announce=3D2
net.ipv4.conf.p2p2.rp_filter=3D2=20
BQ_END
BQ_BEGIN
Hello Nelson,=20
nice to see another environment based on EQL iSCSI SAN.=20
The systctl.conf customized lines above are not what you mean with "we must=
patch the linux kernel", is it?=20
What do you mean exactly?=20
BQ_END
Sorry, by "patch" I meant "sysctl.conf configuration" indeed.=20
This configuration is what we use to be able to configure 2 network interfa=
ces in the same subnet.=20
BQ_BEGIN
---------------------------------------------------------------------------=
-----------------=20
BQ_END
BQ_BEGIN
This works great in most setups, but for a strange reason, on some of our s=
etups, the sysctl configuration is updated by VDSM when activating a host a=
nd the second interface stops working immeadiatly :=20
---------------------------------------------------------------------------=
-----------------=20
vdsm.log=20
2017-06-07 11:51:51,063+0200 INFO (jsonrpc/5) [storage.ISCSI] Setting stri=
ct mode rp_filter for device 'p2p2'. (iscsi:602)
2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not conn=
ect to storageServer (hsm:2392)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer
conObj.connect()
File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect
iscsi.addIscsiNode(self._iface, self._target, self._cred)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, =
in addIscsiNode
iscsiadm.node_login( iface.name , target.address, target.iqn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 33=
7, in node_login
raise IscsiNodeError(rc, out, err)=20
In my environment I don't see these kinds of problems.=20
I'm using plain CentOS 7.3 hosts as hypervisors; is this your case too or a=
re you using ovirt-ng nodes?=20
BQ_END
Plain CentOS 7.3 hosts also. Built from a Minimal centos and adding oVirt r=
epositories.=20
BQ_BEGIN
---------------------------------------------------------------------------=
-----------------=20
"strict mode" is enforced for second interface, and it no longuer works...=
=20
Which means - at least - that there is no redundancy in case of hardware fa=
illure and this is not acceptable for our production needs.=20
What is really strange is that we have another "twin" site on another geogr=
aphic region with simillar hardware configuration and same oVirt installati=
on, and this problem does not happen.=20
What can be the root cause of this behaviour? How can I correct it?=20
BQ_END
BQ_BEGIN
Actually it is very strange to me to understand how you don't have problems=
on the other side.=20
My workaround to get multipath with EQL in oVirt was the one explained here=
with screenshots:=20
http://lists.ovirt.org/pipermail/users/2017-March/080992.html=20
It has been working now for about 3 months without problems. We have severa=
l Oracle RDBMS server VMs on it, from 2virtual threads and 8Gb of ram to 16=
threads (2-4-2) and 64Gb of ram=20
How did you configure multipath instead? As far as I know oVirt will not le=
t you do it.=20
The only way would be to use bonding but that is not certified/supported by=
Dell.=20
HIH sharing experiences,=20
Gianluca=20
BQ_END
I've read the link you gave and it seems to me that you are using the same =
physical interface for SAN and vm network.=20
In my setup :=20
- SAN network is using two disctinct physical interfaces (p2pX) on same sub=
net (therefore the need for sysctl patch)=20
- VM network (p1pX) is using 2 other distinct physical interfaces on "trunk=
" mode=20
- Administration network (emX) is using 2 distinct bonded network interface=
s=20
Our setup, using a total of 6 network interfaces is "simpler" than yours si=
nce each interface has a "disctinct" role.=20
To make multipath work with oVirt on this setup, we had to configure SAN ne=
twork on system before installing oVirt (using sysctl.conf above),=20
- next declare 2 SAN networks on oVirt that are "dragued" to each SAN inter=
face in host network configuration,=20
- and finally, in datacenter listing, chosing a datacenter and selecting "i=
SCSI Multipathing" tab below I can add a multipath configuration concerning=
both SAN oVirt networks and relevant iqn storage targets.=20
Hope it's clear, if not, please ask me.=20
It's very strange to me also since this configuration did work great on our=
labs. But half of production sites have the problem decribed above : secon=
d SAN interface stops working uppon host activation because vdsm forces sys=
ctl net.ipv4.conf.p2p2.rp_filter to 1 (strict mode).=20
Still searching for clues as why this behaviour...=20
------=_Part_57900106_686621279.1496910345349
Content-Type: multipart/related;
boundary="----=_Part_57900107_1954785857.1496910345350"
------=_Part_57900107_1954785857.1496910345350
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable
<html><body><div style=3D"font-family: arial, helvetica, sans-serif;
font-s=
ize: 12pt; color: #000000"><div>Hi
Gianluca,<br></div><div><br data-mce-bog=
us=3D"1"></div><div>My answers below,<br
data-mce-bogus=3D"1"></div><div><b=
r></div><div data-marker=3D"__SIG_PRE__"><div>cordialement,
regards,</div><=
div><br data-mce-bogus=3D"1"></div><table
style=3D"margin: 0px; table-layou=
t: fixed; border: none; width: 345pt; padding: 0px;" data-mce-style=3D"marg=
in: 0px; table-layout: fixed; border: none; width: 345pt; padding: 0px;" da=
ta-mce-selected=3D"1" width=3D"460" cellspacing=3D"0"
cellpadding=3D"0" bor=
der=3D"0"><tbody><tr><td style=3D"margin: 0; font-size:
11px; font-family: =
Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 86.25pt; text-a=
lign: left;" data-mce-style=3D"margin: 0; font-size: 11px; font-family: Ari=
al,sans-serif; color: #2b3c7a; line-height: 16px; width: 86.25pt; text-alig=
n: left;" width=3D"115" valign=3D"top"><a
href=3D"https://www.lyra-network.=
com/" style=3D"border: none; text-decoration: none; width: 100%; display: b=
lock; clear: both;" data-mce-href=3D"https://www.lyra-network.com/"
data-mc=
e-style=3D"border: none; text-decoration: none; width: 100%; display: block=
; clear: both;"><img
src=3D"cid:25cd0e595e0f025605ddb995496484dcb70629d9@zi=
mbra" alt=3D""
data-mce-src=3D"/home/admin(a)lyra-network.com/corporate-docum=
ents/element-signature_logo_lyra_115x94.jpg" doc=3D"corporate-documents/ele=
ment-signature_logo_lyra_115x94.jpg" height=3D"94" width=3D"115"
border=3D"=
0"></a><br data-mce-bogus=3D"1"></td><td
style=3D"margin: 0; font-size: 11p=
x; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width:=
258.75pt;" data-mce-style=3D"margin: 0; font-size: 11px; font-family: Aria=
l,sans-serif; color: #2b3c7a; line-height: 16px; width: 258.75pt;" width=3D=
"345" valign=3D"top"><table style=3D"margin: 0; width:
258.75pt; table-layo=
ut: fixed; border: none;" data-mce-style=3D"margin: 0; width: 258.75pt; tab=
le-layout: fixed; border: none;" width=3D"345" cellspacing=3D"0"
cellpaddin=
g=3D"0" border=3D"0"><tbody><tr><td
style=3D"font-size: 16px; font-family: =
Arial,sans-serif; color: #2b3c7a; line-height: 18px; font-weight: bold;" da=
ta-mce-style=3D"font-size: 16px; font-family: Arial,sans-serif; color: #2b3=
c7a; line-height: 18px; font-weight: bold;">Nelson
LAMEIRAS</td></tr><tr><t=
d style=3D"font-family: Arial,sans-serif; color: #2b3c7a; font-size: 11px; =
line-height: 16px;" data-mce-style=3D"font-family: Arial,sans-serif; color:=
#2b3c7a; font-size: 11px; line-height: 16px;">Ing=C3=A9nieur Syst=C3=A8mes=
et R=C3=A9seaux<span style=3D"font-family: Arial,sans-serif; color: #337ab=
7; font-size: 11px;" data-mce-style=3D"font-family: Arial,sans-serif; color=
: #337ab7; font-size: 11px;"> / Systems and Networks
engineer</span></td></=
tr><tr><td style=3D"line-height: 16px; font-family:
'Arial',sans-serif; col=
or: #2b3c7a; font-size: 11px;" data-mce-style=3D"line-height: 16px; font-fa=
mily: 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;"><span
style=3D"=
font-family: Arial,sans-serif; color: #2b3c7a; font-size: 11px;" data-mce-s=
tyle=3D"font-family: Arial,sans-serif; color: #2b3c7a; font-size: 11px;">Te=
l: +33 5 32 09 09 70</span></td></tr><tr><td
style=3D"line-height: 16px; fo=
nt-family: 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;" data-mce-s=
tyle=3D"line-height: 16px; font-family: 'Arial',sans-serif; color: #2b3c7a;=
font-size: 11px;"><a
href=3D"mailto:nelson.lameiras@lyra-network.com" styl=
e=3D"font-size: 11px; font-family: Arial,sans-serif; line-height: 16px; tex=
t-decoration: none;" data-mce-href=3D"mailto:nelson.lameiras@lyra-network.c=
om" data-mce-style=3D"font-size: 11px; font-family: Arial,sans-serif; line-=
height: 16px; text-decoration:
none;">nelson.lameiras(a)lyra-network.com</a><=
br data-mce-bogus=3D"1"></td></tr><tr><td
style=3D"line-height: 16px; font-=
family: 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;" data-mce-styl=
e=3D"line-height: 16px; font-family: 'Arial',sans-serif; color: #2b3c7a; fo=
nt-size: 11px;"><a href=3D"https://www.lyra-network.com/"
style=3D"font-siz=
e: 11px; font-family: Arial,sans-serif; line-height: 16px; border: none; te=
xt-decoration: none;" data-mce-href=3D"https://www.lyra-network.com/"
data-=
mce-style=3D"font-size: 11px; font-family: Arial,sans-serif; line-height: 1=
6px; border: none; text-decoration: none;">www.lyra-network.com</a> <a
href=
=3D"https://payzen.eu" style=3D"font-size: 11px; font-family:
Arial,sans-se=
rif; line-height: 16px; border: none; text-decoration: none;" data-mce-href=
=3D"https://payzen.eu" data-mce-style=3D"font-size: 11px; font-family:
Aria=
l,sans-serif; line-height: 16px; border: none; text-decoration: none;"> | w=
ww.payzen.eu</a><br
data-mce-bogus=3D"1"></td></tr></tbody></table></td></t=
r></tbody></table><table style=3D"margin: 0; table-layout: fixed;
border: n=
one; width: 345pt;" data-mce-style=3D"margin: 0; table-layout: fixed; borde=
r: none; width: 345pt;" data-mce-selected=3D"1" width=3D"460"
cellspacing=
=3D"0" cellpadding=3D"0"><tbody><tr><td
style=3D"margin: 0; font-size: 11px=
; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: =
24pt; text-align: left;" data-mce-style=3D"margin: 0; font-size: 11px; font=
-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 24pt; =
text-align: left;" width=3D"32" valign=3D"top"><a
href=3D"https://www.youtu=
be.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA" style=3D"border: none; text-decora=
tion: none; margin: 0; float: left; display: inline-block; width: 32px;" da=
ta-mce-href=3D"https://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ... da=
ta-mce-style=3D"border: none; text-decoration: none; margin: 0; float: left=
; display: inline-block; width: 32px;"><img
src=3D"cid:4c5918308790526a8092=
b3bb40ae8dab65d1777b@zimbra" alt=3D""
data-mce-src=3D"/home/admin@lyra-netw=
ork.com/corporate-documents/element-signature_logo_YouTube_32x28.jpg" doc=
=3D"corporate-documents/element-signature_logo_YouTube_32x28.jpg" height=3D=
"28" width=3D"32" border=3D"0"></a><br
data-mce-bogus=3D"1"></td><td style=
=3D"margin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c=
7a; line-height: 16px; width: 30.75pt; text-align: left;" data-mce-style=3D=
"margin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a;=
line-height: 16px; width: 30.75pt; text-align: left;" width=3D"41"
valign=
=3D"top"><a
href=3D"https://www.linkedin.com/company/lyra-network_2" style=
=3D"border: none; text-decoration: none; margin: 0; float: left; display: i=
nline-block; width: 41px;" data-mce-href=3D"https://www.linkedin.com/compan=
y/lyra-network_2" data-mce-style=3D"border: none; text-decoration: none; ma=
rgin: 0; float: left; display: inline-block; width: 41px;"><img
src=3D"cid:=
0c1a4f5470e19b606b0a0890da695ad9d2808a60@zimbra" data-mce-src=3D"/home/admi=
n(a)lyra-network.com/corporate-documents/element-signature_logo_LinkedIn_41x2=
8.jpg" doc=3D"corporate-documents/element-signature_logo_LinkedIn_41x28.jpg=
" height=3D"28" width=3D"41"
border=3D"0"></a><br data-mce-bogus=3D"1"></td=
<td style=3D"margin: 0; font-size: 11px; font-family:
Arial,sans-serif; co=
lor: #2b3c7a; line-height: 16px; width: 31.5pt; text-align:
left;" data-mce=
-style=3D"margin: 0; font-size: 11px; font-family: Arial,sans-serif; color:=
#2b3c7a; line-height: 16px; width: 31.5pt; text-align: left;"
width=3D"42"=
valign=3D"top"><a href=3D"https://twitter.com/LyraNetwork"
style=3D"border=
: none; text-decoration: none; margin: 0; float: left; display: inline-bloc=
k; width: 42px;" data-mce-href=3D"https://twitter.com/LyraNetwork"
data-mce=
-style=3D"border: none; text-decoration: none; margin: 0; float: left; disp=
lay: inline-block; width: 42px;"><img
src=3D"cid:5f6dbbaff6dac1a2e25c3c96f0=
2d8020be9c5268@zimbra" alt=3D""
data-mce-src=3D"/home/admin(a)lyra-network.co=
m/corporate-documents/element-signature_logo_Twitter_42x28.jpg" doc=3D"corp=
orate-documents/element-signature_logo_Twitter_42x28.jpg" height=3D"28"
wid=
th=3D"42" border=3D"0"></a><br
data-mce-bogus=3D"1"></td><td style=3D"margi=
n: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-=
height: 16px; width: 258.75pt; text-align: left;" data-mce-style=3D"margin:=
0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-he=
ight: 16px; width: 258.75pt; text-align: left;" width=3D"345"
valign=3D"top=
"><a href=3D"https://payzen.eu" style=3D"border: none;
text-decoration: non=
e; margin: 0; float: left; display: inline-block; width: 61px;" data-mce-hr=
ef=3D"https://payzen.eu" data-mce-style=3D"border: none; text-decoration:
n=
one; margin: 0; float: left; display: inline-block; width: 61px;"><img src=
=3D"cid:7003ef2927fa834b99af5bd467d27eb6d701e6b1@zimbra" alt=3D""
data-mce-=
src=3D"/home/admin(a)lyra-network.com/corporate-documents/element-signature_p=
ayzen_61x28.jpg" doc=3D"corporate-documents/element-signature_payzen_61x28.=
jpg" height=3D"28" width=3D"61"
border=3D"0"></a><br data-mce-bogus=3D"1"><=
/td></tr></tbody></table><table style=3D"margin: 0;
table-layout: fixed; bo=
rder: none; width: 345pt;" data-mce-style=3D"margin: 0; table-layout: fixed=
; border: none; width: 345pt;" width=3D"460" cellspacing=3D"0"
cellpadding=
=3D"0"><tbody><tr><td><hr style=3D"background:
#293b79 none repeat scroll 0=
% 0%; width: 100%; height: 4px; color: #293b79; margin: 0px;" data-mce-styl=
e=3D"background: #293b79 none repeat scroll 0% 0%; width: 100%; height: 4px=
; color: #293b79; margin:
0px;"></td></tr></tbody></table><table
style=3D"m=
argin: 0; table-layout: fixed; border: none; width: 345pt;" data-mce-style=
=3D"margin: 0; table-layout: fixed; border: none; width: 345pt;"
width=3D"4=
60" cellspacing=3D"0"
cellpadding=3D"0"><tbody><tr><td style=3D"margin: 0; =
font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height=
: 16px; width: 30.75pt; text-align: left;" data-mce-style=3D"margin: 0; fon=
t-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 1=
6px; width: 30.75pt; text-align: left;"><div>Lyra Network, 109 rue de
l'inn=
ovation, 31670 Lab=C3=A8ge,
FRANCE</div></td></tr></tbody></table><div><br>=
</div></div><div><br></div><hr id=3D"zwchr"
data-marker=3D"__DIVIDER__"><di=
v data-marker=3D"__HEADERS__"><b>From: </b>"Gianluca
Cecchi" <gianluca.c=
ecchi(a)gmail.com&gt;<br><b>To: </b>"Nelson Lameiras"
<nelson.lameiras@lyr=
a-network.com><br><b>Cc: </b>"ovirt users"
&lt;users(a)ovirt.org&gt;<br><b=
Sent: </b>Thursday, June 8, 2017 9:36:33
AM<br><b>Subject: </b>Re: [ovirt-=
users] ISCSI storage with
multiple nics on same subnet disabled on host act=
ivation<br></div><div><br></div><div
data-marker=3D"__QUOTED_TEXT__"><div d=
ir=3D"ltr"><div class=3D"gmail_extra"><div
class=3D"gmail_quote"><blockquot=
e>On Wed, Jun 7, 2017 at 2:59 PM, Nelson Lameiras <span
dir=3D"ltr"><<a =
href=3D"mailto:nelson.lameiras@lyra-network.com"
target=3D"_blank">nelson.l=
ameiras(a)lyra-network.com</a>&gt;</span>
wrote:</blockquote><blockquote clas=
s=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid
r=
gb(204,204,204);padding-left:1ex"><div><div
style=3D"font-family:arial,helv=
etica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><div>Hello,<br></div><br>=
<div>In our oVirt hosts, we are using DELL equallogic SAN with each server =
connecting to SAN via 2 physical interfaces. Since both interfaces share th=
e same network (Equalogic limitation) we must patch the linux kernel to to =
allow iSCSI multipath with multiple NICs in the same subnet with sysctl :<b=
r></div><br><div>----------------------------------------------------------=
----------------------------------<br></div><div><pre>net.ipv4.conf.p2p1.ar=
p_ignore=3D1
net.ipv4.conf.p2p1.arp_announce=3D2
net.ipv4.conf.p2p1.rp_filter=3D2
net.ipv4.conf.p2p2.arp_ignore=3D1
net.ipv4.conf.p2p2.arp_announce=3D2
net.ipv4.conf.p2p2.rp_filter=3D2<br><br></pre></div></div></div></blockquot=
e><blockquote><div>Hello Nelson,</div><div>nice to see another
environment =
based on EQL iSCSI SAN.</div><div>The systctl.conf customized lines above a=
re not what you mean with "we must patch the linux kernel", is
it?</div><di=
v>What do you mean exactly?</div></blockquote></div><div
class=3D"gmail_quo=
te"><br data-mce-bogus=3D"1"></div><div
class=3D"gmail_quote">Sorry, by "pa=
tch" I meant "sysctl.conf configuration" indeed.
<br></div><div class=3D"gm=
ail_quote">This configuration is what we use to be able to configure 2 netw=
ork interfaces in the same
subnet.<br><div> <br></div><div><div style=
=3D"font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"=
<div><blockquote><p>------------------------------------------------------=
--------------------------------------<br><br></p></blockquote></div></div>=
</div><blockquote><div><div
style=3D"font-family:arial,helvetica,sans-serif=
;font-size:12pt;color:rgb(0,0,0)"><div><p>This works great in most
setups, =
but for a strange reason, on some of our setups, the sysctl configuration i=
s updated by VDSM when activating a host and the second interface stops wor=
king immeadiatly :</p><div>------------------------------------------------=
--------------------------------------------</div><div>vdsm.log<br></div><d=
iv><pre
id=3D"gmail-m_3172160333806361248preview"><code>2017-06-07 11:51:51=
,063+0200 INFO (jsonrpc/5) [storage.ISCSI] Setting strict mode rp_filter f=
or device 'p2p2'. (iscsi:602)
2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not conn=
ect to storageServer (hsm:2392)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer
conObj.connect()
File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect
iscsi.addIscsiNode(self._iface, self._target, self._cred)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, =
in addIscsiNode
iscsiadm.node_login(<a href=3D"http://iface.name"
target=3D"_blank">ifa=
ce.name</a>, target.address, target.iqn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 33=
7, in node_login
raise IscsiNodeError(rc, out,
err)</code></pre><pre><br></pre></div></d=
iv></div></div><div>In my environment I don't see these kinds of
problems.<=
/div><div>I'm using plain CentOS 7.3 hosts as hypervisors; is this your cas=
e too or are you using ovirt-ng nodes?</div><div><br
data-mce-bogus=3D"1"><=
/div></blockquote><div>Plain CentOS 7.3 hosts also. Built from a Minimal
ce=
ntos and adding oVirt repositories.<br></div><div><br
data-mce-bogus=3D"1">=
</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px
0px 0.8ex;b=
order-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div
style=3D"=
font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><di=
v><div><p
style=3D"margin:0px">--------------------------------------------=
------------------------------------------------</p></div></div><div><br><d=
iv>"strict mode" is enforced for second interface, and it no longuer works.=
..</div><div>Which means - at least - that there is no redundancy in case o=
f hardware faillure and this is not acceptable for our production needs.<br=
</div><br><div>What is really strange is that we
have another "twin" site =
on another geographic region with simillar
hardware configuration and same =
oVirt installation, and this problem does not happen.</div><br><div>What
ca=
n be the root cause of this behaviour? How can I correct
it?<br></div><br><=
br></div></div></div></blockquote><blockquote><div>Actually
it is very stra=
nge to me to understand how you don't have problems on the other side.</div=
<div>My workaround to get multipath with EQL in oVirt was the
one explaine=
d here with screenshots:</div><div><a
href=3D"http://lists.ovirt.org/piperm=
ail/users/2017-March/080992.html"
target=3D"_blank">http://lists.ovirt.org/=
pipermail/users/2017-March/080992.html</a><br
data-mce-bogus=3D"1"></div><d=
iv>It has been working now for about 3 months without problems. We have sev=
eral Oracle RDBMS server VMs on it, from 2virtual threads and 8Gb of ram to=
16 threads (2-4-2) and 64Gb of ram </div><div>How did you configure
=
multipath instead? As far as I know oVirt will not let you do it.</div><div=
The only way would be to use bonding but that is not
certified/supported b=
y Dell.</div><div>HIH sharing
experiences,</div><div>Gianluca</div><div><br=
data-mce-bogus=3D"1"></div><div><br
data-mce-bogus=3D"1"></div></blockquot=
e><div>I've read the link you gave and it seems to me that you are using th=
e same physical interface for SAN and vm network.<br></div><div>In my
setup=
: </div><div> - SAN network is using two disctinct physical
interfac=
es (p2pX) on same subnet (therefore the need for sysctl
patch)<br></div><di=
v> - VM network (p1pX) is using 2 other distinct physical interfaces =
on "trunk" mode<br
data-mce-bogus=3D"1"></div><div> - Administration =
network (emX) is using 2 distinct bonded network interfaces <br data-mce-bo=
gus=3D"1"></div><div><br
data-mce-bogus=3D"1"></div><div>Our setup, using a=
total of 6 network interfaces is "simpler" than yours since each interface=
has a "disctinct" role.<br
data-mce-bogus=3D"1"></div><div><br data-mce-bo=
gus=3D"1"></div><div>To make multipath work with oVirt on this
setup, we ha=
d to configure SAN network on system before installing oVirt (using sysctl.=
conf above), </div><div> - next declare 2 SAN networks on oVirt that
=
are "dragued" to each SAN interface in host network configuration,
</div><d=
iv> - and finally, in datacenter listing, chosing a datacenter and se=
lecting "iSCSI Multipathing" tab below I can add a multipath configuration =
concerning both SAN oVirt networks and relevant iqn storage targets.<br dat=
a-mce-bogus=3D"1"></div><div><br
data-mce-bogus=3D"1"></div><div>Hope it's =
clear, if not, please ask me.<br
data-mce-bogus=3D"1"></div><div><br data-m=
ce-bogus=3D"1"></div><div>It's very strange to me also since
this configura=
tion did work great on our labs. But half of production sites have the prob=
lem decribed above : second SAN interface stops working uppon host activati=
on because vdsm forces sysctl net.ipv4.conf.p2p2.rp_filter to 1 (strict mod=
e).<br data-mce-bogus=3D"1"></div><div><br
data-mce-bogus=3D"1"></div><div>=
Still searching for clues as why this behaviour...<br
data-mce-bogus=3D"1">=
</div></div></div></div><br></div></div></body></html>
------=_Part_57900107_1954785857.1496910345350
Content-Type: image/jpeg; name=element-signature_logo_lyra_115x94.jpg
Content-Disposition: attachment;
filename=element-signature_logo_lyra_115x94.jpg
Content-Transfer-Encoding: base64
Content-ID: <25cd0e595e0f025605ddb995496484dcb70629d9@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOkU1QUU4Nzk1OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOkU1QUU4Nzk0OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIAF4AcwMBEQACEQEDEQH/xACdAAEBAAIDAQEAAAAAAAAAAAAACAUH
AgMGBAEBAQADAQEAAAAAAAAAAAAAAAACBAUBAxAAAAUDAgAHDQcCBwAAAAAAAAECAwQRBQYSByFR
E7RWFzcxQSKS0nOT03RVlQgYYXEyQiMUFYEzsVJicmODOBEAAgECAwcEAgIDAAAAAAAAAAECEQNR
EwQhgbESMlIzMUGhFHEVwQUiQzT/2gAMAwEAAhEDEQA/AKpAE9b/AGQZdF3FtNpsd4l29M2DHSll
iS6w0bzsp5slqJCiLvERnTuENTRQg7bckntK15vmojj1WfMP0rc+Jy/JD7Fjt+EMueJ6bbnBN4bR
lkWdkt+XPtDaHSfjKnSHyUpTZkg+TcLSdFGRjyv3rUo0iqP8EoQkntZ7fcOy5VdrXGYxuccGW2+S
3nCecYq3oUWnU2RmfhGXAPPSXLcJNzVVT8kNVbnKKUHRmppW1/zBOSXVtZS4lpSjNtP8nLKhfcSR
alqdPXp+ERhauKKTe38nX1WfMP0qc+Jy/JHPsWO34RPLniOqz5h+lTnxOX5IfYsdvwhlzxHVZ8w/
Spz4nL8kPsWO34Qy54jqs+YfpU58Tl+SH2LHb8IZc8R1WfMP0qc+Jy/JD7Fjt+EMueJ2bCX3MH9y
LxZr7eJdxKBCkoW0/JdfaJ5mUy2a0EszL/MRHTuGGshDLTiqVf8AAst81GUGMssgAAAAE5b89s+L
+zwefPDV0fhlv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACc
t+e2fF/Z4PPnhq6Pwy38Crd60UaMotAAAAAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLI
AAAABOW/PbPi/s8Hnzw1dH4Zb+BVu9aKNGUWjAKzrGk5cnEjkq/nVo5RMbk3NOnkzd/uU0fgKvdH
rky5eb2I86rQz48iQAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLIAAAABOW/PbPi/s8Hn
zw1dH4Zb+BVu9aKNGUWjSD//AKkj+xnzJQ0V/wA2/wDkr/7D3e7GX5DiOMfzlniMTEMPIRORIJfg
tOeCladCk/nNJH94raa1GcuVnpck4qqOrJdz4dr2yazKOhDjkxhlUGMszop98i/TOlD/AE/CNX+0
x23p27nIJXKRqeeyjdnLMd27sGSzLdEO4Xd4uWifqkhDDiFut0qrUSzQlNa9wz7g9bemjK44puiI
yuNRTOzGs53hud+gvzsYjwMVnqNfLuqUTrEck69bitZmStPc1NJIz4hy5ZtKLpKskIzk36bDHRt2
9yMsuMw9v8ejybNBc5M5s5Rp5Uy4aEZuMJSai4dPCZFStBN6a3BLne1nMyT6Uffgu8GQ5DnysXuN
mbtZx461TG1ms3kyGiLWRHXToMz8Hg7nfMQvaWMYcydTsLjcqG2BSPYnbY/twy3zVw5+yNXV+GO7
gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLROmdZbAxL5g/5yey6/GjREJW2wSTc
M3YpoKms0F3VcY1bNpzscqKs5UnU9pad4cG3Fefw1UWXFO7x3mULkpaJJnoM6JNC10WREak/aXGK
8tLO1/ns2HorilsNTYzar/kOQWTa+5tmmBjk+W9cKGqimkrJS68RcCkoV/yC7clGMXcXrJI8Yptq
OBsj5oEpTiNmSRESSuKSJJFwUJlzgoKv9f1P8Hrf9Db1wh/vbVJh6tH7lhbOvi5RBpr/AEqKMXR1
PZrYaC2x3Jgbawp+IZjBkwpkWS480603rSslkRU7qTOuiqFlUlEfepw6WosO61ODK9ufLsZ27f36
RkG/8q9OwXbe3PgKciMPp0uHHS22hpxRf60o1f4VC9DlsUrWjEHWdSghllknbY/twy3zVw5+yNXV
+GO7gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLRxU00o6qQlR8ZkRhUGMyO7W7H
7FOvUpCSZgMrfMqERqNJeCgj41qokvtMTtxcpJL3OSdFU1n8vFimPwbrnF18O55DIXybhlw8ihZm
s08RLdqVOJJC5rZqqgvSJ42V7v3MpM3dTBnyWbtBQmHFnXFpLzdXOWjW9tWo2irwPpd0pWhVPBPU
R0qILTVWx+y+STuYmQj7sxHH1Kctb7NuahnMflKW2akk3J/bvGlojqtpv8ZuJP8ACR8Ag9M8dtTu
YdErdfEJTCZKrY/L5JSySbrbCdKmicdWRLeWlJK/bs8skq+ElSacJ8HVppr3OZiP2RuxFK4W5xi3
LK2T5D8NmY8baHHnWySSeTSa6to1K8JTumn3gtM6PbtQzDYYqnqTtsf24Zb5q4c/ZGrq/DHdwK1r
rZRIyiyAAAAATlvz2z4v7PB588NXR+GW/gVbvWijRlFoADGZHjVmyO2Ltd4YOTAcUlbjJLcbJRoO
qam2pCuA+GlROFxwdV6nJRT2M+q226DbLfHt8BkmIcRtLMdlNaJQgqEVTqZ/eYjKTbqwlQ5Hb4Bp
Uk4zRpUpa1FoTQ1uEaVqPg7qiUZGffDmYodSbNZ0toaTBjk02hbTaCaQSUtu/wBxCSpQkr/MXfHe
d4iiP12z2h5k2XYMdxlSkuKbW0hSTW2kkIUZGVKpSkiI+8RApPEURydtltdQpDsRlxCtepKm0mR8
qdXKkZfn/NxjnMxQ+kcOk7bH9uGW+auHP2Rq6vwx3cCta62USMosgAAAAE5b89s+L+zwefPDV0fh
lv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACb9/3mWN4cbff
WTTLUWEt1xX4UpTNeNSj+wiGtolW1LfwKt7qRuTrY226RwfSkKH1rnaz3zI4jrZ226RwfSkH1rna
xmRxHWztt0jg+lIPrXO1jMjiOtnbbpHB9KQfWudrGZHEdbO23SOD6Ug+tc7WMyOJ+o3W24WtKE5F
CUpR0SknCqZmH1rnazjuxW2pnbTfbPeGnHbXMamNtK0OLaVqJKjKtDELlqUNklQ7C5GfS6mOu+f4
XZpy4F0vMWHMbJKlsOrJKyJRVTUvtIdjZnJVSOuaXqfF1sbbdI4PpSEvrXO1nMyOJp7YiQxJ3oyi
RHcS9Hejz3GXUcKVoVOZNKi+wyOovaxUsx3cDwtdbKNGUWgAAAADX+4GzGPZveWbtcpsuO+xGTFS
iObRINCFrcIz1oWdaun3xas6qVtUSR5ztKTqeZ+lrCvely8Zj1Q9f2M8EQyEPpawr3pcvGY9UH7G
eCGQh9LWFe9Ll4zHqg/YzwQyEPpawr3pcvGY9UH7GeCGQh9LWFe9Ll4zHqg/YzwQyEc2flfwxl5t
1NzuJqbUSiI1MUqX/UOr+xmn6I5LTRaaxNg4XhNuxOHIiwX3n0SXCdWb5pMyMk6eDSlPEPDU6mV5
pteg0+nVpNI8zm+xeNZfkDt7nzprEl5Dbam2FNEgibTpKmttR97jErWslCNEkTlaUnUwP0tYV70u
XjMeqHp+xngiOQj023+zGPYReXrtbZsuQ+/GVFUiQbRoJC1ocMy0IQdatF3x5XtVK4qNInC0oups
AVT0AAAAAAAAAAAAAAAAAAAAAAAAAAAAD//Z
------=_Part_57900107_1954785857.1496910345350
Content-Type: image/jpeg; name=element-signature_logo_YouTube_32x28.jpg
Content-Disposition: attachment;
filename=element-signature_logo_YouTube_32x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <4c5918308790526a8092b3bb40ae8dab65d1777b@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOjQ1REY5Njk1OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOjQ1REY5Njk0OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAIAMBEQACEQEDEQH/xAB5AAADAQEAAAAAAAAAAAAAAAAABwgF
BgEAAgMBAAAAAAAAAAAAAAAAAAQCAwUBEAABBAECBAUFAAAAAAAAAAABAgMEBQARBiESEwcxUdIU
CEEik1UYEQACAQMCBQUAAAAAAAAAAAAAAQIRAwQhMVFhEiITQaHBYhX/2gAMAwEAAhEDEQA/AKpw
AnD5Hbg3JA35XxKqxmxUPVrJEaI862FuqkPp4IbI5lHQDNXBhFwbaW4tebqcMIvfYjURtz6eesv1
YzWz9fYh38zR7eX2+mu5lHV3VhaNOe8aTJgTHnweVXEBba1cQRkb8IeNtJbHYN9WpW+YY2T53inQ
4HfzZk6a4lmHGaiOSHl8EoT7mSOZR+gBIzUxU3Yklz+Be4+9DRl30N3dMOxjbyro9EzHW3MqupHc
L7qlaoX1SsFvlHl44moPpa6XXiW113FPuu3qrT5I7dfrJLUxllMRl15hQWjqBbiuXmTqCQlQx23F
rHdSqTrNFFZlDAou7vZa43xuSNaw7CPFZZhIiKbeSsqKkOuuE/aCNNHcexspW40a9Sm5bcmcH/J1
5+0rvxOenGf0o8GQ8DN7ZXxzv9vbpqrhy0huMQJCHlstocBKU+ITqANcru50ZRao9SUbLTqPvMwv
DAAwAMAP/9k=
------=_Part_57900107_1954785857.1496910345350
Content-Type: image/jpeg; name=element-signature_logo_LinkedIn_41x28.jpg
Content-Disposition: attachment;
filename=element-signature_logo_LinkedIn_41x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <0c1a4f5470e19b606b0a0890da695ad9d2808a60@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOjY2MEYzRTFGOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOjY2MEYzRTFFOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAKQMBEQACEQEDEQH/xAB2AAEBAQEAAAAAAAAAAAAAAAAABggH
AQEAAwEAAAAAAAAAAAAAAAAAAwQFAhAAAQMEAgICAgMBAAAAAAAAAQIDBAARBQYhEjEHMghBE1Fh
IhcRAAICAQIDCQAAAAAAAAAAAAABAgMREgQxkTIhQVGhscETMxX/2gAMAwEAAhEDEQA/ANU0Bl33
zsuzQfZcmHj8tMiMKaihDLMl1psKcQkXslSUjk8mtjZ1xdeWkVbW9RDZ/ZN8wOXkYnI7LKMuL1Lq
o+RddaIWkLHVwLAPCuf4NWYVwksqPkcNtd5V+ldr2ef7MwkaZmJsqK6p/uy9JdcbUBGcULpUog8i
9Q7uuKreEjquT1GsKxC2KAUBk37B2/6y72t1tB7X8dbIvf8Aq3mtvZfVzKlvUWWw7Dr+Dfy8rCt4
d153Z8ew0FNx3kohuxmUyFtAfFPyBUOBz+arwhKWE89L9SRtLmTuns4xj7LLaxYaGNE6SYojlJZ6
rhLUf19f89eyj4qa1t7ft449ziPWairGLQoBQHFvZ/onObht0jNxclFjMPNtNpZdS4Vj9aAk36gj
mtDb7yMIaWiGdTbySSfqnsKbdctAFh1Fm3Bwfx8fFT/ox8GcfAyk9dfX7OapuGNzj+SiPRoJcKmG
kuBRDjS2x1uAPK71DfvYzg444nUKmnk7lWcTigFAKAUAoBQCgP/Z
------=_Part_57900107_1954785857.1496910345350
Content-Type: image/jpeg; name=element-signature_logo_Twitter_42x28.jpg
Content-Disposition: attachment;
filename=element-signature_logo_Twitter_42x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <5f6dbbaff6dac1a2e25c3c96f02d8020be9c5268@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOjdDMkI3MjMyOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOjdDMkI3MjMxOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAKgMBEQACEQEDEQH/xAB3AAEBAQEBAAAAAAAAAAAAAAAABwUI
BgEBAAMBAQAAAAAAAAAAAAAAAAMEBQIBEAABAwIEBwEBAQAAAAAAAAABAgMEAAUREhMGITFBFBUH
CIEiFhEAAgIBAgQHAAAAAAAAAAAAAAERAgMxBCHBEgVBUWGBQhMV/9oADAMBAAIRAxEAPwDC9g7u
3dH33uJiNeri2wxcJQQ01JfCUNodVwCUqwSlI/AK3sGKropS0Kd7OWZMXcPsyZDdnQ7he5UFhQQ/
LZelOMoURiEqWlRAOFdumNOGqyeTYtPzFfL1df8ASm5XCTP0ey0e5eceyZu4zZc5VhjlGOFZ/cKJ
dMKNeRNhbclzrOJxQCgOSW7o5a/oKXMRFXNT5iYw/EaAUtbL61tOBKTwXglWOXrW30zgj0RUmLlb
9p3rZ/r/ANd3GxWJtm3XC8B1EO2RsEqDkng88pAP8JSnEk8ugqlt6Xy5Fa3FImyNVUI8z8nJCU7o
SOSRAA/O5qbuXx9+Rxg8ToGsssCgFAc/7q+b9xXrc11u7V3iNN3CW/JbQUuhaUuuFYBIHMA9K08e
+rWqUPgivbC25MofK25Qoq81DKjwKlJeUT+kE13+jXyZ59DKX6a9WXPYfmO+msTPJdtp6AWMuhq4
5swHPVFVN1uFkiFoS46dJSqqEgoBQCgFAKAUAoBQH//Z
------=_Part_57900107_1954785857.1496910345350
Content-Type: image/jpeg; name=element-signature_payzen_61x28.jpg
Content-Disposition: attachment; filename=element-signature_payzen_61x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <7003ef2927fa834b99af5bd467d27eb6d701e6b1@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOkU1NjczQjM2OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOkU1NjczQjM1OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAPQMBEQACEQEDEQH/xABvAAABBAMAAAAAAAAAAAAAAAAAAQUG
BwIDBAEBAAAAAAAAAAAAAAAAAAAAABAAAQQABAQEAgsBAAAAAAAAAQIDBAUAERIGITETFEEiFQfS
I1FhcYGhMlIkRMQWhhEBAAAAAAAAAAAAAAAAAAAAAP/aAAwDAQACEQMRAD8AvqZ7q7OhzZMSS9IQ
Yjy2JT/avqZbW2rSvU6lBSACOeeAljLzTzSHmVhxpxIW24k5pUlQzBBHgRgM8BBJ9fKuvcGwr3La
wgxIlfGeaagyVsJ1uOOBRUBmOSRgI0bnZgfQwdx7qDrroYZSVzBrcJ0hKM2xq+7AWBU7PTXTEShc
2svRx6MuWp5o5gjigj68BombFRKmPyTfXTPXWpzoszloaRqOelCQPKkeAwEP6lp6f6V6vYaP9Z6b
3fcr7ntuhno6vPnxwGyohb2ntbpr6Y17NdKt7Bp5+YHlPJLitKyltI0KGk+XPANVgzV1Eyyqbe3s
IMmnhsMbVaYcebDraY+XVbQ15XXFOghQVnkMBkbjcVPTUs6M5IkPbnpWa1hKlrWG7QZBp0hROnUh
xRUfHTgJDWuwtsbptVTpC1xaqhgJfkuFTji9DjoKiTmpSlq/HAcO2d11d/cuXUySJG4C0+NvbfCH
CmK2hClHUop0GQ6E+dWfAeUYBi2lY2E+zgW7twhVlHDk+7AkzFO9u2lRdjORFIEZrTmAkA+GYwDl
TwLJhjZFouxsF3NzN6r7Lj7imExHUOSHWi0TpyCdORIzzwHT/J/7n+vgLVAA5YBChBUlRSCpOelR
HEZ88sA2WG3YVhb1lnJW4VVJcXFjAgM9VxOjqKTlmVJTnp48M8Ax2+3N3J3TJu6KVAbRKisxnWpr
brh+SpasxoKf14BBD91hxE2kB+nt5Px4AEL3VGoiZSAr4qyjyOP2+fAL2nuvw/e0vDl8iT8eAbv8
Lur0vPvYXrPrfrfU6bvb59Lp6NOev83HngLBwBgDAGAMAYAwBgP/2Q==
------=_Part_57900107_1954785857.1496910345350--
------=_Part_57900106_686621279.1496910345349--