2:59 p.m.
------=_Part_57353689_1234561196.1496840388734
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Hello,=20
In our oVirt hosts, we are using DELL equallogic SAN with each server conne=
cting to SAN via 2 physical interfaces. Since both interfaces share the sam=
e network (Equalogic limitation) we must patch the linux kernel to to allow=
iSCSI multipath with multiple NICs in the same subnet with sysctl :=20
---------------------------------------------------------------------------=
-----------------=20
net.ipv4.conf.p2p1.arp_ignore=3D1
net.ipv4.conf.p2p1.arp_announce=3D2
net.ipv4.conf.p2p1.rp_filter=3D2
net.ipv4.conf.p2p2.arp_ignore=3D1
net.ipv4.conf.p2p2.arp_announce=3D2
net.ipv4.conf.p2p2.rp_filter=3D2=20
---------------------------------------------------------------------------=
-----------------=20
This works great in most setups, but for a strange reason, on some of our s=
etups, the sysctl configuration is updated by VDSM when activating a host a=
nd the second interface stops working immeadiatly :=20
---------------------------------------------------------------------------=
-----------------=20
vdsm.log=20
2017-06-07 11:51:51,063+0200 INFO (jsonrpc/5) [storage.ISCSI] Setting stri=
ct mode rp_filter for device 'p2p2'. (iscsi:602)
2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not conn=
ect to storageServer (hsm:2392)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer
conObj.connect()
File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect
iscsi.addIscsiNode(self._iface, self._target, self._cred)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, =
in addIscsiNode
iscsiadm.node_login(iface.name, target.address, target.iqn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 33=
7, in node_login
raise IscsiNodeError(rc, out, err)=20
---------------------------------------------------------------------------=
-----------------=20
"strict mode" is enforced for second interface, and it no longuer works...=
=20
Which means - at least - that there is no redundancy in case of hardware fa=
illure and this is not acceptable for our production needs.=20
What is really strange is that we have another "twin" site on another geogr=
aphic region with simillar hardware configuration and same oVirt installati=
on, and this problem does not happen.=20
What can be the root cause of this behaviour? How can I correct it?=20
our setup:=20
hostedEngine : Centor 7.3, ovirt 4.1.2=20
3 physical nodes centos 7.3, ovirt 4.1.2=20
SAN DELL Equalogic=20
cordialement, regards,=20
=09Nelson LAMEIRAS=20
Ing=C3=A9nieur Syst=C3=A8mes et R=C3=A9seaux / Systems and Networks enginee=
r=20
Tel: +33 5 32 09 09 70=20
nelson.lameiras(a)lyra-network.com=20
www.lyra-network.com | www.payzen.eu=20
=09
=09
=09
Lyra Network, 109 rue de l'innovation, 31670 Lab=C3=A8ge, FRANCE=20
------=_Part_57353689_1234561196.1496840388734
Content-Type: multipart/related;
boundary="----=_Part_57353690_157338222.1496840388734"
------=_Part_57353690_157338222.1496840388734
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable
<html><body><div style=3D"font-family: arial, helvetica, sans-serif;
font-s=
ize: 12pt; color:
#000000"><div>Hello,<br></div><div><br
data-mce-bogus=3D"=
1"></div><div>In our oVirt hosts, we are using DELL equallogic SAN
with eac=
h server connecting to SAN via 2 physical interfaces. Since both interfaces=
share the same network (Equalogic limitation) we must patch the linux kern=
el to to allow iSCSI multipath with multiple NICs in the same subnet with s=
ysctl :<br data-mce-bogus=3D"1"></div><div><br
data-mce-bogus=3D"1"></div><=
div>-----------------------------------------------------------------------=
---------------------<br
data-mce-bogus=3D"1"></div><div><pre>net.ipv4.conf=
.p2p1.arp_ignore=3D1
net.ipv4.conf.p2p1.arp_announce=3D2
net.ipv4.conf.p2p1.rp_filter=3D2
net.ipv4.conf.p2p2.arp_ignore=3D1
net.ipv4.conf.p2p2.arp_announce=3D2
net.ipv4.conf.p2p2.rp_filter=3D2<br><br></pre><p>--------------------------=
------------------------------------------------------------------<br><br><=
/p><p>This works great in most setups, but for a strange reason, on some of=
our setups, the sysctl configuration is updated by VDSM when activating a =
host and the second interface stops working immeadiatly :</p><div>---------=
---------------------------------------------------------------------------=
--------</div><div>vdsm.log<br
data-mce-bogus=3D"1"></div><div><pre id=3D"p=
review"><code>2017-06-07 11:51:51,063+0200 INFO (jsonrpc/5)
[storage.ISCSI=
] Setting strict mode rp_filter for device 'p2p2'. (iscsi:602)
2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not conn=
ect to storageServer (hsm:2392)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer
conObj.connect()
File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect
iscsi.addIscsiNode(self._iface, self._target, self._cred)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, =
in addIscsiNode
iscsiadm.node_login(iface.name, target.address, target.iqn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 33=
7, in node_login
raise IscsiNodeError(rc, out,
err)</code></pre><pre><br><br></pre><p st=
yle=3D"margin: 0px;" data-mce-style=3D"margin:
0px;">----------------------=
----------------------------------------------------------------------</p><=
/div></div><div><div><br
data-mce-bogus=3D"1"></div><div>"strict mode" is e=
nforced for second interface, and it no longuer works...</div><div>Which me=
ans - at least - that there is no redundancy in case of hardware faillure a=
nd this is not acceptable for our production needs.<br data-mce-bogus=3D"1"=
</div><div><br
data-mce-bogus=3D"1"></div><div>What is really strange is t=
hat we have another "twin" site on another geographic region with simillar
=
hardware configuration and same oVirt installation, and this problem does n=
ot happen.</div><div><br
data-mce-bogus=3D"1"></div><div>What can be the ro=
ot cause of this behaviour? How can I correct it?<br
data-mce-bogus=3D"1"><=
/div><div><br data-mce-bogus=3D"1"></div><div>our
setup:</div><div>hostedEn=
gine : Centor 7.3, ovirt 4.1.2<br></div><div>3 physical nodes centos
7.3, o=
virt 4.1.2<br data-mce-bogus=3D"1"></div><div>SAN DELL
Equalogic<br data-mc=
e-bogus=3D"1"></div></div><div><br
data-mce-bogus=3D"1"></div><div data-mar=
ker=3D"__SIG_PRE__"><div>cordialement,
regards,</div><div><br data-mce-bogu=
s=3D"1"></div><table style=3D"margin: 0px; table-layout: fixed;
border: non=
e; width: 345pt; padding: 0px;" data-mce-style=3D"margin: 0px; table-layout=
: fixed; border: none; width: 345pt; padding: 0px;" width=3D"460"
cellspaci=
ng=3D"0" cellpadding=3D"0"
border=3D"0"><tbody><tr><td style=3D"margin: 0; =
font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height=
: 16px; width: 86.25pt; text-align: left;" data-mce-style=3D"margin: 0; fon=
t-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 1=
6px; width: 86.25pt; text-align: left;" width=3D"115"
valign=3D"top"><a hre=
f=3D"https://www.lyra-network.com/" style=3D"border: none;
text-decoration:=
none; width: 100%; display: block; clear: both;" data-mce-href=3D"https://=
www.lyra-network.com/" data-mce-style=3D"border: none; text-decoration: non=
e; width: 100%; display: block; clear: both;"><img
src=3D"cid:6fa8666d679e1=
8cab5e566228d0ca3519aabec27@zimbra" alt=3D""
data-mce-src=3D"/home/admin@ly=
ra-network.com/corporate-documents/element-signature_logo_lyra_115x94.jpg... =
doc=3D"corporate-documents/element-signature_logo_lyra_115x94.jpg" height=
=3D"94" width=3D"115" border=3D"0"></a><br
data-mce-bogus=3D"1"></td><td st=
yle=3D"margin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2=
b3c7a; line-height: 16px; width: 258.75pt;" data-mce-style=3D"margin: 0; fo=
nt-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: =
16px; width: 258.75pt;" width=3D"345" valign=3D"top"><table
style=3D"margin=
: 0; width: 258.75pt; table-layout: fixed; border: none;" data-mce-style=3D=
"margin: 0; width: 258.75pt; table-layout: fixed; border: none;"
width=3D"3=
45" cellspacing=3D"0" cellpadding=3D"0"
border=3D"0"><tbody><tr><td style=
=3D"font-size: 16px; font-family: Arial,sans-serif; color: #2b3c7a; line-he=
ight: 18px; font-weight: bold;" data-mce-style=3D"font-size: 16px; font-fam=
ily: Arial,sans-serif; color: #2b3c7a; line-height: 18px; font-weight: bold=
;">Nelson LAMEIRAS</td></tr><tr><td
style=3D"font-family: Arial,sans-serif;=
color: #2b3c7a; font-size: 11px; line-height: 16px;" data-mce-style=3D"fon=
t-family: Arial,sans-serif; color: #2b3c7a; font-size: 11px; line-height: 1=
6px;">Ing=C3=A9nieur Syst=C3=A8mes et R=C3=A9seaux<span
style=3D"font-famil=
y: Arial,sans-serif; color: #337ab7; font-size: 11px;" data-mce-style=3D"fo=
nt-family: Arial,sans-serif; color: #337ab7; font-size: 11px;"> / Systems a=
nd Networks engineer</span></td></tr><tr><td
style=3D"line-height: 16px; fo=
nt-family: 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;" data-mce-s=
tyle=3D"line-height: 16px; font-family: 'Arial',sans-serif; color: #2b3c7a;=
font-size: 11px;"><span style=3D"font-family: Arial,sans-serif; color:
#2b=
3c7a; font-size: 11px;" data-mce-style=3D"font-family: Arial,sans-serif; co=
lor: #2b3c7a; font-size: 11px;">Tel: +33 5 32 09 09
70</span></td></tr><tr>=
<td style=3D"line-height: 16px; font-family: 'Arial',sans-serif; color:
#2b=
3c7a; font-size: 11px;" data-mce-style=3D"line-height: 16px; font-family:
'=
Arial',sans-serif; color: #2b3c7a; font-size: 11px;"><a
href=3D"mailto:nels=
on.lameiras(a)lyra-network.com" style=3D"font-size: 11px; font-family: Arial,=
sans-serif; line-height: 16px; text-decoration: none;" data-mce-href=3D"mai=
lto:nelson.lameiras@lyra-network.com" data-mce-style=3D"font-size: 11px; fo=
nt-family: Arial,sans-serif; line-height: 16px; text-decoration: none;">nel=
son.lameiras(a)lyra-network.com</a><br
data-mce-bogus=3D"1"></td></tr><tr><td=
style=3D"line-height: 16px; font-family: 'Arial',sans-serif; color: #2b3c7=
a; font-size: 11px;" data-mce-style=3D"line-height: 16px; font-family:
'Ari=
al',sans-serif; color: #2b3c7a; font-size: 11px;"><a
href=3D"https://www.ly=
ra-network.com/" style=3D"font-size: 11px; font-family: Arial,sans-serif; l=
ine-height: 16px; border: none; text-decoration: none;" data-mce-href=3D"ht=
tps://www.lyra-network.com/" data-mce-style=3D"font-size: 11px; font-family=
: Arial,sans-serif; line-height: 16px; border: none; text-decoration: none;=
">www.lyra-network.com</a> <a href=3D"https://payzen.eu"
style=3D"font-size=
: 11px; font-family: Arial,sans-serif; line-height: 16px; border: none; tex=
t-decoration: none;" data-mce-href=3D"https://payzen.eu"
data-mce-style=3D"=
font-size: 11px; font-family: Arial,sans-serif; line-height: 16px; border: =
none; text-decoration: none;"> | www.payzen.eu</a><br
data-mce-bogus=3D"1">=
</td></tr></tbody></table></td></tr></tbody></table><table
style=3D"margin:=
0; table-layout: fixed; border: none; width: 345pt;" data-mce-style=3D"mar=
gin: 0; table-layout: fixed; border: none; width: 345pt;" data-mce-selected=
=3D"1" width=3D"460" cellspacing=3D"0"
cellpadding=3D"0"><tbody><tr><td sty=
le=3D"margin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b=
3c7a; line-height: 16px; width: 24pt; text-align: left;" data-mce-style=3D"=
margin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; =
line-height: 16px; width: 24pt; text-align: left;" width=3D"32"
valign=3D"t=
op"><a
href=3D"https://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA" st=
yle=3D"border: none; text-decoration: none; margin: 0; float: left; display=
: inline-block; width: 32px;" data-mce-href=3D"https://www.youtube.com/chan=
nel/UCrVl1CO_Jlu3KbiRH-tQ_vA" data-mce-style=3D"border: none; text-decorati=
on: none; margin: 0; float: left; display: inline-block; width: 32px;"><img=
src=3D"cid:2806a89cb9fbba6f879bd3fd54a93b515defd6d2@zimbra" alt=3D""
data-=
mce-src=3D"/home/admin(a)lyra-network.com/corporate-documents/element-signatu=
re_logo_YouTube_32x28.jpg" doc=3D"corporate-documents/element-signature_log=
o_YouTube_32x28.jpg" height=3D"28" width=3D"32"
border=3D"0"></a><br data-m=
ce-bogus=3D"1"></td><td style=3D"margin: 0; font-size: 11px;
font-family: A=
rial,sans-serif; color: #2b3c7a; line-height: 16px; width: 30.75pt; text-al=
ign: left;" data-mce-style=3D"margin: 0; font-size: 11px; font-family: Aria=
l,sans-serif; color: #2b3c7a; line-height: 16px; width: 30.75pt; text-align=
: left;" width=3D"41" valign=3D"top"><a
href=3D"https://www.linkedin.com/co=
mpany/lyra-network_2" style=3D"border: none; text-decoration: none; margin:=
0; float: left; display: inline-block; width: 41px;" data-mce-href=3D"http=
s://www.linkedin.com/company/lyra-network_2" data-mce-style=3D"border: none=
; text-decoration: none; margin: 0; float: left; display: inline-block; wid=
th: 41px;"><img
src=3D"cid:878634ba055bb6ae851a7ef4540f754afa3ff86c@zimbra"=
data-mce-src=3D"/home/admin(a)lyra-network.com/corporate-documents/element-s=
ignature_logo_LinkedIn_41x28.jpg" doc=3D"corporate-documents/element-signat=
ure_logo_LinkedIn_41x28.jpg" height=3D"28" width=3D"41"
border=3D"0"></a><b=
r data-mce-bogus=3D"1"></td><td style=3D"margin: 0; font-size:
11px; font-f=
amily: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 31.5pt; =
text-align: left;" data-mce-style=3D"margin: 0; font-size: 11px; font-famil=
y: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 31.5pt; text=
-align: left;" width=3D"42" valign=3D"top"><a
href=3D"https://twitter.com/L=
yraNetwork" style=3D"border: none; text-decoration: none; margin: 0; float:=
left; display: inline-block; width: 42px;" data-mce-href=3D"https://twitte=
r.com/LyraNetwork" data-mce-style=3D"border: none; text-decoration: none; m=
argin: 0; float: left; display: inline-block; width: 42px;"><img
src=3D"cid=
:18a767bbdad6a1133fc4152b03ea78228ce25a14@zimbra" alt=3D""
data-mce-src=3D"=
/home/admin(a)lyra-network.com/corporate-documents/element-signature_logo_Twi=
tter_42x28.jpg" doc=3D"corporate-documents/element-signature_logo_Twitter_4=
2x28.jpg" height=3D"28" width=3D"42"
border=3D"0"></a><br data-mce-bogus=3D=
"1"></td><td style=3D"margin: 0; font-size: 11px; font-family:
Arial,sans-s=
erif; color: #2b3c7a; line-height: 16px; width: 258.75pt; text-align: left;=
" data-mce-style=3D"margin: 0; font-size: 11px; font-family: Arial,sans-ser=
if; color: #2b3c7a; line-height: 16px; width: 258.75pt; text-align: left;" =
width=3D"345" valign=3D"top"><a
href=3D"https://payzen.eu" style=3D"border:=
none; text-decoration: none; margin: 0; float: left; display: inline-block=
; width: 61px;" data-mce-href=3D"https://payzen.eu"
data-mce-style=3D"borde=
r: none; text-decoration: none; margin: 0; float: left; display: inline-blo=
ck; width: 61px;"><img
src=3D"cid:d3fc119a0aa72bf985c19ca559c993b5b4b54ca3@=
zimbra" alt=3D""
data-mce-src=3D"/home/admin(a)lyra-network.com/corporate-doc=
uments/element-signature_payzen_61x28.jpg" doc=3D"corporate-documents/eleme=
nt-signature_payzen_61x28.jpg" height=3D"28" width=3D"61"
border=3D"0"></a>=
<br
data-mce-bogus=3D"1"></td></tr></tbody></table><table
style=3D"margin: =
0; table-layout: fixed; border: none; width: 345pt;" data-mce-style=3D"marg=
in: 0; table-layout: fixed; border: none; width: 345pt;" width=3D"460"
cell=
spacing=3D"0"
cellpadding=3D"0"><tbody><tr><td><hr
style=3D"background: #29=
3b79 none repeat scroll 0% 0%; width: 100%; height: 4px; color: #293b79; ma=
rgin: 0px;" data-mce-style=3D"background: #293b79 none repeat scroll 0% 0%;=
width: 100%; height: 4px; color: #293b79; margin:
0px;"></td></tr></tbody>=
</table><table style=3D"margin: 0; table-layout: fixed; border: none;
width=
: 345pt;" data-mce-style=3D"margin: 0; table-layout: fixed; border: none; w=
idth: 345pt;" data-mce-selected=3D"1" width=3D"460"
cellspacing=3D"0" cellp=
adding=3D"0"><tbody><tr><td style=3D"margin: 0;
font-size: 11px; font-famil=
y: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 30.75pt; tex=
t-align: left;" data-mce-style=3D"margin: 0; font-size: 11px; font-family: =
Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 30.75pt; text-a=
lign: left;"><div>Lyra Network, 109 rue de l'innovation, 31670
Lab=C3=A8ge,=
FRANCE</div></td></tr></tbody></table><div><br></div></div></div></body></=
html>
------=_Part_57353690_157338222.1496840388734
Content-Type: image/jpeg; name=element-signature_logo_lyra_115x94.jpg
Content-Disposition: attachment;
filename=element-signature_logo_lyra_115x94.jpg
Content-Transfer-Encoding: base64
Content-ID: <6fa8666d679e18cab5e566228d0ca3519aabec27@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOkU1QUU4Nzk1OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOkU1QUU4Nzk0OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIAF4AcwMBEQACEQEDEQH/xACdAAEBAAIDAQEAAAAAAAAAAAAACAUH
AgMGBAEBAQADAQEAAAAAAAAAAAAAAAACBAUBAxAAAAUDAgAHDQcCBwAAAAAAAAECAwQRBQYSByFR
E7RWFzcxQSKS0nOT03RVlQgYYXEyQiMUFYEzsVJicmODOBEAAgECAwcEAgIDAAAAAAAAAAECEQNR
EwQhgbESMlIzMUGhFHEVwQUiQzT/2gAMAwEAAhEDEQA/AKpAE9b/AGQZdF3FtNpsd4l29M2DHSll
iS6w0bzsp5slqJCiLvERnTuENTRQg7bckntK15vmojj1WfMP0rc+Jy/JD7Fjt+EMueJ6bbnBN4bR
lkWdkt+XPtDaHSfjKnSHyUpTZkg+TcLSdFGRjyv3rUo0iqP8EoQkntZ7fcOy5VdrXGYxuccGW2+S
3nCecYq3oUWnU2RmfhGXAPPSXLcJNzVVT8kNVbnKKUHRmppW1/zBOSXVtZS4lpSjNtP8nLKhfcSR
alqdPXp+ERhauKKTe38nX1WfMP0qc+Jy/JHPsWO34RPLniOqz5h+lTnxOX5IfYsdvwhlzxHVZ8w/
Spz4nL8kPsWO34Qy54jqs+YfpU58Tl+SH2LHb8IZc8R1WfMP0qc+Jy/JD7Fjt+EMueJ2bCX3MH9y
LxZr7eJdxKBCkoW0/JdfaJ5mUy2a0EszL/MRHTuGGshDLTiqVf8AAst81GUGMssgAAAAE5b89s+L
+zwefPDV0fhlv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACc
t+e2fF/Z4PPnhq6Pwy38Crd60UaMotAAAAAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLI
AAAABOW/PbPi/s8Hnzw1dH4Zb+BVu9aKNGUWjAKzrGk5cnEjkq/nVo5RMbk3NOnkzd/uU0fgKvdH
rky5eb2I86rQz48iQAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLIAAAABOW/PbPi/s8Hn
zw1dH4Zb+BVu9aKNGUWjSD//AKkj+xnzJQ0V/wA2/wDkr/7D3e7GX5DiOMfzlniMTEMPIRORIJfg
tOeCladCk/nNJH94raa1GcuVnpck4qqOrJdz4dr2yazKOhDjkxhlUGMszop98i/TOlD/AE/CNX+0
x23p27nIJXKRqeeyjdnLMd27sGSzLdEO4Xd4uWifqkhDDiFut0qrUSzQlNa9wz7g9bemjK44puiI
yuNRTOzGs53hud+gvzsYjwMVnqNfLuqUTrEck69bitZmStPc1NJIz4hy5ZtKLpKskIzk36bDHRt2
9yMsuMw9v8ejybNBc5M5s5Rp5Uy4aEZuMJSai4dPCZFStBN6a3BLne1nMyT6Uffgu8GQ5DnysXuN
mbtZx461TG1ms3kyGiLWRHXToMz8Hg7nfMQvaWMYcydTsLjcqG2BSPYnbY/twy3zVw5+yNXV+GO7
gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLROmdZbAxL5g/5yey6/GjREJW2wSTc
M3YpoKms0F3VcY1bNpzscqKs5UnU9pad4cG3Fefw1UWXFO7x3mULkpaJJnoM6JNC10WREak/aXGK
8tLO1/ns2HorilsNTYzar/kOQWTa+5tmmBjk+W9cKGqimkrJS68RcCkoV/yC7clGMXcXrJI8Yptq
OBsj5oEpTiNmSRESSuKSJJFwUJlzgoKv9f1P8Hrf9Db1wh/vbVJh6tH7lhbOvi5RBpr/AEqKMXR1
PZrYaC2x3Jgbawp+IZjBkwpkWS480603rSslkRU7qTOuiqFlUlEfepw6WosO61ODK9ufLsZ27f36
RkG/8q9OwXbe3PgKciMPp0uHHS22hpxRf60o1f4VC9DlsUrWjEHWdSghllknbY/twy3zVw5+yNXV
+GO7gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLRxU00o6qQlR8ZkRhUGMyO7W7H
7FOvUpCSZgMrfMqERqNJeCgj41qokvtMTtxcpJL3OSdFU1n8vFimPwbrnF18O55DIXybhlw8ihZm
s08RLdqVOJJC5rZqqgvSJ42V7v3MpM3dTBnyWbtBQmHFnXFpLzdXOWjW9tWo2irwPpd0pWhVPBPU
R0qILTVWx+y+STuYmQj7sxHH1Kctb7NuahnMflKW2akk3J/bvGlojqtpv8ZuJP8ACR8Ag9M8dtTu
YdErdfEJTCZKrY/L5JSySbrbCdKmicdWRLeWlJK/bs8skq+ElSacJ8HVppr3OZiP2RuxFK4W5xi3
LK2T5D8NmY8baHHnWySSeTSa6to1K8JTumn3gtM6PbtQzDYYqnqTtsf24Zb5q4c/ZGrq/DHdwK1r
rZRIyiyAAAAATlvz2z4v7PB588NXR+GW/gVbvWijRlFoADGZHjVmyO2Ltd4YOTAcUlbjJLcbJRoO
qam2pCuA+GlROFxwdV6nJRT2M+q226DbLfHt8BkmIcRtLMdlNaJQgqEVTqZ/eYjKTbqwlQ5Hb4Bp
Uk4zRpUpa1FoTQ1uEaVqPg7qiUZGffDmYodSbNZ0toaTBjk02hbTaCaQSUtu/wBxCSpQkr/MXfHe
d4iiP12z2h5k2XYMdxlSkuKbW0hSTW2kkIUZGVKpSkiI+8RApPEURydtltdQpDsRlxCtepKm0mR8
qdXKkZfn/NxjnMxQ+kcOk7bH9uGW+auHP2Rq6vwx3cCta62USMosgAAAAE5b89s+L+zwefPDV0fh
lv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACb9/3mWN4cbff
WTTLUWEt1xX4UpTNeNSj+wiGtolW1LfwKt7qRuTrY226RwfSkKH1rnaz3zI4jrZ226RwfSkH1rna
xmRxHWztt0jg+lIPrXO1jMjiOtnbbpHB9KQfWudrGZHEdbO23SOD6Ug+tc7WMyOJ+o3W24WtKE5F
CUpR0SknCqZmH1rnazjuxW2pnbTfbPeGnHbXMamNtK0OLaVqJKjKtDELlqUNklQ7C5GfS6mOu+f4
XZpy4F0vMWHMbJKlsOrJKyJRVTUvtIdjZnJVSOuaXqfF1sbbdI4PpSEvrXO1nMyOJp7YiQxJ3oyi
RHcS9Hejz3GXUcKVoVOZNKi+wyOovaxUsx3cDwtdbKNGUWgAAAADX+4GzGPZveWbtcpsuO+xGTFS
iObRINCFrcIz1oWdaun3xas6qVtUSR5ztKTqeZ+lrCvely8Zj1Q9f2M8EQyEPpawr3pcvGY9UH7G
eCGQh9LWFe9Ll4zHqg/YzwQyEPpawr3pcvGY9UH7GeCGQh9LWFe9Ll4zHqg/YzwQyEc2flfwxl5t
1NzuJqbUSiI1MUqX/UOr+xmn6I5LTRaaxNg4XhNuxOHIiwX3n0SXCdWb5pMyMk6eDSlPEPDU6mV5
pteg0+nVpNI8zm+xeNZfkDt7nzprEl5Dbam2FNEgibTpKmttR97jErWslCNEkTlaUnUwP0tYV70u
XjMeqHp+xngiOQj023+zGPYReXrtbZsuQ+/GVFUiQbRoJC1ocMy0IQdatF3x5XtVK4qNInC0oups
AVT0AAAAAAAAAAAAAAAAAAAAAAAAAAAAD//Z
------=_Part_57353690_157338222.1496840388734
Content-Type: image/jpeg; name=element-signature_logo_YouTube_32x28.jpg
Content-Disposition: attachment;
filename=element-signature_logo_YouTube_32x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <2806a89cb9fbba6f879bd3fd54a93b515defd6d2@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOjQ1REY5Njk1OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOjQ1REY5Njk0OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAIAMBEQACEQEDEQH/xAB5AAADAQEAAAAAAAAAAAAAAAAABwgF
BgEAAgMBAAAAAAAAAAAAAAAAAAQCAwUBEAABBAECBAUFAAAAAAAAAAABAgMEBQARBiESEwcxUdIU
CEEik1UYEQACAQMCBQUAAAAAAAAAAAAAAQIRAwQhMVFhEiITQaHBYhX/2gAMAwEAAhEDEQA/AKpw
AnD5Hbg3JA35XxKqxmxUPVrJEaI862FuqkPp4IbI5lHQDNXBhFwbaW4tebqcMIvfYjURtz6eesv1
YzWz9fYh38zR7eX2+mu5lHV3VhaNOe8aTJgTHnweVXEBba1cQRkb8IeNtJbHYN9WpW+YY2T53inQ
4HfzZk6a4lmHGaiOSHl8EoT7mSOZR+gBIzUxU3Yklz+Be4+9DRl30N3dMOxjbyro9EzHW3MqupHc
L7qlaoX1SsFvlHl44moPpa6XXiW113FPuu3qrT5I7dfrJLUxllMRl15hQWjqBbiuXmTqCQlQx23F
rHdSqTrNFFZlDAou7vZa43xuSNaw7CPFZZhIiKbeSsqKkOuuE/aCNNHcexspW40a9Sm5bcmcH/J1
5+0rvxOenGf0o8GQ8DN7ZXxzv9vbpqrhy0huMQJCHlstocBKU+ITqANcru50ZRao9SUbLTqPvMwv
DAAwAMAP/9k=
------=_Part_57353690_157338222.1496840388734
Content-Type: image/jpeg; name=element-signature_logo_LinkedIn_41x28.jpg
Content-Disposition: attachment;
filename=element-signature_logo_LinkedIn_41x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <878634ba055bb6ae851a7ef4540f754afa3ff86c@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOjY2MEYzRTFGOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOjY2MEYzRTFFOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAKQMBEQACEQEDEQH/xAB2AAEBAQEAAAAAAAAAAAAAAAAABggH
AQEAAwEAAAAAAAAAAAAAAAAAAwQFAhAAAQMEAgICAgMBAAAAAAAAAQIDBAARBQYhEjEHMghBE1Fh
IhcRAAICAQIDCQAAAAAAAAAAAAABAgMREgQxkTIhQVGhscETMxX/2gAMAwEAAhEDEQA/ANU0Bl33
zsuzQfZcmHj8tMiMKaihDLMl1psKcQkXslSUjk8mtjZ1xdeWkVbW9RDZ/ZN8wOXkYnI7LKMuL1Lq
o+RddaIWkLHVwLAPCuf4NWYVwksqPkcNtd5V+ldr2ef7MwkaZmJsqK6p/uy9JdcbUBGcULpUog8i
9Q7uuKreEjquT1GsKxC2KAUBk37B2/6y72t1tB7X8dbIvf8Aq3mtvZfVzKlvUWWw7Dr+Dfy8rCt4
d153Z8ew0FNx3kohuxmUyFtAfFPyBUOBz+arwhKWE89L9SRtLmTuns4xj7LLaxYaGNE6SYojlJZ6
rhLUf19f89eyj4qa1t7ft449ziPWairGLQoBQHFvZ/onObht0jNxclFjMPNtNpZdS4Vj9aAk36gj
mtDb7yMIaWiGdTbySSfqnsKbdctAFh1Fm3Bwfx8fFT/ox8GcfAyk9dfX7OapuGNzj+SiPRoJcKmG
kuBRDjS2x1uAPK71DfvYzg444nUKmnk7lWcTigFAKAUAoBQCgP/Z
------=_Part_57353690_157338222.1496840388734
Content-Type: image/jpeg; name=element-signature_logo_Twitter_42x28.jpg
Content-Disposition: attachment;
filename=element-signature_logo_Twitter_42x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <18a767bbdad6a1133fc4152b03ea78228ce25a14@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOjdDMkI3MjMyOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOjdDMkI3MjMxOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAKgMBEQACEQEDEQH/xAB3AAEBAQEBAAAAAAAAAAAAAAAABwUI
BgEBAAMBAQAAAAAAAAAAAAAAAAMEBQIBEAABAwIEBwEBAQAAAAAAAAABAgMEAAUREhMGITFBFBUH
CIEiFhEAAgIBAgQHAAAAAAAAAAAAAAERAgMxBCHBEgVBUWGBQhMV/9oADAMBAAIRAxEAPwDC9g7u
3dH33uJiNeri2wxcJQQ01JfCUNodVwCUqwSlI/AK3sGKropS0Kd7OWZMXcPsyZDdnQ7he5UFhQQ/
LZelOMoURiEqWlRAOFdumNOGqyeTYtPzFfL1df8ASm5XCTP0ey0e5eceyZu4zZc5VhjlGOFZ/cKJ
dMKNeRNhbclzrOJxQCgOSW7o5a/oKXMRFXNT5iYw/EaAUtbL61tOBKTwXglWOXrW30zgj0RUmLlb
9p3rZ/r/ANd3GxWJtm3XC8B1EO2RsEqDkng88pAP8JSnEk8ugqlt6Xy5Fa3FImyNVUI8z8nJCU7o
SOSRAA/O5qbuXx9+Rxg8ToGsssCgFAc/7q+b9xXrc11u7V3iNN3CW/JbQUuhaUuuFYBIHMA9K08e
+rWqUPgivbC25MofK25Qoq81DKjwKlJeUT+kE13+jXyZ59DKX6a9WXPYfmO+msTPJdtp6AWMuhq4
5swHPVFVN1uFkiFoS46dJSqqEgoBQCgFAKAUAoBQH//Z
------=_Part_57353690_157338222.1496840388734
Content-Type: image/jpeg; name=element-signature_payzen_61x28.jpg
Content-Disposition: attachment; filename=element-signature_payzen_61x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <d3fc119a0aa72bf985c19ca559c993b5b4b54ca3@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOkU1NjczQjM2OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOkU1NjczQjM1OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAPQMBEQACEQEDEQH/xABvAAABBAMAAAAAAAAAAAAAAAAAAQUG
BwIDBAEBAAAAAAAAAAAAAAAAAAAAABAAAQQABAQEAgsBAAAAAAAAAQIDBAUAERIGITETFEEiFQfS
I1FhcYGhMlIkRMQWhhEBAAAAAAAAAAAAAAAAAAAAAP/aAAwDAQACEQMRAD8AvqZ7q7OhzZMSS9IQ
Yjy2JT/avqZbW2rSvU6lBSACOeeAljLzTzSHmVhxpxIW24k5pUlQzBBHgRgM8BBJ9fKuvcGwr3La
wgxIlfGeaagyVsJ1uOOBRUBmOSRgI0bnZgfQwdx7qDrroYZSVzBrcJ0hKM2xq+7AWBU7PTXTEShc
2svRx6MuWp5o5gjigj68BombFRKmPyTfXTPXWpzoszloaRqOelCQPKkeAwEP6lp6f6V6vYaP9Z6b
3fcr7ntuhno6vPnxwGyohb2ntbpr6Y17NdKt7Bp5+YHlPJLitKyltI0KGk+XPANVgzV1Eyyqbe3s
IMmnhsMbVaYcebDraY+XVbQ15XXFOghQVnkMBkbjcVPTUs6M5IkPbnpWa1hKlrWG7QZBp0hROnUh
xRUfHTgJDWuwtsbptVTpC1xaqhgJfkuFTji9DjoKiTmpSlq/HAcO2d11d/cuXUySJG4C0+NvbfCH
CmK2hClHUop0GQ6E+dWfAeUYBi2lY2E+zgW7twhVlHDk+7AkzFO9u2lRdjORFIEZrTmAkA+GYwDl
TwLJhjZFouxsF3NzN6r7Lj7imExHUOSHWi0TpyCdORIzzwHT/J/7n+vgLVAA5YBChBUlRSCpOelR
HEZ88sA2WG3YVhb1lnJW4VVJcXFjAgM9VxOjqKTlmVJTnp48M8Ax2+3N3J3TJu6KVAbRKisxnWpr
brh+SpasxoKf14BBD91hxE2kB+nt5Px4AEL3VGoiZSAr4qyjyOP2+fAL2nuvw/e0vDl8iT8eAbv8
Lur0vPvYXrPrfrfU6bvb59Lp6NOev83HngLBwBgDAGAMAYAwBgP/2Q==
------=_Part_57353690_157338222.1496840388734--
------=_Part_57353689_1234561196.1496840388734--
9:36 a.m.
On Wed, Jun 7, 2017 at 2:59 PM, Nelson Lameiras <
nelson.lameiras(a)lyra-network.com> wrote:
Hello,
In our oVirt hosts, we are using DELL equallogic SAN with each server
connecting to SAN via 2 physical interfaces. Since both interfaces share
the same network (Equalogic limitation) we must patch the linux kernel to
to allow iSCSI multipath with multiple NICs in the same subnet with sysctl :
------------------------------------------------------------
--------------------------------
net.ipv4.conf.p2p1.arp_ignore=1
net.ipv4.conf.p2p1.arp_announce=2
net.ipv4.conf.p2p1.rp_filter=2
net.ipv4.conf.p2p2.arp_ignore=1
net.ipv4.conf.p2p2.arp_announce=2
net.ipv4.conf.p2p2.rp_filter=2
Hello Nelson,
nice to see another environment based on EQL iSCSI SAN.
The systctl.conf customized lines above are not what you mean with "we must
patch the linux kernel", is it?
What do you mean exactly?
------------------------------------------------------------
--------------------------------
This works great in most setups, but for a strange reason, on some of our
setups, the sysctl configuration is updated by VDSM when activating a host
and the second interface stops working immeadiatly :
------------------------------------------------------------
--------------------------------
vdsm.log
2017-06-07 11:51:51,063+0200 INFO (jsonrpc/5) [storage.ISCSI] Setting strict mode
rp_filter for device 'p2p2'. (iscsi:602)
2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not connect to
storageServer (hsm:2392)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer
conObj.connect()
File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect
iscsi.addIscsiNode(self._iface, self._target, self._cred)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, in
addIscsiNode
iscsiadm.node_login(iface.name, target.address, target.iqn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 337,
in node_login
raise IscsiNodeError(rc, out, err)
In my environment I don't see these kinds of problems.
I'm using plain CentOS 7.3 hosts as hypervisors; is this your case too or
are you using ovirt-ng nodes?
------------------------------------------------------------
--------------------------------
"strict mode" is enforced for second interface, and it no longuer works...
Which means - at least - that there is no redundancy in case of hardware
faillure and this is not acceptable for our production needs.
What is really strange is that we have another "twin" site on another
geographic region with simillar hardware configuration and same oVirt
installation, and this problem does not happen.
What can be the root cause of this behaviour? How can I correct it?
Actually it is very strange to me to understand how you don't have problems
on the other side.
My workaround to get multipath with EQL in oVirt was the one explained here
with screenshots:
http://lists.ovirt.org/pipermail/users/2017-March/080992.html
It has been working now for about 3 months without problems. We have
several Oracle RDBMS server VMs on it, from 2virtual threads and 8Gb of ram
to 16 threads (2-4-2) and 64Gb of ram
How did you configure multipath instead? As far as I know oVirt will not
let you do it.
The only way would be to use bonding but that is not certified/supported by
Dell.
HIH sharing experiences,
Gianluca
10:25 a.m.
------=_Part_57900106_686621279.1496910345349
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Hi Gianluca,=20
My answers below,=20
cordialement, regards,=20
=09Nelson LAMEIRAS=20
Ing=C3=A9nieur Syst=C3=A8mes et R=C3=A9seaux / Systems and Networks enginee=
r=20
Tel: +33 5 32 09 09 70=20
nelson.lameiras(a)lyra-network.com=20
www.lyra-network.com | www.payzen.eu=20
=09
=09
=09
Lyra Network, 109 rue de l'innovation, 31670 Lab=C3=A8ge, FRANCE=20
From: "Gianluca Cecchi" <gianluca.cecchi(a)gmail.com>=20
To: "Nelson Lameiras" <nelson.lameiras(a)lyra-network.com>=20
Cc: "ovirt users" <users(a)ovirt.org>=20
Sent: Thursday, June 8, 2017 9:36:33 AM=20
Subject: Re: [ovirt-users] ISCSI storage with multiple nics on same subnet =
disabled on host activation=20
On Wed, Jun 7, 2017 at 2:59 PM, Nelson Lameiras < nelson.lameiras@lyra-netw=
ork.com > wrote:=20
BQ_BEGIN
Hello,=20
In our oVirt hosts, we are using DELL equallogic SAN with each server conne=
cting to SAN via 2 physical interfaces. Since both interfaces share the sam=
e network (Equalogic limitation) we must patch the linux kernel to to allow=
iSCSI multipath with multiple NICs in the same subnet with sysctl :=20
---------------------------------------------------------------------------=
-----------------=20
net.ipv4.conf.p2p1.arp_ignore=3D1
net.ipv4.conf.p2p1.arp_announce=3D2
net.ipv4.conf.p2p1.rp_filter=3D2
net.ipv4.conf.p2p2.arp_ignore=3D1
net.ipv4.conf.p2p2.arp_announce=3D2
net.ipv4.conf.p2p2.rp_filter=3D2=20
BQ_END
BQ_BEGIN
Hello Nelson,=20
nice to see another environment based on EQL iSCSI SAN.=20
The systctl.conf customized lines above are not what you mean with "we must=
patch the linux kernel", is it?=20
What do you mean exactly?=20
BQ_END
Sorry, by "patch" I meant "sysctl.conf configuration" indeed.=20
This configuration is what we use to be able to configure 2 network interfa=
ces in the same subnet.=20
BQ_BEGIN
---------------------------------------------------------------------------=
-----------------=20
BQ_END
BQ_BEGIN
This works great in most setups, but for a strange reason, on some of our s=
etups, the sysctl configuration is updated by VDSM when activating a host a=
nd the second interface stops working immeadiatly :=20
---------------------------------------------------------------------------=
-----------------=20
vdsm.log=20
2017-06-07 11:51:51,063+0200 INFO (jsonrpc/5) [storage.ISCSI] Setting stri=
ct mode rp_filter for device 'p2p2'. (iscsi:602)
2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not conn=
ect to storageServer (hsm:2392)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer
conObj.connect()
File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect
iscsi.addIscsiNode(self._iface, self._target, self._cred)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, =
in addIscsiNode
iscsiadm.node_login( iface.name , target.address, target.iqn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 33=
7, in node_login
raise IscsiNodeError(rc, out, err)=20
In my environment I don't see these kinds of problems.=20
I'm using plain CentOS 7.3 hosts as hypervisors; is this your case too or a=
re you using ovirt-ng nodes?=20
BQ_END
Plain CentOS 7.3 hosts also. Built from a Minimal centos and adding oVirt r=
epositories.=20
BQ_BEGIN
---------------------------------------------------------------------------=
-----------------=20
"strict mode" is enforced for second interface, and it no longuer works...=
=20
Which means - at least - that there is no redundancy in case of hardware fa=
illure and this is not acceptable for our production needs.=20
What is really strange is that we have another "twin" site on another geogr=
aphic region with simillar hardware configuration and same oVirt installati=
on, and this problem does not happen.=20
What can be the root cause of this behaviour? How can I correct it?=20
BQ_END
BQ_BEGIN
Actually it is very strange to me to understand how you don't have problems=
on the other side.=20
My workaround to get multipath with EQL in oVirt was the one explained here=
with screenshots:=20
http://lists.ovirt.org/pipermail/users/2017-March/080992.html=20
It has been working now for about 3 months without problems. We have severa=
l Oracle RDBMS server VMs on it, from 2virtual threads and 8Gb of ram to 16=
threads (2-4-2) and 64Gb of ram=20
How did you configure multipath instead? As far as I know oVirt will not le=
t you do it.=20
The only way would be to use bonding but that is not certified/supported by=
Dell.=20
HIH sharing experiences,=20
Gianluca=20
BQ_END
I've read the link you gave and it seems to me that you are using the same =
physical interface for SAN and vm network.=20
In my setup :=20
- SAN network is using two disctinct physical interfaces (p2pX) on same sub=
net (therefore the need for sysctl patch)=20
- VM network (p1pX) is using 2 other distinct physical interfaces on "trunk=
" mode=20
- Administration network (emX) is using 2 distinct bonded network interface=
s=20
Our setup, using a total of 6 network interfaces is "simpler" than yours si=
nce each interface has a "disctinct" role.=20
To make multipath work with oVirt on this setup, we had to configure SAN ne=
twork on system before installing oVirt (using sysctl.conf above),=20
- next declare 2 SAN networks on oVirt that are "dragued" to each SAN inter=
face in host network configuration,=20
- and finally, in datacenter listing, chosing a datacenter and selecting "i=
SCSI Multipathing" tab below I can add a multipath configuration concerning=
both SAN oVirt networks and relevant iqn storage targets.=20
Hope it's clear, if not, please ask me.=20
It's very strange to me also since this configuration did work great on our=
labs. But half of production sites have the problem decribed above : secon=
d SAN interface stops working uppon host activation because vdsm forces sys=
ctl net.ipv4.conf.p2p2.rp_filter to 1 (strict mode).=20
Still searching for clues as why this behaviour...=20
------=_Part_57900106_686621279.1496910345349
Content-Type: multipart/related;
boundary="----=_Part_57900107_1954785857.1496910345350"
------=_Part_57900107_1954785857.1496910345350
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable
<html><body><div style=3D"font-family: arial, helvetica, sans-serif;
font-s=
ize: 12pt; color: #000000"><div>Hi
Gianluca,<br></div><div><br data-mce-bog=
us=3D"1"></div><div>My answers below,<br
data-mce-bogus=3D"1"></div><div><b=
r></div><div data-marker=3D"__SIG_PRE__"><div>cordialement,
regards,</div><=
div><br data-mce-bogus=3D"1"></div><table
style=3D"margin: 0px; table-layou=
t: fixed; border: none; width: 345pt; padding: 0px;" data-mce-style=3D"marg=
in: 0px; table-layout: fixed; border: none; width: 345pt; padding: 0px;" da=
ta-mce-selected=3D"1" width=3D"460" cellspacing=3D"0"
cellpadding=3D"0" bor=
der=3D"0"><tbody><tr><td style=3D"margin: 0; font-size:
11px; font-family: =
Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 86.25pt; text-a=
lign: left;" data-mce-style=3D"margin: 0; font-size: 11px; font-family: Ari=
al,sans-serif; color: #2b3c7a; line-height: 16px; width: 86.25pt; text-alig=
n: left;" width=3D"115" valign=3D"top"><a
href=3D"https://www.lyra-network.=
com/" style=3D"border: none; text-decoration: none; width: 100%; display: b=
lock; clear: both;" data-mce-href=3D"https://www.lyra-network.com/"
data-mc=
e-style=3D"border: none; text-decoration: none; width: 100%; display: block=
; clear: both;"><img
src=3D"cid:25cd0e595e0f025605ddb995496484dcb70629d9@zi=
mbra" alt=3D""
data-mce-src=3D"/home/admin(a)lyra-network.com/corporate-docum=
ents/element-signature_logo_lyra_115x94.jpg" doc=3D"corporate-documents/ele=
ment-signature_logo_lyra_115x94.jpg" height=3D"94" width=3D"115"
border=3D"=
0"></a><br data-mce-bogus=3D"1"></td><td
style=3D"margin: 0; font-size: 11p=
x; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width:=
258.75pt;" data-mce-style=3D"margin: 0; font-size: 11px; font-family: Aria=
l,sans-serif; color: #2b3c7a; line-height: 16px; width: 258.75pt;" width=3D=
"345" valign=3D"top"><table style=3D"margin: 0; width:
258.75pt; table-layo=
ut: fixed; border: none;" data-mce-style=3D"margin: 0; width: 258.75pt; tab=
le-layout: fixed; border: none;" width=3D"345" cellspacing=3D"0"
cellpaddin=
g=3D"0" border=3D"0"><tbody><tr><td
style=3D"font-size: 16px; font-family: =
Arial,sans-serif; color: #2b3c7a; line-height: 18px; font-weight: bold;" da=
ta-mce-style=3D"font-size: 16px; font-family: Arial,sans-serif; color: #2b3=
c7a; line-height: 18px; font-weight: bold;">Nelson
LAMEIRAS</td></tr><tr><t=
d style=3D"font-family: Arial,sans-serif; color: #2b3c7a; font-size: 11px; =
line-height: 16px;" data-mce-style=3D"font-family: Arial,sans-serif; color:=
#2b3c7a; font-size: 11px; line-height: 16px;">Ing=C3=A9nieur Syst=C3=A8mes=
et R=C3=A9seaux<span style=3D"font-family: Arial,sans-serif; color: #337ab=
7; font-size: 11px;" data-mce-style=3D"font-family: Arial,sans-serif; color=
: #337ab7; font-size: 11px;"> / Systems and Networks
engineer</span></td></=
tr><tr><td style=3D"line-height: 16px; font-family:
'Arial',sans-serif; col=
or: #2b3c7a; font-size: 11px;" data-mce-style=3D"line-height: 16px; font-fa=
mily: 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;"><span
style=3D"=
font-family: Arial,sans-serif; color: #2b3c7a; font-size: 11px;" data-mce-s=
tyle=3D"font-family: Arial,sans-serif; color: #2b3c7a; font-size: 11px;">Te=
l: +33 5 32 09 09 70</span></td></tr><tr><td
style=3D"line-height: 16px; fo=
nt-family: 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;" data-mce-s=
tyle=3D"line-height: 16px; font-family: 'Arial',sans-serif; color: #2b3c7a;=
font-size: 11px;"><a
href=3D"mailto:nelson.lameiras@lyra-network.com" styl=
e=3D"font-size: 11px; font-family: Arial,sans-serif; line-height: 16px; tex=
t-decoration: none;" data-mce-href=3D"mailto:nelson.lameiras@lyra-network.c=
om" data-mce-style=3D"font-size: 11px; font-family: Arial,sans-serif; line-=
height: 16px; text-decoration:
none;">nelson.lameiras(a)lyra-network.com</a><=
br data-mce-bogus=3D"1"></td></tr><tr><td
style=3D"line-height: 16px; font-=
family: 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;" data-mce-styl=
e=3D"line-height: 16px; font-family: 'Arial',sans-serif; color: #2b3c7a; fo=
nt-size: 11px;"><a href=3D"https://www.lyra-network.com/"
style=3D"font-siz=
e: 11px; font-family: Arial,sans-serif; line-height: 16px; border: none; te=
xt-decoration: none;" data-mce-href=3D"https://www.lyra-network.com/"
data-=
mce-style=3D"font-size: 11px; font-family: Arial,sans-serif; line-height: 1=
6px; border: none; text-decoration: none;">www.lyra-network.com</a> <a
href=
=3D"https://payzen.eu" style=3D"font-size: 11px; font-family:
Arial,sans-se=
rif; line-height: 16px; border: none; text-decoration: none;" data-mce-href=
=3D"https://payzen.eu" data-mce-style=3D"font-size: 11px; font-family:
Aria=
l,sans-serif; line-height: 16px; border: none; text-decoration: none;"> | w=
ww.payzen.eu</a><br
data-mce-bogus=3D"1"></td></tr></tbody></table></td></t=
r></tbody></table><table style=3D"margin: 0; table-layout: fixed;
border: n=
one; width: 345pt;" data-mce-style=3D"margin: 0; table-layout: fixed; borde=
r: none; width: 345pt;" data-mce-selected=3D"1" width=3D"460"
cellspacing=
=3D"0" cellpadding=3D"0"><tbody><tr><td
style=3D"margin: 0; font-size: 11px=
; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: =
24pt; text-align: left;" data-mce-style=3D"margin: 0; font-size: 11px; font=
-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 24pt; =
text-align: left;" width=3D"32" valign=3D"top"><a
href=3D"https://www.youtu=
be.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA" style=3D"border: none; text-decora=
tion: none; margin: 0; float: left; display: inline-block; width: 32px;" da=
ta-mce-href=3D"https://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ... da=
ta-mce-style=3D"border: none; text-decoration: none; margin: 0; float: left=
; display: inline-block; width: 32px;"><img
src=3D"cid:4c5918308790526a8092=
b3bb40ae8dab65d1777b@zimbra" alt=3D""
data-mce-src=3D"/home/admin@lyra-netw=
ork.com/corporate-documents/element-signature_logo_YouTube_32x28.jpg" doc=
=3D"corporate-documents/element-signature_logo_YouTube_32x28.jpg" height=3D=
"28" width=3D"32" border=3D"0"></a><br
data-mce-bogus=3D"1"></td><td style=
=3D"margin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c=
7a; line-height: 16px; width: 30.75pt; text-align: left;" data-mce-style=3D=
"margin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a;=
line-height: 16px; width: 30.75pt; text-align: left;" width=3D"41"
valign=
=3D"top"><a
href=3D"https://www.linkedin.com/company/lyra-network_2" style=
=3D"border: none; text-decoration: none; margin: 0; float: left; display: i=
nline-block; width: 41px;" data-mce-href=3D"https://www.linkedin.com/compan=
y/lyra-network_2" data-mce-style=3D"border: none; text-decoration: none; ma=
rgin: 0; float: left; display: inline-block; width: 41px;"><img
src=3D"cid:=
0c1a4f5470e19b606b0a0890da695ad9d2808a60@zimbra" data-mce-src=3D"/home/admi=
n(a)lyra-network.com/corporate-documents/element-signature_logo_LinkedIn_41x2=
8.jpg" doc=3D"corporate-documents/element-signature_logo_LinkedIn_41x28.jpg=
" height=3D"28" width=3D"41"
border=3D"0"></a><br data-mce-bogus=3D"1"></td=
<td style=3D"margin: 0; font-size: 11px; font-family:
Arial,sans-serif; co=
lor: #2b3c7a; line-height: 16px; width: 31.5pt; text-align:
left;" data-mce=
-style=3D"margin: 0; font-size: 11px; font-family: Arial,sans-serif; color:=
#2b3c7a; line-height: 16px; width: 31.5pt; text-align: left;"
width=3D"42"=
valign=3D"top"><a href=3D"https://twitter.com/LyraNetwork"
style=3D"border=
: none; text-decoration: none; margin: 0; float: left; display: inline-bloc=
k; width: 42px;" data-mce-href=3D"https://twitter.com/LyraNetwork"
data-mce=
-style=3D"border: none; text-decoration: none; margin: 0; float: left; disp=
lay: inline-block; width: 42px;"><img
src=3D"cid:5f6dbbaff6dac1a2e25c3c96f0=
2d8020be9c5268@zimbra" alt=3D""
data-mce-src=3D"/home/admin(a)lyra-network.co=
m/corporate-documents/element-signature_logo_Twitter_42x28.jpg" doc=3D"corp=
orate-documents/element-signature_logo_Twitter_42x28.jpg" height=3D"28"
wid=
th=3D"42" border=3D"0"></a><br
data-mce-bogus=3D"1"></td><td style=3D"margi=
n: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-=
height: 16px; width: 258.75pt; text-align: left;" data-mce-style=3D"margin:=
0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-he=
ight: 16px; width: 258.75pt; text-align: left;" width=3D"345"
valign=3D"top=
"><a href=3D"https://payzen.eu" style=3D"border: none;
text-decoration: non=
e; margin: 0; float: left; display: inline-block; width: 61px;" data-mce-hr=
ef=3D"https://payzen.eu" data-mce-style=3D"border: none; text-decoration:
n=
one; margin: 0; float: left; display: inline-block; width: 61px;"><img src=
=3D"cid:7003ef2927fa834b99af5bd467d27eb6d701e6b1@zimbra" alt=3D""
data-mce-=
src=3D"/home/admin(a)lyra-network.com/corporate-documents/element-signature_p=
ayzen_61x28.jpg" doc=3D"corporate-documents/element-signature_payzen_61x28.=
jpg" height=3D"28" width=3D"61"
border=3D"0"></a><br data-mce-bogus=3D"1"><=
/td></tr></tbody></table><table style=3D"margin: 0;
table-layout: fixed; bo=
rder: none; width: 345pt;" data-mce-style=3D"margin: 0; table-layout: fixed=
; border: none; width: 345pt;" width=3D"460" cellspacing=3D"0"
cellpadding=
=3D"0"><tbody><tr><td><hr style=3D"background:
#293b79 none repeat scroll 0=
% 0%; width: 100%; height: 4px; color: #293b79; margin: 0px;" data-mce-styl=
e=3D"background: #293b79 none repeat scroll 0% 0%; width: 100%; height: 4px=
; color: #293b79; margin:
0px;"></td></tr></tbody></table><table
style=3D"m=
argin: 0; table-layout: fixed; border: none; width: 345pt;" data-mce-style=
=3D"margin: 0; table-layout: fixed; border: none; width: 345pt;"
width=3D"4=
60" cellspacing=3D"0"
cellpadding=3D"0"><tbody><tr><td style=3D"margin: 0; =
font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height=
: 16px; width: 30.75pt; text-align: left;" data-mce-style=3D"margin: 0; fon=
t-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 1=
6px; width: 30.75pt; text-align: left;"><div>Lyra Network, 109 rue de
l'inn=
ovation, 31670 Lab=C3=A8ge,
FRANCE</div></td></tr></tbody></table><div><br>=
</div></div><div><br></div><hr id=3D"zwchr"
data-marker=3D"__DIVIDER__"><di=
v data-marker=3D"__HEADERS__"><b>From: </b>"Gianluca
Cecchi" <gianluca.c=
ecchi(a)gmail.com&gt;<br><b>To: </b>"Nelson Lameiras"
<nelson.lameiras@lyr=
a-network.com><br><b>Cc: </b>"ovirt users"
&lt;users(a)ovirt.org&gt;<br><b=
Sent: </b>Thursday, June 8, 2017 9:36:33
AM<br><b>Subject: </b>Re: [ovirt-=
users] ISCSI storage with
multiple nics on same subnet disabled on host act=
ivation<br></div><div><br></div><div
data-marker=3D"__QUOTED_TEXT__"><div d=
ir=3D"ltr"><div class=3D"gmail_extra"><div
class=3D"gmail_quote"><blockquot=
e>On Wed, Jun 7, 2017 at 2:59 PM, Nelson Lameiras <span
dir=3D"ltr"><<a =
href=3D"mailto:nelson.lameiras@lyra-network.com"
target=3D"_blank">nelson.l=
ameiras(a)lyra-network.com</a>&gt;</span>
wrote:</blockquote><blockquote clas=
s=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid
r=
gb(204,204,204);padding-left:1ex"><div><div
style=3D"font-family:arial,helv=
etica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><div>Hello,<br></div><br>=
<div>In our oVirt hosts, we are using DELL equallogic SAN with each server =
connecting to SAN via 2 physical interfaces. Since both interfaces share th=
e same network (Equalogic limitation) we must patch the linux kernel to to =
allow iSCSI multipath with multiple NICs in the same subnet with sysctl :<b=
r></div><br><div>----------------------------------------------------------=
----------------------------------<br></div><div><pre>net.ipv4.conf.p2p1.ar=
p_ignore=3D1
net.ipv4.conf.p2p1.arp_announce=3D2
net.ipv4.conf.p2p1.rp_filter=3D2
net.ipv4.conf.p2p2.arp_ignore=3D1
net.ipv4.conf.p2p2.arp_announce=3D2
net.ipv4.conf.p2p2.rp_filter=3D2<br><br></pre></div></div></div></blockquot=
e><blockquote><div>Hello Nelson,</div><div>nice to see another
environment =
based on EQL iSCSI SAN.</div><div>The systctl.conf customized lines above a=
re not what you mean with "we must patch the linux kernel", is
it?</div><di=
v>What do you mean exactly?</div></blockquote></div><div
class=3D"gmail_quo=
te"><br data-mce-bogus=3D"1"></div><div
class=3D"gmail_quote">Sorry, by "pa=
tch" I meant "sysctl.conf configuration" indeed.
<br></div><div class=3D"gm=
ail_quote">This configuration is what we use to be able to configure 2 netw=
ork interfaces in the same
subnet.<br><div> <br></div><div><div style=
=3D"font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"=
<div><blockquote><p>------------------------------------------------------=
--------------------------------------<br><br></p></blockquote></div></div>=
</div><blockquote><div><div
style=3D"font-family:arial,helvetica,sans-serif=
;font-size:12pt;color:rgb(0,0,0)"><div><p>This works great in most
setups, =
but for a strange reason, on some of our setups, the sysctl configuration i=
s updated by VDSM when activating a host and the second interface stops wor=
king immeadiatly :</p><div>------------------------------------------------=
--------------------------------------------</div><div>vdsm.log<br></div><d=
iv><pre
id=3D"gmail-m_3172160333806361248preview"><code>2017-06-07 11:51:51=
,063+0200 INFO (jsonrpc/5) [storage.ISCSI] Setting strict mode rp_filter f=
or device 'p2p2'. (iscsi:602)
2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not conn=
ect to storageServer (hsm:2392)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer
conObj.connect()
File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect
iscsi.addIscsiNode(self._iface, self._target, self._cred)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, =
in addIscsiNode
iscsiadm.node_login(<a href=3D"http://iface.name"
target=3D"_blank">ifa=
ce.name</a>, target.address, target.iqn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 33=
7, in node_login
raise IscsiNodeError(rc, out,
err)</code></pre><pre><br></pre></div></d=
iv></div></div><div>In my environment I don't see these kinds of
problems.<=
/div><div>I'm using plain CentOS 7.3 hosts as hypervisors; is this your cas=
e too or are you using ovirt-ng nodes?</div><div><br
data-mce-bogus=3D"1"><=
/div></blockquote><div>Plain CentOS 7.3 hosts also. Built from a Minimal
ce=
ntos and adding oVirt repositories.<br></div><div><br
data-mce-bogus=3D"1">=
</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px
0px 0.8ex;b=
order-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div
style=3D"=
font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><di=
v><div><p
style=3D"margin:0px">--------------------------------------------=
------------------------------------------------</p></div></div><div><br><d=
iv>"strict mode" is enforced for second interface, and it no longuer works.=
..</div><div>Which means - at least - that there is no redundancy in case o=
f hardware faillure and this is not acceptable for our production needs.<br=
</div><br><div>What is really strange is that we
have another "twin" site =
on another geographic region with simillar
hardware configuration and same =
oVirt installation, and this problem does not happen.</div><br><div>What
ca=
n be the root cause of this behaviour? How can I correct
it?<br></div><br><=
br></div></div></div></blockquote><blockquote><div>Actually
it is very stra=
nge to me to understand how you don't have problems on the other side.</div=
<div>My workaround to get multipath with EQL in oVirt was the
one explaine=
d here with screenshots:</div><div><a
href=3D"http://lists.ovirt.org/piperm=
ail/users/2017-March/080992.html"
target=3D"_blank">http://lists.ovirt.org/=
pipermail/users/2017-March/080992.html</a><br
data-mce-bogus=3D"1"></div><d=
iv>It has been working now for about 3 months without problems. We have sev=
eral Oracle RDBMS server VMs on it, from 2virtual threads and 8Gb of ram to=
16 threads (2-4-2) and 64Gb of ram </div><div>How did you configure
=
multipath instead? As far as I know oVirt will not let you do it.</div><div=
The only way would be to use bonding but that is not
certified/supported b=
y Dell.</div><div>HIH sharing
experiences,</div><div>Gianluca</div><div><br=
data-mce-bogus=3D"1"></div><div><br
data-mce-bogus=3D"1"></div></blockquot=
e><div>I've read the link you gave and it seems to me that you are using th=
e same physical interface for SAN and vm network.<br></div><div>In my
setup=
: </div><div> - SAN network is using two disctinct physical
interfac=
es (p2pX) on same subnet (therefore the need for sysctl
patch)<br></div><di=
v> - VM network (p1pX) is using 2 other distinct physical interfaces =
on "trunk" mode<br
data-mce-bogus=3D"1"></div><div> - Administration =
network (emX) is using 2 distinct bonded network interfaces <br data-mce-bo=
gus=3D"1"></div><div><br
data-mce-bogus=3D"1"></div><div>Our setup, using a=
total of 6 network interfaces is "simpler" than yours since each interface=
has a "disctinct" role.<br
data-mce-bogus=3D"1"></div><div><br data-mce-bo=
gus=3D"1"></div><div>To make multipath work with oVirt on this
setup, we ha=
d to configure SAN network on system before installing oVirt (using sysctl.=
conf above), </div><div> - next declare 2 SAN networks on oVirt that
=
are "dragued" to each SAN interface in host network configuration,
</div><d=
iv> - and finally, in datacenter listing, chosing a datacenter and se=
lecting "iSCSI Multipathing" tab below I can add a multipath configuration =
concerning both SAN oVirt networks and relevant iqn storage targets.<br dat=
a-mce-bogus=3D"1"></div><div><br
data-mce-bogus=3D"1"></div><div>Hope it's =
clear, if not, please ask me.<br
data-mce-bogus=3D"1"></div><div><br data-m=
ce-bogus=3D"1"></div><div>It's very strange to me also since
this configura=
tion did work great on our labs. But half of production sites have the prob=
lem decribed above : second SAN interface stops working uppon host activati=
on because vdsm forces sysctl net.ipv4.conf.p2p2.rp_filter to 1 (strict mod=
e).<br data-mce-bogus=3D"1"></div><div><br
data-mce-bogus=3D"1"></div><div>=
Still searching for clues as why this behaviour...<br
data-mce-bogus=3D"1">=
</div></div></div></div><br></div></div></body></html>
------=_Part_57900107_1954785857.1496910345350
Content-Type: image/jpeg; name=element-signature_logo_lyra_115x94.jpg
Content-Disposition: attachment;
filename=element-signature_logo_lyra_115x94.jpg
Content-Transfer-Encoding: base64
Content-ID: <25cd0e595e0f025605ddb995496484dcb70629d9@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOkU1QUU4Nzk1OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOkU1QUU4Nzk0OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIAF4AcwMBEQACEQEDEQH/xACdAAEBAAIDAQEAAAAAAAAAAAAACAUH
AgMGBAEBAQADAQEAAAAAAAAAAAAAAAACBAUBAxAAAAUDAgAHDQcCBwAAAAAAAAECAwQRBQYSByFR
E7RWFzcxQSKS0nOT03RVlQgYYXEyQiMUFYEzsVJicmODOBEAAgECAwcEAgIDAAAAAAAAAAECEQNR
EwQhgbESMlIzMUGhFHEVwQUiQzT/2gAMAwEAAhEDEQA/AKpAE9b/AGQZdF3FtNpsd4l29M2DHSll
iS6w0bzsp5slqJCiLvERnTuENTRQg7bckntK15vmojj1WfMP0rc+Jy/JD7Fjt+EMueJ6bbnBN4bR
lkWdkt+XPtDaHSfjKnSHyUpTZkg+TcLSdFGRjyv3rUo0iqP8EoQkntZ7fcOy5VdrXGYxuccGW2+S
3nCecYq3oUWnU2RmfhGXAPPSXLcJNzVVT8kNVbnKKUHRmppW1/zBOSXVtZS4lpSjNtP8nLKhfcSR
alqdPXp+ERhauKKTe38nX1WfMP0qc+Jy/JHPsWO34RPLniOqz5h+lTnxOX5IfYsdvwhlzxHVZ8w/
Spz4nL8kPsWO34Qy54jqs+YfpU58Tl+SH2LHb8IZc8R1WfMP0qc+Jy/JD7Fjt+EMueJ2bCX3MH9y
LxZr7eJdxKBCkoW0/JdfaJ5mUy2a0EszL/MRHTuGGshDLTiqVf8AAst81GUGMssgAAAAE5b89s+L
+zwefPDV0fhlv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACc
t+e2fF/Z4PPnhq6Pwy38Crd60UaMotAAAAAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLI
AAAABOW/PbPi/s8Hnzw1dH4Zb+BVu9aKNGUWjAKzrGk5cnEjkq/nVo5RMbk3NOnkzd/uU0fgKvdH
rky5eb2I86rQz48iQAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLIAAAABOW/PbPi/s8Hn
zw1dH4Zb+BVu9aKNGUWjSD//AKkj+xnzJQ0V/wA2/wDkr/7D3e7GX5DiOMfzlniMTEMPIRORIJfg
tOeCladCk/nNJH94raa1GcuVnpck4qqOrJdz4dr2yazKOhDjkxhlUGMszop98i/TOlD/AE/CNX+0
x23p27nIJXKRqeeyjdnLMd27sGSzLdEO4Xd4uWifqkhDDiFut0qrUSzQlNa9wz7g9bemjK44puiI
yuNRTOzGs53hud+gvzsYjwMVnqNfLuqUTrEck69bitZmStPc1NJIz4hy5ZtKLpKskIzk36bDHRt2
9yMsuMw9v8ejybNBc5M5s5Rp5Uy4aEZuMJSai4dPCZFStBN6a3BLne1nMyT6Uffgu8GQ5DnysXuN
mbtZx461TG1ms3kyGiLWRHXToMz8Hg7nfMQvaWMYcydTsLjcqG2BSPYnbY/twy3zVw5+yNXV+GO7
gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLROmdZbAxL5g/5yey6/GjREJW2wSTc
M3YpoKms0F3VcY1bNpzscqKs5UnU9pad4cG3Fefw1UWXFO7x3mULkpaJJnoM6JNC10WREak/aXGK
8tLO1/ns2HorilsNTYzar/kOQWTa+5tmmBjk+W9cKGqimkrJS68RcCkoV/yC7clGMXcXrJI8Yptq
OBsj5oEpTiNmSRESSuKSJJFwUJlzgoKv9f1P8Hrf9Db1wh/vbVJh6tH7lhbOvi5RBpr/AEqKMXR1
PZrYaC2x3Jgbawp+IZjBkwpkWS480603rSslkRU7qTOuiqFlUlEfepw6WosO61ODK9ufLsZ27f36
RkG/8q9OwXbe3PgKciMPp0uHHS22hpxRf60o1f4VC9DlsUrWjEHWdSghllknbY/twy3zVw5+yNXV
+GO7gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLRxU00o6qQlR8ZkRhUGMyO7W7H
7FOvUpCSZgMrfMqERqNJeCgj41qokvtMTtxcpJL3OSdFU1n8vFimPwbrnF18O55DIXybhlw8ihZm
s08RLdqVOJJC5rZqqgvSJ42V7v3MpM3dTBnyWbtBQmHFnXFpLzdXOWjW9tWo2irwPpd0pWhVPBPU
R0qILTVWx+y+STuYmQj7sxHH1Kctb7NuahnMflKW2akk3J/bvGlojqtpv8ZuJP8ACR8Ag9M8dtTu
YdErdfEJTCZKrY/L5JSySbrbCdKmicdWRLeWlJK/bs8skq+ElSacJ8HVppr3OZiP2RuxFK4W5xi3
LK2T5D8NmY8baHHnWySSeTSa6to1K8JTumn3gtM6PbtQzDYYqnqTtsf24Zb5q4c/ZGrq/DHdwK1r
rZRIyiyAAAAATlvz2z4v7PB588NXR+GW/gVbvWijRlFoADGZHjVmyO2Ltd4YOTAcUlbjJLcbJRoO
qam2pCuA+GlROFxwdV6nJRT2M+q226DbLfHt8BkmIcRtLMdlNaJQgqEVTqZ/eYjKTbqwlQ5Hb4Bp
Uk4zRpUpa1FoTQ1uEaVqPg7qiUZGffDmYodSbNZ0toaTBjk02hbTaCaQSUtu/wBxCSpQkr/MXfHe
d4iiP12z2h5k2XYMdxlSkuKbW0hSTW2kkIUZGVKpSkiI+8RApPEURydtltdQpDsRlxCtepKm0mR8
qdXKkZfn/NxjnMxQ+kcOk7bH9uGW+auHP2Rq6vwx3cCta62USMosgAAAAE5b89s+L+zwefPDV0fh
lv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACb9/3mWN4cbff
WTTLUWEt1xX4UpTNeNSj+wiGtolW1LfwKt7qRuTrY226RwfSkKH1rnaz3zI4jrZ226RwfSkH1rna
xmRxHWztt0jg+lIPrXO1jMjiOtnbbpHB9KQfWudrGZHEdbO23SOD6Ug+tc7WMyOJ+o3W24WtKE5F
CUpR0SknCqZmH1rnazjuxW2pnbTfbPeGnHbXMamNtK0OLaVqJKjKtDELlqUNklQ7C5GfS6mOu+f4
XZpy4F0vMWHMbJKlsOrJKyJRVTUvtIdjZnJVSOuaXqfF1sbbdI4PpSEvrXO1nMyOJp7YiQxJ3oyi
RHcS9Hejz3GXUcKVoVOZNKi+wyOovaxUsx3cDwtdbKNGUWgAAAADX+4GzGPZveWbtcpsuO+xGTFS
iObRINCFrcIz1oWdaun3xas6qVtUSR5ztKTqeZ+lrCvely8Zj1Q9f2M8EQyEPpawr3pcvGY9UH7G
eCGQh9LWFe9Ll4zHqg/YzwQyEPpawr3pcvGY9UH7GeCGQh9LWFe9Ll4zHqg/YzwQyEc2flfwxl5t
1NzuJqbUSiI1MUqX/UOr+xmn6I5LTRaaxNg4XhNuxOHIiwX3n0SXCdWb5pMyMk6eDSlPEPDU6mV5
pteg0+nVpNI8zm+xeNZfkDt7nzprEl5Dbam2FNEgibTpKmttR97jErWslCNEkTlaUnUwP0tYV70u
XjMeqHp+xngiOQj023+zGPYReXrtbZsuQ+/GVFUiQbRoJC1ocMy0IQdatF3x5XtVK4qNInC0oups
AVT0AAAAAAAAAAAAAAAAAAAAAAAAAAAAD//Z
------=_Part_57900107_1954785857.1496910345350
Content-Type: image/jpeg; name=element-signature_logo_YouTube_32x28.jpg
Content-Disposition: attachment;
filename=element-signature_logo_YouTube_32x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <4c5918308790526a8092b3bb40ae8dab65d1777b@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOjQ1REY5Njk1OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOjQ1REY5Njk0OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAIAMBEQACEQEDEQH/xAB5AAADAQEAAAAAAAAAAAAAAAAABwgF
BgEAAgMBAAAAAAAAAAAAAAAAAAQCAwUBEAABBAECBAUFAAAAAAAAAAABAgMEBQARBiESEwcxUdIU
CEEik1UYEQACAQMCBQUAAAAAAAAAAAAAAQIRAwQhMVFhEiITQaHBYhX/2gAMAwEAAhEDEQA/AKpw
AnD5Hbg3JA35XxKqxmxUPVrJEaI862FuqkPp4IbI5lHQDNXBhFwbaW4tebqcMIvfYjURtz6eesv1
YzWz9fYh38zR7eX2+mu5lHV3VhaNOe8aTJgTHnweVXEBba1cQRkb8IeNtJbHYN9WpW+YY2T53inQ
4HfzZk6a4lmHGaiOSHl8EoT7mSOZR+gBIzUxU3Yklz+Be4+9DRl30N3dMOxjbyro9EzHW3MqupHc
L7qlaoX1SsFvlHl44moPpa6XXiW113FPuu3qrT5I7dfrJLUxllMRl15hQWjqBbiuXmTqCQlQx23F
rHdSqTrNFFZlDAou7vZa43xuSNaw7CPFZZhIiKbeSsqKkOuuE/aCNNHcexspW40a9Sm5bcmcH/J1
5+0rvxOenGf0o8GQ8DN7ZXxzv9vbpqrhy0huMQJCHlstocBKU+ITqANcru50ZRao9SUbLTqPvMwv
DAAwAMAP/9k=
------=_Part_57900107_1954785857.1496910345350
Content-Type: image/jpeg; name=element-signature_logo_LinkedIn_41x28.jpg
Content-Disposition: attachment;
filename=element-signature_logo_LinkedIn_41x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <0c1a4f5470e19b606b0a0890da695ad9d2808a60@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOjY2MEYzRTFGOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOjY2MEYzRTFFOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAKQMBEQACEQEDEQH/xAB2AAEBAQEAAAAAAAAAAAAAAAAABggH
AQEAAwEAAAAAAAAAAAAAAAAAAwQFAhAAAQMEAgICAgMBAAAAAAAAAQIDBAARBQYhEjEHMghBE1Fh
IhcRAAICAQIDCQAAAAAAAAAAAAABAgMREgQxkTIhQVGhscETMxX/2gAMAwEAAhEDEQA/ANU0Bl33
zsuzQfZcmHj8tMiMKaihDLMl1psKcQkXslSUjk8mtjZ1xdeWkVbW9RDZ/ZN8wOXkYnI7LKMuL1Lq
o+RddaIWkLHVwLAPCuf4NWYVwksqPkcNtd5V+ldr2ef7MwkaZmJsqK6p/uy9JdcbUBGcULpUog8i
9Q7uuKreEjquT1GsKxC2KAUBk37B2/6y72t1tB7X8dbIvf8Aq3mtvZfVzKlvUWWw7Dr+Dfy8rCt4
d153Z8ew0FNx3kohuxmUyFtAfFPyBUOBz+arwhKWE89L9SRtLmTuns4xj7LLaxYaGNE6SYojlJZ6
rhLUf19f89eyj4qa1t7ft449ziPWairGLQoBQHFvZ/onObht0jNxclFjMPNtNpZdS4Vj9aAk36gj
mtDb7yMIaWiGdTbySSfqnsKbdctAFh1Fm3Bwfx8fFT/ox8GcfAyk9dfX7OapuGNzj+SiPRoJcKmG
kuBRDjS2x1uAPK71DfvYzg444nUKmnk7lWcTigFAKAUAoBQCgP/Z
------=_Part_57900107_1954785857.1496910345350
Content-Type: image/jpeg; name=element-signature_logo_Twitter_42x28.jpg
Content-Disposition: attachment;
filename=element-signature_logo_Twitter_42x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <5f6dbbaff6dac1a2e25c3c96f02d8020be9c5268@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOjdDMkI3MjMyOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOjdDMkI3MjMxOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAKgMBEQACEQEDEQH/xAB3AAEBAQEBAAAAAAAAAAAAAAAABwUI
BgEBAAMBAQAAAAAAAAAAAAAAAAMEBQIBEAABAwIEBwEBAQAAAAAAAAABAgMEAAUREhMGITFBFBUH
CIEiFhEAAgIBAgQHAAAAAAAAAAAAAAERAgMxBCHBEgVBUWGBQhMV/9oADAMBAAIRAxEAPwDC9g7u
3dH33uJiNeri2wxcJQQ01JfCUNodVwCUqwSlI/AK3sGKropS0Kd7OWZMXcPsyZDdnQ7he5UFhQQ/
LZelOMoURiEqWlRAOFdumNOGqyeTYtPzFfL1df8ASm5XCTP0ey0e5eceyZu4zZc5VhjlGOFZ/cKJ
dMKNeRNhbclzrOJxQCgOSW7o5a/oKXMRFXNT5iYw/EaAUtbL61tOBKTwXglWOXrW30zgj0RUmLlb
9p3rZ/r/ANd3GxWJtm3XC8B1EO2RsEqDkng88pAP8JSnEk8ugqlt6Xy5Fa3FImyNVUI8z8nJCU7o
SOSRAA/O5qbuXx9+Rxg8ToGsssCgFAc/7q+b9xXrc11u7V3iNN3CW/JbQUuhaUuuFYBIHMA9K08e
+rWqUPgivbC25MofK25Qoq81DKjwKlJeUT+kE13+jXyZ59DKX6a9WXPYfmO+msTPJdtp6AWMuhq4
5swHPVFVN1uFkiFoS46dJSqqEgoBQCgFAKAUAoBQH//Z
------=_Part_57900107_1954785857.1496910345350
Content-Type: image/jpeg; name=element-signature_payzen_61x28.jpg
Content-Disposition: attachment; filename=element-signature_payzen_61x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <7003ef2927fa834b99af5bd467d27eb6d701e6b1@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOkU1NjczQjM2OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOkU1NjczQjM1OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAPQMBEQACEQEDEQH/xABvAAABBAMAAAAAAAAAAAAAAAAAAQUG
BwIDBAEBAAAAAAAAAAAAAAAAAAAAABAAAQQABAQEAgsBAAAAAAAAAQIDBAUAERIGITETFEEiFQfS
I1FhcYGhMlIkRMQWhhEBAAAAAAAAAAAAAAAAAAAAAP/aAAwDAQACEQMRAD8AvqZ7q7OhzZMSS9IQ
Yjy2JT/avqZbW2rSvU6lBSACOeeAljLzTzSHmVhxpxIW24k5pUlQzBBHgRgM8BBJ9fKuvcGwr3La
wgxIlfGeaagyVsJ1uOOBRUBmOSRgI0bnZgfQwdx7qDrroYZSVzBrcJ0hKM2xq+7AWBU7PTXTEShc
2svRx6MuWp5o5gjigj68BombFRKmPyTfXTPXWpzoszloaRqOelCQPKkeAwEP6lp6f6V6vYaP9Z6b
3fcr7ntuhno6vPnxwGyohb2ntbpr6Y17NdKt7Bp5+YHlPJLitKyltI0KGk+XPANVgzV1Eyyqbe3s
IMmnhsMbVaYcebDraY+XVbQ15XXFOghQVnkMBkbjcVPTUs6M5IkPbnpWa1hKlrWG7QZBp0hROnUh
xRUfHTgJDWuwtsbptVTpC1xaqhgJfkuFTji9DjoKiTmpSlq/HAcO2d11d/cuXUySJG4C0+NvbfCH
CmK2hClHUop0GQ6E+dWfAeUYBi2lY2E+zgW7twhVlHDk+7AkzFO9u2lRdjORFIEZrTmAkA+GYwDl
TwLJhjZFouxsF3NzN6r7Lj7imExHUOSHWi0TpyCdORIzzwHT/J/7n+vgLVAA5YBChBUlRSCpOelR
HEZ88sA2WG3YVhb1lnJW4VVJcXFjAgM9VxOjqKTlmVJTnp48M8Ax2+3N3J3TJu6KVAbRKisxnWpr
brh+SpasxoKf14BBD91hxE2kB+nt5Px4AEL3VGoiZSAr4qyjyOP2+fAL2nuvw/e0vDl8iT8eAbv8
Lur0vPvYXrPrfrfU6bvb59Lp6NOev83HngLBwBgDAGAMAYAwBgP/2Q==
------=_Part_57900107_1954785857.1496910345350--
------=_Part_57900106_686621279.1496910345349--
11:10 a.m.
On Thu, Jun 8, 2017 at 10:25 AM, Nelson Lameiras <
nelson.lameiras(a)lyra-network.com> wrote:
> Hello,
>
> In our oVirt hosts, we are using DELL equallogic SAN with each server
> connecting to SAN via 2 physical interfaces. Since both interfaces share
> the same network (Equalogic limitation) we must patch the linux kernel to
> to allow iSCSI multipath with multiple NICs in the same subnet with sysctl :
>
> ------------------------------------------------------------
> --------------------------------
>
> net.ipv4.conf.p2p1.arp_ignore=1
> net.ipv4.conf.p2p1.arp_announce=2
> net.ipv4.conf.p2p1.rp_filter=2
>
> net.ipv4.conf.p2p2.arp_ignore=1
> net.ipv4.conf.p2p2.arp_announce=2
> net.ipv4.conf.p2p2.rp_filter=2
>
> Hello Nelson,
nice to see another environment based on EQL iSCSI SAN.
The systctl.conf customized lines above are not what you mean with "we
must patch the linux kernel", is it?
What do you mean exactly?
Sorry, by "patch" I meant "sysctl.conf configuration" indeed.
This configuration is what we use to be able to configure 2 network
interfaces in the same subnet.
OK.
I'm using that configuration too.
And also with physical hosts, used as physical RDBMS servers, that connect
to Dell EQL iSCSI SAN, not only for oVirt.
It is a requirement so that you are sure that the packets come back to the
same interface, because of more that 1 network interface on the same lan.
That said, I noticed that oVirt creates the file "vdsm.conf" under
/etc/sysctl.d, with these contents
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
So this settings doesn't need to be overwritten by you. Do you have that
file too?
In my case I left the oVirt generated file and created two more files in
/etc/sysctl.d, named 50-iscsi.conf and equallogic.conf
Their contents:
- 50-iscsi.conf
net.ipv4.conf.p1p1/100.arp_announce=2
net.ipv4.conf.p1p1/100.arp_ignore=1
#
net.ipv4.conf.p1p1/100.rp_filter=2
net.ipv4.conf.p1p2.rp_filter=2
The reason of the first two lines is that vlan interfaces apparently didn't
inherited the arp_announce and arp_ignore settings so I put them for the
vlan set interface, no need for the other one (p1p2 in my case)
The reason of the second two lines is to add the third sysctl setting
needed by EQL
So I think you can have a 50-iscsi.conf file only composed by:
net.ipv4.conf.p2p1.rp_filter=2
net.ipv4.conf.p2p2.rp_filter=2
- equallogic.conf
net.core.rmem_default = 262144
net.core.rmem_max = 16777216
net.core.wmem_default = 262144
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
# Defaults were:
#net.core.rmem_default = 212992
#net.core.rmem_max = 212992
#net.core.wmem_default = 212992
#net.core.wmem_max = 212992
#net.ipv4.tcp_rmem = 4096 87380 6291456
#net.ipv4.tcp_wmem = 4096 16384 4194304
I've read the link you gave and it seems to me that you are using
the same
physical interface for SAN and vm network.
In my setup :
- SAN network is using two disctinct physical interfaces (p2pX) on same
subnet (therefore the need for sysctl patch)
- VM network (p1pX) is using 2 other distinct physical interfaces on
"trunk" mode
- Administration network (emX) is using 2 distinct bonded network
interfaces
Our setup, using a total of 6 network interfaces is "simpler" than yours
since each interface has a "disctinct" role.
To make multipath work with oVirt on this setup, we had to configure SAN
network on system before installing oVirt (using sysctl.conf above),
- next declare 2 SAN networks on oVirt that are "dragued" to each SAN
interface in host network configuration,
- and finally, in datacenter listing, chosing a datacenter and selecting
"iSCSI Multipathing" tab below I can add a multipath configuration
concerning both SAN oVirt networks and relevant iqn storage targets.
Hope it's clear, if not, please ask me.
It's very strange to me also since this configuration did work great on
our labs. But half of production sites have the problem decribed above :
second SAN interface stops working uppon host activation because vdsm
forces sysctl net.ipv4.conf.p2p2.rp_filter to 1 (strict mode).
Still searching for clues as why this behaviour...
I had the further need to use both interfaces for VM network, but, also if
not, I couldn't setup multipathing.
I have a requirement to have iSCSI on vlan and when setting the two
interfaces on the same vlan (so also in oVirt), oVirt complained about that.
So this lets me think that if instead you have both network interfaces in
access mode, without vlan, oVirt doesn't complain when you try toset the
multipathing????
If this is the case, I think I could have tested (probably I didn't, I
don't remember) this scenario
Both interfaces on vlan 100 but the network switch port configured with
default vlan as 100, so that form OS point of view, and oVirt point of
view, no vlan in place and perhaps it doesn't complain....
Gianluca
4:56 p.m.
New subject: Fwd: ISCSI storage with multiple nics on same subnet disabled on host activation
------=_Part_88180575_227580591.1500303388932
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Hello, Can any one please help us with the problem described below?=20
Nir, I'm including you since a quick search on the internet led me to think=
that you have worked on this part of the project. Please forgive me if I'm=
off topic.=20
(I incorrectly used below the expression "patch" when I meant
"configure". =
it's corrected now)=20
cordialement, regards,=20
=09Nelson LAMEIRAS=20
Ing=C3=A9nieur Syst=C3=A8mes et R=C3=A9seaux / Systems and Networks enginee=
r=20
Tel: +33 5 32 09 09 70=20
nelson.lameiras(a)lyra-network.com=20
www.lyra-network.com | www.payzen.eu=20
=09
=09
=09
Lyra Network, 109 rue de l'innovation, 31670 Lab=C3=A8ge, FRANCE=20
De: "Nelson Lameiras" <nelson.lameiras(a)lyra-network.com>=20
=C3=80: "ovirt users" <users(a)ovirt.org>=20
Envoy=C3=A9: Mercredi 7 Juin 2017 14:59:48=20
Objet: [ovirt-users] ISCSI storage with multiple nics on same subnet disabl=
ed on host activation=20
Hello,=20
In our oVirt hosts, we are using DELL equallogic SAN with each server conne=
cting to SAN via 2 physical interfaces. Since both interfaces share the sam=
e network (Equalogic limitation) we must configure sysctl to to allow iSCSI=
multipath with multiple NICs in the same subnet :=20
---------------------------------------------------------------------------=
-----------------=20
net.ipv4.conf.p2p1.arp_ignore=3D1
net.ipv4.conf.p2p1.arp_announce=3D2
net.ipv4.conf.p2p1.rp_filter=3D2
net.ipv4.conf.p2p2.arp_ignore=3D1
net.ipv4.conf.p2p2.arp_announce=3D2
net.ipv4.conf.p2p2.rp_filter=3D2=20
---------------------------------------------------------------------------=
-----------------=20
This works great in most setups, but for a strange reason, on some of our s=
etups, the sysctl configuration is updated by VDSM when activating a host a=
nd the second interface stops working immeadiatly :=20
---------------------------------------------------------------------------=
-----------------=20
vdsm.log=20
2017-06-07 11:51:51,063+0200 INFO (jsonrpc/5) [storage.ISCSI] Setting stri=
ct mode rp_filter for device 'p2p2'. (iscsi:602)
2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not conn=
ect to storageServer (hsm:2392)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer
conObj.connect()
File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect
iscsi.addIscsiNode(self._iface, self._target, self._cred)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, =
in addIscsiNode
iscsiadm.node_login(iface.name, target.address, target.iqn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 33=
7, in node_login
raise IscsiNodeError(rc, out, err)=20
---------------------------------------------------------------------------=
-----------------=20
"strict mode" is enforced for second interface, and it no longuer works...=
=20
Which means - at least - that there is no redundancy in case of hardware fa=
illure and this is not acceptable for our production needs.=20
What is really strange is that we have another "twin" site on another geogr=
aphic region with simillar hardware configuration and same oVirt installati=
on, and this problem does not happen.=20
Can this be really random?=20
What can be the root cause of this behaviour? How can I correct it?=20
our setup:=20
oVirt hostedEngine : Centor 7.3, ovirt 4.1.2=20
3 physical oVirt nodes centos 7.3, ovirt 4.1.2=20
SAN DELL Equalogic=20
cordialement, regards,=20
=09Nelson LAMEIRAS=20
Ing=C3=A9nieur Syst=C3=A8mes et R=C3=A9seaux / Systems and Networks enginee=
r=20
Tel: +33 5 32 09 09 70=20
nelson.lameiras(a)lyra-network.com=20
www.lyra-network.com | www.payzen.eu=20
=09
=09
=09
Lyra Network, 109 rue de l'innovation, 31670 Lab=C3=A8ge, FRANCE=20
_______________________________________________=20
Users mailing list=20
Users(a)ovirt.org=20
http://lists.ovirt.org/mailman/listinfo/users=20
------=_Part_88180575_227580591.1500303388932
Content-Type: multipart/related;
boundary="----=_Part_88180576_1156318871.1500303388932"
------=_Part_88180576_1156318871.1500303388932
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable
<html><body><div style=3D"font-family: arial, helvetica, sans-serif;
font-s=
ize: 12pt; color: #000000"><div>Hello, Can any one please help us with the
=
problem described below?<br></div><div><br
data-mce-bogus=3D"1"></div><div>=
Nir, I'm including you since a quick search on the internet led me to think=
that you have worked on this part of the project. Please forgive me if I'm=
off topic.<br></div><div><br
data-mce-bogus=3D"1"></div><div>(I incorrectl=
y used below the expression "patch" when I meant "configure". it's
correcte=
d now)<br
data-mce-bogus=3D"1"></div><div><br></div><div
data-marker=3D"__S=
IG_PRE__"><div>cordialement, regards,</div><div><br
data-mce-bogus=3D"1"></=
div><table style=3D"margin: 0px; table-layout: fixed; border: none; width: =
345pt; padding: 0px;" data-mce-style=3D"margin: 0px; table-layout: fixed; b=
order: none; width: 345pt; padding: 0px;" width=3D"460"
cellspacing=3D"0" c=
ellpadding=3D"0" border=3D"0"><tbody><tr><td
style=3D"margin: 0; font-size:=
11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; wi=
dth: 86.25pt; text-align: left;" data-mce-style=3D"margin: 0; font-size: 11=
px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width=
: 86.25pt; text-align: left;" width=3D"115"
valign=3D"top"><a href=3D"https=
://www.lyra-network.com/" style=3D"border: none; text-decoration: none; wid=
th: 100%; display: block; clear: both;" data-mce-href=3D"https://www.lyra-n=
etwork.com/" data-mce-style=3D"border: none; text-decoration: none; width: =
100%; display: block; clear: both;"><img
src=3D"cid:ac318af44fd2ff936fd390a=
2dfa5ac7f8bf624d9@zimbra" alt=3D""
data-mce-src=3D"/home/admin@lyra-network=
.com/corporate-documents/element-signature_logo_lyra_115x94.jpg" doc=3D"cor=
porate-documents/element-signature_logo_lyra_115x94.jpg" width=3D"115"
bord=
er=3D"0" height=3D"94"></a><br
data-mce-bogus=3D"1"></td><td style=3D"margi=
n: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-=
height: 16px; width: 258.75pt;" data-mce-style=3D"margin: 0; font-size: 11p=
x; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width:=
258.75pt;" width=3D"345" valign=3D"top"><table
style=3D"margin: 0; width: =
258.75pt; table-layout: fixed; border: none;" data-mce-style=3D"margin: 0; =
width: 258.75pt; table-layout: fixed; border: none;" width=3D"345"
cellspac=
ing=3D"0" cellpadding=3D"0"
border=3D"0"><tbody><tr><td style=3D"font-size:=
16px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 18px; fo=
nt-weight: bold;" data-mce-style=3D"font-size: 16px; font-family: Arial,san=
s-serif; color: #2b3c7a; line-height: 18px; font-weight: bold;">Nelson LAME=
IRAS</td></tr><tr><td style=3D"font-family: Arial,sans-serif;
color: #2b3c7=
a; font-size: 11px; line-height: 16px;" data-mce-style=3D"font-family: Aria=
l,sans-serif; color: #2b3c7a; font-size: 11px; line-height: 16px;">Ing=C3=
=A9nieur Syst=C3=A8mes et R=C3=A9seaux<span style=3D"font-family: Arial,san=
s-serif; color: #337ab7; font-size: 11px;" data-mce-style=3D"font-family: A=
rial,sans-serif; color: #337ab7; font-size: 11px;"> / Systems and Networks =
engineer</span></td></tr><tr><td style=3D"line-height:
16px; font-family: '=
Arial',sans-serif; color: #2b3c7a; font-size: 11px;"
data-mce-style=3D"line=
-height: 16px; font-family: 'Arial',sans-serif; color: #2b3c7a; font-size: =
11px;"><span style=3D"font-family: Arial,sans-serif; color: #2b3c7a;
font-s=
ize: 11px;" data-mce-style=3D"font-family: Arial,sans-serif; color: #2b3c7a=
; font-size: 11px;">Tel: +33 5 32 09 09
70</span></td></tr><tr><td style=3D=
"line-height: 16px; font-family: 'Arial',sans-serif; color: #2b3c7a; font-s=
ize: 11px;" data-mce-style=3D"line-height: 16px; font-family:
'Arial',sans-=
serif; color: #2b3c7a; font-size: 11px;"><a
href=3D"mailto:nelson.lameiras@=
lyra-network.com" style=3D"font-size: 11px; font-family: Arial,sans-serif; =
line-height: 16px; text-decoration: none;" data-mce-href=3D"mailto:nelson.l=
ameiras(a)lyra-network.com" data-mce-style=3D"font-size: 11px; font-family: A=
rial,sans-serif; line-height: 16px; text-decoration: none;">nelson.lameiras=
@lyra-network.com</a><br
data-mce-bogus=3D"1"></td></tr><tr><td
style=3D"li=
ne-height: 16px; font-family: 'Arial',sans-serif; color: #2b3c7a; font-size=
: 11px;" data-mce-style=3D"line-height: 16px; font-family:
'Arial',sans-ser=
if; color: #2b3c7a; font-size: 11px;"><a
href=3D"https://www.lyra-network.c=
om/" style=3D"font-size: 11px; font-family: Arial,sans-serif; line-height: =
16px; border: none; text-decoration: none;" data-mce-href=3D"https://www.ly=
ra-network.com/" data-mce-style=3D"font-size: 11px; font-family: Arial,sans=
-serif; line-height: 16px; border: none; text-decoration: none;">www.lyra-n=
etwork.com</a> <a href=3D"https://payzen.eu" style=3D"font-size:
11px; font=
-family: Arial,sans-serif; line-height: 16px; border: none; text-decoration=
: none;" data-mce-href=3D"https://payzen.eu"
data-mce-style=3D"font-size: 1=
1px; font-family: Arial,sans-serif; line-height: 16px; border: none; text-d=
ecoration: none;"> | www.payzen.eu</a><br
data-mce-bogus=3D"1"></td></tr></=
tbody></table></td></tr></tbody></table><table
style=3D"margin: 0; table-la=
yout: fixed; border: none; width: 345pt;" data-mce-style=3D"margin: 0; tabl=
e-layout: fixed; border: none; width: 345pt;" width=3D"460"
cellspacing=3D"=
0" cellpadding=3D"0"><tbody><tr><td style=3D"margin:
0; font-size: 11px; fo=
nt-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 24pt=
; text-align: left;" data-mce-style=3D"margin: 0; font-size: 11px; font-fam=
ily: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 24pt; text=
-align: left;" width=3D"32" valign=3D"top"><a
href=3D"https://www.youtube.c=
om/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA" style=3D"border: none; text-decoration=
: none; margin: 0; float: left; display: inline-block; width: 32px;" data-m=
ce-href=3D"https://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA&... data-m=
ce-style=3D"border: none; text-decoration: none; margin: 0; float: left; di=
splay: inline-block; width: 32px;"><img
src=3D"cid:6f3cc57ba362f7a2942a16e8=
8d1ea9ce686c7da2@zimbra" alt=3D""
data-mce-src=3D"/home/admin(a)lyra-network.=
com/corporate-documents/element-signature_logo_YouTube_32x28.jpg" doc=3D"co=
rporate-documents/element-signature_logo_YouTube_32x28.jpg" width=3D"32"
bo=
rder=3D"0" height=3D"28"></a><br
data-mce-bogus=3D"1"></td><td style=3D"mar=
gin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; lin=
e-height: 16px; width: 30.75pt; text-align: left;" data-mce-style=3D"margin=
: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-h=
eight: 16px; width: 30.75pt; text-align: left;" width=3D"41"
valign=3D"top"=
<a
href=3D"https://www.linkedin.com/company/lyra-network_2" style=3D"borde=
r: none; text-decoration: none; margin: 0; float: left; display: inline-blo=
ck; width: 41px;" data-mce-href=3D"https://www.linkedin.com/company/lyra-ne=
twork_2" data-mce-style=3D"border: none; text-decoration: none; margin: 0; =
float: left; display: inline-block; width: 41px;"><img
src=3D"cid:95ac2affd=
b66bc26fd0e254578e26d8574139b42@zimbra" data-mce-src=3D"/home/admin@lyra-ne=
twork.com/corporate-documents/element-signature_logo_LinkedIn_41x28.jpg&q... do=
c=3D"corporate-documents/element-signature_logo_LinkedIn_41x28.jpg" width=
=3D"41" border=3D"0" height=3D"28"></a><br
data-mce-bogus=3D"1"></td><td st=
yle=3D"margin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2=
b3c7a; line-height: 16px; width: 31.5pt; text-align: left;" data-mce-style=
=3D"margin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c=
7a; line-height: 16px; width: 31.5pt; text-align: left;" width=3D"42"
valig=
n=3D"top"><a href=3D"https://twitter.com/LyraNetwork"
style=3D"border: none=
; text-decoration: none; margin: 0; float: left; display: inline-block; wid=
th: 42px;" data-mce-href=3D"https://twitter.com/LyraNetwork"
data-mce-style=
=3D"border: none; text-decoration: none; margin: 0; float: left; display: i=
nline-block; width: 42px;"><img
src=3D"cid:5e9e243634081dcf099c93fd9ca78f8b=
346f57b3@zimbra" alt=3D""
data-mce-src=3D"/home/admin(a)lyra-network.com/corp=
orate-documents/element-signature_logo_Twitter_42x28.jpg" doc=3D"corporate-=
documents/element-signature_logo_Twitter_42x28.jpg" width=3D"42"
border=3D"=
0" height=3D"28"></a><br
data-mce-bogus=3D"1"></td><td style=3D"margin: 0; =
font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height=
: 16px; width: 258.75pt; text-align: left;" data-mce-style=3D"margin: 0; fo=
nt-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: =
16px; width: 258.75pt; text-align: left;" width=3D"345"
valign=3D"top"><a h=
ref=3D"https://payzen.eu" style=3D"border: none; text-decoration: none;
mar=
gin: 0; float: left; display: inline-block; width: 61px;" data-mce-href=3D"=
https://payzen.eu" data-mce-style=3D"border: none; text-decoration: none; m=
argin: 0; float: left; display: inline-block; width: 61px;"><img
src=3D"cid=
:72acbdf9e9761a33ad68ac239cb304da61f1cda1@zimbra" alt=3D""
data-mce-src=3D"=
/home/admin(a)lyra-network.com/corporate-documents/element-signature_payzen_6=
1x28.jpg" doc=3D"corporate-documents/element-signature_payzen_61x28.jpg"
wi=
dth=3D"61" border=3D"0" height=3D"28"></a><br
data-mce-bogus=3D"1"></td></t=
r></tbody></table><table style=3D"margin: 0; table-layout: fixed;
border: n=
one; width: 345pt;" data-mce-style=3D"margin: 0; table-layout: fixed; borde=
r: none; width: 345pt;" width=3D"460" cellspacing=3D"0"
cellpadding=3D"0"><=
tbody><tr><td><hr style=3D"background: #293b79 none repeat scroll
0% 0%; wi=
dth: 100%; height: 4px; color: #293b79; margin: 0px;" data-mce-style=3D"bac=
kground: #293b79 none repeat scroll 0% 0%; width: 100%; height: 4px; color:=
#293b79; margin:
0px;"></td></tr></tbody></table><table
style=3D"margin: 0=
; table-layout: fixed; border: none; width: 345pt;" data-mce-style=3D"margi=
n: 0; table-layout: fixed; border: none; width: 345pt;" width=3D"460"
cells=
pacing=3D"0" cellpadding=3D"0"><tbody><tr><td
style=3D"margin: 0; font-size=
: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; w=
idth: 30.75pt; text-align: left;" data-mce-style=3D"margin: 0; font-size: 1=
1px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; widt=
h: 30.75pt; text-align: left;"><div>Lyra Network, 109 rue de
l'innovation, =
31670 Lab=C3=A8ge,
FRANCE</div></td></tr></tbody></table><div><br></div></d=
iv><br><hr id=3D"zwchr"
data-marker=3D"__DIVIDER__"><div data-marker=3D"__H=
EADERS__"><b>De: </b>"Nelson Lameiras"
&lt;nelson.lameiras(a)lyra-network.com=
><br><b>=C3=80: </b>"ovirt users"
&lt;users(a)ovirt.org&gt;<br><b>Envoy=C3=
=A9: </b>Mercredi 7 Juin 2017 14:59:48<br><b>Objet:
</b>[ovirt-users] ISCSI=
storage with multiple nics on same subnet disabled on host activation<br><=
/div><br><div data-marker=3D"__QUOTED_TEXT__"><div
style=3D"font-family: ar=
ial, helvetica, sans-serif; font-size: 12pt; color:
#000000"><div>Hello,<br=
</div><br><div>In our oVirt hosts, we are using DELL
equallogic SAN with e=
ach server connecting to SAN via 2 physical interfaces. Since
both interfac=
es share the same network (Equalogic limitation) we must configure sysctl t=
o to allow iSCSI multipath with multiple NICs in the same subnet :<br></div=
<br><div>-----------------------------------------------------------------=
---------------------------<br></div><div><pre>net.ipv4.conf.p2p1.arp_ignor=
e=3D1
net.ipv4.conf.p2p1.arp_announce=3D2
net.ipv4.conf.p2p1.rp_filter=3D2
net.ipv4.conf.p2p2.arp_ignore=3D1
net.ipv4.conf.p2p2.arp_announce=3D2
net.ipv4.conf.p2p2.rp_filter=3D2<br><br></pre><p>--------------------------=
------------------------------------------------------------------<br><br><=
/p><p>This works great in most setups, but for a strange reason, on some of=
our setups, the sysctl configuration is updated by VDSM when activating a =
host and the second interface stops working immeadiatly :</p><div>---------=
---------------------------------------------------------------------------=
--------</div><div>vdsm.log<br></div><div><pre
id=3D"preview"><code>2017-06=
-07 11:51:51,063+0200 INFO (jsonrpc/5) [storage.ISCSI] Setting strict mode=
rp_filter for device 'p2p2'. (iscsi:602)
2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not conn=
ect to storageServer (hsm:2392)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer
conObj.connect()
File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect
iscsi.addIscsiNode(self._iface, self._target, self._cred)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, =
in addIscsiNode
iscsiadm.node_login(iface.name, target.address, target.iqn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 33=
7, in node_login
raise IscsiNodeError(rc, out,
err)</code></pre><pre><br><br></pre><p st=
yle=3D"margin: 0px;">------------------------------------------------------=
--------------------------------------</p></div></div><div><br><div>"strict=
mode" is enforced for second interface, and it no longuer
works...</div><d=
iv>Which means - at least - that there is no redundancy in case of hardware=
faillure and this is not acceptable for our production
needs.<br></div><br=
<div>What is really strange is that we have another
"twin" site on another=
geographic region with simillar hardware
configuration and same oVirt inst=
allation, and this problem does not happen.<br></div><div>Can this be
reall=
y random?<br></div>What can be the root cause of this behaviour? How can I =
correct it?<br><br><div>our setup:</div><div>oVirt
hostedEngine : Centor 7.=
3, ovirt 4.1.2<br></div><div>3 physical oVirt nodes centos 7.3, ovirt
4.1.2=
<br></div><div>SAN DELL
Equalogic<br></div></div><br><div><div>cordialement=
, regards,</div><br><table style=3D"margin: 0px; table-layout: fixed;
borde=
r: none; width: 345pt; padding: 0px;" width=3D"460"
cellspacing=3D"0" cellp=
adding=3D"0" border=3D"0"><tbody><tr><td
style=3D"margin: 0; font-size: 11p=
x; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width:=
86.25pt; text-align: left;" width=3D"115"
valign=3D"top"><a href=3D"https:=
//www.lyra-network.com/" style=3D"border: none; text-decoration: none; widt=
h: 100%; display: block; clear: both;" target=3D"_blank"><img
src=3D"cid:6f=
a8666d679e18cab5e566228d0ca3519aabec27@zimbra" alt=3D""
width=3D"115" borde=
r=3D"0" height=3D"94"></a><br></td><td
style=3D"margin: 0; font-size: 11px;=
font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 2=
58.75pt;" width=3D"345" valign=3D"top"><table
style=3D"margin: 0; width: 25=
8.75pt; table-layout: fixed; border: none;" width=3D"345"
cellspacing=3D"0"=
cellpadding=3D"0" border=3D"0"><tbody><tr><td
style=3D"font-size: 16px; fo=
nt-family: Arial,sans-serif; color: #2b3c7a; line-height: 18px; font-weight=
: bold;">Nelson LAMEIRAS</td></tr><tr><td
style=3D"font-family: Arial,sans-=
serif; color: #2b3c7a; font-size: 11px; line-height: 16px;">Ing=C3=A9nieur =
Syst=C3=A8mes et R=C3=A9seaux<span style=3D"font-family: Arial,sans-serif; =
color: #337ab7; font-size: 11px;"> / Systems and Networks
engineer</span></=
td></tr><tr><td style=3D"line-height: 16px; font-family:
'Arial',sans-serif=
; color: #2b3c7a; font-size: 11px;"><span style=3D"font-family:
Arial,sans-=
serif; color: #2b3c7a; font-size: 11px;">Tel: +33 5 32 09 09
70</span></td>=
</tr><tr><td style=3D"line-height: 16px; font-family:
'Arial',sans-serif; c=
olor: #2b3c7a; font-size: 11px;"><a
href=3D"mailto:nelson.lameiras@lyra-net=
work.com" style=3D"font-size: 11px; font-family: Arial,sans-serif; line-hei=
ght: 16px; text-decoration: none;"
target=3D"_blank">nelson.lameiras@lyra-n=
etwork.com</a><br></td></tr><tr><td
style=3D"line-height: 16px; font-family=
: 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;"><a
href=3D"https://=
www.lyra-network.com/" style=3D"font-size: 11px; font-family: Arial,sans-se=
rif; line-height: 16px; border: none; text-decoration: none;" target=3D"_bl=
ank">www.lyra-network.com</a> <a href=3D"https://payzen.eu"
style=3D"font-s=
ize: 11px; font-family: Arial,sans-serif; line-height: 16px; border: none; =
text-decoration: none;" target=3D"_blank"> |
www.payzen.eu</a><br></td></tr=
</tbody></table></td></tr></tbody></table><table
style=3D"margin: 0; table=
-layout: fixed; border: none; width: 345pt;"
data-mce-selected=3D"1" width=
=3D"460" cellspacing=3D"0"
cellpadding=3D"0"><tbody><tr><td style=3D"margin=
: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-h=
eight: 16px; width: 24pt; text-align: left;" width=3D"32"
valign=3D"top"><a=
href=3D"https://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA" style=3D=
"border: none; text-decoration: none; margin: 0; float: left; display: inli=
ne-block; width: 32px;" target=3D"_blank"><img
src=3D"cid:2806a89cb9fbba6f8=
79bd3fd54a93b515defd6d2@zimbra" alt=3D"" width=3D"32"
border=3D"0" height=
=3D"28"></a><br></td><td style=3D"margin: 0;
font-size: 11px; font-family: =
Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 30.75pt; text-a=
lign: left;" width=3D"41" valign=3D"top"><a
href=3D"https://www.linkedin.co=
m/company/lyra-network_2" style=3D"border: none; text-decoration: none; mar=
gin: 0; float: left; display: inline-block; width: 41px;"
target=3D"_blank"=
<img
src=3D"cid:878634ba055bb6ae851a7ef4540f754afa3ff86c@zimbra" width=3D"=
41" border=3D"0"
height=3D"28"></a><br></td><td style=3D"margin: 0;
font-si=
ze: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px;=
width: 31.5pt; text-align: left;" width=3D"42"
valign=3D"top"><a href=3D"h=
ttps://twitter.com/LyraNetwork" style=3D"border: none; text-decoration: non=
e; margin: 0; float: left; display: inline-block; width: 42px;" target=3D"_=
blank"><img
src=3D"cid:18a767bbdad6a1133fc4152b03ea78228ce25a14@zimbra" alt=
=3D"" width=3D"42" border=3D"0"
height=3D"28"></a><br></td><td style=3D"mar=
gin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; lin=
e-height: 16px; width: 258.75pt; text-align: left;" width=3D"345"
valign=3D=
"top"><a href=3D"https://payzen.eu" style=3D"border: none;
text-decoration:=
none; margin: 0; float: left; display: inline-block; width: 61px;" target=
=3D"_blank"><img
src=3D"cid:d3fc119a0aa72bf985c19ca559c993b5b4b54ca3@zimbra=
" alt=3D"" width=3D"61" border=3D"0"
height=3D"28"></a><br></td></tr></tbod=
y></table><table style=3D"margin: 0; table-layout: fixed; border: none;
wid=
th: 345pt;" width=3D"460" cellspacing=3D"0"
cellpadding=3D"0"><tbody><tr><t=
d><hr style=3D"background: #293b79 none repeat scroll 0% 0%; width: 100%; h=
eight: 4px; color: #293b79; margin:
0px;"></td></tr></tbody></table><table =
style=3D"margin: 0; table-layout: fixed; border: none; width: 345pt;" data-=
mce-selected=3D"1" width=3D"460" cellspacing=3D"0"
cellpadding=3D"0"><tbody=
<tr><td style=3D"margin: 0; font-size: 11px;
font-family: Arial,sans-serif=
; color: #2b3c7a; line-height: 16px; width: 30.75pt;
text-align: left;"><di=
v>Lyra Network, 109 rue de l'innovation, 31670 Lab=C3=A8ge,
FRANCE</div></t=
d></tr></tbody></table><br></div></div><br>________________________________=
_______________<br>Users mailing
list<br>Users@ovirt.org<br>http://lists.ov=
irt.org/mailman/listinfo/users<br></div></div></body...
------=_Part_88180576_1156318871.1500303388932
Content-Type: image/jpeg; name=element-signature_logo_lyra_115x94.jpg
Content-Disposition: attachment;
filename=element-signature_logo_lyra_115x94.jpg
Content-Transfer-Encoding: base64
Content-ID: <6fa8666d679e18cab5e566228d0ca3519aabec27@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOkU1QUU4Nzk1OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOkU1QUU4Nzk0OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIAF4AcwMBEQACEQEDEQH/xACdAAEBAAIDAQEAAAAAAAAAAAAACAUH
AgMGBAEBAQADAQEAAAAAAAAAAAAAAAACBAUBAxAAAAUDAgAHDQcCBwAAAAAAAAECAwQRBQYSByFR
E7RWFzcxQSKS0nOT03RVlQgYYXEyQiMUFYEzsVJicmODOBEAAgECAwcEAgIDAAAAAAAAAAECEQNR
EwQhgbESMlIzMUGhFHEVwQUiQzT/2gAMAwEAAhEDEQA/AKpAE9b/AGQZdF3FtNpsd4l29M2DHSll
iS6w0bzsp5slqJCiLvERnTuENTRQg7bckntK15vmojj1WfMP0rc+Jy/JD7Fjt+EMueJ6bbnBN4bR
lkWdkt+XPtDaHSfjKnSHyUpTZkg+TcLSdFGRjyv3rUo0iqP8EoQkntZ7fcOy5VdrXGYxuccGW2+S
3nCecYq3oUWnU2RmfhGXAPPSXLcJNzVVT8kNVbnKKUHRmppW1/zBOSXVtZS4lpSjNtP8nLKhfcSR
alqdPXp+ERhauKKTe38nX1WfMP0qc+Jy/JHPsWO34RPLniOqz5h+lTnxOX5IfYsdvwhlzxHVZ8w/
Spz4nL8kPsWO34Qy54jqs+YfpU58Tl+SH2LHb8IZc8R1WfMP0qc+Jy/JD7Fjt+EMueJ2bCX3MH9y
LxZr7eJdxKBCkoW0/JdfaJ5mUy2a0EszL/MRHTuGGshDLTiqVf8AAst81GUGMssgAAAAE5b89s+L
+zwefPDV0fhlv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACc
t+e2fF/Z4PPnhq6Pwy38Crd60UaMotAAAAAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLI
AAAABOW/PbPi/s8Hnzw1dH4Zb+BVu9aKNGUWjAKzrGk5cnEjkq/nVo5RMbk3NOnkzd/uU0fgKvdH
rky5eb2I86rQz48iQAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLIAAAABOW/PbPi/s8Hn
zw1dH4Zb+BVu9aKNGUWjSD//AKkj+xnzJQ0V/wA2/wDkr/7D3e7GX5DiOMfzlniMTEMPIRORIJfg
tOeCladCk/nNJH94raa1GcuVnpck4qqOrJdz4dr2yazKOhDjkxhlUGMszop98i/TOlD/AE/CNX+0
x23p27nIJXKRqeeyjdnLMd27sGSzLdEO4Xd4uWifqkhDDiFut0qrUSzQlNa9wz7g9bemjK44puiI
yuNRTOzGs53hud+gvzsYjwMVnqNfLuqUTrEck69bitZmStPc1NJIz4hy5ZtKLpKskIzk36bDHRt2
9yMsuMw9v8ejybNBc5M5s5Rp5Uy4aEZuMJSai4dPCZFStBN6a3BLne1nMyT6Uffgu8GQ5DnysXuN
mbtZx461TG1ms3kyGiLWRHXToMz8Hg7nfMQvaWMYcydTsLjcqG2BSPYnbY/twy3zVw5+yNXV+GO7
gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLROmdZbAxL5g/5yey6/GjREJW2wSTc
M3YpoKms0F3VcY1bNpzscqKs5UnU9pad4cG3Fefw1UWXFO7x3mULkpaJJnoM6JNC10WREak/aXGK
8tLO1/ns2HorilsNTYzar/kOQWTa+5tmmBjk+W9cKGqimkrJS68RcCkoV/yC7clGMXcXrJI8Yptq
OBsj5oEpTiNmSRESSuKSJJFwUJlzgoKv9f1P8Hrf9Db1wh/vbVJh6tH7lhbOvi5RBpr/AEqKMXR1
PZrYaC2x3Jgbawp+IZjBkwpkWS480603rSslkRU7qTOuiqFlUlEfepw6WosO61ODK9ufLsZ27f36
RkG/8q9OwXbe3PgKciMPp0uHHS22hpxRf60o1f4VC9DlsUrWjEHWdSghllknbY/twy3zVw5+yNXV
+GO7gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLRxU00o6qQlR8ZkRhUGMyO7W7H
7FOvUpCSZgMrfMqERqNJeCgj41qokvtMTtxcpJL3OSdFU1n8vFimPwbrnF18O55DIXybhlw8ihZm
s08RLdqVOJJC5rZqqgvSJ42V7v3MpM3dTBnyWbtBQmHFnXFpLzdXOWjW9tWo2irwPpd0pWhVPBPU
R0qILTVWx+y+STuYmQj7sxHH1Kctb7NuahnMflKW2akk3J/bvGlojqtpv8ZuJP8ACR8Ag9M8dtTu
YdErdfEJTCZKrY/L5JSySbrbCdKmicdWRLeWlJK/bs8skq+ElSacJ8HVppr3OZiP2RuxFK4W5xi3
LK2T5D8NmY8baHHnWySSeTSa6to1K8JTumn3gtM6PbtQzDYYqnqTtsf24Zb5q4c/ZGrq/DHdwK1r
rZRIyiyAAAAATlvz2z4v7PB588NXR+GW/gVbvWijRlFoADGZHjVmyO2Ltd4YOTAcUlbjJLcbJRoO
qam2pCuA+GlROFxwdV6nJRT2M+q226DbLfHt8BkmIcRtLMdlNaJQgqEVTqZ/eYjKTbqwlQ5Hb4Bp
Uk4zRpUpa1FoTQ1uEaVqPg7qiUZGffDmYodSbNZ0toaTBjk02hbTaCaQSUtu/wBxCSpQkr/MXfHe
d4iiP12z2h5k2XYMdxlSkuKbW0hSTW2kkIUZGVKpSkiI+8RApPEURydtltdQpDsRlxCtepKm0mR8
qdXKkZfn/NxjnMxQ+kcOk7bH9uGW+auHP2Rq6vwx3cCta62USMosgAAAAE5b89s+L+zwefPDV0fh
lv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACb9/3mWN4cbff
WTTLUWEt1xX4UpTNeNSj+wiGtolW1LfwKt7qRuTrY226RwfSkKH1rnaz3zI4jrZ226RwfSkH1rna
xmRxHWztt0jg+lIPrXO1jMjiOtnbbpHB9KQfWudrGZHEdbO23SOD6Ug+tc7WMyOJ+o3W24WtKE5F
CUpR0SknCqZmH1rnazjuxW2pnbTfbPeGnHbXMamNtK0OLaVqJKjKtDELlqUNklQ7C5GfS6mOu+f4
XZpy4F0vMWHMbJKlsOrJKyJRVTUvtIdjZnJVSOuaXqfF1sbbdI4PpSEvrXO1nMyOJp7YiQxJ3oyi
RHcS9Hejz3GXUcKVoVOZNKi+wyOovaxUsx3cDwtdbKNGUWgAAAADX+4GzGPZveWbtcpsuO+xGTFS
iObRINCFrcIz1oWdaun3xas6qVtUSR5ztKTqeZ+lrCvely8Zj1Q9f2M8EQyEPpawr3pcvGY9UH7G
eCGQh9LWFe9Ll4zHqg/YzwQyEPpawr3pcvGY9UH7GeCGQh9LWFe9Ll4zHqg/YzwQyEc2flfwxl5t
1NzuJqbUSiI1MUqX/UOr+xmn6I5LTRaaxNg4XhNuxOHIiwX3n0SXCdWb5pMyMk6eDSlPEPDU6mV5
pteg0+nVpNI8zm+xeNZfkDt7nzprEl5Dbam2FNEgibTpKmttR97jErWslCNEkTlaUnUwP0tYV70u
XjMeqHp+xngiOQj023+zGPYReXrtbZsuQ+/GVFUiQbRoJC1ocMy0IQdatF3x5XtVK4qNInC0oups
AVT0AAAAAAAAAAAAAAAAAAAAAAAAAAAAD//Z
------=_Part_88180576_1156318871.1500303388932
Content-Type: image/jpeg; name=element-signature_logo_YouTube_32x28.jpg
Content-Disposition: attachment;
filename=element-signature_logo_YouTube_32x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <2806a89cb9fbba6f879bd3fd54a93b515defd6d2@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOjQ1REY5Njk1OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOjQ1REY5Njk0OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAIAMBEQACEQEDEQH/xAB5AAADAQEAAAAAAAAAAAAAAAAABwgF
BgEAAgMBAAAAAAAAAAAAAAAAAAQCAwUBEAABBAECBAUFAAAAAAAAAAABAgMEBQARBiESEwcxUdIU
CEEik1UYEQACAQMCBQUAAAAAAAAAAAAAAQIRAwQhMVFhEiITQaHBYhX/2gAMAwEAAhEDEQA/AKpw
AnD5Hbg3JA35XxKqxmxUPVrJEaI862FuqkPp4IbI5lHQDNXBhFwbaW4tebqcMIvfYjURtz6eesv1
YzWz9fYh38zR7eX2+mu5lHV3VhaNOe8aTJgTHnweVXEBba1cQRkb8IeNtJbHYN9WpW+YY2T53inQ
4HfzZk6a4lmHGaiOSHl8EoT7mSOZR+gBIzUxU3Yklz+Be4+9DRl30N3dMOxjbyro9EzHW3MqupHc
L7qlaoX1SsFvlHl44moPpa6XXiW113FPuu3qrT5I7dfrJLUxllMRl15hQWjqBbiuXmTqCQlQx23F
rHdSqTrNFFZlDAou7vZa43xuSNaw7CPFZZhIiKbeSsqKkOuuE/aCNNHcexspW40a9Sm5bcmcH/J1
5+0rvxOenGf0o8GQ8DN7ZXxzv9vbpqrhy0huMQJCHlstocBKU+ITqANcru50ZRao9SUbLTqPvMwv
DAAwAMAP/9k=
------=_Part_88180576_1156318871.1500303388932
Content-Type: image/jpeg; name=element-signature_logo_LinkedIn_41x28.jpg
Content-Disposition: attachment;
filename=element-signature_logo_LinkedIn_41x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <878634ba055bb6ae851a7ef4540f754afa3ff86c@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOjY2MEYzRTFGOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOjY2MEYzRTFFOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAKQMBEQACEQEDEQH/xAB2AAEBAQEAAAAAAAAAAAAAAAAABggH
AQEAAwEAAAAAAAAAAAAAAAAAAwQFAhAAAQMEAgICAgMBAAAAAAAAAQIDBAARBQYhEjEHMghBE1Fh
IhcRAAICAQIDCQAAAAAAAAAAAAABAgMREgQxkTIhQVGhscETMxX/2gAMAwEAAhEDEQA/ANU0Bl33
zsuzQfZcmHj8tMiMKaihDLMl1psKcQkXslSUjk8mtjZ1xdeWkVbW9RDZ/ZN8wOXkYnI7LKMuL1Lq
o+RddaIWkLHVwLAPCuf4NWYVwksqPkcNtd5V+ldr2ef7MwkaZmJsqK6p/uy9JdcbUBGcULpUog8i
9Q7uuKreEjquT1GsKxC2KAUBk37B2/6y72t1tB7X8dbIvf8Aq3mtvZfVzKlvUWWw7Dr+Dfy8rCt4
d153Z8ew0FNx3kohuxmUyFtAfFPyBUOBz+arwhKWE89L9SRtLmTuns4xj7LLaxYaGNE6SYojlJZ6
rhLUf19f89eyj4qa1t7ft449ziPWairGLQoBQHFvZ/onObht0jNxclFjMPNtNpZdS4Vj9aAk36gj
mtDb7yMIaWiGdTbySSfqnsKbdctAFh1Fm3Bwfx8fFT/ox8GcfAyk9dfX7OapuGNzj+SiPRoJcKmG
kuBRDjS2x1uAPK71DfvYzg444nUKmnk7lWcTigFAKAUAoBQCgP/Z
------=_Part_88180576_1156318871.1500303388932
Content-Type: image/jpeg; name=element-signature_logo_Twitter_42x28.jpg
Content-Disposition: attachment;
filename=element-signature_logo_Twitter_42x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <18a767bbdad6a1133fc4152b03ea78228ce25a14@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOjdDMkI3MjMyOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOjdDMkI3MjMxOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAKgMBEQACEQEDEQH/xAB3AAEBAQEBAAAAAAAAAAAAAAAABwUI
BgEBAAMBAQAAAAAAAAAAAAAAAAMEBQIBEAABAwIEBwEBAQAAAAAAAAABAgMEAAUREhMGITFBFBUH
CIEiFhEAAgIBAgQHAAAAAAAAAAAAAAERAgMxBCHBEgVBUWGBQhMV/9oADAMBAAIRAxEAPwDC9g7u
3dH33uJiNeri2wxcJQQ01JfCUNodVwCUqwSlI/AK3sGKropS0Kd7OWZMXcPsyZDdnQ7he5UFhQQ/
LZelOMoURiEqWlRAOFdumNOGqyeTYtPzFfL1df8ASm5XCTP0ey0e5eceyZu4zZc5VhjlGOFZ/cKJ
dMKNeRNhbclzrOJxQCgOSW7o5a/oKXMRFXNT5iYw/EaAUtbL61tOBKTwXglWOXrW30zgj0RUmLlb
9p3rZ/r/ANd3GxWJtm3XC8B1EO2RsEqDkng88pAP8JSnEk8ugqlt6Xy5Fa3FImyNVUI8z8nJCU7o
SOSRAA/O5qbuXx9+Rxg8ToGsssCgFAc/7q+b9xXrc11u7V3iNN3CW/JbQUuhaUuuFYBIHMA9K08e
+rWqUPgivbC25MofK25Qoq81DKjwKlJeUT+kE13+jXyZ59DKX6a9WXPYfmO+msTPJdtp6AWMuhq4
5swHPVFVN1uFkiFoS46dJSqqEgoBQCgFAKAUAoBQH//Z
------=_Part_88180576_1156318871.1500303388932
Content-Type: image/jpeg; name=element-signature_payzen_61x28.jpg
Content-Disposition: attachment; filename=element-signature_payzen_61x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <d3fc119a0aa72bf985c19ca559c993b5b4b54ca3@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOkU1NjczQjM2OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOkU1NjczQjM1OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAPQMBEQACEQEDEQH/xABvAAABBAMAAAAAAAAAAAAAAAAAAQUG
BwIDBAEBAAAAAAAAAAAAAAAAAAAAABAAAQQABAQEAgsBAAAAAAAAAQIDBAUAERIGITETFEEiFQfS
I1FhcYGhMlIkRMQWhhEBAAAAAAAAAAAAAAAAAAAAAP/aAAwDAQACEQMRAD8AvqZ7q7OhzZMSS9IQ
Yjy2JT/avqZbW2rSvU6lBSACOeeAljLzTzSHmVhxpxIW24k5pUlQzBBHgRgM8BBJ9fKuvcGwr3La
wgxIlfGeaagyVsJ1uOOBRUBmOSRgI0bnZgfQwdx7qDrroYZSVzBrcJ0hKM2xq+7AWBU7PTXTEShc
2svRx6MuWp5o5gjigj68BombFRKmPyTfXTPXWpzoszloaRqOelCQPKkeAwEP6lp6f6V6vYaP9Z6b
3fcr7ntuhno6vPnxwGyohb2ntbpr6Y17NdKt7Bp5+YHlPJLitKyltI0KGk+XPANVgzV1Eyyqbe3s
IMmnhsMbVaYcebDraY+XVbQ15XXFOghQVnkMBkbjcVPTUs6M5IkPbnpWa1hKlrWG7QZBp0hROnUh
xRUfHTgJDWuwtsbptVTpC1xaqhgJfkuFTji9DjoKiTmpSlq/HAcO2d11d/cuXUySJG4C0+NvbfCH
CmK2hClHUop0GQ6E+dWfAeUYBi2lY2E+zgW7twhVlHDk+7AkzFO9u2lRdjORFIEZrTmAkA+GYwDl
TwLJhjZFouxsF3NzN6r7Lj7imExHUOSHWi0TpyCdORIzzwHT/J/7n+vgLVAA5YBChBUlRSCpOelR
HEZ88sA2WG3YVhb1lnJW4VVJcXFjAgM9VxOjqKTlmVJTnp48M8Ax2+3N3J3TJu6KVAbRKisxnWpr
brh+SpasxoKf14BBD91hxE2kB+nt5Px4AEL3VGoiZSAr4qyjyOP2+fAL2nuvw/e0vDl8iT8eAbv8
Lur0vPvYXrPrfrfU6bvb59Lp6NOev83HngLBwBgDAGAMAYAwBgP/2Q==
------=_Part_88180576_1156318871.1500303388932
Content-Type: image/jpeg; name=element-signature_logo_lyra_115x94.jpg
Content-Disposition: attachment;
filename=element-signature_logo_lyra_115x94.jpg
Content-Transfer-Encoding: base64
Content-ID: <ac318af44fd2ff936fd390a2dfa5ac7f8bf624d9@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOkU1QUU4Nzk1OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOkU1QUU4Nzk0OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIAF4AcwMBEQACEQEDEQH/xACdAAEBAAIDAQEAAAAAAAAAAAAACAUH
AgMGBAEBAQADAQEAAAAAAAAAAAAAAAACBAUBAxAAAAUDAgAHDQcCBwAAAAAAAAECAwQRBQYSByFR
E7RWFzcxQSKS0nOT03RVlQgYYXEyQiMUFYEzsVJicmODOBEAAgECAwcEAgIDAAAAAAAAAAECEQNR
EwQhgbESMlIzMUGhFHEVwQUiQzT/2gAMAwEAAhEDEQA/AKpAE9b/AGQZdF3FtNpsd4l29M2DHSll
iS6w0bzsp5slqJCiLvERnTuENTRQg7bckntK15vmojj1WfMP0rc+Jy/JD7Fjt+EMueJ6bbnBN4bR
lkWdkt+XPtDaHSfjKnSHyUpTZkg+TcLSdFGRjyv3rUo0iqP8EoQkntZ7fcOy5VdrXGYxuccGW2+S
3nCecYq3oUWnU2RmfhGXAPPSXLcJNzVVT8kNVbnKKUHRmppW1/zBOSXVtZS4lpSjNtP8nLKhfcSR
alqdPXp+ERhauKKTe38nX1WfMP0qc+Jy/JHPsWO34RPLniOqz5h+lTnxOX5IfYsdvwhlzxHVZ8w/
Spz4nL8kPsWO34Qy54jqs+YfpU58Tl+SH2LHb8IZc8R1WfMP0qc+Jy/JD7Fjt+EMueJ2bCX3MH9y
LxZr7eJdxKBCkoW0/JdfaJ5mUy2a0EszL/MRHTuGGshDLTiqVf8AAst81GUGMssgAAAAE5b89s+L
+zwefPDV0fhlv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACc
t+e2fF/Z4PPnhq6Pwy38Crd60UaMotAAAAAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLI
AAAABOW/PbPi/s8Hnzw1dH4Zb+BVu9aKNGUWjAKzrGk5cnEjkq/nVo5RMbk3NOnkzd/uU0fgKvdH
rky5eb2I86rQz48iQAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLIAAAABOW/PbPi/s8Hn
zw1dH4Zb+BVu9aKNGUWjSD//AKkj+xnzJQ0V/wA2/wDkr/7D3e7GX5DiOMfzlniMTEMPIRORIJfg
tOeCladCk/nNJH94raa1GcuVnpck4qqOrJdz4dr2yazKOhDjkxhlUGMszop98i/TOlD/AE/CNX+0
x23p27nIJXKRqeeyjdnLMd27sGSzLdEO4Xd4uWifqkhDDiFut0qrUSzQlNa9wz7g9bemjK44puiI
yuNRTOzGs53hud+gvzsYjwMVnqNfLuqUTrEck69bitZmStPc1NJIz4hy5ZtKLpKskIzk36bDHRt2
9yMsuMw9v8ejybNBc5M5s5Rp5Uy4aEZuMJSai4dPCZFStBN6a3BLne1nMyT6Uffgu8GQ5DnysXuN
mbtZx461TG1ms3kyGiLWRHXToMz8Hg7nfMQvaWMYcydTsLjcqG2BSPYnbY/twy3zVw5+yNXV+GO7
gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLROmdZbAxL5g/5yey6/GjREJW2wSTc
M3YpoKms0F3VcY1bNpzscqKs5UnU9pad4cG3Fefw1UWXFO7x3mULkpaJJnoM6JNC10WREak/aXGK
8tLO1/ns2HorilsNTYzar/kOQWTa+5tmmBjk+W9cKGqimkrJS68RcCkoV/yC7clGMXcXrJI8Yptq
OBsj5oEpTiNmSRESSuKSJJFwUJlzgoKv9f1P8Hrf9Db1wh/vbVJh6tH7lhbOvi5RBpr/AEqKMXR1
PZrYaC2x3Jgbawp+IZjBkwpkWS480603rSslkRU7qTOuiqFlUlEfepw6WosO61ODK9ufLsZ27f36
RkG/8q9OwXbe3PgKciMPp0uHHS22hpxRf60o1f4VC9DlsUrWjEHWdSghllknbY/twy3zVw5+yNXV
+GO7gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLRxU00o6qQlR8ZkRhUGMyO7W7H
7FOvUpCSZgMrfMqERqNJeCgj41qokvtMTtxcpJL3OSdFU1n8vFimPwbrnF18O55DIXybhlw8ihZm
s08RLdqVOJJC5rZqqgvSJ42V7v3MpM3dTBnyWbtBQmHFnXFpLzdXOWjW9tWo2irwPpd0pWhVPBPU
R0qILTVWx+y+STuYmQj7sxHH1Kctb7NuahnMflKW2akk3J/bvGlojqtpv8ZuJP8ACR8Ag9M8dtTu
YdErdfEJTCZKrY/L5JSySbrbCdKmicdWRLeWlJK/bs8skq+ElSacJ8HVppr3OZiP2RuxFK4W5xi3
LK2T5D8NmY8baHHnWySSeTSa6to1K8JTumn3gtM6PbtQzDYYqnqTtsf24Zb5q4c/ZGrq/DHdwK1r
rZRIyiyAAAAATlvz2z4v7PB588NXR+GW/gVbvWijRlFoADGZHjVmyO2Ltd4YOTAcUlbjJLcbJRoO
qam2pCuA+GlROFxwdV6nJRT2M+q226DbLfHt8BkmIcRtLMdlNaJQgqEVTqZ/eYjKTbqwlQ5Hb4Bp
Uk4zRpUpa1FoTQ1uEaVqPg7qiUZGffDmYodSbNZ0toaTBjk02hbTaCaQSUtu/wBxCSpQkr/MXfHe
d4iiP12z2h5k2XYMdxlSkuKbW0hSTW2kkIUZGVKpSkiI+8RApPEURydtltdQpDsRlxCtepKm0mR8
qdXKkZfn/NxjnMxQ+kcOk7bH9uGW+auHP2Rq6vwx3cCta62USMosgAAAAE5b89s+L+zwefPDV0fh
lv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACb9/3mWN4cbff
WTTLUWEt1xX4UpTNeNSj+wiGtolW1LfwKt7qRuTrY226RwfSkKH1rnaz3zI4jrZ226RwfSkH1rna
xmRxHWztt0jg+lIPrXO1jMjiOtnbbpHB9KQfWudrGZHEdbO23SOD6Ug+tc7WMyOJ+o3W24WtKE5F
CUpR0SknCqZmH1rnazjuxW2pnbTfbPeGnHbXMamNtK0OLaVqJKjKtDELlqUNklQ7C5GfS6mOu+f4
XZpy4F0vMWHMbJKlsOrJKyJRVTUvtIdjZnJVSOuaXqfF1sbbdI4PpSEvrXO1nMyOJp7YiQxJ3oyi
RHcS9Hejz3GXUcKVoVOZNKi+wyOovaxUsx3cDwtdbKNGUWgAAAADX+4GzGPZveWbtcpsuO+xGTFS
iObRINCFrcIz1oWdaun3xas6qVtUSR5ztKTqeZ+lrCvely8Zj1Q9f2M8EQyEPpawr3pcvGY9UH7G
eCGQh9LWFe9Ll4zHqg/YzwQyEPpawr3pcvGY9UH7GeCGQh9LWFe9Ll4zHqg/YzwQyEc2flfwxl5t
1NzuJqbUSiI1MUqX/UOr+xmn6I5LTRaaxNg4XhNuxOHIiwX3n0SXCdWb5pMyMk6eDSlPEPDU6mV5
pteg0+nVpNI8zm+xeNZfkDt7nzprEl5Dbam2FNEgibTpKmttR97jErWslCNEkTlaUnUwP0tYV70u
XjMeqHp+xngiOQj023+zGPYReXrtbZsuQ+/GVFUiQbRoJC1ocMy0IQdatF3x5XtVK4qNInC0oups
AVT0AAAAAAAAAAAAAAAAAAAAAAAAAAAAD//Z
------=_Part_88180576_1156318871.1500303388932
Content-Type: image/jpeg; name=element-signature_logo_YouTube_32x28.jpg
Content-Disposition: attachment;
filename=element-signature_logo_YouTube_32x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <6f3cc57ba362f7a2942a16e88d1ea9ce686c7da2@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOjQ1REY5Njk1OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOjQ1REY5Njk0OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAIAMBEQACEQEDEQH/xAB5AAADAQEAAAAAAAAAAAAAAAAABwgF
BgEAAgMBAAAAAAAAAAAAAAAAAAQCAwUBEAABBAECBAUFAAAAAAAAAAABAgMEBQARBiESEwcxUdIU
CEEik1UYEQACAQMCBQUAAAAAAAAAAAAAAQIRAwQhMVFhEiITQaHBYhX/2gAMAwEAAhEDEQA/AKpw
AnD5Hbg3JA35XxKqxmxUPVrJEaI862FuqkPp4IbI5lHQDNXBhFwbaW4tebqcMIvfYjURtz6eesv1
YzWz9fYh38zR7eX2+mu5lHV3VhaNOe8aTJgTHnweVXEBba1cQRkb8IeNtJbHYN9WpW+YY2T53inQ
4HfzZk6a4lmHGaiOSHl8EoT7mSOZR+gBIzUxU3Yklz+Be4+9DRl30N3dMOxjbyro9EzHW3MqupHc
L7qlaoX1SsFvlHl44moPpa6XXiW113FPuu3qrT5I7dfrJLUxllMRl15hQWjqBbiuXmTqCQlQx23F
rHdSqTrNFFZlDAou7vZa43xuSNaw7CPFZZhIiKbeSsqKkOuuE/aCNNHcexspW40a9Sm5bcmcH/J1
5+0rvxOenGf0o8GQ8DN7ZXxzv9vbpqrhy0huMQJCHlstocBKU+ITqANcru50ZRao9SUbLTqPvMwv
DAAwAMAP/9k=
------=_Part_88180576_1156318871.1500303388932
Content-Type: image/jpeg; name=element-signature_logo_LinkedIn_41x28.jpg
Content-Disposition: attachment;
filename=element-signature_logo_LinkedIn_41x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <95ac2affdb66bc26fd0e254578e26d8574139b42@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOjY2MEYzRTFGOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOjY2MEYzRTFFOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAKQMBEQACEQEDEQH/xAB2AAEBAQEAAAAAAAAAAAAAAAAABggH
AQEAAwEAAAAAAAAAAAAAAAAAAwQFAhAAAQMEAgICAgMBAAAAAAAAAQIDBAARBQYhEjEHMghBE1Fh
IhcRAAICAQIDCQAAAAAAAAAAAAABAgMREgQxkTIhQVGhscETMxX/2gAMAwEAAhEDEQA/ANU0Bl33
zsuzQfZcmHj8tMiMKaihDLMl1psKcQkXslSUjk8mtjZ1xdeWkVbW9RDZ/ZN8wOXkYnI7LKMuL1Lq
o+RddaIWkLHVwLAPCuf4NWYVwksqPkcNtd5V+ldr2ef7MwkaZmJsqK6p/uy9JdcbUBGcULpUog8i
9Q7uuKreEjquT1GsKxC2KAUBk37B2/6y72t1tB7X8dbIvf8Aq3mtvZfVzKlvUWWw7Dr+Dfy8rCt4
d153Z8ew0FNx3kohuxmUyFtAfFPyBUOBz+arwhKWE89L9SRtLmTuns4xj7LLaxYaGNE6SYojlJZ6
rhLUf19f89eyj4qa1t7ft449ziPWairGLQoBQHFvZ/onObht0jNxclFjMPNtNpZdS4Vj9aAk36gj
mtDb7yMIaWiGdTbySSfqnsKbdctAFh1Fm3Bwfx8fFT/ox8GcfAyk9dfX7OapuGNzj+SiPRoJcKmG
kuBRDjS2x1uAPK71DfvYzg444nUKmnk7lWcTigFAKAUAoBQCgP/Z
------=_Part_88180576_1156318871.1500303388932
Content-Type: image/jpeg; name=element-signature_logo_Twitter_42x28.jpg
Content-Disposition: attachment;
filename=element-signature_logo_Twitter_42x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <5e9e243634081dcf099c93fd9ca78f8b346f57b3@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOjdDMkI3MjMyOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOjdDMkI3MjMxOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAKgMBEQACEQEDEQH/xAB3AAEBAQEBAAAAAAAAAAAAAAAABwUI
BgEBAAMBAQAAAAAAAAAAAAAAAAMEBQIBEAABAwIEBwEBAQAAAAAAAAABAgMEAAUREhMGITFBFBUH
CIEiFhEAAgIBAgQHAAAAAAAAAAAAAAERAgMxBCHBEgVBUWGBQhMV/9oADAMBAAIRAxEAPwDC9g7u
3dH33uJiNeri2wxcJQQ01JfCUNodVwCUqwSlI/AK3sGKropS0Kd7OWZMXcPsyZDdnQ7he5UFhQQ/
LZelOMoURiEqWlRAOFdumNOGqyeTYtPzFfL1df8ASm5XCTP0ey0e5eceyZu4zZc5VhjlGOFZ/cKJ
dMKNeRNhbclzrOJxQCgOSW7o5a/oKXMRFXNT5iYw/EaAUtbL61tOBKTwXglWOXrW30zgj0RUmLlb
9p3rZ/r/ANd3GxWJtm3XC8B1EO2RsEqDkng88pAP8JSnEk8ugqlt6Xy5Fa3FImyNVUI8z8nJCU7o
SOSRAA/O5qbuXx9+Rxg8ToGsssCgFAc/7q+b9xXrc11u7V3iNN3CW/JbQUuhaUuuFYBIHMA9K08e
+rWqUPgivbC25MofK25Qoq81DKjwKlJeUT+kE13+jXyZ59DKX6a9WXPYfmO+msTPJdtp6AWMuhq4
5swHPVFVN1uFkiFoS46dJSqqEgoBQCgFAKAUAoBQH//Z
------=_Part_88180576_1156318871.1500303388932
Content-Type: image/jpeg; name=element-signature_payzen_61x28.jpg
Content-Disposition: attachment; filename=element-signature_payzen_61x28.jpg
Content-Transfer-Encoding: base64
Content-ID: <72acbdf9e9761a33ad68ac239cb304da61f1cda1@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov
L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl
aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4
OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6
MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5
OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht
bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i
aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w
PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9
InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1
bWVudElEPSJ4bXAuZGlkOkU1NjczQjM2OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXBNTTpJ
bnN0YW5jZUlEPSJ4bXAuaWlkOkU1NjczQjM1OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXA6
Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N
OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1
LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4
YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG
PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE
BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f
Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f
Hx8fHx8fHx8fHx8fH//AABEIABwAPQMBEQACEQEDEQH/xABvAAABBAMAAAAAAAAAAAAAAAAAAQUG
BwIDBAEBAAAAAAAAAAAAAAAAAAAAABAAAQQABAQEAgsBAAAAAAAAAQIDBAUAERIGITETFEEiFQfS
I1FhcYGhMlIkRMQWhhEBAAAAAAAAAAAAAAAAAAAAAP/aAAwDAQACEQMRAD8AvqZ7q7OhzZMSS9IQ
Yjy2JT/avqZbW2rSvU6lBSACOeeAljLzTzSHmVhxpxIW24k5pUlQzBBHgRgM8BBJ9fKuvcGwr3La
wgxIlfGeaagyVsJ1uOOBRUBmOSRgI0bnZgfQwdx7qDrroYZSVzBrcJ0hKM2xq+7AWBU7PTXTEShc
2svRx6MuWp5o5gjigj68BombFRKmPyTfXTPXWpzoszloaRqOelCQPKkeAwEP6lp6f6V6vYaP9Z6b
3fcr7ntuhno6vPnxwGyohb2ntbpr6Y17NdKt7Bp5+YHlPJLitKyltI0KGk+XPANVgzV1Eyyqbe3s
IMmnhsMbVaYcebDraY+XVbQ15XXFOghQVnkMBkbjcVPTUs6M5IkPbnpWa1hKlrWG7QZBp0hROnUh
xRUfHTgJDWuwtsbptVTpC1xaqhgJfkuFTji9DjoKiTmpSlq/HAcO2d11d/cuXUySJG4C0+NvbfCH
CmK2hClHUop0GQ6E+dWfAeUYBi2lY2E+zgW7twhVlHDk+7AkzFO9u2lRdjORFIEZrTmAkA+GYwDl
TwLJhjZFouxsF3NzN6r7Lj7imExHUOSHWi0TpyCdORIzzwHT/J/7n+vgLVAA5YBChBUlRSCpOelR
HEZ88sA2WG3YVhb1lnJW4VVJcXFjAgM9VxOjqKTlmVJTnp48M8Ax2+3N3J3TJu6KVAbRKisxnWpr
brh+SpasxoKf14BBD91hxE2kB+nt5Px4AEL3VGoiZSAr4qyjyOP2+fAL2nuvw/e0vDl8iT8eAbv8
Lur0vPvYXrPrfrfU6bvb59Lp6NOev83HngLBwBgDAGAMAYAwBgP/2Q==
------=_Part_88180576_1156318871.1500303388932--
------=_Part_88180575_227580591.1500303388932--
9:38 p.m.
New subject: Fwd: ISCSI storage with multiple nics on same subnet disabled on host activation
On Mon, Jul 17, 2017 at 10:56 AM, Nelson Lameiras <
nelson.lameiras(a)lyra-network.com> wrote:
Hello, Can any one please help us with the problem described below?
Nir, I'm including you since a quick search on the internet led me to
think that you have worked on this part of the project. Please forgive me
if I'm off topic.
(I incorrectly used below the expression "patch" when I meant
"configure".
it's corrected now)
VDSM may indeed change the IP filter. From the function that sets it[1]:
def setRpFilterIfNeeded(netIfaceName, hostname, loose_mode):
"""
Set rp_filter to loose or strict mode if there's no session using the
netIfaceName device and it's not the device used by the OS to reach the
'hostname'.
loose mode is needed to allow multiple iSCSI connections in a multiple
NIC
per subnet configuration. strict mode is needed to avoid the security
breach where an untrusted VM can DoS the host by sending it packets with
spoofed random sources.
Arguments:
netIfaceName: the device used by the iSCSI session
target: iSCSI target object cointaining the portal hostname
loose_mode: boolean
I think it sets it to strict mode when disconnecting or removing an iSCSI
session.
Perhaps something in the check we are doing is incorrect? Do you have other
sessions open?
Y.
[1]
https://github.com/oVirt/vdsm/blob/321233bea649fb6d1e72baa1b1164c8c1bc852...
cordialement, regards,
<https://www.lyra-network.com/>
Nelson LAMEIRAS
Ingénieur Systèmes et Réseaux / Systems and Networks engineer
Tel: +33 5 32 09 09 70 <+33%205%2032%2009%2009%2070>
nelson.lameiras(a)lyra-network.com
www.lyra-network.com | www.payzen.eu <https://payzen.eu>
<https://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA>
<https://www.linkedin.com/company/lyra-network_2>
<https://twitter.com/LyraNetwork>
<https://payzen.eu>
------------------------------
Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE
------------------------------
*De: *"Nelson Lameiras" <nelson.lameiras(a)lyra-network.com>
*À: *"ovirt users" <users(a)ovirt.org>
*Envoyé: *Mercredi 7 Juin 2017 14:59:48
*Objet: *[ovirt-users] ISCSI storage with multiple nics on same subnet
disabled on host activation
Hello,
In our oVirt hosts, we are using DELL equallogic SAN with each server
connecting to SAN via 2 physical interfaces. Since both interfaces share
the same network (Equalogic limitation) we must configure sysctl to to
allow iSCSI multipath with multiple NICs in the same subnet :
------------------------------------------------------------
--------------------------------
net.ipv4.conf.p2p1.arp_ignore=1
net.ipv4.conf.p2p1.arp_announce=2
net.ipv4.conf.p2p1.rp_filter=2
net.ipv4.conf.p2p2.arp_ignore=1
net.ipv4.conf.p2p2.arp_announce=2
net.ipv4.conf.p2p2.rp_filter=2
------------------------------------------------------------
--------------------------------
This works great in most setups, but for a strange reason, on some of our
setups, the sysctl configuration is updated by VDSM when activating a host
and the second interface stops working immeadiatly :
------------------------------------------------------------
--------------------------------
vdsm.log
2017-06-07 11:51:51,063+0200 INFO (jsonrpc/5) [storage.ISCSI] Setting strict mode
rp_filter for device 'p2p2'. (iscsi:602)
2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not connect to
storageServer (hsm:2392)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer
conObj.connect()
File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect
iscsi.addIscsiNode(self._iface, self._target, self._cred)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, in
addIscsiNode
iscsiadm.node_login(iface.name, target.address, target.iqn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 337,
in node_login
raise IscsiNodeError(rc, out, err)
------------------------------------------------------------
--------------------------------
"strict mode" is enforced for second interface, and it no longuer works...
Which means - at least - that there is no redundancy in case of hardware
faillure and this is not acceptable for our production needs.
What is really strange is that we have another "twin" site on another
geographic region with simillar hardware configuration and same oVirt
installation, and this problem does not happen.
Can this be really random?
What can be the root cause of this behaviour? How can I correct it?
our setup:
oVirt hostedEngine : Centor 7.3, ovirt 4.1.2
3 physical oVirt nodes centos 7.3, ovirt 4.1.2
SAN DELL Equalogic
cordialement, regards,
<https://www.lyra-network.com/>
Nelson LAMEIRAS
Ingénieur Systèmes et Réseaux / Systems and Networks engineer
Tel: +33 5 32 09 09 70 <+33%205%2032%2009%2009%2070>
nelson.lameiras(a)lyra-network.com
www.lyra-network.com | www.payzen.eu <https://payzen.eu>
<https://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA>
<https://www.linkedin.com/company/lyra-network_2>
<https://twitter.com/LyraNetwork>
<https://payzen.eu>
------------------------------
Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
2467
days inactive
2507
days old
5 comments
3 participants
participants (3)
-
Gianluca Cecchi -
Nelson Lameiras -
Yaniv Kaul