ISCSI storage with multiple nics on same subnet disabled on host activation
------=_Part_57353689_1234561196.1496840388734 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello,=20 In our oVirt hosts, we are using DELL equallogic SAN with each server conne= cting to SAN via 2 physical interfaces. Since both interfaces share the sam= e network (Equalogic limitation) we must patch the linux kernel to to allow= iSCSI multipath with multiple NICs in the same subnet with sysctl :=20 ---------------------------------------------------------------------------= -----------------=20 net.ipv4.conf.p2p1.arp_ignore=3D1 net.ipv4.conf.p2p1.arp_announce=3D2 net.ipv4.conf.p2p1.rp_filter=3D2 net.ipv4.conf.p2p2.arp_ignore=3D1 net.ipv4.conf.p2p2.arp_announce=3D2 net.ipv4.conf.p2p2.rp_filter=3D2=20 ---------------------------------------------------------------------------= -----------------=20 This works great in most setups, but for a strange reason, on some of our s= etups, the sysctl configuration is updated by VDSM when activating a host a= nd the second interface stops working immeadiatly :=20 ---------------------------------------------------------------------------= -----------------=20 vdsm.log=20 2017-06-07 11:51:51,063+0200 INFO (jsonrpc/5) [storage.ISCSI] Setting stri= ct mode rp_filter for device 'p2p2'. (iscsi:602) 2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not conn= ect to storageServer (hsm:2392) Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer conObj.connect() File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect iscsi.addIscsiNode(self._iface, self._target, self._cred) File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, = in addIscsiNode iscsiadm.node_login(iface.name, target.address, target.iqn) File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 33= 7, in node_login raise IscsiNodeError(rc, out, err)=20 ---------------------------------------------------------------------------= -----------------=20 "strict mode" is enforced for second interface, and it no longuer works...= =20 Which means - at least - that there is no redundancy in case of hardware fa= illure and this is not acceptable for our production needs.=20 What is really strange is that we have another "twin" site on another geogr= aphic region with simillar hardware configuration and same oVirt installati= on, and this problem does not happen.=20 What can be the root cause of this behaviour? How can I correct it?=20 our setup:=20 hostedEngine : Centor 7.3, ovirt 4.1.2=20 3 physical nodes centos 7.3, ovirt 4.1.2=20 SAN DELL Equalogic=20 cordialement, regards,=20 =09Nelson LAMEIRAS=20 Ing=C3=A9nieur Syst=C3=A8mes et R=C3=A9seaux / Systems and Networks enginee= r=20 Tel: +33 5 32 09 09 70=20 nelson.lameiras@lyra-network.com=20 www.lyra-network.com | www.payzen.eu=20 =09 =09 =09 Lyra Network, 109 rue de l'innovation, 31670 Lab=C3=A8ge, FRANCE=20 ------=_Part_57353689_1234561196.1496840388734 Content-Type: multipart/related; boundary="----=_Part_57353690_157338222.1496840388734" ------=_Part_57353690_157338222.1496840388734 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: arial, helvetica, sans-serif; font-s= ize: 12pt; color: #000000"><div>Hello,<br></div><div><br data-mce-bogus=3D"= 1"></div><div>In our oVirt hosts, we are using DELL equallogic SAN with eac= h server connecting to SAN via 2 physical interfaces. Since both interfaces= share the same network (Equalogic limitation) we must patch the linux kern= el to to allow iSCSI multipath with multiple NICs in the same subnet with s= ysctl :<br data-mce-bogus=3D"1"></div><div><br data-mce-bogus=3D"1"></div><= div>-----------------------------------------------------------------------= ---------------------<br data-mce-bogus=3D"1"></div><div><pre>net.ipv4.conf= .p2p1.arp_ignore=3D1 net.ipv4.conf.p2p1.arp_announce=3D2 net.ipv4.conf.p2p1.rp_filter=3D2 net.ipv4.conf.p2p2.arp_ignore=3D1 net.ipv4.conf.p2p2.arp_announce=3D2 net.ipv4.conf.p2p2.rp_filter=3D2<br><br></pre><p>--------------------------= ------------------------------------------------------------------<br><br><= /p><p>This works great in most setups, but for a strange reason, on some of= our setups, the sysctl configuration is updated by VDSM when activating a = host and the second interface stops working immeadiatly :</p><div>---------= ---------------------------------------------------------------------------= --------</div><div>vdsm.log<br data-mce-bogus=3D"1"></div><div><pre id=3D"p= review"><code>2017-06-07 11:51:51,063+0200 INFO (jsonrpc/5) [storage.ISCSI= ] Setting strict mode rp_filter for device 'p2p2'. (iscsi:602) 2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not conn= ect to storageServer (hsm:2392) Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer conObj.connect() File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect iscsi.addIscsiNode(self._iface, self._target, self._cred) File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, = in addIscsiNode iscsiadm.node_login(iface.name, target.address, target.iqn) File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 33= 7, in node_login raise IscsiNodeError(rc, out, err)</code></pre><pre><br><br></pre><p st= yle=3D"margin: 0px;" data-mce-style=3D"margin: 0px;">----------------------= ----------------------------------------------------------------------</p><= /div></div><div><div><br data-mce-bogus=3D"1"></div><div>"strict mode" is e= nforced for second interface, and it no longuer works...</div><div>Which me= ans - at least - that there is no redundancy in case of hardware faillure a= nd this is not acceptable for our production needs.<br data-mce-bogus=3D"1"=
</div><div><br data-mce-bogus=3D"1"></div><div>What is really strange is t= hat we have another "twin" site on another geographic region with simillar = hardware configuration and same oVirt installation, and this problem does n= ot happen.</div><div><br data-mce-bogus=3D"1"></div><div>What can be the ro= ot cause of this behaviour? How can I correct it?<br data-mce-bogus=3D"1"><= /div><div><br data-mce-bogus=3D"1"></div><div>our setup:</div><div>hostedEn= gine : Centor 7.3, ovirt 4.1.2<br></div><div>3 physical nodes centos 7.3, o= virt 4.1.2<br data-mce-bogus=3D"1"></div><div>SAN DELL Equalogic<br data-mc= e-bogus=3D"1"></div></div><div><br data-mce-bogus=3D"1"></div><div data-mar= ker=3D"__SIG_PRE__"><div>cordialement, regards,</div><div><br data-mce-bogu= s=3D"1"></div><table style=3D"margin: 0px; table-layout: fixed; border: non= e; width: 345pt; padding: 0px;" data-mce-style=3D"margin: 0px; table-layout= : fixed; border: none; width: 345pt; padding: 0px;" width=3D"460" cellspaci= ng=3D"0" cellpadding=3D"0" border=3D"0"><tbody><tr><td style=3D"margin: 0; = font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height= : 16px; width: 86.25pt; text-align: left;" data-mce-style=3D"margin: 0; fon= t-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 1= 6px; width: 86.25pt; text-align: left;" width=3D"115" valign=3D"top"><a hre= f=3D"https://www.lyra-network.com/" style=3D"border: none; text-decoration:= none; width: 100%; display: block; clear: both;" data-mce-href=3D"https://= www.lyra-network.com/" data-mce-style=3D"border: none; text-decoration: non= e; width: 100%; display: block; clear: both;"><img src=3D"cid:6fa8666d679e1= 8cab5e566228d0ca3519aabec27@zimbra" alt=3D"" data-mce-src=3D"/home/admin@ly= ra-network.com/corporate-documents/element-signature_logo_lyra_115x94.jpg" = doc=3D"corporate-documents/element-signature_logo_lyra_115x94.jpg" height= =3D"94" width=3D"115" border=3D"0"></a><br data-mce-bogus=3D"1"></td><td st= yle=3D"margin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2= b3c7a; line-height: 16px; width: 258.75pt;" data-mce-style=3D"margin: 0; fo= nt-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: = 16px; width: 258.75pt;" width=3D"345" valign=3D"top"><table style=3D"margin= : 0; width: 258.75pt; table-layout: fixed; border: none;" data-mce-style=3D= "margin: 0; width: 258.75pt; table-layout: fixed; border: none;" width=3D"3= 45" cellspacing=3D"0" cellpadding=3D"0" border=3D"0"><tbody><tr><td style= =3D"font-size: 16px; font-family: Arial,sans-serif; color: #2b3c7a; line-he= ight: 18px; font-weight: bold;" data-mce-style=3D"font-size: 16px; font-fam= ily: Arial,sans-serif; color: #2b3c7a; line-height: 18px; font-weight: bold= ;">Nelson LAMEIRAS</td></tr><tr><td style=3D"font-family: Arial,sans-serif;=
color: #2b3c7a; font-size: 11px; line-height: 16px;" data-mce-style=3D"fon= t-family: Arial,sans-serif; color: #2b3c7a; font-size: 11px; line-height: 1= 6px;">Ing=C3=A9nieur Syst=C3=A8mes et R=C3=A9seaux<span style=3D"font-famil= y: Arial,sans-serif; color: #337ab7; font-size: 11px;" data-mce-style=3D"fo= nt-family: Arial,sans-serif; color: #337ab7; font-size: 11px;"> / Systems a= nd Networks engineer</span></td></tr><tr><td style=3D"line-height: 16px; fo= nt-family: 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;" data-mce-s= tyle=3D"line-height: 16px; font-family: 'Arial',sans-serif; color: #2b3c7a;= font-size: 11px;"><span style=3D"font-family: Arial,sans-serif; color: #2b= 3c7a; font-size: 11px;" data-mce-style=3D"font-family: Arial,sans-serif; co= lor: #2b3c7a; font-size: 11px;">Tel: +33 5 32 09 09 70</span></td></tr><tr>= <td style=3D"line-height: 16px; font-family: 'Arial',sans-serif; color: #2b= 3c7a; font-size: 11px;" data-mce-style=3D"line-height: 16px; font-family: '= Arial',sans-serif; color: #2b3c7a; font-size: 11px;"><a href=3D"mailto:nels= on.lameiras@lyra-network.com" style=3D"font-size: 11px; font-family: Arial,= sans-serif; line-height: 16px; text-decoration: none;" data-mce-href=3D"mai= lto:nelson.lameiras@lyra-network.com" data-mce-style=3D"font-size: 11px; fo= nt-family: Arial,sans-serif; line-height: 16px; text-decoration: none;">nel= son.lameiras@lyra-network.com</a><br data-mce-bogus=3D"1"></td></tr><tr><td= style=3D"line-height: 16px; font-family: 'Arial',sans-serif; color: #2b3c7= a; font-size: 11px;" data-mce-style=3D"line-height: 16px; font-family: 'Ari= al',sans-serif; color: #2b3c7a; font-size: 11px;"><a href=3D"https://www.ly= ra-network.com/" style=3D"font-size: 11px; font-family: Arial,sans-serif; l= ine-height: 16px; border: none; text-decoration: none;" data-mce-href=3D"ht= tps://www.lyra-network.com/" data-mce-style=3D"font-size: 11px; font-family= : Arial,sans-serif; line-height: 16px; border: none; text-decoration: none;= ">www.lyra-network.com</a> <a href=3D"https://payzen.eu" style=3D"font-size= : 11px; font-family: Arial,sans-serif; line-height: 16px; border: none; tex= t-decoration: none;" data-mce-href=3D"https://payzen.eu" data-mce-style=3D"= font-size: 11px; font-family: Arial,sans-serif; line-height: 16px; border: = none; text-decoration: none;"> | www.payzen.eu</a><br data-mce-bogus=3D"1">= </td></tr></tbody></table></td></tr></tbody></table><table style=3D"margin:= 0; table-layout: fixed; border: none; width: 345pt;" data-mce-style=3D"mar= gin: 0; table-layout: fixed; border: none; width: 345pt;" data-mce-selected= =3D"1" width=3D"460" cellspacing=3D"0" cellpadding=3D"0"><tbody><tr><td sty= le=3D"margin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b= 3c7a; line-height: 16px; width: 24pt; text-align: left;" data-mce-style=3D"= margin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; = line-height: 16px; width: 24pt; text-align: left;" width=3D"32" valign=3D"t= op"><a href=3D"https://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA" st= yle=3D"border: none; text-decoration: none; margin: 0; float: left; display= : inline-block; width: 32px;" data-mce-href=3D"https://www.youtube.com/chan= nel/UCrVl1CO_Jlu3KbiRH-tQ_vA" data-mce-style=3D"border: none; text-decorati= on: none; margin: 0; float: left; display: inline-block; width: 32px;"><img= src=3D"cid:2806a89cb9fbba6f879bd3fd54a93b515defd6d2@zimbra" alt=3D"" data-= mce-src=3D"/home/admin@lyra-network.com/corporate-documents/element-signatu= re_logo_YouTube_32x28.jpg" doc=3D"corporate-documents/element-signature_log= o_YouTube_32x28.jpg" height=3D"28" width=3D"32" border=3D"0"></a><br data-m= ce-bogus=3D"1"></td><td style=3D"margin: 0; font-size: 11px; font-family: A= rial,sans-serif; color: #2b3c7a; line-height: 16px; width: 30.75pt; text-al= ign: left;" data-mce-style=3D"margin: 0; font-size: 11px; font-family: Aria= l,sans-serif; color: #2b3c7a; line-height: 16px; width: 30.75pt; text-align= : left;" width=3D"41" valign=3D"top"><a href=3D"https://www.linkedin.com/co= mpany/lyra-network_2" style=3D"border: none; text-decoration: none; margin:= 0; float: left; display: inline-block; width: 41px;" data-mce-href=3D"http= s://www.linkedin.com/company/lyra-network_2" data-mce-style=3D"border: none= ; text-decoration: none; margin: 0; float: left; display: inline-block; wid= th: 41px;"><img src=3D"cid:878634ba055bb6ae851a7ef4540f754afa3ff86c@zimbra"= data-mce-src=3D"/home/admin@lyra-network.com/corporate-documents/element-s= ignature_logo_LinkedIn_41x28.jpg" doc=3D"corporate-documents/element-signat= ure_logo_LinkedIn_41x28.jpg" height=3D"28" width=3D"41" border=3D"0"></a><b= r data-mce-bogus=3D"1"></td><td style=3D"margin: 0; font-size: 11px; font-f= amily: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 31.5pt; = text-align: left;" data-mce-style=3D"margin: 0; font-size: 11px; font-famil= y: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 31.5pt; text= -align: left;" width=3D"42" valign=3D"top"><a href=3D"https://twitter.com/L= yraNetwork" style=3D"border: none; text-decoration: none; margin: 0; float:= left; display: inline-block; width: 42px;" data-mce-href=3D"https://twitte= r.com/LyraNetwork" data-mce-style=3D"border: none; text-decoration: none; m= argin: 0; float: left; display: inline-block; width: 42px;"><img src=3D"cid= :18a767bbdad6a1133fc4152b03ea78228ce25a14@zimbra" alt=3D"" data-mce-src=3D"= /home/admin@lyra-network.com/corporate-documents/element-signature_logo_Twi= tter_42x28.jpg" doc=3D"corporate-documents/element-signature_logo_Twitter_4= 2x28.jpg" height=3D"28" width=3D"42" border=3D"0"></a><br data-mce-bogus=3D= "1"></td><td style=3D"margin: 0; font-size: 11px; font-family: Arial,sans-s= erif; color: #2b3c7a; line-height: 16px; width: 258.75pt; text-align: left;= " data-mce-style=3D"margin: 0; font-size: 11px; font-family: Arial,sans-ser= if; color: #2b3c7a; line-height: 16px; width: 258.75pt; text-align: left;" = width=3D"345" valign=3D"top"><a href=3D"https://payzen.eu" style=3D"border:= none; text-decoration: none; margin: 0; float: left; display: inline-block= ; width: 61px;" data-mce-href=3D"https://payzen.eu" data-mce-style=3D"borde= r: none; text-decoration: none; margin: 0; float: left; display: inline-blo= ck; width: 61px;"><img src=3D"cid:d3fc119a0aa72bf985c19ca559c993b5b4b54ca3@= zimbra" alt=3D"" data-mce-src=3D"/home/admin@lyra-network.com/corporate-doc= uments/element-signature_payzen_61x28.jpg" doc=3D"corporate-documents/eleme= nt-signature_payzen_61x28.jpg" height=3D"28" width=3D"61" border=3D"0"></a>= <br data-mce-bogus=3D"1"></td></tr></tbody></table><table style=3D"margin: = 0; table-layout: fixed; border: none; width: 345pt;" data-mce-style=3D"marg= in: 0; table-layout: fixed; border: none; width: 345pt;" width=3D"460" cell= spacing=3D"0" cellpadding=3D"0"><tbody><tr><td><hr style=3D"background: #29= 3b79 none repeat scroll 0% 0%; width: 100%; height: 4px; color: #293b79; ma= rgin: 0px;" data-mce-style=3D"background: #293b79 none repeat scroll 0% 0%;= width: 100%; height: 4px; color: #293b79; margin: 0px;"></td></tr></tbody>= </table><table style=3D"margin: 0; table-layout: fixed; border: none; width= : 345pt;" data-mce-style=3D"margin: 0; table-layout: fixed; border: none; w= idth: 345pt;" data-mce-selected=3D"1" width=3D"460" cellspacing=3D"0" cellp= adding=3D"0"><tbody><tr><td style=3D"margin: 0; font-size: 11px; font-famil= y: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 30.75pt; tex= t-align: left;" data-mce-style=3D"margin: 0; font-size: 11px; font-family: = Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 30.75pt; text-a= lign: left;"><div>Lyra Network, 109 rue de l'innovation, 31670 Lab=C3=A8ge,= FRANCE</div></td></tr></tbody></table><div><br></div></div></div></body></= html> ------=_Part_57353690_157338222.1496840388734 Content-Type: image/jpeg; name=element-signature_logo_lyra_115x94.jpg Content-Disposition: attachment; filename=element-signature_logo_lyra_115x94.jpg Content-Transfer-Encoding: base64 Content-ID: <6fa8666d679e18cab5e566228d0ca3519aabec27@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOkU1QUU4Nzk1OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOkU1QUU4Nzk0OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIAF4AcwMBEQACEQEDEQH/xACdAAEBAAIDAQEAAAAAAAAAAAAACAUH AgMGBAEBAQADAQEAAAAAAAAAAAAAAAACBAUBAxAAAAUDAgAHDQcCBwAAAAAAAAECAwQRBQYSByFR E7RWFzcxQSKS0nOT03RVlQgYYXEyQiMUFYEzsVJicmODOBEAAgECAwcEAgIDAAAAAAAAAAECEQNR EwQhgbESMlIzMUGhFHEVwQUiQzT/2gAMAwEAAhEDEQA/AKpAE9b/AGQZdF3FtNpsd4l29M2DHSll iS6w0bzsp5slqJCiLvERnTuENTRQg7bckntK15vmojj1WfMP0rc+Jy/JD7Fjt+EMueJ6bbnBN4bR lkWdkt+XPtDaHSfjKnSHyUpTZkg+TcLSdFGRjyv3rUo0iqP8EoQkntZ7fcOy5VdrXGYxuccGW2+S 3nCecYq3oUWnU2RmfhGXAPPSXLcJNzVVT8kNVbnKKUHRmppW1/zBOSXVtZS4lpSjNtP8nLKhfcSR alqdPXp+ERhauKKTe38nX1WfMP0qc+Jy/JHPsWO34RPLniOqz5h+lTnxOX5IfYsdvwhlzxHVZ8w/ Spz4nL8kPsWO34Qy54jqs+YfpU58Tl+SH2LHb8IZc8R1WfMP0qc+Jy/JD7Fjt+EMueJ2bCX3MH9y LxZr7eJdxKBCkoW0/JdfaJ5mUy2a0EszL/MRHTuGGshDLTiqVf8AAst81GUGMssgAAAAE5b89s+L +zwefPDV0fhlv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACc t+e2fF/Z4PPnhq6Pwy38Crd60UaMotAAAAAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLI AAAABOW/PbPi/s8Hnzw1dH4Zb+BVu9aKNGUWjAKzrGk5cnEjkq/nVo5RMbk3NOnkzd/uU0fgKvdH rky5eb2I86rQz48iQAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLIAAAABOW/PbPi/s8Hn zw1dH4Zb+BVu9aKNGUWjSD//AKkj+xnzJQ0V/wA2/wDkr/7D3e7GX5DiOMfzlniMTEMPIRORIJfg tOeCladCk/nNJH94raa1GcuVnpck4qqOrJdz4dr2yazKOhDjkxhlUGMszop98i/TOlD/AE/CNX+0 x23p27nIJXKRqeeyjdnLMd27sGSzLdEO4Xd4uWifqkhDDiFut0qrUSzQlNa9wz7g9bemjK44puiI yuNRTOzGs53hud+gvzsYjwMVnqNfLuqUTrEck69bitZmStPc1NJIz4hy5ZtKLpKskIzk36bDHRt2 9yMsuMw9v8ejybNBc5M5s5Rp5Uy4aEZuMJSai4dPCZFStBN6a3BLne1nMyT6Uffgu8GQ5DnysXuN mbtZx461TG1ms3kyGiLWRHXToMz8Hg7nfMQvaWMYcydTsLjcqG2BSPYnbY/twy3zVw5+yNXV+GO7 gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLROmdZbAxL5g/5yey6/GjREJW2wSTc M3YpoKms0F3VcY1bNpzscqKs5UnU9pad4cG3Fefw1UWXFO7x3mULkpaJJnoM6JNC10WREak/aXGK 8tLO1/ns2HorilsNTYzar/kOQWTa+5tmmBjk+W9cKGqimkrJS68RcCkoV/yC7clGMXcXrJI8Yptq OBsj5oEpTiNmSRESSuKSJJFwUJlzgoKv9f1P8Hrf9Db1wh/vbVJh6tH7lhbOvi5RBpr/AEqKMXR1 PZrYaC2x3Jgbawp+IZjBkwpkWS480603rSslkRU7qTOuiqFlUlEfepw6WosO61ODK9ufLsZ27f36 RkG/8q9OwXbe3PgKciMPp0uHHS22hpxRf60o1f4VC9DlsUrWjEHWdSghllknbY/twy3zVw5+yNXV +GO7gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLRxU00o6qQlR8ZkRhUGMyO7W7H 7FOvUpCSZgMrfMqERqNJeCgj41qokvtMTtxcpJL3OSdFU1n8vFimPwbrnF18O55DIXybhlw8ihZm s08RLdqVOJJC5rZqqgvSJ42V7v3MpM3dTBnyWbtBQmHFnXFpLzdXOWjW9tWo2irwPpd0pWhVPBPU R0qILTVWx+y+STuYmQj7sxHH1Kctb7NuahnMflKW2akk3J/bvGlojqtpv8ZuJP8ACR8Ag9M8dtTu YdErdfEJTCZKrY/L5JSySbrbCdKmicdWRLeWlJK/bs8skq+ElSacJ8HVppr3OZiP2RuxFK4W5xi3 LK2T5D8NmY8baHHnWySSeTSa6to1K8JTumn3gtM6PbtQzDYYqnqTtsf24Zb5q4c/ZGrq/DHdwK1r rZRIyiyAAAAATlvz2z4v7PB588NXR+GW/gVbvWijRlFoADGZHjVmyO2Ltd4YOTAcUlbjJLcbJRoO qam2pCuA+GlROFxwdV6nJRT2M+q226DbLfHt8BkmIcRtLMdlNaJQgqEVTqZ/eYjKTbqwlQ5Hb4Bp Uk4zRpUpa1FoTQ1uEaVqPg7qiUZGffDmYodSbNZ0toaTBjk02hbTaCaQSUtu/wBxCSpQkr/MXfHe d4iiP12z2h5k2XYMdxlSkuKbW0hSTW2kkIUZGVKpSkiI+8RApPEURydtltdQpDsRlxCtepKm0mR8 qdXKkZfn/NxjnMxQ+kcOk7bH9uGW+auHP2Rq6vwx3cCta62USMosgAAAAE5b89s+L+zwefPDV0fh lv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACb9/3mWN4cbff WTTLUWEt1xX4UpTNeNSj+wiGtolW1LfwKt7qRuTrY226RwfSkKH1rnaz3zI4jrZ226RwfSkH1rna xmRxHWztt0jg+lIPrXO1jMjiOtnbbpHB9KQfWudrGZHEdbO23SOD6Ug+tc7WMyOJ+o3W24WtKE5F CUpR0SknCqZmH1rnazjuxW2pnbTfbPeGnHbXMamNtK0OLaVqJKjKtDELlqUNklQ7C5GfS6mOu+f4 XZpy4F0vMWHMbJKlsOrJKyJRVTUvtIdjZnJVSOuaXqfF1sbbdI4PpSEvrXO1nMyOJp7YiQxJ3oyi RHcS9Hejz3GXUcKVoVOZNKi+wyOovaxUsx3cDwtdbKNGUWgAAAADX+4GzGPZveWbtcpsuO+xGTFS iObRINCFrcIz1oWdaun3xas6qVtUSR5ztKTqeZ+lrCvely8Zj1Q9f2M8EQyEPpawr3pcvGY9UH7G eCGQh9LWFe9Ll4zHqg/YzwQyEPpawr3pcvGY9UH7GeCGQh9LWFe9Ll4zHqg/YzwQyEc2flfwxl5t 1NzuJqbUSiI1MUqX/UOr+xmn6I5LTRaaxNg4XhNuxOHIiwX3n0SXCdWb5pMyMk6eDSlPEPDU6mV5 pteg0+nVpNI8zm+xeNZfkDt7nzprEl5Dbam2FNEgibTpKmttR97jErWslCNEkTlaUnUwP0tYV70u XjMeqHp+xngiOQj023+zGPYReXrtbZsuQ+/GVFUiQbRoJC1ocMy0IQdatF3x5XtVK4qNInC0oups AVT0AAAAAAAAAAAAAAAAAAAAAAAAAAAAD//Z ------=_Part_57353690_157338222.1496840388734 Content-Type: image/jpeg; name=element-signature_logo_YouTube_32x28.jpg Content-Disposition: attachment; filename=element-signature_logo_YouTube_32x28.jpg Content-Transfer-Encoding: base64 Content-ID: <2806a89cb9fbba6f879bd3fd54a93b515defd6d2@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOjQ1REY5Njk1OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOjQ1REY5Njk0OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAIAMBEQACEQEDEQH/xAB5AAADAQEAAAAAAAAAAAAAAAAABwgF BgEAAgMBAAAAAAAAAAAAAAAAAAQCAwUBEAABBAECBAUFAAAAAAAAAAABAgMEBQARBiESEwcxUdIU CEEik1UYEQACAQMCBQUAAAAAAAAAAAAAAQIRAwQhMVFhEiITQaHBYhX/2gAMAwEAAhEDEQA/AKpw AnD5Hbg3JA35XxKqxmxUPVrJEaI862FuqkPp4IbI5lHQDNXBhFwbaW4tebqcMIvfYjURtz6eesv1 YzWz9fYh38zR7eX2+mu5lHV3VhaNOe8aTJgTHnweVXEBba1cQRkb8IeNtJbHYN9WpW+YY2T53inQ 4HfzZk6a4lmHGaiOSHl8EoT7mSOZR+gBIzUxU3Yklz+Be4+9DRl30N3dMOxjbyro9EzHW3MqupHc L7qlaoX1SsFvlHl44moPpa6XXiW113FPuu3qrT5I7dfrJLUxllMRl15hQWjqBbiuXmTqCQlQx23F rHdSqTrNFFZlDAou7vZa43xuSNaw7CPFZZhIiKbeSsqKkOuuE/aCNNHcexspW40a9Sm5bcmcH/J1 5+0rvxOenGf0o8GQ8DN7ZXxzv9vbpqrhy0huMQJCHlstocBKU+ITqANcru50ZRao9SUbLTqPvMwv DAAwAMAP/9k= ------=_Part_57353690_157338222.1496840388734 Content-Type: image/jpeg; name=element-signature_logo_LinkedIn_41x28.jpg Content-Disposition: attachment; filename=element-signature_logo_LinkedIn_41x28.jpg Content-Transfer-Encoding: base64 Content-ID: <878634ba055bb6ae851a7ef4540f754afa3ff86c@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOjY2MEYzRTFGOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOjY2MEYzRTFFOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAKQMBEQACEQEDEQH/xAB2AAEBAQEAAAAAAAAAAAAAAAAABggH AQEAAwEAAAAAAAAAAAAAAAAAAwQFAhAAAQMEAgICAgMBAAAAAAAAAQIDBAARBQYhEjEHMghBE1Fh IhcRAAICAQIDCQAAAAAAAAAAAAABAgMREgQxkTIhQVGhscETMxX/2gAMAwEAAhEDEQA/ANU0Bl33 zsuzQfZcmHj8tMiMKaihDLMl1psKcQkXslSUjk8mtjZ1xdeWkVbW9RDZ/ZN8wOXkYnI7LKMuL1Lq o+RddaIWkLHVwLAPCuf4NWYVwksqPkcNtd5V+ldr2ef7MwkaZmJsqK6p/uy9JdcbUBGcULpUog8i 9Q7uuKreEjquT1GsKxC2KAUBk37B2/6y72t1tB7X8dbIvf8Aq3mtvZfVzKlvUWWw7Dr+Dfy8rCt4 d153Z8ew0FNx3kohuxmUyFtAfFPyBUOBz+arwhKWE89L9SRtLmTuns4xj7LLaxYaGNE6SYojlJZ6 rhLUf19f89eyj4qa1t7ft449ziPWairGLQoBQHFvZ/onObht0jNxclFjMPNtNpZdS4Vj9aAk36gj mtDb7yMIaWiGdTbySSfqnsKbdctAFh1Fm3Bwfx8fFT/ox8GcfAyk9dfX7OapuGNzj+SiPRoJcKmG kuBRDjS2x1uAPK71DfvYzg444nUKmnk7lWcTigFAKAUAoBQCgP/Z ------=_Part_57353690_157338222.1496840388734 Content-Type: image/jpeg; name=element-signature_logo_Twitter_42x28.jpg Content-Disposition: attachment; filename=element-signature_logo_Twitter_42x28.jpg Content-Transfer-Encoding: base64 Content-ID: <18a767bbdad6a1133fc4152b03ea78228ce25a14@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOjdDMkI3MjMyOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOjdDMkI3MjMxOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAKgMBEQACEQEDEQH/xAB3AAEBAQEBAAAAAAAAAAAAAAAABwUI BgEBAAMBAQAAAAAAAAAAAAAAAAMEBQIBEAABAwIEBwEBAQAAAAAAAAABAgMEAAUREhMGITFBFBUH CIEiFhEAAgIBAgQHAAAAAAAAAAAAAAERAgMxBCHBEgVBUWGBQhMV/9oADAMBAAIRAxEAPwDC9g7u 3dH33uJiNeri2wxcJQQ01JfCUNodVwCUqwSlI/AK3sGKropS0Kd7OWZMXcPsyZDdnQ7he5UFhQQ/ LZelOMoURiEqWlRAOFdumNOGqyeTYtPzFfL1df8ASm5XCTP0ey0e5eceyZu4zZc5VhjlGOFZ/cKJ dMKNeRNhbclzrOJxQCgOSW7o5a/oKXMRFXNT5iYw/EaAUtbL61tOBKTwXglWOXrW30zgj0RUmLlb 9p3rZ/r/ANd3GxWJtm3XC8B1EO2RsEqDkng88pAP8JSnEk8ugqlt6Xy5Fa3FImyNVUI8z8nJCU7o SOSRAA/O5qbuXx9+Rxg8ToGsssCgFAc/7q+b9xXrc11u7V3iNN3CW/JbQUuhaUuuFYBIHMA9K08e +rWqUPgivbC25MofK25Qoq81DKjwKlJeUT+kE13+jXyZ59DKX6a9WXPYfmO+msTPJdtp6AWMuhq4 5swHPVFVN1uFkiFoS46dJSqqEgoBQCgFAKAUAoBQH//Z ------=_Part_57353690_157338222.1496840388734 Content-Type: image/jpeg; name=element-signature_payzen_61x28.jpg Content-Disposition: attachment; filename=element-signature_payzen_61x28.jpg Content-Transfer-Encoding: base64 Content-ID: <d3fc119a0aa72bf985c19ca559c993b5b4b54ca3@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOkU1NjczQjM2OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOkU1NjczQjM1OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAPQMBEQACEQEDEQH/xABvAAABBAMAAAAAAAAAAAAAAAAAAQUG BwIDBAEBAAAAAAAAAAAAAAAAAAAAABAAAQQABAQEAgsBAAAAAAAAAQIDBAUAERIGITETFEEiFQfS I1FhcYGhMlIkRMQWhhEBAAAAAAAAAAAAAAAAAAAAAP/aAAwDAQACEQMRAD8AvqZ7q7OhzZMSS9IQ Yjy2JT/avqZbW2rSvU6lBSACOeeAljLzTzSHmVhxpxIW24k5pUlQzBBHgRgM8BBJ9fKuvcGwr3La wgxIlfGeaagyVsJ1uOOBRUBmOSRgI0bnZgfQwdx7qDrroYZSVzBrcJ0hKM2xq+7AWBU7PTXTEShc 2svRx6MuWp5o5gjigj68BombFRKmPyTfXTPXWpzoszloaRqOelCQPKkeAwEP6lp6f6V6vYaP9Z6b 3fcr7ntuhno6vPnxwGyohb2ntbpr6Y17NdKt7Bp5+YHlPJLitKyltI0KGk+XPANVgzV1Eyyqbe3s IMmnhsMbVaYcebDraY+XVbQ15XXFOghQVnkMBkbjcVPTUs6M5IkPbnpWa1hKlrWG7QZBp0hROnUh xRUfHTgJDWuwtsbptVTpC1xaqhgJfkuFTji9DjoKiTmpSlq/HAcO2d11d/cuXUySJG4C0+NvbfCH CmK2hClHUop0GQ6E+dWfAeUYBi2lY2E+zgW7twhVlHDk+7AkzFO9u2lRdjORFIEZrTmAkA+GYwDl TwLJhjZFouxsF3NzN6r7Lj7imExHUOSHWi0TpyCdORIzzwHT/J/7n+vgLVAA5YBChBUlRSCpOelR HEZ88sA2WG3YVhb1lnJW4VVJcXFjAgM9VxOjqKTlmVJTnp48M8Ax2+3N3J3TJu6KVAbRKisxnWpr brh+SpasxoKf14BBD91hxE2kB+nt5Px4AEL3VGoiZSAr4qyjyOP2+fAL2nuvw/e0vDl8iT8eAbv8 Lur0vPvYXrPrfrfU6bvb59Lp6NOev83HngLBwBgDAGAMAYAwBgP/2Q== ------=_Part_57353690_157338222.1496840388734-- ------=_Part_57353689_1234561196.1496840388734--
On Wed, Jun 7, 2017 at 2:59 PM, Nelson Lameiras < nelson.lameiras@lyra-network.com> wrote:
Hello,
In our oVirt hosts, we are using DELL equallogic SAN with each server connecting to SAN via 2 physical interfaces. Since both interfaces share the same network (Equalogic limitation) we must patch the linux kernel to to allow iSCSI multipath with multiple NICs in the same subnet with sysctl :
------------------------------------------------------------ --------------------------------
net.ipv4.conf.p2p1.arp_ignore=1 net.ipv4.conf.p2p1.arp_announce=2 net.ipv4.conf.p2p1.rp_filter=2
net.ipv4.conf.p2p2.arp_ignore=1 net.ipv4.conf.p2p2.arp_announce=2 net.ipv4.conf.p2p2.rp_filter=2
Hello Nelson, nice to see another environment based on EQL iSCSI SAN. The systctl.conf customized lines above are not what you mean with "we must patch the linux kernel", is it? What do you mean exactly?
------------------------------------------------------------ --------------------------------
This works great in most setups, but for a strange reason, on some of our setups, the sysctl configuration is updated by VDSM when activating a host and the second interface stops working immeadiatly : ------------------------------------------------------------ -------------------------------- vdsm.log
2017-06-07 11:51:51,063+0200 INFO (jsonrpc/5) [storage.ISCSI] Setting strict mode rp_filter for device 'p2p2'. (iscsi:602) 2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not connect to storageServer (hsm:2392) Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer conObj.connect() File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect iscsi.addIscsiNode(self._iface, self._target, self._cred) File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, in addIscsiNode iscsiadm.node_login(iface.name, target.address, target.iqn) File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 337, in node_login raise IscsiNodeError(rc, out, err)
In my environment I don't see these kinds of problems. I'm using plain CentOS 7.3 hosts as hypervisors; is this your case too or are you using ovirt-ng nodes?
------------------------------------------------------------ --------------------------------
"strict mode" is enforced for second interface, and it no longuer works... Which means - at least - that there is no redundancy in case of hardware faillure and this is not acceptable for our production needs.
What is really strange is that we have another "twin" site on another geographic region with simillar hardware configuration and same oVirt installation, and this problem does not happen.
What can be the root cause of this behaviour? How can I correct it?
Actually it is very strange to me to understand how you don't have problems on the other side. My workaround to get multipath with EQL in oVirt was the one explained here with screenshots: http://lists.ovirt.org/pipermail/users/2017-March/080992.html It has been working now for about 3 months without problems. We have several Oracle RDBMS server VMs on it, from 2virtual threads and 8Gb of ram to 16 threads (2-4-2) and 64Gb of ram How did you configure multipath instead? As far as I know oVirt will not let you do it. The only way would be to use bonding but that is not certified/supported by Dell. HIH sharing experiences, Gianluca
<td style=3D"margin: 0; font-size: 11px; font-family: Arial,sans-serif; co= lor: #2b3c7a; line-height: 16px; width: 31.5pt; text-align: left;" data-mce= -style=3D"margin: 0; font-size: 11px; font-family: Arial,sans-serif; color:= #2b3c7a; line-height: 16px; width: 31.5pt; text-align: left;" width=3D"42"= valign=3D"top"><a href=3D"https://twitter.com/LyraNetwork" style=3D"border= : none; text-decoration: none; margin: 0; float: left; display: inline-bloc= k; width: 42px;" data-mce-href=3D"https://twitter.com/LyraNetwork" data-mce= -style=3D"border: none; text-decoration: none; margin: 0; float: left; disp= lay: inline-block; width: 42px;"><img src=3D"cid:5f6dbbaff6dac1a2e25c3c96f0= 2d8020be9c5268@zimbra" alt=3D"" data-mce-src=3D"/home/admin@lyra-network.co= m/corporate-documents/element-signature_logo_Twitter_42x28.jpg" doc=3D"corp= orate-documents/element-signature_logo_Twitter_42x28.jpg" height=3D"28" wid=
Sent: </b>Thursday, June 8, 2017 9:36:33 AM<br><b>Subject: </b>Re: [ovirt-= users] ISCSI storage with multiple nics on same subnet disabled on host act= ivation<br></div><div><br></div><div data-marker=3D"__QUOTED_TEXT__"><div d= ir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote"><blockquot= e>On Wed, Jun 7, 2017 at 2:59 PM, Nelson Lameiras <span dir=3D"ltr"><<a =
<div><blockquote><p>------------------------------------------------------= --------------------------------------<br><br></p></blockquote></div></div>= </div><blockquote><div><div style=3D"font-family:arial,helvetica,sans-serif= ;font-size:12pt;color:rgb(0,0,0)"><div><p>This works great in most setups, = but for a strange reason, on some of our setups, the sysctl configuration i= s updated by VDSM when activating a host and the second interface stops wor= king immeadiatly :</p><div>------------------------------------------------= --------------------------------------------</div><div>vdsm.log<br></div><d= iv><pre id=3D"gmail-m_3172160333806361248preview"><code>2017-06-07 11:51:51= ,063+0200 INFO (jsonrpc/5) [storage.ISCSI] Setting strict mode rp_filter f= or device 'p2p2'. (iscsi:602) 2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not conn= ect to storageServer (hsm:2392) Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer conObj.connect() File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect iscsi.addIscsiNode(self._iface, self._target, self._cred) File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, = in addIscsiNode iscsiadm.node_login(<a href=3D"http://iface.name" target=3D"_blank">ifa= ce.name</a>, target.address, target.iqn) File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 33= 7, in node_login raise IscsiNodeError(rc, out, err)</code></pre><pre><br></pre></div></d= iv></div></div><div>In my environment I don't see these kinds of problems.<= /div><div>I'm using plain CentOS 7.3 hosts as hypervisors; is this your cas= e too or are you using ovirt-ng nodes?</div><div><br data-mce-bogus=3D"1"><= /div></blockquote><div>Plain CentOS 7.3 hosts also. Built from a Minimal ce= ntos and adding oVirt repositories.<br></div><div><br data-mce-bogus=3D"1">= </div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;b= order-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style=3D"= font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><di= v><div><p style=3D"margin:0px">--------------------------------------------= ------------------------------------------------</p></div></div><div><br><d= iv>"strict mode" is enforced for second interface, and it no longuer works.= ..</div><div>Which means - at least - that there is no redundancy in case o= f hardware faillure and this is not acceptable for our production needs.<br= </div><br><div>What is really strange is that we have another "twin" site = on another geographic region with simillar hardware configuration and same = oVirt installation, and this problem does not happen.</div><br><div>What ca= n be the root cause of this behaviour? How can I correct it?<br></div><br><= br></div></div></div></blockquote><blockquote><div>Actually it is very stra= nge to me to understand how you don't have problems on the other side.</div= <div>My workaround to get multipath with EQL in oVirt was the one explaine= d here with screenshots:</div><div><a href=3D"http://lists.ovirt.org/piperm= ail/users/2017-March/080992.html" target=3D"_blank">http://lists.ovirt.org/=
------=_Part_57900106_686621279.1496910345349 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Gianluca,=20 My answers below,=20 cordialement, regards,=20 =09Nelson LAMEIRAS=20 Ing=C3=A9nieur Syst=C3=A8mes et R=C3=A9seaux / Systems and Networks enginee= r=20 Tel: +33 5 32 09 09 70=20 nelson.lameiras@lyra-network.com=20 www.lyra-network.com | www.payzen.eu=20 =09 =09 =09 Lyra Network, 109 rue de l'innovation, 31670 Lab=C3=A8ge, FRANCE=20 From: "Gianluca Cecchi" <gianluca.cecchi@gmail.com>=20 To: "Nelson Lameiras" <nelson.lameiras@lyra-network.com>=20 Cc: "ovirt users" <users@ovirt.org>=20 Sent: Thursday, June 8, 2017 9:36:33 AM=20 Subject: Re: [ovirt-users] ISCSI storage with multiple nics on same subnet = disabled on host activation=20 On Wed, Jun 7, 2017 at 2:59 PM, Nelson Lameiras < nelson.lameiras@lyra-netw= ork.com > wrote:=20 BQ_BEGIN Hello,=20 In our oVirt hosts, we are using DELL equallogic SAN with each server conne= cting to SAN via 2 physical interfaces. Since both interfaces share the sam= e network (Equalogic limitation) we must patch the linux kernel to to allow= iSCSI multipath with multiple NICs in the same subnet with sysctl :=20 ---------------------------------------------------------------------------= -----------------=20 net.ipv4.conf.p2p1.arp_ignore=3D1 net.ipv4.conf.p2p1.arp_announce=3D2 net.ipv4.conf.p2p1.rp_filter=3D2 net.ipv4.conf.p2p2.arp_ignore=3D1 net.ipv4.conf.p2p2.arp_announce=3D2 net.ipv4.conf.p2p2.rp_filter=3D2=20 BQ_END BQ_BEGIN Hello Nelson,=20 nice to see another environment based on EQL iSCSI SAN.=20 The systctl.conf customized lines above are not what you mean with "we must= patch the linux kernel", is it?=20 What do you mean exactly?=20 BQ_END Sorry, by "patch" I meant "sysctl.conf configuration" indeed.=20 This configuration is what we use to be able to configure 2 network interfa= ces in the same subnet.=20 BQ_BEGIN ---------------------------------------------------------------------------= -----------------=20 BQ_END BQ_BEGIN This works great in most setups, but for a strange reason, on some of our s= etups, the sysctl configuration is updated by VDSM when activating a host a= nd the second interface stops working immeadiatly :=20 ---------------------------------------------------------------------------= -----------------=20 vdsm.log=20 2017-06-07 11:51:51,063+0200 INFO (jsonrpc/5) [storage.ISCSI] Setting stri= ct mode rp_filter for device 'p2p2'. (iscsi:602) 2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not conn= ect to storageServer (hsm:2392) Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer conObj.connect() File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect iscsi.addIscsiNode(self._iface, self._target, self._cred) File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, = in addIscsiNode iscsiadm.node_login( iface.name , target.address, target.iqn) File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 33= 7, in node_login raise IscsiNodeError(rc, out, err)=20 In my environment I don't see these kinds of problems.=20 I'm using plain CentOS 7.3 hosts as hypervisors; is this your case too or a= re you using ovirt-ng nodes?=20 BQ_END Plain CentOS 7.3 hosts also. Built from a Minimal centos and adding oVirt r= epositories.=20 BQ_BEGIN ---------------------------------------------------------------------------= -----------------=20 "strict mode" is enforced for second interface, and it no longuer works...= =20 Which means - at least - that there is no redundancy in case of hardware fa= illure and this is not acceptable for our production needs.=20 What is really strange is that we have another "twin" site on another geogr= aphic region with simillar hardware configuration and same oVirt installati= on, and this problem does not happen.=20 What can be the root cause of this behaviour? How can I correct it?=20 BQ_END BQ_BEGIN Actually it is very strange to me to understand how you don't have problems= on the other side.=20 My workaround to get multipath with EQL in oVirt was the one explained here= with screenshots:=20 http://lists.ovirt.org/pipermail/users/2017-March/080992.html=20 It has been working now for about 3 months without problems. We have severa= l Oracle RDBMS server VMs on it, from 2virtual threads and 8Gb of ram to 16= threads (2-4-2) and 64Gb of ram=20 How did you configure multipath instead? As far as I know oVirt will not le= t you do it.=20 The only way would be to use bonding but that is not certified/supported by= Dell.=20 HIH sharing experiences,=20 Gianluca=20 BQ_END I've read the link you gave and it seems to me that you are using the same = physical interface for SAN and vm network.=20 In my setup :=20 - SAN network is using two disctinct physical interfaces (p2pX) on same sub= net (therefore the need for sysctl patch)=20 - VM network (p1pX) is using 2 other distinct physical interfaces on "trunk= " mode=20 - Administration network (emX) is using 2 distinct bonded network interface= s=20 Our setup, using a total of 6 network interfaces is "simpler" than yours si= nce each interface has a "disctinct" role.=20 To make multipath work with oVirt on this setup, we had to configure SAN ne= twork on system before installing oVirt (using sysctl.conf above),=20 - next declare 2 SAN networks on oVirt that are "dragued" to each SAN inter= face in host network configuration,=20 - and finally, in datacenter listing, chosing a datacenter and selecting "i= SCSI Multipathing" tab below I can add a multipath configuration concerning= both SAN oVirt networks and relevant iqn storage targets.=20 Hope it's clear, if not, please ask me.=20 It's very strange to me also since this configuration did work great on our= labs. But half of production sites have the problem decribed above : secon= d SAN interface stops working uppon host activation because vdsm forces sys= ctl net.ipv4.conf.p2p2.rp_filter to 1 (strict mode).=20 Still searching for clues as why this behaviour...=20 ------=_Part_57900106_686621279.1496910345349 Content-Type: multipart/related; boundary="----=_Part_57900107_1954785857.1496910345350" ------=_Part_57900107_1954785857.1496910345350 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: arial, helvetica, sans-serif; font-s= ize: 12pt; color: #000000"><div>Hi Gianluca,<br></div><div><br data-mce-bog= us=3D"1"></div><div>My answers below,<br data-mce-bogus=3D"1"></div><div><b= r></div><div data-marker=3D"__SIG_PRE__"><div>cordialement, regards,</div><= div><br data-mce-bogus=3D"1"></div><table style=3D"margin: 0px; table-layou= t: fixed; border: none; width: 345pt; padding: 0px;" data-mce-style=3D"marg= in: 0px; table-layout: fixed; border: none; width: 345pt; padding: 0px;" da= ta-mce-selected=3D"1" width=3D"460" cellspacing=3D"0" cellpadding=3D"0" bor= der=3D"0"><tbody><tr><td style=3D"margin: 0; font-size: 11px; font-family: = Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 86.25pt; text-a= lign: left;" data-mce-style=3D"margin: 0; font-size: 11px; font-family: Ari= al,sans-serif; color: #2b3c7a; line-height: 16px; width: 86.25pt; text-alig= n: left;" width=3D"115" valign=3D"top"><a href=3D"https://www.lyra-network.= com/" style=3D"border: none; text-decoration: none; width: 100%; display: b= lock; clear: both;" data-mce-href=3D"https://www.lyra-network.com/" data-mc= e-style=3D"border: none; text-decoration: none; width: 100%; display: block= ; clear: both;"><img src=3D"cid:25cd0e595e0f025605ddb995496484dcb70629d9@zi= mbra" alt=3D"" data-mce-src=3D"/home/admin@lyra-network.com/corporate-docum= ents/element-signature_logo_lyra_115x94.jpg" doc=3D"corporate-documents/ele= ment-signature_logo_lyra_115x94.jpg" height=3D"94" width=3D"115" border=3D"= 0"></a><br data-mce-bogus=3D"1"></td><td style=3D"margin: 0; font-size: 11p= x; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width:= 258.75pt;" data-mce-style=3D"margin: 0; font-size: 11px; font-family: Aria= l,sans-serif; color: #2b3c7a; line-height: 16px; width: 258.75pt;" width=3D= "345" valign=3D"top"><table style=3D"margin: 0; width: 258.75pt; table-layo= ut: fixed; border: none;" data-mce-style=3D"margin: 0; width: 258.75pt; tab= le-layout: fixed; border: none;" width=3D"345" cellspacing=3D"0" cellpaddin= g=3D"0" border=3D"0"><tbody><tr><td style=3D"font-size: 16px; font-family: = Arial,sans-serif; color: #2b3c7a; line-height: 18px; font-weight: bold;" da= ta-mce-style=3D"font-size: 16px; font-family: Arial,sans-serif; color: #2b3= c7a; line-height: 18px; font-weight: bold;">Nelson LAMEIRAS</td></tr><tr><t= d style=3D"font-family: Arial,sans-serif; color: #2b3c7a; font-size: 11px; = line-height: 16px;" data-mce-style=3D"font-family: Arial,sans-serif; color:= #2b3c7a; font-size: 11px; line-height: 16px;">Ing=C3=A9nieur Syst=C3=A8mes= et R=C3=A9seaux<span style=3D"font-family: Arial,sans-serif; color: #337ab= 7; font-size: 11px;" data-mce-style=3D"font-family: Arial,sans-serif; color= : #337ab7; font-size: 11px;"> / Systems and Networks engineer</span></td></= tr><tr><td style=3D"line-height: 16px; font-family: 'Arial',sans-serif; col= or: #2b3c7a; font-size: 11px;" data-mce-style=3D"line-height: 16px; font-fa= mily: 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;"><span style=3D"= font-family: Arial,sans-serif; color: #2b3c7a; font-size: 11px;" data-mce-s= tyle=3D"font-family: Arial,sans-serif; color: #2b3c7a; font-size: 11px;">Te= l: +33 5 32 09 09 70</span></td></tr><tr><td style=3D"line-height: 16px; fo= nt-family: 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;" data-mce-s= tyle=3D"line-height: 16px; font-family: 'Arial',sans-serif; color: #2b3c7a;= font-size: 11px;"><a href=3D"mailto:nelson.lameiras@lyra-network.com" styl= e=3D"font-size: 11px; font-family: Arial,sans-serif; line-height: 16px; tex= t-decoration: none;" data-mce-href=3D"mailto:nelson.lameiras@lyra-network.c= om" data-mce-style=3D"font-size: 11px; font-family: Arial,sans-serif; line-= height: 16px; text-decoration: none;">nelson.lameiras@lyra-network.com</a><= br data-mce-bogus=3D"1"></td></tr><tr><td style=3D"line-height: 16px; font-= family: 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;" data-mce-styl= e=3D"line-height: 16px; font-family: 'Arial',sans-serif; color: #2b3c7a; fo= nt-size: 11px;"><a href=3D"https://www.lyra-network.com/" style=3D"font-siz= e: 11px; font-family: Arial,sans-serif; line-height: 16px; border: none; te= xt-decoration: none;" data-mce-href=3D"https://www.lyra-network.com/" data-= mce-style=3D"font-size: 11px; font-family: Arial,sans-serif; line-height: 1= 6px; border: none; text-decoration: none;">www.lyra-network.com</a> <a href= =3D"https://payzen.eu" style=3D"font-size: 11px; font-family: Arial,sans-se= rif; line-height: 16px; border: none; text-decoration: none;" data-mce-href= =3D"https://payzen.eu" data-mce-style=3D"font-size: 11px; font-family: Aria= l,sans-serif; line-height: 16px; border: none; text-decoration: none;"> | w= ww.payzen.eu</a><br data-mce-bogus=3D"1"></td></tr></tbody></table></td></t= r></tbody></table><table style=3D"margin: 0; table-layout: fixed; border: n= one; width: 345pt;" data-mce-style=3D"margin: 0; table-layout: fixed; borde= r: none; width: 345pt;" data-mce-selected=3D"1" width=3D"460" cellspacing= =3D"0" cellpadding=3D"0"><tbody><tr><td style=3D"margin: 0; font-size: 11px= ; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: = 24pt; text-align: left;" data-mce-style=3D"margin: 0; font-size: 11px; font= -family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 24pt; = text-align: left;" width=3D"32" valign=3D"top"><a href=3D"https://www.youtu= be.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA" style=3D"border: none; text-decora= tion: none; margin: 0; float: left; display: inline-block; width: 32px;" da= ta-mce-href=3D"https://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA" da= ta-mce-style=3D"border: none; text-decoration: none; margin: 0; float: left= ; display: inline-block; width: 32px;"><img src=3D"cid:4c5918308790526a8092= b3bb40ae8dab65d1777b@zimbra" alt=3D"" data-mce-src=3D"/home/admin@lyra-netw= ork.com/corporate-documents/element-signature_logo_YouTube_32x28.jpg" doc= =3D"corporate-documents/element-signature_logo_YouTube_32x28.jpg" height=3D= "28" width=3D"32" border=3D"0"></a><br data-mce-bogus=3D"1"></td><td style= =3D"margin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c= 7a; line-height: 16px; width: 30.75pt; text-align: left;" data-mce-style=3D= "margin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a;= line-height: 16px; width: 30.75pt; text-align: left;" width=3D"41" valign= =3D"top"><a href=3D"https://www.linkedin.com/company/lyra-network_2" style= =3D"border: none; text-decoration: none; margin: 0; float: left; display: i= nline-block; width: 41px;" data-mce-href=3D"https://www.linkedin.com/compan= y/lyra-network_2" data-mce-style=3D"border: none; text-decoration: none; ma= rgin: 0; float: left; display: inline-block; width: 41px;"><img src=3D"cid:= 0c1a4f5470e19b606b0a0890da695ad9d2808a60@zimbra" data-mce-src=3D"/home/admi= n@lyra-network.com/corporate-documents/element-signature_logo_LinkedIn_41x2= 8.jpg" doc=3D"corporate-documents/element-signature_logo_LinkedIn_41x28.jpg= " height=3D"28" width=3D"41" border=3D"0"></a><br data-mce-bogus=3D"1"></td= th=3D"42" border=3D"0"></a><br data-mce-bogus=3D"1"></td><td style=3D"margi= n: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-= height: 16px; width: 258.75pt; text-align: left;" data-mce-style=3D"margin:= 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-he= ight: 16px; width: 258.75pt; text-align: left;" width=3D"345" valign=3D"top= "><a href=3D"https://payzen.eu" style=3D"border: none; text-decoration: non= e; margin: 0; float: left; display: inline-block; width: 61px;" data-mce-hr= ef=3D"https://payzen.eu" data-mce-style=3D"border: none; text-decoration: n= one; margin: 0; float: left; display: inline-block; width: 61px;"><img src= =3D"cid:7003ef2927fa834b99af5bd467d27eb6d701e6b1@zimbra" alt=3D"" data-mce-= src=3D"/home/admin@lyra-network.com/corporate-documents/element-signature_p= ayzen_61x28.jpg" doc=3D"corporate-documents/element-signature_payzen_61x28.= jpg" height=3D"28" width=3D"61" border=3D"0"></a><br data-mce-bogus=3D"1"><= /td></tr></tbody></table><table style=3D"margin: 0; table-layout: fixed; bo= rder: none; width: 345pt;" data-mce-style=3D"margin: 0; table-layout: fixed= ; border: none; width: 345pt;" width=3D"460" cellspacing=3D"0" cellpadding= =3D"0"><tbody><tr><td><hr style=3D"background: #293b79 none repeat scroll 0= % 0%; width: 100%; height: 4px; color: #293b79; margin: 0px;" data-mce-styl= e=3D"background: #293b79 none repeat scroll 0% 0%; width: 100%; height: 4px= ; color: #293b79; margin: 0px;"></td></tr></tbody></table><table style=3D"m= argin: 0; table-layout: fixed; border: none; width: 345pt;" data-mce-style= =3D"margin: 0; table-layout: fixed; border: none; width: 345pt;" width=3D"4= 60" cellspacing=3D"0" cellpadding=3D"0"><tbody><tr><td style=3D"margin: 0; = font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height= : 16px; width: 30.75pt; text-align: left;" data-mce-style=3D"margin: 0; fon= t-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 1= 6px; width: 30.75pt; text-align: left;"><div>Lyra Network, 109 rue de l'inn= ovation, 31670 Lab=C3=A8ge, FRANCE</div></td></tr></tbody></table><div><br>= </div></div><div><br></div><hr id=3D"zwchr" data-marker=3D"__DIVIDER__"><di= v data-marker=3D"__HEADERS__"><b>From: </b>"Gianluca Cecchi" <gianluca.c= ecchi@gmail.com><br><b>To: </b>"Nelson Lameiras" <nelson.lameiras@lyr= a-network.com><br><b>Cc: </b>"ovirt users" <users@ovirt.org><br><b= href=3D"mailto:nelson.lameiras@lyra-network.com" target=3D"_blank">nelson.l= ameiras@lyra-network.com</a>></span> wrote:</blockquote><blockquote clas= s=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid r= gb(204,204,204);padding-left:1ex"><div><div style=3D"font-family:arial,helv= etica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><div>Hello,<br></div><br>= <div>In our oVirt hosts, we are using DELL equallogic SAN with each server = connecting to SAN via 2 physical interfaces. Since both interfaces share th= e same network (Equalogic limitation) we must patch the linux kernel to to = allow iSCSI multipath with multiple NICs in the same subnet with sysctl :<b= r></div><br><div>----------------------------------------------------------= ----------------------------------<br></div><div><pre>net.ipv4.conf.p2p1.ar= p_ignore=3D1 net.ipv4.conf.p2p1.arp_announce=3D2 net.ipv4.conf.p2p1.rp_filter=3D2 net.ipv4.conf.p2p2.arp_ignore=3D1 net.ipv4.conf.p2p2.arp_announce=3D2 net.ipv4.conf.p2p2.rp_filter=3D2<br><br></pre></div></div></div></blockquot= e><blockquote><div>Hello Nelson,</div><div>nice to see another environment = based on EQL iSCSI SAN.</div><div>The systctl.conf customized lines above a= re not what you mean with "we must patch the linux kernel", is it?</div><di= v>What do you mean exactly?</div></blockquote></div><div class=3D"gmail_quo= te"><br data-mce-bogus=3D"1"></div><div class=3D"gmail_quote">Sorry, by "pa= tch" I meant "sysctl.conf configuration" indeed. <br></div><div class=3D"gm= ail_quote">This configuration is what we use to be able to configure 2 netw= ork interfaces in the same subnet.<br><div> <br></div><div><div style= =3D"font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"= pipermail/users/2017-March/080992.html</a><br data-mce-bogus=3D"1"></div><d= iv>It has been working now for about 3 months without problems. We have sev= eral Oracle RDBMS server VMs on it, from 2virtual threads and 8Gb of ram to= 16 threads (2-4-2) and 64Gb of ram </div><div>How did you configure = multipath instead? As far as I know oVirt will not let you do it.</div><div=
The only way would be to use bonding but that is not certified/supported b= y Dell.</div><div>HIH sharing experiences,</div><div>Gianluca</div><div><br= data-mce-bogus=3D"1"></div><div><br data-mce-bogus=3D"1"></div></blockquot= e><div>I've read the link you gave and it seems to me that you are using th= e same physical interface for SAN and vm network.<br></div><div>In my setup= : </div><div> - SAN network is using two disctinct physical interfac= es (p2pX) on same subnet (therefore the need for sysctl patch)<br></div><di= v> - VM network (p1pX) is using 2 other distinct physical interfaces = on "trunk" mode<br data-mce-bogus=3D"1"></div><div> - Administration = network (emX) is using 2 distinct bonded network interfaces <br data-mce-bo= gus=3D"1"></div><div><br data-mce-bogus=3D"1"></div><div>Our setup, using a= total of 6 network interfaces is "simpler" than yours since each interface= has a "disctinct" role.<br data-mce-bogus=3D"1"></div><div><br data-mce-bo= gus=3D"1"></div><div>To make multipath work with oVirt on this setup, we ha= d to configure SAN network on system before installing oVirt (using sysctl.= conf above), </div><div> - next declare 2 SAN networks on oVirt that = are "dragued" to each SAN interface in host network configuration, </div><d= iv> - and finally, in datacenter listing, chosing a datacenter and se= lecting "iSCSI Multipathing" tab below I can add a multipath configuration = concerning both SAN oVirt networks and relevant iqn storage targets.<br dat= a-mce-bogus=3D"1"></div><div><br data-mce-bogus=3D"1"></div><div>Hope it's = clear, if not, please ask me.<br data-mce-bogus=3D"1"></div><div><br data-m= ce-bogus=3D"1"></div><div>It's very strange to me also since this configura= tion did work great on our labs. But half of production sites have the prob= lem decribed above : second SAN interface stops working uppon host activati= on because vdsm forces sysctl net.ipv4.conf.p2p2.rp_filter to 1 (strict mod= e).<br data-mce-bogus=3D"1"></div><div><br data-mce-bogus=3D"1"></div><div>= Still searching for clues as why this behaviour...<br data-mce-bogus=3D"1">= </div></div></div></div><br></div></div></body></html> ------=_Part_57900107_1954785857.1496910345350 Content-Type: image/jpeg; name=element-signature_logo_lyra_115x94.jpg Content-Disposition: attachment; filename=element-signature_logo_lyra_115x94.jpg Content-Transfer-Encoding: base64 Content-ID: <25cd0e595e0f025605ddb995496484dcb70629d9@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOkU1QUU4Nzk1OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOkU1QUU4Nzk0OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIAF4AcwMBEQACEQEDEQH/xACdAAEBAAIDAQEAAAAAAAAAAAAACAUH AgMGBAEBAQADAQEAAAAAAAAAAAAAAAACBAUBAxAAAAUDAgAHDQcCBwAAAAAAAAECAwQRBQYSByFR E7RWFzcxQSKS0nOT03RVlQgYYXEyQiMUFYEzsVJicmODOBEAAgECAwcEAgIDAAAAAAAAAAECEQNR EwQhgbESMlIzMUGhFHEVwQUiQzT/2gAMAwEAAhEDEQA/AKpAE9b/AGQZdF3FtNpsd4l29M2DHSll iS6w0bzsp5slqJCiLvERnTuENTRQg7bckntK15vmojj1WfMP0rc+Jy/JD7Fjt+EMueJ6bbnBN4bR lkWdkt+XPtDaHSfjKnSHyUpTZkg+TcLSdFGRjyv3rUo0iqP8EoQkntZ7fcOy5VdrXGYxuccGW2+S 3nCecYq3oUWnU2RmfhGXAPPSXLcJNzVVT8kNVbnKKUHRmppW1/zBOSXVtZS4lpSjNtP8nLKhfcSR alqdPXp+ERhauKKTe38nX1WfMP0qc+Jy/JHPsWO34RPLniOqz5h+lTnxOX5IfYsdvwhlzxHVZ8w/ Spz4nL8kPsWO34Qy54jqs+YfpU58Tl+SH2LHb8IZc8R1WfMP0qc+Jy/JD7Fjt+EMueJ2bCX3MH9y LxZr7eJdxKBCkoW0/JdfaJ5mUy2a0EszL/MRHTuGGshDLTiqVf8AAst81GUGMssgAAAAE5b89s+L +zwefPDV0fhlv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACc t+e2fF/Z4PPnhq6Pwy38Crd60UaMotAAAAAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLI AAAABOW/PbPi/s8Hnzw1dH4Zb+BVu9aKNGUWjAKzrGk5cnEjkq/nVo5RMbk3NOnkzd/uU0fgKvdH rky5eb2I86rQz48iQAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLIAAAABOW/PbPi/s8Hn zw1dH4Zb+BVu9aKNGUWjSD//AKkj+xnzJQ0V/wA2/wDkr/7D3e7GX5DiOMfzlniMTEMPIRORIJfg tOeCladCk/nNJH94raa1GcuVnpck4qqOrJdz4dr2yazKOhDjkxhlUGMszop98i/TOlD/AE/CNX+0 x23p27nIJXKRqeeyjdnLMd27sGSzLdEO4Xd4uWifqkhDDiFut0qrUSzQlNa9wz7g9bemjK44puiI yuNRTOzGs53hud+gvzsYjwMVnqNfLuqUTrEck69bitZmStPc1NJIz4hy5ZtKLpKskIzk36bDHRt2 9yMsuMw9v8ejybNBc5M5s5Rp5Uy4aEZuMJSai4dPCZFStBN6a3BLne1nMyT6Uffgu8GQ5DnysXuN mbtZx461TG1ms3kyGiLWRHXToMz8Hg7nfMQvaWMYcydTsLjcqG2BSPYnbY/twy3zVw5+yNXV+GO7 gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLROmdZbAxL5g/5yey6/GjREJW2wSTc M3YpoKms0F3VcY1bNpzscqKs5UnU9pad4cG3Fefw1UWXFO7x3mULkpaJJnoM6JNC10WREak/aXGK 8tLO1/ns2HorilsNTYzar/kOQWTa+5tmmBjk+W9cKGqimkrJS68RcCkoV/yC7clGMXcXrJI8Yptq OBsj5oEpTiNmSRESSuKSJJFwUJlzgoKv9f1P8Hrf9Db1wh/vbVJh6tH7lhbOvi5RBpr/AEqKMXR1 PZrYaC2x3Jgbawp+IZjBkwpkWS480603rSslkRU7qTOuiqFlUlEfepw6WosO61ODK9ufLsZ27f36 RkG/8q9OwXbe3PgKciMPp0uHHS22hpxRf60o1f4VC9DlsUrWjEHWdSghllknbY/twy3zVw5+yNXV +GO7gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLRxU00o6qQlR8ZkRhUGMyO7W7H 7FOvUpCSZgMrfMqERqNJeCgj41qokvtMTtxcpJL3OSdFU1n8vFimPwbrnF18O55DIXybhlw8ihZm s08RLdqVOJJC5rZqqgvSJ42V7v3MpM3dTBnyWbtBQmHFnXFpLzdXOWjW9tWo2irwPpd0pWhVPBPU R0qILTVWx+y+STuYmQj7sxHH1Kctb7NuahnMflKW2akk3J/bvGlojqtpv8ZuJP8ACR8Ag9M8dtTu YdErdfEJTCZKrY/L5JSySbrbCdKmicdWRLeWlJK/bs8skq+ElSacJ8HVppr3OZiP2RuxFK4W5xi3 LK2T5D8NmY8baHHnWySSeTSa6to1K8JTumn3gtM6PbtQzDYYqnqTtsf24Zb5q4c/ZGrq/DHdwK1r rZRIyiyAAAAATlvz2z4v7PB588NXR+GW/gVbvWijRlFoADGZHjVmyO2Ltd4YOTAcUlbjJLcbJRoO qam2pCuA+GlROFxwdV6nJRT2M+q226DbLfHt8BkmIcRtLMdlNaJQgqEVTqZ/eYjKTbqwlQ5Hb4Bp Uk4zRpUpa1FoTQ1uEaVqPg7qiUZGffDmYodSbNZ0toaTBjk02hbTaCaQSUtu/wBxCSpQkr/MXfHe d4iiP12z2h5k2XYMdxlSkuKbW0hSTW2kkIUZGVKpSkiI+8RApPEURydtltdQpDsRlxCtepKm0mR8 qdXKkZfn/NxjnMxQ+kcOk7bH9uGW+auHP2Rq6vwx3cCta62USMosgAAAAE5b89s+L+zwefPDV0fh lv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACb9/3mWN4cbff WTTLUWEt1xX4UpTNeNSj+wiGtolW1LfwKt7qRuTrY226RwfSkKH1rnaz3zI4jrZ226RwfSkH1rna xmRxHWztt0jg+lIPrXO1jMjiOtnbbpHB9KQfWudrGZHEdbO23SOD6Ug+tc7WMyOJ+o3W24WtKE5F CUpR0SknCqZmH1rnazjuxW2pnbTfbPeGnHbXMamNtK0OLaVqJKjKtDELlqUNklQ7C5GfS6mOu+f4 XZpy4F0vMWHMbJKlsOrJKyJRVTUvtIdjZnJVSOuaXqfF1sbbdI4PpSEvrXO1nMyOJp7YiQxJ3oyi RHcS9Hejz3GXUcKVoVOZNKi+wyOovaxUsx3cDwtdbKNGUWgAAAADX+4GzGPZveWbtcpsuO+xGTFS iObRINCFrcIz1oWdaun3xas6qVtUSR5ztKTqeZ+lrCvely8Zj1Q9f2M8EQyEPpawr3pcvGY9UH7G eCGQh9LWFe9Ll4zHqg/YzwQyEPpawr3pcvGY9UH7GeCGQh9LWFe9Ll4zHqg/YzwQyEc2flfwxl5t 1NzuJqbUSiI1MUqX/UOr+xmn6I5LTRaaxNg4XhNuxOHIiwX3n0SXCdWb5pMyMk6eDSlPEPDU6mV5 pteg0+nVpNI8zm+xeNZfkDt7nzprEl5Dbam2FNEgibTpKmttR97jErWslCNEkTlaUnUwP0tYV70u XjMeqHp+xngiOQj023+zGPYReXrtbZsuQ+/GVFUiQbRoJC1ocMy0IQdatF3x5XtVK4qNInC0oups AVT0AAAAAAAAAAAAAAAAAAAAAAAAAAAAD//Z ------=_Part_57900107_1954785857.1496910345350 Content-Type: image/jpeg; name=element-signature_logo_YouTube_32x28.jpg Content-Disposition: attachment; filename=element-signature_logo_YouTube_32x28.jpg Content-Transfer-Encoding: base64 Content-ID: <4c5918308790526a8092b3bb40ae8dab65d1777b@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOjQ1REY5Njk1OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOjQ1REY5Njk0OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAIAMBEQACEQEDEQH/xAB5AAADAQEAAAAAAAAAAAAAAAAABwgF BgEAAgMBAAAAAAAAAAAAAAAAAAQCAwUBEAABBAECBAUFAAAAAAAAAAABAgMEBQARBiESEwcxUdIU CEEik1UYEQACAQMCBQUAAAAAAAAAAAAAAQIRAwQhMVFhEiITQaHBYhX/2gAMAwEAAhEDEQA/AKpw AnD5Hbg3JA35XxKqxmxUPVrJEaI862FuqkPp4IbI5lHQDNXBhFwbaW4tebqcMIvfYjURtz6eesv1 YzWz9fYh38zR7eX2+mu5lHV3VhaNOe8aTJgTHnweVXEBba1cQRkb8IeNtJbHYN9WpW+YY2T53inQ 4HfzZk6a4lmHGaiOSHl8EoT7mSOZR+gBIzUxU3Yklz+Be4+9DRl30N3dMOxjbyro9EzHW3MqupHc L7qlaoX1SsFvlHl44moPpa6XXiW113FPuu3qrT5I7dfrJLUxllMRl15hQWjqBbiuXmTqCQlQx23F rHdSqTrNFFZlDAou7vZa43xuSNaw7CPFZZhIiKbeSsqKkOuuE/aCNNHcexspW40a9Sm5bcmcH/J1 5+0rvxOenGf0o8GQ8DN7ZXxzv9vbpqrhy0huMQJCHlstocBKU+ITqANcru50ZRao9SUbLTqPvMwv DAAwAMAP/9k= ------=_Part_57900107_1954785857.1496910345350 Content-Type: image/jpeg; name=element-signature_logo_LinkedIn_41x28.jpg Content-Disposition: attachment; filename=element-signature_logo_LinkedIn_41x28.jpg Content-Transfer-Encoding: base64 Content-ID: <0c1a4f5470e19b606b0a0890da695ad9d2808a60@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOjY2MEYzRTFGOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOjY2MEYzRTFFOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAKQMBEQACEQEDEQH/xAB2AAEBAQEAAAAAAAAAAAAAAAAABggH AQEAAwEAAAAAAAAAAAAAAAAAAwQFAhAAAQMEAgICAgMBAAAAAAAAAQIDBAARBQYhEjEHMghBE1Fh IhcRAAICAQIDCQAAAAAAAAAAAAABAgMREgQxkTIhQVGhscETMxX/2gAMAwEAAhEDEQA/ANU0Bl33 zsuzQfZcmHj8tMiMKaihDLMl1psKcQkXslSUjk8mtjZ1xdeWkVbW9RDZ/ZN8wOXkYnI7LKMuL1Lq o+RddaIWkLHVwLAPCuf4NWYVwksqPkcNtd5V+ldr2ef7MwkaZmJsqK6p/uy9JdcbUBGcULpUog8i 9Q7uuKreEjquT1GsKxC2KAUBk37B2/6y72t1tB7X8dbIvf8Aq3mtvZfVzKlvUWWw7Dr+Dfy8rCt4 d153Z8ew0FNx3kohuxmUyFtAfFPyBUOBz+arwhKWE89L9SRtLmTuns4xj7LLaxYaGNE6SYojlJZ6 rhLUf19f89eyj4qa1t7ft449ziPWairGLQoBQHFvZ/onObht0jNxclFjMPNtNpZdS4Vj9aAk36gj mtDb7yMIaWiGdTbySSfqnsKbdctAFh1Fm3Bwfx8fFT/ox8GcfAyk9dfX7OapuGNzj+SiPRoJcKmG kuBRDjS2x1uAPK71DfvYzg444nUKmnk7lWcTigFAKAUAoBQCgP/Z ------=_Part_57900107_1954785857.1496910345350 Content-Type: image/jpeg; name=element-signature_logo_Twitter_42x28.jpg Content-Disposition: attachment; filename=element-signature_logo_Twitter_42x28.jpg Content-Transfer-Encoding: base64 Content-ID: <5f6dbbaff6dac1a2e25c3c96f02d8020be9c5268@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOjdDMkI3MjMyOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOjdDMkI3MjMxOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAKgMBEQACEQEDEQH/xAB3AAEBAQEBAAAAAAAAAAAAAAAABwUI BgEBAAMBAQAAAAAAAAAAAAAAAAMEBQIBEAABAwIEBwEBAQAAAAAAAAABAgMEAAUREhMGITFBFBUH CIEiFhEAAgIBAgQHAAAAAAAAAAAAAAERAgMxBCHBEgVBUWGBQhMV/9oADAMBAAIRAxEAPwDC9g7u 3dH33uJiNeri2wxcJQQ01JfCUNodVwCUqwSlI/AK3sGKropS0Kd7OWZMXcPsyZDdnQ7he5UFhQQ/ LZelOMoURiEqWlRAOFdumNOGqyeTYtPzFfL1df8ASm5XCTP0ey0e5eceyZu4zZc5VhjlGOFZ/cKJ dMKNeRNhbclzrOJxQCgOSW7o5a/oKXMRFXNT5iYw/EaAUtbL61tOBKTwXglWOXrW30zgj0RUmLlb 9p3rZ/r/ANd3GxWJtm3XC8B1EO2RsEqDkng88pAP8JSnEk8ugqlt6Xy5Fa3FImyNVUI8z8nJCU7o SOSRAA/O5qbuXx9+Rxg8ToGsssCgFAc/7q+b9xXrc11u7V3iNN3CW/JbQUuhaUuuFYBIHMA9K08e +rWqUPgivbC25MofK25Qoq81DKjwKlJeUT+kE13+jXyZ59DKX6a9WXPYfmO+msTPJdtp6AWMuhq4 5swHPVFVN1uFkiFoS46dJSqqEgoBQCgFAKAUAoBQH//Z ------=_Part_57900107_1954785857.1496910345350 Content-Type: image/jpeg; name=element-signature_payzen_61x28.jpg Content-Disposition: attachment; filename=element-signature_payzen_61x28.jpg Content-Transfer-Encoding: base64 Content-ID: <7003ef2927fa834b99af5bd467d27eb6d701e6b1@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOkU1NjczQjM2OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOkU1NjczQjM1OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAPQMBEQACEQEDEQH/xABvAAABBAMAAAAAAAAAAAAAAAAAAQUG BwIDBAEBAAAAAAAAAAAAAAAAAAAAABAAAQQABAQEAgsBAAAAAAAAAQIDBAUAERIGITETFEEiFQfS I1FhcYGhMlIkRMQWhhEBAAAAAAAAAAAAAAAAAAAAAP/aAAwDAQACEQMRAD8AvqZ7q7OhzZMSS9IQ Yjy2JT/avqZbW2rSvU6lBSACOeeAljLzTzSHmVhxpxIW24k5pUlQzBBHgRgM8BBJ9fKuvcGwr3La wgxIlfGeaagyVsJ1uOOBRUBmOSRgI0bnZgfQwdx7qDrroYZSVzBrcJ0hKM2xq+7AWBU7PTXTEShc 2svRx6MuWp5o5gjigj68BombFRKmPyTfXTPXWpzoszloaRqOelCQPKkeAwEP6lp6f6V6vYaP9Z6b 3fcr7ntuhno6vPnxwGyohb2ntbpr6Y17NdKt7Bp5+YHlPJLitKyltI0KGk+XPANVgzV1Eyyqbe3s IMmnhsMbVaYcebDraY+XVbQ15XXFOghQVnkMBkbjcVPTUs6M5IkPbnpWa1hKlrWG7QZBp0hROnUh xRUfHTgJDWuwtsbptVTpC1xaqhgJfkuFTji9DjoKiTmpSlq/HAcO2d11d/cuXUySJG4C0+NvbfCH CmK2hClHUop0GQ6E+dWfAeUYBi2lY2E+zgW7twhVlHDk+7AkzFO9u2lRdjORFIEZrTmAkA+GYwDl TwLJhjZFouxsF3NzN6r7Lj7imExHUOSHWi0TpyCdORIzzwHT/J/7n+vgLVAA5YBChBUlRSCpOelR HEZ88sA2WG3YVhb1lnJW4VVJcXFjAgM9VxOjqKTlmVJTnp48M8Ax2+3N3J3TJu6KVAbRKisxnWpr brh+SpasxoKf14BBD91hxE2kB+nt5Px4AEL3VGoiZSAr4qyjyOP2+fAL2nuvw/e0vDl8iT8eAbv8 Lur0vPvYXrPrfrfU6bvb59Lp6NOev83HngLBwBgDAGAMAYAwBgP/2Q== ------=_Part_57900107_1954785857.1496910345350-- ------=_Part_57900106_686621279.1496910345349--
On Thu, Jun 8, 2017 at 10:25 AM, Nelson Lameiras < nelson.lameiras@lyra-network.com> wrote:
Hello,
In our oVirt hosts, we are using DELL equallogic SAN with each server connecting to SAN via 2 physical interfaces. Since both interfaces share the same network (Equalogic limitation) we must patch the linux kernel to to allow iSCSI multipath with multiple NICs in the same subnet with sysctl :
------------------------------------------------------------ --------------------------------
net.ipv4.conf.p2p1.arp_ignore=1 net.ipv4.conf.p2p1.arp_announce=2 net.ipv4.conf.p2p1.rp_filter=2
net.ipv4.conf.p2p2.arp_ignore=1 net.ipv4.conf.p2p2.arp_announce=2 net.ipv4.conf.p2p2.rp_filter=2
Hello Nelson, nice to see another environment based on EQL iSCSI SAN. The systctl.conf customized lines above are not what you mean with "we must patch the linux kernel", is it? What do you mean exactly?
Sorry, by "patch" I meant "sysctl.conf configuration" indeed. This configuration is what we use to be able to configure 2 network interfaces in the same subnet.
OK. I'm using that configuration too. And also with physical hosts, used as physical RDBMS servers, that connect to Dell EQL iSCSI SAN, not only for oVirt. It is a requirement so that you are sure that the packets come back to the same interface, because of more that 1 network interface on the same lan. That said, I noticed that oVirt creates the file "vdsm.conf" under /etc/sysctl.d, with these contents net.ipv4.conf.default.arp_ignore = 1 net.ipv4.conf.default.arp_announce = 2 So this settings doesn't need to be overwritten by you. Do you have that file too? In my case I left the oVirt generated file and created two more files in /etc/sysctl.d, named 50-iscsi.conf and equallogic.conf Their contents: - 50-iscsi.conf net.ipv4.conf.p1p1/100.arp_announce=2 net.ipv4.conf.p1p1/100.arp_ignore=1 # net.ipv4.conf.p1p1/100.rp_filter=2 net.ipv4.conf.p1p2.rp_filter=2 The reason of the first two lines is that vlan interfaces apparently didn't inherited the arp_announce and arp_ignore settings so I put them for the vlan set interface, no need for the other one (p1p2 in my case) The reason of the second two lines is to add the third sysctl setting needed by EQL So I think you can have a 50-iscsi.conf file only composed by: net.ipv4.conf.p2p1.rp_filter=2 net.ipv4.conf.p2p2.rp_filter=2 - equallogic.conf net.core.rmem_default = 262144 net.core.rmem_max = 16777216 net.core.wmem_default = 262144 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 8192 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216 # Defaults were: #net.core.rmem_default = 212992 #net.core.rmem_max = 212992 #net.core.wmem_default = 212992 #net.core.wmem_max = 212992 #net.ipv4.tcp_rmem = 4096 87380 6291456 #net.ipv4.tcp_wmem = 4096 16384 4194304
I've read the link you gave and it seems to me that you are using the same physical interface for SAN and vm network.
In my setup : - SAN network is using two disctinct physical interfaces (p2pX) on same subnet (therefore the need for sysctl patch) - VM network (p1pX) is using 2 other distinct physical interfaces on "trunk" mode - Administration network (emX) is using 2 distinct bonded network interfaces
Our setup, using a total of 6 network interfaces is "simpler" than yours since each interface has a "disctinct" role.
To make multipath work with oVirt on this setup, we had to configure SAN network on system before installing oVirt (using sysctl.conf above), - next declare 2 SAN networks on oVirt that are "dragued" to each SAN interface in host network configuration, - and finally, in datacenter listing, chosing a datacenter and selecting "iSCSI Multipathing" tab below I can add a multipath configuration concerning both SAN oVirt networks and relevant iqn storage targets.
Hope it's clear, if not, please ask me.
It's very strange to me also since this configuration did work great on our labs. But half of production sites have the problem decribed above : second SAN interface stops working uppon host activation because vdsm forces sysctl net.ipv4.conf.p2p2.rp_filter to 1 (strict mode).
Still searching for clues as why this behaviour...
I had the further need to use both interfaces for VM network, but, also if not, I couldn't setup multipathing. I have a requirement to have iSCSI on vlan and when setting the two interfaces on the same vlan (so also in oVirt), oVirt complained about that. So this lets me think that if instead you have both network interfaces in access mode, without vlan, oVirt doesn't complain when you try toset the multipathing???? If this is the case, I think I could have tested (probably I didn't, I don't remember) this scenario Both interfaces on vlan 100 but the network switch port configured with default vlan as 100, so that form OS point of view, and oVirt point of view, no vlan in place and perhaps it doesn't complain.... Gianluca
<a href=3D"https://www.linkedin.com/company/lyra-network_2" style=3D"borde= r: none; text-decoration: none; margin: 0; float: left; display: inline-blo= ck; width: 41px;" data-mce-href=3D"https://www.linkedin.com/company/lyra-ne= twork_2" data-mce-style=3D"border: none; text-decoration: none; margin: 0; = float: left; display: inline-block; width: 41px;"><img src=3D"cid:95ac2affd= b66bc26fd0e254578e26d8574139b42@zimbra" data-mce-src=3D"/home/admin@lyra-ne= twork.com/corporate-documents/element-signature_logo_LinkedIn_41x28.jpg" do= c=3D"corporate-documents/element-signature_logo_LinkedIn_41x28.jpg" width= =3D"41" border=3D"0" height=3D"28"></a><br data-mce-bogus=3D"1"></td><td st= yle=3D"margin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2= b3c7a; line-height: 16px; width: 31.5pt; text-align: left;" data-mce-style= =3D"margin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c= 7a; line-height: 16px; width: 31.5pt; text-align: left;" width=3D"42" valig= n=3D"top"><a href=3D"https://twitter.com/LyraNetwork" style=3D"border: none= ; text-decoration: none; margin: 0; float: left; display: inline-block; wid=
------=_Part_88180575_227580591.1500303388932 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello, Can any one please help us with the problem described below?=20 Nir, I'm including you since a quick search on the internet led me to think= that you have worked on this part of the project. Please forgive me if I'm= off topic.=20 (I incorrectly used below the expression "patch" when I meant "configure". = it's corrected now)=20 cordialement, regards,=20 =09Nelson LAMEIRAS=20 Ing=C3=A9nieur Syst=C3=A8mes et R=C3=A9seaux / Systems and Networks enginee= r=20 Tel: +33 5 32 09 09 70=20 nelson.lameiras@lyra-network.com=20 www.lyra-network.com | www.payzen.eu=20 =09 =09 =09 Lyra Network, 109 rue de l'innovation, 31670 Lab=C3=A8ge, FRANCE=20 De: "Nelson Lameiras" <nelson.lameiras@lyra-network.com>=20 =C3=80: "ovirt users" <users@ovirt.org>=20 Envoy=C3=A9: Mercredi 7 Juin 2017 14:59:48=20 Objet: [ovirt-users] ISCSI storage with multiple nics on same subnet disabl= ed on host activation=20 Hello,=20 In our oVirt hosts, we are using DELL equallogic SAN with each server conne= cting to SAN via 2 physical interfaces. Since both interfaces share the sam= e network (Equalogic limitation) we must configure sysctl to to allow iSCSI= multipath with multiple NICs in the same subnet :=20 ---------------------------------------------------------------------------= -----------------=20 net.ipv4.conf.p2p1.arp_ignore=3D1 net.ipv4.conf.p2p1.arp_announce=3D2 net.ipv4.conf.p2p1.rp_filter=3D2 net.ipv4.conf.p2p2.arp_ignore=3D1 net.ipv4.conf.p2p2.arp_announce=3D2 net.ipv4.conf.p2p2.rp_filter=3D2=20 ---------------------------------------------------------------------------= -----------------=20 This works great in most setups, but for a strange reason, on some of our s= etups, the sysctl configuration is updated by VDSM when activating a host a= nd the second interface stops working immeadiatly :=20 ---------------------------------------------------------------------------= -----------------=20 vdsm.log=20 2017-06-07 11:51:51,063+0200 INFO (jsonrpc/5) [storage.ISCSI] Setting stri= ct mode rp_filter for device 'p2p2'. (iscsi:602) 2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not conn= ect to storageServer (hsm:2392) Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer conObj.connect() File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect iscsi.addIscsiNode(self._iface, self._target, self._cred) File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, = in addIscsiNode iscsiadm.node_login(iface.name, target.address, target.iqn) File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 33= 7, in node_login raise IscsiNodeError(rc, out, err)=20 ---------------------------------------------------------------------------= -----------------=20 "strict mode" is enforced for second interface, and it no longuer works...= =20 Which means - at least - that there is no redundancy in case of hardware fa= illure and this is not acceptable for our production needs.=20 What is really strange is that we have another "twin" site on another geogr= aphic region with simillar hardware configuration and same oVirt installati= on, and this problem does not happen.=20 Can this be really random?=20 What can be the root cause of this behaviour? How can I correct it?=20 our setup:=20 oVirt hostedEngine : Centor 7.3, ovirt 4.1.2=20 3 physical oVirt nodes centos 7.3, ovirt 4.1.2=20 SAN DELL Equalogic=20 cordialement, regards,=20 =09Nelson LAMEIRAS=20 Ing=C3=A9nieur Syst=C3=A8mes et R=C3=A9seaux / Systems and Networks enginee= r=20 Tel: +33 5 32 09 09 70=20 nelson.lameiras@lyra-network.com=20 www.lyra-network.com | www.payzen.eu=20 =09 =09 =09 Lyra Network, 109 rue de l'innovation, 31670 Lab=C3=A8ge, FRANCE=20 _______________________________________________=20 Users mailing list=20 Users@ovirt.org=20 http://lists.ovirt.org/mailman/listinfo/users=20 ------=_Part_88180575_227580591.1500303388932 Content-Type: multipart/related; boundary="----=_Part_88180576_1156318871.1500303388932" ------=_Part_88180576_1156318871.1500303388932 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: arial, helvetica, sans-serif; font-s= ize: 12pt; color: #000000"><div>Hello, Can any one please help us with the = problem described below?<br></div><div><br data-mce-bogus=3D"1"></div><div>= Nir, I'm including you since a quick search on the internet led me to think= that you have worked on this part of the project. Please forgive me if I'm= off topic.<br></div><div><br data-mce-bogus=3D"1"></div><div>(I incorrectl= y used below the expression "patch" when I meant "configure". it's correcte= d now)<br data-mce-bogus=3D"1"></div><div><br></div><div data-marker=3D"__S= IG_PRE__"><div>cordialement, regards,</div><div><br data-mce-bogus=3D"1"></= div><table style=3D"margin: 0px; table-layout: fixed; border: none; width: = 345pt; padding: 0px;" data-mce-style=3D"margin: 0px; table-layout: fixed; b= order: none; width: 345pt; padding: 0px;" width=3D"460" cellspacing=3D"0" c= ellpadding=3D"0" border=3D"0"><tbody><tr><td style=3D"margin: 0; font-size:= 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; wi= dth: 86.25pt; text-align: left;" data-mce-style=3D"margin: 0; font-size: 11= px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width= : 86.25pt; text-align: left;" width=3D"115" valign=3D"top"><a href=3D"https= ://www.lyra-network.com/" style=3D"border: none; text-decoration: none; wid= th: 100%; display: block; clear: both;" data-mce-href=3D"https://www.lyra-n= etwork.com/" data-mce-style=3D"border: none; text-decoration: none; width: = 100%; display: block; clear: both;"><img src=3D"cid:ac318af44fd2ff936fd390a= 2dfa5ac7f8bf624d9@zimbra" alt=3D"" data-mce-src=3D"/home/admin@lyra-network= .com/corporate-documents/element-signature_logo_lyra_115x94.jpg" doc=3D"cor= porate-documents/element-signature_logo_lyra_115x94.jpg" width=3D"115" bord= er=3D"0" height=3D"94"></a><br data-mce-bogus=3D"1"></td><td style=3D"margi= n: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-= height: 16px; width: 258.75pt;" data-mce-style=3D"margin: 0; font-size: 11p= x; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width:= 258.75pt;" width=3D"345" valign=3D"top"><table style=3D"margin: 0; width: = 258.75pt; table-layout: fixed; border: none;" data-mce-style=3D"margin: 0; = width: 258.75pt; table-layout: fixed; border: none;" width=3D"345" cellspac= ing=3D"0" cellpadding=3D"0" border=3D"0"><tbody><tr><td style=3D"font-size:= 16px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 18px; fo= nt-weight: bold;" data-mce-style=3D"font-size: 16px; font-family: Arial,san= s-serif; color: #2b3c7a; line-height: 18px; font-weight: bold;">Nelson LAME= IRAS</td></tr><tr><td style=3D"font-family: Arial,sans-serif; color: #2b3c7= a; font-size: 11px; line-height: 16px;" data-mce-style=3D"font-family: Aria= l,sans-serif; color: #2b3c7a; font-size: 11px; line-height: 16px;">Ing=C3= =A9nieur Syst=C3=A8mes et R=C3=A9seaux<span style=3D"font-family: Arial,san= s-serif; color: #337ab7; font-size: 11px;" data-mce-style=3D"font-family: A= rial,sans-serif; color: #337ab7; font-size: 11px;"> / Systems and Networks = engineer</span></td></tr><tr><td style=3D"line-height: 16px; font-family: '= Arial',sans-serif; color: #2b3c7a; font-size: 11px;" data-mce-style=3D"line= -height: 16px; font-family: 'Arial',sans-serif; color: #2b3c7a; font-size: = 11px;"><span style=3D"font-family: Arial,sans-serif; color: #2b3c7a; font-s= ize: 11px;" data-mce-style=3D"font-family: Arial,sans-serif; color: #2b3c7a= ; font-size: 11px;">Tel: +33 5 32 09 09 70</span></td></tr><tr><td style=3D= "line-height: 16px; font-family: 'Arial',sans-serif; color: #2b3c7a; font-s= ize: 11px;" data-mce-style=3D"line-height: 16px; font-family: 'Arial',sans-= serif; color: #2b3c7a; font-size: 11px;"><a href=3D"mailto:nelson.lameiras@= lyra-network.com" style=3D"font-size: 11px; font-family: Arial,sans-serif; = line-height: 16px; text-decoration: none;" data-mce-href=3D"mailto:nelson.l= ameiras@lyra-network.com" data-mce-style=3D"font-size: 11px; font-family: A= rial,sans-serif; line-height: 16px; text-decoration: none;">nelson.lameiras= @lyra-network.com</a><br data-mce-bogus=3D"1"></td></tr><tr><td style=3D"li= ne-height: 16px; font-family: 'Arial',sans-serif; color: #2b3c7a; font-size= : 11px;" data-mce-style=3D"line-height: 16px; font-family: 'Arial',sans-ser= if; color: #2b3c7a; font-size: 11px;"><a href=3D"https://www.lyra-network.c= om/" style=3D"font-size: 11px; font-family: Arial,sans-serif; line-height: = 16px; border: none; text-decoration: none;" data-mce-href=3D"https://www.ly= ra-network.com/" data-mce-style=3D"font-size: 11px; font-family: Arial,sans= -serif; line-height: 16px; border: none; text-decoration: none;">www.lyra-n= etwork.com</a> <a href=3D"https://payzen.eu" style=3D"font-size: 11px; font= -family: Arial,sans-serif; line-height: 16px; border: none; text-decoration= : none;" data-mce-href=3D"https://payzen.eu" data-mce-style=3D"font-size: 1= 1px; font-family: Arial,sans-serif; line-height: 16px; border: none; text-d= ecoration: none;"> | www.payzen.eu</a><br data-mce-bogus=3D"1"></td></tr></= tbody></table></td></tr></tbody></table><table style=3D"margin: 0; table-la= yout: fixed; border: none; width: 345pt;" data-mce-style=3D"margin: 0; tabl= e-layout: fixed; border: none; width: 345pt;" width=3D"460" cellspacing=3D"= 0" cellpadding=3D"0"><tbody><tr><td style=3D"margin: 0; font-size: 11px; fo= nt-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 24pt= ; text-align: left;" data-mce-style=3D"margin: 0; font-size: 11px; font-fam= ily: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 24pt; text= -align: left;" width=3D"32" valign=3D"top"><a href=3D"https://www.youtube.c= om/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA" style=3D"border: none; text-decoration= : none; margin: 0; float: left; display: inline-block; width: 32px;" data-m= ce-href=3D"https://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA" data-m= ce-style=3D"border: none; text-decoration: none; margin: 0; float: left; di= splay: inline-block; width: 32px;"><img src=3D"cid:6f3cc57ba362f7a2942a16e8= 8d1ea9ce686c7da2@zimbra" alt=3D"" data-mce-src=3D"/home/admin@lyra-network.= com/corporate-documents/element-signature_logo_YouTube_32x28.jpg" doc=3D"co= rporate-documents/element-signature_logo_YouTube_32x28.jpg" width=3D"32" bo= rder=3D"0" height=3D"28"></a><br data-mce-bogus=3D"1"></td><td style=3D"mar= gin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; lin= e-height: 16px; width: 30.75pt; text-align: left;" data-mce-style=3D"margin= : 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-h= eight: 16px; width: 30.75pt; text-align: left;" width=3D"41" valign=3D"top"= th: 42px;" data-mce-href=3D"https://twitter.com/LyraNetwork" data-mce-style= =3D"border: none; text-decoration: none; margin: 0; float: left; display: i= nline-block; width: 42px;"><img src=3D"cid:5e9e243634081dcf099c93fd9ca78f8b= 346f57b3@zimbra" alt=3D"" data-mce-src=3D"/home/admin@lyra-network.com/corp= orate-documents/element-signature_logo_Twitter_42x28.jpg" doc=3D"corporate-= documents/element-signature_logo_Twitter_42x28.jpg" width=3D"42" border=3D"= 0" height=3D"28"></a><br data-mce-bogus=3D"1"></td><td style=3D"margin: 0; = font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height= : 16px; width: 258.75pt; text-align: left;" data-mce-style=3D"margin: 0; fo= nt-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: = 16px; width: 258.75pt; text-align: left;" width=3D"345" valign=3D"top"><a h= ref=3D"https://payzen.eu" style=3D"border: none; text-decoration: none; mar= gin: 0; float: left; display: inline-block; width: 61px;" data-mce-href=3D"= https://payzen.eu" data-mce-style=3D"border: none; text-decoration: none; m= argin: 0; float: left; display: inline-block; width: 61px;"><img src=3D"cid= :72acbdf9e9761a33ad68ac239cb304da61f1cda1@zimbra" alt=3D"" data-mce-src=3D"= /home/admin@lyra-network.com/corporate-documents/element-signature_payzen_6= 1x28.jpg" doc=3D"corporate-documents/element-signature_payzen_61x28.jpg" wi= dth=3D"61" border=3D"0" height=3D"28"></a><br data-mce-bogus=3D"1"></td></t= r></tbody></table><table style=3D"margin: 0; table-layout: fixed; border: n= one; width: 345pt;" data-mce-style=3D"margin: 0; table-layout: fixed; borde= r: none; width: 345pt;" width=3D"460" cellspacing=3D"0" cellpadding=3D"0"><= tbody><tr><td><hr style=3D"background: #293b79 none repeat scroll 0% 0%; wi= dth: 100%; height: 4px; color: #293b79; margin: 0px;" data-mce-style=3D"bac= kground: #293b79 none repeat scroll 0% 0%; width: 100%; height: 4px; color:= #293b79; margin: 0px;"></td></tr></tbody></table><table style=3D"margin: 0= ; table-layout: fixed; border: none; width: 345pt;" data-mce-style=3D"margi= n: 0; table-layout: fixed; border: none; width: 345pt;" width=3D"460" cells= pacing=3D"0" cellpadding=3D"0"><tbody><tr><td style=3D"margin: 0; font-size= : 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; w= idth: 30.75pt; text-align: left;" data-mce-style=3D"margin: 0; font-size: 1= 1px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; widt= h: 30.75pt; text-align: left;"><div>Lyra Network, 109 rue de l'innovation, = 31670 Lab=C3=A8ge, FRANCE</div></td></tr></tbody></table><div><br></div></d= iv><br><hr id=3D"zwchr" data-marker=3D"__DIVIDER__"><div data-marker=3D"__H= EADERS__"><b>De: </b>"Nelson Lameiras" <nelson.lameiras@lyra-network.com= ><br><b>=C3=80: </b>"ovirt users" <users@ovirt.org><br><b>Envoy=C3= =A9: </b>Mercredi 7 Juin 2017 14:59:48<br><b>Objet: </b>[ovirt-users] ISCSI= storage with multiple nics on same subnet disabled on host activation<br><= /div><br><div data-marker=3D"__QUOTED_TEXT__"><div style=3D"font-family: ar= ial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div>Hello,<br=
</div><br><div>In our oVirt hosts, we are using DELL equallogic SAN with e= ach server connecting to SAN via 2 physical interfaces. Since both interfac= es share the same network (Equalogic limitation) we must configure sysctl t= o to allow iSCSI multipath with multiple NICs in the same subnet :<br></div= <br><div>-----------------------------------------------------------------= ---------------------------<br></div><div><pre>net.ipv4.conf.p2p1.arp_ignor= e=3D1 net.ipv4.conf.p2p1.arp_announce=3D2 net.ipv4.conf.p2p1.rp_filter=3D2
net.ipv4.conf.p2p2.arp_ignore=3D1 net.ipv4.conf.p2p2.arp_announce=3D2 net.ipv4.conf.p2p2.rp_filter=3D2<br><br></pre><p>--------------------------= ------------------------------------------------------------------<br><br><= /p><p>This works great in most setups, but for a strange reason, on some of= our setups, the sysctl configuration is updated by VDSM when activating a = host and the second interface stops working immeadiatly :</p><div>---------= ---------------------------------------------------------------------------= --------</div><div>vdsm.log<br></div><div><pre id=3D"preview"><code>2017-06= -07 11:51:51,063+0200 INFO (jsonrpc/5) [storage.ISCSI] Setting strict mode= rp_filter for device 'p2p2'. (iscsi:602) 2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not conn= ect to storageServer (hsm:2392) Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer conObj.connect() File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect iscsi.addIscsiNode(self._iface, self._target, self._cred) File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, = in addIscsiNode iscsiadm.node_login(iface.name, target.address, target.iqn) File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 33= 7, in node_login raise IscsiNodeError(rc, out, err)</code></pre><pre><br><br></pre><p st= yle=3D"margin: 0px;">------------------------------------------------------= --------------------------------------</p></div></div><div><br><div>"strict= mode" is enforced for second interface, and it no longuer works...</div><d= iv>Which means - at least - that there is no redundancy in case of hardware= faillure and this is not acceptable for our production needs.<br></div><br=
<div>What is really strange is that we have another "twin" site on another= geographic region with simillar hardware configuration and same oVirt inst= allation, and this problem does not happen.<br></div><div>Can this be reall= y random?<br></div>What can be the root cause of this behaviour? How can I = correct it?<br><br><div>our setup:</div><div>oVirt hostedEngine : Centor 7.= 3, ovirt 4.1.2<br></div><div>3 physical oVirt nodes centos 7.3, ovirt 4.1.2= <br></div><div>SAN DELL Equalogic<br></div></div><br><div><div>cordialement= , regards,</div><br><table style=3D"margin: 0px; table-layout: fixed; borde= r: none; width: 345pt; padding: 0px;" width=3D"460" cellspacing=3D"0" cellp= adding=3D"0" border=3D"0"><tbody><tr><td style=3D"margin: 0; font-size: 11p= x; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width:= 86.25pt; text-align: left;" width=3D"115" valign=3D"top"><a href=3D"https:= //www.lyra-network.com/" style=3D"border: none; text-decoration: none; widt= h: 100%; display: block; clear: both;" target=3D"_blank"><img src=3D"cid:6f= a8666d679e18cab5e566228d0ca3519aabec27@zimbra" alt=3D"" width=3D"115" borde= r=3D"0" height=3D"94"></a><br></td><td style=3D"margin: 0; font-size: 11px;= font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 2= 58.75pt;" width=3D"345" valign=3D"top"><table style=3D"margin: 0; width: 25= 8.75pt; table-layout: fixed; border: none;" width=3D"345" cellspacing=3D"0"= cellpadding=3D"0" border=3D"0"><tbody><tr><td style=3D"font-size: 16px; fo= nt-family: Arial,sans-serif; color: #2b3c7a; line-height: 18px; font-weight= : bold;">Nelson LAMEIRAS</td></tr><tr><td style=3D"font-family: Arial,sans-= serif; color: #2b3c7a; font-size: 11px; line-height: 16px;">Ing=C3=A9nieur = Syst=C3=A8mes et R=C3=A9seaux<span style=3D"font-family: Arial,sans-serif; =
color: #337ab7; font-size: 11px;"> / Systems and Networks engineer</span></= td></tr><tr><td style=3D"line-height: 16px; font-family: 'Arial',sans-serif= ; color: #2b3c7a; font-size: 11px;"><span style=3D"font-family: Arial,sans-= serif; color: #2b3c7a; font-size: 11px;">Tel: +33 5 32 09 09 70</span></td>= </tr><tr><td style=3D"line-height: 16px; font-family: 'Arial',sans-serif; c= olor: #2b3c7a; font-size: 11px;"><a href=3D"mailto:nelson.lameiras@lyra-net= work.com" style=3D"font-size: 11px; font-family: Arial,sans-serif; line-hei= ght: 16px; text-decoration: none;" target=3D"_blank">nelson.lameiras@lyra-n= etwork.com</a><br></td></tr><tr><td style=3D"line-height: 16px; font-family= : 'Arial',sans-serif; color: #2b3c7a; font-size: 11px;"><a href=3D"https://= www.lyra-network.com/" style=3D"font-size: 11px; font-family: Arial,sans-se= rif; line-height: 16px; border: none; text-decoration: none;" target=3D"_bl= ank">www.lyra-network.com</a> <a href=3D"https://payzen.eu" style=3D"font-s= ize: 11px; font-family: Arial,sans-serif; line-height: 16px; border: none; = text-decoration: none;" target=3D"_blank"> | www.payzen.eu</a><br></td></tr= ></tbody></table></td></tr></tbody></table><table style=3D"margin: 0; table= -layout: fixed; border: none; width: 345pt;" data-mce-selected=3D"1" width= =3D"460" cellspacing=3D"0" cellpadding=3D"0"><tbody><tr><td style=3D"margin= : 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-h= eight: 16px; width: 24pt; text-align: left;" width=3D"32" valign=3D"top"><a= href=3D"https://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA" style=3D= "border: none; text-decoration: none; margin: 0; float: left; display: inli= ne-block; width: 32px;" target=3D"_blank"><img src=3D"cid:2806a89cb9fbba6f8= 79bd3fd54a93b515defd6d2@zimbra" alt=3D"" width=3D"32" border=3D"0" height= =3D"28"></a><br></td><td style=3D"margin: 0; font-size: 11px; font-family: = Arial,sans-serif; color: #2b3c7a; line-height: 16px; width: 30.75pt; text-a= lign: left;" width=3D"41" valign=3D"top"><a href=3D"https://www.linkedin.co= m/company/lyra-network_2" style=3D"border: none; text-decoration: none; mar= gin: 0; float: left; display: inline-block; width: 41px;" target=3D"_blank"= ><img src=3D"cid:878634ba055bb6ae851a7ef4540f754afa3ff86c@zimbra" width=3D"= 41" border=3D"0" height=3D"28"></a><br></td><td style=3D"margin: 0; font-si= ze: 11px; font-family: Arial,sans-serif; color: #2b3c7a; line-height: 16px;= width: 31.5pt; text-align: left;" width=3D"42" valign=3D"top"><a href=3D"h= ttps://twitter.com/LyraNetwork" style=3D"border: none; text-decoration: non= e; margin: 0; float: left; display: inline-block; width: 42px;" target=3D"_= blank"><img src=3D"cid:18a767bbdad6a1133fc4152b03ea78228ce25a14@zimbra" alt= =3D"" width=3D"42" border=3D"0" height=3D"28"></a><br></td><td style=3D"mar= gin: 0; font-size: 11px; font-family: Arial,sans-serif; color: #2b3c7a; lin= e-height: 16px; width: 258.75pt; text-align: left;" width=3D"345" valign=3D= "top"><a href=3D"https://payzen.eu" style=3D"border: none; text-decoration:= none; margin: 0; float: left; display: inline-block; width: 61px;" target= =3D"_blank"><img src=3D"cid:d3fc119a0aa72bf985c19ca559c993b5b4b54ca3@zimbra= " alt=3D"" width=3D"61" border=3D"0" height=3D"28"></a><br></td></tr></tbod= y></table><table style=3D"margin: 0; table-layout: fixed; border: none; wid= th: 345pt;" width=3D"460" cellspacing=3D"0" cellpadding=3D"0"><tbody><tr><t= d><hr style=3D"background: #293b79 none repeat scroll 0% 0%; width: 100%; h= eight: 4px; color: #293b79; margin: 0px;"></td></tr></tbody></table><table = style=3D"margin: 0; table-layout: fixed; border: none; width: 345pt;" data-= mce-selected=3D"1" width=3D"460" cellspacing=3D"0" cellpadding=3D"0"><tbody= ><tr><td style=3D"margin: 0; font-size: 11px; font-family: Arial,sans-serif= ; color: #2b3c7a; line-height: 16px; width: 30.75pt; text-align: left;"><di= v>Lyra Network, 109 rue de l'innovation, 31670 Lab=C3=A8ge, FRANCE</div></t= d></tr></tbody></table><br></div></div><br>________________________________= _______________<br>Users mailing list<br>Users@ovirt.org<br>http://lists.ov= irt.org/mailman/listinfo/users<br></div></div></body></html> ------=_Part_88180576_1156318871.1500303388932 Content-Type: image/jpeg; name=element-signature_logo_lyra_115x94.jpg Content-Disposition: attachment; filename=element-signature_logo_lyra_115x94.jpg Content-Transfer-Encoding: base64 Content-ID: <6fa8666d679e18cab5e566228d0ca3519aabec27@zimbra>
/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOkU1QUU4Nzk1OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOkU1QUU4Nzk0OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIAF4AcwMBEQACEQEDEQH/xACdAAEBAAIDAQEAAAAAAAAAAAAACAUH AgMGBAEBAQADAQEAAAAAAAAAAAAAAAACBAUBAxAAAAUDAgAHDQcCBwAAAAAAAAECAwQRBQYSByFR E7RWFzcxQSKS0nOT03RVlQgYYXEyQiMUFYEzsVJicmODOBEAAgECAwcEAgIDAAAAAAAAAAECEQNR EwQhgbESMlIzMUGhFHEVwQUiQzT/2gAMAwEAAhEDEQA/AKpAE9b/AGQZdF3FtNpsd4l29M2DHSll iS6w0bzsp5slqJCiLvERnTuENTRQg7bckntK15vmojj1WfMP0rc+Jy/JD7Fjt+EMueJ6bbnBN4bR lkWdkt+XPtDaHSfjKnSHyUpTZkg+TcLSdFGRjyv3rUo0iqP8EoQkntZ7fcOy5VdrXGYxuccGW2+S 3nCecYq3oUWnU2RmfhGXAPPSXLcJNzVVT8kNVbnKKUHRmppW1/zBOSXVtZS4lpSjNtP8nLKhfcSR alqdPXp+ERhauKKTe38nX1WfMP0qc+Jy/JHPsWO34RPLniOqz5h+lTnxOX5IfYsdvwhlzxHVZ8w/ Spz4nL8kPsWO34Qy54jqs+YfpU58Tl+SH2LHb8IZc8R1WfMP0qc+Jy/JD7Fjt+EMueJ2bCX3MH9y LxZr7eJdxKBCkoW0/JdfaJ5mUy2a0EszL/MRHTuGGshDLTiqVf8AAst81GUGMssgAAAAE5b89s+L +zwefPDV0fhlv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACc t+e2fF/Z4PPnhq6Pwy38Crd60UaMotAAAAAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLI AAAABOW/PbPi/s8Hnzw1dH4Zb+BVu9aKNGUWjAKzrGk5cnEjkq/nVo5RMbk3NOnkzd/uU0fgKvdH rky5eb2I86rQz48iQAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLIAAAABOW/PbPi/s8Hn zw1dH4Zb+BVu9aKNGUWjSD//AKkj+xnzJQ0V/wA2/wDkr/7D3e7GX5DiOMfzlniMTEMPIRORIJfg tOeCladCk/nNJH94raa1GcuVnpck4qqOrJdz4dr2yazKOhDjkxhlUGMszop98i/TOlD/AE/CNX+0 x23p27nIJXKRqeeyjdnLMd27sGSzLdEO4Xd4uWifqkhDDiFut0qrUSzQlNa9wz7g9bemjK44puiI yuNRTOzGs53hud+gvzsYjwMVnqNfLuqUTrEck69bitZmStPc1NJIz4hy5ZtKLpKskIzk36bDHRt2 9yMsuMw9v8ejybNBc5M5s5Rp5Uy4aEZuMJSai4dPCZFStBN6a3BLne1nMyT6Uffgu8GQ5DnysXuN mbtZx461TG1ms3kyGiLWRHXToMz8Hg7nfMQvaWMYcydTsLjcqG2BSPYnbY/twy3zVw5+yNXV+GO7 gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLROmdZbAxL5g/5yey6/GjREJW2wSTc M3YpoKms0F3VcY1bNpzscqKs5UnU9pad4cG3Fefw1UWXFO7x3mULkpaJJnoM6JNC10WREak/aXGK 8tLO1/ns2HorilsNTYzar/kOQWTa+5tmmBjk+W9cKGqimkrJS68RcCkoV/yC7clGMXcXrJI8Yptq OBsj5oEpTiNmSRESSuKSJJFwUJlzgoKv9f1P8Hrf9Db1wh/vbVJh6tH7lhbOvi5RBpr/AEqKMXR1 PZrYaC2x3Jgbawp+IZjBkwpkWS480603rSslkRU7qTOuiqFlUlEfepw6WosO61ODK9ufLsZ27f36 RkG/8q9OwXbe3PgKciMPp0uHHS22hpxRf60o1f4VC9DlsUrWjEHWdSghllknbY/twy3zVw5+yNXV +GO7gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLRxU00o6qQlR8ZkRhUGMyO7W7H 7FOvUpCSZgMrfMqERqNJeCgj41qokvtMTtxcpJL3OSdFU1n8vFimPwbrnF18O55DIXybhlw8ihZm s08RLdqVOJJC5rZqqgvSJ42V7v3MpM3dTBnyWbtBQmHFnXFpLzdXOWjW9tWo2irwPpd0pWhVPBPU R0qILTVWx+y+STuYmQj7sxHH1Kctb7NuahnMflKW2akk3J/bvGlojqtpv8ZuJP8ACR8Ag9M8dtTu YdErdfEJTCZKrY/L5JSySbrbCdKmicdWRLeWlJK/bs8skq+ElSacJ8HVppr3OZiP2RuxFK4W5xi3 LK2T5D8NmY8baHHnWySSeTSa6to1K8JTumn3gtM6PbtQzDYYqnqTtsf24Zb5q4c/ZGrq/DHdwK1r rZRIyiyAAAAATlvz2z4v7PB588NXR+GW/gVbvWijRlFoADGZHjVmyO2Ltd4YOTAcUlbjJLcbJRoO qam2pCuA+GlROFxwdV6nJRT2M+q226DbLfHt8BkmIcRtLMdlNaJQgqEVTqZ/eYjKTbqwlQ5Hb4Bp Uk4zRpUpa1FoTQ1uEaVqPg7qiUZGffDmYodSbNZ0toaTBjk02hbTaCaQSUtu/wBxCSpQkr/MXfHe d4iiP12z2h5k2XYMdxlSkuKbW0hSTW2kkIUZGVKpSkiI+8RApPEURydtltdQpDsRlxCtepKm0mR8 qdXKkZfn/NxjnMxQ+kcOk7bH9uGW+auHP2Rq6vwx3cCta62USMosgAAAAE5b89s+L+zwefPDV0fh lv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACb9/3mWN4cbff WTTLUWEt1xX4UpTNeNSj+wiGtolW1LfwKt7qRuTrY226RwfSkKH1rnaz3zI4jrZ226RwfSkH1rna xmRxHWztt0jg+lIPrXO1jMjiOtnbbpHB9KQfWudrGZHEdbO23SOD6Ug+tc7WMyOJ+o3W24WtKE5F CUpR0SknCqZmH1rnazjuxW2pnbTfbPeGnHbXMamNtK0OLaVqJKjKtDELlqUNklQ7C5GfS6mOu+f4 XZpy4F0vMWHMbJKlsOrJKyJRVTUvtIdjZnJVSOuaXqfF1sbbdI4PpSEvrXO1nMyOJp7YiQxJ3oyi RHcS9Hejz3GXUcKVoVOZNKi+wyOovaxUsx3cDwtdbKNGUWgAAAADX+4GzGPZveWbtcpsuO+xGTFS iObRINCFrcIz1oWdaun3xas6qVtUSR5ztKTqeZ+lrCvely8Zj1Q9f2M8EQyEPpawr3pcvGY9UH7G eCGQh9LWFe9Ll4zHqg/YzwQyEPpawr3pcvGY9UH7GeCGQh9LWFe9Ll4zHqg/YzwQyEc2flfwxl5t 1NzuJqbUSiI1MUqX/UOr+xmn6I5LTRaaxNg4XhNuxOHIiwX3n0SXCdWb5pMyMk6eDSlPEPDU6mV5 pteg0+nVpNI8zm+xeNZfkDt7nzprEl5Dbam2FNEgibTpKmttR97jErWslCNEkTlaUnUwP0tYV70u XjMeqHp+xngiOQj023+zGPYReXrtbZsuQ+/GVFUiQbRoJC1ocMy0IQdatF3x5XtVK4qNInC0oups AVT0AAAAAAAAAAAAAAAAAAAAAAAAAAAAD//Z ------=_Part_88180576_1156318871.1500303388932 Content-Type: image/jpeg; name=element-signature_logo_YouTube_32x28.jpg Content-Disposition: attachment; filename=element-signature_logo_YouTube_32x28.jpg Content-Transfer-Encoding: base64 Content-ID: <2806a89cb9fbba6f879bd3fd54a93b515defd6d2@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOjQ1REY5Njk1OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOjQ1REY5Njk0OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAIAMBEQACEQEDEQH/xAB5AAADAQEAAAAAAAAAAAAAAAAABwgF BgEAAgMBAAAAAAAAAAAAAAAAAAQCAwUBEAABBAECBAUFAAAAAAAAAAABAgMEBQARBiESEwcxUdIU CEEik1UYEQACAQMCBQUAAAAAAAAAAAAAAQIRAwQhMVFhEiITQaHBYhX/2gAMAwEAAhEDEQA/AKpw AnD5Hbg3JA35XxKqxmxUPVrJEaI862FuqkPp4IbI5lHQDNXBhFwbaW4tebqcMIvfYjURtz6eesv1 YzWz9fYh38zR7eX2+mu5lHV3VhaNOe8aTJgTHnweVXEBba1cQRkb8IeNtJbHYN9WpW+YY2T53inQ 4HfzZk6a4lmHGaiOSHl8EoT7mSOZR+gBIzUxU3Yklz+Be4+9DRl30N3dMOxjbyro9EzHW3MqupHc L7qlaoX1SsFvlHl44moPpa6XXiW113FPuu3qrT5I7dfrJLUxllMRl15hQWjqBbiuXmTqCQlQx23F rHdSqTrNFFZlDAou7vZa43xuSNaw7CPFZZhIiKbeSsqKkOuuE/aCNNHcexspW40a9Sm5bcmcH/J1 5+0rvxOenGf0o8GQ8DN7ZXxzv9vbpqrhy0huMQJCHlstocBKU+ITqANcru50ZRao9SUbLTqPvMwv DAAwAMAP/9k= ------=_Part_88180576_1156318871.1500303388932 Content-Type: image/jpeg; name=element-signature_logo_LinkedIn_41x28.jpg Content-Disposition: attachment; filename=element-signature_logo_LinkedIn_41x28.jpg Content-Transfer-Encoding: base64 Content-ID: <878634ba055bb6ae851a7ef4540f754afa3ff86c@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOjY2MEYzRTFGOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOjY2MEYzRTFFOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAKQMBEQACEQEDEQH/xAB2AAEBAQEAAAAAAAAAAAAAAAAABggH AQEAAwEAAAAAAAAAAAAAAAAAAwQFAhAAAQMEAgICAgMBAAAAAAAAAQIDBAARBQYhEjEHMghBE1Fh IhcRAAICAQIDCQAAAAAAAAAAAAABAgMREgQxkTIhQVGhscETMxX/2gAMAwEAAhEDEQA/ANU0Bl33 zsuzQfZcmHj8tMiMKaihDLMl1psKcQkXslSUjk8mtjZ1xdeWkVbW9RDZ/ZN8wOXkYnI7LKMuL1Lq o+RddaIWkLHVwLAPCuf4NWYVwksqPkcNtd5V+ldr2ef7MwkaZmJsqK6p/uy9JdcbUBGcULpUog8i 9Q7uuKreEjquT1GsKxC2KAUBk37B2/6y72t1tB7X8dbIvf8Aq3mtvZfVzKlvUWWw7Dr+Dfy8rCt4 d153Z8ew0FNx3kohuxmUyFtAfFPyBUOBz+arwhKWE89L9SRtLmTuns4xj7LLaxYaGNE6SYojlJZ6 rhLUf19f89eyj4qa1t7ft449ziPWairGLQoBQHFvZ/onObht0jNxclFjMPNtNpZdS4Vj9aAk36gj mtDb7yMIaWiGdTbySSfqnsKbdctAFh1Fm3Bwfx8fFT/ox8GcfAyk9dfX7OapuGNzj+SiPRoJcKmG kuBRDjS2x1uAPK71DfvYzg444nUKmnk7lWcTigFAKAUAoBQCgP/Z ------=_Part_88180576_1156318871.1500303388932 Content-Type: image/jpeg; name=element-signature_logo_Twitter_42x28.jpg Content-Disposition: attachment; filename=element-signature_logo_Twitter_42x28.jpg Content-Transfer-Encoding: base64 Content-ID: <18a767bbdad6a1133fc4152b03ea78228ce25a14@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOjdDMkI3MjMyOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOjdDMkI3MjMxOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAKgMBEQACEQEDEQH/xAB3AAEBAQEBAAAAAAAAAAAAAAAABwUI BgEBAAMBAQAAAAAAAAAAAAAAAAMEBQIBEAABAwIEBwEBAQAAAAAAAAABAgMEAAUREhMGITFBFBUH CIEiFhEAAgIBAgQHAAAAAAAAAAAAAAERAgMxBCHBEgVBUWGBQhMV/9oADAMBAAIRAxEAPwDC9g7u 3dH33uJiNeri2wxcJQQ01JfCUNodVwCUqwSlI/AK3sGKropS0Kd7OWZMXcPsyZDdnQ7he5UFhQQ/ LZelOMoURiEqWlRAOFdumNOGqyeTYtPzFfL1df8ASm5XCTP0ey0e5eceyZu4zZc5VhjlGOFZ/cKJ dMKNeRNhbclzrOJxQCgOSW7o5a/oKXMRFXNT5iYw/EaAUtbL61tOBKTwXglWOXrW30zgj0RUmLlb 9p3rZ/r/ANd3GxWJtm3XC8B1EO2RsEqDkng88pAP8JSnEk8ugqlt6Xy5Fa3FImyNVUI8z8nJCU7o SOSRAA/O5qbuXx9+Rxg8ToGsssCgFAc/7q+b9xXrc11u7V3iNN3CW/JbQUuhaUuuFYBIHMA9K08e +rWqUPgivbC25MofK25Qoq81DKjwKlJeUT+kE13+jXyZ59DKX6a9WXPYfmO+msTPJdtp6AWMuhq4 5swHPVFVN1uFkiFoS46dJSqqEgoBQCgFAKAUAoBQH//Z ------=_Part_88180576_1156318871.1500303388932 Content-Type: image/jpeg; name=element-signature_payzen_61x28.jpg Content-Disposition: attachment; filename=element-signature_payzen_61x28.jpg Content-Transfer-Encoding: base64 Content-ID: <d3fc119a0aa72bf985c19ca559c993b5b4b54ca3@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOkU1NjczQjM2OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOkU1NjczQjM1OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAPQMBEQACEQEDEQH/xABvAAABBAMAAAAAAAAAAAAAAAAAAQUG BwIDBAEBAAAAAAAAAAAAAAAAAAAAABAAAQQABAQEAgsBAAAAAAAAAQIDBAUAERIGITETFEEiFQfS I1FhcYGhMlIkRMQWhhEBAAAAAAAAAAAAAAAAAAAAAP/aAAwDAQACEQMRAD8AvqZ7q7OhzZMSS9IQ Yjy2JT/avqZbW2rSvU6lBSACOeeAljLzTzSHmVhxpxIW24k5pUlQzBBHgRgM8BBJ9fKuvcGwr3La wgxIlfGeaagyVsJ1uOOBRUBmOSRgI0bnZgfQwdx7qDrroYZSVzBrcJ0hKM2xq+7AWBU7PTXTEShc 2svRx6MuWp5o5gjigj68BombFRKmPyTfXTPXWpzoszloaRqOelCQPKkeAwEP6lp6f6V6vYaP9Z6b 3fcr7ntuhno6vPnxwGyohb2ntbpr6Y17NdKt7Bp5+YHlPJLitKyltI0KGk+XPANVgzV1Eyyqbe3s IMmnhsMbVaYcebDraY+XVbQ15XXFOghQVnkMBkbjcVPTUs6M5IkPbnpWa1hKlrWG7QZBp0hROnUh xRUfHTgJDWuwtsbptVTpC1xaqhgJfkuFTji9DjoKiTmpSlq/HAcO2d11d/cuXUySJG4C0+NvbfCH CmK2hClHUop0GQ6E+dWfAeUYBi2lY2E+zgW7twhVlHDk+7AkzFO9u2lRdjORFIEZrTmAkA+GYwDl TwLJhjZFouxsF3NzN6r7Lj7imExHUOSHWi0TpyCdORIzzwHT/J/7n+vgLVAA5YBChBUlRSCpOelR HEZ88sA2WG3YVhb1lnJW4VVJcXFjAgM9VxOjqKTlmVJTnp48M8Ax2+3N3J3TJu6KVAbRKisxnWpr brh+SpasxoKf14BBD91hxE2kB+nt5Px4AEL3VGoiZSAr4qyjyOP2+fAL2nuvw/e0vDl8iT8eAbv8 Lur0vPvYXrPrfrfU6bvb59Lp6NOev83HngLBwBgDAGAMAYAwBgP/2Q== ------=_Part_88180576_1156318871.1500303388932 Content-Type: image/jpeg; name=element-signature_logo_lyra_115x94.jpg Content-Disposition: attachment; filename=element-signature_logo_lyra_115x94.jpg Content-Transfer-Encoding: base64 Content-ID: <ac318af44fd2ff936fd390a2dfa5ac7f8bf624d9@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOkU1QUU4Nzk1OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOkU1QUU4Nzk0OEExMzExRTZBNTFCRjVENEYyNDYxNTQ1IiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIAF4AcwMBEQACEQEDEQH/xACdAAEBAAIDAQEAAAAAAAAAAAAACAUH AgMGBAEBAQADAQEAAAAAAAAAAAAAAAACBAUBAxAAAAUDAgAHDQcCBwAAAAAAAAECAwQRBQYSByFR E7RWFzcxQSKS0nOT03RVlQgYYXEyQiMUFYEzsVJicmODOBEAAgECAwcEAgIDAAAAAAAAAAECEQNR EwQhgbESMlIzMUGhFHEVwQUiQzT/2gAMAwEAAhEDEQA/AKpAE9b/AGQZdF3FtNpsd4l29M2DHSll iS6w0bzsp5slqJCiLvERnTuENTRQg7bckntK15vmojj1WfMP0rc+Jy/JD7Fjt+EMueJ6bbnBN4bR lkWdkt+XPtDaHSfjKnSHyUpTZkg+TcLSdFGRjyv3rUo0iqP8EoQkntZ7fcOy5VdrXGYxuccGW2+S 3nCecYq3oUWnU2RmfhGXAPPSXLcJNzVVT8kNVbnKKUHRmppW1/zBOSXVtZS4lpSjNtP8nLKhfcSR alqdPXp+ERhauKKTe38nX1WfMP0qc+Jy/JHPsWO34RPLniOqz5h+lTnxOX5IfYsdvwhlzxHVZ8w/ Spz4nL8kPsWO34Qy54jqs+YfpU58Tl+SH2LHb8IZc8R1WfMP0qc+Jy/JD7Fjt+EMueJ2bCX3MH9y LxZr7eJdxKBCkoW0/JdfaJ5mUy2a0EszL/MRHTuGGshDLTiqVf8AAst81GUGMssgAAAAE5b89s+L +zwefPDV0fhlv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACc t+e2fF/Z4PPnhq6Pwy38Crd60UaMotAAAAAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLI AAAABOW/PbPi/s8Hnzw1dH4Zb+BVu9aKNGUWjAKzrGk5cnEjkq/nVo5RMbk3NOnkzd/uU0fgKvdH rky5eb2I86rQz48iQAAAAAAAAABO2x/bhlvmrhz9kaur8Md3ArWutlEjKLIAAAABOW/PbPi/s8Hn zw1dH4Zb+BVu9aKNGUWjSD//AKkj+xnzJQ0V/wA2/wDkr/7D3e7GX5DiOMfzlniMTEMPIRORIJfg tOeCladCk/nNJH94raa1GcuVnpck4qqOrJdz4dr2yazKOhDjkxhlUGMszop98i/TOlD/AE/CNX+0 x23p27nIJXKRqeeyjdnLMd27sGSzLdEO4Xd4uWifqkhDDiFut0qrUSzQlNa9wz7g9bemjK44puiI yuNRTOzGs53hud+gvzsYjwMVnqNfLuqUTrEck69bitZmStPc1NJIz4hy5ZtKLpKskIzk36bDHRt2 9yMsuMw9v8ejybNBc5M5s5Rp5Uy4aEZuMJSai4dPCZFStBN6a3BLne1nMyT6Uffgu8GQ5DnysXuN mbtZx461TG1ms3kyGiLWRHXToMz8Hg7nfMQvaWMYcydTsLjcqG2BSPYnbY/twy3zVw5+yNXV+GO7 gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLROmdZbAxL5g/5yey6/GjREJW2wSTc M3YpoKms0F3VcY1bNpzscqKs5UnU9pad4cG3Fefw1UWXFO7x3mULkpaJJnoM6JNC10WREak/aXGK 8tLO1/ns2HorilsNTYzar/kOQWTa+5tmmBjk+W9cKGqimkrJS68RcCkoV/yC7clGMXcXrJI8Yptq OBsj5oEpTiNmSRESSuKSJJFwUJlzgoKv9f1P8Hrf9Db1wh/vbVJh6tH7lhbOvi5RBpr/AEqKMXR1 PZrYaC2x3Jgbawp+IZjBkwpkWS480603rSslkRU7qTOuiqFlUlEfepw6WosO61ODK9ufLsZ27f36 RkG/8q9OwXbe3PgKciMPp0uHHS22hpxRf60o1f4VC9DlsUrWjEHWdSghllknbY/twy3zVw5+yNXV +GO7gVrXWyiRlFkAAAAAnLfntnxf2eDz54auj8Mt/Aq3etFGjKLRxU00o6qQlR8ZkRhUGMyO7W7H 7FOvUpCSZgMrfMqERqNJeCgj41qokvtMTtxcpJL3OSdFU1n8vFimPwbrnF18O55DIXybhlw8ihZm s08RLdqVOJJC5rZqqgvSJ42V7v3MpM3dTBnyWbtBQmHFnXFpLzdXOWjW9tWo2irwPpd0pWhVPBPU R0qILTVWx+y+STuYmQj7sxHH1Kctb7NuahnMflKW2akk3J/bvGlojqtpv8ZuJP8ACR8Ag9M8dtTu YdErdfEJTCZKrY/L5JSySbrbCdKmicdWRLeWlJK/bs8skq+ElSacJ8HVppr3OZiP2RuxFK4W5xi3 LK2T5D8NmY8baHHnWySSeTSa6to1K8JTumn3gtM6PbtQzDYYqnqTtsf24Zb5q4c/ZGrq/DHdwK1r rZRIyiyAAAAATlvz2z4v7PB588NXR+GW/gVbvWijRlFoADGZHjVmyO2Ltd4YOTAcUlbjJLcbJRoO qam2pCuA+GlROFxwdV6nJRT2M+q226DbLfHt8BkmIcRtLMdlNaJQgqEVTqZ/eYjKTbqwlQ5Hb4Bp Uk4zRpUpa1FoTQ1uEaVqPg7qiUZGffDmYodSbNZ0toaTBjk02hbTaCaQSUtu/wBxCSpQkr/MXfHe d4iiP12z2h5k2XYMdxlSkuKbW0hSTW2kkIUZGVKpSkiI+8RApPEURydtltdQpDsRlxCtepKm0mR8 qdXKkZfn/NxjnMxQ+kcOk7bH9uGW+auHP2Rq6vwx3cCta62USMosgAAAAE5b89s+L+zwefPDV0fh lv4FW71oo0ZRaAAAAAAAAAAAAACdtj+3DLfNXDn7I1dX4Y7uBWtdbKJGUWQAAAACb9/3mWN4cbff WTTLUWEt1xX4UpTNeNSj+wiGtolW1LfwKt7qRuTrY226RwfSkKH1rnaz3zI4jrZ226RwfSkH1rna xmRxHWztt0jg+lIPrXO1jMjiOtnbbpHB9KQfWudrGZHEdbO23SOD6Ug+tc7WMyOJ+o3W24WtKE5F CUpR0SknCqZmH1rnazjuxW2pnbTfbPeGnHbXMamNtK0OLaVqJKjKtDELlqUNklQ7C5GfS6mOu+f4 XZpy4F0vMWHMbJKlsOrJKyJRVTUvtIdjZnJVSOuaXqfF1sbbdI4PpSEvrXO1nMyOJp7YiQxJ3oyi RHcS9Hejz3GXUcKVoVOZNKi+wyOovaxUsx3cDwtdbKNGUWgAAAADX+4GzGPZveWbtcpsuO+xGTFS iObRINCFrcIz1oWdaun3xas6qVtUSR5ztKTqeZ+lrCvely8Zj1Q9f2M8EQyEPpawr3pcvGY9UH7G eCGQh9LWFe9Ll4zHqg/YzwQyEPpawr3pcvGY9UH7GeCGQh9LWFe9Ll4zHqg/YzwQyEc2flfwxl5t 1NzuJqbUSiI1MUqX/UOr+xmn6I5LTRaaxNg4XhNuxOHIiwX3n0SXCdWb5pMyMk6eDSlPEPDU6mV5 pteg0+nVpNI8zm+xeNZfkDt7nzprEl5Dbam2FNEgibTpKmttR97jErWslCNEkTlaUnUwP0tYV70u XjMeqHp+xngiOQj023+zGPYReXrtbZsuQ+/GVFUiQbRoJC1ocMy0IQdatF3x5XtVK4qNInC0oups AVT0AAAAAAAAAAAAAAAAAAAAAAAAAAAAD//Z ------=_Part_88180576_1156318871.1500303388932 Content-Type: image/jpeg; name=element-signature_logo_YouTube_32x28.jpg Content-Disposition: attachment; filename=element-signature_logo_YouTube_32x28.jpg Content-Transfer-Encoding: base64 Content-ID: <6f3cc57ba362f7a2942a16e88d1ea9ce686c7da2@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOjQ1REY5Njk1OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOjQ1REY5Njk0OEExNzExRTY5OUVDOTI1QTU3QzAwNzI0IiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAIAMBEQACEQEDEQH/xAB5AAADAQEAAAAAAAAAAAAAAAAABwgF BgEAAgMBAAAAAAAAAAAAAAAAAAQCAwUBEAABBAECBAUFAAAAAAAAAAABAgMEBQARBiESEwcxUdIU CEEik1UYEQACAQMCBQUAAAAAAAAAAAAAAQIRAwQhMVFhEiITQaHBYhX/2gAMAwEAAhEDEQA/AKpw AnD5Hbg3JA35XxKqxmxUPVrJEaI862FuqkPp4IbI5lHQDNXBhFwbaW4tebqcMIvfYjURtz6eesv1 YzWz9fYh38zR7eX2+mu5lHV3VhaNOe8aTJgTHnweVXEBba1cQRkb8IeNtJbHYN9WpW+YY2T53inQ 4HfzZk6a4lmHGaiOSHl8EoT7mSOZR+gBIzUxU3Yklz+Be4+9DRl30N3dMOxjbyro9EzHW3MqupHc L7qlaoX1SsFvlHl44moPpa6XXiW113FPuu3qrT5I7dfrJLUxllMRl15hQWjqBbiuXmTqCQlQx23F rHdSqTrNFFZlDAou7vZa43xuSNaw7CPFZZhIiKbeSsqKkOuuE/aCNNHcexspW40a9Sm5bcmcH/J1 5+0rvxOenGf0o8GQ8DN7ZXxzv9vbpqrhy0huMQJCHlstocBKU+ITqANcru50ZRao9SUbLTqPvMwv DAAwAMAP/9k= ------=_Part_88180576_1156318871.1500303388932 Content-Type: image/jpeg; name=element-signature_logo_LinkedIn_41x28.jpg Content-Disposition: attachment; filename=element-signature_logo_LinkedIn_41x28.jpg Content-Transfer-Encoding: base64 Content-ID: <95ac2affdb66bc26fd0e254578e26d8574139b42@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOjY2MEYzRTFGOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOjY2MEYzRTFFOEExNzExRTY5QzI5OUNGQkY1MzM2MUQ1IiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAKQMBEQACEQEDEQH/xAB2AAEBAQEAAAAAAAAAAAAAAAAABggH AQEAAwEAAAAAAAAAAAAAAAAAAwQFAhAAAQMEAgICAgMBAAAAAAAAAQIDBAARBQYhEjEHMghBE1Fh IhcRAAICAQIDCQAAAAAAAAAAAAABAgMREgQxkTIhQVGhscETMxX/2gAMAwEAAhEDEQA/ANU0Bl33 zsuzQfZcmHj8tMiMKaihDLMl1psKcQkXslSUjk8mtjZ1xdeWkVbW9RDZ/ZN8wOXkYnI7LKMuL1Lq o+RddaIWkLHVwLAPCuf4NWYVwksqPkcNtd5V+ldr2ef7MwkaZmJsqK6p/uy9JdcbUBGcULpUog8i 9Q7uuKreEjquT1GsKxC2KAUBk37B2/6y72t1tB7X8dbIvf8Aq3mtvZfVzKlvUWWw7Dr+Dfy8rCt4 d153Z8ew0FNx3kohuxmUyFtAfFPyBUOBz+arwhKWE89L9SRtLmTuns4xj7LLaxYaGNE6SYojlJZ6 rhLUf19f89eyj4qa1t7ft449ziPWairGLQoBQHFvZ/onObht0jNxclFjMPNtNpZdS4Vj9aAk36gj mtDb7yMIaWiGdTbySSfqnsKbdctAFh1Fm3Bwfx8fFT/ox8GcfAyk9dfX7OapuGNzj+SiPRoJcKmG kuBRDjS2x1uAPK71DfvYzg444nUKmnk7lWcTigFAKAUAoBQCgP/Z ------=_Part_88180576_1156318871.1500303388932 Content-Type: image/jpeg; name=element-signature_logo_Twitter_42x28.jpg Content-Disposition: attachment; filename=element-signature_logo_Twitter_42x28.jpg Content-Transfer-Encoding: base64 Content-ID: <5e9e243634081dcf099c93fd9ca78f8b346f57b3@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOjdDMkI3MjMyOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOjdDMkI3MjMxOEExNzExRTY5MDUyRTk1NEIyREUwNjVCIiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAKgMBEQACEQEDEQH/xAB3AAEBAQEBAAAAAAAAAAAAAAAABwUI BgEBAAMBAQAAAAAAAAAAAAAAAAMEBQIBEAABAwIEBwEBAQAAAAAAAAABAgMEAAUREhMGITFBFBUH CIEiFhEAAgIBAgQHAAAAAAAAAAAAAAERAgMxBCHBEgVBUWGBQhMV/9oADAMBAAIRAxEAPwDC9g7u 3dH33uJiNeri2wxcJQQ01JfCUNodVwCUqwSlI/AK3sGKropS0Kd7OWZMXcPsyZDdnQ7he5UFhQQ/ LZelOMoURiEqWlRAOFdumNOGqyeTYtPzFfL1df8ASm5XCTP0ey0e5eceyZu4zZc5VhjlGOFZ/cKJ dMKNeRNhbclzrOJxQCgOSW7o5a/oKXMRFXNT5iYw/EaAUtbL61tOBKTwXglWOXrW30zgj0RUmLlb 9p3rZ/r/ANd3GxWJtm3XC8B1EO2RsEqDkng88pAP8JSnEk8ugqlt6Xy5Fa3FImyNVUI8z8nJCU7o SOSRAA/O5qbuXx9+Rxg8ToGsssCgFAc/7q+b9xXrc11u7V3iNN3CW/JbQUuhaUuuFYBIHMA9K08e +rWqUPgivbC25MofK25Qoq81DKjwKlJeUT+kE13+jXyZ59DKX6a9WXPYfmO+msTPJdtp6AWMuhq4 5swHPVFVN1uFkiFoS46dJSqqEgoBQCgFAKAUAoBQH//Z ------=_Part_88180576_1156318871.1500303388932 Content-Type: image/jpeg; name=element-signature_payzen_61x28.jpg Content-Disposition: attachment; filename=element-signature_payzen_61x28.jpg Content-Transfer-Encoding: base64 Content-ID: <72acbdf9e9761a33ad68ac239cb304da61f1cda1@zimbra> /9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QOBaHR0cDov L25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENl aGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4 OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjYtYzEzMiA3OS4xNTkyODQsIDIwMTYvMDQvMTktMTM6 MTM6NDAgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5 OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHht bG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIiB4bWxuczpzdFJlZj0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL3NUeXBlL1Jlc291cmNlUmVmIyIgeG1sbnM6eG1w PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9 InhtcC5kaWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1LTgxYmQtMTYwMTFjZjk5YWVjIiB4bXBNTTpEb2N1 bWVudElEPSJ4bXAuZGlkOkU1NjczQjM2OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXBNTTpJ bnN0YW5jZUlEPSJ4bXAuaWlkOkU1NjczQjM1OEEzNDExRTZCOEJBQ0Y4Mzg3RTEzODkyIiB4bXA6 Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE1LjUgKFdpbmRvd3MpIj4gPHhtcE1N OkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6ZGQwZjkyYzUtNzhhZi1jZTQ1 LTgxYmQtMTYwMTFjZjk5YWVjIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOmRkMGY5MmM1LTc4 YWYtY2U0NS04MWJkLTE2MDExY2Y5OWFlYyIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRG PiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAYE BAQFBAYFBQYJBgUGCQsIBgYICwwKCgsKCgwQDAwMDAwMEAwODxAPDgwTExQUExMcGxsbHB8fHx8f Hx8fHx8BBwcHDQwNGBAQGBoVERUaHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f Hx8fHx8fHx8fHx8fH//AABEIABwAPQMBEQACEQEDEQH/xABvAAABBAMAAAAAAAAAAAAAAAAAAQUG BwIDBAEBAAAAAAAAAAAAAAAAAAAAABAAAQQABAQEAgsBAAAAAAAAAQIDBAUAERIGITETFEEiFQfS I1FhcYGhMlIkRMQWhhEBAAAAAAAAAAAAAAAAAAAAAP/aAAwDAQACEQMRAD8AvqZ7q7OhzZMSS9IQ Yjy2JT/avqZbW2rSvU6lBSACOeeAljLzTzSHmVhxpxIW24k5pUlQzBBHgRgM8BBJ9fKuvcGwr3La wgxIlfGeaagyVsJ1uOOBRUBmOSRgI0bnZgfQwdx7qDrroYZSVzBrcJ0hKM2xq+7AWBU7PTXTEShc 2svRx6MuWp5o5gjigj68BombFRKmPyTfXTPXWpzoszloaRqOelCQPKkeAwEP6lp6f6V6vYaP9Z6b 3fcr7ntuhno6vPnxwGyohb2ntbpr6Y17NdKt7Bp5+YHlPJLitKyltI0KGk+XPANVgzV1Eyyqbe3s IMmnhsMbVaYcebDraY+XVbQ15XXFOghQVnkMBkbjcVPTUs6M5IkPbnpWa1hKlrWG7QZBp0hROnUh xRUfHTgJDWuwtsbptVTpC1xaqhgJfkuFTji9DjoKiTmpSlq/HAcO2d11d/cuXUySJG4C0+NvbfCH CmK2hClHUop0GQ6E+dWfAeUYBi2lY2E+zgW7twhVlHDk+7AkzFO9u2lRdjORFIEZrTmAkA+GYwDl TwLJhjZFouxsF3NzN6r7Lj7imExHUOSHWi0TpyCdORIzzwHT/J/7n+vgLVAA5YBChBUlRSCpOelR HEZ88sA2WG3YVhb1lnJW4VVJcXFjAgM9VxOjqKTlmVJTnp48M8Ax2+3N3J3TJu6KVAbRKisxnWpr brh+SpasxoKf14BBD91hxE2kB+nt5Px4AEL3VGoiZSAr4qyjyOP2+fAL2nuvw/e0vDl8iT8eAbv8 Lur0vPvYXrPrfrfU6bvb59Lp6NOev83HngLBwBgDAGAMAYAwBgP/2Q== ------=_Part_88180576_1156318871.1500303388932-- ------=_Part_88180575_227580591.1500303388932--
On Mon, Jul 17, 2017 at 10:56 AM, Nelson Lameiras < nelson.lameiras@lyra-network.com> wrote:
Hello, Can any one please help us with the problem described below?
Nir, I'm including you since a quick search on the internet led me to think that you have worked on this part of the project. Please forgive me if I'm off topic.
(I incorrectly used below the expression "patch" when I meant "configure". it's corrected now)
VDSM may indeed change the IP filter. From the function that sets it[1]: def setRpFilterIfNeeded(netIfaceName, hostname, loose_mode): """ Set rp_filter to loose or strict mode if there's no session using the netIfaceName device and it's not the device used by the OS to reach the 'hostname'. loose mode is needed to allow multiple iSCSI connections in a multiple NIC per subnet configuration. strict mode is needed to avoid the security breach where an untrusted VM can DoS the host by sending it packets with spoofed random sources. Arguments: netIfaceName: the device used by the iSCSI session target: iSCSI target object cointaining the portal hostname loose_mode: boolean I think it sets it to strict mode when disconnecting or removing an iSCSI session. Perhaps something in the check we are doing is incorrect? Do you have other sessions open? Y. [1] https://github.com/oVirt/vdsm/blob/321233bea649fb6d1e72baa1b1164c8c1bc852af/...
cordialement, regards,
<https://www.lyra-network.com/> Nelson LAMEIRAS Ingénieur Systèmes et Réseaux / Systems and Networks engineer Tel: +33 5 32 09 09 70 <+33%205%2032%2009%2009%2070> nelson.lameiras@lyra-network.com www.lyra-network.com | www.payzen.eu <https://payzen.eu> <https://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA> <https://www.linkedin.com/company/lyra-network_2> <https://twitter.com/LyraNetwork> <https://payzen.eu> ------------------------------ Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE
------------------------------ *De: *"Nelson Lameiras" <nelson.lameiras@lyra-network.com> *À: *"ovirt users" <users@ovirt.org> *Envoyé: *Mercredi 7 Juin 2017 14:59:48 *Objet: *[ovirt-users] ISCSI storage with multiple nics on same subnet disabled on host activation
Hello,
In our oVirt hosts, we are using DELL equallogic SAN with each server connecting to SAN via 2 physical interfaces. Since both interfaces share the same network (Equalogic limitation) we must configure sysctl to to allow iSCSI multipath with multiple NICs in the same subnet :
------------------------------------------------------------ --------------------------------
net.ipv4.conf.p2p1.arp_ignore=1 net.ipv4.conf.p2p1.arp_announce=2 net.ipv4.conf.p2p1.rp_filter=2
net.ipv4.conf.p2p2.arp_ignore=1 net.ipv4.conf.p2p2.arp_announce=2 net.ipv4.conf.p2p2.rp_filter=2
------------------------------------------------------------ --------------------------------
This works great in most setups, but for a strange reason, on some of our setups, the sysctl configuration is updated by VDSM when activating a host and the second interface stops working immeadiatly : ------------------------------------------------------------ -------------------------------- vdsm.log
2017-06-07 11:51:51,063+0200 INFO (jsonrpc/5) [storage.ISCSI] Setting strict mode rp_filter for device 'p2p2'. (iscsi:602) 2017-06-07 11:51:51,064+0200 ERROR (jsonrpc/5) [storage.HSM] Could not connect to storageServer (hsm:2392) Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 2389, in connectStorageServer conObj.connect() File "/usr/share/vdsm/storage/storageServer.py", line 433, in connect iscsi.addIscsiNode(self._iface, self._target, self._cred) File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 232, in addIscsiNode iscsiadm.node_login(iface.name, target.address, target.iqn) File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 337, in node_login raise IscsiNodeError(rc, out, err)
------------------------------------------------------------ --------------------------------
"strict mode" is enforced for second interface, and it no longuer works... Which means - at least - that there is no redundancy in case of hardware faillure and this is not acceptable for our production needs.
What is really strange is that we have another "twin" site on another geographic region with simillar hardware configuration and same oVirt installation, and this problem does not happen. Can this be really random? What can be the root cause of this behaviour? How can I correct it?
our setup: oVirt hostedEngine : Centor 7.3, ovirt 4.1.2 3 physical oVirt nodes centos 7.3, ovirt 4.1.2 SAN DELL Equalogic
cordialement, regards,
<https://www.lyra-network.com/> Nelson LAMEIRAS Ingénieur Systèmes et Réseaux / Systems and Networks engineer Tel: +33 5 32 09 09 70 <+33%205%2032%2009%2009%2070> nelson.lameiras@lyra-network.com www.lyra-network.com | www.payzen.eu <https://payzen.eu> <https://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA> <https://www.linkedin.com/company/lyra-network_2> <https://twitter.com/LyraNetwork> <https://payzen.eu> ------------------------------ Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (3)
-
Gianluca Cecchi -
Nelson Lameiras -
Yaniv Kaul