4:48 p.m.
--_004_CY4PR14MB1687A36A14A995B3A6DA5BABE9B10CY4PR14MB1687namp_
Content-Type: multipart/alternative;
boundary="_000_CY4PR14MB1687A36A14A995B3A6DA5BABE9B10CY4PR14MB1687namp_"
--_000_CY4PR14MB1687A36A14A995B3A6DA5BABE9B10CY4PR14MB1687namp_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Hello again,
Just to provide an update for future reference, I've been unable to sort th=
is problem out.
However, Ovirt4 does not show this failure.
My current approach to move VMs from one installation to another is:
1) Shutdown VM in ovirt3.
2) Maintenance VM storage
3) Clone VM storage at LUN level
4) Import cloned storage in ovirt4
5) Import VM in ovirt4
If everything is ok:
6a) detach + remove original VM storage in ovirt3
7a) remove mappings for original VM storage LUN in ovirt3; this im=
ply cluster instability (sanlock crash on hosts);
8a) recover cluster; this implies shutting down host (and its VM) =
one by one
9a) delete unmapped original LUN from storage manager
When Sanlock freeze on host, the only solution found so far is to reboot it=
. This has to be done on all host experiencing this problem (which basicall=
y means: all hosts in the cluster)
Cheers
AG
From: Andrea Ghelardi
Sent: Tuesday, November 08, 2016 16:40
To: users(a)ovirt.org
Subject: [HELP] unmapping a deleted storage domain triggers crash ovirt
Hello people,
something's not right in my ovirt infrastructure.
I currently have two different ovirt installation:
Ovirt3: 7 hosts linked to Compellent iscsi storage running ovirt 3.5.6
Ovirt4: 4 hosts linked to (same) Compellent iscsi storage running ovirt 4.0=
.4
I'm currently moving my guests from ovirt3 to ovirt4.
Since iscsi storage is linked to both installations, my high level approach=
is:
1) Shutdown VM in ovirt3.
2) Maintenance + detach + remove VM storage in ovirt3
3) Change LUN mapping via iscsi storage manager from ovirt3 to ovirt4
4) Import storage in ovirt4
5) Import VM in ovirt4
6) Run and cheers with high grade liquor. GOTO step 1 for different VM=
.
Now, as soon as I perform step 3 (remove mappings from LUN), ovirt3 goes cr=
azy and eventually forces me to reboot all hosts one by one.
I tried different low level approaches to unmap LUN mpath'ed from hosts wit=
h inconsistent results.
A notable error log extract is:
Nov 8 15:30:52 sovana vdsm root ERROR Process failed with rc=3D1 out=3D'\n=
udevadm settle - timeout of 5 seconds reached, the event queue contains:\n =
/sys/devices/virtual/block/dm-39 (8603)\n /sys/devices/virtual/block/dm-3=
9 (8604)\n /sys
/devices/virtual/block/dm-39 (8605)\n /sys/devices/virtual/block/dm-39 (86=
06)\n /sys/devices/virtual/block/dm-39 (8607)\n /sys/devices/virtual/bloc=
k/dm-39 (8608)\n /sys/devices/virtual/block/dm-39 (8609)\n /sys/devices/v=
irtual/block
/dm-39 (8610)\n /sys/devices/virtual/block/dm-39 (8611)\n /sys/devices/vi=
rtual/block/dm-39 (8612)\n /sys/devices/virtual/block/dm-39 (8613)\n /sys=
/devices/virtual/block/dm-39 (8614)\n /sys/devices/virtual/block/dm-39 (86=
15)\n /sys/
devices/virtual/block/dm-39 (8616)\n /sys/devices/virtual/block/dm-39 (861=
7)\n /sys/devices/virtual/block/dm-39 (8618)\n /sys/devices/virtual/block=
/dm-39 (8619)\n /sys/devices/virtual/block/dm-39 (8620)\n /sys/devices/vi=
rtual/block/
dm-39 (8621)\n /sys/devices/virtual/block/dm-39 (8622)\n /sys/devices/vir=
tual/block/dm-39 (8623)\n /sys/devices/virtual/block/dm-39 (8624)\n /sys/=
devices/virtual/block/dm-39 (8625)\n /sys/devices/virtual/block/dm-39 (862=
6)\n /sys/d
evices/virtual/block/dm-39 (8627)\n /sys/devices/virtual/block/dm-39 (8628=
)\n /sys/devices/virtual/block/dm-39 (8629)\n /sys/devices/virtual/block/=
dm-39 (8630)\n /sys/devices/virtual/block/dm-39 (8631)\n /sys/devices/vir=
tual/block/d
m-39 (8632)\n /sys/devices/virtual/block/dm-39 (8633)\n /sys/devices/virt=
ual/block/dm-39 (8634)\n /sys/devices/virtual/block/dm-39 (8635)\n /sys/d=
evices/virtual/block/dm-39 (8636)\n /sys/devices/virtual/block/dm-39 (8637=
)\n /sys/de
vices/virtual/block/dm-39 (8638)\n /sys/devices/virtual/block/dm-39 (8639)=
\n /sys/devices/virtual/block/dm-39 (8640)\n /sys/devices/virtual/block/d=
m-39 (8641)\n /sys/devices/virtual/block/dm-39 (8642)\n /sys/devices/virt=
ual/block/dm
-39 (8643)\n /sys/devices/virtual/block/dm-39 (8644)\n /sys/devices/virtu=
al/block/dm-39 (8645)\n /sys/devices/virtual/block/dm-39 (8646)\n /sys/de=
vices/virtual/block/d
I really need your help to sort this out as I'm actually blocked in my task=
.
Why mapping changes triggers ovirt crash on a storage which ovirt should no=
t care about?
Thanks
Andrea Ghelardi
+39 050 2203 71 | www.iongroup.com<http://www.iongroup.com/> | a.ghelardi@i=
ontrading.com<mailto:a.ghelardi@iontrading.com Via San
Martino, 52 - 56125 Pisa - ITALY
This email and any attachments may contain information which is confidentia=
l and/or privileged. The information is intended exclusively for the addres=
see and the views expressed may not be official policy, but the personal vi=
ews of the originator. If you are not the intended recipient, be aware that=
any disclosure, copying, distribution or use of the contents is prohibited=
. If you have received this email and any file transmitted with it in error=
, please notify the sender by telephone or return email immediately and del=
ete the material from your computer. Internet communications are not secure=
and ION Trading is not responsible for their abuse by third parties, nor f=
or any alteration or corruption in transmission, nor for any damage or loss=
caused by any virus or other defect. ION Trading accepts no liability or r=
esponsibility arising out of or in any way connected to this email.
[iON_HBlu_small]
Automation through innovation
--_000_CY4PR14MB1687A36A14A995B3A6DA5BABE9B10CY4PR14MB1687namp_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml"
xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word"
=
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml"
xmlns=3D"http:=
//www.w3.org/TR/REC-html40" <head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3Dus-ascii"=
<meta name=3D"Generator" content=3D"Microsoft
Word 15 (filtered medium)" <!--[if
!mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:ION;
panose-1:2 0 5 5 0 0 0 2 0 4;}
@font-face
{font-family:"Source Sans Pro";
panose-1:2 11 5 3 3 4 3 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1446727994;
mso-list-type:hybrid;
mso-list-template-ids:438582106 67698705 67698713 67698715 67698703 676987=
13 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" / </xml><![endif]--><!--[if gte mso
9]><xml <o:shapelayout
v:ext=3D"edit" <o:idmap v:ext=3D"edit"
data=3D"1" /
</o:shapelayout></xml><![endif]--
</head <body lang=3D"EN-US"
link=3D"#0563C1" vlink=3D"#954F72" <div
class=3D"WordSection1" <p
class=3D"MsoNormal">Hello again,<o:p></o:p></p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal">Just to provide an
update for future reference, I=
217;ve been unable to sort this problem out.<o:p></o:p></p <p class=3D"MsoNormal">However, Ovirt4 does not
show this failure.<o:p></o:=
p></p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal">My current approach to
move VMs from one installatio=
n to another is:<o:p></o:p></p <p
class=3D"MsoNormal">1) &n=
bsp; Shutdown VM in ovirt3.<o:p></o:p></p <p
class=3D"MsoNormal">2) &n=
bsp; Maintenance VM storage<o:p></o:p></p <p
class=3D"MsoNormal">3) &n=
bsp; Clone VM storage at LUN level<o:p></o:p></p <p
class=3D"MsoNormal">4) &n=
bsp; Import cloned storage in ovirt4<o:p></o:p></p <p
class=3D"MsoNormal">5) &n=
bsp; Import VM in ovirt4<o:p></o:p></p <p class=3D"MsoNormal">If everything is ok:
<o:p></o:p></p <p
class=3D"MsoNormal">6a) &=
nbsp; detach + remove original VM storage in
ovirt3<o:p></o:p></p <p
class=3D"MsoNormal">7a) &=
nbsp; remove mappings for original VM storage LUN in ovirt3; this imply clu=
ster instability (sanlock crash on hosts);
<o:p></o:p></p <p
class=3D"MsoNormal">8a)
=
recover cluster; this implies shutting down host (and its VM) one by one<o=
:p></o:p></p <p
class=3D"MsoNormal">9a)
=
delete unmapped original LUN from storage manager<o:p></o:p></p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal">When Sanlock freeze on
host, the only solution found=
so far is to reboot it. This has to be done on all host experiencing this =
problem (which basically means: all hosts in the cluster)<o:p></o:p></p <p
class=3D"MsoNormal"><o:p> </o:p></p <p
class=3D"MsoNormal">Cheers<o:p></o:p></p <p
class=3D"MsoNormal">AG<o:p></o:p></p <p
class=3D"MsoNormal"><o:p> </o:p></p <p
class=3D"MsoNormal"><o:p> </o:p></p <div <div
style=3D"border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm =
0cm 0cm" <p
class=3D"MsoNormal"><b>From:</b> Andrea Ghelardi <br <b>Sent:</b> Tuesday, November 08, 2016
16:40<br <b>To:</b>
users(a)ovirt.org<br <b>Subject:</b> [HELP]
unmapping a deleted storage domain triggers crash ov=
irt<o:p></o:p></p </div </div <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal"><span
lang=3D"IT">Hello people,<o:p></o:p></span></p=
<p class=3D"MsoNormal"><span
lang=3D"IT"><o:p> </o:p></span></p <p class=3D"MsoNormal">something’s
not right in my ovirt infrastructu=
re.<o:p></o:p></p <p
class=3D"MsoNormal">I currently have two different ovirt installation:<o=
:p></o:p></p <p
class=3D"MsoNormal">Ovirt3: 7 hosts linked to Compellent iscsi storage r=
unning ovirt 3.5.6<o:p></o:p></p <p
class=3D"MsoNormal">Ovirt4: 4 hosts linked to (same) Compellent iscsi st=
orage running ovirt 4.0.4<o:p></o:p></p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal">I’m currently
moving my guests from ovirt3 to =
ovirt4.<o:p></o:p></p <p
class=3D"MsoNormal">Since iscsi storage is linked to both installations,=
my high level approach is:<o:p></o:p></p <p
class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve=
l1 lfo2"><![if !supportLists]><span
style=3D"mso-list:Ignore">1)<span style=
=3D"font:7.0pt "Times New
Roman"">
</span></span><![endif]>Shutdown VM in
ovirt3.<o:p></o:p></p <p
class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve=
l1 lfo2"><![if !supportLists]><span
style=3D"mso-list:Ignore">2)<span style=
=3D"font:7.0pt "Times New
Roman"">
</span></span><![endif]>Maintenance + detach + remove VM
storage in=
ovirt3<o:p></o:p></p <p
class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve=
l1 lfo2"><![if !supportLists]><span
style=3D"mso-list:Ignore">3)<span style=
=3D"font:7.0pt "Times New
Roman"">
</span></span><![endif]>Change LUN mapping via iscsi storage manager
from o=
virt3 to ovirt4<o:p></o:p></p <p
class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve=
l1 lfo2"><![if !supportLists]><span
style=3D"mso-list:Ignore">4)<span style=
=3D"font:7.0pt "Times New
Roman"">
</span></span><![endif]>Import storage in
ovirt4<o:p></o:p></p <p
class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve=
l1 lfo2"><![if !supportLists]><span
style=3D"mso-list:Ignore">5)<span style=
=3D"font:7.0pt "Times New
Roman"">
</span></span><![endif]>Import VM in
ovirt4<o:p></o:p></p <p
class=3D"MsoListParagraph" style=3D"text-indent:-18.0pt;mso-list:l0 leve=
l1 lfo2"><![if !supportLists]><span
style=3D"mso-list:Ignore">6)<span style=
=3D"font:7.0pt "Times New
Roman"">
</span></span><![endif]>Run and cheers with high grade liquor. GOTO step
1 =
for different VM.<o:p></o:p></p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal">Now, as soon as I
perform step 3 (remove mappings fr=
om LUN), ovirt3 goes crazy and eventually forces me to reboot all hosts one=
by one.<o:p></o:p></p <p
class=3D"MsoNormal">I tried different low level approaches to unmap LUN =
mpath’ed from hosts with inconsistent results.
<o:p></o:p></p <p
class=3D"MsoNormal">A notable error log extract
is:<o:p></o:p></p <p
class=3D"MsoNormal">Nov 8 15:30:52 sovana vdsm root ERROR Process
=
failed with rc=3D1 out=3D'\nudevadm settle - timeout of 5 seconds reached, =
the event queue contains:\n /sys/devices/virtual/block/dm-39 (8603)\n=
/sys/devices/virtual/block/dm-39 (8604)\n
/sys<o:p></o:p></p <p
class=3D"MsoNormal">/devices/virtual/block/dm-39 (8605)\n
/sys/dev=
ices/virtual/block/dm-39 (8606)\n /sys/devices/virtual/block/dm-39 (8=
607)\n /sys/devices/virtual/block/dm-39 (8608)\n /sys/devices/v=
irtual/block/dm-39 (8609)\n
/sys/devices/virtual/block<o:p></o:p></p <p
class=3D"MsoNormal">/dm-39 (8610)\n
/sys/devices/virtual/block/dm-=
39 (8611)\n /sys/devices/virtual/block/dm-39 (8612)\n /sys/devi=
ces/virtual/block/dm-39 (8613)\n /sys/devices/virtual/block/dm-39 (86=
14)\n /sys/devices/virtual/block/dm-39 (8615)\n
/sys/<o:p></o:p=
></p <p
class=3D"MsoNormal">devices/virtual/block/dm-39 (8616)\n
/sys/devi=
ces/virtual/block/dm-39 (8617)\n /sys/devices/virtual/block/dm-39 (86=
18)\n /sys/devices/virtual/block/dm-39 (8619)\n /sys/devices/vi=
rtual/block/dm-39 (8620)\n
/sys/devices/virtual/block/<o:p></o:p></p <p
class=3D"MsoNormal">dm-39 (8621)\n
/sys/devices/virtual/block/dm-3=
9 (8622)\n /sys/devices/virtual/block/dm-39 (8623)\n /sys/devic=
es/virtual/block/dm-39 (8624)\n /sys/devices/virtual/block/dm-39 (862=
5)\n /sys/devices/virtual/block/dm-39 (8626)\n
/sys/d<o:p></o:p=
></p <p
class=3D"MsoNormal">evices/virtual/block/dm-39 (8627)\n
/sys/devic=
es/virtual/block/dm-39 (8628)\n /sys/devices/virtual/block/dm-39 (862=
9)\n /sys/devices/virtual/block/dm-39 (8630)\n /sys/devices/vir=
tual/block/dm-39 (8631)\n
/sys/devices/virtual/block/d<o:p></o:p></p <p
class=3D"MsoNormal">m-39 (8632)\n
/sys/devices/virtual/block/dm-39=
(8633)\n /sys/devices/virtual/block/dm-39 (8634)\n /sys/device=
s/virtual/block/dm-39 (8635)\n /sys/devices/virtual/block/dm-39 (8636=
)\n /sys/devices/virtual/block/dm-39 (8637)\n
/sys/de<o:p></o:p=
></p <p
class=3D"MsoNormal">vices/virtual/block/dm-39 (8638)\n
/sys/device=
s/virtual/block/dm-39 (8639)\n /sys/devices/virtual/block/dm-39 (8640=
)\n /sys/devices/virtual/block/dm-39 (8641)\n /sys/devices/virt=
ual/block/dm-39 (8642)\n
/sys/devices/virtual/block/dm<o:p></o:p></p <p
class=3D"MsoNormal">-39 (8643)\n /sys/devices/virtual/block/dm-39
=
(8644)\n /sys/devices/virtual/block/dm-39 (8645)\n /sys/devices=
/virtual/block/dm-39 (8646)\n
/sys/devices/virtual/block/d<o:p></o:p>=
</p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal">I really need your help
to sort this out as I’=
m actually blocked in my task.<o:p></o:p></p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal">Why mapping changes
triggers ovirt crash on a storag=
e which ovirt should not care about?<o:p></o:p></p <p
class=3D"MsoNormal">Thanks<o:p></o:p></p <p
class=3D"MsoNormal"><o:p> </o:p></p <p
class=3D"MsoNormal"><o:p> </o:p></p <p class=3D"MsoNormal"><b><span
lang=3D"IT" style=3D"font-size:9.0pt;font-f=
amily:ION;color:black">Andrea
Ghelardi<o:p></o:p></span></b></p <p
class=3D"MsoNormal"><span lang=3D"IT"
style=3D"font-size:8.0pt;font-fami=
ly:"Source Sans
Pro",sans-serif;color:gray"><o:p> </o:p></sp=
an></p <p
class=3D"MsoNormal"><span lang=3D"IT"
style=3D"font-size:8.0pt;font-fami=
ly:"Source Sans Pro",sans-serif;color:gray">+39 050 2203
71</=
span><span lang=3D"IT"
style=3D"font-size:8.0pt;font-family:"Source Sa=
ns Pro",sans-serif" <b><span
style=3D"color:#33CCFF">| </span></b></span><span
lang=3D"EN-GB" s=
tyle=3D"font-size:8.0pt;font-family:"Source Sans Pro",sans-serif;=
color:gray"><a href=3D"http://www.iongroup.com/"><span
lang=3D"IT" style=3D=
"color:gray">www.iongroup.com</span></a></span><span
lang=3D"EN-GB" style=
=3D"font-size:8.0pt;font-family:"Source Sans Pro",sans-serif;colo=
r:#33CCFF" </span><b><span
lang=3D"IT" style=3D"font-size:8.0pt;font-family:"Sour=
ce Sans Pro",sans-serif;color:#33CCFF">|
</span></b><u><span lang=3D"EN-GB"
style=3D"font-size:8.0pt;font-family:&qu=
ot;Source Sans Pro",sans-serif;color:#0563C1"><a
href=3D"mailto:a.ghel=
ardi(a)iontrading.com"><span
lang=3D"IT">a.ghelardi(a)iontrading.com</span></a>=
</span></u><span lang=3D"IT"
style=3D"font-size:8.0pt;font-family:"Sou=
rce Sans
Pro",sans-serif;color:gray"><o:p></o:p></span></p <p class=3D"MsoNormal"><span
lang=3D"IT" style=3D"font-size:8.0pt;font-fami=
ly:"Source Sans Pro",sans-serif;color:gray">Via San Martino, 52
&=
#8211; 56125 Pisa - ITALY<o:p></o:p></span></p <p class=3D"MsoNormal"><b><span
lang=3D"IT" style=3D"font-family:ION"><o:p>=
</o:p></span></b></p <p
class=3D"MsoNormal"><i><span
style=3D"font-size:7.5pt;font-family:"=
Source Sans Pro",sans-serif;color:silver">This email and any attachmen=
ts may contain information which is confidential and/or privileged. The inf=
ormation is intended exclusively for the addressee
and the views expressed may not be official policy, but the personal views=
of the originator. If you are not the intended recipient, be aware that an=
y disclosure, copying, distribution or use of the contents is prohibited. I=
f you have received this email and
any file transmitted with it in error, please notify the sender by telepho=
ne or return email immediately and delete the material from your computer. =
Internet communications are not secure and ION Trading is not responsible f=
or their abuse by third parties,
nor for any alteration or corruption in transmission, nor for any damage o=
r loss caused by any virus or other defect. ION Trading accepts no liabilit=
y or responsibility arising out of or in any way connected to this email.</=
span></i><span style=3D"font-size:7.5pt;font-family:"Source Sans
Pro&q=
uot;,sans-serif"><o:p></o:p></span></p <p class=3D"MsoNormal"><b><span
style=3D"font-family:ION"><o:p> </o:p>=
</span></b></p <p
class=3D"MsoNormal"><img border=3D"0" width=3D"120"
height=3D"66" style=
=3D"width:1.25in;height:.6875in" id=3D"Picture_x0020_1"
src=3D"cid:image001=
.png(a)01D240F1.5E6E2B20"
alt=3D"iON_HBlu_small"><o:p></o:p></p <p class=3D"MsoNormal"><span
style=3D"font-size:6.0pt;font-family:ION;color=
:black">Automation through innovation</span><i><span
style=3D"font-size:7.5=
pt;font-family:"Source Sans
Pro",sans-serif;color:silver"><o:p></=
o:p></span></i></p <p
class=3D"MsoNormal"><o:p> </o:p></p </div
</body </html
--_000_CY4PR14MB1687A36A14A995B3A6DA5BABE9B10CY4PR14MB1687namp_--
--_004_CY4PR14MB1687A36A14A995B3A6DA5BABE9B10CY4PR14MB1687namp_
Content-Type: image/png; name="image001.png"
Content-Description: image001.png
Content-Disposition: inline; filename="image001.png"; size=5300;
creation-date="Thu, 17 Nov 2016 15:48:07 GMT";
modification-date="Thu, 17 Nov 2016 15:48:07 GMT"
Content-ID: <image001.png(a)01D240F1.5E6E2B20
Content-Transfer-Encoding: base64
iVBORw0KGgoAAAANSUhEUgAAAHgAAABCCAYAAACchRIZAAAABGdBTUEAALGPC/xhBQAAAAlwSFlz
AAAOvwAADr8BOAVTJAAAABl0RVh0U29mdHdhcmUAcGFpbnQubmV0IDQuMC4xMK0KCsAAABQxSURB
VHhe7V0JeJNVuv5adgTbbC17UVARV2j+Py1LCyRpCyIIUhmQRdRhREfcGIXRMY4KbZKWigpaAaVN
C4h3HL0z4qAiOjreRwVEBESUy9aUrdA2SYFC23Pfc5pAS/+mCa30xifv83xPQnr293zLWf4fAiIo
19mZ8oqvp0Ln3VTgzCZH8QpyFD2L7+NpzeG+tI615wnDCEWs39OBCorGUn7RP0HqSUgNFRYz8elw
nqR85/tUeMQk0oURgnjfHUsFxStBphsaywS5daWg+AzIfo9WHeiH1BG1mcIIHaw9fCMVFm0BubWa
qyQOZyk+x5CFRXpzhREyKDySCHL3NSC1rhQUVeNzNnxxG2+uMEIGecV6EPxTA1LrSkHRWXIcmULr
1oUJDjnkH74KgdQn8LVViuRyKXDuQWQtEWNhHxxyeKs0GgS+CC3lflaB3OJTmAA2KixTeXOEEVKw
sLZCOx3OPJB8FGRzf1vrd/MPHaGCQytpTdG14QArlMHXuIXOAYimZ4HgpVgWFVLhwUxyHJpEb5f0
DAdXvwVYLJG0vqYDvV2mpr+5Y+i9410pl7UL+90wwggjjDDCCBqMUQRbl95mhyW9/Q7LwPb8Owu7
1N8GWG58O/cyc0zZYpPeYzfeVmE33+6yjhp+OiO1L4gOn/6FMpgluaMnK22Qy2r6s8tm+shtM+/y
2M273Zmmbzx20+uezNRUZknv4k0eRiiBWSjyTI7peo/NtMRjNe8HuWc9WeaaiqwU5rabqqHJLo/N
/AV+T2ez49t5s4URKtizpH8Ht8041WNL4VpbxYm9WNxZ5kpOckX2qJ7ebGGECo5bx3UFsc+A4FIl
cn3itqec9lhTxnqzhREqYLmmKJfdaPdkpXiUiPUJ/s41+XFvtjBCBTXQYLfd/By0uFyJWJ/AH1e5
bOYJ3mxhhAr2IYIut5tmuezmA9BSEVwpCaLp3RUvhn1wyIFZLJFlVpMePvY9RMpnGhCbZebaW+qy
mx5oThQdQZTclogXkM5Pmn7t3RNePupBnQMHYhHP5bLVDfB6eH28z82ur3bsRD8EAUGXx820y546
0WU1f+i2mktgrs+A2EqsiyvcWaZfoN0Lal4xarzJg0VyR9ImDiKt4Xekle4njf52itJfTZTW0tdq
0fH+HUgzqAdpJL2oR6efThr5IdIYHiS1dA/FSBNIbTCQZkgP6o+0vwbZ6oSe6O9YUsv3kTZ+CmkH
x+P7lfhLsHVFkCo+CvkHi3J4P9SGe0iXMFT8HkR5DGmPWpK7lGakDYLWzgGhL3uyTEs9mSPn4zOV
5SRH8zTe5EEhAg27Ax3cRFrZie/HMPh7Mdj5pEk0YkZ29qZrJkCWWh5YO4HkN/H9W3zuB7GHISWY
XCX491G0oQjyA9qANAkzqavhmlrNaBFEUIzhZpCwHHXtFX3lfVZLnwmyYxNjRJrAEEnaYdeinU+R
Wv8VyioW/eB90EhbMXkfQl2xSBcUKRYLRR7MTuhUlp2iLofGHrabr+Am3PvnoIEZOCgRDfoeHa3C
IDOv1KCxFWjsejEba83YpUM3sAsGcQLKexdlOyGVoo4L9V0sqF8+gzwHMPDroOUTqesAbp6acwMl
gqKkW1HmRpRdv36NfA71fEfRCXfDcnQVaZuCKr4PJoqdVBImpKHu2PHyqkA6yjNM85bXSogy9EVj
vga5yoPNB1kjrxImLciZWAv4I55XbZiLSbQTZfGBbVhP48KJrsTg/4g2PgsCbiJK6ISCg28L1yaV
/p2LJvIF0UjnUNd6fOq9cUDjiINLU+tnIe0ulHdOsTy1wSMmZyDl/WpQSY+jgacVG+gTlXSUVIlD
gm4kn7kqKQ2+CaZeOoJ6qhXLD0iQVyPDlMtrIOOD9W8CqsGpaMd+5fK9opFPIs0c6jy4O/XyM5G6
x2thgl9B2lLFcrgILZa3Q2YgfQu5uWChkpc2OqN9ooYGqaTU4Mw0/K1OnoROfolBOIVy/JnjQKUG
beFa8QW0Y2LQsUG0BPMrH1cot47Akqnkj6GdGfiO4As+Vqnf2uHdUdYq9M2tXA4XPimln9HeP7ae
mdbIT6Ix/s2mGrNUFT8MqQPX4CsQyGikj1D+GcUymyXyWUGCDv40GJ+sNYyBHKpfloJoJO6WyvC5
C7ICscgI5K6vyV2gwVr9MqQva5DfJ1yDNQgW1dJMir35Cm/Oywzu04R/U2igEPgXleG/Kermq5A6
MJMYlRwNX/cyOuZSLrMFRANN1iLA4QMdKFRD+4hJF4yr4PWo5a+8k+k8YkFYjz4Jf+rcTToUoTMo
ltdWJ1de2V3+pGfckJHNDlIvHQiCVAl3oRO7G3RcI5+CfI4ZPM4b2AQAHlTB56jlfSijJcxyY4Ky
pUNwA+mBxwYYZBW0WCPBbTQSGCkJ10SVlCMCKy92YL36X78fueA+4zBnr6sSatro6udpG2NgNwxI
ODv/9uGb/jV3lJnlzr7k3afmgwcAGv209rGG/+naTa6I6ilXdYw1HG+jk98WGxFxt0QjVWDaqzZc
D//1dwxKgKYZPo/7eLXsFhrPvyumUxIReH1MPaQgdndAEpZ9EfCfndDHqJ6G6i49Elg7EBKhVIcu
gVH3JEZx5s8o7el+lLtZEOW2m2NKMoyvbZ0/qjx7ygg2bNBQFt0rURDbqXsCG3zjEPbK3clVu58x
7iy3ptx79NXk1r2JYRk3pEfmXcl/tf9u+J6sKcPcWXclbbJOGn63ZYKZL/wD83PJ0BC1fC+I+gWD
07QZFBYCfk7NJ5L8IuQFkFYAi7ENv/mP7H3ClyI8qg4C8fHx7TImDb8ta/KwjehvxYI7klmqYSjT
9Ulg9TSxt5GRPJPRxL8ymptXTMt2WKnQmUqrj8T+nDW5t9tqzivLMHl+/IuZLQbJSYOHspi4BHbj
wCHsmQnJbMfTxmqXzby3wm6cW7Mkje+StQ4YZld5pmleaaZp9/EM45mji0zVJzJM5fhtoycjZUbN
IqMmoC2ybsN00N5sDDhfOvg3zzyIUUurKVq+DWT2oqtNUUJib44B6Xr8/Q2kazyAuSDcAnyI2gOa
hMwytnN5lmkc+rrhJPrI+1q80MQ2zzexP4Po/tdACznJ/W9jNGMJo6yvGS3/idGbe6vJUVRKBc5t
+Fw0btV7SQcXpeZ+/sSo0nm3J7HrByQybgm4BnfslsD6oRzLhKSqbQtGbS+3YQzt5tYJsjbzW3x2
0xMum+mI255SzQ+Ua08v+NmjudJjTfnBY02dzrfLvFkaB9+GVMnvghy+LFIio1Y0iILV0nr40CTv
+vDiyRNJmvgBSPMe0jZtsjXQ4m636rx5GwWzJLf1WI33or+70ddzHvuF0xr8m+2xmNj8cUms+zXJ
LGLWK4xyf+RPYNZ5aA/fHUVVEOeVeXvsjz0w0w7NP6junVATeZEP5pagb7+EypnGYRvevD9pOK/b
24zLi9Ic860um3F/Y2eQ/H6QK8u48fRi49VNarFaTgAp/4aZ9R/A8E0PFZZntRGwcpmceK38GAg+
pljGxdItUfLmbBSnXzZf5bEb/+3OMlcr9ZWT/OU8I0u6azZrmwPNLSiqQ24dcRRVRyzfs6VTyuMf
dOg+5Bh8t6K1aoPlXKfuhk+jeoj9/NYJslxZpvnwE5VKHfaJJ9NUjnTD+AVsbzYl8AOLkfCJm9E5
PxsnIjDaQZrEO4jizkelDcGjcb53Le1EvqajcRHp+we/y+Symg4p9dEnxxaZ2VTrUtZx1S/K5Ppk
6S4PmeedhFs6q9geIWJVspN0hlmttg5224zLG7vF55Nyq+msJ9s0hm3ya2YixW6XWtqGTjUeYGmk
KkyCbygm3ux/VqMunSiPT5imCVbLk7wZG4UnwzgDBJco9dEncEs19+U6qjs69isT65OlP9TQmPk1
1DNZuT1cNHwyi1O5uWhf6wRZHlvKvHKFGwR1pcxmKnEtNA9l6YFosCDEnwbXoNO7SKe/s+66siEG
tsfMvxPpdyFf0wTzXaom4LEZ0/iVGKU++gR/L5ud+9bODvn7Kur734tkyVZGpsewhBqu1BavyFUR
Wql1d7K+z5h603b71BOerFTFDpdmj2E/LZ786Q9LJvfjz8x4sylDHS8jcOJnyv4DI41cgiDqOdIO
7o5cymXyvVu14WmRVqmM+lJz8U6TEk7ax8S5M83fIrhq9M6T22r8LuNV+5/a5h/4DgQrv/7CAd/8
lw8YDZ7GKGaoUnuEtIuRK7W95Y/j+hlG8KWZtxmXF+3y9g2amvv2iR0vTWOurLTzHeVR9InssWzj
kofYjDdWfzYo/6v+TT5TrJWuxWxdC6lQ6vB54WZag2CM7yo1vEWB77AUfKmklj4FwU3vOGmkcu9m
jF+UIVCEid7sl2Cbads++52jO+bvfwpB1oEGJOcfhPZuYZS+kFFfc+1GiEKb+HIJS6fKJ24f/skn
D48aubnVdrIKnAvgbypnLF/L1mfPYTsXjmN7nk9jWzPS2ZtLnmHjVvyDRef/VNrW4Rze5BsBrkxQ
U7Q+A6Q1rXVqyY10/4DcR9qEwRRr6CvOplX6G7GWvhOyGsT5Oak5L3wdvBa1+598QLnVOBbxRJES
sT5x28xuvm5NXPGfOHI4HyHHoX9hjH6mt/Ydg1muomfXg9xFjAaMh/Ym1rZBh88e8MVXpbLI/qNZ
p2tSWXz8KJYzdUT1Tr6TZTfNarV1MBUUv4QOnOv66jdMfvB5NnHcRDYlLYXdNvkedt385eyK17ax
iFUHz6CzKWTZ1MRaju9k6SeKnaimjiAFMRLf6/4Z3/kNi3Ug6m2QugGfu/Hpfy3tE7XsJK0+2dsA
v3BbR06BBh9VItYn8MHnXDbT7zdbxnamdSejaHXRIIzPeHr6w+co7cmjpIdZ5prLyeXay4OsePw2
8XlGD6xk7R91sJjHV7Kpzyxkn2TMqD5mG/1zhd38YI21tY4L84tm0eu7T9G0xazNDRNYx95JrFPP
Iax9XyOLNMxiNGclo5e3H6S8A/qAXtjC/apWXgZzHehJEiJumGFxPUfcHuEmueltTi4aw1nkWRbo
Wasrw5wEAn/xbeYoicdmPuK2paXvedh72ZChz+t2tKebpsZjSbQVxF4I+HqNQKD1KKOn3mX06veM
Vu6FMuxj7VftZX3yvmcP5Oad2/jSnC3HcsZOrsmeFOBhTUvD8vG1NC3nBPUbXWtq6g5gNwQQt0xm
NCXjI/pjfoDHhZZI0gwdgQBpawBa3AzhhxTSVlLFD0WlAW1TVmQm93LbTV83vqnDCTZ+Cy1PbrDz
pJO6od58TMDaPXKuwfI9jBa8z+hNrJkvirgjC52sR972yvFv/G3DwpetwzdZLK2zk4UwP4V6m1yN
BQsUOwT+ZcR26j3yBqQO7NCB70LVrv2KUUYgS5zgRc1vLuofIl3gpzSe7JRbXDbjjsYI5uKBhnNf
zZ+u92bzIq4j6kwHwfyosYJ6jqihqXa+Hm5A7nmSHQfPdcrf92XXvL1jaUlrvdVXLVkxYH52YyDi
KM9gFj42UPB7zyp5CTSt5Q/9+QBrpBy0qxdqCsCq1KLMZk5HkHVEiVifIIo+48ky3rvvrQZr9Aix
3y2uF0uvU//RW+jRNWXiEEKBXCH83WT8dZJ5RQ/QiuNdveVcZqj5qU0gd7L4RkIQBHNt76q/Dn5y
dYuaah588fvSmsQBwV4CPGVNne6ympvYyTLxI74/8HvJ3mx1ESmsU7dBcTTykXGUs+UjLJtOK5LL
RRBctBcB6sPk2NNKx4XclDZ1J0slnYQGG5A6MBN9AZEUnRgHTV4KcxrIksef8OuzpWhvrlhKBTfZ
BBBApYA8vztZMNFl7izj3TVL/D7REUHLtsVQwYGlWEaViVMmZYKxhi7aTmsOTaf8w620TOKXtzUG
vqGvNKjcHEL75EJS33KJ96KRh29maPTzUNZBlBVYhFxPxAHFAciLYjPlEk9mQJquwmZc72nkNIn/
7rGZNp7NTJGbOFghrIs7QjPvJ8fBPeL1kEoEO5yn8Ld3xSslW/GtghGkG5ICEvixXP2AiF8EV0v/
wdrW1OzjrqhbolHWTJD0BcQTGNFis74cbfscbZglItlmXCDnW61nbamJbpt5tyvLVI9kty2lGtHz
L/g+uzRnfJO7YigtgtYc7QcCl5O4COD9LxN8wknn5OcXzRHr6dYFzF20NBUDvwOD6QIRp6FxJRjc
D8SNC2qp+0Twa9GDb6rVZmkD/Pr/4hPmH/WpxVVV/qgKliFoQzS0XaPfAI2di8/rvEeLl2JB6oE/
4+O2m28EwetBdInbaj6Ff7tA7g8IsB71LE7t3uSeuw/8Pdtr+H96UrQYhHIy3SD7DD5PU75zL35/
mlYc6PH/5LWRGMBoKUkcxHNTqJL/UBvItNhDX3XRhrqIiHQUCJ4DQi2QTJBphX99HpPsMRHU1T4I
FqzGYjC5f0Z/+BJK3BgRfagXP7BXx3QDqfeD1Bfgl5/ka1++23RJT+9xc7328EhyHJ5Pq4szKc/5
Ar3lvIMcJa13D8sPMBDCDF6+WcdnOD9tqT1xaU69bcRSRqUfIky6uNQvPY7P6bAC0sWPu/jeKNec
J/fqge/08Re9hl/R/Ksgonab1DBbuBbxDJJ8AuSWwOzvg/wTJE+nPjeFX4AemoD280N//pyvVubH
lXUDRu+jqPHfkG5wKhKHNSzkwJdiavhyjXSiDrH1hR9OqAzveG9yhhFSEEGblAdNbeo2SSnxV0OE
EWK4whAL87xWaKkSsT7ht0k6JfM97DBCCl0lDWn1r0FD/V8UUCH4igoHWqEHcUSpfwQaekSRWC4a
QyX+/px4OD2MkEMbkBgPAt8BkQ0v/YmXrEh/J11Cf6S9fGv8MFoQXItjDGbS8mea6pxeqaRKQW70
oJuRqtU2/cNoESS3FW/10fGdLD2iaukVkDxa3Pj8zax/if4PjTMQFfXnvswAAAAASUVORK5CYII=
--_004_CY4PR14MB1687A36A14A995B3A6DA5BABE9B10CY4PR14MB1687namp_--