From fromani at redhat.com Wed Feb 22 14:32:18 2017 Content-Type: multipart/mixed; boundary="===============2879356379117104388==" MIME-Version: 1.0 From: Francesco Romani To: devel at ovirt.org Subject: Re: [ovirt-devel] OST: HE vm does not restart on HC setup Date: Wed, 22 Feb 2017 15:32:14 +0100 Message-ID: <03625763-4b0d-f3c1-e103-979e66355a0c@redhat.com> In-Reply-To: CAN8-ONooweDRBtBrPRZ6OgwcTOfRSB4S96eK8=pCo3mv85+C+w@mail.gmail.com --===============2879356379117104388== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------2DA879FFF9040AC877FCC61C Content-Type: text/plain; charset=3Dutf-8 Content-Transfer-Encoding: 7bit On 02/22/2017 01:53 PM, Simone Tiraboschi wrote: > > > On Wed, Feb 22, 2017 at 1:33 PM, Simone Tiraboschi > > wrote: > > When ovirt-ha-agent checks the status of the engine VM we get: > > 2017-02-21 22:21:14,738-0500 ERROR (jsonrpc/2) [api] FINISH getStats = error=3DVirtual machine does not exist: {'vmId': u'2ccc0ef0-cc31-45b8-8e91-= a78fa4cad671'} (api:69) > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 67= , in method > ret =3D func(*args, **kwargs) > File "/usr/share/vdsm/API.py", line 335, in getStats > vm =3D self.vm > File "/usr/share/vdsm/API.py", line 130, in vm > raise exception.NoSuchVM(vmId=3Dself._UUID) > NoSuchVM: Virtual machine does not exist: {'vmId': u'2ccc0ef0-cc31-45= b8-8e91-a78fa4cad671'} > > While in ovirt-ha-agent logs we have: > > MainThread::INFO::2017-02-21 22:21:18,583::hosted_engine::453::ovirt_= hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Curre= nt state UnknownLocalVmState (score: 3400) > > ... > > MainThread::INFO::2017-02-21 22:21:31,199::state_decorators::25::ovir= t_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Unknown local = engine vm status no actions taken > > Probably it's a bug or a regression somewhere on master. > > On ovirt-ha-broker side the detection is based on a strict string > match on the error message that is expected to be exactly 'Virtual > machine does not exist' to set down status otherwise we set unknown > status as in this case: > https://gerrit.ovirt.org/gitweb?p=3Dovirt-hosted-engine-ha.git;a=3Dblob;f= =3Dovirt_hosted_engine_ha/broker/submonitors/engine_health.py;h=3Dd633cb860= b811e84021221771bf706a9a4ac1d63;hb=3Drefs/heads/master#l54 > > = > Adding Francesco here to understand if something has recently changed > there on vdsm side. It has changed indeed; we had a series of changes which added context to some exceptions. I believe the straw who broke the camel's back was I32ec3f86f8d53f8412f4c0526fc85e2a42e30ea5 It is unfortunate that this change broke HA. Could you perhaps fixing it checking that the message *begins* with that string, and/or checking the error code. bests, -- = Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani --------------2DA879FFF9040AC877FCC61C Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit On 02/22/2017 01:53 PM, Simone Tiraboschi wrote:


On Wed, Feb 22, 2017 at 1:33 PM, Simone Tiraboschi <stirabos(a)redhat.com> wrote:
When ovirt-ha-agent checks the status of the engine VM we get: 
2017-02-21 22:21:14,738-0=
500 ERROR (jsonrpc/2) [api] FINISH getStats error=3DVirtual machine does no=
t exist: {'vmId': u'2ccc0ef0-cc31-45b8-8e91-a78fa4cad671'} (api:69)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 67,=
 in method
    ret =3D func(*args, **kwargs)
  File "/usr/share/vdsm/API.py", line 335, in getStats
    vm =3D self.vm
  File "/usr/share/vdsm/API.py", line 130, in vm
    raise exception.NoSuchVM(vmId=3Dself._UUID)
NoSuchVM: Virtual machine does not exist: {'vmId': u'2ccc0ef0-cc31-45b8-8e9=
1-a78fa4cad671'}
While in ovirt-ha-agent logs we have:
MainThread::INFO::2017-02-21 22:21:18,583::hosted_engine::453::ovirt_h=
osted_engine_ha.agent.hosted_engine.HostedEngine::(start_mon=
itoring) Current state UnknownLocalVmState (score: 3400)
...
MainThread::INFO::2017-02-21 22:21:31,199::sta=
te_decorators::25::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Unknown local engine vm status no actions taken
Probably it's a bug or a regression somewhere on master.


On ovirt-ha-broker side the detection is based on a strict strin=
g match on the error message that is expected to be exactly 'Virtual machin=
e does not exist' to set down status otherwise we set unknown status as in =
this case:
=C2=A0
Adding Francesco here to understand if somethin=
g has recently changed there on vdsm side.

It has changed indeed; we had a series of changes which added context to so=
me exceptions. I believe the straw who broke the camel's back was I32ec3f86=
f8d53f8412f4c0526fc85e2a42e30ea5

It is unfortunate that this change broke HA.

Could you perhaps fixing it checking that the message *begins* with that st=
ring, and/or checking the error code.

bests,


-- =

Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani

--------------2DA879FFF9040AC877FCC61C--

--===============2879356379117104388==
Content-Type: multipart/alternative
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="attachment.bin"

VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t
LS0yREE4NzlGRkY5MDQwQUM4NzdGQ0M2MUMKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy
c2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDdiaXQKCk9uIDAyLzIyLzIwMTcg
MDE6NTMgUE0sIFNpbW9uZSBUaXJhYm9zY2hpIHdyb3RlOgo+Cj4KPiBPbiBXZWQsIEZlYiAyMiwg
MjAxNyBhdCAxOjMzIFBNLCBTaW1vbmUgVGlyYWJvc2NoaQo+IDxzdGlyYWJvc0ByZWRoYXQuY29t
IDxtYWlsdG86c3RpcmFib3NAcmVkaGF0LmNvbT4+IHdyb3RlOgo+Cj4gICAgIFdoZW4gb3ZpcnQt
aGEtYWdlbnQgY2hlY2tzIHRoZSBzdGF0dXMgb2YgdGhlIGVuZ2luZSBWTSB3ZSBnZXQ6Cj4KPiAg
ICAgMjAxNy0wMi0yMSAyMjoyMToxNCw3MzgtMDUwMCBFUlJPUiAoanNvbnJwYy8yKSBbYXBpXSBG
SU5JU0ggZ2V0U3RhdHMgZXJyb3I9VmlydHVhbCBtYWNoaW5lIGRvZXMgbm90IGV4aXN0OiB7J3Zt
SWQnOiB1JzJjY2MwZWYwLWNjMzEtNDViOC04ZTkxLWE3OGZhNGNhZDY3MSd9IChhcGk6NjkpCj4g
ICAgIFRyYWNlYmFjayAobW9zdCByZWNlbnQgY2FsbCBsYXN0KToKPiAgICAgICBGaWxlICIvdXNy
L2xpYi9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy92ZHNtL2NvbW1vbi9hcGkucHkiLCBsaW5lIDY3
LCBpbiBtZXRob2QKPiAgICAgICAgIHJldCA9IGZ1bmMoKmFyZ3MsICoqa3dhcmdzKQo+ICAgICAg
IEZpbGUgIi91c3Ivc2hhcmUvdmRzbS9BUEkucHkiLCBsaW5lIDMzNSwgaW4gZ2V0U3RhdHMKPiAg
ICAgICAgIHZtID0gc2VsZi52bQo+ICAgICAgIEZpbGUgIi91c3Ivc2hhcmUvdmRzbS9BUEkucHki
LCBsaW5lIDEzMCwgaW4gdm0KPiAgICAgICAgIHJhaXNlIGV4Y2VwdGlvbi5Ob1N1Y2hWTSh2bUlk
PXNlbGYuX1VVSUQpCj4gICAgIE5vU3VjaFZNOiBWaXJ0dWFsIG1hY2hpbmUgZG9lcyBub3QgZXhp
c3Q6IHsndm1JZCc6IHUnMmNjYzBlZjAtY2MzMS00NWI4LThlOTEtYTc4ZmE0Y2FkNjcxJ30KPgo+
ICAgICBXaGlsZSBpbiBvdmlydC1oYS1hZ2VudCBsb2dzIHdlIGhhdmU6Cj4KPiAgICAgTWFpblRo
cmVhZDo6SU5GTzo6MjAxNy0wMi0yMSAyMjoyMToxOCw1ODM6Omhvc3RlZF9lbmdpbmU6OjQ1Mzo6
b3ZpcnRfaG9zdGVkX2VuZ2luZV9oYS5hZ2VudC5ob3N0ZWRfZW5naW5lLkhvc3RlZEVuZ2luZTo6
KHN0YXJ0X21vbml0b3JpbmcpIEN1cnJlbnQgc3RhdGUgVW5rbm93bkxvY2FsVm1TdGF0ZSAoc2Nv
cmU6IDM0MDApCj4KPiAgICAgLi4uCj4KPiAgICAgTWFpblRocmVhZDo6SU5GTzo6MjAxNy0wMi0y
MSAyMjoyMTozMSwxOTk6OnN0YXRlX2RlY29yYXRvcnM6OjI1OjpvdmlydF9ob3N0ZWRfZW5naW5l
X2hhLmFnZW50Lmhvc3RlZF9lbmdpbmUuSG9zdGVkRW5naW5lOjooY2hlY2spIFVua25vd24gbG9j
YWwgZW5naW5lIHZtIHN0YXR1cyBubyBhY3Rpb25zIHRha2VuCj4KPiAgICAgUHJvYmFibHkgaXQn
cyBhIGJ1ZyBvciBhIHJlZ3Jlc3Npb24gc29tZXdoZXJlIG9uIG1hc3Rlci4KPgo+IE9uIG92aXJ0
LWhhLWJyb2tlciBzaWRlIHRoZSBkZXRlY3Rpb24gaXMgYmFzZWQgb24gYSBzdHJpY3Qgc3RyaW5n
Cj4gbWF0Y2ggb24gdGhlIGVycm9yIG1lc3NhZ2UgdGhhdCBpcyBleHBlY3RlZCB0byBiZSBleGFj
dGx5ICdWaXJ0dWFsCj4gbWFjaGluZSBkb2VzIG5vdCBleGlzdCcgdG8gc2V0IGRvd24gc3RhdHVz
IG90aGVyd2lzZSB3ZSBzZXQgdW5rbm93bgo+IHN0YXR1cyBhcyBpbiB0aGlzIGNhc2U6Cj4gaHR0
cHM6Ly9nZXJyaXQub3ZpcnQub3JnL2dpdHdlYj9wPW92aXJ0LWhvc3RlZC1lbmdpbmUtaGEuZ2l0
O2E9YmxvYjtmPW92aXJ0X2hvc3RlZF9lbmdpbmVfaGEvYnJva2VyL3N1Ym1vbml0b3JzL2VuZ2lu
ZV9oZWFsdGgucHk7aD1kNjMzY2I4NjBiODExZTg0MDIxMjIxNzcxYmY3MDZhOWE0YWMxZDYzO2hi
PXJlZnMvaGVhZHMvbWFzdGVyI2w1NAo+Cj4gIAo+IEFkZGluZyBGcmFuY2VzY28gaGVyZSB0byB1
bmRlcnN0YW5kIGlmIHNvbWV0aGluZyBoYXMgcmVjZW50bHkgY2hhbmdlZAo+IHRoZXJlIG9uIHZk
c20gc2lkZS4KSXQgaGFzIGNoYW5nZWQgaW5kZWVkOyB3ZSBoYWQgYSBzZXJpZXMgb2YgY2hhbmdl
cyB3aGljaCBhZGRlZCBjb250ZXh0IHRvCnNvbWUgZXhjZXB0aW9ucy4gSSBiZWxpZXZlIHRoZSBz
dHJhdyB3aG8gYnJva2UgdGhlIGNhbWVsJ3MgYmFjayB3YXMKSTMyZWMzZjg2ZjhkNTNmODQxMmY0
YzA1MjZmYzg1ZTJhNDJlMzBlYTUgSXQgaXMgdW5mb3J0dW5hdGUgdGhhdCB0aGlzCmNoYW5nZSBi
cm9rZSBIQS4gQ291bGQgeW91IHBlcmhhcHMgZml4aW5nIGl0IGNoZWNraW5nIHRoYXQgdGhlIG1l
c3NhZ2UKKmJlZ2lucyogd2l0aCB0aGF0IHN0cmluZywgYW5kL29yIGNoZWNraW5nIHRoZSBlcnJv
ciBjb2RlLiBiZXN0cywKCi0tIApGcmFuY2VzY28gUm9tYW5pClJlZCBIYXQgRW5naW5lZXJpbmcg
VmlydHVhbGl6YXRpb24gUiAmIEQKSVJDOiBmcm9tYW5pCgotLS0tLS0tLS0tLS0tLTJEQTg3OUZG
RjkwNDBBQzg3N0ZDQzYxQwpDb250ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD11dGYtOApD
b250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiA4Yml0Cgo8aHRtbD4KICA8aGVhZD4KICAgIDxtZXRh
IGNvbnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11dGYtOCIgaHR0cC1lcXVpdj0iQ29udGVudC1U
eXBlIj4KICA8L2hlYWQ+CiAgPGJvZHkgYmdjb2xvcj0iI0ZGRkZGRiIgdGV4dD0iIzAwMDAwMCI+
CiAgICBPbiAwMi8yMi8yMDE3IDAxOjUzIFBNLCBTaW1vbmUgVGlyYWJvc2NoaSB3cm90ZTo8YnI+
CiAgICA8YmxvY2txdW90ZQpjaXRlPSJtaWQ6Q0FOOC1PTm9vd2VEUkJ0QnJQUlo2T2d3Y1RPZlJT
QjRTOTZlSzg9cENvM212ODUrQyt3QG1haWwuZ21haWwuY29tIgogICAgICB0eXBlPSJjaXRlIj4K
ICAgICAgPGRpdiBkaXI9Imx0ciI+PGJyPgogICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX2V4dHJh
Ij48YnI+CiAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+T24gV2VkLCBGZWIgMjIs
IDIwMTcgYXQgMTozMyBQTSwKICAgICAgICAgICAgU2ltb25lIFRpcmFib3NjaGkgPHNwYW4gZGly
PSJsdHIiPiZsdDs8YQogICAgICAgICAgICAgICAgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIiBocmVm
PSJtYWlsdG86c3RpcmFib3NAcmVkaGF0LmNvbSIKICAgICAgICAgICAgICAgIHRhcmdldD0iX2Js
YW5rIj5zdGlyYWJvc0ByZWRoYXQuY29tPC9hPiZndDs8L3NwYW4+CiAgICAgICAgICAgIHdyb3Rl
Ojxicj4KICAgICAgICAgICAgPGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0i
bWFyZ2luOjBweCAwcHggMHB4CiAgICAgICAgICAgICAgMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNv
bGlkCiAgICAgICAgICAgICAgcmdiKDIwNCwyMDQsMjA0KTtwYWRkaW5nLWxlZnQ6MWV4Ij4KICAg
ICAgICAgICAgICA8ZGl2IGRpcj0ibHRyIj5XaGVuIG92aXJ0LWhhLWFnZW50IGNoZWNrcyB0aGUg
c3RhdHVzIG9mCiAgICAgICAgICAgICAgICB0aGUgZW5naW5lIFZNIHdlIGdldDoKICAgICAgICAg
ICAgICAgIDxkaXY+CiAgICAgICAgICAgICAgICAgIDxwcmUgc3R5bGU9ImNvbG9yOnJnYigwLDAs
MCkiPjIwMTctMDItMjEgMjI6MjE6MTQsNzM4LTA1MDAgRVJST1IgKGpzb25ycGMvMikgW2FwaV0g
RklOSVNIIGdldFN0YXRzIGVycm9yPVZpcnR1YWwgbWFjaGluZSBkb2VzIG5vdCBleGlzdDogeyd2
bUlkJzogdScyY2NjMGVmMC1jYzMxLTQ1YjgtOGU5MS08d2JyPmE3OGZhNGNhZDY3MSd9IChhcGk6
NjkpClRyYWNlYmFjayAobW9zdCByZWNlbnQgY2FsbCBsYXN0KToKICBGaWxlICIvdXNyL2xpYi9w
eXRob24yLjcvc2l0ZS08d2JyPnBhY2thZ2VzL3Zkc20vY29tbW9uL2FwaS5weSIsIGxpbmUgNjcs
IGluIG1ldGhvZAogICAgcmV0ID0gZnVuYygqYXJncywgKiprd2FyZ3MpCiAgRmlsZSAiL3Vzci9z
aGFyZS92ZHNtL0FQSS5weSIsIGxpbmUgMzM1LCBpbiBnZXRTdGF0cwogICAgdm0gPSBzZWxmLnZt
CiAgRmlsZSAiL3Vzci9zaGFyZS92ZHNtL0FQSS5weSIsIGxpbmUgMTMwLCBpbiB2bQogICAgcmFp
c2UgZXhjZXB0aW9uLk5vU3VjaFZNKHZtSWQ9c2VsZi5fPHdicj5VVUlEKQpOb1N1Y2hWTTogVmly
dHVhbCBtYWNoaW5lIGRvZXMgbm90IGV4aXN0OiB7J3ZtSWQnOiB1JzJjY2MwZWYwLWNjMzEtNDVi
OC04ZTkxLTx3YnI+YTc4ZmE0Y2FkNjcxJ308L3ByZT48cHJlIHN0eWxlPSJjb2xvcjpyZ2IoMCww
LDApIj4KPC9wcmU+PHByZT5XaGlsZSBpbiBvdmlydC1oYS1hZ2VudCBsb2dzIHdlIGhhdmU6PHBy
ZSBzdHlsZT0iY29sb3I6cmdiKDAsMCwwKSI+PHByZT5NYWluVGhyZWFkOjo8YSBjbGFzcz0ibW96
LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJJTkZPOjoyMDE3LTAyLTIxIj5JTkZPOjoyMDE3LTAy
LTIxPC9hPiAyMjoyMToxOCw1ODM6Omhvc3RlZF9lbmdpbmU6Ojx3YnI+NDUzOjpvdmlydF9ob3N0
ZWRfZW5naW5lX2hhLjx3YnI+YWdlbnQuaG9zdGVkX2VuZ2luZS48d2JyPkhvc3RlZEVuZ2luZTo6
KHN0YXJ0Xzx3YnI+bW9uaXRvcmluZykgQ3VycmVudCBzdGF0ZSBVbmtub3duTG9jYWxWbVN0YXRl
IChzY29yZTogMzQwMCk8L3ByZT48cHJlPi4uLjwvcHJlPjwvcHJlPjxwcmUgc3R5bGU9ImNvbG9y
OnJnYigwLDAsMCkiPk1haW5UaHJlYWQ6OjxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQi
IGhyZWY9IklORk86OjIwMTctMDItMjEiPklORk86OjIwMTctMDItMjE8L2E+IDIyOjIxOjMxLDE5
OTo6c3RhdGVfPHdicj5kZWNvcmF0b3JzOjoyNTo6b3ZpcnRfaG9zdGVkXzx3YnI+ZW5naW5lX2hh
LmFnZW50Lmhvc3RlZF9lbmdpbmUuPHdicj5Ib3N0ZWRFbmdpbmU6OihjaGVjaykgVW5rbm93biBs
b2NhbCBlbmdpbmUgdm0gc3RhdHVzIG5vIGFjdGlvbnMgdGFrZW48L3ByZT48L3ByZT5Qcm9iYWJs
eSBpdCdzIGEgYnVnIG9yIGEgcmVncmVzc2lvbiBzb21ld2hlcmUgb24gbWFzdGVyLjwvZGl2Pjwv
ZGl2PjwvYmxvY2txdW90ZT48ZGl2Pgo8L2Rpdj48ZGl2Pk9uIG92aXJ0LWhhLWJyb2tlciBzaWRl
IHRoZSBkZXRlY3Rpb24gaXMgYmFzZWQgb24gYSBzdHJpY3Qgc3RyaW5nIG1hdGNoIG9uIHRoZSBl
cnJvciBtZXNzYWdlIHRoYXQgaXMgZXhwZWN0ZWQgdG8gYmUgZXhhY3RseSAnVmlydHVhbCBtYWNo
aW5lIGRvZXMgbm90IGV4aXN0JyB0byBzZXQgZG93biBzdGF0dXMgb3RoZXJ3aXNlIHdlIHNldCB1
bmtub3duIHN0YXR1cyBhcyBpbiB0aGlzIGNhc2U6PC9kaXY+PGRpdj48YSBtb3otZG8tbm90LXNl
bmQ9InRydWUiIGhyZWY9Imh0dHBzOi8vZ2Vycml0Lm92aXJ0Lm9yZy9naXR3ZWI/cD1vdmlydC1o
b3N0ZWQtZW5naW5lLWhhLmdpdDthPWJsb2I7Zj1vdmlydF9ob3N0ZWRfZW5naW5lX2hhL2Jyb2tl
ci9zdWJtb25pdG9ycy9lbmdpbmVfaGVhbHRoLnB5O2g9ZDYzM2NiODYwYjgxMWU4NDAyMTIyMTc3
MWJmNzA2YTlhNGFjMWQ2MztoYj1yZWZzL2hlYWRzL21hc3RlciNsNTQiPmh0dHBzOi8vZ2Vycml0
Lm92aXJ0Lm9yZy9naXR3ZWI/cD1vdmlydC1ob3N0ZWQtZW5naW5lLWhhLmdpdDthPWJsb2I7Zj1v
dmlydF9ob3N0ZWRfZW5naW5lX2hhL2Jyb2tlci9zdWJtb25pdG9ycy9lbmdpbmVfaGVhbHRoLnB5
O2g9ZDYzM2NiODYwYjgxMWU4NDAyMTIyMTc3MWJmNzA2YTlhNGFjMWQ2MztoYj1yZWZzL2hlYWRz
L21hc3RlciNsNTQ8L2E+CjwvZGl2PjxkaXY+wqA8L2Rpdj48ZGl2PkFkZGluZyBGcmFuY2VzY28g
aGVyZSB0byB1bmRlcnN0YW5kIGlmIHNvbWV0aGluZyBoYXMgcmVjZW50bHkgY2hhbmdlZCB0aGVy
ZSBvbiB2ZHNtIHNpZGUuPC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9ibG9ja3F1b3RlPgpJdCBo
YXMgY2hhbmdlZCBpbmRlZWQ7IHdlIGhhZCBhIHNlcmllcyBvZiBjaGFuZ2VzIHdoaWNoIGFkZGVk
IGNvbnRleHQgdG8gc29tZSBleGNlcHRpb25zLiBJIGJlbGlldmUgdGhlIHN0cmF3IHdobyBicm9r
ZSB0aGUgY2FtZWwncyBiYWNrIHdhcyBJMzJlYzNmODZmOGQ1M2Y4NDEyZjRjMDUyNmZjODVlMmE0
MmUzMGVhNQoKSXQgaXMgdW5mb3J0dW5hdGUgdGhhdCB0aGlzIGNoYW5nZSBicm9rZSBIQS4KCkNv
dWxkIHlvdSBwZXJoYXBzIGZpeGluZyBpdCBjaGVja2luZyB0aGF0IHRoZSBtZXNzYWdlICpiZWdp
bnMqIHdpdGggdGhhdCBzdHJpbmcsIGFuZC9vciBjaGVja2luZyB0aGUgZXJyb3IgY29kZS4KCmJl
c3RzLAoKPHByZSBjbGFzcz0ibW96LXNpZ25hdHVyZSIgY29scz0iNzIiPi0tIApGcmFuY2VzY28g
Um9tYW5pClJlZCBIYXQgRW5naW5lZXJpbmcgVmlydHVhbGl6YXRpb24gUiAmYW1wOyBECklSQzog
ZnJvbWFuaTwvcHJlPjwvYm9keT48L2h0bWw+Ci0tLS0tLS0tLS0tLS0tMkRBODc5RkZGOTA0MEFD
ODc3RkNDNjFDLS0K

--===============2879356379117104388==--