From michal.skrivanek at redhat.com Thu Apr 19 17:23:41 2018 Content-Type: multipart/mixed; boundary="===============3352768505449549620==" MIME-Version: 1.0 From: Michal Skrivanek To: users at ovirt.org Subject: Re: [ovirt-users] Frequent vm migration failure Date: Thu, 19 Apr 2018 19:23:35 +0200 Message-ID: <91B56ED6-A1D5-4DB2-87BE-25AB251CC489@redhat.com> In-Reply-To: 1523957290.6441.1.camel@prismatelecomtesting.com --===============3352768505449549620== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --Apple-Mail=3D_F611F34D-66F8-4B72-8CA1-0836FCB767CA Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=3Dutf-8 > On 17 Apr 2018, at 11:28, Stefano Stagnaro =3D wrote: >=3D20 > On Thu, 2018-04-12 at 20:20 +0200, Michal Skrivanek wrote: >>=3D20 >>=3D20 >>> On 12 Apr 2018, at 18:26, Stefano Stagnaro =3D > wrote: >>> Hi, >>>=3D20 >>> I recently upgraded an oVirt deployment from 3.6 to 4.0 and then =3D 4.1.9 (my actual release). Since then, when migrating many hosts =3D simultaneously I always experience few migrations failure like 1 on 10 =3D vms. The failure can occur on any host; moreover, after a couple of =3D failure the destination host fall in Error status and I have to manually = =3D re-activate or wait 30 min. Are they all in a 3.6 cluster?=3D20 >>>=3D20 >>> Tipical error found on vdsm log is (from the source host): >>> 2018-04-12 17:01:32,097+0200 ERROR (migsrc/3192dfe7) [virt.vm] =3D (vmId=3D3D'3192dfe7-eeac-4626-8c86-e49facc9006f') migration destination =3D error: Fatal error during migration (migration:287) >>>=3D20 >>> Please find the logs of source host (v15.ovirt), destination host =3D (v14.ovirt) and engine here: =3D https://www.dropbox.com/sh/xhf8ry4ih40poxd/AABxiFCIxDe14HSx2DqLE61ya?dl=3D3= D=3D 0 =3D >>>=3D20 >>> Some of the vm affected from the migration failure are: >>> svn 3192dfe7-eeac-4626-8c86-e49facc9006f >>> wood a8e83ff0-dfed-4074-b6b6-e947b8ebb952 >>> qnx66 5697c4a4-9e40-4dd6-aba2-c8ab9904a584 >>=3D20 >> can you also include qemu log from /var/log/libvirt/qemu/? >=3D20 > Hi Michal, I've added libvirt logs for relevant VMs on the previous =3D Dropbox share. I do not see anything wrong. It=3DE2=3D80=3D99s a bit too much data to go = =3D through, can you pinpoint the time and VM name when you see a failure? Thanks, michal >=3D20 >>=3D20 >> btw you seem to be using the legacy migration policy throttling the =3D speed significantly. Please read into the migration enhancements in 4.0 >> =3D https://www.ovirt.org/develop/release-management/features/virt/migration-e= =3D nhancements/ =3D > I've already moved to Minimal Downtime and then to Post-copy with same = =3D results. VM migrations continue to fail randomly. >=3D20 >>=3D20 >> Thanks, >> michal >=3D20 > Thanks, > Stefano. >=3D20 >=3D20 >>=3D20 >>>=3D20 >>> Thank you very much for your help. >>>=3D20 >>> --=3D20 >>> Stefano Stagnaro >>>=3D20 >>> Prisma Telecom Testing S.r.l. >>> Via Petrocchi, 4 >>> 20127 Milano =3DE2=3D80=3D93 Italy >>>=3D20 >>> Tel. 02 26113507 int 339 >>> e-mail: stefanos(a)prismatelecomtesting.com =3D >>> skype: stefano.stagnaro >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>=3D20 >=3D20 --Apple-Mail=3D_F611F34D-66F8-4B72-8CA1-0836FCB767CA Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=3Dutf-8

On 17 Apr 2018, at 11:28, Stefano Stagnaro <stefanos(a)prismatelecomtesting.com> wrote:

On Thu, 2018-04-12 at 20:2= 0 =3D +0200, Michal Skrivanek wrote:

On 12 Apr 2018, at 18:26, = =3D Stefano Stagnaro <stefanos(a)prismatelecomtesting.com> wrote:
Hi,

I recen= tly =3D upgraded an oVirt deployment from 3.6 to 4.0 and then 4.1.9 (my actual =3D release). Since then, when migrating many hosts simultaneously I always =3D experience few migrations failure like 1 on 10 vms. The failure can =3D occur on any host; moreover, after a couple of failure the destination =3D host fall in Error status and I have to manually re-activate or wait 30 =3D min.

Are they all in a 3.6 =3D cluster?

Tipical error found on vdsm= =3D log is (from the source host):
2018-04-12 =3D 17:01:32,097+0200 ERROR (migsrc/3192dfe7) [virt.vm] =3D (vmId=3D3D'3192dfe7-eeac-4626-8c86-e49facc9006f') migration destination =3D error: Fatal error during migration (migration:287)

Please find the logs of source host (v15.ovirt), destination = =3D host (v14.ovirt) and engine here: https://www.dropbox.com/sh/xhf8ry4ih40poxd/AABxiFCIxDe14HSx2Dq= L=3D E61ya?dl=3D3D0

Some of the vm affected= =3D from the migration failure are:
svn =3D 3192dfe7-eeac-4626-8c86-e49facc9006f
wood =3D a8e83ff0-dfed-4074-b6b6-e947b8ebb952
qnx66 =3D 5697c4a4-9e40-4dd6-aba2-c8ab9904a584

can you also include qemu log from =3D /var/log/libvirt/qemu/<vmname>?

Hi Michal, I've added= =3D libvirt logs for relevant VMs on the previous Dropbox =3D share.

I do not = =3D see anything wrong. It=3DE2=3D80=3D99s a bit too much data to go through, c= an =3D you pinpoint the time and VM name when you see a failure?

Thanks,

michal

btw you seem to be us= ing =3D the legacy migration policy throttling the speed significantly. Please =3D read into the migration enhancements in 4.0
https://www.ovirt.org/develop/release-management/features/virt= /=3D migration-enhancements/

I've already moved to Minimal Downtime= =3D and then to Post-copy with same results. VM migrations continue to fail =3D randomly.

Thanks,
michal

Thanks,

Stefano.

Thank you very much for you= r =3D help.

--
Stefano Stagnaro=

Prisma Telecom Testing S.r.l.
Via =3D Petrocchi, 4
20127 Milano =3DE2=3D80=3D93 Italy

Tel. 02 26113507 int 339
e-mail: = =3D stefanos(a)prismatelecomtesting.com
skype: =3D stefano.stagnaro
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

<= div=3D class=3D3D"">

=3D --Apple-Mail=3D_F611F34D-66F8-4B72-8CA1-0836FCB767CA-- --===============3352768505449549620== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" Ci0tQXBwbGUtTWFpbD1fRjYxMUYzNEQtNjZGOC00QjcyLThDQTEtMDgzNkZDQjc2N0NBCkNvbnRl bnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKQ29udGVudC1UeXBlOiB0ZXh0 L3BsYWluOwoJY2hhcnNldD11dGYtOAoKCgo+IE9uIDE3IEFwciAyMDE4LCBhdCAxMToyOCwgU3Rl ZmFubyBTdGFnbmFybyA9CjxzdGVmYW5vc0BwcmlzbWF0ZWxlY29tdGVzdGluZy5jb20+IHdyb3Rl Ogo+PTIwCj4gT24gVGh1LCAyMDE4LTA0LTEyIGF0IDIwOjIwICswMjAwLCBNaWNoYWwgU2tyaXZh bmVrIHdyb3RlOgo+Pj0yMAo+Pj0yMAo+Pj4gT24gMTIgQXByIDIwMTgsIGF0IDE4OjI2LCBTdGVm YW5vIFN0YWduYXJvID0KPHN0ZWZhbm9zQHByaXNtYXRlbGVjb210ZXN0aW5nLmNvbSA9CjxtYWls dG86c3RlZmFub3NAcHJpc21hdGVsZWNvbXRlc3RpbmcuY29tPj4gd3JvdGU6Cj4+PiBIaSwKPj4+ PTIwCj4+PiBJIHJlY2VudGx5IHVwZ3JhZGVkIGFuIG9WaXJ0IGRlcGxveW1lbnQgZnJvbSAzLjYg dG8gNC4wIGFuZCB0aGVuID0KNC4xLjkgKG15IGFjdHVhbCByZWxlYXNlKS4gU2luY2UgdGhlbiwg d2hlbiBtaWdyYXRpbmcgbWFueSBob3N0cyA9CnNpbXVsdGFuZW91c2x5IEkgYWx3YXlzIGV4cGVy aWVuY2UgZmV3IG1pZ3JhdGlvbnMgZmFpbHVyZSBsaWtlIDEgb24gMTAgPQp2bXMuIFRoZSBmYWls dXJlIGNhbiBvY2N1ciBvbiBhbnkgaG9zdDsgbW9yZW92ZXIsIGFmdGVyIGEgY291cGxlIG9mID0K ZmFpbHVyZSB0aGUgZGVzdGluYXRpb24gaG9zdCBmYWxsIGluIEVycm9yIHN0YXR1cyBhbmQgSSBo YXZlIHRvIG1hbnVhbGx5ID0KcmUtYWN0aXZhdGUgb3Igd2FpdCAzMCBtaW4uCgpBcmUgdGhleSBh bGwgaW4gYSAzLjYgY2x1c3Rlcj89MjAKCj4+Pj0yMAo+Pj4gVGlwaWNhbCBlcnJvciBmb3VuZCBv biB2ZHNtIGxvZyBpcyAoZnJvbSB0aGUgc291cmNlIGhvc3QpOgo+Pj4gMjAxOC0wNC0xMiAxNzow MTozMiwwOTcrMDIwMCBFUlJPUiAobWlnc3JjLzMxOTJkZmU3KSBbdmlydC52bV0gPQoodm1JZD0z RCczMTkyZGZlNy1lZWFjLTQ2MjYtOGM4Ni1lNDlmYWNjOTAwNmYnKSBtaWdyYXRpb24gZGVzdGlu YXRpb24gPQplcnJvcjogRmF0YWwgZXJyb3IgZHVyaW5nIG1pZ3JhdGlvbiAobWlncmF0aW9uOjI4 NykKPj4+PTIwCj4+PiBQbGVhc2UgZmluZCB0aGUgbG9ncyBvZiBzb3VyY2UgaG9zdCAodjE1Lm92 aXJ0KSwgZGVzdGluYXRpb24gaG9zdCA9Cih2MTQub3ZpcnQpIGFuZCBlbmdpbmUgaGVyZTogPQpo dHRwczovL3d3dy5kcm9wYm94LmNvbS9zaC94aGY4cnk0aWg0MHBveGQvQUFCeGlGQ0l4RGUxNEhT eDJEcUxFNjF5YT9kbD0zRD0KMCA9CjxodHRwczovL3d3dy5kcm9wYm94LmNvbS9zaC94aGY4cnk0 aWg0MHBveGQvQUFCeGlGQ0l4RGUxNEhTeDJEcUxFNjF5YT9kbD0zRD0KMD4KPj4+PTIwCj4+PiBT b21lIG9mIHRoZSB2bSBhZmZlY3RlZCBmcm9tIHRoZSBtaWdyYXRpb24gZmFpbHVyZSBhcmU6Cj4+ PiBzdm4JMzE5MmRmZTctZWVhYy00NjI2LThjODYtZTQ5ZmFjYzkwMDZmCj4+PiB3b29kCWE4ZTgz ZmYwLWRmZWQtNDA3NC1iNmI2LWU5NDdiOGViYjk1Mgo+Pj4gcW54NjYJNTY5N2M0YTQtOWU0MC00 ZGQ2LWFiYTItYzhhYjk5MDRhNTg0Cj4+PTIwCj4+IGNhbiB5b3UgYWxzbyBpbmNsdWRlIHFlbXUg bG9nIGZyb20gL3Zhci9sb2cvbGlidmlydC9xZW11Lzx2bW5hbWU+Pwo+PTIwCj4gSGkgTWljaGFs LCBJJ3ZlIGFkZGVkIGxpYnZpcnQgbG9ncyBmb3IgcmVsZXZhbnQgVk1zIG9uIHRoZSBwcmV2aW91 cyA9CkRyb3Bib3ggc2hhcmUuCgpJIGRvIG5vdCBzZWUgYW55dGhpbmcgd3JvbmcuIEl0PUUyPTgw PTk5cyBhIGJpdCB0b28gbXVjaCBkYXRhIHRvIGdvID0KdGhyb3VnaCwgY2FuIHlvdSBwaW5wb2lu dCB0aGUgdGltZSBhbmQgVk0gbmFtZSB3aGVuIHlvdSBzZWUgYSBmYWlsdXJlPwoKVGhhbmtzLApt aWNoYWwKPj0yMAo+Pj0yMAo+PiBidHcgeW91IHNlZW0gdG8gYmUgdXNpbmcgdGhlIGxlZ2FjeSBt aWdyYXRpb24gcG9saWN5IHRocm90dGxpbmcgdGhlID0Kc3BlZWQgc2lnbmlmaWNhbnRseS4gUGxl YXNlIHJlYWQgaW50byB0aGUgbWlncmF0aW9uIGVuaGFuY2VtZW50cyBpbiA0LjAKPj4gPQpodHRw czovL3d3dy5vdmlydC5vcmcvZGV2ZWxvcC9yZWxlYXNlLW1hbmFnZW1lbnQvZmVhdHVyZXMvdmly dC9taWdyYXRpb24tZT0KbmhhbmNlbWVudHMvID0KPGh0dHBzOi8vd3d3Lm92aXJ0Lm9yZy9kZXZl bG9wL3JlbGVhc2UtbWFuYWdlbWVudC9mZWF0dXJlcy92aXJ0L21pZ3JhdGlvbi09CmVuaGFuY2Vt ZW50cy8+Cj4gSSd2ZSBhbHJlYWR5IG1vdmVkIHRvIE1pbmltYWwgRG93bnRpbWUgYW5kIHRoZW4g dG8gUG9zdC1jb3B5IHdpdGggc2FtZSA9CnJlc3VsdHMuIFZNIG1pZ3JhdGlvbnMgY29udGludWUg dG8gZmFpbCByYW5kb21seS4KPj0yMAo+Pj0yMAo+PiBUaGFua3MsCj4+IG1pY2hhbAo+PTIwCj4g VGhhbmtzLAo+IFN0ZWZhbm8uCj49MjAKPj0yMAo+Pj0yMAo+Pj49MjAKPj4+IFRoYW5rIHlvdSB2 ZXJ5IG11Y2ggZm9yIHlvdXIgaGVscC4KPj4+PTIwCj4+PiAtLT0yMAo+Pj4gU3RlZmFubyBTdGFn bmFybwo+Pj49MjAKPj4+IFByaXNtYSBUZWxlY29tIFRlc3RpbmcgUy5yLmwuCj4+PiBWaWEgUGV0 cm9jY2hpLCA0Cj4+PiAyMDEyNyBNaWxhbm8gPUUyPTgwPTkzIEl0YWx5Cj4+Pj0yMAo+Pj4gVGVs LiAwMiAyNjExMzUwNyBpbnQgMzM5Cj4+PiBlLW1haWw6IHN0ZWZhbm9zQHByaXNtYXRlbGVjb210 ZXN0aW5nLmNvbSA9CjxtYWlsdG86c3RlZmFub3NAcHJpc21hdGVsZWNvbXRlc3RpbmcuY29tPgo+ Pj4gc2t5cGU6IHN0ZWZhbm8uc3RhZ25hcm8KPj4+IF9fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fCj4+PiBVc2VycyBtYWlsaW5nIGxpc3QKPj4+IFVzZXJzQG92 aXJ0Lm9yZwo+Pj4gaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJz Cj4+PTIwCj49MjAKCgotLUFwcGxlLU1haWw9X0Y2MTFGMzRELTY2RjgtNEI3Mi04Q0ExLTA4MzZG Q0I3NjdDQQpDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiBxdW90ZWQtcHJpbnRhYmxlCkNvbnRl bnQtVHlwZTogdGV4dC9odG1sOwoJY2hhcnNldD11dGYtOAoKPGh0bWw+PGhlYWQ+PG1ldGEgaHR0 cC1lcXVpdj0zRCJDb250ZW50LVR5cGUiIGNvbnRlbnQ9M0QidGV4dC9odG1sOyA9CmNoYXJzZXQ9 M0R1dGYtOCI+PC9oZWFkPjxib2R5IHN0eWxlPTNEIndvcmQtd3JhcDogYnJlYWstd29yZDsgPQot d2Via2l0LW5ic3AtbW9kZTogc3BhY2U7IGxpbmUtYnJlYWs6IGFmdGVyLXdoaXRlLXNwYWNlOyIg Y2xhc3M9M0QiIj48YnIgPQpjbGFzcz0zRCIiPjxkaXY+PGJyIGNsYXNzPTNEIiI+PGJsb2NrcXVv dGUgdHlwZT0zRCJjaXRlIiBjbGFzcz0zRCIiPjxkaXYgPQpjbGFzcz0zRCIiPk9uIDE3IEFwciAy MDE4LCBhdCAxMToyOCwgU3RlZmFubyBTdGFnbmFybyAmbHQ7PGEgPQpocmVmPTNEIm1haWx0bzpz dGVmYW5vc0BwcmlzbWF0ZWxlY29tdGVzdGluZy5jb20iID0KY2xhc3M9M0QiIj5zdGVmYW5vc0Bw cmlzbWF0ZWxlY29tdGVzdGluZy5jb208L2E+Jmd0OyB3cm90ZTo8L2Rpdj48YnIgPQpjbGFzcz0z RCJBcHBsZS1pbnRlcmNoYW5nZS1uZXdsaW5lIj48ZGl2IGNsYXNzPTNEIiI+PG1ldGEgPQpodHRw LWVxdWl2PTNEIkNvbnRlbnQtVHlwZSIgY29udGVudD0zRCJ0ZXh0L2h0bWw7IGNoYXJzZXQ9M0R1 dGYtOCIgPQpjbGFzcz0zRCIiPjxkaXYgY2xhc3M9M0QiIj48ZGl2IGNsYXNzPTNEIiI+T24gVGh1 LCAyMDE4LTA0LTEyIGF0IDIwOjIwID0KKzAyMDAsIE1pY2hhbCBTa3JpdmFuZWsgd3JvdGU6PC9k aXY+PGJsb2NrcXVvdGUgdHlwZT0zRCJjaXRlIiA9CmNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+ PGRpdiBjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjxibG9ja3F1b3RlID0KdHlwZT0zRCJjaXRl IiBjbGFzcz0zRCIiPjxkaXYgY2xhc3M9M0QiIj5PbiAxMiBBcHIgMjAxOCwgYXQgMTg6MjYsID0K U3RlZmFubyBTdGFnbmFybyAmbHQ7PGEgaHJlZj0zRCJtYWlsdG86c3RlZmFub3NAcHJpc21hdGVs ZWNvbXRlc3RpbmcuY29tIiA9CmNsYXNzPTNEIiI+c3RlZmFub3NAcHJpc21hdGVsZWNvbXRlc3Rp bmcuY29tPC9hPiZndDsgd3JvdGU6PC9kaXY+PGRpdiA9CmNsYXNzPTNEIiI+PGRpdiBjbGFzcz0z RCIiPkhpLDxiciBjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPkkgcmVjZW50bHkgPQp1cGdyYWRl ZCBhbiBvVmlydCBkZXBsb3ltZW50IGZyb20gMy42IHRvIDQuMCBhbmQgdGhlbiA0LjEuOSAobXkg YWN0dWFsID0KcmVsZWFzZSkuIFNpbmNlIHRoZW4sIHdoZW4gbWlncmF0aW5nIG1hbnkgaG9zdHMg c2ltdWx0YW5lb3VzbHkgSSBhbHdheXMgPQpleHBlcmllbmNlIGZldyBtaWdyYXRpb25zIGZhaWx1 cmUgbGlrZSAxIG9uIDEwIHZtcy4gVGhlIGZhaWx1cmUgY2FuID0Kb2NjdXIgb24gYW55IGhvc3Q7 IG1vcmVvdmVyLCBhZnRlciBhIGNvdXBsZSBvZiBmYWlsdXJlIHRoZSBkZXN0aW5hdGlvbiA9Cmhv c3QgZmFsbCBpbiBFcnJvciBzdGF0dXMgYW5kIEkgaGF2ZSB0byBtYW51YWxseSByZS1hY3RpdmF0 ZSBvciB3YWl0IDMwID0KbWluLjxiciA9CmNsYXNzPTNEIiI+PC9kaXY+PC9kaXY+PC9ibG9ja3F1 b3RlPjwvZGl2PjwvYmxvY2txdW90ZT48L2Rpdj48L2Rpdj48L2Jsb2NrPQpxdW90ZT48ZGl2Pjxi ciBjbGFzcz0zRCIiPjwvZGl2PkFyZSB0aGV5IGFsbCBpbiBhIDMuNiA9CmNsdXN0ZXI/Jm5ic3A7 PC9kaXY+PGRpdj48YnIgY2xhc3M9M0QiIj48YmxvY2txdW90ZSB0eXBlPTNEImNpdGUiID0KY2xh c3M9M0QiIj48ZGl2IGNsYXNzPTNEIiI+PGRpdiBjbGFzcz0zRCIiPjxibG9ja3F1b3RlIHR5cGU9 M0QiY2l0ZSIgPQpjbGFzcz0zRCIiPjxkaXYgY2xhc3M9M0QiIj48YmxvY2txdW90ZSB0eXBlPTNE ImNpdGUiIGNsYXNzPTNEIiI+PGRpdiA9CmNsYXNzPTNEIiI+PGRpdiBjbGFzcz0zRCIiPjxiciBj bGFzcz0zRCIiPlRpcGljYWwgZXJyb3IgZm91bmQgb24gdmRzbSA9CmxvZyBpcyAoZnJvbSB0aGUg c291cmNlIGhvc3QpOjxiciBjbGFzcz0zRCIiPjIwMTgtMDQtMTIgPQoxNzowMTozMiwwOTcrMDIw MCBFUlJPUiAobWlnc3JjLzMxOTJkZmU3KSBbdmlydC52bV0gPQoodm1JZD0zRCczMTkyZGZlNy1l ZWFjLTQ2MjYtOGM4Ni1lNDlmYWNjOTAwNmYnKSBtaWdyYXRpb24gZGVzdGluYXRpb24gPQplcnJv cjogRmF0YWwgZXJyb3IgZHVyaW5nIG1pZ3JhdGlvbiAobWlncmF0aW9uOjI4Nyk8YnIgY2xhc3M9 M0QiIj48YnIgPQpjbGFzcz0zRCIiPlBsZWFzZSBmaW5kIHRoZSBsb2dzIG9mIHNvdXJjZSBob3N0 ICh2MTUub3ZpcnQpLCBkZXN0aW5hdGlvbiA9Cmhvc3QgKHYxNC5vdmlydCkgYW5kIGVuZ2luZSBo ZXJlOiA8YSA9CmhyZWY9M0QiaHR0cHM6Ly93d3cuZHJvcGJveC5jb20vc2gveGhmOHJ5NGloNDBw b3hkL0FBQnhpRkNJeERlMTRIU3gyRHFMRTYxPQp5YT9kbD0zRDAiID0KY2xhc3M9M0QiIj5odHRw czovL3d3dy5kcm9wYm94LmNvbS9zaC94aGY4cnk0aWg0MHBveGQvQUFCeGlGQ0l4RGUxNEhTeDJE cUw9CkU2MXlhP2RsPTNEMDwvYT48YnIgY2xhc3M9M0QiIj48YnIgY2xhc3M9M0QiIj5Tb21lIG9m IHRoZSB2bSBhZmZlY3RlZCA9CmZyb20gdGhlIG1pZ3JhdGlvbiBmYWlsdXJlIGFyZTo8YnIgY2xh c3M9M0QiIj5zdm4JPQozMTkyZGZlNy1lZWFjLTQ2MjYtOGM4Ni1lNDlmYWNjOTAwNmY8YnIgY2xh c3M9M0QiIj53b29kCT0KYThlODNmZjAtZGZlZC00MDc0LWI2YjYtZTk0N2I4ZWJiOTUyPGJyIGNs YXNzPTNEIiI+cW54NjYJPQo1Njk3YzRhNC05ZTQwLTRkZDYtYWJhMi1jOGFiOTkwNGE1ODQ8YnIg PQpjbGFzcz0zRCIiPjwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2IGNsYXNzPTNEIiI+PGJy ID0KY2xhc3M9M0QiIj48L2Rpdj5jYW4geW91IGFsc28gaW5jbHVkZSBxZW11IGxvZyBmcm9tID0K L3Zhci9sb2cvbGlidmlydC9xZW11LyZsdDt2bW5hbWUmZ3Q7PzwvZGl2PjwvYmxvY2txdW90ZT48 ZGl2ID0KY2xhc3M9M0QiIj48YnIgY2xhc3M9M0QiIj48L2Rpdj48ZGl2IGNsYXNzPTNEIiI+SGkg TWljaGFsLCBJJ3ZlIGFkZGVkID0KbGlidmlydCBsb2dzIGZvciByZWxldmFudCBWTXMgb24gdGhl IHByZXZpb3VzIERyb3Bib3ggPQpzaGFyZS48L2Rpdj48L2Rpdj48L2Rpdj48L2Jsb2NrcXVvdGU+ PGRpdj48YnIgY2xhc3M9M0QiIj48L2Rpdj5JIGRvIG5vdCA9CnNlZSBhbnl0aGluZyB3cm9uZy4g SXQ9RTI9ODA9OTlzIGEgYml0IHRvbyBtdWNoIGRhdGEgdG8gZ28gdGhyb3VnaCwgY2FuID0KeW91 IHBpbnBvaW50IHRoZSB0aW1lIGFuZCBWTSBuYW1lIHdoZW4geW91IHNlZSBhIGZhaWx1cmU/PC9k aXY+PGRpdj48YnIgPQpjbGFzcz0zRCIiPjwvZGl2PjxkaXY+VGhhbmtzLDwvZGl2PjxkaXY+bWlj aGFsPGJyIGNsYXNzPTNEIiI+PGJsb2NrcXVvdGUgPQp0eXBlPTNEImNpdGUiIGNsYXNzPTNEIiI+ PGRpdiBjbGFzcz0zRCIiPjxkaXYgY2xhc3M9M0QiIj48ZGl2ID0KY2xhc3M9M0QiIj48YnIgY2xh c3M9M0QiIj48L2Rpdj48YmxvY2txdW90ZSB0eXBlPTNEImNpdGUiIGNsYXNzPTNEIiI+PGRpdiA9 CmNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PC9kaXY+PGRpdiBjbGFzcz0zRCIiPmJ0dyB5b3Ug c2VlbSB0byBiZSB1c2luZyA9CnRoZSBsZWdhY3kgbWlncmF0aW9uIHBvbGljeSB0aHJvdHRsaW5n IHRoZSBzcGVlZCBzaWduaWZpY2FudGx5LiBQbGVhc2UgPQpyZWFkIGludG8gdGhlIG1pZ3JhdGlv biBlbmhhbmNlbWVudHMgaW4gNC4wPC9kaXY+PGRpdiBjbGFzcz0zRCIiPjxhID0KaHJlZj0zRCJo dHRwczovL3d3dy5vdmlydC5vcmcvZGV2ZWxvcC9yZWxlYXNlLW1hbmFnZW1lbnQvZmVhdHVyZXMv dmlydC9taWc9CnJhdGlvbi1lbmhhbmNlbWVudHMvIiA9CmNsYXNzPTNEIiI+aHR0cHM6Ly93d3cu b3ZpcnQub3JnL2RldmVsb3AvcmVsZWFzZS1tYW5hZ2VtZW50L2ZlYXR1cmVzL3ZpcnQvPQptaWdy YXRpb24tZW5oYW5jZW1lbnRzLzwvYT48L2Rpdj48L2Jsb2NrcXVvdGU+PGRpdiBjbGFzcz0zRCIi PjxiciA9CmNsYXNzPTNEIiI+PC9kaXY+PGRpdiBjbGFzcz0zRCIiPkkndmUgYWxyZWFkeSBtb3Zl ZCB0byBNaW5pbWFsIERvd250aW1lID0KYW5kIHRoZW4gdG8gUG9zdC1jb3B5IHdpdGggc2FtZSBy ZXN1bHRzLiBWTSBtaWdyYXRpb25zIGNvbnRpbnVlIHRvIGZhaWwgPQpyYW5kb21seS48L2Rpdj48 ZGl2IGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PC9kaXY+PGJsb2NrcXVvdGUgPQp0eXBlPTNE ImNpdGUiIGNsYXNzPTNEIiI+PGRpdiBjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjwvZGl2Pjxk aXYgPQpjbGFzcz0zRCIiPlRoYW5rcyw8L2Rpdj48ZGl2IGNsYXNzPTNEIiI+bWljaGFsPC9kaXY+ PC9ibG9ja3F1b3RlPjxkaXYgPQpjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjwvZGl2PjxkaXYg Y2xhc3M9M0QiIj5UaGFua3MsPC9kaXY+PGRpdiA9CmNsYXNzPTNEIiI+U3RlZmFuby48L2Rpdj48 ZGl2IGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PC9kaXY+PGRpdiA9CmNsYXNzPTNEIiI+PGJy IGNsYXNzPTNEIiI+PC9kaXY+PGJsb2NrcXVvdGUgdHlwZT0zRCJjaXRlIiBjbGFzcz0zRCIiPjxk aXYgPQpjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjxibG9ja3F1b3RlIHR5cGU9M0QiY2l0ZSIg Y2xhc3M9M0QiIj48ZGl2ID0KY2xhc3M9M0QiIj48ZGl2IGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNE IiI+VGhhbmsgeW91IHZlcnkgbXVjaCBmb3IgeW91ciA9CmhlbHAuPGJyIGNsYXNzPTNEIiI+PGJy IGNsYXNzPTNEIiI+LS0gPGJyIGNsYXNzPTNEIiI+U3RlZmFubyBTdGFnbmFybzxiciA9CmNsYXNz PTNEIiI+PGJyIGNsYXNzPTNEIiI+UHJpc21hIFRlbGVjb20gVGVzdGluZyBTLnIubC48YnIgY2xh c3M9M0QiIj5WaWEgPQpQZXRyb2NjaGksIDQ8YnIgY2xhc3M9M0QiIj4yMDEyNyBNaWxhbm8gPUUy PTgwPTkzIEl0YWx5PGJyIGNsYXNzPTNEIiI+PGJyID0KY2xhc3M9M0QiIj5UZWwuIDAyIDI2MTEz NTA3IGludCAzMzk8YnIgY2xhc3M9M0QiIj48YSA9CmhyZWY9M0QibWFpbHRvOnN0ZWZhbm9zQHBy aXNtYXRlbGVjb210ZXN0aW5nLmNvbSIgY2xhc3M9M0QiIj5lLW1haWw6ID0Kc3RlZmFub3NAcHJp c21hdGVsZWNvbXRlc3RpbmcuY29tPC9hPjxiciBjbGFzcz0zRCIiPnNreXBlOiA9CnN0ZWZhbm8u c3RhZ25hcm88YnIgPQpjbGFzcz0zRCIiPl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fPGJyID0KY2xhc3M9M0QiIj5Vc2VycyBtYWlsaW5nIGxpc3Q8YnIgY2xh c3M9M0QiIj48YSA9CmhyZWY9M0QibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgY2xhc3M9M0QiIj5V c2Vyc0BvdmlydC5vcmc8L2E+PGJyID0KY2xhc3M9M0QiIj5odHRwOi8vbGlzdHMub3ZpcnQub3Jn L21haWxtYW4vbGlzdGluZm8vdXNlcnM8YnIgPQpjbGFzcz0zRCIiPjwvZGl2PjwvZGl2PjwvYmxv Y2txdW90ZT48L2Rpdj48YnIgY2xhc3M9M0QiIj48L2Jsb2NrcXVvdGU+PGRpdj0KIGNsYXNzPTNE IiI+PHNwYW4gY2xhc3M9M0QiIj48cHJlIGNsYXNzPTNEIiI+PHByZSBjbGFzcz0zRCIiPjxiciA9 CmNsYXNzPTNEIiI+PC9wcmU+PC9wcmU+PC9zcGFuPjwvZGl2PjwvZGl2PjwvZGl2PjwvYmxvY2tx dW90ZT48L2Rpdj48YnIgPQpjbGFzcz0zRCIiPjwvYm9keT48L2h0bWw+PQoKLS1BcHBsZS1NYWls PV9GNjExRjM0RC02NkY4LTRCNzItOENBMS0wODM2RkNCNzY3Q0EtLQo= --===============3352768505449549620==--