From ravishankar at redhat.com Fri Jul 21 11:10:49 2017 Content-Type: multipart/mixed; boundary="===============2853443058635076698==" MIME-Version: 1.0 From: Ravishankar N To: users at ovirt.org Subject: Re: [ovirt-users] ovirt 4.1 hosted engine hyper converged on glusterfs 3.8.10 : "engine" storage domain alway complain about "unsynced" elements Date: Fri, 21 Jul 2017 16:40:40 +0530 Message-ID: <83e15279-b42f-0947-7745-89d16bc52bc2@redhat.com> In-Reply-To: CAGK=3kx=XufQ3_bbmF1+R3g+gqCa-nTUf05KNbA00eBcHM92+g@mail.gmail.com --===============2853443058635076698== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------84EB6DEB84AAFAA57D0538D6 Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 7bit On 07/21/2017 02:55 PM, yayo (j) wrote: > 2017-07-20 14:48 GMT+02:00 Ravishankar N >: > > > But it does say something. All these gfids of completed heals in > the log below are the for the ones that you have given the > getfattr output of. So what is likely happening is there is an > intermittent connection problem between your mount and the brick > process, leading to pending heals again after the heal gets > completed, which is why the numbers are varying each time. You > would need to check why that is the case. > Hope this helps, > Ravi > > >> >> /[2017-07-20 09:58:46.573079] I [MSGID: 108026] >> [afr-self-heal-common.c:1254:afr_log_selfheal] >> 0-engine-replicate-0: Completed data selfheal on >> e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=3D[0] 1 sinks=3D2/ >> /[2017-07-20 09:59:22.995003] I [MSGID: 108026] >> [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] >> 0-engine-replicate-0: performing metadata selfheal on >> f05b9742-2771-484a-85fc-5b6974bcef81/ >> /[2017-07-20 09:59:22.999372] I [MSGID: 108026] >> [afr-self-heal-common.c:1254:afr_log_selfheal] >> 0-engine-replicate-0: Completed metadata selfheal on >> f05b9742-2771-484a-85fc-5b6974bcef81. sources=3D[0] 1 sinks=3D2/ >> > > > Hi, > > But we ha1e 2 gluster volume on the same network and the other one = > (the "Data" gluster) don't have any problems. Why you think there is a = > network problem? Because pending self-heals come into the picture when I/O from the = clients (mounts) do not succeed on some bricks. They are mostly due to (a) the client losing connection to some bricks (likely), (b) the I/O failing on the bricks themselves (unlikely). If most of the i/o is also going to the 3rd brick (since you say the = files are already present on all bricks and I/O is successful) , then it = is likely to be (a). > How to check this on a gluster infrastructure? > In the fuse mount logs for the engine volume, check if there are any = messages for brick disconnects. Something along the lines of = "disconnected from volname-client-x". Just guessing here, but maybe even the 'data' volume did experience = disconnects and self-heals later but you did not observe it when you ran = heal info. See the glustershd log or mount log for for self-heal = completion messages on /0-data-replicate-0 /also. Regards, Ravi > Thank you > > > --------------84EB6DEB84AAFAA57D0538D6 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit

On 07/21/2017 02:55 PM, yayo (j) wrote:<= br>

2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishankar(a)redhat.com>:

But it does=C2=A0 say something. All these gfids of completed heals in the log below are the for the ones that you have given the getfattr output of. So what is likely happening is there is an intermittent connection problem between your mount and the brick process, leading to pending heals again after the heal gets completed, which is why the numbers are varying each time. You would need to check why that is the case.
Hope this helps,
Ravi

[2017-07-20 09:58:46.573079] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_l= og_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd743= 27. sources=3D[0] 1 =C2=A0sinks=3D2

[2017-07-20 09:59:22.995003] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr= _selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef= 81

[2017-07-20 09:59:22.999372] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_l= og_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef= 81. sources=3D[0] 1 =C2=A0sinks=3D2

Hi,

But we ha1e 2 gluster volume on the same network and the other one (the "Data" gluster) don't have any problems. Why you think there is a network problem?

Because pending self-heals come into the picture when I/O from the clients (mounts) do not succeed on some bricks. They are mostly due to
(a) the client losing connection to some bricks (likely),
(b) the I/O failing on the bricks themselves (unlikely).

If most of the i/o is also going to the 3rd brick (since you say the files are already present on all bricks and I/O is successful) , then it is likely to be (a).

=C2=A0 How to check this on a gluster infrastructure?

In the fuse mount logs for the engine volume, check if there are any messages for brick disconnects. Something along the lines of "disconnected from volname-client-x".
Just guessing here, but maybe even the 'data' volume did experience disconnects and self-heals later but you did not observe it when you ran heal info. See the glustershd log or mount log for for self-heal completion messages on 0-data-replicate-0 also.

Regards,
Ravi

Thank you

--------------84EB6DEB84AAFAA57D0538D6-- --===============2853443058635076698== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS04NEVCNkRFQjg0QUFGQUE1N0QwNTM4RDYKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDdiaXQK CgpPbiAwNy8yMS8yMDE3IDAyOjU1IFBNLCB5YXlvIChqKSB3cm90ZToKPiAyMDE3LTA3LTIwIDE0 OjQ4IEdNVCswMjowMCBSYXZpc2hhbmthciBOIDxyYXZpc2hhbmthckByZWRoYXQuY29tIAo+IDxt YWlsdG86cmF2aXNoYW5rYXJAcmVkaGF0LmNvbT4+Ogo+Cj4KPiAgICAgQnV0IGl0IGRvZXMgIHNh eSBzb21ldGhpbmcuIEFsbCB0aGVzZSBnZmlkcyBvZiBjb21wbGV0ZWQgaGVhbHMgaW4KPiAgICAg dGhlIGxvZyBiZWxvdyBhcmUgdGhlIGZvciB0aGUgb25lcyB0aGF0IHlvdSBoYXZlIGdpdmVuIHRo ZQo+ICAgICBnZXRmYXR0ciBvdXRwdXQgb2YuIFNvIHdoYXQgaXMgbGlrZWx5IGhhcHBlbmluZyBp cyB0aGVyZSBpcyBhbgo+ICAgICBpbnRlcm1pdHRlbnQgY29ubmVjdGlvbiBwcm9ibGVtIGJldHdl ZW4geW91ciBtb3VudCBhbmQgdGhlIGJyaWNrCj4gICAgIHByb2Nlc3MsIGxlYWRpbmcgdG8gcGVu ZGluZyBoZWFscyBhZ2FpbiBhZnRlciB0aGUgaGVhbCBnZXRzCj4gICAgIGNvbXBsZXRlZCwgd2hp Y2ggaXMgd2h5IHRoZSBudW1iZXJzIGFyZSB2YXJ5aW5nIGVhY2ggdGltZS4gWW91Cj4gICAgIHdv dWxkIG5lZWQgdG8gY2hlY2sgd2h5IHRoYXQgaXMgdGhlIGNhc2UuCj4gICAgIEhvcGUgdGhpcyBo ZWxwcywKPiAgICAgUmF2aQo+Cj4KPj4KPj4gICAgICAgICAvWzIwMTctMDctMjAgMDk6NTg6NDYu NTczMDc5XSBJIFtNU0dJRDogMTA4MDI2XQo+PiAgICAgICAgIFthZnItc2VsZi1oZWFsLWNvbW1v bi5jOjEyNTQ6YWZyX2xvZ19zZWxmaGVhbF0KPj4gICAgICAgICAwLWVuZ2luZS1yZXBsaWNhdGUt MDogQ29tcGxldGVkIGRhdGEgc2VsZmhlYWwgb24KPj4gICAgICAgICBlNmRmZDU1Ni0zNDBiLTRi NzYtYjQ3Yi03YjZmNWJkNzQzMjcuIHNvdXJjZXM9WzBdIDEgIHNpbmtzPTIvCj4+ICAgICAgICAg L1syMDE3LTA3LTIwIDA5OjU5OjIyLjk5NTAwM10gSSBbTVNHSUQ6IDEwODAyNl0KPj4gICAgICAg ICBbYWZyLXNlbGYtaGVhbC1tZXRhZGF0YS5jOjUxOl9fYWZyX3NlbGZoZWFsX21ldGFkYXRhX2Rv XQo+PiAgICAgICAgIDAtZW5naW5lLXJlcGxpY2F0ZS0wOiBwZXJmb3JtaW5nIG1ldGFkYXRhIHNl bGZoZWFsIG9uCj4+ICAgICAgICAgZjA1Yjk3NDItMjc3MS00ODRhLTg1ZmMtNWI2OTc0YmNlZjgx Lwo+PiAgICAgICAgIC9bMjAxNy0wNy0yMCAwOTo1OToyMi45OTkzNzJdIEkgW01TR0lEOiAxMDgw MjZdCj4+ICAgICAgICAgW2Fmci1zZWxmLWhlYWwtY29tbW9uLmM6MTI1NDphZnJfbG9nX3NlbGZo ZWFsXQo+PiAgICAgICAgIDAtZW5naW5lLXJlcGxpY2F0ZS0wOiBDb21wbGV0ZWQgbWV0YWRhdGEg c2VsZmhlYWwgb24KPj4gICAgICAgICBmMDViOTc0Mi0yNzcxLTQ4NGEtODVmYy01YjY5NzRiY2Vm ODEuIHNvdXJjZXM9WzBdIDEgIHNpbmtzPTIvCj4+Cj4KPgo+IEhpLAo+Cj4gQnV0IHdlIGhhMWUg MiBnbHVzdGVyIHZvbHVtZSBvbiB0aGUgc2FtZSBuZXR3b3JrIGFuZCB0aGUgb3RoZXIgb25lIAo+ ICh0aGUgIkRhdGEiIGdsdXN0ZXIpIGRvbid0IGhhdmUgYW55IHByb2JsZW1zLiBXaHkgeW91IHRo aW5rIHRoZXJlIGlzIGEgCj4gbmV0d29yayBwcm9ibGVtPwoKQmVjYXVzZSBwZW5kaW5nIHNlbGYt aGVhbHMgY29tZSBpbnRvIHRoZSBwaWN0dXJlIHdoZW4gSS9PIGZyb20gdGhlIApjbGllbnRzICht b3VudHMpIGRvIG5vdCBzdWNjZWVkIG9uIHNvbWUgYnJpY2tzLiBUaGV5IGFyZSBtb3N0bHkgZHVl IHRvCihhKSB0aGUgY2xpZW50IGxvc2luZyBjb25uZWN0aW9uIHRvIHNvbWUgYnJpY2tzIChsaWtl bHkpLAooYikgdGhlIEkvTyBmYWlsaW5nIG9uIHRoZSBicmlja3MgdGhlbXNlbHZlcyAodW5saWtl bHkpLgoKSWYgbW9zdCBvZiB0aGUgaS9vIGlzIGFsc28gZ29pbmcgdG8gdGhlIDNyZCBicmljayAo c2luY2UgeW91IHNheSB0aGUgCmZpbGVzIGFyZSBhbHJlYWR5IHByZXNlbnQgb24gYWxsIGJyaWNr cyBhbmQgSS9PIGlzIHN1Y2Nlc3NmdWwpICwgdGhlbiBpdCAKaXMgbGlrZWx5IHRvIGJlIChhKS4K Cj4gSG93IHRvIGNoZWNrIHRoaXMgb24gYSBnbHVzdGVyIGluZnJhc3RydWN0dXJlPwo+CkluIHRo ZSBmdXNlIG1vdW50IGxvZ3MgZm9yIHRoZSBlbmdpbmUgdm9sdW1lLCBjaGVjayBpZiB0aGVyZSBh cmUgYW55IAptZXNzYWdlcyBmb3IgYnJpY2sgZGlzY29ubmVjdHMuIFNvbWV0aGluZyBhbG9uZyB0 aGUgbGluZXMgb2YgCiJkaXNjb25uZWN0ZWQgZnJvbSB2b2xuYW1lLWNsaWVudC14Ii4KSnVzdCBn dWVzc2luZyBoZXJlLCBidXQgbWF5YmUgZXZlbiB0aGUgJ2RhdGEnIHZvbHVtZSBkaWQgZXhwZXJp ZW5jZSAKZGlzY29ubmVjdHMgYW5kIHNlbGYtaGVhbHMgbGF0ZXIgYnV0IHlvdSBkaWQgbm90IG9i c2VydmUgaXQgd2hlbiB5b3UgcmFuIApoZWFsIGluZm8uIFNlZSB0aGUgZ2x1c3RlcnNoZCBsb2cg b3IgbW91bnQgbG9nIGZvciBmb3Igc2VsZi1oZWFsIApjb21wbGV0aW9uIG1lc3NhZ2VzIG9uIC8w LWRhdGEtcmVwbGljYXRlLTAgL2Fsc28uCgpSZWdhcmRzLApSYXZpCj4gVGhhbmsgeW91Cj4KPgo+ CgoKLS0tLS0tLS0tLS0tLS04NEVCNkRFQjg0QUFGQUE1N0QwNTM4RDYKQ29udGVudC1UeXBlOiB0 ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgKQ29udGVudC1UcmFuc2Zlci1FbmNvZGluZzogOGJpdAoK PGh0bWw+CiAgPGhlYWQ+CiAgICA8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNvbnRl bnQ9InRleHQvaHRtbDsgY2hhcnNldD11dGYtOCI+CiAgPC9oZWFkPgogIDxib2R5IHRleHQ9IiMw MDAwMDAiIGJnY29sb3I9IiNGRkZGRkYiPgogICAgPGJyPgogICAgPGRpdiBjbGFzcz0ibW96LWNp dGUtcHJlZml4Ij5PbiAwNy8yMS8yMDE3IDAyOjU1IFBNLCB5YXlvIChqKSB3cm90ZTo8YnI+CiAg ICA8L2Rpdj4KICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiCmNpdGU9Im1pZDpDQUdLPTNreD1Y dWZRM19iYm1GMStSM2crZ3FDYS1uVFVmMDVLTmJBMDBlQmNITTkyK2dAbWFpbC5nbWFpbC5jb20i PgogICAgICA8ZGl2IGRpcj0ibHRyIj4KICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+ CiAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+MjAxNy0wNy0yMCAxNDo0OCBHTVQr MDI6MDAKICAgICAgICAgICAgUmF2aXNoYW5rYXIgTiA8c3BhbiBkaXI9Imx0ciI+Jmx0OzxhCiAg ICAgICAgICAgICAgICBocmVmPSJtYWlsdG86cmF2aXNoYW5rYXJAcmVkaGF0LmNvbSIgdGFyZ2V0 PSJfYmxhbmsiCiAgICAgICAgICAgICAgICBtb3otZG8tbm90LXNlbmQ9InRydWUiPnJhdmlzaGFu a2FyQHJlZGhhdC5jb208L2E+Jmd0Ozwvc3Bhbj46PGJyPgogICAgICAgICAgICA8YmxvY2txdW90 ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAKICAgICAgICAgICAgICAu OGV4O2JvcmRlci1sZWZ0OjFweCAjY2NjIHNvbGlkO3BhZGRpbmctbGVmdDoxZXgiPgogICAgICAg ICAgICAgIDxkaXYgdGV4dD0iIzAwMDAwMCIgYmdjb2xvcj0iI0ZGRkZGRiI+PHNwYW4gY2xhc3M9 IiI+CiAgICAgICAgICAgICAgICAgIDxwPjxicj4KICAgICAgICAgICAgICAgICAgPC9wPgogICAg ICAgICAgICAgICAgPC9zcGFuPiBCdXQgaXQgZG9lc8KgIHNheSBzb21ldGhpbmcuIEFsbCB0aGVz ZSBnZmlkcyBvZgogICAgICAgICAgICAgICAgY29tcGxldGVkIGhlYWxzIGluIHRoZSBsb2cgYmVs b3cgYXJlIHRoZSBmb3IgdGhlIG9uZXMKICAgICAgICAgICAgICAgIHRoYXQgeW91IGhhdmUgZ2l2 ZW4gdGhlIGdldGZhdHRyIG91dHB1dCBvZi4gU28gd2hhdCBpcwogICAgICAgICAgICAgICAgbGlr ZWx5IGhhcHBlbmluZyBpcyB0aGVyZSBpcyBhbiBpbnRlcm1pdHRlbnQgY29ubmVjdGlvbgogICAg ICAgICAgICAgICAgcHJvYmxlbSBiZXR3ZWVuIHlvdXIgbW91bnQgYW5kIHRoZSBicmljayBwcm9j ZXNzLAogICAgICAgICAgICAgICAgbGVhZGluZyB0byBwZW5kaW5nIGhlYWxzIGFnYWluIGFmdGVy IHRoZSBoZWFsIGdldHMKICAgICAgICAgICAgICAgIGNvbXBsZXRlZCwgd2hpY2ggaXMgd2h5IHRo ZSBudW1iZXJzIGFyZSB2YXJ5aW5nIGVhY2gKICAgICAgICAgICAgICAgIHRpbWUuIFlvdSB3b3Vs ZCBuZWVkIHRvIGNoZWNrIHdoeSB0aGF0IGlzIHRoZSBjYXNlLjxicj4KICAgICAgICAgICAgICAg IEhvcGUgdGhpcyBoZWxwcyw8YnI+CiAgICAgICAgICAgICAgICBSYXZpCiAgICAgICAgICAgICAg ICA8ZGl2PgogICAgICAgICAgICAgICAgICA8ZGl2IGNsYXNzPSJoNSI+PGJyPgogICAgICAgICAg ICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRl Ij4KICAgICAgICAgICAgICAgICAgICAgIDxkaXYgZGlyPSJsdHIiPgogICAgICAgICAgICAgICAg ICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+CiAgICAgICAgICAgICAgICAgICAgICAg ICAgPGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPgogICAgICAgICAgICAgICAgICAgICAgICAgICAg PGRpdj48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAg ICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAg ICAgICAgICAgICAgICAgICAgIDxibG9ja3F1b3RlIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHgK ICAgICAgICAgICAgICAgICAgICAgICAgICA0MHB4O2JvcmRlcjpub25lO3BhZGRpbmc6MHB4Ij4K ICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+CiAgICAg ICAgICAgICAgICAgICAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+CiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIDxkaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgPGRpdj48aT5bMjAxNy0wNy0yMCAwOTo1ODo0Ni41NzMwNzldIEkKICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgW01TR0lEOiAxMDgwMjZdCiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIFthZnItc2VsZi1oZWFsLWNvbW1vbi5jOjEyNTQ6YTx3YnI+ZnJfbG9n X3NlbGZoZWFsXQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAwLWVuZ2luZS1y ZXBsaWNhdGUtMDogQ29tcGxldGVkIGRhdGEKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgc2VsZmhlYWwgb24KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZTZk ZmQ1NTYtMzQwYi00Yjc2LWI0N2ItN2I2ZjViPHdicj5kNzQzMjcuCiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIHNvdXJjZXM9WzBdIDEgwqBzaW5rcz0yPC9pPjwvZGl2PgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAg ICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAg ICAgICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj4KICAgICAgICAgICAgICAgICAg ICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj4KICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgPGRpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2PjxpPlsy MDE3LTA3LTIwIDA5OjU5OjIyLjk5NTAwM10gSQogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICBbTVNHSUQ6IDEwODAyNl0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgW2Fmci1zZWxmLWhlYWwtbWV0YWRhdGEuYzo1MTpfPHdicj5fYWZyX3NlbGZoZWFsX21ldGFk YXRhX2RvXQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAwLWVuZ2luZS1yZXBs aWNhdGUtMDogcGVyZm9ybWluZwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBt ZXRhZGF0YSBzZWxmaGVhbCBvbgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBm MDViOTc0Mi0yNzcxLTQ4NGEtODVmYy01YjY5NzQ8d2JyPmJjZWY4MTwvaT48L2Rpdj4KICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAg ICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAg ICAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+CiAgICAgICAgICAgICAgICAgICAg ICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+CiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIDxkaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdj48aT5bMjAx Ny0wNy0yMCAwOTo1OToyMi45OTkzNzJdIEkKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgW01TR0lEOiAxMDgwMjZdCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IFthZnItc2VsZi1oZWFsLWNvbW1vbi5jOjEyNTQ6YTx3YnI+ZnJfbG9nX3NlbGZoZWFsXQogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAwLWVuZ2luZS1yZXBsaWNhdGUtMDogQ29t cGxldGVkCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1ldGFkYXRhIHNlbGZo ZWFsIG9uCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGYwNWI5NzQyLTI3NzEt NDg0YS04NWZjLTViNjk3NDx3YnI+YmNlZjgxLgogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICBzb3VyY2VzPVswXSAxIMKgc2lua3M9MjwvaT48L2Rpdj4KICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4K ICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAg PC9ibG9ja3F1b3RlPgogICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAg ICAgICAgPC9ibG9ja3F1b3RlPgogICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAg ICAgIDwvZGl2PgogICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICA8L2Jsb2NrcXVvdGU+ CiAgICAgICAgICA8L2Rpdj4KICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX3NpZ25hdHVyZSIg ZGF0YS1zbWFydG1haWw9ImdtYWlsX3NpZ25hdHVyZSI+PGJyPgogICAgICAgICAgPC9kaXY+CiAg ICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9zaWduYXR1cmUiIGRhdGEtc21hcnRtYWlsPSJnbWFp bF9zaWduYXR1cmUiPjxicj4KICAgICAgICAgIDwvZGl2PgogICAgICAgICAgPGRpdiBjbGFzcz0i Z21haWxfc2lnbmF0dXJlIiBkYXRhLXNtYXJ0bWFpbD0iZ21haWxfc2lnbmF0dXJlIj5IaSw8L2Rp dj4KICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX3NpZ25hdHVyZSIgZGF0YS1zbWFydG1haWw9 ImdtYWlsX3NpZ25hdHVyZSI+PGJyPgogICAgICAgICAgPC9kaXY+CiAgICAgICAgICA8ZGl2IGNs YXNzPSJnbWFpbF9zaWduYXR1cmUiIGRhdGEtc21hcnRtYWlsPSJnbWFpbF9zaWduYXR1cmUiPkJ1 dAogICAgICAgICAgICB3ZSBoYTFlIDIgZ2x1c3RlciB2b2x1bWUgb24gdGhlIHNhbWUgbmV0d29y ayBhbmQgdGhlIG90aGVyCiAgICAgICAgICAgIG9uZSAodGhlICJEYXRhIiBnbHVzdGVyKSBkb24n dCBoYXZlIGFueSBwcm9ibGVtcy4gV2h5IHlvdQogICAgICAgICAgICB0aGluayB0aGVyZSBpcyBh IG5ldHdvcmsgcHJvYmxlbT88L2Rpdj4KICAgICAgICA8L2Rpdj4KICAgICAgPC9kaXY+CiAgICA8 L2Jsb2NrcXVvdGU+CiAgICA8YnI+CiAgICBCZWNhdXNlIHBlbmRpbmcgc2VsZi1oZWFscyBjb21l IGludG8gdGhlIHBpY3R1cmUgd2hlbiBJL08gZnJvbSB0aGUKICAgIGNsaWVudHMgKG1vdW50cykg ZG8gbm90IHN1Y2NlZWQgb24gc29tZSBicmlja3MuIFRoZXkgYXJlIG1vc3RseSBkdWUKICAgIHRv IDxicj4KICAgIChhKSB0aGUgY2xpZW50IGxvc2luZyBjb25uZWN0aW9uIHRvIHNvbWUgYnJpY2tz IChsaWtlbHkpLDxicj4KICAgIChiKSB0aGUgSS9PIGZhaWxpbmcgb24gdGhlIGJyaWNrcyB0aGVt c2VsdmVzICh1bmxpa2VseSkuPGJyPgogICAgPGJyPgogICAgSWYgbW9zdCBvZiB0aGUgaS9vIGlz IGFsc28gZ29pbmcgdG8gdGhlIDNyZCBicmljayAoc2luY2UgeW91IHNheSB0aGUKICAgIGZpbGVz IGFyZSBhbHJlYWR5IHByZXNlbnQgb24gYWxsIGJyaWNrcyBhbmQgSS9PIGlzIHN1Y2Nlc3NmdWwp ICwKICAgIHRoZW4gaXQgaXMgbGlrZWx5IHRvIGJlIChhKS48YnI+CiAgICA8YnI+CiAgICA8Ymxv Y2txdW90ZSB0eXBlPSJjaXRlIgpjaXRlPSJtaWQ6Q0FHSz0za3g9WHVmUTNfYmJtRjErUjNnK2dx Q2EtblRVZjA1S05iQTAwZUJjSE05MitnQG1haWwuZ21haWwuY29tIj4KICAgICAgPGRpdiBkaXI9 Imx0ciI+CiAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPgogICAgICAgICAgPGRpdiBj bGFzcz0iZ21haWxfc2lnbmF0dXJlIiBkYXRhLXNtYXJ0bWFpbD0iZ21haWxfc2lnbmF0dXJlIj7C oAogICAgICAgICAgICBIb3cgdG8gY2hlY2sgdGhpcyBvbiBhIGdsdXN0ZXIgaW5mcmFzdHJ1Y3R1 cmU/PC9kaXY+CiAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9zaWduYXR1cmUiIGRhdGEtc21h cnRtYWlsPSJnbWFpbF9zaWduYXR1cmUiPjxicj4KICAgICAgICAgIDwvZGl2PgogICAgICAgIDwv ZGl2PgogICAgICA8L2Rpdj4KICAgIDwvYmxvY2txdW90ZT4KICAgIEluIHRoZSBmdXNlIG1vdW50 IGxvZ3MgZm9yIHRoZSBlbmdpbmUgdm9sdW1lLCBjaGVjayBpZiB0aGVyZSBhcmUgYW55CiAgICBt ZXNzYWdlcyBmb3IgYnJpY2sgZGlzY29ubmVjdHMuIFNvbWV0aGluZyBhbG9uZyB0aGUgbGluZXMg b2YKICAgICJkaXNjb25uZWN0ZWQgZnJvbSB2b2xuYW1lLWNsaWVudC14Ii48YnI+CiAgICBKdXN0 IGd1ZXNzaW5nIGhlcmUsIGJ1dCBtYXliZSBldmVuIHRoZSAnZGF0YScgdm9sdW1lIGRpZCBleHBl cmllbmNlCiAgICBkaXNjb25uZWN0cyBhbmQgc2VsZi1oZWFscyBsYXRlciBidXQgeW91IGRpZCBu b3Qgb2JzZXJ2ZSBpdCB3aGVuIHlvdQogICAgcmFuIGhlYWwgaW5mby4gU2VlIHRoZSBnbHVzdGVy c2hkIGxvZyBvciBtb3VudCBsb2cgZm9yIGZvciBzZWxmLWhlYWwKICAgIGNvbXBsZXRpb24gbWVz c2FnZXMgb24gPGk+IDAtZGF0YS1yZXBsaWNhdGUtMCA8L2k+YWxzby48YnI+CiAgICA8YnI+CiAg ICBSZWdhcmRzLDxicj4KICAgIFJhdmk8YnI+CiAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIgpj aXRlPSJtaWQ6Q0FHSz0za3g9WHVmUTNfYmJtRjErUjNnK2dxQ2EtblRVZjA1S05iQTAwZUJjSE05 MitnQG1haWwuZ21haWwuY29tIj4KICAgICAgPGRpdiBkaXI9Imx0ciI+CiAgICAgICAgPGRpdiBj bGFzcz0iZ21haWxfZXh0cmEiPgogICAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxfc2lnbmF0dXJl IiBkYXRhLXNtYXJ0bWFpbD0iZ21haWxfc2lnbmF0dXJlIj5UaGFuawogICAgICAgICAgICB5b3U8 L2Rpdj4KICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX3NpZ25hdHVyZSIgZGF0YS1zbWFydG1h aWw9ImdtYWlsX3NpZ25hdHVyZSI+PGJyPgogICAgICAgICAgPC9kaXY+CiAgICAgICAgICA8ZGl2 IGNsYXNzPSJnbWFpbF9zaWduYXR1cmUiIGRhdGEtc21hcnRtYWlsPSJnbWFpbF9zaWduYXR1cmUi Pjxicj4KICAgICAgICAgIDwvZGl2PgogICAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxfc2lnbmF0 dXJlIiBkYXRhLXNtYXJ0bWFpbD0iZ21haWxfc2lnbmF0dXJlIj48YnI+CiAgICAgICAgICA8L2Rp dj4KICAgICAgICA8L2Rpdj4KICAgICAgPC9kaXY+CiAgICA8L2Jsb2NrcXVvdGU+CiAgICA8YnI+ CiAgPC9ib2R5Pgo8L2h0bWw+CgotLS0tLS0tLS0tLS0tLTg0RUI2REVCODRBQUZBQTU3RDA1MzhE Ni0tCg== --===============2853443058635076698==--