From paf1 at email.cz Thu Mar 31 09:12:51 2016 Content-Type: multipart/mixed; boundary="===============4394930642073249396==" MIME-Version: 1.0 From: paf1 at email.cz To: users at ovirt.org Subject: Re: [ovirt-users] ovirt with glusterfs - big test - unwanted results Date: Thu, 31 Mar 2016 15:11:59 +0200 Message-ID: <56FD221F.30707@email.cz> In-Reply-To: CAJgorsaOUQ_42GUSPh-H1vGUgJ114JYcUHR8vHwvmcWR+w8Jmw@mail.gmail.com --===============4394930642073249396== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------050907030908030304090009 Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 7bit Hi, rest of logs: www.uschovna.cz/en/zasilka/HYGXR57CNHM3TP39-L3W = The TEST is the last big event in logs .... TEST TIME : about 14:00-14:30 CET regs.Pavel On 31.3.2016 14:30, Yaniv Kaul wrote: > Hi Pavel, > > Thanks for the report. Can you begin with a more accurate description = > of your environment? > Begin with host, oVirt and Gluster versions. Then continue with the = > exact setup (what are 'A', 'B', 'C' - domains? Volumes? What is the = > mapping between domains and volumes?). > > Are there any logs you can share with us? > > I'm sure with more information, we'd be happy to look at the issue. > Y. > > > On Thu, Mar 31, 2016 at 3:09 PM, paf1(a)email.cz = > > wrote: > > Hello, > we tried the following test - with unwanted results > > input: > 5 node gluster > A =3D replica 3 with arbiter 1 ( node1+node2+arbiter on node 5 ) > B =3D replica 3 with arbiter 1 ( node3+node4+arbiter on node 5 ) > C =3D distributed replica 3 arbiter 1 ( node1+node2, node3+node4, > each arbiter on node 5) > node 5 has only arbiter replica ( 4x ) > > TEST: > 1) directly reboot one node - OK ( is not important which ( data > node or arbiter node )) > 2) directly reboot two nodes - OK ( if nodes are not from the > same replica ) > 3) directly reboot three nodes - yes, this is the main problem > and a questions .... > - rebooted all three nodes from replica "B" ( not so > possible, but who knows ... ) > - all VMs with data on this replica was paused ( no data > access ) - OK > - all VMs running on replica "B" nodes lost ( started > manually, later )( datas on other replicas ) - acceptable > BUT > - !!! all oVIrt domains went down !! - master domain is on > replica "A" which lost only one member from three !!! > so we are not expecting that all domain will go down, > especially master with 2 live members. > > Results: > - the whole cluster unreachable until at all domains up - > depent of all nodes up !!! > - all paused VMs started back - OK > - rest of all VMs rebooted and runnig - OK > > Questions: > 1) why all domains down if master domain ( on replica "A" ) > has two runnig members ( 2 of 3 ) ?? > 2) how to fix that colaps without waiting to all nodes up ? ( > in worste case if node has HW error eg. ) ?? > 3) which oVirt cluster policy can prevent that situation ?? > ( if any ) > > regs. > Pavel > > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > --------------050907030908030304090009 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit Hi,
rest of logs:
www.uschovna.cz/en/zasi= lka/HYGXR57CNHM3TP39-L3W

The TEST is the last big event in logs ....
TEST TIME : about 14:00-14:30=C2=A0 CET

regs.Pavel

On 31.3.2016 14:30, Yaniv Kaul wrote:
Hi Pavel,

Thanks for the report. Can you begin with a more accurate description of your environment?
Begin with host, oVirt and Gluster versions. Then continue with the exact setup (what are 'A', 'B', 'C' - domains? Volumes? What is the mapping between domains and volumes?).

Are there any logs you can share with us?

I'm sure with more information, we'd be happy to look at the issue.
Y.


On Thu, Mar 31, 2016 at 3:09 PM, paf1(a= )email.cz <paf1(a)emai= l.cz> wrote:
Hello,
we tried the=C2=A0 following test - with unwanted results

input:
5 node gluster
A =3D replica 3 with arbiter 1 ( node1+node2+arbiter on node 5 )
B =3D replica 3 with arbiter 1 ( node3+node4+arbiter on node 5 )
C =3D distributed replica 3 arbiter 1=C2=A0 ( node1+node2, node3+node4, each arbiter on node 5)
node 5 has only arbiter replica ( 4x )

TEST:
1)=C2=A0 directly reboot one node - OK ( is not important whi= ch ( data node or arbiter node ))
2)=C2=A0 directly reboot two nodes - OK ( if=C2=A0 nodes are = not from the same replica )
3)=C2=A0 directly reboot three nodes - yes, this is the main problem and a questions ....
=C2=A0=C2=A0=C2=A0 - rebooted all three nodes from replica "B= "=C2=A0 ( not so possible, but who knows ... )
=C2=A0=C2=A0=C2=A0 - all VMs with data on this replica was pa= used ( no data access ) - OK
=C2=A0=C2=A0=C2=A0 - all VMs running on replica "B" nodes los= t (=C2=A0 started manually, later )( datas on other replicas ) - acceptable
BUT
=C2=A0=C2=A0=C2=A0 - !!! all oVIrt domains went down !! - mas= ter domain is on replica "A" which lost only one member from three !!!
=C2=A0=C2=A0=C2=A0 so we are not expecting that all domain wi= ll go down, especially master with 2 live members.
=C2=A0=C2=A0=C2=A0
Results:
=C2=A0=C2=A0=C2=A0 - the whole cluster unreachable until at a= ll domains up - depent of all nodes up !!!
=C2=A0=C2=A0=C2=A0 - all paused VMs started back - OK
=C2=A0=C2=A0=C2=A0 - rest of all VMs rebooted and runnig - OK=

Questions:
=C2=A0=C2=A0=C2=A0 1) why all domains down if master domain (= on replica "A" ) has two runnig members ( 2 of 3 )=C2=A0 ??
=C2=A0=C2=A0=C2=A0 2) how to fix that colaps without waiting = to all nodes up ? ( in worste case if node has HW error eg. ) ??
=C2=A0=C2=A0=C2=A0 3) which oVirt=C2=A0 cluster=C2=A0 policy= =C2=A0 can prevent that situation ?? ( if any )

regs.
Pavel



_______________________________________________
Users mailing list
U= sers(a)ovirt.org
http://lists.ovirt.org/m= ailman/listinfo/users



--------------050907030908030304090009-- --===============4394930642073249396== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS0wNTA5MDcwMzA5MDgwMzAzMDQwOTAwMDkKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDdiaXQK CkhpLApyZXN0IG9mIGxvZ3M6Cnd3dy51c2Nob3ZuYS5jei9lbi96YXNpbGthL0hZR1hSNTdDTkhN M1RQMzktTDNXIAo8aHR0cDovL3d3dy51c2Nob3ZuYS5jei9lbi96YXNpbGthL0hZR1hSNTdDTkhN M1RQMzktTDNXPgoKVGhlIFRFU1QgaXMgdGhlIGxhc3QgYmlnIGV2ZW50IGluIGxvZ3MgLi4uLgpU RVNUIFRJTUUgOiBhYm91dCAxNDowMC0xNDozMCAgQ0VUCgpyZWdzLlBhdmVsCgpPbiAzMS4zLjIw MTYgMTQ6MzAsIFlhbml2IEthdWwgd3JvdGU6Cj4gSGkgUGF2ZWwsCj4KPiBUaGFua3MgZm9yIHRo ZSByZXBvcnQuIENhbiB5b3UgYmVnaW4gd2l0aCBhIG1vcmUgYWNjdXJhdGUgZGVzY3JpcHRpb24g Cj4gb2YgeW91ciBlbnZpcm9ubWVudD8KPiBCZWdpbiB3aXRoIGhvc3QsIG9WaXJ0IGFuZCBHbHVz dGVyIHZlcnNpb25zLiBUaGVuIGNvbnRpbnVlIHdpdGggdGhlIAo+IGV4YWN0IHNldHVwICh3aGF0 IGFyZSAnQScsICdCJywgJ0MnIC0gZG9tYWlucz8gVm9sdW1lcz8gV2hhdCBpcyB0aGUgCj4gbWFw cGluZyBiZXR3ZWVuIGRvbWFpbnMgYW5kIHZvbHVtZXM/KS4KPgo+IEFyZSB0aGVyZSBhbnkgbG9n cyB5b3UgY2FuIHNoYXJlIHdpdGggdXM/Cj4KPiBJJ20gc3VyZSB3aXRoIG1vcmUgaW5mb3JtYXRp b24sIHdlJ2QgYmUgaGFwcHkgdG8gbG9vayBhdCB0aGUgaXNzdWUuCj4gWS4KPgo+Cj4gT24gVGh1 LCBNYXIgMzEsIDIwMTYgYXQgMzowOSBQTSwgcGFmMUBlbWFpbC5jeiA8bWFpbHRvOnBhZjFAZW1h aWwuY3o+IAo+IDxwYWYxQGVtYWlsLmN6IDxtYWlsdG86cGFmMUBlbWFpbC5jej4+IHdyb3RlOgo+ Cj4gICAgIEhlbGxvLAo+ICAgICB3ZSB0cmllZCB0aGUgIGZvbGxvd2luZyB0ZXN0IC0gd2l0aCB1 bndhbnRlZCByZXN1bHRzCj4KPiAgICAgaW5wdXQ6Cj4gICAgIDUgbm9kZSBnbHVzdGVyCj4gICAg IEEgPSByZXBsaWNhIDMgd2l0aCBhcmJpdGVyIDEgKCBub2RlMStub2RlMithcmJpdGVyIG9uIG5v ZGUgNSApCj4gICAgIEIgPSByZXBsaWNhIDMgd2l0aCBhcmJpdGVyIDEgKCBub2RlMytub2RlNCth cmJpdGVyIG9uIG5vZGUgNSApCj4gICAgIEMgPSBkaXN0cmlidXRlZCByZXBsaWNhIDMgYXJiaXRl ciAxICAoIG5vZGUxK25vZGUyLCBub2RlMytub2RlNCwKPiAgICAgZWFjaCBhcmJpdGVyIG9uIG5v ZGUgNSkKPiAgICAgbm9kZSA1IGhhcyBvbmx5IGFyYml0ZXIgcmVwbGljYSAoIDR4ICkKPgo+ICAg ICBURVNUOgo+ICAgICAxKSAgZGlyZWN0bHkgcmVib290IG9uZSBub2RlIC0gT0sgKCBpcyBub3Qg aW1wb3J0YW50IHdoaWNoICggZGF0YQo+ICAgICBub2RlIG9yIGFyYml0ZXIgbm9kZSApKQo+ICAg ICAyKSAgZGlyZWN0bHkgcmVib290IHR3byBub2RlcyAtIE9LICggaWYgIG5vZGVzIGFyZSBub3Qg ZnJvbSB0aGUKPiAgICAgc2FtZSByZXBsaWNhICkKPiAgICAgMykgIGRpcmVjdGx5IHJlYm9vdCB0 aHJlZSBub2RlcyAtIHllcywgdGhpcyBpcyB0aGUgbWFpbiBwcm9ibGVtCj4gICAgIGFuZCBhIHF1 ZXN0aW9ucyAuLi4uCj4gICAgICAgICAtIHJlYm9vdGVkIGFsbCB0aHJlZSBub2RlcyBmcm9tIHJl cGxpY2EgIkIiICAoIG5vdCBzbwo+ICAgICBwb3NzaWJsZSwgYnV0IHdobyBrbm93cyAuLi4gKQo+ ICAgICAgICAgLSBhbGwgVk1zIHdpdGggZGF0YSBvbiB0aGlzIHJlcGxpY2Egd2FzIHBhdXNlZCAo IG5vIGRhdGEKPiAgICAgYWNjZXNzICkgLSBPSwo+ICAgICAgICAgLSBhbGwgVk1zIHJ1bm5pbmcg b24gcmVwbGljYSAiQiIgbm9kZXMgbG9zdCAoICBzdGFydGVkCj4gICAgIG1hbnVhbGx5LCBsYXRl ciApKCBkYXRhcyBvbiBvdGhlciByZXBsaWNhcyApIC0gYWNjZXB0YWJsZQo+ICAgICBCVVQKPiAg ICAgICAgIC0gISEhIGFsbCBvVklydCBkb21haW5zIHdlbnQgZG93biAhISAtIG1hc3RlciBkb21h aW4gaXMgb24KPiAgICAgcmVwbGljYSAiQSIgd2hpY2ggbG9zdCBvbmx5IG9uZSBtZW1iZXIgZnJv bSB0aHJlZSAhISEKPiAgICAgICAgIHNvIHdlIGFyZSBub3QgZXhwZWN0aW5nIHRoYXQgYWxsIGRv bWFpbiB3aWxsIGdvIGRvd24sCj4gICAgIGVzcGVjaWFsbHkgbWFzdGVyIHdpdGggMiBsaXZlIG1l bWJlcnMuCj4KPiAgICAgUmVzdWx0czoKPiAgICAgICAgIC0gdGhlIHdob2xlIGNsdXN0ZXIgdW5y ZWFjaGFibGUgdW50aWwgYXQgYWxsIGRvbWFpbnMgdXAgLQo+ICAgICBkZXBlbnQgb2YgYWxsIG5v ZGVzIHVwICEhIQo+ICAgICAgICAgLSBhbGwgcGF1c2VkIFZNcyBzdGFydGVkIGJhY2sgLSBPSwo+ ICAgICAgICAgLSByZXN0IG9mIGFsbCBWTXMgcmVib290ZWQgYW5kIHJ1bm5pZyAtIE9LCj4KPiAg ICAgUXVlc3Rpb25zOgo+ICAgICAgICAgMSkgd2h5IGFsbCBkb21haW5zIGRvd24gaWYgbWFzdGVy IGRvbWFpbiAoIG9uIHJlcGxpY2EgIkEiICkKPiAgICAgaGFzIHR3byBydW5uaWcgbWVtYmVycyAo IDIgb2YgMyApICA/Pwo+ICAgICAgICAgMikgaG93IHRvIGZpeCB0aGF0IGNvbGFwcyB3aXRob3V0 IHdhaXRpbmcgdG8gYWxsIG5vZGVzIHVwID8gKAo+ICAgICBpbiB3b3JzdGUgY2FzZSBpZiBub2Rl IGhhcyBIVyBlcnJvciBlZy4gKSA/Pwo+ICAgICAgICAgMykgd2hpY2ggb1ZpcnQgIGNsdXN0ZXIg IHBvbGljeSAgY2FuIHByZXZlbnQgdGhhdCBzaXR1YXRpb24gPz8KPiAgICAgKCBpZiBhbnkgKQo+ Cj4gICAgIHJlZ3MuCj4gICAgIFBhdmVsCj4KPgo+Cj4gICAgIF9fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fCj4gICAgIFVzZXJzIG1haWxpbmcgbGlzdAo+ICAg ICBVc2Vyc0BvdmlydC5vcmcgPG1haWx0bzpVc2Vyc0BvdmlydC5vcmc+Cj4gICAgIGh0dHA6Ly9s aXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vycwo+Cj4KCgotLS0tLS0tLS0tLS0t LTA1MDkwNzAzMDkwODAzMDMwNDA5MDAwOQpDb250ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNl dD11dGYtOApDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiA4Yml0Cgo8aHRtbD4KICA8aGVhZD4K ICAgIDxtZXRhIGNvbnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11dGYtOCIgaHR0cC1lcXVpdj0i Q29udGVudC1UeXBlIj4KICA8L2hlYWQ+CiAgPGJvZHkgdGV4dD0iIzAwMDA2NiIgYmdjb2xvcj0i I0ZGRkZGRiI+CiAgICBIaSwgPGJyPgogICAgcmVzdCBvZiBsb2dzOjxicj4KICAgIDxhIGhyZWY9 Imh0dHA6Ly93d3cudXNjaG92bmEuY3ovZW4vemFzaWxrYS9IWUdYUjU3Q05ITTNUUDM5LUwzVyIK ICAgICAgc3R5bGU9InRleHQtZGVjb3JhdGlvbjpub25lO2NvbG9yOiNmZjljMDA7Ij53d3cudXNj aG92bmEuY3ovZW4vemFzaWxrYS9IWUdYUjU3Q05ITTNUUDM5LUwzVzwvYT48YnI+CiAgICA8YnI+ CiAgICBUaGUgVEVTVCBpcyB0aGUgbGFzdCBiaWcgZXZlbnQgaW4gbG9ncyAuLi4uPGJyPgogICAg VEVTVCBUSU1FIDogYWJvdXQgMTQ6MDAtMTQ6MzDCoCBDRVQ8YnI+CiAgICA8YnI+CiAgICByZWdz LlBhdmVsPGJyPgogICAgPGJyPgogICAgPGRpdiBjbGFzcz0ibW96LWNpdGUtcHJlZml4Ij5PbiAz MS4zLjIwMTYgMTQ6MzAsIFlhbml2IEthdWwgd3JvdGU6PGJyPgogICAgPC9kaXY+CiAgICA8Ymxv Y2txdW90ZQpjaXRlPSJtaWQ6Q0FKZ29yc2FPVVFfNDJHVVNQaC1IMXZHVWdKMTE0SlljVUhSOHZI d3ZtY1dSK3c4Sm13QG1haWwuZ21haWwuY29tIgogICAgICB0eXBlPSJjaXRlIj4KICAgICAgPGRp diBkaXI9Imx0ciI+SGkgUGF2ZWwsCiAgICAgICAgPGRpdj48YnI+CiAgICAgICAgPC9kaXY+CiAg ICAgICAgPGRpdj5UaGFua3MgZm9yIHRoZSByZXBvcnQuIENhbiB5b3UgYmVnaW4gd2l0aCBhIG1v cmUgYWNjdXJhdGUKICAgICAgICAgIGRlc2NyaXB0aW9uIG9mIHlvdXIgZW52aXJvbm1lbnQ/PC9k aXY+CiAgICAgICAgPGRpdj5CZWdpbiB3aXRoIGhvc3QsIG9WaXJ0IGFuZCBHbHVzdGVyIHZlcnNp b25zLiBUaGVuIGNvbnRpbnVlCiAgICAgICAgICB3aXRoIHRoZSBleGFjdCBzZXR1cCAod2hhdCBh cmUgJ0EnLCAnQicsICdDJyAtIGRvbWFpbnM/CiAgICAgICAgICBWb2x1bWVzPyBXaGF0IGlzIHRo ZSBtYXBwaW5nIGJldHdlZW4gZG9tYWlucyBhbmQgdm9sdW1lcz8pLjwvZGl2PgogICAgICAgIDxk aXY+PGJyPgogICAgICAgIDwvZGl2PgogICAgICAgIDxkaXY+QXJlIHRoZXJlIGFueSBsb2dzIHlv dSBjYW4gc2hhcmUgd2l0aCB1cz88L2Rpdj4KICAgICAgICA8ZGl2Pjxicj4KICAgICAgICA8L2Rp dj4KICAgICAgICA8ZGl2PkknbSBzdXJlIHdpdGggbW9yZSBpbmZvcm1hdGlvbiwgd2UnZCBiZSBo YXBweSB0byBsb29rIGF0CiAgICAgICAgICB0aGUgaXNzdWUuPC9kaXY+CiAgICAgICAgPGRpdj5Z LjwvZGl2PgogICAgICAgIDxkaXY+PGJyPgogICAgICAgIDwvZGl2PgogICAgICA8L2Rpdj4KICAg ICAgPGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPjxicj4KICAgICAgICA8ZGl2IGNsYXNzPSJnbWFp bF9xdW90ZSI+T24gVGh1LCBNYXIgMzEsIDIwMTYgYXQgMzowOSBQTSwgPGEKICAgICAgICAgICAg bW96LWRvLW5vdC1zZW5kPSJ0cnVlIiBocmVmPSJtYWlsdG86cGFmMUBlbWFpbC5jeiI+cGFmMUBl bWFpbC5jejwvYT4KICAgICAgICAgIDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgbW96LWRvLW5vdC1z ZW5kPSJ0cnVlIgogICAgICAgICAgICAgIGhyZWY9Im1haWx0bzpwYWYxQGVtYWlsLmN6IiB0YXJn ZXQ9Il9ibGFuayI+cGFmMUBlbWFpbC5jejwvYT4mZ3Q7PC9zcGFuPgogICAgICAgICAgd3JvdGU6 PGJyPgogICAgICAgICAgPGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFy Z2luOjAgMCAwCiAgICAgICAgICAgIC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFk ZGluZy1sZWZ0OjFleCI+CiAgICAgICAgICAgIDxkaXYgdGV4dD0iIzAwMDA2NiIgYmdjb2xvcj0i I0ZGRkZGRiI+IEhlbGxvLCA8YnI+CiAgICAgICAgICAgICAgd2UgdHJpZWQgdGhlwqAgZm9sbG93 aW5nIHRlc3QgLSB3aXRoIHVud2FudGVkIHJlc3VsdHM8YnI+CiAgICAgICAgICAgICAgPGJyPgog ICAgICAgICAgICAgIGlucHV0Ojxicj4KICAgICAgICAgICAgICA1IG5vZGUgZ2x1c3Rlcjxicj4K ICAgICAgICAgICAgICBBID0gcmVwbGljYSAzIHdpdGggYXJiaXRlciAxICggbm9kZTErbm9kZTIr YXJiaXRlciBvbiBub2RlCiAgICAgICAgICAgICAgNSApPGJyPgogICAgICAgICAgICAgIEIgPSBy ZXBsaWNhIDMgd2l0aCBhcmJpdGVyIDEgKCBub2RlMytub2RlNCthcmJpdGVyIG9uIG5vZGUKICAg ICAgICAgICAgICA1ICk8YnI+CiAgICAgICAgICAgICAgQyA9IGRpc3RyaWJ1dGVkIHJlcGxpY2Eg MyBhcmJpdGVyIDHCoCAoIG5vZGUxK25vZGUyLAogICAgICAgICAgICAgIG5vZGUzK25vZGU0LCBl YWNoIGFyYml0ZXIgb24gbm9kZSA1KTxicj4KICAgICAgICAgICAgICBub2RlIDUgaGFzIG9ubHkg YXJiaXRlciByZXBsaWNhICggNHggKTxicj4KICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAg ICAgVEVTVDo8YnI+CiAgICAgICAgICAgICAgMSnCoCBkaXJlY3RseSByZWJvb3Qgb25lIG5vZGUg LSBPSyAoIGlzIG5vdCBpbXBvcnRhbnQgd2hpY2gKICAgICAgICAgICAgICAoIGRhdGEgbm9kZSBv ciBhcmJpdGVyIG5vZGUgKSk8YnI+CiAgICAgICAgICAgICAgMinCoCBkaXJlY3RseSByZWJvb3Qg dHdvIG5vZGVzIC0gT0sgKCBpZsKgIG5vZGVzIGFyZSBub3QKICAgICAgICAgICAgICBmcm9tIHRo ZSBzYW1lIHJlcGxpY2EgKSA8YnI+CiAgICAgICAgICAgICAgMynCoCBkaXJlY3RseSByZWJvb3Qg dGhyZWUgbm9kZXMgLSB5ZXMsIHRoaXMgaXMgdGhlIG1haW4KICAgICAgICAgICAgICBwcm9ibGVt IGFuZCBhIHF1ZXN0aW9ucyAuLi4uPGJyPgogICAgICAgICAgICAgIMKgwqDCoCAtIHJlYm9vdGVk IGFsbCB0aHJlZSBub2RlcyBmcm9tIHJlcGxpY2EgIkIiwqAgKCBub3Qgc28KICAgICAgICAgICAg ICBwb3NzaWJsZSwgYnV0IHdobyBrbm93cyAuLi4gKTxicj4KICAgICAgICAgICAgICDCoMKgwqAg LSBhbGwgVk1zIHdpdGggZGF0YSBvbiB0aGlzIHJlcGxpY2Egd2FzIHBhdXNlZCAoIG5vCiAgICAg ICAgICAgICAgZGF0YSBhY2Nlc3MgKSAtIE9LPGJyPgogICAgICAgICAgICAgIMKgwqDCoCAtIGFs bCBWTXMgcnVubmluZyBvbiByZXBsaWNhICJCIiBub2RlcyBsb3N0ICjCoCBzdGFydGVkCiAgICAg ICAgICAgICAgbWFudWFsbHksIGxhdGVyICkoIGRhdGFzIG9uIG90aGVyIHJlcGxpY2FzICkgLSBh Y2NlcHRhYmxlPGJyPgogICAgICAgICAgICAgIEJVVDxicj4KICAgICAgICAgICAgICDCoMKgwqAg LSAhISEgYWxsIG9WSXJ0IGRvbWFpbnMgd2VudCBkb3duICEhIC0gbWFzdGVyIGRvbWFpbgogICAg ICAgICAgICAgIGlzIG9uIHJlcGxpY2EgIkEiIHdoaWNoIGxvc3Qgb25seSBvbmUgbWVtYmVyIGZy b20gdGhyZWUKICAgICAgICAgICAgICAhISE8YnI+CiAgICAgICAgICAgICAgwqDCoMKgIHNvIHdl IGFyZSBub3QgZXhwZWN0aW5nIHRoYXQgYWxsIGRvbWFpbiB3aWxsIGdvIGRvd24sCiAgICAgICAg ICAgICAgZXNwZWNpYWxseSBtYXN0ZXIgd2l0aCAyIGxpdmUgbWVtYmVycy48YnI+CiAgICAgICAg ICAgICAgwqDCoMKgIDxicj4KICAgICAgICAgICAgICBSZXN1bHRzOiA8YnI+CiAgICAgICAgICAg ICAgwqDCoMKgIC0gdGhlIHdob2xlIGNsdXN0ZXIgdW5yZWFjaGFibGUgdW50aWwgYXQgYWxsIGRv bWFpbnMKICAgICAgICAgICAgICB1cCAtIGRlcGVudCBvZiBhbGwgbm9kZXMgdXAgISEhPGJyPgog ICAgICAgICAgICAgIMKgwqDCoCAtIGFsbCBwYXVzZWQgVk1zIHN0YXJ0ZWQgYmFjayAtIE9LPGJy PgogICAgICAgICAgICAgIMKgwqDCoCAtIHJlc3Qgb2YgYWxsIFZNcyByZWJvb3RlZCBhbmQgcnVu bmlnIC0gT0s8YnI+CiAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgIFF1ZXN0aW9uczo8 YnI+CiAgICAgICAgICAgICAgwqDCoMKgIDEpIHdoeSBhbGwgZG9tYWlucyBkb3duIGlmIG1hc3Rl ciBkb21haW4gKCBvbiByZXBsaWNhCiAgICAgICAgICAgICAgIkEiICkgaGFzIHR3byBydW5uaWcg bWVtYmVycyAoIDIgb2YgMyApwqAgPz88YnI+CiAgICAgICAgICAgICAgwqDCoMKgIDIpIGhvdyB0 byBmaXggdGhhdCBjb2xhcHMgd2l0aG91dCB3YWl0aW5nIHRvIGFsbCBub2RlcwogICAgICAgICAg ICAgIHVwID8gKCBpbiB3b3JzdGUgY2FzZSBpZiBub2RlIGhhcyBIVyBlcnJvciBlZy4gKSA/Pzxi cj4KICAgICAgICAgICAgICDCoMKgwqAgMykgd2hpY2ggb1ZpcnTCoCBjbHVzdGVywqAgcG9saWN5 wqAgY2FuIHByZXZlbnQgdGhhdAogICAgICAgICAgICAgIHNpdHVhdGlvbiA/PyAoIGlmIGFueSAp PGJyPgogICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICByZWdzLjxicj4KICAgICAgICAg ICAgICBQYXZlbDxicj4KICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgPGJyPgogICAg ICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgPGJyPgogICAgICAgICAgICBfX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxicj4KICAgICAgICAgICAgVXNlcnMg bWFpbGluZyBsaXN0PGJyPgogICAgICAgICAgICA8YSBtb3otZG8tbm90LXNlbmQ9InRydWUiIGhy ZWY9Im1haWx0bzpVc2Vyc0BvdmlydC5vcmciPlVzZXJzQG92aXJ0Lm9yZzwvYT48YnI+CiAgICAg ICAgICAgIDxhIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSIKICAgICAgICAgICAgICBocmVmPSJodHRw Oi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiCiAgICAgICAgICAgICAg cmVsPSJub3JlZmVycmVyIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9t YWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPjxicj4KICAgICAgICAgICAgPGJyPgogICAgICAgICAg PC9ibG9ja3F1b3RlPgogICAgICAgIDwvZGl2PgogICAgICAgIDxicj4KICAgICAgPC9kaXY+CiAg ICA8L2Jsb2NrcXVvdGU+CiAgICA8YnI+CiAgPC9ib2R5Pgo8L2h0bWw+CgotLS0tLS0tLS0tLS0t LTA1MDkwNzAzMDkwODAzMDMwNDA5MDAwOS0tCg== --===============4394930642073249396==--