From paf1 at email.cz Thu May 28 16:11:31 2015 Content-Type: multipart/mixed; boundary="===============8310744172657421607==" MIME-Version: 1.0 From: paf1 at email.cz To: users at ovirt.org Subject: [ovirt-users] gluster config in 4 node cluster Date: Thu, 28 May 2015 22:11:25 +0200 Message-ID: <5567766D.1060104@email.cz> --===============8310744172657421607== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------060707050503070803070103 Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 7bit Hello, How to optimal configure 4 node cluster for any one node goes to = maintenance without stopping VM ?? a) replica 4 - but it takes a lot of space b) disperse 3+1 ( raid 5 ) - but bad performance and not visible by = oVirt 3.7.2 c) stripe2+replica2 =3D but VM paused any other idea ? regs. Pa. --------------060707050503070803070103 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit Hello,
How to optimal configure=C2=A0 4 node cluster for any one node goes to= =C2=A0 maintenance without stopping VM ??

a) replica 4=C2=A0 - but it takes a lot of space
b) disperse 3+1 ( raid 5 ) - but bad performance and not visible by oVirt 3.7.2
c) stripe2+replica2 =3D but VM paused

any other idea ?
regs.
Pa.
--------------060707050503070803070103-- --===============8310744172657421607== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS0wNjA3MDcwNTA1MDMwNzA4MDMwNzAxMDMKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDdiaXQK CkhlbGxvLApIb3cgdG8gb3B0aW1hbCBjb25maWd1cmUgIDQgbm9kZSBjbHVzdGVyIGZvciBhbnkg b25lIG5vZGUgZ29lcyB0byAKbWFpbnRlbmFuY2Ugd2l0aG91dCBzdG9wcGluZyBWTSA/PwoKYSkg cmVwbGljYSA0ICAtIGJ1dCBpdCB0YWtlcyBhIGxvdCBvZiBzcGFjZQpiKSBkaXNwZXJzZSAzKzEg KCByYWlkIDUgKSAtIGJ1dCBiYWQgcGVyZm9ybWFuY2UgYW5kIG5vdCB2aXNpYmxlIGJ5IApvVmly dCAzLjcuMgpjKSBzdHJpcGUyK3JlcGxpY2EyID0gYnV0IFZNIHBhdXNlZAoKYW55IG90aGVyIGlk ZWEgPwpyZWdzLgpQYS4KCi0tLS0tLS0tLS0tLS0tMDYwNzA3MDUwNTAzMDcwODAzMDcwMTAzCkNv bnRlbnQtVHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNmZXItRW5j b2Rpbmc6IDhiaXQKCjxodG1sPgogIDxoZWFkPgoKICAgIDxtZXRhIGh0dHAtZXF1aXY9ImNvbnRl bnQtdHlwZSIgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4KICA8L2hlYWQ+CiAg PGJvZHkgdGV4dD0iIzAwMDA2NiIgYmdjb2xvcj0iI0ZGRkZGRiI+CiAgICBIZWxsbyw8YnI+CiAg ICBIb3cgdG8gb3B0aW1hbCBjb25maWd1cmXCoCA0IG5vZGUgY2x1c3RlciBmb3IgYW55IG9uZSBu b2RlIGdvZXMgdG/CoAogICAgbWFpbnRlbmFuY2Ugd2l0aG91dCBzdG9wcGluZyBWTSA/Pzxicj4K ICAgIDxicj4KICAgIGEpIHJlcGxpY2EgNMKgIC0gYnV0IGl0IHRha2VzIGEgbG90IG9mIHNwYWNl PGJyPgogICAgYikgZGlzcGVyc2UgMysxICggcmFpZCA1ICkgLSBidXQgYmFkIHBlcmZvcm1hbmNl IGFuZCBub3QgdmlzaWJsZSBieQogICAgb1ZpcnQgMy43LjI8YnI+CiAgICBjKSBzdHJpcGUyK3Jl cGxpY2EyID0gYnV0IFZNIHBhdXNlZCA8YnI+CiAgICA8YnI+CiAgICBhbnkgb3RoZXIgaWRlYSA/ PGJyPgogICAgcmVncy48YnI+CiAgICBQYS48YnI+CiAgPC9ib2R5Pgo8L2h0bWw+CgotLS0tLS0t LS0tLS0tLTA2MDcwNzA1MDUwMzA3MDgwMzA3MDEwMy0tCg== --===============8310744172657421607==-- From y.poltoratskiy at gmail.com Sun May 31 12:32:43 2015 Content-Type: multipart/mixed; boundary="===============3117237807987459780==" MIME-Version: 1.0 From: =?utf-8?q?=D0=AE=D1=80=D0=B8=D0=B9_=D0=9F=D0=BE=D0=BB=D1=82=D0=BE=D1=80?= =?utf-8?q?=D0=B0=D1=86=D0=BA=D0=B8=D0=B9_=3Cy=2Epoltoratskiy_at_gmail=2Ec?= =?utf-8?q?om=3E?= To: users at ovirt.org Subject: Re: [ovirt-users] gluster config in 4 node cluster Date: Sun, 31 May 2015 19:32:39 +0300 Message-ID: <556B37A7.1000508@gmail.com> In-Reply-To: 5567766D.1060104@email.cz --===============3117237807987459780== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------050100050502090900030707 Content-Type: text/plain; charset=3DUTF-8; format=3Dflowed Content-Transfer-Encoding: 8bit Hi, As for me, I would build one cluster with gluster service only based on = two nodes (replica 2), and the other one with virt service only based on = other two nodes. I think this variant is more scalable in future. PS. I am a new in oVirt, so do not except that I am wrong. 28.05.2015 23:11, paf1(a)email.cz =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > Hello, > How to optimal configure 4 node cluster for any one node goes to = > maintenance without stopping VM ?? > > a) replica 4 - but it takes a lot of space > b) disperse 3+1 ( raid 5 ) - but bad performance and not visible by = > oVirt 3.7.2 > c) stripe2+replica2 =3D but VM paused > > any other idea ? > regs. > Pa. > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users --------------050100050502090900030707 Content-Type: text/html; charset=3DUTF-8 Content-Transfer-Encoding: 8bit
Hi,

As for me, I would build one cluster with gluster service only based on two nodes (replica 2), and the other one with virt service only based on other two nodes. I think this variant is more scalable in future.

PS. I am a new in oVirt, so do not except that I am wrong.


28.05.2015 23:11, paf1(a)email.cz =D0=BF= =D0=B8=D1=88=D0=B5=D1=82:
Hello,
How to optimal configure=C2=A0 4 node cluster for any one node goes t= o=C2=A0 maintenance without stopping VM ??

a) replica 4=C2=A0 - but it takes a lot of space
b) disperse 3+1 ( raid 5 ) - but bad performance and not visible by oVirt 3.7.2
c) stripe2+replica2 =3D but VM paused

any other idea ?
regs.
Pa.


_______________________________________________
Users mailing list
Use=
rs(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

--------------050100050502090900030707-- --===============3117237807987459780== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS0wNTAxMDAwNTA1MDIwOTA5MDAwMzA3MDcKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PVVURi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDhiaXQK CgpIaSwKCkFzIGZvciBtZSwgSSB3b3VsZCBidWlsZCBvbmUgY2x1c3RlciB3aXRoIGdsdXN0ZXIg c2VydmljZSBvbmx5IGJhc2VkIG9uIAp0d28gbm9kZXMgKHJlcGxpY2EgMiksIGFuZCB0aGUgb3Ro ZXIgb25lIHdpdGggdmlydCBzZXJ2aWNlIG9ubHkgYmFzZWQgb24gCm90aGVyIHR3byBub2Rlcy4g SSB0aGluayB0aGlzIHZhcmlhbnQgaXMgbW9yZSBzY2FsYWJsZSBpbiBmdXR1cmUuCgpQUy4gSSBh bSBhIG5ldyBpbiBvVmlydCwgc28gZG8gbm90IGV4Y2VwdCB0aGF0IEkgYW0gd3JvbmcuCgoKMjgu MDUuMjAxNSAyMzoxMSwgcGFmMUBlbWFpbC5jeiDQv9C40YjQtdGCOgo+IEhlbGxvLAo+IEhvdyB0 byBvcHRpbWFsIGNvbmZpZ3VyZSAgNCBub2RlIGNsdXN0ZXIgZm9yIGFueSBvbmUgbm9kZSBnb2Vz IHRvIAo+IG1haW50ZW5hbmNlIHdpdGhvdXQgc3RvcHBpbmcgVk0gPz8KPgo+IGEpIHJlcGxpY2Eg NCAgLSBidXQgaXQgdGFrZXMgYSBsb3Qgb2Ygc3BhY2UKPiBiKSBkaXNwZXJzZSAzKzEgKCByYWlk IDUgKSAtIGJ1dCBiYWQgcGVyZm9ybWFuY2UgYW5kIG5vdCB2aXNpYmxlIGJ5IAo+IG9WaXJ0IDMu Ny4yCj4gYykgc3RyaXBlMityZXBsaWNhMiA9IGJ1dCBWTSBwYXVzZWQKPgo+IGFueSBvdGhlciBp ZGVhID8KPiByZWdzLgo+IFBhLgo+Cj4KPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fXwo+IFVzZXJzIG1haWxpbmcgbGlzdAo+IFVzZXJzQG92aXJ0Lm9yZwo+ IGh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycwoKCi0tLS0tLS0t LS0tLS0tMDUwMTAwMDUwNTAyMDkwOTAwMDMwNzA3CkNvbnRlbnQtVHlwZTogdGV4dC9odG1sOyBj aGFyc2V0PVVURi04CkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDhiaXQKCjxodG1sPgogIDxo ZWFkPgogICAgPG1ldGEgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PVVURi04IiBodHRwLWVx dWl2PSJDb250ZW50LVR5cGUiPgogIDwvaGVhZD4KICA8Ym9keSB0ZXh0PSIjMDAwMDAwIiBiZ2Nv bG9yPSIjRkZGRkZGIj4KICAgIDxicj4KICAgIEhpLDxicj4KICAgIDxicj4KICAgIEFzIGZvciBt ZSwgSSB3b3VsZCBidWlsZCBvbmUgY2x1c3RlciB3aXRoIGdsdXN0ZXIgc2VydmljZSBvbmx5IGJh c2VkCiAgICBvbiB0d28gbm9kZXMgKHJlcGxpY2EgMiksIGFuZCB0aGUgb3RoZXIgb25lIHdpdGgg dmlydCBzZXJ2aWNlIG9ubHkKICAgIGJhc2VkIG9uIG90aGVyIHR3byBub2Rlcy4gSSB0aGluayB0 aGlzIHZhcmlhbnQgaXMgbW9yZSBzY2FsYWJsZSBpbgogICAgZnV0dXJlLiA8YnI+CiAgICA8YnI+ CiAgICBQUy4gSSBhbSBhIG5ldyBpbiBvVmlydCwgc28gZG8gbm90IGV4Y2VwdCB0aGF0IEkgYW0g d3JvbmcuPGJyPgogICAgPGJyPgogICAgPGJyPgogICAgPGRpdiBjbGFzcz0ibW96LWNpdGUtcHJl Zml4Ij4yOC4wNS4yMDE1IDIzOjExLCA8YSBjbGFzcz0ibW96LXR4dC1saW5rLWFiYnJldmlhdGVk IiBocmVmPSJtYWlsdG86cGFmMUBlbWFpbC5jeiI+cGFmMUBlbWFpbC5jejwvYT4g0L/QuNGI0LXR gjo8YnI+CiAgICA8L2Rpdj4KICAgIDxibG9ja3F1b3RlIGNpdGU9Im1pZDo1NTY3NzY2RC4xMDYw MTA0QGVtYWlsLmN6IiB0eXBlPSJjaXRlIj4KICAgICAgPG1ldGEgaHR0cC1lcXVpdj0iY29udGVu dC10eXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9VVRGLTgiPgogICAgICBIZWxsbyw8 YnI+CiAgICAgIEhvdyB0byBvcHRpbWFsIGNvbmZpZ3VyZcKgIDQgbm9kZSBjbHVzdGVyIGZvciBh bnkgb25lIG5vZGUgZ29lcyB0b8KgCiAgICAgIG1haW50ZW5hbmNlIHdpdGhvdXQgc3RvcHBpbmcg Vk0gPz88YnI+CiAgICAgIDxicj4KICAgICAgYSkgcmVwbGljYSA0wqAgLSBidXQgaXQgdGFrZXMg YSBsb3Qgb2Ygc3BhY2U8YnI+CiAgICAgIGIpIGRpc3BlcnNlIDMrMSAoIHJhaWQgNSApIC0gYnV0 IGJhZCBwZXJmb3JtYW5jZSBhbmQgbm90IHZpc2libGUKICAgICAgYnkgb1ZpcnQgMy43LjI8YnI+ CiAgICAgIGMpIHN0cmlwZTIrcmVwbGljYTIgPSBidXQgVk0gcGF1c2VkIDxicj4KICAgICAgPGJy PgogICAgICBhbnkgb3RoZXIgaWRlYSA/PGJyPgogICAgICByZWdzLjxicj4KICAgICAgUGEuPGJy PgogICAgICA8YnI+CiAgICAgIDxmaWVsZHNldCBjbGFzcz0ibWltZUF0dGFjaG1lbnRIZWFkZXIi PjwvZmllbGRzZXQ+CiAgICAgIDxicj4KICAgICAgPHByZSB3cmFwPSIiPl9fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fClVzZXJzIG1haWxpbmcgbGlzdAo8YSBj bGFzcz0ibW96LXR4dC1saW5rLWFiYnJldmlhdGVkIiBocmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQu b3JnIj5Vc2Vyc0BvdmlydC5vcmc8L2E+CjxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQi IGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyI+aHR0 cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPgo8L3ByZT4KICAg IDwvYmxvY2txdW90ZT4KICAgIDxicj4KICA8L2JvZHk+CjwvaHRtbD4KCi0tLS0tLS0tLS0tLS0t MDUwMTAwMDUwNTAyMDkwOTAwMDMwNzA3LS0K --===============3117237807987459780==-- From soeren.malchow at mcon.net Sun May 31 19:32:57 2015 Content-Type: multipart/mixed; boundary="===============6287059662962659540==" MIME-Version: 1.0 From: Soeren Malchow To: users at ovirt.org Subject: Re: [ovirt-users] gluster config in 4 node cluster Date: Sun, 31 May 2015 23:32:53 +0000 Message-ID: In-Reply-To: 556B37A7.1000508@gmail.com --===============6287059662962659540== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --_000_D19165E2D96Esoerenmalchowmconnet_ Content-Type: text/plain; charset=3D"windows-1251" Content-Transfer-Encoding: quoted-printable Hi For a production environment i would not build a 2 node gluster, i would bu= =3D ild at least 3 nodes to have it much easier with the quorum. Taking into account that you can use commodity hardware i would also sugges= =3D t to split the services, but i would go for at least 3 gluster nodes =3D96 = wh=3D ich add up to at least 5 nodes for a HA system. If you want 4 in any case, then my suggestion would be to go for 4 replicas= =3D , each node has anything that it needs to run and you can basically use NFS= =3D to localhost for the storage which would make the nodes always access the = =3D local storage for the VMS, availability wise it makes no difference. Cheers Soeren From: =3DDE=3DF0=3DE8=3DE9 =3DCF=3DEE=3DEB=3DF2=3DEE=3DF0=3DE0=3DF6=3DEA=3D= E8=3DE9 > Date: Sunday 31 May 2015 18:32 To: "users(a)ovirt.org" > Subject: Re: [ovirt-users] gluster config in 4 node cluster Hi, As for me, I would build one cluster with gluster service only based on two= =3D nodes (replica 2), and the other one with virt service only based on other= =3D two nodes. I think this variant is more scalable in future. PS. I am a new in oVirt, so do not except that I am wrong. 28.05.2015 23:11, paf1(a)email.cz =3DEF=3DE8=3DF8= =3DE5=3DF2: Hello, How to optimal configure 4 node cluster for any one node goes to maintena= =3D nce without stopping VM ?? a) replica 4 - but it takes a lot of space b) disperse 3+1 ( raid 5 ) - but bad performance and not visible by oVirt 3= =3D .7.2 c) stripe2+replica2 =3D3D but VM paused any other idea ? regs. Pa. _______________________________________________ Users mailing list Users(a)ovirt.orghttp://lists.ovirt.org/mailman/l= isti=3D nfo/users --_000_D19165E2D96Esoerenmalchowmconnet_ Content-Type: text/html; charset=3D"windows-1251" Content-ID: <4CD1DEC785440646AA8474A539146A4D(a)liquidcampaign.com> Content-Transfer-Encoding: quoted-printable
Hi

For a production environment i would not build a 2 node gluster, i wou= =3D ld build at least 3 nodes to have it much easier with the quorum.

Taking into account that you can use commodity hardware i would also s= =3D uggest to split the services, but i would go for at least 3 gluster nodes = =3D =3D96 which add up to at least 5 nodes for a HA system.

If you want 4 in any case, then my suggestion would be to go for 4 rep= =3D licas, each node has anything that it needs to run and you can basically us= =3D e NFS to localhost for the storage which would make the nodes always access= =3D the local storage for the VMS, availability wise it makes no difference.

Cheers
Soeren 

From: =3DDE=3DF0=3DE8=3DE9 =3DCF= =3DEE=3DEB=3DF2=3DEE=3D =3DF0=3DE0=3DF6=3DEA=3DE8=3DE9 <y.poltor=3D atskiy(a)gmail.com>
Date: Sunday 31 May 2015 18:32
To: "users(a)ovirt.org" <=3D users(a)ovirt.org>
Subject: Re: [ovirt-users] gluste= r =3D config in 4 node cluster


Hi,

As for me, I would build one cluster with gluster service only based on two= =3D nodes (replica 2), and the other one with virt service only based on other= =3D two nodes. I think this variant is more scalable in future.

PS. I am a new in oVirt, so do not except that I am wrong.


28.05.2015 23:11, paf1(a)email.cz =3DEF=3DE8=3DF8=3DE5=3DF2:
Hell= o,
How to optimal configure  4 node cluster for any one node goes to = =3D ; maintenance without stopping VM ??

a) replica 4  - but it takes a lot of space
b) disperse 3+1 ( raid 5 ) - but bad performance and not visible by oVi= =3D rt 3.7.2
c) stripe2+replica2 =3D3D but VM paused

any other idea ?
regs.
Pa.


_______________________________________________
Users mailing list
Users=3D
@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users<=
=3D
/a>

--_000_D19165E2D96Esoerenmalchowmconnet_-- --===============6287059662962659540== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" LS1fMDAwX0QxOTE2NUUyRDk2RXNvZXJlbm1hbGNob3dtY29ubmV0XwpDb250ZW50LVR5cGU6IHRl eHQvcGxhaW47IGNoYXJzZXQ9IndpbmRvd3MtMTI1MSIKQ29udGVudC1UcmFuc2Zlci1FbmNvZGlu ZzogcXVvdGVkLXByaW50YWJsZQoKSGkKCkZvciBhIHByb2R1Y3Rpb24gZW52aXJvbm1lbnQgaSB3 b3VsZCBub3QgYnVpbGQgYSAyIG5vZGUgZ2x1c3RlciwgaSB3b3VsZCBidT0KaWxkIGF0IGxlYXN0 IDMgbm9kZXMgdG8gaGF2ZSBpdCBtdWNoIGVhc2llciB3aXRoIHRoZSBxdW9ydW0uCgpUYWtpbmcg aW50byBhY2NvdW50IHRoYXQgeW91IGNhbiB1c2UgY29tbW9kaXR5IGhhcmR3YXJlIGkgd291bGQg YWxzbyBzdWdnZXM9CnQgdG8gc3BsaXQgdGhlIHNlcnZpY2VzLCBidXQgaSB3b3VsZCBnbyBmb3Ig YXQgbGVhc3QgMyBnbHVzdGVyIG5vZGVzID05NiB3aD0KaWNoIGFkZCB1cCB0byBhdCBsZWFzdCA1 IG5vZGVzIGZvciBhIEhBIHN5c3RlbS4KCklmIHlvdSB3YW50IDQgaW4gYW55IGNhc2UsIHRoZW4g bXkgc3VnZ2VzdGlvbiB3b3VsZCBiZSB0byBnbyBmb3IgNCByZXBsaWNhcz0KLCBlYWNoIG5vZGUg aGFzIGFueXRoaW5nIHRoYXQgaXQgbmVlZHMgdG8gcnVuIGFuZCB5b3UgY2FuIGJhc2ljYWxseSB1 c2UgTkZTPQogdG8gbG9jYWxob3N0IGZvciB0aGUgc3RvcmFnZSB3aGljaCB3b3VsZCBtYWtlIHRo ZSBub2RlcyBhbHdheXMgYWNjZXNzIHRoZSA9CmxvY2FsIHN0b3JhZ2UgZm9yIHRoZSBWTVMsIGF2 YWlsYWJpbGl0eSB3aXNlIGl0IG1ha2VzIG5vIGRpZmZlcmVuY2UuCgpDaGVlcnMKU29lcmVuCgpG cm9tOiA9REU9RjA9RTg9RTkgPUNGPUVFPUVCPUYyPUVFPUYwPUUwPUY2PUVBPUU4PUU5IDx5LnBv bHRvcmF0c2tpeUBnbWFpbC49CmNvbTxtYWlsdG86eS5wb2x0b3JhdHNraXlAZ21haWwuY29tPj4K RGF0ZTogU3VuZGF5IDMxIE1heSAyMDE1IDE4OjMyClRvOiAidXNlcnNAb3ZpcnQub3JnPG1haWx0 bzp1c2Vyc0BvdmlydC5vcmc+IiA8dXNlcnNAb3ZpcnQub3JnPG1haWx0bzp1c2Vycz0KQG92aXJ0 Lm9yZz4+ClN1YmplY3Q6IFJlOiBbb3ZpcnQtdXNlcnNdIGdsdXN0ZXIgY29uZmlnIGluIDQgbm9k ZSBjbHVzdGVyCgoKSGksCgpBcyBmb3IgbWUsIEkgd291bGQgYnVpbGQgb25lIGNsdXN0ZXIgd2l0 aCBnbHVzdGVyIHNlcnZpY2Ugb25seSBiYXNlZCBvbiB0d289CiBub2RlcyAocmVwbGljYSAyKSwg YW5kIHRoZSBvdGhlciBvbmUgd2l0aCB2aXJ0IHNlcnZpY2Ugb25seSBiYXNlZCBvbiBvdGhlcj0K IHR3byBub2Rlcy4gSSB0aGluayB0aGlzIHZhcmlhbnQgaXMgbW9yZSBzY2FsYWJsZSBpbiBmdXR1 cmUuCgpQUy4gSSBhbSBhIG5ldyBpbiBvVmlydCwgc28gZG8gbm90IGV4Y2VwdCB0aGF0IEkgYW0g d3JvbmcuCgoKMjguMDUuMjAxNSAyMzoxMSwgcGFmMUBlbWFpbC5jejxtYWlsdG86cGFmMUBlbWFp bC5jej4gPUVGPUU4PUY4PUU1PUYyOgpIZWxsbywKSG93IHRvIG9wdGltYWwgY29uZmlndXJlICA0 IG5vZGUgY2x1c3RlciBmb3IgYW55IG9uZSBub2RlIGdvZXMgdG8gIG1haW50ZW5hPQpuY2Ugd2l0 aG91dCBzdG9wcGluZyBWTSA/PwoKYSkgcmVwbGljYSA0ICAtIGJ1dCBpdCB0YWtlcyBhIGxvdCBv ZiBzcGFjZQpiKSBkaXNwZXJzZSAzKzEgKCByYWlkIDUgKSAtIGJ1dCBiYWQgcGVyZm9ybWFuY2Ug YW5kIG5vdCB2aXNpYmxlIGJ5IG9WaXJ0IDM9Ci43LjIKYykgc3RyaXBlMityZXBsaWNhMiA9M0Qg YnV0IFZNIHBhdXNlZAoKYW55IG90aGVyIGlkZWEgPwpyZWdzLgpQYS4KCgoKX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KVXNlcnMgbWFpbGluZyBsaXN0ClVz ZXJzQG92aXJ0Lm9yZzxtYWlsdG86VXNlcnNAb3ZpcnQub3JnPmh0dHA6Ly9saXN0cy5vdmlydC5v cmcvbWFpbG1hbi9saXN0aT0KbmZvL3VzZXJzCgoKLS1fMDAwX0QxOTE2NUUyRDk2RXNvZXJlbm1h bGNob3dtY29ubmV0XwpDb250ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD0id2luZG93cy0x MjUxIgpDb250ZW50LUlEOiA8NENEMURFQzc4NTQ0MDY0NkFBODQ3NEE1MzkxNDZBNERAbGlxdWlk Y2FtcGFpZ24uY29tPgpDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiBxdW90ZWQtcHJpbnRhYmxl Cgo8aHRtbD4KPGhlYWQ+CjxtZXRhIGh0dHAtZXF1aXY9M0QiQ29udGVudC1UeXBlIiBjb250ZW50 PTNEInRleHQvaHRtbDsgY2hhcnNldD0zRHdpbmRvd3MtMT0KMjUxIj4KPC9oZWFkPgo8Ym9keSBz dHlsZT0zRCJ3b3JkLXdyYXA6IGJyZWFrLXdvcmQ7IC13ZWJraXQtbmJzcC1tb2RlOiBzcGFjZTsg LXdlYmtpdC1saW49CmUtYnJlYWs6IGFmdGVyLXdoaXRlLXNwYWNlOyBjb2xvcjogcmdiKDAsIDAs IDApOyBmb250LXNpemU6IDE0cHg7IGZvbnQtZmFtaT0KbHk6IENhbGlicmksIHNhbnMtc2VyaWY7 Ij4KPGRpdj5IaTwvZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+Rm9yIGEgcHJvZHVjdGlvbiBl bnZpcm9ubWVudCBpIHdvdWxkIG5vdCBidWlsZCBhIDIgbm9kZSBnbHVzdGVyLCBpIHdvdT0KbGQg YnVpbGQgYXQgbGVhc3QgMyBub2RlcyB0byBoYXZlIGl0IG11Y2ggZWFzaWVyIHdpdGggdGhlIHF1 b3J1bS48L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PlRha2luZyBpbnRvIGFjY291bnQgdGhh dCB5b3UgY2FuIHVzZSBjb21tb2RpdHkgaGFyZHdhcmUgaSB3b3VsZCBhbHNvIHM9CnVnZ2VzdCB0 byBzcGxpdCB0aGUgc2VydmljZXMsIGJ1dCBpIHdvdWxkIGdvIGZvciBhdCBsZWFzdCAzIGdsdXN0 ZXIgbm9kZXMgPQo9OTYgd2hpY2ggYWRkIHVwIHRvIGF0IGxlYXN0IDUgbm9kZXMgZm9yIGEgSEEg c3lzdGVtLjwvZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+SWYgeW91IHdhbnQgNCBpbiBhbnkg Y2FzZSwgdGhlbiBteSBzdWdnZXN0aW9uIHdvdWxkIGJlIHRvIGdvIGZvciA0IHJlcD0KbGljYXMs IGVhY2ggbm9kZSBoYXMgYW55dGhpbmcgdGhhdCBpdCBuZWVkcyB0byBydW4gYW5kIHlvdSBjYW4g YmFzaWNhbGx5IHVzPQplIE5GUyB0byBsb2NhbGhvc3QgZm9yIHRoZSBzdG9yYWdlIHdoaWNoIHdv dWxkIG1ha2UgdGhlIG5vZGVzIGFsd2F5cyBhY2Nlc3M9CiB0aGUgbG9jYWwgc3RvcmFnZSBmb3Ig dGhlIFZNUywKIGF2YWlsYWJpbGl0eSB3aXNlIGl0IG1ha2VzIG5vIGRpZmZlcmVuY2UuPC9kaXY+ CjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj5DaGVlcnM8L2Rpdj4KPGRpdj5Tb2VyZW4mbmJzcDs8L2Rp dj4KPGRpdj48YnI+CjwvZGl2Pgo8c3BhbiBpZD0zRCJPTEtfU1JDX0JPRFlfU0VDVElPTiI+Cjxk aXYgc3R5bGU9M0QiZm9udC1mYW1pbHk6Q2FsaWJyaTsgZm9udC1zaXplOjExcHQ7IHRleHQtYWxp Z246bGVmdDsgY29sb3I6Yj0KbGFjazsgQk9SREVSLUJPVFRPTTogbWVkaXVtIG5vbmU7IEJPUkRF Ui1MRUZUOiBtZWRpdW0gbm9uZTsgUEFERElORy1CT1RUT006PQogMGluOyBQQURESU5HLUxFRlQ6 IDBpbjsgUEFERElORy1SSUdIVDogMGluOyBCT1JERVItVE9QOiAjYjVjNGRmIDFwdCBzb2xpZDs9 CiBCT1JERVItUklHSFQ6IG1lZGl1bSBub25lOyBQQURESU5HLVRPUDogM3B0Ij4KPHNwYW4gc3R5 bGU9M0QiZm9udC13ZWlnaHQ6Ym9sZCI+RnJvbTogPC9zcGFuPj1ERT1GMD1FOD1FOSA9Q0Y9RUU9 RUI9RjI9RUU9Cj1GMD1FMD1GNj1FQT1FOD1FOSAmbHQ7PGEgaHJlZj0zRCJtYWlsdG86eS5wb2x0 b3JhdHNraXlAZ21haWwuY29tIj55LnBvbHRvcj0KYXRza2l5QGdtYWlsLmNvbTwvYT4mZ3Q7PGJy Pgo8c3BhbiBzdHlsZT0zRCJmb250LXdlaWdodDpib2xkIj5EYXRlOiA8L3NwYW4+U3VuZGF5IDMx IE1heSAyMDE1IDE4OjMyPGJyPgo8c3BhbiBzdHlsZT0zRCJmb250LXdlaWdodDpib2xkIj5Ubzog PC9zcGFuPiZxdW90OzxhIGhyZWY9M0QibWFpbHRvOnVzZXJzQG89CnZpcnQub3JnIj51c2Vyc0Bv dmlydC5vcmc8L2E+JnF1b3Q7ICZsdDs8YSBocmVmPTNEIm1haWx0bzp1c2Vyc0BvdmlydC5vcmci Pj0KdXNlcnNAb3ZpcnQub3JnPC9hPiZndDs8YnI+CjxzcGFuIHN0eWxlPTNEImZvbnQtd2VpZ2h0 OmJvbGQiPlN1YmplY3Q6IDwvc3Bhbj5SZTogW292aXJ0LXVzZXJzXSBnbHVzdGVyID0KY29uZmln IGluIDQgbm9kZSBjbHVzdGVyPGJyPgo8L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2Pgo8ZGl2 IHRleHQ9M0QiIzAwMDAwMCIgYmdjb2xvcj0zRCIjRkZGRkZGIj48YnI+CkhpLDxicj4KPGJyPgpB cyBmb3IgbWUsIEkgd291bGQgYnVpbGQgb25lIGNsdXN0ZXIgd2l0aCBnbHVzdGVyIHNlcnZpY2Ug b25seSBiYXNlZCBvbiB0d289CiBub2RlcyAocmVwbGljYSAyKSwgYW5kIHRoZSBvdGhlciBvbmUg d2l0aCB2aXJ0IHNlcnZpY2Ugb25seSBiYXNlZCBvbiBvdGhlcj0KIHR3byBub2Rlcy4gSSB0aGlu ayB0aGlzIHZhcmlhbnQgaXMgbW9yZSBzY2FsYWJsZSBpbiBmdXR1cmUuCjxicj4KPGJyPgpQUy4g SSBhbSBhIG5ldyBpbiBvVmlydCwgc28gZG8gbm90IGV4Y2VwdCB0aGF0IEkgYW0gd3JvbmcuPGJy Pgo8YnI+Cjxicj4KPGRpdiBjbGFzcz0zRCJtb3otY2l0ZS1wcmVmaXgiPjI4LjA1LjIwMTUgMjM6 MTEsIDxhIGNsYXNzPTNEIm1vei10eHQtbGluay1hPQpiYnJldmlhdGVkIiBocmVmPTNEIm1haWx0 bzpwYWYxQGVtYWlsLmN6Ij4KcGFmMUBlbWFpbC5jejwvYT4gPUVGPUU4PUY4PUU1PUYyOjxicj4K PC9kaXY+CjxibG9ja3F1b3RlIGNpdGU9M0QibWlkOjU1Njc3NjZELjEwNjAxMDRAZW1haWwuY3oi IHR5cGU9M0QiY2l0ZSI+SGVsbG8sPGJyPgpIb3cgdG8gb3B0aW1hbCBjb25maWd1cmUmbmJzcDsg NCBub2RlIGNsdXN0ZXIgZm9yIGFueSBvbmUgbm9kZSBnb2VzIHRvJm5ic3A9CjsgbWFpbnRlbmFu Y2Ugd2l0aG91dCBzdG9wcGluZyBWTSA/Pzxicj4KPGJyPgphKSByZXBsaWNhIDQmbmJzcDsgLSBi dXQgaXQgdGFrZXMgYSBsb3Qgb2Ygc3BhY2U8YnI+CmIpIGRpc3BlcnNlIDMmIzQzOzEgKCByYWlk IDUgKSAtIGJ1dCBiYWQgcGVyZm9ybWFuY2UgYW5kIG5vdCB2aXNpYmxlIGJ5IG9WaT0KcnQgMy43 LjI8YnI+CmMpIHN0cmlwZTImIzQzO3JlcGxpY2EyID0zRCBidXQgVk0gcGF1c2VkIDxicj4KPGJy Pgphbnkgb3RoZXIgaWRlYSA/PGJyPgpyZWdzLjxicj4KUGEuPGJyPgo8YnI+CjxmaWVsZHNldCBj bGFzcz0zRCJtaW1lQXR0YWNobWVudEhlYWRlciI+PC9maWVsZHNldD4gPGJyPgo8cHJlIHdyYXA9 M0QiIj5fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpVc2Vy cyBtYWlsaW5nIGxpc3QKPGEgY2xhc3M9M0QibW96LXR4dC1saW5rLWFiYnJldmlhdGVkIiBocmVm PTNEIm1haWx0bzpVc2Vyc0BvdmlydC5vcmciPlVzZXJzPQpAb3ZpcnQub3JnPC9hPjxhIGNsYXNz PTNEIm1vei10eHQtbGluay1mcmVldGV4dCIgaHJlZj0zRCJodHRwOi8vbGlzdHMub3ZpcnQ9Ci5v cmcvbWFpbG1hbi9saXN0aW5mby91c2VycyI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFu L2xpc3RpbmZvL3VzZXJzPD0KL2E+PC9wcmU+CjwvYmxvY2txdW90ZT4KPGJyPgo8L2Rpdj4KPC9k aXY+Cjwvc3Bhbj4KPC9ib2R5Pgo8L2h0bWw+CgotLV8wMDBfRDE5MTY1RTJEOTZFc29lcmVubWFs Y2hvd21jb25uZXRfLS0K --===============6287059662962659540==-- From y.poltoratskiy at gmail.com Mon Jun 1 02:56:10 2015 Content-Type: multipart/mixed; boundary="===============7981074770304979143==" MIME-Version: 1.0 From: =?utf-8?q?=D0=AE=D1=80=D0=B8=D0=B9_=D0=9F=D0=BE=D0=BB=D1=82=D0=BE=D1=80?= =?utf-8?q?=D0=B0=D1=86=D0=BA=D0=B8=D0=B9_=3Cy=2Epoltoratskiy_at_gmail=2Ec?= =?utf-8?q?om=3E?= To: users at ovirt.org Subject: Re: [ovirt-users] gluster config in 4 node cluster Date: Mon, 01 Jun 2015 09:56:08 +0300 Message-ID: In-Reply-To: D19165E2.D96E%soeren.malchow@mcon.net --===============7981074770304979143== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable 2015-06-01 2:32 GMT+03:00 Soeren Malchow : > Hi > > For a production environment i would not build a 2 node gluster, i would > build at least 3 nodes to have it much easier with the quorum. > > Taking into account that you can use commodity hardware i would also > suggest to split the services, but i would go for at least 3 gluster nodes > =E2=80=93 which add up to at least 5 nodes for a HA system. > > If you want 4 in any case, then my suggestion would be to go for 4 > replicas, each node has anything that it needs to run and you can basical= ly > use NFS to localhost for the storage which would make the nodes always > access the local storage for the VMS, availability wise it makes no > difference. > I have data center with one cluster based on 3 nodes with both services: virt and gluster, and GlusterFS as a data domain. Every node has a single HDD as a brick (replica 3) and 2x1G nics with bonding on top (src-dst-ip). I have 2 vlans: 1 - management, 2 - VMs. If in a VM I run something like 'dd if=3D/dev/zero of=3D/tmp/tmp' I have ar= ound 80+ MB/s write speed, at this moment on SPM node i run 'iftop -i bond0.1' and can see two flows with around 900 Mb/s each of them to the second and to the third nodes (thanks to the bonding that every flow can be done through separate nic). All of this is expected result and in this case we reach a limit of bandwidth our nics. And even if HDD of a node has 150+ MB/s linear write speed I can not reach such speed of write in VMs. What result would I get with 4 nodes and gluster replica 4? I expect that write speed in VM will be much slower, because now SPM node will generate three flows, and two of them will go through one nic and we will have only 500Mb/s per flow and in this way the write speed of VM will be twice lower. Am I wrong? > > Cheers > Soeren > > From: =D0=AE=D1=80=D0=B8=D0=B9 =D0=9F=D0=BE=D0=BB=D1=82=D0=BE=D1=80=D0= =B0=D1=86=D0=BA=D0=B8=D0=B9 > Date: Sunday 31 May 2015 18:32 > To: "users(a)ovirt.org" > Subject: Re: [ovirt-users] gluster config in 4 node cluster > > > Hi, > > As for me, I would build one cluster with gluster service only based on > two nodes (replica 2), and the other one with virt service only based on > other two nodes. I think this variant is more scalable in future. > > PS. I am a new in oVirt, so do not except that I am wrong. > > > 28.05.2015 23:11, paf1(a)email.cz =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > > Hello, > How to optimal configure 4 node cluster for any one node goes to > maintenance without stopping VM ?? > > a) replica 4 - but it takes a lot of space > b) disperse 3+1 ( raid 5 ) - but bad performance and not visible by oVirt > 3.7.2 > c) stripe2+replica2 =3D but VM paused > > any other idea ? > regs. > Pa. > > > _______________________________________________ > Users mailing listUsers(a)ovirt.orghttp://lists.ovirt.org/mailman/listinf= o/users > > > --===============7981074770304979143== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPjxicj48ZGl2IGNsYXNzPSJn bWFpbF9xdW90ZSI+MjAxNS0wNi0wMSAyOjMyIEdNVCswMzowMCBTb2VyZW4gTWFsY2hvdyA8c3Bh biBkaXI9Imx0ciI+Jmx0OzxhIGhyZWY9Im1haWx0bzpzb2VyZW4ubWFsY2hvd0BtY29uLm5ldCIg dGFyZ2V0PSJfYmxhbmsiPnNvZXJlbi5tYWxjaG93QG1jb24ubmV0PC9hPiZndDs8L3NwYW4+Ojxi cj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MHB4IDBweCAw cHggMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1s ZWZ0OjFleCI+CgoKCjxkaXYgc3R5bGU9IndvcmQtd3JhcDpicmVhay13b3JkO2NvbG9yOnJnYigw LDAsMCk7Zm9udC1zaXplOjE0cHg7Zm9udC1mYW1pbHk6Q2FsaWJyaSxzYW5zLXNlcmlmIj4KPGRp dj5IaTwvZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+Rm9yIGEgcHJvZHVjdGlvbiBlbnZpcm9u bWVudCBpIHdvdWxkIG5vdCBidWlsZCBhIDIgbm9kZSBnbHVzdGVyLCBpIHdvdWxkIGJ1aWxkIGF0 IGxlYXN0IDMgbm9kZXMgdG8gaGF2ZSBpdCBtdWNoIGVhc2llciB3aXRoIHRoZSBxdW9ydW0uPC9k aXY+CjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj5UYWtpbmcgaW50byBhY2NvdW50IHRoYXQgeW91IGNh biB1c2UgY29tbW9kaXR5IGhhcmR3YXJlIGkgd291bGQgYWxzbyBzdWdnZXN0IHRvIHNwbGl0IHRo ZSBzZXJ2aWNlcywgYnV0IGkgd291bGQgZ28gZm9yIGF0IGxlYXN0IDMgZ2x1c3RlciBub2RlcyDi gJMgd2hpY2ggYWRkIHVwIHRvIGF0IGxlYXN0IDUgbm9kZXMgZm9yIGEgSEEgc3lzdGVtLjwvZGl2 Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+SWYgeW91IHdhbnQgNCBpbiBhbnkgY2FzZSwgdGhlbiBt eSBzdWdnZXN0aW9uIHdvdWxkIGJlIHRvIGdvIGZvciA0IHJlcGxpY2FzLCBlYWNoIG5vZGUgaGFz IGFueXRoaW5nIHRoYXQgaXQgbmVlZHMgdG8gcnVuIGFuZCB5b3UgY2FuIGJhc2ljYWxseSB1c2Ug TkZTIHRvIGxvY2FsaG9zdCBmb3IgdGhlIHN0b3JhZ2Ugd2hpY2ggd291bGQgbWFrZSB0aGUgbm9k ZXMgYWx3YXlzIGFjY2VzcyB0aGUgbG9jYWwgc3RvcmFnZSBmb3IgdGhlIFZNUywKIGF2YWlsYWJp bGl0eSB3aXNlIGl0IG1ha2VzIG5vIGRpZmZlcmVuY2UuPC9kaXY+PC9kaXY+PC9ibG9ja3F1b3Rl PjxkaXY+PGJyPjwvZGl2PjxkaXY+SSBoYXZlIGRhdGEgY2VudGVyIHdpdGggb25lIGNsdXN0ZXIg YmFzZWQgb24gMyBub2RlcyB3aXRoIGJvdGggc2VydmljZXM6IHZpcnQgYW5kIGdsdXN0ZXIsIGFu ZCBHbHVzdGVyRlMgYXMgYSBkYXRhIGRvbWFpbi4gRXZlcnkgbm9kZSBoYXMgYSBzaW5nbGUgSERE IGFzIGEgYnJpY2sgKHJlcGxpY2EgMykgYW5kIDJ4MUcgbmljcyB3aXRoIGJvbmRpbmcgb24gdG9w IChzcmMtZHN0LWlwKS4gSSBoYXZlIDIgdmxhbnM6IDEgLSBtYW5hZ2VtZW50LCAyIC0gVk1zLjxi cj48YnI+PC9kaXY+PGRpdj5JZiBpbiBhIFZNIEkgcnVuIHNvbWV0aGluZyBsaWtlICYjMzk7ZGQg aWY9L2Rldi96ZXJvIG9mPS90bXAvdG1wJiMzOTsgSSBoYXZlIGFyb3VuZCA4MCsgTUIvcyB3cml0 ZSBzcGVlZCwgYXQgdGhpcyBtb21lbnQgb24gU1BNIG5vZGUgaSBydW4gJiMzOTtpZnRvcCAtaSBi b25kMC4xJiMzOTsgYW5kIGNhbiBzZWUgdHdvIGZsb3dzIHdpdGggYXJvdW5kIDkwMCBNYi9zIGVh Y2ggb2YgdGhlbSB0byB0aGUgc2Vjb25kIGFuZCB0byB0aGUgdGhpcmQgbm9kZXMgKHRoYW5rcyB0 byB0aGUgYm9uZGluZyB0aGF0IGV2ZXJ5IGZsb3cgY2FuIGJlIGRvbmUgdGhyb3VnaCBzZXBhcmF0 ZSBuaWMpLiBBbGwgb2YgdGhpcyBpcyBleHBlY3RlZCByZXN1bHQgYW5kIGluIHRoaXMgY2FzZSB3 ZSByZWFjaCBhIDxzcGFuIGlkPSJyZXN1bHRfYm94IiBjbGFzcz0iIiBsYW5nPSJlbiI+PHNwYW4g Y2xhc3M9IiI+bGltaXQ8L3NwYW4+IG9mIDxzcGFuIGNsYXNzPSIiPmJhbmR3aWR0aDwvc3Bhbj48 L3NwYW4+IG91ciBuaWNzLiBBbmQgZXZlbiBpZiBIREQgb2YgYSBub2RlIGhhcyAxNTArIE1CL3Mg bGluZWFyIHdyaXRlIHNwZWVkIEkgY2FuIG5vdCByZWFjaCBzdWNoIHNwZWVkIG9mIHdyaXRlIGlu IFZNcy48YnI+PGJyPjwvZGl2PjxkaXY+V2hhdCByZXN1bHQgd291bGQgSSBnZXQgd2l0aCA0IG5v ZGVzIGFuZCBnbHVzdGVyIHJlcGxpY2EgND8gSSBleHBlY3QgdGhhdCB3cml0ZSBzcGVlZCBpbiBW TSB3aWxsIGJlIG11Y2ggc2xvd2VyLCBiZWNhdXNlIG5vdyBTUE0gbm9kZSB3aWxsIGdlbmVyYXRl IHRocmVlIGZsb3dzLCBhbmQgdHdvIG9mIHRoZW0gd2lsbCBnbyB0aHJvdWdoIG9uZSBuaWMgYW5k IHdlIHdpbGwgaGF2ZSBvbmx5IDUwME1iL3MgcGVyIGZsb3cgYW5kIGluIHRoaXMgd2F5IHRoZSB3 cml0ZSBzcGVlZCBvZiBWTSB3aWxsIGJlIHR3aWNlIGxvd2VyLiA8YnI+PGJyPjwvZGl2PjxkaXY+ QW0gSSB3cm9uZz88YnI+PC9kaXY+PGRpdj48YnI+wqA8L2Rpdj48YmxvY2txdW90ZSBjbGFzcz0i Z21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQ6 MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCI+PGRpdiBzdHlsZT0i d29yZC13cmFwOmJyZWFrLXdvcmQ7Y29sb3I6cmdiKDAsMCwwKTtmb250LXNpemU6MTRweDtmb250 LWZhbWlseTpDYWxpYnJpLHNhbnMtc2VyaWYiPgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+Q2hlZXJz PC9kaXY+CjxkaXY+U29lcmVuwqA8L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8c3Bhbj4KPGRpdiBz dHlsZT0iZm9udC1mYW1pbHk6Q2FsaWJyaTtmb250LXNpemU6MTFwdDt0ZXh0LWFsaWduOmxlZnQ7 Y29sb3I6YmxhY2s7Ym9yZGVyLXdpZHRoOjFwdCBtZWRpdW0gbWVkaXVtO2JvcmRlci1zdHlsZTpz b2xpZCBub25lIG5vbmU7Ym9yZGVyLWNvbG9yOnJnYigxODEsMTk2LDIyMykgLW1vei11c2UtdGV4 dC1jb2xvciAtbW96LXVzZS10ZXh0LWNvbG9yO3BhZGRpbmc6M3B0IDBpbiAwaW4iPgo8c3BhbiBz dHlsZT0iZm9udC13ZWlnaHQ6Ym9sZCI+RnJvbTogPC9zcGFuPtCu0YDQuNC5INCf0L7Qu9GC0L7R gNCw0YbQutC40LkgJmx0OzxhIGhyZWY9Im1haWx0bzp5LnBvbHRvcmF0c2tpeUBnbWFpbC5jb20i IHRhcmdldD0iX2JsYW5rIj55LnBvbHRvcmF0c2tpeUBnbWFpbC5jb208L2E+Jmd0Ozxicj4KPHNw YW4gc3R5bGU9ImZvbnQtd2VpZ2h0OmJvbGQiPkRhdGU6IDwvc3Bhbj5TdW5kYXkgMzEgTWF5IDIw MTUgMTg6MzI8YnI+CjxzcGFuIHN0eWxlPSJmb250LXdlaWdodDpib2xkIj5UbzogPC9zcGFuPiZx dW90OzxhIGhyZWY9Im1haWx0bzp1c2Vyc0BvdmlydC5vcmciIHRhcmdldD0iX2JsYW5rIj51c2Vy c0BvdmlydC5vcmc8L2E+JnF1b3Q7ICZsdDs8YSBocmVmPSJtYWlsdG86dXNlcnNAb3ZpcnQub3Jn IiB0YXJnZXQ9Il9ibGFuayI+dXNlcnNAb3ZpcnQub3JnPC9hPiZndDs8YnI+CjxzcGFuIHN0eWxl PSJmb250LXdlaWdodDpib2xkIj5TdWJqZWN0OiA8L3NwYW4+UmU6IFtvdmlydC11c2Vyc10gZ2x1 c3RlciBjb25maWcgaW4gNCBub2RlIGNsdXN0ZXI8YnI+CjwvZGl2PjxkaXY+PGRpdiBjbGFzcz0i aDUiPgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+CjxkaXYgdGV4dD0iIzAwMDAwMCIgYmdjb2xvcj0i I0ZGRkZGRiI+PGJyPgpIaSw8YnI+Cjxicj4KQXMgZm9yIG1lLCBJIHdvdWxkIGJ1aWxkIG9uZSBj bHVzdGVyIHdpdGggZ2x1c3RlciBzZXJ2aWNlIG9ubHkgYmFzZWQgb24gdHdvIG5vZGVzIChyZXBs aWNhIDIpLCBhbmQgdGhlIG90aGVyIG9uZSB3aXRoIHZpcnQgc2VydmljZSBvbmx5IGJhc2VkIG9u IG90aGVyIHR3byBub2Rlcy4gSSB0aGluayB0aGlzIHZhcmlhbnQgaXMgbW9yZSBzY2FsYWJsZSBp biBmdXR1cmUuCjxicj4KPGJyPgpQUy4gSSBhbSBhIG5ldyBpbiBvVmlydCwgc28gZG8gbm90IGV4 Y2VwdCB0aGF0IEkgYW0gd3JvbmcuPGJyPgo8YnI+Cjxicj4KPGRpdj4yOC4wNS4yMDE1IDIzOjEx LCA8YSBocmVmPSJtYWlsdG86cGFmMUBlbWFpbC5jeiIgdGFyZ2V0PSJfYmxhbmsiPgpwYWYxQGVt YWlsLmN6PC9hPiDQv9C40YjQtdGCOjxicj4KPC9kaXY+CjxibG9ja3F1b3RlIHR5cGU9ImNpdGUi PkhlbGxvLDxicj4KSG93IHRvIG9wdGltYWwgY29uZmlndXJlwqAgNCBub2RlIGNsdXN0ZXIgZm9y IGFueSBvbmUgbm9kZSBnb2VzIHRvwqAgbWFpbnRlbmFuY2Ugd2l0aG91dCBzdG9wcGluZyBWTSA/ Pzxicj4KPGJyPgphKSByZXBsaWNhIDTCoCAtIGJ1dCBpdCB0YWtlcyBhIGxvdCBvZiBzcGFjZTxi cj4KYikgZGlzcGVyc2UgMysxICggcmFpZCA1ICkgLSBidXQgYmFkIHBlcmZvcm1hbmNlIGFuZCBu b3QgdmlzaWJsZSBieSBvVmlydCAzLjcuMjxicj4KYykgc3RyaXBlMityZXBsaWNhMiA9IGJ1dCBW TSBwYXVzZWQgPGJyPgo8YnI+CmFueSBvdGhlciBpZGVhID88YnI+CnJlZ3MuPGJyPgpQYS48YnI+ Cjxicj4KPGZpZWxkc2V0PjwvZmllbGRzZXQ+IDxicj4KPHByZT5fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fXwpVc2VycyBtYWlsaW5nIGxpc3QKPGEgaHJlZj0i bWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgdGFyZ2V0PSJfYmxhbmsiPlVzZXJzQG92aXJ0Lm9yZzwv YT48YSBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMi IHRhcmdldD0iX2JsYW5rIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8v dXNlcnM8L2E+PC9wcmU+CjwvYmxvY2txdW90ZT4KPGJyPgo8L2Rpdj4KPC9kaXY+CjwvZGl2Pjwv ZGl2Pjwvc3Bhbj4KPC9kaXY+Cgo8L2Jsb2NrcXVvdGU+PC9kaXY+PGJyPjwvZGl2PjwvZGl2Pgo= --===============7981074770304979143==-- From ml at ohnewald.net Mon Jun 1 03:27:25 2015 Content-Type: multipart/mixed; boundary="===============1068303624957683982==" MIME-Version: 1.0 From: ml at ohnewald.net To: users at ovirt.org Subject: Re: [ovirt-users] gluster config in 4 node cluster Date: Mon, 01 Jun 2015 09:27:23 +0200 Message-ID: <556C095B.6020509@ohnewald.net> In-Reply-To: D19165E2.D96E%soeren.malchow@mcon.net --===============1068303624957683982== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Am 01.06.15 um 01:32 schrieb Soeren Malchow: > Hi > > For a production environment i would not build a 2 node gluster, i would > build at least 3 nodes to have it much easier with the quorum. > Yes, dont build a 2 Node Gluster. Its a pain.... I started of with a two node gluster. In theory it works, but in = production it will give you headace. I switched to 3 Node Gluster. Each connected to each other with 2GBit = Bonding nics and i am very happy with it and it seems very stable. Even = if one node goes down. > Taking into account that you can use commodity hardware i would also > suggest to split the services, but i would go for at least 3 gluster > nodes =E2=80=93 which add up to at least 5 nodes for a HA system. > > If you want 4 in any case, then my suggestion would be to go for 4 > replicas, each node has anything that it needs to run and you can > basically use NFS to localhost for the storage which would make the > nodes always access the local storage for the VMS, availability wise it > makes no difference. > > Cheers > Soeren > > From: =D0=AE=D1=80=D0=B8=D0=B9 =D0=9F=D0=BE=D0=BB=D1=82=D0=BE=D1=80=D0=B0= =D1=86=D0=BA=D0=B8=D0=B9 > > Date: Sunday 31 May 2015 18:32 > To: "users(a)ovirt.org " > > Subject: Re: [ovirt-users] gluster config in 4 node cluster > > > Hi, > > As for me, I would build one cluster with gluster service only based on > two nodes (replica 2), and the other one with virt service only based on > other two nodes. I think this variant is more scalable in future. > > PS. I am a new in oVirt, so do not except that I am wrong. > > > 28.05.2015 23:11, paf1(a)email.cz =D0=BF=D0=B8=D1=88=D0=B5=D1=82: >> Hello, >> How to optimal configure 4 node cluster for any one node goes to >> maintenance without stopping VM ?? >> >> a) replica 4 - but it takes a lot of space >> b) disperse 3+1 ( raid 5 ) - but bad performance and not visible by >> oVirt 3.7.2 >> c) stripe2+replica2 =3D but VM paused >> >> any other idea ? >> regs. >> Pa. >> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users > > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > --===============1068303624957683982==-- From soeren.malchow at mcon.net Mon Jun 1 08:00:55 2015 Content-Type: multipart/mixed; boundary="===============7364879371235814889==" MIME-Version: 1.0 From: Soeren Malchow To: users at ovirt.org Subject: Re: [ovirt-users] gluster config in 4 node cluster Date: Mon, 01 Jun 2015 12:00:50 +0000 Message-ID: In-Reply-To: CANgBB_vkRhZtEpxKqJGM9QMDKO9+QuxuE7cfXdunyk_e3OJ4+Q@mail.gmail.com --===============7364879371235814889== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --_000_D1921574DA47soerenmalchowmconnet_ Content-Type: text/plain; charset=3D"windows-1251" Content-Transfer-Encoding: quoted-printable Hi You can have 150MB/s for a while, but evetually the replication has to catc= =3D h up and you will have more and more blocks out of sync. If you really need that much speed you need to go for 10GBit links There are some switches e.g .the ones compatible with Open Network Linux th= =3D at are relativley cheap when it comes to 10Gbit ( http://opennetlinux.org/h= =3D cl ) Cheers Soeren From: =3DDE=3DF0=3DE8=3DE9 =3DCF=3DEE=3DEB=3DF2=3DEE=3DF0=3DE0=3DF6=3DEA=3D= E8=3DE9 > Date: Monday 1 June 2015 08:56 To: Soeren Malchow =3D > Cc: "users(a)ovirt.org" > Subject: Re: [ovirt-users] gluster config in 4 node cluster 2015-06-01 2:32 GMT+03:00 Soeren Malchow >: Hi For a production environment i would not build a 2 node gluster, i would bu= =3D ild at least 3 nodes to have it much easier with the quorum. Taking into account that you can use commodity hardware i would also sugges= =3D t to split the services, but i would go for at least 3 gluster nodes =3D96 = wh=3D ich add up to at least 5 nodes for a HA system. If you want 4 in any case, then my suggestion would be to go for 4 replicas= =3D , each node has anything that it needs to run and you can basically use NFS= =3D to localhost for the storage which would make the nodes always access the = =3D local storage for the VMS, availability wise it makes no difference. I have data center with one cluster based on 3 nodes with both services: vi= =3D rt and gluster, and GlusterFS as a data domain. Every node has a single HDD= =3D as a brick (replica 3) and 2x1G nics with bonding on top (src-dst-ip). I h= =3D ave 2 vlans: 1 - management, 2 - VMs. If in a VM I run something like 'dd if=3D3D/dev/zero of=3D3D/tmp/tmp' I hav= e ar=3D ound 80+ MB/s write speed, at this moment on SPM node i run 'iftop -i bond0= =3D .1' and can see two flows with around 900 Mb/s each of them to the second a= =3D nd to the third nodes (thanks to the bonding that every flow can be done th= =3D rough separate nic). All of this is expected result and in this case we rea= =3D ch a limit of bandwidth our nics. And even if HDD of a node has 150+ MB/s l= =3D inear write speed I can not reach such speed of write in VMs. What result would I get with 4 nodes and gluster replica 4? I expect that w= =3D rite speed in VM will be much slower, because now SPM node will generate th= =3D ree flows, and two of them will go through one nic and we will have only 50= =3D 0Mb/s per flow and in this way the write speed of VM will be twice lower. Am I wrong? Cheers Soeren From: =3DDE=3DF0=3DE8=3DE9 =3DCF=3DEE=3DEB=3DF2=3DEE=3DF0=3DE0=3DF6=3DEA=3D= E8=3DE9 > Date: Sunday 31 May 2015 18:32 To: "users(a)ovirt.org" > Subject: Re: [ovirt-users] gluster config in 4 node cluster Hi, As for me, I would build one cluster with gluster service only based on two= =3D nodes (replica 2), and the other one with virt service only based on other= =3D two nodes. I think this variant is more scalable in future. PS. I am a new in oVirt, so do not except that I am wrong. 28.05.2015 23:11, paf1(a)email.cz =3DEF=3DE8=3DF8= =3DE5=3DF2: Hello, How to optimal configure 4 node cluster for any one node goes to maintena= =3D nce without stopping VM ?? a) replica 4 - but it takes a lot of space b) disperse 3+1 ( raid 5 ) - but bad performance and not visible by oVirt 3= =3D .7.2 c) stripe2+replica2 =3D3D but VM paused any other idea ? regs. Pa. _______________________________________________ Users mailing list Users(a)ovirt.orghttp://lists.ovirt.org/mailman/l= isti=3D nfo/users --_000_D1921574DA47soerenmalchowmconnet_ Content-Type: text/html; charset=3D"windows-1251" Content-ID: <506E2D25886427489658B38CE569CCCB(a)liquidcampaign.com> Content-Transfer-Encoding: quoted-printable
Hi

You can have 150MB/s for a while, but evetually the replication has to= =3D catch up and you will have more and more blocks out of sync.

If you really need that much speed you need to go for 10GBit links


Cheers
Soeren 

From: =3DDE=3DF0=3DE8=3DE9 =3DCF= =3DEE=3DEB=3DF2=3DEE=3D =3DF0=3DE0=3DF6=3DEA=3DE8=3DE9 <y.poltor=3D atskiy(a)gmail.com>
Date: Monday 1 June 2015 08:56
To: Soeren Malchow <soeren.malchow(a)mcon.net>
Cc: "users(a)ovirt.org" <=3D users(a)ovirt.org>
Subject: Re: [ovirt-users] gluste= r =3D config in 4 node cluster


2015-06-01 2:32 GMT+03:00 Soeren Malchow <= sp=3D an dir=3D3D"ltr"> <soer= en.mal=3D chow(a)mcon.net>:
Hi

For a production environment i would not build a 2 node gluster, i wou= =3D ld build at least 3 nodes to have it much easier with the quorum.

Taking into account that you can use commodity hardware i would also s= =3D uggest to split the services, but i would go for at least 3 gluster nodes = =3D =3D96 which add up to at least 5 nodes for a HA system.

If you want 4 in any case, then my suggestion would be to go for 4 rep= =3D licas, each node has anything that it needs to run and you can basically us= =3D e NFS to localhost for the storage which would make the nodes always access= =3D the local storage for the VMS, availability wise it makes no difference.

I have data center with one cluster based on 3 nodes with both service= =3D s: virt and gluster, and GlusterFS as a data domain. Every node has a singl= =3D e HDD as a brick (replica 3) and 2x1G nics with bonding on top (src-dst-ip)= =3D . I have 2 vlans: 1 - management, 2 - VMs.

If in a VM I run something like 'dd if=3D3D/dev/zero of=3D3D/tmp/tmp' = I ha=3D ve around 80+ MB/s write speed, at this moment on SPM node i run 'iftop= =3D -i bond0.1' and can see two flows with around 900 Mb/s each of them to the= =3D second and to the third nodes (thanks to the bonding that every flow can be done through separate nic). All of this= =3D is expected result and in this case we reach a lim= it of bandwidth our nics. And even if HDD of a node has 150+ MB= =3D /s linear write speed I can not reach such speed of write in VMs.

What result would I get with 4 nodes and gluster replica 4? I expect t= =3D hat write speed in VM will be much slower, because now SPM node will genera= =3D te three flows, and two of them will go through one nic and we will have on= =3D ly 500Mb/s per flow and in this way the write speed of VM will be twice lower.

Am I wrong?

 

Cheers
Soeren 

From: =3DDE=3DF0=3DE8=3DE9 =3DCF= =3DEE=3DEB=3DF2=3DEE=3D =3DF0=3DE0=3DF6=3DEA=3DE8=3DE9 <y.poltoratskiy(a)gmail.com>
Date: Sunday 31 May 2015 18:32
To: "users(a)ovirt.org" <users(a)ovirt.org>
Subject: Re: [ovirt-users] gluste= r =3D config in 4 node cluster


Hi,

As for me, I would build one cluster with gluster service only based on two= =3D nodes (replica 2), and the other one with virt service only based on other= =3D two nodes. I think this variant is more scalable in future.

PS. I am a new in oVirt, so do not except that I am wrong.


28.05.2015 23:11, p=3D af1(a)email.cz =3DEF=3DE8=3DF8=3DE5=3DF2:
Hello,
How to optimal configure  4 node cluster for any one node goes to = =3D ; maintenance without stopping VM ??

a) replica 4  - but it takes a lot of space
b) disperse 3+1 ( raid 5 ) - but bad performance and not visible by oVi= =3D rt 3.7.2
c) stripe2+replica2 =3D3D but VM paused

any other idea ?
regs.
Pa.


_______________________________________________
Users mailing list
Users(a)ovirt.or=
ght=3D
tp://lists.ovirt.org/mailman/listinfo/users


--_000_D1921574DA47soerenmalchowmconnet_-- --===============7364879371235814889== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" LS1fMDAwX0QxOTIxNTc0REE0N3NvZXJlbm1hbGNob3dtY29ubmV0XwpDb250ZW50LVR5cGU6IHRl eHQvcGxhaW47IGNoYXJzZXQ9IndpbmRvd3MtMTI1MSIKQ29udGVudC1UcmFuc2Zlci1FbmNvZGlu ZzogcXVvdGVkLXByaW50YWJsZQoKSGkKCllvdSBjYW4gaGF2ZSAxNTBNQi9zIGZvciBhIHdoaWxl LCBidXQgZXZldHVhbGx5IHRoZSByZXBsaWNhdGlvbiBoYXMgdG8gY2F0Yz0KaCB1cCBhbmQgeW91 IHdpbGwgaGF2ZSBtb3JlIGFuZCBtb3JlIGJsb2NrcyBvdXQgb2Ygc3luYy4KCklmIHlvdSByZWFs bHkgbmVlZCB0aGF0IG11Y2ggc3BlZWQgeW91IG5lZWQgdG8gZ28gZm9yIDEwR0JpdCBsaW5rcwoK VGhlcmUgYXJlIHNvbWUgc3dpdGNoZXMgZS5nIC50aGUgb25lcyBjb21wYXRpYmxlIHdpdGggT3Bl biBOZXR3b3JrIExpbnV4IHRoPQphdCBhcmUgcmVsYXRpdmxleSBjaGVhcCB3aGVuIGl0IGNvbWVz IHRvIDEwR2JpdCAoIGh0dHA6Ly9vcGVubmV0bGludXgub3JnL2g9CmNsICkKCkNoZWVycwpTb2Vy ZW4KCkZyb206ID1ERT1GMD1FOD1FOSA9Q0Y9RUU9RUI9RjI9RUU9RjA9RTA9RjY9RUE9RTg9RTkg PHkucG9sdG9yYXRza2l5QGdtYWlsLj0KY29tPG1haWx0bzp5LnBvbHRvcmF0c2tpeUBnbWFpbC5j b20+PgpEYXRlOiBNb25kYXkgMSBKdW5lIDIwMTUgMDg6NTYKVG86IFNvZXJlbiBNYWxjaG93IDxz b2VyZW4ubWFsY2hvd0BtY29uLm5ldDxtYWlsdG86c29lcmVuLm1hbGNob3dAbWNvbi5uZXQ+PQo+ CkNjOiAidXNlcnNAb3ZpcnQub3JnPG1haWx0bzp1c2Vyc0BvdmlydC5vcmc+IiA8dXNlcnNAb3Zp cnQub3JnPG1haWx0bzp1c2Vycz0KQG92aXJ0Lm9yZz4+ClN1YmplY3Q6IFJlOiBbb3ZpcnQtdXNl cnNdIGdsdXN0ZXIgY29uZmlnIGluIDQgbm9kZSBjbHVzdGVyCgoKMjAxNS0wNi0wMSAyOjMyIEdN VCswMzowMCBTb2VyZW4gTWFsY2hvdyA8c29lcmVuLm1hbGNob3dAbWNvbi5uZXQ8bWFpbHRvOnNv PQplcmVuLm1hbGNob3dAbWNvbi5uZXQ+PjoKSGkKCkZvciBhIHByb2R1Y3Rpb24gZW52aXJvbm1l bnQgaSB3b3VsZCBub3QgYnVpbGQgYSAyIG5vZGUgZ2x1c3RlciwgaSB3b3VsZCBidT0KaWxkIGF0 IGxlYXN0IDMgbm9kZXMgdG8gaGF2ZSBpdCBtdWNoIGVhc2llciB3aXRoIHRoZSBxdW9ydW0uCgpU YWtpbmcgaW50byBhY2NvdW50IHRoYXQgeW91IGNhbiB1c2UgY29tbW9kaXR5IGhhcmR3YXJlIGkg d291bGQgYWxzbyBzdWdnZXM9CnQgdG8gc3BsaXQgdGhlIHNlcnZpY2VzLCBidXQgaSB3b3VsZCBn byBmb3IgYXQgbGVhc3QgMyBnbHVzdGVyIG5vZGVzID05NiB3aD0KaWNoIGFkZCB1cCB0byBhdCBs ZWFzdCA1IG5vZGVzIGZvciBhIEhBIHN5c3RlbS4KCklmIHlvdSB3YW50IDQgaW4gYW55IGNhc2Us IHRoZW4gbXkgc3VnZ2VzdGlvbiB3b3VsZCBiZSB0byBnbyBmb3IgNCByZXBsaWNhcz0KLCBlYWNo IG5vZGUgaGFzIGFueXRoaW5nIHRoYXQgaXQgbmVlZHMgdG8gcnVuIGFuZCB5b3UgY2FuIGJhc2lj YWxseSB1c2UgTkZTPQogdG8gbG9jYWxob3N0IGZvciB0aGUgc3RvcmFnZSB3aGljaCB3b3VsZCBt YWtlIHRoZSBub2RlcyBhbHdheXMgYWNjZXNzIHRoZSA9CmxvY2FsIHN0b3JhZ2UgZm9yIHRoZSBW TVMsIGF2YWlsYWJpbGl0eSB3aXNlIGl0IG1ha2VzIG5vIGRpZmZlcmVuY2UuCgpJIGhhdmUgZGF0 YSBjZW50ZXIgd2l0aCBvbmUgY2x1c3RlciBiYXNlZCBvbiAzIG5vZGVzIHdpdGggYm90aCBzZXJ2 aWNlczogdmk9CnJ0IGFuZCBnbHVzdGVyLCBhbmQgR2x1c3RlckZTIGFzIGEgZGF0YSBkb21haW4u IEV2ZXJ5IG5vZGUgaGFzIGEgc2luZ2xlIEhERD0KIGFzIGEgYnJpY2sgKHJlcGxpY2EgMykgYW5k IDJ4MUcgbmljcyB3aXRoIGJvbmRpbmcgb24gdG9wIChzcmMtZHN0LWlwKS4gSSBoPQphdmUgMiB2 bGFuczogMSAtIG1hbmFnZW1lbnQsIDIgLSBWTXMuCgpJZiBpbiBhIFZNIEkgcnVuIHNvbWV0aGlu ZyBsaWtlICdkZCBpZj0zRC9kZXYvemVybyBvZj0zRC90bXAvdG1wJyBJIGhhdmUgYXI9Cm91bmQg ODArIE1CL3Mgd3JpdGUgc3BlZWQsIGF0IHRoaXMgbW9tZW50IG9uIFNQTSBub2RlIGkgcnVuICdp ZnRvcCAtaSBib25kMD0KLjEnIGFuZCBjYW4gc2VlIHR3byBmbG93cyB3aXRoIGFyb3VuZCA5MDAg TWIvcyBlYWNoIG9mIHRoZW0gdG8gdGhlIHNlY29uZCBhPQpuZCB0byB0aGUgdGhpcmQgbm9kZXMg KHRoYW5rcyB0byB0aGUgYm9uZGluZyB0aGF0IGV2ZXJ5IGZsb3cgY2FuIGJlIGRvbmUgdGg9CnJv dWdoIHNlcGFyYXRlIG5pYykuIEFsbCBvZiB0aGlzIGlzIGV4cGVjdGVkIHJlc3VsdCBhbmQgaW4g dGhpcyBjYXNlIHdlIHJlYT0KY2ggYSBsaW1pdCBvZiBiYW5kd2lkdGggb3VyIG5pY3MuIEFuZCBl dmVuIGlmIEhERCBvZiBhIG5vZGUgaGFzIDE1MCsgTUIvcyBsPQppbmVhciB3cml0ZSBzcGVlZCBJ IGNhbiBub3QgcmVhY2ggc3VjaCBzcGVlZCBvZiB3cml0ZSBpbiBWTXMuCgpXaGF0IHJlc3VsdCB3 b3VsZCBJIGdldCB3aXRoIDQgbm9kZXMgYW5kIGdsdXN0ZXIgcmVwbGljYSA0PyBJIGV4cGVjdCB0 aGF0IHc9CnJpdGUgc3BlZWQgaW4gVk0gd2lsbCBiZSBtdWNoIHNsb3dlciwgYmVjYXVzZSBub3cg U1BNIG5vZGUgd2lsbCBnZW5lcmF0ZSB0aD0KcmVlIGZsb3dzLCBhbmQgdHdvIG9mIHRoZW0gd2ls bCBnbyB0aHJvdWdoIG9uZSBuaWMgYW5kIHdlIHdpbGwgaGF2ZSBvbmx5IDUwPQowTWIvcyBwZXIg ZmxvdyBhbmQgaW4gdGhpcyB3YXkgdGhlIHdyaXRlIHNwZWVkIG9mIFZNIHdpbGwgYmUgdHdpY2Ug bG93ZXIuCgpBbSBJIHdyb25nPwoKCgpDaGVlcnMKU29lcmVuCgpGcm9tOiA9REU9RjA9RTg9RTkg PUNGPUVFPUVCPUYyPUVFPUYwPUUwPUY2PUVBPUU4PUU5IDx5LnBvbHRvcmF0c2tpeUBnbWFpbC49 CmNvbTxtYWlsdG86eS5wb2x0b3JhdHNraXlAZ21haWwuY29tPj4KRGF0ZTogU3VuZGF5IDMxIE1h eSAyMDE1IDE4OjMyClRvOiAidXNlcnNAb3ZpcnQub3JnPG1haWx0bzp1c2Vyc0BvdmlydC5vcmc+ IiA8dXNlcnNAb3ZpcnQub3JnPG1haWx0bzp1c2Vycz0KQG92aXJ0Lm9yZz4+ClN1YmplY3Q6IFJl OiBbb3ZpcnQtdXNlcnNdIGdsdXN0ZXIgY29uZmlnIGluIDQgbm9kZSBjbHVzdGVyCgoKSGksCgpB cyBmb3IgbWUsIEkgd291bGQgYnVpbGQgb25lIGNsdXN0ZXIgd2l0aCBnbHVzdGVyIHNlcnZpY2Ug b25seSBiYXNlZCBvbiB0d289CiBub2RlcyAocmVwbGljYSAyKSwgYW5kIHRoZSBvdGhlciBvbmUg d2l0aCB2aXJ0IHNlcnZpY2Ugb25seSBiYXNlZCBvbiBvdGhlcj0KIHR3byBub2Rlcy4gSSB0aGlu ayB0aGlzIHZhcmlhbnQgaXMgbW9yZSBzY2FsYWJsZSBpbiBmdXR1cmUuCgpQUy4gSSBhbSBhIG5l dyBpbiBvVmlydCwgc28gZG8gbm90IGV4Y2VwdCB0aGF0IEkgYW0gd3JvbmcuCgoKMjguMDUuMjAx NSAyMzoxMSwgcGFmMUBlbWFpbC5jejxtYWlsdG86cGFmMUBlbWFpbC5jej4gPUVGPUU4PUY4PUU1 PUYyOgpIZWxsbywKSG93IHRvIG9wdGltYWwgY29uZmlndXJlICA0IG5vZGUgY2x1c3RlciBmb3Ig YW55IG9uZSBub2RlIGdvZXMgdG8gIG1haW50ZW5hPQpuY2Ugd2l0aG91dCBzdG9wcGluZyBWTSA/ PwoKYSkgcmVwbGljYSA0ICAtIGJ1dCBpdCB0YWtlcyBhIGxvdCBvZiBzcGFjZQpiKSBkaXNwZXJz ZSAzKzEgKCByYWlkIDUgKSAtIGJ1dCBiYWQgcGVyZm9ybWFuY2UgYW5kIG5vdCB2aXNpYmxlIGJ5 IG9WaXJ0IDM9Ci43LjIKYykgc3RyaXBlMityZXBsaWNhMiA9M0QgYnV0IFZNIHBhdXNlZAoKYW55 IG90aGVyIGlkZWEgPwpyZWdzLgpQYS4KCgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX18KVXNlcnMgbWFpbGluZyBsaXN0ClVzZXJzQG92aXJ0Lm9yZzxtYWls dG86VXNlcnNAb3ZpcnQub3JnPmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aT0K bmZvL3VzZXJzCgoKCi0tXzAwMF9EMTkyMTU3NERBNDdzb2VyZW5tYWxjaG93bWNvbm5ldF8KQ29u dGVudC1UeXBlOiB0ZXh0L2h0bWw7IGNoYXJzZXQ9IndpbmRvd3MtMTI1MSIKQ29udGVudC1JRDog PDUwNkUyRDI1ODg2NDI3NDg5NjU4QjM4Q0U1NjlDQ0NCQGxpcXVpZGNhbXBhaWduLmNvbT4KQ29u dGVudC1UcmFuc2Zlci1FbmNvZGluZzogcXVvdGVkLXByaW50YWJsZQoKPGh0bWw+CjxoZWFkPgo8 bWV0YSBodHRwLWVxdWl2PTNEIkNvbnRlbnQtVHlwZSIgY29udGVudD0zRCJ0ZXh0L2h0bWw7IGNo YXJzZXQ9M0R3aW5kb3dzLTE9CjI1MSI+CjwvaGVhZD4KPGJvZHkgc3R5bGU9M0Qid29yZC13cmFw OiBicmVhay13b3JkOyAtd2Via2l0LW5ic3AtbW9kZTogc3BhY2U7IC13ZWJraXQtbGluPQplLWJy ZWFrOiBhZnRlci13aGl0ZS1zcGFjZTsgY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1zaXplOiAx NHB4OyBmb250LWZhbWk9Cmx5OiBDYWxpYnJpLCBzYW5zLXNlcmlmOyI+CjxkaXY+SGk8L2Rpdj4K PGRpdj48YnI+CjwvZGl2Pgo8ZGl2PllvdSBjYW4gaGF2ZSAxNTBNQi9zIGZvciBhIHdoaWxlLCBi dXQgZXZldHVhbGx5IHRoZSByZXBsaWNhdGlvbiBoYXMgdG89CiBjYXRjaCB1cCBhbmQgeW91IHdp bGwgaGF2ZSBtb3JlIGFuZCBtb3JlIGJsb2NrcyBvdXQgb2Ygc3luYy48L2Rpdj4KPGRpdj48YnI+ CjwvZGl2Pgo8ZGl2PklmIHlvdSByZWFsbHkgbmVlZCB0aGF0IG11Y2ggc3BlZWQgeW91IG5lZWQg dG8gZ28gZm9yIDEwR0JpdCBsaW5rczwvZGk9CnY+CjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj5UaGVy ZSBhcmUgc29tZSBzd2l0Y2hlcyBlLmcgLnRoZSBvbmVzIGNvbXBhdGlibGUgd2l0aCBPcGVuIE5l dHdvcmsgTGluPQp1eCB0aGF0IGFyZSByZWxhdGl2bGV5IGNoZWFwIHdoZW4gaXQgY29tZXMgdG8g MTBHYml0ICgmbmJzcDs8YSBocmVmPTNEImh0dHA9CjovL29wZW5uZXRsaW51eC5vcmcvaGNsIj5o dHRwOi8vb3Blbm5ldGxpbnV4Lm9yZy9oY2w8L2E+Jm5ic3A7KTwvZGl2Pgo8ZGl2Pjxicj4KPC9k aXY+CjxkaXY+Q2hlZXJzPC9kaXY+CjxkaXY+U29lcmVuJm5ic3A7PC9kaXY+CjxkaXY+PGJyPgo8 L2Rpdj4KPHNwYW4gaWQ9M0QiT0xLX1NSQ19CT0RZX1NFQ1RJT04iPgo8ZGl2IHN0eWxlPTNEImZv bnQtZmFtaWx5OkNhbGlicmk7IGZvbnQtc2l6ZToxMXB0OyB0ZXh0LWFsaWduOmxlZnQ7IGNvbG9y OmI9CmxhY2s7IEJPUkRFUi1CT1RUT006IG1lZGl1bSBub25lOyBCT1JERVItTEVGVDogbWVkaXVt IG5vbmU7IFBBRERJTkctQk9UVE9NOj0KIDBpbjsgUEFERElORy1MRUZUOiAwaW47IFBBRERJTkct UklHSFQ6IDBpbjsgQk9SREVSLVRPUDogI2I1YzRkZiAxcHQgc29saWQ7PQogQk9SREVSLVJJR0hU OiBtZWRpdW0gbm9uZTsgUEFERElORy1UT1A6IDNwdCI+CjxzcGFuIHN0eWxlPTNEImZvbnQtd2Vp Z2h0OmJvbGQiPkZyb206IDwvc3Bhbj49REU9RjA9RTg9RTkgPUNGPUVFPUVCPUYyPUVFPQo9RjA9 RTA9RjY9RUE9RTg9RTkgJmx0OzxhIGhyZWY9M0QibWFpbHRvOnkucG9sdG9yYXRza2l5QGdtYWls LmNvbSI+eS5wb2x0b3I9CmF0c2tpeUBnbWFpbC5jb208L2E+Jmd0Ozxicj4KPHNwYW4gc3R5bGU9 M0QiZm9udC13ZWlnaHQ6Ym9sZCI+RGF0ZTogPC9zcGFuPk1vbmRheSAxIEp1bmUgMjAxNSAwODo1 Njxicj4KPHNwYW4gc3R5bGU9M0QiZm9udC13ZWlnaHQ6Ym9sZCI+VG86IDwvc3Bhbj5Tb2VyZW4g TWFsY2hvdyAmbHQ7PGEgaHJlZj0zRCJtPQphaWx0bzpzb2VyZW4ubWFsY2hvd0BtY29uLm5ldCI+ c29lcmVuLm1hbGNob3dAbWNvbi5uZXQ8L2E+Jmd0Ozxicj4KPHNwYW4gc3R5bGU9M0QiZm9udC13 ZWlnaHQ6Ym9sZCI+Q2M6IDwvc3Bhbj4mcXVvdDs8YSBocmVmPTNEIm1haWx0bzp1c2Vyc0BvPQp2 aXJ0Lm9yZyI+dXNlcnNAb3ZpcnQub3JnPC9hPiZxdW90OyAmbHQ7PGEgaHJlZj0zRCJtYWlsdG86 dXNlcnNAb3ZpcnQub3JnIj49CnVzZXJzQG92aXJ0Lm9yZzwvYT4mZ3Q7PGJyPgo8c3BhbiBzdHls ZT0zRCJmb250LXdlaWdodDpib2xkIj5TdWJqZWN0OiA8L3NwYW4+UmU6IFtvdmlydC11c2Vyc10g Z2x1c3RlciA9CmNvbmZpZyBpbiA0IG5vZGUgY2x1c3Rlcjxicj4KPC9kaXY+CjxkaXY+PGJyPgo8 L2Rpdj4KPGRpdj4KPGRpdj4KPGRpdiBkaXI9M0QibHRyIj4KPGRpdiBjbGFzcz0zRCJnbWFpbF9l eHRyYSI+PGJyPgo8ZGl2IGNsYXNzPTNEImdtYWlsX3F1b3RlIj4yMDE1LTA2LTAxIDI6MzIgR01U JiM0MzswMzowMCBTb2VyZW4gTWFsY2hvdyA8c3A9CmFuIGRpcj0zRCJsdHIiPgombHQ7PGEgaHJl Zj0zRCJtYWlsdG86c29lcmVuLm1hbGNob3dAbWNvbi5uZXQiIHRhcmdldD0zRCJfYmxhbmsiPnNv ZXJlbi5tYWw9CmNob3dAbWNvbi5uZXQ8L2E+Jmd0Ozwvc3Bhbj46PGJyPgo8YmxvY2txdW90ZSBj bGFzcz0zRCJnbWFpbF9xdW90ZSIgc3R5bGU9M0QibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2Jv cmRlci09CmxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCI+ CjxkaXYgc3R5bGU9M0Qid29yZC13cmFwOmJyZWFrLXdvcmQ7Y29sb3I6cmdiKDAsMCwwKTtmb250 LXNpemU6MTRweDtmb250LWZhbT0KaWx5OkNhbGlicmksc2Fucy1zZXJpZiI+CjxkaXY+SGk8L2Rp dj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PkZvciBhIHByb2R1Y3Rpb24gZW52aXJvbm1lbnQgaSB3 b3VsZCBub3QgYnVpbGQgYSAyIG5vZGUgZ2x1c3RlciwgaSB3b3U9CmxkIGJ1aWxkIGF0IGxlYXN0 IDMgbm9kZXMgdG8gaGF2ZSBpdCBtdWNoIGVhc2llciB3aXRoIHRoZSBxdW9ydW0uPC9kaXY+Cjxk aXY+PGJyPgo8L2Rpdj4KPGRpdj5UYWtpbmcgaW50byBhY2NvdW50IHRoYXQgeW91IGNhbiB1c2Ug Y29tbW9kaXR5IGhhcmR3YXJlIGkgd291bGQgYWxzbyBzPQp1Z2dlc3QgdG8gc3BsaXQgdGhlIHNl cnZpY2VzLCBidXQgaSB3b3VsZCBnbyBmb3IgYXQgbGVhc3QgMyBnbHVzdGVyIG5vZGVzID0KPTk2 IHdoaWNoIGFkZCB1cCB0byBhdCBsZWFzdCA1IG5vZGVzIGZvciBhIEhBIHN5c3RlbS48L2Rpdj4K PGRpdj48YnI+CjwvZGl2Pgo8ZGl2PklmIHlvdSB3YW50IDQgaW4gYW55IGNhc2UsIHRoZW4gbXkg c3VnZ2VzdGlvbiB3b3VsZCBiZSB0byBnbyBmb3IgNCByZXA9CmxpY2FzLCBlYWNoIG5vZGUgaGFz IGFueXRoaW5nIHRoYXQgaXQgbmVlZHMgdG8gcnVuIGFuZCB5b3UgY2FuIGJhc2ljYWxseSB1cz0K ZSBORlMgdG8gbG9jYWxob3N0IGZvciB0aGUgc3RvcmFnZSB3aGljaCB3b3VsZCBtYWtlIHRoZSBu b2RlcyBhbHdheXMgYWNjZXNzPQogdGhlIGxvY2FsIHN0b3JhZ2UgZm9yIHRoZSBWTVMsCiBhdmFp bGFiaWxpdHkgd2lzZSBpdCBtYWtlcyBubyBkaWZmZXJlbmNlLjwvZGl2Pgo8L2Rpdj4KPC9ibG9j a3F1b3RlPgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+SSBoYXZlIGRhdGEgY2VudGVyIHdpdGggb25l IGNsdXN0ZXIgYmFzZWQgb24gMyBub2RlcyB3aXRoIGJvdGggc2VydmljZT0KczogdmlydCBhbmQg Z2x1c3RlciwgYW5kIEdsdXN0ZXJGUyBhcyBhIGRhdGEgZG9tYWluLiBFdmVyeSBub2RlIGhhcyBh IHNpbmdsPQplIEhERCBhcyBhIGJyaWNrIChyZXBsaWNhIDMpIGFuZCAyeDFHIG5pY3Mgd2l0aCBi b25kaW5nIG9uIHRvcCAoc3JjLWRzdC1pcCk9Ci4gSSBoYXZlIDIgdmxhbnM6IDEgLSBtYW5hZ2Vt ZW50LAogMiAtIFZNcy48YnI+Cjxicj4KPC9kaXY+CjxkaXY+SWYgaW4gYSBWTSBJIHJ1biBzb21l dGhpbmcgbGlrZSAnZGQgaWY9M0QvZGV2L3plcm8gb2Y9M0QvdG1wL3RtcCcgSSBoYT0KdmUgYXJv dW5kIDgwJiM0MzsgTUIvcyB3cml0ZSBzcGVlZCwgYXQgdGhpcyBtb21lbnQgb24gU1BNIG5vZGUg aSBydW4gJ2lmdG9wPQogLWkgYm9uZDAuMScgYW5kIGNhbiBzZWUgdHdvIGZsb3dzIHdpdGggYXJv dW5kIDkwMCBNYi9zIGVhY2ggb2YgdGhlbSB0byB0aGU9CiBzZWNvbmQgYW5kIHRvIHRoZSB0aGly ZCBub2RlcyAodGhhbmtzIHRvCiB0aGUgYm9uZGluZyB0aGF0IGV2ZXJ5IGZsb3cgY2FuIGJlIGRv bmUgdGhyb3VnaCBzZXBhcmF0ZSBuaWMpLiBBbGwgb2YgdGhpcz0KIGlzIGV4cGVjdGVkIHJlc3Vs dCBhbmQgaW4gdGhpcyBjYXNlIHdlIHJlYWNoIGEKPHNwYW4gaWQ9M0QicmVzdWx0X2JveCIgY2xh c3M9M0QiIiBsYW5nPTNEImVuIj48c3BhbiBjbGFzcz0zRCIiPmxpbWl0PC9zcGFuPQo+IG9mIDxz cGFuIGNsYXNzPTNEIiI+CmJhbmR3aWR0aDwvc3Bhbj48L3NwYW4+IG91ciBuaWNzLiBBbmQgZXZl biBpZiBIREQgb2YgYSBub2RlIGhhcyAxNTAmIzQzOyBNQj0KL3MgbGluZWFyIHdyaXRlIHNwZWVk IEkgY2FuIG5vdCByZWFjaCBzdWNoIHNwZWVkIG9mIHdyaXRlIGluIFZNcy48YnI+Cjxicj4KPC9k aXY+CjxkaXY+V2hhdCByZXN1bHQgd291bGQgSSBnZXQgd2l0aCA0IG5vZGVzIGFuZCBnbHVzdGVy IHJlcGxpY2EgND8gSSBleHBlY3QgdD0KaGF0IHdyaXRlIHNwZWVkIGluIFZNIHdpbGwgYmUgbXVj aCBzbG93ZXIsIGJlY2F1c2Ugbm93IFNQTSBub2RlIHdpbGwgZ2VuZXJhPQp0ZSB0aHJlZSBmbG93 cywgYW5kIHR3byBvZiB0aGVtIHdpbGwgZ28gdGhyb3VnaCBvbmUgbmljIGFuZCB3ZSB3aWxsIGhh dmUgb249Cmx5IDUwME1iL3MgcGVyIGZsb3cgYW5kIGluIHRoaXMKIHdheSB0aGUgd3JpdGUgc3Bl ZWQgb2YgVk0gd2lsbCBiZSB0d2ljZSBsb3dlci4gPGJyPgo8YnI+CjwvZGl2Pgo8ZGl2PkFtIEkg d3Jvbmc/PGJyPgo8L2Rpdj4KPGRpdj48YnI+CiZuYnNwOzwvZGl2Pgo8YmxvY2txdW90ZSBjbGFz cz0zRCJnbWFpbF9xdW90ZSIgc3R5bGU9M0QibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRl ci09CmxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCI+Cjxk aXYgc3R5bGU9M0Qid29yZC13cmFwOmJyZWFrLXdvcmQ7Y29sb3I6cmdiKDAsMCwwKTtmb250LXNp emU6MTRweDtmb250LWZhbT0KaWx5OkNhbGlicmksc2Fucy1zZXJpZiI+CjxkaXY+PGJyPgo8L2Rp dj4KPGRpdj5DaGVlcnM8L2Rpdj4KPGRpdj5Tb2VyZW4mbmJzcDs8L2Rpdj4KPGRpdj48YnI+Cjwv ZGl2Pgo8c3Bhbj4KPGRpdiBzdHlsZT0zRCJmb250LWZhbWlseTpDYWxpYnJpO2ZvbnQtc2l6ZTox MXB0O3RleHQtYWxpZ246bGVmdDtjb2xvcjpibGFjPQprO2JvcmRlci13aWR0aDoxcHQgbWVkaXVt IG1lZGl1bTtib3JkZXItc3R5bGU6c29saWQgbm9uZSBub25lO2JvcmRlci1jb2xvcjo9CnJnYigx ODEsMTk2LDIyMykgLW1vei11c2UtdGV4dC1jb2xvciAtbW96LXVzZS10ZXh0LWNvbG9yO3BhZGRp bmc6M3B0IDBpbiAwaT0KbiI+CjxzcGFuIHN0eWxlPTNEImZvbnQtd2VpZ2h0OmJvbGQiPkZyb206 IDwvc3Bhbj49REU9RjA9RTg9RTkgPUNGPUVFPUVCPUYyPUVFPQo9RjA9RTA9RjY9RUE9RTg9RTkg Jmx0OzxhIGhyZWY9M0QibWFpbHRvOnkucG9sdG9yYXRza2l5QGdtYWlsLmNvbSIgdGFyZ2V0PQo9 M0QiX2JsYW5rIj55LnBvbHRvcmF0c2tpeUBnbWFpbC5jb208L2E+Jmd0Ozxicj4KPHNwYW4gc3R5 bGU9M0QiZm9udC13ZWlnaHQ6Ym9sZCI+RGF0ZTogPC9zcGFuPlN1bmRheSAzMSBNYXkgMjAxNSAx ODozMjxicj4KPHNwYW4gc3R5bGU9M0QiZm9udC13ZWlnaHQ6Ym9sZCI+VG86IDwvc3Bhbj4mcXVv dDs8YSBocmVmPTNEIm1haWx0bzp1c2Vyc0BvPQp2aXJ0Lm9yZyIgdGFyZ2V0PTNEIl9ibGFuayI+ dXNlcnNAb3ZpcnQub3JnPC9hPiZxdW90OyAmbHQ7PGEgaHJlZj0zRCJtYWlsdG89Cjp1c2Vyc0Bv dmlydC5vcmciIHRhcmdldD0zRCJfYmxhbmsiPnVzZXJzQG92aXJ0Lm9yZzwvYT4mZ3Q7PGJyPgo8 c3BhbiBzdHlsZT0zRCJmb250LXdlaWdodDpib2xkIj5TdWJqZWN0OiA8L3NwYW4+UmU6IFtvdmly dC11c2Vyc10gZ2x1c3RlciA9CmNvbmZpZyBpbiA0IG5vZGUgY2x1c3Rlcjxicj4KPC9kaXY+Cjxk aXY+CjxkaXYgY2xhc3M9M0QiaDUiPgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+CjxkaXYgdGV4dD0z RCIjMDAwMDAwIiBiZ2NvbG9yPTNEIiNGRkZGRkYiPjxicj4KSGksPGJyPgo8YnI+CkFzIGZvciBt ZSwgSSB3b3VsZCBidWlsZCBvbmUgY2x1c3RlciB3aXRoIGdsdXN0ZXIgc2VydmljZSBvbmx5IGJh c2VkIG9uIHR3bz0KIG5vZGVzIChyZXBsaWNhIDIpLCBhbmQgdGhlIG90aGVyIG9uZSB3aXRoIHZp cnQgc2VydmljZSBvbmx5IGJhc2VkIG9uIG90aGVyPQogdHdvIG5vZGVzLiBJIHRoaW5rIHRoaXMg dmFyaWFudCBpcyBtb3JlIHNjYWxhYmxlIGluIGZ1dHVyZS4KPGJyPgo8YnI+ClBTLiBJIGFtIGEg bmV3IGluIG9WaXJ0LCBzbyBkbyBub3QgZXhjZXB0IHRoYXQgSSBhbSB3cm9uZy48YnI+Cjxicj4K PGJyPgo8ZGl2PjI4LjA1LjIwMTUgMjM6MTEsIDxhIGhyZWY9M0QibWFpbHRvOnBhZjFAZW1haWwu Y3oiIHRhcmdldD0zRCJfYmxhbmsiPnA9CmFmMUBlbWFpbC5jejwvYT4gPUVGPUU4PUY4PUU1PUYy Ojxicj4KPC9kaXY+CjxibG9ja3F1b3RlIHR5cGU9M0QiY2l0ZSI+SGVsbG8sPGJyPgpIb3cgdG8g b3B0aW1hbCBjb25maWd1cmUmbmJzcDsgNCBub2RlIGNsdXN0ZXIgZm9yIGFueSBvbmUgbm9kZSBn b2VzIHRvJm5ic3A9CjsgbWFpbnRlbmFuY2Ugd2l0aG91dCBzdG9wcGluZyBWTSA/Pzxicj4KPGJy PgphKSByZXBsaWNhIDQmbmJzcDsgLSBidXQgaXQgdGFrZXMgYSBsb3Qgb2Ygc3BhY2U8YnI+CmIp IGRpc3BlcnNlIDMmIzQzOzEgKCByYWlkIDUgKSAtIGJ1dCBiYWQgcGVyZm9ybWFuY2UgYW5kIG5v dCB2aXNpYmxlIGJ5IG9WaT0KcnQgMy43LjI8YnI+CmMpIHN0cmlwZTImIzQzO3JlcGxpY2EyID0z RCBidXQgVk0gcGF1c2VkIDxicj4KPGJyPgphbnkgb3RoZXIgaWRlYSA/PGJyPgpyZWdzLjxicj4K UGEuPGJyPgo8YnI+CjxmaWVsZHNldD48L2ZpZWxkc2V0PiA8YnI+CjxwcmU+X19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KVXNlcnMgbWFpbGluZyBsaXN0Cjxh IGhyZWY9M0QibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgdGFyZ2V0PTNEIl9ibGFuayI+VXNlcnNA b3ZpcnQub3JnPC9hPjxhID0KaHJlZj0zRCJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4v bGlzdGluZm8vdXNlcnMiIHRhcmdldD0zRCJfYmxhbmsiPmh0PQp0cDovL2xpc3RzLm92aXJ0Lm9y Zy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPjwvcHJlPgo8L2Jsb2NrcXVvdGU+Cjxicj4KPC9k aXY+CjwvZGl2Pgo8L2Rpdj4KPC9kaXY+Cjwvc3Bhbj48L2Rpdj4KPC9ibG9ja3F1b3RlPgo8L2Rp dj4KPGJyPgo8L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8L2Rpdj4KPC9zcGFuPgo8L2JvZHk+CjwvaHRt bD4KCi0tXzAwMF9EMTkyMTU3NERBNDdzb2VyZW5tYWxjaG93bWNvbm5ldF8tLQo= --===============7364879371235814889==--