Devin,

Many, many thaks for your response. I will read the doc you sent and if = =3D I still have questions I will post them here.

But why would I use a RAIDed brick if Gluster, by itself, already "= =3D protects" the data by making replicas. You see, that is what is confus= =3D ing to me...

Thanks,

Moacir

From: Devin Acosta &l= t;de=3D vin(a)pabstatencio.com>
Sent: Monday, August 7, 2017 7:46 AM
To: Moacir Ferreira; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices

Moacir,

I have recently installed multiple Red Hat Virtualizat= io=3D n hosts for several different companies, and have dealt with the Red Hat Su= =3D pport Team in depth about optimal configuration in regards to setting up GlusterFS most efficiently and I wanted to share = =3D with you what I learned.

In general Red Hat Virtualization team frowns upon usi= ng=3D each DISK of the system as just a JBOD, sure there is some protection by h= =3D aving the data replicated, however, the recommendation is to use RAID 6 (preferred) or RAID-5, or at least RAID-1 = =3D at the very least.

Here is the direct quote from Red Hat when I asked about RAID and Bri= =3D cks:

"A typical Gluster configuration would use RAID underneath the bri= =3D cks. RAID 6 is most typical as it gives you 2 disk failure protection, but = =3D RAID 5 could be used too. Once you have the RAIDed bricks, you'd then apply the desired replication on top of that. Th= =3D e most popular way of doing this would be distributed replicated with 2x re= =3D plication. In general you'll get better performance with larger bricks= =3D . 12 drives is often a sweet spot. Another option would be to create a separate tier using all SSD=3D92s.=3D94 <= /i><=3D /font>

In order to SSD tiering from my understanding you would need 1 x N= =3D VMe drive in each server, or 4 x SSD hot tier (it needs to be distributed, = =3D replicated for the hot tier if not using NVME). So with you only having 1 SSD drive in each server, I=3D92d suggest m= ay=3D be looking into the NVME option.

Since your using only 3-servers, what I=3D92d probably suggest is to do= (=3D 2 Replicas + Arbiter Node), this setup actually doesn=3D92t require the= 3=3D rd server to have big drives at all as it only stores meta-data about the files and not actually a full copy.

Please see the attached document that was given to me by Red Hat to get= =3D more information on this. Hope this information helps you.

--

Devin Acosta, RHCA, RHVCA

Red Hat Certified Architect

On August 6, 2017 at 7:29:29 PM, Moacir Ferreira = (<=3D a href=3D3D"mailto:moacirferreira(a)hotmail.com">moacirferreira(a)hotmail.c= om=3D ) wrote:

I am willing to assemble a oVirt "pod", made of 3 server= =3D s, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The id= =3D ea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual= =3D 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop like a server triangle using the 40Gb NICs for virtual= =3D ization files (VMs .qcow2) access and to move VMs around the pod (east /wes= =3D t traffic) while using the 10Gb interfaces for giving services to the outsi= =3D de world (north/south traffic).

This said, my first question is: How should I deploy GlusterFS in such o= =3D Virt scenario? My questions are:

1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and t= =3D hen create a GlusterFS using them?

2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not c= =3D onsuming too much disk space?

4 - Does a oVirt hypervisor pod like I am planning to build, and the vir= =3D tualization environment, benefits from tiering when using a SSD disk? And y= =3D es, will Gluster do it by default or I have to configure it to do so?

At the bottom line, what is the good practice for using GlusterFS in sma= =3D ll pods for enterprises?

You opinion/feedback will be really appreciated!

Moacir

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ov= ir=3D t.org/mailman/listinfo/users

--_000_VI1P190MB0285197D4F878904201462F9C8B50VI1P190MB0285EURP_-- --===============2702624409860843717== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" LS1fMDAwX1ZJMVAxOTBNQjAyODUxOTdENEY4Nzg5MDQyMDE0NjJGOUM4QjUwVkkxUDE5ME1CMDI4 NUVVUlBfCkNvbnRlbnQtVHlwZTogdGV4dC9wbGFpbjsgY2hhcnNldD0iV2luZG93cy0xMjUyIgpD b250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiBxdW90ZWQtcHJpbnRhYmxlCgpEZXZpbiwKCgpNYW55 LCBtYW55IHRoYWtzIGZvciB5b3VyIHJlc3BvbnNlLiBJIHdpbGwgcmVhZCB0aGUgZG9jIHlvdSBz ZW50IGFuZCBpZiBJIHM9CnRpbGwgaGF2ZSBxdWVzdGlvbnMgSSB3aWxsIHBvc3QgdGhlbSBoZXJl LgoKCkJ1dCB3aHkgd291bGQgSSB1c2UgYSBSQUlEZWQgYnJpY2sgaWYgR2x1c3RlciwgYnkgaXRz ZWxmLCBhbHJlYWR5ICJwcm90ZWN0cz0KIiB0aGUgZGF0YSBieSBtYWtpbmcgcmVwbGljYXMuIFlv dSBzZWUsIHRoYXQgaXMgd2hhdCBpcyBjb25mdXNpbmcgdG8gbWUuLi4KCgpUaGFua3MsCgpNb2Fj aXIKCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpGcm9tOiBEZXZpbiBBY29zdGEg PGRldmluQHBhYnN0YXRlbmNpby5jb20+ClNlbnQ6IE1vbmRheSwgQXVndXN0IDcsIDIwMTcgNzo0 NiBBTQpUbzogTW9hY2lyIEZlcnJlaXJhOyB1c2Vyc0BvdmlydC5vcmcKU3ViamVjdDogUmU6IFtv dmlydC11c2Vyc10gR29vZCBwcmFjdGljZXMKCgpNb2FjaXIsCgpJIGhhdmUgcmVjZW50bHkgaW5z dGFsbGVkIG11bHRpcGxlIFJlZCBIYXQgVmlydHVhbGl6YXRpb24gaG9zdHMgZm9yIHNldmVyYWw9 CiBkaWZmZXJlbnQgY29tcGFuaWVzLCBhbmQgaGF2ZSBkZWFsdCB3aXRoIHRoZSBSZWQgSGF0IFN1 cHBvcnQgVGVhbSBpbiBkZXB0aD0KIGFib3V0IG9wdGltYWwgY29uZmlndXJhdGlvbiBpbiByZWdh cmRzIHRvIHNldHRpbmcgdXAgR2x1c3RlckZTIG1vc3QgZWZmaWNpPQplbnRseSBhbmQgSSB3YW50 ZWQgdG8gc2hhcmUgd2l0aCB5b3Ugd2hhdCBJIGxlYXJuZWQuCgpJbiBnZW5lcmFsIFJlZCBIYXQg VmlydHVhbGl6YXRpb24gdGVhbSBmcm93bnMgdXBvbiB1c2luZyBlYWNoIERJU0sgb2YgdGhlIHM9 CnlzdGVtIGFzIGp1c3QgYSBKQk9ELCBzdXJlIHRoZXJlIGlzIHNvbWUgcHJvdGVjdGlvbiBieSBo YXZpbmcgdGhlIGRhdGEgcmVwbD0KaWNhdGVkLCBob3dldmVyLCB0aGUgcmVjb21tZW5kYXRpb24g aXMgdG8gdXNlIFJBSUQgNiAocHJlZmVycmVkKSBvciBSQUlELTUsPQogb3IgYXQgbGVhc3QgUkFJ RC0xIGF0IHRoZSB2ZXJ5IGxlYXN0LgoKSGVyZSBpcyB0aGUgZGlyZWN0IHF1b3RlIGZyb20gUmVk IEhhdCB3aGVuIEkgYXNrZWQgYWJvdXQgUkFJRCBhbmQgQnJpY2tzOgoKIkEgdHlwaWNhbCBHbHVz dGVyIGNvbmZpZ3VyYXRpb24gd291bGQgdXNlIFJBSUQgdW5kZXJuZWF0aCB0aGUgYnJpY2tzLiBS QUlEPQogNiBpcyBtb3N0IHR5cGljYWwgYXMgaXQgZ2l2ZXMgeW91IDIgZGlzayBmYWlsdXJlIHBy b3RlY3Rpb24sIGJ1dCBSQUlEIDUgY289CnVsZCBiZSB1c2VkIHRvby4gT25jZSB5b3UgaGF2ZSB0 aGUgUkFJRGVkIGJyaWNrcywgeW91J2QgdGhlbiBhcHBseSB0aGUgZGVzaT0KcmVkIHJlcGxpY2F0 aW9uIG9uIHRvcCBvZiB0aGF0LiBUaGUgbW9zdCBwb3B1bGFyIHdheSBvZiBkb2luZyB0aGlzIHdv dWxkIGJlPQogZGlzdHJpYnV0ZWQgcmVwbGljYXRlZCB3aXRoIDJ4IHJlcGxpY2F0aW9uLiBJbiBn ZW5lcmFsIHlvdSdsbCBnZXQgYmV0dGVyIHA9CmVyZm9ybWFuY2Ugd2l0aCBsYXJnZXIgYnJpY2tz LiAxMiBkcml2ZXMgaXMgb2Z0ZW4gYSBzd2VldCBzcG90LiBBbm90aGVyIG9wdD0KaW9uIHdvdWxk IGJlIHRvIGNyZWF0ZSBhIHNlcGFyYXRlIHRpZXIgdXNpbmcgYWxsIFNTRD05MnMuPTk0CgpJbiBv cmRlciB0byBTU0QgdGllcmluZyBmcm9tIG15IHVuZGVyc3RhbmRpbmcgeW91IHdvdWxkIG5lZWQg MSB4IE5WTWUgZHJpdmU9CiBpbiBlYWNoIHNlcnZlciwgb3IgNCB4IFNTRCBob3QgdGllciAoaXQg bmVlZHMgdG8gYmUgZGlzdHJpYnV0ZWQsIHJlcGxpY2F0ZT0KZCBmb3IgdGhlIGhvdCB0aWVyIGlm IG5vdCB1c2luZyBOVk1FKS4gU28gd2l0aCB5b3Ugb25seSBoYXZpbmcgMSBTU0QgZHJpdmUgPQpp biBlYWNoIHNlcnZlciwgST05MmQgc3VnZ2VzdCBtYXliZSBsb29raW5nIGludG8gdGhlIE5WTUUg b3B0aW9uLgoKU2luY2UgeW91ciB1c2luZyBvbmx5IDMtc2VydmVycywgd2hhdCBJPTkyZCBwcm9i YWJseSBzdWdnZXN0IGlzIHRvIGRvICgyIFJlPQpwbGljYXMgKyBBcmJpdGVyIE5vZGUpLCB0aGlz IHNldHVwIGFjdHVhbGx5IGRvZXNuPTkydCByZXF1aXJlIHRoZSAzcmQgc2VydmU9CnIgdG8gaGF2 ZSBiaWcgZHJpdmVzIGF0IGFsbCBhcyBpdCBvbmx5IHN0b3JlcyBtZXRhLWRhdGEgYWJvdXQgdGhl IGZpbGVzIGFuZD0KIG5vdCBhY3R1YWxseSBhIGZ1bGwgY29weS4KClBsZWFzZSBzZWUgdGhlIGF0 dGFjaGVkIGRvY3VtZW50IHRoYXQgd2FzIGdpdmVuIHRvIG1lIGJ5IFJlZCBIYXQgdG8gZ2V0IG1v cj0KZSBpbmZvcm1hdGlvbiBvbiB0aGlzLiBIb3BlIHRoaXMgaW5mb3JtYXRpb24gaGVscHMgeW91 LgoKCi0tCgpEZXZpbiBBY29zdGEsIFJIQ0EsIFJIVkNBClJlZCBIYXQgQ2VydGlmaWVkIEFyY2hp dGVjdAoKCk9uIEF1Z3VzdCA2LCAyMDE3IGF0IDc6Mjk6MjkgUE0sIE1vYWNpciBGZXJyZWlyYSAo bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jbz0KbTxtYWlsdG86bW9hY2lyZmVycmVpcmFAaG90bWFp bC5jb20+KSB3cm90ZToKCkkgYW0gd2lsbGluZyB0byBhc3NlbWJsZSBhIG9WaXJ0ICJwb2QiLCBt YWRlIG9mIDMgc2VydmVycywgZWFjaCB3aXRoIDIgQ1BVID0Kc29ja2V0cyBvZiAxMiBjb3Jlcywg MjU2R0IgUkFNLCA3IEhERCAxMEssIDEgU1NELiBUaGUgaWRlYSBpcyB0byB1c2UgR2x1c3RlPQpy RlMgdG8gcHJvdmlkZSBIQSBmb3IgdGhlIFZNcy4gVGhlIDMgc2VydmVycyBoYXZlIGEgZHVhbCA0 MEdiIE5JQyBhbmQgYSBkdWE9CmwgMTBHYiBOSUMuIFNvIG15IGludGVudGlvbiBpcyB0byBjcmVh dGUgYSBsb29wIGxpa2UgYSBzZXJ2ZXIgdHJpYW5nbGUgdXNpbj0KZyB0aGUgNDBHYiBOSUNzIGZv ciB2aXJ0dWFsaXphdGlvbiBmaWxlcyAoVk1zIC5xY293MikgYWNjZXNzIGFuZCB0byBtb3ZlIFZN PQpzIGFyb3VuZCB0aGUgcG9kIChlYXN0IC93ZXN0IHRyYWZmaWMpIHdoaWxlIHVzaW5nIHRoZSAx MEdiIGludGVyZmFjZXMgZm9yIGc9Cml2aW5nIHNlcnZpY2VzIHRvIHRoZSBvdXRzaWRlIHdvcmxk IChub3J0aC9zb3V0aCB0cmFmZmljKS4KCgpUaGlzIHNhaWQsIG15IGZpcnN0IHF1ZXN0aW9uIGlz OiBIb3cgc2hvdWxkIEkgZGVwbG95IEdsdXN0ZXJGUyBpbiBzdWNoIG9WaXI9CnQgc2NlbmFyaW8/ IE15IHF1ZXN0aW9ucyBhcmU6CgoKMSAtIFNob3VsZCBJIGNyZWF0ZSAzIFJBSUQgKGkuZS46IFJB SUQgNSksIG9uZSBvbiBlYWNoIG9WaXJ0IG5vZGUsIGFuZCB0aGVuPQogY3JlYXRlIGEgR2x1c3Rl ckZTIHVzaW5nIHRoZW0/CgoyIC0gSW5zdGVhZCwgc2hvdWxkIEkgY3JlYXRlIGEgSkJPRCBhcnJh eSBtYWRlIG9mIGFsbCBzZXJ2ZXIncyBkaXNrcz8KCjMgLSBXaGF0IGlzIHRoZSBiZXN0IEdsdXN0 ZXIgY29uZmlndXJhdGlvbiB0byBwcm92aWRlIGZvciBIQSB3aGlsZSBub3QgY29ucz0KdW1pbmcg dG9vIG11Y2ggZGlzayBzcGFjZT8KCjQgLSBEb2VzIGEgb1ZpcnQgaHlwZXJ2aXNvciBwb2QgbGlr ZSBJIGFtIHBsYW5uaW5nIHRvIGJ1aWxkLCBhbmQgdGhlIHZpcnR1YT0KbGl6YXRpb24gZW52aXJv bm1lbnQsIGJlbmVmaXRzIGZyb20gdGllcmluZyB3aGVuIHVzaW5nIGEgU1NEIGRpc2s/IEFuZCB5 ZXMsPQogd2lsbCBHbHVzdGVyIGRvIGl0IGJ5IGRlZmF1bHQgb3IgSSBoYXZlIHRvIGNvbmZpZ3Vy ZSBpdCB0byBkbyBzbz8KCgpBdCB0aGUgYm90dG9tIGxpbmUsIHdoYXQgaXMgdGhlIGdvb2QgcHJh Y3RpY2UgZm9yIHVzaW5nIEdsdXN0ZXJGUyBpbiBzbWFsbCA9CnBvZHMgZm9yIGVudGVycHJpc2Vz PwoKCllvdSBvcGluaW9uL2ZlZWRiYWNrIHdpbGwgYmUgcmVhbGx5IGFwcHJlY2lhdGVkIQoKTW9h Y2lyCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpVc2Vy cyBtYWlsaW5nIGxpc3QKVXNlcnNAb3ZpcnQub3JnPG1haWx0bzpVc2Vyc0BvdmlydC5vcmc+Cmh0 dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycwoKLS1fMDAwX1ZJMVAx OTBNQjAyODUxOTdENEY4Nzg5MDQyMDE0NjJGOUM4QjUwVkkxUDE5ME1CMDI4NUVVUlBfCkNvbnRl bnQtVHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PSJXaW5kb3dzLTEyNTIiCkNvbnRlbnQtVHJhbnNm ZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKCjxodG1sPgo8aGVhZD4KPG1ldGEgaHR0cC1l cXVpdj0zRCJDb250ZW50LVR5cGUiIGNvbnRlbnQ9M0QidGV4dC9odG1sOyBjaGFyc2V0PTNEV2lu ZG93cy0xPQoyNTIiPgo8c3R5bGUgdHlwZT0zRCJ0ZXh0L2NzcyIgc3R5bGU9M0QiZGlzcGxheTpu b25lOyI+PCEtLSBQIHttYXJnaW4tdG9wOjA7bWFyZ2k9Cm4tYm90dG9tOjA7fSAtLT48L3N0eWxl Pgo8L2hlYWQ+Cjxib2R5IGRpcj0zRCJsdHIiPgo8ZGl2IGlkPTNEImRpdnRhZ2RlZmF1bHR3cmFw cGVyIiBzdHlsZT0zRCJmb250LXNpemU6MTJwdDtjb2xvcjojMDAwMDAwO2ZvbnQ9Ci1mYW1pbHk6 Q2FsaWJyaSxIZWx2ZXRpY2Esc2Fucy1zZXJpZjsiIGRpcj0zRCJsdHIiPgo8cD5EZXZpbiw8L3A+ CjxwPjxicj4KPC9wPgo8cD5NYW55LCBtYW55IHRoYWtzIGZvciB5b3VyIHJlc3BvbnNlLiBJIHdp bGwgcmVhZCB0aGUgZG9jIHlvdSBzZW50IGFuZCBpZiA9Ckkgc3RpbGwgaGF2ZSBxdWVzdGlvbnMg SSB3aWxsIHBvc3QgdGhlbSBoZXJlLjwvcD4KPHA+PGJyPgo8L3A+CjxwPkJ1dCB3aHkgd291bGQg SSB1c2UgYSBSQUlEZWQgYnJpY2sgaWYgR2x1c3RlciwgYnkgaXRzZWxmLCBhbHJlYWR5ICZxdW90 Oz0KcHJvdGVjdHMmcXVvdDsgdGhlIGRhdGEgYnkgbWFraW5nIHJlcGxpY2FzLiBZb3Ugc2VlLCB0 aGF0IGlzIHdoYXQgaXMgY29uZnVzPQppbmcgdG8gbWUuLi4KPGJyPgo8L3A+Cjxicj4KPHA+VGhh bmtzLDwvcD4KPHA+TW9hY2lyPGJyPgo8L3A+Cjxicj4KPGJyPgo8ZGl2IHN0eWxlPTNEImNvbG9y OiByZ2IoNDksIDU1LCA1Nyk7Ij4KPGhyIHRhYmluZGV4PTNEIi0xIiBzdHlsZT0zRCJkaXNwbGF5 OmlubGluZS1ibG9jazsgd2lkdGg6OTglIj4KPGRpdiBpZD0zRCJkaXZScGx5RndkTXNnIiBkaXI9 M0QibHRyIj48Zm9udCBzdHlsZT0zRCJmb250LXNpemU6MTFwdCIgZmFjZT0KPTNEIkNhbGlicmks IHNhbnMtc2VyaWYiIGNvbG9yPTNEIiMwMDAwMDAiPjxiPkZyb206PC9iPiBEZXZpbiBBY29zdGEg Jmx0O2RlPQp2aW5AcGFic3RhdGVuY2lvLmNvbSZndDs8YnI+CjxiPlNlbnQ6PC9iPiBNb25kYXks IEF1Z3VzdCA3LCAyMDE3IDc6NDYgQU08YnI+CjxiPlRvOjwvYj4gTW9hY2lyIEZlcnJlaXJhOyB1 c2Vyc0BvdmlydC5vcmc8YnI+CjxiPlN1YmplY3Q6PC9iPiBSZTogW292aXJ0LXVzZXJzXSBHb29k IHByYWN0aWNlczwvZm9udD4KPGRpdj4mbmJzcDs8L2Rpdj4KPC9kaXY+CjxkaXY+CjxkaXYgaWQ9 M0QiYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9M0QiY29sb3I6cmdiKDAsMCwwKTsgbWFyZ2luOjBw eCI+PGZvbnQgZj0KYWNlPTNEIklucHV0IE1vbm8iPjxicj4KPC9mb250PjwvZGl2Pgo8ZGl2IGlk PTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEImNvbG9yOnJnYigwLDAsMCk7IG1hcmdpbjow cHgiPjxmb250IGY9CmFjZT0zRCJJbnB1dCBNb25vIj5Nb2FjaXIsPC9mb250PjwvZGl2Pgo8ZGl2 IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEImNvbG9yOnJnYigwLDAsMCk7IG1hcmdp bjowcHgiPjxmb250IGY9CmFjZT0zRCJJbnB1dCBNb25vIj48YnI+CjwvZm9udD48L2Rpdj4KPGRp diBpZD0zRCJibG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJjb2xvcjpyZ2IoMCwwLDApOyBtYXJn aW46MHB4Ij48Zm9udCBmPQphY2U9M0QiSW5wdXQgTW9ubyI+SSBoYXZlIHJlY2VudGx5IGluc3Rh bGxlZCBtdWx0aXBsZSBSZWQgSGF0IFZpcnR1YWxpemF0aW89Cm4gaG9zdHMgZm9yIHNldmVyYWwg ZGlmZmVyZW50IGNvbXBhbmllcywgYW5kIGhhdmUgZGVhbHQgd2l0aCB0aGUgUmVkIEhhdCBTdT0K cHBvcnQgVGVhbSBpbiBkZXB0aCBhYm91dCBvcHRpbWFsIGNvbmZpZ3VyYXRpb24KIGluIHJlZ2Fy ZHMgdG8gc2V0dGluZyB1cCBHbHVzdGVyRlMgbW9zdCBlZmZpY2llbnRseSBhbmQgSSB3YW50ZWQg dG8gc2hhcmUgPQp3aXRoIHlvdSB3aGF0IEkgbGVhcm5lZC48L2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9 M0QiYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9M0QiY29sb3I6cmdiKDAsMCwwKTsgbWFyZ2luOjBw eCI+PGZvbnQgZj0KYWNlPTNEIklucHV0IE1vbm8iPjxicj4KPC9mb250PjwvZGl2Pgo8ZGl2IGlk PTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEImNvbG9yOnJnYigwLDAsMCk7IG1hcmdpbjow cHgiPjxmb250IGY9CmFjZT0zRCJJbnB1dCBNb25vIj5JbiBnZW5lcmFsIFJlZCBIYXQgVmlydHVh bGl6YXRpb24gdGVhbSBmcm93bnMgdXBvbiB1c2luZz0KIGVhY2ggRElTSyBvZiB0aGUgc3lzdGVt IGFzIGp1c3QgYSBKQk9ELCBzdXJlIHRoZXJlIGlzIHNvbWUgcHJvdGVjdGlvbiBieSBoPQphdmlu ZyB0aGUgZGF0YSByZXBsaWNhdGVkLCBob3dldmVyLCB0aGUKIHJlY29tbWVuZGF0aW9uIGlzIHRv IHVzZSBSQUlEIDYgKHByZWZlcnJlZCkgb3IgUkFJRC01LCBvciBhdCBsZWFzdCBSQUlELTEgPQph dCB0aGUgdmVyeSBsZWFzdC48L2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9tZm9u dCIgc3R5bGU9M0QiY29sb3I6cmdiKDAsMCwwKTsgbWFyZ2luOjBweCI+PGZvbnQgZj0KYWNlPTNE IklucHV0IE1vbm8iPjxicj4KPC9mb250PjwvZGl2Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZv bnQiIHN0eWxlPTNEIm1hcmdpbjowcHgiPjxmb250IGZhY2U9M0QiSW5wdXQgTW9ubyI9Cj5IZXJl IGlzIHRoZSZuYnNwO2RpcmVjdCBxdW90ZSBmcm9tIFJlZCBIYXQgd2hlbiBJIGFza2VkIGFib3V0 IFJBSUQgYW5kIEJyaT0KY2tzOjwvZm9udD48L2Rpdj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21m b250IiBzdHlsZT0zRCJtYXJnaW46MHB4Ij48Zm9udCBmYWNlPTNEIklucHV0IE1vbm8iPQo+PGk+ PGJyPgo8L2k+PC9mb250PjwvZGl2Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxl PTNEIm1hcmdpbjowcHgiPjxmb250IGZhY2U9M0QiSW5wdXQgTW9ubyI9Cj48aT4mcXVvdDtBIHR5 cGljYWwgR2x1c3RlciBjb25maWd1cmF0aW9uIHdvdWxkIHVzZSBSQUlEIHVuZGVybmVhdGggdGhl IGJyaT0KY2tzLiBSQUlEIDYgaXMgbW9zdCB0eXBpY2FsIGFzIGl0IGdpdmVzIHlvdSAyIGRpc2sg ZmFpbHVyZSBwcm90ZWN0aW9uLCBidXQgPQpSQUlEIDUgY291bGQgYmUgdXNlZCB0b28uJm5ic3A7 T25jZSB5b3UgaGF2ZSB0aGUKIFJBSURlZCBicmlja3MsIHlvdSdkIHRoZW4gYXBwbHkgdGhlIGRl c2lyZWQgcmVwbGljYXRpb24gb24gdG9wIG9mIHRoYXQuIFRoPQplIG1vc3QgcG9wdWxhciB3YXkg b2YgZG9pbmcgdGhpcyB3b3VsZCBiZSBkaXN0cmlidXRlZCByZXBsaWNhdGVkIHdpdGggMnggcmU9 CnBsaWNhdGlvbi4mbmJzcDtJbiBnZW5lcmFsIHlvdSdsbCBnZXQgYmV0dGVyIHBlcmZvcm1hbmNl IHdpdGggbGFyZ2VyIGJyaWNrcz0KLiAxMiBkcml2ZXMgaXMgb2Z0ZW4gYSBzd2VldCBzcG90LiBB bm90aGVyCiBvcHRpb24gd291bGQgYmUgdG8gY3JlYXRlIGEgc2VwYXJhdGUgdGllciB1c2luZyBh bGwgU1NEPTkycy49OTQmbmJzcDs8L2k+PD0KL2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3Bf Y3VzdG9tZm9udCIgc3R5bGU9M0QibWFyZ2luOjBweCI+PGJyPgo8L2Rpdj4KPGRpdiBpZD0zRCJi bG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJtYXJnaW46MHB4Ij48Zm9udCBmYWNlPTNEIklucHV0 IE1vbm8iPQo+PGk+SW4gb3JkZXIgdG8gU1NEJm5ic3A7dGllcmluZyBmcm9tIG15IHVuZGVyc3Rh bmRpbmcgeW91IHdvdWxkIG5lZWQgMSB4IE49ClZNZSBkcml2ZSBpbiBlYWNoIHNlcnZlciwgb3Ig NCB4IFNTRCBob3QgdGllciAoaXQgbmVlZHMgdG8gYmUgZGlzdHJpYnV0ZWQsID0KcmVwbGljYXRl ZCBmb3IgdGhlIGhvdCB0aWVyIGlmIG5vdCB1c2luZyBOVk1FKS4KIFNvIHdpdGggeW91IG9ubHkg aGF2aW5nIDEgU1NEIGRyaXZlIGluIGVhY2ggc2VydmVyLCZuYnNwO0k9OTJkIHN1Z2dlc3QgbWF5 PQpiZSBsb29raW5nIGludG8gdGhlIE5WTUUgb3B0aW9uLiZuYnNwOzwvaT48L2ZvbnQ+PC9kaXY+ CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9M0QibWFyZ2luOjBweCI+PGZvbnQg ZmFjZT0zRCJJbnB1dCBNb25vIj0KPjxpPjxicj4KPC9pPjwvZm9udD48L2Rpdj4KPGRpdiBpZD0z RCJibG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJtYXJnaW46MHB4Ij48Zm9udCBmYWNlPTNEIklu cHV0IE1vbm8iPQo+PGk+U2luY2UgeW91ciB1c2luZyBvbmx5IDMtc2VydmVycywgd2hhdCBJPTky ZCBwcm9iYWJseSBzdWdnZXN0IGlzIHRvIGRvICg9CjIgUmVwbGljYXMgJiM0MzsgQXJiaXRlciBO b2RlKSwgdGhpcyBzZXR1cCBhY3R1YWxseSBkb2Vzbj05MnQgcmVxdWlyZSB0aGUgMz0KcmQgc2Vy dmVyIHRvIGhhdmUgYmlnIGRyaXZlcyBhdCBhbGwgYXMgaXQgb25seQogc3RvcmVzIG1ldGEtZGF0 YSBhYm91dCB0aGUgZmlsZXMgYW5kIG5vdCBhY3R1YWxseSBhIGZ1bGwgY29weS4mbmJzcDs8L2k+ PC89CmZvbnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9M0QibWFy Z2luOjBweCI+PGZvbnQgZmFjZT0zRCJJbnB1dCBNb25vIj0KPjxpPjxicj4KPC9pPjwvZm9udD48 L2Rpdj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJtYXJnaW46MHB4Ij48 Zm9udCBmYWNlPTNEIklucHV0IE1vbm8iPQo+PGk+UGxlYXNlIHNlZSB0aGUgYXR0YWNoZWQgZG9j dW1lbnQgdGhhdCB3YXMgZ2l2ZW4gdG8gbWUgYnkgUmVkIEhhdCB0byBnZXQ9CiBtb3JlIGluZm9y bWF0aW9uIG9uIHRoaXMuIEhvcGUgdGhpcyBpbmZvcm1hdGlvbiBoZWxwcyB5b3UuPC9pPjwvZm9u dD48L2Rpdj0KPgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEIm1hcmdpbjow cHgiPjxmb250IGZhY2U9M0QiSW5wdXQgTW9ubyI9Cj48aT48YnI+CjwvaT48L2ZvbnQ+PC9kaXY+ Cjxicj4KPGRpdiBpZD0zRCJibG9vcF9zaWduXzE1MDIwODczNzY3MjU0NjkxODQiIGNsYXNzPTNE ImJsb29wX3NpZ24iPjxzcGFuIHN0eWxlPQo9M0QiZm9udC1mYW1pbHk6J2hlbHZldGljYSBOZXVl JyxoZWx2ZXRpY2E7IGZvbnQtc2l6ZToxNHB4Ij4tLTwvc3Bhbj48YnIgc3Q9CnlsZT0zRCJmb250 LWZhbWlseTonaGVsdmV0aWNhIE5ldWUnLGhlbHZldGljYTsgZm9udC1zaXplOjE0cHgiPgo8ZGl2 IGNsYXNzPTNEImdtYWlsX3NpZ25hdHVyZSIgc3R5bGU9M0QiZm9udC1mYW1pbHk6J2hlbHZldGlj YSBOZXVlJyxoZWx2ZXQ9CmljYTsgZm9udC1zaXplOjE0cHgiPgo8ZGl2IGRpcj0zRCJsdHIiPgo8 ZGl2Pjxicj4KPC9kaXY+CjxkaXY+RGV2aW4gQWNvc3RhLCBSSENBLCBSSFZDQTwvZGl2Pgo8ZGl2 PlJlZCBIYXQgQ2VydGlmaWVkIEFyY2hpdGVjdDwvZGl2Pgo8ZGl2PjwvZGl2Pgo8L2Rpdj4KPC9k aXY+CjwvZGl2Pgo8YnI+CjxwIGNsYXNzPTNEImFpcm1haWxfb24iPk9uIEF1Z3VzdCA2LCAyMDE3 IGF0IDc6Mjk6MjkgUE0sIE1vYWNpciBGZXJyZWlyYSAoPD0KYSBocmVmPTNEIm1haWx0bzptb2Fj aXJmZXJyZWlyYUBob3RtYWlsLmNvbSI+bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb208L2E+PQop IHdyb3RlOjwvcD4KPGJsb2NrcXVvdGUgdHlwZT0zRCJjaXRlIiBjbGFzcz0zRCJjbGVhbl9icSI+ PHNwYW4+CjxkaXYgZGlyPTNEImx0ciI+CjxkaXY+PC9kaXY+CjxkaXY+CjxkaXYgaWQ9M0QiZGl2 dGFnZGVmYXVsdHdyYXBwZXIiIGRpcj0zRCJsdHIiIHN0eWxlPTNEImZvbnQtc2l6ZToxMnB0OyBj b2xvcj0KOiMwMDAwMDA7IGZvbnQtZmFtaWx5OkNhbGlicmksSGVsdmV0aWNhLHNhbnMtc2VyaWYi Pgo8cD48c3Bhbj5JIGFtIHdpbGxpbmcgdG8gYXNzZW1ibGUgYSBvVmlydCAmcXVvdDtwb2QmcXVv dDssIG1hZGUgb2YgMyBzZXJ2ZXI9CnMsIGVhY2ggd2l0aCAyIENQVSBzb2NrZXRzIG9mIDEyIGNv cmVzLCAyNTZHQiBSQU0sIDcgSEREIDEwSywgMSBTU0QuIFRoZSBpZD0KZWEgaXMgdG8gdXNlIEds dXN0ZXJGUyB0byBwcm92aWRlIEhBIGZvciB0aGUgVk1zLiBUaGUgMyBzZXJ2ZXJzIGhhdmUgYSBk dWFsPQogNDBHYiBOSUMgYW5kIGEgZHVhbCAxMEdiIE5JQy4gU28gbXkgaW50ZW50aW9uCiBpcyB0 byBjcmVhdGUgYSBsb29wIGxpa2UgYSBzZXJ2ZXIgdHJpYW5nbGUgdXNpbmcgdGhlIDQwR2IgTklD cyBmb3IgdmlydHVhbD0KaXphdGlvbiBmaWxlcyAoVk1zIC5xY293MikgYWNjZXNzIGFuZCB0byBt b3ZlIFZNcyBhcm91bmQgdGhlIHBvZCAoZWFzdCAvd2VzPQp0IHRyYWZmaWMpIHdoaWxlIHVzaW5n IHRoZSAxMEdiIGludGVyZmFjZXMgZm9yIGdpdmluZyBzZXJ2aWNlcyB0byB0aGUgb3V0c2k9CmRl IHdvcmxkIChub3J0aC9zb3V0aCB0cmFmZmljKS48L3NwYW4+PC9wPgo8cD48YnI+CjwvcD4KPHA+ VGhpcyBzYWlkLCBteSBmaXJzdCBxdWVzdGlvbiBpczogSG93IHNob3VsZCBJIGRlcGxveSBHbHVz dGVyRlMgaW4gc3VjaCBvPQpWaXJ0IHNjZW5hcmlvPyBNeSBxdWVzdGlvbnMgYXJlOjwvcD4KPHA+ PGJyPgo8L3A+CjxwPjEgLSBTaG91bGQgSSBjcmVhdGUgMyBSQUlEIChpLmUuOiBSQUlEIDUpLCBv bmUgb24gZWFjaCBvVmlydCBub2RlLCBhbmQgdD0KaGVuIGNyZWF0ZSBhIEdsdXN0ZXJGUyB1c2lu ZyB0aGVtPzwvcD4KPHA+MiAtIEluc3RlYWQsIHNob3VsZCBJIGNyZWF0ZSBhIEpCT0QgYXJyYXkg bWFkZSBvZiBhbGwgc2VydmVyJ3MgZGlza3M/PC9wPQo+CjxwPjMgLSBXaGF0IGlzIHRoZSBiZXN0 IEdsdXN0ZXIgY29uZmlndXJhdGlvbiB0byBwcm92aWRlIGZvciBIQSB3aGlsZSBub3QgYz0Kb25z dW1pbmcgdG9vIG11Y2ggZGlzayBzcGFjZT88YnI+CjwvcD4KPHA+NCAtIERvZXMgYSBvVmlydCBo eXBlcnZpc29yIHBvZCBsaWtlIEkgYW0gcGxhbm5pbmcgdG8gYnVpbGQsIGFuZCB0aGUgdmlyPQp0 dWFsaXphdGlvbiBlbnZpcm9ubWVudCwgYmVuZWZpdHMgZnJvbSB0aWVyaW5nIHdoZW4gdXNpbmcg YSBTU0QgZGlzaz8gQW5kIHk9CmVzLCB3aWxsIEdsdXN0ZXIgZG8gaXQgYnkgZGVmYXVsdCBvciBJ IGhhdmUgdG8gY29uZmlndXJlIGl0IHRvIGRvIHNvPzwvcD4KPHA+PGJyPgo8L3A+CjxwPkF0IHRo ZSBib3R0b20gbGluZSwgd2hhdCBpcyB0aGUgZ29vZCBwcmFjdGljZSBmb3IgdXNpbmcgR2x1c3Rl ckZTIGluIHNtYT0KbGwgcG9kcyBmb3IgZW50ZXJwcmlzZXM/PGJyPgo8L3A+CjxwPjxicj4KPC9w Pgo8cD5Zb3Ugb3Bpbmlvbi9mZWVkYmFjayB3aWxsIGJlIHJlYWxseSBhcHByZWNpYXRlZCE8L3A+ CjxwPk1vYWNpcjxicj4KPC9wPgo8L2Rpdj4KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX18gPGJyPgpVc2VycyBtYWlsaW5nIGxpc3QgPGJyPgo8YSBocmVmPTNE Im1haWx0bzpVc2Vyc0BvdmlydC5vcmciPlVzZXJzQG92aXJ0Lm9yZzwvYT4gPGJyPgo8YSBocmVm PTNEImh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyI+aHR0cDov L2xpc3RzLm92aXI9CnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnM8L2E+Cjxicj4KPC9kaXY+ CjwvZGl2Pgo8L3NwYW4+PC9ibG9ja3F1b3RlPgo8L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8L2JvZHk+ CjwvaHRtbD4KCi0tXzAwMF9WSTFQMTkwTUIwMjg1MTk3RDRGODc4OTA0MjAxNDYyRjlDOEI1MFZJ MVAxOTBNQjAyODVFVVJQXy0tCg== --===============2702624409860843717==-- From colin.coe at gmail.com Mon Aug 7 11:41:20 2017 Content-Type: multipart/mixed; boundary="===============3901340037170499249==" MIME-Version: 1.0 From: Colin Coe To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Mon, 07 Aug 2017 19:41:19 +0800 Message-ID: In-Reply-To: VI1P190MB02858CCC4D7DCD6A86090CC9C8B50@VI1P190MB0285.EURP190.PROD.OUTLOOK.COM --===============3901340037170499249== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi I just thought that you'd do hardware RAID if you had the controller or JBOD if you didn't. In hindsight, a server with 40Gbps NICs is pretty likely to have a hardware RAID controller. I've never done JBOD with hardware RAID. I think having a single gluster brick on hardware JBOD would be riskier than multiple bricks, each on a single disk, but thats not based on anything other than my prejudices. I thought gluster tiering was for the most frequently accessed files, in which case all the VMs disks would end up in the hot tier. However, I have been wrong before... I just wanted to know where the OS was going as I didn't see it mentioned in the OP. Normally, I'd have the OS on a RAID1 but in your case thats a lot of wasted disk. Honestly, I think Yaniv's answer was far better than my own and made the important point about having an arbiter. Thanks On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreira wrote: > Hi Colin, > > > I am in Portugal, so sorry for this late response. It is quite confusing > for me, please consider: > > > 1* - *What if the RAID is done by the server's disk controller, not by > software? > > 2 - For JBOD I am just using gdeploy to deploy it. However, I am not > using the oVirt node GUI to do this. > > > 3 - As the VM .qcow2 files are quite big, tiering would only help if made > by an intelligent system that uses SSD for chunks of data not for the > entire .qcow2 file. But I guess this is a problem everybody else has. So, > Do you know how tiering works in Gluster? > > > 4 - I am putting the OS on the first disk. However, would you do > differently? > > > Moacir > > ------------------------------ > *From:* Colin Coe > *Sent:* Monday, August 7, 2017 4:48 AM > *To:* Moacir Ferreira > *Cc:* users(a)ovirt.org > *Subject:* Re: [ovirt-users] Good practices > > 1) RAID5 may be a performance hit- > > 2) I'd be inclined to do this as JBOD by creating a distributed disperse > volume on each server. Something like > > echo gluster volume create dispersevol disperse-data 5 redundancy 2 \ > $(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e > "server${SERVER}:/brick/brick-${SERVER}${BRICK}/brick \c"; done; done) > > 3) I think the above. > > 4) Gluster does support tiering, but IIRC you'd need the same number of > SSD as spindle drives. There may be another way to use the SSD as a fast > cache. > > Where are you putting the OS? > > Hope I understood the question... > > Thanks > > On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira < > moacirferreira(a)hotmail.com> wrote: > >> I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 >> CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use >> GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC = and >> a dual 10Gb NIC. So my intention is to create a loop like a server trian= gle >> using the 40Gb NICs for virtualization files (VMs .qcow2) access and to >> move VMs around the pod (east /west traffic) while using the 10Gb >> interfaces for giving services to the outside world (north/south traffic= ). >> >> >> This said, my first question is: How should I deploy GlusterFS in such >> oVirt scenario? My questions are: >> >> >> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and >> then create a GlusterFS using them? >> >> 2 - Instead, should I create a JBOD array made of all server's disks? >> >> 3 - What is the best Gluster configuration to provide for HA while not >> consuming too much disk space? >> >> 4 - Does a oVirt hypervisor pod like I am planning to build, and the >> virtualization environment, benefits from tiering when using a SSD disk? >> And yes, will Gluster do it by default or I have to configure it to do s= o? >> >> >> At the bottom line, what is the good practice for using GlusterFS in >> small pods for enterprises? >> >> >> You opinion/feedback will be really appreciated! >> >> Moacir >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> > --===============3901340037170499249== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+SGk8ZGl2Pjxicj48L2Rpdj48ZGl2PkkganVzdCB0aG91Z2h0IHRoYXQg eW91JiMzOTtkIGRvIGhhcmR3YXJlIFJBSUQgaWYgeW91IGhhZCB0aGUgY29udHJvbGxlciBvciBK Qk9EIGlmIHlvdSBkaWRuJiMzOTt0LsKgIEluIGhpbmRzaWdodCwgYSBzZXJ2ZXIgd2l0aCA0MEdi cHMgTklDcyBpcyBwcmV0dHkgbGlrZWx5IHRvIGhhdmUgYSBoYXJkd2FyZSBSQUlEIGNvbnRyb2xs ZXIuwqAgSSYjMzk7dmUgbmV2ZXIgZG9uZSBKQk9EIHdpdGggaGFyZHdhcmUgUkFJRC7CoCBJIHRo aW5rIGhhdmluZyBhIHNpbmdsZSBnbHVzdGVyIGJyaWNrIG9uIGhhcmR3YXJlIEpCT0Qgd291bGQg YmUgcmlza2llciB0aGFuIG11bHRpcGxlIGJyaWNrcywgZWFjaCBvbiBhIHNpbmdsZSBkaXNrLCBi dXQgdGhhdHMgbm90IGJhc2VkIG9uIGFueXRoaW5nIG90aGVyIHRoYW4gbXkgcHJlanVkaWNlcy48 L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PkkgdGhvdWdodCBnbHVzdGVyIHRpZXJpbmcgd2FzIGZv ciB0aGUgbW9zdCBmcmVxdWVudGx5IGFjY2Vzc2VkIGZpbGVzLCBpbiB3aGljaCBjYXNlIGFsbCB0 aGUgVk1zIGRpc2tzIHdvdWxkIGVuZCB1cCBpbiB0aGUgaG90IHRpZXIuwqAgSG93ZXZlciwgSSBo YXZlIGJlZW4gd3JvbmcgYmVmb3JlLi4uPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5JIGp1c3Qg d2FudGVkIHRvIGtub3cgd2hlcmUgdGhlIE9TIHdhcyBnb2luZyBhcyBJIGRpZG4mIzM5O3Qgc2Vl IGl0IG1lbnRpb25lZCBpbiB0aGUgT1AuwqAgTm9ybWFsbHksIEkmIzM5O2QgaGF2ZSB0aGUgT1Mg b24gYSBSQUlEMSBidXQgaW4geW91ciBjYXNlIHRoYXRzIGEgbG90IG9mIHdhc3RlZCBkaXNrLjwv ZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+SG9uZXN0bHksIEkgdGhpbmsgWWFuaXYmIzM5O3MgYW5z d2VyIHdhcyBmYXIgYmV0dGVyIHRoYW4gbXkgb3duIGFuZCBtYWRlIHRoZSBpbXBvcnRhbnQgcG9p bnQgYWJvdXQgaGF2aW5nIGFuIGFyYml0ZXIuwqA8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PlRo YW5rczwvZGl2PjwvZGl2PjxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+PGRpdiBjbGFzcz0i Z21haWxfcXVvdGUiPk9uIE1vbiwgQXVnIDcsIDIwMTcgYXQgNTo1NiBQTSwgTW9hY2lyIEZlcnJl aXJhIDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgaHJlZj0ibWFpbHRvOm1vYWNpcmZlcnJlaXJhQGhv dG1haWwuY29tIiB0YXJnZXQ9Il9ibGFuayI+bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb208L2E+ Jmd0Ozwvc3Bhbj4gd3JvdGU6PGJyPjxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5 bGU9Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRlci1sZWZ0OjFweCAjY2NjIHNvbGlkO3BhZGRpbmct bGVmdDoxZXgiPgoKCgoKPGRpdiBkaXI9Imx0ciI+CjxkaXYgaWQ9Im1fNDgzMTg4Njg4MzMwNTY3 MjcxOGRpdnRhZ2RlZmF1bHR3cmFwcGVyIiBzdHlsZT0iZm9udC1zaXplOjEycHQ7Y29sb3I6IzAw MDAwMDtmb250LWZhbWlseTpDYWxpYnJpLEhlbHZldGljYSxzYW5zLXNlcmlmIiBkaXI9Imx0ciI+ CjxwPkhpIENvbGluLDwvcD4KPHA+PGJyPgo8L3A+CjxwPjxzcGFuPkkgYW0gaW4gUG9ydHVnYWw8 L3NwYW4+LCBzbyBzb3JyeSBmb3IgdGhpcyBsYXRlIHJlc3BvbnNlLiBJdCBpcyBxdWl0ZSBjb25m dXNpbmcgZm9yIG1lLCBwbGVhc2UgY29uc2lkZXI6PC9wPgo8cD48L3A+CjxkaXY+PGI+PGJyPgo8 L2I+MTxiPiAtIDwvYj5XaGF0IGlmIHRoZSBSQUlEIGlzIGRvbmUgYnkgdGhlIHNlcnZlciYjMzk7 cyBkaXNrIGNvbnRyb2xsZXIsIG5vdCBieSBzb2Z0d2FyZT88L2Rpdj4KPGJyPgo8cD48L3A+Cjxw PjIgLTxiPiA8L2I+Rm9yIEpCT0QgSSBhbSBqdXN0IHVzaW5nIGdkZXBsb3kgdG8gZGVwbG95IGl0 LiBIb3dldmVyLCBJIGFtIG5vdCB1c2luZyB0aGUgb1ZpcnQgbm9kZSBHVUkgdG8gZG8gdGhpcy48 L3A+CjxwPjxicj4KPC9wPgo8cD4zIC08Yj4gPC9iPkFzIHRoZSBWTSAucWNvdzLCoGZpbGVzIGFy ZSBxdWl0ZSBiaWcsIHRpZXJpbmcgd291bGQgb25seSBoZWxwIGlmIG1hZGUgYnkgYW4gaW50ZWxs aWdlbnQgc3lzdGVtIHRoYXQgdXNlcyBTU0QgZm9yIGNodW5rcyBvZiBkYXRhIG5vdCBmb3IgdGhl IGVudGlyZSAucWNvdzIgZmlsZS4gQnV0IEkgZ3Vlc3MgdGhpcyBpcyBhIHByb2JsZW0gZXZlcnli b2R5IGVsc2UgaGFzLiBTbywgRG8geW91IGtub3cgaG93IHRpZXJpbmcgd29ya3MKIGluIEdsdXN0 ZXI/PGJyPgo8L3A+CjxwPjxicj4KPC9wPgo8cD40IC3CoEkgYW0gcHV0dGluZyB0aGUgT1Mgb24g dGhlIGZpcnN0IGRpc2suIEhvd2V2ZXIsIHdvdWxkIHlvdSBkbyBkaWZmZXJlbnRseT88YnI+Cjwv cD4KPHA+PGJyPgo8L3A+Ck1vYWNpcjxicj4KPGJyPgo8ZGl2IHN0eWxlPSJjb2xvcjpyZ2IoNDks NTUsNTcpIj4KPGhyIHN0eWxlPSJkaXNwbGF5OmlubGluZS1ibG9jazt3aWR0aDo5OCUiPgo8ZGl2 IGlkPSJtXzQ4MzE4ODY4ODMzMDU2NzI3MThkaXZScGx5RndkTXNnIiBkaXI9Imx0ciI+PGZvbnQg c3R5bGU9ImZvbnQtc2l6ZToxMXB0IiBmYWNlPSJDYWxpYnJpLCBzYW5zLXNlcmlmIiBjb2xvcj0i IzAwMDAwMCI+PGI+RnJvbTo8L2I+IENvbGluIENvZSAmbHQ7PGEgaHJlZj0ibWFpbHRvOmNvbGlu LmNvZUBnbWFpbC5jb20iIHRhcmdldD0iX2JsYW5rIj5jb2xpbi5jb2VAZ21haWwuY29tPC9hPiZn dDs8YnI+CjxiPlNlbnQ6PC9iPiBNb25kYXksIEF1Z3VzdCA3LCAyMDE3IDQ6NDggQU08YnI+Cjxi PlRvOjwvYj4gTW9hY2lyIEZlcnJlaXJhPGJyPgo8Yj5DYzo8L2I+IDxhIGhyZWY9Im1haWx0bzp1 c2Vyc0BvdmlydC5vcmciIHRhcmdldD0iX2JsYW5rIj51c2Vyc0BvdmlydC5vcmc8L2E+PGJyPgo8 Yj5TdWJqZWN0OjwvYj4gUmU6IFtvdmlydC11c2Vyc10gR29vZCBwcmFjdGljZXM8L2ZvbnQ+Cjxk aXY+wqA8L2Rpdj4KPC9kaXY+CjxkaXY+CjxkaXYgZGlyPSJsdHIiPjEpIFJBSUQ1IG1heSBiZSBh IHBlcmZvcm1hbmNlIGhpdC0gPGJyPgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+PHNwYW4gY2xhc3M9 IiI+MikgSSYjMzk7ZCBiZSBpbmNsaW5lZCB0byBkbyB0aGlzIGFzIEpCT0QgYnkgY3JlYXRpbmcg YSBkaXN0cmlidXRlZCBkaXNwZXJzZSB2b2x1bWUgb24gZWFjaCBzZXJ2ZXIuwqAgU29tZXRoaW5n IGxpa2UKPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PmVjaG8gZ2x1c3RlciB2b2x1bWUgY3JlYXRlIGRp c3BlcnNldm9sIGRpc3BlcnNlLWRhdGEgNSByZWR1bmRhbmN5IDIgXDwvZGl2Pgo8ZGl2PiQoZm9y IFNFUlZFUiBpbiBhIGIgYzsgZG8gZm9yIEJSSUNLIGluICQoc2VxIDEgNSk7IGRvIGVjaG8gLWUg JnF1b3Q7c2VydmVyJHtTRVJWRVJ9Oi9icmljay9icmljay08d2JyPiR7U0VSVkVSfSR7QlJJQ0t9 L2JyaWNrIFxjJnF1b3Q7OyBkb25lOyBkb25lKTwvZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+Cjwvc3Bh bj48ZGl2PjMpIEkgdGhpbmsgdGhlIGFib3ZlLiA8Yj48L2I+PC9kaXY+PHNwYW4gY2xhc3M9IiI+ CjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj40KSBHbHVzdGVyIGRvZXMgc3VwcG9ydCB0aWVyaW5nLCBi dXQgSUlSQyB5b3UmIzM5O2QgbmVlZCB0aGUgc2FtZSBudW1iZXIgb2YgU1NEIGFzIHNwaW5kbGUg ZHJpdmVzLsKgIFRoZXJlIG1heSBiZSBhbm90aGVyIHdheSB0byB1c2UgdGhlIFNTRCBhcyBhIGZh c3QgY2FjaGUuwqA8L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PldoZXJlIGFyZSB5b3UgcHV0 dGluZyB0aGUgT1M/PC9kaXY+CjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj5Ib3BlIEkgdW5kZXJzdG9v ZCB0aGUgcXVlc3Rpb24uLi48L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PlRoYW5rczwvZGl2 Pgo8L3NwYW4+PC9kaXY+CjwvZGl2PjxzcGFuIGNsYXNzPSIiPgo8ZGl2IGNsYXNzPSJnbWFpbF9l eHRyYSI+PGJyPgo8ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+T24gU3VuLCBBdWcgNiwgMjAxNyBh dCAxMDo0OSBQTSwgTW9hY2lyIEZlcnJlaXJhIDxzcGFuIGRpcj0ibHRyIj4KJmx0OzxhIGhyZWY9 Im1haWx0bzptb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbSIgdGFyZ2V0PSJfYmxhbmsiPm1vYWNp cmZlcnJlaXJhQGhvdG1haWwuY29tPC9hPiZndDs8L3NwYW4+IHdyb3RlOjxicj4KPGJsb2NrcXVv dGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAwIC44ZXg7Ym9yZGVyLWxl ZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+CjxkaXYgZGlyPSJsdHIiPgo8ZGl2 IGlkPSJtXzQ4MzE4ODY4ODMzMDU2NzI3MThtXzI0NjA5ODU2OTE3NDY0OTgzMjJkaXZ0YWdkZWZh dWx0d3JhcHBlciIgZGlyPSJsdHIiIHN0eWxlPSJmb250LXNpemU6MTJwdDtjb2xvcjojMDAwMDAw O2ZvbnQtZmFtaWx5OkNhbGlicmksSGVsdmV0aWNhLHNhbnMtc2VyaWYiPgo8cD48c3Bhbj5JIGFt IHdpbGxpbmcgdG8gYXNzZW1ibGUgYSBvVmlydCAmcXVvdDtwb2QmcXVvdDssIG1hZGUgb2YgMyBz ZXJ2ZXJzLCBlYWNoIHdpdGggMiBDUFUgc29ja2V0cyBvZiAxMiBjb3JlcywgMjU2R0IgUkFNLCA3 IEhERCAxMEssIDEgU1NELiBUaGUgaWRlYSBpcyB0byB1c2UgR2x1c3RlckZTIHRvIHByb3ZpZGUg SEEgZm9yIHRoZSBWTXMuIFRoZSAzIHNlcnZlcnMgaGF2ZSBhIGR1YWwgNDBHYiBOSUMgYW5kIGEg ZHVhbCAxMEdiIE5JQy4gU28gbXkgaW50ZW50aW9uCiBpcyB0byBjcmVhdGUgYSBsb29wIGxpa2Ug YSBzZXJ2ZXIgdHJpYW5nbGUgdXNpbmcgdGhlIDQwR2IgTklDcyBmb3IgdmlydHVhbGl6YXRpb24g ZmlsZXMgKFZNcyAucWNvdzIpIGFjY2VzcyBhbmQgdG8gbW92ZSBWTXMgYXJvdW5kIHRoZSBwb2Qg KGVhc3QgL3dlc3QgdHJhZmZpYykgd2hpbGUgdXNpbmcgdGhlIDEwR2IgaW50ZXJmYWNlcyBmb3Ig Z2l2aW5nIHNlcnZpY2VzIHRvIHRoZSBvdXRzaWRlIHdvcmxkIChub3J0aC9zb3V0aCB0cmFmZmlj KS48L3NwYW4+PC9wPgo8cD48YnI+CjxzcGFuPjwvc3Bhbj48L3A+CjxwPlRoaXMgc2FpZCwgbXkg Zmlyc3QgcXVlc3Rpb24gaXM6IEhvdyBzaG91bGQgSSBkZXBsb3kgR2x1c3RlckZTIGluIHN1Y2gg b1ZpcnQgc2NlbmFyaW8/IE15IHF1ZXN0aW9ucyBhcmU6PC9wPgo8cD48YnI+CjwvcD4KPHA+MSAt IFNob3VsZCBJIGNyZWF0ZSAzIFJBSUQgKGkuZS46IFJBSUQgNSksIG9uZSBvbiBlYWNoIG9WaXJ0 IG5vZGUsIGFuZCB0aGVuIGNyZWF0ZSBhIEdsdXN0ZXJGUyB1c2luZyB0aGVtPzwvcD4KPHA+MiAt IEluc3RlYWQsIHNob3VsZCBJIGNyZWF0ZSBhIEpCT0QgYXJyYXkgbWFkZSBvZiBhbGwgc2VydmVy JiMzOTtzIGRpc2tzPzwvcD4KPHA+MyAtIFdoYXQgaXMgdGhlIGJlc3QgR2x1c3RlciBjb25maWd1 cmF0aW9uIHRvIHByb3ZpZGUgZm9yIEhBIHdoaWxlIG5vdCBjb25zdW1pbmcgdG9vIG11Y2ggZGlz ayBzcGFjZT88YnI+CjwvcD4KPHA+NCAtIERvZXMgYSBvVmlydCBoeXBlcnZpc29yIHBvZCBsaWtl IEkgYW0gcGxhbm5pbmcgdG8gYnVpbGQsIGFuZCB0aGUgdmlydHVhbGl6YXRpb24gZW52aXJvbm1l bnQsIGJlbmVmaXRzIGZyb20gdGllcmluZyB3aGVuIHVzaW5nIGEgU1NEIGRpc2s/IEFuZCB5ZXMs IHdpbGwgR2x1c3RlciBkbyBpdCBieSBkZWZhdWx0IG9yIEkgaGF2ZSB0byBjb25maWd1cmUgaXQg dG8gZG8gc28/PC9wPgo8cD48YnI+CjwvcD4KPHA+QXQgdGhlIGJvdHRvbSBsaW5lLCB3aGF0IGlz IHRoZSBnb29kIHByYWN0aWNlIGZvciB1c2luZyBHbHVzdGVyRlMgaW4gc21hbGwgcG9kcyBmb3Ig ZW50ZXJwcmlzZXM/PGJyPgo8L3A+CjxwPjxicj4KPC9wPgo8cD5Zb3Ugb3Bpbmlvbi9mZWVkYmFj ayB3aWxsIGJlIHJlYWxseSBhcHByZWNpYXRlZCE8L3A+CjxzcGFuIGNsYXNzPSJtXzQ4MzE4ODY4 ODMzMDU2NzI3MThIT0VuWmIiPjxmb250IGNvbG9yPSIjODg4ODg4Ij4KPHA+TW9hY2lyPGJyPgo8 L3A+CjwvZm9udD48L3NwYW4+PC9kaXY+CjwvZGl2Pgo8YnI+Cl9fX19fX19fX19fX19fX19fX19f X19fX19fX19fXzx3YnI+X19fX19fX19fX19fX19fX188YnI+ClVzZXJzIG1haWxpbmcgbGlzdDxi cj4KPGEgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgdGFyZ2V0PSJfYmxhbmsiPlVzZXJz QG92aXJ0Lm9yZzwvYT48YnI+CjxhIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1h bi9saXN0aW5mby91c2VycyIgcmVsPSJub3JlZmVycmVyIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDov L2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuPHdicj4vbGlzdGluZm8vdXNlcnM8L2E+PGJyPgo8YnI+ CjwvYmxvY2txdW90ZT4KPC9kaXY+Cjxicj4KPC9kaXY+Cjwvc3Bhbj48L2Rpdj4KPC9kaXY+Cjwv ZGl2Pgo8L2Rpdj4KCjwvYmxvY2txdW90ZT48L2Rpdj48YnI+PC9kaXY+Cg== --===============3901340037170499249==-- From moacirferreira at hotmail.com Mon Aug 7 12:45:33 2017 Content-Type: multipart/mixed; boundary="===============7703353261016013733==" MIME-Version: 1.0 From: Moacir Ferreira To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Mon, 07 Aug 2017 10:13:16 +0000 Message-ID: In-Reply-To: CAJgorsacD72hveexi99Wq-bEqmyYQUctKGU5-C83HDqavvmZoQ@mail.gmail.com --===============7703353261016013733== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --_000_VI1P190MB02856C6CA513853E8EDA2DB2C8B50VI1P190MB0285EURP_ Content-Type: text/plain; charset=3D"iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi, in-line responses. Thanks, Moacir ________________________________ From: Yaniv Kaul Sent: Monday, August 7, 2017 7:42 AM To: Moacir Ferreira Cc: users(a)ovirt.org Subject: Re: [ovirt-users] Good practices On Sun, Aug 6, 2017 at 5:49 PM, Moacir Ferreira > wrote: I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU = =3D sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use Gluste= =3D rFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dua= =3D l 10Gb NIC. So my intention is to create a loop like a server triangle usin= =3D g the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VM= =3D s around the pod (east /west traffic) while using the 10Gb interfaces for g= =3D iving services to the outside world (north/south traffic). Very nice gear. How are you planning the network exactly? Without a switch,= =3D back-to-back? (sounds OK to me, just wanted to ensure this is what the 'du= =3D al' is used for). However, I'm unsure if you have the correct balance betwe= =3D en the interface speeds (40g) and the disks (too many HDDs?). Moacir: The idea is to have a very high performance network for the distrib= =3D uted file system and to prevent bottlenecks when we move one VM from a node= =3D to another. Using 40Gb NICs I can just connect the servers back-to-back. I= =3D n this case I don't need the expensive 40Gb switch, I get very high speed a= =3D nd no contention between north/south traffic with east/west. This said, my first question is: How should I deploy GlusterFS in such oVir= =3D t scenario? My questions are: 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then= =3D create a GlusterFS using them? I would assume RAID 1 for the operating system (you don't want a single poi= =3D nt of failure there?) and the rest JBODs. The SSD will be used for caching,= =3D I reckon? (I personally would add more SSDs instead of HDDs, but it does d= =3D epend on the disk sizes and your space requirements. Moacir: Yes, I agree that I need a RAID-1 for the OS. Now, generic JBOD or = =3D a JBOD assembled using RAID-5 "disks" created by the server's disk controll= =3D er? 2 - Instead, should I create a JBOD array made of all server's disks? 3 - What is the best Gluster configuration to provide for HA while not cons= =3D uming too much disk space? Replica 2 + Arbiter sounds good to me. Moacir: I agree, and that is what I am using. 4 - Does a oVirt hypervisor pod like I am planning to build, and the virtua= =3D lization environment, benefits from tiering when using a SSD disk? And yes,= =3D will Gluster do it by default or I have to configure it to do so? Yes, I believe using lvmcache is the best way to go. Moacir: Are you sure? I say that because the qcow2 files will be quite big.= =3D So if tiering is "file based" the SSD would have to be very, very big unle= =3D ss Gluster tiering do it by "chunks of data". At the bottom line, what is the good practice for using GlusterFS in small = =3D pods for enterprises? Don't forget jumbo frames. libgfapi (coming hopefully in 4.1.5). Sharding (= =3D enabled out of the box if you use a hyper-converged setup via gdeploy). Moacir: Yes! This is another reason to have separate networks for north/sou= =3D th and east/west. In that way I can use the standard MTU on the 10Gb NICs a= =3D nd jumbo frames on the file/move 40Gb NICs. Y. You opinion/feedback will be really appreciated! Moacir _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users --_000_VI1P190MB02856C6CA513853E8EDA2DB2C8B50VI1P190MB0285EURP_ Content-Type: text/html; charset=3D"iso-8859-1" Content-Transfer-Encoding: quoted-printable

Hi, in-line responses.

Thanks,

Moacir

From: Yaniv Kaul <= ykau=3D l(a)redhat.com>
Sent: Monday, August 7, 2017 7:42 AM
To: Moacir Ferreira
Cc: users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices

On Sun, Aug 6, 2017 at 5:49 PM, Moacir Ferreir= a =3D <m= oacirf=3D erreira(a)hotmail.com> wrote:

I am willing to assemble a oVirt "pod", made of 3 server= =3D s, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The id= =3D ea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual= =3D 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop like a server triangle using the 40Gb NICs for virtual= =3D ization files (VMs .qcow2) access and to move VMs around the pod (east /wes= =3D t traffic) while using the 10Gb interfaces for giving services to the outsi= =3D de world (north/south traffic).

Very nice gear. How are you planning the network exactly? Without a sw= =3D itch, back-to-back? (sounds OK to me, just wanted to ensure this is what th= =3D e 'dual' is used for). However, I'm unsure if you have the correct balance = =3D between the interface speeds (40g) and the disks (too many HDDs?).

Moaci=3D r: The idea is to have a very high performance network for the distributed fi= =3D le system and to prevent bottlenecks when we move one VM from a node to ano= =3D ther. Using 40Gb NICs I can just connect = th=3D e servers back-to-back. In this case I don't need the expensive 40Gb switch= =3D , I get very high speed and no contention between north/south traffic with = =3D east/west.

This said, my first question is: How should I deploy GlusterFS in such o= =3D Virt scenario? My questions are:

1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and t= =3D hen create a GlusterFS using them?

I would assume RAID 1 for the operating system (you don't want a singl= =3D e point of failure there?) and the rest JBODs. The SSD will be used for cac= =3D hing, I reckon? (I personally would add more SSDs instead of HDDs, but it d= =3D oes depend on the disk sizes and your space requirements.

Moacir: Yes, I agree that I need a R= AI=3D D-1 for the OS. Now, generic JBOD or a JBOD assembled using RAID-5 "di= =3D sks" created by the serv= er=3D 's disk controller?

2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not c= =3D onsuming too much disk space?

Replica 2 + Arbiter sounds good to me.

Moacir: I agree, and that is what I a= m =3D using.

4 - Does a oVirt hypervisor pod like I am planning to build, and the vir= =3D tualization environment, benefits from tiering when using a SSD disk? And y= =3D es, will Gluster do it by default or I have to configure it to do so?

Yes, I believe using lvmcache is the best way to go.

Moacir: Are you sure? I say that because the qcow2 files will be quite b= =3D ig. So if tiering is "file based" the SSD would have to be very, = =3D very big unless Gluster tiering do it by "chunks of data".

At the bottom line, what is the good practice for using GlusterFS in sma= =3D ll pods for enterprises?

Don't forget jumbo frames. libgfapi (coming hopefully in 4.1.5). Shard= =3D ing (enabled out of the box if you use a hyper-converged setup via gdeploy)= =3D .
Moacir: Yes! This is another reason to have separate networks for no= =3D rth/south and east/west. In that way I can use the standard MTU on the 10Gb= =3D NICs and jumbo frames on the file/move 40Gb NICs.

Y.

You opinion/feedback will be really appreciated!

Moacir

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users<=3D br>

--_000_VI1P190MB02856C6CA513853E8EDA2DB2C8B50VI1P190MB0285EURP_-- --===============7703353261016013733== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" LS1fMDAwX1ZJMVAxOTBNQjAyODU2QzZDQTUxMzg1M0U4RURBMkRCMkM4QjUwVkkxUDE5ME1CMDI4 NUVVUlBfCkNvbnRlbnQtVHlwZTogdGV4dC9wbGFpbjsgY2hhcnNldD0iaXNvLTg4NTktMSIKQ29u dGVudC1UcmFuc2Zlci1FbmNvZGluZzogcXVvdGVkLXByaW50YWJsZQoKSGksIGluLWxpbmUgcmVz cG9uc2VzLgoKClRoYW5rcywKCk1vYWNpcgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X18KRnJvbTogWWFuaXYgS2F1bCA8eWthdWxAcmVkaGF0LmNvbT4KU2VudDogTW9uZGF5LCBBdWd1 c3QgNywgMjAxNyA3OjQyIEFNClRvOiBNb2FjaXIgRmVycmVpcmEKQ2M6IHVzZXJzQG92aXJ0Lm9y ZwpTdWJqZWN0OiBSZTogW292aXJ0LXVzZXJzXSBHb29kIHByYWN0aWNlcwoKCgpPbiBTdW4sIEF1 ZyA2LCAyMDE3IGF0IDU6NDkgUE0sIE1vYWNpciBGZXJyZWlyYSA8bW9hY2lyZmVycmVpcmFAaG90 bWFpbC5jb209CjxtYWlsdG86bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb20+PiB3cm90ZToKCkkg YW0gd2lsbGluZyB0byBhc3NlbWJsZSBhIG9WaXJ0ICJwb2QiLCBtYWRlIG9mIDMgc2VydmVycywg ZWFjaCB3aXRoIDIgQ1BVID0Kc29ja2V0cyBvZiAxMiBjb3JlcywgMjU2R0IgUkFNLCA3IEhERCAx MEssIDEgU1NELiBUaGUgaWRlYSBpcyB0byB1c2UgR2x1c3RlPQpyRlMgdG8gcHJvdmlkZSBIQSBm b3IgdGhlIFZNcy4gVGhlIDMgc2VydmVycyBoYXZlIGEgZHVhbCA0MEdiIE5JQyBhbmQgYSBkdWE9 CmwgMTBHYiBOSUMuIFNvIG15IGludGVudGlvbiBpcyB0byBjcmVhdGUgYSBsb29wIGxpa2UgYSBz ZXJ2ZXIgdHJpYW5nbGUgdXNpbj0KZyB0aGUgNDBHYiBOSUNzIGZvciB2aXJ0dWFsaXphdGlvbiBm aWxlcyAoVk1zIC5xY293MikgYWNjZXNzIGFuZCB0byBtb3ZlIFZNPQpzIGFyb3VuZCB0aGUgcG9k IChlYXN0IC93ZXN0IHRyYWZmaWMpIHdoaWxlIHVzaW5nIHRoZSAxMEdiIGludGVyZmFjZXMgZm9y IGc9Cml2aW5nIHNlcnZpY2VzIHRvIHRoZSBvdXRzaWRlIHdvcmxkIChub3J0aC9zb3V0aCB0cmFm ZmljKS4KClZlcnkgbmljZSBnZWFyLiBIb3cgYXJlIHlvdSBwbGFubmluZyB0aGUgbmV0d29yayBl eGFjdGx5PyBXaXRob3V0IGEgc3dpdGNoLD0KIGJhY2stdG8tYmFjaz8gKHNvdW5kcyBPSyB0byBt ZSwganVzdCB3YW50ZWQgdG8gZW5zdXJlIHRoaXMgaXMgd2hhdCB0aGUgJ2R1PQphbCcgaXMgdXNl ZCBmb3IpLiBIb3dldmVyLCBJJ20gdW5zdXJlIGlmIHlvdSBoYXZlIHRoZSBjb3JyZWN0IGJhbGFu Y2UgYmV0d2U9CmVuIHRoZSBpbnRlcmZhY2Ugc3BlZWRzICg0MGcpIGFuZCB0aGUgZGlza3MgKHRv byBtYW55IEhERHM/KS4KCk1vYWNpcjogVGhlIGlkZWEgaXMgdG8gaGF2ZSBhIHZlcnkgaGlnaCBw ZXJmb3JtYW5jZSBuZXR3b3JrIGZvciB0aGUgZGlzdHJpYj0KdXRlZCBmaWxlIHN5c3RlbSBhbmQg dG8gcHJldmVudCBib3R0bGVuZWNrcyB3aGVuIHdlIG1vdmUgb25lIFZNIGZyb20gYSBub2RlPQog dG8gYW5vdGhlci4gVXNpbmcgNDBHYiBOSUNzIEkgY2FuIGp1c3QgY29ubmVjdCB0aGUgc2VydmVy cyBiYWNrLXRvLWJhY2suIEk9Cm4gdGhpcyBjYXNlIEkgZG9uJ3QgbmVlZCB0aGUgZXhwZW5zaXZl IDQwR2Igc3dpdGNoLCBJIGdldCB2ZXJ5IGhpZ2ggc3BlZWQgYT0KbmQgbm8gY29udGVudGlvbiBi ZXR3ZWVuIG5vcnRoL3NvdXRoIHRyYWZmaWMgd2l0aCBlYXN0L3dlc3QuCgoKClRoaXMgc2FpZCwg bXkgZmlyc3QgcXVlc3Rpb24gaXM6IEhvdyBzaG91bGQgSSBkZXBsb3kgR2x1c3RlckZTIGluIHN1 Y2ggb1Zpcj0KdCBzY2VuYXJpbz8gTXkgcXVlc3Rpb25zIGFyZToKCgoxIC0gU2hvdWxkIEkgY3Jl YXRlIDMgUkFJRCAoaS5lLjogUkFJRCA1KSwgb25lIG9uIGVhY2ggb1ZpcnQgbm9kZSwgYW5kIHRo ZW49CiBjcmVhdGUgYSBHbHVzdGVyRlMgdXNpbmcgdGhlbT8KCkkgd291bGQgYXNzdW1lIFJBSUQg MSBmb3IgdGhlIG9wZXJhdGluZyBzeXN0ZW0gKHlvdSBkb24ndCB3YW50IGEgc2luZ2xlIHBvaT0K bnQgb2YgZmFpbHVyZSB0aGVyZT8pIGFuZCB0aGUgcmVzdCBKQk9Ecy4gVGhlIFNTRCB3aWxsIGJl IHVzZWQgZm9yIGNhY2hpbmcsPQogSSByZWNrb24/IChJIHBlcnNvbmFsbHkgd291bGQgYWRkIG1v cmUgU1NEcyBpbnN0ZWFkIG9mIEhERHMsIGJ1dCBpdCBkb2VzIGQ9CmVwZW5kIG9uIHRoZSBkaXNr IHNpemVzIGFuZCB5b3VyIHNwYWNlIHJlcXVpcmVtZW50cy4KCk1vYWNpcjogWWVzLCBJIGFncmVl IHRoYXQgSSBuZWVkIGEgUkFJRC0xIGZvciB0aGUgT1MuIE5vdywgZ2VuZXJpYyBKQk9EIG9yID0K YSBKQk9EIGFzc2VtYmxlZCB1c2luZyBSQUlELTUgImRpc2tzIiBjcmVhdGVkIGJ5IHRoZSBzZXJ2 ZXIncyBkaXNrIGNvbnRyb2xsPQplcj8KCgoyIC0gSW5zdGVhZCwgc2hvdWxkIEkgY3JlYXRlIGEg SkJPRCBhcnJheSBtYWRlIG9mIGFsbCBzZXJ2ZXIncyBkaXNrcz8KCjMgLSBXaGF0IGlzIHRoZSBi ZXN0IEdsdXN0ZXIgY29uZmlndXJhdGlvbiB0byBwcm92aWRlIGZvciBIQSB3aGlsZSBub3QgY29u cz0KdW1pbmcgdG9vIG11Y2ggZGlzayBzcGFjZT8KClJlcGxpY2EgMiArIEFyYml0ZXIgc291bmRz IGdvb2QgdG8gbWUuCk1vYWNpcjogSSBhZ3JlZSwgYW5kIHRoYXQgaXMgd2hhdCBJIGFtIHVzaW5n LgoKCjQgLSBEb2VzIGEgb1ZpcnQgaHlwZXJ2aXNvciBwb2QgbGlrZSBJIGFtIHBsYW5uaW5nIHRv IGJ1aWxkLCBhbmQgdGhlIHZpcnR1YT0KbGl6YXRpb24gZW52aXJvbm1lbnQsIGJlbmVmaXRzIGZy b20gdGllcmluZyB3aGVuIHVzaW5nIGEgU1NEIGRpc2s/IEFuZCB5ZXMsPQogd2lsbCBHbHVzdGVy IGRvIGl0IGJ5IGRlZmF1bHQgb3IgSSBoYXZlIHRvIGNvbmZpZ3VyZSBpdCB0byBkbyBzbz8KClll cywgSSBiZWxpZXZlIHVzaW5nIGx2bWNhY2hlIGlzIHRoZSBiZXN0IHdheSB0byBnby4KCk1vYWNp cjogQXJlIHlvdSBzdXJlPyBJIHNheSB0aGF0IGJlY2F1c2UgdGhlIHFjb3cyIGZpbGVzIHdpbGwg YmUgcXVpdGUgYmlnLj0KIFNvIGlmIHRpZXJpbmcgaXMgImZpbGUgYmFzZWQiIHRoZSBTU0Qgd291 bGQgaGF2ZSB0byBiZSB2ZXJ5LCB2ZXJ5IGJpZyB1bmxlPQpzcyBHbHVzdGVyIHRpZXJpbmcgZG8g aXQgYnkgImNodW5rcyBvZiBkYXRhIi4KCgpBdCB0aGUgYm90dG9tIGxpbmUsIHdoYXQgaXMgdGhl IGdvb2QgcHJhY3RpY2UgZm9yIHVzaW5nIEdsdXN0ZXJGUyBpbiBzbWFsbCA9CnBvZHMgZm9yIGVu dGVycHJpc2VzPwoKRG9uJ3QgZm9yZ2V0IGp1bWJvIGZyYW1lcy4gbGliZ2ZhcGkgKGNvbWluZyBo b3BlZnVsbHkgaW4gNC4xLjUpLiBTaGFyZGluZyAoPQplbmFibGVkIG91dCBvZiB0aGUgYm94IGlm IHlvdSB1c2UgYSBoeXBlci1jb252ZXJnZWQgc2V0dXAgdmlhIGdkZXBsb3kpLgpNb2FjaXI6IFll cyEgVGhpcyBpcyBhbm90aGVyIHJlYXNvbiB0byBoYXZlIHNlcGFyYXRlIG5ldHdvcmtzIGZvciBu b3J0aC9zb3U9CnRoIGFuZCBlYXN0L3dlc3QuIEluIHRoYXQgd2F5IEkgY2FuIHVzZSB0aGUgc3Rh bmRhcmQgTVRVIG9uIHRoZSAxMEdiIE5JQ3MgYT0KbmQganVtYm8gZnJhbWVzIG9uIHRoZSBmaWxl L21vdmUgNDBHYiBOSUNzLgoKWS4KCgoKWW91IG9waW5pb24vZmVlZGJhY2sgd2lsbCBiZSByZWFs bHkgYXBwcmVjaWF0ZWQhCgpNb2FjaXIKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fClVzZXJzIG1haWxpbmcgbGlzdApVc2Vyc0BvdmlydC5vcmc8bWFpbHRv OlVzZXJzQG92aXJ0Lm9yZz4KaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZv L3VzZXJzCgoKCi0tXzAwMF9WSTFQMTkwTUIwMjg1NkM2Q0E1MTM4NTNFOEVEQTJEQjJDOEI1MFZJ MVAxOTBNQjAyODVFVVJQXwpDb250ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD0iaXNvLTg4 NTktMSIKQ29udGVudC1UcmFuc2Zlci1FbmNvZGluZzogcXVvdGVkLXByaW50YWJsZQoKPGh0bWw+ CjxoZWFkPgo8bWV0YSBodHRwLWVxdWl2PTNEIkNvbnRlbnQtVHlwZSIgY29udGVudD0zRCJ0ZXh0 L2h0bWw7IGNoYXJzZXQ9M0Rpc28tODg1OS09CjEiPgo8c3R5bGUgdHlwZT0zRCJ0ZXh0L2NzcyIg c3R5bGU9M0QiZGlzcGxheTpub25lOyI+PCEtLSBQIHttYXJnaW4tdG9wOjA7bWFyZ2k9Cm4tYm90 dG9tOjA7fSAtLT48L3N0eWxlPgo8L2hlYWQ+Cjxib2R5IGRpcj0zRCJsdHIiPgo8ZGl2IGlkPTNE ImRpdnRhZ2RlZmF1bHR3cmFwcGVyIiBzdHlsZT0zRCJmb250LXNpemU6MTJwdDtjb2xvcjojMDAw MDAwO2ZvbnQ9Ci1mYW1pbHk6Q2FsaWJyaSxIZWx2ZXRpY2Esc2Fucy1zZXJpZjsiIGRpcj0zRCJs dHIiPgo8cD5IaSwgaW4tbGluZSByZXNwb25zZXMuPGJyPgo8L3A+Cjxicj4KPHA+VGhhbmtzLDwv cD4KPHA+TW9hY2lyPGJyPgo8L3A+Cjxicj4KPGRpdiBzdHlsZT0zRCJjb2xvcjogcmdiKDQ5LCA1 NSwgNTcpOyI+CjxociB0YWJpbmRleD0zRCItMSIgc3R5bGU9M0QiZGlzcGxheTppbmxpbmUtYmxv Y2s7IHdpZHRoOjk4JSI+CjxkaXYgaWQ9M0QiZGl2UnBseUZ3ZE1zZyIgZGlyPTNEImx0ciI+PGZv bnQgc3R5bGU9M0QiZm9udC1zaXplOjExcHQiIGZhY2U9Cj0zRCJDYWxpYnJpLCBzYW5zLXNlcmlm IiBjb2xvcj0zRCIjMDAwMDAwIj48Yj5Gcm9tOjwvYj4gWWFuaXYgS2F1bCAmbHQ7eWthdT0KbEBy ZWRoYXQuY29tJmd0Ozxicj4KPGI+U2VudDo8L2I+IE1vbmRheSwgQXVndXN0IDcsIDIwMTcgNzo0 MiBBTTxicj4KPGI+VG86PC9iPiBNb2FjaXIgRmVycmVpcmE8YnI+CjxiPkNjOjwvYj4gdXNlcnNA b3ZpcnQub3JnPGJyPgo8Yj5TdWJqZWN0OjwvYj4gUmU6IFtvdmlydC11c2Vyc10gR29vZCBwcmFj dGljZXM8L2ZvbnQ+CjxkaXY+Jm5ic3A7PC9kaXY+CjwvZGl2Pgo8ZGl2Pgo8ZGl2IGRpcj0zRCJs dHIiPjxicj4KPGRpdiBjbGFzcz0zRCJnbWFpbF9leHRyYSI+PGJyPgo8ZGl2IGNsYXNzPTNEImdt YWlsX3F1b3RlIj5PbiBTdW4sIEF1ZyA2LCAyMDE3IGF0IDU6NDkgUE0sIE1vYWNpciBGZXJyZWly YSA9CjxzcGFuIGRpcj0zRCJsdHIiPgombHQ7PGEgaHJlZj0zRCJtYWlsdG86bW9hY2lyZmVycmVp cmFAaG90bWFpbC5jb20iIHRhcmdldD0zRCJfYmxhbmsiPm1vYWNpcmY9CmVycmVpcmFAaG90bWFp bC5jb208L2E+Jmd0Ozwvc3Bhbj4gd3JvdGU6PGJyPgo8YmxvY2txdW90ZSBjbGFzcz0zRCJnbWFp bF9xdW90ZSIgc3R5bGU9M0QibWFyZ2luOjAgMCAwIC44ZXg7IGJvcmRlci1sZWZ0OjE9CnB4ICNj Y2Mgc29saWQ7IHBhZGRpbmctbGVmdDoxZXgiPgo8ZGl2IGRpcj0zRCJsdHIiPgo8ZGl2IGlkPTNE Im1fNTUwOTU4NTg4OTU2OTY5MDc5MWRpdnRhZ2RlZmF1bHR3cmFwcGVyIiBkaXI9M0QibHRyIiBz dHlsZT0zRCI9CmZvbnQtc2l6ZToxMnB0OyBjb2xvcjojMDAwMDAwOyBmb250LWZhbWlseTpDYWxp YnJpLEhlbHZldGljYSxzYW5zLXNlcmlmIj4KPHA+PHNwYW4+SSBhbSB3aWxsaW5nIHRvIGFzc2Vt YmxlIGEgb1ZpcnQgJnF1b3Q7cG9kJnF1b3Q7LCBtYWRlIG9mIDMgc2VydmVyPQpzLCBlYWNoIHdp dGggMiBDUFUgc29ja2V0cyBvZiAxMiBjb3JlcywgMjU2R0IgUkFNLCA3IEhERCAxMEssIDEgU1NE LiBUaGUgaWQ9CmVhIGlzIHRvIHVzZSBHbHVzdGVyRlMgdG8gcHJvdmlkZSBIQSBmb3IgdGhlIFZN cy4gVGhlIDMgc2VydmVycyBoYXZlIGEgZHVhbD0KIDQwR2IgTklDIGFuZCBhIGR1YWwgMTBHYiBO SUMuIFNvIG15IGludGVudGlvbgogaXMgdG8gY3JlYXRlIGEgbG9vcCBsaWtlIGEgc2VydmVyIHRy aWFuZ2xlIHVzaW5nIHRoZSA0MEdiIE5JQ3MgZm9yIHZpcnR1YWw9Cml6YXRpb24gZmlsZXMgKFZN cyAucWNvdzIpIGFjY2VzcyBhbmQgdG8gbW92ZSBWTXMgYXJvdW5kIHRoZSBwb2QgKGVhc3QgL3dl cz0KdCB0cmFmZmljKSB3aGlsZSB1c2luZyB0aGUgMTBHYiBpbnRlcmZhY2VzIGZvciBnaXZpbmcg c2VydmljZXMgdG8gdGhlIG91dHNpPQpkZSB3b3JsZCAobm9ydGgvc291dGggdHJhZmZpYykuPC9z cGFuPjwvcD4KPC9kaXY+CjwvZGl2Pgo8L2Jsb2NrcXVvdGU+CjxkaXY+PGJyPgo8L2Rpdj4KPGRp dj5WZXJ5IG5pY2UgZ2Vhci4gSG93IGFyZSB5b3UgcGxhbm5pbmcgdGhlIG5ldHdvcmsgZXhhY3Rs eT8gV2l0aG91dCBhIHN3PQppdGNoLCBiYWNrLXRvLWJhY2s/IChzb3VuZHMgT0sgdG8gbWUsIGp1 c3Qgd2FudGVkIHRvIGVuc3VyZSB0aGlzIGlzIHdoYXQgdGg9CmUgJ2R1YWwnIGlzIHVzZWQgZm9y KS4gSG93ZXZlciwgSSdtIHVuc3VyZSBpZiB5b3UgaGF2ZSB0aGUgY29ycmVjdCBiYWxhbmNlID0K YmV0d2VlbiB0aGUgaW50ZXJmYWNlIHNwZWVkcyAoNDBnKQogYW5kIHRoZSBkaXNrcyAodG9vIG1h bnkgSEREcz8pLjwvZGl2Pgo8ZGl2Pjxicj4KPHNwYW4gc3R5bGU9M0QiY29sb3I6IHJnYigwLCAw LCAwKTsiPk08L3NwYW4+PHNwYW4gc3R5bGU9M0QiY29sb3I6IHJnYigwLCAwPQosIDApOyI+bzwv c3Bhbj48c3BhbiBzdHlsZT0zRCJjb2xvcjogcmdiKDAsIDAsIDApOyI+YTwvc3Bhbj48c3BhbiBz dHlsZT0zRCI9CmNvbG9yOiByZ2IoMCwgMCwgMCk7Ij5jPC9zcGFuPjxzcGFuIHN0eWxlPTNEImNv bG9yOiByZ2IoMCwgMCwgMCk7Ij5pPC9zcGFuPj0KPHNwYW4gc3R5bGU9M0QiY29sb3I6IHJnYigw LCAwLCAwKTsiPnI8L3NwYW4+PHNwYW4gc3R5bGU9M0QiY29sb3I6IHJnYigwLCAwPQosIDApOyI+ Ojwvc3Bhbj48c3BhbiBzdHlsZT0zRCJjb2xvcjogcmdiKDAsIDAsIDApOyI+CiBUaGUgaWRlYSBp cyB0byBoYXZlIGEgdmVyeSBoaWdoIHBlcmZvcm1hbmNlIG5ldHdvcmsgZm9yIHRoZSBkaXN0cmli dXRlZCBmaT0KbGUgc3lzdGVtIGFuZCB0byBwcmV2ZW50IGJvdHRsZW5lY2tzIHdoZW4gd2UgbW92 ZSBvbmUgVk0gZnJvbSBhIG5vZGUgdG8gYW5vPQp0aGVyLiBVc2luZwo8L3NwYW4+PHNwYW4gc3R5 bGU9M0QiY29sb3I6IHJnYigwLCAwLCAwKTsiPjQwR2IgTklDcyBJIGNhbiBqdXN0IGNvbm5lY3Qg dGg9CmUgc2VydmVycyBiYWNrLXRvLWJhY2suIEluIHRoaXMgY2FzZSBJIGRvbid0IG5lZWQgdGhl IGV4cGVuc2l2ZSA0MEdiIHN3aXRjaD0KLCBJIGdldCB2ZXJ5IGhpZ2ggc3BlZWQgYW5kIG5vIGNv bnRlbnRpb24gYmV0d2VlbiBub3J0aC9zb3V0aCB0cmFmZmljIHdpdGggPQplYXN0L3dlc3QuPC9z cGFuPjxicj4KPGJyPgo8L2Rpdj4KPGJsb2NrcXVvdGUgY2xhc3M9M0QiZ21haWxfcXVvdGUiIHN0 eWxlPTNEIm1hcmdpbjowIDAgMCAuOGV4OyBib3JkZXItbGVmdDoxPQpweCAjY2NjIHNvbGlkOyBw YWRkaW5nLWxlZnQ6MWV4Ij4KPGRpdiBkaXI9M0QibHRyIj4KPGRpdiBpZD0zRCJtXzU1MDk1ODU4 ODk1Njk2OTA3OTFkaXZ0YWdkZWZhdWx0d3JhcHBlciIgZGlyPTNEImx0ciIgc3R5bGU9M0QiPQpm b250LXNpemU6MTJwdDsgY29sb3I6IzAwMDAwMDsgZm9udC1mYW1pbHk6Q2FsaWJyaSxIZWx2ZXRp Y2Esc2Fucy1zZXJpZiI+CjxwPjxicj4KPHNwYW4+PC9zcGFuPjwvcD4KPHA+VGhpcyBzYWlkLCBt eSBmaXJzdCBxdWVzdGlvbiBpczogSG93IHNob3VsZCBJIGRlcGxveSBHbHVzdGVyRlMgaW4gc3Vj aCBvPQpWaXJ0IHNjZW5hcmlvPyBNeSBxdWVzdGlvbnMgYXJlOjwvcD4KPHA+PGJyPgo8L3A+Cjxw PjEgLSBTaG91bGQgSSBjcmVhdGUgMyBSQUlEIChpLmUuOiBSQUlEIDUpLCBvbmUgb24gZWFjaCBv VmlydCBub2RlLCBhbmQgdD0KaGVuIGNyZWF0ZSBhIEdsdXN0ZXJGUyB1c2luZyB0aGVtPzwvcD4K PC9kaXY+CjwvZGl2Pgo8L2Jsb2NrcXVvdGU+CjxkaXY+SSB3b3VsZCBhc3N1bWUgUkFJRCAxIGZv ciB0aGUgb3BlcmF0aW5nIHN5c3RlbSAoeW91IGRvbid0IHdhbnQgYSBzaW5nbD0KZSBwb2ludCBv ZiBmYWlsdXJlIHRoZXJlPykgYW5kIHRoZSByZXN0IEpCT0RzLiBUaGUgU1NEIHdpbGwgYmUgdXNl ZCBmb3IgY2FjPQpoaW5nLCBJIHJlY2tvbj8gKEkgcGVyc29uYWxseSB3b3VsZCBhZGQgbW9yZSBT U0RzIGluc3RlYWQgb2YgSEREcywgYnV0IGl0IGQ9Cm9lcyBkZXBlbmQgb24gdGhlIGRpc2sgc2l6 ZXMgYW5kCiB5b3VyIHNwYWNlIHJlcXVpcmVtZW50cy48YnI+Cjxicj4KPHNwYW4gc3R5bGU9M0Qi Y29sb3I6IHJnYigwLCAwLCAwKTsiPk1vYWNpcjogWWVzLCBJIGFncmVlIHRoYXQgSSBuZWVkIGEg UkFJPQpELTEgZm9yIHRoZSBPUy4gTm93LCBnZW5lcmljIEpCT0Qgb3IgYSBKQk9EIGFzc2VtYmxl ZCB1c2luZyBSQUlELTUgJnF1b3Q7ZGk9CnNrcyZxdW90OyBjcmVhdGVkPC9zcGFuPjxzcGFuIHN0 eWxlPTNEImNvbG9yOiByZ2IoMCwgMCwgMCk7Ij4gYnkgdGhlIHNlcnZlcj0KJ3MgZGlzawo8L3Nw YW4+PHNwYW4gc3R5bGU9M0QiY29sb3I6IHJnYigwLCAwLCAwKTsiPmNvbnRyb2xsZXI/PC9zcGFu Pjxicj4KPC9kaXY+CjxzcGFuIHN0eWxlPTNEImNvbG9yOiByZ2IoMCwgMCwgMCk7Ij48L3NwYW4+ CjxkaXY+Jm5ic3A7PGJyPgo8L2Rpdj4KPGJsb2NrcXVvdGUgY2xhc3M9M0QiZ21haWxfcXVvdGUi IHN0eWxlPTNEIm1hcmdpbjowIDAgMCAuOGV4OyBib3JkZXItbGVmdDoxPQpweCAjY2NjIHNvbGlk OyBwYWRkaW5nLWxlZnQ6MWV4Ij4KPGRpdiBkaXI9M0QibHRyIj4KPGRpdiBpZD0zRCJtXzU1MDk1 ODU4ODk1Njk2OTA3OTFkaXZ0YWdkZWZhdWx0d3JhcHBlciIgZGlyPTNEImx0ciIgc3R5bGU9M0Qi PQpmb250LXNpemU6MTJwdDsgY29sb3I6IzAwMDAwMDsgZm9udC1mYW1pbHk6Q2FsaWJyaSxIZWx2 ZXRpY2Esc2Fucy1zZXJpZiI+CjxwPjIgLSBJbnN0ZWFkLCBzaG91bGQgSSBjcmVhdGUgYSBKQk9E IGFycmF5IG1hZGUgb2YgYWxsIHNlcnZlcidzIGRpc2tzPzwvcD0KPgo8cD4zIC0gV2hhdCBpcyB0 aGUgYmVzdCBHbHVzdGVyIGNvbmZpZ3VyYXRpb24gdG8gcHJvdmlkZSBmb3IgSEEgd2hpbGUgbm90 IGM9Cm9uc3VtaW5nIHRvbyBtdWNoIGRpc2sgc3BhY2U/PGJyPgo8L3A+CjwvZGl2Pgo8L2Rpdj4K PC9ibG9ja3F1b3RlPgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+UmVwbGljYSAyICYjNDM7IEFyYml0 ZXIgc291bmRzIGdvb2QgdG8gbWUuPC9kaXY+CjxkaXY+PHNwYW4gc3R5bGU9M0QiY29sb3I6IHJn YigwLCAwLCAwKTsiPk08L3NwYW4+PHNwYW4gc3R5bGU9M0QiY29sb3I6IHJnYj0KKDAsIDAsIDAp OyI+bzwvc3Bhbj48c3BhbiBzdHlsZT0zRCJjb2xvcjogcmdiKDAsIDAsIDApOyI+YTwvc3Bhbj48 c3BhbiBzdHlsPQplPTNEImNvbG9yOiByZ2IoMCwgMCwgMCk7Ij5jPC9zcGFuPjxzcGFuIHN0eWxl PTNEImNvbG9yOiByZ2IoMCwgMCwgMCk7Ij5pPC89CnNwYW4+PHNwYW4gc3R5bGU9M0QiY29sb3I6 IHJnYigwLCAwLCAwKTsiPnI8L3NwYW4+PHNwYW4gc3R5bGU9M0QiY29sb3I6IHJnYj0KKDAsIDAs IDApOyI+Ojwvc3Bhbj48c3BhbiBzdHlsZT0zRCJjb2xvcjogcmdiKDAsIDAsIDApOyI+Cjwvc3Bh bj48c3BhbiBzdHlsZT0zRCJjb2xvcjogcmdiKDAsIDAsIDApOyI+SSBhZ3JlZSwgYW5kIHRoYXQg aXMgd2hhdCBJIGFtID0KdXNpbmcuPC9zcGFuPjxicj4KPGJyPgo8L2Rpdj4KPGJsb2NrcXVvdGUg Y2xhc3M9M0QiZ21haWxfcXVvdGUiIHN0eWxlPTNEIm1hcmdpbjowIDAgMCAuOGV4OyBib3JkZXIt bGVmdDoxPQpweCAjY2NjIHNvbGlkOyBwYWRkaW5nLWxlZnQ6MWV4Ij4KPGRpdiBkaXI9M0QibHRy Ij4KPGRpdiBpZD0zRCJtXzU1MDk1ODU4ODk1Njk2OTA3OTFkaXZ0YWdkZWZhdWx0d3JhcHBlciIg ZGlyPTNEImx0ciIgc3R5bGU9M0QiPQpmb250LXNpemU6MTJwdDsgY29sb3I6IzAwMDAwMDsgZm9u dC1mYW1pbHk6Q2FsaWJyaSxIZWx2ZXRpY2Esc2Fucy1zZXJpZiI+CjxwPjwvcD4KPHA+NCAtIERv ZXMgYSBvVmlydCBoeXBlcnZpc29yIHBvZCBsaWtlIEkgYW0gcGxhbm5pbmcgdG8gYnVpbGQsIGFu ZCB0aGUgdmlyPQp0dWFsaXphdGlvbiBlbnZpcm9ubWVudCwgYmVuZWZpdHMgZnJvbSB0aWVyaW5n IHdoZW4gdXNpbmcgYSBTU0QgZGlzaz8gQW5kIHk9CmVzLCB3aWxsIEdsdXN0ZXIgZG8gaXQgYnkg ZGVmYXVsdCBvciBJIGhhdmUgdG8gY29uZmlndXJlIGl0IHRvIGRvIHNvPzwvcD4KPC9kaXY+Cjwv ZGl2Pgo8L2Jsb2NrcXVvdGU+CjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj5ZZXMsIEkgYmVsaWV2ZSB1 c2luZyBsdm1jYWNoZSBpcyB0aGUgYmVzdCB3YXkgdG8gZ28uJm5ic3A7PC9kaXY+CjxibG9ja3F1 b3RlIGNsYXNzPTNEImdtYWlsX3F1b3RlIiBzdHlsZT0zRCJtYXJnaW46MCAwIDAgLjhleDsgYm9y ZGVyLWxlZnQ6MT0KcHggI2NjYyBzb2xpZDsgcGFkZGluZy1sZWZ0OjFleCI+CjxkaXYgZGlyPTNE Imx0ciI+CjxkaXYgaWQ9M0QibV81NTA5NTg1ODg5NTY5NjkwNzkxZGl2dGFnZGVmYXVsdHdyYXBw ZXIiIGRpcj0zRCJsdHIiIHN0eWxlPTNEIj0KZm9udC1zaXplOjEycHQ7IGNvbG9yOiMwMDAwMDA7 IGZvbnQtZmFtaWx5OkNhbGlicmksSGVsdmV0aWNhLHNhbnMtc2VyaWYiPgo8cD5Nb2FjaXI6IEFy ZSB5b3Ugc3VyZT8gSSBzYXkgdGhhdCBiZWNhdXNlIHRoZSBxY293MiBmaWxlcyB3aWxsIGJlIHF1 aXRlIGI9CmlnLiBTbyBpZiB0aWVyaW5nIGlzICZxdW90O2ZpbGUgYmFzZWQmcXVvdDsgdGhlIFNT RCB3b3VsZCBoYXZlIHRvIGJlIHZlcnksID0KdmVyeSBiaWcgdW5sZXNzIEdsdXN0ZXIgdGllcmlu ZyBkbyBpdCBieSAmcXVvdDtjaHVua3Mgb2YgZGF0YSZxdW90Oy48YnI+CjwvcD4KPHA+PGJyPgo8 L3A+CjxwPkF0IHRoZSBib3R0b20gbGluZSwgd2hhdCBpcyB0aGUgZ29vZCBwcmFjdGljZSBmb3Ig dXNpbmcgR2x1c3RlckZTIGluIHNtYT0KbGwgcG9kcyBmb3IgZW50ZXJwcmlzZXM/PGJyPgo8L3A+ CjwvZGl2Pgo8L2Rpdj4KPC9ibG9ja3F1b3RlPgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+RG9uJ3Qg Zm9yZ2V0IGp1bWJvIGZyYW1lcy4gbGliZ2ZhcGkgKGNvbWluZyBob3BlZnVsbHkgaW4gNC4xLjUp LiBTaGFyZD0KaW5nIChlbmFibGVkIG91dCBvZiB0aGUgYm94IGlmIHlvdSB1c2UgYSBoeXBlci1j b252ZXJnZWQgc2V0dXAgdmlhIGdkZXBsb3kpPQouPGJyPgo8Yj5Nb2FjaXI6PC9iPiBZZXMhIFRo aXMgaXMgYW5vdGhlciByZWFzb24gdG8gaGF2ZSBzZXBhcmF0ZSBuZXR3b3JrcyBmb3Igbm89CnJ0 aC9zb3V0aCBhbmQgZWFzdC93ZXN0LiBJbiB0aGF0IHdheSBJIGNhbiB1c2UgdGhlIHN0YW5kYXJk IE1UVSBvbiB0aGUgMTBHYj0KIE5JQ3MgYW5kIGp1bWJvIGZyYW1lcyBvbiB0aGUgZmlsZS9tb3Zl IDQwR2IgTklDcy48YnI+Cjxicj4KPC9kaXY+CjxkaXY+WS48L2Rpdj4KPGRpdj4mbmJzcDs8L2Rp dj4KPGJsb2NrcXVvdGUgY2xhc3M9M0QiZ21haWxfcXVvdGUiIHN0eWxlPTNEIm1hcmdpbjowIDAg MCAuOGV4OyBib3JkZXItbGVmdDoxPQpweCAjY2NjIHNvbGlkOyBwYWRkaW5nLWxlZnQ6MWV4Ij4K PGRpdiBkaXI9M0QibHRyIj4KPGRpdiBpZD0zRCJtXzU1MDk1ODU4ODk1Njk2OTA3OTFkaXZ0YWdk ZWZhdWx0d3JhcHBlciIgZGlyPTNEImx0ciIgc3R5bGU9M0QiPQpmb250LXNpemU6MTJwdDsgY29s b3I6IzAwMDAwMDsgZm9udC1mYW1pbHk6Q2FsaWJyaSxIZWx2ZXRpY2Esc2Fucy1zZXJpZiI+Cjxw PjwvcD4KPHA+PGJyPgo8L3A+CjxwPllvdSBvcGluaW9uL2ZlZWRiYWNrIHdpbGwgYmUgcmVhbGx5 IGFwcHJlY2lhdGVkITwvcD4KPHNwYW4gY2xhc3M9M0QiSE9FblpiIj48Zm9udCBjb2xvcj0zRCIj ODg4ODg4Ij4KPHA+TW9hY2lyPGJyPgo8L3A+CjwvZm9udD48L3NwYW4+PC9kaXY+CjwvZGl2Pgo8 YnI+Cl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzx3YnI+X19fX19fX19fX19fX19fX188 YnI+ClVzZXJzIG1haWxpbmcgbGlzdDxicj4KPGEgaHJlZj0zRCJtYWlsdG86VXNlcnNAb3ZpcnQu b3JnIj5Vc2Vyc0BvdmlydC5vcmc8L2E+PGJyPgo8YSBocmVmPTNEImh0dHA6Ly9saXN0cy5vdmly dC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyIgcmVsPTNEIm5vcmVmZXJyZXI9CiIgdGFyZ2V0 PTNEIl9ibGFuayI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy88d2JyPm1haWxtYW4vbGlzdGluZm8v dXNlcnM8L2E+PD0KYnI+Cjxicj4KPC9ibG9ja3F1b3RlPgo8L2Rpdj4KPGJyPgo8L2Rpdj4KPC9k aXY+CjwvZGl2Pgo8L2Rpdj4KPC9kaXY+CjwvYm9keT4KPC9odG1sPgoKLS1fMDAwX1ZJMVAxOTBN QjAyODU2QzZDQTUxMzg1M0U4RURBMkRCMkM4QjUwVkkxUDE5ME1CMDI4NUVVUlBfLS0K --===============7703353261016013733==-- From fernando.frediani at upx.com Mon Aug 7 13:04:29 2017 Content-Type: multipart/mixed; boundary="===============2131844939483290490==" MIME-Version: 1.0 From: FERNANDO FREDIANI To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Mon, 07 Aug 2017 10:04:23 -0300 Message-ID: <63bac47b-afe6-0258-d3d7-e545a5004c30@upx.com> In-Reply-To: CANCGKEp4XGs0U+Qs78eEmqCNtvpLY-Azjb5DcGhZ9yiKTBEEfw@mail.gmail.com --===============2131844939483290490== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------ECDF126785BB7A19D9FCBAF5 Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 8bit For any RAID 5 or 6 configuration I normally follow a simple gold rule = which gave good results so far: - up to 4 disks RAID 5 - 5 or more disks RAID 6 However I didn't really understand well the recommendation to use any = RAID with GlusterFS. I always thought that GlusteFS likes to work in = JBOD mode and control the disks (bricks) directlly so you can create = whatever distribution rule you wish, and if a single disk fails you just = replace it and which obviously have the data replicated from another. = The only downside of using in this way is that the replication data will = be flow accross all servers but that is not much a big issue. Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS. Thanks Regards Fernando On 07/08/2017 03:46, Devin Acosta wrote: > > Moacir, > > I have recently installed multiple Red Hat Virtualization hosts for = > several different companies, and have dealt with the Red Hat Support = > Team in depth about optimal configuration in regards to setting up = > GlusterFS most efficiently and I wanted to share with you what I learned. > > In general Red Hat Virtualization team frowns upon using each DISK of = > the system as just a JBOD, sure there is some protection by having the = > data replicated, however, the recommendation is to use RAID 6 = > (preferred) or RAID-5, or at least RAID-1 at the very least. > > Here is the direct quote from Red Hat when I asked about RAID and Bricks: > / > / > /"A typical Gluster configuration would use RAID underneath the = > bricks. RAID 6 is most typical as it gives you 2 disk failure = > protection, but RAID 5 could be used too. Once you have the RAIDed = > bricks, you'd then apply the desired replication on top of that. The = > most popular way of doing this would be distributed replicated with 2x = > replication. In general you'll get better performance with larger = > bricks. 12 drives is often a sweet spot. Another option would be to = > create a separate tier using all SSD=E2=80=99s.=E2=80=9D / > > /In order to SSD tiering from my understanding you would need 1 x NVMe = > drive in each server, or 4 x SSD hot tier (it needs to be distributed, = > replicated for the hot tier if not using NVME). So with you only = > having 1 SSD drive in each server, I=E2=80=99d suggest maybe looking into= the = > NVME option. / > / > / > /Since your using only 3-servers, what I=E2=80=99d probably suggest is to= do = > (2 Replicas + Arbiter Node), this setup actually doesn=E2=80=99t require = the = > 3rd server to have big drives at all as it only stores meta-data about = > the files and not actually a full copy. / > / > / > /Please see the attached document that was given to me by Red Hat to = > get more information on this. Hope this information helps you./ > / > / > > -- > > Devin Acosta, RHCA, RHVCA > Red Hat Certified Architect > > On August 6, 2017 at 7:29:29 PM, Moacir Ferreira = > (moacirferreira(a)hotmail.com ) wrot= e: > >> I am willing to assemble a oVirt "pod", made of 3 servers, each with = >> 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is = >> to use GlusterFS to provide HA for the VMs. The 3 servers have a dual = >> 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop = >> like a server triangle using the 40Gb NICs for virtualization files = >> (VMs .qcow2) access and to move VMs around the pod (east /west = >> traffic) while using the 10Gb interfaces for giving services to the = >> outside world (north/south traffic). >> >> >> This said, my first question is: How should I deploy GlusterFS in = >> such oVirt scenario? My questions are: >> >> >> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, = >> and then create a GlusterFS using them? >> >> 2 - Instead, should I create a JBOD array made of all server's disks? >> >> 3 - What is the best Gluster configuration to provide for HA while = >> not consuming too much disk space? >> >> 4 - Does a oVirt hypervisor pod like I am planning to build, and the = >> virtualization environment, benefits from tiering when using a SSD = >> disk? And yes, will Gluster do it by default or I have to configure = >> it to do so? >> >> >> At the bottom line, what is the good practice for using GlusterFS in = >> small pods for enterprises? >> >> >> You opinion/feedback will be really appreciated! >> >> Moacir >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users --------------ECDF126785BB7A19D9FCBAF5 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit
For any RAID 5 or 6 configuration I normally follow a simple gold rule which gave good results so far:
- up to 4 disks RAID 5
- 5 or more disks RAID 6

However I didn't really understand well the recommendation to use any RAID with GlusterFS. I always thought that GlusteFS likes to work in JBOD mode and control the disks (bricks) directlly so you can create whatever distribution rule you wish, and if a single disk fails you just replace it and which obviously have the data replicated from another. The only downside of using in this way is that the replication data will be flow accross all servers but that is not much a big issue.

Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS.

Thanks
Regards
Fernando

On 07/08/2017 03:46, Devin Acosta wrote:

Moacir,

I have recently installed multiple Red Hat Virtualization hosts for several different companies, and have dealt with the Red Hat Support Team in depth about optimal configuration in regards to setting up GlusterFS most efficiently and I wanted to share with you what I learned.=

In general Red Hat Virtualization team frowns upon using each DISK of the system as just a JBOD, sure there is some protection by having the data replicated, however, the recommendation is to use RAID 6 (preferred) or RAID-5, or at least RAID-1 at the very least.

Here is the=C2=A0direct quote from Red Hat when I asked abo= ut RAID and Bricks:

"A typical Gluster configuration would use RAID underneath the bricks. RAID 6 is most typical as it gives you 2 disk failure protection, but RAID 5 could be used too.=C2=A0Once you have the RAIDed bricks, you'd then apply the desired replication on top of that. The most popular way of doing this would be distributed replicated with 2x replication.=C2=A0In general you'll get better performance with larger bricks. 12 drives is often a sweet spot. Another option would be to create a separate tier using all SSD=E2=80= =99s.=E2=80=9D=C2=A0

In order to SSD=C2=A0tiering from my understanding you would need 1 x NVMe drive in each server, or 4 x SSD hot tier (it needs to be distributed, replicated for the hot tier if not using NVME). So with you only having 1 SSD drive in each server,=C2=A0I=E2=80=99d suggest maybe looking into the= NVME option.=C2=A0

Since your using only 3-servers, what I=E2=80=99d probab= ly suggest is to do (2 Replicas + Arbiter Node), this setup actually doesn=E2=80=99t require the 3rd server to have big dri= ves at all as it only stores meta-data about the files and not actually a full copy.=C2=A0

Please see the attached document that was given to me by Red Hat to get more information on this. Hope this information helps you.

-= -

Devin Acosta, RHCA, RHVCA

Red Hat Certified Architect

On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (moacirferreira(a)hotmail.com) wrote:=

I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop like a server triangle using the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VMs around the pod (east /west traffic) while using the 10Gb interfaces for giving services to the outside world (north/south traffic).

This said, my first question is: How should I deploy GlusterFS in such oVirt scenario? My questions are:

1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then create a GlusterFS using them?

2 - Instead, should I create a JBOD array made of all server's disks?

3 - What is the best Gluster configuration to provide for HA while not consuming too much disk space?

4 - Does a oVirt hypervisor pod like I am planning to build, and the virtualization environment, benefits from tiering when using a SSD disk? And yes, will Gluster do it by default or I have to configure it to do so?

At the bottom line, what is the good practice for using GlusterFS in small pods for enterprises?

You opinion/feedback will be really appreciated!

Moacir

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/lis= tinfo/users

_______________________________________________ Users mailing list Use= rs(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

--------------ECDF126785BB7A19D9FCBAF5-- --===============2131844939483290490== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS1FQ0RGMTI2Nzg1QkI3QTE5RDlGQ0JBRjUKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDhiaXQK CkZvciBhbnkgUkFJRCA1IG9yIDYgY29uZmlndXJhdGlvbiBJIG5vcm1hbGx5IGZvbGxvdyBhIHNp bXBsZSBnb2xkIHJ1bGUgCndoaWNoIGdhdmUgZ29vZCByZXN1bHRzIHNvIGZhcjoKLSB1cCB0byA0 IGRpc2tzIFJBSUQgNQotIDUgb3IgbW9yZSBkaXNrcyBSQUlEIDYKCkhvd2V2ZXIgSSBkaWRuJ3Qg cmVhbGx5IHVuZGVyc3RhbmQgd2VsbCB0aGUgcmVjb21tZW5kYXRpb24gdG8gdXNlIGFueSAKUkFJ RCB3aXRoIEdsdXN0ZXJGUy4gSSBhbHdheXMgdGhvdWdodCB0aGF0IEdsdXN0ZUZTIGxpa2VzIHRv IHdvcmsgaW4gCkpCT0QgbW9kZSBhbmQgY29udHJvbCB0aGUgZGlza3MgKGJyaWNrcykgZGlyZWN0 bGx5IHNvIHlvdSBjYW4gY3JlYXRlIAp3aGF0ZXZlciBkaXN0cmlidXRpb24gcnVsZSB5b3Ugd2lz aCwgYW5kIGlmIGEgc2luZ2xlIGRpc2sgZmFpbHMgeW91IGp1c3QgCnJlcGxhY2UgaXQgYW5kIHdo aWNoIG9idmlvdXNseSBoYXZlIHRoZSBkYXRhIHJlcGxpY2F0ZWQgZnJvbSBhbm90aGVyLiAKVGhl IG9ubHkgZG93bnNpZGUgb2YgdXNpbmcgaW4gdGhpcyB3YXkgaXMgdGhhdCB0aGUgcmVwbGljYXRp b24gZGF0YSB3aWxsIApiZSBmbG93IGFjY3Jvc3MgYWxsIHNlcnZlcnMgYnV0IHRoYXQgaXMgbm90 IG11Y2ggYSBiaWcgaXNzdWUuCgpBbnlvbmUgY2FuIGVsYWJvcmF0ZSBhYm91dCBVc2luZyBSQUlE ICsgR2x1c3RlckZTIGFuZCBKQk9EICsgR2x1c3RlckZTLgoKVGhhbmtzClJlZ2FyZHMKRmVybmFu ZG8KCgpPbiAwNy8wOC8yMDE3IDAzOjQ2LCBEZXZpbiBBY29zdGEgd3JvdGU6Cj4KPiBNb2FjaXIs Cj4KPiBJIGhhdmUgcmVjZW50bHkgaW5zdGFsbGVkIG11bHRpcGxlIFJlZCBIYXQgVmlydHVhbGl6 YXRpb24gaG9zdHMgZm9yIAo+IHNldmVyYWwgZGlmZmVyZW50IGNvbXBhbmllcywgYW5kIGhhdmUg ZGVhbHQgd2l0aCB0aGUgUmVkIEhhdCBTdXBwb3J0IAo+IFRlYW0gaW4gZGVwdGggYWJvdXQgb3B0 aW1hbCBjb25maWd1cmF0aW9uIGluIHJlZ2FyZHMgdG8gc2V0dGluZyB1cCAKPiBHbHVzdGVyRlMg bW9zdCBlZmZpY2llbnRseSBhbmQgSSB3YW50ZWQgdG8gc2hhcmUgd2l0aCB5b3Ugd2hhdCBJIGxl YXJuZWQuCj4KPiBJbiBnZW5lcmFsIFJlZCBIYXQgVmlydHVhbGl6YXRpb24gdGVhbSBmcm93bnMg dXBvbiB1c2luZyBlYWNoIERJU0sgb2YgCj4gdGhlIHN5c3RlbSBhcyBqdXN0IGEgSkJPRCwgc3Vy ZSB0aGVyZSBpcyBzb21lIHByb3RlY3Rpb24gYnkgaGF2aW5nIHRoZSAKPiBkYXRhIHJlcGxpY2F0 ZWQsIGhvd2V2ZXIsIHRoZSByZWNvbW1lbmRhdGlvbiBpcyB0byB1c2UgUkFJRCA2IAo+IChwcmVm ZXJyZWQpIG9yIFJBSUQtNSwgb3IgYXQgbGVhc3QgUkFJRC0xIGF0IHRoZSB2ZXJ5IGxlYXN0Lgo+ Cj4gSGVyZSBpcyB0aGUgZGlyZWN0IHF1b3RlIGZyb20gUmVkIEhhdCB3aGVuIEkgYXNrZWQgYWJv dXQgUkFJRCBhbmQgQnJpY2tzOgo+IC8KPiAvCj4gLyJBIHR5cGljYWwgR2x1c3RlciBjb25maWd1 cmF0aW9uIHdvdWxkIHVzZSBSQUlEIHVuZGVybmVhdGggdGhlIAo+IGJyaWNrcy4gUkFJRCA2IGlz IG1vc3QgdHlwaWNhbCBhcyBpdCBnaXZlcyB5b3UgMiBkaXNrIGZhaWx1cmUgCj4gcHJvdGVjdGlv biwgYnV0IFJBSUQgNSBjb3VsZCBiZSB1c2VkIHRvby4gT25jZSB5b3UgaGF2ZSB0aGUgUkFJRGVk IAo+IGJyaWNrcywgeW91J2QgdGhlbiBhcHBseSB0aGUgZGVzaXJlZCByZXBsaWNhdGlvbiBvbiB0 b3Agb2YgdGhhdC4gVGhlIAo+IG1vc3QgcG9wdWxhciB3YXkgb2YgZG9pbmcgdGhpcyB3b3VsZCBi ZSBkaXN0cmlidXRlZCByZXBsaWNhdGVkIHdpdGggMnggCj4gcmVwbGljYXRpb24uIEluIGdlbmVy YWwgeW91J2xsIGdldCBiZXR0ZXIgcGVyZm9ybWFuY2Ugd2l0aCBsYXJnZXIgCj4gYnJpY2tzLiAx MiBkcml2ZXMgaXMgb2Z0ZW4gYSBzd2VldCBzcG90LiBBbm90aGVyIG9wdGlvbiB3b3VsZCBiZSB0 byAKPiBjcmVhdGUgYSBzZXBhcmF0ZSB0aWVyIHVzaW5nIGFsbCBTU0TigJlzLuKAnSAvCj4KPiAv SW4gb3JkZXIgdG8gU1NEIHRpZXJpbmcgZnJvbSBteSB1bmRlcnN0YW5kaW5nIHlvdSB3b3VsZCBu ZWVkIDEgeCBOVk1lIAo+IGRyaXZlIGluIGVhY2ggc2VydmVyLCBvciA0IHggU1NEIGhvdCB0aWVy IChpdCBuZWVkcyB0byBiZSBkaXN0cmlidXRlZCwgCj4gcmVwbGljYXRlZCBmb3IgdGhlIGhvdCB0 aWVyIGlmIG5vdCB1c2luZyBOVk1FKS4gU28gd2l0aCB5b3Ugb25seSAKPiBoYXZpbmcgMSBTU0Qg ZHJpdmUgaW4gZWFjaCBzZXJ2ZXIsIEnigJlkIHN1Z2dlc3QgbWF5YmUgbG9va2luZyBpbnRvIHRo ZSAKPiBOVk1FIG9wdGlvbi4gLwo+IC8KPiAvCj4gL1NpbmNlIHlvdXIgdXNpbmcgb25seSAzLXNl cnZlcnMsIHdoYXQgSeKAmWQgcHJvYmFibHkgc3VnZ2VzdCBpcyB0byBkbyAKPiAoMiBSZXBsaWNh cyArIEFyYml0ZXIgTm9kZSksIHRoaXMgc2V0dXAgYWN0dWFsbHkgZG9lc27igJl0IHJlcXVpcmUg dGhlIAo+IDNyZCBzZXJ2ZXIgdG8gaGF2ZSBiaWcgZHJpdmVzIGF0IGFsbCBhcyBpdCBvbmx5IHN0 b3JlcyBtZXRhLWRhdGEgYWJvdXQgCj4gdGhlIGZpbGVzIGFuZCBub3QgYWN0dWFsbHkgYSBmdWxs IGNvcHkuIC8KPiAvCj4gLwo+IC9QbGVhc2Ugc2VlIHRoZSBhdHRhY2hlZCBkb2N1bWVudCB0aGF0 IHdhcyBnaXZlbiB0byBtZSBieSBSZWQgSGF0IHRvIAo+IGdldCBtb3JlIGluZm9ybWF0aW9uIG9u IHRoaXMuIEhvcGUgdGhpcyBpbmZvcm1hdGlvbiBoZWxwcyB5b3UuLwo+IC8KPiAvCj4KPiAtLQo+ Cj4gRGV2aW4gQWNvc3RhLCBSSENBLCBSSFZDQQo+IFJlZCBIYXQgQ2VydGlmaWVkIEFyY2hpdGVj dAo+Cj4gT24gQXVndXN0IDYsIDIwMTcgYXQgNzoyOToyOSBQTSwgTW9hY2lyIEZlcnJlaXJhIAo+ IChtb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbSA8bWFpbHRvOm1vYWNpcmZlcnJlaXJhQGhvdG1h aWwuY29tPikgd3JvdGU6Cj4KPj4gSSBhbSB3aWxsaW5nIHRvIGFzc2VtYmxlIGEgb1ZpcnQgInBv ZCIsIG1hZGUgb2YgMyBzZXJ2ZXJzLCBlYWNoIHdpdGggCj4+IDIgQ1BVIHNvY2tldHMgb2YgMTIg Y29yZXMsIDI1NkdCIFJBTSwgNyBIREQgMTBLLCAxIFNTRC4gVGhlIGlkZWEgaXMgCj4+IHRvIHVz ZSBHbHVzdGVyRlMgdG8gcHJvdmlkZSBIQSBmb3IgdGhlIFZNcy4gVGhlIDMgc2VydmVycyBoYXZl IGEgZHVhbCAKPj4gNDBHYiBOSUMgYW5kIGEgZHVhbCAxMEdiIE5JQy4gU28gbXkgaW50ZW50aW9u IGlzIHRvIGNyZWF0ZSBhIGxvb3AgCj4+IGxpa2UgYSBzZXJ2ZXIgdHJpYW5nbGUgdXNpbmcgdGhl IDQwR2IgTklDcyBmb3IgdmlydHVhbGl6YXRpb24gZmlsZXMgCj4+IChWTXMgLnFjb3cyKSBhY2Nl c3MgYW5kIHRvIG1vdmUgVk1zIGFyb3VuZCB0aGUgcG9kIChlYXN0IC93ZXN0IAo+PiB0cmFmZmlj KSB3aGlsZSB1c2luZyB0aGUgMTBHYiBpbnRlcmZhY2VzIGZvciBnaXZpbmcgc2VydmljZXMgdG8g dGhlIAo+PiBvdXRzaWRlIHdvcmxkIChub3J0aC9zb3V0aCB0cmFmZmljKS4KPj4KPj4KPj4gVGhp cyBzYWlkLCBteSBmaXJzdCBxdWVzdGlvbiBpczogSG93IHNob3VsZCBJIGRlcGxveSBHbHVzdGVy RlMgaW4gCj4+IHN1Y2ggb1ZpcnQgc2NlbmFyaW8/IE15IHF1ZXN0aW9ucyBhcmU6Cj4+Cj4+Cj4+ IDEgLSBTaG91bGQgSSBjcmVhdGUgMyBSQUlEIChpLmUuOiBSQUlEIDUpLCBvbmUgb24gZWFjaCBv VmlydCBub2RlLCAKPj4gYW5kIHRoZW4gY3JlYXRlIGEgR2x1c3RlckZTIHVzaW5nIHRoZW0/Cj4+ Cj4+IDIgLSBJbnN0ZWFkLCBzaG91bGQgSSBjcmVhdGUgYSBKQk9EIGFycmF5IG1hZGUgb2YgYWxs IHNlcnZlcidzIGRpc2tzPwo+Pgo+PiAzIC0gV2hhdCBpcyB0aGUgYmVzdCBHbHVzdGVyIGNvbmZp Z3VyYXRpb24gdG8gcHJvdmlkZSBmb3IgSEEgd2hpbGUgCj4+IG5vdCBjb25zdW1pbmcgdG9vIG11 Y2ggZGlzayBzcGFjZT8KPj4KPj4gNCAtIERvZXMgYSBvVmlydCBoeXBlcnZpc29yIHBvZCBsaWtl IEkgYW0gcGxhbm5pbmcgdG8gYnVpbGQsIGFuZCB0aGUgCj4+IHZpcnR1YWxpemF0aW9uIGVudmly b25tZW50LCBiZW5lZml0cyBmcm9tIHRpZXJpbmcgd2hlbiB1c2luZyBhIFNTRCAKPj4gZGlzaz8g QW5kIHllcywgd2lsbCBHbHVzdGVyIGRvIGl0IGJ5IGRlZmF1bHQgb3IgSSBoYXZlIHRvIGNvbmZp Z3VyZSAKPj4gaXQgdG8gZG8gc28/Cj4+Cj4+Cj4+IEF0IHRoZSBib3R0b20gbGluZSwgd2hhdCBp cyB0aGUgZ29vZCBwcmFjdGljZSBmb3IgdXNpbmcgR2x1c3RlckZTIGluIAo+PiBzbWFsbCBwb2Rz IGZvciBlbnRlcnByaXNlcz8KPj4KPj4KPj4gWW91IG9waW5pb24vZmVlZGJhY2sgd2lsbCBiZSBy ZWFsbHkgYXBwcmVjaWF0ZWQhCj4+Cj4+IE1vYWNpcgo+Pgo+PiBfX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fXwo+PiBVc2VycyBtYWlsaW5nIGxpc3QKPj4gVXNl cnNAb3ZpcnQub3JnIDxtYWlsdG86VXNlcnNAb3ZpcnQub3JnPgo+PiBodHRwOi8vbGlzdHMub3Zp cnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMKPgo+Cj4gX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX18KPiBVc2VycyBtYWlsaW5nIGxpc3QKPiBVc2Vyc0Bv dmlydC5vcmcKPiBodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMK CgotLS0tLS0tLS0tLS0tLUVDREYxMjY3ODVCQjdBMTlEOUZDQkFGNQpDb250ZW50LVR5cGU6IHRl eHQvaHRtbDsgY2hhcnNldD11dGYtOApDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiA4Yml0Cgo8 aHRtbD4KICA8aGVhZD4KICAgIDxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVu dD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4KICA8L2hlYWQ+CiAgPGJvZHkgYmdjb2xvcj0i I0ZGRkZGRiIgdGV4dD0iIzAwMDAwMCI+CiAgICA8cD5Gb3IgYW55IFJBSUQgNSBvciA2IGNvbmZp Z3VyYXRpb24gSSBub3JtYWxseSBmb2xsb3cgYSBzaW1wbGUgZ29sZAogICAgICBydWxlIHdoaWNo IGdhdmUgZ29vZCByZXN1bHRzIHNvIGZhcjo8YnI+CiAgICAgIC0gdXAgdG8gNCBkaXNrcyBSQUlE IDU8YnI+CiAgICAgIC0gNSBvciBtb3JlIGRpc2tzIFJBSUQgNjwvcD4KICAgIDxwPkhvd2V2ZXIg SSBkaWRuJ3QgcmVhbGx5IHVuZGVyc3RhbmQgd2VsbCB0aGUgcmVjb21tZW5kYXRpb24gdG8gdXNl CiAgICAgIGFueSBSQUlEIHdpdGggR2x1c3RlckZTLiBJIGFsd2F5cyB0aG91Z2h0IHRoYXQgR2x1 c3RlRlMgbGlrZXMgdG8KICAgICAgd29yayBpbiBKQk9EIG1vZGUgYW5kIGNvbnRyb2wgdGhlIGRp c2tzIChicmlja3MpIGRpcmVjdGxseSBzbyB5b3UKICAgICAgY2FuIGNyZWF0ZSB3aGF0ZXZlciBk aXN0cmlidXRpb24gcnVsZSB5b3Ugd2lzaCwgYW5kIGlmIGEgc2luZ2xlCiAgICAgIGRpc2sgZmFp bHMgeW91IGp1c3QgcmVwbGFjZSBpdCBhbmQgd2hpY2ggb2J2aW91c2x5IGhhdmUgdGhlIGRhdGEK ICAgICAgcmVwbGljYXRlZCBmcm9tIGFub3RoZXIuIFRoZSBvbmx5IGRvd25zaWRlIG9mIHVzaW5n IGluIHRoaXMgd2F5IGlzCiAgICAgIHRoYXQgdGhlIHJlcGxpY2F0aW9uIGRhdGEgd2lsbCBiZSBm bG93IGFjY3Jvc3MgYWxsIHNlcnZlcnMgYnV0CiAgICAgIHRoYXQgaXMgbm90IG11Y2ggYSBiaWcg aXNzdWUuPC9wPgogICAgPHA+QW55b25lIGNhbiBlbGFib3JhdGUgYWJvdXQgVXNpbmcgUkFJRCAr IEdsdXN0ZXJGUyBhbmQgSkJPRCArCiAgICAgIEdsdXN0ZXJGUy48L3A+CiAgICA8cD5UaGFua3M8 YnI+CiAgICAgIFJlZ2FyZHM8YnI+CiAgICAgIEZlcm5hbmRvPGJyPgogICAgPC9wPgogICAgPGJy PgogICAgPGRpdiBjbGFzcz0ibW96LWNpdGUtcHJlZml4Ij5PbiAwNy8wOC8yMDE3IDAzOjQ2LCBE ZXZpbiBBY29zdGEKICAgICAgd3JvdGU6PGJyPgogICAgPC9kaXY+CiAgICA8YmxvY2txdW90ZSB0 eXBlPSJjaXRlIgpjaXRlPSJtaWQ6Q0FOQ0dLRXA0WEdzMFUrUXM3OGVFbXFDTnR2cExZLUF6amI1 RGNHaFo5eWlLVEJFRWZ3QG1haWwuZ21haWwuY29tIj4KICAgICAgPHN0eWxlPmJvZHl7Zm9udC1m YW1pbHk6SGVsdmV0aWNhLEFyaWFsO2ZvbnQtc2l6ZToxM3B4fTwvc3R5bGU+CiAgICAgIDxkaXYg aWQ9ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJjb2xvcjpyZ2IoMCwwLDApO21hcmdpbjowcHgi Pjxmb250CiAgICAgICAgICBmYWNlPSJJbnB1dCBNb25vIj48YnI+CiAgICAgICAgPC9mb250Pjwv ZGl2PgogICAgICA8ZGl2IGlkPSJibG9vcF9jdXN0b21mb250IiBzdHlsZT0iY29sb3I6cmdiKDAs MCwwKTttYXJnaW46MHB4Ij48Zm9udAogICAgICAgICAgZmFjZT0iSW5wdXQgTW9ubyI+TW9hY2ly LDwvZm9udD48L2Rpdj4KICAgICAgPGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9ImNv bG9yOnJnYigwLDAsMCk7bWFyZ2luOjBweCI+PGZvbnQKICAgICAgICAgIGZhY2U9IklucHV0IE1v bm8iPjxicj4KICAgICAgICA8L2ZvbnQ+PC9kaXY+CiAgICAgIDxkaXYgaWQ9ImJsb29wX2N1c3Rv bWZvbnQiIHN0eWxlPSJjb2xvcjpyZ2IoMCwwLDApO21hcmdpbjowcHgiPjxmb250CiAgICAgICAg ICBmYWNlPSJJbnB1dCBNb25vIj5JIGhhdmUgcmVjZW50bHkgaW5zdGFsbGVkIG11bHRpcGxlIFJl ZCBIYXQKICAgICAgICAgIFZpcnR1YWxpemF0aW9uIGhvc3RzIGZvciBzZXZlcmFsIGRpZmZlcmVu dCBjb21wYW5pZXMsIGFuZCBoYXZlCiAgICAgICAgICBkZWFsdCB3aXRoIHRoZSBSZWQgSGF0IFN1 cHBvcnQgVGVhbSBpbiBkZXB0aCBhYm91dCBvcHRpbWFsCiAgICAgICAgICBjb25maWd1cmF0aW9u IGluIHJlZ2FyZHMgdG8gc2V0dGluZyB1cCBHbHVzdGVyRlMgbW9zdAogICAgICAgICAgZWZmaWNp ZW50bHkgYW5kIEkgd2FudGVkIHRvIHNoYXJlIHdpdGggeW91IHdoYXQgSSBsZWFybmVkLjwvZm9u dD48L2Rpdj4KICAgICAgPGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9ImNvbG9yOnJn YigwLDAsMCk7bWFyZ2luOjBweCI+PGZvbnQKICAgICAgICAgIGZhY2U9IklucHV0IE1vbm8iPjxi cj4KICAgICAgICA8L2ZvbnQ+PC9kaXY+CiAgICAgIDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZvbnQi IHN0eWxlPSJjb2xvcjpyZ2IoMCwwLDApO21hcmdpbjowcHgiPjxmb250CiAgICAgICAgICBmYWNl PSJJbnB1dCBNb25vIj5JbiBnZW5lcmFsIFJlZCBIYXQgVmlydHVhbGl6YXRpb24gdGVhbQogICAg ICAgICAgZnJvd25zIHVwb24gdXNpbmcgZWFjaCBESVNLIG9mIHRoZSBzeXN0ZW0gYXMganVzdCBh IEpCT0QsIHN1cmUKICAgICAgICAgIHRoZXJlIGlzIHNvbWUgcHJvdGVjdGlvbiBieSBoYXZpbmcg dGhlIGRhdGEgcmVwbGljYXRlZCwKICAgICAgICAgIGhvd2V2ZXIsIHRoZSByZWNvbW1lbmRhdGlv biBpcyB0byB1c2UgUkFJRCA2IChwcmVmZXJyZWQpIG9yCiAgICAgICAgICBSQUlELTUsIG9yIGF0 IGxlYXN0IFJBSUQtMSBhdCB0aGUgdmVyeSBsZWFzdC48L2ZvbnQ+PC9kaXY+CiAgICAgIDxkaXYg aWQ9ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJjb2xvcjpyZ2IoMCwwLDApO21hcmdpbjowcHgi Pjxmb250CiAgICAgICAgICBmYWNlPSJJbnB1dCBNb25vIj48YnI+CiAgICAgICAgPC9mb250Pjwv ZGl2PgogICAgICA8ZGl2IGlkPSJibG9vcF9jdXN0b21mb250IiBzdHlsZT0ibWFyZ2luOjBweCI+ PGZvbnQgZmFjZT0iSW5wdXQKICAgICAgICAgIE1vbm8iPkhlcmUgaXMgdGhlwqBkaXJlY3QgcXVv dGUgZnJvbSBSZWQgSGF0IHdoZW4gSSBhc2tlZCBhYm91dAogICAgICAgICAgUkFJRCBhbmQgQnJp Y2tzOjwvZm9udD48L2Rpdj4KICAgICAgPGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9 Im1hcmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0CiAgICAgICAgICBNb25vIj48aT48YnI+CiAg ICAgICAgICA8L2k+PC9mb250PjwvZGl2PgogICAgICA8ZGl2IGlkPSJibG9vcF9jdXN0b21mb250 IiBzdHlsZT0ibWFyZ2luOjBweCI+PGZvbnQgZmFjZT0iSW5wdXQKICAgICAgICAgIE1vbm8iPjxp PiJBIHR5cGljYWwgR2x1c3RlciBjb25maWd1cmF0aW9uIHdvdWxkIHVzZSBSQUlECiAgICAgICAg ICAgIHVuZGVybmVhdGggdGhlIGJyaWNrcy4gUkFJRCA2IGlzIG1vc3QgdHlwaWNhbCBhcyBpdCBn aXZlcwogICAgICAgICAgICB5b3UgMiBkaXNrIGZhaWx1cmUgcHJvdGVjdGlvbiwgYnV0IFJBSUQg NSBjb3VsZCBiZSB1c2VkCiAgICAgICAgICAgIHRvby7CoE9uY2UgeW91IGhhdmUgdGhlIFJBSURl ZCBicmlja3MsIHlvdSdkIHRoZW4gYXBwbHkgdGhlCiAgICAgICAgICAgIGRlc2lyZWQgcmVwbGlj YXRpb24gb24gdG9wIG9mIHRoYXQuIFRoZSBtb3N0IHBvcHVsYXIgd2F5IG9mCiAgICAgICAgICAg IGRvaW5nIHRoaXMgd291bGQgYmUgZGlzdHJpYnV0ZWQgcmVwbGljYXRlZCB3aXRoIDJ4CiAgICAg ICAgICAgIHJlcGxpY2F0aW9uLsKgSW4gZ2VuZXJhbCB5b3UnbGwgZ2V0IGJldHRlciBwZXJmb3Jt YW5jZSB3aXRoCiAgICAgICAgICAgIGxhcmdlciBicmlja3MuIDEyIGRyaXZlcyBpcyBvZnRlbiBh IHN3ZWV0IHNwb3QuIEFub3RoZXIKICAgICAgICAgICAgb3B0aW9uIHdvdWxkIGJlIHRvIGNyZWF0 ZSBhIHNlcGFyYXRlIHRpZXIgdXNpbmcgYWxsIFNTROKAmXMu4oCdwqA8L2k+PC9mb250PjwvZGl2 PgogICAgICA8ZGl2IGlkPSJibG9vcF9jdXN0b21mb250IiBzdHlsZT0ibWFyZ2luOjBweCI+PGJy PgogICAgICA8L2Rpdj4KICAgICAgPGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9Im1h cmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0CiAgICAgICAgICBNb25vIj48aT5JbiBvcmRlciB0 byBTU0TCoHRpZXJpbmcgZnJvbSBteSB1bmRlcnN0YW5kaW5nIHlvdQogICAgICAgICAgICB3b3Vs ZCBuZWVkIDEgeCBOVk1lIGRyaXZlIGluIGVhY2ggc2VydmVyLCBvciA0IHggU1NEIGhvdAogICAg ICAgICAgICB0aWVyIChpdCBuZWVkcyB0byBiZSBkaXN0cmlidXRlZCwgcmVwbGljYXRlZCBmb3Ig dGhlIGhvdAogICAgICAgICAgICB0aWVyIGlmIG5vdCB1c2luZyBOVk1FKS4gU28gd2l0aCB5b3Ug b25seSBoYXZpbmcgMSBTU0QgZHJpdmUKICAgICAgICAgICAgaW4gZWFjaCBzZXJ2ZXIswqBJ4oCZ ZCBzdWdnZXN0IG1heWJlIGxvb2tpbmcgaW50byB0aGUgTlZNRQogICAgICAgICAgICBvcHRpb24u wqA8L2k+PC9mb250PjwvZGl2PgogICAgICA8ZGl2IGlkPSJibG9vcF9jdXN0b21mb250IiBzdHls ZT0ibWFyZ2luOjBweCI+PGZvbnQgZmFjZT0iSW5wdXQKICAgICAgICAgIE1vbm8iPjxpPjxicj4K ICAgICAgICAgIDwvaT48L2ZvbnQ+PC9kaXY+CiAgICAgIDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZv bnQiIHN0eWxlPSJtYXJnaW46MHB4Ij48Zm9udCBmYWNlPSJJbnB1dAogICAgICAgICAgTW9ubyI+ PGk+U2luY2UgeW91ciB1c2luZyBvbmx5IDMtc2VydmVycywgd2hhdCBJ4oCZZCBwcm9iYWJseQog ICAgICAgICAgICBzdWdnZXN0IGlzIHRvIGRvICgyIFJlcGxpY2FzICsgQXJiaXRlciBOb2RlKSwg dGhpcyBzZXR1cAogICAgICAgICAgICBhY3R1YWxseSBkb2VzbuKAmXQgcmVxdWlyZSB0aGUgM3Jk IHNlcnZlciB0byBoYXZlIGJpZyBkcml2ZXMKICAgICAgICAgICAgYXQgYWxsIGFzIGl0IG9ubHkg c3RvcmVzIG1ldGEtZGF0YSBhYm91dCB0aGUgZmlsZXMgYW5kIG5vdAogICAgICAgICAgICBhY3R1 YWxseSBhIGZ1bGwgY29weS7CoDwvaT48L2ZvbnQ+PC9kaXY+CiAgICAgIDxkaXYgaWQ9ImJsb29w X2N1c3RvbWZvbnQiIHN0eWxlPSJtYXJnaW46MHB4Ij48Zm9udCBmYWNlPSJJbnB1dAogICAgICAg ICAgTW9ubyI+PGk+PGJyPgogICAgICAgICAgPC9pPjwvZm9udD48L2Rpdj4KICAgICAgPGRpdiBp ZD0iYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9Im1hcmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0 CiAgICAgICAgICBNb25vIj48aT5QbGVhc2Ugc2VlIHRoZSBhdHRhY2hlZCBkb2N1bWVudCB0aGF0 IHdhcyBnaXZlbiB0byBtZQogICAgICAgICAgICBieSBSZWQgSGF0IHRvIGdldCBtb3JlIGluZm9y bWF0aW9uIG9uIHRoaXMuIEhvcGUgdGhpcwogICAgICAgICAgICBpbmZvcm1hdGlvbiBoZWxwcyB5 b3UuPC9pPjwvZm9udD48L2Rpdj4KICAgICAgPGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9udCIgc3R5 bGU9Im1hcmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0CiAgICAgICAgICBNb25vIj48aT48YnI+ CiAgICAgICAgICA8L2k+PC9mb250PjwvZGl2PgogICAgICA8YnI+CiAgICAgIDxkaXYgaWQ9ImJs b29wX3NpZ25fMTUwMjA4NzM3NjcyNTQ2OTE4NCIgY2xhc3M9ImJsb29wX3NpZ24iPjxzcGFuCiAg ICAgICAgICBzdHlsZT0iZm9udC1mYW1pbHk6J2hlbHZldGljYSBOZXVlJyxoZWx2ZXRpY2E7Zm9u dC1zaXplOjE0cHgiPi0tPC9zcGFuPjxicgogICAgICAgICAgc3R5bGU9ImZvbnQtZmFtaWx5Oido ZWx2ZXRpY2EgTmV1ZScsaGVsdmV0aWNhO2ZvbnQtc2l6ZToxNHB4Ij4KICAgICAgICA8ZGl2IGNs YXNzPSJnbWFpbF9zaWduYXR1cmUiIHN0eWxlPSJmb250LWZhbWlseTonaGVsdmV0aWNhCiAgICAg ICAgICBOZXVlJyxoZWx2ZXRpY2E7Zm9udC1zaXplOjE0cHgiPgogICAgICAgICAgPGRpdiBkaXI9 Imx0ciI+CiAgICAgICAgICAgIDxkaXY+PGJyPgogICAgICAgICAgICA8L2Rpdj4KICAgICAgICAg ICAgPGRpdj5EZXZpbiBBY29zdGEsIFJIQ0EsIFJIVkNBPC9kaXY+CiAgICAgICAgICAgIDxkaXY+ UmVkIEhhdCBDZXJ0aWZpZWQgQXJjaGl0ZWN0PC9kaXY+CiAgICAgICAgICA8L2Rpdj4KICAgICAg ICA8L2Rpdj4KICAgICAgPC9kaXY+CiAgICAgIDxicj4KICAgICAgPHAgY2xhc3M9ImFpcm1haWxf b24iPk9uIEF1Z3VzdCA2LCAyMDE3IGF0IDc6Mjk6MjkgUE0sIE1vYWNpcgogICAgICAgIEZlcnJl aXJhICg8YSBocmVmPSJtYWlsdG86bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb20iCiAgICAgICAg ICBtb3otZG8tbm90LXNlbmQ9InRydWUiPm1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY29tPC9hPikg d3JvdGU6PC9wPgogICAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIiBjbGFzcz0iY2xlYW5fYnEi PjxzcGFuPgogICAgICAgICAgPGRpdiBkaXI9Imx0ciI+CiAgICAgICAgICAgIDxkaXY+CiAgICAg ICAgICAgICAgPHRpdGxlPjwvdGl0bGU+CiAgICAgICAgICAgICAgPGRpdiBpZD0iZGl2dGFnZGVm YXVsdHdyYXBwZXIiCnN0eWxlPSJmb250LXNpemU6MTJwdDtjb2xvcjojMDAwMDAwO2ZvbnQtZmFt aWx5OkNhbGlicmksSGVsdmV0aWNhLHNhbnMtc2VyaWYiCiAgICAgICAgICAgICAgICBkaXI9Imx0 ciI+CiAgICAgICAgICAgICAgICA8cD48c3Bhbj5JIGFtIHdpbGxpbmcgdG8gYXNzZW1ibGUgYSBv VmlydCAicG9kIiwgbWFkZSBvZgogICAgICAgICAgICAgICAgICAgIDMgc2VydmVycywKICAgICAg ICAgICAgICAgICAgICBlYWNoIHdpdGggMiBDUFUgc29ja2V0cyBvZiAxMiBjb3JlcywgMjU2R0Ig UkFNLCA3CiAgICAgICAgICAgICAgICAgICAgSEREIDEwSywgMSBTU0QuCiAgICAgICAgICAgICAg ICAgICAgVGhlIGlkZWEgaXMgdG8gdXNlIEdsdXN0ZXJGUyB0byBwcm92aWRlIEhBIGZvciB0aGUK ICAgICAgICAgICAgICAgICAgICBWTXMuIFRoZSAzCiAgICAgICAgICAgICAgICAgICAgc2VydmVy cyBoYXZlIGEgZHVhbCA0MEdiIE5JQyBhbmQgYSBkdWFsIDEwR2IgTklDLiBTbwogICAgICAgICAg ICAgICAgICAgIG15IGludGVudGlvbgogICAgICAgICAgICAgICAgICAgIGlzIHRvIGNyZWF0ZSBh IGxvb3AgbGlrZSBhIHNlcnZlciB0cmlhbmdsZSB1c2luZyB0aGUKICAgICAgICAgICAgICAgICAg ICA0MEdiIE5JQ3MgZm9yCiAgICAgICAgICAgICAgICAgICAgdmlydHVhbGl6YXRpb24gZmlsZXMg KFZNcyAucWNvdzIpIGFjY2VzcyBhbmQgdG8gbW92ZQogICAgICAgICAgICAgICAgICAgIFZNcyBh cm91bmQgdGhlCiAgICAgICAgICAgICAgICAgICAgcG9kIChlYXN0IC93ZXN0IHRyYWZmaWMpIHdo aWxlIHVzaW5nIHRoZSAxMEdiCiAgICAgICAgICAgICAgICAgICAgaW50ZXJmYWNlcyBmb3IgZ2l2 aW5nCiAgICAgICAgICAgICAgICAgICAgc2VydmljZXMgdG8gdGhlIG91dHNpZGUgd29ybGQgKG5v cnRoL3NvdXRoIHRyYWZmaWMpLjwvc3Bhbj48L3A+CiAgICAgICAgICAgICAgICA8cD48YnI+CiAg ICAgICAgICAgICAgICA8L3A+CiAgICAgICAgICAgICAgICA8cD5UaGlzIHNhaWQsIG15IGZpcnN0 IHF1ZXN0aW9uIGlzOiBIb3cgc2hvdWxkIEkgZGVwbG95CiAgICAgICAgICAgICAgICAgIEdsdXN0 ZXJGUwogICAgICAgICAgICAgICAgICBpbiBzdWNoIG9WaXJ0IHNjZW5hcmlvPyBNeSBxdWVzdGlv bnMgYXJlOjwvcD4KICAgICAgICAgICAgICAgIDxwPjxicj4KICAgICAgICAgICAgICAgIDwvcD4K ICAgICAgICAgICAgICAgIDxwPjEgLSBTaG91bGQgSSBjcmVhdGUgMyBSQUlEIChpLmUuOiBSQUlE IDUpLCBvbmUgb24KICAgICAgICAgICAgICAgICAgZWFjaCBvVmlydAogICAgICAgICAgICAgICAg ICBub2RlLCBhbmQgdGhlbiBjcmVhdGUgYSBHbHVzdGVyRlMgdXNpbmcgdGhlbT88L3A+CiAgICAg ICAgICAgICAgICA8cD4yIC0gSW5zdGVhZCwgc2hvdWxkIEkgY3JlYXRlIGEgSkJPRCBhcnJheSBt YWRlIG9mIGFsbAogICAgICAgICAgICAgICAgICBzZXJ2ZXIncwogICAgICAgICAgICAgICAgICBk aXNrcz88L3A+CiAgICAgICAgICAgICAgICA8cD4zIC0gV2hhdCBpcyB0aGUgYmVzdCBHbHVzdGVy IGNvbmZpZ3VyYXRpb24gdG8gcHJvdmlkZQogICAgICAgICAgICAgICAgICBmb3IgSEEKICAgICAg ICAgICAgICAgICAgd2hpbGUgbm90IGNvbnN1bWluZyB0b28gbXVjaCBkaXNrIHNwYWNlPzxicj4K ICAgICAgICAgICAgICAgIDwvcD4KICAgICAgICAgICAgICAgIDxwPjQgLSBEb2VzIGEgb1ZpcnQg aHlwZXJ2aXNvciBwb2QgbGlrZSBJIGFtIHBsYW5uaW5nIHRvCiAgICAgICAgICAgICAgICAgIGJ1 aWxkLCBhbmQKICAgICAgICAgICAgICAgICAgdGhlIHZpcnR1YWxpemF0aW9uIGVudmlyb25tZW50 LCBiZW5lZml0cyBmcm9tIHRpZXJpbmcKICAgICAgICAgICAgICAgICAgd2hlbiB1c2luZyBhCiAg ICAgICAgICAgICAgICAgIFNTRCBkaXNrPyBBbmQgeWVzLCB3aWxsIEdsdXN0ZXIgZG8gaXQgYnkg ZGVmYXVsdCBvciBJCiAgICAgICAgICAgICAgICAgIGhhdmUgdG8KICAgICAgICAgICAgICAgICAg Y29uZmlndXJlIGl0IHRvIGRvIHNvPzwvcD4KICAgICAgICAgICAgICAgIDxwPjxicj4KICAgICAg ICAgICAgICAgIDwvcD4KICAgICAgICAgICAgICAgIDxwPkF0IHRoZSBib3R0b20gbGluZSwgd2hh dCBpcyB0aGUgZ29vZCBwcmFjdGljZSBmb3IKICAgICAgICAgICAgICAgICAgdXNpbmcKICAgICAg ICAgICAgICAgICAgR2x1c3RlckZTIGluIHNtYWxsIHBvZHMgZm9yIGVudGVycHJpc2VzPzxicj4K ICAgICAgICAgICAgICAgIDwvcD4KICAgICAgICAgICAgICAgIDxwPjxicj4KICAgICAgICAgICAg ICAgIDwvcD4KICAgICAgICAgICAgICAgIDxwPllvdSBvcGluaW9uL2ZlZWRiYWNrIHdpbGwgYmUg cmVhbGx5IGFwcHJlY2lhdGVkITwvcD4KICAgICAgICAgICAgICAgIDxwPk1vYWNpcjxicj4KICAg ICAgICAgICAgICAgIDwvcD4KICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICBfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwogICAgICAgICAgICAg IDxicj4KICAgICAgICAgICAgICBVc2VycyBtYWlsaW5nIGxpc3QKICAgICAgICAgICAgICA8YnI+ CiAgICAgICAgICAgICAgPGEgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgbW96LWRvLW5v dC1zZW5kPSJ0cnVlIj5Vc2Vyc0BvdmlydC5vcmc8L2E+CiAgICAgICAgICAgICAgPGJyPgogICAg ICAgICAgICAgIDxhIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5m by91c2VycyIKICAgICAgICAgICAgICAgIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+aHR0cDovL2xp c3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPgogICAgICAgICAgICAgIDxi cj4KICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICA8L2Rpdj4KICAgICAgICA8L3NwYW4+PC9i bG9ja3F1b3RlPgogICAgICA8YnI+CiAgICAgIDxmaWVsZHNldCBjbGFzcz0ibWltZUF0dGFjaG1l bnRIZWFkZXIiPjwvZmllbGRzZXQ+CiAgICAgIDxicj4KICAgICAgPHByZSB3cmFwPSIiPl9fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fClVzZXJzIG1haWxpbmcg bGlzdAo8YSBjbGFzcz0ibW96LXR4dC1saW5rLWFiYnJldmlhdGVkIiBocmVmPSJtYWlsdG86VXNl cnNAb3ZpcnQub3JnIj5Vc2Vyc0BvdmlydC5vcmc8L2E+CjxhIGNsYXNzPSJtb3otdHh0LWxpbmst ZnJlZXRleHQiIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91 c2VycyI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPgo8 L3ByZT4KICAgIDwvYmxvY2txdW90ZT4KICAgIDxicj4KICA8L2JvZHk+CjwvaHRtbD4KCi0tLS0t LS0tLS0tLS0tRUNERjEyNjc4NUJCN0ExOUQ5RkNCQUY1LS0K --===============2131844939483290490==-- From fernando.frediani at upx.com Mon Aug 7 13:08:37 2017 Content-Type: multipart/mixed; boundary="===============8747593560808751059==" MIME-Version: 1.0 From: FERNANDO FREDIANI To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Mon, 07 Aug 2017 10:08:32 -0300 Message-ID: <8dd168a0-d5ab-bdf4-5c5d-197909afc515@upx.com> In-Reply-To: VI1P190MB02856C6CA513853E8EDA2DB2C8B50@VI1P190MB0285.EURP190.PROD.OUTLOOK.COM --===============8747593560808751059== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------7999E232532992C2B4184FF7 Content-Type: text/plain; charset=3Dwindows-1252; format=3Dflowed Content-Transfer-Encoding: 7bit Moacir, I beleive for using the 3 servers directly connected to each = others without switch you have to have a Bridge on each server for every = 2 physical interfaces to allow the traffic passthrough in layer2 (Is it = possible to create this from the oVirt Engine Web Interface?). If your = ovirtmgmt network is separate from other (should really be) that should = be fine to do. Fernando On 07/08/2017 07:13, Moacir Ferreira wrote: > > Hi, in-line responses. > > > Thanks, > > Moacir > > > ------------------------------------------------------------------------ > *From:* Yaniv Kaul > *Sent:* Monday, August 7, 2017 7:42 AM > *To:* Moacir Ferreira > *Cc:* users(a)ovirt.org > *Subject:* Re: [ovirt-users] Good practices > > > On Sun, Aug 6, 2017 at 5:49 PM, Moacir Ferreira = > > wrot= e: > > I am willing to assemble a oVirt "pod", made of 3 servers, each > with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The > idea is to use GlusterFS to provide HA for the VMs. The 3 servers > have a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to > create a loop like a server triangle using the 40Gb NICs for > virtualization files (VMs .qcow2) access and to move VMs around > the pod (east /west traffic) while using the 10Gb interfaces for > giving services to the outside world (north/south traffic). > > > Very nice gear. How are you planning the network exactly? Without a = > switch, back-to-back? (sounds OK to me, just wanted to ensure this is = > what the 'dual' is used for). However, I'm unsure if you have the = > correct balance between the interface speeds (40g) and the disks (too = > many HDDs?). > > Moacir:The idea is to have a very high performance network for the = > distributed file system and to prevent bottlenecks when we move one VM = > from a node to another. Using 40Gb NICs I can just connect the servers = > back-to-back. In this case I don't need the expensive 40Gb switch, I = > get very high speed and no contention between north/south traffic with = > east/west. > > > This said, my first question is: How should I deploy GlusterFS in > such oVirt scenario? My questions are: > > > 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, > and then create a GlusterFS using them? > > I would assume RAID 1 for the operating system (you don't want a = > single point of failure there?) and the rest JBODs. The SSD will be = > used for caching, I reckon? (I personally would add more SSDs instead = > of HDDs, but it does depend on the disk sizes and your space requirements. > > Moacir: Yes, I agree that I need a RAID-1 for the OS. Now, generic = > JBOD or a JBOD assembled using RAID-5 "disks" createdby the server's = > disk controller? > > 2 - Instead, should I create a JBOD array made of all server's disks? > > 3 - What is the best Gluster configuration to provide for HA while > not consuming too much disk space? > > > Replica 2 + Arbiter sounds good to me. > Moacir:I agree, and that is what I am using. > > 4 - Does a oVirt hypervisor pod like I am planning to build, and > the virtualization environment, benefits from tiering when using a > SSD disk? And yes, will Gluster do it by default or I have to > configure it to do so? > > > Yes, I believe using lvmcache is the best way to go. > > Moacir: Are you sure? I say that because the qcow2 files will be > quite big. So if tiering is "file based" the SSD would have to be > very, very big unless Gluster tiering do it by "chunks of data". > > > At the bottom line, what is the good practice for using GlusterFS > in small pods for enterprises? > > > Don't forget jumbo frames. libgfapi (coming hopefully in 4.1.5). = > Sharding (enabled out of the box if you use a hyper-converged setup = > via gdeploy). > *Moacir:* Yes! This is another reason to have separate networks for = > north/south and east/west. In that way I can use the standard MTU on = > the 10Gb NICs and jumbo frames on the file/move 40Gb NICs. > > Y. > > > You opinion/feedback will be really appreciated! > > Moacir > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > > > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users --------------7999E232532992C2B4184FF7 Content-Type: text/html; charset=3Dwindows-1252 Content-Transfer-Encoding: 8bit
Moacir, I beleive for using the 3 servers directly connected to each others without switch you have to have a Bridge on each server for every 2 physical interfaces to allow the traffic passthrough in layer2 (Is it possible to create this from the oVirt Engine Web Interface?). If your ovirtmgmt network is separate from other (should really be) that should be fine to do.

Fernando

On 07/08/2017 07:13, Moacir Ferreira wrote:

Hi, in-line responses.

Thanks,

Moacir

From: Yaniv Kaul <ykaul(a)redhat.com>
Sent: Monday, August 7, 2017 7:42 AM
To: Moacir Ferreira
Cc: users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices
=C2=A0

On Sun, Aug 6, 2017 at 5:49 PM, Moacir Ferreira <moacirferr= eira(a)hotmail.com> wrote:

I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop like a server triangle using the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VMs around the pod (east /west traffic) while using the 10Gb interfaces for giving services to the outside world (north/south traffic).

Very nice gear. How are you planning the network exactly? Without a switch, back-to-back? (sounds OK to me, just wanted to ensure this is what the 'dual' is used for). However, I'm unsure if you have the correct balance between the interface speeds (40g) and the disks (too many HDDs?).

Moacir: The idea is to have a very high performance network for the distributed file system and to prevent bottlenecks when we move one VM from a node to another. Using 40Gb NICs I can just connect the servers back-to-back. In this case I don't need the expensive 40Gb switch, I get very high speed and no contention between north/south traffic with east/west.

This said, my first question is: How should I deploy GlusterFS in such oVirt scenario? My questions are:

1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then create a GlusterFS using them?

I would assume RAID 1 for the operating system (you don't want a single point of failure there?) and the rest JBODs. The SSD will be used for caching, I reckon? (I personally would add more SSDs instead of HDDs, but it does depend on the disk sizes and your space requirements.

Moacir: Yes, I agree that I need a RAID-1 for the OS. Now, generic JBOD or a JBOD assembled using RAID-5 "disks" created by the server's disk controller?=

=C2=A0

2 - Instead, should I create a JBOD array made of all server's disks?

3 - What is the best Gluster configuration to provide for HA while not consuming too much disk space?

Replica 2 + Arbiter sounds good to me.

Moacir: I agree, and that is what I am using.

4 - Does a oVirt hypervisor pod like I am planning to build, and the virtualization environment, benefits from tiering when using a SSD disk? And yes, will Gluster do it by default or I have to configure it to do so?

Yes, I believe using lvmcache is the best way to go.=C2=A0

Moacir: Are you sure? I say that because the qcow2 files will be quite big. So if tiering is "file based" the SSD would have to be very, very big unless Gluster tiering do it by "chunks of data".

At the bottom line, what is the good practice for using GlusterFS in small pods for enterprises?

Don't forget jumbo frames. libgfapi (coming hopefully in 4.1.5). Sharding (enabled out of the box if you use a hyper-converged setup via gdeploy).
Moacir: Yes! This is another reason to have separate networks for north/south and east/west. In that way I can use the standard MTU on the 10Gb NICs and jumbo frames on the file/move 40Gb NICs.

Y.

=C2=A0

You opinion/feedback will be really appreciated!

Moacir

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/= mailman/listinfo/users

_______________________________________________ Users mailing list Use= rs(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

--------------7999E232532992C2B4184FF7-- --===============8747593560808751059== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS03OTk5RTIzMjUzMjk5MkMyQjQxODRGRjcKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXdpbmRvd3MtMTI1MjsgZm9ybWF0PWZsb3dlZApDb250ZW50LVRyYW5zZmVyLUVuY29kaW5n OiA3Yml0CgpNb2FjaXIsIEkgYmVsZWl2ZSBmb3IgdXNpbmcgdGhlIDMgc2VydmVycyBkaXJlY3Rs eSBjb25uZWN0ZWQgdG8gZWFjaCAKb3RoZXJzIHdpdGhvdXQgc3dpdGNoIHlvdSBoYXZlIHRvIGhh dmUgYSBCcmlkZ2Ugb24gZWFjaCBzZXJ2ZXIgZm9yIGV2ZXJ5IAoyIHBoeXNpY2FsIGludGVyZmFj ZXMgdG8gYWxsb3cgdGhlIHRyYWZmaWMgcGFzc3Rocm91Z2ggaW4gbGF5ZXIyIChJcyBpdCAKcG9z c2libGUgdG8gY3JlYXRlIHRoaXMgZnJvbSB0aGUgb1ZpcnQgRW5naW5lIFdlYiBJbnRlcmZhY2U/ KS4gSWYgeW91ciAKb3ZpcnRtZ210IG5ldHdvcmsgaXMgc2VwYXJhdGUgZnJvbSBvdGhlciAoc2hv dWxkIHJlYWxseSBiZSkgdGhhdCBzaG91bGQgCmJlIGZpbmUgdG8gZG8uCgoKRmVybmFuZG8KCgpP biAwNy8wOC8yMDE3IDA3OjEzLCBNb2FjaXIgRmVycmVpcmEgd3JvdGU6Cj4KPiBIaSwgaW4tbGlu ZSByZXNwb25zZXMuCj4KPgo+IFRoYW5rcywKPgo+IE1vYWNpcgo+Cj4KPiAtLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0KPiAqRnJvbToqIFlhbml2IEthdWwgPHlrYXVsQHJlZGhhdC5jb20+Cj4gKlNlbnQ6KiBNb25k YXksIEF1Z3VzdCA3LCAyMDE3IDc6NDIgQU0KPiAqVG86KiBNb2FjaXIgRmVycmVpcmEKPiAqQ2M6 KiB1c2Vyc0BvdmlydC5vcmcKPiAqU3ViamVjdDoqIFJlOiBbb3ZpcnQtdXNlcnNdIEdvb2QgcHJh Y3RpY2VzCj4KPgo+IE9uIFN1biwgQXVnIDYsIDIwMTcgYXQgNTo0OSBQTSwgTW9hY2lyIEZlcnJl aXJhIAo+IDxtb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbSA8bWFpbHRvOm1vYWNpcmZlcnJlaXJh QGhvdG1haWwuY29tPj4gd3JvdGU6Cj4KPiAgICAgSSBhbSB3aWxsaW5nIHRvIGFzc2VtYmxlIGEg b1ZpcnQgInBvZCIsIG1hZGUgb2YgMyBzZXJ2ZXJzLCBlYWNoCj4gICAgIHdpdGggMiBDUFUgc29j a2V0cyBvZiAxMiBjb3JlcywgMjU2R0IgUkFNLCA3IEhERCAxMEssIDEgU1NELiBUaGUKPiAgICAg aWRlYSBpcyB0byB1c2UgR2x1c3RlckZTIHRvIHByb3ZpZGUgSEEgZm9yIHRoZSBWTXMuIFRoZSAz IHNlcnZlcnMKPiAgICAgaGF2ZSBhIGR1YWwgNDBHYiBOSUMgYW5kIGEgZHVhbCAxMEdiIE5JQy4g U28gbXkgaW50ZW50aW9uIGlzIHRvCj4gICAgIGNyZWF0ZSBhIGxvb3AgbGlrZSBhIHNlcnZlciB0 cmlhbmdsZSB1c2luZyB0aGUgNDBHYiBOSUNzIGZvcgo+ICAgICB2aXJ0dWFsaXphdGlvbiBmaWxl cyAoVk1zIC5xY293MikgYWNjZXNzIGFuZCB0byBtb3ZlIFZNcyBhcm91bmQKPiAgICAgdGhlIHBv ZCAoZWFzdCAvd2VzdCB0cmFmZmljKSB3aGlsZSB1c2luZyB0aGUgMTBHYiBpbnRlcmZhY2VzIGZv cgo+ICAgICBnaXZpbmcgc2VydmljZXMgdG8gdGhlIG91dHNpZGUgd29ybGQgKG5vcnRoL3NvdXRo IHRyYWZmaWMpLgo+Cj4KPiBWZXJ5IG5pY2UgZ2Vhci4gSG93IGFyZSB5b3UgcGxhbm5pbmcgdGhl IG5ldHdvcmsgZXhhY3RseT8gV2l0aG91dCBhIAo+IHN3aXRjaCwgYmFjay10by1iYWNrPyAoc291 bmRzIE9LIHRvIG1lLCBqdXN0IHdhbnRlZCB0byBlbnN1cmUgdGhpcyBpcyAKPiB3aGF0IHRoZSAn ZHVhbCcgaXMgdXNlZCBmb3IpLiBIb3dldmVyLCBJJ20gdW5zdXJlIGlmIHlvdSBoYXZlIHRoZSAK PiBjb3JyZWN0IGJhbGFuY2UgYmV0d2VlbiB0aGUgaW50ZXJmYWNlIHNwZWVkcyAoNDBnKSBhbmQg dGhlIGRpc2tzICh0b28gCj4gbWFueSBIRERzPykuCj4KPiBNb2FjaXI6VGhlIGlkZWEgaXMgdG8g aGF2ZSBhIHZlcnkgaGlnaCBwZXJmb3JtYW5jZSBuZXR3b3JrIGZvciB0aGUgCj4gZGlzdHJpYnV0 ZWQgZmlsZSBzeXN0ZW0gYW5kIHRvIHByZXZlbnQgYm90dGxlbmVja3Mgd2hlbiB3ZSBtb3ZlIG9u ZSBWTSAKPiBmcm9tIGEgbm9kZSB0byBhbm90aGVyLiBVc2luZyA0MEdiIE5JQ3MgSSBjYW4ganVz dCBjb25uZWN0IHRoZSBzZXJ2ZXJzIAo+IGJhY2stdG8tYmFjay4gSW4gdGhpcyBjYXNlIEkgZG9u J3QgbmVlZCB0aGUgZXhwZW5zaXZlIDQwR2Igc3dpdGNoLCBJIAo+IGdldCB2ZXJ5IGhpZ2ggc3Bl ZWQgYW5kIG5vIGNvbnRlbnRpb24gYmV0d2VlbiBub3J0aC9zb3V0aCB0cmFmZmljIHdpdGggCj4g ZWFzdC93ZXN0Lgo+Cj4KPiAgICAgVGhpcyBzYWlkLCBteSBmaXJzdCBxdWVzdGlvbiBpczogSG93 IHNob3VsZCBJIGRlcGxveSBHbHVzdGVyRlMgaW4KPiAgICAgc3VjaCBvVmlydCBzY2VuYXJpbz8g TXkgcXVlc3Rpb25zIGFyZToKPgo+Cj4gICAgIDEgLSBTaG91bGQgSSBjcmVhdGUgMyBSQUlEIChp LmUuOiBSQUlEIDUpLCBvbmUgb24gZWFjaCBvVmlydCBub2RlLAo+ICAgICBhbmQgdGhlbiBjcmVh dGUgYSBHbHVzdGVyRlMgdXNpbmcgdGhlbT8KPgo+IEkgd291bGQgYXNzdW1lIFJBSUQgMSBmb3Ig dGhlIG9wZXJhdGluZyBzeXN0ZW0gKHlvdSBkb24ndCB3YW50IGEgCj4gc2luZ2xlIHBvaW50IG9m IGZhaWx1cmUgdGhlcmU/KSBhbmQgdGhlIHJlc3QgSkJPRHMuIFRoZSBTU0Qgd2lsbCBiZSAKPiB1 c2VkIGZvciBjYWNoaW5nLCBJIHJlY2tvbj8gKEkgcGVyc29uYWxseSB3b3VsZCBhZGQgbW9yZSBT U0RzIGluc3RlYWQgCj4gb2YgSEREcywgYnV0IGl0IGRvZXMgZGVwZW5kIG9uIHRoZSBkaXNrIHNp emVzIGFuZCB5b3VyIHNwYWNlIHJlcXVpcmVtZW50cy4KPgo+IE1vYWNpcjogWWVzLCBJIGFncmVl IHRoYXQgSSBuZWVkIGEgUkFJRC0xIGZvciB0aGUgT1MuIE5vdywgZ2VuZXJpYyAKPiBKQk9EIG9y IGEgSkJPRCBhc3NlbWJsZWQgdXNpbmcgUkFJRC01ICJkaXNrcyIgY3JlYXRlZGJ5IHRoZSBzZXJ2 ZXIncyAKPiBkaXNrIGNvbnRyb2xsZXI/Cj4KPiAgICAgMiAtIEluc3RlYWQsIHNob3VsZCBJIGNy ZWF0ZSBhIEpCT0QgYXJyYXkgbWFkZSBvZiBhbGwgc2VydmVyJ3MgZGlza3M/Cj4KPiAgICAgMyAt IFdoYXQgaXMgdGhlIGJlc3QgR2x1c3RlciBjb25maWd1cmF0aW9uIHRvIHByb3ZpZGUgZm9yIEhB IHdoaWxlCj4gICAgIG5vdCBjb25zdW1pbmcgdG9vIG11Y2ggZGlzayBzcGFjZT8KPgo+Cj4gUmVw bGljYSAyICsgQXJiaXRlciBzb3VuZHMgZ29vZCB0byBtZS4KPiBNb2FjaXI6SSBhZ3JlZSwgYW5k IHRoYXQgaXMgd2hhdCBJIGFtIHVzaW5nLgo+Cj4gICAgIDQgLSBEb2VzIGEgb1ZpcnQgaHlwZXJ2 aXNvciBwb2QgbGlrZSBJIGFtIHBsYW5uaW5nIHRvIGJ1aWxkLCBhbmQKPiAgICAgdGhlIHZpcnR1 YWxpemF0aW9uIGVudmlyb25tZW50LCBiZW5lZml0cyBmcm9tIHRpZXJpbmcgd2hlbiB1c2luZyBh Cj4gICAgIFNTRCBkaXNrPyBBbmQgeWVzLCB3aWxsIEdsdXN0ZXIgZG8gaXQgYnkgZGVmYXVsdCBv ciBJIGhhdmUgdG8KPiAgICAgY29uZmlndXJlIGl0IHRvIGRvIHNvPwo+Cj4KPiBZZXMsIEkgYmVs aWV2ZSB1c2luZyBsdm1jYWNoZSBpcyB0aGUgYmVzdCB3YXkgdG8gZ28uCj4KPiAgICAgTW9hY2ly OiBBcmUgeW91IHN1cmU/IEkgc2F5IHRoYXQgYmVjYXVzZSB0aGUgcWNvdzIgZmlsZXMgd2lsbCBi ZQo+ICAgICBxdWl0ZSBiaWcuIFNvIGlmIHRpZXJpbmcgaXMgImZpbGUgYmFzZWQiIHRoZSBTU0Qg d291bGQgaGF2ZSB0byBiZQo+ICAgICB2ZXJ5LCB2ZXJ5IGJpZyB1bmxlc3MgR2x1c3RlciB0aWVy aW5nIGRvIGl0IGJ5ICJjaHVua3Mgb2YgZGF0YSIuCj4KPgo+ICAgICBBdCB0aGUgYm90dG9tIGxp bmUsIHdoYXQgaXMgdGhlIGdvb2QgcHJhY3RpY2UgZm9yIHVzaW5nIEdsdXN0ZXJGUwo+ICAgICBp biBzbWFsbCBwb2RzIGZvciBlbnRlcnByaXNlcz8KPgo+Cj4gRG9uJ3QgZm9yZ2V0IGp1bWJvIGZy YW1lcy4gbGliZ2ZhcGkgKGNvbWluZyBob3BlZnVsbHkgaW4gNC4xLjUpLiAKPiBTaGFyZGluZyAo ZW5hYmxlZCBvdXQgb2YgdGhlIGJveCBpZiB5b3UgdXNlIGEgaHlwZXItY29udmVyZ2VkIHNldHVw IAo+IHZpYSBnZGVwbG95KS4KPiAqTW9hY2lyOiogWWVzISBUaGlzIGlzIGFub3RoZXIgcmVhc29u IHRvIGhhdmUgc2VwYXJhdGUgbmV0d29ya3MgZm9yIAo+IG5vcnRoL3NvdXRoIGFuZCBlYXN0L3dl c3QuIEluIHRoYXQgd2F5IEkgY2FuIHVzZSB0aGUgc3RhbmRhcmQgTVRVIG9uIAo+IHRoZSAxMEdi IE5JQ3MgYW5kIGp1bWJvIGZyYW1lcyBvbiB0aGUgZmlsZS9tb3ZlIDQwR2IgTklDcy4KPgo+IFku Cj4KPgo+ICAgICBZb3Ugb3Bpbmlvbi9mZWVkYmFjayB3aWxsIGJlIHJlYWxseSBhcHByZWNpYXRl ZCEKPgo+ICAgICBNb2FjaXIKPgo+Cj4gICAgIF9fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fCj4gICAgIFVzZXJzIG1haWxpbmcgbGlzdAo+ICAgICBVc2Vyc0Bv dmlydC5vcmcgPG1haWx0bzpVc2Vyc0BvdmlydC5vcmc+Cj4gICAgIGh0dHA6Ly9saXN0cy5vdmly dC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vycwo+ICAgICA8aHR0cDovL2xpc3RzLm92aXJ0Lm9y Zy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPgo+Cj4KPgo+Cj4gX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX18KPiBVc2VycyBtYWlsaW5nIGxpc3QKPiBVc2Vyc0Bv dmlydC5vcmcKPiBodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMK CgotLS0tLS0tLS0tLS0tLTc5OTlFMjMyNTMyOTkyQzJCNDE4NEZGNwpDb250ZW50LVR5cGU6IHRl eHQvaHRtbDsgY2hhcnNldD13aW5kb3dzLTEyNTIKQ29udGVudC1UcmFuc2Zlci1FbmNvZGluZzog OGJpdAoKPGh0bWw+CiAgPGhlYWQ+CiAgICA8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUi IGNvbnRlbnQ9InRleHQvaHRtbDsKICAgICAgY2hhcnNldD13aW5kb3dzLTEyNTIiPgogIDwvaGVh ZD4KICA8Ym9keSBiZ2NvbG9yPSIjRkZGRkZGIiB0ZXh0PSIjMDAwMDAwIj4KICAgIDxwPk1vYWNp ciwgSSBiZWxlaXZlIGZvciB1c2luZyB0aGUgMyBzZXJ2ZXJzIGRpcmVjdGx5IGNvbm5lY3RlZCB0 bwogICAgICBlYWNoIG90aGVycyB3aXRob3V0IHN3aXRjaCB5b3UgaGF2ZSB0byBoYXZlIGEgQnJp ZGdlIG9uIGVhY2gKICAgICAgc2VydmVyIGZvciBldmVyeSAyIHBoeXNpY2FsIGludGVyZmFjZXMg dG8gYWxsb3cgdGhlIHRyYWZmaWMKICAgICAgcGFzc3Rocm91Z2ggaW4gbGF5ZXIyIChJcyBpdCBw b3NzaWJsZSB0byBjcmVhdGUgdGhpcyBmcm9tIHRoZQogICAgICBvVmlydCBFbmdpbmUgV2ViIElu dGVyZmFjZT8pLiBJZiB5b3VyIG92aXJ0bWdtdCBuZXR3b3JrIGlzCiAgICAgIHNlcGFyYXRlIGZy b20gb3RoZXIgKHNob3VsZCByZWFsbHkgYmUpIHRoYXQgc2hvdWxkIGJlIGZpbmUgdG8gZG8uPC9w PgogICAgPHA+PGJyPgogICAgPC9wPgogICAgPHA+RmVybmFuZG88YnI+CiAgICA8L3A+CiAgICA8 YnI+CiAgICA8ZGl2IGNsYXNzPSJtb3otY2l0ZS1wcmVmaXgiPk9uIDA3LzA4LzIwMTcgMDc6MTMs IE1vYWNpciBGZXJyZWlyYQogICAgICB3cm90ZTo8YnI+CiAgICA8L2Rpdj4KICAgIDxibG9ja3F1 b3RlIHR5cGU9ImNpdGUiCmNpdGU9Im1pZDpWSTFQMTkwTUIwMjg1NkM2Q0E1MTM4NTNFOEVEQTJE QjJDOEI1MEBWSTFQMTkwTUIwMjg1LkVVUlAxOTAuUFJPRC5PVVRMT09LLkNPTSI+CiAgICAgIDxt ZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0idGV4dC9odG1sOwogICAgICAg IGNoYXJzZXQ9d2luZG93cy0xMjUyIj4KICAgICAgPHN0eWxlIHR5cGU9InRleHQvY3NzIiBzdHls ZT0iZGlzcGxheTpub25lOyI+PCEtLSBQIHttYXJnaW4tdG9wOjA7bWFyZ2luLWJvdHRvbTowO30g LS0+PC9zdHlsZT4KICAgICAgPGRpdiBpZD0iZGl2dGFnZGVmYXVsdHdyYXBwZXIiCnN0eWxlPSJm b250LXNpemU6MTJwdDtjb2xvcjojMDAwMDAwO2ZvbnQtZmFtaWx5OkNhbGlicmksSGVsdmV0aWNh LHNhbnMtc2VyaWY7IgogICAgICAgIGRpcj0ibHRyIj4KICAgICAgICA8cD5IaSwgaW4tbGluZSBy ZXNwb25zZXMuPGJyPgogICAgICAgIDwvcD4KICAgICAgICA8YnI+CiAgICAgICAgPHA+VGhhbmtz LDwvcD4KICAgICAgICA8cD5Nb2FjaXI8YnI+CiAgICAgICAgPC9wPgogICAgICAgIDxicj4KICAg ICAgICA8ZGl2IHN0eWxlPSJjb2xvcjogcmdiKDQ5LCA1NSwgNTcpOyI+CiAgICAgICAgICA8aHIg dGFiaW5kZXg9Ii0xIiBzdHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7IHdpZHRoOjk4JSI+CiAg ICAgICAgICA8ZGl2IGlkPSJkaXZScGx5RndkTXNnIiBkaXI9Imx0ciI+PGZvbnQgc3R5bGU9ImZv bnQtc2l6ZToxMXB0IgogICAgICAgICAgICAgIGNvbG9yPSIjMDAwMDAwIiBmYWNlPSJDYWxpYnJp LCBzYW5zLXNlcmlmIj48Yj5Gcm9tOjwvYj4KICAgICAgICAgICAgICBZYW5pdiBLYXVsIDxhIGNs YXNzPSJtb3otdHh0LWxpbmstcmZjMjM5NkUiIGhyZWY9Im1haWx0bzp5a2F1bEByZWRoYXQuY29t Ij4mbHQ7eWthdWxAcmVkaGF0LmNvbSZndDs8L2E+PGJyPgogICAgICAgICAgICAgIDxiPlNlbnQ6 PC9iPiBNb25kYXksIEF1Z3VzdCA3LCAyMDE3IDc6NDIgQU08YnI+CiAgICAgICAgICAgICAgPGI+ VG86PC9iPiBNb2FjaXIgRmVycmVpcmE8YnI+CiAgICAgICAgICAgICAgPGI+Q2M6PC9iPiA8YSBj bGFzcz0ibW96LXR4dC1saW5rLWFiYnJldmlhdGVkIiBocmVmPSJtYWlsdG86dXNlcnNAb3ZpcnQu b3JnIj51c2Vyc0BvdmlydC5vcmc8L2E+PGJyPgogICAgICAgICAgICAgIDxiPlN1YmplY3Q6PC9i PiBSZTogW292aXJ0LXVzZXJzXSBHb29kIHByYWN0aWNlczwvZm9udD4KICAgICAgICAgICAgPGRp dj6gPC9kaXY+CiAgICAgICAgICA8L2Rpdj4KICAgICAgICAgIDxkaXY+CiAgICAgICAgICAgIDxk aXYgZGlyPSJsdHIiPjxicj4KICAgICAgICAgICAgICA8ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+ PGJyPgogICAgICAgICAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPk9uIFN1biwgQXVn IDYsIDIwMTcgYXQgNTo0OSBQTSwKICAgICAgICAgICAgICAgICAgTW9hY2lyIEZlcnJlaXJhIDxz cGFuIGRpcj0ibHRyIj4KICAgICAgICAgICAgICAgICAgICAmbHQ7PGEgaHJlZj0ibWFpbHRvOm1v YWNpcmZlcnJlaXJhQGhvdG1haWwuY29tIgogICAgICAgICAgICAgICAgICAgICAgdGFyZ2V0PSJf YmxhbmsiIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb208 L2E+Jmd0Ozwvc3Bhbj4KICAgICAgICAgICAgICAgICAgd3JvdGU6PGJyPgogICAgICAgICAgICAg ICAgICA8YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAK ICAgICAgICAgICAgICAgICAgICAuOGV4OyBib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDsgcGFk ZGluZy1sZWZ0OjFleCI+CiAgICAgICAgICAgICAgICAgICAgPGRpdiBkaXI9Imx0ciI+CiAgICAg ICAgICAgICAgICAgICAgICA8ZGl2CiAgICAgICAgICAgICAgICAgICAgICAgIGlkPSJtXzU1MDk1 ODU4ODk1Njk2OTA3OTFkaXZ0YWdkZWZhdWx0d3JhcHBlciIKICAgICAgICAgICAgICAgICAgICAg ICAgZGlyPSJsdHIiIHN0eWxlPSJmb250LXNpemU6MTJwdDsgY29sb3I6IzAwMDAwMDsKICAgICAg ICAgICAgICAgICAgICAgICAgZm9udC1mYW1pbHk6Q2FsaWJyaSxIZWx2ZXRpY2Esc2Fucy1zZXJp ZiI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxwPjxzcGFuPkkgYW0gd2lsbGluZyB0byBhc3Nl bWJsZSBhIG9WaXJ0ICJwb2QiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgbWFkZSBvZiAz IHNlcnZlcnMsIGVhY2ggd2l0aCAyIENQVSBzb2NrZXRzCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICBvZiAxMiBjb3JlcywgMjU2R0IgUkFNLCA3IEhERCAxMEssIDEgU1NELgogICAgICAgICAg ICAgICAgICAgICAgICAgICAgVGhlIGlkZWEgaXMgdG8gdXNlIEdsdXN0ZXJGUyB0byBwcm92aWRl IEhBCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBmb3IgdGhlIFZNcy4gVGhlIDMgc2VydmVy cyBoYXZlIGEgZHVhbCA0MEdiCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBOSUMgYW5kIGEg ZHVhbCAxMEdiIE5JQy4gU28gbXkgaW50ZW50aW9uIGlzCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICB0byBjcmVhdGUgYSBsb29wIGxpa2UgYSBzZXJ2ZXIgdHJpYW5nbGUKICAgICAgICAgICAg ICAgICAgICAgICAgICAgIHVzaW5nIHRoZSA0MEdiIE5JQ3MgZm9yIHZpcnR1YWxpemF0aW9uIGZp bGVzCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAoVk1zIC5xY293MikgYWNjZXNzIGFuZCB0 byBtb3ZlIFZNcyBhcm91bmQKICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRoZSBwb2QgKGVh c3QgL3dlc3QgdHJhZmZpYykgd2hpbGUgdXNpbmcgdGhlCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAxMEdiIGludGVyZmFjZXMgZm9yIGdpdmluZyBzZXJ2aWNlcyB0byB0aGUKICAgICAgICAg ICAgICAgICAgICAgICAgICAgIG91dHNpZGUgd29ybGQgKG5vcnRoL3NvdXRoIHRyYWZmaWMpLjwv c3Bhbj48L3A+CiAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAg ICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgPC9ibG9ja3F1b3RlPgogICAgICAgICAgICAgICAg ICA8ZGl2Pjxicj4KICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgIDxk aXY+VmVyeSBuaWNlIGdlYXIuIEhvdyBhcmUgeW91IHBsYW5uaW5nIHRoZSBuZXR3b3JrCiAgICAg ICAgICAgICAgICAgICAgZXhhY3RseT8gV2l0aG91dCBhIHN3aXRjaCwgYmFjay10by1iYWNrPyAo c291bmRzIE9LCiAgICAgICAgICAgICAgICAgICAgdG8gbWUsIGp1c3Qgd2FudGVkIHRvIGVuc3Vy ZSB0aGlzIGlzIHdoYXQgdGhlICdkdWFsJwogICAgICAgICAgICAgICAgICAgIGlzIHVzZWQgZm9y KS4gSG93ZXZlciwgSSdtIHVuc3VyZSBpZiB5b3UgaGF2ZSB0aGUKICAgICAgICAgICAgICAgICAg ICBjb3JyZWN0IGJhbGFuY2UgYmV0d2VlbiB0aGUgaW50ZXJmYWNlIHNwZWVkcyAoNDBnKQogICAg ICAgICAgICAgICAgICAgIGFuZCB0aGUgZGlza3MgKHRvbyBtYW55IEhERHM/KS48L2Rpdj4KICAg ICAgICAgICAgICAgICAgPGRpdj48YnI+CiAgICAgICAgICAgICAgICAgICAgPHNwYW4gc3R5bGU9 ImNvbG9yOiByZ2IoMCwgMCwgMCk7Ij5NPC9zcGFuPjxzcGFuCiAgICAgICAgICAgICAgICAgICAg ICBzdHlsZT0iY29sb3I6IHJnYigwLCAwLCAwKTsiPm88L3NwYW4+PHNwYW4KICAgICAgICAgICAg ICAgICAgICAgIHN0eWxlPSJjb2xvcjogcmdiKDAsIDAsIDApOyI+YTwvc3Bhbj48c3BhbgogICAg ICAgICAgICAgICAgICAgICAgc3R5bGU9ImNvbG9yOiByZ2IoMCwgMCwgMCk7Ij5jPC9zcGFuPjxz cGFuCiAgICAgICAgICAgICAgICAgICAgICBzdHlsZT0iY29sb3I6IHJnYigwLCAwLCAwKTsiPmk8 L3NwYW4+PHNwYW4KICAgICAgICAgICAgICAgICAgICAgIHN0eWxlPSJjb2xvcjogcmdiKDAsIDAs IDApOyI+cjwvc3Bhbj48c3BhbgogICAgICAgICAgICAgICAgICAgICAgc3R5bGU9ImNvbG9yOiBy Z2IoMCwgMCwgMCk7Ij46PC9zcGFuPjxzcGFuCiAgICAgICAgICAgICAgICAgICAgICBzdHlsZT0i Y29sb3I6IHJnYigwLCAwLCAwKTsiPiBUaGUgaWRlYSBpcyB0byBoYXZlCiAgICAgICAgICAgICAg ICAgICAgICBhIHZlcnkgaGlnaCBwZXJmb3JtYW5jZSBuZXR3b3JrIGZvciB0aGUKICAgICAgICAg ICAgICAgICAgICAgIGRpc3RyaWJ1dGVkIGZpbGUgc3lzdGVtIGFuZCB0byBwcmV2ZW50IGJvdHRs ZW5lY2tzCiAgICAgICAgICAgICAgICAgICAgICB3aGVuIHdlIG1vdmUgb25lIFZNIGZyb20gYSBu b2RlIHRvIGFub3RoZXIuIFVzaW5nCiAgICAgICAgICAgICAgICAgICAgPC9zcGFuPjxzcGFuIHN0 eWxlPSJjb2xvcjogcmdiKDAsIDAsIDApOyI+NDBHYiBOSUNzCiAgICAgICAgICAgICAgICAgICAg ICBJIGNhbiBqdXN0IGNvbm5lY3QgdGhlIHNlcnZlcnMgYmFjay10by1iYWNrLiBJbgogICAgICAg ICAgICAgICAgICAgICAgdGhpcyBjYXNlIEkgZG9uJ3QgbmVlZCB0aGUgZXhwZW5zaXZlIDQwR2Ig c3dpdGNoLAogICAgICAgICAgICAgICAgICAgICAgSSBnZXQgdmVyeSBoaWdoIHNwZWVkIGFuZCBu byBjb250ZW50aW9uIGJldHdlZW4KICAgICAgICAgICAgICAgICAgICAgIG5vcnRoL3NvdXRoIHRy YWZmaWMgd2l0aCBlYXN0L3dlc3QuPC9zcGFuPjxicj4KICAgICAgICAgICAgICAgICAgICA8YnI+ CiAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICA8YmxvY2txdW90ZSBj bGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAKICAgICAgICAgICAgICAgICAg ICAuOGV4OyBib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDsgcGFkZGluZy1sZWZ0OjFleCI+CiAg ICAgICAgICAgICAgICAgICAgPGRpdiBkaXI9Imx0ciI+CiAgICAgICAgICAgICAgICAgICAgICA8 ZGl2CiAgICAgICAgICAgICAgICAgICAgICAgIGlkPSJtXzU1MDk1ODU4ODk1Njk2OTA3OTFkaXZ0 YWdkZWZhdWx0d3JhcHBlciIKICAgICAgICAgICAgICAgICAgICAgICAgZGlyPSJsdHIiIHN0eWxl PSJmb250LXNpemU6MTJwdDsgY29sb3I6IzAwMDAwMDsKICAgICAgICAgICAgICAgICAgICAgICAg Zm9udC1mYW1pbHk6Q2FsaWJyaSxIZWx2ZXRpY2Esc2Fucy1zZXJpZiI+CiAgICAgICAgICAgICAg ICAgICAgICAgIDxwPjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICA8c3Bhbj48L3NwYW4+ PC9wPgogICAgICAgICAgICAgICAgICAgICAgICA8cD5UaGlzIHNhaWQsIG15IGZpcnN0IHF1ZXN0 aW9uIGlzOiBIb3cgc2hvdWxkIEkKICAgICAgICAgICAgICAgICAgICAgICAgICBkZXBsb3kgR2x1 c3RlckZTIGluIHN1Y2ggb1ZpcnQgc2NlbmFyaW8/IE15CiAgICAgICAgICAgICAgICAgICAgICAg ICAgcXVlc3Rpb25zIGFyZTo8L3A+CiAgICAgICAgICAgICAgICAgICAgICAgIDxwPjxicj4KICAg ICAgICAgICAgICAgICAgICAgICAgPC9wPgogICAgICAgICAgICAgICAgICAgICAgICA8cD4xIC0g U2hvdWxkIEkgY3JlYXRlIDMgUkFJRCAoaS5lLjogUkFJRCA1KSwKICAgICAgICAgICAgICAgICAg ICAgICAgICBvbmUgb24gZWFjaCBvVmlydCBub2RlLCBhbmQgdGhlbiBjcmVhdGUgYQogICAgICAg ICAgICAgICAgICAgICAgICAgIEdsdXN0ZXJGUyB1c2luZyB0aGVtPzwvcD4KICAgICAgICAgICAg ICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAg ICAgICA8L2Jsb2NrcXVvdGU+CiAgICAgICAgICAgICAgICAgIDxkaXY+SSB3b3VsZCBhc3N1bWUg UkFJRCAxIGZvciB0aGUgb3BlcmF0aW5nIHN5c3RlbQogICAgICAgICAgICAgICAgICAgICh5b3Ug ZG9uJ3Qgd2FudCBhIHNpbmdsZSBwb2ludCBvZiBmYWlsdXJlIHRoZXJlPykKICAgICAgICAgICAg ICAgICAgICBhbmQgdGhlIHJlc3QgSkJPRHMuIFRoZSBTU0Qgd2lsbCBiZSB1c2VkIGZvcgogICAg ICAgICAgICAgICAgICAgIGNhY2hpbmcsIEkgcmVja29uPyAoSSBwZXJzb25hbGx5IHdvdWxkIGFk ZCBtb3JlIFNTRHMKICAgICAgICAgICAgICAgICAgICBpbnN0ZWFkIG9mIEhERHMsIGJ1dCBpdCBk b2VzIGRlcGVuZCBvbiB0aGUgZGlzawogICAgICAgICAgICAgICAgICAgIHNpemVzIGFuZCB5b3Vy IHNwYWNlIHJlcXVpcmVtZW50cy48YnI+CiAgICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAg ICAgICAgICAgICAgIDxzcGFuIHN0eWxlPSJjb2xvcjogcmdiKDAsIDAsIDApOyI+TW9hY2lyOiBZ ZXMsIEkKICAgICAgICAgICAgICAgICAgICAgIGFncmVlIHRoYXQgSSBuZWVkIGEgUkFJRC0xIGZv ciB0aGUgT1MuIE5vdywKICAgICAgICAgICAgICAgICAgICAgIGdlbmVyaWMgSkJPRCBvciBhIEpC T0QgYXNzZW1ibGVkIHVzaW5nIFJBSUQtNQogICAgICAgICAgICAgICAgICAgICAgImRpc2tzIiBj cmVhdGVkPC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjogcmdiKDAsCiAgICAgICAgICAgICAgICAg ICAgICAwLCAwKTsiPiBieSB0aGUgc2VydmVyJ3MgZGlzawogICAgICAgICAgICAgICAgICAgIDwv c3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6IHJnYigwLCAwLCAwKTsiPmNvbnRyb2xsZXI/PC9zcGFu Pjxicj4KICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgIDxzcGFuIHN0 eWxlPSJjb2xvcjogcmdiKDAsIDAsIDApOyI+PC9zcGFuPgogICAgICAgICAgICAgICAgICA8ZGl2 PqA8YnI+CiAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICA8YmxvY2tx dW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAKICAgICAgICAgICAg ICAgICAgICAuOGV4OyBib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDsgcGFkZGluZy1sZWZ0OjFl eCI+CiAgICAgICAgICAgICAgICAgICAgPGRpdiBkaXI9Imx0ciI+CiAgICAgICAgICAgICAgICAg ICAgICA8ZGl2CiAgICAgICAgICAgICAgICAgICAgICAgIGlkPSJtXzU1MDk1ODU4ODk1Njk2OTA3 OTFkaXZ0YWdkZWZhdWx0d3JhcHBlciIKICAgICAgICAgICAgICAgICAgICAgICAgZGlyPSJsdHIi IHN0eWxlPSJmb250LXNpemU6MTJwdDsgY29sb3I6IzAwMDAwMDsKICAgICAgICAgICAgICAgICAg ICAgICAgZm9udC1mYW1pbHk6Q2FsaWJyaSxIZWx2ZXRpY2Esc2Fucy1zZXJpZiI+CiAgICAgICAg ICAgICAgICAgICAgICAgIDxwPjIgLSBJbnN0ZWFkLCBzaG91bGQgSSBjcmVhdGUgYSBKQk9EIGFy cmF5CiAgICAgICAgICAgICAgICAgICAgICAgICAgbWFkZSBvZiBhbGwgc2VydmVyJ3MgZGlza3M/ PC9wPgogICAgICAgICAgICAgICAgICAgICAgICA8cD4zIC0gV2hhdCBpcyB0aGUgYmVzdCBHbHVz dGVyIGNvbmZpZ3VyYXRpb24gdG8KICAgICAgICAgICAgICAgICAgICAgICAgICBwcm92aWRlIGZv ciBIQSB3aGlsZSBub3QgY29uc3VtaW5nIHRvbyBtdWNoCiAgICAgICAgICAgICAgICAgICAgICAg ICAgZGlzayBzcGFjZT88YnI+CiAgICAgICAgICAgICAgICAgICAgICAgIDwvcD4KICAgICAgICAg ICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAg ICAgICAgICA8L2Jsb2NrcXVvdGU+CiAgICAgICAgICAgICAgICAgIDxkaXY+PGJyPgogICAgICAg ICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgPGRpdj5SZXBsaWNhIDIgKyBBcmJp dGVyIHNvdW5kcyBnb29kIHRvIG1lLjwvZGl2PgogICAgICAgICAgICAgICAgICA8ZGl2PjxzcGFu IHN0eWxlPSJjb2xvcjogcmdiKDAsIDAsIDApOyI+TTwvc3Bhbj48c3BhbgogICAgICAgICAgICAg ICAgICAgICAgc3R5bGU9ImNvbG9yOiByZ2IoMCwgMCwgMCk7Ij5vPC9zcGFuPjxzcGFuCiAgICAg ICAgICAgICAgICAgICAgICBzdHlsZT0iY29sb3I6IHJnYigwLCAwLCAwKTsiPmE8L3NwYW4+PHNw YW4KICAgICAgICAgICAgICAgICAgICAgIHN0eWxlPSJjb2xvcjogcmdiKDAsIDAsIDApOyI+Yzwv c3Bhbj48c3BhbgogICAgICAgICAgICAgICAgICAgICAgc3R5bGU9ImNvbG9yOiByZ2IoMCwgMCwg MCk7Ij5pPC9zcGFuPjxzcGFuCiAgICAgICAgICAgICAgICAgICAgICBzdHlsZT0iY29sb3I6IHJn YigwLCAwLCAwKTsiPnI8L3NwYW4+PHNwYW4KICAgICAgICAgICAgICAgICAgICAgIHN0eWxlPSJj b2xvcjogcmdiKDAsIDAsIDApOyI+Ojwvc3Bhbj48c3BhbgogICAgICAgICAgICAgICAgICAgICAg c3R5bGU9ImNvbG9yOiByZ2IoMCwgMCwgMCk7Ij4KICAgICAgICAgICAgICAgICAgICA8L3NwYW4+ PHNwYW4gc3R5bGU9ImNvbG9yOiByZ2IoMCwgMCwgMCk7Ij5JIGFncmVlLAogICAgICAgICAgICAg ICAgICAgICAgYW5kIHRoYXQgaXMgd2hhdCBJIGFtIHVzaW5nLjwvc3Bhbj48YnI+CiAgICAgICAg ICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAg ICAgPGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAwCiAg ICAgICAgICAgICAgICAgICAgLjhleDsgYm9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7IHBhZGRp bmctbGVmdDoxZXgiPgogICAgICAgICAgICAgICAgICAgIDxkaXYgZGlyPSJsdHIiPgogICAgICAg ICAgICAgICAgICAgICAgPGRpdgogICAgICAgICAgICAgICAgICAgICAgICBpZD0ibV81NTA5NTg1 ODg5NTY5NjkwNzkxZGl2dGFnZGVmYXVsdHdyYXBwZXIiCiAgICAgICAgICAgICAgICAgICAgICAg IGRpcj0ibHRyIiBzdHlsZT0iZm9udC1zaXplOjEycHQ7IGNvbG9yOiMwMDAwMDA7CiAgICAgICAg ICAgICAgICAgICAgICAgIGZvbnQtZmFtaWx5OkNhbGlicmksSGVsdmV0aWNhLHNhbnMtc2VyaWYi PgogICAgICAgICAgICAgICAgICAgICAgICA8cD40IC0gRG9lcyBhIG9WaXJ0IGh5cGVydmlzb3Ig cG9kIGxpa2UgSSBhbQogICAgICAgICAgICAgICAgICAgICAgICAgIHBsYW5uaW5nIHRvIGJ1aWxk LCBhbmQgdGhlIHZpcnR1YWxpemF0aW9uCiAgICAgICAgICAgICAgICAgICAgICAgICAgZW52aXJv bm1lbnQsIGJlbmVmaXRzIGZyb20gdGllcmluZyB3aGVuIHVzaW5nCiAgICAgICAgICAgICAgICAg ICAgICAgICAgYSBTU0QgZGlzaz8gQW5kIHllcywgd2lsbCBHbHVzdGVyIGRvIGl0IGJ5CiAgICAg ICAgICAgICAgICAgICAgICAgICAgZGVmYXVsdCBvciBJIGhhdmUgdG8gY29uZmlndXJlIGl0IHRv IGRvIHNvPzwvcD4KICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAg ICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICA8L2Jsb2NrcXVvdGU+CiAgICAgICAgICAgICAg ICAgIDxkaXY+PGJyPgogICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAg PGRpdj5ZZXMsIEkgYmVsaWV2ZSB1c2luZyBsdm1jYWNoZSBpcyB0aGUgYmVzdCB3YXkgdG8KICAg ICAgICAgICAgICAgICAgICBnby6gPC9kaXY+CiAgICAgICAgICAgICAgICAgIDxibG9ja3F1b3Rl IGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1hcmdpbjowIDAgMAogICAgICAgICAgICAgICAg ICAgIC44ZXg7IGJvcmRlci1sZWZ0OjFweCAjY2NjIHNvbGlkOyBwYWRkaW5nLWxlZnQ6MWV4Ij4K ICAgICAgICAgICAgICAgICAgICA8ZGl2IGRpcj0ibHRyIj4KICAgICAgICAgICAgICAgICAgICAg IDxkaXYKICAgICAgICAgICAgICAgICAgICAgICAgaWQ9Im1fNTUwOTU4NTg4OTU2OTY5MDc5MWRp dnRhZ2RlZmF1bHR3cmFwcGVyIgogICAgICAgICAgICAgICAgICAgICAgICBkaXI9Imx0ciIgc3R5 bGU9ImZvbnQtc2l6ZToxMnB0OyBjb2xvcjojMDAwMDAwOwogICAgICAgICAgICAgICAgICAgICAg ICBmb250LWZhbWlseTpDYWxpYnJpLEhlbHZldGljYSxzYW5zLXNlcmlmIj4KICAgICAgICAgICAg ICAgICAgICAgICAgPHA+TW9hY2lyOiBBcmUgeW91IHN1cmU/IEkgc2F5IHRoYXQgYmVjYXVzZSB0 aGUKICAgICAgICAgICAgICAgICAgICAgICAgICBxY293MiBmaWxlcyB3aWxsIGJlIHF1aXRlIGJp Zy4gU28gaWYgdGllcmluZwogICAgICAgICAgICAgICAgICAgICAgICAgIGlzICJmaWxlIGJhc2Vk IiB0aGUgU1NEIHdvdWxkIGhhdmUgdG8gYmUgdmVyeSwKICAgICAgICAgICAgICAgICAgICAgICAg ICB2ZXJ5IGJpZyB1bmxlc3MgR2x1c3RlciB0aWVyaW5nIGRvIGl0IGJ5CiAgICAgICAgICAgICAg ICAgICAgICAgICAgImNodW5rcyBvZiBkYXRhIi48YnI+CiAgICAgICAgICAgICAgICAgICAgICAg IDwvcD4KICAgICAgICAgICAgICAgICAgICAgICAgPHA+PGJyPgogICAgICAgICAgICAgICAgICAg ICAgICA8L3A+CiAgICAgICAgICAgICAgICAgICAgICAgIDxwPkF0IHRoZSBib3R0b20gbGluZSwg d2hhdCBpcyB0aGUgZ29vZCBwcmFjdGljZQogICAgICAgICAgICAgICAgICAgICAgICAgIGZvciB1 c2luZyBHbHVzdGVyRlMgaW4gc21hbGwgcG9kcyBmb3IKICAgICAgICAgICAgICAgICAgICAgICAg ICBlbnRlcnByaXNlcz88YnI+CiAgICAgICAgICAgICAgICAgICAgICAgIDwvcD4KICAgICAgICAg ICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAg ICAgICAgICA8L2Jsb2NrcXVvdGU+CiAgICAgICAgICAgICAgICAgIDxkaXY+PGJyPgogICAgICAg ICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgPGRpdj5Eb24ndCBmb3JnZXQganVt Ym8gZnJhbWVzLiBsaWJnZmFwaSAoY29taW5nCiAgICAgICAgICAgICAgICAgICAgaG9wZWZ1bGx5 IGluIDQuMS41KS4gU2hhcmRpbmcgKGVuYWJsZWQgb3V0IG9mIHRoZQogICAgICAgICAgICAgICAg ICAgIGJveCBpZiB5b3UgdXNlIGEgaHlwZXItY29udmVyZ2VkIHNldHVwIHZpYSBnZGVwbG95KS48 YnI+CiAgICAgICAgICAgICAgICAgICAgPGI+TW9hY2lyOjwvYj4gWWVzISBUaGlzIGlzIGFub3Ro ZXIgcmVhc29uIHRvIGhhdmUKICAgICAgICAgICAgICAgICAgICBzZXBhcmF0ZSBuZXR3b3JrcyBm b3Igbm9ydGgvc291dGggYW5kIGVhc3Qvd2VzdC4gSW4KICAgICAgICAgICAgICAgICAgICB0aGF0 IHdheSBJIGNhbiB1c2UgdGhlIHN0YW5kYXJkIE1UVSBvbiB0aGUgMTBHYiBOSUNzCiAgICAgICAg ICAgICAgICAgICAgYW5kIGp1bWJvIGZyYW1lcyBvbiB0aGUgZmlsZS9tb3ZlIDQwR2IgTklDcy48 YnI+CiAgICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAgICA8L2Rpdj4KICAg ICAgICAgICAgICAgICAgPGRpdj5ZLjwvZGl2PgogICAgICAgICAgICAgICAgICA8ZGl2PqA8L2Rp dj4KICAgICAgICAgICAgICAgICAgPGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHls ZT0ibWFyZ2luOjAgMCAwCiAgICAgICAgICAgICAgICAgICAgLjhleDsgYm9yZGVyLWxlZnQ6MXB4 ICNjY2Mgc29saWQ7IHBhZGRpbmctbGVmdDoxZXgiPgogICAgICAgICAgICAgICAgICAgIDxkaXYg ZGlyPSJsdHIiPgogICAgICAgICAgICAgICAgICAgICAgPGRpdgogICAgICAgICAgICAgICAgICAg ICAgICBpZD0ibV81NTA5NTg1ODg5NTY5NjkwNzkxZGl2dGFnZGVmYXVsdHdyYXBwZXIiCiAgICAg ICAgICAgICAgICAgICAgICAgIGRpcj0ibHRyIiBzdHlsZT0iZm9udC1zaXplOjEycHQ7IGNvbG9y OiMwMDAwMDA7CiAgICAgICAgICAgICAgICAgICAgICAgIGZvbnQtZmFtaWx5OkNhbGlicmksSGVs dmV0aWNhLHNhbnMtc2VyaWYiPgogICAgICAgICAgICAgICAgICAgICAgICA8cD48YnI+CiAgICAg ICAgICAgICAgICAgICAgICAgIDwvcD4KICAgICAgICAgICAgICAgICAgICAgICAgPHA+WW91IG9w aW5pb24vZmVlZGJhY2sgd2lsbCBiZSByZWFsbHkKICAgICAgICAgICAgICAgICAgICAgICAgICBh cHByZWNpYXRlZCE8L3A+CiAgICAgICAgICAgICAgICAgICAgICAgIDxzcGFuIGNsYXNzPSJIT0Vu WmIiPjxmb250IGNvbG9yPSIjODg4ODg4Ij4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxw Pk1vYWNpcjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvcD4KICAgICAgICAgICAg ICAgICAgICAgICAgICA8L2ZvbnQ+PC9zcGFuPjwvZGl2PgogICAgICAgICAgICAgICAgICAgIDwv ZGl2PgogICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAgICBfX19fX19f X19fX19fX19fX19fX19fX19fX19fX188d2JyPl9fX19fX19fX19fX19fX19fPGJyPgogICAgICAg ICAgICAgICAgICAgIFVzZXJzIG1haWxpbmcgbGlzdDxicj4KICAgICAgICAgICAgICAgICAgICA8 YSBocmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQub3JnIgogICAgICAgICAgICAgICAgICAgICAgbW96 LWRvLW5vdC1zZW5kPSJ0cnVlIj5Vc2Vyc0BvdmlydC5vcmc8L2E+PGJyPgogICAgICAgICAgICAg ICAgICAgIDxhCiAgICAgICAgICAgICAgICAgICAgICBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQu b3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiCiAgICAgICAgICAgICAgICAgICAgICByZWw9Im5v cmVmZXJyZXIiIHRhcmdldD0iX2JsYW5rIgogICAgICAgICAgICAgICAgICAgICAgbW96LWRvLW5v dC1zZW5kPSJ0cnVlIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnLzx3YnI+bWFpbG1hbi9saXN0aW5m by91c2VyczwvYT48YnI+CiAgICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAg ICA8L2Jsb2NrcXVvdGU+CiAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgIDxi cj4KICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICA8L2Rp dj4KICAgICAgICA8L2Rpdj4KICAgICAgPC9kaXY+CiAgICAgIDxicj4KICAgICAgPGZpZWxkc2V0 IGNsYXNzPSJtaW1lQXR0YWNobWVudEhlYWRlciI+PC9maWVsZHNldD4KICAgICAgPGJyPgogICAg ICA8cHJlIHdyYXA9IiI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX18KVXNlcnMgbWFpbGluZyBsaXN0CjxhIGNsYXNzPSJtb3otdHh0LWxpbmstYWJicmV2aWF0 ZWQiIGhyZWY9Im1haWx0bzpVc2Vyc0BvdmlydC5vcmciPlVzZXJzQG92aXJ0Lm9yZzwvYT4KPGEg Y2xhc3M9Im1vei10eHQtbGluay1mcmVldGV4dCIgaHJlZj0iaHR0cDovL2xpc3RzLm92aXJ0Lm9y Zy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4v bGlzdGluZm8vdXNlcnM8L2E+CjwvcHJlPgogICAgPC9ibG9ja3F1b3RlPgogICAgPGJyPgogIDwv Ym9keT4KPC9odG1sPgoKLS0tLS0tLS0tLS0tLS03OTk5RTIzMjUzMjk5MkMyQjQxODRGRjctLQo= --===============8747593560808751059==-- From erekle.magradze at recogizer.de Mon Aug 7 13:26:07 2017 Content-Type: multipart/mixed; boundary="===============0412783989436684471==" MIME-Version: 1.0 From: Erekle Magradze To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Mon, 07 Aug 2017 15:26:03 +0200 Message-ID: In-Reply-To: 63bac47b-afe6-0258-d3d7-e545a5004c30@upx.com --===============0412783989436684471== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------C2F37362CE639F36E44EBBD8 Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 8bit Hi Frenando, Here is my experience, if you consider a particular hard drive as a = brick for gluster volume and it dies, i.e. it becomes not accessible = it's a huge hassle to discard that brick and exchange with another one, = since gluster some tries to access that broken brick and it's causing = (at least it cause for me) a big pain, therefore it's better to have a = RAID as brick, i.e. have RAID 1 (mirroring) for each brick, in this case = if the disk is down you can easily exchange it and rebuild the RAID = without going offline, i.e switching off the volume doing brick = manipulations and switching it back on. Cheers Erekle On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote: > > For any RAID 5 or 6 configuration I normally follow a simple gold rule = > which gave good results so far: > - up to 4 disks RAID 5 > - 5 or more disks RAID 6 > > However I didn't really understand well the recommendation to use any = > RAID with GlusterFS. I always thought that GlusteFS likes to work in = > JBOD mode and control the disks (bricks) directlly so you can create = > whatever distribution rule you wish, and if a single disk fails you = > just replace it and which obviously have the data replicated from = > another. The only downside of using in this way is that the = > replication data will be flow accross all servers but that is not much = > a big issue. > > Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS. > > Thanks > Regards > Fernando > > > On 07/08/2017 03:46, Devin Acosta wrote: >> >> Moacir, >> >> I have recently installed multiple Red Hat Virtualization hosts for = >> several different companies, and have dealt with the Red Hat Support = >> Team in depth about optimal configuration in regards to setting up = >> GlusterFS most efficiently and I wanted to share with you what I learned. >> >> In general Red Hat Virtualization team frowns upon using each DISK of = >> the system as just a JBOD, sure there is some protection by having = >> the data replicated, however, the recommendation is to use RAID 6 = >> (preferred) or RAID-5, or at least RAID-1 at the very least. >> >> Here is the direct quote from Red Hat when I asked about RAID and Bricks: >> / >> / >> /"A typical Gluster configuration would use RAID underneath the = >> bricks. RAID 6 is most typical as it gives you 2 disk failure = >> protection, but RAID 5 could be used too. Once you have the RAIDed = >> bricks, you'd then apply the desired replication on top of that. The = >> most popular way of doing this would be distributed replicated with = >> 2x replication. In general you'll get better performance with larger = >> bricks. 12 drives is often a sweet spot. Another option would be to = >> create a separate tier using all SSD=E2=80=99s.=E2=80=9D / >> >> /In order to SSD tiering from my understanding you would need 1 x = >> NVMe drive in each server, or 4 x SSD hot tier (it needs to be = >> distributed, replicated for the hot tier if not using NVME). So with = >> you only having 1 SSD drive in each server, I=E2=80=99d suggest maybe lo= oking = >> into the NVME option. / >> / >> / >> /Since your using only 3-servers, what I=E2=80=99d probably suggest is t= o do = >> (2 Replicas + Arbiter Node), this setup actually doesn=E2=80=99t require= the = >> 3rd server to have big drives at all as it only stores meta-data = >> about the files and not actually a full copy. / >> / >> / >> /Please see the attached document that was given to me by Red Hat to = >> get more information on this. Hope this information helps you./ >> / >> / >> >> -- >> >> Devin Acosta, RHCA, RHVCA >> Red Hat Certified Architect >> >> On August 6, 2017 at 7:29:29 PM, Moacir Ferreira = >> (moacirferreira(a)hotmail.com ) wro= te: >> >>> I am willing to assemble a oVirt "pod", made of 3 servers, each with = >>> 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is = >>> to use GlusterFS to provide HA for the VMs. The 3 servers have a = >>> dual 40Gb NIC and a dual 10Gb NIC. So my intention is to create a = >>> loop like a server triangle using the 40Gb NICs for virtualization = >>> files (VMs .qcow2) access and to move VMs around the pod (east /west = >>> traffic) while using the 10Gb interfaces for giving services to the = >>> outside world (north/south traffic). >>> >>> >>> This said, my first question is: How should I deploy GlusterFS in = >>> such oVirt scenario? My questions are: >>> >>> >>> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, = >>> and then create a GlusterFS using them? >>> >>> 2 - Instead, should I create a JBOD array made of all server's disks? >>> >>> 3 - What is the best Gluster configuration to provide for HA while = >>> not consuming too much disk space? >>> >>> 4 - Does a oVirt hypervisor pod like I am planning to build, and the = >>> virtualization environment, benefits from tiering when using a SSD = >>> disk? And yes, will Gluster do it by default or I have to configure = >>> it to do so? >>> >>> >>> At the bottom line, what is the good practice for using GlusterFS in = >>> small pods for enterprises? >>> >>> >>> You opinion/feedback will be really appreciated! >>> >>> Moacir >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users > > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users --------------C2F37362CE639F36E44EBBD8 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit
Hi Frenando,

Here is my experience, if you consider a particular hard drive as a brick for gluster volume and it dies, i.e. it becomes not accessible it's a huge hassle to discard that brick and exchange with another one, since gluster some tries to access that broken brick and it's causing (at least it cause for me) a big pain, therefore it's better to have a RAID as brick, i.e. have RAID 1 (mirroring) for each brick, in this case if the disk is down you can easily exchange it and rebuild the RAID without going offline, i.e switching off the volume doing brick manipulations and switching it back on.

Cheers

Erekle

On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote:

For any RAID 5 or 6 configuration I normally follow a simple gold rule which gave good results so far:
- up to 4 disks RAID 5
- 5 or more disks RAID 6

However I didn't really understand well the recommendation to use any RAID with GlusterFS. I always thought that GlusteFS likes to work in JBOD mode and control the disks (bricks) directlly so you can create whatever distribution rule you wish, and if a single disk fails you just replace it and which obviously have the data replicated from another. The only downside of using in this way is that the replication data will be flow accross all servers but that is not much a big issue.

Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS.

Thanks
Regards
Fernando

On 07/08/2017 03:46, Devin Acosta wrote:

=

= Moacir,

=

= I have recently installed multiple Red Hat Virtualization hosts for several different companies, and have dealt with the Red Hat Support Team in depth about optimal configuration in regards to setting up GlusterFS most efficiently and I wanted to share with you what I learned.

=

= In general Red Hat Virtualization team frowns upon using each DISK of the system as just a JBOD, sure there is some protection by having the data replicated, however, the recommendation is to use RAID 6 (preferred) or RAID-5, or at least RAID-1 at the very least.

=

Here is the=C2=A0direct quote from Red Hat when I asked about RAID and Bricks:

"A typical Gluster configuration would use RAID underneath the bricks. RAID 6 is most typical as it gives you 2 disk failure protection, but RAID 5 could be used too.=C2=A0Once you have the RAIDed bricks, you'd then apply t= he desired replication on top of that. The most popular way of doing this would be distributed replicated with 2x replication.=C2=A0In general you'll get better performance wi= th larger bricks. 12 drives is often a sweet spot. Another option would be to create a separate tier using all SSD=E2=80=99s.=E2=80=9D=C2=A0

In order to SSD=C2=A0tiering from my understanding you would need 1 x NVMe drive in each server, or 4 x SSD hot tier (it needs to be distributed, replicated for the hot tier if not using NVME). So with you only having 1 SSD drive in each server,=C2=A0I=E2=80=99d suggest maybe looking = into the NVME option.=C2=A0

Since your using only 3-servers, what I=E2=80=99d prob= ably suggest is to do (2 Replicas + Arbiter Node), this setup actually doesn=E2=80=99t require the 3rd server to have big d= rives at all as it only stores meta-data about the files and not actually a full copy.=C2=A0

Please see the attached document that was given to me by Red Hat to get more information on this. Hope this information helps you.

--

Devin Acosta, RHCA, RHVCA

Red Hat Certified Architect

On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (moacirferreira(a)hotmail.com) wrote:

I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop like a server triangle using the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VMs around the pod (east /west traffic) while using the 10Gb interfaces for giving services to the outside world (north/south traffic).

This said, my first question is: How should I deploy GlusterFS in such oVirt scenario? My questions are:

1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then create a GlusterFS using them?

2 - Instead, should I create a JBOD array made of all server's disks?

3 - What is the best Gluster configuration to provide for HA while not consuming too much disk space?

4 - Does a oVirt hypervisor pod like I am planning to build, and the virtualization environment, benefits from tiering when using a SSD disk? And yes, will Gluster do it by default or I have to configure it to do so?

At the bottom line, what is the good practice for using GlusterFS in small pods for enterprises?

You opinion/feedback will be really appreciated!

Moacir

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/l= istinfo/users

_______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/list= info/users

_______________________________________________ Users mailing list Use= rs(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

--------------C2F37362CE639F36E44EBBD8-- --===============0412783989436684471== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS1DMkYzNzM2MkNFNjM5RjM2RTQ0RUJCRDgKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDhiaXQK CkhpIEZyZW5hbmRvLAoKSGVyZSBpcyBteSBleHBlcmllbmNlLCBpZiB5b3UgY29uc2lkZXIgYSBw YXJ0aWN1bGFyIGhhcmQgZHJpdmUgYXMgYSAKYnJpY2sgZm9yIGdsdXN0ZXIgdm9sdW1lIGFuZCBp dCBkaWVzLCBpLmUuIGl0IGJlY29tZXMgbm90IGFjY2Vzc2libGUgCml0J3MgYSBodWdlIGhhc3Ns ZSB0byBkaXNjYXJkIHRoYXQgYnJpY2sgYW5kIGV4Y2hhbmdlIHdpdGggYW5vdGhlciBvbmUsIApz aW5jZSBnbHVzdGVyIHNvbWUgdHJpZXMgdG8gYWNjZXNzIHRoYXQgYnJva2VuIGJyaWNrIGFuZCBp dCdzIGNhdXNpbmcgCihhdCBsZWFzdCBpdCBjYXVzZSBmb3IgbWUpIGEgYmlnIHBhaW4sIHRoZXJl Zm9yZSBpdCdzIGJldHRlciB0byBoYXZlIGEgClJBSUQgYXMgYnJpY2ssIGkuZS4gaGF2ZSBSQUlE IDEgKG1pcnJvcmluZykgZm9yIGVhY2ggYnJpY2ssIGluIHRoaXMgY2FzZSAKaWYgdGhlIGRpc2sg aXMgZG93biB5b3UgY2FuIGVhc2lseSBleGNoYW5nZSBpdCBhbmQgcmVidWlsZCB0aGUgUkFJRCAK d2l0aG91dCBnb2luZyBvZmZsaW5lLCBpLmUgc3dpdGNoaW5nIG9mZiB0aGUgdm9sdW1lIGRvaW5n IGJyaWNrIAptYW5pcHVsYXRpb25zIGFuZCBzd2l0Y2hpbmcgaXQgYmFjayBvbi4KCkNoZWVycwoK RXJla2xlCgoKT24gMDgvMDcvMjAxNyAwMzowNCBQTSwgRkVSTkFORE8gRlJFRElBTkkgd3JvdGU6 Cj4KPiBGb3IgYW55IFJBSUQgNSBvciA2IGNvbmZpZ3VyYXRpb24gSSBub3JtYWxseSBmb2xsb3cg YSBzaW1wbGUgZ29sZCBydWxlIAo+IHdoaWNoIGdhdmUgZ29vZCByZXN1bHRzIHNvIGZhcjoKPiAt IHVwIHRvIDQgZGlza3MgUkFJRCA1Cj4gLSA1IG9yIG1vcmUgZGlza3MgUkFJRCA2Cj4KPiBIb3dl dmVyIEkgZGlkbid0IHJlYWxseSB1bmRlcnN0YW5kIHdlbGwgdGhlIHJlY29tbWVuZGF0aW9uIHRv IHVzZSBhbnkgCj4gUkFJRCB3aXRoIEdsdXN0ZXJGUy4gSSBhbHdheXMgdGhvdWdodCB0aGF0IEds dXN0ZUZTIGxpa2VzIHRvIHdvcmsgaW4gCj4gSkJPRCBtb2RlIGFuZCBjb250cm9sIHRoZSBkaXNr cyAoYnJpY2tzKSBkaXJlY3RsbHkgc28geW91IGNhbiBjcmVhdGUgCj4gd2hhdGV2ZXIgZGlzdHJp YnV0aW9uIHJ1bGUgeW91IHdpc2gsIGFuZCBpZiBhIHNpbmdsZSBkaXNrIGZhaWxzIHlvdSAKPiBq dXN0IHJlcGxhY2UgaXQgYW5kIHdoaWNoIG9idmlvdXNseSBoYXZlIHRoZSBkYXRhIHJlcGxpY2F0 ZWQgZnJvbSAKPiBhbm90aGVyLiBUaGUgb25seSBkb3duc2lkZSBvZiB1c2luZyBpbiB0aGlzIHdh eSBpcyB0aGF0IHRoZSAKPiByZXBsaWNhdGlvbiBkYXRhIHdpbGwgYmUgZmxvdyBhY2Nyb3NzIGFs bCBzZXJ2ZXJzIGJ1dCB0aGF0IGlzIG5vdCBtdWNoIAo+IGEgYmlnIGlzc3VlLgo+Cj4gQW55b25l IGNhbiBlbGFib3JhdGUgYWJvdXQgVXNpbmcgUkFJRCArIEdsdXN0ZXJGUyBhbmQgSkJPRCArIEds dXN0ZXJGUy4KPgo+IFRoYW5rcwo+IFJlZ2FyZHMKPiBGZXJuYW5kbwo+Cj4KPiBPbiAwNy8wOC8y MDE3IDAzOjQ2LCBEZXZpbiBBY29zdGEgd3JvdGU6Cj4+Cj4+IE1vYWNpciwKPj4KPj4gSSBoYXZl IHJlY2VudGx5IGluc3RhbGxlZCBtdWx0aXBsZSBSZWQgSGF0IFZpcnR1YWxpemF0aW9uIGhvc3Rz IGZvciAKPj4gc2V2ZXJhbCBkaWZmZXJlbnQgY29tcGFuaWVzLCBhbmQgaGF2ZSBkZWFsdCB3aXRo IHRoZSBSZWQgSGF0IFN1cHBvcnQgCj4+IFRlYW0gaW4gZGVwdGggYWJvdXQgb3B0aW1hbCBjb25m aWd1cmF0aW9uIGluIHJlZ2FyZHMgdG8gc2V0dGluZyB1cCAKPj4gR2x1c3RlckZTIG1vc3QgZWZm aWNpZW50bHkgYW5kIEkgd2FudGVkIHRvIHNoYXJlIHdpdGggeW91IHdoYXQgSSBsZWFybmVkLgo+ Pgo+PiBJbiBnZW5lcmFsIFJlZCBIYXQgVmlydHVhbGl6YXRpb24gdGVhbSBmcm93bnMgdXBvbiB1 c2luZyBlYWNoIERJU0sgb2YgCj4+IHRoZSBzeXN0ZW0gYXMganVzdCBhIEpCT0QsIHN1cmUgdGhl cmUgaXMgc29tZSBwcm90ZWN0aW9uIGJ5IGhhdmluZyAKPj4gdGhlIGRhdGEgcmVwbGljYXRlZCwg aG93ZXZlciwgdGhlIHJlY29tbWVuZGF0aW9uIGlzIHRvIHVzZSBSQUlEIDYgCj4+IChwcmVmZXJy ZWQpIG9yIFJBSUQtNSwgb3IgYXQgbGVhc3QgUkFJRC0xIGF0IHRoZSB2ZXJ5IGxlYXN0Lgo+Pgo+ PiBIZXJlIGlzIHRoZSBkaXJlY3QgcXVvdGUgZnJvbSBSZWQgSGF0IHdoZW4gSSBhc2tlZCBhYm91 dCBSQUlEIGFuZCBCcmlja3M6Cj4+IC8KPj4gLwo+PiAvIkEgdHlwaWNhbCBHbHVzdGVyIGNvbmZp Z3VyYXRpb24gd291bGQgdXNlIFJBSUQgdW5kZXJuZWF0aCB0aGUgCj4+IGJyaWNrcy4gUkFJRCA2 IGlzIG1vc3QgdHlwaWNhbCBhcyBpdCBnaXZlcyB5b3UgMiBkaXNrIGZhaWx1cmUgCj4+IHByb3Rl Y3Rpb24sIGJ1dCBSQUlEIDUgY291bGQgYmUgdXNlZCB0b28uIE9uY2UgeW91IGhhdmUgdGhlIFJB SURlZCAKPj4gYnJpY2tzLCB5b3UnZCB0aGVuIGFwcGx5IHRoZSBkZXNpcmVkIHJlcGxpY2F0aW9u IG9uIHRvcCBvZiB0aGF0LiBUaGUgCj4+IG1vc3QgcG9wdWxhciB3YXkgb2YgZG9pbmcgdGhpcyB3 b3VsZCBiZSBkaXN0cmlidXRlZCByZXBsaWNhdGVkIHdpdGggCj4+IDJ4IHJlcGxpY2F0aW9uLiBJ biBnZW5lcmFsIHlvdSdsbCBnZXQgYmV0dGVyIHBlcmZvcm1hbmNlIHdpdGggbGFyZ2VyIAo+PiBi cmlja3MuIDEyIGRyaXZlcyBpcyBvZnRlbiBhIHN3ZWV0IHNwb3QuIEFub3RoZXIgb3B0aW9uIHdv dWxkIGJlIHRvIAo+PiBjcmVhdGUgYSBzZXBhcmF0ZSB0aWVyIHVzaW5nIGFsbCBTU0TigJlzLuKA nSAvCj4+Cj4+IC9JbiBvcmRlciB0byBTU0QgdGllcmluZyBmcm9tIG15IHVuZGVyc3RhbmRpbmcg eW91IHdvdWxkIG5lZWQgMSB4IAo+PiBOVk1lIGRyaXZlIGluIGVhY2ggc2VydmVyLCBvciA0IHgg U1NEIGhvdCB0aWVyIChpdCBuZWVkcyB0byBiZSAKPj4gZGlzdHJpYnV0ZWQsIHJlcGxpY2F0ZWQg Zm9yIHRoZSBob3QgdGllciBpZiBub3QgdXNpbmcgTlZNRSkuIFNvIHdpdGggCj4+IHlvdSBvbmx5 IGhhdmluZyAxIFNTRCBkcml2ZSBpbiBlYWNoIHNlcnZlciwgSeKAmWQgc3VnZ2VzdCBtYXliZSBs b29raW5nIAo+PiBpbnRvIHRoZSBOVk1FIG9wdGlvbi4gLwo+PiAvCj4+IC8KPj4gL1NpbmNlIHlv dXIgdXNpbmcgb25seSAzLXNlcnZlcnMsIHdoYXQgSeKAmWQgcHJvYmFibHkgc3VnZ2VzdCBpcyB0 byBkbyAKPj4gKDIgUmVwbGljYXMgKyBBcmJpdGVyIE5vZGUpLCB0aGlzIHNldHVwIGFjdHVhbGx5 IGRvZXNu4oCZdCByZXF1aXJlIHRoZSAKPj4gM3JkIHNlcnZlciB0byBoYXZlIGJpZyBkcml2ZXMg YXQgYWxsIGFzIGl0IG9ubHkgc3RvcmVzIG1ldGEtZGF0YSAKPj4gYWJvdXQgdGhlIGZpbGVzIGFu ZCBub3QgYWN0dWFsbHkgYSBmdWxsIGNvcHkuIC8KPj4gLwo+PiAvCj4+IC9QbGVhc2Ugc2VlIHRo ZSBhdHRhY2hlZCBkb2N1bWVudCB0aGF0IHdhcyBnaXZlbiB0byBtZSBieSBSZWQgSGF0IHRvIAo+ PiBnZXQgbW9yZSBpbmZvcm1hdGlvbiBvbiB0aGlzLiBIb3BlIHRoaXMgaW5mb3JtYXRpb24gaGVs cHMgeW91Li8KPj4gLwo+PiAvCj4+Cj4+IC0tCj4+Cj4+IERldmluIEFjb3N0YSwgUkhDQSwgUkhW Q0EKPj4gUmVkIEhhdCBDZXJ0aWZpZWQgQXJjaGl0ZWN0Cj4+Cj4+IE9uIEF1Z3VzdCA2LCAyMDE3 IGF0IDc6Mjk6MjkgUE0sIE1vYWNpciBGZXJyZWlyYSAKPj4gKG1vYWNpcmZlcnJlaXJhQGhvdG1h aWwuY29tIDxtYWlsdG86bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb20+KSB3cm90ZToKPj4KPj4+ IEkgYW0gd2lsbGluZyB0byBhc3NlbWJsZSBhIG9WaXJ0ICJwb2QiLCBtYWRlIG9mIDMgc2VydmVy cywgZWFjaCB3aXRoIAo+Pj4gMiBDUFUgc29ja2V0cyBvZiAxMiBjb3JlcywgMjU2R0IgUkFNLCA3 IEhERCAxMEssIDEgU1NELiBUaGUgaWRlYSBpcyAKPj4+IHRvIHVzZSBHbHVzdGVyRlMgdG8gcHJv dmlkZSBIQSBmb3IgdGhlIFZNcy4gVGhlIDMgc2VydmVycyBoYXZlIGEgCj4+PiBkdWFsIDQwR2Ig TklDIGFuZCBhIGR1YWwgMTBHYiBOSUMuIFNvIG15IGludGVudGlvbiBpcyB0byBjcmVhdGUgYSAK Pj4+IGxvb3AgbGlrZSBhIHNlcnZlciB0cmlhbmdsZSB1c2luZyB0aGUgNDBHYiBOSUNzIGZvciB2 aXJ0dWFsaXphdGlvbiAKPj4+IGZpbGVzIChWTXMgLnFjb3cyKSBhY2Nlc3MgYW5kIHRvIG1vdmUg Vk1zIGFyb3VuZCB0aGUgcG9kIChlYXN0IC93ZXN0IAo+Pj4gdHJhZmZpYykgd2hpbGUgdXNpbmcg dGhlIDEwR2IgaW50ZXJmYWNlcyBmb3IgZ2l2aW5nIHNlcnZpY2VzIHRvIHRoZSAKPj4+IG91dHNp ZGUgd29ybGQgKG5vcnRoL3NvdXRoIHRyYWZmaWMpLgo+Pj4KPj4+Cj4+PiBUaGlzIHNhaWQsIG15 IGZpcnN0IHF1ZXN0aW9uIGlzOiBIb3cgc2hvdWxkIEkgZGVwbG95IEdsdXN0ZXJGUyBpbiAKPj4+ IHN1Y2ggb1ZpcnQgc2NlbmFyaW8/IE15IHF1ZXN0aW9ucyBhcmU6Cj4+Pgo+Pj4KPj4+IDEgLSBT aG91bGQgSSBjcmVhdGUgMyBSQUlEIChpLmUuOiBSQUlEIDUpLCBvbmUgb24gZWFjaCBvVmlydCBu b2RlLCAKPj4+IGFuZCB0aGVuIGNyZWF0ZSBhIEdsdXN0ZXJGUyB1c2luZyB0aGVtPwo+Pj4KPj4+ IDIgLSBJbnN0ZWFkLCBzaG91bGQgSSBjcmVhdGUgYSBKQk9EIGFycmF5IG1hZGUgb2YgYWxsIHNl cnZlcidzIGRpc2tzPwo+Pj4KPj4+IDMgLSBXaGF0IGlzIHRoZSBiZXN0IEdsdXN0ZXIgY29uZmln dXJhdGlvbiB0byBwcm92aWRlIGZvciBIQSB3aGlsZSAKPj4+IG5vdCBjb25zdW1pbmcgdG9vIG11 Y2ggZGlzayBzcGFjZT8KPj4+Cj4+PiA0IC0gRG9lcyBhIG9WaXJ0IGh5cGVydmlzb3IgcG9kIGxp a2UgSSBhbSBwbGFubmluZyB0byBidWlsZCwgYW5kIHRoZSAKPj4+IHZpcnR1YWxpemF0aW9uIGVu dmlyb25tZW50LCBiZW5lZml0cyBmcm9tIHRpZXJpbmcgd2hlbiB1c2luZyBhIFNTRCAKPj4+IGRp c2s/IEFuZCB5ZXMsIHdpbGwgR2x1c3RlciBkbyBpdCBieSBkZWZhdWx0IG9yIEkgaGF2ZSB0byBj b25maWd1cmUgCj4+PiBpdCB0byBkbyBzbz8KPj4+Cj4+Pgo+Pj4gQXQgdGhlIGJvdHRvbSBsaW5l LCB3aGF0IGlzIHRoZSBnb29kIHByYWN0aWNlIGZvciB1c2luZyBHbHVzdGVyRlMgaW4gCj4+PiBz bWFsbCBwb2RzIGZvciBlbnRlcnByaXNlcz8KPj4+Cj4+Pgo+Pj4gWW91IG9waW5pb24vZmVlZGJh Y2sgd2lsbCBiZSByZWFsbHkgYXBwcmVjaWF0ZWQhCj4+Pgo+Pj4gTW9hY2lyCj4+Pgo+Pj4gX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KPj4+IFVzZXJzIG1h aWxpbmcgbGlzdAo+Pj4gVXNlcnNAb3ZpcnQub3JnIDxtYWlsdG86VXNlcnNAb3ZpcnQub3JnPgo+ Pj4gaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzCj4+Cj4+Cj4+ IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCj4+IFVzZXJz IG1haWxpbmcgbGlzdAo+PiBVc2Vyc0BvdmlydC5vcmcKPj4gaHR0cDovL2xpc3RzLm92aXJ0Lm9y Zy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzCj4KPgo+Cj4gX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX18KPiBVc2VycyBtYWlsaW5nIGxpc3QKPiBVc2Vyc0Bvdmly dC5vcmcKPiBodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMKCgot LS0tLS0tLS0tLS0tLUMyRjM3MzYyQ0U2MzlGMzZFNDRFQkJEOApDb250ZW50LVR5cGU6IHRleHQv aHRtbDsgY2hhcnNldD11dGYtOApDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiA4Yml0Cgo8aHRt bD4KICA8aGVhZD4KICAgIDxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0i dGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4KICA8L2hlYWQ+CiAgPGJvZHkgdGV4dD0iIzAwMDAw MCIgYmdjb2xvcj0iI0ZGRkZGRiI+CiAgICA8cD5IaSBGcmVuYW5kbyw8L3A+CiAgICA8cD5IZXJl IGlzIG15IGV4cGVyaWVuY2UsIGlmIHlvdSBjb25zaWRlciBhIHBhcnRpY3VsYXIgaGFyZCBkcml2 ZSBhcwogICAgICBhIGJyaWNrIGZvciBnbHVzdGVyIHZvbHVtZSBhbmQgaXQgZGllcywgaS5lLiBp dCBiZWNvbWVzIG5vdAogICAgICBhY2Nlc3NpYmxlIGl0J3MgYSBodWdlIGhhc3NsZSB0byBkaXNj YXJkIHRoYXQgYnJpY2sgYW5kIGV4Y2hhbmdlCiAgICAgIHdpdGggYW5vdGhlciBvbmUsIHNpbmNl IGdsdXN0ZXIgc29tZSB0cmllcyB0byBhY2Nlc3MgdGhhdCBicm9rZW4KICAgICAgYnJpY2sgYW5k IGl0J3MgY2F1c2luZyAoYXQgbGVhc3QgaXQgY2F1c2UgZm9yIG1lKSBhIGJpZyBwYWluLAogICAg ICB0aGVyZWZvcmUgaXQncyBiZXR0ZXIgdG8gaGF2ZSBhIFJBSUQgYXMgYnJpY2ssIGkuZS4gaGF2 ZSBSQUlEIDEKICAgICAgKG1pcnJvcmluZykgZm9yIGVhY2ggYnJpY2ssIGluIHRoaXMgY2FzZSBp ZiB0aGUgZGlzayBpcyBkb3duIHlvdQogICAgICBjYW4gZWFzaWx5IGV4Y2hhbmdlIGl0IGFuZCBy ZWJ1aWxkIHRoZSBSQUlEIHdpdGhvdXQgZ29pbmcgb2ZmbGluZSwKICAgICAgaS5lIHN3aXRjaGlu ZyBvZmYgdGhlIHZvbHVtZSBkb2luZyBicmljayBtYW5pcHVsYXRpb25zIGFuZAogICAgICBzd2l0 Y2hpbmcgaXQgYmFjayBvbi48YnI+CiAgICA8L3A+CiAgICA8cD5DaGVlcnM8L3A+CiAgICA8cD5F cmVrbGU8YnI+CiAgICA8L3A+CiAgICA8YnI+CiAgICA8ZGl2IGNsYXNzPSJtb3otY2l0ZS1wcmVm aXgiPk9uIDA4LzA3LzIwMTcgMDM6MDQgUE0sIEZFUk5BTkRPCiAgICAgIEZSRURJQU5JIHdyb3Rl Ojxicj4KICAgIDwvZGl2PgogICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSIKICAgICAgY2l0ZT0i bWlkOjYzYmFjNDdiLWFmZTYtMDI1OC1kM2Q3LWU1NDVhNTAwNGMzMEB1cHguY29tIj4KICAgICAg PG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJz ZXQ9dXRmLTgiPgogICAgICA8cD5Gb3IgYW55IFJBSUQgNSBvciA2IGNvbmZpZ3VyYXRpb24gSSBu b3JtYWxseSBmb2xsb3cgYSBzaW1wbGUKICAgICAgICBnb2xkIHJ1bGUgd2hpY2ggZ2F2ZSBnb29k IHJlc3VsdHMgc28gZmFyOjxicj4KICAgICAgICAtIHVwIHRvIDQgZGlza3MgUkFJRCA1PGJyPgog ICAgICAgIC0gNSBvciBtb3JlIGRpc2tzIFJBSUQgNjwvcD4KICAgICAgPHA+SG93ZXZlciBJIGRp ZG4ndCByZWFsbHkgdW5kZXJzdGFuZCB3ZWxsIHRoZSByZWNvbW1lbmRhdGlvbiB0bwogICAgICAg IHVzZSBhbnkgUkFJRCB3aXRoIEdsdXN0ZXJGUy4gSSBhbHdheXMgdGhvdWdodCB0aGF0IEdsdXN0 ZUZTCiAgICAgICAgbGlrZXMgdG8gd29yayBpbiBKQk9EIG1vZGUgYW5kIGNvbnRyb2wgdGhlIGRp c2tzIChicmlja3MpCiAgICAgICAgZGlyZWN0bGx5IHNvIHlvdSBjYW4gY3JlYXRlIHdoYXRldmVy IGRpc3RyaWJ1dGlvbiBydWxlIHlvdSB3aXNoLAogICAgICAgIGFuZCBpZiBhIHNpbmdsZSBkaXNr IGZhaWxzIHlvdSBqdXN0IHJlcGxhY2UgaXQgYW5kIHdoaWNoCiAgICAgICAgb2J2aW91c2x5IGhh dmUgdGhlIGRhdGEgcmVwbGljYXRlZCBmcm9tIGFub3RoZXIuIFRoZSBvbmx5CiAgICAgICAgZG93 bnNpZGUgb2YgdXNpbmcgaW4gdGhpcyB3YXkgaXMgdGhhdCB0aGUgcmVwbGljYXRpb24gZGF0YSB3 aWxsCiAgICAgICAgYmUgZmxvdyBhY2Nyb3NzIGFsbCBzZXJ2ZXJzIGJ1dCB0aGF0IGlzIG5vdCBt dWNoIGEgYmlnIGlzc3VlLjwvcD4KICAgICAgPHA+QW55b25lIGNhbiBlbGFib3JhdGUgYWJvdXQg VXNpbmcgUkFJRCArIEdsdXN0ZXJGUyBhbmQgSkJPRCArCiAgICAgICAgR2x1c3RlckZTLjwvcD4K ICAgICAgPHA+VGhhbmtzPGJyPgogICAgICAgIFJlZ2FyZHM8YnI+CiAgICAgICAgRmVybmFuZG88 YnI+CiAgICAgIDwvcD4KICAgICAgPGJyPgogICAgICA8ZGl2IGNsYXNzPSJtb3otY2l0ZS1wcmVm aXgiPk9uIDA3LzA4LzIwMTcgMDM6NDYsIERldmluIEFjb3N0YQogICAgICAgIHdyb3RlOjxicj4K ICAgICAgPC9kaXY+CiAgICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiCmNpdGU9Im1pZDpDQU5D R0tFcDRYR3MwVStRczc4ZUVtcUNOdHZwTFktQXpqYjVEY0doWjl5aUtUQkVFZndAbWFpbC5nbWFp bC5jb20iPgogICAgICAgIDxzdHlsZT5ib2R5e2ZvbnQtZmFtaWx5OkhlbHZldGljYSxBcmlhbDtm b250LXNpemU6MTNweH08L3N0eWxlPgogICAgICAgIDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZvbnQi IHN0eWxlPSJjb2xvcjpyZ2IoMCwwLDApO21hcmdpbjowcHgiPjxmb250CiAgICAgICAgICAgIGZh Y2U9IklucHV0IE1vbm8iPjxicj4KICAgICAgICAgIDwvZm9udD48L2Rpdj4KICAgICAgICA8ZGl2 IGlkPSJibG9vcF9jdXN0b21mb250IiBzdHlsZT0iY29sb3I6cmdiKDAsMCwwKTttYXJnaW46MHB4 Ij48Zm9udAogICAgICAgICAgICBmYWNlPSJJbnB1dCBNb25vIj5Nb2FjaXIsPC9mb250PjwvZGl2 PgogICAgICAgIDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJjb2xvcjpyZ2IoMCww LDApO21hcmdpbjowcHgiPjxmb250CiAgICAgICAgICAgIGZhY2U9IklucHV0IE1vbm8iPjxicj4K ICAgICAgICAgIDwvZm9udD48L2Rpdj4KICAgICAgICA8ZGl2IGlkPSJibG9vcF9jdXN0b21mb250 IiBzdHlsZT0iY29sb3I6cmdiKDAsMCwwKTttYXJnaW46MHB4Ij48Zm9udAogICAgICAgICAgICBm YWNlPSJJbnB1dCBNb25vIj5JIGhhdmUgcmVjZW50bHkgaW5zdGFsbGVkIG11bHRpcGxlIFJlZCBI YXQKICAgICAgICAgICAgVmlydHVhbGl6YXRpb24gaG9zdHMgZm9yIHNldmVyYWwgZGlmZmVyZW50 IGNvbXBhbmllcywgYW5kCiAgICAgICAgICAgIGhhdmUgZGVhbHQgd2l0aCB0aGUgUmVkIEhhdCBT dXBwb3J0IFRlYW0gaW4gZGVwdGggYWJvdXQKICAgICAgICAgICAgb3B0aW1hbCBjb25maWd1cmF0 aW9uIGluIHJlZ2FyZHMgdG8gc2V0dGluZyB1cCBHbHVzdGVyRlMKICAgICAgICAgICAgbW9zdCBl ZmZpY2llbnRseSBhbmQgSSB3YW50ZWQgdG8gc2hhcmUgd2l0aCB5b3Ugd2hhdCBJCiAgICAgICAg ICAgIGxlYXJuZWQuPC9mb250PjwvZGl2PgogICAgICAgIDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZv bnQiIHN0eWxlPSJjb2xvcjpyZ2IoMCwwLDApO21hcmdpbjowcHgiPjxmb250CiAgICAgICAgICAg IGZhY2U9IklucHV0IE1vbm8iPjxicj4KICAgICAgICAgIDwvZm9udD48L2Rpdj4KICAgICAgICA8 ZGl2IGlkPSJibG9vcF9jdXN0b21mb250IiBzdHlsZT0iY29sb3I6cmdiKDAsMCwwKTttYXJnaW46 MHB4Ij48Zm9udAogICAgICAgICAgICBmYWNlPSJJbnB1dCBNb25vIj5JbiBnZW5lcmFsIFJlZCBI YXQgVmlydHVhbGl6YXRpb24gdGVhbQogICAgICAgICAgICBmcm93bnMgdXBvbiB1c2luZyBlYWNo IERJU0sgb2YgdGhlIHN5c3RlbSBhcyBqdXN0IGEgSkJPRCwKICAgICAgICAgICAgc3VyZSB0aGVy ZSBpcyBzb21lIHByb3RlY3Rpb24gYnkgaGF2aW5nIHRoZSBkYXRhIHJlcGxpY2F0ZWQsCiAgICAg ICAgICAgIGhvd2V2ZXIsIHRoZSByZWNvbW1lbmRhdGlvbiBpcyB0byB1c2UgUkFJRCA2IChwcmVm ZXJyZWQpIG9yCiAgICAgICAgICAgIFJBSUQtNSwgb3IgYXQgbGVhc3QgUkFJRC0xIGF0IHRoZSB2 ZXJ5IGxlYXN0LjwvZm9udD48L2Rpdj4KICAgICAgICA8ZGl2IGlkPSJibG9vcF9jdXN0b21mb250 IiBzdHlsZT0iY29sb3I6cmdiKDAsMCwwKTttYXJnaW46MHB4Ij48Zm9udAogICAgICAgICAgICBm YWNlPSJJbnB1dCBNb25vIj48YnI+CiAgICAgICAgICA8L2ZvbnQ+PC9kaXY+CiAgICAgICAgPGRp diBpZD0iYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9Im1hcmdpbjowcHgiPjxmb250IGZhY2U9Iklu cHV0CiAgICAgICAgICAgIE1vbm8iPkhlcmUgaXMgdGhlwqBkaXJlY3QgcXVvdGUgZnJvbSBSZWQg SGF0IHdoZW4gSSBhc2tlZAogICAgICAgICAgICBhYm91dCBSQUlEIGFuZCBCcmlja3M6PC9mb250 PjwvZGl2PgogICAgICAgIDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJtYXJnaW46 MHB4Ij48Zm9udCBmYWNlPSJJbnB1dAogICAgICAgICAgICBNb25vIj48aT48YnI+CiAgICAgICAg ICAgIDwvaT48L2ZvbnQ+PC9kaXY+CiAgICAgICAgPGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9udCIg c3R5bGU9Im1hcmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0CiAgICAgICAgICAgIE1vbm8iPjxp PiJBIHR5cGljYWwgR2x1c3RlciBjb25maWd1cmF0aW9uIHdvdWxkIHVzZSBSQUlECiAgICAgICAg ICAgICAgdW5kZXJuZWF0aCB0aGUgYnJpY2tzLiBSQUlEIDYgaXMgbW9zdCB0eXBpY2FsIGFzIGl0 IGdpdmVzCiAgICAgICAgICAgICAgeW91IDIgZGlzayBmYWlsdXJlIHByb3RlY3Rpb24sIGJ1dCBS QUlEIDUgY291bGQgYmUgdXNlZAogICAgICAgICAgICAgIHRvby7CoE9uY2UgeW91IGhhdmUgdGhl IFJBSURlZCBicmlja3MsIHlvdSdkIHRoZW4gYXBwbHkgdGhlCiAgICAgICAgICAgICAgZGVzaXJl ZCByZXBsaWNhdGlvbiBvbiB0b3Agb2YgdGhhdC4gVGhlIG1vc3QgcG9wdWxhciB3YXkKICAgICAg ICAgICAgICBvZiBkb2luZyB0aGlzIHdvdWxkIGJlIGRpc3RyaWJ1dGVkIHJlcGxpY2F0ZWQgd2l0 aCAyeAogICAgICAgICAgICAgIHJlcGxpY2F0aW9uLsKgSW4gZ2VuZXJhbCB5b3UnbGwgZ2V0IGJl dHRlciBwZXJmb3JtYW5jZSB3aXRoCiAgICAgICAgICAgICAgbGFyZ2VyIGJyaWNrcy4gMTIgZHJp dmVzIGlzIG9mdGVuIGEgc3dlZXQgc3BvdC4gQW5vdGhlcgogICAgICAgICAgICAgIG9wdGlvbiB3 b3VsZCBiZSB0byBjcmVhdGUgYSBzZXBhcmF0ZSB0aWVyIHVzaW5nIGFsbAogICAgICAgICAgICAg IFNTROKAmXMu4oCdwqA8L2k+PC9mb250PjwvZGl2PgogICAgICAgIDxkaXYgaWQ9ImJsb29wX2N1 c3RvbWZvbnQiIHN0eWxlPSJtYXJnaW46MHB4Ij48YnI+CiAgICAgICAgPC9kaXY+CiAgICAgICAg PGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9Im1hcmdpbjowcHgiPjxmb250IGZhY2U9 IklucHV0CiAgICAgICAgICAgIE1vbm8iPjxpPkluIG9yZGVyIHRvIFNTRMKgdGllcmluZyBmcm9t IG15IHVuZGVyc3RhbmRpbmcgeW91CiAgICAgICAgICAgICAgd291bGQgbmVlZCAxIHggTlZNZSBk cml2ZSBpbiBlYWNoIHNlcnZlciwgb3IgNCB4IFNTRCBob3QKICAgICAgICAgICAgICB0aWVyIChp dCBuZWVkcyB0byBiZSBkaXN0cmlidXRlZCwgcmVwbGljYXRlZCBmb3IgdGhlIGhvdAogICAgICAg ICAgICAgIHRpZXIgaWYgbm90IHVzaW5nIE5WTUUpLiBTbyB3aXRoIHlvdSBvbmx5IGhhdmluZyAx IFNTRAogICAgICAgICAgICAgIGRyaXZlIGluIGVhY2ggc2VydmVyLMKgSeKAmWQgc3VnZ2VzdCBt YXliZSBsb29raW5nIGludG8gdGhlCiAgICAgICAgICAgICAgTlZNRSBvcHRpb24uwqA8L2k+PC9m b250PjwvZGl2PgogICAgICAgIDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJtYXJn aW46MHB4Ij48Zm9udCBmYWNlPSJJbnB1dAogICAgICAgICAgICBNb25vIj48aT48YnI+CiAgICAg ICAgICAgIDwvaT48L2ZvbnQ+PC9kaXY+CiAgICAgICAgPGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9u dCIgc3R5bGU9Im1hcmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0CiAgICAgICAgICAgIE1vbm8i PjxpPlNpbmNlIHlvdXIgdXNpbmcgb25seSAzLXNlcnZlcnMsIHdoYXQgSeKAmWQgcHJvYmFibHkK ICAgICAgICAgICAgICBzdWdnZXN0IGlzIHRvIGRvICgyIFJlcGxpY2FzICsgQXJiaXRlciBOb2Rl KSwgdGhpcyBzZXR1cAogICAgICAgICAgICAgIGFjdHVhbGx5IGRvZXNu4oCZdCByZXF1aXJlIHRo ZSAzcmQgc2VydmVyIHRvIGhhdmUgYmlnIGRyaXZlcwogICAgICAgICAgICAgIGF0IGFsbCBhcyBp dCBvbmx5IHN0b3JlcyBtZXRhLWRhdGEgYWJvdXQgdGhlIGZpbGVzIGFuZCBub3QKICAgICAgICAg ICAgICBhY3R1YWxseSBhIGZ1bGwgY29weS7CoDwvaT48L2ZvbnQ+PC9kaXY+CiAgICAgICAgPGRp diBpZD0iYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9Im1hcmdpbjowcHgiPjxmb250IGZhY2U9Iklu cHV0CiAgICAgICAgICAgIE1vbm8iPjxpPjxicj4KICAgICAgICAgICAgPC9pPjwvZm9udD48L2Rp dj4KICAgICAgICA8ZGl2IGlkPSJibG9vcF9jdXN0b21mb250IiBzdHlsZT0ibWFyZ2luOjBweCI+ PGZvbnQgZmFjZT0iSW5wdXQKICAgICAgICAgICAgTW9ubyI+PGk+UGxlYXNlIHNlZSB0aGUgYXR0 YWNoZWQgZG9jdW1lbnQgdGhhdCB3YXMgZ2l2ZW4gdG8KICAgICAgICAgICAgICBtZSBieSBSZWQg SGF0IHRvIGdldCBtb3JlIGluZm9ybWF0aW9uIG9uIHRoaXMuIEhvcGUgdGhpcwogICAgICAgICAg ICAgIGluZm9ybWF0aW9uIGhlbHBzIHlvdS48L2k+PC9mb250PjwvZGl2PgogICAgICAgIDxkaXYg aWQ9ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJtYXJnaW46MHB4Ij48Zm9udCBmYWNlPSJJbnB1 dAogICAgICAgICAgICBNb25vIj48aT48YnI+CiAgICAgICAgICAgIDwvaT48L2ZvbnQ+PC9kaXY+ CiAgICAgICAgPGJyPgogICAgICAgIDxkaXYgaWQ9ImJsb29wX3NpZ25fMTUwMjA4NzM3NjcyNTQ2 OTE4NCIgY2xhc3M9ImJsb29wX3NpZ24iPjxzcGFuCiAgICAgICAgICAgIHN0eWxlPSJmb250LWZh bWlseTonaGVsdmV0aWNhCiAgICAgICAgICAgIE5ldWUnLGhlbHZldGljYTtmb250LXNpemU6MTRw eCI+LS08L3NwYW4+PGJyCiAgICAgICAgICAgIHN0eWxlPSJmb250LWZhbWlseTonaGVsdmV0aWNh CiAgICAgICAgICAgIE5ldWUnLGhlbHZldGljYTtmb250LXNpemU6MTRweCI+CiAgICAgICAgICA8 ZGl2IGNsYXNzPSJnbWFpbF9zaWduYXR1cmUiIHN0eWxlPSJmb250LWZhbWlseTonaGVsdmV0aWNh CiAgICAgICAgICAgIE5ldWUnLGhlbHZldGljYTtmb250LXNpemU6MTRweCI+CiAgICAgICAgICAg IDxkaXYgZGlyPSJsdHIiPgogICAgICAgICAgICAgIDxkaXY+PGJyPgogICAgICAgICAgICAgIDwv ZGl2PgogICAgICAgICAgICAgIDxkaXY+RGV2aW4gQWNvc3RhLCBSSENBLCBSSFZDQTwvZGl2Pgog ICAgICAgICAgICAgIDxkaXY+UmVkIEhhdCBDZXJ0aWZpZWQgQXJjaGl0ZWN0PC9kaXY+CiAgICAg ICAgICAgIDwvZGl2PgogICAgICAgICAgPC9kaXY+CiAgICAgICAgPC9kaXY+CiAgICAgICAgPGJy PgogICAgICAgIDxwIGNsYXNzPSJhaXJtYWlsX29uIj5PbiBBdWd1c3QgNiwgMjAxNyBhdCA3OjI5 OjI5IFBNLCBNb2FjaXIKICAgICAgICAgIEZlcnJlaXJhICg8YSBocmVmPSJtYWlsdG86bW9hY2ly ZmVycmVpcmFAaG90bWFpbC5jb20iCiAgICAgICAgICAgIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+ bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb208L2E+KQogICAgICAgICAgd3JvdGU6PC9wPgogICAg ICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiIGNsYXNzPSJjbGVhbl9icSI+PHNwYW4+CiAgICAg ICAgICAgIDxkaXYgZGlyPSJsdHIiPgogICAgICAgICAgICAgIDxkaXY+CiAgICAgICAgICAgICAg ICA8dGl0bGU+PC90aXRsZT4KICAgICAgICAgICAgICAgIDxkaXYgaWQ9ImRpdnRhZ2RlZmF1bHR3 cmFwcGVyIgpzdHlsZT0iZm9udC1zaXplOjEycHQ7Y29sb3I6IzAwMDAwMDtmb250LWZhbWlseTpD YWxpYnJpLEhlbHZldGljYSxzYW5zLXNlcmlmIgogICAgICAgICAgICAgICAgICBkaXI9Imx0ciI+ CiAgICAgICAgICAgICAgICAgIDxwPjxzcGFuPkkgYW0gd2lsbGluZyB0byBhc3NlbWJsZSBhIG9W aXJ0ICJwb2QiLCBtYWRlCiAgICAgICAgICAgICAgICAgICAgICBvZiAzIHNlcnZlcnMsIGVhY2gg d2l0aCAyIENQVSBzb2NrZXRzIG9mIDEyIGNvcmVzLAogICAgICAgICAgICAgICAgICAgICAgMjU2 R0IgUkFNLCA3IEhERCAxMEssIDEgU1NELiBUaGUgaWRlYSBpcyB0byB1c2UKICAgICAgICAgICAg ICAgICAgICAgIEdsdXN0ZXJGUyB0byBwcm92aWRlIEhBIGZvciB0aGUgVk1zLiBUaGUgMyBzZXJ2 ZXJzCiAgICAgICAgICAgICAgICAgICAgICBoYXZlIGEgZHVhbCA0MEdiIE5JQyBhbmQgYSBkdWFs IDEwR2IgTklDLiBTbyBteQogICAgICAgICAgICAgICAgICAgICAgaW50ZW50aW9uIGlzIHRvIGNy ZWF0ZSBhIGxvb3AgbGlrZSBhIHNlcnZlcgogICAgICAgICAgICAgICAgICAgICAgdHJpYW5nbGUg dXNpbmcgdGhlIDQwR2IgTklDcyBmb3IgdmlydHVhbGl6YXRpb24KICAgICAgICAgICAgICAgICAg ICAgIGZpbGVzIChWTXMgLnFjb3cyKSBhY2Nlc3MgYW5kIHRvIG1vdmUgVk1zIGFyb3VuZAogICAg ICAgICAgICAgICAgICAgICAgdGhlIHBvZCAoZWFzdCAvd2VzdCB0cmFmZmljKSB3aGlsZSB1c2lu ZyB0aGUgMTBHYgogICAgICAgICAgICAgICAgICAgICAgaW50ZXJmYWNlcyBmb3IgZ2l2aW5nIHNl cnZpY2VzIHRvIHRoZSBvdXRzaWRlCiAgICAgICAgICAgICAgICAgICAgICB3b3JsZCAobm9ydGgv c291dGggdHJhZmZpYykuPC9zcGFuPjwvcD4KICAgICAgICAgICAgICAgICAgPHA+PGJyPgogICAg ICAgICAgICAgICAgICA8L3A+CiAgICAgICAgICAgICAgICAgIDxwPlRoaXMgc2FpZCwgbXkgZmly c3QgcXVlc3Rpb24gaXM6IEhvdyBzaG91bGQgSQogICAgICAgICAgICAgICAgICAgIGRlcGxveSBH bHVzdGVyRlMgaW4gc3VjaCBvVmlydCBzY2VuYXJpbz8gTXkKICAgICAgICAgICAgICAgICAgICBx dWVzdGlvbnMgYXJlOjwvcD4KICAgICAgICAgICAgICAgICAgPHA+PGJyPgogICAgICAgICAgICAg ICAgICA8L3A+CiAgICAgICAgICAgICAgICAgIDxwPjEgLSBTaG91bGQgSSBjcmVhdGUgMyBSQUlE IChpLmUuOiBSQUlEIDUpLCBvbmUgb24KICAgICAgICAgICAgICAgICAgICBlYWNoIG9WaXJ0IG5v ZGUsIGFuZCB0aGVuIGNyZWF0ZSBhIEdsdXN0ZXJGUyB1c2luZwogICAgICAgICAgICAgICAgICAg IHRoZW0/PC9wPgogICAgICAgICAgICAgICAgICA8cD4yIC0gSW5zdGVhZCwgc2hvdWxkIEkgY3Jl YXRlIGEgSkJPRCBhcnJheSBtYWRlIG9mCiAgICAgICAgICAgICAgICAgICAgYWxsIHNlcnZlcidz IGRpc2tzPzwvcD4KICAgICAgICAgICAgICAgICAgPHA+MyAtIFdoYXQgaXMgdGhlIGJlc3QgR2x1 c3RlciBjb25maWd1cmF0aW9uIHRvCiAgICAgICAgICAgICAgICAgICAgcHJvdmlkZSBmb3IgSEEg d2hpbGUgbm90IGNvbnN1bWluZyB0b28gbXVjaCBkaXNrCiAgICAgICAgICAgICAgICAgICAgc3Bh Y2U/PGJyPgogICAgICAgICAgICAgICAgICA8L3A+CiAgICAgICAgICAgICAgICAgIDxwPjQgLSBE b2VzIGEgb1ZpcnQgaHlwZXJ2aXNvciBwb2QgbGlrZSBJIGFtIHBsYW5uaW5nCiAgICAgICAgICAg ICAgICAgICAgdG8gYnVpbGQsIGFuZCB0aGUgdmlydHVhbGl6YXRpb24gZW52aXJvbm1lbnQsCiAg ICAgICAgICAgICAgICAgICAgYmVuZWZpdHMgZnJvbSB0aWVyaW5nIHdoZW4gdXNpbmcgYSBTU0Qg ZGlzaz8gQW5kCiAgICAgICAgICAgICAgICAgICAgeWVzLCB3aWxsIEdsdXN0ZXIgZG8gaXQgYnkg ZGVmYXVsdCBvciBJIGhhdmUgdG8KICAgICAgICAgICAgICAgICAgICBjb25maWd1cmUgaXQgdG8g ZG8gc28/PC9wPgogICAgICAgICAgICAgICAgICA8cD48YnI+CiAgICAgICAgICAgICAgICAgIDwv cD4KICAgICAgICAgICAgICAgICAgPHA+QXQgdGhlIGJvdHRvbSBsaW5lLCB3aGF0IGlzIHRoZSBn b29kIHByYWN0aWNlIGZvcgogICAgICAgICAgICAgICAgICAgIHVzaW5nIEdsdXN0ZXJGUyBpbiBz bWFsbCBwb2RzIGZvciBlbnRlcnByaXNlcz88YnI+CiAgICAgICAgICAgICAgICAgIDwvcD4KICAg ICAgICAgICAgICAgICAgPHA+PGJyPgogICAgICAgICAgICAgICAgICA8L3A+CiAgICAgICAgICAg ICAgICAgIDxwPllvdSBvcGluaW9uL2ZlZWRiYWNrIHdpbGwgYmUgcmVhbGx5IGFwcHJlY2lhdGVk ITwvcD4KICAgICAgICAgICAgICAgICAgPHA+TW9hY2lyPGJyPgogICAgICAgICAgICAgICAgICA8 L3A+CiAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgIF9fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fIDxicj4KICAgICAgICAgICAgICAgIFVz ZXJzIG1haWxpbmcgbGlzdCA8YnI+CiAgICAgICAgICAgICAgICA8YSBocmVmPSJtYWlsdG86VXNl cnNAb3ZpcnQub3JnIiBtb3otZG8tbm90LXNlbmQ9InRydWUiPlVzZXJzQG92aXJ0Lm9yZzwvYT4K ICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgIDxhIGhyZWY9Imh0dHA6Ly9saXN0 cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyIKICAgICAgICAgICAgICAgICAgbW96 LWRvLW5vdC1zZW5kPSJ0cnVlIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGlu Zm8vdXNlcnM8L2E+CiAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgPC9kaXY+CiAg ICAgICAgICAgIDwvZGl2PgogICAgICAgICAgPC9zcGFuPjwvYmxvY2txdW90ZT4KICAgICAgICA8 YnI+CiAgICAgICAgPGZpZWxkc2V0IGNsYXNzPSJtaW1lQXR0YWNobWVudEhlYWRlciI+PC9maWVs ZHNldD4KICAgICAgICA8YnI+CiAgICAgICAgPHByZSB3cmFwPSIiPl9fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fClVzZXJzIG1haWxpbmcgbGlzdAo8YSBjbGFz cz0ibW96LXR4dC1saW5rLWFiYnJldmlhdGVkIiBocmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQub3Jn IiBtb3otZG8tbm90LXNlbmQ9InRydWUiPlVzZXJzQG92aXJ0Lm9yZzwvYT4KPGEgY2xhc3M9Im1v ei10eHQtbGluay1mcmVldGV4dCIgaHJlZj0iaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFu L2xpc3RpbmZvL3VzZXJzIiBtb3otZG8tbm90LXNlbmQ9InRydWUiPmh0dHA6Ly9saXN0cy5vdmly dC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VyczwvYT4KPC9wcmU+CiAgICAgIDwvYmxvY2txdW90 ZT4KICAgICAgPGJyPgogICAgICA8YnI+CiAgICAgIDxmaWVsZHNldCBjbGFzcz0ibWltZUF0dGFj aG1lbnRIZWFkZXIiPjwvZmllbGRzZXQ+CiAgICAgIDxicj4KICAgICAgPHByZSB3cmFwPSIiPl9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fClVzZXJzIG1haWxp bmcgbGlzdAo8YSBjbGFzcz0ibW96LXR4dC1saW5rLWFiYnJldmlhdGVkIiBocmVmPSJtYWlsdG86 VXNlcnNAb3ZpcnQub3JnIj5Vc2Vyc0BvdmlydC5vcmc8L2E+CjxhIGNsYXNzPSJtb3otdHh0LWxp bmstZnJlZXRleHQiIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5m by91c2VycyI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9h Pgo8L3ByZT4KICAgIDwvYmxvY2txdW90ZT4KICAgIDxicj4KICA8L2JvZHk+CjwvaHRtbD4KCi0t LS0tLS0tLS0tLS0tQzJGMzczNjJDRTYzOUYzNkU0NEVCQkQ4LS0K --===============0412783989436684471==-- From fernando.frediani at upx.com Mon Aug 7 15:41:24 2017 Content-Type: multipart/mixed; boundary="===============3959530968213590397==" MIME-Version: 1.0 From: FERNANDO FREDIANI To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Mon, 07 Aug 2017 12:41:17 -0300 Message-ID: In-Reply-To: aa829d07-fa77-3ed9-2500-e33cc01414b6@recogizer.de --===============3959530968213590397== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------A5088546CDF99006314CCEE8 Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 8bit Thanks for the clarification Erekle. However I get surprised with this way of operating from GlusterFS as it = adds another layer of complexity to the system (either a hardware or = software RAID) before the gluster config and increase the system's = overall costs. An important point to consider is: In RAID configuration you already = have space 'wasted' in order to build redundancy (either RAID 1, 5, or = 6). Then when you have GlusterFS on the top of several RAIDs you have = again more data replicated so you end up with the same data consuming = more space in a group of disks and again on the top of several RAIDs = depending on the Gluster configuration you have (in a RAID 1 config the = same data is replicated 4 times). Yet another downside of having a RAID (specially RAID 5 or 6) is that it = reduces considerably the write speeds as each group of disks will end up = having the write speed of a single disk as all other disks of that group = have to wait for each other to write as well. Therefore if Gluster already replicates data why does it create this big = pain you mentioned if the data is replicated somewhere else, can still = be retrieved to both serve clients and reconstruct the equivalent disk = when it is replaced ? Fernando On 07/08/2017 10:26, Erekle Magradze wrote: > > Hi Frenando, > > Here is my experience, if you consider a particular hard drive as a = > brick for gluster volume and it dies, i.e. it becomes not accessible = > it's a huge hassle to discard that brick and exchange with another = > one, since gluster some tries to access that broken brick and it's = > causing (at least it cause for me) a big pain, therefore it's better = > to have a RAID as brick, i.e. have RAID 1 (mirroring) for each brick, = > in this case if the disk is down you can easily exchange it and = > rebuild the RAID without going offline, i.e switching off the volume = > doing brick manipulations and switching it back on. > > Cheers > > Erekle > > > On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote: >> >> For any RAID 5 or 6 configuration I normally follow a simple gold = >> rule which gave good results so far: >> - up to 4 disks RAID 5 >> - 5 or more disks RAID 6 >> >> However I didn't really understand well the recommendation to use any = >> RAID with GlusterFS. I always thought that GlusteFS likes to work in = >> JBOD mode and control the disks (bricks) directlly so you can create = >> whatever distribution rule you wish, and if a single disk fails you = >> just replace it and which obviously have the data replicated from = >> another. The only downside of using in this way is that the = >> replication data will be flow accross all servers but that is not = >> much a big issue. >> >> Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS. >> >> Thanks >> Regards >> Fernando >> >> >> On 07/08/2017 03:46, Devin Acosta wrote: >>> >>> Moacir, >>> >>> I have recently installed multiple Red Hat Virtualization hosts for = >>> several different companies, and have dealt with the Red Hat Support = >>> Team in depth about optimal configuration in regards to setting up = >>> GlusterFS most efficiently and I wanted to share with you what I = >>> learned. >>> >>> In general Red Hat Virtualization team frowns upon using each DISK = >>> of the system as just a JBOD, sure there is some protection by = >>> having the data replicated, however, the recommendation is to use = >>> RAID 6 (preferred) or RAID-5, or at least RAID-1 at the very least. >>> >>> Here is the direct quote from Red Hat when I asked about RAID and = >>> Bricks: >>> / >>> / >>> /"A typical Gluster configuration would use RAID underneath the = >>> bricks. RAID 6 is most typical as it gives you 2 disk failure = >>> protection, but RAID 5 could be used too. Once you have the RAIDed = >>> bricks, you'd then apply the desired replication on top of that. The = >>> most popular way of doing this would be distributed replicated with = >>> 2x replication. In general you'll get better performance with larger = >>> bricks. 12 drives is often a sweet spot. Another option would be to = >>> create a separate tier using all SSD=E2=80=99s.=E2=80=9D / >>> >>> /In order to SSD tiering from my understanding you would need 1 x = >>> NVMe drive in each server, or 4 x SSD hot tier (it needs to be = >>> distributed, replicated for the hot tier if not using NVME). So with = >>> you only having 1 SSD drive in each server, I=E2=80=99d suggest maybe = >>> looking into the NVME option. / >>> / >>> / >>> /Since your using only 3-servers, what I=E2=80=99d probably suggest is = to do = >>> (2 Replicas + Arbiter Node), this setup actually doesn=E2=80=99t requir= e the = >>> 3rd server to have big drives at all as it only stores meta-data = >>> about the files and not actually a full copy. / >>> / >>> / >>> /Please see the attached document that was given to me by Red Hat to = >>> get more information on this. Hope this information helps you./ >>> / >>> / >>> >>> -- >>> >>> Devin Acosta, RHCA, RHVCA >>> Red Hat Certified Architect >>> >>> On August 6, 2017 at 7:29:29 PM, Moacir Ferreira = >>> (moacirferreira(a)hotmail.com ) wr= ote: >>> >>>> I am willing to assemble a oVirt "pod", made of 3 servers, each = >>>> with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The = >>>> idea is to use GlusterFS to provide HA for the VMs. The 3 servers = >>>> have a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to = >>>> create a loop like a server triangle using the 40Gb NICs for = >>>> virtualization files (VMs .qcow2) access and to move VMs around the = >>>> pod (east /west traffic) while using the 10Gb interfaces for giving = >>>> services to the outside world (north/south traffic). >>>> >>>> >>>> This said, my first question is: How should I deploy GlusterFS in = >>>> such oVirt scenario? My questions are: >>>> >>>> >>>> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, = >>>> and then create a GlusterFS using them? >>>> >>>> 2 - Instead, should I create a JBOD array made of all server's disks? >>>> >>>> 3 - What is the best Gluster configuration to provide for HA while = >>>> not consuming too much disk space? >>>> >>>> 4 - Does a oVirt hypervisor pod like I am planning to build, and = >>>> the virtualization environment, benefits from tiering when using a = >>>> SSD disk? And yes, will Gluster do it by default or I have to = >>>> configure it to do so? >>>> >>>> >>>> At the bottom line, what is the good practice for using GlusterFS = >>>> in small pods for enterprises? >>>> >>>> >>>> You opinion/feedback will be really appreciated! >>>> >>>> Moacir >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >> >> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users > --------------A5088546CDF99006314CCEE8 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit
Thanks for the clarification Erekle.

However I get surprised with this way of operating from GlusterFS as it adds another layer of complexity to the system (either a hardware or software RAID) before the gluster config and increase the system's overall costs.

An important point to consider is: In RAID configuration you already have space 'wasted' in order to build redundancy (either RAID 1, 5, or 6). Then when you have GlusterFS on the top of several RAIDs you have again more data replicated so you end up with the same data consuming more space in a group of disks and again on the top of several RAIDs depending on the Gluster configuration you have (in a RAID 1 config the same data is replicated 4 times).

Yet another downside of having a RAID (specially RAID 5 or 6) is that it reduces considerably the write speeds as each group of disks will end up having the write speed of a single disk as all other disks of that group have to wait for each other to write as well.

Therefore if Gluster already replicates data why does it create this big pain you mentioned if the data is replicated somewhere else, can still be retrieved to both serve clients and reconstruct the equivalent disk when it is replaced ?

Fernando

On 07/08/2017 10:26, Erekle Magradze wrote:

Hi Frenando,

Here is my experience, if you consider a particular hard drive as a brick for gluster volume and it dies, i.e. it becomes not accessible it's a huge hassle to discard that brick and exchange with another one, since gluster some tries to access that broken brick and it's causing (at least it cause for me) a big pain, therefore it's better to have a RAID as brick, i.e. have RAID 1 (mirroring) for each brick, in this case if the disk is down you can easily exchange it and rebuild the RAID without going offline, i.e switching off the volume doing brick manipulations and switching it back on.

Cheers

Erekle

On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote:

For any RAID 5 or 6 configuration I normally follow a simple gold rule which gave good results so far:
- up to 4 disks RAID 5
- 5 or more disks RAID 6

However I didn't really understand well the recommendation to use any RAID with GlusterFS. I always thought that GlusteFS likes to work in JBOD mode and control the disks (bricks) directlly so you can create whatever distribution rule you wish, and if a single disk fails you just replace it and which obviously have the data replicated from another. The only downside of using in this way is that the replication data will be flow accross all servers but that is not much a big issue.

Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS.

Thanks
Regards
Fernando

On 07/08/2017 03:46, Devin Acosta wrote:

Moacir,

I have recently installed multiple Red Hat Virtualization hosts for several different companies, and have dealt with the Red Hat Support Team in depth about optimal configuration in regards to setting up GlusterFS most efficiently and I wanted to share with you what I learned.

In general Red Hat Virtualization team frowns upon using each DISK of the system as just a JBOD, sure there is some protection by having the data replicated, however, the recommendation is to use RAID 6 (preferred) or RAID-5, or at least RAID-1 at the very least.

Here is the=C2=A0direct quote from Red Hat when I asked about RAID and Bricks:

"A typical Gluster configuration would use RAID underneath the bricks. RAID 6 is most typical as it gives you 2 disk failure protection, but RAID 5 could be used too.=C2=A0Once you have the RAIDed bricks, you'd then apply the desired replication on top of that. The most popular way of doing this would be distributed replicated with 2x replication.=C2=A0In general you'll get better performance with larger bricks. 12 drives is often a sweet spot. Another option would be to create a separate tier using all SSD=E2=80=99s.=E2=80=9D=C2= =A0

In order to SSD=C2=A0tiering from my understanding you would need 1 x NVMe drive in each server, or 4 x SSD hot tier (it needs to be distributed, replicated for the hot tier if not using NVME). So with you only having 1 SSD drive in each server,=C2=A0I=E2=80=99= d suggest maybe looking into the NVME option.=C2=A0

Since your using only 3-servers, what I=E2=80=99d probably suggest is to do (2 Replicas + Arbiter Node), this setup actually doesn=E2=80=99t require the 3rd server to have big drives at all as it only stores meta-data about the files and not actually a full copy.=C2= =A0

Please see the attached document that was given to me by Red Hat to get more information on this. Hope this information helps you.

<= span style=3D"font-family:'helvetica Neue',helvetica;font-size:14px">--

Devin Acosta, RHCA, RHVCA

Red Hat Certified Architect

On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (moacirferreira(a)hotmail.com) wrote:

I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop like a server triangle using the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VMs around the pod (east /west traffic) while using the 10Gb interfaces for giving services to the outside world (north/south traffic).

This said, my first question is: How should I deploy GlusterFS in such oVirt scenario? My questions are:

1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then create a GlusterFS using them?

2 - Instead, should I create a JBOD array made of all server's disks?

3 - What is the best Gluster configuration to provide for HA while not consuming too much disk space?

4 - Does a oVirt hypervisor pod like I am planning to build, and the virtualization environment, benefits from tiering when using a SSD disk? And yes, will Gluster do it by default or I have to configure it to do so?

At the bottom line, what is the good practice for using GlusterFS in small pods for enterprises?

You opinion/feedback will be really appreciated!

Moacir

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman= /listinfo/users

_______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/list= info/users

_______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/list= info/users

--------------A5088546CDF99006314CCEE8-- --===============3959530968213590397== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS1BNTA4ODU0NkNERjk5MDA2MzE0Q0NFRTgKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDhiaXQK ClRoYW5rcyBmb3IgdGhlIGNsYXJpZmljYXRpb24gRXJla2xlLgoKSG93ZXZlciBJIGdldCBzdXJw cmlzZWQgd2l0aCB0aGlzIHdheSBvZiBvcGVyYXRpbmcgZnJvbSBHbHVzdGVyRlMgYXMgaXQgCmFk ZHMgYW5vdGhlciBsYXllciBvZiBjb21wbGV4aXR5IHRvIHRoZSBzeXN0ZW0gKGVpdGhlciBhIGhh cmR3YXJlIG9yIApzb2Z0d2FyZSBSQUlEKSBiZWZvcmUgdGhlIGdsdXN0ZXIgY29uZmlnIGFuZCBp bmNyZWFzZSB0aGUgc3lzdGVtJ3MgCm92ZXJhbGwgY29zdHMuCgpBbiBpbXBvcnRhbnQgcG9pbnQg dG8gY29uc2lkZXIgaXM6IEluIFJBSUQgY29uZmlndXJhdGlvbiB5b3UgYWxyZWFkeSAKaGF2ZSBz cGFjZSAnd2FzdGVkJyBpbiBvcmRlciB0byBidWlsZCByZWR1bmRhbmN5IChlaXRoZXIgUkFJRCAx LCA1LCBvciAKNikuIFRoZW4gd2hlbiB5b3UgaGF2ZSBHbHVzdGVyRlMgb24gdGhlIHRvcCBvZiBz ZXZlcmFsIFJBSURzIHlvdSBoYXZlIAphZ2FpbiBtb3JlIGRhdGEgcmVwbGljYXRlZCBzbyB5b3Ug ZW5kIHVwIHdpdGggdGhlIHNhbWUgZGF0YSBjb25zdW1pbmcgCm1vcmUgc3BhY2UgaW4gYSBncm91 cCBvZiBkaXNrcyBhbmQgYWdhaW4gb24gdGhlIHRvcCBvZiBzZXZlcmFsIFJBSURzIApkZXBlbmRp bmcgb24gdGhlIEdsdXN0ZXIgY29uZmlndXJhdGlvbiB5b3UgaGF2ZSAoaW4gYSBSQUlEIDEgY29u ZmlnIHRoZSAKc2FtZSBkYXRhIGlzIHJlcGxpY2F0ZWQgNCB0aW1lcykuCgpZZXQgYW5vdGhlciBk b3duc2lkZSBvZiBoYXZpbmcgYSBSQUlEIChzcGVjaWFsbHkgUkFJRCA1IG9yIDYpIGlzIHRoYXQg aXQgCnJlZHVjZXMgY29uc2lkZXJhYmx5IHRoZSB3cml0ZSBzcGVlZHMgYXMgZWFjaCBncm91cCBv ZiBkaXNrcyB3aWxsIGVuZCB1cCAKaGF2aW5nIHRoZSB3cml0ZSBzcGVlZCBvZiBhIHNpbmdsZSBk aXNrIGFzIGFsbCBvdGhlciBkaXNrcyBvZiB0aGF0IGdyb3VwIApoYXZlIHRvIHdhaXQgZm9yIGVh Y2ggb3RoZXIgdG8gd3JpdGUgYXMgd2VsbC4KClRoZXJlZm9yZSBpZiBHbHVzdGVyIGFscmVhZHkg cmVwbGljYXRlcyBkYXRhIHdoeSBkb2VzIGl0IGNyZWF0ZSB0aGlzIGJpZyAKcGFpbiB5b3UgbWVu dGlvbmVkIGlmIHRoZSBkYXRhIGlzIHJlcGxpY2F0ZWQgc29tZXdoZXJlIGVsc2UsIGNhbiBzdGls bCAKYmUgcmV0cmlldmVkIHRvIGJvdGggc2VydmUgY2xpZW50cyBhbmQgcmVjb25zdHJ1Y3QgdGhl IGVxdWl2YWxlbnQgZGlzayAKd2hlbiBpdCBpcyByZXBsYWNlZCA/CgpGZXJuYW5kbwoKCk9uIDA3 LzA4LzIwMTcgMTA6MjYsIEVyZWtsZSBNYWdyYWR6ZSB3cm90ZToKPgo+IEhpIEZyZW5hbmRvLAo+ Cj4gSGVyZSBpcyBteSBleHBlcmllbmNlLCBpZiB5b3UgY29uc2lkZXIgYSBwYXJ0aWN1bGFyIGhh cmQgZHJpdmUgYXMgYSAKPiBicmljayBmb3IgZ2x1c3RlciB2b2x1bWUgYW5kIGl0IGRpZXMsIGku ZS4gaXQgYmVjb21lcyBub3QgYWNjZXNzaWJsZSAKPiBpdCdzIGEgaHVnZSBoYXNzbGUgdG8gZGlz Y2FyZCB0aGF0IGJyaWNrIGFuZCBleGNoYW5nZSB3aXRoIGFub3RoZXIgCj4gb25lLCBzaW5jZSBn bHVzdGVyIHNvbWUgdHJpZXMgdG8gYWNjZXNzIHRoYXQgYnJva2VuIGJyaWNrIGFuZCBpdCdzIAo+ IGNhdXNpbmcgKGF0IGxlYXN0IGl0IGNhdXNlIGZvciBtZSkgYSBiaWcgcGFpbiwgdGhlcmVmb3Jl IGl0J3MgYmV0dGVyIAo+IHRvIGhhdmUgYSBSQUlEIGFzIGJyaWNrLCBpLmUuIGhhdmUgUkFJRCAx IChtaXJyb3JpbmcpIGZvciBlYWNoIGJyaWNrLCAKPiBpbiB0aGlzIGNhc2UgaWYgdGhlIGRpc2sg aXMgZG93biB5b3UgY2FuIGVhc2lseSBleGNoYW5nZSBpdCBhbmQgCj4gcmVidWlsZCB0aGUgUkFJ RCB3aXRob3V0IGdvaW5nIG9mZmxpbmUsIGkuZSBzd2l0Y2hpbmcgb2ZmIHRoZSB2b2x1bWUgCj4g ZG9pbmcgYnJpY2sgbWFuaXB1bGF0aW9ucyBhbmQgc3dpdGNoaW5nIGl0IGJhY2sgb24uCj4KPiBD aGVlcnMKPgo+IEVyZWtsZQo+Cj4KPiBPbiAwOC8wNy8yMDE3IDAzOjA0IFBNLCBGRVJOQU5ETyBG UkVESUFOSSB3cm90ZToKPj4KPj4gRm9yIGFueSBSQUlEIDUgb3IgNiBjb25maWd1cmF0aW9uIEkg bm9ybWFsbHkgZm9sbG93IGEgc2ltcGxlIGdvbGQgCj4+IHJ1bGUgd2hpY2ggZ2F2ZSBnb29kIHJl c3VsdHMgc28gZmFyOgo+PiAtIHVwIHRvIDQgZGlza3MgUkFJRCA1Cj4+IC0gNSBvciBtb3JlIGRp c2tzIFJBSUQgNgo+Pgo+PiBIb3dldmVyIEkgZGlkbid0IHJlYWxseSB1bmRlcnN0YW5kIHdlbGwg dGhlIHJlY29tbWVuZGF0aW9uIHRvIHVzZSBhbnkgCj4+IFJBSUQgd2l0aCBHbHVzdGVyRlMuIEkg YWx3YXlzIHRob3VnaHQgdGhhdCBHbHVzdGVGUyBsaWtlcyB0byB3b3JrIGluIAo+PiBKQk9EIG1v ZGUgYW5kIGNvbnRyb2wgdGhlIGRpc2tzIChicmlja3MpIGRpcmVjdGxseSBzbyB5b3UgY2FuIGNy ZWF0ZSAKPj4gd2hhdGV2ZXIgZGlzdHJpYnV0aW9uIHJ1bGUgeW91IHdpc2gsIGFuZCBpZiBhIHNp bmdsZSBkaXNrIGZhaWxzIHlvdSAKPj4ganVzdCByZXBsYWNlIGl0IGFuZCB3aGljaCBvYnZpb3Vz bHkgaGF2ZSB0aGUgZGF0YSByZXBsaWNhdGVkIGZyb20gCj4+IGFub3RoZXIuIFRoZSBvbmx5IGRv d25zaWRlIG9mIHVzaW5nIGluIHRoaXMgd2F5IGlzIHRoYXQgdGhlIAo+PiByZXBsaWNhdGlvbiBk YXRhIHdpbGwgYmUgZmxvdyBhY2Nyb3NzIGFsbCBzZXJ2ZXJzIGJ1dCB0aGF0IGlzIG5vdCAKPj4g bXVjaCBhIGJpZyBpc3N1ZS4KPj4KPj4gQW55b25lIGNhbiBlbGFib3JhdGUgYWJvdXQgVXNpbmcg UkFJRCArIEdsdXN0ZXJGUyBhbmQgSkJPRCArIEdsdXN0ZXJGUy4KPj4KPj4gVGhhbmtzCj4+IFJl Z2FyZHMKPj4gRmVybmFuZG8KPj4KPj4KPj4gT24gMDcvMDgvMjAxNyAwMzo0NiwgRGV2aW4gQWNv c3RhIHdyb3RlOgo+Pj4KPj4+IE1vYWNpciwKPj4+Cj4+PiBJIGhhdmUgcmVjZW50bHkgaW5zdGFs bGVkIG11bHRpcGxlIFJlZCBIYXQgVmlydHVhbGl6YXRpb24gaG9zdHMgZm9yIAo+Pj4gc2V2ZXJh bCBkaWZmZXJlbnQgY29tcGFuaWVzLCBhbmQgaGF2ZSBkZWFsdCB3aXRoIHRoZSBSZWQgSGF0IFN1 cHBvcnQgCj4+PiBUZWFtIGluIGRlcHRoIGFib3V0IG9wdGltYWwgY29uZmlndXJhdGlvbiBpbiBy ZWdhcmRzIHRvIHNldHRpbmcgdXAgCj4+PiBHbHVzdGVyRlMgbW9zdCBlZmZpY2llbnRseSBhbmQg SSB3YW50ZWQgdG8gc2hhcmUgd2l0aCB5b3Ugd2hhdCBJIAo+Pj4gbGVhcm5lZC4KPj4+Cj4+PiBJ biBnZW5lcmFsIFJlZCBIYXQgVmlydHVhbGl6YXRpb24gdGVhbSBmcm93bnMgdXBvbiB1c2luZyBl YWNoIERJU0sgCj4+PiBvZiB0aGUgc3lzdGVtIGFzIGp1c3QgYSBKQk9ELCBzdXJlIHRoZXJlIGlz IHNvbWUgcHJvdGVjdGlvbiBieSAKPj4+IGhhdmluZyB0aGUgZGF0YSByZXBsaWNhdGVkLCBob3dl dmVyLCB0aGUgcmVjb21tZW5kYXRpb24gaXMgdG8gdXNlIAo+Pj4gUkFJRCA2IChwcmVmZXJyZWQp IG9yIFJBSUQtNSwgb3IgYXQgbGVhc3QgUkFJRC0xIGF0IHRoZSB2ZXJ5IGxlYXN0Lgo+Pj4KPj4+ IEhlcmUgaXMgdGhlIGRpcmVjdCBxdW90ZSBmcm9tIFJlZCBIYXQgd2hlbiBJIGFza2VkIGFib3V0 IFJBSUQgYW5kIAo+Pj4gQnJpY2tzOgo+Pj4gLwo+Pj4gLwo+Pj4gLyJBIHR5cGljYWwgR2x1c3Rl ciBjb25maWd1cmF0aW9uIHdvdWxkIHVzZSBSQUlEIHVuZGVybmVhdGggdGhlIAo+Pj4gYnJpY2tz LiBSQUlEIDYgaXMgbW9zdCB0eXBpY2FsIGFzIGl0IGdpdmVzIHlvdSAyIGRpc2sgZmFpbHVyZSAK Pj4+IHByb3RlY3Rpb24sIGJ1dCBSQUlEIDUgY291bGQgYmUgdXNlZCB0b28uIE9uY2UgeW91IGhh dmUgdGhlIFJBSURlZCAKPj4+IGJyaWNrcywgeW91J2QgdGhlbiBhcHBseSB0aGUgZGVzaXJlZCBy ZXBsaWNhdGlvbiBvbiB0b3Agb2YgdGhhdC4gVGhlIAo+Pj4gbW9zdCBwb3B1bGFyIHdheSBvZiBk b2luZyB0aGlzIHdvdWxkIGJlIGRpc3RyaWJ1dGVkIHJlcGxpY2F0ZWQgd2l0aCAKPj4+IDJ4IHJl cGxpY2F0aW9uLiBJbiBnZW5lcmFsIHlvdSdsbCBnZXQgYmV0dGVyIHBlcmZvcm1hbmNlIHdpdGgg bGFyZ2VyIAo+Pj4gYnJpY2tzLiAxMiBkcml2ZXMgaXMgb2Z0ZW4gYSBzd2VldCBzcG90LiBBbm90 aGVyIG9wdGlvbiB3b3VsZCBiZSB0byAKPj4+IGNyZWF0ZSBhIHNlcGFyYXRlIHRpZXIgdXNpbmcg YWxsIFNTROKAmXMu4oCdIC8KPj4+Cj4+PiAvSW4gb3JkZXIgdG8gU1NEIHRpZXJpbmcgZnJvbSBt eSB1bmRlcnN0YW5kaW5nIHlvdSB3b3VsZCBuZWVkIDEgeCAKPj4+IE5WTWUgZHJpdmUgaW4gZWFj aCBzZXJ2ZXIsIG9yIDQgeCBTU0QgaG90IHRpZXIgKGl0IG5lZWRzIHRvIGJlIAo+Pj4gZGlzdHJp YnV0ZWQsIHJlcGxpY2F0ZWQgZm9yIHRoZSBob3QgdGllciBpZiBub3QgdXNpbmcgTlZNRSkuIFNv IHdpdGggCj4+PiB5b3Ugb25seSBoYXZpbmcgMSBTU0QgZHJpdmUgaW4gZWFjaCBzZXJ2ZXIsIEni gJlkIHN1Z2dlc3QgbWF5YmUgCj4+PiBsb29raW5nIGludG8gdGhlIE5WTUUgb3B0aW9uLiAvCj4+ PiAvCj4+PiAvCj4+PiAvU2luY2UgeW91ciB1c2luZyBvbmx5IDMtc2VydmVycywgd2hhdCBJ4oCZ ZCBwcm9iYWJseSBzdWdnZXN0IGlzIHRvIGRvIAo+Pj4gKDIgUmVwbGljYXMgKyBBcmJpdGVyIE5v ZGUpLCB0aGlzIHNldHVwIGFjdHVhbGx5IGRvZXNu4oCZdCByZXF1aXJlIHRoZSAKPj4+IDNyZCBz ZXJ2ZXIgdG8gaGF2ZSBiaWcgZHJpdmVzIGF0IGFsbCBhcyBpdCBvbmx5IHN0b3JlcyBtZXRhLWRh dGEgCj4+PiBhYm91dCB0aGUgZmlsZXMgYW5kIG5vdCBhY3R1YWxseSBhIGZ1bGwgY29weS4gLwo+ Pj4gLwo+Pj4gLwo+Pj4gL1BsZWFzZSBzZWUgdGhlIGF0dGFjaGVkIGRvY3VtZW50IHRoYXQgd2Fz IGdpdmVuIHRvIG1lIGJ5IFJlZCBIYXQgdG8gCj4+PiBnZXQgbW9yZSBpbmZvcm1hdGlvbiBvbiB0 aGlzLiBIb3BlIHRoaXMgaW5mb3JtYXRpb24gaGVscHMgeW91Li8KPj4+IC8KPj4+IC8KPj4+Cj4+ PiAtLQo+Pj4KPj4+IERldmluIEFjb3N0YSwgUkhDQSwgUkhWQ0EKPj4+IFJlZCBIYXQgQ2VydGlm aWVkIEFyY2hpdGVjdAo+Pj4KPj4+IE9uIEF1Z3VzdCA2LCAyMDE3IGF0IDc6Mjk6MjkgUE0sIE1v YWNpciBGZXJyZWlyYSAKPj4+IChtb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbSA8bWFpbHRvOm1v YWNpcmZlcnJlaXJhQGhvdG1haWwuY29tPikgd3JvdGU6Cj4+Pgo+Pj4+IEkgYW0gd2lsbGluZyB0 byBhc3NlbWJsZSBhIG9WaXJ0ICJwb2QiLCBtYWRlIG9mIDMgc2VydmVycywgZWFjaCAKPj4+PiB3 aXRoIDIgQ1BVIHNvY2tldHMgb2YgMTIgY29yZXMsIDI1NkdCIFJBTSwgNyBIREQgMTBLLCAxIFNT RC4gVGhlIAo+Pj4+IGlkZWEgaXMgdG8gdXNlIEdsdXN0ZXJGUyB0byBwcm92aWRlIEhBIGZvciB0 aGUgVk1zLiBUaGUgMyBzZXJ2ZXJzIAo+Pj4+IGhhdmUgYSBkdWFsIDQwR2IgTklDIGFuZCBhIGR1 YWwgMTBHYiBOSUMuIFNvIG15IGludGVudGlvbiBpcyB0byAKPj4+PiBjcmVhdGUgYSBsb29wIGxp a2UgYSBzZXJ2ZXIgdHJpYW5nbGUgdXNpbmcgdGhlIDQwR2IgTklDcyBmb3IgCj4+Pj4gdmlydHVh bGl6YXRpb24gZmlsZXMgKFZNcyAucWNvdzIpIGFjY2VzcyBhbmQgdG8gbW92ZSBWTXMgYXJvdW5k IHRoZSAKPj4+PiBwb2QgKGVhc3QgL3dlc3QgdHJhZmZpYykgd2hpbGUgdXNpbmcgdGhlIDEwR2Ig aW50ZXJmYWNlcyBmb3IgZ2l2aW5nIAo+Pj4+IHNlcnZpY2VzIHRvIHRoZSBvdXRzaWRlIHdvcmxk IChub3J0aC9zb3V0aCB0cmFmZmljKS4KPj4+Pgo+Pj4+Cj4+Pj4gVGhpcyBzYWlkLCBteSBmaXJz dCBxdWVzdGlvbiBpczogSG93IHNob3VsZCBJIGRlcGxveSBHbHVzdGVyRlMgaW4gCj4+Pj4gc3Vj aCBvVmlydCBzY2VuYXJpbz8gTXkgcXVlc3Rpb25zIGFyZToKPj4+Pgo+Pj4+Cj4+Pj4gMSAtIFNo b3VsZCBJIGNyZWF0ZSAzIFJBSUQgKGkuZS46IFJBSUQgNSksIG9uZSBvbiBlYWNoIG9WaXJ0IG5v ZGUsIAo+Pj4+IGFuZCB0aGVuIGNyZWF0ZSBhIEdsdXN0ZXJGUyB1c2luZyB0aGVtPwo+Pj4+Cj4+ Pj4gMiAtIEluc3RlYWQsIHNob3VsZCBJIGNyZWF0ZSBhIEpCT0QgYXJyYXkgbWFkZSBvZiBhbGwg c2VydmVyJ3MgZGlza3M/Cj4+Pj4KPj4+PiAzIC0gV2hhdCBpcyB0aGUgYmVzdCBHbHVzdGVyIGNv bmZpZ3VyYXRpb24gdG8gcHJvdmlkZSBmb3IgSEEgd2hpbGUgCj4+Pj4gbm90IGNvbnN1bWluZyB0 b28gbXVjaCBkaXNrIHNwYWNlPwo+Pj4+Cj4+Pj4gNCAtIERvZXMgYSBvVmlydCBoeXBlcnZpc29y IHBvZCBsaWtlIEkgYW0gcGxhbm5pbmcgdG8gYnVpbGQsIGFuZCAKPj4+PiB0aGUgdmlydHVhbGl6 YXRpb24gZW52aXJvbm1lbnQsIGJlbmVmaXRzIGZyb20gdGllcmluZyB3aGVuIHVzaW5nIGEgCj4+ Pj4gU1NEIGRpc2s/IEFuZCB5ZXMsIHdpbGwgR2x1c3RlciBkbyBpdCBieSBkZWZhdWx0IG9yIEkg aGF2ZSB0byAKPj4+PiBjb25maWd1cmUgaXQgdG8gZG8gc28/Cj4+Pj4KPj4+Pgo+Pj4+IEF0IHRo ZSBib3R0b20gbGluZSwgd2hhdCBpcyB0aGUgZ29vZCBwcmFjdGljZSBmb3IgdXNpbmcgR2x1c3Rl ckZTIAo+Pj4+IGluIHNtYWxsIHBvZHMgZm9yIGVudGVycHJpc2VzPwo+Pj4+Cj4+Pj4KPj4+PiBZ b3Ugb3Bpbmlvbi9mZWVkYmFjayB3aWxsIGJlIHJlYWxseSBhcHByZWNpYXRlZCEKPj4+Pgo+Pj4+ IE1vYWNpcgo+Pj4+Cj4+Pj4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX18KPj4+PiBVc2VycyBtYWlsaW5nIGxpc3QKPj4+PiBVc2Vyc0BvdmlydC5vcmcgPG1h aWx0bzpVc2Vyc0BvdmlydC5vcmc+Cj4+Pj4gaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFu L2xpc3RpbmZvL3VzZXJzCj4+Pgo+Pj4KPj4+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fCj4+PiBVc2VycyBtYWlsaW5nIGxpc3QKPj4+IFVzZXJzQG92aXJ0 Lm9yZwo+Pj4gaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzCj4+ Cj4+Cj4+Cj4+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f Cj4+IFVzZXJzIG1haWxpbmcgbGlzdAo+PiBVc2Vyc0BvdmlydC5vcmcKPj4gaHR0cDovL2xpc3Rz Lm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzCj4KCgotLS0tLS0tLS0tLS0tLUE1MDg4 NTQ2Q0RGOTkwMDYzMTRDQ0VFOApDb250ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD11dGYt OApDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiA4Yml0Cgo8aHRtbD4KICA8aGVhZD4KICAgIDxt ZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0 PXV0Zi04Ij4KICA8L2hlYWQ+CiAgPGJvZHkgYmdjb2xvcj0iI0ZGRkZGRiIgdGV4dD0iIzAwMDAw MCI+CiAgICA8cD5UaGFua3MgZm9yIHRoZSBjbGFyaWZpY2F0aW9uIEVyZWtsZS48L3A+CiAgICA8 cD5Ib3dldmVyIEkgZ2V0IHN1cnByaXNlZCB3aXRoIHRoaXMgd2F5IG9mIG9wZXJhdGluZyBmcm9t IEdsdXN0ZXJGUwogICAgICBhcyBpdCBhZGRzIGFub3RoZXIgbGF5ZXIgb2YgY29tcGxleGl0eSB0 byB0aGUgc3lzdGVtIChlaXRoZXIgYQogICAgICBoYXJkd2FyZSBvciBzb2Z0d2FyZSBSQUlEKSBi ZWZvcmUgdGhlIGdsdXN0ZXIgY29uZmlnIGFuZCBpbmNyZWFzZQogICAgICB0aGUgc3lzdGVtJ3Mg b3ZlcmFsbCBjb3N0cy48YnI+CiAgICA8L3A+CiAgICA8cD5BbiBpbXBvcnRhbnQgcG9pbnQgdG8g Y29uc2lkZXIgaXM6IEluIFJBSUQgY29uZmlndXJhdGlvbiB5b3UKICAgICAgYWxyZWFkeSBoYXZl IHNwYWNlICd3YXN0ZWQnIGluIG9yZGVyIHRvIGJ1aWxkIHJlZHVuZGFuY3kgKGVpdGhlcgogICAg ICBSQUlEIDEsIDUsIG9yIDYpLiBUaGVuIHdoZW4geW91IGhhdmUgR2x1c3RlckZTIG9uIHRoZSB0 b3Agb2YKICAgICAgc2V2ZXJhbCBSQUlEcyB5b3UgaGF2ZSBhZ2FpbiBtb3JlIGRhdGEgcmVwbGlj YXRlZCBzbyB5b3UgZW5kIHVwCiAgICAgIHdpdGggdGhlIHNhbWUgZGF0YSBjb25zdW1pbmcgbW9y ZSBzcGFjZSBpbiBhIGdyb3VwIG9mIGRpc2tzIGFuZAogICAgICBhZ2FpbiBvbiB0aGUgdG9wIG9m IHNldmVyYWwgUkFJRHMgZGVwZW5kaW5nIG9uIHRoZSBHbHVzdGVyCiAgICAgIGNvbmZpZ3VyYXRp b24geW91IGhhdmUgKGluIGEgUkFJRCAxIGNvbmZpZyB0aGUgc2FtZSBkYXRhIGlzCiAgICAgIHJl cGxpY2F0ZWQgNCB0aW1lcykuPC9wPgogICAgPHA+WWV0IGFub3RoZXIgZG93bnNpZGUgb2YgaGF2 aW5nIGEgUkFJRCAoc3BlY2lhbGx5IFJBSUQgNSBvciA2KSBpcwogICAgICB0aGF0IGl0IHJlZHVj ZXMgY29uc2lkZXJhYmx5IHRoZSB3cml0ZSBzcGVlZHMgYXMgZWFjaCBncm91cCBvZgogICAgICBk aXNrcyB3aWxsIGVuZCB1cCBoYXZpbmcgdGhlIHdyaXRlIHNwZWVkIG9mIGEgc2luZ2xlIGRpc2sg YXMgYWxsCiAgICAgIG90aGVyIGRpc2tzIG9mIHRoYXQgZ3JvdXAgaGF2ZSB0byB3YWl0IGZvciBl YWNoIG90aGVyIHRvIHdyaXRlIGFzCiAgICAgIHdlbGwuPGJyPgogICAgPC9wPgogICAgPHA+VGhl cmVmb3JlIGlmIEdsdXN0ZXIgYWxyZWFkeSByZXBsaWNhdGVzIGRhdGEgd2h5IGRvZXMgaXQgY3Jl YXRlCiAgICAgIHRoaXMgYmlnIHBhaW4geW91IG1lbnRpb25lZCBpZiB0aGUgZGF0YSBpcyByZXBs aWNhdGVkIHNvbWV3aGVyZQogICAgICBlbHNlLCBjYW4gc3RpbGwgYmUgcmV0cmlldmVkIHRvIGJv dGggc2VydmUgY2xpZW50cyBhbmQgcmVjb25zdHJ1Y3QKICAgICAgdGhlIGVxdWl2YWxlbnQgZGlz ayB3aGVuIGl0IGlzIHJlcGxhY2VkID88L3A+CiAgICA8cD5GZXJuYW5kbzxicj4KICAgIDwvcD4K ICAgIDxicj4KICAgIDxkaXYgY2xhc3M9Im1vei1jaXRlLXByZWZpeCI+T24gMDcvMDgvMjAxNyAx MDoyNiwgRXJla2xlIE1hZ3JhZHplCiAgICAgIHdyb3RlOjxicj4KICAgIDwvZGl2PgogICAgPGJs b2NrcXVvdGUgdHlwZT0iY2l0ZSIKICAgICAgY2l0ZT0ibWlkOmFhODI5ZDA3LWZhNzctM2VkOS0y NTAwLWUzM2NjMDE0MTRiNkByZWNvZ2l6ZXIuZGUiPgogICAgICA8bWV0YSBodHRwLWVxdWl2PSJD b250ZW50LVR5cGUiIGNvbnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11dGYtOCI+CiAgICAgIDxw PkhpIEZyZW5hbmRvLDwvcD4KICAgICAgPHA+SGVyZSBpcyBteSBleHBlcmllbmNlLCBpZiB5b3Ug Y29uc2lkZXIgYSBwYXJ0aWN1bGFyIGhhcmQgZHJpdmUKICAgICAgICBhcyBhIGJyaWNrIGZvciBn bHVzdGVyIHZvbHVtZSBhbmQgaXQgZGllcywgaS5lLiBpdCBiZWNvbWVzIG5vdAogICAgICAgIGFj Y2Vzc2libGUgaXQncyBhIGh1Z2UgaGFzc2xlIHRvIGRpc2NhcmQgdGhhdCBicmljayBhbmQgZXhj aGFuZ2UKICAgICAgICB3aXRoIGFub3RoZXIgb25lLCBzaW5jZSBnbHVzdGVyIHNvbWUgdHJpZXMg dG8gYWNjZXNzIHRoYXQgYnJva2VuCiAgICAgICAgYnJpY2sgYW5kIGl0J3MgY2F1c2luZyAoYXQg bGVhc3QgaXQgY2F1c2UgZm9yIG1lKSBhIGJpZyBwYWluLAogICAgICAgIHRoZXJlZm9yZSBpdCdz IGJldHRlciB0byBoYXZlIGEgUkFJRCBhcyBicmljaywgaS5lLiBoYXZlIFJBSUQgMQogICAgICAg IChtaXJyb3JpbmcpIGZvciBlYWNoIGJyaWNrLCBpbiB0aGlzIGNhc2UgaWYgdGhlIGRpc2sgaXMg ZG93biB5b3UKICAgICAgICBjYW4gZWFzaWx5IGV4Y2hhbmdlIGl0IGFuZCByZWJ1aWxkIHRoZSBS QUlEIHdpdGhvdXQgZ29pbmcKICAgICAgICBvZmZsaW5lLCBpLmUgc3dpdGNoaW5nIG9mZiB0aGUg dm9sdW1lIGRvaW5nIGJyaWNrIG1hbmlwdWxhdGlvbnMKICAgICAgICBhbmQgc3dpdGNoaW5nIGl0 IGJhY2sgb24uPGJyPgogICAgICA8L3A+CiAgICAgIDxwPkNoZWVyczwvcD4KICAgICAgPHA+RXJl a2xlPGJyPgogICAgICA8L3A+CiAgICAgIDxicj4KICAgICAgPGRpdiBjbGFzcz0ibW96LWNpdGUt cHJlZml4Ij5PbiAwOC8wNy8yMDE3IDAzOjA0IFBNLCBGRVJOQU5ETwogICAgICAgIEZSRURJQU5J IHdyb3RlOjxicj4KICAgICAgPC9kaXY+CiAgICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiCiAg ICAgICAgY2l0ZT0ibWlkOjYzYmFjNDdiLWFmZTYtMDI1OC1kM2Q3LWU1NDVhNTAwNGMzMEB1cHgu Y29tIj4KICAgICAgICA8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNvbnRlbnQ9InRl eHQvaHRtbDsKICAgICAgICAgIGNoYXJzZXQ9dXRmLTgiPgogICAgICAgIDxwPkZvciBhbnkgUkFJ RCA1IG9yIDYgY29uZmlndXJhdGlvbiBJIG5vcm1hbGx5IGZvbGxvdyBhIHNpbXBsZQogICAgICAg ICAgZ29sZCBydWxlIHdoaWNoIGdhdmUgZ29vZCByZXN1bHRzIHNvIGZhcjo8YnI+CiAgICAgICAg ICAtIHVwIHRvIDQgZGlza3MgUkFJRCA1PGJyPgogICAgICAgICAgLSA1IG9yIG1vcmUgZGlza3Mg UkFJRCA2PC9wPgogICAgICAgIDxwPkhvd2V2ZXIgSSBkaWRuJ3QgcmVhbGx5IHVuZGVyc3RhbmQg d2VsbCB0aGUgcmVjb21tZW5kYXRpb24gdG8KICAgICAgICAgIHVzZSBhbnkgUkFJRCB3aXRoIEds dXN0ZXJGUy4gSSBhbHdheXMgdGhvdWdodCB0aGF0IEdsdXN0ZUZTCiAgICAgICAgICBsaWtlcyB0 byB3b3JrIGluIEpCT0QgbW9kZSBhbmQgY29udHJvbCB0aGUgZGlza3MgKGJyaWNrcykKICAgICAg ICAgIGRpcmVjdGxseSBzbyB5b3UgY2FuIGNyZWF0ZSB3aGF0ZXZlciBkaXN0cmlidXRpb24gcnVs ZSB5b3UKICAgICAgICAgIHdpc2gsIGFuZCBpZiBhIHNpbmdsZSBkaXNrIGZhaWxzIHlvdSBqdXN0 IHJlcGxhY2UgaXQgYW5kIHdoaWNoCiAgICAgICAgICBvYnZpb3VzbHkgaGF2ZSB0aGUgZGF0YSBy ZXBsaWNhdGVkIGZyb20gYW5vdGhlci4gVGhlIG9ubHkKICAgICAgICAgIGRvd25zaWRlIG9mIHVz aW5nIGluIHRoaXMgd2F5IGlzIHRoYXQgdGhlIHJlcGxpY2F0aW9uIGRhdGEKICAgICAgICAgIHdp bGwgYmUgZmxvdyBhY2Nyb3NzIGFsbCBzZXJ2ZXJzIGJ1dCB0aGF0IGlzIG5vdCBtdWNoIGEgYmln CiAgICAgICAgICBpc3N1ZS48L3A+CiAgICAgICAgPHA+QW55b25lIGNhbiBlbGFib3JhdGUgYWJv dXQgVXNpbmcgUkFJRCArIEdsdXN0ZXJGUyBhbmQgSkJPRCArCiAgICAgICAgICBHbHVzdGVyRlMu PC9wPgogICAgICAgIDxwPlRoYW5rczxicj4KICAgICAgICAgIFJlZ2FyZHM8YnI+CiAgICAgICAg ICBGZXJuYW5kbzxicj4KICAgICAgICA8L3A+CiAgICAgICAgPGJyPgogICAgICAgIDxkaXYgY2xh c3M9Im1vei1jaXRlLXByZWZpeCI+T24gMDcvMDgvMjAxNyAwMzo0NiwgRGV2aW4gQWNvc3RhCiAg ICAgICAgICB3cm90ZTo8YnI+CiAgICAgICAgPC9kaXY+CiAgICAgICAgPGJsb2NrcXVvdGUgdHlw ZT0iY2l0ZSIKY2l0ZT0ibWlkOkNBTkNHS0VwNFhHczBVK1FzNzhlRW1xQ050dnBMWS1BempiNURj R2haOXlpS1RCRUVmd0BtYWlsLmdtYWlsLmNvbSI+CiAgICAgICAgICA8c3R5bGU+Ym9keXtmb250 LWZhbWlseTpIZWx2ZXRpY2EsQXJpYWw7Zm9udC1zaXplOjEzcHh9PC9zdHlsZT4KICAgICAgICAg IDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJjb2xvcjpyZ2IoMCwwLDApO21hcmdp bjowcHgiPjxmb250CiAgICAgICAgICAgICAgZmFjZT0iSW5wdXQgTW9ubyI+PGJyPgogICAgICAg ICAgICA8L2ZvbnQ+PC9kaXY+CiAgICAgICAgICA8ZGl2IGlkPSJibG9vcF9jdXN0b21mb250IiBz dHlsZT0iY29sb3I6cmdiKDAsMCwwKTttYXJnaW46MHB4Ij48Zm9udAogICAgICAgICAgICAgIGZh Y2U9IklucHV0IE1vbm8iPk1vYWNpciw8L2ZvbnQ+PC9kaXY+CiAgICAgICAgICA8ZGl2IGlkPSJi bG9vcF9jdXN0b21mb250IiBzdHlsZT0iY29sb3I6cmdiKDAsMCwwKTttYXJnaW46MHB4Ij48Zm9u dAogICAgICAgICAgICAgIGZhY2U9IklucHV0IE1vbm8iPjxicj4KICAgICAgICAgICAgPC9mb250 PjwvZGl2PgogICAgICAgICAgPGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9ImNvbG9y OnJnYigwLDAsMCk7bWFyZ2luOjBweCI+PGZvbnQKICAgICAgICAgICAgICBmYWNlPSJJbnB1dCBN b25vIj5JIGhhdmUgcmVjZW50bHkgaW5zdGFsbGVkIG11bHRpcGxlIFJlZAogICAgICAgICAgICAg IEhhdCBWaXJ0dWFsaXphdGlvbiBob3N0cyBmb3Igc2V2ZXJhbCBkaWZmZXJlbnQgY29tcGFuaWVz LAogICAgICAgICAgICAgIGFuZCBoYXZlIGRlYWx0IHdpdGggdGhlIFJlZCBIYXQgU3VwcG9ydCBU ZWFtIGluIGRlcHRoCiAgICAgICAgICAgICAgYWJvdXQgb3B0aW1hbCBjb25maWd1cmF0aW9uIGlu IHJlZ2FyZHMgdG8gc2V0dGluZyB1cAogICAgICAgICAgICAgIEdsdXN0ZXJGUyBtb3N0IGVmZmlj aWVudGx5IGFuZCBJIHdhbnRlZCB0byBzaGFyZSB3aXRoIHlvdQogICAgICAgICAgICAgIHdoYXQg SSBsZWFybmVkLjwvZm9udD48L2Rpdj4KICAgICAgICAgIDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZv bnQiIHN0eWxlPSJjb2xvcjpyZ2IoMCwwLDApO21hcmdpbjowcHgiPjxmb250CiAgICAgICAgICAg ICAgZmFjZT0iSW5wdXQgTW9ubyI+PGJyPgogICAgICAgICAgICA8L2ZvbnQ+PC9kaXY+CiAgICAg ICAgICA8ZGl2IGlkPSJibG9vcF9jdXN0b21mb250IiBzdHlsZT0iY29sb3I6cmdiKDAsMCwwKTtt YXJnaW46MHB4Ij48Zm9udAogICAgICAgICAgICAgIGZhY2U9IklucHV0IE1vbm8iPkluIGdlbmVy YWwgUmVkIEhhdCBWaXJ0dWFsaXphdGlvbiB0ZWFtCiAgICAgICAgICAgICAgZnJvd25zIHVwb24g dXNpbmcgZWFjaCBESVNLIG9mIHRoZSBzeXN0ZW0gYXMganVzdCBhIEpCT0QsCiAgICAgICAgICAg ICAgc3VyZSB0aGVyZSBpcyBzb21lIHByb3RlY3Rpb24gYnkgaGF2aW5nIHRoZSBkYXRhCiAgICAg ICAgICAgICAgcmVwbGljYXRlZCwgaG93ZXZlciwgdGhlIHJlY29tbWVuZGF0aW9uIGlzIHRvIHVz ZSBSQUlEIDYKICAgICAgICAgICAgICAocHJlZmVycmVkKSBvciBSQUlELTUsIG9yIGF0IGxlYXN0 IFJBSUQtMSBhdCB0aGUgdmVyeQogICAgICAgICAgICAgIGxlYXN0LjwvZm9udD48L2Rpdj4KICAg ICAgICAgIDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJjb2xvcjpyZ2IoMCwwLDAp O21hcmdpbjowcHgiPjxmb250CiAgICAgICAgICAgICAgZmFjZT0iSW5wdXQgTW9ubyI+PGJyPgog ICAgICAgICAgICA8L2ZvbnQ+PC9kaXY+CiAgICAgICAgICA8ZGl2IGlkPSJibG9vcF9jdXN0b21m b250IiBzdHlsZT0ibWFyZ2luOjBweCI+PGZvbnQKICAgICAgICAgICAgICBmYWNlPSJJbnB1dCBN b25vIj5IZXJlIGlzIHRoZcKgZGlyZWN0IHF1b3RlIGZyb20gUmVkIEhhdAogICAgICAgICAgICAg IHdoZW4gSSBhc2tlZCBhYm91dCBSQUlEIGFuZCBCcmlja3M6PC9mb250PjwvZGl2PgogICAgICAg ICAgPGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9Im1hcmdpbjowcHgiPjxmb250CiAg ICAgICAgICAgICAgZmFjZT0iSW5wdXQgTW9ubyI+PGk+PGJyPgogICAgICAgICAgICAgIDwvaT48 L2ZvbnQ+PC9kaXY+CiAgICAgICAgICA8ZGl2IGlkPSJibG9vcF9jdXN0b21mb250IiBzdHlsZT0i bWFyZ2luOjBweCI+PGZvbnQKICAgICAgICAgICAgICBmYWNlPSJJbnB1dCBNb25vIj48aT4iQSB0 eXBpY2FsIEdsdXN0ZXIgY29uZmlndXJhdGlvbgogICAgICAgICAgICAgICAgd291bGQgdXNlIFJB SUQgdW5kZXJuZWF0aCB0aGUgYnJpY2tzLiBSQUlEIDYgaXMgbW9zdAogICAgICAgICAgICAgICAg dHlwaWNhbCBhcyBpdCBnaXZlcyB5b3UgMiBkaXNrIGZhaWx1cmUgcHJvdGVjdGlvbiwgYnV0CiAg ICAgICAgICAgICAgICBSQUlEIDUgY291bGQgYmUgdXNlZCB0b28uwqBPbmNlIHlvdSBoYXZlIHRo ZSBSQUlEZWQKICAgICAgICAgICAgICAgIGJyaWNrcywgeW91J2QgdGhlbiBhcHBseSB0aGUgZGVz aXJlZCByZXBsaWNhdGlvbiBvbiB0b3AKICAgICAgICAgICAgICAgIG9mIHRoYXQuIFRoZSBtb3N0 IHBvcHVsYXIgd2F5IG9mIGRvaW5nIHRoaXMgd291bGQgYmUKICAgICAgICAgICAgICAgIGRpc3Ry aWJ1dGVkIHJlcGxpY2F0ZWQgd2l0aCAyeCByZXBsaWNhdGlvbi7CoEluIGdlbmVyYWwKICAgICAg ICAgICAgICAgIHlvdSdsbCBnZXQgYmV0dGVyIHBlcmZvcm1hbmNlIHdpdGggbGFyZ2VyIGJyaWNr cy4gMTIKICAgICAgICAgICAgICAgIGRyaXZlcyBpcyBvZnRlbiBhIHN3ZWV0IHNwb3QuIEFub3Ro ZXIgb3B0aW9uIHdvdWxkIGJlIHRvCiAgICAgICAgICAgICAgICBjcmVhdGUgYSBzZXBhcmF0ZSB0 aWVyIHVzaW5nIGFsbCBTU0TigJlzLuKAncKgPC9pPjwvZm9udD48L2Rpdj4KICAgICAgICAgIDxk aXYgaWQ9ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJtYXJnaW46MHB4Ij48YnI+CiAgICAgICAg ICA8L2Rpdj4KICAgICAgICAgIDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJtYXJn aW46MHB4Ij48Zm9udAogICAgICAgICAgICAgIGZhY2U9IklucHV0IE1vbm8iPjxpPkluIG9yZGVy IHRvIFNTRMKgdGllcmluZyBmcm9tIG15CiAgICAgICAgICAgICAgICB1bmRlcnN0YW5kaW5nIHlv dSB3b3VsZCBuZWVkIDEgeCBOVk1lIGRyaXZlIGluIGVhY2gKICAgICAgICAgICAgICAgIHNlcnZl ciwgb3IgNCB4IFNTRCBob3QgdGllciAoaXQgbmVlZHMgdG8gYmUgZGlzdHJpYnV0ZWQsCiAgICAg ICAgICAgICAgICByZXBsaWNhdGVkIGZvciB0aGUgaG90IHRpZXIgaWYgbm90IHVzaW5nIE5WTUUp LiBTbyB3aXRoCiAgICAgICAgICAgICAgICB5b3Ugb25seSBoYXZpbmcgMSBTU0QgZHJpdmUgaW4g ZWFjaCBzZXJ2ZXIswqBJ4oCZZCBzdWdnZXN0CiAgICAgICAgICAgICAgICBtYXliZSBsb29raW5n IGludG8gdGhlIE5WTUUgb3B0aW9uLsKgPC9pPjwvZm9udD48L2Rpdj4KICAgICAgICAgIDxkaXYg aWQ9ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJtYXJnaW46MHB4Ij48Zm9udAogICAgICAgICAg ICAgIGZhY2U9IklucHV0IE1vbm8iPjxpPjxicj4KICAgICAgICAgICAgICA8L2k+PC9mb250Pjwv ZGl2PgogICAgICAgICAgPGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9Im1hcmdpbjow cHgiPjxmb250CiAgICAgICAgICAgICAgZmFjZT0iSW5wdXQgTW9ubyI+PGk+U2luY2UgeW91ciB1 c2luZyBvbmx5IDMtc2VydmVycywgd2hhdAogICAgICAgICAgICAgICAgSeKAmWQgcHJvYmFibHkg c3VnZ2VzdCBpcyB0byBkbyAoMiBSZXBsaWNhcyArIEFyYml0ZXIKICAgICAgICAgICAgICAgIE5v ZGUpLCB0aGlzIHNldHVwIGFjdHVhbGx5IGRvZXNu4oCZdCByZXF1aXJlIHRoZSAzcmQKICAgICAg ICAgICAgICAgIHNlcnZlciB0byBoYXZlIGJpZyBkcml2ZXMgYXQgYWxsIGFzIGl0IG9ubHkgc3Rv cmVzCiAgICAgICAgICAgICAgICBtZXRhLWRhdGEgYWJvdXQgdGhlIGZpbGVzIGFuZCBub3QgYWN0 dWFsbHkgYSBmdWxsIGNvcHkuwqA8L2k+PC9mb250PjwvZGl2PgogICAgICAgICAgPGRpdiBpZD0i Ymxvb3BfY3VzdG9tZm9udCIgc3R5bGU9Im1hcmdpbjowcHgiPjxmb250CiAgICAgICAgICAgICAg ZmFjZT0iSW5wdXQgTW9ubyI+PGk+PGJyPgogICAgICAgICAgICAgIDwvaT48L2ZvbnQ+PC9kaXY+ CiAgICAgICAgICA8ZGl2IGlkPSJibG9vcF9jdXN0b21mb250IiBzdHlsZT0ibWFyZ2luOjBweCI+ PGZvbnQKICAgICAgICAgICAgICBmYWNlPSJJbnB1dCBNb25vIj48aT5QbGVhc2Ugc2VlIHRoZSBh dHRhY2hlZCBkb2N1bWVudCB0aGF0CiAgICAgICAgICAgICAgICB3YXMgZ2l2ZW4gdG8gbWUgYnkg UmVkIEhhdCB0byBnZXQgbW9yZSBpbmZvcm1hdGlvbiBvbgogICAgICAgICAgICAgICAgdGhpcy4g SG9wZSB0aGlzIGluZm9ybWF0aW9uIGhlbHBzIHlvdS48L2k+PC9mb250PjwvZGl2PgogICAgICAg ICAgPGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9Im1hcmdpbjowcHgiPjxmb250CiAg ICAgICAgICAgICAgZmFjZT0iSW5wdXQgTW9ubyI+PGk+PGJyPgogICAgICAgICAgICAgIDwvaT48 L2ZvbnQ+PC9kaXY+CiAgICAgICAgICA8YnI+CiAgICAgICAgICA8ZGl2IGlkPSJibG9vcF9zaWdu XzE1MDIwODczNzY3MjU0NjkxODQiIGNsYXNzPSJibG9vcF9zaWduIj48c3BhbgogICAgICAgICAg ICAgIHN0eWxlPSJmb250LWZhbWlseTonaGVsdmV0aWNhCiAgICAgICAgICAgICAgTmV1ZScsaGVs dmV0aWNhO2ZvbnQtc2l6ZToxNHB4Ij4tLTwvc3Bhbj48YnIKICAgICAgICAgICAgICBzdHlsZT0i Zm9udC1mYW1pbHk6J2hlbHZldGljYQogICAgICAgICAgICAgIE5ldWUnLGhlbHZldGljYTtmb250 LXNpemU6MTRweCI+CiAgICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX3NpZ25hdHVyZSIgc3R5 bGU9ImZvbnQtZmFtaWx5OidoZWx2ZXRpY2EKICAgICAgICAgICAgICBOZXVlJyxoZWx2ZXRpY2E7 Zm9udC1zaXplOjE0cHgiPgogICAgICAgICAgICAgIDxkaXYgZGlyPSJsdHIiPgogICAgICAgICAg ICAgICAgPGRpdj48YnI+CiAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgIDxk aXY+RGV2aW4gQWNvc3RhLCBSSENBLCBSSFZDQTwvZGl2PgogICAgICAgICAgICAgICAgPGRpdj5S ZWQgSGF0IENlcnRpZmllZCBBcmNoaXRlY3Q8L2Rpdj4KICAgICAgICAgICAgICA8L2Rpdj4KICAg ICAgICAgICAgPC9kaXY+CiAgICAgICAgICA8L2Rpdj4KICAgICAgICAgIDxicj4KICAgICAgICAg IDxwIGNsYXNzPSJhaXJtYWlsX29uIj5PbiBBdWd1c3QgNiwgMjAxNyBhdCA3OjI5OjI5IFBNLCBN b2FjaXIKICAgICAgICAgICAgRmVycmVpcmEgKDxhIGhyZWY9Im1haWx0bzptb2FjaXJmZXJyZWly YUBob3RtYWlsLmNvbSIKICAgICAgICAgICAgICBtb3otZG8tbm90LXNlbmQ9InRydWUiPm1vYWNp cmZlcnJlaXJhQGhvdG1haWwuY29tPC9hPikKICAgICAgICAgICAgd3JvdGU6PC9wPgogICAgICAg ICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSIgY2xhc3M9ImNsZWFuX2JxIj48c3Bhbj4KICAgICAg ICAgICAgICA8ZGl2IGRpcj0ibHRyIj4KICAgICAgICAgICAgICAgIDxkaXY+CiAgICAgICAgICAg ICAgICAgIDx0aXRsZT48L3RpdGxlPgogICAgICAgICAgICAgICAgICA8ZGl2IGlkPSJkaXZ0YWdk ZWZhdWx0d3JhcHBlciIKc3R5bGU9ImZvbnQtc2l6ZToxMnB0O2NvbG9yOiMwMDAwMDA7Zm9udC1m YW1pbHk6Q2FsaWJyaSxIZWx2ZXRpY2Esc2Fucy1zZXJpZiIKICAgICAgICAgICAgICAgICAgICBk aXI9Imx0ciI+CiAgICAgICAgICAgICAgICAgICAgPHA+PHNwYW4+SSBhbSB3aWxsaW5nIHRvIGFz c2VtYmxlIGEgb1ZpcnQgInBvZCIsCiAgICAgICAgICAgICAgICAgICAgICAgIG1hZGUgb2YgMyBz ZXJ2ZXJzLCBlYWNoIHdpdGggMiBDUFUgc29ja2V0cyBvZiAxMgogICAgICAgICAgICAgICAgICAg ICAgICBjb3JlcywgMjU2R0IgUkFNLCA3IEhERCAxMEssIDEgU1NELiBUaGUgaWRlYSBpcwogICAg ICAgICAgICAgICAgICAgICAgICB0byB1c2UgR2x1c3RlckZTIHRvIHByb3ZpZGUgSEEgZm9yIHRo ZSBWTXMuIFRoZQogICAgICAgICAgICAgICAgICAgICAgICAzIHNlcnZlcnMgaGF2ZSBhIGR1YWwg NDBHYiBOSUMgYW5kIGEgZHVhbCAxMEdiCiAgICAgICAgICAgICAgICAgICAgICAgIE5JQy4gU28g bXkgaW50ZW50aW9uIGlzIHRvIGNyZWF0ZSBhIGxvb3AgbGlrZSBhCiAgICAgICAgICAgICAgICAg ICAgICAgIHNlcnZlciB0cmlhbmdsZSB1c2luZyB0aGUgNDBHYiBOSUNzIGZvcgogICAgICAgICAg ICAgICAgICAgICAgICB2aXJ0dWFsaXphdGlvbiBmaWxlcyAoVk1zIC5xY293MikgYWNjZXNzIGFu ZCB0bwogICAgICAgICAgICAgICAgICAgICAgICBtb3ZlIFZNcyBhcm91bmQgdGhlIHBvZCAoZWFz dCAvd2VzdCB0cmFmZmljKQogICAgICAgICAgICAgICAgICAgICAgICB3aGlsZSB1c2luZyB0aGUg MTBHYiBpbnRlcmZhY2VzIGZvciBnaXZpbmcKICAgICAgICAgICAgICAgICAgICAgICAgc2Vydmlj ZXMgdG8gdGhlIG91dHNpZGUgd29ybGQgKG5vcnRoL3NvdXRoCiAgICAgICAgICAgICAgICAgICAg ICAgIHRyYWZmaWMpLjwvc3Bhbj48L3A+CiAgICAgICAgICAgICAgICAgICAgPHA+PGJyPgogICAg ICAgICAgICAgICAgICAgIDwvcD4KICAgICAgICAgICAgICAgICAgICA8cD5UaGlzIHNhaWQsIG15 IGZpcnN0IHF1ZXN0aW9uIGlzOiBIb3cgc2hvdWxkIEkKICAgICAgICAgICAgICAgICAgICAgIGRl cGxveSBHbHVzdGVyRlMgaW4gc3VjaCBvVmlydCBzY2VuYXJpbz8gTXkKICAgICAgICAgICAgICAg ICAgICAgIHF1ZXN0aW9ucyBhcmU6PC9wPgogICAgICAgICAgICAgICAgICAgIDxwPjxicj4KICAg ICAgICAgICAgICAgICAgICA8L3A+CiAgICAgICAgICAgICAgICAgICAgPHA+MSAtIFNob3VsZCBJ IGNyZWF0ZSAzIFJBSUQgKGkuZS46IFJBSUQgNSksIG9uZSBvbgogICAgICAgICAgICAgICAgICAg ICAgZWFjaCBvVmlydCBub2RlLCBhbmQgdGhlbiBjcmVhdGUgYSBHbHVzdGVyRlMgdXNpbmcKICAg ICAgICAgICAgICAgICAgICAgIHRoZW0/PC9wPgogICAgICAgICAgICAgICAgICAgIDxwPjIgLSBJ bnN0ZWFkLCBzaG91bGQgSSBjcmVhdGUgYSBKQk9EIGFycmF5IG1hZGUgb2YKICAgICAgICAgICAg ICAgICAgICAgIGFsbCBzZXJ2ZXIncyBkaXNrcz88L3A+CiAgICAgICAgICAgICAgICAgICAgPHA+ MyAtIFdoYXQgaXMgdGhlIGJlc3QgR2x1c3RlciBjb25maWd1cmF0aW9uIHRvCiAgICAgICAgICAg ICAgICAgICAgICBwcm92aWRlIGZvciBIQSB3aGlsZSBub3QgY29uc3VtaW5nIHRvbyBtdWNoIGRp c2sKICAgICAgICAgICAgICAgICAgICAgIHNwYWNlPzxicj4KICAgICAgICAgICAgICAgICAgICA8 L3A+CiAgICAgICAgICAgICAgICAgICAgPHA+NCAtIERvZXMgYSBvVmlydCBoeXBlcnZpc29yIHBv ZCBsaWtlIEkgYW0KICAgICAgICAgICAgICAgICAgICAgIHBsYW5uaW5nIHRvIGJ1aWxkLCBhbmQg dGhlIHZpcnR1YWxpemF0aW9uCiAgICAgICAgICAgICAgICAgICAgICBlbnZpcm9ubWVudCwgYmVu ZWZpdHMgZnJvbSB0aWVyaW5nIHdoZW4gdXNpbmcgYQogICAgICAgICAgICAgICAgICAgICAgU1NE IGRpc2s/IEFuZCB5ZXMsIHdpbGwgR2x1c3RlciBkbyBpdCBieSBkZWZhdWx0CiAgICAgICAgICAg ICAgICAgICAgICBvciBJIGhhdmUgdG8gY29uZmlndXJlIGl0IHRvIGRvIHNvPzwvcD4KICAgICAg ICAgICAgICAgICAgICA8cD48YnI+CiAgICAgICAgICAgICAgICAgICAgPC9wPgogICAgICAgICAg ICAgICAgICAgIDxwPkF0IHRoZSBib3R0b20gbGluZSwgd2hhdCBpcyB0aGUgZ29vZCBwcmFjdGlj ZSBmb3IKICAgICAgICAgICAgICAgICAgICAgIHVzaW5nIEdsdXN0ZXJGUyBpbiBzbWFsbCBwb2Rz IGZvciBlbnRlcnByaXNlcz88YnI+CiAgICAgICAgICAgICAgICAgICAgPC9wPgogICAgICAgICAg ICAgICAgICAgIDxwPjxicj4KICAgICAgICAgICAgICAgICAgICA8L3A+CiAgICAgICAgICAgICAg ICAgICAgPHA+WW91IG9waW5pb24vZmVlZGJhY2sgd2lsbCBiZSByZWFsbHkgYXBwcmVjaWF0ZWQh PC9wPgogICAgICAgICAgICAgICAgICAgIDxwPk1vYWNpcjxicj4KICAgICAgICAgICAgICAgICAg ICA8L3A+CiAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICBfX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXyA8YnI+CiAgICAgICAgICAg ICAgICAgIFVzZXJzIG1haWxpbmcgbGlzdCA8YnI+CiAgICAgICAgICAgICAgICAgIDxhIGhyZWY9 Im1haWx0bzpVc2Vyc0BvdmlydC5vcmciCiAgICAgICAgICAgICAgICAgICAgbW96LWRvLW5vdC1z ZW5kPSJ0cnVlIj5Vc2Vyc0BvdmlydC5vcmc8L2E+IDxicj4KICAgICAgICAgICAgICAgICAgPGEK ICAgICAgICAgICAgICAgICAgICBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4v bGlzdGluZm8vdXNlcnMiCiAgICAgICAgICAgICAgICAgICAgbW96LWRvLW5vdC1zZW5kPSJ0cnVl Ij5odHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnM8L2E+CiAgICAg ICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgIDwv ZGl2PgogICAgICAgICAgICA8L3NwYW4+PC9ibG9ja3F1b3RlPgogICAgICAgICAgPGJyPgogICAg ICAgICAgPGZpZWxkc2V0IGNsYXNzPSJtaW1lQXR0YWNobWVudEhlYWRlciI+PC9maWVsZHNldD4K ICAgICAgICAgIDxicj4KICAgICAgICAgIDxwcmUgd3JhcD0iIj5fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fXwpVc2VycyBtYWlsaW5nIGxpc3QKPGEgY2xhc3M9 Im1vei10eHQtbGluay1hYmJyZXZpYXRlZCIgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIg bW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5Vc2Vyc0BvdmlydC5vcmc8L2E+CjxhIGNsYXNzPSJtb3ot dHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9s aXN0aW5mby91c2VycyIgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5odHRwOi8vbGlzdHMub3ZpcnQu b3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnM8L2E+CjwvcHJlPgogICAgICAgIDwvYmxvY2txdW90 ZT4KICAgICAgICA8YnI+CiAgICAgICAgPGJyPgogICAgICAgIDxmaWVsZHNldCBjbGFzcz0ibWlt ZUF0dGFjaG1lbnRIZWFkZXIiPjwvZmllbGRzZXQ+CiAgICAgICAgPGJyPgogICAgICAgIDxwcmUg d3JhcD0iIj5fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpV c2VycyBtYWlsaW5nIGxpc3QKPGEgY2xhc3M9Im1vei10eHQtbGluay1hYmJyZXZpYXRlZCIgaHJl Zj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5Vc2Vyc0Bv dmlydC5vcmc8L2E+CjxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHA6 Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyIgbW96LWRvLW5vdC1zZW5k PSJ0cnVlIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnM8L2E+ CjwvcHJlPgogICAgICA8L2Jsb2NrcXVvdGU+CiAgICAgIDxicj4KICAgIDwvYmxvY2txdW90ZT4K ICAgIDxicj4KICA8L2JvZHk+CjwvaHRtbD4KCi0tLS0tLS0tLS0tLS0tQTUwODg1NDZDREY5OTAw NjMxNENDRUU4LS0K --===============3959530968213590397==-- From ykaul at redhat.com Mon Aug 7 18:44:12 2017 Content-Type: multipart/mixed; boundary="===============8696520672274949902==" MIME-Version: 1.0 From: Yaniv Kaul To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Mon, 07 Aug 2017 21:43:29 +0300 Message-ID: In-Reply-To: c7a1c2e1-57c3-9fa5-0710-ebee3f3fa069@upx.com --===============8696520672274949902== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Mon, Aug 7, 2017 at 6:41 PM, FERNANDO FREDIANI wrote: > Thanks for the clarification Erekle. > > However I get surprised with this way of operating from GlusterFS as it > adds another layer of complexity to the system (either a hardware or > software RAID) before the gluster config and increase the system's overall > costs. > It does, but with HW based RAID it's not a big deal. The complexity is all the stripe size math... which I personally don't like to calculate. > An important point to consider is: In RAID configuration you already have > space 'wasted' in order to build redundancy (either RAID 1, 5, or 6). Then > when you have GlusterFS on the top of several RAIDs you have again more > data replicated so you end up with the same data consuming more space in a > group of disks and again on the top of several RAIDs depending on the > Gluster configuration you have (in a RAID 1 config the same data is > replicated 4 times). > > Yet another downside of having a RAID (specially RAID 5 or 6) is that it > reduces considerably the write speeds as each group of disks will end up > having the write speed of a single disk as all other disks of that group > have to wait for each other to write as well. > > Therefore if Gluster already replicates data why does it create this big > pain you mentioned if the data is replicated somewhere else, can still be > retrieved to both serve clients and reconstruct the equivalent disk when = it > is replaced ? > I think it's a matter of how fast you can replace a disk (over a long weekend?), how reliably you can do it (please, don't pull the wrong disk! I've seen it happening too many times!) and how much of a performance hit are you willing to accept while in degraded mode (and how long it took to detect it. HDDs, unlike SSDs, die slowly. At least when SSD dies, it dies a quick and determined death. HDDs may accumulate errors and errors and still function). Y. Fernando > > On 07/08/2017 10:26, Erekle Magradze wrote: > > Hi Frenando, > > Here is my experience, if you consider a particular hard drive as a brick > for gluster volume and it dies, i.e. it becomes not accessible it's a huge > hassle to discard that brick and exchange with another one, since gluster > some tries to access that broken brick and it's causing (at least it cause > for me) a big pain, therefore it's better to have a RAID as brick, i.e. > have RAID 1 (mirroring) for each brick, in this case if the disk is down > you can easily exchange it and rebuild the RAID without going offline, i.e > switching off the volume doing brick manipulations and switching it back = on. > > Cheers > > Erekle > > On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote: > > For any RAID 5 or 6 configuration I normally follow a simple gold rule > which gave good results so far: > - up to 4 disks RAID 5 > - 5 or more disks RAID 6 > > However I didn't really understand well the recommendation to use any RAID > with GlusterFS. I always thought that GlusteFS likes to work in JBOD mode > and control the disks (bricks) directlly so you can create whatever > distribution rule you wish, and if a single disk fails you just replace it > and which obviously have the data replicated from another. The only > downside of using in this way is that the replication data will be flow > accross all servers but that is not much a big issue. > > Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS. > > Thanks > Regards > Fernando > > On 07/08/2017 03:46, Devin Acosta wrote: > > > Moacir, > > I have recently installed multiple Red Hat Virtualization hosts for > several different companies, and have dealt with the Red Hat Support Team > in depth about optimal configuration in regards to setting up GlusterFS > most efficiently and I wanted to share with you what I learned. > > In general Red Hat Virtualization team frowns upon using each DISK of the > system as just a JBOD, sure there is some protection by having the data > replicated, however, the recommendation is to use RAID 6 (preferred) or > RAID-5, or at least RAID-1 at the very least. > > Here is the direct quote from Red Hat when I asked about RAID and Bricks: > > *"A typical Gluster configuration would use RAID underneath the bricks. > RAID 6 is most typical as it gives you 2 disk failure protection, but RAID > 5 could be used too. Once you have the RAIDed bricks, you'd then apply the > desired replication on top of that. The most popular way of doing this > would be distributed replicated with 2x replication. In general you'll get > better performance with larger bricks. 12 drives is often a sweet spot. > Another option would be to create a separate tier using all SSD=E2=80=99s= .=E2=80=9D * > > *In order to SSD tiering from my understanding you would need 1 x NVMe > drive in each server, or 4 x SSD hot tier (it needs to be distributed, > replicated for the hot tier if not using NVME). So with you only having 1 > SSD drive in each server, I=E2=80=99d suggest maybe looking into the NVME= option. * > > *Since your using only 3-servers, what I=E2=80=99d probably suggest is to= do (2 > Replicas + Arbiter Node), this setup actually doesn=E2=80=99t require the= 3rd > server to have big drives at all as it only stores meta-data about the > files and not actually a full copy. * > > *Please see the attached document that was given to me by Red Hat to get > more information on this. Hope this information helps you.* > > > -- > > Devin Acosta, RHCA, RHVCA > Red Hat Certified Architect > > On August 6, 2017 at 7:29:29 PM, Moacir Ferreira ( > moacirferreira(a)hotmail.com) wrote: > > I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU > sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use > GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC a= nd > a dual 10Gb NIC. So my intention is to create a loop like a server triang= le > using the 40Gb NICs for virtualization files (VMs .qcow2) access and to > move VMs around the pod (east /west traffic) while using the 10Gb > interfaces for giving services to the outside world (north/south traffic). > > > This said, my first question is: How should I deploy GlusterFS in such > oVirt scenario? My questions are: > > > 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and > then create a GlusterFS using them? > > 2 - Instead, should I create a JBOD array made of all server's disks? > > 3 - What is the best Gluster configuration to provide for HA while not > consuming too much disk space? > > 4 - Does a oVirt hypervisor pod like I am planning to build, and the > virtualization environment, benefits from tiering when using a SSD disk? > And yes, will Gluster do it by default or I have to configure it to do so? > > > At the bottom line, what is the good practice for using GlusterFS in small > pods for enterprises? > > > You opinion/feedback will be really appreciated! > > Moacir > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > > > _______________________________________________ > Users mailing listUsers(a)ovirt.orghttp://lists.ovirt.org/mailman/listinf= o/users > > > > > _______________________________________________ > Users mailing listUsers(a)ovirt.orghttp://lists.ovirt.org/mailman/listinf= o/users > > > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > --===============8696520672274949902== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+PGRpdiBjbGFz cz0iZ21haWxfcXVvdGUiPk9uIE1vbiwgQXVnIDcsIDIwMTcgYXQgNjo0MSBQTSwgRkVSTkFORE8g RlJFRElBTkkgPHNwYW4gZGlyPSJsdHIiPiZsdDs8YSBocmVmPSJtYWlsdG86ZmVybmFuZG8uZnJl ZGlhbmlAdXB4LmNvbSIgdGFyZ2V0PSJfYmxhbmsiPmZlcm5hbmRvLmZyZWRpYW5pQHVweC5jb208 L2E+Jmd0Ozwvc3Bhbj4gd3JvdGU6PGJyPjxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIg c3R5bGU9Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRlci1sZWZ0OjFweCAjY2NjIHNvbGlkO3BhZGRp bmctbGVmdDoxZXgiPgogIAogICAgCiAgCiAgPGRpdiBiZ2NvbG9yPSIjRkZGRkZGIiB0ZXh0PSIj MDAwMDAwIj4KICAgIDxwPlRoYW5rcyBmb3IgdGhlIGNsYXJpZmljYXRpb24gRXJla2xlLjwvcD4K ICAgIDxwPkhvd2V2ZXIgSSBnZXQgc3VycHJpc2VkIHdpdGggdGhpcyB3YXkgb2Ygb3BlcmF0aW5n IGZyb20gR2x1c3RlckZTCiAgICAgIGFzIGl0IGFkZHMgYW5vdGhlciBsYXllciBvZiBjb21wbGV4 aXR5IHRvIHRoZSBzeXN0ZW0gKGVpdGhlciBhCiAgICAgIGhhcmR3YXJlIG9yIHNvZnR3YXJlIFJB SUQpIGJlZm9yZSB0aGUgZ2x1c3RlciBjb25maWcgYW5kIGluY3JlYXNlCiAgICAgIHRoZSBzeXN0 ZW0mIzM5O3Mgb3ZlcmFsbCBjb3N0cy48YnI+PC9wPjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2Pjxi cj48L2Rpdj48ZGl2Pkl0IGRvZXMsIGJ1dCB3aXRoIEhXIGJhc2VkIFJBSUQgaXQmIzM5O3Mgbm90 IGEgYmlnIGRlYWwuIFRoZSBjb21wbGV4aXR5IGlzIGFsbCB0aGUgc3RyaXBlIHNpemUgbWF0aC4u LiB3aGljaCBJIHBlcnNvbmFsbHkgZG9uJiMzOTt0IGxpa2UgdG8gY2FsY3VsYXRlLjwvZGl2Pjxk aXY+wqA8L2Rpdj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46 MCAwIDAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij48 ZGl2IGJnY29sb3I9IiNGRkZGRkYiIHRleHQ9IiMwMDAwMDAiPjxwPgogICAgPC9wPgogICAgPHA+ QW4gaW1wb3J0YW50IHBvaW50IHRvIGNvbnNpZGVyIGlzOiBJbiBSQUlEIGNvbmZpZ3VyYXRpb24g eW91CiAgICAgIGFscmVhZHkgaGF2ZSBzcGFjZSAmIzM5O3dhc3RlZCYjMzk7IGluIG9yZGVyIHRv IGJ1aWxkIHJlZHVuZGFuY3kgKGVpdGhlcgogICAgICBSQUlEIDEsIDUsIG9yIDYpLiBUaGVuIHdo ZW4geW91IGhhdmUgR2x1c3RlckZTIG9uIHRoZSB0b3Agb2YKICAgICAgc2V2ZXJhbCBSQUlEcyB5 b3UgaGF2ZSBhZ2FpbiBtb3JlIGRhdGEgcmVwbGljYXRlZCBzbyB5b3UgZW5kIHVwCiAgICAgIHdp dGggdGhlIHNhbWUgZGF0YSBjb25zdW1pbmcgbW9yZSBzcGFjZSBpbiBhIGdyb3VwIG9mIGRpc2tz IGFuZAogICAgICBhZ2FpbiBvbiB0aGUgdG9wIG9mIHNldmVyYWwgUkFJRHMgZGVwZW5kaW5nIG9u IHRoZSBHbHVzdGVyCiAgICAgIGNvbmZpZ3VyYXRpb24geW91IGhhdmUgKGluIGEgUkFJRCAxIGNv bmZpZyB0aGUgc2FtZSBkYXRhIGlzCiAgICAgIHJlcGxpY2F0ZWQgNCB0aW1lcykuPC9wPgogICAg PHA+WWV0IGFub3RoZXIgZG93bnNpZGUgb2YgaGF2aW5nIGEgUkFJRCAoc3BlY2lhbGx5IFJBSUQg NSBvciA2KSBpcwogICAgICB0aGF0IGl0IHJlZHVjZXMgY29uc2lkZXJhYmx5IHRoZSB3cml0ZSBz cGVlZHMgYXMgZWFjaCBncm91cCBvZgogICAgICBkaXNrcyB3aWxsIGVuZCB1cCBoYXZpbmcgdGhl IHdyaXRlIHNwZWVkIG9mIGEgc2luZ2xlIGRpc2sgYXMgYWxsCiAgICAgIG90aGVyIGRpc2tzIG9m IHRoYXQgZ3JvdXAgaGF2ZSB0byB3YWl0IGZvciBlYWNoIG90aGVyIHRvIHdyaXRlIGFzCiAgICAg IHdlbGwuPGJyPgogICAgPC9wPgogICAgPHA+VGhlcmVmb3JlIGlmIEdsdXN0ZXIgYWxyZWFkeSBy ZXBsaWNhdGVzIGRhdGEgd2h5IGRvZXMgaXQgY3JlYXRlCiAgICAgIHRoaXMgYmlnIHBhaW4geW91 IG1lbnRpb25lZCBpZiB0aGUgZGF0YSBpcyByZXBsaWNhdGVkIHNvbWV3aGVyZQogICAgICBlbHNl LCBjYW4gc3RpbGwgYmUgcmV0cmlldmVkIHRvIGJvdGggc2VydmUgY2xpZW50cyBhbmQgcmVjb25z dHJ1Y3QKICAgICAgdGhlIGVxdWl2YWxlbnQgZGlzayB3aGVuIGl0IGlzIHJlcGxhY2VkID88L3A+ PC9kaXY+PC9ibG9ja3F1b3RlPjxkaXY+PGJyPjwvZGl2PjxkaXY+SSB0aGluayBpdCYjMzk7cyBh IG1hdHRlciBvZiBob3cgZmFzdCB5b3UgY2FuIHJlcGxhY2UgYSBkaXNrIChvdmVyIGEgbG9uZyB3 ZWVrZW5kPyksIGhvdyByZWxpYWJseSB5b3UgY2FuIGRvIGl0IChwbGVhc2UsIGRvbiYjMzk7dCBw dWxsIHRoZSB3cm9uZyBkaXNrISBJJiMzOTt2ZSBzZWVuIGl0IGhhcHBlbmluZyB0b28gbWFueSB0 aW1lcyEpIGFuZCBob3cgbXVjaCBvZiBhIHBlcmZvcm1hbmNlIGhpdCBhcmUgeW91IHdpbGxpbmcg dG8gYWNjZXB0IHdoaWxlIGluIGRlZ3JhZGVkIG1vZGUgKGFuZCBob3cgbG9uZyBpdCB0b29rIHRv IGRldGVjdCBpdC4gSEREcywgdW5saWtlIFNTRHMsIGRpZSBzbG93bHkuIEF0IGxlYXN0IHdoZW4g U1NEIGRpZXMsIGl0IGRpZXMgYSBxdWljayBhbmQgZGV0ZXJtaW5lZCBkZWF0aC4gSEREcyBtYXkg YWNjdW11bGF0ZSBlcnJvcnMgYW5kIGVycm9ycyBhbmQgc3RpbGwgZnVuY3Rpb24pLjwvZGl2Pjxk aXY+WS48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48 YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAgLjhleDti b3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij48ZGl2IGJnY29sb3I9 IiNGRkZGRkYiIHRleHQ9IiMwMDAwMDAiPjxzcGFuIGNsYXNzPSJIT0VuWmIiPjxmb250IGNvbG9y PSIjODg4ODg4Ij4KICAgIDxwPkZlcm5hbmRvPGJyPgogICAgPC9wPjwvZm9udD48L3NwYW4+PGRp dj48ZGl2IGNsYXNzPSJoNSI+CiAgICA8YnI+CiAgICA8ZGl2IGNsYXNzPSJtXy0zOTAyNzA2ODM1 OTAxNzQ3MDY5bW96LWNpdGUtcHJlZml4Ij5PbiAwNy8wOC8yMDE3IDEwOjI2LCBFcmVrbGUgTWFn cmFkemUKICAgICAgd3JvdGU6PGJyPgogICAgPC9kaXY+CiAgICA8YmxvY2txdW90ZSB0eXBlPSJj aXRlIj4KICAgICAgCiAgICAgIDxwPkhpIEZyZW5hbmRvLDwvcD4KICAgICAgPHA+SGVyZSBpcyBt eSBleHBlcmllbmNlLCBpZiB5b3UgY29uc2lkZXIgYSBwYXJ0aWN1bGFyIGhhcmQgZHJpdmUKICAg ICAgICBhcyBhIGJyaWNrIGZvciBnbHVzdGVyIHZvbHVtZSBhbmQgaXQgZGllcywgaS5lLiBpdCBi ZWNvbWVzIG5vdAogICAgICAgIGFjY2Vzc2libGUgaXQmIzM5O3MgYSBodWdlIGhhc3NsZSB0byBk aXNjYXJkIHRoYXQgYnJpY2sgYW5kIGV4Y2hhbmdlCiAgICAgICAgd2l0aCBhbm90aGVyIG9uZSwg c2luY2UgZ2x1c3RlciBzb21lIHRyaWVzIHRvIGFjY2VzcyB0aGF0IGJyb2tlbgogICAgICAgIGJy aWNrIGFuZCBpdCYjMzk7cyBjYXVzaW5nIChhdCBsZWFzdCBpdCBjYXVzZSBmb3IgbWUpIGEgYmln IHBhaW4sCiAgICAgICAgdGhlcmVmb3JlIGl0JiMzOTtzIGJldHRlciB0byBoYXZlIGEgUkFJRCBh cyBicmljaywgaS5lLiBoYXZlIFJBSUQgMQogICAgICAgIChtaXJyb3JpbmcpIGZvciBlYWNoIGJy aWNrLCBpbiB0aGlzIGNhc2UgaWYgdGhlIGRpc2sgaXMgZG93biB5b3UKICAgICAgICBjYW4gZWFz aWx5IGV4Y2hhbmdlIGl0IGFuZCByZWJ1aWxkIHRoZSBSQUlEIHdpdGhvdXQgZ29pbmcKICAgICAg ICBvZmZsaW5lLCBpLmUgc3dpdGNoaW5nIG9mZiB0aGUgdm9sdW1lIGRvaW5nIGJyaWNrIG1hbmlw dWxhdGlvbnMKICAgICAgICBhbmQgc3dpdGNoaW5nIGl0IGJhY2sgb24uPGJyPgogICAgICA8L3A+ CiAgICAgIDxwPkNoZWVyczwvcD4KICAgICAgPHA+RXJla2xlPGJyPgogICAgICA8L3A+CiAgICAg IDxicj4KICAgICAgPGRpdiBjbGFzcz0ibV8tMzkwMjcwNjgzNTkwMTc0NzA2OW1vei1jaXRlLXBy ZWZpeCI+T24gMDgvMDcvMjAxNyAwMzowNCBQTSwgRkVSTkFORE8KICAgICAgICBGUkVESUFOSSB3 cm90ZTo8YnI+CiAgICAgIDwvZGl2PgogICAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIj4KICAg ICAgICAKICAgICAgICA8cD5Gb3IgYW55IFJBSUQgNSBvciA2IGNvbmZpZ3VyYXRpb24gSSBub3Jt YWxseSBmb2xsb3cgYSBzaW1wbGUKICAgICAgICAgIGdvbGQgcnVsZSB3aGljaCBnYXZlIGdvb2Qg cmVzdWx0cyBzbyBmYXI6PGJyPgogICAgICAgICAgLSB1cCB0byA0IGRpc2tzIFJBSUQgNTxicj4K ICAgICAgICAgIC0gNSBvciBtb3JlIGRpc2tzIFJBSUQgNjwvcD4KICAgICAgICA8cD5Ib3dldmVy IEkgZGlkbiYjMzk7dCByZWFsbHkgdW5kZXJzdGFuZCB3ZWxsIHRoZSByZWNvbW1lbmRhdGlvbiB0 bwogICAgICAgICAgdXNlIGFueSBSQUlEIHdpdGggR2x1c3RlckZTLiBJIGFsd2F5cyB0aG91Z2h0 IHRoYXQgR2x1c3RlRlMKICAgICAgICAgIGxpa2VzIHRvIHdvcmsgaW4gSkJPRCBtb2RlIGFuZCBj b250cm9sIHRoZSBkaXNrcyAoYnJpY2tzKQogICAgICAgICAgZGlyZWN0bGx5IHNvIHlvdSBjYW4g Y3JlYXRlIHdoYXRldmVyIGRpc3RyaWJ1dGlvbiBydWxlIHlvdQogICAgICAgICAgd2lzaCwgYW5k IGlmIGEgc2luZ2xlIGRpc2sgZmFpbHMgeW91IGp1c3QgcmVwbGFjZSBpdCBhbmQgd2hpY2gKICAg ICAgICAgIG9idmlvdXNseSBoYXZlIHRoZSBkYXRhIHJlcGxpY2F0ZWQgZnJvbSBhbm90aGVyLiBU aGUgb25seQogICAgICAgICAgZG93bnNpZGUgb2YgdXNpbmcgaW4gdGhpcyB3YXkgaXMgdGhhdCB0 aGUgcmVwbGljYXRpb24gZGF0YQogICAgICAgICAgd2lsbCBiZSBmbG93IGFjY3Jvc3MgYWxsIHNl cnZlcnMgYnV0IHRoYXQgaXMgbm90IG11Y2ggYSBiaWcKICAgICAgICAgIGlzc3VlLjwvcD4KICAg ICAgICA8cD5BbnlvbmUgY2FuIGVsYWJvcmF0ZSBhYm91dCBVc2luZyBSQUlEICsgR2x1c3RlckZT IGFuZCBKQk9EICsKICAgICAgICAgIEdsdXN0ZXJGUy48L3A+CiAgICAgICAgPHA+VGhhbmtzPGJy PgogICAgICAgICAgUmVnYXJkczxicj4KICAgICAgICAgIEZlcm5hbmRvPGJyPgogICAgICAgIDwv cD4KICAgICAgICA8YnI+CiAgICAgICAgPGRpdiBjbGFzcz0ibV8tMzkwMjcwNjgzNTkwMTc0NzA2 OW1vei1jaXRlLXByZWZpeCI+T24gMDcvMDgvMjAxNyAwMzo0NiwgRGV2aW4gQWNvc3RhCiAgICAg ICAgICB3cm90ZTo8YnI+CiAgICAgICAgPC9kaXY+CiAgICAgICAgPGJsb2NrcXVvdGUgdHlwZT0i Y2l0ZSI+CiAgICAgICAgICAKICAgICAgICAgIDxkaXYgaWQ9Im1fLTM5MDI3MDY4MzU5MDE3NDcw NjlibG9vcF9jdXN0b21mb250IiBzdHlsZT0iY29sb3I6cmdiKDAsMCwwKTttYXJnaW46MHB4Ij48 Zm9udCBmYWNlPSJJbnB1dCBNb25vIj48YnI+CiAgICAgICAgICAgIDwvZm9udD48L2Rpdj4KICAg ICAgICAgIDxkaXYgaWQ9Im1fLTM5MDI3MDY4MzU5MDE3NDcwNjlibG9vcF9jdXN0b21mb250IiBz dHlsZT0iY29sb3I6cmdiKDAsMCwwKTttYXJnaW46MHB4Ij48Zm9udCBmYWNlPSJJbnB1dCBNb25v Ij5Nb2FjaXIsPC9mb250PjwvZGl2PgogICAgICAgICAgPGRpdiBpZD0ibV8tMzkwMjcwNjgzNTkw MTc0NzA2OWJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJjb2xvcjpyZ2IoMCwwLDApO21hcmdpbjow cHgiPjxmb250IGZhY2U9IklucHV0IE1vbm8iPjxicj4KICAgICAgICAgICAgPC9mb250PjwvZGl2 PgogICAgICAgICAgPGRpdiBpZD0ibV8tMzkwMjcwNjgzNTkwMTc0NzA2OWJsb29wX2N1c3RvbWZv bnQiIHN0eWxlPSJjb2xvcjpyZ2IoMCwwLDApO21hcmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0 IE1vbm8iPkkgaGF2ZSByZWNlbnRseSBpbnN0YWxsZWQgbXVsdGlwbGUgUmVkCiAgICAgICAgICAg ICAgSGF0IFZpcnR1YWxpemF0aW9uIGhvc3RzIGZvciBzZXZlcmFsIGRpZmZlcmVudCBjb21wYW5p ZXMsCiAgICAgICAgICAgICAgYW5kIGhhdmUgZGVhbHQgd2l0aCB0aGUgUmVkIEhhdCBTdXBwb3J0 IFRlYW0gaW4gZGVwdGgKICAgICAgICAgICAgICBhYm91dCBvcHRpbWFsIGNvbmZpZ3VyYXRpb24g aW4gcmVnYXJkcyB0byBzZXR0aW5nIHVwCiAgICAgICAgICAgICAgR2x1c3RlckZTIG1vc3QgZWZm aWNpZW50bHkgYW5kIEkgd2FudGVkIHRvIHNoYXJlIHdpdGggeW91CiAgICAgICAgICAgICAgd2hh dCBJIGxlYXJuZWQuPC9mb250PjwvZGl2PgogICAgICAgICAgPGRpdiBpZD0ibV8tMzkwMjcwNjgz NTkwMTc0NzA2OWJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJjb2xvcjpyZ2IoMCwwLDApO21hcmdp bjowcHgiPjxmb250IGZhY2U9IklucHV0IE1vbm8iPjxicj4KICAgICAgICAgICAgPC9mb250Pjwv ZGl2PgogICAgICAgICAgPGRpdiBpZD0ibV8tMzkwMjcwNjgzNTkwMTc0NzA2OWJsb29wX2N1c3Rv bWZvbnQiIHN0eWxlPSJjb2xvcjpyZ2IoMCwwLDApO21hcmdpbjowcHgiPjxmb250IGZhY2U9Iklu cHV0IE1vbm8iPkluIGdlbmVyYWwgUmVkIEhhdCBWaXJ0dWFsaXphdGlvbiB0ZWFtCiAgICAgICAg ICAgICAgZnJvd25zIHVwb24gdXNpbmcgZWFjaCBESVNLIG9mIHRoZSBzeXN0ZW0gYXMganVzdCBh IEpCT0QsCiAgICAgICAgICAgICAgc3VyZSB0aGVyZSBpcyBzb21lIHByb3RlY3Rpb24gYnkgaGF2 aW5nIHRoZSBkYXRhCiAgICAgICAgICAgICAgcmVwbGljYXRlZCwgaG93ZXZlciwgdGhlIHJlY29t bWVuZGF0aW9uIGlzIHRvIHVzZSBSQUlEIDYKICAgICAgICAgICAgICAocHJlZmVycmVkKSBvciBS QUlELTUsIG9yIGF0IGxlYXN0IFJBSUQtMSBhdCB0aGUgdmVyeQogICAgICAgICAgICAgIGxlYXN0 LjwvZm9udD48L2Rpdj4KICAgICAgICAgIDxkaXYgaWQ9Im1fLTM5MDI3MDY4MzU5MDE3NDcwNjli bG9vcF9jdXN0b21mb250IiBzdHlsZT0iY29sb3I6cmdiKDAsMCwwKTttYXJnaW46MHB4Ij48Zm9u dCBmYWNlPSJJbnB1dCBNb25vIj48YnI+CiAgICAgICAgICAgIDwvZm9udD48L2Rpdj4KICAgICAg ICAgIDxkaXYgaWQ9Im1fLTM5MDI3MDY4MzU5MDE3NDcwNjlibG9vcF9jdXN0b21mb250IiBzdHls ZT0ibWFyZ2luOjBweCI+PGZvbnQgZmFjZT0iSW5wdXQgTW9ubyI+SGVyZSBpcyB0aGXCoGRpcmVj dCBxdW90ZSBmcm9tIFJlZCBIYXQKICAgICAgICAgICAgICB3aGVuIEkgYXNrZWQgYWJvdXQgUkFJ RCBhbmQgQnJpY2tzOjwvZm9udD48L2Rpdj4KICAgICAgICAgIDxkaXYgaWQ9Im1fLTM5MDI3MDY4 MzU5MDE3NDcwNjlibG9vcF9jdXN0b21mb250IiBzdHlsZT0ibWFyZ2luOjBweCI+PGZvbnQgZmFj ZT0iSW5wdXQgTW9ubyI+PGk+PGJyPgogICAgICAgICAgICAgIDwvaT48L2ZvbnQ+PC9kaXY+CiAg ICAgICAgICA8ZGl2IGlkPSJtXy0zOTAyNzA2ODM1OTAxNzQ3MDY5Ymxvb3BfY3VzdG9tZm9udCIg c3R5bGU9Im1hcmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0IE1vbm8iPjxpPiZxdW90O0EgdHlw aWNhbCBHbHVzdGVyIGNvbmZpZ3VyYXRpb24KICAgICAgICAgICAgICAgIHdvdWxkIHVzZSBSQUlE IHVuZGVybmVhdGggdGhlIGJyaWNrcy4gUkFJRCA2IGlzIG1vc3QKICAgICAgICAgICAgICAgIHR5 cGljYWwgYXMgaXQgZ2l2ZXMgeW91IDIgZGlzayBmYWlsdXJlIHByb3RlY3Rpb24sIGJ1dAogICAg ICAgICAgICAgICAgUkFJRCA1IGNvdWxkIGJlIHVzZWQgdG9vLsKgT25jZSB5b3UgaGF2ZSB0aGUg UkFJRGVkCiAgICAgICAgICAgICAgICBicmlja3MsIHlvdSYjMzk7ZCB0aGVuIGFwcGx5IHRoZSBk ZXNpcmVkIHJlcGxpY2F0aW9uIG9uIHRvcAogICAgICAgICAgICAgICAgb2YgdGhhdC4gVGhlIG1v c3QgcG9wdWxhciB3YXkgb2YgZG9pbmcgdGhpcyB3b3VsZCBiZQogICAgICAgICAgICAgICAgZGlz dHJpYnV0ZWQgcmVwbGljYXRlZCB3aXRoIDJ4IHJlcGxpY2F0aW9uLsKgSW4gZ2VuZXJhbAogICAg ICAgICAgICAgICAgeW91JiMzOTtsbCBnZXQgYmV0dGVyIHBlcmZvcm1hbmNlIHdpdGggbGFyZ2Vy IGJyaWNrcy4gMTIKICAgICAgICAgICAgICAgIGRyaXZlcyBpcyBvZnRlbiBhIHN3ZWV0IHNwb3Qu IEFub3RoZXIgb3B0aW9uIHdvdWxkIGJlIHRvCiAgICAgICAgICAgICAgICBjcmVhdGUgYSBzZXBh cmF0ZSB0aWVyIHVzaW5nIGFsbCBTU0TigJlzLuKAncKgPC9pPjwvZm9udD48L2Rpdj4KICAgICAg ICAgIDxkaXYgaWQ9Im1fLTM5MDI3MDY4MzU5MDE3NDcwNjlibG9vcF9jdXN0b21mb250IiBzdHls ZT0ibWFyZ2luOjBweCI+PGJyPgogICAgICAgICAgPC9kaXY+CiAgICAgICAgICA8ZGl2IGlkPSJt Xy0zOTAyNzA2ODM1OTAxNzQ3MDY5Ymxvb3BfY3VzdG9tZm9udCIgc3R5bGU9Im1hcmdpbjowcHgi Pjxmb250IGZhY2U9IklucHV0IE1vbm8iPjxpPkluIG9yZGVyIHRvIFNTRMKgdGllcmluZyBmcm9t IG15CiAgICAgICAgICAgICAgICB1bmRlcnN0YW5kaW5nIHlvdSB3b3VsZCBuZWVkIDEgeCBOVk1l IGRyaXZlIGluIGVhY2gKICAgICAgICAgICAgICAgIHNlcnZlciwgb3IgNCB4IFNTRCBob3QgdGll ciAoaXQgbmVlZHMgdG8gYmUgZGlzdHJpYnV0ZWQsCiAgICAgICAgICAgICAgICByZXBsaWNhdGVk IGZvciB0aGUgaG90IHRpZXIgaWYgbm90IHVzaW5nIE5WTUUpLiBTbyB3aXRoCiAgICAgICAgICAg ICAgICB5b3Ugb25seSBoYXZpbmcgMSBTU0QgZHJpdmUgaW4gZWFjaCBzZXJ2ZXIswqBJ4oCZZCBz dWdnZXN0CiAgICAgICAgICAgICAgICBtYXliZSBsb29raW5nIGludG8gdGhlIE5WTUUgb3B0aW9u LsKgPC9pPjwvZm9udD48L2Rpdj4KICAgICAgICAgIDxkaXYgaWQ9Im1fLTM5MDI3MDY4MzU5MDE3 NDcwNjlibG9vcF9jdXN0b21mb250IiBzdHlsZT0ibWFyZ2luOjBweCI+PGZvbnQgZmFjZT0iSW5w dXQgTW9ubyI+PGk+PGJyPgogICAgICAgICAgICAgIDwvaT48L2ZvbnQ+PC9kaXY+CiAgICAgICAg ICA8ZGl2IGlkPSJtXy0zOTAyNzA2ODM1OTAxNzQ3MDY5Ymxvb3BfY3VzdG9tZm9udCIgc3R5bGU9 Im1hcmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0IE1vbm8iPjxpPlNpbmNlIHlvdXIgdXNpbmcg b25seSAzLXNlcnZlcnMsIHdoYXQKICAgICAgICAgICAgICAgIEnigJlkIHByb2JhYmx5IHN1Z2dl c3QgaXMgdG8gZG8gKDIgUmVwbGljYXMgKyBBcmJpdGVyCiAgICAgICAgICAgICAgICBOb2RlKSwg dGhpcyBzZXR1cCBhY3R1YWxseSBkb2VzbuKAmXQgcmVxdWlyZSB0aGUgM3JkCiAgICAgICAgICAg ICAgICBzZXJ2ZXIgdG8gaGF2ZSBiaWcgZHJpdmVzIGF0IGFsbCBhcyBpdCBvbmx5IHN0b3Jlcwog ICAgICAgICAgICAgICAgbWV0YS1kYXRhIGFib3V0IHRoZSBmaWxlcyBhbmQgbm90IGFjdHVhbGx5 IGEgZnVsbCBjb3B5LsKgPC9pPjwvZm9udD48L2Rpdj4KICAgICAgICAgIDxkaXYgaWQ9Im1fLTM5 MDI3MDY4MzU5MDE3NDcwNjlibG9vcF9jdXN0b21mb250IiBzdHlsZT0ibWFyZ2luOjBweCI+PGZv bnQgZmFjZT0iSW5wdXQgTW9ubyI+PGk+PGJyPgogICAgICAgICAgICAgIDwvaT48L2ZvbnQ+PC9k aXY+CiAgICAgICAgICA8ZGl2IGlkPSJtXy0zOTAyNzA2ODM1OTAxNzQ3MDY5Ymxvb3BfY3VzdG9t Zm9udCIgc3R5bGU9Im1hcmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0IE1vbm8iPjxpPlBsZWFz ZSBzZWUgdGhlIGF0dGFjaGVkIGRvY3VtZW50IHRoYXQKICAgICAgICAgICAgICAgIHdhcyBnaXZl biB0byBtZSBieSBSZWQgSGF0IHRvIGdldCBtb3JlIGluZm9ybWF0aW9uIG9uCiAgICAgICAgICAg ICAgICB0aGlzLiBIb3BlIHRoaXMgaW5mb3JtYXRpb24gaGVscHMgeW91LjwvaT48L2ZvbnQ+PC9k aXY+CiAgICAgICAgICA8ZGl2IGlkPSJtXy0zOTAyNzA2ODM1OTAxNzQ3MDY5Ymxvb3BfY3VzdG9t Zm9udCIgc3R5bGU9Im1hcmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0IE1vbm8iPjxpPjxicj4K ICAgICAgICAgICAgICA8L2k+PC9mb250PjwvZGl2PgogICAgICAgICAgPGJyPgogICAgICAgICAg PGRpdiBpZD0ibV8tMzkwMjcwNjgzNTkwMTc0NzA2OWJsb29wX3NpZ25fMTUwMjA4NzM3NjcyNTQ2 OTE4NCIgY2xhc3M9Im1fLTM5MDI3MDY4MzU5MDE3NDcwNjlibG9vcF9zaWduIj48c3Bhbj4tLTwv c3Bhbj48YnI+CiAgICAgICAgICAgIDxkaXYgY2xhc3M9Im1fLTM5MDI3MDY4MzU5MDE3NDcwNjln bWFpbF9zaWduYXR1cmUiPgogICAgICAgICAgICAgIDxkaXYgZGlyPSJsdHIiPgogICAgICAgICAg ICAgICAgPGRpdj48YnI+CiAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgIDxk aXY+RGV2aW4gQWNvc3RhLCBSSENBLCBSSFZDQTwvZGl2PgogICAgICAgICAgICAgICAgPGRpdj5S ZWQgSGF0IENlcnRpZmllZCBBcmNoaXRlY3Q8L2Rpdj4KICAgICAgICAgICAgICA8L2Rpdj4KICAg ICAgICAgICAgPC9kaXY+CiAgICAgICAgICA8L2Rpdj4KICAgICAgICAgIDxicj4KICAgICAgICAg IDxwIGNsYXNzPSJtXy0zOTAyNzA2ODM1OTAxNzQ3MDY5YWlybWFpbF9vbiI+T24gQXVndXN0IDYs IDIwMTcgYXQgNzoyOToyOSBQTSwgTW9hY2lyCiAgICAgICAgICAgIEZlcnJlaXJhICg8YSBocmVm PSJtYWlsdG86bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb20iIHRhcmdldD0iX2JsYW5rIj5tb2Fj aXJmZXJyZWlyYUBob3RtYWlsLmNvbTwvYT4pCiAgICAgICAgICAgIHdyb3RlOjwvcD4KICAgICAg ICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiIGNsYXNzPSJtXy0zOTAyNzA2ODM1OTAxNzQ3MDY5 Y2xlYW5fYnEiPjxzcGFuPgogICAgICAgICAgICAgIDxkaXYgZGlyPSJsdHIiPgogICAgICAgICAg ICAgICAgPGRpdj4KICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgIDxkaXYgaWQ9 Im1fLTM5MDI3MDY4MzU5MDE3NDcwNjlkaXZ0YWdkZWZhdWx0d3JhcHBlciIgc3R5bGU9ImZvbnQt c2l6ZToxMnB0O2NvbG9yOiMwMDAwMDA7Zm9udC1mYW1pbHk6Q2FsaWJyaSxIZWx2ZXRpY2Esc2Fu cy1zZXJpZiIgZGlyPSJsdHIiPgogICAgICAgICAgICAgICAgICAgIDxwPjxzcGFuPkkgYW0gd2ls bGluZyB0byBhc3NlbWJsZSBhIG9WaXJ0ICZxdW90O3BvZCZxdW90OywKICAgICAgICAgICAgICAg ICAgICAgICAgbWFkZSBvZiAzIHNlcnZlcnMsIGVhY2ggd2l0aCAyIENQVSBzb2NrZXRzIG9mIDEy CiAgICAgICAgICAgICAgICAgICAgICAgIGNvcmVzLCAyNTZHQiBSQU0sIDcgSEREIDEwSywgMSBT U0QuIFRoZSBpZGVhIGlzCiAgICAgICAgICAgICAgICAgICAgICAgIHRvIHVzZSBHbHVzdGVyRlMg dG8gcHJvdmlkZSBIQSBmb3IgdGhlIFZNcy4gVGhlCiAgICAgICAgICAgICAgICAgICAgICAgIDMg c2VydmVycyBoYXZlIGEgZHVhbCA0MEdiIE5JQyBhbmQgYSBkdWFsIDEwR2IKICAgICAgICAgICAg ICAgICAgICAgICAgTklDLiBTbyBteSBpbnRlbnRpb24gaXMgdG8gY3JlYXRlIGEgbG9vcCBsaWtl IGEKICAgICAgICAgICAgICAgICAgICAgICAgc2VydmVyIHRyaWFuZ2xlIHVzaW5nIHRoZSA0MEdi IE5JQ3MgZm9yCiAgICAgICAgICAgICAgICAgICAgICAgIHZpcnR1YWxpemF0aW9uIGZpbGVzIChW TXMgLnFjb3cyKSBhY2Nlc3MgYW5kIHRvCiAgICAgICAgICAgICAgICAgICAgICAgIG1vdmUgVk1z IGFyb3VuZCB0aGUgcG9kIChlYXN0IC93ZXN0IHRyYWZmaWMpCiAgICAgICAgICAgICAgICAgICAg ICAgIHdoaWxlIHVzaW5nIHRoZSAxMEdiIGludGVyZmFjZXMgZm9yIGdpdmluZwogICAgICAgICAg ICAgICAgICAgICAgICBzZXJ2aWNlcyB0byB0aGUgb3V0c2lkZSB3b3JsZCAobm9ydGgvc291dGgK ICAgICAgICAgICAgICAgICAgICAgICAgdHJhZmZpYykuPC9zcGFuPjwvcD4KICAgICAgICAgICAg ICAgICAgICA8cD48YnI+CiAgICAgICAgICAgICAgICAgICAgPC9wPgogICAgICAgICAgICAgICAg ICAgIDxwPlRoaXMgc2FpZCwgbXkgZmlyc3QgcXVlc3Rpb24gaXM6IEhvdyBzaG91bGQgSQogICAg ICAgICAgICAgICAgICAgICAgZGVwbG95IEdsdXN0ZXJGUyBpbiBzdWNoIG9WaXJ0IHNjZW5hcmlv PyBNeQogICAgICAgICAgICAgICAgICAgICAgcXVlc3Rpb25zIGFyZTo8L3A+CiAgICAgICAgICAg ICAgICAgICAgPHA+PGJyPgogICAgICAgICAgICAgICAgICAgIDwvcD4KICAgICAgICAgICAgICAg ICAgICA8cD4xIC0gU2hvdWxkIEkgY3JlYXRlIDMgUkFJRCAoaS5lLjogUkFJRCA1KSwgb25lIG9u CiAgICAgICAgICAgICAgICAgICAgICBlYWNoIG9WaXJ0IG5vZGUsIGFuZCB0aGVuIGNyZWF0ZSBh IEdsdXN0ZXJGUyB1c2luZwogICAgICAgICAgICAgICAgICAgICAgdGhlbT88L3A+CiAgICAgICAg ICAgICAgICAgICAgPHA+MiAtIEluc3RlYWQsIHNob3VsZCBJIGNyZWF0ZSBhIEpCT0QgYXJyYXkg bWFkZSBvZgogICAgICAgICAgICAgICAgICAgICAgYWxsIHNlcnZlciYjMzk7cyBkaXNrcz88L3A+ CiAgICAgICAgICAgICAgICAgICAgPHA+MyAtIFdoYXQgaXMgdGhlIGJlc3QgR2x1c3RlciBjb25m aWd1cmF0aW9uIHRvCiAgICAgICAgICAgICAgICAgICAgICBwcm92aWRlIGZvciBIQSB3aGlsZSBu b3QgY29uc3VtaW5nIHRvbyBtdWNoIGRpc2sKICAgICAgICAgICAgICAgICAgICAgIHNwYWNlPzxi cj4KICAgICAgICAgICAgICAgICAgICA8L3A+CiAgICAgICAgICAgICAgICAgICAgPHA+NCAtIERv ZXMgYSBvVmlydCBoeXBlcnZpc29yIHBvZCBsaWtlIEkgYW0KICAgICAgICAgICAgICAgICAgICAg IHBsYW5uaW5nIHRvIGJ1aWxkLCBhbmQgdGhlIHZpcnR1YWxpemF0aW9uCiAgICAgICAgICAgICAg ICAgICAgICBlbnZpcm9ubWVudCwgYmVuZWZpdHMgZnJvbSB0aWVyaW5nIHdoZW4gdXNpbmcgYQog ICAgICAgICAgICAgICAgICAgICAgU1NEIGRpc2s/IEFuZCB5ZXMsIHdpbGwgR2x1c3RlciBkbyBp dCBieSBkZWZhdWx0CiAgICAgICAgICAgICAgICAgICAgICBvciBJIGhhdmUgdG8gY29uZmlndXJl IGl0IHRvIGRvIHNvPzwvcD4KICAgICAgICAgICAgICAgICAgICA8cD48YnI+CiAgICAgICAgICAg ICAgICAgICAgPC9wPgogICAgICAgICAgICAgICAgICAgIDxwPkF0IHRoZSBib3R0b20gbGluZSwg d2hhdCBpcyB0aGUgZ29vZCBwcmFjdGljZSBmb3IKICAgICAgICAgICAgICAgICAgICAgIHVzaW5n IEdsdXN0ZXJGUyBpbiBzbWFsbCBwb2RzIGZvciBlbnRlcnByaXNlcz88YnI+CiAgICAgICAgICAg ICAgICAgICAgPC9wPgogICAgICAgICAgICAgICAgICAgIDxwPjxicj4KICAgICAgICAgICAgICAg ICAgICA8L3A+CiAgICAgICAgICAgICAgICAgICAgPHA+WW91IG9waW5pb24vZmVlZGJhY2sgd2ls bCBiZSByZWFsbHkgYXBwcmVjaWF0ZWQhPC9wPgogICAgICAgICAgICAgICAgICAgIDxwPk1vYWNp cjxicj4KICAgICAgICAgICAgICAgICAgICA8L3A+CiAgICAgICAgICAgICAgICAgIDwvZGl2Pgog ICAgICAgICAgICAgICAgICBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX188d2JyPl9fX19f X19fX19fX19fX19fIDxicj4KICAgICAgICAgICAgICAgICAgVXNlcnMgbWFpbGluZyBsaXN0IDxi cj4KICAgICAgICAgICAgICAgICAgPGEgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgdGFy Z2V0PSJfYmxhbmsiPlVzZXJzQG92aXJ0Lm9yZzwvYT4gPGJyPgogICAgICAgICAgICAgICAgICA8 YSBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiIHRh cmdldD0iX2JsYW5rIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnLzx3YnI+bWFpbG1hbi9saXN0aW5m by91c2VyczwvYT4KICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAgPC9kaXY+ CiAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgIDwvc3Bhbj48L2Jsb2NrcXVvdGU+CiAg ICAgICAgICA8YnI+CiAgICAgICAgICA8ZmllbGRzZXQgY2xhc3M9Im1fLTM5MDI3MDY4MzU5MDE3 NDcwNjltaW1lQXR0YWNobWVudEhlYWRlciI+PC9maWVsZHNldD4KICAgICAgICAgIDxicj4KICAg ICAgICAgIDxwcmU+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fPHdicj5fX19fX19fX19f X19fX19fXwpVc2VycyBtYWlsaW5nIGxpc3QKPGEgY2xhc3M9Im1fLTM5MDI3MDY4MzU5MDE3NDcw Njltb3otdHh0LWxpbmstYWJicmV2aWF0ZWQiIGhyZWY9Im1haWx0bzpVc2Vyc0BvdmlydC5vcmci IHRhcmdldD0iX2JsYW5rIj5Vc2Vyc0BvdmlydC5vcmc8L2E+CjxhIGNsYXNzPSJtXy0zOTAyNzA2 ODM1OTAxNzQ3MDY5bW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwOi8vbGlzdHMub3Zp cnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiIHRhcmdldD0iX2JsYW5rIj5odHRwOi8vbGlz dHMub3ZpcnQub3JnLzx3YnI+bWFpbG1hbi9saXN0aW5mby91c2VyczwvYT4KPC9wcmU+CiAgICAg ICAgPC9ibG9ja3F1b3RlPgogICAgICAgIDxicj4KICAgICAgICA8YnI+CiAgICAgICAgPGZpZWxk c2V0IGNsYXNzPSJtXy0zOTAyNzA2ODM1OTAxNzQ3MDY5bWltZUF0dGFjaG1lbnRIZWFkZXIiPjwv ZmllbGRzZXQ+CiAgICAgICAgPGJyPgogICAgICAgIDxwcmU+X19fX19fX19fX19fX19fX19fX19f X19fX19fX19fPHdicj5fX19fX19fX19fX19fX19fXwpVc2VycyBtYWlsaW5nIGxpc3QKPGEgY2xh c3M9Im1fLTM5MDI3MDY4MzU5MDE3NDcwNjltb3otdHh0LWxpbmstYWJicmV2aWF0ZWQiIGhyZWY9 Im1haWx0bzpVc2Vyc0BvdmlydC5vcmciIHRhcmdldD0iX2JsYW5rIj5Vc2Vyc0BvdmlydC5vcmc8 L2E+CjxhIGNsYXNzPSJtXy0zOTAyNzA2ODM1OTAxNzQ3MDY5bW96LXR4dC1saW5rLWZyZWV0ZXh0 IiBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiIHRh cmdldD0iX2JsYW5rIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnLzx3YnI+bWFpbG1hbi9saXN0aW5m by91c2VyczwvYT4KPC9wcmU+CiAgICAgIDwvYmxvY2txdW90ZT4KICAgICAgPGJyPgogICAgPC9i bG9ja3F1b3RlPgogICAgPGJyPgogIDwvZGl2PjwvZGl2PjwvZGl2PgoKPGJyPl9fX19fX19fX19f X19fX19fX19fX19fX19fX19fXzx3YnI+X19fX19fX19fX19fX19fX188YnI+ClVzZXJzIG1haWxp bmcgbGlzdDxicj4KPGEgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyI+VXNlcnNAb3ZpcnQu b3JnPC9hPjxicj4KPGEgaHJlZj0iaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3Rp bmZvL3VzZXJzIiByZWw9Im5vcmVmZXJyZXIiIHRhcmdldD0iX2JsYW5rIj5odHRwOi8vbGlzdHMu b3ZpcnQub3JnLzx3YnI+bWFpbG1hbi9saXN0aW5mby91c2VyczwvYT48YnI+Cjxicj48L2Jsb2Nr cXVvdGU+PC9kaXY+PGJyPjwvZGl2PjwvZGl2Pgo= --===============8696520672274949902==-- From ykaul at redhat.com Mon Aug 7 18:48:51 2017 Content-Type: multipart/mixed; boundary="===============0996246034367816116==" MIME-Version: 1.0 From: Yaniv Kaul To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Mon, 07 Aug 2017 21:48:08 +0300 Message-ID: In-Reply-To: CANvHAxSD0oOUfysYj-sS+fHB6hP3Y3+w8b1EM3m4KJMVmGfLbA@mail.gmail.com --===============0996246034367816116== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Mon, Aug 7, 2017 at 2:41 PM, Colin Coe wrote: > Hi > > I just thought that you'd do hardware RAID if you had the controller or > JBOD if you didn't. In hindsight, a server with 40Gbps NICs is pretty > likely to have a hardware RAID controller. I've never done JBOD with > hardware RAID. I think having a single gluster brick on hardware JBOD > would be riskier than multiple bricks, each on a single disk, but thats n= ot > based on anything other than my prejudices. > > I thought gluster tiering was for the most frequently accessed files, in > which case all the VMs disks would end up in the hot tier. However, I ha= ve > been wrong before... > The most frequent shards, may not be complete files. Y. > I just wanted to know where the OS was going as I didn't see it mentioned > in the OP. Normally, I'd have the OS on a RAID1 but in your case thats a > lot of wasted disk. > > Honestly, I think Yaniv's answer was far better than my own and made the > important point about having an arbiter. > > Thanks > > On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreira < > moacirferreira(a)hotmail.com> wrote: > >> Hi Colin, >> >> >> I am in Portugal, so sorry for this late response. It is quite confusing >> for me, please consider: >> >> >> 1* - *What if the RAID is done by the server's disk controller, not by >> software? >> >> 2 - For JBOD I am just using gdeploy to deploy it. However, I am not >> using the oVirt node GUI to do this. >> >> >> 3 - As the VM .qcow2 files are quite big, tiering would only help if >> made by an intelligent system that uses SSD for chunks of data not for t= he >> entire .qcow2 file. But I guess this is a problem everybody else has. So, >> Do you know how tiering works in Gluster? >> >> >> 4 - I am putting the OS on the first disk. However, would you do >> differently? >> >> >> Moacir >> >> ------------------------------ >> *From:* Colin Coe >> *Sent:* Monday, August 7, 2017 4:48 AM >> *To:* Moacir Ferreira >> *Cc:* users(a)ovirt.org >> *Subject:* Re: [ovirt-users] Good practices >> >> 1) RAID5 may be a performance hit- >> >> 2) I'd be inclined to do this as JBOD by creating a distributed disperse >> volume on each server. Something like >> >> echo gluster volume create dispersevol disperse-data 5 redundancy 2 \ >> $(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e >> "server${SERVER}:/brick/brick-${SERVER}${BRICK}/brick \c"; done; done) >> >> 3) I think the above. >> >> 4) Gluster does support tiering, but IIRC you'd need the same number of >> SSD as spindle drives. There may be another way to use the SSD as a fast >> cache. >> >> Where are you putting the OS? >> >> Hope I understood the question... >> >> Thanks >> >> On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira < >> moacirferreira(a)hotmail.com> wrote: >> >>> I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 >>> CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use >>> GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC= and >>> a dual 10Gb NIC. So my intention is to create a loop like a server tria= ngle >>> using the 40Gb NICs for virtualization files (VMs .qcow2) access and to >>> move VMs around the pod (east /west traffic) while using the 10Gb >>> interfaces for giving services to the outside world (north/south traffi= c). >>> >>> >>> This said, my first question is: How should I deploy GlusterFS in such >>> oVirt scenario? My questions are: >>> >>> >>> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and >>> then create a GlusterFS using them? >>> >>> 2 - Instead, should I create a JBOD array made of all server's disks? >>> >>> 3 - What is the best Gluster configuration to provide for HA while not >>> consuming too much disk space? >>> >>> 4 - Does a oVirt hypervisor pod like I am planning to build, and the >>> virtualization environment, benefits from tiering when using a SSD disk? >>> And yes, will Gluster do it by default or I have to configure it to do = so? >>> >>> >>> At the bottom line, what is the good practice for using GlusterFS in >>> small pods for enterprises? >>> >>> >>> You opinion/feedback will be really appreciated! >>> >>> Moacir >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >> > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > --===============0996246034367816116== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+PGRpdiBjbGFz cz0iZ21haWxfcXVvdGUiPk9uIE1vbiwgQXVnIDcsIDIwMTcgYXQgMjo0MSBQTSwgQ29saW4gQ29l IDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgaHJlZj0ibWFpbHRvOmNvbGluLmNvZUBnbWFpbC5jb20i IHRhcmdldD0iX2JsYW5rIj5jb2xpbi5jb2VAZ21haWwuY29tPC9hPiZndDs8L3NwYW4+IHdyb3Rl Ojxicj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MHB4IDBw eCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGlu Zy1sZWZ0OjFleCI+PGRpdiBkaXI9Imx0ciI+SGk8ZGl2Pjxicj48L2Rpdj48ZGl2PkkganVzdCB0 aG91Z2h0IHRoYXQgeW91JiMzOTtkIGRvIGhhcmR3YXJlIFJBSUQgaWYgeW91IGhhZCB0aGUgY29u dHJvbGxlciBvciBKQk9EIGlmIHlvdSBkaWRuJiMzOTt0LsKgIEluIGhpbmRzaWdodCwgYSBzZXJ2 ZXIgd2l0aCA0MEdicHMgTklDcyBpcyBwcmV0dHkgbGlrZWx5IHRvIGhhdmUgYSBoYXJkd2FyZSBS QUlEIGNvbnRyb2xsZXIuwqAgSSYjMzk7dmUgbmV2ZXIgZG9uZSBKQk9EIHdpdGggaGFyZHdhcmUg UkFJRC7CoCBJIHRoaW5rIGhhdmluZyBhIHNpbmdsZSBnbHVzdGVyIGJyaWNrIG9uIGhhcmR3YXJl IEpCT0Qgd291bGQgYmUgcmlza2llciB0aGFuIG11bHRpcGxlIGJyaWNrcywgZWFjaCBvbiBhIHNp bmdsZSBkaXNrLCBidXQgdGhhdHMgbm90IGJhc2VkIG9uIGFueXRoaW5nIG90aGVyIHRoYW4gbXkg cHJlanVkaWNlcy48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PkkgdGhvdWdodCBnbHVzdGVyIHRp ZXJpbmcgd2FzIGZvciB0aGUgbW9zdCBmcmVxdWVudGx5IGFjY2Vzc2VkIGZpbGVzLCBpbiB3aGlj aCBjYXNlIGFsbCB0aGUgVk1zIGRpc2tzIHdvdWxkIGVuZCB1cCBpbiB0aGUgaG90IHRpZXIuwqAg SG93ZXZlciwgSSBoYXZlIGJlZW4gd3JvbmcgYmVmb3JlLi4uPC9kaXY+PC9kaXY+PC9ibG9ja3F1 b3RlPjxkaXY+PGJyPjwvZGl2PjxkaXY+VGhlIG1vc3QgZnJlcXVlbnQgc2hhcmRzLCBtYXkgbm90 IGJlIGNvbXBsZXRlIGZpbGVzLsKgPC9kaXY+PGRpdj5ZLjxicj48L2Rpdj48ZGl2Pjxicj48L2Rp dj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MHB4IDBweCAw cHggMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1s ZWZ0OjFleCI+PGRpdiBkaXI9Imx0ciI+PGRpdj48YnI+PC9kaXY+PGRpdj5JIGp1c3Qgd2FudGVk IHRvIGtub3cgd2hlcmUgdGhlIE9TIHdhcyBnb2luZyBhcyBJIGRpZG4mIzM5O3Qgc2VlIGl0IG1l bnRpb25lZCBpbiB0aGUgT1AuwqAgTm9ybWFsbHksIEkmIzM5O2QgaGF2ZSB0aGUgT1Mgb24gYSBS QUlEMSBidXQgaW4geW91ciBjYXNlIHRoYXRzIGEgbG90IG9mIHdhc3RlZCBkaXNrLjwvZGl2Pjxk aXY+PGJyPjwvZGl2PjxkaXY+SG9uZXN0bHksIEkgdGhpbmsgWWFuaXYmIzM5O3MgYW5zd2VyIHdh cyBmYXIgYmV0dGVyIHRoYW4gbXkgb3duIGFuZCBtYWRlIHRoZSBpbXBvcnRhbnQgcG9pbnQgYWJv dXQgaGF2aW5nIGFuIGFyYml0ZXIuwqA8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PlRoYW5rczwv ZGl2PjwvZGl2PjxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+PGRpdiBjbGFzcz0iZ21haWxf cXVvdGUiPk9uIE1vbiwgQXVnIDcsIDIwMTcgYXQgNTo1NiBQTSwgTW9hY2lyIEZlcnJlaXJhIDxz cGFuIGRpcj0ibHRyIj4mbHQ7PGEgaHJlZj0ibWFpbHRvOm1vYWNpcmZlcnJlaXJhQGhvdG1haWwu Y29tIiB0YXJnZXQ9Il9ibGFuayI+bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb208L2E+Jmd0Ozwv c3Bhbj4gd3JvdGU6PGJyPjxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1h cmdpbjowcHggMHB4IDBweCAwLjhleDtib3JkZXItbGVmdDoxcHggc29saWQgcmdiKDIwNCwyMDQs MjA0KTtwYWRkaW5nLWxlZnQ6MWV4Ij4KCgoKCjxkaXYgZGlyPSJsdHIiPgo8ZGl2IGlkPSJnbWFp bC1tXy04ODg3ODM4NjU4MjE1OTgyMDMzbV80ODMxODg2ODgzMzA1NjcyNzE4ZGl2dGFnZGVmYXVs dHdyYXBwZXIiIHN0eWxlPSJmb250LXNpemU6MTJwdDtjb2xvcjpyZ2IoMCwwLDApO2ZvbnQtZmFt aWx5OkNhbGlicmksSGVsdmV0aWNhLHNhbnMtc2VyaWYiIGRpcj0ibHRyIj4KPHA+SGkgQ29saW4s PC9wPgo8cD48YnI+CjwvcD4KPHA+PHNwYW4+SSBhbSBpbiBQb3J0dWdhbDwvc3Bhbj4sIHNvIHNv cnJ5IGZvciB0aGlzIGxhdGUgcmVzcG9uc2UuIEl0IGlzIHF1aXRlIGNvbmZ1c2luZyBmb3IgbWUs IHBsZWFzZSBjb25zaWRlcjo8L3A+CjxwPjwvcD4KPGRpdj48Yj48YnI+CjwvYj4xPGI+IC0gPC9i PldoYXQgaWYgdGhlIFJBSUQgaXMgZG9uZSBieSB0aGUgc2VydmVyJiMzOTtzIGRpc2sgY29udHJv bGxlciwgbm90IGJ5IHNvZnR3YXJlPzwvZGl2Pgo8YnI+CjxwPjwvcD4KPHA+MiAtPGI+IDwvYj5G b3IgSkJPRCBJIGFtIGp1c3QgdXNpbmcgZ2RlcGxveSB0byBkZXBsb3kgaXQuIEhvd2V2ZXIsIEkg YW0gbm90IHVzaW5nIHRoZSBvVmlydCBub2RlIEdVSSB0byBkbyB0aGlzLjwvcD4KPHA+PGJyPgo8 L3A+CjxwPjMgLTxiPiA8L2I+QXMgdGhlIFZNIC5xY293MsKgZmlsZXMgYXJlIHF1aXRlIGJpZywg dGllcmluZyB3b3VsZCBvbmx5IGhlbHAgaWYgbWFkZSBieSBhbiBpbnRlbGxpZ2VudCBzeXN0ZW0g dGhhdCB1c2VzIFNTRCBmb3IgY2h1bmtzIG9mIGRhdGEgbm90IGZvciB0aGUgZW50aXJlIC5xY293 MiBmaWxlLiBCdXQgSSBndWVzcyB0aGlzIGlzIGEgcHJvYmxlbSBldmVyeWJvZHkgZWxzZSBoYXMu IFNvLCBEbyB5b3Uga25vdyBob3cgdGllcmluZyB3b3JrcwogaW4gR2x1c3Rlcj88YnI+CjwvcD4K PHA+PGJyPgo8L3A+CjxwPjQgLcKgSSBhbSBwdXR0aW5nIHRoZSBPUyBvbiB0aGUgZmlyc3QgZGlz ay4gSG93ZXZlciwgd291bGQgeW91IGRvIGRpZmZlcmVudGx5Pzxicj4KPC9wPgo8cD48YnI+Cjwv cD4KTW9hY2lyPGJyPgo8YnI+CjxkaXYgc3R5bGU9ImNvbG9yOnJnYig0OSw1NSw1NykiPgo8aHIg c3R5bGU9ImRpc3BsYXk6aW5saW5lLWJsb2NrO3dpZHRoOjk4JSI+CjxkaXYgaWQ9ImdtYWlsLW1f LTg4ODc4Mzg2NTgyMTU5ODIwMzNtXzQ4MzE4ODY4ODMzMDU2NzI3MThkaXZScGx5RndkTXNnIiBk aXI9Imx0ciI+PGZvbnQgc3R5bGU9ImZvbnQtc2l6ZToxMXB0IiBmYWNlPSJDYWxpYnJpLCBzYW5z LXNlcmlmIiBjb2xvcj0iIzAwMDAwMCI+PGI+RnJvbTo8L2I+IENvbGluIENvZSAmbHQ7PGEgaHJl Zj0ibWFpbHRvOmNvbGluLmNvZUBnbWFpbC5jb20iIHRhcmdldD0iX2JsYW5rIj5jb2xpbi5jb2VA Z21haWwuY29tPC9hPiZndDs8YnI+CjxiPlNlbnQ6PC9iPiBNb25kYXksIEF1Z3VzdCA3LCAyMDE3 IDQ6NDggQU08c3BhbiBjbGFzcz0iZ21haWwtIj48YnI+CjxiPlRvOjwvYj4gTW9hY2lyIEZlcnJl aXJhPGJyPgo8Yj5DYzo8L2I+IDxhIGhyZWY9Im1haWx0bzp1c2Vyc0BvdmlydC5vcmciIHRhcmdl dD0iX2JsYW5rIj51c2Vyc0BvdmlydC5vcmc8L2E+PGJyPgo8Yj5TdWJqZWN0OjwvYj4gUmU6IFtv dmlydC11c2Vyc10gR29vZCBwcmFjdGljZXM8L3NwYW4+PC9mb250Pgo8ZGl2PsKgPC9kaXY+Cjwv ZGl2Pgo8ZGl2Pgo8ZGl2IGRpcj0ibHRyIj4xKSBSQUlENSBtYXkgYmUgYSBwZXJmb3JtYW5jZSBo aXQtIDxicj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PjxzcGFuIGNsYXNzPSJnbWFpbC0iPjxzcGFu PjIpIEkmIzM5O2QgYmUgaW5jbGluZWQgdG8gZG8gdGhpcyBhcyBKQk9EIGJ5IGNyZWF0aW5nIGEg ZGlzdHJpYnV0ZWQgZGlzcGVyc2Ugdm9sdW1lIG9uIGVhY2ggc2VydmVyLsKgIFNvbWV0aGluZyBs aWtlCjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj5lY2hvIGdsdXN0ZXIgdm9sdW1lIGNyZWF0ZSBkaXNw ZXJzZXZvbCBkaXNwZXJzZS1kYXRhIDUgcmVkdW5kYW5jeSAyIFw8L2Rpdj4KPGRpdj4kKGZvciBT RVJWRVIgaW4gYSBiIGM7IGRvIGZvciBCUklDSyBpbiAkKHNlcSAxIDUpOyBkbyBlY2hvIC1lICZx dW90O3NlcnZlciR7U0VSVkVSfTovYnJpY2svYnJpY2stPHdicj4ke1NFUlZFUn0ke0JSSUNLfS9i cmljayBcYyZxdW90OzsgZG9uZTsgZG9uZSk8L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8L3NwYW4+ PC9zcGFuPjxkaXY+MykgSSB0aGluayB0aGUgYWJvdmUuIDxiPjwvYj48L2Rpdj48c3BhbiBjbGFz cz0iZ21haWwtIj48c3Bhbj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PjQpIEdsdXN0ZXIgZG9lcyBz dXBwb3J0IHRpZXJpbmcsIGJ1dCBJSVJDIHlvdSYjMzk7ZCBuZWVkIHRoZSBzYW1lIG51bWJlciBv ZiBTU0QgYXMgc3BpbmRsZSBkcml2ZXMuwqAgVGhlcmUgbWF5IGJlIGFub3RoZXIgd2F5IHRvIHVz ZSB0aGUgU1NEIGFzIGEgZmFzdCBjYWNoZS7CoDwvZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+ V2hlcmUgYXJlIHlvdSBwdXR0aW5nIHRoZSBPUz88L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2 PkhvcGUgSSB1bmRlcnN0b29kIHRoZSBxdWVzdGlvbi4uLjwvZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+ CjxkaXY+VGhhbmtzPC9kaXY+Cjwvc3Bhbj48L3NwYW4+PC9kaXY+CjwvZGl2PjxzcGFuIGNsYXNz PSJnbWFpbC0iPjxzcGFuPgo8ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+PGJyPgo8ZGl2IGNsYXNz PSJnbWFpbF9xdW90ZSI+T24gU3VuLCBBdWcgNiwgMjAxNyBhdCAxMDo0OSBQTSwgTW9hY2lyIEZl cnJlaXJhIDxzcGFuIGRpcj0ibHRyIj4KJmx0OzxhIGhyZWY9Im1haWx0bzptb2FjaXJmZXJyZWly YUBob3RtYWlsLmNvbSIgdGFyZ2V0PSJfYmxhbmsiPm1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY29t PC9hPiZndDs8L3NwYW4+IHdyb3RlOjxicj4KPGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3Rl IiBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCBy Z2IoMjA0LDIwNCwyMDQpO3BhZGRpbmctbGVmdDoxZXgiPgo8ZGl2IGRpcj0ibHRyIj4KPGRpdiBp ZD0iZ21haWwtbV8tODg4NzgzODY1ODIxNTk4MjAzM21fNDgzMTg4Njg4MzMwNTY3MjcxOG1fMjQ2 MDk4NTY5MTc0NjQ5ODMyMmRpdnRhZ2RlZmF1bHR3cmFwcGVyIiBkaXI9Imx0ciIgc3R5bGU9ImZv bnQtc2l6ZToxMnB0O2NvbG9yOnJnYigwLDAsMCk7Zm9udC1mYW1pbHk6Q2FsaWJyaSxIZWx2ZXRp Y2Esc2Fucy1zZXJpZiI+CjxwPjxzcGFuPkkgYW0gd2lsbGluZyB0byBhc3NlbWJsZSBhIG9WaXJ0 ICZxdW90O3BvZCZxdW90OywgbWFkZSBvZiAzIHNlcnZlcnMsIGVhY2ggd2l0aCAyIENQVSBzb2Nr ZXRzIG9mIDEyIGNvcmVzLCAyNTZHQiBSQU0sIDcgSEREIDEwSywgMSBTU0QuIFRoZSBpZGVhIGlz IHRvIHVzZSBHbHVzdGVyRlMgdG8gcHJvdmlkZSBIQSBmb3IgdGhlIFZNcy4gVGhlIDMgc2VydmVy cyBoYXZlIGEgZHVhbCA0MEdiIE5JQyBhbmQgYSBkdWFsIDEwR2IgTklDLiBTbyBteSBpbnRlbnRp b24KIGlzIHRvIGNyZWF0ZSBhIGxvb3AgbGlrZSBhIHNlcnZlciB0cmlhbmdsZSB1c2luZyB0aGUg NDBHYiBOSUNzIGZvciB2aXJ0dWFsaXphdGlvbiBmaWxlcyAoVk1zIC5xY293MikgYWNjZXNzIGFu ZCB0byBtb3ZlIFZNcyBhcm91bmQgdGhlIHBvZCAoZWFzdCAvd2VzdCB0cmFmZmljKSB3aGlsZSB1 c2luZyB0aGUgMTBHYiBpbnRlcmZhY2VzIGZvciBnaXZpbmcgc2VydmljZXMgdG8gdGhlIG91dHNp ZGUgd29ybGQgKG5vcnRoL3NvdXRoIHRyYWZmaWMpLjwvc3Bhbj48L3A+CjxwPjxicj4KPHNwYW4+ PC9zcGFuPjwvcD4KPHA+VGhpcyBzYWlkLCBteSBmaXJzdCBxdWVzdGlvbiBpczogSG93IHNob3Vs ZCBJIGRlcGxveSBHbHVzdGVyRlMgaW4gc3VjaCBvVmlydCBzY2VuYXJpbz8gTXkgcXVlc3Rpb25z IGFyZTo8L3A+CjxwPjxicj4KPC9wPgo8cD4xIC0gU2hvdWxkIEkgY3JlYXRlIDMgUkFJRCAoaS5l LjogUkFJRCA1KSwgb25lIG9uIGVhY2ggb1ZpcnQgbm9kZSwgYW5kIHRoZW4gY3JlYXRlIGEgR2x1 c3RlckZTIHVzaW5nIHRoZW0/PC9wPgo8cD4yIC0gSW5zdGVhZCwgc2hvdWxkIEkgY3JlYXRlIGEg SkJPRCBhcnJheSBtYWRlIG9mIGFsbCBzZXJ2ZXImIzM5O3MgZGlza3M/PC9wPgo8cD4zIC0gV2hh dCBpcyB0aGUgYmVzdCBHbHVzdGVyIGNvbmZpZ3VyYXRpb24gdG8gcHJvdmlkZSBmb3IgSEEgd2hp bGUgbm90IGNvbnN1bWluZyB0b28gbXVjaCBkaXNrIHNwYWNlPzxicj4KPC9wPgo8cD40IC0gRG9l cyBhIG9WaXJ0IGh5cGVydmlzb3IgcG9kIGxpa2UgSSBhbSBwbGFubmluZyB0byBidWlsZCwgYW5k IHRoZSB2aXJ0dWFsaXphdGlvbiBlbnZpcm9ubWVudCwgYmVuZWZpdHMgZnJvbSB0aWVyaW5nIHdo ZW4gdXNpbmcgYSBTU0QgZGlzaz8gQW5kIHllcywgd2lsbCBHbHVzdGVyIGRvIGl0IGJ5IGRlZmF1 bHQgb3IgSSBoYXZlIHRvIGNvbmZpZ3VyZSBpdCB0byBkbyBzbz88L3A+CjxwPjxicj4KPC9wPgo8 cD5BdCB0aGUgYm90dG9tIGxpbmUsIHdoYXQgaXMgdGhlIGdvb2QgcHJhY3RpY2UgZm9yIHVzaW5n IEdsdXN0ZXJGUyBpbiBzbWFsbCBwb2RzIGZvciBlbnRlcnByaXNlcz88YnI+CjwvcD4KPHA+PGJy Pgo8L3A+CjxwPllvdSBvcGluaW9uL2ZlZWRiYWNrIHdpbGwgYmUgcmVhbGx5IGFwcHJlY2lhdGVk ITwvcD4KPHNwYW4gY2xhc3M9ImdtYWlsLW1fLTg4ODc4Mzg2NTgyMTU5ODIwMzNtXzQ4MzE4ODY4 ODMzMDU2NzI3MThIT0VuWmIiPjxmb250IGNvbG9yPSIjODg4ODg4Ij4KPHA+TW9hY2lyPGJyPgo8 L3A+CjwvZm9udD48L3NwYW4+PC9kaXY+CjwvZGl2Pgo8YnI+Cl9fX19fX19fX19fX19fX19fX19f X19fX19fX19fXzx3YnI+X19fX19fX19fX19fX19fX188YnI+ClVzZXJzIG1haWxpbmcgbGlzdDxi cj4KPGEgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgdGFyZ2V0PSJfYmxhbmsiPlVzZXJz QG92aXJ0Lm9yZzwvYT48YnI+CjxhIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1h bi9saXN0aW5mby91c2VycyIgcmVsPSJub3JlZmVycmVyIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDov L2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuPHdicj4vbGlzdGluZm8vdXNlcnM8L2E+PGJyPgo8YnI+ CjwvYmxvY2txdW90ZT4KPC9kaXY+Cjxicj4KPC9kaXY+Cjwvc3Bhbj48L3NwYW4+PC9kaXY+Cjwv ZGl2Pgo8L2Rpdj4KPC9kaXY+Cgo8L2Jsb2NrcXVvdGU+PC9kaXY+PGJyPjwvZGl2Pgo8YnI+X19f X19fX19fX19fX19fX19fX19fX19fX19fX19fPHdicj5fX19fX19fX19fX19fX19fXzxicj4KVXNl cnMgbWFpbGluZyBsaXN0PGJyPgo8YSBocmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQub3JnIj5Vc2Vy c0BvdmlydC5vcmc8L2E+PGJyPgo8YSBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxt YW4vbGlzdGluZm8vdXNlcnMiIHJlbD0ibm9yZWZlcnJlciIgdGFyZ2V0PSJfYmxhbmsiPmh0dHA6 Ly9saXN0cy5vdmlydC5vcmcvPHdicj5tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPjxicj4KPGJy PjwvYmxvY2txdW90ZT48L2Rpdj48YnI+PC9kaXY+PC9kaXY+Cg== --===============0996246034367816116==-- From moacirferreira at hotmail.com Mon Aug 7 19:29:00 2017 Content-Type: multipart/mixed; boundary="===============1366104238581514415==" MIME-Version: 1.0 From: Moacir Ferreira To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Mon, 07 Aug 2017 09:56:31 +0000 Message-ID: In-Reply-To: CANvHAxRkxS0izgHBfvgtTio5gEiv2DWk7A7xwsSs-frFrCHd+w@mail.gmail.com --===============1366104238581514415== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --_000_VI1P190MB02858CCC4D7DCD6A86090CC9C8B50VI1P190MB0285EURP_ Content-Type: text/plain; charset=3D"iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi Colin, I am in Portugal, so sorry for this late response. It is quite confusing fo= =3D r me, please consider: 1 - What if the RAID is done by the server's disk controller, not by softwa= =3D re? 2 - For JBOD I am just using gdeploy to deploy it. However, I am not using = =3D the oVirt node GUI to do this. 3 - As the VM .qcow2 files are quite big, tiering would only help if made b= =3D y an intelligent system that uses SSD for chunks of data not for the entire= =3D .qcow2 file. But I guess this is a problem everybody else has. So, Do you = =3D know how tiering works in Gluster? 4 - I am putting the OS on the first disk. However, would you do differentl= =3D y? Moacir ________________________________ From: Colin Coe Sent: Monday, August 7, 2017 4:48 AM To: Moacir Ferreira Cc: users(a)ovirt.org Subject: Re: [ovirt-users] Good practices 1) RAID5 may be a performance hit- 2) I'd be inclined to do this as JBOD by creating a distributed disperse vo= =3D lume on each server. Something like echo gluster volume create dispersevol disperse-data 5 redundancy 2 \ $(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e "server${SERV= =3D ER}:/brick/brick-${SERVER}${BRICK}/brick \c"; done; done) 3) I think the above. 4) Gluster does support tiering, but IIRC you'd need the same number of SSD= =3D as spindle drives. There may be another way to use the SSD as a fast cach= =3D e. Where are you putting the OS? Hope I understood the question... Thanks On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira > wrote: I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU = =3D sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use Gluste= =3D rFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dua= =3D l 10Gb NIC. So my intention is to create a loop like a server triangle usin= =3D g the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VM= =3D s around the pod (east /west traffic) while using the 10Gb interfaces for g= =3D iving services to the outside world (north/south traffic). This said, my first question is: How should I deploy GlusterFS in such oVir= =3D t scenario? My questions are: 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then= =3D create a GlusterFS using them? 2 - Instead, should I create a JBOD array made of all server's disks? 3 - What is the best Gluster configuration to provide for HA while not cons= =3D uming too much disk space? 4 - Does a oVirt hypervisor pod like I am planning to build, and the virtua= =3D lization environment, benefits from tiering when using a SSD disk? And yes,= =3D will Gluster do it by default or I have to configure it to do so? At the bottom line, what is the good practice for using GlusterFS in small = =3D pods for enterprises? You opinion/feedback will be really appreciated! Moacir _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users --_000_VI1P190MB02858CCC4D7DCD6A86090CC9C8B50VI1P190MB0285EURP_ Content-Type: text/html; charset=3D"iso-8859-1" Content-Transfer-Encoding: quoted-printable

Hi Colin,

I am in Portugal, so sorry for this late response. It is qu= =3D ite confusing for me, please consider:

1 - What if the RAID is done by the server's disk controller, no= =3D t by software?

2 - For JBOD I am just using gdeploy to deploy it. However, I am = =3D not using the oVirt node GUI to do this.

3 - As the VM .qcow2 files are quite big, tiering would only= =3D help if made by an intelligent system that uses SSD for chunks of data not= =3D for the entire .qcow2 file. But I guess this is a problem everybody else h= =3D as. So, Do you know how tiering works in Gluster?

4 - I am putting the OS on the first disk. However, would you do di= =3D fferently?

Moacir

From: Colin Coe <c= olin=3D .coe(a)gmail.com>
Sent: Monday, August 7, 2017 4:48 AM
To: Moacir Ferreira
Cc: users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices

1) RAID5 may be a performance hit-

2) I'd be inclined to do this as JBOD by creating a distributed disper= =3D se volume on each server. Something like

echo gluster volume create dispersevol disperse-data 5 redundancy 2 \<= =3D /div>
$(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e "se= =3D rver${SERVER}:/brick/brick-${SERVER}${BRICK}/brick \c"; done; done)

3) I think the above.

4) Gluster does support tiering, but IIRC you'd need the same number o= =3D f SSD as spindle drives. There may be another way to use the SSD as a= =3D fast cache.

Where are you putting the OS?

Hope I understood the question...

Thanks

On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferrei= ra=3D <m= oacirf=3D erreira(a)hotmail.com> wrote:

I am willing to assemble a oVirt "pod", made of 3 server= =3D s, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The id= =3D ea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual= =3D 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop like a server triangle using the 40Gb NICs for virtual= =3D ization files (VMs .qcow2) access and to move VMs around the pod (east /wes= =3D t traffic) while using the 10Gb interfaces for giving services to the outsi= =3D de world (north/south traffic).

This said, my first question is: How should I deploy GlusterFS in such o= =3D Virt scenario? My questions are:

1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and t= =3D hen create a GlusterFS using them?

2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not c= =3D onsuming too much disk space?

4 - Does a oVirt hypervisor pod like I am planning to build, and the vir= =3D tualization environment, benefits from tiering when using a SSD disk? And y= =3D es, will Gluster do it by default or I have to configure it to do so?

At the bottom line, what is the good practice for using GlusterFS in sma= =3D ll pods for enterprises?

You opinion/feedback will be really appreciated!

Moacir

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users<=3D br>

--_000_VI1P190MB02858CCC4D7DCD6A86090CC9C8B50VI1P190MB0285EURP_-- --===============1366104238581514415== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" LS1fMDAwX1ZJMVAxOTBNQjAyODU4Q0NDNEQ3RENENkE4NjA5MENDOUM4QjUwVkkxUDE5ME1CMDI4 NUVVUlBfCkNvbnRlbnQtVHlwZTogdGV4dC9wbGFpbjsgY2hhcnNldD0iaXNvLTg4NTktMSIKQ29u dGVudC1UcmFuc2Zlci1FbmNvZGluZzogcXVvdGVkLXByaW50YWJsZQoKSGkgQ29saW4sCgoKSSBh bSBpbiBQb3J0dWdhbCwgc28gc29ycnkgZm9yIHRoaXMgbGF0ZSByZXNwb25zZS4gSXQgaXMgcXVp dGUgY29uZnVzaW5nIGZvPQpyIG1lLCBwbGVhc2UgY29uc2lkZXI6CgoxIC0gV2hhdCBpZiB0aGUg UkFJRCBpcyBkb25lIGJ5IHRoZSBzZXJ2ZXIncyBkaXNrIGNvbnRyb2xsZXIsIG5vdCBieSBzb2Z0 d2E9CnJlPwoKCjIgLSBGb3IgSkJPRCBJIGFtIGp1c3QgdXNpbmcgZ2RlcGxveSB0byBkZXBsb3kg aXQuIEhvd2V2ZXIsIEkgYW0gbm90IHVzaW5nID0KdGhlIG9WaXJ0IG5vZGUgR1VJIHRvIGRvIHRo aXMuCgoKMyAtIEFzIHRoZSBWTSAucWNvdzIgZmlsZXMgYXJlIHF1aXRlIGJpZywgdGllcmluZyB3 b3VsZCBvbmx5IGhlbHAgaWYgbWFkZSBiPQp5IGFuIGludGVsbGlnZW50IHN5c3RlbSB0aGF0IHVz ZXMgU1NEIGZvciBjaHVua3Mgb2YgZGF0YSBub3QgZm9yIHRoZSBlbnRpcmU9CiAucWNvdzIgZmls ZS4gQnV0IEkgZ3Vlc3MgdGhpcyBpcyBhIHByb2JsZW0gZXZlcnlib2R5IGVsc2UgaGFzLiBTbywg RG8geW91ID0Ka25vdyBob3cgdGllcmluZyB3b3JrcyBpbiBHbHVzdGVyPwoKCjQgLSBJIGFtIHB1 dHRpbmcgdGhlIE9TIG9uIHRoZSBmaXJzdCBkaXNrLiBIb3dldmVyLCB3b3VsZCB5b3UgZG8gZGlm ZmVyZW50bD0KeT8KCgpNb2FjaXIKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCkZy b206IENvbGluIENvZSA8Y29saW4uY29lQGdtYWlsLmNvbT4KU2VudDogTW9uZGF5LCBBdWd1c3Qg NywgMjAxNyA0OjQ4IEFNClRvOiBNb2FjaXIgRmVycmVpcmEKQ2M6IHVzZXJzQG92aXJ0Lm9yZwpT dWJqZWN0OiBSZTogW292aXJ0LXVzZXJzXSBHb29kIHByYWN0aWNlcwoKMSkgUkFJRDUgbWF5IGJl IGEgcGVyZm9ybWFuY2UgaGl0LQoKMikgSSdkIGJlIGluY2xpbmVkIHRvIGRvIHRoaXMgYXMgSkJP RCBieSBjcmVhdGluZyBhIGRpc3RyaWJ1dGVkIGRpc3BlcnNlIHZvPQpsdW1lIG9uIGVhY2ggc2Vy dmVyLiAgU29tZXRoaW5nIGxpa2UKCmVjaG8gZ2x1c3RlciB2b2x1bWUgY3JlYXRlIGRpc3BlcnNl dm9sIGRpc3BlcnNlLWRhdGEgNSByZWR1bmRhbmN5IDIgXAokKGZvciBTRVJWRVIgaW4gYSBiIGM7 IGRvIGZvciBCUklDSyBpbiAkKHNlcSAxIDUpOyBkbyBlY2hvIC1lICJzZXJ2ZXIke1NFUlY9CkVS fTovYnJpY2svYnJpY2stJHtTRVJWRVJ9JHtCUklDS30vYnJpY2sgXGMiOyBkb25lOyBkb25lKQoK MykgSSB0aGluayB0aGUgYWJvdmUuCgo0KSBHbHVzdGVyIGRvZXMgc3VwcG9ydCB0aWVyaW5nLCBi dXQgSUlSQyB5b3UnZCBuZWVkIHRoZSBzYW1lIG51bWJlciBvZiBTU0Q9CiBhcyBzcGluZGxlIGRy aXZlcy4gIFRoZXJlIG1heSBiZSBhbm90aGVyIHdheSB0byB1c2UgdGhlIFNTRCBhcyBhIGZhc3Qg Y2FjaD0KZS4KCldoZXJlIGFyZSB5b3UgcHV0dGluZyB0aGUgT1M/CgpIb3BlIEkgdW5kZXJzdG9v ZCB0aGUgcXVlc3Rpb24uLi4KClRoYW5rcwoKT24gU3VuLCBBdWcgNiwgMjAxNyBhdCAxMDo0OSBQ TSwgTW9hY2lyIEZlcnJlaXJhIDxtb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvPQptPG1haWx0bzpt b2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbT4+IHdyb3RlOgoKSSBhbSB3aWxsaW5nIHRvIGFzc2Vt YmxlIGEgb1ZpcnQgInBvZCIsIG1hZGUgb2YgMyBzZXJ2ZXJzLCBlYWNoIHdpdGggMiBDUFUgPQpz b2NrZXRzIG9mIDEyIGNvcmVzLCAyNTZHQiBSQU0sIDcgSEREIDEwSywgMSBTU0QuIFRoZSBpZGVh IGlzIHRvIHVzZSBHbHVzdGU9CnJGUyB0byBwcm92aWRlIEhBIGZvciB0aGUgVk1zLiBUaGUgMyBz ZXJ2ZXJzIGhhdmUgYSBkdWFsIDQwR2IgTklDIGFuZCBhIGR1YT0KbCAxMEdiIE5JQy4gU28gbXkg aW50ZW50aW9uIGlzIHRvIGNyZWF0ZSBhIGxvb3AgbGlrZSBhIHNlcnZlciB0cmlhbmdsZSB1c2lu PQpnIHRoZSA0MEdiIE5JQ3MgZm9yIHZpcnR1YWxpemF0aW9uIGZpbGVzIChWTXMgLnFjb3cyKSBh Y2Nlc3MgYW5kIHRvIG1vdmUgVk09CnMgYXJvdW5kIHRoZSBwb2QgKGVhc3QgL3dlc3QgdHJhZmZp Yykgd2hpbGUgdXNpbmcgdGhlIDEwR2IgaW50ZXJmYWNlcyBmb3IgZz0KaXZpbmcgc2VydmljZXMg dG8gdGhlIG91dHNpZGUgd29ybGQgKG5vcnRoL3NvdXRoIHRyYWZmaWMpLgoKClRoaXMgc2FpZCwg bXkgZmlyc3QgcXVlc3Rpb24gaXM6IEhvdyBzaG91bGQgSSBkZXBsb3kgR2x1c3RlckZTIGluIHN1 Y2ggb1Zpcj0KdCBzY2VuYXJpbz8gTXkgcXVlc3Rpb25zIGFyZToKCgoxIC0gU2hvdWxkIEkgY3Jl YXRlIDMgUkFJRCAoaS5lLjogUkFJRCA1KSwgb25lIG9uIGVhY2ggb1ZpcnQgbm9kZSwgYW5kIHRo ZW49CiBjcmVhdGUgYSBHbHVzdGVyRlMgdXNpbmcgdGhlbT8KCjIgLSBJbnN0ZWFkLCBzaG91bGQg SSBjcmVhdGUgYSBKQk9EIGFycmF5IG1hZGUgb2YgYWxsIHNlcnZlcidzIGRpc2tzPwoKMyAtIFdo YXQgaXMgdGhlIGJlc3QgR2x1c3RlciBjb25maWd1cmF0aW9uIHRvIHByb3ZpZGUgZm9yIEhBIHdo aWxlIG5vdCBjb25zPQp1bWluZyB0b28gbXVjaCBkaXNrIHNwYWNlPwoKNCAtIERvZXMgYSBvVmly dCBoeXBlcnZpc29yIHBvZCBsaWtlIEkgYW0gcGxhbm5pbmcgdG8gYnVpbGQsIGFuZCB0aGUgdmly dHVhPQpsaXphdGlvbiBlbnZpcm9ubWVudCwgYmVuZWZpdHMgZnJvbSB0aWVyaW5nIHdoZW4gdXNp bmcgYSBTU0QgZGlzaz8gQW5kIHllcyw9CiB3aWxsIEdsdXN0ZXIgZG8gaXQgYnkgZGVmYXVsdCBv ciBJIGhhdmUgdG8gY29uZmlndXJlIGl0IHRvIGRvIHNvPwoKCkF0IHRoZSBib3R0b20gbGluZSwg d2hhdCBpcyB0aGUgZ29vZCBwcmFjdGljZSBmb3IgdXNpbmcgR2x1c3RlckZTIGluIHNtYWxsID0K cG9kcyBmb3IgZW50ZXJwcmlzZXM/CgoKWW91IG9waW5pb24vZmVlZGJhY2sgd2lsbCBiZSByZWFs bHkgYXBwcmVjaWF0ZWQhCgpNb2FjaXIKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fClVzZXJzIG1haWxpbmcgbGlzdApVc2Vyc0BvdmlydC5vcmc8bWFpbHRv OlVzZXJzQG92aXJ0Lm9yZz4KaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZv L3VzZXJzCgoKCi0tXzAwMF9WSTFQMTkwTUIwMjg1OENDQzREN0RDRDZBODYwOTBDQzlDOEI1MFZJ MVAxOTBNQjAyODVFVVJQXwpDb250ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD0iaXNvLTg4 NTktMSIKQ29udGVudC1UcmFuc2Zlci1FbmNvZGluZzogcXVvdGVkLXByaW50YWJsZQoKPGh0bWw+ CjxoZWFkPgo8bWV0YSBodHRwLWVxdWl2PTNEIkNvbnRlbnQtVHlwZSIgY29udGVudD0zRCJ0ZXh0 L2h0bWw7IGNoYXJzZXQ9M0Rpc28tODg1OS09CjEiPgo8c3R5bGUgdHlwZT0zRCJ0ZXh0L2NzcyIg c3R5bGU9M0QiZGlzcGxheTpub25lOyI+PCEtLSBQIHttYXJnaW4tdG9wOjA7bWFyZ2k9Cm4tYm90 dG9tOjA7fSAtLT48L3N0eWxlPgo8L2hlYWQ+Cjxib2R5IGRpcj0zRCJsdHIiPgo8ZGl2IGlkPTNE ImRpdnRhZ2RlZmF1bHR3cmFwcGVyIiBzdHlsZT0zRCJmb250LXNpemU6MTJwdDtjb2xvcjojMDAw MDAwO2ZvbnQ9Ci1mYW1pbHk6Q2FsaWJyaSxIZWx2ZXRpY2Esc2Fucy1zZXJpZjsiIGRpcj0zRCJs dHIiPgo8cD5IaSBDb2xpbiw8L3A+CjxwPjxicj4KPC9wPgo8cD48c3Bhbj5JIGFtIGluIFBvcnR1 Z2FsPC9zcGFuPiwgc28gc29ycnkgZm9yIHRoaXMgbGF0ZSByZXNwb25zZS4gSXQgaXMgcXU9Cml0 ZSBjb25mdXNpbmcgZm9yIG1lLCBwbGVhc2UgY29uc2lkZXI6PC9wPgo8cD48L3A+CjxkaXY+PGI+ PGJyPgo8L2I+MTxiPiAtIDwvYj5XaGF0IGlmIHRoZSBSQUlEIGlzIGRvbmUgYnkgdGhlIHNlcnZl cidzIGRpc2sgY29udHJvbGxlciwgbm89CnQgYnkgc29mdHdhcmU/PC9kaXY+Cjxicj4KPHA+PC9w Pgo8cD4yIC08Yj4gPC9iPkZvciBKQk9EIEkgYW0ganVzdCB1c2luZyBnZGVwbG95IHRvIGRlcGxv eSBpdC4gSG93ZXZlciwgSSBhbSA9Cm5vdCB1c2luZyB0aGUgb1ZpcnQgbm9kZSBHVUkgdG8gZG8g dGhpcy48L3A+CjxwPjxicj4KPC9wPgo8cD4zIC08Yj4gPC9iPkFzIHRoZSBWTSAucWNvdzImbmJz cDtmaWxlcyBhcmUgcXVpdGUgYmlnLCB0aWVyaW5nIHdvdWxkIG9ubHk9CiBoZWxwIGlmIG1hZGUg YnkgYW4gaW50ZWxsaWdlbnQgc3lzdGVtIHRoYXQgdXNlcyBTU0QgZm9yIGNodW5rcyBvZiBkYXRh IG5vdD0KIGZvciB0aGUgZW50aXJlIC5xY293MiBmaWxlLiBCdXQgSSBndWVzcyB0aGlzIGlzIGEg cHJvYmxlbSBldmVyeWJvZHkgZWxzZSBoPQphcy4gU28sIERvIHlvdSBrbm93IGhvdyB0aWVyaW5n IHdvcmtzCiBpbiBHbHVzdGVyPzxicj4KPC9wPgo8cD48YnI+CjwvcD4KPHA+NCAtJm5ic3A7SSBh bSBwdXR0aW5nIHRoZSBPUyBvbiB0aGUgZmlyc3QgZGlzay4gSG93ZXZlciwgd291bGQgeW91IGRv IGRpPQpmZmVyZW50bHk/PGJyPgo8L3A+CjxwPjxicj4KPC9wPgpNb2FjaXI8YnI+Cjxicj4KPGRp diBzdHlsZT0zRCJjb2xvcjogcmdiKDQ5LCA1NSwgNTcpOyI+CjxociB0YWJpbmRleD0zRCItMSIg c3R5bGU9M0QiZGlzcGxheTppbmxpbmUtYmxvY2s7IHdpZHRoOjk4JSI+CjxkaXYgaWQ9M0QiZGl2 UnBseUZ3ZE1zZyIgZGlyPTNEImx0ciI+PGZvbnQgc3R5bGU9M0QiZm9udC1zaXplOjExcHQiIGZh Y2U9Cj0zRCJDYWxpYnJpLCBzYW5zLXNlcmlmIiBjb2xvcj0zRCIjMDAwMDAwIj48Yj5Gcm9tOjwv Yj4gQ29saW4gQ29lICZsdDtjb2xpbj0KLmNvZUBnbWFpbC5jb20mZ3Q7PGJyPgo8Yj5TZW50Ojwv Yj4gTW9uZGF5LCBBdWd1c3QgNywgMjAxNyA0OjQ4IEFNPGJyPgo8Yj5Ubzo8L2I+IE1vYWNpciBG ZXJyZWlyYTxicj4KPGI+Q2M6PC9iPiB1c2Vyc0BvdmlydC5vcmc8YnI+CjxiPlN1YmplY3Q6PC9i PiBSZTogW292aXJ0LXVzZXJzXSBHb29kIHByYWN0aWNlczwvZm9udD4KPGRpdj4mbmJzcDs8L2Rp dj4KPC9kaXY+CjxkaXY+CjxkaXYgZGlyPTNEImx0ciI+MSkgUkFJRDUgbWF5IGJlIGEgcGVyZm9y bWFuY2UgaGl0LSA8YnI+CjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj4yKSBJJ2QgYmUgaW5jbGluZWQg dG8gZG8gdGhpcyBhcyBKQk9EIGJ5IGNyZWF0aW5nIGEgZGlzdHJpYnV0ZWQgZGlzcGVyPQpzZSB2 b2x1bWUgb24gZWFjaCBzZXJ2ZXIuJm5ic3A7IFNvbWV0aGluZyBsaWtlCjxkaXY+PGJyPgo8L2Rp dj4KPGRpdj5lY2hvIGdsdXN0ZXIgdm9sdW1lIGNyZWF0ZSBkaXNwZXJzZXZvbCBkaXNwZXJzZS1k YXRhIDUgcmVkdW5kYW5jeSAyIFw8PQovZGl2Pgo8ZGl2PiQoZm9yIFNFUlZFUiBpbiBhIGIgYzsg ZG8gZm9yIEJSSUNLIGluICQoc2VxIDEgNSk7IGRvIGVjaG8gLWUgJnF1b3Q7c2U9CnJ2ZXIke1NF UlZFUn06L2JyaWNrL2JyaWNrLSR7U0VSVkVSfSR7QlJJQ0t9L2JyaWNrIFxjJnF1b3Q7OyBkb25l OyBkb25lKTwvZD0KaXY+CjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj4zKSBJIHRoaW5rIHRoZSBhYm92 ZS4gPGI+PC9iPjwvZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+NCkgR2x1c3RlciBkb2VzIHN1 cHBvcnQgdGllcmluZywgYnV0IElJUkMgeW91J2QgbmVlZCB0aGUgc2FtZSBudW1iZXIgbz0KZiBT U0QgYXMgc3BpbmRsZSBkcml2ZXMuJm5ic3A7IFRoZXJlIG1heSBiZSBhbm90aGVyIHdheSB0byB1 c2UgdGhlIFNTRCBhcyBhPQogZmFzdCBjYWNoZS4mbmJzcDs8L2Rpdj4KPGRpdj48YnI+CjwvZGl2 Pgo8ZGl2PldoZXJlIGFyZSB5b3UgcHV0dGluZyB0aGUgT1M/PC9kaXY+CjxkaXY+PGJyPgo8L2Rp dj4KPGRpdj5Ib3BlIEkgdW5kZXJzdG9vZCB0aGUgcXVlc3Rpb24uLi48L2Rpdj4KPGRpdj48YnI+ CjwvZGl2Pgo8ZGl2PlRoYW5rczwvZGl2Pgo8L2Rpdj4KPC9kaXY+CjxkaXYgY2xhc3M9M0QiZ21h aWxfZXh0cmEiPjxicj4KPGRpdiBjbGFzcz0zRCJnbWFpbF9xdW90ZSI+T24gU3VuLCBBdWcgNiwg MjAxNyBhdCAxMDo0OSBQTSwgTW9hY2lyIEZlcnJlaXJhPQogPHNwYW4gZGlyPTNEImx0ciI+CiZs dDs8YSBocmVmPTNEIm1haWx0bzptb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbSIgdGFyZ2V0PTNE Il9ibGFuayI+bW9hY2lyZj0KZXJyZWlyYUBob3RtYWlsLmNvbTwvYT4mZ3Q7PC9zcGFuPiB3cm90 ZTo8YnI+CjxibG9ja3F1b3RlIGNsYXNzPTNEImdtYWlsX3F1b3RlIiBzdHlsZT0zRCJtYXJnaW46 MCAwIDAgLjhleDsgYm9yZGVyLWxlZnQ6MT0KcHggI2NjYyBzb2xpZDsgcGFkZGluZy1sZWZ0OjFl eCI+CjxkaXYgZGlyPTNEImx0ciI+CjxkaXYgaWQ9M0QibV8yNDYwOTg1NjkxNzQ2NDk4MzIyZGl2 dGFnZGVmYXVsdHdyYXBwZXIiIGRpcj0zRCJsdHIiIHN0eWxlPTNEIj0KZm9udC1zaXplOjEycHQ7 IGNvbG9yOiMwMDAwMDA7IGZvbnQtZmFtaWx5OkNhbGlicmksSGVsdmV0aWNhLHNhbnMtc2VyaWYi Pgo8cD48c3Bhbj5JIGFtIHdpbGxpbmcgdG8gYXNzZW1ibGUgYSBvVmlydCAmcXVvdDtwb2QmcXVv dDssIG1hZGUgb2YgMyBzZXJ2ZXI9CnMsIGVhY2ggd2l0aCAyIENQVSBzb2NrZXRzIG9mIDEyIGNv cmVzLCAyNTZHQiBSQU0sIDcgSEREIDEwSywgMSBTU0QuIFRoZSBpZD0KZWEgaXMgdG8gdXNlIEds dXN0ZXJGUyB0byBwcm92aWRlIEhBIGZvciB0aGUgVk1zLiBUaGUgMyBzZXJ2ZXJzIGhhdmUgYSBk dWFsPQogNDBHYiBOSUMgYW5kIGEgZHVhbCAxMEdiIE5JQy4gU28gbXkgaW50ZW50aW9uCiBpcyB0 byBjcmVhdGUgYSBsb29wIGxpa2UgYSBzZXJ2ZXIgdHJpYW5nbGUgdXNpbmcgdGhlIDQwR2IgTklD cyBmb3IgdmlydHVhbD0KaXphdGlvbiBmaWxlcyAoVk1zIC5xY293MikgYWNjZXNzIGFuZCB0byBt b3ZlIFZNcyBhcm91bmQgdGhlIHBvZCAoZWFzdCAvd2VzPQp0IHRyYWZmaWMpIHdoaWxlIHVzaW5n IHRoZSAxMEdiIGludGVyZmFjZXMgZm9yIGdpdmluZyBzZXJ2aWNlcyB0byB0aGUgb3V0c2k9CmRl IHdvcmxkIChub3J0aC9zb3V0aCB0cmFmZmljKS48L3NwYW4+PC9wPgo8cD48YnI+CjxzcGFuPjwv c3Bhbj48L3A+CjxwPlRoaXMgc2FpZCwgbXkgZmlyc3QgcXVlc3Rpb24gaXM6IEhvdyBzaG91bGQg SSBkZXBsb3kgR2x1c3RlckZTIGluIHN1Y2ggbz0KVmlydCBzY2VuYXJpbz8gTXkgcXVlc3Rpb25z IGFyZTo8L3A+CjxwPjxicj4KPC9wPgo8cD4xIC0gU2hvdWxkIEkgY3JlYXRlIDMgUkFJRCAoaS5l LjogUkFJRCA1KSwgb25lIG9uIGVhY2ggb1ZpcnQgbm9kZSwgYW5kIHQ9CmhlbiBjcmVhdGUgYSBH bHVzdGVyRlMgdXNpbmcgdGhlbT88L3A+CjxwPjIgLSBJbnN0ZWFkLCBzaG91bGQgSSBjcmVhdGUg YSBKQk9EIGFycmF5IG1hZGUgb2YgYWxsIHNlcnZlcidzIGRpc2tzPzwvcD0KPgo8cD4zIC0gV2hh dCBpcyB0aGUgYmVzdCBHbHVzdGVyIGNvbmZpZ3VyYXRpb24gdG8gcHJvdmlkZSBmb3IgSEEgd2hp bGUgbm90IGM9Cm9uc3VtaW5nIHRvbyBtdWNoIGRpc2sgc3BhY2U/PGJyPgo8L3A+CjxwPjQgLSBE b2VzIGEgb1ZpcnQgaHlwZXJ2aXNvciBwb2QgbGlrZSBJIGFtIHBsYW5uaW5nIHRvIGJ1aWxkLCBh bmQgdGhlIHZpcj0KdHVhbGl6YXRpb24gZW52aXJvbm1lbnQsIGJlbmVmaXRzIGZyb20gdGllcmlu ZyB3aGVuIHVzaW5nIGEgU1NEIGRpc2s/IEFuZCB5PQplcywgd2lsbCBHbHVzdGVyIGRvIGl0IGJ5 IGRlZmF1bHQgb3IgSSBoYXZlIHRvIGNvbmZpZ3VyZSBpdCB0byBkbyBzbz88L3A+CjxwPjxicj4K PC9wPgo8cD5BdCB0aGUgYm90dG9tIGxpbmUsIHdoYXQgaXMgdGhlIGdvb2QgcHJhY3RpY2UgZm9y IHVzaW5nIEdsdXN0ZXJGUyBpbiBzbWE9CmxsIHBvZHMgZm9yIGVudGVycHJpc2VzPzxicj4KPC9w Pgo8cD48YnI+CjwvcD4KPHA+WW91IG9waW5pb24vZmVlZGJhY2sgd2lsbCBiZSByZWFsbHkgYXBw cmVjaWF0ZWQhPC9wPgo8c3BhbiBjbGFzcz0zRCJIT0VuWmIiPjxmb250IGNvbG9yPTNEIiM4ODg4 ODgiPgo8cD5Nb2FjaXI8YnI+CjwvcD4KPC9mb250Pjwvc3Bhbj48L2Rpdj4KPC9kaXY+Cjxicj4K X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fPHdicj5fX19fX19fX19fX19fX19fXzxicj4K VXNlcnMgbWFpbGluZyBsaXN0PGJyPgo8YSBocmVmPTNEIm1haWx0bzpVc2Vyc0BvdmlydC5vcmci PlVzZXJzQG92aXJ0Lm9yZzwvYT48YnI+CjxhIGhyZWY9M0QiaHR0cDovL2xpc3RzLm92aXJ0Lm9y Zy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzIiByZWw9M0Qibm9yZWZlcnJlcj0KIiB0YXJnZXQ9M0Qi X2JsYW5rIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnLzx3YnI+bWFpbG1hbi9saXN0aW5mby91c2Vy czwvYT48PQpicj4KPGJyPgo8L2Jsb2NrcXVvdGU+CjwvZGl2Pgo8YnI+CjwvZGl2Pgo8L2Rpdj4K PC9kaXY+CjwvZGl2Pgo8L2JvZHk+CjwvaHRtbD4KCi0tXzAwMF9WSTFQMTkwTUIwMjg1OENDQzRE N0RDRDZBODYwOTBDQzlDOEI1MFZJMVAxOTBNQjAyODVFVVJQXy0tCg== --===============1366104238581514415==-- From fabrice.bacchella at orange.fr Mon Aug 7 19:49:20 2017 Content-Type: multipart/mixed; boundary="===============2997415206850314559==" MIME-Version: 1.0 From: Fabrice Bacchella To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Mon, 07 Aug 2017 21:49:15 +0200 Message-ID: <034E384B-4B32-4FBC-A760-404EB38EBC86@orange.fr> In-Reply-To: c7a1c2e1-57c3-9fa5-0710-ebee3f3fa069@upx.com --===============2997415206850314559== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --Apple-Mail=3D_912FAEC6-5476-4359-94B5-21FE877C0F94 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=3Dutf-8 > Le 7 ao=3DC3=3DBBt 2017 =3DC3=3DA0 17:41, FERNANDO FREDIANI =3D a =3DC3=3DA9crit : >=3D20 > Yet another downside of having a RAID (specially RAID 5 or 6) is that =3D it reduces considerably the write speeds as each group of disks will end = =3D up having the write speed of a single disk as all other disks of that =3D group have to wait for each other to write as well. >=3D20 That's not true if you have medium to high range hardware raid. For =3D example, HP Smart Array come with a flash cache of about 1 or 2 Gb that =3D hides that from the OS.=3D --Apple-Mail=3D_912FAEC6-5476-4359-94B5-21FE877C0F94 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=3Dutf-8
Le 7 ao=3DC3=3DBBt 2017 =3DC3=3DA0 17:41, FERNANDO FREDIANI &l= t;fernando.frediani(a)upx.com> a =3DC3=3DA9crit :

Yet another downside = =3D of having a RAID (specially RAID 5 or 6) is that it reduces considerably = =3D the write speeds as each group of disks will end up having the write =3D speed of a single disk as all other disks of that group have to wait for = =3D each other to write as well.

That's not true if you have medium to high range =3D hardware raid. For example, HP Smart Array come with a flash cache of =3D about 1 or 2 Gb that hides that from the OS.=3D --Apple-Mail=3D_912FAEC6-5476-4359-94B5-21FE877C0F94-- --===============2997415206850314559== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" Ci0tQXBwbGUtTWFpbD1fOTEyRkFFQzYtNTQ3Ni00MzU5LTk0QjUtMjFGRTg3N0MwRjk0CkNvbnRl bnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKQ29udGVudC1UeXBlOiB0ZXh0 L3BsYWluOwoJY2hhcnNldD11dGYtOAoKCj4gTGUgNyBhbz1DMz1CQnQgMjAxNyA9QzM9QTAgMTc6 NDEsIEZFUk5BTkRPIEZSRURJQU5JID0KPGZlcm5hbmRvLmZyZWRpYW5pQHVweC5jb20+IGEgPUMz PUE5Y3JpdCA6Cj49MjAKCj4gWWV0IGFub3RoZXIgZG93bnNpZGUgb2YgaGF2aW5nIGEgUkFJRCAo c3BlY2lhbGx5IFJBSUQgNSBvciA2KSBpcyB0aGF0ID0KaXQgcmVkdWNlcyBjb25zaWRlcmFibHkg dGhlIHdyaXRlIHNwZWVkcyBhcyBlYWNoIGdyb3VwIG9mIGRpc2tzIHdpbGwgZW5kID0KdXAgaGF2 aW5nIHRoZSB3cml0ZSBzcGVlZCBvZiBhIHNpbmdsZSBkaXNrIGFzIGFsbCBvdGhlciBkaXNrcyBv ZiB0aGF0ID0KZ3JvdXAgaGF2ZSB0byB3YWl0IGZvciBlYWNoIG90aGVyIHRvIHdyaXRlIGFzIHdl bGwuCj49MjAKClRoYXQncyBub3QgdHJ1ZSBpZiB5b3UgaGF2ZSBtZWRpdW0gdG8gaGlnaCByYW5n ZSBoYXJkd2FyZSByYWlkLiBGb3IgPQpleGFtcGxlLCBIUCBTbWFydCBBcnJheSBjb21lIHdpdGgg YSBmbGFzaCBjYWNoZSBvZiBhYm91dCAxIG9yIDIgR2IgdGhhdCA9CmhpZGVzIHRoYXQgZnJvbSB0 aGUgT1MuPQoKLS1BcHBsZS1NYWlsPV85MTJGQUVDNi01NDc2LTQzNTktOTRCNS0yMUZFODc3QzBG OTQKQ29udGVudC1UcmFuc2Zlci1FbmNvZGluZzogcXVvdGVkLXByaW50YWJsZQpDb250ZW50LVR5 cGU6IHRleHQvaHRtbDsKCWNoYXJzZXQ9dXRmLTgKCjxodG1sPjxoZWFkPjxtZXRhIGh0dHAtZXF1 aXY9M0QiQ29udGVudC1UeXBlIiBjb250ZW50PTNEInRleHQvaHRtbCA9CmNoYXJzZXQ9M0R1dGYt OCI+PC9oZWFkPjxib2R5IHN0eWxlPTNEIndvcmQtd3JhcDogYnJlYWstd29yZDsgPQotd2Via2l0 LW5ic3AtbW9kZTogc3BhY2U7IC13ZWJraXQtbGluZS1icmVhazogYWZ0ZXItd2hpdGUtc3BhY2U7 IiA9CmNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PGRpdj48YmxvY2txdW90ZSB0eXBlPTNEImNp dGUiIGNsYXNzPTNEIiI+PGRpdiA9CmNsYXNzPTNEIiI+TGUgNyBhbz1DMz1CQnQgMjAxNyA9QzM9 QTAgMTc6NDEsIEZFUk5BTkRPIEZSRURJQU5JICZsdDs8YSA9CmhyZWY9M0QibWFpbHRvOmZlcm5h bmRvLmZyZWRpYW5pQHVweC5jb20iID0KY2xhc3M9M0QiIj5mZXJuYW5kby5mcmVkaWFuaUB1cHgu Y29tPC9hPiZndDsgYSA9QzM9QTljcml0IDo8L2Rpdj48YnIgPQpjbGFzcz0zRCJBcHBsZS1pbnRl cmNoYW5nZS1uZXdsaW5lIj48ZGl2IGNsYXNzPTNEIiI+PHAgPQpzdHlsZT0zRCJmb250LWZhbWls eTogSGVsdmV0aWNhOyBmb250LXNpemU6IDEycHg7IGZvbnQtc3R5bGU6IG5vcm1hbDsgPQpmb250 LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXdlaWdodDogbm9ybWFsOyBsZXR0ZXItc3BhY2lu Zzogbm9ybWFsOyA9Cm9ycGhhbnM6IGF1dG87IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVu dDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogPQpub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3aWRv d3M6IGF1dG87IHdvcmQtc3BhY2luZzogMHB4OyA9Ci13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6 IDBweDsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDI1NSwgMjU1LCAyNTUpOyIgPQpjbGFzcz0zRCIi PjwvcD48L2Rpdj48L2Jsb2NrcXVvdGU+PGJyIGNsYXNzPTNEIiI+PGJsb2NrcXVvdGUgdHlwZT0z RCJjaXRlIj0KIGNsYXNzPTNEIiI+PGRpdiBjbGFzcz0zRCIiPjxwIHN0eWxlPTNEImZvbnQtZmFt aWx5OiBIZWx2ZXRpY2E7ID0KZm9udC1zaXplOiAxMnB4OyBmb250LXN0eWxlOiBub3JtYWw7IGZv bnQtdmFyaWFudC1jYXBzOiBub3JtYWw7ID0KZm9udC13ZWlnaHQ6IG5vcm1hbDsgbGV0dGVyLXNw YWNpbmc6IG5vcm1hbDsgb3JwaGFuczogYXV0bzsgdGV4dC1hbGlnbjogPQpzdGFydDsgdGV4dC1p bmRlbnQ6IDBweDsgdGV4dC10cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7ID0K d2lkb3dzOiBhdXRvOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0 aDogMHB4OyA9CmJhY2tncm91bmQtY29sb3I6IHJnYigyNTUsIDI1NSwgMjU1KTsiIGNsYXNzPTNE IiI+WWV0IGFub3RoZXIgZG93bnNpZGUgPQpvZiBoYXZpbmcgYSBSQUlEIChzcGVjaWFsbHkgUkFJ RCA1IG9yIDYpIGlzIHRoYXQgaXQgcmVkdWNlcyBjb25zaWRlcmFibHkgPQp0aGUgd3JpdGUgc3Bl ZWRzIGFzIGVhY2ggZ3JvdXAgb2YgZGlza3Mgd2lsbCBlbmQgdXAgaGF2aW5nIHRoZSB3cml0ZSA9 CnNwZWVkIG9mIGEgc2luZ2xlIGRpc2sgYXMgYWxsIG90aGVyIGRpc2tzIG9mIHRoYXQgZ3JvdXAg aGF2ZSB0byB3YWl0IGZvciA9CmVhY2ggb3RoZXIgdG8gd3JpdGUgYXMgd2VsbC48YnIgY2xhc3M9 M0QiIj48L3A+PC9kaXY+PC9ibG9ja3F1b3RlPjxiciA9CmNsYXNzPTNEIiI+PC9kaXY+VGhhdCdz IG5vdCB0cnVlIGlmIHlvdSBoYXZlIG1lZGl1bSB0byBoaWdoIHJhbmdlID0KaGFyZHdhcmUgcmFp ZC4gRm9yIGV4YW1wbGUsIEhQIFNtYXJ0IEFycmF5IGNvbWUgd2l0aCBhIGZsYXNoIGNhY2hlIG9m ID0KYWJvdXQgMSBvciAyIEdiIHRoYXQgaGlkZXMgdGhhdCBmcm9tIHRoZSBPUy48L2JvZHk+PC9o dG1sPj0KCi0tQXBwbGUtTWFpbD1fOTEyRkFFQzYtNTQ3Ni00MzU5LTk0QjUtMjFGRTg3N0MwRjk0 LS0K --===============2997415206850314559==-- From fabrice.bacchella at orange.fr Mon Aug 7 19:50:39 2017 Content-Type: multipart/mixed; boundary="===============3311448484548511065==" MIME-Version: 1.0 From: Fabrice Bacchella To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Mon, 07 Aug 2017 21:50:36 +0200 Message-ID: <4365E3F7-4C77-4FF5-8401-1CDA2F0029EE@orange.fr> In-Reply-To: 8dd168a0-d5ab-bdf4-5c5d-197909afc515@upx.com --===============3311448484548511065== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --Apple-Mail=3D_E33F2A30-713D-4209-8D44-2134E4046305 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=3Dwindows-1252 >> Moacir: Yes! This is another reason to have separate networks for =3D north/south and east/west. In that way I can use the standard MTU on the = =3D 10Gb NICs and jumbo frames on the file/move 40Gb NICs. Why not Jumbo frame every where ?=3D --Apple-Mail=3D_E33F2A30-713D-4209-8D44-2134E4046305 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=3Dwindows-1252
Moacir: Yes! This is another reason = =3D to have separate networks for north/south and east/west. In that way I =3D can use the standard MTU on the 10Gb NICs and jumbo frames on the =3D file/move 40Gb NICs.
<= /=3D blockquote>

Why not Jumbo frame =3D every where ?
=3D --Apple-Mail=3D_E33F2A30-713D-4209-8D44-2134E4046305-- --===============3311448484548511065== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" Ci0tQXBwbGUtTWFpbD1fRTMzRjJBMzAtNzEzRC00MjA5LThENDQtMjEzNEU0MDQ2MzA1CkNvbnRl bnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKQ29udGVudC1UeXBlOiB0ZXh0 L3BsYWluOwoJY2hhcnNldD13aW5kb3dzLTEyNTIKCj4+IE1vYWNpcjogWWVzISBUaGlzIGlzIGFu b3RoZXIgcmVhc29uIHRvIGhhdmUgc2VwYXJhdGUgbmV0d29ya3MgZm9yID0Kbm9ydGgvc291dGgg YW5kIGVhc3Qvd2VzdC4gSW4gdGhhdCB3YXkgSSBjYW4gdXNlIHRoZSBzdGFuZGFyZCBNVFUgb24g dGhlID0KMTBHYiBOSUNzIGFuZCBqdW1ibyBmcmFtZXMgb24gdGhlIGZpbGUvbW92ZSA0MEdiIE5J Q3MuCgpXaHkgbm90IEp1bWJvIGZyYW1lIGV2ZXJ5IHdoZXJlID89CgotLUFwcGxlLU1haWw9X0Uz M0YyQTMwLTcxM0QtNDIwOS04RDQ0LTIxMzRFNDA0NjMwNQpDb250ZW50LVRyYW5zZmVyLUVuY29k aW5nOiBxdW90ZWQtcHJpbnRhYmxlCkNvbnRlbnQtVHlwZTogdGV4dC9odG1sOwoJY2hhcnNldD13 aW5kb3dzLTEyNTIKCjxodG1sPjxoZWFkPjxtZXRhIGh0dHAtZXF1aXY9M0QiQ29udGVudC1UeXBl IiBjb250ZW50PTNEInRleHQvaHRtbCA9CmNoYXJzZXQ9M0R3aW5kb3dzLTEyNTIiPjwvaGVhZD48 Ym9keSBzdHlsZT0zRCJ3b3JkLXdyYXA6IGJyZWFrLXdvcmQ7ID0KLXdlYmtpdC1uYnNwLW1vZGU6 IHNwYWNlOyAtd2Via2l0LWxpbmUtYnJlYWs6IGFmdGVyLXdoaXRlLXNwYWNlOyIgPQpjbGFzcz0z RCIiPjxkaXY+PGJsb2NrcXVvdGUgdHlwZT0zRCJjaXRlIiBjbGFzcz0zRCIiPjxkaXYgPQpjbGFz cz0zRCIiPjxibG9ja3F1b3RlIHR5cGU9M0QiY2l0ZSIgPQpjaXRlPTNEIm1pZDpWSTFQMTkwTUIw Mjg1NkM2Q0E1MTM4NTNFOEVEQTJEQjJDOEI1MEBWSTFQMTkwTUIwMjg1LkVVUlAxOTAuUD0KUk9E Lk9VVExPT0suQ09NIiBzdHlsZT0zRCJmb250LWZhbWlseTogSGVsdmV0aWNhOyBmb250LXNpemU6 IDEycHg7ID0KZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBm b250LXdlaWdodDogbm9ybWFsOyA9CmxldHRlci1zcGFjaW5nOiBub3JtYWw7IG9ycGhhbnM6IGF1 dG87IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogPQowcHg7IHRleHQtdHJhbnNmb3Jt OiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3aWRvd3M6IGF1dG87ID0Kd29yZC1zcGFjaW5n OiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgYmFja2dyb3VuZC1jb2xvcjog PQpyZ2IoMjU1LCAyNTUsIDI1NSk7IiBjbGFzcz0zRCIiPjxkaXYgaWQ9M0QiZGl2dGFnZGVmYXVs dHdyYXBwZXIiID0KZGlyPTNEImx0ciIgc3R5bGU9M0QiZm9udC1zaXplOiAxMnB0OyBmb250LWZh bWlseTogQ2FsaWJyaSwgSGVsdmV0aWNhLCA9CnNhbnMtc2VyaWY7IiBjbGFzcz0zRCIiPjxkaXYg c3R5bGU9M0QiY29sb3I6IHJnYig0OSwgNTUsIDU3KTsiID0KY2xhc3M9M0QiIj48ZGl2IGNsYXNz PTNEIiI+PGRpdiBkaXI9M0QibHRyIiBjbGFzcz0zRCIiPjxkaXYgPQpjbGFzcz0zRCJnbWFpbF9l eHRyYSI+PGRpdiBjbGFzcz0zRCJnbWFpbF9xdW90ZSI+PGRpdiBjbGFzcz0zRCIiPjxiID0KY2xh c3M9M0QiIj5Nb2FjaXI6PC9iPjxzcGFuID0KY2xhc3M9M0QiQXBwbGUtY29udmVydGVkLXNwYWNl Ij4mbmJzcDs8L3NwYW4+WWVzISBUaGlzIGlzIGFub3RoZXIgcmVhc29uID0KdG8gaGF2ZSBzZXBh cmF0ZSBuZXR3b3JrcyBmb3Igbm9ydGgvc291dGggYW5kIGVhc3Qvd2VzdC4gSW4gdGhhdCB3YXkg SSA9CmNhbiB1c2UgdGhlIHN0YW5kYXJkIE1UVSBvbiB0aGUgMTBHYiBOSUNzIGFuZCBqdW1ibyBm cmFtZXMgb24gdGhlID0KZmlsZS9tb3ZlIDQwR2IgTklDcy48YnIgPQpjbGFzcz0zRCIiPjwvZGl2 PjwvZGl2PjwvZGl2PjwvZGl2PjwvZGl2PjwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48L2Rpdj48 Lz0KYmxvY2txdW90ZT48L2Rpdj48YnIgY2xhc3M9M0QiIj48ZGl2IGNsYXNzPTNEIiI+V2h5IG5v dCBKdW1ibyBmcmFtZSA9CmV2ZXJ5IHdoZXJlID88L2Rpdj48L2JvZHk+PC9odG1sPj0KCi0tQXBw bGUtTWFpbD1fRTMzRjJBMzAtNzEzRC00MjA5LThENDQtMjEzNEU0MDQ2MzA1LS0K --===============3311448484548511065==-- From fernando.frediani at upx.com Mon Aug 7 19:52:46 2017 Content-Type: multipart/mixed; boundary="===============8244755700651172869==" MIME-Version: 1.0 From: FERNANDO FREDIANI To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Mon, 07 Aug 2017 16:52:40 -0300 Message-ID: <40d044ae-a41d-082e-131a-bf5fb5503513@upx.com> In-Reply-To: 034E384B-4B32-4FBC-A760-404EB38EBC86@orange.fr --===============8244755700651172869== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------2B4B839116333F84136665DD Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 8bit What you mentioned is a specific case and not a generic situation. The = main point there is that RAID 5 or 6 impacts write performance compared = when you write to only 2 given disks at a time. That was the comparison = made. Fernando On 07/08/2017 16:49, Fabrice Bacchella wrote: > >> Le 7 ao=C3=BBt 2017 =C3=A0 17:41, FERNANDO FREDIANI > > a =C3=A9crit : >> > >> Yet another downside of having a RAID (specially RAID 5 or 6) is that = >> it reduces considerably the write speeds as each group of disks will = >> end up having the write speed of a single disk as all other disks of = >> that group have to wait for each other to write as well. >> > > That's not true if you have medium to high range hardware raid. For = > example, HP Smart Array come with a flash cache of about 1 or 2 Gb = > that hides that from the OS. = --------------2B4B839116333F84136665DD Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit
What you mentioned is a specific case and not a generic situation. The main point there is that RAID 5 or 6 impacts write performance compared when you write to only 2 given disks at a time. That was the comparison made.

Fernando

On 07/08/2017 16:49, Fabrice Bacchella wrote:

Le 7 ao=C3=BBt 2017 =C3=A0 17:41, FERNANDO FREDIA= NI <fernando.frediani(a)upx.com> a =C3=A9crit :

Yet another downside of having a RAID (specially RAID 5 or 6) is that it reduces considerably the write speeds as each group of disks will end up having the write speed of a single disk as all other disks of that group have to wait for each other to write as well.

That's not true if you have medium to high range hardware raid. For example, HP Smart Array come with a flash cache of about 1 or 2 Gb that hides that from the OS.

--------------2B4B839116333F84136665DD-- --===============8244755700651172869== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS0yQjRCODM5MTE2MzMzRjg0MTM2NjY1REQKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDhiaXQK CldoYXQgeW91IG1lbnRpb25lZCBpcyBhIHNwZWNpZmljIGNhc2UgYW5kIG5vdCBhIGdlbmVyaWMg c2l0dWF0aW9uLiBUaGUgCm1haW4gcG9pbnQgdGhlcmUgaXMgdGhhdCBSQUlEIDUgb3IgNiBpbXBh Y3RzIHdyaXRlIHBlcmZvcm1hbmNlIGNvbXBhcmVkIAp3aGVuIHlvdSB3cml0ZSB0byBvbmx5IDIg Z2l2ZW4gZGlza3MgYXQgYSB0aW1lLiBUaGF0IHdhcyB0aGUgY29tcGFyaXNvbiAKbWFkZS4KCkZl cm5hbmRvCgoKT24gMDcvMDgvMjAxNyAxNjo0OSwgRmFicmljZSBCYWNjaGVsbGEgd3JvdGU6Cj4K Pj4gTGUgNyBhb8O7dCAyMDE3IMOgIDE3OjQxLCBGRVJOQU5ETyBGUkVESUFOSSA8ZmVybmFuZG8u ZnJlZGlhbmlAdXB4LmNvbSAKPj4gPG1haWx0bzpmZXJuYW5kby5mcmVkaWFuaUB1cHguY29tPj4g YSDDqWNyaXQgOgo+Pgo+Cj4+IFlldCBhbm90aGVyIGRvd25zaWRlIG9mIGhhdmluZyBhIFJBSUQg KHNwZWNpYWxseSBSQUlEIDUgb3IgNikgaXMgdGhhdCAKPj4gaXQgcmVkdWNlcyBjb25zaWRlcmFi bHkgdGhlIHdyaXRlIHNwZWVkcyBhcyBlYWNoIGdyb3VwIG9mIGRpc2tzIHdpbGwgCj4+IGVuZCB1 cCBoYXZpbmcgdGhlIHdyaXRlIHNwZWVkIG9mIGEgc2luZ2xlIGRpc2sgYXMgYWxsIG90aGVyIGRp c2tzIG9mIAo+PiB0aGF0IGdyb3VwIGhhdmUgdG8gd2FpdCBmb3IgZWFjaCBvdGhlciB0byB3cml0 ZSBhcyB3ZWxsLgo+Pgo+Cj4gVGhhdCdzIG5vdCB0cnVlIGlmIHlvdSBoYXZlIG1lZGl1bSB0byBo aWdoIHJhbmdlIGhhcmR3YXJlIHJhaWQuIEZvciAKPiBleGFtcGxlLCBIUCBTbWFydCBBcnJheSBj b21lIHdpdGggYSBmbGFzaCBjYWNoZSBvZiBhYm91dCAxIG9yIDIgR2IgCj4gdGhhdCBoaWRlcyB0 aGF0IGZyb20gdGhlIE9TLiAKCgotLS0tLS0tLS0tLS0tLTJCNEI4MzkxMTYzMzNGODQxMzY2NjVE RApDb250ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD11dGYtOApDb250ZW50LVRyYW5zZmVy LUVuY29kaW5nOiA4Yml0Cgo8aHRtbD4KICA8aGVhZD4KICAgIDxtZXRhIGh0dHAtZXF1aXY9IkNv bnRlbnQtVHlwZSIgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4KICA8L2hlYWQ+ CiAgPGJvZHkgYmdjb2xvcj0iI0ZGRkZGRiIgdGV4dD0iIzAwMDAwMCI+CiAgICA8cD5XaGF0IHlv dSBtZW50aW9uZWQgaXMgYSBzcGVjaWZpYyBjYXNlIGFuZCBub3QgYSBnZW5lcmljCiAgICAgIHNp dHVhdGlvbi4gVGhlIG1haW4gcG9pbnQgdGhlcmUgaXMgdGhhdCBSQUlEIDUgb3IgNiBpbXBhY3Rz IHdyaXRlCiAgICAgIHBlcmZvcm1hbmNlIGNvbXBhcmVkIHdoZW4geW91IHdyaXRlIHRvIG9ubHkg MiBnaXZlbiBkaXNrcyBhdCBhCiAgICAgIHRpbWUuIFRoYXQgd2FzIHRoZSBjb21wYXJpc29uIG1h ZGUuPGJyPgogICAgPC9wPgogICAgPHA+RmVybmFuZG88YnI+CiAgICA8L3A+CiAgICA8YnI+CiAg ICA8ZGl2IGNsYXNzPSJtb3otY2l0ZS1wcmVmaXgiPk9uIDA3LzA4LzIwMTcgMTY6NDksIEZhYnJp Y2UgQmFjY2hlbGxhCiAgICAgIHdyb3RlOjxicj4KICAgIDwvZGl2PgogICAgPGJsb2NrcXVvdGUg dHlwZT0iY2l0ZSIKICAgICAgY2l0ZT0ibWlkOjAzNEUzODRCLTRCMzItNEZCQy1BNzYwLTQwNEVC MzhFQkM4NkBvcmFuZ2UuZnIiPgogICAgICA8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUi IGNvbnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11dGYtOCI+CiAgICAgIDxiciBjbGFzcz0iIj4K ICAgICAgPGRpdj4KICAgICAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIiBjbGFzcz0iIj4KICAg ICAgICAgIDxkaXYgY2xhc3M9IiI+TGUgNyBhb8O7dCAyMDE3IMOgIDE3OjQxLCBGRVJOQU5ETyBG UkVESUFOSSAmbHQ7PGEKICAgICAgICAgICAgICBocmVmPSJtYWlsdG86ZmVybmFuZG8uZnJlZGlh bmlAdXB4LmNvbSIgY2xhc3M9IiIKICAgICAgICAgICAgICBtb3otZG8tbm90LXNlbmQ9InRydWUi PmZlcm5hbmRvLmZyZWRpYW5pQHVweC5jb208L2E+Jmd0OyBhCiAgICAgICAgICAgIMOpY3JpdCA6 PC9kaXY+CiAgICAgICAgICA8YnIgY2xhc3M9IkFwcGxlLWludGVyY2hhbmdlLW5ld2xpbmUiPgog ICAgICAgIDwvYmxvY2txdW90ZT4KICAgICAgICA8YnIgY2xhc3M9IiI+CiAgICAgICAgPGJsb2Nr cXVvdGUgdHlwZT0iY2l0ZSIgY2xhc3M9IiI+CiAgICAgICAgICA8ZGl2IGNsYXNzPSIiPgogICAg ICAgICAgICA8cCBzdHlsZT0iZm9udC1mYW1pbHk6IEhlbHZldGljYTsgZm9udC1zaXplOiAxMnB4 OwogICAgICAgICAgICAgIGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5v cm1hbDsKICAgICAgICAgICAgICBmb250LXdlaWdodDogbm9ybWFsOyBsZXR0ZXItc3BhY2luZzog bm9ybWFsOyBvcnBoYW5zOgogICAgICAgICAgICAgIGF1dG87IHRleHQtYWxpZ246IHN0YXJ0OyB0 ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybToKICAgICAgICAgICAgICBub25lOyB3aGl0 ZS1zcGFjZTogbm9ybWFsOyB3aWRvd3M6IGF1dG87IHdvcmQtc3BhY2luZzoKICAgICAgICAgICAg ICAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgYmFja2dyb3VuZC1jb2xvcjoK ICAgICAgICAgICAgICByZ2IoMjU1LCAyNTUsIDI1NSk7IiBjbGFzcz0iIj5ZZXQgYW5vdGhlciBk b3duc2lkZSBvZgogICAgICAgICAgICAgIGhhdmluZyBhIFJBSUQgKHNwZWNpYWxseSBSQUlEIDUg b3IgNikgaXMgdGhhdCBpdCByZWR1Y2VzCiAgICAgICAgICAgICAgY29uc2lkZXJhYmx5IHRoZSB3 cml0ZSBzcGVlZHMgYXMgZWFjaCBncm91cCBvZiBkaXNrcyB3aWxsCiAgICAgICAgICAgICAgZW5k IHVwIGhhdmluZyB0aGUgd3JpdGUgc3BlZWQgb2YgYSBzaW5nbGUgZGlzayBhcyBhbGwKICAgICAg ICAgICAgICBvdGhlciBkaXNrcyBvZiB0aGF0IGdyb3VwIGhhdmUgdG8gd2FpdCBmb3IgZWFjaCBv dGhlciB0bwogICAgICAgICAgICAgIHdyaXRlIGFzIHdlbGwuPGJyIGNsYXNzPSIiPgogICAgICAg ICAgICA8L3A+CiAgICAgICAgICA8L2Rpdj4KICAgICAgICA8L2Jsb2NrcXVvdGU+CiAgICAgICAg PGJyIGNsYXNzPSIiPgogICAgICA8L2Rpdj4KICAgICAgVGhhdCdzIG5vdCB0cnVlIGlmIHlvdSBo YXZlIG1lZGl1bSB0byBoaWdoIHJhbmdlIGhhcmR3YXJlIHJhaWQuCiAgICAgIEZvciBleGFtcGxl LCBIUCBTbWFydCBBcnJheSBjb21lIHdpdGggYSBmbGFzaCBjYWNoZSBvZiBhYm91dCAxIG9yCiAg ICAgIDIgR2IgdGhhdCBoaWRlcyB0aGF0IGZyb20gdGhlIE9TLgogICAgPC9ibG9ja3F1b3RlPgog ICAgPGJyPgogIDwvYm9keT4KPC9odG1sPgoKLS0tLS0tLS0tLS0tLS0yQjRCODM5MTE2MzMzRjg0 MTM2NjY1REQtLQo= --===============8244755700651172869==-- From erekle.magradze at recogizer.de Mon Aug 7 20:05:23 2017 Content-Type: multipart/mixed; boundary="===============5337389779839608455==" MIME-Version: 1.0 From: Erekle Magradze To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Mon, 07 Aug 2017 22:05:19 +0200 Message-ID: In-Reply-To: c7a1c2e1-57c3-9fa5-0710-ebee3f3fa069@upx.com --===============5337389779839608455== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------A0081E07F314CF9B687DDA86 Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 8bit Hi Franando, So let's go with the following scenarios: 1. Let's say you have two servers (replication factor is 2), i.e. two = bricks per volume, in this case it is strongly recommended to have the = arbiter node, the metadata storage that will guarantee avoiding the = split brain situation, in this case for arbiter you don't even need a = disk with lots of space, it's enough to have a tiny ssd but hosted on a = separate server. Advantage of such setup is that you don't need the RAID = 1 for each brick, you have the metadata information stored in arbiter = node and brick replacement is easy. 2. If you have odd number of bricks (let's say 3, i.e. replication = factor is 3) in your volume and you didn't create the arbiter node as = well as you didn't configure the quorum, in this case the entire load = for keeping the consistency of the volume resides on all 3 servers, each = of them is important and each brick contains key information, they need = to cross-check each other (that's what people usually do with the first = try of gluster :) ), in this case replacing a brick is a big pain and in = this case RAID 1 is a good option to have (that's the disadvantage, i.e. = loosing the space and not having the JBOD option) advantage is that you = don't have the to have additional arbiter node. 3. You have odd number of bricks and configured arbiter node, in this = case you can easily go with JBOD, however a good practice would be to = have a RAID 1 for arbiter disks (tiny 128GB SSD-s ar perfectly = sufficient for volumes with 10s of TB-s in size.) That's basically it The rest about the reliability and setup scenarios you can find in = gluster documentation, especially look for quorum and arbiter node = configs+options. Cheers Erekle P.S. What I was mentioning, regarding a good practice is mostly related = to the operations of gluster not installation or deployment, i.e. not = the conceptual understanding of gluster (conceptually it's a JBOD system). On 08/07/2017 05:41 PM, FERNANDO FREDIANI wrote: > > Thanks for the clarification Erekle. > > However I get surprised with this way of operating from GlusterFS as = > it adds another layer of complexity to the system (either a hardware = > or software RAID) before the gluster config and increase the system's = > overall costs. > > An important point to consider is: In RAID configuration you already = > have space 'wasted' in order to build redundancy (either RAID 1, 5, or = > 6). Then when you have GlusterFS on the top of several RAIDs you have = > again more data replicated so you end up with the same data consuming = > more space in a group of disks and again on the top of several RAIDs = > depending on the Gluster configuration you have (in a RAID 1 config = > the same data is replicated 4 times). > > Yet another downside of having a RAID (specially RAID 5 or 6) is that = > it reduces considerably the write speeds as each group of disks will = > end up having the write speed of a single disk as all other disks of = > that group have to wait for each other to write as well. > > Therefore if Gluster already replicates data why does it create this = > big pain you mentioned if the data is replicated somewhere else, can = > still be retrieved to both serve clients and reconstruct the = > equivalent disk when it is replaced ? > > Fernando > > > On 07/08/2017 10:26, Erekle Magradze wrote: >> >> Hi Frenando, >> >> Here is my experience, if you consider a particular hard drive as a = >> brick for gluster volume and it dies, i.e. it becomes not accessible = >> it's a huge hassle to discard that brick and exchange with another = >> one, since gluster some tries to access that broken brick and it's = >> causing (at least it cause for me) a big pain, therefore it's better = >> to have a RAID as brick, i.e. have RAID 1 (mirroring) for each brick, = >> in this case if the disk is down you can easily exchange it and = >> rebuild the RAID without going offline, i.e switching off the volume = >> doing brick manipulations and switching it back on. >> >> Cheers >> >> Erekle >> >> >> On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote: >>> >>> For any RAID 5 or 6 configuration I normally follow a simple gold = >>> rule which gave good results so far: >>> - up to 4 disks RAID 5 >>> - 5 or more disks RAID 6 >>> >>> However I didn't really understand well the recommendation to use = >>> any RAID with GlusterFS. I always thought that GlusteFS likes to = >>> work in JBOD mode and control the disks (bricks) directlly so you = >>> can create whatever distribution rule you wish, and if a single disk = >>> fails you just replace it and which obviously have the data = >>> replicated from another. The only downside of using in this way is = >>> that the replication data will be flow accross all servers but that = >>> is not much a big issue. >>> >>> Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS. >>> >>> Thanks >>> Regards >>> Fernando >>> >>> >>> On 07/08/2017 03:46, Devin Acosta wrote: >>>> >>>> Moacir, >>>> >>>> I have recently installed multiple Red Hat Virtualization hosts for = >>>> several different companies, and have dealt with the Red Hat = >>>> Support Team in depth about optimal configuration in regards to = >>>> setting up GlusterFS most efficiently and I wanted to share with = >>>> you what I learned. >>>> >>>> In general Red Hat Virtualization team frowns upon using each DISK = >>>> of the system as just a JBOD, sure there is some protection by = >>>> having the data replicated, however, the recommendation is to use = >>>> RAID 6 (preferred) or RAID-5, or at least RAID-1 at the very least. >>>> >>>> Here is the direct quote from Red Hat when I asked about RAID and = >>>> Bricks: >>>> / >>>> / >>>> /"A typical Gluster configuration would use RAID underneath the = >>>> bricks. RAID 6 is most typical as it gives you 2 disk failure = >>>> protection, but RAID 5 could be used too. Once you have the RAIDed = >>>> bricks, you'd then apply the desired replication on top of that. = >>>> The most popular way of doing this would be distributed replicated = >>>> with 2x replication. In general you'll get better performance with = >>>> larger bricks. 12 drives is often a sweet spot. Another option = >>>> would be to create a separate tier using all SSD=E2=80=99s.=E2=80=9D / >>>> >>>> /In order to SSD tiering from my understanding you would need 1 x = >>>> NVMe drive in each server, or 4 x SSD hot tier (it needs to be = >>>> distributed, replicated for the hot tier if not using NVME). So = >>>> with you only having 1 SSD drive in each server, I=E2=80=99d suggest m= aybe = >>>> looking into the NVME option. / >>>> / >>>> / >>>> /Since your using only 3-servers, what I=E2=80=99d probably suggest is= to = >>>> do (2 Replicas + Arbiter Node), this setup actually doesn=E2=80=99t re= quire = >>>> the 3rd server to have big drives at all as it only stores = >>>> meta-data about the files and not actually a full copy. / >>>> / >>>> / >>>> /Please see the attached document that was given to me by Red Hat = >>>> to get more information on this. Hope this information helps you./ >>>> / >>>> / >>>> >>>> -- >>>> >>>> Devin Acosta, RHCA, RHVCA >>>> Red Hat Certified Architect >>>> >>>> On August 6, 2017 at 7:29:29 PM, Moacir Ferreira = >>>> (moacirferreira(a)hotmail.com ) w= rote: >>>> >>>>> I am willing to assemble a oVirt "pod", made of 3 servers, each = >>>>> with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The = >>>>> idea is to use GlusterFS to provide HA for the VMs. The 3 servers = >>>>> have a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to = >>>>> create a loop like a server triangle using the 40Gb NICs for = >>>>> virtualization files (VMs .qcow2) access and to move VMs around = >>>>> the pod (east /west traffic) while using the 10Gb interfaces for = >>>>> giving services to the outside world (north/south traffic). >>>>> >>>>> >>>>> This said, my first question is: How should I deploy GlusterFS in = >>>>> such oVirt scenario? My questions are: >>>>> >>>>> >>>>> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, = >>>>> and then create a GlusterFS using them? >>>>> >>>>> 2 - Instead, should I create a JBOD array made of all server's disks? >>>>> >>>>> 3 - What is the best Gluster configuration to provide for HA while = >>>>> not consuming too much disk space? >>>>> >>>>> 4 - Does a oVirt hypervisor pod like I am planning to build, and = >>>>> the virtualization environment, benefits from tiering when using a = >>>>> SSD disk? And yes, will Gluster do it by default or I have to = >>>>> configure it to do so? >>>>> >>>>> >>>>> At the bottom line, what is the good practice for using GlusterFS = >>>>> in small pods for enterprises? >>>>> >>>>> >>>>> You opinion/feedback will be really appreciated! >>>>> >>>>> Moacir >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >> > -- = Recogizer Group GmbH Dr.rer.nat. Erekle Magradze Lead Big Data Engineering & DevOps Rheinwerkallee 2, 53227 Bonn Tel: +49 228 29974555 E-Mail erekle.magradze(a)recogizer.de Web: www.recogizer.com = Recogizer auf LinkedIn https://www.linkedin.com/company-beta/10039182/ Folgen Sie uns auf Twitter https://twitter.com/recogizer = ----------------------------------------------------------------- Recogizer Group GmbH Gesch=C3=A4ftsf=C3=BChrer: Oliver Habisch, Carsten Kreutze Handelsregister: Amtsgericht Bonn HRB 20724 Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993 = Diese E-Mail enth=C3=A4lt vertrauliche und/oder rechtlich gesch=C3=BCtzte I= nformationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=C3=BCmlich= erhalten haben, informieren Sie bitte sofort den Absender und l=C3=B6schen Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der = darin enthaltenen Informationen ist nicht gestattet. --------------A0081E07F314CF9B687DDA86 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit
Hi Franando,

So let's go with the following scenarios:

1. Let's say you have two servers (replication factor is 2), i.e. two bricks per volume, in this case it is strongly recommended to have the arbiter node, the metadata storage that will guarantee avoiding the split brain situation, in this case for arbiter you don't even need a disk with lots of space, it's enough to have a tiny ssd but hosted on a separate server. Advantage of such setup is that you don't need the RAID 1 for each brick, you have the metadata information stored in arbiter node and brick replacement is easy.

2. If you have odd number of bricks (let's say 3, i.e. replication factor is 3) in your volume and you didn't create the arbiter node as well as you didn't configure the quorum, in this case the entire load for keeping the consistency of the volume resides on all 3 servers, each of them is important and each brick contains key information, they need to cross-check each other (that's what people usually do with the first try of gluster :) ), in this case replacing a brick is a big pain and in this case RAID 1 is a good option to have (that's the disadvantage, i.e. loosing the space and not having the JBOD option) advantage is that you don't have the to have additional arbiter node.

3. You have odd number of bricks and configured arbiter node, in this case you can easily go with JBOD, however a good practice would be to have a RAID 1 for arbiter disks (tiny 128GB SSD-s ar perfectly sufficient for volumes with 10s of TB-s in size.)

That's basically it

The rest about the reliability and setup scenarios you can find in gluster documentation, especially look for quorum and arbiter node configs+options.

Cheers

Erekle

P.S. What I was mentioning, regarding a good practice is mostly related to the operations of gluster not installation or deployment, i.e. not the conceptual understanding of gluster (conceptually it's a JBOD system).

On 08/07/2017 05:41 PM, FERNANDO FREDIANI wrote:

Thanks for the clarification Erekle.

However I get surprised with this way of operating from GlusterFS as it adds another layer of complexity to the system (either a hardware or software RAID) before the gluster config and increase the system's overall costs.

An important point to consider is: In RAID configuration you already have space 'wasted' in order to build redundancy (either RAID 1, 5, or 6). Then when you have GlusterFS on the top of several RAIDs you have again more data replicated so you end up with the same data consuming more space in a group of disks and again on the top of several RAIDs depending on the Gluster configuration you have (in a RAID 1 config the same data is replicated 4 times).

Yet another downside of having a RAID (specially RAID 5 or 6) is that it reduces considerably the write speeds as each group of disks will end up having the write speed of a single disk as all other disks of that group have to wait for each other to write as well.

Therefore if Gluster already replicates data why does it create this big pain you mentioned if the data is replicated somewhere else, can still be retrieved to both serve clients and reconstruct the equivalent disk when it is replaced ?

Fernando

On 07/08/2017 10:26, Erekle Magradze wrote:

Hi Frenando,

Here is my experience, if you consider a particular hard drive as a brick for gluster volume and it dies, i.e. it becomes not accessible it's a huge hassle to discard that brick and exchange with another one, since gluster some tries to access that broken brick and it's causing (at least it cause for me) a big pain, therefore it's better to have a RAID as brick, i.e. have RAID 1 (mirroring) for each brick, in this case if the disk is down you can easily exchange it and rebuild the RAID without going offline, i.e switching off the volume doing brick manipulations and switching it back on.

Cheers

Erekle

On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote:

For any RAID 5 or 6 configuration I normally follow a simple gold rule which gave good results so far:
- up to 4 disks RAID 5
- 5 or more disks RAID 6

However I didn't really understand well the recommendation to use any RAID with GlusterFS. I always thought that GlusteFS likes to work in JBOD mode and control the disks (bricks) directlly so you can create whatever distribution rule you wish, and if a single disk fails you just replace it and which obviously have the data replicated from another. The only downside of using in this way is that the replication data will be flow accross all servers but that is not much a big issue.

Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS.

Thanks
Regards
Fernando

On 07/08/2017 03:46, Devin Acosta wrote:

Moacir,

I have recently installed multiple Red Hat Virtualization hosts for several different companies, and have dealt with the Red Hat Support Team in depth about optimal configuration in regards to setting up GlusterFS most efficiently and I wanted to share with you what I learned.

In general Red Hat Virtualization team frowns upon using each DISK of the system as just a JBOD, sure there is some protection by having the data replicated, however, the recommendation is to use RAID 6 (preferred) or RAID-5, or at least RAID-1 at the very least.

Here is the=C2=A0direct quote from Red = Hat when I asked about RAID and Bricks:

"A typical Gluster configuration would use RAID underneath the bricks. RAID 6 is most typical as it gives you 2 disk failure protection, but RAID 5 could be used too.=C2=A0Once you have the RAIDed bricks, you'd then apply the desired replication on top of that. The most popular way of doing this would be distributed replicated with 2x replication.=C2=A0In general you'll get better performance with larger bricks. 12 drives is often a sweet spot. Another option would be to create a separate tier using all SSD=E2=80=99s.=E2=80=9D=C2=A0

In order to SSD=C2=A0tiering from my understanding you would need 1 x NVMe drive in each server, or 4 x SSD hot tier (it needs to be distributed, replicated for the hot tier if not using NVME). So with you only having 1 SSD drive in each server,=C2=A0I=E2=80=99d suggest maybe looking into the N= VME option.=C2=A0

Since your using only 3-servers, what I=E2=80=99d probably suggest is to do (2 Replicas + Arbiter Node), this setup actually doesn=E2=80=99t requir= e the 3rd server to have big drives at all as it only stores meta-data about the files and not actually a full copy.=C2=A0

Please see the attached document that was given to me by Red Hat to get more information on this. Hope this information helps you.=

--

Devin Acosta, RHCA, RHVCA

Red Hat Certified Architect

On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (moacirferreira(a)hotmail.com) wrote:

I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop like a server triangle using the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VMs around the pod (east /west traffic) while using the 10Gb interfaces for giving services to the outside world (north/south traffic).

This said, my first question is: How should I deploy GlusterFS in such oVirt scenario? My questions are:

1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then create a GlusterFS using them?

2 - Instead, should I create a JBOD array made of all server's disks?

3 - What is the best Gluster configuration to provide for HA while not consuming too much disk space?

4 - Does a oVirt hypervisor pod like I am planning to build, and the virtualization environment, benefits from tiering when using a SSD disk? And yes, will Gluster do it by default or I have to configure it to do so?

At the bottom line, what is the good practice for using GlusterFS in small pods for enterprises?

You opinion/feedback will be really appreciated!

Moacir

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailm= an/listinfo/users

_______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/list= info/users

_______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/list= info/users

-- = Recogizer Group GmbH Dr.rer.nat. Erekle Magradze Lead Big Data Engineering & DevOps Rheinwerkallee 2, 53227 Bonn Tel: +49 228 29974555 E-Mail erekle.magradze(a)recogizer.de Web: www.recogizer.com = Recogizer auf LinkedIn https://www.linkedin.com/company-be= ta/10039182/ Folgen Sie uns auf Twitter https://twitter.com/recogizer = ----------------------------------------------------------------- Recogizer Group GmbH Gesch=C3=A4ftsf=C3=BChrer: Oliver Habisch, Carsten Kreutze Handelsregister: Amtsgericht Bonn HRB 20724 Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993 = Diese E-Mail enth=C3=A4lt vertrauliche und/oder rechtlich gesch=C3=BCtzte I= nformationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=C3=BCmlich= erhalten haben, informieren Sie bitte sofort den Absender und l=C3=B6schen Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der = darin enthaltenen Informationen ist nicht gestattet.
--------------A0081E07F314CF9B687DDA86-- --===============5337389779839608455== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS1BMDA4MUUwN0YzMTRDRjlCNjg3RERBODYKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDhiaXQK CkhpIEZyYW5hbmRvLAoKU28gbGV0J3MgZ28gd2l0aCB0aGUgZm9sbG93aW5nIHNjZW5hcmlvczoK CjEuIExldCdzIHNheSB5b3UgaGF2ZSB0d28gc2VydmVycyAocmVwbGljYXRpb24gZmFjdG9yIGlz IDIpLCBpLmUuIHR3byAKYnJpY2tzIHBlciB2b2x1bWUsIGluIHRoaXMgY2FzZSBpdCBpcyBzdHJv bmdseSByZWNvbW1lbmRlZCB0byBoYXZlIHRoZSAKYXJiaXRlciBub2RlLCB0aGUgbWV0YWRhdGEg c3RvcmFnZSB0aGF0IHdpbGwgZ3VhcmFudGVlIGF2b2lkaW5nIHRoZSAKc3BsaXQgYnJhaW4gc2l0 dWF0aW9uLCBpbiB0aGlzIGNhc2UgZm9yIGFyYml0ZXIgeW91IGRvbid0IGV2ZW4gbmVlZCBhIApk aXNrIHdpdGggbG90cyBvZiBzcGFjZSwgaXQncyBlbm91Z2ggdG8gaGF2ZSBhIHRpbnkgc3NkIGJ1 dCBob3N0ZWQgb24gYSAKc2VwYXJhdGUgc2VydmVyLiBBZHZhbnRhZ2Ugb2Ygc3VjaCBzZXR1cCBp cyB0aGF0IHlvdSBkb24ndCBuZWVkIHRoZSBSQUlEIAoxIGZvciBlYWNoIGJyaWNrLCB5b3UgaGF2 ZSB0aGUgbWV0YWRhdGEgaW5mb3JtYXRpb24gc3RvcmVkIGluIGFyYml0ZXIgCm5vZGUgYW5kIGJy aWNrIHJlcGxhY2VtZW50IGlzIGVhc3kuCgoyLiBJZiB5b3UgaGF2ZSBvZGQgbnVtYmVyIG9mIGJy aWNrcyAobGV0J3Mgc2F5IDMsIGkuZS4gcmVwbGljYXRpb24gCmZhY3RvciBpcyAzKSBpbiB5b3Vy IHZvbHVtZSBhbmQgeW91IGRpZG4ndCBjcmVhdGUgdGhlIGFyYml0ZXIgbm9kZSBhcyAKd2VsbCBh cyB5b3UgZGlkbid0IGNvbmZpZ3VyZSB0aGUgcXVvcnVtLCBpbiB0aGlzIGNhc2UgdGhlIGVudGly ZSBsb2FkIApmb3Iga2VlcGluZyB0aGUgY29uc2lzdGVuY3kgb2YgdGhlIHZvbHVtZSByZXNpZGVz IG9uIGFsbCAzIHNlcnZlcnMsIGVhY2ggCm9mIHRoZW0gaXMgaW1wb3J0YW50IGFuZCBlYWNoIGJy aWNrIGNvbnRhaW5zIGtleSBpbmZvcm1hdGlvbiwgdGhleSBuZWVkIAp0byBjcm9zcy1jaGVjayBl YWNoIG90aGVyICh0aGF0J3Mgd2hhdCBwZW9wbGUgdXN1YWxseSBkbyB3aXRoIHRoZSBmaXJzdCAK dHJ5IG9mIGdsdXN0ZXIgOikgKSwgaW4gdGhpcyBjYXNlIHJlcGxhY2luZyBhIGJyaWNrIGlzIGEg YmlnIHBhaW4gYW5kIGluIAp0aGlzIGNhc2UgUkFJRCAxIGlzIGEgZ29vZCBvcHRpb24gdG8gaGF2 ZSAodGhhdCdzIHRoZSBkaXNhZHZhbnRhZ2UsIGkuZS4gCmxvb3NpbmcgdGhlIHNwYWNlIGFuZCBu b3QgaGF2aW5nIHRoZSBKQk9EIG9wdGlvbikgYWR2YW50YWdlIGlzIHRoYXQgeW91IApkb24ndCBo YXZlIHRoZSB0byBoYXZlIGFkZGl0aW9uYWwgYXJiaXRlciBub2RlLgoKMy4gWW91IGhhdmUgb2Rk IG51bWJlciBvZiBicmlja3MgYW5kIGNvbmZpZ3VyZWQgYXJiaXRlciBub2RlLCBpbiB0aGlzIApj YXNlIHlvdSBjYW4gZWFzaWx5IGdvIHdpdGggSkJPRCwgaG93ZXZlciBhIGdvb2QgcHJhY3RpY2Ug d291bGQgYmUgdG8gCmhhdmUgYSBSQUlEIDEgZm9yIGFyYml0ZXIgZGlza3MgKHRpbnkgMTI4R0Ig U1NELXMgYXIgcGVyZmVjdGx5IApzdWZmaWNpZW50IGZvciB2b2x1bWVzIHdpdGggMTBzIG9mIFRC LXMgaW4gc2l6ZS4pCgpUaGF0J3MgYmFzaWNhbGx5IGl0CgpUaGUgcmVzdCBhYm91dCB0aGUgcmVs aWFiaWxpdHkgYW5kIHNldHVwIHNjZW5hcmlvcyB5b3UgY2FuIGZpbmQgaW4gCmdsdXN0ZXIgZG9j dW1lbnRhdGlvbiwgZXNwZWNpYWxseSBsb29rIGZvciBxdW9ydW0gYW5kIGFyYml0ZXIgbm9kZSAK Y29uZmlncytvcHRpb25zLgoKQ2hlZXJzCgpFcmVrbGUKClAuUy4gV2hhdCBJIHdhcyBtZW50aW9u aW5nLCByZWdhcmRpbmcgYSBnb29kIHByYWN0aWNlIGlzIG1vc3RseSByZWxhdGVkIAp0byB0aGUg b3BlcmF0aW9ucyBvZiBnbHVzdGVyIG5vdCBpbnN0YWxsYXRpb24gb3IgZGVwbG95bWVudCwgaS5l LiBub3QgCnRoZSBjb25jZXB0dWFsIHVuZGVyc3RhbmRpbmcgb2YgZ2x1c3RlciAoY29uY2VwdHVh bGx5IGl0J3MgYSBKQk9EIHN5c3RlbSkuCgoKT24gMDgvMDcvMjAxNyAwNTo0MSBQTSwgRkVSTkFO RE8gRlJFRElBTkkgd3JvdGU6Cj4KPiBUaGFua3MgZm9yIHRoZSBjbGFyaWZpY2F0aW9uIEVyZWts ZS4KPgo+IEhvd2V2ZXIgSSBnZXQgc3VycHJpc2VkIHdpdGggdGhpcyB3YXkgb2Ygb3BlcmF0aW5n IGZyb20gR2x1c3RlckZTIGFzIAo+IGl0IGFkZHMgYW5vdGhlciBsYXllciBvZiBjb21wbGV4aXR5 IHRvIHRoZSBzeXN0ZW0gKGVpdGhlciBhIGhhcmR3YXJlIAo+IG9yIHNvZnR3YXJlIFJBSUQpIGJl Zm9yZSB0aGUgZ2x1c3RlciBjb25maWcgYW5kIGluY3JlYXNlIHRoZSBzeXN0ZW0ncyAKPiBvdmVy YWxsIGNvc3RzLgo+Cj4gQW4gaW1wb3J0YW50IHBvaW50IHRvIGNvbnNpZGVyIGlzOiBJbiBSQUlE IGNvbmZpZ3VyYXRpb24geW91IGFscmVhZHkgCj4gaGF2ZSBzcGFjZSAnd2FzdGVkJyBpbiBvcmRl ciB0byBidWlsZCByZWR1bmRhbmN5IChlaXRoZXIgUkFJRCAxLCA1LCBvciAKPiA2KS4gVGhlbiB3 aGVuIHlvdSBoYXZlIEdsdXN0ZXJGUyBvbiB0aGUgdG9wIG9mIHNldmVyYWwgUkFJRHMgeW91IGhh dmUgCj4gYWdhaW4gbW9yZSBkYXRhIHJlcGxpY2F0ZWQgc28geW91IGVuZCB1cCB3aXRoIHRoZSBz YW1lIGRhdGEgY29uc3VtaW5nIAo+IG1vcmUgc3BhY2UgaW4gYSBncm91cCBvZiBkaXNrcyBhbmQg YWdhaW4gb24gdGhlIHRvcCBvZiBzZXZlcmFsIFJBSURzIAo+IGRlcGVuZGluZyBvbiB0aGUgR2x1 c3RlciBjb25maWd1cmF0aW9uIHlvdSBoYXZlIChpbiBhIFJBSUQgMSBjb25maWcgCj4gdGhlIHNh bWUgZGF0YSBpcyByZXBsaWNhdGVkIDQgdGltZXMpLgo+Cj4gWWV0IGFub3RoZXIgZG93bnNpZGUg b2YgaGF2aW5nIGEgUkFJRCAoc3BlY2lhbGx5IFJBSUQgNSBvciA2KSBpcyB0aGF0IAo+IGl0IHJl ZHVjZXMgY29uc2lkZXJhYmx5IHRoZSB3cml0ZSBzcGVlZHMgYXMgZWFjaCBncm91cCBvZiBkaXNr cyB3aWxsIAo+IGVuZCB1cCBoYXZpbmcgdGhlIHdyaXRlIHNwZWVkIG9mIGEgc2luZ2xlIGRpc2sg YXMgYWxsIG90aGVyIGRpc2tzIG9mIAo+IHRoYXQgZ3JvdXAgaGF2ZSB0byB3YWl0IGZvciBlYWNo IG90aGVyIHRvIHdyaXRlIGFzIHdlbGwuCj4KPiBUaGVyZWZvcmUgaWYgR2x1c3RlciBhbHJlYWR5 IHJlcGxpY2F0ZXMgZGF0YSB3aHkgZG9lcyBpdCBjcmVhdGUgdGhpcyAKPiBiaWcgcGFpbiB5b3Ug bWVudGlvbmVkIGlmIHRoZSBkYXRhIGlzIHJlcGxpY2F0ZWQgc29tZXdoZXJlIGVsc2UsIGNhbiAK PiBzdGlsbCBiZSByZXRyaWV2ZWQgdG8gYm90aCBzZXJ2ZSBjbGllbnRzIGFuZCByZWNvbnN0cnVj dCB0aGUgCj4gZXF1aXZhbGVudCBkaXNrIHdoZW4gaXQgaXMgcmVwbGFjZWQgPwo+Cj4gRmVybmFu ZG8KPgo+Cj4gT24gMDcvMDgvMjAxNyAxMDoyNiwgRXJla2xlIE1hZ3JhZHplIHdyb3RlOgo+Pgo+ PiBIaSBGcmVuYW5kbywKPj4KPj4gSGVyZSBpcyBteSBleHBlcmllbmNlLCBpZiB5b3UgY29uc2lk ZXIgYSBwYXJ0aWN1bGFyIGhhcmQgZHJpdmUgYXMgYSAKPj4gYnJpY2sgZm9yIGdsdXN0ZXIgdm9s dW1lIGFuZCBpdCBkaWVzLCBpLmUuIGl0IGJlY29tZXMgbm90IGFjY2Vzc2libGUgCj4+IGl0J3Mg YSBodWdlIGhhc3NsZSB0byBkaXNjYXJkIHRoYXQgYnJpY2sgYW5kIGV4Y2hhbmdlIHdpdGggYW5v dGhlciAKPj4gb25lLCBzaW5jZSBnbHVzdGVyIHNvbWUgdHJpZXMgdG8gYWNjZXNzIHRoYXQgYnJv a2VuIGJyaWNrIGFuZCBpdCdzIAo+PiBjYXVzaW5nIChhdCBsZWFzdCBpdCBjYXVzZSBmb3IgbWUp IGEgYmlnIHBhaW4sIHRoZXJlZm9yZSBpdCdzIGJldHRlciAKPj4gdG8gaGF2ZSBhIFJBSUQgYXMg YnJpY2ssIGkuZS4gaGF2ZSBSQUlEIDEgKG1pcnJvcmluZykgZm9yIGVhY2ggYnJpY2ssIAo+PiBp biB0aGlzIGNhc2UgaWYgdGhlIGRpc2sgaXMgZG93biB5b3UgY2FuIGVhc2lseSBleGNoYW5nZSBp dCBhbmQgCj4+IHJlYnVpbGQgdGhlIFJBSUQgd2l0aG91dCBnb2luZyBvZmZsaW5lLCBpLmUgc3dp dGNoaW5nIG9mZiB0aGUgdm9sdW1lIAo+PiBkb2luZyBicmljayBtYW5pcHVsYXRpb25zIGFuZCBz d2l0Y2hpbmcgaXQgYmFjayBvbi4KPj4KPj4gQ2hlZXJzCj4+Cj4+IEVyZWtsZQo+Pgo+Pgo+PiBP biAwOC8wNy8yMDE3IDAzOjA0IFBNLCBGRVJOQU5ETyBGUkVESUFOSSB3cm90ZToKPj4+Cj4+PiBG b3IgYW55IFJBSUQgNSBvciA2IGNvbmZpZ3VyYXRpb24gSSBub3JtYWxseSBmb2xsb3cgYSBzaW1w bGUgZ29sZCAKPj4+IHJ1bGUgd2hpY2ggZ2F2ZSBnb29kIHJlc3VsdHMgc28gZmFyOgo+Pj4gLSB1 cCB0byA0IGRpc2tzIFJBSUQgNQo+Pj4gLSA1IG9yIG1vcmUgZGlza3MgUkFJRCA2Cj4+Pgo+Pj4g SG93ZXZlciBJIGRpZG4ndCByZWFsbHkgdW5kZXJzdGFuZCB3ZWxsIHRoZSByZWNvbW1lbmRhdGlv biB0byB1c2UgCj4+PiBhbnkgUkFJRCB3aXRoIEdsdXN0ZXJGUy4gSSBhbHdheXMgdGhvdWdodCB0 aGF0IEdsdXN0ZUZTIGxpa2VzIHRvIAo+Pj4gd29yayBpbiBKQk9EIG1vZGUgYW5kIGNvbnRyb2wg dGhlIGRpc2tzIChicmlja3MpIGRpcmVjdGxseSBzbyB5b3UgCj4+PiBjYW4gY3JlYXRlIHdoYXRl dmVyIGRpc3RyaWJ1dGlvbiBydWxlIHlvdSB3aXNoLCBhbmQgaWYgYSBzaW5nbGUgZGlzayAKPj4+ IGZhaWxzIHlvdSBqdXN0IHJlcGxhY2UgaXQgYW5kIHdoaWNoIG9idmlvdXNseSBoYXZlIHRoZSBk YXRhIAo+Pj4gcmVwbGljYXRlZCBmcm9tIGFub3RoZXIuIFRoZSBvbmx5IGRvd25zaWRlIG9mIHVz aW5nIGluIHRoaXMgd2F5IGlzIAo+Pj4gdGhhdCB0aGUgcmVwbGljYXRpb24gZGF0YSB3aWxsIGJl IGZsb3cgYWNjcm9zcyBhbGwgc2VydmVycyBidXQgdGhhdCAKPj4+IGlzIG5vdCBtdWNoIGEgYmln IGlzc3VlLgo+Pj4KPj4+IEFueW9uZSBjYW4gZWxhYm9yYXRlIGFib3V0IFVzaW5nIFJBSUQgKyBH bHVzdGVyRlMgYW5kIEpCT0QgKyBHbHVzdGVyRlMuCj4+Pgo+Pj4gVGhhbmtzCj4+PiBSZWdhcmRz Cj4+PiBGZXJuYW5kbwo+Pj4KPj4+Cj4+PiBPbiAwNy8wOC8yMDE3IDAzOjQ2LCBEZXZpbiBBY29z dGEgd3JvdGU6Cj4+Pj4KPj4+PiBNb2FjaXIsCj4+Pj4KPj4+PiBJIGhhdmUgcmVjZW50bHkgaW5z dGFsbGVkIG11bHRpcGxlIFJlZCBIYXQgVmlydHVhbGl6YXRpb24gaG9zdHMgZm9yIAo+Pj4+IHNl dmVyYWwgZGlmZmVyZW50IGNvbXBhbmllcywgYW5kIGhhdmUgZGVhbHQgd2l0aCB0aGUgUmVkIEhh dCAKPj4+PiBTdXBwb3J0IFRlYW0gaW4gZGVwdGggYWJvdXQgb3B0aW1hbCBjb25maWd1cmF0aW9u IGluIHJlZ2FyZHMgdG8gCj4+Pj4gc2V0dGluZyB1cCBHbHVzdGVyRlMgbW9zdCBlZmZpY2llbnRs eSBhbmQgSSB3YW50ZWQgdG8gc2hhcmUgd2l0aCAKPj4+PiB5b3Ugd2hhdCBJIGxlYXJuZWQuCj4+ Pj4KPj4+PiBJbiBnZW5lcmFsIFJlZCBIYXQgVmlydHVhbGl6YXRpb24gdGVhbSBmcm93bnMgdXBv biB1c2luZyBlYWNoIERJU0sgCj4+Pj4gb2YgdGhlIHN5c3RlbSBhcyBqdXN0IGEgSkJPRCwgc3Vy ZSB0aGVyZSBpcyBzb21lIHByb3RlY3Rpb24gYnkgCj4+Pj4gaGF2aW5nIHRoZSBkYXRhIHJlcGxp Y2F0ZWQsIGhvd2V2ZXIsIHRoZSByZWNvbW1lbmRhdGlvbiBpcyB0byB1c2UgCj4+Pj4gUkFJRCA2 IChwcmVmZXJyZWQpIG9yIFJBSUQtNSwgb3IgYXQgbGVhc3QgUkFJRC0xIGF0IHRoZSB2ZXJ5IGxl YXN0Lgo+Pj4+Cj4+Pj4gSGVyZSBpcyB0aGUgZGlyZWN0IHF1b3RlIGZyb20gUmVkIEhhdCB3aGVu IEkgYXNrZWQgYWJvdXQgUkFJRCBhbmQgCj4+Pj4gQnJpY2tzOgo+Pj4+IC8KPj4+PiAvCj4+Pj4g LyJBIHR5cGljYWwgR2x1c3RlciBjb25maWd1cmF0aW9uIHdvdWxkIHVzZSBSQUlEIHVuZGVybmVh dGggdGhlIAo+Pj4+IGJyaWNrcy4gUkFJRCA2IGlzIG1vc3QgdHlwaWNhbCBhcyBpdCBnaXZlcyB5 b3UgMiBkaXNrIGZhaWx1cmUgCj4+Pj4gcHJvdGVjdGlvbiwgYnV0IFJBSUQgNSBjb3VsZCBiZSB1 c2VkIHRvby4gT25jZSB5b3UgaGF2ZSB0aGUgUkFJRGVkIAo+Pj4+IGJyaWNrcywgeW91J2QgdGhl biBhcHBseSB0aGUgZGVzaXJlZCByZXBsaWNhdGlvbiBvbiB0b3Agb2YgdGhhdC4gCj4+Pj4gVGhl IG1vc3QgcG9wdWxhciB3YXkgb2YgZG9pbmcgdGhpcyB3b3VsZCBiZSBkaXN0cmlidXRlZCByZXBs aWNhdGVkIAo+Pj4+IHdpdGggMnggcmVwbGljYXRpb24uIEluIGdlbmVyYWwgeW91J2xsIGdldCBi ZXR0ZXIgcGVyZm9ybWFuY2Ugd2l0aCAKPj4+PiBsYXJnZXIgYnJpY2tzLiAxMiBkcml2ZXMgaXMg b2Z0ZW4gYSBzd2VldCBzcG90LiBBbm90aGVyIG9wdGlvbiAKPj4+PiB3b3VsZCBiZSB0byBjcmVh dGUgYSBzZXBhcmF0ZSB0aWVyIHVzaW5nIGFsbCBTU0TigJlzLuKAnSAvCj4+Pj4KPj4+PiAvSW4g b3JkZXIgdG8gU1NEIHRpZXJpbmcgZnJvbSBteSB1bmRlcnN0YW5kaW5nIHlvdSB3b3VsZCBuZWVk IDEgeCAKPj4+PiBOVk1lIGRyaXZlIGluIGVhY2ggc2VydmVyLCBvciA0IHggU1NEIGhvdCB0aWVy IChpdCBuZWVkcyB0byBiZSAKPj4+PiBkaXN0cmlidXRlZCwgcmVwbGljYXRlZCBmb3IgdGhlIGhv dCB0aWVyIGlmIG5vdCB1c2luZyBOVk1FKS4gU28gCj4+Pj4gd2l0aCB5b3Ugb25seSBoYXZpbmcg MSBTU0QgZHJpdmUgaW4gZWFjaCBzZXJ2ZXIsIEnigJlkIHN1Z2dlc3QgbWF5YmUgCj4+Pj4gbG9v a2luZyBpbnRvIHRoZSBOVk1FIG9wdGlvbi4gLwo+Pj4+IC8KPj4+PiAvCj4+Pj4gL1NpbmNlIHlv dXIgdXNpbmcgb25seSAzLXNlcnZlcnMsIHdoYXQgSeKAmWQgcHJvYmFibHkgc3VnZ2VzdCBpcyB0 byAKPj4+PiBkbyAoMiBSZXBsaWNhcyArIEFyYml0ZXIgTm9kZSksIHRoaXMgc2V0dXAgYWN0dWFs bHkgZG9lc27igJl0IHJlcXVpcmUgCj4+Pj4gdGhlIDNyZCBzZXJ2ZXIgdG8gaGF2ZSBiaWcgZHJp dmVzIGF0IGFsbCBhcyBpdCBvbmx5IHN0b3JlcyAKPj4+PiBtZXRhLWRhdGEgYWJvdXQgdGhlIGZp bGVzIGFuZCBub3QgYWN0dWFsbHkgYSBmdWxsIGNvcHkuIC8KPj4+PiAvCj4+Pj4gLwo+Pj4+IC9Q bGVhc2Ugc2VlIHRoZSBhdHRhY2hlZCBkb2N1bWVudCB0aGF0IHdhcyBnaXZlbiB0byBtZSBieSBS ZWQgSGF0IAo+Pj4+IHRvIGdldCBtb3JlIGluZm9ybWF0aW9uIG9uIHRoaXMuIEhvcGUgdGhpcyBp bmZvcm1hdGlvbiBoZWxwcyB5b3UuLwo+Pj4+IC8KPj4+PiAvCj4+Pj4KPj4+PiAtLQo+Pj4+Cj4+ Pj4gRGV2aW4gQWNvc3RhLCBSSENBLCBSSFZDQQo+Pj4+IFJlZCBIYXQgQ2VydGlmaWVkIEFyY2hp dGVjdAo+Pj4+Cj4+Pj4gT24gQXVndXN0IDYsIDIwMTcgYXQgNzoyOToyOSBQTSwgTW9hY2lyIEZl cnJlaXJhIAo+Pj4+IChtb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbSA8bWFpbHRvOm1vYWNpcmZl cnJlaXJhQGhvdG1haWwuY29tPikgd3JvdGU6Cj4+Pj4KPj4+Pj4gSSBhbSB3aWxsaW5nIHRvIGFz c2VtYmxlIGEgb1ZpcnQgInBvZCIsIG1hZGUgb2YgMyBzZXJ2ZXJzLCBlYWNoIAo+Pj4+PiB3aXRo IDIgQ1BVIHNvY2tldHMgb2YgMTIgY29yZXMsIDI1NkdCIFJBTSwgNyBIREQgMTBLLCAxIFNTRC4g VGhlIAo+Pj4+PiBpZGVhIGlzIHRvIHVzZSBHbHVzdGVyRlMgdG8gcHJvdmlkZSBIQSBmb3IgdGhl IFZNcy4gVGhlIDMgc2VydmVycyAKPj4+Pj4gaGF2ZSBhIGR1YWwgNDBHYiBOSUMgYW5kIGEgZHVh bCAxMEdiIE5JQy4gU28gbXkgaW50ZW50aW9uIGlzIHRvIAo+Pj4+PiBjcmVhdGUgYSBsb29wIGxp a2UgYSBzZXJ2ZXIgdHJpYW5nbGUgdXNpbmcgdGhlIDQwR2IgTklDcyBmb3IgCj4+Pj4+IHZpcnR1 YWxpemF0aW9uIGZpbGVzIChWTXMgLnFjb3cyKSBhY2Nlc3MgYW5kIHRvIG1vdmUgVk1zIGFyb3Vu ZCAKPj4+Pj4gdGhlIHBvZCAoZWFzdCAvd2VzdCB0cmFmZmljKSB3aGlsZSB1c2luZyB0aGUgMTBH YiBpbnRlcmZhY2VzIGZvciAKPj4+Pj4gZ2l2aW5nIHNlcnZpY2VzIHRvIHRoZSBvdXRzaWRlIHdv cmxkIChub3J0aC9zb3V0aCB0cmFmZmljKS4KPj4+Pj4KPj4+Pj4KPj4+Pj4gVGhpcyBzYWlkLCBt eSBmaXJzdCBxdWVzdGlvbiBpczogSG93IHNob3VsZCBJIGRlcGxveSBHbHVzdGVyRlMgaW4gCj4+ Pj4+IHN1Y2ggb1ZpcnQgc2NlbmFyaW8/IE15IHF1ZXN0aW9ucyBhcmU6Cj4+Pj4+Cj4+Pj4+Cj4+ Pj4+IDEgLSBTaG91bGQgSSBjcmVhdGUgMyBSQUlEIChpLmUuOiBSQUlEIDUpLCBvbmUgb24gZWFj aCBvVmlydCBub2RlLCAKPj4+Pj4gYW5kIHRoZW4gY3JlYXRlIGEgR2x1c3RlckZTIHVzaW5nIHRo ZW0/Cj4+Pj4+Cj4+Pj4+IDIgLSBJbnN0ZWFkLCBzaG91bGQgSSBjcmVhdGUgYSBKQk9EIGFycmF5 IG1hZGUgb2YgYWxsIHNlcnZlcidzIGRpc2tzPwo+Pj4+Pgo+Pj4+PiAzIC0gV2hhdCBpcyB0aGUg YmVzdCBHbHVzdGVyIGNvbmZpZ3VyYXRpb24gdG8gcHJvdmlkZSBmb3IgSEEgd2hpbGUgCj4+Pj4+ IG5vdCBjb25zdW1pbmcgdG9vIG11Y2ggZGlzayBzcGFjZT8KPj4+Pj4KPj4+Pj4gNCAtIERvZXMg YSBvVmlydCBoeXBlcnZpc29yIHBvZCBsaWtlIEkgYW0gcGxhbm5pbmcgdG8gYnVpbGQsIGFuZCAK Pj4+Pj4gdGhlIHZpcnR1YWxpemF0aW9uIGVudmlyb25tZW50LCBiZW5lZml0cyBmcm9tIHRpZXJp bmcgd2hlbiB1c2luZyBhIAo+Pj4+PiBTU0QgZGlzaz8gQW5kIHllcywgd2lsbCBHbHVzdGVyIGRv IGl0IGJ5IGRlZmF1bHQgb3IgSSBoYXZlIHRvIAo+Pj4+PiBjb25maWd1cmUgaXQgdG8gZG8gc28/ Cj4+Pj4+Cj4+Pj4+Cj4+Pj4+IEF0IHRoZSBib3R0b20gbGluZSwgd2hhdCBpcyB0aGUgZ29vZCBw cmFjdGljZSBmb3IgdXNpbmcgR2x1c3RlckZTIAo+Pj4+PiBpbiBzbWFsbCBwb2RzIGZvciBlbnRl cnByaXNlcz8KPj4+Pj4KPj4+Pj4KPj4+Pj4gWW91IG9waW5pb24vZmVlZGJhY2sgd2lsbCBiZSBy ZWFsbHkgYXBwcmVjaWF0ZWQhCj4+Pj4+Cj4+Pj4+IE1vYWNpcgo+Pj4+Pgo+Pj4+PiBfX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwo+Pj4+PiBVc2VycyBtYWls aW5nIGxpc3QKPj4+Pj4gVXNlcnNAb3ZpcnQub3JnIDxtYWlsdG86VXNlcnNAb3ZpcnQub3JnPgo+ Pj4+PiBodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMKPj4+Pgo+ Pj4+Cj4+Pj4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18K Pj4+PiBVc2VycyBtYWlsaW5nIGxpc3QKPj4+PiBVc2Vyc0BvdmlydC5vcmcKPj4+PiBodHRwOi8v bGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMKPj4+Cj4+Pgo+Pj4KPj4+IF9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCj4+PiBVc2VycyBt YWlsaW5nIGxpc3QKPj4+IFVzZXJzQG92aXJ0Lm9yZwo+Pj4gaHR0cDovL2xpc3RzLm92aXJ0Lm9y Zy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzCj4+Cj4KCi0tIApSZWNvZ2l6ZXIgR3JvdXAgR21iSAoK RHIucmVyLm5hdC4gRXJla2xlIE1hZ3JhZHplCkxlYWQgQmlnIERhdGEgRW5naW5lZXJpbmcgJiBE ZXZPcHMKUmhlaW53ZXJrYWxsZWUgMiwgNTMyMjcgQm9ubgpUZWw6ICs0OSAyMjggMjk5NzQ1NTUK CkUtTWFpbCBlcmVrbGUubWFncmFkemVAcmVjb2dpemVyLmRlCldlYjogd3d3LnJlY29naXplci5j b20KICAKUmVjb2dpemVyIGF1ZiBMaW5rZWRJbiBodHRwczovL3d3dy5saW5rZWRpbi5jb20vY29t cGFueS1iZXRhLzEwMDM5MTgyLwpGb2xnZW4gU2llIHVucyBhdWYgVHdpdHRlciBodHRwczovL3R3 aXR0ZXIuY29tL3JlY29naXplcgogIAotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQpSZWNvZ2l6ZXIgR3JvdXAgR21iSApHZXNj aMOkZnRzZsO8aHJlcjogT2xpdmVyIEhhYmlzY2gsIENhcnN0ZW4gS3JldXR6ZQpIYW5kZWxzcmVn aXN0ZXI6IEFtdHNnZXJpY2h0IEJvbm4gSFJCIDIwNzI0ClNpdHogZGVyIEdlc2VsbHNjaGFmdDog Qm9ubjsgVVN0LUlELU5yLjogREUyOTQxOTU5OTMKICAKRGllc2UgRS1NYWlsIGVudGjDpGx0IHZl cnRyYXVsaWNoZSB1bmQvb2RlciByZWNodGxpY2ggZ2VzY2jDvHR6dGUgSW5mb3JtYXRpb25lbi4K V2VubiBTaWUgbmljaHQgZGVyIHJpY2h0aWdlIEFkcmVzc2F0IHNpbmQgb2RlciBkaWVzZSBFLU1h aWwgaXJydMO8bWxpY2ggZXJoYWx0ZW4gaGFiZW4sCmluZm9ybWllcmVuIFNpZSBiaXR0ZSBzb2Zv cnQgZGVuIEFic2VuZGVyIHVuZCBsw7ZzY2hlbiBTaWUgZGllc2UgTWFpbC4KRGFzIHVuZXJsYXVi dGUgS29waWVyZW4gc293aWUgZGllIHVuYmVmdWd0ZSBXZWl0ZXJnYWJlIGRpZXNlciBNYWlsIHVu ZCBkZXIgZGFyaW4gZW50aGFsdGVuZW4gSW5mb3JtYXRpb25lbiBpc3QgbmljaHQgZ2VzdGF0dGV0 LgoKCi0tLS0tLS0tLS0tLS0tQTAwODFFMDdGMzE0Q0Y5QjY4N0REQTg2CkNvbnRlbnQtVHlwZTog dGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDhiaXQK CjxodG1sPgogIDxoZWFkPgogICAgPG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIiBjb250 ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPgogIDwvaGVhZD4KICA8Ym9keSB0ZXh0PSIj MDAwMDAwIiBiZ2NvbG9yPSIjRkZGRkZGIj4KICAgIDxwPkhpIEZyYW5hbmRvLDwvcD4KICAgIDxw PlNvIGxldCdzIGdvIHdpdGggdGhlIGZvbGxvd2luZyBzY2VuYXJpb3M6PC9wPgogICAgPHA+MS4g TGV0J3Mgc2F5IHlvdSBoYXZlIHR3byBzZXJ2ZXJzIChyZXBsaWNhdGlvbiBmYWN0b3IgaXMgMiks IGkuZS4KICAgICAgdHdvIGJyaWNrcyBwZXIgdm9sdW1lLCBpbiB0aGlzIGNhc2UgaXQgaXMgc3Ry b25nbHkgcmVjb21tZW5kZWQgdG8KICAgICAgaGF2ZSB0aGUgYXJiaXRlciBub2RlLCB0aGUgbWV0 YWRhdGEgc3RvcmFnZSB0aGF0IHdpbGwgZ3VhcmFudGVlCiAgICAgIGF2b2lkaW5nIHRoZSBzcGxp dCBicmFpbiBzaXR1YXRpb24sIGluIHRoaXMgY2FzZSBmb3IgYXJiaXRlciB5b3UKICAgICAgZG9u J3QgZXZlbiBuZWVkIGEgZGlzayB3aXRoIGxvdHMgb2Ygc3BhY2UsIGl0J3MgZW5vdWdoIHRvIGhh dmUgYQogICAgICB0aW55IHNzZCBidXQgaG9zdGVkIG9uIGEgc2VwYXJhdGUgc2VydmVyLiBBZHZh bnRhZ2Ugb2Ygc3VjaCBzZXR1cAogICAgICBpcyB0aGF0IHlvdSBkb24ndCBuZWVkIHRoZSBSQUlE IDEgZm9yIGVhY2ggYnJpY2ssIHlvdSBoYXZlIHRoZQogICAgICBtZXRhZGF0YSBpbmZvcm1hdGlv biBzdG9yZWQgaW4gYXJiaXRlciBub2RlIGFuZCBicmljayByZXBsYWNlbWVudAogICAgICBpcyBl YXN5LjwvcD4KICAgIDxwPjIuIElmIHlvdSBoYXZlIG9kZCBudW1iZXIgb2YgYnJpY2tzIChsZXQn cyBzYXkgMywgaS5lLgogICAgICByZXBsaWNhdGlvbiBmYWN0b3IgaXMgMykgaW4geW91ciB2b2x1 bWUgYW5kIHlvdSBkaWRuJ3QgY3JlYXRlIHRoZQogICAgICBhcmJpdGVyIG5vZGUgYXMgd2VsbCBh cyB5b3UgZGlkbid0IGNvbmZpZ3VyZSB0aGUgcXVvcnVtLCBpbiB0aGlzCiAgICAgIGNhc2UgdGhl IGVudGlyZSBsb2FkIGZvciBrZWVwaW5nIHRoZSBjb25zaXN0ZW5jeSBvZiB0aGUgdm9sdW1lCiAg ICAgIHJlc2lkZXMgb24gYWxsIDMgc2VydmVycywgZWFjaCBvZiB0aGVtIGlzIGltcG9ydGFudCBh bmQgZWFjaCBicmljawogICAgICBjb250YWlucyBrZXkgaW5mb3JtYXRpb24sIHRoZXkgbmVlZCB0 byBjcm9zcy1jaGVjayBlYWNoIG90aGVyCiAgICAgICh0aGF0J3Mgd2hhdCBwZW9wbGUgdXN1YWxs eSBkbyB3aXRoIHRoZSBmaXJzdCB0cnkgb2YgZ2x1c3RlciA6KSApLAogICAgICBpbiB0aGlzIGNh c2UgcmVwbGFjaW5nIGEgYnJpY2sgaXMgYSBiaWcgcGFpbiBhbmQgaW4gdGhpcyBjYXNlIFJBSUQK ICAgICAgMSBpcyBhIGdvb2Qgb3B0aW9uIHRvIGhhdmUgKHRoYXQncyB0aGUgZGlzYWR2YW50YWdl LCBpLmUuIGxvb3NpbmcKICAgICAgdGhlIHNwYWNlIGFuZCBub3QgaGF2aW5nIHRoZSBKQk9EIG9w dGlvbikgYWR2YW50YWdlIGlzIHRoYXQgeW91CiAgICAgIGRvbid0IGhhdmUgdGhlIHRvIGhhdmUg YWRkaXRpb25hbCBhcmJpdGVyIG5vZGUuPC9wPgogICAgPHA+My4gWW91IGhhdmUgb2RkIG51bWJl ciBvZiBicmlja3MgYW5kIGNvbmZpZ3VyZWQgYXJiaXRlciBub2RlLCBpbgogICAgICB0aGlzIGNh c2UgeW91IGNhbiBlYXNpbHkgZ28gd2l0aCBKQk9ELCBob3dldmVyIGEgZ29vZCBwcmFjdGljZQog ICAgICB3b3VsZCBiZSB0byBoYXZlIGEgUkFJRCAxIGZvciBhcmJpdGVyIGRpc2tzICh0aW55IDEy OEdCIFNTRC1zIGFyCiAgICAgIHBlcmZlY3RseSBzdWZmaWNpZW50IGZvciB2b2x1bWVzIHdpdGgg MTBzIG9mIFRCLXMgaW4gc2l6ZS4pPC9wPgogICAgPHA+VGhhdCdzIGJhc2ljYWxseSBpdDwvcD4K ICAgIDxwPlRoZSByZXN0IGFib3V0IHRoZSByZWxpYWJpbGl0eSBhbmQgc2V0dXAgc2NlbmFyaW9z IHlvdSBjYW4gZmluZAogICAgICBpbiBnbHVzdGVyIGRvY3VtZW50YXRpb24sIGVzcGVjaWFsbHkg bG9vayBmb3IgcXVvcnVtIGFuZCBhcmJpdGVyCiAgICAgIG5vZGUgY29uZmlncytvcHRpb25zLjwv cD4KICAgIDxwPkNoZWVyczwvcD4KICAgIDxwPkVyZWtsZTwvcD4KICAgIDxwPlAuUy4gV2hhdCBJ IHdhcyBtZW50aW9uaW5nLCByZWdhcmRpbmcgYSBnb29kIHByYWN0aWNlIGlzIG1vc3RseQogICAg ICByZWxhdGVkIHRvIHRoZSBvcGVyYXRpb25zIG9mIGdsdXN0ZXIgbm90IGluc3RhbGxhdGlvbiBv cgogICAgICBkZXBsb3ltZW50LCBpLmUuIG5vdCB0aGUgY29uY2VwdHVhbCB1bmRlcnN0YW5kaW5n IG9mIGdsdXN0ZXIKICAgICAgKGNvbmNlcHR1YWxseSBpdCdzIGEgSkJPRCBzeXN0ZW0pLjxicj4K ICAgIDwvcD4KICAgIDxicj4KICAgIDxkaXYgY2xhc3M9Im1vei1jaXRlLXByZWZpeCI+T24gMDgv MDcvMjAxNyAwNTo0MSBQTSwgRkVSTkFORE8KICAgICAgRlJFRElBTkkgd3JvdGU6PGJyPgogICAg PC9kaXY+CiAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIgogICAgICBjaXRlPSJtaWQ6YzdhMWMy ZTEtNTdjMy05ZmE1LTA3MTAtZWJlZTNmM2ZhMDY5QHVweC5jb20iPgogICAgICA8bWV0YSBodHRw LWVxdWl2PSJDb250ZW50LVR5cGUiIGNvbnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11dGYtOCI+ CiAgICAgIDxwPlRoYW5rcyBmb3IgdGhlIGNsYXJpZmljYXRpb24gRXJla2xlLjwvcD4KICAgICAg PHA+SG93ZXZlciBJIGdldCBzdXJwcmlzZWQgd2l0aCB0aGlzIHdheSBvZiBvcGVyYXRpbmcgZnJv bQogICAgICAgIEdsdXN0ZXJGUyBhcyBpdCBhZGRzIGFub3RoZXIgbGF5ZXIgb2YgY29tcGxleGl0 eSB0byB0aGUgc3lzdGVtCiAgICAgICAgKGVpdGhlciBhIGhhcmR3YXJlIG9yIHNvZnR3YXJlIFJB SUQpIGJlZm9yZSB0aGUgZ2x1c3RlciBjb25maWcKICAgICAgICBhbmQgaW5jcmVhc2UgdGhlIHN5 c3RlbSdzIG92ZXJhbGwgY29zdHMuPGJyPgogICAgICA8L3A+CiAgICAgIDxwPkFuIGltcG9ydGFu dCBwb2ludCB0byBjb25zaWRlciBpczogSW4gUkFJRCBjb25maWd1cmF0aW9uIHlvdQogICAgICAg IGFscmVhZHkgaGF2ZSBzcGFjZSAnd2FzdGVkJyBpbiBvcmRlciB0byBidWlsZCByZWR1bmRhbmN5 IChlaXRoZXIKICAgICAgICBSQUlEIDEsIDUsIG9yIDYpLiBUaGVuIHdoZW4geW91IGhhdmUgR2x1 c3RlckZTIG9uIHRoZSB0b3Agb2YKICAgICAgICBzZXZlcmFsIFJBSURzIHlvdSBoYXZlIGFnYWlu IG1vcmUgZGF0YSByZXBsaWNhdGVkIHNvIHlvdSBlbmQgdXAKICAgICAgICB3aXRoIHRoZSBzYW1l IGRhdGEgY29uc3VtaW5nIG1vcmUgc3BhY2UgaW4gYSBncm91cCBvZiBkaXNrcyBhbmQKICAgICAg ICBhZ2FpbiBvbiB0aGUgdG9wIG9mIHNldmVyYWwgUkFJRHMgZGVwZW5kaW5nIG9uIHRoZSBHbHVz dGVyCiAgICAgICAgY29uZmlndXJhdGlvbiB5b3UgaGF2ZSAoaW4gYSBSQUlEIDEgY29uZmlnIHRo ZSBzYW1lIGRhdGEgaXMKICAgICAgICByZXBsaWNhdGVkIDQgdGltZXMpLjwvcD4KICAgICAgPHA+ WWV0IGFub3RoZXIgZG93bnNpZGUgb2YgaGF2aW5nIGEgUkFJRCAoc3BlY2lhbGx5IFJBSUQgNSBv ciA2KQogICAgICAgIGlzIHRoYXQgaXQgcmVkdWNlcyBjb25zaWRlcmFibHkgdGhlIHdyaXRlIHNw ZWVkcyBhcyBlYWNoIGdyb3VwCiAgICAgICAgb2YgZGlza3Mgd2lsbCBlbmQgdXAgaGF2aW5nIHRo ZSB3cml0ZSBzcGVlZCBvZiBhIHNpbmdsZSBkaXNrIGFzCiAgICAgICAgYWxsIG90aGVyIGRpc2tz IG9mIHRoYXQgZ3JvdXAgaGF2ZSB0byB3YWl0IGZvciBlYWNoIG90aGVyIHRvCiAgICAgICAgd3Jp dGUgYXMgd2VsbC48YnI+CiAgICAgIDwvcD4KICAgICAgPHA+VGhlcmVmb3JlIGlmIEdsdXN0ZXIg YWxyZWFkeSByZXBsaWNhdGVzIGRhdGEgd2h5IGRvZXMgaXQgY3JlYXRlCiAgICAgICAgdGhpcyBi aWcgcGFpbiB5b3UgbWVudGlvbmVkIGlmIHRoZSBkYXRhIGlzIHJlcGxpY2F0ZWQgc29tZXdoZXJl CiAgICAgICAgZWxzZSwgY2FuIHN0aWxsIGJlIHJldHJpZXZlZCB0byBib3RoIHNlcnZlIGNsaWVu dHMgYW5kCiAgICAgICAgcmVjb25zdHJ1Y3QgdGhlIGVxdWl2YWxlbnQgZGlzayB3aGVuIGl0IGlz IHJlcGxhY2VkID88L3A+CiAgICAgIDxwPkZlcm5hbmRvPGJyPgogICAgICA8L3A+CiAgICAgIDxi cj4KICAgICAgPGRpdiBjbGFzcz0ibW96LWNpdGUtcHJlZml4Ij5PbiAwNy8wOC8yMDE3IDEwOjI2 LCBFcmVrbGUgTWFncmFkemUKICAgICAgICB3cm90ZTo8YnI+CiAgICAgIDwvZGl2PgogICAgICA8 YmxvY2txdW90ZSB0eXBlPSJjaXRlIgogICAgICAgIGNpdGU9Im1pZDphYTgyOWQwNy1mYTc3LTNl ZDktMjUwMC1lMzNjYzAxNDE0YjZAcmVjb2dpemVyLmRlIj4KICAgICAgICA8bWV0YSBodHRwLWVx dWl2PSJDb250ZW50LVR5cGUiIGNvbnRlbnQ9InRleHQvaHRtbDsKICAgICAgICAgIGNoYXJzZXQ9 dXRmLTgiPgogICAgICAgIDxwPkhpIEZyZW5hbmRvLDwvcD4KICAgICAgICA8cD5IZXJlIGlzIG15 IGV4cGVyaWVuY2UsIGlmIHlvdSBjb25zaWRlciBhIHBhcnRpY3VsYXIgaGFyZAogICAgICAgICAg ZHJpdmUgYXMgYSBicmljayBmb3IgZ2x1c3RlciB2b2x1bWUgYW5kIGl0IGRpZXMsIGkuZS4gaXQK ICAgICAgICAgIGJlY29tZXMgbm90IGFjY2Vzc2libGUgaXQncyBhIGh1Z2UgaGFzc2xlIHRvIGRp c2NhcmQgdGhhdAogICAgICAgICAgYnJpY2sgYW5kIGV4Y2hhbmdlIHdpdGggYW5vdGhlciBvbmUs IHNpbmNlIGdsdXN0ZXIgc29tZSB0cmllcwogICAgICAgICAgdG8gYWNjZXNzIHRoYXQgYnJva2Vu IGJyaWNrIGFuZCBpdCdzIGNhdXNpbmcgKGF0IGxlYXN0IGl0CiAgICAgICAgICBjYXVzZSBmb3Ig bWUpIGEgYmlnIHBhaW4sIHRoZXJlZm9yZSBpdCdzIGJldHRlciB0byBoYXZlIGEgUkFJRAogICAg ICAgICAgYXMgYnJpY2ssIGkuZS4gaGF2ZSBSQUlEIDEgKG1pcnJvcmluZykgZm9yIGVhY2ggYnJp Y2ssIGluIHRoaXMKICAgICAgICAgIGNhc2UgaWYgdGhlIGRpc2sgaXMgZG93biB5b3UgY2FuIGVh c2lseSBleGNoYW5nZSBpdCBhbmQKICAgICAgICAgIHJlYnVpbGQgdGhlIFJBSUQgd2l0aG91dCBn b2luZyBvZmZsaW5lLCBpLmUgc3dpdGNoaW5nIG9mZiB0aGUKICAgICAgICAgIHZvbHVtZSBkb2lu ZyBicmljayBtYW5pcHVsYXRpb25zIGFuZCBzd2l0Y2hpbmcgaXQgYmFjayBvbi48YnI+CiAgICAg ICAgPC9wPgogICAgICAgIDxwPkNoZWVyczwvcD4KICAgICAgICA8cD5FcmVrbGU8YnI+CiAgICAg ICAgPC9wPgogICAgICAgIDxicj4KICAgICAgICA8ZGl2IGNsYXNzPSJtb3otY2l0ZS1wcmVmaXgi Pk9uIDA4LzA3LzIwMTcgMDM6MDQgUE0sIEZFUk5BTkRPCiAgICAgICAgICBGUkVESUFOSSB3cm90 ZTo8YnI+CiAgICAgICAgPC9kaXY+CiAgICAgICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSIKICAg ICAgICAgIGNpdGU9Im1pZDo2M2JhYzQ3Yi1hZmU2LTAyNTgtZDNkNy1lNTQ1YTUwMDRjMzBAdXB4 LmNvbSI+CiAgICAgICAgICA8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNvbnRlbnQ9 InRleHQvaHRtbDsKICAgICAgICAgICAgY2hhcnNldD11dGYtOCI+CiAgICAgICAgICA8cD5Gb3Ig YW55IFJBSUQgNSBvciA2IGNvbmZpZ3VyYXRpb24gSSBub3JtYWxseSBmb2xsb3cgYQogICAgICAg ICAgICBzaW1wbGUgZ29sZCBydWxlIHdoaWNoIGdhdmUgZ29vZCByZXN1bHRzIHNvIGZhcjo8YnI+ CiAgICAgICAgICAgIC0gdXAgdG8gNCBkaXNrcyBSQUlEIDU8YnI+CiAgICAgICAgICAgIC0gNSBv ciBtb3JlIGRpc2tzIFJBSUQgNjwvcD4KICAgICAgICAgIDxwPkhvd2V2ZXIgSSBkaWRuJ3QgcmVh bGx5IHVuZGVyc3RhbmQgd2VsbCB0aGUgcmVjb21tZW5kYXRpb24KICAgICAgICAgICAgdG8gdXNl IGFueSBSQUlEIHdpdGggR2x1c3RlckZTLiBJIGFsd2F5cyB0aG91Z2h0IHRoYXQKICAgICAgICAg ICAgR2x1c3RlRlMgbGlrZXMgdG8gd29yayBpbiBKQk9EIG1vZGUgYW5kIGNvbnRyb2wgdGhlIGRp c2tzCiAgICAgICAgICAgIChicmlja3MpIGRpcmVjdGxseSBzbyB5b3UgY2FuIGNyZWF0ZSB3aGF0 ZXZlciBkaXN0cmlidXRpb24KICAgICAgICAgICAgcnVsZSB5b3Ugd2lzaCwgYW5kIGlmIGEgc2lu Z2xlIGRpc2sgZmFpbHMgeW91IGp1c3QgcmVwbGFjZQogICAgICAgICAgICBpdCBhbmQgd2hpY2gg b2J2aW91c2x5IGhhdmUgdGhlIGRhdGEgcmVwbGljYXRlZCBmcm9tCiAgICAgICAgICAgIGFub3Ro ZXIuIFRoZSBvbmx5IGRvd25zaWRlIG9mIHVzaW5nIGluIHRoaXMgd2F5IGlzIHRoYXQgdGhlCiAg ICAgICAgICAgIHJlcGxpY2F0aW9uIGRhdGEgd2lsbCBiZSBmbG93IGFjY3Jvc3MgYWxsIHNlcnZl cnMgYnV0IHRoYXQKICAgICAgICAgICAgaXMgbm90IG11Y2ggYSBiaWcgaXNzdWUuPC9wPgogICAg ICAgICAgPHA+QW55b25lIGNhbiBlbGFib3JhdGUgYWJvdXQgVXNpbmcgUkFJRCArIEdsdXN0ZXJG UyBhbmQgSkJPRAogICAgICAgICAgICArIEdsdXN0ZXJGUy48L3A+CiAgICAgICAgICA8cD5UaGFu a3M8YnI+CiAgICAgICAgICAgIFJlZ2FyZHM8YnI+CiAgICAgICAgICAgIEZlcm5hbmRvPGJyPgog ICAgICAgICAgPC9wPgogICAgICAgICAgPGJyPgogICAgICAgICAgPGRpdiBjbGFzcz0ibW96LWNp dGUtcHJlZml4Ij5PbiAwNy8wOC8yMDE3IDAzOjQ2LCBEZXZpbiBBY29zdGEKICAgICAgICAgICAg d3JvdGU6PGJyPgogICAgICAgICAgPC9kaXY+CiAgICAgICAgICA8YmxvY2txdW90ZSB0eXBlPSJj aXRlIgpjaXRlPSJtaWQ6Q0FOQ0dLRXA0WEdzMFUrUXM3OGVFbXFDTnR2cExZLUF6amI1RGNHaFo5 eWlLVEJFRWZ3QG1haWwuZ21haWwuY29tIj4KICAgICAgICAgICAgPHN0eWxlPmJvZHl7Zm9udC1m YW1pbHk6SGVsdmV0aWNhLEFyaWFsO2ZvbnQtc2l6ZToxM3B4fTwvc3R5bGU+CiAgICAgICAgICAg IDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZvbnQiCiAgICAgICAgICAgICAgc3R5bGU9ImNvbG9yOnJn YigwLDAsMCk7bWFyZ2luOjBweCI+PGZvbnQgZmFjZT0iSW5wdXQKICAgICAgICAgICAgICAgIE1v bm8iPjxicj4KICAgICAgICAgICAgICA8L2ZvbnQ+PC9kaXY+CiAgICAgICAgICAgIDxkaXYgaWQ9 ImJsb29wX2N1c3RvbWZvbnQiCiAgICAgICAgICAgICAgc3R5bGU9ImNvbG9yOnJnYigwLDAsMCk7 bWFyZ2luOjBweCI+PGZvbnQgZmFjZT0iSW5wdXQKICAgICAgICAgICAgICAgIE1vbm8iPk1vYWNp ciw8L2ZvbnQ+PC9kaXY+CiAgICAgICAgICAgIDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZvbnQiCiAg ICAgICAgICAgICAgc3R5bGU9ImNvbG9yOnJnYigwLDAsMCk7bWFyZ2luOjBweCI+PGZvbnQgZmFj ZT0iSW5wdXQKICAgICAgICAgICAgICAgIE1vbm8iPjxicj4KICAgICAgICAgICAgICA8L2ZvbnQ+ PC9kaXY+CiAgICAgICAgICAgIDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZvbnQiCiAgICAgICAgICAg ICAgc3R5bGU9ImNvbG9yOnJnYigwLDAsMCk7bWFyZ2luOjBweCI+PGZvbnQgZmFjZT0iSW5wdXQK ICAgICAgICAgICAgICAgIE1vbm8iPkkgaGF2ZSByZWNlbnRseSBpbnN0YWxsZWQgbXVsdGlwbGUg UmVkIEhhdAogICAgICAgICAgICAgICAgVmlydHVhbGl6YXRpb24gaG9zdHMgZm9yIHNldmVyYWwg ZGlmZmVyZW50IGNvbXBhbmllcywKICAgICAgICAgICAgICAgIGFuZCBoYXZlIGRlYWx0IHdpdGgg dGhlIFJlZCBIYXQgU3VwcG9ydCBUZWFtIGluIGRlcHRoCiAgICAgICAgICAgICAgICBhYm91dCBv cHRpbWFsIGNvbmZpZ3VyYXRpb24gaW4gcmVnYXJkcyB0byBzZXR0aW5nIHVwCiAgICAgICAgICAg ICAgICBHbHVzdGVyRlMgbW9zdCBlZmZpY2llbnRseSBhbmQgSSB3YW50ZWQgdG8gc2hhcmUgd2l0 aAogICAgICAgICAgICAgICAgeW91IHdoYXQgSSBsZWFybmVkLjwvZm9udD48L2Rpdj4KICAgICAg ICAgICAgPGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9udCIKICAgICAgICAgICAgICBzdHlsZT0iY29s b3I6cmdiKDAsMCwwKTttYXJnaW46MHB4Ij48Zm9udCBmYWNlPSJJbnB1dAogICAgICAgICAgICAg ICAgTW9ubyI+PGJyPgogICAgICAgICAgICAgIDwvZm9udD48L2Rpdj4KICAgICAgICAgICAgPGRp diBpZD0iYmxvb3BfY3VzdG9tZm9udCIKICAgICAgICAgICAgICBzdHlsZT0iY29sb3I6cmdiKDAs MCwwKTttYXJnaW46MHB4Ij48Zm9udCBmYWNlPSJJbnB1dAogICAgICAgICAgICAgICAgTW9ubyI+ SW4gZ2VuZXJhbCBSZWQgSGF0IFZpcnR1YWxpemF0aW9uIHRlYW0gZnJvd25zIHVwb24KICAgICAg ICAgICAgICAgIHVzaW5nIGVhY2ggRElTSyBvZiB0aGUgc3lzdGVtIGFzIGp1c3QgYSBKQk9ELCBz dXJlIHRoZXJlCiAgICAgICAgICAgICAgICBpcyBzb21lIHByb3RlY3Rpb24gYnkgaGF2aW5nIHRo ZSBkYXRhIHJlcGxpY2F0ZWQsCiAgICAgICAgICAgICAgICBob3dldmVyLCB0aGUgcmVjb21tZW5k YXRpb24gaXMgdG8gdXNlIFJBSUQgNiAocHJlZmVycmVkKQogICAgICAgICAgICAgICAgb3IgUkFJ RC01LCBvciBhdCBsZWFzdCBSQUlELTEgYXQgdGhlIHZlcnkgbGVhc3QuPC9mb250PjwvZGl2Pgog ICAgICAgICAgICA8ZGl2IGlkPSJibG9vcF9jdXN0b21mb250IgogICAgICAgICAgICAgIHN0eWxl PSJjb2xvcjpyZ2IoMCwwLDApO21hcmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0CiAgICAgICAg ICAgICAgICBNb25vIj48YnI+CiAgICAgICAgICAgICAgPC9mb250PjwvZGl2PgogICAgICAgICAg ICA8ZGl2IGlkPSJibG9vcF9jdXN0b21mb250IiBzdHlsZT0ibWFyZ2luOjBweCI+PGZvbnQKICAg ICAgICAgICAgICAgIGZhY2U9IklucHV0IE1vbm8iPkhlcmUgaXMgdGhlwqBkaXJlY3QgcXVvdGUg ZnJvbSBSZWQgSGF0CiAgICAgICAgICAgICAgICB3aGVuIEkgYXNrZWQgYWJvdXQgUkFJRCBhbmQg QnJpY2tzOjwvZm9udD48L2Rpdj4KICAgICAgICAgICAgPGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9u dCIgc3R5bGU9Im1hcmdpbjowcHgiPjxmb250CiAgICAgICAgICAgICAgICBmYWNlPSJJbnB1dCBN b25vIj48aT48YnI+CiAgICAgICAgICAgICAgICA8L2k+PC9mb250PjwvZGl2PgogICAgICAgICAg ICA8ZGl2IGlkPSJibG9vcF9jdXN0b21mb250IiBzdHlsZT0ibWFyZ2luOjBweCI+PGZvbnQKICAg ICAgICAgICAgICAgIGZhY2U9IklucHV0IE1vbm8iPjxpPiJBIHR5cGljYWwgR2x1c3RlciBjb25m aWd1cmF0aW9uCiAgICAgICAgICAgICAgICAgIHdvdWxkIHVzZSBSQUlEIHVuZGVybmVhdGggdGhl IGJyaWNrcy4gUkFJRCA2IGlzIG1vc3QKICAgICAgICAgICAgICAgICAgdHlwaWNhbCBhcyBpdCBn aXZlcyB5b3UgMiBkaXNrIGZhaWx1cmUgcHJvdGVjdGlvbiwgYnV0CiAgICAgICAgICAgICAgICAg IFJBSUQgNSBjb3VsZCBiZSB1c2VkIHRvby7CoE9uY2UgeW91IGhhdmUgdGhlIFJBSURlZAogICAg ICAgICAgICAgICAgICBicmlja3MsIHlvdSdkIHRoZW4gYXBwbHkgdGhlIGRlc2lyZWQgcmVwbGlj YXRpb24gb24KICAgICAgICAgICAgICAgICAgdG9wIG9mIHRoYXQuIFRoZSBtb3N0IHBvcHVsYXIg d2F5IG9mIGRvaW5nIHRoaXMgd291bGQKICAgICAgICAgICAgICAgICAgYmUgZGlzdHJpYnV0ZWQg cmVwbGljYXRlZCB3aXRoIDJ4IHJlcGxpY2F0aW9uLsKgSW4KICAgICAgICAgICAgICAgICAgZ2Vu ZXJhbCB5b3UnbGwgZ2V0IGJldHRlciBwZXJmb3JtYW5jZSB3aXRoIGxhcmdlcgogICAgICAgICAg ICAgICAgICBicmlja3MuIDEyIGRyaXZlcyBpcyBvZnRlbiBhIHN3ZWV0IHNwb3QuIEFub3RoZXIK ICAgICAgICAgICAgICAgICAgb3B0aW9uIHdvdWxkIGJlIHRvIGNyZWF0ZSBhIHNlcGFyYXRlIHRp ZXIgdXNpbmcgYWxsCiAgICAgICAgICAgICAgICAgIFNTROKAmXMu4oCdwqA8L2k+PC9mb250Pjwv ZGl2PgogICAgICAgICAgICA8ZGl2IGlkPSJibG9vcF9jdXN0b21mb250IiBzdHlsZT0ibWFyZ2lu OjBweCI+PGJyPgogICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgPGRpdiBpZD0iYmxvb3Bf Y3VzdG9tZm9udCIgc3R5bGU9Im1hcmdpbjowcHgiPjxmb250CiAgICAgICAgICAgICAgICBmYWNl PSJJbnB1dCBNb25vIj48aT5JbiBvcmRlciB0byBTU0TCoHRpZXJpbmcgZnJvbSBteQogICAgICAg ICAgICAgICAgICB1bmRlcnN0YW5kaW5nIHlvdSB3b3VsZCBuZWVkIDEgeCBOVk1lIGRyaXZlIGlu IGVhY2gKICAgICAgICAgICAgICAgICAgc2VydmVyLCBvciA0IHggU1NEIGhvdCB0aWVyIChpdCBu ZWVkcyB0byBiZQogICAgICAgICAgICAgICAgICBkaXN0cmlidXRlZCwgcmVwbGljYXRlZCBmb3Ig dGhlIGhvdCB0aWVyIGlmIG5vdCB1c2luZwogICAgICAgICAgICAgICAgICBOVk1FKS4gU28gd2l0 aCB5b3Ugb25seSBoYXZpbmcgMSBTU0QgZHJpdmUgaW4gZWFjaAogICAgICAgICAgICAgICAgICBz ZXJ2ZXIswqBJ4oCZZCBzdWdnZXN0IG1heWJlIGxvb2tpbmcgaW50byB0aGUgTlZNRQogICAgICAg ICAgICAgICAgICBvcHRpb24uwqA8L2k+PC9mb250PjwvZGl2PgogICAgICAgICAgICA8ZGl2IGlk PSJibG9vcF9jdXN0b21mb250IiBzdHlsZT0ibWFyZ2luOjBweCI+PGZvbnQKICAgICAgICAgICAg ICAgIGZhY2U9IklucHV0IE1vbm8iPjxpPjxicj4KICAgICAgICAgICAgICAgIDwvaT48L2ZvbnQ+ PC9kaXY+CiAgICAgICAgICAgIDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJtYXJn aW46MHB4Ij48Zm9udAogICAgICAgICAgICAgICAgZmFjZT0iSW5wdXQgTW9ubyI+PGk+U2luY2Ug eW91ciB1c2luZyBvbmx5IDMtc2VydmVycywKICAgICAgICAgICAgICAgICAgd2hhdCBJ4oCZZCBw cm9iYWJseSBzdWdnZXN0IGlzIHRvIGRvICgyIFJlcGxpY2FzICsKICAgICAgICAgICAgICAgICAg QXJiaXRlciBOb2RlKSwgdGhpcyBzZXR1cCBhY3R1YWxseSBkb2VzbuKAmXQgcmVxdWlyZSB0aGUK ICAgICAgICAgICAgICAgICAgM3JkIHNlcnZlciB0byBoYXZlIGJpZyBkcml2ZXMgYXQgYWxsIGFz IGl0IG9ubHkgc3RvcmVzCiAgICAgICAgICAgICAgICAgIG1ldGEtZGF0YSBhYm91dCB0aGUgZmls ZXMgYW5kIG5vdCBhY3R1YWxseSBhIGZ1bGwKICAgICAgICAgICAgICAgICAgY29weS7CoDwvaT48 L2ZvbnQ+PC9kaXY+CiAgICAgICAgICAgIDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxl PSJtYXJnaW46MHB4Ij48Zm9udAogICAgICAgICAgICAgICAgZmFjZT0iSW5wdXQgTW9ubyI+PGk+ PGJyPgogICAgICAgICAgICAgICAgPC9pPjwvZm9udD48L2Rpdj4KICAgICAgICAgICAgPGRpdiBp ZD0iYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9Im1hcmdpbjowcHgiPjxmb250CiAgICAgICAgICAg ICAgICBmYWNlPSJJbnB1dCBNb25vIj48aT5QbGVhc2Ugc2VlIHRoZSBhdHRhY2hlZCBkb2N1bWVu dAogICAgICAgICAgICAgICAgICB0aGF0IHdhcyBnaXZlbiB0byBtZSBieSBSZWQgSGF0IHRvIGdl dCBtb3JlCiAgICAgICAgICAgICAgICAgIGluZm9ybWF0aW9uIG9uIHRoaXMuIEhvcGUgdGhpcyBp bmZvcm1hdGlvbiBoZWxwcyB5b3UuPC9pPjwvZm9udD48L2Rpdj4KICAgICAgICAgICAgPGRpdiBp ZD0iYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9Im1hcmdpbjowcHgiPjxmb250CiAgICAgICAgICAg ICAgICBmYWNlPSJJbnB1dCBNb25vIj48aT48YnI+CiAgICAgICAgICAgICAgICA8L2k+PC9mb250 PjwvZGl2PgogICAgICAgICAgICA8YnI+CiAgICAgICAgICAgIDxkaXYgaWQ9ImJsb29wX3NpZ25f MTUwMjA4NzM3NjcyNTQ2OTE4NCIgY2xhc3M9ImJsb29wX3NpZ24iPjxzcGFuCiAgICAgICAgICAg ICAgICBzdHlsZT0iZm9udC1mYW1pbHk6J2hlbHZldGljYQogICAgICAgICAgICAgICAgTmV1ZScs aGVsdmV0aWNhO2ZvbnQtc2l6ZToxNHB4Ij4tLTwvc3Bhbj48YnIKICAgICAgICAgICAgICAgIHN0 eWxlPSJmb250LWZhbWlseTonaGVsdmV0aWNhCiAgICAgICAgICAgICAgICBOZXVlJyxoZWx2ZXRp Y2E7Zm9udC1zaXplOjE0cHgiPgogICAgICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX3NpZ25h dHVyZSIgc3R5bGU9ImZvbnQtZmFtaWx5OidoZWx2ZXRpY2EKICAgICAgICAgICAgICAgIE5ldWUn LGhlbHZldGljYTtmb250LXNpemU6MTRweCI+CiAgICAgICAgICAgICAgICA8ZGl2IGRpcj0ibHRy Ij4KICAgICAgICAgICAgICAgICAgPGRpdj48YnI+CiAgICAgICAgICAgICAgICAgIDwvZGl2Pgog ICAgICAgICAgICAgICAgICA8ZGl2PkRldmluIEFjb3N0YSwgUkhDQSwgUkhWQ0E8L2Rpdj4KICAg ICAgICAgICAgICAgICAgPGRpdj5SZWQgSGF0IENlcnRpZmllZCBBcmNoaXRlY3Q8L2Rpdj4KICAg ICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICA8L2Rp dj4KICAgICAgICAgICAgPGJyPgogICAgICAgICAgICA8cCBjbGFzcz0iYWlybWFpbF9vbiI+T24g QXVndXN0IDYsIDIwMTcgYXQgNzoyOToyOSBQTSwKICAgICAgICAgICAgICBNb2FjaXIgRmVycmVp cmEgKDxhCiAgICAgICAgICAgICAgICBocmVmPSJtYWlsdG86bW9hY2lyZmVycmVpcmFAaG90bWFp bC5jb20iCiAgICAgICAgICAgICAgICBtb3otZG8tbm90LXNlbmQ9InRydWUiPm1vYWNpcmZlcnJl aXJhQGhvdG1haWwuY29tPC9hPikKICAgICAgICAgICAgICB3cm90ZTo8L3A+CiAgICAgICAgICAg IDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiIGNsYXNzPSJjbGVhbl9icSI+PHNwYW4+CiAgICAgICAg ICAgICAgICA8ZGl2IGRpcj0ibHRyIj4KICAgICAgICAgICAgICAgICAgPGRpdj4KICAgICAgICAg ICAgICAgICAgICA8dGl0bGU+PC90aXRsZT4KICAgICAgICAgICAgICAgICAgICA8ZGl2IGlkPSJk aXZ0YWdkZWZhdWx0d3JhcHBlciIKc3R5bGU9ImZvbnQtc2l6ZToxMnB0O2NvbG9yOiMwMDAwMDA7 Zm9udC1mYW1pbHk6Q2FsaWJyaSxIZWx2ZXRpY2Esc2Fucy1zZXJpZiIKICAgICAgICAgICAgICAg ICAgICAgIGRpcj0ibHRyIj4KICAgICAgICAgICAgICAgICAgICAgIDxwPjxzcGFuPkkgYW0gd2ls bGluZyB0byBhc3NlbWJsZSBhIG9WaXJ0ICJwb2QiLAogICAgICAgICAgICAgICAgICAgICAgICAg IG1hZGUgb2YgMyBzZXJ2ZXJzLCBlYWNoIHdpdGggMiBDUFUgc29ja2V0cyBvZgogICAgICAgICAg ICAgICAgICAgICAgICAgIDEyIGNvcmVzLCAyNTZHQiBSQU0sIDcgSEREIDEwSywgMSBTU0QuIFRo ZQogICAgICAgICAgICAgICAgICAgICAgICAgIGlkZWEgaXMgdG8gdXNlIEdsdXN0ZXJGUyB0byBw cm92aWRlIEhBIGZvciB0aGUKICAgICAgICAgICAgICAgICAgICAgICAgICBWTXMuIFRoZSAzIHNl cnZlcnMgaGF2ZSBhIGR1YWwgNDBHYiBOSUMgYW5kIGEKICAgICAgICAgICAgICAgICAgICAgICAg ICBkdWFsIDEwR2IgTklDLiBTbyBteSBpbnRlbnRpb24gaXMgdG8gY3JlYXRlIGEKICAgICAgICAg ICAgICAgICAgICAgICAgICBsb29wIGxpa2UgYSBzZXJ2ZXIgdHJpYW5nbGUgdXNpbmcgdGhlIDQw R2IKICAgICAgICAgICAgICAgICAgICAgICAgICBOSUNzIGZvciB2aXJ0dWFsaXphdGlvbiBmaWxl cyAoVk1zIC5xY293MikKICAgICAgICAgICAgICAgICAgICAgICAgICBhY2Nlc3MgYW5kIHRvIG1v dmUgVk1zIGFyb3VuZCB0aGUgcG9kIChlYXN0CiAgICAgICAgICAgICAgICAgICAgICAgICAgL3dl c3QgdHJhZmZpYykgd2hpbGUgdXNpbmcgdGhlIDEwR2IgaW50ZXJmYWNlcwogICAgICAgICAgICAg ICAgICAgICAgICAgIGZvciBnaXZpbmcgc2VydmljZXMgdG8gdGhlIG91dHNpZGUgd29ybGQKICAg ICAgICAgICAgICAgICAgICAgICAgICAobm9ydGgvc291dGggdHJhZmZpYykuPC9zcGFuPjwvcD4K ICAgICAgICAgICAgICAgICAgICAgIDxwPjxicj4KICAgICAgICAgICAgICAgICAgICAgIDwvcD4K ICAgICAgICAgICAgICAgICAgICAgIDxwPlRoaXMgc2FpZCwgbXkgZmlyc3QgcXVlc3Rpb24gaXM6 IEhvdyBzaG91bGQgSQogICAgICAgICAgICAgICAgICAgICAgICBkZXBsb3kgR2x1c3RlckZTIGlu IHN1Y2ggb1ZpcnQgc2NlbmFyaW8/IE15CiAgICAgICAgICAgICAgICAgICAgICAgIHF1ZXN0aW9u cyBhcmU6PC9wPgogICAgICAgICAgICAgICAgICAgICAgPHA+PGJyPgogICAgICAgICAgICAgICAg ICAgICAgPC9wPgogICAgICAgICAgICAgICAgICAgICAgPHA+MSAtIFNob3VsZCBJIGNyZWF0ZSAz IFJBSUQgKGkuZS46IFJBSUQgNSksIG9uZQogICAgICAgICAgICAgICAgICAgICAgICBvbiBlYWNo IG9WaXJ0IG5vZGUsIGFuZCB0aGVuIGNyZWF0ZSBhIEdsdXN0ZXJGUwogICAgICAgICAgICAgICAg ICAgICAgICB1c2luZyB0aGVtPzwvcD4KICAgICAgICAgICAgICAgICAgICAgIDxwPjIgLSBJbnN0 ZWFkLCBzaG91bGQgSSBjcmVhdGUgYSBKQk9EIGFycmF5IG1hZGUKICAgICAgICAgICAgICAgICAg ICAgICAgb2YgYWxsIHNlcnZlcidzIGRpc2tzPzwvcD4KICAgICAgICAgICAgICAgICAgICAgIDxw PjMgLSBXaGF0IGlzIHRoZSBiZXN0IEdsdXN0ZXIgY29uZmlndXJhdGlvbiB0bwogICAgICAgICAg ICAgICAgICAgICAgICBwcm92aWRlIGZvciBIQSB3aGlsZSBub3QgY29uc3VtaW5nIHRvbyBtdWNo IGRpc2sKICAgICAgICAgICAgICAgICAgICAgICAgc3BhY2U/PGJyPgogICAgICAgICAgICAgICAg ICAgICAgPC9wPgogICAgICAgICAgICAgICAgICAgICAgPHA+NCAtIERvZXMgYSBvVmlydCBoeXBl cnZpc29yIHBvZCBsaWtlIEkgYW0KICAgICAgICAgICAgICAgICAgICAgICAgcGxhbm5pbmcgdG8g YnVpbGQsIGFuZCB0aGUgdmlydHVhbGl6YXRpb24KICAgICAgICAgICAgICAgICAgICAgICAgZW52 aXJvbm1lbnQsIGJlbmVmaXRzIGZyb20gdGllcmluZyB3aGVuIHVzaW5nIGEKICAgICAgICAgICAg ICAgICAgICAgICAgU1NEIGRpc2s/IEFuZCB5ZXMsIHdpbGwgR2x1c3RlciBkbyBpdCBieSBkZWZh dWx0CiAgICAgICAgICAgICAgICAgICAgICAgIG9yIEkgaGF2ZSB0byBjb25maWd1cmUgaXQgdG8g ZG8gc28/PC9wPgogICAgICAgICAgICAgICAgICAgICAgPHA+PGJyPgogICAgICAgICAgICAgICAg ICAgICAgPC9wPgogICAgICAgICAgICAgICAgICAgICAgPHA+QXQgdGhlIGJvdHRvbSBsaW5lLCB3 aGF0IGlzIHRoZSBnb29kIHByYWN0aWNlCiAgICAgICAgICAgICAgICAgICAgICAgIGZvciB1c2lu ZyBHbHVzdGVyRlMgaW4gc21hbGwgcG9kcyBmb3IKICAgICAgICAgICAgICAgICAgICAgICAgZW50 ZXJwcmlzZXM/PGJyPgogICAgICAgICAgICAgICAgICAgICAgPC9wPgogICAgICAgICAgICAgICAg ICAgICAgPHA+PGJyPgogICAgICAgICAgICAgICAgICAgICAgPC9wPgogICAgICAgICAgICAgICAg ICAgICAgPHA+WW91IG9waW5pb24vZmVlZGJhY2sgd2lsbCBiZSByZWFsbHkKICAgICAgICAgICAg ICAgICAgICAgICAgYXBwcmVjaWF0ZWQhPC9wPgogICAgICAgICAgICAgICAgICAgICAgPHA+TW9h Y2lyPGJyPgogICAgICAgICAgICAgICAgICAgICAgPC9wPgogICAgICAgICAgICAgICAgICAgIDwv ZGl2PgogICAgICAgICAgICAgICAgICAgIF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fIDxicj4KICAgICAgICAgICAgICAgICAgICBVc2VycyBtYWlsaW5nIGxp c3QgPGJyPgogICAgICAgICAgICAgICAgICAgIDxhIGhyZWY9Im1haWx0bzpVc2Vyc0BvdmlydC5v cmciCiAgICAgICAgICAgICAgICAgICAgICBtb3otZG8tbm90LXNlbmQ9InRydWUiPlVzZXJzQG92 aXJ0Lm9yZzwvYT4gPGJyPgogICAgICAgICAgICAgICAgICAgIDxhCiAgICAgICAgICAgICAgICAg ICAgICBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMi CiAgICAgICAgICAgICAgICAgICAgICBtb3otZG8tbm90LXNlbmQ9InRydWUiPmh0dHA6Ly9saXN0 cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VyczwvYT4KICAgICAgICAgICAgICAgICAg ICA8YnI+CiAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgPC9kaXY+CiAg ICAgICAgICAgICAgPC9zcGFuPjwvYmxvY2txdW90ZT4KICAgICAgICAgICAgPGJyPgogICAgICAg ICAgICA8ZmllbGRzZXQgY2xhc3M9Im1pbWVBdHRhY2htZW50SGVhZGVyIj48L2ZpZWxkc2V0Pgog ICAgICAgICAgICA8YnI+CiAgICAgICAgICAgIDxwcmUgd3JhcD0iIj5fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpVc2VycyBtYWlsaW5nIGxpc3QKPGEgY2xh c3M9Im1vei10eHQtbGluay1hYmJyZXZpYXRlZCIgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9y ZyIgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5Vc2Vyc0BvdmlydC5vcmc8L2E+CjxhIGNsYXNzPSJt b3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1h bi9saXN0aW5mby91c2VycyIgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5odHRwOi8vbGlzdHMub3Zp cnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnM8L2E+CjwvcHJlPgogICAgICAgICAgPC9ibG9j a3F1b3RlPgogICAgICAgICAgPGJyPgogICAgICAgICAgPGJyPgogICAgICAgICAgPGZpZWxkc2V0 IGNsYXNzPSJtaW1lQXR0YWNobWVudEhlYWRlciI+PC9maWVsZHNldD4KICAgICAgICAgIDxicj4K ICAgICAgICAgIDxwcmUgd3JhcD0iIj5fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fXwpVc2VycyBtYWlsaW5nIGxpc3QKPGEgY2xhc3M9Im1vei10eHQtbGluay1h YmJyZXZpYXRlZCIgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgbW96LWRvLW5vdC1zZW5k PSJ0cnVlIj5Vc2Vyc0BvdmlydC5vcmc8L2E+CjxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRl eHQiIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyIg bW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlz dGluZm8vdXNlcnM8L2E+CjwvcHJlPgogICAgICAgIDwvYmxvY2txdW90ZT4KICAgICAgICA8YnI+ CiAgICAgIDwvYmxvY2txdW90ZT4KICAgICAgPGJyPgogICAgPC9ibG9ja3F1b3RlPgogICAgPGJy PgogICAgPHByZSBjbGFzcz0ibW96LXNpZ25hdHVyZSIgY29scz0iNzIiPi0tIApSZWNvZ2l6ZXIg R3JvdXAgR21iSAoKRHIucmVyLm5hdC4gRXJla2xlIE1hZ3JhZHplCkxlYWQgQmlnIERhdGEgRW5n aW5lZXJpbmcgJmFtcDsgRGV2T3BzClJoZWlud2Vya2FsbGVlIDIsIDUzMjI3IEJvbm4KVGVsOiAr NDkgMjI4IDI5OTc0NTU1CgpFLU1haWwgPGEgY2xhc3M9Im1vei10eHQtbGluay1hYmJyZXZpYXRl ZCIgaHJlZj0ibWFpbHRvOmVyZWtsZS5tYWdyYWR6ZUByZWNvZ2l6ZXIuZGUiPmVyZWtsZS5tYWdy YWR6ZUByZWNvZ2l6ZXIuZGU8L2E+CldlYjogPGEgY2xhc3M9Im1vei10eHQtbGluay1hYmJyZXZp YXRlZCIgaHJlZj0iaHR0cDovL3d3dy5yZWNvZ2l6ZXIuY29tIj53d3cucmVjb2dpemVyLmNvbTwv YT4KIApSZWNvZ2l6ZXIgYXVmIExpbmtlZEluIDxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRl eHQiIGhyZWY9Imh0dHBzOi8vd3d3LmxpbmtlZGluLmNvbS9jb21wYW55LWJldGEvMTAwMzkxODIv Ij5odHRwczovL3d3dy5saW5rZWRpbi5jb20vY29tcGFueS1iZXRhLzEwMDM5MTgyLzwvYT4KRm9s Z2VuIFNpZSB1bnMgYXVmIFR3aXR0ZXIgPGEgY2xhc3M9Im1vei10eHQtbGluay1mcmVldGV4dCIg aHJlZj0iaHR0cHM6Ly90d2l0dGVyLmNvbS9yZWNvZ2l6ZXIiPmh0dHBzOi8vdHdpdHRlci5jb20v cmVjb2dpemVyPC9hPgogCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tClJlY29naXplciBHcm91cCBHbWJICkdlc2Now6RmdHNm w7xocmVyOiBPbGl2ZXIgSGFiaXNjaCwgQ2Fyc3RlbiBLcmV1dHplCkhhbmRlbHNyZWdpc3Rlcjog QW10c2dlcmljaHQgQm9ubiBIUkIgMjA3MjQKU2l0eiBkZXIgR2VzZWxsc2NoYWZ0OiBCb25uOyBV U3QtSUQtTnIuOiBERTI5NDE5NTk5MwogCkRpZXNlIEUtTWFpbCBlbnRow6RsdCB2ZXJ0cmF1bGlj aGUgdW5kL29kZXIgcmVjaHRsaWNoIGdlc2Now7x0enRlIEluZm9ybWF0aW9uZW4uCldlbm4gU2ll IG5pY2h0IGRlciByaWNodGlnZSBBZHJlc3NhdCBzaW5kIG9kZXIgZGllc2UgRS1NYWlsIGlycnTD vG1saWNoIGVyaGFsdGVuIGhhYmVuLAppbmZvcm1pZXJlbiBTaWUgYml0dGUgc29mb3J0IGRlbiBB YnNlbmRlciB1bmQgbMO2c2NoZW4gU2llIGRpZXNlIE1haWwuCkRhcyB1bmVybGF1YnRlIEtvcGll cmVuIHNvd2llIGRpZSB1bmJlZnVndGUgV2VpdGVyZ2FiZSBkaWVzZXIgTWFpbCB1bmQgZGVyIGRh cmluIGVudGhhbHRlbmVuIEluZm9ybWF0aW9uZW4gaXN0IG5pY2h0IGdlc3RhdHRldC48L3ByZT4K ICA8L2JvZHk+CjwvaHRtbD4KCi0tLS0tLS0tLS0tLS0tQTAwODFFMDdGMzE0Q0Y5QjY4N0REQTg2 LS0K --===============5337389779839608455==-- From erekle.magradze at recogizer.de Mon Aug 7 20:07:05 2017 Content-Type: multipart/mixed; boundary="===============3381618162231106505==" MIME-Version: 1.0 From: Erekle Magradze To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Mon, 07 Aug 2017 22:07:02 +0200 Message-ID: In-Reply-To: c7a1c2e1-57c3-9fa5-0710-ebee3f3fa069@upx.com --===============3381618162231106505== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------0DC09401E4EDC25E68CD6208 Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 8bit Hi Fernando (sorry for misspelling your name, I used a different keyboard), So let's go with the following scenarios: 1. Let's say you have two servers (replication factor is 2), i.e. two = bricks per volume, in this case it is strongly recommended to have the = arbiter node, the metadata storage that will guarantee avoiding the = split brain situation, in this case for arbiter you don't even need a = disk with lots of space, it's enough to have a tiny ssd but hosted on a = separate server. Advantage of such setup is that you don't need the RAID = 1 for each brick, you have the metadata information stored in arbiter = node and brick replacement is easy. 2. If you have odd number of bricks (let's say 3, i.e. replication = factor is 3) in your volume and you didn't create the arbiter node as = well as you didn't configure the quorum, in this case the entire load = for keeping the consistency of the volume resides on all 3 servers, each = of them is important and each brick contains key information, they need = to cross-check each other (that's what people usually do with the first = try of gluster :) ), in this case replacing a brick is a big pain and in = this case RAID 1 is a good option to have (that's the disadvantage, i.e. = loosing the space and not having the JBOD option) advantage is that you = don't have the to have additional arbiter node. 3. You have odd number of bricks and configured arbiter node, in this = case you can easily go with JBOD, however a good practice would be to = have a RAID 1 for arbiter disks (tiny 128GB SSD-s ar perfectly = sufficient for volumes with 10s of TB-s in size.) That's basically it The rest about the reliability and setup scenarios you can find in = gluster documentation, especially look for quorum and arbiter node = configs+options. Cheers Erekle P.S. What I was mentioning, regarding a good practice is mostly related = to the operations of gluster not installation or deployment, i.e. not = the conceptual understanding of gluster (conceptually it's a JBOD system). On 08/07/2017 05:41 PM, FERNANDO FREDIANI wrote: > > Thanks for the clarification Erekle. > > However I get surprised with this way of operating from GlusterFS as = > it adds another layer of complexity to the system (either a hardware = > or software RAID) before the gluster config and increase the system's = > overall costs. > > An important point to consider is: In RAID configuration you already = > have space 'wasted' in order to build redundancy (either RAID 1, 5, or = > 6). Then when you have GlusterFS on the top of several RAIDs you have = > again more data replicated so you end up with the same data consuming = > more space in a group of disks and again on the top of several RAIDs = > depending on the Gluster configuration you have (in a RAID 1 config = > the same data is replicated 4 times). > > Yet another downside of having a RAID (specially RAID 5 or 6) is that = > it reduces considerably the write speeds as each group of disks will = > end up having the write speed of a single disk as all other disks of = > that group have to wait for each other to write as well. > > Therefore if Gluster already replicates data why does it create this = > big pain you mentioned if the data is replicated somewhere else, can = > still be retrieved to both serve clients and reconstruct the = > equivalent disk when it is replaced ? > > Fernando > > > On 07/08/2017 10:26, Erekle Magradze wrote: >> >> Hi Frenando, >> >> Here is my experience, if you consider a particular hard drive as a = >> brick for gluster volume and it dies, i.e. it becomes not accessible = >> it's a huge hassle to discard that brick and exchange with another = >> one, since gluster some tries to access that broken brick and it's = >> causing (at least it cause for me) a big pain, therefore it's better = >> to have a RAID as brick, i.e. have RAID 1 (mirroring) for each brick, = >> in this case if the disk is down you can easily exchange it and = >> rebuild the RAID without going offline, i.e switching off the volume = >> doing brick manipulations and switching it back on. >> >> Cheers >> >> Erekle >> >> >> On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote: >>> >>> For any RAID 5 or 6 configuration I normally follow a simple gold = >>> rule which gave good results so far: >>> - up to 4 disks RAID 5 >>> - 5 or more disks RAID 6 >>> >>> However I didn't really understand well the recommendation to use = >>> any RAID with GlusterFS. I always thought that GlusteFS likes to = >>> work in JBOD mode and control the disks (bricks) directlly so you = >>> can create whatever distribution rule you wish, and if a single disk = >>> fails you just replace it and which obviously have the data = >>> replicated from another. The only downside of using in this way is = >>> that the replication data will be flow accross all servers but that = >>> is not much a big issue. >>> >>> Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS. >>> >>> Thanks >>> Regards >>> Fernando >>> >>> >>> On 07/08/2017 03:46, Devin Acosta wrote: >>>> >>>> Moacir, >>>> >>>> I have recently installed multiple Red Hat Virtualization hosts for = >>>> several different companies, and have dealt with the Red Hat = >>>> Support Team in depth about optimal configuration in regards to = >>>> setting up GlusterFS most efficiently and I wanted to share with = >>>> you what I learned. >>>> >>>> In general Red Hat Virtualization team frowns upon using each DISK = >>>> of the system as just a JBOD, sure there is some protection by = >>>> having the data replicated, however, the recommendation is to use = >>>> RAID 6 (preferred) or RAID-5, or at least RAID-1 at the very least. >>>> >>>> Here is the direct quote from Red Hat when I asked about RAID and = >>>> Bricks: >>>> / >>>> / >>>> /"A typical Gluster configuration would use RAID underneath the = >>>> bricks. RAID 6 is most typical as it gives you 2 disk failure = >>>> protection, but RAID 5 could be used too. Once you have the RAIDed = >>>> bricks, you'd then apply the desired replication on top of that. = >>>> The most popular way of doing this would be distributed replicated = >>>> with 2x replication. In general you'll get better performance with = >>>> larger bricks. 12 drives is often a sweet spot. Another option = >>>> would be to create a separate tier using all SSD=E2=80=99s.=E2=80=9D / >>>> >>>> /In order to SSD tiering from my understanding you would need 1 x = >>>> NVMe drive in each server, or 4 x SSD hot tier (it needs to be = >>>> distributed, replicated for the hot tier if not using NVME). So = >>>> with you only having 1 SSD drive in each server, I=E2=80=99d suggest m= aybe = >>>> looking into the NVME option. / >>>> / >>>> / >>>> /Since your using only 3-servers, what I=E2=80=99d probably suggest is= to = >>>> do (2 Replicas + Arbiter Node), this setup actually doesn=E2=80=99t re= quire = >>>> the 3rd server to have big drives at all as it only stores = >>>> meta-data about the files and not actually a full copy. / >>>> / >>>> / >>>> /Please see the attached document that was given to me by Red Hat = >>>> to get more information on this. Hope this information helps you./ >>>> / >>>> / >>>> >>>> -- >>>> >>>> Devin Acosta, RHCA, RHVCA >>>> Red Hat Certified Architect >>>> >>>> On August 6, 2017 at 7:29:29 PM, Moacir Ferreira = >>>> (moacirferreira(a)hotmail.com ) w= rote: >>>> >>>>> I am willing to assemble a oVirt "pod", made of 3 servers, each = >>>>> with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The = >>>>> idea is to use GlusterFS to provide HA for the VMs. The 3 servers = >>>>> have a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to = >>>>> create a loop like a server triangle using the 40Gb NICs for = >>>>> virtualization files (VMs .qcow2) access and to move VMs around = >>>>> the pod (east /west traffic) while using the 10Gb interfaces for = >>>>> giving services to the outside world (north/south traffic). >>>>> >>>>> >>>>> This said, my first question is: How should I deploy GlusterFS in = >>>>> such oVirt scenario? My questions are: >>>>> >>>>> >>>>> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, = >>>>> and then create a GlusterFS using them? >>>>> >>>>> 2 - Instead, should I create a JBOD array made of all server's disks? >>>>> >>>>> 3 - What is the best Gluster configuration to provide for HA while = >>>>> not consuming too much disk space? >>>>> >>>>> 4 - Does a oVirt hypervisor pod like I am planning to build, and = >>>>> the virtualization environment, benefits from tiering when using a = >>>>> SSD disk? And yes, will Gluster do it by default or I have to = >>>>> configure it to do so? >>>>> >>>>> >>>>> At the bottom line, what is the good practice for using GlusterFS = >>>>> in small pods for enterprises? >>>>> >>>>> >>>>> You opinion/feedback will be really appreciated! >>>>> >>>>> Moacir >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >> > -- = Recogizer Group GmbH Dr.rer.nat. Erekle Magradze Lead Big Data Engineering & DevOps Rheinwerkallee 2, 53227 Bonn Tel: +49 228 29974555 E-Mail erekle.magradze(a)recogizer.de Web: www.recogizer.com = Recogizer auf LinkedIn https://www.linkedin.com/company-beta/10039182/ Folgen Sie uns auf Twitter https://twitter.com/recogizer = ----------------------------------------------------------------- Recogizer Group GmbH Gesch=C3=A4ftsf=C3=BChrer: Oliver Habisch, Carsten Kreutze Handelsregister: Amtsgericht Bonn HRB 20724 Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993 = Diese E-Mail enth=C3=A4lt vertrauliche und/oder rechtlich gesch=C3=BCtzte I= nformationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=C3=BCmlich= erhalten haben, informieren Sie bitte sofort den Absender und l=C3=B6schen Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der = darin enthaltenen Informationen ist nicht gestattet. --------------0DC09401E4EDC25E68CD6208 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit
Hi Fernando (sorry for misspelling your name, I used a different keyboard),

So let's go with the following scenarios:

1. Let's say you have two servers (replication factor is 2), i.e. two bricks per volume, in this case it is strongly recommended to have the arbiter node, the metadata storage that will guarantee avoiding the split brain situation, in this case for arbiter you don't even need a disk with lots of space, it's enough to have a tiny ssd but hosted on a separate server. Advantage of such setup is that you don't need the RAID 1 for each brick, you have the metadata information stored in arbiter node and brick replacement is easy.

2. If you have odd number of bricks (let's say 3, i.e. replication factor is 3) in your volume and you didn't create the arbiter node as well as you didn't configure the quorum, in this case the entire load for keeping the consistency of the volume resides on all 3 servers, each of them is important and each brick contains key information, they need to cross-check each other (that's what people usually do with the first try of gluster :) ), in this case replacing a brick is a big pain and in this case RAID 1 is a good option to have (that's the disadvantage, i.e. loosing the space and not having the JBOD option) advantage is that you don't have the to have additional arbiter node.

3. You have odd number of bricks and configured arbiter node, in this case you can easily go with JBOD, however a good practice would be to have a RAID 1 for arbiter disks (tiny 128GB SSD-s ar perfectly sufficient for volumes with 10s of TB-s in size.)

That's basically it

The rest about the reliability and setup scenarios you can find in gluster documentation, especially look for quorum and arbiter node configs+options.

Cheers

Erekle
P.S. What I was mentioning, regarding a good practice is mostly related to the operations of gluster not installation or deployment, i.e. not the conceptual understanding of gluster (conceptually it's a JBOD system).

On 08/07/2017 05:41 PM, FERNANDO FREDIANI wrote:

Thanks for the clarification Erekle.

However I get surprised with this way of operating from GlusterFS as it adds another layer of complexity to the system (either a hardware or software RAID) before the gluster config and increase the system's overall costs.

An important point to consider is: In RAID configuration you already have space 'wasted' in order to build redundancy (either RAID 1, 5, or 6). Then when you have GlusterFS on the top of several RAIDs you have again more data replicated so you end up with the same data consuming more space in a group of disks and again on the top of several RAIDs depending on the Gluster configuration you have (in a RAID 1 config the same data is replicated 4 times).

Yet another downside of having a RAID (specially RAID 5 or 6) is that it reduces considerably the write speeds as each group of disks will end up having the write speed of a single disk as all other disks of that group have to wait for each other to write as well.

Therefore if Gluster already replicates data why does it create this big pain you mentioned if the data is replicated somewhere else, can still be retrieved to both serve clients and reconstruct the equivalent disk when it is replaced ?

Fernando

On 07/08/2017 10:26, Erekle Magradze wrote:

Hi Frenando,

Here is my experience, if you consider a particular hard drive as a brick for gluster volume and it dies, i.e. it becomes not accessible it's a huge hassle to discard that brick and exchange with another one, since gluster some tries to access that broken brick and it's causing (at least it cause for me) a big pain, therefore it's better to have a RAID as brick, i.e. have RAID 1 (mirroring) for each brick, in this case if the disk is down you can easily exchange it and rebuild the RAID without going offline, i.e switching off the volume doing brick manipulations and switching it back on.

Cheers

Erekle

On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote:

For any RAID 5 or 6 configuration I normally follow a simple gold rule which gave good results so far:
- up to 4 disks RAID 5
- 5 or more disks RAID 6

However I didn't really understand well the recommendation to use any RAID with GlusterFS. I always thought that GlusteFS likes to work in JBOD mode and control the disks (bricks) directlly so you can create whatever distribution rule you wish, and if a single disk fails you just replace it and which obviously have the data replicated from another. The only downside of using in this way is that the replication data will be flow accross all servers but that is not much a big issue.

Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS.

Thanks
Regards
Fernando

On 07/08/2017 03:46, Devin Acosta wrote:

Moacir,

I have recently installed multiple Red Hat Virtualization hosts for several different companies, and have dealt with the Red Hat Support Team in depth about optimal configuration in regards to setting up GlusterFS most efficiently and I wanted to share with you what I learned.

In general Red Hat Virtualization team frowns upon using each DISK of the system as just a JBOD, sure there is some protection by having the data replicated, however, the recommendation is to use RAID 6 (preferred) or RAID-5, or at least RAID-1 at the very least.

Here is the=C2=A0direct quote from Red = Hat when I asked about RAID and Bricks:

"A typical Gluster configuration would use RAID underneath the bricks. RAID 6 is most typical as it gives you 2 disk failure protection, but RAID 5 could be used too.=C2=A0Once you have the RAIDed bricks, you'd then apply the desired replication on top of that. The most popular way of doing this would be distributed replicated with 2x replication.=C2=A0In general you'll get better performance with larger bricks. 12 drives is often a sweet spot. Another option would be to create a separate tier using all SSD=E2=80=99s.=E2=80=9D=C2=A0

In order to SSD=C2=A0tiering from my understanding you would need 1 x NVMe drive in each server, or 4 x SSD hot tier (it needs to be distributed, replicated for the hot tier if not using NVME). So with you only having 1 SSD drive in each server,=C2=A0I=E2=80=99d suggest maybe looking into the N= VME option.=C2=A0

Since your using only 3-servers, what I=E2=80=99d probably suggest is to do (2 Replicas + Arbiter Node), this setup actually doesn=E2=80=99t requir= e the 3rd server to have big drives at all as it only stores meta-data about the files and not actually a full copy.=C2=A0

Please see the attached document that was given to me by Red Hat to get more information on this. Hope this information helps you.=

--

Devin Acosta, RHCA, RHVCA

Red Hat Certified Architect

On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (moacirferreira(a)hotmail.com) wrote:

I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop like a server triangle using the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VMs around the pod (east /west traffic) while using the 10Gb interfaces for giving services to the outside world (north/south traffic).

This said, my first question is: How should I deploy GlusterFS in such oVirt scenario? My questions are:

1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then create a GlusterFS using them?

2 - Instead, should I create a JBOD array made of all server's disks?

3 - What is the best Gluster configuration to provide for HA while not consuming too much disk space?

4 - Does a oVirt hypervisor pod like I am planning to build, and the virtualization environment, benefits from tiering when using a SSD disk? And yes, will Gluster do it by default or I have to configure it to do so?

At the bottom line, what is the good practice for using GlusterFS in small pods for enterprises?

You opinion/feedback will be really appreciated!

Moacir

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailm= an/listinfo/users

_______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/list= info/users

_______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/list= info/users

-- = Recogizer Group GmbH Dr.rer.nat. Erekle Magradze Lead Big Data Engineering & DevOps Rheinwerkallee 2, 53227 Bonn Tel: +49 228 29974555 E-Mail erekle.magradze(a)recogizer.de Web: www.recogizer.com = Recogizer auf LinkedIn https://www.linkedin.com/company-be= ta/10039182/ Folgen Sie uns auf Twitter https://twitter.com/recogizer = ----------------------------------------------------------------- Recogizer Group GmbH Gesch=C3=A4ftsf=C3=BChrer: Oliver Habisch, Carsten Kreutze Handelsregister: Amtsgericht Bonn HRB 20724 Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993 = Diese E-Mail enth=C3=A4lt vertrauliche und/oder rechtlich gesch=C3=BCtzte I= nformationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=C3=BCmlich= erhalten haben, informieren Sie bitte sofort den Absender und l=C3=B6schen Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der = darin enthaltenen Informationen ist nicht gestattet.
--------------0DC09401E4EDC25E68CD6208-- --===============3381618162231106505== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS0wREMwOTQwMUU0RURDMjVFNjhDRDYyMDgKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDhiaXQK CkhpIEZlcm5hbmRvIChzb3JyeSBmb3IgbWlzc3BlbGxpbmcgeW91ciBuYW1lLCBJIHVzZWQgYSBk aWZmZXJlbnQga2V5Ym9hcmQpLAoKU28gbGV0J3MgZ28gd2l0aCB0aGUgZm9sbG93aW5nIHNjZW5h cmlvczoKCjEuIExldCdzIHNheSB5b3UgaGF2ZSB0d28gc2VydmVycyAocmVwbGljYXRpb24gZmFj dG9yIGlzIDIpLCBpLmUuIHR3byAKYnJpY2tzIHBlciB2b2x1bWUsIGluIHRoaXMgY2FzZSBpdCBp cyBzdHJvbmdseSByZWNvbW1lbmRlZCB0byBoYXZlIHRoZSAKYXJiaXRlciBub2RlLCB0aGUgbWV0 YWRhdGEgc3RvcmFnZSB0aGF0IHdpbGwgZ3VhcmFudGVlIGF2b2lkaW5nIHRoZSAKc3BsaXQgYnJh aW4gc2l0dWF0aW9uLCBpbiB0aGlzIGNhc2UgZm9yIGFyYml0ZXIgeW91IGRvbid0IGV2ZW4gbmVl ZCBhIApkaXNrIHdpdGggbG90cyBvZiBzcGFjZSwgaXQncyBlbm91Z2ggdG8gaGF2ZSBhIHRpbnkg c3NkIGJ1dCBob3N0ZWQgb24gYSAKc2VwYXJhdGUgc2VydmVyLiBBZHZhbnRhZ2Ugb2Ygc3VjaCBz ZXR1cCBpcyB0aGF0IHlvdSBkb24ndCBuZWVkIHRoZSBSQUlEIAoxIGZvciBlYWNoIGJyaWNrLCB5 b3UgaGF2ZSB0aGUgbWV0YWRhdGEgaW5mb3JtYXRpb24gc3RvcmVkIGluIGFyYml0ZXIgCm5vZGUg YW5kIGJyaWNrIHJlcGxhY2VtZW50IGlzIGVhc3kuCgoyLiBJZiB5b3UgaGF2ZSBvZGQgbnVtYmVy IG9mIGJyaWNrcyAobGV0J3Mgc2F5IDMsIGkuZS4gcmVwbGljYXRpb24gCmZhY3RvciBpcyAzKSBp biB5b3VyIHZvbHVtZSBhbmQgeW91IGRpZG4ndCBjcmVhdGUgdGhlIGFyYml0ZXIgbm9kZSBhcyAK d2VsbCBhcyB5b3UgZGlkbid0IGNvbmZpZ3VyZSB0aGUgcXVvcnVtLCBpbiB0aGlzIGNhc2UgdGhl IGVudGlyZSBsb2FkIApmb3Iga2VlcGluZyB0aGUgY29uc2lzdGVuY3kgb2YgdGhlIHZvbHVtZSBy ZXNpZGVzIG9uIGFsbCAzIHNlcnZlcnMsIGVhY2ggCm9mIHRoZW0gaXMgaW1wb3J0YW50IGFuZCBl YWNoIGJyaWNrIGNvbnRhaW5zIGtleSBpbmZvcm1hdGlvbiwgdGhleSBuZWVkIAp0byBjcm9zcy1j aGVjayBlYWNoIG90aGVyICh0aGF0J3Mgd2hhdCBwZW9wbGUgdXN1YWxseSBkbyB3aXRoIHRoZSBm aXJzdCAKdHJ5IG9mIGdsdXN0ZXIgOikgKSwgaW4gdGhpcyBjYXNlIHJlcGxhY2luZyBhIGJyaWNr IGlzIGEgYmlnIHBhaW4gYW5kIGluIAp0aGlzIGNhc2UgUkFJRCAxIGlzIGEgZ29vZCBvcHRpb24g dG8gaGF2ZSAodGhhdCdzIHRoZSBkaXNhZHZhbnRhZ2UsIGkuZS4gCmxvb3NpbmcgdGhlIHNwYWNl IGFuZCBub3QgaGF2aW5nIHRoZSBKQk9EIG9wdGlvbikgYWR2YW50YWdlIGlzIHRoYXQgeW91IApk b24ndCBoYXZlIHRoZSB0byBoYXZlIGFkZGl0aW9uYWwgYXJiaXRlciBub2RlLgoKMy4gWW91IGhh dmUgb2RkIG51bWJlciBvZiBicmlja3MgYW5kIGNvbmZpZ3VyZWQgYXJiaXRlciBub2RlLCBpbiB0 aGlzIApjYXNlIHlvdSBjYW4gZWFzaWx5IGdvIHdpdGggSkJPRCwgaG93ZXZlciBhIGdvb2QgcHJh Y3RpY2Ugd291bGQgYmUgdG8gCmhhdmUgYSBSQUlEIDEgZm9yIGFyYml0ZXIgZGlza3MgKHRpbnkg MTI4R0IgU1NELXMgYXIgcGVyZmVjdGx5IApzdWZmaWNpZW50IGZvciB2b2x1bWVzIHdpdGggMTBz IG9mIFRCLXMgaW4gc2l6ZS4pCgpUaGF0J3MgYmFzaWNhbGx5IGl0CgpUaGUgcmVzdCBhYm91dCB0 aGUgcmVsaWFiaWxpdHkgYW5kIHNldHVwIHNjZW5hcmlvcyB5b3UgY2FuIGZpbmQgaW4gCmdsdXN0 ZXIgZG9jdW1lbnRhdGlvbiwgZXNwZWNpYWxseSBsb29rIGZvciBxdW9ydW0gYW5kIGFyYml0ZXIg bm9kZSAKY29uZmlncytvcHRpb25zLgoKQ2hlZXJzCgpFcmVrbGUKClAuUy4gV2hhdCBJIHdhcyBt ZW50aW9uaW5nLCByZWdhcmRpbmcgYSBnb29kIHByYWN0aWNlIGlzIG1vc3RseSByZWxhdGVkIAp0 byB0aGUgb3BlcmF0aW9ucyBvZiBnbHVzdGVyIG5vdCBpbnN0YWxsYXRpb24gb3IgZGVwbG95bWVu dCwgaS5lLiBub3QgCnRoZSBjb25jZXB0dWFsIHVuZGVyc3RhbmRpbmcgb2YgZ2x1c3RlciAoY29u Y2VwdHVhbGx5IGl0J3MgYSBKQk9EIHN5c3RlbSkuCgpPbiAwOC8wNy8yMDE3IDA1OjQxIFBNLCBG RVJOQU5ETyBGUkVESUFOSSB3cm90ZToKPgo+IFRoYW5rcyBmb3IgdGhlIGNsYXJpZmljYXRpb24g RXJla2xlLgo+Cj4gSG93ZXZlciBJIGdldCBzdXJwcmlzZWQgd2l0aCB0aGlzIHdheSBvZiBvcGVy YXRpbmcgZnJvbSBHbHVzdGVyRlMgYXMgCj4gaXQgYWRkcyBhbm90aGVyIGxheWVyIG9mIGNvbXBs ZXhpdHkgdG8gdGhlIHN5c3RlbSAoZWl0aGVyIGEgaGFyZHdhcmUgCj4gb3Igc29mdHdhcmUgUkFJ RCkgYmVmb3JlIHRoZSBnbHVzdGVyIGNvbmZpZyBhbmQgaW5jcmVhc2UgdGhlIHN5c3RlbSdzIAo+ IG92ZXJhbGwgY29zdHMuCj4KPiBBbiBpbXBvcnRhbnQgcG9pbnQgdG8gY29uc2lkZXIgaXM6IElu IFJBSUQgY29uZmlndXJhdGlvbiB5b3UgYWxyZWFkeSAKPiBoYXZlIHNwYWNlICd3YXN0ZWQnIGlu IG9yZGVyIHRvIGJ1aWxkIHJlZHVuZGFuY3kgKGVpdGhlciBSQUlEIDEsIDUsIG9yIAo+IDYpLiBU aGVuIHdoZW4geW91IGhhdmUgR2x1c3RlckZTIG9uIHRoZSB0b3Agb2Ygc2V2ZXJhbCBSQUlEcyB5 b3UgaGF2ZSAKPiBhZ2FpbiBtb3JlIGRhdGEgcmVwbGljYXRlZCBzbyB5b3UgZW5kIHVwIHdpdGgg dGhlIHNhbWUgZGF0YSBjb25zdW1pbmcgCj4gbW9yZSBzcGFjZSBpbiBhIGdyb3VwIG9mIGRpc2tz IGFuZCBhZ2FpbiBvbiB0aGUgdG9wIG9mIHNldmVyYWwgUkFJRHMgCj4gZGVwZW5kaW5nIG9uIHRo ZSBHbHVzdGVyIGNvbmZpZ3VyYXRpb24geW91IGhhdmUgKGluIGEgUkFJRCAxIGNvbmZpZyAKPiB0 aGUgc2FtZSBkYXRhIGlzIHJlcGxpY2F0ZWQgNCB0aW1lcykuCj4KPiBZZXQgYW5vdGhlciBkb3du c2lkZSBvZiBoYXZpbmcgYSBSQUlEIChzcGVjaWFsbHkgUkFJRCA1IG9yIDYpIGlzIHRoYXQgCj4g aXQgcmVkdWNlcyBjb25zaWRlcmFibHkgdGhlIHdyaXRlIHNwZWVkcyBhcyBlYWNoIGdyb3VwIG9m IGRpc2tzIHdpbGwgCj4gZW5kIHVwIGhhdmluZyB0aGUgd3JpdGUgc3BlZWQgb2YgYSBzaW5nbGUg ZGlzayBhcyBhbGwgb3RoZXIgZGlza3Mgb2YgCj4gdGhhdCBncm91cCBoYXZlIHRvIHdhaXQgZm9y IGVhY2ggb3RoZXIgdG8gd3JpdGUgYXMgd2VsbC4KPgo+IFRoZXJlZm9yZSBpZiBHbHVzdGVyIGFs cmVhZHkgcmVwbGljYXRlcyBkYXRhIHdoeSBkb2VzIGl0IGNyZWF0ZSB0aGlzIAo+IGJpZyBwYWlu IHlvdSBtZW50aW9uZWQgaWYgdGhlIGRhdGEgaXMgcmVwbGljYXRlZCBzb21ld2hlcmUgZWxzZSwg Y2FuIAo+IHN0aWxsIGJlIHJldHJpZXZlZCB0byBib3RoIHNlcnZlIGNsaWVudHMgYW5kIHJlY29u c3RydWN0IHRoZSAKPiBlcXVpdmFsZW50IGRpc2sgd2hlbiBpdCBpcyByZXBsYWNlZCA/Cj4KPiBG ZXJuYW5kbwo+Cj4KPiBPbiAwNy8wOC8yMDE3IDEwOjI2LCBFcmVrbGUgTWFncmFkemUgd3JvdGU6 Cj4+Cj4+IEhpIEZyZW5hbmRvLAo+Pgo+PiBIZXJlIGlzIG15IGV4cGVyaWVuY2UsIGlmIHlvdSBj b25zaWRlciBhIHBhcnRpY3VsYXIgaGFyZCBkcml2ZSBhcyBhIAo+PiBicmljayBmb3IgZ2x1c3Rl ciB2b2x1bWUgYW5kIGl0IGRpZXMsIGkuZS4gaXQgYmVjb21lcyBub3QgYWNjZXNzaWJsZSAKPj4g aXQncyBhIGh1Z2UgaGFzc2xlIHRvIGRpc2NhcmQgdGhhdCBicmljayBhbmQgZXhjaGFuZ2Ugd2l0 aCBhbm90aGVyIAo+PiBvbmUsIHNpbmNlIGdsdXN0ZXIgc29tZSB0cmllcyB0byBhY2Nlc3MgdGhh dCBicm9rZW4gYnJpY2sgYW5kIGl0J3MgCj4+IGNhdXNpbmcgKGF0IGxlYXN0IGl0IGNhdXNlIGZv ciBtZSkgYSBiaWcgcGFpbiwgdGhlcmVmb3JlIGl0J3MgYmV0dGVyIAo+PiB0byBoYXZlIGEgUkFJ RCBhcyBicmljaywgaS5lLiBoYXZlIFJBSUQgMSAobWlycm9yaW5nKSBmb3IgZWFjaCBicmljaywg Cj4+IGluIHRoaXMgY2FzZSBpZiB0aGUgZGlzayBpcyBkb3duIHlvdSBjYW4gZWFzaWx5IGV4Y2hh bmdlIGl0IGFuZCAKPj4gcmVidWlsZCB0aGUgUkFJRCB3aXRob3V0IGdvaW5nIG9mZmxpbmUsIGku ZSBzd2l0Y2hpbmcgb2ZmIHRoZSB2b2x1bWUgCj4+IGRvaW5nIGJyaWNrIG1hbmlwdWxhdGlvbnMg YW5kIHN3aXRjaGluZyBpdCBiYWNrIG9uLgo+Pgo+PiBDaGVlcnMKPj4KPj4gRXJla2xlCj4+Cj4+ Cj4+IE9uIDA4LzA3LzIwMTcgMDM6MDQgUE0sIEZFUk5BTkRPIEZSRURJQU5JIHdyb3RlOgo+Pj4K Pj4+IEZvciBhbnkgUkFJRCA1IG9yIDYgY29uZmlndXJhdGlvbiBJIG5vcm1hbGx5IGZvbGxvdyBh IHNpbXBsZSBnb2xkIAo+Pj4gcnVsZSB3aGljaCBnYXZlIGdvb2QgcmVzdWx0cyBzbyBmYXI6Cj4+ PiAtIHVwIHRvIDQgZGlza3MgUkFJRCA1Cj4+PiAtIDUgb3IgbW9yZSBkaXNrcyBSQUlEIDYKPj4+ Cj4+PiBIb3dldmVyIEkgZGlkbid0IHJlYWxseSB1bmRlcnN0YW5kIHdlbGwgdGhlIHJlY29tbWVu ZGF0aW9uIHRvIHVzZSAKPj4+IGFueSBSQUlEIHdpdGggR2x1c3RlckZTLiBJIGFsd2F5cyB0aG91 Z2h0IHRoYXQgR2x1c3RlRlMgbGlrZXMgdG8gCj4+PiB3b3JrIGluIEpCT0QgbW9kZSBhbmQgY29u dHJvbCB0aGUgZGlza3MgKGJyaWNrcykgZGlyZWN0bGx5IHNvIHlvdSAKPj4+IGNhbiBjcmVhdGUg d2hhdGV2ZXIgZGlzdHJpYnV0aW9uIHJ1bGUgeW91IHdpc2gsIGFuZCBpZiBhIHNpbmdsZSBkaXNr IAo+Pj4gZmFpbHMgeW91IGp1c3QgcmVwbGFjZSBpdCBhbmQgd2hpY2ggb2J2aW91c2x5IGhhdmUg dGhlIGRhdGEgCj4+PiByZXBsaWNhdGVkIGZyb20gYW5vdGhlci4gVGhlIG9ubHkgZG93bnNpZGUg b2YgdXNpbmcgaW4gdGhpcyB3YXkgaXMgCj4+PiB0aGF0IHRoZSByZXBsaWNhdGlvbiBkYXRhIHdp bGwgYmUgZmxvdyBhY2Nyb3NzIGFsbCBzZXJ2ZXJzIGJ1dCB0aGF0IAo+Pj4gaXMgbm90IG11Y2gg YSBiaWcgaXNzdWUuCj4+Pgo+Pj4gQW55b25lIGNhbiBlbGFib3JhdGUgYWJvdXQgVXNpbmcgUkFJ RCArIEdsdXN0ZXJGUyBhbmQgSkJPRCArIEdsdXN0ZXJGUy4KPj4+Cj4+PiBUaGFua3MKPj4+IFJl Z2FyZHMKPj4+IEZlcm5hbmRvCj4+Pgo+Pj4KPj4+IE9uIDA3LzA4LzIwMTcgMDM6NDYsIERldmlu IEFjb3N0YSB3cm90ZToKPj4+Pgo+Pj4+IE1vYWNpciwKPj4+Pgo+Pj4+IEkgaGF2ZSByZWNlbnRs eSBpbnN0YWxsZWQgbXVsdGlwbGUgUmVkIEhhdCBWaXJ0dWFsaXphdGlvbiBob3N0cyBmb3IgCj4+ Pj4gc2V2ZXJhbCBkaWZmZXJlbnQgY29tcGFuaWVzLCBhbmQgaGF2ZSBkZWFsdCB3aXRoIHRoZSBS ZWQgSGF0IAo+Pj4+IFN1cHBvcnQgVGVhbSBpbiBkZXB0aCBhYm91dCBvcHRpbWFsIGNvbmZpZ3Vy YXRpb24gaW4gcmVnYXJkcyB0byAKPj4+PiBzZXR0aW5nIHVwIEdsdXN0ZXJGUyBtb3N0IGVmZmlj aWVudGx5IGFuZCBJIHdhbnRlZCB0byBzaGFyZSB3aXRoIAo+Pj4+IHlvdSB3aGF0IEkgbGVhcm5l ZC4KPj4+Pgo+Pj4+IEluIGdlbmVyYWwgUmVkIEhhdCBWaXJ0dWFsaXphdGlvbiB0ZWFtIGZyb3du cyB1cG9uIHVzaW5nIGVhY2ggRElTSyAKPj4+PiBvZiB0aGUgc3lzdGVtIGFzIGp1c3QgYSBKQk9E LCBzdXJlIHRoZXJlIGlzIHNvbWUgcHJvdGVjdGlvbiBieSAKPj4+PiBoYXZpbmcgdGhlIGRhdGEg cmVwbGljYXRlZCwgaG93ZXZlciwgdGhlIHJlY29tbWVuZGF0aW9uIGlzIHRvIHVzZSAKPj4+PiBS QUlEIDYgKHByZWZlcnJlZCkgb3IgUkFJRC01LCBvciBhdCBsZWFzdCBSQUlELTEgYXQgdGhlIHZl cnkgbGVhc3QuCj4+Pj4KPj4+PiBIZXJlIGlzIHRoZSBkaXJlY3QgcXVvdGUgZnJvbSBSZWQgSGF0 IHdoZW4gSSBhc2tlZCBhYm91dCBSQUlEIGFuZCAKPj4+PiBCcmlja3M6Cj4+Pj4gLwo+Pj4+IC8K Pj4+PiAvIkEgdHlwaWNhbCBHbHVzdGVyIGNvbmZpZ3VyYXRpb24gd291bGQgdXNlIFJBSUQgdW5k ZXJuZWF0aCB0aGUgCj4+Pj4gYnJpY2tzLiBSQUlEIDYgaXMgbW9zdCB0eXBpY2FsIGFzIGl0IGdp dmVzIHlvdSAyIGRpc2sgZmFpbHVyZSAKPj4+PiBwcm90ZWN0aW9uLCBidXQgUkFJRCA1IGNvdWxk IGJlIHVzZWQgdG9vLiBPbmNlIHlvdSBoYXZlIHRoZSBSQUlEZWQgCj4+Pj4gYnJpY2tzLCB5b3Un ZCB0aGVuIGFwcGx5IHRoZSBkZXNpcmVkIHJlcGxpY2F0aW9uIG9uIHRvcCBvZiB0aGF0LiAKPj4+ PiBUaGUgbW9zdCBwb3B1bGFyIHdheSBvZiBkb2luZyB0aGlzIHdvdWxkIGJlIGRpc3RyaWJ1dGVk IHJlcGxpY2F0ZWQgCj4+Pj4gd2l0aCAyeCByZXBsaWNhdGlvbi4gSW4gZ2VuZXJhbCB5b3UnbGwg Z2V0IGJldHRlciBwZXJmb3JtYW5jZSB3aXRoIAo+Pj4+IGxhcmdlciBicmlja3MuIDEyIGRyaXZl cyBpcyBvZnRlbiBhIHN3ZWV0IHNwb3QuIEFub3RoZXIgb3B0aW9uIAo+Pj4+IHdvdWxkIGJlIHRv IGNyZWF0ZSBhIHNlcGFyYXRlIHRpZXIgdXNpbmcgYWxsIFNTROKAmXMu4oCdIC8KPj4+Pgo+Pj4+ IC9JbiBvcmRlciB0byBTU0QgdGllcmluZyBmcm9tIG15IHVuZGVyc3RhbmRpbmcgeW91IHdvdWxk IG5lZWQgMSB4IAo+Pj4+IE5WTWUgZHJpdmUgaW4gZWFjaCBzZXJ2ZXIsIG9yIDQgeCBTU0QgaG90 IHRpZXIgKGl0IG5lZWRzIHRvIGJlIAo+Pj4+IGRpc3RyaWJ1dGVkLCByZXBsaWNhdGVkIGZvciB0 aGUgaG90IHRpZXIgaWYgbm90IHVzaW5nIE5WTUUpLiBTbyAKPj4+PiB3aXRoIHlvdSBvbmx5IGhh dmluZyAxIFNTRCBkcml2ZSBpbiBlYWNoIHNlcnZlciwgSeKAmWQgc3VnZ2VzdCBtYXliZSAKPj4+ PiBsb29raW5nIGludG8gdGhlIE5WTUUgb3B0aW9uLiAvCj4+Pj4gLwo+Pj4+IC8KPj4+PiAvU2lu Y2UgeW91ciB1c2luZyBvbmx5IDMtc2VydmVycywgd2hhdCBJ4oCZZCBwcm9iYWJseSBzdWdnZXN0 IGlzIHRvIAo+Pj4+IGRvICgyIFJlcGxpY2FzICsgQXJiaXRlciBOb2RlKSwgdGhpcyBzZXR1cCBh Y3R1YWxseSBkb2VzbuKAmXQgcmVxdWlyZSAKPj4+PiB0aGUgM3JkIHNlcnZlciB0byBoYXZlIGJp ZyBkcml2ZXMgYXQgYWxsIGFzIGl0IG9ubHkgc3RvcmVzIAo+Pj4+IG1ldGEtZGF0YSBhYm91dCB0 aGUgZmlsZXMgYW5kIG5vdCBhY3R1YWxseSBhIGZ1bGwgY29weS4gLwo+Pj4+IC8KPj4+PiAvCj4+ Pj4gL1BsZWFzZSBzZWUgdGhlIGF0dGFjaGVkIGRvY3VtZW50IHRoYXQgd2FzIGdpdmVuIHRvIG1l IGJ5IFJlZCBIYXQgCj4+Pj4gdG8gZ2V0IG1vcmUgaW5mb3JtYXRpb24gb24gdGhpcy4gSG9wZSB0 aGlzIGluZm9ybWF0aW9uIGhlbHBzIHlvdS4vCj4+Pj4gLwo+Pj4+IC8KPj4+Pgo+Pj4+IC0tCj4+ Pj4KPj4+PiBEZXZpbiBBY29zdGEsIFJIQ0EsIFJIVkNBCj4+Pj4gUmVkIEhhdCBDZXJ0aWZpZWQg QXJjaGl0ZWN0Cj4+Pj4KPj4+PiBPbiBBdWd1c3QgNiwgMjAxNyBhdCA3OjI5OjI5IFBNLCBNb2Fj aXIgRmVycmVpcmEgCj4+Pj4gKG1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY29tIDxtYWlsdG86bW9h Y2lyZmVycmVpcmFAaG90bWFpbC5jb20+KSB3cm90ZToKPj4+Pgo+Pj4+PiBJIGFtIHdpbGxpbmcg dG8gYXNzZW1ibGUgYSBvVmlydCAicG9kIiwgbWFkZSBvZiAzIHNlcnZlcnMsIGVhY2ggCj4+Pj4+ IHdpdGggMiBDUFUgc29ja2V0cyBvZiAxMiBjb3JlcywgMjU2R0IgUkFNLCA3IEhERCAxMEssIDEg U1NELiBUaGUgCj4+Pj4+IGlkZWEgaXMgdG8gdXNlIEdsdXN0ZXJGUyB0byBwcm92aWRlIEhBIGZv ciB0aGUgVk1zLiBUaGUgMyBzZXJ2ZXJzIAo+Pj4+PiBoYXZlIGEgZHVhbCA0MEdiIE5JQyBhbmQg YSBkdWFsIDEwR2IgTklDLiBTbyBteSBpbnRlbnRpb24gaXMgdG8gCj4+Pj4+IGNyZWF0ZSBhIGxv b3AgbGlrZSBhIHNlcnZlciB0cmlhbmdsZSB1c2luZyB0aGUgNDBHYiBOSUNzIGZvciAKPj4+Pj4g dmlydHVhbGl6YXRpb24gZmlsZXMgKFZNcyAucWNvdzIpIGFjY2VzcyBhbmQgdG8gbW92ZSBWTXMg YXJvdW5kIAo+Pj4+PiB0aGUgcG9kIChlYXN0IC93ZXN0IHRyYWZmaWMpIHdoaWxlIHVzaW5nIHRo ZSAxMEdiIGludGVyZmFjZXMgZm9yIAo+Pj4+PiBnaXZpbmcgc2VydmljZXMgdG8gdGhlIG91dHNp ZGUgd29ybGQgKG5vcnRoL3NvdXRoIHRyYWZmaWMpLgo+Pj4+Pgo+Pj4+Pgo+Pj4+PiBUaGlzIHNh aWQsIG15IGZpcnN0IHF1ZXN0aW9uIGlzOiBIb3cgc2hvdWxkIEkgZGVwbG95IEdsdXN0ZXJGUyBp biAKPj4+Pj4gc3VjaCBvVmlydCBzY2VuYXJpbz8gTXkgcXVlc3Rpb25zIGFyZToKPj4+Pj4KPj4+ Pj4KPj4+Pj4gMSAtIFNob3VsZCBJIGNyZWF0ZSAzIFJBSUQgKGkuZS46IFJBSUQgNSksIG9uZSBv biBlYWNoIG9WaXJ0IG5vZGUsIAo+Pj4+PiBhbmQgdGhlbiBjcmVhdGUgYSBHbHVzdGVyRlMgdXNp bmcgdGhlbT8KPj4+Pj4KPj4+Pj4gMiAtIEluc3RlYWQsIHNob3VsZCBJIGNyZWF0ZSBhIEpCT0Qg YXJyYXkgbWFkZSBvZiBhbGwgc2VydmVyJ3MgZGlza3M/Cj4+Pj4+Cj4+Pj4+IDMgLSBXaGF0IGlz IHRoZSBiZXN0IEdsdXN0ZXIgY29uZmlndXJhdGlvbiB0byBwcm92aWRlIGZvciBIQSB3aGlsZSAK Pj4+Pj4gbm90IGNvbnN1bWluZyB0b28gbXVjaCBkaXNrIHNwYWNlPwo+Pj4+Pgo+Pj4+PiA0IC0g RG9lcyBhIG9WaXJ0IGh5cGVydmlzb3IgcG9kIGxpa2UgSSBhbSBwbGFubmluZyB0byBidWlsZCwg YW5kIAo+Pj4+PiB0aGUgdmlydHVhbGl6YXRpb24gZW52aXJvbm1lbnQsIGJlbmVmaXRzIGZyb20g dGllcmluZyB3aGVuIHVzaW5nIGEgCj4+Pj4+IFNTRCBkaXNrPyBBbmQgeWVzLCB3aWxsIEdsdXN0 ZXIgZG8gaXQgYnkgZGVmYXVsdCBvciBJIGhhdmUgdG8gCj4+Pj4+IGNvbmZpZ3VyZSBpdCB0byBk byBzbz8KPj4+Pj4KPj4+Pj4KPj4+Pj4gQXQgdGhlIGJvdHRvbSBsaW5lLCB3aGF0IGlzIHRoZSBn b29kIHByYWN0aWNlIGZvciB1c2luZyBHbHVzdGVyRlMgCj4+Pj4+IGluIHNtYWxsIHBvZHMgZm9y IGVudGVycHJpc2VzPwo+Pj4+Pgo+Pj4+Pgo+Pj4+PiBZb3Ugb3Bpbmlvbi9mZWVkYmFjayB3aWxs IGJlIHJlYWxseSBhcHByZWNpYXRlZCEKPj4+Pj4KPj4+Pj4gTW9hY2lyCj4+Pj4+Cj4+Pj4+IF9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCj4+Pj4+IFVzZXJz IG1haWxpbmcgbGlzdAo+Pj4+PiBVc2Vyc0BvdmlydC5vcmcgPG1haWx0bzpVc2Vyc0BvdmlydC5v cmc+Cj4+Pj4+IGh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vycwo+ Pj4+Cj4+Pj4KPj4+PiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fXwo+Pj4+IFVzZXJzIG1haWxpbmcgbGlzdAo+Pj4+IFVzZXJzQG92aXJ0Lm9yZwo+Pj4+IGh0 dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vycwo+Pj4KPj4+Cj4+Pgo+ Pj4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KPj4+IFVz ZXJzIG1haWxpbmcgbGlzdAo+Pj4gVXNlcnNAb3ZpcnQub3JnCj4+PiBodHRwOi8vbGlzdHMub3Zp cnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMKPj4KPgoKLS0gClJlY29naXplciBHcm91cCBH bWJICgpEci5yZXIubmF0LiBFcmVrbGUgTWFncmFkemUKTGVhZCBCaWcgRGF0YSBFbmdpbmVlcmlu ZyAmIERldk9wcwpSaGVpbndlcmthbGxlZSAyLCA1MzIyNyBCb25uClRlbDogKzQ5IDIyOCAyOTk3 NDU1NQoKRS1NYWlsIGVyZWtsZS5tYWdyYWR6ZUByZWNvZ2l6ZXIuZGUKV2ViOiB3d3cucmVjb2dp emVyLmNvbQogIApSZWNvZ2l6ZXIgYXVmIExpbmtlZEluIGh0dHBzOi8vd3d3LmxpbmtlZGluLmNv bS9jb21wYW55LWJldGEvMTAwMzkxODIvCkZvbGdlbiBTaWUgdW5zIGF1ZiBUd2l0dGVyIGh0dHBz Oi8vdHdpdHRlci5jb20vcmVjb2dpemVyCiAgCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tClJlY29naXplciBHcm91cCBHbWJI Ckdlc2Now6RmdHNmw7xocmVyOiBPbGl2ZXIgSGFiaXNjaCwgQ2Fyc3RlbiBLcmV1dHplCkhhbmRl bHNyZWdpc3RlcjogQW10c2dlcmljaHQgQm9ubiBIUkIgMjA3MjQKU2l0eiBkZXIgR2VzZWxsc2No YWZ0OiBCb25uOyBVU3QtSUQtTnIuOiBERTI5NDE5NTk5MwogIApEaWVzZSBFLU1haWwgZW50aMOk bHQgdmVydHJhdWxpY2hlIHVuZC9vZGVyIHJlY2h0bGljaCBnZXNjaMO8dHp0ZSBJbmZvcm1hdGlv bmVuLgpXZW5uIFNpZSBuaWNodCBkZXIgcmljaHRpZ2UgQWRyZXNzYXQgc2luZCBvZGVyIGRpZXNl IEUtTWFpbCBpcnJ0w7xtbGljaCBlcmhhbHRlbiBoYWJlbiwKaW5mb3JtaWVyZW4gU2llIGJpdHRl IHNvZm9ydCBkZW4gQWJzZW5kZXIgdW5kIGzDtnNjaGVuIFNpZSBkaWVzZSBNYWlsLgpEYXMgdW5l cmxhdWJ0ZSBLb3BpZXJlbiBzb3dpZSBkaWUgdW5iZWZ1Z3RlIFdlaXRlcmdhYmUgZGllc2VyIE1h aWwgdW5kIGRlciBkYXJpbiBlbnRoYWx0ZW5lbiBJbmZvcm1hdGlvbmVuIGlzdCBuaWNodCBnZXN0 YXR0ZXQuCgoKLS0tLS0tLS0tLS0tLS0wREMwOTQwMUU0RURDMjVFNjhDRDYyMDgKQ29udGVudC1U eXBlOiB0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgKQ29udGVudC1UcmFuc2Zlci1FbmNvZGluZzog OGJpdAoKPGh0bWw+CiAgPGhlYWQ+CiAgICA8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUi IGNvbnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11dGYtOCI+CiAgPC9oZWFkPgogIDxib2R5IHRl eHQ9IiMwMDAwMDAiIGJnY29sb3I9IiNGRkZGRkYiPgogICAgPHA+SGkgRmVybmFuZG8gKHNvcnJ5 IGZvciBtaXNzcGVsbGluZyB5b3VyIG5hbWUsIEkgdXNlZCBhIGRpZmZlcmVudAogICAgICBrZXli b2FyZCksPC9wPgogICAgPHA+U28gbGV0J3MgZ28gd2l0aCB0aGUgZm9sbG93aW5nIHNjZW5hcmlv czo8L3A+CiAgICA8cD4xLiBMZXQncyBzYXkgeW91IGhhdmUgdHdvIHNlcnZlcnMgKHJlcGxpY2F0 aW9uIGZhY3RvciBpcyAyKSwgaS5lLgogICAgICB0d28gYnJpY2tzIHBlciB2b2x1bWUsIGluIHRo aXMgY2FzZSBpdCBpcyBzdHJvbmdseSByZWNvbW1lbmRlZCB0bwogICAgICBoYXZlIHRoZSBhcmJp dGVyIG5vZGUsIHRoZSBtZXRhZGF0YSBzdG9yYWdlIHRoYXQgd2lsbCBndWFyYW50ZWUKICAgICAg YXZvaWRpbmcgdGhlIHNwbGl0IGJyYWluIHNpdHVhdGlvbiwgaW4gdGhpcyBjYXNlIGZvciBhcmJp dGVyIHlvdQogICAgICBkb24ndCBldmVuIG5lZWQgYSBkaXNrIHdpdGggbG90cyBvZiBzcGFjZSwg aXQncyBlbm91Z2ggdG8gaGF2ZSBhCiAgICAgIHRpbnkgc3NkIGJ1dCBob3N0ZWQgb24gYSBzZXBh cmF0ZSBzZXJ2ZXIuIEFkdmFudGFnZSBvZiBzdWNoIHNldHVwCiAgICAgIGlzIHRoYXQgeW91IGRv bid0IG5lZWQgdGhlIFJBSUQgMSBmb3IgZWFjaCBicmljaywgeW91IGhhdmUgdGhlCiAgICAgIG1l dGFkYXRhIGluZm9ybWF0aW9uIHN0b3JlZCBpbiBhcmJpdGVyIG5vZGUgYW5kIGJyaWNrIHJlcGxh Y2VtZW50CiAgICAgIGlzIGVhc3kuPC9wPgogICAgPHA+Mi4gSWYgeW91IGhhdmUgb2RkIG51bWJl ciBvZiBicmlja3MgKGxldCdzIHNheSAzLCBpLmUuCiAgICAgIHJlcGxpY2F0aW9uIGZhY3RvciBp cyAzKSBpbiB5b3VyIHZvbHVtZSBhbmQgeW91IGRpZG4ndCBjcmVhdGUgdGhlCiAgICAgIGFyYml0 ZXIgbm9kZSBhcyB3ZWxsIGFzIHlvdSBkaWRuJ3QgY29uZmlndXJlIHRoZSBxdW9ydW0sIGluIHRo aXMKICAgICAgY2FzZSB0aGUgZW50aXJlIGxvYWQgZm9yIGtlZXBpbmcgdGhlIGNvbnNpc3RlbmN5 IG9mIHRoZSB2b2x1bWUKICAgICAgcmVzaWRlcyBvbiBhbGwgMyBzZXJ2ZXJzLCBlYWNoIG9mIHRo ZW0gaXMgaW1wb3J0YW50IGFuZCBlYWNoIGJyaWNrCiAgICAgIGNvbnRhaW5zIGtleSBpbmZvcm1h dGlvbiwgdGhleSBuZWVkIHRvIGNyb3NzLWNoZWNrIGVhY2ggb3RoZXIKICAgICAgKHRoYXQncyB3 aGF0IHBlb3BsZSB1c3VhbGx5IGRvIHdpdGggdGhlIGZpcnN0IHRyeSBvZiBnbHVzdGVyIDopICks CiAgICAgIGluIHRoaXMgY2FzZSByZXBsYWNpbmcgYSBicmljayBpcyBhIGJpZyBwYWluIGFuZCBp biB0aGlzIGNhc2UgUkFJRAogICAgICAxIGlzIGEgZ29vZCBvcHRpb24gdG8gaGF2ZSAodGhhdCdz IHRoZSBkaXNhZHZhbnRhZ2UsIGkuZS4gbG9vc2luZwogICAgICB0aGUgc3BhY2UgYW5kIG5vdCBo YXZpbmcgdGhlIEpCT0Qgb3B0aW9uKSBhZHZhbnRhZ2UgaXMgdGhhdCB5b3UKICAgICAgZG9uJ3Qg aGF2ZSB0aGUgdG8gaGF2ZSBhZGRpdGlvbmFsIGFyYml0ZXIgbm9kZS48L3A+CiAgICA8cD4zLiBZ b3UgaGF2ZSBvZGQgbnVtYmVyIG9mIGJyaWNrcyBhbmQgY29uZmlndXJlZCBhcmJpdGVyIG5vZGUs IGluCiAgICAgIHRoaXMgY2FzZSB5b3UgY2FuIGVhc2lseSBnbyB3aXRoIEpCT0QsIGhvd2V2ZXIg YSBnb29kIHByYWN0aWNlCiAgICAgIHdvdWxkIGJlIHRvIGhhdmUgYSBSQUlEIDEgZm9yIGFyYml0 ZXIgZGlza3MgKHRpbnkgMTI4R0IgU1NELXMgYXIKICAgICAgcGVyZmVjdGx5IHN1ZmZpY2llbnQg Zm9yIHZvbHVtZXMgd2l0aCAxMHMgb2YgVEItcyBpbiBzaXplLik8L3A+CiAgICA8cD5UaGF0J3Mg YmFzaWNhbGx5IGl0PC9wPgogICAgPHA+VGhlIHJlc3QgYWJvdXQgdGhlIHJlbGlhYmlsaXR5IGFu ZCBzZXR1cCBzY2VuYXJpb3MgeW91IGNhbiBmaW5kCiAgICAgIGluIGdsdXN0ZXIgZG9jdW1lbnRh dGlvbiwgZXNwZWNpYWxseSBsb29rIGZvciBxdW9ydW0gYW5kIGFyYml0ZXIKICAgICAgbm9kZSBj b25maWdzK29wdGlvbnMuPC9wPgogICAgPHA+Q2hlZXJzPC9wPgogICAgPHA+RXJla2xlPC9wPgog ICAgUC5TLiBXaGF0IEkgd2FzIG1lbnRpb25pbmcsIHJlZ2FyZGluZyBhIGdvb2QgcHJhY3RpY2Ug aXMgbW9zdGx5CiAgICByZWxhdGVkIHRvIHRoZSBvcGVyYXRpb25zIG9mIGdsdXN0ZXIgbm90IGlu c3RhbGxhdGlvbiBvciBkZXBsb3ltZW50LAogICAgaS5lLiBub3QgdGhlIGNvbmNlcHR1YWwgdW5k ZXJzdGFuZGluZyBvZiBnbHVzdGVyIChjb25jZXB0dWFsbHkgaXQncwogICAgYSBKQk9EIHN5c3Rl bSkuPGJyPgogICAgPGJyPgogICAgPGRpdiBjbGFzcz0ibW96LWNpdGUtcHJlZml4Ij5PbiAwOC8w Ny8yMDE3IDA1OjQxIFBNLCBGRVJOQU5ETwogICAgICBGUkVESUFOSSB3cm90ZTo8YnI+CiAgICA8 L2Rpdj4KICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiCiAgICAgIGNpdGU9Im1pZDpjN2ExYzJl MS01N2MzLTlmYTUtMDcxMC1lYmVlM2YzZmEwNjlAdXB4LmNvbSI+CiAgICAgIDxtZXRhIGh0dHAt ZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4K ICAgICAgPHA+VGhhbmtzIGZvciB0aGUgY2xhcmlmaWNhdGlvbiBFcmVrbGUuPC9wPgogICAgICA8 cD5Ib3dldmVyIEkgZ2V0IHN1cnByaXNlZCB3aXRoIHRoaXMgd2F5IG9mIG9wZXJhdGluZyBmcm9t CiAgICAgICAgR2x1c3RlckZTIGFzIGl0IGFkZHMgYW5vdGhlciBsYXllciBvZiBjb21wbGV4aXR5 IHRvIHRoZSBzeXN0ZW0KICAgICAgICAoZWl0aGVyIGEgaGFyZHdhcmUgb3Igc29mdHdhcmUgUkFJ RCkgYmVmb3JlIHRoZSBnbHVzdGVyIGNvbmZpZwogICAgICAgIGFuZCBpbmNyZWFzZSB0aGUgc3lz dGVtJ3Mgb3ZlcmFsbCBjb3N0cy48YnI+CiAgICAgIDwvcD4KICAgICAgPHA+QW4gaW1wb3J0YW50 IHBvaW50IHRvIGNvbnNpZGVyIGlzOiBJbiBSQUlEIGNvbmZpZ3VyYXRpb24geW91CiAgICAgICAg YWxyZWFkeSBoYXZlIHNwYWNlICd3YXN0ZWQnIGluIG9yZGVyIHRvIGJ1aWxkIHJlZHVuZGFuY3kg KGVpdGhlcgogICAgICAgIFJBSUQgMSwgNSwgb3IgNikuIFRoZW4gd2hlbiB5b3UgaGF2ZSBHbHVz dGVyRlMgb24gdGhlIHRvcCBvZgogICAgICAgIHNldmVyYWwgUkFJRHMgeW91IGhhdmUgYWdhaW4g bW9yZSBkYXRhIHJlcGxpY2F0ZWQgc28geW91IGVuZCB1cAogICAgICAgIHdpdGggdGhlIHNhbWUg ZGF0YSBjb25zdW1pbmcgbW9yZSBzcGFjZSBpbiBhIGdyb3VwIG9mIGRpc2tzIGFuZAogICAgICAg IGFnYWluIG9uIHRoZSB0b3Agb2Ygc2V2ZXJhbCBSQUlEcyBkZXBlbmRpbmcgb24gdGhlIEdsdXN0 ZXIKICAgICAgICBjb25maWd1cmF0aW9uIHlvdSBoYXZlIChpbiBhIFJBSUQgMSBjb25maWcgdGhl IHNhbWUgZGF0YSBpcwogICAgICAgIHJlcGxpY2F0ZWQgNCB0aW1lcykuPC9wPgogICAgICA8cD5Z ZXQgYW5vdGhlciBkb3duc2lkZSBvZiBoYXZpbmcgYSBSQUlEIChzcGVjaWFsbHkgUkFJRCA1IG9y IDYpCiAgICAgICAgaXMgdGhhdCBpdCByZWR1Y2VzIGNvbnNpZGVyYWJseSB0aGUgd3JpdGUgc3Bl ZWRzIGFzIGVhY2ggZ3JvdXAKICAgICAgICBvZiBkaXNrcyB3aWxsIGVuZCB1cCBoYXZpbmcgdGhl IHdyaXRlIHNwZWVkIG9mIGEgc2luZ2xlIGRpc2sgYXMKICAgICAgICBhbGwgb3RoZXIgZGlza3Mg b2YgdGhhdCBncm91cCBoYXZlIHRvIHdhaXQgZm9yIGVhY2ggb3RoZXIgdG8KICAgICAgICB3cml0 ZSBhcyB3ZWxsLjxicj4KICAgICAgPC9wPgogICAgICA8cD5UaGVyZWZvcmUgaWYgR2x1c3RlciBh bHJlYWR5IHJlcGxpY2F0ZXMgZGF0YSB3aHkgZG9lcyBpdCBjcmVhdGUKICAgICAgICB0aGlzIGJp ZyBwYWluIHlvdSBtZW50aW9uZWQgaWYgdGhlIGRhdGEgaXMgcmVwbGljYXRlZCBzb21ld2hlcmUK ICAgICAgICBlbHNlLCBjYW4gc3RpbGwgYmUgcmV0cmlldmVkIHRvIGJvdGggc2VydmUgY2xpZW50 cyBhbmQKICAgICAgICByZWNvbnN0cnVjdCB0aGUgZXF1aXZhbGVudCBkaXNrIHdoZW4gaXQgaXMg cmVwbGFjZWQgPzwvcD4KICAgICAgPHA+RmVybmFuZG88YnI+CiAgICAgIDwvcD4KICAgICAgPGJy PgogICAgICA8ZGl2IGNsYXNzPSJtb3otY2l0ZS1wcmVmaXgiPk9uIDA3LzA4LzIwMTcgMTA6MjYs IEVyZWtsZSBNYWdyYWR6ZQogICAgICAgIHdyb3RlOjxicj4KICAgICAgPC9kaXY+CiAgICAgIDxi bG9ja3F1b3RlIHR5cGU9ImNpdGUiCiAgICAgICAgY2l0ZT0ibWlkOmFhODI5ZDA3LWZhNzctM2Vk OS0yNTAwLWUzM2NjMDE0MTRiNkByZWNvZ2l6ZXIuZGUiPgogICAgICAgIDxtZXRhIGh0dHAtZXF1 aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0idGV4dC9odG1sOwogICAgICAgICAgY2hhcnNldD11 dGYtOCI+CiAgICAgICAgPHA+SGkgRnJlbmFuZG8sPC9wPgogICAgICAgIDxwPkhlcmUgaXMgbXkg ZXhwZXJpZW5jZSwgaWYgeW91IGNvbnNpZGVyIGEgcGFydGljdWxhciBoYXJkCiAgICAgICAgICBk cml2ZSBhcyBhIGJyaWNrIGZvciBnbHVzdGVyIHZvbHVtZSBhbmQgaXQgZGllcywgaS5lLiBpdAog ICAgICAgICAgYmVjb21lcyBub3QgYWNjZXNzaWJsZSBpdCdzIGEgaHVnZSBoYXNzbGUgdG8gZGlz Y2FyZCB0aGF0CiAgICAgICAgICBicmljayBhbmQgZXhjaGFuZ2Ugd2l0aCBhbm90aGVyIG9uZSwg c2luY2UgZ2x1c3RlciBzb21lIHRyaWVzCiAgICAgICAgICB0byBhY2Nlc3MgdGhhdCBicm9rZW4g YnJpY2sgYW5kIGl0J3MgY2F1c2luZyAoYXQgbGVhc3QgaXQKICAgICAgICAgIGNhdXNlIGZvciBt ZSkgYSBiaWcgcGFpbiwgdGhlcmVmb3JlIGl0J3MgYmV0dGVyIHRvIGhhdmUgYSBSQUlECiAgICAg ICAgICBhcyBicmljaywgaS5lLiBoYXZlIFJBSUQgMSAobWlycm9yaW5nKSBmb3IgZWFjaCBicmlj aywgaW4gdGhpcwogICAgICAgICAgY2FzZSBpZiB0aGUgZGlzayBpcyBkb3duIHlvdSBjYW4gZWFz aWx5IGV4Y2hhbmdlIGl0IGFuZAogICAgICAgICAgcmVidWlsZCB0aGUgUkFJRCB3aXRob3V0IGdv aW5nIG9mZmxpbmUsIGkuZSBzd2l0Y2hpbmcgb2ZmIHRoZQogICAgICAgICAgdm9sdW1lIGRvaW5n IGJyaWNrIG1hbmlwdWxhdGlvbnMgYW5kIHN3aXRjaGluZyBpdCBiYWNrIG9uLjxicj4KICAgICAg ICA8L3A+CiAgICAgICAgPHA+Q2hlZXJzPC9wPgogICAgICAgIDxwPkVyZWtsZTxicj4KICAgICAg ICA8L3A+CiAgICAgICAgPGJyPgogICAgICAgIDxkaXYgY2xhc3M9Im1vei1jaXRlLXByZWZpeCI+ T24gMDgvMDcvMjAxNyAwMzowNCBQTSwgRkVSTkFORE8KICAgICAgICAgIEZSRURJQU5JIHdyb3Rl Ojxicj4KICAgICAgICA8L2Rpdj4KICAgICAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIgogICAg ICAgICAgY2l0ZT0ibWlkOjYzYmFjNDdiLWFmZTYtMDI1OC1kM2Q3LWU1NDVhNTAwNGMzMEB1cHgu Y29tIj4KICAgICAgICAgIDxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0i dGV4dC9odG1sOwogICAgICAgICAgICBjaGFyc2V0PXV0Zi04Ij4KICAgICAgICAgIDxwPkZvciBh bnkgUkFJRCA1IG9yIDYgY29uZmlndXJhdGlvbiBJIG5vcm1hbGx5IGZvbGxvdyBhCiAgICAgICAg ICAgIHNpbXBsZSBnb2xkIHJ1bGUgd2hpY2ggZ2F2ZSBnb29kIHJlc3VsdHMgc28gZmFyOjxicj4K ICAgICAgICAgICAgLSB1cCB0byA0IGRpc2tzIFJBSUQgNTxicj4KICAgICAgICAgICAgLSA1IG9y IG1vcmUgZGlza3MgUkFJRCA2PC9wPgogICAgICAgICAgPHA+SG93ZXZlciBJIGRpZG4ndCByZWFs bHkgdW5kZXJzdGFuZCB3ZWxsIHRoZSByZWNvbW1lbmRhdGlvbgogICAgICAgICAgICB0byB1c2Ug YW55IFJBSUQgd2l0aCBHbHVzdGVyRlMuIEkgYWx3YXlzIHRob3VnaHQgdGhhdAogICAgICAgICAg ICBHbHVzdGVGUyBsaWtlcyB0byB3b3JrIGluIEpCT0QgbW9kZSBhbmQgY29udHJvbCB0aGUgZGlz a3MKICAgICAgICAgICAgKGJyaWNrcykgZGlyZWN0bGx5IHNvIHlvdSBjYW4gY3JlYXRlIHdoYXRl dmVyIGRpc3RyaWJ1dGlvbgogICAgICAgICAgICBydWxlIHlvdSB3aXNoLCBhbmQgaWYgYSBzaW5n bGUgZGlzayBmYWlscyB5b3UganVzdCByZXBsYWNlCiAgICAgICAgICAgIGl0IGFuZCB3aGljaCBv YnZpb3VzbHkgaGF2ZSB0aGUgZGF0YSByZXBsaWNhdGVkIGZyb20KICAgICAgICAgICAgYW5vdGhl ci4gVGhlIG9ubHkgZG93bnNpZGUgb2YgdXNpbmcgaW4gdGhpcyB3YXkgaXMgdGhhdCB0aGUKICAg ICAgICAgICAgcmVwbGljYXRpb24gZGF0YSB3aWxsIGJlIGZsb3cgYWNjcm9zcyBhbGwgc2VydmVy cyBidXQgdGhhdAogICAgICAgICAgICBpcyBub3QgbXVjaCBhIGJpZyBpc3N1ZS48L3A+CiAgICAg ICAgICA8cD5BbnlvbmUgY2FuIGVsYWJvcmF0ZSBhYm91dCBVc2luZyBSQUlEICsgR2x1c3RlckZT IGFuZCBKQk9ECiAgICAgICAgICAgICsgR2x1c3RlckZTLjwvcD4KICAgICAgICAgIDxwPlRoYW5r czxicj4KICAgICAgICAgICAgUmVnYXJkczxicj4KICAgICAgICAgICAgRmVybmFuZG88YnI+CiAg ICAgICAgICA8L3A+CiAgICAgICAgICA8YnI+CiAgICAgICAgICA8ZGl2IGNsYXNzPSJtb3otY2l0 ZS1wcmVmaXgiPk9uIDA3LzA4LzIwMTcgMDM6NDYsIERldmluIEFjb3N0YQogICAgICAgICAgICB3 cm90ZTo8YnI+CiAgICAgICAgICA8L2Rpdj4KICAgICAgICAgIDxibG9ja3F1b3RlIHR5cGU9ImNp dGUiCmNpdGU9Im1pZDpDQU5DR0tFcDRYR3MwVStRczc4ZUVtcUNOdHZwTFktQXpqYjVEY0doWjl5 aUtUQkVFZndAbWFpbC5nbWFpbC5jb20iPgogICAgICAgICAgICA8c3R5bGU+Ym9keXtmb250LWZh bWlseTpIZWx2ZXRpY2EsQXJpYWw7Zm9udC1zaXplOjEzcHh9PC9zdHlsZT4KICAgICAgICAgICAg PGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9udCIKICAgICAgICAgICAgICBzdHlsZT0iY29sb3I6cmdi KDAsMCwwKTttYXJnaW46MHB4Ij48Zm9udCBmYWNlPSJJbnB1dAogICAgICAgICAgICAgICAgTW9u byI+PGJyPgogICAgICAgICAgICAgIDwvZm9udD48L2Rpdj4KICAgICAgICAgICAgPGRpdiBpZD0i Ymxvb3BfY3VzdG9tZm9udCIKICAgICAgICAgICAgICBzdHlsZT0iY29sb3I6cmdiKDAsMCwwKTtt YXJnaW46MHB4Ij48Zm9udCBmYWNlPSJJbnB1dAogICAgICAgICAgICAgICAgTW9ubyI+TW9hY2ly LDwvZm9udD48L2Rpdj4KICAgICAgICAgICAgPGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9udCIKICAg ICAgICAgICAgICBzdHlsZT0iY29sb3I6cmdiKDAsMCwwKTttYXJnaW46MHB4Ij48Zm9udCBmYWNl PSJJbnB1dAogICAgICAgICAgICAgICAgTW9ubyI+PGJyPgogICAgICAgICAgICAgIDwvZm9udD48 L2Rpdj4KICAgICAgICAgICAgPGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9udCIKICAgICAgICAgICAg ICBzdHlsZT0iY29sb3I6cmdiKDAsMCwwKTttYXJnaW46MHB4Ij48Zm9udCBmYWNlPSJJbnB1dAog ICAgICAgICAgICAgICAgTW9ubyI+SSBoYXZlIHJlY2VudGx5IGluc3RhbGxlZCBtdWx0aXBsZSBS ZWQgSGF0CiAgICAgICAgICAgICAgICBWaXJ0dWFsaXphdGlvbiBob3N0cyBmb3Igc2V2ZXJhbCBk aWZmZXJlbnQgY29tcGFuaWVzLAogICAgICAgICAgICAgICAgYW5kIGhhdmUgZGVhbHQgd2l0aCB0 aGUgUmVkIEhhdCBTdXBwb3J0IFRlYW0gaW4gZGVwdGgKICAgICAgICAgICAgICAgIGFib3V0IG9w dGltYWwgY29uZmlndXJhdGlvbiBpbiByZWdhcmRzIHRvIHNldHRpbmcgdXAKICAgICAgICAgICAg ICAgIEdsdXN0ZXJGUyBtb3N0IGVmZmljaWVudGx5IGFuZCBJIHdhbnRlZCB0byBzaGFyZSB3aXRo CiAgICAgICAgICAgICAgICB5b3Ugd2hhdCBJIGxlYXJuZWQuPC9mb250PjwvZGl2PgogICAgICAg ICAgICA8ZGl2IGlkPSJibG9vcF9jdXN0b21mb250IgogICAgICAgICAgICAgIHN0eWxlPSJjb2xv cjpyZ2IoMCwwLDApO21hcmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0CiAgICAgICAgICAgICAg ICBNb25vIj48YnI+CiAgICAgICAgICAgICAgPC9mb250PjwvZGl2PgogICAgICAgICAgICA8ZGl2 IGlkPSJibG9vcF9jdXN0b21mb250IgogICAgICAgICAgICAgIHN0eWxlPSJjb2xvcjpyZ2IoMCww LDApO21hcmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0CiAgICAgICAgICAgICAgICBNb25vIj5J biBnZW5lcmFsIFJlZCBIYXQgVmlydHVhbGl6YXRpb24gdGVhbSBmcm93bnMgdXBvbgogICAgICAg ICAgICAgICAgdXNpbmcgZWFjaCBESVNLIG9mIHRoZSBzeXN0ZW0gYXMganVzdCBhIEpCT0QsIHN1 cmUgdGhlcmUKICAgICAgICAgICAgICAgIGlzIHNvbWUgcHJvdGVjdGlvbiBieSBoYXZpbmcgdGhl IGRhdGEgcmVwbGljYXRlZCwKICAgICAgICAgICAgICAgIGhvd2V2ZXIsIHRoZSByZWNvbW1lbmRh dGlvbiBpcyB0byB1c2UgUkFJRCA2IChwcmVmZXJyZWQpCiAgICAgICAgICAgICAgICBvciBSQUlE LTUsIG9yIGF0IGxlYXN0IFJBSUQtMSBhdCB0aGUgdmVyeSBsZWFzdC48L2ZvbnQ+PC9kaXY+CiAg ICAgICAgICAgIDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZvbnQiCiAgICAgICAgICAgICAgc3R5bGU9 ImNvbG9yOnJnYigwLDAsMCk7bWFyZ2luOjBweCI+PGZvbnQgZmFjZT0iSW5wdXQKICAgICAgICAg ICAgICAgIE1vbm8iPjxicj4KICAgICAgICAgICAgICA8L2ZvbnQ+PC9kaXY+CiAgICAgICAgICAg IDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJtYXJnaW46MHB4Ij48Zm9udAogICAg ICAgICAgICAgICAgZmFjZT0iSW5wdXQgTW9ubyI+SGVyZSBpcyB0aGXCoGRpcmVjdCBxdW90ZSBm cm9tIFJlZCBIYXQKICAgICAgICAgICAgICAgIHdoZW4gSSBhc2tlZCBhYm91dCBSQUlEIGFuZCBC cmlja3M6PC9mb250PjwvZGl2PgogICAgICAgICAgICA8ZGl2IGlkPSJibG9vcF9jdXN0b21mb250 IiBzdHlsZT0ibWFyZ2luOjBweCI+PGZvbnQKICAgICAgICAgICAgICAgIGZhY2U9IklucHV0IE1v bm8iPjxpPjxicj4KICAgICAgICAgICAgICAgIDwvaT48L2ZvbnQ+PC9kaXY+CiAgICAgICAgICAg IDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJtYXJnaW46MHB4Ij48Zm9udAogICAg ICAgICAgICAgICAgZmFjZT0iSW5wdXQgTW9ubyI+PGk+IkEgdHlwaWNhbCBHbHVzdGVyIGNvbmZp Z3VyYXRpb24KICAgICAgICAgICAgICAgICAgd291bGQgdXNlIFJBSUQgdW5kZXJuZWF0aCB0aGUg YnJpY2tzLiBSQUlEIDYgaXMgbW9zdAogICAgICAgICAgICAgICAgICB0eXBpY2FsIGFzIGl0IGdp dmVzIHlvdSAyIGRpc2sgZmFpbHVyZSBwcm90ZWN0aW9uLCBidXQKICAgICAgICAgICAgICAgICAg UkFJRCA1IGNvdWxkIGJlIHVzZWQgdG9vLsKgT25jZSB5b3UgaGF2ZSB0aGUgUkFJRGVkCiAgICAg ICAgICAgICAgICAgIGJyaWNrcywgeW91J2QgdGhlbiBhcHBseSB0aGUgZGVzaXJlZCByZXBsaWNh dGlvbiBvbgogICAgICAgICAgICAgICAgICB0b3Agb2YgdGhhdC4gVGhlIG1vc3QgcG9wdWxhciB3 YXkgb2YgZG9pbmcgdGhpcyB3b3VsZAogICAgICAgICAgICAgICAgICBiZSBkaXN0cmlidXRlZCBy ZXBsaWNhdGVkIHdpdGggMnggcmVwbGljYXRpb24uwqBJbgogICAgICAgICAgICAgICAgICBnZW5l cmFsIHlvdSdsbCBnZXQgYmV0dGVyIHBlcmZvcm1hbmNlIHdpdGggbGFyZ2VyCiAgICAgICAgICAg ICAgICAgIGJyaWNrcy4gMTIgZHJpdmVzIGlzIG9mdGVuIGEgc3dlZXQgc3BvdC4gQW5vdGhlcgog ICAgICAgICAgICAgICAgICBvcHRpb24gd291bGQgYmUgdG8gY3JlYXRlIGEgc2VwYXJhdGUgdGll ciB1c2luZyBhbGwKICAgICAgICAgICAgICAgICAgU1NE4oCZcy7igJ3CoDwvaT48L2ZvbnQ+PC9k aXY+CiAgICAgICAgICAgIDxkaXYgaWQ9ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJtYXJnaW46 MHB4Ij48YnI+CiAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICA8ZGl2IGlkPSJibG9vcF9j dXN0b21mb250IiBzdHlsZT0ibWFyZ2luOjBweCI+PGZvbnQKICAgICAgICAgICAgICAgIGZhY2U9 IklucHV0IE1vbm8iPjxpPkluIG9yZGVyIHRvIFNTRMKgdGllcmluZyBmcm9tIG15CiAgICAgICAg ICAgICAgICAgIHVuZGVyc3RhbmRpbmcgeW91IHdvdWxkIG5lZWQgMSB4IE5WTWUgZHJpdmUgaW4g ZWFjaAogICAgICAgICAgICAgICAgICBzZXJ2ZXIsIG9yIDQgeCBTU0QgaG90IHRpZXIgKGl0IG5l ZWRzIHRvIGJlCiAgICAgICAgICAgICAgICAgIGRpc3RyaWJ1dGVkLCByZXBsaWNhdGVkIGZvciB0 aGUgaG90IHRpZXIgaWYgbm90IHVzaW5nCiAgICAgICAgICAgICAgICAgIE5WTUUpLiBTbyB3aXRo IHlvdSBvbmx5IGhhdmluZyAxIFNTRCBkcml2ZSBpbiBlYWNoCiAgICAgICAgICAgICAgICAgIHNl cnZlcizCoEnigJlkIHN1Z2dlc3QgbWF5YmUgbG9va2luZyBpbnRvIHRoZSBOVk1FCiAgICAgICAg ICAgICAgICAgIG9wdGlvbi7CoDwvaT48L2ZvbnQ+PC9kaXY+CiAgICAgICAgICAgIDxkaXYgaWQ9 ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJtYXJnaW46MHB4Ij48Zm9udAogICAgICAgICAgICAg ICAgZmFjZT0iSW5wdXQgTW9ubyI+PGk+PGJyPgogICAgICAgICAgICAgICAgPC9pPjwvZm9udD48 L2Rpdj4KICAgICAgICAgICAgPGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9Im1hcmdp bjowcHgiPjxmb250CiAgICAgICAgICAgICAgICBmYWNlPSJJbnB1dCBNb25vIj48aT5TaW5jZSB5 b3VyIHVzaW5nIG9ubHkgMy1zZXJ2ZXJzLAogICAgICAgICAgICAgICAgICB3aGF0IEnigJlkIHBy b2JhYmx5IHN1Z2dlc3QgaXMgdG8gZG8gKDIgUmVwbGljYXMgKwogICAgICAgICAgICAgICAgICBB cmJpdGVyIE5vZGUpLCB0aGlzIHNldHVwIGFjdHVhbGx5IGRvZXNu4oCZdCByZXF1aXJlIHRoZQog ICAgICAgICAgICAgICAgICAzcmQgc2VydmVyIHRvIGhhdmUgYmlnIGRyaXZlcyBhdCBhbGwgYXMg aXQgb25seSBzdG9yZXMKICAgICAgICAgICAgICAgICAgbWV0YS1kYXRhIGFib3V0IHRoZSBmaWxl cyBhbmQgbm90IGFjdHVhbGx5IGEgZnVsbAogICAgICAgICAgICAgICAgICBjb3B5LsKgPC9pPjwv Zm9udD48L2Rpdj4KICAgICAgICAgICAgPGRpdiBpZD0iYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9 Im1hcmdpbjowcHgiPjxmb250CiAgICAgICAgICAgICAgICBmYWNlPSJJbnB1dCBNb25vIj48aT48 YnI+CiAgICAgICAgICAgICAgICA8L2k+PC9mb250PjwvZGl2PgogICAgICAgICAgICA8ZGl2IGlk PSJibG9vcF9jdXN0b21mb250IiBzdHlsZT0ibWFyZ2luOjBweCI+PGZvbnQKICAgICAgICAgICAg ICAgIGZhY2U9IklucHV0IE1vbm8iPjxpPlBsZWFzZSBzZWUgdGhlIGF0dGFjaGVkIGRvY3VtZW50 CiAgICAgICAgICAgICAgICAgIHRoYXQgd2FzIGdpdmVuIHRvIG1lIGJ5IFJlZCBIYXQgdG8gZ2V0 IG1vcmUKICAgICAgICAgICAgICAgICAgaW5mb3JtYXRpb24gb24gdGhpcy4gSG9wZSB0aGlzIGlu Zm9ybWF0aW9uIGhlbHBzIHlvdS48L2k+PC9mb250PjwvZGl2PgogICAgICAgICAgICA8ZGl2IGlk PSJibG9vcF9jdXN0b21mb250IiBzdHlsZT0ibWFyZ2luOjBweCI+PGZvbnQKICAgICAgICAgICAg ICAgIGZhY2U9IklucHV0IE1vbm8iPjxpPjxicj4KICAgICAgICAgICAgICAgIDwvaT48L2ZvbnQ+ PC9kaXY+CiAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgPGRpdiBpZD0iYmxvb3Bfc2lnbl8x NTAyMDg3Mzc2NzI1NDY5MTg0IiBjbGFzcz0iYmxvb3Bfc2lnbiI+PHNwYW4KICAgICAgICAgICAg ICAgIHN0eWxlPSJmb250LWZhbWlseTonaGVsdmV0aWNhCiAgICAgICAgICAgICAgICBOZXVlJyxo ZWx2ZXRpY2E7Zm9udC1zaXplOjE0cHgiPi0tPC9zcGFuPjxicgogICAgICAgICAgICAgICAgc3R5 bGU9ImZvbnQtZmFtaWx5OidoZWx2ZXRpY2EKICAgICAgICAgICAgICAgIE5ldWUnLGhlbHZldGlj YTtmb250LXNpemU6MTRweCI+CiAgICAgICAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxfc2lnbmF0 dXJlIiBzdHlsZT0iZm9udC1mYW1pbHk6J2hlbHZldGljYQogICAgICAgICAgICAgICAgTmV1ZScs aGVsdmV0aWNhO2ZvbnQtc2l6ZToxNHB4Ij4KICAgICAgICAgICAgICAgIDxkaXYgZGlyPSJsdHIi PgogICAgICAgICAgICAgICAgICA8ZGl2Pjxicj4KICAgICAgICAgICAgICAgICAgPC9kaXY+CiAg ICAgICAgICAgICAgICAgIDxkaXY+RGV2aW4gQWNvc3RhLCBSSENBLCBSSFZDQTwvZGl2PgogICAg ICAgICAgICAgICAgICA8ZGl2PlJlZCBIYXQgQ2VydGlmaWVkIEFyY2hpdGVjdDwvZGl2PgogICAg ICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgIDwvZGl2 PgogICAgICAgICAgICA8YnI+CiAgICAgICAgICAgIDxwIGNsYXNzPSJhaXJtYWlsX29uIj5PbiBB dWd1c3QgNiwgMjAxNyBhdCA3OjI5OjI5IFBNLAogICAgICAgICAgICAgIE1vYWNpciBGZXJyZWly YSAoPGEKICAgICAgICAgICAgICAgIGhyZWY9Im1haWx0bzptb2FjaXJmZXJyZWlyYUBob3RtYWls LmNvbSIKICAgICAgICAgICAgICAgIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+bW9hY2lyZmVycmVp cmFAaG90bWFpbC5jb208L2E+KQogICAgICAgICAgICAgIHdyb3RlOjwvcD4KICAgICAgICAgICAg PGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSIgY2xhc3M9ImNsZWFuX2JxIj48c3Bhbj4KICAgICAgICAg ICAgICAgIDxkaXYgZGlyPSJsdHIiPgogICAgICAgICAgICAgICAgICA8ZGl2PgogICAgICAgICAg ICAgICAgICAgIDx0aXRsZT48L3RpdGxlPgogICAgICAgICAgICAgICAgICAgIDxkaXYgaWQ9ImRp dnRhZ2RlZmF1bHR3cmFwcGVyIgpzdHlsZT0iZm9udC1zaXplOjEycHQ7Y29sb3I6IzAwMDAwMDtm b250LWZhbWlseTpDYWxpYnJpLEhlbHZldGljYSxzYW5zLXNlcmlmIgogICAgICAgICAgICAgICAg ICAgICAgZGlyPSJsdHIiPgogICAgICAgICAgICAgICAgICAgICAgPHA+PHNwYW4+SSBhbSB3aWxs aW5nIHRvIGFzc2VtYmxlIGEgb1ZpcnQgInBvZCIsCiAgICAgICAgICAgICAgICAgICAgICAgICAg bWFkZSBvZiAzIHNlcnZlcnMsIGVhY2ggd2l0aCAyIENQVSBzb2NrZXRzIG9mCiAgICAgICAgICAg ICAgICAgICAgICAgICAgMTIgY29yZXMsIDI1NkdCIFJBTSwgNyBIREQgMTBLLCAxIFNTRC4gVGhl CiAgICAgICAgICAgICAgICAgICAgICAgICAgaWRlYSBpcyB0byB1c2UgR2x1c3RlckZTIHRvIHBy b3ZpZGUgSEEgZm9yIHRoZQogICAgICAgICAgICAgICAgICAgICAgICAgIFZNcy4gVGhlIDMgc2Vy dmVycyBoYXZlIGEgZHVhbCA0MEdiIE5JQyBhbmQgYQogICAgICAgICAgICAgICAgICAgICAgICAg IGR1YWwgMTBHYiBOSUMuIFNvIG15IGludGVudGlvbiBpcyB0byBjcmVhdGUgYQogICAgICAgICAg ICAgICAgICAgICAgICAgIGxvb3AgbGlrZSBhIHNlcnZlciB0cmlhbmdsZSB1c2luZyB0aGUgNDBH YgogICAgICAgICAgICAgICAgICAgICAgICAgIE5JQ3MgZm9yIHZpcnR1YWxpemF0aW9uIGZpbGVz IChWTXMgLnFjb3cyKQogICAgICAgICAgICAgICAgICAgICAgICAgIGFjY2VzcyBhbmQgdG8gbW92 ZSBWTXMgYXJvdW5kIHRoZSBwb2QgKGVhc3QKICAgICAgICAgICAgICAgICAgICAgICAgICAvd2Vz dCB0cmFmZmljKSB3aGlsZSB1c2luZyB0aGUgMTBHYiBpbnRlcmZhY2VzCiAgICAgICAgICAgICAg ICAgICAgICAgICAgZm9yIGdpdmluZyBzZXJ2aWNlcyB0byB0aGUgb3V0c2lkZSB3b3JsZAogICAg ICAgICAgICAgICAgICAgICAgICAgIChub3J0aC9zb3V0aCB0cmFmZmljKS48L3NwYW4+PC9wPgog ICAgICAgICAgICAgICAgICAgICAgPHA+PGJyPgogICAgICAgICAgICAgICAgICAgICAgPC9wPgog ICAgICAgICAgICAgICAgICAgICAgPHA+VGhpcyBzYWlkLCBteSBmaXJzdCBxdWVzdGlvbiBpczog SG93IHNob3VsZCBJCiAgICAgICAgICAgICAgICAgICAgICAgIGRlcGxveSBHbHVzdGVyRlMgaW4g c3VjaCBvVmlydCBzY2VuYXJpbz8gTXkKICAgICAgICAgICAgICAgICAgICAgICAgcXVlc3Rpb25z IGFyZTo8L3A+CiAgICAgICAgICAgICAgICAgICAgICA8cD48YnI+CiAgICAgICAgICAgICAgICAg ICAgICA8L3A+CiAgICAgICAgICAgICAgICAgICAgICA8cD4xIC0gU2hvdWxkIEkgY3JlYXRlIDMg UkFJRCAoaS5lLjogUkFJRCA1KSwgb25lCiAgICAgICAgICAgICAgICAgICAgICAgIG9uIGVhY2gg b1ZpcnQgbm9kZSwgYW5kIHRoZW4gY3JlYXRlIGEgR2x1c3RlckZTCiAgICAgICAgICAgICAgICAg ICAgICAgIHVzaW5nIHRoZW0/PC9wPgogICAgICAgICAgICAgICAgICAgICAgPHA+MiAtIEluc3Rl YWQsIHNob3VsZCBJIGNyZWF0ZSBhIEpCT0QgYXJyYXkgbWFkZQogICAgICAgICAgICAgICAgICAg ICAgICBvZiBhbGwgc2VydmVyJ3MgZGlza3M/PC9wPgogICAgICAgICAgICAgICAgICAgICAgPHA+ MyAtIFdoYXQgaXMgdGhlIGJlc3QgR2x1c3RlciBjb25maWd1cmF0aW9uIHRvCiAgICAgICAgICAg ICAgICAgICAgICAgIHByb3ZpZGUgZm9yIEhBIHdoaWxlIG5vdCBjb25zdW1pbmcgdG9vIG11Y2gg ZGlzawogICAgICAgICAgICAgICAgICAgICAgICBzcGFjZT88YnI+CiAgICAgICAgICAgICAgICAg ICAgICA8L3A+CiAgICAgICAgICAgICAgICAgICAgICA8cD40IC0gRG9lcyBhIG9WaXJ0IGh5cGVy dmlzb3IgcG9kIGxpa2UgSSBhbQogICAgICAgICAgICAgICAgICAgICAgICBwbGFubmluZyB0byBi dWlsZCwgYW5kIHRoZSB2aXJ0dWFsaXphdGlvbgogICAgICAgICAgICAgICAgICAgICAgICBlbnZp cm9ubWVudCwgYmVuZWZpdHMgZnJvbSB0aWVyaW5nIHdoZW4gdXNpbmcgYQogICAgICAgICAgICAg ICAgICAgICAgICBTU0QgZGlzaz8gQW5kIHllcywgd2lsbCBHbHVzdGVyIGRvIGl0IGJ5IGRlZmF1 bHQKICAgICAgICAgICAgICAgICAgICAgICAgb3IgSSBoYXZlIHRvIGNvbmZpZ3VyZSBpdCB0byBk byBzbz88L3A+CiAgICAgICAgICAgICAgICAgICAgICA8cD48YnI+CiAgICAgICAgICAgICAgICAg ICAgICA8L3A+CiAgICAgICAgICAgICAgICAgICAgICA8cD5BdCB0aGUgYm90dG9tIGxpbmUsIHdo YXQgaXMgdGhlIGdvb2QgcHJhY3RpY2UKICAgICAgICAgICAgICAgICAgICAgICAgZm9yIHVzaW5n IEdsdXN0ZXJGUyBpbiBzbWFsbCBwb2RzIGZvcgogICAgICAgICAgICAgICAgICAgICAgICBlbnRl cnByaXNlcz88YnI+CiAgICAgICAgICAgICAgICAgICAgICA8L3A+CiAgICAgICAgICAgICAgICAg ICAgICA8cD48YnI+CiAgICAgICAgICAgICAgICAgICAgICA8L3A+CiAgICAgICAgICAgICAgICAg ICAgICA8cD5Zb3Ugb3Bpbmlvbi9mZWVkYmFjayB3aWxsIGJlIHJlYWxseQogICAgICAgICAgICAg ICAgICAgICAgICBhcHByZWNpYXRlZCE8L3A+CiAgICAgICAgICAgICAgICAgICAgICA8cD5Nb2Fj aXI8YnI+CiAgICAgICAgICAgICAgICAgICAgICA8L3A+CiAgICAgICAgICAgICAgICAgICAgPC9k aXY+CiAgICAgICAgICAgICAgICAgICAgX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX18gPGJyPgogICAgICAgICAgICAgICAgICAgIFVzZXJzIG1haWxpbmcgbGlz dCA8YnI+CiAgICAgICAgICAgICAgICAgICAgPGEgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9y ZyIKICAgICAgICAgICAgICAgICAgICAgIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+VXNlcnNAb3Zp cnQub3JnPC9hPiA8YnI+CiAgICAgICAgICAgICAgICAgICAgPGEKICAgICAgICAgICAgICAgICAg ICAgIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyIK ICAgICAgICAgICAgICAgICAgICAgIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+aHR0cDovL2xpc3Rz Lm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPgogICAgICAgICAgICAgICAgICAg IDxicj4KICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICA8L2Rpdj4KICAg ICAgICAgICAgICA8L3NwYW4+PC9ibG9ja3F1b3RlPgogICAgICAgICAgICA8YnI+CiAgICAgICAg ICAgIDxmaWVsZHNldCBjbGFzcz0ibWltZUF0dGFjaG1lbnRIZWFkZXIiPjwvZmllbGRzZXQ+CiAg ICAgICAgICAgIDxicj4KICAgICAgICAgICAgPHByZSB3cmFwPSIiPl9fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fClVzZXJzIG1haWxpbmcgbGlzdAo8YSBjbGFz cz0ibW96LXR4dC1saW5rLWFiYnJldmlhdGVkIiBocmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQub3Jn IiBtb3otZG8tbm90LXNlbmQ9InRydWUiPlVzZXJzQG92aXJ0Lm9yZzwvYT4KPGEgY2xhc3M9Im1v ei10eHQtbGluay1mcmVldGV4dCIgaHJlZj0iaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFu L2xpc3RpbmZvL3VzZXJzIiBtb3otZG8tbm90LXNlbmQ9InRydWUiPmh0dHA6Ly9saXN0cy5vdmly dC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VyczwvYT4KPC9wcmU+CiAgICAgICAgICA8L2Jsb2Nr cXVvdGU+CiAgICAgICAgICA8YnI+CiAgICAgICAgICA8YnI+CiAgICAgICAgICA8ZmllbGRzZXQg Y2xhc3M9Im1pbWVBdHRhY2htZW50SGVhZGVyIj48L2ZpZWxkc2V0PgogICAgICAgICAgPGJyPgog ICAgICAgICAgPHByZSB3cmFwPSIiPl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fClVzZXJzIG1haWxpbmcgbGlzdAo8YSBjbGFzcz0ibW96LXR4dC1saW5rLWFi YnJldmlhdGVkIiBocmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQub3JnIiBtb3otZG8tbm90LXNlbmQ9 InRydWUiPlVzZXJzQG92aXJ0Lm9yZzwvYT4KPGEgY2xhc3M9Im1vei10eHQtbGluay1mcmVldGV4 dCIgaHJlZj0iaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzIiBt b3otZG8tbm90LXNlbmQ9InRydWUiPmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0 aW5mby91c2VyczwvYT4KPC9wcmU+CiAgICAgICAgPC9ibG9ja3F1b3RlPgogICAgICAgIDxicj4K ICAgICAgPC9ibG9ja3F1b3RlPgogICAgICA8YnI+CiAgICA8L2Jsb2NrcXVvdGU+CiAgICA8YnI+ CiAgICA8cHJlIGNsYXNzPSJtb3otc2lnbmF0dXJlIiBjb2xzPSI3MiI+LS0gClJlY29naXplciBH cm91cCBHbWJICgpEci5yZXIubmF0LiBFcmVrbGUgTWFncmFkemUKTGVhZCBCaWcgRGF0YSBFbmdp bmVlcmluZyAmYW1wOyBEZXZPcHMKUmhlaW53ZXJrYWxsZWUgMiwgNTMyMjcgQm9ubgpUZWw6ICs0 OSAyMjggMjk5NzQ1NTUKCkUtTWFpbCA8YSBjbGFzcz0ibW96LXR4dC1saW5rLWFiYnJldmlhdGVk IiBocmVmPSJtYWlsdG86ZXJla2xlLm1hZ3JhZHplQHJlY29naXplci5kZSI+ZXJla2xlLm1hZ3Jh ZHplQHJlY29naXplci5kZTwvYT4KV2ViOiA8YSBjbGFzcz0ibW96LXR4dC1saW5rLWFiYnJldmlh dGVkIiBocmVmPSJodHRwOi8vd3d3LnJlY29naXplci5jb20iPnd3dy5yZWNvZ2l6ZXIuY29tPC9h PgogClJlY29naXplciBhdWYgTGlua2VkSW4gPGEgY2xhc3M9Im1vei10eHQtbGluay1mcmVldGV4 dCIgaHJlZj0iaHR0cHM6Ly93d3cubGlua2VkaW4uY29tL2NvbXBhbnktYmV0YS8xMDAzOTE4Mi8i Pmh0dHBzOi8vd3d3LmxpbmtlZGluLmNvbS9jb21wYW55LWJldGEvMTAwMzkxODIvPC9hPgpGb2xn ZW4gU2llIHVucyBhdWYgVHdpdHRlciA8YSBjbGFzcz0ibW96LXR4dC1saW5rLWZyZWV0ZXh0IiBo cmVmPSJodHRwczovL3R3aXR0ZXIuY29tL3JlY29naXplciI+aHR0cHM6Ly90d2l0dGVyLmNvbS9y ZWNvZ2l6ZXI8L2E+CiAKLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KUmVjb2dpemVyIEdyb3VwIEdtYkgKR2VzY2jDpGZ0c2bD vGhyZXI6IE9saXZlciBIYWJpc2NoLCBDYXJzdGVuIEtyZXV0emUKSGFuZGVsc3JlZ2lzdGVyOiBB bXRzZ2VyaWNodCBCb25uIEhSQiAyMDcyNApTaXR6IGRlciBHZXNlbGxzY2hhZnQ6IEJvbm47IFVT dC1JRC1Oci46IERFMjk0MTk1OTkzCiAKRGllc2UgRS1NYWlsIGVudGjDpGx0IHZlcnRyYXVsaWNo ZSB1bmQvb2RlciByZWNodGxpY2ggZ2VzY2jDvHR6dGUgSW5mb3JtYXRpb25lbi4KV2VubiBTaWUg bmljaHQgZGVyIHJpY2h0aWdlIEFkcmVzc2F0IHNpbmQgb2RlciBkaWVzZSBFLU1haWwgaXJydMO8 bWxpY2ggZXJoYWx0ZW4gaGFiZW4sCmluZm9ybWllcmVuIFNpZSBiaXR0ZSBzb2ZvcnQgZGVuIEFi c2VuZGVyIHVuZCBsw7ZzY2hlbiBTaWUgZGllc2UgTWFpbC4KRGFzIHVuZXJsYXVidGUgS29waWVy ZW4gc293aWUgZGllIHVuYmVmdWd0ZSBXZWl0ZXJnYWJlIGRpZXNlciBNYWlsIHVuZCBkZXIgZGFy aW4gZW50aGFsdGVuZW4gSW5mb3JtYXRpb25lbiBpc3QgbmljaHQgZ2VzdGF0dGV0LjwvcHJlPgog IDwvYm9keT4KPC9odG1sPgoKLS0tLS0tLS0tLS0tLS0wREMwOTQwMUU0RURDMjVFNjhDRDYyMDgt LQo= --===============3381618162231106505==-- From fernando.frediani at upx.com Mon Aug 7 21:03:48 2017 Content-Type: multipart/mixed; boundary="===============2306564352647578365==" MIME-Version: 1.0 From: FERNANDO FREDIANI To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Mon, 07 Aug 2017 18:03:40 -0300 Message-ID: <47b0f3b5-a836-d5c2-7cf4-c0147aa3948f@upx.com> In-Reply-To: b61c0164-c933-8204-a949-0aa303983548@recogizer.de --===============2306564352647578365== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------D415F827B0DC677345E35156 Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 8bit Thanks for the detailed answer Erekle. I conclude that it is worth in any scenario to have a arbiter node in = order to avoid wasting more disk space to RAID X + Gluster Replication = on the top of it. The cost seems much lower if you consider running = costs of the whole storage and compare it with the cost to build the = arbiter node. Even having a fully redundant arbiter service with 2 nodes = would make it wort on a larger deployment. Regards Fernando On 07/08/2017 17:07, Erekle Magradze wrote: > > Hi Fernando (sorry for misspelling your name, I used a different = > keyboard), > > So let's go with the following scenarios: > > 1. Let's say you have two servers (replication factor is 2), i.e. two = > bricks per volume, in this case it is strongly recommended to have the = > arbiter node, the metadata storage that will guarantee avoiding the = > split brain situation, in this case for arbiter you don't even need a = > disk with lots of space, it's enough to have a tiny ssd but hosted on = > a separate server. Advantage of such setup is that you don't need the = > RAID 1 for each brick, you have the metadata information stored in = > arbiter node and brick replacement is easy. > > 2. If you have odd number of bricks (let's say 3, i.e. replication = > factor is 3) in your volume and you didn't create the arbiter node as = > well as you didn't configure the quorum, in this case the entire load = > for keeping the consistency of the volume resides on all 3 servers, = > each of them is important and each brick contains key information, = > they need to cross-check each other (that's what people usually do = > with the first try of gluster :) ), in this case replacing a brick is = > a big pain and in this case RAID 1 is a good option to have (that's = > the disadvantage, i.e. loosing the space and not having the JBOD = > option) advantage is that you don't have the to have additional = > arbiter node. > > 3. You have odd number of bricks and configured arbiter node, in this = > case you can easily go with JBOD, however a good practice would be to = > have a RAID 1 for arbiter disks (tiny 128GB SSD-s ar perfectly = > sufficient for volumes with 10s of TB-s in size.) > > That's basically it > > The rest about the reliability and setup scenarios you can find in = > gluster documentation, especially look for quorum and arbiter node = > configs+options. > > Cheers > > Erekle > > P.S. What I was mentioning, regarding a good practice is mostly = > related to the operations of gluster not installation or deployment, = > i.e. not the conceptual understanding of gluster (conceptually it's a = > JBOD system). > > On 08/07/2017 05:41 PM, FERNANDO FREDIANI wrote: >> >> Thanks for the clarification Erekle. >> >> However I get surprised with this way of operating from GlusterFS as = >> it adds another layer of complexity to the system (either a hardware = >> or software RAID) before the gluster config and increase the system's = >> overall costs. >> >> An important point to consider is: In RAID configuration you already = >> have space 'wasted' in order to build redundancy (either RAID 1, 5, = >> or 6). Then when you have GlusterFS on the top of several RAIDs you = >> have again more data replicated so you end up with the same data = >> consuming more space in a group of disks and again on the top of = >> several RAIDs depending on the Gluster configuration you have (in a = >> RAID 1 config the same data is replicated 4 times). >> >> Yet another downside of having a RAID (specially RAID 5 or 6) is that = >> it reduces considerably the write speeds as each group of disks will = >> end up having the write speed of a single disk as all other disks of = >> that group have to wait for each other to write as well. >> >> Therefore if Gluster already replicates data why does it create this = >> big pain you mentioned if the data is replicated somewhere else, can = >> still be retrieved to both serve clients and reconstruct the = >> equivalent disk when it is replaced ? >> >> Fernando >> >> >> On 07/08/2017 10:26, Erekle Magradze wrote: >>> >>> Hi Frenando, >>> >>> Here is my experience, if you consider a particular hard drive as a = >>> brick for gluster volume and it dies, i.e. it becomes not accessible = >>> it's a huge hassle to discard that brick and exchange with another = >>> one, since gluster some tries to access that broken brick and it's = >>> causing (at least it cause for me) a big pain, therefore it's better = >>> to have a RAID as brick, i.e. have RAID 1 (mirroring) for each = >>> brick, in this case if the disk is down you can easily exchange it = >>> and rebuild the RAID without going offline, i.e switching off the = >>> volume doing brick manipulations and switching it back on. >>> >>> Cheers >>> >>> Erekle >>> >>> >>> On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote: >>>> >>>> For any RAID 5 or 6 configuration I normally follow a simple gold = >>>> rule which gave good results so far: >>>> - up to 4 disks RAID 5 >>>> - 5 or more disks RAID 6 >>>> >>>> However I didn't really understand well the recommendation to use = >>>> any RAID with GlusterFS. I always thought that GlusteFS likes to = >>>> work in JBOD mode and control the disks (bricks) directlly so you = >>>> can create whatever distribution rule you wish, and if a single = >>>> disk fails you just replace it and which obviously have the data = >>>> replicated from another. The only downside of using in this way is = >>>> that the replication data will be flow accross all servers but that = >>>> is not much a big issue. >>>> >>>> Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS. >>>> >>>> Thanks >>>> Regards >>>> Fernando >>>> >>>> >>>> On 07/08/2017 03:46, Devin Acosta wrote: >>>>> >>>>> Moacir, >>>>> >>>>> I have recently installed multiple Red Hat Virtualization hosts = >>>>> for several different companies, and have dealt with the Red Hat = >>>>> Support Team in depth about optimal configuration in regards to = >>>>> setting up GlusterFS most efficiently and I wanted to share with = >>>>> you what I learned. >>>>> >>>>> In general Red Hat Virtualization team frowns upon using each DISK = >>>>> of the system as just a JBOD, sure there is some protection by = >>>>> having the data replicated, however, the recommendation is to use = >>>>> RAID 6 (preferred) or RAID-5, or at least RAID-1 at the very least. >>>>> >>>>> Here is the direct quote from Red Hat when I asked about RAID and = >>>>> Bricks: >>>>> / >>>>> / >>>>> /"A typical Gluster configuration would use RAID underneath the = >>>>> bricks. RAID 6 is most typical as it gives you 2 disk failure = >>>>> protection, but RAID 5 could be used too. Once you have the RAIDed = >>>>> bricks, you'd then apply the desired replication on top of that. = >>>>> The most popular way of doing this would be distributed replicated = >>>>> with 2x replication. In general you'll get better performance with = >>>>> larger bricks. 12 drives is often a sweet spot. Another option = >>>>> would be to create a separate tier using all SSD=E2=80=99s.=E2=80=9D / >>>>> >>>>> /In order to SSD tiering from my understanding you would need 1 x = >>>>> NVMe drive in each server, or 4 x SSD hot tier (it needs to be = >>>>> distributed, replicated for the hot tier if not using NVME). So = >>>>> with you only having 1 SSD drive in each server, I=E2=80=99d suggest = maybe = >>>>> looking into the NVME option. / >>>>> / >>>>> / >>>>> /Since your using only 3-servers, what I=E2=80=99d probably suggest i= s to = >>>>> do (2 Replicas + Arbiter Node), this setup actually doesn=E2=80=99t = >>>>> require the 3rd server to have big drives at all as it only stores = >>>>> meta-data about the files and not actually a full copy. / >>>>> / >>>>> / >>>>> /Please see the attached document that was given to me by Red Hat = >>>>> to get more information on this. Hope this information helps you./ >>>>> / >>>>> / >>>>> >>>>> -- >>>>> >>>>> Devin Acosta, RHCA, RHVCA >>>>> Red Hat Certified Architect >>>>> >>>>> On August 6, 2017 at 7:29:29 PM, Moacir Ferreira = >>>>> (moacirferreira(a)hotmail.com ) = >>>>> wrote: >>>>> >>>>>> I am willing to assemble a oVirt "pod", made of 3 servers, each = >>>>>> with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The = >>>>>> idea is to use GlusterFS to provide HA for the VMs. The 3 servers = >>>>>> have a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to = >>>>>> create a loop like a server triangle using the 40Gb NICs for = >>>>>> virtualization files (VMs .qcow2) access and to move VMs around = >>>>>> the pod (east /west traffic) while using the 10Gb interfaces for = >>>>>> giving services to the outside world (north/south traffic). >>>>>> >>>>>> >>>>>> This said, my first question is: How should I deploy GlusterFS in = >>>>>> such oVirt scenario? My questions are: >>>>>> >>>>>> >>>>>> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt = >>>>>> node, and then create a GlusterFS using them? >>>>>> >>>>>> 2 - Instead, should I create a JBOD array made of all server's disks? >>>>>> >>>>>> 3 - What is the best Gluster configuration to provide for HA = >>>>>> while not consuming too much disk space? >>>>>> >>>>>> 4 - Does a oVirt hypervisor pod like I am planning to build, and = >>>>>> the virtualization environment, benefits from tiering when using = >>>>>> a SSD disk? And yes, will Gluster do it by default or I have to = >>>>>> configure it to do so? >>>>>> >>>>>> >>>>>> At the bottom line, what is the good practice for using GlusterFS = >>>>>> in small pods for enterprises? >>>>>> >>>>>> >>>>>> You opinion/feedback will be really appreciated! >>>>>> >>>>>> Moacir >>>>>> >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>> >> > > -- = > Recogizer Group GmbH > > Dr.rer.nat. Erekle Magradze > Lead Big Data Engineering & DevOps > Rheinwerkallee 2, 53227 Bonn > Tel: +49 228 29974555 > > E-Mailerekle.magradze(a)recogizer.de > Web:www.recogizer.com > = > Recogizer auf LinkedInhttps://www.linkedin.com/company-beta/10039182/ > Folgen Sie uns auf Twitterhttps://twitter.com/recogizer > = > ----------------------------------------------------------------- > Recogizer Group GmbH > Gesch=C3=A4ftsf=C3=BChrer: Oliver Habisch, Carsten Kreutze > Handelsregister: Amtsgericht Bonn HRB 20724 > Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993 > = > Diese E-Mail enth=C3=A4lt vertrauliche und/oder rechtlich gesch=C3=BCtzte= Informationen. > Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=C3=BCmli= ch erhalten haben, > informieren Sie bitte sofort den Absender und l=C3=B6schen Sie diese Mail. > Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und de= r darin enthaltenen Informationen ist nicht gestattet. --------------D415F827B0DC677345E35156 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit
Thanks for the detailed answer Erekle.

I conclude that it is worth in any scenario to have a arbiter node in order to avoid wasting more disk space to RAID X + Gluster Replication on the top of it. The cost seems much lower if you consider running costs of the whole storage and compare it with the cost to build the arbiter node. Even having a fully redundant arbiter service with 2 nodes would make it wort on a larger deployment.

Regards
Fernando

On 07/08/2017 17:07, Erekle Magradze wrote:

Hi Fernando (sorry for misspelling your name, I used a different keyboard),

So let's go with the following scenarios:

1. Let's say you have two servers (replication factor is 2), i.e. two bricks per volume, in this case it is strongly recommended to have the arbiter node, the metadata storage that will guarantee avoiding the split brain situation, in this case for arbiter you don't even need a disk with lots of space, it's enough to have a tiny ssd but hosted on a separate server. Advantage of such setup is that you don't need the RAID 1 for each brick, you have the metadata information stored in arbiter node and brick replacement is easy.

2. If you have odd number of bricks (let's say 3, i.e. replication factor is 3) in your volume and you didn't create the arbiter node as well as you didn't configure the quorum, in this case the entire load for keeping the consistency of the volume resides on all 3 servers, each of them is important and each brick contains key information, they need to cross-check each other (that's what people usually do with the first try of gluster :) ), in this case replacing a brick is a big pain and in this case RAID 1 is a good option to have (that's the disadvantage, i.e. loosing the space and not having the JBOD option) advantage is that you don't have the to have additional arbiter node.

3. You have odd number of bricks and configured arbiter node, in this case you can easily go with JBOD, however a good practice would be to have a RAID 1 for arbiter disks (tiny 128GB SSD-s ar perfectly sufficient for volumes with 10s of TB-s in size.)

That's basically it

The rest about the reliability and setup scenarios you can find in gluster documentation, especially look for quorum and arbiter node configs+options.

Cheers

Erekle
P.S. What I was mentioning, regarding a good practice is mostly related to the operations of gluster not installation or deployment, i.e. not the conceptual understanding of gluster (conceptually it's a JBOD system).

On 08/07/2017 05:41 PM, FERNANDO FREDIANI wrote:

Thanks for the clarification Erekle.

However I get surprised with this way of operating from GlusterFS as it adds another layer of complexity to the system (either a hardware or software RAID) before the gluster config and increase the system's overall costs.

An important point to consider is: In RAID configuration you already have space 'wasted' in order to build redundancy (either RAID 1, 5, or 6). Then when you have GlusterFS on the top of several RAIDs you have again more data replicated so you end up with the same data consuming more space in a group of disks and again on the top of several RAIDs depending on the Gluster configuration you have (in a RAID 1 config the same data is replicated 4 times).

Yet another downside of having a RAID (specially RAID 5 or 6) is that it reduces considerably the write speeds as each group of disks will end up having the write speed of a single disk as all other disks of that group have to wait for each other to write as well.

Therefore if Gluster already replicates data why does it create this big pain you mentioned if the data is replicated somewhere else, can still be retrieved to both serve clients and reconstruct the equivalent disk when it is replaced ?

Fernando

On 07/08/2017 10:26, Erekle Magradze wrote:

Hi Frenando,

Here is my experience, if you consider a particular hard drive as a brick for gluster volume and it dies, i.e. it becomes not accessible it's a huge hassle to discard that brick and exchange with another one, since gluster some tries to access that broken brick and it's causing (at least it cause for me) a big pain, therefore it's better to have a RAID as brick, i.e. have RAID 1 (mirroring) for each brick, in this case if the disk is down you can easily exchange it and rebuild the RAID without going offline, i.e switching off the volume doing brick manipulations and switching it back on.

Cheers

Erekle

On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote:

For any RAID 5 or 6 configuration I normally follow a simple gold rule which gave good results so far:
- up to 4 disks RAID 5
- 5 or more disks RAID 6

However I didn't really understand well the recommendation to use any RAID with GlusterFS. I always thought that GlusteFS likes to work in JBOD mode and control the disks (bricks) directlly so you can create whatever distribution rule you wish, and if a single disk fails you just replace it and which obviously have the data replicated from another. The only downside of using in this way is that the replication data will be flow accross all servers but that is not much a big issue.

Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS.

Thanks
Regards
Fernando

On 07/08/2017 03:46, Devin Acosta wrote:

Hi Colin,

Take a look on Devin's response. Also, read the doc he shared that gives= =3D some hints on how to deploy Gluster.

It is more like that if you want high-performance you should have the br= =3D icks created as RAID (5 or 6) by the server's disk controller and them= =3D assemble a JBOD GlusterFS. The attached document is Gluster specific and n= =3D ot for oVirt. But at this point I think that having SSD will not be a plus as using the RAID controller Gluster wi= =3D ll not be aware of the SSD. Regarding the OS, my idea is to have a RAID 1, = =3D made of 2 low cost HDDs, to install it.

So far, based on the information received I should create a si= =3D ngle RAID 5 or 6 on each server and then use this disk as a brick to create= =3D my Gluster cluster, made of 2 replicas + 1 arbiter. What is new for me= =3D is the detail that the arbiter does not need a lot of space as it only keeps meta data.

Thanks for your response!

Moacir

From: Colin Coe <c= olin=3D .coe(a)gmail.com>
Sent: Monday, August 7, 2017 12:41 PM
To: Moacir Ferreira
Cc: users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices

Hi

I just thought that you'd do hardware RAID if you had the controller o= =3D r JBOD if you didn't. In hindsight, a server with 40Gbps NICs is pret= =3D ty likely to have a hardware RAID controller. I've never done JBOD wi= =3D th hardware RAID. I think having a single gluster brick on hardware JBOD would be riskier than multiple bricks, each= =3D on a single disk, but thats not based on anything other than my prejudices= =3D .

I thought gluster tiering was for the most frequently accessed files, = =3D in which case all the VMs disks would end up in the hot tier. However= =3D , I have been wrong before...

I just wanted to know where the OS was going as I didn't see it mentio= =3D ned in the OP. Normally, I'd have the OS on a RAID1 but in your case = =3D thats a lot of wasted disk.

Honestly, I think Yaniv's answer was far better than my own and made t= =3D he important point about having an arbiter.

Thanks

On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreir= a =3D <m= oacirf=3D erreira(a)hotmail.com> wrote:

Hi Colin,

I am in Portugal, so sorry for this late response. It is qu= =3D ite confusing for me, please consider:

1 - What if the RAID is done by the server's disk controller, no= =3D t by software?

2 - For JBOD I am just using gdeploy to deploy it. However, I am = =3D not using the oVirt node GUI to do this.

3 - As the VM .qcow2 files are quite big, tiering would only= =3D help if made by an intelligent system that uses SSD for chunks of data not= =3D for the entire .qcow2 file. But I guess this is a problem everybody else h= =3D as. So, Do you know how tiering works in Gluster?

4 - I am putting the OS on the first disk. However, would you do di= =3D fferently?

Moacir

From:<= /b> =3D Colin Coe <coli=3D n.coe(a)gmail.com>
Sent: Monday, August 7, 2017 4:48 AM
To: Moacir Ferreira
Cc: users= (a)ovirt=3D .org
Subject: Re: [ovirt-users] Good practices

1) RAID5 may be a performance hit-

2) I'd be inclined to do this as JBOD by creating a= d=3D istributed disperse volume on each server. Something like

echo gluster volume create dispersevol disperse-data 5 redundancy 2 \<= =3D /div>
$(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e "se= =3D rver${SERVER}:/brick/brick-${SERVER}${BRICK}/brick \c"; done; don= =3D e)

3) I think the above.

4) Gluster does support tiering, but IIRC you'd need the same number o= =3D f SSD as spindle drives. There may be another way to use the SSD as a= =3D fast cache.

Where are you putting the OS?

Hope I understood the question...

Thanks

On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferrei= ra=3D <m= oacirf=3D erreira(a)hotmail.com> wrote:

I am willing to assemble a oVirt "pod", made of 3 server= =3D s, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The id= =3D ea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual= =3D 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop like a server triangle using the 40Gb NICs for virtual= =3D ization files (VMs .qcow2) access and to move VMs around the pod (east /wes= =3D t traffic) while using the 10Gb interfaces for giving services to the outsi= =3D de world (north/south traffic).

This said, my first question is: How should I deploy GlusterFS in such o= =3D Virt scenario? My questions are:

1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and t= =3D hen create a GlusterFS using them?

2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not c= =3D onsuming too much disk space?

4 - Does a oVirt hypervisor pod like I am planning to build, and the vir= =3D tualization environment, benefits from tiering when using a SSD disk? And y= =3D es, will Gluster do it by default or I have to configure it to do so?

At the bottom line, what is the good practice for using GlusterFS in sma= =3D ll pods for enterprises?

You opinion/feedback will be really appreciated!

Moacir

_______________________________________________
Users mailing list
Users(a)ovirt.or= g http://lists.ovirt.org/mailman/listinfo/users<=3D br>

--_000_VI1P190MB02854F23232C41E2AD3C7223C8B50VI1P190MB0285EURP_-- --===============5017108521715457652== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" LS1fMDAwX1ZJMVAxOTBNQjAyODU0RjIzMjMyQzQxRTJBRDNDNzIyM0M4QjUwVkkxUDE5ME1CMDI4 NUVVUlBfCkNvbnRlbnQtVHlwZTogdGV4dC9wbGFpbjsgY2hhcnNldD0iaXNvLTg4NTktMSIKQ29u dGVudC1UcmFuc2Zlci1FbmNvZGluZzogcXVvdGVkLXByaW50YWJsZQoKSGkgQ29saW4sCgoKVGFr ZSBhIGxvb2sgb24gRGV2aW4ncyByZXNwb25zZS4gQWxzbywgcmVhZCB0aGUgZG9jIGhlIHNoYXJl ZCB0aGF0IGdpdmVzIHNvPQptZSBoaW50cyBvbiBob3cgdG8gZGVwbG95IEdsdXN0ZXIuCgoKSXQg aXMgbW9yZSBsaWtlIHRoYXQgaWYgeW91IHdhbnQgaGlnaC1wZXJmb3JtYW5jZSB5b3Ugc2hvdWxk IGhhdmUgdGhlIGJyaWNrPQpzIGNyZWF0ZWQgYXMgUkFJRCAoNSBvciA2KSBieSB0aGUgc2VydmVy J3MgZGlzayBjb250cm9sbGVyIGFuZCB0aGVtIGFzc2VtYmw9CmUgYSBKQk9EIEdsdXN0ZXJGUy4g VGhlIGF0dGFjaGVkIGRvY3VtZW50IGlzIEdsdXN0ZXIgc3BlY2lmaWMgYW5kIG5vdCBmb3Igbz0K VmlydC4gQnV0IGF0IHRoaXMgcG9pbnQgSSB0aGluayB0aGF0IGhhdmluZyBTU0Qgd2lsbCBub3Qg YmUgYSBwbHVzIGFzIHVzaW5nPQogdGhlIFJBSUQgY29udHJvbGxlciBHbHVzdGVyIHdpbGwgbm90 IGJlIGF3YXJlIG9mIHRoZSBTU0QuIFJlZ2FyZGluZyB0aGUgT1M9CiwgbXkgaWRlYSBpcyB0byBo YXZlIGEgUkFJRCAxLCBtYWRlIG9mIDIgbG93IGNvc3QgSEREcywgdG8gaW5zdGFsbCBpdC4KCgpT byBmYXIsIGJhc2VkIG9uIHRoZSBpbmZvcm1hdGlvbiByZWNlaXZlZCBJIHNob3VsZCBjcmVhdGUg YSBzaW5nbGUgUkFJRCA1IG89CnIgNiBvbiBlYWNoIHNlcnZlciBhbmQgdGhlbiB1c2UgdGhpcyBk aXNrIGFzIGEgYnJpY2sgdG8gY3JlYXRlIG15IEdsdXN0ZXIgYz0KbHVzdGVyLCBtYWRlIG9mIDIg cmVwbGljYXMgKyAxIGFyYml0ZXIuIFdoYXQgaXMgbmV3IGZvciBtZSBpcyB0aGUgZGV0YWlsIHRo PQphdCB0aGUgYXJiaXRlciBkb2VzIG5vdCBuZWVkIGEgbG90IG9mIHNwYWNlIGFzIGl0IG9ubHkg a2VlcHMgbWV0YSBkYXRhLgoKClRoYW5rcyBmb3IgeW91ciByZXNwb25zZSEKCk1vYWNpcgoKX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX18KRnJvbTogQ29saW4gQ29lIDxjb2xpbi5jb2VA Z21haWwuY29tPgpTZW50OiBNb25kYXksIEF1Z3VzdCA3LCAyMDE3IDEyOjQxIFBNClRvOiBNb2Fj aXIgRmVycmVpcmEKQ2M6IHVzZXJzQG92aXJ0Lm9yZwpTdWJqZWN0OiBSZTogW292aXJ0LXVzZXJz XSBHb29kIHByYWN0aWNlcwoKSGkKCkkganVzdCB0aG91Z2h0IHRoYXQgeW91J2QgZG8gaGFyZHdh cmUgUkFJRCBpZiB5b3UgaGFkIHRoZSBjb250cm9sbGVyIG9yIEpCTz0KRCBpZiB5b3UgZGlkbid0 LiAgSW4gaGluZHNpZ2h0LCBhIHNlcnZlciB3aXRoIDQwR2JwcyBOSUNzIGlzIHByZXR0eSBsaWtl bHkgPQp0byBoYXZlIGEgaGFyZHdhcmUgUkFJRCBjb250cm9sbGVyLiAgSSd2ZSBuZXZlciBkb25l IEpCT0Qgd2l0aCBoYXJkd2FyZSBSQUk9CkQuICBJIHRoaW5rIGhhdmluZyBhIHNpbmdsZSBnbHVz dGVyIGJyaWNrIG9uIGhhcmR3YXJlIEpCT0Qgd291bGQgYmUgcmlza2llcj0KIHRoYW4gbXVsdGlw bGUgYnJpY2tzLCBlYWNoIG9uIGEgc2luZ2xlIGRpc2ssIGJ1dCB0aGF0cyBub3QgYmFzZWQgb24g YW55dGhpPQpuZyBvdGhlciB0aGFuIG15IHByZWp1ZGljZXMuCgpJIHRob3VnaHQgZ2x1c3RlciB0 aWVyaW5nIHdhcyBmb3IgdGhlIG1vc3QgZnJlcXVlbnRseSBhY2Nlc3NlZCBmaWxlcywgaW4gd2g9 CmljaCBjYXNlIGFsbCB0aGUgVk1zIGRpc2tzIHdvdWxkIGVuZCB1cCBpbiB0aGUgaG90IHRpZXIu ICBIb3dldmVyLCBJIGhhdmUgYj0KZWVuIHdyb25nIGJlZm9yZS4uLgoKSSBqdXN0IHdhbnRlZCB0 byBrbm93IHdoZXJlIHRoZSBPUyB3YXMgZ29pbmcgYXMgSSBkaWRuJ3Qgc2VlIGl0IG1lbnRpb25l ZCBpPQpuIHRoZSBPUC4gIE5vcm1hbGx5LCBJJ2QgaGF2ZSB0aGUgT1Mgb24gYSBSQUlEMSBidXQg aW4geW91ciBjYXNlIHRoYXRzIGEgbG89CnQgb2Ygd2FzdGVkIGRpc2suCgpIb25lc3RseSwgSSB0 aGluayBZYW5pdidzIGFuc3dlciB3YXMgZmFyIGJldHRlciB0aGFuIG15IG93biBhbmQgbWFkZSB0 aGUgaW09CnBvcnRhbnQgcG9pbnQgYWJvdXQgaGF2aW5nIGFuIGFyYml0ZXIuCgpUaGFua3MKCk9u IE1vbiwgQXVnIDcsIDIwMTcgYXQgNTo1NiBQTSwgTW9hY2lyIEZlcnJlaXJhIDxtb2FjaXJmZXJy ZWlyYUBob3RtYWlsLmNvbT0KPG1haWx0bzptb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbT4+IHdy b3RlOgoKSGkgQ29saW4sCgoKSSBhbSBpbiBQb3J0dWdhbCwgc28gc29ycnkgZm9yIHRoaXMgbGF0 ZSByZXNwb25zZS4gSXQgaXMgcXVpdGUgY29uZnVzaW5nIGZvPQpyIG1lLCBwbGVhc2UgY29uc2lk ZXI6CgoxIC0gV2hhdCBpZiB0aGUgUkFJRCBpcyBkb25lIGJ5IHRoZSBzZXJ2ZXIncyBkaXNrIGNv bnRyb2xsZXIsIG5vdCBieSBzb2Z0d2E9CnJlPwoKCjIgLSBGb3IgSkJPRCBJIGFtIGp1c3QgdXNp bmcgZ2RlcGxveSB0byBkZXBsb3kgaXQuIEhvd2V2ZXIsIEkgYW0gbm90IHVzaW5nID0KdGhlIG9W aXJ0IG5vZGUgR1VJIHRvIGRvIHRoaXMuCgoKMyAtIEFzIHRoZSBWTSAucWNvdzIgZmlsZXMgYXJl IHF1aXRlIGJpZywgdGllcmluZyB3b3VsZCBvbmx5IGhlbHAgaWYgbWFkZSBiPQp5IGFuIGludGVs bGlnZW50IHN5c3RlbSB0aGF0IHVzZXMgU1NEIGZvciBjaHVua3Mgb2YgZGF0YSBub3QgZm9yIHRo ZSBlbnRpcmU9CiAucWNvdzIgZmlsZS4gQnV0IEkgZ3Vlc3MgdGhpcyBpcyBhIHByb2JsZW0gZXZl cnlib2R5IGVsc2UgaGFzLiBTbywgRG8geW91ID0Ka25vdyBob3cgdGllcmluZyB3b3JrcyBpbiBH bHVzdGVyPwoKCjQgLSBJIGFtIHB1dHRpbmcgdGhlIE9TIG9uIHRoZSBmaXJzdCBkaXNrLiBIb3dl dmVyLCB3b3VsZCB5b3UgZG8gZGlmZmVyZW50bD0KeT8KCgpNb2FjaXIKCl9fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fCkZyb206IENvbGluIENvZSA8Y29saW4uY29lQGdtYWlsLmNvbTxt YWlsdG86Y29saW4uY29lQGdtYWlsLmNvbT4+ClNlbnQ6IE1vbmRheSwgQXVndXN0IDcsIDIwMTcg NDo0OCBBTQpUbzogTW9hY2lyIEZlcnJlaXJhCkNjOiB1c2Vyc0BvdmlydC5vcmc8bWFpbHRvOnVz ZXJzQG92aXJ0Lm9yZz4KU3ViamVjdDogUmU6IFtvdmlydC11c2Vyc10gR29vZCBwcmFjdGljZXMK CjEpIFJBSUQ1IG1heSBiZSBhIHBlcmZvcm1hbmNlIGhpdC0KCjIpIEknZCBiZSBpbmNsaW5lZCB0 byBkbyB0aGlzIGFzIEpCT0QgYnkgY3JlYXRpbmcgYSBkaXN0cmlidXRlZCBkaXNwZXJzZSB2bz0K bHVtZSBvbiBlYWNoIHNlcnZlci4gIFNvbWV0aGluZyBsaWtlCgplY2hvIGdsdXN0ZXIgdm9sdW1l IGNyZWF0ZSBkaXNwZXJzZXZvbCBkaXNwZXJzZS1kYXRhIDUgcmVkdW5kYW5jeSAyIFwKJChmb3Ig U0VSVkVSIGluIGEgYiBjOyBkbyBmb3IgQlJJQ0sgaW4gJChzZXEgMSA1KTsgZG8gZWNobyAtZSAi c2VydmVyJHtTRVJWPQpFUn06L2JyaWNrL2JyaWNrLSR7U0VSVkVSfSR7QlJJQ0t9L2JyaWNrIFxj IjsgZG9uZTsgZG9uZSkKCjMpIEkgdGhpbmsgdGhlIGFib3ZlLgoKNCkgR2x1c3RlciBkb2VzIHN1 cHBvcnQgdGllcmluZywgYnV0IElJUkMgeW91J2QgbmVlZCB0aGUgc2FtZSBudW1iZXIgb2YgU1NE PQogYXMgc3BpbmRsZSBkcml2ZXMuICBUaGVyZSBtYXkgYmUgYW5vdGhlciB3YXkgdG8gdXNlIHRo ZSBTU0QgYXMgYSBmYXN0IGNhY2g9CmUuCgpXaGVyZSBhcmUgeW91IHB1dHRpbmcgdGhlIE9TPwoK SG9wZSBJIHVuZGVyc3Rvb2QgdGhlIHF1ZXN0aW9uLi4uCgpUaGFua3MKCk9uIFN1biwgQXVnIDYs IDIwMTcgYXQgMTA6NDkgUE0sIE1vYWNpciBGZXJyZWlyYSA8bW9hY2lyZmVycmVpcmFAaG90bWFp bC5jbz0KbTxtYWlsdG86bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb20+PiB3cm90ZToKCkkgYW0g d2lsbGluZyB0byBhc3NlbWJsZSBhIG9WaXJ0ICJwb2QiLCBtYWRlIG9mIDMgc2VydmVycywgZWFj aCB3aXRoIDIgQ1BVID0Kc29ja2V0cyBvZiAxMiBjb3JlcywgMjU2R0IgUkFNLCA3IEhERCAxMEss IDEgU1NELiBUaGUgaWRlYSBpcyB0byB1c2UgR2x1c3RlPQpyRlMgdG8gcHJvdmlkZSBIQSBmb3Ig dGhlIFZNcy4gVGhlIDMgc2VydmVycyBoYXZlIGEgZHVhbCA0MEdiIE5JQyBhbmQgYSBkdWE9Cmwg MTBHYiBOSUMuIFNvIG15IGludGVudGlvbiBpcyB0byBjcmVhdGUgYSBsb29wIGxpa2UgYSBzZXJ2 ZXIgdHJpYW5nbGUgdXNpbj0KZyB0aGUgNDBHYiBOSUNzIGZvciB2aXJ0dWFsaXphdGlvbiBmaWxl cyAoVk1zIC5xY293MikgYWNjZXNzIGFuZCB0byBtb3ZlIFZNPQpzIGFyb3VuZCB0aGUgcG9kIChl YXN0IC93ZXN0IHRyYWZmaWMpIHdoaWxlIHVzaW5nIHRoZSAxMEdiIGludGVyZmFjZXMgZm9yIGc9 Cml2aW5nIHNlcnZpY2VzIHRvIHRoZSBvdXRzaWRlIHdvcmxkIChub3J0aC9zb3V0aCB0cmFmZmlj KS4KCgpUaGlzIHNhaWQsIG15IGZpcnN0IHF1ZXN0aW9uIGlzOiBIb3cgc2hvdWxkIEkgZGVwbG95 IEdsdXN0ZXJGUyBpbiBzdWNoIG9WaXI9CnQgc2NlbmFyaW8/IE15IHF1ZXN0aW9ucyBhcmU6CgoK MSAtIFNob3VsZCBJIGNyZWF0ZSAzIFJBSUQgKGkuZS46IFJBSUQgNSksIG9uZSBvbiBlYWNoIG9W aXJ0IG5vZGUsIGFuZCB0aGVuPQogY3JlYXRlIGEgR2x1c3RlckZTIHVzaW5nIHRoZW0/CgoyIC0g SW5zdGVhZCwgc2hvdWxkIEkgY3JlYXRlIGEgSkJPRCBhcnJheSBtYWRlIG9mIGFsbCBzZXJ2ZXIn cyBkaXNrcz8KCjMgLSBXaGF0IGlzIHRoZSBiZXN0IEdsdXN0ZXIgY29uZmlndXJhdGlvbiB0byBw cm92aWRlIGZvciBIQSB3aGlsZSBub3QgY29ucz0KdW1pbmcgdG9vIG11Y2ggZGlzayBzcGFjZT8K CjQgLSBEb2VzIGEgb1ZpcnQgaHlwZXJ2aXNvciBwb2QgbGlrZSBJIGFtIHBsYW5uaW5nIHRvIGJ1 aWxkLCBhbmQgdGhlIHZpcnR1YT0KbGl6YXRpb24gZW52aXJvbm1lbnQsIGJlbmVmaXRzIGZyb20g dGllcmluZyB3aGVuIHVzaW5nIGEgU1NEIGRpc2s/IEFuZCB5ZXMsPQogd2lsbCBHbHVzdGVyIGRv IGl0IGJ5IGRlZmF1bHQgb3IgSSBoYXZlIHRvIGNvbmZpZ3VyZSBpdCB0byBkbyBzbz8KCgpBdCB0 aGUgYm90dG9tIGxpbmUsIHdoYXQgaXMgdGhlIGdvb2QgcHJhY3RpY2UgZm9yIHVzaW5nIEdsdXN0 ZXJGUyBpbiBzbWFsbCA9CnBvZHMgZm9yIGVudGVycHJpc2VzPwoKCllvdSBvcGluaW9uL2ZlZWRi YWNrIHdpbGwgYmUgcmVhbGx5IGFwcHJlY2lhdGVkIQoKTW9hY2lyCgpfX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpVc2VycyBtYWlsaW5nIGxpc3QKVXNlcnNA b3ZpcnQub3JnPG1haWx0bzpVc2Vyc0BvdmlydC5vcmc+Cmh0dHA6Ly9saXN0cy5vdmlydC5vcmcv bWFpbG1hbi9saXN0aW5mby91c2VycwoKCgoKLS1fMDAwX1ZJMVAxOTBNQjAyODU0RjIzMjMyQzQx RTJBRDNDNzIyM0M4QjUwVkkxUDE5ME1CMDI4NUVVUlBfCkNvbnRlbnQtVHlwZTogdGV4dC9odG1s OyBjaGFyc2V0PSJpc28tODg1OS0xIgpDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiBxdW90ZWQt cHJpbnRhYmxlCgo8aHRtbD4KPGhlYWQ+CjxtZXRhIGh0dHAtZXF1aXY9M0QiQ29udGVudC1UeXBl IiBjb250ZW50PTNEInRleHQvaHRtbDsgY2hhcnNldD0zRGlzby04ODU5LT0KMSI+CjxzdHlsZSB0 eXBlPTNEInRleHQvY3NzIiBzdHlsZT0zRCJkaXNwbGF5Om5vbmU7Ij48IS0tIFAge21hcmdpbi10 b3A6MDttYXJnaT0Kbi1ib3R0b206MDt9IC0tPjwvc3R5bGU+CjwvaGVhZD4KPGJvZHkgZGlyPTNE Imx0ciI+CjxkaXYgaWQ9M0QiZGl2dGFnZGVmYXVsdHdyYXBwZXIiIHN0eWxlPTNEImZvbnQtc2l6 ZToxMnB0O2NvbG9yOiMwMDAwMDA7Zm9udD0KLWZhbWlseTpDYWxpYnJpLEhlbHZldGljYSxzYW5z LXNlcmlmOyIgZGlyPTNEImx0ciI+CjxwPkhpIENvbGluLDwvcD4KPHA+PGJyPgo8L3A+CjxwPlRh a2UgYSBsb29rIG9uIERldmluJ3MgcmVzcG9uc2UuIEFsc28sIHJlYWQgdGhlIGRvYyBoZSBzaGFy ZWQgdGhhdCBnaXZlcz0KIHNvbWUgaGludHMgb24gaG93IHRvIGRlcGxveSBHbHVzdGVyLjwvcD4K PHA+PGJyPgo8L3A+CjxwPkl0IGlzIG1vcmUgbGlrZSB0aGF0IGlmIHlvdSB3YW50IGhpZ2gtcGVy Zm9ybWFuY2UgeW91IHNob3VsZCBoYXZlIHRoZSBicj0KaWNrcyBjcmVhdGVkIGFzIFJBSUQgKDUg b3IgNikmbmJzcDtieSB0aGUgc2VydmVyJ3MgZGlzayBjb250cm9sbGVyIGFuZCB0aGVtPQogYXNz ZW1ibGUgYSBKQk9EIEdsdXN0ZXJGUy4gVGhlIGF0dGFjaGVkIGRvY3VtZW50IGlzIEdsdXN0ZXIg c3BlY2lmaWMgYW5kIG49Cm90IGZvciBvVmlydC4gQnV0IGF0IHRoaXMgcG9pbnQgSSB0aGluawog dGhhdCBoYXZpbmcgU1NEIHdpbGwgbm90IGJlIGEgcGx1cyBhcyB1c2luZyB0aGUgUkFJRCBjb250 cm9sbGVyIEdsdXN0ZXIgd2k9CmxsIG5vdCBiZSBhd2FyZSBvZiB0aGUgU1NELiBSZWdhcmRpbmcg dGhlIE9TLCBteSBpZGVhIGlzIHRvIGhhdmUgYSBSQUlEIDEsID0KbWFkZSBvZiAyIGxvdyBjb3N0 IEhERHMsIHRvIGluc3RhbGwgaXQuPC9wPgo8cD48YnI+CjwvcD4KPHA+U28gZmFyLCBiYXNlZCBv biB0aGUgaW5mb3JtYXRpb24gcmVjZWl2ZWQmbmJzcDtJIHNob3VsZCZuYnNwO2NyZWF0ZSBhIHNp PQpuZ2xlIFJBSUQgNSBvciA2IG9uIGVhY2ggc2VydmVyIGFuZCB0aGVuIHVzZSB0aGlzIGRpc2sg YXMgYSBicmljayB0byBjcmVhdGU9CiBteSBHbHVzdGVyIGNsdXN0ZXIsIG1hZGUgb2YgMiByZXBs aWNhcyAmIzQzOyAxIGFyYml0ZXIuIFdoYXQgaXMgbmV3IGZvciBtZT0KIGlzIHRoZSBkZXRhaWwg dGhhdCB0aGUgYXJiaXRlciBkb2VzIG5vdCBuZWVkCiBhIGxvdCBvZiBzcGFjZSBhcyBpdCBvbmx5 IGtlZXBzIG1ldGEgZGF0YS48L3A+CjxwPjxicj4KPC9wPgo8cD5UaGFua3MgZm9yIHlvdXIgcmVz cG9uc2UhPGJyPgo8L3A+Ck1vYWNpcjxicj4KPGJyPgo8ZGl2IHN0eWxlPTNEImNvbG9yOiByZ2Io NDksIDU1LCA1Nyk7Ij4KPGhyIHRhYmluZGV4PTNEIi0xIiBzdHlsZT0zRCJkaXNwbGF5OmlubGlu ZS1ibG9jazsgd2lkdGg6OTglIj4KPGRpdiBpZD0zRCJkaXZScGx5RndkTXNnIiBkaXI9M0QibHRy Ij48Zm9udCBzdHlsZT0zRCJmb250LXNpemU6MTFwdCIgZmFjZT0KPTNEIkNhbGlicmksIHNhbnMt c2VyaWYiIGNvbG9yPTNEIiMwMDAwMDAiPjxiPkZyb206PC9iPiBDb2xpbiBDb2UgJmx0O2NvbGlu PQouY29lQGdtYWlsLmNvbSZndDs8YnI+CjxiPlNlbnQ6PC9iPiBNb25kYXksIEF1Z3VzdCA3LCAy MDE3IDEyOjQxIFBNPGJyPgo8Yj5Ubzo8L2I+IE1vYWNpciBGZXJyZWlyYTxicj4KPGI+Q2M6PC9i PiB1c2Vyc0BvdmlydC5vcmc8YnI+CjxiPlN1YmplY3Q6PC9iPiBSZTogW292aXJ0LXVzZXJzXSBH b29kIHByYWN0aWNlczwvZm9udD4KPGRpdj4mbmJzcDs8L2Rpdj4KPC9kaXY+CjxkaXY+CjxkaXYg ZGlyPTNEImx0ciI+SGkKPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PkkganVzdCB0aG91Z2h0IHRoYXQg eW91J2QgZG8gaGFyZHdhcmUgUkFJRCBpZiB5b3UgaGFkIHRoZSBjb250cm9sbGVyIG89CnIgSkJP RCBpZiB5b3UgZGlkbid0LiZuYnNwOyBJbiBoaW5kc2lnaHQsIGEgc2VydmVyIHdpdGggNDBHYnBz IE5JQ3MgaXMgcHJldD0KdHkgbGlrZWx5IHRvIGhhdmUgYSBoYXJkd2FyZSBSQUlEIGNvbnRyb2xs ZXIuJm5ic3A7IEkndmUgbmV2ZXIgZG9uZSBKQk9EIHdpPQp0aCBoYXJkd2FyZSBSQUlELiZuYnNw OyBJIHRoaW5rIGhhdmluZyBhIHNpbmdsZQogZ2x1c3RlciBicmljayBvbiBoYXJkd2FyZSBKQk9E IHdvdWxkIGJlIHJpc2tpZXIgdGhhbiBtdWx0aXBsZSBicmlja3MsIGVhY2g9CiBvbiBhIHNpbmds ZSBkaXNrLCBidXQgdGhhdHMgbm90IGJhc2VkIG9uIGFueXRoaW5nIG90aGVyIHRoYW4gbXkgcHJl anVkaWNlcz0KLjwvZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+SSB0aG91Z2h0IGdsdXN0ZXIg dGllcmluZyB3YXMgZm9yIHRoZSBtb3N0IGZyZXF1ZW50bHkgYWNjZXNzZWQgZmlsZXMsID0KaW4g d2hpY2ggY2FzZSBhbGwgdGhlIFZNcyBkaXNrcyB3b3VsZCBlbmQgdXAgaW4gdGhlIGhvdCB0aWVy LiZuYnNwOyBIb3dldmVyPQosIEkgaGF2ZSBiZWVuIHdyb25nIGJlZm9yZS4uLjwvZGl2Pgo8ZGl2 Pjxicj4KPC9kaXY+CjxkaXY+SSBqdXN0IHdhbnRlZCB0byBrbm93IHdoZXJlIHRoZSBPUyB3YXMg Z29pbmcgYXMgSSBkaWRuJ3Qgc2VlIGl0IG1lbnRpbz0KbmVkIGluIHRoZSBPUC4mbmJzcDsgTm9y bWFsbHksIEknZCBoYXZlIHRoZSBPUyBvbiBhIFJBSUQxIGJ1dCBpbiB5b3VyIGNhc2UgPQp0aGF0 cyBhIGxvdCBvZiB3YXN0ZWQgZGlzay48L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PkhvbmVz dGx5LCBJIHRoaW5rIFlhbml2J3MgYW5zd2VyIHdhcyBmYXIgYmV0dGVyIHRoYW4gbXkgb3duIGFu ZCBtYWRlIHQ9CmhlIGltcG9ydGFudCBwb2ludCBhYm91dCBoYXZpbmcgYW4gYXJiaXRlci4mbmJz cDs8L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PlRoYW5rczwvZGl2Pgo8L2Rpdj4KPGRpdiBj bGFzcz0zRCJnbWFpbF9leHRyYSI+PGJyPgo8ZGl2IGNsYXNzPTNEImdtYWlsX3F1b3RlIj5PbiBN b24sIEF1ZyA3LCAyMDE3IGF0IDU6NTYgUE0sIE1vYWNpciBGZXJyZWlyYSA9CjxzcGFuIGRpcj0z RCJsdHIiPgombHQ7PGEgaHJlZj0zRCJtYWlsdG86bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb20i IHRhcmdldD0zRCJfYmxhbmsiPm1vYWNpcmY9CmVycmVpcmFAaG90bWFpbC5jb208L2E+Jmd0Ozwv c3Bhbj4gd3JvdGU6PGJyPgo8YmxvY2txdW90ZSBjbGFzcz0zRCJnbWFpbF9xdW90ZSIgc3R5bGU9 M0QibWFyZ2luOjAgMCAwIC44ZXg7IGJvcmRlci1sZWZ0OjE9CnB4ICNjY2Mgc29saWQ7IHBhZGRp bmctbGVmdDoxZXgiPgo8ZGl2IGRpcj0zRCJsdHIiPgo8ZGl2IGlkPTNEIm1fNDgzMTg4Njg4MzMw NTY3MjcxOGRpdnRhZ2RlZmF1bHR3cmFwcGVyIiBkaXI9M0QibHRyIiBzdHlsZT0zRCI9CmZvbnQt c2l6ZToxMnB0OyBjb2xvcjojMDAwMDAwOyBmb250LWZhbWlseTpDYWxpYnJpLEhlbHZldGljYSxz YW5zLXNlcmlmIj4KPHA+SGkgQ29saW4sPC9wPgo8cD48YnI+CjwvcD4KPHA+PHNwYW4+SSBhbSBp biBQb3J0dWdhbDwvc3Bhbj4sIHNvIHNvcnJ5IGZvciB0aGlzIGxhdGUgcmVzcG9uc2UuIEl0IGlz IHF1PQppdGUgY29uZnVzaW5nIGZvciBtZSwgcGxlYXNlIGNvbnNpZGVyOjwvcD4KPHA+PC9wPgo8 ZGl2PjxiPjxicj4KPC9iPjE8Yj4gLSA8L2I+V2hhdCBpZiB0aGUgUkFJRCBpcyBkb25lIGJ5IHRo ZSBzZXJ2ZXIncyBkaXNrIGNvbnRyb2xsZXIsIG5vPQp0IGJ5IHNvZnR3YXJlPzwvZGl2Pgo8YnI+ CjxwPjwvcD4KPHA+MiAtPGI+IDwvYj5Gb3IgSkJPRCBJIGFtIGp1c3QgdXNpbmcgZ2RlcGxveSB0 byBkZXBsb3kgaXQuIEhvd2V2ZXIsIEkgYW0gPQpub3QgdXNpbmcgdGhlIG9WaXJ0IG5vZGUgR1VJ IHRvIGRvIHRoaXMuPC9wPgo8cD48YnI+CjwvcD4KPHA+MyAtPGI+IDwvYj5BcyB0aGUgVk0gLnFj b3cyJm5ic3A7ZmlsZXMgYXJlIHF1aXRlIGJpZywgdGllcmluZyB3b3VsZCBvbmx5PQogaGVscCBp ZiBtYWRlIGJ5IGFuIGludGVsbGlnZW50IHN5c3RlbSB0aGF0IHVzZXMgU1NEIGZvciBjaHVua3Mg b2YgZGF0YSBub3Q9CiBmb3IgdGhlIGVudGlyZSAucWNvdzIgZmlsZS4gQnV0IEkgZ3Vlc3MgdGhp cyBpcyBhIHByb2JsZW0gZXZlcnlib2R5IGVsc2UgaD0KYXMuIFNvLCBEbyB5b3Uga25vdyBob3cg dGllcmluZyB3b3JrcwogaW4gR2x1c3Rlcj88YnI+CjwvcD4KPHA+PGJyPgo8L3A+CjxwPjQgLSZu YnNwO0kgYW0gcHV0dGluZyB0aGUgT1Mgb24gdGhlIGZpcnN0IGRpc2suIEhvd2V2ZXIsIHdvdWxk IHlvdSBkbyBkaT0KZmZlcmVudGx5Pzxicj4KPC9wPgo8cD48YnI+CjwvcD4KTW9hY2lyPGJyPgo8 YnI+CjxkaXYgc3R5bGU9M0QiY29sb3I6cmdiKDQ5LDU1LDU3KSI+CjxociBzdHlsZT0zRCJkaXNw bGF5OmlubGluZS1ibG9jazsgd2lkdGg6OTglIj4KPGRpdiBpZD0zRCJtXzQ4MzE4ODY4ODMzMDU2 NzI3MThkaXZScGx5RndkTXNnIiBkaXI9M0QibHRyIj48Zm9udCBzdHlsZT0zRCJmPQpvbnQtc2l6 ZToxMXB0IiBmYWNlPTNEIkNhbGlicmksIHNhbnMtc2VyaWYiIGNvbG9yPTNEIiMwMDAwMDAiPjxi PkZyb206PC9iPiA9CkNvbGluIENvZSAmbHQ7PGEgaHJlZj0zRCJtYWlsdG86Y29saW4uY29lQGdt YWlsLmNvbSIgdGFyZ2V0PTNEIl9ibGFuayI+Y29saT0Kbi5jb2VAZ21haWwuY29tPC9hPiZndDs8 YnI+CjxiPlNlbnQ6PC9iPiBNb25kYXksIEF1Z3VzdCA3LCAyMDE3IDQ6NDggQU08YnI+CjxiPlRv OjwvYj4gTW9hY2lyIEZlcnJlaXJhPGJyPgo8Yj5DYzo8L2I+IDxhIGhyZWY9M0QibWFpbHRvOnVz ZXJzQG92aXJ0Lm9yZyIgdGFyZ2V0PTNEIl9ibGFuayI+dXNlcnNAb3ZpcnQ9Ci5vcmc8L2E+PGJy Pgo8Yj5TdWJqZWN0OjwvYj4gUmU6IFtvdmlydC11c2Vyc10gR29vZCBwcmFjdGljZXM8L2ZvbnQ+ CjxkaXY+Jm5ic3A7PC9kaXY+CjwvZGl2Pgo8ZGl2Pgo8ZGl2IGRpcj0zRCJsdHIiPjEpIFJBSUQ1 IG1heSBiZSBhIHBlcmZvcm1hbmNlIGhpdC0gPGJyPgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+PHNw YW4gY2xhc3M9M0QiIj4yKSBJJ2QgYmUgaW5jbGluZWQgdG8gZG8gdGhpcyBhcyBKQk9EIGJ5IGNy ZWF0aW5nIGEgZD0KaXN0cmlidXRlZCBkaXNwZXJzZSB2b2x1bWUgb24gZWFjaCBzZXJ2ZXIuJm5i c3A7IFNvbWV0aGluZyBsaWtlCjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj5lY2hvIGdsdXN0ZXIgdm9s dW1lIGNyZWF0ZSBkaXNwZXJzZXZvbCBkaXNwZXJzZS1kYXRhIDUgcmVkdW5kYW5jeSAyIFw8PQov ZGl2Pgo8ZGl2PiQoZm9yIFNFUlZFUiBpbiBhIGIgYzsgZG8gZm9yIEJSSUNLIGluICQoc2VxIDEg NSk7IGRvIGVjaG8gLWUgJnF1b3Q7c2U9CnJ2ZXIke1NFUlZFUn06L2JyaWNrL2JyaWNrLTx3YnI+ JHtTRVJWRVJ9JHtCUklDS30vYnJpY2sgXGMmcXVvdDs7IGRvbmU7IGRvbj0KZSk8L2Rpdj4KPGRp dj48YnI+CjwvZGl2Pgo8L3NwYW4+CjxkaXY+MykgSSB0aGluayB0aGUgYWJvdmUuIDxiPjwvYj48 L2Rpdj4KPHNwYW4gY2xhc3M9M0QiIj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PjQpIEdsdXN0ZXIg ZG9lcyBzdXBwb3J0IHRpZXJpbmcsIGJ1dCBJSVJDIHlvdSdkIG5lZWQgdGhlIHNhbWUgbnVtYmVy IG89CmYgU1NEIGFzIHNwaW5kbGUgZHJpdmVzLiZuYnNwOyBUaGVyZSBtYXkgYmUgYW5vdGhlciB3 YXkgdG8gdXNlIHRoZSBTU0QgYXMgYT0KIGZhc3QgY2FjaGUuJm5ic3A7PC9kaXY+CjxkaXY+PGJy Pgo8L2Rpdj4KPGRpdj5XaGVyZSBhcmUgeW91IHB1dHRpbmcgdGhlIE9TPzwvZGl2Pgo8ZGl2Pjxi cj4KPC9kaXY+CjxkaXY+SG9wZSBJIHVuZGVyc3Rvb2QgdGhlIHF1ZXN0aW9uLi4uPC9kaXY+Cjxk aXY+PGJyPgo8L2Rpdj4KPGRpdj5UaGFua3M8L2Rpdj4KPC9zcGFuPjwvZGl2Pgo8L2Rpdj4KPHNw YW4gY2xhc3M9M0QiIj4KPGRpdiBjbGFzcz0zRCJnbWFpbF9leHRyYSI+PGJyPgo8ZGl2IGNsYXNz PTNEImdtYWlsX3F1b3RlIj5PbiBTdW4sIEF1ZyA2LCAyMDE3IGF0IDEwOjQ5IFBNLCBNb2FjaXIg RmVycmVpcmE9CiA8c3BhbiBkaXI9M0QibHRyIj4KJmx0OzxhIGhyZWY9M0QibWFpbHRvOm1vYWNp cmZlcnJlaXJhQGhvdG1haWwuY29tIiB0YXJnZXQ9M0QiX2JsYW5rIj5tb2FjaXJmPQplcnJlaXJh QGhvdG1haWwuY29tPC9hPiZndDs8L3NwYW4+IHdyb3RlOjxicj4KPGJsb2NrcXVvdGUgY2xhc3M9 M0QiZ21haWxfcXVvdGUiIHN0eWxlPTNEIm1hcmdpbjowIDAgMCAuOGV4OyBib3JkZXItbGVmdDox PQpweCAjY2NjIHNvbGlkOyBwYWRkaW5nLWxlZnQ6MWV4Ij4KPGRpdiBkaXI9M0QibHRyIj4KPGRp diBpZD0zRCJtXzQ4MzE4ODY4ODMzMDU2NzI3MThtXzI0NjA5ODU2OTE3NDY0OTgzMjJkaXZ0YWdk ZWZhdWx0d3JhcHBlciIgPQpkaXI9M0QibHRyIiBzdHlsZT0zRCJmb250LXNpemU6MTJwdDsgY29s b3I6IzAwMDAwMDsgZm9udC1mYW1pbHk6Q2FsaWJyaSxIZWw9CnZldGljYSxzYW5zLXNlcmlmIj4K PHA+PHNwYW4+SSBhbSB3aWxsaW5nIHRvIGFzc2VtYmxlIGEgb1ZpcnQgJnF1b3Q7cG9kJnF1b3Q7 LCBtYWRlIG9mIDMgc2VydmVyPQpzLCBlYWNoIHdpdGggMiBDUFUgc29ja2V0cyBvZiAxMiBjb3Jl cywgMjU2R0IgUkFNLCA3IEhERCAxMEssIDEgU1NELiBUaGUgaWQ9CmVhIGlzIHRvIHVzZSBHbHVz dGVyRlMgdG8gcHJvdmlkZSBIQSBmb3IgdGhlIFZNcy4gVGhlIDMgc2VydmVycyBoYXZlIGEgZHVh bD0KIDQwR2IgTklDIGFuZCBhIGR1YWwgMTBHYiBOSUMuIFNvIG15IGludGVudGlvbgogaXMgdG8g Y3JlYXRlIGEgbG9vcCBsaWtlIGEgc2VydmVyIHRyaWFuZ2xlIHVzaW5nIHRoZSA0MEdiIE5JQ3Mg Zm9yIHZpcnR1YWw9Cml6YXRpb24gZmlsZXMgKFZNcyAucWNvdzIpIGFjY2VzcyBhbmQgdG8gbW92 ZSBWTXMgYXJvdW5kIHRoZSBwb2QgKGVhc3QgL3dlcz0KdCB0cmFmZmljKSB3aGlsZSB1c2luZyB0 aGUgMTBHYiBpbnRlcmZhY2VzIGZvciBnaXZpbmcgc2VydmljZXMgdG8gdGhlIG91dHNpPQpkZSB3 b3JsZCAobm9ydGgvc291dGggdHJhZmZpYykuPC9zcGFuPjwvcD4KPHA+PGJyPgo8c3Bhbj48L3Nw YW4+PC9wPgo8cD5UaGlzIHNhaWQsIG15IGZpcnN0IHF1ZXN0aW9uIGlzOiBIb3cgc2hvdWxkIEkg ZGVwbG95IEdsdXN0ZXJGUyBpbiBzdWNoIG89ClZpcnQgc2NlbmFyaW8/IE15IHF1ZXN0aW9ucyBh cmU6PC9wPgo8cD48YnI+CjwvcD4KPHA+MSAtIFNob3VsZCBJIGNyZWF0ZSAzIFJBSUQgKGkuZS46 IFJBSUQgNSksIG9uZSBvbiBlYWNoIG9WaXJ0IG5vZGUsIGFuZCB0PQpoZW4gY3JlYXRlIGEgR2x1 c3RlckZTIHVzaW5nIHRoZW0/PC9wPgo8cD4yIC0gSW5zdGVhZCwgc2hvdWxkIEkgY3JlYXRlIGEg SkJPRCBhcnJheSBtYWRlIG9mIGFsbCBzZXJ2ZXIncyBkaXNrcz88L3A9Cj4KPHA+MyAtIFdoYXQg aXMgdGhlIGJlc3QgR2x1c3RlciBjb25maWd1cmF0aW9uIHRvIHByb3ZpZGUgZm9yIEhBIHdoaWxl IG5vdCBjPQpvbnN1bWluZyB0b28gbXVjaCBkaXNrIHNwYWNlPzxicj4KPC9wPgo8cD40IC0gRG9l cyBhIG9WaXJ0IGh5cGVydmlzb3IgcG9kIGxpa2UgSSBhbSBwbGFubmluZyB0byBidWlsZCwgYW5k IHRoZSB2aXI9CnR1YWxpemF0aW9uIGVudmlyb25tZW50LCBiZW5lZml0cyBmcm9tIHRpZXJpbmcg d2hlbiB1c2luZyBhIFNTRCBkaXNrPyBBbmQgeT0KZXMsIHdpbGwgR2x1c3RlciBkbyBpdCBieSBk ZWZhdWx0IG9yIEkgaGF2ZSB0byBjb25maWd1cmUgaXQgdG8gZG8gc28/PC9wPgo8cD48YnI+Cjwv cD4KPHA+QXQgdGhlIGJvdHRvbSBsaW5lLCB3aGF0IGlzIHRoZSBnb29kIHByYWN0aWNlIGZvciB1 c2luZyBHbHVzdGVyRlMgaW4gc21hPQpsbCBwb2RzIGZvciBlbnRlcnByaXNlcz88YnI+CjwvcD4K PHA+PGJyPgo8L3A+CjxwPllvdSBvcGluaW9uL2ZlZWRiYWNrIHdpbGwgYmUgcmVhbGx5IGFwcHJl Y2lhdGVkITwvcD4KPHNwYW4gY2xhc3M9M0QibV80ODMxODg2ODgzMzA1NjcyNzE4SE9FblpiIj48 Zm9udCBjb2xvcj0zRCIjODg4ODg4Ij4KPHA+TW9hY2lyPGJyPgo8L3A+CjwvZm9udD48L3NwYW4+ PC9kaXY+CjwvZGl2Pgo8YnI+Cl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzx3YnI+X19f X19fX19fX19fX19fX188YnI+ClVzZXJzIG1haWxpbmcgbGlzdDxicj4KPGEgaHJlZj0zRCJtYWls dG86VXNlcnNAb3ZpcnQub3JnIiB0YXJnZXQ9M0QiX2JsYW5rIj5Vc2Vyc0BvdmlydC5vcmc8L2E+ PGJyPQo+CjxhIGhyZWY9M0QiaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZv L3VzZXJzIiByZWw9M0Qibm9yZWZlcnJlcj0KIiB0YXJnZXQ9M0QiX2JsYW5rIj5odHRwOi8vbGlz dHMub3ZpcnQub3JnL21haWxtYW48d2JyPi9saXN0aW5mby91c2VyczwvYT48PQpicj4KPGJyPgo8 L2Jsb2NrcXVvdGU+CjwvZGl2Pgo8YnI+CjwvZGl2Pgo8L3NwYW4+PC9kaXY+CjwvZGl2Pgo8L2Rp dj4KPC9kaXY+CjwvYmxvY2txdW90ZT4KPC9kaXY+Cjxicj4KPC9kaXY+CjwvZGl2Pgo8L2Rpdj4K PC9kaXY+CjwvYm9keT4KPC9odG1sPgoKLS1fMDAwX1ZJMVAxOTBNQjAyODU0RjIzMjMyQzQxRTJB RDNDNzIyM0M4QjUwVkkxUDE5ME1CMDI4NUVVUlBfLS0K --===============5017108521715457652==-- From fernando.frediani at upx.com Tue Aug 8 02:08:34 2017 Content-Type: multipart/mixed; boundary="===============5550542770423793846==" MIME-Version: 1.0 From: FERNANDO FREDIANI To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Mon, 07 Aug 2017 23:08:31 -0300 Message-ID: In-Reply-To: VI1P190MB02854F23232C41E2AD3C7223C8B50@VI1P190MB0285.EURP190.PROD.OUTLOOK.COM --===============5550542770423793846== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Moacir, I understand that if you do this type of configuration you will be severely impacted on storage performance, specially for writes. Even if you have a Hardware RAID Controller with Writeback cache you will have a significant performance penalty and may not fully use all the resources you mentioned you have. Fernando 2017-08-07 10:03 GMT-03:00 Moacir Ferreira : > Hi Colin, > > > Take a look on Devin's response. Also, read the doc he shared that gives > some hints on how to deploy Gluster. > > > It is more like that if you want high-performance you should have the > bricks created as RAID (5 or 6) by the server's disk controller and them > assemble a JBOD GlusterFS. The attached document is Gluster specific and > not for oVirt. But at this point I think that having SSD will not be a pl= us > as using the RAID controller Gluster will not be aware of the SSD. > Regarding the OS, my idea is to have a RAID 1, made of 2 low cost HDDs, to > install it. > > > So far, based on the information received I should create a single RAID 5 > or 6 on each server and then use this disk as a brick to create my Gluster > cluster, made of 2 replicas + 1 arbiter. What is new for me is the detail > that the arbiter does not need a lot of space as it only keeps meta data. > > > Thanks for your response! > Moacir > > ------------------------------ > *From:* Colin Coe > *Sent:* Monday, August 7, 2017 12:41 PM > > *To:* Moacir Ferreira > *Cc:* users(a)ovirt.org > *Subject:* Re: [ovirt-users] Good practices > > Hi > > I just thought that you'd do hardware RAID if you had the controller or > JBOD if you didn't. In hindsight, a server with 40Gbps NICs is pretty > likely to have a hardware RAID controller. I've never done JBOD with > hardware RAID. I think having a single gluster brick on hardware JBOD > would be riskier than multiple bricks, each on a single disk, but thats n= ot > based on anything other than my prejudices. > > I thought gluster tiering was for the most frequently accessed files, in > which case all the VMs disks would end up in the hot tier. However, I ha= ve > been wrong before... > > I just wanted to know where the OS was going as I didn't see it mentioned > in the OP. Normally, I'd have the OS on a RAID1 but in your case thats a > lot of wasted disk. > > Honestly, I think Yaniv's answer was far better than my own and made the > important point about having an arbiter. > > Thanks > > On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreira < > moacirferreira(a)hotmail.com> wrote: > >> Hi Colin, >> >> >> I am in Portugal, so sorry for this late response. It is quite confusing >> for me, please consider: >> >> >> 1* - *What if the RAID is done by the server's disk controller, not by >> software? >> >> 2 - For JBOD I am just using gdeploy to deploy it. However, I am not >> using the oVirt node GUI to do this. >> >> >> 3 - As the VM .qcow2 files are quite big, tiering would only help if >> made by an intelligent system that uses SSD for chunks of data not for t= he >> entire .qcow2 file. But I guess this is a problem everybody else has. So, >> Do you know how tiering works in Gluster? >> >> >> 4 - I am putting the OS on the first disk. However, would you do >> differently? >> >> >> Moacir >> >> ------------------------------ >> *From:* Colin Coe >> *Sent:* Monday, August 7, 2017 4:48 AM >> *To:* Moacir Ferreira >> *Cc:* users(a)ovirt.org >> *Subject:* Re: [ovirt-users] Good practices >> >> 1) RAID5 may be a performance hit- >> >> 2) I'd be inclined to do this as JBOD by creating a distributed disperse >> volume on each server. Something like >> >> echo gluster volume create dispersevol disperse-data 5 redundancy 2 \ >> $(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e >> "server${SERVER}:/brick/brick-${SERVER}${BRICK}/brick \c"; done; done) >> >> 3) I think the above. >> >> 4) Gluster does support tiering, but IIRC you'd need the same number of >> SSD as spindle drives. There may be another way to use the SSD as a fast >> cache. >> >> Where are you putting the OS? >> >> Hope I understood the question... >> >> Thanks >> >> On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira < >> moacirferreira(a)hotmail.com> wrote: >> >>> I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 >>> CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use >>> GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC= and >>> a dual 10Gb NIC. So my intention is to create a loop like a server tria= ngle >>> using the 40Gb NICs for virtualization files (VMs .qcow2) access and to >>> move VMs around the pod (east /west traffic) while using the 10Gb >>> interfaces for giving services to the outside world (north/south traffi= c). >>> >>> >>> This said, my first question is: How should I deploy GlusterFS in such >>> oVirt scenario? My questions are: >>> >>> >>> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and >>> then create a GlusterFS using them? >>> >>> 2 - Instead, should I create a JBOD array made of all server's disks? >>> >>> 3 - What is the best Gluster configuration to provide for HA while not >>> consuming too much disk space? >>> >>> 4 - Does a oVirt hypervisor pod like I am planning to build, and the >>> virtualization environment, benefits from tiering when using a SSD disk? >>> And yes, will Gluster do it by default or I have to configure it to do = so? >>> >>> >>> At the bottom line, what is the good practice for using GlusterFS in >>> small pods for enterprises? >>> >>> >>> You opinion/feedback will be really appreciated! >>> >>> Moacir >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >> > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > --===============5550542770423793846== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGRpdj5Nb2FjaXIsIEkgdW5kZXJzdGFuZCB0aGF0IGlmIHlvdSBkbyB0 aGlzIHR5cGUgb2YgY29uZmlndXJhdGlvbiB5b3Ugd2lsbCBiZSBzZXZlcmVseSBpbXBhY3RlZCBv biBzdG9yYWdlIHBlcmZvcm1hbmNlLCBzcGVjaWFsbHkgZm9yIHdyaXRlcy4gRXZlbiBpZiB5b3Ug aGF2ZSBhIEhhcmR3YXJlIFJBSUQgQ29udHJvbGxlciB3aXRoIFdyaXRlYmFjayBjYWNoZSB5b3Ug d2lsbCBoYXZlIGEgc2lnbmlmaWNhbnQgcGVyZm9ybWFuY2UgcGVuYWx0eSBhbmQgbWF5IG5vdCBm dWxseSB1c2UgYWxsIHRoZSByZXNvdXJjZXMgeW91IG1lbnRpb25lZCB5b3UgaGF2ZS48YnI+PGJy PjwvZGl2PkZlcm5hbmRvPGJyPjwvZGl2PjxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+PGRp diBjbGFzcz0iZ21haWxfcXVvdGUiPjIwMTctMDgtMDcgMTA6MDMgR01ULTAzOjAwIE1vYWNpciBG ZXJyZWlyYSA8c3BhbiBkaXI9Imx0ciI+Jmx0OzxhIGhyZWY9Im1haWx0bzptb2FjaXJmZXJyZWly YUBob3RtYWlsLmNvbSIgdGFyZ2V0PSJfYmxhbmsiPm1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY29t PC9hPiZndDs8L3NwYW4+Ojxicj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxl PSJtYXJnaW46MCAwIDAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxl ZnQ6MWV4Ij4KCgoKCjxkaXYgZGlyPSJsdHIiPgo8ZGl2IGlkPSJtXzE4NzQ4NTk4NTk4NDI3NjMx MDRkaXZ0YWdkZWZhdWx0d3JhcHBlciIgc3R5bGU9ImZvbnQtc2l6ZToxMnB0O2NvbG9yOiMwMDAw MDA7Zm9udC1mYW1pbHk6Q2FsaWJyaSxIZWx2ZXRpY2Esc2Fucy1zZXJpZiIgZGlyPSJsdHIiPgo8 cD5IaSBDb2xpbiw8L3A+CjxwPjxicj4KPC9wPgo8cD5UYWtlIGEgbG9vayBvbiBEZXZpbiYjMzk7 cyByZXNwb25zZS4gQWxzbywgcmVhZCB0aGUgZG9jIGhlIHNoYXJlZCB0aGF0IGdpdmVzIHNvbWUg aGludHMgb24gaG93IHRvIGRlcGxveSBHbHVzdGVyLjwvcD4KPHA+PGJyPgo8L3A+CjxwPkl0IGlz IG1vcmUgbGlrZSB0aGF0IGlmIHlvdSB3YW50IGhpZ2gtcGVyZm9ybWFuY2UgeW91IHNob3VsZCBo YXZlIHRoZSBicmlja3MgY3JlYXRlZCBhcyBSQUlEICg1IG9yIDYpwqBieSB0aGUgc2VydmVyJiMz OTtzIGRpc2sgY29udHJvbGxlciBhbmQgdGhlbSBhc3NlbWJsZSBhIEpCT0QgR2x1c3RlckZTLiBU aGUgYXR0YWNoZWQgZG9jdW1lbnQgaXMgR2x1c3RlciBzcGVjaWZpYyBhbmQgbm90IGZvciBvVmly dC4gQnV0IGF0IHRoaXMgcG9pbnQgSSB0aGluawogdGhhdCBoYXZpbmcgU1NEIHdpbGwgbm90IGJl IGEgcGx1cyBhcyB1c2luZyB0aGUgUkFJRCBjb250cm9sbGVyIEdsdXN0ZXIgd2lsbCBub3QgYmUg YXdhcmUgb2YgdGhlIFNTRC4gUmVnYXJkaW5nIHRoZSBPUywgbXkgaWRlYSBpcyB0byBoYXZlIGEg UkFJRCAxLCBtYWRlIG9mIDIgbG93IGNvc3QgSEREcywgdG8gaW5zdGFsbCBpdC48L3A+CjxwPjxi cj4KPC9wPgo8cD5TbyBmYXIsIGJhc2VkIG9uIHRoZSBpbmZvcm1hdGlvbiByZWNlaXZlZMKgSSBz aG91bGTCoGNyZWF0ZSBhIHNpbmdsZSBSQUlEIDUgb3IgNiBvbiBlYWNoIHNlcnZlciBhbmQgdGhl biB1c2UgdGhpcyBkaXNrIGFzIGEgYnJpY2sgdG8gY3JlYXRlIG15IEdsdXN0ZXIgY2x1c3Rlciwg bWFkZSBvZiAyIHJlcGxpY2FzICsgMSBhcmJpdGVyLiBXaGF0IGlzIG5ldyBmb3IgbWUgaXMgdGhl IGRldGFpbCB0aGF0IHRoZSBhcmJpdGVyIGRvZXMgbm90IG5lZWQKIGEgbG90IG9mIHNwYWNlIGFz IGl0IG9ubHkga2VlcHMgbWV0YSBkYXRhLjwvcD4KPHA+PGJyPgo8L3A+CjxwPlRoYW5rcyBmb3Ig eW91ciByZXNwb25zZSE8YnI+CjwvcD4KTW9hY2lyPGJyPgo8YnI+CjxkaXYgc3R5bGU9ImNvbG9y OnJnYig0OSw1NSw1NykiPgo8aHIgc3R5bGU9ImRpc3BsYXk6aW5saW5lLWJsb2NrO3dpZHRoOjk4 JSI+CjxkaXYgaWQ9Im1fMTg3NDg1OTg1OTg0Mjc2MzEwNGRpdlJwbHlGd2RNc2ciIGRpcj0ibHRy Ij48Zm9udCBzdHlsZT0iZm9udC1zaXplOjExcHQiIGNvbG9yPSIjMDAwMDAwIiBmYWNlPSJDYWxp YnJpLCBzYW5zLXNlcmlmIj48Yj5Gcm9tOjwvYj4gQ29saW4gQ29lICZsdDs8YSBocmVmPSJtYWls dG86Y29saW4uY29lQGdtYWlsLmNvbSIgdGFyZ2V0PSJfYmxhbmsiPmNvbGluLmNvZUBnbWFpbC5j b208L2E+Jmd0Ozxicj4KPGI+U2VudDo8L2I+IE1vbmRheSwgQXVndXN0IDcsIDIwMTcgMTI6NDEg UE08ZGl2PjxkaXYgY2xhc3M9Img1Ij48YnI+CjxiPlRvOjwvYj4gTW9hY2lyIEZlcnJlaXJhPGJy Pgo8Yj5DYzo8L2I+IDxhIGhyZWY9Im1haWx0bzp1c2Vyc0BvdmlydC5vcmciIHRhcmdldD0iX2Js YW5rIj51c2Vyc0BvdmlydC5vcmc8L2E+PGJyPgo8Yj5TdWJqZWN0OjwvYj4gUmU6IFtvdmlydC11 c2Vyc10gR29vZCBwcmFjdGljZXM8L2Rpdj48L2Rpdj48L2ZvbnQ+CjxkaXY+wqA8L2Rpdj4KPC9k aXY+PGRpdj48ZGl2IGNsYXNzPSJoNSI+CjxkaXY+CjxkaXYgZGlyPSJsdHIiPkhpCjxkaXY+PGJy Pgo8L2Rpdj4KPGRpdj5JIGp1c3QgdGhvdWdodCB0aGF0IHlvdSYjMzk7ZCBkbyBoYXJkd2FyZSBS QUlEIGlmIHlvdSBoYWQgdGhlIGNvbnRyb2xsZXIgb3IgSkJPRCBpZiB5b3UgZGlkbiYjMzk7dC7C oCBJbiBoaW5kc2lnaHQsIGEgc2VydmVyIHdpdGggNDBHYnBzIE5JQ3MgaXMgcHJldHR5IGxpa2Vs eSB0byBoYXZlIGEgaGFyZHdhcmUgUkFJRCBjb250cm9sbGVyLsKgIEkmIzM5O3ZlIG5ldmVyIGRv bmUgSkJPRCB3aXRoIGhhcmR3YXJlIFJBSUQuwqAgSSB0aGluayBoYXZpbmcgYSBzaW5nbGUKIGds dXN0ZXIgYnJpY2sgb24gaGFyZHdhcmUgSkJPRCB3b3VsZCBiZSByaXNraWVyIHRoYW4gbXVsdGlw bGUgYnJpY2tzLCBlYWNoIG9uIGEgc2luZ2xlIGRpc2ssIGJ1dCB0aGF0cyBub3QgYmFzZWQgb24g YW55dGhpbmcgb3RoZXIgdGhhbiBteSBwcmVqdWRpY2VzLjwvZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+ CjxkaXY+SSB0aG91Z2h0IGdsdXN0ZXIgdGllcmluZyB3YXMgZm9yIHRoZSBtb3N0IGZyZXF1ZW50 bHkgYWNjZXNzZWQgZmlsZXMsIGluIHdoaWNoIGNhc2UgYWxsIHRoZSBWTXMgZGlza3Mgd291bGQg ZW5kIHVwIGluIHRoZSBob3QgdGllci7CoCBIb3dldmVyLCBJIGhhdmUgYmVlbiB3cm9uZyBiZWZv cmUuLi48L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PkkganVzdCB3YW50ZWQgdG8ga25vdyB3 aGVyZSB0aGUgT1Mgd2FzIGdvaW5nIGFzIEkgZGlkbiYjMzk7dCBzZWUgaXQgbWVudGlvbmVkIGlu IHRoZSBPUC7CoCBOb3JtYWxseSwgSSYjMzk7ZCBoYXZlIHRoZSBPUyBvbiBhIFJBSUQxIGJ1dCBp biB5b3VyIGNhc2UgdGhhdHMgYSBsb3Qgb2Ygd2FzdGVkIGRpc2suPC9kaXY+CjxkaXY+PGJyPgo8 L2Rpdj4KPGRpdj5Ib25lc3RseSwgSSB0aGluayBZYW5pdiYjMzk7cyBhbnN3ZXIgd2FzIGZhciBi ZXR0ZXIgdGhhbiBteSBvd24gYW5kIG1hZGUgdGhlIGltcG9ydGFudCBwb2ludCBhYm91dCBoYXZp bmcgYW4gYXJiaXRlci7CoDwvZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+VGhhbmtzPC9kaXY+ CjwvZGl2Pgo8ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+PGJyPgo8ZGl2IGNsYXNzPSJnbWFpbF9x dW90ZSI+T24gTW9uLCBBdWcgNywgMjAxNyBhdCA1OjU2IFBNLCBNb2FjaXIgRmVycmVpcmEgPHNw YW4gZGlyPSJsdHIiPgombHQ7PGEgaHJlZj0ibWFpbHRvOm1vYWNpcmZlcnJlaXJhQGhvdG1haWwu Y29tIiB0YXJnZXQ9Il9ibGFuayI+bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb208L2E+Jmd0Ozwv c3Bhbj4gd3JvdGU6PGJyPgo8YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJt YXJnaW46MCAwIDAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6 MWV4Ij4KPGRpdiBkaXI9Imx0ciI+CjxkaXYgaWQ9Im1fMTg3NDg1OTg1OTg0Mjc2MzEwNG1fNDgz MTg4Njg4MzMwNTY3MjcxOGRpdnRhZ2RlZmF1bHR3cmFwcGVyIiBkaXI9Imx0ciIgc3R5bGU9ImZv bnQtc2l6ZToxMnB0O2NvbG9yOiMwMDAwMDA7Zm9udC1mYW1pbHk6Q2FsaWJyaSxIZWx2ZXRpY2Es c2Fucy1zZXJpZiI+CjxwPkhpIENvbGluLDwvcD4KPHA+PGJyPgo8L3A+CjxwPjxzcGFuPkkgYW0g aW4gUG9ydHVnYWw8L3NwYW4+LCBzbyBzb3JyeSBmb3IgdGhpcyBsYXRlIHJlc3BvbnNlLiBJdCBp cyBxdWl0ZSBjb25mdXNpbmcgZm9yIG1lLCBwbGVhc2UgY29uc2lkZXI6PC9wPgo8cD48L3A+Cjxk aXY+PGI+PGJyPgo8L2I+MTxiPiAtIDwvYj5XaGF0IGlmIHRoZSBSQUlEIGlzIGRvbmUgYnkgdGhl IHNlcnZlciYjMzk7cyBkaXNrIGNvbnRyb2xsZXIsIG5vdCBieSBzb2Z0d2FyZT88L2Rpdj4KPGJy Pgo8cD48L3A+CjxwPjIgLTxiPiA8L2I+Rm9yIEpCT0QgSSBhbSBqdXN0IHVzaW5nIGdkZXBsb3kg dG8gZGVwbG95IGl0LiBIb3dldmVyLCBJIGFtIG5vdCB1c2luZyB0aGUgb1ZpcnQgbm9kZSBHVUkg dG8gZG8gdGhpcy48L3A+CjxwPjxicj4KPC9wPgo8cD4zIC08Yj4gPC9iPkFzIHRoZSBWTSAucWNv dzLCoGZpbGVzIGFyZSBxdWl0ZSBiaWcsIHRpZXJpbmcgd291bGQgb25seSBoZWxwIGlmIG1hZGUg YnkgYW4gaW50ZWxsaWdlbnQgc3lzdGVtIHRoYXQgdXNlcyBTU0QgZm9yIGNodW5rcyBvZiBkYXRh IG5vdCBmb3IgdGhlIGVudGlyZSAucWNvdzIgZmlsZS4gQnV0IEkgZ3Vlc3MgdGhpcyBpcyBhIHBy b2JsZW0gZXZlcnlib2R5IGVsc2UgaGFzLiBTbywgRG8geW91IGtub3cgaG93IHRpZXJpbmcgd29y a3MKIGluIEdsdXN0ZXI/PGJyPgo8L3A+CjxwPjxicj4KPC9wPgo8cD40IC3CoEkgYW0gcHV0dGlu ZyB0aGUgT1Mgb24gdGhlIGZpcnN0IGRpc2suIEhvd2V2ZXIsIHdvdWxkIHlvdSBkbyBkaWZmZXJl bnRseT88YnI+CjwvcD4KPHA+PGJyPgo8L3A+Ck1vYWNpcjxicj4KPGJyPgo8ZGl2IHN0eWxlPSJj b2xvcjpyZ2IoNDksNTUsNTcpIj4KPGhyIHN0eWxlPSJkaXNwbGF5OmlubGluZS1ibG9jazt3aWR0 aDo5OCUiPgo8ZGl2IGlkPSJtXzE4NzQ4NTk4NTk4NDI3NjMxMDRtXzQ4MzE4ODY4ODMzMDU2NzI3 MThkaXZScGx5RndkTXNnIiBkaXI9Imx0ciI+PGZvbnQgc3R5bGU9ImZvbnQtc2l6ZToxMXB0IiBj b2xvcj0iIzAwMDAwMCIgZmFjZT0iQ2FsaWJyaSwgc2Fucy1zZXJpZiI+PGI+RnJvbTo8L2I+IENv bGluIENvZSAmbHQ7PGEgaHJlZj0ibWFpbHRvOmNvbGluLmNvZUBnbWFpbC5jb20iIHRhcmdldD0i X2JsYW5rIj5jb2xpbi5jb2VAZ21haWwuY29tPC9hPiZndDs8YnI+CjxiPlNlbnQ6PC9iPiBNb25k YXksIEF1Z3VzdCA3LCAyMDE3IDQ6NDggQU08YnI+CjxiPlRvOjwvYj4gTW9hY2lyIEZlcnJlaXJh PGJyPgo8Yj5DYzo8L2I+IDxhIGhyZWY9Im1haWx0bzp1c2Vyc0BvdmlydC5vcmciIHRhcmdldD0i X2JsYW5rIj51c2Vyc0BvdmlydC5vcmc8L2E+PGJyPgo8Yj5TdWJqZWN0OjwvYj4gUmU6IFtvdmly dC11c2Vyc10gR29vZCBwcmFjdGljZXM8L2ZvbnQ+CjxkaXY+wqA8L2Rpdj4KPC9kaXY+CjxkaXY+ CjxkaXYgZGlyPSJsdHIiPjEpIFJBSUQ1IG1heSBiZSBhIHBlcmZvcm1hbmNlIGhpdC0gPGJyPgo8 ZGl2Pjxicj4KPC9kaXY+CjxkaXY+PHNwYW4+MikgSSYjMzk7ZCBiZSBpbmNsaW5lZCB0byBkbyB0 aGlzIGFzIEpCT0QgYnkgY3JlYXRpbmcgYSBkaXN0cmlidXRlZCBkaXNwZXJzZSB2b2x1bWUgb24g ZWFjaCBzZXJ2ZXIuwqAgU29tZXRoaW5nIGxpa2UKPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PmVjaG8g Z2x1c3RlciB2b2x1bWUgY3JlYXRlIGRpc3BlcnNldm9sIGRpc3BlcnNlLWRhdGEgNSByZWR1bmRh bmN5IDIgXDwvZGl2Pgo8ZGl2PiQoZm9yIFNFUlZFUiBpbiBhIGIgYzsgZG8gZm9yIEJSSUNLIGlu ICQoc2VxIDEgNSk7IGRvIGVjaG8gLWUgJnF1b3Q7c2VydmVyJHtTRVJWRVJ9Oi9icmljay9icmlj ay08d2JyPiR7U0VSVkVSfSR7QlJJQ0t9L2JyaWNrIFxjJnF1b3Q7OyBkb25lOyBkb25lKTwvZGl2 Pgo8ZGl2Pjxicj4KPC9kaXY+Cjwvc3Bhbj4KPGRpdj4zKSBJIHRoaW5rIHRoZSBhYm92ZS4gPGI+ PC9iPjwvZGl2Pgo8c3Bhbj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PjQpIEdsdXN0ZXIgZG9lcyBz dXBwb3J0IHRpZXJpbmcsIGJ1dCBJSVJDIHlvdSYjMzk7ZCBuZWVkIHRoZSBzYW1lIG51bWJlciBv ZiBTU0QgYXMgc3BpbmRsZSBkcml2ZXMuwqAgVGhlcmUgbWF5IGJlIGFub3RoZXIgd2F5IHRvIHVz ZSB0aGUgU1NEIGFzIGEgZmFzdCBjYWNoZS7CoDwvZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+ V2hlcmUgYXJlIHlvdSBwdXR0aW5nIHRoZSBPUz88L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2 PkhvcGUgSSB1bmRlcnN0b29kIHRoZSBxdWVzdGlvbi4uLjwvZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+ CjxkaXY+VGhhbmtzPC9kaXY+Cjwvc3Bhbj48L2Rpdj4KPC9kaXY+CjxzcGFuPgo8ZGl2IGNsYXNz PSJnbWFpbF9leHRyYSI+PGJyPgo8ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+T24gU3VuLCBBdWcg NiwgMjAxNyBhdCAxMDo0OSBQTSwgTW9hY2lyIEZlcnJlaXJhIDxzcGFuIGRpcj0ibHRyIj4KJmx0 OzxhIGhyZWY9Im1haWx0bzptb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbSIgdGFyZ2V0PSJfYmxh bmsiPm1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY29tPC9hPiZndDs8L3NwYW4+IHdyb3RlOjxicj4K PGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAwIC44ZXg7 Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+CjxkaXYgZGlyPSJs dHIiPgo8ZGl2IGlkPSJtXzE4NzQ4NTk4NTk4NDI3NjMxMDRtXzQ4MzE4ODY4ODMzMDU2NzI3MTht XzI0NjA5ODU2OTE3NDY0OTgzMjJkaXZ0YWdkZWZhdWx0d3JhcHBlciIgZGlyPSJsdHIiIHN0eWxl PSJmb250LXNpemU6MTJwdDtjb2xvcjojMDAwMDAwO2ZvbnQtZmFtaWx5OkNhbGlicmksSGVsdmV0 aWNhLHNhbnMtc2VyaWYiPgo8cD48c3Bhbj5JIGFtIHdpbGxpbmcgdG8gYXNzZW1ibGUgYSBvVmly dCAmcXVvdDtwb2QmcXVvdDssIG1hZGUgb2YgMyBzZXJ2ZXJzLCBlYWNoIHdpdGggMiBDUFUgc29j a2V0cyBvZiAxMiBjb3JlcywgMjU2R0IgUkFNLCA3IEhERCAxMEssIDEgU1NELiBUaGUgaWRlYSBp cyB0byB1c2UgR2x1c3RlckZTIHRvIHByb3ZpZGUgSEEgZm9yIHRoZSBWTXMuIFRoZSAzIHNlcnZl cnMgaGF2ZSBhIGR1YWwgNDBHYiBOSUMgYW5kIGEgZHVhbCAxMEdiIE5JQy4gU28gbXkgaW50ZW50 aW9uCiBpcyB0byBjcmVhdGUgYSBsb29wIGxpa2UgYSBzZXJ2ZXIgdHJpYW5nbGUgdXNpbmcgdGhl IDQwR2IgTklDcyBmb3IgdmlydHVhbGl6YXRpb24gZmlsZXMgKFZNcyAucWNvdzIpIGFjY2VzcyBh bmQgdG8gbW92ZSBWTXMgYXJvdW5kIHRoZSBwb2QgKGVhc3QgL3dlc3QgdHJhZmZpYykgd2hpbGUg dXNpbmcgdGhlIDEwR2IgaW50ZXJmYWNlcyBmb3IgZ2l2aW5nIHNlcnZpY2VzIHRvIHRoZSBvdXRz aWRlIHdvcmxkIChub3J0aC9zb3V0aCB0cmFmZmljKS48L3NwYW4+PC9wPgo8cD48YnI+CjxzcGFu Pjwvc3Bhbj48L3A+CjxwPlRoaXMgc2FpZCwgbXkgZmlyc3QgcXVlc3Rpb24gaXM6IEhvdyBzaG91 bGQgSSBkZXBsb3kgR2x1c3RlckZTIGluIHN1Y2ggb1ZpcnQgc2NlbmFyaW8/IE15IHF1ZXN0aW9u cyBhcmU6PC9wPgo8cD48YnI+CjwvcD4KPHA+MSAtIFNob3VsZCBJIGNyZWF0ZSAzIFJBSUQgKGku ZS46IFJBSUQgNSksIG9uZSBvbiBlYWNoIG9WaXJ0IG5vZGUsIGFuZCB0aGVuIGNyZWF0ZSBhIEds dXN0ZXJGUyB1c2luZyB0aGVtPzwvcD4KPHA+MiAtIEluc3RlYWQsIHNob3VsZCBJIGNyZWF0ZSBh IEpCT0QgYXJyYXkgbWFkZSBvZiBhbGwgc2VydmVyJiMzOTtzIGRpc2tzPzwvcD4KPHA+MyAtIFdo YXQgaXMgdGhlIGJlc3QgR2x1c3RlciBjb25maWd1cmF0aW9uIHRvIHByb3ZpZGUgZm9yIEhBIHdo aWxlIG5vdCBjb25zdW1pbmcgdG9vIG11Y2ggZGlzayBzcGFjZT88YnI+CjwvcD4KPHA+NCAtIERv ZXMgYSBvVmlydCBoeXBlcnZpc29yIHBvZCBsaWtlIEkgYW0gcGxhbm5pbmcgdG8gYnVpbGQsIGFu ZCB0aGUgdmlydHVhbGl6YXRpb24gZW52aXJvbm1lbnQsIGJlbmVmaXRzIGZyb20gdGllcmluZyB3 aGVuIHVzaW5nIGEgU1NEIGRpc2s/IEFuZCB5ZXMsIHdpbGwgR2x1c3RlciBkbyBpdCBieSBkZWZh dWx0IG9yIEkgaGF2ZSB0byBjb25maWd1cmUgaXQgdG8gZG8gc28/PC9wPgo8cD48YnI+CjwvcD4K PHA+QXQgdGhlIGJvdHRvbSBsaW5lLCB3aGF0IGlzIHRoZSBnb29kIHByYWN0aWNlIGZvciB1c2lu ZyBHbHVzdGVyRlMgaW4gc21hbGwgcG9kcyBmb3IgZW50ZXJwcmlzZXM/PGJyPgo8L3A+CjxwPjxi cj4KPC9wPgo8cD5Zb3Ugb3Bpbmlvbi9mZWVkYmFjayB3aWxsIGJlIHJlYWxseSBhcHByZWNpYXRl ZCE8L3A+CjxzcGFuIGNsYXNzPSJtXzE4NzQ4NTk4NTk4NDI3NjMxMDRtXzQ4MzE4ODY4ODMzMDU2 NzI3MThIT0VuWmIiPjxmb250IGNvbG9yPSIjODg4ODg4Ij4KPHA+TW9hY2lyPGJyPgo8L3A+Cjwv Zm9udD48L3NwYW4+PC9kaXY+CjwvZGl2Pgo8YnI+Cl9fX19fX19fX19fX19fX19fX19fX19fX19f X19fXzx3YnI+X19fX19fX19fX19fX19fX188YnI+ClVzZXJzIG1haWxpbmcgbGlzdDxicj4KPGEg aHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgdGFyZ2V0PSJfYmxhbmsiPlVzZXJzQG92aXJ0 Lm9yZzwvYT48YnI+CjxhIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0 aW5mby91c2VycyIgcmVsPSJub3JlZmVycmVyIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL2xpc3Rz Lm92aXJ0Lm9yZy9tYWlsbWFuPHdicj4vbGlzdGluZm8vdXNlcnM8L2E+PGJyPgo8YnI+CjwvYmxv Y2txdW90ZT4KPC9kaXY+Cjxicj4KPC9kaXY+Cjwvc3Bhbj48L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8 L2Rpdj4KPC9ibG9ja3F1b3RlPgo8L2Rpdj4KPGJyPgo8L2Rpdj4KPC9kaXY+CjwvZGl2PjwvZGl2 PjwvZGl2Pgo8L2Rpdj4KPC9kaXY+Cgo8YnI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19f PHdicj5fX19fX19fX19fX19fX19fXzxicj4KVXNlcnMgbWFpbGluZyBsaXN0PGJyPgo8YSBocmVm PSJtYWlsdG86VXNlcnNAb3ZpcnQub3JnIj5Vc2Vyc0BvdmlydC5vcmc8L2E+PGJyPgo8YSBocmVm PSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiIHJlbD0ibm9y ZWZlcnJlciIgdGFyZ2V0PSJfYmxhbmsiPmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvPHdicj5tYWls bWFuL2xpc3RpbmZvL3VzZXJzPC9hPjxicj4KPGJyPjwvYmxvY2txdW90ZT48L2Rpdj48YnI+PC9k aXY+Cg== --===============5550542770423793846==-- From moacirferreira at hotmail.com Tue Aug 8 05:06:14 2017 Content-Type: multipart/mixed; boundary="===============6238324484136124314==" MIME-Version: 1.0 From: Moacir Ferreira To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Mon, 07 Aug 2017 13:29:17 +0000 Message-ID: In-Reply-To: CANCGKEp4XGs0U+Qs78eEmqCNtvpLY-Azjb5DcGhZ9yiKTBEEfw@mail.gmail.com --===============6238324484136124314== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --_000_VI1P190MB0285AF47BE9B7FEE3EEB5067C8B50VI1P190MB0285EURP_ Content-Type: text/plain; charset=3D"Windows-1252" Content-Transfer-Encoding: quoted-printable Hi Devin, Please consider that for the OS I have a RAID 1. Now, lets say I use RAID 5= =3D to assemble a single disk on each server. In this case, the SSD will not m= =3D ake any difference, right? I guess that to be possible to use it, the SSD s= =3D hould not be part of the RAID 5. In this case I could create a logical volu= =3D me made of the RAIDed brick and then extend it using the SSD. I.e.: Using g= =3D deploy: [disktype] jbod .... [pv1] action=3D3Dcreate devices=3D3Dsdb, sdc wipefs=3D3Dyes ignore_vg_erros=3D3Dno [vg1] action=3D3Dcreate vgname=3D3Dgluster_vg_jbod pvname=3D3Dsdb ignore_vg_erros=3D3Dno [vg2] action=3D3Dextend vgname=3D3Dgluster_vg_jbod pvname=3D3Dsdc ignore_vg_erros=3D3Dno But will Gluster be able to auto-detect and use this SSD brick for tiering?= =3D Do I have to do some other configurations? Also, as the VM files (.qcow2) = =3D are quite big will I benefit from tiering? This is wrong and my approach sh= =3D ould be other? Thanks, Moacir ________________________________ From: Devin Acosta Sent: Monday, August 7, 2017 7:46 AM To: Moacir Ferreira; users(a)ovirt.org Subject: Re: [ovirt-users] Good practices Moacir, I have recently installed multiple Red Hat Virtualization hosts for several= =3D different companies, and have dealt with the Red Hat Support Team in depth= =3D about optimal configuration in regards to setting up GlusterFS most effici= =3D ently and I wanted to share with you what I learned. In general Red Hat Virtualization team frowns upon using each DISK of the s= =3D ystem as just a JBOD, sure there is some protection by having the data repl= =3D icated, however, the recommendation is to use RAID 6 (preferred) or RAID-5,= =3D or at least RAID-1 at the very least. Here is the direct quote from Red Hat when I asked about RAID and Bricks: "A typical Gluster configuration would use RAID underneath the bricks. RAID= =3D 6 is most typical as it gives you 2 disk failure protection, but RAID 5 co= =3D uld be used too. Once you have the RAIDed bricks, you'd then apply the desi= =3D red replication on top of that. The most popular way of doing this would be= =3D distributed replicated with 2x replication. In general you'll get better p= =3D erformance with larger bricks. 12 drives is often a sweet spot. Another opt= =3D ion would be to create a separate tier using all SSD=3D92s.=3D94 In order to SSD tiering from my understanding you would need 1 x NVMe drive= =3D in each server, or 4 x SSD hot tier (it needs to be distributed, replicate= =3D d for the hot tier if not using NVME). So with you only having 1 SSD drive = =3D in each server, I=3D92d suggest maybe looking into the NVME option. Since your using only 3-servers, what I=3D92d probably suggest is to do (2 = Re=3D plicas + Arbiter Node), this setup actually doesn=3D92t require the 3rd ser= ve=3D r to have big drives at all as it only stores meta-data about the files and= =3D not actually a full copy. Please see the attached document that was given to me by Red Hat to get mor= =3D e information on this. Hope this information helps you. -- Devin Acosta, RHCA, RHVCA Red Hat Certified Architect On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (moacirferreira(a)hotmail.= co=3D m) wrote: I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU = =3D sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use Gluste= =3D rFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dua= =3D l 10Gb NIC. So my intention is to create a loop like a server triangle usin= =3D g the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VM= =3D s around the pod (east /west traffic) while using the 10Gb interfaces for g= =3D iving services to the outside world (north/south traffic). This said, my first question is: How should I deploy GlusterFS in such oVir= =3D t scenario? My questions are: 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then= =3D create a GlusterFS using them? 2 - Instead, should I create a JBOD array made of all server's disks? 3 - What is the best Gluster configuration to provide for HA while not cons= =3D uming too much disk space? 4 - Does a oVirt hypervisor pod like I am planning to build, and the virtua= =3D lization environment, benefits from tiering when using a SSD disk? And yes,= =3D will Gluster do it by default or I have to configure it to do so? At the bottom line, what is the good practice for using GlusterFS in small = =3D pods for enterprises? You opinion/feedback will be really appreciated! Moacir _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users --_000_VI1P190MB0285AF47BE9B7FEE3EEB5067C8B50VI1P190MB0285EURP_ Content-Type: text/html; charset=3D"Windows-1252" Content-Transfer-Encoding: quoted-printable

Hi Devin,

Please consider that for the OS I have a RAID 1. Now, lets say I use RAI= =3D D 5 to assemble a single disk on each server. In this case, the SSD will no= =3D t make any difference, right? I guess that to be possible to use it, the SS= =3D D should not be part of the RAID 5. In this case I could create a logical volume made of the RAIDed brick a= =3D nd then extend it using the SSD. I.e.: Using gdeploy:

[disktype]

jbod

....

[pv1]

action=3D3Dcreate

devices=3D3Dsdb, sdc

wipefs=3D3Dyes

ignore_vg_erros=3D3Dno

[vg1]

action=3D3Dcreate

vgname=3D3Dgluster_vg_jbod

pvname=3D3Dsdb

ignore_vg_erros=3D3Dno

[vg2]

action=3D3Dextend

vgname=3D3Dgluster_vg_jbod

pvname=3D3Dsdc

ignore_vg_erros=3D3Dno

But will Gluster be able to auto-detect and use this SSD brick for tieri= =3D ng? Do I have to do some other configurations? Also, as the VM files (.qcow= =3D 2) are quite big will I benefit from tiering? This is wrong and my approach= =3D should be other?

Thanks,

Moacir

From: Devin Acosta &l= t;de=3D vin(a)pabstatencio.com>
Sent: Monday, August 7, 2017 7:46 AM
To: Moacir Ferreira; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices

Moacir,

I have recently installed multiple Red Hat Virtualizat= io=3D n hosts for several different companies, and have dealt with the Red Hat Su= =3D pport Team in depth about optimal configuration in regards to setting up GlusterFS most efficiently and I wanted to share = =3D with you what I learned.

In general Red Hat Virtualization team frowns upon usi= ng=3D each DISK of the system as just a JBOD, sure there is some protection by h= =3D aving the data replicated, however, the recommendation is to use RAID 6 (preferred) or RAID-5, or at least RAID-1 = =3D at the very least.

Here is the direct quote from Red Hat when I asked about RAID and Bri= =3D cks:

"A typical Gluster configuration would use RAID underneath the bri= =3D cks. RAID 6 is most typical as it gives you 2 disk failure protection, but = =3D RAID 5 could be used too. Once you have the RAIDed bricks, you'd then apply the desired replication on top of that. Th= =3D e most popular way of doing this would be distributed replicated with 2x re= =3D plication. In general you'll get better performance with larger bricks= =3D . 12 drives is often a sweet spot. Another option would be to create a separate tier using all SSD=3D92s.=3D94 <= /i><=3D /font>

In order to SSD tiering from my understanding you would need 1 x N= =3D VMe drive in each server, or 4 x SSD hot tier (it needs to be distributed, = =3D replicated for the hot tier if not using NVME). So with you only having 1 SSD drive in each server, I=3D92d suggest m= ay=3D be looking into the NVME option.

Since your using only 3-servers, what I=3D92d probably suggest is to do= (=3D 2 Replicas + Arbiter Node), this setup actually doesn=3D92t require the= 3=3D rd server to have big drives at all as it only stores meta-data about the files and not actually a full copy.

Please see the attached document that was given to me by Red Hat to get= =3D more information on this. Hope this information helps you.

--

Devin Acosta, RHCA, RHVCA

Red Hat Certified Architect

On August 6, 2017 at 7:29:29 PM, Moacir Ferreira = (<=3D a href=3D3D"mailto:moacirferreira(a)hotmail.com">moacirferreira(a)hotmail.c= om=3D ) wrote:

I am willing to assemble a oVirt "pod", made of 3 server= =3D s, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The id= =3D ea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual= =3D 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop like a server triangle using the 40Gb NICs for virtual= =3D ization files (VMs .qcow2) access and to move VMs around the pod (east /wes= =3D t traffic) while using the 10Gb interfaces for giving services to the outsi= =3D de world (north/south traffic).

This said, my first question is: How should I deploy GlusterFS in such o= =3D Virt scenario? My questions are:

1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and t= =3D hen create a GlusterFS using them?

2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not c= =3D onsuming too much disk space?

4 - Does a oVirt hypervisor pod like I am planning to build, and the vir= =3D tualization environment, benefits from tiering when using a SSD disk? And y= =3D es, will Gluster do it by default or I have to configure it to do so?

At the bottom line, what is the good practice for using GlusterFS in sma= =3D ll pods for enterprises?

You opinion/feedback will be really appreciated!

Moacir

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ov= ir=3D t.org/mailman/listinfo/users

--_000_VI1P190MB0285AF47BE9B7FEE3EEB5067C8B50VI1P190MB0285EURP_-- --===============6238324484136124314== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" LS1fMDAwX1ZJMVAxOTBNQjAyODVBRjQ3QkU5QjdGRUUzRUVCNTA2N0M4QjUwVkkxUDE5ME1CMDI4 NUVVUlBfCkNvbnRlbnQtVHlwZTogdGV4dC9wbGFpbjsgY2hhcnNldD0iV2luZG93cy0xMjUyIgpD b250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiBxdW90ZWQtcHJpbnRhYmxlCgpIaSBEZXZpbiwKCgpQ bGVhc2UgY29uc2lkZXIgdGhhdCBmb3IgdGhlIE9TIEkgaGF2ZSBhIFJBSUQgMS4gTm93LCBsZXRz IHNheSBJIHVzZSBSQUlEIDU9CiB0byBhc3NlbWJsZSBhIHNpbmdsZSBkaXNrIG9uIGVhY2ggc2Vy dmVyLiBJbiB0aGlzIGNhc2UsIHRoZSBTU0Qgd2lsbCBub3QgbT0KYWtlIGFueSBkaWZmZXJlbmNl LCByaWdodD8gSSBndWVzcyB0aGF0IHRvIGJlIHBvc3NpYmxlIHRvIHVzZSBpdCwgdGhlIFNTRCBz PQpob3VsZCBub3QgYmUgcGFydCBvZiB0aGUgUkFJRCA1LiBJbiB0aGlzIGNhc2UgSSBjb3VsZCBj cmVhdGUgYSBsb2dpY2FsIHZvbHU9Cm1lIG1hZGUgb2YgdGhlIFJBSURlZCBicmljayBhbmQgdGhl biBleHRlbmQgaXQgdXNpbmcgdGhlIFNTRC4gSS5lLjogVXNpbmcgZz0KZGVwbG95OgoKCltkaXNr dHlwZV0KCmpib2QKCi4uLi4KCltwdjFdCgphY3Rpb249M0RjcmVhdGUKCmRldmljZXM9M0RzZGIs IHNkYwoKd2lwZWZzPTNEeWVzCgppZ25vcmVfdmdfZXJyb3M9M0RubwoKClt2ZzFdCgphY3Rpb249 M0RjcmVhdGUKCnZnbmFtZT0zRGdsdXN0ZXJfdmdfamJvZAoKcHZuYW1lPTNEc2RiCgppZ25vcmVf dmdfZXJyb3M9M0RubwoKClt2ZzJdCgphY3Rpb249M0RleHRlbmQKCnZnbmFtZT0zRGdsdXN0ZXJf dmdfamJvZAoKcHZuYW1lPTNEc2RjCgppZ25vcmVfdmdfZXJyb3M9M0RubwoKCkJ1dCB3aWxsIEds dXN0ZXIgYmUgYWJsZSB0byBhdXRvLWRldGVjdCBhbmQgdXNlIHRoaXMgU1NEIGJyaWNrIGZvciB0 aWVyaW5nPz0KIERvIEkgaGF2ZSB0byBkbyBzb21lIG90aGVyIGNvbmZpZ3VyYXRpb25zPyBBbHNv LCBhcyB0aGUgVk0gZmlsZXMgKC5xY293MikgPQphcmUgcXVpdGUgYmlnIHdpbGwgSSBiZW5lZml0 IGZyb20gdGllcmluZz8gVGhpcyBpcyB3cm9uZyBhbmQgbXkgYXBwcm9hY2ggc2g9Cm91bGQgYmUg b3RoZXI/CgoKVGhhbmtzLAoKTW9hY2lyCgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X18KRnJvbTogRGV2aW4gQWNvc3RhIDxkZXZpbkBwYWJzdGF0ZW5jaW8uY29tPgpTZW50OiBNb25k YXksIEF1Z3VzdCA3LCAyMDE3IDc6NDYgQU0KVG86IE1vYWNpciBGZXJyZWlyYTsgdXNlcnNAb3Zp cnQub3JnClN1YmplY3Q6IFJlOiBbb3ZpcnQtdXNlcnNdIEdvb2QgcHJhY3RpY2VzCgoKTW9hY2ly LAoKSSBoYXZlIHJlY2VudGx5IGluc3RhbGxlZCBtdWx0aXBsZSBSZWQgSGF0IFZpcnR1YWxpemF0 aW9uIGhvc3RzIGZvciBzZXZlcmFsPQogZGlmZmVyZW50IGNvbXBhbmllcywgYW5kIGhhdmUgZGVh bHQgd2l0aCB0aGUgUmVkIEhhdCBTdXBwb3J0IFRlYW0gaW4gZGVwdGg9CiBhYm91dCBvcHRpbWFs IGNvbmZpZ3VyYXRpb24gaW4gcmVnYXJkcyB0byBzZXR0aW5nIHVwIEdsdXN0ZXJGUyBtb3N0IGVm ZmljaT0KZW50bHkgYW5kIEkgd2FudGVkIHRvIHNoYXJlIHdpdGggeW91IHdoYXQgSSBsZWFybmVk LgoKSW4gZ2VuZXJhbCBSZWQgSGF0IFZpcnR1YWxpemF0aW9uIHRlYW0gZnJvd25zIHVwb24gdXNp bmcgZWFjaCBESVNLIG9mIHRoZSBzPQp5c3RlbSBhcyBqdXN0IGEgSkJPRCwgc3VyZSB0aGVyZSBp cyBzb21lIHByb3RlY3Rpb24gYnkgaGF2aW5nIHRoZSBkYXRhIHJlcGw9CmljYXRlZCwgaG93ZXZl ciwgdGhlIHJlY29tbWVuZGF0aW9uIGlzIHRvIHVzZSBSQUlEIDYgKHByZWZlcnJlZCkgb3IgUkFJ RC01LD0KIG9yIGF0IGxlYXN0IFJBSUQtMSBhdCB0aGUgdmVyeSBsZWFzdC4KCkhlcmUgaXMgdGhl IGRpcmVjdCBxdW90ZSBmcm9tIFJlZCBIYXQgd2hlbiBJIGFza2VkIGFib3V0IFJBSUQgYW5kIEJy aWNrczoKCiJBIHR5cGljYWwgR2x1c3RlciBjb25maWd1cmF0aW9uIHdvdWxkIHVzZSBSQUlEIHVu ZGVybmVhdGggdGhlIGJyaWNrcy4gUkFJRD0KIDYgaXMgbW9zdCB0eXBpY2FsIGFzIGl0IGdpdmVz IHlvdSAyIGRpc2sgZmFpbHVyZSBwcm90ZWN0aW9uLCBidXQgUkFJRCA1IGNvPQp1bGQgYmUgdXNl ZCB0b28uIE9uY2UgeW91IGhhdmUgdGhlIFJBSURlZCBicmlja3MsIHlvdSdkIHRoZW4gYXBwbHkg dGhlIGRlc2k9CnJlZCByZXBsaWNhdGlvbiBvbiB0b3Agb2YgdGhhdC4gVGhlIG1vc3QgcG9wdWxh ciB3YXkgb2YgZG9pbmcgdGhpcyB3b3VsZCBiZT0KIGRpc3RyaWJ1dGVkIHJlcGxpY2F0ZWQgd2l0 aCAyeCByZXBsaWNhdGlvbi4gSW4gZ2VuZXJhbCB5b3UnbGwgZ2V0IGJldHRlciBwPQplcmZvcm1h bmNlIHdpdGggbGFyZ2VyIGJyaWNrcy4gMTIgZHJpdmVzIGlzIG9mdGVuIGEgc3dlZXQgc3BvdC4g QW5vdGhlciBvcHQ9CmlvbiB3b3VsZCBiZSB0byBjcmVhdGUgYSBzZXBhcmF0ZSB0aWVyIHVzaW5n IGFsbCBTU0Q9OTJzLj05NAoKSW4gb3JkZXIgdG8gU1NEIHRpZXJpbmcgZnJvbSBteSB1bmRlcnN0 YW5kaW5nIHlvdSB3b3VsZCBuZWVkIDEgeCBOVk1lIGRyaXZlPQogaW4gZWFjaCBzZXJ2ZXIsIG9y IDQgeCBTU0QgaG90IHRpZXIgKGl0IG5lZWRzIHRvIGJlIGRpc3RyaWJ1dGVkLCByZXBsaWNhdGU9 CmQgZm9yIHRoZSBob3QgdGllciBpZiBub3QgdXNpbmcgTlZNRSkuIFNvIHdpdGggeW91IG9ubHkg aGF2aW5nIDEgU1NEIGRyaXZlID0KaW4gZWFjaCBzZXJ2ZXIsIEk9OTJkIHN1Z2dlc3QgbWF5YmUg bG9va2luZyBpbnRvIHRoZSBOVk1FIG9wdGlvbi4KClNpbmNlIHlvdXIgdXNpbmcgb25seSAzLXNl cnZlcnMsIHdoYXQgST05MmQgcHJvYmFibHkgc3VnZ2VzdCBpcyB0byBkbyAoMiBSZT0KcGxpY2Fz ICsgQXJiaXRlciBOb2RlKSwgdGhpcyBzZXR1cCBhY3R1YWxseSBkb2Vzbj05MnQgcmVxdWlyZSB0 aGUgM3JkIHNlcnZlPQpyIHRvIGhhdmUgYmlnIGRyaXZlcyBhdCBhbGwgYXMgaXQgb25seSBzdG9y ZXMgbWV0YS1kYXRhIGFib3V0IHRoZSBmaWxlcyBhbmQ9CiBub3QgYWN0dWFsbHkgYSBmdWxsIGNv cHkuCgpQbGVhc2Ugc2VlIHRoZSBhdHRhY2hlZCBkb2N1bWVudCB0aGF0IHdhcyBnaXZlbiB0byBt ZSBieSBSZWQgSGF0IHRvIGdldCBtb3I9CmUgaW5mb3JtYXRpb24gb24gdGhpcy4gSG9wZSB0aGlz IGluZm9ybWF0aW9uIGhlbHBzIHlvdS4KCgotLQoKRGV2aW4gQWNvc3RhLCBSSENBLCBSSFZDQQpS ZWQgSGF0IENlcnRpZmllZCBBcmNoaXRlY3QKCgpPbiBBdWd1c3QgNiwgMjAxNyBhdCA3OjI5OjI5 IFBNLCBNb2FjaXIgRmVycmVpcmEgKG1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY289Cm08bWFpbHRv Om1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY29tPikgd3JvdGU6CgpJIGFtIHdpbGxpbmcgdG8gYXNz ZW1ibGUgYSBvVmlydCAicG9kIiwgbWFkZSBvZiAzIHNlcnZlcnMsIGVhY2ggd2l0aCAyIENQVSA9 CnNvY2tldHMgb2YgMTIgY29yZXMsIDI1NkdCIFJBTSwgNyBIREQgMTBLLCAxIFNTRC4gVGhlIGlk ZWEgaXMgdG8gdXNlIEdsdXN0ZT0KckZTIHRvIHByb3ZpZGUgSEEgZm9yIHRoZSBWTXMuIFRoZSAz IHNlcnZlcnMgaGF2ZSBhIGR1YWwgNDBHYiBOSUMgYW5kIGEgZHVhPQpsIDEwR2IgTklDLiBTbyBt eSBpbnRlbnRpb24gaXMgdG8gY3JlYXRlIGEgbG9vcCBsaWtlIGEgc2VydmVyIHRyaWFuZ2xlIHVz aW49CmcgdGhlIDQwR2IgTklDcyBmb3IgdmlydHVhbGl6YXRpb24gZmlsZXMgKFZNcyAucWNvdzIp IGFjY2VzcyBhbmQgdG8gbW92ZSBWTT0KcyBhcm91bmQgdGhlIHBvZCAoZWFzdCAvd2VzdCB0cmFm ZmljKSB3aGlsZSB1c2luZyB0aGUgMTBHYiBpbnRlcmZhY2VzIGZvciBnPQppdmluZyBzZXJ2aWNl cyB0byB0aGUgb3V0c2lkZSB3b3JsZCAobm9ydGgvc291dGggdHJhZmZpYykuCgoKVGhpcyBzYWlk LCBteSBmaXJzdCBxdWVzdGlvbiBpczogSG93IHNob3VsZCBJIGRlcGxveSBHbHVzdGVyRlMgaW4g c3VjaCBvVmlyPQp0IHNjZW5hcmlvPyBNeSBxdWVzdGlvbnMgYXJlOgoKCjEgLSBTaG91bGQgSSBj cmVhdGUgMyBSQUlEIChpLmUuOiBSQUlEIDUpLCBvbmUgb24gZWFjaCBvVmlydCBub2RlLCBhbmQg dGhlbj0KIGNyZWF0ZSBhIEdsdXN0ZXJGUyB1c2luZyB0aGVtPwoKMiAtIEluc3RlYWQsIHNob3Vs ZCBJIGNyZWF0ZSBhIEpCT0QgYXJyYXkgbWFkZSBvZiBhbGwgc2VydmVyJ3MgZGlza3M/CgozIC0g V2hhdCBpcyB0aGUgYmVzdCBHbHVzdGVyIGNvbmZpZ3VyYXRpb24gdG8gcHJvdmlkZSBmb3IgSEEg d2hpbGUgbm90IGNvbnM9CnVtaW5nIHRvbyBtdWNoIGRpc2sgc3BhY2U/Cgo0IC0gRG9lcyBhIG9W aXJ0IGh5cGVydmlzb3IgcG9kIGxpa2UgSSBhbSBwbGFubmluZyB0byBidWlsZCwgYW5kIHRoZSB2 aXJ0dWE9CmxpemF0aW9uIGVudmlyb25tZW50LCBiZW5lZml0cyBmcm9tIHRpZXJpbmcgd2hlbiB1 c2luZyBhIFNTRCBkaXNrPyBBbmQgeWVzLD0KIHdpbGwgR2x1c3RlciBkbyBpdCBieSBkZWZhdWx0 IG9yIEkgaGF2ZSB0byBjb25maWd1cmUgaXQgdG8gZG8gc28/CgoKQXQgdGhlIGJvdHRvbSBsaW5l LCB3aGF0IGlzIHRoZSBnb29kIHByYWN0aWNlIGZvciB1c2luZyBHbHVzdGVyRlMgaW4gc21hbGwg PQpwb2RzIGZvciBlbnRlcnByaXNlcz8KCgpZb3Ugb3Bpbmlvbi9mZWVkYmFjayB3aWxsIGJlIHJl YWxseSBhcHByZWNpYXRlZCEKCk1vYWNpcgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX18KVXNlcnMgbWFpbGluZyBsaXN0ClVzZXJzQG92aXJ0Lm9yZzxtYWls dG86VXNlcnNAb3ZpcnQub3JnPgpodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGlu Zm8vdXNlcnMKCi0tXzAwMF9WSTFQMTkwTUIwMjg1QUY0N0JFOUI3RkVFM0VFQjUwNjdDOEI1MFZJ MVAxOTBNQjAyODVFVVJQXwpDb250ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD0iV2luZG93 cy0xMjUyIgpDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiBxdW90ZWQtcHJpbnRhYmxlCgo8aHRt bD4KPGhlYWQ+CjxtZXRhIGh0dHAtZXF1aXY9M0QiQ29udGVudC1UeXBlIiBjb250ZW50PTNEInRl eHQvaHRtbDsgY2hhcnNldD0zRFdpbmRvd3MtMT0KMjUyIj4KPHN0eWxlIHR5cGU9M0QidGV4dC9j c3MiIHN0eWxlPTNEImRpc3BsYXk6bm9uZTsiPjwhLS0gUCB7bWFyZ2luLXRvcDowO21hcmdpPQpu LWJvdHRvbTowO30gLS0+PC9zdHlsZT4KPC9oZWFkPgo8Ym9keSBkaXI9M0QibHRyIj4KPGRpdiBp ZD0zRCJkaXZ0YWdkZWZhdWx0d3JhcHBlciIgc3R5bGU9M0QiZm9udC1zaXplOjEycHQ7Y29sb3I6 IzAwMDAwMDtmb250PQotZmFtaWx5OkNhbGlicmksSGVsdmV0aWNhLHNhbnMtc2VyaWY7IiBkaXI9 M0QibHRyIj4KPHA+SGkgRGV2aW4sPC9wPgo8cD48YnI+CjwvcD4KPHA+UGxlYXNlIGNvbnNpZGVy IHRoYXQgZm9yIHRoZSBPUyBJIGhhdmUgYSBSQUlEIDEuIE5vdywgbGV0cyBzYXkgSSB1c2UgUkFJ PQpEIDUgdG8gYXNzZW1ibGUgYSBzaW5nbGUgZGlzayBvbiBlYWNoIHNlcnZlci4gSW4gdGhpcyBj YXNlLCB0aGUgU1NEIHdpbGwgbm89CnQgbWFrZSBhbnkgZGlmZmVyZW5jZSwgcmlnaHQ/IEkgZ3Vl c3MgdGhhdCB0byBiZSBwb3NzaWJsZSB0byB1c2UgaXQsIHRoZSBTUz0KRCBzaG91bGQgbm90IGJl IHBhcnQgb2YgdGhlIFJBSUQKIDUuIEluIHRoaXMgY2FzZSBJIGNvdWxkIGNyZWF0ZSBhIGxvZ2lj YWwgdm9sdW1lIG1hZGUgb2YgdGhlIFJBSURlZCBicmljayBhPQpuZCB0aGVuIGV4dGVuZCBpdCB1 c2luZyB0aGUgU1NELiBJLmUuOiBVc2luZyBnZGVwbG95OjwvcD4KPHA+PGJyPgo8L3A+CjxwPltk aXNrdHlwZV08L3A+CjxwPmpib2Q8L3A+CjxwPi4uLi48L3A+CjxwPltwdjFdPC9wPgo8cD5hY3Rp b249M0RjcmVhdGU8L3A+CjxwPmRldmljZXM9M0RzZGIsIHNkYzwvcD4KPHA+d2lwZWZzPTNEeWVz PC9wPgo8cD48L3A+CjxwPmlnbm9yZV92Z19lcnJvcz0zRG5vPGJyPgo8L3A+CjxwPjxicj4KPC9w Pgo8cD48L3A+CjxwPlt2ZzFdPC9wPgo8cD48Yj5hY3Rpb249M0RjcmVhdGU8L2I+PC9wPgo8cD52 Z25hbWU9M0RnbHVzdGVyX3ZnX2pib2Q8L3A+CjxwPnB2bmFtZT0zRHNkYjwvcD4KPHA+aWdub3Jl X3ZnX2Vycm9zPTNEbm88YnI+CjwvcD4KPHA+PGJyPgo8L3A+CjxwPlt2ZzJdPC9wPgo8cD48Yj5h Y3Rpb249M0RleHRlbmQ8L2I+PC9wPgo8cD52Z25hbWU9M0RnbHVzdGVyX3ZnX2pib2Q8L3A+Cjxw PnB2bmFtZT0zRHNkYzwvcD4KPHA+aWdub3JlX3ZnX2Vycm9zPTNEbm88YnI+CjwvcD4KPGJyPgo8 cD5CdXQgd2lsbCBHbHVzdGVyIGJlIGFibGUgdG8gYXV0by1kZXRlY3QgYW5kIHVzZSB0aGlzIFNT RCBicmljayBmb3IgdGllcmk9Cm5nPyBEbyBJIGhhdmUgdG8gZG8gc29tZSBvdGhlciBjb25maWd1 cmF0aW9ucz8gQWxzbywgYXMgdGhlIFZNIGZpbGVzICgucWNvdz0KMikgYXJlIHF1aXRlIGJpZyB3 aWxsIEkgYmVuZWZpdCBmcm9tIHRpZXJpbmc/IFRoaXMgaXMgd3JvbmcgYW5kIG15IGFwcHJvYWNo PQogc2hvdWxkIGJlIG90aGVyPzwvcD4KPHA+PGJyPgo8L3A+CjxwPlRoYW5rcyw8L3A+CjxwPk1v YWNpcjxicj4KPC9wPgo8cD48YnI+CjwvcD4KPGJyPgo8ZGl2IHN0eWxlPTNEImNvbG9yOiByZ2Io NDksIDU1LCA1Nyk7Ij4KPGhyIHRhYmluZGV4PTNEIi0xIiBzdHlsZT0zRCJkaXNwbGF5OmlubGlu ZS1ibG9jazsgd2lkdGg6OTglIj4KPGRpdiBpZD0zRCJkaXZScGx5RndkTXNnIiBkaXI9M0QibHRy Ij48Zm9udCBzdHlsZT0zRCJmb250LXNpemU6MTFwdCIgZmFjZT0KPTNEIkNhbGlicmksIHNhbnMt c2VyaWYiIGNvbG9yPTNEIiMwMDAwMDAiPjxiPkZyb206PC9iPiBEZXZpbiBBY29zdGEgJmx0O2Rl PQp2aW5AcGFic3RhdGVuY2lvLmNvbSZndDs8YnI+CjxiPlNlbnQ6PC9iPiBNb25kYXksIEF1Z3Vz dCA3LCAyMDE3IDc6NDYgQU08YnI+CjxiPlRvOjwvYj4gTW9hY2lyIEZlcnJlaXJhOyB1c2Vyc0Bv dmlydC5vcmc8YnI+CjxiPlN1YmplY3Q6PC9iPiBSZTogW292aXJ0LXVzZXJzXSBHb29kIHByYWN0 aWNlczwvZm9udD4KPGRpdj4mbmJzcDs8L2Rpdj4KPC9kaXY+CjxkaXY+CjxkaXYgaWQ9M0QiYmxv b3BfY3VzdG9tZm9udCIgc3R5bGU9M0QiY29sb3I6cmdiKDAsMCwwKTsgbWFyZ2luOjBweCI+PGZv bnQgZj0KYWNlPTNEIklucHV0IE1vbm8iPjxicj4KPC9mb250PjwvZGl2Pgo8ZGl2IGlkPTNEImJs b29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEImNvbG9yOnJnYigwLDAsMCk7IG1hcmdpbjowcHgiPjxm b250IGY9CmFjZT0zRCJJbnB1dCBNb25vIj5Nb2FjaXIsPC9mb250PjwvZGl2Pgo8ZGl2IGlkPTNE ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEImNvbG9yOnJnYigwLDAsMCk7IG1hcmdpbjowcHgi Pjxmb250IGY9CmFjZT0zRCJJbnB1dCBNb25vIj48YnI+CjwvZm9udD48L2Rpdj4KPGRpdiBpZD0z RCJibG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJjb2xvcjpyZ2IoMCwwLDApOyBtYXJnaW46MHB4 Ij48Zm9udCBmPQphY2U9M0QiSW5wdXQgTW9ubyI+SSBoYXZlIHJlY2VudGx5IGluc3RhbGxlZCBt dWx0aXBsZSBSZWQgSGF0IFZpcnR1YWxpemF0aW89Cm4gaG9zdHMgZm9yIHNldmVyYWwgZGlmZmVy ZW50IGNvbXBhbmllcywgYW5kIGhhdmUgZGVhbHQgd2l0aCB0aGUgUmVkIEhhdCBTdT0KcHBvcnQg VGVhbSBpbiBkZXB0aCBhYm91dCBvcHRpbWFsIGNvbmZpZ3VyYXRpb24KIGluIHJlZ2FyZHMgdG8g c2V0dGluZyB1cCBHbHVzdGVyRlMgbW9zdCBlZmZpY2llbnRseSBhbmQgSSB3YW50ZWQgdG8gc2hh cmUgPQp3aXRoIHlvdSB3aGF0IEkgbGVhcm5lZC48L2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxv b3BfY3VzdG9tZm9udCIgc3R5bGU9M0QiY29sb3I6cmdiKDAsMCwwKTsgbWFyZ2luOjBweCI+PGZv bnQgZj0KYWNlPTNEIklucHV0IE1vbm8iPjxicj4KPC9mb250PjwvZGl2Pgo8ZGl2IGlkPTNEImJs b29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEImNvbG9yOnJnYigwLDAsMCk7IG1hcmdpbjowcHgiPjxm b250IGY9CmFjZT0zRCJJbnB1dCBNb25vIj5JbiBnZW5lcmFsIFJlZCBIYXQgVmlydHVhbGl6YXRp b24gdGVhbSBmcm93bnMgdXBvbiB1c2luZz0KIGVhY2ggRElTSyBvZiB0aGUgc3lzdGVtIGFzIGp1 c3QgYSBKQk9ELCBzdXJlIHRoZXJlIGlzIHNvbWUgcHJvdGVjdGlvbiBieSBoPQphdmluZyB0aGUg ZGF0YSByZXBsaWNhdGVkLCBob3dldmVyLCB0aGUKIHJlY29tbWVuZGF0aW9uIGlzIHRvIHVzZSBS QUlEIDYgKHByZWZlcnJlZCkgb3IgUkFJRC01LCBvciBhdCBsZWFzdCBSQUlELTEgPQphdCB0aGUg dmVyeSBsZWFzdC48L2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIgc3R5 bGU9M0QiY29sb3I6cmdiKDAsMCwwKTsgbWFyZ2luOjBweCI+PGZvbnQgZj0KYWNlPTNEIklucHV0 IE1vbm8iPjxicj4KPC9mb250PjwvZGl2Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0 eWxlPTNEIm1hcmdpbjowcHgiPjxmb250IGZhY2U9M0QiSW5wdXQgTW9ubyI9Cj5IZXJlIGlzIHRo ZSZuYnNwO2RpcmVjdCBxdW90ZSBmcm9tIFJlZCBIYXQgd2hlbiBJIGFza2VkIGFib3V0IFJBSUQg YW5kIEJyaT0KY2tzOjwvZm9udD48L2Rpdj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21mb250IiBz dHlsZT0zRCJtYXJnaW46MHB4Ij48Zm9udCBmYWNlPTNEIklucHV0IE1vbm8iPQo+PGk+PGJyPgo8 L2k+PC9mb250PjwvZGl2Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEIm1h cmdpbjowcHgiPjxmb250IGZhY2U9M0QiSW5wdXQgTW9ubyI9Cj48aT4mcXVvdDtBIHR5cGljYWwg R2x1c3RlciBjb25maWd1cmF0aW9uIHdvdWxkIHVzZSBSQUlEIHVuZGVybmVhdGggdGhlIGJyaT0K Y2tzLiBSQUlEIDYgaXMgbW9zdCB0eXBpY2FsIGFzIGl0IGdpdmVzIHlvdSAyIGRpc2sgZmFpbHVy ZSBwcm90ZWN0aW9uLCBidXQgPQpSQUlEIDUgY291bGQgYmUgdXNlZCB0b28uJm5ic3A7T25jZSB5 b3UgaGF2ZSB0aGUKIFJBSURlZCBicmlja3MsIHlvdSdkIHRoZW4gYXBwbHkgdGhlIGRlc2lyZWQg cmVwbGljYXRpb24gb24gdG9wIG9mIHRoYXQuIFRoPQplIG1vc3QgcG9wdWxhciB3YXkgb2YgZG9p bmcgdGhpcyB3b3VsZCBiZSBkaXN0cmlidXRlZCByZXBsaWNhdGVkIHdpdGggMnggcmU9CnBsaWNh dGlvbi4mbmJzcDtJbiBnZW5lcmFsIHlvdSdsbCBnZXQgYmV0dGVyIHBlcmZvcm1hbmNlIHdpdGgg bGFyZ2VyIGJyaWNrcz0KLiAxMiBkcml2ZXMgaXMgb2Z0ZW4gYSBzd2VldCBzcG90LiBBbm90aGVy CiBvcHRpb24gd291bGQgYmUgdG8gY3JlYXRlIGEgc2VwYXJhdGUgdGllciB1c2luZyBhbGwgU1NE PTkycy49OTQmbmJzcDs8L2k+PD0KL2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9t Zm9udCIgc3R5bGU9M0QibWFyZ2luOjBweCI+PGJyPgo8L2Rpdj4KPGRpdiBpZD0zRCJibG9vcF9j dXN0b21mb250IiBzdHlsZT0zRCJtYXJnaW46MHB4Ij48Zm9udCBmYWNlPTNEIklucHV0IE1vbm8i PQo+PGk+SW4gb3JkZXIgdG8gU1NEJm5ic3A7dGllcmluZyBmcm9tIG15IHVuZGVyc3RhbmRpbmcg eW91IHdvdWxkIG5lZWQgMSB4IE49ClZNZSBkcml2ZSBpbiBlYWNoIHNlcnZlciwgb3IgNCB4IFNT RCBob3QgdGllciAoaXQgbmVlZHMgdG8gYmUgZGlzdHJpYnV0ZWQsID0KcmVwbGljYXRlZCBmb3Ig dGhlIGhvdCB0aWVyIGlmIG5vdCB1c2luZyBOVk1FKS4KIFNvIHdpdGggeW91IG9ubHkgaGF2aW5n IDEgU1NEIGRyaXZlIGluIGVhY2ggc2VydmVyLCZuYnNwO0k9OTJkIHN1Z2dlc3QgbWF5PQpiZSBs b29raW5nIGludG8gdGhlIE5WTUUgb3B0aW9uLiZuYnNwOzwvaT48L2ZvbnQ+PC9kaXY+CjxkaXYg aWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9M0QibWFyZ2luOjBweCI+PGZvbnQgZmFjZT0z RCJJbnB1dCBNb25vIj0KPjxpPjxicj4KPC9pPjwvZm9udD48L2Rpdj4KPGRpdiBpZD0zRCJibG9v cF9jdXN0b21mb250IiBzdHlsZT0zRCJtYXJnaW46MHB4Ij48Zm9udCBmYWNlPTNEIklucHV0IE1v bm8iPQo+PGk+U2luY2UgeW91ciB1c2luZyBvbmx5IDMtc2VydmVycywgd2hhdCBJPTkyZCBwcm9i YWJseSBzdWdnZXN0IGlzIHRvIGRvICg9CjIgUmVwbGljYXMgJiM0MzsgQXJiaXRlciBOb2RlKSwg dGhpcyBzZXR1cCBhY3R1YWxseSBkb2Vzbj05MnQgcmVxdWlyZSB0aGUgMz0KcmQgc2VydmVyIHRv IGhhdmUgYmlnIGRyaXZlcyBhdCBhbGwgYXMgaXQgb25seQogc3RvcmVzIG1ldGEtZGF0YSBhYm91 dCB0aGUgZmlsZXMgYW5kIG5vdCBhY3R1YWxseSBhIGZ1bGwgY29weS4mbmJzcDs8L2k+PC89CmZv bnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9M0QibWFyZ2luOjBw eCI+PGZvbnQgZmFjZT0zRCJJbnB1dCBNb25vIj0KPjxpPjxicj4KPC9pPjwvZm9udD48L2Rpdj4K PGRpdiBpZD0zRCJibG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJtYXJnaW46MHB4Ij48Zm9udCBm YWNlPTNEIklucHV0IE1vbm8iPQo+PGk+UGxlYXNlIHNlZSB0aGUgYXR0YWNoZWQgZG9jdW1lbnQg dGhhdCB3YXMgZ2l2ZW4gdG8gbWUgYnkgUmVkIEhhdCB0byBnZXQ9CiBtb3JlIGluZm9ybWF0aW9u IG9uIHRoaXMuIEhvcGUgdGhpcyBpbmZvcm1hdGlvbiBoZWxwcyB5b3UuPC9pPjwvZm9udD48L2Rp dj0KPgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEIm1hcmdpbjowcHgiPjxm b250IGZhY2U9M0QiSW5wdXQgTW9ubyI9Cj48aT48YnI+CjwvaT48L2ZvbnQ+PC9kaXY+Cjxicj4K PGRpdiBpZD0zRCJibG9vcF9zaWduXzE1MDIwODczNzY3MjU0NjkxODQiIGNsYXNzPTNEImJsb29w X3NpZ24iPjxzcGFuIHN0eWxlPQo9M0QiZm9udC1mYW1pbHk6J2hlbHZldGljYSBOZXVlJyxoZWx2 ZXRpY2E7IGZvbnQtc2l6ZToxNHB4Ij4tLTwvc3Bhbj48YnIgc3Q9CnlsZT0zRCJmb250LWZhbWls eTonaGVsdmV0aWNhIE5ldWUnLGhlbHZldGljYTsgZm9udC1zaXplOjE0cHgiPgo8ZGl2IGNsYXNz PTNEImdtYWlsX3NpZ25hdHVyZSIgc3R5bGU9M0QiZm9udC1mYW1pbHk6J2hlbHZldGljYSBOZXVl JyxoZWx2ZXQ9CmljYTsgZm9udC1zaXplOjE0cHgiPgo8ZGl2IGRpcj0zRCJsdHIiPgo8ZGl2Pjxi cj4KPC9kaXY+CjxkaXY+RGV2aW4gQWNvc3RhLCBSSENBLCBSSFZDQTwvZGl2Pgo8ZGl2PlJlZCBI YXQgQ2VydGlmaWVkIEFyY2hpdGVjdDwvZGl2Pgo8ZGl2PjwvZGl2Pgo8L2Rpdj4KPC9kaXY+Cjwv ZGl2Pgo8YnI+CjxwIGNsYXNzPTNEImFpcm1haWxfb24iPk9uIEF1Z3VzdCA2LCAyMDE3IGF0IDc6 Mjk6MjkgUE0sIE1vYWNpciBGZXJyZWlyYSAoPD0KYSBocmVmPTNEIm1haWx0bzptb2FjaXJmZXJy ZWlyYUBob3RtYWlsLmNvbSI+bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb208L2E+PQopIHdyb3Rl OjwvcD4KPGJsb2NrcXVvdGUgdHlwZT0zRCJjaXRlIiBjbGFzcz0zRCJjbGVhbl9icSI+PHNwYW4+ CjxkaXYgZGlyPTNEImx0ciI+CjxkaXY+PC9kaXY+CjxkaXY+CjxkaXYgaWQ9M0QiZGl2dGFnZGVm YXVsdHdyYXBwZXIiIGRpcj0zRCJsdHIiIHN0eWxlPTNEImZvbnQtc2l6ZToxMnB0OyBjb2xvcj0K OiMwMDAwMDA7IGZvbnQtZmFtaWx5OkNhbGlicmksSGVsdmV0aWNhLHNhbnMtc2VyaWYiPgo8cD48 c3Bhbj5JIGFtIHdpbGxpbmcgdG8gYXNzZW1ibGUgYSBvVmlydCAmcXVvdDtwb2QmcXVvdDssIG1h ZGUgb2YgMyBzZXJ2ZXI9CnMsIGVhY2ggd2l0aCAyIENQVSBzb2NrZXRzIG9mIDEyIGNvcmVzLCAy NTZHQiBSQU0sIDcgSEREIDEwSywgMSBTU0QuIFRoZSBpZD0KZWEgaXMgdG8gdXNlIEdsdXN0ZXJG UyB0byBwcm92aWRlIEhBIGZvciB0aGUgVk1zLiBUaGUgMyBzZXJ2ZXJzIGhhdmUgYSBkdWFsPQog NDBHYiBOSUMgYW5kIGEgZHVhbCAxMEdiIE5JQy4gU28gbXkgaW50ZW50aW9uCiBpcyB0byBjcmVh dGUgYSBsb29wIGxpa2UgYSBzZXJ2ZXIgdHJpYW5nbGUgdXNpbmcgdGhlIDQwR2IgTklDcyBmb3Ig dmlydHVhbD0KaXphdGlvbiBmaWxlcyAoVk1zIC5xY293MikgYWNjZXNzIGFuZCB0byBtb3ZlIFZN cyBhcm91bmQgdGhlIHBvZCAoZWFzdCAvd2VzPQp0IHRyYWZmaWMpIHdoaWxlIHVzaW5nIHRoZSAx MEdiIGludGVyZmFjZXMgZm9yIGdpdmluZyBzZXJ2aWNlcyB0byB0aGUgb3V0c2k9CmRlIHdvcmxk IChub3J0aC9zb3V0aCB0cmFmZmljKS48L3NwYW4+PC9wPgo8cD48YnI+CjwvcD4KPHA+VGhpcyBz YWlkLCBteSBmaXJzdCBxdWVzdGlvbiBpczogSG93IHNob3VsZCBJIGRlcGxveSBHbHVzdGVyRlMg aW4gc3VjaCBvPQpWaXJ0IHNjZW5hcmlvPyBNeSBxdWVzdGlvbnMgYXJlOjwvcD4KPHA+PGJyPgo8 L3A+CjxwPjEgLSBTaG91bGQgSSBjcmVhdGUgMyBSQUlEIChpLmUuOiBSQUlEIDUpLCBvbmUgb24g ZWFjaCBvVmlydCBub2RlLCBhbmQgdD0KaGVuIGNyZWF0ZSBhIEdsdXN0ZXJGUyB1c2luZyB0aGVt PzwvcD4KPHA+MiAtIEluc3RlYWQsIHNob3VsZCBJIGNyZWF0ZSBhIEpCT0QgYXJyYXkgbWFkZSBv ZiBhbGwgc2VydmVyJ3MgZGlza3M/PC9wPQo+CjxwPjMgLSBXaGF0IGlzIHRoZSBiZXN0IEdsdXN0 ZXIgY29uZmlndXJhdGlvbiB0byBwcm92aWRlIGZvciBIQSB3aGlsZSBub3QgYz0Kb25zdW1pbmcg dG9vIG11Y2ggZGlzayBzcGFjZT88YnI+CjwvcD4KPHA+NCAtIERvZXMgYSBvVmlydCBoeXBlcnZp c29yIHBvZCBsaWtlIEkgYW0gcGxhbm5pbmcgdG8gYnVpbGQsIGFuZCB0aGUgdmlyPQp0dWFsaXph dGlvbiBlbnZpcm9ubWVudCwgYmVuZWZpdHMgZnJvbSB0aWVyaW5nIHdoZW4gdXNpbmcgYSBTU0Qg ZGlzaz8gQW5kIHk9CmVzLCB3aWxsIEdsdXN0ZXIgZG8gaXQgYnkgZGVmYXVsdCBvciBJIGhhdmUg dG8gY29uZmlndXJlIGl0IHRvIGRvIHNvPzwvcD4KPHA+PGJyPgo8L3A+CjxwPkF0IHRoZSBib3R0 b20gbGluZSwgd2hhdCBpcyB0aGUgZ29vZCBwcmFjdGljZSBmb3IgdXNpbmcgR2x1c3RlckZTIGlu IHNtYT0KbGwgcG9kcyBmb3IgZW50ZXJwcmlzZXM/PGJyPgo8L3A+CjxwPjxicj4KPC9wPgo8cD5Z b3Ugb3Bpbmlvbi9mZWVkYmFjayB3aWxsIGJlIHJlYWxseSBhcHByZWNpYXRlZCE8L3A+CjxwPk1v YWNpcjxicj4KPC9wPgo8L2Rpdj4KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX18gPGJyPgpVc2VycyBtYWlsaW5nIGxpc3QgPGJyPgo8YSBocmVmPTNEIm1haWx0 bzpVc2Vyc0BvdmlydC5vcmciPlVzZXJzQG92aXJ0Lm9yZzwvYT4gPGJyPgo8YSBocmVmPTNEImh0 dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyI+aHR0cDovL2xpc3Rz Lm92aXI9CnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnM8L2E+Cjxicj4KPC9kaXY+CjwvZGl2 Pgo8L3NwYW4+PC9ibG9ja3F1b3RlPgo8L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8L2JvZHk+CjwvaHRt bD4KCi0tXzAwMF9WSTFQMTkwTUIwMjg1QUY0N0JFOUI3RkVFM0VFQjUwNjdDOEI1MFZJMVAxOTBN QjAyODVFVVJQXy0tCg== --===============6238324484136124314==-- From johan at kafit.se Tue Aug 8 06:04:04 2017 Content-Type: multipart/mixed; boundary="===============0182984489577361123==" MIME-Version: 1.0 From: Johan Bernhardsson To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Tue, 08 Aug 2017 08:03:54 +0200 Message-ID: <15dc0713d28.2768.ffe2d6cdf294efc1bb86526523af9945@kafit.se> In-Reply-To: VI1P190MB0285AF47BE9B7FEE3EEB5067C8B50@VI1P190MB0285.EURP190.PROD.OUTLOOK.COM --===============0182984489577361123== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. ------------15dc07141763cbc27688fec2dc Content-Type: text/plain; format=3Dflowed; charset=3D"UTF-8" Content-Transfer-Encoding: quoted-printable You attach the ssd as a hot tier with a gluster command. I don't think th= =3D at=3D20 gdeploy or ovirt gui can do it. The gluster docs and redhat docs explains tiering quite good. /Johan On August 8, 2017 07:06:42 Moacir Ferreira w= =3D rote: > Hi Devin, > > > Please consider that for the OS I have a RAID 1. Now, lets say I use RA= =3D ID 5=3D20 > to assemble a single disk on each server. In this case, the SSD will no= =3D t=3D20 > make any difference, right? I guess that to be possible to use it, the = =3D SSD=3D20 > should not be part of the RAID 5. In this case I could create a logical= =3D =3D20 > volume made of the RAIDed brick and then extend it using the SSD. I.e.:= =3D =3D20 > Using gdeploy: > > > [disktype] > > jbod > > .... > > [pv1] > > action=3D3Dcreate > > devices=3D3Dsdb, sdc > > wipefs=3D3Dyes > > ignore_vg_erros=3D3Dno > > > [vg1] > > action=3D3Dcreate > > vgname=3D3Dgluster_vg_jbod > > pvname=3D3Dsdb > > ignore_vg_erros=3D3Dno > > > [vg2] > > action=3D3Dextend > > vgname=3D3Dgluster_vg_jbod > > pvname=3D3Dsdc > > ignore_vg_erros=3D3Dno > > > But will Gluster be able to auto-detect and use this SSD brick for tier= =3D ing?=3D20 > Do I have to do some other configurations? Also, as the VM files (.qcow= =3D 2)=3D20 > are quite big will I benefit from tiering? This is wrong and my approac= =3D h=3D20 > should be other? > > > Thanks, > > Moacir > > > ________________________________ > From: Devin Acosta > Sent: Monday, August 7, 2017 7:46 AM > To: Moacir Ferreira; users(a)ovirt.org > Subject: Re: [ovirt-users] Good practices > > > Moacir, > > I have recently installed multiple Red Hat Virtualization hosts for sev= =3D eral=3D20 > different companies, and have dealt with the Red Hat Support Team in de= =3D pth=3D20 > about optimal configuration in regards to setting up GlusterFS most=3D20 > efficiently and I wanted to share with you what I learned. > > In general Red Hat Virtualization team frowns upon using each DISK of t= =3D he=3D20 > system as just a JBOD, sure there is some protection by having the data= =3D =3D20 > replicated, however, the recommendation is to use RAID 6 (preferred) or= =3D =3D20 > RAID-5, or at least RAID-1 at the very least. > > Here is the direct quote from Red Hat when I asked about RAID and Brick= =3D s: > > "A typical Gluster configuration would use RAID underneath the bricks. = =3D RAID=3D20 > 6 is most typical as it gives you 2 disk failure protection, but RAID 5= =3D =3D20 > could be used too. Once you have the RAIDed bricks, you'd then apply th= =3D e=3D20 > desired replication on top of that. The most popular way of doing this=3D= 20 > would be distributed replicated with 2x replication. In general you'll = =3D get=3D20 > better performance with larger bricks. 12 drives is often a sweet spot.= =3D =3D20 > Another option would be to create a separate tier using all SSD=3DE2=3D80= =3D99=3D s.=3DE2=3D80=3D9D > > In order to SSD tiering from my understanding you would need 1 x NVMe d= =3D rive=3D20 > in each server, or 4 x SSD hot tier (it needs to be distributed, replic= =3D ated=3D20 > for the hot tier if not using NVME). So with you only having 1 SSD driv= =3D e in=3D20 > each server, I=3DE2=3D80=3D99d suggest maybe looking into the NVME option. > > Since your using only 3-servers, what I=3DE2=3D80=3D99d probably suggest = is t=3D o do (2=3D20 > Replicas + Arbiter Node), this setup actually doesn=3DE2=3D80=3D99t requi= re t=3D he 3rd=3D20 > server to have big drives at all as it only stores meta-data about the=3D= 20 > files and not actually a full copy. > > Please see the attached document that was given to me by Red Hat to get= =3D =3D20 > more information on this. Hope this information helps you. > > > -- > > Devin Acosta, RHCA, RHVCA > Red Hat Certified Architect > > > On August 6, 2017 at 7:29:29 PM, Moacir Ferreira=3D20 > (moacirferreira(a)hotmail.com) wrote: > > I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 = =3D CPU=3D20 > sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use=3D20 > GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC= =3D and=3D20 > a dual 10Gb NIC. So my intention is to create a loop like a server tria= =3D ngle=3D20 > using the 40Gb NICs for virtualization files (VMs .qcow2) access and to= =3D =3D20 > move VMs around the pod (east /west traffic) while using the 10Gb=3D20 > interfaces for giving services to the outside world (north/south traffi= =3D c). > > > This said, my first question is: How should I deploy GlusterFS in such=3D= 20 > oVirt scenario? My questions are: > > > 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and = =3D then=3D20 > create a GlusterFS using them? > > 2 - Instead, should I create a JBOD array made of all server's disks? > > 3 - What is the best Gluster configuration to provide for HA while not=3D= 20 > consuming too much disk space? > > 4 - Does a oVirt hypervisor pod like I am planning to build, and the=3D20 > virtualization environment, benefits from tiering when using a SSD disk= =3D ?=3D20 > And yes, will Gluster do it by default or I have to configure it to do = =3D so? > > > At the bottom line, what is the good practice for using GlusterFS in sm= =3D all=3D20 > pods for enterprises? > > > You opinion/feedback will be really appreciated! > > Moacir > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > > > ---------- > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > ------------15dc07141763cbc27688fec2dc Content-Type: text/html; charset=3D"UTF-8" Content-Transfer-Encoding: quoted-printable

You attach the ssd as a hot tier with a gluster command. I don't think that gdeploy or ovirt gui can = =3D do it.

The gluster docs and redhat docs explains tiering quite good.

/Johan

On August 8, 2017 07:06:42 Moacir Ferreira <moacirferreira(a)hotmail.com>= =3D ; wrote:

Hi Devin,

Please consider that for the OS I have a RAID 1. Now, lets say I use RAID 5 to assemble a single disk on each server. In this case, the SSD wi= =3D ll not make any difference, right? I guess that to be possible to use it, th= =3D e SSD should not be part of the RAID 5. In this case I could create a logical volume made of the RAIDed brick and then extend it using the SSD. I.e.: Using gdeploy:

[disktype]

jbod

....

[pv1]

action=3D3Dcreate

devices=3D3Dsdb, sdc

wipefs=3D3Dyes

ignore_vg_erros=3D3Dno

[vg1]

action=3D3Dcreate

vgname=3D3Dgluster_vg_jbod

pvname=3D3Dsdb

ignore_vg_erros=3D3Dno

[vg2]

action=3D3Dextend

vgname=3D3Dgluster_vg_jbod

pvname=3D3Dsdc

ignore_vg_erros=3D3Dno

But will Gluster be able to auto-detect and use this SSD brick for tiering? Do I have to do some other configurations? Also, as the VM files (.qcow2) are quite big will I benefit from tiering? This is wrong and my approach should be other?

Thanks,

Moacir

From: Devin Acosta <devin(a)pabstatencio.com>
Sent: Monday, August 7, 2017 7:46 AM
To: Moacir Ferreira; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices

Moacir,

I have recently installed multiple Red Hat Virtualiza= =3D tion hosts for several different companies, and have dealt with the Red Hat Support Team in depth about optimal configuration in regards to setting up GlusterFS most efficiently and I wanted to shar= =3D e with you what I learned.

In general Red Hat Virtualization team frowns upon us= =3D ing each DISK of the system as just a JBOD, sure there is some protection by having the data replicated, however, the recommendation is to use RAID 6 (preferred) or RAID-5, or at least RAID-= =3D 1 at the very least.

Here is the direct quote from Red Hat when I asked about RAID and Bricks:

"A typical Gluster configuration would use RA= =3D ID underneath the bricks. RAID 6 is most typical as it gives you 2 disk failure protection, but RAID 5 could be used too. Once you have the RAIDed bricks, you'd then apply the desired replication on top of that. The most popular way of doing this would be distributed replicated with 2= =3D x replication. In general you'll get better performance with larger bricks. 12 drives is often a sweet spot. Another option would be to create a separate tier using all SSD=3DE2=3D80=3D99s.=3DE2=3D80=3D9D

In order to SSD tiering from my understanding you would need 1 x NVMe drive in each server, or 4 x SSD hot tier (it needs to be distributed, replicated for the hot tier if not using NVME). So with you only having 1 SSD drive in each server, I=3DE2=3D80=3D99d= sug=3D gest maybe looking into the NVME option.

Since your using only 3-servers, what I=3DE2=3D80= =3D99d =3D probably suggest is to do (2 Replicas + Arbiter Node), this setup actually doesn=3DE2=3D80=3D99t require the 3rd server to have big drives at all as i= t on=3D ly stores meta-data about the files and not actually a full copy.

Please see the attached document that was given to= =3D me by Red Hat to get more information on this. Hope this information helps you.

--

Devin Acosta, RHCA, RHVCA

Red Hat Certified Architect

On August 6, 2017 at 7:29:29 PM, Moacir Ferreira = =3D (moacirferreira(a)hotmail.com= =3D ) wrote:

I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD= =3D . The idea is to use GlusterFS to provide HA for the VMs. The 3 servers hav= =3D e a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop like a server triangle using the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VMs around the pod (east /west traffic) while using the 10Gb interfaces for giving services = =3D to the outside world (north/south traffic).

This said, my first question is: How should I deploy GlusterFS in such oVirt scenario? My questions are:

1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then create a GlusterFS using them?

2 - Instead, should I create a JBOD array made of all server's disks?<= =3D /p>
3 - What is the best Gluster configuration to provide for HA while not consuming too much disk space?

4 - Does a oVirt hypervisor pod like I am planning to build, and the virtualization environment, benefits from tiering when using a SSD disk? And yes, will Gluster do it by default or I have to configure it to do so= =3D ?

At the bottom line, what is the good practice for using GlusterFS in small pods for enterprises?

You opinion/feedback will be really appreciated!

Moacir

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt= =3D .org/mailman/listinfo/users

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt= =3D .org/mailman/listinfo/users

------------15dc07141763cbc27688fec2dc-- --===============0182984489577361123== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t MTVkYzA3MTQxNzYzY2JjMjc2ODhmZWMyZGMKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBmb3Jt YXQ9Zmxvd2VkOyBjaGFyc2V0PSJVVEYtOCIKQ29udGVudC1UcmFuc2Zlci1FbmNvZGluZzogcXVv dGVkLXByaW50YWJsZQoKWW91IGF0dGFjaCB0aGUgc3NkIGFzIGEgaG90IHRpZXIgd2l0aCBhIGds dXN0ZXIgY29tbWFuZC4gSSBkb24ndCB0aGluayB0aD0KYXQ9MjAKZ2RlcGxveSBvciBvdmlydCBn dWkgY2FuIGRvIGl0LgoKVGhlIGdsdXN0ZXIgZG9jcyBhbmQgcmVkaGF0IGRvY3MgZXhwbGFpbnMg dGllcmluZyBxdWl0ZSBnb29kLgoKL0pvaGFuCgoKT24gQXVndXN0IDgsIDIwMTcgMDc6MDY6NDIg TW9hY2lyIEZlcnJlaXJhIDxtb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbT4gdz0Kcm90ZToKCj4g SGkgRGV2aW4sCj4KPgo+IFBsZWFzZSBjb25zaWRlciB0aGF0IGZvciB0aGUgT1MgSSBoYXZlIGEg UkFJRCAxLiBOb3csIGxldHMgc2F5IEkgdXNlIFJBPQpJRCA1PTIwCj4gdG8gYXNzZW1ibGUgYSBz aW5nbGUgZGlzayBvbiBlYWNoIHNlcnZlci4gSW4gdGhpcyBjYXNlLCB0aGUgU1NEIHdpbGwgbm89 CnQ9MjAKPiBtYWtlIGFueSBkaWZmZXJlbmNlLCByaWdodD8gSSBndWVzcyB0aGF0IHRvIGJlIHBv c3NpYmxlIHRvIHVzZSBpdCwgdGhlID0KU1NEPTIwCj4gc2hvdWxkIG5vdCBiZSBwYXJ0IG9mIHRo ZSBSQUlEIDUuIEluIHRoaXMgY2FzZSBJIGNvdWxkIGNyZWF0ZSBhIGxvZ2ljYWw9Cj0yMAo+IHZv bHVtZSBtYWRlIG9mIHRoZSBSQUlEZWQgYnJpY2sgYW5kIHRoZW4gZXh0ZW5kIGl0IHVzaW5nIHRo ZSBTU0QuIEkuZS46PQo9MjAKPiBVc2luZyBnZGVwbG95Ogo+Cj4KPiBbZGlza3R5cGVdCj4KPiBq Ym9kCj4KPiAuLi4uCj4KPiBbcHYxXQo+Cj4gYWN0aW9uPTNEY3JlYXRlCj4KPiBkZXZpY2VzPTNE c2RiLCBzZGMKPgo+IHdpcGVmcz0zRHllcwo+Cj4gaWdub3JlX3ZnX2Vycm9zPTNEbm8KPgo+Cj4g W3ZnMV0KPgo+IGFjdGlvbj0zRGNyZWF0ZQo+Cj4gdmduYW1lPTNEZ2x1c3Rlcl92Z19qYm9kCj4K PiBwdm5hbWU9M0RzZGIKPgo+IGlnbm9yZV92Z19lcnJvcz0zRG5vCj4KPgo+IFt2ZzJdCj4KPiBh Y3Rpb249M0RleHRlbmQKPgo+IHZnbmFtZT0zRGdsdXN0ZXJfdmdfamJvZAo+Cj4gcHZuYW1lPTNE c2RjCj4KPiBpZ25vcmVfdmdfZXJyb3M9M0Rubwo+Cj4KPiBCdXQgd2lsbCBHbHVzdGVyIGJlIGFi bGUgdG8gYXV0by1kZXRlY3QgYW5kIHVzZSB0aGlzIFNTRCBicmljayBmb3IgdGllcj0KaW5nPz0y MAo+IERvIEkgaGF2ZSB0byBkbyBzb21lIG90aGVyIGNvbmZpZ3VyYXRpb25zPyBBbHNvLCBhcyB0 aGUgVk0gZmlsZXMgKC5xY293PQoyKT0yMAo+IGFyZSBxdWl0ZSBiaWcgd2lsbCBJIGJlbmVmaXQg ZnJvbSB0aWVyaW5nPyBUaGlzIGlzIHdyb25nIGFuZCBteSBhcHByb2FjPQpoPTIwCj4gc2hvdWxk IGJlIG90aGVyPwo+Cj4KPiBUaGFua3MsCj4KPiBNb2FjaXIKPgo+Cj4gX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX18KPiBGcm9tOiBEZXZpbiBBY29zdGEgPGRldmluQHBhYnN0YXRlbmNp by5jb20+Cj4gU2VudDogTW9uZGF5LCBBdWd1c3QgNywgMjAxNyA3OjQ2IEFNCj4gVG86IE1vYWNp ciBGZXJyZWlyYTsgdXNlcnNAb3ZpcnQub3JnCj4gU3ViamVjdDogUmU6IFtvdmlydC11c2Vyc10g R29vZCBwcmFjdGljZXMKPgo+Cj4gTW9hY2lyLAo+Cj4gSSBoYXZlIHJlY2VudGx5IGluc3RhbGxl ZCBtdWx0aXBsZSBSZWQgSGF0IFZpcnR1YWxpemF0aW9uIGhvc3RzIGZvciBzZXY9CmVyYWw9MjAK PiBkaWZmZXJlbnQgY29tcGFuaWVzLCBhbmQgaGF2ZSBkZWFsdCB3aXRoIHRoZSBSZWQgSGF0IFN1 cHBvcnQgVGVhbSBpbiBkZT0KcHRoPTIwCj4gYWJvdXQgb3B0aW1hbCBjb25maWd1cmF0aW9uIGlu IHJlZ2FyZHMgdG8gc2V0dGluZyB1cCBHbHVzdGVyRlMgbW9zdD0yMAo+IGVmZmljaWVudGx5IGFu ZCBJIHdhbnRlZCB0byBzaGFyZSB3aXRoIHlvdSB3aGF0IEkgbGVhcm5lZC4KPgo+IEluIGdlbmVy YWwgUmVkIEhhdCBWaXJ0dWFsaXphdGlvbiB0ZWFtIGZyb3ducyB1cG9uIHVzaW5nIGVhY2ggRElT SyBvZiB0PQpoZT0yMAo+IHN5c3RlbSBhcyBqdXN0IGEgSkJPRCwgc3VyZSB0aGVyZSBpcyBzb21l IHByb3RlY3Rpb24gYnkgaGF2aW5nIHRoZSBkYXRhPQo9MjAKPiByZXBsaWNhdGVkLCBob3dldmVy LCB0aGUgcmVjb21tZW5kYXRpb24gaXMgdG8gdXNlIFJBSUQgNiAocHJlZmVycmVkKSBvcj0KPTIw Cj4gUkFJRC01LCBvciBhdCBsZWFzdCBSQUlELTEgYXQgdGhlIHZlcnkgbGVhc3QuCj4KPiBIZXJl IGlzIHRoZSBkaXJlY3QgcXVvdGUgZnJvbSBSZWQgSGF0IHdoZW4gSSBhc2tlZCBhYm91dCBSQUlE IGFuZCBCcmljaz0KczoKPgo+ICJBIHR5cGljYWwgR2x1c3RlciBjb25maWd1cmF0aW9uIHdvdWxk IHVzZSBSQUlEIHVuZGVybmVhdGggdGhlIGJyaWNrcy4gPQpSQUlEPTIwCj4gNiBpcyBtb3N0IHR5 cGljYWwgYXMgaXQgZ2l2ZXMgeW91IDIgZGlzayBmYWlsdXJlIHByb3RlY3Rpb24sIGJ1dCBSQUlE IDU9Cj0yMAo+IGNvdWxkIGJlIHVzZWQgdG9vLiBPbmNlIHlvdSBoYXZlIHRoZSBSQUlEZWQgYnJp Y2tzLCB5b3UnZCB0aGVuIGFwcGx5IHRoPQplPTIwCj4gZGVzaXJlZCByZXBsaWNhdGlvbiBvbiB0 b3Agb2YgdGhhdC4gVGhlIG1vc3QgcG9wdWxhciB3YXkgb2YgZG9pbmcgdGhpcz0yMAo+IHdvdWxk IGJlIGRpc3RyaWJ1dGVkIHJlcGxpY2F0ZWQgd2l0aCAyeCByZXBsaWNhdGlvbi4gSW4gZ2VuZXJh bCB5b3UnbGwgPQpnZXQ9MjAKPiBiZXR0ZXIgcGVyZm9ybWFuY2Ugd2l0aCBsYXJnZXIgYnJpY2tz LiAxMiBkcml2ZXMgaXMgb2Z0ZW4gYSBzd2VldCBzcG90Lj0KPTIwCj4gQW5vdGhlciBvcHRpb24g d291bGQgYmUgdG8gY3JlYXRlIGEgc2VwYXJhdGUgdGllciB1c2luZyBhbGwgU1NEPUUyPTgwPTk5 PQpzLj1FMj04MD05RAo+Cj4gSW4gb3JkZXIgdG8gU1NEIHRpZXJpbmcgZnJvbSBteSB1bmRlcnN0 YW5kaW5nIHlvdSB3b3VsZCBuZWVkIDEgeCBOVk1lIGQ9CnJpdmU9MjAKPiBpbiBlYWNoIHNlcnZl ciwgb3IgNCB4IFNTRCBob3QgdGllciAoaXQgbmVlZHMgdG8gYmUgZGlzdHJpYnV0ZWQsIHJlcGxp Yz0KYXRlZD0yMAo+IGZvciB0aGUgaG90IHRpZXIgaWYgbm90IHVzaW5nIE5WTUUpLiBTbyB3aXRo IHlvdSBvbmx5IGhhdmluZyAxIFNTRCBkcml2PQplIGluPTIwCj4gZWFjaCBzZXJ2ZXIsIEk9RTI9 ODA9OTlkIHN1Z2dlc3QgbWF5YmUgbG9va2luZyBpbnRvIHRoZSBOVk1FIG9wdGlvbi4KPgo+IFNp bmNlIHlvdXIgdXNpbmcgb25seSAzLXNlcnZlcnMsIHdoYXQgST1FMj04MD05OWQgcHJvYmFibHkg c3VnZ2VzdCBpcyB0PQpvIGRvICgyPTIwCj4gUmVwbGljYXMgKyBBcmJpdGVyIE5vZGUpLCB0aGlz IHNldHVwIGFjdHVhbGx5IGRvZXNuPUUyPTgwPTk5dCByZXF1aXJlIHQ9CmhlIDNyZD0yMAo+IHNl cnZlciB0byBoYXZlIGJpZyBkcml2ZXMgYXQgYWxsIGFzIGl0IG9ubHkgc3RvcmVzIG1ldGEtZGF0 YSBhYm91dCB0aGU9MjAKPiBmaWxlcyBhbmQgbm90IGFjdHVhbGx5IGEgZnVsbCBjb3B5Lgo+Cj4g UGxlYXNlIHNlZSB0aGUgYXR0YWNoZWQgZG9jdW1lbnQgdGhhdCB3YXMgZ2l2ZW4gdG8gbWUgYnkg UmVkIEhhdCB0byBnZXQ9Cj0yMAo+IG1vcmUgaW5mb3JtYXRpb24gb24gdGhpcy4gSG9wZSB0aGlz IGluZm9ybWF0aW9uIGhlbHBzIHlvdS4KPgo+Cj4gLS0KPgo+IERldmluIEFjb3N0YSwgUkhDQSwg UkhWQ0EKPiBSZWQgSGF0IENlcnRpZmllZCBBcmNoaXRlY3QKPgo+Cj4gT24gQXVndXN0IDYsIDIw MTcgYXQgNzoyOToyOSBQTSwgTW9hY2lyIEZlcnJlaXJhPTIwCj4gKG1vYWNpcmZlcnJlaXJhQGhv dG1haWwuY29tPG1haWx0bzptb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbT4pIHdyb3RlOgo+Cj4g SSBhbSB3aWxsaW5nIHRvIGFzc2VtYmxlIGEgb1ZpcnQgInBvZCIsIG1hZGUgb2YgMyBzZXJ2ZXJz LCBlYWNoIHdpdGggMiA9CkNQVT0yMAo+IHNvY2tldHMgb2YgMTIgY29yZXMsIDI1NkdCIFJBTSwg NyBIREQgMTBLLCAxIFNTRC4gVGhlIGlkZWEgaXMgdG8gdXNlPTIwCj4gR2x1c3RlckZTIHRvIHBy b3ZpZGUgSEEgZm9yIHRoZSBWTXMuIFRoZSAzIHNlcnZlcnMgaGF2ZSBhIGR1YWwgNDBHYiBOSUM9 CiBhbmQ9MjAKPiBhIGR1YWwgMTBHYiBOSUMuIFNvIG15IGludGVudGlvbiBpcyB0byBjcmVhdGUg YSBsb29wIGxpa2UgYSBzZXJ2ZXIgdHJpYT0KbmdsZT0yMAo+IHVzaW5nIHRoZSA0MEdiIE5JQ3Mg Zm9yIHZpcnR1YWxpemF0aW9uIGZpbGVzIChWTXMgLnFjb3cyKSBhY2Nlc3MgYW5kIHRvPQo9MjAK PiBtb3ZlIFZNcyBhcm91bmQgdGhlIHBvZCAoZWFzdCAvd2VzdCB0cmFmZmljKSB3aGlsZSB1c2lu ZyB0aGUgMTBHYj0yMAo+IGludGVyZmFjZXMgZm9yIGdpdmluZyBzZXJ2aWNlcyB0byB0aGUgb3V0 c2lkZSB3b3JsZCAobm9ydGgvc291dGggdHJhZmZpPQpjKS4KPgo+Cj4gVGhpcyBzYWlkLCBteSBm aXJzdCBxdWVzdGlvbiBpczogSG93IHNob3VsZCBJIGRlcGxveSBHbHVzdGVyRlMgaW4gc3VjaD0y MAo+IG9WaXJ0IHNjZW5hcmlvPyBNeSBxdWVzdGlvbnMgYXJlOgo+Cj4KPiAxIC0gU2hvdWxkIEkg Y3JlYXRlIDMgUkFJRCAoaS5lLjogUkFJRCA1KSwgb25lIG9uIGVhY2ggb1ZpcnQgbm9kZSwgYW5k ID0KdGhlbj0yMAo+IGNyZWF0ZSBhIEdsdXN0ZXJGUyB1c2luZyB0aGVtPwo+Cj4gMiAtIEluc3Rl YWQsIHNob3VsZCBJIGNyZWF0ZSBhIEpCT0QgYXJyYXkgbWFkZSBvZiBhbGwgc2VydmVyJ3MgZGlz a3M/Cj4KPiAzIC0gV2hhdCBpcyB0aGUgYmVzdCBHbHVzdGVyIGNvbmZpZ3VyYXRpb24gdG8gcHJv dmlkZSBmb3IgSEEgd2hpbGUgbm90PTIwCj4gY29uc3VtaW5nIHRvbyBtdWNoIGRpc2sgc3BhY2U/ Cj4KPiA0IC0gRG9lcyBhIG9WaXJ0IGh5cGVydmlzb3IgcG9kIGxpa2UgSSBhbSBwbGFubmluZyB0 byBidWlsZCwgYW5kIHRoZT0yMAo+IHZpcnR1YWxpemF0aW9uIGVudmlyb25tZW50LCBiZW5lZml0 cyBmcm9tIHRpZXJpbmcgd2hlbiB1c2luZyBhIFNTRCBkaXNrPQo/PTIwCj4gQW5kIHllcywgd2ls bCBHbHVzdGVyIGRvIGl0IGJ5IGRlZmF1bHQgb3IgSSBoYXZlIHRvIGNvbmZpZ3VyZSBpdCB0byBk byA9CnNvPwo+Cj4KPiBBdCB0aGUgYm90dG9tIGxpbmUsIHdoYXQgaXMgdGhlIGdvb2QgcHJhY3Rp Y2UgZm9yIHVzaW5nIEdsdXN0ZXJGUyBpbiBzbT0KYWxsPTIwCj4gcG9kcyBmb3IgZW50ZXJwcmlz ZXM/Cj4KPgo+IFlvdSBvcGluaW9uL2ZlZWRiYWNrIHdpbGwgYmUgcmVhbGx5IGFwcHJlY2lhdGVk IQo+Cj4gTW9hY2lyCj4KPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fXwo+IFVzZXJzIG1haWxpbmcgbGlzdAo+IFVzZXJzQG92aXJ0Lm9yZzxtYWlsdG86VXNl cnNAb3ZpcnQub3JnPgo+IGh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91 c2Vycwo+Cj4KPgo+IC0tLS0tLS0tLS0KPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fXwo+IFVzZXJzIG1haWxpbmcgbGlzdAo+IFVzZXJzQG92aXJ0Lm9yZwo+ IGh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vycwo+CgotLS0tLS0t LS0tLS0xNWRjMDcxNDE3NjNjYmMyNzY4OGZlYzJkYwpDb250ZW50LVR5cGU6IHRleHQvaHRtbDsg Y2hhcnNldD0iVVRGLTgiCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFi bGUKCjxodG1sPgo8aGVhZD4KPHN0eWxlIHR5cGU9M0QidGV4dC9jc3MiIHN0eWxlPTNEImRpc3Bs YXk6bm9uZTsiPjwhLS0gUAp7bWFyZ2luLXRvcDowO21hcmdpbi1ib3R0b206MDt9IC0tPjwvc3R5 bGU+CgoKPC9oZWFkPgo8Ym9keT4KPGRpdiBzdHlsZT0zRCJjb2xvcjogYmxhY2s7Ij4KPGRpdiBz dHlsZT0zRCJjb2xvcjogYmxhY2s7Ij4KPHAgc3R5bGU9M0QibWFyZ2luOiAwIDAgMWVtIDA7IGNv bG9yOiBibGFjazsiPllvdSBhdHRhY2ggdGhlIHNzZCBhcyBhIGhvdAp0aWVyIHdpdGggYSBnbHVz dGVyIGNvbW1hbmQuIEkgZG9uJ3QgdGhpbmsgdGhhdCBnZGVwbG95IG9yIG92aXJ0IGd1aSBjYW4g PQpkbwppdC48L3A+CjxwIHN0eWxlPTNEIm1hcmdpbjogMCAwIDFlbSAwOyBjb2xvcjogYmxhY2s7 Ij5UaGUgZ2x1c3RlciBkb2NzIGFuZCByZWRoYXQKZG9jcyBleHBsYWlucyB0aWVyaW5nIHF1aXRl IGdvb2QuPC9wPgo8cCBzdHlsZT0zRCJtYXJnaW46IDAgMCAxZW0gMDsgY29sb3I6IGJsYWNrOyI+ L0pvaGFuPC9wPgo8L2Rpdj4KPGRpdiBzdHlsZT0zRCJjb2xvcjogYmxhY2s7Ij4KPHAKc3R5bGU9 M0QiY29sb3I6IGJsYWNrOyBmb250LXNpemU6IDEwcHQ7IGZvbnQtZmFtaWx5OiBBcmlhbCwgc2Fu cy1zZXJpZjsgbT0KYXJnaW46IDEwcHQgMDsiPk9uCkF1Z3VzdCA4LCAyMDE3IDA3OjA2OjQyIE1v YWNpciBGZXJyZWlyYSAmbHQ7bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb20mZ3Q9CjsKd3JvdGU6 PC9wPgo8YmxvY2txdW90ZSB0eXBlPTNEImNpdGUiIGNsYXNzPTNEImdtYWlsX3F1b3RlIgpzdHls ZT0zRCJtYXJnaW46IDAgMCAwIDAuNzVleDsgYm9yZGVyLWxlZnQ6IDFweCBzb2xpZCAjODA4MDgw OyBwYWRkaW5nLWxlPQpmdDogMC43NWV4OyI+CjxkaXYgaWQ9M0QiZGl2dGFnZGVmYXVsdHdyYXBw ZXIiCnN0eWxlPTNEImZvbnQtc2l6ZToxMnB0O2NvbG9yOiMwMDAwMDA7Zm9udC1mYW1pbHk6Q2Fs aWJyaSxIZWx2ZXRpY2Esc2Fucy09CnNlcmlmOyIKZGlyPTNEImx0ciI+CjxwPkhpIERldmluLDwv cD4KPHA+PGJyPgo8L3A+CjxwPlBsZWFzZSBjb25zaWRlciB0aGF0IGZvciB0aGUgT1MgSSBoYXZl IGEgUkFJRCAxLiBOb3csIGxldHMgc2F5IEkgdXNlClJBSUQgNSB0byBhc3NlbWJsZSBhIHNpbmds ZSBkaXNrIG9uIGVhY2ggc2VydmVyLiBJbiB0aGlzIGNhc2UsIHRoZSBTU0Qgd2k9CmxsCm5vdCBt YWtlIGFueSBkaWZmZXJlbmNlLCByaWdodD8gSSBndWVzcyB0aGF0IHRvIGJlIHBvc3NpYmxlIHRv IHVzZSBpdCwgdGg9CmUKU1NEIHNob3VsZCBub3QgYmUgcGFydCBvZiB0aGUgUkFJRAogNS4gSW4g dGhpcyBjYXNlIEkgY291bGQgY3JlYXRlIGEgbG9naWNhbCB2b2x1bWUgbWFkZSBvZiB0aGUgUkFJ RGVkIGJyaWNrCmFuZCB0aGVuIGV4dGVuZCBpdCB1c2luZyB0aGUgU1NELiBJLmUuOiBVc2luZyBn ZGVwbG95OjwvcD4KPHA+PGJyPgo8L3A+CjxwPltkaXNrdHlwZV08L3A+CjxwPmpib2Q8L3A+Cjxw Pi4uLi48L3A+CjxwPltwdjFdPC9wPgo8cD5hY3Rpb249M0RjcmVhdGU8L3A+CjxwPmRldmljZXM9 M0RzZGIsIHNkYzwvcD4KPHA+d2lwZWZzPTNEeWVzPC9wPgo8cD48L3A+CjxwPmlnbm9yZV92Z19l cnJvcz0zRG5vPGJyPgo8L3A+CjxwPjxicj4KPC9wPgo8cD48L3A+CjxwPlt2ZzFdPC9wPgo8cD48 Yj5hY3Rpb249M0RjcmVhdGU8L2I+PC9wPgo8cD52Z25hbWU9M0RnbHVzdGVyX3ZnX2pib2Q8L3A+ CjxwPnB2bmFtZT0zRHNkYjwvcD4KPHA+aWdub3JlX3ZnX2Vycm9zPTNEbm88YnI+CjwvcD4KPHA+ PGJyPgo8L3A+CjxwPlt2ZzJdPC9wPgo8cD48Yj5hY3Rpb249M0RleHRlbmQ8L2I+PC9wPgo8cD52 Z25hbWU9M0RnbHVzdGVyX3ZnX2pib2Q8L3A+CjxwPnB2bmFtZT0zRHNkYzwvcD4KPHA+aWdub3Jl X3ZnX2Vycm9zPTNEbm88YnI+CjwvcD4KPGJyPgo8cD5CdXQgd2lsbCBHbHVzdGVyIGJlIGFibGUg dG8gYXV0by1kZXRlY3QgYW5kIHVzZSB0aGlzIFNTRCBicmljayBmb3IKdGllcmluZz8gRG8gSSBo YXZlIHRvIGRvIHNvbWUgb3RoZXIgY29uZmlndXJhdGlvbnM/IEFsc28sIGFzIHRoZSBWTSBmaWxl cwooLnFjb3cyKSBhcmUgcXVpdGUgYmlnIHdpbGwgSSBiZW5lZml0IGZyb20gdGllcmluZz8gVGhp cyBpcyB3cm9uZyBhbmQgbXkKYXBwcm9hY2ggc2hvdWxkIGJlIG90aGVyPzwvcD4KPHA+PGJyPgo8 L3A+CjxwPlRoYW5rcyw8L3A+CjxwPk1vYWNpcjxicj4KPC9wPgo8cD48YnI+CjwvcD4KPGJyPgo8 ZGl2IHN0eWxlPTNEImNvbG9yOiByZ2IoNDksIDU1LCA1Nyk7Ij4KPGhyIHRhYmluZGV4PTNEIi0x IiBzdHlsZT0zRCJkaXNwbGF5OmlubGluZS1ibG9jazsgd2lkdGg6OTglIj4KPGRpdiBpZD0zRCJk aXZScGx5RndkTXNnIiBkaXI9M0QibHRyIj48Zm9udCBzdHlsZT0zRCJmb250LXNpemU6MTFwdCIK ZmFjZT0zRCJDYWxpYnJpLCBzYW5zLXNlcmlmIiBjb2xvcj0zRCIjMDAwMDAwIj48Yj5Gcm9tOjwv Yj4gRGV2aW4gQWNvc3RhCiZsdDtkZXZpbkBwYWJzdGF0ZW5jaW8uY29tJmd0Ozxicj4KPGI+U2Vu dDo8L2I+IE1vbmRheSwgQXVndXN0IDcsIDIwMTcgNzo0NiBBTTxicj4KPGI+VG86PC9iPiBNb2Fj aXIgRmVycmVpcmE7IHVzZXJzQG92aXJ0Lm9yZzxicj4KPGI+U3ViamVjdDo8L2I+IFJlOiBbb3Zp cnQtdXNlcnNdIEdvb2QgcHJhY3RpY2VzPC9mb250Pgo8ZGl2PiZuYnNwOzwvZGl2Pgo8L2Rpdj4K PGRpdj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJjb2xvcjpyZ2IoMCww LDApOyBtYXJnaW46MHB4Ij48Zm9udApmYWNlPTNEIklucHV0IE1vbm8iPjxicj4KPC9mb250Pjwv ZGl2Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEImNvbG9yOnJnYigwLDAs MCk7IG1hcmdpbjowcHgiPjxmb250CmZhY2U9M0QiSW5wdXQgTW9ubyI+TW9hY2lyLDwvZm9udD48 L2Rpdj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJjb2xvcjpyZ2IoMCww LDApOyBtYXJnaW46MHB4Ij48Zm9udApmYWNlPTNEIklucHV0IE1vbm8iPjxicj4KPC9mb250Pjwv ZGl2Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEImNvbG9yOnJnYigwLDAs MCk7IG1hcmdpbjowcHgiPjxmb250CmZhY2U9M0QiSW5wdXQgTW9ubyI+SSBoYXZlIHJlY2VudGx5 IGluc3RhbGxlZCBtdWx0aXBsZSBSZWQgSGF0IFZpcnR1YWxpemE9CnRpb24KaG9zdHMgZm9yIHNl dmVyYWwgZGlmZmVyZW50IGNvbXBhbmllcywgYW5kIGhhdmUgZGVhbHQgd2l0aCB0aGUgUmVkIEhh dApTdXBwb3J0IFRlYW0gaW4gZGVwdGggYWJvdXQgb3B0aW1hbCBjb25maWd1cmF0aW9uCiBpbiBy ZWdhcmRzIHRvIHNldHRpbmcgdXAgR2x1c3RlckZTIG1vc3QgZWZmaWNpZW50bHkgYW5kIEkgd2Fu dGVkIHRvIHNoYXI9CmUKd2l0aCB5b3Ugd2hhdCBJIGxlYXJuZWQuPC9mb250PjwvZGl2Pgo8ZGl2 IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEImNvbG9yOnJnYigwLDAsMCk7IG1hcmdp bjowcHgiPjxmb250CmZhY2U9M0QiSW5wdXQgTW9ubyI+PGJyPgo8L2ZvbnQ+PC9kaXY+CjxkaXYg aWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9M0QiY29sb3I6cmdiKDAsMCwwKTsgbWFyZ2lu OjBweCI+PGZvbnQKZmFjZT0zRCJJbnB1dCBNb25vIj5JbiBnZW5lcmFsIFJlZCBIYXQgVmlydHVh bGl6YXRpb24gdGVhbSBmcm93bnMgdXBvbiB1cz0KaW5nCmVhY2ggRElTSyBvZiB0aGUgc3lzdGVt IGFzIGp1c3QgYSBKQk9ELCBzdXJlIHRoZXJlIGlzIHNvbWUgcHJvdGVjdGlvbiBieQpoYXZpbmcg dGhlIGRhdGEgcmVwbGljYXRlZCwgaG93ZXZlciwgdGhlCiByZWNvbW1lbmRhdGlvbiBpcyB0byB1 c2UgUkFJRCA2IChwcmVmZXJyZWQpIG9yIFJBSUQtNSwgb3IgYXQgbGVhc3QgUkFJRC09CjEKYXQg dGhlIHZlcnkgbGVhc3QuPC9mb250PjwvZGl2Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQi IHN0eWxlPTNEImNvbG9yOnJnYigwLDAsMCk7IG1hcmdpbjowcHgiPjxmb250CmZhY2U9M0QiSW5w dXQgTW9ubyI+PGJyPgo8L2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIg c3R5bGU9M0QibWFyZ2luOjBweCI+PGZvbnQgZmFjZT0zRCJJbnB1dCBNb249Cm8iPkhlcmUKaXMg dGhlJm5ic3A7ZGlyZWN0IHF1b3RlIGZyb20gUmVkIEhhdCB3aGVuIEkgYXNrZWQgYWJvdXQgUkFJ RCBhbmQKQnJpY2tzOjwvZm9udD48L2Rpdj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21mb250IiBz dHlsZT0zRCJtYXJnaW46MHB4Ij48Zm9udCBmYWNlPTNEIklucHV0IE1vbj0KbyI+PGk+PGJyPgo8 L2k+PC9mb250PjwvZGl2Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEIm1h cmdpbjowcHgiPjxmb250CmZhY2U9M0QiSW5wdXQgTW9ubyI+PGk+JnF1b3Q7QSB0eXBpY2FsIEds dXN0ZXIgY29uZmlndXJhdGlvbiB3b3VsZCB1c2UgUkE9CklECnVuZGVybmVhdGggdGhlIGJyaWNr cy4gUkFJRCA2IGlzIG1vc3QgdHlwaWNhbCBhcyBpdCBnaXZlcyB5b3UgMiBkaXNrCmZhaWx1cmUg cHJvdGVjdGlvbiwgYnV0IFJBSUQgNSBjb3VsZCBiZSB1c2VkIHRvby4mbmJzcDtPbmNlIHlvdSBo YXZlIHRoZQogUkFJRGVkIGJyaWNrcywgeW91J2QgdGhlbiBhcHBseSB0aGUgZGVzaXJlZCByZXBs aWNhdGlvbiBvbiB0b3Agb2YgdGhhdC4KVGhlIG1vc3QgcG9wdWxhciB3YXkgb2YgZG9pbmcgdGhp cyB3b3VsZCBiZSBkaXN0cmlidXRlZCByZXBsaWNhdGVkIHdpdGggMj0KeApyZXBsaWNhdGlvbi4m bmJzcDtJbiBnZW5lcmFsIHlvdSdsbCBnZXQgYmV0dGVyIHBlcmZvcm1hbmNlIHdpdGggbGFyZ2Vy CmJyaWNrcy4gMTIgZHJpdmVzIGlzIG9mdGVuIGEgc3dlZXQgc3BvdC4gQW5vdGhlcgogb3B0aW9u IHdvdWxkIGJlIHRvIGNyZWF0ZSBhIHNlcGFyYXRlIHRpZXIgdXNpbmcgYWxsClNTRD1FMj04MD05 OXMuPUUyPTgwPTlEJm5ic3A7PC9pPjwvZm9udD48L2Rpdj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0 b21mb250IiBzdHlsZT0zRCJtYXJnaW46MHB4Ij48YnI+CjwvZGl2Pgo8ZGl2IGlkPTNEImJsb29w X2N1c3RvbWZvbnQiIHN0eWxlPTNEIm1hcmdpbjowcHgiPjxmb250IGZhY2U9M0QiSW5wdXQgTW9u PQpvIj48aT5JbgpvcmRlciB0byBTU0QmbmJzcDt0aWVyaW5nIGZyb20gbXkgdW5kZXJzdGFuZGlu ZyB5b3Ugd291bGQgbmVlZCAxIHggTlZNZQpkcml2ZSBpbiBlYWNoIHNlcnZlciwgb3IgNCB4IFNT RCBob3QgdGllciAoaXQgbmVlZHMgdG8gYmUgZGlzdHJpYnV0ZWQsCnJlcGxpY2F0ZWQgZm9yIHRo ZSBob3QgdGllciBpZiBub3QgdXNpbmcgTlZNRSkuCiBTbyB3aXRoIHlvdSBvbmx5IGhhdmluZyAx IFNTRCBkcml2ZSBpbiBlYWNoIHNlcnZlciwmbmJzcDtJPUUyPTgwPTk5ZCBzdWc9Cmdlc3QgbWF5 YmUKbG9va2luZyBpbnRvIHRoZSBOVk1FIG9wdGlvbi4mbmJzcDs8L2k+PC9mb250PjwvZGl2Pgo8 ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEIm1hcmdpbjowcHgiPjxmb250IGZh Y2U9M0QiSW5wdXQgTW9uPQpvIj48aT48YnI+CjwvaT48L2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9M0Qi Ymxvb3BfY3VzdG9tZm9udCIgc3R5bGU9M0QibWFyZ2luOjBweCI+PGZvbnQKZmFjZT0zRCJJbnB1 dCBNb25vIj48aT5TaW5jZSB5b3VyIHVzaW5nIG9ubHkgMy1zZXJ2ZXJzLCB3aGF0IEk9RTI9ODA9 OTlkID0KcHJvYmFibHkKc3VnZ2VzdCBpcyB0byBkbyAoMiBSZXBsaWNhcyAmIzQzOyBBcmJpdGVy IE5vZGUpLCB0aGlzIHNldHVwIGFjdHVhbGx5CmRvZXNuPUUyPTgwPTk5dCByZXF1aXJlIHRoZSAz cmQgc2VydmVyIHRvIGhhdmUgYmlnIGRyaXZlcyBhdCBhbGwgYXMgaXQgb249Cmx5CiBzdG9yZXMg bWV0YS1kYXRhIGFib3V0IHRoZSBmaWxlcyBhbmQgbm90IGFjdHVhbGx5IGEgZnVsbApjb3B5LiZu YnNwOzwvaT48L2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9 M0QibWFyZ2luOjBweCI+PGZvbnQgZmFjZT0zRCJJbnB1dCBNb249Cm8iPjxpPjxicj4KPC9pPjwv Zm9udD48L2Rpdj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJtYXJnaW46 MHB4Ij48Zm9udApmYWNlPTNEIklucHV0IE1vbm8iPjxpPlBsZWFzZSBzZWUgdGhlIGF0dGFjaGVk IGRvY3VtZW50IHRoYXQgd2FzIGdpdmVuIHRvPQogbWUKYnkgUmVkIEhhdCB0byBnZXQgbW9yZSBp bmZvcm1hdGlvbiBvbiB0aGlzLiBIb3BlIHRoaXMgaW5mb3JtYXRpb24gaGVscHMKeW91LjwvaT48 L2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9M0QibWFyZ2lu OjBweCI+PGZvbnQgZmFjZT0zRCJJbnB1dCBNb249Cm8iPjxpPjxicj4KPC9pPjwvZm9udD48L2Rp dj4KPGJyPgo8ZGl2IGlkPTNEImJsb29wX3NpZ25fMTUwMjA4NzM3NjcyNTQ2OTE4NCIgY2xhc3M9 M0QiYmxvb3Bfc2lnbiI+PHNwYW4Kc3R5bGU9M0QiZm9udC1mYW1pbHk6J2hlbHZldGljYSBOZXVl JyxoZWx2ZXRpY2E7IGZvbnQtc2l6ZToxNHB4Ij4tLTwvc3Bhbj0KPjxicgpzdHlsZT0zRCJmb250 LWZhbWlseTonaGVsdmV0aWNhIE5ldWUnLGhlbHZldGljYTsgZm9udC1zaXplOjE0cHgiPgo8ZGl2 IGNsYXNzPTNEImdtYWlsX3NpZ25hdHVyZSIKc3R5bGU9M0QiZm9udC1mYW1pbHk6J2hlbHZldGlj YSBOZXVlJyxoZWx2ZXRpY2E7IGZvbnQtc2l6ZToxNHB4Ij4KPGRpdiBkaXI9M0QibHRyIj4KPGRp dj48YnI+CjwvZGl2Pgo8ZGl2PkRldmluIEFjb3N0YSwgUkhDQSwgUkhWQ0E8L2Rpdj4KPGRpdj5S ZWQgSGF0IENlcnRpZmllZCBBcmNoaXRlY3Q8L2Rpdj4KPGRpdj48L2Rpdj4KPC9kaXY+CjwvZGl2 Pgo8L2Rpdj4KPGJyPgo8cCBjbGFzcz0zRCJhaXJtYWlsX29uIj5PbiBBdWd1c3QgNiwgMjAxNyBh dCA3OjI5OjI5IFBNLCBNb2FjaXIgRmVycmVpcmEgPQooPGEKaHJlZj0zRCJtYWlsdG86bW9hY2ly ZmVycmVpcmFAaG90bWFpbC5jb20iPm1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY29tPC9hPj0KKQp3 cm90ZTo8L3A+CjxibG9ja3F1b3RlIHR5cGU9M0QiY2l0ZSIgY2xhc3M9M0QiY2xlYW5fYnEiPjxz cGFuPgo8ZGl2IGRpcj0zRCJsdHIiPgo8ZGl2PjwvZGl2Pgo8ZGl2Pgo8ZGl2IGlkPTNEImRpdnRh Z2RlZmF1bHR3cmFwcGVyIiBkaXI9M0QibHRyIgpzdHlsZT0zRCJmb250LXNpemU6MTJwdDsgY29s b3I6IzAwMDAwMDsgZm9udC1mYW1pbHk6Q2FsaWJyaSxIZWx2ZXRpY2Esc2FuPQpzLXNlcmlmIj4K PHA+PHNwYW4+SSBhbSB3aWxsaW5nIHRvIGFzc2VtYmxlIGEgb1ZpcnQgJnF1b3Q7cG9kJnF1b3Q7 LCBtYWRlIG9mIDMKc2VydmVycywgZWFjaCB3aXRoIDIgQ1BVIHNvY2tldHMgb2YgMTIgY29yZXMs IDI1NkdCIFJBTSwgNyBIREQgMTBLLCAxIFNTRD0KLgpUaGUgaWRlYSBpcyB0byB1c2UgR2x1c3Rl ckZTIHRvIHByb3ZpZGUgSEEgZm9yIHRoZSBWTXMuIFRoZSAzIHNlcnZlcnMgaGF2PQplCmEgZHVh bCA0MEdiIE5JQyBhbmQgYSBkdWFsIDEwR2IgTklDLiBTbyBteSBpbnRlbnRpb24KIGlzIHRvIGNy ZWF0ZSBhIGxvb3AgbGlrZSBhIHNlcnZlciB0cmlhbmdsZSB1c2luZyB0aGUgNDBHYiBOSUNzIGZv cgp2aXJ0dWFsaXphdGlvbiBmaWxlcyAoVk1zIC5xY293MikgYWNjZXNzIGFuZCB0byBtb3ZlIFZN cyBhcm91bmQgdGhlIHBvZAooZWFzdCAvd2VzdCB0cmFmZmljKSB3aGlsZSB1c2luZyB0aGUgMTBH YiBpbnRlcmZhY2VzIGZvciBnaXZpbmcgc2VydmljZXMgPQp0bwp0aGUgb3V0c2lkZSB3b3JsZCAo bm9ydGgvc291dGggdHJhZmZpYykuPC9zcGFuPjwvcD4KPHA+PGJyPgo8L3A+CjxwPlRoaXMgc2Fp ZCwgbXkgZmlyc3QgcXVlc3Rpb24gaXM6IEhvdyBzaG91bGQgSSBkZXBsb3kgR2x1c3RlckZTIGlu IHN1Y2gKb1ZpcnQgc2NlbmFyaW8/IE15IHF1ZXN0aW9ucyBhcmU6PC9wPgo8cD48YnI+CjwvcD4K PHA+MSAtIFNob3VsZCBJIGNyZWF0ZSAzIFJBSUQgKGkuZS46IFJBSUQgNSksIG9uZSBvbiBlYWNo IG9WaXJ0IG5vZGUsIGFuZAp0aGVuIGNyZWF0ZSBhIEdsdXN0ZXJGUyB1c2luZyB0aGVtPzwvcD4K PHA+MiAtIEluc3RlYWQsIHNob3VsZCBJIGNyZWF0ZSBhIEpCT0QgYXJyYXkgbWFkZSBvZiBhbGwg c2VydmVyJ3MgZGlza3M/PD0KL3A+CjxwPjMgLSBXaGF0IGlzIHRoZSBiZXN0IEdsdXN0ZXIgY29u ZmlndXJhdGlvbiB0byBwcm92aWRlIGZvciBIQSB3aGlsZSBub3QKY29uc3VtaW5nIHRvbyBtdWNo IGRpc2sgc3BhY2U/PGJyPgo8L3A+CjxwPjQgLSBEb2VzIGEgb1ZpcnQgaHlwZXJ2aXNvciBwb2Qg bGlrZSBJIGFtIHBsYW5uaW5nIHRvIGJ1aWxkLCBhbmQgdGhlCnZpcnR1YWxpemF0aW9uIGVudmly b25tZW50LCBiZW5lZml0cyBmcm9tIHRpZXJpbmcgd2hlbiB1c2luZyBhIFNTRCBkaXNrPwpBbmQg eWVzLCB3aWxsIEdsdXN0ZXIgZG8gaXQgYnkgZGVmYXVsdCBvciBJIGhhdmUgdG8gY29uZmlndXJl IGl0IHRvIGRvIHNvPQo/PC9wPgo8cD48YnI+CjwvcD4KPHA+QXQgdGhlIGJvdHRvbSBsaW5lLCB3 aGF0IGlzIHRoZSBnb29kIHByYWN0aWNlIGZvciB1c2luZyBHbHVzdGVyRlMgaW4Kc21hbGwgcG9k cyBmb3IgZW50ZXJwcmlzZXM/PGJyPgo8L3A+CjxwPjxicj4KPC9wPgo8cD5Zb3Ugb3Bpbmlvbi9m ZWVkYmFjayB3aWxsIGJlIHJlYWxseSBhcHByZWNpYXRlZCE8L3A+CjxwPk1vYWNpcjxicj4KPC9w Pgo8L2Rpdj4KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18g PGJyPgpVc2VycyBtYWlsaW5nIGxpc3QgPGJyPgo8YSBocmVmPTNEIm1haWx0bzpVc2Vyc0Bvdmly dC5vcmciPlVzZXJzQG92aXJ0Lm9yZzwvYT4gPGJyPgo8YQpocmVmPTNEImh0dHA6Ly9saXN0cy5v dmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyI+aHR0cDovL2xpc3RzLm92aXJ0PQoub3Jn L21haWxtYW4vbGlzdGluZm8vdXNlcnM8L2E+Cjxicj4KPC9kaXY+CjwvZGl2Pgo8L3NwYW4+PC9i bG9ja3F1b3RlPgo8L2Rpdj4KPC9kaXY+CjwvZGl2PgoKX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX188YnI+ClVzZXJzIG1haWxpbmcgbGlzdDxicj4KPGEgY2xh c3M9M0QiYXFtLWF1dG9saW5rIGFxbS1hdXRvd3JhcCIKaHJlZj0zRCJtYWlsdG86VXNlcnMlNDBv dmlydC5vcmciPlVzZXJzQG92aXJ0Lm9yZzwvYT48YnI+CjxhIGNsYXNzPTNEImFxbS1hdXRvbGlu ayBhcW0tYXV0b3dyYXAiCmhyZWY9M0QiaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xp c3RpbmZvL3VzZXJzIj5odHRwOi8vbGlzdHMub3ZpcnQ9Ci5vcmcvbWFpbG1hbi9saXN0aW5mby91 c2VyczwvYT48YnI+Cjxicj48L2Jsb2NrcXVvdGU+CjwvZGl2Pgo8L2Rpdj4KPC9ib2R5Pgo8L2h0 bWw+CgotLS0tLS0tLS0tLS0xNWRjMDcxNDE3NjNjYmMyNzY4OGZlYzJkYy0tCgo= --===============0182984489577361123==-- From fabrice.bacchella at orange.fr Tue Aug 8 06:16:39 2017 Content-Type: multipart/mixed; boundary="===============6824931101398144345==" MIME-Version: 1.0 From: Fabrice Bacchella To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Tue, 08 Aug 2017 08:16:32 +0200 Message-ID: In-Reply-To: CAD7b65Kw9QKoa09mVf+ThZL-2phLPPoQ5ABCpO5nhxn=gqTS7A@mail.gmail.com --===============6824931101398144345== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable > Le 8 ao=C3=BBt 2017 =C3=A0 04:08, FERNANDO FREDIANI a =C3=A9crit : > Even if you have a Hardware RAID Controller with Writeback cache you will= have a significant performance penalty and may not fully use all the resou= rces you mentioned you have. > = Nope again,from my experience with HP Smart Array and write back cache, wri= te, that goes in the cache, are even faster that read that must goes to the= disks. of course if the write are too fast and to big, they will over over= flow the cache. But on todays controller they are multi-gigabyte cache, you= must write a lot to fill them. And if you can afford 40Gb card, you can af= ford decent controller. --===============6824931101398144345==-- From ykaul at redhat.com Tue Aug 8 06:30:30 2017 Content-Type: multipart/mixed; boundary="===============3876214987069104049==" MIME-Version: 1.0 From: Yaniv Kaul To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Tue, 08 Aug 2017 09:29:47 +0300 Message-ID: In-Reply-To: 47b0f3b5-a836-d5c2-7cf4-c0147aa3948f@upx.com --===============3876214987069104049== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Tue, Aug 8, 2017 at 12:03 AM, FERNANDO FREDIANI < fernando.frediani(a)upx.com> wrote: > Thanks for the detailed answer Erekle. > > I conclude that it is worth in any scenario to have a arbiter node in > order to avoid wasting more disk space to RAID X + Gluster Replication on > the top of it. The cost seems much lower if you consider running costs of > the whole storage and compare it with the cost to build the arbiter node. > Even having a fully redundant arbiter service with 2 nodes would make it > wort on a larger deployment. > Note that although you get the same consistency as a replica 3 setup, a 2+arbiter gives you data availability as a replica 2 setup. May or may not be OK with your high availability requirements. Y. > Regards > Fernando > On 07/08/2017 17:07, Erekle Magradze wrote: > > Hi Fernando (sorry for misspelling your name, I used a different keyboard= ), > > So let's go with the following scenarios: > > 1. Let's say you have two servers (replication factor is 2), i.e. two > bricks per volume, in this case it is strongly recommended to have the > arbiter node, the metadata storage that will guarantee avoiding the split > brain situation, in this case for arbiter you don't even need a disk with > lots of space, it's enough to have a tiny ssd but hosted on a separate > server. Advantage of such setup is that you don't need the RAID 1 for each > brick, you have the metadata information stored in arbiter node and brick > replacement is easy. > > 2. If you have odd number of bricks (let's say 3, i.e. replication factor > is 3) in your volume and you didn't create the arbiter node as well as you > didn't configure the quorum, in this case the entire load for keeping the > consistency of the volume resides on all 3 servers, each of them is > important and each brick contains key information, they need to cross-che= ck > each other (that's what people usually do with the first try of gluster :) > ), in this case replacing a brick is a big pain and in this case RAID 1 is > a good option to have (that's the disadvantage, i.e. loosing the space and > not having the JBOD option) advantage is that you don't have the to have > additional arbiter node. > > 3. You have odd number of bricks and configured arbiter node, in this case > you can easily go with JBOD, however a good practice would be to have a > RAID 1 for arbiter disks (tiny 128GB SSD-s ar perfectly sufficient for > volumes with 10s of TB-s in size.) > > That's basically it > > The rest about the reliability and setup scenarios you can find in gluster > documentation, especially look for quorum and arbiter node configs+option= s. > > Cheers > > Erekle > P.S. What I was mentioning, regarding a good practice is mostly related to > the operations of gluster not installation or deployment, i.e. not the > conceptual understanding of gluster (conceptually it's a JBOD system). > > On 08/07/2017 05:41 PM, FERNANDO FREDIANI wrote: > > Thanks for the clarification Erekle. > > However I get surprised with this way of operating from GlusterFS as it > adds another layer of complexity to the system (either a hardware or > software RAID) before the gluster config and increase the system's overall > costs. > > An important point to consider is: In RAID configuration you already have > space 'wasted' in order to build redundancy (either RAID 1, 5, or 6). Then > when you have GlusterFS on the top of several RAIDs you have again more > data replicated so you end up with the same data consuming more space in a > group of disks and again on the top of several RAIDs depending on the > Gluster configuration you have (in a RAID 1 config the same data is > replicated 4 times). > > Yet another downside of having a RAID (specially RAID 5 or 6) is that it > reduces considerably the write speeds as each group of disks will end up > having the write speed of a single disk as all other disks of that group > have to wait for each other to write as well. > > Therefore if Gluster already replicates data why does it create this big > pain you mentioned if the data is replicated somewhere else, can still be > retrieved to both serve clients and reconstruct the equivalent disk when = it > is replaced ? > > Fernando > > On 07/08/2017 10:26, Erekle Magradze wrote: > > Hi Frenando, > > Here is my experience, if you consider a particular hard drive as a brick > for gluster volume and it dies, i.e. it becomes not accessible it's a huge > hassle to discard that brick and exchange with another one, since gluster > some tries to access that broken brick and it's causing (at least it cause > for me) a big pain, therefore it's better to have a RAID as brick, i.e. > have RAID 1 (mirroring) for each brick, in this case if the disk is down > you can easily exchange it and rebuild the RAID without going offline, i.e > switching off the volume doing brick manipulations and switching it back = on. > > Cheers > > Erekle > > On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote: > > For any RAID 5 or 6 configuration I normally follow a simple gold rule > which gave good results so far: > - up to 4 disks RAID 5 > - 5 or more disks RAID 6 > > However I didn't really understand well the recommendation to use any RAID > with GlusterFS. I always thought that GlusteFS likes to work in JBOD mode > and control the disks (bricks) directlly so you can create whatever > distribution rule you wish, and if a single disk fails you just replace it > and which obviously have the data replicated from another. The only > downside of using in this way is that the replication data will be flow > accross all servers but that is not much a big issue. > > Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS. > > Thanks > Regards > Fernando > > On 07/08/2017 03:46, Devin Acosta wrote: > > > Moacir, > > I have recently installed multiple Red Hat Virtualization hosts for > several different companies, and have dealt with the Red Hat Support Team > in depth about optimal configuration in regards to setting up GlusterFS > most efficiently and I wanted to share with you what I learned. > > In general Red Hat Virtualization team frowns upon using each DISK of the > system as just a JBOD, sure there is some protection by having the data > replicated, however, the recommendation is to use RAID 6 (preferred) or > RAID-5, or at least RAID-1 at the very least. > > Here is the direct quote from Red Hat when I asked about RAID and Bricks: > > *"A typical Gluster configuration would use RAID underneath the bricks. > RAID 6 is most typical as it gives you 2 disk failure protection, but RAID > 5 could be used too. Once you have the RAIDed bricks, you'd then apply the > desired replication on top of that. The most popular way of doing this > would be distributed replicated with 2x replication. In general you'll get > better performance with larger bricks. 12 drives is often a sweet spot. > Another option would be to create a separate tier using all SSD=E2=80=99s= .=E2=80=9D * > > *In order to SSD tiering from my understanding you would need 1 x NVMe > drive in each server, or 4 x SSD hot tier (it needs to be distributed, > replicated for the hot tier if not using NVME). So with you only having 1 > SSD drive in each server, I=E2=80=99d suggest maybe looking into the NVME= option. * > > *Since your using only 3-servers, what I=E2=80=99d probably suggest is to= do (2 > Replicas + Arbiter Node), this setup actually doesn=E2=80=99t require the= 3rd > server to have big drives at all as it only stores meta-data about the > files and not actually a full copy. * > > *Please see the attached document that was given to me by Red Hat to get > more information on this. Hope this information helps you.* > > > -- > > Devin Acosta, RHCA, RHVCA > Red Hat Certified Architect > > On August 6, 2017 at 7:29:29 PM, Moacir Ferreira ( > moacirferreira(a)hotmail.com) wrote: > > I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU > sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use > GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC a= nd > a dual 10Gb NIC. So my intention is to create a loop like a server triang= le > using the 40Gb NICs for virtualization files (VMs .qcow2) access and to > move VMs around the pod (east /west traffic) while using the 10Gb > interfaces for giving services to the outside world (north/south traffic). > > > This said, my first question is: How should I deploy GlusterFS in such > oVirt scenario? My questions are: > > > 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and > then create a GlusterFS using them? > > 2 - Instead, should I create a JBOD array made of all server's disks? > > 3 - What is the best Gluster configuration to provide for HA while not > consuming too much disk space? > > 4 - Does a oVirt hypervisor pod like I am planning to build, and the > virtualization environment, benefits from tiering when using a SSD disk? > And yes, will Gluster do it by default or I have to configure it to do so? > > > At the bottom line, what is the good practice for using GlusterFS in small > pods for enterprises? > > > You opinion/feedback will be really appreciated! > > Moacir > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > > > _______________________________________________ > Users mailing listUsers(a)ovirt.orghttp://lists.ovirt.org/mailman/listinf= o/users > > > > > _______________________________________________ > Users mailing listUsers(a)ovirt.orghttp://lists.ovirt.org/mailman/listinf= o/users > > > > > -- > Recogizer Group GmbH > > Dr.rer.nat. Erekle Magradze > Lead Big Data Engineering & DevOps > Rheinwerkallee 2, 53227 Bonn > Tel: +49 228 29974555 <+49%20228%2029974555> > > E-Mail erekle.magradze(a)recogizer.de > Web: www.recogizer.com > > Recogizer auf LinkedIn https://www.linkedin.com/company-beta/10039182/ > Folgen Sie uns auf Twitter https://twitter.com/recogizer > > ----------------------------------------------------------------- > Recogizer Group GmbH > Gesch=C3=A4ftsf=C3=BChrer: Oliver Habisch, Carsten Kreutze > Handelsregister: Amtsgericht Bonn HRB 20724 > Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993 > > Diese E-Mail enth=C3=A4lt vertrauliche und/oder rechtlich gesch=C3=BCtzte= Informationen. > Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=C3=BCmli= ch erhalten haben, > informieren Sie bitte sofort den Absender und l=C3=B6schen Sie diese Mail. > Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und de= r darin enthaltenen Informationen ist nicht gestattet. > > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > --===============3876214987069104049== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+PGRpdiBjbGFz cz0iZ21haWxfcXVvdGUiPk9uIFR1ZSwgQXVnIDgsIDIwMTcgYXQgMTI6MDMgQU0sIEZFUk5BTkRP IEZSRURJQU5JIDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgaHJlZj0ibWFpbHRvOmZlcm5hbmRvLmZy ZWRpYW5pQHVweC5jb20iIHRhcmdldD0iX2JsYW5rIj5mZXJuYW5kby5mcmVkaWFuaUB1cHguY29t PC9hPiZndDs8L3NwYW4+IHdyb3RlOjxicj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUi IHN0eWxlPSJtYXJnaW46MCAwIDAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRk aW5nLWxlZnQ6MWV4Ij4KICAKICAgIAogIAogIDxkaXYgYmdjb2xvcj0iI0ZGRkZGRiIgdGV4dD0i IzAwMDAwMCI+CiAgICA8cD5UaGFua3MgZm9yIHRoZSBkZXRhaWxlZCBhbnN3ZXIgRXJla2xlLjwv cD4KICAgIDxwPkkgY29uY2x1ZGUgdGhhdCBpdCBpcyB3b3J0aCBpbiBhbnkgc2NlbmFyaW8gdG8g aGF2ZSBhIGFyYml0ZXIKICAgICAgbm9kZSBpbiBvcmRlciB0byBhdm9pZCB3YXN0aW5nIG1vcmUg ZGlzayBzcGFjZSB0byBSQUlEIFggKyBHbHVzdGVyCiAgICAgIFJlcGxpY2F0aW9uIG9uIHRoZSB0 b3Agb2YgaXQuIFRoZSBjb3N0IHNlZW1zIG11Y2ggbG93ZXIgaWYgeW91CiAgICAgIGNvbnNpZGVy IHJ1bm5pbmcgY29zdHMgb2YgdGhlIHdob2xlIHN0b3JhZ2UgYW5kIGNvbXBhcmUgaXQgd2l0aAog ICAgICB0aGUgY29zdCB0byBidWlsZCB0aGUgYXJiaXRlciBub2RlLiBFdmVuIGhhdmluZyBhIGZ1 bGx5IHJlZHVuZGFudAogICAgICBhcmJpdGVyIHNlcnZpY2Ugd2l0aCAyIG5vZGVzIHdvdWxkIG1h a2UgaXQgd29ydCBvbiBhIGxhcmdlcgogICAgICBkZXBsb3ltZW50LjwvcD48L2Rpdj48L2Jsb2Nr cXVvdGU+PGRpdj48YnI+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5Ob3RlIHRoYXQgYWx0aG91 Z2ggeW91IGdldCB0aGUgc2FtZSBjb25zaXN0ZW5jeSBhcyBhIHJlcGxpY2EgMyBzZXR1cCwgYSAy K2FyYml0ZXIgZ2l2ZXMgeW91IGRhdGEgYXZhaWxhYmlsaXR5IGFzIGEgcmVwbGljYSAyIHNldHVw LiBNYXkgb3IgbWF5IG5vdCBiZSBPSyB3aXRoIHlvdXIgaGlnaCBhdmFpbGFiaWxpdHkgcmVxdWly ZW1lbnRzLjwvZGl2PjxkaXY+WS48L2Rpdj48ZGl2PsKgPC9kaXY+PGJsb2NrcXVvdGUgY2xhc3M9 ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAwIC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNj Y2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+PGRpdiBiZ2NvbG9yPSIjRkZGRkZGIiB0ZXh0PSIj MDAwMDAwIj4KICAgIDxwPlJlZ2FyZHM8c3BhbiBjbGFzcz0iSE9FblpiIj48Zm9udCBjb2xvcj0i Izg4ODg4OCI+PGJyPgogICAgICBGZXJuYW5kbzwvZm9udD48L3NwYW4+PC9wPjxkaXY+PGRpdiBj bGFzcz0iaDUiPgogICAgPGRpdiBjbGFzcz0ibV81MjM0OTIwMTM0NzM2ODIyNzJtb3otY2l0ZS1w cmVmaXgiPk9uIDA3LzA4LzIwMTcgMTc6MDcsIEVyZWtsZSBNYWdyYWR6ZQogICAgICB3cm90ZTo8 YnI+CiAgICA8L2Rpdj4KICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiPgogICAgICAKICAgICAg PHA+SGkgRmVybmFuZG8gKHNvcnJ5IGZvciBtaXNzcGVsbGluZyB5b3VyIG5hbWUsIEkgdXNlZCBh CiAgICAgICAgZGlmZmVyZW50IGtleWJvYXJkKSw8L3A+CiAgICAgIDxwPlNvIGxldCYjMzk7cyBn byB3aXRoIHRoZSBmb2xsb3dpbmcgc2NlbmFyaW9zOjwvcD4KICAgICAgPHA+MS4gTGV0JiMzOTtz IHNheSB5b3UgaGF2ZSB0d28gc2VydmVycyAocmVwbGljYXRpb24gZmFjdG9yIGlzIDIpLAogICAg ICAgIGkuZS4gdHdvIGJyaWNrcyBwZXIgdm9sdW1lLCBpbiB0aGlzIGNhc2UgaXQgaXMgc3Ryb25n bHkKICAgICAgICByZWNvbW1lbmRlZCB0byBoYXZlIHRoZSBhcmJpdGVyIG5vZGUsIHRoZSBtZXRh ZGF0YSBzdG9yYWdlIHRoYXQKICAgICAgICB3aWxsIGd1YXJhbnRlZSBhdm9pZGluZyB0aGUgc3Bs aXQgYnJhaW4gc2l0dWF0aW9uLCBpbiB0aGlzIGNhc2UKICAgICAgICBmb3IgYXJiaXRlciB5b3Ug ZG9uJiMzOTt0IGV2ZW4gbmVlZCBhIGRpc2sgd2l0aCBsb3RzIG9mIHNwYWNlLCBpdCYjMzk7cwog ICAgICAgIGVub3VnaCB0byBoYXZlIGEgdGlueSBzc2QgYnV0IGhvc3RlZCBvbiBhIHNlcGFyYXRl IHNlcnZlci4KICAgICAgICBBZHZhbnRhZ2Ugb2Ygc3VjaCBzZXR1cCBpcyB0aGF0IHlvdSBkb24m IzM5O3QgbmVlZCB0aGUgUkFJRCAxIGZvcgogICAgICAgIGVhY2ggYnJpY2ssIHlvdSBoYXZlIHRo ZSBtZXRhZGF0YSBpbmZvcm1hdGlvbiBzdG9yZWQgaW4gYXJiaXRlcgogICAgICAgIG5vZGUgYW5k IGJyaWNrIHJlcGxhY2VtZW50IGlzIGVhc3kuPC9wPgogICAgICA8cD4yLiBJZiB5b3UgaGF2ZSBv ZGQgbnVtYmVyIG9mIGJyaWNrcyAobGV0JiMzOTtzIHNheSAzLCBpLmUuCiAgICAgICAgcmVwbGlj YXRpb24gZmFjdG9yIGlzIDMpIGluIHlvdXIgdm9sdW1lIGFuZCB5b3UgZGlkbiYjMzk7dCBjcmVh dGUKICAgICAgICB0aGUgYXJiaXRlciBub2RlIGFzIHdlbGwgYXMgeW91IGRpZG4mIzM5O3QgY29u ZmlndXJlIHRoZSBxdW9ydW0sIGluCiAgICAgICAgdGhpcyBjYXNlIHRoZSBlbnRpcmUgbG9hZCBm b3Iga2VlcGluZyB0aGUgY29uc2lzdGVuY3kgb2YgdGhlCiAgICAgICAgdm9sdW1lIHJlc2lkZXMg b24gYWxsIDMgc2VydmVycywgZWFjaCBvZiB0aGVtIGlzIGltcG9ydGFudCBhbmQKICAgICAgICBl YWNoIGJyaWNrIGNvbnRhaW5zIGtleSBpbmZvcm1hdGlvbiwgdGhleSBuZWVkIHRvIGNyb3NzLWNo ZWNrCiAgICAgICAgZWFjaCBvdGhlciAodGhhdCYjMzk7cyB3aGF0IHBlb3BsZSB1c3VhbGx5IGRv IHdpdGggdGhlIGZpcnN0IHRyeSBvZgogICAgICAgIGdsdXN0ZXIgOikgKSwgaW4gdGhpcyBjYXNl IHJlcGxhY2luZyBhIGJyaWNrIGlzIGEgYmlnIHBhaW4gYW5kCiAgICAgICAgaW4gdGhpcyBjYXNl IFJBSUQgMSBpcyBhIGdvb2Qgb3B0aW9uIHRvIGhhdmUgKHRoYXQmIzM5O3MgdGhlCiAgICAgICAg ZGlzYWR2YW50YWdlLCBpLmUuIGxvb3NpbmcgdGhlIHNwYWNlIGFuZCBub3QgaGF2aW5nIHRoZSBK Qk9ECiAgICAgICAgb3B0aW9uKSBhZHZhbnRhZ2UgaXMgdGhhdCB5b3UgZG9uJiMzOTt0IGhhdmUg dGhlIHRvIGhhdmUgYWRkaXRpb25hbAogICAgICAgIGFyYml0ZXIgbm9kZS48L3A+CiAgICAgIDxw PjMuIFlvdSBoYXZlIG9kZCBudW1iZXIgb2YgYnJpY2tzIGFuZCBjb25maWd1cmVkIGFyYml0ZXIg bm9kZSwKICAgICAgICBpbiB0aGlzIGNhc2UgeW91IGNhbiBlYXNpbHkgZ28gd2l0aCBKQk9ELCBo b3dldmVyIGEgZ29vZAogICAgICAgIHByYWN0aWNlIHdvdWxkIGJlIHRvIGhhdmUgYSBSQUlEIDEg Zm9yIGFyYml0ZXIgZGlza3MgKHRpbnkgMTI4R0IKICAgICAgICBTU0QtcyBhciBwZXJmZWN0bHkg c3VmZmljaWVudCBmb3Igdm9sdW1lcyB3aXRoIDEwcyBvZiBUQi1zIGluCiAgICAgICAgc2l6ZS4p PC9wPgogICAgICA8cD5UaGF0JiMzOTtzIGJhc2ljYWxseSBpdDwvcD4KICAgICAgPHA+VGhlIHJl c3QgYWJvdXQgdGhlIHJlbGlhYmlsaXR5IGFuZCBzZXR1cCBzY2VuYXJpb3MgeW91IGNhbiBmaW5k CiAgICAgICAgaW4gZ2x1c3RlciBkb2N1bWVudGF0aW9uLCBlc3BlY2lhbGx5IGxvb2sgZm9yIHF1 b3J1bSBhbmQgYXJiaXRlcgogICAgICAgIG5vZGUgY29uZmlncytvcHRpb25zLjwvcD4KICAgICAg PHA+Q2hlZXJzPC9wPgogICAgICA8cD5FcmVrbGU8L3A+CiAgICAgIFAuUy4gV2hhdCBJIHdhcyBt ZW50aW9uaW5nLCByZWdhcmRpbmcgYSBnb29kIHByYWN0aWNlIGlzIG1vc3RseQogICAgICByZWxh dGVkIHRvIHRoZSBvcGVyYXRpb25zIG9mIGdsdXN0ZXIgbm90IGluc3RhbGxhdGlvbiBvcgogICAg ICBkZXBsb3ltZW50LCBpLmUuIG5vdCB0aGUgY29uY2VwdHVhbCB1bmRlcnN0YW5kaW5nIG9mIGds dXN0ZXIKICAgICAgKGNvbmNlcHR1YWxseSBpdCYjMzk7cyBhIEpCT0Qgc3lzdGVtKS48YnI+CiAg ICAgIDxicj4KICAgICAgPGRpdiBjbGFzcz0ibV81MjM0OTIwMTM0NzM2ODIyNzJtb3otY2l0ZS1w cmVmaXgiPk9uIDA4LzA3LzIwMTcgMDU6NDEgUE0sIEZFUk5BTkRPCiAgICAgICAgRlJFRElBTkkg d3JvdGU6PGJyPgogICAgICA8L2Rpdj4KICAgICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSI+CiAg ICAgICAgCiAgICAgICAgPHA+VGhhbmtzIGZvciB0aGUgY2xhcmlmaWNhdGlvbiBFcmVrbGUuPC9w PgogICAgICAgIDxwPkhvd2V2ZXIgSSBnZXQgc3VycHJpc2VkIHdpdGggdGhpcyB3YXkgb2Ygb3Bl cmF0aW5nIGZyb20KICAgICAgICAgIEdsdXN0ZXJGUyBhcyBpdCBhZGRzIGFub3RoZXIgbGF5ZXIg b2YgY29tcGxleGl0eSB0byB0aGUgc3lzdGVtCiAgICAgICAgICAoZWl0aGVyIGEgaGFyZHdhcmUg b3Igc29mdHdhcmUgUkFJRCkgYmVmb3JlIHRoZSBnbHVzdGVyIGNvbmZpZwogICAgICAgICAgYW5k IGluY3JlYXNlIHRoZSBzeXN0ZW0mIzM5O3Mgb3ZlcmFsbCBjb3N0cy48YnI+CiAgICAgICAgPC9w PgogICAgICAgIDxwPkFuIGltcG9ydGFudCBwb2ludCB0byBjb25zaWRlciBpczogSW4gUkFJRCBj b25maWd1cmF0aW9uIHlvdQogICAgICAgICAgYWxyZWFkeSBoYXZlIHNwYWNlICYjMzk7d2FzdGVk JiMzOTsgaW4gb3JkZXIgdG8gYnVpbGQgcmVkdW5kYW5jeQogICAgICAgICAgKGVpdGhlciBSQUlE IDEsIDUsIG9yIDYpLiBUaGVuIHdoZW4geW91IGhhdmUgR2x1c3RlckZTIG9uIHRoZQogICAgICAg ICAgdG9wIG9mIHNldmVyYWwgUkFJRHMgeW91IGhhdmUgYWdhaW4gbW9yZSBkYXRhIHJlcGxpY2F0 ZWQgc28KICAgICAgICAgIHlvdSBlbmQgdXAgd2l0aCB0aGUgc2FtZSBkYXRhIGNvbnN1bWluZyBt b3JlIHNwYWNlIGluIGEgZ3JvdXAKICAgICAgICAgIG9mIGRpc2tzIGFuZCBhZ2FpbiBvbiB0aGUg dG9wIG9mIHNldmVyYWwgUkFJRHMgZGVwZW5kaW5nIG9uCiAgICAgICAgICB0aGUgR2x1c3RlciBj b25maWd1cmF0aW9uIHlvdSBoYXZlIChpbiBhIFJBSUQgMSBjb25maWcgdGhlCiAgICAgICAgICBz YW1lIGRhdGEgaXMgcmVwbGljYXRlZCA0IHRpbWVzKS48L3A+CiAgICAgICAgPHA+WWV0IGFub3Ro ZXIgZG93bnNpZGUgb2YgaGF2aW5nIGEgUkFJRCAoc3BlY2lhbGx5IFJBSUQgNSBvciA2KQogICAg ICAgICAgaXMgdGhhdCBpdCByZWR1Y2VzIGNvbnNpZGVyYWJseSB0aGUgd3JpdGUgc3BlZWRzIGFz IGVhY2ggZ3JvdXAKICAgICAgICAgIG9mIGRpc2tzIHdpbGwgZW5kIHVwIGhhdmluZyB0aGUgd3Jp dGUgc3BlZWQgb2YgYSBzaW5nbGUgZGlzawogICAgICAgICAgYXMgYWxsIG90aGVyIGRpc2tzIG9m IHRoYXQgZ3JvdXAgaGF2ZSB0byB3YWl0IGZvciBlYWNoIG90aGVyCiAgICAgICAgICB0byB3cml0 ZSBhcyB3ZWxsLjxicj4KICAgICAgICA8L3A+CiAgICAgICAgPHA+VGhlcmVmb3JlIGlmIEdsdXN0 ZXIgYWxyZWFkeSByZXBsaWNhdGVzIGRhdGEgd2h5IGRvZXMgaXQKICAgICAgICAgIGNyZWF0ZSB0 aGlzIGJpZyBwYWluIHlvdSBtZW50aW9uZWQgaWYgdGhlIGRhdGEgaXMgcmVwbGljYXRlZAogICAg ICAgICAgc29tZXdoZXJlIGVsc2UsIGNhbiBzdGlsbCBiZSByZXRyaWV2ZWQgdG8gYm90aCBzZXJ2 ZSBjbGllbnRzCiAgICAgICAgICBhbmQgcmVjb25zdHJ1Y3QgdGhlIGVxdWl2YWxlbnQgZGlzayB3 aGVuIGl0IGlzIHJlcGxhY2VkID88L3A+CiAgICAgICAgPHA+RmVybmFuZG88YnI+CiAgICAgICAg PC9wPgogICAgICAgIDxicj4KICAgICAgICA8ZGl2IGNsYXNzPSJtXzUyMzQ5MjAxMzQ3MzY4MjI3 Mm1vei1jaXRlLXByZWZpeCI+T24gMDcvMDgvMjAxNyAxMDoyNiwgRXJla2xlCiAgICAgICAgICBN YWdyYWR6ZSB3cm90ZTo8YnI+CiAgICAgICAgPC9kaXY+CiAgICAgICAgPGJsb2NrcXVvdGUgdHlw ZT0iY2l0ZSI+CiAgICAgICAgICAKICAgICAgICAgIDxwPkhpIEZyZW5hbmRvLDwvcD4KICAgICAg ICAgIDxwPkhlcmUgaXMgbXkgZXhwZXJpZW5jZSwgaWYgeW91IGNvbnNpZGVyIGEgcGFydGljdWxh ciBoYXJkCiAgICAgICAgICAgIGRyaXZlIGFzIGEgYnJpY2sgZm9yIGdsdXN0ZXIgdm9sdW1lIGFu ZCBpdCBkaWVzLCBpLmUuIGl0CiAgICAgICAgICAgIGJlY29tZXMgbm90IGFjY2Vzc2libGUgaXQm IzM5O3MgYSBodWdlIGhhc3NsZSB0byBkaXNjYXJkIHRoYXQKICAgICAgICAgICAgYnJpY2sgYW5k IGV4Y2hhbmdlIHdpdGggYW5vdGhlciBvbmUsIHNpbmNlIGdsdXN0ZXIgc29tZQogICAgICAgICAg ICB0cmllcyB0byBhY2Nlc3MgdGhhdCBicm9rZW4gYnJpY2sgYW5kIGl0JiMzOTtzIGNhdXNpbmcg KGF0IGxlYXN0CiAgICAgICAgICAgIGl0IGNhdXNlIGZvciBtZSkgYSBiaWcgcGFpbiwgdGhlcmVm b3JlIGl0JiMzOTtzIGJldHRlciB0byBoYXZlIGEKICAgICAgICAgICAgUkFJRCBhcyBicmljaywg aS5lLiBoYXZlIFJBSUQgMSAobWlycm9yaW5nKSBmb3IgZWFjaCBicmljaywKICAgICAgICAgICAg aW4gdGhpcyBjYXNlIGlmIHRoZSBkaXNrIGlzIGRvd24geW91IGNhbiBlYXNpbHkgZXhjaGFuZ2Ug aXQKICAgICAgICAgICAgYW5kIHJlYnVpbGQgdGhlIFJBSUQgd2l0aG91dCBnb2luZyBvZmZsaW5l LCBpLmUgc3dpdGNoaW5nCiAgICAgICAgICAgIG9mZiB0aGUgdm9sdW1lIGRvaW5nIGJyaWNrIG1h bmlwdWxhdGlvbnMgYW5kIHN3aXRjaGluZyBpdAogICAgICAgICAgICBiYWNrIG9uLjxicj4KICAg ICAgICAgIDwvcD4KICAgICAgICAgIDxwPkNoZWVyczwvcD4KICAgICAgICAgIDxwPkVyZWtsZTxi cj4KICAgICAgICAgIDwvcD4KICAgICAgICAgIDxicj4KICAgICAgICAgIDxkaXYgY2xhc3M9Im1f NTIzNDkyMDEzNDczNjgyMjcybW96LWNpdGUtcHJlZml4Ij5PbiAwOC8wNy8yMDE3IDAzOjA0IFBN LCBGRVJOQU5ETwogICAgICAgICAgICBGUkVESUFOSSB3cm90ZTo8YnI+CiAgICAgICAgICA8L2Rp dj4KICAgICAgICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiPgogICAgICAgICAgICAKICAgICAg ICAgICAgPHA+Rm9yIGFueSBSQUlEIDUgb3IgNiBjb25maWd1cmF0aW9uIEkgbm9ybWFsbHkgZm9s bG93IGEKICAgICAgICAgICAgICBzaW1wbGUgZ29sZCBydWxlIHdoaWNoIGdhdmUgZ29vZCByZXN1 bHRzIHNvIGZhcjo8YnI+CiAgICAgICAgICAgICAgLSB1cCB0byA0IGRpc2tzIFJBSUQgNTxicj4K ICAgICAgICAgICAgICAtIDUgb3IgbW9yZSBkaXNrcyBSQUlEIDY8L3A+CiAgICAgICAgICAgIDxw Pkhvd2V2ZXIgSSBkaWRuJiMzOTt0IHJlYWxseSB1bmRlcnN0YW5kIHdlbGwgdGhlCiAgICAgICAg ICAgICAgcmVjb21tZW5kYXRpb24gdG8gdXNlIGFueSBSQUlEIHdpdGggR2x1c3RlckZTLiBJIGFs d2F5cwogICAgICAgICAgICAgIHRob3VnaHQgdGhhdCBHbHVzdGVGUyBsaWtlcyB0byB3b3JrIGlu IEpCT0QgbW9kZSBhbmQKICAgICAgICAgICAgICBjb250cm9sIHRoZSBkaXNrcyAoYnJpY2tzKSBk aXJlY3RsbHkgc28geW91IGNhbiBjcmVhdGUKICAgICAgICAgICAgICB3aGF0ZXZlciBkaXN0cmli dXRpb24gcnVsZSB5b3Ugd2lzaCwgYW5kIGlmIGEgc2luZ2xlIGRpc2sKICAgICAgICAgICAgICBm YWlscyB5b3UganVzdCByZXBsYWNlIGl0IGFuZCB3aGljaCBvYnZpb3VzbHkgaGF2ZSB0aGUKICAg ICAgICAgICAgICBkYXRhIHJlcGxpY2F0ZWQgZnJvbSBhbm90aGVyLiBUaGUgb25seSBkb3duc2lk ZSBvZiB1c2luZwogICAgICAgICAgICAgIGluIHRoaXMgd2F5IGlzIHRoYXQgdGhlIHJlcGxpY2F0 aW9uIGRhdGEgd2lsbCBiZSBmbG93CiAgICAgICAgICAgICAgYWNjcm9zcyBhbGwgc2VydmVycyBi dXQgdGhhdCBpcyBub3QgbXVjaCBhIGJpZyBpc3N1ZS48L3A+CiAgICAgICAgICAgIDxwPkFueW9u ZSBjYW4gZWxhYm9yYXRlIGFib3V0IFVzaW5nIFJBSUQgKyBHbHVzdGVyRlMgYW5kCiAgICAgICAg ICAgICAgSkJPRCArIEdsdXN0ZXJGUy48L3A+CiAgICAgICAgICAgIDxwPlRoYW5rczxicj4KICAg ICAgICAgICAgICBSZWdhcmRzPGJyPgogICAgICAgICAgICAgIEZlcm5hbmRvPGJyPgogICAgICAg ICAgICA8L3A+CiAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgPGRpdiBjbGFzcz0ibV81MjM0 OTIwMTM0NzM2ODIyNzJtb3otY2l0ZS1wcmVmaXgiPk9uIDA3LzA4LzIwMTcgMDM6NDYsIERldmlu CiAgICAgICAgICAgICAgQWNvc3RhIHdyb3RlOjxicj4KICAgICAgICAgICAgPC9kaXY+CiAgICAg ICAgICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiPgogICAgICAgICAgICAgIAogICAgICAgICAg ICAgIDxkaXYgaWQ9Im1fNTIzNDkyMDEzNDczNjgyMjcyYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9 ImNvbG9yOnJnYigwLDAsMCk7bWFyZ2luOjBweCI+PGZvbnQgZmFjZT0iSW5wdXQKICAgICAgICAg ICAgICAgICAgTW9ubyI+PGJyPgogICAgICAgICAgICAgICAgPC9mb250PjwvZGl2PgogICAgICAg ICAgICAgIDxkaXYgaWQ9Im1fNTIzNDkyMDEzNDczNjgyMjcyYmxvb3BfY3VzdG9tZm9udCIgc3R5 bGU9ImNvbG9yOnJnYigwLDAsMCk7bWFyZ2luOjBweCI+PGZvbnQgZmFjZT0iSW5wdXQKICAgICAg ICAgICAgICAgICAgTW9ubyI+TW9hY2lyLDwvZm9udD48L2Rpdj4KICAgICAgICAgICAgICA8ZGl2 IGlkPSJtXzUyMzQ5MjAxMzQ3MzY4MjI3MmJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJjb2xvcjpy Z2IoMCwwLDApO21hcmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0CiAgICAgICAgICAgICAgICAg IE1vbm8iPjxicj4KICAgICAgICAgICAgICAgIDwvZm9udD48L2Rpdj4KICAgICAgICAgICAgICA8 ZGl2IGlkPSJtXzUyMzQ5MjAxMzQ3MzY4MjI3MmJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJjb2xv cjpyZ2IoMCwwLDApO21hcmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0CiAgICAgICAgICAgICAg ICAgIE1vbm8iPkkgaGF2ZSByZWNlbnRseSBpbnN0YWxsZWQgbXVsdGlwbGUgUmVkIEhhdAogICAg ICAgICAgICAgICAgICBWaXJ0dWFsaXphdGlvbiBob3N0cyBmb3Igc2V2ZXJhbCBkaWZmZXJlbnQg Y29tcGFuaWVzLAogICAgICAgICAgICAgICAgICBhbmQgaGF2ZSBkZWFsdCB3aXRoIHRoZSBSZWQg SGF0IFN1cHBvcnQgVGVhbSBpbiBkZXB0aAogICAgICAgICAgICAgICAgICBhYm91dCBvcHRpbWFs IGNvbmZpZ3VyYXRpb24gaW4gcmVnYXJkcyB0byBzZXR0aW5nIHVwCiAgICAgICAgICAgICAgICAg IEdsdXN0ZXJGUyBtb3N0IGVmZmljaWVudGx5IGFuZCBJIHdhbnRlZCB0byBzaGFyZSB3aXRoCiAg ICAgICAgICAgICAgICAgIHlvdSB3aGF0IEkgbGVhcm5lZC48L2ZvbnQ+PC9kaXY+CiAgICAgICAg ICAgICAgPGRpdiBpZD0ibV81MjM0OTIwMTM0NzM2ODIyNzJibG9vcF9jdXN0b21mb250IiBzdHls ZT0iY29sb3I6cmdiKDAsMCwwKTttYXJnaW46MHB4Ij48Zm9udCBmYWNlPSJJbnB1dAogICAgICAg ICAgICAgICAgICBNb25vIj48YnI+CiAgICAgICAgICAgICAgICA8L2ZvbnQ+PC9kaXY+CiAgICAg ICAgICAgICAgPGRpdiBpZD0ibV81MjM0OTIwMTM0NzM2ODIyNzJibG9vcF9jdXN0b21mb250IiBz dHlsZT0iY29sb3I6cmdiKDAsMCwwKTttYXJnaW46MHB4Ij48Zm9udCBmYWNlPSJJbnB1dAogICAg ICAgICAgICAgICAgICBNb25vIj5JbiBnZW5lcmFsIFJlZCBIYXQgVmlydHVhbGl6YXRpb24gdGVh bSBmcm93bnMKICAgICAgICAgICAgICAgICAgdXBvbiB1c2luZyBlYWNoIERJU0sgb2YgdGhlIHN5 c3RlbSBhcyBqdXN0IGEgSkJPRCwKICAgICAgICAgICAgICAgICAgc3VyZSB0aGVyZSBpcyBzb21l IHByb3RlY3Rpb24gYnkgaGF2aW5nIHRoZSBkYXRhCiAgICAgICAgICAgICAgICAgIHJlcGxpY2F0 ZWQsIGhvd2V2ZXIsIHRoZSByZWNvbW1lbmRhdGlvbiBpcyB0byB1c2UgUkFJRAogICAgICAgICAg ICAgICAgICA2IChwcmVmZXJyZWQpIG9yIFJBSUQtNSwgb3IgYXQgbGVhc3QgUkFJRC0xIGF0IHRo ZQogICAgICAgICAgICAgICAgICB2ZXJ5IGxlYXN0LjwvZm9udD48L2Rpdj4KICAgICAgICAgICAg ICA8ZGl2IGlkPSJtXzUyMzQ5MjAxMzQ3MzY4MjI3MmJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJj b2xvcjpyZ2IoMCwwLDApO21hcmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0CiAgICAgICAgICAg ICAgICAgIE1vbm8iPjxicj4KICAgICAgICAgICAgICAgIDwvZm9udD48L2Rpdj4KICAgICAgICAg ICAgICA8ZGl2IGlkPSJtXzUyMzQ5MjAxMzQ3MzY4MjI3MmJsb29wX2N1c3RvbWZvbnQiIHN0eWxl PSJtYXJnaW46MHB4Ij48Zm9udCBmYWNlPSJJbnB1dCBNb25vIj5IZXJlIGlzIHRoZcKgZGlyZWN0 IHF1b3RlIGZyb20gUmVkCiAgICAgICAgICAgICAgICAgIEhhdCB3aGVuIEkgYXNrZWQgYWJvdXQg UkFJRCBhbmQgQnJpY2tzOjwvZm9udD48L2Rpdj4KICAgICAgICAgICAgICA8ZGl2IGlkPSJtXzUy MzQ5MjAxMzQ3MzY4MjI3MmJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJtYXJnaW46MHB4Ij48Zm9u dCBmYWNlPSJJbnB1dCBNb25vIj48aT48YnI+CiAgICAgICAgICAgICAgICAgIDwvaT48L2ZvbnQ+ PC9kaXY+CiAgICAgICAgICAgICAgPGRpdiBpZD0ibV81MjM0OTIwMTM0NzM2ODIyNzJibG9vcF9j dXN0b21mb250IiBzdHlsZT0ibWFyZ2luOjBweCI+PGZvbnQgZmFjZT0iSW5wdXQgTW9ubyI+PGk+ JnF1b3Q7QSB0eXBpY2FsIEdsdXN0ZXIgY29uZmlndXJhdGlvbgogICAgICAgICAgICAgICAgICAg IHdvdWxkIHVzZSBSQUlEIHVuZGVybmVhdGggdGhlIGJyaWNrcy4gUkFJRCA2IGlzIG1vc3QKICAg ICAgICAgICAgICAgICAgICB0eXBpY2FsIGFzIGl0IGdpdmVzIHlvdSAyIGRpc2sgZmFpbHVyZSBw cm90ZWN0aW9uLAogICAgICAgICAgICAgICAgICAgIGJ1dCBSQUlEIDUgY291bGQgYmUgdXNlZCB0 b28uwqBPbmNlIHlvdSBoYXZlIHRoZQogICAgICAgICAgICAgICAgICAgIFJBSURlZCBicmlja3Ms IHlvdSYjMzk7ZCB0aGVuIGFwcGx5IHRoZSBkZXNpcmVkCiAgICAgICAgICAgICAgICAgICAgcmVw bGljYXRpb24gb24gdG9wIG9mIHRoYXQuIFRoZSBtb3N0IHBvcHVsYXIgd2F5IG9mCiAgICAgICAg ICAgICAgICAgICAgZG9pbmcgdGhpcyB3b3VsZCBiZSBkaXN0cmlidXRlZCByZXBsaWNhdGVkIHdp dGggMngKICAgICAgICAgICAgICAgICAgICByZXBsaWNhdGlvbi7CoEluIGdlbmVyYWwgeW91JiMz OTtsbCBnZXQgYmV0dGVyCiAgICAgICAgICAgICAgICAgICAgcGVyZm9ybWFuY2Ugd2l0aCBsYXJn ZXIgYnJpY2tzLiAxMiBkcml2ZXMgaXMgb2Z0ZW4gYQogICAgICAgICAgICAgICAgICAgIHN3ZWV0 IHNwb3QuIEFub3RoZXIgb3B0aW9uIHdvdWxkIGJlIHRvIGNyZWF0ZSBhCiAgICAgICAgICAgICAg ICAgICAgc2VwYXJhdGUgdGllciB1c2luZyBhbGwgU1NE4oCZcy7igJ3CoDwvaT48L2ZvbnQ+PC9k aXY+CiAgICAgICAgICAgICAgPGRpdiBpZD0ibV81MjM0OTIwMTM0NzM2ODIyNzJibG9vcF9jdXN0 b21mb250IiBzdHlsZT0ibWFyZ2luOjBweCI+PGJyPgogICAgICAgICAgICAgIDwvZGl2PgogICAg ICAgICAgICAgIDxkaXYgaWQ9Im1fNTIzNDkyMDEzNDczNjgyMjcyYmxvb3BfY3VzdG9tZm9udCIg c3R5bGU9Im1hcmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0IE1vbm8iPjxpPkluIG9yZGVyIHRv IFNTRMKgdGllcmluZyBmcm9tIG15CiAgICAgICAgICAgICAgICAgICAgdW5kZXJzdGFuZGluZyB5 b3Ugd291bGQgbmVlZCAxIHggTlZNZSBkcml2ZSBpbiBlYWNoCiAgICAgICAgICAgICAgICAgICAg c2VydmVyLCBvciA0IHggU1NEIGhvdCB0aWVyIChpdCBuZWVkcyB0byBiZQogICAgICAgICAgICAg ICAgICAgIGRpc3RyaWJ1dGVkLCByZXBsaWNhdGVkIGZvciB0aGUgaG90IHRpZXIgaWYgbm90CiAg ICAgICAgICAgICAgICAgICAgdXNpbmcgTlZNRSkuIFNvIHdpdGggeW91IG9ubHkgaGF2aW5nIDEg U1NEIGRyaXZlIGluCiAgICAgICAgICAgICAgICAgICAgZWFjaCBzZXJ2ZXIswqBJ4oCZZCBzdWdn ZXN0IG1heWJlIGxvb2tpbmcgaW50byB0aGUgTlZNRQogICAgICAgICAgICAgICAgICAgIG9wdGlv bi7CoDwvaT48L2ZvbnQ+PC9kaXY+CiAgICAgICAgICAgICAgPGRpdiBpZD0ibV81MjM0OTIwMTM0 NzM2ODIyNzJibG9vcF9jdXN0b21mb250IiBzdHlsZT0ibWFyZ2luOjBweCI+PGZvbnQgZmFjZT0i SW5wdXQgTW9ubyI+PGk+PGJyPgogICAgICAgICAgICAgICAgICA8L2k+PC9mb250PjwvZGl2Pgog ICAgICAgICAgICAgIDxkaXYgaWQ9Im1fNTIzNDkyMDEzNDczNjgyMjcyYmxvb3BfY3VzdG9tZm9u dCIgc3R5bGU9Im1hcmdpbjowcHgiPjxmb250IGZhY2U9IklucHV0IE1vbm8iPjxpPlNpbmNlIHlv dXIgdXNpbmcgb25seSAzLXNlcnZlcnMsCiAgICAgICAgICAgICAgICAgICAgd2hhdCBJ4oCZZCBw cm9iYWJseSBzdWdnZXN0IGlzIHRvIGRvICgyIFJlcGxpY2FzICsKICAgICAgICAgICAgICAgICAg ICBBcmJpdGVyIE5vZGUpLCB0aGlzIHNldHVwIGFjdHVhbGx5IGRvZXNu4oCZdCByZXF1aXJlCiAg ICAgICAgICAgICAgICAgICAgdGhlIDNyZCBzZXJ2ZXIgdG8gaGF2ZSBiaWcgZHJpdmVzIGF0IGFs bCBhcyBpdCBvbmx5CiAgICAgICAgICAgICAgICAgICAgc3RvcmVzIG1ldGEtZGF0YSBhYm91dCB0 aGUgZmlsZXMgYW5kIG5vdCBhY3R1YWxseSBhCiAgICAgICAgICAgICAgICAgICAgZnVsbCBjb3B5 LsKgPC9pPjwvZm9udD48L2Rpdj4KICAgICAgICAgICAgICA8ZGl2IGlkPSJtXzUyMzQ5MjAxMzQ3 MzY4MjI3MmJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPSJtYXJnaW46MHB4Ij48Zm9udCBmYWNlPSJJ bnB1dCBNb25vIj48aT48YnI+CiAgICAgICAgICAgICAgICAgIDwvaT48L2ZvbnQ+PC9kaXY+CiAg ICAgICAgICAgICAgPGRpdiBpZD0ibV81MjM0OTIwMTM0NzM2ODIyNzJibG9vcF9jdXN0b21mb250 IiBzdHlsZT0ibWFyZ2luOjBweCI+PGZvbnQgZmFjZT0iSW5wdXQgTW9ubyI+PGk+UGxlYXNlIHNl ZSB0aGUgYXR0YWNoZWQgZG9jdW1lbnQKICAgICAgICAgICAgICAgICAgICB0aGF0IHdhcyBnaXZl biB0byBtZSBieSBSZWQgSGF0IHRvIGdldCBtb3JlCiAgICAgICAgICAgICAgICAgICAgaW5mb3Jt YXRpb24gb24gdGhpcy4gSG9wZSB0aGlzIGluZm9ybWF0aW9uIGhlbHBzCiAgICAgICAgICAgICAg ICAgICAgeW91LjwvaT48L2ZvbnQ+PC9kaXY+CiAgICAgICAgICAgICAgPGRpdiBpZD0ibV81MjM0 OTIwMTM0NzM2ODIyNzJibG9vcF9jdXN0b21mb250IiBzdHlsZT0ibWFyZ2luOjBweCI+PGZvbnQg ZmFjZT0iSW5wdXQgTW9ubyI+PGk+PGJyPgogICAgICAgICAgICAgICAgICA8L2k+PC9mb250Pjwv ZGl2PgogICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICA8ZGl2IGlkPSJtXzUyMzQ5MjAx MzQ3MzY4MjI3MmJsb29wX3NpZ25fMTUwMjA4NzM3NjcyNTQ2OTE4NCIgY2xhc3M9Im1fNTIzNDky MDEzNDczNjgyMjcyYmxvb3Bfc2lnbiI+PHNwYW4+LS08L3NwYW4+PGJyPgogICAgICAgICAgICAg ICAgPGRpdiBjbGFzcz0ibV81MjM0OTIwMTM0NzM2ODIyNzJnbWFpbF9zaWduYXR1cmUiPgogICAg ICAgICAgICAgICAgICA8ZGl2IGRpcj0ibHRyIj4KICAgICAgICAgICAgICAgICAgICA8ZGl2Pjxi cj4KICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICA8ZGl2PkRl dmluIEFjb3N0YSwgUkhDQSwgUkhWQ0E8L2Rpdj4KICAgICAgICAgICAgICAgICAgICA8ZGl2PlJl ZCBIYXQgQ2VydGlmaWVkIEFyY2hpdGVjdDwvZGl2PgogICAgICAgICAgICAgICAgICA8L2Rpdj4K ICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAg IDxicj4KICAgICAgICAgICAgICA8cCBjbGFzcz0ibV81MjM0OTIwMTM0NzM2ODIyNzJhaXJtYWls X29uIj5PbiBBdWd1c3QgNiwgMjAxNyBhdCA3OjI5OjI5IFBNLAogICAgICAgICAgICAgICAgTW9h Y2lyIEZlcnJlaXJhICg8YSBocmVmPSJtYWlsdG86bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb20i IHRhcmdldD0iX2JsYW5rIj5tb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbTwvYT4pCiAgICAgICAg ICAgICAgICB3cm90ZTo8L3A+CiAgICAgICAgICAgICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSIg Y2xhc3M9Im1fNTIzNDkyMDEzNDczNjgyMjcyY2xlYW5fYnEiPjxzcGFuPgogICAgICAgICAgICAg ICAgICA8ZGl2IGRpcj0ibHRyIj4KICAgICAgICAgICAgICAgICAgICA8ZGl2PgogICAgICAgICAg ICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICA8ZGl2IGlkPSJtXzUyMzQ5MjAxMzQ3 MzY4MjI3MmRpdnRhZ2RlZmF1bHR3cmFwcGVyIiBzdHlsZT0iZm9udC1zaXplOjEycHQ7Y29sb3I6 IzAwMDAwMDtmb250LWZhbWlseTpDYWxpYnJpLEhlbHZldGljYSxzYW5zLXNlcmlmIiBkaXI9Imx0 ciI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxwPjxzcGFuPkkgYW0gd2lsbGluZyB0byBhc3Nl bWJsZSBhIG9WaXJ0ICZxdW90O3BvZCZxdW90OywKICAgICAgICAgICAgICAgICAgICAgICAgICAg IG1hZGUgb2YgMyBzZXJ2ZXJzLCBlYWNoIHdpdGggMiBDUFUgc29ja2V0cwogICAgICAgICAgICAg ICAgICAgICAgICAgICAgb2YgMTIgY29yZXMsIDI1NkdCIFJBTSwgNyBIREQgMTBLLCAxIFNTRC4K ICAgICAgICAgICAgICAgICAgICAgICAgICAgIFRoZSBpZGVhIGlzIHRvIHVzZSBHbHVzdGVyRlMg dG8gcHJvdmlkZSBIQQogICAgICAgICAgICAgICAgICAgICAgICAgICAgZm9yIHRoZSBWTXMuIFRo ZSAzIHNlcnZlcnMgaGF2ZSBhIGR1YWwgNDBHYgogICAgICAgICAgICAgICAgICAgICAgICAgICAg TklDIGFuZCBhIGR1YWwgMTBHYiBOSUMuIFNvIG15IGludGVudGlvbiBpcwogICAgICAgICAgICAg ICAgICAgICAgICAgICAgdG8gY3JlYXRlIGEgbG9vcCBsaWtlIGEgc2VydmVyIHRyaWFuZ2xlCiAg ICAgICAgICAgICAgICAgICAgICAgICAgICB1c2luZyB0aGUgNDBHYiBOSUNzIGZvciB2aXJ0dWFs aXphdGlvbiBmaWxlcwogICAgICAgICAgICAgICAgICAgICAgICAgICAgKFZNcyAucWNvdzIpIGFj Y2VzcyBhbmQgdG8gbW92ZSBWTXMgYXJvdW5kCiAgICAgICAgICAgICAgICAgICAgICAgICAgICB0 aGUgcG9kIChlYXN0IC93ZXN0IHRyYWZmaWMpIHdoaWxlIHVzaW5nIHRoZQogICAgICAgICAgICAg ICAgICAgICAgICAgICAgMTBHYiBpbnRlcmZhY2VzIGZvciBnaXZpbmcgc2VydmljZXMgdG8gdGhl CiAgICAgICAgICAgICAgICAgICAgICAgICAgICBvdXRzaWRlIHdvcmxkIChub3J0aC9zb3V0aCB0 cmFmZmljKS48L3NwYW4+PC9wPgogICAgICAgICAgICAgICAgICAgICAgICA8cD48YnI+CiAgICAg ICAgICAgICAgICAgICAgICAgIDwvcD4KICAgICAgICAgICAgICAgICAgICAgICAgPHA+VGhpcyBz YWlkLCBteSBmaXJzdCBxdWVzdGlvbiBpczogSG93IHNob3VsZCBJCiAgICAgICAgICAgICAgICAg ICAgICAgICAgZGVwbG95IEdsdXN0ZXJGUyBpbiBzdWNoIG9WaXJ0IHNjZW5hcmlvPyBNeQogICAg ICAgICAgICAgICAgICAgICAgICAgIHF1ZXN0aW9ucyBhcmU6PC9wPgogICAgICAgICAgICAgICAg ICAgICAgICA8cD48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgIDwvcD4KICAgICAgICAgICAg ICAgICAgICAgICAgPHA+MSAtIFNob3VsZCBJIGNyZWF0ZSAzIFJBSUQgKGkuZS46IFJBSUQgNSks CiAgICAgICAgICAgICAgICAgICAgICAgICAgb25lIG9uIGVhY2ggb1ZpcnQgbm9kZSwgYW5kIHRo ZW4gY3JlYXRlIGEKICAgICAgICAgICAgICAgICAgICAgICAgICBHbHVzdGVyRlMgdXNpbmcgdGhl bT88L3A+CiAgICAgICAgICAgICAgICAgICAgICAgIDxwPjIgLSBJbnN0ZWFkLCBzaG91bGQgSSBj cmVhdGUgYSBKQk9EIGFycmF5CiAgICAgICAgICAgICAgICAgICAgICAgICAgbWFkZSBvZiBhbGwg c2VydmVyJiMzOTtzIGRpc2tzPzwvcD4KICAgICAgICAgICAgICAgICAgICAgICAgPHA+MyAtIFdo YXQgaXMgdGhlIGJlc3QgR2x1c3RlciBjb25maWd1cmF0aW9uIHRvCiAgICAgICAgICAgICAgICAg ICAgICAgICAgcHJvdmlkZSBmb3IgSEEgd2hpbGUgbm90IGNvbnN1bWluZyB0b28gbXVjaAogICAg ICAgICAgICAgICAgICAgICAgICAgIGRpc2sgc3BhY2U/PGJyPgogICAgICAgICAgICAgICAgICAg ICAgICA8L3A+CiAgICAgICAgICAgICAgICAgICAgICAgIDxwPjQgLSBEb2VzIGEgb1ZpcnQgaHlw ZXJ2aXNvciBwb2QgbGlrZSBJIGFtCiAgICAgICAgICAgICAgICAgICAgICAgICAgcGxhbm5pbmcg dG8gYnVpbGQsIGFuZCB0aGUgdmlydHVhbGl6YXRpb24KICAgICAgICAgICAgICAgICAgICAgICAg ICBlbnZpcm9ubWVudCwgYmVuZWZpdHMgZnJvbSB0aWVyaW5nIHdoZW4gdXNpbmcKICAgICAgICAg ICAgICAgICAgICAgICAgICBhIFNTRCBkaXNrPyBBbmQgeWVzLCB3aWxsIEdsdXN0ZXIgZG8gaXQg YnkKICAgICAgICAgICAgICAgICAgICAgICAgICBkZWZhdWx0IG9yIEkgaGF2ZSB0byBjb25maWd1 cmUgaXQgdG8gZG8gc28/PC9wPgogICAgICAgICAgICAgICAgICAgICAgICA8cD48YnI+CiAgICAg ICAgICAgICAgICAgICAgICAgIDwvcD4KICAgICAgICAgICAgICAgICAgICAgICAgPHA+QXQgdGhl IGJvdHRvbSBsaW5lLCB3aGF0IGlzIHRoZSBnb29kIHByYWN0aWNlCiAgICAgICAgICAgICAgICAg ICAgICAgICAgZm9yIHVzaW5nIEdsdXN0ZXJGUyBpbiBzbWFsbCBwb2RzIGZvcgogICAgICAgICAg ICAgICAgICAgICAgICAgIGVudGVycHJpc2VzPzxicj4KICAgICAgICAgICAgICAgICAgICAgICAg PC9wPgogICAgICAgICAgICAgICAgICAgICAgICA8cD48YnI+CiAgICAgICAgICAgICAgICAgICAg ICAgIDwvcD4KICAgICAgICAgICAgICAgICAgICAgICAgPHA+WW91IG9waW5pb24vZmVlZGJhY2sg d2lsbCBiZSByZWFsbHkKICAgICAgICAgICAgICAgICAgICAgICAgICBhcHByZWNpYXRlZCE8L3A+ CiAgICAgICAgICAgICAgICAgICAgICAgIDxwPk1vYWNpcjxicj4KICAgICAgICAgICAgICAgICAg ICAgICAgPC9wPgogICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAg ICAgICBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX188d2JyPl9fX19fX19fX19fX19fX19f IDxicj4KICAgICAgICAgICAgICAgICAgICAgIFVzZXJzIG1haWxpbmcgbGlzdCA8YnI+CiAgICAg ICAgICAgICAgICAgICAgICA8YSBocmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQub3JnIiB0YXJnZXQ9 Il9ibGFuayI+VXNlcnNAb3ZpcnQub3JnPC9hPiA8YnI+CiAgICAgICAgICAgICAgICAgICAgICA8 YSBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiIHRh cmdldD0iX2JsYW5rIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnLzx3YnI+bWFpbG1hbi9saXN0aW5m by91c2VyczwvYT4KICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAg ICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICA8L3NwYW4+ PC9ibG9ja3F1b3RlPgogICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICA8ZmllbGRzZXQg Y2xhc3M9Im1fNTIzNDkyMDEzNDczNjgyMjcybWltZUF0dGFjaG1lbnRIZWFkZXIiPjwvZmllbGRz ZXQ+CiAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgIDxwcmU+X19fX19fX19fX19fX19f X19fX19fX19fX19fX19fPHdicj5fX19fX19fX19fX19fX19fXwpVc2VycyBtYWlsaW5nIGxpc3QK PGEgY2xhc3M9Im1fNTIzNDkyMDEzNDczNjgyMjcybW96LXR4dC1saW5rLWFiYnJldmlhdGVkIiBo cmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQub3JnIiB0YXJnZXQ9Il9ibGFuayI+VXNlcnNAb3ZpcnQu b3JnPC9hPgo8YSBjbGFzcz0ibV81MjM0OTIwMTM0NzM2ODIyNzJtb3otdHh0LWxpbmstZnJlZXRl eHQiIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyIg dGFyZ2V0PSJfYmxhbmsiPmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvPHdicj5tYWlsbWFuL2xpc3Rp bmZvL3VzZXJzPC9hPgo8L3ByZT4KICAgICAgICAgICAgPC9ibG9ja3F1b3RlPgogICAgICAgICAg ICA8YnI+CiAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgPGZpZWxkc2V0IGNsYXNzPSJtXzUy MzQ5MjAxMzQ3MzY4MjI3Mm1pbWVBdHRhY2htZW50SGVhZGVyIj48L2ZpZWxkc2V0PgogICAgICAg ICAgICA8YnI+CiAgICAgICAgICAgIDxwcmU+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19f PHdicj5fX19fX19fX19fX19fX19fXwpVc2VycyBtYWlsaW5nIGxpc3QKPGEgY2xhc3M9Im1fNTIz NDkyMDEzNDczNjgyMjcybW96LXR4dC1saW5rLWFiYnJldmlhdGVkIiBocmVmPSJtYWlsdG86VXNl cnNAb3ZpcnQub3JnIiB0YXJnZXQ9Il9ibGFuayI+VXNlcnNAb3ZpcnQub3JnPC9hPgo8YSBjbGFz cz0ibV81MjM0OTIwMTM0NzM2ODIyNzJtb3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHA6 Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyIgdGFyZ2V0PSJfYmxhbmsi Pmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvPHdicj5tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPgo8 L3ByZT4KICAgICAgICAgIDwvYmxvY2txdW90ZT4KICAgICAgICAgIDxicj4KICAgICAgICA8L2Js b2NrcXVvdGU+CiAgICAgICAgPGJyPgogICAgICA8L2Jsb2NrcXVvdGU+CiAgICAgIDxicj4KICAg ICAgPHByZSBjbGFzcz0ibV81MjM0OTIwMTM0NzM2ODIyNzJtb3otc2lnbmF0dXJlIiBjb2xzPSI3 MiI+LS0gClJlY29naXplciBHcm91cCBHbWJICgpEci5yZXIubmF0LiBFcmVrbGUgTWFncmFkemUK TGVhZCBCaWcgRGF0YSBFbmdpbmVlcmluZyAmYW1wOyBEZXZPcHMKUmhlaW53ZXJrYWxsZWUgMiwg NTMyMjcgQm9ubgpUZWw6IDxhIGhyZWY9InRlbDorNDklMjAyMjglMjAyOTk3NDU1NSIgdmFsdWU9 Iis0OTIyODI5OTc0NTU1IiB0YXJnZXQ9Il9ibGFuayI+KzQ5IDIyOCAyOTk3NDU1NTwvYT4KCkUt TWFpbCA8YSBjbGFzcz0ibV81MjM0OTIwMTM0NzM2ODIyNzJtb3otdHh0LWxpbmstYWJicmV2aWF0 ZWQiIGhyZWY9Im1haWx0bzplcmVrbGUubWFncmFkemVAcmVjb2dpemVyLmRlIiB0YXJnZXQ9Il9i bGFuayI+ZXJla2xlLm1hZ3JhZHplQHJlY29naXplci5kZTwvYT4KV2ViOiA8YSBjbGFzcz0ibV81 MjM0OTIwMTM0NzM2ODIyNzJtb3otdHh0LWxpbmstYWJicmV2aWF0ZWQiIGhyZWY9Imh0dHA6Ly93 d3cucmVjb2dpemVyLmNvbSIgdGFyZ2V0PSJfYmxhbmsiPnd3dy5yZWNvZ2l6ZXIuY29tPC9hPgog ClJlY29naXplciBhdWYgTGlua2VkSW4gPGEgY2xhc3M9Im1fNTIzNDkyMDEzNDczNjgyMjcybW96 LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwczovL3d3dy5saW5rZWRpbi5jb20vY29tcGFu eS1iZXRhLzEwMDM5MTgyLyIgdGFyZ2V0PSJfYmxhbmsiPmh0dHBzOi8vd3d3LmxpbmtlZGluLmNv bS88d2JyPmNvbXBhbnktYmV0YS8xMDAzOTE4Mi88L2E+CkZvbGdlbiBTaWUgdW5zIGF1ZiBUd2l0 dGVyIDxhIGNsYXNzPSJtXzUyMzQ5MjAxMzQ3MzY4MjI3Mm1vei10eHQtbGluay1mcmVldGV4dCIg aHJlZj0iaHR0cHM6Ly90d2l0dGVyLmNvbS9yZWNvZ2l6ZXIiIHRhcmdldD0iX2JsYW5rIj5odHRw czovL3R3aXR0ZXIuY29tL3JlY29naXplcjwvYT4KIAotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS08d2JyPi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLTx3YnI+LS0tLS0KUmVjb2dp emVyIEdyb3VwIEdtYkgKR2VzY2jDpGZ0c2bDvGhyZXI6IE9saXZlciBIYWJpc2NoLCBDYXJzdGVu IEtyZXV0emUKSGFuZGVsc3JlZ2lzdGVyOiBBbXRzZ2VyaWNodCBCb25uIEhSQiAyMDcyNApTaXR6 IGRlciBHZXNlbGxzY2hhZnQ6IEJvbm47IFVTdC1JRC1Oci46IERFMjk0MTk1OTkzCiAKRGllc2Ug RS1NYWlsIGVudGjDpGx0IHZlcnRyYXVsaWNoZSB1bmQvb2RlciByZWNodGxpY2ggZ2VzY2jDvHR6 dGUgSW5mb3JtYXRpb25lbi4KV2VubiBTaWUgbmljaHQgZGVyIHJpY2h0aWdlIEFkcmVzc2F0IHNp bmQgb2RlciBkaWVzZSBFLU1haWwgaXJydMO8bWxpY2ggZXJoYWx0ZW4gaGFiZW4sCmluZm9ybWll cmVuIFNpZSBiaXR0ZSBzb2ZvcnQgZGVuIEFic2VuZGVyIHVuZCBsw7ZzY2hlbiBTaWUgZGllc2Ug TWFpbC4KRGFzIHVuZXJsYXVidGUgS29waWVyZW4gc293aWUgZGllIHVuYmVmdWd0ZSBXZWl0ZXJn YWJlIGRpZXNlciBNYWlsIHVuZCBkZXIgZGFyaW4gZW50aGFsdGVuZW4gSW5mb3JtYXRpb25lbiBp c3QgbmljaHQgZ2VzdGF0dGV0LjwvcHJlPgogICAgPC9ibG9ja3F1b3RlPgogICAgPGJyPgogIDwv ZGl2PjwvZGl2PjwvZGl2PgoKPGJyPl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzx3YnI+ X19fX19fX19fX19fX19fX188YnI+ClVzZXJzIG1haWxpbmcgbGlzdDxicj4KPGEgaHJlZj0ibWFp bHRvOlVzZXJzQG92aXJ0Lm9yZyI+VXNlcnNAb3ZpcnQub3JnPC9hPjxicj4KPGEgaHJlZj0iaHR0 cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzIiByZWw9Im5vcmVmZXJy ZXIiIHRhcmdldD0iX2JsYW5rIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnLzx3YnI+bWFpbG1hbi9s aXN0aW5mby91c2VyczwvYT48YnI+Cjxicj48L2Jsb2NrcXVvdGU+PC9kaXY+PGJyPjwvZGl2Pjwv ZGl2Pgo= --===============3876214987069104049==-- From ykaul at redhat.com Tue Aug 8 06:51:10 2017 Content-Type: multipart/mixed; boundary="===============2809832117699048411==" MIME-Version: 1.0 From: Yaniv Kaul To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Tue, 08 Aug 2017 09:50:27 +0300 Message-ID: In-Reply-To: B1592A25-DF71-4625-9271-5E1E2C37599C@orange.fr --===============2809832117699048411== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Tue, Aug 8, 2017 at 9:16 AM, Fabrice Bacchella < fabrice.bacchella(a)orange.fr> wrote: > > > Le 8 ao=C3=BBt 2017 =C3=A0 04:08, FERNANDO FREDIANI a > =C3=A9crit : > > > Even if you have a Hardware RAID Controller with Writeback cache you > will have a significant performance penalty and may not fully use all the > resources you mentioned you have. > > > > Nope again,from my experience with HP Smart Array and write back cache, > write, that goes in the cache, are even faster that read that must goes to > the disks. of course if the write are too fast and to big, they will over > overflow the cache. But on todays controller they are multi-gigabyte cach= e, > you must write a lot to fill them. And if you can afford 40Gb card, you c= an > afford decent controller. > The last sentence raises an excellent point: balance your resources. Don't spend a fortune on one component while another will end up being your bottleneck. Storage is usually the slowest link in the chain. I personally believe that spending the money on NVMe drives makes more sense than 40Gb (except [1], which is suspiciously cheap!) Y. [1] http://a.co/4hsCTqG > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > --===============2809832117699048411== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+PGRpdiBjbGFz cz0iZ21haWxfcXVvdGUiPk9uIFR1ZSwgQXVnIDgsIDIwMTcgYXQgOToxNiBBTSwgRmFicmljZSBC YWNjaGVsbGEgPHNwYW4gZGlyPSJsdHIiPiZsdDs8YSBocmVmPSJtYWlsdG86ZmFicmljZS5iYWNj aGVsbGFAb3JhbmdlLmZyIiB0YXJnZXQ9Il9ibGFuayI+ZmFicmljZS5iYWNjaGVsbGFAb3Jhbmdl LmZyPC9hPiZndDs8L3NwYW4+IHdyb3RlOjxicj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVv dGUiIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlk IHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCI+PHNwYW4+PGJyPgomZ3Q7IExlIDgg YW/Du3QgMjAxNyDDoCAwNDowOCwgRkVSTkFORE8gRlJFRElBTkkgJmx0OzxhIGhyZWY9Im1haWx0 bzpmZXJuYW5kby5mcmVkaWFuaUB1cHguY29tIiB0YXJnZXQ9Il9ibGFuayI+ZmVybmFuZG8uZnJl ZGlhbmlAdXB4LmNvbTwvYT4mZ3Q7IGEgw6ljcml0IDo8YnI+Cjxicj4KJmd0OyBFdmVuIGlmIHlv dSBoYXZlIGEgSGFyZHdhcmUgUkFJRCBDb250cm9sbGVyIHdpdGggV3JpdGViYWNrIGNhY2hlIHlv dSB3aWxsIGhhdmUgYSBzaWduaWZpY2FudCBwZXJmb3JtYW5jZSBwZW5hbHR5IGFuZCBtYXkgbm90 IGZ1bGx5IHVzZSBhbGwgdGhlIHJlc291cmNlcyB5b3UgbWVudGlvbmVkIHlvdSBoYXZlLjxicj4K Jmd0Ozxicj4KPGJyPgo8L3NwYW4+Tm9wZSBhZ2Fpbixmcm9tIG15IGV4cGVyaWVuY2Ugd2l0aCBI UCBTbWFydCBBcnJheSBhbmQgd3JpdGUgYmFjayBjYWNoZSwgd3JpdGUsIHRoYXQgZ29lcyBpbiB0 aGUgY2FjaGUsIGFyZSBldmVuIGZhc3RlciB0aGF0IHJlYWQgdGhhdCBtdXN0IGdvZXMgdG8gdGhl IGRpc2tzLiBvZiBjb3Vyc2UgaWYgdGhlIHdyaXRlIGFyZSB0b28gZmFzdCBhbmQgdG8gYmlnLCB0 aGV5IHdpbGwgb3ZlciBvdmVyZmxvdyB0aGUgY2FjaGUuIEJ1dCBvbiB0b2RheXMgY29udHJvbGxl ciB0aGV5IGFyZSBtdWx0aS1naWdhYnl0ZSBjYWNoZSwgeW91IG11c3Qgd3JpdGUgYSBsb3QgdG8g ZmlsbCB0aGVtLiBBbmQgaWYgeW91IGNhbiBhZmZvcmQgNDBHYiBjYXJkLCB5b3UgY2FuIGFmZm9y ZCBkZWNlbnQgY29udHJvbGxlci48YnI+PC9ibG9ja3F1b3RlPjxkaXY+PGJyPjwvZGl2PjxkaXY+ VGhlIGxhc3Qgc2VudGVuY2UgcmFpc2VzIGFuIGV4Y2VsbGVudCBwb2ludDogYmFsYW5jZSB5b3Vy IHJlc291cmNlcy4gRG9uJiMzOTt0IHNwZW5kIGEgZm9ydHVuZSBvbiBvbmUgY29tcG9uZW50IHdo aWxlIGFub3RoZXIgd2lsbCBlbmQgdXAgYmVpbmcgeW91ciBib3R0bGVuZWNrLsKgPC9kaXY+PGRp dj5TdG9yYWdlIGlzIHVzdWFsbHkgdGhlIHNsb3dlc3QgbGluayBpbiB0aGUgY2hhaW4uIEkgcGVy c29uYWxseSBiZWxpZXZlIHRoYXQgc3BlbmRpbmcgdGhlIG1vbmV5IG9uIE5WTWUgZHJpdmVzIG1h a2VzIG1vcmUgc2Vuc2UgdGhhbiA0MEdiIChleGNlcHQgWzFdLCB3aGljaCBpcyBzdXNwaWNpb3Vz bHkgY2hlYXAhKTwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+WS48L2Rpdj48ZGl2PlsxXcKgPHNw YW4gc3R5bGU9ImNvbG9yOnJnYigxNywxNywxNyk7Zm9udC1mYW1pbHk6JnF1b3Q7QW1hem9uIEVt YmVyJnF1b3Q7LEFyaWFsLHNhbnMtc2VyaWY7Zm9udC1zaXplOjEzcHgiPjxhIGhyZWY9Imh0dHA6 Ly9hLmNvLzRoc0NUcUciPmh0dHA6Ly9hLmNvLzRoc0NUcUc8L2E+PC9zcGFuPjwvZGl2PjxkaXY+ PHNwYW4gc3R5bGU9ImNvbG9yOnJnYigxNywxNywxNyk7Zm9udC1mYW1pbHk6JnF1b3Q7QW1hem9u IEVtYmVyJnF1b3Q7LEFyaWFsLHNhbnMtc2VyaWY7Zm9udC1zaXplOjEzcHgiPjxicj48L3NwYW4+ PC9kaXY+PGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjBweCAw cHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIwNCwyMDQpO3BhZGRp bmctbGVmdDoxZXgiPgo8ZGl2IGNsYXNzPSJnbWFpbC1tXy02Njg2NjkyMTk4NTU3NzEwNjkwSE9F blpiIj48ZGl2IGNsYXNzPSJnbWFpbC1tXy02Njg2NjkyMTk4NTU3NzEwNjkwaDUiPjxicj4KPGJy Pgo8YnI+Cl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzx3YnI+X19fX19fX19fX19fX19f X188YnI+ClVzZXJzIG1haWxpbmcgbGlzdDxicj4KPGEgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0 Lm9yZyIgdGFyZ2V0PSJfYmxhbmsiPlVzZXJzQG92aXJ0Lm9yZzwvYT48YnI+CjxhIGhyZWY9Imh0 dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyIgcmVsPSJub3JlZmVy cmVyIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuPHdicj4v bGlzdGluZm8vdXNlcnM8L2E+PGJyPgo8L2Rpdj48L2Rpdj48L2Jsb2NrcXVvdGU+PC9kaXY+PGJy PjwvZGl2PjwvZGl2Pgo= --===============2809832117699048411==-- From fabrice.bacchella at orange.fr Tue Aug 8 07:34:15 2017 Content-Type: multipart/mixed; boundary="===============5949613573116717547==" MIME-Version: 1.0 From: Fabrice Bacchella To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Tue, 08 Aug 2017 09:34:10 +0200 Message-ID: <323F6F64-EF25-4272-801D-B2D644905800@orange.fr> In-Reply-To: CAJgorsb_Bawvfnd=a25p+m3gXmsQp0fG+4q9JDvw43KjmRi-1g@mail.gmail.com --===============5949613573116717547== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --Apple-Mail=3D_88F0455F-3BE8-4421-89AF-B80F9A90AAF2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=3Dutf-8 > Le 8 ao=3DC3=3DBBt 2017 =3DC3=3DA0 08:50, Yaniv Kaul = a =3D =3DC3=3DA9crit : >=3D20 > Storage is usually the slowest link in the chain. I personally believe = =3D that spending the money on NVMe drives makes more sense than 40Gb =3D (except [1], which is suspiciously cheap!) >=3D20 > Y. > [1] http://a.co/4hsCTqG =3D http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=3D3Demr_na-c043740= 7=3D 8 It's supported on old Gen8 servers (G10 is comming). It must be coming =3D from an attic. --Apple-Mail=3D_88F0455F-3BE8-4421-89AF-B80F9A90AAF2 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=3Dutf-8
Le 8 ao=3DC3=3DBBt 2017 =3DC3=3DA0 08:50, Yaniv Kaul <ykaul(a)redhat.com>= ; a =3D =3DC3=3DA9crit :

<= div =3D class=3D3D"gmail_quote">

Storage is usually the slowest = =3D link in the chain. I personally believe that spending the money on NVMe =3D drives makes more sense than 40Gb (except [1], which is suspiciously =3D cheap!)

Y.
[1] http://a.co/4hsCTqG

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=3D3De= mr_=3D na-c04374078

It's supported on = =3D old Gen8 servers (G10 is comming). It must be coming from an =3D attic.

=3D --Apple-Mail=3D_88F0455F-3BE8-4421-89AF-B80F9A90AAF2-- --===============5949613573116717547== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" Ci0tQXBwbGUtTWFpbD1fODhGMDQ1NUYtM0JFOC00NDIxLTg5QUYtQjgwRjlBOTBBQUYyCkNvbnRl bnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKQ29udGVudC1UeXBlOiB0ZXh0 L3BsYWluOwoJY2hhcnNldD11dGYtOAoKCj4gTGUgOCBhbz1DMz1CQnQgMjAxNyA9QzM9QTAgMDg6 NTAsIFlhbml2IEthdWwgPHlrYXVsQHJlZGhhdC5jb20+IGEgPQo9QzM9QTljcml0IDoKPj0yMAoK PiBTdG9yYWdlIGlzIHVzdWFsbHkgdGhlIHNsb3dlc3QgbGluayBpbiB0aGUgY2hhaW4uIEkgcGVy c29uYWxseSBiZWxpZXZlID0KdGhhdCBzcGVuZGluZyB0aGUgbW9uZXkgb24gTlZNZSBkcml2ZXMg bWFrZXMgbW9yZSBzZW5zZSB0aGFuIDQwR2IgPQooZXhjZXB0IFsxXSwgd2hpY2ggaXMgc3VzcGlj aW91c2x5IGNoZWFwISkKPj0yMAo+IFkuCj4gWzFdIGh0dHA6Ly9hLmNvLzRoc0NUcUcgPGh0dHA6 Ly9hLmNvLzRoc0NUcUc+Cgo9Cmh0dHA6Ly9oMjA1NjQud3d3Mi5ocGUuY29tL2hwc2MvZG9jL3B1 YmxpYy9kaXNwbGF5P2RvY0lkPTNEZW1yX25hLWMwNDM3NDA3PQo4CgpJdCdzIHN1cHBvcnRlZCBv biBvbGQgR2VuOCBzZXJ2ZXJzIChHMTAgaXMgY29tbWluZykuIEl0IG11c3QgYmUgY29taW5nID0K ZnJvbSBhbiBhdHRpYy4KCgoKLS1BcHBsZS1NYWlsPV84OEYwNDU1Ri0zQkU4LTQ0MjEtODlBRi1C ODBGOUE5MEFBRjIKQ29udGVudC1UcmFuc2Zlci1FbmNvZGluZzogcXVvdGVkLXByaW50YWJsZQpD b250ZW50LVR5cGU6IHRleHQvaHRtbDsKCWNoYXJzZXQ9dXRmLTgKCjxodG1sPjxoZWFkPjxtZXRh IGh0dHAtZXF1aXY9M0QiQ29udGVudC1UeXBlIiBjb250ZW50PTNEInRleHQvaHRtbCA9CmNoYXJz ZXQ9M0R1dGYtOCI+PC9oZWFkPjxib2R5IHN0eWxlPTNEIndvcmQtd3JhcDogYnJlYWstd29yZDsg PQotd2Via2l0LW5ic3AtbW9kZTogc3BhY2U7IC13ZWJraXQtbGluZS1icmVhazogYWZ0ZXItd2hp dGUtc3BhY2U7IiA9CmNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PGRpdj48YmxvY2txdW90ZSB0 eXBlPTNEImNpdGUiIGNsYXNzPTNEIiI+PGRpdiA9CmNsYXNzPTNEIiI+TGUgOCBhbz1DMz1CQnQg MjAxNyA9QzM9QTAgMDg6NTAsIFlhbml2IEthdWwgJmx0OzxhID0KaHJlZj0zRCJtYWlsdG86eWth dWxAcmVkaGF0LmNvbSIgY2xhc3M9M0QiIj55a2F1bEByZWRoYXQuY29tPC9hPiZndDsgYSA9Cj1D Mz1BOWNyaXQgOjwvZGl2PjxiciBjbGFzcz0zRCJBcHBsZS1pbnRlcmNoYW5nZS1uZXdsaW5lIj48 ZGl2ID0KY2xhc3M9M0QiIj48ZGl2IGRpcj0zRCJsdHIiIGNsYXNzPTNEIiI+PGRpdiBjbGFzcz0z RCJnbWFpbF9leHRyYSI+PGRpdiA9CmNsYXNzPTNEImdtYWlsX3F1b3RlIj48L2Rpdj48L2Rpdj48 L2Rpdj48L2Rpdj48L2Jsb2NrcXVvdGU+PGJyID0KY2xhc3M9M0QiIj48YmxvY2txdW90ZSB0eXBl PTNEImNpdGUiIGNsYXNzPTNEIiI+PGRpdiBjbGFzcz0zRCIiPjxkaXYgPQpkaXI9M0QibHRyIiBj bGFzcz0zRCIiPjxkaXYgY2xhc3M9M0QiZ21haWxfZXh0cmEiPjxkaXYgPQpjbGFzcz0zRCJnbWFp bF9xdW90ZSI+PGRpdiBjbGFzcz0zRCIiPlN0b3JhZ2UgaXMgdXN1YWxseSB0aGUgc2xvd2VzdCA9 CmxpbmsgaW4gdGhlIGNoYWluLiBJIHBlcnNvbmFsbHkgYmVsaWV2ZSB0aGF0IHNwZW5kaW5nIHRo ZSBtb25leSBvbiBOVk1lID0KZHJpdmVzIG1ha2VzIG1vcmUgc2Vuc2UgdGhhbiA0MEdiIChleGNl cHQgWzFdLCB3aGljaCBpcyBzdXNwaWNpb3VzbHkgPQpjaGVhcCEpPC9kaXY+PGRpdiBjbGFzcz0z RCIiPjxiciBjbGFzcz0zRCIiPjwvZGl2PjxkaXYgPQpjbGFzcz0zRCIiPlkuPC9kaXY+PGRpdiBj bGFzcz0zRCIiPlsxXSZuYnNwOzxzcGFuID0Kc3R5bGU9M0QiY29sb3I6cmdiKDE3LDE3LDE3KTtm b250LWZhbWlseTomcXVvdDtBbWF6b24gPQpFbWJlciZxdW90OyxBcmlhbCxzYW5zLXNlcmlmO2Zv bnQtc2l6ZToxM3B4IiBjbGFzcz0zRCIiPjxhID0KaHJlZj0zRCJodHRwOi8vYS5jby80aHNDVHFH IiA9CmNsYXNzPTNEIiI+aHR0cDovL2EuY28vNGhzQ1RxRzwvYT48L3NwYW4+PC9kaXY+PC9kaXY+ PC9kaXY+PC9kaXY+PC9kaXY+PC9iPQpsb2NrcXVvdGU+PGJyIGNsYXNzPTNEIiI+PC9kaXY+PGRp dj48YnIgY2xhc3M9M0QiIj48L2Rpdj48ZGl2PjxhID0KaHJlZj0zRCJodHRwOi8vaDIwNTY0Lnd3 dzIuaHBlLmNvbS9ocHNjL2RvYy9wdWJsaWMvZGlzcGxheT9kb2NJZD0zRGVtcl9uYS09CmMwNDM3 NDA3OCIgPQpjbGFzcz0zRCIiPmh0dHA6Ly9oMjA1NjQud3d3Mi5ocGUuY29tL2hwc2MvZG9jL3B1 YmxpYy9kaXNwbGF5P2RvY0lkPTNEZW1yXz0KbmEtYzA0Mzc0MDc4PC9hPjwvZGl2PjxkaXY+PGJy IGNsYXNzPTNEIiI+PC9kaXY+PGRpdj5JdCdzIHN1cHBvcnRlZCBvbiA9Cm9sZCBHZW44IHNlcnZl cnMgKEcxMCBpcyBjb21taW5nKS4gSXQgbXVzdCBiZSBjb21pbmcgZnJvbSBhbiA9CmF0dGljLjwv ZGl2PjxkaXY+PGJyIGNsYXNzPTNEIiI+PC9kaXY+PGJyIGNsYXNzPTNEIiI+PC9ib2R5PjwvaHRt bD49CgotLUFwcGxlLU1haWw9Xzg4RjA0NTVGLTNCRTgtNDQyMS04OUFGLUI4MEY5QTkwQUFGMi0t Cg== --===============5949613573116717547==-- From johan at kafit.se Tue Aug 8 10:24:37 2017 Content-Type: multipart/mixed; boundary="===============0948125851385482984==" MIME-Version: 1.0 From: Johan Bernhardsson To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Tue, 08 Aug 2017 12:24:31 +0200 Message-ID: <15dc15fdb18.2768.ffe2d6cdf294efc1bb86526523af9945@kafit.se> In-Reply-To: DB6P190MB0280BD0BB439D0C8A4588D00C88A0@DB6P190MB0280.EURP190.PROD.OUTLOOK.COM --===============0948125851385482984== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. ------------15dc15fdd7045fa2768c750b77 Content-Type: text/plain; format=3Dflowed; charset=3D"UTF-8" Content-Transfer-Encoding: quoted-printable On ovirt gluster uses sharding. So all large files are broken up in small= =3D =3D20 pieces on the gluster bricks. /Johan On August 8, 2017 12:19:39 Moacir Ferreira w= =3D rote: > Thanks Johan, you brought "light" into my darkness! I went looking for = =3D the=3D20 > GlusterFS tiering how-to and it looks like quite simple to attach a SSD= =3D as=3D20 > hot tier. For those willing to read about it, go here:=3D20 > http://blog.gluster.org/2016/03/automated-tiering-in-gluster/ > > > Now, I still have a question: VMs are made of very large .qcow2 files. = =3D My=3D20 > understanding is that files in Gluster are kept all together in a singl= =3D e=3D20 > brick. If so, I will not benefit from tiering as a single SSD will not = =3D be=3D20 > big enough to fit all my large VM .qcow2 files. This would not be true = =3D if=3D20 > Gluster can store "blocks" of data that compose a large file spread on=3D= 20 > several bricks. But if I am not wrong, this is one of key differences i= =3D n=3D20 > between GlusterFS and Ceph. Can you comment? > > > Moacir > > > ________________________________ > From: Johan Bernhardsson > Sent: Tuesday, August 8, 2017 7:03 AM > To: Moacir Ferreira; Devin Acosta; users(a)ovirt.org > Subject: Re: [ovirt-users] Good practices > > > You attach the ssd as a hot tier with a gluster command. I don't think = =3D that=3D20 > gdeploy or ovirt gui can do it. > > The gluster docs and redhat docs explains tiering quite good. > > /Johan > > On August 8, 2017 07:06:42 Moacir Ferreira = =3D wrote: > > Hi Devin, > > > Please consider that for the OS I have a RAID 1. Now, lets say I use RA= =3D ID 5=3D20 > to assemble a single disk on each server. In this case, the SSD will no= =3D t=3D20 > make any difference, right? I guess that to be possible to use it, the = =3D SSD=3D20 > should not be part of the RAID 5. In this case I could create a logical= =3D =3D20 > volume made of the RAIDed brick and then extend it using the SSD. I.e.:= =3D =3D20 > Using gdeploy: > > > [disktype] > > jbod > > .... > > [pv1] > > action=3D3Dcreate > > devices=3D3Dsdb, sdc > > wipefs=3D3Dyes > > ignore_vg_erros=3D3Dno > > > [vg1] > > action=3D3Dcreate > > vgname=3D3Dgluster_vg_jbod > > pvname=3D3Dsdb > > ignore_vg_erros=3D3Dno > > > [vg2] > > action=3D3Dextend > > vgname=3D3Dgluster_vg_jbod > > pvname=3D3Dsdc > > ignore_vg_erros=3D3Dno > > > But will Gluster be able to auto-detect and use this SSD brick for tier= =3D ing?=3D20 > Do I have to do some other configurations? Also, as the VM files (.qcow= =3D 2)=3D20 > are quite big will I benefit from tiering? This is wrong and my approac= =3D h=3D20 > should be other? > > > Thanks, > > Moacir > > > ________________________________ > From: Devin Acosta > Sent: Monday, August 7, 2017 7:46 AM > To: Moacir Ferreira; users(a)ovirt.org > Subject: Re: [ovirt-users] Good practices > > > Moacir, > > I have recently installed multiple Red Hat Virtualization hosts for sev= =3D eral=3D20 > different companies, and have dealt with the Red Hat Support Team in de= =3D pth=3D20 > about optimal configuration in regards to setting up GlusterFS most=3D20 > efficiently and I wanted to share with you what I learned. > > In general Red Hat Virtualization team frowns upon using each DISK of t= =3D he=3D20 > system as just a JBOD, sure there is some protection by having the data= =3D =3D20 > replicated, however, the recommendation is to use RAID 6 (preferred) or= =3D =3D20 > RAID-5, or at least RAID-1 at the very least. > > Here is the direct quote from Red Hat when I asked about RAID and Brick= =3D s: > > "A typical Gluster configuration would use RAID underneath the bricks. = =3D RAID=3D20 > 6 is most typical as it gives you 2 disk failure protection, but RAID 5= =3D =3D20 > could be used too. Once you have the RAIDed bricks, you'd then apply th= =3D e=3D20 > desired replication on top of that. The most popular way of doing this=3D= 20 > would be distributed replicated with 2x replication. In general you'll = =3D get=3D20 > better performance with larger bricks. 12 drives is often a sweet spot.= =3D =3D20 > Another option would be to create a separate tier using all SSD=3DE2=3D80= =3D99=3D s.=3DE2=3D80=3D9D > > In order to SSD tiering from my understanding you would need 1 x NVMe d= =3D rive=3D20 > in each server, or 4 x SSD hot tier (it needs to be distributed, replic= =3D ated=3D20 > for the hot tier if not using NVME). So with you only having 1 SSD driv= =3D e in=3D20 > each server, I=3DE2=3D80=3D99d suggest maybe looking into the NVME option. > > Since your using only 3-servers, what I=3DE2=3D80=3D99d probably suggest = is t=3D o do (2=3D20 > Replicas + Arbiter Node), this setup actually doesn=3DE2=3D80=3D99t requi= re t=3D he 3rd=3D20 > server to have big drives at all as it only stores meta-data about the=3D= 20 > files and not actually a full copy. > > Please see the attached document that was given to me by Red Hat to get= =3D =3D20 > more information on this. Hope this information helps you. > > > -- > > Devin Acosta, RHCA, RHVCA > Red Hat Certified Architect > > > On August 6, 2017 at 7:29:29 PM, Moacir Ferreira=3D20 > (moacirferreira(a)hotmail.com) wrote: > > I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 = =3D CPU=3D20 > sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use=3D20 > GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC= =3D and=3D20 > a dual 10Gb NIC. So my intention is to create a loop like a server tria= =3D ngle=3D20 > using the 40Gb NICs for virtualization files (VMs .qcow2) access and to= =3D =3D20 > move VMs around the pod (east /west traffic) while using the 10Gb=3D20 > interfaces for giving services to the outside world (north/south traffi= =3D c). > > > This said, my first question is: How should I deploy GlusterFS in such=3D= 20 > oVirt scenario? My questions are: > > > 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and = =3D then=3D20 > create a GlusterFS using them? > > 2 - Instead, should I create a JBOD array made of all server's disks? > > 3 - What is the best Gluster configuration to provide for HA while not=3D= 20 > consuming too much disk space? > > 4 - Does a oVirt hypervisor pod like I am planning to build, and the=3D20 > virtualization environment, benefits from tiering when using a SSD disk= =3D ?=3D20 > And yes, will Gluster do it by default or I have to configure it to do = =3D so? > > > At the bottom line, what is the good practice for using GlusterFS in sm= =3D all=3D20 > pods for enterprises? > > > You opinion/feedback will be really appreciated! > > Moacir > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > ------------15dc15fdd7045fa2768c750b77 Content-Type: text/html; charset=3D"UTF-8" Content-Transfer-Encoding: quoted-printable

On ovirt gluster uses shard= =3D ing. So all large files are broken up in small pieces on the gluster bricks.
/Johan

On August 8, 2017 12:19:39 Moacir Ferreira <moacirferreira(a)hotmail.com>= =3D ; wrote:

Thanks Johan, you brought "light" into my darkness! I went looking for the GlusterFS tiering how-to and it looks like quite simple t= =3D o attach a SSD as hot tier. For those willing to read about it, go here: http://blog.gluster.org/2016/03/automated-tiering-in-gluster/

Now, I still have a question: VMs are made of very large .qcow2 files. My understanding is that files in Gluster are kept all together in a sing= =3D le brick. If so, I will not benefit from tiering as a single SSD will not be big enough to fit all my large VM .qcow2 files. This would not be true if Gluster can store "blocks" of data that compose a large file spread on several bricks. But if I am not wrong, this is one of key differences in between GlusterFS and Ceph. Can you comment?

Moacir

From: Johan Bernh= ar=3D dsson <johan(a)kafit.se>
Sent: Tuesday, August 8, 2017 7:03 AM
To: Moacir Ferreira; Devin Acosta; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices

You attach the ssd as a hot ti= =3D er with a gluster command. I don't think that gdeploy or ovirt gui can do it= =3D .

The gluster docs and redhat do= =3D cs explains tiering quite good.

/Johan

On August 8, 2017 07:06:42 Moacir Ferreira <moacirferreira(a)hotmail.com> wrote:

Hi Devin,

Please consider that for the OS I have a RAID 1. Now, lets say I use RAID 5 to assemble a single disk on each server. In this case, the SSD wi= =3D ll not make any difference, right? I guess that to be possible to use it, th= =3D e SSD should not be part of the RAID 5. In this case I could create a logical volume made of the RAIDed brick and then extend it using the SSD. I.e.: Using gdeploy:

[disktype]

jbod

....

[pv1]

action=3D3Dcreate

devices=3D3Dsdb, sdc

wipefs=3D3Dyes

ignore_vg_erros=3D3Dno

[vg1]

action=3D3Dcreate

vgname=3D3Dgluster_vg_jbod

pvname=3D3Dsdb

ignore_vg_erros=3D3Dno

[vg2]

action=3D3Dextend

vgname=3D3Dgluster_vg_jbod

pvname=3D3Dsdc

ignore_vg_erros=3D3Dno

But will Gluster be able to auto-detect and use this SSD brick for tiering? Do I have to do some other configurations? Also, as the VM files (.qcow2) are quite big will I benefit from tiering? This is wrong and my approach should be other?

Thanks,

Moacir

From: Devin Acosta <devin(a)pabstatencio.com>
Sent: Monday, August 7, 2017 7:46 AM
To: Moacir Ferreira; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices

Moacir,

I have recently installed multiple Red Hat Virtualiza= =3D tion hosts for several different companies, and have dealt with the Red Hat Support Team in depth about optimal configuration in regards to setting up GlusterFS most efficiently and I wanted to shar= =3D e with you what I learned.

In general Red Hat Virtualization team frowns upon us= =3D ing each DISK of the system as just a JBOD, sure there is some protection by having the data replicated, however, the recommendation is to use RAID 6 (preferred) or RAID-5, or at least RAID-= =3D 1 at the very least.

Here is the direct quote from Red Hat when I asked about RAID and Bricks:

"A typical Gluster configuration would use RA= =3D ID underneath the bricks. RAID 6 is most typical as it gives you 2 disk failure protection, but RAID 5 could be used too. Once you have the RAIDed bricks, you'd then apply the desired replication on top of that. The most popular way of doing this would be distributed replicated with 2= =3D x replication. In general you'll get better performance with larger bricks. 12 drives is often a sweet spot. Another option would be to create a separate tier using all SSD=3DE2=3D80=3D99s.=3DE2=3D80=3D9D

In order to SSD tiering from my understanding you would need 1 x NVMe drive in each server, or 4 x SSD hot tier (it needs to be distributed, replicated for the hot tier if not using NVME). So with you only having 1 SSD drive in each server, I=3DE2=3D80=3D99d= sug=3D gest maybe looking into the NVME option.

Since your using only 3-servers, what I=3DE2=3D80= =3D99d =3D probably suggest is to do (2 Replicas + Arbiter Node), this setup actually doesn=3DE2=3D80=3D99t require the 3rd server to have big drives at all as i= t on=3D ly stores meta-data about the files and not actually a full copy.

Please see the attached document that was given to= =3D me by Red Hat to get more information on this. Hope this information helps you.

--

Devin Acosta, RHCA, RHVCA

Red Hat Certified Architect

On August 6, 2017 at 7:29:29 PM, Moacir Ferreira = =3D (moacirferreira(a)hotmail.com= =3D ) wrote:

I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD= =3D . The idea is to use GlusterFS to provide HA for the VMs. The 3 servers hav= =3D e a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop like a server triangle using the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VMs around the pod (east /west traffic) while using the 10Gb interfaces for giving services = =3D to the outside world (north/south traffic).

This said, my first question is: How should I deploy GlusterFS in such oVirt scenario? My questions are:

1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then create a GlusterFS using them?

2 - Instead, should I create a JBOD array made of all server's disks?<= =3D /p>
3 - What is the best Gluster configuration to provide for HA while not consuming too much disk space?

4 - Does a oVirt hypervisor pod like I am planning to build, and the virtualization environment, benefits from tiering when using a SSD disk? And yes, will Gluster do it by default or I have to configure it to do so= =3D ?

At the bottom line, what is the good practice for using GlusterFS in small pods for enterprises?

You opinion/feedback will be really appreciated!

Moacir

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt= =3D .org/mailman/listinfo/users

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt= =3D .org/mailman/listinfo/users

------------15dc15fdd7045fa2768c750b77-- --===============0948125851385482984== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t MTVkYzE1ZmRkNzA0NWZhMjc2OGM3NTBiNzcKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBmb3Jt YXQ9Zmxvd2VkOyBjaGFyc2V0PSJVVEYtOCIKQ29udGVudC1UcmFuc2Zlci1FbmNvZGluZzogcXVv dGVkLXByaW50YWJsZQoKT24gb3ZpcnQgZ2x1c3RlciB1c2VzIHNoYXJkaW5nLiBTbyBhbGwgbGFy Z2UgZmlsZXMgYXJlIGJyb2tlbiB1cCBpbiBzbWFsbD0KPTIwCnBpZWNlcyBvbiB0aGUgZ2x1c3Rl ciBicmlja3MuCgovSm9oYW4KCgpPbiBBdWd1c3QgOCwgMjAxNyAxMjoxOTozOSBNb2FjaXIgRmVy cmVpcmEgPG1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY29tPiB3PQpyb3RlOgoKPiBUaGFua3MgSm9o YW4sIHlvdSBicm91Z2h0ICJsaWdodCIgaW50byBteSBkYXJrbmVzcyEgSSB3ZW50IGxvb2tpbmcg Zm9yID0KdGhlPTIwCj4gR2x1c3RlckZTIHRpZXJpbmcgaG93LXRvIGFuZCBpdCBsb29rcyBsaWtl IHF1aXRlIHNpbXBsZSB0byBhdHRhY2ggYSBTU0Q9CiBhcz0yMAo+IGhvdCB0aWVyLiBGb3IgdGhv c2Ugd2lsbGluZyB0byByZWFkIGFib3V0IGl0LCBnbyBoZXJlOj0yMAo+IGh0dHA6Ly9ibG9nLmds dXN0ZXIub3JnLzIwMTYvMDMvYXV0b21hdGVkLXRpZXJpbmctaW4tZ2x1c3Rlci8KPgo+Cj4gTm93 LCBJIHN0aWxsIGhhdmUgYSBxdWVzdGlvbjogVk1zIGFyZSBtYWRlIG9mIHZlcnkgbGFyZ2UgLnFj b3cyIGZpbGVzLiA9Ck15PTIwCj4gdW5kZXJzdGFuZGluZyBpcyB0aGF0IGZpbGVzIGluIEdsdXN0 ZXIgYXJlIGtlcHQgYWxsIHRvZ2V0aGVyIGluIGEgc2luZ2w9CmU9MjAKPiBicmljay4gSWYgc28s IEkgd2lsbCBub3QgYmVuZWZpdCBmcm9tIHRpZXJpbmcgYXMgYSBzaW5nbGUgU1NEIHdpbGwgbm90 ID0KYmU9MjAKPiBiaWcgZW5vdWdoIHRvIGZpdCBhbGwgbXkgbGFyZ2UgVk0gLnFjb3cyIGZpbGVz LiBUaGlzIHdvdWxkIG5vdCBiZSB0cnVlID0KaWY9MjAKPiBHbHVzdGVyIGNhbiBzdG9yZSAiYmxv Y2tzIiBvZiBkYXRhIHRoYXQgY29tcG9zZSBhIGxhcmdlIGZpbGUgc3ByZWFkIG9uPTIwCj4gc2V2 ZXJhbCBicmlja3MuIEJ1dCBpZiBJIGFtIG5vdCB3cm9uZywgdGhpcyBpcyBvbmUgb2Yga2V5IGRp ZmZlcmVuY2VzIGk9Cm49MjAKPiBiZXR3ZWVuIEdsdXN0ZXJGUyBhbmQgQ2VwaC4gQ2FuIHlvdSBj b21tZW50Pwo+Cj4KPiBNb2FjaXIKPgo+Cj4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X18KPiBGcm9tOiBKb2hhbiBCZXJuaGFyZHNzb24gPGpvaGFuQGthZml0LnNlPgo+IFNlbnQ6IFR1 ZXNkYXksIEF1Z3VzdCA4LCAyMDE3IDc6MDMgQU0KPiBUbzogTW9hY2lyIEZlcnJlaXJhOyBEZXZp biBBY29zdGE7IHVzZXJzQG92aXJ0Lm9yZwo+IFN1YmplY3Q6IFJlOiBbb3ZpcnQtdXNlcnNdIEdv b2QgcHJhY3RpY2VzCj4KPgo+IFlvdSBhdHRhY2ggdGhlIHNzZCBhcyBhIGhvdCB0aWVyIHdpdGgg YSBnbHVzdGVyIGNvbW1hbmQuIEkgZG9uJ3QgdGhpbmsgPQp0aGF0PTIwCj4gZ2RlcGxveSBvciBv dmlydCBndWkgY2FuIGRvIGl0Lgo+Cj4gVGhlIGdsdXN0ZXIgZG9jcyBhbmQgcmVkaGF0IGRvY3Mg ZXhwbGFpbnMgdGllcmluZyBxdWl0ZSBnb29kLgo+Cj4gL0pvaGFuCj4KPiBPbiBBdWd1c3QgOCwg MjAxNyAwNzowNjo0MiBNb2FjaXIgRmVycmVpcmEgPG1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY29t Pj0KIHdyb3RlOgo+Cj4gSGkgRGV2aW4sCj4KPgo+IFBsZWFzZSBjb25zaWRlciB0aGF0IGZvciB0 aGUgT1MgSSBoYXZlIGEgUkFJRCAxLiBOb3csIGxldHMgc2F5IEkgdXNlIFJBPQpJRCA1PTIwCj4g dG8gYXNzZW1ibGUgYSBzaW5nbGUgZGlzayBvbiBlYWNoIHNlcnZlci4gSW4gdGhpcyBjYXNlLCB0 aGUgU1NEIHdpbGwgbm89CnQ9MjAKPiBtYWtlIGFueSBkaWZmZXJlbmNlLCByaWdodD8gSSBndWVz cyB0aGF0IHRvIGJlIHBvc3NpYmxlIHRvIHVzZSBpdCwgdGhlID0KU1NEPTIwCj4gc2hvdWxkIG5v dCBiZSBwYXJ0IG9mIHRoZSBSQUlEIDUuIEluIHRoaXMgY2FzZSBJIGNvdWxkIGNyZWF0ZSBhIGxv Z2ljYWw9Cj0yMAo+IHZvbHVtZSBtYWRlIG9mIHRoZSBSQUlEZWQgYnJpY2sgYW5kIHRoZW4gZXh0 ZW5kIGl0IHVzaW5nIHRoZSBTU0QuIEkuZS46PQo9MjAKPiBVc2luZyBnZGVwbG95Ogo+Cj4KPiBb ZGlza3R5cGVdCj4KPiBqYm9kCj4KPiAuLi4uCj4KPiBbcHYxXQo+Cj4gYWN0aW9uPTNEY3JlYXRl Cj4KPiBkZXZpY2VzPTNEc2RiLCBzZGMKPgo+IHdpcGVmcz0zRHllcwo+Cj4gaWdub3JlX3ZnX2Vy cm9zPTNEbm8KPgo+Cj4gW3ZnMV0KPgo+IGFjdGlvbj0zRGNyZWF0ZQo+Cj4gdmduYW1lPTNEZ2x1 c3Rlcl92Z19qYm9kCj4KPiBwdm5hbWU9M0RzZGIKPgo+IGlnbm9yZV92Z19lcnJvcz0zRG5vCj4K Pgo+IFt2ZzJdCj4KPiBhY3Rpb249M0RleHRlbmQKPgo+IHZnbmFtZT0zRGdsdXN0ZXJfdmdfamJv ZAo+Cj4gcHZuYW1lPTNEc2RjCj4KPiBpZ25vcmVfdmdfZXJyb3M9M0Rubwo+Cj4KPiBCdXQgd2ls bCBHbHVzdGVyIGJlIGFibGUgdG8gYXV0by1kZXRlY3QgYW5kIHVzZSB0aGlzIFNTRCBicmljayBm b3IgdGllcj0KaW5nPz0yMAo+IERvIEkgaGF2ZSB0byBkbyBzb21lIG90aGVyIGNvbmZpZ3VyYXRp b25zPyBBbHNvLCBhcyB0aGUgVk0gZmlsZXMgKC5xY293PQoyKT0yMAo+IGFyZSBxdWl0ZSBiaWcg d2lsbCBJIGJlbmVmaXQgZnJvbSB0aWVyaW5nPyBUaGlzIGlzIHdyb25nIGFuZCBteSBhcHByb2Fj PQpoPTIwCj4gc2hvdWxkIGJlIG90aGVyPwo+Cj4KPiBUaGFua3MsCj4KPiBNb2FjaXIKPgo+Cj4g X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KPiBGcm9tOiBEZXZpbiBBY29zdGEgPGRl dmluQHBhYnN0YXRlbmNpby5jb20+Cj4gU2VudDogTW9uZGF5LCBBdWd1c3QgNywgMjAxNyA3OjQ2 IEFNCj4gVG86IE1vYWNpciBGZXJyZWlyYTsgdXNlcnNAb3ZpcnQub3JnCj4gU3ViamVjdDogUmU6 IFtvdmlydC11c2Vyc10gR29vZCBwcmFjdGljZXMKPgo+Cj4gTW9hY2lyLAo+Cj4gSSBoYXZlIHJl Y2VudGx5IGluc3RhbGxlZCBtdWx0aXBsZSBSZWQgSGF0IFZpcnR1YWxpemF0aW9uIGhvc3RzIGZv ciBzZXY9CmVyYWw9MjAKPiBkaWZmZXJlbnQgY29tcGFuaWVzLCBhbmQgaGF2ZSBkZWFsdCB3aXRo IHRoZSBSZWQgSGF0IFN1cHBvcnQgVGVhbSBpbiBkZT0KcHRoPTIwCj4gYWJvdXQgb3B0aW1hbCBj b25maWd1cmF0aW9uIGluIHJlZ2FyZHMgdG8gc2V0dGluZyB1cCBHbHVzdGVyRlMgbW9zdD0yMAo+ IGVmZmljaWVudGx5IGFuZCBJIHdhbnRlZCB0byBzaGFyZSB3aXRoIHlvdSB3aGF0IEkgbGVhcm5l ZC4KPgo+IEluIGdlbmVyYWwgUmVkIEhhdCBWaXJ0dWFsaXphdGlvbiB0ZWFtIGZyb3ducyB1cG9u IHVzaW5nIGVhY2ggRElTSyBvZiB0PQpoZT0yMAo+IHN5c3RlbSBhcyBqdXN0IGEgSkJPRCwgc3Vy ZSB0aGVyZSBpcyBzb21lIHByb3RlY3Rpb24gYnkgaGF2aW5nIHRoZSBkYXRhPQo9MjAKPiByZXBs aWNhdGVkLCBob3dldmVyLCB0aGUgcmVjb21tZW5kYXRpb24gaXMgdG8gdXNlIFJBSUQgNiAocHJl ZmVycmVkKSBvcj0KPTIwCj4gUkFJRC01LCBvciBhdCBsZWFzdCBSQUlELTEgYXQgdGhlIHZlcnkg bGVhc3QuCj4KPiBIZXJlIGlzIHRoZSBkaXJlY3QgcXVvdGUgZnJvbSBSZWQgSGF0IHdoZW4gSSBh c2tlZCBhYm91dCBSQUlEIGFuZCBCcmljaz0KczoKPgo+ICJBIHR5cGljYWwgR2x1c3RlciBjb25m aWd1cmF0aW9uIHdvdWxkIHVzZSBSQUlEIHVuZGVybmVhdGggdGhlIGJyaWNrcy4gPQpSQUlEPTIw Cj4gNiBpcyBtb3N0IHR5cGljYWwgYXMgaXQgZ2l2ZXMgeW91IDIgZGlzayBmYWlsdXJlIHByb3Rl Y3Rpb24sIGJ1dCBSQUlEIDU9Cj0yMAo+IGNvdWxkIGJlIHVzZWQgdG9vLiBPbmNlIHlvdSBoYXZl IHRoZSBSQUlEZWQgYnJpY2tzLCB5b3UnZCB0aGVuIGFwcGx5IHRoPQplPTIwCj4gZGVzaXJlZCBy ZXBsaWNhdGlvbiBvbiB0b3Agb2YgdGhhdC4gVGhlIG1vc3QgcG9wdWxhciB3YXkgb2YgZG9pbmcg dGhpcz0yMAo+IHdvdWxkIGJlIGRpc3RyaWJ1dGVkIHJlcGxpY2F0ZWQgd2l0aCAyeCByZXBsaWNh dGlvbi4gSW4gZ2VuZXJhbCB5b3UnbGwgPQpnZXQ9MjAKPiBiZXR0ZXIgcGVyZm9ybWFuY2Ugd2l0 aCBsYXJnZXIgYnJpY2tzLiAxMiBkcml2ZXMgaXMgb2Z0ZW4gYSBzd2VldCBzcG90Lj0KPTIwCj4g QW5vdGhlciBvcHRpb24gd291bGQgYmUgdG8gY3JlYXRlIGEgc2VwYXJhdGUgdGllciB1c2luZyBh bGwgU1NEPUUyPTgwPTk5PQpzLj1FMj04MD05RAo+Cj4gSW4gb3JkZXIgdG8gU1NEIHRpZXJpbmcg ZnJvbSBteSB1bmRlcnN0YW5kaW5nIHlvdSB3b3VsZCBuZWVkIDEgeCBOVk1lIGQ9CnJpdmU9MjAK PiBpbiBlYWNoIHNlcnZlciwgb3IgNCB4IFNTRCBob3QgdGllciAoaXQgbmVlZHMgdG8gYmUgZGlz dHJpYnV0ZWQsIHJlcGxpYz0KYXRlZD0yMAo+IGZvciB0aGUgaG90IHRpZXIgaWYgbm90IHVzaW5n IE5WTUUpLiBTbyB3aXRoIHlvdSBvbmx5IGhhdmluZyAxIFNTRCBkcml2PQplIGluPTIwCj4gZWFj aCBzZXJ2ZXIsIEk9RTI9ODA9OTlkIHN1Z2dlc3QgbWF5YmUgbG9va2luZyBpbnRvIHRoZSBOVk1F IG9wdGlvbi4KPgo+IFNpbmNlIHlvdXIgdXNpbmcgb25seSAzLXNlcnZlcnMsIHdoYXQgST1FMj04 MD05OWQgcHJvYmFibHkgc3VnZ2VzdCBpcyB0PQpvIGRvICgyPTIwCj4gUmVwbGljYXMgKyBBcmJp dGVyIE5vZGUpLCB0aGlzIHNldHVwIGFjdHVhbGx5IGRvZXNuPUUyPTgwPTk5dCByZXF1aXJlIHQ9 CmhlIDNyZD0yMAo+IHNlcnZlciB0byBoYXZlIGJpZyBkcml2ZXMgYXQgYWxsIGFzIGl0IG9ubHkg c3RvcmVzIG1ldGEtZGF0YSBhYm91dCB0aGU9MjAKPiBmaWxlcyBhbmQgbm90IGFjdHVhbGx5IGEg ZnVsbCBjb3B5Lgo+Cj4gUGxlYXNlIHNlZSB0aGUgYXR0YWNoZWQgZG9jdW1lbnQgdGhhdCB3YXMg Z2l2ZW4gdG8gbWUgYnkgUmVkIEhhdCB0byBnZXQ9Cj0yMAo+IG1vcmUgaW5mb3JtYXRpb24gb24g dGhpcy4gSG9wZSB0aGlzIGluZm9ybWF0aW9uIGhlbHBzIHlvdS4KPgo+Cj4gLS0KPgo+IERldmlu IEFjb3N0YSwgUkhDQSwgUkhWQ0EKPiBSZWQgSGF0IENlcnRpZmllZCBBcmNoaXRlY3QKPgo+Cj4g T24gQXVndXN0IDYsIDIwMTcgYXQgNzoyOToyOSBQTSwgTW9hY2lyIEZlcnJlaXJhPTIwCj4gKG1v YWNpcmZlcnJlaXJhQGhvdG1haWwuY29tPG1haWx0bzptb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNv bT4pIHdyb3RlOgo+Cj4gSSBhbSB3aWxsaW5nIHRvIGFzc2VtYmxlIGEgb1ZpcnQgInBvZCIsIG1h ZGUgb2YgMyBzZXJ2ZXJzLCBlYWNoIHdpdGggMiA9CkNQVT0yMAo+IHNvY2tldHMgb2YgMTIgY29y ZXMsIDI1NkdCIFJBTSwgNyBIREQgMTBLLCAxIFNTRC4gVGhlIGlkZWEgaXMgdG8gdXNlPTIwCj4g R2x1c3RlckZTIHRvIHByb3ZpZGUgSEEgZm9yIHRoZSBWTXMuIFRoZSAzIHNlcnZlcnMgaGF2ZSBh IGR1YWwgNDBHYiBOSUM9CiBhbmQ9MjAKPiBhIGR1YWwgMTBHYiBOSUMuIFNvIG15IGludGVudGlv biBpcyB0byBjcmVhdGUgYSBsb29wIGxpa2UgYSBzZXJ2ZXIgdHJpYT0KbmdsZT0yMAo+IHVzaW5n IHRoZSA0MEdiIE5JQ3MgZm9yIHZpcnR1YWxpemF0aW9uIGZpbGVzIChWTXMgLnFjb3cyKSBhY2Nl c3MgYW5kIHRvPQo9MjAKPiBtb3ZlIFZNcyBhcm91bmQgdGhlIHBvZCAoZWFzdCAvd2VzdCB0cmFm ZmljKSB3aGlsZSB1c2luZyB0aGUgMTBHYj0yMAo+IGludGVyZmFjZXMgZm9yIGdpdmluZyBzZXJ2 aWNlcyB0byB0aGUgb3V0c2lkZSB3b3JsZCAobm9ydGgvc291dGggdHJhZmZpPQpjKS4KPgo+Cj4g VGhpcyBzYWlkLCBteSBmaXJzdCBxdWVzdGlvbiBpczogSG93IHNob3VsZCBJIGRlcGxveSBHbHVz dGVyRlMgaW4gc3VjaD0yMAo+IG9WaXJ0IHNjZW5hcmlvPyBNeSBxdWVzdGlvbnMgYXJlOgo+Cj4K PiAxIC0gU2hvdWxkIEkgY3JlYXRlIDMgUkFJRCAoaS5lLjogUkFJRCA1KSwgb25lIG9uIGVhY2gg b1ZpcnQgbm9kZSwgYW5kID0KdGhlbj0yMAo+IGNyZWF0ZSBhIEdsdXN0ZXJGUyB1c2luZyB0aGVt Pwo+Cj4gMiAtIEluc3RlYWQsIHNob3VsZCBJIGNyZWF0ZSBhIEpCT0QgYXJyYXkgbWFkZSBvZiBh bGwgc2VydmVyJ3MgZGlza3M/Cj4KPiAzIC0gV2hhdCBpcyB0aGUgYmVzdCBHbHVzdGVyIGNvbmZp Z3VyYXRpb24gdG8gcHJvdmlkZSBmb3IgSEEgd2hpbGUgbm90PTIwCj4gY29uc3VtaW5nIHRvbyBt dWNoIGRpc2sgc3BhY2U/Cj4KPiA0IC0gRG9lcyBhIG9WaXJ0IGh5cGVydmlzb3IgcG9kIGxpa2Ug SSBhbSBwbGFubmluZyB0byBidWlsZCwgYW5kIHRoZT0yMAo+IHZpcnR1YWxpemF0aW9uIGVudmly b25tZW50LCBiZW5lZml0cyBmcm9tIHRpZXJpbmcgd2hlbiB1c2luZyBhIFNTRCBkaXNrPQo/PTIw Cj4gQW5kIHllcywgd2lsbCBHbHVzdGVyIGRvIGl0IGJ5IGRlZmF1bHQgb3IgSSBoYXZlIHRvIGNv bmZpZ3VyZSBpdCB0byBkbyA9CnNvPwo+Cj4KPiBBdCB0aGUgYm90dG9tIGxpbmUsIHdoYXQgaXMg dGhlIGdvb2QgcHJhY3RpY2UgZm9yIHVzaW5nIEdsdXN0ZXJGUyBpbiBzbT0KYWxsPTIwCj4gcG9k cyBmb3IgZW50ZXJwcmlzZXM/Cj4KPgo+IFlvdSBvcGluaW9uL2ZlZWRiYWNrIHdpbGwgYmUgcmVh bGx5IGFwcHJlY2lhdGVkIQo+Cj4gTW9hY2lyCj4KPiBfX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fXwo+IFVzZXJzIG1haWxpbmcgbGlzdAo+IFVzZXJzQG92aXJ0 Lm9yZzxtYWlsdG86VXNlcnNAb3ZpcnQub3JnPgo+IGh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFp bG1hbi9saXN0aW5mby91c2Vycwo+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fCj4gVXNlcnMgbWFpbGluZyBsaXN0Cj4gVXNlcnNAb3ZpcnQub3JnPG1haWx0 bzpVc2VycyU0MG92aXJ0Lm9yZz4KPiBodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlz dGluZm8vdXNlcnMKPgoKLS0tLS0tLS0tLS0tMTVkYzE1ZmRkNzA0NWZhMjc2OGM3NTBiNzcKQ29u dGVudC1UeXBlOiB0ZXh0L2h0bWw7IGNoYXJzZXQ9IlVURi04IgpDb250ZW50LVRyYW5zZmVyLUVu Y29kaW5nOiBxdW90ZWQtcHJpbnRhYmxlCgo8aHRtbD4KPGhlYWQ+CjxzdHlsZSB0eXBlPTNEInRl eHQvY3NzIiBzdHlsZT0zRCJkaXNwbGF5Om5vbmU7Ij48IS0tIFAKe21hcmdpbi10b3A6MDttYXJn aW4tYm90dG9tOjA7fSAtLT48L3N0eWxlPgoKCjwvaGVhZD4KPGJvZHk+CjxkaXYgc3R5bGU9M0Qi Y29sb3I6IGJsYWNrOyI+CjxkaXYgc3R5bGU9M0QiY29sb3I6IGJsYWNrOyI+CjxwIHN0eWxlPTNE Im1hcmdpbjogMCAwIDFlbSAwOyBjb2xvcjogYmxhY2s7Ij5PbiBvdmlydCBnbHVzdGVyIHVzZXMg c2hhcmQ9CmluZy4KU28gYWxsIGxhcmdlIGZpbGVzIGFyZSBicm9rZW4gdXAgaW4gc21hbGwgcGll Y2VzIG9uIHRoZSBnbHVzdGVyIGJyaWNrcy48Lz0KcD4KPHAgc3R5bGU9M0QibWFyZ2luOiAwIDAg MWVtIDA7IGNvbG9yOiBibGFjazsiPi9Kb2hhbjwvcD4KPC9kaXY+CjxkaXYgc3R5bGU9M0QiY29s b3I6IGJsYWNrOyI+CjxwCnN0eWxlPTNEImNvbG9yOiBibGFjazsgZm9udC1zaXplOiAxMHB0OyBm b250LWZhbWlseTogQXJpYWwsIHNhbnMtc2VyaWY7IG09CmFyZ2luOiAxMHB0IDA7Ij5PbgpBdWd1 c3QgOCwgMjAxNyAxMjoxOTozOSBNb2FjaXIgRmVycmVpcmEgJmx0O21vYWNpcmZlcnJlaXJhQGhv dG1haWwuY29tJmd0PQo7Cndyb3RlOjwvcD4KPGJsb2NrcXVvdGUgdHlwZT0zRCJjaXRlIiBjbGFz cz0zRCJnbWFpbF9xdW90ZSIKc3R5bGU9M0QibWFyZ2luOiAwIDAgMCAwLjc1ZXg7IGJvcmRlci1s ZWZ0OiAxcHggc29saWQgIzgwODA4MDsgcGFkZGluZy1sZT0KZnQ6IDAuNzVleDsiPgo8ZGl2IGlk PTNEImRpdnRhZ2RlZmF1bHR3cmFwcGVyIgpzdHlsZT0zRCJmb250LXNpemU6MTJwdDtjb2xvcjoj MDAwMDAwO2ZvbnQtZmFtaWx5OkNhbGlicmksSGVsdmV0aWNhLHNhbnMtPQpzZXJpZjsiCmRpcj0z RCJsdHIiPgo8cD5UaGFua3MgSm9oYW4sIHlvdSBicm91Z2h0ICZxdW90O2xpZ2h0JnF1b3Q7IGlu dG8gbXkgZGFya25lc3MhIEkgd2VudApsb29raW5nIGZvciB0aGUgR2x1c3RlckZTIHRpZXJpbmcg aG93LXRvIGFuZCBpdCBsb29rcyBsaWtlIHF1aXRlIHNpbXBsZSB0PQpvCmF0dGFjaCBhIFNTRCBh cyBob3QgdGllci4gRm9yIHRob3NlIHdpbGxpbmcgdG8gcmVhZCBhYm91dCBpdCwgZ28gaGVyZToK PGEgaHJlZj0zRCJodHRwOi8vYmxvZy5nbHVzdGVyLm9yZy8yMDE2LzAzL2F1dG9tYXRlZC10aWVy aW5nLWluLWdsdXN0ZXIvIgpjbGFzcz0zRCJPV0FBdXRvTGluayIgaWQ9M0QiTFBsbms5MzkzNDMi IHByZXZpZXdyZW1vdmVkPTNEInRydWUiPgpodHRwOi8vYmxvZy5nbHVzdGVyLm9yZy8yMDE2LzAz L2F1dG9tYXRlZC10aWVyaW5nLWluLWdsdXN0ZXIvPC9hPjwvcD4KPHA+PGJyPgo8L3A+CjxwPk5v dywgSSBzdGlsbCBoYXZlIGEgcXVlc3Rpb246IFZNcyBhcmUgbWFkZSBvZiB2ZXJ5IGxhcmdlIC5x Y293MiBmaWxlcy4KTXkgdW5kZXJzdGFuZGluZyBpcyB0aGF0IGZpbGVzIGluIEdsdXN0ZXIgYXJl IGtlcHQgYWxsIHRvZ2V0aGVyIGluIGEgc2luZz0KbGUKYnJpY2suIElmIHNvLCBJIHdpbGwgbm90 IGJlbmVmaXQgZnJvbSB0aWVyaW5nIGFzIGEgc2luZ2xlIFNTRCB3aWxsIG5vdCBiZQpiaWcgZW5v dWdoIHRvIGZpdCBhbGwgbXkgbGFyZ2UgVk0KIC5xY293MiBmaWxlcy4gVGhpcyB3b3VsZCBub3Qg YmUgdHJ1ZSBpZiBHbHVzdGVyIGNhbiBzdG9yZQomcXVvdDtibG9ja3MmcXVvdDsgb2YgZGF0YSB0 aGF0IGNvbXBvc2UgYSBsYXJnZSBmaWxlIHNwcmVhZCBvbiBzZXZlcmFsCmJyaWNrcy4gQnV0IGlm IEkgYW0gbm90IHdyb25nLCB0aGlzIGlzIG9uZSBvZiBrZXkgZGlmZmVyZW5jZXMgaW4gYmV0d2Vl bgpHbHVzdGVyRlMgYW5kIENlcGguIENhbiB5b3UgY29tbWVudD88L3A+CjxwPjxicj4KPC9wPgo8 cD5Nb2FjaXI8YnI+CjwvcD4KPGJyPgo8YnI+CjxkaXYgc3R5bGU9M0QiY29sb3I6IHJnYig0OSwg NTUsIDU3KTsiPgo8aHIgdGFiaW5kZXg9M0QiLTEiIHN0eWxlPTNEImRpc3BsYXk6aW5saW5lLWJs b2NrOyB3aWR0aDo5OCUiPgo8ZGl2IGlkPTNEImRpdlJwbHlGd2RNc2ciIGRpcj0zRCJsdHIiPjxm b250IHN0eWxlPTNEImZvbnQtc2l6ZToxMXB0IgpmYWNlPTNEIkNhbGlicmksIHNhbnMtc2VyaWYi IGNvbG9yPTNEIiMwMDAwMDAiPjxiPkZyb206PC9iPiBKb2hhbiBCZXJuaGFyPQpkc3NvbgombHQ7 am9oYW5Aa2FmaXQuc2UmZ3Q7PGJyPgo8Yj5TZW50OjwvYj4gVHVlc2RheSwgQXVndXN0IDgsIDIw MTcgNzowMyBBTTxicj4KPGI+VG86PC9iPiBNb2FjaXIgRmVycmVpcmE7IERldmluIEFjb3N0YTsg dXNlcnNAb3ZpcnQub3JnPGJyPgo8Yj5TdWJqZWN0OjwvYj4gUmU6IFtvdmlydC11c2Vyc10gR29v ZCBwcmFjdGljZXM8L2ZvbnQ+CjxkaXY+Jm5ic3A7PC9kaXY+CjwvZGl2Pgo8ZGl2Pgo8ZGl2IHN0 eWxlPTNEImNvbG9yOmJsYWNrIj4KPGRpdiBzdHlsZT0zRCJjb2xvcjpibGFjayI+CjxwIHN0eWxl PTNEIm1hcmdpbjowIDAgMWVtIDA7IGNvbG9yOmJsYWNrIj5Zb3UgYXR0YWNoIHRoZSBzc2QgYXMg YSBob3QgdGk9CmVyCndpdGggYSBnbHVzdGVyIGNvbW1hbmQuIEkgZG9uJ3QgdGhpbmsgdGhhdCBn ZGVwbG95IG9yIG92aXJ0IGd1aSBjYW4gZG8gaXQ9Ci48L3A+CjxwIHN0eWxlPTNEIm1hcmdpbjow IDAgMWVtIDA7IGNvbG9yOmJsYWNrIj5UaGUgZ2x1c3RlciBkb2NzIGFuZCByZWRoYXQgZG89CmNz CmV4cGxhaW5zIHRpZXJpbmcgcXVpdGUgZ29vZC48L3A+CjxwIHN0eWxlPTNEIm1hcmdpbjowIDAg MWVtIDA7IGNvbG9yOmJsYWNrIj4vSm9oYW48L3A+CjwvZGl2Pgo8ZGl2IHN0eWxlPTNEImNvbG9y OmJsYWNrIj4KPHAKc3R5bGU9M0QiY29sb3I6YmxhY2s7IGZvbnQtc2l6ZToxMHB0OyBmb250LWZh bWlseTpBcmlhbCxzYW5zLXNlcmlmOyBtYXJnaT0KbjoxMHB0IDAiPgpPbiBBdWd1c3QgOCwgMjAx NyAwNzowNjo0MiBNb2FjaXIgRmVycmVpcmEKJmx0O21vYWNpcmZlcnJlaXJhQGhvdG1haWwuY29t Jmd0OyB3cm90ZTo8L3A+CjxibG9ja3F1b3RlIHR5cGU9M0QiY2l0ZSIgY2xhc3M9M0QiZ21haWxf cXVvdGUiCnN0eWxlPTNEIm1hcmdpbjowIDAgMCAwLjc1ZXg7IGJvcmRlci1sZWZ0OjFweCBzb2xp ZCAjODA4MDgwOyBwYWRkaW5nLWxlZnQ9CjowLjc1ZXgiPgo8ZGl2IGlkPTNEImRpdnRhZ2RlZmF1 bHR3cmFwcGVyIiBkaXI9M0QibHRyIgpzdHlsZT0zRCJmb250LXNpemU6MTJwdDsgY29sb3I6IzAw MDAwMDsgZm9udC1mYW1pbHk6Q2FsaWJyaSxIZWx2ZXRpY2Esc2FuPQpzLXNlcmlmIj4KPHA+SGkg RGV2aW4sPC9wPgo8cD48YnI+CjwvcD4KPHA+UGxlYXNlIGNvbnNpZGVyIHRoYXQgZm9yIHRoZSBP UyBJIGhhdmUgYSBSQUlEIDEuIE5vdywgbGV0cyBzYXkgSSB1c2UKUkFJRCA1IHRvIGFzc2VtYmxl IGEgc2luZ2xlIGRpc2sgb24gZWFjaCBzZXJ2ZXIuIEluIHRoaXMgY2FzZSwgdGhlIFNTRCB3aT0K bGwKbm90IG1ha2UgYW55IGRpZmZlcmVuY2UsIHJpZ2h0PyBJIGd1ZXNzIHRoYXQgdG8gYmUgcG9z c2libGUgdG8gdXNlIGl0LCB0aD0KZQpTU0Qgc2hvdWxkIG5vdCBiZSBwYXJ0IG9mIHRoZSBSQUlE CiA1LiBJbiB0aGlzIGNhc2UgSSBjb3VsZCBjcmVhdGUgYSBsb2dpY2FsIHZvbHVtZSBtYWRlIG9m IHRoZSBSQUlEZWQgYnJpY2sKYW5kIHRoZW4gZXh0ZW5kIGl0IHVzaW5nIHRoZSBTU0QuIEkuZS46 IFVzaW5nIGdkZXBsb3k6PC9wPgo8cD48YnI+CjwvcD4KPHA+W2Rpc2t0eXBlXTwvcD4KPHA+amJv ZDwvcD4KPHA+Li4uLjwvcD4KPHA+W3B2MV08L3A+CjxwPmFjdGlvbj0zRGNyZWF0ZTwvcD4KPHA+ ZGV2aWNlcz0zRHNkYiwgc2RjPC9wPgo8cD53aXBlZnM9M0R5ZXM8L3A+CjxwPjwvcD4KPHA+aWdu b3JlX3ZnX2Vycm9zPTNEbm88YnI+CjwvcD4KPHA+PGJyPgo8L3A+CjxwPjwvcD4KPHA+W3ZnMV08 L3A+CjxwPjxiPmFjdGlvbj0zRGNyZWF0ZTwvYj48L3A+CjxwPnZnbmFtZT0zRGdsdXN0ZXJfdmdf amJvZDwvcD4KPHA+cHZuYW1lPTNEc2RiPC9wPgo8cD5pZ25vcmVfdmdfZXJyb3M9M0Rubzxicj4K PC9wPgo8cD48YnI+CjwvcD4KPHA+W3ZnMl08L3A+CjxwPjxiPmFjdGlvbj0zRGV4dGVuZDwvYj48 L3A+CjxwPnZnbmFtZT0zRGdsdXN0ZXJfdmdfamJvZDwvcD4KPHA+cHZuYW1lPTNEc2RjPC9wPgo8 cD5pZ25vcmVfdmdfZXJyb3M9M0Rubzxicj4KPC9wPgo8YnI+CjxwPkJ1dCB3aWxsIEdsdXN0ZXIg YmUgYWJsZSB0byBhdXRvLWRldGVjdCBhbmQgdXNlIHRoaXMgU1NEIGJyaWNrIGZvcgp0aWVyaW5n PyBEbyBJIGhhdmUgdG8gZG8gc29tZSBvdGhlciBjb25maWd1cmF0aW9ucz8gQWxzbywgYXMgdGhl IFZNIGZpbGVzCigucWNvdzIpIGFyZSBxdWl0ZSBiaWcgd2lsbCBJIGJlbmVmaXQgZnJvbSB0aWVy aW5nPyBUaGlzIGlzIHdyb25nIGFuZCBteQphcHByb2FjaCBzaG91bGQgYmUgb3RoZXI/PC9wPgo8 cD48YnI+CjwvcD4KPHA+VGhhbmtzLDwvcD4KPHA+TW9hY2lyPGJyPgo8L3A+CjxwPjxicj4KPC9w Pgo8YnI+CjxkaXYgc3R5bGU9M0QiY29sb3I6cmdiKDQ5LDU1LDU3KSI+CjxociB0YWJpbmRleD0z RCItMSIgc3R5bGU9M0QiZGlzcGxheTppbmxpbmUtYmxvY2s7IHdpZHRoOjk4JSI+CjxkaXYgaWQ9 M0QiZGl2UnBseUZ3ZE1zZyIgZGlyPTNEImx0ciI+PGZvbnQgc3R5bGU9M0QiZm9udC1zaXplOjEx cHQiCmZhY2U9M0QiQ2FsaWJyaSwgc2Fucy1zZXJpZiIgY29sb3I9M0QiIzAwMDAwMCI+PGI+RnJv bTo8L2I+IERldmluIEFjb3N0YQombHQ7ZGV2aW5AcGFic3RhdGVuY2lvLmNvbSZndDs8YnI+Cjxi PlNlbnQ6PC9iPiBNb25kYXksIEF1Z3VzdCA3LCAyMDE3IDc6NDYgQU08YnI+CjxiPlRvOjwvYj4g TW9hY2lyIEZlcnJlaXJhOyB1c2Vyc0BvdmlydC5vcmc8YnI+CjxiPlN1YmplY3Q6PC9iPiBSZTog W292aXJ0LXVzZXJzXSBHb29kIHByYWN0aWNlczwvZm9udD4KPGRpdj4mbmJzcDs8L2Rpdj4KPC9k aXY+CjxkaXY+CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9M0QiY29sb3I6cmdi KDAsMCwwKTsgbWFyZ2luOjBweCI+PGZvbnQKZmFjZT0zRCJJbnB1dCBNb25vIj48YnI+CjwvZm9u dD48L2Rpdj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJjb2xvcjpyZ2Io MCwwLDApOyBtYXJnaW46MHB4Ij48Zm9udApmYWNlPTNEIklucHV0IE1vbm8iPk1vYWNpciw8L2Zv bnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9M0QiY29sb3I6cmdi KDAsMCwwKTsgbWFyZ2luOjBweCI+PGZvbnQKZmFjZT0zRCJJbnB1dCBNb25vIj48YnI+CjwvZm9u dD48L2Rpdj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJjb2xvcjpyZ2Io MCwwLDApOyBtYXJnaW46MHB4Ij48Zm9udApmYWNlPTNEIklucHV0IE1vbm8iPkkgaGF2ZSByZWNl bnRseSBpbnN0YWxsZWQgbXVsdGlwbGUgUmVkIEhhdCBWaXJ0dWFsaXphPQp0aW9uCmhvc3RzIGZv ciBzZXZlcmFsIGRpZmZlcmVudCBjb21wYW5pZXMsIGFuZCBoYXZlIGRlYWx0IHdpdGggdGhlIFJl ZCBIYXQKU3VwcG9ydCBUZWFtIGluIGRlcHRoIGFib3V0IG9wdGltYWwgY29uZmlndXJhdGlvbgog aW4gcmVnYXJkcyB0byBzZXR0aW5nIHVwIEdsdXN0ZXJGUyBtb3N0IGVmZmljaWVudGx5IGFuZCBJ IHdhbnRlZCB0byBzaGFyPQplCndpdGggeW91IHdoYXQgSSBsZWFybmVkLjwvZm9udD48L2Rpdj4K PGRpdiBpZD0zRCJibG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJjb2xvcjpyZ2IoMCwwLDApOyBt YXJnaW46MHB4Ij48Zm9udApmYWNlPTNEIklucHV0IE1vbm8iPjxicj4KPC9mb250PjwvZGl2Pgo8 ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEImNvbG9yOnJnYigwLDAsMCk7IG1h cmdpbjowcHgiPjxmb250CmZhY2U9M0QiSW5wdXQgTW9ubyI+SW4gZ2VuZXJhbCBSZWQgSGF0IFZp cnR1YWxpemF0aW9uIHRlYW0gZnJvd25zIHVwb24gdXM9CmluZwplYWNoIERJU0sgb2YgdGhlIHN5 c3RlbSBhcyBqdXN0IGEgSkJPRCwgc3VyZSB0aGVyZSBpcyBzb21lIHByb3RlY3Rpb24gYnkKaGF2 aW5nIHRoZSBkYXRhIHJlcGxpY2F0ZWQsIGhvd2V2ZXIsIHRoZQogcmVjb21tZW5kYXRpb24gaXMg dG8gdXNlIFJBSUQgNiAocHJlZmVycmVkKSBvciBSQUlELTUsIG9yIGF0IGxlYXN0IFJBSUQtPQox CmF0IHRoZSB2ZXJ5IGxlYXN0LjwvZm9udD48L2Rpdj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21m b250IiBzdHlsZT0zRCJjb2xvcjpyZ2IoMCwwLDApOyBtYXJnaW46MHB4Ij48Zm9udApmYWNlPTNE IklucHV0IE1vbm8iPjxicj4KPC9mb250PjwvZGl2Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZv bnQiIHN0eWxlPTNEIm1hcmdpbjowcHgiPjxmb250IGZhY2U9M0QiSW5wdXQgTW9uPQpvIj5IZXJl CmlzIHRoZSZuYnNwO2RpcmVjdCBxdW90ZSBmcm9tIFJlZCBIYXQgd2hlbiBJIGFza2VkIGFib3V0 IFJBSUQgYW5kCkJyaWNrczo8L2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9tZm9u dCIgc3R5bGU9M0QibWFyZ2luOjBweCI+PGZvbnQgZmFjZT0zRCJJbnB1dCBNb249Cm8iPjxpPjxi cj4KPC9pPjwvZm9udD48L2Rpdj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21mb250IiBzdHlsZT0z RCJtYXJnaW46MHB4Ij48Zm9udApmYWNlPTNEIklucHV0IE1vbm8iPjxpPiZxdW90O0EgdHlwaWNh bCBHbHVzdGVyIGNvbmZpZ3VyYXRpb24gd291bGQgdXNlIFJBPQpJRAp1bmRlcm5lYXRoIHRoZSBi cmlja3MuIFJBSUQgNiBpcyBtb3N0IHR5cGljYWwgYXMgaXQgZ2l2ZXMgeW91IDIgZGlzawpmYWls dXJlIHByb3RlY3Rpb24sIGJ1dCBSQUlEIDUgY291bGQgYmUgdXNlZCB0b28uJm5ic3A7T25jZSB5 b3UgaGF2ZSB0aGUKIFJBSURlZCBicmlja3MsIHlvdSdkIHRoZW4gYXBwbHkgdGhlIGRlc2lyZWQg cmVwbGljYXRpb24gb24gdG9wIG9mIHRoYXQuClRoZSBtb3N0IHBvcHVsYXIgd2F5IG9mIGRvaW5n IHRoaXMgd291bGQgYmUgZGlzdHJpYnV0ZWQgcmVwbGljYXRlZCB3aXRoIDI9CngKcmVwbGljYXRp b24uJm5ic3A7SW4gZ2VuZXJhbCB5b3UnbGwgZ2V0IGJldHRlciBwZXJmb3JtYW5jZSB3aXRoIGxh cmdlcgpicmlja3MuIDEyIGRyaXZlcyBpcyBvZnRlbiBhIHN3ZWV0IHNwb3QuIEFub3RoZXIKIG9w dGlvbiB3b3VsZCBiZSB0byBjcmVhdGUgYSBzZXBhcmF0ZSB0aWVyIHVzaW5nIGFsbApTU0Q9RTI9 ODA9OTlzLj1FMj04MD05RCZuYnNwOzwvaT48L2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3Bf Y3VzdG9tZm9udCIgc3R5bGU9M0QibWFyZ2luOjBweCI+PGJyPgo8L2Rpdj4KPGRpdiBpZD0zRCJi bG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJtYXJnaW46MHB4Ij48Zm9udCBmYWNlPTNEIklucHV0 IE1vbj0KbyI+PGk+SW4Kb3JkZXIgdG8gU1NEJm5ic3A7dGllcmluZyBmcm9tIG15IHVuZGVyc3Rh bmRpbmcgeW91IHdvdWxkIG5lZWQgMSB4IE5WTWUKZHJpdmUgaW4gZWFjaCBzZXJ2ZXIsIG9yIDQg eCBTU0QgaG90IHRpZXIgKGl0IG5lZWRzIHRvIGJlIGRpc3RyaWJ1dGVkLApyZXBsaWNhdGVkIGZv ciB0aGUgaG90IHRpZXIgaWYgbm90IHVzaW5nIE5WTUUpLgogU28gd2l0aCB5b3Ugb25seSBoYXZp bmcgMSBTU0QgZHJpdmUgaW4gZWFjaCBzZXJ2ZXIsJm5ic3A7ST1FMj04MD05OWQgc3VnPQpnZXN0 IG1heWJlCmxvb2tpbmcgaW50byB0aGUgTlZNRSBvcHRpb24uJm5ic3A7PC9pPjwvZm9udD48L2Rp dj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJtYXJnaW46MHB4Ij48Zm9u dCBmYWNlPTNEIklucHV0IE1vbj0KbyI+PGk+PGJyPgo8L2k+PC9mb250PjwvZGl2Pgo8ZGl2IGlk PTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEIm1hcmdpbjowcHgiPjxmb250CmZhY2U9M0Qi SW5wdXQgTW9ubyI+PGk+U2luY2UgeW91ciB1c2luZyBvbmx5IDMtc2VydmVycywgd2hhdCBJPUUy PTgwPTk5ZCA9CnByb2JhYmx5CnN1Z2dlc3QgaXMgdG8gZG8gKDIgUmVwbGljYXMgJiM0MzsgQXJi aXRlciBOb2RlKSwgdGhpcyBzZXR1cCBhY3R1YWxseQpkb2Vzbj1FMj04MD05OXQgcmVxdWlyZSB0 aGUgM3JkIHNlcnZlciB0byBoYXZlIGJpZyBkcml2ZXMgYXQgYWxsIGFzIGl0IG9uPQpseQogc3Rv cmVzIG1ldGEtZGF0YSBhYm91dCB0aGUgZmlsZXMgYW5kIG5vdCBhY3R1YWxseSBhIGZ1bGwKY29w eS4mbmJzcDs8L2k+PC9mb250PjwvZGl2Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0 eWxlPTNEIm1hcmdpbjowcHgiPjxmb250IGZhY2U9M0QiSW5wdXQgTW9uPQpvIj48aT48YnI+Cjwv aT48L2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9M0QibWFy Z2luOjBweCI+PGZvbnQKZmFjZT0zRCJJbnB1dCBNb25vIj48aT5QbGVhc2Ugc2VlIHRoZSBhdHRh Y2hlZCBkb2N1bWVudCB0aGF0IHdhcyBnaXZlbiB0bz0KIG1lCmJ5IFJlZCBIYXQgdG8gZ2V0IG1v cmUgaW5mb3JtYXRpb24gb24gdGhpcy4gSG9wZSB0aGlzIGluZm9ybWF0aW9uIGhlbHBzCnlvdS48 L2k+PC9mb250PjwvZGl2Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEIm1h cmdpbjowcHgiPjxmb250IGZhY2U9M0QiSW5wdXQgTW9uPQpvIj48aT48YnI+CjwvaT48L2ZvbnQ+ PC9kaXY+Cjxicj4KPGRpdiBpZD0zRCJibG9vcF9zaWduXzE1MDIwODczNzY3MjU0NjkxODQiIGNs YXNzPTNEImJsb29wX3NpZ24iPjxzcGFuCnN0eWxlPTNEImZvbnQtZmFtaWx5OidoZWx2ZXRpY2Eg TmV1ZScsaGVsdmV0aWNhOyBmb250LXNpemU6MTRweCI+LS08L3NwYW49Cj48YnIKc3R5bGU9M0Qi Zm9udC1mYW1pbHk6J2hlbHZldGljYSBOZXVlJyxoZWx2ZXRpY2E7IGZvbnQtc2l6ZToxNHB4Ij4K PGRpdiBjbGFzcz0zRCJnbWFpbF9zaWduYXR1cmUiCnN0eWxlPTNEImZvbnQtZmFtaWx5OidoZWx2 ZXRpY2EgTmV1ZScsaGVsdmV0aWNhOyBmb250LXNpemU6MTRweCI+CjxkaXYgZGlyPTNEImx0ciI+ CjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj5EZXZpbiBBY29zdGEsIFJIQ0EsIFJIVkNBPC9kaXY+Cjxk aXY+UmVkIEhhdCBDZXJ0aWZpZWQgQXJjaGl0ZWN0PC9kaXY+CjxkaXY+PC9kaXY+CjwvZGl2Pgo8 L2Rpdj4KPC9kaXY+Cjxicj4KPHAgY2xhc3M9M0QiYWlybWFpbF9vbiI+T24gQXVndXN0IDYsIDIw MTcgYXQgNzoyOToyOSBQTSwgTW9hY2lyIEZlcnJlaXJhID0KKDxhCmhyZWY9M0QibWFpbHRvOm1v YWNpcmZlcnJlaXJhQGhvdG1haWwuY29tIj5tb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbTwvYT49 CikKd3JvdGU6PC9wPgo8YmxvY2txdW90ZSB0eXBlPTNEImNpdGUiIGNsYXNzPTNEImNsZWFuX2Jx Ij48c3Bhbj4KPGRpdiBkaXI9M0QibHRyIj4KPGRpdj48L2Rpdj4KPGRpdj4KPGRpdiBpZD0zRCJk aXZ0YWdkZWZhdWx0d3JhcHBlciIgZGlyPTNEImx0ciIKc3R5bGU9M0QiZm9udC1zaXplOjEycHQ7 IGNvbG9yOiMwMDAwMDA7IGZvbnQtZmFtaWx5OkNhbGlicmksSGVsdmV0aWNhLHNhbj0Kcy1zZXJp ZiI+CjxwPjxzcGFuPkkgYW0gd2lsbGluZyB0byBhc3NlbWJsZSBhIG9WaXJ0ICZxdW90O3BvZCZx dW90OywgbWFkZSBvZiAzCnNlcnZlcnMsIGVhY2ggd2l0aCAyIENQVSBzb2NrZXRzIG9mIDEyIGNv cmVzLCAyNTZHQiBSQU0sIDcgSEREIDEwSywgMSBTU0Q9Ci4KVGhlIGlkZWEgaXMgdG8gdXNlIEds dXN0ZXJGUyB0byBwcm92aWRlIEhBIGZvciB0aGUgVk1zLiBUaGUgMyBzZXJ2ZXJzIGhhdj0KZQph IGR1YWwgNDBHYiBOSUMgYW5kIGEgZHVhbCAxMEdiIE5JQy4gU28gbXkgaW50ZW50aW9uCiBpcyB0 byBjcmVhdGUgYSBsb29wIGxpa2UgYSBzZXJ2ZXIgdHJpYW5nbGUgdXNpbmcgdGhlIDQwR2IgTklD cyBmb3IKdmlydHVhbGl6YXRpb24gZmlsZXMgKFZNcyAucWNvdzIpIGFjY2VzcyBhbmQgdG8gbW92 ZSBWTXMgYXJvdW5kIHRoZSBwb2QKKGVhc3QgL3dlc3QgdHJhZmZpYykgd2hpbGUgdXNpbmcgdGhl IDEwR2IgaW50ZXJmYWNlcyBmb3IgZ2l2aW5nIHNlcnZpY2VzID0KdG8KdGhlIG91dHNpZGUgd29y bGQgKG5vcnRoL3NvdXRoIHRyYWZmaWMpLjwvc3Bhbj48L3A+CjxwPjxicj4KPC9wPgo8cD5UaGlz IHNhaWQsIG15IGZpcnN0IHF1ZXN0aW9uIGlzOiBIb3cgc2hvdWxkIEkgZGVwbG95IEdsdXN0ZXJG UyBpbiBzdWNoCm9WaXJ0IHNjZW5hcmlvPyBNeSBxdWVzdGlvbnMgYXJlOjwvcD4KPHA+PGJyPgo8 L3A+CjxwPjEgLSBTaG91bGQgSSBjcmVhdGUgMyBSQUlEIChpLmUuOiBSQUlEIDUpLCBvbmUgb24g ZWFjaCBvVmlydCBub2RlLCBhbmQKdGhlbiBjcmVhdGUgYSBHbHVzdGVyRlMgdXNpbmcgdGhlbT88 L3A+CjxwPjIgLSBJbnN0ZWFkLCBzaG91bGQgSSBjcmVhdGUgYSBKQk9EIGFycmF5IG1hZGUgb2Yg YWxsIHNlcnZlcidzIGRpc2tzPzw9Ci9wPgo8cD4zIC0gV2hhdCBpcyB0aGUgYmVzdCBHbHVzdGVy IGNvbmZpZ3VyYXRpb24gdG8gcHJvdmlkZSBmb3IgSEEgd2hpbGUgbm90CmNvbnN1bWluZyB0b28g bXVjaCBkaXNrIHNwYWNlPzxicj4KPC9wPgo8cD40IC0gRG9lcyBhIG9WaXJ0IGh5cGVydmlzb3Ig cG9kIGxpa2UgSSBhbSBwbGFubmluZyB0byBidWlsZCwgYW5kIHRoZQp2aXJ0dWFsaXphdGlvbiBl bnZpcm9ubWVudCwgYmVuZWZpdHMgZnJvbSB0aWVyaW5nIHdoZW4gdXNpbmcgYSBTU0QgZGlzaz8K QW5kIHllcywgd2lsbCBHbHVzdGVyIGRvIGl0IGJ5IGRlZmF1bHQgb3IgSSBoYXZlIHRvIGNvbmZp Z3VyZSBpdCB0byBkbyBzbz0KPzwvcD4KPHA+PGJyPgo8L3A+CjxwPkF0IHRoZSBib3R0b20gbGlu ZSwgd2hhdCBpcyB0aGUgZ29vZCBwcmFjdGljZSBmb3IgdXNpbmcgR2x1c3RlckZTIGluCnNtYWxs IHBvZHMgZm9yIGVudGVycHJpc2VzPzxicj4KPC9wPgo8cD48YnI+CjwvcD4KPHA+WW91IG9waW5p b24vZmVlZGJhY2sgd2lsbCBiZSByZWFsbHkgYXBwcmVjaWF0ZWQhPC9wPgo8cD5Nb2FjaXI8YnI+ CjwvcD4KPC9kaXY+Cl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fIDxicj4KVXNlcnMgbWFpbGluZyBsaXN0IDxicj4KPGEgaHJlZj0zRCJtYWlsdG86VXNlcnNA b3ZpcnQub3JnIj5Vc2Vyc0BvdmlydC5vcmc8L2E+IDxicj4KPGEKaHJlZj0zRCJodHRwOi8vbGlz dHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiPmh0dHA6Ly9saXN0cy5vdmlydD0K Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPgo8YnI+CjwvZGl2Pgo8L2Rpdj4KPC9zcGFu PjwvYmxvY2txdW90ZT4KPC9kaXY+CjwvZGl2Pgo8L2Rpdj4KX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX188YnI+ClVzZXJzIG1haWxpbmcgbGlzdDxicj4KPGEg Y2xhc3M9M0QiYXFtLWF1dG9saW5rIGFxbS1hdXRvd3JhcCIKaHJlZj0zRCJtYWlsdG86VXNlcnMl NDBvdmlydC5vcmciPlVzZXJzQG92aXJ0Lm9yZzwvYT48YnI+CjxhIGNsYXNzPTNEImFxbS1hdXRv bGluayBhcW0tYXV0b3dyYXAiCmhyZWY9M0QiaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFu L2xpc3RpbmZvL3VzZXJzIj5odHRwOi8vbGlzdHMub3ZpcnQ9Ci5vcmcvbWFpbG1hbi9saXN0aW5m by91c2VyczwvYT48YnI+Cjxicj4KPC9ibG9ja3F1b3RlPgo8L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8 L2Rpdj4KPC9kaXY+CjwvYmxvY2txdW90ZT4KPC9kaXY+CjwvZGl2Pgo8L2JvZHk+CjwvaHRtbD4K Ci0tLS0tLS0tLS0tLTE1ZGMxNWZkZDcwNDVmYTI3NjhjNzUwYjc3LS0KCg== --===============0948125851385482984==-- From fernando.frediani at upx.com Tue Aug 8 13:24:56 2017 Content-Type: multipart/mixed; boundary="===============4643676370231253402==" MIME-Version: 1.0 From: FERNANDO FREDIANI To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Tue, 08 Aug 2017 10:24:50 -0300 Message-ID: <02172d33-e14e-aa8d-98f5-d920573e281c@upx.com> In-Reply-To: B1592A25-DF71-4625-9271-5E1E2C37599C@orange.fr --===============4643676370231253402== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable That's something on the way RAID works, regardless what most = 'super-ultra' powerfull hardware controller you may have. RAID 5 or 6 = will never have the same write performance as a RAID 10 o 0 for example. = Writeback caches can deal with bursts well but they have a limit = therefore there will always be a penalty compared to what else you could = have. If you have a continuous stream of data (a big VM deployment or a large = data copy) there will be a continuous write and that will likely fill up = the cache making the disks underneath the bottleneck. That's why on some other scenarios, like ZFS people have multiple groups = of RAID 6 (called RAIDZ2) so it improves the write speeds for these type = of scenarios. In the scenario given in this thread with just 3 servers, each with a = RAID 6 there will be a bare limit on the write performance specially for = streammed data for most powerfull your hardware controller can do = write-back. Also I agree the 40Gb NICs may not be used fully and 10Gb can do the job = well, but if they were available at the begining, why not use them. Fernando On 08/08/2017 03:16, Fabrice Bacchella wrote: >> Le 8 ao=C3=BBt 2017 =C3=A0 04:08, FERNANDO FREDIANI a =C3=A9crit : >> Even if you have a Hardware RAID Controller with Writeback cache you wil= l have a significant performance penalty and may not fully use all the reso= urces you mentioned you have. >> > Nope again,from my experience with HP Smart Array and write back cache, w= rite, that goes in the cache, are even faster that read that must goes to t= he disks. of course if the write are too fast and to big, they will over ov= erflow the cache. But on todays controller they are multi-gigabyte cache, y= ou must write a lot to fill them. And if you can afford 40Gb card, you can = afford decent controller. > > > --===============4643676370231253402==-- From fernando.frediani at upx.com Tue Aug 8 13:32:01 2017 Content-Type: multipart/mixed; boundary="===============6994257392034748241==" MIME-Version: 1.0 From: FERNANDO FREDIANI To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Tue, 08 Aug 2017 10:31:55 -0300 Message-ID: <940f2543-1614-eb8a-6d10-810a46a48fae@upx.com> In-Reply-To: DB6P190MB02801EA0892B38F503220896C88A0@DB6P190MB0280.EURP190.PROD.OUTLOOK.COM --===============6994257392034748241== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------A95CF1C0E4312049A540BB6A Content-Type: text/plain; charset=3Dwindows-1252; format=3Dflowed Content-Transfer-Encoding: 7bit Exactly Moacir, that is my point. A proper Distributed FIlesystem should not rely on any type of RAID as = it can make its own redundancy without having to rely on any underneath = layer (look at CEPH). Using RAID may help with management and in certain = scenarios to replace a faulty disk, but at a cost, not cheap by the way. That's why in terms of resourcing saving, if a replica 3 brings those = issues mentioned it is much worth to have a small arbiter somewhere = instead of wasting a significant amount of disk space. Fernando On 08/08/2017 06:09, Moacir Ferreira wrote: > > Fernando, > > > Let's see what people say... But this is what I understood Red Hat = > says is the best performance model. This is the main reason to open = > this discussion because as long as I can see, some of you in the = > community, do not agree. > > > But when I think about a "distributed file system", that can make any = > number of copies you want, it does not make sense using a RAIDed = > brick, what it makes sense is to use JBOD. > > > Moacir > > > ------------------------------------------------------------------------ > *From:* fernando.frediani(a)upx.com.br o= n = > behalf of FERNANDO FREDIANI > *Sent:* Tuesday, August 8, 2017 3:08 AM > *To:* Moacir Ferreira > *Cc:* Colin Coe; users(a)ovirt.org > *Subject:* Re: [ovirt-users] Good practices > Moacir, I understand that if you do this type of configuration you = > will be severely impacted on storage performance, specially for = > writes. Even if you have a Hardware RAID Controller with Writeback = > cache you will have a significant performance penalty and may not = > fully use all the resources you mentioned you have. > > Fernando > > 2017-08-07 10:03 GMT-03:00 Moacir Ferreira >: > > Hi Colin, > > > Take a look on Devin's response. Also, read the doc he shared that > gives some hints on how to deploy Gluster. > > > It is more like that if you want high-performance you should have > the bricks created as RAID (5 or 6) by the server's disk > controller and them assemble a JBOD GlusterFS. The attached > document is Gluster specific and not for oVirt. But at this point > I think that having SSD will not be a plus as using the RAID > controller Gluster will not be aware of the SSD. Regarding the OS, > my idea is to have a RAID 1, made of 2 low cost HDDs, to install it. > > > So far, based on the information received I should create a single > RAID 5 or 6 on each server and then use this disk as a brick to > create my Gluster cluster, made of 2 replicas + 1 arbiter. What is > new for me is the detail that the arbiter does not need a lot of > space as it only keeps meta data. > > > Thanks for your response! > > Moacir > > ---------------------------------------------------------------------= --- > *From:* Colin Coe > > *Sent:* Monday, August 7, 2017 12:41 PM > > *To:* Moacir Ferreira > *Cc:* users(a)ovirt.org > *Subject:* Re: [ovirt-users] Good practices > Hi > > I just thought that you'd do hardware RAID if you had the > controller or JBOD if you didn't. In hindsight, a server with > 40Gbps NICs is pretty likely to have a hardware RAID controller. = > I've never done JBOD with hardware RAID. I think having a single > gluster brick on hardware JBOD would be riskier than multiple > bricks, each on a single disk, but thats not based on anything > other than my prejudices. > > I thought gluster tiering was for the most frequently accessed > files, in which case all the VMs disks would end up in the hot > tier. However, I have been wrong before... > > I just wanted to know where the OS was going as I didn't see it > mentioned in the OP. Normally, I'd have the OS on a RAID1 but in > your case thats a lot of wasted disk. > > Honestly, I think Yaniv's answer was far better than my own and > made the important point about having an arbiter. > > Thanks > > On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreira > > > wrote: > > Hi Colin, > > > I am in Portugal, so sorry for this late response. It is quite > confusing for me, please consider: > > * > *1*- *What if the RAID is done by the server's disk > controller, not by software? > > 2 -**For JBOD I am just using gdeploy to deploy it. However, I > am not using the oVirt node GUI to do this. > > > 3 -**As the VM .qcow2 files are quite big, tiering would only > help if made by an intelligent system that uses SSD for chunks > of data not for the entire .qcow2 file. But I guess this is a > problem everybody else has. So, Do you know how tiering works > in Gluster? > > > 4 - I am putting the OS on the first disk. However, would you > do differently? > > > Moacir > > -----------------------------------------------------------------= ------- > *From:* Colin Coe > > *Sent:* Monday, August 7, 2017 4:48 AM > *To:* Moacir Ferreira > *Cc:* users(a)ovirt.org > *Subject:* Re: [ovirt-users] Good practices > 1) RAID5 may be a performance hit- > > 2) I'd be inclined to do this as JBOD by creating a > distributed disperse volume on each server. Something like > > echo gluster volume create dispersevol disperse-data 5 > redundancy 2 \ > $(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e > "server${SERVER}:/brick/brick-${SERVER}${BRICK}/brick \c"; > done; done) > > 3) I think the above. > > 4) Gluster does support tiering, but IIRC you'd need the same > number of SSD as spindle drives. There may be another way to > use the SSD as a fast cache. > > Where are you putting the OS? > > Hope I understood the question... > > Thanks > > On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira > > wrote: > > I am willing to assemble a oVirt "pod", made of 3 servers, > each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, > 1 SSD. The idea is to use GlusterFS to provide HA for the > VMs. The 3 servers have a dual 40Gb NIC and a dual 10Gb > NIC. So my intention is to create a loop like a server > triangle using the 40Gb NICs for virtualization files (VMs > .qcow2) access and to move VMs around the pod (east /west > traffic) while using the 10Gb interfaces for giving > services to the outside world (north/south traffic). > > > This said, my first question is: How should I deploy > GlusterFS in such oVirt scenario? My questions are: > > > 1 - Should I create 3 RAID (i.e.: RAID 5), one on each > oVirt node, and then create a GlusterFS using them? > > 2 - Instead, should I create a JBOD array made of all > server's disks? > > 3 - What is the best Gluster configuration to provide for > HA while not consuming too much disk space? > > 4 - Does a oVirt hypervisor pod like I am planning to > build, and the virtualization environment, benefits from > tiering when using a SSD disk? And yes, will Gluster do it > by default or I have to configure it to do so? > > > At the bottom line, what is the good practice for using > GlusterFS in small pods for enterprises? > > > You opinion/feedback will be really appreciated! > > Moacir > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > > > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > > --------------A95CF1C0E4312049A540BB6A Content-Type: text/html; charset=3Dwindows-1252 Content-Transfer-Encoding: 8bit
Exactly Moacir, that is my point.

A proper Distributed FIlesystem should not rely on any type of RAID as it can make its own redundancy without having to rely on any underneath layer (look at CEPH). Using RAID may help with management and in certain scenarios to replace a faulty disk, but at a cost, not cheap by the way.
That's why in terms of resourcing saving, if a replica 3 brings those issues mentioned it is much worth to have a small arbiter somewhere instead of wasting a significant amount of disk space.

Fernando

On 08/08/2017 06:09, Moacir Ferreira wrote:

Fernando,

Let's see what people say... But this is what I understood Red Hat says is the best performance model. This is the main reason to open this discussion because as long as I can see, some of you in the community, do not agree.

But when I think about a "distributed file system", that can make any number of copies you want, it does not make sense using a RAIDed brick, what it makes sense is to use JBOD.

Moacir

From: fernando.frediani(a)upx.com.br <fernando.frediani(a)upx.com.br> on behalf o= f FERNANDO FREDIANI <fernando.frediani(a)upx.com>
Sent: Tuesday, August 8, 2017 3:08 AM
To: Moacir Ferreira
Cc: Colin Coe; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices
=C2=A0

Moacir, I understand that if you do this type of configuration you will be severely impacted on storage performance, specially for writes. Even if you have a Hardware RAID Controller with Writeback cache you will have a significant performance penalty and may not fully use all the resources you mentioned you have.

Fernando

2017-08-07 10:03 GMT-03:00 Moacir Ferreira <moacirferrei= ra(a)hotmail.com>:

Hi Colin,

Take a look on Devin's response. Also, read the doc he shared that gives some hints on how to deploy Gluster.

It is more like that if you want high-performance you should have the bricks created as RAID (5 or 6)=C2=A0by the server's disk controller and them assemble a JBOD GlusterFS. The attached document is Gluster specific and not for oVirt. But at this point I think that having SSD will not be a plus as using the RAID controller Gluster will not be aware of the SSD. Regarding the OS, my idea is to have a RAID 1, made of 2 low cost HDDs, to install it.

So far, based on the information received=C2=A0I should=C2=A0create a single RAID 5 or 6 on each server and then use this disk as a brick to create my Gluster cluster, made of 2 replicas + 1 arbiter. What is new for me is the detail that the arbiter does not need a lot of space as it only keeps meta data.

Thanks for your response!

Moacir

= From: Colin Coe <co= lin.coe(a)gmail.com>
Sent: Monday, August 7, 2017 12:41 PM

To: Moacir Ferreira
Cc: users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices

=C2=A0

Hi

I just thought that you'd do hardware RAID if you had the controller or JBOD if you didn't.=C2=A0 In hindsight, a server with 40Gbps NICs is pretty likely to have a hardware RAID controller.=C2=A0 I've never done JB= OD with hardware RAID.=C2=A0 I think having a single gluster brick on hardware JBOD would be riskier than multiple bricks, each on a single disk, but thats not based on anything other than my prejudices.

I thought gluster tiering was for the most frequently accessed files, in which case all the VMs disks would end up in the hot tier.=C2=A0 However, I have been wrong before...

I just wanted to know where the OS was going as I didn't see it mentioned in the OP.=C2=A0 Normally, I'd have the OS on a RAID1 but in your case thats a lot of wasted disk.

Honestly, I think Yaniv's answer was far better than my own and made the important point about having an arbiter.=C2=A0

Thanks

On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreira <moacirferrei= ra(a)hotmail.com> wrote:

Hi Colin,

I am in Portugal, so sorry for this late response. It is quite confusing for me, please consider:

1 - What if the RAID is done by the server's disk controller, not by software?

2 - For JBOD I am just using gdeploy to deploy it. However, I am not using the oVirt node GUI to do this.

3 - As the VM .qcow2=C2=A0files are quite big, tiering would only help if made by an intelligent system that uses SSD for chunks of data not for the entire .qcow2 file. But I guess this is a problem everybody else has. So, Do you know how tiering works in Gluster?

4 -=C2=A0I am putting the OS on the first disk. However, would you do differently?

Moacir

= From: Colin Coe <co= lin.coe(a)gmail.com>
Sent: Monday, August 7, 2017 4:48 AM
To: Moacir Ferreira
Cc: us= ers(a)ovirt.org
Subject: Re: [ovirt-users] Good practices
=C2=A0

1) RAID5 may be a performance hit-

2) I'd be inclined to do this as JBOD by creating a distributed disperse volume on each server.=C2=A0 Something like

echo gluster volume create dispersevol disperse-data 5 redundancy 2 \

$(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e "server${SERVER}:/brick= /brick-${SERVER}${BRICK}/brick \c"; done; done)

3) I think the above.

4) Gluster does support tiering, but IIRC you'd need the same number of SSD as spindle drives.=C2= =A0 There may be another way to use the SSD as a fast cache.=C2=A0<= /div>

Where are you putting the OS?

Hope I understood the question...

Thanks

= On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira <moacirferreira(a)hotmail.com> wrote:

I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop like a server triangle using the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VMs around the pod (east /west traffic) while using the 10Gb interfaces for giving services to the outside world (north/south traffic).<= /p>

This said, my first question is: How should I deploy GlusterFS in such oVirt scenario? My questions are:

1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then create a GlusterFS using them?

2 - Instead, should I create a JBOD array made of all server's disks?

3 - What is the best Gluster configuration to provide for HA while not consuming too much disk space?

4 - Does a oVirt hypervisor pod like I am planning to build, and the virtualization environment, benefits from tiering when using a SSD disk? And yes, will Gluster do it by default or I have to configure it to do so?

At the bottom line, what is the good practice for using GlusterFS in small pods for enterprises?

You opinion/feedback will be really appreciated!

Moacir

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/ma= ilman/listinfo/users

--------------A95CF1C0E4312049A540BB6A-- --===============6994257392034748241== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS1BOTVDRjFDMEU0MzEyMDQ5QTU0MEJCNkEKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXdpbmRvd3MtMTI1MjsgZm9ybWF0PWZsb3dlZApDb250ZW50LVRyYW5zZmVyLUVuY29kaW5n OiA3Yml0CgpFeGFjdGx5IE1vYWNpciwgdGhhdCBpcyBteSBwb2ludC4KCgpBIHByb3BlciBEaXN0 cmlidXRlZCBGSWxlc3lzdGVtIHNob3VsZCBub3QgcmVseSBvbiBhbnkgdHlwZSBvZiBSQUlEIGFz IAppdCBjYW4gbWFrZSBpdHMgb3duIHJlZHVuZGFuY3kgd2l0aG91dCBoYXZpbmcgdG8gcmVseSBv biBhbnkgdW5kZXJuZWF0aCAKbGF5ZXIgKGxvb2sgYXQgQ0VQSCkuIFVzaW5nIFJBSUQgbWF5IGhl bHAgd2l0aCBtYW5hZ2VtZW50IGFuZCBpbiBjZXJ0YWluIApzY2VuYXJpb3MgdG8gcmVwbGFjZSBh IGZhdWx0eSBkaXNrLCBidXQgYXQgYSBjb3N0LCBub3QgY2hlYXAgYnkgdGhlIHdheS4KVGhhdCdz IHdoeSBpbiB0ZXJtcyBvZiByZXNvdXJjaW5nIHNhdmluZywgaWYgYSByZXBsaWNhIDMgYnJpbmdz IHRob3NlIAppc3N1ZXMgbWVudGlvbmVkIGl0IGlzIG11Y2ggd29ydGggdG8gaGF2ZSBhIHNtYWxs IGFyYml0ZXIgc29tZXdoZXJlIAppbnN0ZWFkIG9mIHdhc3RpbmcgYSBzaWduaWZpY2FudCBhbW91 bnQgb2YgZGlzayBzcGFjZS4KCgpGZXJuYW5kbwoKCk9uIDA4LzA4LzIwMTcgMDY6MDksIE1vYWNp ciBGZXJyZWlyYSB3cm90ZToKPgo+IEZlcm5hbmRvLAo+Cj4KPiBMZXQncyBzZWUgd2hhdCBwZW9w bGUgc2F5Li4uIEJ1dCB0aGlzIGlzIHdoYXQgSSB1bmRlcnN0b29kIFJlZCBIYXQgCj4gc2F5cyBp cyB0aGUgYmVzdCBwZXJmb3JtYW5jZSBtb2RlbC4gVGhpcyBpcyB0aGUgbWFpbiByZWFzb24gdG8g b3BlbiAKPiB0aGlzIGRpc2N1c3Npb24gYmVjYXVzZSBhcyBsb25nIGFzIEkgY2FuIHNlZSwgc29t ZSBvZiB5b3UgaW4gdGhlIAo+IGNvbW11bml0eSwgZG8gbm90IGFncmVlLgo+Cj4KPiBCdXQgd2hl biBJIHRoaW5rIGFib3V0IGEgImRpc3RyaWJ1dGVkIGZpbGUgc3lzdGVtIiwgdGhhdCBjYW4gbWFr ZSBhbnkgCj4gbnVtYmVyIG9mIGNvcGllcyB5b3Ugd2FudCwgaXQgZG9lcyBub3QgbWFrZSBzZW5z ZSB1c2luZyBhIFJBSURlZCAKPiBicmljaywgd2hhdCBpdCBtYWtlcyBzZW5zZSBpcyB0byB1c2Ug SkJPRC4KPgo+Cj4gTW9hY2lyCj4KPgo+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQo+ICpGcm9tOiogZmVybmFu ZG8uZnJlZGlhbmlAdXB4LmNvbS5iciA8ZmVybmFuZG8uZnJlZGlhbmlAdXB4LmNvbS5icj4gb24g Cj4gYmVoYWxmIG9mIEZFUk5BTkRPIEZSRURJQU5JIDxmZXJuYW5kby5mcmVkaWFuaUB1cHguY29t Pgo+ICpTZW50OiogVHVlc2RheSwgQXVndXN0IDgsIDIwMTcgMzowOCBBTQo+ICpUbzoqIE1vYWNp ciBGZXJyZWlyYQo+ICpDYzoqIENvbGluIENvZTsgdXNlcnNAb3ZpcnQub3JnCj4gKlN1YmplY3Q6 KiBSZTogW292aXJ0LXVzZXJzXSBHb29kIHByYWN0aWNlcwo+IE1vYWNpciwgSSB1bmRlcnN0YW5k IHRoYXQgaWYgeW91IGRvIHRoaXMgdHlwZSBvZiBjb25maWd1cmF0aW9uIHlvdSAKPiB3aWxsIGJl IHNldmVyZWx5IGltcGFjdGVkIG9uIHN0b3JhZ2UgcGVyZm9ybWFuY2UsIHNwZWNpYWxseSBmb3Ig Cj4gd3JpdGVzLiBFdmVuIGlmIHlvdSBoYXZlIGEgSGFyZHdhcmUgUkFJRCBDb250cm9sbGVyIHdp dGggV3JpdGViYWNrIAo+IGNhY2hlIHlvdSB3aWxsIGhhdmUgYSBzaWduaWZpY2FudCBwZXJmb3Jt YW5jZSBwZW5hbHR5IGFuZCBtYXkgbm90IAo+IGZ1bGx5IHVzZSBhbGwgdGhlIHJlc291cmNlcyB5 b3UgbWVudGlvbmVkIHlvdSBoYXZlLgo+Cj4gRmVybmFuZG8KPgo+IDIwMTctMDgtMDcgMTA6MDMg R01ULTAzOjAwIE1vYWNpciBGZXJyZWlyYSA8bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb20gCj4g PG1haWx0bzptb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbT4+Ogo+Cj4gICAgIEhpIENvbGluLAo+ Cj4KPiAgICAgVGFrZSBhIGxvb2sgb24gRGV2aW4ncyByZXNwb25zZS4gQWxzbywgcmVhZCB0aGUg ZG9jIGhlIHNoYXJlZCB0aGF0Cj4gICAgIGdpdmVzIHNvbWUgaGludHMgb24gaG93IHRvIGRlcGxv eSBHbHVzdGVyLgo+Cj4KPiAgICAgSXQgaXMgbW9yZSBsaWtlIHRoYXQgaWYgeW91IHdhbnQgaGln aC1wZXJmb3JtYW5jZSB5b3Ugc2hvdWxkIGhhdmUKPiAgICAgdGhlIGJyaWNrcyBjcmVhdGVkIGFz IFJBSUQgKDUgb3IgNikgYnkgdGhlIHNlcnZlcidzIGRpc2sKPiAgICAgY29udHJvbGxlciBhbmQg dGhlbSBhc3NlbWJsZSBhIEpCT0QgR2x1c3RlckZTLiBUaGUgYXR0YWNoZWQKPiAgICAgZG9jdW1l bnQgaXMgR2x1c3RlciBzcGVjaWZpYyBhbmQgbm90IGZvciBvVmlydC4gQnV0IGF0IHRoaXMgcG9p bnQKPiAgICAgSSB0aGluayB0aGF0IGhhdmluZyBTU0Qgd2lsbCBub3QgYmUgYSBwbHVzIGFzIHVz aW5nIHRoZSBSQUlECj4gICAgIGNvbnRyb2xsZXIgR2x1c3RlciB3aWxsIG5vdCBiZSBhd2FyZSBv ZiB0aGUgU1NELiBSZWdhcmRpbmcgdGhlIE9TLAo+ICAgICBteSBpZGVhIGlzIHRvIGhhdmUgYSBS QUlEIDEsIG1hZGUgb2YgMiBsb3cgY29zdCBIRERzLCB0byBpbnN0YWxsIGl0Lgo+Cj4KPiAgICAg U28gZmFyLCBiYXNlZCBvbiB0aGUgaW5mb3JtYXRpb24gcmVjZWl2ZWQgSSBzaG91bGQgY3JlYXRl IGEgc2luZ2xlCj4gICAgIFJBSUQgNSBvciA2IG9uIGVhY2ggc2VydmVyIGFuZCB0aGVuIHVzZSB0 aGlzIGRpc2sgYXMgYSBicmljayB0bwo+ICAgICBjcmVhdGUgbXkgR2x1c3RlciBjbHVzdGVyLCBt YWRlIG9mIDIgcmVwbGljYXMgKyAxIGFyYml0ZXIuIFdoYXQgaXMKPiAgICAgbmV3IGZvciBtZSBp cyB0aGUgZGV0YWlsIHRoYXQgdGhlIGFyYml0ZXIgZG9lcyBub3QgbmVlZCBhIGxvdCBvZgo+ICAg ICBzcGFjZSBhcyBpdCBvbmx5IGtlZXBzIG1ldGEgZGF0YS4KPgo+Cj4gICAgIFRoYW5rcyBmb3Ig eW91ciByZXNwb25zZSEKPgo+ICAgICBNb2FjaXIKPgo+ICAgICAtLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KPiAg ICAgKkZyb206KiBDb2xpbiBDb2UgPGNvbGluLmNvZUBnbWFpbC5jb20gPG1haWx0bzpjb2xpbi5j b2VAZ21haWwuY29tPj4KPiAgICAgKlNlbnQ6KiBNb25kYXksIEF1Z3VzdCA3LCAyMDE3IDEyOjQx IFBNCj4KPiAgICAgKlRvOiogTW9hY2lyIEZlcnJlaXJhCj4gICAgICpDYzoqIHVzZXJzQG92aXJ0 Lm9yZyA8bWFpbHRvOnVzZXJzQG92aXJ0Lm9yZz4KPiAgICAgKlN1YmplY3Q6KiBSZTogW292aXJ0 LXVzZXJzXSBHb29kIHByYWN0aWNlcwo+ICAgICBIaQo+Cj4gICAgIEkganVzdCB0aG91Z2h0IHRo YXQgeW91J2QgZG8gaGFyZHdhcmUgUkFJRCBpZiB5b3UgaGFkIHRoZQo+ICAgICBjb250cm9sbGVy IG9yIEpCT0QgaWYgeW91IGRpZG4ndC4gIEluIGhpbmRzaWdodCwgYSBzZXJ2ZXIgd2l0aAo+ICAg ICA0MEdicHMgTklDcyBpcyBwcmV0dHkgbGlrZWx5IHRvIGhhdmUgYSBoYXJkd2FyZSBSQUlEIGNv bnRyb2xsZXIuIAo+ICAgICBJJ3ZlIG5ldmVyIGRvbmUgSkJPRCB3aXRoIGhhcmR3YXJlIFJBSUQu ICBJIHRoaW5rIGhhdmluZyBhIHNpbmdsZQo+ICAgICBnbHVzdGVyIGJyaWNrIG9uIGhhcmR3YXJl IEpCT0Qgd291bGQgYmUgcmlza2llciB0aGFuIG11bHRpcGxlCj4gICAgIGJyaWNrcywgZWFjaCBv biBhIHNpbmdsZSBkaXNrLCBidXQgdGhhdHMgbm90IGJhc2VkIG9uIGFueXRoaW5nCj4gICAgIG90 aGVyIHRoYW4gbXkgcHJlanVkaWNlcy4KPgo+ICAgICBJIHRob3VnaHQgZ2x1c3RlciB0aWVyaW5n IHdhcyBmb3IgdGhlIG1vc3QgZnJlcXVlbnRseSBhY2Nlc3NlZAo+ICAgICBmaWxlcywgaW4gd2hp Y2ggY2FzZSBhbGwgdGhlIFZNcyBkaXNrcyB3b3VsZCBlbmQgdXAgaW4gdGhlIGhvdAo+ICAgICB0 aWVyLiAgSG93ZXZlciwgSSBoYXZlIGJlZW4gd3JvbmcgYmVmb3JlLi4uCj4KPiAgICAgSSBqdXN0 IHdhbnRlZCB0byBrbm93IHdoZXJlIHRoZSBPUyB3YXMgZ29pbmcgYXMgSSBkaWRuJ3Qgc2VlIGl0 Cj4gICAgIG1lbnRpb25lZCBpbiB0aGUgT1AuICBOb3JtYWxseSwgSSdkIGhhdmUgdGhlIE9TIG9u IGEgUkFJRDEgYnV0IGluCj4gICAgIHlvdXIgY2FzZSB0aGF0cyBhIGxvdCBvZiB3YXN0ZWQgZGlz ay4KPgo+ICAgICBIb25lc3RseSwgSSB0aGluayBZYW5pdidzIGFuc3dlciB3YXMgZmFyIGJldHRl ciB0aGFuIG15IG93biBhbmQKPiAgICAgbWFkZSB0aGUgaW1wb3J0YW50IHBvaW50IGFib3V0IGhh dmluZyBhbiBhcmJpdGVyLgo+Cj4gICAgIFRoYW5rcwo+Cj4gICAgIE9uIE1vbiwgQXVnIDcsIDIw MTcgYXQgNTo1NiBQTSwgTW9hY2lyIEZlcnJlaXJhCj4gICAgIDxtb2FjaXJmZXJyZWlyYUBob3Rt YWlsLmNvbSA8bWFpbHRvOm1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY29tPj4KPiAgICAgd3JvdGU6 Cj4KPiAgICAgICAgIEhpIENvbGluLAo+Cj4KPiAgICAgICAgIEkgYW0gaW4gUG9ydHVnYWwsIHNv IHNvcnJ5IGZvciB0aGlzIGxhdGUgcmVzcG9uc2UuIEl0IGlzIHF1aXRlCj4gICAgICAgICBjb25m dXNpbmcgZm9yIG1lLCBwbGVhc2UgY29uc2lkZXI6Cj4KPiAgICAgICAgICoKPiAgICAgICAgICox Ki0gKldoYXQgaWYgdGhlIFJBSUQgaXMgZG9uZSBieSB0aGUgc2VydmVyJ3MgZGlzawo+ICAgICAg ICAgY29udHJvbGxlciwgbm90IGJ5IHNvZnR3YXJlPwo+Cj4gICAgICAgICAyIC0qKkZvciBKQk9E IEkgYW0ganVzdCB1c2luZyBnZGVwbG95IHRvIGRlcGxveSBpdC4gSG93ZXZlciwgSQo+ICAgICAg ICAgYW0gbm90IHVzaW5nIHRoZSBvVmlydCBub2RlIEdVSSB0byBkbyB0aGlzLgo+Cj4KPiAgICAg ICAgIDMgLSoqQXMgdGhlIFZNIC5xY293MiBmaWxlcyBhcmUgcXVpdGUgYmlnLCB0aWVyaW5nIHdv dWxkIG9ubHkKPiAgICAgICAgIGhlbHAgaWYgbWFkZSBieSBhbiBpbnRlbGxpZ2VudCBzeXN0ZW0g dGhhdCB1c2VzIFNTRCBmb3IgY2h1bmtzCj4gICAgICAgICBvZiBkYXRhIG5vdCBmb3IgdGhlIGVu dGlyZSAucWNvdzIgZmlsZS4gQnV0IEkgZ3Vlc3MgdGhpcyBpcyBhCj4gICAgICAgICBwcm9ibGVt IGV2ZXJ5Ym9keSBlbHNlIGhhcy4gU28sIERvIHlvdSBrbm93IGhvdyB0aWVyaW5nIHdvcmtzCj4g ICAgICAgICBpbiBHbHVzdGVyPwo+Cj4KPiAgICAgICAgIDQgLSBJIGFtIHB1dHRpbmcgdGhlIE9T IG9uIHRoZSBmaXJzdCBkaXNrLiBIb3dldmVyLCB3b3VsZCB5b3UKPiAgICAgICAgIGRvIGRpZmZl cmVudGx5Pwo+Cj4KPiAgICAgICAgIE1vYWNpcgo+Cj4gICAgICAgICAtLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0K PiAgICAgICAgICpGcm9tOiogQ29saW4gQ29lIDxjb2xpbi5jb2VAZ21haWwuY29tCj4gICAgICAg ICA8bWFpbHRvOmNvbGluLmNvZUBnbWFpbC5jb20+Pgo+ICAgICAgICAgKlNlbnQ6KiBNb25kYXks IEF1Z3VzdCA3LCAyMDE3IDQ6NDggQU0KPiAgICAgICAgICpUbzoqIE1vYWNpciBGZXJyZWlyYQo+ ICAgICAgICAgKkNjOiogdXNlcnNAb3ZpcnQub3JnIDxtYWlsdG86dXNlcnNAb3ZpcnQub3JnPgo+ ICAgICAgICAgKlN1YmplY3Q6KiBSZTogW292aXJ0LXVzZXJzXSBHb29kIHByYWN0aWNlcwo+ICAg ICAgICAgMSkgUkFJRDUgbWF5IGJlIGEgcGVyZm9ybWFuY2UgaGl0LQo+Cj4gICAgICAgICAyKSBJ J2QgYmUgaW5jbGluZWQgdG8gZG8gdGhpcyBhcyBKQk9EIGJ5IGNyZWF0aW5nIGEKPiAgICAgICAg IGRpc3RyaWJ1dGVkIGRpc3BlcnNlIHZvbHVtZSBvbiBlYWNoIHNlcnZlci4gIFNvbWV0aGluZyBs aWtlCj4KPiAgICAgICAgIGVjaG8gZ2x1c3RlciB2b2x1bWUgY3JlYXRlIGRpc3BlcnNldm9sIGRp c3BlcnNlLWRhdGEgNQo+ICAgICAgICAgcmVkdW5kYW5jeSAyIFwKPiAgICAgICAgICQoZm9yIFNF UlZFUiBpbiBhIGIgYzsgZG8gZm9yIEJSSUNLIGluICQoc2VxIDEgNSk7IGRvIGVjaG8gLWUKPiAg ICAgICAgICJzZXJ2ZXIke1NFUlZFUn06L2JyaWNrL2JyaWNrLSR7U0VSVkVSfSR7QlJJQ0t9L2Jy aWNrIFxjIjsKPiAgICAgICAgIGRvbmU7IGRvbmUpCj4KPiAgICAgICAgIDMpIEkgdGhpbmsgdGhl IGFib3ZlLgo+Cj4gICAgICAgICA0KSBHbHVzdGVyIGRvZXMgc3VwcG9ydCB0aWVyaW5nLCBidXQg SUlSQyB5b3UnZCBuZWVkIHRoZSBzYW1lCj4gICAgICAgICBudW1iZXIgb2YgU1NEIGFzIHNwaW5k bGUgZHJpdmVzLiBUaGVyZSBtYXkgYmUgYW5vdGhlciB3YXkgdG8KPiAgICAgICAgIHVzZSB0aGUg U1NEIGFzIGEgZmFzdCBjYWNoZS4KPgo+ICAgICAgICAgV2hlcmUgYXJlIHlvdSBwdXR0aW5nIHRo ZSBPUz8KPgo+ICAgICAgICAgSG9wZSBJIHVuZGVyc3Rvb2QgdGhlIHF1ZXN0aW9uLi4uCj4KPiAg ICAgICAgIFRoYW5rcwo+Cj4gICAgICAgICBPbiBTdW4sIEF1ZyA2LCAyMDE3IGF0IDEwOjQ5IFBN LCBNb2FjaXIgRmVycmVpcmEKPiAgICAgICAgIDxtb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbQo+ ICAgICAgICAgPG1haWx0bzptb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbT4+IHdyb3RlOgo+Cj4g ICAgICAgICAgICAgSSBhbSB3aWxsaW5nIHRvIGFzc2VtYmxlIGEgb1ZpcnQgInBvZCIsIG1hZGUg b2YgMyBzZXJ2ZXJzLAo+ICAgICAgICAgICAgIGVhY2ggd2l0aCAyIENQVSBzb2NrZXRzIG9mIDEy IGNvcmVzLCAyNTZHQiBSQU0sIDcgSEREIDEwSywKPiAgICAgICAgICAgICAxIFNTRC4gVGhlIGlk ZWEgaXMgdG8gdXNlIEdsdXN0ZXJGUyB0byBwcm92aWRlIEhBIGZvciB0aGUKPiAgICAgICAgICAg ICBWTXMuIFRoZSAzIHNlcnZlcnMgaGF2ZSBhIGR1YWwgNDBHYiBOSUMgYW5kIGEgZHVhbCAxMEdi Cj4gICAgICAgICAgICAgTklDLiBTbyBteSBpbnRlbnRpb24gaXMgdG8gY3JlYXRlIGEgbG9vcCBs aWtlIGEgc2VydmVyCj4gICAgICAgICAgICAgdHJpYW5nbGUgdXNpbmcgdGhlIDQwR2IgTklDcyBm b3IgdmlydHVhbGl6YXRpb24gZmlsZXMgKFZNcwo+ICAgICAgICAgICAgIC5xY293MikgYWNjZXNz IGFuZCB0byBtb3ZlIFZNcyBhcm91bmQgdGhlIHBvZCAoZWFzdCAvd2VzdAo+ICAgICAgICAgICAg IHRyYWZmaWMpIHdoaWxlIHVzaW5nIHRoZSAxMEdiIGludGVyZmFjZXMgZm9yIGdpdmluZwo+ICAg ICAgICAgICAgIHNlcnZpY2VzIHRvIHRoZSBvdXRzaWRlIHdvcmxkIChub3J0aC9zb3V0aCB0cmFm ZmljKS4KPgo+Cj4gICAgICAgICAgICAgVGhpcyBzYWlkLCBteSBmaXJzdCBxdWVzdGlvbiBpczog SG93IHNob3VsZCBJIGRlcGxveQo+ICAgICAgICAgICAgIEdsdXN0ZXJGUyBpbiBzdWNoIG9WaXJ0 IHNjZW5hcmlvPyBNeSBxdWVzdGlvbnMgYXJlOgo+Cj4KPiAgICAgICAgICAgICAxIC0gU2hvdWxk IEkgY3JlYXRlIDMgUkFJRCAoaS5lLjogUkFJRCA1KSwgb25lIG9uIGVhY2gKPiAgICAgICAgICAg ICBvVmlydCBub2RlLCBhbmQgdGhlbiBjcmVhdGUgYSBHbHVzdGVyRlMgdXNpbmcgdGhlbT8KPgo+ ICAgICAgICAgICAgIDIgLSBJbnN0ZWFkLCBzaG91bGQgSSBjcmVhdGUgYSBKQk9EIGFycmF5IG1h ZGUgb2YgYWxsCj4gICAgICAgICAgICAgc2VydmVyJ3MgZGlza3M/Cj4KPiAgICAgICAgICAgICAz IC0gV2hhdCBpcyB0aGUgYmVzdCBHbHVzdGVyIGNvbmZpZ3VyYXRpb24gdG8gcHJvdmlkZSBmb3IK PiAgICAgICAgICAgICBIQSB3aGlsZSBub3QgY29uc3VtaW5nIHRvbyBtdWNoIGRpc2sgc3BhY2U/ Cj4KPiAgICAgICAgICAgICA0IC0gRG9lcyBhIG9WaXJ0IGh5cGVydmlzb3IgcG9kIGxpa2UgSSBh bSBwbGFubmluZyB0bwo+ICAgICAgICAgICAgIGJ1aWxkLCBhbmQgdGhlIHZpcnR1YWxpemF0aW9u IGVudmlyb25tZW50LCBiZW5lZml0cyBmcm9tCj4gICAgICAgICAgICAgdGllcmluZyB3aGVuIHVz aW5nIGEgU1NEIGRpc2s/IEFuZCB5ZXMsIHdpbGwgR2x1c3RlciBkbyBpdAo+ICAgICAgICAgICAg IGJ5IGRlZmF1bHQgb3IgSSBoYXZlIHRvIGNvbmZpZ3VyZSBpdCB0byBkbyBzbz8KPgo+Cj4gICAg ICAgICAgICAgQXQgdGhlIGJvdHRvbSBsaW5lLCB3aGF0IGlzIHRoZSBnb29kIHByYWN0aWNlIGZv ciB1c2luZwo+ICAgICAgICAgICAgIEdsdXN0ZXJGUyBpbiBzbWFsbCBwb2RzIGZvciBlbnRlcnBy aXNlcz8KPgo+Cj4gICAgICAgICAgICAgWW91IG9waW5pb24vZmVlZGJhY2sgd2lsbCBiZSByZWFs bHkgYXBwcmVjaWF0ZWQhCj4KPiAgICAgICAgICAgICBNb2FjaXIKPgo+Cj4gICAgICAgICAgICAg X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KPiAgICAgICAg ICAgICBVc2VycyBtYWlsaW5nIGxpc3QKPiAgICAgICAgICAgICBVc2Vyc0BvdmlydC5vcmcgPG1h aWx0bzpVc2Vyc0BvdmlydC5vcmc+Cj4gICAgICAgICAgICAgaHR0cDovL2xpc3RzLm92aXJ0Lm9y Zy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzCj4gICAgICAgICAgICAgPGh0dHA6Ly9saXN0cy5vdmly dC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vycz4KPgo+Cj4KPgo+ICAgICBfX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwo+ICAgICBVc2VycyBtYWlsaW5nIGxp c3QKPiAgICAgVXNlcnNAb3ZpcnQub3JnIDxtYWlsdG86VXNlcnNAb3ZpcnQub3JnPgo+ICAgICBo dHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMKPiAgICAgPGh0dHA6 Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vycz4KPgo+CgoKLS0tLS0tLS0t LS0tLS1BOTVDRjFDMEU0MzEyMDQ5QTU0MEJCNkEKQ29udGVudC1UeXBlOiB0ZXh0L2h0bWw7IGNo YXJzZXQ9d2luZG93cy0xMjUyCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDhiaXQKCjxodG1s PgogIDxoZWFkPgogICAgPG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0 ZXh0L2h0bWw7CiAgICAgIGNoYXJzZXQ9d2luZG93cy0xMjUyIj4KICA8L2hlYWQ+CiAgPGJvZHkg Ymdjb2xvcj0iI0ZGRkZGRiIgdGV4dD0iIzAwMDAwMCI+CiAgICA8cD5FeGFjdGx5IE1vYWNpciwg dGhhdCBpcyBteSBwb2ludC48L3A+CiAgICA8cD48YnI+CiAgICA8L3A+CiAgICA8cD5BIHByb3Bl ciBEaXN0cmlidXRlZCBGSWxlc3lzdGVtIHNob3VsZCBub3QgcmVseSBvbiBhbnkgdHlwZSBvZgog ICAgICBSQUlEIGFzIGl0IGNhbiBtYWtlIGl0cyBvd24gcmVkdW5kYW5jeSB3aXRob3V0IGhhdmlu ZyB0byByZWx5IG9uCiAgICAgIGFueSB1bmRlcm5lYXRoIGxheWVyIChsb29rIGF0IENFUEgpLiBV c2luZyBSQUlEIG1heSBoZWxwIHdpdGgKICAgICAgbWFuYWdlbWVudCBhbmQgaW4gY2VydGFpbiBz Y2VuYXJpb3MgdG8gcmVwbGFjZSBhIGZhdWx0eSBkaXNrLCBidXQKICAgICAgYXQgYSBjb3N0LCBu b3QgY2hlYXAgYnkgdGhlIHdheS48YnI+CiAgICAgIFRoYXQncyB3aHkgaW4gdGVybXMgb2YgcmVz b3VyY2luZyBzYXZpbmcsIGlmIGEgcmVwbGljYSAzIGJyaW5ncwogICAgICB0aG9zZSBpc3N1ZXMg bWVudGlvbmVkIGl0IGlzIG11Y2ggd29ydGggdG8gaGF2ZSBhIHNtYWxsIGFyYml0ZXIKICAgICAg c29tZXdoZXJlIGluc3RlYWQgb2Ygd2FzdGluZyBhIHNpZ25pZmljYW50IGFtb3VudCBvZiBkaXNr IHNwYWNlLjwvcD4KICAgIDxwPjxicj4KICAgICAgRmVybmFuZG88YnI+CiAgICA8L3A+CiAgICA8 YnI+CiAgICA8ZGl2IGNsYXNzPSJtb3otY2l0ZS1wcmVmaXgiPk9uIDA4LzA4LzIwMTcgMDY6MDks IE1vYWNpciBGZXJyZWlyYQogICAgICB3cm90ZTo8YnI+CiAgICA8L2Rpdj4KICAgIDxibG9ja3F1 b3RlIHR5cGU9ImNpdGUiCmNpdGU9Im1pZDpEQjZQMTkwTUIwMjgwMUVBMDg5MkIzOEY1MDMyMjA4 OTZDODhBMEBEQjZQMTkwTUIwMjgwLkVVUlAxOTAuUFJPRC5PVVRMT09LLkNPTSI+CiAgICAgIDxt ZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0idGV4dC9odG1sOwogICAgICAg IGNoYXJzZXQ9d2luZG93cy0xMjUyIj4KICAgICAgPHN0eWxlIHR5cGU9InRleHQvY3NzIiBzdHls ZT0iZGlzcGxheTpub25lOyI+PCEtLSBQIHttYXJnaW4tdG9wOjA7bWFyZ2luLWJvdHRvbTowO30g LS0+PC9zdHlsZT4KICAgICAgPGRpdiBpZD0iZGl2dGFnZGVmYXVsdHdyYXBwZXIiCnN0eWxlPSJm b250LXNpemU6MTJwdDtjb2xvcjojMDAwMDAwO2ZvbnQtZmFtaWx5OkNhbGlicmksSGVsdmV0aWNh LHNhbnMtc2VyaWY7IgogICAgICAgIGRpcj0ibHRyIj4KICAgICAgICA8cD5GZXJuYW5kbyw8L3A+ CiAgICAgICAgPHA+PGJyPgogICAgICAgIDwvcD4KICAgICAgICA8cD5MZXQncyBzZWUgd2hhdCBw ZW9wbGUgc2F5Li4uIEJ1dCB0aGlzIGlzIHdoYXQgSSB1bmRlcnN0b29kCiAgICAgICAgICBSZWQg SGF0IHNheXMgaXMgdGhlIGJlc3QgcGVyZm9ybWFuY2UgbW9kZWwuIFRoaXMgaXMgdGhlIG1haW4K ICAgICAgICAgIHJlYXNvbiB0byBvcGVuIHRoaXMgZGlzY3Vzc2lvbiBiZWNhdXNlIGFzIGxvbmcg YXMgSSBjYW4gc2VlLAogICAgICAgICAgc29tZSBvZiB5b3UgaW4gdGhlIGNvbW11bml0eSwgZG8g bm90IGFncmVlLjxicj4KICAgICAgICA8L3A+CiAgICAgICAgPGJyPgogICAgICAgIDxwPkJ1dCB3 aGVuIEkgdGhpbmsgYWJvdXQgYSAiZGlzdHJpYnV0ZWQgZmlsZSBzeXN0ZW0iLCB0aGF0IGNhbgog ICAgICAgICAgbWFrZSBhbnkgbnVtYmVyIG9mIGNvcGllcyB5b3Ugd2FudCwgaXQgZG9lcyBub3Qg bWFrZSBzZW5zZQogICAgICAgICAgdXNpbmcgYSBSQUlEZWQgYnJpY2ssIHdoYXQgaXQgbWFrZXMg c2Vuc2UgaXMgdG8gdXNlIEpCT0QuPC9wPgogICAgICAgIDxwPjxicj4KICAgICAgICA8L3A+CiAg ICAgICAgPHA+TW9hY2lyPGJyPgogICAgICAgIDwvcD4KICAgICAgICA8YnI+CiAgICAgICAgPGRp diBzdHlsZT0iY29sb3I6IHJnYig0OSwgNTUsIDU3KTsiPgogICAgICAgICAgPGhyIHRhYmluZGV4 PSItMSIgc3R5bGU9ImRpc3BsYXk6aW5saW5lLWJsb2NrOyB3aWR0aDo5OCUiPgogICAgICAgICAg PGRpdiBpZD0iZGl2UnBseUZ3ZE1zZyIgZGlyPSJsdHIiPjxmb250IHN0eWxlPSJmb250LXNpemU6 MTFwdCIKICAgICAgICAgICAgICBjb2xvcj0iIzAwMDAwMCIgZmFjZT0iQ2FsaWJyaSwgc2Fucy1z ZXJpZiI+PGI+RnJvbTo8L2I+CiAgICAgICAgICAgICAgPGEgY2xhc3M9Im1vei10eHQtbGluay1h YmJyZXZpYXRlZCIgaHJlZj0ibWFpbHRvOmZlcm5hbmRvLmZyZWRpYW5pQHVweC5jb20uYnIiPmZl cm5hbmRvLmZyZWRpYW5pQHVweC5jb20uYnI8L2E+CiAgICAgICAgICAgICAgPGEgY2xhc3M9Im1v ei10eHQtbGluay1yZmMyMzk2RSIgaHJlZj0ibWFpbHRvOmZlcm5hbmRvLmZyZWRpYW5pQHVweC5j b20uYnIiPiZsdDtmZXJuYW5kby5mcmVkaWFuaUB1cHguY29tLmJyJmd0OzwvYT4gb24gYmVoYWxm IG9mIEZFUk5BTkRPCiAgICAgICAgICAgICAgRlJFRElBTkkgPGEgY2xhc3M9Im1vei10eHQtbGlu ay1yZmMyMzk2RSIgaHJlZj0ibWFpbHRvOmZlcm5hbmRvLmZyZWRpYW5pQHVweC5jb20iPiZsdDtm ZXJuYW5kby5mcmVkaWFuaUB1cHguY29tJmd0OzwvYT48YnI+CiAgICAgICAgICAgICAgPGI+U2Vu dDo8L2I+IFR1ZXNkYXksIEF1Z3VzdCA4LCAyMDE3IDM6MDggQU08YnI+CiAgICAgICAgICAgICAg PGI+VG86PC9iPiBNb2FjaXIgRmVycmVpcmE8YnI+CiAgICAgICAgICAgICAgPGI+Q2M6PC9iPiBD b2xpbiBDb2U7IDxhIGNsYXNzPSJtb3otdHh0LWxpbmstYWJicmV2aWF0ZWQiIGhyZWY9Im1haWx0 bzp1c2Vyc0BvdmlydC5vcmciPnVzZXJzQG92aXJ0Lm9yZzwvYT48YnI+CiAgICAgICAgICAgICAg PGI+U3ViamVjdDo8L2I+IFJlOiBbb3ZpcnQtdXNlcnNdIEdvb2QgcHJhY3RpY2VzPC9mb250Pgog ICAgICAgICAgICA8ZGl2PqA8L2Rpdj4KICAgICAgICAgIDwvZGl2PgogICAgICAgICAgPGRpdj4K ICAgICAgICAgICAgPGRpdiBkaXI9Imx0ciI+CiAgICAgICAgICAgICAgPGRpdj5Nb2FjaXIsIEkg dW5kZXJzdGFuZCB0aGF0IGlmIHlvdSBkbyB0aGlzIHR5cGUgb2YKICAgICAgICAgICAgICAgIGNv bmZpZ3VyYXRpb24geW91IHdpbGwgYmUgc2V2ZXJlbHkgaW1wYWN0ZWQgb24gc3RvcmFnZQogICAg ICAgICAgICAgICAgcGVyZm9ybWFuY2UsIHNwZWNpYWxseSBmb3Igd3JpdGVzLiBFdmVuIGlmIHlv dSBoYXZlIGEKICAgICAgICAgICAgICAgIEhhcmR3YXJlIFJBSUQgQ29udHJvbGxlciB3aXRoIFdy aXRlYmFjayBjYWNoZSB5b3Ugd2lsbAogICAgICAgICAgICAgICAgaGF2ZSBhIHNpZ25pZmljYW50 IHBlcmZvcm1hbmNlIHBlbmFsdHkgYW5kIG1heSBub3QgZnVsbHkKICAgICAgICAgICAgICAgIHVz ZSBhbGwgdGhlIHJlc291cmNlcyB5b3UgbWVudGlvbmVkIHlvdSBoYXZlLjxicj4KICAgICAgICAg ICAgICAgIDxicj4KICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICBGZXJuYW5kbzxi cj4KICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX2V4dHJh Ij48YnI+CiAgICAgICAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPjIwMTctMDgtMDcg MTA6MDMgR01ULTAzOjAwIE1vYWNpcgogICAgICAgICAgICAgICAgRmVycmVpcmEgPHNwYW4gZGly PSJsdHIiPgogICAgICAgICAgICAgICAgICAmbHQ7PGEgaHJlZj0ibWFpbHRvOm1vYWNpcmZlcnJl aXJhQGhvdG1haWwuY29tIgogICAgICAgICAgICAgICAgICAgIHRhcmdldD0iX2JsYW5rIiBtb3ot ZG8tbm90LXNlbmQ9InRydWUiPm1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY29tPC9hPiZndDs8L3Nw YW4+Ojxicj4KICAgICAgICAgICAgICAgIDxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIg c3R5bGU9Im1hcmdpbjowIDAgMAogICAgICAgICAgICAgICAgICAuOGV4OyBib3JkZXItbGVmdDox cHggI2NjYyBzb2xpZDsgcGFkZGluZy1sZWZ0OjFleCI+CiAgICAgICAgICAgICAgICAgIDxkaXYg ZGlyPSJsdHIiPgogICAgICAgICAgICAgICAgICAgIDxkaXYgaWQ9Im1fMTg3NDg1OTg1OTg0Mjc2 MzEwNGRpdnRhZ2RlZmF1bHR3cmFwcGVyIgogICAgICAgICAgICAgICAgICAgICAgZGlyPSJsdHIi IHN0eWxlPSJmb250LXNpemU6MTJwdDsgY29sb3I6IzAwMDAwMDsKICAgICAgICAgICAgICAgICAg ICAgIGZvbnQtZmFtaWx5OkNhbGlicmksSGVsdmV0aWNhLHNhbnMtc2VyaWYiPgogICAgICAgICAg ICAgICAgICAgICAgPHA+SGkgQ29saW4sPC9wPgogICAgICAgICAgICAgICAgICAgICAgPHA+PGJy PgogICAgICAgICAgICAgICAgICAgICAgPC9wPgogICAgICAgICAgICAgICAgICAgICAgPHA+VGFr ZSBhIGxvb2sgb24gRGV2aW4ncyByZXNwb25zZS4gQWxzbywgcmVhZCB0aGUKICAgICAgICAgICAg ICAgICAgICAgICAgZG9jIGhlIHNoYXJlZCB0aGF0IGdpdmVzIHNvbWUgaGludHMgb24gaG93IHRv CiAgICAgICAgICAgICAgICAgICAgICAgIGRlcGxveSBHbHVzdGVyLjwvcD4KICAgICAgICAgICAg ICAgICAgICAgIDxwPjxicj4KICAgICAgICAgICAgICAgICAgICAgIDwvcD4KICAgICAgICAgICAg ICAgICAgICAgIDxwPkl0IGlzIG1vcmUgbGlrZSB0aGF0IGlmIHlvdSB3YW50CiAgICAgICAgICAg ICAgICAgICAgICAgIGhpZ2gtcGVyZm9ybWFuY2UgeW91IHNob3VsZCBoYXZlIHRoZSBicmlja3MK ICAgICAgICAgICAgICAgICAgICAgICAgY3JlYXRlZCBhcyBSQUlEICg1IG9yIDYpoGJ5IHRoZSBz ZXJ2ZXIncyBkaXNrCiAgICAgICAgICAgICAgICAgICAgICAgIGNvbnRyb2xsZXIgYW5kIHRoZW0g YXNzZW1ibGUgYSBKQk9EIEdsdXN0ZXJGUy4KICAgICAgICAgICAgICAgICAgICAgICAgVGhlIGF0 dGFjaGVkIGRvY3VtZW50IGlzIEdsdXN0ZXIgc3BlY2lmaWMgYW5kCiAgICAgICAgICAgICAgICAg ICAgICAgIG5vdCBmb3Igb1ZpcnQuIEJ1dCBhdCB0aGlzIHBvaW50IEkgdGhpbmsgdGhhdAogICAg ICAgICAgICAgICAgICAgICAgICBoYXZpbmcgU1NEIHdpbGwgbm90IGJlIGEgcGx1cyBhcyB1c2lu ZyB0aGUgUkFJRAogICAgICAgICAgICAgICAgICAgICAgICBjb250cm9sbGVyIEdsdXN0ZXIgd2ls bCBub3QgYmUgYXdhcmUgb2YgdGhlIFNTRC4KICAgICAgICAgICAgICAgICAgICAgICAgUmVnYXJk aW5nIHRoZSBPUywgbXkgaWRlYSBpcyB0byBoYXZlIGEgUkFJRCAxLAogICAgICAgICAgICAgICAg ICAgICAgICBtYWRlIG9mIDIgbG93IGNvc3QgSEREcywgdG8gaW5zdGFsbCBpdC48L3A+CiAgICAg ICAgICAgICAgICAgICAgICA8cD48YnI+CiAgICAgICAgICAgICAgICAgICAgICA8L3A+CiAgICAg ICAgICAgICAgICAgICAgICA8cD5TbyBmYXIsIGJhc2VkIG9uIHRoZSBpbmZvcm1hdGlvbiByZWNl aXZlZKBJCiAgICAgICAgICAgICAgICAgICAgICAgIHNob3VsZKBjcmVhdGUgYSBzaW5nbGUgUkFJ RCA1IG9yIDYgb24gZWFjaAogICAgICAgICAgICAgICAgICAgICAgICBzZXJ2ZXIgYW5kIHRoZW4g dXNlIHRoaXMgZGlzayBhcyBhIGJyaWNrIHRvCiAgICAgICAgICAgICAgICAgICAgICAgIGNyZWF0 ZSBteSBHbHVzdGVyIGNsdXN0ZXIsIG1hZGUgb2YgMiByZXBsaWNhcyArCiAgICAgICAgICAgICAg ICAgICAgICAgIDEgYXJiaXRlci4gV2hhdCBpcyBuZXcgZm9yIG1lIGlzIHRoZSBkZXRhaWwgdGhh dAogICAgICAgICAgICAgICAgICAgICAgICB0aGUgYXJiaXRlciBkb2VzIG5vdCBuZWVkIGEgbG90 IG9mIHNwYWNlIGFzIGl0CiAgICAgICAgICAgICAgICAgICAgICAgIG9ubHkga2VlcHMgbWV0YSBk YXRhLjwvcD4KICAgICAgICAgICAgICAgICAgICAgIDxwPjxicj4KICAgICAgICAgICAgICAgICAg ICAgIDwvcD4KICAgICAgICAgICAgICAgICAgICAgIDxwPlRoYW5rcyBmb3IgeW91ciByZXNwb25z ZSE8YnI+CiAgICAgICAgICAgICAgICAgICAgICA8L3A+CiAgICAgICAgICAgICAgICAgICAgICBN b2FjaXI8YnI+CiAgICAgICAgICAgICAgICAgICAgICA8YnI+CiAgICAgICAgICAgICAgICAgICAg ICA8ZGl2IHN0eWxlPSJjb2xvcjpyZ2IoNDksNTUsNTcpIj4KICAgICAgICAgICAgICAgICAgICAg ICAgPGhyIHN0eWxlPSJkaXNwbGF5OmlubGluZS1ibG9jazsgd2lkdGg6OTglIj4KICAgICAgICAg ICAgICAgICAgICAgICAgPGRpdiBpZD0ibV8xODc0ODU5ODU5ODQyNzYzMTA0ZGl2UnBseUZ3ZE1z ZyIKICAgICAgICAgICAgICAgICAgICAgICAgICBkaXI9Imx0ciI+PGZvbnQgc3R5bGU9ImZvbnQt c2l6ZToxMXB0IgogICAgICAgICAgICAgICAgICAgICAgICAgICAgY29sb3I9IiMwMDAwMDAiIGZh Y2U9IkNhbGlicmksIHNhbnMtc2VyaWYiPjxiPkZyb206PC9iPgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgQ29saW4gQ29lICZsdDs8YQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBo cmVmPSJtYWlsdG86Y29saW4uY29lQGdtYWlsLmNvbSIKICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgdGFyZ2V0PSJfYmxhbmsiIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+Y29saW4uY29lQGdt YWlsLmNvbTwvYT4mZ3Q7PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgPGI+U2VudDo8 L2I+IE1vbmRheSwgQXVndXN0IDcsIDIwMTcgMTI6NDEgUE0KICAgICAgICAgICAgICAgICAgICAg ICAgICAgIDxkaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXYgY2xhc3M9Img1 Ij48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGI+VG86PC9iPiBNb2FjaXIg RmVycmVpcmE8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGI+Q2M6PC9iPiA8 YQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaHJlZj0ibWFpbHRvOnVzZXJzQG92 aXJ0Lm9yZyIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRhcmdldD0iX2JsYW5r IiBtb3otZG8tbm90LXNlbmQ9InRydWUiPnVzZXJzQG92aXJ0Lm9yZzwvYT48YnI+CiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgPGI+U3ViamVjdDo8L2I+IFJlOiBbb3ZpcnQtdXNlcnNd IEdvb2QKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBwcmFjdGljZXM8L2Rpdj4KICAg ICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAg IDwvZm9udD4KICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2PqA8L2Rpdj4KICAgICAgICAg ICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgIDxkaXY+CiAgICAg ICAgICAgICAgICAgICAgICAgICAgPGRpdiBjbGFzcz0iaDUiPgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgPGRpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBkaXI9Imx0 ciI+SGkKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2Pjxicj4KICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICA8ZGl2PkkganVzdCB0aG91Z2h0IHRoYXQgeW91J2QgZG8KICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIGhhcmR3YXJlIFJBSUQgaWYgeW91IGhhZCB0aGUKICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIGNvbnRyb2xsZXIgb3IgSkJPRCBpZiB5b3UgZGlkbid0LqAg SW4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGhpbmRzaWdodCwgYSBzZXJ2ZXIg d2l0aCA0MEdicHMgTklDcwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaXMgcHJl dHR5IGxpa2VseSB0byBoYXZlIGEgaGFyZHdhcmUKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIFJBSUQgY29udHJvbGxlci6gIEkndmUgbmV2ZXIgZG9uZSBKQk9ECiAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICB3aXRoIGhhcmR3YXJlIFJBSUQuoCBJIHRoaW5rIGhhdmlu ZyBhCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBzaW5nbGUgZ2x1c3RlciBicmlj ayBvbiBoYXJkd2FyZSBKQk9ECiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB3b3Vs ZCBiZSByaXNraWVyIHRoYW4gbXVsdGlwbGUgYnJpY2tzLAogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgZWFjaCBvbiBhIHNpbmdsZSBkaXNrLCBidXQgdGhhdHMgbm90CiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICBiYXNlZCBvbiBhbnl0aGluZyBvdGhlciB0aGFuIG15 CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBwcmVqdWRpY2VzLjwvZGl2PgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXY+PGJyPgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXY+ SSB0aG91Z2h0IGdsdXN0ZXIgdGllcmluZyB3YXMgZm9yCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICB0aGUgbW9zdCBmcmVxdWVudGx5IGFjY2Vzc2VkIGZpbGVzLCBpbgogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgd2hpY2ggY2FzZSBhbGwgdGhlIFZNcyBkaXNrcyB3 b3VsZCBlbmQKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHVwIGluIHRoZSBob3Qg dGllci6gIEhvd2V2ZXIsIEkgaGF2ZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg YmVlbiB3cm9uZyBiZWZvcmUuLi48L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICA8ZGl2Pjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2PkkganVzdCB3YW50ZWQgdG8ga25vdyB3aGVy ZSB0aGUgT1MKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHdhcyBnb2luZyBhcyBJ IGRpZG4ndCBzZWUgaXQgbWVudGlvbmVkCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICBpbiB0aGUgT1AuoCBOb3JtYWxseSwgSSdkIGhhdmUgdGhlIE9TCiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBvbiBhIFJBSUQxIGJ1dCBpbiB5b3VyIGNhc2UgdGhhdHMgYQogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbG90IG9mIHdhc3RlZCBkaXNrLjwvZGl2Pgog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXY+PGJyPgogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxk aXY+SG9uZXN0bHksIEkgdGhpbmsgWWFuaXYncyBhbnN3ZXIKICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHdhcyBmYXIgYmV0dGVyIHRoYW4gbXkgb3duIGFuZCBtYWRlCiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICB0aGUgaW1wb3J0YW50IHBvaW50IGFib3V0IGhhdmlu ZyBhbgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYXJiaXRlci6gPC9kaXY+CiAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdj48YnI+CiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRp dj5UaGFua3M8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+CiAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPk9u IE1vbiwgQXVnIDcsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAyMDE3IGF0IDU6 NTYgUE0sIE1vYWNpciBGZXJyZWlyYSA8c3BhbgogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICBkaXI9Imx0ciI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICZs dDs8YQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGhyZWY9Im1haWx0bzpt b2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbSIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICB0YXJnZXQ9Il9ibGFuayIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICBtb3otZG8tbm90LXNlbmQ9InRydWUiPm1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY29tPC9h PiZndDs8L3NwYW4+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB3cm90ZTo8YnI+ CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8YmxvY2txdW90ZSBjbGFzcz0iZ21h aWxfcXVvdGUiCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHN0eWxlPSJtYXJn aW46MCAwIDAgLjhleDsKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYm9yZGVy LWxlZnQ6MXB4ICNjY2Mgc29saWQ7CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IHBhZGRpbmctbGVmdDoxZXgiPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8 ZGl2IGRpcj0ibHRyIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2 CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpZD0ibV8xODc0ODU5ODU5 ODQyNzYzMTA0bV80ODMxODg2ODgzMzA1NjcyNzE4ZGl2dGFnZGVmYXVsdHdyYXBwZXIiCiAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBkaXI9Imx0ciIgc3R5bGU9ImZvbnQt c2l6ZToxMnB0OwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY29sb3I6 IzAwMDAwMDsKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZvbnQtZmFt aWx5OkNhbGlicmksSGVsdmV0aWNhLHNhbnMtc2VyaWYiPgogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgPHA+SGkgQ29saW4sPC9wPgogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgPHA+PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgPC9wPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPHA+ PHNwYW4+SSBhbSBpbiBQb3J0dWdhbDwvc3Bhbj4sCiAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIHNvIHNvcnJ5IGZvciB0aGlzIGxhdGUKICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgcmVzcG9uc2UuIEl0IGlzIHF1aXRlCiAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbmZ1c2luZyBmb3IgbWUsIHBsZWFzZQog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb25zaWRlcjo8L3A+CiAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2PjxiPjxicj4KICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9iPjE8Yj4gLSA8L2I+V2hhdCBp ZiB0aGUKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgUkFJRCBpcyBk b25lIGJ5IHRoZSBzZXJ2ZXIncwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICBkaXNrIGNvbnRyb2xsZXIsIG5vdCBieQogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICBzb2Z0d2FyZT88L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IDxwPjIgLTxiPiA8L2I+Rm9yIEpCT0QgSSBhbQogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICBqdXN0IHVzaW5nIGdkZXBsb3kgdG8gZGVwbG95CiAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIGl0LiBIb3dldmVyLCBJIGFtIG5vdCB1c2luZwog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0aGUgb1ZpcnQgbm9kZSBH VUkgdG8gZG8gdGhpcy48L3A+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICA8cD48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L3A+CiAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8cD4zIC08Yj4gPC9iPkFzIHRo ZSBWTQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAucWNvdzKgZmls ZXMgYXJlIHF1aXRlIGJpZywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgdGllcmluZyB3b3VsZCBvbmx5IGhlbHAgaWYKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgbWFkZSBieSBhbiBpbnRlbGxpZ2VudCBzeXN0ZW0KICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdGhhdCB1c2VzIFNTRCBmb3IgY2h1bmtzIG9m CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRhdGEgbm90IGZvciB0 aGUgZW50aXJlIC5xY293MgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICBmaWxlLiBCdXQgSSBndWVzcyB0aGlzIGlzIGEKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgcHJvYmxlbSBldmVyeWJvZHkgZWxzZSBoYXMuCiAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIFNvLCBEbyB5b3Uga25vdyBob3cgdGllcmluZwog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB3b3JrcyBpbiBHbHVzdGVy Pzxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvcD4KICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxwPjxicj4KICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIDwvcD4KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIDxwPjQgLaBJIGFtIHB1dHRpbmcgdGhlIE9TIG9uCiAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRoZSBmaXJzdCBkaXNrLiBIb3dldmVyLCB3b3Vs ZAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB5b3UgZG8gZGlmZmVy ZW50bHk/PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9wPgog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPHA+PGJyPgogICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9wPgogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgTW9hY2lyPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg PGRpdiBzdHlsZT0iY29sb3I6cmdiKDQ5LDU1LDU3KSI+CiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIDxocgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIHN0eWxlPSJkaXNwbGF5OmlubGluZS1ibG9jazsKICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICB3aWR0aDo5OCUiPgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICA8ZGl2CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgaWQ9Im1fMTg3NDg1OTg1OTg0Mjc2MzEwNG1fNDgzMTg4Njg4MzMwNTY3 MjcxOGRpdlJwbHlGd2RNc2ciCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgZGlyPSJsdHIiPjxmb250CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBzdHlsZT0iZm9udC1zaXplOjExcHQiCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICBjb2xvcj0iIzAwMDAwMCIKICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZhY2U9IkNhbGlicmksIHNhbnMtc2VyaWYiPjxi PkZyb206PC9iPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg Q29saW4gQ29lICZsdDs8YQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICBocmVmPSJtYWlsdG86Y29saW4uY29lQGdtYWlsLmNvbSIKICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdGFyZ2V0PSJfYmxhbmsiCiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1vei1kby1ub3Qtc2VuZD0i dHJ1ZSI+Y29saW4uY29lQGdtYWlsLmNvbTwvYT4mZ3Q7PGJyPgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgPGI+U2VudDo8L2I+IE1vbmRheSwKICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEF1Z3VzdCA3LCAyMDE3IDQ6NDgg QU08YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8Yj5U bzo8L2I+IE1vYWNpciBGZXJyZWlyYTxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIDxiPkNjOjwvYj4gPGEKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgaHJlZj0ibWFpbHRvOnVzZXJzQG92aXJ0Lm9yZyIKICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdGFyZ2V0PSJfYmxhbmsi CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1vei1kby1u b3Qtc2VuZD0idHJ1ZSI+dXNlcnNAb3ZpcnQub3JnPC9hPjxicj4KICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxiPlN1YmplY3Q6PC9iPiBSZToKICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFtvdmlydC11c2Vyc10gR29vZAog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcHJhY3RpY2VzPC9m b250PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXY+oDwv ZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdj4KICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2IGRpcj0ibHRyIj4xKSBSQUlENSBt YXkKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGJlIGEgcGVy Zm9ybWFuY2UgaGl0LSA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICA8ZGl2Pjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgPGRpdj48c3Bhbj4yKSBJJ2QgYmUKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBpbmNsaW5lZCB0byBkbyB0aGlzIGFzCiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgSkJPRCBieSBjcmVhdGluZyBhCiAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZGlzdHJpYnV0ZWQg ZGlzcGVyc2UKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICB2b2x1bWUgb24gZWFjaAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIHNlcnZlci6gIFNvbWV0aGluZwogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIGxpa2UKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICA8ZGl2Pjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICA8ZGl2PmVjaG8gZ2x1c3RlcgogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdm9sdW1lIGNyZWF0ZQogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZGlzcGVyc2V2b2wK ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRpc3Bl cnNlLWRhdGEgNQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgcmVkdW5kYW5jeSAyIFw8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICA8ZGl2PiQoZm9yIFNFUlZFUiBpbiBhCiAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBiIGM7IGRvIGZvciBCUklDSyBp bgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJChz ZXEgMSA1KTsgZG8gZWNobwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgLWUKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICJzZXJ2ZXIke1NFUlZFUn06L2JyaWNrL2JyaWNrLTx3YnI+JHtTRVJWRVJ9JHtC UklDS30vYnJpY2sKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIFxjIjsgZG9uZTsgZG9uZSk8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICA8ZGl2Pjxicj4KICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgPC9zcGFuPgogICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICA8ZGl2PjMpIEkgdGhpbmsgdGhlCiAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYWJvdmUuIDwvZGl2PgogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8c3Bhbj4KICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2Pjxicj4KICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2PjQpIEdsdXN0 ZXIgZG9lcwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgc3VwcG9ydCB0aWVyaW5nLCBidXQKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIElJUkMgeW91J2QgbmVlZCB0aGUKICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHNhbWUgbnVtYmVyIG9mIFNTRAogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYXMgc3BpbmRs ZSBkcml2ZXMuoAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgVGhlcmUgbWF5IGJlIGFub3RoZXIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIHdheSB0byB1c2UgdGhlIFNTRAogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYXMgYSBmYXN0IGNhY2hlLqA8L2Rp dj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2 Pjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8 L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8 ZGl2PldoZXJlIGFyZSB5b3UKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHB1dHRpbmcgdGhlIE9TPzwvZGl2PgogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXY+PGJyPgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXY+SG9wZSBJIHVuZGVyc3Rvb2QKICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRoZSBxdWVz dGlvbi4uLjwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIDxkaXY+PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIDxkaXY+VGhhbmtzPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIDwvc3Bhbj48L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICA8c3Bhbj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+CiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj5P bgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFN1biwg QXVnIDYsIDIwMTcgYXQKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAxMDo0OSBQTSwgTW9hY2lyCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgRmVycmVpcmEgPHNwYW4KICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRpcj0ibHRyIj4KICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICZsdDs8YQogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBocmVmPSJtYWlsdG86bW9hY2ly ZmVycmVpcmFAaG90bWFpbC5jb20iCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIHRhcmdldD0iX2JsYW5rIgogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtb3otZG8tbm90LXNlbmQ9InRydWUiPm1v YWNpcmZlcnJlaXJhQGhvdG1haWwuY29tPC9hPiZndDs8L3NwYW4+CiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgd3JvdGU6PGJyPgogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxibG9ja3F1b3RlCiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjbGFzcz0iZ21haWxf cXVvdGUiCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICBzdHlsZT0ibWFyZ2luOjAgMCAwCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAuOGV4OwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgYm9yZGVyLWxlZnQ6MXB4ICNjY2MKICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHNvbGlkOwogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcGFkZGluZy1sZWZ0OjFleCI+CiAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2IGRp cj0ibHRyIj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgPGRpdgppZD0ibV8xODc0ODU5ODU5ODQyNzYzMTA0bV80ODMxODg2ODgzMzA1NjcyNzE4 bV8yNDYwOTg1NjkxNzQ2NDk4MzIyZGl2dGFnZGVmYXVsdHdyYXBwZXIiCiAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZGlyPSJsdHIiCiAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgc3R5bGU9 ImZvbnQtc2l6ZToxMnB0OwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIGNvbG9yOiMwMDAwMDA7CiAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZm9udC1mYW1pbHk6Q2FsaWJyaSxIZWx2ZXRp Y2Esc2Fucy1zZXJpZiI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgPHA+PHNwYW4+SSBhbQogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgd2lsbGluZyB0bwogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYXNzZW1ibGUgYQogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgb1Zp cnQgInBvZCIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBtYWRlIG9mIDMKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIHNlcnZlcnMsIGVhY2gKICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHdpdGggMiBDUFUKICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHNvY2tldHMg b2YgMTIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIGNvcmVzLCAyNTZHQgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgUkFNLCA3IEhERAogICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgMTBLLCAxIFNTRC4KICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFRoZSBpZGVhIGlz IHRvCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICB1c2UgR2x1c3RlckZTCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICB0byBwcm92aWRlIEhBCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmb3IgdGhlIFZNcy4KICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFRoZSAzIHNl cnZlcnMKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIGhhdmUgYSBkdWFsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICA0MEdiIE5JQyBhbmQgYQogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZHVhbCAxMEdiIE5JQy4KICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFNvIG15 CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICBpbnRlbnRpb24gaXMKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIHRvIGNyZWF0ZSBhCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICBsb29wIGxpa2UgYQogICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgc2VydmVyCiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0cmlhbmds ZSB1c2luZwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgdGhlIDQwR2IgTklDcwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgZm9yCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICB2aXJ0dWFsaXphdGlvbgogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZmlsZXMgKFZNcwog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg LnFjb3cyKSBhY2Nlc3MKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIGFuZCB0byBtb3ZlCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICBWTXMgYXJvdW5kIHRoZQogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcG9kIChlYXN0CiAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAv d2VzdCB0cmFmZmljKQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgd2hpbGUgdXNpbmcKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRoZSAxMEdiCiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpbnRlcmZhY2VzIGZvcgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZ2l2 aW5nCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICBzZXJ2aWNlcyB0bwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgdGhlIG91dHNpZGUKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHdvcmxkCiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAobm9ydGgvc291dGgKICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRyYWZm aWMpLjwvc3Bhbj48L3A+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgPHA+PGJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgPHNwYW4+PC9zcGFuPjwvcD4KICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8cD5UaGlzIHNhaWQsIG15 CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICBmaXJzdCBxdWVzdGlvbgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgaXM6IEhvdyBzaG91bGQKICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEkgZGVwbG95CiAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBHbHVzdGVyRlMgaW4K ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IHN1Y2ggb1ZpcnQKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHNjZW5hcmlvPyBNeQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgcXVlc3Rpb25zIGFyZTo8L3A+CiAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPHA+PGJyPgogICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvcD4K ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8 cD4xIC0gU2hvdWxkIEkKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIGNyZWF0ZSAzIFJBSUQKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIChpLmUuOiBSQUlECiAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA1KSwgb25lIG9uCiAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBl YWNoIG9WaXJ0CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBub2RlLCBhbmQgdGhlbgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgY3JlYXRlIGEKICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEdsdXN0ZXJGUwogICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdXNpbmcgdGhl bT88L3A+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgPHA+MiAtIEluc3RlYWQsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICBzaG91bGQgSQogICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY3JlYXRlIGEgSkJPRAogICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYXJyYXkgbWFk ZSBvZgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgYWxsIHNlcnZlcidzCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICBkaXNrcz88L3A+CiAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPHA+MyAtIFdoYXQgaXMKICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRoZSBiZXN0CiAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBH bHVzdGVyCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICBjb25maWd1cmF0aW9uCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICB0byBwcm92aWRlIGZvcgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgSEEgd2hpbGUgbm90CiAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb25z dW1pbmcgdG9vCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBtdWNoIGRpc2sKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgIHNwYWNlPzxicj4KICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L3A+CiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPHA+NCAtIERvZXMgYQogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgb1ZpcnQK ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IGh5cGVydmlzb3IgcG9kCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICBsaWtlIEkgYW0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIHBsYW5uaW5nIHRvCiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBidWlsZCwgYW5kIHRoZQog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg dmlydHVhbGl6YXRpb24KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIGVudmlyb25tZW50LAogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYmVuZWZpdHMgZnJvbQogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdGllcmluZyB3aGVu CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICB1c2luZyBhIFNTRAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgZGlzaz8gQW5kIHllcywKICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHdpbGwgR2x1c3RlcgogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZG8gaXQgYnkKICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRl ZmF1bHQgb3IgSQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgaGF2ZSB0bwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgY29uZmlndXJlIGl0CiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0byBkbyBzbz88L3A+CiAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPHA+PGJyPgog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwv cD4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICA8cD5BdCB0aGUgYm90dG9tCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICBsaW5lLCB3aGF0IGlzCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0aGUgZ29vZAogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcHJhY3RpY2UgZm9y CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICB1c2luZwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgR2x1c3RlckZTIGluCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICBzbWFsbCBwb2RzIGZvcgogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZW50ZXJwcmlzZXM/PGJyPgog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwv cD4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICA8cD48YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgPC9wPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIDxwPllvdQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgb3Bpbmlvbi9mZWVkYmFjawogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgd2lsbCBiZSByZWFsbHkKICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGFw cHJlY2lhdGVkITwvcD4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICA8c3BhbgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgY2xhc3M9Im1fMTg3NDg1OTg1OTg0Mjc2MzEwNG1fNDgzMTg4 Njg4MzMwNTY3MjcxOEhPRW5aYiI+PGZvbnQKY29sb3I9IiM4ODg4ODgiPgogICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPHA+TW9hY2lyPGJy PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgPC9wPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgPC9mb250Pjwvc3Bhbj48L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgPGJyPgpfX19fX19fX19fX19fX19fX19fX19fX19f X19fX188d2JyPl9fX19fX19fX19fX19fX19fPGJyPgogICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgVXNlcnMgbWFpbGluZyBsaXN0PGJyPgogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGEKICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaHJlZj0ibWFpbHRv OlVzZXJzQG92aXJ0Lm9yZyIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgdGFyZ2V0PSJfYmxhbmsiCiAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+VXNlcnNA b3ZpcnQub3JnPC9hPjxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIDxhCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5m by91c2VycyIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgcmVsPSJub3JlZmVycmVyIgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICB0YXJnZXQ9Il9ibGFuayIKICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5o dHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW48d2JyPi9saXN0aW5mby91c2VyczwvYT48YnI+ CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8YnI+ CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9ibG9j a3F1b3RlPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8 L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPGJy PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9zcGFuPjwvZGl2Pgog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9i bG9ja3F1b3RlPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAg ICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAg ICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAg ICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAg ICAgX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fPHdicj5fX19fX19fX19fX19fX19fXzxi cj4KICAgICAgICAgICAgICAgICAgVXNlcnMgbWFpbGluZyBsaXN0PGJyPgogICAgICAgICAgICAg ICAgICA8YSBocmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQub3JnIgogICAgICAgICAgICAgICAgICAg IG1vei1kby1ub3Qtc2VuZD0idHJ1ZSI+VXNlcnNAb3ZpcnQub3JnPC9hPjxicj4KICAgICAgICAg ICAgICAgICAgPGEKICAgICAgICAgICAgICAgICAgICBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQu b3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiCiAgICAgICAgICAgICAgICAgICAgcmVsPSJub3Jl ZmVycmVyIiB0YXJnZXQ9Il9ibGFuayIKICAgICAgICAgICAgICAgICAgICBtb3otZG8tbm90LXNl bmQ9InRydWUiPmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvPHdicj5tYWlsbWFuL2xpc3RpbmZvL3Vz ZXJzPC9hPjxicj4KICAgICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAgPC9ibG9j a3F1b3RlPgogICAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICAgIDxicj4KICAgICAgICAg ICAgPC9kaXY+CiAgICAgICAgICA8L2Rpdj4KICAgICAgICA8L2Rpdj4KICAgICAgPC9kaXY+CiAg ICA8L2Jsb2NrcXVvdGU+CiAgICA8YnI+CiAgPC9ib2R5Pgo8L2h0bWw+CgotLS0tLS0tLS0tLS0t LUE5NUNGMUMwRTQzMTIwNDlBNTQwQkI2QS0tCg== --===============6994257392034748241==-- From karli at inparadise.se Tue Aug 8 13:39:51 2017 Content-Type: multipart/mixed; boundary="===============0625199488519932082==" MIME-Version: 1.0 From: =?utf-8?q?Karli_Sj=C3=B6berg_=3Ckarli_at_inparadise=2Ese=3E?= To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Tue, 08 Aug 2017 15:39:45 +0200 Message-ID: <1502199585.14509.28.camel@inparadise.se> In-Reply-To: 02172d33-e14e-aa8d-98f5-d920573e281c@upx.com --===============0625199488519932082== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On tis, 2017-08-08 at 10:24 -0300, FERNANDO FREDIANI wrote: > That's something on the way RAID works, regardless what most=C2=A0 > 'super-ultra' powerfull hardware controller you may have. RAID 5 or > 6=C2=A0 > will never have the same write performance as a RAID 10 o 0 for > example.=C2=A0 > Writeback caches can deal with bursts well but they have a limit=C2=A0 > therefore there will always be a penalty compared to what else you > could=C2=A0 > have. > = > If you have a continuous stream of data (a big VM deployment or a > large=C2=A0 > data copy) there will be a continuous write and that will likely fill > up=C2=A0 > the cache making the disks underneath the bottleneck. > That's why on some other scenarios, like ZFS people have multiple > groups=C2=A0 > of RAID 6 (called RAIDZ2) so it improves the write speeds for these > type=C2=A0 > of scenarios. Just pointing out that it is commonly known as RAID 60, outside of the ZFS lingo: https://en.wikipedia.org/wiki/Nested_RAID_levels#RAID_60 /K > = > In the scenario given in this thread with just 3 servers, each with > a=C2=A0 > RAID 6 there will be a bare limit on the write performance specially > for=C2=A0 > streammed data for most powerfull your hardware controller can do=C2=A0 > write-back. > = > Also I agree the 40Gb NICs may not be used fully and 10Gb can do the > job=C2=A0 > well, but if they were available at the begining, why not use them. > = > Fernando > = > = > On 08/08/2017 03:16, Fabrice Bacchella wrote: > > = > > > = > > > Le 8 ao=C3=BBt 2017 =C3=A0 04:08, FERNANDO FREDIANI > > com> a =C3=A9crit : > > > Even if you have a Hardware RAID Controller with Writeback cache > > > you will have a significant performance penalty and may not fully > > > use all the resources you mentioned you have. > > > = > > Nope again,from my experience with HP Smart Array and write back > > cache, write, that goes in the cache, are even faster that read > > that must goes to the disks. of course if the write are too fast > > and to big, they will over overflow the cache. But on todays > > controller they are multi-gigabyte cache, you must write a lot to > > fill them. And if you can afford 40Gb card, you can afford decent > > controller. > > = > > = > > = > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users --===============0625199488519932082==-- From moacirferreira at hotmail.com Tue Aug 8 13:44:41 2017 Content-Type: multipart/mixed; boundary="===============2459877832682927586==" MIME-Version: 1.0 From: Moacir Ferreira To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Tue, 08 Aug 2017 09:09:47 +0000 Message-ID: In-Reply-To: CAD7b65Kw9QKoa09mVf+ThZL-2phLPPoQ5ABCpO5nhxn=gqTS7A@mail.gmail.com --===============2459877832682927586== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --_000_DB6P190MB02801EA0892B38F503220896C88A0DB6P190MB0280EURP_ Content-Type: text/plain; charset=3D"iso-8859-1" Content-Transfer-Encoding: quoted-printable Fernando, Let's see what people say... But this is what I understood Red Hat says is = =3D the best performance model. This is the main reason to open this discussion= =3D because as long as I can see, some of you in the community, do not agree. But when I think about a "distributed file system", that can make any numbe= =3D r of copies you want, it does not make sense using a RAIDed brick, what it = =3D makes sense is to use JBOD. Moacir ________________________________ From: fernando.frediani(a)upx.com.br on be= half=3D of FERNANDO FREDIANI Sent: Tuesday, August 8, 2017 3:08 AM To: Moacir Ferreira Cc: Colin Coe; users(a)ovirt.org Subject: Re: [ovirt-users] Good practices Moacir, I understand that if you do this type of configuration you will be = =3D severely impacted on storage performance, specially for writes. Even if you= =3D have a Hardware RAID Controller with Writeback cache you will have a signi= =3D ficant performance penalty and may not fully use all the resources you ment= =3D ioned you have. Fernando 2017-08-07 10:03 GMT-03:00 Moacir Ferreira >: Hi Colin, Take a look on Devin's response. Also, read the doc he shared that gives so= =3D me hints on how to deploy Gluster. It is more like that if you want high-performance you should have the brick= =3D s created as RAID (5 or 6) by the server's disk controller and them assembl= =3D e a JBOD GlusterFS. The attached document is Gluster specific and not for o= =3D Virt. But at this point I think that having SSD will not be a plus as using= =3D the RAID controller Gluster will not be aware of the SSD. Regarding the OS= =3D , my idea is to have a RAID 1, made of 2 low cost HDDs, to install it. So far, based on the information received I should create a single RAID 5 o= =3D r 6 on each server and then use this disk as a brick to create my Gluster c= =3D luster, made of 2 replicas + 1 arbiter. What is new for me is the detail th= =3D at the arbiter does not need a lot of space as it only keeps meta data. Thanks for your response! Moacir ________________________________ From: Colin Coe > Sent: Monday, August 7, 2017 12:41 PM To: Moacir Ferreira Cc: users(a)ovirt.org Subject: Re: [ovirt-users] Good practices Hi I just thought that you'd do hardware RAID if you had the controller or JBO= =3D D if you didn't. In hindsight, a server with 40Gbps NICs is pretty likely = =3D to have a hardware RAID controller. I've never done JBOD with hardware RAI= =3D D. I think having a single gluster brick on hardware JBOD would be riskier= =3D than multiple bricks, each on a single disk, but thats not based on anythi= =3D ng other than my prejudices. I thought gluster tiering was for the most frequently accessed files, in wh= =3D ich case all the VMs disks would end up in the hot tier. However, I have b= =3D een wrong before... I just wanted to know where the OS was going as I didn't see it mentioned i= =3D n the OP. Normally, I'd have the OS on a RAID1 but in your case thats a lo= =3D t of wasted disk. Honestly, I think Yaniv's answer was far better than my own and made the im= =3D portant point about having an arbiter. Thanks On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreira > wrote: Hi Colin, I am in Portugal, so sorry for this late response. It is quite confusing fo= =3D r me, please consider: 1 - What if the RAID is done by the server's disk controller, not by softwa= =3D re? 2 - For JBOD I am just using gdeploy to deploy it. However, I am not using = =3D the oVirt node GUI to do this. 3 - As the VM .qcow2 files are quite big, tiering would only help if made b= =3D y an intelligent system that uses SSD for chunks of data not for the entire= =3D .qcow2 file. But I guess this is a problem everybody else has. So, Do you = =3D know how tiering works in Gluster? 4 - I am putting the OS on the first disk. However, would you do differentl= =3D y? Moacir ________________________________ From: Colin Coe > Sent: Monday, August 7, 2017 4:48 AM To: Moacir Ferreira Cc: users(a)ovirt.org Subject: Re: [ovirt-users] Good practices 1) RAID5 may be a performance hit- 2) I'd be inclined to do this as JBOD by creating a distributed disperse vo= =3D lume on each server. Something like echo gluster volume create dispersevol disperse-data 5 redundancy 2 \ $(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e "server${SERV= =3D ER}:/brick/brick-${SERVER}${BRICK}/brick \c"; done; done) 3) I think the above. 4) Gluster does support tiering, but IIRC you'd need the same number of SSD= =3D as spindle drives. There may be another way to use the SSD as a fast cach= =3D e. Where are you putting the OS? Hope I understood the question... Thanks On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira > wrote: I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU = =3D sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use Gluste= =3D rFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dua= =3D l 10Gb NIC. So my intention is to create a loop like a server triangle usin= =3D g the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VM= =3D s around the pod (east /west traffic) while using the 10Gb interfaces for g= =3D iving services to the outside world (north/south traffic). This said, my first question is: How should I deploy GlusterFS in such oVir= =3D t scenario? My questions are: 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then= =3D create a GlusterFS using them? 2 - Instead, should I create a JBOD array made of all server's disks? 3 - What is the best Gluster configuration to provide for HA while not cons= =3D uming too much disk space? 4 - Does a oVirt hypervisor pod like I am planning to build, and the virtua= =3D lization environment, benefits from tiering when using a SSD disk? And yes,= =3D will Gluster do it by default or I have to configure it to do so? At the bottom line, what is the good practice for using GlusterFS in small = =3D pods for enterprises? You opinion/feedback will be really appreciated! Moacir _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users --_000_DB6P190MB02801EA0892B38F503220896C88A0DB6P190MB0280EURP_ Content-Type: text/html; charset=3D"iso-8859-1" Content-Transfer-Encoding: quoted-printable

Fernando,

Let's see what people say... But this is what I understood Red Hat says = =3D is the best performance model. This is the main reason to open this discuss= =3D ion because as long as I can see, some of you in the community, do not agre= =3D e.

But when I think about a "distributed file system", that can m= =3D ake any number of copies you want, it does not make sense using a RAIDed br= =3D ick, what it makes sense is to use JBOD.

Moacir

From: fernando.fredia= ni(a)u=3D px.com.br <fernando.frediani(a)upx.com.br> on behalf of FERNANDO FRED= IA=3D NI <fernando.frediani(a)upx.com>
Sent: Tuesday, August 8, 2017 3:08 AM
To: Moacir Ferreira
Cc: Colin Coe; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices

Moacir, I understand that if you do this type of configuration you wil= =3D l be severely impacted on storage performance, specially for writes. Even i= =3D f you have a Hardware RAID Controller with Writeback cache you will have a = =3D significant performance penalty and may not fully use all the resources you mentioned you have.

Fernando

2017-08-07 10:03 GMT-03:00 Moacir Ferreira <m= oacirf=3D erreira(a)hotmail.com>:

Hi Colin,

Take a look on Devin's response. Also, read the doc he shared that gives= =3D some hints on how to deploy Gluster.

It is more like that if you want high-performance you should have the br= =3D icks created as RAID (5 or 6) by the server's disk controller and them= =3D assemble a JBOD GlusterFS. The attached document is Gluster specific and n= =3D ot for oVirt. But at this point I think that having SSD will not be a plus as using the RAID controller Gluster wi= =3D ll not be aware of the SSD. Regarding the OS, my idea is to have a RAID 1, = =3D made of 2 low cost HDDs, to install it.

So far, based on the information received I should create a si= =3D ngle RAID 5 or 6 on each server and then use this disk as a brick to create= =3D my Gluster cluster, made of 2 replicas + 1 arbiter. What is new for me= =3D is the detail that the arbiter does not need a lot of space as it only keeps meta data.

Thanks for your response!

Moacir

From:<= /b> =3D Colin Coe <coli=3D n.coe(a)gmail.com>
Sent: Monday, August 7, 2017 12:41 PM

To: Moacir Ferreira
Cc: users= (a)ovirt=3D .org
Subject: Re: [ovirt-users] Good practices

Hi

I just thought that you'd do hardware RAID if you had the controller o= =3D r JBOD if you didn't. In hindsight, a server with 40Gbps NICs is pret= =3D ty likely to have a hardware RAID controller. I've never done JBOD wi= =3D th hardware RAID. I think having a single gluster brick on hardware JBOD would be riskier than multiple bricks, each= =3D on a single disk, but thats not based on anything other than my prejudices= =3D .

I thought gluster tiering was for the most frequently accessed files, = =3D in which case all the VMs disks would end up in the hot tier. However= =3D , I have been wrong before...

I just wanted to know where the OS was going as I didn't see it mentio= =3D ned in the OP. Normally, I'd have the OS on a RAID1 but in your case = =3D thats a lot of wasted disk.

Honestly, I think Yaniv's answer was far better than my own and made t= =3D he important point about having an arbiter.

Thanks

On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreir= a =3D <m= oacirf=3D erreira(a)hotmail.com> wrote:

Hi Colin,

I am in Portugal, so sorry for this late response. It is qu= =3D ite confusing for me, please consider:

1 - What if the RAID is done by the server's disk controller, no= =3D t by software?

2 - For JBOD I am just using gdeploy to deploy it. However, I am = =3D not using the oVirt node GUI to do this.

3 - As the VM .qcow2 files are quite big, tiering would only= =3D help if made by an intelligent system that uses SSD for chunks of data not= =3D for the entire .qcow2 file. But I guess this is a problem everybody else h= =3D as. So, Do you know how tiering works in Gluster?

4 - I am putting the OS on the first disk. However, would you do di= =3D fferently?

Moacir

From: Colin Coe <colin.coe(a)gmail.com>
Sent: Monday, August 7, 2017 4:48 AM
To: Moacir Ferreira
Cc: users= (a)ovirt=3D .org
Subject: Re: [ovirt-users] Good practices

1) RAID5 may be a performance hit-

2) I'd be inclined to do this as JBOD by creating a distributed = =3D disperse volume on each server. Something like

echo gluster volume create dispersevol disperse-data 5 redundancy 2 \<= =3D /div>
$(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e "se= =3D rver${SERVER}:/brick/brick-${SERVER}${BRICK}/brick \c"; done; don= =3D e)

3) I think the above.

4) Gluster does support tiering, but IIRC you'd need the same number o= =3D f SSD as spindle drives. There may be another way to use the SSD as a= =3D fast cache.

Where are you putting the OS?

Hope I understood the question...

Thanks

On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferrei= ra=3D <m= oacirf=3D erreira(a)hotmail.com> wrote:

I am willing to assemble a oVirt "pod", made of 3 server= =3D s, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The id= =3D ea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual= =3D 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop like a server triangle using the 40Gb NICs for virtual= =3D ization files (VMs .qcow2) access and to move VMs around the pod (east /wes= =3D t traffic) while using the 10Gb interfaces for giving services to the outsi= =3D de world (north/south traffic).

This said, my first question is: How should I deploy GlusterFS in such o= =3D Virt scenario? My questions are:

1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and t= =3D hen create a GlusterFS using them?

2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not c= =3D onsuming too much disk space?

4 - Does a oVirt hypervisor pod like I am planning to build, and the vir= =3D tualization environment, benefits from tiering when using a SSD disk? And y= =3D es, will Gluster do it by default or I have to configure it to do so?

At the bottom line, what is the good practice for using GlusterFS in sma= =3D ll pods for enterprises?

You opinion/feedback will be really appreciated!

Moacir

_______________________________________________
Users mailing list
Users(a)ovirt.or= g http://lists.ovirt.org/mailman/listinfo/users<=3D br>

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users<=3D br>

--_000_DB6P190MB02801EA0892B38F503220896C88A0DB6P190MB0280EURP_-- --===============2459877832682927586== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" LS1fMDAwX0RCNlAxOTBNQjAyODAxRUEwODkyQjM4RjUwMzIyMDg5NkM4OEEwREI2UDE5ME1CMDI4 MEVVUlBfCkNvbnRlbnQtVHlwZTogdGV4dC9wbGFpbjsgY2hhcnNldD0iaXNvLTg4NTktMSIKQ29u dGVudC1UcmFuc2Zlci1FbmNvZGluZzogcXVvdGVkLXByaW50YWJsZQoKRmVybmFuZG8sCgoKTGV0 J3Mgc2VlIHdoYXQgcGVvcGxlIHNheS4uLiBCdXQgdGhpcyBpcyB3aGF0IEkgdW5kZXJzdG9vZCBS ZWQgSGF0IHNheXMgaXMgPQp0aGUgYmVzdCBwZXJmb3JtYW5jZSBtb2RlbC4gVGhpcyBpcyB0aGUg bWFpbiByZWFzb24gdG8gb3BlbiB0aGlzIGRpc2N1c3Npb249CiBiZWNhdXNlIGFzIGxvbmcgYXMg SSBjYW4gc2VlLCBzb21lIG9mIHlvdSBpbiB0aGUgY29tbXVuaXR5LCBkbyBub3QgYWdyZWUuCgoK QnV0IHdoZW4gSSB0aGluayBhYm91dCBhICJkaXN0cmlidXRlZCBmaWxlIHN5c3RlbSIsIHRoYXQg Y2FuIG1ha2UgYW55IG51bWJlPQpyIG9mIGNvcGllcyB5b3Ugd2FudCwgaXQgZG9lcyBub3QgbWFr ZSBzZW5zZSB1c2luZyBhIFJBSURlZCBicmljaywgd2hhdCBpdCA9Cm1ha2VzIHNlbnNlIGlzIHRv IHVzZSBKQk9ELgoKCk1vYWNpcgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KRnJv bTogZmVybmFuZG8uZnJlZGlhbmlAdXB4LmNvbS5iciA8ZmVybmFuZG8uZnJlZGlhbmlAdXB4LmNv bS5icj4gb24gYmVoYWxmPQogb2YgRkVSTkFORE8gRlJFRElBTkkgPGZlcm5hbmRvLmZyZWRpYW5p QHVweC5jb20+ClNlbnQ6IFR1ZXNkYXksIEF1Z3VzdCA4LCAyMDE3IDM6MDggQU0KVG86IE1vYWNp ciBGZXJyZWlyYQpDYzogQ29saW4gQ29lOyB1c2Vyc0BvdmlydC5vcmcKU3ViamVjdDogUmU6IFtv dmlydC11c2Vyc10gR29vZCBwcmFjdGljZXMKCk1vYWNpciwgSSB1bmRlcnN0YW5kIHRoYXQgaWYg eW91IGRvIHRoaXMgdHlwZSBvZiBjb25maWd1cmF0aW9uIHlvdSB3aWxsIGJlID0Kc2V2ZXJlbHkg aW1wYWN0ZWQgb24gc3RvcmFnZSBwZXJmb3JtYW5jZSwgc3BlY2lhbGx5IGZvciB3cml0ZXMuIEV2 ZW4gaWYgeW91PQogaGF2ZSBhIEhhcmR3YXJlIFJBSUQgQ29udHJvbGxlciB3aXRoIFdyaXRlYmFj ayBjYWNoZSB5b3Ugd2lsbCBoYXZlIGEgc2lnbmk9CmZpY2FudCBwZXJmb3JtYW5jZSBwZW5hbHR5 IGFuZCBtYXkgbm90IGZ1bGx5IHVzZSBhbGwgdGhlIHJlc291cmNlcyB5b3UgbWVudD0KaW9uZWQg eW91IGhhdmUuCgpGZXJuYW5kbwoKMjAxNy0wOC0wNyAxMDowMyBHTVQtMDM6MDAgTW9hY2lyIEZl cnJlaXJhIDxtb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbTxtYWlsPQp0bzptb2FjaXJmZXJyZWly YUBob3RtYWlsLmNvbT4+OgoKSGkgQ29saW4sCgoKVGFrZSBhIGxvb2sgb24gRGV2aW4ncyByZXNw b25zZS4gQWxzbywgcmVhZCB0aGUgZG9jIGhlIHNoYXJlZCB0aGF0IGdpdmVzIHNvPQptZSBoaW50 cyBvbiBob3cgdG8gZGVwbG95IEdsdXN0ZXIuCgoKSXQgaXMgbW9yZSBsaWtlIHRoYXQgaWYgeW91 IHdhbnQgaGlnaC1wZXJmb3JtYW5jZSB5b3Ugc2hvdWxkIGhhdmUgdGhlIGJyaWNrPQpzIGNyZWF0 ZWQgYXMgUkFJRCAoNSBvciA2KSBieSB0aGUgc2VydmVyJ3MgZGlzayBjb250cm9sbGVyIGFuZCB0 aGVtIGFzc2VtYmw9CmUgYSBKQk9EIEdsdXN0ZXJGUy4gVGhlIGF0dGFjaGVkIGRvY3VtZW50IGlz IEdsdXN0ZXIgc3BlY2lmaWMgYW5kIG5vdCBmb3Igbz0KVmlydC4gQnV0IGF0IHRoaXMgcG9pbnQg SSB0aGluayB0aGF0IGhhdmluZyBTU0Qgd2lsbCBub3QgYmUgYSBwbHVzIGFzIHVzaW5nPQogdGhl IFJBSUQgY29udHJvbGxlciBHbHVzdGVyIHdpbGwgbm90IGJlIGF3YXJlIG9mIHRoZSBTU0QuIFJl Z2FyZGluZyB0aGUgT1M9CiwgbXkgaWRlYSBpcyB0byBoYXZlIGEgUkFJRCAxLCBtYWRlIG9mIDIg bG93IGNvc3QgSEREcywgdG8gaW5zdGFsbCBpdC4KCgpTbyBmYXIsIGJhc2VkIG9uIHRoZSBpbmZv cm1hdGlvbiByZWNlaXZlZCBJIHNob3VsZCBjcmVhdGUgYSBzaW5nbGUgUkFJRCA1IG89CnIgNiBv biBlYWNoIHNlcnZlciBhbmQgdGhlbiB1c2UgdGhpcyBkaXNrIGFzIGEgYnJpY2sgdG8gY3JlYXRl IG15IEdsdXN0ZXIgYz0KbHVzdGVyLCBtYWRlIG9mIDIgcmVwbGljYXMgKyAxIGFyYml0ZXIuIFdo YXQgaXMgbmV3IGZvciBtZSBpcyB0aGUgZGV0YWlsIHRoPQphdCB0aGUgYXJiaXRlciBkb2VzIG5v dCBuZWVkIGEgbG90IG9mIHNwYWNlIGFzIGl0IG9ubHkga2VlcHMgbWV0YSBkYXRhLgoKClRoYW5r cyBmb3IgeW91ciByZXNwb25zZSEKCk1vYWNpcgoKX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX18KRnJvbTogQ29saW4gQ29lIDxjb2xpbi5jb2VAZ21haWwuY29tPG1haWx0bzpjb2xpbi5j b2VAZ21haWwuY29tPj4KU2VudDogTW9uZGF5LCBBdWd1c3QgNywgMjAxNyAxMjo0MSBQTQoKVG86 IE1vYWNpciBGZXJyZWlyYQpDYzogdXNlcnNAb3ZpcnQub3JnPG1haWx0bzp1c2Vyc0BvdmlydC5v cmc+ClN1YmplY3Q6IFJlOiBbb3ZpcnQtdXNlcnNdIEdvb2QgcHJhY3RpY2VzCgpIaQoKSSBqdXN0 IHRob3VnaHQgdGhhdCB5b3UnZCBkbyBoYXJkd2FyZSBSQUlEIGlmIHlvdSBoYWQgdGhlIGNvbnRy b2xsZXIgb3IgSkJPPQpEIGlmIHlvdSBkaWRuJ3QuICBJbiBoaW5kc2lnaHQsIGEgc2VydmVyIHdp dGggNDBHYnBzIE5JQ3MgaXMgcHJldHR5IGxpa2VseSA9CnRvIGhhdmUgYSBoYXJkd2FyZSBSQUlE IGNvbnRyb2xsZXIuICBJJ3ZlIG5ldmVyIGRvbmUgSkJPRCB3aXRoIGhhcmR3YXJlIFJBST0KRC4g IEkgdGhpbmsgaGF2aW5nIGEgc2luZ2xlIGdsdXN0ZXIgYnJpY2sgb24gaGFyZHdhcmUgSkJPRCB3 b3VsZCBiZSByaXNraWVyPQogdGhhbiBtdWx0aXBsZSBicmlja3MsIGVhY2ggb24gYSBzaW5nbGUg ZGlzaywgYnV0IHRoYXRzIG5vdCBiYXNlZCBvbiBhbnl0aGk9Cm5nIG90aGVyIHRoYW4gbXkgcHJl anVkaWNlcy4KCkkgdGhvdWdodCBnbHVzdGVyIHRpZXJpbmcgd2FzIGZvciB0aGUgbW9zdCBmcmVx dWVudGx5IGFjY2Vzc2VkIGZpbGVzLCBpbiB3aD0KaWNoIGNhc2UgYWxsIHRoZSBWTXMgZGlza3Mg d291bGQgZW5kIHVwIGluIHRoZSBob3QgdGllci4gIEhvd2V2ZXIsIEkgaGF2ZSBiPQplZW4gd3Jv bmcgYmVmb3JlLi4uCgpJIGp1c3Qgd2FudGVkIHRvIGtub3cgd2hlcmUgdGhlIE9TIHdhcyBnb2lu ZyBhcyBJIGRpZG4ndCBzZWUgaXQgbWVudGlvbmVkIGk9Cm4gdGhlIE9QLiAgTm9ybWFsbHksIEkn ZCBoYXZlIHRoZSBPUyBvbiBhIFJBSUQxIGJ1dCBpbiB5b3VyIGNhc2UgdGhhdHMgYSBsbz0KdCBv ZiB3YXN0ZWQgZGlzay4KCkhvbmVzdGx5LCBJIHRoaW5rIFlhbml2J3MgYW5zd2VyIHdhcyBmYXIg YmV0dGVyIHRoYW4gbXkgb3duIGFuZCBtYWRlIHRoZSBpbT0KcG9ydGFudCBwb2ludCBhYm91dCBo YXZpbmcgYW4gYXJiaXRlci4KClRoYW5rcwoKT24gTW9uLCBBdWcgNywgMjAxNyBhdCA1OjU2IFBN LCBNb2FjaXIgRmVycmVpcmEgPG1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY29tPQo8bWFpbHRvOm1v YWNpcmZlcnJlaXJhQGhvdG1haWwuY29tPj4gd3JvdGU6CgpIaSBDb2xpbiwKCgpJIGFtIGluIFBv cnR1Z2FsLCBzbyBzb3JyeSBmb3IgdGhpcyBsYXRlIHJlc3BvbnNlLiBJdCBpcyBxdWl0ZSBjb25m dXNpbmcgZm89CnIgbWUsIHBsZWFzZSBjb25zaWRlcjoKCjEgLSBXaGF0IGlmIHRoZSBSQUlEIGlz IGRvbmUgYnkgdGhlIHNlcnZlcidzIGRpc2sgY29udHJvbGxlciwgbm90IGJ5IHNvZnR3YT0KcmU/ CgoKMiAtIEZvciBKQk9EIEkgYW0ganVzdCB1c2luZyBnZGVwbG95IHRvIGRlcGxveSBpdC4gSG93 ZXZlciwgSSBhbSBub3QgdXNpbmcgPQp0aGUgb1ZpcnQgbm9kZSBHVUkgdG8gZG8gdGhpcy4KCgoz IC0gQXMgdGhlIFZNIC5xY293MiBmaWxlcyBhcmUgcXVpdGUgYmlnLCB0aWVyaW5nIHdvdWxkIG9u bHkgaGVscCBpZiBtYWRlIGI9CnkgYW4gaW50ZWxsaWdlbnQgc3lzdGVtIHRoYXQgdXNlcyBTU0Qg Zm9yIGNodW5rcyBvZiBkYXRhIG5vdCBmb3IgdGhlIGVudGlyZT0KIC5xY293MiBmaWxlLiBCdXQg SSBndWVzcyB0aGlzIGlzIGEgcHJvYmxlbSBldmVyeWJvZHkgZWxzZSBoYXMuIFNvLCBEbyB5b3Ug PQprbm93IGhvdyB0aWVyaW5nIHdvcmtzIGluIEdsdXN0ZXI/CgoKNCAtIEkgYW0gcHV0dGluZyB0 aGUgT1Mgb24gdGhlIGZpcnN0IGRpc2suIEhvd2V2ZXIsIHdvdWxkIHlvdSBkbyBkaWZmZXJlbnRs PQp5PwoKCk1vYWNpcgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KRnJvbTogQ29s aW4gQ29lIDxjb2xpbi5jb2VAZ21haWwuY29tPG1haWx0bzpjb2xpbi5jb2VAZ21haWwuY29tPj4K U2VudDogTW9uZGF5LCBBdWd1c3QgNywgMjAxNyA0OjQ4IEFNClRvOiBNb2FjaXIgRmVycmVpcmEK Q2M6IHVzZXJzQG92aXJ0Lm9yZzxtYWlsdG86dXNlcnNAb3ZpcnQub3JnPgpTdWJqZWN0OiBSZTog W292aXJ0LXVzZXJzXSBHb29kIHByYWN0aWNlcwoKMSkgUkFJRDUgbWF5IGJlIGEgcGVyZm9ybWFu Y2UgaGl0LQoKMikgSSdkIGJlIGluY2xpbmVkIHRvIGRvIHRoaXMgYXMgSkJPRCBieSBjcmVhdGlu ZyBhIGRpc3RyaWJ1dGVkIGRpc3BlcnNlIHZvPQpsdW1lIG9uIGVhY2ggc2VydmVyLiAgU29tZXRo aW5nIGxpa2UKCmVjaG8gZ2x1c3RlciB2b2x1bWUgY3JlYXRlIGRpc3BlcnNldm9sIGRpc3BlcnNl LWRhdGEgNSByZWR1bmRhbmN5IDIgXAokKGZvciBTRVJWRVIgaW4gYSBiIGM7IGRvIGZvciBCUklD SyBpbiAkKHNlcSAxIDUpOyBkbyBlY2hvIC1lICJzZXJ2ZXIke1NFUlY9CkVSfTovYnJpY2svYnJp Y2stJHtTRVJWRVJ9JHtCUklDS30vYnJpY2sgXGMiOyBkb25lOyBkb25lKQoKMykgSSB0aGluayB0 aGUgYWJvdmUuCgo0KSBHbHVzdGVyIGRvZXMgc3VwcG9ydCB0aWVyaW5nLCBidXQgSUlSQyB5b3Un ZCBuZWVkIHRoZSBzYW1lIG51bWJlciBvZiBTU0Q9CiBhcyBzcGluZGxlIGRyaXZlcy4gIFRoZXJl IG1heSBiZSBhbm90aGVyIHdheSB0byB1c2UgdGhlIFNTRCBhcyBhIGZhc3QgY2FjaD0KZS4KCldo ZXJlIGFyZSB5b3UgcHV0dGluZyB0aGUgT1M/CgpIb3BlIEkgdW5kZXJzdG9vZCB0aGUgcXVlc3Rp b24uLi4KClRoYW5rcwoKT24gU3VuLCBBdWcgNiwgMjAxNyBhdCAxMDo0OSBQTSwgTW9hY2lyIEZl cnJlaXJhIDxtb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvPQptPG1haWx0bzptb2FjaXJmZXJyZWly YUBob3RtYWlsLmNvbT4+IHdyb3RlOgoKSSBhbSB3aWxsaW5nIHRvIGFzc2VtYmxlIGEgb1ZpcnQg InBvZCIsIG1hZGUgb2YgMyBzZXJ2ZXJzLCBlYWNoIHdpdGggMiBDUFUgPQpzb2NrZXRzIG9mIDEy IGNvcmVzLCAyNTZHQiBSQU0sIDcgSEREIDEwSywgMSBTU0QuIFRoZSBpZGVhIGlzIHRvIHVzZSBH bHVzdGU9CnJGUyB0byBwcm92aWRlIEhBIGZvciB0aGUgVk1zLiBUaGUgMyBzZXJ2ZXJzIGhhdmUg YSBkdWFsIDQwR2IgTklDIGFuZCBhIGR1YT0KbCAxMEdiIE5JQy4gU28gbXkgaW50ZW50aW9uIGlz IHRvIGNyZWF0ZSBhIGxvb3AgbGlrZSBhIHNlcnZlciB0cmlhbmdsZSB1c2luPQpnIHRoZSA0MEdi IE5JQ3MgZm9yIHZpcnR1YWxpemF0aW9uIGZpbGVzIChWTXMgLnFjb3cyKSBhY2Nlc3MgYW5kIHRv IG1vdmUgVk09CnMgYXJvdW5kIHRoZSBwb2QgKGVhc3QgL3dlc3QgdHJhZmZpYykgd2hpbGUgdXNp bmcgdGhlIDEwR2IgaW50ZXJmYWNlcyBmb3IgZz0KaXZpbmcgc2VydmljZXMgdG8gdGhlIG91dHNp ZGUgd29ybGQgKG5vcnRoL3NvdXRoIHRyYWZmaWMpLgoKClRoaXMgc2FpZCwgbXkgZmlyc3QgcXVl c3Rpb24gaXM6IEhvdyBzaG91bGQgSSBkZXBsb3kgR2x1c3RlckZTIGluIHN1Y2ggb1Zpcj0KdCBz Y2VuYXJpbz8gTXkgcXVlc3Rpb25zIGFyZToKCgoxIC0gU2hvdWxkIEkgY3JlYXRlIDMgUkFJRCAo aS5lLjogUkFJRCA1KSwgb25lIG9uIGVhY2ggb1ZpcnQgbm9kZSwgYW5kIHRoZW49CiBjcmVhdGUg YSBHbHVzdGVyRlMgdXNpbmcgdGhlbT8KCjIgLSBJbnN0ZWFkLCBzaG91bGQgSSBjcmVhdGUgYSBK Qk9EIGFycmF5IG1hZGUgb2YgYWxsIHNlcnZlcidzIGRpc2tzPwoKMyAtIFdoYXQgaXMgdGhlIGJl c3QgR2x1c3RlciBjb25maWd1cmF0aW9uIHRvIHByb3ZpZGUgZm9yIEhBIHdoaWxlIG5vdCBjb25z PQp1bWluZyB0b28gbXVjaCBkaXNrIHNwYWNlPwoKNCAtIERvZXMgYSBvVmlydCBoeXBlcnZpc29y IHBvZCBsaWtlIEkgYW0gcGxhbm5pbmcgdG8gYnVpbGQsIGFuZCB0aGUgdmlydHVhPQpsaXphdGlv biBlbnZpcm9ubWVudCwgYmVuZWZpdHMgZnJvbSB0aWVyaW5nIHdoZW4gdXNpbmcgYSBTU0QgZGlz az8gQW5kIHllcyw9CiB3aWxsIEdsdXN0ZXIgZG8gaXQgYnkgZGVmYXVsdCBvciBJIGhhdmUgdG8g Y29uZmlndXJlIGl0IHRvIGRvIHNvPwoKCkF0IHRoZSBib3R0b20gbGluZSwgd2hhdCBpcyB0aGUg Z29vZCBwcmFjdGljZSBmb3IgdXNpbmcgR2x1c3RlckZTIGluIHNtYWxsID0KcG9kcyBmb3IgZW50 ZXJwcmlzZXM/CgoKWW91IG9waW5pb24vZmVlZGJhY2sgd2lsbCBiZSByZWFsbHkgYXBwcmVjaWF0 ZWQhCgpNb2FjaXIKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fClVzZXJzIG1haWxpbmcgbGlzdApVc2Vyc0BvdmlydC5vcmc8bWFpbHRvOlVzZXJzQG92aXJ0 Lm9yZz4KaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzCgoKCgpf X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpVc2VycyBtYWls aW5nIGxpc3QKVXNlcnNAb3ZpcnQub3JnPG1haWx0bzpVc2Vyc0BvdmlydC5vcmc+Cmh0dHA6Ly9s aXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycwoKCgotLV8wMDBfREI2UDE5ME1C MDI4MDFFQTA4OTJCMzhGNTAzMjIwODk2Qzg4QTBEQjZQMTkwTUIwMjgwRVVSUF8KQ29udGVudC1U eXBlOiB0ZXh0L2h0bWw7IGNoYXJzZXQ9Imlzby04ODU5LTEiCkNvbnRlbnQtVHJhbnNmZXItRW5j b2Rpbmc6IHF1b3RlZC1wcmludGFibGUKCjxodG1sPgo8aGVhZD4KPG1ldGEgaHR0cC1lcXVpdj0z RCJDb250ZW50LVR5cGUiIGNvbnRlbnQ9M0QidGV4dC9odG1sOyBjaGFyc2V0PTNEaXNvLTg4NTkt PQoxIj4KPHN0eWxlIHR5cGU9M0QidGV4dC9jc3MiIHN0eWxlPTNEImRpc3BsYXk6bm9uZTsiPjwh LS0gUCB7bWFyZ2luLXRvcDowO21hcmdpPQpuLWJvdHRvbTowO30gLS0+PC9zdHlsZT4KPC9oZWFk Pgo8Ym9keSBkaXI9M0QibHRyIj4KPGRpdiBpZD0zRCJkaXZ0YWdkZWZhdWx0d3JhcHBlciIgc3R5 bGU9M0QiZm9udC1zaXplOjEycHQ7Y29sb3I6IzAwMDAwMDtmb250PQotZmFtaWx5OkNhbGlicmks SGVsdmV0aWNhLHNhbnMtc2VyaWY7IiBkaXI9M0QibHRyIj4KPHA+RmVybmFuZG8sPC9wPgo8cD48 YnI+CjwvcD4KPHA+TGV0J3Mgc2VlIHdoYXQgcGVvcGxlIHNheS4uLiBCdXQgdGhpcyBpcyB3aGF0 IEkgdW5kZXJzdG9vZCBSZWQgSGF0IHNheXMgPQppcyB0aGUgYmVzdCBwZXJmb3JtYW5jZSBtb2Rl bC4gVGhpcyBpcyB0aGUgbWFpbiByZWFzb24gdG8gb3BlbiB0aGlzIGRpc2N1c3M9CmlvbiBiZWNh dXNlIGFzIGxvbmcgYXMgSSBjYW4gc2VlLCBzb21lIG9mIHlvdSBpbiB0aGUgY29tbXVuaXR5LCBk byBub3QgYWdyZT0KZS48YnI+CjwvcD4KPGJyPgo8cD5CdXQgd2hlbiBJIHRoaW5rIGFib3V0IGEg JnF1b3Q7ZGlzdHJpYnV0ZWQgZmlsZSBzeXN0ZW0mcXVvdDssIHRoYXQgY2FuIG09CmFrZSBhbnkg bnVtYmVyIG9mIGNvcGllcyB5b3Ugd2FudCwgaXQgZG9lcyBub3QgbWFrZSBzZW5zZSB1c2luZyBh IFJBSURlZCBicj0KaWNrLCB3aGF0IGl0IG1ha2VzIHNlbnNlIGlzIHRvIHVzZSBKQk9ELjwvcD4K PHA+PGJyPgo8L3A+CjxwPk1vYWNpcjxicj4KPC9wPgo8YnI+CjxkaXYgc3R5bGU9M0QiY29sb3I6 IHJnYig0OSwgNTUsIDU3KTsiPgo8aHIgdGFiaW5kZXg9M0QiLTEiIHN0eWxlPTNEImRpc3BsYXk6 aW5saW5lLWJsb2NrOyB3aWR0aDo5OCUiPgo8ZGl2IGlkPTNEImRpdlJwbHlGd2RNc2ciIGRpcj0z RCJsdHIiPjxmb250IHN0eWxlPTNEImZvbnQtc2l6ZToxMXB0IiBmYWNlPQo9M0QiQ2FsaWJyaSwg c2Fucy1zZXJpZiIgY29sb3I9M0QiIzAwMDAwMCI+PGI+RnJvbTo8L2I+IGZlcm5hbmRvLmZyZWRp YW5pQHU9CnB4LmNvbS5iciAmbHQ7ZmVybmFuZG8uZnJlZGlhbmlAdXB4LmNvbS5iciZndDsgb24g YmVoYWxmIG9mIEZFUk5BTkRPIEZSRURJQT0KTkkgJmx0O2Zlcm5hbmRvLmZyZWRpYW5pQHVweC5j b20mZ3Q7PGJyPgo8Yj5TZW50OjwvYj4gVHVlc2RheSwgQXVndXN0IDgsIDIwMTcgMzowOCBBTTxi cj4KPGI+VG86PC9iPiBNb2FjaXIgRmVycmVpcmE8YnI+CjxiPkNjOjwvYj4gQ29saW4gQ29lOyB1 c2Vyc0BvdmlydC5vcmc8YnI+CjxiPlN1YmplY3Q6PC9iPiBSZTogW292aXJ0LXVzZXJzXSBHb29k IHByYWN0aWNlczwvZm9udD4KPGRpdj4mbmJzcDs8L2Rpdj4KPC9kaXY+CjxkaXY+CjxkaXYgZGly PTNEImx0ciI+CjxkaXY+TW9hY2lyLCBJIHVuZGVyc3RhbmQgdGhhdCBpZiB5b3UgZG8gdGhpcyB0 eXBlIG9mIGNvbmZpZ3VyYXRpb24geW91IHdpbD0KbCBiZSBzZXZlcmVseSBpbXBhY3RlZCBvbiBz dG9yYWdlIHBlcmZvcm1hbmNlLCBzcGVjaWFsbHkgZm9yIHdyaXRlcy4gRXZlbiBpPQpmIHlvdSBo YXZlIGEgSGFyZHdhcmUgUkFJRCBDb250cm9sbGVyIHdpdGggV3JpdGViYWNrIGNhY2hlIHlvdSB3 aWxsIGhhdmUgYSA9CnNpZ25pZmljYW50IHBlcmZvcm1hbmNlIHBlbmFsdHkKIGFuZCBtYXkgbm90 IGZ1bGx5IHVzZSBhbGwgdGhlIHJlc291cmNlcyB5b3UgbWVudGlvbmVkIHlvdSBoYXZlLjxicj4K PGJyPgo8L2Rpdj4KRmVybmFuZG88YnI+CjwvZGl2Pgo8ZGl2IGNsYXNzPTNEImdtYWlsX2V4dHJh Ij48YnI+CjxkaXYgY2xhc3M9M0QiZ21haWxfcXVvdGUiPjIwMTctMDgtMDcgMTA6MDMgR01ULTAz OjAwIE1vYWNpciBGZXJyZWlyYSA8c3Bhbj0KIGRpcj0zRCJsdHIiPgombHQ7PGEgaHJlZj0zRCJt YWlsdG86bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb20iIHRhcmdldD0zRCJfYmxhbmsiPm1vYWNp cmY9CmVycmVpcmFAaG90bWFpbC5jb208L2E+Jmd0Ozwvc3Bhbj46PGJyPgo8YmxvY2txdW90ZSBj bGFzcz0zRCJnbWFpbF9xdW90ZSIgc3R5bGU9M0QibWFyZ2luOjAgMCAwIC44ZXg7IGJvcmRlci1s ZWZ0OjE9CnB4ICNjY2Mgc29saWQ7IHBhZGRpbmctbGVmdDoxZXgiPgo8ZGl2IGRpcj0zRCJsdHIi Pgo8ZGl2IGlkPTNEIm1fMTg3NDg1OTg1OTg0Mjc2MzEwNGRpdnRhZ2RlZmF1bHR3cmFwcGVyIiBk aXI9M0QibHRyIiBzdHlsZT0zRCI9CmZvbnQtc2l6ZToxMnB0OyBjb2xvcjojMDAwMDAwOyBmb250 LWZhbWlseTpDYWxpYnJpLEhlbHZldGljYSxzYW5zLXNlcmlmIj4KPHA+SGkgQ29saW4sPC9wPgo8 cD48YnI+CjwvcD4KPHA+VGFrZSBhIGxvb2sgb24gRGV2aW4ncyByZXNwb25zZS4gQWxzbywgcmVh ZCB0aGUgZG9jIGhlIHNoYXJlZCB0aGF0IGdpdmVzPQogc29tZSBoaW50cyBvbiBob3cgdG8gZGVw bG95IEdsdXN0ZXIuPC9wPgo8cD48YnI+CjwvcD4KPHA+SXQgaXMgbW9yZSBsaWtlIHRoYXQgaWYg eW91IHdhbnQgaGlnaC1wZXJmb3JtYW5jZSB5b3Ugc2hvdWxkIGhhdmUgdGhlIGJyPQppY2tzIGNy ZWF0ZWQgYXMgUkFJRCAoNSBvciA2KSZuYnNwO2J5IHRoZSBzZXJ2ZXIncyBkaXNrIGNvbnRyb2xs ZXIgYW5kIHRoZW09CiBhc3NlbWJsZSBhIEpCT0QgR2x1c3RlckZTLiBUaGUgYXR0YWNoZWQgZG9j dW1lbnQgaXMgR2x1c3RlciBzcGVjaWZpYyBhbmQgbj0Kb3QgZm9yIG9WaXJ0LiBCdXQgYXQgdGhp cyBwb2ludCBJIHRoaW5rCiB0aGF0IGhhdmluZyBTU0Qgd2lsbCBub3QgYmUgYSBwbHVzIGFzIHVz aW5nIHRoZSBSQUlEIGNvbnRyb2xsZXIgR2x1c3RlciB3aT0KbGwgbm90IGJlIGF3YXJlIG9mIHRo ZSBTU0QuIFJlZ2FyZGluZyB0aGUgT1MsIG15IGlkZWEgaXMgdG8gaGF2ZSBhIFJBSUQgMSwgPQpt YWRlIG9mIDIgbG93IGNvc3QgSEREcywgdG8gaW5zdGFsbCBpdC48L3A+CjxwPjxicj4KPC9wPgo8 cD5TbyBmYXIsIGJhc2VkIG9uIHRoZSBpbmZvcm1hdGlvbiByZWNlaXZlZCZuYnNwO0kgc2hvdWxk Jm5ic3A7Y3JlYXRlIGEgc2k9Cm5nbGUgUkFJRCA1IG9yIDYgb24gZWFjaCBzZXJ2ZXIgYW5kIHRo ZW4gdXNlIHRoaXMgZGlzayBhcyBhIGJyaWNrIHRvIGNyZWF0ZT0KIG15IEdsdXN0ZXIgY2x1c3Rl ciwgbWFkZSBvZiAyIHJlcGxpY2FzICYjNDM7IDEgYXJiaXRlci4gV2hhdCBpcyBuZXcgZm9yIG1l PQogaXMgdGhlIGRldGFpbCB0aGF0IHRoZSBhcmJpdGVyIGRvZXMgbm90IG5lZWQKIGEgbG90IG9m IHNwYWNlIGFzIGl0IG9ubHkga2VlcHMgbWV0YSBkYXRhLjwvcD4KPHA+PGJyPgo8L3A+CjxwPlRo YW5rcyBmb3IgeW91ciByZXNwb25zZSE8YnI+CjwvcD4KTW9hY2lyPGJyPgo8YnI+CjxkaXYgc3R5 bGU9M0QiY29sb3I6cmdiKDQ5LDU1LDU3KSI+CjxociBzdHlsZT0zRCJkaXNwbGF5OmlubGluZS1i bG9jazsgd2lkdGg6OTglIj4KPGRpdiBpZD0zRCJtXzE4NzQ4NTk4NTk4NDI3NjMxMDRkaXZScGx5 RndkTXNnIiBkaXI9M0QibHRyIj48Zm9udCBzdHlsZT0zRCJmPQpvbnQtc2l6ZToxMXB0IiBmYWNl PTNEIkNhbGlicmksIHNhbnMtc2VyaWYiIGNvbG9yPTNEIiMwMDAwMDAiPjxiPkZyb206PC9iPiA9 CkNvbGluIENvZSAmbHQ7PGEgaHJlZj0zRCJtYWlsdG86Y29saW4uY29lQGdtYWlsLmNvbSIgdGFy Z2V0PTNEIl9ibGFuayI+Y29saT0Kbi5jb2VAZ21haWwuY29tPC9hPiZndDs8YnI+CjxiPlNlbnQ6 PC9iPiBNb25kYXksIEF1Z3VzdCA3LCAyMDE3IDEyOjQxIFBNCjxkaXY+CjxkaXYgY2xhc3M9M0Qi aDUiPjxicj4KPGI+VG86PC9iPiBNb2FjaXIgRmVycmVpcmE8YnI+CjxiPkNjOjwvYj4gPGEgaHJl Zj0zRCJtYWlsdG86dXNlcnNAb3ZpcnQub3JnIiB0YXJnZXQ9M0QiX2JsYW5rIj51c2Vyc0Bvdmly dD0KLm9yZzwvYT48YnI+CjxiPlN1YmplY3Q6PC9iPiBSZTogW292aXJ0LXVzZXJzXSBHb29kIHBy YWN0aWNlczwvZGl2Pgo8L2Rpdj4KPC9mb250Pgo8ZGl2PiZuYnNwOzwvZGl2Pgo8L2Rpdj4KPGRp dj4KPGRpdiBjbGFzcz0zRCJoNSI+CjxkaXY+CjxkaXYgZGlyPTNEImx0ciI+SGkKPGRpdj48YnI+ CjwvZGl2Pgo8ZGl2PkkganVzdCB0aG91Z2h0IHRoYXQgeW91J2QgZG8gaGFyZHdhcmUgUkFJRCBp ZiB5b3UgaGFkIHRoZSBjb250cm9sbGVyIG89CnIgSkJPRCBpZiB5b3UgZGlkbid0LiZuYnNwOyBJ biBoaW5kc2lnaHQsIGEgc2VydmVyIHdpdGggNDBHYnBzIE5JQ3MgaXMgcHJldD0KdHkgbGlrZWx5 IHRvIGhhdmUgYSBoYXJkd2FyZSBSQUlEIGNvbnRyb2xsZXIuJm5ic3A7IEkndmUgbmV2ZXIgZG9u ZSBKQk9EIHdpPQp0aCBoYXJkd2FyZSBSQUlELiZuYnNwOyBJIHRoaW5rIGhhdmluZyBhIHNpbmds ZQogZ2x1c3RlciBicmljayBvbiBoYXJkd2FyZSBKQk9EIHdvdWxkIGJlIHJpc2tpZXIgdGhhbiBt dWx0aXBsZSBicmlja3MsIGVhY2g9CiBvbiBhIHNpbmdsZSBkaXNrLCBidXQgdGhhdHMgbm90IGJh c2VkIG9uIGFueXRoaW5nIG90aGVyIHRoYW4gbXkgcHJlanVkaWNlcz0KLjwvZGl2Pgo8ZGl2Pjxi cj4KPC9kaXY+CjxkaXY+SSB0aG91Z2h0IGdsdXN0ZXIgdGllcmluZyB3YXMgZm9yIHRoZSBtb3N0 IGZyZXF1ZW50bHkgYWNjZXNzZWQgZmlsZXMsID0KaW4gd2hpY2ggY2FzZSBhbGwgdGhlIFZNcyBk aXNrcyB3b3VsZCBlbmQgdXAgaW4gdGhlIGhvdCB0aWVyLiZuYnNwOyBIb3dldmVyPQosIEkgaGF2 ZSBiZWVuIHdyb25nIGJlZm9yZS4uLjwvZGl2Pgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+SSBqdXN0 IHdhbnRlZCB0byBrbm93IHdoZXJlIHRoZSBPUyB3YXMgZ29pbmcgYXMgSSBkaWRuJ3Qgc2VlIGl0 IG1lbnRpbz0KbmVkIGluIHRoZSBPUC4mbmJzcDsgTm9ybWFsbHksIEknZCBoYXZlIHRoZSBPUyBv biBhIFJBSUQxIGJ1dCBpbiB5b3VyIGNhc2UgPQp0aGF0cyBhIGxvdCBvZiB3YXN0ZWQgZGlzay48 L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8ZGl2PkhvbmVzdGx5LCBJIHRoaW5rIFlhbml2J3MgYW5z d2VyIHdhcyBmYXIgYmV0dGVyIHRoYW4gbXkgb3duIGFuZCBtYWRlIHQ9CmhlIGltcG9ydGFudCBw b2ludCBhYm91dCBoYXZpbmcgYW4gYXJiaXRlci4mbmJzcDs8L2Rpdj4KPGRpdj48YnI+CjwvZGl2 Pgo8ZGl2PlRoYW5rczwvZGl2Pgo8L2Rpdj4KPGRpdiBjbGFzcz0zRCJnbWFpbF9leHRyYSI+PGJy Pgo8ZGl2IGNsYXNzPTNEImdtYWlsX3F1b3RlIj5PbiBNb24sIEF1ZyA3LCAyMDE3IGF0IDU6NTYg UE0sIE1vYWNpciBGZXJyZWlyYSA9CjxzcGFuIGRpcj0zRCJsdHIiPgombHQ7PGEgaHJlZj0zRCJt YWlsdG86bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb20iIHRhcmdldD0zRCJfYmxhbmsiPm1vYWNp cmY9CmVycmVpcmFAaG90bWFpbC5jb208L2E+Jmd0Ozwvc3Bhbj4gd3JvdGU6PGJyPgo8YmxvY2tx dW90ZSBjbGFzcz0zRCJnbWFpbF9xdW90ZSIgc3R5bGU9M0QibWFyZ2luOjAgMCAwIC44ZXg7IGJv cmRlci1sZWZ0OjE9CnB4ICNjY2Mgc29saWQ7IHBhZGRpbmctbGVmdDoxZXgiPgo8ZGl2IGRpcj0z RCJsdHIiPgo8ZGl2IGlkPTNEIm1fMTg3NDg1OTg1OTg0Mjc2MzEwNG1fNDgzMTg4Njg4MzMwNTY3 MjcxOGRpdnRhZ2RlZmF1bHR3cmFwcGVyIiA9CmRpcj0zRCJsdHIiIHN0eWxlPTNEImZvbnQtc2l6 ZToxMnB0OyBjb2xvcjojMDAwMDAwOyBmb250LWZhbWlseTpDYWxpYnJpLEhlbD0KdmV0aWNhLHNh bnMtc2VyaWYiPgo8cD5IaSBDb2xpbiw8L3A+CjxwPjxicj4KPC9wPgo8cD48c3Bhbj5JIGFtIGlu IFBvcnR1Z2FsPC9zcGFuPiwgc28gc29ycnkgZm9yIHRoaXMgbGF0ZSByZXNwb25zZS4gSXQgaXMg cXU9Cml0ZSBjb25mdXNpbmcgZm9yIG1lLCBwbGVhc2UgY29uc2lkZXI6PC9wPgo8cD48L3A+Cjxk aXY+PGI+PGJyPgo8L2I+MTxiPiAtIDwvYj5XaGF0IGlmIHRoZSBSQUlEIGlzIGRvbmUgYnkgdGhl IHNlcnZlcidzIGRpc2sgY29udHJvbGxlciwgbm89CnQgYnkgc29mdHdhcmU/PC9kaXY+Cjxicj4K PHA+PC9wPgo8cD4yIC08Yj4gPC9iPkZvciBKQk9EIEkgYW0ganVzdCB1c2luZyBnZGVwbG95IHRv IGRlcGxveSBpdC4gSG93ZXZlciwgSSBhbSA9Cm5vdCB1c2luZyB0aGUgb1ZpcnQgbm9kZSBHVUkg dG8gZG8gdGhpcy48L3A+CjxwPjxicj4KPC9wPgo8cD4zIC08Yj4gPC9iPkFzIHRoZSBWTSAucWNv dzImbmJzcDtmaWxlcyBhcmUgcXVpdGUgYmlnLCB0aWVyaW5nIHdvdWxkIG9ubHk9CiBoZWxwIGlm IG1hZGUgYnkgYW4gaW50ZWxsaWdlbnQgc3lzdGVtIHRoYXQgdXNlcyBTU0QgZm9yIGNodW5rcyBv ZiBkYXRhIG5vdD0KIGZvciB0aGUgZW50aXJlIC5xY293MiBmaWxlLiBCdXQgSSBndWVzcyB0aGlz IGlzIGEgcHJvYmxlbSBldmVyeWJvZHkgZWxzZSBoPQphcy4gU28sIERvIHlvdSBrbm93IGhvdyB0 aWVyaW5nIHdvcmtzCiBpbiBHbHVzdGVyPzxicj4KPC9wPgo8cD48YnI+CjwvcD4KPHA+NCAtJm5i c3A7SSBhbSBwdXR0aW5nIHRoZSBPUyBvbiB0aGUgZmlyc3QgZGlzay4gSG93ZXZlciwgd291bGQg eW91IGRvIGRpPQpmZmVyZW50bHk/PGJyPgo8L3A+CjxwPjxicj4KPC9wPgpNb2FjaXI8YnI+Cjxi cj4KPGRpdiBzdHlsZT0zRCJjb2xvcjpyZ2IoNDksNTUsNTcpIj4KPGhyIHN0eWxlPTNEImRpc3Bs YXk6aW5saW5lLWJsb2NrOyB3aWR0aDo5OCUiPgo8ZGl2IGlkPTNEIm1fMTg3NDg1OTg1OTg0Mjc2 MzEwNG1fNDgzMTg4Njg4MzMwNTY3MjcxOGRpdlJwbHlGd2RNc2ciIGRpcj0zRCI9Cmx0ciI+PGZv bnQgc3R5bGU9M0QiZm9udC1zaXplOjExcHQiIGZhY2U9M0QiQ2FsaWJyaSwgc2Fucy1zZXJpZiIg Y29sb3I9M0QiIz0KMDAwMDAwIj48Yj5Gcm9tOjwvYj4gQ29saW4gQ29lICZsdDs8YSBocmVmPTNE Im1haWx0bzpjb2xpbi5jb2VAZ21haWwuY29tIiB0PQphcmdldD0zRCJfYmxhbmsiPmNvbGluLmNv ZUBnbWFpbC5jb208L2E+Jmd0Ozxicj4KPGI+U2VudDo8L2I+IE1vbmRheSwgQXVndXN0IDcsIDIw MTcgNDo0OCBBTTxicj4KPGI+VG86PC9iPiBNb2FjaXIgRmVycmVpcmE8YnI+CjxiPkNjOjwvYj4g PGEgaHJlZj0zRCJtYWlsdG86dXNlcnNAb3ZpcnQub3JnIiB0YXJnZXQ9M0QiX2JsYW5rIj51c2Vy c0BvdmlydD0KLm9yZzwvYT48YnI+CjxiPlN1YmplY3Q6PC9iPiBSZTogW292aXJ0LXVzZXJzXSBH b29kIHByYWN0aWNlczwvZm9udD4KPGRpdj4mbmJzcDs8L2Rpdj4KPC9kaXY+CjxkaXY+CjxkaXYg ZGlyPTNEImx0ciI+MSkgUkFJRDUgbWF5IGJlIGEgcGVyZm9ybWFuY2UgaGl0LSA8YnI+CjxkaXY+ PGJyPgo8L2Rpdj4KPGRpdj48c3Bhbj4yKSBJJ2QgYmUgaW5jbGluZWQgdG8gZG8gdGhpcyBhcyBK Qk9EIGJ5IGNyZWF0aW5nIGEgZGlzdHJpYnV0ZWQgPQpkaXNwZXJzZSB2b2x1bWUgb24gZWFjaCBz ZXJ2ZXIuJm5ic3A7IFNvbWV0aGluZyBsaWtlCjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj5lY2hvIGds dXN0ZXIgdm9sdW1lIGNyZWF0ZSBkaXNwZXJzZXZvbCBkaXNwZXJzZS1kYXRhIDUgcmVkdW5kYW5j eSAyIFw8PQovZGl2Pgo8ZGl2PiQoZm9yIFNFUlZFUiBpbiBhIGIgYzsgZG8gZm9yIEJSSUNLIGlu ICQoc2VxIDEgNSk7IGRvIGVjaG8gLWUgJnF1b3Q7c2U9CnJ2ZXIke1NFUlZFUn06L2JyaWNrL2Jy aWNrLTx3YnI+JHtTRVJWRVJ9JHtCUklDS30vYnJpY2sgXGMmcXVvdDs7IGRvbmU7IGRvbj0KZSk8 L2Rpdj4KPGRpdj48YnI+CjwvZGl2Pgo8L3NwYW4+CjxkaXY+MykgSSB0aGluayB0aGUgYWJvdmUu IDxiPjwvYj48L2Rpdj4KPHNwYW4+CjxkaXY+PGJyPgo8L2Rpdj4KPGRpdj40KSBHbHVzdGVyIGRv ZXMgc3VwcG9ydCB0aWVyaW5nLCBidXQgSUlSQyB5b3UnZCBuZWVkIHRoZSBzYW1lIG51bWJlciBv PQpmIFNTRCBhcyBzcGluZGxlIGRyaXZlcy4mbmJzcDsgVGhlcmUgbWF5IGJlIGFub3RoZXIgd2F5 IHRvIHVzZSB0aGUgU1NEIGFzIGE9CiBmYXN0IGNhY2hlLiZuYnNwOzwvZGl2Pgo8ZGl2Pjxicj4K PC9kaXY+CjxkaXY+V2hlcmUgYXJlIHlvdSBwdXR0aW5nIHRoZSBPUz88L2Rpdj4KPGRpdj48YnI+ CjwvZGl2Pgo8ZGl2PkhvcGUgSSB1bmRlcnN0b29kIHRoZSBxdWVzdGlvbi4uLjwvZGl2Pgo8ZGl2 Pjxicj4KPC9kaXY+CjxkaXY+VGhhbmtzPC9kaXY+Cjwvc3Bhbj48L2Rpdj4KPC9kaXY+CjxzcGFu Pgo8ZGl2IGNsYXNzPTNEImdtYWlsX2V4dHJhIj48YnI+CjxkaXYgY2xhc3M9M0QiZ21haWxfcXVv dGUiPk9uIFN1biwgQXVnIDYsIDIwMTcgYXQgMTA6NDkgUE0sIE1vYWNpciBGZXJyZWlyYT0KIDxz cGFuIGRpcj0zRCJsdHIiPgombHQ7PGEgaHJlZj0zRCJtYWlsdG86bW9hY2lyZmVycmVpcmFAaG90 bWFpbC5jb20iIHRhcmdldD0zRCJfYmxhbmsiPm1vYWNpcmY9CmVycmVpcmFAaG90bWFpbC5jb208 L2E+Jmd0Ozwvc3Bhbj4gd3JvdGU6PGJyPgo8YmxvY2txdW90ZSBjbGFzcz0zRCJnbWFpbF9xdW90 ZSIgc3R5bGU9M0QibWFyZ2luOjAgMCAwIC44ZXg7IGJvcmRlci1sZWZ0OjE9CnB4ICNjY2Mgc29s aWQ7IHBhZGRpbmctbGVmdDoxZXgiPgo8ZGl2IGRpcj0zRCJsdHIiPgo8ZGl2IGlkPTNEIm1fMTg3 NDg1OTg1OTg0Mjc2MzEwNG1fNDgzMTg4Njg4MzMwNTY3MjcxOG1fMjQ2MDk4NTY5MTc0NjQ5ODMy MmQ9Cml2dGFnZGVmYXVsdHdyYXBwZXIiIGRpcj0zRCJsdHIiIHN0eWxlPTNEImZvbnQtc2l6ZTox MnB0OyBjb2xvcjojMDAwMDAwOyBmbz0KbnQtZmFtaWx5OkNhbGlicmksSGVsdmV0aWNhLHNhbnMt c2VyaWYiPgo8cD48c3Bhbj5JIGFtIHdpbGxpbmcgdG8gYXNzZW1ibGUgYSBvVmlydCAmcXVvdDtw b2QmcXVvdDssIG1hZGUgb2YgMyBzZXJ2ZXI9CnMsIGVhY2ggd2l0aCAyIENQVSBzb2NrZXRzIG9m IDEyIGNvcmVzLCAyNTZHQiBSQU0sIDcgSEREIDEwSywgMSBTU0QuIFRoZSBpZD0KZWEgaXMgdG8g dXNlIEdsdXN0ZXJGUyB0byBwcm92aWRlIEhBIGZvciB0aGUgVk1zLiBUaGUgMyBzZXJ2ZXJzIGhh dmUgYSBkdWFsPQogNDBHYiBOSUMgYW5kIGEgZHVhbCAxMEdiIE5JQy4gU28gbXkgaW50ZW50aW9u CiBpcyB0byBjcmVhdGUgYSBsb29wIGxpa2UgYSBzZXJ2ZXIgdHJpYW5nbGUgdXNpbmcgdGhlIDQw R2IgTklDcyBmb3IgdmlydHVhbD0KaXphdGlvbiBmaWxlcyAoVk1zIC5xY293MikgYWNjZXNzIGFu ZCB0byBtb3ZlIFZNcyBhcm91bmQgdGhlIHBvZCAoZWFzdCAvd2VzPQp0IHRyYWZmaWMpIHdoaWxl IHVzaW5nIHRoZSAxMEdiIGludGVyZmFjZXMgZm9yIGdpdmluZyBzZXJ2aWNlcyB0byB0aGUgb3V0 c2k9CmRlIHdvcmxkIChub3J0aC9zb3V0aCB0cmFmZmljKS48L3NwYW4+PC9wPgo8cD48YnI+Cjxz cGFuPjwvc3Bhbj48L3A+CjxwPlRoaXMgc2FpZCwgbXkgZmlyc3QgcXVlc3Rpb24gaXM6IEhvdyBz aG91bGQgSSBkZXBsb3kgR2x1c3RlckZTIGluIHN1Y2ggbz0KVmlydCBzY2VuYXJpbz8gTXkgcXVl c3Rpb25zIGFyZTo8L3A+CjxwPjxicj4KPC9wPgo8cD4xIC0gU2hvdWxkIEkgY3JlYXRlIDMgUkFJ RCAoaS5lLjogUkFJRCA1KSwgb25lIG9uIGVhY2ggb1ZpcnQgbm9kZSwgYW5kIHQ9CmhlbiBjcmVh dGUgYSBHbHVzdGVyRlMgdXNpbmcgdGhlbT88L3A+CjxwPjIgLSBJbnN0ZWFkLCBzaG91bGQgSSBj cmVhdGUgYSBKQk9EIGFycmF5IG1hZGUgb2YgYWxsIHNlcnZlcidzIGRpc2tzPzwvcD0KPgo8cD4z IC0gV2hhdCBpcyB0aGUgYmVzdCBHbHVzdGVyIGNvbmZpZ3VyYXRpb24gdG8gcHJvdmlkZSBmb3Ig SEEgd2hpbGUgbm90IGM9Cm9uc3VtaW5nIHRvbyBtdWNoIGRpc2sgc3BhY2U/PGJyPgo8L3A+Cjxw PjQgLSBEb2VzIGEgb1ZpcnQgaHlwZXJ2aXNvciBwb2QgbGlrZSBJIGFtIHBsYW5uaW5nIHRvIGJ1 aWxkLCBhbmQgdGhlIHZpcj0KdHVhbGl6YXRpb24gZW52aXJvbm1lbnQsIGJlbmVmaXRzIGZyb20g dGllcmluZyB3aGVuIHVzaW5nIGEgU1NEIGRpc2s/IEFuZCB5PQplcywgd2lsbCBHbHVzdGVyIGRv IGl0IGJ5IGRlZmF1bHQgb3IgSSBoYXZlIHRvIGNvbmZpZ3VyZSBpdCB0byBkbyBzbz88L3A+Cjxw Pjxicj4KPC9wPgo8cD5BdCB0aGUgYm90dG9tIGxpbmUsIHdoYXQgaXMgdGhlIGdvb2QgcHJhY3Rp Y2UgZm9yIHVzaW5nIEdsdXN0ZXJGUyBpbiBzbWE9CmxsIHBvZHMgZm9yIGVudGVycHJpc2VzPzxi cj4KPC9wPgo8cD48YnI+CjwvcD4KPHA+WW91IG9waW5pb24vZmVlZGJhY2sgd2lsbCBiZSByZWFs bHkgYXBwcmVjaWF0ZWQhPC9wPgo8c3BhbiBjbGFzcz0zRCJtXzE4NzQ4NTk4NTk4NDI3NjMxMDRt XzQ4MzE4ODY4ODMzMDU2NzI3MThIT0VuWmIiPjxmb250IGNvbG89CnI9M0QiIzg4ODg4OCI+Cjxw Pk1vYWNpcjxicj4KPC9wPgo8L2ZvbnQ+PC9zcGFuPjwvZGl2Pgo8L2Rpdj4KPGJyPgpfX19fX19f X19fX19fX19fX19fX19fX19fX19fX188d2JyPl9fX19fX19fX19fX19fX19fPGJyPgpVc2VycyBt YWlsaW5nIGxpc3Q8YnI+CjxhIGhyZWY9M0QibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgdGFyZ2V0 PTNEIl9ibGFuayI+VXNlcnNAb3ZpcnQub3JnPC9hPjxicj0KPgo8YSBocmVmPTNEImh0dHA6Ly9s aXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyIgcmVsPTNEIm5vcmVmZXJyZXI9 CiIgdGFyZ2V0PTNEIl9ibGFuayI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuPHdicj4v bGlzdGluZm8vdXNlcnM8L2E+PD0KYnI+Cjxicj4KPC9ibG9ja3F1b3RlPgo8L2Rpdj4KPGJyPgo8 L2Rpdj4KPC9zcGFuPjwvZGl2Pgo8L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8L2Jsb2NrcXVvdGU+Cjwv ZGl2Pgo8YnI+CjwvZGl2Pgo8L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8L2Rpdj4KPC9kaXY+CjwvZGl2 Pgo8YnI+Cl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzx3YnI+X19fX19fX19fX19fX19f X188YnI+ClVzZXJzIG1haWxpbmcgbGlzdDxicj4KPGEgaHJlZj0zRCJtYWlsdG86VXNlcnNAb3Zp cnQub3JnIj5Vc2Vyc0BvdmlydC5vcmc8L2E+PGJyPgo8YSBocmVmPTNEImh0dHA6Ly9saXN0cy5v dmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyIgcmVsPTNEIm5vcmVmZXJyZXI9CiIgdGFy Z2V0PTNEIl9ibGFuayI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy88d2JyPm1haWxtYW4vbGlzdGlu Zm8vdXNlcnM8L2E+PD0KYnI+Cjxicj4KPC9ibG9ja3F1b3RlPgo8L2Rpdj4KPGJyPgo8L2Rpdj4K PC9kaXY+CjwvZGl2Pgo8L2Rpdj4KPC9ib2R5Pgo8L2h0bWw+CgotLV8wMDBfREI2UDE5ME1CMDI4 MDFFQTA4OTJCMzhGNTAzMjIwODk2Qzg4QTBEQjZQMTkwTUIwMjgwRVVSUF8tLQo= --===============2459877832682927586==-- From fabrice.bacchella at orange.fr Tue Aug 8 13:48:59 2017 Content-Type: multipart/mixed; boundary="===============0719399733473702658==" MIME-Version: 1.0 From: Fabrice Bacchella To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Tue, 08 Aug 2017 15:48:54 +0200 Message-ID: <08BEF1AD-557C-4A13-84F1-EE7918150AF7@orange.fr> In-Reply-To: 02172d33-e14e-aa8d-98f5-d920573e281c@upx.com --===============0719399733473702658== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable > Le 8 ao=C3=BBt 2017 =C3=A0 15:24, FERNANDO FREDIANI a =C3=A9crit : > = > That's something on the way RAID works, regardless what most 'super-ultra= ' powerfull hardware controller you may have. RAID 5 or 6 will never have t= he same write performance as a RAID 10 o 0 for example. Writeback caches ca= n deal with bursts well but they have a limit therefore there will always b= e a penalty compared to what else you could have. Hardware RAID5/6 can have better performance with quite common hardware tha= t software RAID0. I have seen many time on on even old servers that write l= atency (hitting the cache) was smaller that read latency that was going dir= ectly to the disk. I'm not talking about 'super-ultra' powerfull hardware. = An HP Smart Array P440ar with 2 GB flash is sell at 560=E2=82=AC, public pr= ice. Not cheap, but not ultra powerfull. It's now a matter of identifying the bootle neck, and how much money you ca= n throw at it. --===============0719399733473702658==-- From moacirferreira at hotmail.com Tue Aug 8 14:13:18 2017 Content-Type: multipart/mixed; boundary="===============4213557402878432470==" MIME-Version: 1.0 From: Moacir Ferreira To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Tue, 08 Aug 2017 11:39:23 +0000 Message-ID: In-Reply-To: 15dc15fdb18.2768.ffe2d6cdf294efc1bb86526523af9945@kafit.se --===============4213557402878432470== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --_000_DB6P190MB0280F066762EF81DE4270280C88A0DB6P190MB0280EURP_ Content-Type: text/plain; charset=3D"Windows-1252" Content-Transfer-Encoding: quoted-printable Thanks once again Johan! What would be your approach: JBOD straight or JBOD made of RAIDed bricks? Moacir ________________________________ From: Johan Bernhardsson Sent: Tuesday, August 8, 2017 11:24 AM To: Moacir Ferreira; Devin Acosta; users(a)ovirt.org Subject: Re: [ovirt-users] Good practices On ovirt gluster uses sharding. So all large files are broken up in small p= =3D ieces on the gluster bricks. /Johan On August 8, 2017 12:19:39 Moacir Ferreira w= ro=3D te: Thanks Johan, you brought "light" into my darkness! I went looking for the = =3D GlusterFS tiering how-to and it looks like quite simple to attach a SSD as = =3D hot tier. For those willing to read about it, go here: http://blog.gluster.= =3D org/2016/03/automated-tiering-in-gluster/ Now, I still have a question: VMs are made of very large .qcow2 files. My u= =3D nderstanding is that files in Gluster are kept all together in a single bri= =3D ck. If so, I will not benefit from tiering as a single SSD will not be big = =3D enough to fit all my large VM .qcow2 files. This would not be true if Glust= =3D er can store "blocks" of data that compose a large file spread on several b= =3D ricks. But if I am not wrong, this is one of key differences in between Glu= =3D sterFS and Ceph. Can you comment? Moacir ________________________________ From: Johan Bernhardsson Sent: Tuesday, August 8, 2017 7:03 AM To: Moacir Ferreira; Devin Acosta; users(a)ovirt.org Subject: Re: [ovirt-users] Good practices You attach the ssd as a hot tier with a gluster command. I don't think that= =3D gdeploy or ovirt gui can do it. The gluster docs and redhat docs explains tiering quite good. /Johan On August 8, 2017 07:06:42 Moacir Ferreira w= ro=3D te: Hi Devin, Please consider that for the OS I have a RAID 1. Now, lets say I use RAID 5= =3D to assemble a single disk on each server. In this case, the SSD will not m= =3D ake any difference, right? I guess that to be possible to use it, the SSD s= =3D hould not be part of the RAID 5. In this case I could create a logical volu= =3D me made of the RAIDed brick and then extend it using the SSD. I.e.: Using g= =3D deploy: [disktype] jbod .... [pv1] action=3D3Dcreate devices=3D3Dsdb, sdc wipefs=3D3Dyes ignore_vg_erros=3D3Dno [vg1] action=3D3Dcreate vgname=3D3Dgluster_vg_jbod pvname=3D3Dsdb ignore_vg_erros=3D3Dno [vg2] action=3D3Dextend vgname=3D3Dgluster_vg_jbod pvname=3D3Dsdc ignore_vg_erros=3D3Dno But will Gluster be able to auto-detect and use this SSD brick for tiering?= =3D Do I have to do some other configurations? Also, as the VM files (.qcow2) = =3D are quite big will I benefit from tiering? This is wrong and my approach sh= =3D ould be other? Thanks, Moacir ________________________________ From: Devin Acosta Sent: Monday, August 7, 2017 7:46 AM To: Moacir Ferreira; users(a)ovirt.org Subject: Re: [ovirt-users] Good practices Moacir, I have recently installed multiple Red Hat Virtualization hosts for several= =3D different companies, and have dealt with the Red Hat Support Team in depth= =3D about optimal configuration in regards to setting up GlusterFS most effici= =3D ently and I wanted to share with you what I learned. In general Red Hat Virtualization team frowns upon using each DISK of the s= =3D ystem as just a JBOD, sure there is some protection by having the data repl= =3D icated, however, the recommendation is to use RAID 6 (preferred) or RAID-5,= =3D or at least RAID-1 at the very least. Here is the direct quote from Red Hat when I asked about RAID and Bricks: "A typical Gluster configuration would use RAID underneath the bricks. RAID= =3D 6 is most typical as it gives you 2 disk failure protection, but RAID 5 co= =3D uld be used too. Once you have the RAIDed bricks, you'd then apply the desi= =3D red replication on top of that. The most popular way of doing this would be= =3D distributed replicated with 2x replication. In general you'll get better p= =3D erformance with larger bricks. 12 drives is often a sweet spot. Another opt= =3D ion would be to create a separate tier using all SSD=3D92s.=3D94 In order to SSD tiering from my understanding you would need 1 x NVMe drive= =3D in each server, or 4 x SSD hot tier (it needs to be distributed, replicate= =3D d for the hot tier if not using NVME). So with you only having 1 SSD drive = =3D in each server, I=3D92d suggest maybe looking into the NVME option. Since your using only 3-servers, what I=3D92d probably suggest is to do (2 = Re=3D plicas + Arbiter Node), this setup actually doesn=3D92t require the 3rd ser= ve=3D r to have big drives at all as it only stores meta-data about the files and= =3D not actually a full copy. Please see the attached document that was given to me by Red Hat to get mor= =3D e information on this. Hope this information helps you. -- Devin Acosta, RHCA, RHVCA Red Hat Certified Architect On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (moacirferreira(a)hotmail.= co=3D m) wrote: I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU = =3D sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use Gluste= =3D rFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dua= =3D l 10Gb NIC. So my intention is to create a loop like a server triangle usin= =3D g the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VM= =3D s around the pod (east /west traffic) while using the 10Gb interfaces for g= =3D iving services to the outside world (north/south traffic). This said, my first question is: How should I deploy GlusterFS in such oVir= =3D t scenario? My questions are: 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then= =3D create a GlusterFS using them? 2 - Instead, should I create a JBOD array made of all server's disks? 3 - What is the best Gluster configuration to provide for HA while not cons= =3D uming too much disk space? 4 - Does a oVirt hypervisor pod like I am planning to build, and the virtua= =3D lization environment, benefits from tiering when using a SSD disk? And yes,= =3D will Gluster do it by default or I have to configure it to do so? At the bottom line, what is the good practice for using GlusterFS in small = =3D pods for enterprises? You opinion/feedback will be really appreciated! Moacir _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users --_000_DB6P190MB0280F066762EF81DE4270280C88A0DB6P190MB0280EURP_ Content-Type: text/html; charset=3D"Windows-1252" Content-Transfer-Encoding: quoted-printable

Thanks once again Johan!

What would be your approach: JBOD straight or JBOD made of RAIDed bricks= =3D ?

Moacir

From: Johan Bernhards= son =3D <johan(a)kafit.se>
Sent: Tuesday, August 8, 2017 11:24 AM
To: Moacir Ferreira; Devin Acosta; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices

On ovirt gluster uses sharding= . =3D So all large files are broken up in small pieces on the gluster bricks.

/Johan

On August 8, 2017 12:19:39 Moacir Ferreira <moacirferreira(a)hotmail.com= &g=3D t; wrote:

Thanks Johan, you brought "light" into my darkness! I went loo= =3D king for the GlusterFS tiering how-to and it looks like quite simple to att= =3D ach a SSD as hot tier. For those willing to read about it, go here: http://blog.gluster.org/2016/03/automated-tiering-in-gluster/

Now, I still have a question: VMs are made of very large .qcow2 files. M= =3D y understanding is that files in Gluster are kept all together in a single = =3D brick. If so, I will not benefit from tiering as a single SSD will not be b= =3D ig enough to fit all my large VM .qcow2 files. This would not be true if Gluster can store "blocks&quo= =3D t; of data that compose a large file spread on several bricks. But if I am = =3D not wrong, this is one of key differences in between GlusterFS and Ceph. Ca= =3D n you comment?

Moacir

From: Johan Bernhards= son =3D <johan(a)kafit.se>
Sent: Tuesday, August 8, 2017 7:03 AM
To: Moacir Ferreira; Devin Acosta; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices

You attach the ssd as a hot ti= er=3D with a gluster command. I don't think that gdeploy or ovirt gui can do it.= =3D

The gluster docs and redhat do= cs=3D explains tiering quite good.

/Johan

On August 8, 2017 07:06:42 Moacir Ferreira <moacirferreira(a)hotmail.com= &g=3D t; wrote:

Hi Devin,

Please consider that for the OS I have a RAID 1. Now, lets say I use RAI= =3D D 5 to assemble a single disk on each server. In this case, the SSD will no= =3D t make any difference, right? I guess that to be possible to use it, the SS= =3D D should not be part of the RAID 5. In this case I could create a logical volume made of the RAIDed brick a= =3D nd then extend it using the SSD. I.e.: Using gdeploy:

[disktype]

jbod

....

[pv1]

action=3D3Dcreate

devices=3D3Dsdb, sdc

wipefs=3D3Dyes

ignore_vg_erros=3D3Dno

[vg1]

action=3D3Dcreate

vgname=3D3Dgluster_vg_jbod

pvname=3D3Dsdb

ignore_vg_erros=3D3Dno

[vg2]

action=3D3Dextend

vgname=3D3Dgluster_vg_jbod

pvname=3D3Dsdc

ignore_vg_erros=3D3Dno

But will Gluster be able to auto-detect and use this SSD brick for tieri= =3D ng? Do I have to do some other configurations? Also, as the VM files (.qcow= =3D 2) are quite big will I benefit from tiering? This is wrong and my approach= =3D should be other?

Thanks,

Moacir

From: Devin Acosta &l= t;de=3D vin(a)pabstatencio.com>
Sent: Monday, August 7, 2017 7:46 AM
To: Moacir Ferreira; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices

Moacir,

I have recently installed multiple Red Hat Virtualizat= io=3D n hosts for several different companies, and have dealt with the Red Hat Su= =3D pport Team in depth about optimal configuration in regards to setting up GlusterFS most efficiently and I wanted to share = =3D with you what I learned.

In general Red Hat Virtualization team frowns upon usi= ng=3D each DISK of the system as just a JBOD, sure there is some protection by h= =3D aving the data replicated, however, the recommendation is to use RAID 6 (preferred) or RAID-5, or at least RAID-1 = =3D at the very least.

Here is the direct quote from Red Hat when I asked about RAID and Bri= =3D cks:

"A typical Gluster configuration would use RAID underneath the bri= =3D cks. RAID 6 is most typical as it gives you 2 disk failure protection, but = =3D RAID 5 could be used too. Once you have the RAIDed bricks, you'd then apply the desired replication on top of that. Th= =3D e most popular way of doing this would be distributed replicated with 2x re= =3D plication. In general you'll get better performance with larger bricks= =3D . 12 drives is often a sweet spot. Another option would be to create a separate tier using all SSD=3D92s.=3D94 <= /i><=3D /font>

In order to SSD tiering from my understanding you would need 1 x N= =3D VMe drive in each server, or 4 x SSD hot tier (it needs to be distributed, = =3D replicated for the hot tier if not using NVME). So with you only having 1 SSD drive in each server, I=3D92d suggest m= ay=3D be looking into the NVME option.

Since your using only 3-servers, what I=3D92d probably suggest is to do= (=3D 2 Replicas + Arbiter Node), this setup actually doesn=3D92t require the= 3=3D rd server to have big drives at all as it only stores meta-data about the files and not actually a full copy.

Please see the attached document that was given to me by Red Hat to get= =3D more information on this. Hope this information helps you.

--

Devin Acosta, RHCA, RHVCA

Red Hat Certified Architect

On August 6, 2017 at 7:29:29 PM, Moacir Ferreira = (<=3D a href=3D3D"mailto:moacirferreira(a)hotmail.com">moacirferreira(a)hotmail.c= om=3D ) wrote:

I am willing to assemble a oVirt "pod", made of 3 server= =3D s, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The id= =3D ea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual= =3D 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop like a server triangle using the 40Gb NICs for virtual= =3D ization files (VMs .qcow2) access and to move VMs around the pod (east /wes= =3D t traffic) while using the 10Gb interfaces for giving services to the outsi= =3D de world (north/south traffic).

This said, my first question is: How should I deploy GlusterFS in such o= =3D Virt scenario? My questions are:

1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and t= =3D hen create a GlusterFS using them?

2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not c= =3D onsuming too much disk space?

4 - Does a oVirt hypervisor pod like I am planning to build, and the vir= =3D tualization environment, benefits from tiering when using a SSD disk? And y= =3D es, will Gluster do it by default or I have to configure it to do so?

At the bottom line, what is the good practice for using GlusterFS in sma= =3D ll pods for enterprises?

You opinion/feedback will be really appreciated!

Moacir

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ov= ir=3D t.org/mailman/listinfo/users

_______________________________________________
Users mailing list
Us=3D ers(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

--_000_DB6P190MB0280F066762EF81DE4270280C88A0DB6P190MB0280EURP_-- --===============4213557402878432470== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" LS1fMDAwX0RCNlAxOTBNQjAyODBGMDY2NzYyRUY4MURFNDI3MDI4MEM4OEEwREI2UDE5ME1CMDI4 MEVVUlBfCkNvbnRlbnQtVHlwZTogdGV4dC9wbGFpbjsgY2hhcnNldD0iV2luZG93cy0xMjUyIgpD b250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiBxdW90ZWQtcHJpbnRhYmxlCgpUaGFua3Mgb25jZSBh Z2FpbiBKb2hhbiEKCgpXaGF0IHdvdWxkIGJlIHlvdXIgYXBwcm9hY2g6IEpCT0Qgc3RyYWlnaHQg b3IgSkJPRCBtYWRlIG9mIFJBSURlZCBicmlja3M/CgoKTW9hY2lyCgpfX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fXwpGcm9tOiBKb2hhbiBCZXJuaGFyZHNzb24gPGpvaGFuQGthZml0LnNl PgpTZW50OiBUdWVzZGF5LCBBdWd1c3QgOCwgMjAxNyAxMToyNCBBTQpUbzogTW9hY2lyIEZlcnJl aXJhOyBEZXZpbiBBY29zdGE7IHVzZXJzQG92aXJ0Lm9yZwpTdWJqZWN0OiBSZTogW292aXJ0LXVz ZXJzXSBHb29kIHByYWN0aWNlcwoKCk9uIG92aXJ0IGdsdXN0ZXIgdXNlcyBzaGFyZGluZy4gU28g YWxsIGxhcmdlIGZpbGVzIGFyZSBicm9rZW4gdXAgaW4gc21hbGwgcD0KaWVjZXMgb24gdGhlIGds dXN0ZXIgYnJpY2tzLgoKL0pvaGFuCgpPbiBBdWd1c3QgOCwgMjAxNyAxMjoxOTozOSBNb2FjaXIg RmVycmVpcmEgPG1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY29tPiB3cm89CnRlOgoKVGhhbmtzIEpv aGFuLCB5b3UgYnJvdWdodCAibGlnaHQiIGludG8gbXkgZGFya25lc3MhIEkgd2VudCBsb29raW5n IGZvciB0aGUgPQpHbHVzdGVyRlMgdGllcmluZyBob3ctdG8gYW5kIGl0IGxvb2tzIGxpa2UgcXVp dGUgc2ltcGxlIHRvIGF0dGFjaCBhIFNTRCBhcyA9CmhvdCB0aWVyLiBGb3IgdGhvc2Ugd2lsbGlu ZyB0byByZWFkIGFib3V0IGl0LCBnbyBoZXJlOiBodHRwOi8vYmxvZy5nbHVzdGVyLj0Kb3JnLzIw MTYvMDMvYXV0b21hdGVkLXRpZXJpbmctaW4tZ2x1c3Rlci8KCgpOb3csIEkgc3RpbGwgaGF2ZSBh IHF1ZXN0aW9uOiBWTXMgYXJlIG1hZGUgb2YgdmVyeSBsYXJnZSAucWNvdzIgZmlsZXMuIE15IHU9 Cm5kZXJzdGFuZGluZyBpcyB0aGF0IGZpbGVzIGluIEdsdXN0ZXIgYXJlIGtlcHQgYWxsIHRvZ2V0 aGVyIGluIGEgc2luZ2xlIGJyaT0KY2suIElmIHNvLCBJIHdpbGwgbm90IGJlbmVmaXQgZnJvbSB0 aWVyaW5nIGFzIGEgc2luZ2xlIFNTRCB3aWxsIG5vdCBiZSBiaWcgPQplbm91Z2ggdG8gZml0IGFs bCBteSBsYXJnZSBWTSAucWNvdzIgZmlsZXMuIFRoaXMgd291bGQgbm90IGJlIHRydWUgaWYgR2x1 c3Q9CmVyIGNhbiBzdG9yZSAiYmxvY2tzIiBvZiBkYXRhIHRoYXQgY29tcG9zZSBhIGxhcmdlIGZp bGUgc3ByZWFkIG9uIHNldmVyYWwgYj0Kcmlja3MuIEJ1dCBpZiBJIGFtIG5vdCB3cm9uZywgdGhp cyBpcyBvbmUgb2Yga2V5IGRpZmZlcmVuY2VzIGluIGJldHdlZW4gR2x1PQpzdGVyRlMgYW5kIENl cGguIENhbiB5b3UgY29tbWVudD8KCgpNb2FjaXIKCgpfX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fXwpGcm9tOiBKb2hhbiBCZXJuaGFyZHNzb24gPGpvaGFuQGthZml0LnNlPgpTZW50OiBU dWVzZGF5LCBBdWd1c3QgOCwgMjAxNyA3OjAzIEFNClRvOiBNb2FjaXIgRmVycmVpcmE7IERldmlu IEFjb3N0YTsgdXNlcnNAb3ZpcnQub3JnClN1YmplY3Q6IFJlOiBbb3ZpcnQtdXNlcnNdIEdvb2Qg cHJhY3RpY2VzCgoKWW91IGF0dGFjaCB0aGUgc3NkIGFzIGEgaG90IHRpZXIgd2l0aCBhIGdsdXN0 ZXIgY29tbWFuZC4gSSBkb24ndCB0aGluayB0aGF0PQogZ2RlcGxveSBvciBvdmlydCBndWkgY2Fu IGRvIGl0LgoKVGhlIGdsdXN0ZXIgZG9jcyBhbmQgcmVkaGF0IGRvY3MgZXhwbGFpbnMgdGllcmlu ZyBxdWl0ZSBnb29kLgoKL0pvaGFuCgpPbiBBdWd1c3QgOCwgMjAxNyAwNzowNjo0MiBNb2FjaXIg RmVycmVpcmEgPG1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY29tPiB3cm89CnRlOgoKSGkgRGV2aW4s CgoKUGxlYXNlIGNvbnNpZGVyIHRoYXQgZm9yIHRoZSBPUyBJIGhhdmUgYSBSQUlEIDEuIE5vdywg bGV0cyBzYXkgSSB1c2UgUkFJRCA1PQogdG8gYXNzZW1ibGUgYSBzaW5nbGUgZGlzayBvbiBlYWNo IHNlcnZlci4gSW4gdGhpcyBjYXNlLCB0aGUgU1NEIHdpbGwgbm90IG09CmFrZSBhbnkgZGlmZmVy ZW5jZSwgcmlnaHQ/IEkgZ3Vlc3MgdGhhdCB0byBiZSBwb3NzaWJsZSB0byB1c2UgaXQsIHRoZSBT U0Qgcz0KaG91bGQgbm90IGJlIHBhcnQgb2YgdGhlIFJBSUQgNS4gSW4gdGhpcyBjYXNlIEkgY291 bGQgY3JlYXRlIGEgbG9naWNhbCB2b2x1PQptZSBtYWRlIG9mIHRoZSBSQUlEZWQgYnJpY2sgYW5k IHRoZW4gZXh0ZW5kIGl0IHVzaW5nIHRoZSBTU0QuIEkuZS46IFVzaW5nIGc9CmRlcGxveToKCgpb ZGlza3R5cGVdCgpqYm9kCgouLi4uCgpbcHYxXQoKYWN0aW9uPTNEY3JlYXRlCgpkZXZpY2VzPTNE c2RiLCBzZGMKCndpcGVmcz0zRHllcwoKaWdub3JlX3ZnX2Vycm9zPTNEbm8KCgpbdmcxXQoKYWN0 aW9uPTNEY3JlYXRlCgp2Z25hbWU9M0RnbHVzdGVyX3ZnX2pib2QKCnB2bmFtZT0zRHNkYgoKaWdu b3JlX3ZnX2Vycm9zPTNEbm8KCgpbdmcyXQoKYWN0aW9uPTNEZXh0ZW5kCgp2Z25hbWU9M0RnbHVz dGVyX3ZnX2pib2QKCnB2bmFtZT0zRHNkYwoKaWdub3JlX3ZnX2Vycm9zPTNEbm8KCgpCdXQgd2ls bCBHbHVzdGVyIGJlIGFibGUgdG8gYXV0by1kZXRlY3QgYW5kIHVzZSB0aGlzIFNTRCBicmljayBm b3IgdGllcmluZz89CiBEbyBJIGhhdmUgdG8gZG8gc29tZSBvdGhlciBjb25maWd1cmF0aW9ucz8g QWxzbywgYXMgdGhlIFZNIGZpbGVzICgucWNvdzIpID0KYXJlIHF1aXRlIGJpZyB3aWxsIEkgYmVu ZWZpdCBmcm9tIHRpZXJpbmc/IFRoaXMgaXMgd3JvbmcgYW5kIG15IGFwcHJvYWNoIHNoPQpvdWxk IGJlIG90aGVyPwoKClRoYW5rcywKCk1vYWNpcgoKCl9fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fCkZyb206IERldmluIEFjb3N0YSA8ZGV2aW5AcGFic3RhdGVuY2lvLmNvbT4KU2VudDog TW9uZGF5LCBBdWd1c3QgNywgMjAxNyA3OjQ2IEFNClRvOiBNb2FjaXIgRmVycmVpcmE7IHVzZXJz QG92aXJ0Lm9yZwpTdWJqZWN0OiBSZTogW292aXJ0LXVzZXJzXSBHb29kIHByYWN0aWNlcwoKCk1v YWNpciwKCkkgaGF2ZSByZWNlbnRseSBpbnN0YWxsZWQgbXVsdGlwbGUgUmVkIEhhdCBWaXJ0dWFs aXphdGlvbiBob3N0cyBmb3Igc2V2ZXJhbD0KIGRpZmZlcmVudCBjb21wYW5pZXMsIGFuZCBoYXZl IGRlYWx0IHdpdGggdGhlIFJlZCBIYXQgU3VwcG9ydCBUZWFtIGluIGRlcHRoPQogYWJvdXQgb3B0 aW1hbCBjb25maWd1cmF0aW9uIGluIHJlZ2FyZHMgdG8gc2V0dGluZyB1cCBHbHVzdGVyRlMgbW9z dCBlZmZpY2k9CmVudGx5IGFuZCBJIHdhbnRlZCB0byBzaGFyZSB3aXRoIHlvdSB3aGF0IEkgbGVh cm5lZC4KCkluIGdlbmVyYWwgUmVkIEhhdCBWaXJ0dWFsaXphdGlvbiB0ZWFtIGZyb3ducyB1cG9u IHVzaW5nIGVhY2ggRElTSyBvZiB0aGUgcz0KeXN0ZW0gYXMganVzdCBhIEpCT0QsIHN1cmUgdGhl cmUgaXMgc29tZSBwcm90ZWN0aW9uIGJ5IGhhdmluZyB0aGUgZGF0YSByZXBsPQppY2F0ZWQsIGhv d2V2ZXIsIHRoZSByZWNvbW1lbmRhdGlvbiBpcyB0byB1c2UgUkFJRCA2IChwcmVmZXJyZWQpIG9y IFJBSUQtNSw9CiBvciBhdCBsZWFzdCBSQUlELTEgYXQgdGhlIHZlcnkgbGVhc3QuCgpIZXJlIGlz IHRoZSBkaXJlY3QgcXVvdGUgZnJvbSBSZWQgSGF0IHdoZW4gSSBhc2tlZCBhYm91dCBSQUlEIGFu ZCBCcmlja3M6CgoiQSB0eXBpY2FsIEdsdXN0ZXIgY29uZmlndXJhdGlvbiB3b3VsZCB1c2UgUkFJ RCB1bmRlcm5lYXRoIHRoZSBicmlja3MuIFJBSUQ9CiA2IGlzIG1vc3QgdHlwaWNhbCBhcyBpdCBn aXZlcyB5b3UgMiBkaXNrIGZhaWx1cmUgcHJvdGVjdGlvbiwgYnV0IFJBSUQgNSBjbz0KdWxkIGJl IHVzZWQgdG9vLiBPbmNlIHlvdSBoYXZlIHRoZSBSQUlEZWQgYnJpY2tzLCB5b3UnZCB0aGVuIGFw cGx5IHRoZSBkZXNpPQpyZWQgcmVwbGljYXRpb24gb24gdG9wIG9mIHRoYXQuIFRoZSBtb3N0IHBv cHVsYXIgd2F5IG9mIGRvaW5nIHRoaXMgd291bGQgYmU9CiBkaXN0cmlidXRlZCByZXBsaWNhdGVk IHdpdGggMnggcmVwbGljYXRpb24uIEluIGdlbmVyYWwgeW91J2xsIGdldCBiZXR0ZXIgcD0KZXJm b3JtYW5jZSB3aXRoIGxhcmdlciBicmlja3MuIDEyIGRyaXZlcyBpcyBvZnRlbiBhIHN3ZWV0IHNw b3QuIEFub3RoZXIgb3B0PQppb24gd291bGQgYmUgdG8gY3JlYXRlIGEgc2VwYXJhdGUgdGllciB1 c2luZyBhbGwgU1NEPTkycy49OTQKCkluIG9yZGVyIHRvIFNTRCB0aWVyaW5nIGZyb20gbXkgdW5k ZXJzdGFuZGluZyB5b3Ugd291bGQgbmVlZCAxIHggTlZNZSBkcml2ZT0KIGluIGVhY2ggc2VydmVy LCBvciA0IHggU1NEIGhvdCB0aWVyIChpdCBuZWVkcyB0byBiZSBkaXN0cmlidXRlZCwgcmVwbGlj YXRlPQpkIGZvciB0aGUgaG90IHRpZXIgaWYgbm90IHVzaW5nIE5WTUUpLiBTbyB3aXRoIHlvdSBv bmx5IGhhdmluZyAxIFNTRCBkcml2ZSA9CmluIGVhY2ggc2VydmVyLCBJPTkyZCBzdWdnZXN0IG1h eWJlIGxvb2tpbmcgaW50byB0aGUgTlZNRSBvcHRpb24uCgpTaW5jZSB5b3VyIHVzaW5nIG9ubHkg My1zZXJ2ZXJzLCB3aGF0IEk9OTJkIHByb2JhYmx5IHN1Z2dlc3QgaXMgdG8gZG8gKDIgUmU9CnBs aWNhcyArIEFyYml0ZXIgTm9kZSksIHRoaXMgc2V0dXAgYWN0dWFsbHkgZG9lc249OTJ0IHJlcXVp cmUgdGhlIDNyZCBzZXJ2ZT0KciB0byBoYXZlIGJpZyBkcml2ZXMgYXQgYWxsIGFzIGl0IG9ubHkg c3RvcmVzIG1ldGEtZGF0YSBhYm91dCB0aGUgZmlsZXMgYW5kPQogbm90IGFjdHVhbGx5IGEgZnVs bCBjb3B5LgoKUGxlYXNlIHNlZSB0aGUgYXR0YWNoZWQgZG9jdW1lbnQgdGhhdCB3YXMgZ2l2ZW4g dG8gbWUgYnkgUmVkIEhhdCB0byBnZXQgbW9yPQplIGluZm9ybWF0aW9uIG9uIHRoaXMuIEhvcGUg dGhpcyBpbmZvcm1hdGlvbiBoZWxwcyB5b3UuCgoKLS0KCkRldmluIEFjb3N0YSwgUkhDQSwgUkhW Q0EKUmVkIEhhdCBDZXJ0aWZpZWQgQXJjaGl0ZWN0CgoKT24gQXVndXN0IDYsIDIwMTcgYXQgNzoy OToyOSBQTSwgTW9hY2lyIEZlcnJlaXJhIChtb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvPQptPG1h aWx0bzptb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNvbT4pIHdyb3RlOgoKSSBhbSB3aWxsaW5nIHRv IGFzc2VtYmxlIGEgb1ZpcnQgInBvZCIsIG1hZGUgb2YgMyBzZXJ2ZXJzLCBlYWNoIHdpdGggMiBD UFUgPQpzb2NrZXRzIG9mIDEyIGNvcmVzLCAyNTZHQiBSQU0sIDcgSEREIDEwSywgMSBTU0QuIFRo ZSBpZGVhIGlzIHRvIHVzZSBHbHVzdGU9CnJGUyB0byBwcm92aWRlIEhBIGZvciB0aGUgVk1zLiBU aGUgMyBzZXJ2ZXJzIGhhdmUgYSBkdWFsIDQwR2IgTklDIGFuZCBhIGR1YT0KbCAxMEdiIE5JQy4g U28gbXkgaW50ZW50aW9uIGlzIHRvIGNyZWF0ZSBhIGxvb3AgbGlrZSBhIHNlcnZlciB0cmlhbmds ZSB1c2luPQpnIHRoZSA0MEdiIE5JQ3MgZm9yIHZpcnR1YWxpemF0aW9uIGZpbGVzIChWTXMgLnFj b3cyKSBhY2Nlc3MgYW5kIHRvIG1vdmUgVk09CnMgYXJvdW5kIHRoZSBwb2QgKGVhc3QgL3dlc3Qg dHJhZmZpYykgd2hpbGUgdXNpbmcgdGhlIDEwR2IgaW50ZXJmYWNlcyBmb3IgZz0KaXZpbmcgc2Vy dmljZXMgdG8gdGhlIG91dHNpZGUgd29ybGQgKG5vcnRoL3NvdXRoIHRyYWZmaWMpLgoKClRoaXMg c2FpZCwgbXkgZmlyc3QgcXVlc3Rpb24gaXM6IEhvdyBzaG91bGQgSSBkZXBsb3kgR2x1c3RlckZT IGluIHN1Y2ggb1Zpcj0KdCBzY2VuYXJpbz8gTXkgcXVlc3Rpb25zIGFyZToKCgoxIC0gU2hvdWxk IEkgY3JlYXRlIDMgUkFJRCAoaS5lLjogUkFJRCA1KSwgb25lIG9uIGVhY2ggb1ZpcnQgbm9kZSwg YW5kIHRoZW49CiBjcmVhdGUgYSBHbHVzdGVyRlMgdXNpbmcgdGhlbT8KCjIgLSBJbnN0ZWFkLCBz aG91bGQgSSBjcmVhdGUgYSBKQk9EIGFycmF5IG1hZGUgb2YgYWxsIHNlcnZlcidzIGRpc2tzPwoK MyAtIFdoYXQgaXMgdGhlIGJlc3QgR2x1c3RlciBjb25maWd1cmF0aW9uIHRvIHByb3ZpZGUgZm9y IEhBIHdoaWxlIG5vdCBjb25zPQp1bWluZyB0b28gbXVjaCBkaXNrIHNwYWNlPwoKNCAtIERvZXMg YSBvVmlydCBoeXBlcnZpc29yIHBvZCBsaWtlIEkgYW0gcGxhbm5pbmcgdG8gYnVpbGQsIGFuZCB0 aGUgdmlydHVhPQpsaXphdGlvbiBlbnZpcm9ubWVudCwgYmVuZWZpdHMgZnJvbSB0aWVyaW5nIHdo ZW4gdXNpbmcgYSBTU0QgZGlzaz8gQW5kIHllcyw9CiB3aWxsIEdsdXN0ZXIgZG8gaXQgYnkgZGVm YXVsdCBvciBJIGhhdmUgdG8gY29uZmlndXJlIGl0IHRvIGRvIHNvPwoKCkF0IHRoZSBib3R0b20g bGluZSwgd2hhdCBpcyB0aGUgZ29vZCBwcmFjdGljZSBmb3IgdXNpbmcgR2x1c3RlckZTIGluIHNt YWxsID0KcG9kcyBmb3IgZW50ZXJwcmlzZXM/CgoKWW91IG9waW5pb24vZmVlZGJhY2sgd2lsbCBi ZSByZWFsbHkgYXBwcmVjaWF0ZWQhCgpNb2FjaXIKCl9fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fClVzZXJzIG1haWxpbmcgbGlzdApVc2Vyc0BvdmlydC5vcmc8 bWFpbHRvOlVzZXJzQG92aXJ0Lm9yZz4KaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xp c3RpbmZvL3VzZXJzCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fClVzZXJzIG1haWxpbmcgbGlzdApVc2Vyc0BvdmlydC5vcmc8bWFpbHRvOlVzZXJzJTQwb3Zp cnQub3JnPgpodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMKCgot LV8wMDBfREI2UDE5ME1CMDI4MEYwNjY3NjJFRjgxREU0MjcwMjgwQzg4QTBEQjZQMTkwTUIwMjgw RVVSUF8KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWw7IGNoYXJzZXQ9IldpbmRvd3MtMTI1MiIKQ29u dGVudC1UcmFuc2Zlci1FbmNvZGluZzogcXVvdGVkLXByaW50YWJsZQoKPGh0bWw+CjxoZWFkPgo8 bWV0YSBodHRwLWVxdWl2PTNEIkNvbnRlbnQtVHlwZSIgY29udGVudD0zRCJ0ZXh0L2h0bWw7IGNo YXJzZXQ9M0RXaW5kb3dzLTE9CjI1MiI+CjxzdHlsZSB0eXBlPTNEInRleHQvY3NzIiBzdHlsZT0z RCJkaXNwbGF5Om5vbmU7Ij48IS0tIFAge21hcmdpbi10b3A6MDttYXJnaT0Kbi1ib3R0b206MDt9 IC0tPjwvc3R5bGU+CjwvaGVhZD4KPGJvZHkgZGlyPTNEImx0ciI+CjxkaXYgaWQ9M0QiZGl2dGFn ZGVmYXVsdHdyYXBwZXIiIHN0eWxlPTNEImZvbnQtc2l6ZToxMnB0O2NvbG9yOiMwMDAwMDA7Zm9u dD0KLWZhbWlseTpDYWxpYnJpLEhlbHZldGljYSxzYW5zLXNlcmlmOyIgZGlyPTNEImx0ciI+Cjxw PlRoYW5rcyBvbmNlIGFnYWluIEpvaGFuITxicj4KPC9wPgo8YnI+CjxwPldoYXQgd291bGQgYmUg eW91ciBhcHByb2FjaDogSkJPRCBzdHJhaWdodCBvciBKQk9EIG1hZGUgb2YgUkFJRGVkIGJyaWNr cz0KPzwvcD4KPHA+PGJyPgo8L3A+CjxwPk1vYWNpcjxicj4KPC9wPgo8YnI+CjxkaXYgc3R5bGU9 M0QiY29sb3I6IHJnYig0OSwgNTUsIDU3KTsiPgo8aHIgdGFiaW5kZXg9M0QiLTEiIHN0eWxlPTNE ImRpc3BsYXk6aW5saW5lLWJsb2NrOyB3aWR0aDo5OCUiPgo8ZGl2IGlkPTNEImRpdlJwbHlGd2RN c2ciIGRpcj0zRCJsdHIiPjxmb250IHN0eWxlPTNEImZvbnQtc2l6ZToxMXB0IiBmYWNlPQo9M0Qi Q2FsaWJyaSwgc2Fucy1zZXJpZiIgY29sb3I9M0QiIzAwMDAwMCI+PGI+RnJvbTo8L2I+IEpvaGFu IEJlcm5oYXJkc3NvbiA9CiZsdDtqb2hhbkBrYWZpdC5zZSZndDs8YnI+CjxiPlNlbnQ6PC9iPiBU dWVzZGF5LCBBdWd1c3QgOCwgMjAxNyAxMToyNCBBTTxicj4KPGI+VG86PC9iPiBNb2FjaXIgRmVy cmVpcmE7IERldmluIEFjb3N0YTsgdXNlcnNAb3ZpcnQub3JnPGJyPgo8Yj5TdWJqZWN0OjwvYj4g UmU6IFtvdmlydC11c2Vyc10gR29vZCBwcmFjdGljZXM8L2ZvbnQ+CjxkaXY+Jm5ic3A7PC9kaXY+ CjwvZGl2Pgo8ZGl2Pgo8ZGl2IHN0eWxlPTNEImNvbG9yOmJsYWNrIj4KPGRpdiBzdHlsZT0zRCJj b2xvcjpibGFjayI+CjxwIHN0eWxlPTNEIm1hcmdpbjowIDAgMWVtIDA7IGNvbG9yOmJsYWNrIj5P biBvdmlydCBnbHVzdGVyIHVzZXMgc2hhcmRpbmcuID0KU28gYWxsIGxhcmdlIGZpbGVzIGFyZSBi cm9rZW4gdXAgaW4gc21hbGwgcGllY2VzIG9uIHRoZSBnbHVzdGVyIGJyaWNrcy48L3A+CjxwIHN0 eWxlPTNEIm1hcmdpbjowIDAgMWVtIDA7IGNvbG9yOmJsYWNrIj4vSm9oYW48L3A+CjwvZGl2Pgo8 ZGl2IHN0eWxlPTNEImNvbG9yOmJsYWNrIj4KPHAgc3R5bGU9M0QiY29sb3I6YmxhY2s7IGZvbnQt c2l6ZToxMHB0OyBmb250LWZhbWlseTpBcmlhbCxzYW5zLXNlcmlmOyBtYXJnPQppbjoxMHB0IDAi PgpPbiBBdWd1c3QgOCwgMjAxNyAxMjoxOTozOSBNb2FjaXIgRmVycmVpcmEgJmx0O21vYWNpcmZl cnJlaXJhQGhvdG1haWwuY29tJmc9CnQ7IHdyb3RlOjwvcD4KPGJsb2NrcXVvdGUgdHlwZT0zRCJj aXRlIiBjbGFzcz0zRCJnbWFpbF9xdW90ZSIgc3R5bGU9M0QibWFyZ2luOjAgMCAwIDAuNzVlPQp4 OyBib3JkZXItbGVmdDoxcHggc29saWQgIzgwODA4MDsgcGFkZGluZy1sZWZ0OjAuNzVleCI+Cjxk aXYgaWQ9M0QiZGl2dGFnZGVmYXVsdHdyYXBwZXIiIGRpcj0zRCJsdHIiIHN0eWxlPTNEImZvbnQt c2l6ZToxMnB0OyBjb2xvcj0KOiMwMDAwMDA7IGZvbnQtZmFtaWx5OkNhbGlicmksSGVsdmV0aWNh LHNhbnMtc2VyaWYiPgo8cD5UaGFua3MgSm9oYW4sIHlvdSBicm91Z2h0ICZxdW90O2xpZ2h0JnF1 b3Q7IGludG8gbXkgZGFya25lc3MhIEkgd2VudCBsb289CmtpbmcgZm9yIHRoZSBHbHVzdGVyRlMg dGllcmluZyBob3ctdG8gYW5kIGl0IGxvb2tzIGxpa2UgcXVpdGUgc2ltcGxlIHRvIGF0dD0KYWNo IGEgU1NEIGFzIGhvdCB0aWVyLiBGb3IgdGhvc2Ugd2lsbGluZyB0byByZWFkIGFib3V0IGl0LCBn byBoZXJlOgo8YSBocmVmPTNEImh0dHA6Ly9ibG9nLmdsdXN0ZXIub3JnLzIwMTYvMDMvYXV0b21h dGVkLXRpZXJpbmctaW4tZ2x1c3Rlci8iIGM9Cmxhc3M9M0QiT1dBQXV0b0xpbmsiIGlkPTNEIkxQ bG5rOTM5MzQzIiBwcmV2aWV3cmVtb3ZlZD0zRCJ0cnVlIj4KaHR0cDovL2Jsb2cuZ2x1c3Rlci5v cmcvMjAxNi8wMy9hdXRvbWF0ZWQtdGllcmluZy1pbi1nbHVzdGVyLzwvYT48L3A+CjxwPjxicj4K PC9wPgo8cD5Ob3csIEkgc3RpbGwgaGF2ZSBhIHF1ZXN0aW9uOiBWTXMgYXJlIG1hZGUgb2YgdmVy eSBsYXJnZSAucWNvdzIgZmlsZXMuIE09CnkgdW5kZXJzdGFuZGluZyBpcyB0aGF0IGZpbGVzIGlu IEdsdXN0ZXIgYXJlIGtlcHQgYWxsIHRvZ2V0aGVyIGluIGEgc2luZ2xlID0KYnJpY2suIElmIHNv LCBJIHdpbGwgbm90IGJlbmVmaXQgZnJvbSB0aWVyaW5nIGFzIGEgc2luZ2xlIFNTRCB3aWxsIG5v dCBiZSBiPQppZyBlbm91Z2ggdG8gZml0IGFsbCBteSBsYXJnZSBWTQogLnFjb3cyIGZpbGVzLiBU aGlzIHdvdWxkIG5vdCBiZSB0cnVlIGlmIEdsdXN0ZXIgY2FuIHN0b3JlICZxdW90O2Jsb2NrcyZx dW89CnQ7IG9mIGRhdGEgdGhhdCBjb21wb3NlIGEgbGFyZ2UgZmlsZSBzcHJlYWQgb24gc2V2ZXJh bCBicmlja3MuIEJ1dCBpZiBJIGFtID0Kbm90IHdyb25nLCB0aGlzIGlzIG9uZSBvZiBrZXkgZGlm ZmVyZW5jZXMgaW4gYmV0d2VlbiBHbHVzdGVyRlMgYW5kIENlcGguIENhPQpuIHlvdSBjb21tZW50 PzwvcD4KPHA+PGJyPgo8L3A+CjxwPk1vYWNpcjxicj4KPC9wPgo8YnI+Cjxicj4KPGRpdiBzdHls ZT0zRCJjb2xvcjpyZ2IoNDksNTUsNTcpIj4KPGhyIHRhYmluZGV4PTNEIi0xIiBzdHlsZT0zRCJk aXNwbGF5OmlubGluZS1ibG9jazsgd2lkdGg6OTglIj4KPGRpdiBpZD0zRCJkaXZScGx5RndkTXNn IiBkaXI9M0QibHRyIj48Zm9udCBzdHlsZT0zRCJmb250LXNpemU6MTFwdCIgZmFjZT0KPTNEIkNh bGlicmksIHNhbnMtc2VyaWYiIGNvbG9yPTNEIiMwMDAwMDAiPjxiPkZyb206PC9iPiBKb2hhbiBC ZXJuaGFyZHNzb24gPQombHQ7am9oYW5Aa2FmaXQuc2UmZ3Q7PGJyPgo8Yj5TZW50OjwvYj4gVHVl c2RheSwgQXVndXN0IDgsIDIwMTcgNzowMyBBTTxicj4KPGI+VG86PC9iPiBNb2FjaXIgRmVycmVp cmE7IERldmluIEFjb3N0YTsgdXNlcnNAb3ZpcnQub3JnPGJyPgo8Yj5TdWJqZWN0OjwvYj4gUmU6 IFtvdmlydC11c2Vyc10gR29vZCBwcmFjdGljZXM8L2ZvbnQ+CjxkaXY+Jm5ic3A7PC9kaXY+Cjwv ZGl2Pgo8ZGl2Pgo8ZGl2IHN0eWxlPTNEImNvbG9yOmJsYWNrIj4KPGRpdiBzdHlsZT0zRCJjb2xv cjpibGFjayI+CjxwIHN0eWxlPTNEIm1hcmdpbjowIDAgMWVtIDA7IGNvbG9yOmJsYWNrIj5Zb3Ug YXR0YWNoIHRoZSBzc2QgYXMgYSBob3QgdGllcj0KIHdpdGggYSBnbHVzdGVyIGNvbW1hbmQuIEkg ZG9uJ3QgdGhpbmsgdGhhdCBnZGVwbG95IG9yIG92aXJ0IGd1aSBjYW4gZG8gaXQuPQo8L3A+Cjxw IHN0eWxlPTNEIm1hcmdpbjowIDAgMWVtIDA7IGNvbG9yOmJsYWNrIj5UaGUgZ2x1c3RlciBkb2Nz IGFuZCByZWRoYXQgZG9jcz0KIGV4cGxhaW5zIHRpZXJpbmcgcXVpdGUgZ29vZC48L3A+CjxwIHN0 eWxlPTNEIm1hcmdpbjowIDAgMWVtIDA7IGNvbG9yOmJsYWNrIj4vSm9oYW48L3A+CjwvZGl2Pgo8 ZGl2IHN0eWxlPTNEImNvbG9yOmJsYWNrIj4KPHAgc3R5bGU9M0QiY29sb3I6YmxhY2s7IGZvbnQt c2l6ZToxMHB0OyBmb250LWZhbWlseTpBcmlhbCxzYW5zLXNlcmlmOyBtYXJnPQppbjoxMHB0IDAi PgpPbiBBdWd1c3QgOCwgMjAxNyAwNzowNjo0MiBNb2FjaXIgRmVycmVpcmEgJmx0O21vYWNpcmZl cnJlaXJhQGhvdG1haWwuY29tJmc9CnQ7IHdyb3RlOjwvcD4KPGJsb2NrcXVvdGUgdHlwZT0zRCJj aXRlIiBjbGFzcz0zRCJnbWFpbF9xdW90ZSIgc3R5bGU9M0QibWFyZ2luOjAgMCAwIDAuNzVlPQp4 OyBib3JkZXItbGVmdDoxcHggc29saWQgIzgwODA4MDsgcGFkZGluZy1sZWZ0OjAuNzVleCI+Cjxk aXYgaWQ9M0QiZGl2dGFnZGVmYXVsdHdyYXBwZXIiIGRpcj0zRCJsdHIiIHN0eWxlPTNEImZvbnQt c2l6ZToxMnB0OyBjb2xvcj0KOiMwMDAwMDA7IGZvbnQtZmFtaWx5OkNhbGlicmksSGVsdmV0aWNh LHNhbnMtc2VyaWYiPgo8cD5IaSBEZXZpbiw8L3A+CjxwPjxicj4KPC9wPgo8cD5QbGVhc2UgY29u c2lkZXIgdGhhdCBmb3IgdGhlIE9TIEkgaGF2ZSBhIFJBSUQgMS4gTm93LCBsZXRzIHNheSBJIHVz ZSBSQUk9CkQgNSB0byBhc3NlbWJsZSBhIHNpbmdsZSBkaXNrIG9uIGVhY2ggc2VydmVyLiBJbiB0 aGlzIGNhc2UsIHRoZSBTU0Qgd2lsbCBubz0KdCBtYWtlIGFueSBkaWZmZXJlbmNlLCByaWdodD8g SSBndWVzcyB0aGF0IHRvIGJlIHBvc3NpYmxlIHRvIHVzZSBpdCwgdGhlIFNTPQpEIHNob3VsZCBu b3QgYmUgcGFydCBvZiB0aGUgUkFJRAogNS4gSW4gdGhpcyBjYXNlIEkgY291bGQgY3JlYXRlIGEg bG9naWNhbCB2b2x1bWUgbWFkZSBvZiB0aGUgUkFJRGVkIGJyaWNrIGE9Cm5kIHRoZW4gZXh0ZW5k IGl0IHVzaW5nIHRoZSBTU0QuIEkuZS46IFVzaW5nIGdkZXBsb3k6PC9wPgo8cD48YnI+CjwvcD4K PHA+W2Rpc2t0eXBlXTwvcD4KPHA+amJvZDwvcD4KPHA+Li4uLjwvcD4KPHA+W3B2MV08L3A+Cjxw PmFjdGlvbj0zRGNyZWF0ZTwvcD4KPHA+ZGV2aWNlcz0zRHNkYiwgc2RjPC9wPgo8cD53aXBlZnM9 M0R5ZXM8L3A+CjxwPjwvcD4KPHA+aWdub3JlX3ZnX2Vycm9zPTNEbm88YnI+CjwvcD4KPHA+PGJy Pgo8L3A+CjxwPjwvcD4KPHA+W3ZnMV08L3A+CjxwPjxiPmFjdGlvbj0zRGNyZWF0ZTwvYj48L3A+ CjxwPnZnbmFtZT0zRGdsdXN0ZXJfdmdfamJvZDwvcD4KPHA+cHZuYW1lPTNEc2RiPC9wPgo8cD5p Z25vcmVfdmdfZXJyb3M9M0Rubzxicj4KPC9wPgo8cD48YnI+CjwvcD4KPHA+W3ZnMl08L3A+Cjxw PjxiPmFjdGlvbj0zRGV4dGVuZDwvYj48L3A+CjxwPnZnbmFtZT0zRGdsdXN0ZXJfdmdfamJvZDwv cD4KPHA+cHZuYW1lPTNEc2RjPC9wPgo8cD5pZ25vcmVfdmdfZXJyb3M9M0Rubzxicj4KPC9wPgo8 YnI+CjxwPkJ1dCB3aWxsIEdsdXN0ZXIgYmUgYWJsZSB0byBhdXRvLWRldGVjdCBhbmQgdXNlIHRo aXMgU1NEIGJyaWNrIGZvciB0aWVyaT0Kbmc/IERvIEkgaGF2ZSB0byBkbyBzb21lIG90aGVyIGNv bmZpZ3VyYXRpb25zPyBBbHNvLCBhcyB0aGUgVk0gZmlsZXMgKC5xY293PQoyKSBhcmUgcXVpdGUg YmlnIHdpbGwgSSBiZW5lZml0IGZyb20gdGllcmluZz8gVGhpcyBpcyB3cm9uZyBhbmQgbXkgYXBw cm9hY2g9CiBzaG91bGQgYmUgb3RoZXI/PC9wPgo8cD48YnI+CjwvcD4KPHA+VGhhbmtzLDwvcD4K PHA+TW9hY2lyPGJyPgo8L3A+CjxwPjxicj4KPC9wPgo8YnI+CjxkaXYgc3R5bGU9M0QiY29sb3I6 cmdiKDQ5LDU1LDU3KSI+CjxociB0YWJpbmRleD0zRCItMSIgc3R5bGU9M0QiZGlzcGxheTppbmxp bmUtYmxvY2s7IHdpZHRoOjk4JSI+CjxkaXYgaWQ9M0QiZGl2UnBseUZ3ZE1zZyIgZGlyPTNEImx0 ciI+PGZvbnQgc3R5bGU9M0QiZm9udC1zaXplOjExcHQiIGZhY2U9Cj0zRCJDYWxpYnJpLCBzYW5z LXNlcmlmIiBjb2xvcj0zRCIjMDAwMDAwIj48Yj5Gcm9tOjwvYj4gRGV2aW4gQWNvc3RhICZsdDtk ZT0KdmluQHBhYnN0YXRlbmNpby5jb20mZ3Q7PGJyPgo8Yj5TZW50OjwvYj4gTW9uZGF5LCBBdWd1 c3QgNywgMjAxNyA3OjQ2IEFNPGJyPgo8Yj5Ubzo8L2I+IE1vYWNpciBGZXJyZWlyYTsgdXNlcnNA b3ZpcnQub3JnPGJyPgo8Yj5TdWJqZWN0OjwvYj4gUmU6IFtvdmlydC11c2Vyc10gR29vZCBwcmFj dGljZXM8L2ZvbnQ+CjxkaXY+Jm5ic3A7PC9kaXY+CjwvZGl2Pgo8ZGl2Pgo8ZGl2IGlkPTNEImJs b29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEImNvbG9yOnJnYigwLDAsMCk7IG1hcmdpbjowcHgiPjxm b250IGY9CmFjZT0zRCJJbnB1dCBNb25vIj48YnI+CjwvZm9udD48L2Rpdj4KPGRpdiBpZD0zRCJi bG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJjb2xvcjpyZ2IoMCwwLDApOyBtYXJnaW46MHB4Ij48 Zm9udCBmPQphY2U9M0QiSW5wdXQgTW9ubyI+TW9hY2lyLDwvZm9udD48L2Rpdj4KPGRpdiBpZD0z RCJibG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJjb2xvcjpyZ2IoMCwwLDApOyBtYXJnaW46MHB4 Ij48Zm9udCBmPQphY2U9M0QiSW5wdXQgTW9ubyI+PGJyPgo8L2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9 M0QiYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9M0QiY29sb3I6cmdiKDAsMCwwKTsgbWFyZ2luOjBw eCI+PGZvbnQgZj0KYWNlPTNEIklucHV0IE1vbm8iPkkgaGF2ZSByZWNlbnRseSBpbnN0YWxsZWQg bXVsdGlwbGUgUmVkIEhhdCBWaXJ0dWFsaXphdGlvPQpuIGhvc3RzIGZvciBzZXZlcmFsIGRpZmZl cmVudCBjb21wYW5pZXMsIGFuZCBoYXZlIGRlYWx0IHdpdGggdGhlIFJlZCBIYXQgU3U9CnBwb3J0 IFRlYW0gaW4gZGVwdGggYWJvdXQgb3B0aW1hbCBjb25maWd1cmF0aW9uCiBpbiByZWdhcmRzIHRv IHNldHRpbmcgdXAgR2x1c3RlckZTIG1vc3QgZWZmaWNpZW50bHkgYW5kIEkgd2FudGVkIHRvIHNo YXJlID0Kd2l0aCB5b3Ugd2hhdCBJIGxlYXJuZWQuPC9mb250PjwvZGl2Pgo8ZGl2IGlkPTNEImJs b29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEImNvbG9yOnJnYigwLDAsMCk7IG1hcmdpbjowcHgiPjxm b250IGY9CmFjZT0zRCJJbnB1dCBNb25vIj48YnI+CjwvZm9udD48L2Rpdj4KPGRpdiBpZD0zRCJi bG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJjb2xvcjpyZ2IoMCwwLDApOyBtYXJnaW46MHB4Ij48 Zm9udCBmPQphY2U9M0QiSW5wdXQgTW9ubyI+SW4gZ2VuZXJhbCBSZWQgSGF0IFZpcnR1YWxpemF0 aW9uIHRlYW0gZnJvd25zIHVwb24gdXNpbmc9CiBlYWNoIERJU0sgb2YgdGhlIHN5c3RlbSBhcyBq dXN0IGEgSkJPRCwgc3VyZSB0aGVyZSBpcyBzb21lIHByb3RlY3Rpb24gYnkgaD0KYXZpbmcgdGhl IGRhdGEgcmVwbGljYXRlZCwgaG93ZXZlciwgdGhlCiByZWNvbW1lbmRhdGlvbiBpcyB0byB1c2Ug UkFJRCA2IChwcmVmZXJyZWQpIG9yIFJBSUQtNSwgb3IgYXQgbGVhc3QgUkFJRC0xID0KYXQgdGhl IHZlcnkgbGVhc3QuPC9mb250PjwvZGl2Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0 eWxlPTNEImNvbG9yOnJnYigwLDAsMCk7IG1hcmdpbjowcHgiPjxmb250IGY9CmFjZT0zRCJJbnB1 dCBNb25vIj48YnI+CjwvZm9udD48L2Rpdj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21mb250IiBz dHlsZT0zRCJtYXJnaW46MHB4Ij48Zm9udCBmYWNlPTNEIklucHV0IE1vbm8iPQo+SGVyZSBpcyB0 aGUmbmJzcDtkaXJlY3QgcXVvdGUgZnJvbSBSZWQgSGF0IHdoZW4gSSBhc2tlZCBhYm91dCBSQUlE IGFuZCBCcmk9CmNrczo8L2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIg c3R5bGU9M0QibWFyZ2luOjBweCI+PGZvbnQgZmFjZT0zRCJJbnB1dCBNb25vIj0KPjxpPjxicj4K PC9pPjwvZm9udD48L2Rpdj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJt YXJnaW46MHB4Ij48Zm9udCBmYWNlPTNEIklucHV0IE1vbm8iPQo+PGk+JnF1b3Q7QSB0eXBpY2Fs IEdsdXN0ZXIgY29uZmlndXJhdGlvbiB3b3VsZCB1c2UgUkFJRCB1bmRlcm5lYXRoIHRoZSBicmk9 CmNrcy4gUkFJRCA2IGlzIG1vc3QgdHlwaWNhbCBhcyBpdCBnaXZlcyB5b3UgMiBkaXNrIGZhaWx1 cmUgcHJvdGVjdGlvbiwgYnV0ID0KUkFJRCA1IGNvdWxkIGJlIHVzZWQgdG9vLiZuYnNwO09uY2Ug eW91IGhhdmUgdGhlCiBSQUlEZWQgYnJpY2tzLCB5b3UnZCB0aGVuIGFwcGx5IHRoZSBkZXNpcmVk IHJlcGxpY2F0aW9uIG9uIHRvcCBvZiB0aGF0LiBUaD0KZSBtb3N0IHBvcHVsYXIgd2F5IG9mIGRv aW5nIHRoaXMgd291bGQgYmUgZGlzdHJpYnV0ZWQgcmVwbGljYXRlZCB3aXRoIDJ4IHJlPQpwbGlj YXRpb24uJm5ic3A7SW4gZ2VuZXJhbCB5b3UnbGwgZ2V0IGJldHRlciBwZXJmb3JtYW5jZSB3aXRo IGxhcmdlciBicmlja3M9Ci4gMTIgZHJpdmVzIGlzIG9mdGVuIGEgc3dlZXQgc3BvdC4gQW5vdGhl cgogb3B0aW9uIHdvdWxkIGJlIHRvIGNyZWF0ZSBhIHNlcGFyYXRlIHRpZXIgdXNpbmcgYWxsIFNT RD05MnMuPTk0Jm5ic3A7PC9pPjw9Ci9mb250PjwvZGl2Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3Rv bWZvbnQiIHN0eWxlPTNEIm1hcmdpbjowcHgiPjxicj4KPC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3Bf Y3VzdG9tZm9udCIgc3R5bGU9M0QibWFyZ2luOjBweCI+PGZvbnQgZmFjZT0zRCJJbnB1dCBNb25v Ij0KPjxpPkluIG9yZGVyIHRvIFNTRCZuYnNwO3RpZXJpbmcgZnJvbSBteSB1bmRlcnN0YW5kaW5n IHlvdSB3b3VsZCBuZWVkIDEgeCBOPQpWTWUgZHJpdmUgaW4gZWFjaCBzZXJ2ZXIsIG9yIDQgeCBT U0QgaG90IHRpZXIgKGl0IG5lZWRzIHRvIGJlIGRpc3RyaWJ1dGVkLCA9CnJlcGxpY2F0ZWQgZm9y IHRoZSBob3QgdGllciBpZiBub3QgdXNpbmcgTlZNRSkuCiBTbyB3aXRoIHlvdSBvbmx5IGhhdmlu ZyAxIFNTRCBkcml2ZSBpbiBlYWNoIHNlcnZlciwmbmJzcDtJPTkyZCBzdWdnZXN0IG1heT0KYmUg bG9va2luZyBpbnRvIHRoZSBOVk1FIG9wdGlvbi4mbmJzcDs8L2k+PC9mb250PjwvZGl2Pgo8ZGl2 IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEIm1hcmdpbjowcHgiPjxmb250IGZhY2U9 M0QiSW5wdXQgTW9ubyI9Cj48aT48YnI+CjwvaT48L2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxv b3BfY3VzdG9tZm9udCIgc3R5bGU9M0QibWFyZ2luOjBweCI+PGZvbnQgZmFjZT0zRCJJbnB1dCBN b25vIj0KPjxpPlNpbmNlIHlvdXIgdXNpbmcgb25seSAzLXNlcnZlcnMsIHdoYXQgST05MmQgcHJv YmFibHkgc3VnZ2VzdCBpcyB0byBkbyAoPQoyIFJlcGxpY2FzICYjNDM7IEFyYml0ZXIgTm9kZSks IHRoaXMgc2V0dXAgYWN0dWFsbHkgZG9lc249OTJ0IHJlcXVpcmUgdGhlIDM9CnJkIHNlcnZlciB0 byBoYXZlIGJpZyBkcml2ZXMgYXQgYWxsIGFzIGl0IG9ubHkKIHN0b3JlcyBtZXRhLWRhdGEgYWJv dXQgdGhlIGZpbGVzIGFuZCBub3QgYWN0dWFsbHkgYSBmdWxsIGNvcHkuJm5ic3A7PC9pPjwvPQpm b250PjwvZGl2Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEIm1hcmdpbjow cHgiPjxmb250IGZhY2U9M0QiSW5wdXQgTW9ubyI9Cj48aT48YnI+CjwvaT48L2ZvbnQ+PC9kaXY+ CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9M0QibWFyZ2luOjBweCI+PGZvbnQg ZmFjZT0zRCJJbnB1dCBNb25vIj0KPjxpPlBsZWFzZSBzZWUgdGhlIGF0dGFjaGVkIGRvY3VtZW50 IHRoYXQgd2FzIGdpdmVuIHRvIG1lIGJ5IFJlZCBIYXQgdG8gZ2V0PQogbW9yZSBpbmZvcm1hdGlv biBvbiB0aGlzLiBIb3BlIHRoaXMgaW5mb3JtYXRpb24gaGVscHMgeW91LjwvaT48L2ZvbnQ+PC9k aXY9Cj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJtYXJnaW46MHB4Ij48 Zm9udCBmYWNlPTNEIklucHV0IE1vbm8iPQo+PGk+PGJyPgo8L2k+PC9mb250PjwvZGl2Pgo8YnI+ CjxkaXYgaWQ9M0QiYmxvb3Bfc2lnbl8xNTAyMDg3Mzc2NzI1NDY5MTg0IiBjbGFzcz0zRCJibG9v cF9zaWduIj48c3BhbiBzdHlsZT0KPTNEImZvbnQtZmFtaWx5OidoZWx2ZXRpY2EgTmV1ZScsaGVs dmV0aWNhOyBmb250LXNpemU6MTRweCI+LS08L3NwYW4+PGJyIHN0PQp5bGU9M0QiZm9udC1mYW1p bHk6J2hlbHZldGljYSBOZXVlJyxoZWx2ZXRpY2E7IGZvbnQtc2l6ZToxNHB4Ij4KPGRpdiBjbGFz cz0zRCJnbWFpbF9zaWduYXR1cmUiIHN0eWxlPTNEImZvbnQtZmFtaWx5OidoZWx2ZXRpY2EgTmV1 ZScsaGVsdmV0PQppY2E7IGZvbnQtc2l6ZToxNHB4Ij4KPGRpdiBkaXI9M0QibHRyIj4KPGRpdj48 YnI+CjwvZGl2Pgo8ZGl2PkRldmluIEFjb3N0YSwgUkhDQSwgUkhWQ0E8L2Rpdj4KPGRpdj5SZWQg SGF0IENlcnRpZmllZCBBcmNoaXRlY3Q8L2Rpdj4KPGRpdj48L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8 L2Rpdj4KPGJyPgo8cCBjbGFzcz0zRCJhaXJtYWlsX29uIj5PbiBBdWd1c3QgNiwgMjAxNyBhdCA3 OjI5OjI5IFBNLCBNb2FjaXIgRmVycmVpcmEgKDw9CmEgaHJlZj0zRCJtYWlsdG86bW9hY2lyZmVy cmVpcmFAaG90bWFpbC5jb20iPm1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY29tPC9hPj0KKSB3cm90 ZTo8L3A+CjxibG9ja3F1b3RlIHR5cGU9M0QiY2l0ZSIgY2xhc3M9M0QiY2xlYW5fYnEiPjxzcGFu Pgo8ZGl2IGRpcj0zRCJsdHIiPgo8ZGl2PjwvZGl2Pgo8ZGl2Pgo8ZGl2IGlkPTNEImRpdnRhZ2Rl ZmF1bHR3cmFwcGVyIiBkaXI9M0QibHRyIiBzdHlsZT0zRCJmb250LXNpemU6MTJwdDsgY29sb3I9 CjojMDAwMDAwOyBmb250LWZhbWlseTpDYWxpYnJpLEhlbHZldGljYSxzYW5zLXNlcmlmIj4KPHA+ PHNwYW4+SSBhbSB3aWxsaW5nIHRvIGFzc2VtYmxlIGEgb1ZpcnQgJnF1b3Q7cG9kJnF1b3Q7LCBt YWRlIG9mIDMgc2VydmVyPQpzLCBlYWNoIHdpdGggMiBDUFUgc29ja2V0cyBvZiAxMiBjb3Jlcywg MjU2R0IgUkFNLCA3IEhERCAxMEssIDEgU1NELiBUaGUgaWQ9CmVhIGlzIHRvIHVzZSBHbHVzdGVy RlMgdG8gcHJvdmlkZSBIQSBmb3IgdGhlIFZNcy4gVGhlIDMgc2VydmVycyBoYXZlIGEgZHVhbD0K IDQwR2IgTklDIGFuZCBhIGR1YWwgMTBHYiBOSUMuIFNvIG15IGludGVudGlvbgogaXMgdG8gY3Jl YXRlIGEgbG9vcCBsaWtlIGEgc2VydmVyIHRyaWFuZ2xlIHVzaW5nIHRoZSA0MEdiIE5JQ3MgZm9y IHZpcnR1YWw9Cml6YXRpb24gZmlsZXMgKFZNcyAucWNvdzIpIGFjY2VzcyBhbmQgdG8gbW92ZSBW TXMgYXJvdW5kIHRoZSBwb2QgKGVhc3QgL3dlcz0KdCB0cmFmZmljKSB3aGlsZSB1c2luZyB0aGUg MTBHYiBpbnRlcmZhY2VzIGZvciBnaXZpbmcgc2VydmljZXMgdG8gdGhlIG91dHNpPQpkZSB3b3Js ZCAobm9ydGgvc291dGggdHJhZmZpYykuPC9zcGFuPjwvcD4KPHA+PGJyPgo8L3A+CjxwPlRoaXMg c2FpZCwgbXkgZmlyc3QgcXVlc3Rpb24gaXM6IEhvdyBzaG91bGQgSSBkZXBsb3kgR2x1c3RlckZT IGluIHN1Y2ggbz0KVmlydCBzY2VuYXJpbz8gTXkgcXVlc3Rpb25zIGFyZTo8L3A+CjxwPjxicj4K PC9wPgo8cD4xIC0gU2hvdWxkIEkgY3JlYXRlIDMgUkFJRCAoaS5lLjogUkFJRCA1KSwgb25lIG9u IGVhY2ggb1ZpcnQgbm9kZSwgYW5kIHQ9CmhlbiBjcmVhdGUgYSBHbHVzdGVyRlMgdXNpbmcgdGhl bT88L3A+CjxwPjIgLSBJbnN0ZWFkLCBzaG91bGQgSSBjcmVhdGUgYSBKQk9EIGFycmF5IG1hZGUg b2YgYWxsIHNlcnZlcidzIGRpc2tzPzwvcD0KPgo8cD4zIC0gV2hhdCBpcyB0aGUgYmVzdCBHbHVz dGVyIGNvbmZpZ3VyYXRpb24gdG8gcHJvdmlkZSBmb3IgSEEgd2hpbGUgbm90IGM9Cm9uc3VtaW5n IHRvbyBtdWNoIGRpc2sgc3BhY2U/PGJyPgo8L3A+CjxwPjQgLSBEb2VzIGEgb1ZpcnQgaHlwZXJ2 aXNvciBwb2QgbGlrZSBJIGFtIHBsYW5uaW5nIHRvIGJ1aWxkLCBhbmQgdGhlIHZpcj0KdHVhbGl6 YXRpb24gZW52aXJvbm1lbnQsIGJlbmVmaXRzIGZyb20gdGllcmluZyB3aGVuIHVzaW5nIGEgU1NE IGRpc2s/IEFuZCB5PQplcywgd2lsbCBHbHVzdGVyIGRvIGl0IGJ5IGRlZmF1bHQgb3IgSSBoYXZl IHRvIGNvbmZpZ3VyZSBpdCB0byBkbyBzbz88L3A+CjxwPjxicj4KPC9wPgo8cD5BdCB0aGUgYm90 dG9tIGxpbmUsIHdoYXQgaXMgdGhlIGdvb2QgcHJhY3RpY2UgZm9yIHVzaW5nIEdsdXN0ZXJGUyBp biBzbWE9CmxsIHBvZHMgZm9yIGVudGVycHJpc2VzPzxicj4KPC9wPgo8cD48YnI+CjwvcD4KPHA+ WW91IG9waW5pb24vZmVlZGJhY2sgd2lsbCBiZSByZWFsbHkgYXBwcmVjaWF0ZWQhPC9wPgo8cD5N b2FjaXI8YnI+CjwvcD4KPC9kaXY+Cl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fIDxicj4KVXNlcnMgbWFpbGluZyBsaXN0IDxicj4KPGEgaHJlZj0zRCJtYWls dG86VXNlcnNAb3ZpcnQub3JnIj5Vc2Vyc0BvdmlydC5vcmc8L2E+IDxicj4KPGEgaHJlZj0zRCJo dHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiPmh0dHA6Ly9saXN0 cy5vdmlyPQp0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPgo8YnI+CjwvZGl2Pgo8L2Rp dj4KPC9zcGFuPjwvYmxvY2txdW90ZT4KPC9kaXY+CjwvZGl2Pgo8L2Rpdj4KX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188YnI+ClVzZXJzIG1haWxpbmcgbGlz dDxicj4KPGEgY2xhc3M9M0QiYXFtLWF1dG9saW5rIGFxbS1hdXRvd3JhcCIgaHJlZj0zRCJtYWls dG86VXNlcnMlNDBvdmlydC5vcmciPlVzPQplcnNAb3ZpcnQub3JnPC9hPjxicj4KPGEgY2xhc3M9 M0QiYXFtLWF1dG9saW5rIGFxbS1hdXRvd3JhcCIgaHJlZj0zRCJodHRwOi8vbGlzdHMub3ZpcnQu b3JnL21haWxtPQphbi9saXN0aW5mby91c2VycyI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWls bWFuL2xpc3RpbmZvL3VzZXJzPC9hPjxicj4KPGJyPgo8L2Jsb2NrcXVvdGU+CjwvZGl2Pgo8L2Rp dj4KPC9kaXY+CjwvZGl2Pgo8L2Rpdj4KPC9ibG9ja3F1b3RlPgo8L2Rpdj4KPC9kaXY+CjwvZGl2 Pgo8L2Rpdj4KPC9kaXY+CjwvYm9keT4KPC9odG1sPgoKLS1fMDAwX0RCNlAxOTBNQjAyODBGMDY2 NzYyRUY4MURFNDI3MDI4MEM4OEEwREI2UDE5ME1CMDI4MEVVUlBfLS0K --===============4213557402878432470==-- From Pax at acronis.com Tue Aug 8 14:44:24 2017 Content-Type: multipart/mixed; boundary="===============8333647322475142532==" MIME-Version: 1.0 From: Pavel Gashev To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Tue, 08 Aug 2017 14:44:20 +0000 Message-ID: <5C006283-1345-4AB4-A2D9-23DB9438B67C@acronis.com> In-Reply-To: 02172d33-e14e-aa8d-98f5-d920573e281c@upx.com --===============8333647322475142532== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Fernando, I agree that RAID is not required here by common sense. The only point to s= etup RAID is a lack of manageability of GlusterFS. So you just buy manageab= ility for extra hardware cost and write performance in some scenarios. That= is it. On 08/08/2017, 16:24, "users-bounces(a)ovirt.org on behalf of FERNANDO FRED= IANI" = wrote: That's something on the way RAID works, regardless what most = 'super-ultra' powerfull hardware controller you may have. RAID 5 or 6 = will never have the same write performance as a RAID 10 o 0 for example= . = Writeback caches can deal with bursts well but they have a limit = therefore there will always be a penalty compared to what else you coul= d = have. = If you have a continuous stream of data (a big VM deployment or a large = data copy) there will be a continuous write and that will likely fill u= p = the cache making the disks underneath the bottleneck. That's why on some other scenarios, like ZFS people have multiple group= s = of RAID 6 (called RAIDZ2) so it improves the write speeds for these typ= e = of scenarios. = In the scenario given in this thread with just 3 servers, each with a = RAID 6 there will be a bare limit on the write performance specially fo= r = streammed data for most powerfull your hardware controller can do = write-back. = Also I agree the 40Gb NICs may not be used fully and 10Gb can do the jo= b = well, but if they were available at the begining, why not use them. = Fernando = = On 08/08/2017 03:16, Fabrice Bacchella wrote: >> Le 8 ao=C3=BBt 2017 =C3=A0 04:08, FERNANDO FREDIANI a =C3=A9crit : >> Even if you have a Hardware RAID Controller with Writeback cache you= will have a significant performance penalty and may not fully use all the = resources you mentioned you have. >> > Nope again,from my experience with HP Smart Array and write back cach= e, write, that goes in the cache, are even faster that read that must goes = to the disks. of course if the write are too fast and to big, they will ove= r overflow the cache. But on todays controller they are multi-gigabyte cach= e, you must write a lot to fill them. And if you can afford 40Gb card, you = can afford decent controller. > > > = _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users = --===============8333647322475142532==-- From moacirferreira at hotmail.com Wed Aug 9 01:52:33 2017 Content-Type: multipart/mixed; boundary="===============7391426873726200533==" MIME-Version: 1.0 From: Moacir Ferreira To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Tue, 08 Aug 2017 10:19:32 +0000 Message-ID: In-Reply-To: 15dc0713d28.2768.ffe2d6cdf294efc1bb86526523af9945@kafit.se --===============7391426873726200533== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --_000_DB6P190MB0280BD0BB439D0C8A4588D00C88A0DB6P190MB0280EURP_ Content-Type: text/plain; charset=3D"Windows-1252" Content-Transfer-Encoding: quoted-printable Thanks Johan, you brought "light" into my darkness! I went looking for the = =3D GlusterFS tiering how-to and it looks like quite simple to attach a SSD as = =3D hot tier. For those willing to read about it, go here: http://blog.gluster.= =3D org/2016/03/automated-tiering-in-gluster/ Now, I still have a question: VMs are made of very large .qcow2 files. My u= =3D nderstanding is that files in Gluster are kept all together in a single bri= =3D ck. If so, I will not benefit from tiering as a single SSD will not be big = =3D enough to fit all my large VM .qcow2 files. This would not be true if Glust= =3D er can store "blocks" of data that compose a large file spread on several b= =3D ricks. But if I am not wrong, this is one of key differences in between Glu= =3D sterFS and Ceph. Can you comment? Moacir ________________________________ From: Johan Bernhardsson Sent: Tuesday, August 8, 2017 7:03 AM To: Moacir Ferreira; Devin Acosta; users(a)ovirt.org Subject: Re: [ovirt-users] Good practices You attach the ssd as a hot tier with a gluster command. I don't think that= =3D gdeploy or ovirt gui can do it. The gluster docs and redhat docs explains tiering quite good. /Johan On August 8, 2017 07:06:42 Moacir Ferreira w= ro=3D te: Hi Devin, Please consider that for the OS I have a RAID 1. Now, lets say I use RAID 5= =3D to assemble a single disk on each server. In this case, the SSD will not m= =3D ake any difference, right? I guess that to be possible to use it, the SSD s= =3D hould not be part of the RAID 5. In this case I could create a logical volu= =3D me made of the RAIDed brick and then extend it using the SSD. I.e.: Using g= =3D deploy: [disktype] jbod .... [pv1] action=3D3Dcreate devices=3D3Dsdb, sdc wipefs=3D3Dyes ignore_vg_erros=3D3Dno [vg1] action=3D3Dcreate vgname=3D3Dgluster_vg_jbod pvname=3D3Dsdb ignore_vg_erros=3D3Dno [vg2] action=3D3Dextend vgname=3D3Dgluster_vg_jbod pvname=3D3Dsdc ignore_vg_erros=3D3Dno But will Gluster be able to auto-detect and use this SSD brick for tiering?= =3D Do I have to do some other configurations? Also, as the VM files (.qcow2) = =3D are quite big will I benefit from tiering? This is wrong and my approach sh= =3D ould be other? Thanks, Moacir ________________________________ From: Devin Acosta Sent: Monday, August 7, 2017 7:46 AM To: Moacir Ferreira; users(a)ovirt.org Subject: Re: [ovirt-users] Good practices Moacir, I have recently installed multiple Red Hat Virtualization hosts for several= =3D different companies, and have dealt with the Red Hat Support Team in depth= =3D about optimal configuration in regards to setting up GlusterFS most effici= =3D ently and I wanted to share with you what I learned. In general Red Hat Virtualization team frowns upon using each DISK of the s= =3D ystem as just a JBOD, sure there is some protection by having the data repl= =3D icated, however, the recommendation is to use RAID 6 (preferred) or RAID-5,= =3D or at least RAID-1 at the very least. Here is the direct quote from Red Hat when I asked about RAID and Bricks: "A typical Gluster configuration would use RAID underneath the bricks. RAID= =3D 6 is most typical as it gives you 2 disk failure protection, but RAID 5 co= =3D uld be used too. Once you have the RAIDed bricks, you'd then apply the desi= =3D red replication on top of that. The most popular way of doing this would be= =3D distributed replicated with 2x replication. In general you'll get better p= =3D erformance with larger bricks. 12 drives is often a sweet spot. Another opt= =3D ion would be to create a separate tier using all SSD=3D92s.=3D94 In order to SSD tiering from my understanding you would need 1 x NVMe drive= =3D in each server, or 4 x SSD hot tier (it needs to be distributed, replicate= =3D d for the hot tier if not using NVME). So with you only having 1 SSD drive = =3D in each server, I=3D92d suggest maybe looking into the NVME option. Since your using only 3-servers, what I=3D92d probably suggest is to do (2 = Re=3D plicas + Arbiter Node), this setup actually doesn=3D92t require the 3rd ser= ve=3D r to have big drives at all as it only stores meta-data about the files and= =3D not actually a full copy. Please see the attached document that was given to me by Red Hat to get mor= =3D e information on this. Hope this information helps you. -- Devin Acosta, RHCA, RHVCA Red Hat Certified Architect On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (moacirferreira(a)hotmail.= co=3D m) wrote: I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU = =3D sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use Gluste= =3D rFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dua= =3D l 10Gb NIC. So my intention is to create a loop like a server triangle usin= =3D g the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VM= =3D s around the pod (east /west traffic) while using the 10Gb interfaces for g= =3D iving services to the outside world (north/south traffic). This said, my first question is: How should I deploy GlusterFS in such oVir= =3D t scenario? My questions are: 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then= =3D create a GlusterFS using them? 2 - Instead, should I create a JBOD array made of all server's disks? 3 - What is the best Gluster configuration to provide for HA while not cons= =3D uming too much disk space? 4 - Does a oVirt hypervisor pod like I am planning to build, and the virtua= =3D lization environment, benefits from tiering when using a SSD disk? And yes,= =3D will Gluster do it by default or I have to configure it to do so? At the bottom line, what is the good practice for using GlusterFS in small = =3D pods for enterprises? You opinion/feedback will be really appreciated! Moacir _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users --_000_DB6P190MB0280BD0BB439D0C8A4588D00C88A0DB6P190MB0280EURP_ Content-Type: text/html; charset=3D"Windows-1252" Content-Transfer-Encoding: quoted-printable
Thanks Johan, you brought "light" into my darkness! I went loo= =3D king for the GlusterFS tiering how-to and it looks like quite simple to att= =3D ach a SSD as hot tier. For those willing to read about it, go here: http://blog.gluster.org/2016/03/automated-tiering-in-gluster/

Now, I still have a question: VMs are made of very large .qcow2 files. M= =3D y understanding is that files in Gluster are kept all together in a single = =3D brick. If so, I will not benefit from tiering as a single SSD will not be b= =3D ig enough to fit all my large VM .qcow2 files. This would not be true if Gluster can store "blocks&quo= =3D t; of data that compose a large file spread on several bricks. But if I am = =3D not wrong, this is one of key differences in between GlusterFS and Ceph. Ca= =3D n you comment?

Moacir

From: Johan Bernhards= son =3D <johan(a)kafit.se>
Sent: Tuesday, August 8, 2017 7:03 AM
To: Moacir Ferreira; Devin Acosta; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices

You attach the ssd as a hot ti= er=3D with a gluster command. I don't think that gdeploy or ovirt gui can do it.= =3D

The gluster docs and redhat do= cs=3D explains tiering quite good.

/Johan

On August 8, 2017 07:06:42 Moacir Ferreira <moacirferreira(a)hotmail.com= &g=3D t; wrote:

Hi Devin,

Please consider that for the OS I have a RAID 1. Now, lets say I use RAI= =3D D 5 to assemble a single disk on each server. In this case, the SSD will no= =3D t make any difference, right? I guess that to be possible to use it, the SS= =3D D should not be part of the RAID 5. In this case I could create a logical volume made of the RAIDed brick a= =3D nd then extend it using the SSD. I.e.: Using gdeploy:

[disktype]

jbod

....

[pv1]

action=3D3Dcreate

devices=3D3Dsdb, sdc

wipefs=3D3Dyes

ignore_vg_erros=3D3Dno

[vg1]

action=3D3Dcreate

vgname=3D3Dgluster_vg_jbod

pvname=3D3Dsdb

ignore_vg_erros=3D3Dno

[vg2]

action=3D3Dextend

vgname=3D3Dgluster_vg_jbod

pvname=3D3Dsdc

ignore_vg_erros=3D3Dno

But will Gluster be able to auto-detect and use this SSD brick for tieri= =3D ng? Do I have to do some other configurations? Also, as the VM files (.qcow= =3D 2) are quite big will I benefit from tiering? This is wrong and my approach= =3D should be other?

Thanks,

Moacir

From: Devin Acosta &l= t;de=3D vin(a)pabstatencio.com>
Sent: Monday, August 7, 2017 7:46 AM
To: Moacir Ferreira; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices

Moacir,

I have recently installed multiple Red Hat Virtualizat= io=3D n hosts for several different companies, and have dealt with the Red Hat Su= =3D pport Team in depth about optimal configuration in regards to setting up GlusterFS most efficiently and I wanted to share = =3D with you what I learned.

In general Red Hat Virtualization team frowns upon usi= ng=3D each DISK of the system as just a JBOD, sure there is some protection by h= =3D aving the data replicated, however, the recommendation is to use RAID 6 (preferred) or RAID-5, or at least RAID-1 = =3D at the very least.

Here is the direct quote from Red Hat when I asked about RAID and Bri= =3D cks:

"A typical Gluster configuration would use RAID underneath the bri= =3D cks. RAID 6 is most typical as it gives you 2 disk failure protection, but = =3D RAID 5 could be used too. Once you have the RAIDed bricks, you'd then apply the desired replication on top of that. Th= =3D e most popular way of doing this would be distributed replicated with 2x re= =3D plication. In general you'll get better performance with larger bricks= =3D . 12 drives is often a sweet spot. Another option would be to create a separate tier using all SSD=3D92s.=3D94 <= /i><=3D /font>

In order to SSD tiering from my understanding you would need 1 x N= =3D VMe drive in each server, or 4 x SSD hot tier (it needs to be distributed, = =3D replicated for the hot tier if not using NVME). So with you only having 1 SSD drive in each server, I=3D92d suggest m= ay=3D be looking into the NVME option.

Since your using only 3-servers, what I=3D92d probably suggest is to do= (=3D 2 Replicas + Arbiter Node), this setup actually doesn=3D92t require the= 3=3D rd server to have big drives at all as it only stores meta-data about the files and not actually a full copy.

Please see the attached document that was given to me by Red Hat to get= =3D more information on this. Hope this information helps you.

--

Devin Acosta, RHCA, RHVCA

Red Hat Certified Architect

On August 6, 2017 at 7:29:29 PM, Moacir Ferreira = (<=3D a href=3D3D"mailto:moacirferreira(a)hotmail.com">moacirferreira(a)hotmail.c= om=3D ) wrote:

I am willing to assemble a oVirt "pod", made of 3 server= =3D s, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The id= =3D ea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual= =3D 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop like a server triangle using the 40Gb NICs for virtual= =3D ization files (VMs .qcow2) access and to move VMs around the pod (east /wes= =3D t traffic) while using the 10Gb interfaces for giving services to the outsi= =3D de world (north/south traffic).

This said, my first question is: How should I deploy GlusterFS in such o= =3D Virt scenario? My questions are:

1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and t= =3D hen create a GlusterFS using them?

2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not c= =3D onsuming too much disk space?

4 - Does a oVirt hypervisor pod like I am planning to build, and the vir= =3D tualization environment, benefits from tiering when using a SSD disk? And y= =3D es, will Gluster do it by default or I have to configure it to do so?

At the bottom line, what is the good practice for using GlusterFS in sma= =3D ll pods for enterprises?

You opinion/feedback will be really appreciated!

Moacir

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ov= ir=3D t.org/mailman/listinfo/users

_______________________________________________
Users mailing list
Us=3D ers(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

--_000_DB6P190MB0280BD0BB439D0C8A4588D00C88A0DB6P190MB0280EURP_-- --===============7391426873726200533== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" LS1fMDAwX0RCNlAxOTBNQjAyODBCRDBCQjQzOUQwQzhBNDU4OEQwMEM4OEEwREI2UDE5ME1CMDI4 MEVVUlBfCkNvbnRlbnQtVHlwZTogdGV4dC9wbGFpbjsgY2hhcnNldD0iV2luZG93cy0xMjUyIgpD b250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiBxdW90ZWQtcHJpbnRhYmxlCgpUaGFua3MgSm9oYW4s IHlvdSBicm91Z2h0ICJsaWdodCIgaW50byBteSBkYXJrbmVzcyEgSSB3ZW50IGxvb2tpbmcgZm9y IHRoZSA9CkdsdXN0ZXJGUyB0aWVyaW5nIGhvdy10byBhbmQgaXQgbG9va3MgbGlrZSBxdWl0ZSBz aW1wbGUgdG8gYXR0YWNoIGEgU1NEIGFzID0KaG90IHRpZXIuIEZvciB0aG9zZSB3aWxsaW5nIHRv IHJlYWQgYWJvdXQgaXQsIGdvIGhlcmU6IGh0dHA6Ly9ibG9nLmdsdXN0ZXIuPQpvcmcvMjAxNi8w My9hdXRvbWF0ZWQtdGllcmluZy1pbi1nbHVzdGVyLwoKCk5vdywgSSBzdGlsbCBoYXZlIGEgcXVl c3Rpb246IFZNcyBhcmUgbWFkZSBvZiB2ZXJ5IGxhcmdlIC5xY293MiBmaWxlcy4gTXkgdT0KbmRl cnN0YW5kaW5nIGlzIHRoYXQgZmlsZXMgaW4gR2x1c3RlciBhcmUga2VwdCBhbGwgdG9nZXRoZXIg aW4gYSBzaW5nbGUgYnJpPQpjay4gSWYgc28sIEkgd2lsbCBub3QgYmVuZWZpdCBmcm9tIHRpZXJp bmcgYXMgYSBzaW5nbGUgU1NEIHdpbGwgbm90IGJlIGJpZyA9CmVub3VnaCB0byBmaXQgYWxsIG15 IGxhcmdlIFZNIC5xY293MiBmaWxlcy4gVGhpcyB3b3VsZCBub3QgYmUgdHJ1ZSBpZiBHbHVzdD0K ZXIgY2FuIHN0b3JlICJibG9ja3MiIG9mIGRhdGEgdGhhdCBjb21wb3NlIGEgbGFyZ2UgZmlsZSBz cHJlYWQgb24gc2V2ZXJhbCBiPQpyaWNrcy4gQnV0IGlmIEkgYW0gbm90IHdyb25nLCB0aGlzIGlz IG9uZSBvZiBrZXkgZGlmZmVyZW5jZXMgaW4gYmV0d2VlbiBHbHU9CnN0ZXJGUyBhbmQgQ2VwaC4g Q2FuIHlvdSBjb21tZW50PwoKCk1vYWNpcgoKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fCkZyb206IEpvaGFuIEJlcm5oYXJkc3NvbiA8am9oYW5Aa2FmaXQuc2U+ClNlbnQ6IFR1ZXNk YXksIEF1Z3VzdCA4LCAyMDE3IDc6MDMgQU0KVG86IE1vYWNpciBGZXJyZWlyYTsgRGV2aW4gQWNv c3RhOyB1c2Vyc0BvdmlydC5vcmcKU3ViamVjdDogUmU6IFtvdmlydC11c2Vyc10gR29vZCBwcmFj dGljZXMKCgpZb3UgYXR0YWNoIHRoZSBzc2QgYXMgYSBob3QgdGllciB3aXRoIGEgZ2x1c3RlciBj b21tYW5kLiBJIGRvbid0IHRoaW5rIHRoYXQ9CiBnZGVwbG95IG9yIG92aXJ0IGd1aSBjYW4gZG8g aXQuCgpUaGUgZ2x1c3RlciBkb2NzIGFuZCByZWRoYXQgZG9jcyBleHBsYWlucyB0aWVyaW5nIHF1 aXRlIGdvb2QuCgovSm9oYW4KCk9uIEF1Z3VzdCA4LCAyMDE3IDA3OjA2OjQyIE1vYWNpciBGZXJy ZWlyYSA8bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb20+IHdybz0KdGU6CgpIaSBEZXZpbiwKCgpQ bGVhc2UgY29uc2lkZXIgdGhhdCBmb3IgdGhlIE9TIEkgaGF2ZSBhIFJBSUQgMS4gTm93LCBsZXRz IHNheSBJIHVzZSBSQUlEIDU9CiB0byBhc3NlbWJsZSBhIHNpbmdsZSBkaXNrIG9uIGVhY2ggc2Vy dmVyLiBJbiB0aGlzIGNhc2UsIHRoZSBTU0Qgd2lsbCBub3QgbT0KYWtlIGFueSBkaWZmZXJlbmNl LCByaWdodD8gSSBndWVzcyB0aGF0IHRvIGJlIHBvc3NpYmxlIHRvIHVzZSBpdCwgdGhlIFNTRCBz PQpob3VsZCBub3QgYmUgcGFydCBvZiB0aGUgUkFJRCA1LiBJbiB0aGlzIGNhc2UgSSBjb3VsZCBj cmVhdGUgYSBsb2dpY2FsIHZvbHU9Cm1lIG1hZGUgb2YgdGhlIFJBSURlZCBicmljayBhbmQgdGhl biBleHRlbmQgaXQgdXNpbmcgdGhlIFNTRC4gSS5lLjogVXNpbmcgZz0KZGVwbG95OgoKCltkaXNr dHlwZV0KCmpib2QKCi4uLi4KCltwdjFdCgphY3Rpb249M0RjcmVhdGUKCmRldmljZXM9M0RzZGIs IHNkYwoKd2lwZWZzPTNEeWVzCgppZ25vcmVfdmdfZXJyb3M9M0RubwoKClt2ZzFdCgphY3Rpb249 M0RjcmVhdGUKCnZnbmFtZT0zRGdsdXN0ZXJfdmdfamJvZAoKcHZuYW1lPTNEc2RiCgppZ25vcmVf dmdfZXJyb3M9M0RubwoKClt2ZzJdCgphY3Rpb249M0RleHRlbmQKCnZnbmFtZT0zRGdsdXN0ZXJf dmdfamJvZAoKcHZuYW1lPTNEc2RjCgppZ25vcmVfdmdfZXJyb3M9M0RubwoKCkJ1dCB3aWxsIEds dXN0ZXIgYmUgYWJsZSB0byBhdXRvLWRldGVjdCBhbmQgdXNlIHRoaXMgU1NEIGJyaWNrIGZvciB0 aWVyaW5nPz0KIERvIEkgaGF2ZSB0byBkbyBzb21lIG90aGVyIGNvbmZpZ3VyYXRpb25zPyBBbHNv LCBhcyB0aGUgVk0gZmlsZXMgKC5xY293MikgPQphcmUgcXVpdGUgYmlnIHdpbGwgSSBiZW5lZml0 IGZyb20gdGllcmluZz8gVGhpcyBpcyB3cm9uZyBhbmQgbXkgYXBwcm9hY2ggc2g9Cm91bGQgYmUg b3RoZXI/CgoKVGhhbmtzLAoKTW9hY2lyCgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X18KRnJvbTogRGV2aW4gQWNvc3RhIDxkZXZpbkBwYWJzdGF0ZW5jaW8uY29tPgpTZW50OiBNb25k YXksIEF1Z3VzdCA3LCAyMDE3IDc6NDYgQU0KVG86IE1vYWNpciBGZXJyZWlyYTsgdXNlcnNAb3Zp cnQub3JnClN1YmplY3Q6IFJlOiBbb3ZpcnQtdXNlcnNdIEdvb2QgcHJhY3RpY2VzCgoKTW9hY2ly LAoKSSBoYXZlIHJlY2VudGx5IGluc3RhbGxlZCBtdWx0aXBsZSBSZWQgSGF0IFZpcnR1YWxpemF0 aW9uIGhvc3RzIGZvciBzZXZlcmFsPQogZGlmZmVyZW50IGNvbXBhbmllcywgYW5kIGhhdmUgZGVh bHQgd2l0aCB0aGUgUmVkIEhhdCBTdXBwb3J0IFRlYW0gaW4gZGVwdGg9CiBhYm91dCBvcHRpbWFs IGNvbmZpZ3VyYXRpb24gaW4gcmVnYXJkcyB0byBzZXR0aW5nIHVwIEdsdXN0ZXJGUyBtb3N0IGVm ZmljaT0KZW50bHkgYW5kIEkgd2FudGVkIHRvIHNoYXJlIHdpdGggeW91IHdoYXQgSSBsZWFybmVk LgoKSW4gZ2VuZXJhbCBSZWQgSGF0IFZpcnR1YWxpemF0aW9uIHRlYW0gZnJvd25zIHVwb24gdXNp bmcgZWFjaCBESVNLIG9mIHRoZSBzPQp5c3RlbSBhcyBqdXN0IGEgSkJPRCwgc3VyZSB0aGVyZSBp cyBzb21lIHByb3RlY3Rpb24gYnkgaGF2aW5nIHRoZSBkYXRhIHJlcGw9CmljYXRlZCwgaG93ZXZl ciwgdGhlIHJlY29tbWVuZGF0aW9uIGlzIHRvIHVzZSBSQUlEIDYgKHByZWZlcnJlZCkgb3IgUkFJ RC01LD0KIG9yIGF0IGxlYXN0IFJBSUQtMSBhdCB0aGUgdmVyeSBsZWFzdC4KCkhlcmUgaXMgdGhl IGRpcmVjdCBxdW90ZSBmcm9tIFJlZCBIYXQgd2hlbiBJIGFza2VkIGFib3V0IFJBSUQgYW5kIEJy aWNrczoKCiJBIHR5cGljYWwgR2x1c3RlciBjb25maWd1cmF0aW9uIHdvdWxkIHVzZSBSQUlEIHVu ZGVybmVhdGggdGhlIGJyaWNrcy4gUkFJRD0KIDYgaXMgbW9zdCB0eXBpY2FsIGFzIGl0IGdpdmVz IHlvdSAyIGRpc2sgZmFpbHVyZSBwcm90ZWN0aW9uLCBidXQgUkFJRCA1IGNvPQp1bGQgYmUgdXNl ZCB0b28uIE9uY2UgeW91IGhhdmUgdGhlIFJBSURlZCBicmlja3MsIHlvdSdkIHRoZW4gYXBwbHkg dGhlIGRlc2k9CnJlZCByZXBsaWNhdGlvbiBvbiB0b3Agb2YgdGhhdC4gVGhlIG1vc3QgcG9wdWxh ciB3YXkgb2YgZG9pbmcgdGhpcyB3b3VsZCBiZT0KIGRpc3RyaWJ1dGVkIHJlcGxpY2F0ZWQgd2l0 aCAyeCByZXBsaWNhdGlvbi4gSW4gZ2VuZXJhbCB5b3UnbGwgZ2V0IGJldHRlciBwPQplcmZvcm1h bmNlIHdpdGggbGFyZ2VyIGJyaWNrcy4gMTIgZHJpdmVzIGlzIG9mdGVuIGEgc3dlZXQgc3BvdC4g QW5vdGhlciBvcHQ9CmlvbiB3b3VsZCBiZSB0byBjcmVhdGUgYSBzZXBhcmF0ZSB0aWVyIHVzaW5n IGFsbCBTU0Q9OTJzLj05NAoKSW4gb3JkZXIgdG8gU1NEIHRpZXJpbmcgZnJvbSBteSB1bmRlcnN0 YW5kaW5nIHlvdSB3b3VsZCBuZWVkIDEgeCBOVk1lIGRyaXZlPQogaW4gZWFjaCBzZXJ2ZXIsIG9y IDQgeCBTU0QgaG90IHRpZXIgKGl0IG5lZWRzIHRvIGJlIGRpc3RyaWJ1dGVkLCByZXBsaWNhdGU9 CmQgZm9yIHRoZSBob3QgdGllciBpZiBub3QgdXNpbmcgTlZNRSkuIFNvIHdpdGggeW91IG9ubHkg aGF2aW5nIDEgU1NEIGRyaXZlID0KaW4gZWFjaCBzZXJ2ZXIsIEk9OTJkIHN1Z2dlc3QgbWF5YmUg bG9va2luZyBpbnRvIHRoZSBOVk1FIG9wdGlvbi4KClNpbmNlIHlvdXIgdXNpbmcgb25seSAzLXNl cnZlcnMsIHdoYXQgST05MmQgcHJvYmFibHkgc3VnZ2VzdCBpcyB0byBkbyAoMiBSZT0KcGxpY2Fz ICsgQXJiaXRlciBOb2RlKSwgdGhpcyBzZXR1cCBhY3R1YWxseSBkb2Vzbj05MnQgcmVxdWlyZSB0 aGUgM3JkIHNlcnZlPQpyIHRvIGhhdmUgYmlnIGRyaXZlcyBhdCBhbGwgYXMgaXQgb25seSBzdG9y ZXMgbWV0YS1kYXRhIGFib3V0IHRoZSBmaWxlcyBhbmQ9CiBub3QgYWN0dWFsbHkgYSBmdWxsIGNv cHkuCgpQbGVhc2Ugc2VlIHRoZSBhdHRhY2hlZCBkb2N1bWVudCB0aGF0IHdhcyBnaXZlbiB0byBt ZSBieSBSZWQgSGF0IHRvIGdldCBtb3I9CmUgaW5mb3JtYXRpb24gb24gdGhpcy4gSG9wZSB0aGlz IGluZm9ybWF0aW9uIGhlbHBzIHlvdS4KCgotLQoKRGV2aW4gQWNvc3RhLCBSSENBLCBSSFZDQQpS ZWQgSGF0IENlcnRpZmllZCBBcmNoaXRlY3QKCgpPbiBBdWd1c3QgNiwgMjAxNyBhdCA3OjI5OjI5 IFBNLCBNb2FjaXIgRmVycmVpcmEgKG1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY289Cm08bWFpbHRv Om1vYWNpcmZlcnJlaXJhQGhvdG1haWwuY29tPikgd3JvdGU6CgpJIGFtIHdpbGxpbmcgdG8gYXNz ZW1ibGUgYSBvVmlydCAicG9kIiwgbWFkZSBvZiAzIHNlcnZlcnMsIGVhY2ggd2l0aCAyIENQVSA9 CnNvY2tldHMgb2YgMTIgY29yZXMsIDI1NkdCIFJBTSwgNyBIREQgMTBLLCAxIFNTRC4gVGhlIGlk ZWEgaXMgdG8gdXNlIEdsdXN0ZT0KckZTIHRvIHByb3ZpZGUgSEEgZm9yIHRoZSBWTXMuIFRoZSAz IHNlcnZlcnMgaGF2ZSBhIGR1YWwgNDBHYiBOSUMgYW5kIGEgZHVhPQpsIDEwR2IgTklDLiBTbyBt eSBpbnRlbnRpb24gaXMgdG8gY3JlYXRlIGEgbG9vcCBsaWtlIGEgc2VydmVyIHRyaWFuZ2xlIHVz aW49CmcgdGhlIDQwR2IgTklDcyBmb3IgdmlydHVhbGl6YXRpb24gZmlsZXMgKFZNcyAucWNvdzIp IGFjY2VzcyBhbmQgdG8gbW92ZSBWTT0KcyBhcm91bmQgdGhlIHBvZCAoZWFzdCAvd2VzdCB0cmFm ZmljKSB3aGlsZSB1c2luZyB0aGUgMTBHYiBpbnRlcmZhY2VzIGZvciBnPQppdmluZyBzZXJ2aWNl cyB0byB0aGUgb3V0c2lkZSB3b3JsZCAobm9ydGgvc291dGggdHJhZmZpYykuCgoKVGhpcyBzYWlk LCBteSBmaXJzdCBxdWVzdGlvbiBpczogSG93IHNob3VsZCBJIGRlcGxveSBHbHVzdGVyRlMgaW4g c3VjaCBvVmlyPQp0IHNjZW5hcmlvPyBNeSBxdWVzdGlvbnMgYXJlOgoKCjEgLSBTaG91bGQgSSBj cmVhdGUgMyBSQUlEIChpLmUuOiBSQUlEIDUpLCBvbmUgb24gZWFjaCBvVmlydCBub2RlLCBhbmQg dGhlbj0KIGNyZWF0ZSBhIEdsdXN0ZXJGUyB1c2luZyB0aGVtPwoKMiAtIEluc3RlYWQsIHNob3Vs ZCBJIGNyZWF0ZSBhIEpCT0QgYXJyYXkgbWFkZSBvZiBhbGwgc2VydmVyJ3MgZGlza3M/CgozIC0g V2hhdCBpcyB0aGUgYmVzdCBHbHVzdGVyIGNvbmZpZ3VyYXRpb24gdG8gcHJvdmlkZSBmb3IgSEEg d2hpbGUgbm90IGNvbnM9CnVtaW5nIHRvbyBtdWNoIGRpc2sgc3BhY2U/Cgo0IC0gRG9lcyBhIG9W aXJ0IGh5cGVydmlzb3IgcG9kIGxpa2UgSSBhbSBwbGFubmluZyB0byBidWlsZCwgYW5kIHRoZSB2 aXJ0dWE9CmxpemF0aW9uIGVudmlyb25tZW50LCBiZW5lZml0cyBmcm9tIHRpZXJpbmcgd2hlbiB1 c2luZyBhIFNTRCBkaXNrPyBBbmQgeWVzLD0KIHdpbGwgR2x1c3RlciBkbyBpdCBieSBkZWZhdWx0 IG9yIEkgaGF2ZSB0byBjb25maWd1cmUgaXQgdG8gZG8gc28/CgoKQXQgdGhlIGJvdHRvbSBsaW5l LCB3aGF0IGlzIHRoZSBnb29kIHByYWN0aWNlIGZvciB1c2luZyBHbHVzdGVyRlMgaW4gc21hbGwg PQpwb2RzIGZvciBlbnRlcnByaXNlcz8KCgpZb3Ugb3Bpbmlvbi9mZWVkYmFjayB3aWxsIGJlIHJl YWxseSBhcHByZWNpYXRlZCEKCk1vYWNpcgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX18KVXNlcnMgbWFpbGluZyBsaXN0ClVzZXJzQG92aXJ0Lm9yZzxtYWls dG86VXNlcnNAb3ZpcnQub3JnPgpodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGlu Zm8vdXNlcnMKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18K VXNlcnMgbWFpbGluZyBsaXN0ClVzZXJzQG92aXJ0Lm9yZzxtYWlsdG86VXNlcnMlNDBvdmlydC5v cmc+Cmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycwoKCi0tXzAw MF9EQjZQMTkwTUIwMjgwQkQwQkI0MzlEMEM4QTQ1ODhEMDBDODhBMERCNlAxOTBNQjAyODBFVVJQ XwpDb250ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD0iV2luZG93cy0xMjUyIgpDb250ZW50 LVRyYW5zZmVyLUVuY29kaW5nOiBxdW90ZWQtcHJpbnRhYmxlCgo8aHRtbD4KPGhlYWQ+CjxtZXRh IGh0dHAtZXF1aXY9M0QiQ29udGVudC1UeXBlIiBjb250ZW50PTNEInRleHQvaHRtbDsgY2hhcnNl dD0zRFdpbmRvd3MtMT0KMjUyIj4KPHN0eWxlIHR5cGU9M0QidGV4dC9jc3MiIHN0eWxlPTNEImRp c3BsYXk6bm9uZTsiPjwhLS0gUCB7bWFyZ2luLXRvcDowO21hcmdpPQpuLWJvdHRvbTowO30gLS0+ PC9zdHlsZT4KPC9oZWFkPgo8Ym9keSBkaXI9M0QibHRyIj4KPGRpdiBpZD0zRCJkaXZ0YWdkZWZh dWx0d3JhcHBlciIgc3R5bGU9M0QiZm9udC1zaXplOjEycHQ7Y29sb3I6IzAwMDAwMDtmb250PQot ZmFtaWx5OkNhbGlicmksSGVsdmV0aWNhLHNhbnMtc2VyaWY7IiBkaXI9M0QibHRyIj4KPHA+VGhh bmtzIEpvaGFuLCB5b3UgYnJvdWdodCAmcXVvdDtsaWdodCZxdW90OyBpbnRvIG15IGRhcmtuZXNz ISBJIHdlbnQgbG9vPQpraW5nIGZvciB0aGUgR2x1c3RlckZTIHRpZXJpbmcgaG93LXRvIGFuZCBp dCBsb29rcyBsaWtlIHF1aXRlIHNpbXBsZSB0byBhdHQ9CmFjaCBhIFNTRCBhcyBob3QgdGllci4g Rm9yIHRob3NlIHdpbGxpbmcgdG8gcmVhZCBhYm91dCBpdCwgZ28gaGVyZToKPGEgaHJlZj0zRCJo dHRwOi8vYmxvZy5nbHVzdGVyLm9yZy8yMDE2LzAzL2F1dG9tYXRlZC10aWVyaW5nLWluLWdsdXN0 ZXIvIiBjPQpsYXNzPTNEIk9XQUF1dG9MaW5rIiBpZD0zRCJMUGxuazkzOTM0MyIgcHJldmlld3Jl bW92ZWQ9M0QidHJ1ZSI+Cmh0dHA6Ly9ibG9nLmdsdXN0ZXIub3JnLzIwMTYvMDMvYXV0b21hdGVk LXRpZXJpbmctaW4tZ2x1c3Rlci88L2E+PC9wPgo8cD48YnI+CjwvcD4KPHA+Tm93LCBJIHN0aWxs IGhhdmUgYSBxdWVzdGlvbjogVk1zIGFyZSBtYWRlIG9mIHZlcnkgbGFyZ2UgLnFjb3cyIGZpbGVz LiBNPQp5IHVuZGVyc3RhbmRpbmcgaXMgdGhhdCBmaWxlcyBpbiBHbHVzdGVyIGFyZSBrZXB0IGFs bCB0b2dldGhlciBpbiBhIHNpbmdsZSA9CmJyaWNrLiBJZiBzbywgSSB3aWxsIG5vdCBiZW5lZml0 IGZyb20gdGllcmluZyBhcyBhIHNpbmdsZSBTU0Qgd2lsbCBub3QgYmUgYj0KaWcgZW5vdWdoIHRv IGZpdCBhbGwgbXkgbGFyZ2UgVk0KIC5xY293MiBmaWxlcy4gVGhpcyB3b3VsZCBub3QgYmUgdHJ1 ZSBpZiBHbHVzdGVyIGNhbiBzdG9yZSAmcXVvdDtibG9ja3MmcXVvPQp0OyBvZiBkYXRhIHRoYXQg Y29tcG9zZSBhIGxhcmdlIGZpbGUgc3ByZWFkIG9uIHNldmVyYWwgYnJpY2tzLiBCdXQgaWYgSSBh bSA9Cm5vdCB3cm9uZywgdGhpcyBpcyBvbmUgb2Yga2V5IGRpZmZlcmVuY2VzIGluIGJldHdlZW4g R2x1c3RlckZTIGFuZCBDZXBoLiBDYT0KbiB5b3UgY29tbWVudD88L3A+CjxwPjxicj4KPC9wPgo8 cD5Nb2FjaXI8YnI+CjwvcD4KPGJyPgo8YnI+CjxkaXYgc3R5bGU9M0QiY29sb3I6IHJnYig0OSwg NTUsIDU3KTsiPgo8aHIgdGFiaW5kZXg9M0QiLTEiIHN0eWxlPTNEImRpc3BsYXk6aW5saW5lLWJs b2NrOyB3aWR0aDo5OCUiPgo8ZGl2IGlkPTNEImRpdlJwbHlGd2RNc2ciIGRpcj0zRCJsdHIiPjxm b250IHN0eWxlPTNEImZvbnQtc2l6ZToxMXB0IiBmYWNlPQo9M0QiQ2FsaWJyaSwgc2Fucy1zZXJp ZiIgY29sb3I9M0QiIzAwMDAwMCI+PGI+RnJvbTo8L2I+IEpvaGFuIEJlcm5oYXJkc3NvbiA9CiZs dDtqb2hhbkBrYWZpdC5zZSZndDs8YnI+CjxiPlNlbnQ6PC9iPiBUdWVzZGF5LCBBdWd1c3QgOCwg MjAxNyA3OjAzIEFNPGJyPgo8Yj5Ubzo8L2I+IE1vYWNpciBGZXJyZWlyYTsgRGV2aW4gQWNvc3Rh OyB1c2Vyc0BvdmlydC5vcmc8YnI+CjxiPlN1YmplY3Q6PC9iPiBSZTogW292aXJ0LXVzZXJzXSBH b29kIHByYWN0aWNlczwvZm9udD4KPGRpdj4mbmJzcDs8L2Rpdj4KPC9kaXY+CjxkaXY+CjxkaXYg c3R5bGU9M0QiY29sb3I6YmxhY2siPgo8ZGl2IHN0eWxlPTNEImNvbG9yOmJsYWNrIj4KPHAgc3R5 bGU9M0QibWFyZ2luOjAgMCAxZW0gMDsgY29sb3I6YmxhY2siPllvdSBhdHRhY2ggdGhlIHNzZCBh cyBhIGhvdCB0aWVyPQogd2l0aCBhIGdsdXN0ZXIgY29tbWFuZC4gSSBkb24ndCB0aGluayB0aGF0 IGdkZXBsb3kgb3Igb3ZpcnQgZ3VpIGNhbiBkbyBpdC49CjwvcD4KPHAgc3R5bGU9M0QibWFyZ2lu OjAgMCAxZW0gMDsgY29sb3I6YmxhY2siPlRoZSBnbHVzdGVyIGRvY3MgYW5kIHJlZGhhdCBkb2Nz PQogZXhwbGFpbnMgdGllcmluZyBxdWl0ZSBnb29kLjwvcD4KPHAgc3R5bGU9M0QibWFyZ2luOjAg MCAxZW0gMDsgY29sb3I6YmxhY2siPi9Kb2hhbjwvcD4KPC9kaXY+CjxkaXYgc3R5bGU9M0QiY29s b3I6YmxhY2siPgo8cCBzdHlsZT0zRCJjb2xvcjpibGFjazsgZm9udC1zaXplOjEwcHQ7IGZvbnQt ZmFtaWx5OkFyaWFsLHNhbnMtc2VyaWY7IG1hcmc9CmluOjEwcHQgMCI+Ck9uIEF1Z3VzdCA4LCAy MDE3IDA3OjA2OjQyIE1vYWNpciBGZXJyZWlyYSAmbHQ7bW9hY2lyZmVycmVpcmFAaG90bWFpbC5j b20mZz0KdDsgd3JvdGU6PC9wPgo8YmxvY2txdW90ZSB0eXBlPTNEImNpdGUiIGNsYXNzPTNEImdt YWlsX3F1b3RlIiBzdHlsZT0zRCJtYXJnaW46MCAwIDAgMC43NWU9Cng7IGJvcmRlci1sZWZ0OjFw eCBzb2xpZCAjODA4MDgwOyBwYWRkaW5nLWxlZnQ6MC43NWV4Ij4KPGRpdiBpZD0zRCJkaXZ0YWdk ZWZhdWx0d3JhcHBlciIgZGlyPTNEImx0ciIgc3R5bGU9M0QiZm9udC1zaXplOjEycHQ7IGNvbG9y PQo6IzAwMDAwMDsgZm9udC1mYW1pbHk6Q2FsaWJyaSxIZWx2ZXRpY2Esc2Fucy1zZXJpZiI+Cjxw PkhpIERldmluLDwvcD4KPHA+PGJyPgo8L3A+CjxwPlBsZWFzZSBjb25zaWRlciB0aGF0IGZvciB0 aGUgT1MgSSBoYXZlIGEgUkFJRCAxLiBOb3csIGxldHMgc2F5IEkgdXNlIFJBST0KRCA1IHRvIGFz c2VtYmxlIGEgc2luZ2xlIGRpc2sgb24gZWFjaCBzZXJ2ZXIuIEluIHRoaXMgY2FzZSwgdGhlIFNT RCB3aWxsIG5vPQp0IG1ha2UgYW55IGRpZmZlcmVuY2UsIHJpZ2h0PyBJIGd1ZXNzIHRoYXQgdG8g YmUgcG9zc2libGUgdG8gdXNlIGl0LCB0aGUgU1M9CkQgc2hvdWxkIG5vdCBiZSBwYXJ0IG9mIHRo ZSBSQUlECiA1LiBJbiB0aGlzIGNhc2UgSSBjb3VsZCBjcmVhdGUgYSBsb2dpY2FsIHZvbHVtZSBt YWRlIG9mIHRoZSBSQUlEZWQgYnJpY2sgYT0KbmQgdGhlbiBleHRlbmQgaXQgdXNpbmcgdGhlIFNT RC4gSS5lLjogVXNpbmcgZ2RlcGxveTo8L3A+CjxwPjxicj4KPC9wPgo8cD5bZGlza3R5cGVdPC9w Pgo8cD5qYm9kPC9wPgo8cD4uLi4uPC9wPgo8cD5bcHYxXTwvcD4KPHA+YWN0aW9uPTNEY3JlYXRl PC9wPgo8cD5kZXZpY2VzPTNEc2RiLCBzZGM8L3A+CjxwPndpcGVmcz0zRHllczwvcD4KPHA+PC9w Pgo8cD5pZ25vcmVfdmdfZXJyb3M9M0Rubzxicj4KPC9wPgo8cD48YnI+CjwvcD4KPHA+PC9wPgo8 cD5bdmcxXTwvcD4KPHA+PGI+YWN0aW9uPTNEY3JlYXRlPC9iPjwvcD4KPHA+dmduYW1lPTNEZ2x1 c3Rlcl92Z19qYm9kPC9wPgo8cD5wdm5hbWU9M0RzZGI8L3A+CjxwPmlnbm9yZV92Z19lcnJvcz0z RG5vPGJyPgo8L3A+CjxwPjxicj4KPC9wPgo8cD5bdmcyXTwvcD4KPHA+PGI+YWN0aW9uPTNEZXh0 ZW5kPC9iPjwvcD4KPHA+dmduYW1lPTNEZ2x1c3Rlcl92Z19qYm9kPC9wPgo8cD5wdm5hbWU9M0Rz ZGM8L3A+CjxwPmlnbm9yZV92Z19lcnJvcz0zRG5vPGJyPgo8L3A+Cjxicj4KPHA+QnV0IHdpbGwg R2x1c3RlciBiZSBhYmxlIHRvIGF1dG8tZGV0ZWN0IGFuZCB1c2UgdGhpcyBTU0QgYnJpY2sgZm9y IHRpZXJpPQpuZz8gRG8gSSBoYXZlIHRvIGRvIHNvbWUgb3RoZXIgY29uZmlndXJhdGlvbnM/IEFs c28sIGFzIHRoZSBWTSBmaWxlcyAoLnFjb3c9CjIpIGFyZSBxdWl0ZSBiaWcgd2lsbCBJIGJlbmVm aXQgZnJvbSB0aWVyaW5nPyBUaGlzIGlzIHdyb25nIGFuZCBteSBhcHByb2FjaD0KIHNob3VsZCBi ZSBvdGhlcj88L3A+CjxwPjxicj4KPC9wPgo8cD5UaGFua3MsPC9wPgo8cD5Nb2FjaXI8YnI+Cjwv cD4KPHA+PGJyPgo8L3A+Cjxicj4KPGRpdiBzdHlsZT0zRCJjb2xvcjpyZ2IoNDksNTUsNTcpIj4K PGhyIHRhYmluZGV4PTNEIi0xIiBzdHlsZT0zRCJkaXNwbGF5OmlubGluZS1ibG9jazsgd2lkdGg6 OTglIj4KPGRpdiBpZD0zRCJkaXZScGx5RndkTXNnIiBkaXI9M0QibHRyIj48Zm9udCBzdHlsZT0z RCJmb250LXNpemU6MTFwdCIgZmFjZT0KPTNEIkNhbGlicmksIHNhbnMtc2VyaWYiIGNvbG9yPTNE IiMwMDAwMDAiPjxiPkZyb206PC9iPiBEZXZpbiBBY29zdGEgJmx0O2RlPQp2aW5AcGFic3RhdGVu Y2lvLmNvbSZndDs8YnI+CjxiPlNlbnQ6PC9iPiBNb25kYXksIEF1Z3VzdCA3LCAyMDE3IDc6NDYg QU08YnI+CjxiPlRvOjwvYj4gTW9hY2lyIEZlcnJlaXJhOyB1c2Vyc0BvdmlydC5vcmc8YnI+Cjxi PlN1YmplY3Q6PC9iPiBSZTogW292aXJ0LXVzZXJzXSBHb29kIHByYWN0aWNlczwvZm9udD4KPGRp dj4mbmJzcDs8L2Rpdj4KPC9kaXY+CjxkaXY+CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIg c3R5bGU9M0QiY29sb3I6cmdiKDAsMCwwKTsgbWFyZ2luOjBweCI+PGZvbnQgZj0KYWNlPTNEIklu cHV0IE1vbm8iPjxicj4KPC9mb250PjwvZGl2Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQi IHN0eWxlPTNEImNvbG9yOnJnYigwLDAsMCk7IG1hcmdpbjowcHgiPjxmb250IGY9CmFjZT0zRCJJ bnB1dCBNb25vIj5Nb2FjaXIsPC9mb250PjwvZGl2Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZv bnQiIHN0eWxlPTNEImNvbG9yOnJnYigwLDAsMCk7IG1hcmdpbjowcHgiPjxmb250IGY9CmFjZT0z RCJJbnB1dCBNb25vIj48YnI+CjwvZm9udD48L2Rpdj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21m b250IiBzdHlsZT0zRCJjb2xvcjpyZ2IoMCwwLDApOyBtYXJnaW46MHB4Ij48Zm9udCBmPQphY2U9 M0QiSW5wdXQgTW9ubyI+SSBoYXZlIHJlY2VudGx5IGluc3RhbGxlZCBtdWx0aXBsZSBSZWQgSGF0 IFZpcnR1YWxpemF0aW89Cm4gaG9zdHMgZm9yIHNldmVyYWwgZGlmZmVyZW50IGNvbXBhbmllcywg YW5kIGhhdmUgZGVhbHQgd2l0aCB0aGUgUmVkIEhhdCBTdT0KcHBvcnQgVGVhbSBpbiBkZXB0aCBh Ym91dCBvcHRpbWFsIGNvbmZpZ3VyYXRpb24KIGluIHJlZ2FyZHMgdG8gc2V0dGluZyB1cCBHbHVz dGVyRlMgbW9zdCBlZmZpY2llbnRseSBhbmQgSSB3YW50ZWQgdG8gc2hhcmUgPQp3aXRoIHlvdSB3 aGF0IEkgbGVhcm5lZC48L2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIg c3R5bGU9M0QiY29sb3I6cmdiKDAsMCwwKTsgbWFyZ2luOjBweCI+PGZvbnQgZj0KYWNlPTNEIklu cHV0IE1vbm8iPjxicj4KPC9mb250PjwvZGl2Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQi IHN0eWxlPTNEImNvbG9yOnJnYigwLDAsMCk7IG1hcmdpbjowcHgiPjxmb250IGY9CmFjZT0zRCJJ bnB1dCBNb25vIj5JbiBnZW5lcmFsIFJlZCBIYXQgVmlydHVhbGl6YXRpb24gdGVhbSBmcm93bnMg dXBvbiB1c2luZz0KIGVhY2ggRElTSyBvZiB0aGUgc3lzdGVtIGFzIGp1c3QgYSBKQk9ELCBzdXJl IHRoZXJlIGlzIHNvbWUgcHJvdGVjdGlvbiBieSBoPQphdmluZyB0aGUgZGF0YSByZXBsaWNhdGVk LCBob3dldmVyLCB0aGUKIHJlY29tbWVuZGF0aW9uIGlzIHRvIHVzZSBSQUlEIDYgKHByZWZlcnJl ZCkgb3IgUkFJRC01LCBvciBhdCBsZWFzdCBSQUlELTEgPQphdCB0aGUgdmVyeSBsZWFzdC48L2Zv bnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9M0QiY29sb3I6cmdi KDAsMCwwKTsgbWFyZ2luOjBweCI+PGZvbnQgZj0KYWNlPTNEIklucHV0IE1vbm8iPjxicj4KPC9m b250PjwvZGl2Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEIm1hcmdpbjow cHgiPjxmb250IGZhY2U9M0QiSW5wdXQgTW9ubyI9Cj5IZXJlIGlzIHRoZSZuYnNwO2RpcmVjdCBx dW90ZSBmcm9tIFJlZCBIYXQgd2hlbiBJIGFza2VkIGFib3V0IFJBSUQgYW5kIEJyaT0KY2tzOjwv Zm9udD48L2Rpdj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21mb250IiBzdHlsZT0zRCJtYXJnaW46 MHB4Ij48Zm9udCBmYWNlPTNEIklucHV0IE1vbm8iPQo+PGk+PGJyPgo8L2k+PC9mb250PjwvZGl2 Pgo8ZGl2IGlkPTNEImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEIm1hcmdpbjowcHgiPjxmb250 IGZhY2U9M0QiSW5wdXQgTW9ubyI9Cj48aT4mcXVvdDtBIHR5cGljYWwgR2x1c3RlciBjb25maWd1 cmF0aW9uIHdvdWxkIHVzZSBSQUlEIHVuZGVybmVhdGggdGhlIGJyaT0KY2tzLiBSQUlEIDYgaXMg bW9zdCB0eXBpY2FsIGFzIGl0IGdpdmVzIHlvdSAyIGRpc2sgZmFpbHVyZSBwcm90ZWN0aW9uLCBi dXQgPQpSQUlEIDUgY291bGQgYmUgdXNlZCB0b28uJm5ic3A7T25jZSB5b3UgaGF2ZSB0aGUKIFJB SURlZCBicmlja3MsIHlvdSdkIHRoZW4gYXBwbHkgdGhlIGRlc2lyZWQgcmVwbGljYXRpb24gb24g dG9wIG9mIHRoYXQuIFRoPQplIG1vc3QgcG9wdWxhciB3YXkgb2YgZG9pbmcgdGhpcyB3b3VsZCBi ZSBkaXN0cmlidXRlZCByZXBsaWNhdGVkIHdpdGggMnggcmU9CnBsaWNhdGlvbi4mbmJzcDtJbiBn ZW5lcmFsIHlvdSdsbCBnZXQgYmV0dGVyIHBlcmZvcm1hbmNlIHdpdGggbGFyZ2VyIGJyaWNrcz0K LiAxMiBkcml2ZXMgaXMgb2Z0ZW4gYSBzd2VldCBzcG90LiBBbm90aGVyCiBvcHRpb24gd291bGQg YmUgdG8gY3JlYXRlIGEgc2VwYXJhdGUgdGllciB1c2luZyBhbGwgU1NEPTkycy49OTQmbmJzcDs8 L2k+PD0KL2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9M0Qi bWFyZ2luOjBweCI+PGJyPgo8L2Rpdj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21mb250IiBzdHls ZT0zRCJtYXJnaW46MHB4Ij48Zm9udCBmYWNlPTNEIklucHV0IE1vbm8iPQo+PGk+SW4gb3JkZXIg dG8gU1NEJm5ic3A7dGllcmluZyBmcm9tIG15IHVuZGVyc3RhbmRpbmcgeW91IHdvdWxkIG5lZWQg MSB4IE49ClZNZSBkcml2ZSBpbiBlYWNoIHNlcnZlciwgb3IgNCB4IFNTRCBob3QgdGllciAoaXQg bmVlZHMgdG8gYmUgZGlzdHJpYnV0ZWQsID0KcmVwbGljYXRlZCBmb3IgdGhlIGhvdCB0aWVyIGlm IG5vdCB1c2luZyBOVk1FKS4KIFNvIHdpdGggeW91IG9ubHkgaGF2aW5nIDEgU1NEIGRyaXZlIGlu IGVhY2ggc2VydmVyLCZuYnNwO0k9OTJkIHN1Z2dlc3QgbWF5PQpiZSBsb29raW5nIGludG8gdGhl IE5WTUUgb3B0aW9uLiZuYnNwOzwvaT48L2ZvbnQ+PC9kaXY+CjxkaXYgaWQ9M0QiYmxvb3BfY3Vz dG9tZm9udCIgc3R5bGU9M0QibWFyZ2luOjBweCI+PGZvbnQgZmFjZT0zRCJJbnB1dCBNb25vIj0K PjxpPjxicj4KPC9pPjwvZm9udD48L2Rpdj4KPGRpdiBpZD0zRCJibG9vcF9jdXN0b21mb250IiBz dHlsZT0zRCJtYXJnaW46MHB4Ij48Zm9udCBmYWNlPTNEIklucHV0IE1vbm8iPQo+PGk+U2luY2Ug eW91ciB1c2luZyBvbmx5IDMtc2VydmVycywgd2hhdCBJPTkyZCBwcm9iYWJseSBzdWdnZXN0IGlz IHRvIGRvICg9CjIgUmVwbGljYXMgJiM0MzsgQXJiaXRlciBOb2RlKSwgdGhpcyBzZXR1cCBhY3R1 YWxseSBkb2Vzbj05MnQgcmVxdWlyZSB0aGUgMz0KcmQgc2VydmVyIHRvIGhhdmUgYmlnIGRyaXZl cyBhdCBhbGwgYXMgaXQgb25seQogc3RvcmVzIG1ldGEtZGF0YSBhYm91dCB0aGUgZmlsZXMgYW5k IG5vdCBhY3R1YWxseSBhIGZ1bGwgY29weS4mbmJzcDs8L2k+PC89CmZvbnQ+PC9kaXY+CjxkaXYg aWQ9M0QiYmxvb3BfY3VzdG9tZm9udCIgc3R5bGU9M0QibWFyZ2luOjBweCI+PGZvbnQgZmFjZT0z RCJJbnB1dCBNb25vIj0KPjxpPjxicj4KPC9pPjwvZm9udD48L2Rpdj4KPGRpdiBpZD0zRCJibG9v cF9jdXN0b21mb250IiBzdHlsZT0zRCJtYXJnaW46MHB4Ij48Zm9udCBmYWNlPTNEIklucHV0IE1v bm8iPQo+PGk+UGxlYXNlIHNlZSB0aGUgYXR0YWNoZWQgZG9jdW1lbnQgdGhhdCB3YXMgZ2l2ZW4g dG8gbWUgYnkgUmVkIEhhdCB0byBnZXQ9CiBtb3JlIGluZm9ybWF0aW9uIG9uIHRoaXMuIEhvcGUg dGhpcyBpbmZvcm1hdGlvbiBoZWxwcyB5b3UuPC9pPjwvZm9udD48L2Rpdj0KPgo8ZGl2IGlkPTNE ImJsb29wX2N1c3RvbWZvbnQiIHN0eWxlPTNEIm1hcmdpbjowcHgiPjxmb250IGZhY2U9M0QiSW5w dXQgTW9ubyI9Cj48aT48YnI+CjwvaT48L2ZvbnQ+PC9kaXY+Cjxicj4KPGRpdiBpZD0zRCJibG9v cF9zaWduXzE1MDIwODczNzY3MjU0NjkxODQiIGNsYXNzPTNEImJsb29wX3NpZ24iPjxzcGFuIHN0 eWxlPQo9M0QiZm9udC1mYW1pbHk6J2hlbHZldGljYSBOZXVlJyxoZWx2ZXRpY2E7IGZvbnQtc2l6 ZToxNHB4Ij4tLTwvc3Bhbj48YnIgc3Q9CnlsZT0zRCJmb250LWZhbWlseTonaGVsdmV0aWNhIE5l dWUnLGhlbHZldGljYTsgZm9udC1zaXplOjE0cHgiPgo8ZGl2IGNsYXNzPTNEImdtYWlsX3NpZ25h dHVyZSIgc3R5bGU9M0QiZm9udC1mYW1pbHk6J2hlbHZldGljYSBOZXVlJyxoZWx2ZXQ9CmljYTsg Zm9udC1zaXplOjE0cHgiPgo8ZGl2IGRpcj0zRCJsdHIiPgo8ZGl2Pjxicj4KPC9kaXY+CjxkaXY+ RGV2aW4gQWNvc3RhLCBSSENBLCBSSFZDQTwvZGl2Pgo8ZGl2PlJlZCBIYXQgQ2VydGlmaWVkIEFy Y2hpdGVjdDwvZGl2Pgo8ZGl2PjwvZGl2Pgo8L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8YnI+CjxwIGNs YXNzPTNEImFpcm1haWxfb24iPk9uIEF1Z3VzdCA2LCAyMDE3IGF0IDc6Mjk6MjkgUE0sIE1vYWNp ciBGZXJyZWlyYSAoPD0KYSBocmVmPTNEIm1haWx0bzptb2FjaXJmZXJyZWlyYUBob3RtYWlsLmNv bSI+bW9hY2lyZmVycmVpcmFAaG90bWFpbC5jb208L2E+PQopIHdyb3RlOjwvcD4KPGJsb2NrcXVv dGUgdHlwZT0zRCJjaXRlIiBjbGFzcz0zRCJjbGVhbl9icSI+PHNwYW4+CjxkaXYgZGlyPTNEImx0 ciI+CjxkaXY+PC9kaXY+CjxkaXY+CjxkaXYgaWQ9M0QiZGl2dGFnZGVmYXVsdHdyYXBwZXIiIGRp cj0zRCJsdHIiIHN0eWxlPTNEImZvbnQtc2l6ZToxMnB0OyBjb2xvcj0KOiMwMDAwMDA7IGZvbnQt ZmFtaWx5OkNhbGlicmksSGVsdmV0aWNhLHNhbnMtc2VyaWYiPgo8cD48c3Bhbj5JIGFtIHdpbGxp bmcgdG8gYXNzZW1ibGUgYSBvVmlydCAmcXVvdDtwb2QmcXVvdDssIG1hZGUgb2YgMyBzZXJ2ZXI9 CnMsIGVhY2ggd2l0aCAyIENQVSBzb2NrZXRzIG9mIDEyIGNvcmVzLCAyNTZHQiBSQU0sIDcgSERE IDEwSywgMSBTU0QuIFRoZSBpZD0KZWEgaXMgdG8gdXNlIEdsdXN0ZXJGUyB0byBwcm92aWRlIEhB IGZvciB0aGUgVk1zLiBUaGUgMyBzZXJ2ZXJzIGhhdmUgYSBkdWFsPQogNDBHYiBOSUMgYW5kIGEg ZHVhbCAxMEdiIE5JQy4gU28gbXkgaW50ZW50aW9uCiBpcyB0byBjcmVhdGUgYSBsb29wIGxpa2Ug YSBzZXJ2ZXIgdHJpYW5nbGUgdXNpbmcgdGhlIDQwR2IgTklDcyBmb3IgdmlydHVhbD0KaXphdGlv biBmaWxlcyAoVk1zIC5xY293MikgYWNjZXNzIGFuZCB0byBtb3ZlIFZNcyBhcm91bmQgdGhlIHBv ZCAoZWFzdCAvd2VzPQp0IHRyYWZmaWMpIHdoaWxlIHVzaW5nIHRoZSAxMEdiIGludGVyZmFjZXMg Zm9yIGdpdmluZyBzZXJ2aWNlcyB0byB0aGUgb3V0c2k9CmRlIHdvcmxkIChub3J0aC9zb3V0aCB0 cmFmZmljKS48L3NwYW4+PC9wPgo8cD48YnI+CjwvcD4KPHA+VGhpcyBzYWlkLCBteSBmaXJzdCBx dWVzdGlvbiBpczogSG93IHNob3VsZCBJIGRlcGxveSBHbHVzdGVyRlMgaW4gc3VjaCBvPQpWaXJ0 IHNjZW5hcmlvPyBNeSBxdWVzdGlvbnMgYXJlOjwvcD4KPHA+PGJyPgo8L3A+CjxwPjEgLSBTaG91 bGQgSSBjcmVhdGUgMyBSQUlEIChpLmUuOiBSQUlEIDUpLCBvbmUgb24gZWFjaCBvVmlydCBub2Rl LCBhbmQgdD0KaGVuIGNyZWF0ZSBhIEdsdXN0ZXJGUyB1c2luZyB0aGVtPzwvcD4KPHA+MiAtIElu c3RlYWQsIHNob3VsZCBJIGNyZWF0ZSBhIEpCT0QgYXJyYXkgbWFkZSBvZiBhbGwgc2VydmVyJ3Mg ZGlza3M/PC9wPQo+CjxwPjMgLSBXaGF0IGlzIHRoZSBiZXN0IEdsdXN0ZXIgY29uZmlndXJhdGlv biB0byBwcm92aWRlIGZvciBIQSB3aGlsZSBub3QgYz0Kb25zdW1pbmcgdG9vIG11Y2ggZGlzayBz cGFjZT88YnI+CjwvcD4KPHA+NCAtIERvZXMgYSBvVmlydCBoeXBlcnZpc29yIHBvZCBsaWtlIEkg YW0gcGxhbm5pbmcgdG8gYnVpbGQsIGFuZCB0aGUgdmlyPQp0dWFsaXphdGlvbiBlbnZpcm9ubWVu dCwgYmVuZWZpdHMgZnJvbSB0aWVyaW5nIHdoZW4gdXNpbmcgYSBTU0QgZGlzaz8gQW5kIHk9CmVz LCB3aWxsIEdsdXN0ZXIgZG8gaXQgYnkgZGVmYXVsdCBvciBJIGhhdmUgdG8gY29uZmlndXJlIGl0 IHRvIGRvIHNvPzwvcD4KPHA+PGJyPgo8L3A+CjxwPkF0IHRoZSBib3R0b20gbGluZSwgd2hhdCBp cyB0aGUgZ29vZCBwcmFjdGljZSBmb3IgdXNpbmcgR2x1c3RlckZTIGluIHNtYT0KbGwgcG9kcyBm b3IgZW50ZXJwcmlzZXM/PGJyPgo8L3A+CjxwPjxicj4KPC9wPgo8cD5Zb3Ugb3Bpbmlvbi9mZWVk YmFjayB3aWxsIGJlIHJlYWxseSBhcHByZWNpYXRlZCE8L3A+CjxwPk1vYWNpcjxicj4KPC9wPgo8 L2Rpdj4KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18gPGJy PgpVc2VycyBtYWlsaW5nIGxpc3QgPGJyPgo8YSBocmVmPTNEIm1haWx0bzpVc2Vyc0BvdmlydC5v cmciPlVzZXJzQG92aXJ0Lm9yZzwvYT4gPGJyPgo8YSBocmVmPTNEImh0dHA6Ly9saXN0cy5vdmly dC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyI+aHR0cDovL2xpc3RzLm92aXI9CnQub3JnL21h aWxtYW4vbGlzdGluZm8vdXNlcnM8L2E+Cjxicj4KPC9kaXY+CjwvZGl2Pgo8L3NwYW4+PC9ibG9j a3F1b3RlPgo8L2Rpdj4KPC9kaXY+CjwvZGl2PgpfX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fXzxicj4KVXNlcnMgbWFpbGluZyBsaXN0PGJyPgo8YSBjbGFzcz0z RCJhcW0tYXV0b2xpbmsgYXFtLWF1dG93cmFwIiBocmVmPTNEIm1haWx0bzpVc2VycyU0MG92aXJ0 Lm9yZyI+VXM9CmVyc0BvdmlydC5vcmc8L2E+PGJyPgo8YSBjbGFzcz0zRCJhcW0tYXV0b2xpbmsg YXFtLWF1dG93cmFwIiBocmVmPTNEImh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG09CmFuL2xp c3RpbmZvL3VzZXJzIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNl cnM8L2E+PGJyPgo8YnI+CjwvYmxvY2txdW90ZT4KPC9kaXY+CjwvZGl2Pgo8L2Rpdj4KPC9kaXY+ CjwvZGl2Pgo8L2JvZHk+CjwvaHRtbD4KCi0tXzAwMF9EQjZQMTkwTUIwMjgwQkQwQkI0MzlEMEM4 QTQ1ODhEMDBDODhBMERCNlAxOTBNQjAyODBFVVJQXy0tCg== --===============7391426873726200533==--