From mahdi.adnan at outlook.com Tue Jun 20 00:22:13 2017 Content-Type: multipart/mixed; boundary="===============4483317745325526287==" MIME-Version: 1.0 From: Mahdi Adnan To: users at ovirt.org Subject: Re: [ovirt-users] Very poor GlusterFS performance Date: Mon, 19 Jun 2017 17:51:19 +0000 Message-ID: In-Reply-To: f8e995d4-1eb2-c9a8-2f35-555c7174dc4a@bootc.net --===============4483317745325526287== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --_000_DM5PR01MB25069E435FC89BDDA5534AB2FFC40DM5PR01MB2506prod_ Content-Type: text/plain; charset=3D"us-ascii" Content-Transfer-Encoding: quoted-printable Hi, Can you put some numbers ? what tests are you doing ? Im running oVirt with Gluster without performance issues, but im running re= =3D plica 2 all SSDs. Gluster logs might help too. -- Respectfully Mahdi A. Mahdi ________________________________ From: users-bounces(a)ovirt.org on behalf of Ch= ris =3D Boot Sent: Monday, June 19, 2017 5:46:08 PM To: oVirt users Subject: [ovirt-users] Very poor GlusterFS performance Hi folks, I have 3x servers in a "hyper-converged" oVirt 4.1.2 + GlusterFS 3.10 configuration. My VMs run off a replica 3 arbiter 1 volume comprised of 6 bricks, which themselves live on two SSDs in each of the servers (one brick per SSD). The bricks are XFS on LVM thin volumes straight onto the SSDs. Connectivity is 10G Ethernet. Performance within the VMs is pretty terrible. I experience very low throughput and random IO is really bad: it feels like a latency issue. On my oVirt nodes the SSDs are not generally very busy. The 10G network seems to run without errors (iperf3 gives bandwidth measurements of >=3D3D 9.20 Gbits/sec between the three servers). To put this into perspective: I was getting better behaviour from NFS4 on a gigabit connection than I am with GlusterFS on 10G: that doesn't feel right at all. My volume configuration looks like this: Volume Name: vmssd Type: Distributed-Replicate Volume ID: d5a5ddd1-a140-4e0d-b514-701cfe464853 Status: Started Snapshot Count: 0 Number of Bricks: 2 x (2 + 1) =3D3D 6 Transport-type: tcp Bricks: Brick1: ovirt3:/gluster/ssd0_vmssd/brick Brick2: ovirt1:/gluster/ssd0_vmssd/brick Brick3: ovirt2:/gluster/ssd0_vmssd/brick (arbiter) Brick4: ovirt3:/gluster/ssd1_vmssd/brick Brick5: ovirt1:/gluster/ssd1_vmssd/brick Brick6: ovirt2:/gluster/ssd1_vmssd/brick (arbiter) Options Reconfigured: nfs.disable: on transport.address-family: inet6 performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off performance.low-prio-threads: 32 network.remote-dio: off cluster.eager-lock: enable cluster.quorum-type: auto cluster.server-quorum-type: server cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-max-threads: 8 cluster.shd-wait-qlength: 10000 features.shard: on user.cifs: off storage.owner-uid: 36 storage.owner-gid: 36 features.shard-block-size: 128MB performance.strict-o-direct: on network.ping-timeout: 30 cluster.granular-entry-heal: enable I would really appreciate some guidance on this to try to improve things because at this rate I will need to reconsider using GlusterFS altogether. Cheers, Chris -- Chris Boot bootc(a)bootc.net _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users --_000_DM5PR01MB25069E435FC89BDDA5534AB2FFC40DM5PR01MB2506prod_ Content-Type: text/html; charset=3D"us-ascii" Content-Transfer-Encoding: quoted-printable

Hi,

Can you put some numbers ? what tests are you doing ?

Im running oVirt with Gluster without performance issues, but im running= =3D replica 2 all SSDs.

Gluster logs might help too.

--

Respectfully
Mahdi A. Mahdi

From: users-bounces(a= )ovirt=3D .org <users-bounces(a)ovirt.org> on behalf of Chris Boot <bootc(a)= boot=3D c.net>
Sent: Monday, June 19, 2017 5:46:08 PM
To: oVirt users
Subject: [ovirt-users] Very poor GlusterFS performance

Hi folks,

I have 3x servers in a "hyper-converged" oVirt 4.1.2 + Gluste= =3D rFS 3.10
configuration. My VMs run off a replica 3 arbiter 1 volume comprised of
6 bricks, which themselves live on two SSDs in each of the servers (one
brick per SSD). The bricks are XFS on LVM thin volumes straight onto the SSDs. Connectivity is 10G Ethernet.

Performance within the VMs is pretty terrible. I experience very low
throughput and random IO is really bad: it feels like a latency issue.
On my oVirt nodes the SSDs are not generally very busy. The 10G network
seems to run without errors (iperf3 gives bandwidth measurements of >=3D= 3D=3D
9.20 Gbits/sec between the three servers).

To put this into perspective: I was getting better behaviour from NFS4
on a gigabit connection than I am with GlusterFS on 10G: that doesn't
feel right at all.

My volume configuration looks like this:

Volume Name: vmssd
Type: Distributed-Replicate
Volume ID: d5a5ddd1-a140-4e0d-b514-701cfe464853
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x (2 + 1) =3D3D 6
Transport-type: tcp
Bricks:
Brick1: ovirt3:/gluster/ssd0_vmssd/brick
Brick2: ovirt1:/gluster/ssd0_vmssd/brick
Brick3: ovirt2:/gluster/ssd0_vmssd/brick (arbiter)
Brick4: ovirt3:/gluster/ssd1_vmssd/brick
Brick5: ovirt1:/gluster/ssd1_vmssd/brick
Brick6: ovirt2:/gluster/ssd1_vmssd/brick (arbiter)
Options Reconfigured:
nfs.disable: on
transport.address-family: inet6
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
performance.low-prio-threads: 32
network.remote-dio: off
cluster.eager-lock: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
features.shard: on
user.cifs: off
storage.owner-uid: 36
storage.owner-gid: 36
features.shard-block-size: 128MB
performance.strict-o-direct: on
network.ping-timeout: 30
cluster.granular-entry-heal: enable

I would really appreciate some guidance on this to try to improve things because at this rate I will need to reconsider using GlusterFS altogether.<= =3D br>
Cheers,
Chris

--
Chris Boot
bootc(a)bootc.net
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ov= ir=3D t.org/mailman/listinfo/users

--_000_DM5PR01MB25069E435FC89BDDA5534AB2FFC40DM5PR01MB2506prod_-- --===============4483317745325526287== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" LS1fMDAwX0RNNVBSMDFNQjI1MDY5RTQzNUZDODlCRERBNTUzNEFCMkZGQzQwRE01UFIwMU1CMjUw NnByb2RfCkNvbnRlbnQtVHlwZTogdGV4dC9wbGFpbjsgY2hhcnNldD0idXMtYXNjaWkiCkNvbnRl bnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKCkhpLAoKCkNhbiB5b3UgcHV0 IHNvbWUgbnVtYmVycyA/IHdoYXQgdGVzdHMgYXJlIHlvdSBkb2luZyA/CgpJbSBydW5uaW5nIG9W aXJ0IHdpdGggR2x1c3RlciB3aXRob3V0IHBlcmZvcm1hbmNlIGlzc3VlcywgYnV0IGltIHJ1bm5p bmcgcmU9CnBsaWNhIDIgYWxsIFNTRHMuCgpHbHVzdGVyIGxvZ3MgbWlnaHQgaGVscCB0b28uCgoK LS0KClJlc3BlY3RmdWxseQpNYWhkaSBBLiBNYWhkaQoKX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX18KRnJvbTogdXNlcnMtYm91bmNlc0BvdmlydC5vcmcgPHVzZXJzLWJvdW5jZXNAb3Zp cnQub3JnPiBvbiBiZWhhbGYgb2YgQ2hyaXMgPQpCb290IDxib290Y0Bib290Yy5uZXQ+ClNlbnQ6 IE1vbmRheSwgSnVuZSAxOSwgMjAxNyA1OjQ2OjA4IFBNClRvOiBvVmlydCB1c2VycwpTdWJqZWN0 OiBbb3ZpcnQtdXNlcnNdIFZlcnkgcG9vciBHbHVzdGVyRlMgcGVyZm9ybWFuY2UKCkhpIGZvbGtz LAoKSSBoYXZlIDN4IHNlcnZlcnMgaW4gYSAiaHlwZXItY29udmVyZ2VkIiBvVmlydCA0LjEuMiAr IEdsdXN0ZXJGUyAzLjEwCmNvbmZpZ3VyYXRpb24uIE15IFZNcyBydW4gb2ZmIGEgcmVwbGljYSAz IGFyYml0ZXIgMSB2b2x1bWUgY29tcHJpc2VkIG9mCjYgYnJpY2tzLCB3aGljaCB0aGVtc2VsdmVz IGxpdmUgb24gdHdvIFNTRHMgaW4gZWFjaCBvZiB0aGUgc2VydmVycyAob25lCmJyaWNrIHBlciBT U0QpLiBUaGUgYnJpY2tzIGFyZSBYRlMgb24gTFZNIHRoaW4gdm9sdW1lcyBzdHJhaWdodCBvbnRv IHRoZQpTU0RzLiBDb25uZWN0aXZpdHkgaXMgMTBHIEV0aGVybmV0LgoKUGVyZm9ybWFuY2Ugd2l0 aGluIHRoZSBWTXMgaXMgcHJldHR5IHRlcnJpYmxlLiBJIGV4cGVyaWVuY2UgdmVyeSBsb3cKdGhy b3VnaHB1dCBhbmQgcmFuZG9tIElPIGlzIHJlYWxseSBiYWQ6IGl0IGZlZWxzIGxpa2UgYSBsYXRl bmN5IGlzc3VlLgpPbiBteSBvVmlydCBub2RlcyB0aGUgU1NEcyBhcmUgbm90IGdlbmVyYWxseSB2 ZXJ5IGJ1c3kuIFRoZSAxMEcgbmV0d29yawpzZWVtcyB0byBydW4gd2l0aG91dCBlcnJvcnMgKGlw ZXJmMyBnaXZlcyBiYW5kd2lkdGggbWVhc3VyZW1lbnRzIG9mID49M0QKOS4yMCBHYml0cy9zZWMg YmV0d2VlbiB0aGUgdGhyZWUgc2VydmVycykuCgpUbyBwdXQgdGhpcyBpbnRvIHBlcnNwZWN0aXZl OiBJIHdhcyBnZXR0aW5nIGJldHRlciBiZWhhdmlvdXIgZnJvbSBORlM0Cm9uIGEgZ2lnYWJpdCBj b25uZWN0aW9uIHRoYW4gSSBhbSB3aXRoIEdsdXN0ZXJGUyBvbiAxMEc6IHRoYXQgZG9lc24ndApm ZWVsIHJpZ2h0IGF0IGFsbC4KCk15IHZvbHVtZSBjb25maWd1cmF0aW9uIGxvb2tzIGxpa2UgdGhp czoKClZvbHVtZSBOYW1lOiB2bXNzZApUeXBlOiBEaXN0cmlidXRlZC1SZXBsaWNhdGUKVm9sdW1l IElEOiBkNWE1ZGRkMS1hMTQwLTRlMGQtYjUxNC03MDFjZmU0NjQ4NTMKU3RhdHVzOiBTdGFydGVk ClNuYXBzaG90IENvdW50OiAwCk51bWJlciBvZiBCcmlja3M6IDIgeCAoMiArIDEpID0zRCA2ClRy YW5zcG9ydC10eXBlOiB0Y3AKQnJpY2tzOgpCcmljazE6IG92aXJ0MzovZ2x1c3Rlci9zc2QwX3Zt c3NkL2JyaWNrCkJyaWNrMjogb3ZpcnQxOi9nbHVzdGVyL3NzZDBfdm1zc2QvYnJpY2sKQnJpY2sz OiBvdmlydDI6L2dsdXN0ZXIvc3NkMF92bXNzZC9icmljayAoYXJiaXRlcikKQnJpY2s0OiBvdmly dDM6L2dsdXN0ZXIvc3NkMV92bXNzZC9icmljawpCcmljazU6IG92aXJ0MTovZ2x1c3Rlci9zc2Qx X3Ztc3NkL2JyaWNrCkJyaWNrNjogb3ZpcnQyOi9nbHVzdGVyL3NzZDFfdm1zc2QvYnJpY2sgKGFy Yml0ZXIpCk9wdGlvbnMgUmVjb25maWd1cmVkOgpuZnMuZGlzYWJsZTogb24KdHJhbnNwb3J0LmFk ZHJlc3MtZmFtaWx5OiBpbmV0NgpwZXJmb3JtYW5jZS5xdWljay1yZWFkOiBvZmYKcGVyZm9ybWFu Y2UucmVhZC1haGVhZDogb2ZmCnBlcmZvcm1hbmNlLmlvLWNhY2hlOiBvZmYKcGVyZm9ybWFuY2Uu c3RhdC1wcmVmZXRjaDogb2ZmCnBlcmZvcm1hbmNlLmxvdy1wcmlvLXRocmVhZHM6IDMyCm5ldHdv cmsucmVtb3RlLWRpbzogb2ZmCmNsdXN0ZXIuZWFnZXItbG9jazogZW5hYmxlCmNsdXN0ZXIucXVv cnVtLXR5cGU6IGF1dG8KY2x1c3Rlci5zZXJ2ZXItcXVvcnVtLXR5cGU6IHNlcnZlcgpjbHVzdGVy LmRhdGEtc2VsZi1oZWFsLWFsZ29yaXRobTogZnVsbApjbHVzdGVyLmxvY2tpbmctc2NoZW1lOiBn cmFudWxhcgpjbHVzdGVyLnNoZC1tYXgtdGhyZWFkczogOApjbHVzdGVyLnNoZC13YWl0LXFsZW5n dGg6IDEwMDAwCmZlYXR1cmVzLnNoYXJkOiBvbgp1c2VyLmNpZnM6IG9mZgpzdG9yYWdlLm93bmVy LXVpZDogMzYKc3RvcmFnZS5vd25lci1naWQ6IDM2CmZlYXR1cmVzLnNoYXJkLWJsb2NrLXNpemU6 IDEyOE1CCnBlcmZvcm1hbmNlLnN0cmljdC1vLWRpcmVjdDogb24KbmV0d29yay5waW5nLXRpbWVv dXQ6IDMwCmNsdXN0ZXIuZ3JhbnVsYXItZW50cnktaGVhbDogZW5hYmxlCgpJIHdvdWxkIHJlYWxs eSBhcHByZWNpYXRlIHNvbWUgZ3VpZGFuY2Ugb24gdGhpcyB0byB0cnkgdG8gaW1wcm92ZSB0aGlu Z3MKYmVjYXVzZSBhdCB0aGlzIHJhdGUgSSB3aWxsIG5lZWQgdG8gcmVjb25zaWRlciB1c2luZyBH bHVzdGVyRlMgYWx0b2dldGhlci4KCkNoZWVycywKQ2hyaXMKCi0tCkNocmlzIEJvb3QKYm9vdGNA Ym9vdGMubmV0Cl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f ClVzZXJzIG1haWxpbmcgbGlzdApVc2Vyc0BvdmlydC5vcmcKaHR0cDovL2xpc3RzLm92aXJ0Lm9y Zy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzCgotLV8wMDBfRE01UFIwMU1CMjUwNjlFNDM1RkM4OUJE REE1NTM0QUIyRkZDNDBETTVQUjAxTUIyNTA2cHJvZF8KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWw7 IGNoYXJzZXQ9InVzLWFzY2lpIgpDb250ZW50LVRyYW5zZmVyLUVuY29kaW5nOiBxdW90ZWQtcHJp bnRhYmxlCgo8aHRtbD4KPGhlYWQ+CjxtZXRhIGh0dHAtZXF1aXY9M0QiQ29udGVudC1UeXBlIiBj b250ZW50PTNEInRleHQvaHRtbDsgY2hhcnNldD0zRHVzLWFzY2lpIj0KPgo8bWV0YSBuYW1lPTNE IkdlbmVyYXRvciIgY29udGVudD0zRCJNaWNyb3NvZnQgRXhjaGFuZ2UgU2VydmVyIj4KPCEtLSBj b252ZXJ0ZWQgZnJvbSB0ZXh0IC0tPjxzdHlsZT48IS0tIC5FbWFpbFF1b3RlIHsgbWFyZ2luLWxl ZnQ6IDFwdDsgcGFkPQpkaW5nLWxlZnQ6IDRwdDsgYm9yZGVyLWxlZnQ6ICM4MDAwMDAgMnB4IHNv bGlkOyB9IC0tPjwvc3R5bGU+CjwvaGVhZD4KPGJvZHk+CjxtZXRhIGNvbnRlbnQ9M0QidGV4dC9o dG1sOyBjaGFyc2V0PTNEVVRGLTgiPgo8c3R5bGUgdHlwZT0zRCJ0ZXh0L2NzcyIgc3R5bGU9M0Qi Ij4KPCEtLQpwCgl7bWFyZ2luLXRvcDowOwoJbWFyZ2luLWJvdHRvbTowfQotLT4KPC9zdHlsZT4K PGRpdiBkaXI9M0QibHRyIj4KPGRpdiBpZD0zRCJ4X2RpdnRhZ2RlZmF1bHR3cmFwcGVyIiBkaXI9 M0QibHRyIiBzdHlsZT0zRCJmb250LXNpemU6MTJwdDsgY29sPQpvcjojMDAwMDAwOyBmb250LWZh bWlseTpDYWxpYnJpLEhlbHZldGljYSxzYW5zLXNlcmlmIj4KPHA+SGksPC9wPgo8cD48YnI+Cjwv cD4KPHA+Q2FuIHlvdSBwdXQgc29tZSBudW1iZXJzID8gd2hhdCB0ZXN0cyBhcmUgeW91IGRvaW5n ID88L3A+CjxwPkltIHJ1bm5pbmcgb1ZpcnQgd2l0aCBHbHVzdGVyIHdpdGhvdXQgcGVyZm9ybWFu Y2UgaXNzdWVzLCBidXQgaW0gcnVubmluZz0KIHJlcGxpY2EgMiBhbGwgU1NEcy48L3A+CjxwPkds dXN0ZXIgbG9ncyBtaWdodCBoZWxwIHRvby48L3A+CjxwPjxicj4KPC9wPgo8ZGl2IGlkPTNEInhf U2lnbmF0dXJlIj48YnI+CjxkaXYgY2xhc3M9M0QieF9lY3htb3otc2lnbmF0dXJlIj4tLSA8YnI+ Cjxicj4KPGZvbnQgY29sb3I9M0QiIzMzNjZmZiI+PGZvbnQgY29sb3I9M0QiIzAwMDAwMCI+UmVz cGVjdGZ1bGx5PGI+PGJyPgo8L2I+PGI+TWFoZGkgQS4gTWFoZGk8L2I+PC9mb250PjwvZm9udD48 Zm9udCBjb2xvcj0zRCIjMzM2NmZmIj48YnI+Cjxicj4KPC9mb250Pjxmb250IGNvbG9yPTNEIiMz MzY2ZmYiPjwvZm9udD48L2Rpdj4KPC9kaXY+CjwvZGl2Pgo8aHIgdGFiaW5kZXg9M0QiLTEiIHN0 eWxlPTNEImRpc3BsYXk6aW5saW5lLWJsb2NrOyB3aWR0aDo5OCUiPgo8ZGl2IGlkPTNEInhfZGl2 UnBseUZ3ZE1zZyIgZGlyPTNEImx0ciI+PGZvbnQgZmFjZT0zRCJDYWxpYnJpLCBzYW5zLXNlcmlm IiA9CmNvbG9yPTNEIiMwMDAwMDAiIHN0eWxlPTNEImZvbnQtc2l6ZToxMXB0Ij48Yj5Gcm9tOjwv Yj4gdXNlcnMtYm91bmNlc0BvdmlydD0KLm9yZyAmbHQ7dXNlcnMtYm91bmNlc0BvdmlydC5vcmcm Z3Q7IG9uIGJlaGFsZiBvZiBDaHJpcyBCb290ICZsdDtib290Y0Bib290PQpjLm5ldCZndDs8YnI+ CjxiPlNlbnQ6PC9iPiBNb25kYXksIEp1bmUgMTksIDIwMTcgNTo0NjowOCBQTTxicj4KPGI+VG86 PC9iPiBvVmlydCB1c2Vyczxicj4KPGI+U3ViamVjdDo8L2I+IFtvdmlydC11c2Vyc10gVmVyeSBw b29yIEdsdXN0ZXJGUyBwZXJmb3JtYW5jZTwvZm9udD4KPGRpdj4mbmJzcDs8L2Rpdj4KPC9kaXY+ CjwvZGl2Pgo8Zm9udCBzaXplPTNEIjIiPjxzcGFuIHN0eWxlPTNEImZvbnQtc2l6ZToxMHB0OyI+ CjxkaXYgY2xhc3M9M0QiUGxhaW5UZXh0Ij5IaSBmb2xrcyw8YnI+Cjxicj4KSSBoYXZlIDN4IHNl cnZlcnMgaW4gYSAmcXVvdDtoeXBlci1jb252ZXJnZWQmcXVvdDsgb1ZpcnQgNC4xLjIgJiM0Mzsg R2x1c3RlPQpyRlMgMy4xMDxicj4KY29uZmlndXJhdGlvbi4gTXkgVk1zIHJ1biBvZmYgYSByZXBs aWNhIDMgYXJiaXRlciAxIHZvbHVtZSBjb21wcmlzZWQgb2Y8YnI+CjYgYnJpY2tzLCB3aGljaCB0 aGVtc2VsdmVzIGxpdmUgb24gdHdvIFNTRHMgaW4gZWFjaCBvZiB0aGUgc2VydmVycyAob25lPGJy PgpicmljayBwZXIgU1NEKS4gVGhlIGJyaWNrcyBhcmUgWEZTIG9uIExWTSB0aGluIHZvbHVtZXMg c3RyYWlnaHQgb250byB0aGU8YnI9Cj4KU1NEcy4gQ29ubmVjdGl2aXR5IGlzIDEwRyBFdGhlcm5l dC48YnI+Cjxicj4KUGVyZm9ybWFuY2Ugd2l0aGluIHRoZSBWTXMgaXMgcHJldHR5IHRlcnJpYmxl LiBJIGV4cGVyaWVuY2UgdmVyeSBsb3c8YnI+CnRocm91Z2hwdXQgYW5kIHJhbmRvbSBJTyBpcyBy ZWFsbHkgYmFkOiBpdCBmZWVscyBsaWtlIGEgbGF0ZW5jeSBpc3N1ZS48YnI+Ck9uIG15IG9WaXJ0 IG5vZGVzIHRoZSBTU0RzIGFyZSBub3QgZ2VuZXJhbGx5IHZlcnkgYnVzeS4gVGhlIDEwRyBuZXR3 b3JrPGJyPgpzZWVtcyB0byBydW4gd2l0aG91dCBlcnJvcnMgKGlwZXJmMyBnaXZlcyBiYW5kd2lk dGggbWVhc3VyZW1lbnRzIG9mICZndDs9M0Q9Cjxicj4KOS4yMCBHYml0cy9zZWMgYmV0d2VlbiB0 aGUgdGhyZWUgc2VydmVycykuPGJyPgo8YnI+ClRvIHB1dCB0aGlzIGludG8gcGVyc3BlY3RpdmU6 IEkgd2FzIGdldHRpbmcgYmV0dGVyIGJlaGF2aW91ciBmcm9tIE5GUzQ8YnI+Cm9uIGEgZ2lnYWJp dCBjb25uZWN0aW9uIHRoYW4gSSBhbSB3aXRoIEdsdXN0ZXJGUyBvbiAxMEc6IHRoYXQgZG9lc24n dDxicj4KZmVlbCByaWdodCBhdCBhbGwuPGJyPgo8YnI+Ck15IHZvbHVtZSBjb25maWd1cmF0aW9u IGxvb2tzIGxpa2UgdGhpczo8YnI+Cjxicj4KVm9sdW1lIE5hbWU6IHZtc3NkPGJyPgpUeXBlOiBE aXN0cmlidXRlZC1SZXBsaWNhdGU8YnI+ClZvbHVtZSBJRDogZDVhNWRkZDEtYTE0MC00ZTBkLWI1 MTQtNzAxY2ZlNDY0ODUzPGJyPgpTdGF0dXM6IFN0YXJ0ZWQ8YnI+ClNuYXBzaG90IENvdW50OiAw PGJyPgpOdW1iZXIgb2YgQnJpY2tzOiAyIHggKDIgJiM0MzsgMSkgPTNEIDY8YnI+ClRyYW5zcG9y dC10eXBlOiB0Y3A8YnI+CkJyaWNrczo8YnI+CkJyaWNrMTogb3ZpcnQzOi9nbHVzdGVyL3NzZDBf dm1zc2QvYnJpY2s8YnI+CkJyaWNrMjogb3ZpcnQxOi9nbHVzdGVyL3NzZDBfdm1zc2QvYnJpY2s8 YnI+CkJyaWNrMzogb3ZpcnQyOi9nbHVzdGVyL3NzZDBfdm1zc2QvYnJpY2sgKGFyYml0ZXIpPGJy PgpCcmljazQ6IG92aXJ0MzovZ2x1c3Rlci9zc2QxX3Ztc3NkL2JyaWNrPGJyPgpCcmljazU6IG92 aXJ0MTovZ2x1c3Rlci9zc2QxX3Ztc3NkL2JyaWNrPGJyPgpCcmljazY6IG92aXJ0MjovZ2x1c3Rl ci9zc2QxX3Ztc3NkL2JyaWNrIChhcmJpdGVyKTxicj4KT3B0aW9ucyBSZWNvbmZpZ3VyZWQ6PGJy PgpuZnMuZGlzYWJsZTogb248YnI+CnRyYW5zcG9ydC5hZGRyZXNzLWZhbWlseTogaW5ldDY8YnI+ CnBlcmZvcm1hbmNlLnF1aWNrLXJlYWQ6IG9mZjxicj4KcGVyZm9ybWFuY2UucmVhZC1haGVhZDog b2ZmPGJyPgpwZXJmb3JtYW5jZS5pby1jYWNoZTogb2ZmPGJyPgpwZXJmb3JtYW5jZS5zdGF0LXBy ZWZldGNoOiBvZmY8YnI+CnBlcmZvcm1hbmNlLmxvdy1wcmlvLXRocmVhZHM6IDMyPGJyPgpuZXR3 b3JrLnJlbW90ZS1kaW86IG9mZjxicj4KY2x1c3Rlci5lYWdlci1sb2NrOiBlbmFibGU8YnI+CmNs dXN0ZXIucXVvcnVtLXR5cGU6IGF1dG88YnI+CmNsdXN0ZXIuc2VydmVyLXF1b3J1bS10eXBlOiBz ZXJ2ZXI8YnI+CmNsdXN0ZXIuZGF0YS1zZWxmLWhlYWwtYWxnb3JpdGhtOiBmdWxsPGJyPgpjbHVz dGVyLmxvY2tpbmctc2NoZW1lOiBncmFudWxhcjxicj4KY2x1c3Rlci5zaGQtbWF4LXRocmVhZHM6 IDg8YnI+CmNsdXN0ZXIuc2hkLXdhaXQtcWxlbmd0aDogMTAwMDA8YnI+CmZlYXR1cmVzLnNoYXJk OiBvbjxicj4KdXNlci5jaWZzOiBvZmY8YnI+CnN0b3JhZ2Uub3duZXItdWlkOiAzNjxicj4Kc3Rv cmFnZS5vd25lci1naWQ6IDM2PGJyPgpmZWF0dXJlcy5zaGFyZC1ibG9jay1zaXplOiAxMjhNQjxi cj4KcGVyZm9ybWFuY2Uuc3RyaWN0LW8tZGlyZWN0OiBvbjxicj4KbmV0d29yay5waW5nLXRpbWVv dXQ6IDMwPGJyPgpjbHVzdGVyLmdyYW51bGFyLWVudHJ5LWhlYWw6IGVuYWJsZTxicj4KPGJyPgpJ IHdvdWxkIHJlYWxseSBhcHByZWNpYXRlIHNvbWUgZ3VpZGFuY2Ugb24gdGhpcyB0byB0cnkgdG8g aW1wcm92ZSB0aGluZ3M8YnI9Cj4KYmVjYXVzZSBhdCB0aGlzIHJhdGUgSSB3aWxsIG5lZWQgdG8g cmVjb25zaWRlciB1c2luZyBHbHVzdGVyRlMgYWx0b2dldGhlci48PQpicj4KPGJyPgpDaGVlcnMs PGJyPgpDaHJpczxicj4KPGJyPgotLSA8YnI+CkNocmlzIEJvb3Q8YnI+CmJvb3RjQGJvb3RjLm5l dDxicj4KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188YnI+ ClVzZXJzIG1haWxpbmcgbGlzdDxicj4KVXNlcnNAb3ZpcnQub3JnPGJyPgo8YSBocmVmPTNEImh0 dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycyI+aHR0cDovL2xpc3Rz Lm92aXI9CnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnM8L2E+PGJyPgo8L2Rpdj4KPC9zcGFu PjwvZm9udD4KPC9ib2R5Pgo8L2h0bWw+CgotLV8wMDBfRE01UFIwMU1CMjUwNjlFNDM1RkM4OUJE REE1NTM0QUIyRkZDNDBETTVQUjAxTUIyNTA2cHJvZF8tLQo= --===============4483317745325526287==--