From kripper at imatronix.cl Thu May 7 05:09:49 2015 Content-Type: multipart/mixed; boundary="===============1193919570093340470==" MIME-Version: 1.0 From: Christopher Pereira To: devel at ovirt.org Subject: [ovirt-devel] Stale "Make Template" tasks and locked templates + SEEK_HOLE optimization for copying images Date: Thu, 07 May 2015 06:09:46 -0300 Message-ID: <554B2BDA.6040005@imatronix.cl> --===============1193919570093340470== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------080700090609090405050009 Content-Type: text/plain; charset=3Dutf-8; format=3Dflowed Content-Transfer-Encoding: 7bit Hi, I'm testing nightly builds. In general, my experience after reinstalling from scratch and having = solved my storage issues is very good. Today I tried to "Make Template" and had some UX problems I would like = to share with you. I would also discuss a possible optimization for copying images using = SEEK_HOLE. 1) "Make Template" UX and performance problems: Source disk and destination disk (template) were on the same StorageDomain. By accident (probably something very common) the SPM was set on an = external host (that was not hosting this StorageDomain), so the whole = image data went out and back to the same source machine. This obviously takes very long (hours), while copying the sparse files = directly only takes about 10 [s] with the below optimization. While making the templates, I believe I restarted VDSM or rebooted the = SPM so the tasks went stale. My fault again. I was able to remove the stale tasks in Engine by suspending VM's, = stopping VDSM to set the host as non responding and using "confirm host = has been rebooted". Setting the host in maintenance to confirm it was rebooted was not = possible because it had async. running tasks. Aren't this tasks PIDs being checked to see if they are still alive? 2) I saw that VDSM was running " /usr/bin/qemu-img convert": In this case, I believe it is enough to just copy the images instead of = converting them. I made some tests and found that using "cp --sparse=3Dalways" is the best = way to copy images to gluster mounts because it is faster and because = the resulting files are still sparse ('du' reports exactly the same sizes). But I also discovered a bottleneck. When copying sparse files (e.g. a 2 TB sparse image that only uses 800 = MB in disk, a common scenario when we create templates from fresh = installs) the 'cp' command behaves differently depending on if we are = reading from a gluster mount or from a filesystem supporting SEEK_HOLE = (available in kernels >=3D 3.1): a) If we read from a gluster mount, 'cp' reads the 2 TB of zeros, even = when it only writes the non-zeros (iotop shows the 'cp' process reading = those 2 TB). Only sparse writing is optimized. b) If we read from a SEEK_HOLE supporting filesystem (ext, xfs, etc), = 'cp' only reads the non-zero content, thus reading and writing takes = like 10 [s] instead of hours. It seems gluster is not using SEEK_HOLE for reading (?!). Considering the source image is not being modified during the "Make = Template" process and we have access to the gluster bricks, it is = possible to 'cp' the source image directly from the bricks (on top of = the SEEK_HOLE supporting FS) instead of reading from the gluster mount. The difference is really impressive (seconds instead of hours). I tested it cloning some VM's and it works. Am I missing something? Maybe a gluster optimization to enable SEEK_HOLE support on gluster mounts? --------------080700090609090405050009 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 7bit Hi,

I'm testing nightly builds.
In general, my experience after reinstalling from scratch and having solved my storage issues is very good.
Today I tried to "Make Template" and had some UX problems I would like to share with you.
I would also discuss a possible optimization for copying images using SEEK_HOLE.

1) "Make Template" UX and performance problems:

Source disk and destination disk (template) were on the same StorageDomain.
By accident (probably something very common) the SPM was set on an external host (that was not hosting this StorageDomain), so the whole image data went out and back to the same source machine.
This obviously takes very long (hours), while copying the sparse files directly only takes about 10 [s] with the below optimization.

While making the templates, I believe I restarted VDSM or rebooted the SPM so the tasks went stale. My fault again.
I was able to remove the stale tasks in Engine by suspending VM's, stopping VDSM to set the host as non responding and using "confirm host has been rebooted".
Setting the host in maintenance to confirm it was rebooted was not possible because it had async. running tasks.
Aren't this tasks PIDs being checked to see if they are still alive?

2) I saw that VDSM was running " /usr/bin/qemu-img convert":

In this case, I believe it is enough to just copy the images instead of converting them.
I made some tests and found that using "cp --sparse=3Dalways" is the best way to copy images to glust= er mounts because it is faster and because the resulting files are still sparse ('du' reports exactly the same sizes).
But I also discovered a bottleneck.
When copying sparse files (e.g. a 2 TB sparse image that only uses 800 MB in disk, a common scenario when we create templates from fresh installs) the 'cp' command behaves differently depending on if we are reading from a gluster mount or from a filesystem supporting SEEK_HOLE (available in kernels >=3D 3.1):

a) If we read from a gluster mount, 'cp' reads the 2 TB of zeros, even when it only writes the non-zeros (iotop shows the 'cp' process reading those 2 TB). Only sparse writing is optimized.

b) If we read from a SEEK_HOLE supporting filesystem (ext, xfs, etc), 'cp' only reads the non-zero content, thus reading and writing takes like 10 [s] instead of hours.

It seems gluster is not using SEEK_HOLE for reading (?!).

Considering the source image is not being modified during the "Make Template" process and we have access to the gluster bricks, it is possible to 'cp' the source image directly from the bricks (on top of the SEEK_HOLE supporting FS) instead of reading from the gluster mount.

The difference is really impressive (seconds instead of hours).
I tested it cloning some VM's and it works.

Am I missing something?
Maybe a gluster optimization to enable SEEK_HOLE support on gluster mounts?

--------------080700090609090405050009-- --===============1193919570093340470== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS0wODA3MDAwOTA2MDkwOTA0MDUwNTAwMDkKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04OyBmb3JtYXQ9Zmxvd2VkCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDdiaXQK CkhpLAoKSSdtIHRlc3RpbmcgbmlnaHRseSBidWlsZHMuCkluIGdlbmVyYWwsIG15IGV4cGVyaWVu Y2UgYWZ0ZXIgcmVpbnN0YWxsaW5nIGZyb20gc2NyYXRjaCBhbmQgaGF2aW5nIApzb2x2ZWQgbXkg c3RvcmFnZSBpc3N1ZXMgaXMgdmVyeSBnb29kLgpUb2RheSBJIHRyaWVkIHRvICJNYWtlIFRlbXBs YXRlIiBhbmQgaGFkIHNvbWUgVVggcHJvYmxlbXMgSSB3b3VsZCBsaWtlIAp0byBzaGFyZSB3aXRo IHlvdS4KSSB3b3VsZCBhbHNvIGRpc2N1c3MgYSBwb3NzaWJsZSBvcHRpbWl6YXRpb24gZm9yIGNv cHlpbmcgaW1hZ2VzIHVzaW5nIApTRUVLX0hPTEUuCgoxKSAiTWFrZSBUZW1wbGF0ZSIgVVggYW5k IHBlcmZvcm1hbmNlIHByb2JsZW1zOgoKU291cmNlIGRpc2sgYW5kIGRlc3RpbmF0aW9uIGRpc2sg KHRlbXBsYXRlKSB3ZXJlIG9uIHRoZSBzYW1lIFN0b3JhZ2VEb21haW4uCkJ5IGFjY2lkZW50IChw cm9iYWJseSBzb21ldGhpbmcgdmVyeSBjb21tb24pIHRoZSBTUE0gd2FzIHNldCBvbiBhbiAKZXh0 ZXJuYWwgaG9zdCAodGhhdCB3YXMgbm90IGhvc3RpbmcgdGhpcyBTdG9yYWdlRG9tYWluKSwgc28g dGhlIHdob2xlIAppbWFnZSBkYXRhIHdlbnQgb3V0IGFuZCBiYWNrIHRvIHRoZSBzYW1lIHNvdXJj ZSBtYWNoaW5lLgpUaGlzIG9idmlvdXNseSB0YWtlcyB2ZXJ5IGxvbmcgKGhvdXJzKSwgd2hpbGUg Y29weWluZyB0aGUgc3BhcnNlIGZpbGVzIApkaXJlY3RseSBvbmx5IHRha2VzIGFib3V0IDEwIFtz XSB3aXRoIHRoZSBiZWxvdyBvcHRpbWl6YXRpb24uCgpXaGlsZSBtYWtpbmcgdGhlIHRlbXBsYXRl cywgSSBiZWxpZXZlIEkgcmVzdGFydGVkIFZEU00gb3IgcmVib290ZWQgdGhlIApTUE0gc28gdGhl IHRhc2tzIHdlbnQgc3RhbGUuIE15IGZhdWx0IGFnYWluLgpJIHdhcyBhYmxlIHRvIHJlbW92ZSB0 aGUgc3RhbGUgdGFza3MgaW4gRW5naW5lIGJ5IHN1c3BlbmRpbmcgVk0ncywgCnN0b3BwaW5nIFZE U00gdG8gc2V0IHRoZSBob3N0IGFzIG5vbiByZXNwb25kaW5nIGFuZCB1c2luZyAiY29uZmlybSBo b3N0IApoYXMgYmVlbiByZWJvb3RlZCIuClNldHRpbmcgdGhlIGhvc3QgaW4gbWFpbnRlbmFuY2Ug dG8gY29uZmlybSBpdCB3YXMgcmVib290ZWQgd2FzIG5vdCAKcG9zc2libGUgYmVjYXVzZSBpdCBo YWQgYXN5bmMuIHJ1bm5pbmcgdGFza3MuCkFyZW4ndCB0aGlzIHRhc2tzIFBJRHMgYmVpbmcgY2hl Y2tlZCB0byBzZWUgaWYgdGhleSBhcmUgc3RpbGwgYWxpdmU/CgoyKSBJIHNhdyB0aGF0IFZEU00g d2FzIHJ1bm5pbmcgIiAvdXNyL2Jpbi9xZW11LWltZyBjb252ZXJ0IjoKCkluIHRoaXMgY2FzZSwg SSBiZWxpZXZlIGl0IGlzIGVub3VnaCB0byBqdXN0IGNvcHkgdGhlIGltYWdlcyBpbnN0ZWFkIG9m IApjb252ZXJ0aW5nIHRoZW0uCkkgbWFkZSBzb21lIHRlc3RzIGFuZCBmb3VuZCB0aGF0IHVzaW5n ICJjcCAtLXNwYXJzZT1hbHdheXMiIGlzIHRoZSBiZXN0IAp3YXkgdG8gY29weSBpbWFnZXMgdG8g Z2x1c3RlciBtb3VudHMgYmVjYXVzZSBpdCBpcyBmYXN0ZXIgYW5kIGJlY2F1c2UgCnRoZSByZXN1 bHRpbmcgZmlsZXMgYXJlIHN0aWxsIHNwYXJzZSAoJ2R1JyByZXBvcnRzIGV4YWN0bHkgdGhlIHNh bWUgc2l6ZXMpLgpCdXQgSSBhbHNvIGRpc2NvdmVyZWQgYSBib3R0bGVuZWNrLgpXaGVuIGNvcHlp bmcgc3BhcnNlIGZpbGVzIChlLmcuIGEgMiBUQiBzcGFyc2UgaW1hZ2UgdGhhdCBvbmx5IHVzZXMg ODAwIApNQiBpbiBkaXNrLCBhIGNvbW1vbiBzY2VuYXJpbyB3aGVuIHdlIGNyZWF0ZSB0ZW1wbGF0 ZXMgZnJvbSBmcmVzaCAKaW5zdGFsbHMpIHRoZSAnY3AnIGNvbW1hbmQgYmVoYXZlcyBkaWZmZXJl bnRseSBkZXBlbmRpbmcgb24gaWYgd2UgYXJlIApyZWFkaW5nIGZyb20gYSBnbHVzdGVyIG1vdW50 IG9yIGZyb20gYSBmaWxlc3lzdGVtIHN1cHBvcnRpbmcgU0VFS19IT0xFIAooYXZhaWxhYmxlIGlu IGtlcm5lbHMgPj0gMy4xKToKCmEpIElmIHdlIHJlYWQgZnJvbSBhIGdsdXN0ZXIgbW91bnQsICdj cCcgcmVhZHMgdGhlIDIgVEIgb2YgemVyb3MsIGV2ZW4gCndoZW4gaXQgb25seSB3cml0ZXMgdGhl IG5vbi16ZXJvcyAoaW90b3Agc2hvd3MgdGhlICdjcCcgcHJvY2VzcyByZWFkaW5nIAp0aG9zZSAy IFRCKS4gT25seSBzcGFyc2Ugd3JpdGluZyBpcyBvcHRpbWl6ZWQuCgpiKSBJZiB3ZSByZWFkIGZy b20gYSBTRUVLX0hPTEUgc3VwcG9ydGluZyBmaWxlc3lzdGVtIChleHQsIHhmcywgZXRjKSwgCidj cCcgb25seSByZWFkcyB0aGUgbm9uLXplcm8gY29udGVudCwgdGh1cyByZWFkaW5nIGFuZCB3cml0 aW5nIHRha2VzIApsaWtlIDEwIFtzXSBpbnN0ZWFkIG9mIGhvdXJzLgoKSXQgc2VlbXMgZ2x1c3Rl ciBpcyBub3QgdXNpbmcgU0VFS19IT0xFIGZvciByZWFkaW5nICg/ISkuCgpDb25zaWRlcmluZyB0 aGUgc291cmNlIGltYWdlIGlzIG5vdCBiZWluZyBtb2RpZmllZCBkdXJpbmcgdGhlICJNYWtlIApU ZW1wbGF0ZSIgcHJvY2VzcyBhbmQgd2UgaGF2ZSBhY2Nlc3MgdG8gdGhlIGdsdXN0ZXIgYnJpY2tz LCBpdCBpcyAKcG9zc2libGUgdG8gJ2NwJyB0aGUgc291cmNlIGltYWdlIGRpcmVjdGx5IGZyb20g dGhlIGJyaWNrcyAob24gdG9wIG9mIAp0aGUgU0VFS19IT0xFIHN1cHBvcnRpbmcgRlMpIGluc3Rl YWQgb2YgcmVhZGluZyBmcm9tIHRoZSBnbHVzdGVyIG1vdW50LgoKVGhlIGRpZmZlcmVuY2UgaXMg cmVhbGx5IGltcHJlc3NpdmUgKHNlY29uZHMgaW5zdGVhZCBvZiBob3VycykuCkkgdGVzdGVkIGl0 IGNsb25pbmcgc29tZSBWTSdzIGFuZCBpdCB3b3Jrcy4KCkFtIEkgbWlzc2luZyBzb21ldGhpbmc/ Ck1heWJlIGEgZ2x1c3RlciBvcHRpbWl6YXRpb24gdG8gZW5hYmxlIFNFRUtfSE9MRSBzdXBwb3J0 IG9uIGdsdXN0ZXIgbW91bnRzPwoKCi0tLS0tLS0tLS0tLS0tMDgwNzAwMDkwNjA5MDkwNDA1MDUw MDA5CkNvbnRlbnQtVHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNm ZXItRW5jb2Rpbmc6IDdiaXQKCjxodG1sPgogIDxoZWFkPgoKICAgIDxtZXRhIGh0dHAtZXF1aXY9 ImNvbnRlbnQtdHlwZSIgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4KICA8L2hl YWQ+CiAgPGJvZHkgYmdjb2xvcj0iI0ZGRkZGRiIgdGV4dD0iIzAwMDAwMCI+CiAgICBIaSw8YnI+ CiAgICA8YnI+CiAgICBJJ20gdGVzdGluZyBuaWdodGx5IGJ1aWxkcy48YnI+CiAgICBJbiBnZW5l cmFsLCBteSBleHBlcmllbmNlIGFmdGVyIHJlaW5zdGFsbGluZyBmcm9tIHNjcmF0Y2ggYW5kIGhh dmluZwogICAgc29sdmVkIG15IHN0b3JhZ2UgaXNzdWVzIGlzIHZlcnkgZ29vZC48YnI+CiAgICBU b2RheSBJIHRyaWVkIHRvICJNYWtlIFRlbXBsYXRlIiBhbmQgaGFkIHNvbWUgVVggcHJvYmxlbXMg SSB3b3VsZAogICAgbGlrZSB0byBzaGFyZSB3aXRoIHlvdS48YnI+CiAgICBJIHdvdWxkIGFsc28g ZGlzY3VzcyBhIHBvc3NpYmxlIG9wdGltaXphdGlvbiBmb3IgY29weWluZyBpbWFnZXMKICAgIHVz aW5nIFNFRUtfSE9MRS48YnI+CiAgICA8YnI+CiAgICAxKSAiTWFrZSBUZW1wbGF0ZSIgVVggYW5k IHBlcmZvcm1hbmNlIHByb2JsZW1zOjxicj4KICAgIDxicj4KICAgIFNvdXJjZSBkaXNrIGFuZCBk ZXN0aW5hdGlvbiBkaXNrICh0ZW1wbGF0ZSkgd2VyZSBvbiB0aGUgc2FtZQogICAgU3RvcmFnZURv bWFpbi48YnI+CiAgICBCeSBhY2NpZGVudCAocHJvYmFibHkgc29tZXRoaW5nIHZlcnkgY29tbW9u KSB0aGUgU1BNIHdhcyBzZXQgb24gYW4KICAgIGV4dGVybmFsIGhvc3QgKHRoYXQgd2FzIG5vdCBo b3N0aW5nIHRoaXMgU3RvcmFnZURvbWFpbiksIHNvIHRoZQogICAgd2hvbGUgaW1hZ2UgZGF0YSB3 ZW50IG91dCBhbmQgYmFjayB0byB0aGUgc2FtZSBzb3VyY2UgbWFjaGluZS48YnI+CiAgICBUaGlz IG9idmlvdXNseSB0YWtlcyB2ZXJ5IGxvbmcgKGhvdXJzKSwgd2hpbGUgY29weWluZyB0aGUgc3Bh cnNlCiAgICBmaWxlcyBkaXJlY3RseSBvbmx5IHRha2VzIGFib3V0IDEwIFtzXSB3aXRoIHRoZSBi ZWxvdyBvcHRpbWl6YXRpb24uPGJyPgogICAgPGJyPgogICAgV2hpbGUgbWFraW5nIHRoZSB0ZW1w bGF0ZXMsIEkgYmVsaWV2ZSBJIHJlc3RhcnRlZCBWRFNNIG9yIHJlYm9vdGVkCiAgICB0aGUgU1BN IHNvIHRoZSB0YXNrcyB3ZW50IHN0YWxlLiBNeSBmYXVsdCBhZ2Fpbi48YnI+CiAgICA8c3BhbiBz dHlsZT0iY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6ICdBcmlhbCBVbmljb2RlIE1T JywKICAgICAgQXJpYWwsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogc21hbGw7IGZvbnQtc3R5bGU6 IG5vcm1hbDsKICAgICAgZm9udC12YXJpYW50OiBub3JtYWw7IGZvbnQtd2VpZ2h0OiBub3JtYWw7 IGxldHRlci1zcGFjaW5nOiBub3JtYWw7CiAgICAgIGxpbmUtaGVpZ2h0OiAyMS42NjY2Njc5Mzgy MzI0cHg7IG9ycGhhbnM6IGF1dG87IHRleHQtYWxpZ246IHN0YXJ0OwogICAgICB0ZXh0LWluZGVu dDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsKICAgICAg d2lkb3dzOiAxOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDog MHB4OwogICAgICBkaXNwbGF5OiBpbmxpbmUgIWltcG9ydGFudDsgZmxvYXQ6IG5vbmU7IGJhY2tn cm91bmQtY29sb3I6CiAgICAgIHJnYigyNTUsIDI1NSwgMjU1KTsiPkkgd2FzIGFibGUgdG8gcmVt b3ZlIHRoZSBzdGFsZSB0YXNrcyBpbgogICAgICBFbmdpbmUgYnkgc3VzcGVuZGluZyBWTSdzLCBz dG9wcGluZyBWRFNNIHRvIHNldCB0aGUgaG9zdCBhcyBub24KICAgICAgcmVzcG9uZGluZyBhbmQg dXNpbmcgImNvbmZpcm0gaG9zdCBoYXMgYmVlbiByZWJvb3RlZCIuPGJyPgogICAgICBTZXR0aW5n IHRoZSBob3N0IGluIG1haW50ZW5hbmNlIHRvIGNvbmZpcm0gaXQgd2FzIHJlYm9vdGVkIHdhcyBu b3QKICAgICAgcG9zc2libGUgYmVjYXVzZSBpdCBoYWQgYXN5bmMuIHJ1bm5pbmcgdGFza3MuPGJy PgogICAgICBBcmVuJ3QgdGhpcyB0YXNrcyBQSURzIGJlaW5nIGNoZWNrZWQgdG8gc2VlIGlmIHRo ZXkgYXJlIHN0aWxsCiAgICAgIGFsaXZlPzxicj4KICAgICAgPGJyPgogICAgICAyKSBJIHNhdyB0 aGF0IFZEU00gd2FzIHJ1bm5pbmcgIiAvdXNyL2Jpbi9xZW11LWltZyBjb252ZXJ0Ijo8YnI+CiAg ICAgIDxicj4KICAgICAgSW4gdGhpcyBjYXNlLCBJIGJlbGlldmUgaXQgaXMgZW5vdWdoIHRvIGp1 c3QgY29weSB0aGUgaW1hZ2VzCiAgICAgIGluc3RlYWQgb2YgY29udmVydGluZyB0aGVtPGJpZz4u PC9iaWc+PGJyPgogICAgICBJIG1hZGUgc29tZSB0ZXN0cyBhbmQgZm91bmQgdGhhdCB1c2luZyAi Y3AgLS1zcGFyc2U9YWx3YXlzIiBpcyB0aGUKICAgICAgYmVzdCB3YXkgdG8gY29weSBpbWFnZXMg PC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjogcmdiKDAsIDAsIDApOwogICAgICBmb250LWZhbWls eTogJ0FyaWFsIFVuaWNvZGUgTVMnLCBBcmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOgogICAg ICBzbWFsbDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQ6IG5vcm1hbDsgZm9udC13 ZWlnaHQ6CiAgICAgIG5vcm1hbDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgbGluZS1oZWlnaHQ6 IDIxLjY2NjY2NzkzODIzMjRweDsKICAgICAgb3JwaGFuczogYXV0bzsgdGV4dC1hbGlnbjogc3Rh cnQ7IHRleHQtaW5kZW50OiAwcHg7CiAgICAgIHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1z cGFjZTogbm9ybWFsOyB3aWRvd3M6IDE7CiAgICAgIHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0 LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IGRpc3BsYXk6IGlubGluZQogICAgICAhaW1wb3J0YW50 OyBmbG9hdDogbm9uZTsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDI1NSwgMjU1LCAyNTUpOyI+PHNw YW4KICAgICAgICBzdHlsZT0iY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6ICdBcmlh bCBVbmljb2RlIE1TJywKICAgICAgICBBcmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiBzbWFs bDsgZm9udC1zdHlsZTogbm9ybWFsOwogICAgICAgIGZvbnQtdmFyaWFudDogbm9ybWFsOyBmb250 LXdlaWdodDogbm9ybWFsOyBsZXR0ZXItc3BhY2luZzoKICAgICAgICBub3JtYWw7IGxpbmUtaGVp Z2h0OiAyMS42NjY2Njc5MzgyMzI0cHg7IG9ycGhhbnM6IGF1dG87CiAgICAgICAgdGV4dC1hbGln bjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiBub25lOwogICAgICAg IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdpZG93czogMTsgd29yZC1zcGFjaW5nOiAwcHg7CiAgICAg ICAgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyBkaXNwbGF5OiBpbmxpbmUgIWltcG9y dGFudDsKICAgICAgICBmbG9hdDogbm9uZTsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDI1NSwgMjU1 LCAyNTUpOyI+dG8gPC9zcGFuPmdsdXN0ZXIKICAgICAgbW91bnRzIDwvc3Bhbj48c3BhbiBzdHls ZT0iY29sb3I6IHJnYigwLCAwLCAwKTsgZm9udC1mYW1pbHk6CiAgICAgICdBcmlhbCBVbmljb2Rl IE1TJywgQXJpYWwsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogc21hbGw7CiAgICAgIGZvbnQtc3R5 bGU6IG5vcm1hbDsgZm9udC12YXJpYW50OiBub3JtYWw7IGZvbnQtd2VpZ2h0OiBub3JtYWw7CiAg ICAgIGxldHRlci1zcGFjaW5nOiBub3JtYWw7IGxpbmUtaGVpZ2h0OiAyMS42NjY2Njc5MzgyMzI0 cHg7IG9ycGhhbnM6CiAgICAgIGF1dG87IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDog MHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsKICAgICAgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd2lk b3dzOiAxOyB3b3JkLXNwYWNpbmc6IDBweDsKICAgICAgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0 aDogMHB4OyBkaXNwbGF5OiBpbmxpbmUgIWltcG9ydGFudDsgZmxvYXQ6CiAgICAgIG5vbmU7IGJh Y2tncm91bmQtY29sb3I6IHJnYigyNTUsIDI1NSwgMjU1KTsiPjxzcGFuIHN0eWxlPSJjb2xvcjoK ICAgICAgICByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiAnQXJpYWwgVW5pY29kZSBNUycsIEFy aWFsLAogICAgICAgIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogc21hbGw7IGZvbnQtc3R5bGU6IG5v cm1hbDsgZm9udC12YXJpYW50OgogICAgICAgIG5vcm1hbDsgZm9udC13ZWlnaHQ6IG5vcm1hbDsg bGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsKICAgICAgICBsaW5lLWhlaWdodDogMjEuNjY2NjY3OTM4 MjMyNHB4OyBvcnBoYW5zOiBhdXRvOyB0ZXh0LWFsaWduOgogICAgICAgIHN0YXJ0OyB0ZXh0LWlu ZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6CiAgICAgICAgbm9y bWFsOyB3aWRvd3M6IDE7IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdp ZHRoOgogICAgICAgIDBweDsgZGlzcGxheTogaW5saW5lICFpbXBvcnRhbnQ7IGZsb2F0OiBub25l OyBiYWNrZ3JvdW5kLWNvbG9yOgogICAgICAgIHJnYigyNTUsIDI1NSwgMjU1KTsiPmJlY2F1c2Ug aXQgaXMgZmFzdGVyIDwvc3Bhbj5hbmQgYmVjYXVzZQogICAgICB0aGUgcmVzdWx0aW5nIGZpbGVz IGFyZSBzdGlsbCBzcGFyc2UgKCdkdScgcmVwb3J0cyBleGFjdGx5IHRoZQogICAgICBzYW1lIHNp emVzKS48YnI+CiAgICAgIEJ1dCBJIGFsc28gZGlzY292ZXJlZCBhIGJvdHRsZW5lY2suPGJyPgog ICAgICBXaGVuIGNvcHlpbmcgc3BhcnNlIGZpbGVzIChlLmcuIGEgMiBUQiBzcGFyc2UgaW1hZ2Ug dGhhdCBvbmx5IHVzZXMKICAgICAgODAwIE1CIGluIGRpc2ssIGEgY29tbW9uIHNjZW5hcmlvIHdo ZW4gd2UgY3JlYXRlIHRlbXBsYXRlcyBmcm9tCiAgICAgIGZyZXNoIGluc3RhbGxzKSB0aGUgJ2Nw JyBjb21tYW5kIGJlaGF2ZXMgZGlmZmVyZW50bHkgZGVwZW5kaW5nIG9uCiAgICAgIGlmIHdlIGFy ZSByZWFkaW5nIGZyb20gYSBnbHVzdGVyIG1vdW50IG9yIGZyb20gYSBmaWxlc3lzdGVtCiAgICAg IHN1cHBvcnRpbmcgU0VFS19IT0xFIChhdmFpbGFibGUgaW4ga2VybmVscyAmZ3Q7PSAzLjEpOjxi cj4KICAgICAgPGJyPgogICAgICBhKSBJZiB3ZSByZWFkIGZyb20gYSBnbHVzdGVyIG1vdW50LCAn Y3AnIHJlYWRzIHRoZSAyIFRCIG9mIHplcm9zLAogICAgICBldmVuIHdoZW4gaXQgb25seSB3cml0 ZXMgdGhlIG5vbi16ZXJvcyAoaW90b3Agc2hvd3MgdGhlICdjcCcKICAgICAgcHJvY2VzcyByZWFk aW5nIHRob3NlIDIgVEIpLiBPbmx5IHNwYXJzZSB3cml0aW5nIGlzIG9wdGltaXplZC48YnI+CiAg ICAgIDxicj4KICAgIDwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6IHJnYigwLCAwLCAwKTsgZm9u dC1mYW1pbHk6ICdBcmlhbCBVbmljb2RlCiAgICAgIE1TJywgQXJpYWwsIHNhbnMtc2VyaWY7IGZv bnQtc2l6ZTogc21hbGw7IGZvbnQtc3R5bGU6IG5vcm1hbDsKICAgICAgZm9udC12YXJpYW50OiBu b3JtYWw7IGZvbnQtd2VpZ2h0OiBub3JtYWw7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7CiAgICAg IGxpbmUtaGVpZ2h0OiAyMS42NjY2Njc5MzgyMzI0cHg7IG9ycGhhbnM6IGF1dG87IHRleHQtYWxp Z246IHN0YXJ0OwogICAgICB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsg d2hpdGUtc3BhY2U6IG5vcm1hbDsKICAgICAgd2lkb3dzOiAxOyB3b3JkLXNwYWNpbmc6IDBweDsg LXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OwogICAgICBkaXNwbGF5OiBpbmxpbmUgIWlt cG9ydGFudDsgZmxvYXQ6IG5vbmU7IGJhY2tncm91bmQtY29sb3I6CiAgICAgIHJnYigyNTUsIDI1 NSwgMjU1KTsiPmIpIElmIHdlIHJlYWQgZnJvbSBhIFNFRUtfSE9MRSBzdXBwb3J0aW5nCiAgICAg IGZpbGVzeXN0ZW0gKGV4dCwgeGZzLCBldGMpLCAnY3AnIG9ubHkgcmVhZHMgdGhlIG5vbi16ZXJv IGNvbnRlbnQsCiAgICAgIHRodXMgcmVhZGluZyBhbmQgd3JpdGluZyB0YWtlcyBsaWtlIDEwIFtz XSBpbnN0ZWFkIG9mIGhvdXJzLjxicj4KICAgICAgPGJyPgogICAgICBJdCBzZWVtcyBnbHVzdGVy IGlzIG5vdCB1c2luZyBTRUVLX0hPTEUgZm9yIHJlYWRpbmcgKD8hKS48YnI+CiAgICAgIDxicj4K ICAgIDwvc3Bhbj5Db25zaWRlcmluZyB0aGUgc291cmNlIGltYWdlIGlzIG5vdCBiZWluZyBtb2Rp ZmllZCBkdXJpbmcgdGhlCiAgICAiTWFrZSBUZW1wbGF0ZSIgcHJvY2VzcyBhbmQgd2UgaGF2ZSBh Y2Nlc3MgdG8gdGhlIGdsdXN0ZXIgYnJpY2tzLCBpdAogICAgaXMgcG9zc2libGUgdG8gJ2NwJyB0 aGUgc291cmNlIGltYWdlIGRpcmVjdGx5IGZyb20gdGhlIGJyaWNrcyAob24KICAgIHRvcCBvZiB0 aGUgU0VFS19IT0xFIHN1cHBvcnRpbmcgRlMpIGluc3RlYWQgb2YgcmVhZGluZyBmcm9tIHRoZQog ICAgZ2x1c3RlciBtb3VudC48YnI+CiAgICA8YnI+CiAgICBUaGUgZGlmZmVyZW5jZSBpcyByZWFs bHkgaW1wcmVzc2l2ZSAoc2Vjb25kcyBpbnN0ZWFkIG9mIGhvdXJzKS48YnI+CiAgICBJIHRlc3Rl ZCBpdCBjbG9uaW5nIHNvbWUgVk0ncyBhbmQgaXQgd29ya3MuPGJyPgogICAgPGJyPgogICAgQW0g SSBtaXNzaW5nIHNvbWV0aGluZz88YnI+CiAgICBNYXliZSBhIGdsdXN0ZXIgb3B0aW1pemF0aW9u IHRvIGVuYWJsZSBTRUVLX0hPTEUgc3VwcG9ydCBvbiBnbHVzdGVyCiAgICBtb3VudHM/PGJyPgog ICAgPGJyPgogIDwvYm9keT4KPC9odG1sPgoKLS0tLS0tLS0tLS0tLS0wODA3MDAwOTA2MDkwOTA0 MDUwNTAwMDktLQo= --===============1193919570093340470==--