From apahim at redhat.com Thu Nov 20 07:20:09 2014 Content-Type: multipart/mixed; boundary="===============5779157013650671914==" MIME-Version: 1.0 From: Amador Pahim To: users at ovirt.org Subject: Re: [ovirt-users] Cumulative VM network usagePERIOD_TIME Date: Thu, 20 Nov 2014 09:20:07 -0300 Message-ID: <546DDC77.3090803@redhat.com> In-Reply-To: 5461DFFE.9060006@redhat.com --===============5779157013650671914== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------080401060806050003020601 Content-Type: text/plain; charset=3Dwindows-1252; format=3Dflowed Content-Transfer-Encoding: 7bit Hi Lior, Thank you for this. Indeed I have seen multiple requests for this. I = also have a bugzilla for it: = https://bugzilla.redhat.com/show_bug.cgi?id=3D1108144. Some comments bellow. On 11/11/2014 07:07 AM, Lior Vernia wrote: > Hello, > > The need to monitor cumulative VM network usage has come up several > times in the past; while this should be handled as part of > (https://bugzilla.redhat.com/show_bug.cgi?id=3D1063343), in the mean time > I've written a small Python script that monitors those statistics, > attached here. > > The script polls the engine via RESTful API periodically and dumps the > up-to-date total usage into a file. The output is a multi-level > map/dictionary in JSON format, where: > * The top level keys are VM names. > * Under each VM, the next level keys are vNIC names. > * Under each vNIC, there are keys for total 'rx' (received) and 'tx' > (transmitted), where the values are in Bytes. > > The script is built to run forever. It may be stopped at any time, but > while it's not running VM network usage data will "be lost". When it's > re-run, it'll go back to accumulating data on top of its previous data. This could be mitigated if along with rx and tx data, vdsm was reporting = a timestamp reflecting the time when data was collected. So, even with = gaps, we should be able to calculate the cumulative information. > > A few disclaimers: > * I haven't tested this with any edge cases (engine service dies, etc.). > * Tested this with tens of VMs, not sure it'll work fine with hundreds. > * The PERIOD_TIME (polling interval) should be set so that it matches > both the engine's and vdsm's polling interval (see comments inside the > script), otherwise data will be either lost or counted multiple times. > From 3.4 onwards, default configuration should be fine with 15 seconds. Here we have another issue. In 3.4, 15 seconds is fine... backend and = vdsm are in line with 15 seconds. But up to 3.3, vdsm is pooling the = data every 5 seconds and backend is collecting data every 15 seconds. So = 2 in 3 vdsm poolings are droped. Since you're handling total bytes, this = might not be a big issue. > * The precision of traffic measurement on a NIC is 0.1% of the > interface's speed over each PERIOD_TIME interval. For example, on a > 1Gbps vNIC, when PERIOD_TIME =3D 15s, data will only be measured in 15Mb > (~2MB) quanta. Specifically what this means is, that in this example, > any traffic smaller than 2MB over a 15-second period would be negligible > and wouldn't be recorded. Looking to the code, if "overhead" is bigger than "PERIOD_TIME", = cumulative data for a given period will never be accurate. Anyway the = script will fall in exception when that is the case (negative value for = time.sleep()). The mentioned timestamp reported by vdsm could drop the = need for the "overhead" calculation. > > Knock yourselves out :) > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users --------------080401060806050003020601 Content-Type: text/html; charset=3Dwindows-1252 Content-Transfer-Encoding: 7bit

Hi Lior,

Thank you for this. Indeed I have seen multiple requests for this. I also have a bugzilla for it: https://bugzilla.redhat.com/show_bug.cgi?id=3D= 1108144. Some comments bellow.

On 11/11/2014 07:07 AM, Lior Vernia wrote:

Hello,

The need to monitor cumulative VM network usage has come up several
times in the past; while this should be handled as part of
(https://bugzilla.redhat.com/show_bug.cgi?id=3D10633=
43), in the mean time
I've written a small Python script that monitors those statistics,
attached here.

The script polls the engine via RESTful API periodically and dumps the
up-to-date total usage into a file. The output is a multi-level
map/dictionary in JSON format, where:
* The top level keys are VM names.
* Under each VM, the next level keys are vNIC names.
* Under each vNIC, there are keys for total 'rx' (received) and 'tx'
(transmitted), where the values are in Bytes.

The script is built to run forever. It may be stopped at any time, but
while it's not running VM network usage data will "be lost". When it's
re-run, it'll go back to accumulating data on top of its previous data.

This could be mitigated if along with rx and tx data, vdsm was reporting a timestamp reflecting the time when data was collected. So, even with gaps, we should be able to calculate the cumulative information.


A few disclaimers:
* I haven't tested this with any edge cases (engine service dies, etc.).
* Tested this with tens of VMs, not sure it'll work fine with hundreds.
* The PERIOD_TIME (polling interval) should be set so that it matches
both the engine's and vdsm's polling interval (see comments inside the
script), otherwise data will be either lost or counted multiple times.
>From 3.4 onwards, default configuration should be fine with 15 seconds.

Here we have another issue. In 3.4, 15 seconds is fine... backend and vdsm are in line with 15 seconds. But up to 3.3, vdsm is pooling the data every 5 seconds and backend is collecting data every 15 seconds. So 2 in 3 vdsm poolings are droped. Since you're handling total bytes, this might not be a big issue.

* The precision of traffic measurement on a NIC is 0.1% of the
interface's speed over each PERIOD_TIME interval. For example, on a
1Gbps vNIC, when PERIOD_TIME =3D 15s, data will only be measured in 15Mb
(~2MB) quanta. Specifically what this means is, that in this example,
any traffic smaller than 2MB over a 15-second period would be negligible
and wouldn't be recorded.

Looking to the code, if "overhead" is bigger than "PERIOD_TIME", cumulative data for a given period will never be accurate. Anyway the script will fall in exception when that is the case (negative value for time.sleep()). The mentioned timestamp reported by vdsm could drop the need for the "overhead" calculation.


Knock yourselves out :)

_______________________________________________
Users mailing list
Use=
rs(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

--------------080401060806050003020601-- --===============5779157013650671914== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS0wODA0MDEwNjA4MDYwNTAwMDMwMjA2MDEKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXdpbmRvd3MtMTI1MjsgZm9ybWF0PWZsb3dlZApDb250ZW50LVRyYW5zZmVyLUVuY29kaW5n OiA3Yml0CgpIaSBMaW9yLAoKVGhhbmsgeW91IGZvciB0aGlzLiBJbmRlZWQgSSBoYXZlIHNlZW4g bXVsdGlwbGUgcmVxdWVzdHMgZm9yIHRoaXMuIEkgCmFsc28gaGF2ZSBhIGJ1Z3ppbGxhIGZvciBp dDogCmh0dHBzOi8vYnVnemlsbGEucmVkaGF0LmNvbS9zaG93X2J1Zy5jZ2k/aWQ9MTEwODE0NC4g U29tZSBjb21tZW50cyBiZWxsb3cuCgpPbiAxMS8xMS8yMDE0IDA3OjA3IEFNLCBMaW9yIFZlcm5p YSB3cm90ZToKPiBIZWxsbywKPgo+IFRoZSBuZWVkIHRvIG1vbml0b3IgY3VtdWxhdGl2ZSBWTSBu ZXR3b3JrIHVzYWdlIGhhcyBjb21lIHVwIHNldmVyYWwKPiB0aW1lcyBpbiB0aGUgcGFzdDsgd2hp bGUgdGhpcyBzaG91bGQgYmUgaGFuZGxlZCBhcyBwYXJ0IG9mCj4gKGh0dHBzOi8vYnVnemlsbGEu cmVkaGF0LmNvbS9zaG93X2J1Zy5jZ2k/aWQ9MTA2MzM0MyksIGluIHRoZSBtZWFuIHRpbWUKPiBJ J3ZlIHdyaXR0ZW4gYSBzbWFsbCBQeXRob24gc2NyaXB0IHRoYXQgbW9uaXRvcnMgdGhvc2Ugc3Rh dGlzdGljcywKPiBhdHRhY2hlZCBoZXJlLgo+Cj4gVGhlIHNjcmlwdCBwb2xscyB0aGUgZW5naW5l IHZpYSBSRVNUZnVsIEFQSSBwZXJpb2RpY2FsbHkgYW5kIGR1bXBzIHRoZQo+IHVwLXRvLWRhdGUg dG90YWwgdXNhZ2UgaW50byBhIGZpbGUuIFRoZSBvdXRwdXQgaXMgYSBtdWx0aS1sZXZlbAo+IG1h cC9kaWN0aW9uYXJ5IGluIEpTT04gZm9ybWF0LCB3aGVyZToKPiAqIFRoZSB0b3AgbGV2ZWwga2V5 cyBhcmUgVk0gbmFtZXMuCj4gKiBVbmRlciBlYWNoIFZNLCB0aGUgbmV4dCBsZXZlbCBrZXlzIGFy ZSB2TklDIG5hbWVzLgo+ICogVW5kZXIgZWFjaCB2TklDLCB0aGVyZSBhcmUga2V5cyBmb3IgdG90 YWwgJ3J4JyAocmVjZWl2ZWQpIGFuZCAndHgnCj4gKHRyYW5zbWl0dGVkKSwgd2hlcmUgdGhlIHZh bHVlcyBhcmUgaW4gQnl0ZXMuCj4KPiBUaGUgc2NyaXB0IGlzIGJ1aWx0IHRvIHJ1biBmb3JldmVy LiBJdCBtYXkgYmUgc3RvcHBlZCBhdCBhbnkgdGltZSwgYnV0Cj4gd2hpbGUgaXQncyBub3QgcnVu bmluZyBWTSBuZXR3b3JrIHVzYWdlIGRhdGEgd2lsbCAiYmUgbG9zdCIuIFdoZW4gaXQncwo+IHJl LXJ1biwgaXQnbGwgZ28gYmFjayB0byBhY2N1bXVsYXRpbmcgZGF0YSBvbiB0b3Agb2YgaXRzIHBy ZXZpb3VzIGRhdGEuCgpUaGlzIGNvdWxkIGJlIG1pdGlnYXRlZCBpZiBhbG9uZyB3aXRoIHJ4IGFu ZCB0eCBkYXRhLCB2ZHNtIHdhcyByZXBvcnRpbmcgCmEgdGltZXN0YW1wIHJlZmxlY3RpbmcgdGhl IHRpbWUgd2hlbiBkYXRhIHdhcyBjb2xsZWN0ZWQuIFNvLCBldmVuIHdpdGggCmdhcHMsIHdlIHNo b3VsZCBiZSBhYmxlIHRvIGNhbGN1bGF0ZSB0aGUgY3VtdWxhdGl2ZSBpbmZvcm1hdGlvbi4KCj4K PiBBIGZldyBkaXNjbGFpbWVyczoKPiAqIEkgaGF2ZW4ndCB0ZXN0ZWQgdGhpcyB3aXRoIGFueSBl ZGdlIGNhc2VzIChlbmdpbmUgc2VydmljZSBkaWVzLCBldGMuKS4KPiAqIFRlc3RlZCB0aGlzIHdp dGggdGVucyBvZiBWTXMsIG5vdCBzdXJlIGl0J2xsIHdvcmsgZmluZSB3aXRoIGh1bmRyZWRzLgo+ ICogVGhlIFBFUklPRF9USU1FIChwb2xsaW5nIGludGVydmFsKSBzaG91bGQgYmUgc2V0IHNvIHRo YXQgaXQgbWF0Y2hlcwo+IGJvdGggdGhlIGVuZ2luZSdzIGFuZCB2ZHNtJ3MgcG9sbGluZyBpbnRl cnZhbCAoc2VlIGNvbW1lbnRzIGluc2lkZSB0aGUKPiBzY3JpcHQpLCBvdGhlcndpc2UgZGF0YSB3 aWxsIGJlIGVpdGhlciBsb3N0IG9yIGNvdW50ZWQgbXVsdGlwbGUgdGltZXMuCj4gIEZyb20gMy40 IG9ud2FyZHMsIGRlZmF1bHQgY29uZmlndXJhdGlvbiBzaG91bGQgYmUgZmluZSB3aXRoIDE1IHNl Y29uZHMuCgpIZXJlIHdlIGhhdmUgYW5vdGhlciBpc3N1ZS4gSW4gMy40LCAxNSBzZWNvbmRzIGlz IGZpbmUuLi4gYmFja2VuZCBhbmQgCnZkc20gYXJlIGluIGxpbmUgd2l0aCAxNSBzZWNvbmRzLiBC dXQgdXAgdG8gMy4zLCB2ZHNtIGlzIHBvb2xpbmcgdGhlIApkYXRhIGV2ZXJ5IDUgc2Vjb25kcyBh bmQgYmFja2VuZCBpcyBjb2xsZWN0aW5nIGRhdGEgZXZlcnkgMTUgc2Vjb25kcy4gU28gCjIgaW4g MyB2ZHNtIHBvb2xpbmdzIGFyZSBkcm9wZWQuIFNpbmNlIHlvdSdyZSBoYW5kbGluZyB0b3RhbCBi eXRlcywgdGhpcyAKbWlnaHQgbm90IGJlIGEgYmlnIGlzc3VlLgoKPiAqIFRoZSBwcmVjaXNpb24g b2YgdHJhZmZpYyBtZWFzdXJlbWVudCBvbiBhIE5JQyBpcyAwLjElIG9mIHRoZQo+IGludGVyZmFj ZSdzIHNwZWVkIG92ZXIgZWFjaCBQRVJJT0RfVElNRSBpbnRlcnZhbC4gRm9yIGV4YW1wbGUsIG9u IGEKPiAxR2JwcyB2TklDLCB3aGVuIFBFUklPRF9USU1FID0gMTVzLCBkYXRhIHdpbGwgb25seSBi ZSBtZWFzdXJlZCBpbiAxNU1iCj4gKH4yTUIpIHF1YW50YS4gU3BlY2lmaWNhbGx5IHdoYXQgdGhp cyBtZWFucyBpcywgdGhhdCBpbiB0aGlzIGV4YW1wbGUsCj4gYW55IHRyYWZmaWMgc21hbGxlciB0 aGFuIDJNQiBvdmVyIGEgMTUtc2Vjb25kIHBlcmlvZCB3b3VsZCBiZSBuZWdsaWdpYmxlCj4gYW5k IHdvdWxkbid0IGJlIHJlY29yZGVkLgoKTG9va2luZyB0byB0aGUgY29kZSwgaWYgIm92ZXJoZWFk IiBpcyBiaWdnZXIgdGhhbiAiUEVSSU9EX1RJTUUiLCAKY3VtdWxhdGl2ZSBkYXRhIGZvciBhIGdp dmVuIHBlcmlvZCB3aWxsIG5ldmVyIGJlIGFjY3VyYXRlLiBBbnl3YXkgdGhlIApzY3JpcHQgd2ls bCBmYWxsIGluIGV4Y2VwdGlvbiB3aGVuIHRoYXQgaXMgdGhlIGNhc2UgKG5lZ2F0aXZlIHZhbHVl IGZvciAKdGltZS5zbGVlcCgpKS4gVGhlIG1lbnRpb25lZCB0aW1lc3RhbXAgcmVwb3J0ZWQgYnkg dmRzbSBjb3VsZCBkcm9wIHRoZSAKbmVlZCBmb3IgdGhlICJvdmVyaGVhZCIgY2FsY3VsYXRpb24u Cgo+Cj4gS25vY2sgeW91cnNlbHZlcyBvdXQgOikKPgo+Cj4gX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX18KPiBVc2VycyBtYWlsaW5nIGxpc3QKPiBVc2Vyc0Bv dmlydC5vcmcKPiBodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMK CgotLS0tLS0tLS0tLS0tLTA4MDQwMTA2MDgwNjA1MDAwMzAyMDYwMQpDb250ZW50LVR5cGU6IHRl eHQvaHRtbDsgY2hhcnNldD13aW5kb3dzLTEyNTIKQ29udGVudC1UcmFuc2Zlci1FbmNvZGluZzog N2JpdAoKPGh0bWw+CiAgPGhlYWQ+CiAgICA8bWV0YSBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJz ZXQ9d2luZG93cy0xMjUyIgogICAgICBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiPgogIDwvaGVh ZD4KICA8Ym9keSB0ZXh0PSIjMDAwMDAwIiBiZ2NvbG9yPSIjRkZGRkZGIj4KICAgIDxkaXYgY2xh c3M9Im1vei1jaXRlLXByZWZpeCI+SGkgTGlvciw8YnI+CiAgICAgIDxicj4KICAgICAgVGhhbmsg eW91IGZvciB0aGlzLiBJbmRlZWQgSSBoYXZlIHNlZW4gbXVsdGlwbGUgcmVxdWVzdHMgZm9yIHRo aXMuCiAgICAgIEkgYWxzbyBoYXZlIGEgYnVnemlsbGEgZm9yIGl0OgogICAgICA8YSBjbGFzcz0i bW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwczovL2J1Z3ppbGxhLnJlZGhhdC5jb20v c2hvd19idWcuY2dpP2lkPTExMDgxNDQiPmh0dHBzOi8vYnVnemlsbGEucmVkaGF0LmNvbS9zaG93 X2J1Zy5jZ2k/aWQ9MTEwODE0NDwvYT4uIFNvbWUgY29tbWVudHMKICAgICAgYmVsbG93Ljxicj4K ICAgICAgPGJyPgogICAgICBPbiAxMS8xMS8yMDE0IDA3OjA3IEFNLCBMaW9yIFZlcm5pYSB3cm90 ZTo8YnI+CiAgICA8L2Rpdj4KICAgIDxibG9ja3F1b3RlIGNpdGU9Im1pZDo1NDYxREZGRS45MDYw MDA2QHJlZGhhdC5jb20iIHR5cGU9ImNpdGUiPgogICAgICA8cHJlIHdyYXA9IiI+SGVsbG8sCgpU aGUgbmVlZCB0byBtb25pdG9yIGN1bXVsYXRpdmUgVk0gbmV0d29yayB1c2FnZSBoYXMgY29tZSB1 cCBzZXZlcmFsCnRpbWVzIGluIHRoZSBwYXN0OyB3aGlsZSB0aGlzIHNob3VsZCBiZSBoYW5kbGVk IGFzIHBhcnQgb2YKKDxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHBz Oi8vYnVnemlsbGEucmVkaGF0LmNvbS9zaG93X2J1Zy5jZ2k/aWQ9MTA2MzM0MyI+aHR0cHM6Ly9i dWd6aWxsYS5yZWRoYXQuY29tL3Nob3dfYnVnLmNnaT9pZD0xMDYzMzQzPC9hPiksIGluIHRoZSBt ZWFuIHRpbWUKSSd2ZSB3cml0dGVuIGEgc21hbGwgUHl0aG9uIHNjcmlwdCB0aGF0IG1vbml0b3Jz IHRob3NlIHN0YXRpc3RpY3MsCmF0dGFjaGVkIGhlcmUuCgpUaGUgc2NyaXB0IHBvbGxzIHRoZSBl bmdpbmUgdmlhIFJFU1RmdWwgQVBJIHBlcmlvZGljYWxseSBhbmQgZHVtcHMgdGhlCnVwLXRvLWRh dGUgdG90YWwgdXNhZ2UgaW50byBhIGZpbGUuIFRoZSBvdXRwdXQgaXMgYSBtdWx0aS1sZXZlbApt YXAvZGljdGlvbmFyeSBpbiBKU09OIGZvcm1hdCwgd2hlcmU6CiogVGhlIHRvcCBsZXZlbCBrZXlz IGFyZSBWTSBuYW1lcy4KKiBVbmRlciBlYWNoIFZNLCB0aGUgbmV4dCBsZXZlbCBrZXlzIGFyZSB2 TklDIG5hbWVzLgoqIFVuZGVyIGVhY2ggdk5JQywgdGhlcmUgYXJlIGtleXMgZm9yIHRvdGFsICdy eCcgKHJlY2VpdmVkKSBhbmQgJ3R4JwoodHJhbnNtaXR0ZWQpLCB3aGVyZSB0aGUgdmFsdWVzIGFy ZSBpbiBCeXRlcy4KClRoZSBzY3JpcHQgaXMgYnVpbHQgdG8gcnVuIGZvcmV2ZXIuIEl0IG1heSBi ZSBzdG9wcGVkIGF0IGFueSB0aW1lLCBidXQKd2hpbGUgaXQncyBub3QgcnVubmluZyBWTSBuZXR3 b3JrIHVzYWdlIGRhdGEgd2lsbCAiYmUgbG9zdCIuIFdoZW4gaXQncwpyZS1ydW4sIGl0J2xsIGdv IGJhY2sgdG8gYWNjdW11bGF0aW5nIGRhdGEgb24gdG9wIG9mIGl0cyBwcmV2aW91cyBkYXRhLjwv cHJlPgogICAgPC9ibG9ja3F1b3RlPgogICAgPGJyPgogICAgVGhpcyBjb3VsZCBiZSBtaXRpZ2F0 ZWQgaWYgYWxvbmcgd2l0aCByeCBhbmQgdHggZGF0YSwgdmRzbSB3YXMKICAgIHJlcG9ydGluZyBh IHRpbWVzdGFtcCByZWZsZWN0aW5nIHRoZSB0aW1lIHdoZW4gZGF0YSB3YXMgY29sbGVjdGVkLgog ICAgU28sIGV2ZW4gd2l0aCBnYXBzLCB3ZSBzaG91bGQgYmUgYWJsZSB0byBjYWxjdWxhdGUgdGhl IGN1bXVsYXRpdmUKICAgIGluZm9ybWF0aW9uLjxicj4KICAgIDxicj4KICAgIDxibG9ja3F1b3Rl IGNpdGU9Im1pZDo1NDYxREZGRS45MDYwMDA2QHJlZGhhdC5jb20iIHR5cGU9ImNpdGUiPgogICAg ICA8cHJlIHdyYXA9IiI+CgpBIGZldyBkaXNjbGFpbWVyczoKKiBJIGhhdmVuJ3QgdGVzdGVkIHRo aXMgd2l0aCBhbnkgZWRnZSBjYXNlcyAoZW5naW5lIHNlcnZpY2UgZGllcywgZXRjLikuCiogVGVz dGVkIHRoaXMgd2l0aCB0ZW5zIG9mIFZNcywgbm90IHN1cmUgaXQnbGwgd29yayBmaW5lIHdpdGgg aHVuZHJlZHMuCiogVGhlIFBFUklPRF9USU1FIChwb2xsaW5nIGludGVydmFsKSBzaG91bGQgYmUg c2V0IHNvIHRoYXQgaXQgbWF0Y2hlcwpib3RoIHRoZSBlbmdpbmUncyBhbmQgdmRzbSdzIHBvbGxp bmcgaW50ZXJ2YWwgKHNlZSBjb21tZW50cyBpbnNpZGUgdGhlCnNjcmlwdCksIG90aGVyd2lzZSBk YXRhIHdpbGwgYmUgZWl0aGVyIGxvc3Qgb3IgY291bnRlZCBtdWx0aXBsZSB0aW1lcy4KPkZyb20g My40IG9ud2FyZHMsIGRlZmF1bHQgY29uZmlndXJhdGlvbiBzaG91bGQgYmUgZmluZSB3aXRoIDE1 IHNlY29uZHMuPC9wcmU+CiAgICA8L2Jsb2NrcXVvdGU+CiAgICA8YnI+CiAgICBIZXJlIHdlIGhh dmUgYW5vdGhlciBpc3N1ZS4gSW4gMy40LCAxNSBzZWNvbmRzIGlzIGZpbmUuLi4gYmFja2VuZAog ICAgYW5kIHZkc20gYXJlIGluIGxpbmUgd2l0aCAxNSBzZWNvbmRzLiBCdXQgdXAgdG8gMy4zLCB2 ZHNtIGlzIHBvb2xpbmcKICAgIHRoZSBkYXRhIGV2ZXJ5IDUgc2Vjb25kcyBhbmQgYmFja2VuZCBp cyBjb2xsZWN0aW5nIGRhdGEgZXZlcnkgMTUKICAgIHNlY29uZHMuIFNvIDIgaW4gMyB2ZHNtIHBv b2xpbmdzIGFyZSBkcm9wZWQuIFNpbmNlIHlvdSdyZSBoYW5kbGluZwogICAgdG90YWwgYnl0ZXMs IHRoaXMgbWlnaHQgbm90IGJlIGEgYmlnIGlzc3VlLjxicj4KICAgIDxicj4KICAgIDxibG9ja3F1 b3RlIGNpdGU9Im1pZDo1NDYxREZGRS45MDYwMDA2QHJlZGhhdC5jb20iIHR5cGU9ImNpdGUiPgog ICAgICA8cHJlIHdyYXA9IiI+CiogVGhlIHByZWNpc2lvbiBvZiB0cmFmZmljIG1lYXN1cmVtZW50 IG9uIGEgTklDIGlzIDAuMSUgb2YgdGhlCmludGVyZmFjZSdzIHNwZWVkIG92ZXIgZWFjaCBQRVJJ T0RfVElNRSBpbnRlcnZhbC4gRm9yIGV4YW1wbGUsIG9uIGEKMUdicHMgdk5JQywgd2hlbiBQRVJJ T0RfVElNRSA9IDE1cywgZGF0YSB3aWxsIG9ubHkgYmUgbWVhc3VyZWQgaW4gMTVNYgoofjJNQikg cXVhbnRhLiBTcGVjaWZpY2FsbHkgd2hhdCB0aGlzIG1lYW5zIGlzLCB0aGF0IGluIHRoaXMgZXhh bXBsZSwKYW55IHRyYWZmaWMgc21hbGxlciB0aGFuIDJNQiBvdmVyIGEgMTUtc2Vjb25kIHBlcmlv ZCB3b3VsZCBiZSBuZWdsaWdpYmxlCmFuZCB3b3VsZG4ndCBiZSByZWNvcmRlZC48L3ByZT4KICAg IDwvYmxvY2txdW90ZT4KICAgIDxicj4KICAgIExvb2tpbmcgdG8gdGhlIGNvZGUsIGlmICJvdmVy aGVhZCIgaXMgYmlnZ2VyIHRoYW4gIlBFUklPRF9USU1FIiwKICAgIGN1bXVsYXRpdmUgZGF0YSBm b3IgYSBnaXZlbiBwZXJpb2Qgd2lsbCBuZXZlciBiZSBhY2N1cmF0ZS4gQW55d2F5CiAgICB0aGUg c2NyaXB0IHdpbGwgZmFsbCBpbiBleGNlcHRpb24gd2hlbiB0aGF0IGlzIHRoZSBjYXNlIChuZWdh dGl2ZQogICAgdmFsdWUgZm9yIHRpbWUuc2xlZXAoKSkuIFRoZSBtZW50aW9uZWQgdGltZXN0YW1w IHJlcG9ydGVkIGJ5IHZkc20KICAgIGNvdWxkIGRyb3AgdGhlIG5lZWQgZm9yIHRoZSAib3Zlcmhl YWQiIGNhbGN1bGF0aW9uLjxicj4KICAgIDxicj4KICAgIDxibG9ja3F1b3RlIGNpdGU9Im1pZDo1 NDYxREZGRS45MDYwMDA2QHJlZGhhdC5jb20iIHR5cGU9ImNpdGUiPgogICAgICA8cHJlIHdyYXA9 IiI+CgpLbm9jayB5b3Vyc2VsdmVzIG91dCA6KQo8L3ByZT4KICAgICAgPGJyPgogICAgICA8Zmll bGRzZXQgY2xhc3M9Im1pbWVBdHRhY2htZW50SGVhZGVyIj48L2ZpZWxkc2V0PgogICAgICA8YnI+ CiAgICAgIDxwcmUgd3JhcD0iIj5fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fXwpVc2VycyBtYWlsaW5nIGxpc3QKPGEgY2xhc3M9Im1vei10eHQtbGluay1hYmJy ZXZpYXRlZCIgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyI+VXNlcnNAb3ZpcnQub3JnPC9h Pgo8YSBjbGFzcz0ibW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwOi8vbGlzdHMub3Zp cnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiPmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFp bG1hbi9saXN0aW5mby91c2VyczwvYT4KPC9wcmU+CiAgICA8L2Jsb2NrcXVvdGU+CiAgICA8YnI+ CiAgPC9ib2R5Pgo8L2h0bWw+CgotLS0tLS0tLS0tLS0tLTA4MDQwMTA2MDgwNjA1MDAwMzAyMDYw MS0tCg== --===============5779157013650671914==--