From fromani at redhat.com Mon Feb 27 08:11:45 2017 Content-Type: multipart/mixed; boundary="===============0044786323360172455==" MIME-Version: 1.0 From: Francesco Romani To: devel at ovirt.org Subject: Re: [ovirt-devel] [monitoring][collectd] the collectd virt plugin is now on par with Vdsm needs Date: Mon, 27 Feb 2017 09:11:41 +0100 Message-ID: <76727e1c-04b1-fead-e36e-0296ebc37266@redhat.com> In-Reply-To: CACKMAy_kfX=LCNe+7EFD2Cjw_Wit+aDS8FR+RPKEo=cMXU+H+A@mail.gmail.com --===============0044786323360172455== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------A2FDAF300209D96688B3E8CC Content-Type: text/plain; charset=3Dutf-8 Content-Transfer-Encoding: 7bit On 02/26/2017 03:13 PM, Yaniv Dary wrote: > >> 2. collectd *intentionally* report metrics as rates, not as >> absolute >> values as Vdsm does. This may be one issue in presence of >> restarts/data >> loss in the link between collectd and the metrics store. >> >> >> How does this work? = >> If we want to show memory usage over time for example, we need to >> have the usage, not the rate. = >> How would this be reported? > > I was imprecise, my fault. > > Let me retry: > collectd intentionally report quite a lot of metrics we care about > as rates, not as absolute values. > Memory is actually ok fine. > > a0/virt/disk_octets-hdc -> rate > a0/virt/disk_octets-vda > a0/virt/disk_ops-hdc -> rate > a0/virt/disk_ops-vda > a0/virt/disk_time-hdc -> rate > a0/virt/disk_time-vda > a0/virt/if_dropped-vnet0 -> rate > a0/virt/if_errors-vnet0 -> rate > a0/virt/if_octets-vnet0 -> rate > a0/virt/if_packets-vnet0 -> rate > a0/virt/memory-actual_balloon -> absolute > a0/virt/memory-rss -> absolute > a0/virt/memory-total -> absolute > a0/virt/ps_cputime -> rate > a0/virt/total_requests-flush-hdc -> rate > a0/virt/total_requests-flush-vda > a0/virt/total_time_in_ms-flush-hdc -> rate > a0/virt/total_time_in_ms-flush-vda > a0/virt/virt_cpu_total -> rate > a0/virt/virt_vcpu-0 -> rate > a0/virt/virt_vcpu-1 > > collectd "just" reports the changes since the last sampling. I'm > not sure which is the best way to handle that; I've sent a mail to > collectd list some time ago, no answer so far. > > > Can you CC on that thread? > I don't know how ES would work with rates at all. = > I want to be able to show CPU usage over time and I need to know if > its 80% or 10%. > Thanks to the awkward gmail interface I can't reply to myself and CC other people, but I can share the link: https://mailman.verplant.org/pipermail/collectd/2017-January/006965.html -- = Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani --------------A2FDAF300209D96688B3E8CC Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit
On 02/26/2017 03:13 PM, Yaniv Dary wrote:

2. collectd *intentionally* report metrics as rates, not as absolute
values as Vdsm does. This may be one issue in presence of restarts/data
loss in the link between collectd and the metrics store.

How does this work?=C2=A0
If we want to show memory usage over time for example, we need to have the usage, not the rate.=C2=A0
How would this be reported?

I was imprecise, my fault.

Let me retry:
collectd intentionally report quite a lot of metrics we care about as rates, not as absolute values.
Memory is actually ok fine.

=C2=A0 a0/virt/disk_octets-hdc -> rate
=C2=A0 a0/virt/disk_octets-vda
=C2=A0 a0/virt/disk_ops-hdc -> rate
=C2=A0 a0/virt/disk_ops-vda
=C2=A0 a0/virt/disk_time-hdc -> rate
=C2=A0 a0/virt/disk_time-vda
=C2=A0 a0/virt/if_dropped-vnet0 -> rate
=C2=A0 a0/virt/if_errors-vnet0 -> rate
=C2=A0 a0/virt/if_octets-vnet0 -> rate
=C2=A0 a0/virt/if_packets-vnet0 -> rate
=C2=A0 a0/virt/memory-actual_balloon -> absolute
=C2=A0 a0/virt/memory-rss -> absolute
=C2=A0 a0/virt/memory-total -> absolute
=C2=A0 a0/virt/ps_cputime -> rate
=C2=A0 a0/virt/total_requests-flush-hdc ->=C2=A0 ra= te
=C2=A0 a0/virt/total_requests-flush-vda
=C2=A0 a0/virt/total_time_in_ms-flush-hdc -> rate =C2=A0 a0/virt/total_time_in_ms-flush-vda
=C2=A0 a0/virt/virt_cpu_total -> rate
=C2=A0 a0/virt/virt_vcpu-0 -> rate
=C2=A0 a0/virt/virt_vcpu-1

collectd "just" reports the changes since the last sampling. I'm not sure which is the best way to handle that; I've sent a mail to collectd list some time ago, no answer so far.

Can you CC on that thread?
I don't know how ES would work with rates at all.=C2=A0
I want to be able to show CPU usage over time and I need to know if its 80% or 10%.


Thanks to the awkward gmail interface I can't reply to myself and CC other people, but I can share the link:

https://mailman.verplant.org/pipe= rmail/collectd/2017-January/006965.html

-- =

Francesco Romani
Red Hat Engineering Virtualization R & D
IRC: fromani
--------------A2FDAF300209D96688B3E8CC-- --===============0044786323360172455== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS1BMkZEQUYzMDAyMDlEOTY2ODhCM0U4Q0MKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDdiaXQKCgpPbiAwMi8yNi8yMDE3 IDAzOjEzIFBNLCBZYW5pdiBEYXJ5IHdyb3RlOgo+Cj4+ICAgICAgICAgMi4gY29sbGVjdGQgKmlu dGVudGlvbmFsbHkqIHJlcG9ydCBtZXRyaWNzIGFzIHJhdGVzLCBub3QgYXMKPj4gICAgICAgICBh YnNvbHV0ZQo+PiAgICAgICAgIHZhbHVlcyBhcyBWZHNtIGRvZXMuIFRoaXMgbWF5IGJlIG9uZSBp c3N1ZSBpbiBwcmVzZW5jZSBvZgo+PiAgICAgICAgIHJlc3RhcnRzL2RhdGEKPj4gICAgICAgICBs b3NzIGluIHRoZSBsaW5rIGJldHdlZW4gY29sbGVjdGQgYW5kIHRoZSBtZXRyaWNzIHN0b3JlLgo+ Pgo+Pgo+PiAgICAgSG93IGRvZXMgdGhpcyB3b3JrPyAKPj4gICAgIElmIHdlIHdhbnQgdG8gc2hv dyBtZW1vcnkgdXNhZ2Ugb3ZlciB0aW1lIGZvciBleGFtcGxlLCB3ZSBuZWVkIHRvCj4+ICAgICBo YXZlIHRoZSB1c2FnZSwgbm90IHRoZSByYXRlLiAKPj4gICAgIEhvdyB3b3VsZCB0aGlzIGJlIHJl cG9ydGVkPwo+Cj4gICAgIEkgd2FzIGltcHJlY2lzZSwgbXkgZmF1bHQuCj4KPiAgICAgTGV0IG1l IHJldHJ5Ogo+ICAgICBjb2xsZWN0ZCBpbnRlbnRpb25hbGx5IHJlcG9ydCBxdWl0ZSBhIGxvdCBv ZiBtZXRyaWNzIHdlIGNhcmUgYWJvdXQKPiAgICAgYXMgcmF0ZXMsIG5vdCBhcyBhYnNvbHV0ZSB2 YWx1ZXMuCj4gICAgIE1lbW9yeSBpcyBhY3R1YWxseSBvayBmaW5lLgo+Cj4gICAgICAgYTAvdmly dC9kaXNrX29jdGV0cy1oZGMgLT4gcmF0ZQo+ICAgICAgIGEwL3ZpcnQvZGlza19vY3RldHMtdmRh Cj4gICAgICAgYTAvdmlydC9kaXNrX29wcy1oZGMgLT4gcmF0ZQo+ICAgICAgIGEwL3ZpcnQvZGlz a19vcHMtdmRhCj4gICAgICAgYTAvdmlydC9kaXNrX3RpbWUtaGRjIC0+IHJhdGUKPiAgICAgICBh MC92aXJ0L2Rpc2tfdGltZS12ZGEKPiAgICAgICBhMC92aXJ0L2lmX2Ryb3BwZWQtdm5ldDAgLT4g cmF0ZQo+ICAgICAgIGEwL3ZpcnQvaWZfZXJyb3JzLXZuZXQwIC0+IHJhdGUKPiAgICAgICBhMC92 aXJ0L2lmX29jdGV0cy12bmV0MCAtPiByYXRlCj4gICAgICAgYTAvdmlydC9pZl9wYWNrZXRzLXZu ZXQwIC0+IHJhdGUKPiAgICAgICBhMC92aXJ0L21lbW9yeS1hY3R1YWxfYmFsbG9vbiAtPiBhYnNv bHV0ZQo+ICAgICAgIGEwL3ZpcnQvbWVtb3J5LXJzcyAtPiBhYnNvbHV0ZQo+ICAgICAgIGEwL3Zp cnQvbWVtb3J5LXRvdGFsIC0+IGFic29sdXRlCj4gICAgICAgYTAvdmlydC9wc19jcHV0aW1lIC0+ IHJhdGUKPiAgICAgICBhMC92aXJ0L3RvdGFsX3JlcXVlc3RzLWZsdXNoLWhkYyAtPiAgcmF0ZQo+ ICAgICAgIGEwL3ZpcnQvdG90YWxfcmVxdWVzdHMtZmx1c2gtdmRhCj4gICAgICAgYTAvdmlydC90 b3RhbF90aW1lX2luX21zLWZsdXNoLWhkYyAtPiByYXRlCj4gICAgICAgYTAvdmlydC90b3RhbF90 aW1lX2luX21zLWZsdXNoLXZkYQo+ICAgICAgIGEwL3ZpcnQvdmlydF9jcHVfdG90YWwgLT4gcmF0 ZQo+ICAgICAgIGEwL3ZpcnQvdmlydF92Y3B1LTAgLT4gcmF0ZQo+ICAgICAgIGEwL3ZpcnQvdmly dF92Y3B1LTEKPgo+ICAgICBjb2xsZWN0ZCAianVzdCIgcmVwb3J0cyB0aGUgY2hhbmdlcyBzaW5j ZSB0aGUgbGFzdCBzYW1wbGluZy4gSSdtCj4gICAgIG5vdCBzdXJlIHdoaWNoIGlzIHRoZSBiZXN0 IHdheSB0byBoYW5kbGUgdGhhdDsgSSd2ZSBzZW50IGEgbWFpbCB0bwo+ICAgICBjb2xsZWN0ZCBs aXN0IHNvbWUgdGltZSBhZ28sIG5vIGFuc3dlciBzbyBmYXIuCj4KPgo+IENhbiB5b3UgQ0Mgb24g dGhhdCB0aHJlYWQ/Cj4gSSBkb24ndCBrbm93IGhvdyBFUyB3b3VsZCB3b3JrIHdpdGggcmF0ZXMg YXQgYWxsLiAKPiBJIHdhbnQgdG8gYmUgYWJsZSB0byBzaG93IENQVSB1c2FnZSBvdmVyIHRpbWUg YW5kIEkgbmVlZCB0byBrbm93IGlmCj4gaXRzIDgwJSBvciAxMCUuCj4KClRoYW5rcyB0byB0aGUg YXdrd2FyZCBnbWFpbCBpbnRlcmZhY2UgSSBjYW4ndCByZXBseSB0byBteXNlbGYgYW5kIENDCm90 aGVyIHBlb3BsZSwgYnV0IEkgY2FuIHNoYXJlIHRoZSBsaW5rOgoKaHR0cHM6Ly9tYWlsbWFuLnZl cnBsYW50Lm9yZy9waXBlcm1haWwvY29sbGVjdGQvMjAxNy1KYW51YXJ5LzAwNjk2NS5odG1sCgot LSAKRnJhbmNlc2NvIFJvbWFuaQpSZWQgSGF0IEVuZ2luZWVyaW5nIFZpcnR1YWxpemF0aW9uIFIg JiBECklSQzogZnJvbWFuaQoKCi0tLS0tLS0tLS0tLS0tQTJGREFGMzAwMjA5RDk2Njg4QjNFOEND CkNvbnRlbnQtVHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNmZXIt RW5jb2Rpbmc6IDhiaXQKCjxodG1sPgogIDxoZWFkPgogICAgPG1ldGEgY29udGVudD0idGV4dC9o dG1sOyBjaGFyc2V0PXV0Zi04IiBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiPgogIDwvaGVhZD4K ICA8Ym9keSBiZ2NvbG9yPSIjRkZGRkZGIiB0ZXh0PSIjMDAwMDAwIj4KICAgIDxicj4KICAgIDxk aXYgY2xhc3M9Im1vei1jaXRlLXByZWZpeCI+T24gMDIvMjYvMjAxNyAwMzoxMyBQTSwgWWFuaXYg RGFyeQogICAgICB3cm90ZTo8YnI+CiAgICA8L2Rpdj4KICAgIDxibG9ja3F1b3RlCmNpdGU9Im1p ZDpDQUNLTUF5X2tmWD1MQ05lKzdFRkQyQ2p3X1dpdCthRFM4RlIrUlBLRW89Y01YVStIK0FAbWFp bC5nbWFpbC5jb20iCiAgICAgIHR5cGU9ImNpdGUiPgogICAgICA8ZGl2IGRpcj0ibHRyIj48YnI+ CiAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPgogICAgICAgICAgPGRpdiBjbGFzcz0i Z21haWxfcXVvdGUiPgogICAgICAgICAgICA8YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUi IHN0eWxlPSJtYXJnaW46MCAwIDAKICAgICAgICAgICAgICAuOGV4O2JvcmRlci1sZWZ0OjFweCAj Y2NjIHNvbGlkO3BhZGRpbmctbGVmdDoxZXgiPgogICAgICAgICAgICAgIDxkaXYgYmdjb2xvcj0i I0ZGRkZGRiIgdGV4dD0iIzAwMDAwMCI+CiAgICAgICAgICAgICAgICA8ZGl2PgogICAgICAgICAg ICAgICAgICA8ZGl2IGNsYXNzPSJoNSI+CiAgICAgICAgICAgICAgICAgICAgPGJsb2NrcXVvdGUg dHlwZT0iY2l0ZSI+CiAgICAgICAgICAgICAgICAgICAgICA8ZGl2IGRpcj0iYXV0byI+CiAgICAg ICAgICAgICAgICAgICAgICAgIDxkaXY+CiAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBj bGFzcz0iZ21haWxfZXh0cmEiPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBjbGFz cz0iZ21haWxfcXVvdGUiPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8YmxvY2txdW90 ZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNsYXNzPSJtXy02ODQ4NTAzOTUxNDY4 ODc1OTJxdW90ZSIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBzdHlsZT0ibWFyZ2lu OjAgMCAwIC44ZXg7Ym9yZGVyLWxlZnQ6MXB4CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij4gMi4KICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBjb2xsZWN0ZCAqaW50ZW50aW9uYWxseSogcmVwb3J0IG1ldHJpY3MKICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICBhcyByYXRlcywgbm90IGFzIGFic29sdXRlPGJyPgog ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHZhbHVlcyBhcyBWZHNtIGRvZXMuIFRoaXMg bWF5IGJlIG9uZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGlzc3VlIGluIHByZXNl bmNlIG9mIHJlc3RhcnRzL2RhdGE8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg bG9zcyBpbiB0aGUgbGluayBiZXR3ZWVuIGNvbGxlY3RkIGFuZAogICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIHRoZSBtZXRyaWNzIHN0b3JlLjxicj4KICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgPC9ibG9ja3F1b3RlPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+ CiAgICAgICAgICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgICAg IDwvZGl2PgogICAgICAgICAgICAgICAgICAgICAgICA8ZGl2IGRpcj0iYXV0byI+PGJyPgogICAg ICAgICAgICAgICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBk aXI9ImF1dG8iPkhvdyBkb2VzIHRoaXMgd29yaz/CoDwvZGl2PgogICAgICAgICAgICAgICAgICAg ICAgICA8ZGl2IGRpcj0iYXV0byI+SWYgd2Ugd2FudCB0byBzaG93IG1lbW9yeSB1c2FnZQogICAg ICAgICAgICAgICAgICAgICAgICAgIG92ZXIgdGltZSBmb3IgZXhhbXBsZSwgd2UgbmVlZCB0byBo YXZlIHRoZQogICAgICAgICAgICAgICAgICAgICAgICAgIHVzYWdlLCBub3QgdGhlIHJhdGUuwqA8 L2Rpdj4KICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBkaXI9ImF1dG8iPkhvdyB3b3VsZCB0 aGlzIGJlIHJlcG9ydGVkPyA8YnI+CiAgICAgICAgICAgICAgICAgICAgICAgIDwvZGl2PgogICAg ICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAgICAgICAgICAgICAgICAgPC9ibG9ja3F1b3Rl PgogICAgICAgICAgICAgICAgICAgIDxicj4KICAgICAgICAgICAgICAgICAgPC9kaXY+CiAgICAg ICAgICAgICAgICA8L2Rpdj4KICAgICAgICAgICAgICAgIEkgd2FzIGltcHJlY2lzZSwgbXkgZmF1 bHQuPGJyPgogICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAgTGV0IG1lIHJldHJ5 Ojxicj4KICAgICAgICAgICAgICAgIGNvbGxlY3RkIGludGVudGlvbmFsbHkgcmVwb3J0IHF1aXRl IGEgbG90IG9mIG1ldHJpY3Mgd2UKICAgICAgICAgICAgICAgIGNhcmUgYWJvdXQgYXMgcmF0ZXMs IG5vdCBhcyBhYnNvbHV0ZSB2YWx1ZXMuPGJyPgogICAgICAgICAgICAgICAgTWVtb3J5IGlzIGFj dHVhbGx5IG9rIGZpbmUuPGJyPgogICAgICAgICAgICAgICAgPGJyPgogICAgICAgICAgICAgICAg wqAgYTAvdmlydC9kaXNrX29jdGV0cy1oZGMgLSZndDsgcmF0ZTxicj4KICAgICAgICAgICAgICAg IMKgIGEwL3ZpcnQvZGlza19vY3RldHMtdmRhPGJyPgogICAgICAgICAgICAgICAgwqAgYTAvdmly dC9kaXNrX29wcy1oZGMgLSZndDsgcmF0ZTxicj4KICAgICAgICAgICAgICAgIMKgIGEwL3ZpcnQv ZGlza19vcHMtdmRhPGJyPgogICAgICAgICAgICAgICAgwqAgYTAvdmlydC9kaXNrX3RpbWUtaGRj IC0mZ3Q7IHJhdGU8YnI+CiAgICAgICAgICAgICAgICDCoCBhMC92aXJ0L2Rpc2tfdGltZS12ZGE8 YnI+CiAgICAgICAgICAgICAgICDCoCBhMC92aXJ0L2lmX2Ryb3BwZWQtdm5ldDAgLSZndDsgcmF0 ZTxicj4KICAgICAgICAgICAgICAgIMKgIGEwL3ZpcnQvaWZfZXJyb3JzLXZuZXQwIC0mZ3Q7IHJh dGU8YnI+CiAgICAgICAgICAgICAgICDCoCBhMC92aXJ0L2lmX29jdGV0cy12bmV0MCAtJmd0OyBy YXRlPGJyPgogICAgICAgICAgICAgICAgwqAgYTAvdmlydC9pZl9wYWNrZXRzLXZuZXQwIC0mZ3Q7 IHJhdGU8YnI+CiAgICAgICAgICAgICAgICDCoCBhMC92aXJ0L21lbW9yeS1hY3R1YWxfYmFsbG9v biAtJmd0OyBhYnNvbHV0ZTxicj4KICAgICAgICAgICAgICAgIMKgIGEwL3ZpcnQvbWVtb3J5LXJz cyAtJmd0OyBhYnNvbHV0ZTxicj4KICAgICAgICAgICAgICAgIMKgIGEwL3ZpcnQvbWVtb3J5LXRv dGFsIC0mZ3Q7IGFic29sdXRlPGJyPgogICAgICAgICAgICAgICAgwqAgYTAvdmlydC9wc19jcHV0 aW1lIC0mZ3Q7IHJhdGU8YnI+CiAgICAgICAgICAgICAgICDCoCBhMC92aXJ0L3RvdGFsX3JlcXVl c3RzLWZsdXNoLTx3YnI+aGRjIC0mZ3Q7wqAgcmF0ZTxicj4KICAgICAgICAgICAgICAgIMKgIGEw L3ZpcnQvdG90YWxfcmVxdWVzdHMtZmx1c2gtPHdicj52ZGE8YnI+CiAgICAgICAgICAgICAgICDC oCBhMC92aXJ0L3RvdGFsX3RpbWVfaW5fbXMtPHdicj5mbHVzaC1oZGMgLSZndDsgcmF0ZTxicj4K ICAgICAgICAgICAgICAgIMKgIGEwL3ZpcnQvdG90YWxfdGltZV9pbl9tcy08d2JyPmZsdXNoLXZk YTxicj4KICAgICAgICAgICAgICAgIMKgIGEwL3ZpcnQvdmlydF9jcHVfdG90YWwgLSZndDsgcmF0 ZTxicj4KICAgICAgICAgICAgICAgIMKgIGEwL3ZpcnQvdmlydF92Y3B1LTAgLSZndDsgcmF0ZTxi cj4KICAgICAgICAgICAgICAgIMKgIGEwL3ZpcnQvdmlydF92Y3B1LTE8YnI+CiAgICAgICAgICAg ICAgICA8YnI+CiAgICAgICAgICAgICAgICBjb2xsZWN0ZCAianVzdCIgcmVwb3J0cyB0aGUgY2hh bmdlcyBzaW5jZSB0aGUgbGFzdAogICAgICAgICAgICAgICAgc2FtcGxpbmcuIEknbSBub3Qgc3Vy ZSB3aGljaCBpcyB0aGUgYmVzdCB3YXkgdG8gaGFuZGxlCiAgICAgICAgICAgICAgICB0aGF0OyBJ J3ZlIHNlbnQgYSBtYWlsIHRvIGNvbGxlY3RkIGxpc3Qgc29tZSB0aW1lIGFnbywKICAgICAgICAg ICAgICAgIG5vIGFuc3dlciBzbyBmYXIuPC9kaXY+CiAgICAgICAgICAgIDwvYmxvY2txdW90ZT4K ICAgICAgICAgICAgPGRpdj48YnI+CiAgICAgICAgICAgIDwvZGl2PgogICAgICAgICAgICA8ZGl2 PkNhbiB5b3UgQ0Mgb24gdGhhdCB0aHJlYWQ/PC9kaXY+CiAgICAgICAgICAgIDxkaXY+SSBkb24n dCBrbm93IGhvdyBFUyB3b3VsZCB3b3JrIHdpdGggcmF0ZXMgYXQgYWxsLsKgPC9kaXY+CiAgICAg ICAgICAgIDxkaXY+SSB3YW50IHRvIGJlIGFibGUgdG8gc2hvdyBDUFUgdXNhZ2Ugb3ZlciB0aW1l IGFuZCBJCiAgICAgICAgICAgICAgbmVlZCB0byBrbm93IGlmIGl0cyA4MCUgb3IgMTAlLjwvZGl2 PgogICAgICAgICAgICA8YnI+CiAgICAgICAgICA8L2Rpdj4KICAgICAgICA8L2Rpdj4KICAgICAg PC9kaXY+CiAgICA8L2Jsb2NrcXVvdGU+CiAgICA8YnI+CiAgICBUaGFua3MgdG8gdGhlIGF3a3dh cmQgZ21haWwgaW50ZXJmYWNlIEkgY2FuJ3QgcmVwbHkgdG8gbXlzZWxmIGFuZCBDQwogICAgb3Ro ZXIgcGVvcGxlLCBidXQgSSBjYW4gc2hhcmUgdGhlIGxpbms6PGJyPgogICAgPGJyPgo8YSBjbGFz cz0ibW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwczovL21haWxtYW4udmVycGxhbnQu b3JnL3BpcGVybWFpbC9jb2xsZWN0ZC8yMDE3LUphbnVhcnkvMDA2OTY1Lmh0bWwiPmh0dHBzOi8v bWFpbG1hbi52ZXJwbGFudC5vcmcvcGlwZXJtYWlsL2NvbGxlY3RkLzIwMTctSmFudWFyeS8wMDY5 NjUuaHRtbDwvYT48YnI+CiAgICA8YnI+CiAgICA8cHJlIGNsYXNzPSJtb3otc2lnbmF0dXJlIiBj b2xzPSI3MiI+LS0gCkZyYW5jZXNjbyBSb21hbmkKUmVkIEhhdCBFbmdpbmVlcmluZyBWaXJ0dWFs aXphdGlvbiBSICZhbXA7IEQKSVJDOiBmcm9tYW5pPC9wcmU+CiAgPC9ib2R5Pgo8L2h0bWw+Cgot LS0tLS0tLS0tLS0tLUEyRkRBRjMwMDIwOUQ5NjY4OEIzRThDQy0tCg== --===============0044786323360172455==--