<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Feb 21, 2017 at 4:21 PM, Michal Skrivanek <span dir="ltr"><<a href="mailto:michal.skrivanek@redhat.com" target="_blank">michal.skrivanek@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><br>
> On 21 Feb 2017, at 14:44, Yaniv Kaul <<a href="mailto:ykaul@redhat.com">ykaul@redhat.com</a>> wrote:<br>
><br>
><br>
><br>
> On Tue, Feb 21, 2017 at 1:06 PM Francesco Romani <<a href="mailto:fromani@redhat.com">fromani@redhat.com</a>> wrote:<br>
> Hello everyone,<br>
><br>
><br>
> in the last weeks I've been submitting PRs to collectd upstream, to<br>
> bring the virt plugin up to date with Vdsm and oVirt needs.<br>
><br>
> Previously, the collectd virt plugin reported only a subset of metrics<br>
> oVirt uses.<br>
><br>
> In current collectd master, the collectd virt plugin provides all the<br>
> data Vdsm (thus Engine) needs. This means that it is now<br>
><br>
> possible for Vdsm or Engine to query collectd, not Vdsm/libvirt, and<br>
> have the same data.<br>
><br>
> Do we wish to ship the unixsock collectd plugin? I'm not sure we do these days (4.1).<br>
> We can do that later, of course, when we ship this.<br>
<br>
</span>we haven’t decided on the actual solution yet, unixsocket is one possibility.<br>
it is tracked in <a href="https://trello.com/c/alAOm1tQ" rel="noreferrer" target="_blank">https://trello.com/c/alAOm1tQ</a><br>
<br>
we can also have engine pulling it from collectd remotely, then we can eliminate periodic get vm stats<br>
or another (crazy) option to use fluentd to push data straight to engine’s postgres:)<br></blockquote><div><br></div><div>why crazy? sounds like where we want to be, without going through vdsm at all</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Thanks,<br>
michal<br>
<div class="HOEnZb"><div class="h5"><br>
> Y.<br>
><br>
><br>
><br>
> There are only two caveats:<br>
><br>
> 1. it is yet to be seen which version of collectd will ship all those<br>
> enhancements<br>
><br>
> 2. collectd *intentionally* report metrics as rates, not as absolute<br>
> values as Vdsm does. This may be one issue in presence of restarts/data<br>
> loss in the link between collectd and the metrics store.<br>
><br>
><br>
> Please keep reading for more details:<br>
><br>
><br>
> How to get the code?<br>
><br>
> ------------------------------<wbr>--<br>
><br>
> This somehow tricky until we get one official release. If one is<br>
> familiar with the RPM build process, it is easy to build one custom packages<br>
><br>
> from a snapshot from collectd master<br>
> (<a href="https://github.com/collectd/collectd" rel="noreferrer" target="_blank">https://github.com/collectd/<wbr>collectd</a>) and a recent 5.7.1 RPM (like<br>
> <a href="https://koji.fedoraproject.org/koji/buildinfo?buildID=835669" rel="noreferrer" target="_blank">https://koji.fedoraproject.<wbr>org/koji/buildinfo?buildID=<wbr>835669</a>)<br>
><br>
><br>
> How to configure it?<br>
><br>
> ------------------------------<br>
><br>
> Most thing work out of the box. One currently in progress Vdsm patch<br>
> ships the recommended configuration<br>
> <a href="https://gerrit.ovirt.org/#/c/71176/6/static/etc/collectd.d/virt.conf" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>71176/6/static/etc/collectd.d/<wbr>virt.conf</a><br>
><br>
> The meaning of the configuration option is documented in man 5 collectd.conf<br>
><br>
><br>
> How it looks like?<br>
><br>
> --------------------------<br>
><br>
><br>
> Let me post one "screenshot" :)<br>
><br>
><br>
><br>
> $ collectdctl listval | grep a0<br>
> a0/virt/disk_octets-hdc<br>
> a0/virt/disk_octets-vda<br>
> a0/virt/disk_ops-hdc<br>
> a0/virt/disk_ops-vda<br>
> a0/virt/disk_time-hdc<br>
> a0/virt/disk_time-vda<br>
> a0/virt/if_dropped-vnet0<br>
> a0/virt/if_errors-vnet0<br>
> a0/virt/if_octets-vnet0<br>
> a0/virt/if_packets-vnet0<br>
> a0/virt/memory-actual_balloon<br>
> a0/virt/memory-rss<br>
> a0/virt/memory-total<br>
> a0/virt/ps_cputime<br>
> a0/virt/total_requests-flush-<wbr>hdc<br>
> a0/virt/total_requests-flush-<wbr>vda<br>
> a0/virt/total_time_in_ms-<wbr>flush-hdc<br>
> a0/virt/total_time_in_ms-<wbr>flush-vda<br>
> a0/virt/virt_cpu_total<br>
> a0/virt/virt_vcpu-0<br>
> a0/virt/virt_vcpu-1<br>
><br>
><br>
> How to consume the data?<br>
> ------------------------------<wbr>-----------<br>
><br>
> Among the ways to query collectd, the two most popular (and most fitting<br>
> for oVirt use case) ways are perhaps the network protocol<br>
> (<a href="https://collectd.org/wiki/index.php/Binary_protocol" rel="noreferrer" target="_blank">https://collectd.org/wiki/<wbr>index.php/Binary_protocol</a>)<br>
> and the plain text protocol<br>
> (<a href="https://collectd.org/wiki/index.php/Plain_text_protocol" rel="noreferrer" target="_blank">https://collectd.org/wiki/<wbr>index.php/Plain_text_protocol</a>)<wbr>. The first<br>
> could be used by Engine to get the data directly, or to consolidate the<br>
> metrics in one database (e.g to run any kind of query, for historical<br>
> series...).<br>
> The latter will be used by Vdsm to keep reporting the metrics (again<br>
> <a href="https://gerrit.ovirt.org/#/c/71176/6" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>71176/6</a>)<br>
><br>
> Please note that the performance of the plain text protocol are known to<br>
> be lower than the binary protocol<br>
><br>
> What about the unresponsive hosts?<br>
> ------------------------------<wbr>-------------------------<br>
><br>
> We know from experience that hosts may become unresponsive, and this can<br>
> disrupt monitoring. however, we do want to keep monitoring the<br>
> responsive hosts, avoiding that one rogue hosts makes us lose all the<br>
> monitoring data.<br>
> To cope with this need, the virt plugin gained support for "partition<br>
> tag". With this, we can group VMs together using one arbitrary tag. This<br>
> is completely transparent to collectd, and also completely optional.<br>
> oVirt can use this tag to group VMs per-storage-domain, or however it<br>
> sees fit, trying to minimize the disruption should one host become<br>
> unresponsive.<br>
><br>
> Read the full docs here:<br>
> <a href="https://github.com/collectd/collectd/commit/999efc28d8e2e96bc15f535254d412a79755ca4f" rel="noreferrer" target="_blank">https://github.com/collectd/<wbr>collectd/commit/<wbr>999efc28d8e2e96bc15f535254d412<wbr>a79755ca4f</a><br>
><br>
><br>
> What about the collectd-ovirt plugin?<br>
> ------------------------------<wbr>--------------------------<br>
><br>
> Some time ago I implemented one out-of-tree collectd plugin leveraging<br>
> the libvirt bulk stats: <a href="https://github.com/fromanirh/collectd-ovirt" rel="noreferrer" target="_blank">https://github.com/fromanirh/<wbr>collectd-ovirt</a><br>
> This plugin is meant to be a modern, drop-in replacement for the<br>
> existing virt plugin.<br>
> The development of that out of tree plugin is now halted, because we<br>
> have everything we need in the upstream collectd plugin.<br>
><br>
> Future work<br>
> ------------------<br>
><br>
> We believe we have reached feature parity, so we are looking for<br>
> bugixes/performance tuning in the near term future. I'll be happy to<br>
> provide more patches/PRs about that.<br>
><br>
><br>
><br>
> Thanks and bests,<br>
><br>
> --<br>
> Francesco Romani<br>
> Red Hat Engineering Virtualization R & D<br>
> IRC: fromani<br>
><br>
> ______________________________<wbr>_________________<br>
> Devel mailing list<br>
> <a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
> ______________________________<wbr>_________________<br>
> Devel mailing list<br>
> <a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
<br>
______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a></div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><span style="font-size:12.8px"><b>Yaniv Bronhaim.</b></span><br></div></div></div></div></div>
</div></div>