<div dir="auto"><div><br><br><div data-smartmail="gmail_signature">Yaniv Dary<br>Technical Product Manager<br>Red Hat Israel Ltd.<br>34 Jerusalem Road<br>Building A, 4th floor<br>Ra'anana, Israel 4350109<br><br>Tel : +972 (9) 7692306<br> 8272306<br>Email: <a href="mailto:ydary@redhat.com">ydary@redhat.com</a><br>IRC : ydary</div><div class="gmail_extra"><br><div class="gmail_quote">On Feb 21, 2017 13:06, "Francesco Romani" <<a href="mailto:fromani@redhat.com">fromani@redhat.com</a>> wrote:<br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello everyone,<br>
<br>
<br>
in the last weeks I've been submitting PRs to collectd upstream, to<br>
bring the virt plugin up to date with Vdsm and oVirt needs.<br>
<br>
Previously, the collectd virt plugin reported only a subset of metrics<br>
oVirt uses.<br>
<br>
In current collectd master, the collectd virt plugin provides all the<br>
data Vdsm (thus Engine) needs. This means that it is now<br>
<br>
possible for Vdsm or Engine to query collectd, not Vdsm/libvirt, and<br>
have the same data.<br>
<br>
<br>
There are only two caveats:<br>
<br>
1. it is yet to be seen which version of collectd will ship all those<br>
enhancements<br>
<br>
2. collectd *intentionally* report metrics as rates, not as absolute<br>
values as Vdsm does. This may be one issue in presence of restarts/data<br>
loss in the link between collectd and the metrics store.<br></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">How does this work? </div><div dir="auto">If we want to show memory usage over time for example, we need to have the usage, not the rate. </div><div dir="auto">How would this be reported? </div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
Please keep reading for more details:<br>
<br>
<br>
How to get the code?<br>
<br>
------------------------------<wbr>--<br>
<br>
This somehow tricky until we get one official release. If one is<br>
familiar with the RPM build process, it is easy to build one custom packages<br>
<br>
from a snapshot from collectd master<br>
(<a href="https://github.com/collectd/collectd" rel="noreferrer" target="_blank">https://github.com/collectd/<wbr>collectd</a>) and a recent 5.7.1 RPM (like<br>
<a href="https://koji.fedoraproject.org/koji/buildinfo?buildID=835669" rel="noreferrer" target="_blank">https://koji.fedoraproject.<wbr>org/koji/buildinfo?buildID=<wbr>835669</a>)<br>
<br>
<br>
How to configure it?<br>
<br>
------------------------------<br>
<br>
Most thing work out of the box. One currently in progress Vdsm patch<br>
ships the recommended configuration<br>
<a href="https://gerrit.ovirt.org/#/c/71176/6/static/etc/collectd.d/virt.conf" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>71176/6/static/etc/collectd.d/<wbr>virt.conf</a><br>
<br>
The meaning of the configuration option is documented in man 5 collectd.conf<br>
<br>
<br>
How it looks like?<br>
<br>
--------------------------<br>
<br>
<br>
Let me post one "screenshot" :)<br>
<br>
<br>
<br>
$ collectdctl listval | grep a0<br>
a0/virt/disk_octets-hdc<br>
a0/virt/disk_octets-vda<br>
a0/virt/disk_ops-hdc<br>
a0/virt/disk_ops-vda<br>
a0/virt/disk_time-hdc<br>
a0/virt/disk_time-vda<br>
a0/virt/if_dropped-vnet0<br>
a0/virt/if_errors-vnet0<br>
a0/virt/if_octets-vnet0<br>
a0/virt/if_packets-vnet0<br>
a0/virt/memory-actual_balloon<br>
a0/virt/memory-rss<br>
a0/virt/memory-total<br>
a0/virt/ps_cputime<br>
a0/virt/total_requests-flush-<wbr>hdc<br>
a0/virt/total_requests-flush-<wbr>vda<br>
a0/virt/total_time_in_ms-<wbr>flush-hdc<br>
a0/virt/total_time_in_ms-<wbr>flush-vda<br>
a0/virt/virt_cpu_total<br>
a0/virt/virt_vcpu-0<br>
a0/virt/virt_vcpu-1<br>
<br>
<br>
How to consume the data?<br>
------------------------------<wbr>-----------<br>
<br>
Among the ways to query collectd, the two most popular (and most fitting<br>
for oVirt use case) ways are perhaps the network protocol<br>
(<a href="https://collectd.org/wiki/index.php/Binary_protocol" rel="noreferrer" target="_blank">https://collectd.org/wiki/<wbr>index.php/Binary_protocol</a>)<br>
and the plain text protocol<br>
(<a href="https://collectd.org/wiki/index.php/Plain_text_protocol" rel="noreferrer" target="_blank">https://collectd.org/wiki/<wbr>index.php/Plain_text_protocol</a>)<wbr>. The first<br>
could be used by Engine to get the data directly, or to consolidate the<br>
metrics in one database (e.g to run any kind of query, for historical<br>
series...).<br>
The latter will be used by Vdsm to keep reporting the metrics (again<br>
<a href="https://gerrit.ovirt.org/#/c/71176/6" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>71176/6</a>)<br>
<br>
Please note that the performance of the plain text protocol are known to<br>
be lower than the binary protocol<br>
<br>
What about the unresponsive hosts?<br>
------------------------------<wbr>-------------------------<br>
<br>
We know from experience that hosts may become unresponsive, and this can<br>
disrupt monitoring. however, we do want to keep monitoring the<br>
responsive hosts, avoiding that one rogue hosts makes us lose all the<br>
monitoring data.<br>
To cope with this need, the virt plugin gained support for "partition<br>
tag". With this, we can group VMs together using one arbitrary tag. This<br>
is completely transparent to collectd, and also completely optional.<br>
oVirt can use this tag to group VMs per-storage-domain, or however it<br>
sees fit, trying to minimize the disruption should one host become<br>
unresponsive.<br>
<br>
Read the full docs here:<br>
<a href="https://github.com/collectd/collectd/commit/999efc28d8e2e96bc15f535254d412a79755ca4f" rel="noreferrer" target="_blank">https://github.com/collectd/<wbr>collectd/commit/<wbr>999efc28d8e2e96bc15f535254d412<wbr>a79755ca4f</a><br>
<br>
<br>
What about the collectd-ovirt plugin?<br>
------------------------------<wbr>--------------------------<br>
<br>
Some time ago I implemented one out-of-tree collectd plugin leveraging<br>
the libvirt bulk stats: <a href="https://github.com/fromanirh/collectd-ovirt" rel="noreferrer" target="_blank">https://github.com/fromanirh/<wbr>collectd-ovirt</a><br>
This plugin is meant to be a modern, drop-in replacement for the<br>
existing virt plugin.<br>
The development of that out of tree plugin is now halted, because we<br>
have everything we need in the upstream collectd plugin.<br>
<br>
Future work<br>
------------------<br>
<br>
We believe we have reached feature parity, so we are looking for<br>
bugixes/performance tuning in the near term future. I'll be happy to<br>
provide more patches/PRs about that.<br>
<br>
<br>
<br>
Thanks and bests,<br>
<font color="#888888"><br>
--<br>
Francesco Romani<br>
Red Hat Engineering Virtualization R & D<br>
IRC: fromani<br>
<br>
______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
</font></blockquote></div><br></div></div></div>