<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Tue, Feb 21, 2017 at 1:06 PM Francesco Romani <<a href="mailto:fromani@redhat.com">fromani@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello everyone,<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
in the last weeks I've been submitting PRs to collectd upstream, to<br class="gmail_msg">
bring the virt plugin up to date with Vdsm and oVirt needs.<br class="gmail_msg">
<br class="gmail_msg">
Previously, the collectd virt plugin reported only a subset of metrics<br class="gmail_msg">
oVirt uses.<br class="gmail_msg">
<br class="gmail_msg">
In current collectd master, the collectd virt plugin provides all the<br class="gmail_msg">
data Vdsm (thus Engine) needs. This means that it is now<br class="gmail_msg">
<br class="gmail_msg">
possible for Vdsm or Engine to query collectd, not Vdsm/libvirt, and<br class="gmail_msg">
have the same data.<br class="gmail_msg"></blockquote><div><br></div><div>Do we wish to ship the unixsock collectd plugin? I'm not sure we do these days (4.1).</div><div>We can do that later, of course, when we ship this.</div><div>Y.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="gmail_msg">
<br class="gmail_msg">
There are only two caveats:<br class="gmail_msg">
<br class="gmail_msg">
1. it is yet to be seen which version of collectd will ship all those<br class="gmail_msg">
enhancements<br class="gmail_msg">
<br class="gmail_msg">
2. collectd *intentionally* report metrics as rates, not as absolute<br class="gmail_msg">
values as Vdsm does. This may be one issue in presence of restarts/data<br class="gmail_msg">
loss in the link between collectd and the metrics store.<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
Please keep reading for more details:<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
How to get the code?<br class="gmail_msg">
<br class="gmail_msg">
--------------------------------<br class="gmail_msg">
<br class="gmail_msg">
This somehow tricky until we get one official release. If one is<br class="gmail_msg">
familiar with the RPM build process, it is easy to build one custom packages<br class="gmail_msg">
<br class="gmail_msg">
from a snapshot from collectd master<br class="gmail_msg">
(<a href="https://github.com/collectd/collectd" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/collectd/collectd</a>) and a recent 5.7.1 RPM (like<br class="gmail_msg">
<a href="https://koji.fedoraproject.org/koji/buildinfo?buildID=835669" rel="noreferrer" class="gmail_msg" target="_blank">https://koji.fedoraproject.org/koji/buildinfo?buildID=835669</a>)<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
How to configure it?<br class="gmail_msg">
<br class="gmail_msg">
------------------------------<br class="gmail_msg">
<br class="gmail_msg">
Most thing work out of the box. One currently in progress Vdsm patch<br class="gmail_msg">
ships the recommended configuration<br class="gmail_msg">
<a href="https://gerrit.ovirt.org/#/c/71176/6/static/etc/collectd.d/virt.conf" rel="noreferrer" class="gmail_msg" target="_blank">https://gerrit.ovirt.org/#/c/71176/6/static/etc/collectd.d/virt.conf</a><br class="gmail_msg">
<br class="gmail_msg">
The meaning of the configuration option is documented in man 5 collectd.conf<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
How it looks like?<br class="gmail_msg">
<br class="gmail_msg">
--------------------------<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
Let me post one "screenshot" :)<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
$ collectdctl listval | grep a0<br class="gmail_msg">
a0/virt/disk_octets-hdc<br class="gmail_msg">
a0/virt/disk_octets-vda<br class="gmail_msg">
a0/virt/disk_ops-hdc<br class="gmail_msg">
a0/virt/disk_ops-vda<br class="gmail_msg">
a0/virt/disk_time-hdc<br class="gmail_msg">
a0/virt/disk_time-vda<br class="gmail_msg">
a0/virt/if_dropped-vnet0<br class="gmail_msg">
a0/virt/if_errors-vnet0<br class="gmail_msg">
a0/virt/if_octets-vnet0<br class="gmail_msg">
a0/virt/if_packets-vnet0<br class="gmail_msg">
a0/virt/memory-actual_balloon<br class="gmail_msg">
a0/virt/memory-rss<br class="gmail_msg">
a0/virt/memory-total<br class="gmail_msg">
a0/virt/ps_cputime<br class="gmail_msg">
a0/virt/total_requests-flush-hdc<br class="gmail_msg">
a0/virt/total_requests-flush-vda<br class="gmail_msg">
a0/virt/total_time_in_ms-flush-hdc<br class="gmail_msg">
a0/virt/total_time_in_ms-flush-vda<br class="gmail_msg">
a0/virt/virt_cpu_total<br class="gmail_msg">
a0/virt/virt_vcpu-0<br class="gmail_msg">
a0/virt/virt_vcpu-1<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
How to consume the data?<br class="gmail_msg">
-----------------------------------------<br class="gmail_msg">
<br class="gmail_msg">
Among the ways to query collectd, the two most popular (and most fitting<br class="gmail_msg">
for oVirt use case) ways are perhaps the network protocol<br class="gmail_msg">
(<a href="https://collectd.org/wiki/index.php/Binary_protocol" rel="noreferrer" class="gmail_msg" target="_blank">https://collectd.org/wiki/index.php/Binary_protocol</a>)<br class="gmail_msg">
and the plain text protocol<br class="gmail_msg">
(<a href="https://collectd.org/wiki/index.php/Plain_text_protocol" rel="noreferrer" class="gmail_msg" target="_blank">https://collectd.org/wiki/index.php/Plain_text_protocol</a>). The first<br class="gmail_msg">
could be used by Engine to get the data directly, or to consolidate the<br class="gmail_msg">
metrics in one database (e.g to run any kind of query, for historical<br class="gmail_msg">
series...).<br class="gmail_msg">
The latter will be used by Vdsm to keep reporting the metrics (again<br class="gmail_msg">
<a href="https://gerrit.ovirt.org/#/c/71176/6" rel="noreferrer" class="gmail_msg" target="_blank">https://gerrit.ovirt.org/#/c/71176/6</a>)<br class="gmail_msg">
<br class="gmail_msg">
Please note that the performance of the plain text protocol are known to<br class="gmail_msg">
be lower than the binary protocol<br class="gmail_msg">
<br class="gmail_msg">
What about the unresponsive hosts?<br class="gmail_msg">
-------------------------------------------------------<br class="gmail_msg">
<br class="gmail_msg">
We know from experience that hosts may become unresponsive, and this can<br class="gmail_msg">
disrupt monitoring. however, we do want to keep monitoring the<br class="gmail_msg">
responsive hosts, avoiding that one rogue hosts makes us lose all the<br class="gmail_msg">
monitoring data.<br class="gmail_msg">
To cope with this need, the virt plugin gained support for "partition<br class="gmail_msg">
tag". With this, we can group VMs together using one arbitrary tag. This<br class="gmail_msg">
is completely transparent to collectd, and also completely optional.<br class="gmail_msg">
oVirt can use this tag to group VMs per-storage-domain, or however it<br class="gmail_msg">
sees fit, trying to minimize the disruption should one host become<br class="gmail_msg">
unresponsive.<br class="gmail_msg">
<br class="gmail_msg">
Read the full docs here:<br class="gmail_msg">
<a href="https://github.com/collectd/collectd/commit/999efc28d8e2e96bc15f535254d412a79755ca4f" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/collectd/collectd/commit/999efc28d8e2e96bc15f535254d412a79755ca4f</a><br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
What about the collectd-ovirt plugin?<br class="gmail_msg">
--------------------------------------------------------<br class="gmail_msg">
<br class="gmail_msg">
Some time ago I implemented one out-of-tree collectd plugin leveraging<br class="gmail_msg">
the libvirt bulk stats: <a href="https://github.com/fromanirh/collectd-ovirt" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/fromanirh/collectd-ovirt</a><br class="gmail_msg">
This plugin is meant to be a modern, drop-in replacement for the<br class="gmail_msg">
existing virt plugin.<br class="gmail_msg">
The development of that out of tree plugin is now halted, because we<br class="gmail_msg">
have everything we need in the upstream collectd plugin.<br class="gmail_msg">
<br class="gmail_msg">
Future work<br class="gmail_msg">
------------------<br class="gmail_msg">
<br class="gmail_msg">
We believe we have reached feature parity, so we are looking for<br class="gmail_msg">
bugixes/performance tuning in the near term future. I'll be happy to<br class="gmail_msg">
provide more patches/PRs about that.<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
Thanks and bests,<br class="gmail_msg">
<br class="gmail_msg">
--<br class="gmail_msg">
Francesco Romani<br class="gmail_msg">
Red Hat Engineering Virtualization R & D<br class="gmail_msg">
IRC: fromani<br class="gmail_msg">
<br class="gmail_msg">
_______________________________________________<br class="gmail_msg">
Devel mailing list<br class="gmail_msg">
<a href="mailto:Devel@ovirt.org" class="gmail_msg" target="_blank">Devel@ovirt.org</a><br class="gmail_msg">
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.ovirt.org/mailman/listinfo/devel</a><br class="gmail_msg">
</blockquote></div></div>