<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Tue, Feb 21, 2017 at 6:40 PM Yaniv Bronheim <<a href="mailto:ybronhei@redhat.com">ybronhei@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg">On Tue, Feb 21, 2017 at 4:21 PM, Michal Skrivanek <span dir="ltr" class="gmail_msg"><<a href="mailto:michal.skrivanek@redhat.com" class="gmail_msg" target="_blank">michal.skrivanek@redhat.com</a>></span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="gmail_msg"><br class="gmail_msg">
> On 21 Feb 2017, at 14:44, Yaniv Kaul <<a href="mailto:ykaul@redhat.com" class="gmail_msg" target="_blank">ykaul@redhat.com</a>> wrote:<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> On Tue, Feb 21, 2017 at 1:06 PM Francesco Romani <<a href="mailto:fromani@redhat.com" class="gmail_msg" target="_blank">fromani@redhat.com</a>> wrote:<br class="gmail_msg">
> Hello everyone,<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> in the last weeks I've been submitting PRs to collectd upstream, to<br class="gmail_msg">
> bring the virt plugin up to date with Vdsm and oVirt needs.<br class="gmail_msg">
><br class="gmail_msg">
> Previously, the collectd virt plugin reported only a subset of metrics<br class="gmail_msg">
> oVirt uses.<br class="gmail_msg">
><br class="gmail_msg">
> In current collectd master, the collectd virt plugin provides all the<br class="gmail_msg">
> data Vdsm (thus Engine) needs. This means that it is now<br class="gmail_msg">
><br class="gmail_msg">
> possible for Vdsm or Engine to query collectd, not Vdsm/libvirt, and<br class="gmail_msg">
> have the same data.<br class="gmail_msg">
><br class="gmail_msg">
> Do we wish to ship the unixsock collectd plugin? I'm not sure we do these days (4.1).<br class="gmail_msg">
> We can do that later, of course, when we ship this.<br class="gmail_msg">
<br class="gmail_msg">
</span>we haven’t decided on the actual solution yet, unixsocket is one possibility.<br class="gmail_msg">
it is tracked in <a href="https://trello.com/c/alAOm1tQ" rel="noreferrer" class="gmail_msg" target="_blank">https://trello.com/c/alAOm1tQ</a><br class="gmail_msg">
<br class="gmail_msg">
we can also have engine pulling it from collectd remotely, then we can eliminate periodic get vm stats<br class="gmail_msg">
or another (crazy) option to use fluentd to push data straight to engine’s postgres:)<br class="gmail_msg"></blockquote></div></div></div></blockquote><div><br></div><div>I think that we should start first with changing engine internal - introduce a client and service to fetch stats in the engine. It's implementation should be flexible but with stable API. First it will fetch all stats from the db, later can change. The engine internals that needs stats should imediatly start to adapt - instead of db calls or direct entity methods, fetch from a (REST probably) service. After that the door is open to whatever needed.<br> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">why crazy? sounds like where we want to be, without going through vdsm at all</div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="gmail_msg">
Thanks,<br class="gmail_msg">
michal<br class="gmail_msg">
<div class="m_-6973989591943094584HOEnZb gmail_msg"><div class="m_-6973989591943094584h5 gmail_msg"><br class="gmail_msg">
> Y.<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> There are only two caveats:<br class="gmail_msg">
><br class="gmail_msg">
> 1. it is yet to be seen which version of collectd will ship all those<br class="gmail_msg">
> enhancements<br class="gmail_msg">
><br class="gmail_msg">
> 2. collectd *intentionally* report metrics as rates, not as absolute<br class="gmail_msg">
> values as Vdsm does. This may be one issue in presence of restarts/data<br class="gmail_msg">
> loss in the link between collectd and the metrics store.<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> Please keep reading for more details:<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> How to get the code?<br class="gmail_msg">
><br class="gmail_msg">
> --------------------------------<br class="gmail_msg">
><br class="gmail_msg">
> This somehow tricky until we get one official release. If one is<br class="gmail_msg">
> familiar with the RPM build process, it is easy to build one custom packages<br class="gmail_msg">
><br class="gmail_msg">
> from a snapshot from collectd master<br class="gmail_msg">
> (<a href="https://github.com/collectd/collectd" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/collectd/collectd</a>) and a recent 5.7.1 RPM (like<br class="gmail_msg">
> <a href="https://koji.fedoraproject.org/koji/buildinfo?buildID=835669" rel="noreferrer" class="gmail_msg" target="_blank">https://koji.fedoraproject.org/koji/buildinfo?buildID=835669</a>)<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> How to configure it?<br class="gmail_msg">
><br class="gmail_msg">
> ------------------------------<br class="gmail_msg">
><br class="gmail_msg">
> Most thing work out of the box. One currently in progress Vdsm patch<br class="gmail_msg">
> ships the recommended configuration<br class="gmail_msg">
> <a href="https://gerrit.ovirt.org/#/c/71176/6/static/etc/collectd.d/virt.conf" rel="noreferrer" class="gmail_msg" target="_blank">https://gerrit.ovirt.org/#/c/71176/6/static/etc/collectd.d/virt.conf</a><br class="gmail_msg">
><br class="gmail_msg">
> The meaning of the configuration option is documented in man 5 collectd.conf<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> How it looks like?<br class="gmail_msg">
><br class="gmail_msg">
> --------------------------<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> Let me post one "screenshot" :)<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> $ collectdctl listval | grep a0<br class="gmail_msg">
> a0/virt/disk_octets-hdc<br class="gmail_msg">
> a0/virt/disk_octets-vda<br class="gmail_msg">
> a0/virt/disk_ops-hdc<br class="gmail_msg">
> a0/virt/disk_ops-vda<br class="gmail_msg">
> a0/virt/disk_time-hdc<br class="gmail_msg">
> a0/virt/disk_time-vda<br class="gmail_msg">
> a0/virt/if_dropped-vnet0<br class="gmail_msg">
> a0/virt/if_errors-vnet0<br class="gmail_msg">
> a0/virt/if_octets-vnet0<br class="gmail_msg">
> a0/virt/if_packets-vnet0<br class="gmail_msg">
> a0/virt/memory-actual_balloon<br class="gmail_msg">
> a0/virt/memory-rss<br class="gmail_msg">
> a0/virt/memory-total<br class="gmail_msg">
> a0/virt/ps_cputime<br class="gmail_msg">
> a0/virt/total_requests-flush-hdc<br class="gmail_msg">
> a0/virt/total_requests-flush-vda<br class="gmail_msg">
> a0/virt/total_time_in_ms-flush-hdc<br class="gmail_msg">
> a0/virt/total_time_in_ms-flush-vda<br class="gmail_msg">
> a0/virt/virt_cpu_total<br class="gmail_msg">
> a0/virt/virt_vcpu-0<br class="gmail_msg">
> a0/virt/virt_vcpu-1<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> How to consume the data?<br class="gmail_msg">
> -----------------------------------------<br class="gmail_msg">
><br class="gmail_msg">
> Among the ways to query collectd, the two most popular (and most fitting<br class="gmail_msg">
> for oVirt use case) ways are perhaps the network protocol<br class="gmail_msg">
> (<a href="https://collectd.org/wiki/index.php/Binary_protocol" rel="noreferrer" class="gmail_msg" target="_blank">https://collectd.org/wiki/index.php/Binary_protocol</a>)<br class="gmail_msg">
> and the plain text protocol<br class="gmail_msg">
> (<a href="https://collectd.org/wiki/index.php/Plain_text_protocol" rel="noreferrer" class="gmail_msg" target="_blank">https://collectd.org/wiki/index.php/Plain_text_protocol</a>). The first<br class="gmail_msg">
> could be used by Engine to get the data directly, or to consolidate the<br class="gmail_msg">
> metrics in one database (e.g to run any kind of query, for historical<br class="gmail_msg">
> series...).<br class="gmail_msg">
> The latter will be used by Vdsm to keep reporting the metrics (again<br class="gmail_msg">
> <a href="https://gerrit.ovirt.org/#/c/71176/6" rel="noreferrer" class="gmail_msg" target="_blank">https://gerrit.ovirt.org/#/c/71176/6</a>)<br class="gmail_msg">
><br class="gmail_msg">
> Please note that the performance of the plain text protocol are known to<br class="gmail_msg">
> be lower than the binary protocol<br class="gmail_msg">
><br class="gmail_msg">
> What about the unresponsive hosts?<br class="gmail_msg">
> -------------------------------------------------------<br class="gmail_msg">
><br class="gmail_msg">
> We know from experience that hosts may become unresponsive, and this can<br class="gmail_msg">
> disrupt monitoring. however, we do want to keep monitoring the<br class="gmail_msg">
> responsive hosts, avoiding that one rogue hosts makes us lose all the<br class="gmail_msg">
> monitoring data.<br class="gmail_msg">
> To cope with this need, the virt plugin gained support for "partition<br class="gmail_msg">
> tag". With this, we can group VMs together using one arbitrary tag. This<br class="gmail_msg">
> is completely transparent to collectd, and also completely optional.<br class="gmail_msg">
> oVirt can use this tag to group VMs per-storage-domain, or however it<br class="gmail_msg">
> sees fit, trying to minimize the disruption should one host become<br class="gmail_msg">
> unresponsive.<br class="gmail_msg">
><br class="gmail_msg">
> Read the full docs here:<br class="gmail_msg">
> <a href="https://github.com/collectd/collectd/commit/999efc28d8e2e96bc15f535254d412a79755ca4f" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/collectd/collectd/commit/999efc28d8e2e96bc15f535254d412a79755ca4f</a><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> What about the collectd-ovirt plugin?<br class="gmail_msg">
> --------------------------------------------------------<br class="gmail_msg">
><br class="gmail_msg">
> Some time ago I implemented one out-of-tree collectd plugin leveraging<br class="gmail_msg">
> the libvirt bulk stats: <a href="https://github.com/fromanirh/collectd-ovirt" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/fromanirh/collectd-ovirt</a><br class="gmail_msg">
> This plugin is meant to be a modern, drop-in replacement for the<br class="gmail_msg">
> existing virt plugin.<br class="gmail_msg">
> The development of that out of tree plugin is now halted, because we<br class="gmail_msg">
> have everything we need in the upstream collectd plugin.<br class="gmail_msg">
><br class="gmail_msg">
> Future work<br class="gmail_msg">
> ------------------<br class="gmail_msg">
><br class="gmail_msg">
> We believe we have reached feature parity, so we are looking for<br class="gmail_msg">
> bugixes/performance tuning in the near term future. I'll be happy to<br class="gmail_msg">
> provide more patches/PRs about that.<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> Thanks and bests,<br class="gmail_msg">
><br class="gmail_msg">
> --<br class="gmail_msg">
> Francesco Romani<br class="gmail_msg">
> Red Hat Engineering Virtualization R & D<br class="gmail_msg">
> IRC: fromani<br class="gmail_msg">
><br class="gmail_msg">
> _______________________________________________<br class="gmail_msg">
> Devel mailing list<br class="gmail_msg">
> <a href="mailto:Devel@ovirt.org" class="gmail_msg" target="_blank">Devel@ovirt.org</a><br class="gmail_msg">
> <a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.ovirt.org/mailman/listinfo/devel</a><br class="gmail_msg">
> _______________________________________________<br class="gmail_msg">
> Devel mailing list<br class="gmail_msg">
> <a href="mailto:Devel@ovirt.org" class="gmail_msg" target="_blank">Devel@ovirt.org</a><br class="gmail_msg">
> <a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.ovirt.org/mailman/listinfo/devel</a><br class="gmail_msg">
<br class="gmail_msg">
_______________________________________________<br class="gmail_msg">
Devel mailing list<br class="gmail_msg">
<a href="mailto:Devel@ovirt.org" class="gmail_msg" target="_blank">Devel@ovirt.org</a><br class="gmail_msg">
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.ovirt.org/mailman/listinfo/devel</a></div></div></blockquote></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><br class="gmail_msg"><br class="gmail_msg" clear="all"><div class="gmail_msg"><br class="gmail_msg"></div>-- <br class="gmail_msg"><div class="m_-6973989591943094584gmail_signature gmail_msg" data-smartmail="gmail_signature"><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><span style="font-size:12.8px" class="gmail_msg"><b class="gmail_msg">Yaniv Bronhaim.</b></span><br class="gmail_msg"></div></div></div></div></div>
</div></div>
_______________________________________________<br class="gmail_msg">
Devel mailing list<br class="gmail_msg">
<a href="mailto:Devel@ovirt.org" class="gmail_msg" target="_blank">Devel@ovirt.org</a><br class="gmail_msg">
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.ovirt.org/mailman/listinfo/devel</a></blockquote></div></div>