On 21 Feb 2017, at 14:44, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: On Tue, Feb 21, 2017 at 1:06 PM Francesco Romani <fromani(a)redhat.com&gt; wrote: Hello everyone, in the last weeks I've been submitting PRs to collectd upstream, to bring the virt plugin up to date with Vdsm and oVirt needs. Previously, the collectd virt plugin reported only a subset of metrics oVirt uses. In current collectd master, the collectd virt plugin provides all the data Vdsm (thus Engine) needs. This means that it is now possible for Vdsm or Engine to query collectd, not Vdsm/libvirt, and have the same data. Do we wish to ship the unixsock collectd plugin? I'm not sure we do these days (4.1). We can do that later, of course, when we ship this.

Y. There are only two caveats: 1. it is yet to be seen which version of collectd will ship all those enhancements 2. collectd *intentionally* report metrics as rates, not as absolute values as Vdsm does. This may be one issue in presence of restarts/data loss in the link between collectd and the metrics store. Please keep reading for more details: How to get the code? -------------------------------- This somehow tricky until we get one official release. If one is familiar with the RPM build process, it is easy to build one custom packages from a snapshot from collectd master (https://github.com/collectd/collectd) and a recent 5.7.1 RPM (like https://koji.fedoraproject.org/koji/buildinfo?buildID=835669) How to configure it? ------------------------------ Most thing work out of the box. One currently in progress Vdsm patch ships the recommended configuration https://gerrit.ovirt.org/#/c/71176/6/static/etc/collectd.d/virt.conf The meaning of the configuration option is documented in man 5 collectd.conf How it looks like? -------------------------- Let me post one "screenshot" :) $ collectdctl listval | grep a0 a0/virt/disk_octets-hdc a0/virt/disk_octets-vda a0/virt/disk_ops-hdc a0/virt/disk_ops-vda a0/virt/disk_time-hdc a0/virt/disk_time-vda a0/virt/if_dropped-vnet0 a0/virt/if_errors-vnet0 a0/virt/if_octets-vnet0 a0/virt/if_packets-vnet0 a0/virt/memory-actual_balloon a0/virt/memory-rss a0/virt/memory-total a0/virt/ps_cputime a0/virt/total_requests-flush-hdc a0/virt/total_requests-flush-vda a0/virt/total_time_in_ms-flush-hdc a0/virt/total_time_in_ms-flush-vda a0/virt/virt_cpu_total a0/virt/virt_vcpu-0 a0/virt/virt_vcpu-1 How to consume the data? ----------------------------------------- Among the ways to query collectd, the two most popular (and most fitting for oVirt use case) ways are perhaps the network protocol (https://collectd.org/wiki/index.php/Binary_protocol) and the plain text protocol (https://collectd.org/wiki/index.php/Plain_text_protocol). The first could be used by Engine to get the data directly, or to consolidate the metrics in one database (e.g to run any kind of query, for historical series...). The latter will be used by Vdsm to keep reporting the metrics (again https://gerrit.ovirt.org/#/c/71176/6) Please note that the performance of the plain text protocol are known to be lower than the binary protocol What about the unresponsive hosts? ------------------------------------------------------- We know from experience that hosts may become unresponsive, and this can disrupt monitoring. however, we do want to keep monitoring the responsive hosts, avoiding that one rogue hosts makes us lose all the monitoring data. To cope with this need, the virt plugin gained support for "partition tag". With this, we can group VMs together using one arbitrary tag. This is completely transparent to collectd, and also completely optional. oVirt can use this tag to group VMs per-storage-domain, or however it sees fit, trying to minimize the disruption should one host become unresponsive. Read the full docs here: https://github.com/collectd/collectd/commit/999efc28d8e2e96bc15f535254d41... What about the collectd-ovirt plugin? -------------------------------------------------------- Some time ago I implemented one out-of-tree collectd plugin leveraging the libvirt bulk stats: https://github.com/fromanirh/collectd-ovirt This plugin is meant to be a modern, drop-in replacement for the existing virt plugin. The development of that out of tree plugin is now halted, because we have everything we need in the upstream collectd plugin. Future work ------------------ We believe we have reached feature parity, so we are looking for bugixes/performance tuning in the near term future. I'll be happy to provide more patches/PRs about that. Thanks and bests, -- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani _______________________________________________ Devel mailing list Devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/devel _______________________________________________ Devel mailing list Devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/devel

On Tue, Feb 21, 2017 at 4:21 PM, Michal Skrivanek < michal.skrivanek(a)redhat.com&gt; wrote: > On 21 Feb 2017, at 14:44, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: > > > > On Tue, Feb 21, 2017 at 1:06 PM Francesco Romani <fromani(a)redhat.com&gt; wrote: > Hello everyone, > > > in the last weeks I've been submitting PRs to collectd upstream, to > bring the virt plugin up to date with Vdsm and oVirt needs. > > Previously, the collectd virt plugin reported only a subset of metrics > oVirt uses. > > In current collectd master, the collectd virt plugin provides all the > data Vdsm (thus Engine) needs. This means that it is now > > possible for Vdsm or Engine to query collectd, not Vdsm/libvirt, and > have the same data. > > Do we wish to ship the unixsock collectd plugin? I'm not sure we do these days (4.1). > We can do that later, of course, when we ship this. we haven’t decided on the actual solution yet, unixsocket is one possibility. it is tracked in https://trello.com/c/alAOm1tQ we can also have engine pulling it from collectd remotely, then we can eliminate periodic get vm stats or another (crazy) option to use fluentd to push data straight to engine’s postgres:)

why crazy? sounds like where we want to be, without going through vdsm at all Thanks, michal > Y. > > > > There are only two caveats: > > 1. it is yet to be seen which version of collectd will ship all those > enhancements > > 2. collectd *intentionally* report metrics as rates, not as absolute > values as Vdsm does. This may be one issue in presence of restarts/data > loss in the link between collectd and the metrics store. > > > Please keep reading for more details: > > > How to get the code? > > -------------------------------- > > This somehow tricky until we get one official release. If one is > familiar with the RPM build process, it is easy to build one custom packages > > from a snapshot from collectd master > (https://github.com/collectd/collectd) and a recent 5.7.1 RPM (like > https://koji.fedoraproject.org/koji/buildinfo?buildID=835669) > > > How to configure it? > > ------------------------------ > > Most thing work out of the box. One currently in progress Vdsm patch > ships the recommended configuration > https://gerrit.ovirt.org/#/c/71176/6/static/etc/collectd.d/virt.conf > > The meaning of the configuration option is documented in man 5 collectd.conf > > > How it looks like? > > -------------------------- > > > Let me post one "screenshot" :) > > > > $ collectdctl listval | grep a0 > a0/virt/disk_octets-hdc > a0/virt/disk_octets-vda > a0/virt/disk_ops-hdc > a0/virt/disk_ops-vda > a0/virt/disk_time-hdc > a0/virt/disk_time-vda > a0/virt/if_dropped-vnet0 > a0/virt/if_errors-vnet0 > a0/virt/if_octets-vnet0 > a0/virt/if_packets-vnet0 > a0/virt/memory-actual_balloon > a0/virt/memory-rss > a0/virt/memory-total > a0/virt/ps_cputime > a0/virt/total_requests-flush-hdc > a0/virt/total_requests-flush-vda > a0/virt/total_time_in_ms-flush-hdc > a0/virt/total_time_in_ms-flush-vda > a0/virt/virt_cpu_total > a0/virt/virt_vcpu-0 > a0/virt/virt_vcpu-1 > > > How to consume the data? > ----------------------------------------- > > Among the ways to query collectd, the two most popular (and most fitting > for oVirt use case) ways are perhaps the network protocol > (https://collectd.org/wiki/index.php/Binary_protocol) > and the plain text protocol > (https://collectd.org/wiki/index.php/Plain_text_protocol). The first > could be used by Engine to get the data directly, or to consolidate the > metrics in one database (e.g to run any kind of query, for historical > series...). > The latter will be used by Vdsm to keep reporting the metrics (again > https://gerrit.ovirt.org/#/c/71176/6) > > Please note that the performance of the plain text protocol are known to > be lower than the binary protocol > > What about the unresponsive hosts? > ------------------------------------------------------- > > We know from experience that hosts may become unresponsive, and this can > disrupt monitoring. however, we do want to keep monitoring the > responsive hosts, avoiding that one rogue hosts makes us lose all the > monitoring data. > To cope with this need, the virt plugin gained support for "partition > tag". With this, we can group VMs together using one arbitrary tag. This > is completely transparent to collectd, and also completely optional. > oVirt can use this tag to group VMs per-storage-domain, or however it > sees fit, trying to minimize the disruption should one host become > unresponsive. > > Read the full docs here: > https://github.com/collectd/collectd/commit/999efc28d8e2e96bc15f535254d41... > > > What about the collectd-ovirt plugin? > -------------------------------------------------------- > > Some time ago I implemented one out-of-tree collectd plugin leveraging > the libvirt bulk stats: https://github.com/fromanirh/collectd-ovirt > This plugin is meant to be a modern, drop-in replacement for the > existing virt plugin. > The development of that out of tree plugin is now halted, because we > have everything we need in the upstream collectd plugin. > > Future work > ------------------ > > We believe we have reached feature parity, so we are looking for > bugixes/performance tuning in the near term future. I'll be happy to > provide more patches/PRs about that. > > > > Thanks and bests, > > -- > Francesco Romani > Red Hat Engineering Virtualization R & D > IRC: fromani > > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel _______________________________________________ Devel mailing list Devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/devel -- *Yaniv Bronhaim.* _______________________________________________ Devel mailing list Devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/devel

On 02/21/2017 02:44 PM, Yaniv Kaul wrote: On Tue, Feb 21, 2017 at 1:06 PM Francesco Romani <fromani(a)redhat.com&gt; wrote: Hello everyone, in the last weeks I've been submitting PRs to collectd upstream, to bring the virt plugin up to date with Vdsm and oVirt needs. Previously, the collectd virt plugin reported only a subset of metrics oVirt uses. In current collectd master, the collectd virt plugin provides all the data Vdsm (thus Engine) needs. This means that it is now possible for Vdsm or Engine to query collectd, not Vdsm/libvirt, and have the same data. Do we wish to ship the unixsock collectd plugin? I'm not sure we do these days (4.1). We can do that later, of course, when we ship this. Y. AFAIR the collectd unixsock plugin it's built and shipped by default by the collectd (even RPMs). It is the way the command line `collectdctl` too uses to talk with the daemon.

Our client module is still work in progress. I'd be happy to just use a third party client module, the (semi-)official one is not shipped by default last time I checked; perhaps just file one RFE about that? -- Francesco Romani Red Hat Engineering Virtualization R & D IRC: fromani _______________________________________________ Devel mailing list Devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/devel