Hi, during my sampling overhaul effort I faced various times a mismatch between what the libvirt-python API offers for bulk stats and what VDSM needs. The return value of bulk stats is a something like [(dom1, {dict_of_bulk_stats1}), {dom2, {dict_of_bulk_stats2}), ...] While VDSM almost all the times really wants to use { dom_uuid_1: dict_of_bulk_stats_1, dom_uuid_2: dict_of_bulk_stats_2 } translation is trivial to be coded, but bulk stats retrieval is quite always in the hot path. Moreover, this is just wasteful.

But in general, there will always be a not-perfect match from libvirt-python and VDSM needs. This is expected: libvirt-python needs to be more generic. So, I coded a Proof of Concept extension module which *complements* and not *replaces* the libvirt-python API.

https://github.com/mojaves/libvirt-python-ovirt It monkeys-patch some APIs and replace them with VDSM-crafted ones. If we have enough of monkey patching, it is even easier to just add new APIs instead of replacing them. It was a nice and fun hack, but now, the question is: is that a concept worth developing further? Do we want to use and depend on this module? I believe that with such a buffer we can gain some agility with respect to libvirt plans and needs, and the maintainership cost _seems_ fairly low. Thoughts welcome.

----- Original Message ----- > From: "Dan Kenigsberg" <danken(a)redhat.com&gt; > To: "Francesco Romani" <fromani(a)redhat.com&gt; > Cc: devel(a)ovirt.org > Sent: Tuesday, February 24, 2015 11:19:20 AM > Subject: Re: [ovirt-devel] [VDSM] about improved libvirt-python bindings > > On Wed, Feb 18, 2015 at 07:53:17AM -0500, Francesco Romani wrote: > > Hi, > > > > during my sampling overhaul effort I faced various times a mismatch between > > what the libvirt-python API offers for bulk stats and what VDSM needs. > > > > The return value of bulk stats is a something like > > > > [(dom1, {dict_of_bulk_stats1}), {dom2, {dict_of_bulk_stats2}), ...] > > > > While VDSM almost all the times really wants to use > > > > { dom_uuid_1: dict_of_bulk_stats_1, dom_uuid_2: dict_of_bulk_stats_2 } > > > > translation is trivial to be coded, but bulk stats retrieval is quite > > always in the hot path. Moreover, this is just wasteful. > > How big is the waste? Will provide hard numbers (and possibly pretty graphs) > > But in general, there will always be a not-perfect match from > > libvirt-python > > and VDSM needs. This is expected: libvirt-python needs to be more generic. > > > > So, I coded a Proof of Concept extension module which *complements* > > and not *replaces* the libvirt-python API. > > Does it require Vdsm to hold two open connections to libvirtd? [...] Nope, it is intended as transparent replacement, much like our pthreading does for threading standard module.

[...] > > It was a nice and fun hack, but now, the question is: is that a concept > > worth developing further? Do we want to use and depend on this module? > > > > I believe that with such a buffer we can gain some agility with respect > > to libvirt plans and needs, and the maintainership cost _seems_ fairly low. > > > > Thoughts welcome. > > To me, adding this module seems quite expensive: new ovirt project, > inclusion in fedora/el/debian, versioning and release-engineering > hassle. Right, but much of this cost should be bootstrap, not recurring, right? Not trying to push things, just to assess the cons.

From: "Francesco Romani" <fromani(a)redhat.com&gt; To: devel(a)ovirt.org Sent: Wednesday, February 18, 2015 2:53:17 PM Subject: [ovirt-devel] [VDSM] about improved libvirt-python bindings Hi, during my sampling overhaul effort I faced various times a mismatch between what the libvirt-python API offers for bulk stats and what VDSM needs. The return value of bulk stats is a something like [(dom1, {dict_of_bulk_stats1}), {dom2, {dict_of_bulk_stats2}), ...] While VDSM almost all the times really wants to use { dom_uuid_1: dict_of_bulk_stats_1, dom_uuid_2: dict_of_bulk_stats_2 } translation is trivial to be coded, but bulk stats retrieval is quite always in the hot path. Moreover, this is just wasteful. But in general, there will always be a not-perfect match from libvirt-python and VDSM needs. This is expected: libvirt-python needs to be more generic. So, I coded a Proof of Concept extension module which *complements* and not *replaces* the libvirt-python API. https://github.com/mojaves/libvirt-python-ovirt It monkeys-patch some APIs and replace them with VDSM-crafted ones. If we have enough of monkey patching, it is even easier to just add new APIs instead of replacing them. It was a nice and fun hack, but now, the question is: is that a concept worth developing further? Do we want to use and depend on this module? I believe that with such a buffer we can gain some agility with respect to libvirt plans and needs, and the maintainership cost _seems_ fairly low. Thoughts welcome.