@Jiri Denemark <jdenemar@redhat.com> @Eduardo Lima <etrunko@redhat.com> can you please have a look on libvirt side? @Martin Perina <mperina@redhat.com> the host/stats part within vdsm was handled by people who are not working anymore on oVirt project, perhaps someone from infra can have a look? Il giorno gio 9 dic 2021 alle ore 11:20 Victor Stinner <vstinner@redhat.com> ha scritto:
On Tue, Dec 7, 2021 at 6:12 PM Chris Adams <cma@cmadams.net> wrote:
Top differences /usr/lib64/python3.6/site-packages/libvirt.py:442: size=295 MiB (+285 MiB), count=5511282 (+5312311), average=56 B /usr/lib64/python3.6/json/decoder.py:355: size=73.9 MiB (+70.2 MiB), count=736108 (+697450), average=105 B /usr/lib64/python3.6/logging/__init__.py:1630: size=44.2 MiB (+43.8 MiB), count=345704 (+342481), average=134 B /usr/lib64/python3.6/site-packages/libvirt.py:5695: size=30.3 MiB (+30.0 MiB), count=190449 (+188665), average=167 B /usr/lib/python3.6/site-packages/vdsm/host/stats.py:138: size=12.1 MiB (+11.4 MiB), count=75366 (+70991), average=168 B /usr/lib/python3.6/site-packages/vdsm/utils.py:358: size=10.4 MiB (+9968 KiB), count=70204 (+65272), average=156 B
That's quite significant!
Top block 5511282 memory blocks: 302589.8 KiB File "/usr/lib64/python3.6/site-packages/libvirt.py", line 442 ret = libvirtmod.virEventRunDefaultImpl() File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 69 libvirt.virEventRunDefaultImpl() File "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 260 ret = func(*args, **kwargs)
You should check where these "ret" objects (of libvirt.py:442) are stored: 5,511,282 is a lot of small objects (average: 56 bytes)! Maybe they are stored in a list and never destroyed.
Maybe it's a reference leak in the libvirtmod.virEventRunDefaultImpl() function of "libvirtmod" C extension: missing Py_DECREF() somewhere.
Or something somehow prevents to delete these projects object. For example, an exception is stored somewhere which keeps all variables alive (in Python 3, an exception stores a traceback object which keeps all variables of all frames alive).
On GitHub and GitLab, I found the following code. Maybe there are minor differences in the versions that you are using.
https://gitlab.com/libvirt/libvirt-python (I built the code locally to get build/libvirt.py)
build/libvirt.c: --- PyObject * libvirt_intWrap(int val) { return PyLong_FromLong((long) val); }
PyObject * libvirt_virEventRunDefaultImpl(PyObject *self ATTRIBUTE_UNUSED, PyObject *args ATTRIBUTE_UNUSED) { PyObject *py_retval; int c_retval; LIBVIRT_BEGIN_ALLOW_THREADS; c_retval = virEventRunDefaultImpl(); LIBVIRT_END_ALLOW_THREADS; py_retval = libvirt_intWrap((int) c_retval); return py_retval; }
static PyMethodDef libvirtMethods[] = { { (char *)"virEventRunDefaultImpl", libvirt_virEventRunDefaultImpl, METH_VARARGS, NULL }, ... {NULL, NULL, 0, NULL} }; ---
This code looks correct and straightforward. Is it possible that internally virEventRunDefaultImpl() calls a Python memory allocator?
build/libvirt.py: --- def virEventRunDefaultImpl(): ret = libvirtmod.virEventRunDefaultImpl() if ret == -1: raise libvirtError('virEventRunDefaultImpl() failed') return ret ---
Again, this code looks correct and straightforward.
https://github.com/oVirt/vdsm/blob/37ed5c279c2dd9c9bb06329d674882e0f98f34d6/...
vdsm/common/libvirtconnection.py: --- def __run(self): try: libvirt.virEventRegisterDefaultImpl() while self.run: libvirt.virEventRunDefaultImpl() finally: self.run = False ---
libvirt.virEventRunDefaultImpl() result is ignored and so I don't see anything obvious which would explain a leak.
Sometimes, looking at the top function is misleading since the explanation can be found in one of the caller functions.
For example, which function creates 70.2 MiB of objects from a JSON document? What calls json/decoder.py:355?
Victor _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/O5OAA6KNLINLRT...
-- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*