On Tue, Dec 7, 2021 at 6:12 PM Chris Adams <cma@cmadams.net> wrote:
Top differences /usr/lib64/python3.6/site-packages/libvirt.py:442: size=295 MiB (+285 MiB), count=5511282 (+5312311), average=56 B /usr/lib64/python3.6/json/decoder.py:355: size=73.9 MiB (+70.2 MiB), count=736108 (+697450), average=105 B /usr/lib64/python3.6/logging/__init__.py:1630: size=44.2 MiB (+43.8 MiB), count=345704 (+342481), average=134 B /usr/lib64/python3.6/site-packages/libvirt.py:5695: size=30.3 MiB (+30.0 MiB), count=190449 (+188665), average=167 B /usr/lib/python3.6/site-packages/vdsm/host/stats.py:138: size=12.1 MiB (+11.4 MiB), count=75366 (+70991), average=168 B /usr/lib/python3.6/site-packages/vdsm/utils.py:358: size=10.4 MiB (+9968 KiB), count=70204 (+65272), average=156 B
That's quite significant!
Top block 5511282 memory blocks: 302589.8 KiB File "/usr/lib64/python3.6/site-packages/libvirt.py", line 442 ret = libvirtmod.virEventRunDefaultImpl() File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 69 libvirt.virEventRunDefaultImpl() File "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 260 ret = func(*args, **kwargs)
You should check where these "ret" objects (of libvirt.py:442) are stored: 5,511,282 is a lot of small objects (average: 56 bytes)! Maybe they are stored in a list and never destroyed. Maybe it's a reference leak in the libvirtmod.virEventRunDefaultImpl() function of "libvirtmod" C extension: missing Py_DECREF() somewhere. Or something somehow prevents to delete these projects object. For example, an exception is stored somewhere which keeps all variables alive (in Python 3, an exception stores a traceback object which keeps all variables of all frames alive). On GitHub and GitLab, I found the following code. Maybe there are minor differences in the versions that you are using. https://gitlab.com/libvirt/libvirt-python (I built the code locally to get build/libvirt.py) build/libvirt.c: --- PyObject * libvirt_intWrap(int val) { return PyLong_FromLong((long) val); } PyObject * libvirt_virEventRunDefaultImpl(PyObject *self ATTRIBUTE_UNUSED, PyObject *args ATTRIBUTE_UNUSED) { PyObject *py_retval; int c_retval; LIBVIRT_BEGIN_ALLOW_THREADS; c_retval = virEventRunDefaultImpl(); LIBVIRT_END_ALLOW_THREADS; py_retval = libvirt_intWrap((int) c_retval); return py_retval; } static PyMethodDef libvirtMethods[] = { { (char *)"virEventRunDefaultImpl", libvirt_virEventRunDefaultImpl, METH_VARARGS, NULL }, ... {NULL, NULL, 0, NULL} }; --- This code looks correct and straightforward. Is it possible that internally virEventRunDefaultImpl() calls a Python memory allocator? build/libvirt.py: --- def virEventRunDefaultImpl(): ret = libvirtmod.virEventRunDefaultImpl() if ret == -1: raise libvirtError('virEventRunDefaultImpl() failed') return ret --- Again, this code looks correct and straightforward. https://github.com/oVirt/vdsm/blob/37ed5c279c2dd9c9bb06329d674882e0f98f34d6/... vdsm/common/libvirtconnection.py: --- def __run(self): try: libvirt.virEventRegisterDefaultImpl() while self.run: libvirt.virEventRunDefaultImpl() finally: self.run = False --- libvirt.virEventRunDefaultImpl() result is ignored and so I don't see anything obvious which would explain a leak. Sometimes, looking at the top function is misleading since the explanation can be found in one of the caller functions. For example, which function creates 70.2 MiB of objects from a JSON document? What calls json/decoder.py:355? Victor