[ovirt-users] Re: Upgraded to oVirt 4.4.9, still have vdsmd memory leak

On Tue, Dec 7, 2021 at 6:12 PM Chris Adams <cma(a)cmadams.net&gt; wrote: > Top differences > /usr/lib64/python3.6/site-packages/libvirt.py:442: size=295 MiB (+285 MiB), count=5511282 (+5312311), average=56 B > /usr/lib64/python3.6/json/decoder.py:355: size=73.9 MiB (+70.2 MiB), count=736108 (+697450), average=105 B > /usr/lib64/python3.6/logging/__init__.py:1630: size=44.2 MiB (+43.8 MiB), count=345704 (+342481), average=134 B > /usr/lib64/python3.6/site-packages/libvirt.py:5695: size=30.3 MiB (+30.0 MiB), count=190449 (+188665), average=167 B > /usr/lib/python3.6/site-packages/vdsm/host/stats.py:138: size=12.1 MiB (+11.4 MiB), count=75366 (+70991), average=168 B > /usr/lib/python3.6/site-packages/vdsm/utils.py:358: size=10.4 MiB (+9968 KiB), count=70204 (+65272), average=156 B That's quite significant! > Top block > 5511282 memory blocks: 302589.8 KiB > File "/usr/lib64/python3.6/site-packages/libvirt.py", line 442 > ret = libvirtmod.virEventRunDefaultImpl() > File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 69 > libvirt.virEventRunDefaultImpl() > File "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 260 > ret = func(*args, **kwargs) You should check where these "ret" objects (of libvirt.py:442) are stored: 5,511,282 is a lot of small objects (average: 56 bytes)! Maybe they are stored in a list and never destroyed. Maybe it's a reference leak in the libvirtmod.virEventRunDefaultImpl() function of "libvirtmod" C extension: missing Py_DECREF() somewhere. Or something somehow prevents to delete these projects object. For example, an exception is stored somewhere which keeps all variables alive (in Python 3, an exception stores a traceback object which keeps all variables of all frames alive). On GitHub and GitLab, I found the following code. Maybe there are minor differences in the versions that you are using. https://gitlab.com/libvirt/libvirt-python (I built the code locally to get build/libvirt.py) build/libvirt.c: --- PyObject * libvirt_intWrap(int val) { return PyLong_FromLong((long) val); } PyObject * libvirt_virEventRunDefaultImpl(PyObject *self ATTRIBUTE_UNUSED, PyObject *args ATTRIBUTE_UNUSED) { PyObject *py_retval; int c_retval; LIBVIRT_BEGIN_ALLOW_THREADS; c_retval = virEventRunDefaultImpl(); LIBVIRT_END_ALLOW_THREADS; py_retval = libvirt_intWrap((int) c_retval); return py_retval; } static PyMethodDef libvirtMethods[] = { { (char *)"virEventRunDefaultImpl", libvirt_virEventRunDefaultImpl, METH_VARARGS, NULL }, ... {NULL, NULL, 0, NULL} }; --- This code looks correct and straightforward. Is it possible that internally virEventRunDefaultImpl() calls a Python memory allocator? build/libvirt.py: --- def virEventRunDefaultImpl(): ret = libvirtmod.virEventRunDefaultImpl() if ret == -1: raise libvirtError('virEventRunDefaultImpl() failed') return ret --- Again, this code looks correct and straightforward. https://github.com/oVirt/vdsm/blob/37ed5c279c2dd9c9bb06329d674882e0f98f34... vdsm/common/libvirtconnection.py: --- def __run(self): try: libvirt.virEventRegisterDefaultImpl() while self.run: libvirt.virEventRunDefaultImpl() finally: self.run = False --- libvirt.virEventRunDefaultImpl() result is ignored and so I don't see anything obvious which would explain a leak. Sometimes, looking at the top function is misleading since the explanation can be found in one of the caller functions. For example, which function creates 70.2 MiB of objects from a JSON document? What calls json/decoder.py:355? Victor _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/O5OAA6KNLIN...