I have seen vdsmd leak memory for years (I've been running oVirt since version 3.5), but never been able to nail it down. I've upgraded a cluster to oVirt 4.4.9 (reloading the hosts with CentOS 8-stream), and I still see it happen. One host in the cluster, which has been up 8 days, has vdsmd with 4.3 GB resident memory. On a couple of other hosts, it's around half a gigabyte. In the past, it seemed more likely to happen on the hosted engine hosts and/or the SPM host... but the host with the 4.3 GB vdsmd is not either of those. I'm not sure what I do that would make my setup "special" compared to others; I loaded a pretty minimal install of CentOS 8-stream, with the only extra thing being I add the core parts of the Dell PowerEdge OpenManage tools (so I can get remote SNMP hardware monitoring). When I run "pmap $(pidof -x vdsmd)", the bulk of the RAM use is a single anonymous block (which I'm guessing is just the python general memory allocator). I thought maybe the switch to CentOS 8 and python 3 might clear something up, but obviously not. Any ideas?

-- Chris Adams <cma(a)cmadams.net&gt; _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3PTE35WMIVG...

Il giorno ven 12 nov 2021 alle ore 09:47 Sandro Bonazzola < sbonazzo(a)redhat.com&gt; ha scritto: > > > Il giorno mer 10 nov 2021 alle ore 15:45 Chris Adams <cma(a)cmadams.net&gt; > ha scritto: > >> I have seen vdsmd leak memory for years (I've been running oVirt since >> version 3.5), but never been able to nail it down. I've upgraded a >> cluster to oVirt 4.4.9 (reloading the hosts with CentOS 8-stream), and I >> still see it happen. One host in the cluster, which has been up 8 days, >> has vdsmd with 4.3 GB resident memory. On a couple of other hosts, it's >> around half a gigabyte. >> >> In the past, it seemed more likely to happen on the hosted engine hosts >> and/or the SPM host... but the host with the 4.3 GB vdsmd is not either >> of those. >> >> I'm not sure what I do that would make my setup "special" compared to >> others; I loaded a pretty minimal install of CentOS 8-stream, with the >> only extra thing being I add the core parts of the Dell PowerEdge >> OpenManage tools (so I can get remote SNMP hardware monitoring). >> >> When I run "pmap $(pidof -x vdsmd)", the bulk of the RAM use is a single >> anonymous block (which I'm guessing is just the python general memory >> allocator). >> >> I thought maybe the switch to CentOS 8 and python 3 might clear >> something up, but obviously not. Any ideas? >> > > I guess we still have the reproducibility issue ( > https://lists.ovirt.org/archives/list/devel@ovirt.org/thread/KO5SEPAZMLBW... > ). > But maybe in the meanwhile there's a new way to track things down. +Marcin > Sobczyk <msobczyk(a)redhat.com&gt; ? > > > Perhaps https://docs.python.org/3.6/library/tracemalloc.html ?

> > >> -- >> Chris Adams <cma(a)cmadams.net&gt; >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/3PTE35WMIVG... >> > > > -- > > Sandro Bonazzola > > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > > Red Hat EMEA <https://www.redhat.com/> > > sbonazzo(a)redhat.com > <https://www.redhat.com/> > > *Red Hat respects your work life balance. Therefore there is no need to > answer this email out of your office hours.* > > > -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

On Fri, 2021-11-12 at 09:54 +0100, Sandro Bonazzola wrote: > Il giorno ven 12 nov 2021 alle ore 09:50 Sandro Bonazzola < > sbonazzo(a)redhat.com&gt; ha scritto: > > > > > > > Il giorno ven 12 nov 2021 alle ore 09:47 Sandro Bonazzola < > > sbonazzo(a)redhat.com&gt; ha scritto: > > > > > > > > > > > Il giorno mer 10 nov 2021 alle ore 15:45 Chris Adams > > > <cma(a)cmadams.net&gt; > > > ha scritto: > > > > > > > I have seen vdsmd leak memory for years (I've been running > > > > oVirt since > > > > version 3.5), but never been able to nail it down. I've > > > > upgraded a > > > > cluster to oVirt 4.4.9 (reloading the hosts with CentOS 8- > > > > stream), and I > > > > still see it happen. One host in the cluster, which has been > > > > up 8 days, > > > > has vdsmd with 4.3 GB resident memory. On a couple of other > > > > hosts, it's > > > > around half a gigabyte. > > > > > > > > In the past, it seemed more likely to happen on the hosted > > > > engine hosts > > > > and/or the SPM host... but the host with the 4.3 GB vdsmd is > > > > not either > > > > of those. > > > > > > > > I'm not sure what I do that would make my setup "special" > > > > compared to > > > > others; I loaded a pretty minimal install of CentOS 8-stream, > > > > with the > > > > only extra thing being I add the core parts of the Dell > > > > PowerEdge > > > > OpenManage tools (so I can get remote SNMP hardware > > > > monitoring). > > > > > > > > When I run "pmap $(pidof -x vdsmd)", the bulk of the RAM use is > > > > a single > > > > anonymous block (which I'm guessing is just the python general > > > > memory > > > > allocator). > > > > > > > > I thought maybe the switch to CentOS 8 and python 3 might clear > > > > something up, but obviously not. Any ideas? > > > > > > > > > > I guess we still have the reproducibility issue ( > > > https://lists.ovirt.org/archives/list/devel@ovirt.org/thread/KO5SEPAZMLBW... > > > ). > > > But maybe in the meanwhile there's a new way to track things > > > down. +Marcin > > > Sobczyk <msobczyk(a)redhat.com&gt; ? > > > > > > > > > > > Perhaps https://docs.python.org/3.6/library/tracemalloc.html ? > > > > +David Malcolm <dmalcolm(a)redhat.com&gt; I saw your slides on python > memory > leak debugging, maybe you can give some suggestions here. I haven't worked on Python itself in > 8 years, so my knowledge is out- of-date here. Adding in Victor Stinner, who has worked on the CPython memory allocators more recently, and, in particular, implemented the tracemalloc library linked to above. Dave

Then use pickle.load() to reload snapshots from files. You can take multiple snapshots and compare snapshot 1 with snapshot 2, compare 1 with 3, etc. If there is a major memory increase between two snapshots, I expect a significant difference between these two snapshots.

On Tue, Dec 7, 2021 at 6:12 PM Chris Adams <cma(a)cmadams.net&gt; wrote: > Top differences > /usr/lib64/python3.6/site-packages/libvirt.py:442: size=295 MiB (+285 MiB), count=5511282 (+5312311), average=56 B > /usr/lib64/python3.6/json/decoder.py:355: size=73.9 MiB (+70.2 MiB), count=736108 (+697450), average=105 B > /usr/lib64/python3.6/logging/__init__.py:1630: size=44.2 MiB (+43.8 MiB), count=345704 (+342481), average=134 B > /usr/lib64/python3.6/site-packages/libvirt.py:5695: size=30.3 MiB (+30.0 MiB), count=190449 (+188665), average=167 B > /usr/lib/python3.6/site-packages/vdsm/host/stats.py:138: size=12.1 MiB (+11.4 MiB), count=75366 (+70991), average=168 B > /usr/lib/python3.6/site-packages/vdsm/utils.py:358: size=10.4 MiB (+9968 KiB), count=70204 (+65272), average=156 B That's quite significant! > Top block > 5511282 memory blocks: 302589.8 KiB > File "/usr/lib64/python3.6/site-packages/libvirt.py", line 442 > ret = libvirtmod.virEventRunDefaultImpl() > File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 69 > libvirt.virEventRunDefaultImpl() > File "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 260 > ret = func(*args, **kwargs) You should check where these "ret" objects (of libvirt.py:442) are stored: 5,511,282 is a lot of small objects (average: 56 bytes)! Maybe they are stored in a list and never destroyed. Maybe it's a reference leak in the libvirtmod.virEventRunDefaultImpl() function of "libvirtmod" C extension: missing Py_DECREF() somewhere. Or something somehow prevents to delete these projects object. For example, an exception is stored somewhere which keeps all variables alive (in Python 3, an exception stores a traceback object which keeps all variables of all frames alive). On GitHub and GitLab, I found the following code. Maybe there are minor differences in the versions that you are using. https://gitlab.com/libvirt/libvirt-python (I built the code locally to get build/libvirt.py) build/libvirt.c: --- PyObject * libvirt_intWrap(int val) { return PyLong_FromLong((long) val); } PyObject * libvirt_virEventRunDefaultImpl(PyObject *self ATTRIBUTE_UNUSED, PyObject *args ATTRIBUTE_UNUSED) { PyObject *py_retval; int c_retval; LIBVIRT_BEGIN_ALLOW_THREADS; c_retval = virEventRunDefaultImpl(); LIBVIRT_END_ALLOW_THREADS; py_retval = libvirt_intWrap((int) c_retval); return py_retval; } static PyMethodDef libvirtMethods[] = { { (char *)"virEventRunDefaultImpl", libvirt_virEventRunDefaultImpl, METH_VARARGS, NULL }, ... {NULL, NULL, 0, NULL} }; --- This code looks correct and straightforward. Is it possible that internally virEventRunDefaultImpl() calls a Python memory allocator? build/libvirt.py: --- def virEventRunDefaultImpl(): ret = libvirtmod.virEventRunDefaultImpl() if ret == -1: raise libvirtError('virEventRunDefaultImpl() failed') return ret --- Again, this code looks correct and straightforward. https://github.com/oVirt/vdsm/blob/37ed5c279c2dd9c9bb06329d674882e0f98f34... vdsm/common/libvirtconnection.py: --- def __run(self): try: libvirt.virEventRegisterDefaultImpl() while self.run: libvirt.virEventRunDefaultImpl() finally: self.run = False --- libvirt.virEventRunDefaultImpl() result is ignored and so I don't see anything obvious which would explain a leak. Sometimes, looking at the top function is misleading since the explanation can be found in one of the caller functions. For example, which function creates 70.2 MiB of objects from a JSON document? What calls json/decoder.py:355? Victor _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/O5OAA6KNLIN...

I have seen vdsmd leak memory for years (I've been running oVirt since version 3.5), but never been able to nail it down. I've upgraded a cluster to oVirt 4.4.9 (reloading the hosts with CentOS 8-stream), and I still see it happen. One host in the cluster, which has been up 8 days, has vdsmd with 4.3 GB resident memory. On a couple of other hosts, it's around half a gigabyte.