On Fri, 2021-11-12 at 09:54 +0100, Sandro Bonazzola wrote:
Il giorno ven 12 nov 2021 alle ore 09:50 Sandro Bonazzola < sbonazzo@redhat.com> ha scritto:
Il giorno ven 12 nov 2021 alle ore 09:47 Sandro Bonazzola < sbonazzo@redhat.com> ha scritto:
Il giorno mer 10 nov 2021 alle ore 15:45 Chris Adams <cma@cmadams.net> ha scritto:
I have seen vdsmd leak memory for years (I've been running oVirt since version 3.5), but never been able to nail it down. I've upgraded a cluster to oVirt 4.4.9 (reloading the hosts with CentOS 8- stream), and I still see it happen. One host in the cluster, which has been up 8 days, has vdsmd with 4.3 GB resident memory. On a couple of other hosts, it's around half a gigabyte.
In the past, it seemed more likely to happen on the hosted engine hosts and/or the SPM host... but the host with the 4.3 GB vdsmd is not either of those.
I'm not sure what I do that would make my setup "special" compared to others; I loaded a pretty minimal install of CentOS 8-stream, with the only extra thing being I add the core parts of the Dell PowerEdge OpenManage tools (so I can get remote SNMP hardware monitoring).
When I run "pmap $(pidof -x vdsmd)", the bulk of the RAM use is a single anonymous block (which I'm guessing is just the python general memory allocator).
I thought maybe the switch to CentOS 8 and python 3 might clear something up, but obviously not. Any ideas?
I guess we still have the reproducibility issue ( https://lists.ovirt.org/archives/list/devel@ovirt.org/thread/KO5SEPAZMLBWSBS... ). But maybe in the meanwhile there's a new way to track things down. +Marcin Sobczyk <msobczyk@redhat.com> ?
Perhaps https://docs.python.org/3.6/library/tracemalloc.html ?
+David Malcolm <dmalcolm@redhat.com> I saw your slides on python memory leak debugging, maybe you can give some suggestions here.
I haven't worked on Python itself in > 8 years, so my knowledge is out- of-date here. Adding in Victor Stinner, who has worked on the CPython memory allocators more recently, and, in particular, implemented the tracemalloc library linked to above. Dave