Hi, I wrote the tracemalloc module which is easy to use on Python 3.4 and newer. If you take tracemalloc snapshots while the memory usage is growing, and comparing snapshots don't show anything obvious, you can maybe suspect memory fragmentation. You're talking about 4 GB of memory usage, I don't think that memory fragmentation can explain it. Do you need my help to use tracemalloc? Quick tutorial in the official documentation: https://docs.python.org/dev/library/tracemalloc.html#compute-differences Victor On Fri, Nov 12, 2021 at 3:51 PM David Malcolm <dmalcolm@redhat.com> wrote:
On Fri, 2021-11-12 at 09:54 +0100, Sandro Bonazzola wrote:
Il giorno ven 12 nov 2021 alle ore 09:50 Sandro Bonazzola < sbonazzo@redhat.com> ha scritto:
Il giorno ven 12 nov 2021 alle ore 09:47 Sandro Bonazzola < sbonazzo@redhat.com> ha scritto:
Il giorno mer 10 nov 2021 alle ore 15:45 Chris Adams <cma@cmadams.net> ha scritto:
I have seen vdsmd leak memory for years (I've been running oVirt since version 3.5), but never been able to nail it down. I've upgraded a cluster to oVirt 4.4.9 (reloading the hosts with CentOS 8- stream), and I still see it happen. One host in the cluster, which has been up 8 days, has vdsmd with 4.3 GB resident memory. On a couple of other hosts, it's around half a gigabyte.
In the past, it seemed more likely to happen on the hosted engine hosts and/or the SPM host... but the host with the 4.3 GB vdsmd is not either of those.
I'm not sure what I do that would make my setup "special" compared to others; I loaded a pretty minimal install of CentOS 8-stream, with the only extra thing being I add the core parts of the Dell PowerEdge OpenManage tools (so I can get remote SNMP hardware monitoring).
When I run "pmap $(pidof -x vdsmd)", the bulk of the RAM use is a single anonymous block (which I'm guessing is just the python general memory allocator).
I thought maybe the switch to CentOS 8 and python 3 might clear something up, but obviously not. Any ideas?
I guess we still have the reproducibility issue ( https://lists.ovirt.org/archives/list/devel@ovirt.org/thread/KO5SEPAZMLBWSBS... ). But maybe in the meanwhile there's a new way to track things down. +Marcin Sobczyk <msobczyk@redhat.com> ?
Perhaps https://docs.python.org/3.6/library/tracemalloc.html ?
+David Malcolm <dmalcolm@redhat.com> I saw your slides on python memory leak debugging, maybe you can give some suggestions here.
I haven't worked on Python itself in > 8 years, so my knowledge is out- of-date here.
Adding in Victor Stinner, who has worked on the CPython memory allocators more recently, and, in particular, implemented the tracemalloc library linked to above.
Dave