Once upon a time, Victor Stinner <vstinner@redhat.com> said:
Then use pickle.load() to reload snapshots from files. You can take multiple snapshots and compare snapshot 1 with snapshot 2, compare 1 with 3, etc. If there is a major memory increase between two snapshots, I expect a significant difference between these two snapshots.
I tried this approach (tried Valgrind but it caused vdsmd to run too slow, so oVirt saw timeouts and moved VMs away), and it does show a pretty big jump overnight. Below is the output of a compare of tracemalloc dump between yesterday afternoon and this morning. The files in the traceback are from RPMs: python3-libvirt-7.6.0-1.el8s.x86_64 vdsm-common-4.40.90.4-1.el8.noarch Looking at the code, I'm not sure what to make of it though. Top differences /usr/lib64/python3.6/site-packages/libvirt.py:442: size=295 MiB (+285 MiB), count=5511282 (+5312311), average=56 B /usr/lib64/python3.6/json/decoder.py:355: size=73.9 MiB (+70.2 MiB), count=736108 (+697450), average=105 B /usr/lib64/python3.6/logging/__init__.py:1630: size=44.2 MiB (+43.8 MiB), count=345704 (+342481), average=134 B /usr/lib64/python3.6/site-packages/libvirt.py:5695: size=30.3 MiB (+30.0 MiB), count=190449 (+188665), average=167 B /usr/lib/python3.6/site-packages/vdsm/host/stats.py:138: size=12.1 MiB (+11.4 MiB), count=75366 (+70991), average=168 B /usr/lib/python3.6/site-packages/vdsm/utils.py:358: size=10.4 MiB (+9968 KiB), count=70204 (+65272), average=156 B /usr/lib64/python3.6/site-packages/libvirt.py:537: size=7676 KiB (+7656 KiB), count=109119 (+108886), average=72 B /usr/lib/python3.6/site-packages/yajsonrpc/betterAsyncore.py:256: size=7813 KiB (+7505 KiB), count=125015 (+120083), average=64 B /usr/lib64/python3.6/asyncore.py:173: size=6934 KiB (+6735 KiB), count=110941 (+107755), average=64 B /usr/lib/python3.6/site-packages/vdsm/virt/vmchannels.py:163: size=5984 KiB (+5631 KiB), count=95744 (+90103), average=64 B Top block 5511282 memory blocks: 302589.8 KiB File "/usr/lib64/python3.6/site-packages/libvirt.py", line 442 ret = libvirtmod.virEventRunDefaultImpl() File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 69 libvirt.virEventRunDefaultImpl() File "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 260 ret = func(*args, **kwargs) File "/usr/lib64/python3.6/threading.py", line 885 self._target(*self._args, **self._kwargs) File "/usr/lib64/python3.6/threading.py", line 937 self.run() File "/usr/lib64/python3.6/threading.py", line 905 self._bootstrap_inner() -- Chris Adams <cma@cmadams.net>