[ovirt-users] Re: Upgraded to oVirt 4.4.9, still have vdsmd memory leak
Once upon a time, Victor Stinner <vstinner(a)redhat.com> said:
Then use pickle.load() to reload snapshots from files. You can take
multiple snapshots and compare snapshot 1 with snapshot 2, compare 1
with 3, etc. If there is a major memory increase between two
snapshots, I expect a significant difference between these two
snapshots.
I tried this approach (tried Valgrind but it caused vdsmd to run too
slow, so oVirt saw timeouts and moved VMs away), and it does show a
pretty big jump overnight. Below is the output of a compare of
tracemalloc dump between yesterday afternoon and this morning. The
files in the traceback are from RPMs:
python3-libvirt-7.6.0-1.el8s.x86_64
vdsm-common-4.40.90.4-1.el8.noarch
Looking at the code, I'm not sure what to make of it though.
Top differences
/usr/lib64/python3.6/site-packages/libvirt.py:442: size=295 MiB (+285 MiB), count=5511282
(+5312311), average=56 B
/usr/lib64/python3.6/json/decoder.py:355: size=73.9 MiB (+70.2 MiB), count=736108
(+697450), average=105 B
/usr/lib64/python3.6/logging/__init__.py:1630: size=44.2 MiB (+43.8 MiB), count=345704
(+342481), average=134 B
/usr/lib64/python3.6/site-packages/libvirt.py:5695: size=30.3 MiB (+30.0 MiB),
count=190449 (+188665), average=167 B
/usr/lib/python3.6/site-packages/vdsm/host/stats.py:138: size=12.1 MiB (+11.4 MiB),
count=75366 (+70991), average=168 B
/usr/lib/python3.6/site-packages/vdsm/utils.py:358: size=10.4 MiB (+9968 KiB), count=70204
(+65272), average=156 B
/usr/lib64/python3.6/site-packages/libvirt.py:537: size=7676 KiB (+7656 KiB), count=109119
(+108886), average=72 B
/usr/lib/python3.6/site-packages/yajsonrpc/betterAsyncore.py:256: size=7813 KiB (+7505
KiB), count=125015 (+120083), average=64 B
/usr/lib64/python3.6/asyncore.py:173: size=6934 KiB (+6735 KiB), count=110941 (+107755),
average=64 B
/usr/lib/python3.6/site-packages/vdsm/virt/vmchannels.py:163: size=5984 KiB (+5631 KiB),
count=95744 (+90103), average=64 B
Top block
5511282 memory blocks: 302589.8 KiB
File "/usr/lib64/python3.6/site-packages/libvirt.py", line 442
ret = libvirtmod.virEventRunDefaultImpl()
File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line
69
libvirt.virEventRunDefaultImpl()
File "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 260
ret = func(*args, **kwargs)
File "/usr/lib64/python3.6/threading.py", line 885
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python3.6/threading.py", line 937
self.run()
File "/usr/lib64/python3.6/threading.py", line 905
self._bootstrap_inner()
--
Chris Adams <cma(a)cmadams.net>