Il giorno lun 16 dic 2019 alle ore 19:41 Chris Adams <cma(a)cmadams.net&gt; ha scritto: > I've seen vdsmd leak memory (RSS increasing) for a while (brought it up > on the lists and opened a BZ ticket), and never gotten anywhere with > diagnosing or resolving it. I reinstalled my dev setup Friday with > up-to-date CentOS 7 (minimal install) and oVirt 4.3, with a hosted > engine on iSCSI (multipath if it matters). > Adding +Martin Perina <mperina(a)redhat.com&gt; and +Milan Zamazal <mzamazal(a)redhat.com&gt; for awareness

On Thu, Jan 30, 2020 at 2:45 PM Chris Adams <cma(a)cmadams.net <mailto:cma@cmadams.net>> wrote: Once upon a time, Sandro Bonazzola <sbonazzo(a)redhat.com <mailto:sbonazzo@redhat.com>> said: > Il giorno lun 16 dic 2019 alle ore 19:41 Chris Adams <cma(a)cmadams.net <mailto:cma@cmadams.net>> ha > scritto: > > > I've seen vdsmd leak memory (RSS increasing) for a while (brought it up > > on the lists and opened a BZ ticket), and never gotten anywhere with > > diagnosing or resolving it.  I reinstalled my dev setup Friday with > > up-to-date CentOS 7 (minimal install) and oVirt 4.3, with a hosted > > engine on iSCSI (multipath if it matters). > > > > Adding +Martin Perina <mperina(a)redhat.com <mailto:mperina@redhat.com>> and +Milan Zamazal > <mzamazal(a)redhat.com <mailto:mzamazal@redhat.com>> for awareness Is there any possibility of someone helping me look at this?  I'm seeing the issue much worse with 4.3 - a cluster I updated to 4.3.7 two months ago has a host (where the hosted engine was running) where vdsmd got to over 20G RSS. Marcin, any suggestions how to investigate it?

-- Chris Adams <cma(a)cmadams.net <mailto:cma@cmadams.net>> -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.

Once upon a time, Marcin Sobczyk <msobczyk(a)redhat.com&gt; said: > Python mem profiling is hard... I already tackled the VDSM memory > leak problem once. > VDSM was growing, but not at a scale that Chris is describing. Tried > out different tools, > but got to a point, where enforcing periodic garbage collecting made > VDSM mem usage > constant, so the conclusion made there was no mem leaks. Yeah, I gave it a try myself (despite not being very good at Python; been a system admin too long so I'm all about perl :) ), and didn't get anywhere. > Chris, if I understood you correctly, a single machine suffices to > reproduce your issue? > One that acts as a host with hosted engine on it + iscsi storage? > If so, maybe I/you could construct a VM with a reproducible > environment and share? > Having something like this would make investigating this issue much > more reliable. I don't think I've tried to reproduce it on a single machine setup. I think I've always gone ahead and added at least a second machine (even if there was no VM other than the engine); stopping with one to see if it happens is a good idea. My dev cluster is actually down at the moment (a dead UPS and nearby road/bridge construction are a bad combination); I'll get it back online (and on a better UPS hopefully!) today. By "construct a VM" - do you mean building a setup inside a VM (with nested virtualization), so everything is local? My dev cluster iSCSI SAN is targetcli on Linux (the prod setups are all EqualLogics), so I know how to set that up.

Thank you for your help.