Re: [ovirt-users] VDSM memory consumption

Hello Everyone, I did create the original BZ on this. In the mean time, lab system I used is dismantled and the production system is yet to deploy. As I wrote in BZ1147148 [1], I experienced two different issues. One, one big mem leak of about 15MiB/h and a smaller one, ~300KiB. These seem unrelated. The larger leak was indeed related to SSL in some way; not necessarily M2Crypto. However, after disabling SSL this was gone leaving the smaller leak. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1147148 On Mo, 2015-03-09 at 23:49 +0100, Matt . wrote: > Hi, > > I also see this on the latest 3.5 version, I'm thinking about setting > up a cronjob to restart vdsm every night. I did the same thing. In general, it seems to be a bad idea as it compromised system stability on the long run. While VMs seem to be fine, engine does not like this very much. > I cannot believe that people say they don't have this issue. This was hard for me to accept as well. I know of Markus Stockhausen and Seven Kieske, both confirmed the small leak. This might also be some special other service; though I started out with a minimal install of Centos 6. > > Can someone of the devs dive in maybe ? > > Thanks! > > Matt > > > > 2015-03-09 23:29 GMT+01:00 Dan Kenigsberg <danken(a)redhat.com&gt;: > > On Mon, Mar 09, 2015 at 10:40:51AM -0500, Darrell Budic wrote: > >> > On Mar 9, 2015, at 4:51 AM, Dan Kenigsberg <danken(a)redhat.com&gt; wrote: > >> > > >> > On Fri, Mar 06, 2015 at 10:58:53AM -0600, Darrell Budic wrote: > >> >> I believe the supervdsm leak was fixed, but 3.5.1 versions of vdsmd still leaks slowly, ~300k/hr, yes. > >> >> > >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1158108 > >> >> > >> >> > >> >>> On Mar 6, 2015, at 10:23 AM, Chris Adams <cma(a)cmadams.net&gt; wrote: > >> >>> > >> >>> Once upon a time, Federico Alberto Sayd <fsayd(a)uncu.edu.ar&gt; said: > >> >>>> I am experiencing troubles with VDSM memory consuption. > >> >>>> > >> >>>> I am running > >> >>>> > >> >>>> Engine: ovirt 3.5.1 > >> >>>> > >> >>>> Nodes: > >> >>>> > >> >>>> Centos 6.6 > >> >>>> VDSM 4.16.10-8 > >> >>>> Libvirt: libvirt-0.10.2-46 > >> >>>> Kernel: 2.6.32 > >> >>>> > >> >>>> When the host boots, memory consuption is normal, but after 2 or 3 > >> >>>> days running, VDSM memory consuption grows and it consumes more > >> >>>> memory that all vm's running in the host. If I restart the vdsm > >> >>>> service, memory consuption normalizes, but then it start growing > >> >>>> again. > >> >>>> > >> >>>> I have seen some BZ about vdsm and supervdsm about memory leaks, but > >> >>>> I don't know if VDSM 4.6.10.8 is still affected by a related bug. > >> >>> > >> >>> Can't help, but I see the same thing with CentOS 7 nodes and the same > >> >>> version of vdsm. > >> >>> -- > >> >>> Chris Adams <cma(a)cmadams.net&gt; > >> >>> _______________________________________________ > >> >>> Users mailing list > >> >>> Users(a)ovirt.org > >> >>> http://lists.ovirt.org/mailman/listinfo/users > >> > > >> > I'm afraid that we are yet to find a solution for this issue, which is > >> > completly different from the horrible leak of supervdsm < 4.16.7. > >> > > >> > Could you corroborate the claim of > >> > Bug 1147148 - M2Crypto usage in vdsm leaks memory > >> > ? Does the leak disappear once you start using plaintext transport? > >> > > >> > Regards, > >> > Dan. > >> > >> I don’t think this is crypto related, but I could try that if you still need some confirmation (and point me at a quick doc on switching to plaintext?). > >> > >> This is from #ovirt around November 18th I think, Saggi thought he’d found something related: > >> > >> 9:58:43 AM saggi: YamakasY: Found the leak > >> 9:58:48 AM saggi: YamakasY: Or at least the flow > >> 9:58:57 AM saggi: YamakasY: The good news is that I can reproduce > >> 9:59:20 AM YamakasY: saggi: that's kewl! > >> 9:59:25 AM YamakasY: saggi: what happens ? > >> 9:59:41 AM YamakasY: I know from Telsin (ping ping!) that he sees it going faster on gluster usage > >> tdosek left the room (quit: Ping timeout: 480 seconds). (10:00:02 AM) > >> djasa left the room (quit: Quit: Leaving). (10:00:24 AM) > >> mlipchuk left the room (quit: Quit: Leaving.). (10:00:29 AM) > >> laravot left the room (quit: Quit: Leaving.). (10:01:19 AM) > >> 10:01:54 AM saggi: YamakasY: it's in getCapabilities(). Here is the RSS graph. The flatlines are when I stopped calling it and called other verbs. http://i.imgur.com/CLm0Q75.png > > > > I do recall what is the issue Saggi and YamakasY were dicussing (CCing > > the pair), or if it reached fruition as a patch. It is certainly > > something other than Bug 1158108, as the latter speak about a leak in a > > normal working state, with no getCapabilities calls. > > > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users -- Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de www.monkeymen.tv Geschäftsführer: Martin Retschitzegger / Michaela Göllner Handeslregister: Amtsgericht Charlottenburg / HRB 112767