On Mar 6, 2015, at 10:23 AM, Chris Adams <cma(a)cmadams.net&gt; wrote: Once upon a time, Federico Alberto Sayd <fsayd(a)uncu.edu.ar&gt; said: > I am experiencing troubles with VDSM memory consuption. > > I am running > > Engine: ovirt 3.5.1 > > Nodes: > > Centos 6.6 > VDSM 4.16.10-8 > Libvirt: libvirt-0.10.2-46 > Kernel: 2.6.32 > > When the host boots, memory consuption is normal, but after 2 or 3 > days running, VDSM memory consuption grows and it consumes more > memory that all vm's running in the host. If I restart the vdsm > service, memory consuption normalizes, but then it start growing > again. > > I have seen some BZ about vdsm and supervdsm about memory leaks, but > I don't know if VDSM 4.6.10.8 is still affected by a related bug. Can't help, but I see the same thing with CentOS 7 nodes and the same version of vdsm. -- Chris Adams <cma(a)cmadams.net&gt; _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

On Mar 9, 2015, at 4:51 AM, Dan Kenigsberg <danken(a)redhat.com&gt; wrote: On Fri, Mar 06, 2015 at 10:58:53AM -0600, Darrell Budic wrote: > I believe the supervdsm leak was fixed, but 3.5.1 versions of vdsmd still leaks slowly, ~300k/hr, yes. > > https://bugzilla.redhat.com/show_bug.cgi?id=1158108 > > >> On Mar 6, 2015, at 10:23 AM, Chris Adams <cma(a)cmadams.net&gt; wrote: >> >> Once upon a time, Federico Alberto Sayd <fsayd(a)uncu.edu.ar&gt; said: >>> I am experiencing troubles with VDSM memory consuption. >>> >>> I am running >>> >>> Engine: ovirt 3.5.1 >>> >>> Nodes: >>> >>> Centos 6.6 >>> VDSM 4.16.10-8 >>> Libvirt: libvirt-0.10.2-46 >>> Kernel: 2.6.32 >>> >>> When the host boots, memory consuption is normal, but after 2 or 3 >>> days running, VDSM memory consuption grows and it consumes more >>> memory that all vm's running in the host. If I restart the vdsm >>> service, memory consuption normalizes, but then it start growing >>> again. >>> >>> I have seen some BZ about vdsm and supervdsm about memory leaks, but >>> I don't know if VDSM 4.6.10.8 is still affected by a related bug. >> >> Can't help, but I see the same thing with CentOS 7 nodes and the same >> version of vdsm. >> -- >> Chris Adams <cma(a)cmadams.net&gt; >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users I'm afraid that we are yet to find a solution for this issue, which is completly different from the horrible leak of supervdsm < 4.16.7. Could you corroborate the claim of Bug 1147148 - M2Crypto usage in vdsm leaks memory ? Does the leak disappear once you start using plaintext transport? Regards, Dan.

On Mon, Mar 09, 2015 at 10:40:51AM -0500, Darrell Budic wrote: > > On Mar 9, 2015, at 4:51 AM, Dan Kenigsberg <danken(a)redhat.com&gt; wrote: > > > > On Fri, Mar 06, 2015 at 10:58:53AM -0600, Darrell Budic wrote: > >> I believe the supervdsm leak was fixed, but 3.5.1 versions of vdsmd still leaks slowly, ~300k/hr, yes. > >> > >> https://bugzilla.redhat.com/show_bug.cgi?id=1158108 > >> > >> > >>> On Mar 6, 2015, at 10:23 AM, Chris Adams <cma(a)cmadams.net&gt; wrote: > >>> > >>> Once upon a time, Federico Alberto Sayd <fsayd(a)uncu.edu.ar&gt; said: > >>>> I am experiencing troubles with VDSM memory consuption. > >>>> > >>>> I am running > >>>> > >>>> Engine: ovirt 3.5.1 > >>>> > >>>> Nodes: > >>>> > >>>> Centos 6.6 > >>>> VDSM 4.16.10-8 > >>>> Libvirt: libvirt-0.10.2-46 > >>>> Kernel: 2.6.32 > >>>> > >>>> When the host boots, memory consuption is normal, but after 2 or 3 > >>>> days running, VDSM memory consuption grows and it consumes more > >>>> memory that all vm's running in the host. If I restart the vdsm > >>>> service, memory consuption normalizes, but then it start growing > >>>> again. > >>>> > >>>> I have seen some BZ about vdsm and supervdsm about memory leaks, but > >>>> I don't know if VDSM 4.6.10.8 is still affected by a related bug. > >>> > >>> Can't help, but I see the same thing with CentOS 7 nodes and the same > >>> version of vdsm. > >>> -- > >>> Chris Adams <cma(a)cmadams.net&gt; > >>> _______________________________________________ > >>> Users mailing list > >>> Users(a)ovirt.org > >>> http://lists.ovirt.org/mailman/listinfo/users > > > > I'm afraid that we are yet to find a solution for this issue, which is > > completly different from the horrible leak of supervdsm < 4.16.7. > > > > Could you corroborate the claim of > > Bug 1147148 - M2Crypto usage in vdsm leaks memory > > ? Does the leak disappear once you start using plaintext transport? > > > > Regards, > > Dan. > > I don’t think this is crypto related, but I could try that if you still need some confirmation (and point me at a quick doc on switching to plaintext?). > > This is from #ovirt around November 18th I think, Saggi thought he’d found something related: > > 9:58:43 AM saggi: YamakasY: Found the leak > 9:58:48 AM saggi: YamakasY: Or at least the flow > 9:58:57 AM saggi: YamakasY: The good news is that I can reproduce > 9:59:20 AM YamakasY: saggi: that's kewl! > 9:59:25 AM YamakasY: saggi: what happens ? > 9:59:41 AM YamakasY: I know from Telsin (ping ping!) that he sees it going faster on gluster usage > tdosek left the room (quit: Ping timeout: 480 seconds). (10:00:02 AM) > djasa left the room (quit: Quit: Leaving). (10:00:24 AM) > mlipchuk left the room (quit: Quit: Leaving.). (10:00:29 AM) > laravot left the room (quit: Quit: Leaving.). (10:01:19 AM) > 10:01:54 AM saggi: YamakasY: it's in getCapabilities(). Here is the RSS graph. The flatlines are when I stopped calling it and called other verbs. http://i.imgur.com/CLm0Q75.png I do recall what is the issue Saggi and YamakasY were dicussing (CCing the pair), or if it reached fruition as a patch. It is certainly something other than Bug 1158108, as the latter speak about a leak in a normal working state, with no getCapabilities calls.

Hi, I also see this on the latest 3.5 version, I'm thinking about setting up a cronjob to restart vdsm every night. I cannot believe that people say they don't have this issue. Can someone of the devs dive in maybe ?

>> 10:01:54 AM saggi: YamakasY: it's in getCapabilities(). Here is the RSS graph. The flatlines are when I stopped calling it and called other verbs. http://i.imgur.com/CLm0Q75.png > > I do ***NOT*** recall what is the issue Saggi and YamakasY were dicussing (CCing > the pair), or if it reached fruition as a patch. It is certainly > something other than Bug 1158108, as the latter speak about a leak in a > normal working state, with no getCapabilities calls.

Hi, I also see this on the latest 3.5 version, I'm thinking about setting up a cronjob to restart vdsm every night.

I cannot believe that people say they don't have this issue.

Can someone of the devs dive in maybe ? Thanks! Matt 2015-03-09 23:29 GMT+01:00 Dan Kenigsberg <danken(a)redhat.com&gt;: > On Mon, Mar 09, 2015 at 10:40:51AM -0500, Darrell Budic wrote: >> > On Mar 9, 2015, at 4:51 AM, Dan Kenigsberg <danken(a)redhat.com&gt; wrote: >> > >> > On Fri, Mar 06, 2015 at 10:58:53AM -0600, Darrell Budic wrote: >> >> I believe the supervdsm leak was fixed, but 3.5.1 versions of vdsmd still leaks slowly, ~300k/hr, yes. >> >> >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1158108 >> >> >> >> >> >>> On Mar 6, 2015, at 10:23 AM, Chris Adams <cma(a)cmadams.net&gt; wrote: >> >>> >> >>> Once upon a time, Federico Alberto Sayd <fsayd(a)uncu.edu.ar&gt; said: >> >>>> I am experiencing troubles with VDSM memory consuption. >> >>>> >> >>>> I am running >> >>>> >> >>>> Engine: ovirt 3.5.1 >> >>>> >> >>>> Nodes: >> >>>> >> >>>> Centos 6.6 >> >>>> VDSM 4.16.10-8 >> >>>> Libvirt: libvirt-0.10.2-46 >> >>>> Kernel: 2.6.32 >> >>>> >> >>>> When the host boots, memory consuption is normal, but after 2 or 3 >> >>>> days running, VDSM memory consuption grows and it consumes more >> >>>> memory that all vm's running in the host. If I restart the vdsm >> >>>> service, memory consuption normalizes, but then it start growing >> >>>> again. >> >>>> >> >>>> I have seen some BZ about vdsm and supervdsm about memory leaks, but >> >>>> I don't know if VDSM 4.6.10.8 is still affected by a related bug. >> >>> >> >>> Can't help, but I see the same thing with CentOS 7 nodes and the same >> >>> version of vdsm. >> >>> -- >> >>> Chris Adams <cma(a)cmadams.net&gt; >> >>> _______________________________________________ >> >>> Users mailing list >> >>> Users(a)ovirt.org >> >>> http://lists.ovirt.org/mailman/listinfo/users >> > >> > I'm afraid that we are yet to find a solution for this issue, which is >> > completly different from the horrible leak of supervdsm < 4.16.7. >> > >> > Could you corroborate the claim of >> > Bug 1147148 - M2Crypto usage in vdsm leaks memory >> > ? Does the leak disappear once you start using plaintext transport? >> > >> > Regards, >> > Dan. >> >> I don’t think this is crypto related, but I could try that if you still need some confirmation (and point me at a quick doc on switching to plaintext?). >> >> This is from #ovirt around November 18th I think, Saggi thought he’d found something related: >> >> 9:58:43 AM saggi: YamakasY: Found the leak >> 9:58:48 AM saggi: YamakasY: Or at least the flow >> 9:58:57 AM saggi: YamakasY: The good news is that I can reproduce >> 9:59:20 AM YamakasY: saggi: that's kewl! >> 9:59:25 AM YamakasY: saggi: what happens ? >> 9:59:41 AM YamakasY: I know from Telsin (ping ping!) that he sees it going faster on gluster usage >> tdosek left the room (quit: Ping timeout: 480 seconds). (10:00:02 AM) >> djasa left the room (quit: Quit: Leaving). (10:00:24 AM) >> mlipchuk left the room (quit: Quit: Leaving.). (10:00:29 AM) >> laravot left the room (quit: Quit: Leaving.). (10:01:19 AM) >> 10:01:54 AM saggi: YamakasY: it's in getCapabilities(). Here is the RSS graph. The flatlines are when I stopped calling it and called other verbs. http://i.imgur.com/CLm0Q75.png > > I do recall what is the issue Saggi and YamakasY were dicussing (CCing > the pair), or if it reached fruition as a patch. It is certainly > something other than Bug 1158108, as the latter speak about a leak in a > normal working state, with no getCapabilities calls. > > _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

Hi Daniel, Great! Thanks. I only see this issue happening on CentOS 7, Joop van de Wege also confirmed he didn't see it on CentOS 6. Cheers, Matt

2015-03-26 13:33 GMT+01:00 Daniel Helgenberger <daniel.helgenberger(a)m-box.de&gt;: > Hello Everyone, > > I did create the original BZ on this. In the mean time, lab system I > used is dismantled and the production system is yet to deploy. > > As I wrote in BZ1147148 [1], I experienced two different issues. One, > one big mem leak of about 15MiB/h and a smaller one, ~300KiB. These seem > unrelated. > > The larger leak was indeed related to SSL in some way; not necessarily > M2Crypto. However, after disabling SSL this was gone leaving the smaller > leak. > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1147148 > On Mo, 2015-03-09 at 23:49 +0100, Matt . wrote: >> Hi, >> >> I also see this on the latest 3.5 version, I'm thinking about setting >> up a cronjob to restart vdsm every night. > I did the same thing. In general, it seems to be a bad idea as it > compromised system stability on the long run. While VMs seem to be fine, > engine does not like this very much. > >> I cannot believe that people say they don't have this issue. > This was hard for me to accept as well. I know of Markus Stockhausen and > Seven Kieske, both confirmed the small leak. This might also be some > special other service; though I started out with a minimal install of > Centos 6. >> Can someone of the devs dive in maybe ? >> >> Thanks! >> >> Matt >> >> >> >> 2015-03-09 23:29 GMT+01:00 Dan Kenigsberg <danken(a)redhat.com&gt;: >>> On Mon, Mar 09, 2015 at 10:40:51AM -0500, Darrell Budic wrote: >>>>> On Mar 9, 2015, at 4:51 AM, Dan Kenigsberg <danken(a)redhat.com&gt; wrote: >>>>> >>>>> On Fri, Mar 06, 2015 at 10:58:53AM -0600, Darrell Budic wrote: >>>>>> I believe the supervdsm leak was fixed, but 3.5.1 versions of vdsmd still leaks slowly, ~300k/hr, yes. >>>>>> >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1158108 >>>>>> >>>>>> >>>>>>> On Mar 6, 2015, at 10:23 AM, Chris Adams <cma(a)cmadams.net&gt; wrote: >>>>>>> >>>>>>> Once upon a time, Federico Alberto Sayd <fsayd(a)uncu.edu.ar&gt; said: >>>>>>>> I am experiencing troubles with VDSM memory consuption. >>>>>>>> >>>>>>>> I am running >>>>>>>> >>>>>>>> Engine: ovirt 3.5.1 >>>>>>>> >>>>>>>> Nodes: >>>>>>>> >>>>>>>> Centos 6.6 >>>>>>>> VDSM 4.16.10-8 >>>>>>>> Libvirt: libvirt-0.10.2-46 >>>>>>>> Kernel: 2.6.32 >>>>>>>> >>>>>>>> When the host boots, memory consuption is normal, but after 2 or 3 >>>>>>>> days running, VDSM memory consuption grows and it consumes more >>>>>>>> memory that all vm's running in the host. If I restart the vdsm >>>>>>>> service, memory consuption normalizes, but then it start growing >>>>>>>> again. >>>>>>>> >>>>>>>> I have seen some BZ about vdsm and supervdsm about memory leaks, but >>>>>>>> I don't know if VDSM 4.6.10.8 is still affected by a related bug. >>>>>>> Can't help, but I see the same thing with CentOS 7 nodes and the same >>>>>>> version of vdsm. >>>>>>> -- >>>>>>> Chris Adams <cma(a)cmadams.net&gt; >>>>>>> _______________________________________________ >>>>>>> Users mailing list >>>>>>> Users(a)ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> I'm afraid that we are yet to find a solution for this issue, which is >>>>> completly different from the horrible leak of supervdsm < 4.16.7. >>>>> >>>>> Could you corroborate the claim of >>>>> Bug 1147148 - M2Crypto usage in vdsm leaks memory >>>>> ? Does the leak disappear once you start using plaintext transport? >>>>> >>>>> Regards, >>>>> Dan. >>>> I don’t think this is crypto related, but I could try that if you still need some confirmation (and point me at a quick doc on switching to plaintext?). >>>> >>>> This is from #ovirt around November 18th I think, Saggi thought he’d found something related: >>>> >>>> 9:58:43 AM saggi: YamakasY: Found the leak >>>> 9:58:48 AM saggi: YamakasY: Or at least the flow >>>> 9:58:57 AM saggi: YamakasY: The good news is that I can reproduce >>>> 9:59:20 AM YamakasY: saggi: that's kewl! >>>> 9:59:25 AM YamakasY: saggi: what happens ? >>>> 9:59:41 AM YamakasY: I know from Telsin (ping ping!) that he sees it going faster on gluster usage >>>> tdosek left the room (quit: Ping timeout: 480 seconds). (10:00:02 AM) >>>> djasa left the room (quit: Quit: Leaving). (10:00:24 AM) >>>> mlipchuk left the room (quit: Quit: Leaving.). (10:00:29 AM) >>>> laravot left the room (quit: Quit: Leaving.). (10:01:19 AM) >>>> 10:01:54 AM saggi: YamakasY: it's in getCapabilities(). Here is the RSS graph. The flatlines are when I stopped calling it and called other verbs. http://i.imgur.com/CLm0Q75.png >>> I do recall what is the issue Saggi and YamakasY were dicussing (CCing >>> the pair), or if it reached fruition as a patch. It is certainly >>> something other than Bug 1158108, as the latter speak about a leak in a >>> normal working state, with no getCapabilities calls. >>> >>> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users > -- > Daniel Helgenberger > m box bewegtbild GmbH > > P: +49/30/2408781-22 > F: +49/30/2408781-10 > > ACKERSTR. 19 > D-10115 BERLIN > > > www.m-box.de www.monkeymen.tv > > Geschäftsführer: Martin Retschitzegger / Michaela Göllner > Handeslregister: Amtsgericht Charlottenburg / HRB 112767 _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

Daniel Helgenberger <daniel.helgenberger(a)m-box.de&gt; writes: > Hello Everyone, > > I did create the original BZ on this. In the mean time, lab system I > used is dismantled and the production system is yet to deploy. > > As I wrote in BZ1147148 [1], I experienced two different issues. One, > one big mem leak of about 15MiB/h and a smaller one, ~300KiB. These seem > unrelated. > > The larger leak was indeed related to SSL in some way; not necessarily > M2Crypto. However, after disabling SSL this was gone leaving the smaller > leak. > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1147148 I think there are, at least for the purpose of this discussion, 3 leaks: 1. the M2Crypto leak 2. a slower leak 3. a large leak that's not M2Crypto related that's part of sampling My efforts have been around finding the source of my larger leak, which I think is #3. I had disabled ssl so I knew that M2Crypto isn't/shouldn't be the problem as in bz1147148, and ssl is beside the point as it happens with a deactived host. It's part of sampling which always runs. What I've found is, after trying to get the smallest reproducer, that it's not the netlink.iter_links that I commented on in [1] that is the problem. But in the _get_intefaces_and_samples loop is the call to create an InterfaceSample and that has getLinkSpeed() which, for vlans, ends up calling ipwrapper.getLink, and that to netlink.get_link(name) netlink.get_link(name) *is* the source of my big leak. This is vdsm 4.16.10, so it is [2] and it's been changed in master for the removal of support for libnl v1 so it might not be a problem anymore. def get_link(name): """Returns the information dictionary of the name specified link.""" with _pool.socket() as sock: with _nl_link_cache(sock) as cache: link = _rtnl_link_get_by_name(cache, name) if not link: raise IOError(errno.ENODEV, '%s is not present in the system' % name) return _link_info(cache, link) The libnl documentation note at [3] says that for the rtnl_link_get_by_name function "Attention The reference counter of the returned link object will be incremented. Use rtnl_link_put() to release the reference." So I took that hint, and made a change that does the rtnl_link_put() in get_link(name) and it looks like it works for me. diff oldnetlink.py netlink.py 67d66 < return _link_info(cache, link) 68a68,70 > li = _link_info(cache, link) > _rtnl_link_put(link) > return li 333a336,337 > > _rtnl_link_put = _none_proto(('rtnl_link_put', LIBNL_ROUTE)) Hope that helps. And if someone else could confirm that would be great. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1158108 [2] https://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=lib/vdsm/netlink.py;h... [3] http://www.infradead.org/~tgr/libnl/doc/api/group__link.html#ga1d583e4f0b...

On Mar 30, 2015, at 4:14 PM, John Taylor <jtt77777(a)yahoo.com&gt; wrote: Dan Kenigsberg <danken(a)redhat.com&gt; writes: > On Sat, Mar 28, 2015 at 10:20:25AM -0400, John Taylor wrote: >> Daniel Helgenberger <daniel.helgenberger(a)m-box.de&gt; writes: >> >>> Hello Everyone, >>> >>> I did create the original BZ on this. In the mean time, lab system I >>> used is dismantled and the production system is yet to deploy. >>> >>> As I wrote in BZ1147148 [1], I experienced two different issues. One, >>> one big mem leak of about 15MiB/h and a smaller one, ~300KiB. These seem >>> unrelated. >>> >>> The larger leak was indeed related to SSL in some way; not necessarily >>> M2Crypto. However, after disabling SSL this was gone leaving the smaller >>> leak. >>> >>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1147148 >> >> >> I think there are, at least for the purpose of this discussion, 3 leaks: >> 1. the M2Crypto leak >> 2. a slower leak >> 3. a large leak that's not M2Crypto related that's part of sampling >> >> My efforts have been around finding the source of my larger leak, which >> I think is #3. I had disabled ssl so I knew that M2Crypto >> isn't/shouldn't be the problem as in bz1147148, and ssl is beside the >> point as it happens with a deactived host. It's part of sampling which >> always runs. >> >> What I've found is, after trying to get the smallest reproducer, that >> it's not the netlink.iter_links that I commented on in [1] that is the >> problem. But in the _get_intefaces_and_samples loop is the call to >> create an InterfaceSample and that has getLinkSpeed() which, for vlans, >> ends up calling ipwrapper.getLink, and that to >> netlink.get_link(name) >> >> netlink.get_link(name) *is* the source of my big leak. This is vdsm >> 4.16.10, so it is [2] and it's been changed in master for the removal of >> support for libnl v1 so it might not be a problem anymore. >> >> def get_link(name): >> """Returns the information dictionary of the name specified link.""" >> with _pool.socket() as sock: >> with _nl_link_cache(sock) as cache: >> link = _rtnl_link_get_by_name(cache, name) >> if not link: >> raise IOError(errno.ENODEV, '%s is not present in the system' % >> name) >> return _link_info(cache, link) >> >> >> The libnl documentation note at [3] says that for the rtnl_link_get_by_name function >> "Attention >> The reference counter of the returned link object will be incremented. Use rtnl_link_put() to release the reference." >> >> So I took that hint, and made a change that does the rtnl_link_put() in >> get_link(name) and it looks like it works for me. >> >> diff oldnetlink.py netlink.py >> 67d66 >> < return _link_info(cache, link) >> 68a68,70 >>> li = _link_info(cache, link) >>> _rtnl_link_put(link) >>> return li >> 333a336,337 >>> >>> _rtnl_link_put = _none_proto(('rtnl_link_put', LIBNL_ROUTE)) >> >> Hope that helps. And if someone else could confirm that would be great. >> >> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1158108 >> [2] >> https://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=lib/vdsm/netlink.py;h... >> [3] http://www.infradead.org/~tgr/libnl/doc/api/group__link.html#ga1d583e4f0b... > > Thanks, John, for a great detective work. > > I'm afraid that with even on the master branch we keep calling > rtnl_link_get_link() and rtnl_link_get_by_name() without clearing the > reference count, so a fix is due there, too. > > Would you consider posting a fully-fledged fix to gerrit? I still need > to understand what is the use of that refcount, so that we do not > release it too early. > > Regards, > Dan. Dan, I'm happy to [1], although I've probably gotten something wrong with how it's supposed to be done :) It's for the version I'm using so it's for branch ovirt-3.5. [1] https://gerrit.ovirt.org/#/c/39372/ Thanks, -John _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

Once upon a time, Dan Kenigsberg <danken(a)redhat.com&gt; said: > I'm afraid that we are yet to find a solution for this issue, which is > completly different from the horrible leak of supervdsm < 4.16.7. > > Could you corroborate the claim of > Bug 1147148 - M2Crypto usage in vdsm leaks memory > ? Does the leak disappear once you start using plaintext transport? So, to confirm, it looks like to do that, the steps would be: - In the [vars] section of /etc/vdsm/vdsm.conf, set "ssl = false". - Restart the vdsmd service. Is that all that is needed?

Is it safe to restart vdsmd on a node with active VMs?

Hello: I am experiencing troubles with VDSM memory consuption. I am running Engine: ovirt 3.5.1 Nodes: Centos 6.6 VDSM 4.16.10-8 Libvirt: libvirt-0.10.2-46 Kernel: 2.6.32 When the host boots, memory consuption is normal, but after 2 or 3 days running, VDSM memory consuption grows and it consumes more memory that all vm's running in the host. If I restart the vdsm service, memory consuption normalizes, but then it start growing again. I have seen some BZ about vdsm and supervdsm about memory leaks, but I don't know if VDSM 4.6.10.8 is still affected by a related bug. Any help? If you need, I can provide more information Thank you

On 13/03/15 12:29, Kapetanakis Giannis wrote: > We also face this problem since 3.5 in two different installations... > Hope it's fixed soon Nothing will get fixed if no one bothers to open BZs and send relevants log files to help track down the problems.

Chris Adams <cma(a)cmadams.net&gt; writes: > Once upon a time, Sven Kieske <s.kieske(a)mittwald.de&gt; said: >> On 13/03/15 12:29, Kapetanakis Giannis wrote: >> > We also face this problem since 3.5 in two different installations... >> > Hope it's fixed soon >> >> Nothing will get fixed if no one bothers to >> open BZs and send relevants log files to help >> track down the problems. > > There's already an open BZ: > > https://bugzilla.redhat.com/show_bug.cgi?id=1158108 > > I'm not sure if that is exactly the same problem I'm seeing or not; my > vdsm process seems to be growing faster (RSS grew 952K in a 5 minute > period just now; VSZ didn't change). For those following this I've added a comment on the bz [1], although in my case the memory leak is, like Chris Adams, a lot more than the 300KiB/h in the original bug report by Daniel Helgenberger . [1] https://bugzilla.redhat.com/show_bug.cgi?id=1158108

> On Mar 24, 2015, at 4:33 AM, Dan Kenigsberg <danken(a)redhat.com&gt; wrote: > > On Mon, Mar 23, 2015 at 04:00:14PM -0400, John Taylor wrote: >> Chris Adams <cma(a)cmadams.net&gt; writes: >> >>> Once upon a time, Sven Kieske <s.kieske(a)mittwald.de&gt; said: >>>> On 13/03/15 12:29, Kapetanakis Giannis wrote: >>>>> We also face this problem since 3.5 in two different installations... >>>>> Hope it's fixed soon >>>> >>>> Nothing will get fixed if no one bothers to >>>> open BZs and send relevants log files to help >>>> track down the problems. >>> >>> There's already an open BZ: >>> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1158108 >>> >>> I'm not sure if that is exactly the same problem I'm seeing or not; my >>> vdsm process seems to be growing faster (RSS grew 952K in a 5 minute >>> period just now; VSZ didn't change). >> >> For those following this I've added a comment on the bz [1], although in >> my case the memory leak is, like Chris Adams, a lot more than the 300KiB/h >> in the original bug report by Daniel Helgenberger . >> >> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1158108 > > That's interesting (and worrying). > Could you check your suggestion by editing sampling.py so that > _get_interfaces_and_samples() returns the empty dict immediately? > Would this make the leak disappear? Looks like you’ve got something there. Just a quick test for now, watching RSS in top. I’ll let it go this way for a while and see what it looks in a few hours. System 1: 13 VMs w/ 24 interfaces between them 11:47 killed a vdsm @ 9.116G RSS (after maybe a week and a half running) 11:47: 97xxx 11:57 135544 and climbing 12:00 136400 restarted with sampling.py modified to just return empty set: def _get_interfaces_and_samples(): links_and_samples = {} return links_and_samples

> On Mar 25, 2015, at 5:34 AM, Dan Kenigsberg <danken(a)redhat.com&gt; wrote: > > On Tue, Mar 24, 2015 at 02:01:40PM -0500, Darrell Budic wrote: >> >>> On Mar 24, 2015, at 4:33 AM, Dan Kenigsberg <danken(a)redhat.com&gt; wrote: >>> >>> On Mon, Mar 23, 2015 at 04:00:14PM -0400, John Taylor wrote: >>>> Chris Adams <cma(a)cmadams.net&gt; writes: >>>> >>>>> Once upon a time, Sven Kieske <s.kieske(a)mittwald.de&gt; said: >>>>>> On 13/03/15 12:29, Kapetanakis Giannis wrote: >>>>>>> We also face this problem since 3.5 in two different installations... >>>>>>> Hope it's fixed soon >>>>>> >>>>>> Nothing will get fixed if no one bothers to >>>>>> open BZs and send relevants log files to help >>>>>> track down the problems. >>>>> >>>>> There's already an open BZ: >>>>> >>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1158108 >>>>> >>>>> I'm not sure if that is exactly the same problem I'm seeing or not; my >>>>> vdsm process seems to be growing faster (RSS grew 952K in a 5 minute >>>>> period just now; VSZ didn't change). >>>> >>>> For those following this I've added a comment on the bz [1], although in >>>> my case the memory leak is, like Chris Adams, a lot more than the 300KiB/h >>>> in the original bug report by Daniel Helgenberger . >>>> >>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1158108 >>> >>> That's interesting (and worrying). >>> Could you check your suggestion by editing sampling.py so that >>> _get_interfaces_and_samples() returns the empty dict immediately? >>> Would this make the leak disappear? >> >> Looks like you’ve got something there. Just a quick test for now, watching RSS in top. I’ll let it go this way for a while and see what it looks in a few hours. >> >> System 1: 13 VMs w/ 24 interfaces between them >> >> 11:47 killed a vdsm @ 9.116G RSS (after maybe a week and a half running) >> >> 11:47: 97xxx >> 11:57 135544 and climbing >> 12:00 136400 >> >> restarted with sampling.py modified to just return empty set: >> >> def _get_interfaces_and_samples(): >> links_and_samples = {} >> return links_and_samples > > Thanks for the input. Just to be a little more certain that the culprit > is _get_interfaces_and_samples() per se, would you please decorate it > with memoized, and add a log line in the end > > @utils.memoized # add this line > def _get_interfaces_and_samples(): > ... > logging.debug('LINKS %s', links_and_samples) ## and this line > return links_and_samples > > I'd like to see what happens when the function is run only once, and > returns a non-empty reasonable dictionary of links and samples. Looks similar, I modified my second server for this test:

Yes, this script leaks quickly. Started out at a RSS of 21000ish, already at 26744 a minute in, about 5 minutes later it’s at 39384 and climbing. Been abusing a production server for those simple tests, but didn’t want to run valgrind against it right this minute. Did run it against the test.py script above though, got this (fpaste.org didn’t like, too long maybe?): http://tower.onholyground.com/valgrind-test.log To comment on some other posts in this thread, I also see leaks on my test system which is running Centos 6.6, but it only has 3 VMs across 2 servers and 3 configured networks and it leaks MUCH slower. I suspect people don’t notice this on test systems because they don’t have a lot of VMs/interfaces running, and don’t leave them up for weeks at a time. That’s why I was running these tests on my production box, to have more VMs up.