[ovirt-users] VDSM memory consumption

Thu Mar 26 12:56:31 UTC 2015

On 26/03/15 09:43, Matt . wrote:
> Hi Daniel,
>
> Great! Thanks.
>
> I only see this issue happening on CentOS 7, Joop van de Wege also
> confirmed he didn't see it on CentOS 6.
>
> Cheers,
>
> Matt
I have experienced the same issue on Centos 6.6 and Centos 7 both 
managed by the same engine.

Cheers

Federico
>
> 2015-03-26 13:33 GMT+01:00 Daniel Helgenberger <daniel.helgenberger at m-box.de>:
>> Hello Everyone,
>>
>> I did create the original BZ on this. In the mean time, lab system I
>> used is dismantled and the production system is yet to deploy.
>>
>> As I wrote in BZ1147148 [1], I experienced two different issues. One,
>> one big mem leak of about 15MiB/h and a smaller one, ~300KiB. These seem
>> unrelated.
>>
>> The larger leak was indeed related to SSL in some way; not necessarily
>> M2Crypto. However, after disabling SSL this was gone leaving the smaller
>> leak.
>>
>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1147148
>> On Mo, 2015-03-09 at 23:49 +0100, Matt . wrote:
>>> Hi,
>>>
>>> I also see this on the latest 3.5 version, I'm thinking about setting
>>> up a cronjob to restart vdsm every night.
>> I did the same thing. In general, it seems to be a bad idea as it
>> compromised system stability on the long run. While VMs seem to be fine,
>> engine does not like this very much.
>>
>>> I cannot believe that people say they don't have this issue.
>> This was hard for me to accept as well. I know of Markus Stockhausen and
>> Seven Kieske, both confirmed the small leak. This might also be some
>> special other service; though I started out with a minimal install of
>> Centos 6.
>>> Can someone of the devs dive in maybe ?
>>>
>>> Thanks!
>>>
>>> Matt
>>>
>>>
>>>
>>> 2015-03-09 23:29 GMT+01:00 Dan Kenigsberg <danken at redhat.com>:
>>>> On Mon, Mar 09, 2015 at 10:40:51AM -0500, Darrell Budic wrote:
>>>>>> On Mar 9, 2015, at 4:51 AM, Dan Kenigsberg <danken at redhat.com> wrote:
>>>>>>
>>>>>> On Fri, Mar 06, 2015 at 10:58:53AM -0600, Darrell Budic wrote:
>>>>>>> I believe the supervdsm leak was fixed, but 3.5.1 versions of vdsmd still leaks slowly, ~300k/hr, yes.
>>>>>>>
>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1158108
>>>>>>>
>>>>>>>
>>>>>>>> On Mar 6, 2015, at 10:23 AM, Chris Adams <cma at cmadams.net> wrote:
>>>>>>>>
>>>>>>>> Once upon a time, Federico Alberto Sayd <fsayd at uncu.edu.ar> said:
>>>>>>>>> I am experiencing troubles with VDSM memory consuption.
>>>>>>>>>
>>>>>>>>> I am running
>>>>>>>>>
>>>>>>>>> Engine: ovirt 3.5.1
>>>>>>>>>
>>>>>>>>> Nodes:
>>>>>>>>>
>>>>>>>>> Centos 6.6
>>>>>>>>> VDSM 4.16.10-8
>>>>>>>>> Libvirt: libvirt-0.10.2-46
>>>>>>>>> Kernel: 2.6.32
>>>>>>>>>
>>>>>>>>> When the host boots, memory consuption is normal, but after 2 or 3
>>>>>>>>> days running, VDSM memory consuption grows and it consumes more
>>>>>>>>> memory that all vm's running in the host. If I restart the vdsm
>>>>>>>>> service, memory consuption normalizes, but then it start growing
>>>>>>>>> again.
>>>>>>>>>
>>>>>>>>> I have seen some BZ about vdsm and supervdsm about memory leaks, but
>>>>>>>>> I don't know if VDSM 4.6.10.8 is still affected by a related bug.
>>>>>>>> Can't help, but I see the same thing with CentOS 7 nodes and the same
>>>>>>>> version of vdsm.
>>>>>>>> --
>>>>>>>> Chris Adams <cma at cmadams.net>
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list
>>>>>>>> Users at ovirt.org
>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>> I'm afraid that we are yet to find a solution for this issue, which is
>>>>>> completly different from the horrible leak of supervdsm < 4.16.7.
>>>>>>
>>>>>> Could you corroborate the claim of
>>>>>>     Bug 1147148 - M2Crypto usage in vdsm leaks memory
>>>>>> ? Does the leak disappear once you start using plaintext transport?
>>>>>>
>>>>>> Regards,
>>>>>> Dan.
>>>>> I don’t think this is crypto related, but I could try that if you still need some confirmation (and point me at a quick doc on switching to plaintext?).
>>>>>
>>>>> This is from #ovirt around November 18th I think, Saggi thought he’d found something related:
>>>>>
>>>>> 9:58:43 AM saggi: YamakasY: Found the leak
>>>>> 9:58:48 AM saggi: YamakasY: Or at least the flow
>>>>> 9:58:57 AM saggi: YamakasY: The good news is that I can reproduce
>>>>> 9:59:20 AM YamakasY: saggi: that's kewl!
>>>>> 9:59:25 AM YamakasY: saggi: what happens ?
>>>>> 9:59:41 AM YamakasY: I know from Telsin (ping ping!) that he sees it going faster on gluster usage
>>>>> tdosek left the room (quit: Ping timeout: 480 seconds). (10:00:02 AM)
>>>>> djasa left the room (quit: Quit: Leaving). (10:00:24 AM)
>>>>> mlipchuk left the room (quit: Quit: Leaving.). (10:00:29 AM)
>>>>> laravot left the room (quit: Quit: Leaving.). (10:01:19 AM)
>>>>> 10:01:54 AM saggi: YamakasY: it's in getCapabilities(). Here is the RSS graph. The flatlines are when I stopped calling it and called other verbs. http://i.imgur.com/CLm0Q75.png
>>>> I do recall what is the issue Saggi and YamakasY were dicussing (CCing
>>>> the pair), or if it reached fruition as a patch. It is certainly
>>>> something other than Bug 1158108, as the latter speak about a leak in a
>>>> normal working state, with no getCapabilities calls.
>>>>
>>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>> --
>> Daniel Helgenberger
>> m box bewegtbild GmbH
>>
>> P: +49/30/2408781-22
>> F: +49/30/2408781-10
>>
>> ACKERSTR. 19
>> D-10115 BERLIN
>>
>>
>> www.m-box.de  www.monkeymen.tv
>>
>> Geschäftsführer: Martin Retschitzegger / Michaela Göllner
>> Handeslregister: Amtsgericht Charlottenburg / HRB 112767
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users