[ovirt-users] memory leak in 3.5.6 - not vdsm

Fri Jan 22 21:24:20 UTC 2016

On Fri, Jan 22, 2016 at 11:08 PM, Charles Kozler <charles at fixflyer.com> wrote:
> Hi Nir -
>
> Thanks for getting back to me. Will the patch to 3.6 be backported to 3.5?

We plan to include them in 3.5.8.

> As you can tell from the images, it takes days and days for it to increase
> over time. I also wasnt sure if that was the right bug because VDSM memory
> shows normal from top ...
>
>    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>   5134 vdsm       0 -20 4368m 111m  10m S  2.0  0.3   3709:28 vdsm

As you wrote, this issue is not related to vdsm.

>
> Res is only 111M. This is from node1 which is showing currently 20GB of 32GB
> used with only 2 VMs running on it - 1 with 4G and another with ~1 GB of RAM
> configured
>
> The images are from nagios and the value here is a direct correlation to
> what you would see in the free command output. See below from an example of
> node 1 and node 2
>
> [compute[root at node1 ~]$ free
>              total       used       free     shared    buffers     cached
> Mem:      32765316   20318156   12447160        252      30884     628948
> -/+ buffers/cache:   19658324   13106992
> Swap:     19247100          0   19247100
> [compute[root at node1 ~]$ free -m
>              total       used       free     shared    buffers     cached
> Mem:         31997      19843      12153          0         30        614
> -/+ buffers/cache:      19199      12798
> Swap:        18795          0      18795
>
> And its correlated image http://i.imgur.com/PZLEgyx.png (~19GB used)
>
> And as a control, node 2 that I just restarted today
>
> [compute[root at node2 ~]$ free
>              total       used       free     shared    buffers     cached
> Mem:      32765316    1815324   30949992        212      35784     717320
> -/+ buffers/cache:    1062220   31703096
> Swap:     19247100          0   19247100

Is this rhel/centos 6?

> [compute[root at node2 ~]$ free -m
>              total       used       free     shared    buffers     cached
> Mem:         31997       1772      30225          0         34        700
> -/+ buffers/cache:       1036      30960
> Swap:        18795          0      18795
>
> And its correlated image http://i.imgur.com/8ldPVqY.png  (~2GB used). Note
> how 1772 in the image is exactly what is registered under 'used' in free
> command

I guess you should start looking at the processes running on these nodes.

Maybe try to collect memory usage per process using ps?

>
> On Fri, Jan 22, 2016 at 3:59 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>>
>> On Fri, Jan 22, 2016 at 9:25 PM, Charles Kozler <charles at fixflyer.com>
>> wrote:
>> > Here is a screenshot of my three nodes and their increased memory usage
>> > over
>> > 30 days. Note that node #2 had 1 single VM that had 4GB of RAM assigned
>> > to
>> > it. I had since shut it down and saw no memory reclamation occur.
>> > Further, I
>> > flushed page caches and inodes and ran 'sync'. I tried everything but
>> > nothing brought the memory usage down. vdsm was low too (couple hundred
>> > MB)
>>
>> Note that there is an old leak in vdsm, will be fixed in next 3.6 build:
>> https://bugzilla.redhat.com/1269424
>>
>> > and there was no qemu-kvm process running so I'm at a loss
>> >
>> > http://imgur.com/a/aFPcK
>> >
>> > Please advise on what I can do to debug this. Note I have restarted node
>> > 2
>> > (which is why you see the drop) to see if it raises in memory use over
>> > tim
>> > even with no VM's running
>>
>> Not sure what is "memory" that you show in the graphs. Theoretically this
>> may be
>> normal memory usage, Linux using free memory for the buffer cache.
>>
>> Can you instead show the output of "free", during one day, maybe run once
>> per hour?
>>
>> You may also like to install sysstat for collecting and monitoring
>> resources usage.
>>
>> >
>> > [compute[root at node2 log]$ rpm -qa | grep -i ovirt
>> > libgovirt-0.3.2-1.el6.x86_64
>> > ovirt-release35-006-1.noarch
>> > ovirt-hosted-engine-ha-1.2.8-1.el6.noarch
>> > ovirt-hosted-engine-setup-1.2.6.1-1.el6.noarch
>> > ovirt-engine-sdk-python-3.5.6.0-1.el6.noarch
>> > ovirt-host-deploy-1.3.2-1.el6.noarch
>> >
>> >
>> > --
>> >
>> > Charles Kozler
>> > Vice President, IT Operations
>> >
>> > FIX Flyer, LLC
>> > 225 Broadway | Suite 1600 | New York, NY 10007
>> > 1-888-349-3593
>> > http://www.fixflyer.com
>> >
>> > NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED
>> > RECIPIENT(S)
>> > OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS
>> > PROPRIETARY TO FIX FLYER LLC.  ANY UNAUTHORIZED USE, COPYING,
>> > DISTRIBUTION,
>> > OR DISSEMINATION IS STRICTLY PROHIBITED.  ALL RIGHTS TO THIS INFORMATION
>> > IS
>> > RESERVED BY FIX FLYER LLC.  IF YOU ARE NOT THE INTENDED RECIPIENT,
>> > PLEASE
>> > CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS E-MAIL FROM
>> > YOUR
>> > SYSTEM AND DESTROY ANY COPIES.
>> >
>> > _______________________________________________
>> > Users mailing list
>> > Users at ovirt.org
>> > http://lists.ovirt.org/mailman/listinfo/users
>> >
>
>
>
>
> --
>
> Charles Kozler
> Vice President, IT Operations
>
> FIX Flyer, LLC
> 225 Broadway | Suite 1600 | New York, NY 10007
> 1-888-349-3593
> http://www.fixflyer.com
>
> NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED RECIPIENT(S)
> OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS
> PROPRIETARY TO FIX FLYER LLC.  ANY UNAUTHORIZED USE, COPYING, DISTRIBUTION,
> OR DISSEMINATION IS STRICTLY PROHIBITED.  ALL RIGHTS TO THIS INFORMATION IS
> RESERVED BY FIX FLYER LLC.  IF YOU ARE NOT THE INTENDED RECIPIENT, PLEASE
> CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS E-MAIL FROM YOUR
> SYSTEM AND DESTROY ANY COPIES.