Re: [ovirt-users] memory leak in 3.5.6 - not vdsm

On Fri, Jan 22, 2016 at 11:08 PM, Charles Kozler <charles(a)fixflyer.com&gt; wrote: > Hi Nir - > > Thanks for getting back to me. Will the patch to 3.6 be backported to 3.5? We plan to include them in 3.5.8. > As you can tell from the images, it takes days and days for it to increase > over time. I also wasnt sure if that was the right bug because VDSM memory > shows normal from top ... > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 5134 vdsm 0 -20 4368m 111m 10m S 2.0 0.3 3709:28 vdsm As you wrote, this issue is not related to vdsm. > > Res is only 111M. This is from node1 which is showing currently 20GB of 32GB > used with only 2 VMs running on it - 1 with 4G and another with ~1 GB of RAM > configured > > The images are from nagios and the value here is a direct correlation to > what you would see in the free command output. See below from an example of > node 1 and node 2 > > [compute[root@node1 ~]$ free > total used free shared buffers cached > Mem: 32765316 20318156 12447160 252 30884 628948 > -/+ buffers/cache: 19658324 13106992 > Swap: 19247100 0 19247100 > [compute[root@node1 ~]$ free -m > total used free shared buffers cached > Mem: 31997 19843 12153 0 30 614 > -/+ buffers/cache: 19199 12798 > Swap: 18795 0 18795 > > And its correlated image http://i.imgur.com/PZLEgyx.png (~19GB used) > > And as a control, node 2 that I just restarted today > > [compute[root@node2 ~]$ free > total used free shared buffers cached > Mem: 32765316 1815324 30949992 212 35784 717320 > -/+ buffers/cache: 1062220 31703096 > Swap: 19247100 0 19247100 Is this rhel/centos 6? > [compute[root@node2 ~]$ free -m > total used free shared buffers cached > Mem: 31997 1772 30225 0 34 700 > -/+ buffers/cache: 1036 30960 > Swap: 18795 0 18795 > > And its correlated image http://i.imgur.com/8ldPVqY.png (~2GB used). Note > how 1772 in the image is exactly what is registered under 'used' in free > command I guess you should start looking at the processes running on these nodes. Maybe try to collect memory usage per process using ps? > > On Fri, Jan 22, 2016 at 3:59 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >> >> On Fri, Jan 22, 2016 at 9:25 PM, Charles Kozler <charles(a)fixflyer.com&gt; >> wrote: >> > Here is a screenshot of my three nodes and their increased memory usage >> > over >> > 30 days. Note that node #2 had 1 single VM that had 4GB of RAM assigned >> > to >> > it. I had since shut it down and saw no memory reclamation occur. >> > Further, I >> > flushed page caches and inodes and ran 'sync'. I tried everything but >> > nothing brought the memory usage down. vdsm was low too (couple hundred >> > MB) >> >> Note that there is an old leak in vdsm, will be fixed in next 3.6 build: >> https://bugzilla.redhat.com/1269424 >> >> > and there was no qemu-kvm process running so I'm at a loss >> > >> > http://imgur.com/a/aFPcK >> > >> > Please advise on what I can do to debug this. Note I have restarted node >> > 2 >> > (which is why you see the drop) to see if it raises in memory use over >> > tim >> > even with no VM's running >> >> Not sure what is "memory" that you show in the graphs. Theoretically this >> may be >> normal memory usage, Linux using free memory for the buffer cache. >> >> Can you instead show the output of "free", during one day, maybe run once >> per hour? >> >> You may also like to install sysstat for collecting and monitoring >> resources usage. >> >> > >> > [compute[root@node2 log]$ rpm -qa | grep -i ovirt >> > libgovirt-0.3.2-1.el6.x86_64 >> > ovirt-release35-006-1.noarch >> > ovirt-hosted-engine-ha-1.2.8-1.el6.noarch >> > ovirt-hosted-engine-setup-1.2.6.1-1.el6.noarch >> > ovirt-engine-sdk-python-3.5.6.0-1.el6.noarch >> > ovirt-host-deploy-1.3.2-1.el6.noarch >> > >> > >> > -- >> > >> > Charles Kozler >> > Vice President, IT Operations >> > >> > FIX Flyer, LLC >> > 225 Broadway | Suite 1600 | New York, NY 10007 >> > 1-888-349-3593 >> > http://www.fixflyer.com >> > >> > NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED >> > RECIPIENT(S) >> > OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS >> > PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, >> > DISTRIBUTION, >> > OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS INFORMATION >> > IS >> > RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED RECIPIENT, >> > PLEASE >> > CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS E-MAIL FROM >> > YOUR >> > SYSTEM AND DESTROY ANY COPIES. >> > >> > _______________________________________________ >> > Users mailing list >> > Users(a)ovirt.org >> > http://lists.ovirt.org/mailman/listinfo/users >> > > > > > > -- > > Charles Kozler > Vice President, IT Operations > > FIX Flyer, LLC > 225 Broadway | Suite 1600 | New York, NY 10007 > 1-888-349-3593 > http://www.fixflyer.com > > NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED RECIPIENT(S) > OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS > PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, DISTRIBUTION, > OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS INFORMATION IS > RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED RECIPIENT, PLEASE > CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS E-MAIL FROM YOUR > SYSTEM AND DESTROY ANY COPIES.