<div dir="ltr">Hi Nir -<div><br></div><div>Thanks for getting back to me. Will the patch to 3.6 be backported to 3.5? As you can tell from the images, it takes days and days for it to increase over time. I also wasnt sure if that was the right bug because VDSM memory shows normal from top ...</div><div><br></div><div><div> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND </div><div> 5134 vdsm 0 -20 4368m 111m 10m S 2.0 0.3 3709:28 vdsm </div></div><div><br></div><div>Res is only 111M. This is from node1 which is showing currently 20GB of 32GB used with only 2 VMs running on it - 1 with 4G and another with ~1 GB of RAM configured</div><div><br></div><div>The images are from nagios and the value here is a direct correlation to what you would see in the free command output. See below from an example of node 1 and node 2</div><div><br></div><div><div>[compute[root@node1 ~]$ free </div><div> total used free shared buffers cached</div><div>Mem: 32765316 20318156 12447160 252 30884 628948</div><div>-/+ buffers/cache: 19658324 13106992 </div><div>Swap: 19247100 0 19247100 </div><div>[compute[root@node1 ~]$ free -m</div><div> total used free shared buffers cached</div><div>Mem: 31997 19843 12153 0 30 614</div><div>-/+ buffers/cache: 19199 12798 </div><div>Swap: 18795 0 18795 </div></div><div><br></div><div>And its correlated image <a href="http://i.imgur.com/PZLEgyx.png">http://i.imgur.com/PZLEgyx.png</a> (~19GB used)</div><div><br></div><div>And as a control, node 2 that I just restarted today </div><div><br></div><div><div>[compute[root@node2 ~]$ free</div><div> total used free shared buffers cached</div><div>Mem: 32765316 1815324 30949992 212 35784 717320</div><div>-/+ buffers/cache: 1062220 31703096 </div><div>Swap: 19247100 0 19247100 </div><div>[compute[root@node2 ~]$ free -m</div><div> total used free shared buffers cached</div><div>Mem: 31997 1772 30225 0 34 700</div><div>-/+ buffers/cache: 1036 30960 </div><div>Swap: 18795 0 18795 </div></div><div><br></div><div>And its correlated image <a href="http://i.imgur.com/8ldPVqY.png">http://i.imgur.com/8ldPVqY.png</a> (~2GB used). Note how 1772 in the image is exactly what is registered under 'used' in free command</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 22, 2016 at 3:59 PM, Nir Soffer <span dir="ltr"><<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Fri, Jan 22, 2016 at 9:25 PM, Charles Kozler <<a href="mailto:charles@fixflyer.com">charles@fixflyer.com</a>> wrote:<br>
> Here is a screenshot of my three nodes and their increased memory usage over<br>
> 30 days. Note that node #2 had 1 single VM that had 4GB of RAM assigned to<br>
> it. I had since shut it down and saw no memory reclamation occur. Further, I<br>
> flushed page caches and inodes and ran 'sync'. I tried everything but<br>
> nothing brought the memory usage down. vdsm was low too (couple hundred MB)<br>
<br>
</span>Note that there is an old leak in vdsm, will be fixed in next 3.6 build:<br>
<a href="https://bugzilla.redhat.com/1269424" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/1269424</a><br>
<span class=""><br>
> and there was no qemu-kvm process running so I'm at a loss<br>
><br>
> <a href="http://imgur.com/a/aFPcK" rel="noreferrer" target="_blank">http://imgur.com/a/aFPcK</a><br>
><br>
> Please advise on what I can do to debug this. Note I have restarted node 2<br>
> (which is why you see the drop) to see if it raises in memory use over tim<br>
> even with no VM's running<br>
<br>
</span>Not sure what is "memory" that you show in the graphs. Theoretically this may be<br>
normal memory usage, Linux using free memory for the buffer cache.<br>
<br>
Can you instead show the output of "free", during one day, maybe run once<br>
per hour?<br>
<br>
You may also like to install sysstat for collecting and monitoring<br>
resources usage.<br>
<span class=""><br>
><br>
> [compute[root@node2 log]$ rpm -qa | grep -i ovirt<br>
> libgovirt-0.3.2-1.el6.x86_64<br>
> ovirt-release35-006-1.noarch<br>
> ovirt-hosted-engine-ha-1.2.8-1.el6.noarch<br>
> ovirt-hosted-engine-setup-1.2.6.1-1.el6.noarch<br>
> ovirt-engine-sdk-python-3.5.6.0-1.el6.noarch<br>
> ovirt-host-deploy-1.3.2-1.el6.noarch<br>
><br>
><br>
> --<br>
><br>
> Charles Kozler<br>
> Vice President, IT Operations<br>
><br>
> FIX Flyer, LLC<br>
> 225 Broadway | Suite 1600 | New York, NY 10007<br>
> 1-888-349-3593<br>
> <a href="http://www.fixflyer.com" rel="noreferrer" target="_blank">http://www.fixflyer.com</a><br>
><br>
> NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED RECIPIENT(S)<br>
> OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS<br>
> PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, DISTRIBUTION,<br>
> OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS INFORMATION IS<br>
> RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED RECIPIENT, PLEASE<br>
> CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS E-MAIL FROM YOUR<br>
> SYSTEM AND DESTROY ANY COPIES.<br>
><br>
</span>> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><br style="color:rgb(0,0,0);font-family:'Times New Roman';font-size:medium"><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><strong><span style="font-family:'times new roman',times,serif"><font size="2">Charles Kozler</font></span></strong></div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><em><span style="font-family:'times new roman',times,serif"><font size="2">Vice President, IT Operations</font></span></em></div><div style="color:rgb(0,0,0);font-family:'Times New Roman';font-size:medium"> </div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><span style="font-family:'times new roman',times,serif"><font size="2">FIX Flyer, LLC</font></span></div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><span style="font-family:'times new roman',times,serif"><font size="2">225 Broadway | Suite 1600 | New York, NY 10007</font></span></div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><span style="font-family:'times new roman',times,serif"><font size="2">1-888-349-3593</font></span></div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><a href="http://fixflyer.com" target="_blank"><span style="font-family:'times new roman',times,serif"><font size="2">http://www.fixflyer.com</font></span></a></div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><font size="2"> </font></div><div style="color:rgb(0,0,0);font-family:'Times New Roman'"><font size="1">NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.</font></div></div></div></div></div></div></div></div>
</div>