Re: [ovirt-users] memory leak in 3.5.6 - not vdsm

So what about the bug that I hit for vdsm as listed above by Nir? Will I have that patch to avoid the memory leak or no? Upgrading an entire node to centos 7 is not actually feasible and was previously outlined above that I just needed to upgrade to ovirt 3.6 and no mention of OS change ...

On Feb 1, 2016 12:30 PM, "Simone Tiraboschi" <stirabos(a)redhat.com&gt; wrote: > > > On Mon, Feb 1, 2016 at 5:40 PM, Charles Kozler <charles(a)fixflyer.com&gt; > wrote: > >> Sandro / Nir - >> >> I followed your steps plus >> >> http://www.ovirt.org/OVirt_3.6_Release_Notes#Fedora_.2F_CentOS_.2F_RHEL >> >> Engine upgraded fine but then when I got to upgrading a node I did: >> >> $ yum install >> http://resources.ovirt.org/pub/yum-repo/ovirt-release36.rpm >> $ yum update -y >> >> And then rebooted the node. I noticed libvirt was updated by a .1 >> release number but vdsm (where the memory leak issue I thought was?) was >> not upgraded. In fact, very little of ovirt packages on the node were >> noticeably not updated >> >> > We are not building vdsm for el6 in 3.6, you need also to upgrade to el7 > if you want that. > > >> Updated node received the following updated packages during the install: >> >> http://pastebin.ca/3362714 >> >> Note specifically the only packages updated via the ovirt3.6 repository >> was ioprocess, otopi, ovirt-engine-sdk-python, ovirt-host-deploy, >> ovirt-release36, and python-ioprocess. I had expected to see some packages >> like vdsm and the likes updated - or was this not the case? >> >> Upgraded node: >> >> [compute[root@node02 yum.repos.d]$ rpm -qa | grep -i vdsm >> vdsm-4.16.30-0.el6.x86_64 >> vdsm-python-zombiereaper-4.16.30-0.el6.noarch >> vdsm-cli-4.16.30-0.el6.noarch >> vdsm-yajsonrpc-4.16.30-0.el6.noarch >> vdsm-jsonrpc-4.16.30-0.el6.noarch >> vdsm-xmlrpc-4.16.30-0.el6.noarch >> vdsm-python-4.16.30-0.el6.noarch >> >> Nonupgraded node >> >> [compute[root@node01 ~]$ rpm -qa | grep -i vdsm >> vdsm-cli-4.16.30-0.el6.noarch >> vdsm-jsonrpc-4.16.30-0.el6.noarch >> vdsm-python-zombiereaper-4.16.30-0.el6.noarch >> vdsm-xmlrpc-4.16.30-0.el6.noarch >> vdsm-yajsonrpc-4.16.30-0.el6.noarch >> vdsm-4.16.30-0.el6.x86_64 >> vdsm-python-4.16.30-0.el6.noarch >> >> Also, the docs stated that the engine VM would migrate to the freshly >> upgraded node since it would have a higher number but it did not >> >> So I cant really confirm whether or not my issue will be resolved? Or >> that if the node was actually updated properly? >> >> Please advise on how to confirm >> >> Thank you! >> >> On Sat, Jan 23, 2016 at 12:55 AM, Charles Kozler <charles(a)fixflyer.com&gt; >> wrote: >> >>> Thanks Sandro. Should clarify my storage is external on a redundant >>> SAN. The steps I was concerned about was the actual upgrade. I tried to >>> upgrade before and it brought my entire stack crumbling down so I'm >>> hesitant. This bug seems like a huge bug that should at least somehow >>> backported if at all possible because, to me, it renders the entire 3.5.6 >>> branch unusable as no VMs can be deployed since OOM will eventually kill >>> them. In any case that's just my opinion and I'm a new user to ovirt. The >>> docs I followed originally got me going how I need and somehow didn't work >>> for 3.6 in the same fashion so naturally I'm hesitant to upgrade but >>> clearly have no option if I want to continue my infrastructure on ovirt. >>> Thank you again for taking the time out to assist me, I truly appreciate >>> it. I will try an upgrade next week and pray it all goes well :-) >>> On Jan 23, 2016 12:40 AM, "Sandro Bonazzola" <sbonazzo(a)redhat.com&gt; >>> wrote: >>> >>>> >>>> >>>> On Fri, Jan 22, 2016 at 10:53 PM, Charles Kozler <charles(a)fixflyer.com >>>> > wrote: >>>> >>>>> Sandro - >>>>> >>>>> Do you have available documentation that can support upgrading self >>>>> hosted? I followed this >>>>> http://community.redhat.com/blog/2014/10/up-and-running-with-ovirt-3-5/ >>>>> >>>>> Would it be as easy as installing the RPM and then running yum >>>>> upgrade? >>>>> >>>>> >>>> Note that mentioned article describes an unsupported hyperconverged >>>> setup running NFS over Gluster. >>>> That said, >>>> 1) put the hosted-engine storage domain into global maintenance mode >>>> 2) upgrade the engine VM >>>> 3) select the first host to upgrade and put it under maintenance from >>>> the engine, wait for the engine vm to migrate if needed. >>>> 4) yum upgrade the first host and wait until ovirt-ha-agent completes >>>> 5) exit global and local maintenance mode >>>> 6) repeat 3-5 on all the other hosts >>>> 7) once all hosts are updated you can increase the cluster >>>> compatibility level to 3.6. At this point the engine will trigger the >>>> auto-import of the hosted-engine storage domain. >>>> >>>> Simone, Roy, can you confirm above steps? Maybe also you can update >>>> http://www.ovirt.org/Hosted_Engine_Howto#Upgrade_Hosted_Engine >>>> >>>> >>>> >>>>> Thanks >>>>> >>>>> On Fri, Jan 22, 2016 at 4:42 PM, Sandro Bonazzola < >>>>> sbonazzo(a)redhat.com&gt; wrote: >>>>> >>>>>> >>>>>> Il 22/Gen/2016 22:31, "Charles Kozler" <charles(a)fixflyer.com&gt; ha >>>>>> scritto: >>>>>> > >>>>>> > Hi Nir - >>>>>> > >>>>>> > do you have a release target date for 3.5.8? Any estimate would >>>>>> help. >>>>>> > >>>>>> >>>>>> There won't be any supported release after 3.5.6. Please update to >>>>>> 3.6.2 next week >>>>>> >>>>>> > If its not VDSM, what is it exactly? Sorry, I understood from the >>>>>> ticket it was something inside vdsm, was I mistaken? >>>>>> > >>>>>> > CentOS 6 is the servers. 6.7 to be exact >>>>>> > >>>>>> > I have done all forms of flushing that I can (page cache, inodes, >>>>>> dentry's, etc) and as well moved VM's around to other nodes and nothing >>>>>> changes the memory. How can I find the leak? Where is the leak? RES shows >>>>>> the following of which, the totals dont add up to 20GB >>>>>> > >>>>>> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>>>> COMMAND >>>>>> >>>>>> > 19044 qemu 20 0 8876m 4.0g 5680 S 3.6 12.9 1571:44 >>>>>> qemu-kvm >>>>>> >>>>>> > 26143 qemu 20 0 5094m 1.1g 5624 S 9.2 3.7 6012:12 >>>>>> qemu-kvm >>>>>> >>>>>> > 5837 root 0 -20 964m 624m 3664 S 0.0 2.0 85:22.09 >>>>>> glusterfs >>>>>> >>>>>> > 14328 root 0 -20 635m 169m 3384 S 0.0 0.5 43:15.23 >>>>>> glusterfs >>>>>> >>>>>> > 5134 vdsm 0 -20 4368m 111m 10m S 5.9 0.3 3710:50 vdsm >>>>>> >>>>>> >>>>>> > 4095 root 15 -5 727m 43m 10m S 0.0 0.1 0:02.00 >>>>>> supervdsmServer >>>>>> > >>>>>> > 4.0G + 1.1G + 624M + 169 + 111M + 43M = ~7GB >>>>>> > >>>>>> > This was top sorted by RES from highest to lowest >>>>>> > >>>>>> > At that point I wouldnt know where else to look except slab / >>>>>> kernel structures. Of which slab shows: >>>>>> > >>>>>> > [compute[root@node1 ~]$ cat /proc/meminfo | grep -i slab >>>>>> > Slab: 2549748 kB >>>>>> > >>>>>> > So roughly 2-3GB. Adding that to the other use of 7GB we have >>>>>> still about 10GB unaccounted for >>>>>> > >>>>>> > On Fri, Jan 22, 2016 at 4:24 PM, Nir Soffer <nsoffer(a)redhat.com&gt; >>>>>> wrote: >>>>>> >> >>>>>> >> On Fri, Jan 22, 2016 at 11:08 PM, Charles Kozler < >>>>>> charles(a)fixflyer.com&gt; wrote: >>>>>> >> > Hi Nir - >>>>>> >> > >>>>>> >> > Thanks for getting back to me. Will the patch to 3.6 be >>>>>> backported to 3.5? >>>>>> >> >>>>>> >> We plan to include them in 3.5.8. >>>>>> >> >>>>>> >> > As you can tell from the images, it takes days and days for it >>>>>> to increase >>>>>> >> > over time. I also wasnt sure if that was the right bug because >>>>>> VDSM memory >>>>>> >> > shows normal from top ... >>>>>> >> > >>>>>> >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>>>> COMMAND >>>>>> >> > 5134 vdsm 0 -20 4368m 111m 10m S 2.0 0.3 3709:28 >>>>>> vdsm >>>>>> >> >>>>>> >> As you wrote, this issue is not related to vdsm. >>>>>> >> >>>>>> >> > >>>>>> >> > Res is only 111M. This is from node1 which is showing currently >>>>>> 20GB of 32GB >>>>>> >> > used with only 2 VMs running on it - 1 with 4G and another with >>>>>> ~1 GB of RAM >>>>>> >> > configured >>>>>> >> > >>>>>> >> > The images are from nagios and the value here is a direct >>>>>> correlation to >>>>>> >> > what you would see in the free command output. See below from >>>>>> an example of >>>>>> >> > node 1 and node 2 >>>>>> >> > >>>>>> >> > [compute[root@node1 ~]$ free >>>>>> >> > total used free shared buffers >>>>>> cached >>>>>> >> > Mem: 32765316 20318156 12447160 252 30884 >>>>>> 628948 >>>>>> >> > -/+ buffers/cache: 19658324 13106992 >>>>>> >> > Swap: 19247100 0 19247100 >>>>>> >> > [compute[root@node1 ~]$ free -m >>>>>> >> > total used free shared buffers >>>>>> cached >>>>>> >> > Mem: 31997 19843 12153 0 30 >>>>>> 614 >>>>>> >> > -/+ buffers/cache: 19199 12798 >>>>>> >> > Swap: 18795 0 18795 >>>>>> >> > >>>>>> >> > And its correlated image http://i.imgur.com/PZLEgyx.png (~19GB >>>>>> used) >>>>>> >> > >>>>>> >> > And as a control, node 2 that I just restarted today >>>>>> >> > >>>>>> >> > [compute[root@node2 ~]$ free >>>>>> >> > total used free shared buffers >>>>>> cached >>>>>> >> > Mem: 32765316 1815324 30949992 212 35784 >>>>>> 717320 >>>>>> >> > -/+ buffers/cache: 1062220 31703096 >>>>>> >> > Swap: 19247100 0 19247100 >>>>>> >> >>>>>> >> Is this rhel/centos 6? >>>>>> >> >>>>>> >> > [compute[root@node2 ~]$ free -m >>>>>> >> > total used free shared buffers >>>>>> cached >>>>>> >> > Mem: 31997 1772 30225 0 34 >>>>>> 700 >>>>>> >> > -/+ buffers/cache: 1036 30960 >>>>>> >> > Swap: 18795 0 18795 >>>>>> >> > >>>>>> >> > And its correlated image http://i.imgur.com/8ldPVqY.png (~2GB >>>>>> used). Note >>>>>> >> > how 1772 in the image is exactly what is registered under >>>>>> 'used' in free >>>>>> >> > command >>>>>> >> >>>>>> >> I guess you should start looking at the processes running on >>>>>> these nodes. >>>>>> >> >>>>>> >> Maybe try to collect memory usage per process using ps? >>>>>> >> >>>>>> >> > >>>>>> >> > On Fri, Jan 22, 2016 at 3:59 PM, Nir Soffer <nsoffer(a)redhat.com&gt; >>>>>> wrote: >>>>>> >> >> >>>>>> >> >> On Fri, Jan 22, 2016 at 9:25 PM, Charles Kozler < >>>>>> charles(a)fixflyer.com&gt; >>>>>> >> >> wrote: >>>>>> >> >> > Here is a screenshot of my three nodes and their increased >>>>>> memory usage >>>>>> >> >> > over >>>>>> >> >> > 30 days. Note that node #2 had 1 single VM that had 4GB of >>>>>> RAM assigned >>>>>> >> >> > to >>>>>> >> >> > it. I had since shut it down and saw no memory reclamation >>>>>> occur. >>>>>> >> >> > Further, I >>>>>> >> >> > flushed page caches and inodes and ran 'sync'. I tried >>>>>> everything but >>>>>> >> >> > nothing brought the memory usage down. vdsm was low too >>>>>> (couple hundred >>>>>> >> >> > MB) >>>>>> >> >> >>>>>> >> >> Note that there is an old leak in vdsm, will be fixed in next >>>>>> 3.6 build: >>>>>> >> >> https://bugzilla.redhat.com/1269424 >>>>>> >> >> >>>>>> >> >> > and there was no qemu-kvm process running so I'm at a loss >>>>>> >> >> > >>>>>> >> >> > http://imgur.com/a/aFPcK >>>>>> >> >> > >>>>>> >> >> > Please advise on what I can do to debug this. Note I have >>>>>> restarted node >>>>>> >> >> > 2 >>>>>> >> >> > (which is why you see the drop) to see if it raises in >>>>>> memory use over >>>>>> >> >> > tim >>>>>> >> >> > even with no VM's running >>>>>> >> >> >>>>>> >> >> Not sure what is "memory" that you show in the graphs. >>>>>> Theoretically this >>>>>> >> >> may be >>>>>> >> >> normal memory usage, Linux using free memory for the buffer >>>>>> cache. >>>>>> >> >> >>>>>> >> >> Can you instead show the output of "free", during one day, >>>>>> maybe run once >>>>>> >> >> per hour? >>>>>> >> >> >>>>>> >> >> You may also like to install sysstat for collecting and >>>>>> monitoring >>>>>> >> >> resources usage. >>>>>> >> >> >>>>>> >> >> > >>>>>> >> >> > [compute[root@node2 log]$ rpm -qa | grep -i ovirt >>>>>> >> >> > libgovirt-0.3.2-1.el6.x86_64 >>>>>> >> >> > ovirt-release35-006-1.noarch >>>>>> >> >> > ovirt-hosted-engine-ha-1.2.8-1.el6.noarch >>>>>> >> >> > ovirt-hosted-engine-setup-1.2.6.1-1.el6.noarch >>>>>> >> >> > ovirt-engine-sdk-python-3.5.6.0-1.el6.noarch >>>>>> >> >> > ovirt-host-deploy-1.3.2-1.el6.noarch >>>>>> >> >> > >>>>>> >> >> > >>>>>> >> >> > -- >>>>>> >> >> > >>>>>> >> >> > Charles Kozler >>>>>> >> >> > Vice President, IT Operations >>>>>> >> >> > >>>>>> >> >> > FIX Flyer, LLC >>>>>> >> >> > 225 Broadway | Suite 1600 | New York, NY 10007 >>>>>> >> >> > 1-888-349-3593 >>>>>> >> >> > http://www.fixflyer.com >>>>>> >> >> > >>>>>> >> >> > NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE >>>>>> INTENDED >>>>>> >> >> > RECIPIENT(S) >>>>>> >> >> > OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION >>>>>> WHICH IS >>>>>> >> >> > PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, >>>>>> >> >> > DISTRIBUTION, >>>>>> >> >> > OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS >>>>>> INFORMATION >>>>>> >> >> > IS >>>>>> >> >> > RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED >>>>>> RECIPIENT, >>>>>> >> >> > PLEASE >>>>>> >> >> > CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS >>>>>> E-MAIL FROM >>>>>> >> >> > YOUR >>>>>> >> >> > SYSTEM AND DESTROY ANY COPIES. >>>>>> >> >> > >>>>>> >> >> > _______________________________________________ >>>>>> >> >> > Users mailing list >>>>>> >> >> > Users(a)ovirt.org >>>>>> >> >> > http://lists.ovirt.org/mailman/listinfo/users >>>>>> >> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > -- >>>>>> >> > >>>>>> >> > Charles Kozler >>>>>> >> > Vice President, IT Operations >>>>>> >> > >>>>>> >> > FIX Flyer, LLC >>>>>> >> > 225 Broadway | Suite 1600 | New York, NY 10007 >>>>>> >> > 1-888-349-3593 >>>>>> >> > http://www.fixflyer.com >>>>>> >> > >>>>>> >> > NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED >>>>>> RECIPIENT(S) >>>>>> >> > OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION >>>>>> WHICH IS >>>>>> >> > PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, >>>>>> DISTRIBUTION, >>>>>> >> > OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS >>>>>> INFORMATION IS >>>>>> >> > RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED >>>>>> RECIPIENT, PLEASE >>>>>> >> > CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS >>>>>> E-MAIL FROM YOUR >>>>>> >> > SYSTEM AND DESTROY ANY COPIES. >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > >>>>>> > Charles Kozler >>>>>> > Vice President, IT Operations >>>>>> > >>>>>> > FIX Flyer, LLC >>>>>> > 225 Broadway | Suite 1600 | New York, NY 10007 >>>>>> > 1-888-349-3593 >>>>>> > http://www.fixflyer.com >>>>>> > >>>>>> > NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED >>>>>> RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION >>>>>> WHICH IS PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, >>>>>> DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS >>>>>> INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED >>>>>> RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS >>>>>> E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES. >>>>>> > >>>>>> > _______________________________________________ >>>>>> > Users mailing list >>>>>> > Users(a)ovirt.org >>>>>> > http://lists.ovirt.org/mailman/listinfo/users >>>>>> > >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> *Charles Kozler* >>>>> *Vice President, IT Operations* >>>>> >>>>> FIX Flyer, LLC >>>>> 225 Broadway | Suite 1600 | New York, NY 10007 >>>>> 1-888-349-3593 >>>>> http://www.fixflyer.com <http://fixflyer.com> >>>>> >>>>> NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED >>>>> RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION >>>>> WHICH IS PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, >>>>> DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS >>>>> INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED >>>>> RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS >>>>> E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES. >>>>> >>>> >>>> >>>> >>>> -- >>>> Sandro Bonazzola >>>> Better technology. Faster innovation. Powered by community >>>> collaboration. >>>> See how it works at redhat.com >>>> >>> >> >> >> -- >> >> *Charles Kozler* >> *Vice President, IT Operations* >> >> FIX Flyer, LLC >> 225 Broadway | Suite 1600 | New York, NY 10007 >> 1-888-349-3593 >> http://www.fixflyer.com <http://fixflyer.com> >> >> NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT ONLY FOR THE INTENDED >> RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION >> WHICH IS PROPRIETARY TO FIX FLYER LLC. ANY UNAUTHORIZED USE, COPYING, >> DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS >> INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED >> RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY E-MAIL AND PLEASE DELETE THIS >> E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES. >> > >