[ovirt-users] Re: Memory usage inclines

On Dec 12, 2018, at 12:34 AM, Tony Brian Albers <tba(a)kb.dk&gt; wrote: I'm not following you on the 42G available, the way I see it there's 400+G available: [root@man-001 ~]# free -h total used free shared buff/cache a vailable Mem: 503G 96G 19G 205M 387G 405G Swap: 4.0G 520K 4.0G And here's top sorted by %mem usage: top - 07:29:00 up 104 days, 20:56, 1 user, load average: 0.59, 0.68, 0.67 Tasks: 564 total, 1 running, 563 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.5 us, 0.2 sy, 0.0 ni, 98.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 52807689+total, 20085144 free, 10132981+used, 40666195+buff/cache KiB Swap: 4194300 total, 4193780 free, 520 used. 42491062+avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5517 qemu 20 0 9128268 8.1g 14084 S 3.0 1.6 5892:07 qemu-kvm 14187 qemu 20 0 9210236 8.1g 14072 S 5.3 1.6 6586:00 qemu-kvm 12791 qemu 20 0 9272448 8.1g 14140 S 14.2 1.6 17452:10 qemu-kvm 135526 qemu 20 0 9117748 8.1g 13664 S 2.3 1.6 5874:48 qemu-kvm 7938 qemu 20 0 9129936 8.1g 13744 S 2.3 1.6 22109:28 qemu-kvm 11764 qemu 20 0 9275520 8.1g 13720 S 3.3 1.6 10679:25 qemu-kvm 12066 qemu 20 0 9360552 8.1g 13708 S 3.0 1.6 10724:34 qemu-kvm 11153 qemu 20 0 9113544 8.1g 13700 S 15.6 1.6 19050:12 qemu-kvm 12436 qemu 20 0 9161800 8.1g 13712 S 16.2 1.6 21268:00 qemu-kvm 6902 qemu 20 0 9110480 8.0g 13580 S 0.7 1.6 1804:16 qemu-kvm 7621 qemu 20 0 9203816 4.8g 14264 S 1.7 1.0 3143:35 qemu-kvm 6587 qemu 20 0 4880980 4.1g 13744 S 0.7 0.8 2354:56 qemu-kvm 7249 qemu 20 0 4913084 1.6g 13712 S 0.7 0.3 1380:38 qemu-kvm 111877 qemu 20 0 1911088 <tel:1911088> 1.1g 14076 S 0.3 0.2 419:58.70 qemu-kvm 4602 vdsm 0 -20 4803160 <tel:20%204803160> 114184 13860 S 1.3 0.0 2143:44 vdsmd 4058 root 15 -5 1154020 <tel:1154020> 38804 9588 S 0.0 0.0 0:00.81 supervdsmd 818 root 20 0 84576 35356 34940 S 0.0 0.0 1:05.60 systemd-journal 3602 root 20 0 1496796 <tel:1496796> 32536 9232 S 0.0 0.0 123:53.70 python 2672 root 20 0 358328 30228 7984 S 0.0 0.0 0:14.76 firewalld 4801 vdsm 20 0 1640996 <tel:1640996> 28904 5484 S 0.0 0.0 1265:14 python Rebooting a host doesn't help, (I've tried that earlier) the only thing that works is to stop all vm's, reboot all hosts at the same time and start vm's again. Then memory usage shown in the dashboard slowly increases over time again. /tony On Tue, 2018-12-11 at 14:09 -0600, Darrell Budic wrote: > That’s only reporting 42G available of your 512, ok but something > still using it. Try sorting the top by memory %, should be ‘>’ while > it’s running. > >> On Dec 11, 2018, at 1:39 AM, Tony Brian Albers <tba(a)kb.dk&gt; wrote: >> >> Looks ok to me: >> >> top - 08:38:07 up 103 days, 22:05, 1 user, load average: 0.68, >> 0.62, >> 0.57 >> Tasks: 565 total, 1 running, 564 sleeping, 0 stopped, 0 >> zombie >> %Cpu(s): 1.0 us, 0.5 sy, 0.0 ni, 98.5 id, 0.0 wa, 0.0 hi, 0.0 >> si, 0.0 st >> KiB Mem : 52807689+total, 22355988 free, 10132873+used, >> 40439219+buff/cache >> KiB Swap: 4194300 total, 4193780 free, 520 used. >> 42492028+avail >> Mem >> >> PID USER PR NI VIRT RES SHR S %CPU >> %MEM TIME+ >> COMMAND >> 14187 qemu 20 0 9144668 8.1g 14072 >> S 12.6 1.6 6506:46 >> qemu-kvm >> 11153 qemu 20 0 9244680 8.1g 13700 >> S 4.3 1.6 18881:11 >> qemu-kvm >> 12436 qemu 20 0 9292936 8.1g 13712 >> S 3.3 1.6 21071:56 >> qemu-kvm >> 5517 qemu 20 0 9128268 8.1g 14084 >> S 3.0 1.6 5801:03 >> qemu-kvm >> 11764 qemu 20 0 9185364 8.1g 13720 >> S 3.0 1.6 10585:14 >> qemu-kvm >> 7938 qemu 20 0 9252876 8.1g 13744 >> S 2.6 1.6 21912:46 >> qemu-kvm >> 12791 qemu 20 0 9182292 8.1g 14140 >> S 2.6 1.6 17299:36 >> qemu-kvm >> 4602 vdsm 0 -20 4803160 <tel:20%204803160> <tel:20%204803160 <tel:20%204803160>> 114132 13860 >> S 2.3 0.0 2123:45 >> vdsmd >> 7621 qemu 20 0 9187424 4.8g 14264 >> S 2.3 1.0 3114:25 >> qemu-kvm >> 12066 qemu 20 0 9188436 8.1g 13708 >> S 2.3 1.6 10629:53 >> qemu-kvm >> 135526 qemu 20 0 9298060 8.1g 13664 >> S 2.0 1.6 5792:05 >> qemu-kvm >> 6587 qemu 20 0 4883036 4.1g 13744 >> S 1.3 0.8 2334:54 >> qemu-kvm >> 3814 root 20 0 1450200 <tel:1450200> <tel:1450200 <tel:1450200>> 25096 14208 >> S 1.0 0.0 368:03.80 >> libvirtd >> 6902 qemu 20 0 9110480 8.0g 13580 >> S 1.0 1.6 1787:57 >> qemu-kvm >> 7249 qemu 20 0 4913084 1.6g 13712 >> S 0.7 0.3 1367:32 >> qemu-kvm >> >> >> It looks like it's only in oVirt-engine that there's an issue. The >> host >> seems happy enough. >> >> /tony >> >> >> >> On Mon, 2018-12-10 at 20:14 -0600, Darrell Budic wrote: >>> Grab a shell on your hosts and check top memory use quick. Could >>> be >>> VDSMD, in which case restarting the process will give you a temp >>> fix. >>> If you’re running hyperconvered, check your gluster version, >>> there >>> was a leak in versions 3.12.7 - 3.1.12 or so, updating >>> ovirt/gluster >>> is the best fix for that. >>> >>>> On Dec 10, 2018, at 7:36 AM, Tony Brian Albers <tba(a)kb.dk&gt; >>>> wrote: >>>> >>>> Hi guys, >>>> >>>> We have a small test installation here running around 30 vms on >>>> 2 >>>> hosts. >>>> >>>> oVirt 4.2.5.3 >>>> >>>> The hosts each have 512 GB memory, and the vms are sized with >>>> 4-8 >>>> GB >>>> each. >>>> >>>> I have noticed that over the last months, the memory usage in >>>> the >>>> dashboard has been increasing and is now showing 946.8 GB used >>>> of >>>> 1007.2 GB. >>>> >>>> What can be causing this? >>>> >>>> TIA, >>>> >>>> -- >>>> -- >>>> Tony Albers >>>> Systems Architect >>>> Systems Director, National Cultural Heritage Cluster >>>> Royal Danish Library, Victor Albecks Vej 1, 8000 Aarhus C, >>>> Denmark. >>>> Tel: +45 2566 2383 / +45 8946 2316 >>>> _______________________________________________ >>>> Users mailing list -- users(a)ovirt.org >>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/co >>>> mmun >>>> ity-guidelines/ >>>> List Archives: https://lists.ovirt.org/archives/list/users@ovir >>>> t.or >>>> g/message/SDDH2OC5RBOVYYCLGPOUF6HO676HWI5U/ >>> >>> >> >> -- >> Tony Albers >> Systems Architect >> Systems Director, National Cultural Heritage Cluster >> Royal Danish Library, Victor Albecks Vej 1, 8000 Aarhus C, Denmark. >> Tel: +45 2566 2383 <tel:+45%202566%202383> <tel:+45%202566%202383 <tel:+45%202566%202383>> / +45 8946 2316 <tel:+45%208946%202316> >> <tel:+45%208946%202316 <tel:+45%208946%202316>> -- -- Tony Albers Systems Architect Systems Director, National Cultural Heritage Cluster Royal Danish Library, Victor Albecks Vej 1, 8000 Aarhus C, Denmark. Tel: +45 2566 2383 <tel:+45%202566%202383> / +45 8946 2316 <tel:+45%208946%202316>