8:06 a.m.
Screenshots attached
We have 2 hosts and 24 running vms, all vms are pretty small.
As you can see, top and the dashboard does not agree.
An interesting thing is that if I view the host itself in the engine,
it says under "General" that
Max free Memory for scheduling new VMs: 413360 MB
So maybe it's some sort of caching that's using the memory.
/tony
On Wed, 2018-12-12 at 09:59 -0600, Darrell Budic wrote:
Yeah, you’re right about 400G, I dropped a digit reading it out of
your top display.
So what are you seeing in the dashboard, I’m not sure I understand
the disconnect between the top you shared and what you’re seeing
there. It shows lots more than 110G in use, I gather? Or are you
seeing this on the hosts page per host mem use?
> On Dec 12, 2018, at 12:34 AM, Tony Brian Albers <tba(a)kb.dk> wrote:
>
> I'm not following you on the 42G available, the way I see it
> there's
> 400+G available:
>
> [root@man-001 ~]# free -h
> total used free shared buff/cache
> a
> vailable
> Mem: 503G 96G 19G 205M 387G
>
> 405G
> Swap: 4.0G 520K 4.0G
>
> And here's top sorted by %mem usage:
>
> top - 07:29:00 up 104 days, 20:56, 1 user, load average: 0.59,
> 0.68,
> 0.67
> Tasks: 564 total, 1 running, 563 sleeping, 0 stopped, 0
> zombie
> %Cpu(s): 1.5 us, 0.2 sy, 0.0 ni, 98.2 id, 0.0 wa, 0.0 hi, 0.0
> si, 0.0 st
> KiB Mem : 52807689+total, 20085144 free, 10132981+used,
> 40666195+buff/cache
> KiB Swap: 4194300 total, 4193780 free, 520 used.
> 42491062+avail
> Mem
>
> PID USER PR NI VIRT RES SHR S %CPU
> %MEM TIME+
> COMMAND
> 5517 qemu 20 0 9128268 8.1g 14084
> S 3.0 1.6 5892:07
> qemu-kvm
> 14187 qemu 20 0 9210236 8.1g 14072
> S 5.3 1.6 6586:00
> qemu-kvm
> 12791 qemu 20 0 9272448 8.1g 14140
> S 14.2 1.6 17452:10
> qemu-kvm
> 135526 qemu 20 0 9117748 8.1g 13664
> S 2.3 1.6 5874:48
> qemu-kvm
> 7938 qemu 20 0 9129936 8.1g 13744
> S 2.3 1.6 22109:28
> qemu-kvm
> 11764 qemu 20 0 9275520 8.1g 13720
> S 3.3 1.6 10679:25
> qemu-kvm
> 12066 qemu 20 0 9360552 8.1g 13708
> S 3.0 1.6 10724:34
> qemu-kvm
> 11153 qemu 20 0 9113544 8.1g 13700
> S 15.6 1.6 19050:12
> qemu-kvm
> 12436 qemu 20 0 9161800 8.1g 13712
> S 16.2 1.6 21268:00
> qemu-kvm
> 6902 qemu 20 0 9110480 8.0g 13580
> S 0.7 1.6 1804:16
> qemu-kvm
> 7621 qemu 20 0 9203816 4.8g 14264
> S 1.7 1.0 3143:35
> qemu-kvm
> 6587 qemu 20 0 4880980 4.1g 13744
> S 0.7 0.8 2354:56
> qemu-kvm
> 7249 qemu 20 0 4913084 1.6g 13712
> S 0.7 0.3 1380:38
> qemu-kvm
> 111877 qemu 20 0 1911088 1.1g 14076 S 0.3 0.2
> 419:58.70
> qemu-kvm
> 4602 vdsm 0 -20 4803160 114184 13860
> S 1.3 0.0 2143:44
> vdsmd
> 4058 root 15 -5 1154020 38804 9588
> S 0.0 0.0 0:00.81
> supervdsmd
> 818 root 20 0 84576 35356 34940
> S 0.0 0.0 1:05.60
> systemd-journal
> 3602 root 20 0 1496796 32536 9232 S 0.0 0.0
> 123:53.70
> python
> 2672 root 20 0 358328 30228 7984
> S 0.0 0.0 0:14.76
> firewalld
> 4801 vdsm 20 0 1640996 28904 5484
> S 0.0 0.0 1265:14
> python
>
>
> Rebooting a host doesn't help, (I've tried that earlier) the only
> thing
> that works is to stop all vm's, reboot all hosts at the same time
> and
> start vm's again. Then memory usage shown in the dashboard slowly
> increases over time again.
>
> /tony
>
>
>
>
>
>
> On Tue, 2018-12-11 at 14:09 -0600, Darrell Budic wrote:
> > That’s only reporting 42G available of your 512, ok but something
> > still using it. Try sorting the top by memory %, should be ‘>’
> > while
> > it’s running.
> >
> > > On Dec 11, 2018, at 1:39 AM, Tony Brian Albers <tba(a)kb.dk>
> > > wrote:
> > >
> > > Looks ok to me:
> > >
> > > top - 08:38:07 up 103 days, 22:05, 1 user, load average:
> > > 0.68,
> > > 0.62,
> > > 0.57
> > > Tasks: 565 total, 1 running, 564 sleeping, 0 stopped, 0
> > > zombie
> > > %Cpu(s): 1.0 us, 0.5 sy, 0.0 ni, 98.5 id, 0.0 wa, 0.0
> > > hi, 0.0
> > > si, 0.0 st
> > > KiB Mem : 52807689+total, 22355988 free, 10132873+used,
> > > 40439219+buff/cache
> > > KiB Swap: 4194300 total, 4193780 free, 520 used.
> > > 42492028+avail
> > > Mem
> > >
> > > PID USER PR NI VIRT RES SHR S %CPU
> > > %MEM TIME+
> > > COMMAND
> > > 14187 qemu 20 0 9144668 8.1g 14072
> > > S 12.6 1.6 6506:46
> > > qemu-kvm
> > > 11153 qemu 20 0 9244680 8.1g 13700
> > > S 4.3 1.6 18881:11
> > > qemu-kvm
> > > 12436 qemu 20 0 9292936 8.1g 13712
> > > S 3.3 1.6 21071:56
> > > qemu-kvm
> > > 5517 qemu 20 0 9128268 8.1g 14084
> > > S 3.0 1.6 5801:03
> > > qemu-kvm
> > > 11764 qemu 20 0 9185364 8.1g 13720
> > > S 3.0 1.6 10585:14
> > > qemu-kvm
> > > 7938 qemu 20 0 9252876 8.1g 13744
> > > S 2.6 1.6 21912:46
> > > qemu-kvm
> > > 12791 qemu 20 0 9182292 8.1g 14140
> > > S 2.6 1.6 17299:36
> > > qemu-kvm
> > > 4602 vdsm 0 -20 4803160 <tel:20%204803160>
> > > 114132 13860
> > > S 2.3 0.0 2123:45
> > > vdsmd
> > > 7621 qemu 20 0 9187424 4.8g 14264
> > > S 2.3 1.0 3114:25
> > > qemu-kvm
> > > 12066 qemu 20 0 9188436 8.1g 13708
> > > S 2.3 1.6 10629:53
> > > qemu-kvm
> > > 135526 qemu 20 0 9298060 8.1g 13664
> > > S 2.0 1.6 5792:05
> > > qemu-kvm
> > > 6587 qemu 20 0 4883036 4.1g 13744
> > > S 1.3 0.8 2334:54
> > > qemu-kvm
> > > 3814 root 20 0 1450200 <tel:1450200> 25096 14208
> > > S 1.0 0.0 368:03.80
> > > libvirtd
> > > 6902 qemu 20 0 9110480 8.0g 13580
> > > S 1.0 1.6 1787:57
> > > qemu-kvm
> > > 7249 qemu 20 0 4913084 1.6g 13712
> > > S 0.7 0.3 1367:32
> > > qemu-kvm
> > >
> > >
> > > It looks like it's only in oVirt-engine that there's an issue.
> > > The
> > > host
> > > seems happy enough.
> > >
> > > /tony
> > >
> > >
> > >
> > > On Mon, 2018-12-10 at 20:14 -0600, Darrell Budic wrote:
> > > > Grab a shell on your hosts and check top memory use quick.
> > > > Could
> > > > be
> > > > VDSMD, in which case restarting the process will give you a
> > > > temp
> > > > fix.
> > > > If you’re running hyperconvered, check your gluster version,
> > > > there
> > > > was a leak in versions 3.12.7 - 3.1.12 or so, updating
> > > > ovirt/gluster
> > > > is the best fix for that.
> > > >
> > > > > On Dec 10, 2018, at 7:36 AM, Tony Brian Albers
<tba(a)kb.dk>
> > > > > wrote:
> > > > >
> > > > > Hi guys,
> > > > >
> > > > > We have a small test installation here running around 30
> > > > > vms on
> > > > > 2
> > > > > hosts.
> > > > >
> > > > > oVirt 4.2.5.3
> > > > >
> > > > > The hosts each have 512 GB memory, and the vms are sized
> > > > > with
> > > > > 4-8
> > > > > GB
> > > > > each.
> > > > >
> > > > > I have noticed that over the last months, the memory usage
> > > > > in
> > > > > the
> > > > > dashboard has been increasing and is now showing 946.8 GB
> > > > > used
> > > > > of
> > > > > 1007.2 GB.
> > > > >
> > > > > What can be causing this?
> > > > >
> > > > > TIA,
> > > > >
> > > > > --
> > > > > --
> > > > > Tony Albers
> > > > > Systems Architect
> > > > > Systems Director, National Cultural Heritage Cluster
> > > > > Royal Danish Library, Victor Albecks Vej 1, 8000 Aarhus C,
> > > > > Denmark.
> > > > > Tel: +45 2566 2383 / +45 8946 2316
> > > > > _______________________________________________
> > > > > Users mailing list -- users(a)ovirt.org
> > > > > To unsubscribe send an email to users-leave(a)ovirt.org
> > > > > Privacy Statement: https://www.ovirt.org/site/privacy-polic
> > > > > y/
> > > > > oVirt Code of Conduct: https://www.ovirt.org/community/abou
> > > > > t/co
> > > > > mmun
> > > > > ity-guidelines/
> > > > > List Archives: https://lists.ovirt.org/archives/list/users@
> > > > > ovir
> > > > > t.or
> > > > > g/message/SDDH2OC5RBOVYYCLGPOUF6HO676HWI5U/
> > >
> > > --
> > > Tony Albers
> > > Systems Architect
> > > Systems Director, National Cultural Heritage Cluster
> > > Royal Danish Library, Victor Albecks Vej 1, 8000 Aarhus C,
> > > Denmark.
> > > Tel: +45 2566 2383 <tel:+45%202566%202383> / +45 8946 2316
> > > <tel:+45%208946%202316>
>
> --
> --
> Tony Albers
> Systems Architect
> Systems Director, National Cultural Heritage Cluster
> Royal Danish Library, Victor Albecks Vej 1, 8000 Aarhus C, Denmark.
> Tel: +45 2566 2383 / +45 8946 2316
--
Tony Albers
Systems Architect
Systems Director, National Cultural Heritage Cluster
Royal Danish Library, Victor Albecks Vej 1, 8000 Aarhus C, Denmark.
Tel: +45 2566 2383 / +45 8946 2316
