[Engine-devel] Shared Memory

[Dan Kenigsberg]

On Wed, May 30, 2012 at 04:39:42PM -0300, Amador Pahim wrote:
> Hello,
> 
> Not sure if it was already discussed, but I'd like to talk about
> oVirt "Shared Memory".
> 
> Webadmin has the Shared Memory percent information in host general
> tab [1]. Initially, I thought that Shared Memory was the KSM
> de-duplication results. But comparing with my KSM stats, it does not
> make sense. My env:
> 
>     3,5 GB virt-host.
>     6 identical VMs running with 1GB RAM each.
> 
>     Webadmin host details:
>         Memory Sharing: Active
>         Shared Memory: 0%
> 
>     KSM - How many shared pages are being used:
> 
>         $ cat /sys/kernel/mm/ksm/pages_sharing
>         109056
> 
>     KSM - How many more sites are sharing them i.e. how much saved
> 
>         $ cat /sys/kernel/mm/ksm/pages_sharing
>         560128
> 
>     Converting to Mbytes:
> 
>         $ echo $(( (109056 * $(getconf PAGE_SIZE)) / (1024 * 1024) ))
>         426
> 
>         $ echo $(( ( 560128 * $(getconf PAGE_SIZE)) / (1024 * 1024) ))
>         2188
> 
> With those KSM results, I could expect something but 0 in "Shared Memory".
> 
> Tracing the origin of "Shared Memory" in oVirt code, I realized it's
> coming from memShared value (from getVdsStats vdsm command),  which
> is provided in Mbytes:
> 
>     $ vdsClient -s 192.168.10.250 getVdsStats | grep memShared
>             memShared = 9
> 
> Finding memShared function in vdsm, we have:
> 
> $VDSM_ROOT/vdsm/API.py
> 
> ...
> 
> stats['memShared'] = self._memShared() / Mbytes
> 
> ...
> 
> def _memShared(self):
>     """
>     Return an approximation of memory shared by VMs thanks to KSM.
>     """
>     shared = 0
>     for v in self._cif.vmContainer.values():
>         if v.conf['pid'] == '0':
>             continue
>         try:
>             statmfile = file('/proc/' + v.conf['pid'] + '/statm')
>             shared += int(statmfile.read().split()[2]) * PAGE_SIZE_BYTES
>         except:
>             pass
>     return shared
> 
> ...
> 
> memShared is calculated adding the shared pages value (3rd field)
> from /proc/<VM_PID>/statm file from all running VMs, converting to
> bytes through PAGE_SIZE value and transforming to Mbytes at the end.
> This field in statm file currently is (it changed along kernel
> history) the count of pages instantiated in the process address
> space which are shared with a file, including executable, library or
> shared memory. Despite vdsm code comment, KSM shared pages are not
> accounted here. KSM works de-duplicating and sharing memory pages
> without process awareness.
> 
> Engine is calculating the percent considering total physical memory
> - memSize value from getVdsCapabilities vdsm command:
> 
>     $ vdsClient -s 192.168.10.250 getVdsCapabilities | grep memSize
>         memSize = 3574
> 
> Calculating the percent:
> 
>     $ echo "scale=2; 9 * 100 / 3574" | bc
>     .25
> 
> So, we have arround 0,25%, rounded to 0%. "Shared Memory" is
> coherent, but not reflecting the real page deduplication benefits.
> And unnoticed administrators - me included - are led to think that
> Shared Memory is related with KSM results.
> 
> IMO, memShared is not providing any representative information. On
> the other hand, the missing KSM results are really important to
> oVirt administrators, providing information about how much memory
> they are over committed by, for capacity management, and how much
> money oVirt is saving in memory.
> 
> In order to offer those KSM stats to engine, I sent a patch [2]
> (waiting approval) adding "ksmShared" and "ksmSharing" values to
> vdsm getVdsStats command in a standard way, with key names that fits
> with existing KSM ones (ksmCpu, ksmPages and ksmState).
> 
> Before patch:
>     $ vdsClient -s 192.168.10.250 getVdsStats | grep ksm
>         ksmCpu = 1          -> the ksmd process cpu load
>         ksmPages = 664      -> pages to scan before ksmd goes to sleep
>         ksmState = True     -> is ksm running?
> 
> With the patch:
>     $ vdsClient -s 192.168.10.250 getVdsStats | grep ksm
>         ksmCpu = 1
>         ksmPages = 664
>         ksmShared = 426     -> how many Mbytes of memory are being shared
>         ksmSharing = 2188   -> how many more sites are sharing them
> i.e. how much Mbytes are being saved
>         ksmState = True
> 
> 
> Finally my questions:
> 
> 1 - Is sharedMem value (from /proc/PID/statm) that significant to be
> kept in host details? If yes, why?

I belive memShared used to report the Right Thing (i.e., an estimate of
memory saved by KSM) in the rhev-2.Y days. Oded, could you confirm?
Has it changed when we moved to EL6, and nobody noticed?

> 2 - What about - and how (%, Mb, ...) - to add the vdsm new
> ksmShared and ksmSharing stats in host details?

I'm not sure if ksmShared/ksmSharing have an "added value" to the the
admin, but if it does, I see no problem in reporting it from vdsm.
But then again, a meaningful memShared seems enough to me.

> 
> Sorry about my long history. I look forward to hearing your comments.

Actually the long history is useful to understand the issue, thanks for
it.