[Engine-devel] Shared Memory

[Amador Pahim]

Hello,

Not sure if it was already discussed, but I'd like to talk about oVirt 
"Shared Memory".

Webadmin has the Shared Memory percent information in host general tab 
[1]. Initially, I thought that Shared Memory was the KSM de-duplication 
results. But comparing with my KSM stats, it does not make sense. My env:

     3,5 GB virt-host.
     6 identical VMs running with 1GB RAM each.

     Webadmin host details:
         Memory Sharing: Active
         Shared Memory: 0%

     KSM - How many shared pages are being used:

         $ cat /sys/kernel/mm/ksm/pages_sharing
         109056

     KSM - How many more sites are sharing them i.e. how much saved

         $ cat /sys/kernel/mm/ksm/pages_sharing
         560128

     Converting to Mbytes:

         $ echo $(( (109056 * $(getconf PAGE_SIZE)) / (1024 * 1024) ))
         426

         $ echo $(( ( 560128 * $(getconf PAGE_SIZE)) / (1024 * 1024) ))
         2188

With those KSM results, I could expect something but 0 in "Shared Memory".

Tracing the origin of "Shared Memory" in oVirt code, I realized it's 
coming from memShared value (from getVdsStats vdsm command),  which is 
provided in Mbytes:

     $ vdsClient -s 192.168.10.250 getVdsStats | grep memShared
             memShared = 9

Finding memShared function in vdsm, we have:

$VDSM_ROOT/vdsm/API.py

...

stats['memShared'] = self._memShared() / Mbytes

...

def _memShared(self):
     """
     Return an approximation of memory shared by VMs thanks to KSM.
     """
     shared = 0
     for v in self._cif.vmContainer.values():
         if v.conf['pid'] == '0':
             continue
         try:
             statmfile = file('/proc/' + v.conf['pid'] + '/statm')
             shared += int(statmfile.read().split()[2]) * PAGE_SIZE_BYTES
         except:
             pass
     return shared

...

memShared is calculated adding the shared pages value (3rd field) from 
/proc/<VM_PID>/statm file from all running VMs, converting to bytes 
through PAGE_SIZE value and transforming to Mbytes at the end. This 
field in statm file currently is (it changed along kernel history) the 
count of pages instantiated in the process address space which are 
shared with a file, including executable, library or shared memory. 
Despite vdsm code comment, KSM shared pages are not accounted here. KSM 
works de-duplicating and sharing memory pages without process awareness.

Engine is calculating the percent considering total physical memory - 
memSize value from getVdsCapabilities vdsm command:

     $ vdsClient -s 192.168.10.250 getVdsCapabilities | grep memSize
         memSize = 3574

Calculating the percent:

     $ echo "scale=2; 9 * 100 / 3574" | bc
     .25

So, we have arround 0,25%, rounded to 0%. "Shared Memory" is coherent, 
but not reflecting the real page deduplication benefits. And unnoticed 
administrators - me included - are led to think that Shared Memory is 
related with KSM results.

IMO, memShared is not providing any representative information. On the 
other hand, the missing KSM results are really important to oVirt 
administrators, providing information about how much memory they are 
over committed by, for capacity management, and how much money oVirt is 
saving in memory.

In order to offer those KSM stats to engine, I sent a patch [2] (waiting 
approval) adding "ksmShared" and "ksmSharing" values to vdsm getVdsStats 
command in a standard way, with key names that fits with existing KSM 
ones (ksmCpu, ksmPages and ksmState).

Before patch:
     $ vdsClient -s 192.168.10.250 getVdsStats | grep ksm
         ksmCpu = 1          -> the ksmd process cpu load
         ksmPages = 664      -> pages to scan before ksmd goes to sleep
         ksmState = True     -> is ksm running?

With the patch:
     $ vdsClient -s 192.168.10.250 getVdsStats | grep ksm
         ksmCpu = 1
         ksmPages = 664
         ksmShared = 426     -> how many Mbytes of memory are being shared
         ksmSharing = 2188   -> how many more sites are sharing them 
i.e. how much Mbytes are being saved
         ksmState = True


Finally my questions:

1 - Is sharedMem value (from /proc/PID/statm) that significant to be 
kept in host details? If yes, why?
2 - What about - and how (%, Mb, ...) - to add the vdsm new ksmShared 
and ksmSharing stats in host details?

Sorry about my long history. I look forward to hearing your comments.

All the best,
--
Pahim

[1] - 
http://www.pahim.org/wp-content/uploads/2012/05/Screenshot-oVirt-Enterprise-Virtualization-Engine-Web-Administration-Mozilla-Firefox-4.png
[2] - http://gerrit.ovirt.org/4755