[Engine-devel] Shared Memory

[Amador Pahim]

On 05/31/2012 06:15 AM, Dan Kenigsberg wrote:
> On Wed, May 30, 2012 at 04:39:42PM -0300, Amador Pahim wrote:
>> Hello,
>>
>> Not sure if it was already discussed, but I'd like to talk about
>> oVirt "Shared Memory".
>>
>> Webadmin has the Shared Memory percent information in host general
>> tab [1]. Initially, I thought that Shared Memory was the KSM
>> de-duplication results. But comparing with my KSM stats, it does not
>> make sense. My env:
>>
>>      3,5 GB virt-host.
>>      6 identical VMs running with 1GB RAM each.
>>
>>      Webadmin host details:
>>          Memory Sharing: Active
>>          Shared Memory: 0%
>>
>>      KSM - How many shared pages are being used:
>>
>>          $ cat /sys/kernel/mm/ksm/pages_sharing
>>          109056
>>
>>      KSM - How many more sites are sharing them i.e. how much saved
>>
>>          $ cat /sys/kernel/mm/ksm/pages_sharing
>>          560128
>>
>>      Converting to Mbytes:
>>
>>          $ echo $(( (109056 * $(getconf PAGE_SIZE)) / (1024 * 1024) ))
>>          426
>>
>>          $ echo $(( ( 560128 * $(getconf PAGE_SIZE)) / (1024 * 1024) ))
>>          2188
>>
>> With those KSM results, I could expect something but 0 in "Shared Memory".
>>
>> Tracing the origin of "Shared Memory" in oVirt code, I realized it's
>> coming from memShared value (from getVdsStats vdsm command),  which
>> is provided in Mbytes:
>>
>>      $ vdsClient -s 192.168.10.250 getVdsStats | grep memShared
>>              memShared = 9
>>
>> Finding memShared function in vdsm, we have:
>>
>> $VDSM_ROOT/vdsm/API.py
>>
>> ...
>>
>> stats['memShared'] = self._memShared() / Mbytes
>>
>> ...
>>
>> def _memShared(self):
>>      """
>>      Return an approximation of memory shared by VMs thanks to KSM.
>>      """
>>      shared = 0
>>      for v in self._cif.vmContainer.values():
>>          if v.conf['pid'] == '0':
>>              continue
>>          try:
>>              statmfile = file('/proc/' + v.conf['pid'] + '/statm')
>>              shared += int(statmfile.read().split()[2]) * PAGE_SIZE_BYTES
>>          except:
>>              pass
>>      return shared
>>
>> ...
>>
>> memShared is calculated adding the shared pages value (3rd field)
>> from /proc/<VM_PID>/statm file from all running VMs, converting to
>> bytes through PAGE_SIZE value and transforming to Mbytes at the end.
>> This field in statm file currently is (it changed along kernel
>> history) the count of pages instantiated in the process address
>> space which are shared with a file, including executable, library or
>> shared memory. Despite vdsm code comment, KSM shared pages are not
>> accounted here. KSM works de-duplicating and sharing memory pages
>> without process awareness.
>>
>> Engine is calculating the percent considering total physical memory
>> - memSize value from getVdsCapabilities vdsm command:
>>
>>      $ vdsClient -s 192.168.10.250 getVdsCapabilities | grep memSize
>>          memSize = 3574
>>
>> Calculating the percent:
>>
>>      $ echo "scale=2; 9 * 100 / 3574" | bc
>>      .25
>>
>> So, we have arround 0,25%, rounded to 0%. "Shared Memory" is
>> coherent, but not reflecting the real page deduplication benefits.
>> And unnoticed administrators - me included - are led to think that
>> Shared Memory is related with KSM results.
>>
>> IMO, memShared is not providing any representative information. On
>> the other hand, the missing KSM results are really important to
>> oVirt administrators, providing information about how much memory
>> they are over committed by, for capacity management, and how much
>> money oVirt is saving in memory.
>>
>> In order to offer those KSM stats to engine, I sent a patch [2]
>> (waiting approval) adding "ksmShared" and "ksmSharing" values to
>> vdsm getVdsStats command in a standard way, with key names that fits
>> with existing KSM ones (ksmCpu, ksmPages and ksmState).
>>
>> Before patch:
>>      $ vdsClient -s 192.168.10.250 getVdsStats | grep ksm
>>          ksmCpu = 1          ->  the ksmd process cpu load
>>          ksmPages = 664      ->  pages to scan before ksmd goes to sleep
>>          ksmState = True     ->  is ksm running?
>>
>> With the patch:
>>      $ vdsClient -s 192.168.10.250 getVdsStats | grep ksm
>>          ksmCpu = 1
>>          ksmPages = 664
>>          ksmShared = 426     ->  how many Mbytes of memory are being shared
>>          ksmSharing = 2188   ->  how many more sites are sharing them
>> i.e. how much Mbytes are being saved
>>          ksmState = True
>>
>>
>> Finally my questions:
>>
>> 1 - Is sharedMem value (from /proc/PID/statm) that significant to be
>> kept in host details? If yes, why?
> I belive memShared used to report the Right Thing (i.e., an estimate of
> memory saved by KSM) in the rhev-2.Y days. Oded, could you confirm?
> Has it changed when we moved to EL6, and nobody noticed?
>
>> 2 - What about - and how (%, Mb, ...) - to add the vdsm new
>> ksmShared and ksmSharing stats in host details?
> I'm not sure if ksmShared/ksmSharing have an "added value" to the the
> admin, but if it does, I see no problem in reporting it from vdsm.
> But then again, a meaningful memShared seems enough to me.
>
>> Sorry about my long history. I look forward to hearing your comments.
> Actually the long history is useful to understand the issue, thanks for
> it.


Two more points here:

- "The KSM module shipped in this release is a different version from 
the KSM module found on the latest upstream kernel versions. Newer 
features, such as exporting statistics on the /sys filesystem, that are 
implemented upstream are not in the version shipped in this release." [3]

This should explain why /proc/<pid>/statm was used to get KSM stats.

- "[PATCH] statm: shared = rss - anon_rss" [4]

This should explain what's changed in statm.


So, I agree with you in just change memShared source.

--
Pahim

[3] - 
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/5.4_Technical_Notes/Known_Issues-kvm.html
[4] - http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.10





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/engine-devel/attachments/20120531/bc022c82/attachment.html>