[Engine-devel] Shared Memory
Hello, Not sure if it was already discussed, but I'd like to talk about oVirt "Shared Memory". Webadmin has the Shared Memory percent information in host general tab [1]. Initially, I thought that Shared Memory was the KSM de-duplication results. But comparing with my KSM stats, it does not make sense. My env: 3,5 GB virt-host. 6 identical VMs running with 1GB RAM each. Webadmin host details: Memory Sharing: Active Shared Memory: 0% KSM - How many shared pages are being used: $ cat /sys/kernel/mm/ksm/pages_sharing 109056 KSM - How many more sites are sharing them i.e. how much saved $ cat /sys/kernel/mm/ksm/pages_sharing 560128 Converting to Mbytes: $ echo $(( (109056 * $(getconf PAGE_SIZE)) / (1024 * 1024) )) 426 $ echo $(( ( 560128 * $(getconf PAGE_SIZE)) / (1024 * 1024) )) 2188 With those KSM results, I could expect something but 0 in "Shared Memory". Tracing the origin of "Shared Memory" in oVirt code, I realized it's coming from memShared value (from getVdsStats vdsm command), which is provided in Mbytes: $ vdsClient -s 192.168.10.250 getVdsStats | grep memShared memShared = 9 Finding memShared function in vdsm, we have: $VDSM_ROOT/vdsm/API.py ... stats['memShared'] = self._memShared() / Mbytes ... def _memShared(self): """ Return an approximation of memory shared by VMs thanks to KSM. """ shared = 0 for v in self._cif.vmContainer.values(): if v.conf['pid'] == '0': continue try: statmfile = file('/proc/' + v.conf['pid'] + '/statm') shared += int(statmfile.read().split()[2]) * PAGE_SIZE_BYTES except: pass return shared ... memShared is calculated adding the shared pages value (3rd field) from /proc/<VM_PID>/statm file from all running VMs, converting to bytes through PAGE_SIZE value and transforming to Mbytes at the end. This field in statm file currently is (it changed along kernel history) the count of pages instantiated in the process address space which are shared with a file, including executable, library or shared memory. Despite vdsm code comment, KSM shared pages are not accounted here. KSM works de-duplicating and sharing memory pages without process awareness. Engine is calculating the percent considering total physical memory - memSize value from getVdsCapabilities vdsm command: $ vdsClient -s 192.168.10.250 getVdsCapabilities | grep memSize memSize = 3574 Calculating the percent: $ echo "scale=2; 9 * 100 / 3574" | bc .25 So, we have arround 0,25%, rounded to 0%. "Shared Memory" is coherent, but not reflecting the real page deduplication benefits. And unnoticed administrators - me included - are led to think that Shared Memory is related with KSM results. IMO, memShared is not providing any representative information. On the other hand, the missing KSM results are really important to oVirt administrators, providing information about how much memory they are over committed by, for capacity management, and how much money oVirt is saving in memory. In order to offer those KSM stats to engine, I sent a patch [2] (waiting approval) adding "ksmShared" and "ksmSharing" values to vdsm getVdsStats command in a standard way, with key names that fits with existing KSM ones (ksmCpu, ksmPages and ksmState). Before patch: $ vdsClient -s 192.168.10.250 getVdsStats | grep ksm ksmCpu = 1 -> the ksmd process cpu load ksmPages = 664 -> pages to scan before ksmd goes to sleep ksmState = True -> is ksm running? With the patch: $ vdsClient -s 192.168.10.250 getVdsStats | grep ksm ksmCpu = 1 ksmPages = 664 ksmShared = 426 -> how many Mbytes of memory are being shared ksmSharing = 2188 -> how many more sites are sharing them i.e. how much Mbytes are being saved ksmState = True Finally my questions: 1 - Is sharedMem value (from /proc/PID/statm) that significant to be kept in host details? If yes, why? 2 - What about - and how (%, Mb, ...) - to add the vdsm new ksmShared and ksmSharing stats in host details? Sorry about my long history. I look forward to hearing your comments. All the best, -- Pahim [1] - http://www.pahim.org/wp-content/uploads/2012/05/Screenshot-oVirt-Enterprise-... [2] - http://gerrit.ovirt.org/4755
On Wed, May 30, 2012 at 04:39:42PM -0300, Amador Pahim wrote:
Hello,
Not sure if it was already discussed, but I'd like to talk about oVirt "Shared Memory".
Webadmin has the Shared Memory percent information in host general tab [1]. Initially, I thought that Shared Memory was the KSM de-duplication results. But comparing with my KSM stats, it does not make sense. My env:
3,5 GB virt-host. 6 identical VMs running with 1GB RAM each.
Webadmin host details: Memory Sharing: Active Shared Memory: 0%
KSM - How many shared pages are being used:
$ cat /sys/kernel/mm/ksm/pages_sharing 109056
KSM - How many more sites are sharing them i.e. how much saved
$ cat /sys/kernel/mm/ksm/pages_sharing 560128
Converting to Mbytes:
$ echo $(( (109056 * $(getconf PAGE_SIZE)) / (1024 * 1024) )) 426
$ echo $(( ( 560128 * $(getconf PAGE_SIZE)) / (1024 * 1024) )) 2188
With those KSM results, I could expect something but 0 in "Shared Memory".
Tracing the origin of "Shared Memory" in oVirt code, I realized it's coming from memShared value (from getVdsStats vdsm command), which is provided in Mbytes:
$ vdsClient -s 192.168.10.250 getVdsStats | grep memShared memShared = 9
Finding memShared function in vdsm, we have:
$VDSM_ROOT/vdsm/API.py
...
stats['memShared'] = self._memShared() / Mbytes
...
def _memShared(self): """ Return an approximation of memory shared by VMs thanks to KSM. """ shared = 0 for v in self._cif.vmContainer.values(): if v.conf['pid'] == '0': continue try: statmfile = file('/proc/' + v.conf['pid'] + '/statm') shared += int(statmfile.read().split()[2]) * PAGE_SIZE_BYTES except: pass return shared
...
memShared is calculated adding the shared pages value (3rd field) from /proc/<VM_PID>/statm file from all running VMs, converting to bytes through PAGE_SIZE value and transforming to Mbytes at the end. This field in statm file currently is (it changed along kernel history) the count of pages instantiated in the process address space which are shared with a file, including executable, library or shared memory. Despite vdsm code comment, KSM shared pages are not accounted here. KSM works de-duplicating and sharing memory pages without process awareness.
Engine is calculating the percent considering total physical memory - memSize value from getVdsCapabilities vdsm command:
$ vdsClient -s 192.168.10.250 getVdsCapabilities | grep memSize memSize = 3574
Calculating the percent:
$ echo "scale=2; 9 * 100 / 3574" | bc .25
So, we have arround 0,25%, rounded to 0%. "Shared Memory" is coherent, but not reflecting the real page deduplication benefits. And unnoticed administrators - me included - are led to think that Shared Memory is related with KSM results.
IMO, memShared is not providing any representative information. On the other hand, the missing KSM results are really important to oVirt administrators, providing information about how much memory they are over committed by, for capacity management, and how much money oVirt is saving in memory.
In order to offer those KSM stats to engine, I sent a patch [2] (waiting approval) adding "ksmShared" and "ksmSharing" values to vdsm getVdsStats command in a standard way, with key names that fits with existing KSM ones (ksmCpu, ksmPages and ksmState).
Before patch: $ vdsClient -s 192.168.10.250 getVdsStats | grep ksm ksmCpu = 1 -> the ksmd process cpu load ksmPages = 664 -> pages to scan before ksmd goes to sleep ksmState = True -> is ksm running?
With the patch: $ vdsClient -s 192.168.10.250 getVdsStats | grep ksm ksmCpu = 1 ksmPages = 664 ksmShared = 426 -> how many Mbytes of memory are being shared ksmSharing = 2188 -> how many more sites are sharing them i.e. how much Mbytes are being saved ksmState = True
Finally my questions:
1 - Is sharedMem value (from /proc/PID/statm) that significant to be kept in host details? If yes, why?
I belive memShared used to report the Right Thing (i.e., an estimate of memory saved by KSM) in the rhev-2.Y days. Oded, could you confirm? Has it changed when we moved to EL6, and nobody noticed?
2 - What about - and how (%, Mb, ...) - to add the vdsm new ksmShared and ksmSharing stats in host details?
I'm not sure if ksmShared/ksmSharing have an "added value" to the the admin, but if it does, I see no problem in reporting it from vdsm. But then again, a meaningful memShared seems enough to me.
Sorry about my long history. I look forward to hearing your comments.
Actually the long history is useful to understand the issue, thanks for it.
This is a multi-part message in MIME format. --------------000602030108050707080602 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 05/31/2012 06:15 AM, Dan Kenigsberg wrote:
On Wed, May 30, 2012 at 04:39:42PM -0300, Amador Pahim wrote:
Hello,
Not sure if it was already discussed, but I'd like to talk about oVirt "Shared Memory".
Webadmin has the Shared Memory percent information in host general tab [1]. Initially, I thought that Shared Memory was the KSM de-duplication results. But comparing with my KSM stats, it does not make sense. My env:
3,5 GB virt-host. 6 identical VMs running with 1GB RAM each.
Webadmin host details: Memory Sharing: Active Shared Memory: 0%
KSM - How many shared pages are being used:
$ cat /sys/kernel/mm/ksm/pages_sharing 109056
KSM - How many more sites are sharing them i.e. how much saved
$ cat /sys/kernel/mm/ksm/pages_sharing 560128
Converting to Mbytes:
$ echo $(( (109056 * $(getconf PAGE_SIZE)) / (1024 * 1024) )) 426
$ echo $(( ( 560128 * $(getconf PAGE_SIZE)) / (1024 * 1024) )) 2188
With those KSM results, I could expect something but 0 in "Shared Memory".
Tracing the origin of "Shared Memory" in oVirt code, I realized it's coming from memShared value (from getVdsStats vdsm command), which is provided in Mbytes:
$ vdsClient -s 192.168.10.250 getVdsStats | grep memShared memShared = 9
Finding memShared function in vdsm, we have:
$VDSM_ROOT/vdsm/API.py
...
stats['memShared'] = self._memShared() / Mbytes
...
def _memShared(self): """ Return an approximation of memory shared by VMs thanks to KSM. """ shared = 0 for v in self._cif.vmContainer.values(): if v.conf['pid'] == '0': continue try: statmfile = file('/proc/' + v.conf['pid'] + '/statm') shared += int(statmfile.read().split()[2]) * PAGE_SIZE_BYTES except: pass return shared
...
memShared is calculated adding the shared pages value (3rd field) from /proc/<VM_PID>/statm file from all running VMs, converting to bytes through PAGE_SIZE value and transforming to Mbytes at the end. This field in statm file currently is (it changed along kernel history) the count of pages instantiated in the process address space which are shared with a file, including executable, library or shared memory. Despite vdsm code comment, KSM shared pages are not accounted here. KSM works de-duplicating and sharing memory pages without process awareness.
Engine is calculating the percent considering total physical memory - memSize value from getVdsCapabilities vdsm command:
$ vdsClient -s 192.168.10.250 getVdsCapabilities | grep memSize memSize = 3574
Calculating the percent:
$ echo "scale=2; 9 * 100 / 3574" | bc .25
So, we have arround 0,25%, rounded to 0%. "Shared Memory" is coherent, but not reflecting the real page deduplication benefits. And unnoticed administrators - me included - are led to think that Shared Memory is related with KSM results.
IMO, memShared is not providing any representative information. On the other hand, the missing KSM results are really important to oVirt administrators, providing information about how much memory they are over committed by, for capacity management, and how much money oVirt is saving in memory.
In order to offer those KSM stats to engine, I sent a patch [2] (waiting approval) adding "ksmShared" and "ksmSharing" values to vdsm getVdsStats command in a standard way, with key names that fits with existing KSM ones (ksmCpu, ksmPages and ksmState).
Before patch: $ vdsClient -s 192.168.10.250 getVdsStats | grep ksm ksmCpu = 1 -> the ksmd process cpu load ksmPages = 664 -> pages to scan before ksmd goes to sleep ksmState = True -> is ksm running?
With the patch: $ vdsClient -s 192.168.10.250 getVdsStats | grep ksm ksmCpu = 1 ksmPages = 664 ksmShared = 426 -> how many Mbytes of memory are being shared ksmSharing = 2188 -> how many more sites are sharing them i.e. how much Mbytes are being saved ksmState = True
Finally my questions:
1 - Is sharedMem value (from /proc/PID/statm) that significant to be kept in host details? If yes, why? I belive memShared used to report the Right Thing (i.e., an estimate of memory saved by KSM) in the rhev-2.Y days. Oded, could you confirm? Has it changed when we moved to EL6, and nobody noticed?
2 - What about - and how (%, Mb, ...) - to add the vdsm new ksmShared and ksmSharing stats in host details? I'm not sure if ksmShared/ksmSharing have an "added value" to the the admin, but if it does, I see no problem in reporting it from vdsm. But then again, a meaningful memShared seems enough to me.
Sorry about my long history. I look forward to hearing your comments. Actually the long history is useful to understand the issue, thanks for it.
Two more points here: - "The KSM module shipped in this release is a different version from the KSM module found on the latest upstream kernel versions. Newer features, such as exporting statistics on the /sys filesystem, that are implemented upstream are not in the version shipped in this release." [3] This should explain why /proc/<pid>/statm was used to get KSM stats. - "[PATCH] statm: shared = rss - anon_rss" [4] This should explain what's changed in statm. So, I agree with you in just change memShared source. -- Pahim [3] - http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/5.4_Techni... [4] - http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.10 --------------000602030108050707080602 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> On 05/31/2012 06:15 AM, Dan Kenigsberg wrote: <blockquote cite="mid:20120531091504.GI28159@redhat.com" type="cite"> <pre wrap="">On Wed, May 30, 2012 at 04:39:42PM -0300, Amador Pahim wrote: </pre> <blockquote type="cite"> <pre wrap="">Hello, Not sure if it was already discussed, but I'd like to talk about oVirt "Shared Memory". Webadmin has the Shared Memory percent information in host general tab [1]. Initially, I thought that Shared Memory was the KSM de-duplication results. But comparing with my KSM stats, it does not make sense. My env: 3,5 GB virt-host. 6 identical VMs running with 1GB RAM each. Webadmin host details: Memory Sharing: Active Shared Memory: 0% KSM - How many shared pages are being used: $ cat /sys/kernel/mm/ksm/pages_sharing 109056 KSM - How many more sites are sharing them i.e. how much saved $ cat /sys/kernel/mm/ksm/pages_sharing 560128 Converting to Mbytes: $ echo $(( (109056 * $(getconf PAGE_SIZE)) / (1024 * 1024) )) 426 $ echo $(( ( 560128 * $(getconf PAGE_SIZE)) / (1024 * 1024) )) 2188 With those KSM results, I could expect something but 0 in "Shared Memory". Tracing the origin of "Shared Memory" in oVirt code, I realized it's coming from memShared value (from getVdsStats vdsm command), which is provided in Mbytes: $ vdsClient -s 192.168.10.250 getVdsStats | grep memShared memShared = 9 Finding memShared function in vdsm, we have: $VDSM_ROOT/vdsm/API.py ... stats['memShared'] = self._memShared() / Mbytes ... def _memShared(self): """ Return an approximation of memory shared by VMs thanks to KSM. """ shared = 0 for v in self._cif.vmContainer.values(): if v.conf['pid'] == '0': continue try: statmfile = file('/proc/' + v.conf['pid'] + '/statm') shared += int(statmfile.read().split()[2]) * PAGE_SIZE_BYTES except: pass return shared ... memShared is calculated adding the shared pages value (3rd field) from /proc/<VM_PID>/statm file from all running VMs, converting to bytes through PAGE_SIZE value and transforming to Mbytes at the end. This field in statm file currently is (it changed along kernel history) the count of pages instantiated in the process address space which are shared with a file, including executable, library or shared memory. Despite vdsm code comment, KSM shared pages are not accounted here. KSM works de-duplicating and sharing memory pages without process awareness. Engine is calculating the percent considering total physical memory - memSize value from getVdsCapabilities vdsm command: $ vdsClient -s 192.168.10.250 getVdsCapabilities | grep memSize memSize = 3574 Calculating the percent: $ echo "scale=2; 9 * 100 / 3574" | bc .25 So, we have arround 0,25%, rounded to 0%. "Shared Memory" is coherent, but not reflecting the real page deduplication benefits. And unnoticed administrators - me included - are led to think that Shared Memory is related with KSM results. IMO, memShared is not providing any representative information. On the other hand, the missing KSM results are really important to oVirt administrators, providing information about how much memory they are over committed by, for capacity management, and how much money oVirt is saving in memory. In order to offer those KSM stats to engine, I sent a patch [2] (waiting approval) adding "ksmShared" and "ksmSharing" values to vdsm getVdsStats command in a standard way, with key names that fits with existing KSM ones (ksmCpu, ksmPages and ksmState). Before patch: $ vdsClient -s 192.168.10.250 getVdsStats | grep ksm ksmCpu = 1 -> the ksmd process cpu load ksmPages = 664 -> pages to scan before ksmd goes to sleep ksmState = True -> is ksm running? With the patch: $ vdsClient -s 192.168.10.250 getVdsStats | grep ksm ksmCpu = 1 ksmPages = 664 ksmShared = 426 -> how many Mbytes of memory are being shared ksmSharing = 2188 -> how many more sites are sharing them i.e. how much Mbytes are being saved ksmState = True Finally my questions: 1 - Is sharedMem value (from /proc/PID/statm) that significant to be kept in host details? If yes, why? </pre> </blockquote> <pre wrap=""> I belive memShared used to report the Right Thing (i.e., an estimate of memory saved by KSM) in the rhev-2.Y days. Oded, could you confirm? Has it changed when we moved to EL6, and nobody noticed? </pre> <blockquote type="cite"> <pre wrap="">2 - What about - and how (%, Mb, ...) - to add the vdsm new ksmShared and ksmSharing stats in host details? </pre> </blockquote> <pre wrap=""> I'm not sure if ksmShared/ksmSharing have an "added value" to the the admin, but if it does, I see no problem in reporting it from vdsm. But then again, a meaningful memShared seems enough to me. </pre> <blockquote type="cite"> <pre wrap=""> Sorry about my long history. I look forward to hearing your comments. </pre> </blockquote> <pre wrap=""> Actually the long history is useful to understand the issue, thanks for it. </pre> </blockquote> <br> <br> Two more points here:<br> <br> - "The KSM module shipped in this release is a different version from the KSM module found on the latest upstream kernel versions. Newer features, such as exporting statistics on the /sys filesystem, that are implemented upstream are not in the version shipped in this release." [3]<br> <br> This should explain why /proc/<pid>/statm was used to get KSM stats.<br> <br> - <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> "[PATCH] statm: shared = rss - anon_rss" [4]<br> <br> This should explain what's changed in statm.<br> <br> <br> So, I agree with you in just change memShared <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> source.<br> <br> --<br> Pahim<br> <br> [3] - <a class="moz-txt-link-freetext" href="http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/5.4_Technical_Notes/Known_Issues-kvm.html">http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/5.4_Technical_Notes/Known_Issues-kvm.html</a><br> [4] - <a class="moz-txt-link-freetext" href="http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.10">http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.10</a><br> <br> <br> <br> <br> <br> </body> </html> --------------000602030108050707080602--
participants (2)
-
Amador Pahim -
Dan Kenigsberg