10:39 p.m.
Hello,
Not sure if it was already discussed, but I'd like to talk about oVirt
"Shared Memory".
Webadmin has the Shared Memory percent information in host general tab
[1]. Initially, I thought that Shared Memory was the KSM de-duplication
results. But comparing with my KSM stats, it does not make sense. My env:
3,5 GB virt-host.
6 identical VMs running with 1GB RAM each.
Webadmin host details:
Memory Sharing: Active
Shared Memory: 0%
KSM - How many shared pages are being used:
$ cat /sys/kernel/mm/ksm/pages_sharing
109056
KSM - How many more sites are sharing them i.e. how much saved
$ cat /sys/kernel/mm/ksm/pages_sharing
560128
Converting to Mbytes:
$ echo $(( (109056 * $(getconf PAGE_SIZE)) / (1024 * 1024) ))
426
$ echo $(( ( 560128 * $(getconf PAGE_SIZE)) / (1024 * 1024) ))
2188
With those KSM results, I could expect something but 0 in "Shared Memory".
Tracing the origin of "Shared Memory" in oVirt code, I realized it's
coming from memShared value (from getVdsStats vdsm command), which is
provided in Mbytes:
$ vdsClient -s 192.168.10.250 getVdsStats | grep memShared
memShared = 9
Finding memShared function in vdsm, we have:
$VDSM_ROOT/vdsm/API.py
...
stats['memShared'] = self._memShared() / Mbytes
...
def _memShared(self):
"""
Return an approximation of memory shared by VMs thanks to KSM.
"""
shared = 0
for v in self._cif.vmContainer.values():
if v.conf['pid'] == '0':
continue
try:
statmfile = file('/proc/' + v.conf['pid'] +
'/statm')
shared += int(statmfile.read().split()[2]) * PAGE_SIZE_BYTES
except:
pass
return shared
...
memShared is calculated adding the shared pages value (3rd field) from
/proc/<VM_PID>/statm file from all running VMs, converting to bytes
through PAGE_SIZE value and transforming to Mbytes at the end. This
field in statm file currently is (it changed along kernel history) the
count of pages instantiated in the process address space which are
shared with a file, including executable, library or shared memory.
Despite vdsm code comment, KSM shared pages are not accounted here. KSM
works de-duplicating and sharing memory pages without process awareness.
Engine is calculating the percent considering total physical memory -
memSize value from getVdsCapabilities vdsm command:
$ vdsClient -s 192.168.10.250 getVdsCapabilities | grep memSize
memSize = 3574
Calculating the percent:
$ echo "scale=2; 9 * 100 / 3574" | bc
.25
So, we have arround 0,25%, rounded to 0%. "Shared Memory" is coherent,
but not reflecting the real page deduplication benefits. And unnoticed
administrators - me included - are led to think that Shared Memory is
related with KSM results.
IMO, memShared is not providing any representative information. On the
other hand, the missing KSM results are really important to oVirt
administrators, providing information about how much memory they are
over committed by, for capacity management, and how much money oVirt is
saving in memory.
In order to offer those KSM stats to engine, I sent a patch [2] (waiting
approval) adding "ksmShared" and "ksmSharing" values to vdsm
getVdsStats
command in a standard way, with key names that fits with existing KSM
ones (ksmCpu, ksmPages and ksmState).
Before patch:
$ vdsClient -s 192.168.10.250 getVdsStats | grep ksm
ksmCpu = 1 -> the ksmd process cpu load
ksmPages = 664 -> pages to scan before ksmd goes to sleep
ksmState = True -> is ksm running?
With the patch:
$ vdsClient -s 192.168.10.250 getVdsStats | grep ksm
ksmCpu = 1
ksmPages = 664
ksmShared = 426 -> how many Mbytes of memory are being shared
ksmSharing = 2188 -> how many more sites are sharing them
i.e. how much Mbytes are being saved
ksmState = True
Finally my questions:
1 - Is sharedMem value (from /proc/PID/statm) that significant to be
kept in host details? If yes, why?
2 - What about - and how (%, Mb, ...) - to add the vdsm new ksmShared
and ksmSharing stats in host details?
Sorry about my long history. I look forward to hearing your comments.
All the best,
--
Pahim
[1] -
http://www.pahim.org/wp-content/uploads/2012/05/Screenshot-oVirt-Enterpri...
[2] - http://gerrit.ovirt.org/4755
12:15 p.m.
On Wed, May 30, 2012 at 04:39:42PM -0300, Amador Pahim wrote:
Hello,
Not sure if it was already discussed, but I'd like to talk about
oVirt "Shared Memory".
Webadmin has the Shared Memory percent information in host general
tab [1]. Initially, I thought that Shared Memory was the KSM
de-duplication results. But comparing with my KSM stats, it does not
make sense. My env:
3,5 GB virt-host.
6 identical VMs running with 1GB RAM each.
Webadmin host details:
Memory Sharing: Active
Shared Memory: 0%
KSM - How many shared pages are being used:
$ cat /sys/kernel/mm/ksm/pages_sharing
109056
KSM - How many more sites are sharing them i.e. how much saved
$ cat /sys/kernel/mm/ksm/pages_sharing
560128
Converting to Mbytes:
$ echo $(( (109056 * $(getconf PAGE_SIZE)) / (1024 * 1024) ))
426
$ echo $(( ( 560128 * $(getconf PAGE_SIZE)) / (1024 * 1024) ))
2188
With those KSM results, I could expect something but 0 in "Shared Memory".
Tracing the origin of "Shared Memory" in oVirt code, I realized it's
coming from memShared value (from getVdsStats vdsm command), which
is provided in Mbytes:
$ vdsClient -s 192.168.10.250 getVdsStats | grep memShared
memShared = 9
Finding memShared function in vdsm, we have:
$VDSM_ROOT/vdsm/API.py
...
stats['memShared'] = self._memShared() / Mbytes
...
def _memShared(self):
"""
Return an approximation of memory shared by VMs thanks to KSM.
"""
shared = 0
for v in self._cif.vmContainer.values():
if v.conf['pid'] == '0':
continue
try:
statmfile = file('/proc/' + v.conf['pid'] +
'/statm')
shared += int(statmfile.read().split()[2]) * PAGE_SIZE_BYTES
except:
pass
return shared
...
memShared is calculated adding the shared pages value (3rd field)
from /proc/<VM_PID>/statm file from all running VMs, converting to
bytes through PAGE_SIZE value and transforming to Mbytes at the end.
This field in statm file currently is (it changed along kernel
history) the count of pages instantiated in the process address
space which are shared with a file, including executable, library or
shared memory. Despite vdsm code comment, KSM shared pages are not
accounted here. KSM works de-duplicating and sharing memory pages
without process awareness.
Engine is calculating the percent considering total physical memory
- memSize value from getVdsCapabilities vdsm command:
$ vdsClient -s 192.168.10.250 getVdsCapabilities | grep memSize
memSize = 3574
Calculating the percent:
$ echo "scale=2; 9 * 100 / 3574" | bc
.25
So, we have arround 0,25%, rounded to 0%. "Shared Memory" is
coherent, but not reflecting the real page deduplication benefits.
And unnoticed administrators - me included - are led to think that
Shared Memory is related with KSM results.
IMO, memShared is not providing any representative information. On
the other hand, the missing KSM results are really important to
oVirt administrators, providing information about how much memory
they are over committed by, for capacity management, and how much
money oVirt is saving in memory.
In order to offer those KSM stats to engine, I sent a patch [2]
(waiting approval) adding "ksmShared" and "ksmSharing" values to
vdsm getVdsStats command in a standard way, with key names that fits
with existing KSM ones (ksmCpu, ksmPages and ksmState).
Before patch:
$ vdsClient -s 192.168.10.250 getVdsStats | grep ksm
ksmCpu = 1 -> the ksmd process cpu load
ksmPages = 664 -> pages to scan before ksmd goes to sleep
ksmState = True -> is ksm running?
With the patch:
$ vdsClient -s 192.168.10.250 getVdsStats | grep ksm
ksmCpu = 1
ksmPages = 664
ksmShared = 426 -> how many Mbytes of memory are being shared
ksmSharing = 2188 -> how many more sites are sharing them
i.e. how much Mbytes are being saved
ksmState = True
Finally my questions:
1 - Is sharedMem value (from /proc/PID/statm) that significant to be
kept in host details? If yes, why?
I belive memShared used to report the Right Thing (i.e., an estimate of
memory saved by KSM) in the rhev-2.Y days. Oded, could you confirm?
Has it changed when we moved to EL6, and nobody noticed?
2 - What about - and how (%, Mb, ...) - to add the vdsm new
ksmShared and ksmSharing stats in host details?
I'm not sure if ksmShared/ksmSharing have an "added value" to the the
admin, but if it does, I see no problem in reporting it from vdsm.
But then again, a meaningful memShared seems enough to me.
Sorry about my long history. I look forward to hearing your comments.
Actually the long history is useful to understand the issue, thanks for
it.
3:40 p.m.
This is a multi-part message in MIME format.
--------------000602030108050707080602
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
On 05/31/2012 06:15 AM, Dan Kenigsberg wrote:
On Wed, May 30, 2012 at 04:39:42PM -0300, Amador Pahim wrote:
> Hello,
>
> Not sure if it was already discussed, but I'd like to talk about
> oVirt "Shared Memory".
>
> Webadmin has the Shared Memory percent information in host general
> tab [1]. Initially, I thought that Shared Memory was the KSM
> de-duplication results. But comparing with my KSM stats, it does not
> make sense. My env:
>
> 3,5 GB virt-host.
> 6 identical VMs running with 1GB RAM each.
>
> Webadmin host details:
> Memory Sharing: Active
> Shared Memory: 0%
>
> KSM - How many shared pages are being used:
>
> $ cat /sys/kernel/mm/ksm/pages_sharing
> 109056
>
> KSM - How many more sites are sharing them i.e. how much saved
>
> $ cat /sys/kernel/mm/ksm/pages_sharing
> 560128
>
> Converting to Mbytes:
>
> $ echo $(( (109056 * $(getconf PAGE_SIZE)) / (1024 * 1024) ))
> 426
>
> $ echo $(( ( 560128 * $(getconf PAGE_SIZE)) / (1024 * 1024) ))
> 2188
>
> With those KSM results, I could expect something but 0 in "Shared Memory".
>
> Tracing the origin of "Shared Memory" in oVirt code, I realized it's
> coming from memShared value (from getVdsStats vdsm command), which
> is provided in Mbytes:
>
> $ vdsClient -s 192.168.10.250 getVdsStats | grep memShared
> memShared = 9
>
> Finding memShared function in vdsm, we have:
>
> $VDSM_ROOT/vdsm/API.py
>
> ...
>
> stats['memShared'] = self._memShared() / Mbytes
>
> ...
>
> def _memShared(self):
> """
> Return an approximation of memory shared by VMs thanks to KSM.
> """
> shared = 0
> for v in self._cif.vmContainer.values():
> if v.conf['pid'] == '0':
> continue
> try:
> statmfile = file('/proc/' + v.conf['pid'] +
'/statm')
> shared += int(statmfile.read().split()[2]) * PAGE_SIZE_BYTES
> except:
> pass
> return shared
>
> ...
>
> memShared is calculated adding the shared pages value (3rd field)
> from /proc/<VM_PID>/statm file from all running VMs, converting to
> bytes through PAGE_SIZE value and transforming to Mbytes at the end.
> This field in statm file currently is (it changed along kernel
> history) the count of pages instantiated in the process address
> space which are shared with a file, including executable, library or
> shared memory. Despite vdsm code comment, KSM shared pages are not
> accounted here. KSM works de-duplicating and sharing memory pages
> without process awareness.
>
> Engine is calculating the percent considering total physical memory
> - memSize value from getVdsCapabilities vdsm command:
>
> $ vdsClient -s 192.168.10.250 getVdsCapabilities | grep memSize
> memSize = 3574
>
> Calculating the percent:
>
> $ echo "scale=2; 9 * 100 / 3574" | bc
> .25
>
> So, we have arround 0,25%, rounded to 0%. "Shared Memory" is
> coherent, but not reflecting the real page deduplication benefits.
> And unnoticed administrators - me included - are led to think that
> Shared Memory is related with KSM results.
>
> IMO, memShared is not providing any representative information. On
> the other hand, the missing KSM results are really important to
> oVirt administrators, providing information about how much memory
> they are over committed by, for capacity management, and how much
> money oVirt is saving in memory.
>
> In order to offer those KSM stats to engine, I sent a patch [2]
> (waiting approval) adding "ksmShared" and "ksmSharing" values to
> vdsm getVdsStats command in a standard way, with key names that fits
> with existing KSM ones (ksmCpu, ksmPages and ksmState).
>
> Before patch:
> $ vdsClient -s 192.168.10.250 getVdsStats | grep ksm
> ksmCpu = 1 -> the ksmd process cpu load
> ksmPages = 664 -> pages to scan before ksmd goes to sleep
> ksmState = True -> is ksm running?
>
> With the patch:
> $ vdsClient -s 192.168.10.250 getVdsStats | grep ksm
> ksmCpu = 1
> ksmPages = 664
> ksmShared = 426 -> how many Mbytes of memory are being shared
> ksmSharing = 2188 -> how many more sites are sharing them
> i.e. how much Mbytes are being saved
> ksmState = True
>
>
> Finally my questions:
>
> 1 - Is sharedMem value (from /proc/PID/statm) that significant to be
> kept in host details? If yes, why?
I belive memShared used to report the Right Thing (i.e., an estimate of
memory saved by KSM) in the rhev-2.Y days. Oded, could you confirm?
Has it changed when we moved to EL6, and nobody noticed?
> 2 - What about - and how (%, Mb, ...) - to add the vdsm new
> ksmShared and ksmSharing stats in host details?
I'm not sure if ksmShared/ksmSharing have an "added value" to the the
admin, but if it does, I see no problem in reporting it from vdsm.
But then again, a meaningful memShared seems enough to me.
> Sorry about my long history. I look forward to hearing your comments.
Actually the long history is useful to understand the issue, thanks for
it.
Two more points here:
- "The KSM module shipped in this release is a different version from
the KSM module found on the latest upstream kernel versions. Newer
features, such as exporting statistics on the /sys filesystem, that are
implemented upstream are not in the version shipped in this release." [3]
This should explain why /proc/<pid>/statm was used to get KSM stats.
- "[PATCH] statm: shared = rss - anon_rss" [4]
This should explain what's changed in statm.
So, I agree with you in just change memShared source.
--
Pahim
[3] -
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/5.4_Tec...
[4] - http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.10
--------------000602030108050707080602
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
On 05/31/2012 06:15 AM, Dan Kenigsberg wrote:
<blockquote cite="mid:20120531091504.GI28159@redhat.com"
type="cite">
<pre wrap="">On Wed, May 30, 2012 at 04:39:42PM -0300, Amador Pahim
wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hello,
Not sure if it was already discussed, but I'd like to talk about
oVirt "Shared Memory".
Webadmin has the Shared Memory percent information in host general
tab [1]. Initially, I thought that Shared Memory was the KSM
de-duplication results. But comparing with my KSM stats, it does not
make sense. My env:
3,5 GB virt-host.
6 identical VMs running with 1GB RAM each.
Webadmin host details:
Memory Sharing: Active
Shared Memory: 0%
KSM - How many shared pages are being used:
$ cat /sys/kernel/mm/ksm/pages_sharing
109056
KSM - How many more sites are sharing them i.e. how much saved
$ cat /sys/kernel/mm/ksm/pages_sharing
560128
Converting to Mbytes:
$ echo $(( (109056 * $(getconf PAGE_SIZE)) / (1024 * 1024) ))
426
$ echo $(( ( 560128 * $(getconf PAGE_SIZE)) / (1024 * 1024) ))
2188
With those KSM results, I could expect something but 0 in "Shared Memory".
Tracing the origin of "Shared Memory" in oVirt code, I realized it's
coming from memShared value (from getVdsStats vdsm command), which
is provided in Mbytes:
$ vdsClient -s 192.168.10.250 getVdsStats | grep memShared
memShared = 9
Finding memShared function in vdsm, we have:
$VDSM_ROOT/vdsm/API.py
...
stats['memShared'] = self._memShared() / Mbytes
...
def _memShared(self):
"""
Return an approximation of memory shared by VMs thanks to KSM.
"""
shared = 0
for v in self._cif.vmContainer.values():
if v.conf['pid'] == '0':
continue
try:
statmfile = file('/proc/' + v.conf['pid'] + '/statm')
shared += int(statmfile.read().split()[2]) * PAGE_SIZE_BYTES
except:
pass
return shared
...
memShared is calculated adding the shared pages value (3rd field)
from /proc/<VM_PID>/statm file from all running VMs, converting to
bytes through PAGE_SIZE value and transforming to Mbytes at the end.
This field in statm file currently is (it changed along kernel
history) the count of pages instantiated in the process address
space which are shared with a file, including executable, library or
shared memory. Despite vdsm code comment, KSM shared pages are not
accounted here. KSM works de-duplicating and sharing memory pages
without process awareness.
Engine is calculating the percent considering total physical memory
- memSize value from getVdsCapabilities vdsm command:
$ vdsClient -s 192.168.10.250 getVdsCapabilities | grep memSize
memSize = 3574
Calculating the percent:
$ echo "scale=2; 9 * 100 / 3574" | bc
.25
So, we have arround 0,25%, rounded to 0%. "Shared Memory" is
coherent, but not reflecting the real page deduplication benefits.
And unnoticed administrators - me included - are led to think that
Shared Memory is related with KSM results.
IMO, memShared is not providing any representative information. On
the other hand, the missing KSM results are really important to
oVirt administrators, providing information about how much memory
they are over committed by, for capacity management, and how much
money oVirt is saving in memory.
In order to offer those KSM stats to engine, I sent a patch [2]
(waiting approval) adding "ksmShared" and "ksmSharing" values to
vdsm getVdsStats command in a standard way, with key names that fits
with existing KSM ones (ksmCpu, ksmPages and ksmState).
Before patch:
$ vdsClient -s 192.168.10.250 getVdsStats | grep ksm
ksmCpu = 1 -> the ksmd process cpu load
ksmPages = 664 -> pages to scan before ksmd goes to sleep
ksmState = True -> is ksm running?
With the patch:
$ vdsClient -s 192.168.10.250 getVdsStats | grep ksm
ksmCpu = 1
ksmPages = 664
ksmShared = 426 -> how many Mbytes of memory are being shared
ksmSharing = 2188 -> how many more sites are sharing them
i.e. how much Mbytes are being saved
ksmState = True
Finally my questions:
1 - Is sharedMem value (from /proc/PID/statm) that significant to be
kept in host details? If yes, why?
</pre>
</blockquote>
<pre wrap="">
I belive memShared used to report the Right Thing (i.e., an estimate of
memory saved by KSM) in the rhev-2.Y days. Oded, could you confirm?
Has it changed when we moved to EL6, and nobody noticed?
</pre>
<blockquote type="cite">
<pre wrap="">2 - What about - and how (%, Mb, ...) - to add the
vdsm new
ksmShared and ksmSharing stats in host details?
</pre>
</blockquote>
<pre wrap="">
I'm not sure if ksmShared/ksmSharing have an "added value" to the the
admin, but if it does, I see no problem in reporting it from vdsm.
But then again, a meaningful memShared seems enough to me.
</pre>
<blockquote type="cite">
<pre wrap="">
Sorry about my long history. I look forward to hearing your comments.
</pre>
</blockquote>
<pre wrap="">
Actually the long history is useful to understand the issue, thanks for
it.
</pre>
</blockquote>
<br>
<br>
Two more points here:<br>
<br>
- "The KSM module shipped in this release is a different version
from the KSM module found on the latest upstream kernel versions.
Newer features, such as exporting statistics on the /sys filesystem,
that are implemented upstream are not in the version shipped in this
release." [3]<br>
<br>
This should explain why /proc/<pid>/statm was used to get KSM
stats.<br>
<br>
-
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
"[PATCH] statm: shared = rss - anon_rss" [4]<br>
<br>
This should explain what's changed in statm.<br>
<br>
<br>
So, I agree with you in just change memShared
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
source.<br>
<br>
--<br>
Pahim<br>
<br>
[3] -
<a class="moz-txt-link-freetext"
href="http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/h...
[4] - <a class="moz-txt-link-freetext"
href="http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.10&q...
<br>
<br>
<br>
<br>
<br>
</body>
</html>
--------------000602030108050707080602--
4557
days inactive
4558
days old
2 comments
2 participants
participants (2)
-
Amador Pahim -
Dan Kenigsberg