3:44 a.m.
--Apple-Mail=_1B8461A0-C40E-47B7-B0E6-39D36113183A
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
On 20 Apr 2017, at 17:39, Gianluca Cecchi
<gianluca.cecchi(a)gmail.com> =
wrote:
=20
On Thu, Apr 20, 2017 at 10:35 AM, Michal Skrivanek =
<mskrivan(a)redhat.com
<mailto:mskrivan@redhat.com>> wrote:
=20
> On 19 Apr 2017, at 16:28, Gianluca Cecchi <gianluca.cecchi(a)gmail.com =
<mailto:gianluca.cecchi@gmail.com>> wrote:
>=20
> On Wed, Apr 19, 2017 at 3:44 PM, Martin Polednik =
<mpolednik(a)redhat.com
<mailto:mpolednik@redhat.com>> wrote:
>=20
>=20
> If you are using recent CentOS (or I guess Fedora), there isn't any
> extra setup required. Just create the custom property:
>=20
> Both my engine and my hosts are CentOS 7.3 + updates
=20
that=E2=80=99s good
=20
> =20
>=20
> On the host where engine is running:
>=20
> $ engine-config -s "UserDefinedVMProperties=3Dhugepages=3D^.*$"
> $ service ovirt-engine restart
>=20
> and you should see 'hugepages' when editing a VM under custom =
properties.
>=20
> So no vdsm hook at all to install?
=20
today you still need the hook.
=20
>=20
> =20
> Set the number to (desired memory / 2048) and you're good to go. The
> VM will run with it's memory backed by hugepages.
>=20
> As in sysctl.conf? So that if I want 4Gb of Huge Pages I have to set =
2048?
=20
yes. there might be some=20
=20
>=20
> =20
> If you need
> hugepages even inside the VM, do whatever you would do on a physical
> host.
>=20
> mpolednik
>=20
>=20
> yes, the main subject is to have Huge Pages inside the guest, so that =
Oracle
RDBMS at startup detect them and use them
=20
yes, so if you do that via sysctl.conf on real HW just do the same =
here, or
modify kernel cmdline.
=20
Note that those are two separate things
the hook is making QEMU process use hugepages memory in the host - =
that improves
performance of any VM
then how it looks in guest is no concern to oVirt, it=E2=80=99s =
guest-side hugepages. You can enable/set them regardless the previous =
step, which may be fine if you just want to expose the capability to =
some app - e.g. in testing that the guest-side Oracle can work with =
hugepages in the guest.
But you probably want both Oracle to see hugepages and also actually
=
use them - then you need both reserve that on host for qemu process and =
then inside guest reserve that for oracle. I.e. you need to add a =
=E2=80=9Cbuffer=E2=80=9D on host side to accommodate the non-hugepages =
parts of the guest e.g. on 24GB host you can reserve 20GB hugepages for =
VMs to use, and then run a VM with 20GB memory, reserving 16GB hugepages =
inside the guest for oracle to use.
=20
Thanks,
michal
=20
>=20
> Gianluca=20
=20
=20
I'm making some tests right now.=20
Steps done:=20
- configure huge pages on hypervisor
=20
[root@ractor ~]# cat /etc/sysctl.d/huge-pages.conf=20
# 20/04/2017 8Gb
vm.nr_hugepages =3D 4096
[root@ractor ~]#=20
=20
rebooted host (I also updated in the mean time it to latest 4.1.1 =
packages with
vdsm-4.19.10.1-1.el7.centos.x86_64 and =
vdsm-hook-hugepages-4.19.10.1-1.el7.centos.noarch)
I also set "transparent_hugepage=3Dnever" boot parameter
because I =
know that they are in conflict with Huge Pages
=20
So the situation is:
=20
[root@ractor ~]# cat /proc/cmdline=20
BOOT_IMAGE=3D/vmlinuz-3.10.0-514.16.1.el7.x86_64 =
root=3D/dev/mapper/centos-root
ro rd.lvm.lv =
<http://rd.lvm.lv/>=3Dcentos/root rd.lvm.lv =
<http://rd.lvm.lv/>=3Dcentos/swap rhgb quiet LANG=3Den_US.UTF-8 =
transparent_hugepage=3Dnever
[root@ractor ~]#=20
=20
[root@ractor ~]# cat /proc/meminfo | grep -i huge
AnonHugePages: 0 kB
HugePages_Total: 4096
HugePages_Free: 4096
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
[root@ractor ~]#=20
=20
I edited a pre-existing CentOS 6 VM setting for it 8Gb of ram and 2048 =
pages
(4Gb) in custom property forhugepages.
=20
When I power on I get this addition in qemu-kvm process definition as =
expected:
=20
-mem-path /dev/hugepages/libvirt/qemu
=20
I noticed that now I have on host....
=20
[root@ractor vdsm]# cat /proc/meminfo | grep -i huge
AnonHugePages: 0 kB
HugePages_Total: 6144
HugePages_Free: 2048
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
[root@ractor vdsm]#=20
=20
So apparently it did allocated 2048 new huge pages...
Does it mean that actually I have not to pre-allocate huge pages at =
all on host
and it eventually will increase them (but not able to remove =
then I suppose) ?
=20
Anyway the count doesn't seem correct... because it seems that a total =
of
4096 pages are in use/locked... (HugePages_Total - HugePages_Free + =
HugePages_Rsvd)
while they should be 2048.....
=20
[root@ractor vdsm]# ll /dev/hugepages/libvirt/qemu/
total 0
[root@ractor vdsm]# ll /hugetlbfs/libvirt/qemu/
total 0
[root@ractor vdsm]#=20
=20
If I power off the VM
=20
[root@ractor vdsm]# cat /proc/meminfo | grep -i huge
AnonHugePages: 0 kB
HugePages_Total: 4096
HugePages_Free: 4096
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
[root@ractor vdsm]#=20
=20
Does this mean that in CentOS 7.3 Huge Pages could be reclaimed....???
it tries to=E2=80=A6well, as I said, the hook is being improved right =
now and in 4.2 it will likely be more consumable
=20
Nevertheless, when I configure huge pages in guest it seems to work as =
expected
=20
[root@dbtest ~]# cat /proc/meminfo | grep -i huge
AnonHugePages: 0 kB
HugePages_Total: 2048
HugePages_Free: 2048
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
=20
Going into Oracle DB initialization, after configuring its dedicated =
memory (SGA)
to 2354Mb, I get this confirmation inside its log file
Yes, but that would always work even without setting things up on host. =
The =E2=80=9Conly=E2=80=9D difference would be the actual performance.
=20
Thu Apr 20 17:16:27 2017
Per process system memlock (soft) limit =3D 4096M
Thu Apr 20 17:16:27 2017
Expected per process system memlock (soft) limit to lock
SHARED GLOBAL AREA (SGA) into memory: 2354M
Thu Apr 20 17:16:27 2017
Available system pagesizes:
4K, 2048K=20
Thu Apr 20 17:16:27 2017
Supported system pagesize(s):
Thu Apr 20 17:16:27 2017
PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)
Thu Apr 20 17:16:27 2017
4K Configured 3 3 =
NONE
Thu Apr 20 17:16:27 2017
2048K 2048 1177 1177 =
NONE
=20
Gianluca
=20
--Apple-Mail=_1B8461A0-C40E-47B7-B0E6-39D36113183A
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><br class=3D""><div><blockquote
type=3D"cite" class=3D""><div =
class=3D"">On 20 Apr 2017, at 17:39, Gianluca Cecchi <<a =
href=3D"mailto:gianluca.cecchi@gmail.com" =
class=3D"">gianluca.cecchi(a)gmail.com</a>&gt;
wrote:</div><br =
class=3D"Apple-interchange-newline"><div class=3D""><div
dir=3D"ltr" =
class=3D""><div class=3D"gmail_extra"><div
class=3D"gmail_quote">On Thu, =
Apr 20, 2017 at 10:35 AM, Michal Skrivanek <span dir=3D"ltr" =
class=3D""><<a href=3D"mailto:mskrivan@redhat.com"
target=3D"_blank" =
class=3D"">mskrivan(a)redhat.com</a>&gt;</span> wrote:<br =
class=3D""><blockquote class=3D"gmail_quote"
style=3D"margin:0px 0px 0px =
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div =
style=3D"word-wrap:break-word" class=3D""><br
class=3D""><div =
class=3D""><span
class=3D"gmail-m_5930415786222383046gmail-"><blockquote =
type=3D"cite" class=3D""><div class=3D"">On 19 Apr
2017, at 16:28, =
Gianluca Cecchi <<a href=3D"mailto:gianluca.cecchi@gmail.com" =
target=3D"_blank"
class=3D"">gianluca.cecchi(a)gmail.com</a>&gt; =
wrote:</div><br =
class=3D"gmail-m_5930415786222383046gmail-m_-6852186268654812432Apple-inte=
rchange-newline"><div class=3D""><div dir=3D"ltr"
class=3D""><div =
class=3D"gmail_extra"><div class=3D"gmail_quote">On Wed, Apr
19, 2017 at =
3:44 PM, Martin Polednik <span dir=3D"ltr"
class=3D""><<a =
href=3D"mailto:mpolednik@redhat.com" target=3D"_blank" =
class=3D"">mpolednik(a)redhat.com</a>&gt;</span> wrote:<br
=
class=3D""><blockquote class=3D"gmail_quote"
style=3D"margin:0px 0px 0px =
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div =
class=3D"gmail-m_5930415786222383046gmail-m_-6852186268654812432HOEnZb"><d=
iv =
class=3D"gmail-m_5930415786222383046gmail-m_-6852186268654812432h5"><block=
quote class=3D"gmail_quote" style=3D"margin:0px 0px 0px =
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br =
class=3D"">
</blockquote>
<br class=3D""></div></div>
If you are using recent CentOS (or I guess Fedora), there isn't any<br =
class=3D"">
extra setup required. Just create the custom property:<br =
class=3D""></blockquote><div class=3D""><br
class=3D""></div><div =
class=3D"">Both my engine and my hosts are CentOS 7.3 + =
updates</div></div></div></div></div></blockquote><div
class=3D""><br =
class=3D""></div></span>that=E2=80=99s good</div><div
class=3D""><span =
class=3D"gmail-m_5930415786222383046gmail-"><br
class=3D""><blockquote =
type=3D"cite" class=3D""><div class=3D""><div
dir=3D"ltr" class=3D""><div =
class=3D"gmail_extra"><div class=3D"gmail_quote"><div =
class=3D""> </div><blockquote
class=3D"gmail_quote" =
style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid =
rgb(204,204,204);padding-left:1ex">
<br class=3D"">
On the host where engine is running:<br class=3D"">
<br class=3D"">
$ engine-config -s "UserDefinedVMProperties=3Dhugep<wbr =
class=3D"">ages=3D^.*$"<br class=3D"">
$ service ovirt-engine restart<br class=3D"">
<br class=3D"">
and you should see 'hugepages' when editing a VM under custom =
properties.<br class=3D""></blockquote><div
class=3D""><br =
class=3D""></div><div class=3D"">So no vdsm hook at all
to =
install?</div></div></div></div></div></blockquote><div
class=3D""><br =
class=3D""></div></span>today you still need the
hook.</div><div =
class=3D""><span
class=3D"gmail-m_5930415786222383046gmail-"><br =
class=3D""><blockquote type=3D"cite"
class=3D""><div class=3D""><div =
dir=3D"ltr" class=3D""><div
class=3D"gmail_extra"><div =
class=3D"gmail_quote"><div class=3D""><br
class=3D""></div><div =
class=3D""> </div><blockquote
class=3D"gmail_quote" =
style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid =
rgb(204,204,204);padding-left:1ex">
Set the number to (desired memory / 2048) and you're good to go. The<br =
class=3D"">
VM will run with it's memory backed by hugepages. </blockquote><div =
class=3D""><br class=3D""></div><div
class=3D"">As in sysctl.conf? So =
that if I want 4Gb of Huge Pages I have to set =
2048?</div></div></div></div></div></blockquote><div
class=3D""><br =
class=3D""></div></span>yes. there might be
some </div><div =
class=3D""><span
class=3D"gmail-m_5930415786222383046gmail-"><br =
class=3D""><blockquote type=3D"cite"
class=3D""><div class=3D""><div =
dir=3D"ltr" class=3D""><div
class=3D"gmail_extra"><div =
class=3D"gmail_quote"><div class=3D""><br
class=3D""></div><div =
class=3D""> </div><blockquote
class=3D"gmail_quote" =
style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid =
rgb(204,204,204);padding-left:1ex">If you need<br class=3D"">
hugepages even inside the VM, do whatever you would do on a physical<br =
class=3D"">
host.<br class=3D"">
<br class=3D"">
mpolednik<div =
class=3D"gmail-m_5930415786222383046gmail-m_-6852186268654812432HOEnZb"><d=
iv
class=3D"gmail-m_5930415786222383046gmail-m_-6852186268654812432h5"><br=
class=3D""></div></div></blockquote><div
class=3D""><br =
class=3D""></div><div class=3D"">yes, the main subject
is to have Huge =
Pages inside the guest, so that Oracle RDBMS at startup detect them and =
use
them</div></div></div></div></div></blockquote><div
class=3D""><br =
class=3D""></div></span>yes, so if you do that via sysctl.conf on
real =
HW just do the same here, or modify kernel cmdline.</div><div =
class=3D""><br class=3D""></div><div
class=3D"">Note that those are two =
separate things</div><div class=3D"">the hook is making QEMU process
use =
hugepages memory in the host - that improves performance of any =
VM</div><div class=3D"">then how it looks in guest is no concern to
=
oVirt, it=E2=80=99s guest-side hugepages. You can enable/set them =
regardless the previous step, which may be fine if you just want to =
expose the capability to some app - e.g. in testing that the =
guest-side Oracle can work with hugepages in the guest.</div><div =
class=3D"">But you probably want both Oracle to see hugepages and also =
actually use them - then you need both reserve that on host for qemu =
process and then inside guest reserve that for oracle. I.e. you need to =
add a =E2=80=9Cbuffer=E2=80=9D on host side to accommodate the =
non-hugepages parts of the guest e.g. on 24GB host you can reserve 20GB =
hugepages for VMs to use, and then run a VM with 20GB memory, reserving =
16GB hugepages inside the guest for oracle to use.</div><div =
class=3D""><br class=3D""></div><div
class=3D"">Thanks,</div><div =
class=3D"">michal</div><div class=3D""><br
class=3D""><blockquote =
type=3D"cite" class=3D""><div class=3D""><div
dir=3D"ltr" class=3D""><div =
class=3D"gmail_extra"><div class=3D"gmail_quote"><div
class=3D""><br =
class=3D""></div><div
class=3D"">Gianluca </div></div></div></div>
</div></blockquote></div><br
class=3D""></div></blockquote><div =
class=3D""><br class=3D""></div><div
class=3D"">I'm making some tests =
right now. </div><div class=3D"">Steps
done: </div><div =
class=3D"">- configure huge pages on hypervisor</div><div
class=3D""><br =
class=3D""></div><div class=3D""><div
class=3D"">[root@ractor ~]# cat =
/etc/sysctl.d/huge-pages.conf </div><div class=3D"">#
20/04/2017 =
8Gb</div><div class=3D"">vm.nr_hugepages =3D 4096</div><div
=
class=3D"">[root@ractor ~]# </div></div><div
class=3D""><br =
class=3D""></div><div class=3D"">rebooted host (I also
updated in the =
mean time it to latest 4.1.1 packages =
with vdsm-4.19.10.1-1.el7.<wbr class=3D"">centos.x86_64 =
and vdsm-hook-hugepages-4.19.<wbr =
class=3D"">10.1-1.el7.centos.noarch)</div><div
class=3D"">I also set =
"transparent_hugepage=3Dnever" boot parameter because I know that they =
are in conflict with Huge Pages</div><div class=3D""><br =
class=3D""></div><div class=3D"">So the situation
is:</div><div =
class=3D""><br class=3D""></div><div
class=3D""><div =
class=3D"">[root@ractor ~]# cat /proc/cmdline </div><div =
class=3D"">BOOT_IMAGE=3D/vmlinuz-3.10.0-<wbr =
class=3D"">514.16.1.el7.x86_64 root=3D/dev/mapper/centos-root ro <a =
href=3D"http://rd.lvm.lv/" target=3D"_blank" =
class=3D"">rd.lvm.lv</a>=3Dcentos/root <a
href=3D"http://rd.lvm.lv/" =
target=3D"_blank" class=3D"">rd.lvm.lv</a>=3Dcentos/swap rhgb
quiet =
LANG=3Den_US.UTF-8 transparent_hugepage=3Dnever</div><div =
class=3D"">[root@ractor ~]# </div></div><div
class=3D""><br =
class=3D""></div><div class=3D""><div
class=3D"">[root@ractor ~]# cat =
/proc/meminfo | grep -i huge</div><div class=3D"">AnonHugePages:
=
0 kB</div><div
class=3D"">HugePages_Total: =
4096</div><div class=3D"">HugePages_Free:
=
4096</div><div class=3D"">HugePages_Rsvd:
=
0</div><div class=3D"">HugePages_Surp:
=
0</div><div class=3D"">Hugepagesize:
2048 =
kB</div><div class=3D"">[root@ractor
~]# </div></div><div =
class=3D""><br class=3D""></div><div
class=3D"">I edited a pre-existing =
CentOS 6 VM setting for it 8Gb of ram and 2048 pages (4Gb) in custom =
property forhugepages.</div><div class=3D""><br
class=3D""></div><div =
class=3D"">When I power on I get this addition in qemu-kvm process =
definition as expected:</div><div class=3D""><br
class=3D""></div><div =
class=3D"">-mem-path /dev/hugepages/libvirt/qemu<br
class=3D""></div><div =
class=3D""><br class=3D""></div><div
class=3D"">I noticed that now I =
have on host....</div><div class=3D""><br
class=3D""></div><div =
class=3D""><div class=3D"">[root@ractor vdsm]# cat
/proc/meminfo | grep =
-i huge</div><div class=3D"">AnonHugePages:
=
0 kB</div><div class=3D"">HugePages_Total:
6144</div><div =
class=3D"">HugePages_Free: 2048</div><div =
class=3D"">HugePages_Rsvd:
0</div><div =
class=3D"">HugePages_Surp:
0</div><div =
class=3D"">Hugepagesize: 2048
kB</div><div =
class=3D"">[root@ractor vdsm]# </div></div><div
class=3D""><br =
class=3D""></div><div class=3D"">So apparently it did
allocated 2048 new =
huge pages...</div></div></div><div
class=3D"gmail_extra"><div =
class=3D"">Does it mean that actually I have not to pre-allocate huge =
pages at all on host and it eventually will increase them (but not able =
to remove then I suppose) ?</div><div class=3D""><br
class=3D""></div><div=
class=3D"">Anyway the count doesn't seem correct... because it seems =
that a total of 4096 pages are in use/locked... (HugePages_Total - =
HugePages_Free + HugePages_Rsvd)</div><div class=3D"">while
they =
should be 2048.....</div><div class=3D""><br
class=3D""></div><div =
class=3D""><div class=3D"">[root@ractor vdsm]# ll =
/dev/hugepages/libvirt/qemu/</div><div class=3D"">total
0</div><div =
class=3D"">[root@ractor vdsm]# ll /hugetlbfs/libvirt/qemu/</div><div
=
class=3D"">total 0</div><div class=3D"">[root@ractor =
vdsm]# </div></div><div class=3D""><br
class=3D""></div><div =
class=3D"">If I power off the VM</div><div
class=3D""><br =
class=3D""></div><div class=3D""><div
class=3D"">[root@ractor vdsm]# cat =
/proc/meminfo | grep -i huge</div><div class=3D"">AnonHugePages:
=
0 kB</div><div
class=3D"">HugePages_Total: =
4096</div><div class=3D"">HugePages_Free:
=
4096</div><div class=3D"">HugePages_Rsvd:
=
0</div><div class=3D"">HugePages_Surp:
=
0</div><div class=3D"">Hugepagesize:
2048 =
kB</div><div class=3D"">[root@ractor
vdsm]# </div></div><div =
class=3D""><br class=3D""></div><div
class=3D"">Does this mean that in =
CentOS 7.3 Huge Pages could be =
reclaimed....???</div></div></div></div></blockquote><div><br
=
class=3D""></div>it tries to=E2=80=A6well, as I said, the hook is being
=
improved right now and in 4.2 it will likely be more =
consumable</div><div><br class=3D""><blockquote
type=3D"cite" =
class=3D""><div class=3D""><div dir=3D"ltr"
class=3D""><div =
class=3D"gmail_extra"><div class=3D""><br
class=3D""></div><div =
class=3D"">Nevertheless, when I configure huge pages in guest it seems =
to work as expected</div><div class=3D""><br
class=3D""></div><div =
class=3D""><div class=3D"">[root@dbtest ~]# cat /proc/meminfo
| grep -i =
huge</div><div class=3D"">AnonHugePages:
0 =
kB</div><div class=3D"">HugePages_Total:
2048</div><div =
class=3D"">HugePages_Free: 2048</div><div =
class=3D"">HugePages_Rsvd:
0</div><div =
class=3D"">HugePages_Surp:
0</div><div =
class=3D"">Hugepagesize: 2048
kB</div><div =
class=3D""><br class=3D""></div></div><div
class=3D"">Going into Oracle =
DB initialization, after configuring its dedicated memory (SGA) to =
2354Mb, I get this confirmation inside its log =
file</div></div></div></div></blockquote><div><br
class=3D""></div>Yes, =
but that would always work even without setting things up on host. The =
=E2=80=9Conly=E2=80=9D difference would be the actual =
performance.</div><div><br class=3D""><blockquote
type=3D"cite" =
class=3D""><div class=3D""><div dir=3D"ltr"
class=3D""><div =
class=3D"gmail_extra"><div class=3D""><br
class=3D""></div><div =
class=3D""><div class=3D"">Thu Apr 20 17:16:27
2017</div><div =
class=3D""> Per process system memlock (soft) limit =3D =
4096M</div><div class=3D"">Thu Apr 20 17:16:27
2017</div><div =
class=3D""> Expected per process system memlock (soft) limit to =
lock</div><div class=3D""> SHARED GLOBAL AREA (SGA) into
memory: =
2354M</div><div class=3D"">Thu Apr 20 17:16:27
2017</div><div =
class=3D""> Available system pagesizes:</div><div
class=3D""> =
4K, 2048K </div><div class=3D"">Thu Apr 20 17:16:27
2017</div><div =
class=3D""> Supported system pagesize(s):</div><div
class=3D"">Thu =
Apr 20 17:16:27 2017</div><div class=3D""> <b =
class=3D"">PAGESIZE</b> AVAILABLE_PAGES
EXPECTED_PAGES =
<b class=3D"">ALLOCATED_PAGES</b>
ERROR(s)</div><div =
class=3D"">Thu Apr 20 17:16:27 2017</div><div
class=3D""> =
4K Configured
=
3
=
3 NONE</div><div
class=3D"">Thu Apr 20 =
17:16:27 2017</div><div class=3D"">
<b class=3D"">2048K=
</b> 2048
=
1177
<b =
class=3D"">1177</b>
NONE</div></div><div =
class=3D""><br class=3D""></div><div
class=3D"">Gianluca</div><div =
class=3D""><br
class=3D""></div></div></div>
</div></blockquote></div><br
class=3D""></body></html>=
--Apple-Mail=_1B8461A0-C40E-47B7-B0E6-39D36113183A--