10:39 a.m.
On Thu, Apr 20, 2017 at 10:35 AM, Michal Skrivanek <mskrivan(a)redhat.com>
wrote:
On 19 Apr 2017, at 16:28, Gianluca Cecchi <gianluca.cecchi(a)gmail.com>
wrote:
On Wed, Apr 19, 2017 at 3:44 PM, Martin Polednik <mpolednik(a)redhat.com>
wrote:
>
>>
> If you are using recent CentOS (or I guess Fedora), there isn't any
> extra setup required. Just create the custom property:
>
Both my engine and my hosts are CentOS 7.3 + updates
that’s good
>
> On the host where engine is running:
>
> $ engine-config -s "UserDefinedVMProperties=hugepages=^.*$"
> $ service ovirt-engine restart
>
> and you should see 'hugepages' when editing a VM under custom properties.
>
So no vdsm hook at all to install?
today you still need the hook.
> Set the number to (desired memory / 2048) and you're good to go. The
> VM will run with it's memory backed by hugepages.
As in sysctl.conf? So that if I want 4Gb of Huge Pages I have to set 2048?
yes. there might be some
> If you need
> hugepages even inside the VM, do whatever you would do on a physical
> host.
>
> mpolednik
>
>
yes, the main subject is to have Huge Pages inside the guest, so that
Oracle RDBMS at startup detect them and use them
yes, so if you do that via sysctl.conf on real HW just do the same here,
or modify kernel cmdline.
Note that those are two separate things
the hook is making QEMU process use hugepages memory in the host - that
improves performance of any VM
then how it looks in guest is no concern to oVirt, it’s guest-side
hugepages. You can enable/set them regardless the previous step, which may
be fine if you just want to expose the capability to some app - e.g. in
testing that the guest-side Oracle can work with hugepages in the guest.
But you probably want both Oracle to see hugepages and also actually use
them - then you need both reserve that on host for qemu process and then
inside guest reserve that for oracle. I.e. you need to add a “buffer” on
host side to accommodate the non-hugepages parts of the guest e.g. on 24GB
host you can reserve 20GB hugepages for VMs to use, and then run a VM with
20GB memory, reserving 16GB hugepages inside the guest for oracle to use.
Thanks,
michal
Gianluca
I'm making some tests right now.
Steps done:
- configure huge pages on hypervisor
[root@ractor ~]# cat /etc/sysctl.d/huge-pages.conf
# 20/04/2017 8Gb
vm.nr_hugepages = 4096
[root@ractor ~]#
rebooted host (I also updated in the mean time it to latest 4.1.1 packages
with vdsm-4.19.10.1-1.el7.centos.x86_64 and vdsm-hook-hugepages-4.19.
10.1-1.el7.centos.noarch)
I also set "transparent_hugepage=never" boot parameter because I know that
they are in conflict with Huge Pages
So the situation is:
[root@ractor ~]# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-514.16.1.el7.x86_64 root=/dev/mapper/centos-root
ro rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_US.UTF-8
transparent_hugepage=never
[root@ractor ~]#
[root@ractor ~]# cat /proc/meminfo | grep -i huge
AnonHugePages: 0 kB
HugePages_Total: 4096
HugePages_Free: 4096
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
[root@ractor ~]#
I edited a pre-existing CentOS 6 VM setting for it 8Gb of ram and 2048
pages (4Gb) in custom property forhugepages.
When I power on I get this addition in qemu-kvm process definition as
expected:
-mem-path /dev/hugepages/libvirt/qemu
I noticed that now I have on host....
[root@ractor vdsm]# cat /proc/meminfo | grep -i huge
AnonHugePages: 0 kB
HugePages_Total: 6144
HugePages_Free: 2048
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
[root@ractor vdsm]#
So apparently it did allocated 2048 new huge pages...
Does it mean that actually I have not to pre-allocate huge pages at all on
host and it eventually will increase them (but not able to remove then I
suppose) ?
Anyway the count doesn't seem correct... because it seems that a total of
4096 pages are in use/locked... (HugePages_Total - HugePages_Free
+ HugePages_Rsvd)
while they should be 2048.....
[root@ractor vdsm]# ll /dev/hugepages/libvirt/qemu/
total 0
[root@ractor vdsm]# ll /hugetlbfs/libvirt/qemu/
total 0
[root@ractor vdsm]#
If I power off the VM
[root@ractor vdsm]# cat /proc/meminfo | grep -i huge
AnonHugePages: 0 kB
HugePages_Total: 4096
HugePages_Free: 4096
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
[root@ractor vdsm]#
Does this mean that in CentOS 7.3 Huge Pages could be reclaimed....???
Nevertheless, when I configure huge pages in guest it seems to work as
expected
[root@dbtest ~]# cat /proc/meminfo | grep -i huge
AnonHugePages: 0 kB
HugePages_Total: 2048
HugePages_Free: 2048
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Going into Oracle DB initialization, after configuring its dedicated memory
(SGA) to 2354Mb, I get this confirmation inside its log file
Thu Apr 20 17:16:27 2017
Per process system memlock (soft) limit = 4096M
Thu Apr 20 17:16:27 2017
Expected per process system memlock (soft) limit to lock
SHARED GLOBAL AREA (SGA) into memory: 2354M
Thu Apr 20 17:16:27 2017
Available system pagesizes:
4K, 2048K
Thu Apr 20 17:16:27 2017
Supported system pagesize(s):
Thu Apr 20 17:16:27 2017
*PAGESIZE* AVAILABLE_PAGES EXPECTED_PAGES *ALLOCATED_PAGES* ERROR(s)
Thu Apr 20 17:16:27 2017
4K Configured 3 3 NONE
Thu Apr 20 17:16:27 2017
*2048K * 2048 1177 *1177* NONE
Gianluca