On Thu, Apr 20, 2017 at 10:35 AM, Michal Skrivanek <mskrivan@redhat.com> wrote:

On 19 Apr 2017, at 16:28, Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Wed, Apr 19, 2017 at 3:44 PM, Martin Polednik <mpolednik@redhat.com> wrote:


If you are using recent CentOS (or I guess Fedora), there isn't any
extra setup required. Just create the custom property:

Both my engine and my hosts are CentOS 7.3 + updates

that’s good

 

On the host where engine is running:

$ engine-config -s "UserDefinedVMProperties=hugepages=^.*$"
$ service ovirt-engine restart

and you should see 'hugepages' when editing a VM under custom properties.

So no vdsm hook at all to install?

today you still need the hook.


 
Set the number to (desired memory / 2048) and you're good to go. The
VM will run with it's memory backed by hugepages.

As in sysctl.conf? So that if I want 4Gb of Huge Pages I have to set 2048?

yes. there might be some 


 
If you need
hugepages even inside the VM, do whatever you would do on a physical
host.

mpolednik


yes, the main subject is to have Huge Pages inside the guest, so that Oracle RDBMS at startup detect them and use them

yes, so if you do that via sysctl.conf on real HW just do the same here, or modify kernel cmdline.

Note that those are two separate things
the hook is making QEMU process use hugepages memory in the host - that improves performance of any VM
then how it looks in guest is no concern to oVirt, it’s guest-side hugepages. You can enable/set them regardless the previous step, which may be fine if you just want to expose the capability to some app  - e.g. in testing that the guest-side Oracle can work with hugepages in the guest.
But you probably want both Oracle to see hugepages and also actually use them - then you need both reserve that on host for qemu process and then inside guest reserve that for oracle. I.e. you need to add a “buffer” on host side to accommodate the non-hugepages parts of the guest e.g. on 24GB host you can reserve 20GB hugepages for VMs to use, and then run a VM with 20GB memory, reserving 16GB hugepages inside the guest for oracle to use.

Thanks,
michal


Gianluca 


I'm making some tests right now. 
Steps done: 
- configure huge pages on hypervisor

[root@ractor ~]# cat /etc/sysctl.d/huge-pages.conf 
# 20/04/2017 8Gb
vm.nr_hugepages = 4096
[root@ractor ~]# 

rebooted host (I also updated in the mean time it to latest 4.1.1 packages with vdsm-4.19.10.1-1.el7.centos.x86_64 and vdsm-hook-hugepages-4.19.10.1-1.el7.centos.noarch)
I also set "transparent_hugepage=never" boot parameter because I know that they are in conflict with Huge Pages

So the situation is:

[root@ractor ~]# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-514.16.1.el7.x86_64 root=/dev/mapper/centos-root ro rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_US.UTF-8 transparent_hugepage=never
[root@ractor ~]# 

[root@ractor ~]# cat /proc/meminfo | grep -i huge
AnonHugePages:         0 kB
HugePages_Total:    4096
HugePages_Free:     4096
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
[root@ractor ~]# 

I edited a pre-existing CentOS 6 VM setting for it 8Gb of ram and 2048 pages (4Gb) in custom property forhugepages.

When I power on I get this addition in qemu-kvm process definition as expected:

-mem-path /dev/hugepages/libvirt/qemu

I noticed that now I have on host....

[root@ractor vdsm]# cat /proc/meminfo | grep -i huge
AnonHugePages:         0 kB
HugePages_Total:    6144
HugePages_Free:     2048
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
[root@ractor vdsm]# 

So apparently it did allocated 2048 new huge pages...
Does it mean that actually I have not to pre-allocate huge pages at all on host and it eventually will increase them (but not able to remove then I suppose) ?

Anyway the count doesn't seem correct... because it seems that a total of 4096 pages are in use/locked... (HugePages_Total - HugePages_Free + HugePages_Rsvd)
while they should be 2048.....

[root@ractor vdsm]# ll /dev/hugepages/libvirt/qemu/
total 0
[root@ractor vdsm]# ll /hugetlbfs/libvirt/qemu/
total 0
[root@ractor vdsm]# 

If I power off the VM

[root@ractor vdsm]# cat /proc/meminfo | grep -i huge
AnonHugePages:         0 kB
HugePages_Total:    4096
HugePages_Free:     4096
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
[root@ractor vdsm]# 

Does this mean that in CentOS 7.3 Huge Pages could be reclaimed....???

Nevertheless, when I configure huge pages in guest it seems to work as expected

[root@dbtest ~]# cat /proc/meminfo | grep -i huge
AnonHugePages:         0 kB
HugePages_Total:    2048
HugePages_Free:     2048
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

Going into Oracle DB initialization, after configuring its dedicated memory (SGA) to 2354Mb, I get this confirmation inside its log file

Thu Apr 20 17:16:27 2017
 Per process system memlock (soft) limit = 4096M
Thu Apr 20 17:16:27 2017
 Expected per process system memlock (soft) limit to lock
 SHARED GLOBAL AREA (SGA) into memory: 2354M
Thu Apr 20 17:16:27 2017
 Available system pagesizes:
  4K, 2048K 
Thu Apr 20 17:16:27 2017
 Supported system pagesize(s):
Thu Apr 20 17:16:27 2017
  PAGESIZE  AVAILABLE_PAGES  EXPECTED_PAGES  ALLOCATED_PAGES  ERROR(s)
Thu Apr 20 17:16:27 2017
        4K       Configured               3               3        NONE
Thu Apr 20 17:16:27 2017
     2048K             2048            1177            1177        NONE

Gianluca