[ovirt-users] Question about Huge Pages

Thu Apr 20 15:39:20 UTC 2017

On Thu, Apr 20, 2017 at 10:35 AM, Michal Skrivanek <mskrivan at redhat.com>
wrote:

>
> On 19 Apr 2017, at 16:28, Gianluca Cecchi <gianluca.cecchi at gmail.com>
> wrote:
>
> On Wed, Apr 19, 2017 at 3:44 PM, Martin Polednik <mpolednik at redhat.com>
> wrote:
>
>>
>>>
>> If you are using recent CentOS (or I guess Fedora), there isn't any
>> extra setup required. Just create the custom property:
>>
>
> Both my engine and my hosts are CentOS 7.3 + updates
>
>
> that’s good
>
>
>
>>
>> On the host where engine is running:
>>
>> $ engine-config -s "UserDefinedVMProperties=hugepages=^.*$"
>> $ service ovirt-engine restart
>>
>> and you should see 'hugepages' when editing a VM under custom properties.
>>
>
> So no vdsm hook at all to install?
>
>
> today you still need the hook.
>
>
>
>
>> Set the number to (desired memory / 2048) and you're good to go. The
>> VM will run with it's memory backed by hugepages.
>
>
> As in sysctl.conf? So that if I want 4Gb of Huge Pages I have to set 2048?
>
>
> yes. there might be some
>
>
>
>
>> If you need
>> hugepages even inside the VM, do whatever you would do on a physical
>> host.
>>
>> mpolednik
>>
>>
> yes, the main subject is to have Huge Pages inside the guest, so that
> Oracle RDBMS at startup detect them and use them
>
>
> yes, so if you do that via sysctl.conf on real HW just do the same here,
> or modify kernel cmdline.
>
> Note that those are two separate things
> the hook is making QEMU process use hugepages memory in the host - that
> improves performance of any VM
> then how it looks in guest is no concern to oVirt, it’s guest-side
> hugepages. You can enable/set them regardless the previous step, which may
> be fine if you just want to expose the capability to some app  - e.g. in
> testing that the guest-side Oracle can work with hugepages in the guest.
> But you probably want both Oracle to see hugepages and also actually use
> them - then you need both reserve that on host for qemu process and then
> inside guest reserve that for oracle. I.e. you need to add a “buffer” on
> host side to accommodate the non-hugepages parts of the guest e.g. on 24GB
> host you can reserve 20GB hugepages for VMs to use, and then run a VM with
> 20GB memory, reserving 16GB hugepages inside the guest for oracle to use.
>
> Thanks,
> michal
>
>
> Gianluca
>
>
>
I'm making some tests right now.
Steps done:
- configure huge pages on hypervisor

[root at ractor ~]# cat /etc/sysctl.d/huge-pages.conf
# 20/04/2017 8Gb
vm.nr_hugepages = 4096
[root at ractor ~]#

rebooted host (I also updated in the mean time it to latest 4.1.1 packages
with vdsm-4.19.10.1-1.el7.centos.x86_64 and vdsm-hook-hugepages-4.19.
10.1-1.el7.centos.noarch)
I also set "transparent_hugepage=never" boot parameter because I know that
they are in conflict with Huge Pages

So the situation is:

[root at ractor ~]# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-514.16.1.el7.x86_64 root=/dev/mapper/centos-root
ro rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_US.UTF-8
transparent_hugepage=never
[root at ractor ~]#

[root at ractor ~]# cat /proc/meminfo | grep -i huge
AnonHugePages:         0 kB
HugePages_Total:    4096
HugePages_Free:     4096
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
[root at ractor ~]#

I edited a pre-existing CentOS 6 VM setting for it 8Gb of ram and 2048
pages (4Gb) in custom property forhugepages.

When I power on I get this addition in qemu-kvm process definition as
expected:

-mem-path /dev/hugepages/libvirt/qemu

I noticed that now I have on host....

[root at ractor vdsm]# cat /proc/meminfo | grep -i huge
AnonHugePages:         0 kB
HugePages_Total:    6144
HugePages_Free:     2048
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
[root at ractor vdsm]#

So apparently it did allocated 2048 new huge pages...
Does it mean that actually I have not to pre-allocate huge pages at all on
host and it eventually will increase them (but not able to remove then I
suppose) ?

Anyway the count doesn't seem correct... because it seems that a total of
4096 pages are in use/locked... (HugePages_Total - HugePages_Free
+ HugePages_Rsvd)
while they should be 2048.....

[root at ractor vdsm]# ll /dev/hugepages/libvirt/qemu/
total 0
[root at ractor vdsm]# ll /hugetlbfs/libvirt/qemu/
total 0
[root at ractor vdsm]#

If I power off the VM

[root at ractor vdsm]# cat /proc/meminfo | grep -i huge
AnonHugePages:         0 kB
HugePages_Total:    4096
HugePages_Free:     4096
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
[root at ractor vdsm]#

Does this mean that in CentOS 7.3 Huge Pages could be reclaimed....???

Nevertheless, when I configure huge pages in guest it seems to work as
expected

[root at dbtest ~]# cat /proc/meminfo | grep -i huge
AnonHugePages:         0 kB
HugePages_Total:    2048
HugePages_Free:     2048
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

Going into Oracle DB initialization, after configuring its dedicated memory
(SGA) to 2354Mb, I get this confirmation inside its log file

Thu Apr 20 17:16:27 2017
 Per process system memlock (soft) limit = 4096M
Thu Apr 20 17:16:27 2017
 Expected per process system memlock (soft) limit to lock
 SHARED GLOBAL AREA (SGA) into memory: 2354M
Thu Apr 20 17:16:27 2017
 Available system pagesizes:
  4K, 2048K
Thu Apr 20 17:16:27 2017
 Supported system pagesize(s):
Thu Apr 20 17:16:27 2017
  *PAGESIZE*  AVAILABLE_PAGES  EXPECTED_PAGES  *ALLOCATED_PAGES*  ERROR(s)
Thu Apr 20 17:16:27 2017
        4K       Configured               3               3        NONE
Thu Apr 20 17:16:27 2017
     *2048K *            2048            1177            *1177*        NONE

Gianluca
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170420/8c2ad362/attachment.html>