<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 20 Apr 2017, at 17:39, Gianluca Cecchi <<a href="mailto:gianluca.cecchi@gmail.com" class="">gianluca.cecchi@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote">On Thu, Apr 20, 2017 at 10:35 AM, Michal Skrivanek <span dir="ltr" class=""><<a href="mailto:mskrivan@redhat.com" target="_blank" class="">mskrivan@redhat.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word" class=""><br class=""><div class=""><span class="gmail-m_5930415786222383046gmail-"><blockquote type="cite" class=""><div class="">On 19 Apr 2017, at 16:28, Gianluca Cecchi <<a href="mailto:gianluca.cecchi@gmail.com" target="_blank" class="">gianluca.cecchi@gmail.com</a>> wrote:</div><br class="gmail-m_5930415786222383046gmail-m_-6852186268654812432Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote">On Wed, Apr 19, 2017 at 3:44 PM, Martin Polednik <span dir="ltr" class=""><<a href="mailto:mpolednik@redhat.com" target="_blank" class="">mpolednik@redhat.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail-m_5930415786222383046gmail-m_-6852186268654812432HOEnZb"><div class="gmail-m_5930415786222383046gmail-m_-6852186268654812432h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br class="">
</blockquote>
<br class=""></div></div>
If you are using recent CentOS (or I guess Fedora), there isn't any<br class="">
extra setup required. Just create the custom property:<br class=""></blockquote><div class=""><br class=""></div><div class="">Both my engine and my hosts are CentOS 7.3 + updates</div></div></div></div></div></blockquote><div class=""><br class=""></div></span>that’s good</div><div class=""><span class="gmail-m_5930415786222383046gmail-"><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br class="">
On the host where engine is running:<br class="">
<br class="">
$ engine-config -s "UserDefinedVMProperties=hugep<wbr class="">ages=^.*$"<br class="">
$ service ovirt-engine restart<br class="">
<br class="">
and you should see 'hugepages' when editing a VM under custom properties.<br class=""></blockquote><div class=""><br class=""></div><div class="">So no vdsm hook at all to install?</div></div></div></div></div></blockquote><div class=""><br class=""></div></span>today you still need the hook.</div><div class=""><span class="gmail-m_5930415786222383046gmail-"><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""><br class=""></div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Set the number to (desired memory / 2048) and you're good to go. The<br class="">
VM will run with it's memory backed by hugepages. </blockquote><div class=""><br class=""></div><div class="">As in sysctl.conf? So that if I want 4Gb of Huge Pages I have to set 2048?</div></div></div></div></div></blockquote><div class=""><br class=""></div></span>yes. there might be some </div><div class=""><span class="gmail-m_5930415786222383046gmail-"><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""><br class=""></div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">If you need<br class="">
hugepages even inside the VM, do whatever you would do on a physical<br class="">
host.<br class="">
<br class="">
mpolednik<div class="gmail-m_5930415786222383046gmail-m_-6852186268654812432HOEnZb"><div class="gmail-m_5930415786222383046gmail-m_-6852186268654812432h5"><br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">yes, the main subject is to have Huge Pages inside the guest, so that Oracle RDBMS at startup detect them and use them</div></div></div></div></div></blockquote><div class=""><br class=""></div></span>yes, so if you do that via sysctl.conf on real HW just do the same here, or modify kernel cmdline.</div><div class=""><br class=""></div><div class="">Note that those are two separate things</div><div class="">the hook is making QEMU process use hugepages memory in the host - that improves performance of any VM</div><div class="">then how it looks in guest is no concern to oVirt, it’s guest-side hugepages. You can enable/set them regardless the previous step, which may be fine if you just want to expose the capability to some app - e.g. in testing that the guest-side Oracle can work with hugepages in the guest.</div><div class="">But you probably want both Oracle to see hugepages and also actually use them - then you need both reserve that on host for qemu process and then inside guest reserve that for oracle. I.e. you need to add a “buffer” on host side to accommodate the non-hugepages parts of the guest e.g. on 24GB host you can reserve 20GB hugepages for VMs to use, and then run a VM with 20GB memory, reserving 16GB hugepages inside the guest for oracle to use.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">michal</div><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""><br class=""></div><div class="">Gianluca </div></div></div></div>
</div></blockquote></div><br class=""></div></blockquote><div class=""><br class=""></div><div class="">I'm making some tests right now. </div><div class="">Steps done: </div><div class="">- configure huge pages on hypervisor</div><div class=""><br class=""></div><div class=""><div class="">[root@ractor ~]# cat /etc/sysctl.d/huge-pages.conf </div><div class=""># 20/04/2017 8Gb</div><div class="">vm.nr_hugepages = 4096</div><div class="">[root@ractor ~]# </div></div><div class=""><br class=""></div><div class="">rebooted host (I also updated in the mean time it to latest 4.1.1 packages with vdsm-4.19.10.1-1.el7.<wbr class="">centos.x86_64 and vdsm-hook-hugepages-4.19.<wbr class="">10.1-1.el7.centos.noarch)</div><div class="">I also set "transparent_hugepage=never" boot parameter because I know that they are in conflict with Huge Pages</div><div class=""><br class=""></div><div class="">So the situation is:</div><div class=""><br class=""></div><div class=""><div class="">[root@ractor ~]# cat /proc/cmdline </div><div class="">BOOT_IMAGE=/vmlinuz-3.10.0-<wbr class="">514.16.1.el7.x86_64 root=/dev/mapper/centos-root ro <a href="http://rd.lvm.lv/" target="_blank" class="">rd.lvm.lv</a>=centos/root <a href="http://rd.lvm.lv/" target="_blank" class="">rd.lvm.lv</a>=centos/swap rhgb quiet LANG=en_US.UTF-8 transparent_hugepage=never</div><div class="">[root@ractor ~]# </div></div><div class=""><br class=""></div><div class=""><div class="">[root@ractor ~]# cat /proc/meminfo | grep -i huge</div><div class="">AnonHugePages: 0 kB</div><div class="">HugePages_Total: 4096</div><div class="">HugePages_Free: 4096</div><div class="">HugePages_Rsvd: 0</div><div class="">HugePages_Surp: 0</div><div class="">Hugepagesize: 2048 kB</div><div class="">[root@ractor ~]# </div></div><div class=""><br class=""></div><div class="">I edited a pre-existing CentOS 6 VM setting for it 8Gb of ram and 2048 pages (4Gb) in custom property forhugepages.</div><div class=""><br class=""></div><div class="">When I power on I get this addition in qemu-kvm process definition as expected:</div><div class=""><br class=""></div><div class="">-mem-path /dev/hugepages/libvirt/qemu<br class=""></div><div class=""><br class=""></div><div class="">I noticed that now I have on host....</div><div class=""><br class=""></div><div class=""><div class="">[root@ractor vdsm]# cat /proc/meminfo | grep -i huge</div><div class="">AnonHugePages: 0 kB</div><div class="">HugePages_Total: 6144</div><div class="">HugePages_Free: 2048</div><div class="">HugePages_Rsvd: 0</div><div class="">HugePages_Surp: 0</div><div class="">Hugepagesize: 2048 kB</div><div class="">[root@ractor vdsm]# </div></div><div class=""><br class=""></div><div class="">So apparently it did allocated 2048 new huge pages...</div></div></div><div class="gmail_extra"><div class="">Does it mean that actually I have not to pre-allocate huge pages at all on host and it eventually will increase them (but not able to remove then I suppose) ?</div><div class=""><br class=""></div><div class="">Anyway the count doesn't seem correct... because it seems that a total of 4096 pages are in use/locked... (HugePages_Total - HugePages_Free + HugePages_Rsvd)</div><div class="">while they should be 2048.....</div><div class=""><br class=""></div><div class=""><div class="">[root@ractor vdsm]# ll /dev/hugepages/libvirt/qemu/</div><div class="">total 0</div><div class="">[root@ractor vdsm]# ll /hugetlbfs/libvirt/qemu/</div><div class="">total 0</div><div class="">[root@ractor vdsm]# </div></div><div class=""><br class=""></div><div class="">If I power off the VM</div><div class=""><br class=""></div><div class=""><div class="">[root@ractor vdsm]# cat /proc/meminfo | grep -i huge</div><div class="">AnonHugePages: 0 kB</div><div class="">HugePages_Total: 4096</div><div class="">HugePages_Free: 4096</div><div class="">HugePages_Rsvd: 0</div><div class="">HugePages_Surp: 0</div><div class="">Hugepagesize: 2048 kB</div><div class="">[root@ractor vdsm]# </div></div><div class=""><br class=""></div><div class="">Does this mean that in CentOS 7.3 Huge Pages could be reclaimed....???</div></div></div></div></blockquote><div><br class=""></div>it tries to…well, as I said, the hook is being improved right now and in 4.2 it will likely be more consumable</div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class=""><br class=""></div><div class="">Nevertheless, when I configure huge pages in guest it seems to work as expected</div><div class=""><br class=""></div><div class=""><div class="">[root@dbtest ~]# cat /proc/meminfo | grep -i huge</div><div class="">AnonHugePages: 0 kB</div><div class="">HugePages_Total: 2048</div><div class="">HugePages_Free: 2048</div><div class="">HugePages_Rsvd: 0</div><div class="">HugePages_Surp: 0</div><div class="">Hugepagesize: 2048 kB</div><div class=""><br class=""></div></div><div class="">Going into Oracle DB initialization, after configuring its dedicated memory (SGA) to 2354Mb, I get this confirmation inside its log file</div></div></div></div></blockquote><div><br class=""></div>Yes, but that would always work even without setting things up on host. The “only” difference would be the actual performance.</div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class=""><br class=""></div><div class=""><div class="">Thu Apr 20 17:16:27 2017</div><div class=""> Per process system memlock (soft) limit = 4096M</div><div class="">Thu Apr 20 17:16:27 2017</div><div class=""> Expected per process system memlock (soft) limit to lock</div><div class=""> SHARED GLOBAL AREA (SGA) into memory: 2354M</div><div class="">Thu Apr 20 17:16:27 2017</div><div class=""> Available system pagesizes:</div><div class=""> 4K, 2048K </div><div class="">Thu Apr 20 17:16:27 2017</div><div class=""> Supported system pagesize(s):</div><div class="">Thu Apr 20 17:16:27 2017</div><div class=""> <b class="">PAGESIZE</b> AVAILABLE_PAGES EXPECTED_PAGES <b class="">ALLOCATED_PAGES</b> ERROR(s)</div><div class="">Thu Apr 20 17:16:27 2017</div><div class=""> 4K Configured 3 3 NONE</div><div class="">Thu Apr 20 17:16:27 2017</div><div class=""> <b class="">2048K </b> 2048 1177 <b class="">1177</b> NONE</div></div><div class=""><br class=""></div><div class="">Gianluca</div><div class=""><br class=""></div></div></div>
</div></blockquote></div><br class=""></body></html>