4:14 a.m.
Hello List,
i ran into problems using numa-pinning and reserved hugepages.
- My EPYC 7281 based Servers (Dual Socket) have 8 Numa-Nodes each having
32 GB of memory for a total of 256 GB System Memory
- I'm using 192 x 1 GB hugepages reserved on the kernel cmdline
default_hugepagesz=1G hugepagesz=1G hugepages=192 This reserves 24
hugepages on each numa-node.
I wanted to pin a MariaDB VM using 32 GB (Custom Property
hugepages=1048576) to numa-nodes 0-3 of CPU-Socket 1. Pinning in GUI
etc. no problem.
When trying to start the vm this can't be done since ovirt claims that
the host can't fullfill the memory requirements - which is simply not
correct since there were > 164 hugepages free.
It should have taken 8 hugepages from each numa node 0-3 to fullfill the
32 GB Memory requirement.
I also freed the system completely from other VM's but that didn't work
either.
Is it possible that the scheduler only takes into account the "free
memory" (as seen in numactl -H below) *not reserved* by hugepages for
its decisions ? Since the host has only < 8 GB of free mem per numa-node
I can understand that VM was not able to start under that condition.
VM is runnig and using 32 hugepages without pinning but a warning states
"VM dbserver01b does not fit to a single NUMA node on host
myhost.mydomain.de. This may negatively impact its performance. Consider
using vNUMA and NUMA pinning for this VM."
This is the numa Hardware Layout and hugepages usage now with other VM's
running:
from cat /proc/meminfo
HugePages_Total: 192
HugePages_Free: 160
HugePages_Rsvd: 0
HugePages_Surp: 0
I can confirm that also under the condition of running other VM's there
are at least 8 hugepages free for each numa-node 0-3:
grep ""
/sys/devices/system/node/*/hugepages/hugepages-1048576kB/free_hugepages
/sys/devices/system/node/node0/hugepages/hugepages-1048576kB/free_hugepages:8
/sys/devices/system/node/node1/hugepages/hugepages-1048576kB/free_hugepages:23
/sys/devices/system/node/node2/hugepages/hugepages-1048576kB/free_hugepages:20
/sys/devices/system/node/node3/hugepages/hugepages-1048576kB/free_hugepages:22
/sys/devices/system/node/node4/hugepages/hugepages-1048576kB/free_hugepages:16
/sys/devices/system/node/node5/hugepages/hugepages-1048576kB/free_hugepages:5
/sys/devices/system/node/node6/hugepages/hugepages-1048576kB/free_hugepages:19
/sys/devices/system/node/node7/hugepages/hugepages-1048576kB/free_hugepages:24
numactl -h:
available: 8 nodes (0-7)
node 0 cpus: 0 1 2 3 32 33 34 35
node 0 size: 32673 MB
node 0 free: 3779 MB
node 1 cpus: 4 5 6 7 36 37 38 39
node 1 size: 32767 MB
node 1 free: 6162 MB
node 2 cpus: 8 9 10 11 40 41 42 43
node 2 size: 32767 MB
node 2 free: 6698 MB
node 3 cpus: 12 13 14 15 44 45 46 47
node 3 size: 32767 MB
node 3 free: 1589 MB
node 4 cpus: 16 17 18 19 48 49 50 51
node 4 size: 32767 MB
node 4 free: 2630 MB
node 5 cpus: 20 21 22 23 52 53 54 55
node 5 size: 32767 MB
node 5 free: 2487 MB
node 6 cpus: 24 25 26 27 56 57 58 59
node 6 size: 32767 MB
node 6 free: 3279 MB
node 7 cpus: 28 29 30 31 60 61 62 63
node 7 size: 32767 MB
node 7 free: 5513 MB
node distances:
node 0 1 2 3 4 5 6 7
0: 10 16 16 16 32 32 32 32
1: 16 10 16 16 32 32 32 32
2: 16 16 10 16 32 32 32 32
3: 16 16 16 10 32 32 32 32
4: 32 32 32 32 10 16 16 16
5: 32 32 32 32 16 10 16 16
6: 32 32 32 32 16 16 10 16
7: 32 32 32 32 16 16 16 10
--
*Ralf Schenk*
fon +49 (0) 24 05 / 40 83 70
fax +49 (0) 24 05 / 40 83 759
mail *rs(a)databay.de* <mailto:rs@databay.de>
*Databay AG*
Jens-Otto-Krag-Straße 11
D-52146 Würselen
*www.databay.de* <http://www.databay.de>
Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202
Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm.
Philipp Hermanns
Aufsichtsratsvorsitzender: Wilhelm Dohmen
------------------------------------------------------------------------
