Hello List,
i ran into problems
using numa-pinning and reserved hugepages.
- My EPYC 7281 based
Servers (Dual Socket) have 8 Numa-Nodes each having 32 GB of
memory for a total of 256 GB System Memory
- I'm using 192 x 1 GB
hugepages reserved on the kernel cmdline default_hugepagesz=1G
hugepagesz=1G hugepages=192 This reserves 24 hugepages on each
numa-node.
I wanted to pin a MariaDB VM using 32 GB (Custom Property hugepages=1048576) to numa-nodes 0-3 of CPU-Socket 1. Pinning in GUI etc. no problem.
When trying to start
the vm this can't be done since ovirt claims that the host can't
fullfill the memory requirements - which is simply not correct
since there were > 164 hugepages free.
It should have taken 8
hugepages from each numa node 0-3 to fullfill the 32 GB Memory
requirement.
I also freed the system completely from other VM's but that didn't work either.
Is it possible that the scheduler only takes into account the "free memory" (as seen in numactl -H below) not reserved by hugepages for its decisions ? Since the host has only < 8 GB of free mem per numa-node I can understand that VM was not able to start under that condition.
VM is runnig and using 32 hugepages without pinning but a warning states "VM dbserver01b does not fit to a single NUMA node on host myhost.mydomain.de. This may negatively impact its performance. Consider using vNUMA and NUMA pinning for this VM."
This is the numa Hardware Layout and hugepages usage now with other VM's running:
from cat /proc/meminfo
HugePages_Total: 192
HugePages_Free: 160
HugePages_Rsvd: 0
HugePages_Surp: 0
I can confirm that also under the condition of running other VM's there are at least 8 hugepages free for each numa-node 0-3:
grep ""
/sys/devices/system/node/*/hugepages/hugepages-1048576kB/free_hugepages
/sys/devices/system/node/node0/hugepages/hugepages-1048576kB/free_hugepages:8
/sys/devices/system/node/node1/hugepages/hugepages-1048576kB/free_hugepages:23
/sys/devices/system/node/node2/hugepages/hugepages-1048576kB/free_hugepages:20
/sys/devices/system/node/node3/hugepages/hugepages-1048576kB/free_hugepages:22
/sys/devices/system/node/node4/hugepages/hugepages-1048576kB/free_hugepages:16
/sys/devices/system/node/node5/hugepages/hugepages-1048576kB/free_hugepages:5
/sys/devices/system/node/node6/hugepages/hugepages-1048576kB/free_hugepages:19
/sys/devices/system/node/node7/hugepages/hugepages-1048576kB/free_hugepages:24
numactl -h:
available: 8 nodes (0-7)
node 0 cpus: 0 1 2 3 32 33 34 35
node 0 size: 32673 MB
node 0 free: 3779 MB
node 1 cpus: 4 5 6 7 36 37 38 39
node 1 size: 32767 MB
node 1 free: 6162 MB
node 2 cpus: 8 9 10 11 40 41 42 43
node 2 size: 32767 MB
node 2 free: 6698 MB
node 3 cpus: 12 13 14 15 44 45 46 47
node 3 size: 32767 MB
node 3 free: 1589 MB
node 4 cpus: 16 17 18 19 48 49 50 51
node 4 size: 32767 MB
node 4 free: 2630 MB
node 5 cpus: 20 21 22 23 52 53 54 55
node 5 size: 32767 MB
node 5 free: 2487 MB
node 6 cpus: 24 25 26 27 56 57 58 59
node 6 size: 32767 MB
node 6 free: 3279 MB
node 7 cpus: 28 29 30 31 60 61 62 63
node 7 size: 32767 MB
node 7 free: 5513 MB
node distances:
node 0 1 2 3 4 5 6 7
0: 10 16 16 16 32 32 32 32
1: 16 10 16 16 32 32 32 32
2: 16 16 10 16 32 32 32 32
3: 16 16 16 10 32 32 32 32
4: 32 32 32 32 10 16 16 16
5: 32 32 32 32 16 10 16 16
6: 32 32 32 32 16 16 10 16
7: 32 32 32 32 16 16 16 10
| Ralf Schenk fon +49 (0) 24 05 / 40 83 70 fax +49 (0) 24 05 / 40 83 759 mail rs@databay.de |
Databay AG Jens-Otto-Krag-Straße 11 D-52146 Würselen www.databay.de |
|
| Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202 Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm. Philipp Hermanns Aufsichtsratsvorsitzender: Wilhelm Dohmen |
||