7:14 p.m.
Hello List,
i ran into problems using numa-pinning and reserved hugepages.
- My EPYC 7281 based Servers (Dual Socket) have 8 Numa-Nodes each having
32 GB of memory for a total of 256 GB System Memory
- I'm using 192 x 1 GB hugepages reserved on the kernel cmdline
default_hugepagesz=1G hugepagesz=1G hugepages=192 This reserves 24
hugepages on each numa-node.
I wanted to pin a MariaDB VM using 32 GB (Custom Property
hugepages=1048576) to numa-nodes 0-3 of CPU-Socket 1. Pinning in GUI
etc. no problem.
When trying to start the vm this can't be done since ovirt claims that
the host can't fullfill the memory requirements - which is simply not
correct since there were > 164 hugepages free.
It should have taken 8 hugepages from each numa node 0-3 to fullfill the
32 GB Memory requirement.
I also freed the system completely from other VM's but that didn't work
either.
Is it possible that the scheduler only takes into account the "free
memory" (as seen in numactl -H below) *not reserved* by hugepages for
its decisions ? Since the host has only < 8 GB of free mem per numa-node
I can understand that VM was not able to start under that condition.
VM is runnig and using 32 hugepages without pinning but a warning states
"VM dbserver01b does not fit to a single NUMA node on host
myhost.mydomain.de. This may negatively impact its performance. Consider
using vNUMA and NUMA pinning for this VM."
This is the numa Hardware Layout and hugepages usage now with other VM's
running:
from cat /proc/meminfo
HugePages_Total: 192
HugePages_Free: 160
HugePages_Rsvd: 0
HugePages_Surp: 0
I can confirm that also under the condition of running other VM's there
are at least 8 hugepages free for each numa-node 0-3:
grep ""
/sys/devices/system/node/*/hugepages/hugepages-1048576kB/free_hugepages
/sys/devices/system/node/node0/hugepages/hugepages-1048576kB/free_hugepages:8
/sys/devices/system/node/node1/hugepages/hugepages-1048576kB/free_hugepages:23
/sys/devices/system/node/node2/hugepages/hugepages-1048576kB/free_hugepages:20
/sys/devices/system/node/node3/hugepages/hugepages-1048576kB/free_hugepages:22
/sys/devices/system/node/node4/hugepages/hugepages-1048576kB/free_hugepages:16
/sys/devices/system/node/node5/hugepages/hugepages-1048576kB/free_hugepages:5
/sys/devices/system/node/node6/hugepages/hugepages-1048576kB/free_hugepages:19
/sys/devices/system/node/node7/hugepages/hugepages-1048576kB/free_hugepages:24
numactl -h:
available: 8 nodes (0-7)
node 0 cpus: 0 1 2 3 32 33 34 35
node 0 size: 32673 MB
node 0 free: 3779 MB
node 1 cpus: 4 5 6 7 36 37 38 39
node 1 size: 32767 MB
node 1 free: 6162 MB
node 2 cpus: 8 9 10 11 40 41 42 43
node 2 size: 32767 MB
node 2 free: 6698 MB
node 3 cpus: 12 13 14 15 44 45 46 47
node 3 size: 32767 MB
node 3 free: 1589 MB
node 4 cpus: 16 17 18 19 48 49 50 51
node 4 size: 32767 MB
node 4 free: 2630 MB
node 5 cpus: 20 21 22 23 52 53 54 55
node 5 size: 32767 MB
node 5 free: 2487 MB
node 6 cpus: 24 25 26 27 56 57 58 59
node 6 size: 32767 MB
node 6 free: 3279 MB
node 7 cpus: 28 29 30 31 60 61 62 63
node 7 size: 32767 MB
node 7 free: 5513 MB
node distances:
node 0 1 2 3 4 5 6 7
0: 10 16 16 16 32 32 32 32
1: 16 10 16 16 32 32 32 32
2: 16 16 10 16 32 32 32 32
3: 16 16 16 10 32 32 32 32
4: 32 32 32 32 10 16 16 16
5: 32 32 32 32 16 10 16 16
6: 32 32 32 32 16 16 10 16
7: 32 32 32 32 16 16 16 10
--
*Ralf Schenk*
fon +49 (0) 24 05 / 40 83 70
fax +49 (0) 24 05 / 40 83 759
mail *rs(a)databay.de* <mailto:rs@databay.de>
*Databay AG*
Jens-Otto-Krag-Straße 11
D-52146 Würselen
*www.databay.de* <http://www.databay.de>
Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202
Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm.
Philipp Hermanns
Aufsichtsratsvorsitzender: Wilhelm Dohmen
------------------------------------------------------------------------
9:27 p.m.
New subject: numa pinning and reserved hugepages (1G) Bug in scheduler calculation or decision ?
Hi,
this is a bug in the scheduler. Currently, it ignores hugepages when
evaluating NUMA pinning.
There is a bugzilla ticket[1] that was originally reported as a similar
case, but then later the reporter changed it.
Could you open a new bugzilla ticket and attach the details from this email?
As a workaround, if you don't want to migrate the VM or you are sure that
it can run on the target host, you can clone a cluster policy and remove
the 'NUMA' filter. (In Administration -> Configure -> Scheduling Policies).
[1] - https://bugzilla.redhat.com/show_bug.cgi?id=1720558
Best regards,
Andrej
On Wed, 21 Aug 2019 at 12:16, Ralf Schenk <rs(a)databay.de> wrote:
Hello List,
i ran into problems using numa-pinning and reserved hugepages.
- My EPYC 7281 based Servers (Dual Socket) have 8 Numa-Nodes each having
32 GB of memory for a total of 256 GB System Memory
- I'm using 192 x 1 GB hugepages reserved on the kernel cmdline
default_hugepagesz=1G hugepagesz=1G hugepages=192 This reserves 24
hugepages on each numa-node.
I wanted to pin a MariaDB VM using 32 GB (Custom Property
hugepages=1048576) to numa-nodes 0-3 of CPU-Socket 1. Pinning in GUI etc.
no problem.
When trying to start the vm this can't be done since ovirt claims that the
host can't fullfill the memory requirements - which is simply not correct
since there were > 164 hugepages free.
It should have taken 8 hugepages from each numa node 0-3 to fullfill the
32 GB Memory requirement.
I also freed the system completely from other VM's but that didn't work
either.
Is it possible that the scheduler only takes into account the "free
memory" (as seen in numactl -H below) *not reserved* by hugepages for its
decisions ? Since the host has only < 8 GB of free mem per numa-node I can
understand that VM was not able to start under that condition.
VM is runnig and using 32 hugepages without pinning but a warning states
"VM dbserver01b does not fit to a single NUMA node on host
myhost.mydomain.de. This may negatively impact its performance. Consider
using vNUMA and NUMA pinning for this VM."
This is the numa Hardware Layout and hugepages usage now with other VM's
running:
from cat /proc/meminfo
HugePages_Total: 192
HugePages_Free: 160
HugePages_Rsvd: 0
HugePages_Surp: 0
I can confirm that also under the condition of running other VM's there
are at least 8 hugepages free for each numa-node 0-3:
grep ""
/sys/devices/system/node/*/hugepages/hugepages-1048576kB/free_hugepages
/sys/devices/system/node/node0/hugepages/hugepages-1048576kB/free_hugepages:8
/sys/devices/system/node/node1/hugepages/hugepages-1048576kB/free_hugepages:23
/sys/devices/system/node/node2/hugepages/hugepages-1048576kB/free_hugepages:20
/sys/devices/system/node/node3/hugepages/hugepages-1048576kB/free_hugepages:22
/sys/devices/system/node/node4/hugepages/hugepages-1048576kB/free_hugepages:16
/sys/devices/system/node/node5/hugepages/hugepages-1048576kB/free_hugepages:5
/sys/devices/system/node/node6/hugepages/hugepages-1048576kB/free_hugepages:19
/sys/devices/system/node/node7/hugepages/hugepages-1048576kB/free_hugepages:24
numactl -h:
available: 8 nodes (0-7)
node 0 cpus: 0 1 2 3 32 33 34 35
node 0 size: 32673 MB
node 0 free: 3779 MB
node 1 cpus: 4 5 6 7 36 37 38 39
node 1 size: 32767 MB
node 1 free: 6162 MB
node 2 cpus: 8 9 10 11 40 41 42 43
node 2 size: 32767 MB
node 2 free: 6698 MB
node 3 cpus: 12 13 14 15 44 45 46 47
node 3 size: 32767 MB
node 3 free: 1589 MB
node 4 cpus: 16 17 18 19 48 49 50 51
node 4 size: 32767 MB
node 4 free: 2630 MB
node 5 cpus: 20 21 22 23 52 53 54 55
node 5 size: 32767 MB
node 5 free: 2487 MB
node 6 cpus: 24 25 26 27 56 57 58 59
node 6 size: 32767 MB
node 6 free: 3279 MB
node 7 cpus: 28 29 30 31 60 61 62 63
node 7 size: 32767 MB
node 7 free: 5513 MB
node distances:
node 0 1 2 3 4 5 6 7
0: 10 16 16 16 32 32 32 32
1: 16 10 16 16 32 32 32 32
2: 16 16 10 16 32 32 32 32
3: 16 16 16 10 32 32 32 32
4: 32 32 32 32 10 16 16 16
5: 32 32 32 32 16 10 16 16
6: 32 32 32 32 16 16 10 16
7: 32 32 32 32 16 16 16 10
--
*Ralf Schenk*
fon +49 (0) 24 05 / 40 83 70
fax +49 (0) 24 05 / 40 83 759
mail *rs(a)databay.de* <rs(a)databay.de>
*Databay AG*
Jens-Otto-Krag-Straße 11
D-52146 Würselen
*www.databay.de* <http://www.databay.de>
Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202
Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm.
Philipp Hermanns
Aufsichtsratsvorsitzender: Wilhelm Dohmen
------------------------------
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/GXIWD6B6E3U...
7:45 p.m.
New subject: numa pinning and reserved hugepages (1G) Bug in scheduler calculation or decision ?
Hello,
reported it (again) as
https://bugzilla.redhat.com/show_bug.cgi?id=1745247 and referenced [1]
Thanks
Am 23.08.2019 um 14:27 schrieb Andrej Krejcir:
Hi,
this is a bug in the scheduler. Currently, it ignores hugepages when
evaluating NUMA pinning.
There is a bugzilla ticket[1] that was originally reported as a
similar case, but then later the reporter changed it.
Could you open a new bugzilla ticket and attach the details from this
email?
As a workaround, if you don't want to migrate the VM or you are sure
that it can run on the target host, you can clone a cluster policy and
remove the 'NUMA' filter. (In Administration -> Configure ->
Scheduling Policies).
[1] - https://bugzilla.redhat.com/show_bug.cgi?id=1720558
Best regards,
Andrej
On Wed, 21 Aug 2019 at 12:16, Ralf Schenk <rs(a)databay.de
<mailto:rs@databay.de>> wrote:
Hello List,
i ran into problems using numa-pinning and reserved hugepages.
- My EPYC 7281 based Servers (Dual Socket) have 8 Numa-Nodes each
having 32 GB of memory for a total of 256 GB System Memory
- I'm using 192 x 1 GB hugepages reserved on the kernel cmdline
default_hugepagesz=1G hugepagesz=1G hugepages=192 This reserves 24
hugepages on each numa-node.
I wanted to pin a MariaDB VM using 32 GB (Custom Property
hugepages=1048576) to numa-nodes 0-3 of CPU-Socket 1. Pinning in
GUI etc. no problem.
When trying to start the vm this can't be done since ovirt claims
that the host can't fullfill the memory requirements - which is
simply not correct since there were > 164 hugepages free.
It should have taken 8 hugepages from each numa node 0-3 to
fullfill the 32 GB Memory requirement.
I also freed the system completely from other VM's but that didn't
work either.
Is it possible that the scheduler only takes into account the
"free memory" (as seen in numactl -H below) *not reserved* by
hugepages for its decisions ? Since the host has only < 8 GB of
free mem per numa-node I can understand that VM was not able to
start under that condition.
VM is runnig and using 32 hugepages without pinning but a warning
states "VM dbserver01b does not fit to a single NUMA node on host
myhost.mydomain.de <http://myhost.mydomain.de>;. This may
negatively impact its performance. Consider using vNUMA and NUMA
pinning for this VM."
This is the numa Hardware Layout and hugepages usage now with
other VM's running:
from cat /proc/meminfo
HugePages_Total: 192
HugePages_Free: 160
HugePages_Rsvd: 0
HugePages_Surp: 0
I can confirm that also under the condition of running other VM's
there are at least 8 hugepages free for each numa-node 0-3:
grep ""
/sys/devices/system/node/*/hugepages/hugepages-1048576kB/free_hugepages
/sys/devices/system/node/node0/hugepages/hugepages-1048576kB/free_hugepages:8
/sys/devices/system/node/node1/hugepages/hugepages-1048576kB/free_hugepages:23
/sys/devices/system/node/node2/hugepages/hugepages-1048576kB/free_hugepages:20
/sys/devices/system/node/node3/hugepages/hugepages-1048576kB/free_hugepages:22
/sys/devices/system/node/node4/hugepages/hugepages-1048576kB/free_hugepages:16
/sys/devices/system/node/node5/hugepages/hugepages-1048576kB/free_hugepages:5
/sys/devices/system/node/node6/hugepages/hugepages-1048576kB/free_hugepages:19
/sys/devices/system/node/node7/hugepages/hugepages-1048576kB/free_hugepages:24
numactl -h:
available: 8 nodes (0-7)
node 0 cpus: 0 1 2 3 32 33 34 35
node 0 size: 32673 MB
node 0 free: 3779 MB
node 1 cpus: 4 5 6 7 36 37 38 39
node 1 size: 32767 MB
node 1 free: 6162 MB
node 2 cpus: 8 9 10 11 40 41 42 43
node 2 size: 32767 MB
node 2 free: 6698 MB
node 3 cpus: 12 13 14 15 44 45 46 47
node 3 size: 32767 MB
node 3 free: 1589 MB
node 4 cpus: 16 17 18 19 48 49 50 51
node 4 size: 32767 MB
node 4 free: 2630 MB
node 5 cpus: 20 21 22 23 52 53 54 55
node 5 size: 32767 MB
node 5 free: 2487 MB
node 6 cpus: 24 25 26 27 56 57 58 59
node 6 size: 32767 MB
node 6 free: 3279 MB
node 7 cpus: 28 29 30 31 60 61 62 63
node 7 size: 32767 MB
node 7 free: 5513 MB
node distances:
node 0 1 2 3 4 5 6 7
0: 10 16 16 16 32 32 32 32
1: 16 10 16 16 32 32 32 32
2: 16 16 10 16 32 32 32 32
3: 16 16 16 10 32 32 32 32
4: 32 32 32 32 10 16 16 16
5: 32 32 32 32 16 10 16 16
6: 32 32 32 32 16 16 10 16
7: 32 32 32 32 16 16 16 10
--
*Ralf Schenk*
fon +49 (0) 24 05 / 40 83 70
fax +49 (0) 24 05 / 40 83 759
mail *rs(a)databay.de* <mailto:rs@databay.de>
*Databay AG*
Jens-Otto-Krag-Straße 11
D-52146 Würselen
*www.databay.de* <http://www.databay.de>
Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202
Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari,
Dipl.-Kfm. Philipp Hermanns
Aufsichtsratsvorsitzender: Wilhelm Dohmen
------------------------------------------------------------------------
_______________________________________________
Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org>
To unsubscribe send an email to users-leave(a)ovirt.org
<mailto:users-leave@ovirt.org>
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/GXIWD6B6E3U...
--
*Ralf Schenk*
fon +49 (0) 24 05 / 40 83 70
fax +49 (0) 24 05 / 40 83 759
mail *rs(a)databay.de* <mailto:rs@databay.de>
*Databay AG*
Jens-Otto-Krag-Straße 11
D-52146 Würselen
*www.databay.de* <http://www.databay.de>
Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202
Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm.
Philipp Hermanns
Aufsichtsratsvorsitzender: Wilhelm Dohmen
------------------------------------------------------------------------
1707
days inactive
1710
days old
2 comments
2 participants
participants (2)
-
Andrej Krejcir -
Ralf Schenk