7:13 p.m.
Hi,
I'm having performance issues with a ovirt installation. It is showing
high steal (5-10%) for a cpu intensive VM. The hypervisor however has
more than 65% of his resources idle while the steal is seen inside of
the VM.
Even when placing only a single VM on a hypervisor it still receives
steal (0-2%), even though the hypervisor is not overcommited.
Hypervisor:
2 Socket system in total 2*28(56HT) cores
VM:
30vCPUs (ovirt seems to think its a good idea to make that 15 sockets *
2 cores)
My questions are:
a) Could it be that the hypervisor is trying to schedule all 30 cores on
a single numa node, ie using the HT cores instead of "real" ones and
this shows up as steal?
b) Do I need to make VMs this big numa-aware and spread the vm over both
numa nodes?
c) Would using the High Performance VM type help in this kind of situation?
d) General advise: how do I reduce steal in an environment where the
hypervisor has idle resources
Any advise would be appreciated.
Greetings
Klaas
3:09 a.m.
On Wed, Dec 4, 2019 at 6:15 PM <klaasdemter(a)gmail.com> wrote:
Hi,
I'm having performance issues with a ovirt installation. It is showing
high steal (5-10%) for a cpu intensive VM. The hypervisor however has
more than 65% of his resources idle while the steal is seen inside of
the VM.
Even when placing only a single VM on a hypervisor it still receives
steal (0-2%), even though the hypervisor is not overcommited.
Hypervisor:
2 Socket system in total 2*28(56HT) cores
VM:
30vCPUs (ovirt seems to think its a good idea to make that 15 sockets *
2 cores)
I think you can control this in oVirt.
My questions are:
a) Could it be that the hypervisor is trying to schedule all 30 cores on
a single numa node, ie using the HT cores instead of "real" ones and
this shows up as steal?
b) Do I need to make VMs this big numa-aware and spread the vm over both
numa nodes?
c) Would using the High Performance VM type help in this kind of situation?
d) General advise: how do I reduce steal in an environment where the
hypervisor has idle resources
Any advise would be appreciated.
These questions are mainly about qemu, so adding qemu-discuss.
I think it will help if you share your vm qemu command line, found in:
/var/log/libvit/qemu/vm-name.log
Nir
11:08 a.m.
Hi,
Thanks for the quick response.
On Thu, Dec 5, 2019, 01:10 Nir Soffer <nsoffer(a)redhat.com> wrote:
On Wed, Dec 4, 2019 at 6:15 PM <klaasdemter(a)gmail.com> wrote:
>
> Hi,
>
> I'm having performance issues with a ovirt installation. It is showing
> high steal (5-10%) for a cpu intensive VM. The hypervisor however has
> more than 65% of his resources idle while the steal is seen inside of
> the VM.
>
> Even when placing only a single VM on a hypervisor it still receives
> steal (0-2%), even though the hypervisor is not overcommited.
>
>
> Hypervisor:
>
> 2 Socket system in total 2*28(56HT) cores
>
>
> VM:
>
> 30vCPUs (ovirt seems to think its a good idea to make that 15 sockets *
> 2 cores)
I think you can control this in oVirt.
Yes, I'm saying ovirt defaults to using 15 sockets * 2 cores if you don't
specify a different "advanced" configuration. This also leads to a funny
behavior that you can't increase the VMs from 30 to 35 cores online.
> My questions are:
>
> a) Could it be that the hypervisor is trying to schedule all 30 cores on
> a single numa node, ie using the HT cores instead of "real" ones and
> this shows up as steal?
>
> b) Do I need to make VMs this big numa-aware and spread the vm over both
> numa nodes?
>
> c) Would using the High Performance VM type help in this kind of
situation?
>
> d) General advise: how do I reduce steal in an environment where the
> hypervisor has idle resources
>
>
> Any advise would be appreciated.
These questions are mainly about qemu, so adding qemu-discuss.
I think it will help if you share your vm qemu command line, found in:
/var/log/libvit/qemu/vm-name.log
Log:
2019-11-16 22:33:33.487+0000: starting up libvirt version: 4.5.0, package:
10.el7_6.6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>;,
2019-02-21-14:30:01, x86-038.build.eng.bos.redhat.com), qemu version:
2.12.0qemu-kvm-rhev-2.12.0-18.el7_6.3, kernel: 3.10.0-957.10.1.el7.x86_64,
hostname: hypervisor.tld
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name guest=vm,debug-threads=on
-S -object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-24-vm/master-key.aes
-machine pc-i440fx-rhel7.5.0,accel=kvm,usb=off,dump-guest-core=off -cpu
Broadwell,spec-ctrl=on,ssbd=on,rtm=on,hle=on -m
size=534773760k,slots=16,maxmem=629145600k -realtime mlock=off -smp
30,maxcpus=32,sockets=16,cores=2,threads=1 -object iothread,id=iothread1
-numa node,nodeid=0,cpus=0-29,mem=522240 -uuid
95285008-8efb-446c-b75c-771f020cf494 -smbios
'type=1,manufacturer=oVirt,product=RHEV
Hypervisor,version=7.6-4.el7,serial=4c4c4544-004b-3510-805a-c3c04f485032,uuid=95285008-8efb-446c-b75c-771f020cf494'
-no-user-config -nodefaults -chardev
socket,id=charmonitor,fd=51,server,nowait -mon
chardev=charmonitor,id=monitor,mode=control -rtc
base=2019-11-16T22:33:33,driftfix=slew -global
kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global
PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
virtio-scsi-pci,iothread=iothread1,id=ua-98c9be1a-c830-4718-9936-28a81e9a07b6,bus=pci.0,addr=0x5
-device
virtio-serial-pci,id=ua-3f6418a3-8b1a-49c3-b79b-ca5f1864cf47,max_ports=16,bus=pci.0,addr=0x4
-drive
if=none,id=drive-ua-641b4623-d2b3-4f7f-a31d-65959f5ffc9b,werror=report,rerror=report,readonly=on
-device
ide-cd,bus=ide.1,unit=0,drive=drive-ua-641b4623-d2b3-4f7f-a31d-65959f5ffc9b,id=ua-641b4623-d2b3-4f7f-a31d-65959f5ffc9b
-drive
file=/rhev/data-center/mnt/storage-domain/0f623c51-66c8-4a5b-a12f-95ab9c6184b4/images/88901d29-e223-457d-a80e-384b1f147cea/0f994c76-d525-4a4f-9513-c6220fe7c31c,format=raw,if=none,id=drive-ua-88901d29-e223-457d-a80e-384b1f147cea,serial=88901d29-e223-457d-a80e-384b1f147cea,werror=stop,rerror=stop,cache=none,aio=threads
-device
scsi-hd,bus=ua-98c9be1a-c830-4718-9936-28a81e9a07b6.0,channel=0,scsi-id=0,lun=0,drive=drive-ua-88901d29-e223-457d-a80e-384b1f147cea,id=ua-88901d29-e223-457d-a80e-384b1f147cea,bootindex=1,write-cache=on
-netdev
tap,fds=54:55:56:57,id=hostua-1fad81fe-8c0b-41dd-8015-7498ebbc2710,vhost=on,vhostfds=58:59:60:61
-device
virtio-net-pci,mq=on,vectors=10,host_mtu=1500,netdev=hostua-1fad81fe-8c0b-41dd-8015-7498ebbc2710,id=ua-1fad81fe-8c0b-41dd-8015-7498ebbc2710,mac=22:44:66:00:00:d0,bus=pci.0,addr=0x3
-chardev socket,id=charchannel0,fd=62,server,nowait -device
virtserialport,bus=ua-3f6418a3-8b1a-49c3-b79b-ca5f1864cf47.0,nr=1,chardev=charchannel0,id=channel0,name=ovirt-guest-agent.0
-chardev socket,id=charchannel1,fd=63,server,nowait -device
virtserialport,bus=ua-3f6418a3-8b1a-49c3-b79b-ca5f1864cf47.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0
-chardev spicevmc,id=charchannel2,name=vdagent -device
virtserialport,bus=ua-3f6418a3-8b1a-49c3-b79b-ca5f1864cf47.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0
-spice
port=5918,tls-port=5919,addr=10.160.69.12,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on
-device
qxl-vga,id=ua-a3bde96a-2165-434e-80f5-5980dbb9f19d,ram_size=67108864,vram_size=33554432,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2
-device
virtio-balloon-pci,id=ua-124c48f3-6bc4-453c-b0fb-107060bbc1f6,bus=pci.0,addr=0x6
-object
rng-random,id=objua-ff7eb44a-85a1-4731-aec0-0b85b16c2644,filename=/dev/urandom
-device
virtio-rng-pci,rng=objua-ff7eb44a-85a1-4731-aec0-0b85b16c2644,id=ua-ff7eb44a-85a1-4731-aec0-0b85b16c2644,bus=pci.0,addr=0x7
-device vmcoreinfo -sandbox
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny
-msg timestamp=on
2019-11-16T22:33:33.624932Z qemu-kvm: -drive
file=/rhev/data-center/mnt/storage-domain/0f623c51-66c8-4a5b-a12f-95ab9c6184b4/images/88901d29-e223-457d-a80e-384b1f147cea/0f994c76-d525-4a4f-9513-c6220fe7c31c,format=raw,if=none,id=drive-ua-88901d29-e223-457d-a80e-384b1f147cea,serial=88901d29-e223-457d-a80e-384b1f147cea,werror=stop,rerror=stop,cache=none,aio=threads:
'serial' is deprecated, please use the corresponding option of '-device'
instead
2019-11-16T22:33:33.630420Z qemu-kvm: warning: CPU(s) not present in any
NUMA nodes: CPU 30 [socket-id: 15, core-id: 0, thread-id: 0], CPU 31
[socket-id: 15, core-id: 1, thread-id: 0]
2019-11-16T22:33:33.630439Z qemu-kvm: warning: All CPU(s) up to maxcpus
should be described in NUMA config, ability to start up with partial NUMA
mappings is obsoleted and will be removed in future
Nir
1814
days inactive
1815
days old
2 comments
3 participants
participants (3)
-
Klaas Demter -
klaasdemter＠gmail.com
-
Nir Soffer