2:08 a.m.
Hi,
Thanks for the quick response.
On Thu, Dec 5, 2019, 01:10 Nir Soffer <nsoffer(a)redhat.com> wrote:
On Wed, Dec 4, 2019 at 6:15 PM <klaasdemter(a)gmail.com> wrote:
>
> Hi,
>
> I'm having performance issues with a ovirt installation. It is showing
> high steal (5-10%) for a cpu intensive VM. The hypervisor however has
> more than 65% of his resources idle while the steal is seen inside of
> the VM.
>
> Even when placing only a single VM on a hypervisor it still receives
> steal (0-2%), even though the hypervisor is not overcommited.
>
>
> Hypervisor:
>
> 2 Socket system in total 2*28(56HT) cores
>
>
> VM:
>
> 30vCPUs (ovirt seems to think its a good idea to make that 15 sockets *
> 2 cores)
I think you can control this in oVirt.
Yes, I'm saying ovirt defaults to using 15 sockets * 2 cores if you don't
specify a different "advanced" configuration. This also leads to a funny
behavior that you can't increase the VMs from 30 to 35 cores online.
> My questions are:
>
> a) Could it be that the hypervisor is trying to schedule all 30 cores on
> a single numa node, ie using the HT cores instead of "real" ones and
> this shows up as steal?
>
> b) Do I need to make VMs this big numa-aware and spread the vm over both
> numa nodes?
>
> c) Would using the High Performance VM type help in this kind of
situation?
>
> d) General advise: how do I reduce steal in an environment where the
> hypervisor has idle resources
>
>
> Any advise would be appreciated.
These questions are mainly about qemu, so adding qemu-discuss.
I think it will help if you share your vm qemu command line, found in:
/var/log/libvit/qemu/vm-name.log
Log:
2019-11-16 22:33:33.487+0000: starting up libvirt version: 4.5.0, package:
10.el7_6.6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>;,
2019-02-21-14:30:01, x86-038.build.eng.bos.redhat.com), qemu version:
2.12.0qemu-kvm-rhev-2.12.0-18.el7_6.3, kernel: 3.10.0-957.10.1.el7.x86_64,
hostname: hypervisor.tld
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name guest=vm,debug-threads=on
-S -object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-24-vm/master-key.aes
-machine pc-i440fx-rhel7.5.0,accel=kvm,usb=off,dump-guest-core=off -cpu
Broadwell,spec-ctrl=on,ssbd=on,rtm=on,hle=on -m
size=534773760k,slots=16,maxmem=629145600k -realtime mlock=off -smp
30,maxcpus=32,sockets=16,cores=2,threads=1 -object iothread,id=iothread1
-numa node,nodeid=0,cpus=0-29,mem=522240 -uuid
95285008-8efb-446c-b75c-771f020cf494 -smbios
'type=1,manufacturer=oVirt,product=RHEV
Hypervisor,version=7.6-4.el7,serial=4c4c4544-004b-3510-805a-c3c04f485032,uuid=95285008-8efb-446c-b75c-771f020cf494'
-no-user-config -nodefaults -chardev
socket,id=charmonitor,fd=51,server,nowait -mon
chardev=charmonitor,id=monitor,mode=control -rtc
base=2019-11-16T22:33:33,driftfix=slew -global
kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global
PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
virtio-scsi-pci,iothread=iothread1,id=ua-98c9be1a-c830-4718-9936-28a81e9a07b6,bus=pci.0,addr=0x5
-device
virtio-serial-pci,id=ua-3f6418a3-8b1a-49c3-b79b-ca5f1864cf47,max_ports=16,bus=pci.0,addr=0x4
-drive
if=none,id=drive-ua-641b4623-d2b3-4f7f-a31d-65959f5ffc9b,werror=report,rerror=report,readonly=on
-device
ide-cd,bus=ide.1,unit=0,drive=drive-ua-641b4623-d2b3-4f7f-a31d-65959f5ffc9b,id=ua-641b4623-d2b3-4f7f-a31d-65959f5ffc9b
-drive
file=/rhev/data-center/mnt/storage-domain/0f623c51-66c8-4a5b-a12f-95ab9c6184b4/images/88901d29-e223-457d-a80e-384b1f147cea/0f994c76-d525-4a4f-9513-c6220fe7c31c,format=raw,if=none,id=drive-ua-88901d29-e223-457d-a80e-384b1f147cea,serial=88901d29-e223-457d-a80e-384b1f147cea,werror=stop,rerror=stop,cache=none,aio=threads
-device
scsi-hd,bus=ua-98c9be1a-c830-4718-9936-28a81e9a07b6.0,channel=0,scsi-id=0,lun=0,drive=drive-ua-88901d29-e223-457d-a80e-384b1f147cea,id=ua-88901d29-e223-457d-a80e-384b1f147cea,bootindex=1,write-cache=on
-netdev
tap,fds=54:55:56:57,id=hostua-1fad81fe-8c0b-41dd-8015-7498ebbc2710,vhost=on,vhostfds=58:59:60:61
-device
virtio-net-pci,mq=on,vectors=10,host_mtu=1500,netdev=hostua-1fad81fe-8c0b-41dd-8015-7498ebbc2710,id=ua-1fad81fe-8c0b-41dd-8015-7498ebbc2710,mac=22:44:66:00:00:d0,bus=pci.0,addr=0x3
-chardev socket,id=charchannel0,fd=62,server,nowait -device
virtserialport,bus=ua-3f6418a3-8b1a-49c3-b79b-ca5f1864cf47.0,nr=1,chardev=charchannel0,id=channel0,name=ovirt-guest-agent.0
-chardev socket,id=charchannel1,fd=63,server,nowait -device
virtserialport,bus=ua-3f6418a3-8b1a-49c3-b79b-ca5f1864cf47.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0
-chardev spicevmc,id=charchannel2,name=vdagent -device
virtserialport,bus=ua-3f6418a3-8b1a-49c3-b79b-ca5f1864cf47.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0
-spice
port=5918,tls-port=5919,addr=10.160.69.12,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on
-device
qxl-vga,id=ua-a3bde96a-2165-434e-80f5-5980dbb9f19d,ram_size=67108864,vram_size=33554432,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2
-device
virtio-balloon-pci,id=ua-124c48f3-6bc4-453c-b0fb-107060bbc1f6,bus=pci.0,addr=0x6
-object
rng-random,id=objua-ff7eb44a-85a1-4731-aec0-0b85b16c2644,filename=/dev/urandom
-device
virtio-rng-pci,rng=objua-ff7eb44a-85a1-4731-aec0-0b85b16c2644,id=ua-ff7eb44a-85a1-4731-aec0-0b85b16c2644,bus=pci.0,addr=0x7
-device vmcoreinfo -sandbox
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny
-msg timestamp=on
2019-11-16T22:33:33.624932Z qemu-kvm: -drive
file=/rhev/data-center/mnt/storage-domain/0f623c51-66c8-4a5b-a12f-95ab9c6184b4/images/88901d29-e223-457d-a80e-384b1f147cea/0f994c76-d525-4a4f-9513-c6220fe7c31c,format=raw,if=none,id=drive-ua-88901d29-e223-457d-a80e-384b1f147cea,serial=88901d29-e223-457d-a80e-384b1f147cea,werror=stop,rerror=stop,cache=none,aio=threads:
'serial' is deprecated, please use the corresponding option of '-device'
instead
2019-11-16T22:33:33.630420Z qemu-kvm: warning: CPU(s) not present in any
NUMA nodes: CPU 30 [socket-id: 15, core-id: 0, thread-id: 0], CPU 31
[socket-id: 15, core-id: 1, thread-id: 0]
2019-11-16T22:33:33.630439Z qemu-kvm: warning: All CPU(s) up to maxcpus
should be described in NUMA config, ability to start up with partial NUMA
mappings is obsoleted and will be removed in future
Nir