numa error after upgrading from 3.5rc2 to 3.5rc3
Hello, after upgrading engine and hypervisor to rc3 I get this message when trying to start VM: VM vm_name is down with error. Exit message: internal error internal error NUMA memory tuning in 'preferred' mode only supports single node. hypervisor is Intel blade MFS5520VI with processor : 15 vendor_id : GenuineIntel cpu family : 6 model : 26 model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz stepping : 5 cpu MHz : 2927.000 cache size : 8192 KB physical id : 1 siblings : 8 core id : 3 cpu cores : 4 apicid : 23 initial apicid : 23 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dts tpr_shadow vnmi flexpriority ept vpid bogomips : 5851.73 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: Did anything change from rc2 to rc3 regarding this? Gianluca
On Thu, Sep 25, 2014 at 1:58 AM, Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
Hello, after upgrading engine and hypervisor to rc3 I get this message when trying to start VM:
VM vm_name is down with error. Exit message: internal error internal error NUMA memory tuning in 'preferred' mode only supports single node.
hypervisor is Intel blade MFS5520VI with
processor : 15 vendor_id : GenuineIntel cpu family : 6 model : 26 model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz stepping : 5 cpu MHz : 2927.000 cache size : 8192 KB physical id : 1 siblings : 8 core id : 3 cpu cores : 4 apicid : 23 initial apicid : 23 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dts tpr_shadow vnmi flexpriority ept vpid bogomips : 5851.73 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management:
Did anything change from rc2 to rc3 regarding this?
Gianluca
If it can be of any help, both engine and node are CentOS 6.5 and on node I have this output from numactl command: [root@ovnode04 vdsm]# numactl --show policy: default preferred node: current physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 cpubind: 0 1 nodebind: 0 1 membind: 0 1 [root@ovnode04 vdsm]# numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 2 4 6 8 10 12 14 node 0 size: 8128 MB node 0 free: 7567 MB node 1 cpus: 1 3 5 7 9 11 13 15 node 1 size: 8192 MB node 1 free: 7747 MB node distances: node 0 1 0: 10 21 1: 21 10 In engine.log: 2014-09-25 01:58:18,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-16) [21714fa2] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,pitReinjection=false,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=7200,transparentHugePages=true,vmId=0ce8ebc0-8464-4e9a-b382-1836234b3560,acpiEnable=true,custom={device_fc07dfb4-c33c-4302-915f-fcafcbaf201fdevice_814694f2-e3d9-4808-87d1-1576acbb6b39device_8e5365f1-1027-498e-9453-0289278434d4device_ef937c2a-c7ca-4405-98d8-72387b983ed7=VmDevice {vmId=0ce8ebc0-8464-4e9a-b382-1836234b3560, deviceId=ef937c2a-c7ca-4405-98d8-72387b983ed7, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={bus=0, controller=0, type=virtio-serial, port=3}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}, device_fc07dfb4-c33c-4302-915f-fcafcbaf201f=VmDevice {vmId=0ce8ebc0-8464-4e9a-b382-1836234b3560, deviceId=fc07dfb4-c33c-4302-915f-fcafcbaf201f, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}, device_fc07dfb4-c33c-4302-915f-fcafcbaf201fdevice_814694f2-e3d9-4808-87d1-1576acbb6b39device_8e5365f1-1027-498e-9453-0289278434d4=VmDevice {vmId=0ce8ebc0-8464-4e9a-b382-1836234b3560, deviceId=8e5365f1-1027-498e-9453-0289278434d4, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={bus=0, controller=0, type=virtio-serial, port=2}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}, device_fc07dfb4-c33c-4302-915f-fcafcbaf201fdevice_814694f2-e3d9-4808-87d1-1576acbb6b39=VmDevice {vmId=0ce8ebc0-8464-4e9a-b382-1836234b3560, deviceId=814694f2-e3d9-4808-87d1-1576acbb6b39, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={bus=0, controller=0, type=virtio-serial, port=1}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}},spiceSslCipherSuite=DEFAULT,memSize=2048,smp=2,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1365,display=qxl,smartcardEnable=false,bootMenuEnable=false,numaTune={mode=preferred, nodeset=0,1},spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,maxVCpus=16,devices=[{address={bus=0x00, domain=0x0000, type=pci, slot=0x02, function=0x0}, specParams={ram=65536, vram=32768, heads=1}, device=qxl, type=video, deviceId=544e064e-f2df-40c8-89e3-9311cc82af45}, {shared=false, iface=ide, index=2, address={unit=0, bus=1, target=0, controller=0, type=drive}, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=00fd82da-3da6-45c1-b18b-54e3ccb52c9b}, {shared=false, index=0, volumeID=72821a56-1806-4b8f-bd73-7b5ef4f3c12f, propagateErrors=off, format=cow, type=disk, iface=scsi, bootOrder=1, address={unit=0, bus=0, target=0, controller=0, type=drive}, domainID=346732c8-e1e9-49de-9ca4-63f4477ef6dd, imageID=f9c73080-8653-44bd-9bd9-acc7a9925971, specParams={}, optional=false, device=disk, poolID=4512e567-f94e-476a-a050-6cd0a15e260a, readonly=false, deviceId=f9c73080-8653-44bd-9bd9-acc7a9925971}, {nicModel=pv, address={bus=0x00, domain=0x0000, type=pci, slot=0x03, function=0x0}, specParams={outbound={}, inbound={}}, macAddr=00:01:a4:a3:42:01, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=2d8e61a1-a759-4115-9c38-081d32bb9d3f}, {specParams={}, device=console, type=console, deviceId=cf5798e1-f7f5-4c7e-b3e3-9903ad6b3b0a}, {address={bus=0x00, domain=0x0000, type=pci, slot=0x06, function=0x0}, specParams={model=virtio}, device=memballoon, type=balloon, deviceId=f9007dbf-da2b-41ce-b589-272af18ef860}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, type=pci, slot=0x04, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=49fe629a-fa54-4eba-9e95-14768b4bec4c}, {address={bus=0x00, domain=0x0000, type=pci, slot=0x05, function=0x0}, specParams={}, device=virtio-serial, type=controller, deviceId=3bf8fbed-9a33-4e25-b07e-356a54cacc86}],vmName=c65new,cpuType=Nehalem,fileTransferEnable=true 2014-09-25 01:58:18,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-16) [21714fa2] FINISH, CreateVDSCommand, log id: 8f7a8dd 2014-09-25 01:58:18,099 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-16) [21714fa2] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 78789fc9 2014-09-25 01:58:18,100 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-16) [21714fa2] Lock freed to object EngineLock [exclusiveLocks= key: 0ce8ebc0-8464-4e9a-b382-1836234b3560 value: VM , sharedLocks= ] 2014-09-25 01:58:18,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-16) [21714fa2] Correlation ID: 21714fa2, Job ID: 8fcd2749-9b69-4cfb-8b13-f3a7fb752895, Call Stack: null, Custom Event ID: -1, Message: VM c65new was started by admin (Host: ovnode04). 2014-09-25 01:58:19,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-1) START, DestroyVDSCommand(HostName = ovnode04, HostId = 36fec87b-c21f-4157-ab2f-434b67c05cb9, vmId=0ce8ebc0-8464-4e9a-b382-1836234b3560, force=false, secondsToWait=0, gracefully=false, reason=), log id: 5be4f2f8 2014-09-25 01:58:20,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-1) FINISH, DestroyVDSCommand, log id: 5be4f2f8 2014-09-25 01:58:20,071 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM c65new is down with error. Exit message: internal error internal error NUMA memory tuning in 'preferred' mode only supports single node. 2014-09-25 01:58:20,072 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) Running on vds during rerun failed vm: null 2014-09-25 01:58:20,072 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) VM c65new (0ce8ebc0-8464-4e9a-b382-1836234b3560) is running in db and not running in VDS ovnode04 2014-09-25 01:58:20,097 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) Rerun vm 0ce8ebc0-8464-4e9a-b382-1836234b3560. Called from vds ovnode04 2014-09-25 01:58:20,107 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) Correlation ID: 21714fa2, Job ID: 8fcd2749-9b69-4cfb-8b13-f3a7fb752895, Call Stack: null, Custom Event ID: -1, Message: Failed to run VM c65new on Host ovnode04.
In /var/log/libvirt/qemu/vm_name.log I find the information below BTW: how to eliminate 2hr offset in timestamp of libvrt logs? Time was 01:58 but in this file I see 23:58... 2014-09-24 23:58:20.162+0000: starting up LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=spice /usr/libexec/ qemu-kvm -name c65new -S -M rhel6.5.0 -cpu Nehalem -enable-kvm -m 2048 -realtime mlock=off -smp 2,maxcpus=16, sockets=16,cores=1,threads=1 -uuid 0ce8ebc0-8464-4e9a-b382-1836234b3560 -smbios type=1,manufacturer=oVirt,pro duct=oVirt Node,version=6-5.el6.centos.11.2,serial=9B1B2F84-3CB9-11DE-B5A1-001517B493E8,uuid=0ce8ebc0-8464-4e 9a-b382-1836234b3560 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/c65ne w.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2014-09-25T01:58:20,driftf ix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device vir tio-scsi-pci,id=scsi0,bus=pci.0,addr= 0x4 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/4512e567-f94e-476a-a050-6cd0a15e260a/346732c8-e1e9-49de-9ca4-63f4477ef6dd/images/f9c73080-8653-44bd-9bd9-acc7a9925971/72821a56-1806-4b8f-bd73-7b5ef4f3c12f,if=none,id=drive-scsi0-0-0-0,format=qcow2,serial=f9c73080-8653-44bd-9bd9-acc7a9925971,cache=none,werror=stop,rerror=stop,aio=native -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:01:a4:a3:42:01,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/0ce8ebc0-8464-4e9a-b382-1836234b3560.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/0ce8ebc0-8464-4e9a-b382-1836234b3560.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -chardev pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0 -spice port=5900,tls-port=5901,addr=192.168.1.74,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -k en-us -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=33554432 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 libvir: QEMU Driver error : internal error NUMA memory tuning in 'preferred' mode only supports single node 2014-09-24 23:58:20.262+0000: shutting down
On Thu, Sep 25, 2014 at 10:29:08AM +0200, Gianluca Cecchi wrote:
In /var/log/libvirt/qemu/vm_name.log I find the information below
BTW: how to eliminate 2hr offset in timestamp of libvrt logs? Time was 01:58 but in this file I see 23:58...
You can move to a country that uses UTC as its local timezone... Some time ago, libvirt started using UTC regardless of the local timezone. I think it's cool - everybody should use UTC everywhere, and Vdsm should follow suit.
2014-09-24 23:58:20.162+0000: starting up LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=spice /usr/libexec/ qemu-kvm -name c65new -S -M rhel6.5.0 -cpu Nehalem -enable-kvm -m 2048 -realtime mlock=off -smp 2,maxcpus=16, sockets=16,cores=1,threads=1 -uuid 0ce8ebc0-8464-4e9a-b382-1836234b3560 -smbios type=1,manufacturer=oVirt,pro duct=oVirt Node,version=6-5.el6.centos.11.2,serial=9B1B2F84-3CB9-11DE-B5A1-001517B493E8,uuid=0ce8ebc0-8464-4e 9a-b382-1836234b3560 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/c65ne w.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2014-09-25T01:58:20,driftf ix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device vir tio-scsi-pci,id=scsi0,bus=pci.0,addr= 0x4 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/4512e567-f94e-476a-a050-6cd0a15e260a/346732c8-e1e9-49de-9ca4-63f4477ef6dd/images/f9c73080-8653-44bd-9bd9-acc7a9925971/72821a56-1806-4b8f-bd73-7b5ef4f3c12f,if=none,id=drive-scsi0-0-0-0,format=qcow2,serial=f9c73080-8653-44bd-9bd9-acc7a9925971,cache=none,werror=stop,rerror=stop,aio=native -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:01:a4:a3:42:01,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/0ce8ebc0-8464-4e9a-b382-1836234b3560.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/0ce8ebc0-8464-4e9a-b382-1836234b3560.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -chardev pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0 -spice port=5900,tls-port=5901,addr=192.168.1.74,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -k en-us -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=33554432 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 libvir: QEMU Driver error : internal error NUMA memory tuning in 'preferred' mode only supports single node 2014-09-24 23:58:20.262+0000: shutting down
Clearly, there's a libvirt bug here. Maybe Michal and Martin (CC) can help with it. I suspect that your `virsh capabilities` and the domxml you're trying to start, would assist him.
On Thu, Sep 25, 2014 at 1:01 PM, Dan Kenigsberg <danken@redhat.com> wrote:
Clearly, there's a libvirt bug here. Maybe Michal and Martin (CC) can help with it. I suspect that your `virsh capabilities` and the domxml you're trying to start, would assist him.
at this moment I have no access through the gui and I'm experimenting wit rest api.... I have to always take opportunities out from problems;-) the command curl --insecure -u "admin@internal:my_password" -H "Content-type: application/xml" -X GET https://ovirtmgr/api/vms gives these lines at the end of the vm I'm using for test (actually all my 3 VMs have the same value for numa_tune_mode) ... <cpu_profile href="/api/cpuprofiles/11c7f359-180c-47ea-8948-5344fbeabded" id="11c7f359-180c-47ea-8948-5344fbeabded"/> <next_run_configuration_exists>false</next_run_configuration_exists> <numa_tune_mode>preferred</numa_tune_mode> </vm> that seems indeed related with my error about "preferred" ... Can I in any way change the "<numa_tune_mode>" parameter that the vm automatically took? WHat would be the rest api command if possible and what are the possible parameters I can give for numa_tune_mode? Or can I in any way delete this parameter for the VM? Gianluca
On Thu, Sep 25, 2014 at 1:38 PM, Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Thu, Sep 25, 2014 at 1:01 PM, Dan Kenigsberg <danken@redhat.com> wrote:
Clearly, there's a libvirt bug here. Maybe Michal and Martin (CC) can help with it. I suspect that your `virsh capabilities` and the domxml you're trying to start, would assist him.
BTW: host capabilities https://drive.google.com/file/d/0BwoPbcrMv8mvZlpTVU92ci1UTFU/edit?usp=sharin... vm generated xml https://drive.google.com/file/d/0BwoPbcrMv8mvMVNYLWFERHNaS3M/edit?usp=sharin... Gianluca
Hello, my suspect was correct. I found some reference here about possible values: http://www.ovirt.org/Features/Detailed_NUMA_and_Virtual_NUMA so I made this [root@ovirtmgr ~]# cat numa_tune.xml <vm><numa_tune_mode>interleave</numa_tune_mode></vm> [root@ovirtmgr ~]# curl --insecure -u "admin@internal:my_password" -H "Content-type: application/xml" -T numa_tune.xml ' https://ovirtmgr/api/vms/168470b1-b7eb-4dab-8fa4-6b744e2ad738' and now start of VM succeeds with [root@ovirtmgr ~]# curl --insecure --request POST -H "Content-type: application/xml" -u "admin@internal:my_password" --header "Accept: application/xml" --data '<action> </action>' " https://ovirtmgr/ovirt-engine/api/vms/168470b1-b7eb-4dab-8fa4-6b744e2ad738/s... " In qemu log file for the VM 2014-09-25 13:55:38.111+0000: starting up LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name ubuntutrusty -S -M rhel6.5.0 -cpu Nehalem -enable-kvm -m 4096 -realtime mlock=off -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 -uuid 168470b1-b7eb-4dab-8fa4-6b744e2ad738 -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=6-5.el6.centos.11.2,serial=9B1B2F84-3CB9-11DE-B5A1-001517B493E8,uuid=168470b1-b7eb-4dab-8fa4-6b744e2ad738 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/ubuntutrusty.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2014-09-25T15:55:37,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x5 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x6 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/4512e567-f94e-476a-a050-6cd0a15e260a/346732c8-e1e9-49de-9ca4-63f4477ef6dd/images/d690177e-eca6-4dd2-9380-9d2d49c35b68/cc46a780-ea50-4a71-8acc-963f703e7321,if=none,id=drive-virtio-disk0,format=qcow2,serial=d690177e-eca6-4dd2-9380-9d2d49c35b68,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:01:a4:a3:42:02,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/168470b1-b7eb-4dab-8fa4-6b744e2ad738.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/168470b1-b7eb-4dab-8fa4-6b744e2ad738.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice port=5900,tls-port=5901,addr=192.168.1.74,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -k en-us -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=33554432 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 The question now is how can I influence configuration from the GUI so that all my VMs are by default created as interleave. I will also try to create VM from scratch in 3.5rc3 to find if the problem could arise only for pre-existing VMs or not... Comments welcome on my findings.... Gianluca
Hi Gianluca, I see the numa mode select box in the Host sub-tab of the VM edit dialog. It was committed three days ago to the 3.5 branch so it should be available in the 3.5 GA release I think. I am not exactly sure if it made the RC3 build though. http://gerrit.ovirt.org/#/q/I05b83028b722088e39ad7156c77ad4eb479dc241,n,z Regards -- Martin Sivák msivak@redhat.com Red Hat Czech RHEV-M SLA / Brno, CZ ----- Original Message -----
Hello, my suspect was correct. I found some reference here about possible values: http://www.ovirt.org/Features/Detailed_NUMA_and_Virtual_NUMA
so I made this [root@ovirtmgr ~]# cat numa_tune.xml <vm><numa_tune_mode>interleave</numa_tune_mode></vm>
[root@ovirtmgr ~]# curl --insecure -u "admin@internal:my_password" -H "Content-type: application/xml" -T numa_tune.xml ' https://ovirtmgr/api/vms/168470b1-b7eb-4dab-8fa4-6b744e2ad738'
and now start of VM succeeds with [root@ovirtmgr ~]# curl --insecure --request POST -H "Content-type: application/xml" -u "admin@internal:my_password" --header "Accept: application/xml" --data '<action> </action>' " https://ovirtmgr/ovirt-engine/api/vms/168470b1-b7eb-4dab-8fa4-6b744e2ad738/s... "
In qemu log file for the VM
2014-09-25 13:55:38.111+0000: starting up LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name ubuntutrusty -S -M rhel6.5.0 -cpu Nehalem -enable-kvm -m 4096 -realtime mlock=off -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 -uuid 168470b1-b7eb-4dab-8fa4-6b744e2ad738 -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=6-5.el6.centos.11.2,serial=9B1B2F84-3CB9-11DE-B5A1-001517B493E8,uuid=168470b1-b7eb-4dab-8fa4-6b744e2ad738 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/ubuntutrusty.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2014-09-25T15:55:37,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x5 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x6 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/4512e567-f94e-476a-a050-6cd0a15e260a/346732c8-e1e9-49de-9ca4-63f4477ef6dd/images/d690177e-eca6-4dd2-9380-9d2d49c35b68/cc46a780-ea50-4a71-8acc-963f703e7321,if=none,id=drive-virtio-disk0,format=qcow2,serial=d690177e-eca6-4dd2-9380-9d2d49c35b68,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:01:a4:a3:42:02,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/168470b1-b7eb-4dab-8fa4-6b744e2ad738.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/168470b1-b7eb-4dab-8fa4-6b744e2ad738.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice port=5900,tls-port=5901,addr=192.168.1.74,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -k en-us -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=33554432 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8
The question now is how can I influence configuration from the GUI so that all my VMs are by default created as interleave.
I will also try to create VM from scratch in 3.5rc3 to find if the problem could arise only for pre-existing VMs or not...
Comments welcome on my findings....
Gianluca
On Thu, Sep 25, 2014 at 8:38 PM, Martin Sivak <msivak@redhat.com> wrote:
Hi Gianluca,
I see the numa mode select box in the Host sub-tab of the VM edit dialog.
It was committed three days ago to the 3.5 branch so it should be available in the 3.5 GA release I think. I am not exactly sure if it made the RC3 build though.
http://gerrit.ovirt.org/#/q/I05b83028b722088e39ad7156c77ad4eb479dc241,n,z
Strange. For my hypervisor it represents a regression. No VM is able to start with default parameters. I can access the GUI again and it seems I cannot change the parameter. See here screenshot: https://drive.google.com/file/d/0BwoPbcrMv8mvTTZCb3RTRUdFcGc/edit?usp=sharin... For the vm where I changed the setting through Rest API I see the field greyed too, but the value inside is "interleave" and not "preferred" Is there any engine-config option to change default? Gianluca
On Thu, Sep 25, 2014 at 10:52 PM, Gianluca Cecchi <gianluca.cecchi@gmail.com
wrote:
On Thu, Sep 25, 2014 at 8:38 PM, Martin Sivak <msivak@redhat.com> wrote:
Hi Gianluca,
I see the numa mode select box in the Host sub-tab of the VM edit dialog.
It was committed three days ago to the 3.5 branch so it should be available in the 3.5 GA release I think. I am not exactly sure if it made the RC3 build though.
http://gerrit.ovirt.org/#/q/I05b83028b722088e39ad7156c77ad4eb479dc241,n,z
Strange. For my hypervisor it represents a regression. No VM is able to start with default parameters. I can access the GUI again and it seems I cannot change the parameter. See here screenshot:
https://drive.google.com/file/d/0BwoPbcrMv8mvTTZCb3RTRUdFcGc/edit?usp=sharin...
For the vm where I changed the setting through Rest API I see the field greyed too, but the value inside is "interleave" and not "preferred"
Is there any engine-config option to change default?
Gianluca
Hello, so on engine I have ovirt-engine-3.5.0-0.0.master.20140923231936.git42065cc.el6.noarch and on node I have vdsm-4.16.5-0.el6.x86_64 libvirt-0.10.2-29.el6_5.12.x86_64 They should be what in 3.5rc3 New VMs created in 3.5rc3 are ok in the sense that I verified that numa_tune_mode is populated by default with "interleave" value that my node is able to manage to correctly start VMs. But all the 3 VMs that was previously created in 3.5rc2 had numa_tune_mode=preferred that lets them unable to be started with the specified error. I don't know if this is a problem only because they were created in 3.5rc2 or if this problem could influence also oVirt 3.4 infras that are upgrading to 3.5 and so a critical regression. Gianluca
Hi, I think the default was changed to INTERLEAVE, because that is the value I see everywhere as preselected in the current 3.5 code. This is the logic that governs if the select boxes are enabled: if (getModel().getMigrationMode().getSelectedItem() != MigrationSupport.PINNED_TO_HOST || getModel().getIsAutoAssign().getEntity() || getModel().getDefaultHost().getSelectedItem() == null || !getModel().getDefaultHost().getSelectedItem().isNumaSupport()) { enabled = false; } So it seems that the following conditions have to be satisfied for the select boxes to be enabled: The VM has to be pinned to a NUMA host (check that please, it might allow you to update the values using UI). Have the default host set (probably satisfied by the previous condition) The host has to support NUMA I am adding Gilad to CC as he knows the code. Unfortunately he will be available some time next week as they have a holiday season in Israel atm. Regards -- Martin Sivák msivak@redhat.com Red Hat Czech RHEV-M SLA / Brno, CZ ----- Original Message -----
On Thu, Sep 25, 2014 at 10:52 PM, Gianluca Cecchi <gianluca.cecchi@gmail.com
wrote:
On Thu, Sep 25, 2014 at 8:38 PM, Martin Sivak <msivak@redhat.com> wrote:
Hi Gianluca,
I see the numa mode select box in the Host sub-tab of the VM edit dialog.
It was committed three days ago to the 3.5 branch so it should be available in the 3.5 GA release I think. I am not exactly sure if it made the RC3 build though.
http://gerrit.ovirt.org/#/q/I05b83028b722088e39ad7156c77ad4eb479dc241,n,z
Strange. For my hypervisor it represents a regression. No VM is able to start with default parameters. I can access the GUI again and it seems I cannot change the parameter. See here screenshot:
https://drive.google.com/file/d/0BwoPbcrMv8mvTTZCb3RTRUdFcGc/edit?usp=sharin...
For the vm where I changed the setting through Rest API I see the field greyed too, but the value inside is "interleave" and not "preferred"
Is there any engine-config option to change default?
Gianluca
Hello, so on engine I have ovirt-engine-3.5.0-0.0.master.20140923231936.git42065cc.el6.noarch and on node I have vdsm-4.16.5-0.el6.x86_64 libvirt-0.10.2-29.el6_5.12.x86_64 They should be what in 3.5rc3 New VMs created in 3.5rc3 are ok in the sense that I verified that numa_tune_mode is populated by default with "interleave" value that my node is able to manage to correctly start VMs.
But all the 3 VMs that was previously created in 3.5rc2 had numa_tune_mode=preferred that lets them unable to be started with the specified error.
I don't know if this is a problem only because they were created in 3.5rc2 or if this problem could influence also oVirt 3.4 infras that are upgrading to 3.5 and so a critical regression.
Gianluca
On Fri, Sep 26, 2014 at 10:56 AM, Martin Sivak <msivak@redhat.com> wrote:
Hi,
I think the default was changed to INTERLEAVE, because that is the value I see everywhere as preselected in the current 3.5 code.
This is the logic that governs if the select boxes are enabled:
if (getModel().getMigrationMode().getSelectedItem() != MigrationSupport.PINNED_TO_HOST || getModel().getIsAutoAssign().getEntity() || getModel().getDefaultHost().getSelectedItem() == null || !getModel().getDefaultHost().getSelectedItem().isNumaSupport()) { enabled = false; }
So it seems that the following conditions have to be satisfied for the select boxes to be enabled:
The VM has to be pinned to a NUMA host (check that please, it might allow you to update the values using UI). Have the default host set (probably satisfied by the previous condition) The host has to support NUMA
I am adding Gilad to CC as he knows the code. Unfortunately he will be available some time next week as they have a holiday season in Israel atm.
The point here is: If I am on oVirt 3.4.x environment, is this numa_tune_mode parameter present in the VM definition? I have not at hand now a 3.4 system so I cannot check. If not present at all in pre-3.5 VMs and I upgrade my environment, what would be done for my pre-existing VMs? Because I verified (not from 3.5rc3 scratch installation but upgrading from 3.5rc2) that for new VMs indeed the parameter is set with "interleave" as default. I don't know if inside the sql upgrade scripts there is something like if current < 3.5 then add parameter and set to interleave So in this case it would make the correct thing for 3.4 environments, but as a corner effect it would do nothing for my 3.5.rc2 environment, that still didn't have the parameter, causing the problem I'm experiencing.... Gianluca
----- Original Message -----
From: "Gianluca Cecchi" <gianluca.cecchi@gmail.com> To: "Martin Sivak" <msivak@redhat.com> Cc: "Dan Kenigsberg" <danken@redhat.com>, "users" <users@ovirt.org>, mprivozn@redhat.com, "Gilad Chaplik" <gchaplik@redhat.com> Sent: Friday, September 26, 2014 12:11:09 PM Subject: Re: [ovirt-users] numa error after upgrading from 3.5rc2 to 3.5rc3
On Fri, Sep 26, 2014 at 10:56 AM, Martin Sivak <msivak@redhat.com> wrote:
Hi,
I think the default was changed to INTERLEAVE, because that is the value I see everywhere as preselected in the current 3.5 code.
This is the logic that governs if the select boxes are enabled:
if (getModel().getMigrationMode().getSelectedItem() != MigrationSupport.PINNED_TO_HOST || getModel().getIsAutoAssign().getEntity() || getModel().getDefaultHost().getSelectedItem() == null || !getModel().getDefaultHost().getSelectedItem().isNumaSupport()) { enabled = false; }
So it seems that the following conditions have to be satisfied for the select boxes to be enabled:
The VM has to be pinned to a NUMA host (check that please, it might allow you to update the values using UI). Have the default host set (probably satisfied by the previous condition) The host has to support NUMA
I am adding Gilad to CC as he knows the code. Unfortunately he will be available some time next week as they have a holiday season in Israel atm.
The point here is: If I am on oVirt 3.4.x environment, is this numa_tune_mode parameter present in the VM definition? I have not at hand now a 3.4 system so I cannot check. If not present at all in pre-3.5 VMs and I upgrade my environment, what would be done for my pre-existing VMs? Because I verified (not from 3.5rc3 scratch installation but upgrading from 3.5rc2) that for new VMs indeed the parameter is set with "interleave" as default. I don't know if inside the sql upgrade scripts there is something like
if current < 3.5 then add parameter and set to interleave
So in this case it would make the correct thing for 3.4 environments, but as a corner effect it would do nothing for my 3.5.rc2 environment, that still didn't have the parameter, causing the problem I'm experiencing....
you're right, I'm changing default value to interleave. for now run the query: update vm_static set numatune_mode ='interleave';
Gianluca
participants (4)
-
Dan Kenigsberg -
Gianluca Cecchi -
Gilad Chaplik -
Martin Sivak