On Thu, Sep 25, 2014 at 1:58 AM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: If it can be of any help, both engine and node are CentOS 6.5 and on node I have this output from numactl command: [root@ovnode04 vdsm]# numactl --show policy: default preferred node: current physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 cpubind: 0 1 nodebind: 0 1 membind: 0 1 [root@ovnode04 vdsm]# numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 2 4 6 8 10 12 14 node 0 size: 8128 MB node 0 free: 7567 MB node 1 cpus: 1 3 5 7 9 11 13 15 node 1 size: 8192 MB node 1 free: 7747 MB node distances: node 0 1 0: 10 21 1: 21 10 In engine.log: 2014-09-25 01:58:18,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-16) [21714fa2] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,pitReinjection=false,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=7200,transparentHugePages=true,vmId=0ce8ebc0-8464-4e9a-b382-1836234b3560,acpiEnable=true,custom={device_fc07dfb4-c33c-4302-915f-fcafcbaf201fdevice_814694f2-e3d9-4808-87d1-1576acbb6b39device_8e5365f1-1027-498e-9453-0289278434d4device_ef937c2a-c7ca-4405-98d8-72387b983ed7=VmDevice {vmId=0ce8ebc0-8464-4e9a-b382-1836234b3560, deviceId=ef937c2a-c7ca-4405-98d8-72387b983ed7, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={bus=0, controller=0, type=virtio-serial, port=3}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}, device_fc07dfb4-c33c-4302-915f-fcafcbaf201f=VmDevice {vmId=0ce8ebc0-8464-4e9a-b382-1836234b3560, deviceId=fc07dfb4-c33c-4302-915f-fcafcbaf201f, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}, device_fc07dfb4-c33c-4302-915f-fcafcbaf201fdevice_814694f2-e3d9-4808-87d1-1576acbb6b39device_8e5365f1-1027-498e-9453-0289278434d4=VmDevice {vmId=0ce8ebc0-8464-4e9a-b382-1836234b3560, deviceId=8e5365f1-1027-498e-9453-0289278434d4, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={bus=0, controller=0, type=virtio-serial, port=2}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}, device_fc07dfb4-c33c-4302-915f-fcafcbaf201fdevice_814694f2-e3d9-4808-87d1-1576acbb6b39=VmDevice {vmId=0ce8ebc0-8464-4e9a-b382-1836234b3560, deviceId=814694f2-e3d9-4808-87d1-1576acbb6b39, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={bus=0, controller=0, type=virtio-serial, port=1}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}},spiceSslCipherSuite=DEFAULT,memSize=2048,smp=2,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1365,display=qxl,smartcardEnable=false,bootMenuEnable=false,numaTune={mode=preferred, nodeset=0,1},spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,maxVCpus=16,devices=[{address={bus=0x00, domain=0x0000, type=pci, slot=0x02, function=0x0}, specParams={ram=65536, vram=32768, heads=1}, device=qxl, type=video, deviceId=544e064e-f2df-40c8-89e3-9311cc82af45}, {shared=false, iface=ide, index=2, address={unit=0, bus=1, target=0, controller=0, type=drive}, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=00fd82da-3da6-45c1-b18b-54e3ccb52c9b}, {shared=false, index=0, volumeID=72821a56-1806-4b8f-bd73-7b5ef4f3c12f, propagateErrors=off, format=cow, type=disk, iface=scsi, bootOrder=1, address={unit=0, bus=0, target=0, controller=0, type=drive}, domainID=346732c8-e1e9-49de-9ca4-63f4477ef6dd, imageID=f9c73080-8653-44bd-9bd9-acc7a9925971, specParams={}, optional=false, device=disk, poolID=4512e567-f94e-476a-a050-6cd0a15e260a, readonly=false, deviceId=f9c73080-8653-44bd-9bd9-acc7a9925971}, {nicModel=pv, address={bus=0x00, domain=0x0000, type=pci, slot=0x03, function=0x0}, specParams={outbound={}, inbound={}}, macAddr=00:01:a4:a3:42:01, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=ovirtmgmt, deviceId=2d8e61a1-a759-4115-9c38-081d32bb9d3f}, {specParams={}, device=console, type=console, deviceId=cf5798e1-f7f5-4c7e-b3e3-9903ad6b3b0a}, {address={bus=0x00, domain=0x0000, type=pci, slot=0x06, function=0x0}, specParams={model=virtio}, device=memballoon, type=balloon, deviceId=f9007dbf-da2b-41ce-b589-272af18ef860}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, type=pci, slot=0x04, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=49fe629a-fa54-4eba-9e95-14768b4bec4c}, {address={bus=0x00, domain=0x0000, type=pci, slot=0x05, function=0x0}, specParams={}, device=virtio-serial, type=controller, deviceId=3bf8fbed-9a33-4e25-b07e-356a54cacc86}],vmName=c65new,cpuType=Nehalem,fileTransferEnable=true 2014-09-25 01:58:18,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-16) [21714fa2] FINISH, CreateVDSCommand, log id: 8f7a8dd 2014-09-25 01:58:18,099 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-16) [21714fa2] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 78789fc9 2014-09-25 01:58:18,100 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-16) [21714fa2] Lock freed to object EngineLock [exclusiveLocks= key: 0ce8ebc0-8464-4e9a-b382-1836234b3560 value: VM , sharedLocks= ] 2014-09-25 01:58:18,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-16) [21714fa2] Correlation ID: 21714fa2, Job ID: 8fcd2749-9b69-4cfb-8b13-f3a7fb752895, Call Stack: null, Custom Event ID: -1, Message: VM c65new was started by admin (Host: ovnode04). 2014-09-25 01:58:19,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-1) START, DestroyVDSCommand(HostName = ovnode04, HostId = 36fec87b-c21f-4157-ab2f-434b67c05cb9, vmId=0ce8ebc0-8464-4e9a-b382-1836234b3560, force=false, secondsToWait=0, gracefully=false, reason=), log id: 5be4f2f8 2014-09-25 01:58:20,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-1) FINISH, DestroyVDSCommand, log id: 5be4f2f8 2014-09-25 01:58:20,071 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM c65new is down with error. Exit message: internal error internal error NUMA memory tuning in 'preferred' mode only supports single node. 2014-09-25 01:58:20,072 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) Running on vds during rerun failed vm: null 2014-09-25 01:58:20,072 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) VM c65new (0ce8ebc0-8464-4e9a-b382-1836234b3560) is running in db and not running in VDS ovnode04 2014-09-25 01:58:20,097 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) Rerun vm 0ce8ebc0-8464-4e9a-b382-1836234b3560. Called from vds ovnode04 2014-09-25 01:58:20,107 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) Correlation ID: 21714fa2, Job ID: 8fcd2749-9b69-4cfb-8b13-f3a7fb752895, Call Stack: null, Custom Event ID: -1, Message: Failed to run VM c65new on Host ovnode04.

On Thu, Sep 25, 2014 at 10:29:08AM +0200, Gianluca Cecchi wrote: You can move to a country that uses UTC as its local timezone... Some time ago, libvirt started using UTC regardless of the local timezone. I think it's cool - everybody should use UTC everywhere, and Vdsm should follow suit. Clearly, there's a libvirt bug here. Maybe Michal and Martin (CC) can help with it. I suspect that your `virsh capabilities` and the domxml you're trying to start, would assist him.

On Thu, Sep 25, 2014 at 1:38 PM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: BTW: host capabilities https://drive.google.com/file/d/0BwoPbcrMv8mvZlpTVU92ci1UTFU/edit?usp=sha... vm generated xml https://drive.google.com/file/d/0BwoPbcrMv8mvMVNYLWFERHNaS3M/edit?usp=sha... Gianluca

Hi Gianluca, I see the numa mode select box in the Host sub-tab of the VM edit dialog. It was committed three days ago to the 3.5 branch so it should be available in the 3.5 GA release I think. I am not exactly sure if it made the RC3 build though. http://gerrit.ovirt.org/#/q/I05b83028b722088e39ad7156c77ad4eb479dc241,n,z Regards -- Martin Sivák msivak(a)redhat.com Red Hat Czech RHEV-M SLA / Brno, CZ ----- Original Message -----

On Thu, Sep 25, 2014 at 10:52 PM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com > On Thu, Sep 25, 2014 at 8:38 PM, Martin Sivak <msivak(a)redhat.com> >> Hi Gianluca, >> >> I see the numa mode select box in the Host sub-tab of the VM edit dialog. >> >> It was committed three days ago to the 3.5 branch so it should be >> available in the 3.5 GA release I think. I am not exactly sure if it made >> the RC3 build though. >> >> http://gerrit.ovirt.org/#/q/I05b83028b722088e39ad7156c77ad4eb479dc241,n,z >> > > Strange. > For my hypervisor it represents a regression. No VM is able to start with > default parameters. > I can access the GUI again and it seems I cannot change the parameter. > See here screenshot: > > https://drive.google.com/file/d/0BwoPbcrMv8mvTTZCb3RTRUdFcGc/edit?usp=sha... > > For the vm where I changed the setting through Rest API I see the field > greyed too, but the value inside is "interleave" and not "preferred" > > Is there any engine-config option to change default? > > Gianluca > > Hello, so on engine I have ovirt-engine-3.5.0-0.0.master.20140923231936.git42065cc.el6.noarch and on node I have vdsm-4.16.5-0.el6.x86_64 libvirt-0.10.2-29.el6_5.12.x86_64 They should be what in 3.5rc3 New VMs created in 3.5rc3 are ok in the sense that I verified that numa_tune_mode is populated by default with "interleave" value that my node is able to manage to correctly start VMs. But all the 3 VMs that was previously created in 3.5rc2 had numa_tune_mode=preferred that lets them unable to be started with the specified error. I don't know if this is a problem only because they were created in 3.5rc2 or if this problem could influence also oVirt 3.4 infras that are upgrading to 3.5 and so a critical regression. Gianluca

On Fri, Sep 26, 2014 at 10:56 AM, Martin Sivak <msivak(a)redhat.com&gt; wrote: The point here is: If I am on oVirt 3.4.x environment, is this numa_tune_mode parameter present in the VM definition? I have not at hand now a 3.4 system so I cannot check. If not present at all in pre-3.5 VMs and I upgrade my environment, what would be done for my pre-existing VMs? Because I verified (not from 3.5rc3 scratch installation but upgrading from 3.5rc2) that for new VMs indeed the parameter is set with "interleave" as default. I don't know if inside the sql upgrade scripts there is something like if current < 3.5 then add parameter and set to interleave So in this case it would make the correct thing for 3.4 environments, but as a corner effect it would do nothing for my 3.5.rc2 environment, that still didn't have the parameter, causing the problem I'm experiencing.... Gianluca