[ovirt-users] Can't start VMs (Unable to get volume size for domain)

Mon Jan 4 12:06:41 EST 2016

I’m running 3.6.2 rc1 with hosted engine on an FCP storage domain.

As of yesterday, I can’t run some VMs. I’ve experience corruption on others (I now have a Windows VM that blue screens on boot).

Here’s the log from my engine.

2016-01-04 16:55:39,446 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (default task-16) [1f1deb62] Lock Acquired to object 'EngineLock:{exclusiveLocks='[3a17534b-e86d-4563-8ca2-2a27c34b4a87=<VM, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2016-01-04 16:55:39,479 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-16) [1f1deb62] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='3a17534b-e86d-4563-8ca2-2a27c34b4a87'}), log id: 299a5052
2016-01-04 16:55:39,479 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-16) [1f1deb62] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 299a5052
2016-01-04 16:55:39,517 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-40) [1f1deb62] Running command: RunVmCommand internal: false. Entities affected :  ID: 3a17534b-e86d-4563-8ca2-2a27c34b4a87 Type: VMAction group RUN_VM with role type USER
2016-01-04 16:55:39,579 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-40) [1f1deb62] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='00000000-0000-0000-0000-000000000000', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic at dadddaa9'}), log id: 6574710a
2016-01-04 16:55:39,582 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-40) [1f1deb62] FINISH, UpdateVmDynamicDataVDSCommand, log id: 6574710a
2016-01-04 16:55:39,585 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-40) [1f1deb62] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='2fe6c27b-9346-4678-8cd3-c9d367ec447f', vmId='3a17534b-e86d-4563-8ca2-2a27c34b4a87', vm='VM [adm1]'}), log id: 55e0849d
2016-01-04 16:55:39,586 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-40) [1f1deb62] START, CreateVDSCommand(HostName = ov-101, CreateVmVDSCommandParameters:{runAsync='true', hostId='2fe6c27b-9346-4678-8cd3-c9d367ec447f', vmId='3a17534b-e86d-4563-8ca2-2a27c34b4a87', vm='VM [adm1]'}), log id: 1d5c1c04
2016-01-04 16:55:39,589 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.VmInfoBuilderBase] (org.ovirt.thread.pool-8-thread-40) [1f1deb62] Bootable disk '9e43c66a-5bf1-44d6-94f4-52178d15c1e6' set to index '0'
2016-01-04 16:55:39,600 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-40) [1f1deb62] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand pitReinjection=false,memGuaranteedSize=4054,smpThreadsPerCore=1,cpuType=SandyBridge,vmId=3a17534b-e86d-4563-8ca2-2a27c34b4a87,acpiEnable=true,numaTune={nodeset=0,1, mode=interleave},tabletEnable=true,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,vmType=kvm,keyboardLayout=en-us,smp=1,smpCoresPerSocket=1,emulatedMachine=pc-i440fx-rhel7.2.0,smartcardEnable=false,guestNumaNodes=[{memory=4054, cpus=0, nodeIndex=0}],transparentHugePages=true,vmName=adm1,maxVCpus=16,kvmEnable=true,devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=cirrus, deviceId=645e99e3-a9fa-4894-baf5-97b539236782}, {type=graphics, specParams={}, device=vnc, deviceId=12845c03-16a3-4bf0-a015-a15201a77673}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=ab048396-5dd8-4594-aa8a-9fe835a04cd1}, {shared=false, address={bus=0, controller=0, unit=0, type=drive, target=0}, imageID=9e43c66a-5bf1-44d6-94f4-52178d15c1e6, format=raw, index=0, optional=false, type=disk, deviceId=9e43c66a-5bf1-44d6-94f4-52178d15c1e6, domainID=1fb79d91-b245-4447-91e0-e57671152a8c, propagateErrors=off, iface=ide, readonly=false, bootOrder=1, poolID=00000001-0001-0001-0001-000000000154, volumeID=c736baca-de76-4593-b3dc-28bb8807e7a3, specParams={}, device=disk}, {shared=false, address={bus=0, controller=0, unit=1, type=drive, target=0}, imageID=a016b350-87ef-4c3b-b150-024907fed9c0, format=raw, optional=false, type=disk, deviceId=a016b350-87ef-4c3b-b150-024907fed9c0, domainID=1fb79d91-b245-4447-91e0-e57671152a8c, propagateErrors=off, iface=ide, readonly=false, poolID=00000001-0001-0001-0001-000000000154, volumeID=20fc4399-0b02-4da1-8aee-68df1629ca94, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=rtl8139, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=8e00d4cc-6a60-4598-82ee-645d742708de, macAddr=FA:0D:49:9E:A2:E6, network=server-vlan10}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=8ac5777e-375f-4ec6-a6fd-856c7cd7363b}],custom={device_8617fb20-b870-45ea-8232-a70dd8b4551c=VmDevice:{id='VmDeviceId:{deviceId='8617fb20-b870-45ea-8232-a70dd8b4551c', vmId='3a17534b-e86d-4563-8ca2-2a27c34b4a87'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}, device_8617fb20-b870-45ea-8232-a70dd8b4551cdevice_f691fc09-31c8-43bf-bd82-c5acac8a1a76device_30bd748e-6ea8-434f-8587-d8ff8db5555e=VmDevice:{id='VmDeviceId:{deviceId='30bd748e-6ea8-434f-8587-d8ff8db5555e', vmId='3a17534b-e86d-4563-8ca2-2a27c34b4a87'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}, device_8617fb20-b870-45ea-8232-a70dd8b4551cdevice_f691fc09-31c8-43bf-bd82-c5acac8a1a76=VmDevice:{id='VmDeviceId:{deviceId='f691fc09-31c8-43bf-bd82-c5acac8a1a76', vmId='3a17534b-e86d-4563-8ca2-2a27c34b4a87'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}},display=vnc,timeOffset=0,spiceSslCipherSuite=DEFAULT,nice=0,maxMemSize=4194304,maxMemSlots=16,bootMenuEnable=false,memSize=4054
2016-01-04 16:55:39,627 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-40) [1f1deb62] FINISH, CreateVDSCommand, log id: 1d5c1c04
2016-01-04 16:55:39,631 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-40) [1f1deb62] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 55e0849d
2016-01-04 16:55:39,631 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-40) [1f1deb62] Lock freed to object 'EngineLock:{exclusiveLocks='[3a17534b-e86d-4563-8ca2-2a27c34b4a87=<VM, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2016-01-04 16:55:39,634 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-40) [1f1deb62] Correlation ID: 1f1deb62, Job ID: 5212e078-cf01-4312-ae00-7e0a136217a5, Call Stack: null, Custom Event ID: -1, Message: VM adm1 was started by jforeman at us.dignitastech.com@Dignitas AD (Host: ov-101).
2016-01-04 16:55:40,724 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-10) [] START, DestroyVDSCommand(HostName = ov-101, DestroyVmVDSCommandParameters:{runAsync='true', hostId='2fe6c27b-9346-4678-8cd3-c9d367ec447f', vmId='3a17534b-e86d-4563-8ca2-2a27c34b4a87', force='false', secondsToWait='0', gracefully='false', reason=''}), log id: 7935781d
2016-01-04 16:55:41,730 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-10) [] FINISH, DestroyVDSCommand, log id: 7935781d
2016-01-04 16:55:41,747 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-10) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM adm1 is down with error. Exit message: Unable to get volume size for domain 1fb79d91-b245-4447-91e0-e57671152a8c volume c736baca-de76-4593-b3dc-28bb8807e7a3.
2016-01-04 16:55:41,747 INFO  [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (ForkJoinPool-1-worker-10) [] Running on vds during rerun failed vm: '2fe6c27b-9346-4678-8cd3-c9d367ec447f'
2016-01-04 16:55:41,747 INFO  [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (ForkJoinPool-1-worker-10) [] VM '3a17534b-e86d-4563-8ca2-2a27c34b4a87(adm1) is running in db and not running in VDS 'ov-101'
2016-01-04 16:55:41,747 INFO  [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (ForkJoinPool-1-worker-10) [] add VM 'adm1' to HA rerun treatment
2016-01-04 16:55:41,752 ERROR [org.ovirt.engine.core.vdsbroker.VmsMonitoring] (ForkJoinPool-1-worker-10) [] Rerun VM '3a17534b-e86d-4563-8ca2-2a27c34b4a87'. Called from VDS 'ov-101'
2016-01-04 16:55:41,756 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-30) [] Correlation ID: 1f1deb62, Job ID: 5212e078-cf01-4312-ae00-7e0a136217a5, Call Stack: null, Custom Event ID: -1, Message: Failed to run VM adm1 on Host ov-101.
2016-01-04 16:55:41,760 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-30) [] Lock Acquired to object 'EngineLock:{exclusiveLocks='[3a17534b-e86d-4563-8ca2-2a27c34b4a87=<VM, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2016-01-04 16:55:41,770 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-8-thread-30) [] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='3a17534b-e86d-4563-8ca2-2a27c34b4a87'}), log id: 2577cd3a
2016-01-04 16:55:41,770 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-8-thread-30) [] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 2577cd3a
2016-01-04 16:55:41,798 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-30) [] Running command: RunVmCommand internal: false. Entities affected :  ID: 3a17534b-e86d-4563-8ca2-2a27c34b4a87 Type: VMAction group RUN_VM with role type USER
2016-01-04 16:55:41,850 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-30) [] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='00000000-0000-0000-0000-000000000000', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic at dbe0ef0a'}), log id: 351fb749
2016-01-04 16:55:41,852 INFO  [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-30) [] FINISH, UpdateVmDynamicDataVDSCommand, log id: 351fb749
2016-01-04 16:55:41,854 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-30) [] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='65555052-9601-4e4f-88f5-a0f14dcc29eb', vmId='3a17534b-e86d-4563-8ca2-2a27c34b4a87', vm='VM [adm1]'}), log id: 3163c7c3
2016-01-04 16:55:41,857 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-30) [] START, CreateVDSCommand(HostName = ov-102, CreateVmVDSCommandParameters:{runAsync='true', hostId='65555052-9601-4e4f-88f5-a0f14dcc29eb', vmId='3a17534b-e86d-4563-8ca2-2a27c34b4a87', vm='VM [adm1]'}), log id: 569ec368
2016-01-04 16:55:41,860 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.VmInfoBuilderBase] (org.ovirt.thread.pool-8-thread-30) [] Bootable disk '9e43c66a-5bf1-44d6-94f4-52178d15c1e6' set to index '0'
2016-01-04 16:55:41,869 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-30) [] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand pitReinjection=false,memGuaranteedSize=4054,smpThreadsPerCore=1,cpuType=SandyBridge,vmId=3a17534b-e86d-4563-8ca2-2a27c34b4a87,acpiEnable=true,numaTune={nodeset=0,1, mode=interleave},tabletEnable=true,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,vmType=kvm,keyboardLayout=en-us,smp=1,smpCoresPerSocket=1,emulatedMachine=pc-i440fx-rhel7.2.0,smartcardEnable=false,guestNumaNodes=[{memory=4054, cpus=0, nodeIndex=0}],transparentHugePages=true,vmName=adm1,maxVCpus=16,kvmEnable=true,devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=cirrus, deviceId=645e99e3-a9fa-4894-baf5-97b539236782}, {type=graphics, specParams={}, device=vnc, deviceId=12845c03-16a3-4bf0-a015-a15201a77673}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=ab048396-5dd8-4594-aa8a-9fe835a04cd1}, {shared=false, address={bus=0, controller=0, unit=0, type=drive, target=0}, imageID=9e43c66a-5bf1-44d6-94f4-52178d15c1e6, format=raw, index=0, optional=false, type=disk, deviceId=9e43c66a-5bf1-44d6-94f4-52178d15c1e6, domainID=1fb79d91-b245-4447-91e0-e57671152a8c, propagateErrors=off, iface=ide, readonly=false, bootOrder=1, poolID=00000001-0001-0001-0001-000000000154, volumeID=c736baca-de76-4593-b3dc-28bb8807e7a3, specParams={}, device=disk}, {shared=false, address={bus=0, controller=0, unit=1, type=drive, target=0}, imageID=a016b350-87ef-4c3b-b150-024907fed9c0, format=raw, optional=false, type=disk, deviceId=a016b350-87ef-4c3b-b150-024907fed9c0, domainID=1fb79d91-b245-4447-91e0-e57671152a8c, propagateErrors=off, iface=ide, readonly=false, poolID=00000001-0001-0001-0001-000000000154, volumeID=20fc4399-0b02-4da1-8aee-68df1629ca94, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=rtl8139, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=8e00d4cc-6a60-4598-82ee-645d742708de, macAddr=FA:0D:49:9E:A2:E6, network=server-vlan10}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=8ac5777e-375f-4ec6-a6fd-856c7cd7363b}],custom={device_8617fb20-b870-45ea-8232-a70dd8b4551c=VmDevice:{id='VmDeviceId:{deviceId='8617fb20-b870-45ea-8232-a70dd8b4551c', vmId='3a17534b-e86d-4563-8ca2-2a27c34b4a87'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}, device_8617fb20-b870-45ea-8232-a70dd8b4551cdevice_f691fc09-31c8-43bf-bd82-c5acac8a1a76device_30bd748e-6ea8-434f-8587-d8ff8db5555e=VmDevice:{id='VmDeviceId:{deviceId='30bd748e-6ea8-434f-8587-d8ff8db5555e', vmId='3a17534b-e86d-4563-8ca2-2a27c34b4a87'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}, device_8617fb20-b870-45ea-8232-a70dd8b4551cdevice_f691fc09-31c8-43bf-bd82-c5acac8a1a76=VmDevice:{id='VmDeviceId:{deviceId='f691fc09-31c8-43bf-bd82-c5acac8a1a76', vmId='3a17534b-e86d-4563-8ca2-2a27c34b4a87'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', usingScsiReservation='false'}},display=vnc,timeOffset=0,spiceSslCipherSuite=DEFAULT,nice=0,maxMemSize=4194304,maxMemSlots=16,bootMenuEnable=false,memSize=4054
2016-01-04 16:55:41,987 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-30) [] FINISH, CreateVDSCommand, log id: 569ec368
2016-01-04 16:55:41,991 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-30) [] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 3163c7c3
2016-01-04 16:55:41,992 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-30) [] Lock freed to object 'EngineLock:{exclusiveLocks='[3a17534b-e86d-4563-8ca2-2a27c34b4a87=<VM, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2016-01-04 16:55:41,994 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-30) [] Correlation ID: 1f1deb62, Job ID: 5212e078-cf01-4312-ae00-7e0a136217a5, Call Stack: null, Custom Event ID: -1, Message: VM adm1 was started by jforeman at us.dignitastech.com@Dignitas AD (Host: ov-102).
2016-01-04 16:55:43,069 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-3) [] START, DestroyVDSCommand(HostName = ov-102, DestroyVmVDSCommandParameters:{runAsync='true', hostId='65555052-9601-4e4f-88f5-a0f14dcc29eb', vmId='3a17534b-e86d-4563-8ca2-2a27c34b4a87', force='false', secondsToWait='0', gracefully='false', reason=''}), log id: 43dd93c5
2016-01-04 16:55:44,075 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-3) [] FINISH, DestroyVDSCommand, log id: 43dd93c5
2016-01-04 16:55:44,091 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-3) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM adm1 is down with error. Exit message: Unable to get volume size for domain 1fb79d91-b245-4447-91e0-e57671152a8c volume c736baca-de76-4593-b3dc-28bb8807e7a3.
2016-01-04 16:55:44,091 INFO  [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (ForkJoinPool-1-worker-3) [] Running on vds during rerun failed vm: '65555052-9601-4e4f-88f5-a0f14dcc29eb'
2016-01-04 16:55:44,092 INFO  [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (ForkJoinPool-1-worker-3) [] VM '3a17534b-e86d-4563-8ca2-2a27c34b4a87(adm1) is running in db and not running in VDS 'ov-102'
2016-01-04 16:55:44,092 INFO  [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (ForkJoinPool-1-worker-3) [] add VM 'adm1' to HA rerun treatment
2016-01-04 16:55:44,096 ERROR [org.ovirt.engine.core.vdsbroker.VmsMonitoring] (ForkJoinPool-1-worker-3) [] Rerun VM '3a17534b-e86d-4563-8ca2-2a27c34b4a87'. Called from VDS 'ov-102'
2016-01-04 16:55:44,128 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-35) [] Correlation ID: 1f1deb62, Job ID: 5212e078-cf01-4312-ae00-7e0a136217a5, Call Stack: null, Custom Event ID: -1, Message: Failed to run VM adm1 on Host ov-102.
2016-01-04 16:55:44,132 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-35) [] Lock Acquired to object 'EngineLock:{exclusiveLocks='[3a17534b-e86d-4563-8ca2-2a27c34b4a87=<VM, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2016-01-04 16:55:44,141 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-8-thread-35) [] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='3a17534b-e86d-4563-8ca2-2a27c34b4a87'}), log id: 545236ca
2016-01-04 16:55:44,141 INFO  [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-8-thread-35) [] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 545236ca
2016-01-04 16:55:44,162 WARN  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-35) [] CanDoAction of action 'RunVm' failed for user jforeman at us.dignitastech.com@Dignitas AD. Reasons: VAR__ACTION__RUN,VAR__TYPE__VM,VAR__ACTION__RUN,VAR__TYPE__VM,VAR__ACTION__RUN,VAR__TYPE__VM,VAR__ACTION__RUN,VAR__TYPE__VM,SCHEDULING_NO_HOSTS
2016-01-04 16:55:44,162 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-35) [] Lock freed to object 'EngineLock:{exclusiveLocks='[3a17534b-e86d-4563-8ca2-2a27c34b4a87=<VM, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2016-01-04 16:55:44,170 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-35) [] Correlation ID: 1f1deb62, Job ID: 5212e078-cf01-4312-ae00-7e0a136217a5, Call Stack: null, Custom Event ID: -1, Message: Failed to run VM adm1 (User: jforeman at us.dignitastech.com@Dignitas AD).
2016-01-04 16:55:44,173 INFO  [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-8-thread-46) [48c1f0bd] Running command: ProcessDownVmCommand internal: true.