Re: [ovirt-users] help.. vm trapped in limbo aka can't acquire exclusive lock

13 Apr 2014

      Hi Jeremiah,

Thanks for providing the relevant information from the engine log.

A fix was just merged to 3.4 branch.
For more info about the bug and the solution see: https://bugzilla.redhat.com/1086280

Note that in the meantime you can unlock the VM by restarting the engine as Roy mentioned.

Regards,
Arik

----- Original Message -----
...
----- Forwarded Message -----
...
From: "Jeremiah Jahn" <jeremiah@goodinassociates.com>
To: "Yair Zaslavsky" <yzaslavs@redhat.com>
Cc: users@ovirt.org
Sent: Thursday, April 10, 2014 5:48:24 AM
Subject: Re: [ovirt-users] help.. vm trapped in limbo aka can't acquire
exclusive lock
here's the start of the problem. If you really want the WHOLE log I
can do that too, but this is where I ask it to migrate, and where it
fails. After which it just keeps repeating that it can't get the lock.
2014-04-09 14:18:48,050 INFO
[org.ovirt.engine.core.bll.MigrateVmCommand]
(DefaultQuartzScheduler_Worker-33) Lock freed to object EngineLock
[exclusiveLocks= key:
0af0aaf6-85cf-4c67-9744-68a63601ab85 value: VM
, sharedLocks= ]
2014-04-09 14:19:32,177 INFO
[org.ovirt.engine.core.bll.MigrateVmCommand] (ajp--127.0.0.1-8702-1)
[4350ffd] Lock Acquired to object EngineLock [exclusiveLocks= key:
b0108933-deb2-4fa0-ae74-e10cefbb0cea value: VM
, sharedLocks= ]
2014-04-09 14:19:32,199 INFO
[org.ovirt.engine.core.bll.MigrateVmCommand]
(org.ovirt.thread.pool-6-thread-16) [4350ffd] Running command:
MigrateVmCommand internal: false. Entities affected :  ID:
b0108933-deb2-4fa0-ae74-e10cefbb0cea Type: VM
2014-04-09 14:19:32,203 INFO
[org.ovirt.engine.core.bll.scheduling.policyunits.HaReservationWeightPolicyUnit]
(org.ovirt.thread.pool-6-thread-16) [4350ffd] Started HA reservation
scoring method
2014-04-09 14:19:32,208 INFO
[org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
(org.ovirt.thread.pool-6-thread-16) [4350ffd] START,
MigrateVDSCommand(HostName = kvm01.virt, HostId =
fa455012-b929-4805-a5c1-50e6c788039a,
vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31,
dstVdsId=ce6b5b29-778f-4812-97b5-950f3d3b6c71,
dstHost=10.10.4.34:54321, migrationMethod=ONLINE,
tunnelMigration=false, migrationDowntime=0), log id: 745cd0ba
2014-04-09 14:19:32,209 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
(org.ovirt.thread.pool-6-thread-16) [4350ffd]
VdsBroker::migrate::Entered
(vm_guid=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31,
dstHost=10.10.4.34:54321,  method=online
2014-04-09 14:19:32,210 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
(org.ovirt.thread.pool-6-thread-16) [4350ffd] START,
MigrateBrokerVDSCommand(HostName = kvm01.virt, HostId =
fa455012-b929-4805-a5c1-50e6c788039a,
vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31,
dstVdsId=ce6b5b29-778f-4812-97b5-950f3d3b6c71,
dstHost=10.10.4.34:54321, migrationMethod=ONLINE,
tunnelMigration=false, migrationDowntime=0), log id: 5de39b66
2014-04-09 14:19:32,224 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
(org.ovirt.thread.pool-6-thread-16) [4350ffd] FINISH,
MigrateBrokerVDSCommand, log id: 5de39b66
2014-04-09 14:19:32,228 INFO
[org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
(org.ovirt.thread.pool-6-thread-16) [4350ffd] FINISH,
MigrateVDSCommand, return: MigratingFrom, log id: 745cd0ba
2014-04-09 14:19:32,233 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-16) [4350ffd] Correlation ID: 4350ffd,
Job ID: 433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom
Event ID: -1, Message: Migration started (VM: web.judici, Source:
kvm01.virt, Destination: kvm04, User: admin).
2014-04-09 14:19:33,465 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-62) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm04
ignoring it in the refresh until migration is done
2014-04-09 14:19:34,268 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-74) VM web.judici
b0108933-deb2-4fa0-ae74-e10cefbb0cea moved from MigratingFrom --> Up
2014-04-09 14:19:34,269 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-74) Adding VM
b0108933-deb2-4fa0-ae74-e10cefbb0cea to re-run list
2014-04-09 14:19:34,269 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand]
(DefaultQuartzScheduler_Worker-74) START, FullListVdsCommand(HostName
= kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a,
vds=Host[kvm01.virt], vmIds=[b0108933-deb2-4fa0-ae74-e10cefbb0cea]),
log id: 6936226f
2014-04-09 14:19:34,279 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand]
(DefaultQuartzScheduler_Worker-74) FINISH, FullListVdsCommand, return:
[{displaySecurePort=5905, kvmEnable=true, nicModel=rtl8139,pv,
keyboardLayout=en-us, displayIp=0, pauseCode=NOERR, nice=0,
pitReinjection=false, timeOffset=0, transparentHugePages=true,
vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, acpiEnable=true, custom={},
spiceSslCipherSuite=DEFAULT, memSize=8192, displayPort=5904, smp=4,
emulatedMachine=rhel6.5.0, status=Up, vmType=kvm,
memGuaranteedSize=1024, display=qxl, pid=30289, smartcardEnable=false,
smpCoresPerSocket=1,
spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,
maxVCpus=160, clientIp=10.10.4.242,
devices=[Ljava.lang.Object;@610edb5f, vmName=web.judici,
cpuType=Penryn}], log id: 6936226f
2014-04-09 14:19:34,281 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-74) Received a console Device without
an address when processing VM b0108933-deb2-4fa0-ae74-e10cefbb0cea
devices, skipping device: {alias=console0, specParams={},
device=console, type=console,
deviceId=05fba965-133e-4810-a87c-1688be8beb5a}
2014-04-09 14:19:34,294 ERROR
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-74) Rerun vm
b0108933-deb2-4fa0-ae74-e10cefbb0cea. Called from vds kvm01.virt
2014-04-09 14:19:34,299 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-46) START,
MigrateStatusVDSCommand(HostName = kvm01.virt, HostId =
fa455012-b929-4805-a5c1-50e6c788039a,
vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea), log id: 16ec0461
2014-04-09 14:19:34,346 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-46) Failed in MigrateStatusVDS method
2014-04-09 14:19:34,346 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-46) Command
org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand
return value
 StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=12,
mMessage=Fatal error during migration]]
2014-04-09 14:19:34,347 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-46) HostName = kvm01.virt
2014-04-09 14:19:34,348 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-46) Command
MigrateStatusVDSCommand(HostName = kvm01.virt, HostId =
fa455012-b929-4805-a5c1-50e6c788039a,
vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea) execution failed.
Exception: VDSErrorException: VDSGenericException: VDSErrorException:
Failed to MigrateStatusVDS, error = Fatal error during migration, code
= 12
2014-04-09 14:19:34,349 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-46) FINISH, MigrateStatusVDSCommand,
log id: 16ec0461
2014-04-09 14:19:34,353 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-46) Correlation ID: 4350ffd, Job ID:
433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event
ID: -1, Message: Migration failed due to Error: Fatal error during
migration. Trying to migrate to another Host (VM: web.judici, Source:
kvm01.virt, Destination: kvm04).
2014-04-09 14:19:34,370 INFO
[org.ovirt.engine.core.bll.MigrateVmCommand]
(org.ovirt.thread.pool-6-thread-46) Running command: MigrateVmCommand
internal: false. Entities affected :  ID:
b0108933-deb2-4fa0-ae74-e10cefbb0cea Type: VM
2014-04-09 14:19:34,374 INFO
[org.ovirt.engine.core.bll.scheduling.policyunits.HaReservationWeightPolicyUnit]
(org.ovirt.thread.pool-6-thread-46) Started HA reservation scoring
method
2014-04-09 14:19:34,378 INFO
[org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
(org.ovirt.thread.pool-6-thread-46) START, MigrateVDSCommand(HostName
= kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a,
vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31,
dstVdsId=071d8ef9-971e-499b-975a-255ee3d3f895,
dstHost=10.10.4.35:54321, migrationMethod=ONLINE,
tunnelMigration=false, migrationDowntime=0), log id: 7ac1f1eb
2014-04-09 14:19:34,379 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
(org.ovirt.thread.pool-6-thread-46) VdsBroker::migrate::Entered
(vm_guid=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31,
dstHost=10.10.4.35:54321,  method=online
2014-04-09 14:19:34,380 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
(org.ovirt.thread.pool-6-thread-46) START,
MigrateBrokerVDSCommand(HostName = kvm01.virt, HostId =
fa455012-b929-4805-a5c1-50e6c788039a,
vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31,
dstVdsId=071d8ef9-971e-499b-975a-255ee3d3f895,
dstHost=10.10.4.35:54321, migrationMethod=ONLINE,
tunnelMigration=false, migrationDowntime=0), log id: 3c9d3817
2014-04-09 14:19:34,431 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
(org.ovirt.thread.pool-6-thread-46) FINISH, MigrateBrokerVDSCommand,
log id: 3c9d3817
2014-04-09 14:19:34,434 INFO
[org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
(org.ovirt.thread.pool-6-thread-46) FINISH, MigrateVDSCommand, return:
MigratingFrom, log id: 7ac1f1eb
2014-04-09 14:19:34,438 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-46) Correlation ID: 4350ffd, Job ID:
433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event
ID: -1, Message: Migration started (VM: web.judici, Source:
kvm01.virt, Destination: kvm05.virt, User: admin).
2014-04-09 14:19:37,582 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-92) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:19:40,617 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-91) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:19:43,640 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-100) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:19:46,680 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-8) [1450e72d] RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:19:49,735 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-26) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:19:52,752 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-23) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:19:55,769 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-35) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:19:58,788 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-50) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:01,837 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-53) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:04,880 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-58) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:07,904 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-68) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:10,921 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-77) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:13,951 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-91) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:16,988 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-100) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:20,033 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-8) [1450e72d] RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:23,052 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-26) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:26,070 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-36) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:29,090 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-45) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:32,135 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-50) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:35,180 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-51) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:38,196 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-71) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:41,215 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-73) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:44,234 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-81) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:47,275 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-97) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:50,316 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-98) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:53,332 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-12) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:56,348 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-18) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:20:59,363 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-36) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:02,404 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-29) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:05,439 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-47) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:08,455 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-60) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:11,476 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-58) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:14,501 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-76) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:17,549 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-77) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:20,598 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-91) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:23,617 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-100) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:26,642 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-6) [5c26eeb2] RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:29,664 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-21) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:32,702 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-25) [e633f03] RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:35,751 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-31) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:38,772 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-48) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:41,790 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-60) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:44,812 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-71) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:47,858 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-72) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:50,903 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-77) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:53,919 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-94) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:56,938 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-5) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:21:59,961 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-14) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:02,997 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-21) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:06,037 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-36) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:09,058 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-45) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:12,075 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-50) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:15,103 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-51) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:17,696 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-58) VM web.judici
b0108933-deb2-4fa0-ae74-e10cefbb0cea moved from MigratingFrom --> Up
2014-04-09 14:22:17,697 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-58) Adding VM
b0108933-deb2-4fa0-ae74-e10cefbb0cea to re-run list
2014-04-09 14:22:17,698 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand]
(DefaultQuartzScheduler_Worker-58) START, FullListVdsCommand(HostName
= kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a,
vds=Host[kvm01.virt], vmIds=[b0108933-deb2-4fa0-ae74-e10cefbb0cea]),
log id: 1688c794
2014-04-09 14:22:17,706 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand]
(DefaultQuartzScheduler_Worker-58) FINISH, FullListVdsCommand, return:
[{displaySecurePort=5905, kvmEnable=true, nicModel=rtl8139,pv,
keyboardLayout=en-us, displayIp=0, pauseCode=NOERR, nice=0,
pitReinjection=false, timeOffset=0, transparentHugePages=true,
vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, acpiEnable=true, custom={},
spiceSslCipherSuite=DEFAULT, memSize=8192, displayPort=5904, smp=4,
emulatedMachine=rhel6.5.0, status=Up, vmType=kvm,
memGuaranteedSize=1024, display=qxl, pid=30289, smartcardEnable=false,
smpCoresPerSocket=1,
spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,
maxVCpus=160, clientIp=10.10.4.242,
devices=[Ljava.lang.Object;@6de6d11a, vmName=web.judici,
cpuType=Penryn}], log id: 1688c794
2014-04-09 14:22:17,709 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-58) Received a console Device without
an address when processing VM b0108933-deb2-4fa0-ae74-e10cefbb0cea
devices, skipping device: {alias=console0, specParams={},
device=console, type=console,
deviceId=05fba965-133e-4810-a87c-1688be8beb5a}
2014-04-09 14:22:17,728 ERROR
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-58) Rerun vm
b0108933-deb2-4fa0-ae74-e10cefbb0cea. Called from vds kvm01.virt
2014-04-09 14:22:17,735 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-29) START,
MigrateStatusVDSCommand(HostName = kvm01.virt, HostId =
fa455012-b929-4805-a5c1-50e6c788039a,
vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea), log id: 15822aeb
2014-04-09 14:22:17,781 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-29) Failed in MigrateStatusVDS method
2014-04-09 14:22:17,782 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-29) Command
org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand
return value
 StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=47,
mMessage=Migration canceled]]
2014-04-09 14:22:17,782 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-29) HostName = kvm01.virt
2014-04-09 14:22:17,783 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-29) Command
MigrateStatusVDSCommand(HostName = kvm01.virt, HostId =
fa455012-b929-4805-a5c1-50e6c788039a,
vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea) execution failed.
Exception: VDSErrorException: VDSGenericException: VDSErrorException:
Failed to MigrateStatusVDS, error = Migration canceled, code = 47
2014-04-09 14:22:17,784 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-29) FINISH, MigrateStatusVDSCommand,
log id: 15822aeb
2014-04-09 14:22:17,791 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-29) Correlation ID: 4350ffd, Job ID:
433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event
ID: -1, Message: Migration failed due to Error: Migration not in
progress. Trying to migrate to another Host (VM: web.judici, Source:
kvm01.virt, Destination: kvm05.virt).
2014-04-09 14:22:17,818 INFO
[org.ovirt.engine.core.bll.MigrateVmCommand]
(org.ovirt.thread.pool-6-thread-29) Running command: MigrateVmCommand
internal: false. Entities affected :  ID:
b0108933-deb2-4fa0-ae74-e10cefbb0cea Type: VM
2014-04-09 14:22:17,824 INFO
[org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
(org.ovirt.thread.pool-6-thread-29) START, MigrateVDSCommand(HostName
= kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a,
vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31,
dstVdsId=d5e2830d-e59c-49be-b6e8-0968dbf357aa,
dstHost=10.10.4.33:54321, migrationMethod=ONLINE,
tunnelMigration=false, migrationDowntime=0), log id: 7dc46595
2014-04-09 14:22:17,825 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
(org.ovirt.thread.pool-6-thread-29) VdsBroker::migrate::Entered
(vm_guid=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31,
dstHost=10.10.4.33:54321,  method=online
2014-04-09 14:22:17,826 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
(org.ovirt.thread.pool-6-thread-29) START,
MigrateBrokerVDSCommand(HostName = kvm01.virt, HostId =
fa455012-b929-4805-a5c1-50e6c788039a,
vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31,
dstVdsId=d5e2830d-e59c-49be-b6e8-0968dbf357aa,
dstHost=10.10.4.33:54321, migrationMethod=ONLINE,
tunnelMigration=false, migrationDowntime=0), log id: 25038825
2014-04-09 14:22:17,848 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
(org.ovirt.thread.pool-6-thread-29) FINISH, MigrateBrokerVDSCommand,
log id: 25038825
2014-04-09 14:22:17,853 INFO
[org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
(org.ovirt.thread.pool-6-thread-29) FINISH, MigrateVDSCommand, return:
MigratingFrom, log id: 7dc46595
2014-04-09 14:22:17,859 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-29) Correlation ID: 4350ffd, Job ID:
433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event
ID: -1, Message: Migration started (VM: web.judici, Source:
kvm01.virt, Destination: kvm03.virt, User: admin).
2014-04-09 14:22:19,226 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-67) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:22,275 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-79) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:25,291 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-83) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:28,314 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-1) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:31,330 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-6) [5c26eeb2] RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:34,360 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-16) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:37,389 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-30) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:40,407 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-31) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:43,425 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-52) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:46,443 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-51) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:49,473 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-71) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:52,514 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-76) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:55,533 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-92) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:22:58,553 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-90) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:01,569 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-10) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:04,596 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-7) [7a463a98] RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:07,626 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-16) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:10,648 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-30) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:13,665 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-41) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:16,693 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-47) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:19,724 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-51) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:22,758 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-65) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:25,778 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-79) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:28,795 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-95) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:31,814 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-87) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:34,848 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-1) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:37,884 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-19) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:40,910 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-22) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:43,930 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-25) [e633f03] RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:46,949 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-45) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:49,982 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-48) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:53,048 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-64) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:56,076 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-66) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:23:59,093 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-72) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:02,112 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-93) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:05,140 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-87) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:08,210 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-3) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:11,240 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-6) [5c26eeb2] RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:14,257 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-21) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:17,277 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-36) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:20,309 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-31) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:23,371 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-48) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:26,391 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-63) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:29,409 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-74) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:32,428 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-72) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:35,467 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-85) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:38,499 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-89) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-2014-04-09 14:24:38,499 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-89) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-
e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh
until migration is done
2014-04-09 14:24:41,534 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-1) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:44,561 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-11) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:47,584 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-21) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:50,631 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-25) [e633f03] RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:53,668 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-37) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:56,683 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-55) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:24:59,702 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-57) RefreshVmList vm id
b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt
ignoring it in the refresh until migration is done
2014-04-09 14:25:00,974 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-59) VM web.judici
b0108933-deb2-4fa0-ae74-e10cefbb0cea moved from MigratingFrom --> Up
2014-04-09 14:25:00,975 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-59) Adding VM
b0108933-deb2-4fa0-ae74-e10cefbb0cea to re-run list
2014-04-09 14:25:00,976 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand]
(DefaultQuartzScheduler_Worker-59) START, FullListVdsCommand(HostName
= kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a,
vds=Host[kvm01.virt], vmIds=[b0108933-deb2-4fa0-ae74-e10cefbb0cea]),
log id: 7ef5612e
2014-04-09 14:25:00,984 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand]
(DefaultQuartzScheduler_Worker-59) FINISH, FullListVdsCommand, return:
[{displaySecurePort=5905, kvmEnable=true, nicModel=rtl8139,pv,
keyboardLayout=en-us, displayIp=0, pauseCode=NOERR, nice=0,
pitReinjection=false, timeOffset=0, transparentHugePages=true,
vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, acpiEnable=true, custom={},
spiceSslCipherSuite=DEFAULT, memSize=8192, displayPort=5904, smp=4,
emulatedMachine=rhel6.5.0, status=Up, vmType=kvm,
memGuaranteedSize=1024, display=qxl, pid=30289, smartcardEnable=false,
smpCoresPerSocket=1,
spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,
maxVCpus=160, clientIp=10.10.4.242,
devices=[Ljava.lang.Object;@1fbf5420, vmName=web.judici,
cpuType=Penryn}], log id: 7ef5612e
2014-04-09 14:25:00,988 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-59) Received a console Device without
an address when processing VM b0108933-deb2-4fa0-ae74-e10cefbb0cea
devices, skipping device: {alias=console0, specParams={},
device=console, type=console,
deviceId=05fba965-133e-4810-a87c-1688be8beb5a}
2014-04-09 14:25:01,030 ERROR
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-59) Rerun vm
b0108933-deb2-4fa0-ae74-e10cefbb0cea. Called from vds kvm01.virt
2014-04-09 14:25:01,038 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-37) START,
MigrateStatusVDSCommand(HostName = kvm01.virt, HostId =
fa455012-b929-4805-a5c1-50e6c788039a,
vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea), log id: 47013f27
2014-04-09 14:25:01,045 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-37) Failed in MigrateStatusVDS method
2014-04-09 14:25:01,045 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-37) Command
org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand
return value
 StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=47,
mMessage=Migration canceled]]
2014-04-09 14:25:01,046 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-37) HostName = kvm01.virt
2014-04-09 14:25:01,046 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-37) Command
MigrateStatusVDSCommand(HostName = kvm01.virt, HostId =
fa455012-b929-4805-a5c1-50e6c788039a,
vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea) execution failed.
Exception: VDSErrorException: VDSGenericException: VDSErrorException:
Failed to MigrateStatusVDS, error = Migration canceled, code = 47
2014-04-09 14:25:01,048 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-37) FINISH, MigrateStatusVDSCommand,
log id: 47013f27
2014-04-09 14:25:01,055 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-37) Correlation ID: 4350ffd, Job ID:
433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event
ID: -1, Message: Migration failed due to Error: Migration not in
progress. Trying to migrate to another Host (VM: web.judici, Source:
kvm01.virt, Destination: kvm03.virt).
2014-04-09 14:25:01,061 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-37) Correlation ID: 4350ffd, Job ID:
433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event
ID: -1, Message: Migration failed due to Error: Migration not in
progress (VM: web.judici, Source: kvm01.virt, Destination:
kvm03.virt).
2014-04-09 14:34:08,665 INFO
[org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand]
(org.ovirt.thread.pool-6-thread-43) [4045389e] Running command:
MaintenanceNumberOfVdssCommand internal: false. Entities affected :
ID: fa455012-b929-4805-a5c1-50e6c788039a Type: VDS
2014-04-09 14:34:08,667 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-43) [4045389e] START,
SetVdsStatusVDSCommand(HostName = kvm01.virt, HostId =
fa455012-b929-4805-a5c1-50e6c788039a, status=PreparingForMaintenance,
nonOperationalReason=NONE, stopSpmFailureLogged=true), log id:
64965c26
2014-04-09 14:34:08,678 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(org.ovirt.thread.pool-6-thread-43) [4045389e] FINISH,
SetVdsStatusVDSCommand, log id: 64965c26
2014-04-09 14:34:08,725 INFO
[org.ovirt.engine.core.bll.MaintenanceVdsCommand]
(org.ovirt.thread.pool-6-thread-43) [4045389e] Running command:
MaintenanceVdsCommand internal: true. Entities affected :  ID:
fa455012-b929-4805-a5c1-50e6c788039a Type: VDS
2014-04-09 14:34:08,744 INFO
[org.ovirt.engine.core.bll.InternalMigrateVmCommand]
(org.ovirt.thread.pool-6-thread-43) [35f06602] Failed to Acquire Lock
to object EngineLock [exclusiveLocks= key:
b0108933-deb2-4fa0-ae74-e10cefbb0cea value: VM
, sharedLocks= ]
On Wed, Apr 9, 2014 at 6:33 PM, Yair Zaslavsky <yzaslavs@redhat.com> wrote:
...
Can you please attach full engine.log?
Many thanks,
Yair
----- Original Message -----
...
From: "Jeremiah Jahn" <jeremiah@goodinassociates.com>
To: users@ovirt.org
Sent: Thursday, April 10, 2014 2:18:48 AM
Subject: [ovirt-users] help.. vm trapped in limbo aka can't acquire
exclusive lock
I can't start it, I can't migrate it. I tried to migrate it before,
but the machine was stuck in a read only state. The migration failed
because the machine it was being migrated to was also in a read only
state. somewhere  in the process the lock obviously got lost, and I
can't get it back...
2014-04-09 18:11:16,675 INFO  [org.ovirt.engine.core.bll.RunVmCommand]
(ajp--127.0.0.1-8702-3) [58b40832] Failed to Acquire Lock to object
EngineLock [exclusiveLocks= key: b0108933-deb2-4fa0-ae74-e10cefbb0cea
value: VM
, sharedLocks= ]
2014-04-09 18:11:16,676 WARN  [org.ovirt.engine.core.bll.RunVmCommand]
(ajp--127.0.0.1-8702-3) [58b40832] CanDoAction of action RunVm failed.
Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_IS_BEING_MIGRATED,$VmName
web.judici
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Arik Hadas

tags

participants (1)