Re: [ovirt-users] help.. vm trapped in limbo aka can't acquire exclusive lock

here's the start of the problem. If you really want the WHOLE log I can do that too, but this is where I ask it to migrate, and where it fails. After which it just keeps repeating that it can't get the lock. 2014-04-09 14:18:48,050 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (DefaultQuartzScheduler_Worker-33) Lock freed to object EngineLock [exclusiveLocks= key: 0af0aaf6-85cf-4c67-9744-68a63601ab85 value: VM , sharedLocks= ] 2014-04-09 14:19:32,177 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (ajp--127.0.0.1-8702-1) [4350ffd] Lock Acquired to object EngineLock [exclusiveLocks= key: b0108933-deb2-4fa0-ae74-e10cefbb0cea value: VM , sharedLocks= ] 2014-04-09 14:19:32,199 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-16) [4350ffd] Running command: MigrateVmCommand internal: false. Entities affected : ID: b0108933-deb2-4fa0-ae74-e10cefbb0cea Type: VM 2014-04-09 14:19:32,203 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.HaReservationWeightPolicyUnit] (org.ovirt.thread.pool-6-thread-16) [4350ffd] Started HA reservation scoring method 2014-04-09 14:19:32,208 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-16) [4350ffd] START, MigrateVDSCommand(HostName = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31, dstVdsId=ce6b5b29-778f-4812-97b5-950f3d3b6c71, dstHost=10.10.4.34:54321, migrationMethod=ONLINE, tunnelMigration=false, migrationDowntime=0), log id: 745cd0ba 2014-04-09 14:19:32,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-16) [4350ffd] VdsBroker::migrate::Entered (vm_guid=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31, dstHost=10.10.4.34:54321, method=online 2014-04-09 14:19:32,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-16) [4350ffd] START, MigrateBrokerVDSCommand(HostName = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31, dstVdsId=ce6b5b29-778f-4812-97b5-950f3d3b6c71, dstHost=10.10.4.34:54321, migrationMethod=ONLINE, tunnelMigration=false, migrationDowntime=0), log id: 5de39b66 2014-04-09 14:19:32,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-16) [4350ffd] FINISH, MigrateBrokerVDSCommand, log id: 5de39b66 2014-04-09 14:19:32,228 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-16) [4350ffd] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 745cd0ba 2014-04-09 14:19:32,233 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-16) [4350ffd] Correlation ID: 4350ffd, Job ID: 433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event ID: -1, Message: Migration started (VM: web.judici, Source: kvm01.virt, Destination: kvm04, User: admin). 2014-04-09 14:19:33,465 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm04 ignoring it in the refresh until migration is done 2014-04-09 14:19:34,268 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) VM web.judici b0108933-deb2-4fa0-ae74-e10cefbb0cea moved from MigratingFrom --> Up 2014-04-09 14:19:34,269 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Adding VM b0108933-deb2-4fa0-ae74-e10cefbb0cea to re-run list 2014-04-09 14:19:34,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-74) START, FullListVdsCommand(HostName = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, vds=Host[kvm01.virt], vmIds=[b0108933-deb2-4fa0-ae74-e10cefbb0cea]), log id: 6936226f 2014-04-09 14:19:34,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-74) FINISH, FullListVdsCommand, return: [{displaySecurePort=5905, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=0, pauseCode=NOERR, nice=0, pitReinjection=false, timeOffset=0, transparentHugePages=true, vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=8192, displayPort=5904, smp=4, emulatedMachine=rhel6.5.0, status=Up, vmType=kvm, memGuaranteedSize=1024, display=qxl, pid=30289, smartcardEnable=false, smpCoresPerSocket=1, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, maxVCpus=160, clientIp=10.10.4.242, devices=[Ljava.lang.Object;@610edb5f, vmName=web.judici, cpuType=Penryn}], log id: 6936226f 2014-04-09 14:19:34,281 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Received a console Device without an address when processing VM b0108933-deb2-4fa0-ae74-e10cefbb0cea devices, skipping device: {alias=console0, specParams={}, device=console, type=console, deviceId=05fba965-133e-4810-a87c-1688be8beb5a} 2014-04-09 14:19:34,294 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) Rerun vm b0108933-deb2-4fa0-ae74-e10cefbb0cea. Called from vds kvm01.virt 2014-04-09 14:19:34,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-46) START, MigrateStatusVDSCommand(HostName = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea), log id: 16ec0461 2014-04-09 14:19:34,346 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-46) Failed in MigrateStatusVDS method 2014-04-09 14:19:34,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=12, mMessage=Fatal error during migration]] 2014-04-09 14:19:34,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-46) HostName = kvm01.virt 2014-04-09 14:19:34,348 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-46) Command MigrateStatusVDSCommand(HostName = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Fatal error during migration, code = 12 2014-04-09 14:19:34,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-46) FINISH, MigrateStatusVDSCommand, log id: 16ec0461 2014-04-09 14:19:34,353 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) Correlation ID: 4350ffd, Job ID: 433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event ID: -1, Message: Migration failed due to Error: Fatal error during migration. Trying to migrate to another Host (VM: web.judici, Source: kvm01.virt, Destination: kvm04). 2014-04-09 14:19:34,370 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-46) Running command: MigrateVmCommand internal: false. Entities affected : ID: b0108933-deb2-4fa0-ae74-e10cefbb0cea Type: VM 2014-04-09 14:19:34,374 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.HaReservationWeightPolicyUnit] (org.ovirt.thread.pool-6-thread-46) Started HA reservation scoring method 2014-04-09 14:19:34,378 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-46) START, MigrateVDSCommand(HostName = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31, dstVdsId=071d8ef9-971e-499b-975a-255ee3d3f895, dstHost=10.10.4.35:54321, migrationMethod=ONLINE, tunnelMigration=false, migrationDowntime=0), log id: 7ac1f1eb 2014-04-09 14:19:34,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-46) VdsBroker::migrate::Entered (vm_guid=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31, dstHost=10.10.4.35:54321, method=online 2014-04-09 14:19:34,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-46) START, MigrateBrokerVDSCommand(HostName = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31, dstVdsId=071d8ef9-971e-499b-975a-255ee3d3f895, dstHost=10.10.4.35:54321, migrationMethod=ONLINE, tunnelMigration=false, migrationDowntime=0), log id: 3c9d3817 2014-04-09 14:19:34,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-46) FINISH, MigrateBrokerVDSCommand, log id: 3c9d3817 2014-04-09 14:19:34,434 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-46) FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7ac1f1eb 2014-04-09 14:19:34,438 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-46) Correlation ID: 4350ffd, Job ID: 433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event ID: -1, Message: Migration started (VM: web.judici, Source: kvm01.virt, Destination: kvm05.virt, User: admin). 2014-04-09 14:19:37,582 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:19:40,617 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:19:43,640 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:19:46,680 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) [1450e72d] RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:19:49,735 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:19:52,752 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:19:55,769 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-35) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:19:58,788 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:01,837 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:04,880 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:07,904 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:10,921 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:13,951 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:16,988 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:20,033 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) [1450e72d] RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:23,052 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:26,070 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:29,090 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:32,135 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:35,180 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:38,196 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:41,215 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:44,234 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:47,275 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:50,316 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:53,332 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:56,348 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:20:59,363 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:02,404 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:05,439 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:08,455 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:11,476 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:14,501 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:17,549 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:20,598 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:23,617 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:26,642 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [5c26eeb2] RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:29,664 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:32,702 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [e633f03] RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:35,751 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:38,772 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:41,790 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:44,812 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:47,858 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:50,903 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:53,919 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:56,938 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:21:59,961 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:02,997 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:06,037 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:09,058 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:12,075 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:15,103 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:17,696 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) VM web.judici b0108933-deb2-4fa0-ae74-e10cefbb0cea moved from MigratingFrom --> Up 2014-04-09 14:22:17,697 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) Adding VM b0108933-deb2-4fa0-ae74-e10cefbb0cea to re-run list 2014-04-09 14:22:17,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-58) START, FullListVdsCommand(HostName = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, vds=Host[kvm01.virt], vmIds=[b0108933-deb2-4fa0-ae74-e10cefbb0cea]), log id: 1688c794 2014-04-09 14:22:17,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-58) FINISH, FullListVdsCommand, return: [{displaySecurePort=5905, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=0, pauseCode=NOERR, nice=0, pitReinjection=false, timeOffset=0, transparentHugePages=true, vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=8192, displayPort=5904, smp=4, emulatedMachine=rhel6.5.0, status=Up, vmType=kvm, memGuaranteedSize=1024, display=qxl, pid=30289, smartcardEnable=false, smpCoresPerSocket=1, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, maxVCpus=160, clientIp=10.10.4.242, devices=[Ljava.lang.Object;@6de6d11a, vmName=web.judici, cpuType=Penryn}], log id: 1688c794 2014-04-09 14:22:17,709 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) Received a console Device without an address when processing VM b0108933-deb2-4fa0-ae74-e10cefbb0cea devices, skipping device: {alias=console0, specParams={}, device=console, type=console, deviceId=05fba965-133e-4810-a87c-1688be8beb5a} 2014-04-09 14:22:17,728 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) Rerun vm b0108933-deb2-4fa0-ae74-e10cefbb0cea. Called from vds kvm01.virt 2014-04-09 14:22:17,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-29) START, MigrateStatusVDSCommand(HostName = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea), log id: 15822aeb 2014-04-09 14:22:17,781 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-29) Failed in MigrateStatusVDS method 2014-04-09 14:22:17,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-29) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=47, mMessage=Migration canceled]] 2014-04-09 14:22:17,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-29) HostName = kvm01.virt 2014-04-09 14:22:17,783 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-29) Command MigrateStatusVDSCommand(HostName = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Migration canceled, code = 47 2014-04-09 14:22:17,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-29) FINISH, MigrateStatusVDSCommand, log id: 15822aeb 2014-04-09 14:22:17,791 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) Correlation ID: 4350ffd, Job ID: 433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event ID: -1, Message: Migration failed due to Error: Migration not in progress. Trying to migrate to another Host (VM: web.judici, Source: kvm01.virt, Destination: kvm05.virt). 2014-04-09 14:22:17,818 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-29) Running command: MigrateVmCommand internal: false. Entities affected : ID: b0108933-deb2-4fa0-ae74-e10cefbb0cea Type: VM 2014-04-09 14:22:17,824 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-29) START, MigrateVDSCommand(HostName = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31, dstVdsId=d5e2830d-e59c-49be-b6e8-0968dbf357aa, dstHost=10.10.4.33:54321, migrationMethod=ONLINE, tunnelMigration=false, migrationDowntime=0), log id: 7dc46595 2014-04-09 14:22:17,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-29) VdsBroker::migrate::Entered (vm_guid=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31, dstHost=10.10.4.33:54321, method=online 2014-04-09 14:22:17,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-29) START, MigrateBrokerVDSCommand(HostName = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31, dstVdsId=d5e2830d-e59c-49be-b6e8-0968dbf357aa, dstHost=10.10.4.33:54321, migrationMethod=ONLINE, tunnelMigration=false, migrationDowntime=0), log id: 25038825 2014-04-09 14:22:17,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-29) FINISH, MigrateBrokerVDSCommand, log id: 25038825 2014-04-09 14:22:17,853 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-29) FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7dc46595 2014-04-09 14:22:17,859 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-29) Correlation ID: 4350ffd, Job ID: 433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event ID: -1, Message: Migration started (VM: web.judici, Source: kvm01.virt, Destination: kvm03.virt, User: admin). 2014-04-09 14:22:19,226 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:22,275 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:25,291 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:28,314 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:31,330 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [5c26eeb2] RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:34,360 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:37,389 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:40,407 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:43,425 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:46,443 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:49,473 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:52,514 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:55,533 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:22:58,553 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:01,569 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:04,596 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) [7a463a98] RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:07,626 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:10,648 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:13,665 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:16,693 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:19,724 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:22,758 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:25,778 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:28,795 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:31,814 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:34,848 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:37,884 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:40,910 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:43,930 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [e633f03] RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:46,949 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:49,982 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:53,048 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:56,076 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:23:59,093 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:02,112 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:05,140 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:08,210 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:11,240 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [5c26eeb2] RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:14,257 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:17,277 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:20,309 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:23,371 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:26,391 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:29,409 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:32,428 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:35,467 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:38,499 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) RefreshVmList vm id b0108933-deb2-4fa0-ae74-2014-04-09 14:24:38,499 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) RefreshVmList vm id b0108933-deb2-4fa0-ae74- e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:41,534 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:44,561 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:47,584 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:50,631 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [e633f03] RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:53,668 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:56,683 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:24:59,702 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) RefreshVmList vm id b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh until migration is done 2014-04-09 14:25:00,974 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) VM web.judici b0108933-deb2-4fa0-ae74-e10cefbb0cea moved from MigratingFrom --> Up 2014-04-09 14:25:00,975 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Adding VM b0108933-deb2-4fa0-ae74-e10cefbb0cea to re-run list 2014-04-09 14:25:00,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-59) START, FullListVdsCommand(HostName = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, vds=Host[kvm01.virt], vmIds=[b0108933-deb2-4fa0-ae74-e10cefbb0cea]), log id: 7ef5612e 2014-04-09 14:25:00,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-59) FINISH, FullListVdsCommand, return: [{displaySecurePort=5905, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=0, pauseCode=NOERR, nice=0, pitReinjection=false, timeOffset=0, transparentHugePages=true, vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=8192, displayPort=5904, smp=4, emulatedMachine=rhel6.5.0, status=Up, vmType=kvm, memGuaranteedSize=1024, display=qxl, pid=30289, smartcardEnable=false, smpCoresPerSocket=1, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, maxVCpus=160, clientIp=10.10.4.242, devices=[Ljava.lang.Object;@1fbf5420, vmName=web.judici, cpuType=Penryn}], log id: 7ef5612e 2014-04-09 14:25:00,988 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Received a console Device without an address when processing VM b0108933-deb2-4fa0-ae74-e10cefbb0cea devices, skipping device: {alias=console0, specParams={}, device=console, type=console, deviceId=05fba965-133e-4810-a87c-1688be8beb5a} 2014-04-09 14:25:01,030 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) Rerun vm b0108933-deb2-4fa0-ae74-e10cefbb0cea. Called from vds kvm01.virt 2014-04-09 14:25:01,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-37) START, MigrateStatusVDSCommand(HostName = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea), log id: 47013f27 2014-04-09 14:25:01,045 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-37) Failed in MigrateStatusVDS method 2014-04-09 14:25:01,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-37) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=47, mMessage=Migration canceled]] 2014-04-09 14:25:01,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-37) HostName = kvm01.virt 2014-04-09 14:25:01,046 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-37) Command MigrateStatusVDSCommand(HostName = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Migration canceled, code = 47 2014-04-09 14:25:01,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-37) FINISH, MigrateStatusVDSCommand, log id: 47013f27 2014-04-09 14:25:01,055 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) Correlation ID: 4350ffd, Job ID: 433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event ID: -1, Message: Migration failed due to Error: Migration not in progress. Trying to migrate to another Host (VM: web.judici, Source: kvm01.virt, Destination: kvm03.virt). 2014-04-09 14:25:01,061 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-37) Correlation ID: 4350ffd, Job ID: 433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event ID: -1, Message: Migration failed due to Error: Migration not in progress (VM: web.judici, Source: kvm01.virt, Destination: kvm03.virt). 2014-04-09 14:34:08,665 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-43) [4045389e] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: fa455012-b929-4805-a5c1-50e6c788039a Type: VDS 2014-04-09 14:34:08,667 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-43) [4045389e] START, SetVdsStatusVDSCommand(HostName = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 64965c26 2014-04-09 14:34:08,678 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-43) [4045389e] FINISH, SetVdsStatusVDSCommand, log id: 64965c26 2014-04-09 14:34:08,725 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-43) [4045389e] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: fa455012-b929-4805-a5c1-50e6c788039a Type: VDS 2014-04-09 14:34:08,744 INFO [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (org.ovirt.thread.pool-6-thread-43) [35f06602] Failed to Acquire Lock to object EngineLock [exclusiveLocks= key: b0108933-deb2-4fa0-ae74-e10cefbb0cea value: VM , sharedLocks= ] On Wed, Apr 9, 2014 at 6:33 PM, Yair Zaslavsky <yzaslavs(a)redhat.com&gt; wrote: