Re: [ovirt-users] help.. vm trapped in limbo aka can't acquire exclusive lock

----- Forwarded Message ----- > From: "Jeremiah Jahn" <jeremiah(a)goodinassociates.com&gt; > To: "Yair Zaslavsky" <yzaslavs(a)redhat.com&gt; > Cc: users(a)ovirt.org > Sent: Thursday, April 10, 2014 5:48:24 AM > Subject: Re: [ovirt-users] help.. vm trapped in limbo aka can't acquire > exclusive lock > > here's the start of the problem. If you really want the WHOLE log I > can do that too, but this is where I ask it to migrate, and where it > fails. After which it just keeps repeating that it can't get the lock. > > > > 2014-04-09 14:18:48,050 INFO > [org.ovirt.engine.core.bll.MigrateVmCommand] > (DefaultQuartzScheduler_Worker-33) Lock freed to object EngineLock > [exclusiveLocks= key: > 0af0aaf6-85cf-4c67-9744-68a63601ab85 value: VM > , sharedLocks= ] > 2014-04-09 14:19:32,177 INFO > [org.ovirt.engine.core.bll.MigrateVmCommand] (ajp--127.0.0.1-8702-1) > [4350ffd] Lock Acquired to object EngineLock [exclusiveLocks= key: > b0108933-deb2-4fa0-ae74-e10cefbb0cea value: VM > , sharedLocks= ] > 2014-04-09 14:19:32,199 INFO > [org.ovirt.engine.core.bll.MigrateVmCommand] > (org.ovirt.thread.pool-6-thread-16) [4350ffd] Running command: > MigrateVmCommand internal: false. Entities affected : ID: > b0108933-deb2-4fa0-ae74-e10cefbb0cea Type: VM > 2014-04-09 14:19:32,203 INFO > [org.ovirt.engine.core.bll.scheduling.policyunits.HaReservationWeightPolicyUnit] > (org.ovirt.thread.pool-6-thread-16) [4350ffd] Started HA reservation > scoring method > 2014-04-09 14:19:32,208 INFO > [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] > (org.ovirt.thread.pool-6-thread-16) [4350ffd] START, > MigrateVDSCommand(HostName = kvm01.virt, HostId = > fa455012-b929-4805-a5c1-50e6c788039a, > vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31, > dstVdsId=ce6b5b29-778f-4812-97b5-950f3d3b6c71, > dstHost=10.10.4.34:54321, migrationMethod=ONLINE, > tunnelMigration=false, migrationDowntime=0), log id: 745cd0ba > 2014-04-09 14:19:32,209 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] > (org.ovirt.thread.pool-6-thread-16) [4350ffd] > VdsBroker::migrate::Entered > (vm_guid=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31, > dstHost=10.10.4.34:54321, method=online > 2014-04-09 14:19:32,210 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] > (org.ovirt.thread.pool-6-thread-16) [4350ffd] START, > MigrateBrokerVDSCommand(HostName = kvm01.virt, HostId = > fa455012-b929-4805-a5c1-50e6c788039a, > vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31, > dstVdsId=ce6b5b29-778f-4812-97b5-950f3d3b6c71, > dstHost=10.10.4.34:54321, migrationMethod=ONLINE, > tunnelMigration=false, migrationDowntime=0), log id: 5de39b66 > 2014-04-09 14:19:32,224 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] > (org.ovirt.thread.pool-6-thread-16) [4350ffd] FINISH, > MigrateBrokerVDSCommand, log id: 5de39b66 > 2014-04-09 14:19:32,228 INFO > [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] > (org.ovirt.thread.pool-6-thread-16) [4350ffd] FINISH, > MigrateVDSCommand, return: MigratingFrom, log id: 745cd0ba > 2014-04-09 14:19:32,233 INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (org.ovirt.thread.pool-6-thread-16) [4350ffd] Correlation ID: 4350ffd, > Job ID: 433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom > Event ID: -1, Message: Migration started (VM: web.judici, Source: > kvm01.virt, Destination: kvm04, User: admin). > 2014-04-09 14:19:33,465 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-62) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm04 > ignoring it in the refresh until migration is done > 2014-04-09 14:19:34,268 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-74) VM web.judici > b0108933-deb2-4fa0-ae74-e10cefbb0cea moved from MigratingFrom --> Up > 2014-04-09 14:19:34,269 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-74) Adding VM > b0108933-deb2-4fa0-ae74-e10cefbb0cea to re-run list > 2014-04-09 14:19:34,269 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] > (DefaultQuartzScheduler_Worker-74) START, FullListVdsCommand(HostName > = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, > vds=Host[kvm01.virt], vmIds=[b0108933-deb2-4fa0-ae74-e10cefbb0cea]), > log id: 6936226f > 2014-04-09 14:19:34,279 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] > (DefaultQuartzScheduler_Worker-74) FINISH, FullListVdsCommand, return: > [{displaySecurePort=5905, kvmEnable=true, nicModel=rtl8139,pv, > keyboardLayout=en-us, displayIp=0, pauseCode=NOERR, nice=0, > pitReinjection=false, timeOffset=0, transparentHugePages=true, > vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, acpiEnable=true, custom={}, > spiceSslCipherSuite=DEFAULT, memSize=8192, displayPort=5904, smp=4, > emulatedMachine=rhel6.5.0, status=Up, vmType=kvm, > memGuaranteedSize=1024, display=qxl, pid=30289, smartcardEnable=false, > smpCoresPerSocket=1, > spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, > maxVCpus=160, clientIp=10.10.4.242, > devices=[Ljava.lang.Object;@610edb5f, vmName=web.judici, > cpuType=Penryn}], log id: 6936226f > 2014-04-09 14:19:34,281 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-74) Received a console Device without > an address when processing VM b0108933-deb2-4fa0-ae74-e10cefbb0cea > devices, skipping device: {alias=console0, specParams={}, > device=console, type=console, > deviceId=05fba965-133e-4810-a87c-1688be8beb5a} > 2014-04-09 14:19:34,294 ERROR > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-74) Rerun vm > b0108933-deb2-4fa0-ae74-e10cefbb0cea. Called from vds kvm01.virt > 2014-04-09 14:19:34,299 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-46) START, > MigrateStatusVDSCommand(HostName = kvm01.virt, HostId = > fa455012-b929-4805-a5c1-50e6c788039a, > vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea), log id: 16ec0461 > 2014-04-09 14:19:34,346 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-46) Failed in MigrateStatusVDS method > 2014-04-09 14:19:34,346 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-46) Command > org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand > return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=12, > mMessage=Fatal error during migration]] > 2014-04-09 14:19:34,347 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-46) HostName = kvm01.virt > 2014-04-09 14:19:34,348 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-46) Command > MigrateStatusVDSCommand(HostName = kvm01.virt, HostId = > fa455012-b929-4805-a5c1-50e6c788039a, > vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea) execution failed. > Exception: VDSErrorException: VDSGenericException: VDSErrorException: > Failed to MigrateStatusVDS, error = Fatal error during migration, code > = 12 > 2014-04-09 14:19:34,349 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-46) FINISH, MigrateStatusVDSCommand, > log id: 16ec0461 > 2014-04-09 14:19:34,353 INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (org.ovirt.thread.pool-6-thread-46) Correlation ID: 4350ffd, Job ID: > 433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event > ID: -1, Message: Migration failed due to Error: Fatal error during > migration. Trying to migrate to another Host (VM: web.judici, Source: > kvm01.virt, Destination: kvm04). > 2014-04-09 14:19:34,370 INFO > [org.ovirt.engine.core.bll.MigrateVmCommand] > (org.ovirt.thread.pool-6-thread-46) Running command: MigrateVmCommand > internal: false. Entities affected : ID: > b0108933-deb2-4fa0-ae74-e10cefbb0cea Type: VM > 2014-04-09 14:19:34,374 INFO > [org.ovirt.engine.core.bll.scheduling.policyunits.HaReservationWeightPolicyUnit] > (org.ovirt.thread.pool-6-thread-46) Started HA reservation scoring > method > 2014-04-09 14:19:34,378 INFO > [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] > (org.ovirt.thread.pool-6-thread-46) START, MigrateVDSCommand(HostName > = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, > vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31, > dstVdsId=071d8ef9-971e-499b-975a-255ee3d3f895, > dstHost=10.10.4.35:54321, migrationMethod=ONLINE, > tunnelMigration=false, migrationDowntime=0), log id: 7ac1f1eb > 2014-04-09 14:19:34,379 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] > (org.ovirt.thread.pool-6-thread-46) VdsBroker::migrate::Entered > (vm_guid=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31, > dstHost=10.10.4.35:54321, method=online > 2014-04-09 14:19:34,380 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] > (org.ovirt.thread.pool-6-thread-46) START, > MigrateBrokerVDSCommand(HostName = kvm01.virt, HostId = > fa455012-b929-4805-a5c1-50e6c788039a, > vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31, > dstVdsId=071d8ef9-971e-499b-975a-255ee3d3f895, > dstHost=10.10.4.35:54321, migrationMethod=ONLINE, > tunnelMigration=false, migrationDowntime=0), log id: 3c9d3817 > 2014-04-09 14:19:34,431 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] > (org.ovirt.thread.pool-6-thread-46) FINISH, MigrateBrokerVDSCommand, > log id: 3c9d3817 > 2014-04-09 14:19:34,434 INFO > [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] > (org.ovirt.thread.pool-6-thread-46) FINISH, MigrateVDSCommand, return: > MigratingFrom, log id: 7ac1f1eb > 2014-04-09 14:19:34,438 INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (org.ovirt.thread.pool-6-thread-46) Correlation ID: 4350ffd, Job ID: > 433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event > ID: -1, Message: Migration started (VM: web.judici, Source: > kvm01.virt, Destination: kvm05.virt, User: admin). > 2014-04-09 14:19:37,582 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-92) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:19:40,617 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-91) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:19:43,640 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-100) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:19:46,680 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-8) [1450e72d] RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:19:49,735 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-26) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:19:52,752 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-23) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:19:55,769 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-35) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:19:58,788 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-50) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:01,837 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-53) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:04,880 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-58) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:07,904 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-68) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:10,921 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-77) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:13,951 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-91) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:16,988 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-100) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:20,033 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-8) [1450e72d] RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:23,052 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-26) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:26,070 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-36) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:29,090 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-45) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:32,135 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-50) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:35,180 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-51) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:38,196 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-71) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:41,215 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-73) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:44,234 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-81) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:47,275 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-97) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:50,316 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-98) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:53,332 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-12) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:56,348 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-18) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:20:59,363 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-36) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:02,404 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-29) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:05,439 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-47) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:08,455 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-60) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:11,476 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-58) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:14,501 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-76) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:17,549 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-77) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:20,598 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-91) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:23,617 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-100) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:26,642 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-6) [5c26eeb2] RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:29,664 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-21) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:32,702 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-25) [e633f03] RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:35,751 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-31) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:38,772 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-48) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:41,790 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-60) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:44,812 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-71) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:47,858 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-72) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:50,903 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-77) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:53,919 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-94) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:56,938 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-5) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:21:59,961 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-14) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:02,997 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-21) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:06,037 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-36) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:09,058 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-45) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:12,075 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-50) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:15,103 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-51) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm05.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:17,696 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-58) VM web.judici > b0108933-deb2-4fa0-ae74-e10cefbb0cea moved from MigratingFrom --> Up > 2014-04-09 14:22:17,697 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-58) Adding VM > b0108933-deb2-4fa0-ae74-e10cefbb0cea to re-run list > 2014-04-09 14:22:17,698 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] > (DefaultQuartzScheduler_Worker-58) START, FullListVdsCommand(HostName > = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, > vds=Host[kvm01.virt], vmIds=[b0108933-deb2-4fa0-ae74-e10cefbb0cea]), > log id: 1688c794 > 2014-04-09 14:22:17,706 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] > (DefaultQuartzScheduler_Worker-58) FINISH, FullListVdsCommand, return: > [{displaySecurePort=5905, kvmEnable=true, nicModel=rtl8139,pv, > keyboardLayout=en-us, displayIp=0, pauseCode=NOERR, nice=0, > pitReinjection=false, timeOffset=0, transparentHugePages=true, > vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, acpiEnable=true, custom={}, > spiceSslCipherSuite=DEFAULT, memSize=8192, displayPort=5904, smp=4, > emulatedMachine=rhel6.5.0, status=Up, vmType=kvm, > memGuaranteedSize=1024, display=qxl, pid=30289, smartcardEnable=false, > smpCoresPerSocket=1, > spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, > maxVCpus=160, clientIp=10.10.4.242, > devices=[Ljava.lang.Object;@6de6d11a, vmName=web.judici, > cpuType=Penryn}], log id: 1688c794 > 2014-04-09 14:22:17,709 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-58) Received a console Device without > an address when processing VM b0108933-deb2-4fa0-ae74-e10cefbb0cea > devices, skipping device: {alias=console0, specParams={}, > device=console, type=console, > deviceId=05fba965-133e-4810-a87c-1688be8beb5a} > 2014-04-09 14:22:17,728 ERROR > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-58) Rerun vm > b0108933-deb2-4fa0-ae74-e10cefbb0cea. Called from vds kvm01.virt > 2014-04-09 14:22:17,735 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-29) START, > MigrateStatusVDSCommand(HostName = kvm01.virt, HostId = > fa455012-b929-4805-a5c1-50e6c788039a, > vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea), log id: 15822aeb > 2014-04-09 14:22:17,781 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-29) Failed in MigrateStatusVDS method > 2014-04-09 14:22:17,782 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-29) Command > org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand > return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=47, > mMessage=Migration canceled]] > 2014-04-09 14:22:17,782 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-29) HostName = kvm01.virt > 2014-04-09 14:22:17,783 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-29) Command > MigrateStatusVDSCommand(HostName = kvm01.virt, HostId = > fa455012-b929-4805-a5c1-50e6c788039a, > vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea) execution failed. > Exception: VDSErrorException: VDSGenericException: VDSErrorException: > Failed to MigrateStatusVDS, error = Migration canceled, code = 47 > 2014-04-09 14:22:17,784 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-29) FINISH, MigrateStatusVDSCommand, > log id: 15822aeb > 2014-04-09 14:22:17,791 INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (org.ovirt.thread.pool-6-thread-29) Correlation ID: 4350ffd, Job ID: > 433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event > ID: -1, Message: Migration failed due to Error: Migration not in > progress. Trying to migrate to another Host (VM: web.judici, Source: > kvm01.virt, Destination: kvm05.virt). > 2014-04-09 14:22:17,818 INFO > [org.ovirt.engine.core.bll.MigrateVmCommand] > (org.ovirt.thread.pool-6-thread-29) Running command: MigrateVmCommand > internal: false. Entities affected : ID: > b0108933-deb2-4fa0-ae74-e10cefbb0cea Type: VM > 2014-04-09 14:22:17,824 INFO > [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] > (org.ovirt.thread.pool-6-thread-29) START, MigrateVDSCommand(HostName > = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, > vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31, > dstVdsId=d5e2830d-e59c-49be-b6e8-0968dbf357aa, > dstHost=10.10.4.33:54321, migrationMethod=ONLINE, > tunnelMigration=false, migrationDowntime=0), log id: 7dc46595 > 2014-04-09 14:22:17,825 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] > (org.ovirt.thread.pool-6-thread-29) VdsBroker::migrate::Entered > (vm_guid=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31, > dstHost=10.10.4.33:54321, method=online > 2014-04-09 14:22:17,826 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] > (org.ovirt.thread.pool-6-thread-29) START, > MigrateBrokerVDSCommand(HostName = kvm01.virt, HostId = > fa455012-b929-4805-a5c1-50e6c788039a, > vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, srcHost=10.10.4.31, > dstVdsId=d5e2830d-e59c-49be-b6e8-0968dbf357aa, > dstHost=10.10.4.33:54321, migrationMethod=ONLINE, > tunnelMigration=false, migrationDowntime=0), log id: 25038825 > 2014-04-09 14:22:17,848 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] > (org.ovirt.thread.pool-6-thread-29) FINISH, MigrateBrokerVDSCommand, > log id: 25038825 > 2014-04-09 14:22:17,853 INFO > [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] > (org.ovirt.thread.pool-6-thread-29) FINISH, MigrateVDSCommand, return: > MigratingFrom, log id: 7dc46595 > 2014-04-09 14:22:17,859 INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (org.ovirt.thread.pool-6-thread-29) Correlation ID: 4350ffd, Job ID: > 433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event > ID: -1, Message: Migration started (VM: web.judici, Source: > kvm01.virt, Destination: kvm03.virt, User: admin). > 2014-04-09 14:22:19,226 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-67) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:22,275 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-79) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:25,291 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-83) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:28,314 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-1) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:31,330 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-6) [5c26eeb2] RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:34,360 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-16) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:37,389 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-30) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:40,407 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-31) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:43,425 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-52) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:46,443 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-51) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:49,473 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-71) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:52,514 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-76) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:55,533 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-92) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:22:58,553 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-90) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:01,569 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-10) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:04,596 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-7) [7a463a98] RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:07,626 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-16) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:10,648 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-30) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:13,665 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-41) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:16,693 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-47) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:19,724 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-51) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:22,758 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-65) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:25,778 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-79) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:28,795 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-95) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:31,814 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-87) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:34,848 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-1) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:37,884 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-19) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:40,910 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-22) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:43,930 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-25) [e633f03] RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:46,949 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-45) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:49,982 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-48) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:53,048 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-64) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:56,076 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-66) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:23:59,093 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-72) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:02,112 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-93) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:05,140 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-87) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:08,210 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-3) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:11,240 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-6) [5c26eeb2] RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:14,257 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-21) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:17,277 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-36) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:20,309 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-31) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:23,371 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-48) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:26,391 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-63) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:29,409 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-74) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:32,428 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-72) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:35,467 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-85) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:38,499 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-89) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-2014-04-09 14:24:38,499 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-89) RefreshVmList vm id > b0108933-deb2-4fa0-ae74- > e10cefbb0cea is migrating to vds kvm03.virt ignoring it in the refresh > until migration is done > 2014-04-09 14:24:41,534 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-1) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:44,561 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-11) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:47,584 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-21) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:50,631 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-25) [e633f03] RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:53,668 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-37) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:56,683 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-55) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:24:59,702 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-57) RefreshVmList vm id > b0108933-deb2-4fa0-ae74-e10cefbb0cea is migrating to vds kvm03.virt > ignoring it in the refresh until migration is done > 2014-04-09 14:25:00,974 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-59) VM web.judici > b0108933-deb2-4fa0-ae74-e10cefbb0cea moved from MigratingFrom --> Up > 2014-04-09 14:25:00,975 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-59) Adding VM > b0108933-deb2-4fa0-ae74-e10cefbb0cea to re-run list > 2014-04-09 14:25:00,976 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] > (DefaultQuartzScheduler_Worker-59) START, FullListVdsCommand(HostName > = kvm01.virt, HostId = fa455012-b929-4805-a5c1-50e6c788039a, > vds=Host[kvm01.virt], vmIds=[b0108933-deb2-4fa0-ae74-e10cefbb0cea]), > log id: 7ef5612e > 2014-04-09 14:25:00,984 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] > (DefaultQuartzScheduler_Worker-59) FINISH, FullListVdsCommand, return: > [{displaySecurePort=5905, kvmEnable=true, nicModel=rtl8139,pv, > keyboardLayout=en-us, displayIp=0, pauseCode=NOERR, nice=0, > pitReinjection=false, timeOffset=0, transparentHugePages=true, > vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea, acpiEnable=true, custom={}, > spiceSslCipherSuite=DEFAULT, memSize=8192, displayPort=5904, smp=4, > emulatedMachine=rhel6.5.0, status=Up, vmType=kvm, > memGuaranteedSize=1024, display=qxl, pid=30289, smartcardEnable=false, > smpCoresPerSocket=1, > spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, > maxVCpus=160, clientIp=10.10.4.242, > devices=[Ljava.lang.Object;@1fbf5420, vmName=web.judici, > cpuType=Penryn}], log id: 7ef5612e > 2014-04-09 14:25:00,988 INFO > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-59) Received a console Device without > an address when processing VM b0108933-deb2-4fa0-ae74-e10cefbb0cea > devices, skipping device: {alias=console0, specParams={}, > device=console, type=console, > deviceId=05fba965-133e-4810-a87c-1688be8beb5a} > 2014-04-09 14:25:01,030 ERROR > [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] > (DefaultQuartzScheduler_Worker-59) Rerun vm > b0108933-deb2-4fa0-ae74-e10cefbb0cea. Called from vds kvm01.virt > 2014-04-09 14:25:01,038 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-37) START, > MigrateStatusVDSCommand(HostName = kvm01.virt, HostId = > fa455012-b929-4805-a5c1-50e6c788039a, > vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea), log id: 47013f27 > 2014-04-09 14:25:01,045 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-37) Failed in MigrateStatusVDS method > 2014-04-09 14:25:01,045 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-37) Command > org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand > return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=47, > mMessage=Migration canceled]] > 2014-04-09 14:25:01,046 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-37) HostName = kvm01.virt > 2014-04-09 14:25:01,046 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-37) Command > MigrateStatusVDSCommand(HostName = kvm01.virt, HostId = > fa455012-b929-4805-a5c1-50e6c788039a, > vmId=b0108933-deb2-4fa0-ae74-e10cefbb0cea) execution failed. > Exception: VDSErrorException: VDSGenericException: VDSErrorException: > Failed to MigrateStatusVDS, error = Migration canceled, code = 47 > 2014-04-09 14:25:01,048 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-37) FINISH, MigrateStatusVDSCommand, > log id: 47013f27 > 2014-04-09 14:25:01,055 INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (org.ovirt.thread.pool-6-thread-37) Correlation ID: 4350ffd, Job ID: > 433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event > ID: -1, Message: Migration failed due to Error: Migration not in > progress. Trying to migrate to another Host (VM: web.judici, Source: > kvm01.virt, Destination: kvm03.virt). > 2014-04-09 14:25:01,061 INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (org.ovirt.thread.pool-6-thread-37) Correlation ID: 4350ffd, Job ID: > 433cbdce-90b3-41ed-bbfe-3a10ea2c6616, Call Stack: null, Custom Event > ID: -1, Message: Migration failed due to Error: Migration not in > progress (VM: web.judici, Source: kvm01.virt, Destination: > kvm03.virt). > 2014-04-09 14:34:08,665 INFO > [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] > (org.ovirt.thread.pool-6-thread-43) [4045389e] Running command: > MaintenanceNumberOfVdssCommand internal: false. Entities affected : > ID: fa455012-b929-4805-a5c1-50e6c788039a Type: VDS > 2014-04-09 14:34:08,667 INFO > [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-43) [4045389e] START, > SetVdsStatusVDSCommand(HostName = kvm01.virt, HostId = > fa455012-b929-4805-a5c1-50e6c788039a, status=PreparingForMaintenance, > nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: > 64965c26 > 2014-04-09 14:34:08,678 INFO > [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] > (org.ovirt.thread.pool-6-thread-43) [4045389e] FINISH, > SetVdsStatusVDSCommand, log id: 64965c26 > 2014-04-09 14:34:08,725 INFO > [org.ovirt.engine.core.bll.MaintenanceVdsCommand] > (org.ovirt.thread.pool-6-thread-43) [4045389e] Running command: > MaintenanceVdsCommand internal: true. Entities affected : ID: > fa455012-b929-4805-a5c1-50e6c788039a Type: VDS > 2014-04-09 14:34:08,744 INFO > [org.ovirt.engine.core.bll.InternalMigrateVmCommand] > (org.ovirt.thread.pool-6-thread-43) [35f06602] Failed to Acquire Lock > to object EngineLock [exclusiveLocks= key: > b0108933-deb2-4fa0-ae74-e10cefbb0cea value: VM > , sharedLocks= ] > > On Wed, Apr 9, 2014 at 6:33 PM, Yair Zaslavsky <yzaslavs(a)redhat.com&gt; wrote: > > Can you please attach full engine.log? > > > > Many thanks, > > Yair > > > > > > ----- Original Message ----- > >> From: "Jeremiah Jahn" <jeremiah(a)goodinassociates.com&gt; > >> To: users(a)ovirt.org > >> Sent: Thursday, April 10, 2014 2:18:48 AM > >> Subject: [ovirt-users] help.. vm trapped in limbo aka can't acquire > >> exclusive lock > >> > >> I can't start it, I can't migrate it. I tried to migrate it before, > >> but the machine was stuck in a read only state. The migration failed > >> because the machine it was being migrated to was also in a read only > >> state. somewhere in the process the lock obviously got lost, and I > >> can't get it back... > >> > >> > >> > >> 2014-04-09 18:11:16,675 INFO [org.ovirt.engine.core.bll.RunVmCommand] > >> (ajp--127.0.0.1-8702-3) [58b40832] Failed to Acquire Lock to object > >> EngineLock [exclusiveLocks= key: b0108933-deb2-4fa0-ae74-e10cefbb0cea > >> value: VM > >> , sharedLocks= ] > >> > >> > >> 2014-04-09 18:11:16,676 WARN [org.ovirt.engine.core.bll.RunVmCommand] > >> (ajp--127.0.0.1-8702-3) [58b40832] CanDoAction of action RunVm failed. > >> Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_IS_BEING_MIGRATED,$VmName > >> web.judici > >> _______________________________________________ > >> Users mailing list > >> Users(a)ovirt.org > >> http://lists.ovirt.org/mailman/listinfo/users > >> > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >