Re: [ovirt-users] Migration failed due to Error: Fatal error during migration

Hi , I am also facing the same issue... here is the engine logs :- 2014-08-29 09:27:45,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-6-thread-24) FINISH, MigrateStatusVDSCommand, log id: 1f3e4161 2014-08-29 09:27:45,439 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) Correlation ID: 84ff2f4, Job ID: 8bc0b78a-c600-4f8d-98f8-66a46c66abe0, Call Stack: null, Custom Event ID: -1, Message: Migration failed due to Error: Fatal error during migration. Trying to migrate to another Host (VM: bc16391ac105b7e68cccb47803906d0b, Source: compute4, Destination: compute3). 2014-08-29 09:27:45,536 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-24) Running command: MigrateVmCommand internal: false. Entities affected : ID: 6134b272-cd7f-43c1-a1b1-eaefe69c6b75 Type: VM 2014-08-29 09:27:45,610 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.HaReservationWeightPolicyUnit] (org.ovirt.thread.pool-6-thread-24) Started HA reservation scoring method 2014-08-29 09:27:45,666 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-24) START, MigrateVDSCommand(HostName = compute4, HostId = 3a7a4504-1434-4fd2-ac00-e8d12c043b37, vmId=6134b272-cd7f-43c1-a1b1-eaefe69c6b75, srcHost=compute4.3linux.com, dstVdsId=bcd2bd85-c501-4be4-9730-a8662462cab5, dstHost= compute2.3linux.com:54321, migrationMethod=ONLINE, tunnelMigration=false, migrationDowntime=0), log id: 640b0ccd 2014-08-29 09:27:45,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-24) VdsBroker::migrate::Entered (vm_guid=6134b272-cd7f-43c1-a1b1-eaefe69c6b75, srcHost=compute4.3linux.com, dstHost=compute2.3linux.com:54321, method=online 2014-08-29 09:27:45,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-24) START, MigrateBrokerVDSCommand(HostName = compute4, HostId = 3a7a4504-1434-4fd2-ac00-e8d12c043b37, vmId=6134b272-cd7f-43c1-a1b1-eaefe69c6b75, srcHost=compute4.3linux.com, dstVdsId=bcd2bd85-c501-4be4-9730-a8662462cab5, dstHost= compute2.3linux.com:54321, migrationMethod=ONLINE, tunnelMigration=false, migrationDowntime=0), log id: 1b3d3891 2014-08-29 09:27:45,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-24) FINISH, MigrateBrokerVDSCommand, log id: 1b3d3891 2014-08-29 09:27:45,707 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-24) FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 640b0ccd 2014-08-29 09:27:45,711 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) Correlation ID: 84ff2f4, Job ID: 8bc0b78a-c600-4f8d-98f8-66a46c66abe0, Call Stack: null, Custom Event ID: -1, Message: Migration started (VM: bc16391ac105b7e68cccb47803906d0b, Source: compute4, Destination: compute2, User: admin). 2014-08-29 09:27:47,143 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-28) START, GlusterVolumesListVDSCommand(HostName = compute4, HostId = 3a7a4504-1434-4fd2-ac00-e8d12c043b37), log id: 9e34612 2014-08-29 09:27:47,300 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-28) FINISH, GlusterVolumesListVDSCommand, return: {e6117925-79b1-417b-9d07-cfc31f68bc51=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b42f1b41}, log id: 9e34612 2014-08-29 09:27:48,306 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) RefreshVmList vm id 6134b272-cd7f-43c1-a1b1-eaefe69c6b75 is migrating to vds compute2 ignoring it in the refresh until migration is done 2014-08-29 09:27:51,349 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) RefreshVmList vm id 6134b272-cd7f-43c1-a1b1-eaefe69c6b75 is migrating to vds compute2 ignoring it in the refresh until migration is done 2014-08-29 09:27:52,470 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-49) START, GlusterVolumesListVDSCommand(HostName = compute4, HostId = 3a7a4504-1434-4fd2-ac00-e8d12c043b37), log id: 7cbb4cd 2014-08-29 09:27:52,624 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, GlusterVolumesListVDSCommand, return: {e6117925-79b1-417b-9d07-cfc31f68bc51=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@debf8bab}, log id: 7cbb4cd 2014-08-29 09:27:53,557 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-14) Running command: LoginUserCommand internal: false. 2014-08-29 09:27:53,561 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-14) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. 2014-08-29 09:27:53,567 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false. 2014-08-29 09:27:53,661 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-14) [52b6e08a] Running command: LogoutUserCommand internal: false. 2014-08-29 09:27:53,666 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-14) [52b6e08a] Correlation ID: 52b6e08a, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. 2014-08-29 09:27:53,667 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-4) [55036013] Running command: LogoutUserCommand internal: false. 2014-08-29 09:27:53,671 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) [55036013] Correlation ID: 55036013, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. 2014-08-29 09:27:54,378 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) RefreshVmList vm id 6134b272-cd7f-43c1-a1b1-eaefe69c6b75 is migrating to vds compute2 ignoring it in the refresh until migration is done 2014-08-29 09:27:57,408 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) RefreshVmList vm id 6134b272-cd7f-43c1-a1b1-eaefe69c6b75 is migrating to vds compute2 ignoring it in the refresh until migration is done 2014-08-29 09:27:57,795 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-2) START, GlusterVolumesListVDSCommand(HostName = compute4, HostId = 3a7a4504-1434-4fd2-ac00-e8d12c043b37), log id: 6e95a40d 2014-08-29 09:27:57,954 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-2) FINISH, GlusterVolumesListVDSCommand, return: {e6117925-79b1-417b-9d07-cfc31f68bc51=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@8146b6ae}, log id: 6e95a40d 2014-08-29 09:27:58,556 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false. 2014-08-29 09:27:58,654 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-8) [6db16c8b] Running command: LogoutUserCommand internal: false. 2014-08-29 09:27:58,662 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-8) [6db16c8b] Correlation ID: 6db16c8b, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. 2014-08-29 09:28:00,440 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) RefreshVmList vm id 6134b272-cd7f-43c1-a1b1-eaefe69c6b75 is migrating to vds compute2 ignoring it in the refresh until migration is done 2014-08-29 09:28:00,915 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-93) START, GlusterTasksListVDSCommand(HostName = compute2, HostId = bcd2bd85-c501-4be4-9730-a8662462cab5), log id: 5b36bfa6 2014-08-29 09:28:00,974 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, GlusterTasksListVDSCommand, return: [GlusterAsyncTask[6c826fa1-8be8-4b4b-b1cc-d93d4299451f-REBALANCE-FINISHED]], log id: 5b36bfa6 2014-08-29 09:28:00,992 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-93) Trying to release exclusive lock which does not exist, lock key: e6117925-79b1-417b-9d07-cfc31f68bc51GLUSTER 2014-08-29 09:28:03,128 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-85) START, GlusterVolumesListVDSCommand(HostName = compute4, HostId = 3a7a4504-1434-4fd2-ac00-e8d12c043b37), log id: 6836617e 2014-08-29 09:28:03,298 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-85) FINISH, GlusterVolumesListVDSCommand, return: {e6117925-79b1-417b-9d07-cfc31f68bc51=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@658c0e64}, log id: 6836617e 2014-08-29 09:28:03,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-55) START, FullListVdsCommand(HostName = compute2, HostId = bcd2bd85-c501-4be4-9730-a8662462cab5, vds=Host[compute2,bcd2bd85-c501-4be4-9730-a8662462cab5], vmIds=[6134b272-cd7f-43c1-a1b1-eaefe69c6b75]), log id: 7bd8658c 2014-08-29 09:28:03,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-55) FINISH, FullListVdsCommand, return: [{displaySecurePort=-1, kvmEnable=true, nicModel=rtl8139,pv, afterMigrationStatus=, keyboardLayout=en-us, displayIp=43.252.176.14, pitReinjection=false, nice=0, displayNetwork=ovirtmgmt, timeOffset=0, transparentHugePages=true, vmId=6134b272-cd7f-43c1-a1b1-eaefe69c6b75, acpiEnable=true, cpuShares=2, custom={}, spiceSslCipherSuite=DEFAULT, memSize=4096, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=512, display=vnc, pid=5718, smartcardEnable=false, tabletEnable=true, smpCoresPerSocket=1, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@53ee8569, vmName=bc16391ac105b7e68cccb47803906d0b, cpuType=Westmere}], log id: 7bd8658c 2014-08-29 09:28:03,548 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) Correlation ID: 84ff2f4, Job ID: 8bc0b78a-c600-4f8d-98f8-66a46c66abe0, Call Stack: null, Custom Event ID: -1, Message: Migration completed (VM: bc16391ac105b7e68cccb47803906d0b, Source: compute4, Destination: compute2, Duration: 17 sec). 2014-08-29 09:28:03,551 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false. 2014-08-29 09:28:03,553 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (DefaultQuartzScheduler_Worker-55) Lock freed to object EngineLock [exclusiveLocks= key: 6134b272-cd7f-43c1-a1b1-eaefe69c6b75 value: VM , sharedLocks= ] 2014-08-29 09:28:03,571 INFO [org.ovirt.engine.core.vdsbroker.FailedToRunVmVDSCommand] (org.ovirt.thread.pool-6-thread-10) START, FailedToRunVmVDSCommand(HostName = compute3, HostId = fb492af0-3489-4c15-bb9d-6e3829cb536c), log id: 52897093 2014-08-29 09:28:03,572 INFO [org.ovirt.engine.core.vdsbroker.FailedToRunVmVDSCommand] (org.ovirt.thread.pool-6-thread-10) FINISH, FailedToRunVmVDSCommand, log id: 52897093 2014-08-29 09:28:03,651 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-1) [5c34822e] Running command: LogoutUserCommand internal: false. 2014-08-29 09:28:03,656 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) [5c34822e] Correlation ID: 5c34822e, Call Stack: null, Custom Event ID: -1, Message: User admin logged out. 2014-08-29 09:28:03,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-22) START, DestroyVDSCommand(HostName = compute4, HostId = 3a7a4504-1434-4fd2-ac00-e8d12c043b37, vmId=6134b272-cd7f-43c1-a1b1-eaefe69c6b75, force=false, secondsToWait=0, gracefully=false), log id: 7484552f 2014-08-29 09:28:03,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-22) FINISH, DestroyVDSCommand, log id: 7484552f 2014-08-29 09:28:03,770 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) RefreshVmList vm id 6134b272-cd7f-43c1-a1b1-eaefe69c6b75 status = Down on vds compute4 ignoring it in the refresh until migration is done 2014-08-29 09:28:08,470 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-71) START, GlusterVolumesListVDSCommand(HostName = compute4, HostId = 3a7a4504-1434-4fd2-ac00-e8d12c043b37), log id: 6293b194 2014-08-29 09:28:08,564 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false. 2014-08-29 09:28:08,571 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. On Thu, Aug 28, 2014 at 7:48 PM, Omer Frenkel <ofrenkel(a)redhat.com&gt; wrote: