[ovirt-users] Migration failed due to Error: Fatal error during migration
Omer Frenkel
ofrenkel at redhat.com
Sun Aug 31 02:52:48 EDT 2014
----- Original Message -----
> From: "Punit Dambiwal" <hypunit at gmail.com>
> To: "Omer Frenkel" <ofrenkel at redhat.com>
> Cc: "Manfred Landauer" <manfred.landauer at fabasoft.com>, users at ovirt.org
> Sent: Friday, August 29, 2014 4:30:38 AM
> Subject: Re: [ovirt-users] Migration failed due to Error: Fatal error during migration
>
> Hi ,
>
> I am also facing the same issue...
>
> here is the engine logs :-
>
> 2014-08-29 09:27:45,432 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
> (org.ovirt.thread.pool-6-thread-24) FINISH, MigrateStatusVDSCommand, log
> id: 1f3e4161
> 2014-08-29 09:27:45,439 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (org.ovirt.thread.pool-6-thread-24) Correlation ID: 84ff2f4, Job ID:
> 8bc0b78a-c600-4f8d-98f8-66a46c66abe0, Call Stack: null, Custom Event ID:
> -1, Message: Migration failed due to Error: Fatal error during migration.
please send vdsm and libvirt log from the src host for this vm
> Trying to migrate to another Host (VM: bc16391ac105b7e68cccb47803906d0b,
> Source: compute4, Destination: compute3).
> 2014-08-29 09:27:45,536 INFO [org.ovirt.engine.core.bll.MigrateVmCommand]
> (org.ovirt.thread.pool-6-thread-24) Running command: MigrateVmCommand
> internal: false. Entities affected : ID:
> 6134b272-cd7f-43c1-a1b1-eaefe69c6b75 Type: VM
i see this vm did migrate successfully, right?
> 2014-08-29 09:27:45,610 INFO
> [org.ovirt.engine.core.bll.scheduling.policyunits.HaReservationWeightPolicyUnit]
> (org.ovirt.thread.pool-6-thread-24) Started HA reservation scoring method
> 2014-08-29 09:27:45,666 INFO
> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
> (org.ovirt.thread.pool-6-thread-24) START, MigrateVDSCommand(HostName =
> compute4, HostId = 3a7a4504-1434-4fd2-ac00-e8d12c043b37,
> vmId=6134b272-cd7f-43c1-a1b1-eaefe69c6b75, srcHost=compute4.3linux.com,
> dstVdsId=bcd2bd85-c501-4be4-9730-a8662462cab5, dstHost=
> compute2.3linux.com:54321, migrationMethod=ONLINE, tunnelMigration=false,
> migrationDowntime=0), log id: 640b0ccd
> 2014-08-29 09:27:45,667 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
> (org.ovirt.thread.pool-6-thread-24) VdsBroker::migrate::Entered
> (vm_guid=6134b272-cd7f-43c1-a1b1-eaefe69c6b75, srcHost=compute4.3linux.com,
> dstHost=compute2.3linux.com:54321, method=online
> 2014-08-29 09:27:45,684 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
> (org.ovirt.thread.pool-6-thread-24) START, MigrateBrokerVDSCommand(HostName
> = compute4, HostId = 3a7a4504-1434-4fd2-ac00-e8d12c043b37,
> vmId=6134b272-cd7f-43c1-a1b1-eaefe69c6b75, srcHost=compute4.3linux.com,
> dstVdsId=bcd2bd85-c501-4be4-9730-a8662462cab5, dstHost=
> compute2.3linux.com:54321, migrationMethod=ONLINE, tunnelMigration=false,
> migrationDowntime=0), log id: 1b3d3891
> 2014-08-29 09:27:45,702 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
> (org.ovirt.thread.pool-6-thread-24) FINISH, MigrateBrokerVDSCommand, log
> id: 1b3d3891
> 2014-08-29 09:27:45,707 INFO
> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
> (org.ovirt.thread.pool-6-thread-24) FINISH, MigrateVDSCommand, return:
> MigratingFrom, log id: 640b0ccd
> 2014-08-29 09:27:45,711 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (org.ovirt.thread.pool-6-thread-24) Correlation ID: 84ff2f4, Job ID:
> 8bc0b78a-c600-4f8d-98f8-66a46c66abe0, Call Stack: null, Custom Event ID:
> -1, Message: Migration started (VM: bc16391ac105b7e68cccb47803906d0b,
> Source: compute4, Destination: compute2, User: admin).
> 2014-08-29 09:27:47,143 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> (DefaultQuartzScheduler_Worker-28) START,
> GlusterVolumesListVDSCommand(HostName = compute4, HostId =
> 3a7a4504-1434-4fd2-ac00-e8d12c043b37), log id: 9e34612
> 2014-08-29 09:27:47,300 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> (DefaultQuartzScheduler_Worker-28) FINISH, GlusterVolumesListVDSCommand,
> return:
> {e6117925-79b1-417b-9d07-cfc31f68bc51=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at b42f1b41},
> log id: 9e34612
> 2014-08-29 09:27:48,306 INFO
> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler_Worker-51) RefreshVmList vm id
> 6134b272-cd7f-43c1-a1b1-eaefe69c6b75 is migrating to vds compute2 ignoring
> it in the refresh until migration is done
> 2014-08-29 09:27:51,349 INFO
> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler_Worker-52) RefreshVmList vm id
> 6134b272-cd7f-43c1-a1b1-eaefe69c6b75 is migrating to vds compute2 ignoring
> it in the refresh until migration is done
> 2014-08-29 09:27:52,470 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> (DefaultQuartzScheduler_Worker-49) START,
> GlusterVolumesListVDSCommand(HostName = compute4, HostId =
> 3a7a4504-1434-4fd2-ac00-e8d12c043b37), log id: 7cbb4cd
> 2014-08-29 09:27:52,624 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> (DefaultQuartzScheduler_Worker-49) FINISH, GlusterVolumesListVDSCommand,
> return:
> {e6117925-79b1-417b-9d07-cfc31f68bc51=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at debf8bab},
> log id: 7cbb4cd
> 2014-08-29 09:27:53,557 INFO [org.ovirt.engine.core.bll.LoginUserCommand]
> (ajp--127.0.0.1-8702-14) Running command: LoginUserCommand internal: false.
> 2014-08-29 09:27:53,561 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (ajp--127.0.0.1-8702-14) Correlation ID: null, Call Stack: null, Custom
> Event ID: -1, Message: User admin logged in.
> 2014-08-29 09:27:53,567 INFO [org.ovirt.engine.core.bll.LoginUserCommand]
> (ajp--127.0.0.1-8702-4) Running command: LoginUserCommand internal: false.
> 2014-08-29 09:27:53,661 INFO [org.ovirt.engine.core.bll.LogoutUserCommand]
> (ajp--127.0.0.1-8702-14) [52b6e08a] Running command: LogoutUserCommand
> internal: false.
> 2014-08-29 09:27:53,666 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (ajp--127.0.0.1-8702-14) [52b6e08a] Correlation ID: 52b6e08a, Call Stack:
> null, Custom Event ID: -1, Message: User admin logged out.
> 2014-08-29 09:27:53,667 INFO [org.ovirt.engine.core.bll.LogoutUserCommand]
> (ajp--127.0.0.1-8702-4) [55036013] Running command: LogoutUserCommand
> internal: false.
> 2014-08-29 09:27:53,671 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (ajp--127.0.0.1-8702-4) [55036013] Correlation ID: 55036013, Call Stack:
> null, Custom Event ID: -1, Message: User admin logged out.
> 2014-08-29 09:27:54,378 INFO
> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler_Worker-79) RefreshVmList vm id
> 6134b272-cd7f-43c1-a1b1-eaefe69c6b75 is migrating to vds compute2 ignoring
> it in the refresh until migration is done
> 2014-08-29 09:27:57,408 INFO
> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler_Worker-16) RefreshVmList vm id
> 6134b272-cd7f-43c1-a1b1-eaefe69c6b75 is migrating to vds compute2 ignoring
> it in the refresh until migration is done
> 2014-08-29 09:27:57,795 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> (DefaultQuartzScheduler_Worker-2) START,
> GlusterVolumesListVDSCommand(HostName = compute4, HostId =
> 3a7a4504-1434-4fd2-ac00-e8d12c043b37), log id: 6e95a40d
> 2014-08-29 09:27:57,954 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> (DefaultQuartzScheduler_Worker-2) FINISH, GlusterVolumesListVDSCommand,
> return:
> {e6117925-79b1-417b-9d07-cfc31f68bc51=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 8146b6ae},
> log id: 6e95a40d
> 2014-08-29 09:27:58,556 INFO [org.ovirt.engine.core.bll.LoginUserCommand]
> (ajp--127.0.0.1-8702-8) Running command: LoginUserCommand internal: false.
> 2014-08-29 09:27:58,654 INFO [org.ovirt.engine.core.bll.LogoutUserCommand]
> (ajp--127.0.0.1-8702-8) [6db16c8b] Running command: LogoutUserCommand
> internal: false.
> 2014-08-29 09:27:58,662 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (ajp--127.0.0.1-8702-8) [6db16c8b] Correlation ID: 6db16c8b, Call Stack:
> null, Custom Event ID: -1, Message: User admin logged out.
> 2014-08-29 09:28:00,440 INFO
> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler_Worker-12) RefreshVmList vm id
> 6134b272-cd7f-43c1-a1b1-eaefe69c6b75 is migrating to vds compute2 ignoring
> it in the refresh until migration is done
> 2014-08-29 09:28:00,915 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand]
> (DefaultQuartzScheduler_Worker-93) START,
> GlusterTasksListVDSCommand(HostName = compute2, HostId =
> bcd2bd85-c501-4be4-9730-a8662462cab5), log id: 5b36bfa6
> 2014-08-29 09:28:00,974 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand]
> (DefaultQuartzScheduler_Worker-93) FINISH, GlusterTasksListVDSCommand,
> return:
> [GlusterAsyncTask[6c826fa1-8be8-4b4b-b1cc-d93d4299451f-REBALANCE-FINISHED]],
> log id: 5b36bfa6
> 2014-08-29 09:28:00,992 WARN
> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (DefaultQuartzScheduler_Worker-93) Trying to release exclusive lock which
> does not exist, lock key: e6117925-79b1-417b-9d07-cfc31f68bc51GLUSTER
> 2014-08-29 09:28:03,128 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> (DefaultQuartzScheduler_Worker-85) START,
> GlusterVolumesListVDSCommand(HostName = compute4, HostId =
> 3a7a4504-1434-4fd2-ac00-e8d12c043b37), log id: 6836617e
> 2014-08-29 09:28:03,298 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> (DefaultQuartzScheduler_Worker-85) FINISH, GlusterVolumesListVDSCommand,
> return:
> {e6117925-79b1-417b-9d07-cfc31f68bc51=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 658c0e64},
> log id: 6836617e
> 2014-08-29 09:28:03,488 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand]
> (DefaultQuartzScheduler_Worker-55) START, FullListVdsCommand(HostName =
> compute2, HostId = bcd2bd85-c501-4be4-9730-a8662462cab5,
> vds=Host[compute2,bcd2bd85-c501-4be4-9730-a8662462cab5],
> vmIds=[6134b272-cd7f-43c1-a1b1-eaefe69c6b75]), log id: 7bd8658c
> 2014-08-29 09:28:03,495 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand]
> (DefaultQuartzScheduler_Worker-55) FINISH, FullListVdsCommand, return:
> [{displaySecurePort=-1, kvmEnable=true, nicModel=rtl8139,pv,
> afterMigrationStatus=, keyboardLayout=en-us, displayIp=43.252.176.14,
> pitReinjection=false, nice=0, displayNetwork=ovirtmgmt, timeOffset=0,
> transparentHugePages=true, vmId=6134b272-cd7f-43c1-a1b1-eaefe69c6b75,
> acpiEnable=true, cpuShares=2, custom={}, spiceSslCipherSuite=DEFAULT,
> memSize=4096, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0,
> vmType=kvm, status=Up, memGuaranteedSize=512, display=vnc, pid=5718,
> smartcardEnable=false, tabletEnable=true, smpCoresPerSocket=1,
> spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,
> maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@53ee8569,
> vmName=bc16391ac105b7e68cccb47803906d0b, cpuType=Westmere}], log id:
> 7bd8658c
> 2014-08-29 09:28:03,548 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler_Worker-55) Correlation ID: 84ff2f4, Job ID:
> 8bc0b78a-c600-4f8d-98f8-66a46c66abe0, Call Stack: null, Custom Event ID:
> -1, Message: Migration completed (VM: bc16391ac105b7e68cccb47803906d0b,
> Source: compute4, Destination: compute2, Duration: 17 sec).
> 2014-08-29 09:28:03,551 INFO [org.ovirt.engine.core.bll.LoginUserCommand]
> (ajp--127.0.0.1-8702-1) Running command: LoginUserCommand internal: false.
> 2014-08-29 09:28:03,553 INFO [org.ovirt.engine.core.bll.MigrateVmCommand]
> (DefaultQuartzScheduler_Worker-55) Lock freed to object EngineLock
> [exclusiveLocks= key: 6134b272-cd7f-43c1-a1b1-eaefe69c6b75 value: VM
> , sharedLocks= ]
> 2014-08-29 09:28:03,571 INFO
> [org.ovirt.engine.core.vdsbroker.FailedToRunVmVDSCommand]
> (org.ovirt.thread.pool-6-thread-10) START, FailedToRunVmVDSCommand(HostName
> = compute3, HostId = fb492af0-3489-4c15-bb9d-6e3829cb536c), log id: 52897093
> 2014-08-29 09:28:03,572 INFO
> [org.ovirt.engine.core.vdsbroker.FailedToRunVmVDSCommand]
> (org.ovirt.thread.pool-6-thread-10) FINISH, FailedToRunVmVDSCommand, log
> id: 52897093
> 2014-08-29 09:28:03,651 INFO [org.ovirt.engine.core.bll.LogoutUserCommand]
> (ajp--127.0.0.1-8702-1) [5c34822e] Running command: LogoutUserCommand
> internal: false.
> 2014-08-29 09:28:03,656 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (ajp--127.0.0.1-8702-1) [5c34822e] Correlation ID: 5c34822e, Call Stack:
> null, Custom Event ID: -1, Message: User admin logged out.
> 2014-08-29 09:28:03,716 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
> (DefaultQuartzScheduler_Worker-22) START, DestroyVDSCommand(HostName =
> compute4, HostId = 3a7a4504-1434-4fd2-ac00-e8d12c043b37,
> vmId=6134b272-cd7f-43c1-a1b1-eaefe69c6b75, force=false, secondsToWait=0,
> gracefully=false), log id: 7484552f
> 2014-08-29 09:28:03,769 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
> (DefaultQuartzScheduler_Worker-22) FINISH, DestroyVDSCommand, log id:
> 7484552f
> 2014-08-29 09:28:03,770 INFO
> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler_Worker-22) RefreshVmList vm id
> 6134b272-cd7f-43c1-a1b1-eaefe69c6b75 status = Down on vds compute4 ignoring
> it in the refresh until migration is done
> 2014-08-29 09:28:08,470 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> (DefaultQuartzScheduler_Worker-71) START,
> GlusterVolumesListVDSCommand(HostName = compute4, HostId =
> 3a7a4504-1434-4fd2-ac00-e8d12c043b37), log id: 6293b194
> 2014-08-29 09:28:08,564 INFO [org.ovirt.engine.core.bll.LoginUserCommand]
> (ajp--127.0.0.1-8702-6) Running command: LoginUserCommand internal: false.
> 2014-08-29 09:28:08,571 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (ajp--127.0.0.1-8702-6) Correlation ID: null, Call Stack: null, Custom
> Event ID: -1, Message: User admin logged in.
>
>
>
> On Thu, Aug 28, 2014 at 7:48 PM, Omer Frenkel <ofrenkel at redhat.com> wrote:
>
> >
> >
> > ----- Original Message -----
> > > From: "Manfred Landauer" <manfred.landauer at fabasoft.com>
> > > To: users at ovirt.org
> > > Sent: Thursday, August 14, 2014 6:29:21 PM
> > > Subject: [ovirt-users] Migration failed due to Error: Fatal error
> > during migration
> > >
> > >
> > >
> > > Hi all
> > >
> > >
> > >
> > > When we try to migrate a VM on oVirt “Engine Version: 3.4.3-1.el6 ” form
> > host
> > > A to host B we’ll get this Errormessage: “Migration failed due to Error:
> > > Fatal error during migration”.
> > >
> > >
> > >
> > > It looks like, this occurs only when thin provisioned HDD’s attached to
> > the
> > > VM. VM’s with preallocated HDD’s attached, migrate without a problem.
> > >
> > >
> > >
> > > Hope someone can help us to solve this issue.
> > >
> > >
> >
> > it looks more like a network error:
> > Thread-3810747::ERROR::2014-08-14 16:48:45,471::vm::337::vm.Vm::(run)
> > vmId=`494f5edc-7edd-4300-a675-f0a8883265e4`::Failed to migrate
> > Traceback (most recent call last):
> > File "/usr/share/vdsm/vm.py", line 323, in run
> > self._startUnderlyingMigration()
> > File "/usr/share/vdsm/vm.py", line 400, in _startUnderlyingMigration
> > None, maxBandwidth)
> > File "/usr/share/vdsm/vm.py", line 838, in f
> > ret = attr(*args, **kwargs)
> > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py",
> > line 76, in wrapper
> > ret = f(*args, **kwargs)
> > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1178, in
> > migrateToURI2
> > if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed',
> > dom=self)
> > libvirtError: Unable to read from monitor: Connection reset by peer
> >
> > could you attach the libvirt log?
> >
> > >
> > > Best regards
> > >
> > > Manfred
> > >
> > >
> > >
> > > _______________________________________________
> > > Users mailing list
> > > Users at ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/users
> > >
> > _______________________________________________
> > Users mailing list
> > Users at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
>
More information about the Users
mailing list