[Users] Failing migration, inconsistent state

[Martijn Grendelman]

Martijn Grendelman schreef op 5-12-2013 11:00:
> Hi,
> 
> I tried to migrate several VMs from one host to another. Two VMs
> migrated without issues, but for one VM, the migration didn't happen. It
> seems to be hanging, but the UI is now in an inconsistent state:
> 
> - The 'Tasks' tab reports 0 active tasks, but the last task (the
> migration in question) is still reported as 'Executing'.
> - The VM status is 'Up' (not migrating)
> - 'Migrate' action is choosable from menu, while 'Cancel Migration' is
> greyed out, but when I choose 'Migrate' and pick a host, I am told
> 'Cannot migrate VM. VM <name> is being migrated.'.
> 
> What is the best way to fix this?

And perhaps this information is useful to oVirt developers:


> Thread-600648::DEBUG::2013-12-03 12:37:47,926::vm::180::vm.Vm::(_setupVdsConnection) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::Destination server is: onode0.isaac.local:54321
> Thread-600648::DEBUG::2013-12-03 12:37:47,927::vm::182::vm.Vm::(_setupVdsConnection) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::Initiating connection with destination
> Thread-600648::DEBUG::2013-12-03 12:37:47,990::vm::232::vm.Vm::(_prepareGuest) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::Migration started
> Thread-600648::DEBUG::2013-12-03 12:37:48,006::vm::299::vm.Vm::(run) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::migration semaphore acquired
> Thread-600648::DEBUG::2013-12-03 12:37:48,115::vm::357::vm.Vm::(_startUnderlyingMigration) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::starting migration to qemu+tls://onode0.isaac.local/system with miguri tcp://onode0.isaac.local
> Thread-600648::DEBUG::2013-12-03 12:43:10,819::libvirtconnection::108::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 9 edom: 10 level: 2 message: operation failed: migration job: unexpectedly failed
> Thread-600648::DEBUG::2013-12-03 12:43:10,819::vm::742::vm.Vm::(cancel) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::canceling migration downtime thread
> Thread-600648::DEBUG::2013-12-03 12:43:10,819::vm::812::vm.Vm::(stop) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::stopping migration monitor thread
> Thread-600648::ERROR::2013-12-03 12:43:10,820::vm::238::vm.Vm::(_recover) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::operation failed: migration job: unexpectedly failed
> Thread-600648::ERROR::2013-12-03 12:43:11,276::vm::321::vm.Vm::(run) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::Failed to migrate
> Traceback (most recent call last):
>   File "/usr/share/vdsm/vm.py", line 308, in run
>     self._startUnderlyingMigration()
>   File "/usr/share/vdsm/vm.py", line 385, in _startUnderlyingMigration
>     None, maxBandwidth)
>   File "/usr/share/vdsm/vm.py", line 835, in f
>     ret = attr(*args, **kwargs)
>   File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper
>     ret = f(*args, **kwargs)
>   File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1178, in migrateToURI2
>     if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
> libvirtError: operation failed: migration job: unexpectedly failed

Cheers,
Martijn.