[Users] Failing migration, inconsistent state

[Roy Golan]

On Thu 05 Dec 2013 12:18:51 PM IST, Martijn Grendelman wrote:
> Martijn Grendelman schreef op 5-12-2013 11:00:
>> Hi,
>>
>> I tried to migrate several VMs from one host to another. Two VMs
>> migrated without issues, but for one VM, the migration didn't happen. It
>> seems to be hanging, but the UI is now in an inconsistent state:
>>
>> - The 'Tasks' tab reports 0 active tasks, but the last task (the
>> migration in question) is still reported as 'Executing'.
>> - The VM status is 'Up' (not migrating)
>> - 'Migrate' action is choosable from menu, while 'Cancel Migration' is
>> greyed out, but when I choose 'Migrate' and pick a host, I am told
>> 'Cannot migrate VM. VM <name> is being migrated.'.
>>
>> What is the best way to fix this?
>
> And perhaps this information is useful to oVirt developers:
>
>
>> Thread-600648::DEBUG::2013-12-03 12:37:47,926::vm::180::vm.Vm::(_setupVdsConnection) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::Destination server is: onode0.isaac.local:54321
>> Thread-600648::DEBUG::2013-12-03 12:37:47,927::vm::182::vm.Vm::(_setupVdsConnection) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::Initiating connection with destination
>> Thread-600648::DEBUG::2013-12-03 12:37:47,990::vm::232::vm.Vm::(_prepareGuest) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::Migration started
>> Thread-600648::DEBUG::2013-12-03 12:37:48,006::vm::299::vm.Vm::(run) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::migration semaphore acquired
>> Thread-600648::DEBUG::2013-12-03 12:37:48,115::vm::357::vm.Vm::(_startUnderlyingMigration) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::starting migration to qemu+tls://onode0.isaac.local/system with miguri tcp://onode0.isaac.local
>> Thread-600648::DEBUG::2013-12-03 12:43:10,819::libvirtconnection::108::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 9 edom: 10 level: 2 message: operation failed: migration job: unexpectedly failed
>> Thread-600648::DEBUG::2013-12-03 12:43:10,819::vm::742::vm.Vm::(cancel) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::canceling migration downtime thread
>> Thread-600648::DEBUG::2013-12-03 12:43:10,819::vm::812::vm.Vm::(stop) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::stopping migration monitor thread
>> Thread-600648::ERROR::2013-12-03 12:43:10,820::vm::238::vm.Vm::(_recover) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::operation failed: migration job: unexpectedly failed
>> Thread-600648::ERROR::2013-12-03 12:43:11,276::vm::321::vm.Vm::(run) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::Failed to migrate
>> Traceback (most recent call last):
>>    File "/usr/share/vdsm/vm.py", line 308, in run
>>      self._startUnderlyingMigration()
>>    File "/usr/share/vdsm/vm.py", line 385, in _startUnderlyingMigration
>>      None, maxBandwidth)
>>    File "/usr/share/vdsm/vm.py", line 835, in f
>>      ret = attr(*args, **kwargs)
>>    File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper
>>      ret = f(*args, **kwargs)
>>    File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1178, in migrateToURI2
>>      if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
>> libvirtError: operation failed: migration job: unexpectedly failed
>
> Cheers,
> Martijn.
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

then engine->vdsm call to migrate succeed but the async underling call 
failed. so the state change
the engine should expect is UP -> MigratingFrom -> UP  --- which means 
migration failure.

can you supply also the /var/log/ovirt-engine/engine.log and vdsm.log?  
I want to know why a restart solved your problem.

Thanks,
Roy