[Users] VM Migration failed

Fri Jan 17 22:21:41 UTC 2014

On 01/18/2014 12:07 AM, Meital Bourvine wrote:
> I opened a bug about missing info in UI:
> https://bugzilla.redhat.com/show_bug.cgi?id=1054994
>
> It actually failed with this error:
> Thread-1417::DEBUG::2014-01-17 17:01:28,344::vm::768::vm.Vm::(run) vmId=`b8787906-187a-4234-a0c9-58fc4ddf2a57`::starting migration monitor thread
> Thread-1415::DEBUG::2014-01-17 17:01:28,409::libvirtconnection::108::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 38 edom: 42 level: 2 message: Failed to inquire lock: No such process
> Thread-1415::DEBUG::2014-01-17 17:01:28,409::vm::745::vm.Vm::(cancel) vmId=`b8787906-187a-4234-a0c9-58fc4ddf2a57`::canceling migration downtime thread
> Thread-1415::DEBUG::2014-01-17 17:01:28,409::vm::815::vm.Vm::(stop) vmId=`b8787906-187a-4234-a0c9-58fc4ddf2a57`::stopping migration monitor thread
> Thread-1416::DEBUG::2014-01-17 17:01:28,409::vm::742::vm.Vm::(run) vmId=`b8787906-187a-4234-a0c9-58fc4ddf2a57`::migration downtime thread exiting
> Thread-1415::ERROR::2014-01-17 17:01:28,410::vm::238::vm.Vm::(_recover) vmId=`b8787906-187a-4234-a0c9-58fc4ddf2a57`::Failed to inquire lock: No such process
> Thread-1415::ERROR::2014-01-17 17:01:28,619::vm::337::vm.Vm::(run) vmId=`b8787906-187a-4234-a0c9-58fc4ddf2a57`::Failed to migrate
> Traceback (most recent call last):
>    File "/usr/share/vdsm/vm.py", line 323, in run
>      self._startUnderlyingMigration()
>    File "/usr/share/vdsm/vm.py", line 400, in _startUnderlyingMigration
>      None, maxBandwidth)
>    File "/usr/share/vdsm/vm.py", line 838, in f
>      ret = attr(*args, **kwargs)
>    File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper
>      ret = f(*args, **kwargs)
>    File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1178, in migrateToURI2
>      if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
> libvirtError: Failed to inquire lock: No such process
> Thread-26::ERROR::2014-01-17 17:01:28,917::sampling::355::vm.Vm::(collect) vmId=`b8787906-187a-4234-a0c9-58fc4ddf2a57`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x26efb58>
>
> The problem is that it doesn't say which process...

this looks like noise post the migration failing, with probably a more 
relevant error in libvirt log

>
> ----- Original Message -----
>> From: "Itamar Heim" <iheim at redhat.com>
>> To: "Edgars M." <edgars.mazurs at gmail.com>, "Meital Bourvine" <mbourvin at redhat.com>
>> Cc: users at ovirt.org, "Michal Skrivanek" <mskrivan at redhat.com>
>> Sent: Friday, January 17, 2014 9:47:11 PM
>> Subject: Re: [Users] VM Migration failed
>>
>> On 01/17/2014 06:25 PM, Edgars M. wrote:
>>> Hi Meital
>>>
>>> I tried to migrate another VM and it also failed.
>>>
>>> This is what I get in UI:
>>>
>>>
>>> Migration started (VM: nophpapp01, Source: novmnode1, Destination:
>>> novmnode2, User: edgarsm).
>>> Migration failed due to Error: Fatal error during migration. Trying to
>>> migrate to another Host (VM: nophpapp01, Source: novmnode1, Destination:
>>> novmnode2).
>>> Migration failed due to Error: Fatal error during migration (VM:
>>> nophpapp01, Source: novmnode1, Destination: novmnode2).
>>>
>>> There is nothing in /var/log/messages, neither on engine server nor nodes.
>>>
>>>
>>> See attachments for engine and vdsm logs. (I am migrating from vmnode1
>>> to vmnode2)
>>
>> whatever the issue is here, can you please open a bug on trying to
>> return more info to user on the migration error itself (for easier
>> troubleshooting).
>>
>> thanks,
>>      Itamar
>>
>>>
>>> Thanks for the help
>>> Edgars
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Jan 17, 2014 at 4:46 PM, Meital Bourvine <mbourvin at redhat.com
>>> <mailto:mbourvin at redhat.com>> wrote:
>>>
>>>      Which error are you getting in the UI?
>>>      Can you please attach the full engine and vdsm logs?
>>>      Also, please check if there is a relevant error in /var/log/messages
>>>
>>>
>>>      ------------------------------------------------------------------------
>>>
>>>          *From: *"Edgars M." <edgars.mazurs at gmail.com
>>>          <mailto:edgars.mazurs at gmail.com>>
>>>          *To: *users at ovirt.org <mailto:users at ovirt.org>
>>>          *Sent: *Friday, January 17, 2014 3:42:37 PM
>>>          *Subject: *[Users] VM Migration failed
>>>
>>>
>>>          Hi
>>>
>>>          I am experiencing issues with manual VM migration. VM fails to
>>>          migrate to other node in the same Cluster. Here are some
>>>          relevant engine.log entries:
>>>
>>>          ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>>>          (DefaultQuartzScheduler_Worker-73) Rerun vm
>>>          a31cfd62-26fc-4396-8a83-1aed68c7fd39. Called from vds novmnode1
>>>
>>>          ERROR
>>>          [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>>>          (pool-6-thread-49) Failed in MigrateStatusVDS method
>>>
>>>          ERROR
>>>          [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>>>          (pool-6-thread-49) Error code migrateErr and error message
>>>          VDSGenericException: VDSErrorException: Failed to
>>>          MigrateStatusVDS, error = Fatal error during migration
>>>
>>>          ERROR
>>>          [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>>>          (pool-6-thread-49) Command MigrateStatusVDS execution failed.
>>>          Exception: VDSErrorException: VDSGenericException:
>>>          VDSErrorException: Failed to MigrateStatusVDS, error = Fatal
>>>          error during migration
>>>
>>>          Both Engine and Node are running CentOS 6.5 x64. oVirt Engine
>>>          Version: 3.3.2-1.el6. VDSM version 4.13.2-1. I restarted engine
>>>          and vdsm, that did not help.
>>>
>>>          Also in vdsm log file I see the following errors related to the
>>>          same VM ID:
>>>
>>>          Thread-27::ERROR::2014-01-17
>>>          16:37:06,271::sampling::355::vm.Vm::(collect)
>>>          vmId=`a31cfd62-26fc-4396-8a83-1aed68c7fd39`::Stats function
>>>          failed: <AdvancedStatsFunction _highWrite at 0x26efb58>
>>>
>>>          Any hints?
>>>
>>>          BR
>>>          Edgars
>>>
>>>          _______________________________________________
>>>          Users mailing list
>>>          Users at ovirt.org <mailto:Users at ovirt.org>
>>>          http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>
>>
>>