[Users] VM migrations failing

Fri Feb 1 08:58:50 UTC 2013

On 01/31/2013 07:07 PM, Dead Horse wrote:
> Here is the content exceprt from libvirtd.log for the command: virsh #
> migrate --p2p sl63 qemu+ssh://192.168.1.2/system
> 

Thanks for testing this.  However, this is another problem.  When
migrating p2p, the destination URI must be reachable from the source,
not the client.  That is probably what failed in first place.  Would you
mind trying migrating p2p (and specifying the destination URI that's
reachable from the source host)?  Thanks.

Also, on both hosts the version are the same?

> 2013-01-31 18:02:53.740+0000: 2832: debug : virDomainFree:2313 :
> dom=0x7f4f88000c80, (VM: name=sl63,
> uuid=887d764a-f835-4112-9eda-836a772ea5eb)
> 2013-01-31 18:02:53.743+0000: 2831: debug : virDomainLookupByName:2146 :
> conn=0x7f4f8c001d80, name=sl63
> 2013-01-31 18:02:53.743+0000: 2831: debug : virDomainFree:2313 :
> dom=0x7f4f84002150, (VM: name=sl63,
> uuid=887d764a-f835-4112-9eda-836a772ea5eb)
> 2013-01-31 18:02:53.747+0000: 2829: debug : virDrvSupportsFeature:1521 :
> conn=0x7f4f8c001d80, feature=4
> 2013-01-31 18:02:53.751+0000: 2826: debug : virDrvSupportsFeature:1521 :
> conn=0x7f4f8c001d80, feature=6
> 2013-01-31 18:02:53.754+0000: 2828: debug : virDomainMigratePerform3:6247 :
> dom=0x7f4f7c0d1430, (VM: name=sl63,
> uuid=887d764a-f835-4112-9eda-836a772ea5eb), xmlin=(null) cookiein=(nil),
> cookieinlen=0, cookieout=0x7f4f998cab30, cookieoutlen=0x7f4f998cab3c,
> dconnuri=qemu+ssh://192.168.1.2/system, uri=(null), flags=2, dname=(null),
> bandwidth=0
> 2013-01-31 18:02:53.755+0000: 2828: debug : qemuMigrationPerform:2821 :
> driver=0x7f4f8c0af820, conn=0x7f4f8c001d80, vm=0x7f4f7c0c7040,
> xmlin=(null), dconnuri=qemu+ssh://192.168.1.2/system<http://3.57.111.32/system>,

Funny how mail clients try to make it "easier" for us and add a link
that makes sense (to them).

[...]

> 
> On Thu, Jan 31, 2013 at 11:27 AM, Dead Horse
> <deadhorseconsulting at gmail.com>wrote:
> 
>> note ignore the IP diff in the ssh host auth --> copy/paste fail ;)
>> - DHC
>>
>>
>> On Thu, Jan 31, 2013 at 11:25 AM, Dead Horse <
>> deadhorseconsulting at gmail.com> wrote:
>>
>>> Doh, brain fart VDSM is not involved here for the purposed of the needed
>>> test.
>>> Here is my initial whack at it:
>>>
>>> Source Node:
>>>
>>> virsh # list
>>>  Id    Name                           State
>>> ----------------------------------------------------
>>>  1     sl63                           running
>>>
>>> virsh # migrate --p2p sl63 qemu+ssh://192.168.1.2/system
>>> error: operation failed: Failed to connect to remote libvirt URI
>>> qemu+ssh://192.168.1.2/system <http://3.57.111.32/system>
>>>
>>> virsh # migrate --live sl63 qemu+ssh://192.168.1.2/system
>>> The authenticity of host '192.168.1.2 (192.168.1.2)' can't be established.
>>> RSA key fingerprint is e5:1d:b3:e5:38:5f:e1:8b:73:26:9e:15:c8:0a:2d:ac.
>>> Are you sure you want to continue connecting (yes/no)? yes
>>> root at 192.168.1.2's password:
>>> Please enter your authentication name: vdsm at ovirt
>>> Please enter your password:
>>>
>>> virsh #
>>>
>>>
>>> Dest Node After migrate --live:
>>> virsh # list
>>>  Id    Name                           State
>>> ----------------------------------------------------
>>>  2     sl63                           running
>>>
>>> virsh #
>>>
>>>
>>>
>>> On Thu, Jan 31, 2013 at 10:38 AM, Dead Horse <
>>> deadhorseconsulting at gmail.com> wrote:
>>>
>>>> Shu,
>>>> I build oVirt Engine and vdsm from source myself. The commits I
>>>> indicated are what I built from.I run the engine under FC17 and my nodes
>>>> are running EL6.x respectively.
>>>>
>>>> Dan,
>>>> I reverted VDSM on my two test nodes to an earlier build of VDSM
>>>> (commit:
>>>> c343e1833f7b6e5428dd90f14f7807dca1baa0b4)
>>>> VDSM after the above commit is broken due to commit:
>>>> fc3a44f71d2ef202cff18d7203b9e4165b546621 however when I built and tested
>>>> from master yesterday I did apply a patch I tested for
>>>> ybronhei which fixed that issue.
>>>>
>>>> I will build VDSM from master, today w/ the supervdsm patch and try the
>>>> manual migration you indicated.
>>>>
>>>>  - DHC
>>>>
>>>>
>>>>
>>>> On Thu, Jan 31, 2013 at 4:56 AM, Dan Kenigsberg <danken at redhat.com>wrote:
>>>>
>>>>> On Thu, Jan 31, 2013 at 11:08:58AM +0100, Martin Kletzander wrote:
>>>>>> On 01/31/2013 10:25 AM, Dan Kenigsberg wrote:
>>>>>>> On Thu, Jan 31, 2013 at 09:43:44AM +0100, Martin Kletzander wrote:
>>>>>>>> On 01/30/2013 08:40 PM, Dead Horse wrote:
>>>>>>>>> The nodes are EL6.3 based.
>>>>>>>>>
>>>>>>>>> Currently installed libvirt packages:
>>>>>>>>>
>>>>>>>>> libvirt-lock-sanlock-0.9.10-21.el6_3.8.x86_64
>>>>>>>>> libvirt-cim-0.6.1-3.el6.x86_64
>>>>>>>>> libvirt-0.9.10-21.el6_3.8.x86_64
>>>>>>>>> libvirt-python-0.9.10-21.el6_3.8.x86_64
>>>>>>>>> libvirt-client-0.9.10-21.el6_3.8.x86_64
>>>>>>>>>
>>>>>>>>> and qemu packages:
>>>>>>>>> qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64
>>>>>>>>> qemu-kvm-tools-0.12.1.2-2.295.el6_3.10.x86_64
>>>>>>>>> qemu-img-0.12.1.2-2.295.el6_3.10.x86_64
>>>>>>>>>
>>>>>>>>> Thus my presumption here given the above is that
>>>>> virDomainMigrateToURI2 has
>>>>>>>>> not yet been patched and/or back-ported into the EL6.x
>>>>> libvirt/qemu?
>>>>>>>>>
>>>>>>>>
>>>>>>>> virDomainMigrateToURI2 is supported since 0.9.2, but is there a
>>>>>>>> possibility the code is requesting direct migration?  That might
>>>>> explain
>>>>>>>> the message, which is then incorrect; this was fixed in [1].
>>>>>>>>
>>>>>>>> Martin
>>>>>>>>
>>>>>>>> [1]
>>>>>>>>
>>>>> http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=3189dfb1636da22d426d2fc07cc9f60304b16c5c
>>>>>>>
>>>>>>> What is "direct migration" exactly, in the context of qemu-kvm?
>>>>>>>
>>>>>>> We are using p2p migration
>>>>>>>
>>>>> http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=vdsm/libvirtvm.py;h=fe140ecbfac665248e2ad5c4bfaebaf54ab884cc;hb=18c24f7c7c27ac732c4a760caa9524e7319cd47e#l501
>>>>>>>
>>>>>>
>>>>>> OK, so that's not the issue, sorry for the confusion.  I was thinking
>>>>> it
>>>>>> would "somehow" get there.  Direct migration doesn't exist in QEMU at
>>>>>> all, so it seemed weird, but I can't seem to find any other reason for
>>>>>> this failure; will keep searching, though.
>>>>>
>>>>> In this case, Dead Horse, would you try to migrate a VM (that you do not
>>>>> care much about) using
>>>>>     virsh -c qemu+tls://hostname/system migrate --p2p dsthost?
>>>>>
>>>>> I'd like to see that the problem reproduces this way, too. More of
>>>>> libvirtd.log may help. You may want to disable iptables for a moment,
>>>>> just to eliminate a common cause of failure.
>>>>>
>>>>
>>>>
>>>
>>
> 