Here is the content exceprt from libvirtd.log for the command: virsh #
migrate --p2p sl63 qemu+ssh://192.168.1.2/system
2013-01-31 18:02:53.740+0000: 2832: debug : virDomainFree:2313 :
dom=0x7f4f88000c80, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:02:53.743+0000: 2831: debug : virDomainLookupByName:2146 :
conn=0x7f4f8c001d80, name=sl63
2013-01-31 18:02:53.743+0000: 2831: debug : virDomainFree:2313 :
dom=0x7f4f84002150, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:02:53.747+0000: 2829: debug : virDrvSupportsFeature:1521 :
conn=0x7f4f8c001d80, feature=4
2013-01-31 18:02:53.751+0000: 2826: debug : virDrvSupportsFeature:1521 :
conn=0x7f4f8c001d80, feature=6
2013-01-31 18:02:53.754+0000: 2828: debug : virDomainMigratePerform3:6247 :
dom=0x7f4f7c0d1430, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb), xmlin=(null) cookiein=(nil),
cookieinlen=0, cookieout=0x7f4f998cab30, cookieoutlen=0x7f4f998cab3c,
dconnuri=qemu+ssh://192.168.1.2/system, uri=(null), flags=2, dname=(null),
bandwidth=0
2013-01-31 18:02:53.755+0000: 2828: debug : qemuMigrationPerform:2821 :
driver=0x7f4f8c0af820, conn=0x7f4f8c001d80, vm=0x7f4f7c0c7040,
xmlin=(null), dconnuri=qemu+ssh://192.168.1.2/system<http://3.57.111.32/system>,
uri=(null), cookiein=(null), cookieinlen=0, cookieout=0x7f4f998cab30,
cookieoutlen=0x7f4f998cab3c, flags=2, dname=(null), resource=0, v3proto=1
2013-01-31 18:02:53.755+0000: 2828: debug :
qemuDomainObjBeginJobInternal:758 : Starting async job: migration out
2013-01-31 18:02:53.779+0000: 2828: debug :
qemuProcessAutoDestroyActive:4226 : vm=sl63
2013-01-31 18:02:53.779+0000: 2828: debug : qemuDriverCloseCallbackGet:605
: vm=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb, conn=(nil)
2013-01-31 18:02:53.779+0000: 2828: debug : qemuDriverCloseCallbackGet:611
: cb=(nil)
2013-01-31 18:02:53.779+0000: 2828: debug : doPeer2PeerMigrate:2528 :
driver=0x7f4f8c0af820, sconn=0x7f4f8c001d80, vm=0x7f4f7c0c7040,
xmlin=(null), dconnuri=qemu+ssh://192.168.1.2/system<http://3.57.111.32/system>,
uri=(null), flags=2, dname=(null), resource=0
2013-01-31 18:02:53.779+0000: 2828: debug : virConnectOpen:1349 :
name=qemu+ssh://192.168.1.2/system <
http://3.57.111.32/system>
2013-01-31 18:02:53.779+0000: 2828: debug :
virConnectOpenResolveURIAlias:1070 : Loading config file
'/etc/libvirt/libvirt.conf'
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1151 : name "qemu+ssh://
192.168.1.2/system <
http://3.57.111.32/system>" to URI components:
scheme qemu+ssh
opaque (null)
authority (null)
server 192.168.1.2
user (null)
port 0
path /system
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1195 : trying driver 0
(Test) ...
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1201 : driver 0 Test
returned DECLINED
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1195 : trying driver 1
(ESX) ...
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1201 : driver 1 ESX
returned DECLINED
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1195 : trying driver 2
(remote) ...
2013-01-31 18:02:53.779+0000: 2828: debug : virCommandRunAsync:2174 : About
to run LC_ALL=C
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin ssh
192.168.1.2 sh -c 'if 'nc' -q 2>&1 | grep "requires an
argument" >/dev/null
2>&1; then ARG=-q0;else ARG=;fi;'nc' $ARG -U
/var/run/libvirt/libvirt-sock'
2013-01-31 18:02:53.780+0000: 2828: debug : virCommandRunAsync:2192 :
Command result 0, with PID 14537
2013-01-31 18:02:53.844+0000: 2828: error : virNetSocketReadWire:988 :
Cannot recv data: Permission denied, please try again.
Permission denied, please try again.
: Connection reset by peerey,gssapi-keyex,gssapi-with-mic,password).
2013-01-31 18:02:53.845+0000: 2828: debug : do_open:1201 : driver 2 remote
returned ERROR
2013-01-31 18:02:53.845+0000: 2828: error : doPeer2PeerMigrate:2539 :
operation failed: Failed to connect to remote libvirt URI qemu+ssh://
192.168.1.2/system <
http://3.57.111.32/system>
2013-01-31 18:02:53.845+0000: 2828: debug : qemuDomainObjEndAsyncJob:888 :
Stopping async job: migration out
2013-01-31 18:02:53.845+0000: 2821: debug : virPidAbort:2341 : aborting
child process 14537
2013-01-31 18:02:53.845+0000: 2821: debug : virPidAbort:2346 : process has
ended: exit status 255
2013-01-31 18:02:53.863+0000: 2828: debug : virDomainFree:2313 :
dom=0x7f4f7c0d1430, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:02:54.459+0000: 2834: debug : virDomainInterfaceStats:7223 :
dom=0x7f4f6c000910, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0,
stats=0x7f4f95cc4b00, size=64
2013-01-31 18:02:54.459+0000: 2834: debug : virDomainFree:2313 :
dom=0x7f4f6c000910, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:02:59.464+0000: 2825: debug : virDomainInterfaceStats:7223 :
dom=0x7f4f8c2455d0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0,
stats=0x7f4f9b6cdb00, size=64
2013-01-31 18:02:59.472+0000: 2825: debug : virDomainFree:2313 :
dom=0x7f4f8c2455d0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:03:04.478+0000: 2829: debug : virDrvSupportsFeature:1521 :
conn=0x7f4f7c000ae0, feature=9
2013-01-31 18:03:04.487+0000: 2826: debug : virDomainGetCPUStats:18676 :
dom=0x7f4f70000dc0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb), params=(nil), nparams=0,
start_cpu=-1, ncpus=1, flags=4
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupNew:603 : New group
/libvirt/qemu/sl63
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected
mount/mapping 0:cpu at /cgroup/cpu in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected
mount/mapping 1:cpuacct at /cgroup/cpuacct in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected
mount/mapping 2:cpuset at /cgroup/cpuset in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected
mount/mapping 3:memory at /cgroup/memory in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected
mount/mapping 4:devices at /cgroup/devices in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected
mount/mapping 5:freezer at /cgroup/freezer in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected
mount/mapping 6:blkio at /cgroup/blkio in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:524 : Make
group /libvirt/qemu/sl63
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/cpu/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/cpuacct/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/cpuset/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/memory/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/devices/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/freezer/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/blkio/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virDomainFree:2313 :
dom=0x7f4f70000dc0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:03:04.491+0000: 2828: debug : virDrvSupportsFeature:1521 :
conn=0x7f4f7c000ae0, feature=9
2013-01-31 18:03:04.495+0000: 2827: debug : virDomainGetCPUStats:18676 :
dom=0x7f4f74000ce0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb), params=0x7f4f74000b20,
nparams=3, start_cpu=-1, ncpus=1, flags=4
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupNew:603 : New group
/libvirt/qemu/sl63
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected
mount/mapping 0:cpu at /cgroup/cpu in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected
mount/mapping 1:cpuacct at /cgroup/cpuacct in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected
mount/mapping 2:cpuset at /cgroup/cpuset in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected
mount/mapping 3:memory at /cgroup/memory in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected
mount/mapping 4:devices at /cgroup/devices in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected
mount/mapping 5:freezer at /cgroup/freezer in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected
mount/mapping 6:blkio at /cgroup/blkio in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:524 : Make
group /libvirt/qemu/sl63
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/cpu/libvirt/qemu/sl63/
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/cpuacct/libvirt/qemu/sl63/
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/cpuset/libvirt/qemu/sl63/
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/memory/libvirt/qemu/sl63/
2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/devices/libvirt/qemu/sl63/
2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/freezer/libvirt/qemu/sl63/
2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/blkio/libvirt/qemu/sl63/
2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupGetValueStr:350 : Get
value /cgroup/cpuacct/libvirt/qemu/sl63/cpuacct.usage
2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupGetValueStr:350 : Get
value /cgroup/cpuacct/libvirt/qemu/sl63/cpuacct.stat
2013-01-31 18:03:04.496+0000: 2827: debug : virDomainFree:2313 :
dom=0x7f4f74000ce0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:03:04.500+0000: 2825: debug : virDomainInterfaceStats:7223 :
dom=0x7f4f8c2455d0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0,
stats=0x7f4f9b6cdb00, size=64
2013-01-31 18:03:04.500+0000: 2825: debug : virDomainFree:2313 :
dom=0x7f4f8c2455d0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:03:09.505+0000: 2829: debug : virDomainInterfaceStats:7223 :
dom=0x7f4f800034a0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0,
stats=0x7f4f98ec9b00, size=64
2013-01-31 18:03:09.520+0000: 2829: debug : virDomainFree:2313 :
dom=0x7f4f800034a0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb)
On Thu, Jan 31, 2013 at 11:27 AM, Dead Horse
<deadhorseconsulting(a)gmail.com>wrote:
note ignore the IP diff in the ssh host auth --> copy/paste fail
;)
- DHC
On Thu, Jan 31, 2013 at 11:25 AM, Dead Horse <
deadhorseconsulting(a)gmail.com> wrote:
> Doh, brain fart VDSM is not involved here for the purposed of the needed
> test.
> Here is my initial whack at it:
>
> Source Node:
>
> virsh # list
> Id Name State
> ----------------------------------------------------
> 1 sl63 running
>
> virsh # migrate --p2p sl63 qemu+ssh://192.168.1.2/system
> error: operation failed: Failed to connect to remote libvirt URI
> qemu+ssh://192.168.1.2/system <
http://3.57.111.32/system>
>
> virsh # migrate --live sl63 qemu+ssh://192.168.1.2/system
> The authenticity of host '192.168.1.2 (192.168.1.2)' can't be
established.
> RSA key fingerprint is e5:1d:b3:e5:38:5f:e1:8b:73:26:9e:15:c8:0a:2d:ac.
> Are you sure you want to continue connecting (yes/no)? yes
> root(a)192.168.1.2's password:
> Please enter your authentication name: vdsm@ovirt
> Please enter your password:
>
> virsh #
>
>
> Dest Node After migrate --live:
> virsh # list
> Id Name State
> ----------------------------------------------------
> 2 sl63 running
>
> virsh #
>
>
>
> On Thu, Jan 31, 2013 at 10:38 AM, Dead Horse <
> deadhorseconsulting(a)gmail.com> wrote:
>
>> Shu,
>> I build oVirt Engine and vdsm from source myself. The commits I
>> indicated are what I built from.I run the engine under FC17 and my nodes
>> are running EL6.x respectively.
>>
>> Dan,
>> I reverted VDSM on my two test nodes to an earlier build of VDSM
>> (commit:
>> c343e1833f7b6e5428dd90f14f7807dca1baa0b4)
>> VDSM after the above commit is broken due to commit:
>> fc3a44f71d2ef202cff18d7203b9e4165b546621 however when I built and tested
>> from master yesterday I did apply a patch I tested for
>> ybronhei which fixed that issue.
>>
>> I will build VDSM from master, today w/ the supervdsm patch and try the
>> manual migration you indicated.
>>
>> - DHC
>>
>>
>>
>> On Thu, Jan 31, 2013 at 4:56 AM, Dan Kenigsberg <danken(a)redhat.com>wrote:
>>
>>> On Thu, Jan 31, 2013 at 11:08:58AM +0100, Martin Kletzander wrote:
>>> > On 01/31/2013 10:25 AM, Dan Kenigsberg wrote:
>>> > > On Thu, Jan 31, 2013 at 09:43:44AM +0100, Martin Kletzander wrote:
>>> > >> On 01/30/2013 08:40 PM, Dead Horse wrote:
>>> > >>> The nodes are EL6.3 based.
>>> > >>>
>>> > >>> Currently installed libvirt packages:
>>> > >>>
>>> > >>> libvirt-lock-sanlock-0.9.10-21.el6_3.8.x86_64
>>> > >>> libvirt-cim-0.6.1-3.el6.x86_64
>>> > >>> libvirt-0.9.10-21.el6_3.8.x86_64
>>> > >>> libvirt-python-0.9.10-21.el6_3.8.x86_64
>>> > >>> libvirt-client-0.9.10-21.el6_3.8.x86_64
>>> > >>>
>>> > >>> and qemu packages:
>>> > >>> qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64
>>> > >>> qemu-kvm-tools-0.12.1.2-2.295.el6_3.10.x86_64
>>> > >>> qemu-img-0.12.1.2-2.295.el6_3.10.x86_64
>>> > >>>
>>> > >>> Thus my presumption here given the above is that
>>> virDomainMigrateToURI2 has
>>> > >>> not yet been patched and/or back-ported into the EL6.x
>>> libvirt/qemu?
>>> > >>>
>>> > >>
>>> > >> virDomainMigrateToURI2 is supported since 0.9.2, but is there
a
>>> > >> possibility the code is requesting direct migration? That
might
>>> explain
>>> > >> the message, which is then incorrect; this was fixed in [1].
>>> > >>
>>> > >> Martin
>>> > >>
>>> > >> [1]
>>> > >>
>>>
http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=3189dfb1636da22d426d...
>>> > >
>>> > > What is "direct migration" exactly, in the context of
qemu-kvm?
>>> > >
>>> > > We are using p2p migration
>>> > >
>>>
http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=vdsm/libvirtvm.py;h=fe...
>>> > >
>>> >
>>> > OK, so that's not the issue, sorry for the confusion. I was
thinking
>>> it
>>> > would "somehow" get there. Direct migration doesn't exist
in QEMU at
>>> > all, so it seemed weird, but I can't seem to find any other reason
for
>>> > this failure; will keep searching, though.
>>>
>>> In this case, Dead Horse, would you try to migrate a VM (that you do not
>>> care much about) using
>>> virsh -c qemu+tls://hostname/system migrate --p2p dsthost?
>>>
>>> I'd like to see that the problem reproduces this way, too. More of
>>> libvirtd.log may help. You may want to disable iptables for a moment,
>>> just to eliminate a common cause of failure.
>>>
>>
>>
>