Here is the content exceprt from libvirtd.log for the command: virsh # migrate --p2p sl63 qemu+ssh://192.168.1.2/system

2013-01-31 18:02:53.740+0000: 2832: debug : virDomainFree:2313 : dom=0x7f4f88000c80, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:02:53.743+0000: 2831: debug : virDomainLookupByName:2146 : conn=0x7f4f8c001d80, name=sl63
2013-01-31 18:02:53.743+0000: 2831: debug : virDomainFree:2313 : dom=0x7f4f84002150, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:02:53.747+0000: 2829: debug : virDrvSupportsFeature:1521 : conn=0x7f4f8c001d80, feature=4
2013-01-31 18:02:53.751+0000: 2826: debug : virDrvSupportsFeature:1521 : conn=0x7f4f8c001d80, feature=6
2013-01-31 18:02:53.754+0000: 2828: debug : virDomainMigratePerform3:6247 : dom=0x7f4f7c0d1430, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), xmlin=(null) cookiein=(nil), cookieinlen=0, cookieout=0x7f4f998cab30, cookieoutlen=0x7f4f998cab3c, dconnuri=qemu+ssh://192.168.1.2/system, uri=(null), flags=2, dname=(null), bandwidth=0
2013-01-31 18:02:53.755+0000: 2828: debug : qemuMigrationPerform:2821 : driver=0x7f4f8c0af820, conn=0x7f4f8c001d80, vm=0x7f4f7c0c7040, xmlin=(null), dconnuri=qemu+ssh://192.168.1.2/system, uri=(null), cookiein=(null), cookieinlen=0, cookieout=0x7f4f998cab30, cookieoutlen=0x7f4f998cab3c, flags=2, dname=(null), resource=0, v3proto=1
2013-01-31 18:02:53.755+0000: 2828: debug : qemuDomainObjBeginJobInternal:758 : Starting async job: migration out
2013-01-31 18:02:53.779+0000: 2828: debug : qemuProcessAutoDestroyActive:4226 : vm=sl63
2013-01-31 18:02:53.779+0000: 2828: debug : qemuDriverCloseCallbackGet:605 : vm=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb, conn=(nil)
2013-01-31 18:02:53.779+0000: 2828: debug : qemuDriverCloseCallbackGet:611 : cb=(nil)
2013-01-31 18:02:53.779+0000: 2828: debug : doPeer2PeerMigrate:2528 : driver=0x7f4f8c0af820, sconn=0x7f4f8c001d80, vm=0x7f4f7c0c7040, xmlin=(null), dconnuri=qemu+ssh://192.168.1.2/system, uri=(null), flags=2, dname=(null), resource=0
2013-01-31 18:02:53.779+0000: 2828: debug : virConnectOpen:1349 : name=qemu+ssh://192.168.1.2/system
2013-01-31 18:02:53.779+0000: 2828: debug : virConnectOpenResolveURIAlias:1070 : Loading config file '/etc/libvirt/libvirt.conf'
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1151 : name "qemu+ssh://192.168.1.2/system" to URI components:
  scheme qemu+ssh
  opaque (null)
  authority (null)
  server 192.168.1.2
  user (null)
  port 0
  path /system

2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1195 : trying driver 0 (Test) ...
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1201 : driver 0 Test returned DECLINED
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1195 : trying driver 1 (ESX) ...
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1201 : driver 1 ESX returned DECLINED
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1195 : trying driver 2 (remote) ...
2013-01-31 18:02:53.779+0000: 2828: debug : virCommandRunAsync:2174 : About to run LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin ssh 192.168.1.2 sh -c 'if 'nc' -q 2>&1 | grep "requires an argument" >/dev/null 2>&1; then ARG=-q0;else ARG=;fi;'nc' $ARG -U /var/run/libvirt/libvirt-sock'
2013-01-31 18:02:53.780+0000: 2828: debug : virCommandRunAsync:2192 : Command result 0, with PID 14537
2013-01-31 18:02:53.844+0000: 2828: error : virNetSocketReadWire:988 : Cannot recv data: Permission denied, please try again.
Permission denied, please try again.
: Connection reset by peerey,gssapi-keyex,gssapi-with-mic,password).
2013-01-31 18:02:53.845+0000: 2828: debug : do_open:1201 : driver 2 remote returned ERROR
2013-01-31 18:02:53.845+0000: 2828: error : doPeer2PeerMigrate:2539 : operation failed: Failed to connect to remote libvirt URI qemu+ssh://192.168.1.2/system
2013-01-31 18:02:53.845+0000: 2828: debug : qemuDomainObjEndAsyncJob:888 : Stopping async job: migration out
2013-01-31 18:02:53.845+0000: 2821: debug : virPidAbort:2341 : aborting child process 14537
2013-01-31 18:02:53.845+0000: 2821: debug : virPidAbort:2346 : process has ended: exit status 255
2013-01-31 18:02:53.863+0000: 2828: debug : virDomainFree:2313 : dom=0x7f4f7c0d1430, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:02:54.459+0000: 2834: debug : virDomainInterfaceStats:7223 : dom=0x7f4f6c000910, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0, stats=0x7f4f95cc4b00, size=64
2013-01-31 18:02:54.459+0000: 2834: debug : virDomainFree:2313 : dom=0x7f4f6c000910, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:02:59.464+0000: 2825: debug : virDomainInterfaceStats:7223 : dom=0x7f4f8c2455d0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0, stats=0x7f4f9b6cdb00, size=64
2013-01-31 18:02:59.472+0000: 2825: debug : virDomainFree:2313 : dom=0x7f4f8c2455d0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:03:04.478+0000: 2829: debug : virDrvSupportsFeature:1521 : conn=0x7f4f7c000ae0, feature=9
2013-01-31 18:03:04.487+0000: 2826: debug : virDomainGetCPUStats:18676 : dom=0x7f4f70000dc0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), params=(nil), nparams=0, start_cpu=-1, ncpus=1, flags=4
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupNew:603 : New group /libvirt/qemu/sl63
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 0:cpu at /cgroup/cpu in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 1:cpuacct at /cgroup/cpuacct in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 2:cpuset at /cgroup/cpuset in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 3:memory at /cgroup/memory in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 4:devices at /cgroup/devices in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 5:freezer at /cgroup/freezer in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 6:blkio at /cgroup/blkio in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:524 : Make group /libvirt/qemu/sl63
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/cpu/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/cpuacct/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/cpuset/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/memory/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/devices/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/freezer/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/blkio/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virDomainFree:2313 : dom=0x7f4f70000dc0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:03:04.491+0000: 2828: debug : virDrvSupportsFeature:1521 : conn=0x7f4f7c000ae0, feature=9
2013-01-31 18:03:04.495+0000: 2827: debug : virDomainGetCPUStats:18676 : dom=0x7f4f74000ce0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), params=0x7f4f74000b20, nparams=3, start_cpu=-1, ncpus=1, flags=4
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupNew:603 : New group /libvirt/qemu/sl63
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 0:cpu at /cgroup/cpu in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 1:cpuacct at /cgroup/cpuacct in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 2:cpuset at /cgroup/cpuset in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 3:memory at /cgroup/memory in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 4:devices at /cgroup/devices in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 5:freezer at /cgroup/freezer in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 6:blkio at /cgroup/blkio in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:524 : Make group /libvirt/qemu/sl63
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/cpu/libvirt/qemu/sl63/
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/cpuacct/libvirt/qemu/sl63/
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/cpuset/libvirt/qemu/sl63/
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/memory/libvirt/qemu/sl63/
2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/devices/libvirt/qemu/sl63/
2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/freezer/libvirt/qemu/sl63/
2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/blkio/libvirt/qemu/sl63/
2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupGetValueStr:350 : Get value /cgroup/cpuacct/libvirt/qemu/sl63/cpuacct.usage
2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupGetValueStr:350 : Get value /cgroup/cpuacct/libvirt/qemu/sl63/cpuacct.stat
2013-01-31 18:03:04.496+0000: 2827: debug : virDomainFree:2313 : dom=0x7f4f74000ce0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:03:04.500+0000: 2825: debug : virDomainInterfaceStats:7223 : dom=0x7f4f8c2455d0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0, stats=0x7f4f9b6cdb00, size=64
2013-01-31 18:03:04.500+0000: 2825: debug : virDomainFree:2313 : dom=0x7f4f8c2455d0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:03:09.505+0000: 2829: debug : virDomainInterfaceStats:7223 : dom=0x7f4f800034a0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0, stats=0x7f4f98ec9b00, size=64
2013-01-31 18:03:09.520+0000: 2829: debug : virDomainFree:2313 : dom=0x7f4f800034a0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb)



On Thu, Jan 31, 2013 at 11:27 AM, Dead Horse <deadhorseconsulting@gmail.com> wrote:
note ignore the IP diff in the ssh host auth --> copy/paste fail ;)
- DHC


On Thu, Jan 31, 2013 at 11:25 AM, Dead Horse <deadhorseconsulting@gmail.com> wrote:
Doh, brain fart VDSM is not involved here for the purposed of the needed test.
Here is my initial whack at it:

Source Node:

virsh # list
 Id    Name                           State
----------------------------------------------------
 1     sl63                           running

virsh # migrate --p2p sl63 qemu+ssh://192.168.1.2/system
error: operation failed: Failed to connect to remote libvirt URI qemu+ssh://192.168.1.2/system

virsh # migrate --live sl63 qemu+ssh://192.168.1.2/system
The authenticity of host '192.168.1.2 (192.168.1.2)' can't be established.
RSA key fingerprint is e5:1d:b3:e5:38:5f:e1:8b:73:26:9e:15:c8:0a:2d:ac.
Are you sure you want to continue connecting (yes/no)? yes
root@192.168.1.2's password:
Please enter your authentication name: vdsm@ovirt
Please enter your password:

virsh #


Dest Node After migrate --live:
virsh # list
 Id    Name                           State
----------------------------------------------------
 2     sl63                           running

virsh #



On Thu, Jan 31, 2013 at 10:38 AM, Dead Horse <deadhorseconsulting@gmail.com> wrote:
Shu,
I build oVirt Engine and vdsm from source myself. The commits I indicated are what I built from.I run the engine under FC17 and my nodes are running EL6.x respectively.

Dan,
I reverted VDSM on my two test nodes to an earlier build of VDSM (commit:
c343e1833f7b6e5428dd90f14f7807dca1baa0b4)
VDSM after the above commit is broken due to commit:
fc3a44f71d2ef202cff18d7203b9e4165b546621 however when I built and tested from master yesterday I did apply a patch I tested for
ybronhei which fixed that issue.

I will build VDSM from master, today w/ the supervdsm patch and try the manual migration you indicated.

 - DHC



On Thu, Jan 31, 2013 at 4:56 AM, Dan Kenigsberg <danken@redhat.com> wrote:
On Thu, Jan 31, 2013 at 11:08:58AM +0100, Martin Kletzander wrote:
> On 01/31/2013 10:25 AM, Dan Kenigsberg wrote:
> > On Thu, Jan 31, 2013 at 09:43:44AM +0100, Martin Kletzander wrote:
> >> On 01/30/2013 08:40 PM, Dead Horse wrote:
> >>> The nodes are EL6.3 based.
> >>>
> >>> Currently installed libvirt packages:
> >>>
> >>> libvirt-lock-sanlock-0.9.10-21.el6_3.8.x86_64
> >>> libvirt-cim-0.6.1-3.el6.x86_64
> >>> libvirt-0.9.10-21.el6_3.8.x86_64
> >>> libvirt-python-0.9.10-21.el6_3.8.x86_64
> >>> libvirt-client-0.9.10-21.el6_3.8.x86_64
> >>>
> >>> and qemu packages:
> >>> qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64
> >>> qemu-kvm-tools-0.12.1.2-2.295.el6_3.10.x86_64
> >>> qemu-img-0.12.1.2-2.295.el6_3.10.x86_64
> >>>
> >>> Thus my presumption here given the above is that virDomainMigrateToURI2 has
> >>> not yet been patched and/or back-ported into the EL6.x libvirt/qemu?
> >>>
> >>
> >> virDomainMigrateToURI2 is supported since 0.9.2, but is there a
> >> possibility the code is requesting direct migration?  That might explain
> >> the message, which is then incorrect; this was fixed in [1].
> >>
> >> Martin
> >>
> >> [1]
> >> http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=3189dfb1636da22d426d2fc07cc9f60304b16c5c
> >
> > What is "direct migration" exactly, in the context of qemu-kvm?
> >
> > We are using p2p migration
> > http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=vdsm/libvirtvm.py;h=fe140ecbfac665248e2ad5c4bfaebaf54ab884cc;hb=18c24f7c7c27ac732c4a760caa9524e7319cd47e#l501
> >
>
> OK, so that's not the issue, sorry for the confusion.  I was thinking it
> would "somehow" get there.  Direct migration doesn't exist in QEMU at
> all, so it seemed weird, but I can't seem to find any other reason for
> this failure; will keep searching, though.

In this case, Dead Horse, would you try to migrate a VM (that you do not
care much about) using
    virsh -c qemu+tls://hostname/system migrate --p2p dsthost?

I'd like to see that the problem reproduces this way, too. More of
libvirtd.log may help. You may want to disable iptables for a moment,
just to eliminate a common cause of failure.