<div dir="ltr">Here is the content exceprt from libvirtd.log for the command: virsh # migrate --p2p sl63 qemu+ssh://<a href="http://192.168.1.2/system" target="_blank">192.168.1.2/system</a><br><br>2013-01-31 18:02:53.740+0000: 2832: debug : virDomainFree:2313 : dom=0x7f4f88000c80, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb)<br>
2013-01-31 18:02:53.743+0000: 2831: debug : virDomainLookupByName:2146 : conn=0x7f4f8c001d80, name=sl63<br>2013-01-31 18:02:53.743+0000: 2831: debug : virDomainFree:2313 : dom=0x7f4f84002150, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb)<br>
2013-01-31 18:02:53.747+0000: 2829: debug : virDrvSupportsFeature:1521 : conn=0x7f4f8c001d80, feature=4<br>2013-01-31 18:02:53.751+0000: 2826: debug : virDrvSupportsFeature:1521 : conn=0x7f4f8c001d80, feature=6<br>2013-01-31 18:02:53.754+0000: 2828: debug : virDomainMigratePerform3:6247 : dom=0x7f4f7c0d1430, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), xmlin=(null) cookiein=(nil), cookieinlen=0, cookieout=0x7f4f998cab30, cookieoutlen=0x7f4f998cab3c, dconnuri=qemu+ssh://<a href="http://192.168.1.2/system" target="_blank">192.168.1.2/system</a>, uri=(null), flags=2, dname=(null), bandwidth=0<br>
2013-01-31 18:02:53.755+0000: 2828: debug : qemuMigrationPerform:2821 : driver=0x7f4f8c0af820, conn=0x7f4f8c001d80, vm=0x7f4f7c0c7040, xmlin=(null), dconnuri=qemu+ssh://<a href="http://3.57.111.32/system" target="_blank">192.168.1.2/system</a>, uri=(null), cookiein=(null), cookieinlen=0, cookieout=0x7f4f998cab30, cookieoutlen=0x7f4f998cab3c, flags=2, dname=(null), resource=0, v3proto=1<br>
2013-01-31 18:02:53.755+0000: 2828: debug : qemuDomainObjBeginJobInternal:758 : Starting async job: migration out<br>2013-01-31 18:02:53.779+0000: 2828: debug : qemuProcessAutoDestroyActive:4226 : vm=sl63<br>2013-01-31 18:02:53.779+0000: 2828: debug : qemuDriverCloseCallbackGet:605 : vm=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb, conn=(nil)<br>
2013-01-31 18:02:53.779+0000: 2828: debug : qemuDriverCloseCallbackGet:611 : cb=(nil)<br>2013-01-31 18:02:53.779+0000: 2828: debug : doPeer2PeerMigrate:2528 : driver=0x7f4f8c0af820, sconn=0x7f4f8c001d80, vm=0x7f4f7c0c7040, xmlin=(null), dconnuri=qemu+ssh://<a href="http://3.57.111.32/system" target="_blank">192.168.1.2/system</a>, uri=(null), flags=2, dname=(null), resource=0<br>
2013-01-31 18:02:53.779+0000: 2828: debug : virConnectOpen:1349 : name=qemu+ssh://<a href="http://3.57.111.32/system" target="_blank">192.168.1.2/system</a><br>2013-01-31 18:02:53.779+0000: 2828: debug : virConnectOpenResolveURIAlias:1070 : Loading config file '/etc/libvirt/libvirt.conf'<br>
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1151 : name "qemu+ssh://<a href="http://3.57.111.32/system" target="_blank">192.168.1.2/system</a>" to URI components:<br> scheme qemu+ssh<br> opaque (null)<br>
authority (null)<br>
server 192.168.1.2<br> user (null)<br> port 0<br> path /system<br><br>2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1195 : trying driver 0 (Test) ...<br>2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1201 : driver 0 Test returned DECLINED<br>
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1195 : trying driver 1 (ESX) ...<br>2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1201 : driver 1 ESX returned DECLINED<br>2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1195 : trying driver 2 (remote) ...<br>
2013-01-31 18:02:53.779+0000: 2828: debug : virCommandRunAsync:2174 : About to run LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin ssh 192.168.1.2 sh -c 'if 'nc' -q 2>&1 | grep "requires an argument" >/dev/null 2>&1; then ARG=-q0;else ARG=;fi;'nc' $ARG -U /var/run/libvirt/libvirt-sock'<br>
2013-01-31 18:02:53.780+0000: 2828: debug : virCommandRunAsync:2192 : Command result 0, with PID 14537<br>2013-01-31 18:02:53.844+0000: 2828: error : virNetSocketReadWire:988 : Cannot recv data: Permission denied, please try again.<br>
Permission denied, please try again.<br>: Connection reset by peerey,gssapi-keyex,gssapi-with-mic,password).<br>2013-01-31 18:02:53.845+0000: 2828: debug : do_open:1201 : driver 2 remote returned ERROR<br>2013-01-31 18:02:53.845+0000: 2828: error : doPeer2PeerMigrate:2539 : operation failed: Failed to connect to remote libvirt URI qemu+ssh://<a href="http://3.57.111.32/system" target="_blank">192.168.1.2/system</a><br>
2013-01-31 18:02:53.845+0000: 2828: debug : qemuDomainObjEndAsyncJob:888 : Stopping async job: migration out<br>2013-01-31 18:02:53.845+0000: 2821: debug : virPidAbort:2341 : aborting child process 14537<br>2013-01-31 18:02:53.845+0000: 2821: debug : virPidAbort:2346 : process has ended: exit status 255<br>
2013-01-31 18:02:53.863+0000: 2828: debug : virDomainFree:2313 : dom=0x7f4f7c0d1430, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb)<br>2013-01-31 18:02:54.459+0000: 2834: debug : virDomainInterfaceStats:7223 : dom=0x7f4f6c000910, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0, stats=0x7f4f95cc4b00, size=64<br>
2013-01-31 18:02:54.459+0000: 2834: debug : virDomainFree:2313 : dom=0x7f4f6c000910, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb)<br>2013-01-31 18:02:59.464+0000: 2825: debug : virDomainInterfaceStats:7223 : dom=0x7f4f8c2455d0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0, stats=0x7f4f9b6cdb00, size=64<br>
2013-01-31 18:02:59.472+0000: 2825: debug : virDomainFree:2313 : dom=0x7f4f8c2455d0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb)<br>2013-01-31 18:03:04.478+0000: 2829: debug : virDrvSupportsFeature:1521 : conn=0x7f4f7c000ae0, feature=9<br>
2013-01-31 18:03:04.487+0000: 2826: debug : virDomainGetCPUStats:18676 : dom=0x7f4f70000dc0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), params=(nil), nparams=0, start_cpu=-1, ncpus=1, flags=4<br>2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupNew:603 : New group /libvirt/qemu/sl63<br>
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 0:cpu at /cgroup/cpu in <br>2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 1:cpuacct at /cgroup/cpuacct in <br>
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 2:cpuset at /cgroup/cpuset in <br>2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 3:memory at /cgroup/memory in <br>
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 4:devices at /cgroup/devices in <br>2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 5:freezer at /cgroup/freezer in <br>
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected mount/mapping 6:blkio at /cgroup/blkio in <br>2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:524 : Make group /libvirt/qemu/sl63<br>
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/cpu/libvirt/qemu/sl63/<br>2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/cpuacct/libvirt/qemu/sl63/<br>
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/cpuset/libvirt/qemu/sl63/<br>2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/memory/libvirt/qemu/sl63/<br>
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/devices/libvirt/qemu/sl63/<br>2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/freezer/libvirt/qemu/sl63/<br>
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make controller /cgroup/blkio/libvirt/qemu/sl63/<br>2013-01-31 18:03:04.487+0000: 2826: debug : virDomainFree:2313 : dom=0x7f4f70000dc0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb)<br>
2013-01-31 18:03:04.491+0000: 2828: debug : virDrvSupportsFeature:1521 : conn=0x7f4f7c000ae0, feature=9<br>2013-01-31 18:03:04.495+0000: 2827: debug : virDomainGetCPUStats:18676 : dom=0x7f4f74000ce0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), params=0x7f4f74000b20, nparams=3, start_cpu=-1, ncpus=1, flags=4<br>
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupNew:603 : New group /libvirt/qemu/sl63<br>2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 0:cpu at /cgroup/cpu in <br>2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 1:cpuacct at /cgroup/cpuacct in <br>
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 2:cpuset at /cgroup/cpuset in <br>2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 3:memory at /cgroup/memory in <br>
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 4:devices at /cgroup/devices in <br>2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 5:freezer at /cgroup/freezer in <br>
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected mount/mapping 6:blkio at /cgroup/blkio in <br>2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:524 : Make group /libvirt/qemu/sl63<br>
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/cpu/libvirt/qemu/sl63/<br>2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/cpuacct/libvirt/qemu/sl63/<br>
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/cpuset/libvirt/qemu/sl63/<br>2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/memory/libvirt/qemu/sl63/<br>
2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/devices/libvirt/qemu/sl63/<br>2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/freezer/libvirt/qemu/sl63/<br>
2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupMakeGroup:546 : Make controller /cgroup/blkio/libvirt/qemu/sl63/<br>2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupGetValueStr:350 : Get value /cgroup/cpuacct/libvirt/qemu/sl63/cpuacct.usage<br>
2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupGetValueStr:350 : Get value /cgroup/cpuacct/libvirt/qemu/sl63/cpuacct.stat<br>2013-01-31 18:03:04.496+0000: 2827: debug : virDomainFree:2313 : dom=0x7f4f74000ce0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb)<br>
2013-01-31 18:03:04.500+0000: 2825: debug : virDomainInterfaceStats:7223 : dom=0x7f4f8c2455d0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0, stats=0x7f4f9b6cdb00, size=64<br>2013-01-31 18:03:04.500+0000: 2825: debug : virDomainFree:2313 : dom=0x7f4f8c2455d0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb)<br>
2013-01-31 18:03:09.505+0000: 2829: debug : virDomainInterfaceStats:7223 : dom=0x7f4f800034a0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0, stats=0x7f4f98ec9b00, size=64<br>2013-01-31 18:03:09.520+0000: 2829: debug : virDomainFree:2313 : dom=0x7f4f800034a0, (VM: name=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb)<br>
<br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jan 31, 2013 at 11:27 AM, Dead Horse <span dir="ltr"><<a href="mailto:deadhorseconsulting@gmail.com" target="_blank">deadhorseconsulting@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>note ignore the IP diff in the ssh host auth --> copy/paste fail ;)<br></div>- DHC<br></div><div>
<div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jan 31, 2013 at 11:25 AM, Dead Horse <span dir="ltr"><<a href="mailto:deadhorseconsulting@gmail.com" target="_blank">deadhorseconsulting@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Doh, brain fart VDSM is not involved here for the purposed of the needed test.<br></div>Here is my initial whack at it:<br>
<br>Source Node:<br><br>virsh # list<br> Id Name State<br>
----------------------------------------------------<br> 1 sl63 running<br><br>virsh # migrate --p2p sl63 qemu+ssh://<a href="http://192.168.1.2/system" target="_blank">192.168.1.2/system</a><br>
error: operation failed: Failed to connect to remote libvirt URI qemu+ssh://<a href="http://3.57.111.32/system" target="_blank">192.168.1.2/system</a><br>
<br>virsh # migrate --live sl63 qemu+ssh://<a href="http://192.168.1.2/system" target="_blank">192.168.1.2/system</a><br>The authenticity of host '192.168.1.2 (192.168.1.2)' can't be established.<br>RSA key fingerprint is e5:1d:b3:e5:38:5f:e1:8b:73:26:9e:15:c8:0a:2d:ac.<br>
Are you sure you want to continue connecting (yes/no)? yes<br><a href="mailto:root@192.168.1.2" target="_blank">root@192.168.1.2</a>'s password: <br>Please enter your authentication name: vdsm@ovirt<br>Please enter your password: <br>
<br>virsh # <br><br><br></div>Dest Node After migrate --live:<br>virsh # list<br> Id Name State<br>----------------------------------------------------<br> 2 sl63 running<br>
<br>virsh # <br><br></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jan 31, 2013 at 10:38 AM, Dead Horse <span dir="ltr"><<a href="mailto:deadhorseconsulting@gmail.com" target="_blank">deadhorseconsulting@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><span name="Shu Ming">Shu,<br></span></div><span name="Shu Ming">I build oVirt Engine and vdsm from source myself. The commits I indicated are what I built from.I run the engine under FC17 and my nodes are running EL6.x respectively. <br>
<br></span></div><span name="Shu Ming">Dan,<br></span></div><span name="Shu Ming">I reverted VDSM on my two test nodes to an earlier build of VDSM (commit: </span><br><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;display:inline!important;font-weight:normal;float:none;line-height:normal;text-transform:none;font-size:12px;white-space:normal;font-family:monospace;word-spacing:0px">c343e1833f7b6e5428dd90f14f7807dca1baa0b4)<br>
VDSM after the above commit is broken due to commit: </span><br><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;display:inline!important;font-weight:normal;float:none;line-height:normal;text-transform:none;font-size:12px;white-space:normal;font-family:monospace;word-spacing:0px">fc3a44f71d2ef202cff18d7203b9e4165b546621 however when I built and tested from master yesterday I did apply a patch I tested for </span><br>
<span name="ybronhei">ybronhei which fixed that issue.<br><br></span></div><span name="ybronhei">I will build VDSM from master, today w/ the supervdsm patch and try the manual migration you indicated.<br>
<br></span></div><span name="ybronhei"> - DHC <br></span><div><div><div><div><span name="Shu Ming"><br></span></div></div></div></div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Thu, Jan 31, 2013 at 4:56 AM, Dan Kenigsberg <span dir="ltr"><<a href="mailto:danken@redhat.com" target="_blank">danken@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>On Thu, Jan 31, 2013 at 11:08:58AM +0100, Martin Kletzander wrote:<br>
> On 01/31/2013 10:25 AM, Dan Kenigsberg wrote:<br>
> > On Thu, Jan 31, 2013 at 09:43:44AM +0100, Martin Kletzander wrote:<br>
> >> On 01/30/2013 08:40 PM, Dead Horse wrote:<br>
> >>> The nodes are EL6.3 based.<br>
> >>><br>
> >>> Currently installed libvirt packages:<br>
> >>><br>
> >>> libvirt-lock-sanlock-0.9.10-21.el6_3.8.x86_64<br>
> >>> libvirt-cim-0.6.1-3.el6.x86_64<br>
> >>> libvirt-0.9.10-21.el6_3.8.x86_64<br>
> >>> libvirt-python-0.9.10-21.el6_3.8.x86_64<br>
> >>> libvirt-client-0.9.10-21.el6_3.8.x86_64<br>
> >>><br>
> >>> and qemu packages:<br>
> >>> qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64<br>
> >>> qemu-kvm-tools-0.12.1.2-2.295.el6_3.10.x86_64<br>
> >>> qemu-img-0.12.1.2-2.295.el6_3.10.x86_64<br>
> >>><br>
> >>> Thus my presumption here given the above is that virDomainMigrateToURI2 has<br>
> >>> not yet been patched and/or back-ported into the EL6.x libvirt/qemu?<br>
> >>><br>
> >><br>
> >> virDomainMigrateToURI2 is supported since 0.9.2, but is there a<br>
> >> possibility the code is requesting direct migration? That might explain<br>
> >> the message, which is then incorrect; this was fixed in [1].<br>
> >><br>
> >> Martin<br>
> >><br>
> >> [1]<br>
> >> <a href="http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=3189dfb1636da22d426d2fc07cc9f60304b16c5c" target="_blank">http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=3189dfb1636da22d426d2fc07cc9f60304b16c5c</a><br>
> ><br>
> > What is "direct migration" exactly, in the context of qemu-kvm?<br>
> ><br>
> > We are using p2p migration<br>
> > <a href="http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=vdsm/libvirtvm.py;h=fe140ecbfac665248e2ad5c4bfaebaf54ab884cc;hb=18c24f7c7c27ac732c4a760caa9524e7319cd47e#l501" target="_blank">http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=vdsm/libvirtvm.py;h=fe140ecbfac665248e2ad5c4bfaebaf54ab884cc;hb=18c24f7c7c27ac732c4a760caa9524e7319cd47e#l501</a><br>
> ><br>
><br>
> OK, so that's not the issue, sorry for the confusion. I was thinking it<br>
> would "somehow" get there. Direct migration doesn't exist in QEMU at<br>
> all, so it seemed weird, but I can't seem to find any other reason for<br>
> this failure; will keep searching, though.<br>
<br>
</div></div>In this case, Dead Horse, would you try to migrate a VM (that you do not<br>
care much about) using<br>
virsh -c qemu+tls://hostname/system migrate --p2p dsthost?<br>
<br>
I'd like to see that the problem reproduces this way, too. More of<br>
libvirtd.log may help. You may want to disable iptables for a moment,<br>
just to eliminate a common cause of failure.<br>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>