[Users] VM migrations failing
Dead Horse
deadhorseconsulting at gmail.com
Thu Jan 31 13:07:38 EST 2013
Here is the content exceprt from libvirtd.log for the command: virsh #
migrate --p2p sl63 qemu+ssh://192.168.1.2/system
2013-01-31 18:02:53.740+0000: 2832: debug : virDomainFree:2313 :
dom=0x7f4f88000c80, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:02:53.743+0000: 2831: debug : virDomainLookupByName:2146 :
conn=0x7f4f8c001d80, name=sl63
2013-01-31 18:02:53.743+0000: 2831: debug : virDomainFree:2313 :
dom=0x7f4f84002150, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:02:53.747+0000: 2829: debug : virDrvSupportsFeature:1521 :
conn=0x7f4f8c001d80, feature=4
2013-01-31 18:02:53.751+0000: 2826: debug : virDrvSupportsFeature:1521 :
conn=0x7f4f8c001d80, feature=6
2013-01-31 18:02:53.754+0000: 2828: debug : virDomainMigratePerform3:6247 :
dom=0x7f4f7c0d1430, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb), xmlin=(null) cookiein=(nil),
cookieinlen=0, cookieout=0x7f4f998cab30, cookieoutlen=0x7f4f998cab3c,
dconnuri=qemu+ssh://192.168.1.2/system, uri=(null), flags=2, dname=(null),
bandwidth=0
2013-01-31 18:02:53.755+0000: 2828: debug : qemuMigrationPerform:2821 :
driver=0x7f4f8c0af820, conn=0x7f4f8c001d80, vm=0x7f4f7c0c7040,
xmlin=(null), dconnuri=qemu+ssh://192.168.1.2/system<http://3.57.111.32/system>,
uri=(null), cookiein=(null), cookieinlen=0, cookieout=0x7f4f998cab30,
cookieoutlen=0x7f4f998cab3c, flags=2, dname=(null), resource=0, v3proto=1
2013-01-31 18:02:53.755+0000: 2828: debug :
qemuDomainObjBeginJobInternal:758 : Starting async job: migration out
2013-01-31 18:02:53.779+0000: 2828: debug :
qemuProcessAutoDestroyActive:4226 : vm=sl63
2013-01-31 18:02:53.779+0000: 2828: debug : qemuDriverCloseCallbackGet:605
: vm=sl63, uuid=887d764a-f835-4112-9eda-836a772ea5eb, conn=(nil)
2013-01-31 18:02:53.779+0000: 2828: debug : qemuDriverCloseCallbackGet:611
: cb=(nil)
2013-01-31 18:02:53.779+0000: 2828: debug : doPeer2PeerMigrate:2528 :
driver=0x7f4f8c0af820, sconn=0x7f4f8c001d80, vm=0x7f4f7c0c7040,
xmlin=(null), dconnuri=qemu+ssh://192.168.1.2/system<http://3.57.111.32/system>,
uri=(null), flags=2, dname=(null), resource=0
2013-01-31 18:02:53.779+0000: 2828: debug : virConnectOpen:1349 :
name=qemu+ssh://192.168.1.2/system <http://3.57.111.32/system>
2013-01-31 18:02:53.779+0000: 2828: debug :
virConnectOpenResolveURIAlias:1070 : Loading config file
'/etc/libvirt/libvirt.conf'
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1151 : name "qemu+ssh://
192.168.1.2/system <http://3.57.111.32/system>" to URI components:
scheme qemu+ssh
opaque (null)
authority (null)
server 192.168.1.2
user (null)
port 0
path /system
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1195 : trying driver 0
(Test) ...
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1201 : driver 0 Test
returned DECLINED
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1195 : trying driver 1
(ESX) ...
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1201 : driver 1 ESX
returned DECLINED
2013-01-31 18:02:53.779+0000: 2828: debug : do_open:1195 : trying driver 2
(remote) ...
2013-01-31 18:02:53.779+0000: 2828: debug : virCommandRunAsync:2174 : About
to run LC_ALL=C
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin ssh
192.168.1.2 sh -c 'if 'nc' -q 2>&1 | grep "requires an argument" >/dev/null
2>&1; then ARG=-q0;else ARG=;fi;'nc' $ARG -U /var/run/libvirt/libvirt-sock'
2013-01-31 18:02:53.780+0000: 2828: debug : virCommandRunAsync:2192 :
Command result 0, with PID 14537
2013-01-31 18:02:53.844+0000: 2828: error : virNetSocketReadWire:988 :
Cannot recv data: Permission denied, please try again.
Permission denied, please try again.
: Connection reset by peerey,gssapi-keyex,gssapi-with-mic,password).
2013-01-31 18:02:53.845+0000: 2828: debug : do_open:1201 : driver 2 remote
returned ERROR
2013-01-31 18:02:53.845+0000: 2828: error : doPeer2PeerMigrate:2539 :
operation failed: Failed to connect to remote libvirt URI qemu+ssh://
192.168.1.2/system <http://3.57.111.32/system>
2013-01-31 18:02:53.845+0000: 2828: debug : qemuDomainObjEndAsyncJob:888 :
Stopping async job: migration out
2013-01-31 18:02:53.845+0000: 2821: debug : virPidAbort:2341 : aborting
child process 14537
2013-01-31 18:02:53.845+0000: 2821: debug : virPidAbort:2346 : process has
ended: exit status 255
2013-01-31 18:02:53.863+0000: 2828: debug : virDomainFree:2313 :
dom=0x7f4f7c0d1430, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:02:54.459+0000: 2834: debug : virDomainInterfaceStats:7223 :
dom=0x7f4f6c000910, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0,
stats=0x7f4f95cc4b00, size=64
2013-01-31 18:02:54.459+0000: 2834: debug : virDomainFree:2313 :
dom=0x7f4f6c000910, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:02:59.464+0000: 2825: debug : virDomainInterfaceStats:7223 :
dom=0x7f4f8c2455d0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0,
stats=0x7f4f9b6cdb00, size=64
2013-01-31 18:02:59.472+0000: 2825: debug : virDomainFree:2313 :
dom=0x7f4f8c2455d0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:03:04.478+0000: 2829: debug : virDrvSupportsFeature:1521 :
conn=0x7f4f7c000ae0, feature=9
2013-01-31 18:03:04.487+0000: 2826: debug : virDomainGetCPUStats:18676 :
dom=0x7f4f70000dc0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb), params=(nil), nparams=0,
start_cpu=-1, ncpus=1, flags=4
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupNew:603 : New group
/libvirt/qemu/sl63
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected
mount/mapping 0:cpu at /cgroup/cpu in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected
mount/mapping 1:cpuacct at /cgroup/cpuacct in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected
mount/mapping 2:cpuset at /cgroup/cpuset in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected
mount/mapping 3:memory at /cgroup/memory in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected
mount/mapping 4:devices at /cgroup/devices in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected
mount/mapping 5:freezer at /cgroup/freezer in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupDetect:262 : Detected
mount/mapping 6:blkio at /cgroup/blkio in
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:524 : Make
group /libvirt/qemu/sl63
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/cpu/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/cpuacct/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/cpuset/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/memory/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/devices/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/freezer/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/blkio/libvirt/qemu/sl63/
2013-01-31 18:03:04.487+0000: 2826: debug : virDomainFree:2313 :
dom=0x7f4f70000dc0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:03:04.491+0000: 2828: debug : virDrvSupportsFeature:1521 :
conn=0x7f4f7c000ae0, feature=9
2013-01-31 18:03:04.495+0000: 2827: debug : virDomainGetCPUStats:18676 :
dom=0x7f4f74000ce0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb), params=0x7f4f74000b20,
nparams=3, start_cpu=-1, ncpus=1, flags=4
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupNew:603 : New group
/libvirt/qemu/sl63
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected
mount/mapping 0:cpu at /cgroup/cpu in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected
mount/mapping 1:cpuacct at /cgroup/cpuacct in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected
mount/mapping 2:cpuset at /cgroup/cpuset in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected
mount/mapping 3:memory at /cgroup/memory in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected
mount/mapping 4:devices at /cgroup/devices in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected
mount/mapping 5:freezer at /cgroup/freezer in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupDetect:262 : Detected
mount/mapping 6:blkio at /cgroup/blkio in
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:524 : Make
group /libvirt/qemu/sl63
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/cpu/libvirt/qemu/sl63/
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/cpuacct/libvirt/qemu/sl63/
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/cpuset/libvirt/qemu/sl63/
2013-01-31 18:03:04.495+0000: 2827: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/memory/libvirt/qemu/sl63/
2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/devices/libvirt/qemu/sl63/
2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/freezer/libvirt/qemu/sl63/
2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupMakeGroup:546 : Make
controller /cgroup/blkio/libvirt/qemu/sl63/
2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupGetValueStr:350 : Get
value /cgroup/cpuacct/libvirt/qemu/sl63/cpuacct.usage
2013-01-31 18:03:04.496+0000: 2827: debug : virCgroupGetValueStr:350 : Get
value /cgroup/cpuacct/libvirt/qemu/sl63/cpuacct.stat
2013-01-31 18:03:04.496+0000: 2827: debug : virDomainFree:2313 :
dom=0x7f4f74000ce0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:03:04.500+0000: 2825: debug : virDomainInterfaceStats:7223 :
dom=0x7f4f8c2455d0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0,
stats=0x7f4f9b6cdb00, size=64
2013-01-31 18:03:04.500+0000: 2825: debug : virDomainFree:2313 :
dom=0x7f4f8c2455d0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb)
2013-01-31 18:03:09.505+0000: 2829: debug : virDomainInterfaceStats:7223 :
dom=0x7f4f800034a0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb), path=vnet0,
stats=0x7f4f98ec9b00, size=64
2013-01-31 18:03:09.520+0000: 2829: debug : virDomainFree:2313 :
dom=0x7f4f800034a0, (VM: name=sl63,
uuid=887d764a-f835-4112-9eda-836a772ea5eb)
On Thu, Jan 31, 2013 at 11:27 AM, Dead Horse
<deadhorseconsulting at gmail.com>wrote:
> note ignore the IP diff in the ssh host auth --> copy/paste fail ;)
> - DHC
>
>
> On Thu, Jan 31, 2013 at 11:25 AM, Dead Horse <
> deadhorseconsulting at gmail.com> wrote:
>
>> Doh, brain fart VDSM is not involved here for the purposed of the needed
>> test.
>> Here is my initial whack at it:
>>
>> Source Node:
>>
>> virsh # list
>> Id Name State
>> ----------------------------------------------------
>> 1 sl63 running
>>
>> virsh # migrate --p2p sl63 qemu+ssh://192.168.1.2/system
>> error: operation failed: Failed to connect to remote libvirt URI
>> qemu+ssh://192.168.1.2/system <http://3.57.111.32/system>
>>
>> virsh # migrate --live sl63 qemu+ssh://192.168.1.2/system
>> The authenticity of host '192.168.1.2 (192.168.1.2)' can't be established.
>> RSA key fingerprint is e5:1d:b3:e5:38:5f:e1:8b:73:26:9e:15:c8:0a:2d:ac.
>> Are you sure you want to continue connecting (yes/no)? yes
>> root at 192.168.1.2's password:
>> Please enter your authentication name: vdsm at ovirt
>> Please enter your password:
>>
>> virsh #
>>
>>
>> Dest Node After migrate --live:
>> virsh # list
>> Id Name State
>> ----------------------------------------------------
>> 2 sl63 running
>>
>> virsh #
>>
>>
>>
>> On Thu, Jan 31, 2013 at 10:38 AM, Dead Horse <
>> deadhorseconsulting at gmail.com> wrote:
>>
>>> Shu,
>>> I build oVirt Engine and vdsm from source myself. The commits I
>>> indicated are what I built from.I run the engine under FC17 and my nodes
>>> are running EL6.x respectively.
>>>
>>> Dan,
>>> I reverted VDSM on my two test nodes to an earlier build of VDSM
>>> (commit:
>>> c343e1833f7b6e5428dd90f14f7807dca1baa0b4)
>>> VDSM after the above commit is broken due to commit:
>>> fc3a44f71d2ef202cff18d7203b9e4165b546621 however when I built and tested
>>> from master yesterday I did apply a patch I tested for
>>> ybronhei which fixed that issue.
>>>
>>> I will build VDSM from master, today w/ the supervdsm patch and try the
>>> manual migration you indicated.
>>>
>>> - DHC
>>>
>>>
>>>
>>> On Thu, Jan 31, 2013 at 4:56 AM, Dan Kenigsberg <danken at redhat.com>wrote:
>>>
>>>> On Thu, Jan 31, 2013 at 11:08:58AM +0100, Martin Kletzander wrote:
>>>> > On 01/31/2013 10:25 AM, Dan Kenigsberg wrote:
>>>> > > On Thu, Jan 31, 2013 at 09:43:44AM +0100, Martin Kletzander wrote:
>>>> > >> On 01/30/2013 08:40 PM, Dead Horse wrote:
>>>> > >>> The nodes are EL6.3 based.
>>>> > >>>
>>>> > >>> Currently installed libvirt packages:
>>>> > >>>
>>>> > >>> libvirt-lock-sanlock-0.9.10-21.el6_3.8.x86_64
>>>> > >>> libvirt-cim-0.6.1-3.el6.x86_64
>>>> > >>> libvirt-0.9.10-21.el6_3.8.x86_64
>>>> > >>> libvirt-python-0.9.10-21.el6_3.8.x86_64
>>>> > >>> libvirt-client-0.9.10-21.el6_3.8.x86_64
>>>> > >>>
>>>> > >>> and qemu packages:
>>>> > >>> qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64
>>>> > >>> qemu-kvm-tools-0.12.1.2-2.295.el6_3.10.x86_64
>>>> > >>> qemu-img-0.12.1.2-2.295.el6_3.10.x86_64
>>>> > >>>
>>>> > >>> Thus my presumption here given the above is that
>>>> virDomainMigrateToURI2 has
>>>> > >>> not yet been patched and/or back-ported into the EL6.x
>>>> libvirt/qemu?
>>>> > >>>
>>>> > >>
>>>> > >> virDomainMigrateToURI2 is supported since 0.9.2, but is there a
>>>> > >> possibility the code is requesting direct migration? That might
>>>> explain
>>>> > >> the message, which is then incorrect; this was fixed in [1].
>>>> > >>
>>>> > >> Martin
>>>> > >>
>>>> > >> [1]
>>>> > >>
>>>> http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=3189dfb1636da22d426d2fc07cc9f60304b16c5c
>>>> > >
>>>> > > What is "direct migration" exactly, in the context of qemu-kvm?
>>>> > >
>>>> > > We are using p2p migration
>>>> > >
>>>> http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=vdsm/libvirtvm.py;h=fe140ecbfac665248e2ad5c4bfaebaf54ab884cc;hb=18c24f7c7c27ac732c4a760caa9524e7319cd47e#l501
>>>> > >
>>>> >
>>>> > OK, so that's not the issue, sorry for the confusion. I was thinking
>>>> it
>>>> > would "somehow" get there. Direct migration doesn't exist in QEMU at
>>>> > all, so it seemed weird, but I can't seem to find any other reason for
>>>> > this failure; will keep searching, though.
>>>>
>>>> In this case, Dead Horse, would you try to migrate a VM (that you do not
>>>> care much about) using
>>>> virsh -c qemu+tls://hostname/system migrate --p2p dsthost?
>>>>
>>>> I'd like to see that the problem reproduces this way, too. More of
>>>> libvirtd.log may help. You may want to disable iptables for a moment,
>>>> just to eliminate a common cause of failure.
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20130131/c47c91f8/attachment-0001.html>
More information about the Users
mailing list