[Users] Migration issues with ovirt 3.3

Wed Oct 9 09:43:57 EDT 2013

On Wed, Oct 09, 2013 at 02:42:22PM +0200, Gianluca Cecchi wrote:
> On Tue, Oct 8, 2013 at 10:40 AM, Dan Kenigsberg wrote:
> 
> >
> >>
> >> But migration still fails
> >>
> >
> > It seems like an unrelated failure. I do not know what's blocking
> > migration traffic. Could you see if libvirtd.log and qemu logs at source
> > and destinaiton have clues?
> >
> 
> It seems that on VM.log under qemu of desdt host I have:
> ...
> -incoming tcp:[::]:49153: Failed to bind socket: Address already in use

Is that port really taken (`ss -ntp` should tell by whom)?

> 
> 
> See all:
> - In libvirtd.log of source host
> 2013-10-07 23:20:54.471+0000: 1209: debug :
> qemuMonitorOpenInternal:751 : QEMU_MONITOR_NEW: mon=0x7fc66412e820
> refs=2 fd=30
> 2013-10-07 23:20:54.472+0000: 1209: warning :
> qemuDomainObjEnterMonitorInternal:1136 : This thread seems to be the
> async job owner; entering monitor without asking for a nested job is
> dangerous
> 2013-10-07 23:20:54.472+0000: 1209: debug :
> qemuMonitorSetCapabilities:1145 : mon=0x7fc66412e820
> 2013-10-07 23:20:54.472+0000: 1209: debug : qemuMonitorSend:887 :
> QEMU_MONITOR_SEND_MSG: mon=0x7fc66412e820
> msg={"execute":"qmp_capabilities","id":"libvirt-1"}
>  fd=-1
> 2013-10-07 23:20:54.769+0000: 1199: error : qemuMonitorIORead:505 :
> Unable to read from monitor: Connection reset by peer
> 2013-10-07 23:20:54.769+0000: 1199: debug : qemuMonitorIO:638 : Error
> on monitor Unable to read from monitor: Connection reset by peer
> 2013-10-07 23:20:54.769+0000: 1199: debug : qemuMonitorIO:672 :
> Triggering error callback
> 2013-10-07 23:20:54.769+0000: 1199: debug :
> qemuProcessHandleMonitorError:351 : Received error on 0x7fc664124fb0
> 'c8again32'
> 2013-10-07 23:20:54.769+0000: 1209: debug : qemuMonitorSend:899 : Send
> command resulted in error Unable to read from monitor: Connection
> reset by peer
> 2013-10-07 23:20:54.770+0000: 1199: debug : qemuMonitorIO:638 : Error
> on monitor Unable to read from monitor: Connection reset by peer
> 2013-10-07 23:20:54.770+0000: 1209: debug : virFileMakePathHelper:1283
> : path=/var/run/libvirt/qemu mode=0777
> 2013-10-07 23:20:54.770+0000: 1199: debug : qemuMonitorIO:661 :
> Triggering EOF callback
> 2013-10-07 23:20:54.770+0000: 1199: debug :
> qemuProcessHandleMonitorEOF:294 : Received EOF on 0x7fc664124fb0
> 'c8again32'
> 2013-10-07 23:20:54.770+0000: 1209: debug : qemuProcessStop:3992 :
> Shutting down VM 'c8again32' pid=18053 flags=0
> 2013-10-07 23:20:54.771+0000: 1209: error :
> virNWFilterDHCPSnoopEnd:2135 : internal error ifname "vnet0" not in
> key map
> 2013-10-07 23:20:54.782+0000: 1209: debug : virCommandRunAsync:2251 :
> About to run /bin/sh -c 'IPT="/usr/sbin/iptables"
> $IPT -D libvirt-out -m physdev --physdev-is-bridged --physdev-out
> vnet0 -g FO-vnet0
> $IPT -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0
> $IPT -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0
> $IPT -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0
> $IPT -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT
> $IPT -F FO-vnet0
> $IPT -X FO-vnet0
> $IPT -F FI-vnet0
> $IPT -X FI-vnet0
> $IPT -F HI-vnet0
> $IPT -X HI-vnet0
> IPT="/usr/sbin/ip6tables"
> $IPT -D libvirt-out -m physdev --physdev-is-bridged --physdev-out
> vnet0 -g FO-vnet0
> $IPT -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0
> $IPT -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0
> $IPT -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0
> $IPT -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT
> $IPT -F FO-vnet0
> $IPT -X FO-vnet0
> $IPT -F FI-vnet0
> $IPT -X FI-vnet0
> $IPT -F HI-vnet0
> $IPT -X HI-vnet0
> EBT="/usr/sbin/ebtables"
> $EBT -t nat -D PREROUTING -i vnet0 -j libvirt-I-vnet0
> $EBT -t nat -D POSTROUTING -o vnet0 -j libvirt-O-vnet0
> EBT="/usr/sbin/ebtables"
> collect_chains()
> {
>   for tmp2 in $*; do
>     for tmp in $($EBT -t nat -L $tmp2 | \
>       sed -n "/Bridge chain/,\$ s/.*-j \\([IO]-.*\\)/\\1/p");
>     do
>       echo $tmp
>       collect_chains $tmp
>     done
>   done
> }
> rm_chains()
> {
>   for tmp in $*; do $EBT -t nat -F $tmp; done
>   for tmp in $*; do $EBT -t nat -X $tmp; done
> }
> tmp='\''
> '\''
> IFS='\'' '\'''\''       '\''$tmp
> chains="$(collect_chains libvirt-I-vnet0 libvirt-O-vnet0)"
> $EBT -t nat -F libvirt-I-vnet0
> $EBT -t nat -F libvirt-O-vnet0
> rm_chains $chains
> $EBT -t nat -F libvirt-I-vnet0
> $EBT -t nat -X libvirt-I-vnet0
> $EBT -t nat -F libvirt-O-vnet0
> $EBT -t nat -X libvirt-O-vnet0
> '
> 2013-10-07 23:20:54.784+0000: 1209: debug : virCommandRunAsync:2256 :
> Command result 0, with PID 18076
> 2013-10-07 23:20:54.863+0000: 1209: debug : virCommandRun:2125 :
> Result exit status 255, stdout: '' stderr: 'iptables v1.4.18: goto
> 'FO-vnet0' is not a chain
> 
> Try `iptables -h' or 'iptables --help' for more information.
> iptables v1.4.18: goto 'FO-vnet0' is not a chain
> 
> Try `iptables -h' or 'iptables --help' for more information.
> iptables v1.4.18: goto 'FI-vnet0' is not a chain
> Try `iptables -h' or 'iptables --help' for more information.
> iptables v1.4.18: goto 'HI-vnet0' is not a chain
> 
> Try `iptables -h' or 'iptables --help' for more information.
> iptables: Bad rule (does a matching rule exist in that chain?).
> iptables: No chain/target/match by that name.
> iptables: No chain/target/match by that name.
> iptables: No chain/target/match by that name.
> iptables: No chain/target/match by that name.
> iptables: No chain/target/match by that name.
> iptables: No chain/target/match by that name.
> ip6tables v1.4.18: goto 'FO-vnet0' is not a chain
> 
> Try `ip6tables -h' or 'ip6tables --help' for more information.
> ip6tables v1.4.18: goto 'FO-vnet0' is not a chain
> 
> Try `ip6tables -h' or 'ip6tables --help' for more information.
> ip6tables v1.4.18: goto 'FI-vnet0' is not a chain
> 
> Try `ip6tables -h' or 'ip6tables --help' for more information.
> ip6tables v1.4.18: goto 'HI-vnet0' is not a chain
> 
> Try `ip6tables -h' or 'ip6tables --help' for more information.
> ip6tables: Bad rule (does a matching rule exist in that chain?).
> ip6tables: No chain/target/match by that name.
> ip6tables: No chain/target/match by that name.
> ip6tables: No chain/target/match by that name.
> ip6tables: No chain/target/match by that name.
> ip6tables: No chain/target/match by that name.
> ip6tables: No chain/target/match by that name.
> Illegal target name 'libvirt-O-vnet0'.
> Chain 'libvirt-O-vnet0' doesn't exist.
> Chain 'libvirt-O-vnet0' doesn't exist.
> Chain 'libvirt-O-vnet0' doesn't exist.
> Chain 'libvirt-O-vnet0' doesn't exist.
> '
> 2013-10-07 23:20:54.863+0000: 1209: debug : qemuMonitorClose:821 :
> QEMU_MONITOR_CLOSE: mon=0x7fc66412e820 refs=2
> 2013-10-07 23:20:54.863+0000: 1209: debug : qemuProcessKill:3951 :
> vm=c8again32 pid=18053 flags=5
> 2013-10-07 23:20:54.863+0000: 1209: debug :
> virProcessKillPainfully:269 : vpid=18053 force=1
> 2013-10-07 23:20:54.863+0000: 1209: debug : qemuDomainCleanupRun:2132
> : driver=0x7fc664024cd0, vm=c8again32
> 2013-10-07 23:20:54.863+0000: 1209: debug :
> qemuProcessAutoDestroyRemove:4504 : vm=c8again32
> 2013-10-07 23:20:54.863+0000: 1209: debug :
> virQEMUCloseCallbacksUnset:744 : vm=c8again32,
> uuid=d54660a2-45ed-41ae-ab99-a6f93ebbdbb1, cb=0x7fc66b6fe570
> 2013-10-07 23:20:54.864+0000: 1209: error :
> virPortAllocatorRelease:174 : port 0 must be in range (5900, 65535)
> 2013-10-07 23:20:54.865+0000: 1209: debug : qemuDomainObjEndJob:1070 :
> Stopping job: none (async=migration in)
> 2013-10-07 23:20:54.865+0000: 1209: debug :
> qemuDomainObjEndAsyncJob:1088 : Stopping async job: migration in
> 2013-10-07 23:20:54.865+0000: 1199: debug :
> qemuProcessHandleMonitorEOF:306 : Domain 0x7fc664124fb0 is not active,
> ignoring EOF
> Caught Segmentation violation dumping internal log buffer:

This line one seems ominous. Can libvir-list help with it?