11:34 p.m.
From: "Devin Acosta" <devin(a)pabstatencio.com>
To: "Lance Richardson" <lrichard(a)redhat.com>
Cc: "Marcin Mirecki" <mmirecki(a)redhat.com>, "users"
<Users(a)ovirt.org>
Sent: Monday, December 5, 2016 4:17:35 PM
Subject: Re: [ovirt-users] oVIRT 4 / OVN / Communication issues of instances between
nodes.
Lance,
I found some interesting logs, we have (3) oVIRT nodes.
We are running:
CentOS Linux release 7.2.1511 (Core)
Linux hostname 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC
2016 x86_64 x86_64 x86_64 GNU/Linux
<snip>
2016-12-05T20:47:56.774Z|00021|ofctrl|INFO|OpenFlow error:
OFPT_ERROR
(OF1.3) (xid=0x17): OFPBAC_BAD_TYPE
This (generally unintelligible message usually indicates that the kernel
openvswitch module doesn't support conntrack.
<snip>
2016-12-05T20:35:04.345Z|00001|vlog|INFO|opened log file
/var/log/openvswitch/ovs-vswitchd.log
2016-12-05T20:35:04.347Z|00002|ovs_numa|INFO|Discovered 16 CPU cores on
NUMA node 0
2016-12-05T20:35:04.347Z|00003|ovs_numa|INFO|Discovered 16 CPU cores on
NUMA node 1
2016-12-05T20:35:04.347Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 32
CPU cores
2016-12-05T20:35:04.348Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
connecting...
2016-12-05T20:35:04.348Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
connected
2016-12-05T20:35:04.350Z|00007|ofproto_dpif|INFO|system@ovs-system:
Datapath supports recirculation
2016-12-05T20:35:04.350Z|00008|ofproto_dpif|INFO|system@ovs-system: MPLS
label stack length probed as 1
2016-12-05T20:35:04.350Z|00009|ofproto_dpif|INFO|system@ovs-system:
Datapath does not support truncate action
2016-12-05T20:35:04.350Z|00010|ofproto_dpif|INFO|system@ovs-system:
Datapath supports unique flow ids
2016-12-05T20:35:04.350Z|00011|ofproto_dpif|INFO|system@ovs-system:
Datapath does not support ct_state
2016-12-05T20:35:04.350Z|00012|ofproto_dpif|INFO|system@ovs-system:
Datapath does not support ct_zone
2016-12-05T20:35:04.350Z|00013|ofproto_dpif|INFO|system@ovs-system:
Datapath does not support ct_mark
2016-12-05T20:35:04.350Z|00014|ofproto_dpif|INFO|system@ovs-system:
Datapath does not support ct_label
2016-12-05T20:35:04.350Z|00015|ofproto_dpif|INFO|system@ovs-system:
Datapath does not support ct_state_nat
OK, "Datapath does not support ct_*" confirms that the kernel openvswitch
module doesn't support the conntrack features needed by OVN.
Most likely the loaded module is the stock CentOS one, you can build
the out-of-tree kernel module RPM from the same source tree where you
built the other OVS/OVN RPMs via:
make rpm-fedora-kmod
This should leave an RPM named something like:
openvswitch-kmod-2.6.90-1.el7.centos.x86_64.rpm
Install that and reboot and things should be working better.
Regards,
Lance
Your help is greatly appreciated!
Devin
On Mon, Dec 5, 2016 at 12:31 PM, Lance Richardson <lrichard(a)redhat.com>
wrote:
> > From: "Devin Acosta" <devin(a)pabstatencio.com>
> > To: "Marcin Mirecki" <mmirecki(a)redhat.com>
> > Cc: "users" <Users(a)ovirt.org>
> > Sent: Monday, December 5, 2016 12:11:46 PM
> > Subject: Re: [ovirt-users] oVIRT 4 / OVN / Communication issues of
> instances between nodes.
> >
> > Marcin,
> >
> > Also I noticed in your original post it mentions:
> >
> > ip link - the result should include a link called genev_sys_ ...
> >
> > I noticed that on my hosts I don't see any links with name: genev_sys_ ??
> > Could this be a problem?
> >
> > lo:
> > enp4s0f0:
> > enp4s0f1:
> > enp7s0f0:
> > enp7s0f1:
> > bond0:
> > DEV-NOC:
> > ovirtmgmt:
> > bond0.700@bond0:
> > DEV-VM-NET:
> > bond0.705@bond0:
> > ;vdsmdummy;:
> > vnet0:
> > vnet1:
> > vnet2:
> > vnet3:
> > vnet4:
> > ovs-system:
> > br-int:
> > vnet5:
> > vnet6:
> >
>
> Hi Devin,
>
> What distribution and kernel version are you using?
>
> One thing you could check is whether the vport_geneve kernel module
> is being loaded, e.g. you should see something like:
>
> $ lsmod | grep vport
> vport_geneve 12560 1
> openvswitch 246755 5 vport_geneve
>
> If vport_geneve is not loaded, you could "sudo modprobe vport_geneve"
> to make sure it's available and can be loaded.
>
> The first 100 lines or so of ovs-vswitchd.log might have some useful
> information about where things are going wrong.
>
> It does sound as though there is some issue with geneve tunnels,
> which would certainly explain issues with inter-node traffic.
>
> Regards,
>
> Lance
>
--
Devin Acosta
Red Hat Certified Architect, LinuxStack
602-354-1220 || devin(a)linuxguru.co