[ovirt-users] oVIRT 4 / OVN / Communication issues of instances between nodes.

[Lance Richardson]

> From: "Devin Acosta" <devin at pabstatencio.com>
> To: "Lance Richardson" <lrichard at redhat.com>
> Cc: "Marcin Mirecki" <mmirecki at redhat.com>, "users" <Users at ovirt.org>
> Sent: Monday, December 5, 2016 4:17:35 PM
> Subject: Re: [ovirt-users] oVIRT 4 / OVN / Communication issues of instances between nodes.
> 
> Lance,
> 
> I found some interesting logs, we have (3) oVIRT nodes.
> 
> We are running:
> CentOS Linux release 7.2.1511 (Core)
> Linux hostname 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC
> 2016 x86_64 x86_64 x86_64 GNU/Linux
> 

<snip>

> 2016-12-05T20:47:56.774Z|00021|ofctrl|INFO|OpenFlow error: OFPT_ERROR
> (OF1.3) (xid=0x17): OFPBAC_BAD_TYPE

This (generally unintelligible message usually indicates that the kernel
openvswitch module doesn't support conntrack.

<snip>

> 
> 2016-12-05T20:35:04.345Z|00001|vlog|INFO|opened log file
> /var/log/openvswitch/ovs-vswitchd.log
> 2016-12-05T20:35:04.347Z|00002|ovs_numa|INFO|Discovered 16 CPU cores on
> NUMA node 0
> 2016-12-05T20:35:04.347Z|00003|ovs_numa|INFO|Discovered 16 CPU cores on
> NUMA node 1
> 2016-12-05T20:35:04.347Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 32
> CPU cores
> 2016-12-05T20:35:04.348Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
> connecting...
> 2016-12-05T20:35:04.348Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
> connected
> 2016-12-05T20:35:04.350Z|00007|ofproto_dpif|INFO|system at ovs-system:
> Datapath supports recirculation
> 2016-12-05T20:35:04.350Z|00008|ofproto_dpif|INFO|system at ovs-system: MPLS
> label stack length probed as 1
> 2016-12-05T20:35:04.350Z|00009|ofproto_dpif|INFO|system at ovs-system:
> Datapath does not support truncate action
> 2016-12-05T20:35:04.350Z|00010|ofproto_dpif|INFO|system at ovs-system:
> Datapath supports unique flow ids
> 2016-12-05T20:35:04.350Z|00011|ofproto_dpif|INFO|system at ovs-system:
> Datapath does not support ct_state
> 2016-12-05T20:35:04.350Z|00012|ofproto_dpif|INFO|system at ovs-system:
> Datapath does not support ct_zone
> 2016-12-05T20:35:04.350Z|00013|ofproto_dpif|INFO|system at ovs-system:
> Datapath does not support ct_mark
> 2016-12-05T20:35:04.350Z|00014|ofproto_dpif|INFO|system at ovs-system:
> Datapath does not support ct_label
> 2016-12-05T20:35:04.350Z|00015|ofproto_dpif|INFO|system at ovs-system:
> Datapath does not support ct_state_nat

OK, "Datapath does not support ct_*" confirms that the kernel openvswitch
module doesn't support the conntrack features needed by OVN.

Most likely the loaded module is the stock CentOS one, you can build
the out-of-tree kernel module RPM from the same source tree where you
built the other OVS/OVN RPMs via:

   make rpm-fedora-kmod

This should leave an RPM named something like:

   openvswitch-kmod-2.6.90-1.el7.centos.x86_64.rpm

Install that and reboot and things should be working better.

Regards,

   Lance


> 
> Your help is greatly appreciated!
> 
> Devin
> 
> On Mon, Dec 5, 2016 at 12:31 PM, Lance Richardson <lrichard at redhat.com>
> wrote:
> 
> > > From: "Devin Acosta" <devin at pabstatencio.com>
> > > To: "Marcin Mirecki" <mmirecki at redhat.com>
> > > Cc: "users" <Users at ovirt.org>
> > > Sent: Monday, December 5, 2016 12:11:46 PM
> > > Subject: Re: [ovirt-users] oVIRT 4 / OVN / Communication issues of
> > instances between nodes.
> > >
> > > Marcin,
> > >
> > > Also I noticed in your original post it mentions:
> > >
> > > ip link - the result should include a link called genev_sys_ ...
> > >
> > > I noticed that on my hosts I don't see any links with name: genev_sys_ ??
> > > Could this be a problem?
> > >
> > > lo:
> > > enp4s0f0:
> > > enp4s0f1:
> > > enp7s0f0:
> > > enp7s0f1:
> > > bond0:
> > > DEV-NOC:
> > > ovirtmgmt:
> > > bond0.700 at bond0:
> > > DEV-VM-NET:
> > > bond0.705 at bond0:
> > > ;vdsmdummy;:
> > > vnet0:
> > > vnet1:
> > > vnet2:
> > > vnet3:
> > > vnet4:
> > > ovs-system:
> > > br-int:
> > > vnet5:
> > > vnet6:
> > >
> >
> > Hi Devin,
> >
> > What distribution and kernel version are you using?
> >
> > One thing you could check is whether the vport_geneve kernel module
> > is being loaded, e.g. you should see something like:
> >
> >     $ lsmod | grep vport
> >     vport_geneve           12560  1
> >     openvswitch           246755  5 vport_geneve
> >
> > If vport_geneve is  not loaded, you could "sudo modprobe vport_geneve"
> > to make sure it's available and can be loaded.
> >
> > The first 100 lines or so of ovs-vswitchd.log might have some useful
> > information about where things are going wrong.
> >
> > It does sound as though there is some issue with geneve tunnels,
> > which would certainly explain issues with inter-node traffic.
> >
> > Regards,
> >
> >     Lance
> >
> 
> 
> 
> --
> 
> Devin Acosta
> Red Hat Certified Architect, LinuxStack
> 602-354-1220 || devin at linuxguru.co
>