Re: [ovirt-users] oVIRT 4 / OVN / Communication issues of instances between nodes.

6 Dec 2016

      Lance,

Well I installed the new kernel module and it cleared up a lot of the
errors I was seeing in the log, but what I notice is that I still can't
ping instances between hosts.  I'm starting to wonder am I missing
something fundamental here? I don't see anything in the ovs-vswitchd.log to
show tunnel?

I do show in the kernel log on reload of the module:

[1056295.308707] openvswitch: module verification failed: signature and/or
required key missing - tainting kernel
[1056295.311034] openvswitch: Open vSwitch switching datapath 2.6.90
[1056295.311145] openvswitch: LISP tunneling driver
[1056295.311147] openvswitch: GRE over IPv4 tunneling driver
[1056295.311153] openvswitch: Geneve tunneling driver
[1056295.311164] openvswitch: VxLAN tunneling driver
[1056295.311166] openvswitch: STT tunneling driver

[node2]

[root@ovirt-node2 openvswitch]# brctl show
bridge name bridge id STP enabled interfaces
;vdsmdummy; 8000.000000000000 no
DEV-NOC 8000.0cc47a1ef306 no bond0
DEV-VM-NET 8000.0cc47a1ef306 no bond0.700
ovirtmgmt 8000.0cc47a08b3c2 no enp7s0f0

-- 

Devin Acosta
Red Hat Certified Architect, LinuxStack
devin@linuxguru.co

On Mon, Dec 5, 2016 at 2:34 PM, Lance Richardson <lrichard@redhat.com>
wrote:
...
...
From: "Devin Acosta" <devin@pabstatencio.com>
To: "Lance Richardson" <lrichard@redhat.com>
Cc: "Marcin Mirecki" <mmirecki@redhat.com>, "users" <Users@ovirt.org>
Sent: Monday, December 5, 2016 4:17:35 PM
Subject: Re: [ovirt-users] oVIRT 4 / OVN / Communication issues of
instances between nodes.
Lance,
I found some interesting logs, we have (3) oVIRT nodes.
We are running:
CentOS Linux release 7.2.1511 (Core)
Linux hostname 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC
2016 x86_64 x86_64 x86_64 GNU/Linux
<snip>
...
2016-12-05T20:47:56.774Z|00021|ofctrl|INFO|OpenFlow error: OFPT_ERROR
(OF1.3) (xid=0x17): OFPBAC_BAD_TYPE
This (generally unintelligible message usually indicates that the kernel
openvswitch module doesn't support conntrack.
<snip>
...
2016-12-05T20:35:04.345Z|00001|vlog|INFO|opened log file
/var/log/openvswitch/ovs-vswitchd.log
2016-12-05T20:35:04.347Z|00002|ovs_numa|INFO|Discovered 16 CPU cores on
NUMA node 0
2016-12-05T20:35:04.347Z|00003|ovs_numa|INFO|Discovered 16 CPU cores on
NUMA node 1
2016-12-05T20:35:04.347Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes
and 32
...
CPU cores
2016-12-05T20:35:04.348Z|00005|reconnect|INFO|unix:/
var/run/openvswitch/db.sock:
connecting...
2016-12-05T20:35:04.348Z|00006|reconnect|INFO|unix:/
var/run/openvswitch/db.sock:
connected
2016-12-05T20:35:04.350Z|00007|ofproto_dpif|INFO|system@ovs-system:
Datapath supports recirculation
2016-12-05T20:35:04.350Z|00008|ofproto_dpif|INFO|system@ovs-system: MPLS
label stack length probed as 1
2016-12-05T20:35:04.350Z|00009|ofproto_dpif|INFO|system@ovs-system:
Datapath does not support truncate action
2016-12-05T20:35:04.350Z|00010|ofproto_dpif|INFO|system@ovs-system:
Datapath supports unique flow ids
2016-12-05T20:35:04.350Z|00011|ofproto_dpif|INFO|system@ovs-system:
Datapath does not support ct_state
2016-12-05T20:35:04.350Z|00012|ofproto_dpif|INFO|system@ovs-system:
Datapath does not support ct_zone
2016-12-05T20:35:04.350Z|00013|ofproto_dpif|INFO|system@ovs-system:
Datapath does not support ct_mark
2016-12-05T20:35:04.350Z|00014|ofproto_dpif|INFO|system@ovs-system:
Datapath does not support ct_label
2016-12-05T20:35:04.350Z|00015|ofproto_dpif|INFO|system@ovs-system:
Datapath does not support ct_state_nat
OK, "Datapath does not support ct_*" confirms that the kernel openvswitch
module doesn't support the conntrack features needed by OVN.
Most likely the loaded module is the stock CentOS one, you can build
the out-of-tree kernel module RPM from the same source tree where you
built the other OVS/OVN RPMs via:
make rpm-fedora-kmod
This should leave an RPM named something like:
openvswitch-kmod-2.6.90-1.el7.centos.x86_64.rpm
Install that and reboot and things should be working better.
Regards,
Lance
...
Your help is greatly appreciated!
Devin
On Mon, Dec 5, 2016 at 12:31 PM, Lance Richardson <lrichard@redhat.com>
wrote:
...
...
From: "Devin Acosta" <devin@pabstatencio.com>
To: "Marcin Mirecki" <mmirecki@redhat.com>
Cc: "users" <Users@ovirt.org>
Sent: Monday, December 5, 2016 12:11:46 PM
Subject: Re: [ovirt-users] oVIRT 4 / OVN / Communication issues of
instances between nodes.
Marcin,
Also I noticed in your original post it mentions:
ip link - the result should include a link called genev_sys_ ...
I noticed that on my hosts I don't see any links with name:
genev_sys_ ??
...
...
...
Could this be a problem?
lo:
enp4s0f0:
enp4s0f1:
enp7s0f0:
enp7s0f1:
bond0:
DEV-NOC:
ovirtmgmt:
bond0.700@bond0:
DEV-VM-NET:
bond0.705@bond0:
;vdsmdummy;:
vnet0:
vnet1:
vnet2:
vnet3:
vnet4:
ovs-system:
br-int:
vnet5:
vnet6:
Hi Devin,
What distribution and kernel version are you using?
One thing you could check is whether the vport_geneve kernel module
is being loaded, e.g. you should see something like:
$ lsmod | grep vport
    vport_geneve           12560  1
    openvswitch           246755  5 vport_geneve
If vport_geneve is  not loaded, you could "sudo modprobe vport_geneve"
to make sure it's available and can be loaded.
The first 100 lines or so of ovs-vswitchd.log might have some useful
information about where things are going wrong.
It does sound as though there is some issue with geneve tunnels,
which would certainly explain issues with inter-node traffic.
Regards,
Lance
--
Devin Acosta
Red Hat Certified Architect, LinuxStack
602-354-1220 || devin@linuxguru.co