From: "Devin Acosta" <devin@pabstatencio.com> To: "Lance Richardson" <lrichard@redhat.com> Cc: "Marcin Mirecki" <mmirecki@redhat.com>, "users" <Users@ovirt.org> Sent: Monday, December 5, 2016 4:17:35 PM Subject: Re: [ovirt-users] oVIRT 4 / OVN / Communication issues of instances between nodes.
Lance,
I found some interesting logs, we have (3) oVIRT nodes.
We are running: CentOS Linux release 7.2.1511 (Core) Linux hostname 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
<snip>
2016-12-05T20:47:56.774Z|00021|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x17): OFPBAC_BAD_TYPE
This (generally unintelligible message usually indicates that the kernel openvswitch module doesn't support conntrack. <snip>
2016-12-05T20:35:04.345Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log 2016-12-05T20:35:04.347Z|00002|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 0 2016-12-05T20:35:04.347Z|00003|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 1 2016-12-05T20:35:04.347Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 32 CPU cores 2016-12-05T20:35:04.348Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2016-12-05T20:35:04.348Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2016-12-05T20:35:04.350Z|00007|ofproto_dpif|INFO|system@ovs-system: Datapath supports recirculation 2016-12-05T20:35:04.350Z|00008|ofproto_dpif|INFO|system@ovs-system: MPLS label stack length probed as 1 2016-12-05T20:35:04.350Z|00009|ofproto_dpif|INFO|system@ovs-system: Datapath does not support truncate action 2016-12-05T20:35:04.350Z|00010|ofproto_dpif|INFO|system@ovs-system: Datapath supports unique flow ids 2016-12-05T20:35:04.350Z|00011|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_state 2016-12-05T20:35:04.350Z|00012|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_zone 2016-12-05T20:35:04.350Z|00013|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_mark 2016-12-05T20:35:04.350Z|00014|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_label 2016-12-05T20:35:04.350Z|00015|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_state_nat
OK, "Datapath does not support ct_*" confirms that the kernel openvswitch module doesn't support the conntrack features needed by OVN. Most likely the loaded module is the stock CentOS one, you can build the out-of-tree kernel module RPM from the same source tree where you built the other OVS/OVN RPMs via: make rpm-fedora-kmod This should leave an RPM named something like: openvswitch-kmod-2.6.90-1.el7.centos.x86_64.rpm Install that and reboot and things should be working better. Regards, Lance
Your help is greatly appreciated!
Devin
On Mon, Dec 5, 2016 at 12:31 PM, Lance Richardson <lrichard@redhat.com> wrote:
From: "Devin Acosta" <devin@pabstatencio.com> To: "Marcin Mirecki" <mmirecki@redhat.com> Cc: "users" <Users@ovirt.org> Sent: Monday, December 5, 2016 12:11:46 PM Subject: Re: [ovirt-users] oVIRT 4 / OVN / Communication issues of instances between nodes.
Marcin,
Also I noticed in your original post it mentions:
ip link - the result should include a link called genev_sys_ ...
I noticed that on my hosts I don't see any links with name: genev_sys_ ?? Could this be a problem?
lo: enp4s0f0: enp4s0f1: enp7s0f0: enp7s0f1: bond0: DEV-NOC: ovirtmgmt: bond0.700@bond0: DEV-VM-NET: bond0.705@bond0: ;vdsmdummy;: vnet0: vnet1: vnet2: vnet3: vnet4: ovs-system: br-int: vnet5: vnet6:
Hi Devin,
What distribution and kernel version are you using?
One thing you could check is whether the vport_geneve kernel module is being loaded, e.g. you should see something like:
$ lsmod | grep vport vport_geneve 12560 1 openvswitch 246755 5 vport_geneve
If vport_geneve is not loaded, you could "sudo modprobe vport_geneve" to make sure it's available and can be loaded.
The first 100 lines or so of ovs-vswitchd.log might have some useful information about where things are going wrong.
It does sound as though there is some issue with geneve tunnels, which would certainly explain issues with inter-node traffic.
Regards,
Lance
--
Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 || devin@linuxguru.co