<div dir="ltr"><br><div>Lance,</div><div><br></div><div>Well I installed the new kernel module and it cleared up a lot of the errors I was seeing in the log, but what I notice is that I still can't ping instances between hosts. I'm starting to wonder am I missing something fundamental here? I don't see anything in the ovs-vswitchd.log to show tunnel? </div><div><br></div><div>I do show in the kernel log on reload of the module:</div><div><br></div><div><div>[1056295.308707] openvswitch: module verification failed: signature and/or required key missing - tainting kernel</div><div>[1056295.311034] openvswitch: Open vSwitch switching datapath 2.6.90</div><div>[1056295.311145] openvswitch: LISP tunneling driver</div><div>[1056295.311147] openvswitch: GRE over IPv4 tunneling driver</div><div>[1056295.311153] openvswitch: Geneve tunneling driver</div><div>[1056295.311164] openvswitch: VxLAN tunneling driver</div><div>[1056295.311166] openvswitch: STT tunneling driver</div></div><div><br></div><div>[node2]</div><div><br></div><div><div>[root@ovirt-node2 openvswitch]# cat ovs-vswitchd.log</div><div>2016-12-06T04:22:23.192Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log</div><div>2016-12-06T04:22:23.194Z|00002|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 0</div><div>2016-12-06T04:22:23.194Z|00003|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 1</div><div>2016-12-06T04:22:23.194Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 32 CPU cores</div><div>2016-12-06T04:22:23.194Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...</div><div>2016-12-06T04:22:23.195Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected</div><div>2016-12-06T04:22:23.197Z|00007|ofproto_dpif|INFO|system@ovs-system: Datapath supports recirculation</div><div>2016-12-06T04:22:23.197Z|00008|ofproto_dpif|INFO|system@ovs-system: MPLS label stack length probed as 1</div><div>2016-12-06T04:22:23.197Z|00009|ofproto_dpif|INFO|system@ovs-system: Datapath supports truncate action</div><div>2016-12-06T04:22:23.197Z|00010|ofproto_dpif|INFO|system@ovs-system: Datapath supports unique flow ids</div><div>2016-12-06T04:22:23.197Z|00011|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_state</div><div>2016-12-06T04:22:23.197Z|00012|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_zone</div><div>2016-12-06T04:22:23.197Z|00013|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_mark</div><div>2016-12-06T04:22:23.197Z|00014|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_label</div><div>2016-12-06T04:22:23.197Z|00015|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_state_nat</div><div>2016-12-06T04:22:23.339Z|00001|ofproto_dpif_upcall(handler1)|INFO|received packet on unassociated datapath port 0</div><div>2016-12-06T04:22:23.339Z|00016|bridge|INFO|bridge br-int: added interface vnet0 on port 5</div><div>2016-12-06T04:22:23.339Z|00017|bridge|INFO|bridge br-int: added interface br-int on port 65534</div><div>2016-12-06T04:22:23.339Z|00018|bridge|INFO|bridge br-int: using datapath ID 000016d6e0b66442</div><div>2016-12-06T04:22:23.339Z|00019|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt"</div><div>2016-12-06T04:22:23.340Z|00020|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.6.90</div><div>2016-12-06T04:22:32.437Z|00021|bridge|INFO|bridge br-int: added interface ovn-c0dc09-0 on port 6</div><div>2016-12-06T04:22:32.437Z|00022|bridge|INFO|bridge br-int: added interface ovn-252778-0 on port 7</div><div>2016-12-06T04:22:33.342Z|00023|memory|INFO|281400 kB peak resident set size after 10.2 seconds</div><div>2016-12-06T04:22:33.342Z|00024|memory|INFO|handlers:23 ofconns:2 ports:4 revalidators:9 rules:79</div></div><div>2016-12-06T04:22:42.440Z|00025|connmgr|INFO|br-int<->unix: 76 flow_mods 10 s ago (75 adds, 1 deletes)<br></div><div><br></div><div><div>[root@ovirt-node2 openvswitch]# cat ovn-controller.log</div><div>2016-12-06T04:22:32.398Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovn-controller.log</div><div>2016-12-06T04:22:32.400Z|00002|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...</div><div>2016-12-06T04:22:32.400Z|00003|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected</div><div>2016-12-06T04:22:32.402Z|00004|reconnect|INFO|tcp:<a href="http://172.20.192.77:6642">172.20.192.77:6642</a>: connecting...</div><div>2016-12-06T04:22:32.403Z|00005|reconnect|INFO|tcp:<a href="http://172.20.192.77:6642">172.20.192.77:6642</a>: connected</div><div>2016-12-06T04:22:32.406Z|00006|binding|INFO|Claiming lport 56432d2b-a96d-4ac7-b0e9-3450a006e1d4 for this chassis.</div><div>2016-12-06T04:22:32.406Z|00007|binding|INFO|Claiming 00:1a:4a:16:01:64 dynamic</div><div>2016-12-06T04:22:32.407Z|00008|ofctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch</div><div>2016-12-06T04:22:32.407Z|00009|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting...</div><div>2016-12-06T04:22:32.407Z|00010|pinctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch</div><div>2016-12-06T04:22:32.407Z|00011|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting...</div><div>2016-12-06T04:22:32.408Z|00012|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected</div><div>2016-12-06T04:22:32.408Z|00013|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected</div><div>2016-12-06T04:22:32.440Z|00014|ofctrl|INFO|dropping duplicate flow: table_id=32, priority=150, reg10=0x2/0x2, actions=resubmit(,33)</div><div>2016-12-06T04:22:32.441Z|00015|ofctrl|INFO|dropping duplicate flow: table_id=32, priority=150, reg10=0x2/0x2, actions=resubmit(,33)</div><div>2016-12-06T04:22:32.441Z|00016|ofctrl|INFO|dropping duplicate flow: table_id=32, priority=150, reg10=0x2/0x2, actions=resubmit(,33)</div><div>2016-12-06T04:22:37.408Z|00017|ofctrl|INFO|dropping duplicate flow: table_id=32, priority=150, reg10=0x2/0x2, actions=resubmit(,33)</div><div>2016-12-06T04:22:42.408Z|00018|ofctrl|INFO|dropping duplicate flow: table_id=32, priority=150, reg10=0x2/0x2, actions=resubmit(,33)</div><div>2016-12-06T04:22:47.409Z|00019|ofctrl|INFO|Dropped 1 log messages in last 5 seconds (most recently, 5 seconds ago) due to excessive rate</div><div>2016-12-06T04:22:47.409Z|00020|ofctrl|INFO|dropping duplicate flow: table_id=32, priority=150, reg10=0x2/0x2, actions=resubmit(,33)</div><div>2016-12-06T04:22:57.411Z|00021|ofctrl|INFO|Dropped 3 log messages in last 10 seconds (most recently, 5 seconds ago) due to excessive rate</div><div>2016-12-06T04:22:57.411Z|00022|ofctrl|INFO|dropping duplicate flow: table_id=32, priority=150, reg10=0x2/0x2, actions=resubmit(,33)</div><div>2016-12-06T04:23:12.413Z|00023|ofctrl|INFO|Dropped 4 log messages in last 10 seconds (most recently, 5 seconds ago) due to excessive rate</div><div>2016-12-06T04:23:12.413Z|00024|ofctrl|INFO|dropping duplicate flow: table_id=32, priority=150, reg10=0x2/0x2, actions=resubmit(,33)</div><div>2016-12-06T04:23:22.415Z|00025|ofctrl|INFO|Dropped 3 log messages in last 10 seconds (most recently, 5 seconds ago) due to excessive rate</div><div>2016-12-06T04:23:22.415Z|00026|ofctrl|INFO|dropping duplicate flow: table_id=32, priority=150, reg10=0x2/0x2, actions=resubmit(,33)</div><div>2016-12-06T04:23:37.417Z|00027|ofctrl|INFO|Dropped 5 log messages in last 10 seconds (most recently, 5 seconds ago) due to excessive rate</div><div>2016-12-06T04:23:37.417Z|00028|ofctrl|INFO|dropping duplicate flow: table_id=32, priority=150, reg10=0x2/0x2, actions=resubmit(,33)</div><div>2016-12-06T04:23:47.419Z|00029|ofctrl|INFO|Dropped 3 log messages in last 10 seconds (most recently, 5 seconds ago) due to excessive rate</div><div>2016-12-06T04:23:47.419Z|00030|ofctrl|INFO|dropping duplicate flow: table_id=32, priority=150, reg10=0x2/0x2, actions=resubmit(,33)</div><div>2016-12-06T04:23:57.421Z|00031|ofctrl|INFO|Dropped 3 log messages in last 10 seconds (most recently, 5 seconds ago) due to excessive rate</div><div>2016-12-06T04:23:57.421Z|00032|ofctrl|INFO|dropping duplicate flow: table_id=32, priority=150, reg10=0x2/0x2, actions=resubmit(,33)</div></div><div><div><br></div></div><div><div>[root@ovirt-node2 openvswitch]# brctl show</div><div>bridge name<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>bridge id<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>STP enabled<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>interfaces</div><div>;vdsmdummy;<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>8000.000000000000<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>no</div><div>DEV-NOC<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>8000.0cc47a1ef306<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>no<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>bond0</div><div>DEV-VM-NET<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>8000.0cc47a1ef306<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>no<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>bond0.700</div><div>ovirtmgmt<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>8000.0cc47a08b3c2<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>no<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>enp7s0f0</div></div><div><br></div><div>-- <br><div class="gmail_signature"><div dir="ltr"><div><br></div><div>Devin Acosta</div><div>Red Hat Certified Architect, LinuxStack </div><div><a href="mailto:devin@linuxguru.co" target="_blank">devin@linuxguru.co</a></div></div></div></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 5, 2016 at 2:34 PM, Lance Richardson <span dir="ltr"><<a href="mailto:lrichard@redhat.com" target="_blank">lrichard@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> From: "Devin Acosta" <<a href="mailto:devin@pabstatencio.com">devin@pabstatencio.com</a>><br>
> To: "Lance Richardson" <<a href="mailto:lrichard@redhat.com">lrichard@redhat.com</a>><br>
> Cc: "Marcin Mirecki" <<a href="mailto:mmirecki@redhat.com">mmirecki@redhat.com</a>>, "users" <<a href="mailto:Users@ovirt.org">Users@ovirt.org</a>><br>
> Sent: Monday, December 5, 2016 4:17:35 PM<br>
<span class="gmail-">> Subject: Re: [ovirt-users] oVIRT 4 / OVN / Communication issues of instances between nodes.<br>
><br>
</span><span class="gmail-">> Lance,<br>
><br>
> I found some interesting logs, we have (3) oVIRT nodes.<br>
><br>
> We are running:<br>
> CentOS Linux release 7.2.1511 (Core)<br>
> Linux hostname 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC<br>
> 2016 x86_64 x86_64 x86_64 GNU/Linux<br>
><br>
<br>
</span><snip><br>
<span class="gmail-"><br>
> 2016-12-05T20:47:56.774Z|<wbr>00021|ofctrl|INFO|OpenFlow error: OFPT_ERROR<br>
> (OF1.3) (xid=0x17): OFPBAC_BAD_TYPE<br>
<br>
</span>This (generally unintelligible message usually indicates that the kernel<br>
openvswitch module doesn't support conntrack.<br>
<br>
<snip><br>
<div><div class="gmail-h5"><br>
><br>
> 2016-12-05T20:35:04.345Z|<wbr>00001|vlog|INFO|opened log file<br>
> /var/log/openvswitch/ovs-<wbr>vswitchd.log<br>
> 2016-12-05T20:35:04.347Z|<wbr>00002|ovs_numa|INFO|Discovered 16 CPU cores on<br>
> NUMA node 0<br>
> 2016-12-05T20:35:04.347Z|<wbr>00003|ovs_numa|INFO|Discovered 16 CPU cores on<br>
> NUMA node 1<br>
> 2016-12-05T20:35:04.347Z|<wbr>00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 32<br>
> CPU cores<br>
> 2016-12-05T20:35:04.348Z|<wbr>00005|reconnect|INFO|unix:/<wbr>var/run/openvswitch/db.sock:<br>
> connecting...<br>
> 2016-12-05T20:35:04.348Z|<wbr>00006|reconnect|INFO|unix:/<wbr>var/run/openvswitch/db.sock:<br>
> connected<br>
> 2016-12-05T20:35:04.350Z|<wbr>00007|ofproto_dpif|INFO|<wbr>system@ovs-system:<br>
> Datapath supports recirculation<br>
> 2016-12-05T20:35:04.350Z|<wbr>00008|ofproto_dpif|INFO|<wbr>system@ovs-system: MPLS<br>
> label stack length probed as 1<br>
> 2016-12-05T20:35:04.350Z|<wbr>00009|ofproto_dpif|INFO|<wbr>system@ovs-system:<br>
> Datapath does not support truncate action<br>
> 2016-12-05T20:35:04.350Z|<wbr>00010|ofproto_dpif|INFO|<wbr>system@ovs-system:<br>
> Datapath supports unique flow ids<br>
> 2016-12-05T20:35:04.350Z|<wbr>00011|ofproto_dpif|INFO|<wbr>system@ovs-system:<br>
> Datapath does not support ct_state<br>
> 2016-12-05T20:35:04.350Z|<wbr>00012|ofproto_dpif|INFO|<wbr>system@ovs-system:<br>
> Datapath does not support ct_zone<br>
> 2016-12-05T20:35:04.350Z|<wbr>00013|ofproto_dpif|INFO|<wbr>system@ovs-system:<br>
> Datapath does not support ct_mark<br>
> 2016-12-05T20:35:04.350Z|<wbr>00014|ofproto_dpif|INFO|<wbr>system@ovs-system:<br>
> Datapath does not support ct_label<br>
> 2016-12-05T20:35:04.350Z|<wbr>00015|ofproto_dpif|INFO|<wbr>system@ovs-system:<br>
> Datapath does not support ct_state_nat<br>
<br>
</div></div>OK, "Datapath does not support ct_*" confirms that the kernel openvswitch<br>
module doesn't support the conntrack features needed by OVN.<br>
<br>
Most likely the loaded module is the stock CentOS one, you can build<br>
the out-of-tree kernel module RPM from the same source tree where you<br>
built the other OVS/OVN RPMs via:<br>
<br>
make rpm-fedora-kmod<br>
<br>
This should leave an RPM named something like:<br>
<br>
openvswitch-kmod-2.6.90-1.el7.<wbr>centos.x86_64.rpm<br>
<br>
Install that and reboot and things should be working better.<br>
<br>
Regards,<br>
<br>
Lance<br>
<div class="gmail-HOEnZb"><div class="gmail-h5"><br>
<br>
><br>
> Your help is greatly appreciated!<br>
><br>
> Devin<br>
><br>
> On Mon, Dec 5, 2016 at 12:31 PM, Lance Richardson <<a href="mailto:lrichard@redhat.com">lrichard@redhat.com</a>><br>
> wrote:<br>
><br>
> > > From: "Devin Acosta" <<a href="mailto:devin@pabstatencio.com">devin@pabstatencio.com</a>><br>
> > > To: "Marcin Mirecki" <<a href="mailto:mmirecki@redhat.com">mmirecki@redhat.com</a>><br>
> > > Cc: "users" <<a href="mailto:Users@ovirt.org">Users@ovirt.org</a>><br>
> > > Sent: Monday, December 5, 2016 12:11:46 PM<br>
> > > Subject: Re: [ovirt-users] oVIRT 4 / OVN / Communication issues of<br>
> > instances between nodes.<br>
> > ><br>
> > > Marcin,<br>
> > ><br>
> > > Also I noticed in your original post it mentions:<br>
> > ><br>
> > > ip link - the result should include a link called genev_sys_ ...<br>
> > ><br>
> > > I noticed that on my hosts I don't see any links with name: genev_sys_ ??<br>
> > > Could this be a problem?<br>
> > ><br>
> > > lo:<br>
> > > enp4s0f0:<br>
> > > enp4s0f1:<br>
> > > enp7s0f0:<br>
> > > enp7s0f1:<br>
> > > bond0:<br>
> > > DEV-NOC:<br>
> > > ovirtmgmt:<br>
> > > bond0.700@bond0:<br>
> > > DEV-VM-NET:<br>
> > > bond0.705@bond0:<br>
> > > ;vdsmdummy;:<br>
> > > vnet0:<br>
> > > vnet1:<br>
> > > vnet2:<br>
> > > vnet3:<br>
> > > vnet4:<br>
> > > ovs-system:<br>
> > > br-int:<br>
> > > vnet5:<br>
> > > vnet6:<br>
> > ><br>
> ><br>
> > Hi Devin,<br>
> ><br>
> > What distribution and kernel version are you using?<br>
> ><br>
> > One thing you could check is whether the vport_geneve kernel module<br>
> > is being loaded, e.g. you should see something like:<br>
> ><br>
> > $ lsmod | grep vport<br>
> > vport_geneve 12560 1<br>
> > openvswitch 246755 5 vport_geneve<br>
> ><br>
> > If vport_geneve is not loaded, you could "sudo modprobe vport_geneve"<br>
> > to make sure it's available and can be loaded.<br>
> ><br>
> > The first 100 lines or so of ovs-vswitchd.log might have some useful<br>
> > information about where things are going wrong.<br>
> ><br>
> > It does sound as though there is some issue with geneve tunnels,<br>
> > which would certainly explain issues with inter-node traffic.<br>
> ><br>
> > Regards,<br>
> ><br>
> > Lance<br>
> ><br>
><br>
><br>
><br>
> --<br>
><br>
> Devin Acosta<br>
> Red Hat Certified Architect, LinuxStack<br>
> 602-354-1220 || <a href="mailto:devin@linuxguru.co">devin@linuxguru.co</a><br>
><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div><br>
</div></div>