Lance,

I found some interesting logs, we have (3) oVIRT nodes.

We are running:

CentOS Linux release 7.2.1511 (Core)

Linux hostname 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

What is interesting now is my 2nd node now won't hand out DHCP any more, however it does have a strange "tunnel" error message on node2 also, see below. What is bizarre node1 nor node3 show that bizarre tunnel message?

[ovirt-node1]

[root@ovirt-node1 ~]# lsmod | grep vport

vport_geneve 12815 1

geneve 13381 1 vport_geneve

openvswitch 84535 1 vport_geneve

[ovirt-node1 / ovn-controller.log]

2016-12-05T20:47:56.761Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovn-controller.log

2016-12-05T20:47:56.762Z|00002|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...

2016-12-05T20:47:56.762Z|00003|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected

2016-12-05T20:47:56.764Z|00004|reconnect|INFO|tcp:172.20.192.77:6642: connecting...

2016-12-05T20:47:56.764Z|00005|reconnect|INFO|tcp:172.20.192.77:6642: connected

2016-12-05T20:47:56.768Z|00006|binding|INFO|Claiming lport 91d4f4f5-4b9f-42c0-aa2c-8a101474bb84 for this chassis.

2016-12-05T20:47:56.768Z|00007|binding|INFO|Claiming 00:1a:4a:16:01:5e dynamic

2016-12-05T20:47:56.768Z|00008|binding|INFO|Claiming lport 71ef81f1-7c20-4c68-b536-d274703f7541 for this chassis.

2016-12-05T20:47:56.768Z|00009|binding|INFO|Claiming 00:1a:4a:16:01:61 dynamic

2016-12-05T20:47:56.768Z|00010|ofctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch

2016-12-05T20:47:56.768Z|00011|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting...

2016-12-05T20:47:56.768Z|00012|pinctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch

2016-12-05T20:47:56.768Z|00013|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting...

2016-12-05T20:47:56.768Z|00014|ofctrl|INFO|dropping duplicate flow: table_id=29, priority=50, metadata=0x2,dl_dst=00:1a:4a:16:01:62, actions=set_field:0x5->reg15,resubmit(,32)

2016-12-05T20:47:56.770Z|00015|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected

2016-12-05T20:47:56.770Z|00016|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected

2016-12-05T20:47:56.770Z|00017|ofctrl|INFO|dropping duplicate flow: table_id=29, priority=50, metadata=0x2,dl_dst=00:1a:4a:16:01:62, actions=set_field:0x5->reg15,resubmit(,32)

2016-12-05T20:47:56.771Z|00018|ofctrl|INFO|dropping duplicate flow: table_id=29, priority=50, metadata=0x2,dl_dst=00:1a:4a:16:01:62, actions=set_field:0x5->reg15,resubmit(,32)

2016-12-05T20:47:56.772Z|00019|ofctrl|INFO|dropping duplicate flow: table_id=29, priority=50, metadata=0x2,dl_dst=00:1a:4a:16:01:62, actions=set_field:0x5->reg15,resubmit(,32)

2016-12-05T20:47:56.773Z|00020|ofctrl|INFO|dropping duplicate flow: table_id=32, priority=150, reg10=0x2/0x2, actions=resubmit(,33)

2016-12-05T20:47:56.774Z|00021|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x17): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x17):

(***truncated to 64 bytes from 120***)

00000000 04 0e 00 78 00 00 00 17-00 00 00 00 00 00 00 00 |...x............|

00000010 00 00 00 00 00 00 00 00-32 00 00 00 00 00 00 64 |........2......d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|

2016-12-05T20:47:56.774Z|00022|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x1f): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x1f):

(***truncated to 64 bytes from 160***)

00000000 04 0e 00 a0 00 00 00 1f-00 00 00 00 00 00 00 00 |................|

00000010 00 00 00 00 00 00 00 00-36 00 00 00 00 00 00 64 |........6......d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|

2016-12-05T20:47:56.774Z|00023|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x21): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x21):

(***truncated to 64 bytes from 136***)

00000000 04 0e 00 88 00 00 00 21-00 00 00 00 00 00 00 00 |.......!........|

00000010 00 00 00 00 00 00 00 00-19 00 00 00 00 00 00 64 |...............d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|

2016-12-05T20:47:56.774Z|00024|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x2c): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x2c):

(***truncated to 64 bytes from 160***)

00000000 04 0e 00 a0 00 00 00 2c-00 00 00 00 00 00 00 00 |.......,........|

00000010 00 00 00 00 00 00 00 00-19 00 00 00 00 00 00 64 |...............d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|

2016-12-05T20:47:56.774Z|00025|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x2d): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x2d):

(***truncated to 64 bytes from 136***)

00000000 04 0e 00 88 00 00 00 2d-00 00 00 00 00 00 00 00 |.......-........|

00000010 00 00 00 00 00 00 00 00-36 00 00 00 00 00 00 64 |........6......d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|

2016-12-05T20:47:56.774Z|00026|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x31): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x31):

(***truncated to 64 bytes from 120***)

00000000 04 0e 00 78 00 00 00 31-00 00 00 00 00 00 00 00 |...x...1........|

00000010 00 00 00 00 00 00 00 00-32 00 00 00 00 00 00 64 |........2......d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|

2016-12-05T20:47:56.774Z|00027|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x39): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x39):

(***truncated to 64 bytes from 136***)

00000000 04 0e 00 88 00 00 00 39-00 00 00 00 00 00 00 00 |.......9........|

00000010 00 00 00 00 00 00 00 00-36 00 00 00 00 00 00 64 |........6......d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|

2016-12-05T20:47:56.774Z|00028|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x47): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x47):

(***truncated to 64 bytes from 160***)

00000000 04 0e 00 a0 00 00 00 47-00 00 00 00 00 00 00 00 |.......G........|

00000010 00 00 00 00 00 00 00 00-19 00 00 00 00 00 00 64 |...............d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|

2016-12-05T20:47:56.774Z|00029|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x49): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x49):

(***truncated to 64 bytes from 120***)

00000000 04 0e 00 78 00 00 00 49-00 00 00 00 00 00 00 00 |...x...I........|

00000010 00 00 00 00 00 00 00 00-15 00 00 00 00 00 00 64 |...............d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|

2016-12-05T20:47:56.774Z|00030|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x4a): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x4a):

(***truncated to 64 bytes from 160***)

00000000 04 0e 00 a0 00 00 00 4a-00 00 00 00 00 00 00 00 |.......J........|

00000010 00 00 00 00 00 00 00 00-36 00 00 00 00 00 00 64 |........6......d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|

2016-12-05T20:47:56.774Z|00031|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x53): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x53):

(***truncated to 64 bytes from 136***)

00000000 04 0e 00 88 00 00 00 53-00 00 00 00 00 00 00 00 |.......S........|

00000010 00 00 00 00 00 00 00 00-19 00 00 00 00 00 00 64 |...............d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|

2016-12-05T20:47:56.774Z|00032|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x61): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x61):

(***truncated to 64 bytes from 120***)

00000000 04 0e 00 78 00 00 00 61-00 00 00 00 00 00 00 00 |...x...a........|

00000010 00 00 00 00 00 00 00 00-15 00 00 00 00 00 00 64 |...............d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|

[ovirt-node2]

# lsmod | grep vport

vport_geneve 12815 1

geneve 13381 1 vport_geneve

openvswitch 84535 1 vport_geneve

[ovirt-node2 in the ovs-vswitchd.log]

What is of interest, is the "tunnel" error line, not sure where it's getting IP 50.48.48.48, not even an IP we use on this network, nor do i know where it's getting IP 47.51.48.55 ?

2016-12-05T20:35:04.499Z|00001|tunnel(revalidator25)|WARN|receive tunnel port not found (tun_id=0x3230203838373438,tun_src=47.51.48.55,tun_dst=50.48.48.48,tun_ipv6_src=::,tun_ipv6_dst=::,tun_gbp_id=0,tun_gbp_flags=0,tun_tos=20,tun_ttl=114,tun_flags=csum|key,vlan_tci=0x0000,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,dl_type=0x1234)

2016-12-05T20:35:04.345Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log

2016-12-05T20:35:04.347Z|00002|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 0

2016-12-05T20:35:04.347Z|00003|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 1

2016-12-05T20:35:04.347Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 32 CPU cores

2016-12-05T20:35:04.348Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...

2016-12-05T20:35:04.348Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected

2016-12-05T20:35:04.350Z|00007|ofproto_dpif|INFO|system@ovs-system: Datapath supports recirculation

2016-12-05T20:35:04.350Z|00008|ofproto_dpif|INFO|system@ovs-system: MPLS label stack length probed as 1

2016-12-05T20:35:04.350Z|00009|ofproto_dpif|INFO|system@ovs-system: Datapath does not support truncate action

2016-12-05T20:35:04.350Z|00010|ofproto_dpif|INFO|system@ovs-system: Datapath supports unique flow ids

2016-12-05T20:35:04.350Z|00011|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_state

2016-12-05T20:35:04.350Z|00012|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_zone

2016-12-05T20:35:04.350Z|00013|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_mark

2016-12-05T20:35:04.350Z|00014|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_label

2016-12-05T20:35:04.350Z|00015|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_state_nat

2016-12-05T20:35:04.493Z|00016|bridge|INFO|bridge br-int: added interface vnet0 on port 5

2016-12-05T20:35:04.493Z|00001|ofproto_dpif_upcall(handler1)|INFO|received packet on unassociated datapath port 0

2016-12-05T20:35:04.493Z|00017|bridge|INFO|bridge br-int: added interface br-int on port 65534

2016-12-05T20:35:04.493Z|00018|bridge|INFO|bridge br-int: using datapath ID 000016d6e0b66442

2016-12-05T20:35:04.493Z|00019|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt"

2016-12-05T20:35:04.494Z|00020|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.6.90

2016-12-05T20:35:09.657Z|00021|bridge|INFO|bridge br-int: added interface ovn-c0dc09-0 on port 6

2016-12-05T20:35:09.657Z|00022|bridge|INFO|bridge br-int: added interface ovn-252778-0 on port 7

2016-12-05T20:35:09.660Z|00023|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)

2016-12-05T20:35:09.660Z|00024|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T20:35:09.660Z|00025|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)

2016-12-05T20:35:09.660Z|00026|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T20:35:09.660Z|00027|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)

2016-12-05T20:35:09.660Z|00028|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T20:35:09.660Z|00029|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)

2016-12-05T20:35:09.660Z|00030|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T20:35:09.660Z|00031|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)

2016-12-05T20:35:09.660Z|00032|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T20:35:09.660Z|00033|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T20:35:09.660Z|00034|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T20:35:09.660Z|00035|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T20:35:09.660Z|00036|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T20:35:09.660Z|00037|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T20:35:14.495Z|00038|memory|INFO|281084 kB peak resident set size after 10.2 seconds

2016-12-05T20:35:14.495Z|00039|memory|INFO|handlers:23 ofconns:2 ports:4 revalidators:9 rules:76 udpif keys:1

2016-12-05T20:35:19.659Z|00040|connmgr|INFO|br-int<->unix: 73 flow_mods 10 s ago (72 adds, 1 deletes)

2016-12-05T20:37:43.716Z|00041|connmgr|INFO|br-int<->unix: 8 flow_mods 10 s ago (7 deletes, 1 modifications)

2016-12-05T20:39:02.163Z|00042|bridge|INFO|bridge br-int: added interface ovn-c0dc09-0 on port 8

2016-12-05T20:39:12.165Z|00043|connmgr|INFO|br-int<->unix: 8 flow_mods 10 s ago (7 adds, 1 modifications)

2016-12-05T20:45:34.604Z|00044|connmgr|INFO|br-int<->unix: 8 flow_mods 10 s ago (7 deletes, 1 modifications)

2016-12-05T20:47:56.774Z|00045|bridge|INFO|bridge br-int: added interface ovn-c0dc09-0 on port 9

2016-12-05T20:48:06.776Z|00046|connmgr|INFO|br-int<->unix: 8 flow_mods 10 s ago (7 adds, 1 modifications)

[ovirt-node3 / ovs-vswitchd.log]

2016-12-05T21:15:19.969Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log

2016-12-05T21:15:19.973Z|00002|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 0

2016-12-05T21:15:19.973Z|00003|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 1

2016-12-05T21:15:19.973Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 32 CPU cores

2016-12-05T21:15:19.973Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...

2016-12-05T21:15:19.973Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected

2016-12-05T21:15:19.978Z|00007|ofproto_dpif|INFO|system@ovs-system: Datapath supports recirculation

2016-12-05T21:15:19.978Z|00008|ofproto_dpif|INFO|system@ovs-system: MPLS label stack length probed as 1

2016-12-05T21:15:19.978Z|00009|ofproto_dpif|INFO|system@ovs-system: Datapath does not support truncate action

2016-12-05T21:15:19.978Z|00010|ofproto_dpif|INFO|system@ovs-system: Datapath supports unique flow ids

2016-12-05T21:15:19.978Z|00011|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_state

2016-12-05T21:15:19.978Z|00012|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_zone

2016-12-05T21:15:19.978Z|00013|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_mark

2016-12-05T21:15:19.978Z|00014|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_label

2016-12-05T21:15:19.978Z|00015|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_state_nat

2016-12-05T21:15:20.148Z|00001|ofproto_dpif_upcall(handler1)|INFO|received packet on unassociated datapath port 0

2016-12-05T21:15:20.148Z|00016|bridge|INFO|bridge br-int: added interface vnet0 on port 7

2016-12-05T21:15:20.148Z|00017|bridge|INFO|bridge br-int: added interface br-int on port 65534

2016-12-05T21:15:20.148Z|00018|bridge|INFO|bridge br-int: using datapath ID 0000921726222e4b

2016-12-05T21:15:20.148Z|00019|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt"

2016-12-05T21:15:20.150Z|00020|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.6.90

2016-12-05T21:15:30.150Z|00021|memory|INFO|281052 kB peak resident set size after 10.2 seconds

2016-12-05T21:15:30.150Z|00022|memory|INFO|handlers:23 ports:2 revalidators:9 rules:4

2016-12-05T21:15:47.900Z|00023|bridge|INFO|bridge br-int: added interface ovn-456949-0 on port 8

2016-12-05T21:15:47.900Z|00024|bridge|INFO|bridge br-int: added interface ovn-c0dc09-0 on port 9

2016-12-05T21:15:47.901Z|00025|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)

2016-12-05T21:15:47.901Z|00026|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T21:15:47.901Z|00027|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)

2016-12-05T21:15:47.901Z|00028|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T21:15:47.901Z|00029|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)

2016-12-05T21:15:47.901Z|00030|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T21:15:47.901Z|00031|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)

2016-12-05T21:15:47.901Z|00032|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T21:15:47.901Z|00033|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)

2016-12-05T21:15:47.901Z|00034|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T21:15:47.901Z|00035|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T21:15:47.901Z|00036|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T21:15:47.902Z|00037|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T21:15:47.902Z|00038|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T21:15:47.902Z|00039|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message

2016-12-05T21:15:57.901Z|00040|connmgr|INFO|br-int<->unix: 73 flow_mods 10 s ago (72 adds, 1 deletes)

[ovirt-node3 ovn-controller.log]

2016-12-05T21:15:47.858Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovn-controller.log

2016-12-05T21:15:47.860Z|00002|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...

2016-12-05T21:15:47.860Z|00003|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected

2016-12-05T21:15:47.864Z|00004|reconnect|INFO|tcp:172.20.192.77:6642: connecting...

2016-12-05T21:15:47.864Z|00005|reconnect|INFO|tcp:172.20.192.77:6642: connected

2016-12-05T21:15:47.867Z|00006|binding|INFO|Claiming lport 6b289418-8b8e-42b4-8334-c71584afcd3e for this chassis.

2016-12-05T21:15:47.867Z|00007|binding|INFO|Claiming 00:1a:4a:16:01:5c dynamic

2016-12-05T21:15:47.867Z|00008|ofctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch

2016-12-05T21:15:47.867Z|00009|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting...

2016-12-05T21:15:47.868Z|00010|pinctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch

2016-12-05T21:15:47.868Z|00011|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting...

2016-12-05T21:15:47.868Z|00012|ofctrl|INFO|dropping duplicate flow: table_id=29, priority=50, metadata=0x2,dl_dst=00:1a:4a:16:01:62, actions=set_field:0x5->reg15,resubmit(,32)

2016-12-05T21:15:47.869Z|00013|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected

2016-12-05T21:15:47.869Z|00014|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected

2016-12-05T21:15:47.870Z|00015|ofctrl|INFO|dropping duplicate flow: table_id=29, priority=50, metadata=0x2,dl_dst=00:1a:4a:16:01:62, actions=set_field:0x5->reg15,resubmit(,32)

2016-12-05T21:15:47.871Z|00016|ofctrl|INFO|dropping duplicate flow: table_id=29, priority=50, metadata=0x2,dl_dst=00:1a:4a:16:01:62, actions=set_field:0x5->reg15,resubmit(,32)

2016-12-05T21:15:47.872Z|00017|ofctrl|INFO|dropping duplicate flow: table_id=29, priority=50, metadata=0x2,dl_dst=00:1a:4a:16:01:62, actions=set_field:0x5->reg15,resubmit(,32)

2016-12-05T21:15:47.901Z|00018|ofctrl|INFO|dropping duplicate flow: table_id=29, priority=50, metadata=0x2,dl_dst=00:1a:4a:16:01:62, actions=set_field:0x5->reg15,resubmit(,32)

2016-12-05T21:15:47.901Z|00019|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x16): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x16):

(***truncated to 64 bytes from 120***)

00000000 04 0e 00 78 00 00 00 16-00 00 00 00 00 00 00 00 |...x............|

00000010 00 00 00 00 00 00 00 00-32 00 00 00 00 00 00 64 |........2......d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|

2016-12-05T21:15:47.902Z|00020|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x1c): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x1c):

(***truncated to 64 bytes from 160***)

00000000 04 0e 00 a0 00 00 00 1c-00 00 00 00 00 00 00 00 |................|

00000010 00 00 00 00 00 00 00 00-36 00 00 00 00 00 00 64 |........6......d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|

2016-12-05T21:15:47.902Z|00021|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x1e): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x1e):

(***truncated to 64 bytes from 136***)

00000000 04 0e 00 88 00 00 00 1e-00 00 00 00 00 00 00 00 |................|

00000010 00 00 00 00 00 00 00 00-19 00 00 00 00 00 00 64 |...............d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|

2016-12-05T21:15:47.902Z|00022|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x27): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x27):

(***truncated to 64 bytes from 160***)

00000000 04 0e 00 a0 00 00 00 27-00 00 00 00 00 00 00 00 |.......'........|

00000010 00 00 00 00 00 00 00 00-19 00 00 00 00 00 00 64 |...............d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|

2016-12-05T21:15:47.902Z|00023|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x28): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x28):

(***truncated to 64 bytes from 136***)

00000000 04 0e 00 88 00 00 00 28-00 00 00 00 00 00 00 00 |.......(........|

00000010 00 00 00 00 00 00 00 00-36 00 00 00 00 00 00 64 |........6......d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|

2016-12-05T21:15:47.902Z|00024|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x2b): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x2b):

(***truncated to 64 bytes from 120***)

00000000 04 0e 00 78 00 00 00 2b-00 00 00 00 00 00 00 00 |...x...+........|

00000010 00 00 00 00 00 00 00 00-32 00 00 00 00 00 00 64 |........2......d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|

2016-12-05T21:15:47.902Z|00025|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x31): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x31):

(***truncated to 64 bytes from 136***)

00000000 04 0e 00 88 00 00 00 31-00 00 00 00 00 00 00 00 |.......1........|

00000010 00 00 00 00 00 00 00 00-36 00 00 00 00 00 00 64 |........6......d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|

2016-12-05T21:15:47.902Z|00026|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x3d): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x3d):

(***truncated to 64 bytes from 160***)

00000000 04 0e 00 a0 00 00 00 3d-00 00 00 00 00 00 00 00 |.......=........|

00000010 00 00 00 00 00 00 00 00-19 00 00 00 00 00 00 64 |...............d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|

2016-12-05T21:15:47.902Z|00027|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x3f): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x3f):

(***truncated to 64 bytes from 120***)

00000000 04 0e 00 78 00 00 00 3f-00 00 00 00 00 00 00 00 |...x...?........|

00000010 00 00 00 00 00 00 00 00-15 00 00 00 00 00 00 64 |...............d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|

2016-12-05T21:15:47.902Z|00028|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x40): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x40):

(***truncated to 64 bytes from 160***)

00000000 04 0e 00 a0 00 00 00 40-00 00 00 00 00 00 00 00 |.......@........|

00000010 00 00 00 00 00 00 00 00-36 00 00 00 00 00 00 64 |........6......d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|

2016-12-05T21:15:47.902Z|00029|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x46): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x46):

(***truncated to 64 bytes from 136***)

00000000 04 0e 00 88 00 00 00 46-00 00 00 00 00 00 00 00 |.......F........|

00000010 00 00 00 00 00 00 00 00-19 00 00 00 00 00 00 64 |...............d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|

2016-12-05T21:15:47.902Z|00030|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x54): OFPBAC_BAD_TYPE

OFPT_FLOW_MOD (OF1.3) (xid=0x54):

(***truncated to 64 bytes from 120***)

00000000 04 0e 00 78 00 00 00 54-00 00 00 00 00 00 00 00 |...x...T........|

00000010 00 00 00 00 00 00 00 00-15 00 00 00 00 00 00 64 |...............d|

00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|

00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|

Your help is greatly appreciated!

Devin

On Mon, Dec 5, 2016 at 12:31 PM, Lance Richardson <lrichard@redhat.com> wrote:

> From: "Devin Acosta" <devin@pabstatencio.com>
> To: "Marcin Mirecki" <mmirecki@redhat.com>
> Cc: "users" <Users@ovirt.org>
> Sent: Monday, December 5, 2016 12:11:46 PM
> Subject: Re: [ovirt-users] oVIRT 4 / OVN / Communication issues of instances between nodes.

>
> Marcin,
>
> Also I noticed in your original post it mentions:
>
> ip link - the result should include a link called genev_sys_ ...
>
> I noticed that on my hosts I don't see any links with name: genev_sys_ ??
> Could this be a problem?
>
> lo:
> enp4s0f0:
> enp4s0f1:
> enp7s0f0:
> enp7s0f1:
> bond0:
> DEV-NOC:
> ovirtmgmt:
> bond0.700@bond0:
> DEV-VM-NET:
> bond0.705@bond0:
> ;vdsmdummy;:
> vnet0:
> vnet1:
> vnet2:
> vnet3:
> vnet4:
> ovs-system:
> br-int:
> vnet5:
> vnet6:
>

Hi Devin,

What distribution and kernel version are you using?

One thing you could check is whether the vport_geneve kernel module
is being loaded, e.g. you should see something like:

$ lsmod | grep vport
vport_geneve 12560 1
openvswitch 246755 5 vport_geneve

If vport_geneve is not loaded, you could "sudo modprobe vport_geneve"
to make sure it's available and can be loaded.

The first 100 lines or so of ovs-vswitchd.log might have some useful
information about where things are going wrong.

It does sound as though there is some issue with geneve tunnels,
which would certainly explain issues with inter-node traffic.

Regards,

Lance

Devin Acosta

Red Hat Certified Architect, LinuxStack

602-354-1220 || devin@linuxguru.co