<div dir="ltr"><br><div>Lance,</div><div><br></div><div>I found some interesting logs, we have (3) oVIRT nodes.</div><div><div><br></div><div>We are running:</div><div>CentOS Linux release 7.2.1511 (Core)<br></div><div>Linux hostname 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux</div></div><div><br></div><div>What is interesting now is my 2nd node now won't hand out DHCP any more, however it does have a strange "tunnel" error message on node2 also, see below. What is bizarre node1 nor node3 show that bizarre tunnel message? </div><div><br></div><div><br></div><div>[ovirt-node1]</div><div><br></div><div><div>[root@ovirt-node1 ~]# lsmod | grep vport</div><div>vport_geneve 12815 1</div><div>geneve 13381 1 vport_geneve</div><div>openvswitch 84535 1 vport_geneve</div></div><div><br></div><div>[ovirt-node1 / ovn-controller.log]</div><div><br></div><div><div>2016-12-05T20:47:56.761Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovn-controller.log</div><div>2016-12-05T20:47:56.762Z|00002|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...</div><div>2016-12-05T20:47:56.762Z|00003|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected</div><div>2016-12-05T20:47:56.764Z|00004|reconnect|INFO|tcp:<a href="http://172.20.192.77:6642">172.20.192.77:6642</a>: connecting...</div><div>2016-12-05T20:47:56.764Z|00005|reconnect|INFO|tcp:<a href="http://172.20.192.77:6642">172.20.192.77:6642</a>: connected</div><div>2016-12-05T20:47:56.768Z|00006|binding|INFO|Claiming lport 91d4f4f5-4b9f-42c0-aa2c-8a101474bb84 for this chassis.</div><div>2016-12-05T20:47:56.768Z|00007|binding|INFO|Claiming 00:1a:4a:16:01:5e dynamic</div><div>2016-12-05T20:47:56.768Z|00008|binding|INFO|Claiming lport 71ef81f1-7c20-4c68-b536-d274703f7541 for this chassis.</div><div>2016-12-05T20:47:56.768Z|00009|binding|INFO|Claiming 00:1a:4a:16:01:61 dynamic</div><div>2016-12-05T20:47:56.768Z|00010|ofctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch</div><div>2016-12-05T20:47:56.768Z|00011|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting...</div><div>2016-12-05T20:47:56.768Z|00012|pinctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch</div><div>2016-12-05T20:47:56.768Z|00013|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting...</div><div>2016-12-05T20:47:56.768Z|00014|ofctrl|INFO|dropping duplicate flow: table_id=29, priority=50, metadata=0x2,dl_dst=00:1a:4a:16:01:62, actions=set_field:0x5->reg15,resubmit(,32)</div><div>2016-12-05T20:47:56.770Z|00015|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected</div><div>2016-12-05T20:47:56.770Z|00016|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected</div><div>2016-12-05T20:47:56.770Z|00017|ofctrl|INFO|dropping duplicate flow: table_id=29, priority=50, metadata=0x2,dl_dst=00:1a:4a:16:01:62, actions=set_field:0x5->reg15,resubmit(,32)</div><div>2016-12-05T20:47:56.771Z|00018|ofctrl|INFO|dropping duplicate flow: table_id=29, priority=50, metadata=0x2,dl_dst=00:1a:4a:16:01:62, actions=set_field:0x5->reg15,resubmit(,32)</div><div>2016-12-05T20:47:56.772Z|00019|ofctrl|INFO|dropping duplicate flow: table_id=29, priority=50, metadata=0x2,dl_dst=00:1a:4a:16:01:62, actions=set_field:0x5->reg15,resubmit(,32)</div><div>2016-12-05T20:47:56.773Z|00020|ofctrl|INFO|dropping duplicate flow: table_id=32, priority=150, reg10=0x2/0x2, actions=resubmit(,33)</div><div>2016-12-05T20:47:56.774Z|00021|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x17): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x17):</div><div>(***truncated to 64 bytes from 120***)</div><div>00000000 04 0e 00 78 00 00 00 17-00 00 00 00 00 00 00 00 |...x............|</div><div>00000010 00 00 00 00 00 00 00 00-32 00 00 00 00 00 00 64 |........2......d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T20:47:56.774Z|00022|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x1f): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x1f):</div><div>(***truncated to 64 bytes from 160***)</div><div>00000000 04 0e 00 a0 00 00 00 1f-00 00 00 00 00 00 00 00 |................|</div><div>00000010 00 00 00 00 00 00 00 00-36 00 00 00 00 00 00 64 |........6......d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T20:47:56.774Z|00023|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x21): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x21):</div><div>(***truncated to 64 bytes from 136***)</div><div>00000000 04 0e 00 88 00 00 00 21-00 00 00 00 00 00 00 00 |.......!........|</div><div>00000010 00 00 00 00 00 00 00 00-19 00 00 00 00 00 00 64 |...............d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T20:47:56.774Z|00024|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x2c): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x2c):</div><div>(***truncated to 64 bytes from 160***)</div><div>00000000 04 0e 00 a0 00 00 00 2c-00 00 00 00 00 00 00 00 |.......,........|</div><div>00000010 00 00 00 00 00 00 00 00-19 00 00 00 00 00 00 64 |...............d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T20:47:56.774Z|00025|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x2d): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x2d):</div><div>(***truncated to 64 bytes from 136***)</div><div>00000000 04 0e 00 88 00 00 00 2d-00 00 00 00 00 00 00 00 |.......-........|</div><div>00000010 00 00 00 00 00 00 00 00-36 00 00 00 00 00 00 64 |........6......d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T20:47:56.774Z|00026|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x31): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x31):</div><div>(***truncated to 64 bytes from 120***)</div><div>00000000 04 0e 00 78 00 00 00 31-00 00 00 00 00 00 00 00 |...x...1........|</div><div>00000010 00 00 00 00 00 00 00 00-32 00 00 00 00 00 00 64 |........2......d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T20:47:56.774Z|00027|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x39): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x39):</div><div>(***truncated to 64 bytes from 136***)</div><div>00000000 04 0e 00 88 00 00 00 39-00 00 00 00 00 00 00 00 |.......9........|</div><div>00000010 00 00 00 00 00 00 00 00-36 00 00 00 00 00 00 64 |........6......d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T20:47:56.774Z|00028|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x47): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x47):</div><div>(***truncated to 64 bytes from 160***)</div><div>00000000 04 0e 00 a0 00 00 00 47-00 00 00 00 00 00 00 00 |.......G........|</div><div>00000010 00 00 00 00 00 00 00 00-19 00 00 00 00 00 00 64 |...............d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T20:47:56.774Z|00029|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x49): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x49):</div><div>(***truncated to 64 bytes from 120***)</div><div>00000000 04 0e 00 78 00 00 00 49-00 00 00 00 00 00 00 00 |...x...I........|</div><div>00000010 00 00 00 00 00 00 00 00-15 00 00 00 00 00 00 64 |...............d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T20:47:56.774Z|00030|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x4a): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x4a):</div><div>(***truncated to 64 bytes from 160***)</div><div>00000000 04 0e 00 a0 00 00 00 4a-00 00 00 00 00 00 00 00 |.......J........|</div><div>00000010 00 00 00 00 00 00 00 00-36 00 00 00 00 00 00 64 |........6......d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T20:47:56.774Z|00031|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x53): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x53):</div><div>(***truncated to 64 bytes from 136***)</div><div>00000000 04 0e 00 88 00 00 00 53-00 00 00 00 00 00 00 00 |.......S........|</div><div>00000010 00 00 00 00 00 00 00 00-19 00 00 00 00 00 00 64 |...............d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T20:47:56.774Z|00032|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x61): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x61):</div><div>(***truncated to 64 bytes from 120***)</div><div>00000000 04 0e 00 78 00 00 00 61-00 00 00 00 00 00 00 00 |...x...a........|</div><div>00000010 00 00 00 00 00 00 00 00-15 00 00 00 00 00 00 64 |...............d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|</div></div><div><br></div><div>[ovirt-node2]</div><div># lsmod | grep vport<br></div><div><div>vport_geneve 12815 1</div><div>geneve 13381 1 vport_geneve</div><div>openvswitch 84535 1 vport_geneve</div></div><div><br></div><div>[ovirt-node2 in the ovs-vswitchd.log]</div><div><br></div><div>What is of interest, is the "tunnel" error line, not sure where it's getting IP 50.48.48.48, not even an IP we use on this network, nor do i know where it's getting IP 47.51.48.55 ?</div><div><br></div><div>2016-12-05T20:35:04.499Z|00001|tunnel(revalidator25)|WARN|receive tunnel port not found (tun_id=0x3230203838373438,tun_src=47.51.48.55,tun_dst=50.48.48.48,tun_ipv6_src=::,tun_ipv6_dst=::,tun_gbp_id=0,tun_gbp_flags=0,tun_tos=20,tun_ttl=114,tun_flags=csum|key,vlan_tci=0x0000,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,dl_type=0x1234)<br></div><div><br></div><div><br></div><div><div>2016-12-05T20:35:04.345Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log</div><div>2016-12-05T20:35:04.347Z|00002|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 0</div><div>2016-12-05T20:35:04.347Z|00003|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 1</div><div>2016-12-05T20:35:04.347Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 32 CPU cores</div><div>2016-12-05T20:35:04.348Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...</div><div>2016-12-05T20:35:04.348Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected</div><div>2016-12-05T20:35:04.350Z|00007|ofproto_dpif|INFO|system@ovs-system: Datapath supports recirculation</div><div>2016-12-05T20:35:04.350Z|00008|ofproto_dpif|INFO|system@ovs-system: MPLS label stack length probed as 1</div><div>2016-12-05T20:35:04.350Z|00009|ofproto_dpif|INFO|system@ovs-system: Datapath does not support truncate action</div><div>2016-12-05T20:35:04.350Z|00010|ofproto_dpif|INFO|system@ovs-system: Datapath supports unique flow ids</div><div>2016-12-05T20:35:04.350Z|00011|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_state</div><div>2016-12-05T20:35:04.350Z|00012|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_zone</div><div>2016-12-05T20:35:04.350Z|00013|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_mark</div><div>2016-12-05T20:35:04.350Z|00014|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_label</div><div>2016-12-05T20:35:04.350Z|00015|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_state_nat</div><div>2016-12-05T20:35:04.493Z|00016|bridge|INFO|bridge br-int: added interface vnet0 on port 5</div><div>2016-12-05T20:35:04.493Z|00001|ofproto_dpif_upcall(handler1)|INFO|received packet on unassociated datapath port 0</div><div>2016-12-05T20:35:04.493Z|00017|bridge|INFO|bridge br-int: added interface br-int on port 65534</div><div>2016-12-05T20:35:04.493Z|00018|bridge|INFO|bridge br-int: using datapath ID 000016d6e0b66442</div><div>2016-12-05T20:35:04.493Z|00019|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt"</div><div>2016-12-05T20:35:04.494Z|00020|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.6.90</div><div>2016-12-05T20:35:04.499Z|00001|tunnel(revalidator25)|WARN|receive tunnel port not found (tun_id=0x3230203838373438,tun_src=47.51.48.55,tun_dst=50.48.48.48,tun_ipv6_src=::,tun_ipv6_dst=::,tun_gbp_id=0,tun_gbp_flags=0,tun_tos=20,tun_ttl=114,tun_flags=csum|key,vlan_tci=0x0000,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,dl_type=0x1234)</div><div>2016-12-05T20:35:09.657Z|00021|bridge|INFO|bridge br-int: added interface ovn-c0dc09-0 on port 6</div><div>2016-12-05T20:35:09.657Z|00022|bridge|INFO|bridge br-int: added interface ovn-252778-0 on port 7</div><div>2016-12-05T20:35:09.660Z|00023|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)</div><div>2016-12-05T20:35:09.660Z|00024|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T20:35:09.660Z|00025|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)</div><div>2016-12-05T20:35:09.660Z|00026|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T20:35:09.660Z|00027|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)</div><div>2016-12-05T20:35:09.660Z|00028|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T20:35:09.660Z|00029|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)</div><div>2016-12-05T20:35:09.660Z|00030|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T20:35:09.660Z|00031|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)</div><div>2016-12-05T20:35:09.660Z|00032|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T20:35:09.660Z|00033|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T20:35:09.660Z|00034|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T20:35:09.660Z|00035|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T20:35:09.660Z|00036|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T20:35:09.660Z|00037|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T20:35:14.495Z|00038|memory|INFO|281084 kB peak resident set size after 10.2 seconds</div><div>2016-12-05T20:35:14.495Z|00039|memory|INFO|handlers:23 ofconns:2 ports:4 revalidators:9 rules:76 udpif keys:1</div><div>2016-12-05T20:35:19.659Z|00040|connmgr|INFO|br-int<->unix: 73 flow_mods 10 s ago (72 adds, 1 deletes)</div><div>2016-12-05T20:37:43.716Z|00041|connmgr|INFO|br-int<->unix: 8 flow_mods 10 s ago (7 deletes, 1 modifications)</div><div>2016-12-05T20:39:02.163Z|00042|bridge|INFO|bridge br-int: added interface ovn-c0dc09-0 on port 8</div><div>2016-12-05T20:39:12.165Z|00043|connmgr|INFO|br-int<->unix: 8 flow_mods 10 s ago (7 adds, 1 modifications)</div><div>2016-12-05T20:45:34.604Z|00044|connmgr|INFO|br-int<->unix: 8 flow_mods 10 s ago (7 deletes, 1 modifications)</div><div>2016-12-05T20:47:56.774Z|00045|bridge|INFO|bridge br-int: added interface ovn-c0dc09-0 on port 9</div><div>2016-12-05T20:48:06.776Z|00046|connmgr|INFO|br-int<->unix: 8 flow_mods 10 s ago (7 adds, 1 modifications)</div></div><div><br></div><div>[ovirt-node3 / ovs-vswitchd.log]</div><div><br></div><div><div>2016-12-05T21:15:19.969Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log</div><div>2016-12-05T21:15:19.973Z|00002|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 0</div><div>2016-12-05T21:15:19.973Z|00003|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 1</div><div>2016-12-05T21:15:19.973Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 32 CPU cores</div><div>2016-12-05T21:15:19.973Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...</div><div>2016-12-05T21:15:19.973Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected</div><div>2016-12-05T21:15:19.978Z|00007|ofproto_dpif|INFO|system@ovs-system: Datapath supports recirculation</div><div>2016-12-05T21:15:19.978Z|00008|ofproto_dpif|INFO|system@ovs-system: MPLS label stack length probed as 1</div><div>2016-12-05T21:15:19.978Z|00009|ofproto_dpif|INFO|system@ovs-system: Datapath does not support truncate action</div><div>2016-12-05T21:15:19.978Z|00010|ofproto_dpif|INFO|system@ovs-system: Datapath supports unique flow ids</div><div>2016-12-05T21:15:19.978Z|00011|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_state</div><div>2016-12-05T21:15:19.978Z|00012|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_zone</div><div>2016-12-05T21:15:19.978Z|00013|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_mark</div><div>2016-12-05T21:15:19.978Z|00014|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_label</div><div>2016-12-05T21:15:19.978Z|00015|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_state_nat</div><div>2016-12-05T21:15:20.148Z|00001|ofproto_dpif_upcall(handler1)|INFO|received packet on unassociated datapath port 0</div><div>2016-12-05T21:15:20.148Z|00016|bridge|INFO|bridge br-int: added interface vnet0 on port 7</div><div>2016-12-05T21:15:20.148Z|00017|bridge|INFO|bridge br-int: added interface br-int on port 65534</div><div>2016-12-05T21:15:20.148Z|00018|bridge|INFO|bridge br-int: using datapath ID 0000921726222e4b</div><div>2016-12-05T21:15:20.148Z|00019|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt"</div><div>2016-12-05T21:15:20.150Z|00020|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.6.90</div><div>2016-12-05T21:15:30.150Z|00021|memory|INFO|281052 kB peak resident set size after 10.2 seconds</div><div>2016-12-05T21:15:30.150Z|00022|memory|INFO|handlers:23 ports:2 revalidators:9 rules:4</div><div>2016-12-05T21:15:47.900Z|00023|bridge|INFO|bridge br-int: added interface ovn-456949-0 on port 8</div><div>2016-12-05T21:15:47.900Z|00024|bridge|INFO|bridge br-int: added interface ovn-c0dc09-0 on port 9</div><div>2016-12-05T21:15:47.901Z|00025|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)</div><div>2016-12-05T21:15:47.901Z|00026|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T21:15:47.901Z|00027|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)</div><div>2016-12-05T21:15:47.901Z|00028|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T21:15:47.901Z|00029|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)</div><div>2016-12-05T21:15:47.901Z|00030|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T21:15:47.901Z|00031|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)</div><div>2016-12-05T21:15:47.901Z|00032|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T21:15:47.901Z|00033|ofproto_dpif|WARN|Rejecting ct action because datapath does not support ct action (your kernel module may be out of date)</div><div>2016-12-05T21:15:47.901Z|00034|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T21:15:47.901Z|00035|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T21:15:47.901Z|00036|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T21:15:47.902Z|00037|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T21:15:47.902Z|00038|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T21:15:47.902Z|00039|connmgr|INFO|br-int<->unix: sending OFPBAC_BAD_TYPE error reply to OFPT_FLOW_MOD message</div><div>2016-12-05T21:15:57.901Z|00040|connmgr|INFO|br-int<->unix: 73 flow_mods 10 s ago (72 adds, 1 deletes)</div></div><div><br></div><div>[ovirt-node3 ovn-controller.log]</div><div><br></div><div><div>2016-12-05T21:15:47.858Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovn-controller.log</div><div>2016-12-05T21:15:47.860Z|00002|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...</div><div>2016-12-05T21:15:47.860Z|00003|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected</div><div>2016-12-05T21:15:47.864Z|00004|reconnect|INFO|tcp:<a href="http://172.20.192.77:6642">172.20.192.77:6642</a>: connecting...</div><div>2016-12-05T21:15:47.864Z|00005|reconnect|INFO|tcp:<a href="http://172.20.192.77:6642">172.20.192.77:6642</a>: connected</div><div>2016-12-05T21:15:47.867Z|00006|binding|INFO|Claiming lport 6b289418-8b8e-42b4-8334-c71584afcd3e for this chassis.</div><div>2016-12-05T21:15:47.867Z|00007|binding|INFO|Claiming 00:1a:4a:16:01:5c dynamic</div><div>2016-12-05T21:15:47.867Z|00008|ofctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch</div><div>2016-12-05T21:15:47.867Z|00009|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting...</div><div>2016-12-05T21:15:47.868Z|00010|pinctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch</div><div>2016-12-05T21:15:47.868Z|00011|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting...</div><div>2016-12-05T21:15:47.868Z|00012|ofctrl|INFO|dropping duplicate flow: table_id=29, priority=50, metadata=0x2,dl_dst=00:1a:4a:16:01:62, actions=set_field:0x5->reg15,resubmit(,32)</div><div>2016-12-05T21:15:47.869Z|00013|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected</div><div>2016-12-05T21:15:47.869Z|00014|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected</div><div>2016-12-05T21:15:47.870Z|00015|ofctrl|INFO|dropping duplicate flow: table_id=29, priority=50, metadata=0x2,dl_dst=00:1a:4a:16:01:62, actions=set_field:0x5->reg15,resubmit(,32)</div><div>2016-12-05T21:15:47.871Z|00016|ofctrl|INFO|dropping duplicate flow: table_id=29, priority=50, metadata=0x2,dl_dst=00:1a:4a:16:01:62, actions=set_field:0x5->reg15,resubmit(,32)</div><div>2016-12-05T21:15:47.872Z|00017|ofctrl|INFO|dropping duplicate flow: table_id=29, priority=50, metadata=0x2,dl_dst=00:1a:4a:16:01:62, actions=set_field:0x5->reg15,resubmit(,32)</div><div>2016-12-05T21:15:47.901Z|00018|ofctrl|INFO|dropping duplicate flow: table_id=29, priority=50, metadata=0x2,dl_dst=00:1a:4a:16:01:62, actions=set_field:0x5->reg15,resubmit(,32)</div><div>2016-12-05T21:15:47.901Z|00019|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x16): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x16):</div><div>(***truncated to 64 bytes from 120***)</div><div>00000000 04 0e 00 78 00 00 00 16-00 00 00 00 00 00 00 00 |...x............|</div><div>00000010 00 00 00 00 00 00 00 00-32 00 00 00 00 00 00 64 |........2......d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T21:15:47.902Z|00020|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x1c): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x1c):</div><div>(***truncated to 64 bytes from 160***)</div><div>00000000 04 0e 00 a0 00 00 00 1c-00 00 00 00 00 00 00 00 |................|</div><div>00000010 00 00 00 00 00 00 00 00-36 00 00 00 00 00 00 64 |........6......d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T21:15:47.902Z|00021|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x1e): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x1e):</div><div>(***truncated to 64 bytes from 136***)</div><div>00000000 04 0e 00 88 00 00 00 1e-00 00 00 00 00 00 00 00 |................|</div><div>00000010 00 00 00 00 00 00 00 00-19 00 00 00 00 00 00 64 |...............d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T21:15:47.902Z|00022|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x27): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x27):</div><div>(***truncated to 64 bytes from 160***)</div><div>00000000 04 0e 00 a0 00 00 00 27-00 00 00 00 00 00 00 00 |.......'........|</div><div>00000010 00 00 00 00 00 00 00 00-19 00 00 00 00 00 00 64 |...............d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T21:15:47.902Z|00023|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x28): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x28):</div><div>(***truncated to 64 bytes from 136***)</div><div>00000000 04 0e 00 88 00 00 00 28-00 00 00 00 00 00 00 00 |.......(........|</div><div>00000010 00 00 00 00 00 00 00 00-36 00 00 00 00 00 00 64 |........6......d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T21:15:47.902Z|00024|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x2b): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x2b):</div><div>(***truncated to 64 bytes from 120***)</div><div>00000000 04 0e 00 78 00 00 00 2b-00 00 00 00 00 00 00 00 |...x...+........|</div><div>00000010 00 00 00 00 00 00 00 00-32 00 00 00 00 00 00 64 |........2......d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T21:15:47.902Z|00025|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x31): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x31):</div><div>(***truncated to 64 bytes from 136***)</div><div>00000000 04 0e 00 88 00 00 00 31-00 00 00 00 00 00 00 00 |.......1........|</div><div>00000010 00 00 00 00 00 00 00 00-36 00 00 00 00 00 00 64 |........6......d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T21:15:47.902Z|00026|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x3d): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x3d):</div><div>(***truncated to 64 bytes from 160***)</div><div>00000000 04 0e 00 a0 00 00 00 3d-00 00 00 00 00 00 00 00 |.......=........|</div><div>00000010 00 00 00 00 00 00 00 00-19 00 00 00 00 00 00 64 |...............d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T21:15:47.902Z|00027|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x3f): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x3f):</div><div>(***truncated to 64 bytes from 120***)</div><div>00000000 04 0e 00 78 00 00 00 3f-00 00 00 00 00 00 00 00 |...x...?........|</div><div>00000010 00 00 00 00 00 00 00 00-15 00 00 00 00 00 00 64 |...............d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T21:15:47.902Z|00028|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x40): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x40):</div><div>(***truncated to 64 bytes from 160***)</div><div>00000000 04 0e 00 a0 00 00 00 40-00 00 00 00 00 00 00 00 |.......@........|</div><div>00000010 00 00 00 00 00 00 00 00-36 00 00 00 00 00 00 64 |........6......d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T21:15:47.902Z|00029|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x46): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x46):</div><div>(***truncated to 64 bytes from 136***)</div><div>00000000 04 0e 00 88 00 00 00 46-00 00 00 00 00 00 00 00 |.......F........|</div><div>00000010 00 00 00 00 00 00 00 00-19 00 00 00 00 00 00 64 |...............d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-08 00 00 01 01 08 00 00 |..."............|</div><div>2016-12-05T21:15:47.902Z|00030|ofctrl|INFO|OpenFlow error: OFPT_ERROR (OF1.3) (xid=0x54): OFPBAC_BAD_TYPE</div><div>OFPT_FLOW_MOD (OF1.3) (xid=0x54):</div><div>(***truncated to 64 bytes from 120***)</div><div>00000000 04 0e 00 78 00 00 00 54-00 00 00 00 00 00 00 00 |...x...T........|</div><div>00000010 00 00 00 00 00 00 00 00-15 00 00 00 00 00 00 64 |...............d|</div><div>00000020 ff ff ff ff ff ff ff ff-ff ff ff ff 00 00 00 00 |................|</div><div>00000030 00 01 00 22 80 00 0a 02-86 dd 00 01 01 08 00 00 |..."............|</div></div><div><br></div><div>Your help is greatly appreciated!</div><div><br></div><div>Devin</div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 5, 2016 at 12:31 PM, Lance Richardson <span dir="ltr"><<a href="mailto:lrichard@redhat.com" target="_blank">lrichard@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> From: "Devin Acosta" <<a href="mailto:devin@pabstatencio.com">devin@pabstatencio.com</a>><br>
> To: "Marcin Mirecki" <<a href="mailto:mmirecki@redhat.com">mmirecki@redhat.com</a>><br>
> Cc: "users" <<a href="mailto:Users@ovirt.org">Users@ovirt.org</a>><br>
> Sent: Monday, December 5, 2016 12:11:46 PM<br>
> Subject: Re: [ovirt-users] oVIRT 4 / OVN / Communication issues of instances between nodes.<br>
<div><div class="gmail-h5">><br>
> Marcin,<br>
><br>
> Also I noticed in your original post it mentions:<br>
><br>
> ip link - the result should include a link called genev_sys_ ...<br>
><br>
> I noticed that on my hosts I don't see any links with name: genev_sys_ ??<br>
> Could this be a problem?<br>
><br>
> lo:<br>
> enp4s0f0:<br>
> enp4s0f1:<br>
> enp7s0f0:<br>
> enp7s0f1:<br>
> bond0:<br>
> DEV-NOC:<br>
> ovirtmgmt:<br>
> bond0.700@bond0:<br>
> DEV-VM-NET:<br>
> bond0.705@bond0:<br>
> ;vdsmdummy;:<br>
> vnet0:<br>
> vnet1:<br>
> vnet2:<br>
> vnet3:<br>
> vnet4:<br>
> ovs-system:<br>
> br-int:<br>
> vnet5:<br>
> vnet6:<br>
><br>
<br>
</div></div>Hi Devin,<br>
<br>
What distribution and kernel version are you using?<br>
<br>
One thing you could check is whether the vport_geneve kernel module<br>
is being loaded, e.g. you should see something like:<br>
<br>
$ lsmod | grep vport<br>
vport_geneve 12560 1<br>
openvswitch 246755 5 vport_geneve<br>
<br>
If vport_geneve is not loaded, you could "sudo modprobe vport_geneve"<br>
to make sure it's available and can be loaded.<br>
<br>
The first 100 lines or so of ovs-vswitchd.log might have some useful<br>
information about where things are going wrong.<br>
<br>
It does sound as though there is some issue with geneve tunnels,<br>
which would certainly explain issues with inter-node traffic.<br>
<br>
Regards,<br>
<br>
Lance<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><br></div><div>Devin Acosta</div><div>Red Hat Certified Architect, LinuxStack </div><div>602-354-1220 || <a href="mailto:devin@linuxguru.co" target="_blank">devin@linuxguru.co</a></div></div></div>
</div></div>