Chassis "c0dc0909-4274-400d-826e-82348fee47e9" hostname: "host1" Encap geneve ip: "172.20.192.73" options: {csum="true"} Port_Binding "f409d17f-38bc-4ee9-922b-63dbfc46cd91" Port_Binding "29dce6d5-77bd-4a3c-b077-bb002c4347d6" Port_Binding "e4f23f60-cc16-4399-a15b-fb05c09c4409" Port_Binding "a6d2008f-bb73-437b-bbac-b50487ea5257" Port_Binding "ee971db2-cdaf-4fa0-b83c-993371533af9" Port_Binding "f5aed0a0-03e0-46ec-92c6-07e5f03e9cc9" Port_Binding "4f1cf249-7fb6-4ff2-a7a9-a98bf45af6b8" Port_Binding "49725a3e-ef3f-474d-b84a-4975ce55e8fb" Port_Binding "bd7c66e9-81d5-4bfe-9331-203908048a6f" Port_Binding "301bf657-b6c4-459f-bd88-d21885190651" Port_Binding "0745689c-2c70-4b48-bda2-d0c1f8127133" Chassis "252778b5-bc63-486d-8d82-c3e4350ee650" hostname: "host2" Encap geneve ip: "172.20.192.75" options: {csum="true"} Port_Binding "d2e19c41-f8ff-49d2-bd8e-01d833e59a59" Chassis "45694909-7882-4d7a-948a-bdf34e6472cb" hostname: "host3" Encap geneve ip: "172.20.192.74" options: {csum="true"} Port_Binding "70443ae2-a04b-4d1f-90e4-f12e7e48fe82" Port_Binding "fcb6df47-32e6-4f7c-96fd-b598c33bff97" I see on host #3 2016-12-20T21:35:01.280Z|173706735|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.283Z|173706736|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.286Z|173706737|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.288Z|173706738|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.291Z|173706739|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.294Z|173706740|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.297Z|173706741|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.299Z|173706742|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.302Z|173706743|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.305Z|173706744|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.308Z|173706745|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.310Z|173706746|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.313Z|173706747|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.316Z|173706748|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.319Z|173706749|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.321Z|173706750|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.324Z|173706751|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.327Z|173706752|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. on host #, running ovs-vsctl show 61af799c-a621-445e-8183-23dcb38ea3cc Bridge br-int fail_mode: secure Port "vnet0" Interface "vnet0" error: "could not open network device vnet0 (No such device)" Port "ovn-c0dc09-0" Interface "ovn-c0dc09-0" type: geneve options: {csum="true", key=flow, remote_ip="172.20.192.73"} Port "ovn-456949-0" Interface "ovn-456949-0" type: geneve options: {csum="true", key=flow, remote_ip="172.20.192.74"} Port br-int Interface br-int type: internal ovs_version: "2.6.90" So some port got stuck on a box? Any idea what would cause this and what i should do to remove it? On Tue, Dec 20, 2016 at 2:29 PM, Russell Bryant <rbryant@redhat.com> wrote:
On Tue, Dec 20, 2016 at 4:18 PM, Devin Acosta <devin@pabstatencio.com> wrote:
One interesting thing I noticed today after looking at the logs from the 'ovn-controller' on the oVIRT nodes, is node1 is logging like crazy, the file is over 1.2GB in size already, seems to be looping like crazy? What should i check to see why it is logging and maxing CPU like this?
2016-12-20T21:16:17.517Z|91605|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.521Z|91606|poll_loop|INFO|Dropped 5666 log messages in last 6 seconds (most recently, 0 seconds ago) due to excessive rate 2016-12-20T21:16:17.521Z|91607|poll_loop|INFO|wakeup due to [POLLIN] on fd 12 (172.20.192.73:54710<->172.20.192.77:6642) at lib/stream-fd.c:155 (94% CPU usage) 2016-12-20T21:16:17.521Z|91608|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9.
Check "ovn-sbctl show" to figure out which host is chassis 252778b5-bc63-486d-8d82-c3e4350ee650. See if the other host has similar entries in its log.
This could happen if a port is configured on two hosts at the same time. They will constantly fight over which chassis owns it.
-- Russell Bryant
-- Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 || devin@linuxguru.co