[ovirt-users] Re: Problems with OVN

Hi Marcin, Here is ip addr output from virtual machines: [root@testi2 ~]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:1a:4a:16:01:05 brd ff:ff:ff:ff:ff:ff inet 10.0.1.25/24 brd 10.0.1.255 scope global dynamic eth0 valid_lft 86331sec preferred_lft 86331sec inet6 fe80::21a:4aff:fe16:105/64 scope link valid_lft forever preferred_lft forever 3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:1a:4a:16:01:03 brd ff:ff:ff:ff:ff:ff inet 10.0.200.10/24 brd 10.0.200.255 scope global dynamic eth2 valid_lft 86334sec preferred_lft 86334sec inet6 fe80::21a:4aff:fe16:103/64 scope link valid_lft forever preferred_lft forever eth0 connected to network ovirtmgmt eth2 connected to OVN network vm-public [root@testi6 ~]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:1a:4a:16:01:0b brd ff:ff:ff:ff:ff:ff inet 10.0.1.27/24 brd 10.0.1.255 scope global dynamic eth0 valid_lft 86187sec preferred_lft 86187sec inet6 fe80::21a:4aff:fe16:10b/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:1a:4a:16:01:0c brd ff:ff:ff:ff:ff:ff inet 10.0.200.11/24 brd 10.0.200.255 scope global dynamic eth1 valid_lft 86301sec preferred_lft 86301sec inet6 fe80::21a:4aff:fe16:10c/64 scope link valid_lft forever preferred_lft forever eth0 connected to network ovirtmgmt eth1 connected to OVN network vm-public Best regards, Samuli Marcin Mirecki kirjoitti 08.05.2018 10:14: > Hi Samuli, > > Your configuration looks correct. > Can you also send me the result of 'ip addr' on your vm's? > > Thanks, > Marcin > > On Mon, May 7, 2018 at 7:44 PM, Samuli Heinonen > <samppah(a)neutraali.net&gt; wrote: > > Hi Marcin, >> >> Thank you for your response. >> >> I used engine-setup to do the configuration. Only exception is that >> I had to run "vdsm-tool ovn-config engine-ip local-ip" (ie. >> vdsm-tool ovn-config 10.0.1.101 10.0.1.21) on hypervisors. >> >> Here is the output of requested commands: >> >> [root@oe ~]# ovn-sbctl show >> Chassis "049183d5-61b6-4b9c-bae3-c7b10d30f8cb" >> hostname: "o2.hirundinidae.local" >> Encap geneve >> ip: "10.0.1.18" >> options: {csum="true"} >> Port_Binding "87c5e44a-7c8b-41b2-89a6-fa52f27643ed" >> Chassis "972f1b7b-10de-4e4f-a5f9-f080890f087d" >> hostname: "o3.hirundinidae.local" >> Encap geneve >> ip: "10.0.1.21" >> options: {csum="true"} >> Port_Binding "ccea5185-3efa-4d9c-9475-9e46009fea4f" >> Port_Binding "e868219c-f16c-45c6-b7b1-72d044fee602" >> >> [root@oe ~]# ovn-nbctl show >> switch 7d264a6c-ea48-4a6d-9663-5244102dc9bb (vm-private) >> port 4ec3ecf6-d04a-406c-8354-c5e195ffde05 >> addresses: ["00:1a:4a:16:01:06 dynamic"] >> switch 40aedb7d-b1c3-400e-9ddb-16bee3bb312a (vm-public) >> port 87c5e44a-7c8b-41b2-89a6-fa52f27643ed >> addresses: ["00:1a:4a:16:01:03"] >> port ccea5185-3efa-4d9c-9475-9e46009fea4f >> addresses: ["00:1a:4a:16:01:0c"] >> port e868219c-f16c-45c6-b7b1-72d044fee602 >> addresses: ["00:1a:4a:16:01:0a"] >> >> [root@o2 ~]# ip addr >> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN >> qlen 1 >> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 >> inet 127.0.0.1/8 [1] scope host lo >> valid_lft forever preferred_lft forever >> inet6 ::1/128 scope host >> valid_lft forever preferred_lft forever >> 2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc >> pfifo_fast master ovirtmgmt state UP qlen 1000 >> link/ether 78:f2:9e:90:bc:64 brd ff:ff:ff:ff:ff:ff >> 3: enp0s20f0u5c2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc >> pfifo_fast master public state UNKNOWN qlen 1000 >> link/ether 50:3e:aa:4c:9b:01 brd ff:ff:ff:ff:ff:ff >> 4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >> qlen 1000 >> link/ether 82:49:e1:15:af:56 brd ff:ff:ff:ff:ff:ff >> 5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen >> 1000 >> link/ether a2:bb:78:7e:35:4b brd ff:ff:ff:ff:ff:ff >> 21: public: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue >> state UP qlen 1000 >> link/ether 50:3e:aa:4c:9b:01 brd ff:ff:ff:ff:ff:ff >> inet6 fe80::523e:aaff:fe4c:9b01/64 scope link >> valid_lft forever preferred_lft forever >> 22: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc >> noqueue state UP qlen 1000 >> link/ether 78:f2:9e:90:bc:64 brd ff:ff:ff:ff:ff:ff >> inet 10.0.1.18/24 [2] brd 10.0.1.255 scope global ovirtmgmt >> >> valid_lft forever preferred_lft forever >> inet6 fe80::7af2:9eff:fe90:bc64/64 scope link >> valid_lft forever preferred_lft forever >> 23: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 >> qdisc noqueue master ovs-system state UNKNOWN qlen 1000 >> link/ether 02:c0:7a:e3:4e:76 brd ff:ff:ff:ff:ff:ff >> inet6 fe80::c0:7aff:fee3:4e76/64 scope link >> valid_lft forever preferred_lft forever >> 24: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state >> DOWN qlen 1000 >> link/ether a2:2f:f2:58:88:da brd ff:ff:ff:ff:ff:ff >> 26: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc >> pfifo_fast master ovs-system state UNKNOWN qlen 1000 >> link/ether fe:1a:4a:16:01:03 brd ff:ff:ff:ff:ff:ff >> inet6 fe80::fc1a:4aff:fe16:103/64 scope link >> valid_lft forever preferred_lft forever >> 29: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc >> pfifo_fast master ovirtmgmt state UNKNOWN qlen 1000 >> link/ether fe:1a:4a:16:01:05 brd ff:ff:ff:ff:ff:ff >> inet6 fe80::fc1a:4aff:fe16:105/64 scope link >> valid_lft forever preferred_lft forever >> >> [root@o2 ~]# ovs-vsctl show >> 6be6d37c-74cf-485e-9957-f8eb4bddb2ca >> Bridge br-int >> fail_mode: secure >> Port br-int >> Interface br-int >> type: internal >> Port "ovn-972f1b-0" >> Interface "ovn-972f1b-0" >> type: geneve >> options: {csum="true", key=flow, >> remote_ip="10.0.1.21"} >> Port "vnet0" >> Interface "vnet0" >> ovs_version: "2.9.0" >> >> [root@o3 ~]# ip addr >> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN >> qlen 1 >> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 >> inet 127.0.0.1/8 [1] scope host lo >> valid_lft forever preferred_lft forever >> inet6 ::1/128 scope host >> valid_lft forever preferred_lft forever >> 2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc >> pfifo_fast master ovirtmgmt state UP qlen 1000 >> link/ether 78:f2:9e:90:bc:50 brd ff:ff:ff:ff:ff:ff >> 3: enp0s20f0u5c2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc >> pfifo_fast master public state UNKNOWN qlen 1000 >> link/ether 50:3e:aa:4c:9c:03 brd ff:ff:ff:ff:ff:ff >> 4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >> qlen 1000 >> link/ether 7e:43:c1:b0:48:73 brd ff:ff:ff:ff:ff:ff >> 5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen >> 1000 >> link/ether 3a:fe:68:34:31:4c brd ff:ff:ff:ff:ff:ff >> 21: public: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue >> state UP qlen 1000 >> link/ether 50:3e:aa:4c:9c:03 brd ff:ff:ff:ff:ff:ff >> inet6 fe80::523e:aaff:fe4c:9c03/64 scope link >> valid_lft forever preferred_lft forever >> 22: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc >> noqueue state UP qlen 1000 >> link/ether 78:f2:9e:90:bc:50 brd ff:ff:ff:ff:ff:ff >> inet 10.0.1.21/24 [3] brd 10.0.1.255 scope global ovirtmgmt >> >> valid_lft forever preferred_lft forever >> inet6 fe80::7af2:9eff:fe90:bc50/64 scope link >> valid_lft forever preferred_lft forever >> 24: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state >> DOWN qlen 1000 >> link/ether 02:92:3f:89:f2:c7 brd ff:ff:ff:ff:ff:ff >> 25: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc >> pfifo_fast master ovirtmgmt state UNKNOWN qlen 1000 >> link/ether fe:16:3e:0b:b1:2d brd ff:ff:ff:ff:ff:ff >> inet6 fe80::fc16:3eff:fe0b:b12d/64 scope link >> valid_lft forever preferred_lft forever >> 27: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc >> pfifo_fast master ovirtmgmt state UNKNOWN qlen 1000 >> link/ether fe:1a:4a:16:01:0b brd ff:ff:ff:ff:ff:ff >> inet6 fe80::fc1a:4aff:fe16:10b/64 scope link >> valid_lft forever preferred_lft forever >> 29: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc >> pfifo_fast master ovs-system state UNKNOWN qlen 1000 >> link/ether fe:1a:4a:16:01:0c brd ff:ff:ff:ff:ff:ff >> inet6 fe80::fc1a:4aff:fe16:10c/64 scope link >> valid_lft forever preferred_lft forever >> 31: vnet6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc >> pfifo_fast master ovirtmgmt state UNKNOWN qlen 1000 >> link/ether fe:1a:4a:16:01:07 brd ff:ff:ff:ff:ff:ff >> inet6 fe80::fc1a:4aff:fe16:107/64 scope link >> valid_lft forever preferred_lft forever >> 32: vnet7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc >> pfifo_fast master public state UNKNOWN qlen 1000 >> link/ether fe:1a:4a:16:01:09 brd ff:ff:ff:ff:ff:ff >> inet6 fe80::fc1a:4aff:fe16:109/64 scope link >> valid_lft forever preferred_lft forever >> 33: vnet8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc >> pfifo_fast master ovs-system state UNKNOWN qlen 1000 >> link/ether fe:1a:4a:16:01:0a brd ff:ff:ff:ff:ff:ff >> inet6 fe80::fc1a:4aff:fe16:10a/64 scope link >> valid_lft forever preferred_lft forever >> 34: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 >> qdisc noqueue master ovs-system state UNKNOWN qlen 1000 >> link/ether 46:88:1c:22:6f:c3 brd ff:ff:ff:ff:ff:ff >> inet6 fe80::4488:1cff:fe22:6fc3/64 scope link >> valid_lft forever preferred_lft forever >> >> [root@o3 ~]# ovs-vsctl show >> 8c2c19fc-d9e4-423d-afcb-f5ecff602ca7 >> Bridge br-int >> fail_mode: secure >> Port "vnet4" >> Interface "vnet4" >> Port "ovn-049183-0" >> Interface "ovn-049183-0" >> type: geneve >> options: {csum="true", key=flow, >> remote_ip="10.0.1.18"} >> Port "vnet8" >> Interface "vnet8" >> Port br-int >> Interface br-int >> type: internal >> ovs_version: "2.9.0" >> >> Best regards, >> Samuli >> >> Marcin Mirecki wrote: >> >> Hi Samuli, >> >> Let's first make sure the configuration is correct. >> How did you configure the env? Did you use the automatic >> engine-setup >> configuration? >> >> Can you please send me the output of the following: >> >> on engine: >> ovn-sbctl show >> ovn-nbctl show >> >> on hosts: >> ip addr >> ovs-vsctl show >> >> The 'vdsm-tool ovn-config' command configures the ovn controller to >> use the >> first ip as the ovn central, and the local tunnel to use the second >> one. >> >> Regards, >> Marcin >> >> On Sun, May 6, 2018 at 10:42 AM, Samuli >> Heinonen&lt;samppah(a)neutraali.net&gt; >> wrote: >> >> Hi all, >> >> I'm building a home lab using oVirt+GlusterFS in hyperconverged(ish) >> setup. >> >> My setup consists of 2x nodes with ASRock H110M-STX motherboard, >> Intel >> Pentium G4560 3,5 GHz CPU and 16 GB RAM. Motherboard has integrated >> Intel >> Gigabit I219V LAN. At the moment I'm using RaspberryPi as Gluster >> arbiter >> node. Nodes are connected to basic "desktop switch" without any >> management >> available. >> >> Hardware is nowhere near perfect, but it get its job done and is >> enough >> for playing around. However I'm having problems getting OVN to work >> properly and I'm clueless where to look next. >> >> oVirt is setup like this: >> oVirt engine host oe / 10.0.1.101 >> oVirt hypervisor host o2 / 10.0.1.18 >> oVirt hypervisor host o3 / 10.0.1.21 >> OVN network 10.0.200.0/24 [4] >> >> When I spin up a VM in o2 and o3 with IP address in network >> 10.0.1.0/24 [5] >> >> everything works fine. VMs can interact between each other without >> any >> problems. >> >> Problems show up when I try to use OVN based network between virtual >> machines. If virtual machines are on same hypervisor then everything >> seems >> to work ok. But if I have virtual machine on hypervisor o2 and >> another one >> on hypervisor o3 then TCP connections doesn't work very well. UDP >> seems to >> be ok and it's possible to ping hosts, do dns& ntp queries and so >> on. >> >> Problem with TCP is that for example when taking SSH connection to >> another >> host at some point connection just hangs and most of the time it's >> not even >> possible to even log in before connectiong hangs. If I look into >> tcpdump at >> that point it looks like packets never reach destination. Also, if I >> have >> multiple connections, then all of them hang at the same time. >> >> I have tried switching off tx checksum and other similar settings, >> but it >> didn't make any difference. >> >> I'm suspecting that hardware is not good enough. Before >> investigating into >> new hardware I'd like to get some confirmation that everything is >> setup >> correctly. >> >> When setting up oVirt/OVN I had to run following undocumented >> command to >> get it working at all: vdsm-tool ovn-config 10.0.1.101 10.0.1.21 >> (oVirt >> engine IP, hypervisor IP). Especially this makes me think that I >> have >> missed some crucial part in configuration. >> >> On oVirt engine in /var/log/openvswitch/ovsdb-server-nb.log there >> are >> error messages: >> 2018-05-06T08:30:05.418Z|00913|stream_ssl|WARN|SSL_read: unexpected >> SSL >> connection close >> 2018-05-06T08:30:05.418Z|00914|jsonrpc|WARN|ssl:127.0.0.1:53152 [6]: >> receive >> error: Protocol error >> 2018-05-06T08:30:05.419Z|00915|reconnect|WARN|ssl:127.0.0.1:53152 >> [6]: >> connection dropped (Protocol error) >> >> To be honest, I'm not sure what's causing those error messages or >> are they >> related. I found out some bug reports stating that they are not >> critical. >> >> Any ideas what to do next or should I just get better hardware? :) >> >> Best regards, >> Samuli Heinonen >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users [7] >> > > > > Links: > ------ > [1] http://127.0.0.1/8 > [2] http://10.0.1.18/24 > [3] http://10.0.1.21/24 > [4] http://10.0.200.0/24 > [5] http://10.0.1.0/24 > [6] http://127.0.0.1:53152 > [7] http://lists.ovirt.org/mailman/listinfo/users >