[ovirt-users] Re: OVN Geneve tunnels not been established

Hi Dominik Just saw the below on host dc01-host02 ovs-vsctl show f3b13557-dfb4-45a4-b6af-c995ccf68720 Bridge br-int Port "ovn-95ccb0-0" Interface "ovn-95ccb0-0" type: geneve options: {csum="true", key=flow, remote_ip="dc01-host01"} Port "vnet10" Interface "vnet10" Port "vnet11" Interface "vnet11" Port "vnet0" Interface "vnet0" Port "vnet9" Interface "vnet9" Port "vnet8" Interface "vnet8" Port br-int Interface br-int type: internal Port "vnet12" Interface "vnet12" Port "ovn-be3abc-0" Interface "ovn-be3abc-0" type: geneve options: {csum="true", key=flow, remote_ip="dc01-host02"} Port "vnet7" Interface "vnet7" Port "ovn-c4b238-0" Interface "ovn-c4b238-0" type: geneve options: {csum="true", key=flow, remote_ip="dc02-host01"} Port "vnet6" Interface "vnet6" ovs_version: "2.11.0" Why would this node establish a geneve tunnel to himself? Other nodes do not exhibit this behavior. On Wed, Sep 16, 2020 at 12:21 PM Konstantinos Betsis <k.betsis(a)gmail.com&gt; wrote: > Hi Dominik > > Below is the output of the ovs-vsctl list interface > > _uuid : bdaf92c1-4389-4ddf-aab0-93975076ebb2 > admin_state : up > bfd : {} > bfd_status : {} > cfm_fault : [] > cfm_fault_status : [] > cfm_flap_count : [] > cfm_health : [] > cfm_mpid : [] > cfm_remote_mpids : [] > cfm_remote_opstate : [] > duplex : full > error : [] > external_ids : {attached-mac="56:6f:77:61:00:02", > iface-id="5d03a7a5-82a1-40f9-b50c-353a26167fa3", iface-status=active, > vm-id="e45b7b34-24f2-41ed-b95e-cd1ad532e8d3"} > ifindex : 34 > ingress_policing_burst: 0 > ingress_policing_rate: 0 > lacp_current : [] > link_resets : 1 > link_speed : 10000000 > link_state : up > lldp : {} > mac : [] > mac_in_use : "fe:6f:77:61:00:02" > mtu : 1442 > mtu_request : [] > name : "vnet6" > ofport : 2 > ofport_request : [] > options : {} > other_config : {} > statistics : {collisions=0, rx_bytes=10828495, rx_crc_err=0, > rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, > rx_packets=117713, tx_bytes=20771797, tx_dropped=0, tx_errors=0, > tx_packets=106954} > status : {driver_name=tun, driver_version="1.6", > firmware_version=""} > type : "" > > _uuid : bad80911-3993-4085-a0b0-962b6c9156cd > admin_state : up > bfd : {} > bfd_status : {} > cfm_fault : [] > cfm_fault_status : [] > cfm_flap_count : [] > cfm_health : [] > cfm_mpid : [] > cfm_remote_mpids : [] > cfm_remote_opstate : [] > duplex : [] > error : [] > external_ids : {} > ifindex : 39 > ingress_policing_burst: 0 > ingress_policing_rate: 0 > lacp_current : [] > link_resets : 0 > link_speed : [] > link_state : up > lldp : {} > mac : [] > mac_in_use : "fe:37:52:c4:cb:03" > mtu : [] > mtu_request : [] > name : "ovn-c4b238-0" > ofport : 7 > ofport_request : [] > options : {csum="true", key=flow, remote_ip="192.168.121.164"} > other_config : {} > statistics : {rx_bytes=0, rx_packets=0, tx_bytes=0, tx_packets=0} > status : {tunnel_egress_iface="ovirtmgmt-ams03", > tunnel_egress_iface_carrier=up} > type : geneve > > _uuid : 8e7705d1-0b9d-4e30-8277-c339e7e1c27a > admin_state : up > bfd : {} > bfd_status : {} > cfm_fault : [] > cfm_fault_status : [] > cfm_flap_count : [] > cfm_health : [] > cfm_mpid : [] > cfm_remote_mpids : [] > cfm_remote_opstate : [] > duplex : full > error : [] > external_ids : {attached-mac="56:6f:77:61:00:0d", > iface-id="b7ff5f2b-4bb4-4250-8ad8-8a7e19d2b4c7", iface-status=active, > vm-id="8d73f333-bca4-4b32-9b87-2e7ee07eda84"} > ifindex : 28 > ingress_policing_burst: 0 > ingress_policing_rate: 0 > lacp_current : [] > link_resets : 1 > link_speed : 10000000 > link_state : up > lldp : {} > mac : [] > mac_in_use : "fe:6f:77:61:00:0d" > mtu : 1442 > mtu_request : [] > name : "vnet0" > ofport : 1 > ofport_request : [] > options : {} > other_config : {} > statistics : {collisions=0, rx_bytes=20609787, rx_crc_err=0, > rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, > rx_packets=104535, tx_bytes=10830007, tx_dropped=0, tx_errors=0, > tx_packets=117735} > status : {driver_name=tun, driver_version="1.6", > firmware_version=""} > type : "" > > _uuid : 86dcc68a-63e4-4445-9373-81c1f4502c17 > admin_state : up > bfd : {} > bfd_status : {} > cfm_fault : [] > cfm_fault_status : [] > cfm_flap_count : [] > cfm_health : [] > cfm_mpid : [] > cfm_remote_mpids : [] > cfm_remote_opstate : [] > duplex : full > error : [] > external_ids : {attached-mac="56:6f:77:61:00:10", > iface-id="4e8d5636-4110-41b2-906d-f9b04c2e62cd", iface-status=active, > vm-id="9a002a9b-5f09-4def-a531-d50ff683470b"} > ifindex : 40 > ingress_policing_burst: 0 > ingress_policing_rate: 0 > lacp_current : [] > link_resets : 1 > link_speed : 10000000 > link_state : up > lldp : {} > mac : [] > mac_in_use : "fe:6f:77:61:00:10" > mtu : 1442 > mtu_request : [] > name : "vnet11" > ofport : 10 > ofport_request : [] > options : {} > other_config : {} > statistics : {collisions=0, rx_bytes=3311352, rx_crc_err=0, > rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=51012, > tx_bytes=5514116, tx_dropped=0, tx_errors=0, tx_packets=103456} > status : {driver_name=tun, driver_version="1.6", > firmware_version=""} > type : "" > > _uuid : e8d5e4a2-b9a0-4146-8d98-34713cb443de > admin_state : up > bfd : {} > bfd_status : {} > cfm_fault : [] > cfm_fault_status : [] > cfm_flap_count : [] > cfm_health : [] > cfm_mpid : [] > cfm_remote_mpids : [] > cfm_remote_opstate : [] > duplex : full > error : [] > external_ids : {attached-mac="56:6f:77:61:00:15", > iface-id="b88de6e4-6d77-4e42-b734-4cc676728910", iface-status=active, > vm-id="e45b7b34-24f2-41ed-b95e-cd1ad532e8d3"} > ifindex : 37 > ingress_policing_burst: 0 > ingress_policing_rate: 0 > lacp_current : [] > link_resets : 1 > link_speed : 10000000 > link_state : up > lldp : {} > mac : [] > mac_in_use : "fe:6f:77:61:00:15" > mtu : 1442 > mtu_request : [] > name : "vnet9" > ofport : 5 > ofport_request : [] > options : {} > other_config : {} > statistics : {collisions=0, rx_bytes=180, rx_crc_err=0, > rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=2, > tx_bytes=4500, tx_dropped=0, tx_errors=0, tx_packets=74} > status : {driver_name=tun, driver_version="1.6", > firmware_version=""} > type : "" > > _uuid : 6a2974b3-cd72-4688-a630-0a7e9c779b21 > admin_state : up > bfd : {} > bfd_status : {} > cfm_fault : [] > cfm_fault_status : [] > cfm_flap_count : [] > cfm_health : [] > cfm_mpid : [] > cfm_remote_mpids : [] > cfm_remote_opstate : [] > duplex : full > error : [] > external_ids : {attached-mac="56:6f:77:61:00:17", > iface-id="64681036-26e2-41d7-b73f-ab5302610145", iface-status=active, > vm-id="bf0dc78c-dad5-41a0-914c-ae0da0f9a388"} > ifindex : 41 > ingress_policing_burst: 0 > ingress_policing_rate: 0 > lacp_current : [] > link_resets : 1 > link_speed : 10000000 > link_state : up > lldp : {} > mac : [] > mac_in_use : "fe:6f:77:61:00:17" > mtu : 1442 > mtu_request : [] > name : "vnet12" > ofport : 11 > ofport_request : [] > options : {} > other_config : {} > statistics : {collisions=0, rx_bytes=5513640, rx_crc_err=0, > rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, > rx_packets=103450, tx_bytes=3311868, tx_dropped=0, tx_errors=0, > tx_packets=51018} > status : {driver_name=tun, driver_version="1.6", > firmware_version=""} > type : "" > > _uuid : 44498e54-f122-41a0-a41a-7a88ba2dba9b > admin_state : down > bfd : {} > bfd_status : {} > cfm_fault : [] > cfm_fault_status : [] > cfm_flap_count : [] > cfm_health : [] > cfm_mpid : [] > cfm_remote_mpids : [] > cfm_remote_opstate : [] > duplex : [] > error : [] > external_ids : {} > ifindex : 7 > ingress_policing_burst: 0 > ingress_policing_rate: 0 > lacp_current : [] > link_resets : 0 > link_speed : [] > link_state : down > lldp : {} > mac : [] > mac_in_use : "32:0a:69:67:07:4f" > mtu : 1442 > mtu_request : [] > name : br-int > ofport : 65534 > ofport_request : [] > options : {} > other_config : {} > statistics : {collisions=0, rx_bytes=0, rx_crc_err=0, > rx_dropped=326, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=0, > tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0} > status : {driver_name=openvswitch} > type : internal > > _uuid : e2114584-8ceb-43d6-817b-e457738ead8a > admin_state : up > bfd : {} > bfd_status : {} > cfm_fault : [] > cfm_fault_status : [] > cfm_flap_count : [] > cfm_health : [] > cfm_mpid : [] > cfm_remote_mpids : [] > cfm_remote_opstate : [] > duplex : full > error : [] > external_ids : {attached-mac="56:6f:77:61:00:03", > iface-id="16162721-c815-4cd8-ab57-f22e6e482c7f", iface-status=active, > vm-id="e45b7b34-24f2-41ed-b95e-cd1ad532e8d3"} > ifindex : 35 > ingress_policing_burst: 0 > ingress_policing_rate: 0 > lacp_current : [] > link_resets : 1 > link_speed : 10000000 > link_state : up > lldp : {} > mac : [] > mac_in_use : "fe:6f:77:61:00:03" > mtu : 1442 > mtu_request : [] > name : "vnet7" > ofport : 3 > ofport_request : [] > options : {} > other_config : {} > statistics : {collisions=0, rx_bytes=180, rx_crc_err=0, > rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=2, > tx_bytes=4730, tx_dropped=0, tx_errors=0, tx_packets=77} > status : {driver_name=tun, driver_version="1.6", > firmware_version=""} > type : "" > > _uuid : ee16943e-d145-4080-893f-464098a6388f > admin_state : up > bfd : {} > bfd_status : {} > cfm_fault : [] > cfm_fault_status : [] > cfm_flap_count : [] > cfm_health : [] > cfm_mpid : [] > cfm_remote_mpids : [] > cfm_remote_opstate : [] > duplex : [] > error : [] > external_ids : {} > ifindex : 39 > ingress_policing_burst: 0 > ingress_policing_rate: 0 > lacp_current : [] > link_resets : 0 > link_speed : [] > link_state : up > lldp : {} > mac : [] > mac_in_use : "1e:50:3f:a8:42:d1" > mtu : [] > mtu_request : [] > name : "ovn-be3abc-0" > ofport : 8 > ofport_request : [] > options : {csum="true", key=flow, remote_ip="DC01-host02"} > other_config : {} > statistics : {rx_bytes=0, rx_packets=0, tx_bytes=0, tx_packets=0} > status : {tunnel_egress_iface="ovirtmgmt-ams03", > tunnel_egress_iface_carrier=up} > type : geneve > > _uuid : 86a229be-373e-4c43-b2f1-6190523ed73a > admin_state : up > bfd : {} > bfd_status : {} > cfm_fault : [] > cfm_fault_status : [] > cfm_flap_count : [] > cfm_health : [] > cfm_mpid : [] > cfm_remote_mpids : [] > cfm_remote_opstate : [] > duplex : full > error : [] > external_ids : {attached-mac="56:6f:77:61:00:1c", > iface-id="12d829c3-64eb-44bc-a0bd-d7219991f35f", iface-status=active, > vm-id="e45b7b34-24f2-41ed-b95e-cd1ad532e8d3"} > ifindex : 38 > ingress_policing_burst: 0 > ingress_policing_rate: 0 > lacp_current : [] > link_resets : 1 > link_speed : 10000000 > link_state : up > lldp : {} > mac : [] > mac_in_use : "fe:6f:77:61:00:1c" > mtu : 1442 > mtu_request : [] > name : "vnet10" > ofport : 6 > ofport_request : [] > options : {} > other_config : {} > statistics : {collisions=0, rx_bytes=117912, rx_crc_err=0, > rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=2195, > tx_bytes=4204, tx_dropped=0, tx_errors=0, tx_packets=66} > status : {driver_name=tun, driver_version="1.6", > firmware_version=""} > type : "" > > _uuid : fa4b8d96-bffe-4b56-930e-0e7fcc5f68ac > admin_state : up > bfd : {} > bfd_status : {} > cfm_fault : [] > cfm_fault_status : [] > cfm_flap_count : [] > cfm_health : [] > cfm_mpid : [] > cfm_remote_mpids : [] > cfm_remote_opstate : [] > duplex : [] > error : [] > external_ids : {} > ifindex : 39 > ingress_policing_burst: 0 > ingress_policing_rate: 0 > lacp_current : [] > link_resets : 0 > link_speed : [] > link_state : up > lldp : {} > mac : [] > mac_in_use : "7a:28:24:eb:ec:d2" > mtu : [] > mtu_request : [] > name : "ovn-95ccb0-0" > ofport : 9 > ofport_request : [] > options : {csum="true", key=flow, remote_ip="DC01-host01"} > other_config : {} > statistics : {rx_bytes=0, rx_packets=0, tx_bytes=12840478, > tx_packets=224029} > status : {tunnel_egress_iface="ovirtmgmt-ams03", > tunnel_egress_iface_carrier=up} > type : geneve > > _uuid : 5e3df5c7-958c-491d-8d41-0ae83c613f1d > admin_state : up > bfd : {} > bfd_status : {} > cfm_fault : [] > cfm_fault_status : [] > cfm_flap_count : [] > cfm_health : [] > cfm_mpid : [] > cfm_remote_mpids : [] > cfm_remote_opstate : [] > duplex : full > error : [] > external_ids : {attached-mac="56:6f:77:61:00:06", > iface-id="9a6cc189-0934-4468-97ae-09f90fa4598d", iface-status=active, > vm-id="e45b7b34-24f2-41ed-b95e-cd1ad532e8d3"} > ifindex : 36 > ingress_policing_burst: 0 > ingress_policing_rate: 0 > lacp_current : [] > link_resets : 1 > link_speed : 10000000 > link_state : up > lldp : {} > mac : [] > mac_in_use : "fe:6f:77:61:00:06" > mtu : 1442 > mtu_request : [] > name : "vnet8" > ofport : 4 > ofport_request : [] > options : {} > other_config : {} > statistics : {collisions=0, rx_bytes=180, rx_crc_err=0, > rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=2, > tx_bytes=8829812, tx_dropped=0, tx_errors=0, tx_packets=154540} > status : {driver_name=tun, driver_version="1.6", > firmware_version=""} > type : "" > > > I've identified which VMs have these MAC addresses but i do not see any > "conflict" with any other VM's MAC address. > > I really do not understand why these will create a conflict. > > On Wed, Sep 16, 2020 at 12:06 PM Dominik Holler <dholler(a)redhat.com&gt; > wrote: > >> >> >> On Tue, Sep 15, 2020 at 6:53 PM Konstantinos Betsis <k.betsis(a)gmail.com&gt; >> wrote: >> >>> So a new test-net was created under DC01 and was depicted in the >>> networks tab under both DC01 and DC02. >>> I believe for some reason networks are duplicated in DCs, maybe for >>> future use??? Don't know. >>> If one tries to delete the network from the other DC it gets an error, >>> while if deleted from the once initially created it gets deleted from both. >>> >>> >> In oVirt a logical network is an entity in a data center. If the >> automatic synchronization is enabled on the ovirt-provider-ovn entity in >> oVirt Engine, the OVN networks are reflected to all data centers. If you do >> not like this, you can disable the automatic synchronization of the >> ovirt-provider-ovn in Admin Portal. >> >> >>> From the DC01-node02 i get the following errors: >>> >>> 2020-09-15T16:48:49.904Z|22748|main|INFO|OVNSB commit failed, force >>> recompute next time. >>> 2020-09-15T16:48:49.905Z|22749|binding|INFO|Claiming lport >>> 9a6cc189-0934-4468-97ae-09f90fa4598d for this chassis. >>> 2020-09-15T16:48:49.905Z|22750|binding|INFO|9a6cc189-0934-4468-97ae-09f90fa4598d: >>> Claiming 56:6f:77:61:00:06 >>> 2020-09-15T16:48:49.905Z|22751|binding|INFO|Claiming lport >>> 16162721-c815-4cd8-ab57-f22e6e482c7f for this chassis. >>> 2020-09-15T16:48:49.905Z|22752|binding|INFO|16162721-c815-4cd8-ab57-f22e6e482c7f: >>> Claiming 56:6f:77:61:00:03 >>> 2020-09-15T16:48:49.905Z|22753|binding|INFO|Claiming lport >>> b88de6e4-6d77-4e42-b734-4cc676728910 for this chassis. >>> 2020-09-15T16:48:49.905Z|22754|binding|INFO|b88de6e4-6d77-4e42-b734-4cc676728910: >>> Claiming 56:6f:77:61:00:15 >>> 2020-09-15T16:48:49.905Z|22755|binding|INFO|Claiming lport >>> b7ff5f2b-4bb4-4250-8ad8-8a7e19d2b4c7 for this chassis. >>> 2020-09-15T16:48:49.905Z|22756|binding|INFO|b7ff5f2b-4bb4-4250-8ad8-8a7e19d2b4c7: >>> Claiming 56:6f:77:61:00:0d >>> 2020-09-15T16:48:49.905Z|22757|binding|INFO|Claiming lport >>> 5d03a7a5-82a1-40f9-b50c-353a26167fa3 for this chassis. >>> 2020-09-15T16:48:49.905Z|22758|binding|INFO|5d03a7a5-82a1-40f9-b50c-353a26167fa3: >>> Claiming 56:6f:77:61:00:02 >>> 2020-09-15T16:48:49.905Z|22759|binding|INFO|Claiming lport >>> 12d829c3-64eb-44bc-a0bd-d7219991f35f for this chassis. >>> 2020-09-15T16:48:49.905Z|22760|binding|INFO|12d829c3-64eb-44bc-a0bd-d7219991f35f: >>> Claiming 56:6f:77:61:00:1c >>> 2020-09-15T16:48:49.959Z|22761|main|INFO|OVNSB commit failed, force >>> recompute next time. >>> 2020-09-15T16:48:49.960Z|22762|binding|INFO|Claiming lport >>> 9a6cc189-0934-4468-97ae-09f90fa4598d for this chassis. >>> 2020-09-15T16:48:49.960Z|22763|binding|INFO|9a6cc189-0934-4468-97ae-09f90fa4598d: >>> Claiming 56:6f:77:61:00:06 >>> 2020-09-15T16:48:49.960Z|22764|binding|INFO|Claiming lport >>> 16162721-c815-4cd8-ab57-f22e6e482c7f for this chassis. >>> 2020-09-15T16:48:49.960Z|22765|binding|INFO|16162721-c815-4cd8-ab57-f22e6e482c7f: >>> Claiming 56:6f:77:61:00:03 >>> 2020-09-15T16:48:49.960Z|22766|binding|INFO|Claiming lport >>> b88de6e4-6d77-4e42-b734-4cc676728910 for this chassis. >>> 2020-09-15T16:48:49.960Z|22767|binding|INFO|b88de6e4-6d77-4e42-b734-4cc676728910: >>> Claiming 56:6f:77:61:00:15 >>> 2020-09-15T16:48:49.960Z|22768|binding|INFO|Claiming lport >>> b7ff5f2b-4bb4-4250-8ad8-8a7e19d2b4c7 for this chassis. >>> 2020-09-15T16:48:49.960Z|22769|binding|INFO|b7ff5f2b-4bb4-4250-8ad8-8a7e19d2b4c7: >>> Claiming 56:6f:77:61:00:0d >>> 2020-09-15T16:48:49.960Z|22770|binding|INFO|Claiming lport >>> 5d03a7a5-82a1-40f9-b50c-353a26167fa3 for this chassis. >>> 2020-09-15T16:48:49.960Z|22771|binding|INFO|5d03a7a5-82a1-40f9-b50c-353a26167fa3: >>> Claiming 56:6f:77:61:00:02 >>> 2020-09-15T16:48:49.960Z|22772|binding|INFO|Claiming lport >>> 12d829c3-64eb-44bc-a0bd-d7219991f35f for this chassis. >>> 2020-09-15T16:48:49.960Z|22773|binding|INFO|12d829c3-64eb-44bc-a0bd-d7219991f35f: >>> Claiming 56:6f:77:61:00:1c >>> >>> >>> And this repeats forever. >>> >>> >> Looks like the southbound db is confused. >> >> Can you try to delete all chassis listed by >> sudo ovn-sbctl show >> via >> sudo /usr/share/ovirt-provider-ovn/scripts/remove_chassis.sh dev-host0 >> ? >> if the script remove_chassis.sh is not installed, you can use >> >> https://github.com/oVirt/ovirt-provider-ovn/blob/master/provider/scripts/... >> instead. >> >> Can you please also share the output of >> ovs-vsctl list Interface >> on the host which produced the logfile above? >> >> >> >> >>> The connections to ovn-sbctl is ok and the geneve tunnels are depicted >>> under ovs-vsctl ok. >>> VMs still not able to ping each other. >>> >>> On Tue, Sep 15, 2020 at 7:22 PM Dominik Holler <dholler(a)redhat.com&gt; >>> wrote: >>> >>>> >>>> >>>> On Tue, Sep 15, 2020 at 6:18 PM Konstantinos Betsis < >>>> k.betsis(a)gmail.com&gt; wrote: >>>> >>>>> Hi Dominik >>>>> >>>>> Fixed the issue. >>>>> >>>> >>>> Thanks. >>>> >>>> >>>>> I believe the /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf >>>>> needed update also. >>>>> The package is upgraded to the latest version. >>>>> >>>>> Once the provider was updated with the following it functioned >>>>> perfectly: >>>>> >>>>> Name: ovirt-provider-ovn >>>>> Description: oVirt network provider for OVN >>>>> Type: External Network Provider >>>>> Network Plugin: oVirt Network Provider for OVN >>>>> Automatic Synchronization: Checked >>>>> Unmanaged: Unchecked >>>>> Provider URL: https:dc02-ovirt01.testdomain.com:9696 >>>>> Requires Authentication: Checked >>>>> Username: admin@internal >>>>> Password: "The admin password" >>>>> Protocol: HTTPS >>>>> Host Name: dc02-ovirt01.testdomain.com >>>>> API Port: 35357 >>>>> API Version: v2.0 >>>>> Tenant Name: "Empty" >>>>> >>>>> For some reason the TLS certificate was in conflict with the ovn >>>>> provider details, i would bet the "host" entry. >>>>> >>>>> So now geneve tunnels are established. >>>>> OVN provider is working. >>>>> >>>>> But VMs still do not communicated on the same VM network spanning >>>>> different hosts. >>>>> >>>>> So if we have a VM network test-net on both dc01-host01 and >>>>> dc01-host02 and each host has a VM with IP addresses on the same network, >>>>> VMs on the same VM network should communicate directly. >>>>> But traffic does not reach each other. >>>>> >>>>> >>>> Can you create a new external network, with port security disabled, >>>> and an IPv4 subnet? >>>> If the VMs get an IP address via DHCP, ovn is working, and should be >>>> able to ping each other, too. >>>> If not, there should be a helpful entry in the ovn-controller.log of >>>> the host the VM is running. >>>> >>>> >>>>> On Tue, Sep 15, 2020 at 7:07 PM Dominik Holler <dholler(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> Can you try again with: >>>>>> >>>>>> [OVN REMOTE] >>>>>> ovn-remote=ssl:127.0.0.1:6641 >>>>>> [SSL] >>>>>> https-enabled=false >>>>>> ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem >>>>>> ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer >>>>>> ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass >>>>>> [OVIRT] >>>>>> ovirt-sso-client-secret=*random_test* >>>>>> ovirt-host=https://dc02-ovirt01.testdomain.com:443 >>>>>> <https://dc02-ovirt01.testdomain.com/> >>>>>> ovirt-sso-client-id=ovirt-provider-ovn >>>>>> ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem >>>>>> [NETWORK] >>>>>> port-security-enabled-default=True >>>>>> [PROVIDER] >>>>>> >>>>>> provider-host=dc02-ovirt01.testdomain.com >>>>>> >>>>>> >>>>>> >>>>>> Please note that the should match the HTTP or HTTPS in the of the >>>>>> ovirt-prover-ovn configuration in oVirt Engine. >>>>>> So if the ovirt-provider-ovn entity in Engine is on HTTP, the config >>>>>> file should use >>>>>> https-enabled=false >>>>>> >>>>>> >>>>>> On Tue, Sep 15, 2020 at 5:56 PM Konstantinos Betsis < >>>>>> k.betsis(a)gmail.com&gt; wrote: >>>>>> >>>>>>> This is the updated one: >>>>>>> >>>>>>> # This file is automatically generated by engine-setup. Please do >>>>>>> not edit manually >>>>>>> [OVN REMOTE] >>>>>>> ovn-remote=ssl:127.0.0.1:6641 >>>>>>> [SSL] >>>>>>> https-enabled=true >>>>>>> ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem >>>>>>> ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer >>>>>>> >>>>>>> ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass >>>>>>> [OVIRT] >>>>>>> ovirt-sso-client-secret=*random_text* >>>>>>> ovirt-host=https://dc02-ovirt01.testdomain.com:443 >>>>>>> ovirt-sso-client-id=ovirt-provider-ovn >>>>>>> ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem >>>>>>> [NETWORK] >>>>>>> port-security-enabled-default=True >>>>>>> [PROVIDER] >>>>>>> provider-host=dc02-ovirt01.testdomain.com >>>>>>> [AUTH] >>>>>>> auth-plugin=auth.plugins.static_token:NoAuthPlugin >>>>>>> >>>>>>> >>>>>>> However, it still does not connect. >>>>>>> It prompts for the certificate but then fails and prompts to see >>>>>>> the log but the ovirt-provider-ovn.log does not list anything. >>>>>>> >>>>>>> Yes we've got ovirt for about a year now from about version 4.1 >>>>>>> >>>>>>> >>>>>> This might explain the trouble. Upgrade of ovirt-provider-ovn should >>>>>> work flawlessly starting from oVirt 4.2. >>>>>> >>>>>> >>>>>>> On Tue, Sep 15, 2020 at 6:44 PM Dominik Holler <dholler(a)redhat.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Sep 15, 2020 at 5:34 PM Konstantinos Betsis < >>>>>>>> k.betsis(a)gmail.com&gt; wrote: >>>>>>>> >>>>>>>>> There is a file with the below entries >>>>>>>>> >>>>>>>> >>>>>>>> Impressive, do you know when this config file was created and if >>>>>>>> it was manually modified? >>>>>>>> Is this an upgrade from oVirt 4.1? >>>>>>>> >>>>>>>> >>>>>>>>> [root@dc02-ovirt01 log]# cat >>>>>>>>> /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf >>>>>>>>> # This file is automatically generated by engine-setup. Please do >>>>>>>>> not edit manually >>>>>>>>> [OVN REMOTE] >>>>>>>>> ovn-remote=tcp:127.0.0.1:6641 >>>>>>>>> [SSL] >>>>>>>>> https-enabled=false >>>>>>>>> ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem >>>>>>>>> ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer >>>>>>>>> >>>>>>>>> ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass >>>>>>>>> [OVIRT] >>>>>>>>> ovirt-sso-client-secret=*random_test* >>>>>>>>> ovirt-host=https://dc02-ovirt01.testdomain.com:443 >>>>>>>>> ovirt-sso-client-id=ovirt-provider-ovn >>>>>>>>> ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem >>>>>>>>> [NETWORK] >>>>>>>>> port-security-enabled-default=True >>>>>>>>> [PROVIDER] >>>>>>>>> >>>>>>>>> provider-host=dc02-ovirt01.testdomain.com >>>>>>>>> >>>>>>>>> The only entry missing is the [AUTH] and under [SSL] the >>>>>>>>> https-enabled is false. Should I edit this in this file or is this going to >>>>>>>>> break everything? >>>>>>>>> >>>>>>>>> >>>>>>>> Changing the file should improve, but better create a backup into >>>>>>>> another diretory before modification. >>>>>>>> The only required change is >>>>>>>> from >>>>>>>> ovn-remote=tcp:127.0.0.1:6641 >>>>>>>> to >>>>>>>> ovn-remote=ssl:127.0.0.1:6641 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Tue, Sep 15, 2020 at 6:27 PM Dominik Holler < >>>>>>>>> dholler(a)redhat.com&gt; wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Sep 15, 2020 at 5:11 PM Konstantinos Betsis < >>>>>>>>>> k.betsis(a)gmail.com&gt; wrote: >>>>>>>>>> >>>>>>>>>>> Hi Dominik >>>>>>>>>>> >>>>>>>>>>> That immediately fixed the geneve tunnels between all hosts. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> thanks for the feedback. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> However, the ovn provider is not broken. >>>>>>>>>>> After fixing the networks we tried to move a VM to the >>>>>>>>>>> DC01-host01 so we powered it down and simply configured it to run on >>>>>>>>>>> dc01-node01. >>>>>>>>>>> >>>>>>>>>>> While checking the logs on the ovirt engine i noticed the below: >>>>>>>>>>> Failed to synchronize networks of Provider ovirt-provider-ovn. >>>>>>>>>>> >>>>>>>>>>> The ovn-provider configure on the engine is the below: >>>>>>>>>>> Name: ovirt-provider-ovn >>>>>>>>>>> Description: oVirt network provider for OVN >>>>>>>>>>> Type: External Network Provider >>>>>>>>>>> Network Plugin: oVirt Network Provider for OVN >>>>>>>>>>> Automatic Synchronization: Checked >>>>>>>>>>> Unmanaged: Unchecked >>>>>>>>>>> Provider URL: http:localhost:9696 >>>>>>>>>>> Requires Authentication: Checked >>>>>>>>>>> Username: admin@internal >>>>>>>>>>> Password: "The admin password" >>>>>>>>>>> Protocol: hTTP >>>>>>>>>>> Host Name: dc02-ovirt01 >>>>>>>>>>> API Port: 35357 >>>>>>>>>>> API Version: v2.0 >>>>>>>>>>> Tenant Name: "Empty" >>>>>>>>>>> >>>>>>>>>>> In the past this was deleted by an engineer and recreated as >>>>>>>>>>> per the documentation, and it worked. Do we need to update something due to >>>>>>>>>>> the SSL on the ovn? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> Is there a file in /etc/ovirt-provider-ovn/conf.d/ ? >>>>>>>>>> engine-setup should have created one. >>>>>>>>>> If the file is missing, for testing purposes, you can create a >>>>>>>>>> file /etc/ovirt-provider-ovn/conf.d/00-setup-ovirt-provider-ovn-test.conf : >>>>>>>>>> [PROVIDER] >>>>>>>>>> provider-host=REPLACE_WITH_FQDN >>>>>>>>>> [SSL] >>>>>>>>>> ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer >>>>>>>>>> >>>>>>>>>> ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass >>>>>>>>>> ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem >>>>>>>>>> https-enabled=true >>>>>>>>>> [OVN REMOTE] >>>>>>>>>> ovn-remote=ssl:127.0.0.1:6641 >>>>>>>>>> [AUTH] >>>>>>>>>> auth-plugin=auth.plugins.static_token:NoAuthPlugin >>>>>>>>>> [NETWORK] >>>>>>>>>> port-security-enabled-default=True >>>>>>>>>> >>>>>>>>>> and restart the ovirt-provider-ovn service. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> From the ovn-provider logs the below is generated after a >>>>>>>>>>> service restart and when the start VM is triggered >>>>>>>>>>> >>>>>>>>>>> 2020-09-15 15:07:33,579 root Starting server >>>>>>>>>>> 2020-09-15 15:07:33,579 root Version: 1.2.29-1 >>>>>>>>>>> 2020-09-15 15:07:33,579 root Build date: 20191217125241 >>>>>>>>>>> 2020-09-15 15:07:33,579 root Githash: cb5a80d >>>>>>>>>>> 2020-09-15 15:08:26,582 root From: ::ffff:127.0.0.1:59980 >>>>>>>>>>> Request: GET /v2.0/ports >>>>>>>>>>> 2020-09-15 15:08:26,582 root Could not retrieve schema from tcp: >>>>>>>>>>> 127.0.0.1:6641: Unknown error -1 >>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>> File >>>>>>>>>>> "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 138, in >>>>>>>>>>> _handle_request >>>>>>>>>>> method, path_parts, content >>>>>>>>>>> File >>>>>>>>>>> "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 175, in >>>>>>>>>>> handle_request >>>>>>>>>>> return self.call_response_handler(handler, content, >>>>>>>>>>> parameters) >>>>>>>>>>> File "/usr/share/ovirt-provider-ovn/handlers/neutron.py", >>>>>>>>>>> line 35, in call_response_handler >>>>>>>>>>> with NeutronApi() as ovn_north: >>>>>>>>>>> File "/usr/share/ovirt-provider-ovn/neutron/neutron_api.py", >>>>>>>>>>> line 95, in __init__ >>>>>>>>>>> self.ovsidl, self.idl = ovn_connection.connect() >>>>>>>>>>> File "/usr/share/ovirt-provider-ovn/ovn_connection.py", line >>>>>>>>>>> 46, in connect >>>>>>>>>>> ovnconst.OVN_NORTHBOUND >>>>>>>>>>> File >>>>>>>>>>> "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", >>>>>>>>>>> line 127, in from_server >>>>>>>>>>> helper = idlutils.get_schema_helper(connection_string, >>>>>>>>>>> schema_name) >>>>>>>>>>> File >>>>>>>>>>> "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", >>>>>>>>>>> line 128, in get_schema_helper >>>>>>>>>>> 'err': os.strerror(err)}) >>>>>>>>>>> Exception: Could not retrieve schema from tcp:127.0.0.1:6641: >>>>>>>>>>> Unknown error -1 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> When i update the ovn provider from the GUI to have >>>>>>>>>>> https://localhost:9696/ and HTTPS as the protocol the test >>>>>>>>>>> fails. >>>>>>>>>>> >>>>>>>>>>> On Tue, Sep 15, 2020 at 5:35 PM Dominik Holler < >>>>>>>>>>> dholler(a)redhat.com&gt; wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Sep 14, 2020 at 9:25 AM Konstantinos Betsis < >>>>>>>>>>>> k.betsis(a)gmail.com&gt; wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Dominik >>>>>>>>>>>>> >>>>>>>>>>>>> When these commands are used on the ovirt-engine host the >>>>>>>>>>>>> output is the one depicted in your email. >>>>>>>>>>>>> For your reference see also below: >>>>>>>>>>>>> >>>>>>>>>>>>> [root@ath01-ovirt01 certs]# ovn-nbctl get-ssl >>>>>>>>>>>>> Private key: /etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass >>>>>>>>>>>>> Certificate: /etc/pki/ovirt-engine/certs/ovn-ndb.cer >>>>>>>>>>>>> CA Certificate: /etc/pki/ovirt-engine/ca.pem >>>>>>>>>>>>> Bootstrap: false >>>>>>>>>>>>> [root@ath01-ovirt01 certs]# ovn-nbctl get-connection >>>>>>>>>>>>> ptcp:6641 >>>>>>>>>>>>> >>>>>>>>>>>>> [root@ath01-ovirt01 certs]# ovn-sbctl get-ssl >>>>>>>>>>>>> Private key: /etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass >>>>>>>>>>>>> Certificate: /etc/pki/ovirt-engine/certs/ovn-sdb.cer >>>>>>>>>>>>> CA Certificate: /etc/pki/ovirt-engine/ca.pem >>>>>>>>>>>>> Bootstrap: false >>>>>>>>>>>>> [root@ath01-ovirt01 certs]# ovn-sbctl get-connection >>>>>>>>>>>>> read-write role="" ptcp:6642 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> ^^^ the line above points to the problem: ovn-central is >>>>>>>>>>>> configured to use plain TCP without ssl. >>>>>>>>>>>> engine-setup usually configures ovn-central to use SSL. That >>>>>>>>>>>> the files /etc/pki/ovirt-engine/keys/ovn-* exist, shows, >>>>>>>>>>>> that engine-setup was triggered correctly. Looks like the ovn >>>>>>>>>>>> db was dropped somehow, this should not happen. >>>>>>>>>>>> This can be fixed manually by executing the following commands >>>>>>>>>>>> on engine's machine: >>>>>>>>>>>> ovn-nbctl set-ssl >>>>>>>>>>>> /etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass >>>>>>>>>>>> /etc/pki/ovirt-engine/certs/ovn-ndb.cer /etc/pki/ovirt-engine/ca.pem >>>>>>>>>>>> ovn-nbctl set-connection pssl:6641 >>>>>>>>>>>> ovn-sbctl set-ssl >>>>>>>>>>>> /etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass >>>>>>>>>>>> /etc/pki/ovirt-engine/certs/ovn-sdb.cer /etc/pki/ovirt-engine/ca.pem >>>>>>>>>>>> ovn-sbctl set-connection pssl:6642 >>>>>>>>>>>> >>>>>>>>>>>> The /var/log/openvswitch/ovn-controller.log on the hosts >>>>>>>>>>>> should tell that br-int.mgmt is connected now. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> [root@ath01-ovirt01 certs]# ls -l >>>>>>>>>>>>> /etc/pki/ovirt-engine/keys/ovn-* >>>>>>>>>>>>> -rw-r-----. 1 root hugetlbfs 1828 Jun 25 11:08 >>>>>>>>>>>>> /etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass >>>>>>>>>>>>> -rw-------. 1 root root 2893 Jun 25 11:08 >>>>>>>>>>>>> /etc/pki/ovirt-engine/keys/ovn-ndb.p12 >>>>>>>>>>>>> -rw-r-----. 1 root hugetlbfs 1828 Jun 25 11:08 >>>>>>>>>>>>> /etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass >>>>>>>>>>>>> -rw-------. 1 root root 2893 Jun 25 11:08 >>>>>>>>>>>>> /etc/pki/ovirt-engine/keys/ovn-sdb.p12 >>>>>>>>>>>>> >>>>>>>>>>>>> When i try the above commands on the node hosts the following >>>>>>>>>>>>> happens: >>>>>>>>>>>>> ovn-nbctl get-ssl / get-connection >>>>>>>>>>>>> ovn-nbctl: unix:/var/run/openvswitch/ovnnb_db.sock: database >>>>>>>>>>>>> connection failed (No such file or directory) >>>>>>>>>>>>> The above i believe is expected since no northbound >>>>>>>>>>>>> connections should be established from the host nodes. >>>>>>>>>>>>> >>>>>>>>>>>>> ovn-sbctl get-ssl /get-connection >>>>>>>>>>>>> The output is stuck till i terminate it. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> Yes, the ovn-* commands works only on engine's machine, which >>>>>>>>>>>> has the role ovn-central. >>>>>>>>>>>> On the hosts, there is only the ovn-controller, which connects >>>>>>>>>>>> the ovn southbound to openvswitch on the host. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> For the requested logs the below are found in the >>>>>>>>>>>>> ovsdb-server-sb.log >>>>>>>>>>>>> >>>>>>>>>>>>> 2020-09-14T07:18:38.187Z|219636|reconnect|WARN|tcp:DC02-host01:33146: >>>>>>>>>>>>> connection dropped (Protocol error) >>>>>>>>>>>>> 2020-09-14T07:18:41.946Z|219637|reconnect|WARN|tcp:DC01-host01:51188: >>>>>>>>>>>>> connection dropped (Protocol error) >>>>>>>>>>>>> 2020-09-14T07:18:43.033Z|219638|reconnect|WARN|tcp:DC01-host02:37044: >>>>>>>>>>>>> connection dropped (Protocol error) >>>>>>>>>>>>> 2020-09-14T07:18:46.198Z|219639|reconnect|WARN|tcp:DC02-host01:33148: >>>>>>>>>>>>> connection dropped (Protocol error) >>>>>>>>>>>>> 2020-09-14T07:18:50.069Z|219640|jsonrpc|WARN|Dropped 4 log >>>>>>>>>>>>> messages in last 12 seconds (most recently, 4 seconds ago) due to excessive >>>>>>>>>>>>> rate >>>>>>>>>>>>> 2020-09-14T07:18:50.069Z|219641|jsonrpc|WARN|tcp:DC01-host01:51190: >>>>>>>>>>>>> error parsing stream: line 0, column 0, byte 0: invalid character U+0016 >>>>>>>>>>>>> 2020-09-14T07:18:50.069Z|219642|jsonrpc|WARN|Dropped 4 log >>>>>>>>>>>>> messages in last 12 seconds (most recently, 4 seconds ago) due to excessive >>>>>>>>>>>>> rate >>>>>>>>>>>>> 2020-09-14T07:18:50.069Z|219643|jsonrpc|WARN|tcp:DC01-host01:51190: >>>>>>>>>>>>> received SSL data on JSON-RPC channel >>>>>>>>>>>>> 2020-09-14T07:18:50.070Z|219644|reconnect|WARN|tcp:DC01-host01:51190: >>>>>>>>>>>>> connection dropped (Protocol error) >>>>>>>>>>>>> 2020-09-14T07:18:51.147Z|219645|reconnect|WARN|tcp:DC01-host02:37046: >>>>>>>>>>>>> connection dropped (Protocol error) >>>>>>>>>>>>> 2020-09-14T07:18:54.209Z|219646|reconnect|WARN|tcp:DC02-host01:33150: >>>>>>>>>>>>> connection dropped (Protocol error) >>>>>>>>>>>>> 2020-09-14T07:18:58.192Z|219647|reconnect|WARN|tcp:DC01-host01:51192: >>>>>>>>>>>>> connection dropped (Protocol error) >>>>>>>>>>>>> 2020-09-14T07:18:59.262Z|219648|jsonrpc|WARN|Dropped 3 log >>>>>>>>>>>>> messages in last 8 seconds (most recently, 1 seconds ago) due to excessive >>>>>>>>>>>>> rate >>>>>>>>>>>>> 2020-09-14T07:18:59.262Z|219649|jsonrpc|WARN|tcp:DC01-host02:37048: >>>>>>>>>>>>> error parsing stream: line 0, column 0, byte 0: invalid character U+0016 >>>>>>>>>>>>> 2020-09-14T07:18:59.263Z|219650|jsonrpc|WARN|Dropped 3 log >>>>>>>>>>>>> messages in last 8 seconds (most recently, 1 seconds ago) due to excessive >>>>>>>>>>>>> rate >>>>>>>>>>>>> 2020-09-14T07:18:59.263Z|219651|jsonrpc|WARN|tcp:DC01-host02:37048: >>>>>>>>>>>>> received SSL data on JSON-RPC channel >>>>>>>>>>>>> 2020-09-14T07:18:59.263Z|219652|reconnect|WARN|tcp:DC01-host02:37048: >>>>>>>>>>>>> connection dropped (Protocol error) >>>>>>>>>>>>> 2020-09-14T07:19:02.220Z|219653|reconnect|WARN|tcp:DC02-host01:33152: >>>>>>>>>>>>> connection dropped (Protocol error) >>>>>>>>>>>>> 2020-09-14T07:19:06.316Z|219654|reconnect|WARN|tcp:DC01-host01:51194: >>>>>>>>>>>>> connection dropped (Protocol error) >>>>>>>>>>>>> 2020-09-14T07:19:07.386Z|219655|reconnect|WARN|tcp:DC01-host02:37050: >>>>>>>>>>>>> connection dropped (Protocol error) >>>>>>>>>>>>> 2020-09-14T07:19:10.232Z|219656|reconnect|WARN|tcp:DC02-host01:33154: >>>>>>>>>>>>> connection dropped (Protocol error) >>>>>>>>>>>>> 2020-09-14T07:19:14.439Z|219657|jsonrpc|WARN|Dropped 4 log >>>>>>>>>>>>> messages in last 12 seconds (most recently, 4 seconds ago) due to excessive >>>>>>>>>>>>> rate >>>>>>>>>>>>> 2020-09-14T07:19:14.439Z|219658|jsonrpc|WARN|tcp:DC01-host01:51196: >>>>>>>>>>>>> error parsing stream: line 0, column 0, byte 0: invalid character U+0016 >>>>>>>>>>>>> 2020-09-14T07:19:14.439Z|219659|jsonrpc|WARN|Dropped 4 log >>>>>>>>>>>>> messages in last 12 seconds (most recently, 4 seconds ago) due to excessive >>>>>>>>>>>>> rate >>>>>>>>>>>>> 2020-09-14T07:19:14.439Z|219660|jsonrpc|WARN|tcp:DC01-host01:51196: >>>>>>>>>>>>> received SSL data on JSON-RPC channel >>>>>>>>>>>>> 2020-09-14T07:19:14.440Z|219661|reconnect|WARN|tcp:DC01-host01:51196: >>>>>>>>>>>>> connection dropped (Protocol error) >>>>>>>>>>>>> 2020-09-14T07:19:15.505Z|219662|reconnect|WARN|tcp:DC01-host02:37052: >>>>>>>>>>>>> connection dropped (Protocol error) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> How can we fix these SSL errors? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I addressed this above. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> I thought vdsm did the certificate provisioning on the host >>>>>>>>>>>>> nodes as to communicate to the engine host node. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> Yes, this seems to work in your scenario, just the SSL >>>>>>>>>>>> configuration on the ovn-central was lost. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Sep 11, 2020 at 6:39 PM Dominik Holler < >>>>>>>>>>>>> dholler(a)redhat.com&gt; wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Looks still like the ovn-controller on the host has problems >>>>>>>>>>>>>> communicating with ovn-southbound. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Are there any hints in /var/log/openvswitch/*.log, >>>>>>>>>>>>>> especially in /var/log/openvswitch/ovsdb-server-sb.log ? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can you please check the output of >>>>>>>>>>>>>> >>>>>>>>>>>>>> ovn-nbctl get-ssl >>>>>>>>>>>>>> ovn-nbctl get-connection >>>>>>>>>>>>>> ovn-sbctl get-ssl >>>>>>>>>>>>>> ovn-sbctl get-connection >>>>>>>>>>>>>> ls -l /etc/pki/ovirt-engine/keys/ovn-* >>>>>>>>>>>>>> >>>>>>>>>>>>>> it should be similar to >>>>>>>>>>>>>> >>>>>>>>>>>>>> [root@ovirt-43 ~]# ovn-nbctl get-ssl >>>>>>>>>>>>>> Private key: /etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass >>>>>>>>>>>>>> Certificate: /etc/pki/ovirt-engine/certs/ovn-ndb.cer >>>>>>>>>>>>>> CA Certificate: /etc/pki/ovirt-engine/ca.pem >>>>>>>>>>>>>> Bootstrap: false >>>>>>>>>>>>>> [root@ovirt-43 ~]# ovn-nbctl get-connection >>>>>>>>>>>>>> pssl:6641:[::] >>>>>>>>>>>>>> [root@ovirt-43 ~]# ovn-sbctl get-ssl >>>>>>>>>>>>>> Private key: /etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass >>>>>>>>>>>>>> Certificate: /etc/pki/ovirt-engine/certs/ovn-sdb.cer >>>>>>>>>>>>>> CA Certificate: /etc/pki/ovirt-engine/ca.pem >>>>>>>>>>>>>> Bootstrap: false >>>>>>>>>>>>>> [root@ovirt-43 ~]# ovn-sbctl get-connection >>>>>>>>>>>>>> read-write role="" pssl:6642:[::] >>>>>>>>>>>>>> [root@ovirt-43 ~]# ls -l /etc/pki/ovirt-engine/keys/ovn-* >>>>>>>>>>>>>> -rw-r-----. 1 root hugetlbfs 1828 Oct 14 2019 >>>>>>>>>>>>>> /etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass >>>>>>>>>>>>>> -rw-------. 1 root root 2709 Oct 14 2019 >>>>>>>>>>>>>> /etc/pki/ovirt-engine/keys/ovn-ndb.p12 >>>>>>>>>>>>>> -rw-r-----. 1 root hugetlbfs 1828 Oct 14 2019 >>>>>>>>>>>>>> /etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass >>>>>>>>>>>>>> -rw-------. 1 root root 2709 Oct 14 2019 >>>>>>>>>>>>>> /etc/pki/ovirt-engine/keys/ovn-sdb.p12 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Sep 11, 2020 at 1:10 PM Konstantinos Betsis < >>>>>>>>>>>>>> k.betsis(a)gmail.com&gt; wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I did a restart of the ovn-controller, this is the output >>>>>>>>>>>>>>> of the ovn-controller.log >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2020-09-11T10:54:07.566Z|00001|vlog|INFO|opened log file >>>>>>>>>>>>>>> /var/log/openvswitch/ovn-controller.log >>>>>>>>>>>>>>> 2020-09-11T10:54:07.568Z|00002|reconnect|INFO|unix:/var/run/openvswitch/db.sock: >>>>>>>>>>>>>>> connecting... >>>>>>>>>>>>>>> 2020-09-11T10:54:07.568Z|00003|reconnect|INFO|unix:/var/run/openvswitch/db.sock: >>>>>>>>>>>>>>> connected >>>>>>>>>>>>>>> 2020-09-11T10:54:07.570Z|00004|main|INFO|OVS IDL >>>>>>>>>>>>>>> reconnected, force recompute. >>>>>>>>>>>>>>> 2020-09-11T10:54:07.571Z|00005|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>>>>>>>>>>>> connecting... >>>>>>>>>>>>>>> 2020-09-11T10:54:07.571Z|00006|main|INFO|OVNSB IDL >>>>>>>>>>>>>>> reconnected, force recompute. >>>>>>>>>>>>>>> 2020-09-11T10:54:07.685Z|00007|stream_ssl|WARN|SSL_connect: >>>>>>>>>>>>>>> unexpected SSL connection close >>>>>>>>>>>>>>> 2020-09-11T10:54:07.685Z|00008|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>>>>>>>>>>>> connection attempt failed (Protocol error) >>>>>>>>>>>>>>> 2020-09-11T10:54:08.685Z|00009|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>>>>>>>>>>>> connecting... >>>>>>>>>>>>>>> 2020-09-11T10:54:08.800Z|00010|stream_ssl|WARN|SSL_connect: >>>>>>>>>>>>>>> unexpected SSL connection close >>>>>>>>>>>>>>> 2020-09-11T10:54:08.800Z|00011|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>>>>>>>>>>>> connection attempt failed (Protocol error) >>>>>>>>>>>>>>> 2020-09-11T10:54:08.800Z|00012|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>>>>>>>>>>>> waiting 2 seconds before reconnect >>>>>>>>>>>>>>> 2020-09-11T10:54:10.802Z|00013|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>>>>>>>>>>>> connecting... >>>>>>>>>>>>>>> 2020-09-11T10:54:10.917Z|00014|stream_ssl|WARN|SSL_connect: >>>>>>>>>>>>>>> unexpected SSL connection close >>>>>>>>>>>>>>> 2020-09-11T10:54:10.917Z|00015|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>>>>>>>>>>>> connection attempt failed (Protocol error) >>>>>>>>>>>>>>> 2020-09-11T10:54:10.917Z|00016|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>>>>>>>>>>>> waiting 4 seconds before reconnect >>>>>>>>>>>>>>> 2020-09-11T10:54:14.921Z|00017|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>>>>>>>>>>>> connecting... >>>>>>>>>>>>>>> 2020-09-11T10:54:15.036Z|00018|stream_ssl|WARN|SSL_connect: >>>>>>>>>>>>>>> unexpected SSL connection close >>>>>>>>>>>>>>> 2020-09-11T10:54:15.036Z|00019|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>>>>>>>>>>>> connection attempt failed (Protocol error) >>>>>>>>>>>>>>> 2020-09-11T10:54:15.036Z|00020|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>>>>>>>>>>>> continuing to reconnect in the background but suppressing further logging >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have also done the vdsm-tool ovn-config OVIRT_ENGINE_IP >>>>>>>>>>>>>>> OVIRTMGMT_NETWORK_DC >>>>>>>>>>>>>>> This is how the OVIRT_ENGINE_IP is provided in the ovn >>>>>>>>>>>>>>> controller, i can redo it if you wan. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> After the restart of the ovn-controller the OVIRT ENGINE >>>>>>>>>>>>>>> still shows only two geneve connections one with DC01-host02 and >>>>>>>>>>>>>>> DC02-host01. >>>>>>>>>>>>>>> Chassis "c4b23834-aec7-4bf8-8be7-aa94a50a6144" >>>>>>>>>>>>>>> hostname: "dc02-host01" >>>>>>>>>>>>>>> Encap geneve >>>>>>>>>>>>>>> ip: "DC02-host01_IP" >>>>>>>>>>>>>>> options: {csum="true"} >>>>>>>>>>>>>>> Chassis "be3abcc9-7358-4040-a37b-8d8a782f239c" >>>>>>>>>>>>>>> hostname: "DC01-host02" >>>>>>>>>>>>>>> Encap geneve >>>>>>>>>>>>>>> ip: "DC01-host02" >>>>>>>>>>>>>>> options: {csum="true"} >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I've re-done the vdsm-tool command and nothing changed.... >>>>>>>>>>>>>>> again....with the same errors as the systemctl restart ovn-controller >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Sep 11, 2020 at 1:49 PM Dominik Holler < >>>>>>>>>>>>>>> dholler(a)redhat.com&gt; wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Please include ovirt-users list in your reply, to share >>>>>>>>>>>>>>>> the knowledge and experience with the community! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Sep 11, 2020 at 12:12 PM Konstantinos Betsis < >>>>>>>>>>>>>>>> k.betsis(a)gmail.com&gt; wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Ok below the output per node and DC >>>>>>>>>>>>>>>>> DC01 >>>>>>>>>>>>>>>>> node01 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [root@dc01-node01 ~]# ovs-vsctl --no-wait get open . >>>>>>>>>>>>>>>>> external-ids:ovn-remote >>>>>>>>>>>>>>>>> "ssl:*OVIRT_ENGINE_IP*:6642" >>>>>>>>>>>>>>>>> [root@ dc01-node01 ~]# ovs-vsctl --no-wait get open . >>>>>>>>>>>>>>>>> external-ids:ovn-encap-type >>>>>>>>>>>>>>>>> geneve >>>>>>>>>>>>>>>>> [root@ dc01-node01 ~]# ovs-vsctl --no-wait get open . >>>>>>>>>>>>>>>>> external-ids:ovn-encap-ip >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> "*OVIRTMGMT_IP_DC01-NODE01*" >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> node02 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [root@dc01-node02 ~]# ovs-vsctl --no-wait get open . >>>>>>>>>>>>>>>>> external-ids:ovn-remote >>>>>>>>>>>>>>>>> "ssl:*OVIRT_ENGINE_IP*:6642" >>>>>>>>>>>>>>>>> [root@ dc01-node02 ~]# ovs-vsctl --no-wait get open . >>>>>>>>>>>>>>>>> external-ids:ovn-encap-type >>>>>>>>>>>>>>>>> geneve >>>>>>>>>>>>>>>>> [root@ dc01-node02 ~]# ovs-vsctl --no-wait get open . >>>>>>>>>>>>>>>>> external-ids:ovn-encap-ip >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> "*OVIRTMGMT_IP_DC01-NODE02*" >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> DC02 >>>>>>>>>>>>>>>>> node01 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [root@dc02-node01 ~]# ovs-vsctl --no-wait get open . >>>>>>>>>>>>>>>>> external-ids:ovn-remote >>>>>>>>>>>>>>>>> "ssl:*OVIRT_ENGINE_IP*:6642" >>>>>>>>>>>>>>>>> [root@ dc02-node01 ~]# ovs-vsctl --no-wait get open . >>>>>>>>>>>>>>>>> external-ids:ovn-encap-type >>>>>>>>>>>>>>>>> geneve >>>>>>>>>>>>>>>>> [root@ dc02-node01 ~]# ovs-vsctl --no-wait get open . >>>>>>>>>>>>>>>>> external-ids:ovn-encap-ip >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> "*OVIRTMGMT_IP_DC02-NODE01*" >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Looks good. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> DC01 node01 and node02 share the same VM networks and VMs >>>>>>>>>>>>>>>>> deployed on top of them cannot talk to VM on the other hypervisor. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Maybe there is a hint on ovn-controller.log on dc01-node02 >>>>>>>>>>>>>>>> ? Maybe restarting ovn-controller creates more helpful log messages? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> You can also try restart the ovn configuration on all >>>>>>>>>>>>>>>> hosts by executing >>>>>>>>>>>>>>>> vdsm-tool ovn-config OVIRT_ENGINE_IP LOCAL_OVIRTMGMT_IP >>>>>>>>>>>>>>>> on each host, this would trigger >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://github.com/oVirt/ovirt-provider-ovn/blob/master/driver/scripts/se... >>>>>>>>>>>>>>>> internally. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> So I would expect to see the same output for node01 to >>>>>>>>>>>>>>>>> have a geneve tunnel to node02 and vice versa. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Me too. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Fri, Sep 11, 2020 at 12:14 PM Dominik Holler < >>>>>>>>>>>>>>>>> dholler(a)redhat.com&gt; wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Fri, Sep 11, 2020 at 10:53 AM Konstantinos Betsis < >>>>>>>>>>>>>>>>>> k.betsis(a)gmail.com&gt; wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Dominik >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> OVN is selected as the default network provider on the >>>>>>>>>>>>>>>>>>> clusters and the hosts. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> sounds good. >>>>>>>>>>>>>>>>>> This configuration is required already during the host >>>>>>>>>>>>>>>>>> is added to oVirt Engine, because OVN is configured during this step. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The "ovn-sbctl show" works on the ovirt engine and >>>>>>>>>>>>>>>>>>> shows only two hosts, 1 per DC. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Chassis "c4b23834-aec7-4bf8-8be7-aa94a50a6144" >>>>>>>>>>>>>>>>>>> hostname: "dc01-node02" >>>>>>>>>>>>>>>>>>> Encap geneve >>>>>>>>>>>>>>>>>>> ip: "X.X.X.X" >>>>>>>>>>>>>>>>>>> options: {csum="true"} >>>>>>>>>>>>>>>>>>> Chassis "be3abcc9-7358-4040-a37b-8d8a782f239c" >>>>>>>>>>>>>>>>>>> hostname: "dc02-node1" >>>>>>>>>>>>>>>>>>> Encap geneve >>>>>>>>>>>>>>>>>>> ip: "A.A.A.A" >>>>>>>>>>>>>>>>>>> options: {csum="true"} >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The new node is not listed (dc01-node1). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> When executed on the nodes the same command (ovn-sbctl >>>>>>>>>>>>>>>>>>> show) times-out on all nodes..... >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The output of the >>>>>>>>>>>>>>>>>>> /var/log/openvswitch/ovn-conntroller.log lists on all logs >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 2020-09-11T08:46:55.197Z|07361|stream_ssl|WARN|SSL_connect: >>>>>>>>>>>>>>>>>>> unexpected SSL connection close >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Can you please compare the output of >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> ovs-vsctl --no-wait get open . external-ids:ovn-remote >>>>>>>>>>>>>>>>>> ovs-vsctl --no-wait get open . >>>>>>>>>>>>>>>>>> external-ids:ovn-encap-type >>>>>>>>>>>>>>>>>> ovs-vsctl --no-wait get open . external-ids:ovn-encap-ip >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> of the working hosts, e.g. dc01-node02, and the failing >>>>>>>>>>>>>>>>>> host dc01-node1? >>>>>>>>>>>>>>>>>> This should point us the relevant difference in the >>>>>>>>>>>>>>>>>> configuration. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Please include ovirt-users list in your replay, to share >>>>>>>>>>>>>>>>>> the knowledge and experience with the community. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thank you >>>>>>>>>>>>>>>>>>> Best regards >>>>>>>>>>>>>>>>>>> Konstantinos Betsis >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Fri, Sep 11, 2020 at 11:01 AM Dominik Holler < >>>>>>>>>>>>>>>>>>> dholler(a)redhat.com&gt; wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Thu, Sep 10, 2020 at 6:26 PM Konstantinos B < >>>>>>>>>>>>>>>>>>>> k.betsis(a)gmail.com&gt; wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi all >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> We have a small installation based on OVIRT 4.3. >>>>>>>>>>>>>>>>>>>>> 1 Cluster is based on Centos 7 and the other on OVIRT >>>>>>>>>>>>>>>>>>>>> NG Node image. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> The environment was stable till an upgrade took place >>>>>>>>>>>>>>>>>>>>> a couple of months ago. >>>>>>>>>>>>>>>>>>>>> As such we had to re-install one of the Centos 7 node >>>>>>>>>>>>>>>>>>>>> and start from scratch. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> To trigger the automatic configuration of the host, it >>>>>>>>>>>>>>>>>>>> is required to configure ovirt-provider-ovn as the default network provider >>>>>>>>>>>>>>>>>>>> for the cluster before adding the host to oVirt. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Even though the installation completed successfully >>>>>>>>>>>>>>>>>>>>> and VMs are created, the following are not working as expected: >>>>>>>>>>>>>>>>>>>>> 1. ovn geneve tunnels are not established with the >>>>>>>>>>>>>>>>>>>>> other Centos 7 node in the cluster. >>>>>>>>>>>>>>>>>>>>> 2. Centos 7 node is configured by ovirt engine >>>>>>>>>>>>>>>>>>>>> however no geneve tunnel is established when "ovn-sbctl show" is issued on >>>>>>>>>>>>>>>>>>>>> the engine. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Does "ovn-sbctl show" list the hosts? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> 3. no flows are shown on the engine on port 6642 for >>>>>>>>>>>>>>>>>>>>> the ovs db. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Does anyone have any experience on how to >>>>>>>>>>>>>>>>>>>>> troubleshoot OVN on ovirt? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> /var/log/openvswitch/ovncontroller.log on the host >>>>>>>>>>>>>>>>>>>> should contain a helpful hint. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thank you >>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>>>>>>>>>>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>>>>>>>>>>>>>>>> Privacy Statement: >>>>>>>>>>>>>>>>>>>>> https://www.ovirt.org/privacy-policy.html >>>>>>>>>>>>>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>>>>>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>>>>>>>>>>>>> List Archives: >>>>>>>>>>>>>>>>>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/LBVGLQJBWJF... >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>