[ovirt-users] Re: OVN Geneve tunnels not been established

On Tue, Sep 15, 2020 at 5:11 PM Konstantinos Betsis <k.betsis(a)gmail.com&gt; wrote: > Hi Dominik > > That immediately fixed the geneve tunnels between all hosts. > > thanks for the feedback. > However, the ovn provider is not broken. > After fixing the networks we tried to move a VM to the DC01-host01 so we > powered it down and simply configured it to run on dc01-node01. > > While checking the logs on the ovirt engine i noticed the below: > Failed to synchronize networks of Provider ovirt-provider-ovn. > > The ovn-provider configure on the engine is the below: > Name: ovirt-provider-ovn > Description: oVirt network provider for OVN > Type: External Network Provider > Network Plugin: oVirt Network Provider for OVN > Automatic Synchronization: Checked > Unmanaged: Unchecked > Provider URL: http:localhost:9696 > Requires Authentication: Checked > Username: admin@internal > Password: "The admin password" > Protocol: hTTP > Host Name: dc02-ovirt01 > API Port: 35357 > API Version: v2.0 > Tenant Name: "Empty" > > In the past this was deleted by an engineer and recreated as per the > documentation, and it worked. Do we need to update something due to the SSL > on the ovn? > > Is there a file in /etc/ovirt-provider-ovn/conf.d/ ? engine-setup should have created one. If the file is missing, for testing purposes, you can create a file /etc/ovirt-provider-ovn/conf.d/00-setup-ovirt-provider-ovn-test.conf : [PROVIDER] provider-host=REPLACE_WITH_FQDN [SSL] ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem https-enabled=true [OVN REMOTE] ovn-remote=ssl:127.0.0.1:6641 [AUTH] auth-plugin=auth.plugins.static_token:NoAuthPlugin [NETWORK] port-security-enabled-default=True and restart the ovirt-provider-ovn service. > From the ovn-provider logs the below is generated after a service restart > and when the start VM is triggered > > 2020-09-15 15:07:33,579 root Starting server > 2020-09-15 15:07:33,579 root Version: 1.2.29-1 > 2020-09-15 15:07:33,579 root Build date: 20191217125241 > 2020-09-15 15:07:33,579 root Githash: cb5a80d > 2020-09-15 15:08:26,582 root From: ::ffff:127.0.0.1:59980 Request: GET > /v2.0/ports > 2020-09-15 15:08:26,582 root Could not retrieve schema from tcp: > 127.0.0.1:6641: Unknown error -1 > Traceback (most recent call last): > File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line > 138, in _handle_request > method, path_parts, content > File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", > line 175, in handle_request > return self.call_response_handler(handler, content, parameters) > File "/usr/share/ovirt-provider-ovn/handlers/neutron.py", line 35, in > call_response_handler > with NeutronApi() as ovn_north: > File "/usr/share/ovirt-provider-ovn/neutron/neutron_api.py", line 95, > in __init__ > self.ovsidl, self.idl = ovn_connection.connect() > File "/usr/share/ovirt-provider-ovn/ovn_connection.py", line 46, in > connect > ovnconst.OVN_NORTHBOUND > File > "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", > line 127, in from_server > helper = idlutils.get_schema_helper(connection_string, schema_name) > File > "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", > line 128, in get_schema_helper > 'err': os.strerror(err)}) > Exception: Could not retrieve schema from tcp:127.0.0.1:6641: Unknown > error -1 > > > When i update the ovn provider from the GUI to have > https://localhost:9696/ and HTTPS as the protocol the test fails. > > On Tue, Sep 15, 2020 at 5:35 PM Dominik Holler <dholler(a)redhat.com&gt; > wrote: > >> >> >> On Mon, Sep 14, 2020 at 9:25 AM Konstantinos Betsis <k.betsis(a)gmail.com&gt; >> wrote: >> >>> Hi Dominik >>> >>> When these commands are used on the ovirt-engine host the output is the >>> one depicted in your email. >>> For your reference see also below: >>> >>> [root@ath01-ovirt01 certs]# ovn-nbctl get-ssl >>> Private key: /etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass >>> Certificate: /etc/pki/ovirt-engine/certs/ovn-ndb.cer >>> CA Certificate: /etc/pki/ovirt-engine/ca.pem >>> Bootstrap: false >>> [root@ath01-ovirt01 certs]# ovn-nbctl get-connection >>> ptcp:6641 >>> >>> [root@ath01-ovirt01 certs]# ovn-sbctl get-ssl >>> Private key: /etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass >>> Certificate: /etc/pki/ovirt-engine/certs/ovn-sdb.cer >>> CA Certificate: /etc/pki/ovirt-engine/ca.pem >>> Bootstrap: false >>> [root@ath01-ovirt01 certs]# ovn-sbctl get-connection >>> read-write role="" ptcp:6642 >>> >>> >> ^^^ the line above points to the problem: ovn-central is configured to >> use plain TCP without ssl. >> engine-setup usually configures ovn-central to use SSL. That the files >> /etc/pki/ovirt-engine/keys/ovn-* exist, shows, >> that engine-setup was triggered correctly. Looks like the ovn db was >> dropped somehow, this should not happen. >> This can be fixed manually by executing the following commands on >> engine's machine: >> ovn-nbctl set-ssl /etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass >> /etc/pki/ovirt-engine/certs/ovn-ndb.cer /etc/pki/ovirt-engine/ca.pem >> ovn-nbctl set-connection pssl:6641 >> ovn-sbctl set-ssl /etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass >> /etc/pki/ovirt-engine/certs/ovn-sdb.cer /etc/pki/ovirt-engine/ca.pem >> ovn-sbctl set-connection pssl:6642 >> >> The /var/log/openvswitch/ovn-controller.log on the hosts should tell >> that br-int.mgmt is connected now. >> >> >> >>> [root@ath01-ovirt01 certs]# ls -l /etc/pki/ovirt-engine/keys/ovn-* >>> -rw-r-----. 1 root hugetlbfs 1828 Jun 25 11:08 >>> /etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass >>> -rw-------. 1 root root 2893 Jun 25 11:08 >>> /etc/pki/ovirt-engine/keys/ovn-ndb.p12 >>> -rw-r-----. 1 root hugetlbfs 1828 Jun 25 11:08 >>> /etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass >>> -rw-------. 1 root root 2893 Jun 25 11:08 >>> /etc/pki/ovirt-engine/keys/ovn-sdb.p12 >>> >>> When i try the above commands on the node hosts the following happens: >>> ovn-nbctl get-ssl / get-connection >>> ovn-nbctl: unix:/var/run/openvswitch/ovnnb_db.sock: database connection >>> failed (No such file or directory) >>> The above i believe is expected since no northbound connections should >>> be established from the host nodes. >>> >>> ovn-sbctl get-ssl /get-connection >>> The output is stuck till i terminate it. >>> >>> >> Yes, the ovn-* commands works only on engine's machine, which has the >> role ovn-central. >> On the hosts, there is only the ovn-controller, which connects the >> ovn southbound to openvswitch on the host. >> >> >>> For the requested logs the below are found in the ovsdb-server-sb.log >>> >>> 2020-09-14T07:18:38.187Z|219636|reconnect|WARN|tcp:DC02-host01:33146: >>> connection dropped (Protocol error) >>> 2020-09-14T07:18:41.946Z|219637|reconnect|WARN|tcp:DC01-host01:51188: >>> connection dropped (Protocol error) >>> 2020-09-14T07:18:43.033Z|219638|reconnect|WARN|tcp:DC01-host02:37044: >>> connection dropped (Protocol error) >>> 2020-09-14T07:18:46.198Z|219639|reconnect|WARN|tcp:DC02-host01:33148: >>> connection dropped (Protocol error) >>> 2020-09-14T07:18:50.069Z|219640|jsonrpc|WARN|Dropped 4 log messages in >>> last 12 seconds (most recently, 4 seconds ago) due to excessive rate >>> 2020-09-14T07:18:50.069Z|219641|jsonrpc|WARN|tcp:DC01-host01:51190: >>> error parsing stream: line 0, column 0, byte 0: invalid character U+0016 >>> 2020-09-14T07:18:50.069Z|219642|jsonrpc|WARN|Dropped 4 log messages in >>> last 12 seconds (most recently, 4 seconds ago) due to excessive rate >>> 2020-09-14T07:18:50.069Z|219643|jsonrpc|WARN|tcp:DC01-host01:51190: >>> received SSL data on JSON-RPC channel >>> 2020-09-14T07:18:50.070Z|219644|reconnect|WARN|tcp:DC01-host01:51190: >>> connection dropped (Protocol error) >>> 2020-09-14T07:18:51.147Z|219645|reconnect|WARN|tcp:DC01-host02:37046: >>> connection dropped (Protocol error) >>> 2020-09-14T07:18:54.209Z|219646|reconnect|WARN|tcp:DC02-host01:33150: >>> connection dropped (Protocol error) >>> 2020-09-14T07:18:58.192Z|219647|reconnect|WARN|tcp:DC01-host01:51192: >>> connection dropped (Protocol error) >>> 2020-09-14T07:18:59.262Z|219648|jsonrpc|WARN|Dropped 3 log messages in >>> last 8 seconds (most recently, 1 seconds ago) due to excessive rate >>> 2020-09-14T07:18:59.262Z|219649|jsonrpc|WARN|tcp:DC01-host02:37048: >>> error parsing stream: line 0, column 0, byte 0: invalid character U+0016 >>> 2020-09-14T07:18:59.263Z|219650|jsonrpc|WARN|Dropped 3 log messages in >>> last 8 seconds (most recently, 1 seconds ago) due to excessive rate >>> 2020-09-14T07:18:59.263Z|219651|jsonrpc|WARN|tcp:DC01-host02:37048: >>> received SSL data on JSON-RPC channel >>> 2020-09-14T07:18:59.263Z|219652|reconnect|WARN|tcp:DC01-host02:37048: >>> connection dropped (Protocol error) >>> 2020-09-14T07:19:02.220Z|219653|reconnect|WARN|tcp:DC02-host01:33152: >>> connection dropped (Protocol error) >>> 2020-09-14T07:19:06.316Z|219654|reconnect|WARN|tcp:DC01-host01:51194: >>> connection dropped (Protocol error) >>> 2020-09-14T07:19:07.386Z|219655|reconnect|WARN|tcp:DC01-host02:37050: >>> connection dropped (Protocol error) >>> 2020-09-14T07:19:10.232Z|219656|reconnect|WARN|tcp:DC02-host01:33154: >>> connection dropped (Protocol error) >>> 2020-09-14T07:19:14.439Z|219657|jsonrpc|WARN|Dropped 4 log messages in >>> last 12 seconds (most recently, 4 seconds ago) due to excessive rate >>> 2020-09-14T07:19:14.439Z|219658|jsonrpc|WARN|tcp:DC01-host01:51196: >>> error parsing stream: line 0, column 0, byte 0: invalid character U+0016 >>> 2020-09-14T07:19:14.439Z|219659|jsonrpc|WARN|Dropped 4 log messages in >>> last 12 seconds (most recently, 4 seconds ago) due to excessive rate >>> 2020-09-14T07:19:14.439Z|219660|jsonrpc|WARN|tcp:DC01-host01:51196: >>> received SSL data on JSON-RPC channel >>> 2020-09-14T07:19:14.440Z|219661|reconnect|WARN|tcp:DC01-host01:51196: >>> connection dropped (Protocol error) >>> 2020-09-14T07:19:15.505Z|219662|reconnect|WARN|tcp:DC01-host02:37052: >>> connection dropped (Protocol error) >>> >>> >>> How can we fix these SSL errors? >>> >> >> I addressed this above. >> >> >>> I thought vdsm did the certificate provisioning on the host nodes as to >>> communicate to the engine host node. >>> >>> >> Yes, this seems to work in your scenario, just the SSL configuration on >> the ovn-central was lost. >> >> >>> On Fri, Sep 11, 2020 at 6:39 PM Dominik Holler <dholler(a)redhat.com&gt; >>> wrote: >>> >>>> Looks still like the ovn-controller on the host has problems >>>> communicating with ovn-southbound. >>>> >>>> Are there any hints in /var/log/openvswitch/*.log, >>>> especially in /var/log/openvswitch/ovsdb-server-sb.log ? >>>> >>>> Can you please check the output of >>>> >>>> ovn-nbctl get-ssl >>>> ovn-nbctl get-connection >>>> ovn-sbctl get-ssl >>>> ovn-sbctl get-connection >>>> ls -l /etc/pki/ovirt-engine/keys/ovn-* >>>> >>>> it should be similar to >>>> >>>> [root@ovirt-43 ~]# ovn-nbctl get-ssl >>>> Private key: /etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass >>>> Certificate: /etc/pki/ovirt-engine/certs/ovn-ndb.cer >>>> CA Certificate: /etc/pki/ovirt-engine/ca.pem >>>> Bootstrap: false >>>> [root@ovirt-43 ~]# ovn-nbctl get-connection >>>> pssl:6641:[::] >>>> [root@ovirt-43 ~]# ovn-sbctl get-ssl >>>> Private key: /etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass >>>> Certificate: /etc/pki/ovirt-engine/certs/ovn-sdb.cer >>>> CA Certificate: /etc/pki/ovirt-engine/ca.pem >>>> Bootstrap: false >>>> [root@ovirt-43 ~]# ovn-sbctl get-connection >>>> read-write role="" pssl:6642:[::] >>>> [root@ovirt-43 ~]# ls -l /etc/pki/ovirt-engine/keys/ovn-* >>>> -rw-r-----. 1 root hugetlbfs 1828 Oct 14 2019 >>>> /etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass >>>> -rw-------. 1 root root 2709 Oct 14 2019 >>>> /etc/pki/ovirt-engine/keys/ovn-ndb.p12 >>>> -rw-r-----. 1 root hugetlbfs 1828 Oct 14 2019 >>>> /etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass >>>> -rw-------. 1 root root 2709 Oct 14 2019 >>>> /etc/pki/ovirt-engine/keys/ovn-sdb.p12 >>>> >>>> >>>> >>>> >>>> On Fri, Sep 11, 2020 at 1:10 PM Konstantinos Betsis < >>>> k.betsis(a)gmail.com&gt; wrote: >>>> >>>>> I did a restart of the ovn-controller, this is the output of the >>>>> ovn-controller.log >>>>> >>>>> 2020-09-11T10:54:07.566Z|00001|vlog|INFO|opened log file >>>>> /var/log/openvswitch/ovn-controller.log >>>>> 2020-09-11T10:54:07.568Z|00002|reconnect|INFO|unix:/var/run/openvswitch/db.sock: >>>>> connecting... >>>>> 2020-09-11T10:54:07.568Z|00003|reconnect|INFO|unix:/var/run/openvswitch/db.sock: >>>>> connected >>>>> 2020-09-11T10:54:07.570Z|00004|main|INFO|OVS IDL reconnected, force >>>>> recompute. >>>>> 2020-09-11T10:54:07.571Z|00005|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>> connecting... >>>>> 2020-09-11T10:54:07.571Z|00006|main|INFO|OVNSB IDL reconnected, force >>>>> recompute. >>>>> 2020-09-11T10:54:07.685Z|00007|stream_ssl|WARN|SSL_connect: >>>>> unexpected SSL connection close >>>>> 2020-09-11T10:54:07.685Z|00008|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>> connection attempt failed (Protocol error) >>>>> 2020-09-11T10:54:08.685Z|00009|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>> connecting... >>>>> 2020-09-11T10:54:08.800Z|00010|stream_ssl|WARN|SSL_connect: >>>>> unexpected SSL connection close >>>>> 2020-09-11T10:54:08.800Z|00011|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>> connection attempt failed (Protocol error) >>>>> 2020-09-11T10:54:08.800Z|00012|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>> waiting 2 seconds before reconnect >>>>> 2020-09-11T10:54:10.802Z|00013|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>> connecting... >>>>> 2020-09-11T10:54:10.917Z|00014|stream_ssl|WARN|SSL_connect: >>>>> unexpected SSL connection close >>>>> 2020-09-11T10:54:10.917Z|00015|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>> connection attempt failed (Protocol error) >>>>> 2020-09-11T10:54:10.917Z|00016|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>> waiting 4 seconds before reconnect >>>>> 2020-09-11T10:54:14.921Z|00017|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>> connecting... >>>>> 2020-09-11T10:54:15.036Z|00018|stream_ssl|WARN|SSL_connect: >>>>> unexpected SSL connection close >>>>> 2020-09-11T10:54:15.036Z|00019|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>> connection attempt failed (Protocol error) >>>>> 2020-09-11T10:54:15.036Z|00020|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642: >>>>> continuing to reconnect in the background but suppressing further logging >>>>> >>>>> >>>>> I have also done the vdsm-tool ovn-config OVIRT_ENGINE_IP >>>>> OVIRTMGMT_NETWORK_DC >>>>> This is how the OVIRT_ENGINE_IP is provided in the ovn controller, i >>>>> can redo it if you wan. >>>>> >>>>> After the restart of the ovn-controller the OVIRT ENGINE still shows >>>>> only two geneve connections one with DC01-host02 and DC02-host01. >>>>> Chassis "c4b23834-aec7-4bf8-8be7-aa94a50a6144" >>>>> hostname: "dc02-host01" >>>>> Encap geneve >>>>> ip: "DC02-host01_IP" >>>>> options: {csum="true"} >>>>> Chassis "be3abcc9-7358-4040-a37b-8d8a782f239c" >>>>> hostname: "DC01-host02" >>>>> Encap geneve >>>>> ip: "DC01-host02" >>>>> options: {csum="true"} >>>>> >>>>> I've re-done the vdsm-tool command and nothing changed.... >>>>> again....with the same errors as the systemctl restart ovn-controller >>>>> >>>>> On Fri, Sep 11, 2020 at 1:49 PM Dominik Holler <dholler(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> Please include ovirt-users list in your reply, to share >>>>>> the knowledge and experience with the community! >>>>>> >>>>>> On Fri, Sep 11, 2020 at 12:12 PM Konstantinos Betsis < >>>>>> k.betsis(a)gmail.com&gt; wrote: >>>>>> >>>>>>> Ok below the output per node and DC >>>>>>> DC01 >>>>>>> node01 >>>>>>> >>>>>>> [root@dc01-node01 ~]# ovs-vsctl --no-wait get open . >>>>>>> external-ids:ovn-remote >>>>>>> "ssl:*OVIRT_ENGINE_IP*:6642" >>>>>>> [root@ dc01-node01 ~]# ovs-vsctl --no-wait get open . >>>>>>> external-ids:ovn-encap-type >>>>>>> geneve >>>>>>> [root@ dc01-node01 ~]# ovs-vsctl --no-wait get open . >>>>>>> external-ids:ovn-encap-ip >>>>>>> >>>>>>> "*OVIRTMGMT_IP_DC01-NODE01*" >>>>>>> >>>>>>> node02 >>>>>>> >>>>>>> [root@dc01-node02 ~]# ovs-vsctl --no-wait get open . >>>>>>> external-ids:ovn-remote >>>>>>> "ssl:*OVIRT_ENGINE_IP*:6642" >>>>>>> [root@ dc01-node02 ~]# ovs-vsctl --no-wait get open . >>>>>>> external-ids:ovn-encap-type >>>>>>> geneve >>>>>>> [root@ dc01-node02 ~]# ovs-vsctl --no-wait get open . >>>>>>> external-ids:ovn-encap-ip >>>>>>> >>>>>>> "*OVIRTMGMT_IP_DC01-NODE02*" >>>>>>> >>>>>>> DC02 >>>>>>> node01 >>>>>>> >>>>>>> [root@dc02-node01 ~]# ovs-vsctl --no-wait get open . >>>>>>> external-ids:ovn-remote >>>>>>> "ssl:*OVIRT_ENGINE_IP*:6642" >>>>>>> [root@ dc02-node01 ~]# ovs-vsctl --no-wait get open . >>>>>>> external-ids:ovn-encap-type >>>>>>> geneve >>>>>>> [root@ dc02-node01 ~]# ovs-vsctl --no-wait get open . >>>>>>> external-ids:ovn-encap-ip >>>>>>> >>>>>>> "*OVIRTMGMT_IP_DC02-NODE01*" >>>>>>> >>>>>>> >>>>>> Looks good. >>>>>> >>>>>> >>>>>>> DC01 node01 and node02 share the same VM networks and VMs deployed >>>>>>> on top of them cannot talk to VM on the other hypervisor. >>>>>>> >>>>>> >>>>>> Maybe there is a hint on ovn-controller.log on dc01-node02 ? Maybe >>>>>> restarting ovn-controller creates more helpful log messages? >>>>>> >>>>>> You can also try restart the ovn configuration on all hosts by >>>>>> executing >>>>>> vdsm-tool ovn-config OVIRT_ENGINE_IP LOCAL_OVIRTMGMT_IP >>>>>> on each host, this would trigger >>>>>> >>>>>> https://github.com/oVirt/ovirt-provider-ovn/blob/master/driver/scripts/se... >>>>>> internally. >>>>>> >>>>>> >>>>>>> So I would expect to see the same output for node01 to have a >>>>>>> geneve tunnel to node02 and vice versa. >>>>>>> >>>>>>> >>>>>> Me too. >>>>>> >>>>>> >>>>>>> On Fri, Sep 11, 2020 at 12:14 PM Dominik Holler <dholler(a)redhat.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Sep 11, 2020 at 10:53 AM Konstantinos Betsis < >>>>>>>> k.betsis(a)gmail.com&gt; wrote: >>>>>>>> >>>>>>>>> Hi Dominik >>>>>>>>> >>>>>>>>> OVN is selected as the default network provider on the clusters >>>>>>>>> and the hosts. >>>>>>>>> >>>>>>>>> >>>>>>>> sounds good. >>>>>>>> This configuration is required already during the host is added to >>>>>>>> oVirt Engine, because OVN is configured during this step. >>>>>>>> >>>>>>>> >>>>>>>>> The "ovn-sbctl show" works on the ovirt engine and shows only two >>>>>>>>> hosts, 1 per DC. >>>>>>>>> >>>>>>>>> Chassis "c4b23834-aec7-4bf8-8be7-aa94a50a6144" >>>>>>>>> hostname: "dc01-node02" >>>>>>>>> Encap geneve >>>>>>>>> ip: "X.X.X.X" >>>>>>>>> options: {csum="true"} >>>>>>>>> Chassis "be3abcc9-7358-4040-a37b-8d8a782f239c" >>>>>>>>> hostname: "dc02-node1" >>>>>>>>> Encap geneve >>>>>>>>> ip: "A.A.A.A" >>>>>>>>> options: {csum="true"} >>>>>>>>> >>>>>>>>> >>>>>>>>> The new node is not listed (dc01-node1). >>>>>>>>> >>>>>>>>> When executed on the nodes the same command (ovn-sbctl show) >>>>>>>>> times-out on all nodes..... >>>>>>>>> >>>>>>>>> The output of the /var/log/openvswitch/ovn-conntroller.log lists >>>>>>>>> on all logs >>>>>>>>> >>>>>>>>> 2020-09-11T08:46:55.197Z|07361|stream_ssl|WARN|SSL_connect: >>>>>>>>> unexpected SSL connection close >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> Can you please compare the output of >>>>>>>> >>>>>>>> ovs-vsctl --no-wait get open . external-ids:ovn-remote >>>>>>>> ovs-vsctl --no-wait get open . external-ids:ovn-encap-type >>>>>>>> ovs-vsctl --no-wait get open . external-ids:ovn-encap-ip >>>>>>>> >>>>>>>> of the working hosts, e.g. dc01-node02, and the failing host >>>>>>>> dc01-node1? >>>>>>>> This should point us the relevant difference in the configuration. >>>>>>>> >>>>>>>> Please include ovirt-users list in your replay, to share >>>>>>>> the knowledge and experience with the community. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Thank you >>>>>>>>> Best regards >>>>>>>>> Konstantinos Betsis >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Sep 11, 2020 at 11:01 AM Dominik Holler < >>>>>>>>> dholler(a)redhat.com&gt; wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Sep 10, 2020 at 6:26 PM Konstantinos B < >>>>>>>>>> k.betsis(a)gmail.com&gt; wrote: >>>>>>>>>> >>>>>>>>>>> Hi all >>>>>>>>>>> >>>>>>>>>>> We have a small installation based on OVIRT 4.3. >>>>>>>>>>> 1 Cluster is based on Centos 7 and the other on OVIRT NG Node >>>>>>>>>>> image. >>>>>>>>>>> >>>>>>>>>>> The environment was stable till an upgrade took place a couple >>>>>>>>>>> of months ago. >>>>>>>>>>> As such we had to re-install one of the Centos 7 node and start >>>>>>>>>>> from scratch. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> To trigger the automatic configuration of the host, it is >>>>>>>>>> required to configure ovirt-provider-ovn as the default network provider >>>>>>>>>> for the cluster before adding the host to oVirt. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Even though the installation completed successfully and VMs are >>>>>>>>>>> created, the following are not working as expected: >>>>>>>>>>> 1. ovn geneve tunnels are not established with the other Centos >>>>>>>>>>> 7 node in the cluster. >>>>>>>>>>> 2. Centos 7 node is configured by ovirt engine however no >>>>>>>>>>> geneve tunnel is established when "ovn-sbctl show" is issued on the engine. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Does "ovn-sbctl show" list the hosts? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> 3. no flows are shown on the engine on port 6642 for the ovs db. >>>>>>>>>>> >>>>>>>>>>> Does anyone have any experience on how to troubleshoot OVN on >>>>>>>>>>> ovirt? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> /var/log/openvswitch/ovncontroller.log on the host should >>>>>>>>>> contain a helpful hint. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Thank you >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>>> List Archives: >>>>>>>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/LBVGLQJBWJF... >>>>>>>>>>> >>>>>>>>>>