6:33 a.m.
Sure
I've attached it for easier reference.
On Wed, Sep 30, 2020 at 2:21 PM Dominik Holler <dholler(a)redhat.com> wrote:
On Wed, Sep 30, 2020 at 1:16 PM Konstantinos Betsis <k.betsis(a)gmail.com>
wrote:
> Hi Dominik
>
> The DC01-node02 was formatted and reinstalled and then attached to ovirt
> environment.
> Unfortunately we exhibit the same issue.
> The new DC01-node02 tries to establish geneve tunnels to his own IP.
>
> [root@dc01-node02 ~]# ovs-vsctl show
> eff2663e-cb10-41b0-93ba-605bb5c7bd78
> Bridge br-int
> fail_mode: secure
> Port "ovn-95ccb0-0"
> Interface "ovn-95ccb0-0"
> type: geneve
> options: {csum="true", key=flow,
> remote_ip="dc01-node01_IP"}
> Port "ovn-be3abc-0"
> Interface "ovn-be3abc-0"
> type: geneve
> options: {csum="true", key=flow,
> remote_ip="dc01-node02_IP"}
> Port "ovn-c4b238-0"
> Interface "ovn-c4b238-0"
> type: geneve
> options: {csum="true", key=flow,
> remote_ip="dc02-node01_IP"}
> Port br-int
> Interface br-int
> type: internal
> ovs_version: "2.11.0"
>
>
> Is there a way to fix this on the Ovirt engine since this is where the
> information resides?
> Something is broken there.
>
>
I suspect that there is an inconsistency in the OVN SB DB.
Is there a way to share your /var/lib/openvswitch/ovnsb_db.db with us?
> Thank you
> Best Regards
> Konstantinos Betsis
>
>
> On Wed, Sep 16, 2020, 13:25 Dominik Holler <dholler(a)redhat.com> wrote:
>
>>
>>
>> On Wed, Sep 16, 2020 at 12:15 PM Konstantinos Betsis <k.betsis(a)gmail.com>
>> wrote:
>>
>>> I have a better solution.
>>> I am currently migrating all VMs over to dc01-node01 and then i'll
>>> format it as to fix the partitioning as well.
>>>
>>>
>>
>>> In theory the ovs sbdb will be fixed once it is re-installed....
>>>
>>> If not we can then check if there is a stale entry in the ovirt host
>>> where the sb db is managed.
>>>
>>>
>> Maybe you could ensure that there is no entry anymore during dc01-host02
>> is reinstalling, before the host is added to oVirt again?
>>
>>
>>> Do you agree with this?
>>>
>>>
>> Sounds good, but OVN should be not the reason to reinstall.
>>
>>
>>> On Wed, Sep 16, 2020 at 1:00 PM Dominik Holler <dholler(a)redhat.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> Maybe because of a duplicated entry in the ovn sb db?
>>>> Can you please stop the ovn-ctrontroller on this host, remove the host
>>>> from the ovn sb db, ensure it is gone and restart the ovn-controller on
the
>>>> host?
>>>>
>>>>
>>>> On Wed, Sep 16, 2020 at 11:55 AM Konstantinos Betsis <
>>>> k.betsis(a)gmail.com> wrote:
>>>>
>>>>> Hi Dominik
>>>>>
>>>>> Just saw the below on host dc01-host02
>>>>>
>>>>> ovs-vsctl show
>>>>> f3b13557-dfb4-45a4-b6af-c995ccf68720
>>>>> Bridge br-int
>>>>> Port "ovn-95ccb0-0"
>>>>> Interface "ovn-95ccb0-0"
>>>>> type: geneve
>>>>> options: {csum="true", key=flow,
>>>>> remote_ip="dc01-host01"}
>>>>> Port "vnet10"
>>>>> Interface "vnet10"
>>>>> Port "vnet11"
>>>>> Interface "vnet11"
>>>>> Port "vnet0"
>>>>> Interface "vnet0"
>>>>> Port "vnet9"
>>>>> Interface "vnet9"
>>>>> Port "vnet8"
>>>>> Interface "vnet8"
>>>>> Port br-int
>>>>> Interface br-int
>>>>> type: internal
>>>>> Port "vnet12"
>>>>> Interface "vnet12"
>>>>> Port "ovn-be3abc-0"
>>>>> Interface "ovn-be3abc-0"
>>>>> type: geneve
>>>>> options: {csum="true", key=flow,
>>>>> remote_ip="dc01-host02"}
>>>>> Port "vnet7"
>>>>> Interface "vnet7"
>>>>> Port "ovn-c4b238-0"
>>>>> Interface "ovn-c4b238-0"
>>>>> type: geneve
>>>>> options: {csum="true", key=flow,
>>>>> remote_ip="dc02-host01"}
>>>>> Port "vnet6"
>>>>> Interface "vnet6"
>>>>> ovs_version: "2.11.0"
>>>>>
>>>>>
>>>>> Why would this node establish a geneve tunnel to himself?
>>>>> Other nodes do not exhibit this behavior.
>>>>>
>>>>> On Wed, Sep 16, 2020 at 12:21 PM Konstantinos Betsis <
>>>>> k.betsis(a)gmail.com> wrote:
>>>>>
>>>>>> Hi Dominik
>>>>>>
>>>>>> Below is the output of the ovs-vsctl list interface
>>>>>>
>>>>>> _uuid : bdaf92c1-4389-4ddf-aab0-93975076ebb2
>>>>>> admin_state : up
>>>>>> bfd : {}
>>>>>> bfd_status : {}
>>>>>> cfm_fault : []
>>>>>> cfm_fault_status : []
>>>>>> cfm_flap_count : []
>>>>>> cfm_health : []
>>>>>> cfm_mpid : []
>>>>>> cfm_remote_mpids : []
>>>>>> cfm_remote_opstate : []
>>>>>> duplex : full
>>>>>> error : []
>>>>>> external_ids :
{attached-mac="56:6f:77:61:00:02",
>>>>>> iface-id="5d03a7a5-82a1-40f9-b50c-353a26167fa3",
iface-status=active,
>>>>>> vm-id="e45b7b34-24f2-41ed-b95e-cd1ad532e8d3"}
>>>>>> ifindex : 34
>>>>>> ingress_policing_burst: 0
>>>>>> ingress_policing_rate: 0
>>>>>> lacp_current : []
>>>>>> link_resets : 1
>>>>>> link_speed : 10000000
>>>>>> link_state : up
>>>>>> lldp : {}
>>>>>> mac : []
>>>>>> mac_in_use : "fe:6f:77:61:00:02"
>>>>>> mtu : 1442
>>>>>> mtu_request : []
>>>>>> name : "vnet6"
>>>>>> ofport : 2
>>>>>> ofport_request : []
>>>>>> options : {}
>>>>>> other_config : {}
>>>>>> statistics : {collisions=0, rx_bytes=10828495,
>>>>>> rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0,
rx_over_err=0,
>>>>>> rx_packets=117713, tx_bytes=20771797, tx_dropped=0, tx_errors=0,
>>>>>> tx_packets=106954}
>>>>>> status : {driver_name=tun,
driver_version="1.6",
>>>>>> firmware_version=""}
>>>>>> type : ""
>>>>>>
>>>>>> _uuid : bad80911-3993-4085-a0b0-962b6c9156cd
>>>>>> admin_state : up
>>>>>> bfd : {}
>>>>>> bfd_status : {}
>>>>>> cfm_fault : []
>>>>>> cfm_fault_status : []
>>>>>> cfm_flap_count : []
>>>>>> cfm_health : []
>>>>>> cfm_mpid : []
>>>>>> cfm_remote_mpids : []
>>>>>> cfm_remote_opstate : []
>>>>>> duplex : []
>>>>>> error : []
>>>>>> external_ids : {}
>>>>>> ifindex : 39
>>>>>> ingress_policing_burst: 0
>>>>>> ingress_policing_rate: 0
>>>>>> lacp_current : []
>>>>>> link_resets : 0
>>>>>> link_speed : []
>>>>>> link_state : up
>>>>>> lldp : {}
>>>>>> mac : []
>>>>>> mac_in_use : "fe:37:52:c4:cb:03"
>>>>>> mtu : []
>>>>>> mtu_request : []
>>>>>> name : "ovn-c4b238-0"
>>>>>> ofport : 7
>>>>>> ofport_request : []
>>>>>> options : {csum="true", key=flow,
>>>>>> remote_ip="192.168.121.164"}
>>>>>> other_config : {}
>>>>>> statistics : {rx_bytes=0, rx_packets=0, tx_bytes=0,
>>>>>> tx_packets=0}
>>>>>> status :
{tunnel_egress_iface="ovirtmgmt-ams03",
>>>>>> tunnel_egress_iface_carrier=up}
>>>>>> type : geneve
>>>>>>
>>>>>> _uuid : 8e7705d1-0b9d-4e30-8277-c339e7e1c27a
>>>>>> admin_state : up
>>>>>> bfd : {}
>>>>>> bfd_status : {}
>>>>>> cfm_fault : []
>>>>>> cfm_fault_status : []
>>>>>> cfm_flap_count : []
>>>>>> cfm_health : []
>>>>>> cfm_mpid : []
>>>>>> cfm_remote_mpids : []
>>>>>> cfm_remote_opstate : []
>>>>>> duplex : full
>>>>>> error : []
>>>>>> external_ids :
{attached-mac="56:6f:77:61:00:0d",
>>>>>> iface-id="b7ff5f2b-4bb4-4250-8ad8-8a7e19d2b4c7",
iface-status=active,
>>>>>> vm-id="8d73f333-bca4-4b32-9b87-2e7ee07eda84"}
>>>>>> ifindex : 28
>>>>>> ingress_policing_burst: 0
>>>>>> ingress_policing_rate: 0
>>>>>> lacp_current : []
>>>>>> link_resets : 1
>>>>>> link_speed : 10000000
>>>>>> link_state : up
>>>>>> lldp : {}
>>>>>> mac : []
>>>>>> mac_in_use : "fe:6f:77:61:00:0d"
>>>>>> mtu : 1442
>>>>>> mtu_request : []
>>>>>> name : "vnet0"
>>>>>> ofport : 1
>>>>>> ofport_request : []
>>>>>> options : {}
>>>>>> other_config : {}
>>>>>> statistics : {collisions=0, rx_bytes=20609787,
>>>>>> rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0,
rx_over_err=0,
>>>>>> rx_packets=104535, tx_bytes=10830007, tx_dropped=0, tx_errors=0,
>>>>>> tx_packets=117735}
>>>>>> status : {driver_name=tun,
driver_version="1.6",
>>>>>> firmware_version=""}
>>>>>> type : ""
>>>>>>
>>>>>> _uuid : 86dcc68a-63e4-4445-9373-81c1f4502c17
>>>>>> admin_state : up
>>>>>> bfd : {}
>>>>>> bfd_status : {}
>>>>>> cfm_fault : []
>>>>>> cfm_fault_status : []
>>>>>> cfm_flap_count : []
>>>>>> cfm_health : []
>>>>>> cfm_mpid : []
>>>>>> cfm_remote_mpids : []
>>>>>> cfm_remote_opstate : []
>>>>>> duplex : full
>>>>>> error : []
>>>>>> external_ids :
{attached-mac="56:6f:77:61:00:10",
>>>>>> iface-id="4e8d5636-4110-41b2-906d-f9b04c2e62cd",
iface-status=active,
>>>>>> vm-id="9a002a9b-5f09-4def-a531-d50ff683470b"}
>>>>>> ifindex : 40
>>>>>> ingress_policing_burst: 0
>>>>>> ingress_policing_rate: 0
>>>>>> lacp_current : []
>>>>>> link_resets : 1
>>>>>> link_speed : 10000000
>>>>>> link_state : up
>>>>>> lldp : {}
>>>>>> mac : []
>>>>>> mac_in_use : "fe:6f:77:61:00:10"
>>>>>> mtu : 1442
>>>>>> mtu_request : []
>>>>>> name : "vnet11"
>>>>>> ofport : 10
>>>>>> ofport_request : []
>>>>>> options : {}
>>>>>> other_config : {}
>>>>>> statistics : {collisions=0, rx_bytes=3311352,
rx_crc_err=0,
>>>>>> rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0,
rx_packets=51012,
>>>>>> tx_bytes=5514116, tx_dropped=0, tx_errors=0, tx_packets=103456}
>>>>>> status : {driver_name=tun,
driver_version="1.6",
>>>>>> firmware_version=""}
>>>>>> type : ""
>>>>>>
>>>>>> _uuid : e8d5e4a2-b9a0-4146-8d98-34713cb443de
>>>>>> admin_state : up
>>>>>> bfd : {}
>>>>>> bfd_status : {}
>>>>>> cfm_fault : []
>>>>>> cfm_fault_status : []
>>>>>> cfm_flap_count : []
>>>>>> cfm_health : []
>>>>>> cfm_mpid : []
>>>>>> cfm_remote_mpids : []
>>>>>> cfm_remote_opstate : []
>>>>>> duplex : full
>>>>>> error : []
>>>>>> external_ids :
{attached-mac="56:6f:77:61:00:15",
>>>>>> iface-id="b88de6e4-6d77-4e42-b734-4cc676728910",
iface-status=active,
>>>>>> vm-id="e45b7b34-24f2-41ed-b95e-cd1ad532e8d3"}
>>>>>> ifindex : 37
>>>>>> ingress_policing_burst: 0
>>>>>> ingress_policing_rate: 0
>>>>>> lacp_current : []
>>>>>> link_resets : 1
>>>>>> link_speed : 10000000
>>>>>> link_state : up
>>>>>> lldp : {}
>>>>>> mac : []
>>>>>> mac_in_use : "fe:6f:77:61:00:15"
>>>>>> mtu : 1442
>>>>>> mtu_request : []
>>>>>> name : "vnet9"
>>>>>> ofport : 5
>>>>>> ofport_request : []
>>>>>> options : {}
>>>>>> other_config : {}
>>>>>> statistics : {collisions=0, rx_bytes=180, rx_crc_err=0,
>>>>>> rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0,
rx_packets=2,
>>>>>> tx_bytes=4500, tx_dropped=0, tx_errors=0, tx_packets=74}
>>>>>> status : {driver_name=tun,
driver_version="1.6",
>>>>>> firmware_version=""}
>>>>>> type : ""
>>>>>>
>>>>>> _uuid : 6a2974b3-cd72-4688-a630-0a7e9c779b21
>>>>>> admin_state : up
>>>>>> bfd : {}
>>>>>> bfd_status : {}
>>>>>> cfm_fault : []
>>>>>> cfm_fault_status : []
>>>>>> cfm_flap_count : []
>>>>>> cfm_health : []
>>>>>> cfm_mpid : []
>>>>>> cfm_remote_mpids : []
>>>>>> cfm_remote_opstate : []
>>>>>> duplex : full
>>>>>> error : []
>>>>>> external_ids :
{attached-mac="56:6f:77:61:00:17",
>>>>>> iface-id="64681036-26e2-41d7-b73f-ab5302610145",
iface-status=active,
>>>>>> vm-id="bf0dc78c-dad5-41a0-914c-ae0da0f9a388"}
>>>>>> ifindex : 41
>>>>>> ingress_policing_burst: 0
>>>>>> ingress_policing_rate: 0
>>>>>> lacp_current : []
>>>>>> link_resets : 1
>>>>>> link_speed : 10000000
>>>>>> link_state : up
>>>>>> lldp : {}
>>>>>> mac : []
>>>>>> mac_in_use : "fe:6f:77:61:00:17"
>>>>>> mtu : 1442
>>>>>> mtu_request : []
>>>>>> name : "vnet12"
>>>>>> ofport : 11
>>>>>> ofport_request : []
>>>>>> options : {}
>>>>>> other_config : {}
>>>>>> statistics : {collisions=0, rx_bytes=5513640,
rx_crc_err=0,
>>>>>> rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0,
>>>>>> rx_packets=103450, tx_bytes=3311868, tx_dropped=0, tx_errors=0,
>>>>>> tx_packets=51018}
>>>>>> status : {driver_name=tun,
driver_version="1.6",
>>>>>> firmware_version=""}
>>>>>> type : ""
>>>>>>
>>>>>> _uuid : 44498e54-f122-41a0-a41a-7a88ba2dba9b
>>>>>> admin_state : down
>>>>>> bfd : {}
>>>>>> bfd_status : {}
>>>>>> cfm_fault : []
>>>>>> cfm_fault_status : []
>>>>>> cfm_flap_count : []
>>>>>> cfm_health : []
>>>>>> cfm_mpid : []
>>>>>> cfm_remote_mpids : []
>>>>>> cfm_remote_opstate : []
>>>>>> duplex : []
>>>>>> error : []
>>>>>> external_ids : {}
>>>>>> ifindex : 7
>>>>>> ingress_policing_burst: 0
>>>>>> ingress_policing_rate: 0
>>>>>> lacp_current : []
>>>>>> link_resets : 0
>>>>>> link_speed : []
>>>>>> link_state : down
>>>>>> lldp : {}
>>>>>> mac : []
>>>>>> mac_in_use : "32:0a:69:67:07:4f"
>>>>>> mtu : 1442
>>>>>> mtu_request : []
>>>>>> name : br-int
>>>>>> ofport : 65534
>>>>>> ofport_request : []
>>>>>> options : {}
>>>>>> other_config : {}
>>>>>> statistics : {collisions=0, rx_bytes=0, rx_crc_err=0,
>>>>>> rx_dropped=326, rx_errors=0, rx_frame_err=0, rx_over_err=0,
rx_packets=0,
>>>>>> tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0}
>>>>>> status : {driver_name=openvswitch}
>>>>>> type : internal
>>>>>>
>>>>>> _uuid : e2114584-8ceb-43d6-817b-e457738ead8a
>>>>>> admin_state : up
>>>>>> bfd : {}
>>>>>> bfd_status : {}
>>>>>> cfm_fault : []
>>>>>> cfm_fault_status : []
>>>>>> cfm_flap_count : []
>>>>>> cfm_health : []
>>>>>> cfm_mpid : []
>>>>>> cfm_remote_mpids : []
>>>>>> cfm_remote_opstate : []
>>>>>> duplex : full
>>>>>> error : []
>>>>>> external_ids :
{attached-mac="56:6f:77:61:00:03",
>>>>>> iface-id="16162721-c815-4cd8-ab57-f22e6e482c7f",
iface-status=active,
>>>>>> vm-id="e45b7b34-24f2-41ed-b95e-cd1ad532e8d3"}
>>>>>> ifindex : 35
>>>>>> ingress_policing_burst: 0
>>>>>> ingress_policing_rate: 0
>>>>>> lacp_current : []
>>>>>> link_resets : 1
>>>>>> link_speed : 10000000
>>>>>> link_state : up
>>>>>> lldp : {}
>>>>>> mac : []
>>>>>> mac_in_use : "fe:6f:77:61:00:03"
>>>>>> mtu : 1442
>>>>>> mtu_request : []
>>>>>> name : "vnet7"
>>>>>> ofport : 3
>>>>>> ofport_request : []
>>>>>> options : {}
>>>>>> other_config : {}
>>>>>> statistics : {collisions=0, rx_bytes=180, rx_crc_err=0,
>>>>>> rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0,
rx_packets=2,
>>>>>> tx_bytes=4730, tx_dropped=0, tx_errors=0, tx_packets=77}
>>>>>> status : {driver_name=tun,
driver_version="1.6",
>>>>>> firmware_version=""}
>>>>>> type : ""
>>>>>>
>>>>>> _uuid : ee16943e-d145-4080-893f-464098a6388f
>>>>>> admin_state : up
>>>>>> bfd : {}
>>>>>> bfd_status : {}
>>>>>> cfm_fault : []
>>>>>> cfm_fault_status : []
>>>>>> cfm_flap_count : []
>>>>>> cfm_health : []
>>>>>> cfm_mpid : []
>>>>>> cfm_remote_mpids : []
>>>>>> cfm_remote_opstate : []
>>>>>> duplex : []
>>>>>> error : []
>>>>>> external_ids : {}
>>>>>> ifindex : 39
>>>>>> ingress_policing_burst: 0
>>>>>> ingress_policing_rate: 0
>>>>>> lacp_current : []
>>>>>> link_resets : 0
>>>>>> link_speed : []
>>>>>> link_state : up
>>>>>> lldp : {}
>>>>>> mac : []
>>>>>> mac_in_use : "1e:50:3f:a8:42:d1"
>>>>>> mtu : []
>>>>>> mtu_request : []
>>>>>> name : "ovn-be3abc-0"
>>>>>> ofport : 8
>>>>>> ofport_request : []
>>>>>> options : {csum="true", key=flow,
>>>>>> remote_ip="DC01-host02"}
>>>>>> other_config : {}
>>>>>> statistics : {rx_bytes=0, rx_packets=0, tx_bytes=0,
>>>>>> tx_packets=0}
>>>>>> status :
{tunnel_egress_iface="ovirtmgmt-ams03",
>>>>>> tunnel_egress_iface_carrier=up}
>>>>>> type : geneve
>>>>>>
>>>>>> _uuid : 86a229be-373e-4c43-b2f1-6190523ed73a
>>>>>> admin_state : up
>>>>>> bfd : {}
>>>>>> bfd_status : {}
>>>>>> cfm_fault : []
>>>>>> cfm_fault_status : []
>>>>>> cfm_flap_count : []
>>>>>> cfm_health : []
>>>>>> cfm_mpid : []
>>>>>> cfm_remote_mpids : []
>>>>>> cfm_remote_opstate : []
>>>>>> duplex : full
>>>>>> error : []
>>>>>> external_ids :
{attached-mac="56:6f:77:61:00:1c",
>>>>>> iface-id="12d829c3-64eb-44bc-a0bd-d7219991f35f",
iface-status=active,
>>>>>> vm-id="e45b7b34-24f2-41ed-b95e-cd1ad532e8d3"}
>>>>>> ifindex : 38
>>>>>> ingress_policing_burst: 0
>>>>>> ingress_policing_rate: 0
>>>>>> lacp_current : []
>>>>>> link_resets : 1
>>>>>> link_speed : 10000000
>>>>>> link_state : up
>>>>>> lldp : {}
>>>>>> mac : []
>>>>>> mac_in_use : "fe:6f:77:61:00:1c"
>>>>>> mtu : 1442
>>>>>> mtu_request : []
>>>>>> name : "vnet10"
>>>>>> ofport : 6
>>>>>> ofport_request : []
>>>>>> options : {}
>>>>>> other_config : {}
>>>>>> statistics : {collisions=0, rx_bytes=117912,
rx_crc_err=0,
>>>>>> rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0,
rx_packets=2195,
>>>>>> tx_bytes=4204, tx_dropped=0, tx_errors=0, tx_packets=66}
>>>>>> status : {driver_name=tun,
driver_version="1.6",
>>>>>> firmware_version=""}
>>>>>> type : ""
>>>>>>
>>>>>> _uuid : fa4b8d96-bffe-4b56-930e-0e7fcc5f68ac
>>>>>> admin_state : up
>>>>>> bfd : {}
>>>>>> bfd_status : {}
>>>>>> cfm_fault : []
>>>>>> cfm_fault_status : []
>>>>>> cfm_flap_count : []
>>>>>> cfm_health : []
>>>>>> cfm_mpid : []
>>>>>> cfm_remote_mpids : []
>>>>>> cfm_remote_opstate : []
>>>>>> duplex : []
>>>>>> error : []
>>>>>> external_ids : {}
>>>>>> ifindex : 39
>>>>>> ingress_policing_burst: 0
>>>>>> ingress_policing_rate: 0
>>>>>> lacp_current : []
>>>>>> link_resets : 0
>>>>>> link_speed : []
>>>>>> link_state : up
>>>>>> lldp : {}
>>>>>> mac : []
>>>>>> mac_in_use : "7a:28:24:eb:ec:d2"
>>>>>> mtu : []
>>>>>> mtu_request : []
>>>>>> name : "ovn-95ccb0-0"
>>>>>> ofport : 9
>>>>>> ofport_request : []
>>>>>> options : {csum="true", key=flow,
>>>>>> remote_ip="DC01-host01"}
>>>>>> other_config : {}
>>>>>> statistics : {rx_bytes=0, rx_packets=0,
tx_bytes=12840478,
>>>>>> tx_packets=224029}
>>>>>> status :
{tunnel_egress_iface="ovirtmgmt-ams03",
>>>>>> tunnel_egress_iface_carrier=up}
>>>>>> type : geneve
>>>>>>
>>>>>> _uuid : 5e3df5c7-958c-491d-8d41-0ae83c613f1d
>>>>>> admin_state : up
>>>>>> bfd : {}
>>>>>> bfd_status : {}
>>>>>> cfm_fault : []
>>>>>> cfm_fault_status : []
>>>>>> cfm_flap_count : []
>>>>>> cfm_health : []
>>>>>> cfm_mpid : []
>>>>>> cfm_remote_mpids : []
>>>>>> cfm_remote_opstate : []
>>>>>> duplex : full
>>>>>> error : []
>>>>>> external_ids :
{attached-mac="56:6f:77:61:00:06",
>>>>>> iface-id="9a6cc189-0934-4468-97ae-09f90fa4598d",
iface-status=active,
>>>>>> vm-id="e45b7b34-24f2-41ed-b95e-cd1ad532e8d3"}
>>>>>> ifindex : 36
>>>>>> ingress_policing_burst: 0
>>>>>> ingress_policing_rate: 0
>>>>>> lacp_current : []
>>>>>> link_resets : 1
>>>>>> link_speed : 10000000
>>>>>> link_state : up
>>>>>> lldp : {}
>>>>>> mac : []
>>>>>> mac_in_use : "fe:6f:77:61:00:06"
>>>>>> mtu : 1442
>>>>>> mtu_request : []
>>>>>> name : "vnet8"
>>>>>> ofport : 4
>>>>>> ofport_request : []
>>>>>> options : {}
>>>>>> other_config : {}
>>>>>> statistics : {collisions=0, rx_bytes=180, rx_crc_err=0,
>>>>>> rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0,
rx_packets=2,
>>>>>> tx_bytes=8829812, tx_dropped=0, tx_errors=0, tx_packets=154540}
>>>>>> status : {driver_name=tun,
driver_version="1.6",
>>>>>> firmware_version=""}
>>>>>> type : ""
>>>>>>
>>>>>>
>>>>>> I've identified which VMs have these MAC addresses but i do
not see
>>>>>> any "conflict" with any other VM's MAC address.
>>>>>>
>>>>>> I really do not understand why these will create a conflict.
>>>>>>
>>>>>> On Wed, Sep 16, 2020 at 12:06 PM Dominik Holler
<dholler(a)redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Sep 15, 2020 at 6:53 PM Konstantinos Betsis <
>>>>>>> k.betsis(a)gmail.com> wrote:
>>>>>>>
>>>>>>>> So a new test-net was created under DC01 and was depicted
in the
>>>>>>>> networks tab under both DC01 and DC02.
>>>>>>>> I believe for some reason networks are duplicated in DCs,
maybe
>>>>>>>> for future use??? Don't know.
>>>>>>>> If one tries to delete the network from the other DC it
gets an
>>>>>>>> error, while if deleted from the once initially created
it gets deleted
>>>>>>>> from both.
>>>>>>>>
>>>>>>>>
>>>>>>> In oVirt a logical network is an entity in a data center. If
the
>>>>>>> automatic synchronization is enabled on the
ovirt-provider-ovn entity in
>>>>>>> oVirt Engine, the OVN networks are reflected to all data
centers. If you do
>>>>>>> not like this, you can disable the automatic synchronization
of the
>>>>>>> ovirt-provider-ovn in Admin Portal.
>>>>>>>
>>>>>>>
>>>>>>>> From the DC01-node02 i get the following errors:
>>>>>>>>
>>>>>>>> 2020-09-15T16:48:49.904Z|22748|main|INFO|OVNSB commit
failed,
>>>>>>>> force recompute next time.
>>>>>>>> 2020-09-15T16:48:49.905Z|22749|binding|INFO|Claiming
lport
>>>>>>>> 9a6cc189-0934-4468-97ae-09f90fa4598d for this chassis.
>>>>>>>>
2020-09-15T16:48:49.905Z|22750|binding|INFO|9a6cc189-0934-4468-97ae-09f90fa4598d:
>>>>>>>> Claiming 56:6f:77:61:00:06
>>>>>>>> 2020-09-15T16:48:49.905Z|22751|binding|INFO|Claiming
lport
>>>>>>>> 16162721-c815-4cd8-ab57-f22e6e482c7f for this chassis.
>>>>>>>>
2020-09-15T16:48:49.905Z|22752|binding|INFO|16162721-c815-4cd8-ab57-f22e6e482c7f:
>>>>>>>> Claiming 56:6f:77:61:00:03
>>>>>>>> 2020-09-15T16:48:49.905Z|22753|binding|INFO|Claiming
lport
>>>>>>>> b88de6e4-6d77-4e42-b734-4cc676728910 for this chassis.
>>>>>>>>
2020-09-15T16:48:49.905Z|22754|binding|INFO|b88de6e4-6d77-4e42-b734-4cc676728910:
>>>>>>>> Claiming 56:6f:77:61:00:15
>>>>>>>> 2020-09-15T16:48:49.905Z|22755|binding|INFO|Claiming
lport
>>>>>>>> b7ff5f2b-4bb4-4250-8ad8-8a7e19d2b4c7 for this chassis.
>>>>>>>>
2020-09-15T16:48:49.905Z|22756|binding|INFO|b7ff5f2b-4bb4-4250-8ad8-8a7e19d2b4c7:
>>>>>>>> Claiming 56:6f:77:61:00:0d
>>>>>>>> 2020-09-15T16:48:49.905Z|22757|binding|INFO|Claiming
lport
>>>>>>>> 5d03a7a5-82a1-40f9-b50c-353a26167fa3 for this chassis.
>>>>>>>>
2020-09-15T16:48:49.905Z|22758|binding|INFO|5d03a7a5-82a1-40f9-b50c-353a26167fa3:
>>>>>>>> Claiming 56:6f:77:61:00:02
>>>>>>>> 2020-09-15T16:48:49.905Z|22759|binding|INFO|Claiming
lport
>>>>>>>> 12d829c3-64eb-44bc-a0bd-d7219991f35f for this chassis.
>>>>>>>>
2020-09-15T16:48:49.905Z|22760|binding|INFO|12d829c3-64eb-44bc-a0bd-d7219991f35f:
>>>>>>>> Claiming 56:6f:77:61:00:1c
>>>>>>>> 2020-09-15T16:48:49.959Z|22761|main|INFO|OVNSB commit
failed,
>>>>>>>> force recompute next time.
>>>>>>>> 2020-09-15T16:48:49.960Z|22762|binding|INFO|Claiming
lport
>>>>>>>> 9a6cc189-0934-4468-97ae-09f90fa4598d for this chassis.
>>>>>>>>
2020-09-15T16:48:49.960Z|22763|binding|INFO|9a6cc189-0934-4468-97ae-09f90fa4598d:
>>>>>>>> Claiming 56:6f:77:61:00:06
>>>>>>>> 2020-09-15T16:48:49.960Z|22764|binding|INFO|Claiming
lport
>>>>>>>> 16162721-c815-4cd8-ab57-f22e6e482c7f for this chassis.
>>>>>>>>
2020-09-15T16:48:49.960Z|22765|binding|INFO|16162721-c815-4cd8-ab57-f22e6e482c7f:
>>>>>>>> Claiming 56:6f:77:61:00:03
>>>>>>>> 2020-09-15T16:48:49.960Z|22766|binding|INFO|Claiming
lport
>>>>>>>> b88de6e4-6d77-4e42-b734-4cc676728910 for this chassis.
>>>>>>>>
2020-09-15T16:48:49.960Z|22767|binding|INFO|b88de6e4-6d77-4e42-b734-4cc676728910:
>>>>>>>> Claiming 56:6f:77:61:00:15
>>>>>>>> 2020-09-15T16:48:49.960Z|22768|binding|INFO|Claiming
lport
>>>>>>>> b7ff5f2b-4bb4-4250-8ad8-8a7e19d2b4c7 for this chassis.
>>>>>>>>
2020-09-15T16:48:49.960Z|22769|binding|INFO|b7ff5f2b-4bb4-4250-8ad8-8a7e19d2b4c7:
>>>>>>>> Claiming 56:6f:77:61:00:0d
>>>>>>>> 2020-09-15T16:48:49.960Z|22770|binding|INFO|Claiming
lport
>>>>>>>> 5d03a7a5-82a1-40f9-b50c-353a26167fa3 for this chassis.
>>>>>>>>
2020-09-15T16:48:49.960Z|22771|binding|INFO|5d03a7a5-82a1-40f9-b50c-353a26167fa3:
>>>>>>>> Claiming 56:6f:77:61:00:02
>>>>>>>> 2020-09-15T16:48:49.960Z|22772|binding|INFO|Claiming
lport
>>>>>>>> 12d829c3-64eb-44bc-a0bd-d7219991f35f for this chassis.
>>>>>>>>
2020-09-15T16:48:49.960Z|22773|binding|INFO|12d829c3-64eb-44bc-a0bd-d7219991f35f:
>>>>>>>> Claiming 56:6f:77:61:00:1c
>>>>>>>>
>>>>>>>>
>>>>>>>> And this repeats forever.
>>>>>>>>
>>>>>>>>
>>>>>>> Looks like the southbound db is confused.
>>>>>>>
>>>>>>> Can you try to delete all chassis listed by
>>>>>>> sudo ovn-sbctl show
>>>>>>> via
>>>>>>> sudo /usr/share/ovirt-provider-ovn/scripts/remove_chassis.sh
>>>>>>> dev-host0
>>>>>>> ?
>>>>>>> if the script remove_chassis.sh is not installed, you can
use
>>>>>>>
>>>>>>>
https://github.com/oVirt/ovirt-provider-ovn/blob/master/provider/scripts/...
>>>>>>> instead.
>>>>>>>
>>>>>>> Can you please also share the output of
>>>>>>> ovs-vsctl list Interface
>>>>>>> on the host which produced the logfile above?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> The connections to ovn-sbctl is ok and the geneve tunnels
are
>>>>>>>> depicted under ovs-vsctl ok.
>>>>>>>> VMs still not able to ping each other.
>>>>>>>>
>>>>>>>> On Tue, Sep 15, 2020 at 7:22 PM Dominik Holler
<dholler(a)redhat.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Sep 15, 2020 at 6:18 PM Konstantinos Betsis
<
>>>>>>>>> k.betsis(a)gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Dominik
>>>>>>>>>>
>>>>>>>>>> Fixed the issue.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I believe the
/etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf
>>>>>>>>>> needed update also.
>>>>>>>>>> The package is upgraded to the latest version.
>>>>>>>>>>
>>>>>>>>>> Once the provider was updated with the following
it functioned
>>>>>>>>>> perfectly:
>>>>>>>>>>
>>>>>>>>>> Name: ovirt-provider-ovn
>>>>>>>>>> Description: oVirt network provider for OVN
>>>>>>>>>> Type: External Network Provider
>>>>>>>>>> Network Plugin: oVirt Network Provider for OVN
>>>>>>>>>> Automatic Synchronization: Checked
>>>>>>>>>> Unmanaged: Unchecked
>>>>>>>>>> Provider URL:
https:dc02-ovirt01.testdomain.com:9696
>>>>>>>>>> Requires Authentication: Checked
>>>>>>>>>> Username: admin@internal
>>>>>>>>>> Password: "The admin password"
>>>>>>>>>> Protocol: HTTPS
>>>>>>>>>> Host Name: dc02-ovirt01.testdomain.com
>>>>>>>>>> API Port: 35357
>>>>>>>>>> API Version: v2.0
>>>>>>>>>> Tenant Name: "Empty"
>>>>>>>>>>
>>>>>>>>>> For some reason the TLS certificate was in
conflict with the ovn
>>>>>>>>>> provider details, i would bet the
"host" entry.
>>>>>>>>>>
>>>>>>>>>> So now geneve tunnels are established.
>>>>>>>>>> OVN provider is working.
>>>>>>>>>>
>>>>>>>>>> But VMs still do not communicated on the same VM
network
>>>>>>>>>> spanning different hosts.
>>>>>>>>>>
>>>>>>>>>> So if we have a VM network test-net on both
dc01-host01 and
>>>>>>>>>> dc01-host02 and each host has a VM with IP
addresses on the same network,
>>>>>>>>>> VMs on the same VM network should communicate
directly.
>>>>>>>>>> But traffic does not reach each other.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Can you create a new external network, with port
security
>>>>>>>>> disabled, and an IPv4 subnet?
>>>>>>>>> If the VMs get an IP address via DHCP, ovn is
working, and should
>>>>>>>>> be able to ping each other, too.
>>>>>>>>> If not, there should be a helpful entry in the
ovn-controller.log
>>>>>>>>> of the host the VM is running.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Tue, Sep 15, 2020 at 7:07 PM Dominik Holler
<
>>>>>>>>>> dholler(a)redhat.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Can you try again with:
>>>>>>>>>>>
>>>>>>>>>>> [OVN REMOTE]
>>>>>>>>>>> ovn-remote=ssl:127.0.0.1:6641
>>>>>>>>>>> [SSL]
>>>>>>>>>>> https-enabled=false
>>>>>>>>>>> ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem
>>>>>>>>>>>
ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer
>>>>>>>>>>>
>>>>>>>>>>>
ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass
>>>>>>>>>>> [OVIRT]
>>>>>>>>>>> ovirt-sso-client-secret=*random_test*
>>>>>>>>>>>
ovirt-host=https://dc02-ovirt01.testdomain.com:443
>>>>>>>>>>> <https://dc02-ovirt01.testdomain.com/>
>>>>>>>>>>> ovirt-sso-client-id=ovirt-provider-ovn
>>>>>>>>>>>
ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem
>>>>>>>>>>> [NETWORK]
>>>>>>>>>>> port-security-enabled-default=True
>>>>>>>>>>> [PROVIDER]
>>>>>>>>>>>
>>>>>>>>>>> provider-host=dc02-ovirt01.testdomain.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Please note that the should match the HTTP or
HTTPS in the of
>>>>>>>>>>> the ovirt-prover-ovn configuration in oVirt
Engine.
>>>>>>>>>>> So if the ovirt-provider-ovn entity in Engine
is on HTTP, the
>>>>>>>>>>> config file should use
>>>>>>>>>>> https-enabled=false
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 15, 2020 at 5:56 PM Konstantinos
Betsis <
>>>>>>>>>>> k.betsis(a)gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> This is the updated one:
>>>>>>>>>>>>
>>>>>>>>>>>> # This file is automatically generated by
engine-setup. Please
>>>>>>>>>>>> do not edit manually
>>>>>>>>>>>> [OVN REMOTE]
>>>>>>>>>>>> ovn-remote=ssl:127.0.0.1:6641
>>>>>>>>>>>> [SSL]
>>>>>>>>>>>> https-enabled=true
>>>>>>>>>>>>
ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem
>>>>>>>>>>>>
>>>>>>>>>>>>
ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer
>>>>>>>>>>>>
>>>>>>>>>>>>
ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass
>>>>>>>>>>>> [OVIRT]
>>>>>>>>>>>> ovirt-sso-client-secret=*random_text*
>>>>>>>>>>>>
ovirt-host=https://dc02-ovirt01.testdomain.com:443
>>>>>>>>>>>> ovirt-sso-client-id=ovirt-provider-ovn
>>>>>>>>>>>>
ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem
>>>>>>>>>>>> [NETWORK]
>>>>>>>>>>>> port-security-enabled-default=True
>>>>>>>>>>>> [PROVIDER]
>>>>>>>>>>>>
provider-host=dc02-ovirt01.testdomain.com
>>>>>>>>>>>> [AUTH]
>>>>>>>>>>>>
auth-plugin=auth.plugins.static_token:NoAuthPlugin
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> However, it still does not connect.
>>>>>>>>>>>> It prompts for the certificate but then
fails and prompts to
>>>>>>>>>>>> see the log but the
ovirt-provider-ovn.log does not list anything.
>>>>>>>>>>>>
>>>>>>>>>>>> Yes we've got ovirt for about a year
now from about version 4.1
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> This might explain the trouble. Upgrade of
ovirt-provider-ovn
>>>>>>>>>>> should work flawlessly starting from oVirt
4.2.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Sep 15, 2020 at 6:44 PM Dominik
Holler <
>>>>>>>>>>>> dholler(a)redhat.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Sep 15, 2020 at 5:34 PM
Konstantinos Betsis <
>>>>>>>>>>>>> k.betsis(a)gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> There is a file with the below
entries
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Impressive, do you know when this
config file was created and
>>>>>>>>>>>>> if it was manually modified?
>>>>>>>>>>>>> Is this an upgrade from oVirt 4.1?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> [root@dc02-ovirt01 log]# cat
>>>>>>>>>>>>>>
/etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf
>>>>>>>>>>>>>> # This file is automatically
generated by engine-setup.
>>>>>>>>>>>>>> Please do not edit manually
>>>>>>>>>>>>>> [OVN REMOTE]
>>>>>>>>>>>>>> ovn-remote=tcp:127.0.0.1:6641
>>>>>>>>>>>>>> [SSL]
>>>>>>>>>>>>>> https-enabled=false
>>>>>>>>>>>>>>
ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass
>>>>>>>>>>>>>> [OVIRT]
>>>>>>>>>>>>>>
ovirt-sso-client-secret=*random_test*
>>>>>>>>>>>>>>
ovirt-host=https://dc02-ovirt01.testdomain.com:443
>>>>>>>>>>>>>>
ovirt-sso-client-id=ovirt-provider-ovn
>>>>>>>>>>>>>>
ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem
>>>>>>>>>>>>>> [NETWORK]
>>>>>>>>>>>>>>
port-security-enabled-default=True
>>>>>>>>>>>>>> [PROVIDER]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
provider-host=dc02-ovirt01.testdomain.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The only entry missing is the
[AUTH] and under [SSL] the
>>>>>>>>>>>>>> https-enabled is false. Should I
edit this in this file or is this going to
>>>>>>>>>>>>>> break everything?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Changing the file should improve, but
better create a backup
>>>>>>>>>>>>> into another diretory before
modification.
>>>>>>>>>>>>> The only required change is
>>>>>>>>>>>>> from
>>>>>>>>>>>>> ovn-remote=tcp:127.0.0.1:6641
>>>>>>>>>>>>> to
>>>>>>>>>>>>> ovn-remote=ssl:127.0.0.1:6641
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Sep 15, 2020 at 6:27 PM
Dominik Holler <
>>>>>>>>>>>>>> dholler(a)redhat.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Sep 15, 2020 at 5:11
PM Konstantinos Betsis <
>>>>>>>>>>>>>>> k.betsis(a)gmail.com>
wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Dominik
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> That immediately fixed
the geneve tunnels between all
>>>>>>>>>>>>>>>> hosts.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> thanks for the feedback.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> However, the ovn provider
is not broken.
>>>>>>>>>>>>>>>> After fixing the networks
we tried to move a VM to the
>>>>>>>>>>>>>>>> DC01-host01 so we powered
it down and simply configured it to run on
>>>>>>>>>>>>>>>> dc01-node01.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> While checking the logs
on the ovirt engine i noticed the
>>>>>>>>>>>>>>>> below:
>>>>>>>>>>>>>>>> Failed to synchronize
networks of Provider
>>>>>>>>>>>>>>>> ovirt-provider-ovn.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The ovn-provider
configure on the engine is the below:
>>>>>>>>>>>>>>>> Name: ovirt-provider-ovn
>>>>>>>>>>>>>>>> Description: oVirt
network provider for OVN
>>>>>>>>>>>>>>>> Type: External Network
Provider
>>>>>>>>>>>>>>>> Network Plugin: oVirt
Network Provider for OVN
>>>>>>>>>>>>>>>> Automatic
Synchronization: Checked
>>>>>>>>>>>>>>>> Unmanaged: Unchecked
>>>>>>>>>>>>>>>> Provider URL:
http:localhost:9696
>>>>>>>>>>>>>>>> Requires Authentication:
Checked
>>>>>>>>>>>>>>>> Username: admin@internal
>>>>>>>>>>>>>>>> Password: "The admin
password"
>>>>>>>>>>>>>>>> Protocol: hTTP
>>>>>>>>>>>>>>>> Host Name: dc02-ovirt01
>>>>>>>>>>>>>>>> API Port: 35357
>>>>>>>>>>>>>>>> API Version: v2.0
>>>>>>>>>>>>>>>> Tenant Name:
"Empty"
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In the past this was
deleted by an engineer and recreated
>>>>>>>>>>>>>>>> as per the documentation,
and it worked. Do we need to update something due
>>>>>>>>>>>>>>>> to the SSL on the ovn?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is there a file in
/etc/ovirt-provider-ovn/conf.d/ ?
>>>>>>>>>>>>>>> engine-setup should have
created one.
>>>>>>>>>>>>>>> If the file is missing, for
testing purposes, you can
>>>>>>>>>>>>>>> create a
>>>>>>>>>>>>>>> file
/etc/ovirt-provider-ovn/conf.d/00-setup-ovirt-provider-ovn-test.conf :
>>>>>>>>>>>>>>> [PROVIDER]
>>>>>>>>>>>>>>>
provider-host=REPLACE_WITH_FQDN
>>>>>>>>>>>>>>> [SSL]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass
>>>>>>>>>>>>>>>
ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem
>>>>>>>>>>>>>>> https-enabled=true
>>>>>>>>>>>>>>> [OVN REMOTE]
>>>>>>>>>>>>>>>
ovn-remote=ssl:127.0.0.1:6641
>>>>>>>>>>>>>>> [AUTH]
>>>>>>>>>>>>>>>
auth-plugin=auth.plugins.static_token:NoAuthPlugin
>>>>>>>>>>>>>>> [NETWORK]
>>>>>>>>>>>>>>>
port-security-enabled-default=True
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and restart the
ovirt-provider-ovn service.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> From the ovn-provider
logs the below is generated after a
>>>>>>>>>>>>>>>> service restart and when
the start VM is triggered
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2020-09-15 15:07:33,579
root Starting server
>>>>>>>>>>>>>>>> 2020-09-15 15:07:33,579
root Version: 1.2.29-1
>>>>>>>>>>>>>>>> 2020-09-15 15:07:33,579
root Build date: 20191217125241
>>>>>>>>>>>>>>>> 2020-09-15 15:07:33,579
root Githash: cb5a80d
>>>>>>>>>>>>>>>> 2020-09-15 15:08:26,582
root From: ::ffff:127.0.0.1:59980
>>>>>>>>>>>>>>>> Request: GET /v2.0/ports
>>>>>>>>>>>>>>>> 2020-09-15 15:08:26,582
root Could not retrieve schema
>>>>>>>>>>>>>>>> from tcp:127.0.0.1:6641:
Unknown error -1
>>>>>>>>>>>>>>>> Traceback (most recent
call last):
>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>
"/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 138, in
>>>>>>>>>>>>>>>> _handle_request
>>>>>>>>>>>>>>>> method, path_parts,
content
>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>
"/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 175, in
>>>>>>>>>>>>>>>> handle_request
>>>>>>>>>>>>>>>> return
self.call_response_handler(handler, content,
>>>>>>>>>>>>>>>> parameters)
>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>
"/usr/share/ovirt-provider-ovn/handlers/neutron.py", line 35, in
>>>>>>>>>>>>>>>> call_response_handler
>>>>>>>>>>>>>>>> with NeutronApi() as
ovn_north:
>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>
"/usr/share/ovirt-provider-ovn/neutron/neutron_api.py", line 95, in __init__
>>>>>>>>>>>>>>>> self.ovsidl, self.idl
= ovn_connection.connect()
>>>>>>>>>>>>>>>> File
"/usr/share/ovirt-provider-ovn/ovn_connection.py",
>>>>>>>>>>>>>>>> line 46, in connect
>>>>>>>>>>>>>>>>
ovnconst.OVN_NORTHBOUND
>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>
"/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py",
>>>>>>>>>>>>>>>> line 127, in from_server
>>>>>>>>>>>>>>>> helper =
idlutils.get_schema_helper(connection_string,
>>>>>>>>>>>>>>>> schema_name)
>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>
"/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py",
>>>>>>>>>>>>>>>> line 128, in
get_schema_helper
>>>>>>>>>>>>>>>> 'err':
os.strerror(err)})
>>>>>>>>>>>>>>>> Exception: Could not
retrieve schema from tcp:
>>>>>>>>>>>>>>>> 127.0.0.1:6641: Unknown
error -1
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When i update the ovn
provider from the GUI to have
>>>>>>>>>>>>>>>> https://localhost:9696/
and HTTPS as the protocol the
>>>>>>>>>>>>>>>> test fails.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Sep 15, 2020 at
5:35 PM Dominik Holler <
>>>>>>>>>>>>>>>> dholler(a)redhat.com>
wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Sep 14, 2020
at 9:25 AM Konstantinos Betsis <
>>>>>>>>>>>>>>>>>
k.betsis(a)gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Dominik
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> When these
commands are used on the ovirt-engine host
>>>>>>>>>>>>>>>>>> the output is the
one depicted in your email.
>>>>>>>>>>>>>>>>>> For your
reference see also below:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
[root@ath01-ovirt01 certs]# ovn-nbctl get-ssl
>>>>>>>>>>>>>>>>>> Private key:
>>>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass
>>>>>>>>>>>>>>>>>> Certificate:
/etc/pki/ovirt-engine/certs/ovn-ndb.cer
>>>>>>>>>>>>>>>>>> CA Certificate:
/etc/pki/ovirt-engine/ca.pem
>>>>>>>>>>>>>>>>>> Bootstrap: false
>>>>>>>>>>>>>>>>>>
[root@ath01-ovirt01 certs]# ovn-nbctl get-connection
>>>>>>>>>>>>>>>>>> ptcp:6641
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
[root@ath01-ovirt01 certs]# ovn-sbctl get-ssl
>>>>>>>>>>>>>>>>>> Private key:
>>>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass
>>>>>>>>>>>>>>>>>> Certificate:
/etc/pki/ovirt-engine/certs/ovn-sdb.cer
>>>>>>>>>>>>>>>>>> CA Certificate:
/etc/pki/ovirt-engine/ca.pem
>>>>>>>>>>>>>>>>>> Bootstrap: false
>>>>>>>>>>>>>>>>>>
[root@ath01-ovirt01 certs]# ovn-sbctl get-connection
>>>>>>>>>>>>>>>>>> read-write
role="" ptcp:6642
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ^^^ the line above
points to the problem: ovn-central is
>>>>>>>>>>>>>>>>> configured to use
plain TCP without ssl.
>>>>>>>>>>>>>>>>> engine-setup usually
configures ovn-central to use SSL.
>>>>>>>>>>>>>>>>> That the files
/etc/pki/ovirt-engine/keys/ovn-* exist, shows,
>>>>>>>>>>>>>>>>> that engine-setup was
triggered correctly. Looks like the
>>>>>>>>>>>>>>>>> ovn db was dropped
somehow, this should not happen.
>>>>>>>>>>>>>>>>> This can be fixed
manually by executing the following
>>>>>>>>>>>>>>>>> commands on
engine's machine:
>>>>>>>>>>>>>>>>> ovn-nbctl set-ssl
>>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass
>>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/certs/ovn-ndb.cer /etc/pki/ovirt-engine/ca.pem
>>>>>>>>>>>>>>>>> ovn-nbctl
set-connection pssl:6641
>>>>>>>>>>>>>>>>> ovn-sbctl set-ssl
>>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass
>>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/certs/ovn-sdb.cer /etc/pki/ovirt-engine/ca.pem
>>>>>>>>>>>>>>>>> ovn-sbctl
set-connection pssl:6642
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The
/var/log/openvswitch/ovn-controller.log on the hosts
>>>>>>>>>>>>>>>>> should tell that
br-int.mgmt is connected now.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
[root@ath01-ovirt01 certs]# ls -l
>>>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-*
>>>>>>>>>>>>>>>>>> -rw-r-----. 1
root hugetlbfs 1828 Jun 25 11:08
>>>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass
>>>>>>>>>>>>>>>>>> -rw-------. 1
root root 2893 Jun 25 11:08
>>>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-ndb.p12
>>>>>>>>>>>>>>>>>> -rw-r-----. 1
root hugetlbfs 1828 Jun 25 11:08
>>>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass
>>>>>>>>>>>>>>>>>> -rw-------. 1
root root 2893 Jun 25 11:08
>>>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-sdb.p12
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> When i try the
above commands on the node hosts the
>>>>>>>>>>>>>>>>>> following
happens:
>>>>>>>>>>>>>>>>>> ovn-nbctl get-ssl
/ get-connection
>>>>>>>>>>>>>>>>>> ovn-nbctl:
unix:/var/run/openvswitch/ovnnb_db.sock:
>>>>>>>>>>>>>>>>>> database
connection failed (No such file or directory)
>>>>>>>>>>>>>>>>>> The above i
believe is expected since no northbound
>>>>>>>>>>>>>>>>>> connections
should be established from the host nodes.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ovn-sbctl get-ssl
/get-connection
>>>>>>>>>>>>>>>>>> The output is
stuck till i terminate it.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yes, the ovn-*
commands works only on engine's machine,
>>>>>>>>>>>>>>>>> which has the role
ovn-central.
>>>>>>>>>>>>>>>>> On the hosts, there
is only the ovn-controller, which
>>>>>>>>>>>>>>>>> connects the ovn
southbound to openvswitch on the host.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> For the requested
logs the below are found in the
>>>>>>>>>>>>>>>>>>
ovsdb-server-sb.log
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
2020-09-14T07:18:38.187Z|219636|reconnect|WARN|tcp:DC02-host01:33146:
>>>>>>>>>>>>>>>>>> connection
dropped (Protocol error)
>>>>>>>>>>>>>>>>>>
2020-09-14T07:18:41.946Z|219637|reconnect|WARN|tcp:DC01-host01:51188:
>>>>>>>>>>>>>>>>>> connection
dropped (Protocol error)
>>>>>>>>>>>>>>>>>>
2020-09-14T07:18:43.033Z|219638|reconnect|WARN|tcp:DC01-host02:37044:
>>>>>>>>>>>>>>>>>> connection
dropped (Protocol error)
>>>>>>>>>>>>>>>>>>
2020-09-14T07:18:46.198Z|219639|reconnect|WARN|tcp:DC02-host01:33148:
>>>>>>>>>>>>>>>>>> connection
dropped (Protocol error)
>>>>>>>>>>>>>>>>>>
2020-09-14T07:18:50.069Z|219640|jsonrpc|WARN|Dropped 4
>>>>>>>>>>>>>>>>>> log messages in
last 12 seconds (most recently, 4 seconds ago) due to
>>>>>>>>>>>>>>>>>> excessive rate
>>>>>>>>>>>>>>>>>>
2020-09-14T07:18:50.069Z|219641|jsonrpc|WARN|tcp:DC01-host01:51190:
>>>>>>>>>>>>>>>>>> error parsing
stream: line 0, column 0, byte 0: invalid character U+0016
>>>>>>>>>>>>>>>>>>
2020-09-14T07:18:50.069Z|219642|jsonrpc|WARN|Dropped 4
>>>>>>>>>>>>>>>>>> log messages in
last 12 seconds (most recently, 4 seconds ago) due to
>>>>>>>>>>>>>>>>>> excessive rate
>>>>>>>>>>>>>>>>>>
2020-09-14T07:18:50.069Z|219643|jsonrpc|WARN|tcp:DC01-host01:51190:
>>>>>>>>>>>>>>>>>> received SSL data
on JSON-RPC channel
>>>>>>>>>>>>>>>>>>
2020-09-14T07:18:50.070Z|219644|reconnect|WARN|tcp:DC01-host01:51190:
>>>>>>>>>>>>>>>>>> connection
dropped (Protocol error)
>>>>>>>>>>>>>>>>>>
2020-09-14T07:18:51.147Z|219645|reconnect|WARN|tcp:DC01-host02:37046:
>>>>>>>>>>>>>>>>>> connection
dropped (Protocol error)
>>>>>>>>>>>>>>>>>>
2020-09-14T07:18:54.209Z|219646|reconnect|WARN|tcp:DC02-host01:33150:
>>>>>>>>>>>>>>>>>> connection
dropped (Protocol error)
>>>>>>>>>>>>>>>>>>
2020-09-14T07:18:58.192Z|219647|reconnect|WARN|tcp:DC01-host01:51192:
>>>>>>>>>>>>>>>>>> connection
dropped (Protocol error)
>>>>>>>>>>>>>>>>>>
2020-09-14T07:18:59.262Z|219648|jsonrpc|WARN|Dropped 3
>>>>>>>>>>>>>>>>>> log messages in
last 8 seconds (most recently, 1 seconds ago) due to
>>>>>>>>>>>>>>>>>> excessive rate
>>>>>>>>>>>>>>>>>>
2020-09-14T07:18:59.262Z|219649|jsonrpc|WARN|tcp:DC01-host02:37048:
>>>>>>>>>>>>>>>>>> error parsing
stream: line 0, column 0, byte 0: invalid character U+0016
>>>>>>>>>>>>>>>>>>
2020-09-14T07:18:59.263Z|219650|jsonrpc|WARN|Dropped 3
>>>>>>>>>>>>>>>>>> log messages in
last 8 seconds (most recently, 1 seconds ago) due to
>>>>>>>>>>>>>>>>>> excessive rate
>>>>>>>>>>>>>>>>>>
2020-09-14T07:18:59.263Z|219651|jsonrpc|WARN|tcp:DC01-host02:37048:
>>>>>>>>>>>>>>>>>> received SSL data
on JSON-RPC channel
>>>>>>>>>>>>>>>>>>
2020-09-14T07:18:59.263Z|219652|reconnect|WARN|tcp:DC01-host02:37048:
>>>>>>>>>>>>>>>>>> connection
dropped (Protocol error)
>>>>>>>>>>>>>>>>>>
2020-09-14T07:19:02.220Z|219653|reconnect|WARN|tcp:DC02-host01:33152:
>>>>>>>>>>>>>>>>>> connection
dropped (Protocol error)
>>>>>>>>>>>>>>>>>>
2020-09-14T07:19:06.316Z|219654|reconnect|WARN|tcp:DC01-host01:51194:
>>>>>>>>>>>>>>>>>> connection
dropped (Protocol error)
>>>>>>>>>>>>>>>>>>
2020-09-14T07:19:07.386Z|219655|reconnect|WARN|tcp:DC01-host02:37050:
>>>>>>>>>>>>>>>>>> connection
dropped (Protocol error)
>>>>>>>>>>>>>>>>>>
2020-09-14T07:19:10.232Z|219656|reconnect|WARN|tcp:DC02-host01:33154:
>>>>>>>>>>>>>>>>>> connection
dropped (Protocol error)
>>>>>>>>>>>>>>>>>>
2020-09-14T07:19:14.439Z|219657|jsonrpc|WARN|Dropped 4
>>>>>>>>>>>>>>>>>> log messages in
last 12 seconds (most recently, 4 seconds ago) due to
>>>>>>>>>>>>>>>>>> excessive rate
>>>>>>>>>>>>>>>>>>
2020-09-14T07:19:14.439Z|219658|jsonrpc|WARN|tcp:DC01-host01:51196:
>>>>>>>>>>>>>>>>>> error parsing
stream: line 0, column 0, byte 0: invalid character U+0016
>>>>>>>>>>>>>>>>>>
2020-09-14T07:19:14.439Z|219659|jsonrpc|WARN|Dropped 4
>>>>>>>>>>>>>>>>>> log messages in
last 12 seconds (most recently, 4 seconds ago) due to
>>>>>>>>>>>>>>>>>> excessive rate
>>>>>>>>>>>>>>>>>>
2020-09-14T07:19:14.439Z|219660|jsonrpc|WARN|tcp:DC01-host01:51196:
>>>>>>>>>>>>>>>>>> received SSL data
on JSON-RPC channel
>>>>>>>>>>>>>>>>>>
2020-09-14T07:19:14.440Z|219661|reconnect|WARN|tcp:DC01-host01:51196:
>>>>>>>>>>>>>>>>>> connection
dropped (Protocol error)
>>>>>>>>>>>>>>>>>>
2020-09-14T07:19:15.505Z|219662|reconnect|WARN|tcp:DC01-host02:37052:
>>>>>>>>>>>>>>>>>> connection
dropped (Protocol error)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> How can we fix
these SSL errors?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I addressed this
above.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I thought vdsm
did the certificate provisioning on the
>>>>>>>>>>>>>>>>>> host nodes as to
communicate to the engine host node.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yes, this seems to
work in your scenario, just the SSL
>>>>>>>>>>>>>>>>> configuration on the
ovn-central was lost.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Sep 11,
2020 at 6:39 PM Dominik Holler <
>>>>>>>>>>>>>>>>>>
dholler(a)redhat.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Looks still
like the ovn-controller on the host
>>>>>>>>>>>>>>>>>>> has problems
communicating with ovn-southbound.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Are there any
hints in /var/log/openvswitch/*.log,
>>>>>>>>>>>>>>>>>>> especially in
/var/log/openvswitch/ovsdb-server-sb.log ?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Can you
please check the output of
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ovn-nbctl
get-ssl
>>>>>>>>>>>>>>>>>>> ovn-nbctl
get-connection
>>>>>>>>>>>>>>>>>>> ovn-sbctl
get-ssl
>>>>>>>>>>>>>>>>>>> ovn-sbctl
get-connection
>>>>>>>>>>>>>>>>>>> ls -l
/etc/pki/ovirt-engine/keys/ovn-*
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> it should be
similar to
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
[root@ovirt-43 ~]# ovn-nbctl get-ssl
>>>>>>>>>>>>>>>>>>> Private key:
>>>>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass
>>>>>>>>>>>>>>>>>>> Certificate:
/etc/pki/ovirt-engine/certs/ovn-ndb.cer
>>>>>>>>>>>>>>>>>>> CA
Certificate: /etc/pki/ovirt-engine/ca.pem
>>>>>>>>>>>>>>>>>>> Bootstrap:
false
>>>>>>>>>>>>>>>>>>>
[root@ovirt-43 ~]# ovn-nbctl get-connection
>>>>>>>>>>>>>>>>>>>
pssl:6641:[::]
>>>>>>>>>>>>>>>>>>>
[root@ovirt-43 ~]# ovn-sbctl get-ssl
>>>>>>>>>>>>>>>>>>> Private key:
>>>>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass
>>>>>>>>>>>>>>>>>>> Certificate:
/etc/pki/ovirt-engine/certs/ovn-sdb.cer
>>>>>>>>>>>>>>>>>>> CA
Certificate: /etc/pki/ovirt-engine/ca.pem
>>>>>>>>>>>>>>>>>>> Bootstrap:
false
>>>>>>>>>>>>>>>>>>>
[root@ovirt-43 ~]# ovn-sbctl get-connection
>>>>>>>>>>>>>>>>>>> read-write
role="" pssl:6642:[::]
>>>>>>>>>>>>>>>>>>>
[root@ovirt-43 ~]# ls -l
>>>>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-*
>>>>>>>>>>>>>>>>>>> -rw-r-----. 1
root hugetlbfs 1828 Oct 14 2019
>>>>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass
>>>>>>>>>>>>>>>>>>> -rw-------. 1
root root 2709 Oct 14 2019
>>>>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-ndb.p12
>>>>>>>>>>>>>>>>>>> -rw-r-----. 1
root hugetlbfs 1828 Oct 14 2019
>>>>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass
>>>>>>>>>>>>>>>>>>> -rw-------. 1
root root 2709 Oct 14 2019
>>>>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-sdb.p12
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Sep
11, 2020 at 1:10 PM Konstantinos Betsis <
>>>>>>>>>>>>>>>>>>>
k.betsis(a)gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I did a
restart of the ovn-controller, this is the
>>>>>>>>>>>>>>>>>>>> output of
the ovn-controller.log
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:07.566Z|00001|vlog|INFO|opened log
>>>>>>>>>>>>>>>>>>>> file
/var/log/openvswitch/ovn-controller.log
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:07.568Z|00002|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
>>>>>>>>>>>>>>>>>>>>
connecting...
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:07.568Z|00003|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
>>>>>>>>>>>>>>>>>>>>
connected
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:07.570Z|00004|main|INFO|OVS IDL
>>>>>>>>>>>>>>>>>>>>
reconnected, force recompute.
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:07.571Z|00005|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>>>>>
connecting...
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:07.571Z|00006|main|INFO|OVNSB IDL
>>>>>>>>>>>>>>>>>>>>
reconnected, force recompute.
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:07.685Z|00007|stream_ssl|WARN|SSL_connect:
>>>>>>>>>>>>>>>>>>>>
unexpected SSL connection close
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:07.685Z|00008|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>>>>>
connection attempt failed (Protocol error)
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:08.685Z|00009|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>>>>>
connecting...
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:08.800Z|00010|stream_ssl|WARN|SSL_connect:
>>>>>>>>>>>>>>>>>>>>
unexpected SSL connection close
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:08.800Z|00011|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>>>>>
connection attempt failed (Protocol error)
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:08.800Z|00012|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>>>>> waiting 2
seconds before reconnect
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:10.802Z|00013|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>>>>>
connecting...
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:10.917Z|00014|stream_ssl|WARN|SSL_connect:
>>>>>>>>>>>>>>>>>>>>
unexpected SSL connection close
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:10.917Z|00015|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>>>>>
connection attempt failed (Protocol error)
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:10.917Z|00016|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>>>>> waiting 4
seconds before reconnect
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:14.921Z|00017|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>>>>>
connecting...
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:15.036Z|00018|stream_ssl|WARN|SSL_connect:
>>>>>>>>>>>>>>>>>>>>
unexpected SSL connection close
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:15.036Z|00019|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>>>>>
connection attempt failed (Protocol error)
>>>>>>>>>>>>>>>>>>>>
2020-09-11T10:54:15.036Z|00020|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>>>>>
continuing to reconnect in the background but suppressing further logging
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I have
also done the vdsm-tool ovn-config
>>>>>>>>>>>>>>>>>>>>
OVIRT_ENGINE_IP OVIRTMGMT_NETWORK_DC
>>>>>>>>>>>>>>>>>>>> This is
how the OVIRT_ENGINE_IP is provided in the ovn
>>>>>>>>>>>>>>>>>>>>
controller, i can redo it if you wan.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> After the
restart of the ovn-controller the OVIRT
>>>>>>>>>>>>>>>>>>>> ENGINE
still shows only two geneve connections one with DC01-host02 and
>>>>>>>>>>>>>>>>>>>>
DC02-host01.
>>>>>>>>>>>>>>>>>>>> Chassis
"c4b23834-aec7-4bf8-8be7-aa94a50a6144"
>>>>>>>>>>>>>>>>>>>>
hostname: "dc02-host01"
>>>>>>>>>>>>>>>>>>>> Encap
geneve
>>>>>>>>>>>>>>>>>>>>
ip: "DC02-host01_IP"
>>>>>>>>>>>>>>>>>>>>
options: {csum="true"}
>>>>>>>>>>>>>>>>>>>> Chassis
"be3abcc9-7358-4040-a37b-8d8a782f239c"
>>>>>>>>>>>>>>>>>>>>
hostname: "DC01-host02"
>>>>>>>>>>>>>>>>>>>> Encap
geneve
>>>>>>>>>>>>>>>>>>>>
ip: "DC01-host02"
>>>>>>>>>>>>>>>>>>>>
options: {csum="true"}
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I've
re-done the vdsm-tool command and nothing
>>>>>>>>>>>>>>>>>>>>
changed.... again....with the same errors as the systemctl restart
>>>>>>>>>>>>>>>>>>>>
ovn-controller
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri,
Sep 11, 2020 at 1:49 PM Dominik Holler <
>>>>>>>>>>>>>>>>>>>>
dholler(a)redhat.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
Please include ovirt-users list in your reply, to
>>>>>>>>>>>>>>>>>>>>> share
the knowledge and experience with the community!
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On
Fri, Sep 11, 2020 at 12:12 PM Konstantinos Betsis <
>>>>>>>>>>>>>>>>>>>>>
k.betsis(a)gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
Ok below the output per node and DC
>>>>>>>>>>>>>>>>>>>>>>
DC01
>>>>>>>>>>>>>>>>>>>>>>
node01
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
[root@dc01-node01 ~]# ovs-vsctl --no-wait get open
>>>>>>>>>>>>>>>>>>>>>> .
external-ids:ovn-remote
>>>>>>>>>>>>>>>>>>>>>>
"ssl:*OVIRT_ENGINE_IP*:6642"
>>>>>>>>>>>>>>>>>>>>>>
[root@ dc01-node01 ~]# ovs-vsctl --no-wait get open
>>>>>>>>>>>>>>>>>>>>>> .
external-ids:ovn-encap-type
>>>>>>>>>>>>>>>>>>>>>>
geneve
>>>>>>>>>>>>>>>>>>>>>>
[root@ dc01-node01 ~]# ovs-vsctl --no-wait get open
>>>>>>>>>>>>>>>>>>>>>> .
external-ids:ovn-encap-ip
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
"*OVIRTMGMT_IP_DC01-NODE01*"
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
node02
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
[root@dc01-node02 ~]# ovs-vsctl --no-wait get open
>>>>>>>>>>>>>>>>>>>>>> .
external-ids:ovn-remote
>>>>>>>>>>>>>>>>>>>>>>
"ssl:*OVIRT_ENGINE_IP*:6642"
>>>>>>>>>>>>>>>>>>>>>>
[root@ dc01-node02 ~]# ovs-vsctl --no-wait get open
>>>>>>>>>>>>>>>>>>>>>> .
external-ids:ovn-encap-type
>>>>>>>>>>>>>>>>>>>>>>
geneve
>>>>>>>>>>>>>>>>>>>>>>
[root@ dc01-node02 ~]# ovs-vsctl --no-wait get open
>>>>>>>>>>>>>>>>>>>>>> .
external-ids:ovn-encap-ip
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
"*OVIRTMGMT_IP_DC01-NODE02*"
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
DC02
>>>>>>>>>>>>>>>>>>>>>>
node01
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
[root@dc02-node01 ~]# ovs-vsctl --no-wait get open
>>>>>>>>>>>>>>>>>>>>>> .
external-ids:ovn-remote
>>>>>>>>>>>>>>>>>>>>>>
"ssl:*OVIRT_ENGINE_IP*:6642"
>>>>>>>>>>>>>>>>>>>>>>
[root@ dc02-node01 ~]# ovs-vsctl --no-wait get open
>>>>>>>>>>>>>>>>>>>>>> .
external-ids:ovn-encap-type
>>>>>>>>>>>>>>>>>>>>>>
geneve
>>>>>>>>>>>>>>>>>>>>>>
[root@ dc02-node01 ~]# ovs-vsctl --no-wait get open
>>>>>>>>>>>>>>>>>>>>>> .
external-ids:ovn-encap-ip
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
"*OVIRTMGMT_IP_DC02-NODE01*"
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Looks
good.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
DC01 node01 and node02 share the same VM networks
>>>>>>>>>>>>>>>>>>>>>>
and VMs deployed on top of them cannot talk to VM on the other hypervisor.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Maybe
there is a hint on ovn-controller.log on
>>>>>>>>>>>>>>>>>>>>>
dc01-node02 ? Maybe restarting ovn-controller creates more helpful log
>>>>>>>>>>>>>>>>>>>>>
messages?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> You
can also try restart the ovn configuration on all
>>>>>>>>>>>>>>>>>>>>> hosts
by executing
>>>>>>>>>>>>>>>>>>>>>
vdsm-tool ovn-config OVIRT_ENGINE_IP
>>>>>>>>>>>>>>>>>>>>>
LOCAL_OVIRTMGMT_IP
>>>>>>>>>>>>>>>>>>>>> on
each host, this would trigger
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
https://github.com/oVirt/ovirt-provider-ovn/blob/master/driver/scripts/se...
>>>>>>>>>>>>>>>>>>>>>
internally.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
So I would expect to see the same output for node01
>>>>>>>>>>>>>>>>>>>>>>
to have a geneve tunnel to node02 and vice versa.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Me
too.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
On Fri, Sep 11, 2020 at 12:14 PM Dominik Holler <
>>>>>>>>>>>>>>>>>>>>>>
dholler(a)redhat.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
On Fri, Sep 11, 2020 at 10:53 AM Konstantinos
>>>>>>>>>>>>>>>>>>>>>>>
Betsis <k.betsis(a)gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
Hi Dominik
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
OVN is selected as the default network provider on
>>>>>>>>>>>>>>>>>>>>>>>>
the clusters and the hosts.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
sounds good.
>>>>>>>>>>>>>>>>>>>>>>>
This configuration is required already during the
>>>>>>>>>>>>>>>>>>>>>>>
host is added to oVirt Engine, because OVN is configured during this step.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
The "ovn-sbctl show" works on the ovirt engine and
>>>>>>>>>>>>>>>>>>>>>>>>
shows only two hosts, 1 per DC.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
Chassis "c4b23834-aec7-4bf8-8be7-aa94a50a6144"
>>>>>>>>>>>>>>>>>>>>>>>>
hostname: "dc01-node02"
>>>>>>>>>>>>>>>>>>>>>>>>
Encap geneve
>>>>>>>>>>>>>>>>>>>>>>>>
ip: "X.X.X.X"
>>>>>>>>>>>>>>>>>>>>>>>>
options: {csum="true"}
>>>>>>>>>>>>>>>>>>>>>>>>
Chassis "be3abcc9-7358-4040-a37b-8d8a782f239c"
>>>>>>>>>>>>>>>>>>>>>>>>
hostname: "dc02-node1"
>>>>>>>>>>>>>>>>>>>>>>>>
Encap geneve
>>>>>>>>>>>>>>>>>>>>>>>>
ip: "A.A.A.A"
>>>>>>>>>>>>>>>>>>>>>>>>
options: {csum="true"}
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
The new node is not listed (dc01-node1).
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
When executed on the nodes the same command
>>>>>>>>>>>>>>>>>>>>>>>>
(ovn-sbctl show) times-out on all nodes.....
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
The output of the
>>>>>>>>>>>>>>>>>>>>>>>>
/var/log/openvswitch/ovn-conntroller.log lists on all logs
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
2020-09-11T08:46:55.197Z|07361|stream_ssl|WARN|SSL_connect:
>>>>>>>>>>>>>>>>>>>>>>>>
unexpected SSL connection close
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
Can you please compare the output of
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
ovs-vsctl --no-wait get open .
>>>>>>>>>>>>>>>>>>>>>>>
external-ids:ovn-remote
>>>>>>>>>>>>>>>>>>>>>>>
ovs-vsctl --no-wait get open .
>>>>>>>>>>>>>>>>>>>>>>>
external-ids:ovn-encap-type
>>>>>>>>>>>>>>>>>>>>>>>
ovs-vsctl --no-wait get open .
>>>>>>>>>>>>>>>>>>>>>>>
external-ids:ovn-encap-ip
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
of the working hosts, e.g. dc01-node02, and the
>>>>>>>>>>>>>>>>>>>>>>>
failing host dc01-node1?
>>>>>>>>>>>>>>>>>>>>>>>
This should point us the relevant difference in the
>>>>>>>>>>>>>>>>>>>>>>>
configuration.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
Please include ovirt-users list in your replay, to
>>>>>>>>>>>>>>>>>>>>>>>
share the knowledge and experience with the community.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
Thank you
>>>>>>>>>>>>>>>>>>>>>>>>
Best regards
>>>>>>>>>>>>>>>>>>>>>>>>
Konstantinos Betsis
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
On Fri, Sep 11, 2020 at 11:01 AM Dominik Holler <
>>>>>>>>>>>>>>>>>>>>>>>>
dholler(a)redhat.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
On Thu, Sep 10, 2020 at 6:26 PM Konstantinos B <
>>>>>>>>>>>>>>>>>>>>>>>>>
k.betsis(a)gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
Hi all
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
We have a small installation based on OVIRT 4.3.
>>>>>>>>>>>>>>>>>>>>>>>>>>
1 Cluster is based on Centos 7 and the other on
>>>>>>>>>>>>>>>>>>>>>>>>>>
OVIRT NG Node image.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
The environment was stable till an upgrade took
>>>>>>>>>>>>>>>>>>>>>>>>>>
place a couple of months ago.
>>>>>>>>>>>>>>>>>>>>>>>>>>
As such we had to re-install one of the Centos 7
>>>>>>>>>>>>>>>>>>>>>>>>>>
node and start from scratch.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
To trigger the automatic configuration of the
>>>>>>>>>>>>>>>>>>>>>>>>>
host, it is required to configure ovirt-provider-ovn as the default network
>>>>>>>>>>>>>>>>>>>>>>>>>
provider for the cluster before adding the host to oVirt.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
Even though the installation completed
>>>>>>>>>>>>>>>>>>>>>>>>>>
successfully and VMs are created, the following are not working as expected:
>>>>>>>>>>>>>>>>>>>>>>>>>>
1. ovn geneve tunnels are not established with
>>>>>>>>>>>>>>>>>>>>>>>>>>
the other Centos 7 node in the cluster.
>>>>>>>>>>>>>>>>>>>>>>>>>>
2. Centos 7 node is configured by ovirt engine
>>>>>>>>>>>>>>>>>>>>>>>>>>
however no geneve tunnel is established when "ovn-sbctl show" is issued on
>>>>>>>>>>>>>>>>>>>>>>>>>>
the engine.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
Does "ovn-sbctl show" list the hosts?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
3. no flows are shown on the engine on port 6642
>>>>>>>>>>>>>>>>>>>>>>>>>>
for the ovs db.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
Does anyone have any experience on how to
>>>>>>>>>>>>>>>>>>>>>>>>>>
troubleshoot OVN on ovirt?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
/var/log/openvswitch/ovncontroller.log on the
>>>>>>>>>>>>>>>>>>>>>>>>>
host should contain a helpful hint.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
Thank you
>>>>>>>>>>>>>>>>>>>>>>>>>>
_______________________________________________
>>>>>>>>>>>>>>>>>>>>>>>>>>
Users mailing list -- users(a)ovirt.org
>>>>>>>>>>>>>>>>>>>>>>>>>>
To unsubscribe send an email to
>>>>>>>>>>>>>>>>>>>>>>>>>>
users-leave(a)ovirt.org
>>>>>>>>>>>>>>>>>>>>>>>>>>
Privacy Statement:
>>>>>>>>>>>>>>>>>>>>>>>>>>
https://www.ovirt.org/privacy-policy.html
>>>>>>>>>>>>>>>>>>>>>>>>>>
oVirt Code of Conduct:
>>>>>>>>>>>>>>>>>>>>>>>>>>
https://www.ovirt.org/community/about/community-guidelines/
>>>>>>>>>>>>>>>>>>>>>>>>>>
List Archives:
>>>>>>>>>>>>>>>>>>>>>>>>>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/LBVGLQJBWJF...
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>