6:24 a.m.
On Wed, Sep 16, 2020 at 12:15 PM Konstantinos Betsis <k.betsis(a)gmail.com>
wrote:
I have a better solution.
I am currently migrating all VMs over to dc01-node01 and then i'll format
it as to fix the partitioning as well.
In theory the ovs sbdb will be fixed once it is re-installed....
If not we can then check if there is a stale entry in the ovirt host where
the sb db is managed.
Maybe you could ensure that there is no entry anymore during dc01-host02 is
reinstalling, before the host is added to oVirt again?
Do you agree with this?
Sounds good, but OVN should be not the reason to reinstall.
On Wed, Sep 16, 2020 at 1:00 PM Dominik Holler
<dholler(a)redhat.com> wrote:
>
>
> Maybe because of a duplicated entry in the ovn sb db?
> Can you please stop the ovn-ctrontroller on this host, remove the host
> from the ovn sb db, ensure it is gone and restart the ovn-controller on the
> host?
>
>
> On Wed, Sep 16, 2020 at 11:55 AM Konstantinos Betsis <k.betsis(a)gmail.com>
> wrote:
>
>> Hi Dominik
>>
>> Just saw the below on host dc01-host02
>>
>> ovs-vsctl show
>> f3b13557-dfb4-45a4-b6af-c995ccf68720
>> Bridge br-int
>> Port "ovn-95ccb0-0"
>> Interface "ovn-95ccb0-0"
>> type: geneve
>> options: {csum="true", key=flow,
remote_ip="dc01-host01"}
>> Port "vnet10"
>> Interface "vnet10"
>> Port "vnet11"
>> Interface "vnet11"
>> Port "vnet0"
>> Interface "vnet0"
>> Port "vnet9"
>> Interface "vnet9"
>> Port "vnet8"
>> Interface "vnet8"
>> Port br-int
>> Interface br-int
>> type: internal
>> Port "vnet12"
>> Interface "vnet12"
>> Port "ovn-be3abc-0"
>> Interface "ovn-be3abc-0"
>> type: geneve
>> options: {csum="true", key=flow,
remote_ip="dc01-host02"}
>> Port "vnet7"
>> Interface "vnet7"
>> Port "ovn-c4b238-0"
>> Interface "ovn-c4b238-0"
>> type: geneve
>> options: {csum="true", key=flow,
remote_ip="dc02-host01"}
>> Port "vnet6"
>> Interface "vnet6"
>> ovs_version: "2.11.0"
>>
>>
>> Why would this node establish a geneve tunnel to himself?
>> Other nodes do not exhibit this behavior.
>>
>> On Wed, Sep 16, 2020 at 12:21 PM Konstantinos Betsis <k.betsis(a)gmail.com>
>> wrote:
>>
>>> Hi Dominik
>>>
>>> Below is the output of the ovs-vsctl list interface
>>>
>>> _uuid : bdaf92c1-4389-4ddf-aab0-93975076ebb2
>>> admin_state : up
>>> bfd : {}
>>> bfd_status : {}
>>> cfm_fault : []
>>> cfm_fault_status : []
>>> cfm_flap_count : []
>>> cfm_health : []
>>> cfm_mpid : []
>>> cfm_remote_mpids : []
>>> cfm_remote_opstate : []
>>> duplex : full
>>> error : []
>>> external_ids : {attached-mac="56:6f:77:61:00:02",
>>> iface-id="5d03a7a5-82a1-40f9-b50c-353a26167fa3",
iface-status=active,
>>> vm-id="e45b7b34-24f2-41ed-b95e-cd1ad532e8d3"}
>>> ifindex : 34
>>> ingress_policing_burst: 0
>>> ingress_policing_rate: 0
>>> lacp_current : []
>>> link_resets : 1
>>> link_speed : 10000000
>>> link_state : up
>>> lldp : {}
>>> mac : []
>>> mac_in_use : "fe:6f:77:61:00:02"
>>> mtu : 1442
>>> mtu_request : []
>>> name : "vnet6"
>>> ofport : 2
>>> ofport_request : []
>>> options : {}
>>> other_config : {}
>>> statistics : {collisions=0, rx_bytes=10828495, rx_crc_err=0,
>>> rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0,
>>> rx_packets=117713, tx_bytes=20771797, tx_dropped=0, tx_errors=0,
>>> tx_packets=106954}
>>> status : {driver_name=tun, driver_version="1.6",
>>> firmware_version=""}
>>> type : ""
>>>
>>> _uuid : bad80911-3993-4085-a0b0-962b6c9156cd
>>> admin_state : up
>>> bfd : {}
>>> bfd_status : {}
>>> cfm_fault : []
>>> cfm_fault_status : []
>>> cfm_flap_count : []
>>> cfm_health : []
>>> cfm_mpid : []
>>> cfm_remote_mpids : []
>>> cfm_remote_opstate : []
>>> duplex : []
>>> error : []
>>> external_ids : {}
>>> ifindex : 39
>>> ingress_policing_burst: 0
>>> ingress_policing_rate: 0
>>> lacp_current : []
>>> link_resets : 0
>>> link_speed : []
>>> link_state : up
>>> lldp : {}
>>> mac : []
>>> mac_in_use : "fe:37:52:c4:cb:03"
>>> mtu : []
>>> mtu_request : []
>>> name : "ovn-c4b238-0"
>>> ofport : 7
>>> ofport_request : []
>>> options : {csum="true", key=flow,
>>> remote_ip="192.168.121.164"}
>>> other_config : {}
>>> statistics : {rx_bytes=0, rx_packets=0, tx_bytes=0,
>>> tx_packets=0}
>>> status : {tunnel_egress_iface="ovirtmgmt-ams03",
>>> tunnel_egress_iface_carrier=up}
>>> type : geneve
>>>
>>> _uuid : 8e7705d1-0b9d-4e30-8277-c339e7e1c27a
>>> admin_state : up
>>> bfd : {}
>>> bfd_status : {}
>>> cfm_fault : []
>>> cfm_fault_status : []
>>> cfm_flap_count : []
>>> cfm_health : []
>>> cfm_mpid : []
>>> cfm_remote_mpids : []
>>> cfm_remote_opstate : []
>>> duplex : full
>>> error : []
>>> external_ids : {attached-mac="56:6f:77:61:00:0d",
>>> iface-id="b7ff5f2b-4bb4-4250-8ad8-8a7e19d2b4c7",
iface-status=active,
>>> vm-id="8d73f333-bca4-4b32-9b87-2e7ee07eda84"}
>>> ifindex : 28
>>> ingress_policing_burst: 0
>>> ingress_policing_rate: 0
>>> lacp_current : []
>>> link_resets : 1
>>> link_speed : 10000000
>>> link_state : up
>>> lldp : {}
>>> mac : []
>>> mac_in_use : "fe:6f:77:61:00:0d"
>>> mtu : 1442
>>> mtu_request : []
>>> name : "vnet0"
>>> ofport : 1
>>> ofport_request : []
>>> options : {}
>>> other_config : {}
>>> statistics : {collisions=0, rx_bytes=20609787, rx_crc_err=0,
>>> rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0,
>>> rx_packets=104535, tx_bytes=10830007, tx_dropped=0, tx_errors=0,
>>> tx_packets=117735}
>>> status : {driver_name=tun, driver_version="1.6",
>>> firmware_version=""}
>>> type : ""
>>>
>>> _uuid : 86dcc68a-63e4-4445-9373-81c1f4502c17
>>> admin_state : up
>>> bfd : {}
>>> bfd_status : {}
>>> cfm_fault : []
>>> cfm_fault_status : []
>>> cfm_flap_count : []
>>> cfm_health : []
>>> cfm_mpid : []
>>> cfm_remote_mpids : []
>>> cfm_remote_opstate : []
>>> duplex : full
>>> error : []
>>> external_ids : {attached-mac="56:6f:77:61:00:10",
>>> iface-id="4e8d5636-4110-41b2-906d-f9b04c2e62cd",
iface-status=active,
>>> vm-id="9a002a9b-5f09-4def-a531-d50ff683470b"}
>>> ifindex : 40
>>> ingress_policing_burst: 0
>>> ingress_policing_rate: 0
>>> lacp_current : []
>>> link_resets : 1
>>> link_speed : 10000000
>>> link_state : up
>>> lldp : {}
>>> mac : []
>>> mac_in_use : "fe:6f:77:61:00:10"
>>> mtu : 1442
>>> mtu_request : []
>>> name : "vnet11"
>>> ofport : 10
>>> ofport_request : []
>>> options : {}
>>> other_config : {}
>>> statistics : {collisions=0, rx_bytes=3311352, rx_crc_err=0,
>>> rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=51012,
>>> tx_bytes=5514116, tx_dropped=0, tx_errors=0, tx_packets=103456}
>>> status : {driver_name=tun, driver_version="1.6",
>>> firmware_version=""}
>>> type : ""
>>>
>>> _uuid : e8d5e4a2-b9a0-4146-8d98-34713cb443de
>>> admin_state : up
>>> bfd : {}
>>> bfd_status : {}
>>> cfm_fault : []
>>> cfm_fault_status : []
>>> cfm_flap_count : []
>>> cfm_health : []
>>> cfm_mpid : []
>>> cfm_remote_mpids : []
>>> cfm_remote_opstate : []
>>> duplex : full
>>> error : []
>>> external_ids : {attached-mac="56:6f:77:61:00:15",
>>> iface-id="b88de6e4-6d77-4e42-b734-4cc676728910",
iface-status=active,
>>> vm-id="e45b7b34-24f2-41ed-b95e-cd1ad532e8d3"}
>>> ifindex : 37
>>> ingress_policing_burst: 0
>>> ingress_policing_rate: 0
>>> lacp_current : []
>>> link_resets : 1
>>> link_speed : 10000000
>>> link_state : up
>>> lldp : {}
>>> mac : []
>>> mac_in_use : "fe:6f:77:61:00:15"
>>> mtu : 1442
>>> mtu_request : []
>>> name : "vnet9"
>>> ofport : 5
>>> ofport_request : []
>>> options : {}
>>> other_config : {}
>>> statistics : {collisions=0, rx_bytes=180, rx_crc_err=0,
>>> rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=2,
>>> tx_bytes=4500, tx_dropped=0, tx_errors=0, tx_packets=74}
>>> status : {driver_name=tun, driver_version="1.6",
>>> firmware_version=""}
>>> type : ""
>>>
>>> _uuid : 6a2974b3-cd72-4688-a630-0a7e9c779b21
>>> admin_state : up
>>> bfd : {}
>>> bfd_status : {}
>>> cfm_fault : []
>>> cfm_fault_status : []
>>> cfm_flap_count : []
>>> cfm_health : []
>>> cfm_mpid : []
>>> cfm_remote_mpids : []
>>> cfm_remote_opstate : []
>>> duplex : full
>>> error : []
>>> external_ids : {attached-mac="56:6f:77:61:00:17",
>>> iface-id="64681036-26e2-41d7-b73f-ab5302610145",
iface-status=active,
>>> vm-id="bf0dc78c-dad5-41a0-914c-ae0da0f9a388"}
>>> ifindex : 41
>>> ingress_policing_burst: 0
>>> ingress_policing_rate: 0
>>> lacp_current : []
>>> link_resets : 1
>>> link_speed : 10000000
>>> link_state : up
>>> lldp : {}
>>> mac : []
>>> mac_in_use : "fe:6f:77:61:00:17"
>>> mtu : 1442
>>> mtu_request : []
>>> name : "vnet12"
>>> ofport : 11
>>> ofport_request : []
>>> options : {}
>>> other_config : {}
>>> statistics : {collisions=0, rx_bytes=5513640, rx_crc_err=0,
>>> rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0,
>>> rx_packets=103450, tx_bytes=3311868, tx_dropped=0, tx_errors=0,
>>> tx_packets=51018}
>>> status : {driver_name=tun, driver_version="1.6",
>>> firmware_version=""}
>>> type : ""
>>>
>>> _uuid : 44498e54-f122-41a0-a41a-7a88ba2dba9b
>>> admin_state : down
>>> bfd : {}
>>> bfd_status : {}
>>> cfm_fault : []
>>> cfm_fault_status : []
>>> cfm_flap_count : []
>>> cfm_health : []
>>> cfm_mpid : []
>>> cfm_remote_mpids : []
>>> cfm_remote_opstate : []
>>> duplex : []
>>> error : []
>>> external_ids : {}
>>> ifindex : 7
>>> ingress_policing_burst: 0
>>> ingress_policing_rate: 0
>>> lacp_current : []
>>> link_resets : 0
>>> link_speed : []
>>> link_state : down
>>> lldp : {}
>>> mac : []
>>> mac_in_use : "32:0a:69:67:07:4f"
>>> mtu : 1442
>>> mtu_request : []
>>> name : br-int
>>> ofport : 65534
>>> ofport_request : []
>>> options : {}
>>> other_config : {}
>>> statistics : {collisions=0, rx_bytes=0, rx_crc_err=0,
>>> rx_dropped=326, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=0,
>>> tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0}
>>> status : {driver_name=openvswitch}
>>> type : internal
>>>
>>> _uuid : e2114584-8ceb-43d6-817b-e457738ead8a
>>> admin_state : up
>>> bfd : {}
>>> bfd_status : {}
>>> cfm_fault : []
>>> cfm_fault_status : []
>>> cfm_flap_count : []
>>> cfm_health : []
>>> cfm_mpid : []
>>> cfm_remote_mpids : []
>>> cfm_remote_opstate : []
>>> duplex : full
>>> error : []
>>> external_ids : {attached-mac="56:6f:77:61:00:03",
>>> iface-id="16162721-c815-4cd8-ab57-f22e6e482c7f",
iface-status=active,
>>> vm-id="e45b7b34-24f2-41ed-b95e-cd1ad532e8d3"}
>>> ifindex : 35
>>> ingress_policing_burst: 0
>>> ingress_policing_rate: 0
>>> lacp_current : []
>>> link_resets : 1
>>> link_speed : 10000000
>>> link_state : up
>>> lldp : {}
>>> mac : []
>>> mac_in_use : "fe:6f:77:61:00:03"
>>> mtu : 1442
>>> mtu_request : []
>>> name : "vnet7"
>>> ofport : 3
>>> ofport_request : []
>>> options : {}
>>> other_config : {}
>>> statistics : {collisions=0, rx_bytes=180, rx_crc_err=0,
>>> rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=2,
>>> tx_bytes=4730, tx_dropped=0, tx_errors=0, tx_packets=77}
>>> status : {driver_name=tun, driver_version="1.6",
>>> firmware_version=""}
>>> type : ""
>>>
>>> _uuid : ee16943e-d145-4080-893f-464098a6388f
>>> admin_state : up
>>> bfd : {}
>>> bfd_status : {}
>>> cfm_fault : []
>>> cfm_fault_status : []
>>> cfm_flap_count : []
>>> cfm_health : []
>>> cfm_mpid : []
>>> cfm_remote_mpids : []
>>> cfm_remote_opstate : []
>>> duplex : []
>>> error : []
>>> external_ids : {}
>>> ifindex : 39
>>> ingress_policing_burst: 0
>>> ingress_policing_rate: 0
>>> lacp_current : []
>>> link_resets : 0
>>> link_speed : []
>>> link_state : up
>>> lldp : {}
>>> mac : []
>>> mac_in_use : "1e:50:3f:a8:42:d1"
>>> mtu : []
>>> mtu_request : []
>>> name : "ovn-be3abc-0"
>>> ofport : 8
>>> ofport_request : []
>>> options : {csum="true", key=flow,
remote_ip="DC01-host02"}
>>> other_config : {}
>>> statistics : {rx_bytes=0, rx_packets=0, tx_bytes=0,
>>> tx_packets=0}
>>> status : {tunnel_egress_iface="ovirtmgmt-ams03",
>>> tunnel_egress_iface_carrier=up}
>>> type : geneve
>>>
>>> _uuid : 86a229be-373e-4c43-b2f1-6190523ed73a
>>> admin_state : up
>>> bfd : {}
>>> bfd_status : {}
>>> cfm_fault : []
>>> cfm_fault_status : []
>>> cfm_flap_count : []
>>> cfm_health : []
>>> cfm_mpid : []
>>> cfm_remote_mpids : []
>>> cfm_remote_opstate : []
>>> duplex : full
>>> error : []
>>> external_ids : {attached-mac="56:6f:77:61:00:1c",
>>> iface-id="12d829c3-64eb-44bc-a0bd-d7219991f35f",
iface-status=active,
>>> vm-id="e45b7b34-24f2-41ed-b95e-cd1ad532e8d3"}
>>> ifindex : 38
>>> ingress_policing_burst: 0
>>> ingress_policing_rate: 0
>>> lacp_current : []
>>> link_resets : 1
>>> link_speed : 10000000
>>> link_state : up
>>> lldp : {}
>>> mac : []
>>> mac_in_use : "fe:6f:77:61:00:1c"
>>> mtu : 1442
>>> mtu_request : []
>>> name : "vnet10"
>>> ofport : 6
>>> ofport_request : []
>>> options : {}
>>> other_config : {}
>>> statistics : {collisions=0, rx_bytes=117912, rx_crc_err=0,
>>> rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=2195,
>>> tx_bytes=4204, tx_dropped=0, tx_errors=0, tx_packets=66}
>>> status : {driver_name=tun, driver_version="1.6",
>>> firmware_version=""}
>>> type : ""
>>>
>>> _uuid : fa4b8d96-bffe-4b56-930e-0e7fcc5f68ac
>>> admin_state : up
>>> bfd : {}
>>> bfd_status : {}
>>> cfm_fault : []
>>> cfm_fault_status : []
>>> cfm_flap_count : []
>>> cfm_health : []
>>> cfm_mpid : []
>>> cfm_remote_mpids : []
>>> cfm_remote_opstate : []
>>> duplex : []
>>> error : []
>>> external_ids : {}
>>> ifindex : 39
>>> ingress_policing_burst: 0
>>> ingress_policing_rate: 0
>>> lacp_current : []
>>> link_resets : 0
>>> link_speed : []
>>> link_state : up
>>> lldp : {}
>>> mac : []
>>> mac_in_use : "7a:28:24:eb:ec:d2"
>>> mtu : []
>>> mtu_request : []
>>> name : "ovn-95ccb0-0"
>>> ofport : 9
>>> ofport_request : []
>>> options : {csum="true", key=flow,
remote_ip="DC01-host01"}
>>> other_config : {}
>>> statistics : {rx_bytes=0, rx_packets=0, tx_bytes=12840478,
>>> tx_packets=224029}
>>> status : {tunnel_egress_iface="ovirtmgmt-ams03",
>>> tunnel_egress_iface_carrier=up}
>>> type : geneve
>>>
>>> _uuid : 5e3df5c7-958c-491d-8d41-0ae83c613f1d
>>> admin_state : up
>>> bfd : {}
>>> bfd_status : {}
>>> cfm_fault : []
>>> cfm_fault_status : []
>>> cfm_flap_count : []
>>> cfm_health : []
>>> cfm_mpid : []
>>> cfm_remote_mpids : []
>>> cfm_remote_opstate : []
>>> duplex : full
>>> error : []
>>> external_ids : {attached-mac="56:6f:77:61:00:06",
>>> iface-id="9a6cc189-0934-4468-97ae-09f90fa4598d",
iface-status=active,
>>> vm-id="e45b7b34-24f2-41ed-b95e-cd1ad532e8d3"}
>>> ifindex : 36
>>> ingress_policing_burst: 0
>>> ingress_policing_rate: 0
>>> lacp_current : []
>>> link_resets : 1
>>> link_speed : 10000000
>>> link_state : up
>>> lldp : {}
>>> mac : []
>>> mac_in_use : "fe:6f:77:61:00:06"
>>> mtu : 1442
>>> mtu_request : []
>>> name : "vnet8"
>>> ofport : 4
>>> ofport_request : []
>>> options : {}
>>> other_config : {}
>>> statistics : {collisions=0, rx_bytes=180, rx_crc_err=0,
>>> rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=2,
>>> tx_bytes=8829812, tx_dropped=0, tx_errors=0, tx_packets=154540}
>>> status : {driver_name=tun, driver_version="1.6",
>>> firmware_version=""}
>>> type : ""
>>>
>>>
>>> I've identified which VMs have these MAC addresses but i do not see any
>>> "conflict" with any other VM's MAC address.
>>>
>>> I really do not understand why these will create a conflict.
>>>
>>> On Wed, Sep 16, 2020 at 12:06 PM Dominik Holler <dholler(a)redhat.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Sep 15, 2020 at 6:53 PM Konstantinos Betsis <
>>>> k.betsis(a)gmail.com> wrote:
>>>>
>>>>> So a new test-net was created under DC01 and was depicted in the
>>>>> networks tab under both DC01 and DC02.
>>>>> I believe for some reason networks are duplicated in DCs, maybe for
>>>>> future use??? Don't know.
>>>>> If one tries to delete the network from the other DC it gets an
>>>>> error, while if deleted from the once initially created it gets
deleted
>>>>> from both.
>>>>>
>>>>>
>>>> In oVirt a logical network is an entity in a data center. If the
>>>> automatic synchronization is enabled on the ovirt-provider-ovn entity in
>>>> oVirt Engine, the OVN networks are reflected to all data centers. If you
do
>>>> not like this, you can disable the automatic synchronization of the
>>>> ovirt-provider-ovn in Admin Portal.
>>>>
>>>>
>>>>> From the DC01-node02 i get the following errors:
>>>>>
>>>>> 2020-09-15T16:48:49.904Z|22748|main|INFO|OVNSB commit failed, force
>>>>> recompute next time.
>>>>> 2020-09-15T16:48:49.905Z|22749|binding|INFO|Claiming lport
>>>>> 9a6cc189-0934-4468-97ae-09f90fa4598d for this chassis.
>>>>>
2020-09-15T16:48:49.905Z|22750|binding|INFO|9a6cc189-0934-4468-97ae-09f90fa4598d:
>>>>> Claiming 56:6f:77:61:00:06
>>>>> 2020-09-15T16:48:49.905Z|22751|binding|INFO|Claiming lport
>>>>> 16162721-c815-4cd8-ab57-f22e6e482c7f for this chassis.
>>>>>
2020-09-15T16:48:49.905Z|22752|binding|INFO|16162721-c815-4cd8-ab57-f22e6e482c7f:
>>>>> Claiming 56:6f:77:61:00:03
>>>>> 2020-09-15T16:48:49.905Z|22753|binding|INFO|Claiming lport
>>>>> b88de6e4-6d77-4e42-b734-4cc676728910 for this chassis.
>>>>>
2020-09-15T16:48:49.905Z|22754|binding|INFO|b88de6e4-6d77-4e42-b734-4cc676728910:
>>>>> Claiming 56:6f:77:61:00:15
>>>>> 2020-09-15T16:48:49.905Z|22755|binding|INFO|Claiming lport
>>>>> b7ff5f2b-4bb4-4250-8ad8-8a7e19d2b4c7 for this chassis.
>>>>>
2020-09-15T16:48:49.905Z|22756|binding|INFO|b7ff5f2b-4bb4-4250-8ad8-8a7e19d2b4c7:
>>>>> Claiming 56:6f:77:61:00:0d
>>>>> 2020-09-15T16:48:49.905Z|22757|binding|INFO|Claiming lport
>>>>> 5d03a7a5-82a1-40f9-b50c-353a26167fa3 for this chassis.
>>>>>
2020-09-15T16:48:49.905Z|22758|binding|INFO|5d03a7a5-82a1-40f9-b50c-353a26167fa3:
>>>>> Claiming 56:6f:77:61:00:02
>>>>> 2020-09-15T16:48:49.905Z|22759|binding|INFO|Claiming lport
>>>>> 12d829c3-64eb-44bc-a0bd-d7219991f35f for this chassis.
>>>>>
2020-09-15T16:48:49.905Z|22760|binding|INFO|12d829c3-64eb-44bc-a0bd-d7219991f35f:
>>>>> Claiming 56:6f:77:61:00:1c
>>>>> 2020-09-15T16:48:49.959Z|22761|main|INFO|OVNSB commit failed, force
>>>>> recompute next time.
>>>>> 2020-09-15T16:48:49.960Z|22762|binding|INFO|Claiming lport
>>>>> 9a6cc189-0934-4468-97ae-09f90fa4598d for this chassis.
>>>>>
2020-09-15T16:48:49.960Z|22763|binding|INFO|9a6cc189-0934-4468-97ae-09f90fa4598d:
>>>>> Claiming 56:6f:77:61:00:06
>>>>> 2020-09-15T16:48:49.960Z|22764|binding|INFO|Claiming lport
>>>>> 16162721-c815-4cd8-ab57-f22e6e482c7f for this chassis.
>>>>>
2020-09-15T16:48:49.960Z|22765|binding|INFO|16162721-c815-4cd8-ab57-f22e6e482c7f:
>>>>> Claiming 56:6f:77:61:00:03
>>>>> 2020-09-15T16:48:49.960Z|22766|binding|INFO|Claiming lport
>>>>> b88de6e4-6d77-4e42-b734-4cc676728910 for this chassis.
>>>>>
2020-09-15T16:48:49.960Z|22767|binding|INFO|b88de6e4-6d77-4e42-b734-4cc676728910:
>>>>> Claiming 56:6f:77:61:00:15
>>>>> 2020-09-15T16:48:49.960Z|22768|binding|INFO|Claiming lport
>>>>> b7ff5f2b-4bb4-4250-8ad8-8a7e19d2b4c7 for this chassis.
>>>>>
2020-09-15T16:48:49.960Z|22769|binding|INFO|b7ff5f2b-4bb4-4250-8ad8-8a7e19d2b4c7:
>>>>> Claiming 56:6f:77:61:00:0d
>>>>> 2020-09-15T16:48:49.960Z|22770|binding|INFO|Claiming lport
>>>>> 5d03a7a5-82a1-40f9-b50c-353a26167fa3 for this chassis.
>>>>>
2020-09-15T16:48:49.960Z|22771|binding|INFO|5d03a7a5-82a1-40f9-b50c-353a26167fa3:
>>>>> Claiming 56:6f:77:61:00:02
>>>>> 2020-09-15T16:48:49.960Z|22772|binding|INFO|Claiming lport
>>>>> 12d829c3-64eb-44bc-a0bd-d7219991f35f for this chassis.
>>>>>
2020-09-15T16:48:49.960Z|22773|binding|INFO|12d829c3-64eb-44bc-a0bd-d7219991f35f:
>>>>> Claiming 56:6f:77:61:00:1c
>>>>>
>>>>>
>>>>> And this repeats forever.
>>>>>
>>>>>
>>>> Looks like the southbound db is confused.
>>>>
>>>> Can you try to delete all chassis listed by
>>>> sudo ovn-sbctl show
>>>> via
>>>> sudo /usr/share/ovirt-provider-ovn/scripts/remove_chassis.sh dev-host0
>>>> ?
>>>> if the script remove_chassis.sh is not installed, you can use
>>>>
>>>>
https://github.com/oVirt/ovirt-provider-ovn/blob/master/provider/scripts/...
>>>> instead.
>>>>
>>>> Can you please also share the output of
>>>> ovs-vsctl list Interface
>>>> on the host which produced the logfile above?
>>>>
>>>>
>>>>
>>>>
>>>>> The connections to ovn-sbctl is ok and the geneve tunnels are
>>>>> depicted under ovs-vsctl ok.
>>>>> VMs still not able to ping each other.
>>>>>
>>>>> On Tue, Sep 15, 2020 at 7:22 PM Dominik Holler
<dholler(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 15, 2020 at 6:18 PM Konstantinos Betsis <
>>>>>> k.betsis(a)gmail.com> wrote:
>>>>>>
>>>>>>> Hi Dominik
>>>>>>>
>>>>>>> Fixed the issue.
>>>>>>>
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>>> I believe the
/etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf
>>>>>>> needed update also.
>>>>>>> The package is upgraded to the latest version.
>>>>>>>
>>>>>>> Once the provider was updated with the following it
functioned
>>>>>>> perfectly:
>>>>>>>
>>>>>>> Name: ovirt-provider-ovn
>>>>>>> Description: oVirt network provider for OVN
>>>>>>> Type: External Network Provider
>>>>>>> Network Plugin: oVirt Network Provider for OVN
>>>>>>> Automatic Synchronization: Checked
>>>>>>> Unmanaged: Unchecked
>>>>>>> Provider URL: https:dc02-ovirt01.testdomain.com:9696
>>>>>>> Requires Authentication: Checked
>>>>>>> Username: admin@internal
>>>>>>> Password: "The admin password"
>>>>>>> Protocol: HTTPS
>>>>>>> Host Name: dc02-ovirt01.testdomain.com
>>>>>>> API Port: 35357
>>>>>>> API Version: v2.0
>>>>>>> Tenant Name: "Empty"
>>>>>>>
>>>>>>> For some reason the TLS certificate was in conflict with the
ovn
>>>>>>> provider details, i would bet the "host" entry.
>>>>>>>
>>>>>>> So now geneve tunnels are established.
>>>>>>> OVN provider is working.
>>>>>>>
>>>>>>> But VMs still do not communicated on the same VM network
spanning
>>>>>>> different hosts.
>>>>>>>
>>>>>>> So if we have a VM network test-net on both dc01-host01 and
>>>>>>> dc01-host02 and each host has a VM with IP addresses on the
same network,
>>>>>>> VMs on the same VM network should communicate directly.
>>>>>>> But traffic does not reach each other.
>>>>>>>
>>>>>>>
>>>>>> Can you create a new external network, with port security
disabled,
>>>>>> and an IPv4 subnet?
>>>>>> If the VMs get an IP address via DHCP, ovn is working, and should
be
>>>>>> able to ping each other, too.
>>>>>> If not, there should be a helpful entry in the ovn-controller.log
of
>>>>>> the host the VM is running.
>>>>>>
>>>>>>
>>>>>>> On Tue, Sep 15, 2020 at 7:07 PM Dominik Holler
<dholler(a)redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Can you try again with:
>>>>>>>>
>>>>>>>> [OVN REMOTE]
>>>>>>>> ovn-remote=ssl:127.0.0.1:6641
>>>>>>>> [SSL]
>>>>>>>> https-enabled=false
>>>>>>>> ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem
>>>>>>>>
ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer
>>>>>>>>
>>>>>>>>
ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass
>>>>>>>> [OVIRT]
>>>>>>>> ovirt-sso-client-secret=*random_test*
>>>>>>>> ovirt-host=https://dc02-ovirt01.testdomain.com:443
>>>>>>>> <https://dc02-ovirt01.testdomain.com/>
>>>>>>>> ovirt-sso-client-id=ovirt-provider-ovn
>>>>>>>> ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem
>>>>>>>> [NETWORK]
>>>>>>>> port-security-enabled-default=True
>>>>>>>> [PROVIDER]
>>>>>>>>
>>>>>>>> provider-host=dc02-ovirt01.testdomain.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Please note that the should match the HTTP or HTTPS in
the of the
>>>>>>>> ovirt-prover-ovn configuration in oVirt Engine.
>>>>>>>> So if the ovirt-provider-ovn entity in Engine is on HTTP,
the
>>>>>>>> config file should use
>>>>>>>> https-enabled=false
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Sep 15, 2020 at 5:56 PM Konstantinos Betsis <
>>>>>>>> k.betsis(a)gmail.com> wrote:
>>>>>>>>
>>>>>>>>> This is the updated one:
>>>>>>>>>
>>>>>>>>> # This file is automatically generated by
engine-setup. Please do
>>>>>>>>> not edit manually
>>>>>>>>> [OVN REMOTE]
>>>>>>>>> ovn-remote=ssl:127.0.0.1:6641
>>>>>>>>> [SSL]
>>>>>>>>> https-enabled=true
>>>>>>>>> ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem
>>>>>>>>>
ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer
>>>>>>>>>
>>>>>>>>>
ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass
>>>>>>>>> [OVIRT]
>>>>>>>>> ovirt-sso-client-secret=*random_text*
>>>>>>>>> ovirt-host=https://dc02-ovirt01.testdomain.com:443
>>>>>>>>> ovirt-sso-client-id=ovirt-provider-ovn
>>>>>>>>> ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem
>>>>>>>>> [NETWORK]
>>>>>>>>> port-security-enabled-default=True
>>>>>>>>> [PROVIDER]
>>>>>>>>> provider-host=dc02-ovirt01.testdomain.com
>>>>>>>>> [AUTH]
>>>>>>>>> auth-plugin=auth.plugins.static_token:NoAuthPlugin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> However, it still does not connect.
>>>>>>>>> It prompts for the certificate but then fails and
prompts to see
>>>>>>>>> the log but the ovirt-provider-ovn.log does not list
anything.
>>>>>>>>>
>>>>>>>>> Yes we've got ovirt for about a year now from
about version 4.1
>>>>>>>>>
>>>>>>>>>
>>>>>>>> This might explain the trouble. Upgrade of
ovirt-provider-ovn
>>>>>>>> should work flawlessly starting from oVirt 4.2.
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Tue, Sep 15, 2020 at 6:44 PM Dominik Holler <
>>>>>>>>> dholler(a)redhat.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 15, 2020 at 5:34 PM Konstantinos
Betsis <
>>>>>>>>>> k.betsis(a)gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> There is a file with the below entries
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Impressive, do you know when this config file was
created and if
>>>>>>>>>> it was manually modified?
>>>>>>>>>> Is this an upgrade from oVirt 4.1?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> [root@dc02-ovirt01 log]# cat
>>>>>>>>>>>
/etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf
>>>>>>>>>>> # This file is automatically generated by
engine-setup. Please
>>>>>>>>>>> do not edit manually
>>>>>>>>>>> [OVN REMOTE]
>>>>>>>>>>> ovn-remote=tcp:127.0.0.1:6641
>>>>>>>>>>> [SSL]
>>>>>>>>>>> https-enabled=false
>>>>>>>>>>> ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem
>>>>>>>>>>>
ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer
>>>>>>>>>>>
>>>>>>>>>>>
ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass
>>>>>>>>>>> [OVIRT]
>>>>>>>>>>> ovirt-sso-client-secret=*random_test*
>>>>>>>>>>>
ovirt-host=https://dc02-ovirt01.testdomain.com:443
>>>>>>>>>>> ovirt-sso-client-id=ovirt-provider-ovn
>>>>>>>>>>>
ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem
>>>>>>>>>>> [NETWORK]
>>>>>>>>>>> port-security-enabled-default=True
>>>>>>>>>>> [PROVIDER]
>>>>>>>>>>>
>>>>>>>>>>> provider-host=dc02-ovirt01.testdomain.com
>>>>>>>>>>>
>>>>>>>>>>> The only entry missing is the [AUTH] and
under [SSL] the
>>>>>>>>>>> https-enabled is false. Should I edit this in
this file or is this going to
>>>>>>>>>>> break everything?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Changing the file should improve, but better
create a backup
>>>>>>>>>> into another diretory before modification.
>>>>>>>>>> The only required change is
>>>>>>>>>> from
>>>>>>>>>> ovn-remote=tcp:127.0.0.1:6641
>>>>>>>>>> to
>>>>>>>>>> ovn-remote=ssl:127.0.0.1:6641
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 15, 2020 at 6:27 PM Dominik
Holler <
>>>>>>>>>>> dholler(a)redhat.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Sep 15, 2020 at 5:11 PM
Konstantinos Betsis <
>>>>>>>>>>>> k.betsis(a)gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Dominik
>>>>>>>>>>>>>
>>>>>>>>>>>>> That immediately fixed the geneve
tunnels between all hosts.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> thanks for the feedback.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> However, the ovn provider is not
broken.
>>>>>>>>>>>>> After fixing the networks we tried to
move a VM to the
>>>>>>>>>>>>> DC01-host01 so we powered it down and
simply configured it to run on
>>>>>>>>>>>>> dc01-node01.
>>>>>>>>>>>>>
>>>>>>>>>>>>> While checking the logs on the ovirt
engine i noticed the
>>>>>>>>>>>>> below:
>>>>>>>>>>>>> Failed to synchronize networks of
Provider ovirt-provider-ovn.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The ovn-provider configure on the
engine is the below:
>>>>>>>>>>>>> Name: ovirt-provider-ovn
>>>>>>>>>>>>> Description: oVirt network provider
for OVN
>>>>>>>>>>>>> Type: External Network Provider
>>>>>>>>>>>>> Network Plugin: oVirt Network
Provider for OVN
>>>>>>>>>>>>> Automatic Synchronization: Checked
>>>>>>>>>>>>> Unmanaged: Unchecked
>>>>>>>>>>>>> Provider URL: http:localhost:9696
>>>>>>>>>>>>> Requires Authentication: Checked
>>>>>>>>>>>>> Username: admin@internal
>>>>>>>>>>>>> Password: "The admin
password"
>>>>>>>>>>>>> Protocol: hTTP
>>>>>>>>>>>>> Host Name: dc02-ovirt01
>>>>>>>>>>>>> API Port: 35357
>>>>>>>>>>>>> API Version: v2.0
>>>>>>>>>>>>> Tenant Name: "Empty"
>>>>>>>>>>>>>
>>>>>>>>>>>>> In the past this was deleted by an
engineer and recreated as
>>>>>>>>>>>>> per the documentation, and it worked.
Do we need to update something due to
>>>>>>>>>>>>> the SSL on the ovn?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Is there a file in
/etc/ovirt-provider-ovn/conf.d/ ?
>>>>>>>>>>>> engine-setup should have created one.
>>>>>>>>>>>> If the file is missing, for testing
purposes, you can create a
>>>>>>>>>>>> file
/etc/ovirt-provider-ovn/conf.d/00-setup-ovirt-provider-ovn-test.conf :
>>>>>>>>>>>> [PROVIDER]
>>>>>>>>>>>> provider-host=REPLACE_WITH_FQDN
>>>>>>>>>>>> [SSL]
>>>>>>>>>>>>
>>>>>>>>>>>>
ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer
>>>>>>>>>>>>
>>>>>>>>>>>>
ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass
>>>>>>>>>>>>
ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem
>>>>>>>>>>>> https-enabled=true
>>>>>>>>>>>> [OVN REMOTE]
>>>>>>>>>>>> ovn-remote=ssl:127.0.0.1:6641
>>>>>>>>>>>> [AUTH]
>>>>>>>>>>>>
auth-plugin=auth.plugins.static_token:NoAuthPlugin
>>>>>>>>>>>> [NETWORK]
>>>>>>>>>>>> port-security-enabled-default=True
>>>>>>>>>>>>
>>>>>>>>>>>> and restart the ovirt-provider-ovn
service.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> From the ovn-provider logs the below
is generated after a
>>>>>>>>>>>>> service restart and when the start VM
is triggered
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2020-09-15 15:07:33,579 root Starting
server
>>>>>>>>>>>>> 2020-09-15 15:07:33,579 root Version:
1.2.29-1
>>>>>>>>>>>>> 2020-09-15 15:07:33,579 root Build
date: 20191217125241
>>>>>>>>>>>>> 2020-09-15 15:07:33,579 root Githash:
cb5a80d
>>>>>>>>>>>>> 2020-09-15 15:08:26,582 root From:
::ffff:127.0.0.1:59980
>>>>>>>>>>>>> Request: GET /v2.0/ports
>>>>>>>>>>>>> 2020-09-15 15:08:26,582 root Could
not retrieve schema from
>>>>>>>>>>>>> tcp:127.0.0.1:6641: Unknown error -1
>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>> File
>>>>>>>>>>>>>
"/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 138, in
>>>>>>>>>>>>> _handle_request
>>>>>>>>>>>>> method, path_parts, content
>>>>>>>>>>>>> File
>>>>>>>>>>>>>
"/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 175, in
>>>>>>>>>>>>> handle_request
>>>>>>>>>>>>> return
self.call_response_handler(handler, content,
>>>>>>>>>>>>> parameters)
>>>>>>>>>>>>> File
"/usr/share/ovirt-provider-ovn/handlers/neutron.py",
>>>>>>>>>>>>> line 35, in call_response_handler
>>>>>>>>>>>>> with NeutronApi() as ovn_north:
>>>>>>>>>>>>> File
>>>>>>>>>>>>>
"/usr/share/ovirt-provider-ovn/neutron/neutron_api.py", line 95, in __init__
>>>>>>>>>>>>> self.ovsidl, self.idl =
ovn_connection.connect()
>>>>>>>>>>>>> File
"/usr/share/ovirt-provider-ovn/ovn_connection.py",
>>>>>>>>>>>>> line 46, in connect
>>>>>>>>>>>>> ovnconst.OVN_NORTHBOUND
>>>>>>>>>>>>> File
>>>>>>>>>>>>>
"/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py",
>>>>>>>>>>>>> line 127, in from_server
>>>>>>>>>>>>> helper =
idlutils.get_schema_helper(connection_string,
>>>>>>>>>>>>> schema_name)
>>>>>>>>>>>>> File
>>>>>>>>>>>>>
"/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py",
>>>>>>>>>>>>> line 128, in get_schema_helper
>>>>>>>>>>>>> 'err':
os.strerror(err)})
>>>>>>>>>>>>> Exception: Could not retrieve schema
from tcp:127.0.0.1:6641:
>>>>>>>>>>>>> Unknown error -1
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> When i update the ovn provider from
the GUI to have
>>>>>>>>>>>>> https://localhost:9696/ and HTTPS as
the protocol the test
>>>>>>>>>>>>> fails.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Sep 15, 2020 at 5:35 PM
Dominik Holler <
>>>>>>>>>>>>> dholler(a)redhat.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Sep 14, 2020 at 9:25 AM
Konstantinos Betsis <
>>>>>>>>>>>>>> k.betsis(a)gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Dominik
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When these commands are used
on the ovirt-engine host the
>>>>>>>>>>>>>>> output is the one depicted in
your email.
>>>>>>>>>>>>>>> For your reference see also
below:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [root@ath01-ovirt01 certs]#
ovn-nbctl get-ssl
>>>>>>>>>>>>>>> Private key:
/etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass
>>>>>>>>>>>>>>> Certificate:
/etc/pki/ovirt-engine/certs/ovn-ndb.cer
>>>>>>>>>>>>>>> CA Certificate:
/etc/pki/ovirt-engine/ca.pem
>>>>>>>>>>>>>>> Bootstrap: false
>>>>>>>>>>>>>>> [root@ath01-ovirt01 certs]#
ovn-nbctl get-connection
>>>>>>>>>>>>>>> ptcp:6641
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [root@ath01-ovirt01 certs]#
ovn-sbctl get-ssl
>>>>>>>>>>>>>>> Private key:
/etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass
>>>>>>>>>>>>>>> Certificate:
/etc/pki/ovirt-engine/certs/ovn-sdb.cer
>>>>>>>>>>>>>>> CA Certificate:
/etc/pki/ovirt-engine/ca.pem
>>>>>>>>>>>>>>> Bootstrap: false
>>>>>>>>>>>>>>> [root@ath01-ovirt01 certs]#
ovn-sbctl get-connection
>>>>>>>>>>>>>>> read-write role=""
ptcp:6642
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ^^^ the line above points to the
problem: ovn-central is
>>>>>>>>>>>>>> configured to use plain TCP
without ssl.
>>>>>>>>>>>>>> engine-setup usually configures
ovn-central to use SSL. That
>>>>>>>>>>>>>> the files
/etc/pki/ovirt-engine/keys/ovn-* exist, shows,
>>>>>>>>>>>>>> that engine-setup was triggered
correctly. Looks like the
>>>>>>>>>>>>>> ovn db was dropped somehow, this
should not happen.
>>>>>>>>>>>>>> This can be fixed manually by
executing the following
>>>>>>>>>>>>>> commands on engine's
machine:
>>>>>>>>>>>>>> ovn-nbctl set-ssl
>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass
>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/certs/ovn-ndb.cer /etc/pki/ovirt-engine/ca.pem
>>>>>>>>>>>>>> ovn-nbctl set-connection
pssl:6641
>>>>>>>>>>>>>> ovn-sbctl set-ssl
>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass
>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/certs/ovn-sdb.cer /etc/pki/ovirt-engine/ca.pem
>>>>>>>>>>>>>> ovn-sbctl set-connection
pssl:6642
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The
/var/log/openvswitch/ovn-controller.log on the hosts
>>>>>>>>>>>>>> should tell that br-int.mgmt is
connected now.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [root@ath01-ovirt01 certs]#
ls -l
>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-*
>>>>>>>>>>>>>>> -rw-r-----. 1 root hugetlbfs
1828 Jun 25 11:08
>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass
>>>>>>>>>>>>>>> -rw-------. 1 root root
2893 Jun 25 11:08
>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-ndb.p12
>>>>>>>>>>>>>>> -rw-r-----. 1 root hugetlbfs
1828 Jun 25 11:08
>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass
>>>>>>>>>>>>>>> -rw-------. 1 root root
2893 Jun 25 11:08
>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-sdb.p12
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When i try the above commands
on the node hosts the
>>>>>>>>>>>>>>> following happens:
>>>>>>>>>>>>>>> ovn-nbctl get-ssl /
get-connection
>>>>>>>>>>>>>>> ovn-nbctl:
unix:/var/run/openvswitch/ovnnb_db.sock:
>>>>>>>>>>>>>>> database connection failed
(No such file or directory)
>>>>>>>>>>>>>>> The above i believe is
expected since no northbound
>>>>>>>>>>>>>>> connections should be
established from the host nodes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ovn-sbctl get-ssl
/get-connection
>>>>>>>>>>>>>>> The output is stuck till i
terminate it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, the ovn-* commands works
only on engine's machine,
>>>>>>>>>>>>>> which has the role ovn-central.
>>>>>>>>>>>>>> On the hosts, there is only the
ovn-controller, which
>>>>>>>>>>>>>> connects the ovn southbound to
openvswitch on the host.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For the requested logs the
below are found in the
>>>>>>>>>>>>>>> ovsdb-server-sb.log
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
2020-09-14T07:18:38.187Z|219636|reconnect|WARN|tcp:DC02-host01:33146:
>>>>>>>>>>>>>>> connection dropped (Protocol
error)
>>>>>>>>>>>>>>>
2020-09-14T07:18:41.946Z|219637|reconnect|WARN|tcp:DC01-host01:51188:
>>>>>>>>>>>>>>> connection dropped (Protocol
error)
>>>>>>>>>>>>>>>
2020-09-14T07:18:43.033Z|219638|reconnect|WARN|tcp:DC01-host02:37044:
>>>>>>>>>>>>>>> connection dropped (Protocol
error)
>>>>>>>>>>>>>>>
2020-09-14T07:18:46.198Z|219639|reconnect|WARN|tcp:DC02-host01:33148:
>>>>>>>>>>>>>>> connection dropped (Protocol
error)
>>>>>>>>>>>>>>>
2020-09-14T07:18:50.069Z|219640|jsonrpc|WARN|Dropped 4 log
>>>>>>>>>>>>>>> messages in last 12 seconds
(most recently, 4 seconds ago) due to excessive
>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>
2020-09-14T07:18:50.069Z|219641|jsonrpc|WARN|tcp:DC01-host01:51190:
>>>>>>>>>>>>>>> error parsing stream: line 0,
column 0, byte 0: invalid character U+0016
>>>>>>>>>>>>>>>
2020-09-14T07:18:50.069Z|219642|jsonrpc|WARN|Dropped 4 log
>>>>>>>>>>>>>>> messages in last 12 seconds
(most recently, 4 seconds ago) due to excessive
>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>
2020-09-14T07:18:50.069Z|219643|jsonrpc|WARN|tcp:DC01-host01:51190:
>>>>>>>>>>>>>>> received SSL data on JSON-RPC
channel
>>>>>>>>>>>>>>>
2020-09-14T07:18:50.070Z|219644|reconnect|WARN|tcp:DC01-host01:51190:
>>>>>>>>>>>>>>> connection dropped (Protocol
error)
>>>>>>>>>>>>>>>
2020-09-14T07:18:51.147Z|219645|reconnect|WARN|tcp:DC01-host02:37046:
>>>>>>>>>>>>>>> connection dropped (Protocol
error)
>>>>>>>>>>>>>>>
2020-09-14T07:18:54.209Z|219646|reconnect|WARN|tcp:DC02-host01:33150:
>>>>>>>>>>>>>>> connection dropped (Protocol
error)
>>>>>>>>>>>>>>>
2020-09-14T07:18:58.192Z|219647|reconnect|WARN|tcp:DC01-host01:51192:
>>>>>>>>>>>>>>> connection dropped (Protocol
error)
>>>>>>>>>>>>>>>
2020-09-14T07:18:59.262Z|219648|jsonrpc|WARN|Dropped 3 log
>>>>>>>>>>>>>>> messages in last 8 seconds
(most recently, 1 seconds ago) due to excessive
>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>
2020-09-14T07:18:59.262Z|219649|jsonrpc|WARN|tcp:DC01-host02:37048:
>>>>>>>>>>>>>>> error parsing stream: line 0,
column 0, byte 0: invalid character U+0016
>>>>>>>>>>>>>>>
2020-09-14T07:18:59.263Z|219650|jsonrpc|WARN|Dropped 3 log
>>>>>>>>>>>>>>> messages in last 8 seconds
(most recently, 1 seconds ago) due to excessive
>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>
2020-09-14T07:18:59.263Z|219651|jsonrpc|WARN|tcp:DC01-host02:37048:
>>>>>>>>>>>>>>> received SSL data on JSON-RPC
channel
>>>>>>>>>>>>>>>
2020-09-14T07:18:59.263Z|219652|reconnect|WARN|tcp:DC01-host02:37048:
>>>>>>>>>>>>>>> connection dropped (Protocol
error)
>>>>>>>>>>>>>>>
2020-09-14T07:19:02.220Z|219653|reconnect|WARN|tcp:DC02-host01:33152:
>>>>>>>>>>>>>>> connection dropped (Protocol
error)
>>>>>>>>>>>>>>>
2020-09-14T07:19:06.316Z|219654|reconnect|WARN|tcp:DC01-host01:51194:
>>>>>>>>>>>>>>> connection dropped (Protocol
error)
>>>>>>>>>>>>>>>
2020-09-14T07:19:07.386Z|219655|reconnect|WARN|tcp:DC01-host02:37050:
>>>>>>>>>>>>>>> connection dropped (Protocol
error)
>>>>>>>>>>>>>>>
2020-09-14T07:19:10.232Z|219656|reconnect|WARN|tcp:DC02-host01:33154:
>>>>>>>>>>>>>>> connection dropped (Protocol
error)
>>>>>>>>>>>>>>>
2020-09-14T07:19:14.439Z|219657|jsonrpc|WARN|Dropped 4 log
>>>>>>>>>>>>>>> messages in last 12 seconds
(most recently, 4 seconds ago) due to excessive
>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>
2020-09-14T07:19:14.439Z|219658|jsonrpc|WARN|tcp:DC01-host01:51196:
>>>>>>>>>>>>>>> error parsing stream: line 0,
column 0, byte 0: invalid character U+0016
>>>>>>>>>>>>>>>
2020-09-14T07:19:14.439Z|219659|jsonrpc|WARN|Dropped 4 log
>>>>>>>>>>>>>>> messages in last 12 seconds
(most recently, 4 seconds ago) due to excessive
>>>>>>>>>>>>>>> rate
>>>>>>>>>>>>>>>
2020-09-14T07:19:14.439Z|219660|jsonrpc|WARN|tcp:DC01-host01:51196:
>>>>>>>>>>>>>>> received SSL data on JSON-RPC
channel
>>>>>>>>>>>>>>>
2020-09-14T07:19:14.440Z|219661|reconnect|WARN|tcp:DC01-host01:51196:
>>>>>>>>>>>>>>> connection dropped (Protocol
error)
>>>>>>>>>>>>>>>
2020-09-14T07:19:15.505Z|219662|reconnect|WARN|tcp:DC01-host02:37052:
>>>>>>>>>>>>>>> connection dropped (Protocol
error)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> How can we fix these SSL
errors?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I addressed this above.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I thought vdsm did the
certificate provisioning on the host
>>>>>>>>>>>>>>> nodes as to communicate to
the engine host node.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, this seems to work in your
scenario, just the SSL
>>>>>>>>>>>>>> configuration on the ovn-central
was lost.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Sep 11, 2020 at 6:39
PM Dominik Holler <
>>>>>>>>>>>>>>> dholler(a)redhat.com>
wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Looks still like the
ovn-controller on the host
>>>>>>>>>>>>>>>> has problems
communicating with ovn-southbound.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Are there any hints in
/var/log/openvswitch/*.log,
>>>>>>>>>>>>>>>> especially in
/var/log/openvswitch/ovsdb-server-sb.log ?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you please check the
output of
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ovn-nbctl get-ssl
>>>>>>>>>>>>>>>> ovn-nbctl get-connection
>>>>>>>>>>>>>>>> ovn-sbctl get-ssl
>>>>>>>>>>>>>>>> ovn-sbctl get-connection
>>>>>>>>>>>>>>>> ls -l
/etc/pki/ovirt-engine/keys/ovn-*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> it should be similar to
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [root@ovirt-43 ~]#
ovn-nbctl get-ssl
>>>>>>>>>>>>>>>> Private key:
/etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass
>>>>>>>>>>>>>>>> Certificate:
/etc/pki/ovirt-engine/certs/ovn-ndb.cer
>>>>>>>>>>>>>>>> CA Certificate:
/etc/pki/ovirt-engine/ca.pem
>>>>>>>>>>>>>>>> Bootstrap: false
>>>>>>>>>>>>>>>> [root@ovirt-43 ~]#
ovn-nbctl get-connection
>>>>>>>>>>>>>>>> pssl:6641:[::]
>>>>>>>>>>>>>>>> [root@ovirt-43 ~]#
ovn-sbctl get-ssl
>>>>>>>>>>>>>>>> Private key:
/etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass
>>>>>>>>>>>>>>>> Certificate:
/etc/pki/ovirt-engine/certs/ovn-sdb.cer
>>>>>>>>>>>>>>>> CA Certificate:
/etc/pki/ovirt-engine/ca.pem
>>>>>>>>>>>>>>>> Bootstrap: false
>>>>>>>>>>>>>>>> [root@ovirt-43 ~]#
ovn-sbctl get-connection
>>>>>>>>>>>>>>>> read-write
role="" pssl:6642:[::]
>>>>>>>>>>>>>>>> [root@ovirt-43 ~]# ls -l
/etc/pki/ovirt-engine/keys/ovn-*
>>>>>>>>>>>>>>>> -rw-r-----. 1 root
hugetlbfs 1828 Oct 14 2019
>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-ndb.key.nopass
>>>>>>>>>>>>>>>> -rw-------. 1 root root
2709 Oct 14 2019
>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-ndb.p12
>>>>>>>>>>>>>>>> -rw-r-----. 1 root
hugetlbfs 1828 Oct 14 2019
>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-sdb.key.nopass
>>>>>>>>>>>>>>>> -rw-------. 1 root root
2709 Oct 14 2019
>>>>>>>>>>>>>>>>
/etc/pki/ovirt-engine/keys/ovn-sdb.p12
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Sep 11, 2020 at
1:10 PM Konstantinos Betsis <
>>>>>>>>>>>>>>>> k.betsis(a)gmail.com>
wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I did a restart of
the ovn-controller, this is the output
>>>>>>>>>>>>>>>>> of the
ovn-controller.log
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:07.566Z|00001|vlog|INFO|opened log file
>>>>>>>>>>>>>>>>>
/var/log/openvswitch/ovn-controller.log
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:07.568Z|00002|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
>>>>>>>>>>>>>>>>> connecting...
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:07.568Z|00003|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
>>>>>>>>>>>>>>>>> connected
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:07.570Z|00004|main|INFO|OVS IDL
>>>>>>>>>>>>>>>>> reconnected, force
recompute.
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:07.571Z|00005|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>> connecting...
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:07.571Z|00006|main|INFO|OVNSB IDL
>>>>>>>>>>>>>>>>> reconnected, force
recompute.
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:07.685Z|00007|stream_ssl|WARN|SSL_connect:
>>>>>>>>>>>>>>>>> unexpected SSL
connection close
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:07.685Z|00008|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>> connection attempt
failed (Protocol error)
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:08.685Z|00009|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>> connecting...
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:08.800Z|00010|stream_ssl|WARN|SSL_connect:
>>>>>>>>>>>>>>>>> unexpected SSL
connection close
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:08.800Z|00011|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>> connection attempt
failed (Protocol error)
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:08.800Z|00012|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>> waiting 2 seconds
before reconnect
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:10.802Z|00013|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>> connecting...
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:10.917Z|00014|stream_ssl|WARN|SSL_connect:
>>>>>>>>>>>>>>>>> unexpected SSL
connection close
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:10.917Z|00015|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>> connection attempt
failed (Protocol error)
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:10.917Z|00016|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>> waiting 4 seconds
before reconnect
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:14.921Z|00017|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>> connecting...
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:15.036Z|00018|stream_ssl|WARN|SSL_connect:
>>>>>>>>>>>>>>>>> unexpected SSL
connection close
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:15.036Z|00019|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>> connection attempt
failed (Protocol error)
>>>>>>>>>>>>>>>>>
2020-09-11T10:54:15.036Z|00020|reconnect|INFO|ssl:OVIRT_ENGINE_IP:6642:
>>>>>>>>>>>>>>>>> continuing to
reconnect in the background but suppressing further logging
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have also done the
vdsm-tool ovn-config OVIRT_ENGINE_IP
>>>>>>>>>>>>>>>>> OVIRTMGMT_NETWORK_DC
>>>>>>>>>>>>>>>>> This is how the
OVIRT_ENGINE_IP is provided in the ovn
>>>>>>>>>>>>>>>>> controller, i can
redo it if you wan.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> After the restart of
the ovn-controller the OVIRT ENGINE
>>>>>>>>>>>>>>>>> still shows only two
geneve connections one with DC01-host02 and
>>>>>>>>>>>>>>>>> DC02-host01.
>>>>>>>>>>>>>>>>> Chassis
"c4b23834-aec7-4bf8-8be7-aa94a50a6144"
>>>>>>>>>>>>>>>>> hostname:
"dc02-host01"
>>>>>>>>>>>>>>>>> Encap geneve
>>>>>>>>>>>>>>>>> ip:
"DC02-host01_IP"
>>>>>>>>>>>>>>>>> options:
{csum="true"}
>>>>>>>>>>>>>>>>> Chassis
"be3abcc9-7358-4040-a37b-8d8a782f239c"
>>>>>>>>>>>>>>>>> hostname:
"DC01-host02"
>>>>>>>>>>>>>>>>> Encap geneve
>>>>>>>>>>>>>>>>> ip:
"DC01-host02"
>>>>>>>>>>>>>>>>> options:
{csum="true"}
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've re-done the
vdsm-tool command and nothing
>>>>>>>>>>>>>>>>> changed....
again....with the same errors as the systemctl restart
>>>>>>>>>>>>>>>>> ovn-controller
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Sep 11, 2020
at 1:49 PM Dominik Holler <
>>>>>>>>>>>>>>>>>
dholler(a)redhat.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Please include
ovirt-users list in your reply, to share
>>>>>>>>>>>>>>>>>> the knowledge and
experience with the community!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Sep 11,
2020 at 12:12 PM Konstantinos Betsis <
>>>>>>>>>>>>>>>>>>
k.betsis(a)gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Ok below the
output per node and DC
>>>>>>>>>>>>>>>>>>> DC01
>>>>>>>>>>>>>>>>>>> node01
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
[root@dc01-node01 ~]# ovs-vsctl --no-wait get open .
>>>>>>>>>>>>>>>>>>>
external-ids:ovn-remote
>>>>>>>>>>>>>>>>>>>
"ssl:*OVIRT_ENGINE_IP*:6642"
>>>>>>>>>>>>>>>>>>> [root@
dc01-node01 ~]# ovs-vsctl --no-wait get open .
>>>>>>>>>>>>>>>>>>>
external-ids:ovn-encap-type
>>>>>>>>>>>>>>>>>>> geneve
>>>>>>>>>>>>>>>>>>> [root@
dc01-node01 ~]# ovs-vsctl --no-wait get open .
>>>>>>>>>>>>>>>>>>>
external-ids:ovn-encap-ip
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
"*OVIRTMGMT_IP_DC01-NODE01*"
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> node02
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
[root@dc01-node02 ~]# ovs-vsctl --no-wait get open .
>>>>>>>>>>>>>>>>>>>
external-ids:ovn-remote
>>>>>>>>>>>>>>>>>>>
"ssl:*OVIRT_ENGINE_IP*:6642"
>>>>>>>>>>>>>>>>>>> [root@
dc01-node02 ~]# ovs-vsctl --no-wait get open .
>>>>>>>>>>>>>>>>>>>
external-ids:ovn-encap-type
>>>>>>>>>>>>>>>>>>> geneve
>>>>>>>>>>>>>>>>>>> [root@
dc01-node02 ~]# ovs-vsctl --no-wait get open .
>>>>>>>>>>>>>>>>>>>
external-ids:ovn-encap-ip
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
"*OVIRTMGMT_IP_DC01-NODE02*"
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> DC02
>>>>>>>>>>>>>>>>>>> node01
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
[root@dc02-node01 ~]# ovs-vsctl --no-wait get open .
>>>>>>>>>>>>>>>>>>>
external-ids:ovn-remote
>>>>>>>>>>>>>>>>>>>
"ssl:*OVIRT_ENGINE_IP*:6642"
>>>>>>>>>>>>>>>>>>> [root@
dc02-node01 ~]# ovs-vsctl --no-wait get open .
>>>>>>>>>>>>>>>>>>>
external-ids:ovn-encap-type
>>>>>>>>>>>>>>>>>>> geneve
>>>>>>>>>>>>>>>>>>> [root@
dc02-node01 ~]# ovs-vsctl --no-wait get open .
>>>>>>>>>>>>>>>>>>>
external-ids:ovn-encap-ip
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
"*OVIRTMGMT_IP_DC02-NODE01*"
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Looks good.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> DC01 node01
and node02 share the same VM networks and
>>>>>>>>>>>>>>>>>>> VMs deployed
on top of them cannot talk to VM on the other hypervisor.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Maybe there is a
hint on ovn-controller.log on
>>>>>>>>>>>>>>>>>> dc01-node02 ?
Maybe restarting ovn-controller creates more helpful log
>>>>>>>>>>>>>>>>>> messages?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> You can also try
restart the ovn configuration on all
>>>>>>>>>>>>>>>>>> hosts by
executing
>>>>>>>>>>>>>>>>>> vdsm-tool
ovn-config OVIRT_ENGINE_IP LOCAL_OVIRTMGMT_IP
>>>>>>>>>>>>>>>>>> on each host,
this would trigger
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
https://github.com/oVirt/ovirt-provider-ovn/blob/master/driver/scripts/se...
>>>>>>>>>>>>>>>>>> internally.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> So I would
expect to see the same output for node01 to
>>>>>>>>>>>>>>>>>>> have a geneve
tunnel to node02 and vice versa.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Me too.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Sep
11, 2020 at 12:14 PM Dominik Holler <
>>>>>>>>>>>>>>>>>>>
dholler(a)redhat.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri,
Sep 11, 2020 at 10:53 AM Konstantinos Betsis <
>>>>>>>>>>>>>>>>>>>>
k.betsis(a)gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi
Dominik
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> OVN
is selected as the default network provider on
>>>>>>>>>>>>>>>>>>>>> the
clusters and the hosts.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> sounds
good.
>>>>>>>>>>>>>>>>>>>> This
configuration is required already during the host
>>>>>>>>>>>>>>>>>>>> is added
to oVirt Engine, because OVN is configured during this step.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The
"ovn-sbctl show" works on the ovirt engine and
>>>>>>>>>>>>>>>>>>>>> shows
only two hosts, 1 per DC.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
Chassis "c4b23834-aec7-4bf8-8be7-aa94a50a6144"
>>>>>>>>>>>>>>>>>>>>>
hostname: "dc01-node02"
>>>>>>>>>>>>>>>>>>>>>
Encap geneve
>>>>>>>>>>>>>>>>>>>>>
ip: "X.X.X.X"
>>>>>>>>>>>>>>>>>>>>>
options: {csum="true"}
>>>>>>>>>>>>>>>>>>>>>
Chassis "be3abcc9-7358-4040-a37b-8d8a782f239c"
>>>>>>>>>>>>>>>>>>>>>
hostname: "dc02-node1"
>>>>>>>>>>>>>>>>>>>>>
Encap geneve
>>>>>>>>>>>>>>>>>>>>>
ip: "A.A.A.A"
>>>>>>>>>>>>>>>>>>>>>
options: {csum="true"}
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The
new node is not listed (dc01-node1).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> When
executed on the nodes the same command
>>>>>>>>>>>>>>>>>>>>>
(ovn-sbctl show) times-out on all nodes.....
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The
output of the
>>>>>>>>>>>>>>>>>>>>>
/var/log/openvswitch/ovn-conntroller.log lists on all logs
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
2020-09-11T08:46:55.197Z|07361|stream_ssl|WARN|SSL_connect:
>>>>>>>>>>>>>>>>>>>>>
unexpected SSL connection close
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Can you
please compare the output of
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ovs-vsctl
--no-wait get open . external-ids:ovn-remote
>>>>>>>>>>>>>>>>>>>> ovs-vsctl
--no-wait get open .
>>>>>>>>>>>>>>>>>>>>
external-ids:ovn-encap-type
>>>>>>>>>>>>>>>>>>>> ovs-vsctl
--no-wait get open .
>>>>>>>>>>>>>>>>>>>>
external-ids:ovn-encap-ip
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> of the
working hosts, e.g. dc01-node02, and the
>>>>>>>>>>>>>>>>>>>> failing
host dc01-node1?
>>>>>>>>>>>>>>>>>>>> This
should point us the relevant difference in the
>>>>>>>>>>>>>>>>>>>>
configuration.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Please
include ovirt-users list in your replay, to
>>>>>>>>>>>>>>>>>>>> share the
knowledge and experience with the community.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thank
you
>>>>>>>>>>>>>>>>>>>>> Best
regards
>>>>>>>>>>>>>>>>>>>>>
Konstantinos Betsis
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On
Fri, Sep 11, 2020 at 11:01 AM Dominik Holler <
>>>>>>>>>>>>>>>>>>>>>
dholler(a)redhat.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
On Thu, Sep 10, 2020 at 6:26 PM Konstantinos B <
>>>>>>>>>>>>>>>>>>>>>>
k.betsis(a)gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
Hi all
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
We have a small installation based on OVIRT 4.3.
>>>>>>>>>>>>>>>>>>>>>>>
1 Cluster is based on Centos 7 and the other on
>>>>>>>>>>>>>>>>>>>>>>>
OVIRT NG Node image.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
The environment was stable till an upgrade took
>>>>>>>>>>>>>>>>>>>>>>>
place a couple of months ago.
>>>>>>>>>>>>>>>>>>>>>>>
As such we had to re-install one of the Centos 7
>>>>>>>>>>>>>>>>>>>>>>>
node and start from scratch.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
To trigger the automatic configuration of the host,
>>>>>>>>>>>>>>>>>>>>>>
it is required to configure ovirt-provider-ovn as the default network
>>>>>>>>>>>>>>>>>>>>>>
provider for the cluster before adding the host to oVirt.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
Even though the installation completed successfully
>>>>>>>>>>>>>>>>>>>>>>>
and VMs are created, the following are not working as expected:
>>>>>>>>>>>>>>>>>>>>>>>
1. ovn geneve tunnels are not established with the
>>>>>>>>>>>>>>>>>>>>>>>
other Centos 7 node in the cluster.
>>>>>>>>>>>>>>>>>>>>>>>
2. Centos 7 node is configured by ovirt engine
>>>>>>>>>>>>>>>>>>>>>>>
however no geneve tunnel is established when "ovn-sbctl show" is issued on
>>>>>>>>>>>>>>>>>>>>>>>
the engine.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
Does "ovn-sbctl show" list the hosts?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
3. no flows are shown on the engine on port 6642
>>>>>>>>>>>>>>>>>>>>>>>
for the ovs db.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
Does anyone have any experience on how to
>>>>>>>>>>>>>>>>>>>>>>>
troubleshoot OVN on ovirt?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
/var/log/openvswitch/ovncontroller.log on the host
>>>>>>>>>>>>>>>>>>>>>>
should contain a helpful hint.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
Thank you
>>>>>>>>>>>>>>>>>>>>>>>
_______________________________________________
>>>>>>>>>>>>>>>>>>>>>>>
Users mailing list -- users(a)ovirt.org
>>>>>>>>>>>>>>>>>>>>>>>
To unsubscribe send an email to
>>>>>>>>>>>>>>>>>>>>>>>
users-leave(a)ovirt.org
>>>>>>>>>>>>>>>>>>>>>>>
Privacy Statement:
>>>>>>>>>>>>>>>>>>>>>>>
https://www.ovirt.org/privacy-policy.html
>>>>>>>>>>>>>>>>>>>>>>>
oVirt Code of Conduct:
>>>>>>>>>>>>>>>>>>>>>>>
https://www.ovirt.org/community/about/community-guidelines/
>>>>>>>>>>>>>>>>>>>>>>>
List Archives:
>>>>>>>>>>>>>>>>>>>>>>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/LBVGLQJBWJF...
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>