On Mon, Mar 18, 2019 at 5:08 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Mon, Mar 18, 2019 at 4:40 PM Miguel Duarte de Mora Barroso < mdbarroso@redhat.com> wrote:

...

On Mon, Mar 18, 2019 at 2:20 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

...

Hello, passing from old manual to current OVN in 4.3.1 it seems I have some

problems with OVN now.

...

I cannot assign network on OVN to VM (powered on or off doesn't change). When I add//edit a vnic, they are not on the possible choices Environment composed by three hosts and one engine (external on vSphere). The mgmt network during time has been configured on network named ovirtmgmntZ2Z3 On engine it seems there are 2 switches for every defined ovn network (ovn192 and ovn172) Below some output of commands in case any inconsistency has remained and I can purge it. Thanks in advance.

I'm very confused here; you mention that on engine there are 2 switches for every ovn network, but, on your ovn-nbctl list logical_switch output I can clearly see the 2 logical switches where the OVN logical networks are stored. Who created those ?

I think it could be related to the situation described here (it is the same environment, in the meantime updated also from 4.2.8 to 4.3.1) and previous configuration not backed up at that time:

https://lists.ovirt.org/archives/list/users@ovirt.org/message/32S5L4JKHGPHE2...

and some steps not done correctly by me. After following indications, I tried to import ovn but probably I did it wrong.

Is it possible that you added new networks, instead of importing the old ones? If so the old networks would just stay in the database, and we would have duplicated networks like you have now.

...

Could you show us the properties of those 2 networks ? (e.g. ovn-nbctl list logical_switch 32367d8a-460f-4447-b35a-abe9ea5187e0 & ovn-nbctl list logical_switch 64c4c17f-cd67-4e29-939e-2b952495159f)

[root@ovmgr1 ~]# ovn-nbctl list logical_switch 32367d8a-460f-4447-b35a-abe9ea5187e0 _uuid : 32367d8a-460f-4447-b35a-abe9ea5187e0 acls : [] dns_records : [] external_ids : {} load_balancer : [] name : "ovn192" other_config : {subnet="192.168.10.0/24"} ports : [affc5570-3e5a-439c-9fdf-d75d6810e3a3, f639d541-2118-4c24-b478-b7a586eb170c] qos_rules : [] [root@ovmgr1 ~]#

[root@ovmgr1 ~]# ovn-nbctl list logical_switch 64c4c17f-cd67-4e29-939e-2b952495159f _uuid : 64c4c17f-cd67-4e29-939e-2b952495159f acls : [] dns_records : [] external_ids : {} load_balancer : [] name : "ovn172" other_config : {subnet="172.16.10.0/24"} ports : [32c348d9-12e9-4bcf-a43f-69338c887cfc, 3c77c2ea-de00-43f9-a5c5-9b3ffea5ec69] qos_rules : [] [root@ovmgr1 ~]#

...

@Gianluca Cecchi , I notice that one of your duplicate networks - 'ovn192' - has no ports attached. That makes it the perfect candidate to be deleted, and see if it becomes 'listable' on engine. That would help rule out the 'duplicate name' theory.

I can try. Can you give me the command to be run? It is a test oVirt so It would be not a big problem in case of failures in this respect.

...

At the moment, I can't think of a better alternative. Let's see if Marcin comes up with a better test / idea / alternative.

Also, please let us know the version of the ovirt-provider-ovn, openvswitch-ovn-central, and openvswitch-ovn-host.

On engine: [root@ovmgr1 ~]# rpm -q ovirt-provider-ovn openvswitch-ovn-central openvswitch-ovn-host ovirt-provider-ovn-1.2.20-1.el7.noarch openvswitch-ovn-central-2.10.1-3.el7.x86_64 package openvswitch-ovn-host is not installed [root@ovmgr1 ~]#

On the 3 hosts I only have this package installed: openvswitch-ovn-host-2.10.1-3.el7.x86_64

Thanks Gianluca

On Mon, Mar 18, 2019 at 5:08 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Mon, Mar 18, 2019 at 4:40 PM Miguel Duarte de Mora Barroso <mdbarroso@redhat.com> wrote:

...

On Mon, Mar 18, 2019 at 2:20 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

...

Hello, passing from old manual to current OVN in 4.3.1 it seems I have some problems with OVN now. I cannot assign network on OVN to VM (powered on or off doesn't change). When I add//edit a vnic, they are not on the possible choices Environment composed by three hosts and one engine (external on vSphere). The mgmt network during time has been configured on network named ovirtmgmntZ2Z3 On engine it seems there are 2 switches for every defined ovn network (ovn192 and ovn172) Below some output of commands in case any inconsistency has remained and I can purge it. Thanks in advance.

I'm very confused here; you mention that on engine there are 2 switches for every ovn network, but, on your ovn-nbctl list logical_switch output I can clearly see the 2 logical switches where the OVN logical networks are stored. Who created those ?

I think it could be related to the situation described here (it is the same environment, in the meantime updated also from 4.2.8 to 4.3.1) and previous configuration not backed up at that time: https://lists.ovirt.org/archives/list/users@ovirt.org/message/32S5L4JKHGPHE2...

and some steps not done correctly by me. After following indications, I tried to import ovn but probably I did it wrong.

...

Could you show us the properties of those 2 networks ? (e.g. ovn-nbctl list logical_switch 32367d8a-460f-4447-b35a-abe9ea5187e0 & ovn-nbctl list logical_switch 64c4c17f-cd67-4e29-939e-2b952495159f)

[root@ovmgr1 ~]# ovn-nbctl list logical_switch 32367d8a-460f-4447-b35a-abe9ea5187e0 _uuid : 32367d8a-460f-4447-b35a-abe9ea5187e0 acls : [] dns_records : [] external_ids : {} load_balancer : [] name : "ovn192" other_config : {subnet="192.168.10.0/24"} ports : [affc5570-3e5a-439c-9fdf-d75d6810e3a3, f639d541-2118-4c24-b478-b7a586eb170c] qos_rules : [] [root@ovmgr1 ~]#

[root@ovmgr1 ~]# ovn-nbctl list logical_switch 64c4c17f-cd67-4e29-939e-2b952495159f _uuid : 64c4c17f-cd67-4e29-939e-2b952495159f acls : [] dns_records : [] external_ids : {} load_balancer : [] name : "ovn172" other_config : {subnet="172.16.10.0/24"} ports : [32c348d9-12e9-4bcf-a43f-69338c887cfc, 3c77c2ea-de00-43f9-a5c5-9b3ffea5ec69] qos_rules : [] [root@ovmgr1 ~]#

...

@Gianluca Cecchi , I notice that one of your duplicate networks - 'ovn192' - has no ports attached. That makes it the perfect candidate to be deleted, and see if it becomes 'listable' on engine. That would help rule out the 'duplicate name' theory.

I can try. Can you give me the command to be run? It is a test oVirt so It would be not a big problem in case of failures in this respect.

You can delete it via the UI; just be sure to delete the one without ports - it's external ID is 6110649a-db2b-4de7-8fbc-601095cfe510. It will ask you if you also want to delete it from the external provider, say yes.

...

At the moment, I can't think of a better alternative. Let's see if Marcin comes up with a better test / idea / alternative.

Also, please let us know the version of the ovirt-provider-ovn, openvswitch-ovn-central, and openvswitch-ovn-host.

On engine: [root@ovmgr1 ~]# rpm -q ovirt-provider-ovn openvswitch-ovn-central openvswitch-ovn-host ovirt-provider-ovn-1.2.20-1.el7.noarch openvswitch-ovn-central-2.10.1-3.el7.x86_64 package openvswitch-ovn-host is not installed [root@ovmgr1 ~]#

On the 3 hosts I only have this package installed: openvswitch-ovn-host-2.10.1-3.el7.x86_64

Thanks Gianluca

On Tue, Mar 19, 2019 at 2:15 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Tue, Mar 19, 2019 at 10:25 AM Miguel Duarte de Mora Barroso <mdbarroso@redhat.com> wrote:

...

...

...

@Gianluca Cecchi , I notice that one of your duplicate networks - 'ovn192' - has no ports attached. That makes it the perfect candidate to be deleted, and see if it becomes 'listable' on engine. That would help rule out the 'duplicate name' theory.

I can try. Can you give me the command to be run? It is a test oVirt so It would be not a big problem in case of failures in this respect.

You can delete it via the UI; just be sure to delete the one without ports - it's external ID is 6110649a-db2b-4de7-8fbc-601095cfe510.

It will ask you if you also want to delete it from the external provider, say yes.

Inside the GUI I see only one ovn192 network and one ovn172 network and their external ids don't match the ones without ports...

- ovn192 Id: 8fd63a10-a2ba-4c56-a8e0-0bc8d70be8b5 External ID: 32367d8a-460f-4447-b35a-abe9ea5187e0

- ovn172 Id: 7546d5d3-a0e3-40d5-9d22-cf355da47d3a External ID: 64c4c17f-cd67-4e29-939e-2b952495159f

So I think I have to delete from command line

Check pastebin [0], with it you can safely delete those 2 networks. Last course of action would be to delete via ovn-nbctl - e.g. ovn-nbctl destroy logical_switch <network_id> - but hopefully it won't come to that. [0] - https://paste.fedoraproject.org/paste/mxVUEJZWxG-QHX0mJO1VhA

Gianluca Cecchi

On Tue, Mar 19, 2019 at 3:09 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Tue, Mar 19, 2019 at 2:51 PM Miguel Duarte de Mora Barroso <mdbarroso@redhat.com> wrote:

...

On Tue, Mar 19, 2019 at 2:15 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

...

On Tue, Mar 19, 2019 at 10:25 AM Miguel Duarte de Mora Barroso <mdbarroso@redhat.com> wrote:

...

...

...

@Gianluca Cecchi , I notice that one of your duplicate networks - 'ovn192' - has no ports attached. That makes it the perfect candidate to be deleted, and see if it becomes 'listable' on engine. That would help rule out the 'duplicate name' theory.

I can try. Can you give me the command to be run? It is a test oVirt so It would be not a big problem in case of failures in this respect.

You can delete it via the UI; just be sure to delete the one without ports - it's external ID is 6110649a-db2b-4de7-8fbc-601095cfe510.

It will ask you if you also want to delete it from the external provider, say yes.

Inside the GUI I see only one ovn192 network and one ovn172 network and their external ids don't match the ones without ports...

- ovn192 Id: 8fd63a10-a2ba-4c56-a8e0-0bc8d70be8b5 External ID: 32367d8a-460f-4447-b35a-abe9ea5187e0

- ovn172 Id: 7546d5d3-a0e3-40d5-9d22-cf355da47d3a External ID: 64c4c17f-cd67-4e29-939e-2b952495159f

So I think I have to delete from command line

Check pastebin [0], with it you can safely delete those 2 networks. Last course of action would be to delete via ovn-nbctl - e.g. ovn-nbctl destroy logical_switch <network_id> - but hopefully it won't come to that.

[0] - https://paste.fedoraproject.org/paste/mxVUEJZWxG-QHX0mJO1VhA

...

Gianluca Cecchi

I get this error from the first part where I should get the token id { "error": { "message": "No JSON object could be decoded", "code": 400, "title": "Bad Request" } }

In your command there is:

-H 'Postman-Token: 87fa50fd-0d06-497d-b2ac-b66b78ad90b8' \

Remove that, sorry for not noticing it before. Also get rid of the 'Cache-Control: no-cache' header. The request thus becomes: curl -k -X POST \ https://localhost:35357/v2.0/tokens \ -H 'Content-Type: application/json' \ -d '{ "auth": { "passwordCredentials": { "username": <your_username@your_domain>, "password": <your_password> } } } '

what is that sequence? where did you get it? Also, inside the credential section

"username": XXXX, "password": YYY

do I have to put my username and password inside single/double quotes or nothing? eg admin@internal or "admin@internal" or what?

Between quotes - e.g. "admin@internal" and "whatever-password-you-have".

Thanks, Gianluca

On Tue, Mar 19, 2019 at 4:44 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Tue, Mar 19, 2019 at 4:31 PM Miguel Duarte de Mora Barroso < mdbarroso@redhat.com> wrote:

[snip]

...

...

...

...

...

>> @Gianluca Cecchi , I notice that one of your duplicate networks - >> 'ovn192' - has no ports attached. That makes it the perfect candidate >> to be deleted, and see if it becomes 'listable' on engine. That would >> help rule out the 'duplicate name' theory. > > > I can try. Can you give me the command to be run? > It is a test oVirt so It would be not a big problem in case of failures in this respect.

You can delete it via the UI; just be sure to delete the one without ports - it's external ID is 6110649a-db2b-4de7-8fbc-601095cfe510.

It will ask you if you also want to delete it from the external provider, say yes.

Inside the GUI I see only one ovn192 network and one ovn172 network and their external ids don't match the ones without ports...

- ovn192 Id: 8fd63a10-a2ba-4c56-a8e0-0bc8d70be8b5 External ID: 32367d8a-460f-4447-b35a-abe9ea5187e0

- ovn172 Id: 7546d5d3-a0e3-40d5-9d22-cf355da47d3a External ID: 64c4c17f-cd67-4e29-939e-2b952495159f

So I think I have to delete from command line

Check pastebin [0], with it you can safely delete those 2 networks. Last course of action would be to delete via ovn-nbctl - e.g. ovn-nbctl destroy logical_switch <network_id> - but hopefully it won't come to that.

[0] - https://paste.fedoraproject.org/paste/mxVUEJZWxG-QHX0mJO1VhA

I get "not found" for both:

[root@ovmgr1 ~]# curl -k -X DELETE ' https://localhost:9696/v2/networks/6110649a-db2b-4de7-8fbc-601095cfe510' -H 'X-Auth-Token: WyutJuakjpSzJ4nj7drptpDfbAb3sKcZWvhF3NqRVXRyUpIHz9QGG_ZeeLi7u7trv7Er2D3vAcSX9LIFpXzz7w' { "error": { "message": "Cannot find Logical_Switch with name=6110649a-db2b-4de7-8fbc-601095cfe510", "code": 404, "title": "Not Found" } } [root@ovmgr1 ~]# curl -k -X DELETE ' https://localhost:9696/v2/networks/8fd63a10-a2ba-4c56-a8e0-0bc8d70be8b5' -H 'X-Auth-Token: WyutJuakjpSzJ4nj7drptpDfbAb3sKcZWvhF3NqRVXRyUpIHz9QGG_ZeeLi7u7trv7Er2D3vAcSX9LIFpXzz7w' { "error": { "message": "Cannot find Logical_Switch with name=8fd63a10-a2ba-4c56-a8e0-0bc8d70be8b5", "code": 404, "title": "Not Found" } } [root@ovmgr1 ~]#

Is there a command to get the supposed list?

Thanks for your help. I'm also available to completely reset the OVN config if there is a way for it...

Gianluca

A GET call outputs this information : [root@ovmgr1 ~]# curl -k -X GET 'https://localhost:9696/v2/networks' -H 'X-Auth-Token: WyutJuakjpSzJ4nj7drptpDfbAb3sKcZWvhF3NqRVXRyUpIHz9QGG_ZeeLi7u7trv7Er2D3vAcSX9LIFpXzz7w' {"networks": [{"status": "ACTIVE", "name": "ovn172", "tenant_id": "00000000000000000000000000000001", "mtu": 1442, "port_security_enabled": false, "id": "64c4c17f-cd67-4e29-939e-2b952495159f"}, {"status": "ACTIVE", "name": "ovn172", "tenant_id": "00000000000000000000000000000001", "mtu": 1442, "port_security_enabled": false, "id": "04501f6b-3977-4ba1-9ead-7096768d796d"}, {"status": "ACTIVE", "name": "ovn192", "tenant_id": "00000000000000000000000000000001", "mtu": 1442, "port_security_enabled": false, "id": "32367d8a-460f-4447-b35a-abe9ea5187e0"}]}[root@ovmgr1 ~]# [root@ovmgr1 ~]#

On Wed, Mar 20, 2019 at 1:26 PM Marcin Mirecki <mmirecki@redhat.com> wrote:

Looking at the original state we had: switch 32367d8a-460f-4447-b35a-abe9ea5187e0 (ovn192) switch 6110649a-db2b-4de7-8fbc-601095cfe510 (ovn192) switch 64c4c17f-cd67-4e29-939e-2b952495159f (ovn172) switch 04501f6b-3977-4ba1-9ead-7096768d796d (ovn172)

In the output of GET, 6110649a-db2b-4de7-8fbc-601095cfe510 is not longer there, so it has been deleted. Did you maybe try to submit the request twice?

With that switch, as it had no ports attached, I tried the command line option with: ovn-nbctl destroy logical_switch 6110649a-db2b-4de7-8fbc-601095cfe510

About 8fd63a10-a2ba-4c56-a8e0-0bc8d70be8b5. There was never a network with that id, so this is correct.

Yes, but that was the id provided by web admin gui for the network.... - ovn192 Id: 8fd63a10-a2ba-4c56-a8e0-0bc8d70be8b5 External ID: 32367d8a-460f-4447-b35a-abe9ea5187e0 or did I misunderstood?

Also note that to delete a network you will first have to delete its ports.

OK. Is there a command to clean all so that I can restart with a new OVN setup in this infra? I think I messed up too many things on it.... Thanks, Gianluca

On Thu, Mar 21, 2019 at 9:31 AM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Wed, Mar 20, 2019 at 2:09 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

...

On Wed, Mar 20, 2019 at 1:26 PM Marcin Mirecki <mmirecki@redhat.com> wrote:

...

Looking at the original state we had: switch 32367d8a-460f-4447-b35a-abe9ea5187e0 (ovn192) switch 6110649a-db2b-4de7-8fbc-601095cfe510 (ovn192) switch 64c4c17f-cd67-4e29-939e-2b952495159f (ovn172) switch 04501f6b-3977-4ba1-9ead-7096768d796d (ovn172)

In the output of GET, 6110649a-db2b-4de7-8fbc-601095cfe510 is not longer there, so it has been deleted. Did you maybe try to submit the request twice?

With that switch, as it had no ports attached, I tried the command line option with: ovn-nbctl destroy logical_switch 6110649a-db2b-4de7-8fbc-601095cfe510

...

About 8fd63a10-a2ba-4c56-a8e0-0bc8d70be8b5. There was never a network with that id, so this is correct.

Yes, but that was the id provided by web admin gui for the network.... - ovn192 Id: 8fd63a10-a2ba-4c56-a8e0-0bc8d70be8b5 External ID: 32367d8a-460f-4447-b35a-abe9ea5187e0

or did I misunderstood?

...

Also note that to delete a network you will first have to delete its ports.

OK. Is there a command to clean all so that I can restart with a new OVN setup in this infra? I think I messed up too many things on it....

Thanks, Gianluca

I have deleted provider from web admin gui and then on manager from command line:

ovn-nbctl lsp-del <port_name> for the ports defined and then ovn-nbctl destroy logical_switch <switch_name> for the defined switches.

Then reboot of manager.

Now I have on it: [root@ovmgr1 ~]# ovs-vsctl show eae54ff9-b86c-4050-8241-46f44336ba94 ovs_version: "2.10.1" [root@ovmgr1 ~]#

[root@ovmgr1 ~]# ovn-nbctl show [root@ovmgr1 ~]#

and no provider and/or networks on OVN in web admin gui.

What could be the sequence to re-add an OVN provider now? From engine-setup or from web admin gui?

Through engine UI, through "Administration" -> "Providers" -> Add

Any quick tip for? Any other commands (eg at db level) to verify all previous config is cleaned?

+Ales Musil could you indicate how to check the network list on the engine's DB ? A query to filter all the external networks is what we're after.

Thanks Gianluca

On Thu, Mar 21, 2019 at 11:16 AM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Thu, Mar 21, 2019 at 9:43 AM Miguel Duarte de Mora Barroso < mdbarroso@redhat.com> wrote:

...

...

...

OK. Is there a command to clean all so that I can restart with a new OVN

setup in this infra?

...

I think I messed up too many things on it....

Thanks, Gianluca

I have deleted provider from web admin gui and then on manager from command line:

ovn-nbctl lsp-del <port_name> for the ports defined and then ovn-nbctl destroy logical_switch <switch_name> for the defined switches.

Then reboot of manager.

Now I have on it: [root@ovmgr1 ~]# ovs-vsctl show eae54ff9-b86c-4050-8241-46f44336ba94 ovs_version: "2.10.1" [root@ovmgr1 ~]#

[root@ovmgr1 ~]# ovn-nbctl show [root@ovmgr1 ~]#

and no provider and/or networks on OVN in web admin gui.

What could be the sequence to re-add an OVN provider now? From engine-setup or from web admin gui?

Through engine UI, through "Administration" -> "Providers" -> Add

...

Any quick tip for? Any other commands (eg at db level) to verify all previous config is cleaned?

+Ales Musil could you indicate how to check the network list on the engine's DB ? A query to filter all the external networks is what we're after.

Hi, simply "select * from network where provider_network_external_id is not null" should do the trick.

...

...

Thanks Gianluca

In the mean time tested this on database, searching around tables:

engine=# select name from providers; name ------------------------ ovirt-image-repository (1 row)

engine=#

engine=# select provider_network_provider_id,provider_network_external_id,provider_physical_network_id from network; provider_network_provider_id | provider_network_external_id | provider_physical_network_id

------------------------------+------------------------------+------------------------------ | | | | | | | | | | | | | | | | | | (9 rows)

engine=# select provider_network_provider_id,provider_network_external_id,provider_physical_network_id from network_view ; provider_network_provider_id | provider_network_external_id | provider_physical_network_id

------------------------------+------------------------------+------------------------------ | | | | | | | | | | | | | | | | | | (9 rows)

engine=# select * from provider_binding_host_id; vds_id | plugin_type | binding_host_id

--------------------------------------+--------------------+-------------------------------------- 8ef1ce6f-4e38-486c-b3a4-58235f1f1d06 | OVIRT_PROVIDER_OVN | b8872ab5-4606-4a79-b77d-9d956a18d349 9001bfea-d7d8-4ae4-aeaf-14a5e2d88d77 | OVIRT_PROVIDER_OVN | ddecf0da-4708-4f93-958b-6af365a5eeca d16e723c-b44c-4c1c-be76-c67911e47ccd | OVIRT_PROVIDER_OVN | 1dce5b7c-a9fc-4ddb-99b4-e2c9e0fa54c5 (3 rows)

The last seems to me the configuration of my 3 hosts... do I need to clean in any way it or can I say that it is a sort of default empty config without any network configured?

Thanks, Gianluca

-- ALES MUSIL Associate Software Engineer - rhv network Red Hat EMEA <https://www.redhat.com/> amusil@redhat.com IM: amusil <https://red.ht/sig>

On Thu, Mar 21, 2019 at 2:04 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Thu, Mar 21, 2019 at 12:39 PM Ales Musil <amusil@redhat.com> wrote:

...

Hi, simply "select * from network where provider_network_external_id is not null" should do the trick.

Thanks. As you can see one of my queries detailed no values:

...

engine=# select

...

provider_network_provider_id,provider_network_external_id,provider_physical_network_id from network; provider_network_provider_id | provider_network_external_id | provider_physical_network_id

------------------------------+------------------------------+------------------------------ | | | | | | | | | | | | | | | | | | (9 rows)

anyway:

engine=# select * from network where provider_network_external_id is not null; id | name | description | type | addr | subnet | gateway | vlan_id | stp | storage_pool_id | mtu | vm_network | provider_network_provider_id | provider_network_external_id | free_text_comment | label | qos_id | vdsm_name | dns_resolver_configuration_id | provider_physical_network_id

----+------+-------------+------+------+--------+---------+---------+-----+-----------------+-----+-

-----------+------------------------------+------------------------------+-------------------+------

-+--------+-----------+-------------------------------+------------------------------ (0 rows)

engine=#

One more question. Suppose the problem caused any stale vnic on any vm, previously atached on OVN, is there a way to see at db level? Currently I only have 4 VMs and "network interfaces" in web admin gui doesn't show any OVN, but I would like to crosscheck also at db level, because I think in previous config before doing damages I has some on OVN.

select iface.name, n.name from vm_interface as iface left join vnic_profiles as vnic on iface.vnic_profile_id = vnic.id left join network n on vnic.network_id = n.id where n.provider_network_external_id is not null A bit longer but should show you the ovn network name that might be attached to the VM. Hopefully this helps

Thanks, Gianluca

-- ALES MUSIL Associate Software Engineer - rhv network Red Hat EMEA <https://www.redhat.com/> amusil@redhat.com IM: amusil <https://red.ht/sig>

On Thu, Mar 21, 2019 at 3:07 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

...

...

Output is this

name | name ------+------ (0 rows)

So I should be in the safe side, I hope. Thanks again for insights! Gianluca

Any way to reset admin@internal password previously set up for OVN? When adding provider I get this during test: Failed to communicate with the external provider, see log for additional details. and in /var/log/ovirt-provider-ovn.log 2019-03-21 15:36:52,735 root Error during SSO authentication Cannot authenticate user 'admin@internal': Unable to log in. Verify your login information or contact the system administrator.. : access_denied Traceback (most recent call last): File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 134, in _handle_request method, path_parts, content File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 175, in handle_request return self.call_response_handler(handler, content, parameters) File "/usr/share/ovirt-provider-ovn/handlers/keystone.py", line 33, in call_response_handler return response_handler(content, parameters) File "/usr/share/ovirt-provider-ovn/handlers/keystone_responses.py", line 62, in post_tokens user_password=user_password) File "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 26, in create_token return auth.core.plugin.create_token(user_at_domain, user_password) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/plugin.py", line 48, in create_token timeout=self._timeout()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 75, in create_token username, password, engine_url, ca_file, timeout) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 91, in _get_sso_token timeout=timeout File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 55, in wrapper _check_for_error(response) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 181, in _check_for_error result['error'], details)) Unauthorized: Error during SSO authentication Cannot authenticate user 'admin@internal': Unable to log in. Verify your login information or contact the system administrator.. : access_denied from last setup log on Februray I see --== PRODUCT OPTIONS ==-- Configure ovirt-provider-ovn (Yes, No) [Yes]: . . . (Yes, No) [No]: Yes oVirt OVN provider user[admin@internal]: oVirt OVN provider password: --== STORAGE CONFIGURATION ==-- . . . From /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf I see # This file is automatically generated by engine-setup. Please do not edit manually [OVN REMOTE] ovn-remote=ssl:127.0.0.1:6641 [SSL] https-enabled=true ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass [OVIRT] ovirt-sso-client-secret=<my_secret_omitted> ovirt-host=https://ovmgr1.mydomain:443 ovirt-sso-client-id=ovirt-provider-ovn ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem [PROVIDER] provider-host=ovmgr1.mydomain [root@ovmgr1 ~]# ll /etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer -rw-r--r--. 1 root root 1953 Feb 6 15:19 /etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer [root@ovmgr1 ~]# ll /etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass -rw-------. 1 root root 1828 Feb 6 15:19 /etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass [root@ovmgr1 ~]# openssl x509 -in /etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer -text -noout Certificate: . . . Validity Not Before: Feb 5 14:19:25 2019 GMT Not After : Jan 11 14:19:25 2024 GMT . . . I'm trying to add with name "MYOVN" from web admin gui: should I use instead another name? Gianluca

On Fri, 22 Mar 2019 10:49:08 +0100 Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Thu, Mar 21, 2019 at 3:46 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

...

. . .

I'm trying to add with name "MYOVN" from web admin gui: should I use instead another name?

Gianluca

Tried also this as detailed by Dominik, renewing certificates:

https://www.mail-archive.com/users@ovirt.org/msg53697.html

Not understood what to do in step 2. Use the SSO_CLIENT_SECRET from the outfile produced by the previous command in /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf "Use" in which way???

use as <my_secret_omitted> in [OVIRT] ovirt-sso-client-secret=<my_secret_omitted>

I named with default "ovirt-provider-ovn" the OVN provider, after enabling debug in OVN I get thsi when I test the connection in web admin gui

2019-03-22 10:40:41,917 root From: ::ffff:10.4.192.43:44744 Request: POST /v2.0/tokens 2019-03-22 10:40:41,918 root Request body: {"auth": {"passwordCredentials": {"username": "admin@internal", "password": "<PASSWORD_HIDDEN>"}}} 2019-03-22 10:40:41,918 auth.plugins.ovirt.sso Connecting to oVirt engine's SSO module: https://ovmgr1.mydomain:443/ovirt-engine/sso/oauth/token 2019-03-22 10:40:41,918 auth.plugins.ovirt.sso Connecting to oVirt engine's SSO module: https://ovmgr1.mydomain:443/ovirt-engine/sso/oauth/token 2019-03-22 10:40:41,921 urllib3.connectionpool Starting new HTTPS connection (1): ovmgr1.mydomain 2019-03-22 10:40:46,961 urllib3.connectionpool https://ovmgr1.mydomain:443 "POST /ovirt-engine/sso/oauth/token HTTP/1.1" 400 148 2019-03-22 10:40:46,964 root From: ::ffff:10.4.192.43:44744 Request: POST /v2.0/tokens 2019-03-22 10:40:46,964 root Request body: {"auth": {"passwordCredentials": {"username": "admin@internal", "password": "<PASSWORD_HIDDEN>"}}} 2019-03-22 10:40:46,964 root Error during SSO authentication Cannot authenticate user 'admin@internal': Unable to log in. Verify your login information or contact the system administrator.. : access_denied Traceback (most recent call last): File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 134, in _handle_request method, path_parts, content File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 175, in handle_request return self.call_response_handler(handler, content, parameters) File "/usr/share/ovirt-provider-ovn/handlers/keystone.py", line 33, in call_response_handler return response_handler(content, parameters) File "/usr/share/ovirt-provider-ovn/handlers/keystone_responses.py", line 62, in post_tokens user_password=user_password) File "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 26, in create_token return auth.core.plugin.create_token(user_at_domain, user_password) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/plugin.py", line 48, in create_token timeout=self._timeout()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 75, in create_token username, password, engine_url, ca_file, timeout) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 91, in _get_sso_token timeout=timeout File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 55, in wrapper _check_for_error(response) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 181, in _check_for_error result['error'], details)) Unauthorized: Error during SSO authentication Cannot authenticate user 'admin@internal': Unable to log in. Verify your login information or contact the system administrator.. : access_denied

It seems I have not completely understood the link between SSO and admin@internal as a user for OVN authentication....

The ovirt-sso-client-id and ovirt-sso-client-secret is required, to allow the ovirt-provider-ovn to connect to Engine's SSO for checking to user visible username, e.g. admin@internal, and password. I guess you are already aware of the doc in https://github.com/oVirt/ovirt-provider-ovn/#section-ovirt ovirt-provider-ovn does not store neither the user, e.g. admin@internal password nor the session token, it is just forwarded to Engine's SSO to check for validity. If you are interested in the details, the session token is generated by _get_sso_token in https://github.com/oVirt/ovirt-provider-ovn/blob/master/provider/auth/plugin... and validated by another method in https://github.com/oVirt/ovirt-provider-ovn/blob/master/provider/auth/plugin... where the ovirt-sso-client-id and ovirt-sso-client-secret are used as client_id, client_secret. In your case _get_sso_token is already failing, which does not use the ovirt-sso-client-secret. To solve this praticular issue, the provider in oVirt web admin ui should use the usual oVirt password for admin@internal.

Gianluca

On Fri, 22 Mar 2019 14:37:49 +0100 Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Fri, Mar 22, 2019 at 12:20 PM Dominik Holler <dholler@redhat.com> wrote: [snip]

...

...

...

Tried also this as detailed by Dominik, renewing certificates: https://www.mail-archive.com/users@ovirt.org/msg53697.html

Not understood what to do in step 2. Use the SSO_CLIENT_SECRET from the outfile produced by the previous command in /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf "Use" in which way???

use as <my_secret_omitted> in

[OVIRT] ovirt-sso-client-secret=<my_secret_omitted>

OK. I misunderstood/misread behavior of the command /usr/share/ovirt-engine/bin/ovirt-register-sso-client-tool.sh

I thought it directly wrote /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf Instead it writes a temporary/transient file such as /var/tmp/ovirt-engine/99_sso_client_1553259220210.conf now I put the value of this generated file "SSO_CLIENT_SECRET=" inside my 10-setup-ovirt-provider-ovn.conf and follow the other steps and I'm able to successfully test the OVN connection

...

...

It seems I have not completely understood the link between SSO and admin@internal as a user for OVN authentication....

The ovirt-sso-client-id and ovirt-sso-client-secret is required, to allow the ovirt-provider-ovn to connect to Engine's SSO for checking to user visible username, e.g. admin@internal, and password.

I guess you are already aware of the doc in https://github.com/oVirt/ovirt-provider-ovn/#section-ovirt

ovirt-provider-ovn does not store neither the user, e.g. admin@internal password nor the session token, it is just forwarded to Engine's SSO to check for validity.

If you are interested in the details, the session token is generated by _get_sso_token in

https://github.com/oVirt/ovirt-provider-ovn/blob/master/provider/auth/plugin... and validated by another method in

https://github.com/oVirt/ovirt-provider-ovn/blob/master/provider/auth/plugin... where the ovirt-sso-client-id and ovirt-sso-client-secret are used as client_id, client_secret.

In your case _get_sso_token is already failing, which does not use the ovirt-sso-client-secret.

To solve this praticular issue, the provider in oVirt web admin ui should use the usual oVirt password for admin@internal.

Thanks for all the corollary information! Now I was also able to create two ovn networks (ovn172 and ovn192) with one subnet each and on engine I have:

[root@ovmgr1 log]# ovn-nbctl show switch fc2fc4e8-ff71-4ec3-ba03-536a870cd483 (ovirt-ovn192-1e252228-ade7-47c8-acda-5209be358fcf) port 84c78095-744c-4415-805f-5f739af3d4d3 addresses: ["00:1a:4a:17:01:53 dynamic"] switch 9e77163a-c4e4-4abf-a554-0388e6b5e4ce (ovirt-ovn172-4ac7ba24-aad5-432d-b1d2-672eaeea7d63) port 899809f2-3ee8-4121-9fff-5e55bb0d5d8b addresses: ["00:1a:4a:17:01:54 dynamic"] [root@ovmgr1 log]#

I'm now able to create/attach an ovn based nic to a VM, but if I try to power on this VM I get an error in web admin gui

Failed to run VM p2vorasvi11

and in engine.log:

2019-03-22 14:30:34,498+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-5) [] EVENT_ID: VM_DOWN_ERROR(119), VM p2vorasvi11 is down with error. Exit message: Hook Error: ('',).

If I put a standard nic the VM is able to start without problem. Full log in engine.log during startup here: https://drive.google.com/file/d/1vDY64QaRkb8LWHJ9gD16bsJlyovISNwr/view?usp=s...

Can you please share the relevant part of vdsm.log, too?

BTW: one strange thing I see in dumped xml in engine.log during startup is:

<graphics type="spice" port="-1" autoport="yes" passwd="*****" passwdValidTo="1970-01-01T00:00:01" tlsPort="-1">

Gianluca

On Fri, 22 Mar 2019 16:14:59 +0100 Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Fri, Mar 22, 2019 at 3:21 PM Dominik Holler <dholler@redhat.com> wrote:

...

...

I'm now able to create/attach an ovn based nic to a VM, but if I try to power on this VM I get an error in web admin gui

Failed to run VM p2vorasvi11

and in engine.log:

2019-03-22 14:30:34,498+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-5) [] EVENT_ID: VM_DOWN_ERROR(119), VM p2vorasvi11 is down with error. Exit message: Hook Error: ('',).

If I put a standard nic the VM is able to start without problem. Full log in engine.log during startup here:

https://drive.google.com/file/d/1vDY64QaRkb8LWHJ9gD16bsJlyovISNwr/view?usp=s...

...

Can you please share the relevant part of vdsm.log, too?

It seems this below regarding sudo and password required for it:

2019-03-22 14:30:30,247+0100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.05 seconds (__init__:312) 2019-03-22 14:30:33,667+0100 INFO (vm/e54f8a2d) [root] /usr/libexec/vdsm/hooks/before_device_create/20_ovirt_provider_ovn_vhostuser_hook: rc=1 err=Traceback (most recent call last): File "/usr/libexec/vdsm/hooks/before_device_create/20_ovirt_provider_ovn_vhostuser_hook", line 134, in <module> main() File "/usr/libexec/vdsm/hooks/before_device_create/20_ovirt_provider_ovn_vhostuser_hook", line 120, in main if not is_netdev_datapath(): File "/usr/libexec/vdsm/hooks/before_device_create/20_ovirt_provider_ovn_vhostuser_hook", line 110, in is_netdev_datapath data, headings = list_ovs_table('bridge') File "/usr/libexec/vdsm/hooks/before_device_create/20_ovirt_provider_ovn_vhostuser_hook", line 42, in list_ovs_table exec_cmd('ovs-vsctl', '--format=json', 'list', table)[0] File "/usr/libexec/vdsm/hooks/before_device_create/20_ovirt_provider_ovn_vhostuser_hook", line 36, in exec_cmd (args, err)) RuntimeError: Failed to execute ('ovs-vsctl', '--format=json', 'list', 'bridge'), due to: ['sudo: a password is required'] (hooks:114) 2019-03-22 14:30:33,733+0100 INFO (vm/e54f8a2d) [root] /usr/libexec/vdsm/hooks/before_device_create/50_macspoof: rc=0 err= (hooks:114) 2019-03-22 14:30:33,802+0100 INFO (vm/e54f8a2d) [root] /usr/libexec/vdsm/hooks/before_device_create/50_vmfex: rc=0 err= (hooks:114) 2019-03-22 14:30:33,803+0100 ERROR (vm/e54f8a2d) [virt.vm] (vmId='e54f8a2d-432f-41f6-95b2-7bca3e5ebb4b') The vm start process failed (vm:937) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 866, in _startUnderlyingVm self._run() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2842, in _run self._buildDomainXML(), File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2265, in _buildDomainXML dom, self.id, self._custom) File "/usr/lib/python2.7/site-packages/vdsm/virt/domxml_preprocess.py", line 243, in replace_device_xml_with_hooks_xml dev_custom) File "/usr/lib/python2.7/site-packages/vdsm/common/hooks.py", line 138, in before_device_create params=customProperties) File "/usr/lib/python2.7/site-packages/vdsm/common/hooks.py", line 124, in _runHooksDir raise exception.HookError(err) HookError: Hook Error: ('',)

Thanks for raising this. I created https://bugzilla.redhat.com/1691933 to track this. Do you uninstalled vdsm-hook-openstacknet?

On Sat, 23 Mar 2019 15:19:06 +0100 Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Fri, Mar 22, 2019 at 10:19 PM Dominik Holler <dholler@redhat.com> wrote:

...

...

in _runHooksDir raise exception.HookError(err) HookError: Hook Error: ('',)

Thanks for raising this. I created https://bugzilla.redhat.com/1691933 to track this.

Do you uninstalled vdsm-hook-openstacknet?

No. It seems to me this package had never got installed but in 4.2.x OVN external network provider worked. The environment was created at beginning 2017 with 4.0.6 and then gradually updated, now at 4.3.2. OVN originally installed when in 4.1.0 with the manual way before official inclusion in engine-setup

[root@ov300 ~]# rpm -q vdsm-hook-openstacknet package vdsm-hook-openstacknet is not installed [root@ov300 ~]#

[root@ov300 ~]# ll -rt /var/log/yum.log* -rw-------. 1 root root 63893 Sep 29 2017 /var/log/yum.log-20180101 -rw-------. 1 root root 13840 Feb 9 2018 /var/log/yum.log-20180326 -rw-------. 1 root root 43106 Nov 22 11:47 /var/log/yum.log-20190101 -rw-------. 1 root root 38473 Mar 5 13:46 /var/log/yum.log-20190306 -rw-------. 1 root root 5018 Mar 22 14:11 /var/log/yum.log [root@ov300 ~]#

[root@ov300 ~]# grep vdsm-hook-openstacknet /var/log/yum.log* [root@ov300 ~]#

And the same for the other two hosts I can confirm that if I install that package (no vdsm restart):

Installing: vdsm-hook-openstacknet noarch 4.30.11-1.el7 ovirt-4.3 14 k

The VM with OVN network card on ovn192 is able to boot now and I have the vnet1 interface on ov300

[root@ov300 ~]# ovs-vsctl show f1a41e9c-16fb-4aa2-a386-2f366ade4d3c Bridge br-int fail_mode: secure Port br-int Interface br-int type: internal Port "ovn-b8872a-0" Interface "ovn-b8872a-0" type: geneve options: {csum="true", key=flow, remote_ip="10.4.192.34"} Port "ovn-1dce5b-0" Interface "ovn-1dce5b-0" type: geneve options: {csum="true", key=flow, remote_ip="10.4.192.32"} Port "vnet1" Interface "vnet1" ovs_version: "2.10.1" [root@ov300 ~]#

[root@ovmgr1 ~]# ovn-sbctl show Chassis "ddecf0da-4708-4f93-958b-6af365a5eeca" hostname: "ov300.datacenter.polimi.it" Encap geneve ip: "10.4.192.33" options: {csum="true"} Port_Binding "84c78095-744c-4415-805f-5f739af3d4d3" Chassis "1dce5b7c-a9fc-4ddb-99b4-e2c9e0fa54c5" hostname: "ov200.datacenter.polimi.it" Encap geneve ip: "10.4.192.32" options: {csum="true"} Chassis "b8872ab5-4606-4a79-b77d-9d956a18d349" hostname: "ov301.datacenter.polimi.it" Encap geneve ip: "10.4.192.34" options: {csum="true"} [root@ovmgr1 ~]#

And on engine: [root@ovmgr1 ~]# ovn-nbctl show switch fc2fc4e8-ff71-4ec3-ba03-536a870cd483 (ovirt-ovn192-1e252228-ade7-47c8-acda-5209be358fcf) port 84c78095-744c-4415-805f-5f739af3d4d3 addresses: ["00:1a:4a:17:01:53 dynamic"] switch 9e77163a-c4e4-4abf-a554-0388e6b5e4ce (ovirt-ovn172-4ac7ba24-aad5-432d-b1d2-672eaeea7d63) [root@ovmgr1 ~]#

So at the end it could be a missing dependency during install of new packages?

Not by intention. If vdsm-hook-openstacknet is installed, a file in /etc/sudoers.d/ is created, which allows vdsm to call ovs-vsctl without restricted parameters. /etc/sudoers.d/50_vdsm_hook_ovirt_provider_ovn_hook of ovirt-provider-ovn-driver should allow vdsm to call ovs-vsctl with all required parameters, but it does not. This is why I created bug 1691933. In the newer installations I checked vdsm-hook-openstacknet was installed and hides the bug. Maybe there are upgrade paths, which results in scenarios, where vdsm-hook-openstacknet is not installed, which should be fine, but shows the bug.

I have to dig a bit more, because from first tests if I start another VM on the same ovn192 network also on the same host they are not able to communicate Possibly an iptables misconfiguration on host?

Just to understand the error, would you please check if /var/log/openvswitch/ovn-controller.log or any other logfile in the same directory contains any hints? Would communication using a new created ovn network without port security enabled work? If there are not further hints, I suggest to re-configure the ovirt-provider-ovn-driver on the host via vdsm-tool ovn-config OVN_Central_IP Tunneling_IP_or_Network_Name (please find more details on https://ovirt.org/documentation/admin-guide/chap-External_Providers.html#con... ) and check if this fixed the issue.

I have vnet1 and vnet2 on host now

[root@ov300 ~]# ovs-vsctl show f1a41e9c-16fb-4aa2-a386-2f366ade4d3c Bridge br-int fail_mode: secure Port br-int Interface br-int type: internal Port "vnet2" Interface "vnet2" Port "ovn-b8872a-0" Interface "ovn-b8872a-0" type: geneve options: {csum="true", key=flow, remote_ip="10.4.192.34"} Port "ovn-1dce5b-0" Interface "ovn-1dce5b-0" type: geneve options: {csum="true", key=flow, remote_ip="10.4.192.32"} Port "vnet1" Interface "vnet1" ovs_version: "2.10.1" [root@ov300 ~]#

Thanks for the moment Gianluca

On Thu, Apr 4, 2019 at 11:27 AM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Sat, Mar 23, 2019 at 7:44 PM Dominik Holler <dholler@redhat.com> wrote:

Sorry for late reply Dominik.... busy on other (interesting at least ;-) things

...

...

I have to dig a bit more, because from first tests if I start another VM on the same ovn192 network also on the same host they are not able to communicate Possibly an iptables misconfiguration on host?

Just to understand the error, would you please check if /var/log/openvswitch/ovn-controller.log or any other logfile in the same directory contains any hints?

It seems not

...

Would communication using a new created ovn network without port security enabled work?

I confirm that if I create a new ovn with security port "Disabled" the VMs can communicate both when running on the same host and on hosts even in different datacenters ;-) I unplug vnic / change ovn network of vms to match the new one / plug vnics again and they communicate. I unplug vnic / change ovn network of vms to the old one with port securty enabled / plug vnics again and they don't communicate.

Questions: - what is the role of the "Network port security" option for an OVN network?

It means that newly created ports under that network will inherit the port security value from the network - e.g. if the network's port security attribute is active, so will the newly created port's port security. Port security on a port means 2 things: #1 - security group rules *will* apply to the VM having that port attached #2 - only the specified mac address will be allowed to send/receive through that port. MAC spoofing protection is applied.

- what is the meaning of "Undefined" option for it other than "Enabled" and "Disabled"?

It means that the network will inherit the value from the provider's configuration - you can check what it translates to in /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf

- it seems I cannot edit the value for "Network port security" option of an existing OVN network, is it correct?

You cannot do it *through the UI*. You can use ansible / REST api to update the network - or ports - port_security_enabled value. I am working on creating a couple of playbooks for this; hopefully I can provide those early next week. It would be helpful to agilize this process.

Thanks again, Gianluca

There is a notion of 'default' group, that ensures connectivity to all VMs whose ports belong to that group - and all ports with active port security, by default do. I'm not sure how you reached that situation, but let's first make sure of a couple of things; please provider the output of: - ovn-nbctl list logical_switch_port # this will feature info of the port security value, and of which groups the port belongs to - the latter in the 'external_ids' column. - ovn-nbctl list port_group # this is where the security groups are stored; it has associations to the ACLs belonging to the group, and of the ports that are using it - ovn-nbctl list address_set # this is where the IPs per group are stored. security groups are an L3 concept. A pastebin with the aforementioned info is welcome.

On Thu, Apr 4, 2019 at 2:04 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Thu, Apr 4, 2019 at 12:07 PM Miguel Duarte de Mora Barroso <mdbarroso@redhat.com> wrote:

...

...

Questions: - what is the role of the "Network port security" option for an OVN network?

It means that newly created ports under that network will inherit the port security value from the network - e.g. if the network's port security attribute is active, so will the newly created port's port security.

Port security on a port means 2 things: #1 - security group rules *will* apply to the VM having that port attached #2 - only the specified mac address will be allowed to send/receive through that port. MAC spoofing protection is applied.

...

- what is the meaning of "Undefined" option for it other than "Enabled" and "Disabled"?

It means that the network will inherit the value from the provider's configuration - you can check what it translates to in /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf

Thanks for clarifications. Digging around RHV 4.2 vs 4.3beta docs I see now that "Network Port Security" will be also one of the new features for it In 4.3 beta the third option is explictly defined as "Inherited" (reflecting your explanation) and not "Undefined" as in current oVirt 4.3.2)

...

...

- it seems I cannot edit the value for "Network port security" option of an existing OVN network, is it correct?

You cannot do it *through the UI*. You can use ansible / REST api to update the network - or ports - port_security_enabled value.

...

I am working on creating a couple of playbooks for this; hopefully I can provide those early next week. It would be helpful to agilize this process.

Indeed. Because in Openstack web mgmt interface all the settings related to security groups are simplified and intuitive, but here we have not... Also because it seems from rhv 4.3beta manual that creation of security groups themselves will not be possible through web gui...

...

There is a notion of 'default' group, that ensures connectivity to all VMs whose ports belong to that group - and all ports with active port security, by default do.

I'm not sure how you reached that situation, but let's first make sure of a couple of things; please provider the output of: - ovn-nbctl list logical_switch_port # this will feature info of the port security value, and of which groups the port belongs to - the latter in the 'external_ids' column. - ovn-nbctl list port_group # this is where the security groups are stored; it has associations to the ACLs belonging to the group, and of the ports that are using it - ovn-nbctl list address_set # this is where the IPs per group are stored. security groups are an L3 concept.

A pastebin with the aforementioned info is welcome.

See here: https://drive.google.com/file/d/1hgXMGttMgb0oaDEy5k6aWFdb01dYsjwq/view?usp=s...

From the data you supply, everything looks as is should: both the ports are members of the default port group, and both their IPs are featured in the ip4 address set. Mind sharing the created ACLs ? (which I'm quite positive will be the default ones, but I just have to be sure). Can be done via "ovn-nbctl list acl" . With that I can check the ACLs assigned to the default group, and assure they are correct.

Gianluca

Hi Gianluca, First of, sorry for the late reply, been very busy this past week. Regarding the lack of security group support on oVirt, I agree it's unfortunate. Please take a look at this repo [0]; you'll find playbooks to update the port's / networks port security, security groups, and a couple of examples on how to create new security groups and rules via ansible. You can follow the README, it features all the information you need to install the requirements, and use the playbooks. Comments are welcome. You can find answers to your questions inline. [0] - https://github.com/maiqueb/ovirt-security-groups-demo/ On Fri, Apr 5, 2019 at 10:25 AM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:

On Fri, Apr 5, 2019 at 9:56 AM Miguel Duarte de Mora Barroso <mdbarroso@redhat.com> wrote:

...

Mind sharing the created ACLs ? (which I'm quite positive will be the default ones, but I just have to be sure). Can be done via "ovn-nbctl list acl" . With that I can check the ACLs assigned to the default group, and assure they are correct.

The question is: previous networks (in the sense of already existing before the port security feature had been introduced in 4.3) seems inherited the "Enabled" option and this prevents communication between VMs on the same OVN network. Is this expected?

Previous networks are unchanged; nothing updates any of those during the upgrade. Now, newly created ports on existing networks *will* inherit the value from the configuration - since the network itself doesn't have the port security attribute set. Can you share what's the current port-security-enabled value on your configuration ? (/etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf)

Otherwise other people in 4.2 using OVN will have the same problem migrating to 4.3 If I create now n 4.3.2 a new OVN based network, if I select "Create an external provider", I get as default "ovirt-provider-ovn" as External Provider and "Enabled" as Network Port Security. Is this expected?

Yes.

Is it expected that a new OVN network with default values (Enabled port security) is made so that by default 2 VMs don't communicate if I don't set a special security group rule (that in tis moment requires REST api)?

No, the exact purpose of the default group is for the VMs to communicate out of the box. The ACLs you provide match all the ACLs present on the port groups you've previously shared, and ; from my perspective, your VMs should be able to communicate. Could you share the output of 'ovs-ofctl dump-flows br-int' on the ovirt node where your VMs are located ? That could indicate why the packets are being dropped. Please provide that in a pastebin (this email is already hard to follow). A further question: your cluster switch type is ovs, right? This would only matter if your VMs run in different nodes, but hey, best to get that sorted out early. Lastly, are your VMs able to receive an IP address via dhcp ?

As far as ACLs currently in place are concerned, here they are for my current environment.

[root@ovmgr1 ~]# ovn-nbctl list acl _uuid : 239f0fa4-a66e-4cce-8df2-05630f11e052 action : drop direction : to-lport external_ids : {description="drop all ingress ip traffic", ovirt_port_group_id="79d3d3a0-7a57-4903-8646-f678ea53aeca"} log : false match : "outport == @DropAll && ip" meter : [] name : "" priority : 1000 severity : alert

_uuid : 141aa336-0549-47d0-b09f-c2cb0dd78dd2 action : allow-related direction : from-lport external_ids : {description="automatically added allow all egress ip traffic", ovirt_ethertype="IPv4", ovirt_port_group_id="1fd8cacf-35cf-4aa3-b245-fec9c2e6e616"} log : false match : "inport == @Default && ip4" meter : [] name : "" priority : 1001 severity : alert

_uuid : ac7d5a16-a596-43dc-88ec-e9d47512e7ce action : drop direction : from-lport external_ids : {description="drop all egress ip traffic", ovirt_port_group_id="79d3d3a0-7a57-4903-8646-f678ea53aeca"} log : false match : "inport == @DropAll && ip" meter : [] name : "" priority : 1000 severity : alert

_uuid : ef7f32f2-8b78-433f-a831-0e801c9d8b3e action : allow-related direction : to-lport external_ids : {ovirt_ethertype="IPv4", ovirt_port_group_id="1fd8cacf-35cf-4aa3-b245-fec9c2e6e616", ovirt_remote_group_id="1fd8cacf-35cf-4aa3-b245-fec9c2e6e616"} log : false match : "outport == @Default && ip4 && ip4.src == $pg_ip4_Default" meter : [] name : "" priority : 1001 severity : alert

_uuid : 70c7114b-1be6-49c1-9bbd-966c52751e79 action : allow-related direction : from-lport external_ids : {description="automatically added allow all egress ip traffic", ovirt_ethertype="IPv6", ovirt_port_group_id="1fd8cacf-35cf-4aa3-b245-fec9c2e6e616"} log : false match : "inport == @Default && ip6" meter : [] name : "" priority : 1001 severity : alert

_uuid : 264111cf-4f66-4b4c-b3c9-693bbca53a70 action : allow-related direction : to-lport external_ids : {ovirt_ethertype="IPv6", ovirt_port_group_id="1fd8cacf-35cf-4aa3-b245-fec9c2e6e616", ovirt_remote_group_id="1fd8cacf-35cf-4aa3-b245-fec9c2e6e616"} log : false match : "outport == @Default && ip6 && ip6.src == $pg_ip6_Default" meter : [] name : "" priority : 1001 severity : alert [root@ovmgr1 ~]#

Gianluca