Hello, I installed some months ago a test setup in 4.1.0 with ovn. Now after updating engine and host to 4.1.1 it seems the services are up but it doesn't work. If I run a VM with a network device in OVN external provider, it cant' boot and I get this in engine.log: 2017-04-20 15:17:42,285+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-11) [e55e0971-f1e5-4fda-8666-3ce23797027f] EVENT_ID: USER_FAILED_RUN_VM(54), Correlation ID: e55e0971-f1e5-4fda-8666-3ce23797027f, Job ID: 067f6e70-9e70-48bc-be44-d5bd1d9485fd, Call Stack: null, Custom Event ID: -1, Message: Failed to run VM c6 (User: admin@internal-authz). 2017-04-20 15:17:42,317+02 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-11) [e55e0971-f1e5-4fda-8666-3ce23797027f] Lock freed to object 'EngineLock:{exclusiveLocks='[50194eea-f96d-4ebb-bf64-55cef13f4309=<VM, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}' 2017-04-20 15:17:42,317+02 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-11) [e55e0971-f1e5-4fda-8666-3ce23797027f] Command 'org.ovirt.engine.core.bll.RunVmCommand' failed: EngineException: (Failed with error PROVIDER_FAILURE and code 5050) Firewall is disabled/stopped at host and engine side (where I installed the central server too) and should not be the problem.... On engine server I get this into /var/log/ovirt-provider-ovn.log 2017-04-20 16:36:25,355 Request: GET : /v2.0/ports 2017-04-20 16:36:25,355 Connecting to remote ovn database: tcp: 127.0.0.1:6641 2017-04-20 16:36:28,422 Failed to connect! 2017-04-20 16:36:28,422 Failed to connect! Traceback (most recent call last): File "/usr/share/ovirt-provider-ovn/neutron.py", line 76, in _handle_request content) File "/usr/share/ovirt-provider-ovn/neutron.py", line 132, in handle_request with OvnNbDb(self.remote) as nb_db: File "/usr/share/ovirt-provider-ovn/ovndb/ndb_api.py", line 56, in __init__ self.connect(tables, remote, self.OVN_NB_OVSSCHEMA_FILE) File "/usr/share/ovirt-provider-ovn/ovndb/ovsdb_api.py", line 110, in connect OvsDb._connect(self._ovsdb_connection) File "/usr/share/ovirt-provider-ovn/ovndb/ovsdb_api.py", line 47, in block raise OvsDBConnectionFailed('Failed to connect!') OvsDBConnectionFailed: Failed to connect! Initial working versions on engine, where I configured the central server: Feb 14 17:55:57 Installed: openvswitch-2.6.90-1.el7.centos.x86_64 Feb 14 17:55:57 Installed: openvswitch-ovn-common-2.6.90-1.el7.centos.x86_64 Feb 14 17:55:58 Installed: openvswitch-ovn-central-2.6.90-1.el7.centos.x86_64 Feb 14 17:55:59 Installed: python-openvswitch-2.6.90-1.el7.centos.noarch Feb 14 17:56:52 Installed: ovirt-provider-ovn-1.0-1.20161219125609.git.el7.centos.noarch Today as part of the update I got: Apr 20 11:30:06 Updated: openvswitch-2.7.0-1.el7.centos.x86_64 Apr 20 11:30:06 Updated: openvswitch-ovn-common-2.7.0-1.el7.centos.x86_64 Apr 20 11:30:07 Updated: openvswitch-ovn-central-2.7.0-1.el7.centos.x86_64 Apr 20 11:30:24 Installed: python-openvswitch-2.7.0-1.el7.centos.noarch Apr 20 11:31:00 Updated: ovirt-provider-ovn-1.0-6.el7.centos.noarch At the page https://www.ovirt.org/blog/2016/11/ovirt-provider-ovn/ I see this note about ports: " Since OVS 2.7, OVN central must be configured to listen to requests on appropriate ports: ovn-sbctl set-connection ptcp:6642 ovn-nbctl set-connection ptcp:6641 " and in my case I indeed passed from 2.6.90 to 2.7.0... Do I need to run these two commands? Or any other configuration settings? Thanks in advance, Gianluca
On Thu, Apr 20, 2017 at 6:54 PM, Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
Hello, I installed some months ago a test setup in 4.1.0 with ovn. Now after updating engine and host to 4.1.1 it seems the services are up but it doesn't work. If I run a VM with a network device in OVN external provider, it cant' boot and I get this in engine.log:
[snip]
At the page https://www.ovirt.org/blog/2016/11/ovirt-provider-ovn/
I see this note about ports:
" Since OVS 2.7, OVN central must be configured to listen to requests on appropriate ports:
ovn-sbctl set-connection ptcp:6642 ovn-nbctl set-connection ptcp:6641 "
and in my case I indeed passed from 2.6.90 to 2.7.0...
Do I need to run these two commands? Or any other configuration settings?
Thanks in advance, Gianluca
I confirm that after running these two commands all work ok again and I'm able to start a VM with a vnic provided by the OVN provider [root@ractorshe ~]# ovn-sbctl set-connection ptcp:6642 [root@ractorshe ~]# ovn-nbctl set-connection ptcp:6641 [root@ractorshe ~]#
Hello Gianluca, Can you please check the ovn north db log. This is placed in /var/log/openvswitch/ovsdb-server-nb.log Please check if the logs has any new entries when you try to connect and when you issue the 'ovn-nbctl set-connection ptcp:6641' command. If the connection attempt is getting through, pvs db should print an error to the log. Please also try restarting the ovn-northd service. Do the ovn-controllers connect to the south-db? You can verify this by looking at /var/log/openvswitch/ovn-controller.log on the ovn-controller host (please look for entries saying "... <north ip>:6642 connected") Marcin On Fri, Apr 21, 2017 at 1:09 PM, Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Thu, Apr 20, 2017 at 6:54 PM, Gianluca Cecchi < gianluca.cecchi@gmail.com> wrote:
Hello, I installed some months ago a test setup in 4.1.0 with ovn. Now after updating engine and host to 4.1.1 it seems the services are up but it doesn't work. If I run a VM with a network device in OVN external provider, it cant' boot and I get this in engine.log:
[snip]
At the page https://www.ovirt.org/blog/2016/11/ovirt-provider-ovn/
I see this note about ports:
" Since OVS 2.7, OVN central must be configured to listen to requests on appropriate ports:
ovn-sbctl set-connection ptcp:6642 ovn-nbctl set-connection ptcp:6641 "
and in my case I indeed passed from 2.6.90 to 2.7.0...
Do I need to run these two commands? Or any other configuration settings?
Thanks in advance, Gianluca
I confirm that after running these two commands all work ok again and I'm able to start a VM with a vnic provided by the OVN provider
[root@ractorshe ~]# ovn-sbctl set-connection ptcp:6642 [root@ractorshe ~]# ovn-nbctl set-connection ptcp:6641 [root@ractorshe ~]#
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- MARCIN mIRECKI Red Hat <https://www.redhat.com> <https://red.ht/sig>
On Sun, Apr 23, 2017 at 11:32 PM, Marcin Mirecki <mmirecki@redhat.com> wrote:
Hello Gianluca,
Can you please check the ovn north db log. This is placed in /var/log/openvswitch/ovsdb-server-nb.log Please check if the logs has any new entries when you try to connect and when you issue the 'ovn-nbctl set-connection ptcp:6641' command. If the connection attempt is getting through, pvs db should print an error to the log.
Please also try restarting the ovn-northd service.
Do the ovn-controllers connect to the south-db? You can verify this by looking at /var/log/openvswitch/ovn-controller.log on the ovn-controller host (please look for entries saying "... <north ip>:6642 connected")
Marcin
The ovirt nb log contains: 2017-04-24T07:46:51.541Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server-nb.log 2017-04-24T07:46:51.550Z|00002|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.7.0 2017-04-24T07:47:01.560Z|00003|memory|INFO|2268 kB peak resident set size after 10.0 seconds 2017-04-24T07:47:01.560Z|00004|memory|INFO|cells:100 json-caches:1 monitors:1 ses In my ovn-controller.log of my 3 hosts I have this, when I run the 2 commands below on the provider host ovn-sbctl set-connection ptcp:6642 ovn-nbctl set-connection ptcp:6641 2017-04-24T07:56:23.178Z|00247|reconnect|INFO|tcp:10.4.192.43:6642: connecting... 2017-04-24T07:56:23.178Z|00248|reconnect|INFO|tcp:10.4.192.43:6642: connection attempt failed (Connection refused) 2017-04-24T07:56:23.178Z|00249|reconnect|INFO|tcp:10.4.192.43:6642: waiting 8 seconds before reconnect 2017-04-24T07:56:31.187Z|00250|reconnect|INFO|tcp:10.4.192.43:6642: connecting... 2017-04-24T07:56:31.188Z|00251|reconnect|INFO|tcp:10.4.192.43:6642: connected 2017-04-24T07:56:31.193Z|00252|ofctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch 2017-04-24T07:56:31.193Z|00253|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2017-04-24T07:56:31.201Z|00254|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected 2017-04-24T07:56:31.201Z|00255|pinctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch 2017-04-24T07:56:31.201Z|00256|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2017-04-24T07:56:31.201Z|00257|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected 2017-04-24T07:56:31.202Z|00258|binding|INFO|Releasing lport 0a2a47bc-ea0d-4f1d-8f49-ec903e519983 from this chassis. On my provider I see then the 3 oVirt hosts connected: [root@ovmgr1 openvswitch]# netstat -tpn|grep 66 tcp 0 0 10.4.192.43:6642 10.4.168.76:38882 ESTABLISHED 6699/ovsdb-server tcp 0 0 10.4.192.43:6642 10.4.168.75:45486 ESTABLISHED 6699/ovsdb-server tcp 0 0 127.0.0.1:5432 127.0.0.1:37074 ESTABLISHED 16696/postgres: eng tcp 0 0 10.4.192.43:6642 10.4.168.74:59176 ESTABLISHED 6699/ovsdb-server [root@ovmgr1 openvswitch]# But it seems that the "set" command above is not persistent across reboot of the provider host.... that in my case is the oVirt engine server
Looks like the south db works properly. The north db uses the same mechanism, just a different schema and port. Looking at the netstat output it looks like ovn north db is not even listening, or is there anything for 6641? On Mon, Apr 24, 2017 at 10:37 AM, Gianluca Cecchi <gianluca.cecchi@gmail.com
wrote:
On Sun, Apr 23, 2017 at 11:32 PM, Marcin Mirecki <mmirecki@redhat.com> wrote:
Hello Gianluca,
Can you please check the ovn north db log. This is placed in /var/log/openvswitch/ovsdb-server-nb.log Please check if the logs has any new entries when you try to connect and when you issue the 'ovn-nbctl set-connection ptcp:6641' command. If the connection attempt is getting through, pvs db should print an error to the log.
Please also try restarting the ovn-northd service.
Do the ovn-controllers connect to the south-db? You can verify this by looking at /var/log/openvswitch/ovn-controller.log on the ovn-controller host (please look for entries saying "... <north ip>:6642 connected")
Marcin
The ovirt nb log contains: 2017-04-24T07:46:51.541Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server-nb.log 2017-04-24T07:46:51.550Z|00002|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.7.0 2017-04-24T07:47:01.560Z|00003|memory|INFO|2268 kB peak resident set size after 10.0 seconds 2017-04-24T07:47:01.560Z|00004|memory|INFO|cells:100 json-caches:1 monitors:1 ses
In my ovn-controller.log of my 3 hosts I have this, when I run the 2 commands below on the provider host
ovn-sbctl set-connection ptcp:6642 ovn-nbctl set-connection ptcp:6641
2017-04-24T07:56:23.178Z|00247|reconnect|INFO|tcp:10.4.192.43:6642: connecting... 2017-04-24T07:56:23.178Z|00248|reconnect|INFO|tcp:10.4.192.43:6642: connection attempt failed (Connection refused) 2017-04-24T07:56:23.178Z|00249|reconnect|INFO|tcp:10.4.192.43:6642: waiting 8 seconds before reconnect 2017-04-24T07:56:31.187Z|00250|reconnect|INFO|tcp:10.4.192.43:6642: connecting... 2017-04-24T07:56:31.188Z|00251|reconnect|INFO|tcp:10.4.192.43:6642: connected 2017-04-24T07:56:31.193Z|00252|ofctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch 2017-04-24T07:56:31.193Z|00253|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2017-04-24T07:56:31.201Z|00254|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected 2017-04-24T07:56:31.201Z|00255|pinctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting to switch 2017-04-24T07:56:31.201Z|00256|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting... 2017-04-24T07:56:31.201Z|00257|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected 2017-04-24T07:56:31.202Z|00258|binding|INFO|Releasing lport 0a2a47bc-ea0d-4f1d-8f49-ec903e519983 from this chassis.
On my provider I see then the 3 oVirt hosts connected: [root@ovmgr1 openvswitch]# netstat -tpn|grep 66 tcp 0 0 10.4.192.43:6642 10.4.168.76:38882 ESTABLISHED 6699/ovsdb-server tcp 0 0 10.4.192.43:6642 10.4.168.75:45486 ESTABLISHED 6699/ovsdb-server tcp 0 0 127.0.0.1:5432 127.0.0.1:37074 ESTABLISHED 16696/postgres: eng tcp 0 0 10.4.192.43:6642 10.4.168.74:59176 ESTABLISHED 6699/ovsdb-server [root@ovmgr1 openvswitch]#
But it seems that the "set" command above is not persistent across reboot of the provider host.... that in my case is the oVirt engine server
-- MARCIN mIRECKI Red Hat <https://www.redhat.com> <https://red.ht/sig>
On Mon, Apr 24, 2017 at 10:03 PM, Marcin Mirecki <mmirecki@redhat.com> wrote:
Looks like the south db works properly. The north db uses the same mechanism, just a different schema and port.
Looking at the netstat output it looks like ovn north db is not even listening, or is there anything for 6641?
Actually yes... it seems that the switch "-t" with the "-p" doesn't catch the 6641 and 6642 "LISTEN" lines, while if I use "-a" instead of "-t" I get them too... with "-a" root@ovmgr1 ~]# netstat -apn | grep 664 tcp 0 0 0.0.0.0:6641 0.0.0.0:* LISTEN 6691/ovsdb-server tcp 0 0 0.0.0.0:6642 0.0.0.0:* LISTEN 6699/ovsdb-server tcp 0 0 10.4.192.43:6642 10.4.168.76:38882 ESTABLISHED 6699/ovsdb-server tcp 0 0 10.4.192.43:6642 10.4.168.75:45486 ESTABLISHED 6699/ovsdb-server tcp 0 0 10.4.192.43:6642 10.4.168.74:59176 ESTABLISHED 6699/ovsdb-server unix 3 [ ] STREAM CONNECTED 14119 664/vmtoolsd with "-t" [root@ovmgr1 ~]# netstat -tpn | grep 664 tcp 0 0 10.4.192.43:6642 10.4.168.76:38882 ESTABLISHED 6699/ovsdb-server tcp 0 0 10.4.192.43:6642 10.4.168.75:45486 ESTABLISHED 6699/ovsdb-server tcp 0 0 10.4.192.43:6642 10.4.168.74:59176 ESTABLISHED 6699/ovsdb-server
It looks like ndb is listening (netstat shows this), but not receiving anything (log is empty). This looks like a connectivity issue. Can the provider host ping the ovn host? Is firewalld/iptables disabled on both hosts? Let's make sure that OVN is working fine. I attach a short python script that checks the connection from the localhost. Please check if it can connect to ovn from localhost (just execute this on the host with ovn: "python tcp_connection_test.py" ). It should connect and print out all the OVN networks. Does this work? On Mon, Apr 24, 2017 at 11:23 PM, Gianluca Cecchi <gianluca.cecchi@gmail.com
wrote:
On Mon, Apr 24, 2017 at 10:03 PM, Marcin Mirecki <mmirecki@redhat.com> wrote:
Looks like the south db works properly. The north db uses the same mechanism, just a different schema and port.
Looking at the netstat output it looks like ovn north db is not even listening, or is there anything for 6641?
Actually yes... it seems that the switch "-t" with the "-p" doesn't catch the 6641 and 6642 "LISTEN" lines, while if I use "-a" instead of "-t" I get them too...
with "-a" root@ovmgr1 ~]# netstat -apn | grep 664 tcp 0 0 0.0.0.0:6641 0.0.0.0:* LISTEN 6691/ovsdb-server tcp 0 0 0.0.0.0:6642 0.0.0.0:* LISTEN 6699/ovsdb-server tcp 0 0 10.4.192.43:6642 10.4.168.76:38882 ESTABLISHED 6699/ovsdb-server tcp 0 0 10.4.192.43:6642 10.4.168.75:45486 ESTABLISHED 6699/ovsdb-server tcp 0 0 10.4.192.43:6642 10.4.168.74:59176 ESTABLISHED 6699/ovsdb-server unix 3 [ ] STREAM CONNECTED 14119 664/vmtoolsd
with "-t" [root@ovmgr1 ~]# netstat -tpn | grep 664 tcp 0 0 10.4.192.43:6642 10.4.168.76:38882 ESTABLISHED 6699/ovsdb-server tcp 0 0 10.4.192.43:6642 10.4.168.75:45486 ESTABLISHED 6699/ovsdb-server tcp 0 0 10.4.192.43:6642 10.4.168.74:59176 ESTABLISHED 6699/ovsdb-server
-- MARCIN mIRECKI Red Hat <https://www.redhat.com> <https://red.ht/sig>
On Tue, Apr 25, 2017 at 9:34 AM, Marcin Mirecki <mmirecki@redhat.com> wrote:
It looks like ndb is listening (netstat shows this), but not receiving anything (log is empty). This looks like a connectivity issue. Can the provider host ping the ovn host?
Yes. The provider host is the oVirt self hosted engine VM
Is firewalld/iptables disabled on both hosts?
Firewall is disabled on all 3 oVirt hosts. While on engine VM side the firewalld service is active but with needed ports enabled. [g.cecchi@ovmgr1 ~]$ sudo firewall-cmd --get-default-zone public [g.cecchi@ovmgr1 ~]$ sudo firewall-cmd --get-active-zones public interfaces: ens192 [g.cecchi@ovmgr1 ~]$ sudo firewall-cmd --list-all public (active) target: default icmp-block-inversion: no interfaces: ens192 sources: services: dhcpv6-client ovirt-fence-kdump-listener ovirt-http ovirt-https ovirt-imageio-proxy ovirt-postgres ovirt-provider-ovn ovirt-vmconsole-proxy ovirt-websocket-proxy ssh ports: protocols: masquerade: no forward-ports: sourceports: icmp-blocks: rich rules: rule family="ipv4" port port="6641" protocol="tcp" accept rule family="ipv4" port port="6642" protocol="tcp" accept [g.cecchi@ovmgr1 ~]$
Let's make sure that OVN is working fine. I attach a short python script that checks the connection from the localhost. Please check if it can connect to ovn from localhost (just execute this on the host with ovn: "python tcp_connection_test.py" ). It should connect and print out all the OVN networks. Does this work? <https://www.redhat.com>
It seems so. If I run on my ovm provider host: [g.cecchi@ovmgr1 ~]$ python /tmp/tcp_connection_test.py connecting 1 connecting 2 CONNECTED! ROW: uuid:04501f6b-3977-4ba1-9ead-7096768d796d name:ovn172 ROW: uuid:6110649a-db2b-4de7-8fbc-601095cfe510 name:ovn192 [g.cecchi@ovmgr1 ~]$ Perhaps I was not clear at the beginning.... I try to reformulate. I noticed that ovn didn't work after upgrade from 4.1.0 to 4.1.1. Then in oVirt OVN documentation I found about the "set" commands to give in case of openvswitch 2.7 And in fact after giving the commands I was able to start VM with OVN vnics and they was able to communicate through openvswitch. The problem was the persistence of the set command. It seems that after restarting engine VM (that is my provider host), the setting is not maintained.... Gianluca
participants (2)
-
Gianluca Cecchi -
Marcin Mirecki