9:54 a.m.
Hello,
I installed some months ago a test setup in 4.1.0 with ovn.
Now after updating engine and host to 4.1.1 it seems the services are up
but it doesn't work.
If I run a VM with a network device in OVN external provider, it cant' boot
and I get this in engine.log:
2017-04-20 15:17:42,285+02 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-11) [e55e0971-f1e5-4fda-8666-3ce23797027f]
EVENT_ID: USER_FAILED_RUN_VM(54), Correlation ID:
e55e0971-f1e5-4fda-8666-3ce23797027f, Job ID:
067f6e70-9e70-48bc-be44-d5bd1d9485fd, Call Stack: null, Custom Event ID:
-1, Message: Failed to run VM c6 (User: admin@internal-authz).
2017-04-20 15:17:42,317+02 INFO [org.ovirt.engine.core.bll.RunVmCommand]
(org.ovirt.thread.pool-6-thread-11) [e55e0971-f1e5-4fda-8666-3ce23797027f]
Lock freed to object
'EngineLock:{exclusiveLocks='[50194eea-f96d-4ebb-bf64-55cef13f4309=<VM,
ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'
2017-04-20 15:17:42,317+02 ERROR [org.ovirt.engine.core.bll.RunVmCommand]
(org.ovirt.thread.pool-6-thread-11) [e55e0971-f1e5-4fda-8666-3ce23797027f]
Command 'org.ovirt.engine.core.bll.RunVmCommand' failed: EngineException:
(Failed with error PROVIDER_FAILURE and code 5050)
Firewall is disabled/stopped at host and engine side (where I installed the
central server too) and should not be the problem....
On engine server I get this into /var/log/ovirt-provider-ovn.log
2017-04-20 16:36:25,355 Request: GET : /v2.0/ports
2017-04-20 16:36:25,355 Connecting to remote ovn database: tcp:
127.0.0.1:6641
2017-04-20 16:36:28,422 Failed to connect!
2017-04-20 16:36:28,422 Failed to connect!
Traceback (most recent call last):
File "/usr/share/ovirt-provider-ovn/neutron.py", line 76, in
_handle_request
content)
File "/usr/share/ovirt-provider-ovn/neutron.py", line 132, in
handle_request
with OvnNbDb(self.remote) as nb_db:
File "/usr/share/ovirt-provider-ovn/ovndb/ndb_api.py", line 56, in
__init__
self.connect(tables, remote, self.OVN_NB_OVSSCHEMA_FILE)
File "/usr/share/ovirt-provider-ovn/ovndb/ovsdb_api.py", line 110, in
connect
OvsDb._connect(self._ovsdb_connection)
File "/usr/share/ovirt-provider-ovn/ovndb/ovsdb_api.py", line 47, in block
raise OvsDBConnectionFailed('Failed to connect!')
OvsDBConnectionFailed: Failed to connect!
Initial working versions on engine, where I configured the central server:
Feb 14 17:55:57 Installed: openvswitch-2.6.90-1.el7.centos.x86_64
Feb 14 17:55:57 Installed: openvswitch-ovn-common-2.6.90-1.el7.centos.x86_64
Feb 14 17:55:58 Installed:
openvswitch-ovn-central-2.6.90-1.el7.centos.x86_64
Feb 14 17:55:59 Installed: python-openvswitch-2.6.90-1.el7.centos.noarch
Feb 14 17:56:52 Installed:
ovirt-provider-ovn-1.0-1.20161219125609.git.el7.centos.noarch
Today as part of the update I got:
Apr 20 11:30:06 Updated: openvswitch-2.7.0-1.el7.centos.x86_64
Apr 20 11:30:06 Updated: openvswitch-ovn-common-2.7.0-1.el7.centos.x86_64
Apr 20 11:30:07 Updated: openvswitch-ovn-central-2.7.0-1.el7.centos.x86_64
Apr 20 11:30:24 Installed: python-openvswitch-2.7.0-1.el7.centos.noarch
Apr 20 11:31:00 Updated: ovirt-provider-ovn-1.0-6.el7.centos.noarch
At the page
https://www.ovirt.org/blog/2016/11/ovirt-provider-ovn/
I see this note about ports:
"
Since OVS 2.7, OVN central must be configured to listen to requests on
appropriate ports:
ovn-sbctl set-connection ptcp:6642
ovn-nbctl set-connection ptcp:6641
"
and in my case I indeed passed from 2.6.90 to 2.7.0...
Do I need to run these two commands?
Or any other configuration settings?
Thanks in advance,
Gianluca
4:09 a.m.
On Thu, Apr 20, 2017 at 6:54 PM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com>
wrote:
Hello,
I installed some months ago a test setup in 4.1.0 with ovn.
Now after updating engine and host to 4.1.1 it seems the services are up
but it doesn't work.
If I run a VM with a network device in OVN external provider, it cant'
boot and I get this in engine.log:
[snip]
At the page
https://www.ovirt.org/blog/2016/11/ovirt-provider-ovn/
I see this note about ports:
"
Since OVS 2.7, OVN central must be configured to listen to requests on
appropriate ports:
ovn-sbctl set-connection ptcp:6642
ovn-nbctl set-connection ptcp:6641
"
and in my case I indeed passed from 2.6.90 to 2.7.0...
Do I need to run these two commands?
Or any other configuration settings?
Thanks in advance,
Gianluca
I confirm that after running these two commands all work ok again and I'm
able to start a VM with a vnic provided by the OVN provider
[root@ractorshe ~]# ovn-sbctl set-connection ptcp:6642
[root@ractorshe ~]# ovn-nbctl set-connection ptcp:6641
[root@ractorshe ~]#
2:32 p.m.
Hello Gianluca,
Can you please check the ovn north db log.
This is placed in /var/log/openvswitch/ovsdb-server-nb.log
Please check if the logs has any new entries when you try to connect and
when you issue the 'ovn-nbctl set-connection ptcp:6641' command.
If the connection attempt is getting through, pvs db should print an error
to the log.
Please also try restarting the ovn-northd service.
Do the ovn-controllers connect to the south-db?
You can verify this by looking at /var/log/openvswitch/ovn-controller.log
on the ovn-controller host (please look for entries saying "... <north
ip>:6642 connected")
Marcin
On Fri, Apr 21, 2017 at 1:09 PM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com>
wrote:
On Thu, Apr 20, 2017 at 6:54 PM, Gianluca Cecchi <
gianluca.cecchi(a)gmail.com> wrote:
> Hello,
> I installed some months ago a test setup in 4.1.0 with ovn.
> Now after updating engine and host to 4.1.1 it seems the services are up
> but it doesn't work.
> If I run a VM with a network device in OVN external provider, it cant'
> boot and I get this in engine.log:
>
>
> [snip]
>
> At the page
> https://www.ovirt.org/blog/2016/11/ovirt-provider-ovn/
>
> I see this note about ports:
>
> "
> Since OVS 2.7, OVN central must be configured to listen to requests on
> appropriate ports:
>
> ovn-sbctl set-connection ptcp:6642
> ovn-nbctl set-connection ptcp:6641
> "
>
> and in my case I indeed passed from 2.6.90 to 2.7.0...
>
> Do I need to run these two commands?
> Or any other configuration settings?
>
> Thanks in advance,
> Gianluca
>
I confirm that after running these two commands all work ok again and I'm
able to start a VM with a vnic provided by the OVN provider
[root@ractorshe ~]# ovn-sbctl set-connection ptcp:6642
[root@ractorshe ~]# ovn-nbctl set-connection ptcp:6641
[root@ractorshe ~]#
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
MARCIN mIRECKI
Red Hat
<https://www.redhat.com>
<https://red.ht/sig>
1:37 a.m.
On Sun, Apr 23, 2017 at 11:32 PM, Marcin Mirecki <mmirecki(a)redhat.com>
wrote:
Hello Gianluca,
Can you please check the ovn north db log.
This is placed in /var/log/openvswitch/ovsdb-server-nb.log
Please check if the logs has any new entries when you try to connect and
when you issue the 'ovn-nbctl set-connection ptcp:6641' command.
If the connection attempt is getting through, pvs db should print an error
to the log.
Please also try restarting the ovn-northd service.
Do the ovn-controllers connect to the south-db?
You can verify this by looking at /var/log/openvswitch/ovn-controller.log
on the ovn-controller host (please look for entries saying "... <north
ip>:6642 connected")
Marcin
The ovirt nb log contains:
2017-04-24T07:46:51.541Z|00001|vlog|INFO|opened log file
/var/log/openvswitch/ovsdb-server-nb.log
2017-04-24T07:46:51.550Z|00002|ovsdb_server|INFO|ovsdb-server (Open
vSwitch) 2.7.0
2017-04-24T07:47:01.560Z|00003|memory|INFO|2268 kB peak resident set size
after 10.0 seconds
2017-04-24T07:47:01.560Z|00004|memory|INFO|cells:100 json-caches:1
monitors:1 ses
In my ovn-controller.log of my 3 hosts I have this, when I run the 2
commands below on the provider host
ovn-sbctl set-connection ptcp:6642
ovn-nbctl set-connection ptcp:6641
2017-04-24T07:56:23.178Z|00247|reconnect|INFO|tcp:10.4.192.43:6642:
connecting...
2017-04-24T07:56:23.178Z|00248|reconnect|INFO|tcp:10.4.192.43:6642:
connection attempt failed (Connection refused)
2017-04-24T07:56:23.178Z|00249|reconnect|INFO|tcp:10.4.192.43:6642: waiting
8 seconds before reconnect
2017-04-24T07:56:31.187Z|00250|reconnect|INFO|tcp:10.4.192.43:6642:
connecting...
2017-04-24T07:56:31.188Z|00251|reconnect|INFO|tcp:10.4.192.43:6642:
connected
2017-04-24T07:56:31.193Z|00252|ofctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt:
connecting to switch
2017-04-24T07:56:31.193Z|00253|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt:
connecting...
2017-04-24T07:56:31.201Z|00254|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt:
connected
2017-04-24T07:56:31.201Z|00255|pinctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt:
connecting to switch
2017-04-24T07:56:31.201Z|00256|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt:
connecting...
2017-04-24T07:56:31.201Z|00257|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt:
connected
2017-04-24T07:56:31.202Z|00258|binding|INFO|Releasing lport
0a2a47bc-ea0d-4f1d-8f49-ec903e519983 from this chassis.
On my provider I see then the 3 oVirt hosts connected:
[root@ovmgr1 openvswitch]# netstat -tpn|grep 66
tcp 0 0 10.4.192.43:6642 10.4.168.76:38882
ESTABLISHED 6699/ovsdb-server
tcp 0 0 10.4.192.43:6642 10.4.168.75:45486
ESTABLISHED 6699/ovsdb-server
tcp 0 0 127.0.0.1:5432 127.0.0.1:37074
ESTABLISHED 16696/postgres: eng
tcp 0 0 10.4.192.43:6642 10.4.168.74:59176
ESTABLISHED 6699/ovsdb-server
[root@ovmgr1 openvswitch]#
But it seems that the "set" command above is not persistent across reboot
of the provider host.... that in my case is the oVirt engine server
1:03 p.m.
Looks like the south db works properly. The north db uses the same
mechanism, just a different schema and port.
Looking at the netstat output it looks like ovn north db is not even
listening, or is there anything for 6641?
On Mon, Apr 24, 2017 at 10:37 AM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com
wrote:
>
>
> On Sun, Apr 23, 2017 at 11:32 PM, Marcin Mirecki <mmirecki(a)redhat.com>
wrote:
>
>> Hello Gianluca,
>>
>> Can you please check the ovn north db log.
>> This is placed in /var/log/openvswitch/ovsdb-server-nb.log
>> Please check if the logs has any new entries when you try to connect and
>> when you issue the 'ovn-nbctl set-connection ptcp:6641' command.
>> If the connection attempt is getting through, pvs db should print an
>> error to the log.
>>
>> Please also try restarting the ovn-northd service.
>>
>> Do the ovn-controllers connect to the south-db?
>> You can verify this by looking at /var/log/openvswitch/ovn-controller.log
>> on the ovn-controller host (please look for entries saying "... <north
>> ip>:6642 connected")
>>
>> Marcin
>>
>>
>>
> The ovirt nb log contains:
> 2017-04-24T07:46:51.541Z|00001|vlog|INFO|opened log file
> /var/log/openvswitch/ovsdb-server-nb.log
> 2017-04-24T07:46:51.550Z|00002|ovsdb_server|INFO|ovsdb-server (Open
> vSwitch) 2.7.0
> 2017-04-24T07:47:01.560Z|00003|memory|INFO|2268 kB peak resident set size
> after 10.0 seconds
> 2017-04-24T07:47:01.560Z|00004|memory|INFO|cells:100 json-caches:1
> monitors:1 ses
>
> In my ovn-controller.log of my 3 hosts I have this, when I run the 2
> commands below on the provider host
>
> ovn-sbctl set-connection ptcp:6642
> ovn-nbctl set-connection ptcp:6641
>
>
> 2017-04-24T07:56:23.178Z|00247|reconnect|INFO|tcp:10.4.192.43:6642:
> connecting...
> 2017-04-24T07:56:23.178Z|00248|reconnect|INFO|tcp:10.4.192.43:6642:
> connection attempt failed (Connection refused)
> 2017-04-24T07:56:23.178Z|00249|reconnect|INFO|tcp:10.4.192.43:6642:
> waiting 8 seconds before reconnect
> 2017-04-24T07:56:31.187Z|00250|reconnect|INFO|tcp:10.4.192.43:6642:
> connecting...
> 2017-04-24T07:56:31.188Z|00251|reconnect|INFO|tcp:10.4.192.43:6642:
> connected
> 2017-04-24T07:56:31.193Z|00252|ofctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt:
> connecting to switch
> 2017-04-24T07:56:31.193Z|00253|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt:
> connecting...
> 2017-04-24T07:56:31.201Z|00254|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt:
> connected
> 2017-04-24T07:56:31.201Z|00255|pinctrl|INFO|unix:/var/run/openvswitch/br-int.mgmt:
> connecting to switch
> 2017-04-24T07:56:31.201Z|00256|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt:
> connecting...
> 2017-04-24T07:56:31.201Z|00257|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt:
> connected
> 2017-04-24T07:56:31.202Z|00258|binding|INFO|Releasing lport
> 0a2a47bc-ea0d-4f1d-8f49-ec903e519983 from this chassis.
>
> On my provider I see then the 3 oVirt hosts connected:
> [root@ovmgr1 openvswitch]# netstat -tpn|grep 66
> tcp 0 0 10.4.192.43:6642 10.4.168.76:38882
> ESTABLISHED 6699/ovsdb-server
> tcp 0 0 10.4.192.43:6642 10.4.168.75:45486
> ESTABLISHED 6699/ovsdb-server
> tcp 0 0 127.0.0.1:5432 127.0.0.1:37074
> ESTABLISHED 16696/postgres: eng
> tcp 0 0 10.4.192.43:6642 10.4.168.74:59176
> ESTABLISHED 6699/ovsdb-server
> [root@ovmgr1 openvswitch]#
>
> But it seems that the "set" command above is not persistent across reboot
> of the provider host.... that in my case is the oVirt engine server
>
>
--
MARCIN mIRECKI
Red Hat
<https://www.redhat.com>
<https://red.ht/sig>
2:23 p.m.
On Mon, Apr 24, 2017 at 10:03 PM, Marcin Mirecki <mmirecki(a)redhat.com>
wrote:
Looks like the south db works properly. The north db uses the same
mechanism, just a different schema and port.
Looking at the netstat output it looks like ovn north db is not even
listening, or is there anything for 6641?
Actually yes... it seems that the switch "-t" with the "-p"
doesn't catch
the 6641 and 6642 "LISTEN" lines, while if I use "-a" instead of
"-t" I get
them too...
with "-a"
root@ovmgr1 ~]# netstat -apn | grep 664
tcp 0 0 0.0.0.0:6641 0.0.0.0:*
LISTEN 6691/ovsdb-server
tcp 0 0 0.0.0.0:6642 0.0.0.0:*
LISTEN 6699/ovsdb-server
tcp 0 0 10.4.192.43:6642 10.4.168.76:38882
ESTABLISHED 6699/ovsdb-server
tcp 0 0 10.4.192.43:6642 10.4.168.75:45486
ESTABLISHED 6699/ovsdb-server
tcp 0 0 10.4.192.43:6642 10.4.168.74:59176
ESTABLISHED 6699/ovsdb-server
unix 3 [ ] STREAM CONNECTED 14119
664/vmtoolsd
with "-t"
[root@ovmgr1 ~]# netstat -tpn | grep 664
tcp 0 0 10.4.192.43:6642 10.4.168.76:38882
ESTABLISHED 6699/ovsdb-server
tcp 0 0 10.4.192.43:6642 10.4.168.75:45486
ESTABLISHED 6699/ovsdb-server
tcp 0 0 10.4.192.43:6642 10.4.168.74:59176
ESTABLISHED 6699/ovsdb-server
12:34 a.m.
It looks like ndb is listening (netstat shows this), but not receiving
anything (log is empty).
This looks like a connectivity issue.
Can the provider host ping the ovn host?
Is firewalld/iptables disabled on both hosts?
Let's make sure that OVN is working fine.
I attach a short python script that checks the connection from the
localhost.
Please check if it can connect to ovn from localhost (just execute this on
the host with ovn: "python tcp_connection_test.py" ).
It should connect and print out all the OVN networks.
Does this work?
On Mon, Apr 24, 2017 at 11:23 PM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com
wrote:
>
> On Mon, Apr 24, 2017 at 10:03 PM, Marcin Mirecki <mmirecki(a)redhat.com>
wrote:
>
>> Looks like the south db works properly. The north db uses the same
>> mechanism, just a different schema and port.
>>
>> Looking at the netstat output it looks like ovn north db is not even
>> listening, or is there anything for 6641?
>>
>>
>>
>>
> Actually yes... it seems that the switch "-t" with the "-p"
doesn't catch
> the 6641 and 6642 "LISTEN" lines, while if I use "-a" instead of
"-t" I get
> them too...
>
> with "-a"
> root@ovmgr1 ~]# netstat -apn | grep 664
> tcp 0 0 0.0.0.0:6641 0.0.0.0:*
> LISTEN 6691/ovsdb-server
> tcp 0 0 0.0.0.0:6642 0.0.0.0:*
> LISTEN 6699/ovsdb-server
> tcp 0 0 10.4.192.43:6642 10.4.168.76:38882
> ESTABLISHED 6699/ovsdb-server
> tcp 0 0 10.4.192.43:6642 10.4.168.75:45486
> ESTABLISHED 6699/ovsdb-server
> tcp 0 0 10.4.192.43:6642 10.4.168.74:59176
> ESTABLISHED 6699/ovsdb-server
> unix 3 [ ] STREAM CONNECTED 14119
> 664/vmtoolsd
>
> with "-t"
> [root@ovmgr1 ~]# netstat -tpn | grep 664
> tcp 0 0 10.4.192.43:6642 10.4.168.76:38882
> ESTABLISHED 6699/ovsdb-server
> tcp 0 0 10.4.192.43:6642 10.4.168.75:45486
> ESTABLISHED 6699/ovsdb-server
> tcp 0 0 10.4.192.43:6642 10.4.168.74:59176
> ESTABLISHED 6699/ovsdb-server
>
>
>
--
MARCIN mIRECKI
Red Hat
<https://www.redhat.com>
<https://red.ht/sig>
3:52 a.m.
On Tue, Apr 25, 2017 at 9:34 AM, Marcin Mirecki <mmirecki(a)redhat.com> wrote:
It looks like ndb is listening (netstat shows this), but not
receiving
anything (log is empty).
This looks like a connectivity issue.
Can the provider host ping the ovn host?
Yes. The provider host is the oVirt self hosted engine VM
Is firewalld/iptables disabled on both hosts?
Firewall is disabled on all 3 oVirt hosts. While on engine VM side the
firewalld service is active but with needed ports enabled.
[g.cecchi@ovmgr1 ~]$ sudo firewall-cmd --get-default-zone
public
[g.cecchi@ovmgr1 ~]$ sudo firewall-cmd --get-active-zones
public
interfaces: ens192
[g.cecchi@ovmgr1 ~]$ sudo firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: ens192
sources:
services: dhcpv6-client ovirt-fence-kdump-listener ovirt-http ovirt-https
ovirt-imageio-proxy ovirt-postgres ovirt-provider-ovn ovirt-vmconsole-proxy
ovirt-websocket-proxy ssh
ports:
protocols:
masquerade: no
forward-ports:
sourceports:
icmp-blocks:
rich rules:
rule family="ipv4" port port="6641" protocol="tcp"
accept
rule family="ipv4" port port="6642" protocol="tcp"
accept
[g.cecchi@ovmgr1 ~]$
Let's make sure that OVN is working fine.
I attach a short python script that checks the connection from the
localhost.
Please check if it can connect to ovn from localhost (just execute this on
the host with ovn: "python tcp_connection_test.py" ).
It should connect and print out all the OVN networks.
Does this work?
<https://www.redhat.com>
It seems so. If I run on my ovm provider host:
[g.cecchi@ovmgr1 ~]$ python /tmp/tcp_connection_test.py
connecting 1
connecting 2
CONNECTED!
ROW: uuid:04501f6b-3977-4ba1-9ead-7096768d796d name:ovn172
ROW: uuid:6110649a-db2b-4de7-8fbc-601095cfe510 name:ovn192
[g.cecchi@ovmgr1 ~]$
Perhaps I was not clear at the beginning.... I try to reformulate.
I noticed that ovn didn't work after upgrade from 4.1.0 to 4.1.1.
Then in oVirt OVN documentation I found about the "set" commands to give
in case of openvswitch 2.7
And in fact after giving the commands I was able to start VM with OVN vnics
and they was able to communicate through openvswitch.
The problem was the persistence of the set command. It seems that after
restarting engine VM (that is my provider host), the setting is not
maintained....
Gianluca
2558
days inactive
2563
days old
7 comments
2 participants
participants (2)
-
Gianluca Cecchi -
Marcin Mirecki