ovn bad gateway after update from 4.2.7 to 4.2.8
Hello, at this moment (about two days ago) I have updated only engine (external, not self hosted) from 4.2.7.5 to 4.2.8.2 As soon as I'm starting for the first time a VM with an ovn based nic I get what below in ovirt-provider-ovn.log In admin gui, if I try for example to start via "run once" I get: " Error while executing action Run VM once: Failed to communicate with the external provider, see log for additional details. " Any clue? Thanks, Gianluca 2019-01-29 17:23:20,554 root Starting server 2019-01-29 17:23:20,554 root Version: 1.2.18-1 2019-01-29 17:23:20,555 root Build date: 20190114151850 2019-01-29 17:23:20,555 root Githash: dae4c1d 2019-01-29 18:04:15,575 root Starting server 2019-01-29 18:04:15,576 root Version: 1.2.18-1 2019-01-29 18:04:15,576 root Build date: 20190114151850 2019-01-29 18:04:15,576 root Githash: dae4c1d 2019-02-01 14:26:58,316 root From: ::ffff:127.0.0.1:49582 Request: GET /v2.0/ports 2019-02-01 14:26:58,317 root HTTPSConnectionPool(host='engine-host', port=443): Max retries exceeded with url: /ovirt-engine/sso/oauth/token-info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe806166b90>: Failed to establish a new connection: [Errno -2] Name or service not known',)) Traceback (most recent call last): File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 134, in _handle_request method, path_parts, content File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 175, in handle_request return self.call_response_handler(handler, content, parameters) File "/usr/share/ovirt-provider-ovn/handlers/neutron.py", line 33, in call_response_handler TOKEN_HTTP_HEADER_FIELD_NAME, '')): File "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 31, in validate_token return auth.core.plugin.validate_token(token) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/authorization_by_username.py", line 36, in validate_token return self._is_user_name(token, _admin_user_name()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/authorization_by_username.py", line 47, in _is_user_name timeout=AuthorizationByUserName._timeout()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 131, in get_token_info timeout=timeout File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 54, in wrapper response = func(*args, **kwargs) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 47, in wrapper raise BadGateway(e) BadGateway: HTTPSConnectionPool(host='engine-host', port=443): Max retries exceeded with url: /ovirt-engine/sso/oauth/token-info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe806166b90>: Failed to establish a new connection: [Errno -2] Name or service not known',)) 2019-02-01 14:27:26,968 root From: ::ffff:127.0.0.1:49590 Request: GET /v2.0/ports 2019-02-01 14:27:26,969 root HTTPSConnectionPool(host='engine-host', port=443): Max retries exceeded with url: /ovirt-engine/sso/oauth/token-info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe80618df50>: Failed to establish a new connection: [Errno -2] Name or service not known',)) Traceback (most recent call last): File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 134, in _handle_request method, path_parts, content File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 175, in handle_request return self.call_response_handler(handler, content, parameters) File "/usr/share/ovirt-provider-ovn/handlers/neutron.py", line 33, in call_response_handler TOKEN_HTTP_HEADER_FIELD_NAME, '')): File "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 31, in validate_token return auth.core.plugin.validate_token(token) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/authorization_by_username.py", line 36, in validate_token return self._is_user_name(token, _admin_user_name()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/authorization_by_username.py", line 47, in _is_user_name timeout=AuthorizationByUserName._timeout()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 131, in get_token_info timeout=timeout File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 54, in wrapper response = func(*args, **kwargs) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 47, in wrapper raise BadGateway(e) BadGateway: HTTPSConnectionPool(host='engine-host', port=443): Max retries exceeded with url: /ovirt-engine/sso/oauth/token-info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe80618df50>: Failed to establish a new connection: [Errno -2] Name or service not known',)) 2019-02-01 14:29:17,412 root From: ::ffff:127.0.0.1:49616 Request: GET /v2.0/ports 2019-02-01 14:29:17,412 root HTTPSConnectionPool(host='engine-host', port=443): Max retries exceeded with url: /ovirt-engine/sso/oauth/token-info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe80618de50>: Failed to establish a new connection: [Errno -2] Name or service not known',)) Traceback (most recent call last): File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 134, in _handle_request method, path_parts, content File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 175, in handle_request return self.call_response_handler(handler, content, parameters) File "/usr/share/ovirt-provider-ovn/handlers/neutron.py", line 33, in call_response_handler TOKEN_HTTP_HEADER_FIELD_NAME, '')): File "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 31, in validate_token return auth.core.plugin.validate_token(token) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/authorization_by_username.py", line 36, in validate_token return self._is_user_name(token, _admin_user_name()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/authorization_by_username.py", line 47, in _is_user_name timeout=AuthorizationByUserName._timeout()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 131, in get_token_info timeout=timeout File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 54, in wrapper response = func(*args, **kwargs) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 47, in wrapper raise BadGateway(e) BadGateway: HTTPSConnectionPool(host='engine-host', port=443): Max retries exceeded with url: /ovirt-engine/sso/oauth/token-info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe80618de50>: Failed to establish a new connection: [Errno -2] Name or service not known',))
On Fri, 1 Feb 2019 14:37:10 +0100 Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
Hello, at this moment (about two days ago) I have updated only engine (external, not self hosted) from 4.2.7.5 to 4.2.8.2
As soon as I'm starting for the first time a VM with an ovn based nic I get what below in ovirt-provider-ovn.log
In admin gui, if I try for example to start via "run once" I get: " Error while executing action Run VM once: Failed to communicate with the external provider, see log for additional details. " Any clue?
The ovirt-provider-ovn fails during checking the credentials at engine's sso because of a networking problem. Can you please check if the url of the config value ovirt-host in /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf can be reached from engine's host? If this does not explain the problem, can you please increase the logging of the ovirt-provider-ovn by sudo sed -i.$(date +%F-%H-%M) 's/INFO/DEBUG/gi' /etc/ovirt-provider-ovn/logger.conf systemctl restart ovirt-provider-ovn and share a new detailed error in ovirt-provider-ovn.log? Thanks.
Thanks, Gianluca
2019-01-29 17:23:20,554 root Starting server 2019-01-29 17:23:20,554 root Version: 1.2.18-1 2019-01-29 17:23:20,555 root Build date: 20190114151850 2019-01-29 17:23:20,555 root Githash: dae4c1d 2019-01-29 18:04:15,575 root Starting server 2019-01-29 18:04:15,576 root Version: 1.2.18-1 2019-01-29 18:04:15,576 root Build date: 20190114151850 2019-01-29 18:04:15,576 root Githash: dae4c1d 2019-02-01 14:26:58,316 root From: ::ffff:127.0.0.1:49582 Request: GET /v2.0/ports 2019-02-01 14:26:58,317 root HTTPSConnectionPool(host='engine-host', port=443): Max retries exceeded with url: /ovirt-engine/sso/oauth/token-info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe806166b90>: Failed to establish a new connection: [Errno -2] 0x7fe806166b90>Name or service not known',)) Traceback (most recent call last): File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 134, in _handle_request method, path_parts, content File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 175, in handle_request return self.call_response_handler(handler, content, parameters) File "/usr/share/ovirt-provider-ovn/handlers/neutron.py", line 33, in call_response_handler TOKEN_HTTP_HEADER_FIELD_NAME, '')): File "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 31, in validate_token return auth.core.plugin.validate_token(token) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/authorization_by_username.py", line 36, in validate_token return self._is_user_name(token, _admin_user_name()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/authorization_by_username.py", line 47, in _is_user_name timeout=AuthorizationByUserName._timeout()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 131, in get_token_info timeout=timeout File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 54, in wrapper response = func(*args, **kwargs) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 47, in wrapper raise BadGateway(e) BadGateway: HTTPSConnectionPool(host='engine-host', port=443): Max retries exceeded with url: /ovirt-engine/sso/oauth/token-info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe806166b90>: Failed to establish a new connection: [Errno -2] 0x7fe806166b90>Name or service not known',)) 2019-02-01 14:27:26,968 root From: ::ffff:127.0.0.1:49590 Request: GET /v2.0/ports 2019-02-01 14:27:26,969 root HTTPSConnectionPool(host='engine-host', port=443): Max retries exceeded with url: /ovirt-engine/sso/oauth/token-info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe80618df50>: Failed to establish a new connection: [Errno -2] 0x7fe80618df50>Name or service not known',)) Traceback (most recent call last): File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 134, in _handle_request method, path_parts, content File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 175, in handle_request return self.call_response_handler(handler, content, parameters) File "/usr/share/ovirt-provider-ovn/handlers/neutron.py", line 33, in call_response_handler TOKEN_HTTP_HEADER_FIELD_NAME, '')): File "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 31, in validate_token return auth.core.plugin.validate_token(token) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/authorization_by_username.py", line 36, in validate_token return self._is_user_name(token, _admin_user_name()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/authorization_by_username.py", line 47, in _is_user_name timeout=AuthorizationByUserName._timeout()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 131, in get_token_info timeout=timeout File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 54, in wrapper response = func(*args, **kwargs) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 47, in wrapper raise BadGateway(e) BadGateway: HTTPSConnectionPool(host='engine-host', port=443): Max retries exceeded with url: /ovirt-engine/sso/oauth/token-info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe80618df50>: Failed to establish a new connection: [Errno -2] 0x7fe80618df50>Name or service not known',)) 2019-02-01 14:29:17,412 root From: ::ffff:127.0.0.1:49616 Request: GET /v2.0/ports 2019-02-01 14:29:17,412 root HTTPSConnectionPool(host='engine-host', port=443): Max retries exceeded with url: /ovirt-engine/sso/oauth/token-info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe80618de50>: Failed to establish a new connection: [Errno -2] 0x7fe80618de50>Name or service not known',)) Traceback (most recent call last): File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 134, in _handle_request method, path_parts, content File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 175, in handle_request return self.call_response_handler(handler, content, parameters) File "/usr/share/ovirt-provider-ovn/handlers/neutron.py", line 33, in call_response_handler TOKEN_HTTP_HEADER_FIELD_NAME, '')): File "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 31, in validate_token return auth.core.plugin.validate_token(token) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/authorization_by_username.py", line 36, in validate_token return self._is_user_name(token, _admin_user_name()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/authorization_by_username.py", line 47, in _is_user_name timeout=AuthorizationByUserName._timeout()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 131, in get_token_info timeout=timeout File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 54, in wrapper response = func(*args, **kwargs) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 47, in wrapper raise BadGateway(e) BadGateway: HTTPSConnectionPool(host='engine-host', port=443): Max retries exceeded with url: /ovirt-engine/sso/oauth/token-info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe80618de50>: Failed to establish a new connection: [Errno -2] 0x7fe80618de50>Name or service not known',))
On Fri, Feb 1, 2019 at 10:18 PM Dominik Holler <dholler@redhat.com> wrote:
On Fri, 1 Feb 2019 14:37:10 +0100 Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
Hello, at this moment (about two days ago) I have updated only engine (external, not self hosted) from 4.2.7.5 to 4.2.8.2
As soon as I'm starting for the first time a VM with an ovn based nic I get what below in ovirt-provider-ovn.log
In admin gui, if I try for example to start via "run once" I get: " Error while executing action Run VM once: Failed to communicate with the external provider, see log for additional details. " Any clue?
The ovirt-provider-ovn fails during checking the credentials at engine's sso because of a networking problem.
That would be odd - after all we're using the loopback interface From: ::ffff:127.0.0.1:49582 Request: GET /v2.0/ports but please try the url.
Can you please check if the url of the config value ovirt-host in /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf can be reached from engine's host? If this does not explain the problem, can you please increase the logging of the ovirt-provider-ovn by sudo sed -i.$(date +%F-%H-%M) 's/INFO/DEBUG/gi' /etc/ovirt-provider-ovn/logger.conf systemctl restart ovirt-provider-ovn and share a new detailed error in ovirt-provider-ovn.log? Thanks.
Dominik, could it possibly be related to our hardening TLS ciphers? If it is, setting (an insecure) ssl-ciphers-string=DEFAULT in /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf would help.
Thanks, Gianluca
2019-01-29 17:23:20,554 root Starting server 2019-01-29 17:23:20,554 root Version: 1.2.18-1 2019-01-29 17:23:20,555 root Build date: 20190114151850 2019-01-29 17:23:20,555 root Githash: dae4c1d 2019-01-29 18:04:15,575 root Starting server 2019-01-29 18:04:15,576 root Version: 1.2.18-1 2019-01-29 18:04:15,576 root Build date: 20190114151850 2019-01-29 18:04:15,576 root Githash: dae4c1d 2019-02-01 14:26:58,316 root From: ::ffff:127.0.0.1:49582 Request: GET /v2.0/ports 2019-02-01 14:26:58,317 root HTTPSConnectionPool(host='engine-host', port=443): Max retries exceeded with url: /ovirt-engine/sso/oauth/token-info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe806166b90>: Failed to establish a new connection: [Errno -2] 0x7fe806166b90>Name or service not known',)) Traceback (most recent call last): File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 134, in _handle_request method, path_parts, content File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 175, in handle_request return self.call_response_handler(handler, content, parameters) File "/usr/share/ovirt-provider-ovn/handlers/neutron.py", line 33, in call_response_handler TOKEN_HTTP_HEADER_FIELD_NAME, '')): File "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 31, in validate_token return auth.core.plugin.validate_token(token) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/authorization_by_username.py", line 36, in validate_token return self._is_user_name(token, _admin_user_name()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/authorization_by_username.py", line 47, in _is_user_name timeout=AuthorizationByUserName._timeout()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 131, in get_token_info timeout=timeout File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 54, in wrapper response = func(*args, **kwargs) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 47, in wrapper raise BadGateway(e) BadGateway: HTTPSConnectionPool(host='engine-host', port=443): Max retries exceeded with url: /ovirt-engine/sso/oauth/token-info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe806166b90>: Failed to establish a new connection: [Errno -2] 0x7fe806166b90>Name or service not known',))
On Fri, 1 Feb 2019 22:35:00 +0200 Dan Kenigsberg <danken@redhat.com> wrote:
On Fri, Feb 1, 2019 at 10:18 PM Dominik Holler <dholler@redhat.com> wrote:
On Fri, 1 Feb 2019 14:37:10 +0100 Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
Hello, at this moment (about two days ago) I have updated only engine (external, not self hosted) from 4.2.7.5 to 4.2.8.2
As soon as I'm starting for the first time a VM with an ovn based nic I get what below in ovirt-provider-ovn.log
In admin gui, if I try for example to start via "run once" I get: " Error while executing action Run VM once: Failed to communicate with the external provider, see log for additional details. " Any clue?
The ovirt-provider-ovn fails during checking the credentials at engine's sso because of a networking problem.
That would be odd - after all we're using the loopback interface From: ::ffff:127.0.0.1:49582 Request: GET /v2.0/ports but please try the url.
Communication from Engine to ovirt-provider-ovn via OpenStack API looks good. The problem seems to be in the communication from ovirt-provider-ovn to engine's sso. The hostname to resolve seems to be 'engine-host'.
Can you please check if the url of the config value ovirt-host in /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf can be reached from engine's host? If this does not explain the problem, can you please increase the logging of the ovirt-provider-ovn by sudo sed -i.$(date +%F-%H-%M) 's/INFO/DEBUG/gi' /etc/ovirt-provider-ovn/logger.conf systemctl restart ovirt-provider-ovn and share a new detailed error in ovirt-provider-ovn.log? Thanks.
Dominik, could it possibly be related to our hardening TLS ciphers? If it is, setting (an insecure) ssl-ciphers-string=DEFAULT in /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf would help.
I do not expect this, because this setting should be applied only to OpenStack API related communication, which looks good here.
Thanks, Gianluca
2019-01-29 17:23:20,554 root Starting server 2019-01-29 17:23:20,554 root Version: 1.2.18-1 2019-01-29 17:23:20,555 root Build date: 20190114151850 2019-01-29 17:23:20,555 root Githash: dae4c1d 2019-01-29 18:04:15,575 root Starting server 2019-01-29 18:04:15,576 root Version: 1.2.18-1 2019-01-29 18:04:15,576 root Build date: 20190114151850 2019-01-29 18:04:15,576 root Githash: dae4c1d 2019-02-01 14:26:58,316 root From: ::ffff:127.0.0.1:49582 Request: GET /v2.0/ports 2019-02-01 14:26:58,317 root HTTPSConnectionPool(host='engine-host', port=443): Max retries exceeded with url: /ovirt-engine/sso/oauth/token-info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe806166b90>: Failed to establish a new connection: [Errno -2] 0x7fe806166b90>Name or service not known',)) Traceback (most recent call last): File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 134, in _handle_request method, path_parts, content File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 175, in handle_request return self.call_response_handler(handler, content, parameters) File "/usr/share/ovirt-provider-ovn/handlers/neutron.py", line 33, in call_response_handler TOKEN_HTTP_HEADER_FIELD_NAME, '')): File "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 31, in validate_token return auth.core.plugin.validate_token(token) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/authorization_by_username.py", line 36, in validate_token return self._is_user_name(token, _admin_user_name()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/authorization_by_username.py", line 47, in _is_user_name timeout=AuthorizationByUserName._timeout()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 131, in get_token_info timeout=timeout File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 54, in wrapper response = func(*args, **kwargs) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 47, in wrapper raise BadGateway(e) BadGateway: HTTPSConnectionPool(host='engine-host', port=443): Max retries exceeded with url: /ovirt-engine/sso/oauth/token-info (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fe806166b90>: Failed to establish a new connection: [Errno -2] 0x7fe806166b90>Name or service not known',))
On Fri, Feb 1, 2019 at 10:51 PM Dominik Holler <dholler@redhat.com> wrote:
On Fri, 1 Feb 2019 22:35:00 +0200 Dan Kenigsberg <danken@redhat.com> wrote:
On Fri, Feb 1, 2019 at 10:18 PM Dominik Holler <dholler@redhat.com> wrote:
On Fri, 1 Feb 2019 14:37:10 +0100 Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
Hello, at this moment (about two days ago) I have updated only engine (external, not self hosted) from 4.2.7.5 to 4.2.8.2
As soon as I'm starting for the first time a VM with an ovn based nic I get what below in ovirt-provider-ovn.log
In admin gui, if I try for example to start via "run once" I get: " Error while executing action Run VM once: Failed to communicate with the external provider, see log for additional details. " Any clue?
The ovirt-provider-ovn fails during checking the credentials at engine's sso because of a networking problem.
That would be odd - after all we're using the loopback interface From: ::ffff:127.0.0.1:49582 Request: GET /v2.0/ports but please try the url.
Communication from Engine to ovirt-provider-ovn via OpenStack API looks good. The problem seems to be in the communication from ovirt-provider-ovn to engine's sso. The hostname to resolve seems to be 'engine-host'.
Hi all, so it seems that updating from 4.2.7 to 4.2.8 wiped the file entirely... In fact now I only have: [root@ovmgr1 ~]# ll /etc/ovirt-provider-ovn/conf.d/ total 4 -rw-r--r--. 1 root root 194 Jan 14 16:18 README [root@ovmgr1 ~]# While on another similar environment in 4.2.7 I still have: [root@ovirt ~]# ll /etc/ovirt-provider-ovn/conf.d/ total 8 -rw-r--r--. 1 root root 561 Dec 31 2017 10-setup-ovirt-provider-ovn.conf -rw-r--r--. 1 root root 194 Oct 17 05:56 README [root@ovirt ~]# So in my updated environment I think it tries to use the default file values in /etc/ovirt-provider-ovn/ovirt-provider-ovn.conf [OVIRT] ovirt-host=https://engine-host The question now is why it has been wiped out? This environment has external VM working as ovirt engine, so not SHE. I see that engine-backup doesn't run backup of the /etc/ovirt-provider-ovn/ ... could it be useful to include it? I should have a VSPhere image backup of the VM so I should be able to restore the file. I'll see... In the mean time could it be the update of the package has been the responsible? Eg, in my 4.2.7 env I currently have ovirt-provider-ovn-1.2.16-1.el7.noarch The update to 4.2.8 in the other system has put in ovirt-provider-ovn-1.2.18-1.el7.noarch HIH debugging Gianluca
On Mon, Feb 4, 2019 at 2:47 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Fri, Feb 1, 2019 at 10:51 PM Dominik Holler <dholler@redhat.com> wrote:
On Fri, 1 Feb 2019 22:35:00 +0200 Dan Kenigsberg <danken@redhat.com> wrote:
On Fri, Feb 1, 2019 at 10:18 PM Dominik Holler <dholler@redhat.com> wrote:
On Fri, 1 Feb 2019 14:37:10 +0100 Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
Hello, at this moment (about two days ago) I have updated only engine (external, not self hosted) from 4.2.7.5 to 4.2.8.2
As soon as I'm starting for the first time a VM with an ovn based nic I get what below in ovirt-provider-ovn.log
In admin gui, if I try for example to start via "run once" I get: " Error while executing action Run VM once: Failed to communicate with the external provider, see log for additional details. " Any clue?
The ovirt-provider-ovn fails during checking the credentials at engine's sso because of a networking problem.
That would be odd - after all we're using the loopback interface From: ::ffff:127.0.0.1:49582 Request: GET /v2.0/ports but please try the url.
Communication from Engine to ovirt-provider-ovn via OpenStack API looks good. The problem seems to be in the communication from ovirt-provider-ovn to engine's sso. The hostname to resolve seems to be 'engine-host'.
Hi all, so it seems that updating from 4.2.7 to 4.2.8 wiped the file entirely...
In fact now I only have:
[root@ovmgr1 ~]# ll /etc/ovirt-provider-ovn/conf.d/ total 4 -rw-r--r--. 1 root root 194 Jan 14 16:18 README [root@ovmgr1 ~]#
While on another similar environment in 4.2.7 I still have:
[root@ovirt ~]# ll /etc/ovirt-provider-ovn/conf.d/ total 8 -rw-r--r--. 1 root root 561 Dec 31 2017 10-setup-ovirt-provider-ovn.conf -rw-r--r--. 1 root root 194 Oct 17 05:56 README [root@ovirt ~]#
So in my updated environment I think it tries to use the default file values in /etc/ovirt-provider-ovn/ovirt-provider-ovn.conf [OVIRT] ovirt-host=https://engine-host
The question now is why it has been wiped out? This environment has external VM working as ovirt engine, so not SHE.
I see that engine-backup doesn't run backup of the /etc/ovirt-provider-ovn/ ... could it be useful to include it? I should have a VSPhere image backup of the VM so I should be able to restore the file. I'll see...
In the mean time could it be the update of the package has been the responsible? Eg, in my 4.2.7 env I currently have ovirt-provider-ovn-1.2.16-1.el7.noarch The update to 4.2.8 in the other system has put in ovirt-provider-ovn-1.2.18-1.el7.noarch
HIH debugging Gianluca
My update from 4.2.7 to 4.2.8 has been executed on 29th of January around 18:00 I have restored a VM image the day before and it seems that the /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf was not there even before the update.... The diff between the previous /etc/ovirt-provider-ovn/ovirt-provider-ovn.conf and the new one put in place by the update is [root@ovmgr1 ovirt-provider-ovn]# diff old_ovirt-provider-ovn.conf ovirt-provider-ovn.conf 11a12
url_filter_exception=limit,page_reverse,next,previous [root@ovmgr1 ovirt-provider-ovn]#
At this point, as it has been some time I didn't power up VMs on OVN in this environment, I go back with past updates. On this environment the OVN part was installed manually when not yet included in standard setup. So I find this thread of mine about update from 4.1.9 to 4.2.3 on May 2018 I had a doubt about OVN and update and posted here: https://www.mail-archive.com/users@ovirt.org/msg48491.html So based on Marcin feedback I selected NO for ovirt-ovn-provider question and perhaps the following updates caused file to be removed... perhaps... How can I solve now? Could it be that updating to 4.3 and selecting yes for ovn now, I get a new config for it? Or running engine-setup for 4.2.8? Thanks, Gianluca
On Mon, 4 Feb 2019 15:57:24 +0100 Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Mon, Feb 4, 2019 at 2:47 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Fri, Feb 1, 2019 at 10:51 PM Dominik Holler <dholler@redhat.com> wrote:
On Fri, 1 Feb 2019 22:35:00 +0200 Dan Kenigsberg <danken@redhat.com> wrote:
On Fri, Feb 1, 2019 at 10:18 PM Dominik Holler <dholler@redhat.com> wrote:
On Fri, 1 Feb 2019 14:37:10 +0100 Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
Hello, at this moment (about two days ago) I have updated only engine (external, not self hosted) from 4.2.7.5 to 4.2.8.2
As soon as I'm starting for the first time a VM with an ovn based nic I get what below in ovirt-provider-ovn.log
In admin gui, if I try for example to start via "run once" I get: " Error while executing action Run VM once: Failed to communicate with the external provider, see log for additional details. " Any clue?
The ovirt-provider-ovn fails during checking the credentials at engine's sso because of a networking problem.
That would be odd - after all we're using the loopback interface From: ::ffff:127.0.0.1:49582 Request: GET /v2.0/ports but please try the url.
Communication from Engine to ovirt-provider-ovn via OpenStack API looks good. The problem seems to be in the communication from ovirt-provider-ovn to engine's sso. The hostname to resolve seems to be 'engine-host'.
Hi all, so it seems that updating from 4.2.7 to 4.2.8 wiped the file entirely...
In fact now I only have:
[root@ovmgr1 ~]# ll /etc/ovirt-provider-ovn/conf.d/ total 4 -rw-r--r--. 1 root root 194 Jan 14 16:18 README [root@ovmgr1 ~]#
While on another similar environment in 4.2.7 I still have:
[root@ovirt ~]# ll /etc/ovirt-provider-ovn/conf.d/ total 8 -rw-r--r--. 1 root root 561 Dec 31 2017 10-setup-ovirt-provider-ovn.conf -rw-r--r--. 1 root root 194 Oct 17 05:56 README [root@ovirt ~]#
So in my updated environment I think it tries to use the default file values in /etc/ovirt-provider-ovn/ovirt-provider-ovn.conf [OVIRT] ovirt-host=https://engine-host
The question now is why it has been wiped out? This environment has external VM working as ovirt engine, so not SHE.
I see that engine-backup doesn't run backup of the /etc/ovirt-provider-ovn/ ... could it be useful to include it?
Sounds like a valuable idea, would you create a bug?
I should have a VSPhere image backup of the VM so I should be able to restore the file. I'll see...
In the mean time could it be the update of the package has been the responsible? Eg, in my 4.2.7 env I currently have ovirt-provider-ovn-1.2.16-1.el7.noarch The update to 4.2.8 in the other system has put in ovirt-provider-ovn-1.2.18-1.el7.noarch
HIH debugging Gianluca
My update from 4.2.7 to 4.2.8 has been executed on 29th of January around 18:00 I have restored a VM image the day before and it seems that the /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf was not there even before the update....
The diff between the previous /etc/ovirt-provider-ovn/ovirt-provider-ovn.conf and the new one put in place by the update is
[root@ovmgr1 ovirt-provider-ovn]# diff old_ovirt-provider-ovn.conf ovirt-provider-ovn.conf 11a12
url_filter_exception=limit,page_reverse,next,previous [root@ovmgr1 ovirt-provider-ovn]#
At this point, as it has been some time I didn't power up VMs on OVN in this environment, I go back with past updates. On this environment the OVN part was installed manually when not yet included in standard setup. So I find this thread of mine about update from 4.1.9 to 4.2.3 on May 2018 I had a doubt about OVN and update and posted here: https://www.mail-archive.com/users@ovirt.org/msg48491.html
Thanks for figuring this out.
So based on Marcin feedback I selected NO for ovirt-ovn-provider question and perhaps the following updates caused file to be removed... perhaps...
This file is generated by engine-setup if Yes is selected for ovirt-ovn-provider, so I expect the file never existed.
How can I solve now?
Remove or rename the ovirt-provider-ovn provider in oVirt Engine, if it already exsits and run engine-setup with answer-file that has 'OVESETUP_OVN/ovirtProviderOvn' enabled: 'OVESETUP_OVN/ovirtProviderOvn=bool:True' should do the trick. Didi, can you please ack that this will not destroy something except ovirt-provider-ovn related things?
Could it be that updating to 4.3 and selecting yes for ovn now, I get a new config for it? Or running engine-setup for 4.2.8? Thanks, Gianluca
On Mon, Feb 4, 2019 at 5:16 PM Dominik Holler <dholler@redhat.com> wrote:
On Mon, 4 Feb 2019 15:57:24 +0100 Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Mon, Feb 4, 2019 at 2:47 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Fri, Feb 1, 2019 at 10:51 PM Dominik Holler <dholler@redhat.com> wrote:
On Fri, 1 Feb 2019 22:35:00 +0200 Dan Kenigsberg <danken@redhat.com> wrote:
On Fri, Feb 1, 2019 at 10:18 PM Dominik Holler <dholler@redhat.com> wrote:
On Fri, 1 Feb 2019 14:37:10 +0100 Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
> Hello, > at this moment (about two days ago) I have updated only engine > (external, not self hosted) from 4.2.7.5 to 4.2.8.2 > > As soon as I'm starting for the first time a VM with an ovn based > nic I get what below in ovirt-provider-ovn.log > > In admin gui, if I try for example to start via "run once" I get: > " > Error while executing action Run VM once: Failed to communicate > with the external provider, see log for additional details. > " > Any clue?
The ovirt-provider-ovn fails during checking the credentials at engine's sso because of a networking problem.
That would be odd - after all we're using the loopback interface From: ::ffff:127.0.0.1:49582 Request: GET /v2.0/ports but please try the url.
Communication from Engine to ovirt-provider-ovn via OpenStack API looks good. The problem seems to be in the communication from ovirt-provider-ovn to engine's sso. The hostname to resolve seems to be 'engine-host'.
Hi all, so it seems that updating from 4.2.7 to 4.2.8 wiped the file entirely...
In fact now I only have:
[root@ovmgr1 ~]# ll /etc/ovirt-provider-ovn/conf.d/ total 4 -rw-r--r--. 1 root root 194 Jan 14 16:18 README [root@ovmgr1 ~]#
While on another similar environment in 4.2.7 I still have:
[root@ovirt ~]# ll /etc/ovirt-provider-ovn/conf.d/ total 8 -rw-r--r--. 1 root root 561 Dec 31 2017 10-setup-ovirt-provider-ovn.conf -rw-r--r--. 1 root root 194 Oct 17 05:56 README [root@ovirt ~]#
So in my updated environment I think it tries to use the default file values in /etc/ovirt-provider-ovn/ovirt-provider-ovn.conf [OVIRT] ovirt-host=https://engine-host
The question now is why it has been wiped out? This environment has external VM working as ovirt engine, so not SHE.
I see that engine-backup doesn't run backup of the /etc/ovirt-provider-ovn/ ... could it be useful to include it?
Sounds like a valuable idea, would you create a bug?
I should have a VSPhere image backup of the VM so I should be able to restore the file. I'll see...
In the mean time could it be the update of the package has been the responsible? Eg, in my 4.2.7 env I currently have ovirt-provider-ovn-1.2.16-1.el7.noarch The update to 4.2.8 in the other system has put in ovirt-provider-ovn-1.2.18-1.el7.noarch
HIH debugging Gianluca
My update from 4.2.7 to 4.2.8 has been executed on 29th of January around 18:00 I have restored a VM image the day before and it seems that the /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf was not there even before the update....
The diff between the previous /etc/ovirt-provider-ovn/ovirt-provider-ovn.conf and the new one put in place by the update is
[root@ovmgr1 ovirt-provider-ovn]# diff old_ovirt-provider-ovn.conf ovirt-provider-ovn.conf 11a12
url_filter_exception=limit,page_reverse,next,previous [root@ovmgr1 ovirt-provider-ovn]#
At this point, as it has been some time I didn't power up VMs on OVN in this environment, I go back with past updates. On this environment the OVN part was installed manually when not yet included in standard setup. So I find this thread of mine about update from 4.1.9 to 4.2.3 on May 2018 I had a doubt about OVN and update and posted here: https://www.mail-archive.com/users@ovirt.org/msg48491.html
Thanks for figuring this out.
So based on Marcin feedback I selected NO for ovirt-ovn-provider question and perhaps the following updates caused file to be removed... perhaps...
This file is generated by engine-setup if Yes is selected for ovirt-ovn-provider, so I expect the file never existed.
How can I solve now?
Remove or rename the ovirt-provider-ovn provider in oVirt Engine, if it already exsits and run engine-setup with answer-file that has 'OVESETUP_OVN/ovirtProviderOvn' enabled: 'OVESETUP_OVN/ovirtProviderOvn=bool:True' should do the trick.
Didi, can you please ack that this will not destroy something except ovirt-provider-ovn related things?
I think this should work. Also this should work: engine-setup --reconfigure-optional-components This option to engine-setup "clears" the saved 'No' answers to all questions that their key is marked reconfigurable=True in the relevant constants.py file. This does include OVESETUP_OVN/ovirtProviderOvn , so passing this option should make it ask you again (as if no answer was provided). That said, I didn't test any of these two options, so you better try first on a test system, or have good backups.
Could it be that updating to 4.3 and selecting yes for ovn now, I get a new config for it? Or running engine-setup for 4.2.8?
None of these would ask you again, because the answer is already saved (in /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf ). Best regards, -- Didi
On Mon, 4 Feb 2019 16:15:56 +0100 Dominik Holler <dholler@redhat.com> wrote:
On Mon, 4 Feb 2019 15:57:24 +0100 Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Mon, Feb 4, 2019 at 2:47 PM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Fri, Feb 1, 2019 at 10:51 PM Dominik Holler <dholler@redhat.com> wrote:
On Fri, 1 Feb 2019 22:35:00 +0200 Dan Kenigsberg <danken@redhat.com> wrote:
On Fri, Feb 1, 2019 at 10:18 PM Dominik Holler <dholler@redhat.com> wrote:
On Fri, 1 Feb 2019 14:37:10 +0100 Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
> Hello, > at this moment (about two days ago) I have updated only engine > (external, not self hosted) from 4.2.7.5 to 4.2.8.2 > > As soon as I'm starting for the first time a VM with an ovn based > nic I get what below in ovirt-provider-ovn.log > > In admin gui, if I try for example to start via "run once" I get: > " > Error while executing action Run VM once: Failed to communicate > with the external provider, see log for additional details. > " > Any clue?
The ovirt-provider-ovn fails during checking the credentials at engine's sso because of a networking problem.
That would be odd - after all we're using the loopback interface From: ::ffff:127.0.0.1:49582 Request: GET /v2.0/ports but please try the url.
Communication from Engine to ovirt-provider-ovn via OpenStack API looks good. The problem seems to be in the communication from ovirt-provider-ovn to engine's sso. The hostname to resolve seems to be 'engine-host'.
Hi all, so it seems that updating from 4.2.7 to 4.2.8 wiped the file entirely...
In fact now I only have:
[root@ovmgr1 ~]# ll /etc/ovirt-provider-ovn/conf.d/ total 4 -rw-r--r--. 1 root root 194 Jan 14 16:18 README [root@ovmgr1 ~]#
While on another similar environment in 4.2.7 I still have:
[root@ovirt ~]# ll /etc/ovirt-provider-ovn/conf.d/ total 8 -rw-r--r--. 1 root root 561 Dec 31 2017 10-setup-ovirt-provider-ovn.conf -rw-r--r--. 1 root root 194 Oct 17 05:56 README [root@ovirt ~]#
So in my updated environment I think it tries to use the default file values in /etc/ovirt-provider-ovn/ovirt-provider-ovn.conf [OVIRT] ovirt-host=https://engine-host
The question now is why it has been wiped out? This environment has external VM working as ovirt engine, so not SHE.
I see that engine-backup doesn't run backup of the /etc/ovirt-provider-ovn/ ... could it be useful to include it?
Sounds like a valuable idea, would you create a bug?
Bug 1630824 - engine-backup should backup ovirt-provider-ovn https://bugzilla.redhat.com/1630824
I should have a VSPhere image backup of the VM so I should be able to restore the file. I'll see...
In the mean time could it be the update of the package has been the responsible? Eg, in my 4.2.7 env I currently have ovirt-provider-ovn-1.2.16-1.el7.noarch The update to 4.2.8 in the other system has put in ovirt-provider-ovn-1.2.18-1.el7.noarch
HIH debugging Gianluca
My update from 4.2.7 to 4.2.8 has been executed on 29th of January around 18:00 I have restored a VM image the day before and it seems that the /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf was not there even before the update....
The diff between the previous /etc/ovirt-provider-ovn/ovirt-provider-ovn.conf and the new one put in place by the update is
[root@ovmgr1 ovirt-provider-ovn]# diff old_ovirt-provider-ovn.conf ovirt-provider-ovn.conf 11a12
url_filter_exception=limit,page_reverse,next,previous [root@ovmgr1 ovirt-provider-ovn]#
At this point, as it has been some time I didn't power up VMs on OVN in this environment, I go back with past updates. On this environment the OVN part was installed manually when not yet included in standard setup. So I find this thread of mine about update from 4.1.9 to 4.2.3 on May 2018 I had a doubt about OVN and update and posted here: https://www.mail-archive.com/users@ovirt.org/msg48491.html
Thanks for figuring this out.
So based on Marcin feedback I selected NO for ovirt-ovn-provider question and perhaps the following updates caused file to be removed... perhaps...
This file is generated by engine-setup if Yes is selected for ovirt-ovn-provider, so I expect the file never existed.
How can I solve now?
Remove or rename the ovirt-provider-ovn provider in oVirt Engine, if it already exsits and run engine-setup with answer-file that has 'OVESETUP_OVN/ovirtProviderOvn' enabled: 'OVESETUP_OVN/ovirtProviderOvn=bool:True' should do the trick.
Didi, can you please ack that this will not destroy something except ovirt-provider-ovn related things?
Could it be that updating to 4.3 and selecting yes for ovn now, I get a new config for it? Or running engine-setup for 4.2.8? Thanks, Gianluca
participants (4)
-
Dan Kenigsberg -
Dominik Holler -
Gianluca Cecchi -
Yedidyah Bar David