5:46 a.m.
Hi,
After upgrading to 4.2.1 I have problems with ovn provider.
I'm getting "Failed to synchronize networks of Provider
ovirt-provider-ovn."
I use custom SSL certificate in apache and I guess this is the reason.
I've tried to update ovirt-provider-ovn.conf with
[OVIRT]
#ovirt-ca-file=/etc/pki/ovirt-engine/ca.pem
ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem
but still no go
Any tips on this?
thanks
G
8:21 a.m.
On Fri, 16 Mar 2018 12:46:13 +0200
Kapetanakis Giannis <bilias(a)edu.physics.uoc.gr> wrote:
Would you share the lines in engine.log produced by clicking the "Test"
button in the "Edit Provider" dialog?
On Clicking the test button, are you asked about "Import provider
certificate"?
10:40 a.m.
On 16/03/18 15:21, Dominik Holler wrote:
I get ok in test:
Test succeeded, managed to access provider.
2018-03-16 17:35:20,024+02 INFO
[org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] (default task-28)
[9920f622-b878-45e1-a421-e76c0ab23470] Running command: TestProviderConnectivityCommand
internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type:
SystemAction group CREATE_STORAGE_POOL with role type ADMIN
However a little bit later:
ovirt-provider-ovn.log:
2018-03-16 17:37:27,827 requests.packages.urllib3.connectionpool Starting new HTTPS
connection (1): engine-host
2018-03-16 17:37:27,827 requests.packages.urllib3.connectionpool Starting new HTTPS
connection (1): engine-host
2018-03-16 17:37:27,832 root [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
(_ssl.c:579)
Traceback (most recent call last):
File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 131, in
_handle_request
method, path_parts, content)
File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 175,
in handle_request
return self.call_response_handler(handler, content, parameters)
File "/usr/share/ovirt-provider-ovn/handlers/keystone.py", line 33, in
call_response_handler
return response_handler(content, parameters)
File "/usr/share/ovirt-provider-ovn/handlers/keystone_responses.py", line 62,
in post_tokens
user_password=user_password)
File "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 26, in
create_token
return auth.core.plugin.create_token(user_at_domain, user_password)
File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/plugin.py", line 48, in
create_token
timeout=self._timeout())
File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 75, in
create_token
username, password, engine_url, ca_file, timeout)
File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 91, in
_get_sso_token
timeout=timeout
File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 54, in
wrapper
response = func(*args, **kwargs)
File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 47, in
wrapper
raise BadGateway(e)
BadGateway: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
and in engine log:
2018-03-16 17:37:27,834+02 ERROR
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-27) [621c2b23] Command
'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' failed:
EngineException: (Failed with error PROVIDER_FAILURE and code 5050)
2018-03-16 17:37:27,850+02 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-27) [621c2b23] EVENT_ID:
PROVIDER_SYNCHRONIZED_FAILED(216), Failed to synchronize networks of Provider
ovirt-provider-ovn.
So the engine can talk with ovn but not the other way around as I understand.
I think it might have to do with [SSL] settings of ovirt-provider-ovn.conf
G
10:46 a.m.
On 16/03/18 17:40, Kapetanakis Giannis wrote:
SORRY wrong provider.
It asks for the cert.
Failed to communicate with the external provider, see log for additional details.
2018-03-16 17:44:08,262+02 INFO
[org.ovirt.engine.core.bll.provider.ImportProviderCertificateCommand] (default task-52)
[4731d25d-fce3-4408-99ea-8f9d1b5ee5b6] Running command: ImportProviderCertificateCommand
internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type:
SystemAction group CREATE_STORAGE_POOL with role type ADMIN
2018-03-16 17:44:08,275+02 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-52)
[4731d25d-fce3-4408-99ea-8f9d1b5ee5b6] EVENT_ID: PROVIDER_CERTIFICATE_IMPORTED(213),
Certificate for provider ovirt-provider-ovn was imported. (User: admin@internal)
2018-03-16 17:44:08,302+02 INFO
[org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] (default task-44)
[f4b2c57b-60c7-4ef9-a59f-0c5b22fa0356] Running command: TestProviderConnectivityCommand
internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type:
SystemAction group CREATE_STORAGE_POOL with role type ADMIN
2018-03-16 17:44:08,360+02 ERROR
[org.ovirt.engine.core.bll.provider.network.openstack.BaseNetworkProviderProxy] (default
task-44) [f4b2c57b-60c7-4ef9-a59f-0c5b22fa0356] Bad Gateway (OpenStack response error
code: 502)
2018-03-16 17:44:08,360+02 ERROR
[org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] (default task-44)
[f4b2c57b-60c7-4ef9-a59f-0c5b22fa0356] Command
'org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand' failed:
EngineException: (Failed with error PROVIDER_FAILURE and code 5050)
and in provider log:
2018-03-16 17:45:33,961 requests.packages.urllib3.connectionpool Starting new HTTPS
connection (1): engine-host
2018-03-16 17:45:33,961 requests.packages.urllib3.connectionpool Starting new HTTPS
connection (1): engine-host
2018-03-16 17:45:33,966 root [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
(_ssl.c:579)
Traceback (most recent call last):
File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 131, in
_handle_request
method, path_parts, content)
File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 175,
in handle_request
return self.call_response_handler(handler, content, parameters)
File "/usr/share/ovirt-provider-ovn/handlers/keystone.py", line 33, in
call_response_handler
return response_handler(content, parameters)
File "/usr/share/ovirt-provider-ovn/handlers/keystone_responses.py", line 62,
in post_tokens
user_password=user_password)
File "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 26, in
create_token
return auth.core.plugin.create_token(user_at_domain, user_password)
File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/plugin.py", line 48, in
create_token
timeout=self._timeout())
File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 75, in
create_token
username, password, engine_url, ca_file, timeout)
File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 91, in
_get_sso_token
timeout=timeout
File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 54, in
wrapper
response = func(*args, **kwargs)
File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 47, in
wrapper
raise BadGateway(e)
BadGateway: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
11:40 a.m.
On Fri, 16 Mar 2018 17:46:36 +0200
Kapetanakis Giannis <bilias(a)edu.physics.uoc.gr> wrote:
Thanks. Yes, the ovirt-provider-ovn refuses to connect to ovirt-engine
for authentication because ovirt-provider-ovn does not trust the
ssl-certificate and propagates this as the BadGateway error.
Please not that engine-setup creates the file
/etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf
which overwrites the default values from
/etc/ovirt-provider-ovn/ovirt-provider-ovn.conf
If you want to check if the referenced
/etc/pki/ovirt-engine/apache-ca.pem is correct, you can use the
following python snippet:
import requests
response = requests.get('https://ENGINE_FQDN/',
verify='/etc/pki/ovirt-engine/apache-ca.pem')
assert response.status_code == 200
Does this help to solve the issue?
9:42 a.m.
New subject: Failed to synchronize networks of Provider ovirt-provider-ovn
I have a same issue with OVN provider and SSL, but certificate changes not helps to
resolve it.
I use following
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/...
to replace my cert, and after reboot get this error.
ovirt-ca-file= is a same SSL file which use WebUI.
I restart ovirt-provider-ovn, i restart engine, i restart everything what i can restart.
Nothing helps...
Logs below.
[root@engine ~]# tail -n 50 /var/log/ovirt-provider-ovn.log
2018-09-12 14:10:23,828 root [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
(_ssl.c:579)
Traceback (most recent call last):
File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 133, in
_handle_request
method, path_parts, content
File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 175,
in handle_request
return self.call_response_handler(handler, content, parameters)
File "/usr/share/ovirt-provider-ovn/handlers/keystone.py", line 33, in
call_response_handler
return response_handler(content, parameters)
File "/usr/share/ovirt-provider-ovn/handlers/keystone_responses.py", line 62,
in post_tokens
user_password=user_password)
File "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 26, in
create_token
return auth.core.plugin.create_token(user_at_domain, user_password)
File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/plugin.py", line 48, in
create_token
timeout=self._timeout())
File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 75, in
create_token
username, password, engine_url, ca_file, timeout)
File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 91, in
_get_sso_token
timeout=timeout
File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 54, in
wrapper
response = func(*args, **kwargs)
File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 47, in
wrapper
raise BadGateway(e)
BadGateway: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
[root@engine ~]# tail -n 20 /var/log/ovirt-engine/engine.log
2018-09-12 14:10:23,773+03 INFO
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] Lock Acquired to object
'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]',
sharedLocks=''}'
2018-09-12 14:10:23,778+03 INFO
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] Running command:
SyncNetworkProviderCommand internal: true.
2018-09-12 14:10:23,836+03 ERROR
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] Command
'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' failed:
EngineException: (Failed with error Bad Gateway and code 5050)
2018-09-12 14:10:23,837+03 INFO
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] Lock freed to object
'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]',
sharedLocks=''}'
2018-09-12 14:14:12,477+03 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils]
(default task-6) [] User admin@internal successfully logged in with scopes:
ovirt-app-admin ovirt-app-api ovirt-app-portal ovirt-ext=auth:sequence-priority=~
ovirt-ext=revoke:revoke-all ovirt-ext=token-info:authz-search
ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate
ovirt-ext=token:password-access
2018-09-12 14:14:12,587+03 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand]
(default task-6) [1bf1b763] Running command: CreateUserSessionCommand internal: false.
2018-09-12 14:14:12,628+03 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-6)
[1bf1b763] EVENT_ID: USER_VDC_LOGIN(30), User admin@internal-authz connecting from
'10.0.3.61' using session
's8jAm7BUJGlicthm6yZBA3CUM8QpRdtwFaK3M/IppfhB3fHFB9gmNf0cAlbl1xIhcJ2WX+ww7e71Ri+MxJSsIg=='
logged in.
2018-09-12 14:14:30,972+03 INFO
[org.ovirt.engine.core.bll.provider.ImportProviderCertificateCommand] (default task-6)
[ee3cc8a7-4485-4fdf-a0c2-e9d67b5cfcd3] Running command: ImportProviderCertificateCommand
internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type:
SystemAction group CREATE_STORAGE_POOL with role type ADMIN
2018-09-12 14:14:30,982+03 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-6)
[ee3cc8a7-4485-4fdf-a0c2-e9d67b5cfcd3] EVENT_ID: PROVIDER_CERTIFICATE_IMPORTED(213),
Certificate for provider ovirt-provider-ovn was imported. (User: admin@internal-authz)
2018-09-12 14:14:31,006+03 INFO
[org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] (default task-6)
[a48d94ab-b0b2-42a2-a667-0525b4c652ea] Running command: TestProviderConnectivityCommand
internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type:
SystemAction group CREATE_STORAGE_POOL with role type ADMIN
2018-09-12 14:14:31,058+03 ERROR
[org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] (default task-6)
[a48d94ab-b0b2-42a2-a667-0525b4c652ea] Command
'org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand' failed:
EngineException: (Failed with error Bad Gateway and code 5050)
2018-09-12 14:15:10,954+03 INFO
[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'default'
is using 0 threads out of 1, 5 threads waiting for tasks.
2018-09-12 14:15:10,954+03 INFO
[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engine'
is using 0 threads out of 500, 16 threads waiting for tasks and 0 tasks in queue.
2018-09-12 14:15:10,954+03 INFO
[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool
'engineScheduled' is using 0 threads out of 100, 100 threads waiting for tasks.
2018-09-12 14:15:10,954+03 INFO
[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool
'engineThreadMonitoring' is using 1 threads out of 1, 0 threads waiting for
tasks.
2018-09-12 14:15:10,954+03 INFO
[org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
(EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool
'hostUpdatesChecker' is using 0 threads out of 5, 2 threads waiting for tasks.
2018-09-12 14:15:23,843+03 INFO
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] Lock Acquired to object
'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]',
sharedLocks=''}'
2018-09-12 14:15:23,849+03 INFO
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] Running command:
SyncNetworkProviderCommand internal: true.
2018-09-12 14:15:23,900+03 ERROR
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] Command
'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' failed:
EngineException: (Failed with error Bad Gateway and code 5050)
2018-09-12 14:15:23,901+03 INFO
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] Lock freed to object
'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]',
sharedLocks=''}'
[root@engine ~]# cat /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf
# This file is automatically generated by engine-setup. Please do not edit manually
[OVN REMOTE]
ovn-remote=ssl:127.0.0.1:6641
[SSL]
https-enabled=true
ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem
ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer
ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass
[OVIRT]
ovirt-sso-client-secret=Ms7Gw9qNT6IkXu7oA54tDmxaZDIukABV
ovirt-host=https://engine.set.local:443
ovirt-sso-client-id=ovirt-provider-ovn
ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem
[PROVIDER]
provider-host=engine.set.local
10:46 a.m.
New subject: Failed to synchronize networks of Provider ovirt-provider-ovn
On Wed, 12 Sep 2018 14:42:15 -0000
mail(a)set-pro.net wrote:
The config looks good.
You can check if the webserver is using the cert the ovirt-provider-ovn
is expecting by comparing the output of
openssl s_client -connect engine.set.local:443 -servername ssotest \
-showcerts | openssl x509 -text -noout
and
cat /etc/pki/ovirt-engine/apache-ca.pem | openssl x509 -text -noout
From a technical point of view, the provider uses the requests library,
so that you can easily check if the provider would like a cert on
command line by:
python -c "import requests; \
print requests.get('https://engine.set.local', \
verify='/etc/pki/ovirt-engine/apache-ca.pem')"
10:58 a.m.
New subject: Failed to synchronize networks of Provider ovirt-provider-ovn
I found what happens with OVN.
The wrong contents of a /etc/pki/ovirt-engine/apache-ca.pem I update it with my root-CA
cert file and get a success test:
"Test succeeded, managed to access provider."
Thank's to all reply!
4:08 a.m.
New subject: Failed to synchronize networks of Provider ovirt-provider-ovn
Hello,
I have a simmilar issue with ovirt-provider-ovn.
But in my config I see:
ovirt-sso-client-secret=to_be_set
Where do I find / how do I generate this token?
Thanks,
Robert O'Kane
On 09/12/2018 04:42 PM, mail(a)set-pro.net wrote:
--
Robert O'Kane
Systems Administrator
Kunsthochschule für Medien Köln
Peter-Welter-Platz 2
50676 Köln
fon: +49(221)20189-223
fax: +49(221)20189-49223
5:25 a.m.
New subject: Failed to synchronize networks of Provider ovirt-provider-ovn
On Thu, 13 Sep 2018 11:08:28 +0200
Robert O'Kane <okane(a)khm.de> wrote:
Usually engine-setup will generate an appropriate automatically.
/etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf.
If you want to (or have to?) generate manually the client secrete,
follow this steps:
1. Run /usr/share/ovirt-engine/bin/ovirt-register-sso-client-tool.sh
with
Client Id: ovirt-provider-ovn
Client CA Certificate File Location: /etc/pki/ovirt-engine/certs/engine.cer
Callback Prefix URL: https://<ENGINE_FQDN>:443/ovirt-engine/
2. Use the SSO_CLIENT_SECRET from the outfile produced by the previous
command in
/etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf
3. Restart ovirt-engine and ovirt-provider-ovn
systemctl restart ovirt-engine
systemctl restart ovirt-provider-ovn
2391
days inactive
2573
days old
11 comments
4 participants
participants (4)
-
Dominik Holler -
Kapetanakis Giannis -
mail＠set-pro.net -
Robert O'Kane