[ovirt-users] Re: Failed to synchronize networks of Provider ovirt-provider-ovn

I have a same issue with OVN provider and SSL, but certificate changes not helps to resolve it. I use following https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/... to replace my cert, and after reboot get this error. ovirt-ca-file= is a same SSL file which use WebUI. I restart ovirt-provider-ovn, i restart engine, i restart everything what i can restart. Nothing helps... Logs below. [root@engine ~]# tail -n 50 /var/log/ovirt-provider-ovn.log 2018-09-12 14:10:23,828 root [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) Traceback (most recent call last): File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 133, in _handle_request method, path_parts, content File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 175, in handle_request return self.call_response_handler(handler, content, parameters) File "/usr/share/ovirt-provider-ovn/handlers/keystone.py", line 33, in call_response_handler return response_handler(content, parameters) File "/usr/share/ovirt-provider-ovn/handlers/keystone_responses.py", line 62, in post_tokens user_password=user_password) File "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 26, in create_token return auth.core.plugin.create_token(user_at_domain, user_password) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/plugin.py", line 48, in create_token timeout=self._timeout()) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 75, in create_token username, password, engine_url, ca_file, timeout) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 91, in _get_sso_token timeout=timeout File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 54, in wrapper response = func(*args, **kwargs) File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 47, in wrapper raise BadGateway(e) BadGateway: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) [root@engine ~]# tail -n 20 /var/log/ovirt-engine/engine.log 2018-09-12 14:10:23,773+03 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] Lock Acquired to object 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', sharedLocks=''}' 2018-09-12 14:10:23,778+03 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] Running command: SyncNetworkProviderCommand internal: true. 2018-09-12 14:10:23,836+03 ERROR [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] Command 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' failed: EngineException: (Failed with error Bad Gateway and code 5050) 2018-09-12 14:10:23,837+03 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] Lock freed to object 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', sharedLocks=''}' 2018-09-12 14:14:12,477+03 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-6) [] User admin@internal successfully logged in with scopes: ovirt-app-admin ovirt-app-api ovirt-app-portal ovirt-ext=auth:sequence-priority=~ ovirt-ext=revoke:revoke-all ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2018-09-12 14:14:12,587+03 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-6) [1bf1b763] Running command: CreateUserSessionCommand internal: false. 2018-09-12 14:14:12,628+03 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-6) [1bf1b763] EVENT_ID: USER_VDC_LOGIN(30), User admin@internal-authz connecting from '10.0.3.61' using session 's8jAm7BUJGlicthm6yZBA3CUM8QpRdtwFaK3M/IppfhB3fHFB9gmNf0cAlbl1xIhcJ2WX+ww7e71Ri+MxJSsIg==' logged in. 2018-09-12 14:14:30,972+03 INFO [org.ovirt.engine.core.bll.provider.ImportProviderCertificateCommand] (default task-6) [ee3cc8a7-4485-4fdf-a0c2-e9d67b5cfcd3] Running command: ImportProviderCertificateCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_POOL with role type ADMIN 2018-09-12 14:14:30,982+03 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-6) [ee3cc8a7-4485-4fdf-a0c2-e9d67b5cfcd3] EVENT_ID: PROVIDER_CERTIFICATE_IMPORTED(213), Certificate for provider ovirt-provider-ovn was imported. (User: admin@internal-authz) 2018-09-12 14:14:31,006+03 INFO [org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] (default task-6) [a48d94ab-b0b2-42a2-a667-0525b4c652ea] Running command: TestProviderConnectivityCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_POOL with role type ADMIN 2018-09-12 14:14:31,058+03 ERROR [org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] (default task-6) [a48d94ab-b0b2-42a2-a667-0525b4c652ea] Command 'org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand' failed: EngineException: (Failed with error Bad Gateway and code 5050) 2018-09-12 14:15:10,954+03 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'default' is using 0 threads out of 1, 5 threads waiting for tasks. 2018-09-12 14:15:10,954+03 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engine' is using 0 threads out of 500, 16 threads waiting for tasks and 0 tasks in queue. 2018-09-12 14:15:10,954+03 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engineScheduled' is using 0 threads out of 100, 100 threads waiting for tasks. 2018-09-12 14:15:10,954+03 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engineThreadMonitoring' is using 1 threads out of 1, 0 threads waiting for tasks. 2018-09-12 14:15:10,954+03 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'hostUpdatesChecker' is using 0 threads out of 5, 2 threads waiting for tasks. 2018-09-12 14:15:23,843+03 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', sharedLocks=''}' 2018-09-12 14:15:23,849+03 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] Running command: SyncNetworkProviderCommand internal: true. 2018-09-12 14:15:23,900+03 ERROR [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] Command 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' failed: EngineException: (Failed with error Bad Gateway and code 5050) 2018-09-12 14:15:23,901+03 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] Lock freed to object 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', sharedLocks=''}' [root@engine ~]# cat /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf # This file is automatically generated by engine-setup. Please do not edit manually [OVN REMOTE] ovn-remote=ssl:127.0.0.1:6641 [SSL] https-enabled=true ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass [OVIRT] ovirt-sso-client-secret=Ms7Gw9qNT6IkXu7oA54tDmxaZDIukABV ovirt-host=https://engine.set.local:443 ovirt-sso-client-id=ovirt-provider-ovn ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem [PROVIDER] provider-host=engine.set.local