not able to upload disks, iso - paused by the system error -- Version 4.4.6.7-1.el8
Hi Team in one of the cluster infra, we are unable to upload the images or disks via gui. up on checking the /var/log/ovirt-imageio/daemon.log found that throwing ssl connection failure, help us to check what are we missing.. We are using thirdparty CA approved SSL for web GUI.. 2021-10-11 22:45:42,812 INFO (Thread-6) [http] OPEN connection=6 client=127.0.0.1 2021-10-11 22:45:42,812 INFO (Thread-6) [tickets] [127.0.0.1] REMOVE ticket=f18cff91-1fc4-43b6-91ea-ca2a11d409a6 2021-10-11 22:45:42,813 INFO (Thread-6) [http] CLOSE connection=6 client=127.0.0.1 [connection 1 ops, 0.000539 s] [dispatch 1 ops, 0.000216 s] 2021-10-11 22:45:43,621 INFO (Thread-4) [images] [::ffff:10.12.23.212] OPTIONS ticket=53ff98f9-f429-4880-abe6-06c6c01473de 2021-10-11 22:45:43,621 INFO (Thread-4) [backends.http] Open backend netloc='renlovkvma01.test.lab:54322' path='/images/53ff98f9-f429-4880-abe6-06c6c01473de' cafile='/etc/pki/ovirt-engine/ca.pem' secure=True 2021-10-11 22:45:43,626 ERROR (Thread-4) [http] Server error Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py", line 66, in get return ticket.get_context(req.connection_id) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/auth.py", line 146, in get_context return self._connections[con_id] KeyError: 4 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 774, in __call__ self.dispatch(req, resp) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 819, in dispatch return method(req, resp, *match.groups()) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/cors.py", line 84, in wrapper return func(self, req, resp, *args) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/images.py", line 246, in options ctx = backends.get(req, ticket, self.config) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py", line 85, in get cafile=ca_file) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 48, in open return Backend(url, **options) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 76, in __init__ self._connect() File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 117, in _connect self._con = self._create_tcp_connection() File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 379, in _create_tcp_connection con.connect() File "/usr/lib64/python3.6/http/client.py", line 1437, in connect server_hostname=server_hostname) File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket _context=self, _session=session) File "/usr/lib64/python3.6/ssl.py", line 776, in __init__ self.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake self._sslobj.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
On Tuesday, 12 October 2021 07:53:46 CEST dhanaraj.ramesh--- via Users wrote:
Hi Team
in one of the cluster infra, we are unable to upload the images or disks via gui. up on checking the /var/log/ovirt-imageio/daemon.log found that throwing ssl connection failure, help us to check what are we missing..
looks like certificate is not trusted (invalid/missing root certificates on given machine?)
We are using thirdparty CA approved SSL for web GUI..
did you try "Test connection" button on the upload page? If it fails, you need to download the certificate from [1] and make it trusted in your browser [1] https://$ENGINE_ADDRESS/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA
2021-10-11 22:45:42,812 INFO (Thread-6) [http] OPEN connection=6 client=127.0.0.1 2021-10-11 22:45:42,812 INFO (Thread-6) [tickets] [127.0.0.1] REMOVE ticket=f18cff91-1fc4-43b6-91ea-ca2a11d409a6 2021-10-11 22:45:42,813 INFO (Thread-6) [http] CLOSE connection=6 client=127.0.0.1 [connection 1 ops, 0.000539 s] [dispatch 1 ops, 0.000216 s] 2021-10-11 22:45:43,621 INFO (Thread-4) [images] [::ffff:10.12.23.212] OPTIONS ticket=53ff98f9-f429-4880-abe6-06c6c01473de 2021-10-11 22:45:43,621 INFO (Thread-4) [backends.http] Open backend netloc='renlovkvma01.test.lab:54322' path='/images/53ff98f9-f429-4880-abe6-06c6c01473de' cafile='/etc/pki/ovirt-engine/ca.pem' secure=True 2021-10-11 22:45:43,626 ERROR (Thread-4) [http] Server error Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init __.py", line 66, in get return ticket.get_context(req.connection_id) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/auth.py", line 146, in get_context return self._connections[con_id] KeyError: 4
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 774, in __call__ self.dispatch(req, resp) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 819, in dispatch return method(req, resp, *match.groups()) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/cors.py", line 84, in wrapper return func(self, req, resp, *args) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/images.py", line 246, in options ctx = backends.get(req, ticket, self.config) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init __.py", line 85, in get cafile=ca_file) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.p y", line 48, in open return Backend(url, **options) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.p y", line 76, in __init__ self._connect() File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.p y", line 117, in _connect self._con = self._create_tcp_connection() File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.p y", line 379, in _create_tcp_connection con.connect() File "/usr/lib64/python3.6/http/client.py", line 1437, in connect server_hostname=server_hostname) File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket _context=self, _session=session) File "/usr/lib64/python3.6/ssl.py", line 776, in __init__ self.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake self._sslobj.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/XMBNYE7E7C5XY UKO6FARQQAHCT4HOYSJ/
On Tue, Oct 12, 2021 at 8:55 AM dhanaraj.ramesh--- via Users <users@ovirt.org> wrote:
Hi Team
in one of the cluster infra, we are unable to upload the images or disks via gui. up on checking the /var/log/ovirt-imageio/daemon.log found that throwing ssl connection failure, help us to check what are we missing..
Which version? If you are on ovirt 4.4, please share output of: ovirt-imageio --show-config on engine.
We are using thirdparty CA approved SSL for web GUI..
2021-10-11 22:45:42,812 INFO (Thread-6) [http] OPEN connection=6 client=127.0.0.1 2021-10-11 22:45:42,812 INFO (Thread-6) [tickets] [127.0.0.1] REMOVE ticket=f18cff91-1fc4-43b6-91ea-ca2a11d409a6 2021-10-11 22:45:42,813 INFO (Thread-6) [http] CLOSE connection=6 client=127.0.0.1 [connection 1 ops, 0.000539 s] [dispatch 1 ops, 0.000216 s] 2021-10-11 22:45:43,621 INFO (Thread-4) [images] [::ffff:10.12.23.212] OPTIONS ticket=53ff98f9-f429-4880-abe6-06c6c01473de 2021-10-11 22:45:43,621 INFO (Thread-4) [backends.http] Open backend netloc='renlovkvma01.test.lab:54322' path='/images/53ff98f9-f429-4880-abe6-06c6c01473de' cafile='/etc/pki/ovirt-engine/ca.pem' secure=True
Looks like the host is configured correctly - the http backend is using the right CA file to access the host.
2021-10-11 22:45:43,626 ERROR (Thread-4) [http] Server error ... self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
The CA file on engine side (/etc/pki/ovirt-engine/ca.pem) does not match the CA file on the host (/etc/pki/vdsm/certs/cacert.pem). Which files did you change when we added the thirdparty CA approved SSL? Nir
Hi Nir, sorry for reply late, here we go ovirt-imageio --show-config { "backend_file": { "buffer_size": 8388608 }, "backend_http": { "buffer_size": 8388608, "ca_file": "/etc/pki/ovirt-engine/ca.pem" }, "backend_nbd": { "buffer_size": 8388608 }, "control": { "port": 54324, "prefer_ipv4": true, "remove_timeout": 60, "socket": "/run/ovirt-imageio/sock", "transport": "tcp" }, "daemon": { "drop_privileges": true, "group_name": "ovirtimg", "max_connections": 8, "poll_interval": 1.0, "run_dir": "/run/ovirt-imageio", "user_name": "ovirtimg" }, "formatter_long": { "format": "%(asctime)s %(levelname)-7s (%(threadName)s) [%(name)s] %(message)s" }, "formatters": { "keys": "long" }, "handler_logfile": { "args": "(\"/var/log/ovirt-imageio/daemon.log\",)", "formatter": "long", "class": "logging.handlers.RotatingFileHandler", "kwargs": "{\"maxBytes\": 20971520, \"backupCount\": 10}", "level": "DEBUG" }, "handler_stderr": { "formatter": "long", "class": "logging.StreamHandler", "level": "DEBUG" }, "handlers": { "keys": "logfile" }, "local": { "enable": false, "socket": "\u0000/org/ovirt/imageio" }, "logger_root": { "handlers": "logfile", "level": "INFO", "propagate": 0 }, "loggers": { "keys": "root" }, "profile": { "filename": "/run/ovirt-imageio/profile" }, "remote": { "host": "::", "port": 54323 }, "tls": { "ca_file": "", "cert_file": "/etc/pki/ovirt-engine/certs/apache.cer", "enable": true, "enable_tls1_1": false, "key_file": "/etc/pki/ovirt-engine/keys/apache.key.nopass" } } Yes, The CA file on engine side (/etc/pki/ovirt-engine/ca.pem) does not matching with the CA file on the host (/etc/pki/vdsm/certs/cacert.pem) because we made the changes only on the engine side ca.pem with third party cert. we followed this doc from Replacing the oVirt Engine Apache CA Certificate, It is working for rest all cluster https://www.ovirt.org/documentation/administration_guide/index.html#appe-Red... Replacing the oVirt Engine Apache CA Certificate If you are using a self-hosted engine, put the environment into global maintenance mode. # hosted-engine --set-maintenance --mode=global For more information, see Maintaining the Self-hosted engine. Add your CA certificate to the host-wide trust store: # cp /tmp/3rd-party-ca-cert.pem /etc/pki/ca-trust/source/anchors # update-ca-trust The Engine has been configured to use /etc/pki/ovirt-engine/apache-ca.pem, which is symbolically linked to /etc/pki/ovirt-engine/ca.pem. Remove the symbolic link: # rm /etc/pki/ovirt-engine/apache-ca.pem Save your CA certificate as /etc/pki/ovirt-engine/apache-ca.pem: # cp /tmp/3rd-party-ca-cert.pem /etc/pki/ovirt-engine/apache-ca.pem Back up the existing private key and certificate: # cp /etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/keys/apache.key.nopass.bck # cp /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/certs/apache.cer.bck Copy the private key to the required location: # cp /tmp/apache.key /etc/pki/ovirt-engine/keys/apache.key.nopass Set the private key owner to root and set the permissions to 0640: # chown root:ovirt /etc/pki/ovirt-engine/keys/apache.key.nopass # chmod 640 /etc/pki/ovirt-engine/keys/apache.key.nopass Copy the certificate to the required location: # cp /tmp/apache.cer /etc/pki/ovirt-engine/certs/apache.cer Set the certificate owner to root and set the permissions to 0644: # chown root:ovirt /etc/pki/ovirt-engine/certs/apache.cer # chmod 644 /etc/pki/ovirt-engine/certs/apache.cer Restart the Apache server: # systemctl restart httpd.service Create a new trust store configuration file, /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf, with the following parameters: ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts" ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD="" Copy the /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf file, and rename it with an index number that is greater than 10 (for example, 99-setup.conf). Add the following parameters to the new file: SSL_CERTIFICATE=/etc/pki/ovirt-engine/certs/apache.cer SSL_KEY=/etc/pki/ovirt-engine/keys/apache.key.nopass Restart the websocket-proxy service: # systemctl restart ovirt-websocket-proxy.service If you manually changed the /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf file, or are using a configuration file from an older installation, make sure that the Engine is still configured to use /etc/pki/ovirt-engine/apache-ca.pem as the certificate source. Enable engine-backup to update the system on restore by creating a new file, /etc/ovirt-engine-backup/engine-backup-config.d/update-system-wide-pki.sh, with the following content: BACKUP_PATHS="${BACKUP_PATHS} /etc/ovirt-engine-backup" cp -f /etc/pki/ovirt-engine/apache-ca.pem \ /etc/pki/ca-trust/source/anchors/3rd-party-ca-cert.pem update-ca-trust Restart the ovirt-provider-ovn service: # systemctl restart ovirt-provider-ovn.service Restart the ovirt-imageio service: # systemctl restart ovirt-imageio.service Restart the ovirt-engine service: # systemctl restart ovirt-engine.service If you are using a self-hosted engine, turn off global maintenance mode. # hosted-engine --set-maintenance --mode=none Your users can now connect to the Administration Portal and VM Portal, without seeing a warning about the authenticity of the certificate used to encrypt HTTPS traffic.
participants (3)
-
dhanaraj.ramesh＠yahoo.com -
Nir Soffer -
Vojtech Juranek