Hi
I did a fresh installation of version 4.4.0.3. After the engine setup I replaced the apache certificate with a custom certificate. I used this article to do it: https://myhomelab.gr/linux/2020/01/20/replacing_ovirt_ssl.html
To summarize, I replaced those files with my own authority and the signed custom certificate
/etc/pki/ovirt-engine/keys/apache.key.nopass
/etc/pki/ovirt-engine/certs/apache.cer
/etc/pki/ovirt-engine/apache-ca.pem
That worked so far, apache uses now my certificate, login is possible. To setup a new machine, I need to upload an iso image, which failed. I found this error in /var/log/ovirt-imageio/daemon.log
2020-07-08 20:43:23,750 INFO (Thread-10) [http] OPEN client=192.168.1.228
2020-07-08 20:43:23,767 INFO (Thread-10) [backends.http] Open backend netloc='the_secret_hostname:54322' path='/images/ef60404c-dc69-4a3d-bfaa-8571f675f3e1' cafile='/etc/pki/ovirt-engine/apache-ca.pem' secure=True
2020-07-08 20:43:23,770 ERROR (Thread-10) [http] Server error
Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 699, in __call__
self.dispatch(req, resp)
File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 744, in dispatch
return method(req, resp, *match.groups())
File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/cors.py", line 84, in wrapper
return func(self, req, resp, *args)
File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/images.py", line 66, in put
backends.get(req, ticket, self.config),
File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py", line 53, in get
cafile=config.tls.ca_file)
File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 48, in open
secure=options.get("secure", True))
File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 63, in __init__
options = self._options()
File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 364, in _options
self._con.request("OPTIONS", self.url.path)
File "/usr/lib64/python3.6/http/client.py", line 1254, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output
self.send(msg)
File "/usr/lib64/python3.6/http/client.py", line 974, in send
self.connect()
File "/usr/lib64/python3.6/http/client.py", line 1422, in connect
server_hostname=server_hostname)
File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket
_context=self, _session=session)
File "/usr/lib64/python3.6/ssl.py", line 776, in __init__
self.do_handshake()
File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
2020-07-08 20:43:23,770 INFO (Thread-10) [http] CLOSE client=192.168.1.228 [connection 1 ops, 0.019775 s] [dispatch 1 ops, 0.003114 s]
I'm a python developer so I had no problem reading the traceback.
The SSL handshake fails when image-io tries to connect to what I think is called an ovn-provider. But it is using my new authority certificate cafile='/etc/pki/ovirt-engine/apache-ca.pem' which does not validate the certificate generated by the ovirt engine setup, which the ovn-provider probably uses.
I didn't exactly know where the parameter for the validation ca file is. Probably it is the ca_file parameter in /etc/ovirt-imageio/conf.d/50-engine.conf. But that needs to be set to my own authority ca file.
I modified the python file to set the ca_file parameter to the engine setups ca_file directly
/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py
So the function call around line 50 looks like this:
backend = module.open(
ticket.url,
mode,
sparse=ticket.sparse,
dirty=ticket.dirty,
cafile='/etc/pki/ovirt-engine/ca.pem' #config.tls.ca_file
)
Reading this again, the problem is clear now.
The imageio proxy is trying to use your CA to verify the the host imageio daemon certificate. This cannot work because the host certificate is signed by engine CA, and the imageio daemon on the host is using vdsm certificates
With the current version you will have to create certificates for each host imageio daemon and configure it, which is probably not practical.
So looks like we need to add additional ca_file configuration, which must be used when connecting to host using the http backend. Using the ca_file used for the http server is wrong, it works only for the default configuration.
Please file imageio bug for this.
You can keep your local change until we fix this.
Now the image upload works, but obviously this is not the way to fix things. Is there an other way to make image-io accept the certificate from the engine setup, while using my custom certificate? I don't want to replace the certificates of all ovirt components with custom certificates.
This is also not supported.