On Thu, Jul 23, 2020 at 7:47 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Thu, Jul 23, 2020 at 5:14 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Sun, Jul 19, 2020 at 5:23 PM <ramon@clematide.ch> wrote:
Hi
I did a fresh installation of version 4.4.0.3. After the engine setup I replaced the apache certificate with a custom certificate. I used this article to do it: https://myhomelab.gr/linux/2020/01/20/replacing_ovirt_ssl.html
To summarize, I replaced those files with my own authority and the signed custom certificate
/etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/apache-ca.pem
That worked so far, apache uses now my certificate, login is possible. To setup a new machine, I need to upload an iso image, which failed. I found this error in /var/log/ovirt-imageio/daemon.log
2020-07-08 20:43:23,750 INFO (Thread-10) [http] OPEN client=192.168.1.228 2020-07-08 20:43:23,767 INFO (Thread-10) [backends.http] Open backend netloc='the_secret_hostname:54322' path='/images/ef60404c-dc69-4a3d-bfaa-8571f675f3e1' cafile='/etc/pki/ovirt-engine/apache-ca.pem' secure=True 2020-07-08 20:43:23,770 ERROR (Thread-10) [http] Server error Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 699, in __call__ self.dispatch(req, resp) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 744, in dispatch return method(req, resp, *match.groups()) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/cors.py", line 84, in wrapper return func(self, req, resp, *args) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/images.py", line 66, in put backends.get(req, ticket, self.config), File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py", line 53, in get cafile=config.tls.ca_file) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 48, in open secure=options.get("secure", True)) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 63, in __init__ options = self._options() File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 364, in _options self._con.request("OPTIONS", self.url.path) File "/usr/lib64/python3.6/http/client.py", line 1254, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output self.send(msg) File "/usr/lib64/python3.6/http/client.py", line 974, in send self.connect() File "/usr/lib64/python3.6/http/client.py", line 1422, in connect server_hostname=server_hostname) File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket _context=self, _session=session) File "/usr/lib64/python3.6/ssl.py", line 776, in __init__ self.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake self._sslobj.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897) 2020-07-08 20:43:23,770 INFO (Thread-10) [http] CLOSE client=192.168.1.228 [connection 1 ops, 0.019775 s] [dispatch 1 ops, 0.003114 s]
I'm a python developer so I had no problem reading the traceback.
The SSL handshake fails when image-io tries to connect to what I think is called an ovn-provider. But it is using my new authority certificate cafile='/etc/pki/ovirt-engine/apache-ca.pem' which does not validate the certificate generated by the ovirt engine setup, which the ovn-provider probably uses.
I didn't exactly know where the parameter for the validation ca file is. Probably it is the ca_file parameter in /etc/ovirt-imageio/conf.d/50-engine.conf. But that needs to be set to my own authority ca file.
I modified the python file to set the ca_file parameter to the engine setups ca_file directly
/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py
So the function call around line 50 looks like this:
backend = module.open( ticket.url, mode, sparse=ticket.sparse, dirty=ticket.dirty, cafile='/etc/pki/ovirt-engine/ca.pem' #config.tls.ca_file )
Now the image upload works, but obviously this is not the way to fix things. Is there an other way to make image-io accept the certificate from the engine setup, while using my custom certificate? I don't want to replace the certificates of all ovirt components with custom certificates. I only need the weblogin with my custom certificate.
Adding Nir.
It's been quite some time since I checked imageio and using 3rd-party CAs, not sure about current status.
Last time I tried this (before the work done on imageio for 4.4), it was enough to make imageio use apache keypair and restart it, see also this bug and its dependencies:
https://bugzilla.redhat.com/show_bug.cgi?id=1385617
Nir - did you try this recently? If it's indeed broken, do we need a doc change, or imageio, or perhaps both?
I never tried to change the certificates on engine host since I don't have any setup with non-default certificates, but we do have automated tests using various configurations and certificates generated for imageio tests.
See here how we generate the certificates: https://github.com/oVirt/ovirt-imageio/blob/master/daemon/test/genpki
imageio automated tests run a server using these certificates, using a client configured to verify server certificates: https://github.com/oVirt/ovirt-imageio/blob/acf69ee00411e3b1cdd1c13c584759ec...
If you think this does not cover all cases, please suggest how to test it better.
I admit I do not know imageio well enough, sorry. I assume that you do not run an engine in your tests. Are you sure you can still cover all cases that involve imageio? Does the flow "upload an image from web admin ui" require an engine? Not only in the interaction between browser code and imageio, but also between engine/httpd and imageio (if any). I guess it would be best to add to OST code to change the certs to 3rd-party ones, and then try imageio upload via a browser. Best regards, -- Didi