Hi I did a fresh installation of version 4.4.0.3. After the engine setup I replaced the apache certificate with a custom certificate. I used this article to do it: https://myhomelab.gr/linux/2020/01/20/replacing_ovirt_ssl.html To summarize, I replaced those files with my own authority and the signed custom certificate /etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/apache-ca.pem That worked so far, apache uses now my certificate, login is possible. To setup a new machine, I need to upload an iso image, which failed. I found this error in /var/log/ovirt-imageio/daemon.log 2020-07-08 20:43:23,750 INFO (Thread-10) [http] OPEN client=192.168.1.228 2020-07-08 20:43:23,767 INFO (Thread-10) [backends.http] Open backend netloc='the_secret_hostname:54322' path='/images/ef60404c-dc69-4a3d-bfaa-8571f675f3e1' cafile='/etc/pki/ovirt-engine/apache-ca.pem' secure=True 2020-07-08 20:43:23,770 ERROR (Thread-10) [http] Server error Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 699, in __call__ self.dispatch(req, resp) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 744, in dispatch return method(req, resp, *match.groups()) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/cors.py", line 84, in wrapper return func(self, req, resp, *args) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/images.py", line 66, in put backends.get(req, ticket, self.config), File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py", line 53, in get cafile=config.tls.ca_file) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 48, in open secure=options.get("secure", True)) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 63, in __init__ options = self._options() File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 364, in _options self._con.request("OPTIONS", self.url.path) File "/usr/lib64/python3.6/http/client.py", line 1254, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output self.send(msg) File "/usr/lib64/python3.6/http/client.py", line 974, in send self.connect() File "/usr/lib64/python3.6/http/client.py", line 1422, in connect server_hostname=server_hostname) File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket _context=self, _session=session) File "/usr/lib64/python3.6/ssl.py", line 776, in __init__ self.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake self._sslobj.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897) 2020-07-08 20:43:23,770 INFO (Thread-10) [http] CLOSE client=192.168.1.228 [connection 1 ops, 0.019775 s] [dispatch 1 ops, 0.003114 s] I'm a python developer so I had no problem reading the traceback. The SSL handshake fails when image-io tries to connect to what I think is called an ovn-provider. But it is using my new authority certificate cafile='/etc/pki/ovirt-engine/apache-ca.pem' which does not validate the certificate generated by the ovirt engine setup, which the ovn-provider probably uses. I didn't exactly know where the parameter for the validation ca file is. Probably it is the ca_file parameter in /etc/ovirt-imageio/conf.d/50-engine.conf. But that needs to be set to my own authority ca file. I modified the python file to set the ca_file parameter to the engine setups ca_file directly /usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py So the function call around line 50 looks like this: backend = module.open( ticket.url, mode, sparse=ticket.sparse, dirty=ticket.dirty, cafile='/etc/pki/ovirt-engine/ca.pem' #config.tls.ca_file ) Now the image upload works, but obviously this is not the way to fix things. Is there an other way to make image-io accept the certificate from the engine setup, while using my custom certificate? I don't want to replace the certificates of all ovirt components with custom certificates. I only need the weblogin with my custom certificate. Regards
On Sun, Jul 19, 2020 at 5:23 PM <ramon@clematide.ch> wrote:
Hi
I did a fresh installation of version 4.4.0.3. After the engine setup I replaced the apache certificate with a custom certificate. I used this article to do it: https://myhomelab.gr/linux/2020/01/20/replacing_ovirt_ssl.html
To summarize, I replaced those files with my own authority and the signed custom certificate
/etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/apache-ca.pem
That worked so far, apache uses now my certificate, login is possible. To setup a new machine, I need to upload an iso image, which failed. I found this error in /var/log/ovirt-imageio/daemon.log
2020-07-08 20:43:23,750 INFO (Thread-10) [http] OPEN client=192.168.1.228 2020-07-08 20:43:23,767 INFO (Thread-10) [backends.http] Open backend netloc='the_secret_hostname:54322' path='/images/ef60404c-dc69-4a3d-bfaa-8571f675f3e1' cafile='/etc/pki/ovirt-engine/apache-ca.pem' secure=True 2020-07-08 20:43:23,770 ERROR (Thread-10) [http] Server error Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 699, in __call__ self.dispatch(req, resp) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 744, in dispatch return method(req, resp, *match.groups()) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/cors.py", line 84, in wrapper return func(self, req, resp, *args) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/images.py", line 66, in put backends.get(req, ticket, self.config), File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py", line 53, in get cafile=config.tls.ca_file) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 48, in open secure=options.get("secure", True)) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 63, in __init__ options = self._options() File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 364, in _options self._con.request("OPTIONS", self.url.path) File "/usr/lib64/python3.6/http/client.py", line 1254, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output self.send(msg) File "/usr/lib64/python3.6/http/client.py", line 974, in send self.connect() File "/usr/lib64/python3.6/http/client.py", line 1422, in connect server_hostname=server_hostname) File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket _context=self, _session=session) File "/usr/lib64/python3.6/ssl.py", line 776, in __init__ self.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake self._sslobj.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897) 2020-07-08 20:43:23,770 INFO (Thread-10) [http] CLOSE client=192.168.1.228 [connection 1 ops, 0.019775 s] [dispatch 1 ops, 0.003114 s]
I'm a python developer so I had no problem reading the traceback.
The SSL handshake fails when image-io tries to connect to what I think is called an ovn-provider. But it is using my new authority certificate cafile='/etc/pki/ovirt-engine/apache-ca.pem' which does not validate the certificate generated by the ovirt engine setup, which the ovn-provider probably uses.
I didn't exactly know where the parameter for the validation ca file is. Probably it is the ca_file parameter in /etc/ovirt-imageio/conf.d/50-engine.conf. But that needs to be set to my own authority ca file.
I modified the python file to set the ca_file parameter to the engine setups ca_file directly
/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py
So the function call around line 50 looks like this:
backend = module.open( ticket.url, mode, sparse=ticket.sparse, dirty=ticket.dirty, cafile='/etc/pki/ovirt-engine/ca.pem' #config.tls.ca_file )
Now the image upload works, but obviously this is not the way to fix things. Is there an other way to make image-io accept the certificate from the engine setup, while using my custom certificate? I don't want to replace the certificates of all ovirt components with custom certificates. I only need the weblogin with my custom certificate.
Adding Nir. It's been quite some time since I checked imageio and using 3rd-party CAs, not sure about current status. Last time I tried this (before the work done on imageio for 4.4), it was enough to make imageio use apache keypair and restart it, see also this bug and its dependencies: https://bugzilla.redhat.com/show_bug.cgi?id=1385617 Nir - did you try this recently? If it's indeed broken, do we need a doc change, or imageio, or perhaps both? Best regards, -- Didi
On Thu, Jul 23, 2020 at 5:14 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Sun, Jul 19, 2020 at 5:23 PM <ramon@clematide.ch> wrote:
Hi
I did a fresh installation of version 4.4.0.3. After the engine setup I replaced the apache certificate with a custom certificate. I used this article to do it: https://myhomelab.gr/linux/2020/01/20/replacing_ovirt_ssl.html
To summarize, I replaced those files with my own authority and the signed custom certificate
/etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/apache-ca.pem
That worked so far, apache uses now my certificate, login is possible. To setup a new machine, I need to upload an iso image, which failed. I found this error in /var/log/ovirt-imageio/daemon.log
2020-07-08 20:43:23,750 INFO (Thread-10) [http] OPEN client=192.168.1.228 2020-07-08 20:43:23,767 INFO (Thread-10) [backends.http] Open backend netloc='the_secret_hostname:54322' path='/images/ef60404c-dc69-4a3d-bfaa-8571f675f3e1' cafile='/etc/pki/ovirt-engine/apache-ca.pem' secure=True 2020-07-08 20:43:23,770 ERROR (Thread-10) [http] Server error Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 699, in __call__ self.dispatch(req, resp) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 744, in dispatch return method(req, resp, *match.groups()) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/cors.py", line 84, in wrapper return func(self, req, resp, *args) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/images.py", line 66, in put backends.get(req, ticket, self.config), File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py", line 53, in get cafile=config.tls.ca_file) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 48, in open secure=options.get("secure", True)) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 63, in __init__ options = self._options() File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 364, in _options self._con.request("OPTIONS", self.url.path) File "/usr/lib64/python3.6/http/client.py", line 1254, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output self.send(msg) File "/usr/lib64/python3.6/http/client.py", line 974, in send self.connect() File "/usr/lib64/python3.6/http/client.py", line 1422, in connect server_hostname=server_hostname) File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket _context=self, _session=session) File "/usr/lib64/python3.6/ssl.py", line 776, in __init__ self.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake self._sslobj.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897) 2020-07-08 20:43:23,770 INFO (Thread-10) [http] CLOSE client=192.168.1.228 [connection 1 ops, 0.019775 s] [dispatch 1 ops, 0.003114 s]
I'm a python developer so I had no problem reading the traceback.
The SSL handshake fails when image-io tries to connect to what I think is called an ovn-provider. But it is using my new authority certificate cafile='/etc/pki/ovirt-engine/apache-ca.pem' which does not validate the certificate generated by the ovirt engine setup, which the ovn-provider probably uses.
I didn't exactly know where the parameter for the validation ca file is. Probably it is the ca_file parameter in /etc/ovirt-imageio/conf.d/50-engine.conf. But that needs to be set to my own authority ca file.
I modified the python file to set the ca_file parameter to the engine setups ca_file directly
/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py
So the function call around line 50 looks like this:
backend = module.open( ticket.url, mode, sparse=ticket.sparse, dirty=ticket.dirty, cafile='/etc/pki/ovirt-engine/ca.pem' #config.tls.ca_file )
Now the image upload works, but obviously this is not the way to fix things. Is there an other way to make image-io accept the certificate from the engine setup, while using my custom certificate? I don't want to replace the certificates of all ovirt components with custom certificates. I only need the weblogin with my custom certificate.
Adding Nir.
It's been quite some time since I checked imageio and using 3rd-party CAs, not sure about current status.
Last time I tried this (before the work done on imageio for 4.4), it was enough to make imageio use apache keypair and restart it, see also this bug and its dependencies:
https://bugzilla.redhat.com/show_bug.cgi?id=1385617
Nir - did you try this recently? If it's indeed broken, do we need a doc change, or imageio, or perhaps both?
I never tried to change the certificates on engine host since I don't have any setup with non-default certificates, but we do have automated tests using various configurations and certificates generated for imageio tests. See here how we generate the certificates: https://github.com/oVirt/ovirt-imageio/blob/master/daemon/test/genpki imageio automated tests run a server using these certificates, using a client configured to verify server certificates: https://github.com/oVirt/ovirt-imageio/blob/acf69ee00411e3b1cdd1c13c584759ec... If you think this does not cover all cases, please suggest how to test it better. Nir
On Thu, Jul 23, 2020 at 7:47 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Thu, Jul 23, 2020 at 5:14 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Sun, Jul 19, 2020 at 5:23 PM <ramon@clematide.ch> wrote:
Hi
I did a fresh installation of version 4.4.0.3. After the engine setup I replaced the apache certificate with a custom certificate. I used this article to do it: https://myhomelab.gr/linux/2020/01/20/replacing_ovirt_ssl.html
To summarize, I replaced those files with my own authority and the signed custom certificate
/etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/apache-ca.pem
That worked so far, apache uses now my certificate, login is possible. To setup a new machine, I need to upload an iso image, which failed. I found this error in /var/log/ovirt-imageio/daemon.log
2020-07-08 20:43:23,750 INFO (Thread-10) [http] OPEN client=192.168.1.228 2020-07-08 20:43:23,767 INFO (Thread-10) [backends.http] Open backend netloc='the_secret_hostname:54322' path='/images/ef60404c-dc69-4a3d-bfaa-8571f675f3e1' cafile='/etc/pki/ovirt-engine/apache-ca.pem' secure=True 2020-07-08 20:43:23,770 ERROR (Thread-10) [http] Server error Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 699, in __call__ self.dispatch(req, resp) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 744, in dispatch return method(req, resp, *match.groups()) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/cors.py", line 84, in wrapper return func(self, req, resp, *args) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/images.py", line 66, in put backends.get(req, ticket, self.config), File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py", line 53, in get cafile=config.tls.ca_file) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 48, in open secure=options.get("secure", True)) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 63, in __init__ options = self._options() File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 364, in _options self._con.request("OPTIONS", self.url.path) File "/usr/lib64/python3.6/http/client.py", line 1254, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output self.send(msg) File "/usr/lib64/python3.6/http/client.py", line 974, in send self.connect() File "/usr/lib64/python3.6/http/client.py", line 1422, in connect server_hostname=server_hostname) File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket _context=self, _session=session) File "/usr/lib64/python3.6/ssl.py", line 776, in __init__ self.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake self._sslobj.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897) 2020-07-08 20:43:23,770 INFO (Thread-10) [http] CLOSE client=192.168.1.228 [connection 1 ops, 0.019775 s] [dispatch 1 ops, 0.003114 s]
I'm a python developer so I had no problem reading the traceback.
The SSL handshake fails when image-io tries to connect to what I think is called an ovn-provider. But it is using my new authority certificate cafile='/etc/pki/ovirt-engine/apache-ca.pem' which does not validate the certificate generated by the ovirt engine setup, which the ovn-provider probably uses.
I didn't exactly know where the parameter for the validation ca file is. Probably it is the ca_file parameter in /etc/ovirt-imageio/conf.d/50-engine.conf. But that needs to be set to my own authority ca file.
I modified the python file to set the ca_file parameter to the engine setups ca_file directly
/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py
So the function call around line 50 looks like this:
backend = module.open( ticket.url, mode, sparse=ticket.sparse, dirty=ticket.dirty, cafile='/etc/pki/ovirt-engine/ca.pem' #config.tls.ca_file )
Now the image upload works, but obviously this is not the way to fix things. Is there an other way to make image-io accept the certificate from the engine setup, while using my custom certificate? I don't want to replace the certificates of all ovirt components with custom certificates. I only need the weblogin with my custom certificate.
Adding Nir.
It's been quite some time since I checked imageio and using 3rd-party CAs, not sure about current status.
Last time I tried this (before the work done on imageio for 4.4), it was enough to make imageio use apache keypair and restart it, see also this bug and its dependencies:
https://bugzilla.redhat.com/show_bug.cgi?id=1385617
Nir - did you try this recently? If it's indeed broken, do we need a doc change, or imageio, or perhaps both?
I never tried to change the certificates on engine host since I don't have any setup with non-default certificates, but we do have automated tests using various configurations and certificates generated for imageio tests.
See here how we generate the certificates: https://github.com/oVirt/ovirt-imageio/blob/master/daemon/test/genpki
imageio automated tests run a server using these certificates, using a client configured to verify server certificates: https://github.com/oVirt/ovirt-imageio/blob/acf69ee00411e3b1cdd1c13c584759ec...
If you think this does not cover all cases, please suggest how to test it better.
I admit I do not know imageio well enough, sorry. I assume that you do not run an engine in your tests. Are you sure you can still cover all cases that involve imageio? Does the flow "upload an image from web admin ui" require an engine? Not only in the interaction between browser code and imageio, but also between engine/httpd and imageio (if any). I guess it would be best to add to OST code to change the certs to 3rd-party ones, and then try imageio upload via a browser. Best regards, -- Didi
On Sun, Jul 19, 2020 at 5:22 PM <ramon@clematide.ch> wrote:
Hi
I did a fresh installation of version 4.4.0.3. After the engine setup I replaced the apache certificate with a custom certificate. I used this article to do it: https://myhomelab.gr/linux/2020/01/20/replacing_ovirt_ssl.html
To summarize, I replaced those files with my own authority and the signed custom certificate
/etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/apache-ca.pem
That worked so far, apache uses now my certificate, login is possible. To setup a new machine, I need to upload an iso image, which failed. I found this error in /var/log/ovirt-imageio/daemon.log
2020-07-08 20:43:23,750 INFO (Thread-10) [http] OPEN client=192.168.1.228 2020-07-08 20:43:23,767 INFO (Thread-10) [backends.http] Open backend netloc='the_secret_hostname:54322' path='/images/ef60404c-dc69-4a3d-bfaa-8571f675f3e1' cafile='/etc/pki/ovirt-engine/apache-ca.pem' secure=True 2020-07-08 20:43:23,770 ERROR (Thread-10) [http] Server error Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 699, in __call__ self.dispatch(req, resp) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 744, in dispatch return method(req, resp, *match.groups()) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/cors.py", line 84, in wrapper return func(self, req, resp, *args) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/images.py", line 66, in put backends.get(req, ticket, self.config), File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py", line 53, in get cafile=config.tls.ca_file) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 48, in open secure=options.get("secure", True)) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 63, in __init__ options = self._options() File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 364, in _options self._con.request("OPTIONS", self.url.path) File "/usr/lib64/python3.6/http/client.py", line 1254, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output self.send(msg) File "/usr/lib64/python3.6/http/client.py", line 974, in send self.connect() File "/usr/lib64/python3.6/http/client.py", line 1422, in connect server_hostname=server_hostname) File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket _context=self, _session=session) File "/usr/lib64/python3.6/ssl.py", line 776, in __init__ self.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake self._sslobj.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897) 2020-07-08 20:43:23,770 INFO (Thread-10) [http] CLOSE client=192.168.1.228 [connection 1 ops, 0.019775 s] [dispatch 1 ops, 0.003114 s]
I'm a python developer so I had no problem reading the traceback.
The SSL handshake fails when image-io tries to connect to what I think is called an ovn-provider. But it is using my new authority certificate cafile='/etc/pki/ovirt-engine/apache-ca.pem' which does not validate the certificate generated by the ovirt engine setup, which the ovn-provider probably uses.
I didn't exactly know where the parameter for the validation ca file is. Probably it is the ca_file parameter in /etc/ovirt-imageio/conf.d/50-engine.conf.
Right
But that needs to be set to my own authority ca file.
Right, but you should not modify this file, it is owned by engine and your changes will be lost on the next upgrade. As documented in the top of the file, you need to create a drop in file: $ cat /etc/ovirt-imageio/cond.d/99-local.conf [tls] ca_file = ... I think you need to change the key_file and cert_file, otherwise clients connected to imageio server may fail to verify the server certificate. And restart the ovirt-imageio service.
I modified the python file to set the ca_file parameter to the engine setups ca_file directly
/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py
So the function call around line 50 looks like this:
backend = module.open( ticket.url, mode, sparse=ticket.sparse, dirty=ticket.dirty, cafile='/etc/pki/ovirt-engine/ca.pem' #config.tls.ca_file )
Now the image upload works, but obviously this is not the way to fix things. Is there an other way to make image-io accept the certificate from the engine setup, while using my custom certificate? I don't want to replace the certificates of all ovirt components with custom certificates. I only need the weblogin with my custom certificate.
This works but you will lose the change on the next update, and as you know modifying python code in production is risky. Nir
Hi Nir I did not modify /etc/ovirt-imageio/conf.d/50-engine.conf I only replaced those files: /etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/apache-ca.pem ovirt-imageio has the apache certificates configured by default. I found certificates generated by the engine setup for imageio (but not used?) So I switched to those certificates: cat /etc/ovirt-imageio/conf.d/99-locl.conf [tls] key_file = /etc/pki/ovirt-engine/keys/imageio-proxy.key.nopass cert_file = /etc/pki/ovirt-engine/certs/imageio-proxy.cer ca_file = /etc/pki/ovirt-engine/ca.pem When I test the connection in the image upload screen, now my browser does not validate the imageio's certificate. When import the ca generated by the engine setup, upload works. But I don't want to import the ca generated by the engine setup.
On Thu, Jul 30, 2020, 09:31 Ramon Clematide <ramon@clematide.ch> wrote:
Hi Nir
I did not modify /etc/ovirt-imageio/conf.d/50-engine.conf
I only replaced those files:
/etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/apache-ca.pem
ovirt-imageio has the apache certificates configured by default.
So why did you change the code using the default configuration?
I found certificates generated by the engine setup for imageio (but not used?)
So I switched to those certificates:
cat /etc/ovirt-imageio/conf.d/99-locl.conf [tls] key_file = /etc/pki/ovirt-engine/keys/imageio-proxy.key.nopass cert_file = /etc/pki/ovirt-engine/certs/imageio-proxy.cer ca_file = /etc/pki/ovirt-engine/ca.pem
When I test the connection in the image upload screen, now my browser does not validate the imageio's certificate. When import the ca generated by the engine setup, upload works. But I don't want to import the ca generated by the engine setup.
Why did you switch to engine ca if you don't want to use it? When you change certificates, you need to restart the ovirt-imageio service since it loads the certificates during startup. Did you restart it? _______________________________________________
Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/GRKFPQKHKODCJU...
On Sun, Jul 19, 2020, 17:22 <ramon@clematide.ch> wrote:
Hi
I did a fresh installation of version 4.4.0.3. After the engine setup I replaced the apache certificate with a custom certificate. I used this article to do it: https://myhomelab.gr/linux/2020/01/20/replacing_ovirt_ssl.html
To summarize, I replaced those files with my own authority and the signed custom certificate
/etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/apache-ca.pem
That worked so far, apache uses now my certificate, login is possible. To setup a new machine, I need to upload an iso image, which failed. I found this error in /var/log/ovirt-imageio/daemon.log
2020-07-08 20:43:23,750 INFO (Thread-10) [http] OPEN client=192.168.1.228 2020-07-08 20:43:23,767 INFO (Thread-10) [backends.http] Open backend netloc='the_secret_hostname:54322' path='/images/ef60404c-dc69-4a3d-bfaa-8571f675f3e1' cafile='/etc/pki/ovirt-engine/apache-ca.pem' secure=True 2020-07-08 20:43:23,770 ERROR (Thread-10) [http] Server error Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 699, in __call__ self.dispatch(req, resp) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 744, in dispatch return method(req, resp, *match.groups()) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/cors.py", line 84, in wrapper return func(self, req, resp, *args) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/images.py", line 66, in put backends.get(req, ticket, self.config), File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py", line 53, in get cafile=config.tls.ca_file) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 48, in open secure=options.get("secure", True)) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 63, in __init__ options = self._options() File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 364, in _options self._con.request("OPTIONS", self.url.path) File "/usr/lib64/python3.6/http/client.py", line 1254, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output self.send(msg) File "/usr/lib64/python3.6/http/client.py", line 974, in send self.connect() File "/usr/lib64/python3.6/http/client.py", line 1422, in connect server_hostname=server_hostname) File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket _context=self, _session=session) File "/usr/lib64/python3.6/ssl.py", line 776, in __init__ self.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake self._sslobj.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897) 2020-07-08 20:43:23,770 INFO (Thread-10) [http] CLOSE client=192.168.1.228 [connection 1 ops, 0.019775 s] [dispatch 1 ops, 0.003114 s]
I'm a python developer so I had no problem reading the traceback.
The SSL handshake fails when image-io tries to connect to what I think is called an ovn-provider. But it is using my new authority certificate cafile='/etc/pki/ovirt-engine/apache-ca.pem' which does not validate the certificate generated by the ovirt engine setup, which the ovn-provider probably uses.
I didn't exactly know where the parameter for the validation ca file is. Probably it is the ca_file parameter in /etc/ovirt-imageio/conf.d/50-engine.conf. But that needs to be set to my own authority ca file.
I modified the python file to set the ca_file parameter to the engine setups ca_file directly
/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py
So the function call around line 50 looks like this:
backend = module.open( ticket.url, mode, sparse=ticket.sparse, dirty=ticket.dirty, cafile='/etc/pki/ovirt-engine/ca.pem' #config.tls.ca_file )
Reading this again, the problem is clear now. The imageio proxy is trying to use your CA to verify the the host imageio daemon certificate. This cannot work because the host certificate is signed by engine CA, and the imageio daemon on the host is using vdsm certificates With the current version you will have to create certificates for each host imageio daemon and configure it, which is probably not practical. So looks like we need to add additional ca_file configuration, which must be used when connecting to host using the http backend. Using the ca_file used for the http server is wrong, it works only for the default configuration. Please file imageio bug for this. You can keep your local change until we fix this.
Now the image upload works, but obviously this is not the way to fix things. Is there an other way to make image-io accept the certificate from the engine setup, while using my custom certificate? I don't want to replace the certificates of all ovirt components with custom certificates.
This is also not supported. I only need the weblogin with my custom certificate.
Regards _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/365ISJ7JHAKYIG...
On Thu, Jul 30, 2020 at 12:53 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Jul 19, 2020, 17:22 <ramon@clematide.ch> wrote:
Hi
I did a fresh installation of version 4.4.0.3. After the engine setup I replaced the apache certificate with a custom certificate. I used this article to do it: https://myhomelab.gr/linux/2020/01/20/replacing_ovirt_ssl.html
To summarize, I replaced those files with my own authority and the signed custom certificate
/etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/apache-ca.pem
That worked so far, apache uses now my certificate, login is possible. To setup a new machine, I need to upload an iso image, which failed. I found this error in /var/log/ovirt-imageio/daemon.log
2020-07-08 20:43:23,750 INFO (Thread-10) [http] OPEN client=192.168.1.228 2020-07-08 20:43:23,767 INFO (Thread-10) [backends.http] Open backend netloc='the_secret_hostname:54322' path='/images/ef60404c-dc69-4a3d-bfaa-8571f675f3e1' cafile='/etc/pki/ovirt-engine/apache-ca.pem' secure=True 2020-07-08 20:43:23,770 ERROR (Thread-10) [http] Server error Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 699, in __call__ self.dispatch(req, resp) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 744, in dispatch return method(req, resp, *match.groups()) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/cors.py", line 84, in wrapper return func(self, req, resp, *args) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/images.py", line 66, in put backends.get(req, ticket, self.config), File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py", line 53, in get cafile=config.tls.ca_file) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 48, in open secure=options.get("secure", True)) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 63, in __init__ options = self._options() File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 364, in _options self._con.request("OPTIONS", self.url.path) File "/usr/lib64/python3.6/http/client.py", line 1254, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output self.send(msg) File "/usr/lib64/python3.6/http/client.py", line 974, in send self.connect() File "/usr/lib64/python3.6/http/client.py", line 1422, in connect server_hostname=server_hostname) File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket _context=self, _session=session) File "/usr/lib64/python3.6/ssl.py", line 776, in __init__ self.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake self._sslobj.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897) 2020-07-08 20:43:23,770 INFO (Thread-10) [http] CLOSE client=192.168.1.228 [connection 1 ops, 0.019775 s] [dispatch 1 ops, 0.003114 s]
I'm a python developer so I had no problem reading the traceback.
The SSL handshake fails when image-io tries to connect to what I think is called an ovn-provider. But it is using my new authority certificate cafile='/etc/pki/ovirt-engine/apache-ca.pem' which does not validate the certificate generated by the ovirt engine setup, which the ovn-provider probably uses.
I didn't exactly know where the parameter for the validation ca file is. Probably it is the ca_file parameter in /etc/ovirt-imageio/conf.d/50-engine.conf. But that needs to be set to my own authority ca file.
I modified the python file to set the ca_file parameter to the engine setups ca_file directly
/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py
So the function call around line 50 looks like this:
backend = module.open( ticket.url, mode, sparse=ticket.sparse, dirty=ticket.dirty, cafile='/etc/pki/ovirt-engine/ca.pem' #config.tls.ca_file )
Reading this again, the problem is clear now.
The imageio proxy is trying to use your CA to verify the the host imageio daemon certificate. This cannot work because the host certificate is signed by engine CA, and the imageio daemon on the host is using vdsm certificates
With the current version you will have to create certificates for each host imageio daemon and configure it, which is probably not practical.
So looks like we need to add additional ca_file configuration, which must be used when connecting to host using the http backend. Using the ca_file used for the http server is wrong, it works only for the default configuration.
Please file imageio bug for this.
I filed this bug for you: https://bugzilla.redhat.com/1862107 I hope we can get it fixed in 4.4.2.
You can keep your local change until we fix this.
Now the image upload works, but obviously this is not the way to fix things. Is there an other way to make image-io accept the certificate from the engine setup, while using my custom certificate? I don't want to replace the certificates of all ovirt components with custom certificates.
This is also not supported.
I only need the weblogin with my custom certificate.
Regards _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/365ISJ7JHAKYIG...
HI Nir Thank you. Yes exactly, another parameter different to config.tls.ca_file would be nice. Regards
participants (4)
-
Nir Soffer -
Ramon Clematide -
ramon＠clematide.ch
-
Yedidyah Bar David