That's typically one of three issues: 1. You've replaced the certificate used by apache, but haven't updated the configuration for the ovirt-imageio-proxy. 2. You're using an older web browser. (Pale Moon, Waterfox, older versions of Chrome, etc.) 3. There's an issue that causes the initial transfers (upload or download) to fail once or twice every ~24 hours. If it's the first issue, a quick fix is to simply add your new certificate's CA cert to the hidden /etc/pki/ovirt-engine/.truststore file (It's a java keystore.) then restart the engine and imageio-proxy. If it's the second issue, try using a newer browser. Current versions of Chrome are known to work. If it's the third issue, abort the download and try again. Assuming you've ruled out the other two issues, it should work. If not, we might need your logs to determine what's wrong. If your engine is stuck with an aborted download, try restarting both the ovirt-engine and ovirt-imageio-proxy services. (That should get rid of the bad download state.) -Patrick Hibbs On Wed, 2022-03-02 at 23:42 +0000, louisb(a)ameritech.net wrote:

On Sun, Mar 6, 2022 at 9:42 PM <louisb(a)ameritech.net&gt; wrote: In oVirt 4.4 the ovirt-imageio-proxy service was replaced with the ovirt-imageio service. The built-in configuration should work with the default (self signed) CA and with custom CA without any configuration change. Is this all-in-one installation, when ovirt-engine is installed on the single hypervisor, and the same host is added later as an hypervisor? To make sure you configured the browser correctly, please open the "upload" dialog and click the "Test connection" button. If the testing the connection works the browser can communicate with the ovirt-imageio service and your system is ready for upload from the browser. Nir

I set up a new ovirt test instance on a clean Rocky Linux 8.5 server with a custom apache cert about two weeks ago. Uploading a test image via the browser didn't work until I changed the .truststore file. I should also point out that I also had to set the cert in apache's config. Simply replacing the symlink in the cert directory didn't work as it wasn't pointing to it at all. (Instead it was pointing at some snakeoil cert generated by apache.) Granted, the apache issue is specific to Rocky, but the imageio service is definitely in ovirt's full control. If the imageio service is supposed to work out of the box with a custom certificate, there's something amiss. WARNING: Small rant follows: Yes, I could have changed a config file instead of changing .truststore, but it's just another way to achieve the same result. (And the one I discovered back in ovirt 3.x.) It doesn't make the process any eaiser, if anything it's just another option to check if something goes wrong. Instead of checking only .truststore, Now we have to check .truststore, and any number of extra config files for a redirect statement, and the load ordering of those config files, *and* whether or not those redirect statements point to a valid cert or not. Instead of having just one place to troubleshoot, now there's at least four. The config file change also doesn't make it any eaiser to perform those changes. You still need to manually make these changes via ssh on the engine host. Why would I want to advice changing a config file, and risk that much of an additional mess to deal with in support, when I can tell them one specific file to fix that has none of these extras to deal with? Personally, I would choose the option with less chance for human error. /rant -Patrick Hibbs On Sun, 2022-03-06 at 21:54 +0200, Nir Soffer wrote:

The new server is a standalone engine with 2 (also clean Rocky Linux 8.5 installs) hypervisor hosts using GlusterFS in client mode. (I.e. *NOT* hyperconverged.) It's basically the same setup I have for my production engine. (The difference being that the production engine is older and running on CentOS 8 Stream.) As for the .truststore file, I meant this one on the engine host: ---snip--- $ ls -la /etc/pki/ovirt-engine/.truststore -rw-r--r--. 1 root root 4170 Feb 14 18:08 /etc/pki/ovirt- engine/.truststore ---snip--- If we peek into this on a clean installation we get something like this: ---snip--- sudo keytool --list --keystore /etc/pki/ovirt- engine/.truststore.generated Enter keystore password: Keystore type: PKCS12 Keystore provider: SUN Your keystore contains 2 entries cacert, Mar 6, 2022, trustedCertEntry, Certificate fingerprint (SHA-256): E9:8E:FC:CF:AE:8C:27:3A:FD:BD:8E:0E:72:DA:99:84:7E:65:55:E6:86:74:D8:3C :FE:AE:1C:D7:3C:36:D8:C1 qemu-cacert, Mar 6, 2022, trustedCertEntry, Certificate fingerprint (SHA-256): 65:D1:E3:14:2D:0D:44:28:B2:A7:EC:AA:B6:84:DA:D6:EC:8A:86:E7:FF:D9:FC:07 :A6:22:8A:A4:86:AD:0E:EB ---snip--- By default, the two certs in /etc/pki/ovirt-engine/.truststore are the generated CA cert and the host certificate it signed. If we change this file to include an additional certificate and mark it as a trustedCertEntry, the imageio service will work without any additional configuration. This is because oVirt (Used to??? Does for me.) defaults to using this file out of the box. The instructions at https://www.ovirt.org/documentation/administration_guide/index.html#Repla... don't mention this at all. Instead, they instruct the administrator to create additional config files which creates additional chances for human error to cause problems. In oVirt 3.x, modifying /etc/pki/ovirt-engine/.truststore was the *only* way to get the image uploads working via the Web Admin interface if you changed apache's certificate. This solution was also not documented back then. I remember spending about a week back then trying to hunt down where the upload was failing only to finally stumble over .truststore by miss typing a ls command. Good times :) General CA Issue thoughts below: As for the CA issue in general, this is my take of it. (Keep in mind it's solely from an end-user perspective, so I'd love to hear what others have to say about it.): As of this email, looking at https://www.ovirt.org/documentation/administration_guide/index.html and searching for the term "certificate" on that page gives 104 results. Of those results, we get about 3 different major methods for managing the various certificates: 1. Manual updating of the certs via ssh. (Apache, imageio, Various SSO solutions, Various storage protocols) 2. VDSM manages the certs, and administrator changes *will* be overwritten. (Host Cockpit Console, Host SPICE Console, Host VNC Console, VDSM internal connections) 3. Uploading the certificates for use via the Web Admin interface (Interop with other services: ESX / k8 / openstack etc.) Notice: That there is no overlap for any of the given methods. Each of these certs *MUST* be managed in their own specific way, and each has their own checks that must be met (or lack thereof.) There is also no clear dividing line between what certs are considered "internal" to oVirt, and what certs should be encountered by end-users on a regular basis. I.e. The Apache cert is clearly for end users, but image transfers will fail if imageio isn't dealt with seperately. SPICE Consoles will work out of the box regardless of changing Apache's cert, but VNC Consoles will fail entirely unless the web viewer is used. Host Cockpit consoles also fail if Apache's cert is changed, and this cannot be fixed at all by the administrator. The only recourse is to install the engine's generated CA on the end-user terminals. Further, as the mailing list will attest, sometimes the VDSM managed certs don't update properly, and expiry notifications are easily missed by administrators. Leading to multiple headaches and downtime. In short, the issue in-general is a lack of centralized universal management for certificates. A more centralized management system would do away with the multiple management methods, and focus on providing a consistent view of the entire PKI infrastructure regardless of who or what accessed it from anywhere. (I.e. VDSM or an Administrator.) This would cut out the current configuration headaches, and make it a single check to figure what's up when things go wrong. Such a solution would also, in my opinion, include a management tab in the Web Admin GUI. Such a tab could be used to deploy replacement certs for the oVirt services that allowed substitutions, or regenerating certs for the services that do not. In addition to seeing the current (in-use) certificate information for each service. If I was going to truly make wishes, I'd also like to see an external notification mechanism added too. Like an e-mail notification when something gets too close to expiry. (Could just use sendmail configured by the host initially, but I'm sure others would love to configure that via the Web GUI too.) Also, maybe make the expiries configurable from the GUI tab too. As to what such a system would look like, I'd say use VDSM's database to store the certificates / keys (along with a backup of the previous key to prevent an upgrade failure from requiring a host reinstall) and provide a REST API / VDSM command to access and manage them. With only VDSM's cert and a client key permanently stored on the host file systems. (VDSM needs to connect to the DB anyway to do something useful.) When VDSM starts, it could connect to the DB and pull down any updated certs / keys from it's database for use by the host's services and itself. That way any changes to the keys or certs are picked up when VDSM resets, and changes simply require the administrator to put the affected hosts into and out of maintenance. As a bonus, this design also gives us auditing of certificate changes (regulatory / legal / policy compliance) for free. As stated before, I'd love to hear other solutions / viewpoints. -Patrick Hibbs On Sun, 2022-03-06 at 23:58 +0200, Nir Soffer wrote: