9:14 p.m.
The new server is a standalone engine with 2 (also clean Rocky Linux
8.5 installs) hypervisor hosts using GlusterFS in client mode. (I.e.
*NOT* hyperconverged.)
It's basically the same setup I have for my production engine. (The
difference being that the production engine is older and running on
CentOS 8 Stream.)
As for the .truststore file, I meant this one on the engine host:
---snip---
$ ls -la /etc/pki/ovirt-engine/.truststore
-rw-r--r--. 1 root root 4170 Feb 14 18:08 /etc/pki/ovirt-
engine/.truststore
---snip---
If we peek into this on a clean installation we get something like
this:
---snip---
sudo keytool --list --keystore /etc/pki/ovirt-
engine/.truststore.generated
Enter keystore password:
Keystore type: PKCS12
Keystore provider: SUN
Your keystore contains 2 entries
cacert, Mar 6, 2022, trustedCertEntry,
Certificate fingerprint (SHA-256):
E9:8E:FC:CF:AE:8C:27:3A:FD:BD:8E:0E:72:DA:99:84:7E:65:55:E6:86:74:D8:3C
:FE:AE:1C:D7:3C:36:D8:C1
qemu-cacert, Mar 6, 2022, trustedCertEntry,
Certificate fingerprint (SHA-256):
65:D1:E3:14:2D:0D:44:28:B2:A7:EC:AA:B6:84:DA:D6:EC:8A:86:E7:FF:D9:FC:07
:A6:22:8A:A4:86:AD:0E:EB
---snip---
By default, the two certs in /etc/pki/ovirt-engine/.truststore are the
generated CA cert and the host certificate it signed.
If we change this file to include an additional certificate and mark it
as a trustedCertEntry, the imageio service will work without any
additional configuration. This is because oVirt (Used to??? Does for
me.) defaults to using this file out of the box.
The instructions at
https://www.ovirt.org/documentation/administration_guide/index.html#Repla...
don't mention this at all. Instead, they instruct the administrator to
create additional config files which creates additional chances for
human error to cause problems.
In oVirt 3.x, modifying /etc/pki/ovirt-engine/.truststore was the
*only* way to get the image uploads working via the Web Admin interface
if you changed apache's certificate. This solution was also not
documented back then. I remember spending about a week back then trying
to hunt down where the upload was failing only to finally stumble over
.truststore by miss typing a ls command. Good times :)
General CA Issue thoughts below:
As for the CA issue in general, this is my take of it. (Keep in mind
it's solely from an end-user perspective, so I'd love to hear what
others have to say about it.):
As of this email, looking at
https://www.ovirt.org/documentation/administration_guide/index.html
and searching for the term "certificate" on that page gives 104
results.
Of those results, we get about 3 different major methods for managing
the various certificates:
1. Manual updating of the certs via ssh. (Apache, imageio, Various SSO
solutions, Various storage protocols)
2. VDSM manages the certs, and administrator changes *will* be
overwritten. (Host Cockpit Console, Host SPICE Console, Host VNC
Console, VDSM internal connections)
3. Uploading the certificates for use via the Web Admin interface
(Interop with other services: ESX / k8 / openstack etc.)
Notice: That there is no overlap for any of the given methods. Each of
these certs *MUST* be managed in their own specific way, and each has
their own checks that must be met (or lack thereof.)
There is also no clear dividing line between what certs are considered
"internal" to oVirt, and what certs should be encountered by end-users
on a regular basis. I.e. The Apache cert is clearly for end users, but
image transfers will fail if imageio isn't dealt with seperately. SPICE
Consoles will work out of the box regardless of changing Apache's cert,
but VNC Consoles will fail entirely unless the web viewer is used. Host
Cockpit consoles also fail if Apache's cert is changed, and this cannot
be fixed at all by the administrator. The only recourse is to install
the engine's generated CA on the end-user terminals.
Further, as the mailing list will attest, sometimes the VDSM managed
certs don't update properly, and expiry notifications are easily missed
by administrators. Leading to multiple headaches and downtime.
In short, the issue in-general is a lack of centralized universal
management for certificates.
A more centralized management system would do away with the multiple
management methods, and focus on providing a consistent view of the
entire PKI infrastructure regardless of who or what accessed it from
anywhere. (I.e. VDSM or an Administrator.) This would cut out the
current configuration headaches, and make it a single check to figure
what's up when things go wrong.
Such a solution would also, in my opinion, include a management tab in
the Web Admin GUI. Such a tab could be used to deploy replacement certs
for the oVirt services that allowed substitutions, or regenerating
certs for the services that do not. In addition to seeing the current
(in-use) certificate information for each service. If I was going to
truly make wishes, I'd also like to see an external notification
mechanism added too. Like an e-mail notification when something gets
too close to expiry. (Could just use sendmail configured by the host
initially, but I'm sure others would love to configure that via the Web
GUI too.) Also, maybe make the expiries configurable from the GUI tab
too.
As to what such a system would look like, I'd say use VDSM's database
to store the certificates / keys (along with a backup of the previous
key to prevent an upgrade failure from requiring a host reinstall) and
provide a REST API / VDSM command to access and manage them. With only
VDSM's cert and a client key permanently stored on the host file
systems. (VDSM needs to connect to the DB anyway to do something
useful.) When VDSM starts, it could connect to the DB and pull down any
updated certs / keys from it's database for use by the host's services
and itself. That way any changes to the keys or certs are picked up
when VDSM resets, and changes simply require the administrator to put
the affected hosts into and out of maintenance. As a bonus, this design
also gives us auditing of certificate changes (regulatory / legal /
policy compliance) for free.
As stated before, I'd love to hear other solutions / viewpoints.
-Patrick Hibbs
On Sun, 2022-03-06 at 23:58 +0200, Nir Soffer wrote:
On Sun, Mar 6, 2022 at 11:42 PM Patrick Hibbs
<hibbsncc1701(a)gmail.com> wrote:
>
> I set up a new ovirt test instance on a clean Rocky Linux 8.5
> server
> with a custom apache cert about two weeks ago.
Do you have single server used both for running ovirt-engine and
as a hypervisor? This requires special configuration. If engine is
not running on the hypervisor, for example engine runs in a VM or
on another host, everything should work out of the box.
> Uploading a test image
> via the browser didn't work until I changed the .truststore file.
.truststore file where?
> I should also point out that I also had to set the cert in apache's
> config. Simply replacing the symlink in the cert directory didn't
> work
> as it wasn't pointing to it at all. (Instead it was pointing at
> some
> snakeoil cert generated by apache.) Granted, the apache issue is
> specific to Rocky, but the imageio service is definitely in ovirt's
> full control.
>
> If the imageio service is supposed to work out of the box with a
> custom
> certificate, there's something amiss.
These are the defaults:
$ ovirt-imageio --show-config | jq '.tls'
{
"ca_file": "/etc/pki/ovirt-engine/apache-ca.pem",
"cert_file": "/etc/pki/ovirt-engine/certs/apache.cer",
"enable": true,
"enable_tls1_1": false,
"key_file": "/etc/pki/ovirt-engine/keys/apache.key.nopass"
}
The ovirt-imageio service works with apache configuration files.
If these symlinks point to the right files, everything should work
out of the box.
If you change the apache PKI files, you need to modify ovirt-imageio
configuration by adding a drop-in configuration file with the right
configuration:
$ cat /etc/ovirt-imageio/conf.d/99-local.conf
[tls]
key_file = /path/to/keyfile
cert_file = /path/to/certfile
ca_file = /path/to/cafile
Note: the following configuration *must not* change:
$ ovirt-imageio --show-config | jq '.backend_http'
{
"buffer_size": 8388608,
"ca_file": "/etc/pki/ovirt-engine/ca.pem"
}
This CA file is used to access the hosts, which are managed by
ovirt-engine self signed CA, and cannot be replaced.
> WARNING: Small rant follows:
>
> Yes, I could have changed a config file instead of changing
> .truststore, but it's just another way to achieve the same result.
> (And
> the one I discovered back in ovirt 3.x.) It doesn't make the
> process
> any eaiser, if anything it's just another option to check if
> something
> goes wrong. Instead of checking only .truststore, Now we have to
> check
> .truststore, and any number of extra config files for a redirect
> statement, and the load ordering of those config files, *and*
> whether
> or not those redirect statements point to a valid cert or not.
> Instead
> of having just one place to troubleshoot, now there's at least
> four.
> The config file change also doesn't make it any eaiser to perform
> those
> changes. You still need to manually make these changes via ssh on
> the
> engine host. Why would I want to advice changing a config file, and
> risk that much of an additional mess to deal with in support, when
> I
> can tell them one specific file to fix that has none of these
> extras to
> deal with? Personally, I would choose the option with less chance
> for
> human error.
It is clear that you think we can have a better way to configure the
system,
but it is not clear what is the issue and what is the better
solution.
Can you explain in more details what is the problem with the
documented
solution on using custom PKI files, and what is the easier way to do
this?
If we have an easier way, it should be documented.
Nir
> On Sun, 2022-03-06 at 21:54 +0200, Nir Soffer wrote:
> > On Sun, Mar 6, 2022 at 9:42 PM <louisb(a)ameritech.net> wrote:
> > >
> > > I don't have the file "ovirt-imageio-proxy" on my system,
is
> > > there
> > > another file that I should be looking at? Once I locate the
> > > correct file what content in the file needs to change?
> > >
> > > I'm using the latest release of "Firefox/91.6.0" as my
> > > browser,
> > > and i import the "Engine CA" after the fact. However, after
> > > the
> > > import I tried again and got the same results.
> >
> > In oVirt 4.4 the ovirt-imageio-proxy service was replaced with
> > the
> > ovirt-imageio service.
> >
> > The built-in configuration should work with the default (self
> > signed)
> > CA and with custom
> > CA without any configuration change.
> >
> > Is this all-in-one installation, when ovirt-engine is installed
> > on
> > the
> > single hypervisor,
> > and the same host is added later as an hypervisor?
> >
> > To make sure you configured the browser correctly, please open
> > the
> > "upload" dialog
> > and click the "Test connection" button. If the testing the
> > connection
> > works the browser
> > can communicate with the ovirt-imageio service and your system is
> > ready for upload
> > from the browser.
> >
> > Nir