On Sun, Dec 6, 2020 at 12:34 AM Derek Atkins <derek(a)ihtfp.com&gt; wrote: Indeed. I don't think you'll find anything like this. People did come up in the past with various procedure to hack pki like what you want, but these are, generally speaking, quite fragile - usually do not get updated over versions etc. I am pretty certain the only way to do this using "official" tools/docs is: 1. Stop all VMs except for the engine one. 2. Take a backup with engine-backup. 3. Stop the engine VM. 4. Reinstall the host OS from scratch or use ovirt-hosted-engine-cleanup. 5. Provision the host again as a hosted-engine host, using '--restore-from-file'. Either using new storage for the engine, or after cleaning up the existing hosted-engine storage. If you still want to try doing this manually, then the tool to use is pki-enroll-request.sh. IIRC it's documented. You should find what keys/certs you want to replace, generate new keys and CSRs (or use existing keys and generate CSRs, or even use existing CSRs if you find them), copy to the engine, sign with pki-enroll-request.sh, then copy the generated cert to the host. I am almost certain there is no way to tell vdsm (and other processes) to reload the certs, so you'll have to restart it (them) - and this usually requires putting the host in maintenance (and therefore stop (migrate) all VMs). Which ones, specifically? AFAIR no process uses these certs as such. There are only processes that use the ssh-format keys extracted from them, which do not include a signature (sha1 or whatever). If you think I am wrong, and/or notice other certs that need to be regenerated, that's a bug - please open one. Thanks! Re remote-viewer/spice: You didn't say if you tried again after engine-setup and what happened. In any case, this is unrelated to vmconsole (which is for serial consoles, using ssh). But you might still need to regenerate the host cert. BTW: You can try using novnc and websocket-proxy - engine-setup does update the cert for the latter, so this might work as-is. Best regards, -- Didi

Hi Didi, One more question: Can you verify that etc/pki/libvirt/clientcert.pem, etc/pki/vdsm/certs/vdsmcert.pem, and etc/pki/vdsm/libvirt-spice/server-cert.pem are all supposed to be same certificate (on the host)? By a quick find | grep all three of these files appear to be the <Host>.cer certificate file? -derek On Sun, December 6, 2020 12:25 pm, Derek Atkins wrote: -- Derek Atkins 617-623-3745 derek(a)ihtfp.com www.ihtfp.com Computer and Internet Security Consultant

On Sun, Dec 6, 2020 at 7:25 PM Derek Atkins <derek(a)ihtfp.com&gt; wrote: I assume you'll do that anyway, some time later, no? So why not now? I think 4.4.3 is a pretty good version. I agree it's "dangerous" in the sense that it involves lots of rather complex actions, some of which are hard to debug/fix if they fail. But: You can plan and test ahead on a test machine somewhere - even a VM, with nested-kvm. Just make sure it has no access to your host and storage (network-wise), so that it does not try to manage/access them. You are right - for a single host, you should do something like this: 1. Stop all VMs except for the engine 2. Put it to global maintenance 3. Stop the engine VM ('poweroff' from inside it, or 'hosted-engine --vm-shutdown') 4. Restart whatever services you want, or just reboot 5. Exit global maintenance 6. Engine VM should be started automatically in a few minutes. If it does not, you can 'hosted-engine --vm-start'. You can monitor agent.log in the meanwhile. engine-setup is not designed to touch hosts. Hosts should be managed by the engine, usually. ca.der is not used by anything, you can ignore it. The private key of the CA is in /etc/pki/ovirt-engine/private/ca.pem, and the public key is in /etc/pki/ovirt-engine/ca.pem. That's what all tools use. OK. I agree it would be nice, but I think it would require quite a lot of work. Generally speaking, the project considers the "standard" use case to be a setup of at least two hosts, and at least one host "extra" (in terms of capacity), so that if a host fails, you can still keep everything up. In that regard, a single-host setup is considered a kind of "corner case", meant mainly for testing/development, not production. Is there such a big advantage in using oVirt for a single host, compared to virt-manager? Good luck and best regards, -- Didi

On Sun, Dec 6, 2020 at 8:14 PM Derek Atkins <derek(a)ihtfp.com&gt; wrote: You are right - we didn't make engine-setup recreate existing certs for this - "Renew" deals with other stuff [1]. We only change the default for new ones [2], and wrote a procedure [3][4] for doing this manually. At the time, this wasn't mandatory - browsers didn't reject sha1. Perhaps now it should be. [1] https://www.ovirt.org/develop/release-management/features/infra/pki-renew... [2] https://gerrit.ovirt.org/c/ovirt-engine/+/65927 (I see that I even didn't open a bug for this at the time, not sure why) [3] https://www.ovirt.org/documentation/upgrade_guide/#Replacing_SHA-1_Certif... [4] https://bugzilla.redhat.com/show_bug.cgi?id=1420577 (This is a RHV doc bug, and the result was an update to RHV docs, which eventually were also published as [3] above) As I said, at the time it didn't seem to be mandatory, and docs seemed to be enough. If you feel otherwise, please open a bug. I think there is a difference, or at least there was, between what browsers did/do with https certs, and what they did with CA certs. If you had a CA cert already accepted/imported/trusted by the browser, and then you entered a site with a cert signed by this CA, but with a SHA1 signature, this was one separate case. Browsers started warning/ rejecting them earlier. If you have a CA cert with a SHA1 signature, and want to import that to a browser, that's another case. I didn't test recently (or much over time, other than working on these bugs) with recent browsers, but I think it took longer until they rejected (if indeed they do - not sure all of them do). Best regards, -- Didi

On Mon, Dec 7, 2020 at 2:22 PM Derek Atkins <derek(a)ihtfp.com&gt; wrote: [snip] +1 here. And I think developers should put more attention in single host environments than lastly done. Derek explained very well what could be many common situations to have a single host environment and the reason not to use virt-manager and such. At time there was the all-in-one and then it was deprecated/abandoned in favour of single host deployment. Now due to perhaps ansible playbook or new logic in host upgrades it seems to see more and more messages about single host not supported. In my opinion to do a reinstall every minor update is not feasible. The free version of ESXi could become a more appealing alternative in the near future for these kind of usages, with the risk of potentially eating away also the bigger shape scenario then. Perhaps to support again the all-in-one effort could be a good approach. If you think it can get more attention I can open an RFE for revamping all-in-one or an RFE to provide a mechanism to have an host update itself through a locally executed playbook and then "acquired" by the SHE in a second time when exiting global maintenance. I don't know the internals enough and I have not the coding skills, but I can support testing the functionality Thanks for listening Gianluca

Hi Michal, On Mon, December 7, 2020 11:43 am, Michal Skrivanek wrote: [snip] As of several years ago, I don't think any of these options worked with multiple, distributed (remote) users with different capabilities on the same VM Host. Has that changed? I think there are enough users who want this configuration (or, gasp, are actually using this configuration) that it might warrant a little more testing. Yes, we understand that there will be times that we need to shut down VMs and reboot the system, and those times can be scheduled (like I've done). However, that WOULD require a little more support, to at least have a recipe that works on a single-host hosted-engine solution. For host cert renewal, that recipe didn't really exist. [snip] Of course. (Personally, I think it should have been called 5.0 instead of 4.4, as it requires a full re-install to migrate from 4.3). Exactly. Some tested playbook recipes that allows a single-host hosted-engine deployment to perform these operations is really what I'm asking for. Yes, I know it will require rebooting. But reboot is much less risky that re-install! And for the record, after putting the new certificates into place by hand, just restarting a VM was sufficient to get Spice to pull in the new cert(s). So, technically, it LOOKS like I don't have to reboot the whole system (although I plan to do that tonight) -- I could just shutdown and re-run each VM. Thank you for all your support and everything you do for this project, Michal. We very much appreciate it! -derek -- Derek Atkins 617-623-3745 derek(a)ihtfp.com www.ihtfp.com Computer and Internet Security Consultant

This is the only place we found the answer we needed concerning how to sign a host key from the engine! Saved our bacon! The only tweak I would make is that there were a few more destinations at the end to copy the files into, and instead of rebooting the host we found that just restarting vdsmd and libvirtd got everything working without any existing VMs having to stop. Here's a complete update of what you have above intermingling some extra bits for anyone who has to do this in the future. If the certs on your oVirt host expire, it can be a PITA to figure out how to fix it. It's actually simple, but takes a LOT of manual work. Make sure every part of every script makes sense as I made some modifications as I documented it! I do not warrant that I didn't make a mistake somewhere. :) ########### On the ENGINE HOST ########### ## Check the CA Cert on the ENGINE openssl x509 -in /etc/pki/ovirt-engine/ca.pem -noout -dates ## If it is expired, then on the ENGINE, rebuild the CA Cert with this in a script: ## NOTE: these are steps from another post I didn't need to do them -- could be dangerous... ``` set -x ## Make the script echo everything out, so if it fails you know where set -e ## Make the script STOP on any error set my_date="$(date +"%Y%m%d%H%M%S")" # Backup the existing CA files /bin/cp -p /etc/pki/ovirt-engine/private/ca.pem /etc/pki/ovirt-engine/private/ca.pem.$my_date /bin/cp -p /etc/pki/ovirt-engine/ca.pem{,.$my_date} /bin/mv /etc/pki/ovirt-engine/certs/ca.der{,.$my_date} # Sign the key openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in /etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256 openssl x509 -in /etc/pki/ovirt-engine/ca.pem.new -text > /etc/pki/ovirt-engine/ca.pem.new.full # Put the files into place /bin/mv -f /etc/pki/ovirt-engine/ca.pem.new.full /etc/pki/ovirt-engine/ca.pem /bin/cp -p /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/certs/ca.der ``` Now you need to copy the new CA file over to the host: Source: ENGINE `/etc/pki/ovirt-engine/ca.pem` Dest: HOST `/tmp/new-ca.pem` ########### On the oVirt Host ########### # Create a CSR using the information from the existing certificate and the existing key: openssl x509 -x509toreq -in /etc/pki/libvirt/clientcert.pem -out /tmp/HOST.csr -signkey /etc/pki/libvirt/private/clientkey.pem Now you need to copy the new CA file over to the host: Source: HOST `/tmp/HOST.csr` Dest: ENGINE `/etc/pki/ovirt-engine/requests/full.hostname.com.req` ########### On the ENGINE HOST ########### # Now sign it: /usr/share/ovirt-engine/bin/pki-enroll-request.sh --name=full.hostname.com # NB -- adding --san results in an error: --san=host.na.me (So no Subject Alternate Names) Now you need to copy the new Certificate file over to the host: Source: ENGINE /etc/pki/ovirt-engine/certs/full.hostname.com.cer Dest: HOST /tmp/new-cert.pem ########### On the oVirt Host ########### Run this script to put the cert and CA in place. Note if you don't put a ca into /tmp/new-ca.pem it skips that step. ``` set -x set -e set my_date="$(date +"%Y%m%d%H%M%S")" for x in /etc/pki/libvirt/clientcert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-migrate/server-cert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem /etc/pki/vdsm/libvirt-vnc/server-cert.pem; do /bin/mv -n $x ${x}.${mydate} /bin/cp /tmp/new-cert.pem ${x} chmod 644 ${x} chown root:kvm ${x} done if -f /tmp/new-ca.pem; then for x in /etc/pki/vdsm/libvirt-migrate/ca-cert.pem /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-vnc/ca-cert.pem /etc/pki/vdsm/libvirt-spice/ca-cert.pem /etc/pki/CA/cacert.pem; do /bin/mv -n $x ${x}.${mydate} /bin/cp /tmp/new-ca.pem ${x} chmod 644 ${x} chown root:kvm ${x} done fi ``` Now you're ready to restart two vital services -- some people say "reboot the host" -- but we found that unecessary. Running this restart was safe for us and didn't cause any running VMs to crash or reboot -- they kept going without issue. Once we did this, waited a few minutes, the host came back up on the engine and everything was happy. Specifically a VM we were unable to get running was a click away from full functionality again! # Restart the two services affected by the key/cert changes systemctl restart vdsmd libvirtd

Ah, the steps I list here work for non-single node environments as well. The problem with vdsm-tool doing the certificate renewal in that setup is that the certificate needs to be signed by the engine host and not the local host. Apologies if this causes any confusion for single-node users. ________________________________ From: Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; Sent: Saturday, January 13, 2024 6:57 PM To: Michael Douglass <mikedoug(a)certida.com>; mikedoug--- via Users <users(a)ovirt.org&gt; Subject: [EXT]Re: [ovirt-users] Re: [SOLVED] Re: Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment? CAUTION: Be careful of links - This email originated from outside of Certida. Hi, Recently I found another way to renew the certificates on a one-node self-hosted environment. The trick is to stop vdsmd and wait till the engine shows the system unresponsive. Then you can trigger a certificate renewal and just power on vdsmd again. Best Regards, Strahil Nikolov On Sun, Jan 14, 2024 at 2:54, mikedoug--- via Users <users(a)ovirt.org&gt; wrote: This is the only place we found the answer we needed concerning how to sign a host key from the engine! Saved our bacon! The only tweak I would make is that there were a few more destinations at the end to copy the files into, and instead of rebooting the host we found that just restarting vdsmd and libvirtd got everything working without any existing VMs having to stop. Here's a complete update of what you have above intermingling some extra bits for anyone who has to do this in the future. If the certs on your oVirt host expire, it can be a PITA to figure out how to fix it. It's actually simple, but takes a LOT of manual work. Make sure every part of every script makes sense as I made some modifications as I documented it! I do not warrant that I didn't make a mistake somewhere. :) ########### On the ENGINE HOST ########### ## Check the CA Cert on the ENGINE openssl x509 -in /etc/pki/ovirt-engine/ca.pem -noout -dates ## If it is expired, then on the ENGINE, rebuild the CA Cert with this in a script: ## NOTE: these are steps from another post I didn't need to do them -- could be dangerous... ``` set -x ## Make the script echo everything out, so if it fails you know where set -e ## Make the script STOP on any error set my_date="$(date +"%Y%m%d%H%M%S")" # Backup the existing CA files /bin/cp -p /etc/pki/ovirt-engine/private/ca.pem /etc/pki/ovirt-engine/private/ca.pem.$my_date /bin/cp -p /etc/pki/ovirt-engine/ca.pem{,.$my_date} /bin/mv /etc/pki/ovirt-engine/certs/ca.der{,.$my_date} # Sign the key openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in /etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256 openssl x509 -in /etc/pki/ovirt-engine/ca.pem.new -text > /etc/pki/ovirt-engine/ca.pem.new.full # Put the files into place /bin/mv -f /etc/pki/ovirt-engine/ca.pem.new.full /etc/pki/ovirt-engine/ca.pem /bin/cp -p /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/certs/ca.der ``` Now you need to copy the new CA file over to the host: Source: ENGINE `/etc/pki/ovirt-engine/ca.pem` Dest: HOST `/tmp/new-ca.pem` ########### On the oVirt Host ########### # Create a CSR using the information from the existing certificate and the existing key: openssl x509 -x509toreq -in /etc/pki/libvirt/clientcert.pem -out /tmp/HOST.csr -signkey /etc/pki/libvirt/private/clientkey.pem Now you need to copy the new CA file over to the host: Source: HOST `/tmp/HOST.csr` Dest: ENGINE `/etc/pki/ovirt-engine/requests/full.hostname.com.req` ########### On the ENGINE HOST ########### # Now sign it: /usr/share/ovirt-engine/bin/pki-enroll-request.sh --name=full.hostname.com # NB -- adding --san results in an error: --san=host.na.me (So no Subject Alternate Names) Now you need to copy the new Certificate file over to the host: Source: ENGINE /etc/pki/ovirt-engine/certs/full.hostname.com.cer Dest: HOST /tmp/new-cert.pem ########### On the oVirt Host ########### Run this script to put the cert and CA in place. Note if you don't put a ca into /tmp/new-ca.pem it skips that step. ``` set -x set -e set my_date="$(date +"%Y%m%d%H%M%S")" for x in /etc/pki/libvirt/clientcert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-migrate/server-cert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem /etc/pki/vdsm/libvirt-vnc/server-cert.pem; do /bin/mv -n $x ${x}.${mydate} /bin/cp /tmp/new-cert.pem ${x} chmod 644 ${x} chown root:kvm ${x} done if -f /tmp/new-ca.pem; then for x in /etc/pki/vdsm/libvirt-migrate/ca-cert.pem /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-vnc/ca-cert.pem /etc/pki/vdsm/libvirt-spice/ca-cert.pem /etc/pki/CA/cacert.pem; do /bin/mv -n $x ${x}.${mydate} /bin/cp /tmp/new-ca.pem ${x} chmod 644 ${x} chown root:kvm ${x} done fi ``` Now you're ready to restart two vital services -- some people say "reboot the host" -- but we found that unecessary. Running this restart was safe for us and didn't cause any running VMs to crash or reboot -- they kept going without issue. Once we did this, waited a few minutes, the host came back up on the engine and everything was happy. Specifically a VM we were unable to get running was a click away from full functionality again! # Restart the two services affected by the key/cert changes systemctl restart vdsmd libvirtd _______________________________________________ Users mailing list -- users@ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html<https://url.avanan.click/v2/... oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/<https://ur... List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2EBSBWXFCEA...