Re: [EXT]Re: Re: [SOLVED] Re: Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?
Actually , the vdsmd trick is needed only to fool the engine that the host has failed (you need to disable the fencing or prevent it somehow - like changing the ipmi user's password).The engine itself uses ssh to distribute the new certificate to the host. Best Regards,Strahil Nikolov On Wed, Jan 17, 2024 at 18:07, Michael Douglass<mikedoug@certida.com> wrote: Ah, the steps I list here work for non-single node environments as well. The problem with vdsm-tool doing the certificate renewal in that setup is that the certificate needs to be signed by the engine host and not the local host. Apologies if this causes any confusion for single-node users. From: Strahil Nikolov <hunter86_bg@yahoo.com> Sent: Saturday, January 13, 2024 6:57 PM To: Michael Douglass <mikedoug@certida.com>; mikedoug--- via Users <users@ovirt.org> Subject: [EXT]Re: [ovirt-users] Re: [SOLVED] Re: Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment? CAUTION: Be careful of links - This email originated from outside of Certida.Hi, Recently I found another way to renew the certificates on a one-node self-hosted environment.The trick is to stop vdsmd and wait till the engine shows the system unresponsive. Then you can trigger a certificate renewal and just power on vdsmd again. Best Regards,Strahil Nikolov On Sun, Jan 14, 2024 at 2:54, mikedoug--- via Users<users@ovirt.org> wrote:This is the only place we found the answer we needed concerning how to sign a host key from the engine! Saved our bacon! The only tweak I would make is that there were a few more destinations at the end to copy the files into, and instead of rebooting the host we found that just restarting vdsmd and libvirtd got everything working without any existing VMs having to stop. Here's a complete update of what you have above intermingling some extra bits for anyone who has to do this in the future. If the certs on your oVirt host expire, it can be a PITA to figure out how to fix it. It's actually simple, but takes a LOT of manual work. Make sure every part of every script makes sense as I made some modifications as I documented it! I do not warrant that I didn't make a mistake somewhere. :) ########### On the ENGINE HOST ########### ## Check the CA Cert on the ENGINE openssl x509 -in /etc/pki/ovirt-engine/ca.pem -noout -dates ## If it is expired, then on the ENGINE, rebuild the CA Cert with this in a script: ## NOTE: these are steps from another post I didn't need to do them -- could be dangerous... ``` set -x ## Make the script echo everything out, so if it fails you know where set -e ## Make the script STOP on any error set my_date="$(date +"%Y%m%d%H%M%S")" # Backup the existing CA files /bin/cp -p /etc/pki/ovirt-engine/private/ca.pem /etc/pki/ovirt-engine/private/ca.pem.$my_date /bin/cp -p /etc/pki/ovirt-engine/ca.pem{,.$my_date} /bin/mv /etc/pki/ovirt-engine/certs/ca.der{,.$my_date} # Sign the key openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in /etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256 openssl x509 -in /etc/pki/ovirt-engine/ca.pem.new -text > /etc/pki/ovirt-engine/ca.pem.new.full # Put the files into place /bin/mv -f /etc/pki/ovirt-engine/ca.pem.new.full /etc/pki/ovirt-engine/ca.pem /bin/cp -p /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/certs/ca.der ``` Now you need to copy the new CA file over to the host: Source: ENGINE `/etc/pki/ovirt-engine/ca.pem` Dest: HOST `/tmp/new-ca.pem` ########### On the oVirt Host ########### # Create a CSR using the information from the existing certificate and the existing key: openssl x509 -x509toreq -in /etc/pki/libvirt/clientcert.pem -out /tmp/HOST.csr -signkey /etc/pki/libvirt/private/clientkey.pem Now you need to copy the new CA file over to the host: Source: HOST `/tmp/HOST.csr` Dest: ENGINE `/etc/pki/ovirt-engine/requests/full.hostname.com.req` ########### On the ENGINE HOST ########### # Now sign it: /usr/share/ovirt-engine/bin/pki-enroll-request.sh --name=full.hostname.com # NB -- adding --san results in an error: --san=host.na.me (So no Subject Alternate Names) Now you need to copy the new Certificate file over to the host: Source: ENGINE /etc/pki/ovirt-engine/certs/full.hostname.com.cer Dest: HOST /tmp/new-cert.pem ########### On the oVirt Host ########### Run this script to put the cert and CA in place. Note if you don't put a ca into /tmp/new-ca.pem it skips that step. ``` set -x set -e set my_date="$(date +"%Y%m%d%H%M%S")" for x in /etc/pki/libvirt/clientcert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-migrate/server-cert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem /etc/pki/vdsm/libvirt-vnc/server-cert.pem; do /bin/mv -n $x ${x}.${mydate} /bin/cp /tmp/new-cert.pem ${x} chmod 644 ${x} chown root:kvm ${x} done if -f /tmp/new-ca.pem; then for x in /etc/pki/vdsm/libvirt-migrate/ca-cert.pem /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-vnc/ca-cert.pem /etc/pki/vdsm/libvirt-spice/ca-cert.pem /etc/pki/CA/cacert.pem; do /bin/mv -n $x ${x}.${mydate} /bin/cp /tmp/new-ca.pem ${x} chmod 644 ${x} chown root:kvm ${x} done fi ``` Now you're ready to restart two vital services -- some people say "reboot the host" -- but we found that unecessary. Running this restart was safe for us and didn't cause any running VMs to crash or reboot -- they kept going without issue. Once we did this, waited a few minutes, the host came back up on the engine and everything was happy. Specifically a VM we were unable to get running was a click away from full functionality again! # Restart the two services affected by the key/cert changes systemctl restart vdsmd libvirtd _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2EBSBWXFCEANDO...
Oh, that wasn't something we found easily through the engine UI – our hosts were already "failed" because their certificates had expired. If there had been some easy-to-use button that says "renew/update host certificates" where the engine would use SSH to go update them on its own – that would have been very nice. Though really... The engine can easily see "the certificates the host had are about to or already have expired" – even nicer would be that the engine proactively just go update the host certificates without any intervention. It took a long time for us to google and find this post that showed how to manually sign using the Engine's key. But, in the end, we're all good here and just leaving our notes for others that might suffer the same fate. :) ________________________________ From: Strahil Nikolov <hunter86_bg@yahoo.com> Sent: Thursday, January 18, 2024 2:59 PM To: Michael Douglass <mikedoug@certida.com>; mikedoug--- via Users <users@ovirt.org> Subject: Re: [EXT]Re: [ovirt-users] Re: [SOLVED] Re: Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment? Actually , the vdsmd trick is needed only to fool the engine that the host has failed (you need to disable the fencing or prevent it somehow - like changing the ipmi user's password). The engine itself uses ssh to distribute the new certificate to the host. Best Regards, Strahil Nikolov On Wed, Jan 17, 2024 at 18:07, Michael Douglass <mikedoug@certida.com> wrote: Ah, the steps I list here work for non-single node environments as well. The problem with vdsm-tool doing the certificate renewal in that setup is that the certificate needs to be signed by the engine host and not the local host. Apologies if this causes any confusion for single-node users. ________________________________ From: Strahil Nikolov <hunter86_bg@yahoo.com> Sent: Saturday, January 13, 2024 6:57 PM To: Michael Douglass <mikedoug@certida.com>; mikedoug--- via Users <users@ovirt.org> Subject: [EXT]Re: [ovirt-users] Re: [SOLVED] Re: Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment? CAUTION: Be careful of links - This email originated from outside of Certida. Hi, Recently I found another way to renew the certificates on a one-node self-hosted environment. The trick is to stop vdsmd and wait till the engine shows the system unresponsive. Then you can trigger a certificate renewal and just power on vdsmd again. Best Regards, Strahil Nikolov On Sun, Jan 14, 2024 at 2:54, mikedoug--- via Users <users@ovirt.org> wrote: This is the only place we found the answer we needed concerning how to sign a host key from the engine! Saved our bacon! The only tweak I would make is that there were a few more destinations at the end to copy the files into, and instead of rebooting the host we found that just restarting vdsmd and libvirtd got everything working without any existing VMs having to stop. Here's a complete update of what you have above intermingling some extra bits for anyone who has to do this in the future. If the certs on your oVirt host expire, it can be a PITA to figure out how to fix it. It's actually simple, but takes a LOT of manual work. Make sure every part of every script makes sense as I made some modifications as I documented it! I do not warrant that I didn't make a mistake somewhere. :) ########### On the ENGINE HOST ########### ## Check the CA Cert on the ENGINE openssl x509 -in /etc/pki/ovirt-engine/ca.pem -noout -dates ## If it is expired, then on the ENGINE, rebuild the CA Cert with this in a script: ## NOTE: these are steps from another post I didn't need to do them -- could be dangerous... ``` set -x ## Make the script echo everything out, so if it fails you know where set -e ## Make the script STOP on any error set my_date="$(date +"%Y%m%d%H%M%S")" # Backup the existing CA files /bin/cp -p /etc/pki/ovirt-engine/private/ca.pem /etc/pki/ovirt-engine/private/ca.pem.$my_date /bin/cp -p /etc/pki/ovirt-engine/ca.pem{,.$my_date} /bin/mv /etc/pki/ovirt-engine/certs/ca.der{,.$my_date} # Sign the key openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in /etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256 openssl x509 -in /etc/pki/ovirt-engine/ca.pem.new -text > /etc/pki/ovirt-engine/ca.pem.new.full # Put the files into place /bin/mv -f /etc/pki/ovirt-engine/ca.pem.new.full /etc/pki/ovirt-engine/ca.pem /bin/cp -p /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/certs/ca.der ``` Now you need to copy the new CA file over to the host: Source: ENGINE `/etc/pki/ovirt-engine/ca.pem` Dest: HOST `/tmp/new-ca.pem` ########### On the oVirt Host ########### # Create a CSR using the information from the existing certificate and the existing key: openssl x509 -x509toreq -in /etc/pki/libvirt/clientcert.pem -out /tmp/HOST.csr -signkey /etc/pki/libvirt/private/clientkey.pem Now you need to copy the new CA file over to the host: Source: HOST `/tmp/HOST.csr` Dest: ENGINE `/etc/pki/ovirt-engine/requests/full.hostname.com.req` ########### On the ENGINE HOST ########### # Now sign it: /usr/share/ovirt-engine/bin/pki-enroll-request.sh --name=full.hostname.com # NB -- adding --san results in an error: --san=host.na.me (So no Subject Alternate Names) Now you need to copy the new Certificate file over to the host: Source: ENGINE /etc/pki/ovirt-engine/certs/full.hostname.com.cer Dest: HOST /tmp/new-cert.pem ########### On the oVirt Host ########### Run this script to put the cert and CA in place. Note if you don't put a ca into /tmp/new-ca.pem it skips that step. ``` set -x set -e set my_date="$(date +"%Y%m%d%H%M%S")" for x in /etc/pki/libvirt/clientcert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-migrate/server-cert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem /etc/pki/vdsm/libvirt-vnc/server-cert.pem; do /bin/mv -n $x ${x}.${mydate} /bin/cp /tmp/new-cert.pem ${x} chmod 644 ${x} chown root:kvm ${x} done if -f /tmp/new-ca.pem; then for x in /etc/pki/vdsm/libvirt-migrate/ca-cert.pem /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-vnc/ca-cert.pem /etc/pki/vdsm/libvirt-spice/ca-cert.pem /etc/pki/CA/cacert.pem; do /bin/mv -n $x ${x}.${mydate} /bin/cp /tmp/new-ca.pem ${x} chmod 644 ${x} chown root:kvm ${x} done fi ``` Now you're ready to restart two vital services -- some people say "reboot the host" -- but we found that unecessary. Running this restart was safe for us and didn't cause any running VMs to crash or reboot -- they kept going without issue. Once we did this, waited a few minutes, the host came back up on the engine and everything was happy. Specifically a VM we were unable to get running was a click away from full functionality again! # Restart the two services affected by the key/cert changes systemctl restart vdsmd libvirtd _______________________________________________ Users mailing list -- users@ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html<https://url.avanan.click/v2/___https://www.ovirt.org/privacy-policy.html___.YXAzOmNlcnRpZGE6YTpvOjQ3MTQ2MDVmNTZmNzQxNmI5YjU3NmE5ZTdiYmRmMzUyOjY6OTU1ODoxZDg1ZTdlMTA3ZDg0M2U1MGY0M2Q1NTk1OTc0NzBiMzE5MWEzNjVkM2JmNDEwZGMzYTExMjRmMmE0Zjk2NGZkOmg6VA> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/<https://url.avanan.click/v2/___https://www.ovirt.org/community/about/community-guidelines/___.YXAzOmNlcnRpZGE6YTpvOjQ3MTQ2MDVmNTZmNzQxNmI5YjU3NmE5ZTdiYmRmMzUyOjY6OTI4NjpkOTk3MzA1YThkNThlYTBhMzIyOGRhY2VjMjM2ZTFjNDcwOTk2YWQ2MzdmYWJhYzAwMDM2MjQ0ZDIxYjhlMTU0Omg6VA> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2EBSBWXFCEANDOH3ZLHQENVTAMVDFMOE/<https://url.avanan.click/v2/___https://lists.ovirt.org/archives/list/users@ovirt.org/message/2EBSBWXFCEANDOH3ZLHQENVTAMVDFMOE/___.YXAzOmNlcnRpZGE6YTpvOjQ3MTQ2MDVmNTZmNzQxNmI5YjU3NmE5ZTdiYmRmMzUyOjY6NGIxYTozZDVmZGE1YTJhOTM2NTRjZTc4ZDgxMzE2ZDFmZDg3NjQzNTdmMzhkNmRkYTA4Mjg4MWE2Njk0OGM3MWY1YjAyOmg6VA>
participants (2)
-
Michael Douglass -
Strahil Nikolov