Re: [EXT]Re: Re: [SOLVED] Re: Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?
11:59 p.m.
Actually , the vdsmd trick is needed only to fool the engine that the host has failed (you
need to disable the fencing or prevent it somehow - like changing the ipmi user's
password).The engine itself uses ssh to distribute the new certificate to the host.
Best Regards,Strahil Nikolov
On Wed, Jan 17, 2024 at 18:07, Michael Douglass<mikedoug(a)certida.com> wrote: Ah,
the steps I list here work for non-single node environments as well. The problem with
vdsm-tool doing the certificate renewal in that setup is that the certificate needs to be
signed by the engine host and not the local host. Apologies if this causes any confusion
for single-node users.
From: Strahil Nikolov <hunter86_bg(a)yahoo.com>
Sent: Saturday, January 13, 2024 6:57 PM
To: Michael Douglass <mikedoug(a)certida.com>; mikedoug--- via Users
<users(a)ovirt.org>
Subject: [EXT]Re: [ovirt-users] Re: [SOLVED] Re: Re: How to re-enroll (or renew) host
certificates for a single-host hosted-engine deployment? CAUTION: Be careful of links -
This email originated from outside of Certida.Hi,
Recently I found another way to renew the certificates on a one-node self-hosted
environment.The trick is to stop vdsmd and wait till the engine shows the system
unresponsive. Then you can trigger a certificate renewal and just power on vdsmd again.
Best Regards,Strahil Nikolov
On Sun, Jan 14, 2024 at 2:54, mikedoug--- via Users<users(a)ovirt.org> wrote:This is
the only place we found the answer we needed concerning how to sign a host key from the
engine! Saved our bacon!
The only tweak I would make is that there were a few more destinations at the end to copy
the files into, and instead of rebooting the host we found that just restarting vdsmd and
libvirtd got everything working without any existing VMs having to stop. Here's a
complete update of what you have above intermingling some extra bits for anyone who has to
do this in the future.
If the certs on your oVirt host expire, it can be a PITA to figure out how to fix it.
It's actually simple, but takes a LOT of manual work.
Make sure every part of every script makes sense as I made some modifications as I
documented it! I do not warrant that I didn't make a mistake somewhere. :)
########### On the ENGINE HOST ###########
## Check the CA Cert on the ENGINE
openssl x509 -in /etc/pki/ovirt-engine/ca.pem -noout -dates
## If it is expired, then on the ENGINE, rebuild the CA Cert with this in a script:
## NOTE: these are steps from another post I didn't need to do them -- could be
dangerous...
```
set -x ## Make the script echo everything out, so if it fails you know where
set -e ## Make the script STOP on any error
set my_date="$(date +"%Y%m%d%H%M%S")"
# Backup the existing CA files
/bin/cp -p /etc/pki/ovirt-engine/private/ca.pem
/etc/pki/ovirt-engine/private/ca.pem.$my_date
/bin/cp -p /etc/pki/ovirt-engine/ca.pem{,.$my_date}
/bin/mv /etc/pki/ovirt-engine/certs/ca.der{,.$my_date}
# Sign the key
openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in
/etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256
openssl x509 -in /etc/pki/ovirt-engine/ca.pem.new -text >
/etc/pki/ovirt-engine/ca.pem.new.full
# Put the files into place
/bin/mv -f /etc/pki/ovirt-engine/ca.pem.new.full /etc/pki/ovirt-engine/ca.pem
/bin/cp -p /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/certs/ca.der
```
Now you need to copy the new CA file over to the host:
Source: ENGINE `/etc/pki/ovirt-engine/ca.pem`
Dest: HOST `/tmp/new-ca.pem`
########### On the oVirt Host ###########
# Create a CSR using the information from the existing certificate and the existing key:
openssl x509 -x509toreq -in /etc/pki/libvirt/clientcert.pem -out /tmp/HOST.csr -signkey
/etc/pki/libvirt/private/clientkey.pem
Now you need to copy the new CA file over to the host:
Source: HOST `/tmp/HOST.csr`
Dest: ENGINE `/etc/pki/ovirt-engine/requests/full.hostname.com.req`
########### On the ENGINE HOST ###########
# Now sign it:
/usr/share/ovirt-engine/bin/pki-enroll-request.sh --name=full.hostname.com
# NB -- adding --san results in an error: --san=host.na.me (So no Subject Alternate
Names)
Now you need to copy the new Certificate file over to the host:
Source: ENGINE /etc/pki/ovirt-engine/certs/full.hostname.com.cer
Dest: HOST /tmp/new-cert.pem
########### On the oVirt Host ###########
Run this script to put the cert and CA in place. Note if you don't put a ca into
/tmp/new-ca.pem it skips that step.
```
set -x
set -e
set my_date="$(date +"%Y%m%d%H%M%S")"
for x in /etc/pki/libvirt/clientcert.pem /etc/pki/vdsm/certs/vdsmcert.pem
/etc/pki/vdsm/libvirt-migrate/server-cert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem
/etc/pki/vdsm/libvirt-vnc/server-cert.pem; do
/bin/mv -n $x ${x}.${mydate}
/bin/cp /tmp/new-cert.pem ${x}
chmod 644 ${x}
chown root:kvm ${x}
done
if -f /tmp/new-ca.pem; then
for x in /etc/pki/vdsm/libvirt-migrate/ca-cert.pem /etc/pki/vdsm/certs/cacert.pem
/etc/pki/vdsm/libvirt-vnc/ca-cert.pem /etc/pki/vdsm/libvirt-spice/ca-cert.pem
/etc/pki/CA/cacert.pem; do
/bin/mv -n $x ${x}.${mydate}
/bin/cp /tmp/new-ca.pem ${x}
chmod 644 ${x}
chown root:kvm ${x}
done
fi
```
Now you're ready to restart two vital services -- some people say "reboot the
host" -- but we found that unecessary. Running this restart was safe for us and
didn't cause any running VMs to crash or reboot -- they kept going without issue.
Once we did this, waited a few minutes, the host came back up on the engine and everything
was happy. Specifically a VM we were unable to get running was a click away from full
functionality again!
# Restart the two services affected by the key/cert changes
systemctl restart vdsmd libvirtd
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/2EBSBWXFCEA...
12:04 a.m.
New subject: [EXT]Re: Re: [SOLVED] Re: Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?
Oh, that wasn't something we found easily through the engine UI – our hosts were
already "failed" because their certificates had expired. If there had been some
easy-to-use button that says "renew/update host certificates" where the engine
would use SSH to go update them on its own – that would have been very nice. Though
really... The engine can easily see "the certificates the host had are about to or
already have expired" – even nicer would be that the engine proactively just go
update the host certificates without any intervention.
It took a long time for us to google and find this post that showed how to manually sign
using the Engine's key. But, in the end, we're all good here and just leaving our
notes for others that might suffer the same fate. :)
________________________________
From: Strahil Nikolov <hunter86_bg(a)yahoo.com>
Sent: Thursday, January 18, 2024 2:59 PM
To: Michael Douglass <mikedoug(a)certida.com>; mikedoug--- via Users
<users(a)ovirt.org>
Subject: Re: [EXT]Re: [ovirt-users] Re: [SOLVED] Re: Re: How to re-enroll (or renew) host
certificates for a single-host hosted-engine deployment?
Actually , the vdsmd trick is needed only to fool the engine that the host has failed (you
need to disable the fencing or prevent it somehow - like changing the ipmi user's
password).
The engine itself uses ssh to distribute the new certificate to the host.
Best Regards,
Strahil Nikolov
On Wed, Jan 17, 2024 at 18:07, Michael Douglass
<mikedoug(a)certida.com> wrote:
Ah, the steps I list here work for non-single node environments as well. The problem with
vdsm-tool doing the certificate renewal in that setup is that the certificate needs to be
signed by the engine host and not the local host. Apologies if this causes any confusion
for single-node users.
________________________________
From: Strahil Nikolov <hunter86_bg(a)yahoo.com>
Sent: Saturday, January 13, 2024 6:57 PM
To: Michael Douglass <mikedoug(a)certida.com>; mikedoug--- via Users
<users(a)ovirt.org>
Subject: [EXT]Re: [ovirt-users] Re: [SOLVED] Re: Re: How to re-enroll (or renew) host
certificates for a single-host hosted-engine deployment?
CAUTION: Be careful of links - This email originated from outside of Certida.
Hi,
Recently I found another way to renew the certificates on a one-node self-hosted
environment.
The trick is to stop vdsmd and wait till the engine shows the system unresponsive. Then
you can trigger a certificate renewal and just power on vdsmd again.
Best Regards,
Strahil Nikolov
On Sun, Jan 14, 2024 at 2:54, mikedoug--- via Users
<users(a)ovirt.org> wrote:
This is the only place we found the answer we needed concerning how to sign a host key
from the engine! Saved our bacon!
The only tweak I would make is that there were a few more destinations at the end to copy
the files into, and instead of rebooting the host we found that just restarting vdsmd and
libvirtd got everything working without any existing VMs having to stop. Here's a
complete update of what you have above intermingling some extra bits for anyone who has to
do this in the future.
If the certs on your oVirt host expire, it can be a PITA to figure out how to fix it.
It's actually simple, but takes a LOT of manual work.
Make sure every part of every script makes sense as I made some modifications as I
documented it! I do not warrant that I didn't make a mistake somewhere. :)
########### On the ENGINE HOST ###########
## Check the CA Cert on the ENGINE
openssl x509 -in /etc/pki/ovirt-engine/ca.pem -noout -dates
## If it is expired, then on the ENGINE, rebuild the CA Cert with this in a script:
## NOTE: these are steps from another post I didn't need to do them -- could be
dangerous...
```
set -x ## Make the script echo everything out, so if it fails you know where
set -e ## Make the script STOP on any error
set my_date="$(date +"%Y%m%d%H%M%S")"
# Backup the existing CA files
/bin/cp -p /etc/pki/ovirt-engine/private/ca.pem
/etc/pki/ovirt-engine/private/ca.pem.$my_date
/bin/cp -p /etc/pki/ovirt-engine/ca.pem{,.$my_date}
/bin/mv /etc/pki/ovirt-engine/certs/ca.der{,.$my_date}
# Sign the key
openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in
/etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256
openssl x509 -in /etc/pki/ovirt-engine/ca.pem.new -text >
/etc/pki/ovirt-engine/ca.pem.new.full
# Put the files into place
/bin/mv -f /etc/pki/ovirt-engine/ca.pem.new.full /etc/pki/ovirt-engine/ca.pem
/bin/cp -p /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/certs/ca.der
```
Now you need to copy the new CA file over to the host:
Source: ENGINE `/etc/pki/ovirt-engine/ca.pem`
Dest: HOST `/tmp/new-ca.pem`
########### On the oVirt Host ###########
# Create a CSR using the information from the existing certificate and the existing key:
openssl x509 -x509toreq -in /etc/pki/libvirt/clientcert.pem -out /tmp/HOST.csr -signkey
/etc/pki/libvirt/private/clientkey.pem
Now you need to copy the new CA file over to the host:
Source: HOST `/tmp/HOST.csr`
Dest: ENGINE `/etc/pki/ovirt-engine/requests/full.hostname.com.req`
########### On the ENGINE HOST ###########
# Now sign it:
/usr/share/ovirt-engine/bin/pki-enroll-request.sh --name=full.hostname.com
# NB -- adding --san results in an error: --san=host.na.me (So no Subject Alternate
Names)
Now you need to copy the new Certificate file over to the host:
Source: ENGINE /etc/pki/ovirt-engine/certs/full.hostname.com.cer
Dest: HOST /tmp/new-cert.pem
########### On the oVirt Host ###########
Run this script to put the cert and CA in place. Note if you don't put a ca into
/tmp/new-ca.pem it skips that step.
```
set -x
set -e
set my_date="$(date +"%Y%m%d%H%M%S")"
for x in /etc/pki/libvirt/clientcert.pem /etc/pki/vdsm/certs/vdsmcert.pem
/etc/pki/vdsm/libvirt-migrate/server-cert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem
/etc/pki/vdsm/libvirt-vnc/server-cert.pem; do
/bin/mv -n $x ${x}.${mydate}
/bin/cp /tmp/new-cert.pem ${x}
chmod 644 ${x}
chown root:kvm ${x}
done
if -f /tmp/new-ca.pem; then
for x in /etc/pki/vdsm/libvirt-migrate/ca-cert.pem /etc/pki/vdsm/certs/cacert.pem
/etc/pki/vdsm/libvirt-vnc/ca-cert.pem /etc/pki/vdsm/libvirt-spice/ca-cert.pem
/etc/pki/CA/cacert.pem; do
/bin/mv -n $x ${x}.${mydate}
/bin/cp /tmp/new-ca.pem ${x}
chmod 644 ${x}
chown root:kvm ${x}
done
fi
```
Now you're ready to restart two vital services -- some people say "reboot the
host" -- but we found that unecessary. Running this restart was safe for us and
didn't cause any running VMs to crash or reboot -- they kept going without issue.
Once we did this, waited a few minutes, the host came back up on the engine and everything
was happy. Specifically a VM we were unable to get running was a click away from full
functionality again!
# Restart the two services affected by the key/cert changes
systemctl restart vdsmd libvirtd
_______________________________________________
Users mailing list -- users@ovirt.org<mailto:users@ovirt.org>
To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org>
Privacy Statement:
https://www.ovirt.org/privacy-policy.html<https://url.avanan.click/v2/...
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/<https://ur...
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/2EBSBWXFCEA...
123
days inactive
123
days old
1 comments
2 participants
participants (2)
-
Michael Douglass -
Strahil Nikolov