How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?
Hi, I've got a single-host hosted-engine deployment that I originally installed with 4.0 and have upgraded over the years to 4.3.10. I and some of my users have upgraded remote-viewer and now I get an error when I try to view the console of my VMs: (remote-viewer:8252): Spice-WARNING **: 11:30:41.806: ../subprojects/spice-common/common/ssl_verify.c:477:openssl_verify: Error in server certificate verification: CA signature digest algorithm too weak (num=68:depth0:/O=<My Org Name>/CN=<Host's Name>) I am 99.99% sure this is because the old certs use SHA1. I reran engine-setup on the engine and it asked me if I wanted to renew the PKI, and I answered yes. This replaced many[1] of the certificates in /etc/pki/ovirt-engine/certs on the engine, but it did not update the Host's certificate. All the documentation I've seen says that to refresh this certificate I need to put the host into maintenance mode and then re-enroll.. However I cannot do that, because this is a single-host system so I cannot put the host in local mode -- there is no place to migrate the VMs (let alone the Engine VM). So.... Is there a command-line way to re-enroll manually and update the host certs? Or some other way to get all the leftover certs renewed? Thanks, -derek [1] Not only did it not update the Host's cert, it did not update any of the vmconsole-proxy certs, nor the certs in /etc/pki/ovirt-vmconsole/, and obviously nothing in /etc/pki/ on the host itself. -- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant
On Sun, Dec 6, 2020 at 12:34 AM Derek Atkins <derek@ihtfp.com> wrote:
Hi,
I've got a single-host hosted-engine deployment that I originally installed with 4.0 and have upgraded over the years to 4.3.10. I and some of my users have upgraded remote-viewer and now I get an error when I try to view the console of my VMs:
(remote-viewer:8252): Spice-WARNING **: 11:30:41.806: ../subprojects/spice-common/common/ssl_verify.c:477:openssl_verify: Error in server certificate verification: CA signature digest algorithm too weak (num=68:depth0:/O=<My Org Name>/CN=<Host's Name>)
I am 99.99% sure this is because the old certs use SHA1.
I reran engine-setup on the engine and it asked me if I wanted to renew the PKI, and I answered yes. This replaced many[1] of the certificates in /etc/pki/ovirt-engine/certs on the engine, but it did not update the Host's certificate.
Indeed.
All the documentation I've seen says that to refresh this certificate I need to put the host into maintenance mode and then re-enroll.. However I cannot do that, because this is a single-host system so I cannot put the host in local mode -- there is no place to migrate the VMs (let alone the Engine VM).
So.... Is there a command-line way to re-enroll manually and update the host certs?
I don't think you'll find anything like this. People did come up in the past with various procedure to hack pki like what you want, but these are, generally speaking, quite fragile - usually do not get updated over versions etc. I am pretty certain the only way to do this using "official" tools/docs is: 1. Stop all VMs except for the engine one. 2. Take a backup with engine-backup. 3. Stop the engine VM. 4. Reinstall the host OS from scratch or use ovirt-hosted-engine-cleanup. 5. Provision the host again as a hosted-engine host, using '--restore-from-file'. Either using new storage for the engine, or after cleaning up the existing hosted-engine storage. If you still want to try doing this manually, then the tool to use is pki-enroll-request.sh. IIRC it's documented. You should find what keys/certs you want to replace, generate new keys and CSRs (or use existing keys and generate CSRs, or even use existing CSRs if you find them), copy to the engine, sign with pki-enroll-request.sh, then copy the generated cert to the host. I am almost certain there is no way to tell vdsm (and other processes) to reload the certs, so you'll have to restart it (them) - and this usually requires putting the host in maintenance (and therefore stop (migrate) all VMs).
Or some other way to get all the leftover certs renewed?
Which ones, specifically?
Thanks,
-derek
[1] Not only did it not update the Host's cert, it did not update any of the vmconsole-proxy certs, nor the certs in /etc/pki/ovirt-vmconsole/, and obviously nothing in /etc/pki/ on the host itself.
AFAIR no process uses these certs as such. There are only processes that use the ssh-format keys extracted from them, which do not include a signature (sha1 or whatever). If you think I am wrong, and/or notice other certs that need to be regenerated, that's a bug - please open one. Thanks! Re remote-viewer/spice: You didn't say if you tried again after engine-setup and what happened. In any case, this is unrelated to vmconsole (which is for serial consoles, using ssh). But you might still need to regenerate the host cert. BTW: You can try using novnc and websocket-proxy - engine-setup does update the cert for the latter, so this might work as-is. Best regards, -- Didi
HI, On Sun, December 6, 2020 7:44 am, Yedidyah Bar David wrote:
On Sun, Dec 6, 2020 at 12:34 AM Derek Atkins <derek@ihtfp.com> wrote: [snip]
So.... Is there a command-line way to re-enroll manually and update the host certs?
I don't think you'll find anything like this.
People did come up in the past with various procedure to hack pki like what you want, but these are, generally speaking, quite fragile - usually do not get updated over versions etc.
I am pretty certain the only way to do this using "official" tools/docs is:
1. Stop all VMs except for the engine one.
2. Take a backup with engine-backup.
3. Stop the engine VM.
4. Reinstall the host OS from scratch or use ovirt-hosted-engine-cleanup.
5. Provision the host again as a hosted-engine host, using '--restore-from-file'. Either using new storage for the engine, or after cleaning up the existing hosted-engine storage.
If I were to go this route I might as well upgrade to EL8 / 4.4 at the same time. However, I would rather not do that; I consider that a very dangerous operation, with a generally too-high probability of failure.
If you still want to try doing this manually, then the tool to use is pki-enroll-request.sh. IIRC it's documented. You should find what keys/certs you want to replace, generate new keys and CSRs (or use existing keys and generate CSRs, or even use existing CSRs if you find them), copy to the engine, sign with pki-enroll-request.sh, then copy the generated cert to the host.
Thanks. I will look into this method.
I am almost certain there is no way to tell vdsm (and other processes) to reload the certs, so you'll have to restart it (them) - and this usually requires putting the host in maintenance (and therefore stop (migrate) all VMs).
I don't mind stopping the VMs in order to reboot the host if I can plan that. My understanding is that because there is no place to migrate the hosted-engine, that implies even I stop all the other VMs, I still cannot put the host into maintenance mode. Is my understanding correct?
Or some other way to get all the leftover certs renewed?
Which ones, specifically?
I think I listed them all: <host>*.cer and vmconsole*.cer on the engine, and of course everything on the host itself. Does it matter that ca.der didn't change? I don't know if that is a self-signed cert that might be problematic?
Thanks,
-derek
[1] Not only did it not update the Host's cert, it did not update any of the vmconsole-proxy certs, nor the certs in /etc/pki/ovirt-vmconsole/, and obviously nothing in /etc/pki/ on the host itself.
AFAIR no process uses these certs as such. There are only processes that use the ssh-format keys extracted from them, which do not include a signature (sha1 or whatever).
If you think I am wrong, and/or notice other certs that need to be regenerated, that's a bug - please open one. Thanks!
I have not noticed anything, yet, but I have not restarted the host or vdsm since I re-ran engine-setup.
Re remote-viewer/spice: You didn't say if you tried again after engine-setup and what happened. In any case, this is unrelated to vmconsole (which is for serial consoles, using ssh). But you might still need to regenerate the host cert.
Sorry, I thought I did. Yes, I did try re-running remote-viewer after running engine-setup. There was no change in the console.vv file (except of course for the password and sso-token), so yes, it failed in the same way. Note, however, that I did not restart vdsm or the host after running engine-setup.
BTW: You can try using novnc and websocket-proxy - engine-setup does update the cert for the latter, so this might work as-is.
Yes, that does work indeed, so as a short-term solution that can work for me. I'll ask my colleague on a Mac if that works for him. But it would be nice to get remote-viewer working, IMHO, which would require a way to renew / refresh the host cert -- which of course would be nice to do without having to re-install! Thanks!!!
Best regards, -- Didi
-derek -- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant
Hi Didi, One more question: Can you verify that etc/pki/libvirt/clientcert.pem, etc/pki/vdsm/certs/vdsmcert.pem, and etc/pki/vdsm/libvirt-spice/server-cert.pem are all supposed to be same certificate (on the host)? By a quick find | grep all three of these files appear to be the <Host>.cer certificate file? -derek On Sun, December 6, 2020 12:25 pm, Derek Atkins wrote:
HI,
On Sun, December 6, 2020 7:44 am, Yedidyah Bar David wrote:
On Sun, Dec 6, 2020 at 12:34 AM Derek Atkins <derek@ihtfp.com> wrote: [snip]
So.... Is there a command-line way to re-enroll manually and update the host certs?
I don't think you'll find anything like this.
People did come up in the past with various procedure to hack pki like what you want, but these are, generally speaking, quite fragile - usually do not get updated over versions etc.
I am pretty certain the only way to do this using "official" tools/docs is:
1. Stop all VMs except for the engine one.
2. Take a backup with engine-backup.
3. Stop the engine VM.
4. Reinstall the host OS from scratch or use ovirt-hosted-engine-cleanup.
5. Provision the host again as a hosted-engine host, using '--restore-from-file'. Either using new storage for the engine, or after cleaning up the existing hosted-engine storage.
If I were to go this route I might as well upgrade to EL8 / 4.4 at the same time. However, I would rather not do that; I consider that a very dangerous operation, with a generally too-high probability of failure.
If you still want to try doing this manually, then the tool to use is pki-enroll-request.sh. IIRC it's documented. You should find what keys/certs you want to replace, generate new keys and CSRs (or use existing keys and generate CSRs, or even use existing CSRs if you find them), copy to the engine, sign with pki-enroll-request.sh, then copy the generated cert to the host.
Thanks. I will look into this method.
I am almost certain there is no way to tell vdsm (and other processes) to reload the certs, so you'll have to restart it (them) - and this usually requires putting the host in maintenance (and therefore stop (migrate) all VMs).
I don't mind stopping the VMs in order to reboot the host if I can plan that. My understanding is that because there is no place to migrate the hosted-engine, that implies even I stop all the other VMs, I still cannot put the host into maintenance mode. Is my understanding correct?
Or some other way to get all the leftover certs renewed?
Which ones, specifically?
I think I listed them all: <host>*.cer and vmconsole*.cer on the engine, and of course everything on the host itself.
Does it matter that ca.der didn't change? I don't know if that is a self-signed cert that might be problematic?
Thanks,
-derek
[1] Not only did it not update the Host's cert, it did not update any of the vmconsole-proxy certs, nor the certs in /etc/pki/ovirt-vmconsole/, and obviously nothing in /etc/pki/ on the host itself.
AFAIR no process uses these certs as such. There are only processes that use the ssh-format keys extracted from them, which do not include a signature (sha1 or whatever).
If you think I am wrong, and/or notice other certs that need to be regenerated, that's a bug - please open one. Thanks!
I have not noticed anything, yet, but I have not restarted the host or vdsm since I re-ran engine-setup.
Re remote-viewer/spice: You didn't say if you tried again after engine-setup and what happened. In any case, this is unrelated to vmconsole (which is for serial consoles, using ssh). But you might still need to regenerate the host cert.
Sorry, I thought I did. Yes, I did try re-running remote-viewer after running engine-setup. There was no change in the console.vv file (except of course for the password and sso-token), so yes, it failed in the same way.
Note, however, that I did not restart vdsm or the host after running engine-setup.
BTW: You can try using novnc and websocket-proxy - engine-setup does update the cert for the latter, so this might work as-is.
Yes, that does work indeed, so as a short-term solution that can work for me. I'll ask my colleague on a Mac if that works for him.
But it would be nice to get remote-viewer working, IMHO, which would require a way to renew / refresh the host cert -- which of course would be nice to do without having to re-install!
Thanks!!!
Best regards, -- Didi
-derek
-- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant
-- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant
On Sun, Dec 6, 2020 at 7:49 PM Derek Atkins <derek@ihtfp.com> wrote:
Hi Didi,
One more question:
Can you verify that etc/pki/libvirt/clientcert.pem, etc/pki/vdsm/certs/vdsmcert.pem, and etc/pki/vdsm/libvirt-spice/server-cert.pem are all supposed to be same certificate (on the host)? By a quick find | grep all three of these files appear to be the <Host>.cer certificate file?
Yes, and also vdsm/libvirt-vnc/server-cert.pem . -- Didi
On Sun, Dec 6, 2020 at 7:25 PM Derek Atkins <derek@ihtfp.com> wrote:
HI,
On Sun, December 6, 2020 7:44 am, Yedidyah Bar David wrote:
On Sun, Dec 6, 2020 at 12:34 AM Derek Atkins <derek@ihtfp.com> wrote: [snip]
So.... Is there a command-line way to re-enroll manually and update the host certs?
I don't think you'll find anything like this.
People did come up in the past with various procedure to hack pki like what you want, but these are, generally speaking, quite fragile - usually do not get updated over versions etc.
I am pretty certain the only way to do this using "official" tools/docs is:
1. Stop all VMs except for the engine one.
2. Take a backup with engine-backup.
3. Stop the engine VM.
4. Reinstall the host OS from scratch or use ovirt-hosted-engine-cleanup.
5. Provision the host again as a hosted-engine host, using '--restore-from-file'. Either using new storage for the engine, or after cleaning up the existing hosted-engine storage.
If I were to go this route I might as well upgrade to EL8 / 4.4 at the same time. However, I would rather not do that; I consider that a very dangerous operation, with a generally too-high probability of failure.
I assume you'll do that anyway, some time later, no? So why not now? I think 4.4.3 is a pretty good version. I agree it's "dangerous" in the sense that it involves lots of rather complex actions, some of which are hard to debug/fix if they fail. But: You can plan and test ahead on a test machine somewhere - even a VM, with nested-kvm. Just make sure it has no access to your host and storage (network-wise), so that it does not try to manage/access them.
If you still want to try doing this manually, then the tool to use is pki-enroll-request.sh. IIRC it's documented. You should find what keys/certs you want to replace, generate new keys and CSRs (or use existing keys and generate CSRs, or even use existing CSRs if you find them), copy to the engine, sign with pki-enroll-request.sh, then copy the generated cert to the host.
Thanks. I will look into this method.
I am almost certain there is no way to tell vdsm (and other processes) to reload the certs, so you'll have to restart it (them) - and this usually requires putting the host in maintenance (and therefore stop (migrate) all VMs).
I don't mind stopping the VMs in order to reboot the host if I can plan that. My understanding is that because there is no place to migrate the hosted-engine, that implies even I stop all the other VMs, I still cannot put the host into maintenance mode. Is my understanding correct?
You are right - for a single host, you should do something like this: 1. Stop all VMs except for the engine 2. Put it to global maintenance 3. Stop the engine VM ('poweroff' from inside it, or 'hosted-engine --vm-shutdown') 4. Restart whatever services you want, or just reboot 5. Exit global maintenance 6. Engine VM should be started automatically in a few minutes. If it does not, you can 'hosted-engine --vm-start'. You can monitor agent.log in the meanwhile.
Or some other way to get all the leftover certs renewed?
Which ones, specifically?
I think I listed them all: <host>*.cer and vmconsole*.cer on the engine, and of course everything on the host itself.
engine-setup is not designed to touch hosts. Hosts should be managed by the engine, usually.
Does it matter that ca.der didn't change? I don't know if that is a self-signed cert that might be problematic?
ca.der is not used by anything, you can ignore it. The private key of the CA is in /etc/pki/ovirt-engine/private/ca.pem, and the public key is in /etc/pki/ovirt-engine/ca.pem. That's what all tools use.
Thanks,
-derek
[1] Not only did it not update the Host's cert, it did not update any of the vmconsole-proxy certs, nor the certs in /etc/pki/ovirt-vmconsole/, and obviously nothing in /etc/pki/ on the host itself.
AFAIR no process uses these certs as such. There are only processes that use the ssh-format keys extracted from them, which do not include a signature (sha1 or whatever).
If you think I am wrong, and/or notice other certs that need to be regenerated, that's a bug - please open one. Thanks!
I have not noticed anything, yet, but I have not restarted the host or vdsm since I re-ran engine-setup.
Re remote-viewer/spice: You didn't say if you tried again after engine-setup and what happened. In any case, this is unrelated to vmconsole (which is for serial consoles, using ssh). But you might still need to regenerate the host cert.
Sorry, I thought I did. Yes, I did try re-running remote-viewer after running engine-setup. There was no change in the console.vv file (except of course for the password and sso-token), so yes, it failed in the same way.
Note, however, that I did not restart vdsm or the host after running engine-setup.
OK.
BTW: You can try using novnc and websocket-proxy - engine-setup does update the cert for the latter, so this might work as-is.
Yes, that does work indeed, so as a short-term solution that can work for me. I'll ask my colleague on a Mac if that works for him.
But it would be nice to get remote-viewer working, IMHO, which would require a way to renew / refresh the host cert -- which of course would be nice to do without having to re-install!
I agree it would be nice, but I think it would require quite a lot of work. Generally speaking, the project considers the "standard" use case to be a setup of at least two hosts, and at least one host "extra" (in terms of capacity), so that if a host fails, you can still keep everything up. In that regard, a single-host setup is considered a kind of "corner case", meant mainly for testing/development, not production. Is there such a big advantage in using oVirt for a single host, compared to virt-manager? Good luck and best regards, -- Didi
Hi again, I also noticed that ca.pem was not updated -- it's still using Sha1. I don't know if this will be an issue with remote-viewer if I wind up refreshing the host cert? -derek On Sun, December 6, 2020 7:44 am, Yedidyah Bar David wrote:
On Sun, Dec 6, 2020 at 12:34 AM Derek Atkins <derek@ihtfp.com> wrote:
Hi,
I've got a single-host hosted-engine deployment that I originally installed with 4.0 and have upgraded over the years to 4.3.10. I and some of my users have upgraded remote-viewer and now I get an error when I try to view the console of my VMs:
(remote-viewer:8252): Spice-WARNING **: 11:30:41.806: ../subprojects/spice-common/common/ssl_verify.c:477:openssl_verify: Error in server certificate verification: CA signature digest algorithm too weak (num=68:depth0:/O=<My Org Name>/CN=<Host's Name>)
I am 99.99% sure this is because the old certs use SHA1.
I reran engine-setup on the engine and it asked me if I wanted to renew the PKI, and I answered yes. This replaced many[1] of the certificates in /etc/pki/ovirt-engine/certs on the engine, but it did not update the Host's certificate.
Indeed.
All the documentation I've seen says that to refresh this certificate I need to put the host into maintenance mode and then re-enroll.. However I cannot do that, because this is a single-host system so I cannot put the host in local mode -- there is no place to migrate the VMs (let alone the Engine VM).
So.... Is there a command-line way to re-enroll manually and update the host certs?
I don't think you'll find anything like this.
People did come up in the past with various procedure to hack pki like what you want, but these are, generally speaking, quite fragile - usually do not get updated over versions etc.
I am pretty certain the only way to do this using "official" tools/docs is:
1. Stop all VMs except for the engine one.
2. Take a backup with engine-backup.
3. Stop the engine VM.
4. Reinstall the host OS from scratch or use ovirt-hosted-engine-cleanup.
5. Provision the host again as a hosted-engine host, using '--restore-from-file'. Either using new storage for the engine, or after cleaning up the existing hosted-engine storage.
If you still want to try doing this manually, then the tool to use is pki-enroll-request.sh. IIRC it's documented. You should find what keys/certs you want to replace, generate new keys and CSRs (or use existing keys and generate CSRs, or even use existing CSRs if you find them), copy to the engine, sign with pki-enroll-request.sh, then copy the generated cert to the host. I am almost certain there is no way to tell vdsm (and other processes) to reload the certs, so you'll have to restart it (them) - and this usually requires putting the host in maintenance (and therefore stop (migrate) all VMs).
Or some other way to get all the leftover certs renewed?
Which ones, specifically?
Thanks,
-derek
[1] Not only did it not update the Host's cert, it did not update any of the vmconsole-proxy certs, nor the certs in /etc/pki/ovirt-vmconsole/, and obviously nothing in /etc/pki/ on the host itself.
AFAIR no process uses these certs as such. There are only processes that use the ssh-format keys extracted from them, which do not include a signature (sha1 or whatever).
If you think I am wrong, and/or notice other certs that need to be regenerated, that's a bug - please open one. Thanks!
Re remote-viewer/spice: You didn't say if you tried again after engine-setup and what happened. In any case, this is unrelated to vmconsole (which is for serial consoles, using ssh). But you might still need to regenerate the host cert.
BTW: You can try using novnc and websocket-proxy - engine-setup does update the cert for the latter, so this might work as-is.
Best regards, -- Didi
-- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant
On Sun, Dec 6, 2020 at 8:14 PM Derek Atkins <derek@ihtfp.com> wrote:
Hi again,
I also noticed that ca.pem was not updated -- it's still using Sha1.
You are right - we didn't make engine-setup recreate existing certs for this - "Renew" deals with other stuff [1]. We only change the default for new ones [2], and wrote a procedure [3][4] for doing this manually. At the time, this wasn't mandatory - browsers didn't reject sha1. Perhaps now it should be. [1] https://www.ovirt.org/develop/release-management/features/infra/pki-renew.ht... [2] https://gerrit.ovirt.org/c/ovirt-engine/+/65927 (I see that I even didn't open a bug for this at the time, not sure why) [3] https://www.ovirt.org/documentation/upgrade_guide/#Replacing_SHA-1_Certifica... [4] https://bugzilla.redhat.com/show_bug.cgi?id=1420577 (This is a RHV doc bug, and the result was an update to RHV docs, which eventually were also published as [3] above)
I don't know if this will be an issue with remote-viewer if I wind up refreshing the host cert?
As I said, at the time it didn't seem to be mandatory, and docs seemed to be enough. If you feel otherwise, please open a bug. I think there is a difference, or at least there was, between what browsers did/do with https certs, and what they did with CA certs. If you had a CA cert already accepted/imported/trusted by the browser, and then you entered a site with a cert signed by this CA, but with a SHA1 signature, this was one separate case. Browsers started warning/ rejecting them earlier. If you have a CA cert with a SHA1 signature, and want to import that to a browser, that's another case. I didn't test recently (or much over time, other than working on these bugs) with recent browsers, but I think it took longer until they rejected (if indeed they do - not sure all of them do). Best regards,
-derek
On Sun, December 6, 2020 7:44 am, Yedidyah Bar David wrote:
On Sun, Dec 6, 2020 at 12:34 AM Derek Atkins <derek@ihtfp.com> wrote:
Hi,
I've got a single-host hosted-engine deployment that I originally installed with 4.0 and have upgraded over the years to 4.3.10. I and some of my users have upgraded remote-viewer and now I get an error when I try to view the console of my VMs:
(remote-viewer:8252): Spice-WARNING **: 11:30:41.806: ../subprojects/spice-common/common/ssl_verify.c:477:openssl_verify: Error in server certificate verification: CA signature digest algorithm too weak (num=68:depth0:/O=<My Org Name>/CN=<Host's Name>)
I am 99.99% sure this is because the old certs use SHA1.
I reran engine-setup on the engine and it asked me if I wanted to renew the PKI, and I answered yes. This replaced many[1] of the certificates in /etc/pki/ovirt-engine/certs on the engine, but it did not update the Host's certificate.
Indeed.
All the documentation I've seen says that to refresh this certificate I need to put the host into maintenance mode and then re-enroll.. However I cannot do that, because this is a single-host system so I cannot put the host in local mode -- there is no place to migrate the VMs (let alone the Engine VM).
So.... Is there a command-line way to re-enroll manually and update the host certs?
I don't think you'll find anything like this.
People did come up in the past with various procedure to hack pki like what you want, but these are, generally speaking, quite fragile - usually do not get updated over versions etc.
I am pretty certain the only way to do this using "official" tools/docs is:
1. Stop all VMs except for the engine one.
2. Take a backup with engine-backup.
3. Stop the engine VM.
4. Reinstall the host OS from scratch or use ovirt-hosted-engine-cleanup.
5. Provision the host again as a hosted-engine host, using '--restore-from-file'. Either using new storage for the engine, or after cleaning up the existing hosted-engine storage.
If you still want to try doing this manually, then the tool to use is pki-enroll-request.sh. IIRC it's documented. You should find what keys/certs you want to replace, generate new keys and CSRs (or use existing keys and generate CSRs, or even use existing CSRs if you find them), copy to the engine, sign with pki-enroll-request.sh, then copy the generated cert to the host. I am almost certain there is no way to tell vdsm (and other processes) to reload the certs, so you'll have to restart it (them) - and this usually requires putting the host in maintenance (and therefore stop (migrate) all VMs).
Or some other way to get all the leftover certs renewed?
Which ones, specifically?
Thanks,
-derek
[1] Not only did it not update the Host's cert, it did not update any of the vmconsole-proxy certs, nor the certs in /etc/pki/ovirt-vmconsole/, and obviously nothing in /etc/pki/ on the host itself.
AFAIR no process uses these certs as such. There are only processes that use the ssh-format keys extracted from them, which do not include a signature (sha1 or whatever).
If you think I am wrong, and/or notice other certs that need to be regenerated, that's a bug - please open one. Thanks!
Re remote-viewer/spice: You didn't say if you tried again after engine-setup and what happened. In any case, this is unrelated to vmconsole (which is for serial consoles, using ssh). But you might still need to regenerate the host cert.
BTW: You can try using novnc and websocket-proxy - engine-setup does update the cert for the latter, so this might work as-is.
Best regards, -- Didi
-- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant
-- Didi
Hi Didi, Sorry for the multiple emails yesterday. I'm going to respond to all of your responses in this one. On Mon, December 7, 2020 3:31 am, Yedidyah Bar David wrote:
On Sun, Dec 6, 2020 at 8:14 PM Derek Atkins <derek@ihtfp.com> wrote:
Hi again,
I also noticed that ca.pem was not updated -- it's still using Sha1.
You are right - we didn't make engine-setup recreate existing certs for this - "Renew" deals with other stuff [1]. We only change the default for new ones [2], and wrote a procedure [3][4] for doing this manually. At the time, this wasn't mandatory - browsers didn't reject sha1. Perhaps now it should be.
I should point out that it's not the browsers that are rejecting SHA1, but it is remote-viewer that is. My Fedora-33 firefox connected to my Sha1-using Ovirt HTTPS just fine, without any complaints. Granted, as I note later, these certs were already imported and accepted in firefox years ago, so that could be why there was not complaints. However, the console.vv file sent to remote-viewer includes the CA cert -- but I'm not sure if it's complaining about that or the host cert that gets sent during the connection; I can't tell from the output. I know it's the CA cert that's sent in the .vv file, but I'm not 100% sure which particular source is being used, and I'm not sure which cert is considered "bad" by the viewer. [snip]
I don't know if this will be an issue with remote-viewer if I wind up refreshing the host cert?
As I said, at the time it didn't seem to be mandatory, and docs seemed to be enough. If you feel otherwise, please open a bug.
I already refreshed the CA cert, so unfortunately I wont be able to test this for sure. I know that refreshing the CA cert on the engine alone is not sufficient -- the console.vv file still has the old one, even after an engine restart.
I think there is a difference, or at least there was, between what browsers did/do with https certs, and what they did with CA certs.
Probably true, but firefox was not complaining with the Sha1 certs.
If you had a CA cert already accepted/imported/trusted by the browser, and then you entered a site with a cert signed by this CA, but with a SHA1 signature, this was one separate case. Browsers started warning/ rejecting them earlier.
I think I had already accepted the site cert which is probably why it wasn't complaining about it being SHA1.
If you have a CA cert with a SHA1 signature, and want to import that to a browser, that's another case. I didn't test recently (or much over time, other than working on these bugs) with recent browsers, but I think it took longer until they rejected (if indeed they do - not sure all of them do).
Indeed. I already had both the SHA1 CA cert and the SHA1 host cert accepted in my Firefox trust before I upgraded, so perhaps that's why I didn't see any issues in F33. I removed both and re-imported the (new) CA cert.
One more question:
Can you verify that etc/pki/libvirt/clientcert.pem, etc/pki/vdsm/certs/vdsmcert.pem, and etc/pki/vdsm/libvirt-spice/server-cert.pem are all supposed to be same certificate (on the host)? By a quick find | grep all three of these files appear to be the <Host>.cer certificate file?
Yes, and also vdsm/libvirt-vnc/server-cert.pem .
I don't see this directly in /etc/pki on the host? All I see is: # ls -l /etc/pki/vdsm/ total 8 drwxr-xr-x. 2 vdsm kvm 4096 Dec 6 17:16 certs drwxr-xr-x. 2 vdsm kvm 80 Jun 7 2020 keys drwxr-xr-x. 2 vdsm kvm 4096 Dec 6 17:18 libvirt-spice And /etc/pki/vdsm does not exist on the engine. Indeed: # find /etc/pki -name server-cert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem
Does it matter that ca.der didn't change? I don't know if that is a self-signed cert that might be problematic?
ca.der is not used by anything, you can ignore it. The private key of the CA is in /etc/pki/ovirt-engine/private/ca.pem, and the public key is in /etc/pki/ovirt-engine/ca.pem. That's what all tools use.
Actually, I verified that ca.der *IS* used -- that's what gets sent out if you access http://your-manager-fqdn/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA -- so I had to update that in order to make the new cert available.
Generally speaking, the project considers the "standard" use case to be a setup of at least two hosts, and at least one host "extra" (in terms of capacity), so that if a host fails, you can still keep everything up. In that regard, a single-host setup is considered a kind of "corner case", meant mainly for testing/development, not production. Is there such a big advantage in using oVirt for a single host, compared to virt-manager?
The main advantages of ovirt over virt-manager is the access-control and remote-access capabilities. Specifically, I have several users which have different access to different VMs and their consoles. Without providing ssh access to the host, I wasn't sure how to provide that access in a clean way via virt-manager. I *used* to use vmware-server, and kept that running as long as I could (on a single server). But that hardware was running long in the tooth, so I migrated to ovirt because it gave me a similar web-based UI interface to my users so it was relatively easy to migrate them. When I migrated 4-5 years ago, virt-manager was still in relative infancy and, IIRC, didn't have much remote capability. Of course, because of this view that ovirt "requires" multiple hosts, there are several upgrade issues here. I don't mind having to reboot the host periodically (like once or twice a year); I can schedule that kind of downtime. But it would cost about $10-12k to add two more hosts and the 10Gb infrastructure necessary to have a true 3-host ovirt deployment, and my users don't want to spend that (and I cannot afford that on my own). When I asked them, the response was "we can live with periodic downtime; it's not worth $10k to gain one more nine"). I should add that this deployment is 30% personal, 5% for friends, and 65% for various open-source projects on a volunteer basis, so there is not a funding source for the resources. I donate my time, my electricity, and my network to run and maintain it. There's still multiple single points of failure -- I had a 53-hour power outage after Hurricane (well, tropical storm by the time it hit us) Eta came through, and I've had several network outages, some due to local fiber cuts and one due to an upstream router failure. So even if I had 2 more ovirt hosts, it wouldn't solve THOSE issues, which frankly hit much more often than me having to reboot! I still love this project. I think ovirt is the best (OSS) VM management solution out there. But I wish there was more support for smaller production deployments like mine, and that the development team considered single-host production deployments in their plans and testing. Again, thank you for all your support. I have a scheduled reboot tonight so I'll let you know how it goes after I restart. -derek PS: Happy (almost) Hannukah! -- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant
On Mon, Dec 7, 2020 at 2:22 PM Derek Atkins <derek@ihtfp.com> wrote: [snip]
The main advantages of ovirt over virt-manager is the access-control and remote-access capabilities. Specifically, I have several users which have different access to different VMs and their consoles. Without providing ssh access to the host, I wasn't sure how to provide that access in a clean way via virt-manager.
I *used* to use vmware-server, and kept that running as long as I could (on a single server). But that hardware was running long in the tooth, so I migrated to ovirt because it gave me a similar web-based UI interface to my users so it was relatively easy to migrate them. When I migrated 4-5 years ago, virt-manager was still in relative infancy and, IIRC, didn't have much remote capability.
+1 here. And I think developers should put more attention in single host environments than lastly done. Derek explained very well what could be many common situations to have a single host environment and the reason not to use virt-manager and such. At time there was the all-in-one and then it was deprecated/abandoned in favour of single host deployment. Now due to perhaps ansible playbook or new logic in host upgrades it seems to see more and more messages about single host not supported. In my opinion to do a reinstall every minor update is not feasible. The free version of ESXi could become a more appealing alternative in the near future for these kind of usages, with the risk of potentially eating away also the bigger shape scenario then. Perhaps to support again the all-in-one effort could be a good approach. If you think it can get more attention I can open an RFE for revamping all-in-one or an RFE to provide a mechanism to have an host update itself through a locally executed playbook and then "acquired" by the SHE in a second time when exiting global maintenance. I don't know the internals enough and I have not the coding skills, but I can support testing the functionality Thanks for listening Gianluca
On 7 Dec 2020, at 15:35, Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Mon, Dec 7, 2020 at 2:22 PM Derek Atkins <derek@ihtfp.com <mailto:derek@ihtfp.com>> wrote: [snip]
The main advantages of ovirt over virt-manager is the access-control and remote-access capabilities. Specifically, I have several users which have different access to different VMs and their consoles. Without providing ssh access to the host, I wasn't sure how to provide that access in a clean way via virt-manager.
I *used* to use vmware-server, and kept that running as long as I could (on a single server). But that hardware was running long in the tooth, so I migrated to ovirt because it gave me a similar web-based UI interface to my users so it was relatively easy to migrate them. When I migrated 4-5 years ago, virt-manager was still in relative infancy and, IIRC, didn't have much remote capability.
+1 here. And I think developers should put more attention in single host environments than lastly done.
well, the truth is it is a corner case. I’m not saying it shouldn’t work but as Didi said a single host management was never the main goal. We’ve built oVirt around shared storage and DC scalability, that it sort of happens to work with single host is….nice, but it’s really not that typical. There are better options for desktop-like virtualization in OSS world, there’s virt-manager, there’s VM management in cockpit UI, gnome-boxes.
Derek explained very well what could be many common situations to have a single host environment and the reason not to use virt-manager and such. At time there was the all-in-one and then it was deprecated/abandoned in favour of single host deployment.
yes. but it was never meant to be a real thing in a first place, it was created just for demo purposes so it can run on a single laptop.
Now due to perhaps ansible playbook or new logic in host upgrades it seems to see more and more messages about single host not supported.
it’s not intentional, just not tested enough so it keeps breaking. we really can’t test every use case in automation.
In my opinion to do a reinstall every minor update is not feasible. The free version of ESXi could become a more appealing alternative in the near future for these kind of usages, with the risk of potentially eating away also the bigger shape scenario then. Perhaps to support again the all-in-one effort could be a good approach.
all-in-one was awful, i’d rather fix those few little things around single host deployment:)
If you think it can get more attention I can open an RFE for revamping all-in-one or an RFE to provide a mechanism to have an host update itself through a locally executed playbook and then "acquired" by the SHE in a second time when exiting global maintenance. I don't know the internals enough and I have not the coding skills, but I can support testing the functionality
I don’t think it would take too much attention, TBH. We’re still dealing with 4.4 and el8 complications (it’s still fairly early since GA of a major release) What would make sense, I think, is to identify the actual issues/complications and do them differently, like indeed a special local playbooks or whatnot, or “special” hacks. And then document on oVirt wiki. But otherwise I do not really see them supportable - the amount of work to e.g. re-enroll certs on a running host is just too much to do properly, and everyone has a different level of “risk” they accept. HTH, michal
Thanks for listening Gianluca
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/IXXAXY7IMAWSL7...
Hi Michal, On Mon, December 7, 2020 11:43 am, Michal Skrivanek wrote:
On 7 Dec 2020, at 15:35, Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Mon, Dec 7, 2020 at 2:22 PM Derek Atkins <derek@ihtfp.com <mailto:derek@ihtfp.com>> wrote: [snip]
The main advantages of ovirt over virt-manager is the access-control and remote-access capabilities. Specifically, I have several users which have different access to different VMs and their consoles. Without providing ssh access to the host, I wasn't sure how to provide that access in a clean way via virt-manager.
[snip]
+1 here. And I think developers should put more attention in single host environments than lastly done.
well, the truth is it is a corner case. I’m not saying it shouldn’t work but as Didi said a single host management was never the main goal. We’ve built oVirt around shared storage and DC scalability, that it sort of happens to work with single host is….nice, but it’s really not that typical. There are better options for desktop-like virtualization in OSS world, there’s virt-manager, there’s VM management in cockpit UI, gnome-boxes.
As of several years ago, I don't think any of these options worked with multiple, distributed (remote) users with different capabilities on the same VM Host. Has that changed?
Derek explained very well what could be many common situations to have a single host environment and the reason not to use virt-manager and such. At time there was the all-in-one and then it was deprecated/abandoned in favour of single host deployment.
yes. but it was never meant to be a real thing in a first place, it was created just for demo purposes so it can run on a single laptop.
Now due to perhaps ansible playbook or new logic in host upgrades it seems to see more and more messages about single host not supported.
it’s not intentional, just not tested enough so it keeps breaking. we really can’t test every use case in automation.
I think there are enough users who want this configuration (or, gasp, are actually using this configuration) that it might warrant a little more testing. Yes, we understand that there will be times that we need to shut down VMs and reboot the system, and those times can be scheduled (like I've done). However, that WOULD require a little more support, to at least have a recipe that works on a single-host hosted-engine solution. For host cert renewal, that recipe didn't really exist. [snip]
I don’t think it would take too much attention, TBH. We’re still dealing with 4.4 and el8 complications (it’s still fairly early since GA of a major release)
Of course. (Personally, I think it should have been called 5.0 instead of 4.4, as it requires a full re-install to migrate from 4.3).
What would make sense, I think, is to identify the actual issues/complications and do them differently, like indeed a special local playbooks or whatnot, or “special” hacks. And then document on oVirt wiki. But otherwise I do not really see them supportable - the amount of work to e.g. re-enroll certs on a running host is just too much to do properly, and everyone has a different level of “risk” they accept.
Exactly. Some tested playbook recipes that allows a single-host hosted-engine deployment to perform these operations is really what I'm asking for. Yes, I know it will require rebooting. But reboot is much less risky that re-install! And for the record, after putting the new certificates into place by hand, just restarting a VM was sufficient to get Spice to pull in the new cert(s). So, technically, it LOOKS like I don't have to reboot the whole system (although I plan to do that tonight) -- I could just shutdown and re-run each VM.
HTH, michal
Thank you for all your support and everything you do for this project, Michal. We very much appreciate it! -derek -- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant
Hi, On Mon, December 7, 2020 4:02 pm, Derek Atkins wrote:
Hi Michal,
On Mon, December 7, 2020 11:43 am, Michal Skrivanek wrote:
[snip] And for the record, after putting the new certificates into place by hand, just restarting a VM was sufficient to get Spice to pull in the new cert(s). So, technically, it LOOKS like I don't have to reboot the whole system (although I plan to do that tonight) -- I could just shutdown and re-run each VM.
HTH, michal
Thank you for all your support and everything you do for this project, Michal. We very much appreciate it!
For the record, I rebooted the host last night and once everything came back, the new certs were all in place and everything was happy.... Except for the fact that my host cert does not have a SAN (SubjectAltName) so the engine is *still* complaining about it. See my other email about that. FYI, here are the commands I used to refresh everything (modulo restarting everything): set my_date="$(date +"%Y%m%d%H%M%S")" ## On the ENGINE, rebuild the CA Cert: cp -p /etc/pki/ovirt-engine/private/ca.pem /etc/pki/ovirt-engine/private/ca.pem.$my_date cp -p /etc/pki/ovirt-engine/ca.pem{,.$my_date} openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in /etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256 openssl x509 -in /etc/pki/ovirt-engine/ca.pem.new -text > /etc/pki/ovirt-engine/ca.pem.new.full mv /etc/pki/ovirt-engine/ca.pem.new.full /etc/pki/ovirt-engine/ca.pem mv /etc/pki/ovirt-engine/certs/ca.der{,.$my_date} cp -p /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/certs/ca.der # On ovirt host, create a CSR: # openssl x509 -x509toreq -in /etc/pki/libvirt/clientcert.pem -out /tmp/HOST.csr -signkey /etc/pki/libvirt/private/clientkey.pem mv /etc/pki/ovirt-engine/certs/host.na.me.cer{,.$my_date} mv /etc/pki/ovirt-engine/requests/host.na.me.req{,.$my_date} # copy new CSR into place on the engine: # /etc/pki/ovirt-engine/requests/host.na.me.req # and sign it: /usr/share/ovirt-engine/bin/pki-enroll-request.sh --name=host.na.me # NB -- adding --san results in an error: --san=host.na.me # copy new Host cert from /etc/pki/ovirt-engine/certs/host.na.me.cer # to host:new_cert # and copy CA cert to host:cacert.pem # ON OVIRT Host: mv /etc/pki/libvirt/clientcert.pem{,.$my_date} mv /etc/pki/vdsm/certs/vdsmcert.pem{,.$my_date} mv /etc/pki/vdsm/libvirt-spice/server-cert.pem{,.$my_date} cp -p new_cert /etc/pki/libvirt/clientcert.pem cp -p new_cert /etc/pki/vdsm/certs/vdsmcert.pem cp -p new_cert /etc/pki/vdsm/libvirt-spice/server-cert.pem chown root:kvm /etc/pki/libvirt/clientcert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem # # Copy new CA cert into place on Host: mv /etc/pki/CA/cacert.pem{,$my_date} cp -p cacert.pem /etc/pki/CA/cacert.pem chgrp kvm /etc/pki/CA/cacert.pem mv /etc/pki/vdsm/certs/cacert.pem{,.$my_date} mv /etc/pki/vdsm/libvirt-spice/ca-cert.pem{,.$my_date} mv /etc/pki/ovirt-engine/ca.pem{,.$my_date} cp -p /etc/pki/CA/cacert.pem /etc/pki/vdsm/certs/cacert.pem cp -p /etc/pki/CA/cacert.pem /etc/pki/vdsm/libvirt-spice/ca-cert.pem cp -p /etc/pki/CA/cacert.pem /etc/pki/ovirt-engine/ca.pem At this point I shut down all VMs, rebooted the host, and restarted all the VMs and everything came back happy (except for the lack of the SubjectAltName). Also note that you will need to remove the trusted cert from your browser(s) and re-add the new CA cert -- otherwise you will get a browser error complaining about the change in certificate from the same Issuer and with the same Serial#. -derek -- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant
This is the only place we found the answer we needed concerning how to sign a host key from the engine! Saved our bacon! The only tweak I would make is that there were a few more destinations at the end to copy the files into, and instead of rebooting the host we found that just restarting vdsmd and libvirtd got everything working without any existing VMs having to stop. Here's a complete update of what you have above intermingling some extra bits for anyone who has to do this in the future. If the certs on your oVirt host expire, it can be a PITA to figure out how to fix it. It's actually simple, but takes a LOT of manual work. Make sure every part of every script makes sense as I made some modifications as I documented it! I do not warrant that I didn't make a mistake somewhere. :) ########### On the ENGINE HOST ########### ## Check the CA Cert on the ENGINE openssl x509 -in /etc/pki/ovirt-engine/ca.pem -noout -dates ## If it is expired, then on the ENGINE, rebuild the CA Cert with this in a script: ## NOTE: these are steps from another post I didn't need to do them -- could be dangerous... ``` set -x ## Make the script echo everything out, so if it fails you know where set -e ## Make the script STOP on any error set my_date="$(date +"%Y%m%d%H%M%S")" # Backup the existing CA files /bin/cp -p /etc/pki/ovirt-engine/private/ca.pem /etc/pki/ovirt-engine/private/ca.pem.$my_date /bin/cp -p /etc/pki/ovirt-engine/ca.pem{,.$my_date} /bin/mv /etc/pki/ovirt-engine/certs/ca.der{,.$my_date} # Sign the key openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in /etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256 openssl x509 -in /etc/pki/ovirt-engine/ca.pem.new -text > /etc/pki/ovirt-engine/ca.pem.new.full # Put the files into place /bin/mv -f /etc/pki/ovirt-engine/ca.pem.new.full /etc/pki/ovirt-engine/ca.pem /bin/cp -p /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/certs/ca.der ``` Now you need to copy the new CA file over to the host: Source: ENGINE `/etc/pki/ovirt-engine/ca.pem` Dest: HOST `/tmp/new-ca.pem` ########### On the oVirt Host ########### # Create a CSR using the information from the existing certificate and the existing key: openssl x509 -x509toreq -in /etc/pki/libvirt/clientcert.pem -out /tmp/HOST.csr -signkey /etc/pki/libvirt/private/clientkey.pem Now you need to copy the new CA file over to the host: Source: HOST `/tmp/HOST.csr` Dest: ENGINE `/etc/pki/ovirt-engine/requests/full.hostname.com.req` ########### On the ENGINE HOST ########### # Now sign it: /usr/share/ovirt-engine/bin/pki-enroll-request.sh --name=full.hostname.com # NB -- adding --san results in an error: --san=host.na.me (So no Subject Alternate Names) Now you need to copy the new Certificate file over to the host: Source: ENGINE /etc/pki/ovirt-engine/certs/full.hostname.com.cer Dest: HOST /tmp/new-cert.pem ########### On the oVirt Host ########### Run this script to put the cert and CA in place. Note if you don't put a ca into /tmp/new-ca.pem it skips that step. ``` set -x set -e set my_date="$(date +"%Y%m%d%H%M%S")" for x in /etc/pki/libvirt/clientcert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-migrate/server-cert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem /etc/pki/vdsm/libvirt-vnc/server-cert.pem; do /bin/mv -n $x ${x}.${mydate} /bin/cp /tmp/new-cert.pem ${x} chmod 644 ${x} chown root:kvm ${x} done if -f /tmp/new-ca.pem; then for x in /etc/pki/vdsm/libvirt-migrate/ca-cert.pem /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-vnc/ca-cert.pem /etc/pki/vdsm/libvirt-spice/ca-cert.pem /etc/pki/CA/cacert.pem; do /bin/mv -n $x ${x}.${mydate} /bin/cp /tmp/new-ca.pem ${x} chmod 644 ${x} chown root:kvm ${x} done fi ``` Now you're ready to restart two vital services -- some people say "reboot the host" -- but we found that unecessary. Running this restart was safe for us and didn't cause any running VMs to crash or reboot -- they kept going without issue. Once we did this, waited a few minutes, the host came back up on the engine and everything was happy. Specifically a VM we were unable to get running was a click away from full functionality again! # Restart the two services affected by the key/cert changes systemctl restart vdsmd libvirtd
Hi, Recently I found another way to renew the certificates on a one-node self-hosted environment.The trick is to stop vdsmd and wait till the engine shows the system unresponsive. Then you can trigger a certificate renewal and just power on vdsmd again. Best Regards,Strahil Nikolov On Sun, Jan 14, 2024 at 2:54, mikedoug--- via Users<users@ovirt.org> wrote: This is the only place we found the answer we needed concerning how to sign a host key from the engine! Saved our bacon! The only tweak I would make is that there were a few more destinations at the end to copy the files into, and instead of rebooting the host we found that just restarting vdsmd and libvirtd got everything working without any existing VMs having to stop. Here's a complete update of what you have above intermingling some extra bits for anyone who has to do this in the future. If the certs on your oVirt host expire, it can be a PITA to figure out how to fix it. It's actually simple, but takes a LOT of manual work. Make sure every part of every script makes sense as I made some modifications as I documented it! I do not warrant that I didn't make a mistake somewhere. :) ########### On the ENGINE HOST ########### ## Check the CA Cert on the ENGINE openssl x509 -in /etc/pki/ovirt-engine/ca.pem -noout -dates ## If it is expired, then on the ENGINE, rebuild the CA Cert with this in a script: ## NOTE: these are steps from another post I didn't need to do them -- could be dangerous... ``` set -x ## Make the script echo everything out, so if it fails you know where set -e ## Make the script STOP on any error set my_date="$(date +"%Y%m%d%H%M%S")" # Backup the existing CA files /bin/cp -p /etc/pki/ovirt-engine/private/ca.pem /etc/pki/ovirt-engine/private/ca.pem.$my_date /bin/cp -p /etc/pki/ovirt-engine/ca.pem{,.$my_date} /bin/mv /etc/pki/ovirt-engine/certs/ca.der{,.$my_date} # Sign the key openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in /etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256 openssl x509 -in /etc/pki/ovirt-engine/ca.pem.new -text > /etc/pki/ovirt-engine/ca.pem.new.full # Put the files into place /bin/mv -f /etc/pki/ovirt-engine/ca.pem.new.full /etc/pki/ovirt-engine/ca.pem /bin/cp -p /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/certs/ca.der ``` Now you need to copy the new CA file over to the host: Source: ENGINE `/etc/pki/ovirt-engine/ca.pem` Dest: HOST `/tmp/new-ca.pem` ########### On the oVirt Host ########### # Create a CSR using the information from the existing certificate and the existing key: openssl x509 -x509toreq -in /etc/pki/libvirt/clientcert.pem -out /tmp/HOST.csr -signkey /etc/pki/libvirt/private/clientkey.pem Now you need to copy the new CA file over to the host: Source: HOST `/tmp/HOST.csr` Dest: ENGINE `/etc/pki/ovirt-engine/requests/full.hostname.com.req` ########### On the ENGINE HOST ########### # Now sign it: /usr/share/ovirt-engine/bin/pki-enroll-request.sh --name=full.hostname.com # NB -- adding --san results in an error: --san=host.na.me (So no Subject Alternate Names) Now you need to copy the new Certificate file over to the host: Source: ENGINE /etc/pki/ovirt-engine/certs/full.hostname.com.cer Dest: HOST /tmp/new-cert.pem ########### On the oVirt Host ########### Run this script to put the cert and CA in place. Note if you don't put a ca into /tmp/new-ca.pem it skips that step. ``` set -x set -e set my_date="$(date +"%Y%m%d%H%M%S")" for x in /etc/pki/libvirt/clientcert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-migrate/server-cert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem /etc/pki/vdsm/libvirt-vnc/server-cert.pem; do /bin/mv -n $x ${x}.${mydate} /bin/cp /tmp/new-cert.pem ${x} chmod 644 ${x} chown root:kvm ${x} done if -f /tmp/new-ca.pem; then for x in /etc/pki/vdsm/libvirt-migrate/ca-cert.pem /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-vnc/ca-cert.pem /etc/pki/vdsm/libvirt-spice/ca-cert.pem /etc/pki/CA/cacert.pem; do /bin/mv -n $x ${x}.${mydate} /bin/cp /tmp/new-ca.pem ${x} chmod 644 ${x} chown root:kvm ${x} done fi ``` Now you're ready to restart two vital services -- some people say "reboot the host" -- but we found that unecessary. Running this restart was safe for us and didn't cause any running VMs to crash or reboot -- they kept going without issue. Once we did this, waited a few minutes, the host came back up on the engine and everything was happy. Specifically a VM we were unable to get running was a click away from full functionality again! # Restart the two services affected by the key/cert changes systemctl restart vdsmd libvirtd _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2EBSBWXFCEANDO...
Ah, the steps I list here work for non-single node environments as well. The problem with vdsm-tool doing the certificate renewal in that setup is that the certificate needs to be signed by the engine host and not the local host. Apologies if this causes any confusion for single-node users. ________________________________ From: Strahil Nikolov <hunter86_bg@yahoo.com> Sent: Saturday, January 13, 2024 6:57 PM To: Michael Douglass <mikedoug@certida.com>; mikedoug--- via Users <users@ovirt.org> Subject: [EXT]Re: [ovirt-users] Re: [SOLVED] Re: Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment? CAUTION: Be careful of links - This email originated from outside of Certida. Hi, Recently I found another way to renew the certificates on a one-node self-hosted environment. The trick is to stop vdsmd and wait till the engine shows the system unresponsive. Then you can trigger a certificate renewal and just power on vdsmd again. Best Regards, Strahil Nikolov On Sun, Jan 14, 2024 at 2:54, mikedoug--- via Users <users@ovirt.org> wrote: This is the only place we found the answer we needed concerning how to sign a host key from the engine! Saved our bacon! The only tweak I would make is that there were a few more destinations at the end to copy the files into, and instead of rebooting the host we found that just restarting vdsmd and libvirtd got everything working without any existing VMs having to stop. Here's a complete update of what you have above intermingling some extra bits for anyone who has to do this in the future. If the certs on your oVirt host expire, it can be a PITA to figure out how to fix it. It's actually simple, but takes a LOT of manual work. Make sure every part of every script makes sense as I made some modifications as I documented it! I do not warrant that I didn't make a mistake somewhere. :) ########### On the ENGINE HOST ########### ## Check the CA Cert on the ENGINE openssl x509 -in /etc/pki/ovirt-engine/ca.pem -noout -dates ## If it is expired, then on the ENGINE, rebuild the CA Cert with this in a script: ## NOTE: these are steps from another post I didn't need to do them -- could be dangerous... ``` set -x ## Make the script echo everything out, so if it fails you know where set -e ## Make the script STOP on any error set my_date="$(date +"%Y%m%d%H%M%S")" # Backup the existing CA files /bin/cp -p /etc/pki/ovirt-engine/private/ca.pem /etc/pki/ovirt-engine/private/ca.pem.$my_date /bin/cp -p /etc/pki/ovirt-engine/ca.pem{,.$my_date} /bin/mv /etc/pki/ovirt-engine/certs/ca.der{,.$my_date} # Sign the key openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in /etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256 openssl x509 -in /etc/pki/ovirt-engine/ca.pem.new -text > /etc/pki/ovirt-engine/ca.pem.new.full # Put the files into place /bin/mv -f /etc/pki/ovirt-engine/ca.pem.new.full /etc/pki/ovirt-engine/ca.pem /bin/cp -p /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/certs/ca.der ``` Now you need to copy the new CA file over to the host: Source: ENGINE `/etc/pki/ovirt-engine/ca.pem` Dest: HOST `/tmp/new-ca.pem` ########### On the oVirt Host ########### # Create a CSR using the information from the existing certificate and the existing key: openssl x509 -x509toreq -in /etc/pki/libvirt/clientcert.pem -out /tmp/HOST.csr -signkey /etc/pki/libvirt/private/clientkey.pem Now you need to copy the new CA file over to the host: Source: HOST `/tmp/HOST.csr` Dest: ENGINE `/etc/pki/ovirt-engine/requests/full.hostname.com.req` ########### On the ENGINE HOST ########### # Now sign it: /usr/share/ovirt-engine/bin/pki-enroll-request.sh --name=full.hostname.com # NB -- adding --san results in an error: --san=host.na.me (So no Subject Alternate Names) Now you need to copy the new Certificate file over to the host: Source: ENGINE /etc/pki/ovirt-engine/certs/full.hostname.com.cer Dest: HOST /tmp/new-cert.pem ########### On the oVirt Host ########### Run this script to put the cert and CA in place. Note if you don't put a ca into /tmp/new-ca.pem it skips that step. ``` set -x set -e set my_date="$(date +"%Y%m%d%H%M%S")" for x in /etc/pki/libvirt/clientcert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-migrate/server-cert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem /etc/pki/vdsm/libvirt-vnc/server-cert.pem; do /bin/mv -n $x ${x}.${mydate} /bin/cp /tmp/new-cert.pem ${x} chmod 644 ${x} chown root:kvm ${x} done if -f /tmp/new-ca.pem; then for x in /etc/pki/vdsm/libvirt-migrate/ca-cert.pem /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-vnc/ca-cert.pem /etc/pki/vdsm/libvirt-spice/ca-cert.pem /etc/pki/CA/cacert.pem; do /bin/mv -n $x ${x}.${mydate} /bin/cp /tmp/new-ca.pem ${x} chmod 644 ${x} chown root:kvm ${x} done fi ``` Now you're ready to restart two vital services -- some people say "reboot the host" -- but we found that unecessary. Running this restart was safe for us and didn't cause any running VMs to crash or reboot -- they kept going without issue. Once we did this, waited a few minutes, the host came back up on the engine and everything was happy. Specifically a VM we were unable to get running was a click away from full functionality again! # Restart the two services affected by the key/cert changes systemctl restart vdsmd libvirtd _______________________________________________ Users mailing list -- users@ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html<https://url.avanan.click/v2/___https://www.ovirt.org/privacy-policy.html___.YXAzOmNlcnRpZGE6YTpvOjQ3MTQ2MDVmNTZmNzQxNmI5YjU3NmE5ZTdiYmRmMzUyOjY6OTU1ODoxZDg1ZTdlMTA3ZDg0M2U1MGY0M2Q1NTk1OTc0NzBiMzE5MWEzNjVkM2JmNDEwZGMzYTExMjRmMmE0Zjk2NGZkOmg6VA> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/<https://url.avanan.click/v2/___https://www.ovirt.org/community/about/community-guidelines/___.YXAzOmNlcnRpZGE6YTpvOjQ3MTQ2MDVmNTZmNzQxNmI5YjU3NmE5ZTdiYmRmMzUyOjY6OTI4NjpkOTk3MzA1YThkNThlYTBhMzIyOGRhY2VjMjM2ZTFjNDcwOTk2YWQ2MzdmYWJhYzAwMDM2MjQ0ZDIxYjhlMTU0Omg6VA> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2EBSBWXFCEANDOH3ZLHQENVTAMVDFMOE/<https://url.avanan.click/v2/___https://lists.ovirt.org/archives/list/users@ovirt.org/message/2EBSBWXFCEANDOH3ZLHQENVTAMVDFMOE/___.YXAzOmNlcnRpZGE6YTpvOjQ3MTQ2MDVmNTZmNzQxNmI5YjU3NmE5ZTdiYmRmMzUyOjY6NGIxYTozZDVmZGE1YTJhOTM2NTRjZTc4ZDgxMzE2ZDFmZDg3NjQzNTdmMzhkNmRkYTA4Mjg4MWE2Njk0OGM3MWY1YjAyOmg6VA>
Hi, Can you please walk your "trick" method step-by-step for me? I have a single-hosted hosted-engine deployment. I have renewed the engine certificates using "engine-setup --offline" but struggling with enrolling new certificates for host. Host certificates are not expired for now, but i'd like to renew them. Thanks in advance!
participants (8)
-
Derek Atkins -
Gianluca Cecchi -
Michael Douglass -
Michal Skrivanek -
mikedoug＠certida.com
-
p.vinnik＠sibproauto.ru -
Strahil Nikolov -
Yedidyah Bar David