On Sun, Jan 30, 2022 at 8:16 PM Diggy Mc <d03(a)bornfree.org&gt; wrote: If it's a certificate created by engine-setup for you, you can run 'engine-setup' and it can recreate it for you. If you do not want to update the system, you can run it with 'engine-setup --offline'. Otherwise, if it's a certificate you got elsewhere, you should update it manually, perhaps following some of the steps of the procedure to replace the certificate - the one you followed originally. Good luck and best regards,

> On Sun, Jan 30, 2022 at 8:16 PM Diggy Mc <d03(a)bornfree.org&gt; wrote: > > If it's a certificate created by engine-setup for you, you can run > 'engine-setup' and it can recreate it for you. If you do not want to > update the system, you can run it with 'engine-setup --offline'. > Otherwise, if it's a certificate you got elsewhere, you should update > it manually, perhaps following some of the steps of the procedure to > replace the certificate - the one you followed originally. > > Good luck and best regards, It is the original certificate created during initial install/setup. If possible, I would like to have another oVirt generated certificate without upgrading the engine's version. Where can I find instructions on how to do that? What would be the pros and cons of generating my own self-signed certificate

with a longer validity period?

Where can I find instructions on that?

Again, thanks for your help.

So, what is the problem in restorib the engine from backup ?

Hello, On Mon, Feb 7, 2022 at 8:45 AM Yedidyah Bar David <didi(a)redhat.com&gt; wrote: > > On Sun, Feb 6, 2022 at 5:09 PM Gilboa Davara <gilboad(a)gmail.com&gt; wrote: > > > > Unlike my predecessor, I not only lost my vmengine, I also lost the vdsm services on all hosts. > > All seem to be hitting the same issue - read, the certs under /etc/pki/vdsm/certs and /etc/pki/ovirt* all expired a couple of days ago. > > As such, the hosted engine cannot go into global maintenance mode, > > What do you mean by that? What happens if you 'hosted-engine > --set-maintenance --mode=global'? Failed, stating the cluster is not in global maintenance mode.

(Understandable, given two of 3 hosts were offline due to certificate issues...) > > > > preventing engine-setup --offline from running. > > Actually just a few days ago I pushed a patch for: > > https://bugzilla.redhat.com/show_bug.cgi?id=1700460 > > But: > > If you really have a problem that you can't set global maintenance, > using this is a risk - HA might intervene in the middle and shutdown > the VM. So either make sure global maintenance does work, or stop > all HA services on all hosts. > > > Two questions: > > 1. Is there any automated method to renew the vdsm certificates? > > You mean, without an engine? > > I think that if you have a functional engine one way or another, > you can automate this somehow, didn't check. Try checking e.g. the > python sdk examples - there might be there something you can base > on. > > > 2. Assuming the previous answer is "no", assuming I'm somewhat versed in using openssl, how can I manually renew them? > > I'd rather not try to invent from memory how this is supposed to work, > and doing this methodically and verifying before replying is quite > an effort. > > If this is really what you want, I suggest something like: > > 1. Set up a test env with an engine and one host > 2. Backup (or use git on) /etc on both > 3. Renew the host cert from the UI > 4. Check what changed > > You should find, IMO, that the key(s) on the host didn't > change. I guess you might also find CSRs on one or both of them. > So basically it should be something like: > 1. Create a CSR on the host for the existing key (one or more, > not sure). > 2. Copy and sign this on the engine using pki-enroll-request.sh > (I think you can find examples for it scattered around, perhaps > even in the main guides) > 3. Copy back the generated certs to the host > 4. Perhaps restart one or more services there (vdsm, imageio?, > ovn, etc.) > > You can check the code in > /usr/share/ovirt-engine/ansible-runner-service-project/project > to see how it's done when initiated from the UI. > > Good luck and best regards, I more of less found a document stating the above somewhere in the middle of the night. Tried it. Got the WebUI working again. However, for the life of me I couldn't get the hosts to work to talk to the engine. (Even though I could use openssl s_client -showcerts -connect host and got valid certs). In the end, @around ~4am, I decided to take the brute force route, clean the hosts, upgrade them to -streams, and redeploy the engine again (3'rd attempt, after sufficient amount of coffee reminded me the qemu-6.1 is broken, and needed to be downgraded before trying to deploy the HE...). Either way, when I finish importing the VMs, I'll open a RFE to add BIG-WARNING-IN-BOLD-LETTERS in the WebUI to notify the admin that the certificates are about to expire.

On Mon, Feb 7, 2022 at 1:27 PM Gilboa Davara <gilboad(a)gmail.com&gt; wrote: > > Hello, > > On Mon, Feb 7, 2022 at 8:45 AM Yedidyah Bar David <didi(a)redhat.com&gt; wrote: >> >> On Sun, Feb 6, 2022 at 5:09 PM Gilboa Davara <gilboad(a)gmail.com&gt; wrote: >> > >> > Unlike my predecessor, I not only lost my vmengine, I also lost the vdsm services on all hosts. >> > All seem to be hitting the same issue - read, the certs under /etc/pki/vdsm/certs and /etc/pki/ovirt* all expired a couple of days ago. >> > As such, the hosted engine cannot go into global maintenance mode, >> >> What do you mean by that? What happens if you 'hosted-engine >> --set-maintenance --mode=global'? > > > Failed, stating the cluster is not in global maintenance mode. Please clarify, and/or share relevant logs, if you have them.

You had a semi-working existing HE cluster. You ran engine-backup on it, took a backup, while it was _not_ in global maintenance.

That's ok and expected. Then you took one of the hosts and evacuated it (or just a new one), (re)installed the OS (or somehow cleaned it up), and ran 'hosted-engine --deploy --import-from-file' with the backup you took. This failed? Where exactly and with what error?

If it's the engine-setup running inside the engine VM, with the same error as when running 'engine-setup' (perhaps with --offline) manually, then this shouldn't happen at this point: - engine-backup --mode=restore sets vdc option in the db 'DbJustRestored' - engine-setup checks this and sets its own env[JUST_RESTORED] accordingly > (Understandable, given two of 3 hosts were offline due to certificate issues...) > > >> >> >> > preventing engine-setup --offline from running. >> >> Actually just a few days ago I pushed a patch for: >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1700460 >> >> But: >> >> If you really have a problem that you can't set global maintenance, >> using this is a risk - HA might intervene in the middle and shutdown >> the VM. So either make sure global maintenance does work, or stop >> all HA services on all hosts. >> >> > Two questions: >> > 1. Is there any automated method to renew the vdsm certificates? >> >> You mean, without an engine? >> >> I think that if you have a functional engine one way or another, >> you can automate this somehow, didn't check. Try checking e.g. the >> python sdk examples - there might be there something you can base >> on. >> >> > 2. Assuming the previous answer is "no", assuming I'm somewhat versed in using openssl, how can I manually renew them? >> >> I'd rather not try to invent from memory how this is supposed to work, >> and doing this methodically and verifying before replying is quite >> an effort. >> >> If this is really what you want, I suggest something like: >> >> 1. Set up a test env with an engine and one host >> 2. Backup (or use git on) /etc on both >> 3. Renew the host cert from the UI >> 4. Check what changed >> >> You should find, IMO, that the key(s) on the host didn't >> change. I guess you might also find CSRs on one or both of them. >> So basically it should be something like: >> 1. Create a CSR on the host for the existing key (one or more, >> not sure). >> 2. Copy and sign this on the engine using pki-enroll-request.sh >> (I think you can find examples for it scattered around, perhaps >> even in the main guides) >> 3. Copy back the generated certs to the host >> 4. Perhaps restart one or more services there (vdsm, imageio?, >> ovn, etc.) >> >> You can check the code in >> /usr/share/ovirt-engine/ansible-runner-service-project/project >> to see how it's done when initiated from the UI. >> >> Good luck and best regards, > > > I more of less found a document stating the above somewhere in the middle of the night. > Tried it. > Got the WebUI working again. > However, for the life of me I couldn't get the hosts to work to talk to the engine. (Even though I could use openssl s_client -showcerts -connect host and got valid certs). > In the end, @around ~4am, I decided to take the brute force route, clean the hosts, upgrade them to -streams, and redeploy the engine again (3'rd attempt, after sufficient amount of coffee reminded me the qemu-6.1 is broken, and needed to be downgraded before trying to deploy the HE...). > Either way, when I finish importing the VMs, I'll open a RFE to add BIG-WARNING-IN-BOLD-LETTERS in the WebUI to notify the admin that the certificates are about to expire. You should have already received them, no? https://bugzilla.redhat.com/show_bug.cgi?id=1258585 Best regards, -- Didi

On Mon, Feb 7, 2022 at 4:03 PM Martin Perina <mperina(a)redhat.com&gt; wrote: > > > On Mon, Feb 7, 2022 at 12:33 PM Gilboa Davara <gilboad(a)gmail.com&gt; wrote: > >> Hello, >> >> On Mon, Feb 7, 2022 at 8:45 AM Yedidyah Bar David <didi(a)redhat.com&gt; >> wrote: >> >>> On Sun, Feb 6, 2022 at 5:09 PM Gilboa Davara <gilboad(a)gmail.com&gt; wrote: >>> > >>> > Unlike my predecessor, I not only lost my vmengine, I also lost the >>> vdsm services on all hosts. >>> > All seem to be hitting the same issue - read, the certs under >>> /etc/pki/vdsm/certs and /etc/pki/ovirt* all expired a couple of days ago. >>> > As such, the hosted engine cannot go into global maintenance mode, >>> >>> What do you mean by that? What happens if you 'hosted-engine >>> --set-maintenance --mode=global'? >>> >> >> Failed, stating the cluster is not in global maintenance mode. >> (Understandable, given two of 3 hosts were offline due to certificate >> issues...) >> >> >> >>> >>> > preventing engine-setup --offline from running. >>> >>> Actually just a few days ago I pushed a patch for: >>> >>> https://bugzilla.redhat.com/show_bug.cgi?id=1700460 >>> >>> But: >>> >>> If you really have a problem that you can't set global maintenance, >>> using this is a risk - HA might intervene in the middle and shutdown >>> the VM. So either make sure global maintenance does work, or stop >>> all HA services on all hosts. >>> >>> > Two questions: >>> > 1. Is there any automated method to renew the vdsm certificates? >>> >>> You mean, without an engine? >>> >>> I think that if you have a functional engine one way or another, >>> you can automate this somehow, didn't check. Try checking e.g. the >>> python sdk examples - there might be there something you can base >>> on. >>> >>> > 2. Assuming the previous answer is "no", assuming I'm somewhat versed >>> in using openssl, how can I manually renew them? >>> >>> I'd rather not try to invent from memory how this is supposed to work, >>> and doing this methodically and verifying before replying is quite >>> an effort. >>> >>> If this is really what you want, I suggest something like: >>> >>> 1. Set up a test env with an engine and one host >>> 2. Backup (or use git on) /etc on both >>> 3. Renew the host cert from the UI >>> 4. Check what changed >>> >>> You should find, IMO, that the key(s) on the host didn't >>> change. I guess you might also find CSRs on one or both of them. >>> So basically it should be something like: >>> 1. Create a CSR on the host for the existing key (one or more, >>> not sure). >>> 2. Copy and sign this on the engine using pki-enroll-request.sh >>> (I think you can find examples for it scattered around, perhaps >>> even in the main guides) >>> 3. Copy back the generated certs to the host >>> 4. Perhaps restart one or more services there (vdsm, imageio?, >>> ovn, etc.) >>> >>> You can check the code in >>> /usr/share/ovirt-engine/ansible-runner-service-project/project >>> to see how it's done when initiated from the UI. >>> >>> Good luck and best regards, >>> >> >> I more of less found a document stating the above somewhere in the >> middle of the night. >> Tried it. >> Got the WebUI working again. >> However, for the life of me I couldn't get the hosts to work to talk to >> the engine. (Even though I could use openssl s_client -showcerts -connect >> host and got valid certs). >> In the end, @around ~4am, I decided to take the brute force route, clean >> the hosts, upgrade them to -streams, and redeploy the engine again (3'rd >> attempt, after sufficient amount of coffee reminded me the qemu-6.1 is >> broken, and needed to be downgraded before trying to deploy the HE...). >> Either way, when I finish importing the VMs, I'll open a RFE to add >> BIG-WARNING-IN-BOLD-LETTERS in the WebUI to notify the admin that the >> certificates are about to expire. >> > > We already have quite a lot of warnings/alters about certificates which > are going to expire soon: > > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/TMJVAJMH5MK... > > So what exactly are you missing here? > I don't know how, but the only errors I saw in the WebUI were update related (failed to check updates on host).

late.

- Gilboa > >> Thanks for the help! >> >> - Gilboa >> >> >> >>> -- >>> Didi >>> >>> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/CGSAB7NPWWO... >> > > > -- > Martin Perina > Manager, Software Engineering > Red Hat Czech s.r.o. >

On Wed, Feb 9, 2022 at 1:05 AM Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; wrote: > I have no clue , but I would give vdsm.service a restart. > Thanks again for the prompt response. Tried that, restarted all services and the all the VMS, didn't work. Any idea how I can verify the certificate information actually being used by qemu for the spice console? remote-viewer just fails, without giving any meaningful error message. - Gilboa   > > Best Regards, > Strahil Nikolov > > > On Tue, Feb 8, 2022 at 18:19, Gilboa Davara > > <gilboad(a)gmail.com&gt; wrote: > > _______________________________________________ > > Users mailing list -- users(a)ovirt.org > > To unsubscribe send an email to users-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/privacy-policy.html > > oVirt Code of Conduct: > > https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: > > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/2GAQH44QD6K... > > _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/OG57VT2XGDT...

On Wed, Feb 9, 2022 at 7:52 AM Patrick Hibbs <hibbsncc1701(a)gmail.com&gt; wrote: > The certificates used in SPICE connections are stored on the VM hosts. By > default they are at /etc/pki/vdsm/libvirt-spice, and configured by VDSM in > /etc/libvirt/qemu.conf. Their default names are ca-cert.pem, > server-cert.pem, and server-key.pem. Using openssl x509 -noout -text -in > </path/to/cert-file> should show you the certificate's expiration info. > > Note: Don't try to change anything, it will be overwritten by VDSM on the > next host update / reinstall. > > As for remote-viewer, if you run it manually from the console with > "remote-viewer --debug </path/to/console.vv>" or "remote-viewer --verbose > </path/to/console.vv>" it will print log information about the connection > it's trying to establish. > > -Patrick Hibbs > > Hello, You must have missed my answer above. (Understandable, given the length of this thread...) I replaced and verified /etc/pki/vdsm/libvirt-spice/server-cert.pem Restarted all the services on the host. $ openssl x509 -in /etc/pki/vdsm/libvirt-spice/server-cert.pem -noout -dates notBefore=Feb 7 13:59:14 2022 GMT notAfter=Feb 7 13:59:14 2027 GMT $ openssl x509 -in /etc/pki/vdsm/libvirt-spice/ca-cert.pem -noout -dates notBefore=Dec 26 16:25:01 2020 GMT notAfter=Dec 25 16:25:01 2030 GMT However, remote-viewer still fails: $ remote-viewer --debug console.vv ... (remote-viewer:14874): Spice-WARNING **: 18:14:33.500: ../subprojects/spice-common/common/ssl_verify.c:506:openssl _verify: ssl: subject 'O=localdomain,CN=gilboa-wx-srv.localdomain' verification failed The main problem here is that while we assume the problem is expired certificates, it can be something else (Subject, CN, etc). The error is not informative.. - Gilboa.

> On Wed, 2022-02-09 at 06:58 +0200, Gilboa Davara wrote: > > > > On Wed, Feb 9, 2022 at 1:05 AM Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; > wrote: > > I have no clue , but I would give vdsm.service a restart. > > > Thanks again for the prompt response. > Tried that, restarted all services and the all the VMS, didn't work. > > Any idea how I can verify the certificate information actually being used > by qemu for the spice console? > remote-viewer just fails, without giving any meaningful error message. > > - Gilboa > > > > Best Regards, > Strahil Nikolov > > On Tue, Feb 8, 2022 at 18:19, Gilboa Davara > <gilboad(a)gmail.com&gt; wrote: > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/2GAQH44QD6K... > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/OG57VT2XGDT... > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/AKQVBARD4EW... >