On Mon, Feb 7, 2022 at 1:27 PM Gilboa Davara <gilboad@gmail.com> wrote:
>
> Hello,
>
> On Mon, Feb 7, 2022 at 8:45 AM Yedidyah Bar David <didi@redhat.com> wrote:
>>
>> On Sun, Feb 6, 2022 at 5:09 PM Gilboa Davara <gilboad@gmail.com> wrote:
>> >
>> > Unlike my predecessor, I not only lost my vmengine, I also lost the vdsm services on all hosts.
>> > All seem to be hitting the same issue - read, the certs under /etc/pki/vdsm/certs and /etc/pki/ovirt* all expired a couple of days ago.
>> > As such, the hosted engine cannot go into global maintenance mode,
>>
>> What do you mean by that? What happens if you 'hosted-engine
>> --set-maintenance --mode=global'?
>
>
> Failed, stating the cluster is not in global maintenance mode.
Please clarify, and/or share relevant logs, if you have them.
Sadly enough, no.
When I zapped the old vmegine and hosts configuration, I forgot to save the logs.
(In my defense, it was 4am...)
That said, the fix proposed in BZ#1700460 (Let the user skip the global maintenance check) might have saved my cluster.
You had a semi-working existing HE cluster.
You ran engine-backup on it, took a backup, while it was _not_ in
global maintenance.
It was rather odd.
One of the hosts was still active and running the HE engine.
After I updated the apache certs, I could connect to the WebUI, but the WebUI failed to access the nodes, spewing SSL handshake errors.
I then processed to replace the hosts certs, which seems to work, (E.g. vdsm-client Host getCapabilities worked), hosted-engine --vm-status worked and I could see all 3 hosts, but the engine failed to communicate with the hosts, hence, even though I had a working cluster and engine, and I could get the cluster into global maintenance mode, engine-setup --offline continued to spew "not-in-global-maintenance-mode' errors.
At this stage I decided to simply zap the hosted engine and ovirt-hosted-engine-cleanup the hosts.
As my brain was half dead, I decided to do a fresh deployment, and not use the daily backup.
That's ok and expected.
Then you took one of the hosts and evacuated it (or just a new one),
(re)installed the OS (or somehow cleaned it up), and ran
'hosted-engine --deploy --import-from-file' with the backup you took.
This failed? Where exactly and with what error?
Didn't use the backup.
Clean hosted-engine --deploy failed due to qemu-6.1 failure. (I believe it's a known BZ#).
Once I remembered to downgrade it to 6.0, everything worked as advertised (minus one export domain, see another email).
If it's the engine-setup running inside the engine VM, with the same
error as when running 'engine-setup' (perhaps with --offline) manually,
then this shouldn't happen at this point:
- engine-backup --mode=restore sets vdc option in the db 'DbJustRestored'
- engine-setup checks this and sets its own env[JUST_RESTORED] accordingly
> (Understandable, given two of 3 hosts were offline due to certificate issues...)
>
>
>>
>>
>> > preventing engine-setup --offline from running.
>>
>> Actually just a few days ago I pushed a patch for:
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1700460
>>
>> But:
>>
>> If you really have a problem that you can't set global maintenance,
>> using this is a risk - HA might intervene in the middle and shutdown
>> the VM. So either make sure global maintenance does work, or stop
>> all HA services on all hosts.
>>
>> > Two questions:
>> > 1. Is there any automated method to renew the vdsm certificates?
>>
>> You mean, without an engine?
>>
>> I think that if you have a functional engine one way or another,
>> you can automate this somehow, didn't check. Try checking e.g. the
>> python sdk examples - there might be there something you can base
>> on.
>>
>> > 2. Assuming the previous answer is "no", assuming I'm somewhat versed in using openssl, how can I manually renew them?
>>
>> I'd rather not try to invent from memory how this is supposed to work,
>> and doing this methodically and verifying before replying is quite
>> an effort.
>>
>> If this is really what you want, I suggest something like:
>>
>> 1. Set up a test env with an engine and one host
>> 2. Backup (or use git on) /etc on both
>> 3. Renew the host cert from the UI
>> 4. Check what changed
>>
>> You should find, IMO, that the key(s) on the host didn't
>> change. I guess you might also find CSRs on one or both of them.
>> So basically it should be something like:
>> 1. Create a CSR on the host for the existing key (one or more,
>> not sure).
>> 2. Copy and sign this on the engine using pki-enroll-request.sh
>> (I think you can find examples for it scattered around, perhaps
>> even in the main guides)
>> 3. Copy back the generated certs to the host
>> 4. Perhaps restart one or more services there (vdsm, imageio?,
>> ovn, etc.)
>>
>> You can check the code in
>> /usr/share/ovirt-engine/ansible-runner-service-project/project
>> to see how it's done when initiated from the UI.
>>
>> Good luck and best regards,
>
>
> I more of less found a document stating the above somewhere in the middle of the night.
> Tried it.
> Got the WebUI working again.
> However, for the life of me I couldn't get the hosts to work to talk to the engine. (Even though I could use openssl s_client -showcerts -connect host and got valid certs).
> In the end, @around ~4am, I decided to take the brute force route, clean the hosts, upgrade them to -streams, and redeploy the engine again (3'rd attempt, after sufficient amount of coffee reminded me the qemu-6.1 is broken, and needed to be downgraded before trying to deploy the HE...).
> Either way, when I finish importing the VMs, I'll open a RFE to add BIG-WARNING-IN-BOLD-LETTERS in the WebUI to notify the admin that the certificates are about to expire.
You should have already received them, no?
https://bugzilla.redhat.com/show_bug.cgi?id=1258585
Best regards,
--
Didi