On Mon, Feb 7, 2022 at 12:33 PM Gilboa Davara <gilboad(a)gmail.com> wrote:
> Hello,
>
> On Mon, Feb 7, 2022 at 8:45 AM Yedidyah Bar David <didi(a)redhat.com>
> wrote:
>
>> On Sun, Feb 6, 2022 at 5:09 PM Gilboa Davara <gilboad(a)gmail.com> wrote:
>> >
>> > Unlike my predecessor, I not only lost my vmengine, I also lost the
>> vdsm services on all hosts.
>> > All seem to be hitting the same issue - read, the certs under
>> /etc/pki/vdsm/certs and /etc/pki/ovirt* all expired a couple of days ago.
>> > As such, the hosted engine cannot go into global maintenance mode,
>>
>> What do you mean by that? What happens if you 'hosted-engine
>> --set-maintenance --mode=global'?
>>
>
> Failed, stating the cluster is not in global maintenance mode.
> (Understandable, given two of 3 hosts were offline due to certificate
> issues...)
>
>
>
>>
>> > preventing engine-setup --offline from running.
>>
>> Actually just a few days ago I pushed a patch for:
>>
>>
https://bugzilla.redhat.com/show_bug.cgi?id=1700460
>>
>> But:
>>
>> If you really have a problem that you can't set global maintenance,
>> using this is a risk - HA might intervene in the middle and shutdown
>> the VM. So either make sure global maintenance does work, or stop
>> all HA services on all hosts.
>>
>> > Two questions:
>> > 1. Is there any automated method to renew the vdsm certificates?
>>
>> You mean, without an engine?
>>
>> I think that if you have a functional engine one way or another,
>> you can automate this somehow, didn't check. Try checking e.g. the
>> python sdk examples - there might be there something you can base
>> on.
>>
>> > 2. Assuming the previous answer is "no", assuming I'm somewhat
versed
>> in using openssl, how can I manually renew them?
>>
>> I'd rather not try to invent from memory how this is supposed to work,
>> and doing this methodically and verifying before replying is quite
>> an effort.
>>
>> If this is really what you want, I suggest something like:
>>
>> 1. Set up a test env with an engine and one host
>> 2. Backup (or use git on) /etc on both
>> 3. Renew the host cert from the UI
>> 4. Check what changed
>>
>> You should find, IMO, that the key(s) on the host didn't
>> change. I guess you might also find CSRs on one or both of them.
>> So basically it should be something like:
>> 1. Create a CSR on the host for the existing key (one or more,
>> not sure).
>> 2. Copy and sign this on the engine using pki-enroll-request.sh
>> (I think you can find examples for it scattered around, perhaps
>> even in the main guides)
>> 3. Copy back the generated certs to the host
>> 4. Perhaps restart one or more services there (vdsm, imageio?,
>> ovn, etc.)
>>
>> You can check the code in
>> /usr/share/ovirt-engine/ansible-runner-service-project/project
>> to see how it's done when initiated from the UI.
>>
>> Good luck and best regards,
>>
>
> I more of less found a document stating the above somewhere in the middle
> of the night.
> Tried it.
> Got the WebUI working again.
> However, for the life of me I couldn't get the hosts to work to talk to
> the engine. (Even though I could use openssl s_client -showcerts -connect
> host and got valid certs).
> In the end, @around ~4am, I decided to take the brute force route, clean
> the hosts, upgrade them to -streams, and redeploy the engine again (3'rd
> attempt, after sufficient amount of coffee reminded me the qemu-6.1 is
> broken, and needed to be downgraded before trying to deploy the HE...).
> Either way, when I finish importing the VMs, I'll open a RFE to add
> BIG-WARNING-IN-BOLD-LETTERS in the WebUI to notify the admin that the
> certificates are about to expire.
>
We already have quite a lot of warnings/alters about certificates which
are going to expire soon:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TMJVAJMH5MK...
So what exactly are you missing here?
I don't know how, but the only errors I saw in the WebUI were update
related (failed to check updates on host).
There was an error in engine-setup, but at this stage it was far, far too
late.
- Gilboa
> Thanks for the help!
>
> - Gilboa
>
>
>
>> --
>> Didi
>>
>> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement:
https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
>
https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/CGSAB7NPWWO...
>
--
Martin Perina
Manager, Software Engineering
Red Hat Czech s.r.o.