Good evening all,
I have a three host installation with a separate dedicated bare metal system for the engine, running Ovirt 4.5.2.4-1.el8.
This afternoon, the engine lost communication with one of the hosts. The engine log says the certificate is expired.
The official solution appears to be to put the host into maintenance mode then re-enroll it.
Unfortunately, because the certificate is expired, the engine cannot switch to maintenance mode or control the VM's to shut them down.
Error while executing action: Cannot switch Host to Maintenance mode.
Host still has running VMs on it and is in Non Responsive state.
See log excerpt below
What is the correct way to update/reinstate a certificate in a running cluster when the engine does not acknowledge the host is operational due to an expired certificate?
Thank you.
2023-07-20 16:27:46,904-05 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to /192.168.2.18
2023-07-20 16:27:46,904-05 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connected to /192.168.2.18:54321
2023-07-20 16:27:46,912-05 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Unable to process messages Received fatal alert: certificate_expired
2023-07-20 16:27:46,914-05 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-52) [] Unable to RefreshCapabilities: VDSNetworkException: VDSGenericException: VDSNetworkException: Received fatal alert: certificate_expired
2023-07-20 16:27:47,356-05 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-34) [] Unable to RefreshCapabilities: ClientConnectionException: SSL session is invalid
2023-07-20 16:27:47,356-05 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-34) [] Trying to release exclusive lock which does not exist, lock key: 'f69d35b2-7666-4ac6-8645-2f119cf2ce1cVDS_INIT'
2023-07-20 16:27:47,356-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-34) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@7d03f4f0'
2023-07-20 16:27:47,356-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-34) [] HostName = ovirt-host-03
2023-07-20 16:27:47,356-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-34) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = ovirt-host-03, VdsIdAndVdsVDSCommandParametersBase:{hostId='f69d35b2-7666-4ac6-8645-2f119cf2ce1c', vds='Host[ovirt-host-03,f69d35b2-7666-4ac6-8645-2f119cf2ce1c]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: SSL session is invalid