I know this is old/dated, but I recently came across a similar situation and was able to
resolve it. Thought I'd share my steps in case it helps someone else.
In my environment, I had a 4 node cluster and the apache cert (and host certs) had
expired. The hosted-engine cert was still valid.
I had two hosts (node B and node D) that would not go into Global Maintenance as they
could not connect to the HA-agent. Running engine-setup from within the hosted engine
would error out that the cluster was not in Global Maintenance. I was able to tell Hosted
Engine to forget about those two hosts.
When you initiate "hosted-engine --vm-status" pay attention to the host ID of
the hosts that aren't in Global Maintenance.
To remove the hosts that were unable to receive up-to-date config (global maintenance)
from the hosted-engine availability list, I executed the following from a host that IS
showing Global Maintenance (node C):
'hosted-engine --clean-metadata --host-id=<host_id> --force-clean'
eg: 'hosted-engine --clean-metadata --host-id=4 --force-clean'
Now "hosted-engine --vm-status" will be consistent. However, in my case, this
_still_ did not allow me to run engine-setup & continued to error out, saying that
Global Maintenance was not set.
Knowing that the remaining hosts were in fact in global maintenance, I issued:
engine-setup --otopi-environment=OVESETUP_CONFIG/continueSetupOnHEVM=bool:True --offline
This allowed the setup to execute properly and I was able to renew the certs and configure
the hosts accordingly. The hosts that I had cleared the meta-data from previously, were
automatically re-added once I exited global maintenance.
This cluster was properly recovered while still serving guests.
-Andrew