[ovirt-users] Unable to remove host from ovirt engine

Recently our host and ovirt engine certificates expired and with some ideas from Strahil we were able to get 2 of the 3 ovirt hosts updated with usable certificates and move all of our VMs to those two nodes. https://lists.ovirt.org/archives/list/users@ovirt.org/thread/QCFPKQ3OKPOU... Not having any luck with the last host we figured we'd just try to remove it from ovirt engine and re-add it. While it seems `hosted-engine --vm-status` on one node no longer shows the removed host, the other good host and the web interface still show ovirt-1 in the mix. What is the best way to remove a NonRespnsive host from ovirt and re-add it? [root@ovirt-1 ~]# hosted-engine --vm-status The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. [root@ovirt-2 ~]# hosted-engine --vm-status !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host ovirt-3.xxxxx.com (id: 2) status ==-- Host ID : 2 Host timestamp : 12515451 Score : 3274 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : ovirt-3.xxxxx.com Local maintenance : False stopped : False crc32 : 9cf92792 conf_on_shared_storage : True local_conf_timestamp : 12515451 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=12515451 (Mon Apr 25 14:08:51 2022) host-id=2 score=3274 vm_conf_refresh_time=12515451 (Mon Apr 25 14:08:51 2022) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False --== Host ovirt-2.xxxxx.com (id: 3) status ==-- Host ID : 3 Host timestamp : 12513269 Score : 3400 Engine status : {"vm": "up", "health": "good", "detail": "Up"} Hostname : ovirt-2.xxxxx.com Local maintenance : False stopped : False crc32 : 4a89d706 conf_on_shared_storage : True local_conf_timestamp : 12513269 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=12513269 (Mon Apr 25 14:09:00 2022) host-id=3 score=3400 vm_conf_refresh_time=12513269 (Mon Apr 25 14:09:00 2022) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False !! Cluster is in GLOBAL MAINTENANCE mode !! [root@ovirt-3 ~]# hosted-engine --vm-status !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host ovirt-1.xxxxx.com (id: 1) status ==-- Host ID : 1 Host timestamp : 6750990 Score : 0 Engine status : unknown stale-data Hostname : ovirt-1.xxxxx.com Local maintenance : False stopped : True crc32 : 5290657b conf_on_shared_storage : True local_conf_timestamp : 6750950 Status up-to-date : False Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=6750990 (Thu Feb 17 22:17:53 2022) host-id=1 score=0 vm_conf_refresh_time=6750950 (Thu Feb 17 22:17:12 2022) conf_on_shared_storage=True maintenance=False state=AgentStopped stopped=True --== Host ovirt-3.xxxxx.com (id: 2) status ==-- Host ID : 2 Host timestamp : 12515501 Score : 3279 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : ovirt-3.xxxxx.com Local maintenance : False stopped : False crc32 : 0845cd93 conf_on_shared_storage : True local_conf_timestamp : 12515501 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=12515501 (Mon Apr 25 14:09:42 2022) host-id=2 score=3279 vm_conf_refresh_time=12515501 (Mon Apr 25 14:09:42 2022) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False --== Host ovirt-2.xxxxx.com (id: 3) status ==-- Host ID : 3 Host timestamp : 12513309 Score : 3400 Engine status : {"vm": "up", "health": "good", "detail": "Up"} Hostname : ovirt-2.xxxxx.com Local maintenance : False stopped : False crc32 : 237726e0 conf_on_shared_storage : True local_conf_timestamp : 12513309 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=12513309 (Mon Apr 25 14:09:39 2022) host-id=3 score=3400 vm_conf_refresh_time=12513309 (Mon Apr 25 14:09:39 2022) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False !! Cluster is in GLOBAL MAINTENANCE mode !!