Recovering from corrupted hosted-engine
by timothy.dilbert@bmt.ky
Hi Guys,
We had a 2-node self-supported RHEV cluster that we used for our development environment. We're in the middle of a migration from RHEV over to VMware - one host has already been converted to VMware and we have actively been running VM migrations over to VMware. During the migration, we had an extended power outage and had to improperly shut down the RHEV host. Since bringing it back up we've not been able to start the hosted-engine. Each time we try to start the hosted engine we're getting the following messages:
## START
[root@bmrhev01 ~]# hosted-engine --vm-start
VM exists and is down, cleaning up and restarting
VM in WaitForLaunch
[root@bmrhev01 ~]# hosted-engine --vm-status
--== Host bmrhev01 (id: 1) status ==--
conf_on_shared_storage : True
Status up-to-date : True
Hostname : bmrhev01
Host ID : 1
Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down_unexpected", "detail": "unknown"}
Score : 3400
stopped : False
Local maintenance : False
crc32 : 325ff4b3
local_conf_timestamp : 7920
Host timestamp : 7920
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=7920 (Tue Jun 15 14:59:32 2021)
host-id=1
score=3400
vm_conf_refresh_time=7920 (Tue Jun 15 14:59:32 2021)
conf_on_shared_storage=True
maintenance=False
state=EngineStarting
stopped=False
END ##
We've tried rebooting a number of times. Restarting various oVirt services. Nothing appears to be resolving the issue. At this point, the only thing I truly care about is migrating the guest VMs over to VMware. There's nothing else in the RHEV environment I care about. I'm happy to settle with any of the following, providing they can salvage the guest VMs:
a) Deleting and redeploying the hosted-engine.
b) Abandoning the hosted-engine and some how converting the guest VMs over to VMware.
I don't have enough experience to know if any of the above is possible or the repercussions of either of them. Any help from anyone would be very helpful. I'm sorry to be a leach, rather than a contributor. I'm seriously in trouble here and a kind heart would be much appreciated.
Thanks, Tim.