Hi Guys, We had a 2-node self-supported RHEV cluster that we used for our development environment. We're in the middle of a migration from RHEV over to VMware - one host has already been converted to VMware and we have actively been running VM migrations over to VMware. During the migration, we had an extended power outage and had to improperly shut down the RHEV host. Since bringing it back up we've not been able to start the hosted-engine. Each time we try to start the hosted engine we're getting the following messages: ## START [root@bmrhev01 ~]# hosted-engine --vm-start VM exists and is down, cleaning up and restarting VM in WaitForLaunch [root@bmrhev01 ~]# hosted-engine --vm-status --== Host bmrhev01 (id: 1) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : bmrhev01 Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down_unexpected", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 325ff4b3 local_conf_timestamp : 7920 Host timestamp : 7920 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=7920 (Tue Jun 15 14:59:32 2021) host-id=1 score=3400 vm_conf_refresh_time=7920 (Tue Jun 15 14:59:32 2021) conf_on_shared_storage=True maintenance=False state=EngineStarting stopped=False END ## We've tried rebooting a number of times. Restarting various oVirt services. Nothing appears to be resolving the issue. At this point, the only thing I truly care about is migrating the guest VMs over to VMware. There's nothing else in the RHEV environment I care about. I'm happy to settle with any of the following, providing they can salvage the guest VMs: a) Deleting and redeploying the hosted-engine. b) Abandoning the hosted-engine and some how converting the guest VMs over to VMware. I don't have enough experience to know if any of the above is possible or the repercussions of either of them. Any help from anyone would be very helpful. I'm sorry to be a leach, rather than a contributor. I'm seriously in trouble here and a kind heart would be much appreciated. Thanks, Tim.
If anyone else runs into this issue, I've been able to recover one of the VMs using the "qemu-img convert -O vmdk vmdiskfile vmdiskfile.vmdk" command outlined in Howard Johnson's response linked below: https://access.redhat.com/discussions/881153 Tried the same actions on a Linux machine, but CentOS is not loading, throwing errors saying it can't find the /root or /boot partitions. I'm hopeful that I can at least salvage a few of the Windows VMs currently held hostage in RHEV. The UUID folder structure is hard to navigate for someone that doesn't know what he's looking at (i.e. me) so I'm likely to have mixed results. Other articles I've come across suggest that it should be possible to redeploy the hosted-engine and import the existing Storage Domain, and the VMs should reappear. I've not had success with this because I haven't been able to make "IP.Address:/path/to/storage/domain" accessible, which is likely what faulted the original hosted-engine. Now that I've nuked the original hosted-engine, I'm stuck down the path I've created for myself, either a) continuing with "qemu-img convert", or b) figuring how to to make "cd IP.Address:/path/to/storage/domain" stop saying: "No such file or directory". If anyone has any tips, that would be much appreciated.
OK. To bring this to a close... There were some casualties, but ultimately I was able to salvage a dozen or so virtual machines using the "qemu-img convert -O vmdk vmdiskfile vmdiskfile.vmdk" command. In case anyone else finds themself in a similar situation, this involved the following steps: 1) Give up on the hosted-engine. 2) SSH into RHEV and navigate to the VM storage domain. 3) Go into each UUID folder and run # qemu-img convert -O vmdk vmdiskfile vmdiskfile.vmdk 4) Secure-copy vmdiskfile_converted.vmdk to VMware. 5) SSH onto the VMware host. 6) Run # vmkfstool -i vmdiskfile.vmdk new_file_name.vmdk. 7) Attach new_file_name.vmdk to a new VM in VMware. Make sure "Firmware" is set to "BIOS", under Edit Settings > VM Options > Boot Options. 8) Boot the VMware VM, and Windows should automatically load and you should be good to start removing the RHEV-related VM tools and install the VMware tools. ** Note: Remember my situation involved migrating away from RHEV over to VMware. So, I really wasn't concerned about recovering the hosted-engine. And, prior the hosted-engine going down, in preparation to migration to VMware I had already removed all snapshots from all RHEV VMs. This all made what's below possible, and a lot easier. Also, these steps didn't always work. I was able to recover 75% of Windows VMs. Unfortunately, I experienced a 100% lost for Linux VMs.
And why did you loose your Linux VMs ? The disks are still there unless you lost storage domain . Also, the restore procedure of the hosted engine is straight forward. Best Regards,Strahil Nikolov OK. To bring this to a close... There were some casualties, but ultimately I was able to salvage a dozen or so virtual machines using the "qemu-img convert -O vmdk vmdiskfile vmdiskfile.vmdk" command. In case anyone else finds themself in a similar situation, this involved the following steps: 1) Give up on the hosted-engine. 2) SSH into RHEV and navigate to the VM storage domain. 3) Go into each UUID folder and run # qemu-img convert -O vmdk vmdiskfile vmdiskfile.vmdk 4) Secure-copy vmdiskfile_converted.vmdk to VMware. 5) SSH onto the VMware host. 6) Run # vmkfstool -i vmdiskfile.vmdk new_file_name.vmdk. 7) Attach new_file_name.vmdk to a new VM in VMware. Make sure "Firmware" is set to "BIOS", under Edit Settings > VM Options > Boot Options. 8) Boot the VMware VM, and Windows should automatically load and you should be good to start removing the RHEV-related VM tools and install the VMware tools. ** Note: Remember my situation involved migrating away from RHEV over to VMware. So, I really wasn't concerned about recovering the hosted-engine. And, prior the hosted-engine going down, in preparation to migration to VMware I had already removed all snapshots from all RHEV VMs. This all made what's below possible, and a lot easier. Also, these steps didn't always work. I was able to recover 75% of Windows VMs. Unfortunately, I experienced a 100% lost for Linux VMs. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/YY7DIWFCOGKSOD...
participants (2)
-
Strahil Nikolov -
timothy.dilbert＠bmt.ky