3:05 p.m.
Hi Guys,
We had a 2-node self-supported RHEV cluster that we used for our development environment.
We're in the middle of a migration from RHEV over to VMware - one host has already
been converted to VMware and we have actively been running VM migrations over to VMware.
During the migration, we had an extended power outage and had to improperly shut down the
RHEV host. Since bringing it back up we've not been able to start the hosted-engine.
Each time we try to start the hosted engine we're getting the following messages:
## START
[root@bmrhev01 ~]# hosted-engine --vm-start
VM exists and is down, cleaning up and restarting
VM in WaitForLaunch
[root@bmrhev01 ~]# hosted-engine --vm-status
--== Host bmrhev01 (id: 1) status ==--
conf_on_shared_storage : True
Status up-to-date : True
Hostname : bmrhev01
Host ID : 1
Engine status : {"reason": "vm not running on this
host", "health": "bad", "vm":
"down_unexpected", "detail": "unknown"}
Score : 3400
stopped : False
Local maintenance : False
crc32 : 325ff4b3
local_conf_timestamp : 7920
Host timestamp : 7920
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=7920 (Tue Jun 15 14:59:32 2021)
host-id=1
score=3400
vm_conf_refresh_time=7920 (Tue Jun 15 14:59:32 2021)
conf_on_shared_storage=True
maintenance=False
state=EngineStarting
stopped=False
END ##
We've tried rebooting a number of times. Restarting various oVirt services. Nothing
appears to be resolving the issue. At this point, the only thing I truly care about is
migrating the guest VMs over to VMware. There's nothing else in the RHEV environment I
care about. I'm happy to settle with any of the following, providing they can salvage
the guest VMs:
a) Deleting and redeploying the hosted-engine.
b) Abandoning the hosted-engine and some how converting the guest VMs over to VMware.
I don't have enough experience to know if any of the above is possible or the
repercussions of either of them. Any help from anyone would be very helpful. I'm sorry
to be a leach, rather than a contributor. I'm seriously in trouble here and a kind
heart would be much appreciated.
Thanks, Tim.
1:32 p.m.
If anyone else runs into this issue, I've been able to recover one of the VMs using
the "qemu-img convert -O vmdk vmdiskfile vmdiskfile.vmdk" command outlined in
Howard Johnson's response linked below:
https://access.redhat.com/discussions/881153
Tried the same actions on a Linux machine, but CentOS is not loading, throwing errors
saying it can't find the /root or /boot partitions. I'm hopeful that I can at
least salvage a few of the Windows VMs currently held hostage in RHEV. The UUID folder
structure is hard to navigate for someone that doesn't know what he's looking at
(i.e. me) so I'm likely to have mixed results.
Other articles I've come across suggest that it should be possible to redeploy the
hosted-engine and import the existing Storage Domain, and the VMs should reappear.
I've not had success with this because I haven't been able to make
"IP.Address:/path/to/storage/domain" accessible, which is likely what faulted
the original hosted-engine.
Now that I've nuked the original hosted-engine, I'm stuck down the path I've
created for myself, either a) continuing with "qemu-img convert", or b) figuring
how to to make "cd IP.Address:/path/to/storage/domain" stop saying: "No
such file or directory".
If anyone has any tips, that would be much appreciated.
9:26 a.m.
OK. To bring this to a close...
There were some casualties, but ultimately I was able to salvage a dozen or so virtual
machines using the "qemu-img convert -O vmdk vmdiskfile vmdiskfile.vmdk"
command. In case anyone else finds themself in a similar situation, this involved the
following steps:
1) Give up on the hosted-engine.
2) SSH into RHEV and navigate to the VM storage domain.
3) Go into each UUID folder and run # qemu-img convert -O vmdk vmdiskfile vmdiskfile.vmdk
4) Secure-copy vmdiskfile_converted.vmdk to VMware.
5) SSH onto the VMware host.
6) Run # vmkfstool -i vmdiskfile.vmdk new_file_name.vmdk.
7) Attach new_file_name.vmdk to a new VM in VMware. Make sure "Firmware" is set
to "BIOS", under Edit Settings > VM Options > Boot Options.
8) Boot the VMware VM, and Windows should automatically load and you should be good to
start removing the RHEV-related VM tools and install the VMware tools.
** Note: Remember my situation involved migrating away from RHEV over to VMware. So, I
really wasn't concerned about recovering the hosted-engine. And, prior the
hosted-engine going down, in preparation to migration to VMware I had already removed all
snapshots from all RHEV VMs. This all made what's below possible, and a lot easier.
Also, these steps didn't always work. I was able to recover 75% of Windows VMs.
Unfortunately, I experienced a 100% lost for Linux VMs.
1 p.m.
And why did you loose your Linux VMs ?
The disks are still there unless you lost storage domain .
Also, the restore procedure of the hosted engine is straight forward.
Best Regards,Strahil Nikolov
OK. To bring this to a close...
There were some casualties, but ultimately I was able to salvage a dozen or so virtual
machines using the "qemu-img convert -O vmdk vmdiskfile vmdiskfile.vmdk"
command. In case anyone else finds themself in a similar situation, this involved the
following steps:
1) Give up on the hosted-engine.
2) SSH into RHEV and navigate to the VM storage domain.
3) Go into each UUID folder and run # qemu-img convert -O vmdk vmdiskfile vmdiskfile.vmdk
4) Secure-copy vmdiskfile_converted.vmdk to VMware.
5) SSH onto the VMware host.
6) Run # vmkfstool -i vmdiskfile.vmdk new_file_name.vmdk.
7) Attach new_file_name.vmdk to a new VM in VMware. Make sure "Firmware" is set
to "BIOS", under Edit Settings > VM Options > Boot Options.
8) Boot the VMware VM, and Windows should automatically load and you should be good to
start removing the RHEV-related VM tools and install the VMware tools.
** Note: Remember my situation involved migrating away from RHEV over to VMware. So, I
really wasn't concerned about recovering the hosted-engine. And, prior the
hosted-engine going down, in preparation to migration to VMware I had already removed all
snapshots from all RHEV VMs. This all made what's below possible, and a lot easier.
Also, these steps didn't always work. I was able to recover 75% of Windows VMs.
Unfortunately, I experienced a 100% lost for Linux VMs.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/YY7DIWFCOGK...
1253
days inactive
1259
days old
3 comments
2 participants
participants (2)
-
Strahil Nikolov -
timothy.dilbert＠bmt.ky