<div dir="ltr"><div>It depends on how 'toasted' the engine is.<br><br></div>It is possible to mount the disk, if for example it's a kernel upgrade that's gone wrong and stopping the machine form booting<br></div><div class="gmail_extra"><br><div class="gmail_quote">On 3 July 2017 at 15:23, Yedidyah Bar David <span dir="ltr"><<a href="mailto:didi@redhat.com" target="_blank">didi@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Mon, Jul 3, 2017 at 4:40 PM, Andrew Dent <<a href="mailto:adent@ctcroydon.com.au">adent@ctcroydon.com.au</a>> wrote:<br>
> Hi Didi<br>
><br>
> Fair enough.<br>
> If I'm in this situation.....<br>
> I have 3 hosts with 6 production VMs.<br>
> The hosted-engine VM is completely toast and not recoverable.<br>
<br>
Meaning? It does not even boot? If so, then I am afraid that you<br>
need to follow the linked procedure, including deploying hosted-engine<br>
again on new (or fully cleaned) storage.<br>
<br>
Are all 3 hosts also hosted-engine hosts? And you want to keep them<br>
this way? If so, you'll have to reinstall them.<br>
<br>
I think, didn't try, that after you deploy a new hosted-engine host,<br>
and restore the engine, the new engine will see your existing hosts,<br>
but hosted-engine-ha will not work.<br>
<br>
If you want to try and keep your VMs running, I guess your best bet<br>
is to try to add a new non-hosted-engine host (one or more, as needed),<br>
migrate your VMs to it (them), then remove the hosted-engine hosts and<br>
deploy them again as described there.<br>
<br>
> However I have a backup of the hosted-engine database (do I need anything<br>
> else).<br>
<br>
Only database? Or a backup taken by engine-backup? You also need various<br>
configuration files. To see the full list, in case you didn't use<br>
engine-backup but can restore individual files, check BACKUP_PATHS inside<br>
the script. But this is definitely not supported - you should first try<br>
verifying by restoring on a test VM and see what happens.<br>
<br>
><br>
> Is it possible to build a new VM, import the backup of the previous<br>
> hosted-engine database and reconnect the storage domains and VMs in their<br>
> running state without any VMs experiencing an outage?<br>
<br>
As I wrote above, I think it is, but hosted-engine-ha will not work.<br>
<br>
If at all possible, I strongly suggest to try to simulate this in a<br>
test env. Can be done using nested-kvm VMs for hosts, if you do not have<br>
enough real hardware.<br>
<br>
><br>
> The URL<br>
> <a href="http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/" rel="noreferrer" target="_blank">http://www.ovirt.org/<wbr>documentation/self-hosted/<wbr>chap-Backing_up_and_Restoring_<wbr>an_EL-Based_Self-Hosted_<wbr>Environment/</a><br>
> looks to be longer now.<br>
> I'll review, test and see if this will give me what I'm looking for.<br>
><br>
> The broken link still seems to be broken.<br>
> When I click the link, the browser ends up at this<br>
> <a href="http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment#documentation%20self%20hosted%20chap%20Backing%20up%20and%20Restoring%20an%20EL%20Based%20Self%20Hosted%20Environment%20chap%20Installing%20Additional%20Hosts%20to%20a%20Self%20Hosted%20Environment" rel="noreferrer" target="_blank">http://www.ovirt.org/<wbr>documentation/self-hosted/<wbr>chap-Backing_up_and_Restoring_<wbr>an_EL-Based_Self-Hosted_<wbr>Environment/chap-Installing_<wbr>Additional_Hosts_to_a_Self-<wbr>Hosted_Environment#<wbr>documentation%20self%20hosted%<wbr>20chap%20Backing%20up%20and%<wbr>20Restoring%20an%20EL%20Based%<wbr>20Self%20Hosted%20Environment%<wbr>20chap%20Installing%<wbr>20Additional%20Hosts%20to%20a%<wbr>20Self%20Hosted%20Environment</a><br>
><br>
> But I suspect it should be<br>
> <a href="http://www.ovirt.org/documentation/self-hosted/chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment/" rel="noreferrer" target="_blank">http://www.ovirt.org/<wbr>documentation/self-hosted/<wbr>chap-Installing_Additional_<wbr>Hosts_to_a_Self-Hosted_<wbr>Environment/</a><br>
<br>
In my previous mail I just wrote I sent a pull request, not that<br>
the site is updated already. I updated it since and it now seems<br>
ok to me.<br>
<br>
Best,<br>
<br>
><br>
> Kind regards<br>
><br>
> Andrew<br>
><br>
> ------ Original Message ------<br>
> From: "Yedidyah Bar David" <<a href="mailto:didi@redhat.com">didi@redhat.com</a>><br>
> To: "Andrew Dent" <<a href="mailto:adent@ctcroydon.com.au">adent@ctcroydon.com.au</a>><br>
> Cc: "users" <<a href="mailto:users@ovirt.org">users@ovirt.org</a>><br>
> Sent: 3/07/2017 11:12:05 PM<br>
> Subject: Re: [ovirt-users] Recovering hosted-engine<br>
><br>
>> On Mon, Jul 3, 2017 at 3:46 PM, Andrew Dent <<a href="mailto:adent@ctcroydon.com.au">adent@ctcroydon.com.au</a>><br>
>> wrote:<br>
>>><br>
>>> Has anyone successfully completed a hosted-engine recovery on a multiple<br>
>>> host setup with production VMs?<br>
>><br>
>><br>
>> I'd like to clarify that "recovery" can span a large spectrum of<br>
>> flows, from a trivial "I did some change to the engine database<br>
>> that broke stuff and I want to restore a backup I took prior to<br>
>> this change" to a full system restoration including purchasing<br>
>> and deploying new (perhaps different) hosts/network/storage<br>
>> hardware, including many other flows in between.<br>
>><br>
>> So when you plan for recovery, you should define very well what<br>
>> flows you plan to handle, and how you handle each.<br>
>><br>
>> The linked procedure correctly says it's "providing an example".<br>
>><br>
>>><br>
>>> Kind regards<br>
>>><br>
>>><br>
>>> Andrew<br>
>>><br>
>>><br>
>>><br>
>>> ------ Original Message ------<br>
>>> From: "Andrew Dent" <<a href="mailto:adent@ctcroydon.com.au">adent@ctcroydon.com.au</a>><br>
>>> To: "users" <<a href="mailto:users@ovirt.org">users@ovirt.org</a>><br>
>>> Sent: 2/07/2017 2:22:16 PM<br>
>>> Subject: [ovirt-users] Recovering hosted-engine<br>
>>><br>
>>> Hi<br>
>>><br>
>>> A couple of questions about hosted-engine recovery.<br>
>>> Part way through this URL, in the section "Workflow for Restoring the<br>
>>> Self-Hosted Engine Environment"<br>
>>><br>
>>> <a href="http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/" rel="noreferrer" target="_blank">http://www.ovirt.org/<wbr>documentation/self-hosted/<wbr>chap-Backing_up_and_Restoring_<wbr>an_EL-Based_Self-Hosted_<wbr>Environment/</a><br>
>>> it looks like once the hosted-engine is recovered on Host 1, the VMs on<br>
>>> Host<br>
>>> 2 and 3 will be running, but not accessible to the recovered Hosted<br>
>>> Engine.<br>
>>> Is that correct?<br>
>><br>
>><br>
>> I am pretty certain that the procedure assumed that all hosts need<br>
>> restoration,<br>
>> not that some are still up-and-running.<br>
>><br>
>>> If so, how to you remove host 2 and host 3 from the environment, then<br>
>>> add<br>
>>> back in again while keeping the VMs running?<br>
>><br>
>><br>
>> That's a good question.<br>
>><br>
>> Please try to describe the exact flow you have in mind. What's broken and<br>
>> needs restoration, and how do you plan to do that?<br>
>><br>
>>><br>
>>> Host 2 and Host 3 are not recoverable in their current state. These<br>
>>> hosts<br>
>>> need to be removed from the environment, and then added again to the<br>
>>> environment using the hosted-engine deployment script. For more<br>
>>> information<br>
>>> on these actions, see the Removing Non-Operational Hosts from a Restored<br>
>>> Self-Hosted Engine Environment section below and Chapter 7: Installing<br>
>>> Additional Hosts to a Self-Hosted Environment.<br>
>>><br>
>>> BTW: The link referring to chapter 7 is broken.<br>
>><br>
>><br>
>> You are right. The link in the bottom of the page seems working.<br>
>> Now pushed [1] to fix. Thanks for the report!<br>
>><br>
>> [1]<br>
>><br>
>> Best,<br>
<span class="HOEnZb"><font color="#888888">>> --<br>
>> Didi<br>
><br>
><br>
<br>
<br>
<br>
--<br>
Didi<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
</font></span></blockquote></div><br></div>