[ovirt-users] Recovering hosted-engine

[Yedidyah Bar David]

On Mon, Jul 3, 2017 at 4:40 PM, Andrew Dent <adent at ctcroydon.com.au> wrote:
> Hi Didi
>
> Fair enough.
> If I'm in this situation.....
> I have 3 hosts with 6 production VMs.
> The hosted-engine VM is completely toast and not recoverable.

Meaning? It does not even boot? If so, then I am afraid that you
need to follow the linked procedure, including deploying hosted-engine
again on new (or fully cleaned) storage.

Are all 3 hosts also hosted-engine hosts? And you want to keep them
this way? If so, you'll have to reinstall them.

I think, didn't try, that after you deploy a new hosted-engine host,
and restore the engine, the new engine will see your existing hosts,
but hosted-engine-ha will not work.

If you want to try and keep your VMs running, I guess your best bet
is to try to add a new non-hosted-engine host (one or more, as needed),
migrate your VMs to it (them), then remove the hosted-engine hosts and
deploy them again as described there.

> However I have a backup of the hosted-engine database (do I need anything
> else).

Only database? Or a backup taken by engine-backup? You also need various
configuration files. To see the full list, in case you didn't use
engine-backup but can restore individual files, check BACKUP_PATHS inside
the script. But this is definitely not supported - you should first try
verifying by restoring on a test VM and see what happens.

>
> Is it possible to build a new VM, import the backup of the previous
> hosted-engine database and reconnect the storage domains and VMs in their
> running state without any VMs experiencing an outage?

As I wrote above, I think it is, but hosted-engine-ha will not work.

If at all possible, I strongly suggest to try to simulate this in a
test env. Can be done using nested-kvm VMs for hosts, if you do not have
enough real hardware.

>
> The URL
> http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/
> looks to be longer now.
> I'll review, test and see if this will give me what I'm looking for.
>
> The broken link still seems to be broken.
> When I click the link, the browser ends up at this
> http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment#documentation%20self%20hosted%20chap%20Backing%20up%20and%20Restoring%20an%20EL%20Based%20Self%20Hosted%20Environment%20chap%20Installing%20Additional%20Hosts%20to%20a%20Self%20Hosted%20Environment
>
> But I suspect it should be
> http://www.ovirt.org/documentation/self-hosted/chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment/

In my previous mail I just wrote I sent a pull request, not that
the site is updated already. I updated it since and it now seems
ok to me.

Best,

>
> Kind regards
>
> Andrew
>
> ------ Original Message ------
> From: "Yedidyah Bar David" <didi at redhat.com>
> To: "Andrew Dent" <adent at ctcroydon.com.au>
> Cc: "users" <users at ovirt.org>
> Sent: 3/07/2017 11:12:05 PM
> Subject: Re: [ovirt-users] Recovering hosted-engine
>
>> On Mon, Jul 3, 2017 at 3:46 PM, Andrew Dent <adent at ctcroydon.com.au>
>> wrote:
>>>
>>>  Has anyone successfully completed a hosted-engine recovery on a multiple
>>>  host setup with production VMs?
>>
>>
>> I'd like to clarify that "recovery" can span a large spectrum of
>> flows, from a trivial "I did some change to the engine database
>> that broke stuff and I want to restore a backup I took prior to
>> this change" to a full system restoration including purchasing
>> and deploying new (perhaps different) hosts/network/storage
>> hardware, including many other flows in between.
>>
>> So when you plan for recovery, you should define very well what
>> flows you plan to handle, and how you handle each.
>>
>> The linked procedure correctly says it's "providing an example".
>>
>>>
>>>  Kind regards
>>>
>>>
>>>  Andrew
>>>
>>>
>>>
>>>  ------ Original Message ------
>>>  From: "Andrew Dent" <adent at ctcroydon.com.au>
>>>  To: "users" <users at ovirt.org>
>>>  Sent: 2/07/2017 2:22:16 PM
>>>  Subject: [ovirt-users] Recovering hosted-engine
>>>
>>>  Hi
>>>
>>>  A couple of questions about hosted-engine recovery.
>>>  Part way through this URL, in the section "Workflow for Restoring the
>>>  Self-Hosted Engine Environment"
>>>
>>> http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/
>>>  it looks like once the hosted-engine is recovered on Host 1, the VMs on
>>> Host
>>>  2 and 3 will be running, but not accessible to the recovered Hosted
>>> Engine.
>>>  Is that correct?
>>
>>
>> I am pretty certain that the procedure assumed that all hosts need
>> restoration,
>> not that some are still up-and-running.
>>
>>>  If so, how to you remove host 2 and host 3 from the environment, then
>>> add
>>>  back in again while keeping the VMs running?
>>
>>
>> That's a good question.
>>
>> Please try to describe the exact flow you have in mind. What's broken and
>> needs restoration, and how do you plan to do that?
>>
>>>
>>>  Host 2 and Host 3 are not recoverable in their current state. These
>>> hosts
>>>  need to be removed from the environment, and then added again to the
>>>  environment using the hosted-engine deployment script. For more
>>> information
>>>  on these actions, see the Removing Non-Operational Hosts from a Restored
>>>  Self-Hosted Engine Environment section below and Chapter 7: Installing
>>>  Additional Hosts to a Self-Hosted Environment.
>>>
>>>  BTW: The link referring to chapter 7 is broken.
>>
>>
>> You are right. The link in the bottom of the page seems working.
>> Now pushed [1] to fix. Thanks for the report!
>>
>> [1]
>>
>> Best,
>> --
>> Didi
>
>



-- 
Didi