On Wed, Oct 5, 2016 at 10:30 AM, <aleksey.maksimov@it-kb.ru> wrote:

Well. Then, in the case of conditions:

1) the vm is not available anymore due to storage corruption 2) an empty shared storage is available 3) engine backup exists 4) all VMs still running on the hosts in the cluster

The recovery plan will be like this (as I understand it):

1) On all the hosts (if they are still available):

# service ovirt-ha-broker stop # service ovirt-ha-agent stop # chkconfig --del ovirt-ha-broker # chkconfig --del ovirt-ha-agent

2) On first host (if the original host is not available anymore, provision a new host from scratch and proceed on this new host):

2.1) # hosted-engine --deploy

◾use same fqdn you had previously in the HE VM. ◾point to the new shared storage ◾provide the same admin password you used in previous setup ◾install the OS on the vm

I'd suggest to use the engine appliance also for this. You can just say No when it asks about automatically running engine-setup.

◾confirm it has been installed

on Hosted Engine VM:

a) Install the ovirt-engine rpms on the vm but don't run engine-setup: # yum install http://resources.ovirt.org/pub/yum-repo/ovirt-release40. rpm # yum install epel-release # yum install ovirt-engine b) Restore the backup: # engine-backup --mode=restore --file=file_name --log=log_file_name --provision-db --provision-dwh-db --restore-permissions

In order to let the engine auto-import the new hosted-engine storage domain, you have to remove the old one. The same for the engine VM. Unfortunately you cannot do that from the engine since they are somehow protected to avoid unintentional damages. The easiest way is to remove them from the DB before running engine-setup. I'm working on a helper utility to make it easiser: https://gerrit.ovirt.org/#/c/64966/ I think I'll integrate it with engine-backup to simply do it with an additional CLI flag.

c) Run "engine-setup"

2.2) Open Administration Portal and remove the all old hosts used for Hosted Engine

Right, we can also integrate this step in the HE cleaning helper.

2.3) Confirm that the engine has been installed (Return to the host and continue the hosted-engine deployment script by selecting option 1) and then finish the deploy.

2.4) In Administration Portal activate new host

3) On all additional hosts run "hosted-engine --deploy".

I strongly suggest to deploy them from the engine and not from CLI. CLI deploy support for additional HE host is deprecated an it will be removed in 4.1.

Right?

On Wed, Oct 5, 2016 at 11:56 AM, <aleksey.maksimov@it-kb.ru> wrote:

Weird. The RHV4 guides not contain the information that we need to clean the database from old storage domain before running the command engine-setup. What specific actions do we need?

You can check this as a reference but take care because it's still not fully tested: https://gerrit.ovirt.org/#/c/64966/3/packaging/setup/dbutils/hecleaner_sp.sq...

Eventually I want to get a full recovery plan at the moment for oVirt 4.0.

05.10.2016, 12:07, "Simone Tiraboschi" <stirabos@redhat.com>:

On Wed, Oct 5, 2016 at 10:30 AM, <aleksey.maksimov@it-kb.ru> wrote:

Well. Then, in the case of conditions:

1) the vm is not available anymore due to storage corruption 2) an empty shared storage is available 3) engine backup exists 4) all VMs still running on the hosts in the cluster

The recovery plan will be like this (as I understand it):

1) On all the hosts (if they are still available):

# service ovirt-ha-broker stop # service ovirt-ha-agent stop # chkconfig --del ovirt-ha-broker # chkconfig --del ovirt-ha-agent

2) On first host (if the original host is not available anymore, provision a new host from scratch and proceed on this new host):

2.1) # hosted-engine --deploy

◾use same fqdn you had previously in the HE VM. ◾point to the new shared storage ◾provide the same admin password you used in previous setup ◾install the OS on the vm

I'd suggest to use the engine appliance also for this. You can just say No when it asks about automatically running engine-setup.

◾confirm it has been installed

on Hosted Engine VM:

a) Install the ovirt-engine rpms on the vm but don't run engine-setup: # yum install http://resources.ovirt.org/pub/yum-repo/ovirt-release40. rpm # yum install epel-release # yum install ovirt-engine b) Restore the backup: # engine-backup --mode=restore --file=file_name --log=log_file_name --provision-db --provision-dwh-db --restore-permissions

In order to let the engine auto-import the new hosted-engine storage domain, you have to remove the old one. The same for the engine VM. Unfortunately you cannot do that from the engine since they are somehow protected to avoid unintentional damages. The easiest way is to remove them from the DB before running engine-setup. I'm working on a helper utility to make it easiser: https://gerrit.ovirt.org/#/c/64966/ I think I'll integrate it with engine-backup to simply do it with an additional CLI flag.

c) Run "engine-setup"

2.2) Open Administration Portal and remove the all old hosts used for Hosted Engine

Right, we can also integrate this step in the HE cleaning helper.

2.3) Confirm that the engine has been installed (Return to the host and continue the hosted-engine deployment script by selecting option 1) and then finish the deploy.

2.4) In Administration Portal activate new host

3) On all additional hosts run "hosted-engine --deploy".

I strongly suggest to deploy them from the engine and not from CLI. CLI deploy support for additional HE host is deprecated an it will be removed in 4.1.

Right?

On Wed, Oct 5, 2016 at 12:40 PM, <aleksey.maksimov@it-kb.ru> wrote:

Ouch. It is beyond my understanding.

Thus, it appears that described in the RHV4 guide ( https://access.redhat.com/documentation/en/red-hat- virtualization/4.0/single/self-hosted-engine-guide/# sect-Restoring_SHE_bkup) recovery procedure in fact incomplete?

Yes, you are right although this is a kind of special case since we are moving/restoring to a different storage domain while you are not asked to remove the old storage if you are restoring in place.

On Thu, Oct 6, 2016 at 7:32 AM, <aleksey.maksimov@it-kb.ru> wrote:

Hi Simone. When can we expect a new version of the engine-backup with built-in cleaning helper?

That bug is targeted to 4.1

05.10.2016, 13:52, "Simone Tiraboschi" <stirabos@redhat.com>:

...

On Wed, Oct 5, 2016 at 12:40 PM, <aleksey.maksimov@it-kb.ru> wrote:

...

Ouch. It is beyond my understanding.

Thus, it appears that described in the RHV4 guide ( https://access.redhat.com/documentation/en/red-hat- virtualization/4.0/single/self-hosted-engine-guide/# sect-Restoring_SHE_bkup) recovery procedure in fact incomplete? Yes, you are right although this is a kind of special case since we are moving/restoring to a different storage domain while you are not asked to remove the old storage if you are restoring in place.