Re: [ovirt-users] Recovering hosted-engine
Hi Didi Fair enough. If I'm in this situation..... I have 3 hosts with 6 production VMs. The hosted-engine VM is completely toast and not recoverable. However I have a backup of the hosted-engine database (do I need anything else). Is it possible to build a new VM, import the backup of the previous hosted-engine database and reconnect the storage domains and VMs in their running state without any VMs experiencing an outage? The URL http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring... looks to be longer now. I'll review, test and see if this will give me what I'm looking for. The broken link still seems to be broken. When I click the link, the browser ends up at this http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring... But I suspect it should be http://www.ovirt.org/documentation/self-hosted/chap-Installing_Additional_Ho... Kind regards Andrew ------ Original Message ------ From: "Yedidyah Bar David" <didi@redhat.com> To: "Andrew Dent" <adent@ctcroydon.com.au> Cc: "users" <users@ovirt.org> Sent: 3/07/2017 11:12:05 PM Subject: Re: [ovirt-users] Recovering hosted-engine
On Mon, Jul 3, 2017 at 3:46 PM, Andrew Dent <adent@ctcroydon.com.au> wrote:
Has anyone successfully completed a hosted-engine recovery on a multiple host setup with production VMs?
I'd like to clarify that "recovery" can span a large spectrum of flows, from a trivial "I did some change to the engine database that broke stuff and I want to restore a backup I took prior to this change" to a full system restoration including purchasing and deploying new (perhaps different) hosts/network/storage hardware, including many other flows in between.
So when you plan for recovery, you should define very well what flows you plan to handle, and how you handle each.
The linked procedure correctly says it's "providing an example".
Kind regards
Andrew
------ Original Message ------ From: "Andrew Dent" <adent@ctcroydon.com.au> To: "users" <users@ovirt.org> Sent: 2/07/2017 2:22:16 PM Subject: [ovirt-users] Recovering hosted-engine
Hi
A couple of questions about hosted-engine recovery. Part way through this URL, in the section "Workflow for Restoring the Self-Hosted Engine Environment"
http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring... it looks like once the hosted-engine is recovered on Host 1, the VMs on Host 2 and 3 will be running, but not accessible to the recovered Hosted Engine. Is that correct?
I am pretty certain that the procedure assumed that all hosts need restoration, not that some are still up-and-running.
If so, how to you remove host 2 and host 3 from the environment, then add back in again while keeping the VMs running?
That's a good question.
Please try to describe the exact flow you have in mind. What's broken and needs restoration, and how do you plan to do that?
Host 2 and Host 3 are not recoverable in their current state. These hosts need to be removed from the environment, and then added again to the environment using the hosted-engine deployment script. For more information on these actions, see the Removing Non-Operational Hosts from a Restored Self-Hosted Engine Environment section below and Chapter 7: Installing Additional Hosts to a Self-Hosted Environment.
BTW: The link referring to chapter 7 is broken.
You are right. The link in the bottom of the page seems working. Now pushed [1] to fix. Thanks for the report!
[1]
Best, -- Didi
Hi andrew, This is my personal experience on recovery. Yes you can recover the hosted-engine from a backup (almost every action you did on on oVirt after backup will have to be manually reproduced) This can be tricky depending on your conditions (and there are some subtleties which can have significant impact later on) - Do you have still SSH access to engine ? - If so, do you trust engine VM OS, or should you reinstall it ? If both questions are positive, then you just need to do a engine restore as explained here : http://www.ovirt.org/documentation/admin-guide/chap-Backups_and_Migration/ cordialement, regards, Nelson LAMEIRAS Ingénieur Systèmes et Réseaux / Systems and Networks engineer Tel: +33 5 32 09 09 70 nelson.lameiras@lyra-network.com www.lyra-network.com | www.payzen.eu Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE ----- Original Message ----- From: "Andrew Dent" <adent@ctcroydon.com.au> To: "Yedidyah Bar David" <didi@redhat.com>, "users" <users@ovirt.org> Sent: Monday, July 3, 2017 3:40:38 PM Subject: Re: [ovirt-users] Recovering hosted-engine Hi Didi Fair enough. If I'm in this situation..... I have 3 hosts with 6 production VMs. The hosted-engine VM is completely toast and not recoverable. However I have a backup of the hosted-engine database (do I need anything else). Is it possible to build a new VM, import the backup of the previous hosted-engine database and reconnect the storage domains and VMs in their running state without any VMs experiencing an outage? The URL http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring... looks to be longer now. I'll review, test and see if this will give me what I'm looking for. The broken link still seems to be broken. When I click the link, the browser ends up at this http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring... But I suspect it should be http://www.ovirt.org/documentation/self-hosted/chap-Installing_Additional_Ho... Kind regards Andrew ------ Original Message ------ From: "Yedidyah Bar David" <didi@redhat.com> To: "Andrew Dent" <adent@ctcroydon.com.au> Cc: "users" <users@ovirt.org> Sent: 3/07/2017 11:12:05 PM Subject: Re: [ovirt-users] Recovering hosted-engine
On Mon, Jul 3, 2017 at 3:46 PM, Andrew Dent <adent@ctcroydon.com.au> wrote:
Has anyone successfully completed a hosted-engine recovery on a multiple host setup with production VMs?
I'd like to clarify that "recovery" can span a large spectrum of flows, from a trivial "I did some change to the engine database that broke stuff and I want to restore a backup I took prior to this change" to a full system restoration including purchasing and deploying new (perhaps different) hosts/network/storage hardware, including many other flows in between.
So when you plan for recovery, you should define very well what flows you plan to handle, and how you handle each.
The linked procedure correctly says it's "providing an example".
Kind regards
Andrew
------ Original Message ------ From: "Andrew Dent" <adent@ctcroydon.com.au> To: "users" <users@ovirt.org> Sent: 2/07/2017 2:22:16 PM Subject: [ovirt-users] Recovering hosted-engine
Hi
A couple of questions about hosted-engine recovery. Part way through this URL, in the section "Workflow for Restoring the Self-Hosted Engine Environment"
http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring... it looks like once the hosted-engine is recovered on Host 1, the VMs on Host 2 and 3 will be running, but not accessible to the recovered Hosted Engine. Is that correct?
I am pretty certain that the procedure assumed that all hosts need restoration, not that some are still up-and-running.
If so, how to you remove host 2 and host 3 from the environment, then add back in again while keeping the VMs running?
That's a good question.
Please try to describe the exact flow you have in mind. What's broken and needs restoration, and how do you plan to do that?
Host 2 and Host 3 are not recoverable in their current state. These hosts need to be removed from the environment, and then added again to the environment using the hosted-engine deployment script. For more information on these actions, see the Removing Non-Operational Hosts from a Restored Self-Hosted Engine Environment section below and Chapter 7: Installing Additional Hosts to a Self-Hosted Environment.
BTW: The link referring to chapter 7 is broken.
You are right. The link in the bottom of the page seems working. Now pushed [1] to fix. Thanks for the report!
[1]
Best, -- Didi
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Mon, Jul 3, 2017 at 4:40 PM, Andrew Dent <adent@ctcroydon.com.au> wrote:
Hi Didi
Fair enough. If I'm in this situation..... I have 3 hosts with 6 production VMs. The hosted-engine VM is completely toast and not recoverable.
Meaning? It does not even boot? If so, then I am afraid that you need to follow the linked procedure, including deploying hosted-engine again on new (or fully cleaned) storage. Are all 3 hosts also hosted-engine hosts? And you want to keep them this way? If so, you'll have to reinstall them. I think, didn't try, that after you deploy a new hosted-engine host, and restore the engine, the new engine will see your existing hosts, but hosted-engine-ha will not work. If you want to try and keep your VMs running, I guess your best bet is to try to add a new non-hosted-engine host (one or more, as needed), migrate your VMs to it (them), then remove the hosted-engine hosts and deploy them again as described there.
However I have a backup of the hosted-engine database (do I need anything else).
Only database? Or a backup taken by engine-backup? You also need various configuration files. To see the full list, in case you didn't use engine-backup but can restore individual files, check BACKUP_PATHS inside the script. But this is definitely not supported - you should first try verifying by restoring on a test VM and see what happens.
Is it possible to build a new VM, import the backup of the previous hosted-engine database and reconnect the storage domains and VMs in their running state without any VMs experiencing an outage?
As I wrote above, I think it is, but hosted-engine-ha will not work. If at all possible, I strongly suggest to try to simulate this in a test env. Can be done using nested-kvm VMs for hosts, if you do not have enough real hardware.
The URL http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring... looks to be longer now. I'll review, test and see if this will give me what I'm looking for.
The broken link still seems to be broken. When I click the link, the browser ends up at this http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring...
But I suspect it should be http://www.ovirt.org/documentation/self-hosted/chap-Installing_Additional_Ho...
In my previous mail I just wrote I sent a pull request, not that the site is updated already. I updated it since and it now seems ok to me. Best,
Kind regards
Andrew
------ Original Message ------ From: "Yedidyah Bar David" <didi@redhat.com> To: "Andrew Dent" <adent@ctcroydon.com.au> Cc: "users" <users@ovirt.org> Sent: 3/07/2017 11:12:05 PM Subject: Re: [ovirt-users] Recovering hosted-engine
On Mon, Jul 3, 2017 at 3:46 PM, Andrew Dent <adent@ctcroydon.com.au> wrote:
Has anyone successfully completed a hosted-engine recovery on a multiple host setup with production VMs?
I'd like to clarify that "recovery" can span a large spectrum of flows, from a trivial "I did some change to the engine database that broke stuff and I want to restore a backup I took prior to this change" to a full system restoration including purchasing and deploying new (perhaps different) hosts/network/storage hardware, including many other flows in between.
So when you plan for recovery, you should define very well what flows you plan to handle, and how you handle each.
The linked procedure correctly says it's "providing an example".
Kind regards
Andrew
------ Original Message ------ From: "Andrew Dent" <adent@ctcroydon.com.au> To: "users" <users@ovirt.org> Sent: 2/07/2017 2:22:16 PM Subject: [ovirt-users] Recovering hosted-engine
Hi
A couple of questions about hosted-engine recovery. Part way through this URL, in the section "Workflow for Restoring the Self-Hosted Engine Environment"
http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restoring... it looks like once the hosted-engine is recovered on Host 1, the VMs on Host 2 and 3 will be running, but not accessible to the recovered Hosted Engine. Is that correct?
I am pretty certain that the procedure assumed that all hosts need restoration, not that some are still up-and-running.
If so, how to you remove host 2 and host 3 from the environment, then add back in again while keeping the VMs running?
That's a good question.
Please try to describe the exact flow you have in mind. What's broken and needs restoration, and how do you plan to do that?
Host 2 and Host 3 are not recoverable in their current state. These hosts need to be removed from the environment, and then added again to the environment using the hosted-engine deployment script. For more information on these actions, see the Removing Non-Operational Hosts from a Restored Self-Hosted Engine Environment section below and Chapter 7: Installing Additional Hosts to a Self-Hosted Environment.
BTW: The link referring to chapter 7 is broken.
You are right. The link in the bottom of the page seems working. Now pushed [1] to fix. Thanks for the report!
[1]
Best, -- Didi
-- Didi
It depends on how 'toasted' the engine is. It is possible to mount the disk, if for example it's a kernel upgrade that's gone wrong and stopping the machine form booting On 3 July 2017 at 15:23, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Jul 3, 2017 at 4:40 PM, Andrew Dent <adent@ctcroydon.com.au> wrote:
Hi Didi
Fair enough. If I'm in this situation..... I have 3 hosts with 6 production VMs. The hosted-engine VM is completely toast and not recoverable.
Meaning? It does not even boot? If so, then I am afraid that you need to follow the linked procedure, including deploying hosted-engine again on new (or fully cleaned) storage.
Are all 3 hosts also hosted-engine hosts? And you want to keep them this way? If so, you'll have to reinstall them.
I think, didn't try, that after you deploy a new hosted-engine host, and restore the engine, the new engine will see your existing hosts, but hosted-engine-ha will not work.
If you want to try and keep your VMs running, I guess your best bet is to try to add a new non-hosted-engine host (one or more, as needed), migrate your VMs to it (them), then remove the hosted-engine hosts and deploy them again as described there.
However I have a backup of the hosted-engine database (do I need anything else).
Only database? Or a backup taken by engine-backup? You also need various configuration files. To see the full list, in case you didn't use engine-backup but can restore individual files, check BACKUP_PATHS inside the script. But this is definitely not supported - you should first try verifying by restoring on a test VM and see what happens.
Is it possible to build a new VM, import the backup of the previous hosted-engine database and reconnect the storage domains and VMs in their running state without any VMs experiencing an outage?
As I wrote above, I think it is, but hosted-engine-ha will not work.
If at all possible, I strongly suggest to try to simulate this in a test env. Can be done using nested-kvm VMs for hosts, if you do not have enough real hardware.
chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/
looks to be longer now. I'll review, test and see if this will give me what I'm looking for.
The broken link still seems to be broken. When I click the link, the browser ends up at this http://www.ovirt.org/documentation/self-hosted/ chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_ Environment/chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment# documentation%20self%20hosted%20chap%20Backing%20up%20and% 20Restoring%20an%20EL%20Based%20Self%20Hosted%20Environment% 20chap%20Installing%20Additional%20Hosts%20to%20a% 20Self%20Hosted%20Environment
But I suspect it should be http://www.ovirt.org/documentation/self-hosted/ chap-Installing_Additional_Hosts_to_a_Self-Hosted_Environment/
In my previous mail I just wrote I sent a pull request, not that the site is updated already. I updated it since and it now seems ok to me.
Best,
Kind regards
Andrew
------ Original Message ------ From: "Yedidyah Bar David" <didi@redhat.com> To: "Andrew Dent" <adent@ctcroydon.com.au> Cc: "users" <users@ovirt.org> Sent: 3/07/2017 11:12:05 PM Subject: Re: [ovirt-users] Recovering hosted-engine
On Mon, Jul 3, 2017 at 3:46 PM, Andrew Dent <adent@ctcroydon.com.au> wrote:
Has anyone successfully completed a hosted-engine recovery on a
multiple
host setup with production VMs?
I'd like to clarify that "recovery" can span a large spectrum of flows, from a trivial "I did some change to the engine database that broke stuff and I want to restore a backup I took prior to this change" to a full system restoration including purchasing and deploying new (perhaps different) hosts/network/storage hardware, including many other flows in between.
So when you plan for recovery, you should define very well what flows you plan to handle, and how you handle each.
The linked procedure correctly says it's "providing an example".
Kind regards
Andrew
------ Original Message ------ From: "Andrew Dent" <adent@ctcroydon.com.au> To: "users" <users@ovirt.org> Sent: 2/07/2017 2:22:16 PM Subject: [ovirt-users] Recovering hosted-engine
Hi
A couple of questions about hosted-engine recovery. Part way through this URL, in the section "Workflow for Restoring the Self-Hosted Engine Environment"
chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/
it looks like once the hosted-engine is recovered on Host 1, the VMs on Host 2 and 3 will be running, but not accessible to the recovered Hosted Engine. Is that correct?
I am pretty certain that the procedure assumed that all hosts need restoration, not that some are still up-and-running.
If so, how to you remove host 2 and host 3 from the environment, then add back in again while keeping the VMs running?
That's a good question.
Please try to describe the exact flow you have in mind. What's broken and needs restoration, and how do you plan to do that?
Host 2 and Host 3 are not recoverable in their current state. These hosts need to be removed from the environment, and then added again to the environment using the hosted-engine deployment script. For more information on these actions, see the Removing Non-Operational Hosts from a
Restored
Self-Hosted Engine Environment section below and Chapter 7: Installing Additional Hosts to a Self-Hosted Environment.
BTW: The link referring to chapter 7 is broken.
You are right. The link in the bottom of the page seems working. Now pushed [1] to fix. Thanks for the report!
[1]
Best, -- Didi
-- Didi _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (4)
-
Andrew Dent -
Maton, Brett -
Nelson Lameiras -
Yedidyah Bar David