8:30 p.m.
Hi all,
One of our engines has had a DB failure* & it seems there was an unnoticed
problem in its backup routine, meaning the last backup I've got is a couple
of weeks old.
Luckily, VDSM has kept the underlying VMs running without any
interruptions, so my objective is to get the HE back online & get the hosts
& VMs back under its control with minimal downtime.
So, my questions are the following...
1. What problems can I expect to have with VMs added/modified since the
last backup?
2. As it's only the DB that's been affected, can I skip redeploying the
Engine & jump straight to restoring the DB & rerunning engine-setup?
3. The original docs I read didn't mention that it's best to leave a
host in maintenance mode before running the engine backup, so my plan is to
install a new temporary host on a separate server, re-add the old hosts &
then once everything's back up, remove the temporary host. Are there any
faults in this plan?
4. When it comes to deleting the old HE VM, the docs point to a
paywalled guide on redhat.com...?
CentOS 7
oVirt 4.0.4
Gluster 3.8
* Apparently a write somehow cleared fsync, despite not actually having
been written to disk?! No idea how that happened...
Many thanks,
--
Doug
3:49 p.m.
Hey guys,
Just giving this a bump in the hope that someone might be able to advise...
Hi all,
One of our engines has had a DB failure* & it seems there was
an
unnoticed problem in its backup routine, meaning the last backup I've got
is a couple of weeks old.
Luckily, VDSM has kept the underlying VMs running without any
interruptions, so my objective is to get the HE back online & get the hosts
& VMs back under its control with minimal downtime.
So, my questions are the following...
1. What problems can I expect to have with VMs added/modified since
the last backup?
2. As it's only the DB that's been affected, can I skip redeploying
the Engine & jump straight to restoring the DB & rerunning engine-setup?
3. The original docs I read didn't mention that it's best to leave a
host in maintenance mode before running the engine backup, so my plan is to
install a new temporary host on a separate server, re-add the old hosts &
then once everything's back up, remove the temporary host. Are there any
faults in this plan?
4. When it comes to deleting the old HE VM, the docs point to a
paywalled guide on redhat.com...?
To add a bit more info to 4), I'm referring to the following...
Note: If the Engine database is restored successfully, but the Engine
virtual machine appears to be Down and cannot be migrated to another
self-hosted engine host, you can enable a new Engine virtual machine and
remove the dead Engine virtual machine from the environment by following
the steps provided in https://access.redhat.com/solutions/1517683.
Source:
http://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restor...
CentOS 7
oVirt 4.0.4
Gluster 3.8
* Apparently a write somehow cleared fsync, despite not actually having
been written to disk?! No idea how that happened...
Many thanks,
--
Doug
Cheers,
--
Doug
4:11 p.m.
On Tue, Jan 24, 2017 at 1:49 PM, Doug Ingham <dougti(a)gmail.com> wrote:
Hey guys,
Just giving this a bump in the hope that someone might be able to
advise...
Hi all,
> One of our engines has had a DB failure* & it seems there was an
> unnoticed problem in its backup routine, meaning the last backup I've got
> is a couple of weeks old.
> Luckily, VDSM has kept the underlying VMs running without any
> interruptions, so my objective is to get the HE back online & get the hosts
> & VMs back under its control with minimal downtime.
>
> So, my questions are the following...
>
> 1. What problems can I expect to have with VMs added/modified since
> the last backup?
>
> Modified VMs will be reverted to the previous configuration; additional
VMs
should be seen as external VMs, then you could import.
> 1. As it's only the DB that's been affected, can I skip redeploying
> the Engine & jump straight to restoring the DB & rerunning engine-setup?
>
>
Yes, if the engine VM is fine, you could just import the previous backup
and run engine-setup again.
Please set the global maintenance mode for hosted-engine since
engine-backup and engine-setup are going to bring down the engine.
> 1. The original docs I read didn't mention that it's best to leave a
> host in maintenance mode before running the engine backup, so my plan is to
> install a new temporary host on a separate server, re-add the old hosts &
> then once everything's back up, remove the temporary host. Are there any
> faults in this plan?
> 2. When it comes to deleting the old HE VM, the docs point to a
> paywalled guide on redhat.com...?
>
> To add a bit more info to 4), I'm referring to the following...
Note: If the Engine database is restored successfully, but the Engine
> virtual machine appears to be Down and cannot be migrated to another
> self-hosted engine host, you can enable a new Engine virtual machine and
> remove the dead Engine virtual machine from the environment by following
> the steps provided in https://access.redhat.com/solutions/1517683.
>
Source: http://www.ovirt.org/documentation/self-hosted/
chap-Backing_up_and_Restoring_an_EL-Based_Self-Hosted_Environment/
If you are re-importing the backup in place on the initial engine VM you
don't have to.
The point is just if you are migrating to a new engine VM and so you have
to remove the entry of the previous one to let the auto-import process
trigger again.
CentOS 7
> oVirt 4.0.4
> Gluster 3.8
>
> * Apparently a write somehow cleared fsync, despite not actually having
> been written to disk?! No idea how that happened...
>
> Many thanks,
> --
> Doug
>
Cheers,
--
Doug
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
3:52 p.m.
Hi All, Simone,
On 24 January 2017 at 10:11, Simone Tiraboschi <stirabos(a)redhat.com> wrote:
On Tue, Jan 24, 2017 at 1:49 PM, Doug Ingham <dougti(a)gmail.com> wrote:
> Hey guys,
> Just giving this a bump in the hope that someone might be able to
> advise...
>
> Hi all,
>> One of our engines has had a DB failure* & it seems there was an
>> unnoticed problem in its backup routine, meaning the last backup I've got
>> is a couple of weeks old.
>> Luckily, VDSM has kept the underlying VMs running without any
>> interruptions, so my objective is to get the HE back online & get the hosts
>> & VMs back under its control with minimal downtime.
>>
>> So, my questions are the following...
>>
>> 1. What problems can I expect to have with VMs added/modified since
>> the last backup?
>>
>> Modified VMs will be reverted to the previous configuration; additional
VMs should be seen as external VMs, then you could import.
Given VDSM kept the VMs up whilst the HE's been down, how will the running
VMs that were present before & after the backup be affected?
Many of the VMs that were present during the last backup are now on
different hosts, including the HE VM. Will that cause any issues?
>
>> 1. As it's only the DB that's been affected, can I skip redeploying
>> the Engine & jump straight to restoring the DB & rerunning
engine-setup?
>>
>>
Yes, if the engine VM is fine, you could just import the previous backup
and run engine-setup again.
Please set the global maintenance mode for hosted-engine since
engine-backup and engine-setup are going to bring down the engine.
As per above, do I still only need to import the previous backup even if
the all of the VMs (including the HE VM) are now on different hosts to when
the backup was made?
And as for the future, is it going to be necessary to always keep an unused
host in the cluster to allow for emergency restores? I'm a bit concerned
that if we ever utilised all of our hosts for running VMs, then we'd be
completely stuck if the HE ever imploded again.
Cheers,
--
Doug
7:30 p.m.
On Mon, Feb 6, 2017 at 1:52 PM, Doug Ingham <dougti(a)gmail.com> wrote:
Hi All, Simone,
On 24 January 2017 at 10:11, Simone Tiraboschi <stirabos(a)redhat.com>
wrote:
>
>
> On Tue, Jan 24, 2017 at 1:49 PM, Doug Ingham <dougti(a)gmail.com> wrote:
>
>> Hey guys,
>> Just giving this a bump in the hope that someone might be able to
>> advise...
>>
>> Hi all,
>>> One of our engines has had a DB failure* & it seems there was an
>>> unnoticed problem in its backup routine, meaning the last backup I've
got
>>> is a couple of weeks old.
>>> Luckily, VDSM has kept the underlying VMs running without any
>>> interruptions, so my objective is to get the HE back online & get the
hosts
>>> & VMs back under its control with minimal downtime.
>>>
>>> So, my questions are the following...
>>>
>>> 1. What problems can I expect to have with VMs added/modified since
>>> the last backup?
>>>
>>> Modified VMs will be reverted to the previous configuration; additional
> VMs should be seen as external VMs, then you could import.
>
Given VDSM kept the VMs up whilst the HE's been down, how will the running
VMs that were present before & after the backup be affected?
Many of the VMs that were present during the last backup are now on
different hosts, including the HE VM. Will that cause any issues?
For normal VMs I don't expect any issue: the engine will simply update the
correspondent record once it will find them on the managed hosts.
A serious issue could instead happen with HA VMs:
if the engine finds earlier an HA VM as running on a different host it will
simply update its record, the issue is if it finds earlier the VM a not on
the original host since it will try to restart it causing a split brain and
probably a VM corruption.
I opened a bug to track it:
https://bugzilla.redhat.com/show_bug.cgi?id=1419649
>
>
>>
>>> 1. As it's only the DB that's been affected, can I skip
redeploying
>>> the Engine & jump straight to restoring the DB & rerunning
engine-setup?
>>>
>>>
> Yes, if the engine VM is fine, you could just import the previous backup
> and run engine-setup again.
> Please set the global maintenance mode for hosted-engine since
> engine-backup and engine-setup are going to bring down the engine.
>
As per above, do I still only need to import the previous backup even if
the all of the VMs (including the HE VM) are now on different hosts to when
the backup was made?
Please take care of the HA VMs.
And as for the future, is it going to be necessary to always keep an
unused host in the cluster to allow for emergency restores? I'm a bit
concerned that if we ever utilised all of our hosts for running VMs, then
we'd be completely stuck if the HE ever imploded again.
Honestly I don't see any special issue there.
Cheers,
--
Doug
7:45 p.m.
On 6 February 2017 at 13:30, Simone Tiraboschi <stirabos(a)redhat.com> wrote:
>>>> 1. What problems can I expect to have with VMs added/modified
>>>> since the last backup?
>>>>
>>>> Modified VMs will be reverted to the previous configuration;
>> additional VMs should be seen as external VMs, then you could import.
>>
>
> Given VDSM kept the VMs up whilst the HE's been down, how will the
> running VMs that were present before & after the backup be affected?
>
> Many of the VMs that were present during the last backup are now on
> different hosts, including the HE VM. Will that cause any issues?
>
For normal VMs I don't expect any issue: the engine will simply update the
correspondent record once it will find them on the managed hosts.
A serious issue could instead happen with HA VMs:
if the engine finds earlier an HA VM as running on a different host it
will simply update its record, the issue is if it finds earlier the VM a
not on the original host since it will try to restart it causing a split
brain and probably a VM corruption.
I opened a bug to track it:
https://bugzilla.redhat.com/show_bug.cgi?id=1419649
Ouch. *All* of our VMs are HA by default.
So the simplest current solution would be to shutdown the running VMs in
VDSM, before restoring the backup & running engine-setup?
Cheers,
--
Doug
7:50 p.m.
> A serious issue could instead happen with HA VMs:
> if the engine finds earlier an HA VM as running on a different host it
> will simply update its record, the issue is if it finds earlier the VM a not
> on the original host since it will try to restart it causing a split brain
> and probably a VM corruption.
Uh? Why should this be the case? I think we need Arik to confirm this,
but the engine only does HA VM restarts during fencing. And fencing is
only activated 5 minutes after the engine start to make sure we
already have all host reports.
Martin
On Mon, Feb 6, 2017 at 5:45 PM, Doug Ingham <dougti(a)gmail.com> wrote:
> On 6 February 2017 at 13:30, Simone Tiraboschi <stirabos(a)redhat.com> wrote:
>>
>>
>>>>>> What problems can I expect to have with VMs added/modified since
the
>>>>>> last backup?
>>>>
>>>> Modified VMs will be reverted to the previous configuration; additional
>>>> VMs should be seen as external VMs, then you could import.
>>>
>>>
>>> Given VDSM kept the VMs up whilst the HE's been down, how will the
>>> running VMs that were present before & after the backup be affected?
>>>
>>> Many of the VMs that were present during the last backup are now on
>>> different hosts, including the HE VM. Will that cause any issues?
>>
>>
>> For normal VMs I don't expect any issue: the engine will simply update the
>> correspondent record once it will find them on the managed hosts.
> A serious issue could instead happen with HA VMs:
> if the engine finds earlier an HA VM as running on a different host it
> will simply update its record, the issue is if it finds earlier the VM a not
> on the original host since it will try to restart it causing a split brain
> and probably a VM corruption.
>> I opened a bug to track it:
>
>
> Ouch. *All* of our VMs are HA by default.
>
> So the simplest current solution would be to shutdown the running VMs in
> VDSM, before restoring the backup & running engine-setup?
>
> Cheers,
> --
> Doug
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
2846
days inactive
2863
days old
6 comments
3 participants
participants (3)
-
Doug Ingham -
Martin Sivak -
Simone Tiraboschi