10:53 p.m.
Hello users,
After a painful experience with a crashed engine and a problematic recovery
from a backup, I was thinking to create one more engine in standby mode in
the same network, that will automatically recover backup files from the
active engine on a daily/hourly basis.
This way I'll run recovery tests every day automatically and easily switch
in case of engine failure with a DNS record change to the passive engine.
So the main question, is it harmful to have more then one engine alive if
only one is been modified? can the standby engine affect the host nodes?
8:47 a.m.
On Thu, Jan 10, 2019 at 10:56 PM maoz zadok <maozza(a)gmail.com> wrote:
Hello users,
After a painful experience with a crashed engine and a problematic recovery from a
backup, I was thinking to create one more engine in standby mode in the same network, that
will automatically recover backup files from the active engine on a daily/hourly basis.
This way I'll run recovery tests every day automatically and easily switch in case of
engine failure with a DNS record change to the passive engine.
Makes sense. I think I heard about people doing this, do not have details.
So the main question, is it harmful to have more then one engine
alive if only one is been modified?
It's harmful even if they are identical, because they do not know
about each other. Each thinks it's the only one controlling the hosts,
and they might give hosts conflicting commands to do.
can the standby engine affect the host nodes?
Yes, that's the main problem.
If you go this way, I suggest to not let the backup server access
your hosts normally, and think about some other way to test it.
When you need to activate it, open access to the hosts (and change
DNS).
Obviously, this requires very careful planning, implementation and
on-going operation.
Most people that need HA engine use ovirt-hosted-engine, and some
others use a single engine vm managed by some other means (e.g.
some other virt solution, another instance of oVirt, or plain kvm
and libvirt plus some custom HA stuff).
Best regards,
--
Didi
3:26 p.m.
13.01.2019 10:47, Yedidyah Bar David пишет:
Most people that need HA engine use ovirt-hosted-engine,
HA hosted-engine cannot help if VM image are broken. HA runs same image
on different nodes, and if current running VM corrupt FS, for example,
it cannot run on other nodes also.
I wrote about this experience here some time ago.
--
Mike
4:37 p.m.
New subject: EXTERNAL: Re: multiple engines (active passive)
In that case, my understanding is that you use the backup feature within hosted-engine,
and if you need to recover use that backup, which is an option during hosted-engine
installation.
I’m all new to this, and I know I haven’t read all the administration documentation
(though what I have read is very good). It might be helpful, if it doesn’t exist, to have
a separate “disaster recovery” practices page to cover all the terrible things that could
happen to a running system. The first part would be activities that administrators should
be doing regularly while the system is healthy, then a section on testing the validity of
those backups. Finally, multiple subsections on “if this happens, then use this, this, and
this that you have for section A to recover your system.”
I’d be willing to help create that, since I need to create the documents for our business
systems.
Tom Albrecht III
Cyber Architect
Lockheed Martin RMS
Sent from my iPhone
On Jan 13, 2019, at 8:34 AM, Mike <combr(a)ya.ru> wrote:
13.01.2019 10:47, Yedidyah Bar David пишет:
> Most people that need HA engine use ovirt-hosted-engine,
HA hosted-engine cannot help if VM image are broken. HA runs same image on different
nodes, and if current running VM corrupt FS, for example, it cannot run on other nodes
also.
I wrote about this experience here some time ago.
--
Mike
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WK4IT3NMVOW...
9:33 p.m.
is it good enough to disable the ovirt-engine "systemctl disable
ovirt-engine" on the standby node?
if it does, what about the other services, do I have to disable the
following as well? :
*ovirt-engine-dwhd.service enabled *
*ovirt-engine.service enabled *
*ovirt-fence-kdump-listener.service enabled *
*ovirt-imageio-proxy.service enabled *
*ovirt-provider-ovn.service enabled *
*ovirt-vmconsole-proxy-sshd.service enabled *
*ovirt-websocket-proxy.service enabled *
*Thank you! *
*maoz*
On Sun, Jan 13, 2019 at 3:34 PM Mike <combr(a)ya.ru> wrote:
13.01.2019 10:47, Yedidyah Bar David пишет:
> Most people that need HA engine use ovirt-hosted-engine,
HA hosted-engine cannot help if VM image are broken. HA runs same image
on different nodes, and if current running VM corrupt FS, for example,
it cannot run on other nodes also.
I wrote about this experience here some time ago.
--
Mike
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WK4IT3NMVOW...
10:15 a.m.
On Sun, Jan 13, 2019 at 9:40 PM <alex(a)triadic.us> wrote:
For what it's worth we do active passive with pacemaker,
corosync, and
drbd. all the configuration files stay synced by drbd, pacemaker ensures
the services are only running on one node. Works pretty well.
For more demanding scenarios there is also an ansible role to handle the
disaster recovery of an oVirt environment:
https://github.com/oVirt/ovirt-ansible-disaster-recovery
Here a demo about how to handle the mapping between the two sites:
https://youtu.be/mEOgH-Tk09c
On Jan 13, 2019 2:33 PM, maoz zadok <maozza(a)gmail.com> wrote:
is it good enough to disable the ovirt-engine "systemctl disable
ovirt-engine" on the standby node?
if it does, what about the other services, do I have to disable the
following as well? :
*ovirt-engine-dwhd.service enabled *
*ovirt-engine.service enabled *
*ovirt-fence-kdump-listener.service enabled *
*ovirt-imageio-proxy.service enabled *
*ovirt-provider-ovn.service enabled *
*ovirt-vmconsole-proxy-sshd.service enabled *
*ovirt-websocket-proxy.service enabled *
*Thank you! *
*maoz*
On Sun, Jan 13, 2019 at 3:34 PM Mike <combr(a)ya.ru> wrote:
13.01.2019 10:47, Yedidyah Bar David пишет:
> Most people that need HA engine use ovirt-hosted-engine,
HA hosted-engine cannot help if VM image are broken. HA runs same image
on different nodes, and if current running VM corrupt FS, for example,
it cannot run on other nodes also.
I wrote about this experience here some time ago.
--
Mike
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WK4IT3NMVOW...
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/CAXQSP3DG5X...
8:41 p.m.
I'm still sort of new to ovirt, but I went through a similar things. I had my
original engine fail and had to recover, so here is my "oVirt HA plan"
1. I do NOT use hosted ovirt, I had issues getting it deployed correctly, and that
doesn't help if they engine VM itself has issues. My engine is hosted on a completely
separated 2-node hyper-converged Hyper-V Cluster. Unless you have a cluster larger than 3
hosts, I really don't think hosted ovirt is a good idea, it would make more sense to
just load ovirt on another PC by itself.
2. I plan on loading another copy of ovirt in a "cold storage" configuration.
Where it will be loaded on centos, and configured as close as I can without adding in any
hosts. I'll probably keep it turned off and try to updated it once a month or so.
3. If I have another oVirt failure, I know to log in, put the storage domains into
maintenance mode if possible, and copy out any needed config items i may have missed. I
will the shut it off and delete it. I then will reboot all the hosts one at a time to
clear out any sanlock issues, then add the hosts to the new copy of ovirt and import the
storage. I estimate that process to take around 2-4hrs.
9:16 p.m.
Well, I really love oVirt, but I don't know.. All the solutions mentioned
here are complicated and or dangerous. Including hosted ha engine that
fails while deploying(for me). I think that test the backup for recovery is
very important and need to be done on a regular basis, What good is a
backup if you cannot restore?? I worked for a whole night trying to
recover the failed engine...recovery from backup was very painful. does
anyone here have a solution for testing backups (not in crises mode)?
On Mon, Jan 14, 2019, 20:41 <michael(a)wanderingmad.com wrote:
I'm still sort of new to ovirt, but I went through a similar
things. I
had my original engine fail and had to recover, so here is my "oVirt HA
plan"
1. I do NOT use hosted ovirt, I had issues getting it deployed correctly,
and that doesn't help if they engine VM itself has issues. My engine is
hosted on a completely separated 2-node hyper-converged Hyper-V Cluster.
Unless you have a cluster larger than 3 hosts, I really don't think hosted
ovirt is a good idea, it would make more sense to just load ovirt on
another PC by itself.
2. I plan on loading another copy of ovirt in a "cold storage"
configuration. Where it will be loaded on centos, and configured as close
as I can without adding in any hosts. I'll probably keep it turned off and
try to updated it once a month or so.
3. If I have another oVirt failure, I know to log in, put the storage
domains into maintenance mode if possible, and copy out any needed config
items i may have missed. I will the shut it off and delete it. I then
will reboot all the hosts one at a time to clear out any sanlock issues,
then add the hosts to the new copy of ovirt and import the storage. I
estimate that process to take around 2-4hrs.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AHD3OABM65F...
3:19 a.m.
real HA is complicated, no way around that...
As stated earlier, we also run engine bare metal using pacemaker /
corosync / drbd to keep both nodes in perfect sync, failover happens in
a few seconds. We also do daily backups of the engine, but in the 4
years or so that we have been running ovirt, we have luckily never had
to use them with this setup. STONITH is pretty important to setup if you
are running less than three nodes as the engine, just to keep split
brain from corrupting everything.
On 2019-01-14 14:16, maoz zadok wrote:
Well, I really love oVirt, but I don't know.. All the solutions
mentioned here are complicated and or dangerous. Including hosted ha engine that fails
while deploying(for me). I think that test the backup for recovery is very important and
need to be done on a regular basis, What good is a backup if you cannot restore?? I
worked for a whole night trying to recover the failed engine...recovery from backup was
very painful. does anyone here have a solution for testing backups (not in crises mode)?
On Mon, Jan 14, 2019, 20:41 <michael(a)wanderingmad.com wrote:
> I'm still sort of new to ovirt, but I went through a similar things. I had my
original engine fail and had to recover, so here is my "oVirt HA plan"
>
> 1. I do NOT use hosted ovirt, I had issues getting it deployed correctly, and that
doesn't help if they engine VM itself has issues. My engine is hosted on a completely
separated 2-node hyper-converged Hyper-V Cluster. Unless you have a cluster larger than 3
hosts, I really don't think hosted ovirt is a good idea, it would make more sense to
just load ovirt on another PC by itself.
> 2. I plan on loading another copy of ovirt in a "cold storage"
configuration. Where it will be loaded on centos, and configured as close as I can
without adding in any hosts. I'll probably keep it turned off and try to updated it
once a month or so.
> 3. If I have another oVirt failure, I know to log in, put the storage domains into
maintenance mode if possible, and copy out any needed config items i may have missed. I
will the shut it off and delete it. I then will reboot all the hosts one at a time to
clear out any sanlock issues, then add the hosts to the new copy of ovirt and import the
storage. I estimate that process to take around 2-4hrs.
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AHD3OABM65F...
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6XC6QPSLBEI...
12:27 p.m.
Hi Alex,
could you share more details on your setup ? I also had issues with restoring the engine
from a backup and it made me think of running engine not a self hosted HA. I don’t have
any experience with pacemaker/corrosync and drbd, but I’m willing to test and find out how
to use these for the engine. I think details on your config would provide a good starting
point. Of course a working backup of any solution is necessary.
Thank you, Sven
Von: Alex McWhirter [mailto:alex@triadic.us]
Gesendet: Dienstag, 15. Januar 2019 02:19
An: maoz zadok <maozza(a)gmail.com>
Cc: michael(a)wanderingmad.com; users <users(a)ovirt.org>
Betreff: [ovirt-users] Re: multiple engines (active passive)
real HA is complicated, no way around that...
As stated earlier, we also run engine bare metal using pacemaker / corosync / drbd to keep
both nodes in perfect sync, failover happens in a few seconds. We also do daily backups of
the engine, but in the 4 years or so that we have been running ovirt, we have luckily
never had to use them with this setup. STONITH is pretty important to setup if you are
running less than three nodes as the engine, just to keep split brain from corrupting
everything.
On 2019-01-14 14:16, maoz zadok wrote:
Well, I really love oVirt, but I don't know.. All the solutions mentioned here are
complicated and or dangerous. Including hosted ha engine that fails while deploying(for
me). I think that test the backup for recovery is very important and need to be done on a
regular basis, What good is a backup if you cannot restore?? I worked for a whole night
trying to recover the failed engine...recovery from backup was very painful. does anyone
here have a solution for testing backups (not in crises mode)?
On Mon, Jan 14, 2019, 20:41
<michael@wanderingmad.com<mailto:michael@wanderingmad.com> wrote:
I'm still sort of new to ovirt, but I went through a similar things. I had my
original engine fail and had to recover, so here is my "oVirt HA plan"
1. I do NOT use hosted ovirt, I had issues getting it deployed correctly, and that
doesn't help if they engine VM itself has issues. My engine is hosted on a completely
separated 2-node hyper-converged Hyper-V Cluster. Unless you have a cluster larger than 3
hosts, I really don't think hosted ovirt is a good idea, it would make more sense to
just load ovirt on another PC by itself.
2. I plan on loading another copy of ovirt in a "cold storage" configuration.
Where it will be loaded on centos, and configured as close as I can without adding in any
hosts. I'll probably keep it turned off and try to updated it once a month or so.
3. If I have another oVirt failure, I know to log in, put the storage domains into
maintenance mode if possible, and copy out any needed config items i may have missed. I
will the shut it off and delete it. I then will reboot all the hosts one at a time to
clear out any sanlock issues, then add the hosts to the new copy of ovirt and import the
storage. I estimate that process to take around 2-4hrs.
_______________________________________________
Users mailing list -- users@ovirt.org<mailto:users@ovirt.org>
To unsubscribe send an email to
users-leave@ovirt.org<mailto:users-leave@ovirt.org>
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AHD3OABM65F...
_______________________________________________
Users mailing list -- users@ovirt.org<mailto:users@ovirt.org>
To unsubscribe send an email to
users-leave@ovirt.org<mailto:users-leave@ovirt.org>
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6XC6QPSLBEI...
2133
days inactive
2144
days old
10 comments
9 participants
participants (9)
-
Albrecht, Thomas C -
Alex McWhirter -
alex＠triadic.us
-
maoz zadok -
michael＠wanderingmad.com -
Mike -
Simone Tiraboschi -
Sven Achtelik -
Yedidyah Bar David