7:08 p.m.
On 11 February 2017 at 13:32, Bartosiak-Jentys, Chris <
chris.bartosiak-jentys(a)certico.co.uk> wrote:
Hello list,
Just wanted to get your opinion on my ovirt home lab setup. While this is
not a production setup I would like it to run relatively reliably so please
tell me if the following storage configuration is likely to result in
corruption or just bat s**t insane.
I have a 3 node hosted engine setup, VM data store and engine data store
are both replica 3 gluster volumes (one brick on each host).
I do not want to run all 3 hosts 24/7 due to electricity costs, I only
power up the larger hosts (2 Dell R710's) when I need additional resources
for VM's.
I read about using CTDB and floating/virtual IP's to allow the storage
mount point to transition between available hosts but after some thought
decided to go about this another, simpler, way:
I created a common hostname for the storage mount points: gfs-data and
gfs-engine
On each host I edited /etc/hosts file to have these hostnames resolve to
each hosts IP i.e. on host1 gfs-data & gfs-engine --> host1 IP
on host2 gfs-data & gfs-engine --> host2 IP
etc.
In ovirt engine each storage domain is mounted as gfs-data:/data and
gfs-engine:/engine
My thinking is that this way no matter which host is up and acting as SPM
it will be able to mount the storage as its only dependent on that host
being up.
I changed gluster options for server-quorum-ratio so that the volumes
remain up even if quorum is not met, I know this is risky but its just a
lab setup after all.
So, any thoughts on the /etc/hosts method to ensure the storage mount
point is always available? Is data corruption more or less inevitable with
this setup? Am I insane ;) ?
Why not just use localhost? And no need for CTDB with a floating IP, oVirt
uses libgfapi for Gluster which deals with that all natively.
As for the quorum issue, I would most definitely *not* run with quorum
disabled when you're running more than one node. As you say you
specifically plan for when the other 2 nodes of the replica 3 set will be
active or not, I'd do something along the lines of the following...
Going from 3 nodes to 1 node:
- Put nodes 2 & 3 in maintenance to offload their virtual load;
- Once the 2 nodes are free of load, disable quorum on the Gluster volumes;
- Power down the 2 nodes.
Going from 1 node to 3 nodes:
- Power on *only* 1 of the pair of nodes (if you power on both & self-heal
is enabled, Gluster will "heal" the files on the main node with the older
files on the 2 nodes which were powered down);
- Allow Gluster some time to detect that the files are in split-brain;
- Tell Gluster to heal the files in split-brain based on modification time;
- Once the 2 nodes are in sync, re-enable quorum & power on the last node,
which will be resynchronised automatically;
- Take the 2 hosts out of maintenance mode.
If you want to power on the 2nd two nodes at the same time, make absolutely
sure self-heal is disabled first! If you don't, Gluster will see the 2nd
two nodes as in quorum & heal the data on your 1st node with the
out-of-date data.
--
Doug