On Fri, Mar 3, 2017 at 7:00 PM, Darrell Budic <budic(a)onholyground.com>
wrote:
Why are you using an arbitrator if all your HW configs are identical?
I’d
use a true replica 3 in this case.
This was just GIU suggestion when I was creating the cluster it was asking
for the 3 Hosts , I did not knew even that an Arbiter does not keep the
data.
I am not so sure if I can change the type of the glusterfs to triplicated
one in the running system, probably I need to destroy whole cluster.
Also in my experience with gluster and vm hosting, the ZIL/slog
degrades
write performance unless it’s a truly dedicated disk. But I have 8 spinners
backing my ZFS volumes, so trying to share a sata disk wasn’t a good zil.
If yours is dedicated SAS, keep it, if it’s SATA, try testing without it.
We have also several huge systems running with zfs quite successful over
the years. This was an idea to use zfs + glusterfs for the HA solutions.
You don’t have compression enabled on your zfs volume, and I’d
recommend
enabling relatime on it. Depending on the amount of RAM in these boxes, you
probably want to limit your zfs arc size to 8G or so (1/4 total ram or
less). Gluster just works volumes hard during a rebuild, what’s the problem
you’re seeing? If it’s affecting your VMs, using shading and tuning client
& server threads can help avoid interruptions to your VMs while repairs are
running. If you really need to limit it, you can use cgroups to keep it
from hogging all the CPU, but it takes longer to heal, of course. There are
a couple older posts and blogs about it, if you go back a while.
Yes I saw that glusterfs is CPU/RAM hugry!!! 99% of all 16 cores used just
for healing 500GB vm disks. It was taking almost infinity compare with nfs
storage (single disk+zfs ssd cache, for sure one get an penalty for the
HA:) )