On Fri, Mar 3, 2017 at 7:00 PM, Darrell Budic <budic@onholyground.com> wrote:

Why are you using an arbitrator if all your HW configs are identical? I’d use a true replica 3 in this case.

This was just GIU suggestion when I was creating the cluster it was asking for the 3 Hosts , I did not knew even that an Arbiter does not keep the data.

I am not so sure if I can change the type of the glusterfs to triplicated one in the running system, probably I need to destroy whole cluster.

Also in my experience with gluster and vm hosting, the ZIL/slog degrades write performance unless it’s a truly dedicated disk. But I have 8 spinners backing my ZFS volumes, so trying to share a sata disk wasn’t a good zil. If yours is dedicated SAS, keep it, if it’s SATA, try testing without it.

We have also several huge systems running with zfs quite successful over the years. This was an idea to use zfs + glusterfs for the HA solutions.

You don’t have compression enabled on your zfs volume, and I’d recommend enabling relatime on it. Depending on the amount of RAM in these boxes, you probably want to limit your zfs arc size to 8G or so (1/4 total ram or less). Gluster just works volumes hard during a rebuild, what’s the problem you’re seeing? If it’s affecting your VMs, using shading and tuning client & server threads can help avoid interruptions to your VMs while repairs are running. If you really need to limit it, you can use cgroups to keep it from hogging all the CPU, but it takes longer to heal, of course. There are a couple older posts and blogs about it, if you go back a while.

Yes I saw that glusterfs is CPU/RAM hugry!!! 99% of all 16 cores used just for healing 500GB vm disks. It was taking almost infinity compare with nfs storage (single disk+zfs ssd cache, for sure one get an penalty for the HA:) )