On Fri, Apr 5, 2019 at 2:18 PM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
>This definitively helps, but for my experience the network speed is really determinant here.Can you describe your network >configuration?
>A 10 Gbps net is definitively fine here.
>A few bonded 1 Gbps nics could work.
>A single 1 Gbps nic could be an issue.


I have a gigabit interface on my workstations and sadly I have no option for upgrade without switching the hardware.
I have observed my network traffic for days with iftop and gtop and I have never reached my Gbit interface's maximum bandwidth, not even the half of it.

Even when reseting my bricks (gluster volume reset-brick) and running a full heal - I do not observe more than 50GiB/s utilization. I am not sure if FUSE is using network for accessing the local brick - but I  hope that it is not true.

GlusterFS is a scalable *network* filesystem: the network is always there.
You can use caching technique to read from the local peer first but sooner or later you will have to compare it with data from other peers or sync the data to other peers on writes.

According to gluster administration guide:
https://docs.gluster.org/en/latest/Administrator%20Guide/Network%20Configurations%20Techniques/
 
in the "when to bond" section we can read:
network throughput limit of client/server \<\< storage throughput limit

1 GbE (almost always)
10-Gbps links or faster -- for writes, replication doubles the load on the network and replicas are usually on different peers to which the client can transmit in parallel.

So if you are using oVirt hyper-converged in replica 3 you have to transmit everything two times over the storage network to sync it with other peers.

I'm not really in that details, but if https://bugzilla.redhat.com/1673058 is really like it's described, we even have an 5x overhead with current gluster 5.x.

This means that with a 1000 Mbps nic we cannot expect more than:
1000 Mbps / 2 (other replicas) / 5 (overhead in Gluster 5.x ???) / 8 (bit per bytes) = 12.5 MByte per seconds and this is definitively enough to have sanlock failing especially because we don't have just the sanlock load as you can imagine.

I'd strongly advice to move to 10 Gigabit Ethernet (nowadays with a few hundred dollars you can buy a 4/5 ports 10GBASE-T copper switch plus 3 nics and the cables just for the gluster network) or to bond a few 1 Gigabit Ethernet links.


Checking disk performance - everything is in the expected ranges.

I suspect that the Gluster v5 enhancements are increasing both network and IOPS requirements and my setup was not dealing with it properly.


>It's definitively planned, see: https://bugzilla.redhat.com/1693998
>I'm not really sure about its time plan.

I will try to get involved and provide feedback both to oVirt and Gluster dev teams.

Best Regards,
Strahil Nikolov