4:02 p.m.
On Fri, Apr 5, 2019 at 2:18 PM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
>This definitively helps, but for my experience the network speed
is
really determinant here.Can you describe your network >configuration?
>A 10 Gbps net is definitively fine here.
>A few bonded 1 Gbps nics could work.
>A single 1 Gbps nic could be an issue.
I have a gigabit interface on my workstations and sadly I have no option
for upgrade without switching the hardware.
I have observed my network traffic for days with iftop and gtop and I have
never reached my Gbit interface's maximum bandwidth, not even the half of
it.
Even when reseting my bricks (gluster volume reset-brick) and running a
full heal - I do not observe more than 50GiB/s utilization. I am not sure
if FUSE is using network for accessing the local brick - but I hope that
it is not true.
GlusterFS is a scalable *network* filesystem: the network is always there.
You can use caching technique to read from the local peer first but sooner
or later you will have to compare it with data from other peers or sync the
data to other peers on writes.
According to gluster administration guide:
https://docs.gluster.org/en/latest/Administrator%20Guide/Network%20Config...
in the "when to bond" section we can read:
network throughput limit of client/server \<\< storage throughput limit
1 GbE (almost always)
10-Gbps links or faster -- for writes, replication doubles the load on the
network and replicas are usually on different peers to which the client can
transmit in parallel.
So if you are using oVirt hyper-converged in replica 3 you have to transmit
everything two times over the storage network to sync it with other peers.
I'm not really in that details, but if https://bugzilla.redhat.com/1673058
is really like it's described, we even have an 5x overhead with current
gluster 5.x.
This means that with a 1000 Mbps nic we cannot expect more than:
1000 Mbps / 2 (other replicas) / 5 (overhead in Gluster 5.x ???) / 8 (bit
per bytes) = 12.5 MByte per seconds and this is definitively enough to have
sanlock failing especially because we don't have just the sanlock load as
you can imagine.
I'd strongly advice to move to 10 Gigabit Ethernet (nowadays with a few
hundred dollars you can buy a 4/5 ports 10GBASE-T copper switch plus 3 nics
and the cables just for the gluster network) or to bond a few 1 Gigabit
Ethernet links.
Checking disk performance - everything is in the expected ranges.
I suspect that the Gluster v5 enhancements are increasing both network and
IOPS requirements and my setup was not dealing with it properly.
>It's definitively planned, see: https://bugzilla.redhat.com/1693998
<https://bugzilla.redhat.com/show_bug.cgi?id=1693998>
>I'm not really sure about its time plan.
I will try to get involved and provide feedback both to oVirt and Gluster
dev teams.
Best Regards,
Strahil Nikolov