Hi all,

I'm running a glusterFS setup v 8.6 with two node and one arbiter. Both nodes and arbiter are CentOS 8 Stream with oVirt 4.4. Under gluster I have a LVM thin partition.

VMs running in this cluster have really poor write performance, when a test directly performend on the disk score about 300 MB/s

dd test on host1:

[root@ovirt-host1 tmp]# dd if=/dev/zero of=./foo.dat bs=256M count=1 oflag=dsync
1+0 records in
1+0 records out
268435456 bytes (268 MB, 256 MiB) copied, 0.839861 s, 320 MB/s

dd test on host1 on gluster:

[root@ovirt-host1 tmp]# dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/ovirt-host1:_data/foo.dat bs=256M count=1 oflag=dsync
1+0 records in
1+0 records out
268435456 bytes (268 MB, 256 MiB) copied, 50.6889 s, 5.3 MB/s

Nontheless, the write results in a VM inside the cluster is a little bit faster (dd results vary from 15 MB/s to 60 MB/s)  and this is very strange to me:

root@vm1-ha:/tmp# dd if=/dev/zero of=./foo.dat bs=256M count=1 oflag=dsync; rm -f ./foo.dat
1+0 records in
1+0 records out
268435456 bytes (268 MB, 256 MiB) copied, 5.58727 s, 48.0 MB/s


Here's the actual gluster configuration, I also applied  some paramaters in /var/lib/glusterd/groups/virt as mentioned in other ovirt thread related I found.


gluster volume info data

Volume Name: data
Type: Replicate
Volume ID: 09b532eb-57de-4c29-862d-93993c990e32
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: ovirt-host1:/gluster_bricks/data/data
Brick2: ovirt-host2:/gluster_bricks/data/data
Brick3: ovirt-arbiter:/gluster_bricks/data/data (arbiter)
Options Reconfigured:
server.event-threads: 4
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
cluster.server-quorum-type: server
cluster.lookup-optimize: off
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.choose-local: off
client.event-threads: 4
performance.client-io-threads: on
nfs.disable: on
transport.address-family: inet
storage.fips-mode-rchecksum: on
storage.owner-uid: 36
storage.owner-gid: 36
features.shard: on
performance.low-prio-threads: 32
performance.strict-o-direct: on
network.remote-dio: off
network.ping-timeout: 30
user.cifs: off
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
cluster.eager-lock: enable


The speed between two hosts is about 1Gb/s:

[root@ovirt-host1 ~]# iperf3 -c ovirt-host2 -p 5002
Connecting to host ovirt-host2 port 5002
[  5] local x.x.x.x port 58072 connected to y.y.y.y port 5002
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   112 MBytes   938 Mbits/sec  117    375 KBytes
[  5]   1.00-2.00   sec   112 MBytes   937 Mbits/sec    0    397 KBytes
[  5]   2.00-3.00   sec   110 MBytes   924 Mbits/sec   18    344 KBytes
[  5]   3.00-4.00   sec   112 MBytes   936 Mbits/sec    0    369 KBytes
[  5]   4.00-5.00   sec   111 MBytes   927 Mbits/sec   12    386 KBytes
[  5]   5.00-6.00   sec   112 MBytes   938 Mbits/sec    0    471 KBytes
[  5]   6.00-7.00   sec   108 MBytes   909 Mbits/sec   34    382 KBytes
[  5]   7.00-8.00   sec   112 MBytes   942 Mbits/sec    0    438 KBytes
[  5]   8.00-9.00   sec   111 MBytes   928 Mbits/sec   38    372 KBytes
[  5]   9.00-10.00  sec   111 MBytes   934 Mbits/sec    0    481 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.08 GBytes   931 Mbits/sec  219             sender
[  5]   0.00-10.04  sec  1.08 GBytes   926 Mbits/sec                  receiver

iperf Done.

Between nodes and arbiter about 200MB/s

[  5] local ovirt-arbiter port 45220 connected to ovirt-host1 port 5002
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  30.6 MBytes   257 Mbits/sec  1177    281 KBytes
[  5]   1.00-2.00   sec  26.2 MBytes   220 Mbits/sec    0    344 KBytes
[  5]   2.00-3.00   sec  28.8 MBytes   241 Mbits/sec   15    288 KBytes
[  5]   3.00-4.00   sec  26.2 MBytes   220 Mbits/sec    0    352 KBytes
[  5]   4.00-5.00   sec  30.0 MBytes   252 Mbits/sec   32    293 KBytes
[  5]   5.00-6.00   sec  26.2 MBytes   220 Mbits/sec    0    354 KBytes
[  5]   6.00-7.00   sec  30.0 MBytes   252 Mbits/sec   32    293 KBytes
[  5]   7.00-8.00   sec  27.5 MBytes   231 Mbits/sec    0    355 KBytes
[  5]   8.00-9.00   sec  28.8 MBytes   241 Mbits/sec   30    294 KBytes
[  5]   9.00-10.00  sec  26.2 MBytes   220 Mbits/sec    3    250 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   281 MBytes   235 Mbits/sec  1289             sender
[  5]   0.00-10.03  sec   277 MBytes   232 Mbits/sec                  receiver

iperf Done.



I definitely missing something obvious and I'm not a gluster/ovirt black bealt... Can anyone point me to the right way?

Thank you for your time.

Regards,
Francesco
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RWOSI4BLJMJ5WBCF5TQBXW2VB77N5GZU/

--

--
		Francesco Lorenzini System Administrator & DevOps Engineer Shellrent Srl Via dell'Edilizia, 19 - 36100 Vicenza Tel. 0444321155