5:09 a.m.
Hi all,
I'm running a glusterFS setup v 8.6 with two node and one arbiter. Both nodes and
arbiter are CentOS 8 Stream with oVirt 4.4. Under gluster I have a LVM thin partition.
VMs running in this cluster have really poor write performance, when a test directly
performend on the disk score about 300 MB/s
dd test on host1:
[root@ovirt-host1 tmp]# dd if=/dev/zero of=./foo.dat bs=256M count=1 oflag=dsync
1+0 records in
1+0 records out
268435456 bytes (268 MB, 256 MiB) copied, 0.839861 s, 320 MB/s
dd test on host1 on gluster:
[root@ovirt-host1 tmp]# dd if=/dev/zero
of=/rhev/data-center/mnt/glusterSD/ovirt-host1:_data/foo.dat bs=256M count=1 oflag=dsync
1+0 records in
1+0 records out
268435456 bytes (268 MB, 256 MiB) copied, 50.6889 s, 5.3 MB/s
Nontheless, the write results in a VM inside the cluster is a little bit faster (dd
results vary from 15 MB/s to 60 MB/s) and this is very strange to me:
root@vm1-ha:/tmp# dd if=/dev/zero of=./foo.dat bs=256M count=1 oflag=dsync; rm -f
./foo.dat
1+0 records in
1+0 records out
268435456 bytes (268 MB, 256 MiB) copied, 5.58727 s, 48.0 MB/s
Here's the actual gluster configuration, I also applied some paramaters in
/var/lib/glusterd/groups/virt as mentioned in other ovirt thread related I found.
gluster volume info data
Volume Name: data
Type: Replicate
Volume ID: 09b532eb-57de-4c29-862d-93993c990e32
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: ovirt-host1:/gluster_bricks/data/data
Brick2: ovirt-host2:/gluster_bricks/data/data
Brick3: ovirt-arbiter:/gluster_bricks/data/data (arbiter)
Options Reconfigured:
server.event-threads: 4
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
cluster.server-quorum-type: server
cluster.lookup-optimize: off
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.choose-local: off
client.event-threads: 4
performance.client-io-threads: on
nfs.disable: on
transport.address-family: inet
storage.fips-mode-rchecksum: on
storage.owner-uid: 36
storage.owner-gid: 36
features.shard: on
performance.low-prio-threads: 32
performance.strict-o-direct: on
network.remote-dio: off
network.ping-timeout: 30
user.cifs: off
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
cluster.eager-lock: enable
The speed between two hosts is about 1Gb/s:
[root@ovirt-host1 ~]# iperf3 -c ovirt-host2 -p 5002
Connecting to host ovirt-host2 port 5002
[ 5] local x.x.x.x port 58072 connected to y.y.y.y port 5002
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 112 MBytes 938 Mbits/sec 117 375 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 937 Mbits/sec 0 397 KBytes
[ 5] 2.00-3.00 sec 110 MBytes 924 Mbits/sec 18 344 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 936 Mbits/sec 0 369 KBytes
[ 5] 4.00-5.00 sec 111 MBytes 927 Mbits/sec 12 386 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 938 Mbits/sec 0 471 KBytes
[ 5] 6.00-7.00 sec 108 MBytes 909 Mbits/sec 34 382 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 0 438 KBytes
[ 5] 8.00-9.00 sec 111 MBytes 928 Mbits/sec 38 372 KBytes
[ 5] 9.00-10.00 sec 111 MBytes 934 Mbits/sec 0 481 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.08 GBytes 931 Mbits/sec 219 sender
[ 5] 0.00-10.04 sec 1.08 GBytes 926 Mbits/sec receiver
iperf Done.
Between nodes and arbiter about 200MB/s
[ 5] local ovirt-arbiter port 45220 connected to ovirt-host1 port 5002
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 30.6 MBytes 257 Mbits/sec 1177 281 KBytes
[ 5] 1.00-2.00 sec 26.2 MBytes 220 Mbits/sec 0 344 KBytes
[ 5] 2.00-3.00 sec 28.8 MBytes 241 Mbits/sec 15 288 KBytes
[ 5] 3.00-4.00 sec 26.2 MBytes 220 Mbits/sec 0 352 KBytes
[ 5] 4.00-5.00 sec 30.0 MBytes 252 Mbits/sec 32 293 KBytes
[ 5] 5.00-6.00 sec 26.2 MBytes 220 Mbits/sec 0 354 KBytes
[ 5] 6.00-7.00 sec 30.0 MBytes 252 Mbits/sec 32 293 KBytes
[ 5] 7.00-8.00 sec 27.5 MBytes 231 Mbits/sec 0 355 KBytes
[ 5] 8.00-9.00 sec 28.8 MBytes 241 Mbits/sec 30 294 KBytes
[ 5] 9.00-10.00 sec 26.2 MBytes 220 Mbits/sec 3 250 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 281 MBytes 235 Mbits/sec 1289 sender
[ 5] 0.00-10.03 sec 277 MBytes 232 Mbits/sec receiver
iperf Done.
I definitely missing something obvious and I'm not a gluster/ovirt black bealt... Can
anyone point me to the right way?
Thank you for your time.
Regards,
Francesco