[ovirt-users] Re: GlusterFS poor performance

Hi all, I'm running a glusterFS setup v 8.6 with two node and one arbiter. Both nodes and arbiter are CentOS 8 Stream with oVirt 4.4. Under gluster I have a LVM thin partition. VMs running in this cluster have really poor write performance, when a test directly performend on the disk score about 300 MB/s dd test on host1: [root@ovirt-host1 tmp]# dd if=/dev/zero of=./foo.dat bs=256M count=1 oflag=dsync 1+0 records in 1+0 records out 268435456 bytes (268 MB, 256 MiB) copied, 0.839861 s, 320 MB/s dd test on host1 on gluster: [root@ovirt-host1 tmp]# dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/ovirt-host1:_data/foo.dat bs=256M count=1 oflag=dsync 1+0 records in 1+0 records out 268435456 bytes (268 MB, 256 MiB) copied, 50.6889 s, 5.3 MB/s Nontheless, the write results in a VM inside the cluster is a little bit faster (dd results vary from 15 MB/s to 60 MB/s) and this is very strange to me: root@vm1-ha:/tmp# dd if=/dev/zero of=./foo.dat bs=256M count=1 oflag=dsync; rm -f ./foo.dat 1+0 records in 1+0 records out 268435456 bytes (268 MB, 256 MiB) copied, 5.58727 s, 48.0 MB/s Here's the actual gluster configuration, I also applied some paramaters in /var/lib/glusterd/groups/virt as mentioned in other ovirt thread related I found. gluster volume info data Volume Name: data Type: Replicate Volume ID: 09b532eb-57de-4c29-862d-93993c990e32 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: ovirt-host1:/gluster_bricks/data/data Brick2: ovirt-host2:/gluster_bricks/data/data Brick3: ovirt-arbiter:/gluster_bricks/data/data (arbiter) Options Reconfigured: server.event-threads: 4 cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.server-quorum-type: server cluster.lookup-optimize: off cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.choose-local: off client.event-threads: 4 performance.client-io-threads: on nfs.disable: on transport.address-family: inet storage.fips-mode-rchecksum: on storage.owner-uid: 36 storage.owner-gid: 36 features.shard: on performance.low-prio-threads: 32 performance.strict-o-direct: on network.remote-dio: off network.ping-timeout: 30 user.cifs: off performance.quick-read: off performance.read-ahead: off performance.io-cache: off cluster.eager-lock: enable The speed between two hosts is about 1Gb/s: [root@ovirt-host1 ~]# iperf3 -c ovirt-host2 -p 5002 Connecting to host ovirt-host2 port 5002 [ 5] local x.x.x.x port 58072 connected to y.y.y.y port 5002 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 112 MBytes 938 Mbits/sec 117 375 KBytes [ 5] 1.00-2.00 sec 112 MBytes 937 Mbits/sec 0 397 KBytes [ 5] 2.00-3.00 sec 110 MBytes 924 Mbits/sec 18 344 KBytes [ 5] 3.00-4.00 sec 112 MBytes 936 Mbits/sec 0 369 KBytes [ 5] 4.00-5.00 sec 111 MBytes 927 Mbits/sec 12 386 KBytes [ 5] 5.00-6.00 sec 112 MBytes 938 Mbits/sec 0 471 KBytes [ 5] 6.00-7.00 sec 108 MBytes 909 Mbits/sec 34 382 KBytes [ 5] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 0 438 KBytes [ 5] 8.00-9.00 sec 111 MBytes 928 Mbits/sec 38 372 KBytes [ 5] 9.00-10.00 sec 111 MBytes 934 Mbits/sec 0 481 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.08 GBytes 931 Mbits/sec 219 sender [ 5] 0.00-10.04 sec 1.08 GBytes 926 Mbits/sec receiver iperf Done. Between nodes and arbiter about 200MB/s [ 5] local ovirt-arbiter port 45220 connected to ovirt-host1 port 5002 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 30.6 MBytes 257 Mbits/sec 1177 281 KBytes [ 5] 1.00-2.00 sec 26.2 MBytes 220 Mbits/sec 0 344 KBytes [ 5] 2.00-3.00 sec 28.8 MBytes 241 Mbits/sec 15 288 KBytes [ 5] 3.00-4.00 sec 26.2 MBytes 220 Mbits/sec 0 352 KBytes [ 5] 4.00-5.00 sec 30.0 MBytes 252 Mbits/sec 32 293 KBytes [ 5] 5.00-6.00 sec 26.2 MBytes 220 Mbits/sec 0 354 KBytes [ 5] 6.00-7.00 sec 30.0 MBytes 252 Mbits/sec 32 293 KBytes [ 5] 7.00-8.00 sec 27.5 MBytes 231 Mbits/sec 0 355 KBytes [ 5] 8.00-9.00 sec 28.8 MBytes 241 Mbits/sec 30 294 KBytes [ 5] 9.00-10.00 sec 26.2 MBytes 220 Mbits/sec 3 250 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 281 MBytes 235 Mbits/sec 1289 sender [ 5] 0.00-10.03 sec 277 MBytes 232 Mbits/sec receiver iperf Done. I definitely missing something obvious and I'm not a gluster/ovirt black bealt... Can anyone point me to the right way? Thank you for your time. Regards, Francesco _______________________________________________ Users mailing list --users(a)ovirt.org To unsubscribe send an email tousers-leave(a)ovirt.org Privacy Statement:https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct:https://www.ovirt.org/community/about/community-guidelines/ List Archives:https://lists.ovirt.org/archives/list/users@ovirt.org/message/RW...