Hi Mathieu, How are you measuring the Gluster disk performance? also using dd you should use the oflag=dsync to avoid buffer caching. Regards, Paul S ________________________________ From: Mathieu Valois <mvalois(a)teicee.com&gt; Sent: 08 September 2021 10:12 To: users <users(a)ovirt.org&gt; Subject: [ovirt-users] Poor gluster performances over 10Gbps network Caution External Mail: Do not click any links or open any attachments unless you trust the sender and know that the content is safe. Sorry for double post but I don't know if this mail has been received. Hello everyone, I know this issue was already treated on this mailing list. However none of the proposed solutions is satisfying me. Here is my situation : I've got 3 hyperconverged gluster ovirt nodes, with 6 network interfaces, bounded in bunches of 2 (management, VMs and gluster). The gluster network is on a dedicated bound where the 2 interfaces are directly connected to the 2 other ovirt nodes. Gluster is apparently using it : # gluster volume status vmstore Status of volume: vmstore Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick gluster-ov1:/gluster_bricks /vmstore/vmstore 49152 0 Y 3019 Brick gluster-ov2:/gluster_bricks /vmstore/vmstore 49152 0 Y 3009 Brick gluster-ov3:/gluster_bricks /vmstore/vmstore where 'gluster-ov{1,2,3}' are domain names referencing nodes in the gluster network. This networks has 10Gbps capabilities : # iperf3 -c gluster-ov3 Connecting to host gluster-ov3, port 5201 [ 5] local 10.20.0.50 port 46220 connected to 10.20.0.51 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.16 GBytes 9.92 Gbits/sec 17 900 KBytes [ 5] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec 0 900 KBytes [ 5] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec 4 996 KBytes [ 5] 3.00-4.00 sec 1.15 GBytes 9.90 Gbits/sec 1 996 KBytes [ 5] 4.00-5.00 sec 1.15 GBytes 9.89 Gbits/sec 0 996 KBytes [ 5] 5.00-6.00 sec 1.15 GBytes 9.90 Gbits/sec 0 996 KBytes [ 5] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec 0 996 KBytes [ 5] 7.00-8.00 sec 1.15 GBytes 9.91 Gbits/sec 0 996 KBytes [ 5] 8.00-9.00 sec 1.15 GBytes 9.90 Gbits/sec 0 996 KBytes [ 5] 9.00-10.00 sec 1.15 GBytes 9.90 Gbits/sec 0 996 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.5 GBytes 9.90 Gbits/sec 22 sender [ 5] 0.00-10.04 sec 11.5 GBytes 9.86 Gbits/sec receiver iperf Done. However, VMs stored on the vmstore gluster volume has poor write performances, oscillating between 100KBps and 30MBps. I almost always observe a write spike (180Mbps) at the beginning until around 500MB written, then it drastically falls at 10MBps, sometimes even less (100KBps). Hypervisors have 32 threads (2 sockets, 8 cores per socket, 2 threads per core). Here is the volume settings : Volume Name: vmstore Type: Replicate Volume ID: XXX Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: gluster-ov1:/gluster_bricks/vmstore/vmstore Brick2: gluster-ov2:/gluster_bricks/vmstore/vmstore Brick3: gluster-ov3:/gluster_bricks/vmstore/vmstore Options Reconfigured: performance.io-thread-count: 32 # was 16 by default. cluster.granular-entry-heal: enable storage.owner-gid: 36 storage.owner-uid: 36 cluster.lookup-optimize: off server.keepalive-count: 5 server.keepalive-interval: 2 server.keepalive-time: 10 server.tcp-user-timeout: 20 network.ping-timeout: 30 server.event-threads: 4 client.event-threads: 8 # was 4 by default cluster.choose-local: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable performance.strict-o-direct: on network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off auth.allow: * user.cifs: off storage.fips-mode-rchecksum: on transport.address-family: inet nfs.disable: on performance.client-io-threads: on When I naively write directly on the logical volume, which is mounted on a material RAID5 3-disks array, I have interesting performances: # dd if=/dev/zero of=a bs=4M count=2048 2048+0 records in 2048+0 records out 8589934592 bytes (8.6 GB, 8.0 GiB) copied, 17.2485 s, 498 MB/s #urandom gives around 200MBps Moreover, hypervisors have SSD which have been configured as lvcache, but I'm unsure how to test it efficiently. I can't find where is the problem, as every piece of the chain is apparently doing well ... Thanks anyone for helping me :) -- [téïcée]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2... Mathieu Valois Bureau Caen: Quartier Kœnig - 153, rue Géraldine MOCK - 14760 Bretteville-sur-Odon Bureau Vitré: Zone de la baratière - 12, route de Domalain - 35500 Vitré 02 72 34 13 20 | www.teicee.com<https://eur02.safelinks.protection.outlook.com/?url=htt... [téïcée sur facebook]<https://eur02.safelinks.protection.outlook.com/?url=https%3A... [téïcée sur twitter] <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter... [téïcée sur linkedin] <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.lin... [téïcée sur viadeo] <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffr.viad... [Datadocké] To view the terms under which this email is distributed, please go to:- https://leedsbeckett.ac.uk/disclaimer/email

You can find attached the benchmarks on the host and guest. I find the differences not so big though... Le 09/09/2021 à 13:40, Nir Soffer a écrit : -- téïcée <https://www.teicee.com/?pk_campaign=Email> *Mathieu Valois* Bureau Caen: Quartier Kœnig - 153, rue Géraldine MOCK - 14760 Bretteville-sur-Odon Bureau Vitré: Zone de la baratière - 12, route de Domalain - 35500 Vitré 02 72 34 13 20 | www.teicee.com <https://www.teicee.com/?pk_campaign=Email> téïcée sur facebook <https://www.facebook.com/teicee> téïcée sur twitter <https://twitter.com/Teicee_fr> téïcée sur linkedin <https://www.linkedin.com/company/t-c-e> téïcée sur viadeo <https://fr.viadeo.com/fr/company/teicee> Datadocké

First of all, many thanks for your analysis! Le 09/09/2021 à 17:06, Nir Soffer a écrit : Into the brick :   # fio --filename=/gluster_bricks/vmstore/fio.data --output=/root/test.out /root/bench.fio Yes, LVM is on hard disk RAID with lvcache using SSD. I've configured hyperconverged gluster with 372Go of SSD cache, but 7GiB/s bandwidth seems surprisingly huge. This is what I feel too. Yes, this is very likely. I've analyzed the gluster volume profiling tool, and I the most time-costly operation is FSYNC : # gluster volume profile vmstore info  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop  ---------   -----------   -----------   ----------- ------------        ----       0.00       0.00 us       0.00 us       0.00 us              6      FORGET       0.00       0.00 us       0.00 us       0.00 us 210     RELEASE       0.00       0.00 us       0.00 us       0.00 us 42  RELEASEDIR       0.00     212.06 us      86.19 us     396.24 us              4    READDIRP       0.00     105.13 us      10.31 us     241.91 us 14    GETXATTR       0.00     316.95 us     196.80 us     400.89 us              6      CREATE       0.00     234.14 us      15.50 us     805.85 us 10     READDIR       0.00     213.87 us     137.89 us     294.45 us 12      UNLINK       0.00      73.30 us       1.11 us     154.41 us 42     OPENDIR       0.00     318.69 us     211.02 us     465.69 us 23       MKNOD       0.00      46.98 us      13.80 us     140.04 us 201       FLUSH       0.00      98.34 us      43.90 us     308.65 us 204        OPEN       0.00      66.89 us      21.80 us     140.42 us 398      STATFS       0.00     588.67 us      47.60 us   38236.37 us 78       FSTAT       0.02     908.06 us      14.01 us   10798.16 us 249     ENTRYLK       0.02     167.16 us      15.54 us     476.21 us 1787      LOOKUP       0.15   31851.02 us      22.39 us  366250.86 us 57     INODELK       0.16     120.31 us      32.84 us   71638.49 us 15827    FXATTROP       0.68    1430.51 us      11.70 us 1152275.20 us 5719    FINODELK       0.71    3211.50 us      47.48 us  191016.03 us 2667        READ      38.44   50693.63 us     329.41 us  617663.12 us 9134       FSYNC      59.80   21063.30 us      56.86 us  663984.57 us 34197       WRITE       0.00       0.00 us       0.00 us       0.00 us 31      UPCALL I'll test that script. -- téïcée <https://www.teicee.com/?pk_campaign=Email> *Mathieu Valois* Bureau Caen: Quartier Kœnig - 153, rue Géraldine MOCK - 14760 Bretteville-sur-Odon Bureau Vitré: Zone de la baratière - 12, route de Domalain - 35500 Vitré 02 72 34 13 20 | www.teicee.com <https://www.teicee.com/?pk_campaign=Email> téïcée sur facebook <https://www.facebook.com/teicee> téïcée sur twitter <https://twitter.com/Teicee_fr> téïcée sur linkedin <https://www.linkedin.com/company/t-c-e> téïcée sur viadeo <https://fr.viadeo.com/fr/company/teicee> Datadocké

It's my understanding that libgfapi is currently unsupported in oVirt because there were a few long standing bugs. Unfortunately, the performance improvements that users have reported on this mailing list haven't been seen or replicated by Red Hat, so recently most of these bugzilla tickets have been closed as WONTFIX https://bugzilla.redhat.com/show_bug.cgi?id=1633642 https://bugzilla.redhat.com/show_bug.cgi?id=1465810 https://bugzilla.redhat.com/show_bug.cgi?id=1552344 This is a shame as I think the performance improvements from libgfapi as reported in user benchmarks were impressive, and the underlying qemu/libvirt bugs are now fixed or were close to be. Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group On Thu, Sep 9, 2021 at 7:00 PM Strahil Nikolov via Users <users(a)ovirt.org&gt; wrote: -- Ce message et toutes les pièces jointes (ci-après le “message”) sont établis à l’intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse. L’internet ne permettant pas d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse ou il aurait été modifié. IT, ES, UK.  <https://interactiv-group.com/disclaimer.html>