First of all, many thanks for your analysis!

On Thu, Sep 9, 2021 at 4:12 PM Mathieu Valois <mvalois@teicee.com> wrote:

Host is using the gluster mount (/rhev/data-center/mnt/glusterSD/server:_path/...)

or writing directly into the same filesystem used by gluster (/bricks/brick1/...)?

Into the brick :

  # fio --filename=/gluster_bricks/vmstore/fio.data --output=/root/test.out /root/bench.fio

Comparing host and guest:

seq-write: (groupid=0, jobs=4): err= 0: pid=294433: Thu Sep  9 14:30:14 2021
  write: IOPS=151, BW=153MiB/s (160MB/s)(4628MiB/30280msec); 0 zone resets

I guess the underlying storage is hard disk - 150 MiB/s is not bad but

very low compared with fast SSD.

seq-read: (groupid=1, jobs=4): err= 0: pid=294778: Thu Sep  9 14:30:14 2021
  read: IOPS=7084, BW=7086MiB/s (7430MB/s)(208GiB/30016msec)

You have crazy caching (ignoring the direct I/O?), 7GiB/s read?

...

So you have very fast reads (seq/random), with very slow seq/random write.

Also would be interesting to test fsync - this benchmark does not do any fsync

but your slow yum/rpm upgrade likey do one of more fsyncs per package

upgrade.

Yes, this is very likely. I've analyzed the gluster volume profiling tool, and I the most time-costly operation is FSYNC :

# gluster volume profile vmstore info

 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              6      FORGET
      0.00       0.00 us       0.00 us       0.00 us            210     RELEASE
      0.00       0.00 us       0.00 us       0.00 us             42  RELEASEDIR
      0.00     212.06 us      86.19 us     396.24 us              4    READDIRP
      0.00     105.13 us      10.31 us     241.91 us             14    GETXATTR
      0.00     316.95 us     196.80 us     400.89 us              6      CREATE
      0.00     234.14 us      15.50 us     805.85 us             10     READDIR
      0.00     213.87 us     137.89 us     294.45 us             12      UNLINK
      0.00      73.30 us       1.11 us     154.41 us             42     OPENDIR
      0.00     318.69 us     211.02 us     465.69 us             23       MKNOD
      0.00      46.98 us      13.80 us     140.04 us            201       FLUSH
      0.00      98.34 us      43.90 us     308.65 us            204        OPEN
      0.00      66.89 us      21.80 us     140.42 us            398      STATFS
      0.00     588.67 us      47.60 us   38236.37 us             78       FSTAT
      0.02     908.06 us      14.01 us   10798.16 us            249     ENTRYLK
      0.02     167.16 us      15.54 us     476.21 us           1787      LOOKUP
      0.15   31851.02 us      22.39 us  366250.86 us             57     INODELK
      0.16     120.31 us      32.84 us   71638.49 us          15827    FXATTROP
      0.68    1430.51 us      11.70 us 1152275.20 us           5719    FINODELK
      0.71    3211.50 us      47.48 us  191016.03 us           2667        READ
     38.44   50693.63 us     329.41 us  617663.12 us           9134       FSYNC
     59.80   21063.30 us      56.86 us  663984.57 us          34197       WRITE
      0.00       0.00 us       0.00 us       0.00 us             31      UPCALL

There is an example sync test script here:

https://www.ibm.com/cloud/blog/using-fio-to-tell-whether-your-storage-is-fast-enough-for-etcd

I'll test that script.

--

Mathieu Valois Bureau Caen: Quartier Kœnig - 153, rue Géraldine MOCK - 14760 Bretteville-sur-Odon Bureau Vitré: Zone de la baratière - 12, route de Domalain - 35500 Vitré 02 72 34 13 20 | www.teicee.com