First of all, many thanks for your analysis! Le 09/09/2021 à 17:06, Nir Soffer a écrit :
On Thu, Sep 9, 2021 at 4:12 PM Mathieu Valois <mvalois@teicee.com <mailto:mvalois@teicee.com>> wrote:
Host is using the gluster mount (/rhev/data-center/mnt/glusterSD/server:_path/...) or writing directly into the same filesystem used by gluster (/bricks/brick1/...)?
Into the brick : # fio --filename=/gluster_bricks/vmstore/fio.data --output=/root/test.out /root/bench.fio
Comparing host and guest:
seq-write: (groupid=0, jobs=4): err= 0: pid=294433: Thu Sep 9 14:30:14 2021 write: IOPS=151, BW=153MiB/s (160MB/s)(4628MiB/30280msec); 0 zone resets
I guess the underlying storage is hard disk - 150 MiB/s is not bad but very low compared with fast SSD.
Yes, LVM is on hard disk RAID with lvcache using SSD.
seq-read: (groupid=1, jobs=4): err= 0: pid=294778: Thu Sep 9 14:30:14 2021 read: IOPS=7084, BW=7086MiB/s (7430MB/s)(208GiB/30016msec)
You have crazy caching (ignoring the direct I/O?), 7GiB/s read?
I've configured hyperconverged gluster with 372Go of SSD cache, but 7GiB/s bandwidth seems surprisingly huge.
...
So you have very fast reads (seq/random), with very slow seq/random write. This is what I feel too.
Also would be interesting to test fsync - this benchmark does not do any fsync but your slow yum/rpm upgrade likey do one of more fsyncs per package upgrade.
Yes, this is very likely. I've analyzed the gluster volume profiling tool, and I the most time-costly operation is FSYNC : # gluster volume profile vmstore info %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 6 FORGET 0.00 0.00 us 0.00 us 0.00 us 210 RELEASE 0.00 0.00 us 0.00 us 0.00 us 42 RELEASEDIR 0.00 212.06 us 86.19 us 396.24 us 4 READDIRP 0.00 105.13 us 10.31 us 241.91 us 14 GETXATTR 0.00 316.95 us 196.80 us 400.89 us 6 CREATE 0.00 234.14 us 15.50 us 805.85 us 10 READDIR 0.00 213.87 us 137.89 us 294.45 us 12 UNLINK 0.00 73.30 us 1.11 us 154.41 us 42 OPENDIR 0.00 318.69 us 211.02 us 465.69 us 23 MKNOD 0.00 46.98 us 13.80 us 140.04 us 201 FLUSH 0.00 98.34 us 43.90 us 308.65 us 204 OPEN 0.00 66.89 us 21.80 us 140.42 us 398 STATFS 0.00 588.67 us 47.60 us 38236.37 us 78 FSTAT 0.02 908.06 us 14.01 us 10798.16 us 249 ENTRYLK 0.02 167.16 us 15.54 us 476.21 us 1787 LOOKUP 0.15 31851.02 us 22.39 us 366250.86 us 57 INODELK 0.16 120.31 us 32.84 us 71638.49 us 15827 FXATTROP 0.68 1430.51 us 11.70 us 1152275.20 us 5719 FINODELK 0.71 3211.50 us 47.48 us 191016.03 us 2667 READ 38.44 50693.63 us 329.41 us 617663.12 us 9134 FSYNC 59.80 21063.30 us 56.86 us 663984.57 us 34197 WRITE 0.00 0.00 us 0.00 us 0.00 us 31 UPCALL
There is an example sync test script here: https://www.ibm.com/cloud/blog/using-fio-to-tell-whether-your-storage-is-fas... <https://www.ibm.com/cloud/blog/using-fio-to-tell-whether-your-storage-is-fast-enough-for-etcd>
I'll test that script. -- téïcée <https://www.teicee.com/?pk_campaign=Email> *Mathieu Valois* Bureau Caen: Quartier Kœnig - 153, rue Géraldine MOCK - 14760 Bretteville-sur-Odon Bureau Vitré: Zone de la baratière - 12, route de Domalain - 35500 Vitré 02 72 34 13 20 | www.teicee.com <https://www.teicee.com/?pk_campaign=Email> téïcée sur facebook <https://www.facebook.com/teicee> téïcée sur twitter <https://twitter.com/Teicee_fr> téïcée sur linkedin <https://www.linkedin.com/company/t-c-e> téïcée sur viadeo <https://fr.viadeo.com/fr/company/teicee> Datadocké
