11:36 a.m.
First of all, many thanks for your analysis!
Le 09/09/2021 à 17:06, Nir Soffer a écrit :
On Thu, Sep 9, 2021 at 4:12 PM Mathieu Valois <mvalois(a)teicee.com
<mailto:mvalois@teicee.com>> wrote:
Host is using the gluster mount
(/rhev/data-center/mnt/glusterSD/server:_path/...)
or writing directly into the same filesystem used by gluster
(/bricks/brick1/...)?
Into the brick :
# fio --filename=/gluster_bricks/vmstore/fio.data
--output=/root/test.out /root/bench.fio
Comparing host and guest:
seq-write: (groupid=0, jobs=4): err= 0: pid=294433: Thu Sep 9
14:30:14 2021
write: IOPS=151, BW=153MiB/s (160MB/s)(4628MiB/30280msec); 0 zone resets
I guess the underlying storage is hard disk - 150 MiB/s is not bad but
very low compared with fast SSD.
Yes, LVM is on hard disk RAID with lvcache using
SSD.
seq-read: (groupid=1, jobs=4): err= 0: pid=294778: Thu Sep 9 14:30:14
2021
read: IOPS=7084, BW=7086MiB/s (7430MB/s)(208GiB/30016msec)
You have crazy caching (ignoring the direct I/O?), 7GiB/s read?
I've configured
hyperconverged gluster with 372Go of SSD cache, but
7GiB/s bandwidth seems surprisingly huge.
...
So you have very fast reads (seq/random), with very slow seq/random write.
This is
what I feel too.
Also would be interesting to test fsync - this benchmark does not do
any fsync
but your slow yum/rpm upgrade likey do one of more fsyncs per package
upgrade.
Yes, this is very likely. I've analyzed the gluster volume profiling
tool, and I the most time-costly operation is FSYNC :
# gluster volume profile vmstore info
%-latency Avg-latency Min-Latency Max-Latency No. of
calls Fop
--------- ----------- ----------- -----------
------------ ----
0.00 0.00 us 0.00 us 0.00 us
6 FORGET
0.00 0.00 us 0.00 us 0.00 us 210 RELEASE
0.00 0.00 us 0.00 us 0.00 us 42 RELEASEDIR
0.00 212.06 us 86.19 us 396.24 us
4 READDIRP
0.00 105.13 us 10.31 us 241.91 us 14 GETXATTR
0.00 316.95 us 196.80 us 400.89 us
6 CREATE
0.00 234.14 us 15.50 us 805.85 us 10 READDIR
0.00 213.87 us 137.89 us 294.45 us 12 UNLINK
0.00 73.30 us 1.11 us 154.41 us 42 OPENDIR
0.00 318.69 us 211.02 us 465.69 us 23 MKNOD
0.00 46.98 us 13.80 us 140.04 us 201 FLUSH
0.00 98.34 us 43.90 us 308.65 us 204 OPEN
0.00 66.89 us 21.80 us 140.42 us 398 STATFS
0.00 588.67 us 47.60 us 38236.37 us 78 FSTAT
0.02 908.06 us 14.01 us 10798.16 us 249 ENTRYLK
0.02 167.16 us 15.54 us 476.21 us 1787 LOOKUP
0.15 31851.02 us 22.39 us 366250.86 us 57 INODELK
0.16 120.31 us 32.84 us 71638.49 us 15827 FXATTROP
0.68 1430.51 us 11.70 us 1152275.20 us 5719 FINODELK
0.71 3211.50 us 47.48 us 191016.03 us 2667 READ
38.44 50693.63 us 329.41 us 617663.12 us 9134 FSYNC
59.80 21063.30 us 56.86 us 663984.57 us 34197 WRITE
0.00 0.00 us 0.00 us 0.00 us 31 UPCALL
There is an example sync test script here:
https://www.ibm.com/cloud/blog/using-fio-to-tell-whether-your-storage-is-...
<https://www.ibm.com/cloud/blog/using-fio-to-tell-whether-your-storage-is-...
I'll test that script.
--
téïcée <https://www.teicee.com/?pk_campaign=Email> *Mathieu Valois*
Bureau Caen: Quartier Kœnig - 153, rue Géraldine MOCK - 14760
Bretteville-sur-Odon
Bureau Vitré: Zone de la baratière - 12, route de Domalain - 35500 Vitré
02 72 34 13 20 | www.teicee.com <https://www.teicee.com/?pk_campaign=Email>
téïcée sur facebook <https://www.facebook.com/teicee> téïcée sur twitter
<https://twitter.com/Teicee_fr> téïcée sur linkedin
<https://www.linkedin.com/company/t-c-e> téïcée sur viadeo
<https://fr.viadeo.com/fr/company/teicee> Datadocké
