8:09 a.m.
Hello All,
We have 3 servers with a raid 50 array each, we are having extreme
performance issues with our gluster, writes on gluster seem to take at
least 3 times longer than on the raid directly. Can this be improved? I've
read through several other performance issues threads but have been unable
to make any improvements
"gluster volume info" and "gluster volume profile vmstore info" is
below
=========================================================================================================================
-Inside Gluster - test took 35+ hours:
[root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G
-n 0 -m TEST -f -b -u root
Using uid:0, gid:0.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Version 1.98 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
TEST 600G 35.7m 17 5824k 7 112m 13 182.7
6
Latency 5466ms 12754ms 3499ms
1589ms
1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,,,,,,,,,,,,,,,,,5466ms,12754ms,,3499ms,1589ms,,,,,,
=========================================================================================================================
-Outside Gluster - test took 18 minutes:
[root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G
-n 0 -m TEST -f -b -u root
Using uid:0, gid:0.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Version 1.98 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
TEST 600G 567m 78 149m 30 307m 37 83.0
57
Latency 205ms 4630ms 1450ms
679ms
1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,,,,,,,,,,,,,,,,,205ms,4630ms,,1450ms,679ms,,,,,,
=========================================================================================================================
[root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume info
Volume Name: engine
Type: Replicate
Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine
Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine
Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine
Options Reconfigured:
cluster.granular-entry-heal: enable
performance.strict-o-direct: on
network.ping-timeout: 30
storage.owner-gid: 36
storage.owner-uid: 36
server.event-threads: 4
client.event-threads: 4
cluster.choose-local: off
user.cifs: off
features.shard: on
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.server-quorum-type: server
cluster.quorum-type: auto
cluster.eager-lock: enable
network.remote-dio: off
performance.low-prio-threads: 32
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: on
diagnostics.latency-measurement: on
diagnostics.count-fop-hits: on
Volume Name: vmstore
Type: Replicate
Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore
Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore
Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore
Options Reconfigured:
cluster.granular-entry-heal: enable
performance.strict-o-direct: on
network.ping-timeout: 20
storage.owner-gid: 36
storage.owner-uid: 36
server.event-threads: 4
client.event-threads: 4
cluster.choose-local: off
user.cifs: off
features.shard: on
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.server-quorum-type: server
cluster.quorum-type: auto
cluster.eager-lock: enable
network.remote-dio: off
performance.low-prio-threads: 32
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: on
diagnostics.latency-measurement: on
diagnostics.count-fop-hits: on
server.tcp-user-timeout: 20
server.keepalive-time: 10
server.keepalive-interval: 2
server.keepalive-count: 5
cluster.lookup-optimize: off
=========================================================================================================================
[root@ovirt1 ~]# gluster volume profile vmstore info
Brick: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore
---------------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 4b+
8b+
No. of Reads: 0 0
0
No. of Writes: 4021 2
64
Block Size: 64b+ 128b+
256b+
No. of Reads: 0 134
405
No. of Writes: 2 32
119
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 2890 2617
3059
No. of Writes: 113881 93551
131776
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 1096597 164931
181569
No. of Writes: 41091001 20499776
3230472
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 212970 96574
244017
No. of Writes: 1304688 991154
3412866
Block Size: 262144b+ 524288b+
1048576b+
No. of Reads: 0 0
0
No. of Writes: 15 18
65166
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 6396
FORGET
0.00 0.00 us 0.00 us 0.00 us 189759
RELEASE
0.00 0.00 us 0.00 us 0.00 us 31292
RELEASEDIR
0.00 159.99 us 44.37 us 278.66 us 79
STAT
0.00 364.42 us 174.87 us 644.45 us 96
LINK
0.00 2714.10 us 203.53 us 47397.28 us 26
RMDIR
0.00 688.26 us 72.39 us 53857.63 us 186
SETXATTR
0.00 974.01 us 271.02 us 27298.52 us 176
RENAME
0.00 5254.64 us 159.48 us 84454.46 us 35
DISCARD
0.00 1133.84 us 86.47 us 87096.92 us 186
REMOVEXATTR
0.00 7907.47 us 221.82 us 182276.21 us 29
FALLOCATE
0.00 8304.51 us 311.89 us 42725.81 us 34
MKDIR
0.00 5758.19 us 296.39 us 21950.99 us 81
TRUNCATE
0.00 401.16 us 23.68 us 132668.93 us 3621
LK
0.00 545.48 us 14.22 us 1076517.76 us 6796
READDIR
0.00 2250.78 us 336.61 us 1071764.61 us 4257
CREATE
0.00 531.76 us 1.35 us 1723046.94 us 31292
OPENDIR
0.01 2058.67 us 196.74 us 1587752.27 us 21986
MKNOD
0.01 1241.28 us 22.61 us 1548642.70 us 52674
READDIRP
0.01 67508.32 us 100.01 us 3002069.81 us 1100
FTRUNCATE
0.01 412.74 us 10.94 us 5111334.35 us 209731
FLUSH
0.02 78668.59 us 183.06 us 3012629.87 us 1222
SETATTR
0.03 761.13 us 21.13 us 4611854.15 us 249094
STATFS
0.03 1132.33 us 45.51 us 3159531.87 us 185654
OPEN
0.03 2010.33 us 14.43 us 3158098.85 us 107824
GETXATTR
0.04 2765.29 us 61.38 us 3159208.13 us 100484
XATTROP
0.13 1367.04 us 51.46 us 8450420.87 us 595299
FSTAT
0.46 157440.90 us 47.62 us 13405712.93 us 18218
UNLINK
0.60 1274.67 us 19.60 us 11985005.14 us 2980806
LOOKUP
1.67 16645.99 us 12.80 us 6869488.64 us 631331
ENTRYLK
4.10 7504.86 us 39.52 us 11749528.82 us 3430506
READ
4.31 56942.36 us 16.57 us 23654127.65 us 475563
INODELK
6.09 660.89 us 11.02 us 20571410.32 us 57908331
FINODELK
20.59 1823.35 us 63.26 us 23639762.45 us 70934143
WRITE
25.70 2680.82 us 35.37 us 13636394.75 us 60223005
FXATTROP
36.14 3164.86 us 52.35 us 16805886.48 us 71739477
FSYNC
0.00 0.00 us 0.00 us 0.00 us 78458
UPCALL
Duration: 404667 seconds
Data Read: 59850935724 bytes
Data Written: 1070158035683 bytes
Interval 43 Stats:
Block Size: 1b+ 256b+
512b+
No. of Reads: 0 14
3
No. of Writes: 12 4
252
Block Size: 1024b+ 2048b+
4096b+
No. of Reads: 0 0
8105
No. of Writes: 100 34
147355
Block Size: 8192b+ 16384b+
32768b+
No. of Reads: 1559 1630
1494
No. of Writes: 86013 17318
8687
Block Size: 65536b+ 131072b+
No. of Reads: 532 764
No. of Writes: 8617 36317
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 16
FORGET
0.00 0.00 us 0.00 us 0.00 us 665
RELEASE
0.00 0.00 us 0.00 us 0.00 us 52
RELEASEDIR
0.00 181.75 us 181.75 us 181.75 us 1
STAT
0.00 914.55 us 858.55 us 970.56 us 2
TRUNCATE
0.00 792.11 us 681.24 us 1051.32 us 4
RENAME
0.00 374.44 us 31.86 us 792.10 us 20
READDIR
0.00 399.48 us 306.97 us 562.69 us 28
UNLINK
0.00 752.40 us 636.14 us 907.29 us 16
CREATE
0.00 196.62 us 33.27 us 2586.77 us 336
GETXATTR
0.00 1107.98 us 288.51 us 29692.53 us 80
MKNOD
0.00 1074.80 us 74.41 us 2121.03 us 126
READDIRP
0.00 3119.52 us 3.17 us 157121.44 us 52
OPENDIR
0.01 972.14 us 21.72 us 329342.21 us 635
FLUSH
0.03 2122.15 us 31.97 us 377538.16 us 732
STATFS
0.06 4478.89 us 66.09 us 521289.10 us 649
OPEN
0.10 10273.00 us 88.19 us 478524.77 us 472
XATTROP
0.16 6969.92 us 16.14 us 385156.98 us 1138
ENTRYLK
0.20 4635.40 us 64.29 us 1098319.92 us 2097
FSTAT
0.65 4098.07 us 46.85 us 919617.66 us 7797
LOOKUP
2.36 87465.33 us 34.75 us 1224052.07 us 1317
INODELK
4.37 13897.55 us 57.89 us 1984970.27 us 15336
READ
5.24 1265.46 us 16.22 us 1638678.32 us 202165
FINODELK
21.05 3370.28 us 144.44 us 2264130.49 us 304724
WRITE
31.00 6411.68 us 39.84 us 2190355.42 us 235969
FXATTROP
34.75 6107.95 us 60.26 us 2270252.44 us 277643
FSYNC
0.00 0.00 us 0.00 us 0.00 us 368
UPCALL
Duration: 1132 seconds
Data Read: 292367566 bytes
Data Written: 7495339692 bytes
Brick: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore
---------------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 4b+
8b+
No. of Reads: 0 0
0
No. of Writes: 4802 2
64
Block Size: 64b+ 128b+
256b+
No. of Reads: 0 138
828
No. of Writes: 2 32
131
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 3595 4590
6161
No. of Writes: 128976 99680
132413
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 930031 128995
150468
No. of Writes: 48357551 23957030
3689474
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 154039 71949
790010
No. of Writes: 1438859 1052045
3715427
Block Size: 262144b+ 524288b+
1048576b+
No. of Reads: 9 11
40269
No. of Writes: 5 20
50757
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 7629
FORGET
0.00 0.00 us 0.00 us 0.00 us 228577
RELEASE
0.00 0.00 us 0.00 us 0.00 us 41524
RELEASEDIR
0.00 374.22 us 219.35 us 574.87 us 96
LINK
0.00 1712.22 us 164.90 us 22337.79 us 41
DISCARD
0.00 2840.25 us 245.24 us 46839.63 us 26
RMDIR
0.00 3660.92 us 229.86 us 86252.56 us 35
FALLOCATE
0.00 876.48 us 297.53 us 16685.63 us 188
RENAME
0.00 6660.10 us 379.21 us 52962.79 us 34
MKDIR
0.00 957.48 us 80.36 us 218059.88 us 355
REMOVEXATTR
0.00 1234.85 us 76.50 us 322955.27 us 355
SETXATTR
0.00 2326.62 us 84.07 us 250041.25 us 196
STAT
0.00 9660.45 us 289.50 us 295106.01 us 87
TRUNCATE
0.00 486.10 us 24.12 us 869966.79 us 5260
LK
0.00 453.69 us 18.32 us 426653.18 us 8135
READDIR
0.00 1832.38 us 296.61 us 261299.81 us 5050
CREATE
0.00 386.77 us 1.31 us 1300836.12 us 41524
OPENDIR
0.00 712.06 us 29.85 us 1441115.02 us 34874
READDIRP
0.01 1469.51 us 231.67 us 440065.71 us 22845
MKNOD
0.01 44229.36 us 75.58 us 1803963.42 us 871
FTRUNCATE
0.01 84433.20 us 149.47 us 3614869.24 us 1001
SETATTR
0.02 418.11 us 13.39 us 3515000.81 us 243820
FLUSH
0.03 1108.47 us 14.77 us 1658743.41 us 130647
GETXATTR
0.03 562.86 us 26.74 us 5043949.33 us 297799
STATFS
0.03 845.50 us 48.16 us 1998938.03 us 223680
OPEN
0.05 2236.13 us 56.08 us 5295998.22 us 120682
XATTROP
0.16 890.82 us 43.51 us 3653292.88 us 1012585
FSTAT
0.34 2974.20 us 12.92 us 7782497.74 us 642555
ENTRYLK
0.47 135831.21 us 70.35 us 11033867.90 us 19800
UNLINK
0.57 954.05 us 21.50 us 4553393.38 us 3414605
LOOKUP
1.79 14461.54 us 13.45 us 32841452.98 us 702915
INODELK
5.14 8014.78 us 40.70 us 5439109.56 us 3644063
READ
5.37 443.04 us 11.53 us 32863652.53 us 68909131
FINODELK
22.31 1780.31 us 33.59 us 11318712.62 us 71235991
FXATTROP
22.83 1571.16 us 74.86 us 32615055.19 us 82622840
WRITE
40.84 2762.75 us 52.77 us 8859115.35 us 84039509
FSYNC
0.00 0.00 us 0.00 us 0.00 us 95492
UPCALL
Duration: 484169 seconds
Data Read: 167149718723 bytes
Data Written: 1177141649872 bytes
Interval 43 Stats:
Block Size: 1b+ 256b+
512b+
No. of Reads: 0 6
4
No. of Writes: 12 4
252
Block Size: 1024b+ 2048b+
4096b+
No. of Reads: 0 0
5668
No. of Writes: 100 34
147357
Block Size: 8192b+ 16384b+
32768b+
No. of Reads: 1178 783
1215
No. of Writes: 86014 17318
8687
Block Size: 65536b+ 131072b+
No. of Reads: 264 4109
No. of Writes: 8617 36317
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 16
FORGET
0.00 0.00 us 0.00 us 0.00 us 665
RELEASE
0.00 0.00 us 0.00 us 0.00 us 52
RELEASEDIR
0.00 866.18 us 849.57 us 882.78 us 2
TRUNCATE
0.00 1016.14 us 868.35 us 1182.48 us 4
RENAME
0.00 455.65 us 41.13 us 1679.37 us 20
READDIR
0.00 373.33 us 173.54 us 538.40 us 28
UNLINK
0.00 722.01 us 635.91 us 853.71 us 16
CREATE
0.00 335.52 us 38.15 us 1381.97 us 54
READDIRP
0.00 39218.92 us 214.34 us 78223.50 us 2
STAT
0.00 1730.77 us 2.59 us 84964.69 us 52
OPENDIR
0.00 1304.96 us 393.64 us 29724.09 us 80
MKNOD
0.01 422.75 us 22.85 us 134336.60 us 635
FLUSH
0.01 699.21 us 37.54 us 137813.64 us 732
STATFS
0.01 468.63 us 23.71 us 260961.83 us 1141
ENTRYLK
0.04 2041.83 us 69.80 us 163743.98 us 649
OPEN
0.08 1253.65 us 72.52 us 290508.15 us 2354
FSTAT
0.09 6913.19 us 69.01 us 212479.24 us 472
XATTROP
0.13 3558.26 us 32.70 us 195896.10 us 1317
INODELK
0.13 9793.37 us 36.11 us 212755.58 us 499
GETXATTR
0.34 1615.15 us 55.42 us 711310.28 us 7797
LOOKUP
4.92 13025.92 us 59.83 us 1061585.06 us 13884
READ
5.26 955.95 us 16.81 us 1069148.49 us 202174
FINODELK
23.28 2809.44 us 137.21 us 1134076.13 us 304713
WRITE
26.46 4124.05 us 43.30 us 1142167.27 us 235954
FXATTROP
39.23 5195.62 us 65.47 us 1250469.35 us 277632
FSYNC
0.00 0.00 us 0.00 us 0.00 us 365
UPCALL
Duration: 1132 seconds
Data Read: 668951066 bytes
Data Written: 7495356076 bytes