Hello All,
We have 3 servers with a raid 50 array each, we are having extreme performance issues with our gluster, writes on gluster seem to take at least 3 times longer than on the raid directly. Can this be improved? I've read through several other performance issues threads but have been unable to make any improvements
"gluster volume info" and "gluster volume profile vmstore info" is below
=========================================================================================================================
-Inside Gluster - test took 35+ hours:
[root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root
Using uid:0, gid:0.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Version 1.98 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
TEST 600G 35.7m 17 5824k 7 112m 13 182.7 6
Latency 5466ms 12754ms 3499ms 1589ms
1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,,,,,,,,,,,,,,,,,5466ms,12754ms,,3499ms,1589ms,,,,,,
=========================================================================================================================
-Outside Gluster - test took 18 minutes:
[root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root
Using uid:0, gid:0.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Version 1.98 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
TEST 600G 567m 78 149m 30 307m 37 83.0 57
Latency 205ms 4630ms 1450ms 679ms
1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,,,,,,,,,,,,,,,,,205ms,4630ms,,1450ms,679ms,,,,,,
=========================================================================================================================
[root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume info
Volume Name: engine
Type: Replicate
Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine
Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine
Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine
Options Reconfigured:
cluster.granular-entry-heal: enable
performance.strict-o-direct: on
network.ping-timeout: 30
storage.owner-gid: 36
storage.owner-uid: 36
server.event-threads: 4
client.event-threads: 4
cluster.choose-local: off
user.cifs: off
features.shard: on
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.server-quorum-type: server
cluster.quorum-type: auto
cluster.eager-lock: enable
network.remote-dio: off
performance.low-prio-threads: 32
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: on
diagnostics.latency-measurement: on
diagnostics.count-fop-hits: on
Volume Name: vmstore
Type: Replicate
Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore
Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore
Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore
Options Reconfigured:
cluster.granular-entry-heal: enable
performance.strict-o-direct: on
network.ping-timeout: 20
storage.owner-gid: 36
storage.owner-uid: 36
server.event-threads: 4
client.event-threads: 4
cluster.choose-local: off
user.cifs: off
features.shard: on
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.server-quorum-type: server
cluster.quorum-type: auto
cluster.eager-lock: enable
network.remote-dio: off
performance.low-prio-threads: 32
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: on
diagnostics.latency-measurement: on
diagnostics.count-fop-hits: on
server.tcp-user-timeout: 20
server.keepalive-time: 10
server.keepalive-interval: 2
server.keepalive-count: 5
cluster.lookup-optimize: off
=========================================================================================================================
[root@ovirt1 ~]# gluster volume profile vmstore info
Brick: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore
---------------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 4021 2 64
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 134 405
No. of Writes: 2 32 119
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 2890 2617 3059
No. of Writes: 113881 93551 131776
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 1096597 164931 181569
No. of Writes: 41091001 20499776 3230472
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 212970 96574 244017
No. of Writes: 1304688 991154 3412866
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 0 0
No. of Writes: 15 18 65166
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6396 FORGET
0.00 0.00 us 0.00 us 0.00 us 189759 RELEASE
0.00 0.00 us 0.00 us 0.00 us 31292 RELEASEDIR
0.00 159.99 us 44.37 us 278.66 us 79 STAT
0.00 364.42 us 174.87 us 644.45 us 96 LINK
0.00 2714.10 us 203.53 us 47397.28 us 26 RMDIR
0.00 688.26 us 72.39 us 53857.63 us 186 SETXATTR
0.00 974.01 us 271.02 us 27298.52 us 176 RENAME
0.00 5254.64 us 159.48 us 84454.46 us 35 DISCARD
0.00 1133.84 us 86.47 us 87096.92 us 186 REMOVEXATTR
0.00 7907.47 us 221.82 us 182276.21 us 29 FALLOCATE
0.00 8304.51 us 311.89 us 42725.81 us 34 MKDIR
0.00 5758.19 us 296.39 us 21950.99 us 81 TRUNCATE
0.00 401.16 us 23.68 us 132668.93 us 3621 LK
0.00 545.48 us 14.22 us 1076517.76 us 6796 READDIR
0.00 2250.78 us 336.61 us 1071764.61 us 4257 CREATE
0.00 531.76 us 1.35 us 1723046.94 us 31292 OPENDIR
0.01 2058.67 us 196.74 us 1587752.27 us 21986 MKNOD
0.01 1241.28 us 22.61 us 1548642.70 us 52674 READDIRP
0.01 67508.32 us 100.01 us 3002069.81 us 1100 FTRUNCATE
0.01 412.74 us 10.94 us 5111334.35 us 209731 FLUSH
0.02 78668.59 us 183.06 us 3012629.87 us 1222 SETATTR
0.03 761.13 us 21.13 us 4611854.15 us 249094 STATFS
0.03 1132.33 us 45.51 us 3159531.87 us 185654 OPEN
0.03 2010.33 us 14.43 us 3158098.85 us 107824 GETXATTR
0.04 2765.29 us 61.38 us 3159208.13 us 100484 XATTROP
0.13 1367.04 us 51.46 us 8450420.87 us 595299 FSTAT
0.46 157440.90 us 47.62 us 13405712.93 us 18218 UNLINK
0.60 1274.67 us 19.60 us 11985005.14 us 2980806 LOOKUP
1.67 16645.99 us 12.80 us 6869488.64 us 631331 ENTRYLK
4.10 7504.86 us 39.52 us 11749528.82 us 3430506 READ
4.31 56942.36 us 16.57 us 23654127.65 us 475563 INODELK
6.09 660.89 us 11.02 us 20571410.32 us 57908331 FINODELK
20.59 1823.35 us 63.26 us 23639762.45 us 70934143 WRITE
25.70 2680.82 us 35.37 us 13636394.75 us 60223005 FXATTROP
36.14 3164.86 us 52.35 us 16805886.48 us 71739477 FSYNC
0.00 0.00 us 0.00 us 0.00 us 78458 UPCALL
Duration: 404667 seconds
Data Read: 59850935724 bytes
Data Written: 1070158035683 bytes
Interval 43 Stats:
Block Size: 1b+ 256b+ 512b+
No. of Reads: 0 14 3
No. of Writes: 12 4 252
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 0 0 8105
No. of Writes: 100 34 147355
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 1559 1630 1494
No. of Writes: 86013 17318 8687
Block Size: 65536b+ 131072b+
No. of Reads: 532 764
No. of Writes: 8617 36317
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 16 FORGET
0.00 0.00 us 0.00 us 0.00 us 665 RELEASE
0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR
0.00 181.75 us 181.75 us 181.75 us 1 STAT
0.00 914.55 us 858.55 us 970.56 us 2 TRUNCATE
0.00 792.11 us 681.24 us 1051.32 us 4 RENAME
0.00 374.44 us 31.86 us 792.10 us 20 READDIR
0.00 399.48 us 306.97 us 562.69 us 28 UNLINK
0.00 752.40 us 636.14 us 907.29 us 16 CREATE
0.00 196.62 us 33.27 us 2586.77 us 336 GETXATTR
0.00 1107.98 us 288.51 us 29692.53 us 80 MKNOD
0.00 1074.80 us 74.41 us 2121.03 us 126 READDIRP
0.00 3119.52 us 3.17 us 157121.44 us 52 OPENDIR
0.01 972.14 us 21.72 us 329342.21 us 635 FLUSH
0.03 2122.15 us 31.97 us 377538.16 us 732 STATFS
0.06 4478.89 us 66.09 us 521289.10 us 649 OPEN
0.10 10273.00 us 88.19 us 478524.77 us 472 XATTROP
0.16 6969.92 us 16.14 us 385156.98 us 1138 ENTRYLK
0.20 4635.40 us 64.29 us 1098319.92 us 2097 FSTAT
0.65 4098.07 us 46.85 us 919617.66 us 7797 LOOKUP
2.36 87465.33 us 34.75 us 1224052.07 us 1317 INODELK
4.37 13897.55 us 57.89 us 1984970.27 us 15336 READ
5.24 1265.46 us 16.22 us 1638678.32 us 202165 FINODELK
21.05 3370.28 us 144.44 us 2264130.49 us 304724 WRITE
31.00 6411.68 us 39.84 us 2190355.42 us 235969 FXATTROP
34.75 6107.95 us 60.26 us 2270252.44 us 277643 FSYNC
0.00 0.00 us 0.00 us 0.00 us 368 UPCALL
Duration: 1132 seconds
Data Read: 292367566 bytes
Data Written: 7495339692 bytes
Brick: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore
---------------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 4802 2 64
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 138 828
No. of Writes: 2 32 131
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 3595 4590 6161
No. of Writes: 128976 99680 132413
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 930031 128995 150468
No. of Writes: 48357551 23957030 3689474
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 154039 71949 790010
No. of Writes: 1438859 1052045 3715427
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 9 11 40269
No. of Writes: 5 20 50757
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 7629 FORGET
0.00 0.00 us 0.00 us 0.00 us 228577 RELEASE
0.00 0.00 us 0.00 us 0.00 us 41524 RELEASEDIR
0.00 374.22 us 219.35 us 574.87 us 96 LINK
0.00 1712.22 us 164.90 us 22337.79 us 41 DISCARD
0.00 2840.25 us 245.24 us 46839.63 us 26 RMDIR
0.00 3660.92 us 229.86 us 86252.56 us 35 FALLOCATE
0.00 876.48 us 297.53 us 16685.63 us 188 RENAME
0.00 6660.10 us 379.21 us 52962.79 us 34 MKDIR
0.00 957.48 us 80.36 us 218059.88 us 355 REMOVEXATTR
0.00 1234.85 us 76.50 us 322955.27 us 355 SETXATTR
0.00 2326.62 us 84.07 us 250041.25 us 196 STAT
0.00 9660.45 us 289.50 us 295106.01 us 87 TRUNCATE
0.00 486.10 us 24.12 us 869966.79 us 5260 LK
0.00 453.69 us 18.32 us 426653.18 us 8135 READDIR
0.00 1832.38 us 296.61 us 261299.81 us 5050 CREATE
0.00 386.77 us 1.31 us 1300836.12 us 41524 OPENDIR
0.00 712.06 us 29.85 us 1441115.02 us 34874 READDIRP
0.01 1469.51 us 231.67 us 440065.71 us 22845 MKNOD
0.01 44229.36 us 75.58 us 1803963.42 us 871 FTRUNCATE
0.01 84433.20 us 149.47 us 3614869.24 us 1001 SETATTR
0.02 418.11 us 13.39 us 3515000.81 us 243820 FLUSH
0.03 1108.47 us 14.77 us 1658743.41 us 130647 GETXATTR
0.03 562.86 us 26.74 us 5043949.33 us 297799 STATFS
0.03 845.50 us 48.16 us 1998938.03 us 223680 OPEN
0.05 2236.13 us 56.08 us 5295998.22 us 120682 XATTROP
0.16 890.82 us 43.51 us 3653292.88 us 1012585 FSTAT
0.34 2974.20 us 12.92 us 7782497.74 us 642555 ENTRYLK
0.47 135831.21 us 70.35 us 11033867.90 us 19800 UNLINK
0.57 954.05 us 21.50 us 4553393.38 us 3414605 LOOKUP
1.79 14461.54 us 13.45 us 32841452.98 us 702915 INODELK
5.14 8014.78 us 40.70 us 5439109.56 us 3644063 READ
5.37 443.04 us 11.53 us 32863652.53 us 68909131 FINODELK
22.31 1780.31 us 33.59 us 11318712.62 us 71235991 FXATTROP
22.83 1571.16 us 74.86 us 32615055.19 us 82622840 WRITE
40.84 2762.75 us 52.77 us 8859115.35 us 84039509 FSYNC
0.00 0.00 us 0.00 us 0.00 us 95492 UPCALL
Duration: 484169 seconds
Data Read: 167149718723 bytes
Data Written: 1177141649872 bytes
Interval 43 Stats:
Block Size: 1b+ 256b+ 512b+
No. of Reads: 0 6 4
No. of Writes: 12 4 252
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 0 0 5668
No. of Writes: 100 34 147357
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 1178 783 1215
No. of Writes: 86014 17318 8687
Block Size: 65536b+ 131072b+
No. of Reads: 264 4109
No. of Writes: 8617 36317
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 16 FORGET
0.00 0.00 us 0.00 us 0.00 us 665 RELEASE
0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR
0.00 866.18 us 849.57 us 882.78 us 2 TRUNCATE
0.00 1016.14 us 868.35 us 1182.48 us 4 RENAME
0.00 455.65 us 41.13 us 1679.37 us 20 READDIR
0.00 373.33 us 173.54 us 538.40 us 28 UNLINK
0.00 722.01 us 635.91 us 853.71 us 16 CREATE
0.00 335.52 us 38.15 us 1381.97 us 54 READDIRP
0.00 39218.92 us 214.34 us 78223.50 us 2 STAT
0.00 1730.77 us 2.59 us 84964.69 us 52 OPENDIR
0.00 1304.96 us 393.64 us 29724.09 us 80 MKNOD
0.01 422.75 us 22.85 us 134336.60 us 635 FLUSH
0.01 699.21 us 37.54 us 137813.64 us 732 STATFS
0.01 468.63 us 23.71 us 260961.83 us 1141 ENTRYLK
0.04 2041.83 us 69.80 us 163743.98 us 649 OPEN
0.08 1253.65 us 72.52 us 290508.15 us 2354 FSTAT
0.09 6913.19 us 69.01 us 212479.24 us 472 XATTROP
0.13 3558.26 us 32.70 us 195896.10 us 1317 INODELK
0.13 9793.37 us 36.11 us 212755.58 us 499 GETXATTR
0.34 1615.15 us 55.42 us 711310.28 us 7797 LOOKUP
4.92 13025.92 us 59.83 us 1061585.06 us 13884 READ
5.26 955.95 us 16.81 us 1069148.49 us 202174 FINODELK
23.28 2809.44 us 137.21 us 1134076.13 us 304713 WRITE
26.46 4124.05 us 43.30 us 1142167.27 us 235954 FXATTROP
39.23 5195.62 us 65.47 us 1250469.35 us 277632 FSYNC
0.00 0.00 us 0.00 us 0.00 us 365 UPCALL
Duration: 1132 seconds
Data Read: 668951066 bytes
Data Written: 7495356076 bytes