Hello All, We have 3 servers with a raid 50 array each, we are having extreme performance issues with our gluster, writes on gluster seem to take at least 3 times longer than on the raid directly. Can this be improved? I've read through several other performance issues threads but have been unable to make any improvements "gluster volume info" and "gluster volume profile vmstore info" is below ========================================================================================================================= -Inside Gluster - test took 35+ hours: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 35.7m 17 5824k 7 112m 13 182.7 6 Latency 5466ms 12754ms 3499ms 1589ms 1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,,,,,,,,,,,,,,,,,5466ms,12754ms,,3499ms,1589ms,,,,,, ========================================================================================================================= -Outside Gluster - test took 18 minutes: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 567m 78 149m 30 307m 37 83.0 57 Latency 205ms 4630ms 1450ms 679ms 1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,,,,,,,,,,,,,,,,,205ms,4630ms,,1450ms,679ms,,,,,, ========================================================================================================================= [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume info Volume Name: engine Type: Replicate Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 30 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on Volume Name: vmstore Type: Replicate Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 20 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on server.tcp-user-timeout: 20 server.keepalive-time: 10 server.keepalive-interval: 2 server.keepalive-count: 5 cluster.lookup-optimize: off ========================================================================================================================= [root@ovirt1 ~]# gluster volume profile vmstore info Brick: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore --------------------------------------------------------- Cumulative Stats: Block Size: 1b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 4021 2 64 Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 134 405 No. of Writes: 2 32 119 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 2890 2617 3059 No. of Writes: 113881 93551 131776 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 1096597 164931 181569 No. of Writes: 41091001 20499776 3230472 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 212970 96574 244017 No. of Writes: 1304688 991154 3412866 Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 0 0 0 No. of Writes: 15 18 65166 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 6396 FORGET 0.00 0.00 us 0.00 us 0.00 us 189759 RELEASE 0.00 0.00 us 0.00 us 0.00 us 31292 RELEASEDIR 0.00 159.99 us 44.37 us 278.66 us 79 STAT 0.00 364.42 us 174.87 us 644.45 us 96 LINK 0.00 2714.10 us 203.53 us 47397.28 us 26 RMDIR 0.00 688.26 us 72.39 us 53857.63 us 186 SETXATTR 0.00 974.01 us 271.02 us 27298.52 us 176 RENAME 0.00 5254.64 us 159.48 us 84454.46 us 35 DISCARD 0.00 1133.84 us 86.47 us 87096.92 us 186 REMOVEXATTR 0.00 7907.47 us 221.82 us 182276.21 us 29 FALLOCATE 0.00 8304.51 us 311.89 us 42725.81 us 34 MKDIR 0.00 5758.19 us 296.39 us 21950.99 us 81 TRUNCATE 0.00 401.16 us 23.68 us 132668.93 us 3621 LK 0.00 545.48 us 14.22 us 1076517.76 us 6796 READDIR 0.00 2250.78 us 336.61 us 1071764.61 us 4257 CREATE 0.00 531.76 us 1.35 us 1723046.94 us 31292 OPENDIR 0.01 2058.67 us 196.74 us 1587752.27 us 21986 MKNOD 0.01 1241.28 us 22.61 us 1548642.70 us 52674 READDIRP 0.01 67508.32 us 100.01 us 3002069.81 us 1100 FTRUNCATE 0.01 412.74 us 10.94 us 5111334.35 us 209731 FLUSH 0.02 78668.59 us 183.06 us 3012629.87 us 1222 SETATTR 0.03 761.13 us 21.13 us 4611854.15 us 249094 STATFS 0.03 1132.33 us 45.51 us 3159531.87 us 185654 OPEN 0.03 2010.33 us 14.43 us 3158098.85 us 107824 GETXATTR 0.04 2765.29 us 61.38 us 3159208.13 us 100484 XATTROP 0.13 1367.04 us 51.46 us 8450420.87 us 595299 FSTAT 0.46 157440.90 us 47.62 us 13405712.93 us 18218 UNLINK 0.60 1274.67 us 19.60 us 11985005.14 us 2980806 LOOKUP 1.67 16645.99 us 12.80 us 6869488.64 us 631331 ENTRYLK 4.10 7504.86 us 39.52 us 11749528.82 us 3430506 READ 4.31 56942.36 us 16.57 us 23654127.65 us 475563 INODELK 6.09 660.89 us 11.02 us 20571410.32 us 57908331 FINODELK 20.59 1823.35 us 63.26 us 23639762.45 us 70934143 WRITE 25.70 2680.82 us 35.37 us 13636394.75 us 60223005 FXATTROP 36.14 3164.86 us 52.35 us 16805886.48 us 71739477 FSYNC 0.00 0.00 us 0.00 us 0.00 us 78458 UPCALL Duration: 404667 seconds Data Read: 59850935724 bytes Data Written: 1070158035683 bytes Interval 43 Stats: Block Size: 1b+ 256b+ 512b+ No. of Reads: 0 14 3 No. of Writes: 12 4 252 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 8105 No. of Writes: 100 34 147355 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1559 1630 1494 No. of Writes: 86013 17318 8687 Block Size: 65536b+ 131072b+ No. of Reads: 532 764 No. of Writes: 8617 36317 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16 FORGET 0.00 0.00 us 0.00 us 0.00 us 665 RELEASE 0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR 0.00 181.75 us 181.75 us 181.75 us 1 STAT 0.00 914.55 us 858.55 us 970.56 us 2 TRUNCATE 0.00 792.11 us 681.24 us 1051.32 us 4 RENAME 0.00 374.44 us 31.86 us 792.10 us 20 READDIR 0.00 399.48 us 306.97 us 562.69 us 28 UNLINK 0.00 752.40 us 636.14 us 907.29 us 16 CREATE 0.00 196.62 us 33.27 us 2586.77 us 336 GETXATTR 0.00 1107.98 us 288.51 us 29692.53 us 80 MKNOD 0.00 1074.80 us 74.41 us 2121.03 us 126 READDIRP 0.00 3119.52 us 3.17 us 157121.44 us 52 OPENDIR 0.01 972.14 us 21.72 us 329342.21 us 635 FLUSH 0.03 2122.15 us 31.97 us 377538.16 us 732 STATFS 0.06 4478.89 us 66.09 us 521289.10 us 649 OPEN 0.10 10273.00 us 88.19 us 478524.77 us 472 XATTROP 0.16 6969.92 us 16.14 us 385156.98 us 1138 ENTRYLK 0.20 4635.40 us 64.29 us 1098319.92 us 2097 FSTAT 0.65 4098.07 us 46.85 us 919617.66 us 7797 LOOKUP 2.36 87465.33 us 34.75 us 1224052.07 us 1317 INODELK 4.37 13897.55 us 57.89 us 1984970.27 us 15336 READ 5.24 1265.46 us 16.22 us 1638678.32 us 202165 FINODELK 21.05 3370.28 us 144.44 us 2264130.49 us 304724 WRITE 31.00 6411.68 us 39.84 us 2190355.42 us 235969 FXATTROP 34.75 6107.95 us 60.26 us 2270252.44 us 277643 FSYNC 0.00 0.00 us 0.00 us 0.00 us 368 UPCALL Duration: 1132 seconds Data Read: 292367566 bytes Data Written: 7495339692 bytes Brick: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore --------------------------------------------------------- Cumulative Stats: Block Size: 1b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 4802 2 64 Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 138 828 No. of Writes: 2 32 131 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 3595 4590 6161 No. of Writes: 128976 99680 132413 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 930031 128995 150468 No. of Writes: 48357551 23957030 3689474 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 154039 71949 790010 No. of Writes: 1438859 1052045 3715427 Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 9 11 40269 No. of Writes: 5 20 50757 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 7629 FORGET 0.00 0.00 us 0.00 us 0.00 us 228577 RELEASE 0.00 0.00 us 0.00 us 0.00 us 41524 RELEASEDIR 0.00 374.22 us 219.35 us 574.87 us 96 LINK 0.00 1712.22 us 164.90 us 22337.79 us 41 DISCARD 0.00 2840.25 us 245.24 us 46839.63 us 26 RMDIR 0.00 3660.92 us 229.86 us 86252.56 us 35 FALLOCATE 0.00 876.48 us 297.53 us 16685.63 us 188 RENAME 0.00 6660.10 us 379.21 us 52962.79 us 34 MKDIR 0.00 957.48 us 80.36 us 218059.88 us 355 REMOVEXATTR 0.00 1234.85 us 76.50 us 322955.27 us 355 SETXATTR 0.00 2326.62 us 84.07 us 250041.25 us 196 STAT 0.00 9660.45 us 289.50 us 295106.01 us 87 TRUNCATE 0.00 486.10 us 24.12 us 869966.79 us 5260 LK 0.00 453.69 us 18.32 us 426653.18 us 8135 READDIR 0.00 1832.38 us 296.61 us 261299.81 us 5050 CREATE 0.00 386.77 us 1.31 us 1300836.12 us 41524 OPENDIR 0.00 712.06 us 29.85 us 1441115.02 us 34874 READDIRP 0.01 1469.51 us 231.67 us 440065.71 us 22845 MKNOD 0.01 44229.36 us 75.58 us 1803963.42 us 871 FTRUNCATE 0.01 84433.20 us 149.47 us 3614869.24 us 1001 SETATTR 0.02 418.11 us 13.39 us 3515000.81 us 243820 FLUSH 0.03 1108.47 us 14.77 us 1658743.41 us 130647 GETXATTR 0.03 562.86 us 26.74 us 5043949.33 us 297799 STATFS 0.03 845.50 us 48.16 us 1998938.03 us 223680 OPEN 0.05 2236.13 us 56.08 us 5295998.22 us 120682 XATTROP 0.16 890.82 us 43.51 us 3653292.88 us 1012585 FSTAT 0.34 2974.20 us 12.92 us 7782497.74 us 642555 ENTRYLK 0.47 135831.21 us 70.35 us 11033867.90 us 19800 UNLINK 0.57 954.05 us 21.50 us 4553393.38 us 3414605 LOOKUP 1.79 14461.54 us 13.45 us 32841452.98 us 702915 INODELK 5.14 8014.78 us 40.70 us 5439109.56 us 3644063 READ 5.37 443.04 us 11.53 us 32863652.53 us 68909131 FINODELK 22.31 1780.31 us 33.59 us 11318712.62 us 71235991 FXATTROP 22.83 1571.16 us 74.86 us 32615055.19 us 82622840 WRITE 40.84 2762.75 us 52.77 us 8859115.35 us 84039509 FSYNC 0.00 0.00 us 0.00 us 0.00 us 95492 UPCALL Duration: 484169 seconds Data Read: 167149718723 bytes Data Written: 1177141649872 bytes Interval 43 Stats: Block Size: 1b+ 256b+ 512b+ No. of Reads: 0 6 4 No. of Writes: 12 4 252 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 5668 No. of Writes: 100 34 147357 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1178 783 1215 No. of Writes: 86014 17318 8687 Block Size: 65536b+ 131072b+ No. of Reads: 264 4109 No. of Writes: 8617 36317 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16 FORGET 0.00 0.00 us 0.00 us 0.00 us 665 RELEASE 0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR 0.00 866.18 us 849.57 us 882.78 us 2 TRUNCATE 0.00 1016.14 us 868.35 us 1182.48 us 4 RENAME 0.00 455.65 us 41.13 us 1679.37 us 20 READDIR 0.00 373.33 us 173.54 us 538.40 us 28 UNLINK 0.00 722.01 us 635.91 us 853.71 us 16 CREATE 0.00 335.52 us 38.15 us 1381.97 us 54 READDIRP 0.00 39218.92 us 214.34 us 78223.50 us 2 STAT 0.00 1730.77 us 2.59 us 84964.69 us 52 OPENDIR 0.00 1304.96 us 393.64 us 29724.09 us 80 MKNOD 0.01 422.75 us 22.85 us 134336.60 us 635 FLUSH 0.01 699.21 us 37.54 us 137813.64 us 732 STATFS 0.01 468.63 us 23.71 us 260961.83 us 1141 ENTRYLK 0.04 2041.83 us 69.80 us 163743.98 us 649 OPEN 0.08 1253.65 us 72.52 us 290508.15 us 2354 FSTAT 0.09 6913.19 us 69.01 us 212479.24 us 472 XATTROP 0.13 3558.26 us 32.70 us 195896.10 us 1317 INODELK 0.13 9793.37 us 36.11 us 212755.58 us 499 GETXATTR 0.34 1615.15 us 55.42 us 711310.28 us 7797 LOOKUP 4.92 13025.92 us 59.83 us 1061585.06 us 13884 READ 5.26 955.95 us 16.81 us 1069148.49 us 202174 FINODELK 23.28 2809.44 us 137.21 us 1134076.13 us 304713 WRITE 26.46 4124.05 us 43.30 us 1142167.27 us 235954 FXATTROP 39.23 5195.62 us 65.47 us 1250469.35 us 277632 FSYNC 0.00 0.00 us 0.00 us 0.00 us 365 UPCALL Duration: 1132 seconds Data Read: 668951066 bytes Data Written: 7495356076 bytes