Re: Gluster Performance issues
Hello Sunil, [root@ovirt1 ~]# gluster --version glusterfs 8.6 same on all hosts On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya < sheggodu@redhat.com> wrote:
Hi,
Which version of gluster is in use?
Regards,
Sunil kumar Acharya
Red Hat
T: +91-8067935170 <http://redhatemailsignature-marketing.itos.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison <alex@discoverygarden.ca> wrote:
Hello All,
We have 3 servers with a raid 50 array each, we are having extreme performance issues with our gluster, writes on gluster seem to take at least 3 times longer than on the raid directly. Can this be improved? I've read through several other performance issues threads but have been unable to make any improvements
"gluster volume info" and "gluster volume profile vmstore info" is below
=========================================================================================================================
-Inside Gluster - test took 35+ hours: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 35.7m 17 5824k 7 112m 13 182.7 6 Latency 5466ms 12754ms 3499ms 1589ms
1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,,,,,,,,,,,,,,,,,5466ms,12754ms,,3499ms,1589ms,,,,,,
=========================================================================================================================
-Outside Gluster - test took 18 minutes: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 567m 78 149m 30 307m 37 83.0 57 Latency 205ms 4630ms 1450ms 679ms
1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,,,,,,,,,,,,,,,,,205ms,4630ms,,1450ms,679ms,,,,,,
=========================================================================================================================
[root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume info Volume Name: engine Type: Replicate Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 30 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on
Volume Name: vmstore Type: Replicate Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 20 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on server.tcp-user-timeout: 20 server.keepalive-time: 10 server.keepalive-interval: 2 server.keepalive-count: 5 cluster.lookup-optimize: off
=========================================================================================================================
[root@ovirt1 ~]# gluster volume profile vmstore info Brick: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore --------------------------------------------------------- Cumulative Stats: Block Size: 1b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 4021 2 64
Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 134 405 No. of Writes: 2 32 119
Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 2890 2617 3059 No. of Writes: 113881 93551 131776
Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 1096597 164931 181569 No. of Writes: 41091001 20499776 3230472
Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 212970 96574 244017 No. of Writes: 1304688 991154 3412866
Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 0 0 0 No. of Writes: 15 18 65166
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 6396 FORGET 0.00 0.00 us 0.00 us 0.00 us 189759 RELEASE 0.00 0.00 us 0.00 us 0.00 us 31292 RELEASEDIR 0.00 159.99 us 44.37 us 278.66 us 79 STAT 0.00 364.42 us 174.87 us 644.45 us 96 LINK 0.00 2714.10 us 203.53 us 47397.28 us 26 RMDIR 0.00 688.26 us 72.39 us 53857.63 us 186 SETXATTR 0.00 974.01 us 271.02 us 27298.52 us 176 RENAME 0.00 5254.64 us 159.48 us 84454.46 us 35 DISCARD 0.00 1133.84 us 86.47 us 87096.92 us 186 REMOVEXATTR 0.00 7907.47 us 221.82 us 182276.21 us 29 FALLOCATE 0.00 8304.51 us 311.89 us 42725.81 us 34 MKDIR 0.00 5758.19 us 296.39 us 21950.99 us 81 TRUNCATE 0.00 401.16 us 23.68 us 132668.93 us 3621 LK 0.00 545.48 us 14.22 us 1076517.76 us 6796 READDIR 0.00 2250.78 us 336.61 us 1071764.61 us 4257 CREATE 0.00 531.76 us 1.35 us 1723046.94 us 31292 OPENDIR 0.01 2058.67 us 196.74 us 1587752.27 us 21986 MKNOD 0.01 1241.28 us 22.61 us 1548642.70 us 52674 READDIRP 0.01 67508.32 us 100.01 us 3002069.81 us 1100 FTRUNCATE 0.01 412.74 us 10.94 us 5111334.35 us 209731 FLUSH 0.02 78668.59 us 183.06 us 3012629.87 us 1222 SETATTR 0.03 761.13 us 21.13 us 4611854.15 us 249094 STATFS 0.03 1132.33 us 45.51 us 3159531.87 us 185654 OPEN 0.03 2010.33 us 14.43 us 3158098.85 us 107824 GETXATTR 0.04 2765.29 us 61.38 us 3159208.13 us 100484 XATTROP 0.13 1367.04 us 51.46 us 8450420.87 us 595299 FSTAT 0.46 157440.90 us 47.62 us 13405712.93 us 18218 UNLINK 0.60 1274.67 us 19.60 us 11985005.14 us 2980806 LOOKUP 1.67 16645.99 us 12.80 us 6869488.64 us 631331 ENTRYLK 4.10 7504.86 us 39.52 us 11749528.82 us 3430506 READ 4.31 56942.36 us 16.57 us 23654127.65 us 475563 INODELK 6.09 660.89 us 11.02 us 20571410.32 us 57908331 FINODELK 20.59 1823.35 us 63.26 us 23639762.45 us 70934143 WRITE 25.70 2680.82 us 35.37 us 13636394.75 us 60223005 FXATTROP 36.14 3164.86 us 52.35 us 16805886.48 us 71739477 FSYNC 0.00 0.00 us 0.00 us 0.00 us 78458 UPCALL
Duration: 404667 seconds Data Read: 59850935724 bytes Data Written: 1070158035683 bytes
Interval 43 Stats: Block Size: 1b+ 256b+ 512b+ No. of Reads: 0 14 3 No. of Writes: 12 4 252
Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 8105 No. of Writes: 100 34 147355
Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1559 1630 1494 No. of Writes: 86013 17318 8687
Block Size: 65536b+ 131072b+ No. of Reads: 532 764 No. of Writes: 8617 36317 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16 FORGET 0.00 0.00 us 0.00 us 0.00 us 665 RELEASE 0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR 0.00 181.75 us 181.75 us 181.75 us 1 STAT 0.00 914.55 us 858.55 us 970.56 us 2 TRUNCATE 0.00 792.11 us 681.24 us 1051.32 us 4 RENAME 0.00 374.44 us 31.86 us 792.10 us 20 READDIR 0.00 399.48 us 306.97 us 562.69 us 28 UNLINK 0.00 752.40 us 636.14 us 907.29 us 16 CREATE 0.00 196.62 us 33.27 us 2586.77 us 336 GETXATTR 0.00 1107.98 us 288.51 us 29692.53 us 80 MKNOD 0.00 1074.80 us 74.41 us 2121.03 us 126 READDIRP 0.00 3119.52 us 3.17 us 157121.44 us 52 OPENDIR 0.01 972.14 us 21.72 us 329342.21 us 635 FLUSH 0.03 2122.15 us 31.97 us 377538.16 us 732 STATFS 0.06 4478.89 us 66.09 us 521289.10 us 649 OPEN 0.10 10273.00 us 88.19 us 478524.77 us 472 XATTROP 0.16 6969.92 us 16.14 us 385156.98 us 1138 ENTRYLK 0.20 4635.40 us 64.29 us 1098319.92 us 2097 FSTAT 0.65 4098.07 us 46.85 us 919617.66 us 7797 LOOKUP 2.36 87465.33 us 34.75 us 1224052.07 us 1317 INODELK 4.37 13897.55 us 57.89 us 1984970.27 us 15336 READ 5.24 1265.46 us 16.22 us 1638678.32 us 202165 FINODELK 21.05 3370.28 us 144.44 us 2264130.49 us 304724 WRITE 31.00 6411.68 us 39.84 us 2190355.42 us 235969 FXATTROP 34.75 6107.95 us 60.26 us 2270252.44 us 277643 FSYNC 0.00 0.00 us 0.00 us 0.00 us 368 UPCALL
Duration: 1132 seconds Data Read: 292367566 bytes Data Written: 7495339692 bytes
Brick: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore --------------------------------------------------------- Cumulative Stats: Block Size: 1b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 4802 2 64
Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 138 828 No. of Writes: 2 32 131
Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 3595 4590 6161 No. of Writes: 128976 99680 132413
Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 930031 128995 150468 No. of Writes: 48357551 23957030 3689474
Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 154039 71949 790010 No. of Writes: 1438859 1052045 3715427
Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 9 11 40269 No. of Writes: 5 20 50757
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 7629 FORGET 0.00 0.00 us 0.00 us 0.00 us 228577 RELEASE 0.00 0.00 us 0.00 us 0.00 us 41524 RELEASEDIR 0.00 374.22 us 219.35 us 574.87 us 96 LINK 0.00 1712.22 us 164.90 us 22337.79 us 41 DISCARD 0.00 2840.25 us 245.24 us 46839.63 us 26 RMDIR 0.00 3660.92 us 229.86 us 86252.56 us 35 FALLOCATE 0.00 876.48 us 297.53 us 16685.63 us 188 RENAME 0.00 6660.10 us 379.21 us 52962.79 us 34 MKDIR 0.00 957.48 us 80.36 us 218059.88 us 355 REMOVEXATTR 0.00 1234.85 us 76.50 us 322955.27 us 355 SETXATTR 0.00 2326.62 us 84.07 us 250041.25 us 196 STAT 0.00 9660.45 us 289.50 us 295106.01 us 87 TRUNCATE 0.00 486.10 us 24.12 us 869966.79 us 5260 LK 0.00 453.69 us 18.32 us 426653.18 us 8135 READDIR 0.00 1832.38 us 296.61 us 261299.81 us 5050 CREATE 0.00 386.77 us 1.31 us 1300836.12 us 41524 OPENDIR 0.00 712.06 us 29.85 us 1441115.02 us 34874 READDIRP 0.01 1469.51 us 231.67 us 440065.71 us 22845 MKNOD 0.01 44229.36 us 75.58 us 1803963.42 us 871 FTRUNCATE 0.01 84433.20 us 149.47 us 3614869.24 us 1001 SETATTR 0.02 418.11 us 13.39 us 3515000.81 us 243820 FLUSH 0.03 1108.47 us 14.77 us 1658743.41 us 130647 GETXATTR 0.03 562.86 us 26.74 us 5043949.33 us 297799 STATFS 0.03 845.50 us 48.16 us 1998938.03 us 223680 OPEN 0.05 2236.13 us 56.08 us 5295998.22 us 120682 XATTROP 0.16 890.82 us 43.51 us 3653292.88 us 1012585 FSTAT 0.34 2974.20 us 12.92 us 7782497.74 us 642555 ENTRYLK 0.47 135831.21 us 70.35 us 11033867.90 us 19800 UNLINK 0.57 954.05 us 21.50 us 4553393.38 us 3414605 LOOKUP 1.79 14461.54 us 13.45 us 32841452.98 us 702915 INODELK 5.14 8014.78 us 40.70 us 5439109.56 us 3644063 READ 5.37 443.04 us 11.53 us 32863652.53 us 68909131 FINODELK 22.31 1780.31 us 33.59 us 11318712.62 us 71235991 FXATTROP 22.83 1571.16 us 74.86 us 32615055.19 us 82622840 WRITE 40.84 2762.75 us 52.77 us 8859115.35 us 84039509 FSYNC 0.00 0.00 us 0.00 us 0.00 us 95492 UPCALL
Duration: 484169 seconds Data Read: 167149718723 bytes Data Written: 1177141649872 bytes
Interval 43 Stats: Block Size: 1b+ 256b+ 512b+ No. of Reads: 0 6 4 No. of Writes: 12 4 252
Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 5668 No. of Writes: 100 34 147357
Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1178 783 1215 No. of Writes: 86014 17318 8687
Block Size: 65536b+ 131072b+ No. of Reads: 264 4109 No. of Writes: 8617 36317 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16 FORGET 0.00 0.00 us 0.00 us 0.00 us 665 RELEASE 0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR 0.00 866.18 us 849.57 us 882.78 us 2 TRUNCATE 0.00 1016.14 us 868.35 us 1182.48 us 4 RENAME 0.00 455.65 us 41.13 us 1679.37 us 20 READDIR 0.00 373.33 us 173.54 us 538.40 us 28 UNLINK 0.00 722.01 us 635.91 us 853.71 us 16 CREATE 0.00 335.52 us 38.15 us 1381.97 us 54 READDIRP 0.00 39218.92 us 214.34 us 78223.50 us 2 STAT 0.00 1730.77 us 2.59 us 84964.69 us 52 OPENDIR 0.00 1304.96 us 393.64 us 29724.09 us 80 MKNOD 0.01 422.75 us 22.85 us 134336.60 us 635 FLUSH 0.01 699.21 us 37.54 us 137813.64 us 732 STATFS 0.01 468.63 us 23.71 us 260961.83 us 1141 ENTRYLK 0.04 2041.83 us 69.80 us 163743.98 us 649 OPEN 0.08 1253.65 us 72.52 us 290508.15 us 2354 FSTAT 0.09 6913.19 us 69.01 us 212479.24 us 472 XATTROP 0.13 3558.26 us 32.70 us 195896.10 us 1317 INODELK 0.13 9793.37 us 36.11 us 212755.58 us 499 GETXATTR 0.34 1615.15 us 55.42 us 711310.28 us 7797 LOOKUP 4.92 13025.92 us 59.83 us 1061585.06 us 13884 READ 5.26 955.95 us 16.81 us 1069148.49 us 202174 FINODELK 23.28 2809.44 us 137.21 us 1134076.13 us 304713 WRITE 26.46 4124.05 us 43.30 us 1142167.27 us 235954 FXATTROP 39.23 5195.62 us 65.47 us 1250469.35 us 277632 FSYNC 0.00 0.00 us 0.00 us 0.00 us 365 UPCALL
Duration: 1132 seconds Data Read: 668951066 bytes Data Written: 7495356076 bytes _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/G3G5HNX3P2Z5E5...
Hi, Another question which I don't see answered: What is the underlying connectivity between the Gluster hosts? -derek On Wed, February 23, 2022 8:39 am, Alex Morrison wrote:
Hello Sunil,
[root@ovirt1 ~]# gluster --version glusterfs 8.6
same on all hosts
On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya < sheggodu@redhat.com> wrote:
Hi,
Which version of gluster is in use?
Regards,
Sunil kumar Acharya
Red Hat
T: +91-8067935170 <http://redhatemailsignature-marketing.itos.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison <alex@discoverygarden.ca> wrote:
Hello All,
We have 3 servers with a raid 50 array each, we are having extreme performance issues with our gluster, writes on gluster seem to take at least 3 times longer than on the raid directly. Can this be improved? I've read through several other performance issues threads but have been unable to make any improvements
"gluster volume info" and "gluster volume profile vmstore info" is below
=========================================================================================================================
-Inside Gluster - test took 35+ hours: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 35.7m 17 5824k 7 112m 13 182.7 6 Latency 5466ms 12754ms 3499ms 1589ms
1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,,,,,,,,,,,,,,,,,5466ms,12754ms,,3499ms,1589ms,,,,,,
=========================================================================================================================
-Outside Gluster - test took 18 minutes: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 567m 78 149m 30 307m 37 83.0 57 Latency 205ms 4630ms 1450ms 679ms
1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,,,,,,,,,,,,,,,,,205ms,4630ms,,1450ms,679ms,,,,,,
=========================================================================================================================
[root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume info Volume Name: engine Type: Replicate Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 30 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on
Volume Name: vmstore Type: Replicate Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 20 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on server.tcp-user-timeout: 20 server.keepalive-time: 10 server.keepalive-interval: 2 server.keepalive-count: 5 cluster.lookup-optimize: off
=========================================================================================================================
[root@ovirt1 ~]# gluster volume profile vmstore info Brick: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore --------------------------------------------------------- Cumulative Stats: Block Size: 1b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 4021 2 64
Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 134 405 No. of Writes: 2 32 119
Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 2890 2617 3059 No. of Writes: 113881 93551 131776
Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 1096597 164931 181569 No. of Writes: 41091001 20499776 3230472
Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 212970 96574 244017 No. of Writes: 1304688 991154 3412866
Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 0 0 0 No. of Writes: 15 18 65166
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 6396 FORGET 0.00 0.00 us 0.00 us 0.00 us 189759 RELEASE 0.00 0.00 us 0.00 us 0.00 us 31292 RELEASEDIR 0.00 159.99 us 44.37 us 278.66 us 79 STAT 0.00 364.42 us 174.87 us 644.45 us 96 LINK 0.00 2714.10 us 203.53 us 47397.28 us 26 RMDIR 0.00 688.26 us 72.39 us 53857.63 us 186 SETXATTR 0.00 974.01 us 271.02 us 27298.52 us 176 RENAME 0.00 5254.64 us 159.48 us 84454.46 us 35 DISCARD 0.00 1133.84 us 86.47 us 87096.92 us 186 REMOVEXATTR 0.00 7907.47 us 221.82 us 182276.21 us 29 FALLOCATE 0.00 8304.51 us 311.89 us 42725.81 us 34 MKDIR 0.00 5758.19 us 296.39 us 21950.99 us 81 TRUNCATE 0.00 401.16 us 23.68 us 132668.93 us 3621 LK 0.00 545.48 us 14.22 us 1076517.76 us 6796 READDIR 0.00 2250.78 us 336.61 us 1071764.61 us 4257 CREATE 0.00 531.76 us 1.35 us 1723046.94 us 31292 OPENDIR 0.01 2058.67 us 196.74 us 1587752.27 us 21986 MKNOD 0.01 1241.28 us 22.61 us 1548642.70 us 52674 READDIRP 0.01 67508.32 us 100.01 us 3002069.81 us 1100 FTRUNCATE 0.01 412.74 us 10.94 us 5111334.35 us 209731 FLUSH 0.02 78668.59 us 183.06 us 3012629.87 us 1222 SETATTR 0.03 761.13 us 21.13 us 4611854.15 us 249094 STATFS 0.03 1132.33 us 45.51 us 3159531.87 us 185654 OPEN 0.03 2010.33 us 14.43 us 3158098.85 us 107824 GETXATTR 0.04 2765.29 us 61.38 us 3159208.13 us 100484 XATTROP 0.13 1367.04 us 51.46 us 8450420.87 us 595299 FSTAT 0.46 157440.90 us 47.62 us 13405712.93 us 18218 UNLINK 0.60 1274.67 us 19.60 us 11985005.14 us 2980806 LOOKUP 1.67 16645.99 us 12.80 us 6869488.64 us 631331 ENTRYLK 4.10 7504.86 us 39.52 us 11749528.82 us 3430506 READ 4.31 56942.36 us 16.57 us 23654127.65 us 475563 INODELK 6.09 660.89 us 11.02 us 20571410.32 us 57908331 FINODELK 20.59 1823.35 us 63.26 us 23639762.45 us 70934143 WRITE 25.70 2680.82 us 35.37 us 13636394.75 us 60223005 FXATTROP 36.14 3164.86 us 52.35 us 16805886.48 us 71739477 FSYNC 0.00 0.00 us 0.00 us 0.00 us 78458 UPCALL
Duration: 404667 seconds Data Read: 59850935724 bytes Data Written: 1070158035683 bytes
Interval 43 Stats: Block Size: 1b+ 256b+ 512b+ No. of Reads: 0 14 3 No. of Writes: 12 4 252
Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 8105 No. of Writes: 100 34 147355
Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1559 1630 1494 No. of Writes: 86013 17318 8687
Block Size: 65536b+ 131072b+ No. of Reads: 532 764 No. of Writes: 8617 36317 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16 FORGET 0.00 0.00 us 0.00 us 0.00 us 665 RELEASE 0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR 0.00 181.75 us 181.75 us 181.75 us 1 STAT 0.00 914.55 us 858.55 us 970.56 us 2 TRUNCATE 0.00 792.11 us 681.24 us 1051.32 us 4 RENAME 0.00 374.44 us 31.86 us 792.10 us 20 READDIR 0.00 399.48 us 306.97 us 562.69 us 28 UNLINK 0.00 752.40 us 636.14 us 907.29 us 16 CREATE 0.00 196.62 us 33.27 us 2586.77 us 336 GETXATTR 0.00 1107.98 us 288.51 us 29692.53 us 80 MKNOD 0.00 1074.80 us 74.41 us 2121.03 us 126 READDIRP 0.00 3119.52 us 3.17 us 157121.44 us 52 OPENDIR 0.01 972.14 us 21.72 us 329342.21 us 635 FLUSH 0.03 2122.15 us 31.97 us 377538.16 us 732 STATFS 0.06 4478.89 us 66.09 us 521289.10 us 649 OPEN 0.10 10273.00 us 88.19 us 478524.77 us 472 XATTROP 0.16 6969.92 us 16.14 us 385156.98 us 1138 ENTRYLK 0.20 4635.40 us 64.29 us 1098319.92 us 2097 FSTAT 0.65 4098.07 us 46.85 us 919617.66 us 7797 LOOKUP 2.36 87465.33 us 34.75 us 1224052.07 us 1317 INODELK 4.37 13897.55 us 57.89 us 1984970.27 us 15336 READ 5.24 1265.46 us 16.22 us 1638678.32 us 202165 FINODELK 21.05 3370.28 us 144.44 us 2264130.49 us 304724 WRITE 31.00 6411.68 us 39.84 us 2190355.42 us 235969 FXATTROP 34.75 6107.95 us 60.26 us 2270252.44 us 277643 FSYNC 0.00 0.00 us 0.00 us 0.00 us 368 UPCALL
Duration: 1132 seconds Data Read: 292367566 bytes Data Written: 7495339692 bytes
Brick: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore --------------------------------------------------------- Cumulative Stats: Block Size: 1b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 4802 2 64
Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 138 828 No. of Writes: 2 32 131
Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 3595 4590 6161 No. of Writes: 128976 99680 132413
Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 930031 128995 150468 No. of Writes: 48357551 23957030 3689474
Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 154039 71949 790010 No. of Writes: 1438859 1052045 3715427
Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 9 11 40269 No. of Writes: 5 20 50757
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 7629 FORGET 0.00 0.00 us 0.00 us 0.00 us 228577 RELEASE 0.00 0.00 us 0.00 us 0.00 us 41524 RELEASEDIR 0.00 374.22 us 219.35 us 574.87 us 96 LINK 0.00 1712.22 us 164.90 us 22337.79 us 41 DISCARD 0.00 2840.25 us 245.24 us 46839.63 us 26 RMDIR 0.00 3660.92 us 229.86 us 86252.56 us 35 FALLOCATE 0.00 876.48 us 297.53 us 16685.63 us 188 RENAME 0.00 6660.10 us 379.21 us 52962.79 us 34 MKDIR 0.00 957.48 us 80.36 us 218059.88 us 355 REMOVEXATTR 0.00 1234.85 us 76.50 us 322955.27 us 355 SETXATTR 0.00 2326.62 us 84.07 us 250041.25 us 196 STAT 0.00 9660.45 us 289.50 us 295106.01 us 87 TRUNCATE 0.00 486.10 us 24.12 us 869966.79 us 5260 LK 0.00 453.69 us 18.32 us 426653.18 us 8135 READDIR 0.00 1832.38 us 296.61 us 261299.81 us 5050 CREATE 0.00 386.77 us 1.31 us 1300836.12 us 41524 OPENDIR 0.00 712.06 us 29.85 us 1441115.02 us 34874 READDIRP 0.01 1469.51 us 231.67 us 440065.71 us 22845 MKNOD 0.01 44229.36 us 75.58 us 1803963.42 us 871 FTRUNCATE 0.01 84433.20 us 149.47 us 3614869.24 us 1001 SETATTR 0.02 418.11 us 13.39 us 3515000.81 us 243820 FLUSH 0.03 1108.47 us 14.77 us 1658743.41 us 130647 GETXATTR 0.03 562.86 us 26.74 us 5043949.33 us 297799 STATFS 0.03 845.50 us 48.16 us 1998938.03 us 223680 OPEN 0.05 2236.13 us 56.08 us 5295998.22 us 120682 XATTROP 0.16 890.82 us 43.51 us 3653292.88 us 1012585 FSTAT 0.34 2974.20 us 12.92 us 7782497.74 us 642555 ENTRYLK 0.47 135831.21 us 70.35 us 11033867.90 us 19800 UNLINK 0.57 954.05 us 21.50 us 4553393.38 us 3414605 LOOKUP 1.79 14461.54 us 13.45 us 32841452.98 us 702915 INODELK 5.14 8014.78 us 40.70 us 5439109.56 us 3644063 READ 5.37 443.04 us 11.53 us 32863652.53 us 68909131 FINODELK 22.31 1780.31 us 33.59 us 11318712.62 us 71235991 FXATTROP 22.83 1571.16 us 74.86 us 32615055.19 us 82622840 WRITE 40.84 2762.75 us 52.77 us 8859115.35 us 84039509 FSYNC 0.00 0.00 us 0.00 us 0.00 us 95492 UPCALL
Duration: 484169 seconds Data Read: 167149718723 bytes Data Written: 1177141649872 bytes
Interval 43 Stats: Block Size: 1b+ 256b+ 512b+ No. of Reads: 0 6 4 No. of Writes: 12 4 252
Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 5668 No. of Writes: 100 34 147357
Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1178 783 1215 No. of Writes: 86014 17318 8687
Block Size: 65536b+ 131072b+ No. of Reads: 264 4109 No. of Writes: 8617 36317 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16 FORGET 0.00 0.00 us 0.00 us 0.00 us 665 RELEASE 0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR 0.00 866.18 us 849.57 us 882.78 us 2 TRUNCATE 0.00 1016.14 us 868.35 us 1182.48 us 4 RENAME 0.00 455.65 us 41.13 us 1679.37 us 20 READDIR 0.00 373.33 us 173.54 us 538.40 us 28 UNLINK 0.00 722.01 us 635.91 us 853.71 us 16 CREATE 0.00 335.52 us 38.15 us 1381.97 us 54 READDIRP 0.00 39218.92 us 214.34 us 78223.50 us 2 STAT 0.00 1730.77 us 2.59 us 84964.69 us 52 OPENDIR 0.00 1304.96 us 393.64 us 29724.09 us 80 MKNOD 0.01 422.75 us 22.85 us 134336.60 us 635 FLUSH 0.01 699.21 us 37.54 us 137813.64 us 732 STATFS 0.01 468.63 us 23.71 us 260961.83 us 1141 ENTRYLK 0.04 2041.83 us 69.80 us 163743.98 us 649 OPEN 0.08 1253.65 us 72.52 us 290508.15 us 2354 FSTAT 0.09 6913.19 us 69.01 us 212479.24 us 472 XATTROP 0.13 3558.26 us 32.70 us 195896.10 us 1317 INODELK 0.13 9793.37 us 36.11 us 212755.58 us 499 GETXATTR 0.34 1615.15 us 55.42 us 711310.28 us 7797 LOOKUP 4.92 13025.92 us 59.83 us 1061585.06 us 13884 READ 5.26 955.95 us 16.81 us 1069148.49 us 202174 FINODELK 23.28 2809.44 us 137.21 us 1134076.13 us 304713 WRITE 26.46 4124.05 us 43.30 us 1142167.27 us 235954 FXATTROP 39.23 5195.62 us 65.47 us 1250469.35 us 277632 FSYNC 0.00 0.00 us 0.00 us 0.00 us 365 UPCALL
Duration: 1132 seconds Data Read: 668951066 bytes Data Written: 7495356076 bytes _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/G3G5HNX3P2Z5E5...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/NKMZYE6EC3IVLF...
-- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant
Hello Derek, We have a 10Gig connection dedicated to the storage network, nothing else is on that switch. On Wed, Feb 23, 2022 at 9:49 AM Derek Atkins <derek@ihtfp.com> wrote:
Hi,
Another question which I don't see answered: What is the underlying connectivity between the Gluster hosts?
-derek
On Wed, February 23, 2022 8:39 am, Alex Morrison wrote:
Hello Sunil,
[root@ovirt1 ~]# gluster --version glusterfs 8.6
same on all hosts
On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya < sheggodu@redhat.com> wrote:
Hi,
Which version of gluster is in use?
Regards,
Sunil kumar Acharya
Red Hat
T: +91-8067935170 <http://redhatemailsignature-marketing.itos.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison <alex@discoverygarden.ca> wrote:
Hello All,
We have 3 servers with a raid 50 array each, we are having extreme performance issues with our gluster, writes on gluster seem to take at least 3 times longer than on the raid directly. Can this be improved? I've read through several other performance issues threads but have been unable to make any improvements
"gluster volume info" and "gluster volume profile vmstore info" is below
=========================================================================================================================
-Inside Gluster - test took 35+ hours: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 35.7m 17 5824k 7 112m 13 182.7 6 Latency 5466ms 12754ms 3499ms 1589ms
1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,,,,,,,,,,,,,,,,,5466ms,12754ms,,3499ms,1589ms,,,,,,
=========================================================================================================================
-Outside Gluster - test took 18 minutes: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 567m 78 149m 30 307m 37 83.0 57 Latency 205ms 4630ms 1450ms 679ms
1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,,,,,,,,,,,,,,,,,205ms,4630ms,,1450ms,679ms,,,,,,
=========================================================================================================================
[root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume
info
Volume Name: engine Type: Replicate Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 30 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on
Volume Name: vmstore Type: Replicate Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 20 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on server.tcp-user-timeout: 20 server.keepalive-time: 10 server.keepalive-interval: 2 server.keepalive-count: 5 cluster.lookup-optimize: off
=========================================================================================================================
[root@ovirt1 ~]# gluster volume profile vmstore info Brick: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore --------------------------------------------------------- Cumulative Stats: Block Size: 1b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 4021 2 64
Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 134 405 No. of Writes: 2 32 119
Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 2890 2617 3059 No. of Writes: 113881 93551 131776
Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 1096597 164931 181569 No. of Writes: 41091001 20499776 3230472
Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 212970 96574 244017 No. of Writes: 1304688 991154 3412866
Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 0 0 0 No. of Writes: 15 18 65166
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 6396 FORGET 0.00 0.00 us 0.00 us 0.00 us 189759 RELEASE 0.00 0.00 us 0.00 us 0.00 us 31292 RELEASEDIR 0.00 159.99 us 44.37 us 278.66 us 79 STAT 0.00 364.42 us 174.87 us 644.45 us 96 LINK 0.00 2714.10 us 203.53 us 47397.28 us 26 RMDIR 0.00 688.26 us 72.39 us 53857.63 us 186 SETXATTR 0.00 974.01 us 271.02 us 27298.52 us 176 RENAME 0.00 5254.64 us 159.48 us 84454.46 us 35 DISCARD 0.00 1133.84 us 86.47 us 87096.92 us 186 REMOVEXATTR 0.00 7907.47 us 221.82 us 182276.21 us 29 FALLOCATE 0.00 8304.51 us 311.89 us 42725.81 us 34 MKDIR 0.00 5758.19 us 296.39 us 21950.99 us 81 TRUNCATE 0.00 401.16 us 23.68 us 132668.93 us 3621 LK 0.00 545.48 us 14.22 us 1076517.76 us 6796 READDIR 0.00 2250.78 us 336.61 us 1071764.61 us 4257 CREATE 0.00 531.76 us 1.35 us 1723046.94 us 31292 OPENDIR 0.01 2058.67 us 196.74 us 1587752.27 us 21986 MKNOD 0.01 1241.28 us 22.61 us 1548642.70 us 52674 READDIRP 0.01 67508.32 us 100.01 us 3002069.81 us 1100 FTRUNCATE 0.01 412.74 us 10.94 us 5111334.35 us 209731 FLUSH 0.02 78668.59 us 183.06 us 3012629.87 us 1222 SETATTR 0.03 761.13 us 21.13 us 4611854.15 us 249094 STATFS 0.03 1132.33 us 45.51 us 3159531.87 us 185654 OPEN 0.03 2010.33 us 14.43 us 3158098.85 us 107824 GETXATTR 0.04 2765.29 us 61.38 us 3159208.13 us 100484 XATTROP 0.13 1367.04 us 51.46 us 8450420.87 us 595299 FSTAT 0.46 157440.90 us 47.62 us 13405712.93 us 18218 UNLINK 0.60 1274.67 us 19.60 us 11985005.14 us 2980806 LOOKUP 1.67 16645.99 us 12.80 us 6869488.64 us 631331 ENTRYLK 4.10 7504.86 us 39.52 us 11749528.82 us 3430506 READ 4.31 56942.36 us 16.57 us 23654127.65 us 475563 INODELK 6.09 660.89 us 11.02 us 20571410.32 us 57908331 FINODELK 20.59 1823.35 us 63.26 us 23639762.45 us 70934143 WRITE 25.70 2680.82 us 35.37 us 13636394.75 us 60223005 FXATTROP 36.14 3164.86 us 52.35 us 16805886.48 us 71739477 FSYNC 0.00 0.00 us 0.00 us 0.00 us 78458 UPCALL
Duration: 404667 seconds Data Read: 59850935724 bytes Data Written: 1070158035683 bytes
Interval 43 Stats: Block Size: 1b+ 256b+ 512b+ No. of Reads: 0 14 3 No. of Writes: 12 4 252
Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 8105 No. of Writes: 100 34 147355
Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1559 1630 1494 No. of Writes: 86013 17318 8687
Block Size: 65536b+ 131072b+ No. of Reads: 532 764 No. of Writes: 8617 36317 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16 FORGET 0.00 0.00 us 0.00 us 0.00 us 665 RELEASE 0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR 0.00 181.75 us 181.75 us 181.75 us 1 STAT 0.00 914.55 us 858.55 us 970.56 us 2 TRUNCATE 0.00 792.11 us 681.24 us 1051.32 us 4 RENAME 0.00 374.44 us 31.86 us 792.10 us 20 READDIR 0.00 399.48 us 306.97 us 562.69 us 28 UNLINK 0.00 752.40 us 636.14 us 907.29 us 16 CREATE 0.00 196.62 us 33.27 us 2586.77 us 336 GETXATTR 0.00 1107.98 us 288.51 us 29692.53 us 80 MKNOD 0.00 1074.80 us 74.41 us 2121.03 us 126 READDIRP 0.00 3119.52 us 3.17 us 157121.44 us 52 OPENDIR 0.01 972.14 us 21.72 us 329342.21 us 635 FLUSH 0.03 2122.15 us 31.97 us 377538.16 us 732 STATFS 0.06 4478.89 us 66.09 us 521289.10 us 649 OPEN 0.10 10273.00 us 88.19 us 478524.77 us 472 XATTROP 0.16 6969.92 us 16.14 us 385156.98 us 1138 ENTRYLK 0.20 4635.40 us 64.29 us 1098319.92 us 2097 FSTAT 0.65 4098.07 us 46.85 us 919617.66 us 7797 LOOKUP 2.36 87465.33 us 34.75 us 1224052.07 us 1317 INODELK 4.37 13897.55 us 57.89 us 1984970.27 us 15336 READ 5.24 1265.46 us 16.22 us 1638678.32 us 202165 FINODELK 21.05 3370.28 us 144.44 us 2264130.49 us 304724 WRITE 31.00 6411.68 us 39.84 us 2190355.42 us 235969 FXATTROP 34.75 6107.95 us 60.26 us 2270252.44 us 277643 FSYNC 0.00 0.00 us 0.00 us 0.00 us 368 UPCALL
Duration: 1132 seconds Data Read: 292367566 bytes Data Written: 7495339692 bytes
Brick: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore --------------------------------------------------------- Cumulative Stats: Block Size: 1b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 4802 2 64
Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 138 828 No. of Writes: 2 32 131
Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 3595 4590 6161 No. of Writes: 128976 99680 132413
Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 930031 128995 150468 No. of Writes: 48357551 23957030 3689474
Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 154039 71949 790010 No. of Writes: 1438859 1052045 3715427
Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 9 11 40269 No. of Writes: 5 20 50757
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 7629 FORGET 0.00 0.00 us 0.00 us 0.00 us 228577 RELEASE 0.00 0.00 us 0.00 us 0.00 us 41524 RELEASEDIR 0.00 374.22 us 219.35 us 574.87 us 96 LINK 0.00 1712.22 us 164.90 us 22337.79 us 41 DISCARD 0.00 2840.25 us 245.24 us 46839.63 us 26 RMDIR 0.00 3660.92 us 229.86 us 86252.56 us 35 FALLOCATE 0.00 876.48 us 297.53 us 16685.63 us 188 RENAME 0.00 6660.10 us 379.21 us 52962.79 us 34 MKDIR 0.00 957.48 us 80.36 us 218059.88 us 355 REMOVEXATTR 0.00 1234.85 us 76.50 us 322955.27 us 355 SETXATTR 0.00 2326.62 us 84.07 us 250041.25 us 196 STAT 0.00 9660.45 us 289.50 us 295106.01 us 87 TRUNCATE 0.00 486.10 us 24.12 us 869966.79 us 5260 LK 0.00 453.69 us 18.32 us 426653.18 us 8135 READDIR 0.00 1832.38 us 296.61 us 261299.81 us 5050 CREATE 0.00 386.77 us 1.31 us 1300836.12 us 41524 OPENDIR 0.00 712.06 us 29.85 us 1441115.02 us 34874 READDIRP 0.01 1469.51 us 231.67 us 440065.71 us 22845 MKNOD 0.01 44229.36 us 75.58 us 1803963.42 us 871 FTRUNCATE 0.01 84433.20 us 149.47 us 3614869.24 us 1001 SETATTR 0.02 418.11 us 13.39 us 3515000.81 us 243820 FLUSH 0.03 1108.47 us 14.77 us 1658743.41 us 130647 GETXATTR 0.03 562.86 us 26.74 us 5043949.33 us 297799 STATFS 0.03 845.50 us 48.16 us 1998938.03 us 223680 OPEN 0.05 2236.13 us 56.08 us 5295998.22 us 120682 XATTROP 0.16 890.82 us 43.51 us 3653292.88 us 1012585 FSTAT 0.34 2974.20 us 12.92 us 7782497.74 us 642555 ENTRYLK 0.47 135831.21 us 70.35 us 11033867.90 us 19800 UNLINK 0.57 954.05 us 21.50 us 4553393.38 us 3414605 LOOKUP 1.79 14461.54 us 13.45 us 32841452.98 us 702915 INODELK 5.14 8014.78 us 40.70 us 5439109.56 us 3644063 READ 5.37 443.04 us 11.53 us 32863652.53 us 68909131 FINODELK 22.31 1780.31 us 33.59 us 11318712.62 us 71235991 FXATTROP 22.83 1571.16 us 74.86 us 32615055.19 us 82622840 WRITE 40.84 2762.75 us 52.77 us 8859115.35 us 84039509 FSYNC 0.00 0.00 us 0.00 us 0.00 us 95492 UPCALL
Duration: 484169 seconds Data Read: 167149718723 bytes Data Written: 1177141649872 bytes
Interval 43 Stats: Block Size: 1b+ 256b+ 512b+ No. of Reads: 0 6 4 No. of Writes: 12 4 252
Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 5668 No. of Writes: 100 34 147357
Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1178 783 1215 No. of Writes: 86014 17318 8687
Block Size: 65536b+ 131072b+ No. of Reads: 264 4109 No. of Writes: 8617 36317 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16 FORGET 0.00 0.00 us 0.00 us 0.00 us 665 RELEASE 0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR 0.00 866.18 us 849.57 us 882.78 us 2 TRUNCATE 0.00 1016.14 us 868.35 us 1182.48 us 4 RENAME 0.00 455.65 us 41.13 us 1679.37 us 20 READDIR 0.00 373.33 us 173.54 us 538.40 us 28 UNLINK 0.00 722.01 us 635.91 us 853.71 us 16 CREATE 0.00 335.52 us 38.15 us 1381.97 us 54 READDIRP 0.00 39218.92 us 214.34 us 78223.50 us 2 STAT 0.00 1730.77 us 2.59 us 84964.69 us 52 OPENDIR 0.00 1304.96 us 393.64 us 29724.09 us 80 MKNOD 0.01 422.75 us 22.85 us 134336.60 us 635 FLUSH 0.01 699.21 us 37.54 us 137813.64 us 732 STATFS 0.01 468.63 us 23.71 us 260961.83 us 1141 ENTRYLK 0.04 2041.83 us 69.80 us 163743.98 us 649 OPEN 0.08 1253.65 us 72.52 us 290508.15 us 2354 FSTAT 0.09 6913.19 us 69.01 us 212479.24 us 472 XATTROP 0.13 3558.26 us 32.70 us 195896.10 us 1317 INODELK 0.13 9793.37 us 36.11 us 212755.58 us 499 GETXATTR 0.34 1615.15 us 55.42 us 711310.28 us 7797 LOOKUP 4.92 13025.92 us 59.83 us 1061585.06 us 13884 READ 5.26 955.95 us 16.81 us 1069148.49 us 202174 FINODELK 23.28 2809.44 us 137.21 us 1134076.13 us 304713 WRITE 26.46 4124.05 us 43.30 us 1142167.27 us 235954 FXATTROP 39.23 5195.62 us 65.47 us 1250469.35 us 277632 FSYNC 0.00 0.00 us 0.00 us 0.00 us 365 UPCALL
Duration: 1132 seconds Data Read: 668951066 bytes Data Written: 7495356076 bytes _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/G3G5HNX3P2Z5E5...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/NKMZYE6EC3IVLF...
-- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant
Have you verified that you're actually getting 10Gbps between the hosts? -derek On Wed, February 23, 2022 9:02 am, Alex Morrison wrote:
Hello Derek,
We have a 10Gig connection dedicated to the storage network, nothing else is on that switch.
On Wed, Feb 23, 2022 at 9:49 AM Derek Atkins <derek@ihtfp.com> wrote:
Hi,
Another question which I don't see answered: What is the underlying connectivity between the Gluster hosts?
-derek
On Wed, February 23, 2022 8:39 am, Alex Morrison wrote:
Hello Sunil,
[root@ovirt1 ~]# gluster --version glusterfs 8.6
same on all hosts
On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya < sheggodu@redhat.com> wrote:
Hi,
Which version of gluster is in use?
Regards,
Sunil kumar Acharya
Red Hat
T: +91-8067935170 <http://redhatemailsignature-marketing.itos.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison <alex@discoverygarden.ca> wrote:
Hello All,
We have 3 servers with a raid 50 array each, we are having extreme performance issues with our gluster, writes on gluster seem to take at least 3 times longer than on the raid directly. Can this be improved? I've read through several other performance issues threads but have been unable to make any improvements
"gluster volume info" and "gluster volume profile vmstore info" is below
=========================================================================================================================
-Inside Gluster - test took 35+ hours: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 ------Sequential Output------ --Sequential
Input-
--Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 35.7m 17 5824k 7 112m 13 182.7 6 Latency 5466ms 12754ms 3499ms 1589ms
1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,,,,,,,,,,,,,,,,,5466ms,12754ms,,3499ms,1589ms,,,,,,
=========================================================================================================================
-Outside Gluster - test took 18 minutes: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 ------Sequential Output------ --Sequential
Input-
--Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 567m 78 149m 30 307m 37 83.0 57 Latency 205ms 4630ms 1450ms 679ms
1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,,,,,,,,,,,,,,,,,205ms,4630ms,,1450ms,679ms,,,,,,
=========================================================================================================================
[root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume
info
Volume Name: engine Type: Replicate Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 30 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on
Volume Name: vmstore Type: Replicate Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 20 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on server.tcp-user-timeout: 20 server.keepalive-time: 10 server.keepalive-interval: 2 server.keepalive-count: 5 cluster.lookup-optimize: off
=========================================================================================================================
[root@ovirt1 ~]# gluster volume profile vmstore info Brick: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore --------------------------------------------------------- Cumulative Stats: Block Size: 1b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 4021 2 64
Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 134 405 No. of Writes: 2 32 119
Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 2890 2617 3059 No. of Writes: 113881 93551 131776
Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 1096597 164931 181569 No. of Writes: 41091001 20499776 3230472
Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 212970 96574 244017 No. of Writes: 1304688 991154 3412866
Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 0 0 0 No. of Writes: 15 18 65166
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 6396 FORGET 0.00 0.00 us 0.00 us 0.00 us 189759 RELEASE 0.00 0.00 us 0.00 us 0.00 us 31292 RELEASEDIR 0.00 159.99 us 44.37 us 278.66 us 79 STAT 0.00 364.42 us 174.87 us 644.45 us 96 LINK 0.00 2714.10 us 203.53 us 47397.28 us 26 RMDIR 0.00 688.26 us 72.39 us 53857.63 us 186 SETXATTR 0.00 974.01 us 271.02 us 27298.52 us 176 RENAME 0.00 5254.64 us 159.48 us 84454.46 us 35 DISCARD 0.00 1133.84 us 86.47 us 87096.92 us 186 REMOVEXATTR 0.00 7907.47 us 221.82 us 182276.21 us 29 FALLOCATE 0.00 8304.51 us 311.89 us 42725.81 us 34 MKDIR 0.00 5758.19 us 296.39 us 21950.99 us 81 TRUNCATE 0.00 401.16 us 23.68 us 132668.93 us 3621 LK 0.00 545.48 us 14.22 us 1076517.76 us 6796 READDIR 0.00 2250.78 us 336.61 us 1071764.61 us 4257 CREATE 0.00 531.76 us 1.35 us 1723046.94 us 31292 OPENDIR 0.01 2058.67 us 196.74 us 1587752.27 us 21986 MKNOD 0.01 1241.28 us 22.61 us 1548642.70 us 52674 READDIRP 0.01 67508.32 us 100.01 us 3002069.81 us 1100 FTRUNCATE 0.01 412.74 us 10.94 us 5111334.35 us 209731 FLUSH 0.02 78668.59 us 183.06 us 3012629.87 us 1222 SETATTR 0.03 761.13 us 21.13 us 4611854.15 us 249094 STATFS 0.03 1132.33 us 45.51 us 3159531.87 us 185654 OPEN 0.03 2010.33 us 14.43 us 3158098.85 us 107824 GETXATTR 0.04 2765.29 us 61.38 us 3159208.13 us 100484 XATTROP 0.13 1367.04 us 51.46 us 8450420.87 us 595299 FSTAT 0.46 157440.90 us 47.62 us 13405712.93 us 18218 UNLINK 0.60 1274.67 us 19.60 us 11985005.14 us 2980806 LOOKUP 1.67 16645.99 us 12.80 us 6869488.64 us 631331 ENTRYLK 4.10 7504.86 us 39.52 us 11749528.82 us 3430506 READ 4.31 56942.36 us 16.57 us 23654127.65 us 475563 INODELK 6.09 660.89 us 11.02 us 20571410.32 us 57908331 FINODELK 20.59 1823.35 us 63.26 us 23639762.45 us 70934143 WRITE 25.70 2680.82 us 35.37 us 13636394.75 us 60223005 FXATTROP 36.14 3164.86 us 52.35 us 16805886.48 us 71739477 FSYNC 0.00 0.00 us 0.00 us 0.00 us 78458 UPCALL
Duration: 404667 seconds Data Read: 59850935724 bytes Data Written: 1070158035683 bytes
Interval 43 Stats: Block Size: 1b+ 256b+ 512b+ No. of Reads: 0 14 3 No. of Writes: 12 4 252
Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 8105 No. of Writes: 100 34 147355
Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1559 1630 1494 No. of Writes: 86013 17318 8687
Block Size: 65536b+ 131072b+ No. of Reads: 532 764 No. of Writes: 8617 36317 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16 FORGET 0.00 0.00 us 0.00 us 0.00 us 665 RELEASE 0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR 0.00 181.75 us 181.75 us 181.75 us 1 STAT 0.00 914.55 us 858.55 us 970.56 us 2 TRUNCATE 0.00 792.11 us 681.24 us 1051.32 us 4 RENAME 0.00 374.44 us 31.86 us 792.10 us 20 READDIR 0.00 399.48 us 306.97 us 562.69 us 28 UNLINK 0.00 752.40 us 636.14 us 907.29 us 16 CREATE 0.00 196.62 us 33.27 us 2586.77 us 336 GETXATTR 0.00 1107.98 us 288.51 us 29692.53 us 80 MKNOD 0.00 1074.80 us 74.41 us 2121.03 us 126 READDIRP 0.00 3119.52 us 3.17 us 157121.44 us 52 OPENDIR 0.01 972.14 us 21.72 us 329342.21 us 635 FLUSH 0.03 2122.15 us 31.97 us 377538.16 us 732 STATFS 0.06 4478.89 us 66.09 us 521289.10 us 649 OPEN 0.10 10273.00 us 88.19 us 478524.77 us 472 XATTROP 0.16 6969.92 us 16.14 us 385156.98 us 1138 ENTRYLK 0.20 4635.40 us 64.29 us 1098319.92 us 2097 FSTAT 0.65 4098.07 us 46.85 us 919617.66 us 7797 LOOKUP 2.36 87465.33 us 34.75 us 1224052.07 us 1317 INODELK 4.37 13897.55 us 57.89 us 1984970.27 us 15336 READ 5.24 1265.46 us 16.22 us 1638678.32 us 202165 FINODELK 21.05 3370.28 us 144.44 us 2264130.49 us 304724 WRITE 31.00 6411.68 us 39.84 us 2190355.42 us 235969 FXATTROP 34.75 6107.95 us 60.26 us 2270252.44 us 277643 FSYNC 0.00 0.00 us 0.00 us 0.00 us 368 UPCALL
Duration: 1132 seconds Data Read: 292367566 bytes Data Written: 7495339692 bytes
Brick: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore --------------------------------------------------------- Cumulative Stats: Block Size: 1b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 4802 2 64
Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 138 828 No. of Writes: 2 32 131
Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 3595 4590 6161 No. of Writes: 128976 99680 132413
Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 930031 128995 150468 No. of Writes: 48357551 23957030 3689474
Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 154039 71949 790010 No. of Writes: 1438859 1052045 3715427
Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 9 11 40269 No. of Writes: 5 20 50757
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 7629 FORGET 0.00 0.00 us 0.00 us 0.00 us 228577 RELEASE 0.00 0.00 us 0.00 us 0.00 us 41524 RELEASEDIR 0.00 374.22 us 219.35 us 574.87 us 96 LINK 0.00 1712.22 us 164.90 us 22337.79 us 41 DISCARD 0.00 2840.25 us 245.24 us 46839.63 us 26 RMDIR 0.00 3660.92 us 229.86 us 86252.56 us 35 FALLOCATE 0.00 876.48 us 297.53 us 16685.63 us 188 RENAME 0.00 6660.10 us 379.21 us 52962.79 us 34 MKDIR 0.00 957.48 us 80.36 us 218059.88 us 355 REMOVEXATTR 0.00 1234.85 us 76.50 us 322955.27 us 355 SETXATTR 0.00 2326.62 us 84.07 us 250041.25 us 196 STAT 0.00 9660.45 us 289.50 us 295106.01 us 87 TRUNCATE 0.00 486.10 us 24.12 us 869966.79 us 5260 LK 0.00 453.69 us 18.32 us 426653.18 us 8135 READDIR 0.00 1832.38 us 296.61 us 261299.81 us 5050 CREATE 0.00 386.77 us 1.31 us 1300836.12 us 41524 OPENDIR 0.00 712.06 us 29.85 us 1441115.02 us 34874 READDIRP 0.01 1469.51 us 231.67 us 440065.71 us 22845 MKNOD 0.01 44229.36 us 75.58 us 1803963.42 us 871 FTRUNCATE 0.01 84433.20 us 149.47 us 3614869.24 us 1001 SETATTR 0.02 418.11 us 13.39 us 3515000.81 us 243820 FLUSH 0.03 1108.47 us 14.77 us 1658743.41 us 130647 GETXATTR 0.03 562.86 us 26.74 us 5043949.33 us 297799 STATFS 0.03 845.50 us 48.16 us 1998938.03 us 223680 OPEN 0.05 2236.13 us 56.08 us 5295998.22 us 120682 XATTROP 0.16 890.82 us 43.51 us 3653292.88 us 1012585 FSTAT 0.34 2974.20 us 12.92 us 7782497.74 us 642555 ENTRYLK 0.47 135831.21 us 70.35 us 11033867.90 us 19800 UNLINK 0.57 954.05 us 21.50 us 4553393.38 us 3414605 LOOKUP 1.79 14461.54 us 13.45 us 32841452.98 us 702915 INODELK 5.14 8014.78 us 40.70 us 5439109.56 us 3644063 READ 5.37 443.04 us 11.53 us 32863652.53 us 68909131 FINODELK 22.31 1780.31 us 33.59 us 11318712.62 us 71235991 FXATTROP 22.83 1571.16 us 74.86 us 32615055.19 us 82622840 WRITE 40.84 2762.75 us 52.77 us 8859115.35 us 84039509 FSYNC 0.00 0.00 us 0.00 us 0.00 us 95492 UPCALL
Duration: 484169 seconds Data Read: 167149718723 bytes Data Written: 1177141649872 bytes
Interval 43 Stats: Block Size: 1b+ 256b+ 512b+ No. of Reads: 0 6 4 No. of Writes: 12 4 252
Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 5668 No. of Writes: 100 34 147357
Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1178 783 1215 No. of Writes: 86014 17318 8687
Block Size: 65536b+ 131072b+ No. of Reads: 264 4109 No. of Writes: 8617 36317 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16 FORGET 0.00 0.00 us 0.00 us 0.00 us 665 RELEASE 0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR 0.00 866.18 us 849.57 us 882.78 us 2 TRUNCATE 0.00 1016.14 us 868.35 us 1182.48 us 4 RENAME 0.00 455.65 us 41.13 us 1679.37 us 20 READDIR 0.00 373.33 us 173.54 us 538.40 us 28 UNLINK 0.00 722.01 us 635.91 us 853.71 us 16 CREATE 0.00 335.52 us 38.15 us 1381.97 us 54 READDIRP 0.00 39218.92 us 214.34 us 78223.50 us 2 STAT 0.00 1730.77 us 2.59 us 84964.69 us 52 OPENDIR 0.00 1304.96 us 393.64 us 29724.09 us 80 MKNOD 0.01 422.75 us 22.85 us 134336.60 us 635 FLUSH 0.01 699.21 us 37.54 us 137813.64 us 732 STATFS 0.01 468.63 us 23.71 us 260961.83 us 1141 ENTRYLK 0.04 2041.83 us 69.80 us 163743.98 us 649 OPEN 0.08 1253.65 us 72.52 us 290508.15 us 2354 FSTAT 0.09 6913.19 us 69.01 us 212479.24 us 472 XATTROP 0.13 3558.26 us 32.70 us 195896.10 us 1317 INODELK 0.13 9793.37 us 36.11 us 212755.58 us 499 GETXATTR 0.34 1615.15 us 55.42 us 711310.28 us 7797 LOOKUP 4.92 13025.92 us 59.83 us 1061585.06 us 13884 READ 5.26 955.95 us 16.81 us 1069148.49 us 202174 FINODELK 23.28 2809.44 us 137.21 us 1134076.13 us 304713 WRITE 26.46 4124.05 us 43.30 us 1142167.27 us 235954 FXATTROP 39.23 5195.62 us 65.47 us 1250469.35 us 277632 FSYNC 0.00 0.00 us 0.00 us 0.00 us 365 UPCALL
Duration: 1132 seconds Data Read: 668951066 bytes Data Written: 7495356076 bytes _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/G3G5HNX3P2Z5E5...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/NKMZYE6EC3IVLF...
-- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant
-- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant
Regards, Sunil On Wed, Feb 23, 2022 at 7:34 PM Derek Atkins <derek@ihtfp.com> wrote:
Have you verified that you're actually getting 10Gbps between the hosts?
-derek
On Wed, February 23, 2022 9:02 am, Alex Morrison wrote:
Hello Derek,
We have a 10Gig connection dedicated to the storage network, nothing else is on that switch.
On Wed, Feb 23, 2022 at 9:49 AM Derek Atkins <derek@ihtfp.com> wrote:
Hi,
Another question which I don't see answered: What is the underlying connectivity between the Gluster hosts?
-derek
On Wed, February 23, 2022 8:39 am, Alex Morrison wrote:
Hello Sunil,
[root@ovirt1 ~]# gluster --version glusterfs 8.6
same on all hosts
Latest Release-10.1( https://lists.gluster.org/pipermail/gluster-users/2022-February/039761.html) has some performance fixes which should help in this situation compared to the older gluster bits.
On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya < sheggodu@redhat.com> wrote:
Hi,
Which version of gluster is in use?
Regards,
Sunil kumar Acharya
Red Hat
T: +91-8067935170 <http://redhatemailsignature-marketing.itos.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison
<alex@discoverygarden.ca>
wrote:
Hello All,
We have 3 servers with a raid 50 array each, we are having extreme performance issues with our gluster, writes on gluster seem to take at least 3 times longer than on the raid directly. Can this be improved? I've read through several other performance issues threads but have been unable to make any improvements
"gluster volume info" and "gluster volume profile vmstore info" is below
=========================================================================================================================
-Inside Gluster - test took 35+ hours: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d .
-s
600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 35.7m 17 5824k 7 112m 13 182.7 6 Latency 5466ms 12754ms 3499ms 1589ms
1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,,,,,,,,,,,,,,,,,5466ms,12754ms,,3499ms,1589ms,,,,,,
=========================================================================================================================
-Outside Gluster - test took 18 minutes: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d .
-s
600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 567m 78 149m 30 307m 37 83.0 57 Latency 205ms 4630ms 1450ms 679ms
1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,,,,,,,,,,,,,,,,,205ms,4630ms,,1450ms,679ms,,,,,,
=========================================================================================================================
[root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume
info
Volume Name: engine Type: Replicate Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 30 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on
Volume Name: vmstore Type: Replicate Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 20 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on server.tcp-user-timeout: 20 server.keepalive-time: 10 server.keepalive-interval: 2 server.keepalive-count: 5 cluster.lookup-optimize: off
=========================================================================================================================
[root@ovirt1 ~]# gluster volume profile vmstore info Brick: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore --------------------------------------------------------- Cumulative Stats: Block Size: 1b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 4021 2 64
Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 134 405 No. of Writes: 2 32 119
Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 2890 2617 3059 No. of Writes: 113881 93551 131776
Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 1096597 164931 181569 No. of Writes: 41091001 20499776 3230472
Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 212970 96574 244017 No. of Writes: 1304688 991154 3412866
Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 0 0 0 No. of Writes: 15 18 65166
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 6396 FORGET 0.00 0.00 us 0.00 us 0.00 us 189759 RELEASE 0.00 0.00 us 0.00 us 0.00 us 31292 RELEASEDIR 0.00 159.99 us 44.37 us 278.66 us 79 STAT 0.00 364.42 us 174.87 us 644.45 us 96 LINK 0.00 2714.10 us 203.53 us 47397.28 us 26 RMDIR 0.00 688.26 us 72.39 us 53857.63 us 186 SETXATTR 0.00 974.01 us 271.02 us 27298.52 us 176 RENAME 0.00 5254.64 us 159.48 us 84454.46 us 35 DISCARD 0.00 1133.84 us 86.47 us 87096.92 us 186 REMOVEXATTR 0.00 7907.47 us 221.82 us 182276.21 us 29 FALLOCATE 0.00 8304.51 us 311.89 us 42725.81 us 34 MKDIR 0.00 5758.19 us 296.39 us 21950.99 us 81 TRUNCATE 0.00 401.16 us 23.68 us 132668.93 us 3621 LK 0.00 545.48 us 14.22 us 1076517.76 us 6796 READDIR 0.00 2250.78 us 336.61 us 1071764.61 us 4257 CREATE 0.00 531.76 us 1.35 us 1723046.94 us 31292 OPENDIR 0.01 2058.67 us 196.74 us 1587752.27 us 21986 MKNOD 0.01 1241.28 us 22.61 us 1548642.70 us 52674 READDIRP 0.01 67508.32 us 100.01 us 3002069.81 us 1100 FTRUNCATE 0.01 412.74 us 10.94 us 5111334.35 us 209731 FLUSH 0.02 78668.59 us 183.06 us 3012629.87 us 1222 SETATTR 0.03 761.13 us 21.13 us 4611854.15 us 249094 STATFS 0.03 1132.33 us 45.51 us 3159531.87 us 185654 OPEN 0.03 2010.33 us 14.43 us 3158098.85 us 107824 GETXATTR 0.04 2765.29 us 61.38 us 3159208.13 us 100484 XATTROP 0.13 1367.04 us 51.46 us 8450420.87 us 595299 FSTAT 0.46 157440.90 us 47.62 us 13405712.93 us 18218 UNLINK 0.60 1274.67 us 19.60 us 11985005.14 us 2980806 LOOKUP 1.67 16645.99 us 12.80 us 6869488.64 us 631331 ENTRYLK 4.10 7504.86 us 39.52 us 11749528.82 us 3430506 READ 4.31 56942.36 us 16.57 us 23654127.65 us 475563 INODELK 6.09 660.89 us 11.02 us 20571410.32 us 57908331 FINODELK 20.59 1823.35 us 63.26 us 23639762.45 us 70934143 WRITE 25.70 2680.82 us 35.37 us 13636394.75 us 60223005 FXATTROP 36.14 3164.86 us 52.35 us 16805886.48 us 71739477 FSYNC 0.00 0.00 us 0.00 us 0.00 us 78458 UPCALL
Duration: 404667 seconds Data Read: 59850935724 bytes Data Written: 1070158035683 bytes
Interval 43 Stats: Block Size: 1b+ 256b+ 512b+ No. of Reads: 0 14 3 No. of Writes: 12 4 252
Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 8105 No. of Writes: 100 34 147355
Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1559 1630 1494 No. of Writes: 86013 17318 8687
Block Size: 65536b+ 131072b+ No. of Reads: 532 764 No. of Writes: 8617 36317 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16 FORGET 0.00 0.00 us 0.00 us 0.00 us 665 RELEASE 0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR 0.00 181.75 us 181.75 us 181.75 us 1 STAT 0.00 914.55 us 858.55 us 970.56 us 2 TRUNCATE 0.00 792.11 us 681.24 us 1051.32 us 4 RENAME 0.00 374.44 us 31.86 us 792.10 us 20 READDIR 0.00 399.48 us 306.97 us 562.69 us 28 UNLINK 0.00 752.40 us 636.14 us 907.29 us 16 CREATE 0.00 196.62 us 33.27 us 2586.77 us 336 GETXATTR 0.00 1107.98 us 288.51 us 29692.53 us 80 MKNOD 0.00 1074.80 us 74.41 us 2121.03 us 126 READDIRP 0.00 3119.52 us 3.17 us 157121.44 us 52 OPENDIR 0.01 972.14 us 21.72 us 329342.21 us 635 FLUSH 0.03 2122.15 us 31.97 us 377538.16 us 732 STATFS 0.06 4478.89 us 66.09 us 521289.10 us 649 OPEN 0.10 10273.00 us 88.19 us 478524.77 us 472 XATTROP 0.16 6969.92 us 16.14 us 385156.98 us 1138 ENTRYLK 0.20 4635.40 us 64.29 us 1098319.92 us 2097 FSTAT 0.65 4098.07 us 46.85 us 919617.66 us 7797 LOOKUP 2.36 87465.33 us 34.75 us 1224052.07 us 1317 INODELK 4.37 13897.55 us 57.89 us 1984970.27 us 15336 READ 5.24 1265.46 us 16.22 us 1638678.32 us 202165 FINODELK 21.05 3370.28 us 144.44 us 2264130.49 us 304724 WRITE 31.00 6411.68 us 39.84 us 2190355.42 us 235969 FXATTROP 34.75 6107.95 us 60.26 us 2270252.44 us 277643 FSYNC 0.00 0.00 us 0.00 us 0.00 us 368 UPCALL
Duration: 1132 seconds Data Read: 292367566 bytes Data Written: 7495339692 bytes
Brick: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore --------------------------------------------------------- Cumulative Stats: Block Size: 1b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 4802 2 64
Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 138 828 No. of Writes: 2 32 131
Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 3595 4590 6161 No. of Writes: 128976 99680 132413
Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 930031 128995 150468 No. of Writes: 48357551 23957030 3689474
Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 154039 71949 790010 No. of Writes: 1438859 1052045 3715427
Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 9 11 40269 No. of Writes: 5 20 50757
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 7629 FORGET 0.00 0.00 us 0.00 us 0.00 us 228577 RELEASE 0.00 0.00 us 0.00 us 0.00 us 41524 RELEASEDIR 0.00 374.22 us 219.35 us 574.87 us 96 LINK 0.00 1712.22 us 164.90 us 22337.79 us 41 DISCARD 0.00 2840.25 us 245.24 us 46839.63 us 26 RMDIR 0.00 3660.92 us 229.86 us 86252.56 us 35 FALLOCATE 0.00 876.48 us 297.53 us 16685.63 us 188 RENAME 0.00 6660.10 us 379.21 us 52962.79 us 34 MKDIR 0.00 957.48 us 80.36 us 218059.88 us 355 REMOVEXATTR 0.00 1234.85 us 76.50 us 322955.27 us 355 SETXATTR 0.00 2326.62 us 84.07 us 250041.25 us 196 STAT 0.00 9660.45 us 289.50 us 295106.01 us 87 TRUNCATE 0.00 486.10 us 24.12 us 869966.79 us 5260 LK 0.00 453.69 us 18.32 us 426653.18 us 8135 READDIR 0.00 1832.38 us 296.61 us 261299.81 us 5050 CREATE 0.00 386.77 us 1.31 us 1300836.12 us 41524 OPENDIR 0.00 712.06 us 29.85 us 1441115.02 us 34874 READDIRP 0.01 1469.51 us 231.67 us 440065.71 us 22845 MKNOD 0.01 44229.36 us 75.58 us 1803963.42 us 871 FTRUNCATE 0.01 84433.20 us 149.47 us 3614869.24 us 1001 SETATTR 0.02 418.11 us 13.39 us 3515000.81 us 243820 FLUSH 0.03 1108.47 us 14.77 us 1658743.41 us 130647 GETXATTR 0.03 562.86 us 26.74 us 5043949.33 us 297799 STATFS 0.03 845.50 us 48.16 us 1998938.03 us 223680 OPEN 0.05 2236.13 us 56.08 us 5295998.22 us 120682 XATTROP 0.16 890.82 us 43.51 us 3653292.88 us 1012585 FSTAT 0.34 2974.20 us 12.92 us 7782497.74 us 642555 ENTRYLK 0.47 135831.21 us 70.35 us 11033867.90 us 19800 UNLINK 0.57 954.05 us 21.50 us 4553393.38 us 3414605 LOOKUP 1.79 14461.54 us 13.45 us 32841452.98 us 702915 INODELK 5.14 8014.78 us 40.70 us 5439109.56 us 3644063 READ 5.37 443.04 us 11.53 us 32863652.53 us 68909131 FINODELK 22.31 1780.31 us 33.59 us 11318712.62 us 71235991 FXATTROP 22.83 1571.16 us 74.86 us 32615055.19 us 82622840 WRITE 40.84 2762.75 us 52.77 us 8859115.35 us 84039509 FSYNC 0.00 0.00 us 0.00 us 0.00 us 95492 UPCALL
Duration: 484169 seconds Data Read: 167149718723 bytes Data Written: 1177141649872 bytes
Interval 43 Stats: Block Size: 1b+ 256b+ 512b+ No. of Reads: 0 6 4 No. of Writes: 12 4 252
Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 5668 No. of Writes: 100 34 147357
Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1178 783 1215 No. of Writes: 86014 17318 8687
Block Size: 65536b+ 131072b+ No. of Reads: 264 4109 No. of Writes: 8617 36317 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16 FORGET 0.00 0.00 us 0.00 us 0.00 us 665 RELEASE 0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR 0.00 866.18 us 849.57 us 882.78 us 2 TRUNCATE 0.00 1016.14 us 868.35 us 1182.48 us 4 RENAME 0.00 455.65 us 41.13 us 1679.37 us 20 READDIR 0.00 373.33 us 173.54 us 538.40 us 28 UNLINK 0.00 722.01 us 635.91 us 853.71 us 16 CREATE 0.00 335.52 us 38.15 us 1381.97 us 54 READDIRP 0.00 39218.92 us 214.34 us 78223.50 us 2 STAT 0.00 1730.77 us 2.59 us 84964.69 us 52 OPENDIR 0.00 1304.96 us 393.64 us 29724.09 us 80 MKNOD 0.01 422.75 us 22.85 us 134336.60 us 635 FLUSH 0.01 699.21 us 37.54 us 137813.64 us 732 STATFS 0.01 468.63 us 23.71 us 260961.83 us 1141 ENTRYLK 0.04 2041.83 us 69.80 us 163743.98 us 649 OPEN 0.08 1253.65 us 72.52 us 290508.15 us 2354 FSTAT 0.09 6913.19 us 69.01 us 212479.24 us 472 XATTROP 0.13 3558.26 us 32.70 us 195896.10 us 1317 INODELK 0.13 9793.37 us 36.11 us 212755.58 us 499 GETXATTR 0.34 1615.15 us 55.42 us 711310.28 us 7797 LOOKUP 4.92 13025.92 us 59.83 us 1061585.06 us 13884 READ 5.26 955.95 us 16.81 us 1069148.49 us 202174 FINODELK 23.28 2809.44 us 137.21 us 1134076.13 us 304713 WRITE 26.46 4124.05 us 43.30 us 1142167.27 us 235954 FXATTROP 39.23 5195.62 us 65.47 us 1250469.35 us 277632 FSYNC 0.00 0.00 us 0.00 us 0.00 us 365 UPCALL
Duration: 1132 seconds Data Read: 668951066 bytes Data Written: 7495356076 bytes _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/G3G5HNX3P2Z5E5...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/NKMZYE6EC3IVLF...
-- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant
-- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant
Hello All, I believe the network is performing as expected, I did an iperf test: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# iperf3 -c 10.10.1.2 Connecting to host 10.10.1.2, port 5201 [ 5] local 10.10.1.1 port 38422 connected to 10.10.1.2 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.08 GBytes 9.24 Gbits/sec 0 2.96 MBytes [ 5] 1.00-2.00 sec 1.03 GBytes 8.81 Gbits/sec 0 2.96 MBytes [ 5] 2.00-3.00 sec 1006 MBytes 8.44 Gbits/sec 101 1.45 MBytes [ 5] 3.00-4.00 sec 1.04 GBytes 8.92 Gbits/sec 5 901 KBytes [ 5] 4.00-5.00 sec 1.05 GBytes 9.01 Gbits/sec 0 957 KBytes [ 5] 5.00-6.00 sec 1.08 GBytes 9.23 Gbits/sec 0 990 KBytes [ 5] 6.00-7.00 sec 1008 MBytes 8.46 Gbits/sec 159 655 KBytes [ 5] 7.00-8.00 sec 1.06 GBytes 9.11 Gbits/sec 0 970 KBytes [ 5] 8.00-9.00 sec 1.03 GBytes 8.85 Gbits/sec 2 829 KBytes [ 5] 9.00-10.00 sec 1.04 GBytes 8.96 Gbits/sec 0 947 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.4 GBytes 8.90 Gbits/sec 267 sender [ 5] 0.00-10.04 sec 10.4 GBytes 8.87 Gbits/sec receiver iperf Done. On Wed, Feb 23, 2022 at 11:45 AM Sunil Kumar Heggodu Gopala Acharya < sheggodu@redhat.com> wrote:
Regards, Sunil
On Wed, Feb 23, 2022 at 7:34 PM Derek Atkins <derek@ihtfp.com> wrote:
Have you verified that you're actually getting 10Gbps between the hosts?
-derek
On Wed, February 23, 2022 9:02 am, Alex Morrison wrote:
Hello Derek,
We have a 10Gig connection dedicated to the storage network, nothing else is on that switch.
On Wed, Feb 23, 2022 at 9:49 AM Derek Atkins <derek@ihtfp.com> wrote:
Hi,
Another question which I don't see answered: What is the underlying connectivity between the Gluster hosts?
-derek
On Wed, February 23, 2022 8:39 am, Alex Morrison wrote:
Hello Sunil,
[root@ovirt1 ~]# gluster --version glusterfs 8.6
same on all hosts
Latest Release-10.1( https://lists.gluster.org/pipermail/gluster-users/2022-February/039761.html) has some performance fixes which should help in this situation compared to the older gluster bits.
On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya < sheggodu@redhat.com> wrote:
Hi,
Which version of gluster is in use?
Regards,
Sunil kumar Acharya
Red Hat
T: +91-8067935170 <http://redhatemailsignature-marketing.itos.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison
<alex@discoverygarden.ca>
wrote:
> Hello All, > > We have 3 servers with a raid 50 array each, we are having extreme > performance issues with our gluster, writes on gluster seem to take at > least 3 times longer than on the raid directly. Can this be improved? > I've > read through several other performance issues threads but have been > unable > to make any improvements > > "gluster volume info" and "gluster volume profile vmstore info" is > below > > >
=========================================================================================================================
> > -Inside Gluster - test took 35+ hours: > [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s > 600G -n 0 -m TEST -f -b -u root > Using uid:0, gid:0. > Writing intelligently...done > Rewriting...done > Reading intelligently...done > start 'em...done...done...done...done...done... > Version 1.98 ------Sequential Output------ --Sequential Input- > --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > /sec %CP > TEST 600G 35.7m 17 5824k 7 112m 13 > 182.7 6 > Latency 5466ms 12754ms 3499ms > 1589ms > > >
1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,,,,,,,,,,,,,,,,,5466ms,12754ms,,3499ms,1589ms,,,,,,
> > >
=========================================================================================================================
> > -Outside Gluster - test took 18 minutes: > [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s > 600G -n 0 -m TEST -f -b -u root > Using uid:0, gid:0. > Writing intelligently...done > Rewriting...done > Reading intelligently...done > start 'em...done...done...done...done...done... > Version 1.98 ------Sequential Output------ --Sequential Input- > --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > /sec %CP > TEST 600G 567m 78 149m 30 307m 37 > 83.0 57 > Latency 205ms 4630ms 1450ms > 679ms > > >
1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,,,,,,,,,,,,,,,,,205ms,4630ms,,1450ms,679ms,,,,,,
> > >
=========================================================================================================================
> > [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume info > Volume Name: engine > Type: Replicate > Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 3 = 3 > Transport-type: tcp > Bricks: > Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine > Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine > Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine > Options Reconfigured: > cluster.granular-entry-heal: enable > performance.strict-o-direct: on > network.ping-timeout: 30 > storage.owner-gid: 36 > storage.owner-uid: 36 > server.event-threads: 4 > client.event-threads: 4 > cluster.choose-local: off > user.cifs: off > features.shard: on > cluster.shd-wait-qlength: 10000 > cluster.shd-max-threads: 8 > cluster.locking-scheme: granular > cluster.data-self-heal-algorithm: full > cluster.server-quorum-type: server > cluster.quorum-type: auto > cluster.eager-lock: enable > network.remote-dio: off > performance.low-prio-threads: 32 > performance.io-cache: off > performance.read-ahead: off > performance.quick-read: off > transport.address-family: inet > storage.fips-mode-rchecksum: on > nfs.disable: on > performance.client-io-threads: on > diagnostics.latency-measurement: on > diagnostics.count-fop-hits: on > > Volume Name: vmstore > Type: Replicate > Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 3 = 3 > Transport-type: tcp > Bricks: > Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore > Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore > Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore > Options Reconfigured: > cluster.granular-entry-heal: enable > performance.strict-o-direct: on > network.ping-timeout: 20 > storage.owner-gid: 36 > storage.owner-uid: 36 > server.event-threads: 4 > client.event-threads: 4 > cluster.choose-local: off > user.cifs: off > features.shard: on > cluster.shd-wait-qlength: 10000 > cluster.shd-max-threads: 8 > cluster.locking-scheme: granular > cluster.data-self-heal-algorithm: full > cluster.server-quorum-type: server > cluster.quorum-type: auto > cluster.eager-lock: enable > network.remote-dio: off > performance.low-prio-threads: 32 > performance.io-cache: off > performance.read-ahead: off > performance.quick-read: off > transport.address-family: inet > storage.fips-mode-rchecksum: on > nfs.disable: on > performance.client-io-threads: on > diagnostics.latency-measurement: on > diagnostics.count-fop-hits: on > server.tcp-user-timeout: 20 > server.keepalive-time: 10 > server.keepalive-interval: 2 > server.keepalive-count: 5 > cluster.lookup-optimize: off > > >
=========================================================================================================================
> > [root@ovirt1 ~]# gluster volume profile vmstore info > Brick: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore > --------------------------------------------------------- > Cumulative Stats: > Block Size: 1b+ 4b+ > 8b+ > No. of Reads: 0 0 > 0 > No. of Writes: 4021 2 > 64 > > Block Size: 64b+ 128b+ > 256b+ > No. of Reads: 0 134 > 405 > No. of Writes: 2 32 > 119 > > Block Size: 512b+ 1024b+ > 2048b+ > No. of Reads: 2890 2617 > 3059 > No. of Writes: 113881 93551 > 131776 > > Block Size: 4096b+ 8192b+ > 16384b+ > No. of Reads: 1096597 164931 > 181569 > No. of Writes: 41091001 20499776 > 3230472 > > Block Size: 32768b+ 65536b+ > 131072b+ > No. of Reads: 212970 96574 > 244017 > No. of Writes: 1304688 991154 > 3412866 > > Block Size: 262144b+ 524288b+ > 1048576b+ > No. of Reads: 0 0 > 0 > No. of Writes: 15 18 > 65166 > > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 6396 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 189759 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 31292 > RELEASEDIR > 0.00 159.99 us 44.37 us 278.66 us 79 > STAT > 0.00 364.42 us 174.87 us 644.45 us 96 > LINK > 0.00 2714.10 us 203.53 us 47397.28 us 26 > RMDIR > 0.00 688.26 us 72.39 us 53857.63 us 186 > SETXATTR > 0.00 974.01 us 271.02 us 27298.52 us 176 > RENAME > 0.00 5254.64 us 159.48 us 84454.46 us 35 > DISCARD > 0.00 1133.84 us 86.47 us 87096.92 us 186 > REMOVEXATTR > 0.00 7907.47 us 221.82 us 182276.21 us 29 > FALLOCATE > 0.00 8304.51 us 311.89 us 42725.81 us 34 > MKDIR > 0.00 5758.19 us 296.39 us 21950.99 us 81 > TRUNCATE > 0.00 401.16 us 23.68 us 132668.93 us 3621 > LK > 0.00 545.48 us 14.22 us 1076517.76 us 6796 > READDIR > 0.00 2250.78 us 336.61 us 1071764.61 us 4257 > CREATE > 0.00 531.76 us 1.35 us 1723046.94 us 31292 > OPENDIR > 0.01 2058.67 us 196.74 us 1587752.27 us 21986 > MKNOD > 0.01 1241.28 us 22.61 us 1548642.70 us 52674 > READDIRP > 0.01 67508.32 us 100.01 us 3002069.81 us 1100 > FTRUNCATE > 0.01 412.74 us 10.94 us 5111334.35 us 209731 > FLUSH > 0.02 78668.59 us 183.06 us 3012629.87 us 1222 > SETATTR > 0.03 761.13 us 21.13 us 4611854.15 us 249094 > STATFS > 0.03 1132.33 us 45.51 us 3159531.87 us 185654 > OPEN > 0.03 2010.33 us 14.43 us 3158098.85 us 107824 > GETXATTR > 0.04 2765.29 us 61.38 us 3159208.13 us 100484 > XATTROP > 0.13 1367.04 us 51.46 us 8450420.87 us 595299 > FSTAT > 0.46 157440.90 us 47.62 us 13405712.93 us 18218 > UNLINK > 0.60 1274.67 us 19.60 us 11985005.14 us 2980806 > LOOKUP > 1.67 16645.99 us 12.80 us 6869488.64 us 631331 > ENTRYLK > 4.10 7504.86 us 39.52 us 11749528.82 us 3430506 > READ > 4.31 56942.36 us 16.57 us 23654127.65 us 475563 > INODELK > 6.09 660.89 us 11.02 us 20571410.32 us 57908331 > FINODELK > 20.59 1823.35 us 63.26 us 23639762.45 us 70934143 > WRITE > 25.70 2680.82 us 35.37 us 13636394.75 us 60223005 > FXATTROP > 36.14 3164.86 us 52.35 us 16805886.48 us 71739477 > FSYNC > 0.00 0.00 us 0.00 us 0.00 us 78458 > UPCALL > > Duration: 404667 seconds > Data Read: 59850935724 bytes > Data Written: 1070158035683 bytes > > Interval 43 Stats: > Block Size: 1b+ 256b+ > 512b+ > No. of Reads: 0 14 > 3 > No. of Writes: 12 4 > 252 > > Block Size: 1024b+ 2048b+ > 4096b+ > No. of Reads: 0 0 > 8105 > No. of Writes: 100 34 > 147355 > > Block Size: 8192b+ 16384b+ > 32768b+ > No. of Reads: 1559 1630 > 1494 > No. of Writes: 86013 17318 > 8687 > > Block Size: 65536b+ 131072b+ > No. of Reads: 532 764 > No. of Writes: 8617 36317 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 16 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 665 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 52 > RELEASEDIR > 0.00 181.75 us 181.75 us 181.75 us 1 > STAT > 0.00 914.55 us 858.55 us 970.56 us 2 > TRUNCATE > 0.00 792.11 us 681.24 us 1051.32 us 4 > RENAME > 0.00 374.44 us 31.86 us 792.10 us 20 > READDIR > 0.00 399.48 us 306.97 us 562.69 us 28 > UNLINK > 0.00 752.40 us 636.14 us 907.29 us 16 > CREATE > 0.00 196.62 us 33.27 us 2586.77 us 336 > GETXATTR > 0.00 1107.98 us 288.51 us 29692.53 us 80 > MKNOD > 0.00 1074.80 us 74.41 us 2121.03 us 126 > READDIRP > 0.00 3119.52 us 3.17 us 157121.44 us 52 > OPENDIR > 0.01 972.14 us 21.72 us 329342.21 us 635 > FLUSH > 0.03 2122.15 us 31.97 us 377538.16 us 732 > STATFS > 0.06 4478.89 us 66.09 us 521289.10 us 649 > OPEN > 0.10 10273.00 us 88.19 us 478524.77 us 472 > XATTROP > 0.16 6969.92 us 16.14 us 385156.98 us 1138 > ENTRYLK > 0.20 4635.40 us 64.29 us 1098319.92 us 2097 > FSTAT > 0.65 4098.07 us 46.85 us 919617.66 us 7797 > LOOKUP > 2.36 87465.33 us 34.75 us 1224052.07 us 1317 > INODELK > 4.37 13897.55 us 57.89 us 1984970.27 us 15336 > READ > 5.24 1265.46 us 16.22 us 1638678.32 us 202165 > FINODELK > 21.05 3370.28 us 144.44 us 2264130.49 us 304724 > WRITE > 31.00 6411.68 us 39.84 us 2190355.42 us 235969 > FXATTROP > 34.75 6107.95 us 60.26 us 2270252.44 us 277643 > FSYNC > 0.00 0.00 us 0.00 us 0.00 us 368 > UPCALL > > Duration: 1132 seconds > Data Read: 292367566 bytes > Data Written: 7495339692 bytes > > Brick: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore > --------------------------------------------------------- > Cumulative Stats: > Block Size: 1b+ 4b+ > 8b+ > No. of Reads: 0 0 > 0 > No. of Writes: 4802 2 > 64 > > Block Size: 64b+ 128b+ > 256b+ > No. of Reads: 0 138 > 828 > No. of Writes: 2 32 > 131 > > Block Size: 512b+ 1024b+ > 2048b+ > No. of Reads: 3595 4590 > 6161 > No. of Writes: 128976 99680 > 132413 > > Block Size: 4096b+ 8192b+ > 16384b+ > No. of Reads: 930031 128995 > 150468 > No. of Writes: 48357551 23957030 > 3689474 > > Block Size: 32768b+ 65536b+ > 131072b+ > No. of Reads: 154039 71949 > 790010 > No. of Writes: 1438859 1052045 > 3715427 > > Block Size: 262144b+ 524288b+ > 1048576b+ > No. of Reads: 9 11 > 40269 > No. of Writes: 5 20 > 50757 > > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 7629 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 228577 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 41524 > RELEASEDIR > 0.00 374.22 us 219.35 us 574.87 us 96 > LINK > 0.00 1712.22 us 164.90 us 22337.79 us 41 > DISCARD > 0.00 2840.25 us 245.24 us 46839.63 us 26 > RMDIR > 0.00 3660.92 us 229.86 us 86252.56 us 35 > FALLOCATE > 0.00 876.48 us 297.53 us 16685.63 us 188 > RENAME > 0.00 6660.10 us 379.21 us 52962.79 us 34 > MKDIR > 0.00 957.48 us 80.36 us 218059.88 us 355 > REMOVEXATTR > 0.00 1234.85 us 76.50 us 322955.27 us 355 > SETXATTR > 0.00 2326.62 us 84.07 us 250041.25 us 196 > STAT > 0.00 9660.45 us 289.50 us 295106.01 us 87 > TRUNCATE > 0.00 486.10 us 24.12 us 869966.79 us 5260 > LK > 0.00 453.69 us 18.32 us 426653.18 us 8135 > READDIR > 0.00 1832.38 us 296.61 us 261299.81 us 5050 > CREATE > 0.00 386.77 us 1.31 us 1300836.12 us 41524 > OPENDIR > 0.00 712.06 us 29.85 us 1441115.02 us 34874 > READDIRP > 0.01 1469.51 us 231.67 us 440065.71 us 22845 > MKNOD > 0.01 44229.36 us 75.58 us 1803963.42 us 871 > FTRUNCATE > 0.01 84433.20 us 149.47 us 3614869.24 us 1001 > SETATTR > 0.02 418.11 us 13.39 us 3515000.81 us 243820 > FLUSH > 0.03 1108.47 us 14.77 us 1658743.41 us 130647 > GETXATTR > 0.03 562.86 us 26.74 us 5043949.33 us 297799 > STATFS > 0.03 845.50 us 48.16 us 1998938.03 us 223680 > OPEN > 0.05 2236.13 us 56.08 us 5295998.22 us 120682 > XATTROP > 0.16 890.82 us 43.51 us 3653292.88 us 1012585 > FSTAT > 0.34 2974.20 us 12.92 us 7782497.74 us 642555 > ENTRYLK > 0.47 135831.21 us 70.35 us 11033867.90 us 19800 > UNLINK > 0.57 954.05 us 21.50 us 4553393.38 us 3414605 > LOOKUP > 1.79 14461.54 us 13.45 us 32841452.98 us 702915 > INODELK > 5.14 8014.78 us 40.70 us 5439109.56 us 3644063 > READ > 5.37 443.04 us 11.53 us 32863652.53 us 68909131 > FINODELK > 22.31 1780.31 us 33.59 us 11318712.62 us 71235991 > FXATTROP > 22.83 1571.16 us 74.86 us 32615055.19 us 82622840 > WRITE > 40.84 2762.75 us 52.77 us 8859115.35 us 84039509 > FSYNC > 0.00 0.00 us 0.00 us 0.00 us 95492 > UPCALL > > Duration: 484169 seconds > Data Read: 167149718723 bytes > Data Written: 1177141649872 bytes > > Interval 43 Stats: > Block Size: 1b+ 256b+ > 512b+ > No. of Reads: 0 6 > 4 > No. of Writes: 12 4 > 252 > > Block Size: 1024b+ 2048b+ > 4096b+ > No. of Reads: 0 0 > 5668 > No. of Writes: 100 34 > 147357 > > Block Size: 8192b+ 16384b+ > 32768b+ > No. of Reads: 1178 783 > 1215 > No. of Writes: 86014 17318 > 8687 > > Block Size: 65536b+ 131072b+ > No. of Reads: 264 4109 > No. of Writes: 8617 36317 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 16 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 665 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 52 > RELEASEDIR > 0.00 866.18 us 849.57 us 882.78 us 2 > TRUNCATE > 0.00 1016.14 us 868.35 us 1182.48 us 4 > RENAME > 0.00 455.65 us 41.13 us 1679.37 us 20 > READDIR > 0.00 373.33 us 173.54 us 538.40 us 28 > UNLINK > 0.00 722.01 us 635.91 us 853.71 us 16 > CREATE > 0.00 335.52 us 38.15 us 1381.97 us 54 > READDIRP > 0.00 39218.92 us 214.34 us 78223.50 us 2 > STAT > 0.00 1730.77 us 2.59 us 84964.69 us 52 > OPENDIR > 0.00 1304.96 us 393.64 us 29724.09 us 80 > MKNOD > 0.01 422.75 us 22.85 us 134336.60 us 635 > FLUSH > 0.01 699.21 us 37.54 us 137813.64 us 732 > STATFS > 0.01 468.63 us 23.71 us 260961.83 us 1141 > ENTRYLK > 0.04 2041.83 us 69.80 us 163743.98 us 649 > OPEN > 0.08 1253.65 us 72.52 us 290508.15 us 2354 > FSTAT > 0.09 6913.19 us 69.01 us 212479.24 us 472 > XATTROP > 0.13 3558.26 us 32.70 us 195896.10 us 1317 > INODELK > 0.13 9793.37 us 36.11 us 212755.58 us 499 > GETXATTR > 0.34 1615.15 us 55.42 us 711310.28 us 7797 > LOOKUP > 4.92 13025.92 us 59.83 us 1061585.06 us 13884 > READ > 5.26 955.95 us 16.81 us 1069148.49 us 202174 > FINODELK > 23.28 2809.44 us 137.21 us 1134076.13 us 304713 > WRITE > 26.46 4124.05 us 43.30 us 1142167.27 us 235954 > FXATTROP > 39.23 5195.62 us 65.47 us 1250469.35 us 277632 > FSYNC > 0.00 0.00 us 0.00 us 0.00 us 365 > UPCALL > > Duration: 1132 seconds > Data Read: 668951066 bytes > Data Written: 7495356076 bytes > _______________________________________________ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: >
https://lists.ovirt.org/archives/list/users@ovirt.org/message/G3G5HNX3P2Z5E5...
>
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/NKMZYE6EC3IVLF...
-- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant
-- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant
You can try to play a little bit with the I/O threads (but don't jump too fast). What is your I/O scheduler and mount options.You can reduce I/O lookups if you specify the 'noatime' and the selinux context on the mount options. A real killer of performance is the lattency. What is the lattency between all nodes ? Best Regards,Strahil Nikolov On Wed, Feb 23, 2022 at 20:26, Alex Morrison<alex@discoverygarden.ca> wrote: Hello All, I believe the network is performing as expected, I did an iperf test: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# iperf3 -c 10.10.1.2 Connecting to host 10.10.1.2, port 5201 [ 5] local 10.10.1.1 port 38422 connected to 10.10.1.2 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.08 GBytes 9.24 Gbits/sec 0 2.96 MBytes [ 5] 1.00-2.00 sec 1.03 GBytes 8.81 Gbits/sec 0 2.96 MBytes [ 5] 2.00-3.00 sec 1006 MBytes 8.44 Gbits/sec 101 1.45 MBytes [ 5] 3.00-4.00 sec 1.04 GBytes 8.92 Gbits/sec 5 901 KBytes [ 5] 4.00-5.00 sec 1.05 GBytes 9.01 Gbits/sec 0 957 KBytes [ 5] 5.00-6.00 sec 1.08 GBytes 9.23 Gbits/sec 0 990 KBytes [ 5] 6.00-7.00 sec 1008 MBytes 8.46 Gbits/sec 159 655 KBytes [ 5] 7.00-8.00 sec 1.06 GBytes 9.11 Gbits/sec 0 970 KBytes [ 5] 8.00-9.00 sec 1.03 GBytes 8.85 Gbits/sec 2 829 KBytes [ 5] 9.00-10.00 sec 1.04 GBytes 8.96 Gbits/sec 0 947 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.4 GBytes 8.90 Gbits/sec 267 sender [ 5] 0.00-10.04 sec 10.4 GBytes 8.87 Gbits/sec receiver iperf Done. On Wed, Feb 23, 2022 at 11:45 AM Sunil Kumar Heggodu Gopala Acharya <sheggodu@redhat.com> wrote: Regards, Sunil On Wed, Feb 23, 2022 at 7:34 PM Derek Atkins <derek@ihtfp.com> wrote: Have you verified that you're actually getting 10Gbps between the hosts? -derek On Wed, February 23, 2022 9:02 am, Alex Morrison wrote:
Hello Derek,
We have a 10Gig connection dedicated to the storage network, nothing else is on that switch.
On Wed, Feb 23, 2022 at 9:49 AM Derek Atkins <derek@ihtfp.com> wrote:
Hi,
Another question which I don't see answered: What is the underlying connectivity between the Gluster hosts?
-derek
On Wed, February 23, 2022 8:39 am, Alex Morrison wrote:
Hello Sunil,
[root@ovirt1 ~]# gluster --version glusterfs 8.6
same on all hosts
Latest Release-10.1(https://lists.gluster.org/pipermail/gluster-users/2022-February/039761.html) has some performance fixes which should help in this situation compared to the older gluster bits.
On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya < sheggodu@redhat.com> wrote:
Hi,
Which version of gluster is in use?
Regards,
Sunil kumar Acharya
Red Hat
T: +91-8067935170 <http://redhatemailsignature-marketing.itos.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison
<alex@discoverygarden.ca>
wrote:
Hello All,
We have 3 servers with a raid 50 array each, we are having extreme performance issues with our gluster, writes on gluster seem to take at least 3 times longer than on the raid directly. Can this be improved? I've read through several other performance issues threads but have been unable to make any improvements
"gluster volume info" and "gluster volume profile vmstore info" is below
=========================================================================================================================
-Inside Gluster - test took 35+ hours: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 ------Sequential Output------ --Sequential
Input-
--Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 35.7m 17 5824k 7 112m 13 182.7 6 Latency 5466ms 12754ms 3499ms 1589ms
1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,,,,,,,,,,,,,,,,,5466ms,12754ms,,3499ms,1589ms,,,,,,
=========================================================================================================================
-Outside Gluster - test took 18 minutes: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 ------Sequential Output------ --Sequential
Input-
--Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 567m 78 149m 30 307m 37 83.0 57 Latency 205ms 4630ms 1450ms 679ms
1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,,,,,,,,,,,,,,,,,205ms,4630ms,,1450ms,679ms,,,,,,
=========================================================================================================================
[root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume
info
Volume Name: engine Type: Replicate Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 30 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on
Volume Name: vmstore Type: Replicate Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 20 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on server.tcp-user-timeout: 20 server.keepalive-time: 10 server.keepalive-interval: 2 server.keepalive-count: 5 cluster.lookup-optimize: off
=========================================================================================================================
[root@ovirt1 ~]# gluster volume profile vmstore info Brick: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore --------------------------------------------------------- Cumulative Stats: Block Size: 1b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 4021 2 64
Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 134 405 No. of Writes: 2 32 119
Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 2890 2617 3059 No. of Writes: 113881 93551 131776
Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 1096597 164931 181569 No. of Writes: 41091001 20499776 3230472
Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 212970 96574 244017 No. of Writes: 1304688 991154 3412866
Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 0 0 0 No. of Writes: 15 18 65166
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 6396 FORGET 0.00 0.00 us 0.00 us 0.00 us 189759 RELEASE 0.00 0.00 us 0.00 us 0.00 us 31292 RELEASEDIR 0.00 159.99 us 44.37 us 278.66 us 79 STAT 0.00 364.42 us 174.87 us 644.45 us 96 LINK 0.00 2714.10 us 203.53 us 47397.28 us 26 RMDIR 0.00 688.26 us 72.39 us 53857.63 us 186 SETXATTR 0.00 974.01 us 271.02 us 27298.52 us 176 RENAME 0.00 5254.64 us 159.48 us 84454.46 us 35 DISCARD 0.00 1133.84 us 86.47 us 87096.92 us 186 REMOVEXATTR 0.00 7907.47 us 221.82 us 182276.21 us 29 FALLOCATE 0.00 8304.51 us 311.89 us 42725.81 us 34 MKDIR 0.00 5758.19 us 296.39 us 21950.99 us 81 TRUNCATE 0.00 401.16 us 23.68 us 132668.93 us 3621 LK 0.00 545.48 us 14.22 us 1076517.76 us 6796 READDIR 0.00 2250.78 us 336.61 us 1071764.61 us 4257 CREATE 0.00 531.76 us 1.35 us 1723046.94 us 31292 OPENDIR 0.01 2058.67 us 196.74 us 1587752.27 us 21986 MKNOD 0.01 1241.28 us 22.61 us 1548642.70 us 52674 READDIRP 0.01 67508.32 us 100.01 us 3002069.81 us 1100 FTRUNCATE 0.01 412.74 us 10.94 us 5111334.35 us 209731 FLUSH 0.02 78668.59 us 183.06 us 3012629.87 us 1222 SETATTR 0.03 761.13 us 21.13 us 4611854.15 us 249094 STATFS 0.03 1132.33 us 45.51 us 3159531.87 us 185654 OPEN 0.03 2010.33 us 14.43 us 3158098.85 us 107824 GETXATTR 0.04 2765.29 us 61.38 us 3159208.13 us 100484 XATTROP 0.13 1367.04 us 51.46 us 8450420.87 us 595299 FSTAT 0.46 157440.90 us 47.62 us 13405712.93 us 18218 UNLINK 0.60 1274.67 us 19.60 us 11985005.14 us 2980806 LOOKUP 1.67 16645.99 us 12.80 us 6869488.64 us 631331 ENTRYLK 4.10 7504.86 us 39.52 us 11749528.82 us 3430506 READ 4.31 56942.36 us 16.57 us 23654127.65 us 475563 INODELK 6.09 660.89 us 11.02 us 20571410.32 us 57908331 FINODELK 20.59 1823.35 us 63.26 us 23639762.45 us 70934143 WRITE 25.70 2680.82 us 35.37 us 13636394.75 us 60223005 FXATTROP 36.14 3164.86 us 52.35 us 16805886.48 us 71739477 FSYNC 0.00 0.00 us 0.00 us 0.00 us 78458 UPCALL
Duration: 404667 seconds Data Read: 59850935724 bytes Data Written: 1070158035683 bytes
Interval 43 Stats: Block Size: 1b+ 256b+ 512b+ No. of Reads: 0 14 3 No. of Writes: 12 4 252
Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 8105 No. of Writes: 100 34 147355
Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1559 1630 1494 No. of Writes: 86013 17318 8687
Block Size: 65536b+ 131072b+ No. of Reads: 532 764 No. of Writes: 8617 36317 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16 FORGET 0.00 0.00 us 0.00 us 0.00 us 665 RELEASE 0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR 0.00 181.75 us 181.75 us 181.75 us 1 STAT 0.00 914.55 us 858.55 us 970.56 us 2 TRUNCATE 0.00 792.11 us 681.24 us 1051.32 us 4 RENAME 0.00 374.44 us 31.86 us 792.10 us 20 READDIR 0.00 399.48 us 306.97 us 562.69 us 28 UNLINK 0.00 752.40 us 636.14 us 907.29 us 16 CREATE 0.00 196.62 us 33.27 us 2586.77 us 336 GETXATTR 0.00 1107.98 us 288.51 us 29692.53 us 80 MKNOD 0.00 1074.80 us 74.41 us 2121.03 us 126 READDIRP 0.00 3119.52 us 3.17 us 157121.44 us 52 OPENDIR 0.01 972.14 us 21.72 us 329342.21 us 635 FLUSH 0.03 2122.15 us 31.97 us 377538.16 us 732 STATFS 0.06 4478.89 us 66.09 us 521289.10 us 649 OPEN 0.10 10273.00 us 88.19 us 478524.77 us 472 XATTROP 0.16 6969.92 us 16.14 us 385156.98 us 1138 ENTRYLK 0.20 4635.40 us 64.29 us 1098319.92 us 2097 FSTAT 0.65 4098.07 us 46.85 us 919617.66 us 7797 LOOKUP 2.36 87465.33 us 34.75 us 1224052.07 us 1317 INODELK 4.37 13897.55 us 57.89 us 1984970.27 us 15336 READ 5.24 1265.46 us 16.22 us 1638678.32 us 202165 FINODELK 21.05 3370.28 us 144.44 us 2264130.49 us 304724 WRITE 31.00 6411.68 us 39.84 us 2190355.42 us 235969 FXATTROP 34.75 6107.95 us 60.26 us 2270252.44 us 277643 FSYNC 0.00 0.00 us 0.00 us 0.00 us 368 UPCALL
Duration: 1132 seconds Data Read: 292367566 bytes Data Written: 7495339692 bytes
Brick: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore --------------------------------------------------------- Cumulative Stats: Block Size: 1b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 4802 2 64
Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 138 828 No. of Writes: 2 32 131
Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 3595 4590 6161 No. of Writes: 128976 99680 132413
Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 930031 128995 150468 No. of Writes: 48357551 23957030 3689474
Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 154039 71949 790010 No. of Writes: 1438859 1052045 3715427
Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 9 11 40269 No. of Writes: 5 20 50757
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 7629 FORGET 0.00 0.00 us 0.00 us 0.00 us 228577 RELEASE 0.00 0.00 us 0.00 us 0.00 us 41524 RELEASEDIR 0.00 374.22 us 219.35 us 574.87 us 96 LINK 0.00 1712.22 us 164.90 us 22337.79 us 41 DISCARD 0.00 2840.25 us 245.24 us 46839.63 us 26 RMDIR 0.00 3660.92 us 229.86 us 86252.56 us 35 FALLOCATE 0.00 876.48 us 297.53 us 16685.63 us 188 RENAME 0.00 6660.10 us 379.21 us 52962.79 us 34 MKDIR 0.00 957.48 us 80.36 us 218059.88 us 355 REMOVEXATTR 0.00 1234.85 us 76.50 us 322955.27 us 355 SETXATTR 0.00 2326.62 us 84.07 us 250041.25 us 196 STAT 0.00 9660.45 us 289.50 us 295106.01 us 87 TRUNCATE 0.00 486.10 us 24.12 us 869966.79 us 5260 LK 0.00 453.69 us 18.32 us 426653.18 us 8135 READDIR 0.00 1832.38 us 296.61 us 261299.81 us 5050 CREATE 0.00 386.77 us 1.31 us 1300836.12 us 41524 OPENDIR 0.00 712.06 us 29.85 us 1441115.02 us 34874 READDIRP 0.01 1469.51 us 231.67 us 440065.71 us 22845 MKNOD 0.01 44229.36 us 75.58 us 1803963.42 us 871 FTRUNCATE 0.01 84433.20 us 149.47 us 3614869.24 us 1001 SETATTR 0.02 418.11 us 13.39 us 3515000.81 us 243820 FLUSH 0.03 1108.47 us 14.77 us 1658743.41 us 130647 GETXATTR 0.03 562.86 us 26.74 us 5043949.33 us 297799 STATFS 0.03 845.50 us 48.16 us 1998938.03 us 223680 OPEN 0.05 2236.13 us 56.08 us 5295998.22 us 120682 XATTROP 0.16 890.82 us 43.51 us 3653292.88 us 1012585 FSTAT 0.34 2974.20 us 12.92 us 7782497.74 us 642555 ENTRYLK 0.47 135831.21 us 70.35 us 11033867.90 us 19800 UNLINK 0.57 954.05 us 21.50 us 4553393.38 us 3414605 LOOKUP 1.79 14461.54 us 13.45 us 32841452.98 us 702915 INODELK 5.14 8014.78 us 40.70 us 5439109.56 us 3644063 READ 5.37 443.04 us 11.53 us 32863652.53 us 68909131 FINODELK 22.31 1780.31 us 33.59 us 11318712.62 us 71235991 FXATTROP 22.83 1571.16 us 74.86 us 32615055.19 us 82622840 WRITE 40.84 2762.75 us 52.77 us 8859115.35 us 84039509 FSYNC 0.00 0.00 us 0.00 us 0.00 us 95492 UPCALL
Duration: 484169 seconds Data Read: 167149718723 bytes Data Written: 1177141649872 bytes
Interval 43 Stats: Block Size: 1b+ 256b+ 512b+ No. of Reads: 0 6 4 No. of Writes: 12 4 252
Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 5668 No. of Writes: 100 34 147357
Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1178 783 1215 No. of Writes: 86014 17318 8687
Block Size: 65536b+ 131072b+ No. of Reads: 264 4109 No. of Writes: 8617 36317 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16 FORGET 0.00 0.00 us 0.00 us 0.00 us 665 RELEASE 0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR 0.00 866.18 us 849.57 us 882.78 us 2 TRUNCATE 0.00 1016.14 us 868.35 us 1182.48 us 4 RENAME 0.00 455.65 us 41.13 us 1679.37 us 20 READDIR 0.00 373.33 us 173.54 us 538.40 us 28 UNLINK 0.00 722.01 us 635.91 us 853.71 us 16 CREATE 0.00 335.52 us 38.15 us 1381.97 us 54 READDIRP 0.00 39218.92 us 214.34 us 78223.50 us 2 STAT 0.00 1730.77 us 2.59 us 84964.69 us 52 OPENDIR 0.00 1304.96 us 393.64 us 29724.09 us 80 MKNOD 0.01 422.75 us 22.85 us 134336.60 us 635 FLUSH 0.01 699.21 us 37.54 us 137813.64 us 732 STATFS 0.01 468.63 us 23.71 us 260961.83 us 1141 ENTRYLK 0.04 2041.83 us 69.80 us 163743.98 us 649 OPEN 0.08 1253.65 us 72.52 us 290508.15 us 2354 FSTAT 0.09 6913.19 us 69.01 us 212479.24 us 472 XATTROP 0.13 3558.26 us 32.70 us 195896.10 us 1317 INODELK 0.13 9793.37 us 36.11 us 212755.58 us 499 GETXATTR 0.34 1615.15 us 55.42 us 711310.28 us 7797 LOOKUP 4.92 13025.92 us 59.83 us 1061585.06 us 13884 READ 5.26 955.95 us 16.81 us 1069148.49 us 202174 FINODELK 23.28 2809.44 us 137.21 us 1134076.13 us 304713 WRITE 26.46 4124.05 us 43.30 us 1142167.27 us 235954 FXATTROP 39.23 5195.62 us 65.47 us 1250469.35 us 277632 FSYNC 0.00 0.00 us 0.00 us 0.00 us 365 UPCALL
Duration: 1132 seconds Data Read: 668951066 bytes Data Written: 7495356076 bytes _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/G3G5HNX3P2Z5E5...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/NKMZYE6EC3IVLF...
-- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant
-- Derek Atkins 617-623-3745 derek@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/EQ47A2RUSF37QU...
Since RAID config add additional io penalties, the RAID should be removed and have to add each of the disks directly to GLUSTER configs. We had similar issues with RAID, after eliminating that we are good with performances Get Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: Strahil Nikolov via Users <users@ovirt.org> Sent: Thursday, February 24, 2022 3:43:39 AM To: Alex Morrison <alex@discoverygarden.ca>; Sunil Kumar Heggodu Gopala Acharya <sheggodu@redhat.com> Cc: users@ovirt.org <users@ovirt.org> Subject: [ovirt-users] Re: Gluster Performance issues You can try to play a little bit with the I/O threads (but don't jump too fast). What is your I/O scheduler and mount options. You can reduce I/O lookups if you specify the 'noatime' and the selinux context on the mount options. A real killer of performance is the lattency. What is the lattency between all nodes ? Best Regards, Strahil Nikolov On Wed, Feb 23, 2022 at 20:26, Alex Morrison <alex@discoverygarden.ca> wrote: Hello All, I believe the network is performing as expected, I did an iperf test: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# iperf3 -c 10.10.1.2 Connecting to host 10.10.1.2, port 5201 [ 5] local 10.10.1.1 port 38422 connected to 10.10.1.2 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.08 GBytes 9.24 Gbits/sec 0 2.96 MBytes [ 5] 1.00-2.00 sec 1.03 GBytes 8.81 Gbits/sec 0 2.96 MBytes [ 5] 2.00-3.00 sec 1006 MBytes 8.44 Gbits/sec 101 1.45 MBytes [ 5] 3.00-4.00 sec 1.04 GBytes 8.92 Gbits/sec 5 901 KBytes [ 5] 4.00-5.00 sec 1.05 GBytes 9.01 Gbits/sec 0 957 KBytes [ 5] 5.00-6.00 sec 1.08 GBytes 9.23 Gbits/sec 0 990 KBytes [ 5] 6.00-7.00 sec 1008 MBytes 8.46 Gbits/sec 159 655 KBytes [ 5] 7.00-8.00 sec 1.06 GBytes 9.11 Gbits/sec 0 970 KBytes [ 5] 8.00-9.00 sec 1.03 GBytes 8.85 Gbits/sec 2 829 KBytes [ 5] 9.00-10.00 sec 1.04 GBytes 8.96 Gbits/sec 0 947 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.4 GBytes 8.90 Gbits/sec 267 sender [ 5] 0.00-10.04 sec 10.4 GBytes 8.87 Gbits/sec receiver iperf Done. On Wed, Feb 23, 2022 at 11:45 AM Sunil Kumar Heggodu Gopala Acharya <sheggodu@redhat.com<mailto:sheggodu@redhat.com>> wrote: Regards, Sunil On Wed, Feb 23, 2022 at 7:34 PM Derek Atkins <derek@ihtfp.com<mailto:derek@ihtfp.com>> wrote: Have you verified that you're actually getting 10Gbps between the hosts? -derek On Wed, February 23, 2022 9:02 am, Alex Morrison wrote:
Hello Derek,
We have a 10Gig connection dedicated to the storage network, nothing else is on that switch.
On Wed, Feb 23, 2022 at 9:49 AM Derek Atkins <derek@ihtfp.com<mailto:derek@ihtfp.com>> wrote:
Hi,
Another question which I don't see answered: What is the underlying connectivity between the Gluster hosts?
-derek
On Wed, February 23, 2022 8:39 am, Alex Morrison wrote:
Hello Sunil,
[root@ovirt1 ~]# gluster --version glusterfs 8.6
same on all hosts Latest Release-10.1(https://lists.gluster.org/pipermail/gluster-users/2022-February/039761.html) has some performance fixes which should help in this situation compared to the older gluster bits.
On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya < sheggodu@redhat.com<mailto:sheggodu@redhat.com>> wrote:
Hi,
Which version of gluster is in use?
Regards,
Sunil kumar Acharya
Red Hat
T: +91-8067935170 <http://redhatemailsignature-marketing.itos.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison <alex@discoverygarden.ca<mailto:alex@discoverygarden.ca>> wrote:
Hello All,
We have 3 servers with a raid 50 array each, we are having extreme performance issues with our gluster, writes on gluster seem to take at least 3 times longer than on the raid directly. Can this be improved? I've read through several other performance issues threads but have been unable to make any improvements
"gluster volume info" and "gluster volume profile vmstore info" is below
=========================================================================================================================
-Inside Gluster - test took 35+ hours: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 ------Sequential Output------ --Sequential
Input-
--Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 35.7m 17 5824k 7 112m 13 182.7 6 Latency 5466ms 12754ms 3499ms 1589ms
1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,,,,,,,,,,,,,,,,,5466ms,12754ms,,3499ms,1589ms,,,,,,
=========================================================================================================================
-Outside Gluster - test took 18 minutes: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 ------Sequential Output------ --Sequential
Input-
--Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 567m 78 149m 30 307m 37 83.0 57 Latency 205ms 4630ms 1450ms 679ms
1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,,,,,,,,,,,,,,,,,205ms,4630ms,,1450ms,679ms,,,,,,
=========================================================================================================================
[root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume
info
Volume Name: engine Type: Replicate Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 30 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on
Volume Name: vmstore Type: Replicate Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 20 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on server.tcp-user-timeout: 20 server.keepalive-time: 10 server.keepalive-interval: 2 server.keepalive-count: 5 cluster.lookup-optimize: off
=========================================================================================================================
[root@ovirt1 ~]# gluster volume profile vmstore info Brick: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore --------------------------------------------------------- Cumulative Stats: Block Size: 1b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 4021 2 64
Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 134 405 No. of Writes: 2 32 119
Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 2890 2617 3059 No. of Writes: 113881 93551 131776
Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 1096597 164931 181569 No. of Writes: 41091001 20499776 3230472
Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 212970 96574 244017 No. of Writes: 1304688 991154 3412866
Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 0 0 0 No. of Writes: 15 18 65166
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 6396 FORGET 0.00 0.00 us 0.00 us 0.00 us 189759 RELEASE 0.00 0.00 us 0.00 us 0.00 us 31292 RELEASEDIR 0.00 159.99 us 44.37 us 278.66 us 79 STAT 0.00 364.42 us 174.87 us 644.45 us 96 LINK 0.00 2714.10 us 203.53 us 47397.28 us 26 RMDIR 0.00 688.26 us 72.39 us 53857.63 us 186 SETXATTR 0.00 974.01 us 271.02 us 27298.52 us 176 RENAME 0.00 5254.64 us 159.48 us 84454.46 us 35 DISCARD 0.00 1133.84 us 86.47 us 87096.92 us 186 REMOVEXATTR 0.00 7907.47 us 221.82 us 182276.21 us 29 FALLOCATE 0.00 8304.51 us 311.89 us 42725.81 us 34 MKDIR 0.00 5758.19 us 296.39 us 21950.99 us 81 TRUNCATE 0.00 401.16 us 23.68 us 132668.93 us 3621 LK 0.00 545.48 us 14.22 us 1076517.76 us 6796 READDIR 0.00 2250.78 us 336.61 us 1071764.61 us 4257 CREATE 0.00 531.76 us 1.35 us 1723046.94 us 31292 OPENDIR 0.01 2058.67 us 196.74 us 1587752.27 us 21986 MKNOD 0.01 1241.28 us 22.61 us 1548642.70 us 52674 READDIRP 0.01 67508.32 us 100.01 us 3002069.81 us 1100 FTRUNCATE 0.01 412.74 us 10.94 us 5111334.35 us 209731 FLUSH 0.02 78668.59 us 183.06 us 3012629.87 us 1222 SETATTR 0.03 761.13 us 21.13 us 4611854.15 us 249094 STATFS 0.03 1132.33 us 45.51 us 3159531.87 us 185654 OPEN 0.03 2010.33 us 14.43 us 3158098.85 us 107824 GETXATTR 0.04 2765.29 us 61.38 us 3159208.13 us 100484 XATTROP 0.13 1367.04 us 51.46 us 8450420.87 us 595299 FSTAT 0.46 157440.90 us 47.62 us 13405712.93 us 18218 UNLINK 0.60 1274.67 us 19.60 us 11985005.14 us 2980806 LOOKUP 1.67 16645.99 us 12.80 us 6869488.64 us 631331 ENTRYLK 4.10 7504.86 us 39.52 us 11749528.82 us 3430506 READ 4.31 56942.36 us 16.57 us 23654127.65 us 475563 INODELK 6.09 660.89 us 11.02 us 20571410.32 us 57908331 FINODELK 20.59 1823.35 us 63.26 us 23639762.45 us 70934143 WRITE 25.70 2680.82 us 35.37 us 13636394.75 us 60223005 FXATTROP 36.14 3164.86 us 52.35 us 16805886.48 us 71739477 FSYNC 0.00 0.00 us 0.00 us 0.00 us 78458 UPCALL
Duration: 404667 seconds Data Read: 59850935724 bytes Data Written: 1070158035683 bytes
Interval 43 Stats: Block Size: 1b+ 256b+ 512b+ No. of Reads: 0 14 3 No. of Writes: 12 4 252
Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 8105 No. of Writes: 100 34 147355
Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1559 1630 1494 No. of Writes: 86013 17318 8687
Block Size: 65536b+ 131072b+ No. of Reads: 532 764 No. of Writes: 8617 36317 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16 FORGET 0.00 0.00 us 0.00 us 0.00 us 665 RELEASE 0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR 0.00 181.75 us 181.75 us 181.75 us 1 STAT 0.00 914.55 us 858.55 us 970.56 us 2 TRUNCATE 0.00 792.11 us 681.24 us 1051.32 us 4 RENAME 0.00 374.44 us 31.86 us 792.10 us 20 READDIR 0.00 399.48 us 306.97 us 562.69 us 28 UNLINK 0.00 752.40 us 636.14 us 907.29 us 16 CREATE 0.00 196.62 us 33.27 us 2586.77 us 336 GETXATTR 0.00 1107.98 us 288.51 us 29692.53 us 80 MKNOD 0.00 1074.80 us 74.41 us 2121.03 us 126 READDIRP 0.00 3119.52 us 3.17 us 157121.44 us 52 OPENDIR 0.01 972.14 us 21.72 us 329342.21 us 635 FLUSH 0.03 2122.15 us 31.97 us 377538.16 us 732 STATFS 0.06 4478.89 us 66.09 us 521289.10 us 649 OPEN 0.10 10273.00 us 88.19 us 478524.77 us 472 XATTROP 0.16 6969.92 us 16.14 us 385156.98 us 1138 ENTRYLK 0.20 4635.40 us 64.29 us 1098319.92 us 2097 FSTAT 0.65 4098.07 us 46.85 us 919617.66 us 7797 LOOKUP 2.36 87465.33 us 34.75 us 1224052.07 us 1317 INODELK 4.37 13897.55 us 57.89 us 1984970.27 us 15336 READ 5.24 1265.46 us 16.22 us 1638678.32 us 202165 FINODELK 21.05 3370.28 us 144.44 us 2264130.49 us 304724 WRITE 31.00 6411.68 us 39.84 us 2190355.42 us 235969 FXATTROP 34.75 6107.95 us 60.26 us 2270252.44 us 277643 FSYNC 0.00 0.00 us 0.00 us 0.00 us 368 UPCALL
Duration: 1132 seconds Data Read: 292367566 bytes Data Written: 7495339692 bytes
Brick: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore --------------------------------------------------------- Cumulative Stats: Block Size: 1b+ 4b+ 8b+ No. of Reads: 0 0 0 No. of Writes: 4802 2 64
Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 138 828 No. of Writes: 2 32 131
Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 3595 4590 6161 No. of Writes: 128976 99680 132413
Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 930031 128995 150468 No. of Writes: 48357551 23957030 3689474
Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 154039 71949 790010 No. of Writes: 1438859 1052045 3715427
Block Size: 262144b+ 524288b+ 1048576b+ No. of Reads: 9 11 40269 No. of Writes: 5 20 50757
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 7629 FORGET 0.00 0.00 us 0.00 us 0.00 us 228577 RELEASE 0.00 0.00 us 0.00 us 0.00 us 41524 RELEASEDIR 0.00 374.22 us 219.35 us 574.87 us 96 LINK 0.00 1712.22 us 164.90 us 22337.79 us 41 DISCARD 0.00 2840.25 us 245.24 us 46839.63 us 26 RMDIR 0.00 3660.92 us 229.86 us 86252.56 us 35 FALLOCATE 0.00 876.48 us 297.53 us 16685.63 us 188 RENAME 0.00 6660.10 us 379.21 us 52962.79 us 34 MKDIR 0.00 957.48 us 80.36 us 218059.88 us 355 REMOVEXATTR 0.00 1234.85 us 76.50 us 322955.27 us 355 SETXATTR 0.00 2326.62 us 84.07 us 250041.25 us 196 STAT 0.00 9660.45 us 289.50 us 295106.01 us 87 TRUNCATE 0.00 486.10 us 24.12 us 869966.79 us 5260 LK 0.00 453.69 us 18.32 us 426653.18 us 8135 READDIR 0.00 1832.38 us 296.61 us 261299.81 us 5050 CREATE 0.00 386.77 us 1.31 us 1300836.12 us 41524 OPENDIR 0.00 712.06 us 29.85 us 1441115.02 us 34874 READDIRP 0.01 1469.51 us 231.67 us 440065.71 us 22845 MKNOD 0.01 44229.36 us 75.58 us 1803963.42 us 871 FTRUNCATE 0.01 84433.20 us 149.47 us 3614869.24 us 1001 SETATTR 0.02 418.11 us 13.39 us 3515000.81 us 243820 FLUSH 0.03 1108.47 us 14.77 us 1658743.41 us 130647 GETXATTR 0.03 562.86 us 26.74 us 5043949.33 us 297799 STATFS 0.03 845.50 us 48.16 us 1998938.03 us 223680 OPEN 0.05 2236.13 us 56.08 us 5295998.22 us 120682 XATTROP 0.16 890.82 us 43.51 us 3653292.88 us 1012585 FSTAT 0.34 2974.20 us 12.92 us 7782497.74 us 642555 ENTRYLK 0.47 135831.21 us 70.35 us 11033867.90 us 19800 UNLINK 0.57 954.05 us 21.50 us 4553393.38 us 3414605 LOOKUP 1.79 14461.54 us 13.45 us 32841452.98 us 702915 INODELK 5.14 8014.78 us 40.70 us 5439109.56 us 3644063 READ 5.37 443.04 us 11.53 us 32863652.53 us 68909131 FINODELK 22.31 1780.31 us 33.59 us 11318712.62 us 71235991 FXATTROP 22.83 1571.16 us 74.86 us 32615055.19 us 82622840 WRITE 40.84 2762.75 us 52.77 us 8859115.35 us 84039509 FSYNC 0.00 0.00 us 0.00 us 0.00 us 95492 UPCALL
Duration: 484169 seconds Data Read: 167149718723 bytes Data Written: 1177141649872 bytes
Interval 43 Stats: Block Size: 1b+ 256b+ 512b+ No. of Reads: 0 6 4 No. of Writes: 12 4 252
Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 0 5668 No. of Writes: 100 34 147357
Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1178 783 1215 No. of Writes: 86014 17318 8687
Block Size: 65536b+ 131072b+ No. of Reads: 264 4109 No. of Writes: 8617 36317 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 16 FORGET 0.00 0.00 us 0.00 us 0.00 us 665 RELEASE 0.00 0.00 us 0.00 us 0.00 us 52 RELEASEDIR 0.00 866.18 us 849.57 us 882.78 us 2 TRUNCATE 0.00 1016.14 us 868.35 us 1182.48 us 4 RENAME 0.00 455.65 us 41.13 us 1679.37 us 20 READDIR 0.00 373.33 us 173.54 us 538.40 us 28 UNLINK 0.00 722.01 us 635.91 us 853.71 us 16 CREATE 0.00 335.52 us 38.15 us 1381.97 us 54 READDIRP 0.00 39218.92 us 214.34 us 78223.50 us 2 STAT 0.00 1730.77 us 2.59 us 84964.69 us 52 OPENDIR 0.00 1304.96 us 393.64 us 29724.09 us 80 MKNOD 0.01 422.75 us 22.85 us 134336.60 us 635 FLUSH 0.01 699.21 us 37.54 us 137813.64 us 732 STATFS 0.01 468.63 us 23.71 us 260961.83 us 1141 ENTRYLK 0.04 2041.83 us 69.80 us 163743.98 us 649 OPEN 0.08 1253.65 us 72.52 us 290508.15 us 2354 FSTAT 0.09 6913.19 us 69.01 us 212479.24 us 472 XATTROP 0.13 3558.26 us 32.70 us 195896.10 us 1317 INODELK 0.13 9793.37 us 36.11 us 212755.58 us 499 GETXATTR 0.34 1615.15 us 55.42 us 711310.28 us 7797 LOOKUP 4.92 13025.92 us 59.83 us 1061585.06 us 13884 READ 5.26 955.95 us 16.81 us 1069148.49 us 202174 FINODELK 23.28 2809.44 us 137.21 us 1134076.13 us 304713 WRITE 26.46 4124.05 us 43.30 us 1142167.27 us 235954 FXATTROP 39.23 5195.62 us 65.47 us 1250469.35 us 277632 FSYNC 0.00 0.00 us 0.00 us 0.00 us 365 UPCALL
Duration: 1132 seconds Data Read: 668951066 bytes Data Written: 7495356076 bytes _______________________________________________ Users mailing list -- users@ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/G3G5HNX3P2Z5E5...
_______________________________________________ Users mailing list -- users@ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/NKMZYE6EC3IVLF...
-- Derek Atkins 617-623-3745 derek@ihtfp.com<mailto:derek@ihtfp.com> www.ihtfp.com<http://www.ihtfp.com> Computer and Internet Security Consultant
-- Derek Atkins 617-623-3745 derek@ihtfp.com<mailto:derek@ihtfp.com> www.ihtfp.com<http://www.ihtfp.com> Computer and Internet Security Consultant _______________________________________________ Users mailing list -- users@ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/EQ47A2RUSF37QU...
participants (5)
-
Alex Morrison -
Derek Atkins -
Dhanaraj Ramesh -
Strahil Nikolov -
Sunil Kumar Heggodu Gopala Acharya