Hello Sunil, [root@ovirt1 ~]# gluster --version glusterfs 8.6 same on all hosts On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya < sheggodu(a)redhat.com&gt; wrote: > Hi, > > Which version of gluster is in use? > > Regards, > > Sunil kumar Acharya > > Red Hat > > <https://www.redhat.com> > > T: +91-8067935170 > <http://redhatemailsignature-marketing.itos.redhat.com/> > > <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > > > > On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison <alex(a)discoverygarden.ca&gt; > wrote: > >> Hello All, >> >> We have 3 servers with a raid 50 array each, we are having extreme >> performance issues with our gluster, writes on gluster seem to take at >> least 3 times longer than on the raid directly. Can this be improved? >> I've >> read through several other performance issues threads but have been >> unable >> to make any improvements >> >> "gluster volume info" and "gluster volume profile vmstore info" is >> below >> >> >> ========================================================================================================================= >> >> -Inside Gluster - test took 35+ hours: >> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s >> 600G -n 0 -m TEST -f -b -u root >> Using uid:0, gid:0. >> Writing intelligently...done >> Rewriting...done >> Reading intelligently...done >> start 'em...done...done...done...done...done... >> Version 1.98 ------Sequential Output------ --Sequential Input- >> --Random- >> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- >> Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >> /sec %CP >> TEST 600G 35.7m 17 5824k 7 112m 13 >> 182.7 6 >> Latency 5466ms 12754ms 3499ms >> 1589ms >> >> >> 1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,,,,,,,,,,,,,,,,,5466ms,12754ms,,3499ms,1589ms,,,,,, >> >> >> ========================================================================================================================= >> >> -Outside Gluster - test took 18 minutes: >> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s >> 600G -n 0 -m TEST -f -b -u root >> Using uid:0, gid:0. >> Writing intelligently...done >> Rewriting...done >> Reading intelligently...done >> start 'em...done...done...done...done...done... >> Version 1.98 ------Sequential Output------ --Sequential Input- >> --Random- >> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- >> Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >> /sec %CP >> TEST 600G 567m 78 149m 30 307m 37 >> 83.0 57 >> Latency 205ms 4630ms 1450ms >> 679ms >> >> >> 1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,,,,,,,,,,,,,,,,,205ms,4630ms,,1450ms,679ms,,,,,, >> >> >> ========================================================================================================================= >> >> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume info >> Volume Name: engine >> Type: Replicate >> Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x 3 = 3 >> Transport-type: tcp >> Bricks: >> Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine >> Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine >> Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine >> Options Reconfigured: >> cluster.granular-entry-heal: enable >> performance.strict-o-direct: on >> network.ping-timeout: 30 >> storage.owner-gid: 36 >> storage.owner-uid: 36 >> server.event-threads: 4 >> client.event-threads: 4 >> cluster.choose-local: off >> user.cifs: off >> features.shard: on >> cluster.shd-wait-qlength: 10000 >> cluster.shd-max-threads: 8 >> cluster.locking-scheme: granular >> cluster.data-self-heal-algorithm: full >> cluster.server-quorum-type: server >> cluster.quorum-type: auto >> cluster.eager-lock: enable >> network.remote-dio: off >> performance.low-prio-threads: 32 >> performance.io-cache: off >> performance.read-ahead: off >> performance.quick-read: off >> transport.address-family: inet >> storage.fips-mode-rchecksum: on >> nfs.disable: on >> performance.client-io-threads: on >> diagnostics.latency-measurement: on >> diagnostics.count-fop-hits: on >> >> Volume Name: vmstore >> Type: Replicate >> Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x 3 = 3 >> Transport-type: tcp >> Bricks: >> Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore >> Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore >> Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore >> Options Reconfigured: >> cluster.granular-entry-heal: enable >> performance.strict-o-direct: on >> network.ping-timeout: 20 >> storage.owner-gid: 36 >> storage.owner-uid: 36 >> server.event-threads: 4 >> client.event-threads: 4 >> cluster.choose-local: off >> user.cifs: off >> features.shard: on >> cluster.shd-wait-qlength: 10000 >> cluster.shd-max-threads: 8 >> cluster.locking-scheme: granular >> cluster.data-self-heal-algorithm: full >> cluster.server-quorum-type: server >> cluster.quorum-type: auto >> cluster.eager-lock: enable >> network.remote-dio: off >> performance.low-prio-threads: 32 >> performance.io-cache: off >> performance.read-ahead: off >> performance.quick-read: off >> transport.address-family: inet >> storage.fips-mode-rchecksum: on >> nfs.disable: on >> performance.client-io-threads: on >> diagnostics.latency-measurement: on >> diagnostics.count-fop-hits: on >> server.tcp-user-timeout: 20 >> server.keepalive-time: 10 >> server.keepalive-interval: 2 >> server.keepalive-count: 5 >> cluster.lookup-optimize: off >> >> >> ========================================================================================================================= >> >> [root@ovirt1 ~]# gluster volume profile vmstore info >> Brick: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore >> --------------------------------------------------------- >> Cumulative Stats: >> Block Size: 1b+ 4b+ >> 8b+ >> No. of Reads: 0 0 >> 0 >> No. of Writes: 4021 2 >> 64 >> >> Block Size: 64b+ 128b+ >> 256b+ >> No. of Reads: 0 134 >> 405 >> No. of Writes: 2 32 >> 119 >> >> Block Size: 512b+ 1024b+ >> 2048b+ >> No. of Reads: 2890 2617 >> 3059 >> No. of Writes: 113881 93551 >> 131776 >> >> Block Size: 4096b+ 8192b+ >> 16384b+ >> No. of Reads: 1096597 164931 >> 181569 >> No. of Writes: 41091001 20499776 >> 3230472 >> >> Block Size: 32768b+ 65536b+ >> 131072b+ >> No. of Reads: 212970 96574 >> 244017 >> No. of Writes: 1304688 991154 >> 3412866 >> >> Block Size: 262144b+ 524288b+ >> 1048576b+ >> No. of Reads: 0 0 >> 0 >> No. of Writes: 15 18 >> 65166 >> >> %-latency Avg-latency Min-Latency Max-Latency No. of calls >> Fop >> --------- ----------- ----------- ----------- ------------ >> ---- >> 0.00 0.00 us 0.00 us 0.00 us 6396 >> FORGET >> 0.00 0.00 us 0.00 us 0.00 us 189759 >> RELEASE >> 0.00 0.00 us 0.00 us 0.00 us 31292 >> RELEASEDIR >> 0.00 159.99 us 44.37 us 278.66 us 79 >> STAT >> 0.00 364.42 us 174.87 us 644.45 us 96 >> LINK >> 0.00 2714.10 us 203.53 us 47397.28 us 26 >> RMDIR >> 0.00 688.26 us 72.39 us 53857.63 us 186 >> SETXATTR >> 0.00 974.01 us 271.02 us 27298.52 us 176 >> RENAME >> 0.00 5254.64 us 159.48 us 84454.46 us 35 >> DISCARD >> 0.00 1133.84 us 86.47 us 87096.92 us 186 >> REMOVEXATTR >> 0.00 7907.47 us 221.82 us 182276.21 us 29 >> FALLOCATE >> 0.00 8304.51 us 311.89 us 42725.81 us 34 >> MKDIR >> 0.00 5758.19 us 296.39 us 21950.99 us 81 >> TRUNCATE >> 0.00 401.16 us 23.68 us 132668.93 us 3621 >> LK >> 0.00 545.48 us 14.22 us 1076517.76 us 6796 >> READDIR >> 0.00 2250.78 us 336.61 us 1071764.61 us 4257 >> CREATE >> 0.00 531.76 us 1.35 us 1723046.94 us 31292 >> OPENDIR >> 0.01 2058.67 us 196.74 us 1587752.27 us 21986 >> MKNOD >> 0.01 1241.28 us 22.61 us 1548642.70 us 52674 >> READDIRP >> 0.01 67508.32 us 100.01 us 3002069.81 us 1100 >> FTRUNCATE >> 0.01 412.74 us 10.94 us 5111334.35 us 209731 >> FLUSH >> 0.02 78668.59 us 183.06 us 3012629.87 us 1222 >> SETATTR >> 0.03 761.13 us 21.13 us 4611854.15 us 249094 >> STATFS >> 0.03 1132.33 us 45.51 us 3159531.87 us 185654 >> OPEN >> 0.03 2010.33 us 14.43 us 3158098.85 us 107824 >> GETXATTR >> 0.04 2765.29 us 61.38 us 3159208.13 us 100484 >> XATTROP >> 0.13 1367.04 us 51.46 us 8450420.87 us 595299 >> FSTAT >> 0.46 157440.90 us 47.62 us 13405712.93 us 18218 >> UNLINK >> 0.60 1274.67 us 19.60 us 11985005.14 us 2980806 >> LOOKUP >> 1.67 16645.99 us 12.80 us 6869488.64 us 631331 >> ENTRYLK >> 4.10 7504.86 us 39.52 us 11749528.82 us 3430506 >> READ >> 4.31 56942.36 us 16.57 us 23654127.65 us 475563 >> INODELK >> 6.09 660.89 us 11.02 us 20571410.32 us 57908331 >> FINODELK >> 20.59 1823.35 us 63.26 us 23639762.45 us 70934143 >> WRITE >> 25.70 2680.82 us 35.37 us 13636394.75 us 60223005 >> FXATTROP >> 36.14 3164.86 us 52.35 us 16805886.48 us 71739477 >> FSYNC >> 0.00 0.00 us 0.00 us 0.00 us 78458 >> UPCALL >> >> Duration: 404667 seconds >> Data Read: 59850935724 bytes >> Data Written: 1070158035683 bytes >> >> Interval 43 Stats: >> Block Size: 1b+ 256b+ >> 512b+ >> No. of Reads: 0 14 >> 3 >> No. of Writes: 12 4 >> 252 >> >> Block Size: 1024b+ 2048b+ >> 4096b+ >> No. of Reads: 0 0 >> 8105 >> No. of Writes: 100 34 >> 147355 >> >> Block Size: 8192b+ 16384b+ >> 32768b+ >> No. of Reads: 1559 1630 >> 1494 >> No. of Writes: 86013 17318 >> 8687 >> >> Block Size: 65536b+ 131072b+ >> No. of Reads: 532 764 >> No. of Writes: 8617 36317 >> %-latency Avg-latency Min-Latency Max-Latency No. of calls >> Fop >> --------- ----------- ----------- ----------- ------------ >> ---- >> 0.00 0.00 us 0.00 us 0.00 us 16 >> FORGET >> 0.00 0.00 us 0.00 us 0.00 us 665 >> RELEASE >> 0.00 0.00 us 0.00 us 0.00 us 52 >> RELEASEDIR >> 0.00 181.75 us 181.75 us 181.75 us 1 >> STAT >> 0.00 914.55 us 858.55 us 970.56 us 2 >> TRUNCATE >> 0.00 792.11 us 681.24 us 1051.32 us 4 >> RENAME >> 0.00 374.44 us 31.86 us 792.10 us 20 >> READDIR >> 0.00 399.48 us 306.97 us 562.69 us 28 >> UNLINK >> 0.00 752.40 us 636.14 us 907.29 us 16 >> CREATE >> 0.00 196.62 us 33.27 us 2586.77 us 336 >> GETXATTR >> 0.00 1107.98 us 288.51 us 29692.53 us 80 >> MKNOD >> 0.00 1074.80 us 74.41 us 2121.03 us 126 >> READDIRP >> 0.00 3119.52 us 3.17 us 157121.44 us 52 >> OPENDIR >> 0.01 972.14 us 21.72 us 329342.21 us 635 >> FLUSH >> 0.03 2122.15 us 31.97 us 377538.16 us 732 >> STATFS >> 0.06 4478.89 us 66.09 us 521289.10 us 649 >> OPEN >> 0.10 10273.00 us 88.19 us 478524.77 us 472 >> XATTROP >> 0.16 6969.92 us 16.14 us 385156.98 us 1138 >> ENTRYLK >> 0.20 4635.40 us 64.29 us 1098319.92 us 2097 >> FSTAT >> 0.65 4098.07 us 46.85 us 919617.66 us 7797 >> LOOKUP >> 2.36 87465.33 us 34.75 us 1224052.07 us 1317 >> INODELK >> 4.37 13897.55 us 57.89 us 1984970.27 us 15336 >> READ >> 5.24 1265.46 us 16.22 us 1638678.32 us 202165 >> FINODELK >> 21.05 3370.28 us 144.44 us 2264130.49 us 304724 >> WRITE >> 31.00 6411.68 us 39.84 us 2190355.42 us 235969 >> FXATTROP >> 34.75 6107.95 us 60.26 us 2270252.44 us 277643 >> FSYNC >> 0.00 0.00 us 0.00 us 0.00 us 368 >> UPCALL >> >> Duration: 1132 seconds >> Data Read: 292367566 bytes >> Data Written: 7495339692 bytes >> >> Brick: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore >> --------------------------------------------------------- >> Cumulative Stats: >> Block Size: 1b+ 4b+ >> 8b+ >> No. of Reads: 0 0 >> 0 >> No. of Writes: 4802 2 >> 64 >> >> Block Size: 64b+ 128b+ >> 256b+ >> No. of Reads: 0 138 >> 828 >> No. of Writes: 2 32 >> 131 >> >> Block Size: 512b+ 1024b+ >> 2048b+ >> No. of Reads: 3595 4590 >> 6161 >> No. of Writes: 128976 99680 >> 132413 >> >> Block Size: 4096b+ 8192b+ >> 16384b+ >> No. of Reads: 930031 128995 >> 150468 >> No. of Writes: 48357551 23957030 >> 3689474 >> >> Block Size: 32768b+ 65536b+ >> 131072b+ >> No. of Reads: 154039 71949 >> 790010 >> No. of Writes: 1438859 1052045 >> 3715427 >> >> Block Size: 262144b+ 524288b+ >> 1048576b+ >> No. of Reads: 9 11 >> 40269 >> No. of Writes: 5 20 >> 50757 >> >> %-latency Avg-latency Min-Latency Max-Latency No. of calls >> Fop >> --------- ----------- ----------- ----------- ------------ >> ---- >> 0.00 0.00 us 0.00 us 0.00 us 7629 >> FORGET >> 0.00 0.00 us 0.00 us 0.00 us 228577 >> RELEASE >> 0.00 0.00 us 0.00 us 0.00 us 41524 >> RELEASEDIR >> 0.00 374.22 us 219.35 us 574.87 us 96 >> LINK >> 0.00 1712.22 us 164.90 us 22337.79 us 41 >> DISCARD >> 0.00 2840.25 us 245.24 us 46839.63 us 26 >> RMDIR >> 0.00 3660.92 us 229.86 us 86252.56 us 35 >> FALLOCATE >> 0.00 876.48 us 297.53 us 16685.63 us 188 >> RENAME >> 0.00 6660.10 us 379.21 us 52962.79 us 34 >> MKDIR >> 0.00 957.48 us 80.36 us 218059.88 us 355 >> REMOVEXATTR >> 0.00 1234.85 us 76.50 us 322955.27 us 355 >> SETXATTR >> 0.00 2326.62 us 84.07 us 250041.25 us 196 >> STAT >> 0.00 9660.45 us 289.50 us 295106.01 us 87 >> TRUNCATE >> 0.00 486.10 us 24.12 us 869966.79 us 5260 >> LK >> 0.00 453.69 us 18.32 us 426653.18 us 8135 >> READDIR >> 0.00 1832.38 us 296.61 us 261299.81 us 5050 >> CREATE >> 0.00 386.77 us 1.31 us 1300836.12 us 41524 >> OPENDIR >> 0.00 712.06 us 29.85 us 1441115.02 us 34874 >> READDIRP >> 0.01 1469.51 us 231.67 us 440065.71 us 22845 >> MKNOD >> 0.01 44229.36 us 75.58 us 1803963.42 us 871 >> FTRUNCATE >> 0.01 84433.20 us 149.47 us 3614869.24 us 1001 >> SETATTR >> 0.02 418.11 us 13.39 us 3515000.81 us 243820 >> FLUSH >> 0.03 1108.47 us 14.77 us 1658743.41 us 130647 >> GETXATTR >> 0.03 562.86 us 26.74 us 5043949.33 us 297799 >> STATFS >> 0.03 845.50 us 48.16 us 1998938.03 us 223680 >> OPEN >> 0.05 2236.13 us 56.08 us 5295998.22 us 120682 >> XATTROP >> 0.16 890.82 us 43.51 us 3653292.88 us 1012585 >> FSTAT >> 0.34 2974.20 us 12.92 us 7782497.74 us 642555 >> ENTRYLK >> 0.47 135831.21 us 70.35 us 11033867.90 us 19800 >> UNLINK >> 0.57 954.05 us 21.50 us 4553393.38 us 3414605 >> LOOKUP >> 1.79 14461.54 us 13.45 us 32841452.98 us 702915 >> INODELK >> 5.14 8014.78 us 40.70 us 5439109.56 us 3644063 >> READ >> 5.37 443.04 us 11.53 us 32863652.53 us 68909131 >> FINODELK >> 22.31 1780.31 us 33.59 us 11318712.62 us 71235991 >> FXATTROP >> 22.83 1571.16 us 74.86 us 32615055.19 us 82622840 >> WRITE >> 40.84 2762.75 us 52.77 us 8859115.35 us 84039509 >> FSYNC >> 0.00 0.00 us 0.00 us 0.00 us 95492 >> UPCALL >> >> Duration: 484169 seconds >> Data Read: 167149718723 bytes >> Data Written: 1177141649872 bytes >> >> Interval 43 Stats: >> Block Size: 1b+ 256b+ >> 512b+ >> No. of Reads: 0 6 >> 4 >> No. of Writes: 12 4 >> 252 >> >> Block Size: 1024b+ 2048b+ >> 4096b+ >> No. of Reads: 0 0 >> 5668 >> No. of Writes: 100 34 >> 147357 >> >> Block Size: 8192b+ 16384b+ >> 32768b+ >> No. of Reads: 1178 783 >> 1215 >> No. of Writes: 86014 17318 >> 8687 >> >> Block Size: 65536b+ 131072b+ >> No. of Reads: 264 4109 >> No. of Writes: 8617 36317 >> %-latency Avg-latency Min-Latency Max-Latency No. of calls >> Fop >> --------- ----------- ----------- ----------- ------------ >> ---- >> 0.00 0.00 us 0.00 us 0.00 us 16 >> FORGET >> 0.00 0.00 us 0.00 us 0.00 us 665 >> RELEASE >> 0.00 0.00 us 0.00 us 0.00 us 52 >> RELEASEDIR >> 0.00 866.18 us 849.57 us 882.78 us 2 >> TRUNCATE >> 0.00 1016.14 us 868.35 us 1182.48 us 4 >> RENAME >> 0.00 455.65 us 41.13 us 1679.37 us 20 >> READDIR >> 0.00 373.33 us 173.54 us 538.40 us 28 >> UNLINK >> 0.00 722.01 us 635.91 us 853.71 us 16 >> CREATE >> 0.00 335.52 us 38.15 us 1381.97 us 54 >> READDIRP >> 0.00 39218.92 us 214.34 us 78223.50 us 2 >> STAT >> 0.00 1730.77 us 2.59 us 84964.69 us 52 >> OPENDIR >> 0.00 1304.96 us 393.64 us 29724.09 us 80 >> MKNOD >> 0.01 422.75 us 22.85 us 134336.60 us 635 >> FLUSH >> 0.01 699.21 us 37.54 us 137813.64 us 732 >> STATFS >> 0.01 468.63 us 23.71 us 260961.83 us 1141 >> ENTRYLK >> 0.04 2041.83 us 69.80 us 163743.98 us 649 >> OPEN >> 0.08 1253.65 us 72.52 us 290508.15 us 2354 >> FSTAT >> 0.09 6913.19 us 69.01 us 212479.24 us 472 >> XATTROP >> 0.13 3558.26 us 32.70 us 195896.10 us 1317 >> INODELK >> 0.13 9793.37 us 36.11 us 212755.58 us 499 >> GETXATTR >> 0.34 1615.15 us 55.42 us 711310.28 us 7797 >> LOOKUP >> 4.92 13025.92 us 59.83 us 1061585.06 us 13884 >> READ >> 5.26 955.95 us 16.81 us 1069148.49 us 202174 >> FINODELK >> 23.28 2809.44 us 137.21 us 1134076.13 us 304713 >> WRITE >> 26.46 4124.05 us 43.30 us 1142167.27 us 235954 >> FXATTROP >> 39.23 5195.62 us 65.47 us 1250469.35 us 277632 >> FSYNC >> 0.00 0.00 us 0.00 us 0.00 us 365 >> UPCALL >> >> Duration: 1132 seconds >> Data Read: 668951066 bytes >> Data Written: 7495356076 bytes >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/G3G5HNX3P2Z... >> > _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/NKMZYE6EC3I...

Hello Derek, We have a 10Gig connection dedicated to the storage network, nothing else is on that switch. On Wed, Feb 23, 2022 at 9:49 AM Derek Atkins <derek(a)ihtfp.com&gt; wrote: > Hi, > > Another question which I don't see answered: What is the underlying > connectivity between the Gluster hosts? > > -derek > > On Wed, February 23, 2022 8:39 am, Alex Morrison wrote: > > Hello Sunil, > > > > [root@ovirt1 ~]# gluster --version > > glusterfs 8.6 > > > > same on all hosts > > > > On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya < > > sheggodu(a)redhat.com&gt; wrote: > > > >> Hi, > >> > >> Which version of gluster is in use? > >> > >> Regards, > >> > >> Sunil kumar Acharya > >> > >> Red Hat > >> > >> <https://www.redhat.com> > >> > >> T: +91-8067935170 > >> <http://redhatemailsignature-marketing.itos.redhat.com/> > >> > >> <https://red.ht/sig> > >> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > >> > >> > >> > >> On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison > <alex(a)discoverygarden.ca&gt; > >> wrote: > >> > >>> Hello All, > >>> > >>> We have 3 servers with a raid 50 array each, we are having extreme > >>> performance issues with our gluster, writes on gluster seem to take > at > >>> least 3 times longer than on the raid directly. Can this be > improved? > >>> I've > >>> read through several other performance issues threads but have been > >>> unable > >>> to make any improvements > >>> > >>> "gluster volume info" and "gluster volume profile vmstore info" is > >>> below > >>> > >>> > >>> > ========================================================================================================================= > >>> > >>> -Inside Gluster - test took 35+ hours: > >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s > >>> 600G -n 0 -m TEST -f -b -u root > >>> Using uid:0, gid:0. > >>> Writing intelligently...done > >>> Rewriting...done > >>> Reading intelligently...done > >>> start 'em...done...done...done...done...done... > >>> Version 1.98 ------Sequential Output------ --Sequential > Input- > >>> --Random- > >>> -Per Chr- --Block-- -Rewrite- -Per Chr- > --Block-- > >>> --Seeks-- > >>> Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec > %CP > >>> /sec %CP > >>> TEST 600G 35.7m 17 5824k 7 112m > 13 > >>> 182.7 6 > >>> Latency 5466ms 12754ms 3499ms > >>> 1589ms > >>> > >>> > >>> > 1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,,,,,,,,,,,,,,,,,5466ms,12754ms,,3499ms,1589ms,,,,,, > >>> > >>> > >>> > ========================================================================================================================= > >>> > >>> -Outside Gluster - test took 18 minutes: > >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s > >>> 600G -n 0 -m TEST -f -b -u root > >>> Using uid:0, gid:0. > >>> Writing intelligently...done > >>> Rewriting...done > >>> Reading intelligently...done > >>> start 'em...done...done...done...done...done... > >>> Version 1.98 ------Sequential Output------ --Sequential > Input- > >>> --Random- > >>> -Per Chr- --Block-- -Rewrite- -Per Chr- > --Block-- > >>> --Seeks-- > >>> Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec > %CP > >>> /sec %CP > >>> TEST 600G 567m 78 149m 30 307m > 37 > >>> 83.0 57 > >>> Latency 205ms 4630ms 1450ms > >>> 679ms > >>> > >>> > >>> > 1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,,,,,,,,,,,,,,,,,205ms,4630ms,,1450ms,679ms,,,,,, > >>> > >>> > >>> > ========================================================================================================================= > >>> > >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume > info > >>> Volume Name: engine > >>> Type: Replicate > >>> Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b > >>> Status: Started > >>> Snapshot Count: 0 > >>> Number of Bricks: 1 x 3 = 3 > >>> Transport-type: tcp > >>> Bricks: > >>> Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine > >>> Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine > >>> Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine > >>> Options Reconfigured: > >>> cluster.granular-entry-heal: enable > >>> performance.strict-o-direct: on > >>> network.ping-timeout: 30 > >>> storage.owner-gid: 36 > >>> storage.owner-uid: 36 > >>> server.event-threads: 4 > >>> client.event-threads: 4 > >>> cluster.choose-local: off > >>> user.cifs: off > >>> features.shard: on > >>> cluster.shd-wait-qlength: 10000 > >>> cluster.shd-max-threads: 8 > >>> cluster.locking-scheme: granular > >>> cluster.data-self-heal-algorithm: full > >>> cluster.server-quorum-type: server > >>> cluster.quorum-type: auto > >>> cluster.eager-lock: enable > >>> network.remote-dio: off > >>> performance.low-prio-threads: 32 > >>> performance.io-cache: off > >>> performance.read-ahead: off > >>> performance.quick-read: off > >>> transport.address-family: inet > >>> storage.fips-mode-rchecksum: on > >>> nfs.disable: on > >>> performance.client-io-threads: on > >>> diagnostics.latency-measurement: on > >>> diagnostics.count-fop-hits: on > >>> > >>> Volume Name: vmstore > >>> Type: Replicate > >>> Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753 > >>> Status: Started > >>> Snapshot Count: 0 > >>> Number of Bricks: 1 x 3 = 3 > >>> Transport-type: tcp > >>> Bricks: > >>> Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore > >>> Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore > >>> Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore > >>> Options Reconfigured: > >>> cluster.granular-entry-heal: enable > >>> performance.strict-o-direct: on > >>> network.ping-timeout: 20 > >>> storage.owner-gid: 36 > >>> storage.owner-uid: 36 > >>> server.event-threads: 4 > >>> client.event-threads: 4 > >>> cluster.choose-local: off > >>> user.cifs: off > >>> features.shard: on > >>> cluster.shd-wait-qlength: 10000 > >>> cluster.shd-max-threads: 8 > >>> cluster.locking-scheme: granular > >>> cluster.data-self-heal-algorithm: full > >>> cluster.server-quorum-type: server > >>> cluster.quorum-type: auto > >>> cluster.eager-lock: enable > >>> network.remote-dio: off > >>> performance.low-prio-threads: 32 > >>> performance.io-cache: off > >>> performance.read-ahead: off > >>> performance.quick-read: off > >>> transport.address-family: inet > >>> storage.fips-mode-rchecksum: on > >>> nfs.disable: on > >>> performance.client-io-threads: on > >>> diagnostics.latency-measurement: on > >>> diagnostics.count-fop-hits: on > >>> server.tcp-user-timeout: 20 > >>> server.keepalive-time: 10 > >>> server.keepalive-interval: 2 > >>> server.keepalive-count: 5 > >>> cluster.lookup-optimize: off > >>> > >>> > >>> > ========================================================================================================================= > >>> > >>> [root@ovirt1 ~]# gluster volume profile vmstore info > >>> Brick: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore > >>> --------------------------------------------------------- > >>> Cumulative Stats: > >>> Block Size: 1b+ 4b+ > >>> 8b+ > >>> No. of Reads: 0 0 > >>> 0 > >>> No. of Writes: 4021 2 > >>> 64 > >>> > >>> Block Size: 64b+ 128b+ > >>> 256b+ > >>> No. of Reads: 0 134 > >>> 405 > >>> No. of Writes: 2 32 > >>> 119 > >>> > >>> Block Size: 512b+ 1024b+ > >>> 2048b+ > >>> No. of Reads: 2890 2617 > >>> 3059 > >>> No. of Writes: 113881 93551 > >>> 131776 > >>> > >>> Block Size: 4096b+ 8192b+ > >>> 16384b+ > >>> No. of Reads: 1096597 164931 > >>> 181569 > >>> No. of Writes: 41091001 20499776 > >>> 3230472 > >>> > >>> Block Size: 32768b+ 65536b+ > >>> 131072b+ > >>> No. of Reads: 212970 96574 > >>> 244017 > >>> No. of Writes: 1304688 991154 > >>> 3412866 > >>> > >>> Block Size: 262144b+ 524288b+ > >>> 1048576b+ > >>> No. of Reads: 0 0 > >>> 0 > >>> No. of Writes: 15 18 > >>> 65166 > >>> > >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls > >>> Fop > >>> --------- ----------- ----------- ----------- ------------ > >>> ---- > >>> 0.00 0.00 us 0.00 us 0.00 us 6396 > >>> FORGET > >>> 0.00 0.00 us 0.00 us 0.00 us 189759 > >>> RELEASE > >>> 0.00 0.00 us 0.00 us 0.00 us 31292 > >>> RELEASEDIR > >>> 0.00 159.99 us 44.37 us 278.66 us 79 > >>> STAT > >>> 0.00 364.42 us 174.87 us 644.45 us 96 > >>> LINK > >>> 0.00 2714.10 us 203.53 us 47397.28 us 26 > >>> RMDIR > >>> 0.00 688.26 us 72.39 us 53857.63 us 186 > >>> SETXATTR > >>> 0.00 974.01 us 271.02 us 27298.52 us 176 > >>> RENAME > >>> 0.00 5254.64 us 159.48 us 84454.46 us 35 > >>> DISCARD > >>> 0.00 1133.84 us 86.47 us 87096.92 us 186 > >>> REMOVEXATTR > >>> 0.00 7907.47 us 221.82 us 182276.21 us 29 > >>> FALLOCATE > >>> 0.00 8304.51 us 311.89 us 42725.81 us 34 > >>> MKDIR > >>> 0.00 5758.19 us 296.39 us 21950.99 us 81 > >>> TRUNCATE > >>> 0.00 401.16 us 23.68 us 132668.93 us 3621 > >>> LK > >>> 0.00 545.48 us 14.22 us 1076517.76 us 6796 > >>> READDIR > >>> 0.00 2250.78 us 336.61 us 1071764.61 us 4257 > >>> CREATE > >>> 0.00 531.76 us 1.35 us 1723046.94 us 31292 > >>> OPENDIR > >>> 0.01 2058.67 us 196.74 us 1587752.27 us 21986 > >>> MKNOD > >>> 0.01 1241.28 us 22.61 us 1548642.70 us 52674 > >>> READDIRP > >>> 0.01 67508.32 us 100.01 us 3002069.81 us 1100 > >>> FTRUNCATE > >>> 0.01 412.74 us 10.94 us 5111334.35 us 209731 > >>> FLUSH > >>> 0.02 78668.59 us 183.06 us 3012629.87 us 1222 > >>> SETATTR > >>> 0.03 761.13 us 21.13 us 4611854.15 us 249094 > >>> STATFS > >>> 0.03 1132.33 us 45.51 us 3159531.87 us 185654 > >>> OPEN > >>> 0.03 2010.33 us 14.43 us 3158098.85 us 107824 > >>> GETXATTR > >>> 0.04 2765.29 us 61.38 us 3159208.13 us 100484 > >>> XATTROP > >>> 0.13 1367.04 us 51.46 us 8450420.87 us 595299 > >>> FSTAT > >>> 0.46 157440.90 us 47.62 us 13405712.93 us 18218 > >>> UNLINK > >>> 0.60 1274.67 us 19.60 us 11985005.14 us 2980806 > >>> LOOKUP > >>> 1.67 16645.99 us 12.80 us 6869488.64 us 631331 > >>> ENTRYLK > >>> 4.10 7504.86 us 39.52 us 11749528.82 us 3430506 > >>> READ > >>> 4.31 56942.36 us 16.57 us 23654127.65 us 475563 > >>> INODELK > >>> 6.09 660.89 us 11.02 us 20571410.32 us 57908331 > >>> FINODELK > >>> 20.59 1823.35 us 63.26 us 23639762.45 us 70934143 > >>> WRITE > >>> 25.70 2680.82 us 35.37 us 13636394.75 us 60223005 > >>> FXATTROP > >>> 36.14 3164.86 us 52.35 us 16805886.48 us 71739477 > >>> FSYNC > >>> 0.00 0.00 us 0.00 us 0.00 us 78458 > >>> UPCALL > >>> > >>> Duration: 404667 seconds > >>> Data Read: 59850935724 bytes > >>> Data Written: 1070158035683 bytes > >>> > >>> Interval 43 Stats: > >>> Block Size: 1b+ 256b+ > >>> 512b+ > >>> No. of Reads: 0 14 > >>> 3 > >>> No. of Writes: 12 4 > >>> 252 > >>> > >>> Block Size: 1024b+ 2048b+ > >>> 4096b+ > >>> No. of Reads: 0 0 > >>> 8105 > >>> No. of Writes: 100 34 > >>> 147355 > >>> > >>> Block Size: 8192b+ 16384b+ > >>> 32768b+ > >>> No. of Reads: 1559 1630 > >>> 1494 > >>> No. of Writes: 86013 17318 > >>> 8687 > >>> > >>> Block Size: 65536b+ 131072b+ > >>> No. of Reads: 532 764 > >>> No. of Writes: 8617 36317 > >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls > >>> Fop > >>> --------- ----------- ----------- ----------- ------------ > >>> ---- > >>> 0.00 0.00 us 0.00 us 0.00 us 16 > >>> FORGET > >>> 0.00 0.00 us 0.00 us 0.00 us 665 > >>> RELEASE > >>> 0.00 0.00 us 0.00 us 0.00 us 52 > >>> RELEASEDIR > >>> 0.00 181.75 us 181.75 us 181.75 us 1 > >>> STAT > >>> 0.00 914.55 us 858.55 us 970.56 us 2 > >>> TRUNCATE > >>> 0.00 792.11 us 681.24 us 1051.32 us 4 > >>> RENAME > >>> 0.00 374.44 us 31.86 us 792.10 us 20 > >>> READDIR > >>> 0.00 399.48 us 306.97 us 562.69 us 28 > >>> UNLINK > >>> 0.00 752.40 us 636.14 us 907.29 us 16 > >>> CREATE > >>> 0.00 196.62 us 33.27 us 2586.77 us 336 > >>> GETXATTR > >>> 0.00 1107.98 us 288.51 us 29692.53 us 80 > >>> MKNOD > >>> 0.00 1074.80 us 74.41 us 2121.03 us 126 > >>> READDIRP > >>> 0.00 3119.52 us 3.17 us 157121.44 us 52 > >>> OPENDIR > >>> 0.01 972.14 us 21.72 us 329342.21 us 635 > >>> FLUSH > >>> 0.03 2122.15 us 31.97 us 377538.16 us 732 > >>> STATFS > >>> 0.06 4478.89 us 66.09 us 521289.10 us 649 > >>> OPEN > >>> 0.10 10273.00 us 88.19 us 478524.77 us 472 > >>> XATTROP > >>> 0.16 6969.92 us 16.14 us 385156.98 us 1138 > >>> ENTRYLK > >>> 0.20 4635.40 us 64.29 us 1098319.92 us 2097 > >>> FSTAT > >>> 0.65 4098.07 us 46.85 us 919617.66 us 7797 > >>> LOOKUP > >>> 2.36 87465.33 us 34.75 us 1224052.07 us 1317 > >>> INODELK > >>> 4.37 13897.55 us 57.89 us 1984970.27 us 15336 > >>> READ > >>> 5.24 1265.46 us 16.22 us 1638678.32 us 202165 > >>> FINODELK > >>> 21.05 3370.28 us 144.44 us 2264130.49 us 304724 > >>> WRITE > >>> 31.00 6411.68 us 39.84 us 2190355.42 us 235969 > >>> FXATTROP > >>> 34.75 6107.95 us 60.26 us 2270252.44 us 277643 > >>> FSYNC > >>> 0.00 0.00 us 0.00 us 0.00 us 368 > >>> UPCALL > >>> > >>> Duration: 1132 seconds > >>> Data Read: 292367566 bytes > >>> Data Written: 7495339692 bytes > >>> > >>> Brick: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore > >>> --------------------------------------------------------- > >>> Cumulative Stats: > >>> Block Size: 1b+ 4b+ > >>> 8b+ > >>> No. of Reads: 0 0 > >>> 0 > >>> No. of Writes: 4802 2 > >>> 64 > >>> > >>> Block Size: 64b+ 128b+ > >>> 256b+ > >>> No. of Reads: 0 138 > >>> 828 > >>> No. of Writes: 2 32 > >>> 131 > >>> > >>> Block Size: 512b+ 1024b+ > >>> 2048b+ > >>> No. of Reads: 3595 4590 > >>> 6161 > >>> No. of Writes: 128976 99680 > >>> 132413 > >>> > >>> Block Size: 4096b+ 8192b+ > >>> 16384b+ > >>> No. of Reads: 930031 128995 > >>> 150468 > >>> No. of Writes: 48357551 23957030 > >>> 3689474 > >>> > >>> Block Size: 32768b+ 65536b+ > >>> 131072b+ > >>> No. of Reads: 154039 71949 > >>> 790010 > >>> No. of Writes: 1438859 1052045 > >>> 3715427 > >>> > >>> Block Size: 262144b+ 524288b+ > >>> 1048576b+ > >>> No. of Reads: 9 11 > >>> 40269 > >>> No. of Writes: 5 20 > >>> 50757 > >>> > >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls > >>> Fop > >>> --------- ----------- ----------- ----------- ------------ > >>> ---- > >>> 0.00 0.00 us 0.00 us 0.00 us 7629 > >>> FORGET > >>> 0.00 0.00 us 0.00 us 0.00 us 228577 > >>> RELEASE > >>> 0.00 0.00 us 0.00 us 0.00 us 41524 > >>> RELEASEDIR > >>> 0.00 374.22 us 219.35 us 574.87 us 96 > >>> LINK > >>> 0.00 1712.22 us 164.90 us 22337.79 us 41 > >>> DISCARD > >>> 0.00 2840.25 us 245.24 us 46839.63 us 26 > >>> RMDIR > >>> 0.00 3660.92 us 229.86 us 86252.56 us 35 > >>> FALLOCATE > >>> 0.00 876.48 us 297.53 us 16685.63 us 188 > >>> RENAME > >>> 0.00 6660.10 us 379.21 us 52962.79 us 34 > >>> MKDIR > >>> 0.00 957.48 us 80.36 us 218059.88 us 355 > >>> REMOVEXATTR > >>> 0.00 1234.85 us 76.50 us 322955.27 us 355 > >>> SETXATTR > >>> 0.00 2326.62 us 84.07 us 250041.25 us 196 > >>> STAT > >>> 0.00 9660.45 us 289.50 us 295106.01 us 87 > >>> TRUNCATE > >>> 0.00 486.10 us 24.12 us 869966.79 us 5260 > >>> LK > >>> 0.00 453.69 us 18.32 us 426653.18 us 8135 > >>> READDIR > >>> 0.00 1832.38 us 296.61 us 261299.81 us 5050 > >>> CREATE > >>> 0.00 386.77 us 1.31 us 1300836.12 us 41524 > >>> OPENDIR > >>> 0.00 712.06 us 29.85 us 1441115.02 us 34874 > >>> READDIRP > >>> 0.01 1469.51 us 231.67 us 440065.71 us 22845 > >>> MKNOD > >>> 0.01 44229.36 us 75.58 us 1803963.42 us 871 > >>> FTRUNCATE > >>> 0.01 84433.20 us 149.47 us 3614869.24 us 1001 > >>> SETATTR > >>> 0.02 418.11 us 13.39 us 3515000.81 us 243820 > >>> FLUSH > >>> 0.03 1108.47 us 14.77 us 1658743.41 us 130647 > >>> GETXATTR > >>> 0.03 562.86 us 26.74 us 5043949.33 us 297799 > >>> STATFS > >>> 0.03 845.50 us 48.16 us 1998938.03 us 223680 > >>> OPEN > >>> 0.05 2236.13 us 56.08 us 5295998.22 us 120682 > >>> XATTROP > >>> 0.16 890.82 us 43.51 us 3653292.88 us 1012585 > >>> FSTAT > >>> 0.34 2974.20 us 12.92 us 7782497.74 us 642555 > >>> ENTRYLK > >>> 0.47 135831.21 us 70.35 us 11033867.90 us 19800 > >>> UNLINK > >>> 0.57 954.05 us 21.50 us 4553393.38 us 3414605 > >>> LOOKUP > >>> 1.79 14461.54 us 13.45 us 32841452.98 us 702915 > >>> INODELK > >>> 5.14 8014.78 us 40.70 us 5439109.56 us 3644063 > >>> READ > >>> 5.37 443.04 us 11.53 us 32863652.53 us 68909131 > >>> FINODELK > >>> 22.31 1780.31 us 33.59 us 11318712.62 us 71235991 > >>> FXATTROP > >>> 22.83 1571.16 us 74.86 us 32615055.19 us 82622840 > >>> WRITE > >>> 40.84 2762.75 us 52.77 us 8859115.35 us 84039509 > >>> FSYNC > >>> 0.00 0.00 us 0.00 us 0.00 us 95492 > >>> UPCALL > >>> > >>> Duration: 484169 seconds > >>> Data Read: 167149718723 bytes > >>> Data Written: 1177141649872 bytes > >>> > >>> Interval 43 Stats: > >>> Block Size: 1b+ 256b+ > >>> 512b+ > >>> No. of Reads: 0 6 > >>> 4 > >>> No. of Writes: 12 4 > >>> 252 > >>> > >>> Block Size: 1024b+ 2048b+ > >>> 4096b+ > >>> No. of Reads: 0 0 > >>> 5668 > >>> No. of Writes: 100 34 > >>> 147357 > >>> > >>> Block Size: 8192b+ 16384b+ > >>> 32768b+ > >>> No. of Reads: 1178 783 > >>> 1215 > >>> No. of Writes: 86014 17318 > >>> 8687 > >>> > >>> Block Size: 65536b+ 131072b+ > >>> No. of Reads: 264 4109 > >>> No. of Writes: 8617 36317 > >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls > >>> Fop > >>> --------- ----------- ----------- ----------- ------------ > >>> ---- > >>> 0.00 0.00 us 0.00 us 0.00 us 16 > >>> FORGET > >>> 0.00 0.00 us 0.00 us 0.00 us 665 > >>> RELEASE > >>> 0.00 0.00 us 0.00 us 0.00 us 52 > >>> RELEASEDIR > >>> 0.00 866.18 us 849.57 us 882.78 us 2 > >>> TRUNCATE > >>> 0.00 1016.14 us 868.35 us 1182.48 us 4 > >>> RENAME > >>> 0.00 455.65 us 41.13 us 1679.37 us 20 > >>> READDIR > >>> 0.00 373.33 us 173.54 us 538.40 us 28 > >>> UNLINK > >>> 0.00 722.01 us 635.91 us 853.71 us 16 > >>> CREATE > >>> 0.00 335.52 us 38.15 us 1381.97 us 54 > >>> READDIRP > >>> 0.00 39218.92 us 214.34 us 78223.50 us 2 > >>> STAT > >>> 0.00 1730.77 us 2.59 us 84964.69 us 52 > >>> OPENDIR > >>> 0.00 1304.96 us 393.64 us 29724.09 us 80 > >>> MKNOD > >>> 0.01 422.75 us 22.85 us 134336.60 us 635 > >>> FLUSH > >>> 0.01 699.21 us 37.54 us 137813.64 us 732 > >>> STATFS > >>> 0.01 468.63 us 23.71 us 260961.83 us 1141 > >>> ENTRYLK > >>> 0.04 2041.83 us 69.80 us 163743.98 us 649 > >>> OPEN > >>> 0.08 1253.65 us 72.52 us 290508.15 us 2354 > >>> FSTAT > >>> 0.09 6913.19 us 69.01 us 212479.24 us 472 > >>> XATTROP > >>> 0.13 3558.26 us 32.70 us 195896.10 us 1317 > >>> INODELK > >>> 0.13 9793.37 us 36.11 us 212755.58 us 499 > >>> GETXATTR > >>> 0.34 1615.15 us 55.42 us 711310.28 us 7797 > >>> LOOKUP > >>> 4.92 13025.92 us 59.83 us 1061585.06 us 13884 > >>> READ > >>> 5.26 955.95 us 16.81 us 1069148.49 us 202174 > >>> FINODELK > >>> 23.28 2809.44 us 137.21 us 1134076.13 us 304713 > >>> WRITE > >>> 26.46 4124.05 us 43.30 us 1142167.27 us 235954 > >>> FXATTROP > >>> 39.23 5195.62 us 65.47 us 1250469.35 us 277632 > >>> FSYNC > >>> 0.00 0.00 us 0.00 us 0.00 us 365 > >>> UPCALL > >>> > >>> Duration: 1132 seconds > >>> Data Read: 668951066 bytes > >>> Data Written: 7495356076 bytes > >>> _______________________________________________ > >>> Users mailing list -- users(a)ovirt.org > >>> To unsubscribe send an email to users-leave(a)ovirt.org > >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html > >>> oVirt Code of Conduct: > >>> https://www.ovirt.org/community/about/community-guidelines/ > >>> List Archives: > >>> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/G3G5HNX3P2Z... > >>> > >> > > _______________________________________________ > > Users mailing list -- users(a)ovirt.org > > To unsubscribe send an email to users-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/privacy-policy.html > > oVirt Code of Conduct: > > https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: > > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/NKMZYE6EC3I... > > > > > -- > Derek Atkins 617-623-3745 > derek(a)ihtfp.com www.ihtfp.com > Computer and Internet Security Consultant > >

Regards, Sunil On Wed, Feb 23, 2022 at 7:34 PM Derek Atkins <derek(a)ihtfp.com&gt; wrote: > Have you verified that you're actually getting 10Gbps between the hosts? > > -derek > > On Wed, February 23, 2022 9:02 am, Alex Morrison wrote: > > Hello Derek, > > > > We have a 10Gig connection dedicated to the storage network, nothing > else > > is on that switch. > > > > On Wed, Feb 23, 2022 at 9:49 AM Derek Atkins <derek(a)ihtfp.com&gt; wrote: > > > >> Hi, > >> > >> Another question which I don't see answered: What is the underlying > >> connectivity between the Gluster hosts? > >> > >> -derek > >> > >> On Wed, February 23, 2022 8:39 am, Alex Morrison wrote: > >> > Hello Sunil, > >> > > >> > [root@ovirt1 ~]# gluster --version > >> > glusterfs 8.6 > >> > > >> > same on all hosts > Latest Release-10.1( https://lists.gluster.org/pipermail/gluster-users/2022-February/039761.html) has some performance fixes which should help in this situation compared to the older gluster bits. > >> > > >> > On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya < > >> > sheggodu(a)redhat.com&gt; wrote: > >> > > >> >> Hi, > >> >> > >> >> Which version of gluster is in use? > >> >> > >> >> Regards, > >> >> > >> >> Sunil kumar Acharya > >> >> > >> >> Red Hat > >> >> > >> >> <https://www.redhat.com> > >> >> > >> >> T: +91-8067935170 > >> >> <http://redhatemailsignature-marketing.itos.redhat.com/> > >> >> > >> >> <https://red.ht/sig> > >> >> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > >> >> > >> >> > >> >> > >> >> On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison > >> <alex(a)discoverygarden.ca&gt; > >> >> wrote: > >> >> > >> >>> Hello All, > >> >>> > >> >>> We have 3 servers with a raid 50 array each, we are having extreme > >> >>> performance issues with our gluster, writes on gluster seem to take > >> at > >> >>> least 3 times longer than on the raid directly. Can this be > >> improved? > >> >>> I've > >> >>> read through several other performance issues threads but have been > >> >>> unable > >> >>> to make any improvements > >> >>> > >> >>> "gluster volume info" and "gluster volume profile vmstore info" is > >> >>> below > >> >>> > >> >>> > >> >>> > >> > ========================================================================================================================= > >> >>> > >> >>> -Inside Gluster - test took 35+ hours: > >> >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . > -s > >> >>> 600G -n 0 -m TEST -f -b -u root > >> >>> Using uid:0, gid:0. > >> >>> Writing intelligently...done > >> >>> Rewriting...done > >> >>> Reading intelligently...done > >> >>> start 'em...done...done...done...done...done... > >> >>> Version 1.98 ------Sequential Output------ --Sequential > >> Input- > >> >>> --Random- > >> >>> -Per Chr- --Block-- -Rewrite- -Per Chr- > >> --Block-- > >> >>> --Seeks-- > >> >>> Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec > >> %CP > >> >>> /sec %CP > >> >>> TEST 600G 35.7m 17 5824k 7 112m > >> 13 > >> >>> 182.7 6 > >> >>> Latency 5466ms 12754ms 3499ms > >> >>> 1589ms > >> >>> > >> >>> > >> >>> > >> > 1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,,,,,,,,,,,,,,,,,5466ms,12754ms,,3499ms,1589ms,,,,,, > >> >>> > >> >>> > >> >>> > >> > ========================================================================================================================= > >> >>> > >> >>> -Outside Gluster - test took 18 minutes: > >> >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . > -s > >> >>> 600G -n 0 -m TEST -f -b -u root > >> >>> Using uid:0, gid:0. > >> >>> Writing intelligently...done > >> >>> Rewriting...done > >> >>> Reading intelligently...done > >> >>> start 'em...done...done...done...done...done... > >> >>> Version 1.98 ------Sequential Output------ --Sequential > >> Input- > >> >>> --Random- > >> >>> -Per Chr- --Block-- -Rewrite- -Per Chr- > >> --Block-- > >> >>> --Seeks-- > >> >>> Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec > >> %CP > >> >>> /sec %CP > >> >>> TEST 600G 567m 78 149m 30 307m > >> 37 > >> >>> 83.0 57 > >> >>> Latency 205ms 4630ms 1450ms > >> >>> 679ms > >> >>> > >> >>> > >> >>> > >> > 1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,,,,,,,,,,,,,,,,,205ms,4630ms,,1450ms,679ms,,,,,, > >> >>> > >> >>> > >> >>> > >> > ========================================================================================================================= > >> >>> > >> >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume > >> info > >> >>> Volume Name: engine > >> >>> Type: Replicate > >> >>> Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b > >> >>> Status: Started > >> >>> Snapshot Count: 0 > >> >>> Number of Bricks: 1 x 3 = 3 > >> >>> Transport-type: tcp > >> >>> Bricks: > >> >>> Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine > >> >>> Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine > >> >>> Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine > >> >>> Options Reconfigured: > >> >>> cluster.granular-entry-heal: enable > >> >>> performance.strict-o-direct: on > >> >>> network.ping-timeout: 30 > >> >>> storage.owner-gid: 36 > >> >>> storage.owner-uid: 36 > >> >>> server.event-threads: 4 > >> >>> client.event-threads: 4 > >> >>> cluster.choose-local: off > >> >>> user.cifs: off > >> >>> features.shard: on > >> >>> cluster.shd-wait-qlength: 10000 > >> >>> cluster.shd-max-threads: 8 > >> >>> cluster.locking-scheme: granular > >> >>> cluster.data-self-heal-algorithm: full > >> >>> cluster.server-quorum-type: server > >> >>> cluster.quorum-type: auto > >> >>> cluster.eager-lock: enable > >> >>> network.remote-dio: off > >> >>> performance.low-prio-threads: 32 > >> >>> performance.io-cache: off > >> >>> performance.read-ahead: off > >> >>> performance.quick-read: off > >> >>> transport.address-family: inet > >> >>> storage.fips-mode-rchecksum: on > >> >>> nfs.disable: on > >> >>> performance.client-io-threads: on > >> >>> diagnostics.latency-measurement: on > >> >>> diagnostics.count-fop-hits: on > >> >>> > >> >>> Volume Name: vmstore > >> >>> Type: Replicate > >> >>> Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753 > >> >>> Status: Started > >> >>> Snapshot Count: 0 > >> >>> Number of Bricks: 1 x 3 = 3 > >> >>> Transport-type: tcp > >> >>> Bricks: > >> >>> Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore > >> >>> Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore > >> >>> Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore > >> >>> Options Reconfigured: > >> >>> cluster.granular-entry-heal: enable > >> >>> performance.strict-o-direct: on > >> >>> network.ping-timeout: 20 > >> >>> storage.owner-gid: 36 > >> >>> storage.owner-uid: 36 > >> >>> server.event-threads: 4 > >> >>> client.event-threads: 4 > >> >>> cluster.choose-local: off > >> >>> user.cifs: off > >> >>> features.shard: on > >> >>> cluster.shd-wait-qlength: 10000 > >> >>> cluster.shd-max-threads: 8 > >> >>> cluster.locking-scheme: granular > >> >>> cluster.data-self-heal-algorithm: full > >> >>> cluster.server-quorum-type: server > >> >>> cluster.quorum-type: auto > >> >>> cluster.eager-lock: enable > >> >>> network.remote-dio: off > >> >>> performance.low-prio-threads: 32 > >> >>> performance.io-cache: off > >> >>> performance.read-ahead: off > >> >>> performance.quick-read: off > >> >>> transport.address-family: inet > >> >>> storage.fips-mode-rchecksum: on > >> >>> nfs.disable: on > >> >>> performance.client-io-threads: on > >> >>> diagnostics.latency-measurement: on > >> >>> diagnostics.count-fop-hits: on > >> >>> server.tcp-user-timeout: 20 > >> >>> server.keepalive-time: 10 > >> >>> server.keepalive-interval: 2 > >> >>> server.keepalive-count: 5 > >> >>> cluster.lookup-optimize: off > >> >>> > >> >>> > >> >>> > >> > ========================================================================================================================= > >> >>> > >> >>> [root@ovirt1 ~]# gluster volume profile vmstore info > >> >>> Brick: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore > >> >>> --------------------------------------------------------- > >> >>> Cumulative Stats: > >> >>> Block Size: 1b+ 4b+ > >> >>> 8b+ > >> >>> No. of Reads: 0 0 > >> >>> 0 > >> >>> No. of Writes: 4021 2 > >> >>> 64 > >> >>> > >> >>> Block Size: 64b+ 128b+ > >> >>> 256b+ > >> >>> No. of Reads: 0 134 > >> >>> 405 > >> >>> No. of Writes: 2 32 > >> >>> 119 > >> >>> > >> >>> Block Size: 512b+ 1024b+ > >> >>> 2048b+ > >> >>> No. of Reads: 2890 2617 > >> >>> 3059 > >> >>> No. of Writes: 113881 93551 > >> >>> 131776 > >> >>> > >> >>> Block Size: 4096b+ 8192b+ > >> >>> 16384b+ > >> >>> No. of Reads: 1096597 164931 > >> >>> 181569 > >> >>> No. of Writes: 41091001 20499776 > >> >>> 3230472 > >> >>> > >> >>> Block Size: 32768b+ 65536b+ > >> >>> 131072b+ > >> >>> No. of Reads: 212970 96574 > >> >>> 244017 > >> >>> No. of Writes: 1304688 991154 > >> >>> 3412866 > >> >>> > >> >>> Block Size: 262144b+ 524288b+ > >> >>> 1048576b+ > >> >>> No. of Reads: 0 0 > >> >>> 0 > >> >>> No. of Writes: 15 18 > >> >>> 65166 > >> >>> > >> >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls > >> >>> Fop > >> >>> --------- ----------- ----------- ----------- ------------ > >> >>> ---- > >> >>> 0.00 0.00 us 0.00 us 0.00 us 6396 > >> >>> FORGET > >> >>> 0.00 0.00 us 0.00 us 0.00 us 189759 > >> >>> RELEASE > >> >>> 0.00 0.00 us 0.00 us 0.00 us 31292 > >> >>> RELEASEDIR > >> >>> 0.00 159.99 us 44.37 us 278.66 us 79 > >> >>> STAT > >> >>> 0.00 364.42 us 174.87 us 644.45 us 96 > >> >>> LINK > >> >>> 0.00 2714.10 us 203.53 us 47397.28 us 26 > >> >>> RMDIR > >> >>> 0.00 688.26 us 72.39 us 53857.63 us 186 > >> >>> SETXATTR > >> >>> 0.00 974.01 us 271.02 us 27298.52 us 176 > >> >>> RENAME > >> >>> 0.00 5254.64 us 159.48 us 84454.46 us 35 > >> >>> DISCARD > >> >>> 0.00 1133.84 us 86.47 us 87096.92 us 186 > >> >>> REMOVEXATTR > >> >>> 0.00 7907.47 us 221.82 us 182276.21 us 29 > >> >>> FALLOCATE > >> >>> 0.00 8304.51 us 311.89 us 42725.81 us 34 > >> >>> MKDIR > >> >>> 0.00 5758.19 us 296.39 us 21950.99 us 81 > >> >>> TRUNCATE > >> >>> 0.00 401.16 us 23.68 us 132668.93 us 3621 > >> >>> LK > >> >>> 0.00 545.48 us 14.22 us 1076517.76 us 6796 > >> >>> READDIR > >> >>> 0.00 2250.78 us 336.61 us 1071764.61 us 4257 > >> >>> CREATE > >> >>> 0.00 531.76 us 1.35 us 1723046.94 us 31292 > >> >>> OPENDIR > >> >>> 0.01 2058.67 us 196.74 us 1587752.27 us 21986 > >> >>> MKNOD > >> >>> 0.01 1241.28 us 22.61 us 1548642.70 us 52674 > >> >>> READDIRP > >> >>> 0.01 67508.32 us 100.01 us 3002069.81 us 1100 > >> >>> FTRUNCATE > >> >>> 0.01 412.74 us 10.94 us 5111334.35 us 209731 > >> >>> FLUSH > >> >>> 0.02 78668.59 us 183.06 us 3012629.87 us 1222 > >> >>> SETATTR > >> >>> 0.03 761.13 us 21.13 us 4611854.15 us 249094 > >> >>> STATFS > >> >>> 0.03 1132.33 us 45.51 us 3159531.87 us 185654 > >> >>> OPEN > >> >>> 0.03 2010.33 us 14.43 us 3158098.85 us 107824 > >> >>> GETXATTR > >> >>> 0.04 2765.29 us 61.38 us 3159208.13 us 100484 > >> >>> XATTROP > >> >>> 0.13 1367.04 us 51.46 us 8450420.87 us 595299 > >> >>> FSTAT > >> >>> 0.46 157440.90 us 47.62 us 13405712.93 us > 18218 > >> >>> UNLINK > >> >>> 0.60 1274.67 us 19.60 us 11985005.14 us > 2980806 > >> >>> LOOKUP > >> >>> 1.67 16645.99 us 12.80 us 6869488.64 us 631331 > >> >>> ENTRYLK > >> >>> 4.10 7504.86 us 39.52 us 11749528.82 us > 3430506 > >> >>> READ > >> >>> 4.31 56942.36 us 16.57 us 23654127.65 us > 475563 > >> >>> INODELK > >> >>> 6.09 660.89 us 11.02 us 20571410.32 us > 57908331 > >> >>> FINODELK > >> >>> 20.59 1823.35 us 63.26 us 23639762.45 us > 70934143 > >> >>> WRITE > >> >>> 25.70 2680.82 us 35.37 us 13636394.75 us > 60223005 > >> >>> FXATTROP > >> >>> 36.14 3164.86 us 52.35 us 16805886.48 us > 71739477 > >> >>> FSYNC > >> >>> 0.00 0.00 us 0.00 us 0.00 us 78458 > >> >>> UPCALL > >> >>> > >> >>> Duration: 404667 seconds > >> >>> Data Read: 59850935724 bytes > >> >>> Data Written: 1070158035683 bytes > >> >>> > >> >>> Interval 43 Stats: > >> >>> Block Size: 1b+ 256b+ > >> >>> 512b+ > >> >>> No. of Reads: 0 14 > >> >>> 3 > >> >>> No. of Writes: 12 4 > >> >>> 252 > >> >>> > >> >>> Block Size: 1024b+ 2048b+ > >> >>> 4096b+ > >> >>> No. of Reads: 0 0 > >> >>> 8105 > >> >>> No. of Writes: 100 34 > >> >>> 147355 > >> >>> > >> >>> Block Size: 8192b+ 16384b+ > >> >>> 32768b+ > >> >>> No. of Reads: 1559 1630 > >> >>> 1494 > >> >>> No. of Writes: 86013 17318 > >> >>> 8687 > >> >>> > >> >>> Block Size: 65536b+ 131072b+ > >> >>> No. of Reads: 532 764 > >> >>> No. of Writes: 8617 36317 > >> >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls > >> >>> Fop > >> >>> --------- ----------- ----------- ----------- ------------ > >> >>> ---- > >> >>> 0.00 0.00 us 0.00 us 0.00 us 16 > >> >>> FORGET > >> >>> 0.00 0.00 us 0.00 us 0.00 us 665 > >> >>> RELEASE > >> >>> 0.00 0.00 us 0.00 us 0.00 us 52 > >> >>> RELEASEDIR > >> >>> 0.00 181.75 us 181.75 us 181.75 us 1 > >> >>> STAT > >> >>> 0.00 914.55 us 858.55 us 970.56 us 2 > >> >>> TRUNCATE > >> >>> 0.00 792.11 us 681.24 us 1051.32 us 4 > >> >>> RENAME > >> >>> 0.00 374.44 us 31.86 us 792.10 us 20 > >> >>> READDIR > >> >>> 0.00 399.48 us 306.97 us 562.69 us 28 > >> >>> UNLINK > >> >>> 0.00 752.40 us 636.14 us 907.29 us 16 > >> >>> CREATE > >> >>> 0.00 196.62 us 33.27 us 2586.77 us 336 > >> >>> GETXATTR > >> >>> 0.00 1107.98 us 288.51 us 29692.53 us 80 > >> >>> MKNOD > >> >>> 0.00 1074.80 us 74.41 us 2121.03 us 126 > >> >>> READDIRP > >> >>> 0.00 3119.52 us 3.17 us 157121.44 us 52 > >> >>> OPENDIR > >> >>> 0.01 972.14 us 21.72 us 329342.21 us 635 > >> >>> FLUSH > >> >>> 0.03 2122.15 us 31.97 us 377538.16 us 732 > >> >>> STATFS > >> >>> 0.06 4478.89 us 66.09 us 521289.10 us 649 > >> >>> OPEN > >> >>> 0.10 10273.00 us 88.19 us 478524.77 us 472 > >> >>> XATTROP > >> >>> 0.16 6969.92 us 16.14 us 385156.98 us 1138 > >> >>> ENTRYLK > >> >>> 0.20 4635.40 us 64.29 us 1098319.92 us 2097 > >> >>> FSTAT > >> >>> 0.65 4098.07 us 46.85 us 919617.66 us 7797 > >> >>> LOOKUP > >> >>> 2.36 87465.33 us 34.75 us 1224052.07 us 1317 > >> >>> INODELK > >> >>> 4.37 13897.55 us 57.89 us 1984970.27 us 15336 > >> >>> READ > >> >>> 5.24 1265.46 us 16.22 us 1638678.32 us 202165 > >> >>> FINODELK > >> >>> 21.05 3370.28 us 144.44 us 2264130.49 us 304724 > >> >>> WRITE > >> >>> 31.00 6411.68 us 39.84 us 2190355.42 us 235969 > >> >>> FXATTROP > >> >>> 34.75 6107.95 us 60.26 us 2270252.44 us 277643 > >> >>> FSYNC > >> >>> 0.00 0.00 us 0.00 us 0.00 us 368 > >> >>> UPCALL > >> >>> > >> >>> Duration: 1132 seconds > >> >>> Data Read: 292367566 bytes > >> >>> Data Written: 7495339692 bytes > >> >>> > >> >>> Brick: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore > >> >>> --------------------------------------------------------- > >> >>> Cumulative Stats: > >> >>> Block Size: 1b+ 4b+ > >> >>> 8b+ > >> >>> No. of Reads: 0 0 > >> >>> 0 > >> >>> No. of Writes: 4802 2 > >> >>> 64 > >> >>> > >> >>> Block Size: 64b+ 128b+ > >> >>> 256b+ > >> >>> No. of Reads: 0 138 > >> >>> 828 > >> >>> No. of Writes: 2 32 > >> >>> 131 > >> >>> > >> >>> Block Size: 512b+ 1024b+ > >> >>> 2048b+ > >> >>> No. of Reads: 3595 4590 > >> >>> 6161 > >> >>> No. of Writes: 128976 99680 > >> >>> 132413 > >> >>> > >> >>> Block Size: 4096b+ 8192b+ > >> >>> 16384b+ > >> >>> No. of Reads: 930031 128995 > >> >>> 150468 > >> >>> No. of Writes: 48357551 23957030 > >> >>> 3689474 > >> >>> > >> >>> Block Size: 32768b+ 65536b+ > >> >>> 131072b+ > >> >>> No. of Reads: 154039 71949 > >> >>> 790010 > >> >>> No. of Writes: 1438859 1052045 > >> >>> 3715427 > >> >>> > >> >>> Block Size: 262144b+ 524288b+ > >> >>> 1048576b+ > >> >>> No. of Reads: 9 11 > >> >>> 40269 > >> >>> No. of Writes: 5 20 > >> >>> 50757 > >> >>> > >> >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls > >> >>> Fop > >> >>> --------- ----------- ----------- ----------- ------------ > >> >>> ---- > >> >>> 0.00 0.00 us 0.00 us 0.00 us 7629 > >> >>> FORGET > >> >>> 0.00 0.00 us 0.00 us 0.00 us 228577 > >> >>> RELEASE > >> >>> 0.00 0.00 us 0.00 us 0.00 us 41524 > >> >>> RELEASEDIR > >> >>> 0.00 374.22 us 219.35 us 574.87 us 96 > >> >>> LINK > >> >>> 0.00 1712.22 us 164.90 us 22337.79 us 41 > >> >>> DISCARD > >> >>> 0.00 2840.25 us 245.24 us 46839.63 us 26 > >> >>> RMDIR > >> >>> 0.00 3660.92 us 229.86 us 86252.56 us 35 > >> >>> FALLOCATE > >> >>> 0.00 876.48 us 297.53 us 16685.63 us 188 > >> >>> RENAME > >> >>> 0.00 6660.10 us 379.21 us 52962.79 us 34 > >> >>> MKDIR > >> >>> 0.00 957.48 us 80.36 us 218059.88 us 355 > >> >>> REMOVEXATTR > >> >>> 0.00 1234.85 us 76.50 us 322955.27 us 355 > >> >>> SETXATTR > >> >>> 0.00 2326.62 us 84.07 us 250041.25 us 196 > >> >>> STAT > >> >>> 0.00 9660.45 us 289.50 us 295106.01 us 87 > >> >>> TRUNCATE > >> >>> 0.00 486.10 us 24.12 us 869966.79 us 5260 > >> >>> LK > >> >>> 0.00 453.69 us 18.32 us 426653.18 us 8135 > >> >>> READDIR > >> >>> 0.00 1832.38 us 296.61 us 261299.81 us 5050 > >> >>> CREATE > >> >>> 0.00 386.77 us 1.31 us 1300836.12 us 41524 > >> >>> OPENDIR > >> >>> 0.00 712.06 us 29.85 us 1441115.02 us 34874 > >> >>> READDIRP > >> >>> 0.01 1469.51 us 231.67 us 440065.71 us 22845 > >> >>> MKNOD > >> >>> 0.01 44229.36 us 75.58 us 1803963.42 us 871 > >> >>> FTRUNCATE > >> >>> 0.01 84433.20 us 149.47 us 3614869.24 us 1001 > >> >>> SETATTR > >> >>> 0.02 418.11 us 13.39 us 3515000.81 us 243820 > >> >>> FLUSH > >> >>> 0.03 1108.47 us 14.77 us 1658743.41 us 130647 > >> >>> GETXATTR > >> >>> 0.03 562.86 us 26.74 us 5043949.33 us 297799 > >> >>> STATFS > >> >>> 0.03 845.50 us 48.16 us 1998938.03 us 223680 > >> >>> OPEN > >> >>> 0.05 2236.13 us 56.08 us 5295998.22 us 120682 > >> >>> XATTROP > >> >>> 0.16 890.82 us 43.51 us 3653292.88 us 1012585 > >> >>> FSTAT > >> >>> 0.34 2974.20 us 12.92 us 7782497.74 us 642555 > >> >>> ENTRYLK > >> >>> 0.47 135831.21 us 70.35 us 11033867.90 us > 19800 > >> >>> UNLINK > >> >>> 0.57 954.05 us 21.50 us 4553393.38 us 3414605 > >> >>> LOOKUP > >> >>> 1.79 14461.54 us 13.45 us 32841452.98 us > 702915 > >> >>> INODELK > >> >>> 5.14 8014.78 us 40.70 us 5439109.56 us 3644063 > >> >>> READ > >> >>> 5.37 443.04 us 11.53 us 32863652.53 us > 68909131 > >> >>> FINODELK > >> >>> 22.31 1780.31 us 33.59 us 11318712.62 us > 71235991 > >> >>> FXATTROP > >> >>> 22.83 1571.16 us 74.86 us 32615055.19 us > 82622840 > >> >>> WRITE > >> >>> 40.84 2762.75 us 52.77 us 8859115.35 us 84039509 > >> >>> FSYNC > >> >>> 0.00 0.00 us 0.00 us 0.00 us 95492 > >> >>> UPCALL > >> >>> > >> >>> Duration: 484169 seconds > >> >>> Data Read: 167149718723 bytes > >> >>> Data Written: 1177141649872 bytes > >> >>> > >> >>> Interval 43 Stats: > >> >>> Block Size: 1b+ 256b+ > >> >>> 512b+ > >> >>> No. of Reads: 0 6 > >> >>> 4 > >> >>> No. of Writes: 12 4 > >> >>> 252 > >> >>> > >> >>> Block Size: 1024b+ 2048b+ > >> >>> 4096b+ > >> >>> No. of Reads: 0 0 > >> >>> 5668 > >> >>> No. of Writes: 100 34 > >> >>> 147357 > >> >>> > >> >>> Block Size: 8192b+ 16384b+ > >> >>> 32768b+ > >> >>> No. of Reads: 1178 783 > >> >>> 1215 > >> >>> No. of Writes: 86014 17318 > >> >>> 8687 > >> >>> > >> >>> Block Size: 65536b+ 131072b+ > >> >>> No. of Reads: 264 4109 > >> >>> No. of Writes: 8617 36317 > >> >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls > >> >>> Fop > >> >>> --------- ----------- ----------- ----------- ------------ > >> >>> ---- > >> >>> 0.00 0.00 us 0.00 us 0.00 us 16 > >> >>> FORGET > >> >>> 0.00 0.00 us 0.00 us 0.00 us 665 > >> >>> RELEASE > >> >>> 0.00 0.00 us 0.00 us 0.00 us 52 > >> >>> RELEASEDIR > >> >>> 0.00 866.18 us 849.57 us 882.78 us 2 > >> >>> TRUNCATE > >> >>> 0.00 1016.14 us 868.35 us 1182.48 us 4 > >> >>> RENAME > >> >>> 0.00 455.65 us 41.13 us 1679.37 us 20 > >> >>> READDIR > >> >>> 0.00 373.33 us 173.54 us 538.40 us 28 > >> >>> UNLINK > >> >>> 0.00 722.01 us 635.91 us 853.71 us 16 > >> >>> CREATE > >> >>> 0.00 335.52 us 38.15 us 1381.97 us 54 > >> >>> READDIRP > >> >>> 0.00 39218.92 us 214.34 us 78223.50 us 2 > >> >>> STAT > >> >>> 0.00 1730.77 us 2.59 us 84964.69 us 52 > >> >>> OPENDIR > >> >>> 0.00 1304.96 us 393.64 us 29724.09 us 80 > >> >>> MKNOD > >> >>> 0.01 422.75 us 22.85 us 134336.60 us 635 > >> >>> FLUSH > >> >>> 0.01 699.21 us 37.54 us 137813.64 us 732 > >> >>> STATFS > >> >>> 0.01 468.63 us 23.71 us 260961.83 us 1141 > >> >>> ENTRYLK > >> >>> 0.04 2041.83 us 69.80 us 163743.98 us 649 > >> >>> OPEN > >> >>> 0.08 1253.65 us 72.52 us 290508.15 us 2354 > >> >>> FSTAT > >> >>> 0.09 6913.19 us 69.01 us 212479.24 us 472 > >> >>> XATTROP > >> >>> 0.13 3558.26 us 32.70 us 195896.10 us 1317 > >> >>> INODELK > >> >>> 0.13 9793.37 us 36.11 us 212755.58 us 499 > >> >>> GETXATTR > >> >>> 0.34 1615.15 us 55.42 us 711310.28 us 7797 > >> >>> LOOKUP > >> >>> 4.92 13025.92 us 59.83 us 1061585.06 us 13884 > >> >>> READ > >> >>> 5.26 955.95 us 16.81 us 1069148.49 us 202174 > >> >>> FINODELK > >> >>> 23.28 2809.44 us 137.21 us 1134076.13 us 304713 > >> >>> WRITE > >> >>> 26.46 4124.05 us 43.30 us 1142167.27 us 235954 > >> >>> FXATTROP > >> >>> 39.23 5195.62 us 65.47 us 1250469.35 us 277632 > >> >>> FSYNC > >> >>> 0.00 0.00 us 0.00 us 0.00 us 365 > >> >>> UPCALL > >> >>> > >> >>> Duration: 1132 seconds > >> >>> Data Read: 668951066 bytes > >> >>> Data Written: 7495356076 bytes > >> >>> _______________________________________________ > >> >>> Users mailing list -- users(a)ovirt.org > >> >>> To unsubscribe send an email to users-leave(a)ovirt.org > >> >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html > >> >>> oVirt Code of Conduct: > >> >>> https://www.ovirt.org/community/about/community-guidelines/ > >> >>> List Archives: > >> >>> > >> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/G3G5HNX3P2Z... > >> >>> > >> >> > >> > _______________________________________________ > >> > Users mailing list -- users(a)ovirt.org > >> > To unsubscribe send an email to users-leave(a)ovirt.org > >> > Privacy Statement: https://www.ovirt.org/privacy-policy.html > >> > oVirt Code of Conduct: > >> > https://www.ovirt.org/community/about/community-guidelines/ > >> > List Archives: > >> > > >> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/NKMZYE6EC3I... > >> > > >> > >> > >> -- > >> Derek Atkins 617-623-3745 > >> derek(a)ihtfp.com www.ihtfp.com > >> Computer and Internet Security Consultant > >> > >> > > > > > -- > Derek Atkins 617-623-3745 > derek(a)ihtfp.com www.ihtfp.com > Computer and Internet Security Consultant > >

Hello Derek, We have a 10Gig connection dedicated to the storage network, nothing else is on that switch. On Wed, Feb 23, 2022 at 9:49 AM Derek Atkins <derek@ihtfp.com<mailto:derek@ihtfp.com>> wrote: > Hi, > > Another question which I don't see answered: What is the underlying > connectivity between the Gluster hosts? > > -derek > > On Wed, February 23, 2022 8:39 am, Alex Morrison wrote: > > Hello Sunil, > > > > [root@ovirt1 ~]# gluster --version > > glusterfs 8.6 > > > > same on all hosts

> > > > On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya < > > sheggodu@redhat.com<mailto:sheggodu@redhat.com>> wrote: > > > >> Hi, > >> > >> Which version of gluster is in use? > >> > >> Regards, > >> > >> Sunil kumar Acharya > >> > >> Red Hat > >> > >> <https://www.redhat.com> > >> > >> T: +91-8067935170 > >> <http://redhatemailsignature-marketing.itos.redhat.com/> > >> > >> <https://red.ht/sig> > >> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > >> > >> > >> > >> On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison > <alex@discoverygarden.ca<mailto:alex@discoverygarden.ca>> > >> wrote: > >> > >>> Hello All, > >>> > >>> We have 3 servers with a raid 50 array each, we are having extreme > >>> performance issues with our gluster, writes on gluster seem to take > at > >>> least 3 times longer than on the raid directly. Can this be > improved? > >>> I've > >>> read through several other performance issues threads but have been > >>> unable > >>> to make any improvements > >>> > >>> "gluster volume info" and "gluster volume profile vmstore info" is > >>> below > >>> > >>> > >>> > ========================================================================================================================= > >>> > >>> -Inside Gluster - test took 35+ hours: > >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s > >>> 600G -n 0 -m TEST -f -b -u root > >>> Using uid:0, gid:0. > >>> Writing intelligently...done > >>> Rewriting...done > >>> Reading intelligently...done > >>> start 'em...done...done...done...done...done... > >>> Version 1.98 ------Sequential Output------ --Sequential > Input- > >>> --Random- > >>> -Per Chr- --Block-- -Rewrite- -Per Chr- > --Block-- > >>> --Seeks-- > >>> Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec > %CP > >>> /sec %CP > >>> TEST 600G 35.7m 17 5824k 7 112m > 13 > >>> 182.7 6 > >>> Latency 5466ms 12754ms 3499ms > >>> 1589ms > >>> > >>> > >>> > 1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,,,,,,,,,,,,,,,,,5466ms,12754ms,,3499ms,1589ms,,,,,, > >>> > >>> > >>> > ========================================================================================================================= > >>> > >>> -Outside Gluster - test took 18 minutes: > >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s > >>> 600G -n 0 -m TEST -f -b -u root > >>> Using uid:0, gid:0. > >>> Writing intelligently...done > >>> Rewriting...done > >>> Reading intelligently...done > >>> start 'em...done...done...done...done...done... > >>> Version 1.98 ------Sequential Output------ --Sequential > Input- > >>> --Random- > >>> -Per Chr- --Block-- -Rewrite- -Per Chr- > --Block-- > >>> --Seeks-- > >>> Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec > %CP > >>> /sec %CP > >>> TEST 600G 567m 78 149m 30 307m > 37 > >>> 83.0 57 > >>> Latency 205ms 4630ms 1450ms > >>> 679ms > >>> > >>> > >>> > 1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,,,,,,,,,,,,,,,,,205ms,4630ms,,1450ms,679ms,,,,,, > >>> > >>> > >>> > ========================================================================================================================= > >>> > >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume > info > >>> Volume Name: engine > >>> Type: Replicate > >>> Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b > >>> Status: Started > >>> Snapshot Count: 0 > >>> Number of Bricks: 1 x 3 = 3 > >>> Transport-type: tcp > >>> Bricks: > >>> Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine > >>> Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine > >>> Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine > >>> Options Reconfigured: > >>> cluster.granular-entry-heal: enable > >>> performance.strict-o-direct: on > >>> network.ping-timeout: 30 > >>> storage.owner-gid: 36 > >>> storage.owner-uid: 36 > >>> server.event-threads: 4 > >>> client.event-threads: 4 > >>> cluster.choose-local: off > >>> user.cifs: off > >>> features.shard: on > >>> cluster.shd-wait-qlength: 10000 > >>> cluster.shd-max-threads: 8 > >>> cluster.locking-scheme: granular > >>> cluster.data-self-heal-algorithm: full > >>> cluster.server-quorum-type: server > >>> cluster.quorum-type: auto > >>> cluster.eager-lock: enable > >>> network.remote-dio: off > >>> performance.low-prio-threads: 32 > >>> performance.io-cache: off > >>> performance.read-ahead: off > >>> performance.quick-read: off > >>> transport.address-family: inet > >>> storage.fips-mode-rchecksum: on > >>> nfs.disable: on > >>> performance.client-io-threads: on > >>> diagnostics.latency-measurement: on > >>> diagnostics.count-fop-hits: on > >>> > >>> Volume Name: vmstore > >>> Type: Replicate > >>> Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753 > >>> Status: Started > >>> Snapshot Count: 0 > >>> Number of Bricks: 1 x 3 = 3 > >>> Transport-type: tcp > >>> Bricks: > >>> Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore > >>> Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore > >>> Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore > >>> Options Reconfigured: > >>> cluster.granular-entry-heal: enable > >>> performance.strict-o-direct: on > >>> network.ping-timeout: 20 > >>> storage.owner-gid: 36 > >>> storage.owner-uid: 36 > >>> server.event-threads: 4 > >>> client.event-threads: 4 > >>> cluster.choose-local: off > >>> user.cifs: off > >>> features.shard: on > >>> cluster.shd-wait-qlength: 10000 > >>> cluster.shd-max-threads: 8 > >>> cluster.locking-scheme: granular > >>> cluster.data-self-heal-algorithm: full > >>> cluster.server-quorum-type: server > >>> cluster.quorum-type: auto > >>> cluster.eager-lock: enable > >>> network.remote-dio: off > >>> performance.low-prio-threads: 32 > >>> performance.io-cache: off > >>> performance.read-ahead: off > >>> performance.quick-read: off > >>> transport.address-family: inet > >>> storage.fips-mode-rchecksum: on > >>> nfs.disable: on > >>> performance.client-io-threads: on > >>> diagnostics.latency-measurement: on > >>> diagnostics.count-fop-hits: on > >>> server.tcp-user-timeout: 20 > >>> server.keepalive-time: 10 > >>> server.keepalive-interval: 2 > >>> server.keepalive-count: 5 > >>> cluster.lookup-optimize: off > >>> > >>> > >>> > ========================================================================================================================= > >>> > >>> [root@ovirt1 ~]# gluster volume profile vmstore info > >>> Brick: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore > >>> --------------------------------------------------------- > >>> Cumulative Stats: > >>> Block Size: 1b+ 4b+ > >>> 8b+ > >>> No. of Reads: 0 0 > >>> 0 > >>> No. of Writes: 4021 2 > >>> 64 > >>> > >>> Block Size: 64b+ 128b+ > >>> 256b+ > >>> No. of Reads: 0 134 > >>> 405 > >>> No. of Writes: 2 32 > >>> 119 > >>> > >>> Block Size: 512b+ 1024b+ > >>> 2048b+ > >>> No. of Reads: 2890 2617 > >>> 3059 > >>> No. of Writes: 113881 93551 > >>> 131776 > >>> > >>> Block Size: 4096b+ 8192b+ > >>> 16384b+ > >>> No. of Reads: 1096597 164931 > >>> 181569 > >>> No. of Writes: 41091001 20499776 > >>> 3230472 > >>> > >>> Block Size: 32768b+ 65536b+ > >>> 131072b+ > >>> No. of Reads: 212970 96574 > >>> 244017 > >>> No. of Writes: 1304688 991154 > >>> 3412866 > >>> > >>> Block Size: 262144b+ 524288b+ > >>> 1048576b+ > >>> No. of Reads: 0 0 > >>> 0 > >>> No. of Writes: 15 18 > >>> 65166 > >>> > >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls > >>> Fop > >>> --------- ----------- ----------- ----------- ------------ > >>> ---- > >>> 0.00 0.00 us 0.00 us 0.00 us 6396 > >>> FORGET > >>> 0.00 0.00 us 0.00 us 0.00 us 189759 > >>> RELEASE > >>> 0.00 0.00 us 0.00 us 0.00 us 31292 > >>> RELEASEDIR > >>> 0.00 159.99 us 44.37 us 278.66 us 79 > >>> STAT > >>> 0.00 364.42 us 174.87 us 644.45 us 96 > >>> LINK > >>> 0.00 2714.10 us 203.53 us 47397.28 us 26 > >>> RMDIR > >>> 0.00 688.26 us 72.39 us 53857.63 us 186 > >>> SETXATTR > >>> 0.00 974.01 us 271.02 us 27298.52 us 176 > >>> RENAME > >>> 0.00 5254.64 us 159.48 us 84454.46 us 35 > >>> DISCARD > >>> 0.00 1133.84 us 86.47 us 87096.92 us 186 > >>> REMOVEXATTR > >>> 0.00 7907.47 us 221.82 us 182276.21 us 29 > >>> FALLOCATE > >>> 0.00 8304.51 us 311.89 us 42725.81 us 34 > >>> MKDIR > >>> 0.00 5758.19 us 296.39 us 21950.99 us 81 > >>> TRUNCATE > >>> 0.00 401.16 us 23.68 us 132668.93 us 3621 > >>> LK > >>> 0.00 545.48 us 14.22 us 1076517.76 us 6796 > >>> READDIR > >>> 0.00 2250.78 us 336.61 us 1071764.61 us 4257 > >>> CREATE > >>> 0.00 531.76 us 1.35 us 1723046.94 us 31292 > >>> OPENDIR > >>> 0.01 2058.67 us 196.74 us 1587752.27 us 21986 > >>> MKNOD > >>> 0.01 1241.28 us 22.61 us 1548642.70 us 52674 > >>> READDIRP > >>> 0.01 67508.32 us 100.01 us 3002069.81 us 1100 > >>> FTRUNCATE > >>> 0.01 412.74 us 10.94 us 5111334.35 us 209731 > >>> FLUSH > >>> 0.02 78668.59 us 183.06 us 3012629.87 us 1222 > >>> SETATTR > >>> 0.03 761.13 us 21.13 us 4611854.15 us 249094 > >>> STATFS > >>> 0.03 1132.33 us 45.51 us 3159531.87 us 185654 > >>> OPEN > >>> 0.03 2010.33 us 14.43 us 3158098.85 us 107824 > >>> GETXATTR > >>> 0.04 2765.29 us 61.38 us 3159208.13 us 100484 > >>> XATTROP > >>> 0.13 1367.04 us 51.46 us 8450420.87 us 595299 > >>> FSTAT > >>> 0.46 157440.90 us 47.62 us 13405712.93 us 18218 > >>> UNLINK > >>> 0.60 1274.67 us 19.60 us 11985005.14 us 2980806 > >>> LOOKUP > >>> 1.67 16645.99 us 12.80 us 6869488.64 us 631331 > >>> ENTRYLK > >>> 4.10 7504.86 us 39.52 us 11749528.82 us 3430506 > >>> READ > >>> 4.31 56942.36 us 16.57 us 23654127.65 us 475563 > >>> INODELK > >>> 6.09 660.89 us 11.02 us 20571410.32 us 57908331 > >>> FINODELK > >>> 20.59 1823.35 us 63.26 us 23639762.45 us 70934143 > >>> WRITE > >>> 25.70 2680.82 us 35.37 us 13636394.75 us 60223005 > >>> FXATTROP > >>> 36.14 3164.86 us 52.35 us 16805886.48 us 71739477 > >>> FSYNC > >>> 0.00 0.00 us 0.00 us 0.00 us 78458 > >>> UPCALL > >>> > >>> Duration: 404667 seconds > >>> Data Read: 59850935724 bytes > >>> Data Written: 1070158035683 bytes > >>> > >>> Interval 43 Stats: > >>> Block Size: 1b+ 256b+ > >>> 512b+ > >>> No. of Reads: 0 14 > >>> 3 > >>> No. of Writes: 12 4 > >>> 252 > >>> > >>> Block Size: 1024b+ 2048b+ > >>> 4096b+ > >>> No. of Reads: 0 0 > >>> 8105 > >>> No. of Writes: 100 34 > >>> 147355 > >>> > >>> Block Size: 8192b+ 16384b+ > >>> 32768b+ > >>> No. of Reads: 1559 1630 > >>> 1494 > >>> No. of Writes: 86013 17318 > >>> 8687 > >>> > >>> Block Size: 65536b+ 131072b+ > >>> No. of Reads: 532 764 > >>> No. of Writes: 8617 36317 > >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls > >>> Fop > >>> --------- ----------- ----------- ----------- ------------ > >>> ---- > >>> 0.00 0.00 us 0.00 us 0.00 us 16 > >>> FORGET > >>> 0.00 0.00 us 0.00 us 0.00 us 665 > >>> RELEASE > >>> 0.00 0.00 us 0.00 us 0.00 us 52 > >>> RELEASEDIR > >>> 0.00 181.75 us 181.75 us 181.75 us 1 > >>> STAT > >>> 0.00 914.55 us 858.55 us 970.56 us 2 > >>> TRUNCATE > >>> 0.00 792.11 us 681.24 us 1051.32 us 4 > >>> RENAME > >>> 0.00 374.44 us 31.86 us 792.10 us 20 > >>> READDIR > >>> 0.00 399.48 us 306.97 us 562.69 us 28 > >>> UNLINK > >>> 0.00 752.40 us 636.14 us 907.29 us 16 > >>> CREATE > >>> 0.00 196.62 us 33.27 us 2586.77 us 336 > >>> GETXATTR > >>> 0.00 1107.98 us 288.51 us 29692.53 us 80 > >>> MKNOD > >>> 0.00 1074.80 us 74.41 us 2121.03 us 126 > >>> READDIRP > >>> 0.00 3119.52 us 3.17 us 157121.44 us 52 > >>> OPENDIR > >>> 0.01 972.14 us 21.72 us 329342.21 us 635 > >>> FLUSH > >>> 0.03 2122.15 us 31.97 us 377538.16 us 732 > >>> STATFS > >>> 0.06 4478.89 us 66.09 us 521289.10 us 649 > >>> OPEN > >>> 0.10 10273.00 us 88.19 us 478524.77 us 472 > >>> XATTROP > >>> 0.16 6969.92 us 16.14 us 385156.98 us 1138 > >>> ENTRYLK > >>> 0.20 4635.40 us 64.29 us 1098319.92 us 2097 > >>> FSTAT > >>> 0.65 4098.07 us 46.85 us 919617.66 us 7797 > >>> LOOKUP > >>> 2.36 87465.33 us 34.75 us 1224052.07 us 1317 > >>> INODELK > >>> 4.37 13897.55 us 57.89 us 1984970.27 us 15336 > >>> READ > >>> 5.24 1265.46 us 16.22 us 1638678.32 us 202165 > >>> FINODELK > >>> 21.05 3370.28 us 144.44 us 2264130.49 us 304724 > >>> WRITE > >>> 31.00 6411.68 us 39.84 us 2190355.42 us 235969 > >>> FXATTROP > >>> 34.75 6107.95 us 60.26 us 2270252.44 us 277643 > >>> FSYNC > >>> 0.00 0.00 us 0.00 us 0.00 us 368 > >>> UPCALL > >>> > >>> Duration: 1132 seconds > >>> Data Read: 292367566 bytes > >>> Data Written: 7495339692 bytes > >>> > >>> Brick: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore > >>> --------------------------------------------------------- > >>> Cumulative Stats: > >>> Block Size: 1b+ 4b+ > >>> 8b+ > >>> No. of Reads: 0 0 > >>> 0 > >>> No. of Writes: 4802 2 > >>> 64 > >>> > >>> Block Size: 64b+ 128b+ > >>> 256b+ > >>> No. of Reads: 0 138 > >>> 828 > >>> No. of Writes: 2 32 > >>> 131 > >>> > >>> Block Size: 512b+ 1024b+ > >>> 2048b+ > >>> No. of Reads: 3595 4590 > >>> 6161 > >>> No. of Writes: 128976 99680 > >>> 132413 > >>> > >>> Block Size: 4096b+ 8192b+ > >>> 16384b+ > >>> No. of Reads: 930031 128995 > >>> 150468 > >>> No. of Writes: 48357551 23957030 > >>> 3689474 > >>> > >>> Block Size: 32768b+ 65536b+ > >>> 131072b+ > >>> No. of Reads: 154039 71949 > >>> 790010 > >>> No. of Writes: 1438859 1052045 > >>> 3715427 > >>> > >>> Block Size: 262144b+ 524288b+ > >>> 1048576b+ > >>> No. of Reads: 9 11 > >>> 40269 > >>> No. of Writes: 5 20 > >>> 50757 > >>> > >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls > >>> Fop > >>> --------- ----------- ----------- ----------- ------------ > >>> ---- > >>> 0.00 0.00 us 0.00 us 0.00 us 7629 > >>> FORGET > >>> 0.00 0.00 us 0.00 us 0.00 us 228577 > >>> RELEASE > >>> 0.00 0.00 us 0.00 us 0.00 us 41524 > >>> RELEASEDIR > >>> 0.00 374.22 us 219.35 us 574.87 us 96 > >>> LINK > >>> 0.00 1712.22 us 164.90 us 22337.79 us 41 > >>> DISCARD > >>> 0.00 2840.25 us 245.24 us 46839.63 us 26 > >>> RMDIR > >>> 0.00 3660.92 us 229.86 us 86252.56 us 35 > >>> FALLOCATE > >>> 0.00 876.48 us 297.53 us 16685.63 us 188 > >>> RENAME > >>> 0.00 6660.10 us 379.21 us 52962.79 us 34 > >>> MKDIR > >>> 0.00 957.48 us 80.36 us 218059.88 us 355 > >>> REMOVEXATTR > >>> 0.00 1234.85 us 76.50 us 322955.27 us 355 > >>> SETXATTR > >>> 0.00 2326.62 us 84.07 us 250041.25 us 196 > >>> STAT > >>> 0.00 9660.45 us 289.50 us 295106.01 us 87 > >>> TRUNCATE > >>> 0.00 486.10 us 24.12 us 869966.79 us 5260 > >>> LK > >>> 0.00 453.69 us 18.32 us 426653.18 us 8135 > >>> READDIR > >>> 0.00 1832.38 us 296.61 us 261299.81 us 5050 > >>> CREATE > >>> 0.00 386.77 us 1.31 us 1300836.12 us 41524 > >>> OPENDIR > >>> 0.00 712.06 us 29.85 us 1441115.02 us 34874 > >>> READDIRP > >>> 0.01 1469.51 us 231.67 us 440065.71 us 22845 > >>> MKNOD > >>> 0.01 44229.36 us 75.58 us 1803963.42 us 871 > >>> FTRUNCATE > >>> 0.01 84433.20 us 149.47 us 3614869.24 us 1001 > >>> SETATTR > >>> 0.02 418.11 us 13.39 us 3515000.81 us 243820 > >>> FLUSH > >>> 0.03 1108.47 us 14.77 us 1658743.41 us 130647 > >>> GETXATTR > >>> 0.03 562.86 us 26.74 us 5043949.33 us 297799 > >>> STATFS > >>> 0.03 845.50 us 48.16 us 1998938.03 us 223680 > >>> OPEN > >>> 0.05 2236.13 us 56.08 us 5295998.22 us 120682 > >>> XATTROP > >>> 0.16 890.82 us 43.51 us 3653292.88 us 1012585 > >>> FSTAT > >>> 0.34 2974.20 us 12.92 us 7782497.74 us 642555 > >>> ENTRYLK > >>> 0.47 135831.21 us 70.35 us 11033867.90 us 19800 > >>> UNLINK > >>> 0.57 954.05 us 21.50 us 4553393.38 us 3414605 > >>> LOOKUP > >>> 1.79 14461.54 us 13.45 us 32841452.98 us 702915 > >>> INODELK > >>> 5.14 8014.78 us 40.70 us 5439109.56 us 3644063 > >>> READ > >>> 5.37 443.04 us 11.53 us 32863652.53 us 68909131 > >>> FINODELK > >>> 22.31 1780.31 us 33.59 us 11318712.62 us 71235991 > >>> FXATTROP > >>> 22.83 1571.16 us 74.86 us 32615055.19 us 82622840 > >>> WRITE > >>> 40.84 2762.75 us 52.77 us 8859115.35 us 84039509 > >>> FSYNC > >>> 0.00 0.00 us 0.00 us 0.00 us 95492 > >>> UPCALL > >>> > >>> Duration: 484169 seconds > >>> Data Read: 167149718723 bytes > >>> Data Written: 1177141649872 bytes > >>> > >>> Interval 43 Stats: > >>> Block Size: 1b+ 256b+ > >>> 512b+ > >>> No. of Reads: 0 6 > >>> 4 > >>> No. of Writes: 12 4 > >>> 252 > >>> > >>> Block Size: 1024b+ 2048b+ > >>> 4096b+ > >>> No. of Reads: 0 0 > >>> 5668 > >>> No. of Writes: 100 34 > >>> 147357 > >>> > >>> Block Size: 8192b+ 16384b+ > >>> 32768b+ > >>> No. of Reads: 1178 783 > >>> 1215 > >>> No. of Writes: 86014 17318 > >>> 8687 > >>> > >>> Block Size: 65536b+ 131072b+ > >>> No. of Reads: 264 4109 > >>> No. of Writes: 8617 36317 > >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls > >>> Fop > >>> --------- ----------- ----------- ----------- ------------ > >>> ---- > >>> 0.00 0.00 us 0.00 us 0.00 us 16 > >>> FORGET > >>> 0.00 0.00 us 0.00 us 0.00 us 665 > >>> RELEASE > >>> 0.00 0.00 us 0.00 us 0.00 us 52 > >>> RELEASEDIR > >>> 0.00 866.18 us 849.57 us 882.78 us 2 > >>> TRUNCATE > >>> 0.00 1016.14 us 868.35 us 1182.48 us 4 > >>> RENAME > >>> 0.00 455.65 us 41.13 us 1679.37 us 20 > >>> READDIR > >>> 0.00 373.33 us 173.54 us 538.40 us 28 > >>> UNLINK > >>> 0.00 722.01 us 635.91 us 853.71 us 16 > >>> CREATE > >>> 0.00 335.52 us 38.15 us 1381.97 us 54 > >>> READDIRP > >>> 0.00 39218.92 us 214.34 us 78223.50 us 2 > >>> STAT > >>> 0.00 1730.77 us 2.59 us 84964.69 us 52 > >>> OPENDIR > >>> 0.00 1304.96 us 393.64 us 29724.09 us 80 > >>> MKNOD > >>> 0.01 422.75 us 22.85 us 134336.60 us 635 > >>> FLUSH > >>> 0.01 699.21 us 37.54 us 137813.64 us 732 > >>> STATFS > >>> 0.01 468.63 us 23.71 us 260961.83 us 1141 > >>> ENTRYLK > >>> 0.04 2041.83 us 69.80 us 163743.98 us 649 > >>> OPEN > >>> 0.08 1253.65 us 72.52 us 290508.15 us 2354 > >>> FSTAT > >>> 0.09 6913.19 us 69.01 us 212479.24 us 472 > >>> XATTROP > >>> 0.13 3558.26 us 32.70 us 195896.10 us 1317 > >>> INODELK > >>> 0.13 9793.37 us 36.11 us 212755.58 us 499 > >>> GETXATTR > >>> 0.34 1615.15 us 55.42 us 711310.28 us 7797 > >>> LOOKUP > >>> 4.92 13025.92 us 59.83 us 1061585.06 us 13884 > >>> READ > >>> 5.26 955.95 us 16.81 us 1069148.49 us 202174 > >>> FINODELK > >>> 23.28 2809.44 us 137.21 us 1134076.13 us 304713 > >>> WRITE > >>> 26.46 4124.05 us 43.30 us 1142167.27 us 235954 > >>> FXATTROP > >>> 39.23 5195.62 us 65.47 us 1250469.35 us 277632 > >>> FSYNC > >>> 0.00 0.00 us 0.00 us 0.00 us 365 > >>> UPCALL > >>> > >>> Duration: 1132 seconds > >>> Data Read: 668951066 bytes > >>> Data Written: 7495356076 bytes > >>> _______________________________________________ > >>> Users mailing list -- users@ovirt.org<mailto:users@ovirt.org> > >>> To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org> > >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html > >>> oVirt Code of Conduct: > >>> https://www.ovirt.org/community/about/community-guidelines/ > >>> List Archives: > >>> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/G3G5HNX3P2Z... > >>> > >> > > _______________________________________________ > > Users mailing list -- users@ovirt.org<mailto:users@ovirt.org> > > To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org> > > Privacy Statement: https://www.ovirt.org/privacy-policy.html > > oVirt Code of Conduct: > > https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: > > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/NKMZYE6EC3I... > > > > > -- > Derek Atkins 617-623-3745 > derek@ihtfp.com<mailto:derek@ihtfp.com> www.ihtfp.com<http://www.ihtfp.com> > Computer and Internet Security Consultant > >