Hi, OK, thanks. I'd also asked for gluster version you're running. Could you share that information as well? -Krutika On Thu, Mar 7, 2019 at 9:38 PM Drew Rash <drew.rash@gmail.com> wrote:
Here is the output for our ssd gluster which exhibits the same issue as the hdd glusters. However, I can replicate the issue on an 8TB WD Gold disk NFS mounted as well ( removed the gluster part ) Which is the reason I'm on the oVirt site. I can start a file copy that writes at max speed, then after a gb or 2 it drops down to 3-10 MBps maxing at 13.3 ish overall. Testing outside of oVirt using dd doesn't have the same behavior. Outside ovirt (directly on the ovirtnode to the gluster or 8tb nfs mount results is max drive speeds consistently for large file copies)
I enabled writeback (as someone suggested) on the virtio-scsi windows disk and one of our windows 10 installs speed up. Still suffers from sustained write issue which causes the whole box to cripple. Opening chrome for example cripples the box or sql server management studio also.
Volume Name: gv1 Type: Replicate Volume ID: 7340a436-d971-4d69-84f9-12a23cd76ec8 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: 10.30.30.121:/gluster_bricks/gv1/brick Brick2: 10.30.30.122:/gluster_bricks/gv1/brick Brick3: 10.30.30.123:/gluster_bricks/gv1/brick (arbiter) Options Reconfigured: network.ping-timeout: 30 performance.strict-o-direct: on storage.owner-gid: 36 storage.owner-uid: 36 user.cifs: off cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off features.shard: off cluster.granular-entry-heal: enable transport.address-family: inet nfs.disable: on performance.client-io-threads: off
On Thu, Mar 7, 2019 at 1:00 AM Krutika Dhananjay <kdhananj@redhat.com> wrote:
So from the profile, it appears the XATTROPs and FINODELKs are way higher than the number of WRITEs:
<excerpt> ... ... %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.43 384.83 us 51.00 us 65375.00 us 13632 FXATTROP 7.54 13535.70 us 225.00 us 210298.00 us 6816 WRITE 45.99 28508.86 us 7.00 us 2591280.00 us 19751 FINODELK
... ... </excerpt>
We'd noticed something similar in our internal tests and found inefficiencies in gluster's eager-lock implementation. This was fixed at https://review.gluster.org/c/glusterfs/+/19503. I need the two things I asked for in the prev mail to confirm if you're hitting the same issue.
-Krutika
On Thu, Mar 7, 2019 at 12:24 PM Krutika Dhananjay <kdhananj@redhat.com> wrote:
Hi,
Could you share the following pieces of information to begin with -
1. output of `gluster volume info $AFFECTED_VOLUME_NAME` 2. glusterfs version you're running
-Krutika
On Sat, Mar 2, 2019 at 3:38 AM Drew R <drew.rash@gmail.com> wrote:
Saw some people asking for profile info. So I had started a migration from a 6TB WDGold 2+1arb replicated gluster to a 1TB samsung ssd 2+1 rep gluster and it's been running a while for a 100GB file thin provisioned with like 28GB actually used. Here is the profile info. I started the profiler like 5 minutes ago. The migration had been running for like 30minutes:
gluster volume profile gv2 info Brick: 10.30.30.122:/gluster_bricks/gv2/brick --------------------------------------------- Cumulative Stats: Block Size: 256b+ 512b+ 1024b+ No. of Reads: 1189 8 12 No. of Writes: 4 3245 883
Block Size: 2048b+ 4096b+ 8192b+ No. of Reads: 10 20 2 No. of Writes: 1087 312228 124080
Block Size: 16384b+ 32768b+ 65536b+ No. of Reads: 0 1 52 No. of Writes: 5188 3617 5532
Block Size: 131072b+ No. of Reads: 70191 No. of Writes: 634192 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 2 FORGET 0.00 0.00 us 0.00 us 0.00 us 202 RELEASE 0.00 0.00 us 0.00 us 0.00 us 1297 RELEASEDIR 0.00 14.50 us 9.00 us 20.00 us 4 READDIR 0.00 38.00 us 9.00 us 120.00 us 7 GETXATTR 0.00 66.00 us 34.00 us 128.00 us 6 OPEN 0.00 137.25 us 52.00 us 195.00 us 4 SETATTR 0.00 23.19 us 11.00 us 46.00 us 26 INODELK 0.00 41.58 us 18.00 us 79.00 us 24 OPENDIR 0.00 166.70 us 15.00 us 775.00 us 27 READDIRP 0.01 135.29 us 12.00 us 11695.00 us 221 STATFS 0.01 176.54 us 22.00 us 22944.00 us 364 FSTAT 0.02 626.21 us 13.00 us 17308.00 us 168 STAT 0.09 834.84 us 9.00 us 34337.00 us 607 LOOKUP 0.73 146.18 us 6.00 us 52255.00 us 29329 FINODELK 1.03 298.20 us 42.00 us 43711.00 us 20204 FXATTROP 15.38 8903.40 us 213.00 us 213832.00 us 10102 WRITE 39.14 26796.37 us 222.00 us 122696.00 us 8538 READ 43.59 39536.79 us 259.00 us 183630.00 us 6446 FSYNC
Duration: 15078 seconds Data Read: 9207377205 bytes Data Written: 86214017762 bytes
Interval 2 Stats: Block Size: 256b+ 512b+ 1024b+ No. of Reads: 17 0 0 No. of Writes: 0 43 7
Block Size: 2048b+ 4096b+ 8192b+ No. of Reads: 0 7 0 No. of Writes: 16 1881 1010
Block Size: 16384b+ 32768b+ 65536b+ No. of Reads: 0 0 6 No. of Writes: 305 586 2359
Block Size: 131072b+ No. of Reads: 7162 No. of Writes: 610 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 6 RELEASE 0.00 0.00 us 0.00 us 0.00 us 20 RELEASEDIR 0.00 14.50 us 9.00 us 20.00 us 4 READDIR 0.00 38.00 us 9.00 us 120.00 us 7 GETXATTR 0.00 66.00 us 34.00 us 128.00 us 6 OPEN 0.00 137.25 us 52.00 us 195.00 us 4 SETATTR 0.00 23.19 us 11.00 us 46.00 us 26 INODELK 0.00 40.05 us 18.00 us 79.00 us 20 OPENDIR 0.00 180.33 us 16.00 us 775.00 us 21 READDIRP 0.01 181.77 us 12.00 us 11695.00 us 149 STATFS 0.01 511.23 us 15.00 us 16954.00 us 111 STAT 0.01 210.22 us 22.00 us 22944.00 us 291 FSTAT 0.09 979.73 us 9.00 us 23539.00 us 404 LOOKUP 0.81 179.58 us 6.00 us 52255.00 us 19786 FINODELK 1.07 342.71 us 42.00 us 43711.00 us 13633 FXATTROP 14.18 9087.30 us 213.00 us 213832.00 us 6817 WRITE 40.00 40227.38 us 259.00 us 157934.00 us 4343 FSYNC 43.81 26601.57 us 222.00 us 118756.00 us 7192 READ
Duration: 177 seconds Data Read: 939534945 bytes Data Written: 363763200 bytes
Brick: 10.30.30.121:/gluster_bricks/gv2/brick --------------------------------------------- Cumulative Stats: Block Size: 8b+ 256b+ 512b+ No. of Reads: 0 142626 329092 No. of Writes: 12 98 123230
Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 21175 18223 3345043 No. of Writes: 58646 69147 6092143
Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 1090937 216037 748595 No. of Writes: 2030403 243915 240942
Block Size: 65536b+ 131072b+ No. of Reads: 1524667 4828662 No. of Writes: 605125 1889892 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 133 FORGET 0.00 0.00 us 0.00 us 0.00 us 4821 RELEASE 0.00 0.00 us 0.00 us 0.00 us 42742 RELEASEDIR 0.00 28.50 us 12.00 us 61.00 us 4 READDIR 0.00 71.75 us 59.00 us 88.00 us 4 SETATTR 0.00 44.57 us 12.00 us 94.00 us 7 GETXATTR 0.00 62.17 us 42.00 us 89.00 us 6 OPEN 0.00 35.92 us 1.00 us 76.00 us 24 OPENDIR 0.00 325.54 us 22.00 us 1117.00 us 13 READDIRP 0.00 482.13 us 19.00 us 12473.00 us 46 STAT 0.00 217.65 us 15.00 us 32841.00 us 221 STATFS 0.01 497.52 us 11.00 us 29179.00 us 539 LOOKUP 0.02 685.44 us 16.00 us 61131.00 us 615 FSTAT 0.04 31691.92 us 17.00 us 349648.00 us 26 INODELK 0.45 451.95 us 51.00 us 65375.00 us 20206 FXATTROP 6.90 13906.03 us 225.00 us 210298.00 us 10103 WRITE 18.47 58375.96 us 389.00 us 218211.00 us 6446 FSYNC 27.45 27279.84 us 184.00 us 241444.00 us 20503 READ 46.66 32462.82 us 7.00 us 2750519.00 us 29283 FINODELK
Duration: 475152 seconds Data Read: 799630169437 bytes Data Written: 362708560796 bytes
Interval 2 Stats: Block Size: 256b+ 512b+ 1024b+ No. of Reads: 53 0 0 No. of Writes: 0 43 7
Block Size: 2048b+ 4096b+ 8192b+ No. of Reads: 0 891 616 No. of Writes: 16 1881 1010
Block Size: 16384b+ 32768b+ 65536b+ No. of Reads: 288 750 52 No. of Writes: 304 586 2359
Block Size: 131072b+ No. of Reads: 9445 No. of Writes: 610 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 6 RELEASE 0.00 0.00 us 0.00 us 0.00 us 20 RELEASEDIR 0.00 28.50 us 12.00 us 61.00 us 4 READDIR 0.00 71.75 us 59.00 us 88.00 us 4 SETATTR 0.00 44.57 us 12.00 us 94.00 us 7 GETXATTR 0.00 62.17 us 42.00 us 89.00 us 6 OPEN 0.00 34.50 us 1.00 us 76.00 us 20 OPENDIR 0.00 55.16 us 19.00 us 128.00 us 25 STAT 0.00 363.18 us 25.00 us 1117.00 us 11 READDIRP 0.00 71.61 us 15.00 us 4323.00 us 149 STATFS 0.01 251.40 us 11.00 us 15344.00 us 336 LOOKUP 0.02 594.11 us 16.00 us 61131.00 us 450 FSTAT 0.07 31691.92 us 17.00 us 349648.00 us 26 INODELK 0.43 384.83 us 51.00 us 65375.00 us 13632 FXATTROP 7.54 13535.70 us 225.00 us 210298.00 us 6816 WRITE 20.03 56497.17 us 394.00 us 218211.00 us 4340 FSYNC 25.91 26229.86 us 184.00 us 241444.00 us 12095 READ 45.99 28508.86 us 7.00 us 2591280.00 us 19751 FINODELK
Duration: 177 seconds Data Read: 1298631621 bytes Data Written: 363744256 bytes
Brick: 10.30.30.123:/gluster_bricks/gv2/brick --------------------------------------------- Cumulative Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 18126558 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 383 FORGET 0.00 0.00 us 0.00 us 0.00 us 9602 RELEASE 0.00 0.00 us 0.00 us 0.00 us 70644 RELEASEDIR 0.00 20.75 us 13.00 us 29.00 us 4 READDIR 0.00 87.00 us 72.00 us 129.00 us 4 SETATTR 0.00 71.00 us 46.00 us 93.00 us 6 OPEN 0.00 66.00 us 9.00 us 114.00 us 7 GETXATTR 0.00 50.58 us 16.00 us 90.00 us 26 INODELK 0.00 56.38 us 1.00 us 111.00 us 24 OPENDIR 0.02 66.23 us 28.00 us 146.00 us 113 FSTAT 0.17 143.88 us 13.00 us 444.00 us 539 LOOKUP 1.17 51.64 us 10.00 us 166.00 us 10103 WRITE 2.89 44.06 us 7.00 us 2177.00 us 29349 FINODELK 5.55 122.83 us 54.00 us 9281.00 us 20206 FXATTROP 90.19 6249.64 us 237.00 us 59996.00 us 6448 FSYNC
Duration: 772792 seconds Data Read: 0 bytes Data Written: 18126558 bytes
Interval 2 Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 6816 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 6 RELEASE 0.00 0.00 us 0.00 us 0.00 us 20 RELEASEDIR 0.00 20.75 us 13.00 us 29.00 us 4 READDIR 0.00 87.00 us 72.00 us 129.00 us 4 SETATTR 0.00 71.00 us 46.00 us 93.00 us 6 OPEN 0.00 66.00 us 9.00 us 114.00 us 7 GETXATTR 0.00 58.10 us 1.00 us 111.00 us 20 OPENDIR 0.00 50.58 us 16.00 us 90.00 us 26 INODELK 0.03 66.23 us 28.00 us 146.00 us 113 FSTAT 0.16 145.84 us 13.00 us 346.00 us 336 LOOKUP 1.18 51.98 us 11.00 us 157.00 us 6816 WRITE 2.91 44.05 us 7.00 us 2177.00 us 19798 FINODELK 5.59 122.70 us 54.00 us 9281.00 us 13632 FXATTROP 90.11 6210.61 us 238.00 us 54711.00 us 4342 FSYNC
Duration: 177 seconds Data Read: 0 bytes Data Written: 6816 bytes _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/XHAUIMGRCVEDCL...