On Thu, Mar 7, 2019 at 12:24 PM Krutika Dhananjay <kdhananj@redhat.com> wrote:

Hi,

Could you share the following pieces of information to begin with -

1. output of `gluster volume info $AFFECTED_VOLUME_NAME`
2. glusterfs version you're running

-Krutika

On Sat, Mar 2, 2019 at 3:38 AM Drew R <drew.rash@gmail.com> wrote:
Saw some people asking for profile info. So I had started a migration from a 6TB WDGold 2+1arb replicated gluster to a 1TB samsung ssd 2+1 rep gluster and it's been running a while for a 100GB file thin provisioned with like 28GB actually used. Here is the profile info. I started the profiler like 5 minutes ago. The migration had been running for like 30minutes:

gluster volume profile gv2 info
Brick: 10.30.30.122:/gluster_bricks/gv2/brick
---------------------------------------------
Cumulative Stats:
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 1189 8 12
No. of Writes: 4 3245 883

Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 10 20 2
No. of Writes: 1087 312228 124080

Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 0 1 52
No. of Writes: 5188 3617 5532

Block Size: 131072b+
No. of Reads: 70191
No. of Writes: 634192
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 2 FORGET
0.00 0.00 us 0.00 us 0.00 us 202 RELEASE
0.00 0.00 us 0.00 us 0.00 us 1297 RELEASEDIR
0.00 14.50 us 9.00 us 20.00 us 4 READDIR
0.00 38.00 us 9.00 us 120.00 us 7 GETXATTR
0.00 66.00 us 34.00 us 128.00 us 6 OPEN
0.00 137.25 us 52.00 us 195.00 us 4 SETATTR
0.00 23.19 us 11.00 us 46.00 us 26 INODELK
0.00 41.58 us 18.00 us 79.00 us 24 OPENDIR
0.00 166.70 us 15.00 us 775.00 us 27 READDIRP
0.01 135.29 us 12.00 us 11695.00 us 221 STATFS
0.01 176.54 us 22.00 us 22944.00 us 364 FSTAT
0.02 626.21 us 13.00 us 17308.00 us 168 STAT
0.09 834.84 us 9.00 us 34337.00 us 607 LOOKUP
0.73 146.18 us 6.00 us 52255.00 us 29329 FINODELK
1.03 298.20 us 42.00 us 43711.00 us 20204 FXATTROP
15.38 8903.40 us 213.00 us 213832.00 us 10102 WRITE
39.14 26796.37 us 222.00 us 122696.00 us 8538 READ
43.59 39536.79 us 259.00 us 183630.00 us 6446 FSYNC

Duration: 15078 seconds
Data Read: 9207377205 bytes
Data Written: 86214017762 bytes

Interval 2 Stats:
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 17 0 0
No. of Writes: 0 43 7

Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 7 0
No. of Writes: 16 1881 1010

Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 0 0 6
No. of Writes: 305 586 2359

Block Size: 131072b+
No. of Reads: 7162
No. of Writes: 610
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6 RELEASE
0.00 0.00 us 0.00 us 0.00 us 20 RELEASEDIR
0.00 14.50 us 9.00 us 20.00 us 4 READDIR
0.00 38.00 us 9.00 us 120.00 us 7 GETXATTR
0.00 66.00 us 34.00 us 128.00 us 6 OPEN
0.00 137.25 us 52.00 us 195.00 us 4 SETATTR
0.00 23.19 us 11.00 us 46.00 us 26 INODELK
0.00 40.05 us 18.00 us 79.00 us 20 OPENDIR
0.00 180.33 us 16.00 us 775.00 us 21 READDIRP
0.01 181.77 us 12.00 us 11695.00 us 149 STATFS
0.01 511.23 us 15.00 us 16954.00 us 111 STAT
0.01 210.22 us 22.00 us 22944.00 us 291 FSTAT
0.09 979.73 us 9.00 us 23539.00 us 404 LOOKUP
0.81 179.58 us 6.00 us 52255.00 us 19786 FINODELK
1.07 342.71 us 42.00 us 43711.00 us 13633 FXATTROP
14.18 9087.30 us 213.00 us 213832.00 us 6817 WRITE
40.00 40227.38 us 259.00 us 157934.00 us 4343 FSYNC
43.81 26601.57 us 222.00 us 118756.00 us 7192 READ

Duration: 177 seconds
Data Read: 939534945 bytes
Data Written: 363763200 bytes

Brick: 10.30.30.121:/gluster_bricks/gv2/brick
---------------------------------------------
Cumulative Stats:
Block Size: 8b+ 256b+ 512b+
No. of Reads: 0 142626 329092
No. of Writes: 12 98 123230

Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 21175 18223 3345043
No. of Writes: 58646 69147 6092143

Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 1090937 216037 748595
No. of Writes: 2030403 243915 240942

Block Size: 65536b+ 131072b+
No. of Reads: 1524667 4828662
No. of Writes: 605125 1889892
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 133 FORGET
0.00 0.00 us 0.00 us 0.00 us 4821 RELEASE
0.00 0.00 us 0.00 us 0.00 us 42742 RELEASEDIR
0.00 28.50 us 12.00 us 61.00 us 4 READDIR
0.00 71.75 us 59.00 us 88.00 us 4 SETATTR
0.00 44.57 us 12.00 us 94.00 us 7 GETXATTR
0.00 62.17 us 42.00 us 89.00 us 6 OPEN
0.00 35.92 us 1.00 us 76.00 us 24 OPENDIR
0.00 325.54 us 22.00 us 1117.00 us 13 READDIRP
0.00 482.13 us 19.00 us 12473.00 us 46 STAT
0.00 217.65 us 15.00 us 32841.00 us 221 STATFS
0.01 497.52 us 11.00 us 29179.00 us 539 LOOKUP
0.02 685.44 us 16.00 us 61131.00 us 615 FSTAT
0.04 31691.92 us 17.00 us 349648.00 us 26 INODELK
0.45 451.95 us 51.00 us 65375.00 us 20206 FXATTROP
6.90 13906.03 us 225.00 us 210298.00 us 10103 WRITE
18.47 58375.96 us 389.00 us 218211.00 us 6446 FSYNC
27.45 27279.84 us 184.00 us 241444.00 us 20503 READ
46.66 32462.82 us 7.00 us 2750519.00 us 29283 FINODELK

Duration: 475152 seconds
Data Read: 799630169437 bytes
Data Written: 362708560796 bytes

Interval 2 Stats:
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 53 0 0
No. of Writes: 0 43 7

Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 891 616
No. of Writes: 16 1881 1010

Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 288 750 52
No. of Writes: 304 586 2359

Block Size: 131072b+
No. of Reads: 9445
No. of Writes: 610
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6 RELEASE
0.00 0.00 us 0.00 us 0.00 us 20 RELEASEDIR
0.00 28.50 us 12.00 us 61.00 us 4 READDIR
0.00 71.75 us 59.00 us 88.00 us 4 SETATTR
0.00 44.57 us 12.00 us 94.00 us 7 GETXATTR
0.00 62.17 us 42.00 us 89.00 us 6 OPEN
0.00 34.50 us 1.00 us 76.00 us 20 OPENDIR
0.00 55.16 us 19.00 us 128.00 us 25 STAT
0.00 363.18 us 25.00 us 1117.00 us 11 READDIRP
0.00 71.61 us 15.00 us 4323.00 us 149 STATFS
0.01 251.40 us 11.00 us 15344.00 us 336 LOOKUP
0.02 594.11 us 16.00 us 61131.00 us 450 FSTAT
0.07 31691.92 us 17.00 us 349648.00 us 26 INODELK
0.43 384.83 us 51.00 us 65375.00 us 13632 FXATTROP
7.54 13535.70 us 225.00 us 210298.00 us 6816 WRITE
20.03 56497.17 us 394.00 us 218211.00 us 4340 FSYNC
25.91 26229.86 us 184.00 us 241444.00 us 12095 READ
45.99 28508.86 us 7.00 us 2591280.00 us 19751 FINODELK

Duration: 177 seconds
Data Read: 1298631621 bytes
Data Written: 363744256 bytes

Brick: 10.30.30.123:/gluster_bricks/gv2/brick
---------------------------------------------
Cumulative Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 18126558
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 383 FORGET
0.00 0.00 us 0.00 us 0.00 us 9602 RELEASE
0.00 0.00 us 0.00 us 0.00 us 70644 RELEASEDIR
0.00 20.75 us 13.00 us 29.00 us 4 READDIR
0.00 87.00 us 72.00 us 129.00 us 4 SETATTR
0.00 71.00 us 46.00 us 93.00 us 6 OPEN
0.00 66.00 us 9.00 us 114.00 us 7 GETXATTR
0.00 50.58 us 16.00 us 90.00 us 26 INODELK
0.00 56.38 us 1.00 us 111.00 us 24 OPENDIR
0.02 66.23 us 28.00 us 146.00 us 113 FSTAT
0.17 143.88 us 13.00 us 444.00 us 539 LOOKUP
1.17 51.64 us 10.00 us 166.00 us 10103 WRITE
2.89 44.06 us 7.00 us 2177.00 us 29349 FINODELK
5.55 122.83 us 54.00 us 9281.00 us 20206 FXATTROP
90.19 6249.64 us 237.00 us 59996.00 us 6448 FSYNC

Duration: 772792 seconds
Data Read: 0 bytes
Data Written: 18126558 bytes

Interval 2 Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 6816
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6 RELEASE
0.00 0.00 us 0.00 us 0.00 us 20 RELEASEDIR
0.00 20.75 us 13.00 us 29.00 us 4 READDIR
0.00 87.00 us 72.00 us 129.00 us 4 SETATTR
0.00 71.00 us 46.00 us 93.00 us 6 OPEN
0.00 66.00 us 9.00 us 114.00 us 7 GETXATTR
0.00 58.10 us 1.00 us 111.00 us 20 OPENDIR
0.00 50.58 us 16.00 us 90.00 us 26 INODELK
0.03 66.23 us 28.00 us 146.00 us 113 FSTAT
0.16 145.84 us 13.00 us 346.00 us 336 LOOKUP
1.18 51.98 us 11.00 us 157.00 us 6816 WRITE
2.91 44.05 us 7.00 us 2177.00 us 19798 FINODELK
5.59 122.70 us 54.00 us 9281.00 us 13632 FXATTROP
90.11 6210.61 us 238.00 us 54711.00 us 4342 FSYNC

Duration: 177 seconds
Data Read: 0 bytes
Data Written: 6816 bytes
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/XHAUIMGRCVEDCLS7F6WALUWJVQ4IXAWU/