[ovirt-users] Re: oVirt Performance (Horrific)

So from the profile, it appears the XATTROPs and FINODELKs are way higher than the number of WRITEs: <excerpt> ... ... %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.43 384.83 us 51.00 us 65375.00 us 13632 FXATTROP 7.54 13535.70 us 225.00 us 210298.00 us 6816 WRITE 45.99 28508.86 us 7.00 us 2591280.00 us 19751 FINODELK ... ... </excerpt> We'd noticed something similar in our internal tests and found inefficiencies in gluster's eager-lock implementation. This was fixed at https://review.gluster.org/c/glusterfs/+/19503. I need the two things I asked for in the prev mail to confirm if you're hitting the same issue. -Krutika On Thu, Mar 7, 2019 at 12:24 PM Krutika Dhananjay <kdhananj(a)redhat.com&gt; wrote: > Hi, > > Could you share the following pieces of information to begin with - > > 1. output of `gluster volume info $AFFECTED_VOLUME_NAME` > 2. glusterfs version you're running > > -Krutika > > > On Sat, Mar 2, 2019 at 3:38 AM Drew R <drew.rash(a)gmail.com&gt; wrote: > >> Saw some people asking for profile info. So I had started a migration >> from a 6TB WDGold 2+1arb replicated gluster to a 1TB samsung ssd 2+1 rep >> gluster and it's been running a while for a 100GB file thin provisioned >> with like 28GB actually used. Here is the profile info. I started the >> profiler like 5 minutes ago. The migration had been running for like >> 30minutes: >> >> gluster volume profile gv2 info >> Brick: 10.30.30.122:/gluster_bricks/gv2/brick >> --------------------------------------------- >> Cumulative Stats: >> Block Size: 256b+ 512b+ >> 1024b+ >> No. of Reads: 1189 8 >> 12 >> No. of Writes: 4 3245 >> 883 >> >> Block Size: 2048b+ 4096b+ >> 8192b+ >> No. of Reads: 10 20 >> 2 >> No. of Writes: 1087 312228 >> 124080 >> >> Block Size: 16384b+ 32768b+ >> 65536b+ >> No. of Reads: 0 1 >> 52 >> No. of Writes: 5188 3617 >> 5532 >> >> Block Size: 131072b+ >> No. of Reads: 70191 >> No. of Writes: 634192 >> %-latency Avg-latency Min-Latency Max-Latency No. of calls >> Fop >> --------- ----------- ----------- ----------- ------------ >> ---- >> 0.00 0.00 us 0.00 us 0.00 us 2 >> FORGET >> 0.00 0.00 us 0.00 us 0.00 us 202 >> RELEASE >> 0.00 0.00 us 0.00 us 0.00 us 1297 >> RELEASEDIR >> 0.00 14.50 us 9.00 us 20.00 us 4 >> READDIR >> 0.00 38.00 us 9.00 us 120.00 us 7 >> GETXATTR >> 0.00 66.00 us 34.00 us 128.00 us 6 >> OPEN >> 0.00 137.25 us 52.00 us 195.00 us 4 >> SETATTR >> 0.00 23.19 us 11.00 us 46.00 us 26 >> INODELK >> 0.00 41.58 us 18.00 us 79.00 us 24 >> OPENDIR >> 0.00 166.70 us 15.00 us 775.00 us 27 >> READDIRP >> 0.01 135.29 us 12.00 us 11695.00 us 221 >> STATFS >> 0.01 176.54 us 22.00 us 22944.00 us 364 >> FSTAT >> 0.02 626.21 us 13.00 us 17308.00 us 168 >> STAT >> 0.09 834.84 us 9.00 us 34337.00 us 607 >> LOOKUP >> 0.73 146.18 us 6.00 us 52255.00 us 29329 >> FINODELK >> 1.03 298.20 us 42.00 us 43711.00 us 20204 >> FXATTROP >> 15.38 8903.40 us 213.00 us 213832.00 us 10102 >> WRITE >> 39.14 26796.37 us 222.00 us 122696.00 us 8538 >> READ >> 43.59 39536.79 us 259.00 us 183630.00 us 6446 >> FSYNC >> >> Duration: 15078 seconds >> Data Read: 9207377205 bytes >> Data Written: 86214017762 bytes >> >> Interval 2 Stats: >> Block Size: 256b+ 512b+ >> 1024b+ >> No. of Reads: 17 0 >> 0 >> No. of Writes: 0 43 >> 7 >> >> Block Size: 2048b+ 4096b+ >> 8192b+ >> No. of Reads: 0 7 >> 0 >> No. of Writes: 16 1881 >> 1010 >> >> Block Size: 16384b+ 32768b+ >> 65536b+ >> No. of Reads: 0 0 >> 6 >> No. of Writes: 305 586 >> 2359 >> >> Block Size: 131072b+ >> No. of Reads: 7162 >> No. of Writes: 610 >> %-latency Avg-latency Min-Latency Max-Latency No. of calls >> Fop >> --------- ----------- ----------- ----------- ------------ >> ---- >> 0.00 0.00 us 0.00 us 0.00 us 6 >> RELEASE >> 0.00 0.00 us 0.00 us 0.00 us 20 >> RELEASEDIR >> 0.00 14.50 us 9.00 us 20.00 us 4 >> READDIR >> 0.00 38.00 us 9.00 us 120.00 us 7 >> GETXATTR >> 0.00 66.00 us 34.00 us 128.00 us 6 >> OPEN >> 0.00 137.25 us 52.00 us 195.00 us 4 >> SETATTR >> 0.00 23.19 us 11.00 us 46.00 us 26 >> INODELK >> 0.00 40.05 us 18.00 us 79.00 us 20 >> OPENDIR >> 0.00 180.33 us 16.00 us 775.00 us 21 >> READDIRP >> 0.01 181.77 us 12.00 us 11695.00 us 149 >> STATFS >> 0.01 511.23 us 15.00 us 16954.00 us 111 >> STAT >> 0.01 210.22 us 22.00 us 22944.00 us 291 >> FSTAT >> 0.09 979.73 us 9.00 us 23539.00 us 404 >> LOOKUP >> 0.81 179.58 us 6.00 us 52255.00 us 19786 >> FINODELK >> 1.07 342.71 us 42.00 us 43711.00 us 13633 >> FXATTROP >> 14.18 9087.30 us 213.00 us 213832.00 us 6817 >> WRITE >> 40.00 40227.38 us 259.00 us 157934.00 us 4343 >> FSYNC >> 43.81 26601.57 us 222.00 us 118756.00 us 7192 >> READ >> >> Duration: 177 seconds >> Data Read: 939534945 bytes >> Data Written: 363763200 bytes >> >> Brick: 10.30.30.121:/gluster_bricks/gv2/brick >> --------------------------------------------- >> Cumulative Stats: >> Block Size: 8b+ 256b+ >> 512b+ >> No. of Reads: 0 142626 >> 329092 >> No. of Writes: 12 98 >> 123230 >> >> Block Size: 1024b+ 2048b+ >> 4096b+ >> No. of Reads: 21175 18223 >> 3345043 >> No. of Writes: 58646 69147 >> 6092143 >> >> Block Size: 8192b+ 16384b+ >> 32768b+ >> No. of Reads: 1090937 216037 >> 748595 >> No. of Writes: 2030403 243915 >> 240942 >> >> Block Size: 65536b+ 131072b+ >> No. of Reads: 1524667 4828662 >> No. of Writes: 605125 1889892 >> %-latency Avg-latency Min-Latency Max-Latency No. of calls >> Fop >> --------- ----------- ----------- ----------- ------------ >> ---- >> 0.00 0.00 us 0.00 us 0.00 us 133 >> FORGET >> 0.00 0.00 us 0.00 us 0.00 us 4821 >> RELEASE >> 0.00 0.00 us 0.00 us 0.00 us 42742 >> RELEASEDIR >> 0.00 28.50 us 12.00 us 61.00 us 4 >> READDIR >> 0.00 71.75 us 59.00 us 88.00 us 4 >> SETATTR >> 0.00 44.57 us 12.00 us 94.00 us 7 >> GETXATTR >> 0.00 62.17 us 42.00 us 89.00 us 6 >> OPEN >> 0.00 35.92 us 1.00 us 76.00 us 24 >> OPENDIR >> 0.00 325.54 us 22.00 us 1117.00 us 13 >> READDIRP >> 0.00 482.13 us 19.00 us 12473.00 us 46 >> STAT >> 0.00 217.65 us 15.00 us 32841.00 us 221 >> STATFS >> 0.01 497.52 us 11.00 us 29179.00 us 539 >> LOOKUP >> 0.02 685.44 us 16.00 us 61131.00 us 615 >> FSTAT >> 0.04 31691.92 us 17.00 us 349648.00 us 26 >> INODELK >> 0.45 451.95 us 51.00 us 65375.00 us 20206 >> FXATTROP >> 6.90 13906.03 us 225.00 us 210298.00 us 10103 >> WRITE >> 18.47 58375.96 us 389.00 us 218211.00 us 6446 >> FSYNC >> 27.45 27279.84 us 184.00 us 241444.00 us 20503 >> READ >> 46.66 32462.82 us 7.00 us 2750519.00 us 29283 >> FINODELK >> >> Duration: 475152 seconds >> Data Read: 799630169437 bytes >> Data Written: 362708560796 bytes >> >> Interval 2 Stats: >> Block Size: 256b+ 512b+ >> 1024b+ >> No. of Reads: 53 0 >> 0 >> No. of Writes: 0 43 >> 7 >> >> Block Size: 2048b+ 4096b+ >> 8192b+ >> No. of Reads: 0 891 >> 616 >> No. of Writes: 16 1881 >> 1010 >> >> Block Size: 16384b+ 32768b+ >> 65536b+ >> No. of Reads: 288 750 >> 52 >> No. of Writes: 304 586 >> 2359 >> >> Block Size: 131072b+ >> No. of Reads: 9445 >> No. of Writes: 610 >> %-latency Avg-latency Min-Latency Max-Latency No. of calls >> Fop >> --------- ----------- ----------- ----------- ------------ >> ---- >> 0.00 0.00 us 0.00 us 0.00 us 6 >> RELEASE >> 0.00 0.00 us 0.00 us 0.00 us 20 >> RELEASEDIR >> 0.00 28.50 us 12.00 us 61.00 us 4 >> READDIR >> 0.00 71.75 us 59.00 us 88.00 us 4 >> SETATTR >> 0.00 44.57 us 12.00 us 94.00 us 7 >> GETXATTR >> 0.00 62.17 us 42.00 us 89.00 us 6 >> OPEN >> 0.00 34.50 us 1.00 us 76.00 us 20 >> OPENDIR >> 0.00 55.16 us 19.00 us 128.00 us 25 >> STAT >> 0.00 363.18 us 25.00 us 1117.00 us 11 >> READDIRP >> 0.00 71.61 us 15.00 us 4323.00 us 149 >> STATFS >> 0.01 251.40 us 11.00 us 15344.00 us 336 >> LOOKUP >> 0.02 594.11 us 16.00 us 61131.00 us 450 >> FSTAT >> 0.07 31691.92 us 17.00 us 349648.00 us 26 >> INODELK >> 0.43 384.83 us 51.00 us 65375.00 us 13632 >> FXATTROP >> 7.54 13535.70 us 225.00 us 210298.00 us 6816 >> WRITE >> 20.03 56497.17 us 394.00 us 218211.00 us 4340 >> FSYNC >> 25.91 26229.86 us 184.00 us 241444.00 us 12095 >> READ >> 45.99 28508.86 us 7.00 us 2591280.00 us 19751 >> FINODELK >> >> Duration: 177 seconds >> Data Read: 1298631621 bytes >> Data Written: 363744256 bytes >> >> Brick: 10.30.30.123:/gluster_bricks/gv2/brick >> --------------------------------------------- >> Cumulative Stats: >> Block Size: 1b+ >> No. of Reads: 0 >> No. of Writes: 18126558 >> %-latency Avg-latency Min-Latency Max-Latency No. of calls >> Fop >> --------- ----------- ----------- ----------- ------------ >> ---- >> 0.00 0.00 us 0.00 us 0.00 us 383 >> FORGET >> 0.00 0.00 us 0.00 us 0.00 us 9602 >> RELEASE >> 0.00 0.00 us 0.00 us 0.00 us 70644 >> RELEASEDIR >> 0.00 20.75 us 13.00 us 29.00 us 4 >> READDIR >> 0.00 87.00 us 72.00 us 129.00 us 4 >> SETATTR >> 0.00 71.00 us 46.00 us 93.00 us 6 >> OPEN >> 0.00 66.00 us 9.00 us 114.00 us 7 >> GETXATTR >> 0.00 50.58 us 16.00 us 90.00 us 26 >> INODELK >> 0.00 56.38 us 1.00 us 111.00 us 24 >> OPENDIR >> 0.02 66.23 us 28.00 us 146.00 us 113 >> FSTAT >> 0.17 143.88 us 13.00 us 444.00 us 539 >> LOOKUP >> 1.17 51.64 us 10.00 us 166.00 us 10103 >> WRITE >> 2.89 44.06 us 7.00 us 2177.00 us 29349 >> FINODELK >> 5.55 122.83 us 54.00 us 9281.00 us 20206 >> FXATTROP >> 90.19 6249.64 us 237.00 us 59996.00 us 6448 >> FSYNC >> >> Duration: 772792 seconds >> Data Read: 0 bytes >> Data Written: 18126558 bytes >> >> Interval 2 Stats: >> Block Size: 1b+ >> No. of Reads: 0 >> No. of Writes: 6816 >> %-latency Avg-latency Min-Latency Max-Latency No. of calls >> Fop >> --------- ----------- ----------- ----------- ------------ >> ---- >> 0.00 0.00 us 0.00 us 0.00 us 6 >> RELEASE >> 0.00 0.00 us 0.00 us 0.00 us 20 >> RELEASEDIR >> 0.00 20.75 us 13.00 us 29.00 us 4 >> READDIR >> 0.00 87.00 us 72.00 us 129.00 us 4 >> SETATTR >> 0.00 71.00 us 46.00 us 93.00 us 6 >> OPEN >> 0.00 66.00 us 9.00 us 114.00 us 7 >> GETXATTR >> 0.00 58.10 us 1.00 us 111.00 us 20 >> OPENDIR >> 0.00 50.58 us 16.00 us 90.00 us 26 >> INODELK >> 0.03 66.23 us 28.00 us 146.00 us 113 >> FSTAT >> 0.16 145.84 us 13.00 us 346.00 us 336 >> LOOKUP >> 1.18 51.98 us 11.00 us 157.00 us 6816 >> WRITE >> 2.91 44.05 us 7.00 us 2177.00 us 19798 >> FINODELK >> 5.59 122.70 us 54.00 us 9281.00 us 13632 >> FXATTROP >> 90.11 6210.61 us 238.00 us 54711.00 us 4342 >> FSYNC >> >> Duration: 177 seconds >> Data Read: 0 bytes >> Data Written: 6816 bytes >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/XHAUIMGRCVE... >> >