[ovirt-users] Re: oVirt Performance (Horrific)

Thursday, 7 March 2019

So from the profile, it appears the XATTROPs and FINODELKs are way higher
than the number of WRITEs:

<excerpt>
...
...
%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
 Fop
 ---------   -----------   -----------   -----------   ------------
----
      0.43     384.83 us      51.00 us   65375.00 us          13632
FXATTROP
      7.54   13535.70 us     225.00 us  210298.00 us           6816
 WRITE
     45.99   28508.86 us       7.00 us 2591280.00 us          19751
FINODELK

...
...
</excerpt>

We'd noticed something similar in our internal tests and found
inefficiencies in gluster's eager-lock implementation. This was fixed at
https://review.gluster.org/c/glusterfs/+/19503.
I need the two things I asked for in the prev mail to confirm if you're
hitting the same issue.

-Krutika

On Thu, Mar 7, 2019 at 12:24 PM Krutika Dhananjay <kdhananj(a)redhat.com&gt;
wrote:

...
 Hi,

 Could you share the following pieces of information to begin with -

 1. output of `gluster volume info $AFFECTED_VOLUME_NAME`
 2. glusterfs version you're running

 -Krutika

 On Sat, Mar 2, 2019 at 3:38 AM Drew R <drew.rash(a)gmail.com&gt; wrote:

> Saw some people asking for profile info.  So I had started a migration
> from a 6TB WDGold 2+1arb replicated gluster to a 1TB samsung ssd 2+1 rep
> gluster and it's been running a while for a 100GB file thin provisioned
> with like 28GB actually used.  Here is the profile info.  I started the
> profiler like 5 minutes ago. The migration had been running for like
> 30minutes:
>
> gluster volume profile gv2 info
> Brick: 10.30.30.122:/gluster_bricks/gv2/brick
> ---------------------------------------------
> Cumulative Stats:
>    Block Size:                256b+                 512b+
> 1024b+
>  No. of Reads:                 1189                     8
>     12
> No. of Writes:                    4                  3245
>    883
>
>    Block Size:               2048b+                4096b+
> 8192b+
>  No. of Reads:                   10                    20
>      2
> No. of Writes:                 1087                312228
> 124080
>
>    Block Size:              16384b+               32768b+
>  65536b+
>  No. of Reads:                    0                     1
>     52
> No. of Writes:                 5188                  3617
>   5532
>
>    Block Size:             131072b+
>  No. of Reads:                70191
> No. of Writes:               634192
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
>    Fop
>  ---------   -----------   -----------   -----------   ------------
>   ----
>       0.00       0.00 us       0.00 us       0.00 us              2
> FORGET
>       0.00       0.00 us       0.00 us       0.00 us            202
>  RELEASE
>       0.00       0.00 us       0.00 us       0.00 us           1297
> RELEASEDIR
>       0.00      14.50 us       9.00 us      20.00 us              4
>  READDIR
>       0.00      38.00 us       9.00 us     120.00 us              7
> GETXATTR
>       0.00      66.00 us      34.00 us     128.00 us              6
>   OPEN
>       0.00     137.25 us      52.00 us     195.00 us              4
>  SETATTR
>       0.00      23.19 us      11.00 us      46.00 us             26
>  INODELK
>       0.00      41.58 us      18.00 us      79.00 us             24
>  OPENDIR
>       0.00     166.70 us      15.00 us     775.00 us             27
> READDIRP
>       0.01     135.29 us      12.00 us   11695.00 us            221
> STATFS
>       0.01     176.54 us      22.00 us   22944.00 us            364
>  FSTAT
>       0.02     626.21 us      13.00 us   17308.00 us            168
>   STAT
>       0.09     834.84 us       9.00 us   34337.00 us            607
> LOOKUP
>       0.73     146.18 us       6.00 us   52255.00 us          29329
> FINODELK
>       1.03     298.20 us      42.00 us   43711.00 us          20204
> FXATTROP
>      15.38    8903.40 us     213.00 us  213832.00 us          10102
>  WRITE
>      39.14   26796.37 us     222.00 us  122696.00 us           8538
>   READ
>      43.59   39536.79 us     259.00 us  183630.00 us           6446
>  FSYNC
>
>     Duration: 15078 seconds
>    Data Read: 9207377205 bytes
> Data Written: 86214017762 bytes
>
> Interval 2 Stats:
>    Block Size:                256b+                 512b+
> 1024b+
>  No. of Reads:                   17                     0
>      0
> No. of Writes:                    0                    43
>      7
>
>    Block Size:               2048b+                4096b+
> 8192b+
>  No. of Reads:                    0                     7
>      0
> No. of Writes:                   16                  1881
>   1010
>
>    Block Size:              16384b+               32768b+
>  65536b+
>  No. of Reads:                    0                     0
>      6
> No. of Writes:                  305                   586
>   2359
>
>    Block Size:             131072b+
>  No. of Reads:                 7162
> No. of Writes:                  610
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
>    Fop
>  ---------   -----------   -----------   -----------   ------------
>   ----
>       0.00       0.00 us       0.00 us       0.00 us              6
>  RELEASE
>       0.00       0.00 us       0.00 us       0.00 us             20
> RELEASEDIR
>       0.00      14.50 us       9.00 us      20.00 us              4
>  READDIR
>       0.00      38.00 us       9.00 us     120.00 us              7
> GETXATTR
>       0.00      66.00 us      34.00 us     128.00 us              6
>   OPEN
>       0.00     137.25 us      52.00 us     195.00 us              4
>  SETATTR
>       0.00      23.19 us      11.00 us      46.00 us             26
>  INODELK
>       0.00      40.05 us      18.00 us      79.00 us             20
>  OPENDIR
>       0.00     180.33 us      16.00 us     775.00 us             21
> READDIRP
>       0.01     181.77 us      12.00 us   11695.00 us            149
> STATFS
>       0.01     511.23 us      15.00 us   16954.00 us            111
>   STAT
>       0.01     210.22 us      22.00 us   22944.00 us            291
>  FSTAT
>       0.09     979.73 us       9.00 us   23539.00 us            404
> LOOKUP
>       0.81     179.58 us       6.00 us   52255.00 us          19786
> FINODELK
>       1.07     342.71 us      42.00 us   43711.00 us          13633
> FXATTROP
>      14.18    9087.30 us     213.00 us  213832.00 us           6817
>  WRITE
>      40.00   40227.38 us     259.00 us  157934.00 us           4343
>  FSYNC
>      43.81   26601.57 us     222.00 us  118756.00 us           7192
>   READ
>
>     Duration: 177 seconds
>    Data Read: 939534945 bytes
> Data Written: 363763200 bytes
>
> Brick: 10.30.30.121:/gluster_bricks/gv2/brick
> ---------------------------------------------
> Cumulative Stats:
>    Block Size:                  8b+                 256b+
>  512b+
>  No. of Reads:                    0                142626
> 329092
> No. of Writes:                   12                    98
> 123230
>
>    Block Size:               1024b+                2048b+
> 4096b+
>  No. of Reads:                21175                 18223
>  3345043
> No. of Writes:                58646                 69147
>  6092143
>
>    Block Size:               8192b+               16384b+
>  32768b+
>  No. of Reads:              1090937                216037
> 748595
> No. of Writes:              2030403                243915
> 240942
>
>    Block Size:              65536b+              131072b+
>  No. of Reads:              1524667               4828662
> No. of Writes:               605125               1889892
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
>    Fop
>  ---------   -----------   -----------   -----------   ------------
>   ----
>       0.00       0.00 us       0.00 us       0.00 us            133
> FORGET
>       0.00       0.00 us       0.00 us       0.00 us           4821
>  RELEASE
>       0.00       0.00 us       0.00 us       0.00 us          42742
> RELEASEDIR
>       0.00      28.50 us      12.00 us      61.00 us              4
>  READDIR
>       0.00      71.75 us      59.00 us      88.00 us              4
>  SETATTR
>       0.00      44.57 us      12.00 us      94.00 us              7
> GETXATTR
>       0.00      62.17 us      42.00 us      89.00 us              6
>   OPEN
>       0.00      35.92 us       1.00 us      76.00 us             24
>  OPENDIR
>       0.00     325.54 us      22.00 us    1117.00 us             13
> READDIRP
>       0.00     482.13 us      19.00 us   12473.00 us             46
>   STAT
>       0.00     217.65 us      15.00 us   32841.00 us            221
> STATFS
>       0.01     497.52 us      11.00 us   29179.00 us            539
> LOOKUP
>       0.02     685.44 us      16.00 us   61131.00 us            615
>  FSTAT
>       0.04   31691.92 us      17.00 us  349648.00 us             26
>  INODELK
>       0.45     451.95 us      51.00 us   65375.00 us          20206
> FXATTROP
>       6.90   13906.03 us     225.00 us  210298.00 us          10103
>  WRITE
>      18.47   58375.96 us     389.00 us  218211.00 us           6446
>  FSYNC
>      27.45   27279.84 us     184.00 us  241444.00 us          20503
>   READ
>      46.66   32462.82 us       7.00 us 2750519.00 us          29283
> FINODELK
>
>     Duration: 475152 seconds
>    Data Read: 799630169437 bytes
> Data Written: 362708560796 bytes
>
> Interval 2 Stats:
>    Block Size:                256b+                 512b+
> 1024b+
>  No. of Reads:                   53                     0
>      0
> No. of Writes:                    0                    43
>      7
>
>    Block Size:               2048b+                4096b+
> 8192b+
>  No. of Reads:                    0                   891
>    616
> No. of Writes:                   16                  1881
>   1010
>
>    Block Size:              16384b+               32768b+
>  65536b+
>  No. of Reads:                  288                   750
>     52
> No. of Writes:                  304                   586
>   2359
>
>    Block Size:             131072b+
>  No. of Reads:                 9445
> No. of Writes:                  610
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
>    Fop
>  ---------   -----------   -----------   -----------   ------------
>   ----
>       0.00       0.00 us       0.00 us       0.00 us              6
>  RELEASE
>       0.00       0.00 us       0.00 us       0.00 us             20
> RELEASEDIR
>       0.00      28.50 us      12.00 us      61.00 us              4
>  READDIR
>       0.00      71.75 us      59.00 us      88.00 us              4
>  SETATTR
>       0.00      44.57 us      12.00 us      94.00 us              7
> GETXATTR
>       0.00      62.17 us      42.00 us      89.00 us              6
>   OPEN
>       0.00      34.50 us       1.00 us      76.00 us             20
>  OPENDIR
>       0.00      55.16 us      19.00 us     128.00 us             25
>   STAT
>       0.00     363.18 us      25.00 us    1117.00 us             11
> READDIRP
>       0.00      71.61 us      15.00 us    4323.00 us            149
> STATFS
>       0.01     251.40 us      11.00 us   15344.00 us            336
> LOOKUP
>       0.02     594.11 us      16.00 us   61131.00 us            450
>  FSTAT
>       0.07   31691.92 us      17.00 us  349648.00 us             26
>  INODELK
>       0.43     384.83 us      51.00 us   65375.00 us          13632
> FXATTROP
>       7.54   13535.70 us     225.00 us  210298.00 us           6816
>  WRITE
>      20.03   56497.17 us     394.00 us  218211.00 us           4340
>  FSYNC
>      25.91   26229.86 us     184.00 us  241444.00 us          12095
>   READ
>      45.99   28508.86 us       7.00 us 2591280.00 us          19751
> FINODELK
>
>     Duration: 177 seconds
>    Data Read: 1298631621 bytes
> Data Written: 363744256 bytes
>
> Brick: 10.30.30.123:/gluster_bricks/gv2/brick
> ---------------------------------------------
> Cumulative Stats:
>    Block Size:                  1b+
>  No. of Reads:                    0
> No. of Writes:             18126558
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
>    Fop
>  ---------   -----------   -----------   -----------   ------------
>   ----
>       0.00       0.00 us       0.00 us       0.00 us            383
> FORGET
>       0.00       0.00 us       0.00 us       0.00 us           9602
>  RELEASE
>       0.00       0.00 us       0.00 us       0.00 us          70644
> RELEASEDIR
>       0.00      20.75 us      13.00 us      29.00 us              4
>  READDIR
>       0.00      87.00 us      72.00 us     129.00 us              4
>  SETATTR
>       0.00      71.00 us      46.00 us      93.00 us              6
>   OPEN
>       0.00      66.00 us       9.00 us     114.00 us              7
> GETXATTR
>       0.00      50.58 us      16.00 us      90.00 us             26
>  INODELK
>       0.00      56.38 us       1.00 us     111.00 us             24
>  OPENDIR
>       0.02      66.23 us      28.00 us     146.00 us            113
>  FSTAT
>       0.17     143.88 us      13.00 us     444.00 us            539
> LOOKUP
>       1.17      51.64 us      10.00 us     166.00 us          10103
>  WRITE
>       2.89      44.06 us       7.00 us    2177.00 us          29349
> FINODELK
>       5.55     122.83 us      54.00 us    9281.00 us          20206
> FXATTROP
>      90.19    6249.64 us     237.00 us   59996.00 us           6448
>  FSYNC
>
>     Duration: 772792 seconds
>    Data Read: 0 bytes
> Data Written: 18126558 bytes
>
> Interval 2 Stats:
>    Block Size:                  1b+
>  No. of Reads:                    0
> No. of Writes:                 6816
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
>    Fop
>  ---------   -----------   -----------   -----------   ------------
>   ----
>       0.00       0.00 us       0.00 us       0.00 us              6
>  RELEASE
>       0.00       0.00 us       0.00 us       0.00 us             20
> RELEASEDIR
>       0.00      20.75 us      13.00 us      29.00 us              4
>  READDIR
>       0.00      87.00 us      72.00 us     129.00 us              4
>  SETATTR
>       0.00      71.00 us      46.00 us      93.00 us              6
>   OPEN
>       0.00      66.00 us       9.00 us     114.00 us              7
> GETXATTR
>       0.00      58.10 us       1.00 us     111.00 us             20
>  OPENDIR
>       0.00      50.58 us      16.00 us      90.00 us             26
>  INODELK
>       0.03      66.23 us      28.00 us     146.00 us            113
>  FSTAT
>       0.16     145.84 us      13.00 us     346.00 us            336
> LOOKUP
>       1.18      51.98 us      11.00 us     157.00 us           6816
>  WRITE
>       2.91      44.05 us       7.00 us    2177.00 us          19798
> FINODELK
>       5.59     122.70 us      54.00 us    9281.00 us          13632
> FXATTROP
>      90.11    6210.61 us     238.00 us   54711.00 us           4342
>  FSYNC
>
>     Duration: 177 seconds
>    Data Read: 0 bytes
> Data Written: 6816 bytes
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/XHAUIMGRCVE...
>

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

[ovirt-users] Re: oVirt Performance (Horrific)