seq-write: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8 ... seq-read: (g=1): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8 ... rand-write-qd32: (g=2): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32 ... rand-read-qd32: (g=3): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32 ... rand-write-qd1: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 rand-read-qd1: (g=5): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.16 Starting 18 processes seq-write: Laying out IO file (1 file / 10240MiB) seq-write: (groupid=0, jobs=4): err= 0: pid=46235: Thu Sep 9 14:18:05 2021 write: IOPS=57, BW=58.8MiB/s (61.6MB/s)(1792MiB/30492msec); 0 zone resets slat (usec): min=53, max=335320, avg=1692.05, stdev=21085.55 clat (msec): min=49, max=2472, avg=557.25, stdev=404.74 lat (msec): min=49, max=2473, avg=558.92, stdev=407.28 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 247], 10.00th=[ 321], 20.00th=[ 338], | 30.00th=[ 342], 40.00th=[ 342], 50.00th=[ 342], 60.00th=[ 351], | 70.00th=[ 472], 80.00th=[ 877], 90.00th=[ 1301], 95.00th=[ 1418], | 99.00th=[ 1636], 99.50th=[ 1838], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7391, max=111568, per=100.00%, avg=71150.39, stdev=5677.70, samples=183 iops : min= 4, max= 108, avg=68.11, stdev= 5.58, samples=183 lat (msec) : 50=0.06%, 100=0.51%, 250=4.65%, 500=67.40%, 750=6.41% lat (msec) : 1000=4.37%, 2000=17.91%, >=2000=0.28% cpu : usr=0.12%, sys=0.13%, ctx=1648, majf=0, minf=60 IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=99.8%, 8=0.2%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1764,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 seq-read: (groupid=1, jobs=4): err= 0: pid=46315: Thu Sep 9 14:18:05 2021 read: IOPS=5100, BW=5101MiB/s (5349MB/s)(150GiB/30111msec) slat (usec): min=26, max=4205, avg=86.91, stdev=38.55 clat (nsec): min=815, max=332475k, avg=6186377.89, stdev=26542039.96 lat (usec): min=55, max=332517, avg=6273.58, stdev=26538.34 clat percentiles (nsec): | 1.00th=[ 980], 5.00th=[ 78336], 10.00th=[ 238592], | 20.00th=[ 473088], 30.00th=[ 505856], 40.00th=[ 536576], | 50.00th=[ 577536], 60.00th=[ 634880], 70.00th=[ 692224], | 80.00th=[ 765952], 90.00th=[ 954368], 95.00th=[ 25559040], | 99.00th=[154140672], 99.50th=[177209344], 99.90th=[256901120], | 99.95th=[274726912], 99.99th=[299892736] bw ( MiB/s): min= 121, max=48079, per=98.89%, avg=5044.70, stdev=3000.15, samples=240 iops : min= 120, max=48079, avg=5043.07, stdev=3000.18, samples=240 lat (nsec) : 1000=1.33% lat (usec) : 2=1.38%, 4=0.03%, 10=0.01%, 20=0.02%, 50=0.01% lat (usec) : 100=2.94%, 250=4.61%, 500=17.75%, 750=49.89%, 1000=12.82% lat (msec) : 2=2.81%, 4=0.06%, 10=0.16%, 20=1.01%, 50=0.68% lat (msec) : 100=1.85%, 250=2.54%, 500=0.12% cpu : usr=0.34%, sys=11.26%, ctx=10087, majf=0, minf=60 IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=153582,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 rand-write-qd32: (groupid=2, jobs=4): err= 0: pid=46330: Thu Sep 9 14:18:05 2021 write: IOPS=299, BW=1215KiB/s (1244kB/s)(35.8MiB/30212msec); 0 zone resets slat (usec): min=5, max=167756, avg=368.13, stdev=6422.10 clat (msec): min=22, max=3049, avg=422.46, stdev=559.64 lat (msec): min=22, max=3049, avg=422.82, stdev=560.40 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 97], 10.00th=[ 103], 20.00th=[ 157], | 30.00th=[ 209], 40.00th=[ 220], 50.00th=[ 226], 60.00th=[ 234], | 70.00th=[ 251], 80.00th=[ 359], 90.00th=[ 1003], 95.00th=[ 1787], | 99.00th=[ 2769], 99.50th=[ 3037], 99.90th=[ 3037], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min= 87, max= 2750, per=100.00%, avg=1665.82, stdev=222.17, samples=159 iops : min= 20, max= 686, avg=415.20, stdev=55.54, samples=159 lat (msec) : 50=1.04%, 100=6.55%, 250=63.15%, 500=12.55%, 750=2.97% lat (msec) : 1000=4.97%, 2000=5.37%, >=2000=4.76% cpu : usr=0.08%, sys=0.29%, ctx=9014, majf=0, minf=60 IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=0,9050,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 rand-read-qd32: (groupid=3, jobs=4): err= 0: pid=46538: Thu Sep 9 14:18:05 2021 read: IOPS=456k, BW=1780MiB/s (1867MB/s)(52.2GiB/30005msec) slat (nsec): min=1610, max=18410k, avg=5101.31, stdev=87261.28 clat (nsec): min=683, max=263644k, avg=274516.14, stdev=1220977.05 lat (usec): min=2, max=263649, avg=279.84, stdev=1225.27 clat percentiles (nsec): | 1.00th=[ 884], 5.00th=[ 5216], 10.00th=[ 12992], | 20.00th=[ 28288], 30.00th=[ 45824], 40.00th=[ 67072], | 50.00th=[ 89600], 60.00th=[ 112128], 70.00th=[ 134144], | 80.00th=[ 162816], 90.00th=[ 222208], 95.00th=[ 937984], | 99.00th=[ 4292608], 99.50th=[ 8093696], 99.90th=[10420224], | 99.95th=[12124160], 99.99th=[14221312] bw ( MiB/s): min= 865, max= 2219, per=98.89%, avg=1760.43, stdev=54.09, samples=236 iops : min=221460, max=568171, avg=450667.59, stdev=13847.97, samples=236 lat (nsec) : 750=0.01%, 1000=2.38% lat (usec) : 2=0.73%, 4=0.01%, 10=5.33%, 20=6.17%, 50=17.40% lat (usec) : 100=22.47%, 250=37.21%, 500=1.16%, 750=1.25%, 1000=1.06% lat (msec) : 2=1.47%, 4=2.14%, 10=1.10%, 20=0.12%, 50=0.01% lat (msec) : 500=0.01% cpu : usr=26.98%, sys=44.88%, ctx=37869, majf=0, minf=60 IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=13674271,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 rand-write-qd1: (groupid=4, jobs=1): err= 0: pid=46552: Thu Sep 9 14:18:05 2021 write: IOPS=213, BW=854KiB/s (875kB/s)(25.0MiB/30003msec); 0 zone resets slat (usec): min=14, max=10560, avg=38.97, stdev=184.01 clat (usec): min=599, max=2263.4k, avg=4638.59, stdev=59351.46 lat (usec): min=673, max=2270.0k, avg=4678.12, stdev=59482.51 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 824], 10.00th=[ 898], | 20.00th=[ 2180], 30.00th=[ 2311], 40.00th=[ 2376], | 50.00th=[ 2474], 60.00th=[ 2900], 70.00th=[ 3032], | 80.00th=[ 3130], 90.00th=[ 3228], 95.00th=[ 3326], | 99.00th=[ 3589], 99.50th=[ 4047], 99.90th=[ 265290], | 99.95th=[1853883], 99.99th=[2264925] bw ( KiB/s): min= 7, max= 1696, per=100.00%, avg=1249.83, stdev=622.40, samples=41 iops : min= 1, max= 424, avg=312.44, stdev=155.64, samples=41 lat (usec) : 750=1.14%, 1000=13.53% lat (msec) : 2=0.92%, 4=83.90%, 10=0.08%, 20=0.05%, 50=0.03% lat (msec) : 100=0.09%, 250=0.14%, 500=0.03%, 2000=0.05%, >=2000=0.05% cpu : usr=0.14%, sys=0.85%, ctx=6432, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,6409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 rand-read-qd1: (groupid=5, jobs=1): err= 0: pid=46565: Thu Sep 9 14:18:05 2021 read: IOPS=20.4k, BW=79.8MiB/s (83.7MB/s)(2396MiB/30001msec) slat (nsec): min=1509, max=747904, avg=3997.22, stdev=4935.97 clat (nsec): min=663, max=584288k, avg=44036.11, stdev=762638.81 lat (usec): min=2, max=584300, avg=48.18, stdev=763.33 clat percentiles (nsec): | 1.00th=[ 676], 5.00th=[ 684], 10.00th=[ 692], | 20.00th=[ 724], 30.00th=[ 740], 40.00th=[ 756], | 50.00th=[ 780], 60.00th=[ 812], 70.00th=[ 972], | 80.00th=[ 1704], 90.00th=[ 1864], 95.00th=[ 489472], | 99.00th=[ 782336], 99.50th=[ 823296], 99.90th=[ 888832], | 99.95th=[ 913408], 99.99th=[1019904] bw ( KiB/s): min=29208, max=91600, per=99.93%, avg=81709.86, stdev=9983.04, samples=59 iops : min= 7302, max=22900, avg=20427.44, stdev=2495.74, samples=59 lat (nsec) : 750=35.16%, 1000=35.73% lat (usec) : 2=19.71%, 4=1.79%, 10=0.04%, 20=0.43%, 50=0.05% lat (usec) : 100=0.01%, 250=0.01%, 500=2.24%, 750=3.41%, 1000=1.41% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 750=0.01% cpu : usr=5.37%, sys=8.85%, ctx=43635, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=613263,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=58.8MiB/s (61.6MB/s), 58.8MiB/s-58.8MiB/s (61.6MB/s-61.6MB/s), io=1792MiB (1879MB), run=30492-30492msec Run status group 1 (all jobs): READ: bw=5101MiB/s (5349MB/s), 5101MiB/s-5101MiB/s (5349MB/s-5349MB/s), io=150GiB (161GB), run=30111-30111msec Run status group 2 (all jobs): WRITE: bw=1215KiB/s (1244kB/s), 1215KiB/s-1215KiB/s (1244kB/s-1244kB/s), io=35.8MiB (37.6MB), run=30212-30212msec Run status group 3 (all jobs): READ: bw=1780MiB/s (1867MB/s), 1780MiB/s-1780MiB/s (1867MB/s-1867MB/s), io=52.2GiB (56.0GB), run=30005-30005msec Run status group 4 (all jobs): WRITE: bw=854KiB/s (875kB/s), 854KiB/s-854KiB/s (875kB/s-875kB/s), io=25.0MiB (26.3MB), run=30003-30003msec Run status group 5 (all jobs): READ: bw=79.8MiB/s (83.7MB/s), 79.8MiB/s-79.8MiB/s (83.7MB/s-83.7MB/s), io=2396MiB (2512MB), run=30001-30001msec Disk stats (read/write): sda: ios=1308653/25511, merge=446/2810, ticks=4360315/7045149, in_queue=11467004, util=97.78%