seq-write: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8 ... seq-read: (g=1): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8 ... rand-write-qd32: (g=2): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32 ... rand-read-qd32: (g=3): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32 ... rand-write-qd1: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 rand-read-qd1: (g=5): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.19 Starting 18 processes seq-write: Laying out IO file (1 file / 10240MiB) seq-write: (groupid=0, jobs=4): err= 0: pid=294433: Thu Sep 9 14:30:14 2021 write: IOPS=151, BW=153MiB/s (160MB/s)(4628MiB/30280msec); 0 zone resets slat (usec): min=43, max=826750, avg=2541.28, stdev=26317.99 clat (msec): min=4, max=1103, avg=207.56, stdev=169.55 lat (msec): min=4, max=1103, avg=210.09, stdev=170.68 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 31], 10.00th=[ 36], 20.00th=[ 53], | 30.00th=[ 92], 40.00th=[ 133], 50.00th=[ 174], 60.00th=[ 209], | 70.00th=[ 255], 80.00th=[ 317], 90.00th=[ 447], 95.00th=[ 558], | 99.00th=[ 818], 99.50th=[ 844], 99.90th=[ 1011], 99.95th=[ 1070], | 99.99th=[ 1099] bw ( KiB/s): min= 8172, max=433308, per=100.00%, avg=160585.97, stdev=20005.36, samples=232 iops : min= 4, max= 420, avg=153.17, stdev=19.58, samples=232 lat (msec) : 10=0.13%, 20=0.72%, 50=18.22%, 100=12.85%, 250=37.87% lat (msec) : 500=23.85%, 750=5.72%, 1000=1.15%, 2000=0.11% cpu : usr=0.27%, sys=2.70%, ctx=2384, majf=0, minf=772 IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4600,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 seq-read: (groupid=1, jobs=4): err= 0: pid=294778: Thu Sep 9 14:30:14 2021 read: IOPS=7084, BW=7086MiB/s (7430MB/s)(208GiB/30016msec) slat (usec): min=6, max=118317, avg=68.62, stdev=596.43 clat (nsec): min=839, max=144267k, avg=4447719.83, stdev=10105727.60 lat (usec): min=57, max=144320, avg=4516.54, stdev=10126.98 clat percentiles (usec): | 1.00th=[ 178], 5.00th=[ 408], 10.00th=[ 408], 20.00th=[ 416], | 30.00th=[ 433], 40.00th=[ 445], 50.00th=[ 457], 60.00th=[ 474], | 70.00th=[ 494], 80.00th=[ 537], 90.00th=[ 29492], 95.00th=[ 31065], | 99.00th=[ 32113], 99.50th=[ 33162], 99.90th=[ 35914], 99.95th=[ 39060], | 99.99th=[125305] bw ( MiB/s): min= 826, max=58638, per=99.77%, avg=7069.26, stdev=3497.32, samples=236 iops : min= 824, max=58637, avg=7067.41, stdev=3497.32, samples=236 lat (nsec) : 1000=0.14% lat (usec) : 2=0.08%, 4=0.01%, 20=0.01%, 100=0.46%, 250=0.75% lat (usec) : 500=71.15%, 750=10.75%, 1000=0.36% lat (msec) : 2=1.89%, 4=0.02%, 10=0.63%, 20=0.84%, 50=12.90% lat (msec) : 100=0.01%, 250=0.02% cpu : usr=0.41%, sys=11.20%, ctx=30465, majf=0, minf=3390 IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=212660,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 rand-write-qd32: (groupid=2, jobs=4): err= 0: pid=295141: Thu Sep 9 14:30:14 2021 write: IOPS=228, BW=928KiB/s (951kB/s)(28.1MiB/30971msec); 0 zone resets slat (usec): min=8, max=800895, avg=1067.23, stdev=24781.44 clat (msec): min=11, max=1565, avg=554.93, stdev=256.34 lat (msec): min=12, max=1565, avg=555.98, stdev=256.45 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 157], 10.00th=[ 220], 20.00th=[ 347], | 30.00th=[ 422], 40.00th=[ 481], 50.00th=[ 527], 60.00th=[ 592], | 70.00th=[ 667], 80.00th=[ 776], 90.00th=[ 911], 95.00th=[ 986], | 99.00th=[ 1234], 99.50th=[ 1267], 99.90th=[ 1284], 99.95th=[ 1502], | 99.99th=[ 1569] bw ( KiB/s): min= 28, max= 2944, per=100.00%, avg=932.10, stdev=115.69, samples=240 iops : min= 4, max= 734, avg=230.08, stdev=28.95, samples=240 lat (msec) : 20=0.14%, 50=1.29%, 100=1.46%, 250=8.69%, 500=34.11% lat (msec) : 750=33.73%, 1000=18.02%, 2000=4.32% cpu : usr=0.03%, sys=0.21%, ctx=3729, majf=0, minf=233 IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=0,7065,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 rand-read-qd32: (groupid=3, jobs=4): err= 0: pid=296094: Thu Sep 9 14:30:14 2021 read: IOPS=552k, BW=2157MiB/s (2262MB/s)(63.2GiB/30001msec) slat (nsec): min=1356, max=508918k, avg=5192.02, stdev=265716.52 clat (nsec): min=640, max=1721.8M, avg=229054.00, stdev=4972066.22 lat (usec): min=2, max=1721.9k, avg=234.33, stdev=4979.42 clat percentiles (nsec): | 1.00th=[ 692], 5.00th=[ 7392], 10.00th=[ 31360], | 20.00th=[ 92672], 30.00th=[ 128512], 40.00th=[ 148480], | 50.00th=[ 164864], 60.00th=[ 181248], 70.00th=[ 199680], | 80.00th=[ 228352], 90.00th=[ 280576], 95.00th=[ 321536], | 99.00th=[ 634880], 99.50th=[ 1155072], 99.90th=[ 2007040], | 99.95th=[ 2244608], 99.99th=[173015040] bw ( MiB/s): min= 829, max= 2529, per=100.00%, avg=2157.05, stdev=111.87, samples=236 iops : min=212434, max=647558, avg=552203.37, stdev=28639.46, samples=236 lat (nsec) : 750=2.36%, 1000=0.32% lat (usec) : 2=0.10%, 4=1.10%, 10=1.73%, 20=2.19%, 50=5.42% lat (usec) : 100=8.22%, 250=63.64%, 500=13.60%, 750=0.48%, 1000=0.22% lat (msec) : 2=0.51%, 4=0.07%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=17.44%, sys=60.65%, ctx=75348, majf=0, minf=527 IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=16564797,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 rand-write-qd1: (groupid=4, jobs=1): err= 0: pid=296386: Thu Sep 9 14:30:14 2021 write: IOPS=55, BW=223KiB/s (229kB/s)(6696KiB/30002msec); 0 zone resets slat (usec): min=20, max=81257, avg=190.26, stdev=3316.34 clat (usec): min=194, max=105690, avg=17727.89, stdev=13096.14 lat (usec): min=233, max=142461, avg=17918.55, stdev=13680.78 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 3982], 10.00th=[ 6128], 20.00th=[ 8586], | 30.00th=[ 10290], 40.00th=[ 12387], 50.00th=[ 14484], 60.00th=[ 16450], | 70.00th=[ 20055], 80.00th=[ 23987], 90.00th=[ 33162], 95.00th=[ 45351], | 99.00th=[ 69731], 99.50th=[ 77071], 99.90th=[102237], 99.95th=[105382], | 99.99th=[105382] bw ( KiB/s): min= 111, max= 319, per=99.35%, avg=221.54, stdev=46.34, samples=59 iops : min= 27, max= 79, avg=54.66, stdev=11.60, samples=59 lat (usec) : 250=0.06%, 500=0.42%, 750=0.12%, 1000=0.06% lat (msec) : 2=0.54%, 4=3.88%, 10=23.06%, 20=42.00%, 50=26.28% lat (msec) : 100=3.46%, 250=0.12% cpu : usr=0.09%, sys=0.36%, ctx=1677, majf=0, minf=58 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1674,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 rand-read-qd1: (groupid=5, jobs=1): err= 0: pid=296633: Thu Sep 9 14:30:14 2021 read: IOPS=39.4k, BW=154MiB/s (161MB/s)(4617MiB/30001msec) slat (nsec): min=1353, max=99082k, avg=3917.01, stdev=137827.66 clat (nsec): min=640, max=60535k, avg=20869.92, stdev=194315.47 lat (usec): min=2, max=99181, avg=24.88, stdev=239.10 clat percentiles (nsec): | 1.00th=[ 644], 5.00th=[ 652], 10.00th=[ 652], | 20.00th=[ 652], 30.00th=[ 660], 40.00th=[ 660], | 50.00th=[ 684], 60.00th=[ 716], 70.00th=[ 1592], | 80.00th=[ 1624], 90.00th=[ 105984], 95.00th=[ 118272], | 99.00th=[ 136192], 99.50th=[ 138240], 99.90th=[ 150528], | 99.95th=[ 160768], 99.99th=[9633792] bw ( KiB/s): min=116487, max=172718, per=99.98%, avg=157540.78, stdev=12396.80, samples=59 iops : min=29121, max=43179, avg=39384.81, stdev=3099.20, samples=59 lat (nsec) : 750=64.74%, 1000=3.13% lat (usec) : 2=16.07%, 4=0.12%, 10=0.04%, 20=0.15%, 50=0.01% lat (usec) : 100=4.83%, 250=10.87%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01% cpu : usr=5.78%, sys=12.61%, ctx=186050, majf=0, minf=61 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1181893,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=4628MiB (4853MB), run=30280-30280msec Run status group 1 (all jobs): READ: bw=7086MiB/s (7430MB/s), 7086MiB/s-7086MiB/s (7430MB/s-7430MB/s), io=208GiB (223GB), run=30016-30016msec Run status group 2 (all jobs): WRITE: bw=928KiB/s (951kB/s), 928KiB/s-928KiB/s (951kB/s-951kB/s), io=28.1MiB (29.4MB), run=30971-30971msec Run status group 3 (all jobs): READ: bw=2157MiB/s (2262MB/s), 2157MiB/s-2157MiB/s (2262MB/s-2262MB/s), io=63.2GiB (67.8GB), run=30001-30001msec Run status group 4 (all jobs): WRITE: bw=223KiB/s (229kB/s), 223KiB/s-223KiB/s (229kB/s-229kB/s), io=6696KiB (6857kB), run=30002-30002msec Run status group 5 (all jobs): READ: bw=154MiB/s (161MB/s), 154MiB/s-154MiB/s (161MB/s-161MB/s), io=4617MiB (4841MB), run=30001-30001msec Disk stats (read/write): dm-21: ios=3261529/26212, merge=0/0, ticks=4094670/7275235, in_queue=11369905, util=95.25%, aggrios=3262333/26499, aggrmerge=0/0, aggrticks=4053810/7251327, aggrin_queue=11305137, aggrutil=94.42% dm-18: ios=3262333/26499, merge=0/0, ticks=4053810/7251327, in_queue=11305137, util=94.42%, aggrios=1663895/18267, aggrmerge=0/0, aggrticks=2527558/4306360, aggrin_queue=6833918, aggrutil=94.36% dm-17: ios=3327737/36463, merge=0/0, ticks=5051948/8609589, in_queue=13661537, util=94.36%, aggrios=1113206/27777, aggrmerge=0/0, aggrticks=2259269/2773040, aggrin_queue=5032309, aggrutil=65.79% dm-15: ios=1286/24248, merge=0/0, ticks=1051/11620, in_queue=12671, util=2.65%, aggrios=3313657/47664, aggrmerge=61/1480, aggrticks=3137631/53760, aggrin_queue=3191390, aggrutil=52.34% dm-4: ios=3313657/47664, merge=61/1480, ticks=3137631/53760, in_queue=3191390, util=52.34%, aggrios=3313669/47664, aggrmerge=0/0, aggrticks=3121966/51507, aggrin_queue=3173472, aggrutil=52.24% sdd: ios=3313669/47664, merge=0/0, ticks=3121966/51507, in_queue=3173472, util=52.24% dm-16: ios=25901/34187, merge=0/0, ticks=3632707/8262185, in_queue=11894892, util=65.79%, aggrios=27671/37083, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00% md125: ios=27671/37083, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=15220/25759, aggrmerge=58800/823861, aggrticks=1658275/1124632, aggrin_queue=2782907, aggrutil=51.64% sdf: ios=15090/25795, merge=60713/822931, ticks=1285984/1173500, in_queue=2459484, util=50.38% sde: ios=15300/25639, merge=56966/822517, ticks=1638879/926670, in_queue=2565548, util=48.12% sdc: ios=15272/25843, merge=58721/826137, ticks=2049963/1273728, in_queue=3323690, util=51.64% dm-14: ios=3312432/24896, merge=0/0, ticks=3144050/45315, in_queue=3189365, util=51.88% dm-13: ios=53/72, merge=0/0, ticks=3169/3131, in_queue=6300, util=2.39%