##### Linear read: test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=4954: Fri Jan 8 10:15:01 2021 read: IOPS=124, BW=500MiB/s (524MB/s)(29.3GiB/60132msec) slat (usec): min=299, max=216583, avg=7980.52, stdev=5377.29 clat (msec): min=124, max=468, avg=247.98, stdev=30.38 lat (msec): min=125, max=477, avg=255.96, stdev=30.88 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 243], 10.00th=[ 243], 20.00th=[ 243], | 30.00th=[ 243], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 243], | 70.00th=[ 243], 80.00th=[ 245], 90.00th=[ 251], 95.00th=[ 257], | 99.00th=[ 447], 99.50th=[ 451], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min=294912, max=532480, per=99.79%, avg=510685.62, stdev=44607.45, samples=120 iops : min= 72, max= 130, avg=124.67, stdev=10.89, samples=120 lat (msec) : 250=88.35%, 500=11.65% cpu : usr=0.11%, sys=1.15%, ctx=15018, majf=0, minf=585 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=7513,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): READ: bw=500MiB/s (524MB/s), 500MiB/s-500MiB/s (524MB/s-524MB/s), io=29.3GiB (31.5GB), run=60132-60132msec Disk stats (read/write): sdc: ios=120206/0, merge=0/0, ticks=15163099/0, in_queue=1924080, util=99.91% ##### Linear write: test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=5031: Fri Jan 8 10:16:02 2021 write: IOPS=101, BW=404MiB/s (424MB/s)(23.8GiB/60357msec); 0 zone resets slat (usec): min=91, max=211458, avg=9832.72, stdev=6335.07 clat (msec): min=298, max=643, avg=306.30, stdev=36.22 lat (msec): min=308, max=652, avg=316.13, stdev=36.65 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 584], 99.95th=[ 617], | 99.99th=[ 642] bw ( KiB/s): min=122880, max=425984, per=100.00%, avg=414084.32, stdev=43754.76, samples=120 iops : min= 30, max= 104, avg=101.08, stdev=10.68, samples=120 lat (msec) : 500=97.11%, 750=2.89% cpu : usr=1.86%, sys=0.73%, ctx=12192, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.3%, 32=99.5%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=0,6098,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): WRITE: bw=404MiB/s (424MB/s), 404MiB/s-404MiB/s (424MB/s-424MB/s), io=23.8GiB (25.6GB), run=60357-60357msec Disk stats (read/write): sdc: ios=43/97561, merge=0/0, ticks=12/15196965, in_queue=1931368, util=99.87% #### Peak parallel random read: test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=5325: Fri Jan 8 10:17:02 2021 read: IOPS=55.1k, BW=215MiB/s (226MB/s)(12.6GiB/60003msec) slat (nsec): min=1395, max=73180, avg=2194.08, stdev=1389.08 clat (usec): min=104, max=234906, avg=2321.06, stdev=2442.96 lat (usec): min=106, max=234908, avg=2323.32, stdev=2442.96 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 570], 10.00th=[ 766], 20.00th=[ 1123], | 30.00th=[ 1450], 40.00th=[ 1778], 50.00th=[ 2073], 60.00th=[ 2376], | 70.00th=[ 2704], 80.00th=[ 3130], 90.00th=[ 4146], 95.00th=[ 5080], | 99.00th=[ 6521], 99.50th=[ 8291], 99.90th=[14091], 99.95th=[16712], | 99.99th=[24511] bw ( KiB/s): min=124496, max=292200, per=100.00%, avg=220307.37, stdev=48683.11, samples=120 iops : min=31124, max=73050, avg=55076.87, stdev=12170.77, samples=120 lat (usec) : 250=0.17%, 500=3.16%, 750=6.23%, 1000=6.93% lat (msec) : 2=30.90%, 4=41.46%, 10=10.83%, 20=0.29%, 50=0.01% lat (msec) : 250=0.01% cpu : usr=7.55%, sys=15.38%, ctx=3232576, majf=0, minf=137 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=3304891,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): READ: bw=215MiB/s (226MB/s), 215MiB/s-215MiB/s (226MB/s-226MB/s), io=12.6GiB (13.5GB), run=60003-60003msec Disk stats (read/write): sdc: ios=3298122/0, merge=0/0, ticks=7647186/0, in_queue=1802140, util=98.87% #### Single-threaded read latency: test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=5465: Fri Jan 8 10:18:03 2021 read: IOPS=9443, BW=36.9MiB/s (38.7MB/s)(2213MiB/60001msec) slat (nsec): min=2050, max=41189, avg=3883.80, stdev=2939.43 clat (usec): min=37, max=221787, avg=101.24, stdev=295.26 lat (usec): min=51, max=221789, avg=105.20, stdev=295.28 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 79], | 30.00th=[ 84], 40.00th=[ 95], 50.00th=[ 95], 60.00th=[ 100], | 70.00th=[ 126], 80.00th=[ 127], 90.00th=[ 133], 95.00th=[ 133], | 99.00th=[ 141], 99.50th=[ 145], 99.90th=[ 176], 99.95th=[ 192], | 99.99th=[ 235] bw ( KiB/s): min=22248, max=49208, per=100.00%, avg=37777.96, stdev=2700.14, samples=119 iops : min= 5562, max=12302, avg=9444.49, stdev=675.04, samples=119 lat (usec) : 50=0.01%, 100=60.27%, 250=39.72%, 500=0.01%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 250=0.01% cpu : usr=2.55%, sys=5.24%, ctx=566651, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=566646,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=36.9MiB/s (38.7MB/s), 36.9MiB/s-36.9MiB/s (38.7MB/s-38.7MB/s), io=2213MiB (2321MB), run=60001-60001msec Disk stats (read/write): sdc: ios=565411/0, merge=0/0, ticks=57134/0, in_queue=56212, util=93.79% #### Peak parallel random write: test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=5630: Fri Jan 8 10:19:03 2021 write: IOPS=62.1k, BW=243MiB/s (254MB/s)(14.2GiB/60003msec); 0 zone resets slat (nsec): min=1452, max=134384, avg=2108.72, stdev=932.43 clat (usec): min=407, max=22944, avg=2058.34, stdev=1085.03 lat (usec): min=415, max=22946, avg=2060.48, stdev=1085.03 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 685], 10.00th=[ 832], 20.00th=[ 1123], | 30.00th=[ 1418], 40.00th=[ 1696], 50.00th=[ 1991], 60.00th=[ 2278], | 70.00th=[ 2573], 80.00th=[ 2868], 90.00th=[ 3195], 95.00th=[ 3392], | 99.00th=[ 5407], 99.50th=[ 7504], 99.90th=[11600], 99.95th=[13173], | 99.99th=[16188] bw ( KiB/s): min=244632, max=249912, per=100.00%, avg=248403.63, stdev=928.31, samples=120 iops : min=61158, max=62478, avg=62100.92, stdev=232.11, samples=120 lat (usec) : 500=0.01%, 750=7.35%, 1000=8.55% lat (msec) : 2=34.29%, 4=48.28%, 10=1.32%, 20=0.20%, 50=0.01% cpu : usr=8.89%, sys=17.20%, ctx=3708327, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,3726370,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): WRITE: bw=243MiB/s (254MB/s), 243MiB/s-243MiB/s (254MB/s-254MB/s), io=14.2GiB (15.3GB), run=60003-60003msec Disk stats (read/write): sdc: ios=128/3718437, merge=0/1, ticks=50/7646905, in_queue=1793160, util=97.74% #### Journal write latency (with sync): test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=5715: Fri Jan 8 10:20:04 2021 write: IOPS=26.0k, BW=105MiB/s (111MB/s)(6328MiB/60001msec); 0 zone resets slat (nsec): min=2095, max=49788, avg=2761.91, stdev=1707.49 clat (nsec): min=552, max=222865k, avg=33840.52, stdev=175371.71 lat (usec): min=31, max=222867, avg=36.65, stdev=175.38 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 31], | 70.00th=[ 32], 80.00th=[ 33], 90.00th=[ 39], 95.00th=[ 49], | 99.00th=[ 80], 99.50th=[ 99], 99.90th=[ 127], 99.95th=[ 137], | 99.99th=[ 178] bw ( KiB/s): min=66904, max=110680, per=100.00%, avg=108037.59, stdev=4074.32, samples=119 iops : min=16726, max=27670, avg=27009.39, stdev=1018.58, samples=119 lat (nsec) : 750=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.11%, 50=95.12%, 100=4.30% lat (usec) : 250=0.47%, 500=0.01%, 750=0.01% lat (msec) : 250=0.01% cpu : usr=4.44%, sys=10.73%, ctx=1619991, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1619990,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=105MiB/s (111MB/s), 105MiB/s-105MiB/s (111MB/s-111MB/s), io=6328MiB (6635MB), run=60001-60001msec Disk stats (read/write): sdc: ios=128/1616904, merge=0/0, ticks=56/54592, in_queue=53120, util=88.53% #### Journal write latency (with fsync): test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=5840: Fri Jan 8 10:21:05 2021 write: IOPS=26.1k, BW=102MiB/s (107MB/s)(6129MiB/60001msec); 0 zone resets slat (nsec): min=2029, max=61928, avg=2880.57, stdev=1738.58 clat (nsec): min=538, max=223671k, avg=33914.70, stdev=178905.16 lat (usec): min=31, max=223674, avg=36.84, stdev=178.92 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 31], | 70.00th=[ 32], 80.00th=[ 33], 90.00th=[ 39], 95.00th=[ 50], | 99.00th=[ 83], 99.50th=[ 109], 99.90th=[ 133], 99.95th=[ 141], | 99.99th=[ 239] bw ( KiB/s): min=54128, max=108376, per=99.99%, avg=104585.50, stdev=5328.88, samples=119 iops : min=13532, max=27094, avg=26146.38, stdev=1332.20, samples=119 lat (nsec) : 750=0.01% lat (usec) : 10=0.01%, 20=0.12%, 50=94.90%, 100=4.40%, 250=0.58% lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 250=0.01% fsync/fdatasync/sync_file_range: sync (nsec): min=13, max=17569, avg=26.49, stdev=88.77 sync percentiles (nsec): | 1.00th=[ 15], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 21], 40.00th=[ 22], 50.00th=[ 23], 60.00th=[ 24], | 70.00th=[ 25], 80.00th=[ 27], 90.00th=[ 41], 95.00th=[ 47], | 99.00th=[ 81], 99.50th=[ 94], 99.90th=[ 157], 99.95th=[ 161], | 99.99th=[ 241] cpu : usr=5.87%, sys=11.62%, ctx=1568962, majf=0, minf=15 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1568953,0,1568952 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=102MiB/s (107MB/s), 102MiB/s-102MiB/s (107MB/s-107MB/s), io=6129MiB (6426MB), run=60001-60001msec Disk stats (read/write): sdc: ios=166/1565817, merge=0/0, ticks=84/53152, in_queue=51704, util=86.12% #### Single-threaded random write latency: test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=6835: Fri Jan 8 10:22:05 2021 write: IOPS=26.3k, BW=103MiB/s (108MB/s)(6158MiB/60001msec); 0 zone resets slat (nsec): min=2034, max=68615, avg=2821.14, stdev=1664.94 clat (nsec): min=550, max=942234, avg=34655.93, stdev=12022.53 lat (usec): min=32, max=944, avg=37.52, stdev=12.18 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 32], 60.00th=[ 32], | 70.00th=[ 32], 80.00th=[ 34], 90.00th=[ 41], 95.00th=[ 56], | 99.00th=[ 96], 99.50th=[ 114], 99.90th=[ 137], 99.95th=[ 147], | 99.99th=[ 241] bw ( KiB/s): min=100856, max=108592, per=100.00%, avg=105087.28, stdev=2208.93, samples=119 iops : min=25214, max=27148, avg=26271.78, stdev=552.24, samples=119 lat (nsec) : 750=0.01% lat (usec) : 20=0.08%, 50=94.23%, 100=4.87%, 250=0.81%, 500=0.01% lat (usec) : 750=0.01%, 1000=0.01% cpu : usr=5.06%, sys=10.33%, ctx=1576330, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1576327,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=103MiB/s (108MB/s), 103MiB/s-103MiB/s (108MB/s-108MB/s), io=6158MiB (6457MB), run=60001-60001msec Disk stats (read/write): sdc: ios=43/1573119, merge=0/0, ticks=9/54384, in_queue=52568, util=87.68%