##### Linear read: test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=27953: Thu Jan 7 14:50:59 2021 read: IOPS=123, BW=492MiB/s (516MB/s)(28.9GiB/60113msec) slat (usec): min=23, max=21962, avg=8100.78, stdev=7983.79 clat (msec): min=106, max=333, avg=251.61, stdev=14.00 lat (msec): min=121, max=348, avg=259.71, stdev=13.86 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 228], 10.00th=[ 234], 20.00th=[ 241], | 30.00th=[ 245], 40.00th=[ 247], 50.00th=[ 251], 60.00th=[ 255], | 70.00th=[ 262], 80.00th=[ 266], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 292], 99.95th=[ 321], | 99.99th=[ 334] bw ( KiB/s): min=327680, max=548864, per=99.75%, avg=503023.80, stdev=26692.51, samples=120 iops : min= 80, max= 134, avg=122.78, stdev= 6.52, samples=120 lat (msec) : 250=46.53%, 500=53.47% cpu : usr=0.06%, sys=1.03%, ctx=3779, majf=0, minf=584 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=7401,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): READ: bw=492MiB/s (516MB/s), 492MiB/s-492MiB/s (516MB/s-516MB/s), io=28.9GiB (31.0GB), run=60113-60113msec Disk stats (read/write): sdc: ios=29949/0, merge=0/0, ticks=3619333/0, in_queue=1916624, util=99.90% ##### Linear write: test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=28028: Thu Jan 7 14:51:59 2021 write: IOPS=127, BW=511MiB/s (536MB/s)(30.0GiB/60118msec); 0 zone resets slat (usec): min=80, max=32732, avg=7805.60, stdev=7491.63 clat (msec): min=112, max=342, avg=242.49, stdev=10.61 lat (msec): min=127, max=358, avg=250.30, stdev= 9.38 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 232], 10.00th=[ 232], 20.00th=[ 232], | 30.00th=[ 234], 40.00th=[ 247], 50.00th=[ 247], 60.00th=[ 247], | 70.00th=[ 247], 80.00th=[ 249], 90.00th=[ 249], 95.00th=[ 253], | 99.00th=[ 275], 99.50th=[ 284], 99.90th=[ 296], 99.95th=[ 326], | 99.99th=[ 342] bw ( KiB/s): min=393216, max=540672, per=99.77%, avg=522145.56, stdev=16772.36, samples=120 iops : min= 96, max= 132, avg=127.46, stdev= 4.10, samples=120 lat (msec) : 250=93.52%, 500=6.48% cpu : usr=2.97%, sys=0.88%, ctx=3896, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=0,7681,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): WRITE: bw=511MiB/s (536MB/s), 511MiB/s-511MiB/s (536MB/s-536MB/s), io=30.0GiB (32.2GB), run=60118-60118msec Disk stats (read/write): sdc: ios=16/30960, merge=0/0, ticks=1/3622883, in_queue=1922928, util=99.85% #### Peak parallel random read: test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=28123: Thu Jan 7 14:53:00 2021 read: IOPS=96.5k, BW=377MiB/s (395MB/s)(22.1GiB/60001msec) slat (nsec): min=948, max=1910.6k, avg=9654.97, stdev=23371.44 clat (usec): min=613, max=4981, avg=1316.19, stdev=221.88 lat (usec): min=627, max=4982, avg=1325.92, stdev=223.17 clat percentiles (usec): | 1.00th=[ 1020], 5.00th=[ 1045], 10.00th=[ 1057], 20.00th=[ 1139], | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1287], 60.00th=[ 1352], | 70.00th=[ 1385], 80.00th=[ 1450], 90.00th=[ 1549], 95.00th=[ 1614], | 99.00th=[ 2180], 99.50th=[ 2442], 99.90th=[ 2933], 99.95th=[ 3097], | 99.99th=[ 3458] bw ( KiB/s): min=384608, max=387112, per=99.99%, avg=385987.22, stdev=507.54, samples=120 iops : min=96152, max=96778, avg=96496.79, stdev=126.88, samples=120 lat (usec) : 750=0.01%, 1000=0.65% lat (msec) : 2=97.83%, 4=1.52%, 10=0.01% cpu : usr=6.29%, sys=21.50%, ctx=626981, majf=0, minf=136 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=5790708,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): READ: bw=377MiB/s (395MB/s), 377MiB/s-377MiB/s (395MB/s-395MB/s), io=22.1GiB (23.7GB), run=60001-60001msec Disk stats (read/write): sdc: ios=5779926/0, merge=1/0, ticks=3599475/0, in_queue=1914072, util=99.88% #### Single-threaded read latency: test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=28211: Thu Jan 7 14:54:00 2021 read: IOPS=20.6k, BW=80.4MiB/s (84.3MB/s)(4824MiB/60001msec) slat (nsec): min=1665, max=40544, avg=4722.99, stdev=2451.93 clat (usec): min=12, max=1679, avg=43.03, stdev=28.34 lat (usec): min=25, max=1682, avg=47.92, stdev=28.65 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 29], 10.00th=[ 30], 20.00th=[ 30], | 30.00th=[ 31], 40.00th=[ 32], 50.00th=[ 35], 60.00th=[ 36], | 70.00th=[ 37], 80.00th=[ 37], 90.00th=[ 96], 95.00th=[ 114], | 99.00th=[ 145], 99.50th=[ 145], 99.90th=[ 147], 99.95th=[ 151], | 99.99th=[ 157] bw ( KiB/s): min=79352, max=91312, per=100.00%, avg=82339.18, stdev=2991.17, samples=119 iops : min=19838, max=22828, avg=20584.77, stdev=747.78, samples=119 lat (usec) : 20=0.04%, 50=87.50%, 100=4.21%, 250=8.25%, 500=0.01% lat (usec) : 750=0.01% lat (msec) : 2=0.01% cpu : usr=4.34%, sys=15.26%, ctx=1234843, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1234834,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=80.4MiB/s (84.3MB/s), 80.4MiB/s-80.4MiB/s (84.3MB/s-84.3MB/s), io=4824MiB (5058MB), run=60001-60001msec Disk stats (read/write): sdc: ios=1232638/0, merge=0/0, ticks=53341/0, in_queue=51100, util=85.22% #### Peak parallel random write: test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=28326: Thu Jan 7 14:55:01 2021 write: IOPS=84.2k, BW=329MiB/s (345MB/s)(19.3GiB/60001msec); 0 zone resets slat (nsec): min=1013, max=5687.1k, avg=11119.97, stdev=28311.50 clat (usec): min=670, max=11032, avg=1509.34, stdev=269.48 lat (usec): min=764, max=11036, avg=1520.54, stdev=271.16 clat percentiles (usec): | 1.00th=[ 1172], 5.00th=[ 1188], 10.00th=[ 1237], 20.00th=[ 1303], | 30.00th=[ 1385], 40.00th=[ 1418], 50.00th=[ 1500], 60.00th=[ 1549], | 70.00th=[ 1631], 80.00th=[ 1663], 90.00th=[ 1762], 95.00th=[ 1811], | 99.00th=[ 2474], 99.50th=[ 2802], 99.90th=[ 3523], 99.95th=[ 4015], | 99.99th=[ 7767] bw ( KiB/s): min=317032, max=338600, per=100.00%, avg=336619.59, stdev=2754.24, samples=120 iops : min=79258, max=84650, avg=84154.92, stdev=688.47, samples=120 lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=97.11%, 4=2.84%, 10=0.05%, 20=0.01% cpu : usr=5.59%, sys=19.23%, ctx=610846, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,5049595,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): WRITE: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=19.3GiB (20.7GB), run=60001-60001msec Disk stats (read/write): sdc: ios=43/5040033, merge=0/1, ticks=9/3609103, in_queue=1915432, util=99.89% #### Journal write latency (with sync): test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=28422: Thu Jan 7 14:56:01 2021 write: IOPS=37.8k, BW=148MiB/s (155MB/s)(8855MiB/60001msec); 0 zone resets slat (nsec): min=1729, max=63823, avg=2533.58, stdev=674.96 clat (usec): min=9, max=434, avg=23.50, stdev= 2.50 lat (usec): min=22, max=441, avg=26.15, stdev= 2.76 clat percentiles (nsec): | 1.00th=[21120], 5.00th=[21376], 10.00th=[21632], 20.00th=[22144], | 30.00th=[22656], 40.00th=[22912], 50.00th=[22912], 60.00th=[23424], | 70.00th=[23936], 80.00th=[24192], 90.00th=[25216], 95.00th=[27520], | 99.00th=[30848], 99.50th=[34560], 99.90th=[39168], 99.95th=[40704], | 99.99th=[47360] bw ( KiB/s): min=144872, max=160776, per=99.94%, avg=151038.59, stdev=5603.94, samples=119 iops : min=36218, max=40194, avg=37759.63, stdev=1400.99, samples=119 lat (usec) : 10=0.01%, 20=0.03%, 50=99.97%, 100=0.01%, 250=0.01% lat (usec) : 500=0.01% cpu : usr=4.07%, sys=15.51%, ctx=2266932, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2266910,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=148MiB/s (155MB/s), 148MiB/s-148MiB/s (155MB/s-155MB/s), io=8855MiB (9285MB), run=60001-60001msec Disk stats (read/write): sdc: ios=43/2262624, merge=0/0, ticks=9/53184, in_queue=43648, util=72.78% #### Journal write latency (with fsync): test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=28524: Thu Jan 7 14:57:02 2021 write: IOPS=36.0k, BW=141MiB/s (148MB/s)(8445MiB/60001msec); 0 zone resets slat (nsec): min=1833, max=28038, avg=2601.19, stdev=632.90 clat (usec): min=2, max=1374, avg=24.03, stdev= 4.92 lat (usec): min=22, max=1378, avg=26.75, stdev= 5.04 clat percentiles (usec): | 1.00th=[ 22], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 24], 60.00th=[ 24], | 70.00th=[ 24], 80.00th=[ 25], 90.00th=[ 26], 95.00th=[ 28], | 99.00th=[ 32], 99.50th=[ 36], 99.90th=[ 41], 99.95th=[ 42], | 99.99th=[ 192] bw ( KiB/s): min=130392, max=153560, per=99.94%, avg=144037.34, stdev=3783.04, samples=119 iops : min=32598, max=38390, avg=36009.31, stdev=945.75, samples=119 lat (usec) : 4=0.01%, 10=0.01%, 20=0.02%, 50=99.96%, 100=0.01% lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01% fsync/fdatasync/sync_file_range: sync (nsec): min=16, max=16312, avg=28.00, stdev=41.17 sync percentiles (nsec): | 1.00th=[ 21], 5.00th=[ 23], 10.00th=[ 25], 20.00th=[ 26], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 28], 60.00th=[ 28], | 70.00th=[ 29], 80.00th=[ 29], 90.00th=[ 30], 95.00th=[ 32], | 99.00th=[ 34], 99.50th=[ 38], 99.90th=[ 88], 99.95th=[ 91], | 99.99th=[ 100] cpu : usr=4.96%, sys=16.88%, ctx=2161813, majf=0, minf=14 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2161803,0,2161802 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=141MiB/s (148MB/s), 141MiB/s-141MiB/s (148MB/s-148MB/s), io=8445MiB (8855MB), run=60001-60001msec Disk stats (read/write): sdc: ios=43/2157667, merge=0/0, ticks=10/51899, in_queue=43456, util=72.47% #### Single-threaded random write latency: test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=28635: Thu Jan 7 14:58:03 2021 write: IOPS=37.9k, BW=148MiB/s (155MB/s)(8882MiB/60001msec); 0 zone resets slat (nsec): min=1656, max=43941, avg=2445.47, stdev=628.45 clat (nsec): min=391, max=1886.6k, avg=23405.39, stdev=6353.82 lat (usec): min=22, max=1911, avg=25.96, stdev= 6.51 clat percentiles (usec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 23], 60.00th=[ 24], | 70.00th=[ 24], 80.00th=[ 25], 90.00th=[ 25], 95.00th=[ 28], | 99.00th=[ 31], 99.50th=[ 34], 99.90th=[ 40], 99.95th=[ 41], | 99.99th=[ 151] bw ( KiB/s): min=142552, max=160240, per=100.00%, avg=151598.35, stdev=5952.37, samples=119 iops : min=35638, max=40060, avg=37899.52, stdev=1488.08, samples=119 lat (nsec) : 500=0.01% lat (usec) : 10=0.01%, 20=0.02%, 50=99.97%, 100=0.01%, 250=0.01% lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01% cpu : usr=4.52%, sys=14.90%, ctx=2273682, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2273666,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=148MiB/s (155MB/s), 148MiB/s-148MiB/s (155MB/s-155MB/s), io=8882MiB (9313MB), run=60001-60001msec Disk stats (read/write): sdc: ios=43/2269595, merge=0/0, ticks=10/53114, in_queue=45684, util=76.18%