##### Linear read: test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=3857: Fri Jan 8 10:05:10 2021 read: IOPS=125, BW=501MiB/s (525MB/s)(29.4GiB/60127msec) slat (usec): min=303, max=204274, avg=7958.31, stdev=5094.90 clat (msec): min=126, max=445, avg=247.30, stdev=28.61 lat (msec): min=128, max=453, avg=255.26, stdev=29.08 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 241], 10.00th=[ 243], 20.00th=[ 243], | 30.00th=[ 243], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 243], | 70.00th=[ 243], 80.00th=[ 245], 90.00th=[ 249], 95.00th=[ 253], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min=385024, max=532480, per=99.79%, avg=512066.88, stdev=32153.84, samples=120 iops : min= 94, max= 130, avg=124.97, stdev= 7.84, samples=120 lat (msec) : 250=91.60%, 500=8.40% cpu : usr=0.13%, sys=1.10%, ctx=15060, majf=0, minf=583 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=7533,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): READ: bw=501MiB/s (525MB/s), 501MiB/s-501MiB/s (525MB/s-525MB/s), io=29.4GiB (31.6GB), run=60127-60127msec Disk stats (read/write): sdc: ios=120038/0, merge=0/0, ticks=15094172/0, in_queue=1916476, util=99.90% ##### Linear write: test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=4053: Fri Jan 8 10:06:11 2021 write: IOPS=101, BW=406MiB/s (426MB/s)(23.8GiB/60156msec); 0 zone resets slat (usec): min=73, max=202302, avg=9823.69, stdev=6052.43 clat (msec): min=152, max=492, avg=305.31, stdev=33.95 lat (msec): min=157, max=502, avg=315.13, stdev=34.51 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min=262144, max=425984, per=99.73%, avg=414479.88, stdev=37182.78, samples=120 iops : min= 64, max= 104, avg=101.16, stdev= 9.07, samples=120 lat (msec) : 250=0.34%, 500=99.66% cpu : usr=1.66%, sys=0.90%, ctx=12203, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.3%, 32=99.5%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=0,6104,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): WRITE: bw=406MiB/s (426MB/s), 406MiB/s-406MiB/s (426MB/s-426MB/s), io=23.8GiB (25.6GB), run=60156-60156msec Disk stats (read/write): sdc: ios=43/97574, merge=0/0, ticks=11/15149663, in_queue=1923584, util=99.87% #### Peak parallel random read: test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=4206: Fri Jan 8 10:07:11 2021 read: IOPS=54.1k, BW=211MiB/s (221MB/s)(12.4GiB/60002msec) slat (nsec): min=1364, max=82392, avg=2168.27, stdev=1426.35 clat (usec): min=97, max=228303, avg=2364.89, stdev=2768.74 lat (usec): min=99, max=228309, avg=2367.10, stdev=2768.73 clat percentiles (usec): | 1.00th=[ 363], 5.00th=[ 578], 10.00th=[ 775], 20.00th=[ 1139], | 30.00th=[ 1467], 40.00th=[ 1795], 50.00th=[ 2114], 60.00th=[ 2409], | 70.00th=[ 2737], 80.00th=[ 3195], 90.00th=[ 4228], 95.00th=[ 5145], | 99.00th=[ 6587], 99.50th=[ 8455], 99.90th=[ 14615], 99.95th=[ 17433], | 99.99th=[212861] bw ( KiB/s): min=118528, max=292336, per=99.99%, avg=216230.15, stdev=48763.86, samples=120 iops : min=29632, max=73084, avg=54057.59, stdev=12191.03, samples=120 lat (usec) : 100=0.01%, 250=0.19%, 500=3.11%, 750=6.09%, 1000=6.77% lat (msec) : 2=30.46%, 4=41.48%, 10=11.57%, 20=0.30%, 50=0.02% lat (msec) : 250=0.01% cpu : usr=7.08%, sys=15.30%, ctx=3168002, majf=0, minf=138 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=3243746,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): READ: bw=211MiB/s (221MB/s), 211MiB/s-211MiB/s (221MB/s-221MB/s), io=12.4GiB (13.3GB), run=60002-60002msec Disk stats (read/write): sdc: ios=3236441/0, merge=1/0, ticks=7647585/0, in_queue=1672128, util=98.79% #### Single-threaded read latency: test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=4287: Fri Jan 8 10:08:12 2021 read: IOPS=9435, BW=36.9MiB/s (38.6MB/s)(2211MiB/60001msec) slat (usec): min=2, max=263, avg= 4.03, stdev= 3.10 clat (nsec): min=1060, max=212522k, avg=101135.78, stdev=283141.56 lat (usec): min=51, max=212525, avg=105.26, stdev=283.17 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 79], | 30.00th=[ 84], 40.00th=[ 94], 50.00th=[ 95], 60.00th=[ 100], | 70.00th=[ 126], 80.00th=[ 127], 90.00th=[ 131], 95.00th=[ 133], | 99.00th=[ 141], 99.50th=[ 145], 99.90th=[ 186], 99.95th=[ 200], | 99.99th=[ 318] bw ( KiB/s): min=21120, max=49536, per=100.00%, avg=37739.26, stdev=2690.94, samples=120 iops : min= 5280, max=12384, avg=9434.79, stdev=672.73, samples=120 lat (usec) : 2=0.01%, 50=0.01%, 100=60.83%, 250=39.14%, 500=0.02% lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 250=0.01% cpu : usr=2.62%, sys=5.34%, ctx=566146, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=566139,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=36.9MiB/s (38.6MB/s), 36.9MiB/s-36.9MiB/s (38.6MB/s-38.6MB/s), io=2211MiB (2319MB), run=60001-60001msec Disk stats (read/write): sdc: ios=564938/0, merge=0/0, ticks=57110/0, in_queue=56220, util=93.77% #### Peak parallel random write: test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=4411: Fri Jan 8 10:09:13 2021 write: IOPS=61.6k, BW=241MiB/s (252MB/s)(14.1GiB/60003msec); 0 zone resets slat (nsec): min=1472, max=70965, avg=2144.58, stdev=958.17 clat (usec): min=405, max=24440, avg=2073.72, stdev=1091.09 lat (usec): min=428, max=24441, avg=2075.90, stdev=1091.09 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 685], 10.00th=[ 832], 20.00th=[ 1123], | 30.00th=[ 1418], 40.00th=[ 1713], 50.00th=[ 2008], 60.00th=[ 2311], | 70.00th=[ 2606], 80.00th=[ 2900], 90.00th=[ 3228], 95.00th=[ 3425], | 99.00th=[ 5407], 99.50th=[ 7504], 99.90th=[11600], 99.95th=[13042], | 99.99th=[16188] bw ( KiB/s): min=240552, max=249576, per=99.99%, avg=246554.45, stdev=1674.86, samples=120 iops : min=60138, max=62394, avg=61638.62, stdev=418.72, samples=120 lat (usec) : 500=0.01%, 750=7.18%, 1000=8.48% lat (msec) : 2=34.04%, 4=48.73%, 10=1.37%, 20=0.20%, 50=0.01% cpu : usr=9.13%, sys=17.29%, ctx=3678872, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,3698694,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): WRITE: bw=241MiB/s (252MB/s), 241MiB/s-241MiB/s (252MB/s-252MB/s), io=14.1GiB (15.1GB), run=60003-60003msec Disk stats (read/write): sdc: ios=99/3690768, merge=0/1, ticks=34/7647046, in_queue=1653072, util=95.86% #### Journal write latency (with sync): test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=4511: Fri Jan 8 10:10:13 2021 write: IOPS=21.3k, BW=83.3MiB/s (87.3MB/s)(4996MiB/60001msec); 0 zone resets slat (nsec): min=3175, max=60319, avg=4751.80, stdev=3530.06 clat (nsec): min=560, max=213298k, avg=41597.24, stdev=188908.38 lat (usec): min=40, max=213302, avg=46.44, stdev=188.93 clat percentiles (usec): | 1.00th=[ 34], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 38], | 30.00th=[ 38], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 39], | 70.00th=[ 39], 80.00th=[ 43], 90.00th=[ 49], 95.00th=[ 58], | 99.00th=[ 91], 99.50th=[ 119], 99.90th=[ 145], 99.95th=[ 155], | 99.99th=[ 255] bw ( KiB/s): min=49768, max=87024, per=99.99%, avg=85250.71, stdev=3575.35, samples=119 iops : min=12442, max=21756, avg=21312.71, stdev=893.85, samples=119 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=91.25%, 100=7.94% lat (usec) : 250=0.80%, 500=0.01%, 750=0.01% lat (msec) : 250=0.01% cpu : usr=4.42%, sys=8.52%, ctx=2557800, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1278900,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=83.3MiB/s (87.3MB/s), 83.3MiB/s-83.3MiB/s (87.3MB/s-87.3MB/s), io=4996MiB (5238MB), run=60001-60001msec Disk stats (read/write): sdc: ios=99/1276329, merge=0/0, ticks=35/55090, in_queue=53184, util=88.70% #### Journal write latency (with fsync): test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=4633: Fri Jan 8 10:11:14 2021 write: IOPS=7983, BW=31.2MiB/s (32.7MB/s)(1871MiB/60001msec); 0 zone resets slat (nsec): min=2229, max=44534, avg=3142.52, stdev=2179.77 clat (usec): min=3, max=370, avg=32.41, stdev= 3.78 lat (usec): min=32, max=386, avg=35.60, stdev= 4.59 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 32], | 70.00th=[ 32], 80.00th=[ 33], 90.00th=[ 37], 95.00th=[ 39], | 99.00th=[ 48], 99.50th=[ 50], 99.90th=[ 61], 99.95th=[ 82], | 99.99th=[ 133] bw ( KiB/s): min=30496, max=32496, per=100.00%, avg=31933.42, stdev=354.20, samples=119 iops : min= 7624, max= 8124, avg=7983.34, stdev=88.54, samples=119 lat (usec) : 4=0.01%, 10=0.01%, 20=0.19%, 50=99.37%, 100=0.41% lat (usec) : 250=0.03%, 500=0.01% fsync/fdatasync/sync_file_range: sync (nsec): min=17, max=17430, avg=87.69, stdev=182.92 sync percentiles (nsec): | 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 44], 20.00th=[ 46], | 30.00th=[ 52], 40.00th=[ 83], 50.00th=[ 88], 60.00th=[ 89], | 70.00th=[ 91], 80.00th=[ 92], 90.00th=[ 100], 95.00th=[ 155], | 99.00th=[ 310], 99.50th=[ 310], 99.90th=[ 370], 99.95th=[ 426], | 99.99th=[12864] cpu : usr=2.54%, sys=5.63%, ctx=1437106, majf=0, minf=11 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,479033,0,479033 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=31.2MiB/s (32.7MB/s), 31.2MiB/s-31.2MiB/s (32.7MB/s-32.7MB/s), io=1871MiB (1962MB), run=60001-60001msec Disk stats (read/write): sdc: ios=43/956141, merge=0/0, ticks=14/56785, in_queue=15364, util=25.63% #### Single-threaded random write latency: test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.12 Starting 1 process test: (groupid=0, jobs=1): err= 0: pid=4783: Fri Jan 8 10:12:14 2021 write: IOPS=20.9k, BW=81.7MiB/s (85.6MB/s)(4901MiB/60001msec); 0 zone resets slat (nsec): min=3143, max=88319, avg=4847.39, stdev=3683.61 clat (nsec): min=612, max=218324k, avg=42221.09, stdev=337573.02 lat (usec): min=40, max=218327, avg=47.16, stdev=337.59 clat percentiles (usec): | 1.00th=[ 34], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 39], | 70.00th=[ 40], 80.00th=[ 43], 90.00th=[ 49], 95.00th=[ 58], | 99.00th=[ 89], 99.50th=[ 117], 99.90th=[ 143], 99.95th=[ 151], | 99.99th=[ 225] bw ( KiB/s): min=64544, max=86536, per=99.99%, avg=83623.85, stdev=4178.13, samples=119 iops : min=16136, max=21634, avg=20905.94, stdev=1044.53, samples=119 lat (nsec) : 750=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=90.94%, 100=8.30% lat (usec) : 250=0.75%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 250=0.01% cpu : usr=4.66%, sys=8.69%, ctx=2509075, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1254533,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=81.7MiB/s (85.6MB/s), 81.7MiB/s-81.7MiB/s (85.6MB/s-85.6MB/s), io=4901MiB (5139MB), run=60001-60001msec Disk stats (read/write): sdc: ios=43/1251877, merge=0/0, ticks=13/54787, in_queue=53408, util=89.08%