random-write: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=posixaio, iodepth=16
...
fio-3.28
Starting 16 processes

random-write: (groupid=0, jobs=1): err= 0: pid=732: Tue Dec 19 07:28:14 2023
  write: IOPS=592, BW=37.0MiB/s (38.8MB/s)(2234MiB/60334msec); 0 zone resets
    slat (nsec): min=691, max=3640.3k, avg=8904.35, stdev=35595.01
    clat (usec): min=504, max=408881, avg=26652.32, stdev=23763.99
     lat (usec): min=513, max=408888, avg=26661.23, stdev=23764.01
    clat percentiles (usec):
     |  1.00th=[  1205],  5.00th=[  2507], 10.00th=[ 14484], 20.00th=[ 17695],
     | 30.00th=[ 19530], 40.00th=[ 21103], 50.00th=[ 23200], 60.00th=[ 25560],
     | 70.00th=[ 27657], 80.00th=[ 30802], 90.00th=[ 36963], 95.00th=[ 48497],
     | 99.00th=[137364], 99.50th=[181404], 99.90th=[291505], 99.95th=[291505],
     | 99.99th=[408945]
   bw (  KiB/s): min=13668, max=57984, per=6.24%, avg=38160.66, stdev=8436.85, samples=119
   iops        : min=  213, max=  906, avg=596.14, stdev=131.84, samples=119
  lat (usec)   : 750=0.01%, 1000=0.20%
  lat (msec)   : 2=3.70%, 4=2.17%, 10=2.06%, 20=25.27%, 50=61.91%
  lat (msec)   : 100=3.21%, 250=1.16%, 500=0.31%
  cpu          : usr=1.14%, sys=1.80%, ctx=27681, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=50.7%, 16=48.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.6%, 8=3.6%, 16=0.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,35747,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=733: Tue Dec 19 07:28:14 2023
  write: IOPS=606, BW=37.9MiB/s (39.7MB/s)(2285MiB/60332msec); 0 zone resets
    slat (nsec): min=827, max=2286.9k, avg=9117.18, stdev=33181.47
    clat (usec): min=108, max=282176, avg=26118.03, stdev=20289.74
     lat (usec): min=126, max=282184, avg=26127.15, stdev=20289.53
    clat percentiles (usec):
     |  1.00th=[  1205],  5.00th=[  2147], 10.00th=[ 13435], 20.00th=[ 17695],
     | 30.00th=[ 19530], 40.00th=[ 21365], 50.00th=[ 23462], 60.00th=[ 25297],
     | 70.00th=[ 27657], 80.00th=[ 30802], 90.00th=[ 37487], 95.00th=[ 47973],
     | 99.00th=[127402], 99.50th=[160433], 99.90th=[225444], 99.95th=[261096],
     | 99.99th=[283116]
   bw (  KiB/s): min=12544, max=57728, per=6.38%, avg=39036.28, stdev=8348.92, samples=119
   iops        : min=  196, max=  902, avg=609.82, stdev=130.48, samples=119
  lat (usec)   : 250=0.01%, 750=0.01%, 1000=0.25%
  lat (msec)   : 2=4.42%, 4=2.30%, 10=1.77%, 20=23.74%, 50=62.94%
  lat (msec)   : 100=3.07%, 250=1.41%, 500=0.09%
  cpu          : usr=1.11%, sys=1.70%, ctx=27485, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=50.7%, 16=48.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.0%, 8=2.5%, 16=1.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,36562,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=734: Tue Dec 19 07:28:14 2023
  write: IOPS=597, BW=37.3MiB/s (39.1MB/s)(2248MiB/60213msec); 0 zone resets
    slat (nsec): min=585, max=6849.7k, avg=9863.65, stdev=62985.37
    clat (usec): min=5, max=334963, avg=26615.93, stdev=21890.47
     lat (usec): min=102, max=334969, avg=26625.80, stdev=21890.03
    clat percentiles (usec):
     |  1.00th=[  1254],  5.00th=[  2638], 10.00th=[ 14615], 20.00th=[ 17695],
     | 30.00th=[ 19530], 40.00th=[ 21365], 50.00th=[ 23200], 60.00th=[ 25560],
     | 70.00th=[ 27919], 80.00th=[ 31065], 90.00th=[ 38011], 95.00th=[ 49021],
     | 99.00th=[137364], 99.50th=[162530], 99.90th=[263193], 99.95th=[278922],
     | 99.99th=[333448]
   bw (  KiB/s): min=10219, max=65667, per=6.27%, avg=38387.64, stdev=9340.54, samples=119
   iops        : min=  159, max= 1026, avg=599.70, stdev=145.97, samples=119
  lat (usec)   : 10=0.01%, 250=0.01%, 500=0.01%, 750=0.05%, 1000=0.15%
  lat (msec)   : 2=3.62%, 4=2.19%, 10=2.06%, 20=24.25%, 50=62.80%
  lat (msec)   : 100=3.19%, 250=1.46%, 500=0.19%
  cpu          : usr=1.07%, sys=1.72%, ctx=27716, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.4%, 8=50.6%, 16=48.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=3.1%, 16=1.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,35961,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=735: Tue Dec 19 07:28:14 2023
  write: IOPS=608, BW=38.0MiB/s (39.9MB/s)(2291MiB/60224msec); 0 zone resets
    slat (nsec): min=636, max=12918k, avg=9827.93, stdev=92909.15
    clat (usec): min=205, max=284205, avg=26132.87, stdev=20835.25
     lat (usec): min=224, max=284212, avg=26142.70, stdev=20835.06
    clat percentiles (usec):
     |  1.00th=[  1123],  5.00th=[  1778], 10.00th=[ 11207], 20.00th=[ 17695],
     | 30.00th=[ 19530], 40.00th=[ 21103], 50.00th=[ 22938], 60.00th=[ 25297],
     | 70.00th=[ 27657], 80.00th=[ 30802], 90.00th=[ 38011], 95.00th=[ 50594],
     | 99.00th=[135267], 99.50th=[158335], 99.90th=[258999], 99.95th=[267387],
     | 99.99th=[283116]
   bw (  KiB/s): min=17280, max=73875, per=6.40%, avg=39132.22, stdev=9352.70, samples=119
   iops        : min=  270, max= 1154, avg=611.33, stdev=146.13, samples=119
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.20%
  lat (msec)   : 2=5.58%, 4=1.98%, 10=1.88%, 20=23.65%, 50=61.62%
  lat (msec)   : 100=3.44%, 250=1.54%, 500=0.10%
  cpu          : usr=1.08%, sys=1.72%, ctx=28178, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.7%, 16=49.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.9%, 8=2.9%, 16=1.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,36650,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=736: Tue Dec 19 07:28:14 2023
  write: IOPS=587, BW=36.7MiB/s (38.5MB/s)(2212MiB/60243msec); 0 zone resets
    slat (nsec): min=752, max=6365.5k, avg=9100.03, stdev=40451.54
    clat (usec): min=695, max=383218, avg=26837.68, stdev=22023.42
     lat (usec): min=851, max=383227, avg=26846.78, stdev=22023.57
    clat percentiles (usec):
     |  1.00th=[  1369],  5.00th=[  2409], 10.00th=[ 15139], 20.00th=[ 17957],
     | 30.00th=[ 19792], 40.00th=[ 21627], 50.00th=[ 23462], 60.00th=[ 25560],
     | 70.00th=[ 27919], 80.00th=[ 31065], 90.00th=[ 38011], 95.00th=[ 49546],
     | 99.00th=[135267], 99.50th=[156238], 99.90th=[274727], 99.95th=[325059],
     | 99.99th=[354419]
   bw (  KiB/s): min=15584, max=58550, per=6.17%, avg=37779.19, stdev=8553.77, samples=119
   iops        : min=  243, max=  914, avg=590.16, stdev=133.66, samples=119
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=4.16%, 4=1.84%, 10=1.59%, 20=23.82%, 50=63.62%
  lat (msec)   : 100=3.30%, 250=1.46%, 500=0.19%
  cpu          : usr=1.06%, sys=1.83%, ctx=26747, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.7%, 16=49.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.7%, 8=3.1%, 16=1.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,35394,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=737: Tue Dec 19 07:28:14 2023
  write: IOPS=605, BW=37.8MiB/s (39.7MB/s)(2277MiB/60170msec); 0 zone resets
    slat (nsec): min=626, max=1941.6k, avg=9137.38, stdev=30107.76
    clat (usec): min=6, max=366833, avg=26218.21, stdev=21443.82
     lat (usec): min=107, max=366842, avg=26227.35, stdev=21443.79
    clat percentiles (usec):
     |  1.00th=[  1287],  5.00th=[  2212], 10.00th=[ 12125], 20.00th=[ 17433],
     | 30.00th=[ 19530], 40.00th=[ 21365], 50.00th=[ 23200], 60.00th=[ 25297],
     | 70.00th=[ 27657], 80.00th=[ 31065], 90.00th=[ 38011], 95.00th=[ 49021],
     | 99.00th=[128451], 99.50th=[152044], 99.90th=[278922], 99.95th=[291505],
     | 99.99th=[354419]
   bw (  KiB/s): min=15104, max=85376, per=6.36%, avg=38898.55, stdev=9235.21, samples=119
   iops        : min=  236, max= 1334, avg=607.67, stdev=144.32, samples=119
  lat (usec)   : 10=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.10%
  lat (msec)   : 2=4.32%, 4=2.75%, 10=2.15%, 20=23.78%, 50=62.06%
  lat (msec)   : 100=3.28%, 250=1.34%, 500=0.18%
  cpu          : usr=1.11%, sys=1.71%, ctx=27310, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=50.8%, 16=48.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.7%, 8=3.7%, 16=0.7%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,36435,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=738: Tue Dec 19 07:28:14 2023
  write: IOPS=599, BW=37.5MiB/s (39.3MB/s)(2260MiB/60271msec); 0 zone resets
    slat (nsec): min=753, max=3430.6k, avg=9526.61, stdev=42244.42
    clat (usec): min=286, max=350771, avg=26321.31, stdev=21797.68
     lat (usec): min=295, max=350777, avg=26330.84, stdev=21797.22
    clat percentiles (usec):
     |  1.00th=[  1352],  5.00th=[  2147], 10.00th=[ 13698], 20.00th=[ 17695],
     | 30.00th=[ 19530], 40.00th=[ 21365], 50.00th=[ 23200], 60.00th=[ 25297],
     | 70.00th=[ 27657], 80.00th=[ 30802], 90.00th=[ 37487], 95.00th=[ 49546],
     | 99.00th=[133694], 99.50th=[160433], 99.90th=[254804], 99.95th=[261096],
     | 99.99th=[350225]
   bw (  KiB/s): min=13184, max=76544, per=6.31%, avg=38606.36, stdev=9087.91, samples=119
   iops        : min=  206, max= 1196, avg=603.10, stdev=142.02, samples=119
  lat (usec)   : 500=0.03%, 1000=0.08%
  lat (msec)   : 2=4.39%, 4=2.57%, 10=1.75%, 20=23.62%, 50=62.67%
  lat (msec)   : 100=3.30%, 250=1.43%, 500=0.17%
  cpu          : usr=1.13%, sys=1.71%, ctx=26699, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.4%, 8=50.8%, 16=48.7%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.7%, 8=2.8%, 16=1.4%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,36162,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=739: Tue Dec 19 07:28:14 2023
  write: IOPS=597, BW=37.4MiB/s (39.2MB/s)(2252MiB/60276msec); 0 zone resets
    slat (nsec): min=650, max=4892.7k, avg=9055.54, stdev=41369.21
    clat (usec): min=218, max=315989, avg=26524.74, stdev=21127.27
     lat (usec): min=239, max=315995, avg=26533.79, stdev=21127.01
    clat percentiles (usec):
     |  1.00th=[  1254],  5.00th=[  2343], 10.00th=[ 14615], 20.00th=[ 17957],
     | 30.00th=[ 19792], 40.00th=[ 21365], 50.00th=[ 23200], 60.00th=[ 25560],
     | 70.00th=[ 27919], 80.00th=[ 31065], 90.00th=[ 38011], 95.00th=[ 48497],
     | 99.00th=[135267], 99.50th=[160433], 99.90th=[263193], 99.95th=[270533],
     | 99.99th=[316670]
   bw (  KiB/s): min=11113, max=59264, per=6.29%, avg=38475.47, stdev=8680.87, samples=119
   iops        : min=  173, max=  926, avg=601.08, stdev=135.67, samples=119
  lat (usec)   : 250=0.01%, 750=0.01%, 1000=0.13%
  lat (msec)   : 2=4.23%, 4=2.06%, 10=1.84%, 20=23.25%, 50=63.91%
  lat (msec)   : 100=2.88%, 250=1.54%, 500=0.13%
  cpu          : usr=1.13%, sys=1.70%, ctx=27067, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.8%, 16=48.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=3.7%, 16=0.4%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,36037,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=740: Tue Dec 19 07:28:14 2023
  write: IOPS=603, BW=37.7MiB/s (39.6MB/s)(2274MiB/60293msec); 0 zone resets
    slat (nsec): min=714, max=5091.2k, avg=9105.25, stdev=42879.38
    clat (usec): min=289, max=380994, avg=26191.07, stdev=20524.33
     lat (usec): min=295, max=381005, avg=26200.17, stdev=20524.12
    clat percentiles (usec):
     |  1.00th=[  1221],  5.00th=[  1991], 10.00th=[ 14484], 20.00th=[ 17957],
     | 30.00th=[ 19792], 40.00th=[ 21365], 50.00th=[ 23462], 60.00th=[ 25297],
     | 70.00th=[ 27657], 80.00th=[ 30802], 90.00th=[ 37487], 95.00th=[ 49021],
     | 99.00th=[132645], 99.50th=[156238], 99.90th=[229639], 99.95th=[270533],
     | 99.99th=[379585]
   bw (  KiB/s): min= 8175, max=58752, per=6.35%, avg=38853.77, stdev=8491.11, samples=119
   iops        : min=  127, max=  918, avg=606.99, stdev=132.69, samples=119
  lat (usec)   : 500=0.01%, 750=0.02%, 1000=0.19%
  lat (msec)   : 2=4.83%, 4=2.02%, 10=1.53%, 20=23.21%, 50=63.47%
  lat (msec)   : 100=3.33%, 250=1.32%, 500=0.08%
  cpu          : usr=1.03%, sys=1.81%, ctx=26970, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=50.8%, 16=48.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=3.1%, 16=1.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,36390,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=741: Tue Dec 19 07:28:14 2023
  write: IOPS=603, BW=37.7MiB/s (39.5MB/s)(2273MiB/60302msec); 0 zone resets
    slat (nsec): min=784, max=5662.0k, avg=9857.64, stdev=46722.21
    clat (usec): min=6, max=270132, avg=26205.53, stdev=20788.90
     lat (usec): min=93, max=270136, avg=26215.39, stdev=20788.06
    clat percentiles (usec):
     |  1.00th=[  1205],  5.00th=[  2245], 10.00th=[ 13960], 20.00th=[ 17695],
     | 30.00th=[ 19530], 40.00th=[ 21365], 50.00th=[ 23200], 60.00th=[ 25297],
     | 70.00th=[ 27657], 80.00th=[ 30540], 90.00th=[ 37487], 95.00th=[ 48497],
     | 99.00th=[133694], 99.50th=[164627], 99.90th=[250610], 99.95th=[263193],
     | 99.99th=[270533]
   bw (  KiB/s): min=20096, max=57229, per=6.35%, avg=38830.86, stdev=8097.81, samples=119
   iops        : min=  314, max=  894, avg=606.64, stdev=126.53, samples=119
  lat (usec)   : 10=0.01%, 20=0.01%, 100=0.01%, 250=0.02%, 500=0.05%
  lat (usec)   : 750=0.03%, 1000=0.15%
  lat (msec)   : 2=3.80%, 4=3.02%, 10=1.53%, 20=23.60%, 50=63.29%
  lat (msec)   : 100=2.83%, 250=1.55%, 500=0.11%
  cpu          : usr=1.09%, sys=1.77%, ctx=27654, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.4%, 8=50.7%, 16=48.7%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.9%, 8=2.8%, 16=1.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,36368,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=742: Tue Dec 19 07:28:14 2023
  write: IOPS=602, BW=37.6MiB/s (39.5MB/s)(2265MiB/60193msec); 0 zone resets
    slat (nsec): min=914, max=7603.3k, avg=9315.01, stdev=54831.18
    clat (usec): min=452, max=366626, avg=26320.66, stdev=20500.64
     lat (usec): min=465, max=366637, avg=26329.97, stdev=20500.53
    clat percentiles (usec):
     |  1.00th=[  1188],  5.00th=[  2147], 10.00th=[ 14615], 20.00th=[ 17695],
     | 30.00th=[ 19530], 40.00th=[ 21365], 50.00th=[ 23462], 60.00th=[ 25560],
     | 70.00th=[ 27919], 80.00th=[ 31065], 90.00th=[ 38011], 95.00th=[ 49021],
     | 99.00th=[124257], 99.50th=[158335], 99.90th=[187696], 99.95th=[278922],
     | 99.99th=[367002]
   bw (  KiB/s): min=18944, max=57485, per=6.32%, avg=38689.18, stdev=8240.49, samples=119
   iops        : min=  296, max=  898, avg=604.39, stdev=128.78, samples=119
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.08%
  lat (msec)   : 2=4.48%, 4=2.35%, 10=1.38%, 20=23.97%, 50=62.89%
  lat (msec)   : 100=3.35%, 250=1.38%, 500=0.10%
  cpu          : usr=1.08%, sys=1.72%, ctx=26593, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=50.8%, 16=48.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.0%, 8=3.5%, 16=0.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,36239,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=743: Tue Dec 19 07:28:14 2023
  write: IOPS=584, BW=36.5MiB/s (38.3MB/s)(2204MiB/60321msec); 0 zone resets
    slat (nsec): min=771, max=6812.1k, avg=9417.07, stdev=49961.61
    clat (usec): min=715, max=503179, avg=27022.53, stdev=23421.13
     lat (usec): min=724, max=503186, avg=27031.95, stdev=23421.10
    clat percentiles (usec):
     |  1.00th=[  1369],  5.00th=[  2835], 10.00th=[ 14877], 20.00th=[ 17957],
     | 30.00th=[ 19792], 40.00th=[ 21365], 50.00th=[ 23462], 60.00th=[ 25560],
     | 70.00th=[ 28181], 80.00th=[ 31327], 90.00th=[ 38536], 95.00th=[ 49021],
     | 99.00th=[137364], 99.50th=[160433], 99.90th=[278922], 99.95th=[295699],
     | 99.99th=[501220]
   bw (  KiB/s): min=14690, max=58890, per=6.15%, avg=37643.09, stdev=8477.95, samples=119
   iops        : min=  229, max=  920, avg=588.06, stdev=132.48, samples=119
  lat (usec)   : 750=0.01%, 1000=0.03%
  lat (msec)   : 2=3.38%, 4=2.39%, 10=1.76%, 20=24.19%, 50=63.42%
  lat (msec)   : 100=3.02%, 250=1.61%, 500=0.17%, 750=0.03%
  cpu          : usr=1.06%, sys=1.85%, ctx=27810, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.7%, 16=49.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=3.2%, 16=1.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,35268,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=744: Tue Dec 19 07:28:14 2023
  write: IOPS=593, BW=37.1MiB/s (38.9MB/s)(2236MiB/60301msec); 0 zone resets
    slat (nsec): min=740, max=4226.6k, avg=9182.41, stdev=40401.38
    clat (usec): min=960, max=432668, avg=26653.64, stdev=22136.55
     lat (usec): min=965, max=432670, avg=26662.83, stdev=22136.61
    clat percentiles (usec):
     |  1.00th=[  1254],  5.00th=[  2212], 10.00th=[ 14746], 20.00th=[ 17957],
     | 30.00th=[ 19530], 40.00th=[ 21365], 50.00th=[ 23462], 60.00th=[ 25560],
     | 70.00th=[ 27919], 80.00th=[ 31065], 90.00th=[ 38011], 95.00th=[ 49021],
     | 99.00th=[137364], 99.50th=[158335], 99.90th=[267387], 99.95th=[278922],
     | 99.99th=[429917]
   bw (  KiB/s): min=19840, max=58368, per=6.24%, avg=38192.21, stdev=7997.22, samples=119
   iops        : min=  310, max=  912, avg=596.65, stdev=124.95, samples=119
  lat (usec)   : 1000=0.05%
  lat (msec)   : 2=4.32%, 4=2.03%, 10=1.53%, 20=24.08%, 50=63.37%
  lat (msec)   : 100=3.06%, 250=1.39%, 500=0.18%
  cpu          : usr=1.08%, sys=1.81%, ctx=27019, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=50.7%, 16=48.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.1%, 8=3.2%, 16=0.7%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,35776,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=745: Tue Dec 19 07:28:14 2023
  write: IOPS=596, BW=37.3MiB/s (39.1MB/s)(2245MiB/60191msec); 0 zone resets
    slat (nsec): min=764, max=9648.2k, avg=9026.96, stdev=56527.64
    clat (usec): min=203, max=277325, avg=26567.66, stdev=21269.60
     lat (usec): min=211, max=277333, avg=26576.69, stdev=21269.66
    clat percentiles (usec):
     |  1.00th=[  1237],  5.00th=[  2114], 10.00th=[ 14877], 20.00th=[ 17957],
     | 30.00th=[ 19530], 40.00th=[ 21365], 50.00th=[ 23462], 60.00th=[ 25560],
     | 70.00th=[ 27657], 80.00th=[ 30802], 90.00th=[ 37487], 95.00th=[ 48497],
     | 99.00th=[137364], 99.50th=[158335], 99.90th=[250610], 99.95th=[263193],
     | 99.99th=[278922]
   bw (  KiB/s): min=17920, max=58496, per=6.27%, avg=38338.92, stdev=8286.35, samples=119
   iops        : min=  280, max=  914, avg=598.85, stdev=129.51, samples=119
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.17%
  lat (msec)   : 2=4.44%, 4=1.99%, 10=1.05%, 20=24.49%, 50=63.11%
  lat (msec)   : 100=2.97%, 250=1.66%, 500=0.11%
  cpu          : usr=1.03%, sys=1.77%, ctx=26986, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=50.8%, 16=48.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.7%, 8=2.8%, 16=1.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,35917,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=746: Tue Dec 19 07:28:14 2023
  write: IOPS=597, BW=37.3MiB/s (39.2MB/s)(2250MiB/60249msec); 0 zone resets
    slat (nsec): min=657, max=3055.4k, avg=8853.99, stdev=27455.57
    clat (usec): min=395, max=380903, avg=26492.77, stdev=21952.53
     lat (usec): min=403, max=380910, avg=26501.63, stdev=21952.25
    clat percentiles (usec):
     |  1.00th=[  1336],  5.00th=[  2638], 10.00th=[ 14615], 20.00th=[ 17695],
     | 30.00th=[ 19530], 40.00th=[ 21365], 50.00th=[ 23200], 60.00th=[ 25297],
     | 70.00th=[ 27657], 80.00th=[ 30802], 90.00th=[ 36963], 95.00th=[ 48497],
     | 99.00th=[133694], 99.50th=[164627], 99.90th=[254804], 99.95th=[270533],
     | 99.99th=[379585]
   bw (  KiB/s): min=19800, max=58496, per=6.28%, avg=38425.00, stdev=8149.79, samples=119
   iops        : min=  309, max=  914, avg=600.27, stdev=127.37, samples=119
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.10%
  lat (msec)   : 2=3.73%, 4=2.04%, 10=2.05%, 20=24.83%, 50=62.50%
  lat (msec)   : 100=3.03%, 250=1.59%, 500=0.12%
  cpu          : usr=1.05%, sys=1.72%, ctx=26983, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.7%, 8=3.4%, 16=0.9%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,36004,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=747: Tue Dec 19 07:28:14 2023
  write: IOPS=596, BW=37.3MiB/s (39.1MB/s)(2246MiB/60289msec); 0 zone resets
    slat (nsec): min=858, max=4786.4k, avg=9352.77, stdev=45147.86
    clat (usec): min=8, max=302576, avg=26483.88, stdev=21305.47
     lat (usec): min=94, max=302581, avg=26493.23, stdev=21304.98
    clat percentiles (usec):
     |  1.00th=[  1221],  5.00th=[  2409], 10.00th=[ 14746], 20.00th=[ 17695],
     | 30.00th=[ 19530], 40.00th=[ 21365], 50.00th=[ 23200], 60.00th=[ 25297],
     | 70.00th=[ 27657], 80.00th=[ 31065], 90.00th=[ 38011], 95.00th=[ 50070],
     | 99.00th=[137364], 99.50th=[162530], 99.90th=[270533], 99.95th=[278922],
     | 99.99th=[304088]
   bw (  KiB/s): min=13668, max=59392, per=6.27%, avg=38368.90, stdev=8477.63, samples=119
   iops        : min=  213, max=  928, avg=599.43, stdev=132.46, samples=119
  lat (usec)   : 10=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.16%
  lat (msec)   : 2=3.95%, 4=2.09%, 10=1.79%, 20=24.39%, 50=62.53%
  lat (msec)   : 100=3.44%, 250=1.45%, 500=0.17%
  cpu          : usr=1.05%, sys=1.82%, ctx=27499, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=50.8%, 16=48.7%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.4%, 8=3.3%, 16=1.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,35934,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=598MiB/s (627MB/s), 36.5MiB/s-38.0MiB/s (38.3MB/s-39.9MB/s), io=35.2GiB (37.8GB), run=60170-60334msec

Disk stats (read/write):
    dm-7: ios=0/474604, merge=0/0, ticks=0/140985172, in_queue=140985172, util=98.00%, aggrios=0/487480, aggrmerge=0/0, aggrticks=0/145531753, aggrin_queue=145531753, aggrutil=97.98%
    dm-6: ios=0/487480, merge=0/0, ticks=0/145531753, in_queue=145531753, util=97.98%, aggrios=0/487480, aggrmerge=0/0, aggrticks=0/145528662, aggrin_queue=145528662, aggrutil=97.98%
    dm-4: ios=0/487480, merge=0/0, ticks=0/145528662, in_queue=145528662, util=97.98%, aggrios=0/121884, aggrmerge=0/0, aggrticks=0/36382023, aggrin_queue=36382023, aggrutil=97.97%
  dm-1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
  dm-2: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
    dm-0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    drbd1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/487536, aggrmerge=0/0, aggrticks=0/145526951, aggrin_queue=145526951, aggrutil=97.97%
    md1: ios=0/487536, merge=0/0, ticks=0/145526951, in_queue=145526951, util=97.97%, aggrios=0/239754, aggrmerge=0/4022, aggrticks=0/19940675, aggrin_queue=19941042, aggrutil=91.49%
  sde: ios=0/239866, merge=0/3796, ticks=0/16199136, in_queue=16199382, util=82.01%
  sdc: ios=0/239807, merge=0/3855, ticks=0/18083690, in_queue=18084132, util=84.10%
  sdb: ios=0/239757, merge=0/4135, ticks=0/21295175, in_queue=21295498, util=88.72%
  sda: ios=0/239587, merge=0/4305, ticks=0/24184702, in_queue=24185156, util=91.49%
  dm-3: ios=0/487536, merge=0/0, ticks=0/145528092, in_queue=145528092, util=97.97%