random-write: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=posixaio, iodepth=16
...
fio-3.28
Starting 16 processes

random-write: (groupid=0, jobs=1): err= 0: pid=760: Mon Dec 18 08:42:08 2023
  write: IOPS=662, BW=41.4MiB/s (43.4MB/s)(2494MiB/60221msec); 0 zone resets
    slat (nsec): min=595, max=3403.2k, avg=8315.93, stdev=30836.99
    clat (usec): min=1059, max=162211, avg=23954.76, stdev=10010.71
     lat (usec): min=1064, max=162220, avg=23963.08, stdev=10010.67
    clat percentiles (msec):
     |  1.00th=[    5],  5.00th=[   15], 10.00th=[   17], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   22], 60.00th=[   24],
     | 70.00th=[   26], 80.00th=[   29], 90.00th=[   35], 95.00th=[   41],
     | 99.00th=[   55], 99.50th=[   64], 99.90th=[  128], 99.95th=[  140],
     | 99.99th=[  153]
   bw (  KiB/s): min=20608, max=112384, per=6.35%, avg=42582.08, stdev=11447.14, samples=119
   iops        : min=  322, max= 1756, avg=665.24, stdev=178.85, samples=119
  lat (msec)   : 2=0.22%, 4=0.41%, 10=1.04%, 20=34.79%, 50=61.81%
  lat (msec)   : 100=1.43%, 250=0.30%
  cpu          : usr=1.23%, sys=1.85%, ctx=28097, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.0%, 8=3.2%, 16=0.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,39907,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=761: Mon Dec 18 08:42:08 2023
  write: IOPS=663, BW=41.5MiB/s (43.5MB/s)(2498MiB/60222msec); 0 zone resets
    slat (nsec): min=663, max=1900.5k, avg=8324.35, stdev=28527.55
    clat (usec): min=1305, max=168495, avg=23898.64, stdev=9961.75
     lat (usec): min=1316, max=168504, avg=23906.96, stdev=9961.49
    clat percentiles (msec):
     |  1.00th=[    5],  5.00th=[   15], 10.00th=[   16], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   22], 60.00th=[   24],
     | 70.00th=[   26], 80.00th=[   29], 90.00th=[   35], 95.00th=[   41],
     | 99.00th=[   54], 99.50th=[   62], 99.90th=[  136], 99.95th=[  144],
     | 99.99th=[  167]
   bw (  KiB/s): min=22656, max=103680, per=6.36%, avg=42645.45, stdev=11012.41, samples=119
   iops        : min=  354, max= 1620, avg=666.22, stdev=172.07, samples=119
  lat (msec)   : 2=0.04%, 4=0.35%, 10=1.08%, 20=35.28%, 50=61.73%
  lat (msec)   : 100=1.24%, 250=0.28%
  cpu          : usr=1.19%, sys=1.91%, ctx=28390, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.7%, 16=49.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.5%, 8=4.0%, 16=0.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,39967,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=762: Mon Dec 18 08:42:08 2023
  write: IOPS=661, BW=41.3MiB/s (43.3MB/s)(2488MiB/60207msec); 0 zone resets
    slat (nsec): min=624, max=3960.4k, avg=8463.59, stdev=40292.47
    clat (usec): min=1293, max=176968, avg=23961.72, stdev=9729.91
     lat (usec): min=1312, max=176975, avg=23970.18, stdev=9729.58
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[   15], 10.00th=[   17], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   22], 60.00th=[   24],
     | 70.00th=[   26], 80.00th=[   29], 90.00th=[   35], 95.00th=[   41],
     | 99.00th=[   55], 99.50th=[   62], 99.90th=[  124], 99.95th=[  157],
     | 99.99th=[  174]
   bw (  KiB/s): min=23424, max=86016, per=6.33%, avg=42473.32, stdev=10268.35, samples=119
   iops        : min=  366, max= 1344, avg=663.53, stdev=160.44, samples=119
  lat (msec)   : 2=0.01%, 4=0.11%, 10=1.03%, 20=35.78%, 50=61.45%
  lat (msec)   : 100=1.43%, 250=0.19%
  cpu          : usr=1.19%, sys=1.95%, ctx=28257, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.7%, 16=49.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.9%, 8=2.2%, 16=1.9%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,39809,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=763: Mon Dec 18 08:42:08 2023
  write: IOPS=655, BW=40.9MiB/s (42.9MB/s)(2466MiB/60220msec); 0 zone resets
    slat (nsec): min=654, max=7287.6k, avg=8336.20, stdev=42824.48
    clat (usec): min=1312, max=227695, avg=24189.62, stdev=10611.30
     lat (usec): min=1331, max=227702, avg=24197.95, stdev=10611.01
    clat percentiles (msec):
     |  1.00th=[   13],  5.00th=[   16], 10.00th=[   17], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   22], 60.00th=[   24],
     | 70.00th=[   26], 80.00th=[   29], 90.00th=[   35], 95.00th=[   41],
     | 99.00th=[   54], 99.50th=[   66], 99.90th=[  146], 99.95th=[  157],
     | 99.99th=[  228]
   bw (  KiB/s): min=19584, max=62984, per=6.27%, avg=42081.31, stdev=9710.87, samples=119
   iops        : min=  306, max=  984, avg=657.42, stdev=151.71, samples=119
  lat (msec)   : 2=0.16%, 4=0.03%, 10=0.23%, 20=35.19%, 50=62.92%
  lat (msec)   : 100=1.15%, 250=0.32%
  cpu          : usr=1.15%, sys=1.90%, ctx=27699, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.1%, 8=2.9%, 16=0.9%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,39452,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=764: Mon Dec 18 08:42:08 2023
  write: IOPS=653, BW=40.8MiB/s (42.8MB/s)(2457MiB/60188msec); 0 zone resets
    slat (nsec): min=1861, max=4117.6k, avg=8235.31, stdev=31177.92
    clat (usec): min=1131, max=342840, avg=24182.85, stdev=10680.08
     lat (usec): min=1146, max=342846, avg=24191.09, stdev=10679.93
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   16], 10.00th=[   17], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   22], 60.00th=[   24],
     | 70.00th=[   26], 80.00th=[   29], 90.00th=[   34], 95.00th=[   41],
     | 99.00th=[   53], 99.50th=[   61], 99.90th=[  129], 99.95th=[  205],
     | 99.99th=[  338]
   bw (  KiB/s): min=15360, max=61312, per=6.25%, avg=41930.32, stdev=9999.85, samples=119
   iops        : min=  240, max=  958, avg=655.11, stdev=156.25, samples=119
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.08%, 20=35.51%, 50=62.91%
  lat (msec)   : 100=1.25%, 250=0.22%, 500=0.02%
  cpu          : usr=1.21%, sys=1.72%, ctx=26748, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.7%, 16=49.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.7%, 8=3.5%, 16=0.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,39308,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=765: Mon Dec 18 08:42:08 2023
  write: IOPS=654, BW=40.9MiB/s (42.9MB/s)(2461MiB/60160msec); 0 zone resets
    slat (nsec): min=1870, max=2610.0k, avg=8474.62, stdev=29683.15
    clat (usec): min=1263, max=339809, avg=24151.19, stdev=10907.68
     lat (usec): min=1293, max=339815, avg=24159.67, stdev=10907.22
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   16], 10.00th=[   17], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   22], 60.00th=[   24],
     | 70.00th=[   26], 80.00th=[   29], 90.00th=[   35], 95.00th=[   40],
     | 99.00th=[   53], 99.50th=[   62], 99.90th=[  161], 99.95th=[  171],
     | 99.99th=[  334]
   bw (  KiB/s): min=17664, max=63744, per=6.26%, avg=42007.34, stdev=10179.43, samples=119
   iops        : min=  276, max=  996, avg=656.29, stdev=159.06, samples=119
  lat (msec)   : 2=0.02%, 4=0.08%, 10=0.03%, 20=35.63%, 50=62.87%
  lat (msec)   : 100=1.14%, 250=0.21%, 500=0.03%
  cpu          : usr=1.22%, sys=1.80%, ctx=27003, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.7%, 16=49.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.9%, 8=2.9%, 16=1.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,39372,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=766: Mon Dec 18 08:42:08 2023
  write: IOPS=656, BW=41.1MiB/s (43.1MB/s)(2470MiB/60150msec); 0 zone resets
    slat (nsec): min=1798, max=4229.4k, avg=8610.57, stdev=42563.30
    clat (usec): min=818, max=223608, avg=24155.38, stdev=9906.44
     lat (usec): min=834, max=223618, avg=24163.99, stdev=9906.06
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   16], 10.00th=[   17], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   22], 60.00th=[   24],
     | 70.00th=[   26], 80.00th=[   29], 90.00th=[   35], 95.00th=[   41],
     | 99.00th=[   55], 99.50th=[   65], 99.90th=[  117], 99.95th=[  142],
     | 99.99th=[  218]
   bw (  KiB/s): min=23296, max=62208, per=6.29%, avg=42166.86, stdev=9651.57, samples=119
   iops        : min=  364, max=  972, avg=658.76, stdev=150.86, samples=119
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.08%, 10=0.11%, 20=35.83%, 50=62.33%, 100=1.40%
  lat (msec)   : 250=0.25%
  cpu          : usr=1.22%, sys=1.79%, ctx=27252, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.9%, 8=3.1%, 16=1.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,39517,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=767: Mon Dec 18 08:42:08 2023
  write: IOPS=656, BW=41.0MiB/s (43.0MB/s)(2466MiB/60136msec); 0 zone resets
    slat (nsec): min=1854, max=1351.3k, avg=8157.93, stdev=20602.28
    clat (usec): min=1903, max=209824, avg=24139.28, stdev=9864.22
     lat (usec): min=1918, max=209834, avg=24147.44, stdev=9863.85
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   16], 10.00th=[   17], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   22], 60.00th=[   24],
     | 70.00th=[   26], 80.00th=[   29], 90.00th=[   35], 95.00th=[   41],
     | 99.00th=[   53], 99.50th=[   63], 99.90th=[  142], 99.95th=[  153],
     | 99.99th=[  199]
   bw (  KiB/s): min=24192, max=62208, per=6.28%, avg=42106.38, stdev=9565.84, samples=119
   iops        : min=  378, max=  972, avg=657.83, stdev=149.46, samples=119
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.14%, 20=35.72%, 50=62.80%
  lat (msec)   : 100=1.07%, 250=0.25%
  cpu          : usr=1.17%, sys=1.91%, ctx=28080, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.3%, 8=4.2%, 16=0.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,39459,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=768: Mon Dec 18 08:42:08 2023
  write: IOPS=656, BW=41.0MiB/s (43.0MB/s)(2468MiB/60126msec); 0 zone resets
    slat (nsec): min=1693, max=1945.0k, avg=8254.77, stdev=27117.79
    clat (usec): min=1146, max=191630, avg=24144.40, stdev=9550.07
     lat (usec): min=1160, max=191637, avg=24152.65, stdev=9550.64
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   16], 10.00th=[   17], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   23], 60.00th=[   24],
     | 70.00th=[   26], 80.00th=[   29], 90.00th=[   35], 95.00th=[   41],
     | 99.00th=[   53], 99.50th=[   60], 99.90th=[  125], 99.95th=[  155],
     | 99.99th=[  188]
   bw (  KiB/s): min=23552, max=61952, per=6.28%, avg=42141.94, stdev=9845.83, samples=119
   iops        : min=  368, max=  968, avg=658.38, stdev=153.85, samples=119
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.03%, 20=35.63%, 50=63.04%
  lat (msec)   : 100=1.04%, 250=0.25%
  cpu          : usr=1.19%, sys=1.76%, ctx=26459, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=50.6%, 16=49.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.7%, 8=2.8%, 16=1.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,39491,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=769: Mon Dec 18 08:42:08 2023
  write: IOPS=653, BW=40.9MiB/s (42.9MB/s)(2457MiB/60121msec); 0 zone resets
    slat (nsec): min=1647, max=1600.3k, avg=8216.04, stdev=23961.90
    clat (usec): min=1107, max=221585, avg=24189.81, stdev=10470.72
     lat (usec): min=1126, max=221590, avg=24198.02, stdev=10470.62
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   16], 10.00th=[   17], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   22], 60.00th=[   24],
     | 70.00th=[   26], 80.00th=[   29], 90.00th=[   34], 95.00th=[   40],
     | 99.00th=[   53], 99.50th=[   57], 99.90th=[  163], 99.95th=[  203],
     | 99.99th=[  220]
   bw (  KiB/s): min=19968, max=62464, per=6.25%, avg=41950.53, stdev=10137.04, samples=119
   iops        : min=  312, max=  976, avg=655.39, stdev=158.40, samples=119
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.05%, 20=35.67%, 50=62.97%
  lat (msec)   : 100=1.04%, 250=0.26%
  cpu          : usr=1.19%, sys=1.70%, ctx=26277, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.5%, 8=3.5%, 16=1.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,39312,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=770: Mon Dec 18 08:42:08 2023
  write: IOPS=653, BW=40.8MiB/s (42.8MB/s)(2455MiB/60115msec); 0 zone resets
    slat (nsec): min=1885, max=3495.8k, avg=8672.21, stdev=38132.56
    clat (usec): min=839, max=215484, avg=24260.77, stdev=10135.29
     lat (usec): min=859, max=215492, avg=24269.44, stdev=10135.01
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   16], 10.00th=[   17], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   22], 60.00th=[   24],
     | 70.00th=[   26], 80.00th=[   29], 90.00th=[   35], 95.00th=[   41],
     | 99.00th=[   54], 99.50th=[   64], 99.90th=[  153], 99.95th=[  169],
     | 99.99th=[  213]
   bw (  KiB/s): min=18522, max=63872, per=6.25%, avg=41911.34, stdev=10027.19, samples=119
   iops        : min=  289, max=  998, avg=654.78, stdev=156.69, samples=119
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.04%, 20=34.95%, 50=63.46%
  lat (msec)   : 100=1.31%, 250=0.22%
  cpu          : usr=1.19%, sys=1.71%, ctx=25562, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.1%, 8=2.9%, 16=1.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,39280,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=771: Mon Dec 18 08:42:08 2023
  write: IOPS=652, BW=40.8MiB/s (42.8MB/s)(2452MiB/60103msec); 0 zone resets
    slat (nsec): min=1798, max=3309.4k, avg=8117.01, stdev=26880.54
    clat (usec): min=1301, max=328338, avg=24242.36, stdev=11412.83
     lat (usec): min=1319, max=328345, avg=24250.48, stdev=11412.56
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   16], 10.00th=[   17], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   22], 60.00th=[   24],
     | 70.00th=[   26], 80.00th=[   29], 90.00th=[   34], 95.00th=[   40],
     | 99.00th=[   53], 99.50th=[   59], 99.90th=[  174], 99.95th=[  209],
     | 99.99th=[  317]
   bw (  KiB/s): min=16351, max=63616, per=6.24%, avg=41858.55, stdev=10106.05, samples=119
   iops        : min=  255, max=  994, avg=653.98, stdev=157.92, samples=119
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.06%, 20=35.31%, 50=63.20%
  lat (msec)   : 100=1.11%, 250=0.26%, 500=0.04%
  cpu          : usr=1.25%, sys=1.70%, ctx=26161, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.7%, 8=3.5%, 16=0.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,39233,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=772: Mon Dec 18 08:42:08 2023
  write: IOPS=652, BW=40.8MiB/s (42.8MB/s)(2451MiB/60084msec); 0 zone resets
    slat (nsec): min=798, max=3518.5k, avg=8377.21, stdev=29938.48
    clat (usec): min=1018, max=206605, avg=24299.08, stdev=10927.47
     lat (usec): min=1021, max=206672, avg=24307.46, stdev=10927.22
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   16], 10.00th=[   17], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   22], 60.00th=[   24],
     | 70.00th=[   26], 80.00th=[   29], 90.00th=[   35], 95.00th=[   41],
     | 99.00th=[   54], 99.50th=[   63], 99.90th=[  184], 99.95th=[  197],
     | 99.99th=[  205]
   bw (  KiB/s): min=18395, max=64000, per=6.24%, avg=41836.06, stdev=10141.91, samples=119
   iops        : min=  287, max= 1000, avg=653.62, stdev=158.46, samples=119
  lat (msec)   : 2=0.05%, 4=0.03%, 10=0.07%, 20=35.68%, 50=62.65%
  lat (msec)   : 100=1.23%, 250=0.30%
  cpu          : usr=1.15%, sys=1.81%, ctx=26844, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=3.2%, 16=1.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,39221,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=773: Mon Dec 18 08:42:08 2023
  write: IOPS=655, BW=41.0MiB/s (43.0MB/s)(2464MiB/60108msec); 0 zone resets
    slat (nsec): min=1107, max=2256.1k, avg=8384.01, stdev=29080.57
    clat (usec): min=1213, max=239235, avg=24147.75, stdev=10866.01
     lat (usec): min=1229, max=239242, avg=24156.13, stdev=10865.79
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   16], 10.00th=[   17], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   22], 60.00th=[   24],
     | 70.00th=[   26], 80.00th=[   29], 90.00th=[   34], 95.00th=[   40],
     | 99.00th=[   53], 99.50th=[   62], 99.90th=[  169], 99.95th=[  201],
     | 99.99th=[  234]
   bw (  KiB/s): min=21760, max=63360, per=6.27%, avg=42057.97, stdev=10125.58, samples=119
   iops        : min=  340, max=  990, avg=657.12, stdev=158.21, samples=119
  lat (msec)   : 2=0.04%, 4=0.03%, 10=0.03%, 20=36.13%, 50=62.50%
  lat (msec)   : 100=1.00%, 250=0.27%
  cpu          : usr=1.19%, sys=1.76%, ctx=26867, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.7%, 16=49.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=3.7%, 16=0.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,39423,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=774: Mon Dec 18 08:42:08 2023
  write: IOPS=653, BW=40.9MiB/s (42.8MB/s)(2456MiB/60098msec); 0 zone resets
    slat (nsec): min=1188, max=4049.7k, avg=8376.73, stdev=33557.16
    clat (usec): min=898, max=267453, avg=24241.54, stdev=11828.10
     lat (usec): min=903, max=267457, avg=24249.92, stdev=11827.83
    clat percentiles (msec):
     |  1.00th=[   13],  5.00th=[   16], 10.00th=[   17], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   22], 60.00th=[   24],
     | 70.00th=[   26], 80.00th=[   29], 90.00th=[   35], 95.00th=[   41],
     | 99.00th=[   53], 99.50th=[   60], 99.90th=[  226], 99.95th=[  251],
     | 99.99th=[  266]
   bw (  KiB/s): min=16640, max=64510, per=6.24%, avg=41849.65, stdev=10131.96, samples=119
   iops        : min=  260, max= 1007, avg=653.82, stdev=158.30, samples=119
  lat (usec)   : 1000=0.02%
  lat (msec)   : 2=0.07%, 4=0.06%, 10=0.13%, 20=35.19%, 50=63.18%
  lat (msec)   : 100=1.05%, 250=0.25%, 500=0.05%
  cpu          : usr=1.21%, sys=1.74%, ctx=26447, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.7%, 16=49.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.9%, 8=2.8%, 16=1.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,39295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=775: Mon Dec 18 08:42:08 2023
  write: IOPS=651, BW=40.7MiB/s (42.7MB/s)(2448MiB/60088msec); 0 zone resets
    slat (nsec): min=1905, max=3148.2k, avg=8372.34, stdev=29397.57
    clat (usec): min=1417, max=229597, avg=24334.39, stdev=10432.67
     lat (usec): min=1435, max=229605, avg=24342.76, stdev=10432.51
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   16], 10.00th=[   17], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   22], 60.00th=[   24],
     | 70.00th=[   26], 80.00th=[   29], 90.00th=[   35], 95.00th=[   41],
     | 99.00th=[   55], 99.50th=[   63], 99.90th=[  146], 99.95th=[  157],
     | 99.99th=[  215]
   bw (  KiB/s): min=22272, max=62080, per=6.23%, avg=41804.96, stdev=9999.28, samples=119
   iops        : min=  348, max=  970, avg=653.14, stdev=156.24, samples=119
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.03%, 20=35.42%, 50=62.84%
  lat (msec)   : 100=1.39%, 250=0.31%
  cpu          : usr=1.15%, sys=1.84%, ctx=26872, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.7%, 8=2.9%, 16=1.4%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,39167,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=655MiB/s (687MB/s), 40.7MiB/s-41.5MiB/s (42.7MB/s-43.5MB/s), io=38.5GiB (41.4GB), run=60084-60222msec

Disk stats (read/write):
    dm-7: ios=0/559432, merge=0/0, ticks=0/24104399, in_queue=24104399, util=68.43%, aggrios=0/568957, aggrmerge=0/0, aggrticks=0/24565349, aggrin_queue=24565349, aggrutil=68.48%
    dm-6: ios=0/568957, merge=0/0, ticks=0/24565349, in_queue=24565349, util=68.48%, aggrios=0/568957, aggrmerge=0/0, aggrticks=0/24561729, aggrin_queue=24561729, aggrutil=68.48%
    dm-4: ios=0/568957, merge=0/0, ticks=0/24561729, in_queue=24561729, util=68.48%, aggrios=0/142249, aggrmerge=0/0, aggrticks=0/6140160, aggrin_queue=6140160, aggrutil=68.47%
  dm-1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
  dm-2: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
    dm-0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    drbd1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/568997, aggrmerge=0/0, aggrticks=0/24559163, aggrin_queue=24559163, aggrutil=68.47%
    md1: ios=0/568997, merge=0/0, ticks=0/24559163, in_queue=24559163, util=68.47%, aggrios=0/188479, aggrmerge=0/1336, aggrticks=0/8131576, aggrin_queue=8132280, aggrutil=67.57%
  sdc: ios=0/188601, merge=0/1265, ticks=0/7467246, in_queue=7467946, util=62.62%
  sdb: ios=0/188377, merge=0/1322, ticks=0/8184095, in_queue=8184682, util=65.43%
  sda: ios=0/188459, merge=0/1423, ticks=0/8743389, in_queue=8744213, util=67.57%
  dm-3: ios=0/568997, merge=0/0, ticks=0/24560641, in_queue=24560641, util=68.47%