random-write: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=posixaio, iodepth=16
...
fio-3.28
Starting 16 processes

random-write: (groupid=0, jobs=1): err= 0: pid=771: Mon Dec 18 13:43:25 2023
  write: IOPS=221, BW=13.8MiB/s (14.5MB/s)(833MiB/60260msec); 0 zone resets
    slat (nsec): min=1251, max=5700.1k, avg=9447.99, stdev=57644.79
    clat (usec): min=4, max=661089, avg=71812.45, stdev=33959.01
     lat (usec): min=103, max=661099, avg=71821.90, stdev=33958.57
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   32], 10.00th=[   40], 20.00th=[   52],
     | 30.00th=[   59], 40.00th=[   65], 50.00th=[   71], 60.00th=[   77],
     | 70.00th=[   83], 80.00th=[   90], 90.00th=[  103], 95.00th=[  112],
     | 99.00th=[  140], 99.50th=[  163], 99.90th=[  510], 99.95th=[  642],
     | 99.99th=[  659]
   bw (  KiB/s): min= 4736, max=27720, per=6.35%, avg=14328.50, stdev=2816.03, samples=118
   iops        : min=   74, max=  433, avg=223.64, stdev=44.02, samples=118
  lat (usec)   : 10=0.02%, 250=0.03%, 500=0.02%, 750=0.02%, 1000=0.02%
  lat (msec)   : 2=0.09%, 4=0.02%, 10=0.12%, 20=1.43%, 50=17.16%
  lat (msec)   : 100=69.83%, 250=10.94%, 500=0.20%, 750=0.12%
  cpu          : usr=0.46%, sys=0.81%, ctx=11326, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=51.2%, 16=48.5%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.6%, 8=2.9%, 16=0.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13322,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=772: Mon Dec 18 13:43:25 2023
  write: IOPS=216, BW=13.5MiB/s (14.2MB/s)(816MiB/60270msec); 0 zone resets
    slat (nsec): min=1692, max=3011.6k, avg=9240.89, stdev=39675.19
    clat (msec): min=9, max=976, avg=72.79, stdev=49.54
     lat (msec): min=9, max=976, avg=72.80, stdev=49.54
    clat percentiles (msec):
     |  1.00th=[   19],  5.00th=[   34], 10.00th=[   42], 20.00th=[   52],
     | 30.00th=[   58], 40.00th=[   65], 50.00th=[   70], 60.00th=[   77],
     | 70.00th=[   83], 80.00th=[   90], 90.00th=[  100], 95.00th=[  109],
     | 99.00th=[  133], 99.50th=[  150], 99.90th=[  953], 99.95th=[  969],
     | 99.99th=[  978]
   bw (  KiB/s): min= 3449, max=28288, per=6.33%, avg=14278.09, stdev=2471.76, samples=116
   iops        : min=   53, max=  442, avg=222.84, stdev=38.67, samples=116
  lat (msec)   : 10=0.02%, 20=1.12%, 50=16.98%, 100=72.75%, 250=8.80%
  lat (msec)   : 500=0.05%, 1000=0.28%
  cpu          : usr=0.44%, sys=0.83%, ctx=11009, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=51.2%, 16=48.6%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.8%, 8=2.9%, 16=0.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13052,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=773: Mon Dec 18 13:43:25 2023
  write: IOPS=222, BW=13.9MiB/s (14.6MB/s)(845MiB/60637msec); 0 zone resets
    slat (nsec): min=1138, max=6599.3k, avg=9121.58, stdev=62853.24
    clat (usec): min=205, max=943843, avg=70436.09, stdev=38433.34
     lat (usec): min=229, max=943858, avg=70445.22, stdev=38433.26
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[   27], 10.00th=[   39], 20.00th=[   50],
     | 30.00th=[   58], 40.00th=[   64], 50.00th=[   70], 60.00th=[   75],
     | 70.00th=[   83], 80.00th=[   89], 90.00th=[  100], 95.00th=[  110],
     | 99.00th=[  136], 99.50th=[  163], 99.90th=[  625], 99.95th=[  919],
     | 99.99th=[  944]
   bw (  KiB/s): min=10368, max=54163, per=6.50%, avg=14665.18, stdev=4348.45, samples=117
   iops        : min=  162, max=  846, avg=228.90, stdev=67.95, samples=117
  lat (usec)   : 250=0.01%, 750=0.03%, 1000=0.15%
  lat (msec)   : 2=0.60%, 4=0.47%, 10=0.38%, 20=1.57%, 50=17.36%
  lat (msec)   : 100=69.73%, 250=9.53%, 750=0.11%, 1000=0.07%
  cpu          : usr=0.42%, sys=0.85%, ctx=11819, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.2%, 4=0.5%, 8=51.3%, 16=48.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.9%, 8=3.5%, 16=0.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13516,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=774: Mon Dec 18 13:43:25 2023
  write: IOPS=228, BW=14.3MiB/s (15.0MB/s)(867MiB/60724msec); 0 zone resets
    slat (nsec): min=727, max=17728k, avg=10802.39, stdev=176482.84
    clat (usec): min=504, max=382703, avg=69279.77, stdev=29106.55
     lat (usec): min=510, max=382711, avg=69290.57, stdev=29104.40
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[   27], 10.00th=[   36], 20.00th=[   49],
     | 30.00th=[   57], 40.00th=[   64], 50.00th=[   70], 60.00th=[   75],
     | 70.00th=[   82], 80.00th=[   89], 90.00th=[  100], 95.00th=[  109],
     | 99.00th=[  146], 99.50th=[  197], 99.90th=[  338], 99.95th=[  347],
     | 99.99th=[  380]
   bw (  KiB/s): min= 7936, max=54912, per=6.53%, avg=14746.89, stdev=4471.32, samples=120
   iops        : min=  124, max=  858, avg=230.17, stdev=69.89, samples=120
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.71%, 4=0.46%, 10=0.82%, 20=1.54%, 50=18.05%
  lat (msec)   : 100=69.27%, 250=8.89%, 500=0.22%
  cpu          : usr=0.47%, sys=0.74%, ctx=10611, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.4%, 8=51.5%, 16=48.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=97.1%, 8=2.5%, 16=0.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13869,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=775: Mon Dec 18 13:43:25 2023
  write: IOPS=218, BW=13.7MiB/s (14.3MB/s)(825MiB/60417msec); 0 zone resets
    slat (nsec): min=866, max=6196.6k, avg=9281.92, stdev=64690.55
    clat (usec): min=165, max=1029.0k, avg=72237.27, stdev=55017.02
     lat (usec): min=211, max=1029.0k, avg=72246.55, stdev=55016.67
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   30], 10.00th=[   40], 20.00th=[   51],
     | 30.00th=[   58], 40.00th=[   65], 50.00th=[   71], 60.00th=[   75],
     | 70.00th=[   82], 80.00th=[   89], 90.00th=[  100], 95.00th=[  109],
     | 99.00th=[  129], 99.50th=[  140], 99.90th=[ 1011], 99.95th=[ 1020],
     | 99.99th=[ 1028]
   bw (  KiB/s): min= 8576, max=30720, per=6.40%, avg=14443.41, stdev=2649.43, samples=116
   iops        : min=  134, max=  480, avg=225.45, stdev=41.44, samples=116
  lat (usec)   : 250=0.01%
  lat (msec)   : 2=0.30%, 4=0.19%, 10=0.14%, 20=1.55%, 50=16.95%
  lat (msec)   : 100=71.47%, 250=9.06%, 1000=0.21%, 2000=0.11%
  cpu          : usr=0.45%, sys=0.82%, ctx=10777, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=51.0%, 16=48.6%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.1%, 8=3.4%, 16=0.4%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13202,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=776: Mon Dec 18 13:43:25 2023
  write: IOPS=216, BW=13.6MiB/s (14.2MB/s)(820MiB/60506msec); 0 zone resets
    slat (nsec): min=1217, max=8107.0k, avg=9603.61, stdev=89424.59
    clat (usec): min=1958, max=1017.0k, avg=72729.43, stdev=48003.54
     lat (usec): min=1960, max=1017.0k, avg=72739.03, stdev=48002.58
    clat percentiles (msec):
     |  1.00th=[   18],  5.00th=[   33], 10.00th=[   42], 20.00th=[   51],
     | 30.00th=[   58], 40.00th=[   65], 50.00th=[   70], 60.00th=[   77],
     | 70.00th=[   83], 80.00th=[   90], 90.00th=[  101], 95.00th=[  110],
     | 99.00th=[  140], 99.50th=[  180], 99.90th=[  944], 99.95th=[  969],
     | 99.99th=[ 1011]
   bw (  KiB/s): min= 6131, max=24576, per=6.31%, avg=14236.08, stdev=2320.23, samples=117
   iops        : min=   95, max=  384, avg=222.20, stdev=36.30, samples=117
  lat (msec)   : 2=0.02%, 4=0.08%, 10=0.15%, 20=1.20%, 50=17.76%
  lat (msec)   : 100=71.01%, 250=9.49%, 500=0.07%, 1000=0.21%, 2000=0.02%
  cpu          : usr=0.43%, sys=0.78%, ctx=10805, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=51.3%, 16=48.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.5%, 8=3.2%, 16=0.4%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13127,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=777: Mon Dec 18 13:43:25 2023
  write: IOPS=225, BW=14.1MiB/s (14.8MB/s)(851MiB/60484msec); 0 zone resets
    slat (nsec): min=885, max=5278.3k, avg=9396.60, stdev=63530.96
    clat (usec): min=1470, max=493907, avg=70462.45, stdev=30598.34
     lat (usec): min=1473, max=493913, avg=70471.85, stdev=30599.10
    clat percentiles (msec):
     |  1.00th=[    9],  5.00th=[   28], 10.00th=[   37], 20.00th=[   50],
     | 30.00th=[   58], 40.00th=[   64], 50.00th=[   70], 60.00th=[   75],
     | 70.00th=[   83], 80.00th=[   90], 90.00th=[  102], 95.00th=[  113],
     | 99.00th=[  159], 99.50th=[  199], 99.90th=[  464], 99.95th=[  489],
     | 99.99th=[  493]
   bw (  KiB/s): min= 6463, max=42922, per=6.43%, avg=14520.61, stdev=3960.71, samples=119
   iops        : min=  100, max=  670, avg=226.66, stdev=61.87, samples=119
  lat (msec)   : 2=0.17%, 4=0.42%, 10=0.69%, 20=1.62%, 50=17.83%
  lat (msec)   : 100=68.82%, 250=10.33%, 500=0.12%
  cpu          : usr=0.42%, sys=0.82%, ctx=10558, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.2%, 4=0.6%, 8=51.1%, 16=48.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.7%, 8=1.5%, 16=1.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13615,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=778: Mon Dec 18 13:43:25 2023
  write: IOPS=221, BW=13.9MiB/s (14.5MB/s)(838MiB/60508msec); 0 zone resets
    slat (nsec): min=975, max=9910.0k, avg=9848.13, stdev=106475.73
    clat (usec): min=216, max=971198, avg=71504.15, stdev=44183.77
     lat (usec): min=221, max=971209, avg=71514.00, stdev=44184.56
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   31], 10.00th=[   39], 20.00th=[   51],
     | 30.00th=[   58], 40.00th=[   64], 50.00th=[   70], 60.00th=[   75],
     | 70.00th=[   82], 80.00th=[   90], 90.00th=[  100], 95.00th=[  108],
     | 99.00th=[  138], 99.50th=[  180], 99.90th=[  927], 99.95th=[  961],
     | 99.99th=[  969]
   bw (  KiB/s): min= 7984, max=30403, per=6.43%, avg=14505.11, stdev=2928.61, samples=118
   iops        : min=  124, max=  475, avg=226.42, stdev=45.78, samples=118
  lat (usec)   : 250=0.01%, 750=0.07%, 1000=0.02%
  lat (msec)   : 2=0.12%, 10=0.03%, 20=1.46%, 50=18.27%, 100=71.14%
  lat (msec)   : 250=8.54%, 500=0.07%, 750=0.15%, 1000=0.10%
  cpu          : usr=0.43%, sys=0.78%, ctx=10854, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=51.3%, 16=48.3%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.4%, 8=3.3%, 16=0.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13415,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=779: Mon Dec 18 13:43:25 2023
  write: IOPS=225, BW=14.1MiB/s (14.8MB/s)(855MiB/60545msec); 0 zone resets
    slat (nsec): min=1008, max=5689.3k, avg=9419.63, stdev=68115.67
    clat (usec): min=807, max=378995, avg=70196.51, stdev=28385.25
     lat (usec): min=811, max=379005, avg=70205.93, stdev=28385.67
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   30], 10.00th=[   37], 20.00th=[   50],
     | 30.00th=[   57], 40.00th=[   64], 50.00th=[   70], 60.00th=[   77],
     | 70.00th=[   83], 80.00th=[   90], 90.00th=[  100], 95.00th=[  109],
     | 99.00th=[  144], 99.50th=[  176], 99.90th=[  334], 99.95th=[  347],
     | 99.99th=[  376]
   bw (  KiB/s): min= 6016, max=35456, per=6.46%, avg=14581.55, stdev=3208.81, samples=119
   iops        : min=   94, max=  554, avg=227.60, stdev=50.15, samples=119
  lat (usec)   : 1000=0.07%
  lat (msec)   : 2=0.29%, 4=0.12%, 10=0.15%, 20=1.58%, 50=19.06%
  lat (msec)   : 100=69.08%, 250=9.43%, 500=0.22%
  cpu          : usr=0.47%, sys=0.78%, ctx=10220, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.2%, 4=0.5%, 8=51.4%, 16=47.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.8%, 8=2.8%, 16=0.4%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13675,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=780: Mon Dec 18 13:43:25 2023
  write: IOPS=224, BW=14.0MiB/s (14.7MB/s)(844MiB/60230msec); 0 zone resets
    slat (nsec): min=1251, max=5051.8k, avg=9504.23, stdev=56794.37
    clat (usec): min=1190, max=541809, avg=70737.02, stdev=32016.09
     lat (usec): min=1195, max=541816, avg=70746.53, stdev=32016.12
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   29], 10.00th=[   37], 20.00th=[   50],
     | 30.00th=[   57], 40.00th=[   64], 50.00th=[   70], 60.00th=[   77],
     | 70.00th=[   83], 80.00th=[   90], 90.00th=[  101], 95.00th=[  112],
     | 99.00th=[  146], 99.50th=[  188], 99.90th=[  460], 99.95th=[  523],
     | 99.99th=[  542]
   bw (  KiB/s): min= 2039, max=37750, per=6.39%, avg=14413.35, stdev=3757.24, samples=119
   iops        : min=   31, max=  589, avg=224.98, stdev=58.70, samples=119
  lat (msec)   : 2=0.56%, 4=0.09%, 10=0.33%, 20=1.50%, 50=18.33%
  lat (msec)   : 100=68.77%, 250=10.13%, 500=0.24%, 750=0.06%
  cpu          : usr=0.44%, sys=0.84%, ctx=10842, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=51.2%, 16=48.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.0%, 8=3.1%, 16=0.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13511,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=781: Mon Dec 18 13:43:25 2023
  write: IOPS=226, BW=14.2MiB/s (14.8MB/s)(857MiB/60520msec); 0 zone resets
    slat (nsec): min=724, max=817179, avg=8402.54, stdev=14803.89
    clat (usec): min=432, max=330086, avg=70101.86, stdev=27513.11
     lat (usec): min=437, max=330094, avg=70110.26, stdev=27513.36
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   30], 10.00th=[   39], 20.00th=[   50],
     | 30.00th=[   57], 40.00th=[   64], 50.00th=[   70], 60.00th=[   75],
     | 70.00th=[   82], 80.00th=[   90], 90.00th=[  101], 95.00th=[  111],
     | 99.00th=[  155], 99.50th=[  171], 99.90th=[  292], 99.95th=[  296],
     | 99.99th=[  321]
   bw (  KiB/s): min= 8853, max=36352, per=6.48%, avg=14624.05, stdev=3271.52, samples=119
   iops        : min=  138, max=  568, avg=228.26, stdev=51.13, samples=119
  lat (usec)   : 500=0.02%, 1000=0.02%
  lat (msec)   : 2=0.62%, 4=0.18%, 10=0.14%, 20=1.28%, 50=18.02%
  lat (msec)   : 100=69.64%, 250=9.88%, 500=0.20%
  cpu          : usr=0.41%, sys=0.78%, ctx=9899, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.4%, 8=51.6%, 16=48.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.9%, 8=3.6%, 16=0.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13709,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=782: Mon Dec 18 13:43:25 2023
  write: IOPS=222, BW=13.9MiB/s (14.6MB/s)(844MiB/60682msec); 0 zone resets
    slat (nsec): min=1447, max=1158.6k, avg=8804.38, stdev=22886.36
    clat (usec): min=675, max=944991, avg=70811.92, stdev=39604.52
     lat (usec): min=684, max=944999, avg=70820.73, stdev=39604.43
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   29], 10.00th=[   39], 20.00th=[   50],
     | 30.00th=[   57], 40.00th=[   63], 50.00th=[   69], 60.00th=[   75],
     | 70.00th=[   83], 80.00th=[   90], 90.00th=[  101], 95.00th=[  110],
     | 99.00th=[  140], 99.50th=[  188], 99.90th=[  902], 99.95th=[  927],
     | 99.99th=[  944]
   bw (  KiB/s): min= 5632, max=29354, per=6.43%, avg=14518.38, stdev=2999.52, samples=118
   iops        : min=   88, max=  458, avg=226.62, stdev=46.87, samples=118
  lat (usec)   : 750=0.01%
  lat (msec)   : 2=0.39%, 4=0.20%, 10=0.16%, 20=1.51%, 50=18.72%
  lat (msec)   : 100=69.17%, 250=9.53%, 500=0.21%, 1000=0.10%
  cpu          : usr=0.44%, sys=0.80%, ctx=11486, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.2%, 4=0.5%, 8=51.2%, 16=48.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.4%, 8=2.9%, 16=0.7%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13499,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=783: Mon Dec 18 13:43:25 2023
  write: IOPS=217, BW=13.6MiB/s (14.2MB/s)(824MiB/60670msec); 0 zone resets
    slat (nsec): min=1390, max=5663.2k, avg=9511.05, stdev=58431.24
    clat (usec): min=1530, max=897186, avg=72593.53, stdev=40981.46
     lat (usec): min=1580, max=897191, avg=72603.04, stdev=40980.99
    clat percentiles (msec):
     |  1.00th=[   18],  5.00th=[   33], 10.00th=[   41], 20.00th=[   52],
     | 30.00th=[   58], 40.00th=[   65], 50.00th=[   70], 60.00th=[   77],
     | 70.00th=[   83], 80.00th=[   90], 90.00th=[  101], 95.00th=[  111],
     | 99.00th=[  144], 99.50th=[  180], 99.90th=[  844], 99.95th=[  877],
     | 99.99th=[  894]
   bw (  KiB/s): min= 2427, max=28614, per=6.29%, avg=14189.75, stdev=2763.52, samples=118
   iops        : min=   37, max=  447, avg=221.48, stdev=43.24, samples=118
  lat (msec)   : 2=0.07%, 10=0.04%, 20=0.96%, 50=17.70%, 100=71.20%
  lat (msec)   : 250=9.62%, 500=0.20%, 750=0.11%, 1000=0.11%
  cpu          : usr=0.42%, sys=0.78%, ctx=11470, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=51.2%, 16=48.5%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=97.8%, 8=2.0%, 16=0.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13187,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=784: Mon Dec 18 13:43:25 2023
  write: IOPS=216, BW=13.5MiB/s (14.2MB/s)(819MiB/60480msec); 0 zone resets
    slat (nsec): min=1940, max=1581.5k, avg=8493.79, stdev=19020.29
    clat (usec): min=1996, max=901558, avg=73045.10, stdev=46294.29
     lat (msec): min=2, max=901, avg=73.05, stdev=46.29
    clat percentiles (msec):
     |  1.00th=[   18],  5.00th=[   34], 10.00th=[   42], 20.00th=[   52],
     | 30.00th=[   58], 40.00th=[   65], 50.00th=[   70], 60.00th=[   77],
     | 70.00th=[   83], 80.00th=[   90], 90.00th=[  101], 95.00th=[  111],
     | 99.00th=[  142], 99.50th=[  251], 99.90th=[  894], 99.95th=[  894],
     | 99.99th=[  902]
   bw (  KiB/s): min= 5632, max=30019, per=6.30%, avg=14215.52, stdev=2567.29, samples=117
   iops        : min=   88, max=  469, avg=221.90, stdev=40.11, samples=117
  lat (msec)   : 2=0.01%, 10=0.12%, 20=1.06%, 50=17.77%, 100=70.98%
  lat (msec)   : 250=9.57%, 500=0.24%, 750=0.03%, 1000=0.21%
  cpu          : usr=0.42%, sys=0.81%, ctx=10422, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=51.0%, 16=48.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.7%, 8=2.9%, 16=0.4%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13103,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=785: Mon Dec 18 13:43:25 2023
  write: IOPS=218, BW=13.6MiB/s (14.3MB/s)(827MiB/60643msec); 0 zone resets
    slat (nsec): min=1760, max=9875.1k, avg=10103.36, stdev=96515.16
    clat (usec): min=894, max=709926, avg=72498.82, stdev=37901.14
     lat (usec): min=907, max=709936, avg=72508.92, stdev=37900.53
    clat percentiles (msec):
     |  1.00th=[   18],  5.00th=[   33], 10.00th=[   41], 20.00th=[   52],
     | 30.00th=[   59], 40.00th=[   65], 50.00th=[   71], 60.00th=[   77],
     | 70.00th=[   83], 80.00th=[   90], 90.00th=[  102], 95.00th=[  113],
     | 99.00th=[  140], 99.50th=[  184], 99.90th=[  684], 99.95th=[  693],
     | 99.99th=[  709]
   bw (  KiB/s): min= 2043, max=30909, per=6.31%, avg=14235.32, stdev=3001.27, samples=118
   iops        : min=   31, max=  482, avg=222.19, stdev=46.91, samples=118
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.09%, 4=0.01%, 10=0.05%, 20=1.25%, 50=17.27%
  lat (msec)   : 100=70.53%, 250=10.47%, 500=0.10%, 750=0.23%
  cpu          : usr=0.48%, sys=0.72%, ctx=10113, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=51.2%, 16=48.5%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.4%, 8=3.4%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13230,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=786: Mon Dec 18 13:43:25 2023
  write: IOPS=216, BW=13.5MiB/s (14.2MB/s)(818MiB/60602msec); 0 zone resets
    slat (nsec): min=1195, max=3927.9k, avg=9091.85, stdev=43938.79
    clat (usec): min=1421, max=997082, avg=72975.15, stdev=48251.46
     lat (usec): min=1422, max=997087, avg=72984.24, stdev=48251.67
    clat percentiles (msec):
     |  1.00th=[   18],  5.00th=[   33], 10.00th=[   42], 20.00th=[   52],
     | 30.00th=[   58], 40.00th=[   65], 50.00th=[   70], 60.00th=[   77],
     | 70.00th=[   83], 80.00th=[   90], 90.00th=[  101], 95.00th=[  110],
     | 99.00th=[  138], 99.50th=[  176], 99.90th=[  911], 99.95th=[  944],
     | 99.99th=[  995]
   bw (  KiB/s): min= 1750, max=24478, per=6.24%, avg=14082.92, stdev=2837.91, samples=118
   iops        : min=   27, max=  382, avg=219.81, stdev=44.39, samples=118
  lat (msec)   : 2=0.12%, 4=0.05%, 10=0.11%, 20=0.87%, 50=17.15%
  lat (msec)   : 100=71.79%, 250=9.58%, 750=0.11%, 1000=0.21%
  cpu          : usr=0.43%, sys=0.88%, ctx=10604, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=51.4%, 16=48.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.7%, 8=2.4%, 16=1.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,13092,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=220MiB/s (231MB/s), 13.5MiB/s-14.3MiB/s (14.2MB/s-15.0MB/s), io=13.1GiB (14.0GB), run=60230-60724msec

Disk stats (read/write):
    dm-7: ios=0/179853, merge=0/0, ticks=0/15629810, in_queue=15629810, util=95.95%, aggrios=0/184447, aggrmerge=0/0, aggrticks=0/16105569, aggrin_queue=16105569, aggrutil=95.97%
    dm-6: ios=0/184447, merge=0/0, ticks=0/16105569, in_queue=16105569, util=95.97%, aggrios=0/184447, aggrmerge=0/0, aggrticks=0/16103272, aggrin_queue=16103272, aggrutil=95.97%
    dm-4: ios=0/184447, merge=0/0, ticks=0/16103272, in_queue=16103272, util=95.97%, aggrios=0/46125, aggrmerge=0/0, aggrticks=0/4025703, aggrin_queue=4025703, aggrutil=96.00%
  dm-1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
  dm-2: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
    dm-0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    drbd1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/184501, aggrmerge=0/0, aggrticks=0/16101864, aggrin_queue=16101864, aggrutil=95.98%
    md1: ios=0/184501, merge=0/0, ticks=0/16101864, in_queue=16101864, util=95.98%, aggrios=224480/399737, aggrmerge=1283070/1245157, aggrticks=523712/611985, aggrin_queue=1136146, aggrutil=86.37%
  sde: ios=218904/388225, merge=1288433/1256988, ticks=569285/626406, in_queue=1196131, util=86.28%
  sdc: ios=228630/404307, merge=1277323/1242070, ticks=467129/538719, in_queue=1006268, util=85.87%
  sdb: ios=222294/396563, merge=1285446/1248225, ticks=553078/668806, in_queue=1222341, util=86.37%
  sda: ios=228092/409856, merge=1281080/1233348, ticks=505356/614012, in_queue=1119847, util=86.28%
  dm-3: ios=0/184501, merge=0/0, ticks=0/16102814, in_queue=16102814, util=96.00%