random-write: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=posixaio, iodepth=16
...
fio-3.28
Starting 16 processes
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)

random-write: (groupid=0, jobs=1): err= 0: pid=694: Tue Dec 19 07:26:46 2023
  read: IOPS=1314, BW=82.1MiB/s (86.1MB/s)(4929MiB/60002msec)
    slat (nsec): min=150, max=31462k, avg=2503.26, stdev=128056.28
    clat (usec): min=6, max=784365, avg=11809.67, stdev=39925.42
     lat (usec): min=66, max=784365, avg=11812.17, stdev=39926.33
    clat percentiles (usec):
     |  1.00th=[   594],  5.00th=[   807], 10.00th=[   979], 20.00th=[  1319],
     | 30.00th=[  1860], 40.00th=[  2999], 50.00th=[  4555], 60.00th=[  6128],
     | 70.00th=[  7832], 80.00th=[ 10683], 90.00th=[ 18220], 95.00th=[ 30802],
     | 99.00th=[183501], 99.50th=[278922], 99.90th=[549454], 99.95th=[692061],
     | 99.99th=[784335]
   bw (  KiB/s): min=  896, max=253568, per=5.92%, avg=85232.03, stdev=54405.30, samples=118
   iops        : min=   14, max= 3962, avg=1331.53, stdev=850.09, samples=118
  lat (usec)   : 10=0.01%, 20=0.01%, 100=0.01%, 250=0.02%, 500=0.26%
  lat (usec)   : 750=3.32%, 1000=6.95%
  lat (msec)   : 2=21.20%, 4=14.69%, 10=31.68%, 20=13.02%, 50=5.74%
  lat (msec)   : 100=1.22%, 250=1.29%, 500=0.42%, 750=0.14%, 1000=0.03%
  cpu          : usr=1.09%, sys=2.74%, ctx=55284, majf=0, minf=32
  IO depths    : 1=0.6%, 2=1.3%, 4=3.2%, 8=51.8%, 16=43.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.6%, 8=1.5%, 16=2.9%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=78859,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=695: Tue Dec 19 07:26:46 2023
  read: IOPS=1372, BW=85.8MiB/s (89.9MB/s)(5150MiB/60051msec)
    slat (nsec): min=149, max=14389k, avg=1810.67, stdev=56831.50
    clat (usec): min=8, max=745884, avg=11467.02, stdev=38520.09
     lat (usec): min=117, max=745894, avg=11468.83, stdev=38520.40
    clat percentiles (usec):
     |  1.00th=[   586],  5.00th=[   791], 10.00th=[   963], 20.00th=[  1336],
     | 30.00th=[  1926], 40.00th=[  2933], 50.00th=[  4228], 60.00th=[  5735],
     | 70.00th=[  7570], 80.00th=[ 10552], 90.00th=[ 18220], 95.00th=[ 31327],
     | 99.00th=[183501], 99.50th=[274727], 99.90th=[566232], 99.95th=[700449],
     | 99.99th=[742392]
   bw (  KiB/s): min= 1792, max=210793, per=6.23%, avg=89682.17, stdev=51283.56, samples=117
   iops        : min=   28, max= 3293, avg=1401.07, stdev=801.33, samples=117
  lat (usec)   : 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.23%
  lat (usec)   : 750=3.44%, 1000=7.38%
  lat (msec)   : 2=19.74%, 4=17.61%, 10=30.15%, 20=12.42%, 50=6.27%
  lat (msec)   : 100=0.98%, 250=1.19%, 500=0.45%, 750=0.14%
  cpu          : usr=1.05%, sys=3.01%, ctx=60762, majf=0, minf=36
  IO depths    : 1=0.5%, 2=1.1%, 4=2.8%, 8=51.7%, 16=43.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.5%, 8=1.8%, 16=2.7%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=82404,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=696: Tue Dec 19 07:26:46 2023
  read: IOPS=1500, BW=93.8MiB/s (98.3MB/s)(5632MiB/60071msec)
    slat (nsec): min=152, max=34951k, avg=2318.14, stdev=126273.45
    clat (usec): min=5, max=683863, avg=10358.35, stdev=34401.69
     lat (usec): min=63, max=683864, avg=10360.67, stdev=34402.75
    clat percentiles (usec):
     |  1.00th=[   586],  5.00th=[   799], 10.00th=[   938], 20.00th=[  1237],
     | 30.00th=[  1614], 40.00th=[  2442], 50.00th=[  3621], 60.00th=[  4948],
     | 70.00th=[  6718], 80.00th=[  9372], 90.00th=[ 16319], 95.00th=[ 29230],
     | 99.00th=[158335], 99.50th=[256902], 99.90th=[492831], 99.95th=[599786],
     | 99.99th=[650118]
   bw (  KiB/s): min= 1280, max=346315, per=6.72%, avg=96688.01, stdev=64129.46, samples=119
   iops        : min=   20, max= 5411, avg=1510.54, stdev=1002.04, samples=119
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=0.24%, 750=3.33%, 1000=8.36%
  lat (msec)   : 2=23.11%, 4=17.97%, 10=28.38%, 20=10.80%, 50=4.87%
  lat (msec)   : 100=1.25%, 250=1.14%, 500=0.42%, 750=0.10%
  cpu          : usr=1.21%, sys=3.16%, ctx=67738, majf=0, minf=36
  IO depths    : 1=0.4%, 2=1.0%, 4=2.6%, 8=51.7%, 16=44.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.6%, 8=1.9%, 16=2.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=90113,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=697: Tue Dec 19 07:26:46 2023
  read: IOPS=1364, BW=85.3MiB/s (89.4MB/s)(5120MiB/60039msec)
    slat (nsec): min=156, max=25678k, avg=2445.00, stdev=108235.17
    clat (usec): min=4, max=762631, avg=11378.85, stdev=37839.97
     lat (usec): min=52, max=762632, avg=11381.30, stdev=37840.45
    clat percentiles (usec):
     |  1.00th=[   545],  5.00th=[   799], 10.00th=[  1020], 20.00th=[  1598],
     | 30.00th=[  2540], 40.00th=[  3523], 50.00th=[  4490], 60.00th=[  5735],
     | 70.00th=[  7308], 80.00th=[ 10028], 90.00th=[ 17171], 95.00th=[ 30278],
     | 99.00th=[166724], 99.50th=[265290], 99.90th=[574620], 99.95th=[675283],
     | 99.99th=[759170]
   bw (  KiB/s): min= 2048, max=232448, per=6.21%, avg=89357.00, stdev=50530.01, samples=117
   iops        : min=   32, max= 3632, avg=1395.99, stdev=789.47, samples=117
  lat (usec)   : 10=0.01%, 20=0.01%, 100=0.01%, 250=0.01%, 500=0.54%
  lat (usec)   : 750=3.46%, 1000=5.43%
  lat (msec)   : 2=14.77%, 4=21.01%, 10=34.76%, 20=11.73%, 50=5.31%
  lat (msec)   : 100=1.18%, 250=1.19%, 500=0.45%, 750=0.12%, 1000=0.02%
  cpu          : usr=1.21%, sys=2.83%, ctx=59197, majf=0, minf=35
  IO depths    : 1=0.6%, 2=1.2%, 4=3.3%, 8=51.9%, 16=43.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.3%, 8=1.9%, 16=2.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=81921,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=698: Tue Dec 19 07:26:46 2023
  read: IOPS=1427, BW=89.2MiB/s (93.6MB/s)(5354MiB/60007msec)
    slat (nsec): min=151, max=43688k, avg=2536.77, stdev=172749.73
    clat (usec): min=20, max=766538, avg=10999.48, stdev=35380.27
     lat (usec): min=78, max=766539, avg=11002.02, stdev=35381.45
    clat percentiles (usec):
     |  1.00th=[   578],  5.00th=[   807], 10.00th=[   988], 20.00th=[  1352],
     | 30.00th=[  1876], 40.00th=[  2606], 50.00th=[  3621], 60.00th=[  4883],
     | 70.00th=[  6718], 80.00th=[  9634], 90.00th=[ 17433], 95.00th=[ 31327],
     | 99.00th=[193987], 99.50th=[274727], 99.90th=[429917], 99.95th=[517997],
     | 99.99th=[767558]
   bw (  KiB/s): min= 1788, max=286848, per=6.41%, avg=92271.97, stdev=62271.37, samples=118
   iops        : min=   27, max= 4482, avg=1441.55, stdev=973.07, samples=118
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.35%, 750=3.15%
  lat (usec)   : 1000=6.70%
  lat (msec)   : 2=21.30%, 4=21.80%, 10=27.53%, 20=10.68%, 50=5.42%
  lat (msec)   : 100=1.11%, 250=1.32%, 500=0.58%, 750=0.04%, 1000=0.02%
  cpu          : usr=1.24%, sys=3.06%, ctx=65434, majf=0, minf=36
  IO depths    : 1=0.4%, 2=0.9%, 4=2.3%, 8=51.6%, 16=44.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.6%, 8=2.1%, 16=2.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=85663,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=699: Tue Dec 19 07:26:46 2023
  read: IOPS=1431, BW=89.4MiB/s (93.8MB/s)(5376MiB/60108msec)
    slat (nsec): min=153, max=35373k, avg=2541.55, stdev=161791.41
    clat (usec): min=6, max=784217, avg=10894.50, stdev=37436.57
     lat (usec): min=60, max=784217, avg=10897.04, stdev=37437.23
    clat percentiles (usec):
     |  1.00th=[   537],  5.00th=[   758], 10.00th=[   971], 20.00th=[  1450],
     | 30.00th=[  2212], 40.00th=[  3097], 50.00th=[  4228], 60.00th=[  5538],
     | 70.00th=[  7111], 80.00th=[  9503], 90.00th=[ 16909], 95.00th=[ 29754],
     | 99.00th=[166724], 99.50th=[248513], 99.90th=[541066], 99.95th=[624952],
     | 99.99th=[775947]
   bw (  KiB/s): min= 1026, max=245120, per=6.41%, avg=92300.62, stdev=57201.19, samples=119
   iops        : min=   16, max= 3830, avg=1441.95, stdev=893.84, samples=119
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.46%, 750=4.32%, 1000=5.77%
  lat (msec)   : 2=17.19%, 4=20.38%, 10=33.31%, 20=10.47%, 50=5.27%
  lat (msec)   : 100=1.25%, 250=1.07%, 500=0.31%, 750=0.15%, 1000=0.03%
  cpu          : usr=1.16%, sys=3.06%, ctx=64490, majf=0, minf=36
  IO depths    : 1=0.5%, 2=1.0%, 4=2.6%, 8=51.5%, 16=44.5%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.6%, 8=1.8%, 16=2.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=86017,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=700: Tue Dec 19 07:26:46 2023
  read: IOPS=1470, BW=91.9MiB/s (96.4MB/s)(5515MiB/60012msec)
    slat (nsec): min=153, max=15291k, avg=2164.77, stdev=83424.18
    clat (usec): min=18, max=770643, avg=10636.39, stdev=36827.12
     lat (usec): min=93, max=770643, avg=10638.55, stdev=36827.71
    clat percentiles (usec):
     |  1.00th=[   578],  5.00th=[   791], 10.00th=[   930], 20.00th=[  1237],
     | 30.00th=[  1729], 40.00th=[  2573], 50.00th=[  3752], 60.00th=[  5080],
     | 70.00th=[  6718], 80.00th=[  9372], 90.00th=[ 16188], 95.00th=[ 27919],
     | 99.00th=[166724], 99.50th=[242222], 99.90th=[526386], 99.95th=[633340],
     | 99.99th=[759170]
   bw (  KiB/s): min=  127, max=370459, per=6.56%, avg=94451.41, stdev=61364.47, samples=119
   iops        : min=    1, max= 5788, avg=1475.60, stdev=958.84, samples=119
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.34%
  lat (usec)   : 750=3.64%, 1000=8.37%
  lat (msec)   : 2=21.01%, 4=18.81%, 10=29.33%, 20=10.64%, 50=4.99%
  lat (msec)   : 100=1.01%, 250=1.36%, 500=0.34%, 750=0.13%, 1000=0.02%
  cpu          : usr=1.18%, sys=3.12%, ctx=63536, majf=0, minf=32
  IO depths    : 1=0.4%, 2=0.9%, 4=2.5%, 8=51.7%, 16=44.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.7%, 8=1.9%, 16=2.4%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=88247,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=701: Tue Dec 19 07:26:46 2023
  read: IOPS=1371, BW=85.7MiB/s (89.9MB/s)(5142MiB/60005msec)
    slat (nsec): min=150, max=4844.7k, avg=1554.20, stdev=26089.26
    clat (usec): min=13, max=948105, avg=11486.98, stdev=38139.49
     lat (usec): min=241, max=948106, avg=11488.53, stdev=38139.59
    clat percentiles (usec):
     |  1.00th=[   611],  5.00th=[   807], 10.00th=[   988], 20.00th=[  1418],
     | 30.00th=[  2212], 40.00th=[  3261], 50.00th=[  4424], 60.00th=[  5800],
     | 70.00th=[  7570], 80.00th=[ 10421], 90.00th=[ 17433], 95.00th=[ 30540],
     | 99.00th=[185598], 99.50th=[265290], 99.90th=[517997], 99.95th=[591397],
     | 99.99th=[926942]
   bw (  KiB/s): min= 1777, max=318464, per=6.11%, avg=87876.52, stdev=55476.57, samples=119
   iops        : min=   27, max= 4976, avg=1372.85, stdev=866.82, samples=119
  lat (usec)   : 20=0.01%, 250=0.01%, 500=0.33%, 750=3.17%, 1000=6.81%
  lat (msec)   : 2=17.69%, 4=18.58%, 10=32.41%, 20=12.55%, 50=5.50%
  lat (msec)   : 100=1.06%, 250=1.32%, 500=0.46%, 750=0.09%, 1000=0.02%
  cpu          : usr=1.13%, sys=2.90%, ctx=63944, majf=0, minf=33
  IO depths    : 1=0.4%, 2=0.9%, 4=2.3%, 8=51.4%, 16=45.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.7%, 8=1.8%, 16=2.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=82269,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=702: Tue Dec 19 07:26:46 2023
  read: IOPS=1480, BW=92.5MiB/s (97.0MB/s)(5553MiB/60003msec)
    slat (nsec): min=152, max=7131.6k, avg=1732.74, stdev=38624.89
    clat (usec): min=20, max=904006, avg=10397.48, stdev=36319.83
     lat (usec): min=145, max=904007, avg=10399.21, stdev=36320.06
    clat percentiles (usec):
     |  1.00th=[   519],  5.00th=[   742], 10.00th=[   889], 20.00th=[  1188],
     | 30.00th=[  1614], 40.00th=[  2573], 50.00th=[  3720], 60.00th=[  5014],
     | 70.00th=[  6587], 80.00th=[  9110], 90.00th=[ 15533], 95.00th=[ 27132],
     | 99.00th=[164627], 99.50th=[261096], 99.90th=[513803], 99.95th=[574620],
     | 99.99th=[901776]
   bw (  KiB/s): min=  256, max=262016, per=6.60%, avg=95029.16, stdev=64056.78, samples=118
   iops        : min=    4, max= 4094, avg=1484.62, stdev=1000.96, samples=118
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.67%, 750=4.47%
  lat (usec)   : 1000=8.64%
  lat (msec)   : 2=20.64%, 4=18.03%, 10=29.99%, 20=10.31%, 50=4.37%
  lat (msec)   : 100=1.12%, 250=1.20%, 500=0.41%, 750=0.10%, 1000=0.02%
  cpu          : usr=1.26%, sys=3.04%, ctx=63810, majf=0, minf=33
  IO depths    : 1=0.3%, 2=0.7%, 4=2.1%, 8=51.6%, 16=45.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.5%, 8=1.9%, 16=2.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=88846,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=703: Tue Dec 19 07:26:46 2023
  read: IOPS=1356, BW=84.8MiB/s (88.9MB/s)(5087MiB/60009msec)
    slat (nsec): min=150, max=18078k, avg=1905.33, stdev=73084.59
    clat (usec): min=78, max=596022, avg=11611.57, stdev=36543.68
     lat (usec): min=119, max=596022, avg=11613.47, stdev=36544.86
    clat percentiles (usec):
     |  1.00th=[   578],  5.00th=[   775], 10.00th=[   938], 20.00th=[  1352],
     | 30.00th=[  2245], 40.00th=[  3261], 50.00th=[  4424], 60.00th=[  5866],
     | 70.00th=[  7701], 80.00th=[ 10552], 90.00th=[ 18482], 95.00th=[ 30802],
     | 99.00th=[191890], 99.50th=[283116], 99.90th=[463471], 99.95th=[541066],
     | 99.99th=[599786]
   bw (  KiB/s): min=  896, max=293760, per=6.03%, avg=86792.95, stdev=57121.67, samples=119
   iops        : min=   14, max= 4590, avg=1355.92, stdev=892.56, samples=119
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.33%, 750=4.06%, 1000=7.23%
  lat (msec)   : 2=16.23%, 4=18.95%, 10=31.34%, 20=12.90%, 50=5.90%
  lat (msec)   : 100=1.31%, 250=1.04%, 500=0.62%, 750=0.07%
  cpu          : usr=1.14%, sys=2.80%, ctx=55666, majf=0, minf=33
  IO depths    : 1=0.5%, 2=1.1%, 4=2.8%, 8=51.9%, 16=43.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.6%, 8=1.9%, 16=2.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=81395,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=704: Tue Dec 19 07:26:46 2023
  read: IOPS=1470, BW=91.9MiB/s (96.4MB/s)(5516MiB/60009msec)
    slat (nsec): min=154, max=11570k, avg=1869.48, stdev=55639.35
    clat (usec): min=37, max=552343, avg=10624.13, stdev=31637.42
     lat (usec): min=105, max=552343, avg=10626.00, stdev=31638.11
    clat percentiles (usec):
     |  1.00th=[   611],  5.00th=[   848], 10.00th=[  1074], 20.00th=[  1500],
     | 30.00th=[  2311], 40.00th=[  3326], 50.00th=[  4424], 60.00th=[  5735],
     | 70.00th=[  7373], 80.00th=[  9896], 90.00th=[ 16909], 95.00th=[ 28705],
     | 99.00th=[147850], 99.50th=[246416], 99.90th=[455082], 99.95th=[484443],
     | 99.99th=[549454]
   bw (  KiB/s): min= 2048, max=376320, per=6.56%, avg=94456.41, stdev=62258.08, samples=119
   iops        : min=   32, max= 5880, avg=1475.63, stdev=972.80, samples=119
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.26%, 750=2.56%
  lat (usec)   : 1000=5.71%
  lat (msec)   : 2=18.69%, 4=19.31%, 10=33.80%, 20=11.70%, 50=5.02%
  lat (msec)   : 100=1.22%, 250=1.24%, 500=0.45%, 750=0.03%
  cpu          : usr=1.27%, sys=3.05%, ctx=67292, majf=0, minf=34
  IO depths    : 1=0.4%, 2=1.0%, 4=2.6%, 8=51.6%, 16=44.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.5%, 8=1.9%, 16=2.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=88262,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=705: Tue Dec 19 07:26:46 2023
  read: IOPS=1489, BW=93.1MiB/s (97.6MB/s)(5588MiB/60014msec)
    slat (nsec): min=153, max=12418k, avg=1874.85, stdev=59628.38
    clat (usec): min=29, max=696104, avg=10521.02, stdev=33037.36
     lat (usec): min=117, max=696105, avg=10522.89, stdev=33037.57
    clat percentiles (usec):
     |  1.00th=[   545],  5.00th=[   742], 10.00th=[   930], 20.00th=[  1385],
     | 30.00th=[  2147], 40.00th=[  3130], 50.00th=[  4228], 60.00th=[  5473],
     | 70.00th=[  7177], 80.00th=[  9634], 90.00th=[ 17171], 95.00th=[ 29230],
     | 99.00th=[149947], 99.50th=[233833], 99.90th=[421528], 99.95th=[641729],
     | 99.99th=[692061]
   bw (  KiB/s): min=  127, max=274944, per=6.75%, avg=97145.69, stdev=58625.55, samples=117
   iops        : min=    1, max= 4296, avg=1517.71, stdev=916.06, samples=117
  lat (usec)   : 50=0.01%, 250=0.01%, 500=0.50%, 750=4.73%, 1000=6.32%
  lat (msec)   : 2=16.70%, 4=20.06%, 10=32.59%, 20=10.91%, 50=5.17%
  lat (msec)   : 100=1.38%, 250=1.18%, 500=0.36%, 750=0.08%
  cpu          : usr=1.27%, sys=3.09%, ctx=70436, majf=0, minf=34
  IO depths    : 1=0.4%, 2=0.9%, 4=2.3%, 8=51.3%, 16=45.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.6%, 8=1.7%, 16=2.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=89408,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=706: Tue Dec 19 07:26:46 2023
  read: IOPS=1362, BW=85.1MiB/s (89.3MB/s)(5109MiB/60002msec)
    slat (nsec): min=151, max=16851k, avg=2326.96, stdev=97087.05
    clat (usec): min=19, max=646255, avg=11457.65, stdev=35026.94
     lat (usec): min=104, max=646257, avg=11459.98, stdev=35027.87
    clat percentiles (usec):
     |  1.00th=[   578],  5.00th=[   766], 10.00th=[   906], 20.00th=[  1237],
     | 30.00th=[  1975], 40.00th=[  3163], 50.00th=[  4490], 60.00th=[  5866],
     | 70.00th=[  7701], 80.00th=[ 10814], 90.00th=[ 19268], 95.00th=[ 31327],
     | 99.00th=[175113], 99.50th=[278922], 99.90th=[471860], 99.95th=[557843],
     | 99.99th=[641729]
   bw (  KiB/s): min= 1532, max=250624, per=6.07%, avg=87369.12, stdev=54279.71, samples=119
   iops        : min=   23, max= 3916, avg=1364.96, stdev=848.15, samples=119
  lat (usec)   : 20=0.01%, 250=0.01%, 500=0.38%, 750=4.15%, 1000=8.25%
  lat (msec)   : 2=17.40%, 4=16.23%, 10=31.57%, 20=12.65%, 50=6.17%
  lat (msec)   : 100=1.39%, 250=1.17%, 500=0.55%, 750=0.08%
  cpu          : usr=1.09%, sys=2.77%, ctx=54807, majf=0, minf=36
  IO depths    : 1=0.6%, 2=1.3%, 4=3.3%, 8=52.1%, 16=42.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.6%, 8=1.9%, 16=2.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=81740,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=707: Tue Dec 19 07:26:46 2023
  read: IOPS=1312, BW=82.0MiB/s (86.0MB/s)(4926MiB/60048msec)
    slat (nsec): min=154, max=21828k, avg=2320.93, stdev=98882.18
    clat (usec): min=4, max=1039.6k, avg=11858.39, stdev=40119.68
     lat (usec): min=57, max=1039.6k, avg=11860.71, stdev=40121.40
    clat percentiles (usec):
     |  1.00th=[    586],  5.00th=[    775], 10.00th=[    963],
     | 20.00th=[   1450], 30.00th=[   2212], 40.00th=[   3359],
     | 50.00th=[   4490], 60.00th=[   5735], 70.00th=[   7308],
     | 80.00th=[  10028], 90.00th=[  17695], 95.00th=[  31327],
     | 99.00th=[ 196084], 99.50th=[ 283116], 99.90th=[ 530580],
     | 99.95th=[ 549454], 99.99th=[1035994]
   bw (  KiB/s): min=  256, max=259200, per=5.91%, avg=85081.88, stdev=55845.31, samples=118
   iops        : min=    4, max= 4050, avg=1329.18, stdev=872.59, samples=118
  lat (usec)   : 10=0.01%, 20=0.01%, 100=0.01%, 250=0.03%, 500=0.34%
  lat (usec)   : 750=4.09%, 1000=6.34%
  lat (msec)   : 2=17.12%, 4=17.82%, 10=34.12%, 20=11.56%, 50=5.45%
  lat (msec)   : 100=1.10%, 250=1.40%, 500=0.46%, 750=0.14%, 2000=0.02%
  cpu          : usr=1.09%, sys=2.69%, ctx=58139, majf=0, minf=37
  IO depths    : 1=0.5%, 2=1.2%, 4=3.2%, 8=51.5%, 16=43.6%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.6%, 8=1.9%, 16=2.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=78820,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=708: Tue Dec 19 07:26:46 2023
  read: IOPS=1449, BW=90.6MiB/s (95.0MB/s)(5438MiB/60025msec)
    slat (nsec): min=150, max=19678k, avg=2599.69, stdev=113433.79
    clat (usec): min=4, max=989160, avg=10811.02, stdev=35774.26
     lat (usec): min=57, max=989161, avg=10813.62, stdev=35775.93
    clat percentiles (usec):
     |  1.00th=[   594],  5.00th=[   791], 10.00th=[   971], 20.00th=[  1450],
     | 30.00th=[  2212], 40.00th=[  3261], 50.00th=[  4228], 60.00th=[  5407],
     | 70.00th=[  7111], 80.00th=[  9503], 90.00th=[ 16450], 95.00th=[ 28181],
     | 99.00th=[162530], 99.50th=[261096], 99.90th=[497026], 99.95th=[517997],
     | 99.99th=[868221]
   bw (  KiB/s): min=  896, max=348160, per=6.48%, avg=93293.51, stdev=59844.47, samples=119
   iops        : min=   14, max= 5440, avg=1457.51, stdev=935.09, samples=119
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.03%
  lat (usec)   : 500=0.29%, 750=3.46%, 1000=6.93%
  lat (msec)   : 2=16.89%, 4=20.67%, 10=33.11%, 20=11.00%, 50=4.79%
  lat (msec)   : 100=1.06%, 250=1.22%, 500=0.49%, 750=0.06%, 1000=0.02%
  cpu          : usr=1.30%, sys=3.02%, ctx=65401, majf=0, minf=36
  IO depths    : 1=0.4%, 2=1.0%, 4=2.7%, 8=51.5%, 16=44.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.6%, 8=1.6%, 16=2.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=87009,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=709: Tue Dec 19 07:26:46 2023
  read: IOPS=1341, BW=83.9MiB/s (87.9MB/s)(5033MiB/60021msec)
    slat (nsec): min=151, max=10950k, avg=2367.74, stdev=73893.20
    clat (usec): min=11, max=655030, avg=11666.88, stdev=38905.50
     lat (usec): min=88, max=655032, avg=11669.24, stdev=38905.56
    clat percentiles (usec):
     |  1.00th=[   537],  5.00th=[   742], 10.00th=[   898], 20.00th=[  1254],
     | 30.00th=[  1811], 40.00th=[  2966], 50.00th=[  4359], 60.00th=[  5932],
     | 70.00th=[  7701], 80.00th=[ 10683], 90.00th=[ 17695], 95.00th=[ 31327],
     | 99.00th=[185598], 99.50th=[267387], 99.90th=[557843], 99.95th=[624952],
     | 99.99th=[658506]
   bw (  KiB/s): min= 1920, max=245760, per=6.13%, avg=88206.59, stdev=53362.28, samples=116
   iops        : min=   30, max= 3840, avg=1378.00, stdev=833.79, samples=116
  lat (usec)   : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.57%
  lat (usec)   : 750=4.80%, 1000=7.89%
  lat (msec)   : 2=18.68%, 4=15.46%, 10=30.84%, 20=13.33%, 50=5.42%
  lat (msec)   : 100=1.10%, 250=1.31%, 500=0.41%, 750=0.19%
  cpu          : usr=1.17%, sys=2.64%, ctx=55795, majf=0, minf=32
  IO depths    : 1=0.4%, 2=1.0%, 4=2.6%, 8=51.7%, 16=44.3%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=1.8%, 16=2.4%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=80527,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1405MiB/s (1474MB/s), 82.0MiB/s-93.8MiB/s (86.0MB/s-98.3MB/s), io=82.5GiB (88.6GB), run=60002-60108msec

Disk stats (read/write):
    dm-7: ios=27120/84, merge=0/0, ticks=2965650/5555, in_queue=2971205, util=80.38%, aggrios=197622/89, aggrmerge=0/0, aggrticks=22506403/5688, aggrin_queue=22512091, aggrutil=93.20%
    dm-6: ios=197622/89, merge=0/0, ticks=22506403/5688, in_queue=22512091, util=93.20%, aggrios=197622/89, aggrmerge=0/0, aggrticks=22496998/5686, aggrin_queue=22502684, aggrutil=93.18%
    dm-4: ios=197622/89, merge=0/0, ticks=22496998/5686, in_queue=22502684, util=93.18%, aggrios=49405/27, aggrmerge=0/0, aggrticks=5623633/1480, aggrin_queue=5625114, aggrutil=93.17%
  dm-1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
  dm-2: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
    dm-0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    drbd1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=197622/109, aggrmerge=0/0, aggrticks=22490873/5922, aggrin_queue=22496795, aggrutil=93.17%
    md1: ios=197622/109, merge=0/0, ticks=22490873/5922, in_queue=22496795, util=93.17%, aggrios=47820/106, aggrmerge=1584/32, aggrticks=5471567/4885, aggrin_queue=5479336, aggrutil=92.57%
  sde: ios=50594/105, merge=2182/20, ticks=5464144/4748, in_queue=5471535, util=91.94%
  sdc: ios=47312/105, merge=1526/20, ticks=5592991/5103, in_queue=5600881, util=92.43%
  sdb: ios=48060/107, merge=1437/45, ticks=5371123/4702, in_queue=5378777, util=91.83%
  sda: ios=45317/107, merge=1194/45, ticks=5458010/4989, in_queue=5466154, util=92.57%
  dm-3: ios=197622/109, merge=0/0, ticks=22494534/5923, in_queue=22500457, util=93.17%