random-write: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=posixaio, iodepth=16 ... fio-3.28 Starting 16 processes random-write: (groupid=0, jobs=1): err= 0: pid=760: Mon Dec 18 08:42:08 2023 write: IOPS=662, BW=41.4MiB/s (43.4MB/s)(2494MiB/60221msec); 0 zone resets slat (nsec): min=595, max=3403.2k, avg=8315.93, stdev=30836.99 clat (usec): min=1059, max=162211, avg=23954.76, stdev=10010.71 lat (usec): min=1064, max=162220, avg=23963.08, stdev=10010.67 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 15], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 35], 95.00th=[ 41], | 99.00th=[ 55], 99.50th=[ 64], 99.90th=[ 128], 99.95th=[ 140], | 99.99th=[ 153] bw ( KiB/s): min=20608, max=112384, per=6.35%, avg=42582.08, stdev=11447.14, samples=119 iops : min= 322, max= 1756, avg=665.24, stdev=178.85, samples=119 lat (msec) : 2=0.22%, 4=0.41%, 10=1.04%, 20=34.79%, 50=61.81% lat (msec) : 100=1.43%, 250=0.30% cpu : usr=1.23%, sys=1.85%, ctx=28097, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.2%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=96.0%, 8=3.2%, 16=0.8%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,39907,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=761: Mon Dec 18 08:42:08 2023 write: IOPS=663, BW=41.5MiB/s (43.5MB/s)(2498MiB/60222msec); 0 zone resets slat (nsec): min=663, max=1900.5k, avg=8324.35, stdev=28527.55 clat (usec): min=1305, max=168495, avg=23898.64, stdev=9961.75 lat (usec): min=1316, max=168504, avg=23906.96, stdev=9961.49 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 35], 95.00th=[ 41], | 99.00th=[ 54], 99.50th=[ 62], 99.90th=[ 136], 99.95th=[ 144], | 99.99th=[ 167] bw ( KiB/s): min=22656, max=103680, per=6.36%, avg=42645.45, stdev=11012.41, samples=119 iops : min= 354, max= 1620, avg=666.22, stdev=172.07, samples=119 lat (msec) : 2=0.04%, 4=0.35%, 10=1.08%, 20=35.28%, 50=61.73% lat (msec) : 100=1.24%, 250=0.28% cpu : usr=1.19%, sys=1.91%, ctx=28390, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.7%, 16=49.1%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=4.0%, 16=0.5%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,39967,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=762: Mon Dec 18 08:42:08 2023 write: IOPS=661, BW=41.3MiB/s (43.3MB/s)(2488MiB/60207msec); 0 zone resets slat (nsec): min=624, max=3960.4k, avg=8463.59, stdev=40292.47 clat (usec): min=1293, max=176968, avg=23961.72, stdev=9729.91 lat (usec): min=1312, max=176975, avg=23970.18, stdev=9729.58 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 15], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 35], 95.00th=[ 41], | 99.00th=[ 55], 99.50th=[ 62], 99.90th=[ 124], 99.95th=[ 157], | 99.99th=[ 174] bw ( KiB/s): min=23424, max=86016, per=6.33%, avg=42473.32, stdev=10268.35, samples=119 iops : min= 366, max= 1344, avg=663.53, stdev=160.44, samples=119 lat (msec) : 2=0.01%, 4=0.11%, 10=1.03%, 20=35.78%, 50=61.45% lat (msec) : 100=1.43%, 250=0.19% cpu : usr=1.19%, sys=1.95%, ctx=28257, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.7%, 16=49.1%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.9%, 8=2.2%, 16=1.9%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,39809,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=763: Mon Dec 18 08:42:08 2023 write: IOPS=655, BW=40.9MiB/s (42.9MB/s)(2466MiB/60220msec); 0 zone resets slat (nsec): min=654, max=7287.6k, avg=8336.20, stdev=42824.48 clat (usec): min=1312, max=227695, avg=24189.62, stdev=10611.30 lat (usec): min=1331, max=227702, avg=24197.95, stdev=10611.01 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 35], 95.00th=[ 41], | 99.00th=[ 54], 99.50th=[ 66], 99.90th=[ 146], 99.95th=[ 157], | 99.99th=[ 228] bw ( KiB/s): min=19584, max=62984, per=6.27%, avg=42081.31, stdev=9710.87, samples=119 iops : min= 306, max= 984, avg=657.42, stdev=151.71, samples=119 lat (msec) : 2=0.16%, 4=0.03%, 10=0.23%, 20=35.19%, 50=62.92% lat (msec) : 100=1.15%, 250=0.32% cpu : usr=1.15%, sys=1.90%, ctx=27699, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.2%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=96.1%, 8=2.9%, 16=0.9%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,39452,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=764: Mon Dec 18 08:42:08 2023 write: IOPS=653, BW=40.8MiB/s (42.8MB/s)(2457MiB/60188msec); 0 zone resets slat (nsec): min=1861, max=4117.6k, avg=8235.31, stdev=31177.92 clat (usec): min=1131, max=342840, avg=24182.85, stdev=10680.08 lat (usec): min=1146, max=342846, avg=24191.09, stdev=10679.93 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 34], 95.00th=[ 41], | 99.00th=[ 53], 99.50th=[ 61], 99.90th=[ 129], 99.95th=[ 205], | 99.99th=[ 338] bw ( KiB/s): min=15360, max=61312, per=6.25%, avg=41930.32, stdev=9999.85, samples=119 iops : min= 240, max= 958, avg=655.11, stdev=156.25, samples=119 lat (msec) : 2=0.01%, 4=0.01%, 10=0.08%, 20=35.51%, 50=62.91% lat (msec) : 100=1.25%, 250=0.22%, 500=0.02% cpu : usr=1.21%, sys=1.72%, ctx=26748, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.7%, 16=49.1%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.7%, 8=3.5%, 16=0.8%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,39308,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=765: Mon Dec 18 08:42:08 2023 write: IOPS=654, BW=40.9MiB/s (42.9MB/s)(2461MiB/60160msec); 0 zone resets slat (nsec): min=1870, max=2610.0k, avg=8474.62, stdev=29683.15 clat (usec): min=1263, max=339809, avg=24151.19, stdev=10907.68 lat (usec): min=1293, max=339815, avg=24159.67, stdev=10907.22 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 35], 95.00th=[ 40], | 99.00th=[ 53], 99.50th=[ 62], 99.90th=[ 161], 99.95th=[ 171], | 99.99th=[ 334] bw ( KiB/s): min=17664, max=63744, per=6.26%, avg=42007.34, stdev=10179.43, samples=119 iops : min= 276, max= 996, avg=656.29, stdev=159.06, samples=119 lat (msec) : 2=0.02%, 4=0.08%, 10=0.03%, 20=35.63%, 50=62.87% lat (msec) : 100=1.14%, 250=0.21%, 500=0.03% cpu : usr=1.22%, sys=1.80%, ctx=27003, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.7%, 16=49.1%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.9%, 8=2.9%, 16=1.2%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,39372,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=766: Mon Dec 18 08:42:08 2023 write: IOPS=656, BW=41.1MiB/s (43.1MB/s)(2470MiB/60150msec); 0 zone resets slat (nsec): min=1798, max=4229.4k, avg=8610.57, stdev=42563.30 clat (usec): min=818, max=223608, avg=24155.38, stdev=9906.44 lat (usec): min=834, max=223618, avg=24163.99, stdev=9906.06 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 35], 95.00th=[ 41], | 99.00th=[ 55], 99.50th=[ 65], 99.90th=[ 117], 99.95th=[ 142], | 99.99th=[ 218] bw ( KiB/s): min=23296, max=62208, per=6.29%, avg=42166.86, stdev=9651.57, samples=119 iops : min= 364, max= 972, avg=658.76, stdev=150.86, samples=119 lat (usec) : 1000=0.01% lat (msec) : 2=0.08%, 10=0.11%, 20=35.83%, 50=62.33%, 100=1.40% lat (msec) : 250=0.25% cpu : usr=1.22%, sys=1.79%, ctx=27252, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.1%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.9%, 8=3.1%, 16=1.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,39517,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=767: Mon Dec 18 08:42:08 2023 write: IOPS=656, BW=41.0MiB/s (43.0MB/s)(2466MiB/60136msec); 0 zone resets slat (nsec): min=1854, max=1351.3k, avg=8157.93, stdev=20602.28 clat (usec): min=1903, max=209824, avg=24139.28, stdev=9864.22 lat (usec): min=1918, max=209834, avg=24147.44, stdev=9863.85 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 35], 95.00th=[ 41], | 99.00th=[ 53], 99.50th=[ 63], 99.90th=[ 142], 99.95th=[ 153], | 99.99th=[ 199] bw ( KiB/s): min=24192, max=62208, per=6.28%, avg=42106.38, stdev=9565.84, samples=119 iops : min= 378, max= 972, avg=657.83, stdev=149.46, samples=119 lat (msec) : 2=0.01%, 4=0.01%, 10=0.14%, 20=35.72%, 50=62.80% lat (msec) : 100=1.07%, 250=0.25% cpu : usr=1.17%, sys=1.91%, ctx=28080, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.1%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.3%, 8=4.2%, 16=0.5%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,39459,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=768: Mon Dec 18 08:42:08 2023 write: IOPS=656, BW=41.0MiB/s (43.0MB/s)(2468MiB/60126msec); 0 zone resets slat (nsec): min=1693, max=1945.0k, avg=8254.77, stdev=27117.79 clat (usec): min=1146, max=191630, avg=24144.40, stdev=9550.07 lat (usec): min=1160, max=191637, avg=24152.65, stdev=9550.64 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 35], 95.00th=[ 41], | 99.00th=[ 53], 99.50th=[ 60], 99.90th=[ 125], 99.95th=[ 155], | 99.99th=[ 188] bw ( KiB/s): min=23552, max=61952, per=6.28%, avg=42141.94, stdev=9845.83, samples=119 iops : min= 368, max= 968, avg=658.38, stdev=153.85, samples=119 lat (msec) : 2=0.01%, 4=0.01%, 10=0.03%, 20=35.63%, 50=63.04% lat (msec) : 100=1.04%, 250=0.25% cpu : usr=1.19%, sys=1.76%, ctx=26459, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.3%, 8=50.6%, 16=49.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.7%, 8=2.8%, 16=1.5%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,39491,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=769: Mon Dec 18 08:42:08 2023 write: IOPS=653, BW=40.9MiB/s (42.9MB/s)(2457MiB/60121msec); 0 zone resets slat (nsec): min=1647, max=1600.3k, avg=8216.04, stdev=23961.90 clat (usec): min=1107, max=221585, avg=24189.81, stdev=10470.72 lat (usec): min=1126, max=221590, avg=24198.02, stdev=10470.62 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 34], 95.00th=[ 40], | 99.00th=[ 53], 99.50th=[ 57], 99.90th=[ 163], 99.95th=[ 203], | 99.99th=[ 220] bw ( KiB/s): min=19968, max=62464, per=6.25%, avg=41950.53, stdev=10137.04, samples=119 iops : min= 312, max= 976, avg=655.39, stdev=158.40, samples=119 lat (msec) : 2=0.01%, 4=0.01%, 10=0.05%, 20=35.67%, 50=62.97% lat (msec) : 100=1.04%, 250=0.26% cpu : usr=1.19%, sys=1.70%, ctx=26277, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.2%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=3.5%, 16=1.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,39312,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=770: Mon Dec 18 08:42:08 2023 write: IOPS=653, BW=40.8MiB/s (42.8MB/s)(2455MiB/60115msec); 0 zone resets slat (nsec): min=1885, max=3495.8k, avg=8672.21, stdev=38132.56 clat (usec): min=839, max=215484, avg=24260.77, stdev=10135.29 lat (usec): min=859, max=215492, avg=24269.44, stdev=10135.01 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 35], 95.00th=[ 41], | 99.00th=[ 54], 99.50th=[ 64], 99.90th=[ 153], 99.95th=[ 169], | 99.99th=[ 213] bw ( KiB/s): min=18522, max=63872, per=6.25%, avg=41911.34, stdev=10027.19, samples=119 iops : min= 289, max= 998, avg=654.78, stdev=156.69, samples=119 lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.04%, 20=34.95%, 50=63.46% lat (msec) : 100=1.31%, 250=0.22% cpu : usr=1.19%, sys=1.71%, ctx=25562, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.1%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=96.1%, 8=2.9%, 16=1.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,39280,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=771: Mon Dec 18 08:42:08 2023 write: IOPS=652, BW=40.8MiB/s (42.8MB/s)(2452MiB/60103msec); 0 zone resets slat (nsec): min=1798, max=3309.4k, avg=8117.01, stdev=26880.54 clat (usec): min=1301, max=328338, avg=24242.36, stdev=11412.83 lat (usec): min=1319, max=328345, avg=24250.48, stdev=11412.56 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 34], 95.00th=[ 40], | 99.00th=[ 53], 99.50th=[ 59], 99.90th=[ 174], 99.95th=[ 209], | 99.99th=[ 317] bw ( KiB/s): min=16351, max=63616, per=6.24%, avg=41858.55, stdev=10106.05, samples=119 iops : min= 255, max= 994, avg=653.98, stdev=157.92, samples=119 lat (msec) : 2=0.01%, 4=0.01%, 10=0.06%, 20=35.31%, 50=63.20% lat (msec) : 100=1.11%, 250=0.26%, 500=0.04% cpu : usr=1.25%, sys=1.70%, ctx=26161, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.2%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.7%, 8=3.5%, 16=0.8%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,39233,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=772: Mon Dec 18 08:42:08 2023 write: IOPS=652, BW=40.8MiB/s (42.8MB/s)(2451MiB/60084msec); 0 zone resets slat (nsec): min=798, max=3518.5k, avg=8377.21, stdev=29938.48 clat (usec): min=1018, max=206605, avg=24299.08, stdev=10927.47 lat (usec): min=1021, max=206672, avg=24307.46, stdev=10927.22 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 35], 95.00th=[ 41], | 99.00th=[ 54], 99.50th=[ 63], 99.90th=[ 184], 99.95th=[ 197], | 99.99th=[ 205] bw ( KiB/s): min=18395, max=64000, per=6.24%, avg=41836.06, stdev=10141.91, samples=119 iops : min= 287, max= 1000, avg=653.62, stdev=158.46, samples=119 lat (msec) : 2=0.05%, 4=0.03%, 10=0.07%, 20=35.68%, 50=62.65% lat (msec) : 100=1.23%, 250=0.30% cpu : usr=1.15%, sys=1.81%, ctx=26844, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.2%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.8%, 8=3.2%, 16=1.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,39221,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=773: Mon Dec 18 08:42:08 2023 write: IOPS=655, BW=41.0MiB/s (43.0MB/s)(2464MiB/60108msec); 0 zone resets slat (nsec): min=1107, max=2256.1k, avg=8384.01, stdev=29080.57 clat (usec): min=1213, max=239235, avg=24147.75, stdev=10866.01 lat (usec): min=1229, max=239242, avg=24156.13, stdev=10865.79 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 34], 95.00th=[ 40], | 99.00th=[ 53], 99.50th=[ 62], 99.90th=[ 169], 99.95th=[ 201], | 99.99th=[ 234] bw ( KiB/s): min=21760, max=63360, per=6.27%, avg=42057.97, stdev=10125.58, samples=119 iops : min= 340, max= 990, avg=657.12, stdev=158.21, samples=119 lat (msec) : 2=0.04%, 4=0.03%, 10=0.03%, 20=36.13%, 50=62.50% lat (msec) : 100=1.00%, 250=0.27% cpu : usr=1.19%, sys=1.76%, ctx=26867, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.7%, 16=49.1%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.8%, 8=3.7%, 16=0.5%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,39423,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=774: Mon Dec 18 08:42:08 2023 write: IOPS=653, BW=40.9MiB/s (42.8MB/s)(2456MiB/60098msec); 0 zone resets slat (nsec): min=1188, max=4049.7k, avg=8376.73, stdev=33557.16 clat (usec): min=898, max=267453, avg=24241.54, stdev=11828.10 lat (usec): min=903, max=267457, avg=24249.92, stdev=11827.83 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 35], 95.00th=[ 41], | 99.00th=[ 53], 99.50th=[ 60], 99.90th=[ 226], 99.95th=[ 251], | 99.99th=[ 266] bw ( KiB/s): min=16640, max=64510, per=6.24%, avg=41849.65, stdev=10131.96, samples=119 iops : min= 260, max= 1007, avg=653.82, stdev=158.30, samples=119 lat (usec) : 1000=0.02% lat (msec) : 2=0.07%, 4=0.06%, 10=0.13%, 20=35.19%, 50=63.18% lat (msec) : 100=1.05%, 250=0.25%, 500=0.05% cpu : usr=1.21%, sys=1.74%, ctx=26447, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.7%, 16=49.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.9%, 8=2.8%, 16=1.3%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,39295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=775: Mon Dec 18 08:42:08 2023 write: IOPS=651, BW=40.7MiB/s (42.7MB/s)(2448MiB/60088msec); 0 zone resets slat (nsec): min=1905, max=3148.2k, avg=8372.34, stdev=29397.57 clat (usec): min=1417, max=229597, avg=24334.39, stdev=10432.67 lat (usec): min=1435, max=229605, avg=24342.76, stdev=10432.51 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 35], 95.00th=[ 41], | 99.00th=[ 55], 99.50th=[ 63], 99.90th=[ 146], 99.95th=[ 157], | 99.99th=[ 215] bw ( KiB/s): min=22272, max=62080, per=6.23%, avg=41804.96, stdev=9999.28, samples=119 iops : min= 348, max= 970, avg=653.14, stdev=156.24, samples=119 lat (msec) : 2=0.01%, 4=0.01%, 10=0.03%, 20=35.42%, 50=62.84% lat (msec) : 100=1.39%, 250=0.31% cpu : usr=1.15%, sys=1.84%, ctx=26872, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.1%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.7%, 8=2.9%, 16=1.4%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,39167,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=655MiB/s (687MB/s), 40.7MiB/s-41.5MiB/s (42.7MB/s-43.5MB/s), io=38.5GiB (41.4GB), run=60084-60222msec Disk stats (read/write): dm-7: ios=0/559432, merge=0/0, ticks=0/24104399, in_queue=24104399, util=68.43%, aggrios=0/568957, aggrmerge=0/0, aggrticks=0/24565349, aggrin_queue=24565349, aggrutil=68.48% dm-6: ios=0/568957, merge=0/0, ticks=0/24565349, in_queue=24565349, util=68.48%, aggrios=0/568957, aggrmerge=0/0, aggrticks=0/24561729, aggrin_queue=24561729, aggrutil=68.48% dm-4: ios=0/568957, merge=0/0, ticks=0/24561729, in_queue=24561729, util=68.48%, aggrios=0/142249, aggrmerge=0/0, aggrticks=0/6140160, aggrin_queue=6140160, aggrutil=68.47% dm-1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% dm-2: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% dm-0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00% drbd1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/568997, aggrmerge=0/0, aggrticks=0/24559163, aggrin_queue=24559163, aggrutil=68.47% md1: ios=0/568997, merge=0/0, ticks=0/24559163, in_queue=24559163, util=68.47%, aggrios=0/188479, aggrmerge=0/1336, aggrticks=0/8131576, aggrin_queue=8132280, aggrutil=67.57% sdc: ios=0/188601, merge=0/1265, ticks=0/7467246, in_queue=7467946, util=62.62% sdb: ios=0/188377, merge=0/1322, ticks=0/8184095, in_queue=8184682, util=65.43% sda: ios=0/188459, merge=0/1423, ticks=0/8743389, in_queue=8744213, util=67.57% dm-3: ios=0/568997, merge=0/0, ticks=0/24560641, in_queue=24560641, util=68.47%