random-write: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=posixaio, iodepth=16 ... fio-3.28 Starting 16 processes random-write: (groupid=0, jobs=1): err= 0: pid=740: Tue Dec 19 06:32:06 2023 write: IOPS=202, BW=12.7MiB/s (13.3MB/s)(768MiB/60538msec); 0 zone resets slat (nsec): min=1028, max=3062.6k, avg=8353.62, stdev=36308.43 clat (usec): min=1098, max=501879, avg=78166.56, stdev=30897.07 lat (usec): min=1102, max=501883, avg=78174.91, stdev=30898.23 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 33], 10.00th=[ 45], 20.00th=[ 58], | 30.00th=[ 65], 40.00th=[ 72], 50.00th=[ 78], 60.00th=[ 84], | 70.00th=[ 90], 80.00th=[ 97], 90.00th=[ 110], 95.00th=[ 124], | 99.00th=[ 159], 99.50th=[ 184], 99.90th=[ 279], 99.95th=[ 464], | 99.99th=[ 498] bw ( KiB/s): min= 5482, max=32574, per=6.40%, avg=13065.75, stdev=3032.15, samples=120 iops : min= 85, max= 508, avg=203.91, stdev=47.38, samples=120 lat (msec) : 2=0.77%, 4=0.12%, 10=0.47%, 20=1.32%, 50=10.08% lat (msec) : 100=70.09%, 250=17.01%, 500=0.13%, 750=0.01% cpu : usr=0.38%, sys=0.72%, ctx=10113, majf=0, minf=35 IO depths : 1=0.1%, 2=0.1%, 4=0.3%, 8=50.7%, 16=48.9%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.8%, 8=2.2%, 16=2.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,12289,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=741: Tue Dec 19 06:32:06 2023 write: IOPS=202, BW=12.6MiB/s (13.2MB/s)(765MiB/60595msec); 0 zone resets slat (nsec): min=1379, max=2562.0k, avg=8757.41, stdev=35680.80 clat (usec): min=721, max=278991, avg=78422.22, stdev=28172.45 lat (usec): min=726, max=278998, avg=78430.97, stdev=28172.61 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 35], 10.00th=[ 47], 20.00th=[ 58], | 30.00th=[ 65], 40.00th=[ 72], 50.00th=[ 78], 60.00th=[ 84], | 70.00th=[ 90], 80.00th=[ 99], 90.00th=[ 111], 95.00th=[ 125], | 99.00th=[ 159], 99.50th=[ 169], 99.90th=[ 271], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 7808, max=38323, per=6.38%, avg=13030.45, stdev=3174.96, samples=119 iops : min= 122, max= 598, avg=203.37, stdev=49.59, samples=119 lat (usec) : 750=0.04%, 1000=0.02% lat (msec) : 2=0.07%, 4=0.05%, 10=0.23%, 20=1.69%, 50=10.05% lat (msec) : 100=69.95%, 250=17.75%, 500=0.15% cpu : usr=0.39%, sys=0.50%, ctx=8923, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.7%, 16=49.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.9%, 8=3.1%, 16=1.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,12241,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=742: Tue Dec 19 06:32:06 2023 write: IOPS=199, BW=12.5MiB/s (13.1MB/s)(762MiB/61115msec); 0 zone resets slat (nsec): min=1170, max=2116.6k, avg=8842.74, stdev=34789.31 clat (usec): min=681, max=598790, avg=78085.95, stdev=31026.84 lat (usec): min=687, max=598795, avg=78094.80, stdev=31027.29 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 34], 10.00th=[ 45], 20.00th=[ 58], | 30.00th=[ 65], 40.00th=[ 71], 50.00th=[ 78], 60.00th=[ 84], | 70.00th=[ 91], 80.00th=[ 99], 90.00th=[ 109], 95.00th=[ 122], | 99.00th=[ 150], 99.50th=[ 167], 99.90th=[ 401], 99.95th=[ 584], | 99.99th=[ 600] bw ( KiB/s): min= 2682, max=22573, per=6.36%, avg=12988.50, stdev=2522.84, samples=119 iops : min= 41, max= 352, avg=202.71, stdev=39.46, samples=119 lat (usec) : 750=0.03% lat (msec) : 2=0.21%, 4=0.16%, 10=0.53%, 20=1.15%, 50=11.02% lat (msec) : 100=69.25%, 250=17.46%, 500=0.12%, 750=0.06% cpu : usr=0.38%, sys=0.67%, ctx=9367, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.3%, 8=51.0%, 16=48.6%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=96.8%, 8=1.5%, 16=1.6%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,12199,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=743: Tue Dec 19 06:32:06 2023 write: IOPS=197, BW=12.3MiB/s (12.9MB/s)(754MiB/61114msec); 0 zone resets slat (nsec): min=1472, max=5436.4k, avg=8721.78, stdev=55207.05 clat (msec): min=2, max=426, avg=79.13, stdev=28.65 lat (msec): min=2, max=426, avg=79.14, stdev=28.65 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 37], 10.00th=[ 48], 20.00th=[ 58], | 30.00th=[ 66], 40.00th=[ 72], 50.00th=[ 79], 60.00th=[ 85], | 70.00th=[ 91], 80.00th=[ 99], 90.00th=[ 110], 95.00th=[ 122], | 99.00th=[ 155], 99.50th=[ 169], 99.90th=[ 338], 99.95th=[ 405], | 99.99th=[ 426] bw ( KiB/s): min= 5075, max=22573, per=6.30%, avg=12850.47, stdev=2304.43, samples=119 iops : min= 79, max= 352, avg=200.55, stdev=36.00, samples=119 lat (msec) : 4=0.08%, 10=0.03%, 20=0.91%, 50=10.85%, 100=70.76% lat (msec) : 250=17.17%, 500=0.19% cpu : usr=0.36%, sys=0.60%, ctx=8744, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=51.1%, 16=48.6%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=94.8%, 8=4.9%, 16=0.4%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,12070,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=744: Tue Dec 19 06:32:06 2023 write: IOPS=200, BW=12.6MiB/s (13.2MB/s)(767MiB/61105msec); 0 zone resets slat (nsec): min=630, max=5285.0k, avg=8435.75, stdev=49617.12 clat (usec): min=940, max=217236, avg=78150.40, stdev=27986.13 lat (usec): min=944, max=217244, avg=78158.84, stdev=27986.06 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 33], 10.00th=[ 45], 20.00th=[ 58], | 30.00th=[ 65], 40.00th=[ 72], 50.00th=[ 78], 60.00th=[ 85], | 70.00th=[ 91], 80.00th=[ 99], 90.00th=[ 111], 95.00th=[ 123], | 99.00th=[ 157], 99.50th=[ 174], 99.90th=[ 207], 99.95th=[ 209], | 99.99th=[ 218] bw ( KiB/s): min= 7296, max=50944, per=6.40%, avg=13073.55, stdev=4020.53, samples=119 iops : min= 114, max= 796, avg=204.07, stdev=62.85, samples=119 lat (usec) : 1000=0.03% lat (msec) : 2=0.36%, 4=0.29%, 10=0.50%, 20=1.30%, 50=10.38% lat (msec) : 100=68.48%, 250=18.67% cpu : usr=0.34%, sys=0.57%, ctx=9673, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.1%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.9%, 8=3.9%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,12276,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=745: Tue Dec 19 06:32:06 2023 write: IOPS=196, BW=12.3MiB/s (12.9MB/s)(750MiB/61110msec); 0 zone resets slat (nsec): min=948, max=1081.8k, avg=8419.45, stdev=21439.27 clat (usec): min=1288, max=386792, avg=79732.23, stdev=28352.42 lat (usec): min=1298, max=386798, avg=79740.64, stdev=28352.52 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 37], 10.00th=[ 49], 20.00th=[ 59], | 30.00th=[ 67], 40.00th=[ 72], 50.00th=[ 80], 60.00th=[ 86], | 70.00th=[ 92], 80.00th=[ 100], 90.00th=[ 112], 95.00th=[ 123], | 99.00th=[ 155], 99.50th=[ 182], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 384] bw ( KiB/s): min= 6144, max=22829, per=6.26%, avg=12782.59, stdev=2218.87, samples=119 iops : min= 96, max= 356, avg=199.49, stdev=34.67, samples=119 lat (msec) : 2=0.14%, 4=0.15%, 10=0.26%, 20=0.74%, 50=9.80% lat (msec) : 100=69.58%, 250=19.20%, 500=0.12% cpu : usr=0.35%, sys=0.65%, ctx=9200, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.3%, 8=50.6%, 16=49.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=96.5%, 8=2.1%, 16=1.3%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,12005,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=746: Tue Dec 19 06:32:06 2023 write: IOPS=202, BW=12.7MiB/s (13.3MB/s)(766MiB/60507msec); 0 zone resets slat (nsec): min=1270, max=10596k, avg=9046.31, stdev=97373.78 clat (usec): min=629, max=466345, avg=78190.58, stdev=31550.84 lat (usec): min=638, max=466349, avg=78199.63, stdev=31551.27 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 34], 10.00th=[ 45], 20.00th=[ 57], | 30.00th=[ 65], 40.00th=[ 71], 50.00th=[ 78], 60.00th=[ 85], | 70.00th=[ 91], 80.00th=[ 99], 90.00th=[ 110], 95.00th=[ 122], | 99.00th=[ 153], 99.50th=[ 176], 99.90th=[ 439], 99.95th=[ 456], | 99.99th=[ 468] bw ( KiB/s): min= 5570, max=30208, per=6.39%, avg=13046.78, stdev=2698.10, samples=119 iops : min= 87, max= 472, avg=203.60, stdev=42.20, samples=119 lat (usec) : 750=0.01% lat (msec) : 2=0.12%, 4=0.26%, 10=0.30%, 20=1.31%, 50=11.37% lat (msec) : 100=68.54%, 250=17.84%, 500=0.24% cpu : usr=0.38%, sys=0.55%, ctx=9087, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.3%, 8=51.3%, 16=48.3%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=96.2%, 8=3.3%, 16=0.4%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,12252,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=747: Tue Dec 19 06:32:06 2023 write: IOPS=199, BW=12.5MiB/s (13.1MB/s)(762MiB/61100msec); 0 zone resets slat (nsec): min=762, max=25850k, avg=10680.44, stdev=237770.56 clat (usec): min=1918, max=294306, avg=78470.46, stdev=26552.57 lat (usec): min=1921, max=294313, avg=78481.14, stdev=26550.37 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 36], 10.00th=[ 47], 20.00th=[ 58], | 30.00th=[ 65], 40.00th=[ 71], 50.00th=[ 78], 60.00th=[ 84], | 70.00th=[ 91], 80.00th=[ 100], 90.00th=[ 110], 95.00th=[ 122], | 99.00th=[ 155], 99.50th=[ 169], 99.90th=[ 207], 99.95th=[ 232], | 99.99th=[ 296] bw ( KiB/s): min= 6560, max=29952, per=6.36%, avg=12975.67, stdev=2578.14, samples=119 iops : min= 102, max= 468, avg=202.53, stdev=40.31, samples=119 lat (msec) : 2=0.03%, 4=0.07%, 20=1.09%, 50=10.69%, 100=69.36% lat (msec) : 250=18.71%, 500=0.05% cpu : usr=0.35%, sys=0.61%, ctx=9095, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.7%, 16=49.1%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=96.6%, 8=3.4%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,12188,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=748: Tue Dec 19 06:32:06 2023 write: IOPS=197, BW=12.4MiB/s (13.0MB/s)(754MiB/61026msec); 0 zone resets slat (nsec): min=1052, max=5074.7k, avg=9253.08, stdev=72022.96 clat (msec): min=7, max=346, avg=79.36, stdev=27.28 lat (msec): min=7, max=346, avg=79.37, stdev=27.28 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 37], 10.00th=[ 50], 20.00th=[ 59], | 30.00th=[ 66], 40.00th=[ 72], 50.00th=[ 79], 60.00th=[ 85], | 70.00th=[ 91], 80.00th=[ 99], 90.00th=[ 111], 95.00th=[ 123], | 99.00th=[ 155], 99.50th=[ 176], 99.90th=[ 262], 99.95th=[ 313], | 99.99th=[ 342] bw ( KiB/s): min= 7675, max=23936, per=6.29%, avg=12847.73, stdev=2252.36, samples=119 iops : min= 119, max= 374, avg=200.53, stdev=35.21, samples=119 lat (msec) : 10=0.01%, 20=0.90%, 50=9.90%, 100=71.14%, 250=17.92% lat (msec) : 500=0.12% cpu : usr=0.32%, sys=0.68%, ctx=8983, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.6%, 16=49.1%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=96.3%, 8=2.6%, 16=1.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,12066,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=749: Tue Dec 19 06:32:06 2023 write: IOPS=197, BW=12.4MiB/s (13.0MB/s)(754MiB/60947msec); 0 zone resets slat (nsec): min=914, max=715902, avg=8104.70, stdev=14465.66 clat (usec): min=1406, max=433609, avg=79425.54, stdev=30856.21 lat (usec): min=1408, max=433615, avg=79433.64, stdev=30856.25 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 38], 10.00th=[ 49], 20.00th=[ 58], | 30.00th=[ 65], 40.00th=[ 71], 50.00th=[ 78], 60.00th=[ 85], | 70.00th=[ 92], 80.00th=[ 100], 90.00th=[ 111], 95.00th=[ 123], | 99.00th=[ 159], 99.50th=[ 188], 99.90th=[ 380], 99.95th=[ 414], | 99.99th=[ 430] bw ( KiB/s): min= 6684, max=27904, per=6.29%, avg=12840.03, stdev=2521.04, samples=119 iops : min= 104, max= 436, avg=200.43, stdev=39.41, samples=119 lat (msec) : 2=0.12%, 4=0.05%, 10=0.15%, 20=1.19%, 50=10.26% lat (msec) : 100=69.77%, 250=18.09%, 500=0.37% cpu : usr=0.36%, sys=0.63%, ctx=8996, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.8%, 16=48.8%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=96.0%, 8=2.3%, 16=1.7%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,12061,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=750: Tue Dec 19 06:32:06 2023 write: IOPS=199, BW=12.4MiB/s (13.0MB/s)(759MiB/61006msec); 0 zone resets slat (nsec): min=1966, max=6653.3k, avg=9023.16, stdev=77627.94 clat (usec): min=1819, max=433882, avg=78983.29, stdev=29230.27 lat (usec): min=1826, max=433889, avg=78992.31, stdev=29231.41 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 35], 10.00th=[ 47], 20.00th=[ 58], | 30.00th=[ 66], 40.00th=[ 72], 50.00th=[ 78], 60.00th=[ 85], | 70.00th=[ 91], 80.00th=[ 100], 90.00th=[ 112], 95.00th=[ 123], | 99.00th=[ 153], 99.50th=[ 165], 99.90th=[ 405], 99.95th=[ 414], | 99.99th=[ 435] bw ( KiB/s): min= 7922, max=35200, per=6.34%, avg=12935.36, stdev=2895.33, samples=119 iops : min= 123, max= 550, avg=201.88, stdev=45.28, samples=119 lat (msec) : 2=0.11%, 4=0.10%, 10=0.12%, 20=1.04%, 50=11.10% lat (msec) : 100=68.36%, 250=19.04%, 500=0.13% cpu : usr=0.39%, sys=0.52%, ctx=9008, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.8%, 16=48.9%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=96.4%, 8=3.0%, 16=0.6%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,12146,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=751: Tue Dec 19 06:32:06 2023 write: IOPS=201, BW=12.6MiB/s (13.2MB/s)(767MiB/61064msec); 0 zone resets slat (nsec): min=735, max=12896k, avg=9128.16, stdev=117410.44 clat (usec): min=1306, max=224001, avg=78158.03, stdev=27043.89 lat (usec): min=1311, max=224008, avg=78167.16, stdev=27041.33 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 35], 10.00th=[ 45], 20.00th=[ 58], | 30.00th=[ 66], 40.00th=[ 72], 50.00th=[ 78], 60.00th=[ 85], | 70.00th=[ 91], 80.00th=[ 100], 90.00th=[ 110], 95.00th=[ 122], | 99.00th=[ 150], 99.50th=[ 165], 99.90th=[ 211], 99.95th=[ 213], | 99.99th=[ 222] bw ( KiB/s): min= 8266, max=45056, per=6.40%, avg=13073.12, stdev=3519.21, samples=119 iops : min= 129, max= 704, avg=204.09, stdev=54.99, samples=119 lat (msec) : 2=0.24%, 4=0.33%, 10=0.26%, 20=1.50%, 50=10.26% lat (msec) : 100=68.85%, 250=18.55% cpu : usr=0.38%, sys=0.53%, ctx=9144, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.5%, 16=49.3%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=96.9%, 8=2.8%, 16=0.3%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,12275,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=752: Tue Dec 19 06:32:06 2023 write: IOPS=200, BW=12.6MiB/s (13.2MB/s)(767MiB/61095msec); 0 zone resets slat (nsec): min=863, max=3805.1k, avg=8376.67, stdev=38141.46 clat (usec): min=869, max=291625, avg=78170.45, stdev=27953.56 lat (usec): min=874, max=291634, avg=78178.82, stdev=27953.59 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 34], 10.00th=[ 47], 20.00th=[ 58], | 30.00th=[ 65], 40.00th=[ 71], 50.00th=[ 78], 60.00th=[ 84], | 70.00th=[ 91], 80.00th=[ 100], 90.00th=[ 111], 95.00th=[ 123], | 99.00th=[ 155], 99.50th=[ 171], 99.90th=[ 243], 99.95th=[ 251], | 99.99th=[ 292] bw ( KiB/s): min= 7922, max=47009, per=6.41%, avg=13074.01, stdev=3738.21, samples=119 iops : min= 123, max= 734, avg=204.05, stdev=58.40, samples=119 lat (usec) : 1000=0.08% lat (msec) : 4=0.43%, 10=0.40%, 20=1.58%, 50=10.40%, 100=68.81% lat (msec) : 250=18.24%, 500=0.06% cpu : usr=0.38%, sys=0.50%, ctx=9665, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.5%, 16=49.1%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=96.6%, 8=2.5%, 16=0.9%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,12273,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=753: Tue Dec 19 06:32:06 2023 write: IOPS=201, BW=12.6MiB/s (13.2MB/s)(762MiB/60380msec); 0 zone resets slat (nsec): min=679, max=6813.1k, avg=9187.62, stdev=86190.09 clat (usec): min=1483, max=264794, avg=78609.23, stdev=27909.27 lat (usec): min=1488, max=264803, avg=78618.42, stdev=27909.48 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 37], 10.00th=[ 47], 20.00th=[ 57], | 30.00th=[ 64], 40.00th=[ 71], 50.00th=[ 78], 60.00th=[ 84], | 70.00th=[ 92], 80.00th=[ 100], 90.00th=[ 111], 95.00th=[ 124], | 99.00th=[ 155], 99.50th=[ 178], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 266] bw ( KiB/s): min= 4343, max=30268, per=6.36%, avg=12984.26, stdev=2839.26, samples=119 iops : min= 67, max= 472, avg=202.65, stdev=44.36, samples=119 lat (msec) : 2=0.09%, 4=0.01%, 10=0.01%, 20=1.00%, 50=11.49% lat (msec) : 100=68.75%, 250=18.49%, 500=0.16% cpu : usr=0.37%, sys=0.56%, ctx=8361, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.3%, 8=51.0%, 16=48.7%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.8%, 8=3.1%, 16=1.2%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,12191,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=754: Tue Dec 19 06:32:06 2023 write: IOPS=202, BW=12.7MiB/s (13.3MB/s)(768MiB/60620msec); 0 zone resets slat (nsec): min=619, max=4324.6k, avg=8794.68, stdev=43104.46 clat (usec): min=210, max=757339, avg=78567.43, stdev=35860.16 lat (usec): min=255, max=757346, avg=78576.22, stdev=35860.40 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 33], 10.00th=[ 46], 20.00th=[ 58], | 30.00th=[ 65], 40.00th=[ 71], 50.00th=[ 78], 60.00th=[ 85], | 70.00th=[ 91], 80.00th=[ 100], 90.00th=[ 111], 95.00th=[ 124], | 99.00th=[ 148], 99.50th=[ 171], 99.90th=[ 726], 99.95th=[ 735], | 99.99th=[ 760] bw ( KiB/s): min= 2039, max=50688, per=6.40%, avg=13062.52, stdev=4136.85, samples=120 iops : min= 31, max= 792, avg=203.87, stdev=64.67, samples=120 lat (usec) : 250=0.01% lat (msec) : 2=0.19%, 4=0.18%, 10=0.79%, 20=1.58%, 50=10.09% lat (msec) : 100=68.75%, 250=18.29%, 750=0.11%, 1000=0.02% cpu : usr=0.36%, sys=0.66%, ctx=9061, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=50.9%, 16=48.8%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=96.0%, 8=3.0%, 16=1.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,12289,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=755: Tue Dec 19 06:32:06 2023 write: IOPS=200, BW=12.5MiB/s (13.1MB/s)(755MiB/60399msec); 0 zone resets slat (nsec): min=1425, max=2368.3k, avg=8699.16, stdev=39927.87 clat (msec): min=2, max=795, avg=79.24, stdev=36.21 lat (msec): min=2, max=795, avg=79.25, stdev=36.21 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 36], 10.00th=[ 47], 20.00th=[ 58], | 30.00th=[ 65], 40.00th=[ 71], 50.00th=[ 78], 60.00th=[ 84], | 70.00th=[ 90], 80.00th=[ 99], 90.00th=[ 110], 95.00th=[ 123], | 99.00th=[ 153], 99.50th=[ 169], 99.90th=[ 776], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 6437, max=32319, per=6.36%, avg=12977.36, stdev=2655.67, samples=118 iops : min= 100, max= 504, avg=202.54, stdev=41.47, samples=118 lat (msec) : 4=0.02%, 20=1.02%, 50=11.00%, 100=70.06%, 250=17.72% lat (msec) : 500=0.07%, 1000=0.12% cpu : usr=0.40%, sys=0.54%, ctx=9131, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.3%, 8=50.7%, 16=48.9%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.8%, 8=1.2%, 16=3.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,12086,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=199MiB/s (209MB/s), 12.3MiB/s-12.7MiB/s (12.9MB/s-13.3MB/s), io=11.9GiB (12.8GB), run=60380-61115msec Disk stats (read/write): dm-7: ios=0/163823, merge=0/0, ticks=0/13615856, in_queue=13615856, util=95.14%, aggrios=0/167913, aggrmerge=0/0, aggrticks=0/13958376, aggrin_queue=13958376, aggrutil=95.14% dm-6: ios=0/167913, merge=0/0, ticks=0/13958376, in_queue=13958376, util=95.14%, aggrios=0/167913, aggrmerge=0/0, aggrticks=0/13956531, aggrin_queue=13956531, aggrutil=95.12% dm-4: ios=0/167913, merge=0/0, ticks=0/13956531, in_queue=13956531, util=95.12%, aggrios=0/41998, aggrmerge=0/0, aggrticks=0/3489033, aggrin_queue=3489033, aggrutil=95.13% dm-1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% dm-2: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% dm-0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00% drbd1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/167993, aggrmerge=0/0, aggrticks=0/13955468, aggrin_queue=13955468, aggrutil=95.13% md1: ios=0/167993, merge=0/0, ticks=0/13955468, in_queue=13955468, util=95.13%, aggrios=105378/556500, aggrmerge=611065/1719455, aggrticks=282451/876721, aggrin_queue=1159648, aggrutil=85.52% sde: ios=105030/536131, merge=609992/1741251, ticks=322345/876646, in_queue=1199550, util=85.12% sdc: ios=108143/563490, merge=609070/1711718, ticks=217999/700780, in_queue=919116, util=84.69% sdb: ios=102551/541606, merge=615209/1733049, ticks=345192/1068737, in_queue=1414561, util=85.52% sda: ios=105791/584774, merge=609992/1691803, ticks=244270/860721, in_queue=1105365, util=85.40% dm-3: ios=0/167993, merge=0/0, ticks=0/13956133, in_queue=13956133, util=95.13%