random-write: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=posixaio, iodepth=16 ... fio-3.28 Starting 16 processes random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: (groupid=0, jobs=1): err= 0: pid=702: Tue Dec 19 06:25:01 2023 read: IOPS=1209, BW=75.6MiB/s (79.3MB/s)(4536MiB/60007msec) slat (nsec): min=153, max=13893k, avg=2217.66, stdev=88561.34 clat (usec): min=5, max=1122.8k, avg=13022.28, stdev=67746.78 lat (usec): min=57, max=1122.8k, avg=13024.50, stdev=67747.41 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 644], 10.00th=[ 783], | 20.00th=[ 1045], 30.00th=[ 1303], 40.00th=[ 1729], | 50.00th=[ 2343], 60.00th=[ 3163], 70.00th=[ 4359], | 80.00th=[ 6456], 90.00th=[ 11600], 95.00th=[ 22938], | 99.00th=[ 358613], 99.50th=[ 526386], 99.90th=[1098908], | 99.95th=[1098908], 99.99th=[1115685] bw ( KiB/s): min= 127, max=262272, per=5.42%, avg=80621.34, stdev=71910.46, samples=114 iops : min= 1, max= 4098, avg=1259.56, stdev=1123.63, samples=114 lat (usec) : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.02% lat (usec) : 500=0.69%, 750=8.08%, 1000=9.60% lat (msec) : 2=26.46%, 4=22.79%, 10=20.33%, 20=6.40%, 50=2.71% lat (msec) : 100=0.60%, 250=1.06%, 500=0.70%, 750=0.33%, 1000=0.09% lat (msec) : 2000=0.13% cpu : usr=0.97%, sys=2.35%, ctx=48655, majf=0, minf=34 IO depths : 1=0.4%, 2=1.0%, 4=2.8%, 8=52.1%, 16=43.6%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.6%, 8=2.0%, 16=2.4%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=72573,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=703: Tue Dec 19 06:25:01 2023 read: IOPS=1355, BW=84.7MiB/s (88.9MB/s)(5121MiB/60429msec) slat (nsec): min=152, max=15809k, avg=2151.16, stdev=93040.33 clat (usec): min=9, max=1105.3k, avg=11628.08, stdev=57242.42 lat (usec): min=31, max=1105.6k, avg=11630.23, stdev=57244.43 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 668], 10.00th=[ 807], | 20.00th=[ 1037], 30.00th=[ 1287], 40.00th=[ 1680], | 50.00th=[ 2278], 60.00th=[ 2999], 70.00th=[ 4080], | 80.00th=[ 5866], 90.00th=[ 10552], 95.00th=[ 21103], | 99.00th=[ 316670], 99.50th=[ 467665], 99.90th=[ 725615], | 99.95th=[ 935330], 99.99th=[1098908] bw ( KiB/s): min= 91, max=322944, per=6.02%, avg=89528.95, stdev=80770.31, samples=117 iops : min= 1, max= 5046, avg=1398.80, stdev=1262.05, samples=117 lat (usec) : 10=0.01%, 20=0.01%, 250=0.01%, 500=0.68%, 750=7.15% lat (usec) : 1000=10.52% lat (msec) : 2=27.83%, 4=23.00%, 10=20.32%, 20=5.21%, 50=2.56% lat (msec) : 100=0.55%, 250=0.94%, 500=0.85%, 750=0.29%, 1000=0.08% lat (msec) : 2000=0.02% cpu : usr=1.07%, sys=2.76%, ctx=56731, majf=0, minf=34 IO depths : 1=0.4%, 2=0.9%, 4=2.8%, 8=52.0%, 16=43.9%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.3%, 8=2.0%, 16=2.6%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=81936,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=704: Tue Dec 19 06:25:01 2023 read: IOPS=1540, BW=96.3MiB/s (101MB/s)(5777MiB/60005msec) slat (nsec): min=151, max=6572.2k, avg=1516.62, stdev=30389.77 clat (usec): min=36, max=884692, avg=10190.58, stdev=49096.34 lat (usec): min=131, max=884693, avg=10192.10, stdev=49097.49 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 685], 10.00th=[ 840], 20.00th=[ 1057], | 30.00th=[ 1287], 40.00th=[ 1598], 50.00th=[ 2245], 60.00th=[ 2933], | 70.00th=[ 4015], 80.00th=[ 5800], 90.00th=[ 10552], 95.00th=[ 20317], | 99.00th=[225444], 99.50th=[392168], 99.90th=[683672], 99.95th=[734004], | 99.99th=[884999] bw ( KiB/s): min= 128, max=315008, per=6.70%, avg=99660.21, stdev=80195.65, samples=118 iops : min= 2, max= 4922, avg=1557.07, stdev=1253.07, samples=118 lat (usec) : 50=0.01%, 250=0.04%, 500=0.54%, 750=6.36%, 1000=10.31% lat (msec) : 2=29.77%, 4=22.79%, 10=19.64%, 20=5.48%, 50=2.59% lat (msec) : 100=0.54%, 250=1.01%, 500=0.60%, 750=0.28%, 1000=0.04% cpu : usr=1.13%, sys=3.12%, ctx=61484, majf=0, minf=35 IO depths : 1=0.5%, 2=1.3%, 4=3.4%, 8=52.1%, 16=42.7%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=1.9%, 16=2.7%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=92439,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=705: Tue Dec 19 06:25:01 2023 read: IOPS=1481, BW=92.6MiB/s (97.1MB/s)(5554MiB/60004msec) slat (nsec): min=152, max=33468k, avg=2158.41, stdev=121342.58 clat (usec): min=95, max=1122.7k, avg=10641.35, stdev=51914.13 lat (usec): min=103, max=1122.7k, avg=10643.51, stdev=51916.92 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 709], 10.00th=[ 832], | 20.00th=[ 1037], 30.00th=[ 1237], 40.00th=[ 1500], | 50.00th=[ 2057], 60.00th=[ 2737], 70.00th=[ 3916], | 80.00th=[ 5866], 90.00th=[ 10945], 95.00th=[ 22938], | 99.00th=[ 229639], 99.50th=[ 404751], 99.90th=[ 666895], | 99.95th=[ 742392], 99.99th=[1115685] bw ( KiB/s): min= 128, max=376448, per=6.48%, avg=96391.15, stdev=85577.67, samples=117 iops : min= 2, max= 5882, avg=1506.01, stdev=1337.16, samples=117 lat (usec) : 100=0.01%, 250=0.01%, 500=0.53%, 750=5.91%, 1000=11.89% lat (msec) : 2=30.75%, 4=21.71%, 10=18.26%, 20=5.28%, 50=2.79% lat (msec) : 100=0.91%, 250=1.02%, 500=0.58%, 750=0.33%, 1000=0.01% lat (msec) : 2000=0.03% cpu : usr=1.14%, sys=2.99%, ctx=56874, majf=0, minf=36 IO depths : 1=0.5%, 2=1.1%, 4=3.3%, 8=52.3%, 16=42.7%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.4%, 8=2.2%, 16=2.4%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=88866,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=706: Tue Dec 19 06:25:01 2023 read: IOPS=1499, BW=93.7MiB/s (98.3MB/s)(5638MiB/60147msec) slat (nsec): min=149, max=15710k, avg=1733.54, stdev=58540.14 clat (usec): min=19, max=1115.3k, avg=10312.00, stdev=50228.95 lat (usec): min=128, max=1115.3k, avg=10313.73, stdev=50229.78 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 611], 10.00th=[ 750], | 20.00th=[ 1004], 30.00th=[ 1221], 40.00th=[ 1450], | 50.00th=[ 1729], 60.00th=[ 2311], 70.00th=[ 3490], | 80.00th=[ 5604], 90.00th=[ 11076], 95.00th=[ 21890], | 99.00th=[ 244319], 99.50th=[ 408945], 99.90th=[ 675283], | 99.95th=[ 759170], 99.99th=[1115685] bw ( KiB/s): min= 128, max=386048, per=6.68%, avg=99375.20, stdev=83020.97, samples=116 iops : min= 2, max= 6032, avg=1552.60, stdev=1297.27, samples=116 lat (usec) : 20=0.01%, 50=0.01%, 250=0.01%, 500=0.98%, 750=9.16% lat (usec) : 1000=9.57% lat (msec) : 2=36.01%, 4=17.39%, 10=15.57%, 20=5.81%, 50=2.69% lat (msec) : 100=0.75%, 250=1.08%, 500=0.69%, 750=0.22%, 1000=0.05% lat (msec) : 2000=0.02% cpu : usr=1.18%, sys=3.04%, ctx=60098, majf=0, minf=34 IO depths : 1=0.4%, 2=1.0%, 4=2.9%, 8=52.1%, 16=43.6%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.6%, 8=1.9%, 16=2.6%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=90209,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=707: Tue Dec 19 06:25:01 2023 read: IOPS=1535, BW=96.0MiB/s (101MB/s)(5761MiB/60008msec) slat (nsec): min=152, max=9843.4k, avg=1709.38, stdev=44825.92 clat (usec): min=10, max=869883, avg=10234.03, stdev=49276.46 lat (usec): min=78, max=869884, avg=10235.74, stdev=49276.57 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 627], 10.00th=[ 766], 20.00th=[ 988], | 30.00th=[ 1188], 40.00th=[ 1450], 50.00th=[ 1942], 60.00th=[ 2573], | 70.00th=[ 3720], 80.00th=[ 5342], 90.00th=[ 9634], 95.00th=[ 19792], | 99.00th=[263193], 99.50th=[400557], 99.90th=[633340], 99.95th=[692061], | 99.99th=[868221] bw ( KiB/s): min= 127, max=344576, per=6.67%, avg=99253.09, stdev=85002.08, samples=118 iops : min= 1, max= 5384, avg=1550.69, stdev=1328.23, samples=118 lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.66% lat (usec) : 750=8.70%, 1000=11.27% lat (msec) : 2=30.06%, 4=21.77%, 10=17.94%, 20=4.62%, 50=2.31% lat (msec) : 100=0.62%, 250=0.98%, 500=0.72%, 750=0.29%, 1000=0.03% cpu : usr=1.25%, sys=2.97%, ctx=58550, majf=0, minf=35 IO depths : 1=0.4%, 2=1.0%, 4=2.8%, 8=52.3%, 16=43.5%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.6%, 8=1.9%, 16=2.5%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=92172,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=708: Tue Dec 19 06:25:01 2023 read: IOPS=1580, BW=98.8MiB/s (104MB/s)(5929MiB/60003msec) slat (nsec): min=152, max=7413.8k, avg=1551.95, stdev=36139.44 clat (usec): min=24, max=982052, avg=9874.43, stdev=49022.87 lat (usec): min=78, max=982053, avg=9875.98, stdev=49024.18 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 701], 10.00th=[ 824], 20.00th=[ 1020], | 30.00th=[ 1221], 40.00th=[ 1483], 50.00th=[ 1991], 60.00th=[ 2638], | 70.00th=[ 3589], 80.00th=[ 5276], 90.00th=[ 9634], 95.00th=[ 19530], | 99.00th=[227541], 99.50th=[358613], 99.90th=[750781], 99.95th=[834667], | 99.99th=[977273] bw ( KiB/s): min= 128, max=367488, per=7.01%, avg=104312.05, stdev=89963.32, samples=116 iops : min= 2, max= 5742, avg=1629.76, stdev=1405.66, samples=116 lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.60%, 750=6.15% lat (usec) : 1000=12.19% lat (msec) : 2=31.12%, 4=23.08%, 10=17.23%, 20=4.76%, 50=2.18% lat (msec) : 100=0.70%, 250=1.07%, 500=0.61%, 750=0.20%, 1000=0.10% cpu : usr=1.17%, sys=3.33%, ctx=62055, majf=0, minf=34 IO depths : 1=0.5%, 2=1.2%, 4=3.4%, 8=52.3%, 16=42.6%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.4%, 8=2.3%, 16=2.2%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=94864,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=709: Tue Dec 19 06:25:01 2023 read: IOPS=1448, BW=90.5MiB/s (94.9MB/s)(5438MiB/60075msec) slat (nsec): min=152, max=3110.9k, avg=1499.85, stdev=19925.20 clat (usec): min=68, max=911908, avg=10764.33, stdev=48643.51 lat (usec): min=71, max=911938, avg=10765.83, stdev=48643.85 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 693], 10.00th=[ 848], 20.00th=[ 1106], | 30.00th=[ 1319], 40.00th=[ 1745], 50.00th=[ 2409], 60.00th=[ 3195], | 70.00th=[ 4359], 80.00th=[ 6325], 90.00th=[ 11600], 95.00th=[ 22938], | 99.00th=[246416], 99.50th=[400557], 99.90th=[641729], 99.95th=[742392], | 99.99th=[910164] bw ( KiB/s): min= 512, max=271104, per=6.54%, avg=97208.78, stdev=74185.05, samples=114 iops : min= 8, max= 4236, avg=1518.73, stdev=1159.12, samples=114 lat (usec) : 100=0.01%, 250=0.01%, 500=0.48%, 750=6.36%, 1000=9.19% lat (msec) : 2=27.66%, 4=23.88%, 10=20.70%, 20=6.05%, 50=2.78% lat (msec) : 100=0.85%, 250=1.07%, 500=0.68%, 750=0.24%, 1000=0.05% cpu : usr=1.26%, sys=2.85%, ctx=57212, majf=0, minf=33 IO depths : 1=0.5%, 2=1.2%, 4=3.2%, 8=52.2%, 16=42.9%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=1.9%, 16=2.6%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=87012,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=710: Tue Dec 19 06:25:01 2023 read: IOPS=1530, BW=95.7MiB/s (100MB/s)(5758MiB/60190msec) slat (nsec): min=159, max=7350.5k, avg=1741.22, stdev=41166.55 clat (usec): min=4, max=819988, avg=10223.00, stdev=47536.47 lat (usec): min=56, max=819989, avg=10224.74, stdev=47537.25 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 652], 10.00th=[ 791], 20.00th=[ 1012], | 30.00th=[ 1221], 40.00th=[ 1500], 50.00th=[ 2089], 60.00th=[ 2737], | 70.00th=[ 3851], 80.00th=[ 5604], 90.00th=[ 10683], 95.00th=[ 20579], | 99.00th=[231736], 99.50th=[371196], 99.90th=[700449], 99.95th=[734004], | 99.99th=[817890] bw ( KiB/s): min= 127, max=299520, per=6.71%, avg=99731.22, stdev=77965.15, samples=117 iops : min= 1, max= 4680, avg=1558.15, stdev=1218.23, samples=117 lat (usec) : 10=0.01%, 20=0.01%, 100=0.01%, 250=0.02%, 500=0.88% lat (usec) : 750=7.72%, 1000=10.79% lat (msec) : 2=29.59%, 4=22.29%, 10=18.04%, 20=5.50%, 50=2.32% lat (msec) : 100=0.66%, 250=1.28%, 500=0.65%, 750=0.20%, 1000=0.05% cpu : usr=1.21%, sys=3.14%, ctx=55942, majf=0, minf=34 IO depths : 1=0.5%, 2=1.1%, 4=3.2%, 8=52.6%, 16=42.6%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.4%, 8=2.3%, 16=2.2%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=92132,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=711: Tue Dec 19 06:25:01 2023 read: IOPS=1520, BW=95.1MiB/s (99.7MB/s)(5704MiB/60003msec) slat (nsec): min=156, max=8490.1k, avg=1538.67, stdev=31415.48 clat (usec): min=5, max=850140, avg=10305.19, stdev=48414.50 lat (usec): min=61, max=850376, avg=10306.73, stdev=48414.75 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 635], 10.00th=[ 783], 20.00th=[ 1020], | 30.00th=[ 1221], 40.00th=[ 1483], 50.00th=[ 2114], 60.00th=[ 2769], | 70.00th=[ 3884], 80.00th=[ 5800], 90.00th=[ 10814], 95.00th=[ 20317], | 99.00th=[240124], 99.50th=[404751], 99.90th=[683672], 99.95th=[801113], | 99.99th=[851444] bw ( KiB/s): min= 510, max=306048, per=6.66%, avg=99030.24, stdev=81471.37, samples=117 iops : min= 7, max= 4782, avg=1547.18, stdev=1273.01, samples=117 lat (usec) : 10=0.01%, 20=0.01%, 100=0.01%, 250=0.02%, 500=0.94% lat (usec) : 750=7.94%, 1000=10.02% lat (msec) : 2=29.61%, 4=22.36%, 10=18.15%, 20=5.83%, 50=2.31% lat (msec) : 100=0.70%, 250=1.15%, 500=0.72%, 750=0.17%, 1000=0.08% cpu : usr=1.16%, sys=2.95%, ctx=57056, majf=0, minf=34 IO depths : 1=0.6%, 2=1.3%, 4=3.6%, 8=52.2%, 16=42.3%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=1.6%, 16=2.9%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=91263,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=712: Tue Dec 19 06:25:01 2023 read: IOPS=1493, BW=93.4MiB/s (97.9MB/s)(5633MiB/60331msec) slat (nsec): min=154, max=20793k, avg=1617.64, stdev=71255.69 clat (usec): min=45, max=863550, avg=10506.16, stdev=50463.68 lat (usec): min=193, max=863550, avg=10507.78, stdev=50464.06 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 685], 10.00th=[ 840], 20.00th=[ 1074], | 30.00th=[ 1270], 40.00th=[ 1582], 50.00th=[ 2114], 60.00th=[ 2737], | 70.00th=[ 3752], 80.00th=[ 5604], 90.00th=[ 10421], 95.00th=[ 20055], | 99.00th=[258999], 99.50th=[413139], 99.90th=[641729], 99.95th=[767558], | 99.99th=[859833] bw ( KiB/s): min= 96, max=379008, per=6.62%, avg=98441.08, stdev=81391.34, samples=117 iops : min= 1, max= 5922, avg=1537.99, stdev=1271.74, samples=117 lat (usec) : 50=0.01%, 250=0.01%, 500=0.55%, 750=6.44%, 1000=10.06% lat (msec) : 2=31.34%, 4=23.45%, 10=17.65%, 20=5.51%, 50=2.28% lat (msec) : 100=0.60%, 250=1.09%, 500=0.68%, 750=0.29%, 1000=0.06% cpu : usr=1.23%, sys=3.01%, ctx=62152, majf=0, minf=33 IO depths : 1=0.5%, 2=1.1%, 4=3.0%, 8=52.0%, 16=43.5%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.6%, 8=1.5%, 16=2.9%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=90129,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=713: Tue Dec 19 06:25:01 2023 read: IOPS=1493, BW=93.3MiB/s (97.9MB/s)(5632MiB/60334msec) slat (nsec): min=149, max=29622k, avg=1961.65, stdev=103955.59 clat (usec): min=6, max=803458, avg=10468.32, stdev=48053.15 lat (usec): min=40, max=803459, avg=10470.28, stdev=48054.06 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 676], 10.00th=[ 807], 20.00th=[ 1012], | 30.00th=[ 1205], 40.00th=[ 1467], 50.00th=[ 2040], 60.00th=[ 2737], | 70.00th=[ 3916], 80.00th=[ 5800], 90.00th=[ 10945], 95.00th=[ 21103], | 99.00th=[258999], 99.50th=[396362], 99.90th=[650118], 99.95th=[675283], | 99.99th=[801113] bw ( KiB/s): min= 128, max=305024, per=6.60%, avg=98179.59, stdev=77002.77, samples=117 iops : min= 2, max= 4766, avg=1533.88, stdev=1203.17, samples=117 lat (usec) : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01% lat (usec) : 500=0.61%, 750=7.00%, 1000=11.83% lat (msec) : 2=30.00%, 4=21.24%, 10=18.20%, 20=5.79%, 50=2.46% lat (msec) : 100=0.68%, 250=1.12%, 500=0.74%, 750=0.26%, 1000=0.03% cpu : usr=1.13%, sys=2.99%, ctx=57354, majf=0, minf=34 IO depths : 1=0.5%, 2=1.1%, 4=3.2%, 8=52.4%, 16=42.8%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.4%, 8=2.1%, 16=2.5%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=90113,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=714: Tue Dec 19 06:25:01 2023 read: IOPS=1357, BW=84.9MiB/s (89.0MB/s)(5120MiB/60333msec) slat (nsec): min=154, max=154024k, avg=3529.82, stdev=539749.90 clat (usec): min=41, max=1128.3k, avg=11509.70, stdev=55308.49 lat (usec): min=90, max=1128.3k, avg=11513.23, stdev=55317.01 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 701], 10.00th=[ 857], | 20.00th=[ 1074], 30.00th=[ 1287], 40.00th=[ 1647], | 50.00th=[ 2343], 60.00th=[ 3261], 70.00th=[ 4424], | 80.00th=[ 6194], 90.00th=[ 11731], 95.00th=[ 24511], | 99.00th=[ 295699], 99.50th=[ 379585], 99.90th=[ 784335], | 99.95th=[1019216], 99.99th=[1132463] bw ( KiB/s): min= 127, max=293888, per=5.96%, avg=88701.92, stdev=75594.58, samples=118 iops : min= 1, max= 4592, avg=1385.81, stdev=1181.18, samples=118 lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.83%, 750=5.17% lat (usec) : 1000=10.27% lat (msec) : 2=29.15%, 4=21.27%, 10=21.34%, 20=6.20%, 50=2.80% lat (msec) : 100=0.78%, 250=1.05%, 500=0.85%, 750=0.16%, 1000=0.06% lat (msec) : 2000=0.06% cpu : usr=1.05%, sys=2.80%, ctx=56110, majf=0, minf=34 IO depths : 1=0.4%, 2=0.9%, 4=2.6%, 8=51.9%, 16=44.2%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.6%, 8=1.4%, 16=2.9%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=81921,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=715: Tue Dec 19 06:25:01 2023 read: IOPS=1392, BW=87.1MiB/s (91.3MB/s)(5224MiB/60003msec) slat (nsec): min=153, max=4009.4k, avg=1544.61, stdev=24119.81 clat (usec): min=22, max=1033.9k, avg=11292.83, stdev=54454.24 lat (usec): min=116, max=1033.9k, avg=11294.37, stdev=54455.48 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 693], 10.00th=[ 848], | 20.00th=[ 1074], 30.00th=[ 1336], 40.00th=[ 1762], | 50.00th=[ 2474], 60.00th=[ 3294], 70.00th=[ 4555], | 80.00th=[ 6456], 90.00th=[ 11731], 95.00th=[ 21103], | 99.00th=[ 263193], 99.50th=[ 450888], 99.90th=[ 742392], | 99.95th=[ 935330], 99.99th=[1035994] bw ( KiB/s): min= 128, max=323840, per=6.05%, avg=90016.15, stdev=76585.77, samples=117 iops : min= 2, max= 5060, avg=1406.34, stdev=1196.72, samples=117 lat (usec) : 50=0.01%, 250=0.01%, 500=0.83%, 750=5.81%, 1000=10.09% lat (msec) : 2=26.01%, 4=23.64%, 10=21.61%, 20=6.67%, 50=2.52% lat (msec) : 100=0.71%, 250=1.05%, 500=0.70%, 750=0.25%, 1000=0.08% lat (msec) : 2000=0.02% cpu : usr=1.08%, sys=2.89%, ctx=53129, majf=0, minf=34 IO depths : 1=0.5%, 2=1.2%, 4=3.3%, 8=52.4%, 16=42.6%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=1.9%, 16=2.6%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=83582,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=716: Tue Dec 19 06:25:01 2023 read: IOPS=1427, BW=89.2MiB/s (93.5MB/s)(5378MiB/60286msec) slat (nsec): min=152, max=5822.1k, avg=1536.07, stdev=29435.01 clat (usec): min=13, max=924883, avg=11045.03, stdev=53788.08 lat (usec): min=84, max=924883, avg=11046.57, stdev=53788.75 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 644], 10.00th=[ 775], 20.00th=[ 963], | 30.00th=[ 1172], 40.00th=[ 1434], 50.00th=[ 1909], 60.00th=[ 2540], | 70.00th=[ 3523], 80.00th=[ 5276], 90.00th=[ 10814], 95.00th=[ 23987], | 99.00th=[261096], 99.50th=[425722], 99.90th=[750781], 99.95th=[851444], | 99.99th=[926942] bw ( KiB/s): min= 256, max=265984, per=6.60%, avg=98178.74, stdev=77668.25, samples=112 iops : min= 4, max= 4156, avg=1533.88, stdev=1213.57, samples=112 lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.83% lat (usec) : 750=8.29%, 1000=13.06% lat (msec) : 2=29.01%, 4=22.57%, 10=15.57%, 20=4.75%, 50=2.77% lat (msec) : 100=0.94%, 250=1.10%, 500=0.71%, 750=0.28%, 1000=0.10% cpu : usr=1.11%, sys=2.84%, ctx=55395, majf=0, minf=34 IO depths : 1=0.4%, 2=1.0%, 4=2.8%, 8=52.1%, 16=43.8%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.6%, 8=2.0%, 16=2.4%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=86052,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=717: Tue Dec 19 06:25:01 2023 read: IOPS=1484, BW=92.8MiB/s (97.3MB/s)(5566MiB/60007msec) slat (nsec): min=149, max=18379k, avg=1829.24, stdev=74143.76 clat (usec): min=44, max=999005, avg=10468.25, stdev=48356.94 lat (usec): min=45, max=999006, avg=10470.08, stdev=48358.13 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 701], 10.00th=[ 865], | 20.00th=[ 1074], 30.00th=[ 1287], 40.00th=[ 1631], | 50.00th=[ 2212], 60.00th=[ 2900], 70.00th=[ 4015], | 80.00th=[ 6063], 90.00th=[ 11207], 95.00th=[ 22676], | 99.00th=[ 238027], 99.50th=[ 408945], 99.90th=[ 624952], | 99.95th=[ 750781], 99.99th=[1002439] bw ( KiB/s): min= 128, max=261760, per=6.50%, avg=96615.10, stdev=74070.87, samples=117 iops : min= 2, max= 4090, avg=1509.44, stdev=1157.32, samples=117 lat (usec) : 50=0.01%, 250=0.01%, 500=0.69%, 750=5.49%, 1000=10.29% lat (msec) : 2=30.23%, 4=23.11%, 10=18.86%, 20=5.64%, 50=2.86% lat (msec) : 100=0.83%, 250=1.06%, 500=0.62%, 750=0.26%, 1000=0.06% cpu : usr=1.22%, sys=2.93%, ctx=60407, majf=0, minf=34 IO depths : 1=0.5%, 2=1.1%, 4=3.0%, 8=51.9%, 16=43.5%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=1.7%, 16=2.8%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=89059,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1452MiB/s (1523MB/s), 75.6MiB/s-98.8MiB/s (79.3MB/s-104MB/s), io=85.7GiB (92.0GB), run=60003-60429msec Disk stats (read/write): dm-7: ios=26886/95, merge=0/0, ticks=7513290/16550, in_queue=7529840, util=86.46%, aggrios=203952/95, aggrmerge=0/0, aggrticks=36382937/16550, aggrin_queue=36399487, aggrutil=98.56% dm-6: ios=203952/95, merge=0/0, ticks=36382937/16550, in_queue=36399487, util=98.56%, aggrios=203952/95, aggrmerge=0/0, aggrticks=36376715/16549, aggrin_queue=36393264, aggrutil=98.55% dm-4: ios=203952/95, merge=0/0, ticks=36376715/16549, in_queue=36393264, util=98.55%, aggrios=50988/28, aggrmerge=0/0, aggrticks=9093745/4223, aggrin_queue=9097968, aggrutil=98.55% dm-1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% dm-2: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% dm-0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00% drbd1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=203952/113, aggrmerge=0/0, aggrticks=36372326/16891, aggrin_queue=36389217, aggrutil=98.55% md1: ios=203952/113, merge=0/0, ticks=36372326/16891, in_queue=36389217, util=98.55%, aggrios=45060/86, aggrmerge=5946/25, aggrticks=8196642/4926, aggrin_queue=8204306, aggrutil=98.13% sde: ios=44435/58, merge=6276/11, ticks=4700365/2540, in_queue=4705177, util=91.47% sdc: ios=44760/95, merge=5397/34, ticks=12896254/6590, in_queue=12906056, util=98.13% sdb: ios=46650/99, merge=6508/33, ticks=4951900/4205, in_queue=4958529, util=91.83% sda: ios=44395/95, merge=5606/24, ticks=10238050/6369, in_queue=10247465, util=97.64% dm-3: ios=203952/113, merge=0/0, ticks=36374982/16893, in_queue=36391875, util=98.55%