random-write: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=posixaio, iodepth=16 ... fio-3.28 Starting 16 processes random-write: (groupid=0, jobs=1): err= 0: pid=798: Mon Dec 18 08:45:55 2023 read: IOPS=950, BW=59.4MiB/s (62.3MB/s)(3585MiB/60334msec) slat (nsec): min=150, max=5505.5k, avg=1047.13, stdev=23337.54 clat (usec): min=79, max=2246.5k, avg=16722.94, stdev=116827.16 lat (usec): min=84, max=2246.5k, avg=16723.99, stdev=116827.69 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 465], 10.00th=[ 486], | 20.00th=[ 529], 30.00th=[ 562], 40.00th=[ 603], | 50.00th=[ 685], 60.00th=[ 865], 70.00th=[ 1020], | 80.00th=[ 1287], 90.00th=[ 2245], 95.00th=[ 5538], | 99.00th=[ 650118], 99.50th=[ 742392], 99.90th=[1484784], | 99.95th=[1568670], 99.99th=[2231370] bw ( KiB/s): min= 128, max=267904, per=7.31%, avg=81538.84, stdev=80533.29, samples=90 iops : min= 2, max= 4186, avg=1273.94, stdev=1258.32, samples=90 lat (usec) : 100=0.01%, 250=0.03%, 500=13.21%, 750=40.36%, 1000=15.30% lat (msec) : 2=20.18%, 4=4.77%, 10=2.33%, 20=0.49%, 50=0.68% lat (msec) : 100=0.38%, 250=0.33%, 500=0.42%, 750=1.05%, 1000=0.12% lat (msec) : 2000=0.34%, >=2000=0.03% cpu : usr=0.51%, sys=1.37%, ctx=28404, majf=0, minf=37 IO depths : 1=0.2%, 2=0.5%, 4=2.2%, 8=51.6%, 16=45.6%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.8%, 8=2.4%, 16=1.8%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=57364,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=799: Mon Dec 18 08:45:55 2023 read: IOPS=1018, BW=63.7MiB/s (66.8MB/s)(3841MiB/60337msec) slat (nsec): min=150, max=2532.7k, avg=1041.46, stdev=14763.09 clat (usec): min=10, max=1466.3k, avg=15604.53, stdev=95978.08 lat (usec): min=72, max=1466.3k, avg=15605.57, stdev=95978.21 clat percentiles (usec): | 1.00th=[ 379], 5.00th=[ 461], 10.00th=[ 486], | 20.00th=[ 529], 30.00th=[ 578], 40.00th=[ 652], | 50.00th=[ 783], 60.00th=[ 922], 70.00th=[ 1106], | 80.00th=[ 1549], 90.00th=[ 3261], 95.00th=[ 13042], | 99.00th=[ 557843], 99.50th=[ 717226], 99.90th=[1350566], | 99.95th=[1400898], 99.99th=[1468007] bw ( KiB/s): min= 128, max=267776, per=7.34%, avg=81880.11, stdev=76020.09, samples=96 iops : min= 2, max= 4184, avg=1279.28, stdev=1187.81, samples=96 lat (usec) : 20=0.01%, 100=0.01%, 250=0.05%, 500=13.52%, 750=34.60% lat (usec) : 1000=16.86% lat (msec) : 2=18.44%, 4=8.25%, 10=2.80%, 20=0.98%, 50=0.86% lat (msec) : 100=0.65%, 250=0.98%, 500=0.87%, 750=0.74%, 1000=0.15% lat (msec) : 2000=0.24% cpu : usr=0.56%, sys=1.43%, ctx=31264, majf=0, minf=37 IO depths : 1=0.3%, 2=0.7%, 4=2.6%, 8=51.7%, 16=44.7%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.4%, 8=2.4%, 16=2.2%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=61457,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=800: Mon Dec 18 08:45:55 2023 read: IOPS=1056, BW=66.0MiB/s (69.2MB/s)(3968MiB/60117msec) slat (nsec): min=148, max=227623, avg=989.58, stdev=3456.88 clat (usec): min=3, max=1609.7k, avg=15048.47, stdev=103042.64 lat (usec): min=44, max=1609.7k, avg=15049.46, stdev=103042.69 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 461], 10.00th=[ 486], | 20.00th=[ 523], 30.00th=[ 562], 40.00th=[ 611], | 50.00th=[ 701], 60.00th=[ 865], 70.00th=[ 1037], | 80.00th=[ 1369], 90.00th=[ 2507], 95.00th=[ 8225], | 99.00th=[ 541066], 99.50th=[ 692061], 99.90th=[1518339], | 99.95th=[1535116], 99.99th=[1602225] bw ( KiB/s): min= 128, max=262912, per=7.59%, avg=84587.33, stdev=81658.37, samples=96 iops : min= 2, max= 4108, avg=1321.64, stdev=1275.96, samples=96 lat (usec) : 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.08% lat (usec) : 500=13.67%, 750=38.89%, 1000=15.51% lat (msec) : 2=19.03%, 4=5.54%, 10=2.62%, 20=0.92%, 50=0.75% lat (msec) : 100=0.47%, 250=0.66%, 500=0.61%, 750=0.77%, 1000=0.16% lat (msec) : 2000=0.29% cpu : usr=0.56%, sys=1.38%, ctx=29894, majf=0, minf=34 IO depths : 1=0.4%, 2=0.9%, 4=3.1%, 8=51.8%, 16=43.9%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.6%, 8=2.3%, 16=2.2%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=63490,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=801: Mon Dec 18 08:45:55 2023 read: IOPS=1199, BW=74.9MiB/s (78.6MB/s)(4501MiB/60062msec) slat (nsec): min=148, max=1781.9k, avg=1007.78, stdev=8217.81 clat (usec): min=38, max=1327.2k, avg=13254.61, stdev=80174.65 lat (usec): min=131, max=1327.2k, avg=13255.62, stdev=80174.72 clat percentiles (usec): | 1.00th=[ 396], 5.00th=[ 474], 10.00th=[ 498], | 20.00th=[ 537], 30.00th=[ 586], 40.00th=[ 652], | 50.00th=[ 791], 60.00th=[ 914], 70.00th=[ 1045], | 80.00th=[ 1369], 90.00th=[ 2507], 95.00th=[ 8029], | 99.00th=[ 549454], 99.50th=[ 683672], 99.90th=[ 817890], | 99.95th=[ 834667], 99.99th=[1333789] bw ( KiB/s): min= 127, max=263281, per=7.61%, avg=84872.47, stdev=71510.24, samples=108 iops : min= 1, max= 4113, avg=1326.03, stdev=1117.34, samples=108 lat (usec) : 50=0.01%, 250=0.04%, 500=10.29%, 750=37.06%, 1000=19.69% lat (msec) : 2=20.45%, 4=5.38%, 10=2.35%, 20=0.78%, 50=0.70% lat (msec) : 100=0.68%, 250=0.72%, 500=0.73%, 750=0.89%, 1000=0.21% lat (msec) : 2000=0.03% cpu : usr=0.62%, sys=1.72%, ctx=34532, majf=0, minf=33 IO depths : 1=0.3%, 2=0.7%, 4=2.6%, 8=51.8%, 16=44.6%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.6%, 8=2.0%, 16=2.4%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=72018,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=802: Mon Dec 18 08:45:55 2023 read: IOPS=1136, BW=71.0MiB/s (74.5MB/s)(4261MiB/60001msec) slat (nsec): min=150, max=2539.7k, avg=1061.59, stdev=12388.68 clat (usec): min=12, max=1459.6k, avg=13991.39, stdev=89870.82 lat (usec): min=87, max=1459.6k, avg=13992.45, stdev=89870.93 clat percentiles (usec): | 1.00th=[ 392], 5.00th=[ 469], 10.00th=[ 494], | 20.00th=[ 537], 30.00th=[ 578], 40.00th=[ 644], | 50.00th=[ 775], 60.00th=[ 906], 70.00th=[ 1057], | 80.00th=[ 1401], 90.00th=[ 2573], 95.00th=[ 7439], | 99.00th=[ 509608], 99.50th=[ 675283], 99.90th=[1132463], | 99.95th=[1400898], 99.99th=[1451230] bw ( KiB/s): min= 127, max=360448, per=7.41%, avg=82644.15, stdev=81485.38, samples=103 iops : min= 1, max= 5632, avg=1291.21, stdev=1273.20, samples=103 lat (usec) : 20=0.01%, 100=0.01%, 250=0.06%, 500=11.42%, 750=37.22% lat (usec) : 1000=18.60% lat (msec) : 2=19.49%, 4=6.45%, 10=2.21%, 20=0.58%, 50=0.81% lat (msec) : 100=0.60%, 250=0.76%, 500=0.75%, 750=0.68%, 1000=0.18% lat (msec) : 2000=0.21% cpu : usr=0.64%, sys=1.47%, ctx=33023, majf=0, minf=33 IO depths : 1=0.3%, 2=0.7%, 4=2.6%, 8=51.9%, 16=44.5%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.7%, 8=2.2%, 16=2.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=68176,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=803: Mon Dec 18 08:45:55 2023 read: IOPS=1079, BW=67.5MiB/s (70.8MB/s)(4097MiB/60699msec) slat (nsec): min=149, max=1557.2k, avg=1035.07, stdev=9602.93 clat (usec): min=99, max=1537.0k, avg=14706.32, stdev=97220.45 lat (usec): min=115, max=1537.0k, avg=14707.36, stdev=97220.71 clat percentiles (usec): | 1.00th=[ 375], 5.00th=[ 469], 10.00th=[ 494], | 20.00th=[ 537], 30.00th=[ 578], 40.00th=[ 635], | 50.00th=[ 750], 60.00th=[ 873], 70.00th=[ 1004], | 80.00th=[ 1303], 90.00th=[ 2278], 95.00th=[ 8717], | 99.00th=[ 583009], 99.50th=[ 734004], 99.90th=[1384121], | 99.95th=[1451230], 99.99th=[1535116] bw ( KiB/s): min= 124, max=264192, per=7.38%, avg=82226.83, stdev=81445.62, samples=102 iops : min= 1, max= 4128, avg=1284.67, stdev=1272.61, samples=102 lat (usec) : 100=0.01%, 250=0.06%, 500=11.32%, 750=38.91%, 1000=19.67% lat (msec) : 2=18.89%, 4=4.31%, 10=2.09%, 20=0.49%, 50=1.07% lat (msec) : 100=0.86%, 250=0.56%, 500=0.55%, 750=0.79%, 1000=0.22% lat (msec) : 2000=0.23% cpu : usr=0.61%, sys=1.42%, ctx=31631, majf=0, minf=35 IO depths : 1=0.2%, 2=0.5%, 4=2.3%, 8=51.9%, 16=45.1%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.6%, 8=2.0%, 16=2.3%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=65552,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=804: Mon Dec 18 08:45:55 2023 read: IOPS=1193, BW=74.6MiB/s (78.2MB/s)(4486MiB/60132msec) slat (nsec): min=149, max=2806.8k, avg=1046.11, stdev=11786.27 clat (usec): min=29, max=920858, avg=13314.66, stdev=74954.87 lat (usec): min=136, max=920858, avg=13315.71, stdev=74954.92 clat percentiles (usec): | 1.00th=[ 396], 5.00th=[ 469], 10.00th=[ 498], 20.00th=[ 545], | 30.00th=[ 611], 40.00th=[ 701], 50.00th=[ 824], 60.00th=[ 938], | 70.00th=[ 1074], 80.00th=[ 1352], 90.00th=[ 2507], 95.00th=[ 13173], | 99.00th=[471860], 99.50th=[641729], 99.90th=[775947], 99.95th=[784335], | 99.99th=[918553] bw ( KiB/s): min= 255, max=266368, per=7.42%, avg=82715.06, stdev=71893.75, samples=111 iops : min= 3, max= 4162, avg=1292.31, stdev=1123.38, samples=111 lat (usec) : 50=0.01%, 250=0.02%, 500=10.48%, 750=33.52%, 1000=21.10% lat (msec) : 2=22.73%, 4=4.87%, 10=1.89%, 20=0.78%, 50=0.92% lat (msec) : 100=0.84%, 250=0.93%, 500=0.98%, 750=0.73%, 1000=0.22% cpu : usr=0.67%, sys=1.59%, ctx=37203, majf=0, minf=35 IO depths : 1=0.3%, 2=0.8%, 4=2.5%, 8=51.7%, 16=44.7%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.7%, 8=2.0%, 16=2.3%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=71779,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=805: Mon Dec 18 08:45:55 2023 read: IOPS=1049, BW=65.6MiB/s (68.8MB/s)(3966MiB/60470msec) slat (nsec): min=150, max=3289.4k, avg=1040.44, stdev=13592.32 clat (usec): min=11, max=1508.8k, avg=15140.36, stdev=93150.64 lat (usec): min=82, max=1508.8k, avg=15141.40, stdev=93150.77 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 465], 10.00th=[ 494], | 20.00th=[ 537], 30.00th=[ 594], 40.00th=[ 685], | 50.00th=[ 824], 60.00th=[ 947], 70.00th=[ 1123], | 80.00th=[ 1500], 90.00th=[ 2769], 95.00th=[ 8979], | 99.00th=[ 574620], 99.50th=[ 725615], 99.90th=[1182794], | 99.95th=[1400898], 99.99th=[1501561] bw ( KiB/s): min= 128, max=262144, per=7.35%, avg=81963.69, stdev=72131.71, samples=99 iops : min= 2, max= 4096, avg=1280.58, stdev=1127.08, samples=99 lat (usec) : 20=0.01%, 100=0.01%, 250=0.07%, 500=11.13%, 750=33.65% lat (usec) : 1000=18.67% lat (msec) : 2=22.19%, 4=6.91%, 10=2.49%, 20=0.78%, 50=0.77% lat (msec) : 100=0.59%, 250=0.76%, 500=0.68%, 750=0.96%, 1000=0.16% lat (msec) : 2000=0.16% cpu : usr=0.57%, sys=1.48%, ctx=33223, majf=0, minf=33 IO depths : 1=0.3%, 2=0.8%, 4=2.5%, 8=51.8%, 16=44.5%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=2.1%, 16=2.4%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=63460,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=806: Mon Dec 18 08:45:55 2023 read: IOPS=1079, BW=67.5MiB/s (70.7MB/s)(4097MiB/60724msec) slat (nsec): min=149, max=4974.6k, avg=1161.91, stdev=22899.63 clat (usec): min=46, max=1455.5k, avg=14691.57, stdev=90034.88 lat (usec): min=82, max=1455.5k, avg=14692.73, stdev=90035.35 clat percentiles (usec): | 1.00th=[ 392], 5.00th=[ 465], 10.00th=[ 490], | 20.00th=[ 529], 30.00th=[ 578], 40.00th=[ 660], | 50.00th=[ 791], 60.00th=[ 889], 70.00th=[ 1020], | 80.00th=[ 1270], 90.00th=[ 2278], 95.00th=[ 7046], | 99.00th=[ 501220], 99.50th=[ 700449], 99.90th=[1199571], | 99.95th=[1317012], 99.99th=[1451230] bw ( KiB/s): min= 127, max=273664, per=7.44%, avg=82997.53, stdev=69207.97, samples=101 iops : min= 1, max= 4276, avg=1296.73, stdev=1081.38, samples=101 lat (usec) : 50=0.01%, 100=0.01%, 250=0.02%, 500=12.34%, 750=35.05% lat (usec) : 1000=21.51% lat (msec) : 2=19.63%, 4=4.91%, 10=1.95%, 20=0.42%, 50=0.74% lat (msec) : 100=0.33%, 250=1.10%, 500=0.98%, 750=0.65%, 1000=0.19% lat (msec) : 2000=0.17% cpu : usr=0.57%, sys=1.58%, ctx=31968, majf=0, minf=32 IO depths : 1=0.3%, 2=0.7%, 4=2.3%, 8=51.7%, 16=45.1%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.6%, 8=2.4%, 16=2.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=65552,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=807: Mon Dec 18 08:45:55 2023 read: IOPS=1086, BW=67.9MiB/s (71.2MB/s)(4102MiB/60429msec) slat (nsec): min=150, max=2167.0k, avg=1100.59, stdev=12625.42 clat (usec): min=70, max=1465.3k, avg=14648.05, stdev=91047.41 lat (usec): min=73, max=1465.3k, avg=14649.15, stdev=91047.51 clat percentiles (usec): | 1.00th=[ 375], 5.00th=[ 469], 10.00th=[ 490], | 20.00th=[ 529], 30.00th=[ 570], 40.00th=[ 635], | 50.00th=[ 775], 60.00th=[ 914], 70.00th=[ 1057], | 80.00th=[ 1385], 90.00th=[ 2474], 95.00th=[ 11600], | 99.00th=[ 497026], 99.50th=[ 641729], 99.90th=[1384121], | 99.95th=[1417675], 99.99th=[1468007] bw ( KiB/s): min= 128, max=264448, per=7.46%, avg=83135.96, stdev=66558.42, samples=101 iops : min= 2, max= 4132, avg=1298.86, stdev=1039.99, samples=101 lat (usec) : 100=0.01%, 250=0.06%, 500=12.85%, 750=35.82%, 1000=18.00% lat (msec) : 2=20.39%, 4=5.89%, 10=1.78%, 20=0.63%, 50=0.82% lat (msec) : 100=0.69%, 250=1.22%, 500=0.87%, 750=0.67%, 1000=0.14% lat (msec) : 2000=0.18% cpu : usr=0.60%, sys=1.44%, ctx=30386, majf=0, minf=33 IO depths : 1=0.3%, 2=0.7%, 4=2.7%, 8=51.8%, 16=44.5%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.6%, 8=2.0%, 16=2.5%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=65633,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=808: Mon Dec 18 08:45:55 2023 read: IOPS=1017, BW=63.6MiB/s (66.7MB/s)(3846MiB/60476msec) slat (nsec): min=149, max=1038.2k, avg=1018.53, stdev=6474.86 clat (usec): min=60, max=1577.3k, avg=15613.06, stdev=95038.24 lat (usec): min=72, max=1577.3k, avg=15614.08, stdev=95038.24 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 465], 10.00th=[ 486], | 20.00th=[ 529], 30.00th=[ 578], 40.00th=[ 652], | 50.00th=[ 799], 60.00th=[ 914], 70.00th=[ 1090], | 80.00th=[ 1434], 90.00th=[ 2638], 95.00th=[ 10814], | 99.00th=[ 599786], 99.50th=[ 717226], 99.90th=[1115685], | 99.95th=[1417675], 99.99th=[1568670] bw ( KiB/s): min= 128, max=265856, per=6.86%, avg=76450.02, stdev=71676.30, samples=103 iops : min= 2, max= 4154, avg=1194.44, stdev=1119.94, samples=103 lat (usec) : 100=0.01%, 250=0.05%, 500=13.09%, 750=33.83%, 1000=18.99% lat (msec) : 2=19.79%, 4=7.30%, 10=1.86%, 20=0.74%, 50=0.83% lat (msec) : 100=0.65%, 250=0.91%, 500=0.58%, 750=0.94%, 1000=0.28% lat (msec) : 2000=0.15% cpu : usr=0.51%, sys=1.51%, ctx=30138, majf=0, minf=33 IO depths : 1=0.4%, 2=0.9%, 4=2.8%, 8=52.1%, 16=43.9%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=2.4%, 16=2.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=61540,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=809: Mon Dec 18 08:45:55 2023 read: IOPS=1159, BW=72.4MiB/s (76.0MB/s)(4382MiB/60490msec) slat (nsec): min=149, max=8719.5k, avg=1200.72, stdev=35375.03 clat (usec): min=7, max=1096.9k, avg=13709.02, stdev=78444.61 lat (usec): min=127, max=1096.9k, avg=13710.22, stdev=78444.99 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 469], 10.00th=[ 494], | 20.00th=[ 537], 30.00th=[ 578], 40.00th=[ 652], | 50.00th=[ 783], 60.00th=[ 898], 70.00th=[ 1037], | 80.00th=[ 1336], 90.00th=[ 2704], 95.00th=[ 18482], | 99.00th=[ 501220], 99.50th=[ 700449], 99.90th=[ 809501], | 99.95th=[ 834667], 99.99th=[1098908] bw ( KiB/s): min= 128, max=267904, per=7.66%, avg=85434.72, stdev=67799.14, samples=105 iops : min= 2, max= 4186, avg=1334.81, stdev=1059.36, samples=105 lat (usec) : 10=0.01%, 250=0.04%, 500=11.59%, 750=36.49%, 1000=19.66% lat (msec) : 2=19.06%, 4=5.18%, 10=2.15%, 20=0.89%, 50=1.00% lat (msec) : 100=0.92%, 250=1.33%, 500=0.68%, 750=0.67%, 1000=0.31% lat (msec) : 2000=0.02% cpu : usr=0.65%, sys=1.55%, ctx=33316, majf=0, minf=33 IO depths : 1=0.3%, 2=0.7%, 4=2.6%, 8=52.0%, 16=44.5%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=2.2%, 16=2.2%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=70108,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=810: Mon Dec 18 08:45:55 2023 read: IOPS=1107, BW=69.2MiB/s (72.6MB/s)(4158MiB/60054msec) slat (nsec): min=151, max=2791.6k, avg=995.86, stdev=11379.75 clat (usec): min=95, max=1247.9k, avg=14374.41, stdev=86798.63 lat (usec): min=107, max=1247.9k, avg=14375.41, stdev=86798.67 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 469], 10.00th=[ 498], | 20.00th=[ 537], 30.00th=[ 594], 40.00th=[ 676], | 50.00th=[ 807], 60.00th=[ 930], 70.00th=[ 1074], | 80.00th=[ 1352], 90.00th=[ 2507], 95.00th=[ 14746], | 99.00th=[ 616563], 99.50th=[ 725615], 99.90th=[1010828], | 99.95th=[1082131], 99.99th=[1249903] bw ( KiB/s): min= 128, max=266752, per=7.49%, avg=83460.80, stdev=77464.30, samples=102 iops : min= 2, max= 4168, avg=1303.92, stdev=1210.39, samples=102 lat (usec) : 100=0.01%, 250=0.06%, 500=10.69%, 750=35.44%, 1000=18.96% lat (msec) : 2=22.23%, 4=5.36%, 10=1.78%, 20=0.67%, 50=1.36% lat (msec) : 100=0.78%, 250=0.94%, 500=0.57%, 750=0.73%, 1000=0.31% lat (msec) : 2000=0.11% cpu : usr=0.62%, sys=1.51%, ctx=33107, majf=0, minf=33 IO depths : 1=0.3%, 2=0.6%, 4=2.3%, 8=51.8%, 16=45.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=2.3%, 16=2.2%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=66529,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=811: Mon Dec 18 08:45:55 2023 read: IOPS=1096, BW=68.5MiB/s (71.8MB/s)(4110MiB/60003msec) slat (nsec): min=150, max=1285.8k, avg=1014.65, stdev=6645.51 clat (usec): min=68, max=1625.0k, avg=14517.84, stdev=99299.54 lat (usec): min=94, max=1625.0k, avg=14518.86, stdev=99299.58 clat percentiles (usec): | 1.00th=[ 375], 5.00th=[ 469], 10.00th=[ 494], | 20.00th=[ 537], 30.00th=[ 586], 40.00th=[ 644], | 50.00th=[ 783], 60.00th=[ 922], 70.00th=[ 1057], | 80.00th=[ 1336], 90.00th=[ 2278], 95.00th=[ 6718], | 99.00th=[ 530580], 99.50th=[ 742392], 99.90th=[1535116], | 99.95th=[1585447], 99.99th=[1619002] bw ( KiB/s): min= 128, max=348416, per=7.69%, avg=85680.82, stdev=79918.05, samples=98 iops : min= 2, max= 5444, avg=1338.68, stdev=1248.74, samples=98 lat (usec) : 100=0.01%, 250=0.05%, 500=11.61%, 750=36.66%, 1000=18.32% lat (msec) : 2=21.90%, 4=5.31%, 10=1.62%, 20=0.56%, 50=0.80% lat (msec) : 100=0.60%, 250=0.82%, 500=0.65%, 750=0.61%, 1000=0.28% lat (msec) : 2000=0.21% cpu : usr=0.62%, sys=1.53%, ctx=32689, majf=0, minf=33 IO depths : 1=0.3%, 2=0.7%, 4=2.7%, 8=51.9%, 16=44.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.6%, 8=1.9%, 16=2.5%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=65764,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=812: Mon Dec 18 08:45:55 2023 read: IOPS=1222, BW=76.4MiB/s (80.1MB/s)(4609MiB/60334msec) slat (nsec): min=149, max=1085.2k, avg=971.87, stdev=6790.68 clat (usec): min=15, max=1170.4k, avg=13003.77, stdev=75854.59 lat (usec): min=99, max=1170.4k, avg=13004.74, stdev=75854.67 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 469], 10.00th=[ 494], | 20.00th=[ 537], 30.00th=[ 578], 40.00th=[ 644], | 50.00th=[ 783], 60.00th=[ 914], 70.00th=[ 1045], | 80.00th=[ 1336], 90.00th=[ 2311], 95.00th=[ 10028], | 99.00th=[ 476054], 99.50th=[ 599786], 99.90th=[ 868221], | 99.95th=[ 893387], 99.99th=[1166017] bw ( KiB/s): min= 127, max=294656, per=7.69%, avg=85786.40, stdev=71282.05, samples=110 iops : min= 1, max= 4604, avg=1340.31, stdev=1113.85, samples=110 lat (usec) : 20=0.01%, 100=0.01%, 250=0.04%, 500=11.46%, 750=36.81% lat (usec) : 1000=18.53% lat (msec) : 2=21.52%, 4=4.66%, 10=1.97%, 20=0.62%, 50=0.84% lat (msec) : 100=0.73%, 250=1.01%, 500=0.87%, 750=0.70%, 1000=0.21% lat (msec) : 2000=0.02% cpu : usr=0.61%, sys=1.60%, ctx=36211, majf=0, minf=33 IO depths : 1=0.4%, 2=0.8%, 4=2.8%, 8=51.8%, 16=44.2%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.6%, 8=2.0%, 16=2.4%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=73748,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=813: Mon Dec 18 08:45:55 2023 read: IOPS=1085, BW=67.8MiB/s (71.1MB/s)(4102MiB/60491msec) slat (nsec): min=149, max=761697, avg=965.67, stdev=5017.37 clat (usec): min=45, max=1495.7k, avg=14661.09, stdev=91326.05 lat (usec): min=64, max=1495.7k, avg=14662.05, stdev=91326.13 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 461], 10.00th=[ 486], | 20.00th=[ 529], 30.00th=[ 562], 40.00th=[ 627], | 50.00th=[ 758], 60.00th=[ 881], 70.00th=[ 1004], | 80.00th=[ 1303], 90.00th=[ 2409], 95.00th=[ 10683], | 99.00th=[ 566232], 99.50th=[ 700449], 99.90th=[1182794], | 99.95th=[1367344], 99.99th=[1501561] bw ( KiB/s): min= 128, max=267624, per=7.46%, avg=83159.71, stdev=77029.88, samples=101 iops : min= 2, max= 4181, avg=1299.23, stdev=1203.56, samples=101 lat (usec) : 50=0.01%, 100=0.01%, 250=0.04%, 500=13.14%, 750=36.49% lat (usec) : 1000=19.97% lat (msec) : 2=18.43%, 4=4.76%, 10=2.12%, 20=0.69%, 50=0.87% lat (msec) : 100=0.67%, 250=0.97%, 500=0.67%, 750=0.79%, 1000=0.22% lat (msec) : 2000=0.16% cpu : usr=0.53%, sys=1.49%, ctx=31346, majf=0, minf=33 IO depths : 1=0.3%, 2=0.7%, 4=2.4%, 8=51.9%, 16=44.7%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.7%, 8=2.2%, 16=2.2%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=65636,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1089MiB/s (1142MB/s), 59.4MiB/s-76.4MiB/s (62.3MB/s-80.1MB/s), io=64.6GiB (69.3GB), run=60001-60724msec Disk stats (read/write): dm-7: ios=20239/113, merge=0/0, ticks=11737951/9978, in_queue=11747929, util=78.50%, aggrios=153939/113, aggrmerge=0/0, aggrticks=67288538/9978, aggrin_queue=67298516, aggrutil=99.80% dm-6: ios=153939/113, merge=0/0, ticks=67288538/9978, in_queue=67298516, util=99.80%, aggrios=153939/113, aggrmerge=0/0, aggrticks=67287130/9977, aggrin_queue=67297107, aggrutil=99.81% dm-4: ios=153939/113, merge=0/0, ticks=67287130/9977, in_queue=67297107, util=99.81%, aggrios=38484/33, aggrmerge=0/0, aggrticks=16821637/2604, aggrin_queue=16824241, aggrutil=99.80% dm-1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% dm-2: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% dm-0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00% drbd1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=153939/133, aggrmerge=0/0, aggrticks=67285851/10415, aggrin_queue=67296266, aggrutil=99.80% md1: ios=153939/133, merge=0/0, ticks=67285851/10415, in_queue=67296266, util=99.80%, aggrios=45853/115, aggrmerge=5446/32, aggrticks=20188573/8307, aggrin_queue=20201487, aggrutil=99.65% sdc: ios=45760/121, merge=5565/60, ticks=27832724/9765, in_queue=27847650, util=99.65% sdb: ios=46054/113, merge=5256/19, ticks=6681037/6252, in_queue=6691216, util=91.99% sda: ios=45745/111, merge=5518/18, ticks=26051958/8905, in_queue=26065596, util=99.42% dm-3: ios=153939/133, merge=0/0, ticks=67286550/10416, in_queue=67296966, util=99.80%