random-write: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=posixaio, iodepth=16 ... fio-3.28 Starting 16 processes random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: Laying out IO file (1 file / 256MiB) random-write: (groupid=0, jobs=1): err= 0: pid=733: Mon Dec 18 13:41:58 2023 read: IOPS=1152, BW=72.0MiB/s (75.5MB/s)(4350MiB/60386msec) slat (nsec): min=149, max=9409.5k, avg=1539.14, stdev=48635.52 clat (usec): min=90, max=1296.0k, avg=13660.36, stdev=75890.15 lat (usec): min=98, max=1296.0k, avg=13661.89, stdev=75890.68 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 510], 10.00th=[ 553], | 20.00th=[ 644], 30.00th=[ 816], 40.00th=[ 988], | 50.00th=[ 1156], 60.00th=[ 1418], 70.00th=[ 1991], | 80.00th=[ 3261], 90.00th=[ 8586], 95.00th=[ 20055], | 99.00th=[ 513803], 99.50th=[ 599786], 99.90th=[ 834667], | 99.95th=[1052771], 99.99th=[1300235] bw ( KiB/s): min= 128, max=325760, per=5.57%, avg=79446.21, stdev=81648.89, samples=111 iops : min= 2, max= 5090, avg=1241.21, stdev=1275.72, samples=111 lat (usec) : 100=0.01%, 250=0.03%, 500=3.93%, 750=22.45%, 1000=14.41% lat (msec) : 2=29.29%, 4=12.99%, 10=7.99%, 20=3.88%, 50=1.83% lat (msec) : 100=0.76%, 250=0.73%, 500=0.69%, 750=0.89%, 1000=0.05% lat (msec) : 2000=0.08% cpu : usr=0.80%, sys=2.16%, ctx=41212, majf=0, minf=35 IO depths : 1=0.4%, 2=0.9%, 4=2.7%, 8=52.0%, 16=44.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.8%, 8=2.0%, 16=2.3%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=69598,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=734: Mon Dec 18 13:41:58 2023 read: IOPS=1467, BW=91.7MiB/s (96.1MB/s)(5502MiB/60008msec) slat (nsec): min=150, max=15034k, avg=1417.13, stdev=51275.71 clat (usec): min=6, max=1261.9k, avg=10775.30, stdev=59145.63 lat (usec): min=119, max=1262.1k, avg=10776.71, stdev=59145.79 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 545], 10.00th=[ 594], | 20.00th=[ 709], 30.00th=[ 898], 40.00th=[ 1074], | 50.00th=[ 1254], 60.00th=[ 1467], 70.00th=[ 2040], | 80.00th=[ 3261], 90.00th=[ 7111], 95.00th=[ 16581], | 99.00th=[ 341836], 99.50th=[ 492831], 99.90th=[ 666895], | 99.95th=[ 725615], 99.99th=[1266680] bw ( KiB/s): min= 256, max=302080, per=6.94%, avg=99039.16, stdev=84468.57, samples=113 iops : min= 4, max= 4720, avg=1547.35, stdev=1319.84, samples=113 lat (usec) : 10=0.01%, 250=0.02%, 500=1.94%, 750=20.21%, 1000=13.97% lat (msec) : 2=33.46%, 4=13.78%, 10=8.98%, 20=3.20%, 50=1.47% lat (msec) : 100=0.71%, 250=0.96%, 500=0.83%, 750=0.42%, 1000=0.03% lat (msec) : 2000=0.02% cpu : usr=1.07%, sys=2.66%, ctx=52266, majf=0, minf=37 IO depths : 1=0.5%, 2=1.1%, 4=3.1%, 8=52.1%, 16=43.2%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=2.0%, 16=2.5%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=88036,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=735: Mon Dec 18 13:41:58 2023 read: IOPS=1395, BW=87.2MiB/s (91.5MB/s)(5247MiB/60155msec) slat (nsec): min=151, max=13156k, avg=1740.12, stdev=62589.91 clat (usec): min=111, max=968840, avg=11275.40, stdev=57662.02 lat (usec): min=122, max=968840, avg=11277.14, stdev=57662.58 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 553], 10.00th=[ 619], 20.00th=[ 775], | 30.00th=[ 938], 40.00th=[ 1090], 50.00th=[ 1270], 60.00th=[ 1614], | 70.00th=[ 2474], 80.00th=[ 4359], 90.00th=[ 8979], 95.00th=[ 21103], | 99.00th=[320865], 99.50th=[517997], 99.90th=[650118], 99.95th=[683672], | 99.99th=[968885] bw ( KiB/s): min= 128, max=278355, per=6.61%, avg=94297.58, stdev=82191.44, samples=113 iops : min= 2, max= 4349, avg=1473.26, stdev=1284.21, samples=113 lat (usec) : 250=0.02%, 500=1.88%, 750=16.79%, 1000=15.80% lat (msec) : 2=31.01%, 4=12.86%, 10=12.39%, 20=4.01%, 50=1.95% lat (msec) : 100=0.99%, 250=0.97%, 500=0.76%, 750=0.53%, 1000=0.04% cpu : usr=0.95%, sys=2.61%, ctx=48911, majf=0, minf=36 IO depths : 1=0.6%, 2=1.3%, 4=3.8%, 8=52.3%, 16=42.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.4%, 8=1.6%, 16=2.9%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=83956,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=736: Mon Dec 18 13:41:58 2023 read: IOPS=1429, BW=89.3MiB/s (93.7MB/s)(5377MiB/60197msec) slat (nsec): min=151, max=10170k, avg=1460.07, stdev=37235.63 clat (usec): min=74, max=1121.9k, avg=10976.08, stdev=57907.73 lat (usec): min=84, max=1121.9k, avg=10977.54, stdev=57907.80 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 562], 10.00th=[ 611], | 20.00th=[ 766], 30.00th=[ 971], 40.00th=[ 1156], | 50.00th=[ 1352], 60.00th=[ 1827], 70.00th=[ 2671], | 80.00th=[ 4424], 90.00th=[ 8356], 95.00th=[ 16712], | 99.00th=[ 358613], 99.50th=[ 492831], 99.90th=[ 650118], | 99.95th=[ 666895], 99.99th=[1115685] bw ( KiB/s): min= 128, max=270976, per=6.82%, avg=97339.57, stdev=78433.69, samples=113 iops : min= 2, max= 4234, avg=1520.79, stdev=1225.52, samples=113 lat (usec) : 100=0.01%, 250=0.02%, 500=1.36%, 750=18.14%, 1000=11.92% lat (msec) : 2=31.00%, 4=15.64%, 10=13.62%, 20=3.95%, 50=1.41% lat (msec) : 100=0.78%, 250=0.81%, 500=0.85%, 750=0.45%, 1000=0.03% lat (msec) : 2000=0.01% cpu : usr=1.02%, sys=2.61%, ctx=52661, majf=0, minf=38 IO depths : 1=0.5%, 2=1.2%, 4=3.2%, 8=52.0%, 16=43.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=1.7%, 16=2.8%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=86035,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=737: Mon Dec 18 13:41:58 2023 read: IOPS=1417, BW=88.6MiB/s (92.9MB/s)(5317MiB/60001msec) slat (nsec): min=150, max=4679.0k, avg=1524.05, stdev=28103.37 clat (usec): min=11, max=702191, avg=11104.06, stdev=59628.60 lat (usec): min=94, max=702202, avg=11105.59, stdev=59628.60 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 523], 10.00th=[ 578], 20.00th=[ 709], | 30.00th=[ 873], 40.00th=[ 1057], 50.00th=[ 1237], 60.00th=[ 1532], | 70.00th=[ 2311], 80.00th=[ 3851], 90.00th=[ 8455], 95.00th=[ 16909], | 99.00th=[375391], 99.50th=[541066], 99.90th=[658506], 99.95th=[683672], | 99.99th=[700449] bw ( KiB/s): min= 128, max=317184, per=6.54%, avg=93324.64, stdev=81647.56, samples=115 iops : min= 2, max= 4956, avg=1458.06, stdev=1275.75, samples=115 lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.02%, 500=3.35% lat (usec) : 750=19.14%, 1000=14.43% lat (msec) : 2=30.07%, 4=13.74%, 10=10.71%, 20=4.06%, 50=1.62% lat (msec) : 100=0.73%, 250=0.68%, 500=0.80%, 750=0.62% cpu : usr=0.94%, sys=2.68%, ctx=48008, majf=0, minf=34 IO depths : 1=0.5%, 2=1.1%, 4=3.2%, 8=52.2%, 16=42.9%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=2.0%, 16=2.5%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=85076,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=738: Mon Dec 18 13:41:58 2023 read: IOPS=1452, BW=90.8MiB/s (95.2MB/s)(5449MiB/60002msec) slat (nsec): min=151, max=5401.4k, avg=1488.28, stdev=29735.94 clat (usec): min=69, max=879779, avg=10928.87, stdev=55744.28 lat (usec): min=86, max=879780, avg=10930.36, stdev=55744.52 clat percentiles (usec): | 1.00th=[ 461], 5.00th=[ 545], 10.00th=[ 594], 20.00th=[ 717], | 30.00th=[ 906], 40.00th=[ 1074], 50.00th=[ 1287], 60.00th=[ 1680], | 70.00th=[ 2474], 80.00th=[ 4080], 90.00th=[ 8094], 95.00th=[ 18220], | 99.00th=[304088], 99.50th=[492831], 99.90th=[641729], 99.95th=[683672], | 99.99th=[876610] bw ( KiB/s): min= 128, max=274560, per=6.92%, avg=98726.38, stdev=81261.41, samples=112 iops : min= 2, max= 4290, avg=1542.48, stdev=1269.71, samples=112 lat (usec) : 100=0.01%, 250=0.01%, 500=2.08%, 750=20.03%, 1000=13.10% lat (msec) : 2=29.62%, 4=14.85%, 10=12.17%, 20=3.36%, 50=1.64% lat (msec) : 100=0.73%, 250=1.10%, 500=0.85%, 750=0.44%, 1000=0.02% cpu : usr=0.91%, sys=2.74%, ctx=49258, majf=0, minf=34 IO depths : 1=0.6%, 2=1.3%, 4=3.5%, 8=52.3%, 16=42.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=1.8%, 16=2.7%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=87181,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=739: Mon Dec 18 13:41:58 2023 read: IOPS=1490, BW=93.2MiB/s (97.7MB/s)(5633MiB/60468msec) slat (nsec): min=150, max=11214k, avg=1414.33, stdev=41655.70 clat (usec): min=97, max=1001.4k, avg=10511.75, stdev=56281.32 lat (usec): min=109, max=1001.4k, avg=10513.16, stdev=56281.70 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 553], 10.00th=[ 603], | 20.00th=[ 758], 30.00th=[ 930], 40.00th=[ 1090], | 50.00th=[ 1254], 60.00th=[ 1532], 70.00th=[ 2311], | 80.00th=[ 3523], 90.00th=[ 7242], 95.00th=[ 17171], | 99.00th=[ 308282], 99.50th=[ 459277], 99.90th=[ 683672], | 99.95th=[ 792724], 99.99th=[1002439] bw ( KiB/s): min= 512, max=366208, per=7.09%, avg=101104.82, stdev=84672.13, samples=114 iops : min= 8, max= 5722, avg=1579.62, stdev=1323.05, samples=114 lat (usec) : 100=0.01%, 250=0.01%, 500=1.87%, 750=17.86%, 1000=14.55% lat (msec) : 2=32.77%, 4=15.30%, 10=10.09%, 20=2.99%, 50=1.56% lat (msec) : 100=0.73%, 250=0.99%, 500=0.88%, 750=0.35%, 1000=0.03% lat (msec) : 2000=0.02% cpu : usr=1.02%, sys=2.70%, ctx=54684, majf=0, minf=33 IO depths : 1=0.5%, 2=1.2%, 4=3.3%, 8=52.1%, 16=42.9%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.6%, 8=1.8%, 16=2.6%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=90128,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=740: Mon Dec 18 13:41:58 2023 read: IOPS=1429, BW=89.3MiB/s (93.7MB/s)(5382MiB/60259msec) slat (nsec): min=151, max=7968.3k, avg=1332.46, stdev=29971.48 clat (usec): min=104, max=1063.3k, avg=11060.27, stdev=58748.75 lat (usec): min=111, max=1063.3k, avg=11061.60, stdev=58748.93 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 562], 10.00th=[ 627], | 20.00th=[ 783], 30.00th=[ 971], 40.00th=[ 1156], | 50.00th=[ 1401], 60.00th=[ 1909], 70.00th=[ 2671], | 80.00th=[ 3949], 90.00th=[ 7832], 95.00th=[ 17957], | 99.00th=[ 329253], 99.50th=[ 517997], 99.90th=[ 675283], | 99.95th=[ 717226], 99.99th=[1061159] bw ( KiB/s): min= 125, max=319488, per=6.59%, avg=94002.44, stdev=83524.14, samples=117 iops : min= 1, max= 4992, avg=1468.59, stdev=1305.10, samples=117 lat (usec) : 250=0.01%, 500=1.66%, 750=16.47%, 1000=13.85% lat (msec) : 2=29.07%, 4=19.25%, 10=12.09%, 20=2.99%, 50=1.57% lat (msec) : 100=0.85%, 250=0.88%, 500=0.78%, 750=0.52%, 1000=0.02% lat (msec) : 2000=0.01% cpu : usr=1.04%, sys=2.61%, ctx=57270, majf=0, minf=33 IO depths : 1=0.4%, 2=0.9%, 4=2.4%, 8=51.6%, 16=44.7%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.6%, 8=1.9%, 16=2.5%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=86116,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=741: Mon Dec 18 13:41:58 2023 read: IOPS=1428, BW=89.3MiB/s (93.6MB/s)(5382MiB/60281msec) slat (nsec): min=149, max=10816k, avg=1636.17, stdev=46308.80 clat (usec): min=6, max=1212.3k, avg=11057.34, stdev=59406.94 lat (usec): min=68, max=1212.3k, avg=11058.98, stdev=59407.17 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 545], 10.00th=[ 627], | 20.00th=[ 799], 30.00th=[ 947], 40.00th=[ 1090], | 50.00th=[ 1254], 60.00th=[ 1582], 70.00th=[ 2278], | 80.00th=[ 3818], 90.00th=[ 8029], 95.00th=[ 17433], | 99.00th=[ 350225], 99.50th=[ 501220], 99.90th=[ 700449], | 99.95th=[ 742392], 99.99th=[1216349] bw ( KiB/s): min= 127, max=265856, per=6.76%, avg=96512.33, stdev=78024.62, samples=114 iops : min= 1, max= 4154, avg=1507.84, stdev=1219.12, samples=114 lat (usec) : 10=0.01%, 50=0.01%, 100=0.01%, 250=0.02%, 500=2.61% lat (usec) : 750=14.81%, 1000=16.67% lat (msec) : 2=32.94%, 4=13.83%, 10=11.15%, 20=3.44%, 50=1.63% lat (msec) : 100=0.70%, 250=0.77%, 500=0.91%, 750=0.46%, 1000=0.03% lat (msec) : 2000=0.02% cpu : usr=0.98%, sys=2.57%, ctx=51070, majf=0, minf=34 IO depths : 1=0.5%, 2=1.1%, 4=3.1%, 8=52.3%, 16=43.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=1.8%, 16=2.7%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=86116,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=742: Mon Dec 18 13:41:58 2023 read: IOPS=1366, BW=85.4MiB/s (89.6MB/s)(5137MiB/60127msec) slat (nsec): min=152, max=1901.8k, avg=1283.03, stdev=12222.02 clat (usec): min=126, max=938333, avg=11566.32, stdev=59733.26 lat (usec): min=127, max=938334, avg=11567.60, stdev=59733.36 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 553], 10.00th=[ 611], 20.00th=[ 758], | 30.00th=[ 914], 40.00th=[ 1074], 50.00th=[ 1254], 60.00th=[ 1582], | 70.00th=[ 2409], 80.00th=[ 4228], 90.00th=[ 9896], 95.00th=[ 21103], | 99.00th=[358613], 99.50th=[513803], 99.90th=[700449], 99.95th=[784335], | 99.99th=[935330] bw ( KiB/s): min= 128, max=341632, per=6.45%, avg=91964.23, stdev=79654.46, samples=114 iops : min= 2, max= 5338, avg=1436.78, stdev=1244.63, samples=114 lat (usec) : 250=0.01%, 500=2.02%, 750=17.38%, 1000=16.32% lat (msec) : 2=30.42%, 4=13.02%, 10=10.93%, 20=4.67%, 50=2.18% lat (msec) : 100=0.64%, 250=1.10%, 500=0.76%, 750=0.49%, 1000=0.06% cpu : usr=0.95%, sys=2.50%, ctx=50546, majf=0, minf=34 IO depths : 1=0.4%, 2=1.0%, 4=2.8%, 8=51.9%, 16=44.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=1.8%, 16=2.6%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=82186,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=743: Mon Dec 18 13:41:58 2023 read: IOPS=1360, BW=85.0MiB/s (89.2MB/s)(5121MiB/60218msec) slat (nsec): min=150, max=3040.1k, avg=1246.20, stdev=12560.96 clat (usec): min=50, max=1107.0k, avg=11571.13, stdev=62996.59 lat (usec): min=59, max=1107.0k, avg=11572.38, stdev=62996.70 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 545], 10.00th=[ 594], | 20.00th=[ 717], 30.00th=[ 914], 40.00th=[ 1057], | 50.00th=[ 1237], 60.00th=[ 1500], 70.00th=[ 2245], | 80.00th=[ 3720], 90.00th=[ 7570], 95.00th=[ 16909], | 99.00th=[ 379585], 99.50th=[ 541066], 99.90th=[ 742392], | 99.95th=[ 817890], 99.99th=[1098908] bw ( KiB/s): min= 128, max=268263, per=6.74%, avg=96112.72, stdev=84552.92, samples=109 iops : min= 2, max= 4191, avg=1501.60, stdev=1321.14, samples=109 lat (usec) : 100=0.01%, 250=0.03%, 500=1.85%, 750=19.92%, 1000=14.58% lat (msec) : 2=31.18%, 4=13.96%, 10=10.72%, 20=3.24%, 50=1.61% lat (msec) : 100=0.60%, 250=0.82%, 500=0.85%, 750=0.54%, 1000=0.08% lat (msec) : 2000=0.02% cpu : usr=0.87%, sys=2.54%, ctx=49581, majf=0, minf=33 IO depths : 1=0.4%, 2=1.0%, 4=3.1%, 8=52.1%, 16=43.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=1.6%, 16=2.9%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=81936,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=744: Mon Dec 18 13:41:58 2023 read: IOPS=1298, BW=81.1MiB/s (85.1MB/s)(4894MiB/60318msec) slat (nsec): min=149, max=19068k, avg=1503.55, stdev=72400.63 clat (usec): min=36, max=1249.6k, avg=12039.57, stdev=66626.21 lat (usec): min=103, max=1249.9k, avg=12041.08, stdev=66626.48 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 506], 10.00th=[ 553], | 20.00th=[ 635], 30.00th=[ 775], 40.00th=[ 955], | 50.00th=[ 1139], 60.00th=[ 1369], 70.00th=[ 1926], | 80.00th=[ 3359], 90.00th=[ 7963], 95.00th=[ 19006], | 99.00th=[ 417334], 99.50th=[ 574620], 99.90th=[ 742392], | 99.95th=[ 817890], 99.99th=[1249903] bw ( KiB/s): min= 126, max=324608, per=6.29%, avg=89749.58, stdev=87617.01, samples=111 iops : min= 1, max= 5072, avg=1402.19, stdev=1369.03, samples=111 lat (usec) : 50=0.01%, 250=0.02%, 500=4.56%, 750=24.06%, 1000=13.88% lat (msec) : 2=28.31%, 4=11.49%, 10=9.49%, 20=3.41%, 50=1.79% lat (msec) : 100=0.73%, 250=0.73%, 500=0.83%, 750=0.62%, 1000=0.08% lat (msec) : 2000=0.02% cpu : usr=0.90%, sys=2.32%, ctx=46487, majf=0, minf=34 IO depths : 1=0.4%, 2=0.9%, 4=2.8%, 8=51.9%, 16=44.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=2.1%, 16=2.5%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=78307,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=745: Mon Dec 18 13:41:58 2023 read: IOPS=1445, BW=90.3MiB/s (94.7MB/s)(5421MiB/60001msec) slat (nsec): min=150, max=50714k, avg=1906.54, stdev=173335.52 clat (usec): min=64, max=1233.5k, avg=10872.92, stdev=58552.56 lat (usec): min=71, max=1284.2k, avg=10874.83, stdev=58565.01 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 506], 10.00th=[ 553], | 20.00th=[ 668], 30.00th=[ 857], 40.00th=[ 1004], | 50.00th=[ 1205], 60.00th=[ 1516], 70.00th=[ 2376], | 80.00th=[ 3884], 90.00th=[ 8225], 95.00th=[ 17433], | 99.00th=[ 329253], 99.50th=[ 517997], 99.90th=[ 658506], | 99.95th=[ 725615], 99.99th=[1233126] bw ( KiB/s): min= 506, max=268288, per=7.05%, avg=100514.87, stdev=83674.25, samples=110 iops : min= 7, max= 4192, avg=1570.40, stdev=1307.39, samples=110 lat (usec) : 100=0.01%, 250=0.03%, 500=4.22%, 750=20.15%, 1000=15.26% lat (msec) : 2=26.87%, 4=14.00%, 10=11.18%, 20=3.64%, 50=1.79% lat (msec) : 100=0.58%, 250=0.94%, 500=0.81%, 750=0.52%, 2000=0.02% cpu : usr=0.94%, sys=2.58%, ctx=48631, majf=0, minf=34 IO depths : 1=0.5%, 2=1.2%, 4=3.6%, 8=52.3%, 16=42.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.4%, 8=1.7%, 16=2.9%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=86735,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=746: Mon Dec 18 13:41:58 2023 read: IOPS=1364, BW=85.2MiB/s (89.4MB/s)(5121MiB/60073msec) slat (nsec): min=150, max=9168.5k, avg=1443.43, stdev=37855.07 clat (usec): min=76, max=1208.6k, avg=11592.08, stdev=62724.94 lat (usec): min=107, max=1208.9k, avg=11593.52, stdev=62725.09 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 537], 10.00th=[ 586], | 20.00th=[ 685], 30.00th=[ 848], 40.00th=[ 1004], | 50.00th=[ 1188], 60.00th=[ 1532], 70.00th=[ 2343], | 80.00th=[ 3851], 90.00th=[ 8586], 95.00th=[ 19530], | 99.00th=[ 362808], 99.50th=[ 534774], 99.90th=[ 700449], | 99.95th=[ 851444], 99.99th=[1216349] bw ( KiB/s): min= 512, max=317700, per=6.44%, avg=91918.34, stdev=82593.63, samples=113 iops : min= 8, max= 4964, avg=1436.07, stdev=1290.54, samples=113 lat (usec) : 100=0.01%, 250=0.04%, 500=2.47%, 750=21.88%, 1000=15.11% lat (msec) : 2=27.07%, 4=14.24%, 10=10.39%, 20=3.92%, 50=1.75% lat (msec) : 100=0.84%, 250=0.94%, 500=0.70%, 750=0.57%, 1000=0.03% lat (msec) : 2000=0.04% cpu : usr=0.87%, sys=2.52%, ctx=47315, majf=0, minf=34 IO depths : 1=0.5%, 2=1.2%, 4=3.3%, 8=52.4%, 16=42.6%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.5%, 8=2.0%, 16=2.5%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=81940,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=747: Mon Dec 18 13:41:58 2023 read: IOPS=1499, BW=93.7MiB/s (98.3MB/s)(5633MiB/60087msec) slat (nsec): min=149, max=4115.4k, avg=1404.10, stdev=22280.21 clat (usec): min=38, max=1138.7k, avg=10466.63, stdev=57385.33 lat (usec): min=93, max=1138.7k, avg=10468.04, stdev=57385.47 clat percentiles (usec): | 1.00th=[ 461], 5.00th=[ 537], 10.00th=[ 586], | 20.00th=[ 701], 30.00th=[ 873], 40.00th=[ 1020], | 50.00th=[ 1205], 60.00th=[ 1549], 70.00th=[ 2343], | 80.00th=[ 3490], 90.00th=[ 6718], 95.00th=[ 15401], | 99.00th=[ 333448], 99.50th=[ 488637], 99.90th=[ 692061], | 99.95th=[ 767558], 99.99th=[1132463] bw ( KiB/s): min= 2048, max=282880, per=7.03%, avg=100229.03, stdev=87714.04, samples=114 iops : min= 32, max= 4420, avg=1565.91, stdev=1370.52, samples=114 lat (usec) : 50=0.01%, 100=0.01%, 250=0.02%, 500=2.16%, 750=20.98% lat (usec) : 1000=15.35% lat (msec) : 2=28.02%, 4=16.50%, 10=10.12%, 20=2.55%, 50=1.40% lat (msec) : 100=0.65%, 250=0.98%, 500=0.77%, 750=0.42%, 1000=0.05% lat (msec) : 2000=0.02% cpu : usr=1.05%, sys=2.66%, ctx=52629, majf=0, minf=34 IO depths : 1=0.5%, 2=1.2%, 4=3.3%, 8=52.0%, 16=43.1%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.6%, 8=1.9%, 16=2.5%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=90128,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 random-write: (groupid=0, jobs=1): err= 0: pid=748: Mon Dec 18 13:41:58 2023 read: IOPS=1407, BW=88.0MiB/s (92.3MB/s)(5279MiB/60001msec) slat (nsec): min=150, max=12194k, avg=1663.24, stdev=60254.86 clat (usec): min=49, max=1167.4k, avg=11180.47, stdev=59223.15 lat (usec): min=62, max=1167.4k, avg=11182.13, stdev=59224.71 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 523], 10.00th=[ 570], | 20.00th=[ 685], 30.00th=[ 889], 40.00th=[ 1057], | 50.00th=[ 1237], 60.00th=[ 1598], 70.00th=[ 2409], | 80.00th=[ 3785], 90.00th=[ 7767], 95.00th=[ 17695], | 99.00th=[ 333448], 99.50th=[ 497026], 99.90th=[ 708838], | 99.95th=[ 767558], 99.99th=[1166017] bw ( KiB/s): min= 638, max=428544, per=6.90%, avg=98424.71, stdev=84332.78, samples=109 iops : min= 9, max= 6696, avg=1537.70, stdev=1317.68, samples=109 lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=3.43%, 750=19.86% lat (usec) : 1000=13.83% lat (msec) : 2=28.73%, 4=15.04%, 10=11.07%, 20=3.31%, 50=1.67% lat (msec) : 100=0.71%, 250=0.89%, 500=0.94%, 750=0.42%, 1000=0.06% lat (msec) : 2000=0.02% cpu : usr=0.94%, sys=2.49%, ctx=51177, majf=0, minf=34 IO depths : 1=0.5%, 2=1.0%, 4=2.9%, 8=52.0%, 16=43.6%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=95.6%, 8=2.0%, 16=2.4%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=84468,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1393MiB/s (1461MB/s), 72.0MiB/s-93.7MiB/s (75.5MB/s-98.3MB/s), io=82.3GiB (88.3GB), run=60001-60468msec Disk stats (read/write): dm-7: ios=25499/81, merge=0/0, ticks=10347931/25926, in_queue=10373857, util=89.31%, aggrios=195125/81, aggrmerge=0/0, aggrticks=43903289/25925, aggrin_queue=43929214, aggrutil=99.97% dm-6: ios=195125/81, merge=0/0, ticks=43903289/25925, in_queue=43929214, util=99.97%, aggrios=195125/81, aggrmerge=0/0, aggrticks=43898958/25913, aggrin_queue=43924871, aggrutil=99.98% dm-4: ios=195125/81, merge=0/0, ticks=43898958/25913, in_queue=43924871, util=99.98%, aggrios=48781/24, aggrmerge=0/0, aggrticks=10974435/6587, aggrin_queue=10981022, aggrutil=99.98% dm-1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% dm-2: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% dm-0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00% drbd1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=195125/99, aggrmerge=0/0, aggrticks=43896026/26348, aggrin_queue=43922374, aggrutil=99.98% md1: ios=195125/99, merge=0/0, ticks=43896026/26348, in_queue=43922374, util=99.98%, aggrios=43179/74, aggrmerge=5625/13, aggrticks=9996652/4907, aggrin_queue=10004715, aggrutil=99.51% sde: ios=44779/55, merge=7746/0, ticks=4172863/2371, in_queue=4177399, util=88.04% sdc: ios=42519/85, merge=4647/23, ticks=18838007/6812, in_queue=18848953, util=99.51% sdb: ios=42894/75, merge=6046/3, ticks=4114319/3553, in_queue=4120104, util=88.32% sda: ios=42525/81, merge=4062/26, ticks=12861420/6894, in_queue=12872406, util=98.79% dm-3: ios=195125/99, merge=0/0, ticks=43897742/26348, in_queue=43924090, util=99.98%