Исправление Harliff, (текущая версия) :
И попробуй выключить кэш в ssd.
Выключил (sdparm -s WCE=0,RCD=0 /dev/sdb). Вот результаты:
# fio rw-test-zfs3.fio
readtest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8
writetest: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8
fio-2.16
Starting 2 processes
Jobs: 1 (f=1): [_(1),f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:00s]
readtest: (groupid=0, jobs=1): err= 0: pid=2673: Mon May 6 01:09:53 2019
read : io=102400MB, bw=350484KB/s, iops=87621, runt=299179msec
slat (usec): min=2, max=173091, avg= 4.03, stdev=34.01
clat (usec): min=1, max=207923, avg=86.17, stdev=984.85
lat (usec): min=7, max=207926, avg=90.32, stdev=985.49
clat percentiles (usec):
| 1.00th=[ 14], 5.00th=[ 18], 10.00th=[ 21], 20.00th=[ 25],
| 30.00th=[ 29], 40.00th=[ 36], 50.00th=[ 52], 60.00th=[ 64],
| 70.00th=[ 76], 80.00th=[ 109], 90.00th=[ 173], 95.00th=[ 225],
| 99.00th=[ 362], 99.50th=[ 474], 99.90th=[ 1112], 99.95th=[ 1688],
| 99.99th=[ 6304]
bw (KB /s): min= 19, max=4096000, per=28.88%, avg=101225.98, stdev=68526.22
lat (usec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=7.23%, 50=41.41%
lat (usec) : 100=29.42%, 250=18.36%, 500=3.12%, 750=0.25%, 1000=0.08%
lat (msec) : 2=0.08%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=14.10%, sys=42.36%, ctx=4123857, majf=0, minf=768164
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=26214400/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=8
writetest: (groupid=0, jobs=1): err= 0: pid=2674: Mon May 6 01:09:53 2019
write: io=102400MB, bw=44262KB/s, iops=11065, runt=2369011msec
slat (usec): min=1, max=8735, avg= 4.56, stdev= 5.81
clat (usec): min=0, max=1915.7K, avg=716.90, stdev=19824.25
lat (usec): min=13, max=1915.7K, avg=721.62, stdev=19824.25
clat percentiles (usec):
| 1.00th=[ 19], 5.00th=[ 24], 10.00th=[ 27], 20.00th=[ 36],
| 30.00th=[ 68], 40.00th=[ 95], 50.00th=[ 195], 60.00th=[ 322],
| 70.00th=[ 462], 80.00th=[ 620], 90.00th=[ 820], 95.00th=[ 972],
| 99.00th=[ 1336], 99.50th=[ 1656], 99.90th=[ 5472], 99.95th=[103936],
| 99.99th=[1187840]
bw (KB /s): min= 0, max=4096000, per=100.00%, avg=52425.41, stdev=59253.39
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=1.26%, 50=24.92%
lat (usec) : 100=14.45%, 250=13.78%, 500=18.23%, 750=14.36%, 1000=8.57%
lat (msec) : 2=4.10%, 4=0.19%, 10=0.08%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.02%
cpu : usr=2.66%, sys=6.89%, ctx=11914743, majf=0, minf=768158
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=26214400/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: io=102400MB, aggrb=350484KB/s, minb=350484KB/s, maxb=350484KB/s, mint=299179msec, maxt=299179msec
WRITE: io=102400MB, aggrb=44262KB/s, minb=44262KB/s, maxb=44262KB/s, mint=2369011msec, maxt=2369011msec
fio rw-test-zfs3.fio 174.37s user 291.28s system 19% cpu 40:00.85 total
Исходная версия Harliff, :
И попробуй выключить кэш в ssd.
Выключил (sdparm -s WCE=0,RCD=0 /dev/sdb). Вот результаты:
root@prox ~/fio/rw-test-depth-8 # fio rw-test-zfs3.fio
readtest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8
writetest: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8
fio-2.16
Starting 2 processes
Jobs: 1 (f=1): [_(1),f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:00s]
readtest: (groupid=0, jobs=1): err= 0: pid=2673: Mon May 6 01:09:53 2019
read : io=102400MB, bw=350484KB/s, iops=87621, runt=299179msec
slat (usec): min=2, max=173091, avg= 4.03, stdev=34.01
clat (usec): min=1, max=207923, avg=86.17, stdev=984.85
lat (usec): min=7, max=207926, avg=90.32, stdev=985.49
clat percentiles (usec):
| 1.00th=[ 14], 5.00th=[ 18], 10.00th=[ 21], 20.00th=[ 25],
| 30.00th=[ 29], 40.00th=[ 36], 50.00th=[ 52], 60.00th=[ 64],
| 70.00th=[ 76], 80.00th=[ 109], 90.00th=[ 173], 95.00th=[ 225],
| 99.00th=[ 362], 99.50th=[ 474], 99.90th=[ 1112], 99.95th=[ 1688],
| 99.99th=[ 6304]
bw (KB /s): min= 19, max=4096000, per=28.88%, avg=101225.98, stdev=68526.22
lat (usec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=7.23%, 50=41.41%
lat (usec) : 100=29.42%, 250=18.36%, 500=3.12%, 750=0.25%, 1000=0.08%
lat (msec) : 2=0.08%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=14.10%, sys=42.36%, ctx=4123857, majf=0, minf=768164
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=26214400/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=8
writetest: (groupid=0, jobs=1): err= 0: pid=2674: Mon May 6 01:09:53 2019
write: io=102400MB, bw=44262KB/s, iops=11065, runt=2369011msec
slat (usec): min=1, max=8735, avg= 4.56, stdev= 5.81
clat (usec): min=0, max=1915.7K, avg=716.90, stdev=19824.25
lat (usec): min=13, max=1915.7K, avg=721.62, stdev=19824.25
clat percentiles (usec):
| 1.00th=[ 19], 5.00th=[ 24], 10.00th=[ 27], 20.00th=[ 36],
| 30.00th=[ 68], 40.00th=[ 95], 50.00th=[ 195], 60.00th=[ 322],
| 70.00th=[ 462], 80.00th=[ 620], 90.00th=[ 820], 95.00th=[ 972],
| 99.00th=[ 1336], 99.50th=[ 1656], 99.90th=[ 5472], 99.95th=[103936],
| 99.99th=[1187840]
bw (KB /s): min= 0, max=4096000, per=100.00%, avg=52425.41, stdev=59253.39
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=1.26%, 50=24.92%
lat (usec) : 100=14.45%, 250=13.78%, 500=18.23%, 750=14.36%, 1000=8.57%
lat (msec) : 2=4.10%, 4=0.19%, 10=0.08%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.02%
cpu : usr=2.66%, sys=6.89%, ctx=11914743, majf=0, minf=768158
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=26214400/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: io=102400MB, aggrb=350484KB/s, minb=350484KB/s, maxb=350484KB/s, mint=299179msec, maxt=299179msec
WRITE: io=102400MB, aggrb=44262KB/s, minb=44262KB/s, maxb=44262KB/s, mint=2369011msec, maxt=2369011msec
fio rw-test-zfs3.fio 174.37s user 291.28s system 19% cpu 40:00.85 total