История изменений
Исправление Harliff, (текущая версия) :
primarycache=metadata
Попробуй вернуть в all
Вернул. Чтение улучшилось в разы (и обогнало LVM). С записью, правда, проблемы остаются. Позже попробую с logbias=latency...
# fio rw-test-zfs3.fio
readtest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8
writetest: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8
fio-2.16
Starting 2 processes
Jobs: 1 (f=1): [_(1),f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:00s]
readtest: (groupid=0, jobs=1): err= 0: pid=1205: Sun May 5 20:46:23 2019
read : io=102400MB, bw=181496KB/s, iops=45374, runt=577739msec
slat (usec): min=1, max=21658, avg= 4.32, stdev= 6.57
clat (usec): min=0, max=425349, avg=171.06, stdev=1206.24
lat (usec): min=7, max=425353, avg=175.45, stdev=1206.28
clat percentiles (usec):
| 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 18], 20.00th=[ 24],
| 30.00th=[ 32], 40.00th=[ 49], 50.00th=[ 62], 60.00th=[ 74],
| 70.00th=[ 101], 80.00th=[ 151], 90.00th=[ 225], 95.00th=[ 314],
| 99.00th=[ 3088], 99.50th=[ 4016], 99.90th=[ 5856], 99.95th=[ 6624],
| 99.99th=[76288]
lat (usec) : 2=0.01%, 4=0.01%, 10=0.07%, 20=12.92%, 50=27.23%
lat (usec) : 100=29.51%, 250=22.29%, 500=4.85%, 750=0.44%, 1000=0.20%
lat (msec) : 2=0.73%, 4=1.24%, 10=0.49%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=7.29%, sys=25.53%, ctx=8553085, majf=1, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=26214400/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=8
writetest: (groupid=0, jobs=1): err= 0: pid=1206: Sun May 5 20:46:23 2019
write: io=102400MB, bw=49774KB/s, iops=12443, runt=2106671msec
slat (usec): min=1, max=4143, avg= 4.63, stdev= 5.66
clat (usec): min=0, max=2305.5K, avg=637.18, stdev=18033.53
lat (usec): min=13, max=2305.6K, avg=641.89, stdev=18033.52
clat percentiles (usec):
| 1.00th=[ 19], 5.00th=[ 23], 10.00th=[ 27], 20.00th=[ 34],
| 30.00th=[ 63], 40.00th=[ 84], 50.00th=[ 171], 60.00th=[ 290],
| 70.00th=[ 410], 80.00th=[ 548], 90.00th=[ 732], 95.00th=[ 892],
| 99.00th=[ 1272], 99.50th=[ 1672], 99.90th=[ 5792], 99.95th=[101888],
| 99.99th=[1089536]
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=1.55%, 50=26.46%
lat (usec) : 100=15.72%, 250=12.79%, 500=20.42%, 750=13.68%, 1000=6.41%
lat (msec) : 2=2.58%, 4=0.23%, 10=0.10%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=2.51%, sys=7.49%, ctx=10960464, majf=9, minf=8
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=26214400/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
READ: io=102400MB, aggrb=181496KB/s, minb=181496KB/s, maxb=181496KB/s, mint=577739msec, maxt=577739msec
WRITE: io=102400MB, aggrb=49774KB/s, minb=49774KB/s, maxb=49774KB/s, mint=2106671msec, maxt=2106671msec
fio rw-test-zfs3.fio 97.97s user 308.28s system 19% cpu 35:12.95 total
Исходная версия Harliff, :
primarycache=metadata
Попробуй вернуть в all
Вернул. Чтение улучшилось в разы (и обогнало LVM). С записью, правда, проблемы остаются. Позже попробую с logbias=latency...
# fio rw-test-zfs3.fio readtest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 writetest: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 fio-2.16 Starting 2 processes Jobs: 1 (f=1): [_(1),f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:00s] readtest: (groupid=0, jobs=1): err= 0: pid=1205: Sun May 5 20:46:23 2019 read : io=102400MB, bw=181496KB/s, iops=45374, runt=577739msec slat (usec): min=1, max=21658, avg= 4.32, stdev= 6.57 clat (usec): min=0, max=425349, avg=171.06, stdev=1206.24 lat (usec): min=7, max=425353, avg=175.45, stdev=1206.28 clat percentiles (usec): | 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 18], 20.00th=[ 24], | 30.00th=[ 32], 40.00th=[ 49], 50.00th=[ 62], 60.00th=[ 74], | 70.00th=[ 101], 80.00th=[ 151], 90.00th=[ 225], 95.00th=[ 314], | 99.00th=[ 3088], 99.50th=[ 4016], 99.90th=[ 5856], 99.95th=[ 6624], | 99.99th=[76288] lat (usec) : 2=0.01%, 4=0.01%, 10=0.07%, 20=12.92%, 50=27.23% lat (usec) : 100=29.51%, 250=22.29%, 500=4.85%, 750=0.44%, 1000=0.20% lat (msec) : 2=0.73%, 4=1.24%, 10=0.49%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01% cpu : usr=7.29%, sys=25.53%, ctx=8553085, majf=1, minf=25 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=26214400/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8 writetest: (groupid=0, jobs=1): err= 0: pid=1206: Sun May 5 20:46:23 2019 write: io=102400MB, bw=49774KB/s, iops=12443, runt=2106671msec slat (usec): min=1, max=4143, avg= 4.63, stdev= 5.66 clat (usec): min=0, max=2305.5K, avg=637.18, stdev=18033.53 lat (usec): min=13, max=2305.6K, avg=641.89, stdev=18033.52 clat percentiles (usec): | 1.00th=[ 19], 5.00th=[ 23], 10.00th=[ 27], 20.00th=[ 34], | 30.00th=[ 63], 40.00th=[ 84], 50.00th=[ 171], 60.00th=[ 290], | 70.00th=[ 410], 80.00th=[ 548], 90.00th=[ 732], 95.00th=[ 892], | 99.00th=[ 1272], 99.50th=[ 1672], 99.90th=[ 5792], 99.95th=[101888], | 99.99th=[1089536] lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=1.55%, 50=26.46% lat (usec) : 100=15.72%, 250=12.79%, 500=20.42%, 750=13.68%, 1000=6.41% lat (msec) : 2=2.58%, 4=0.23%, 10=0.10%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.01% cpu : usr=2.51%, sys=7.49%, ctx=10960464, majf=9, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=26214400/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs): READ: io=102400MB, aggrb=181496KB/s, minb=181496KB/s, maxb=181496KB/s, mint=577739msec, maxt=577739msec WRITE: io=102400MB, aggrb=49774KB/s, minb=49774KB/s, maxb=49774KB/s, mint=2106671msec, maxt=2106671msec fio rw-test-zfs3.fio 97.97s user 308.28s system 19% cpu 35:12.95 total