iostat + how to solve latency on disks

we use iostat -x and check for the await column - per device it shows the total time spent waiting plus the actual handling of the request by the disk

iostat -yzx 5
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdm 0.00 0.00 0.00 1.60 0.00 4.20 5.25 0.00 0.75 0.00 0.75 0.62 0.10
sda 0.00 0.40 0.00 3.00 0.00 24.70 16.47 0.11 37.60 0.00 37.60 7.13 2.14
sdc 0.00 0.00 0.00 0.80 0.00 6.10 15.25 0.01 12.75 0.00 12.75 12.75 1.02
sdb 0.00 0.00 0.00 0.40 0.00 2.40 12.00 0.01 19.50 0.00 19.50 19.50 0.78
sdd 0.00 0.00 0.00 0.80 0.00 6.60 16.50 0.01 10.75 0.00 10.75 10.75 0.86
sdg 0.00 0.00 0.00 1.00 0.00 8.90 17.80 0.02 15.80 0.00 15.80 15.80 1.58
sdf 0.00 0.20 0.00 1.00 0.00 8.00 16.00 0.01 12.80 0.00 82.80 12.80 1.28
sde 0.00 0.00 0.00 0.80 0.00 7.50 18.75 0.01 11.25 0.00 11.25 11.25 0.90
sdk 0.00 0.00 0.00 0.80 0.00 6.50 16.25 0.01 14.25 0.00 14.25 14.25 1.14
sdi 0.00 0.00 0.00 1.00 0.00 7.20 14.40 0.01 11.60 0.00 11.60 11.60 1.16
sdj 0.00 0.00 0.00 0.80 0.00 5.90 14.75 0.01 13.50 0.00 13.50 13.50 1.08
sdh 0.00 0.00 0.00 0.80 0.00 5.60 14.00 0.01 13.25 0.00 13.25 13.25 1.06
sdl 0.00 0.00 0.00 1.00 0.00 7.20 14.40 0.01 14.20 0.00 14.20 14.20 1.42
sdn 0.00 0.00 0.00 1.60 0.00 4.20 5.25 0.00 0.88 0.00 0.88 0.75 0.12
md1 0.00 0.00 0.00 0.40 0.00 3.20 16.00
sdq 0.00 0.00 0.00 0.60 0.00 5.20 17.33 0.01 12.67 0.00 82.67 12.67 0.76
sdr 0.00 0.00 0.00 0.60 0.00 4.20 14.00 0.01 10.00 0.00 10.00 10.00 0.60

we can see from the output that await on some disk is high

my question - what we can do about this ?

any option to resolve this ?

1 Answer

Your service times on the relevant disks are around 15ms - this is quite a normal value for spinning rust. Since you have only a very limited amount of outstanding I/O and nearly no reads, you are random write bound. Easiest way to resolve that is using bcache dm-cache or whatever with a decent SSD.