You want few things. First you want to reduce swappiness
sysctl -w vm.swappiness=10
This will save some disk IO; because last thing you need is additional writes to disk when kernel tries to page out some stuff from mem. Goal is to tune things so as little swapping is needed. However don't turn off swappiness by setting it to 0 or disabling. I would recommend extreme action to go as far as setting swappiness to 1. If you observe dstat output for a while you will notice quick how much data actually gets written and read from swap.
Now there is mechanism in newer kernels (3.2+) called writeback throttling. To be able to use it as you said you need to tune dirty ratios. Check for more details
this link. Quote from there that interests you is
Once dirty_ratio (resp. dirty_bytes) limit is hit then the process which
writes gets throttled.
So by defaults dirties are rather high especially if you have a lot of memory and slow disk subsystem. So you need to tune them down; as low as possible not to affect normal use* and yet value will dictate volume of data that will exist in memory before kernel spawns up processes to write that out to disk, when your disk IO bottleneck situation starts to occur. At that point you want that process to be throttled, which kernel does by injecting sleeps in it.
*to figure out what is normal use; I'd recommend to install atop and observe what is going on there; you want to check figures of
dirty there and see the D overview where disk read/writes are tracked. There is column WCANCL; these are actually writes that were handled in memory and never were needed to be written to disk (dirty pages) but for some temporary data. Mysql has those when it does complex queries, compiler when making bunch of small obj files which wont be needed for long etc...
Other than that it may help to switch to deadline disk scheduler and to adjust affinity of reads vs writes to suite your environment better. e.g. if you do 10x more reads than writes you may want to set
to 5 rather than default 2. Setting higher
will also help. Additionally you can gain some latency if you decrease number of
requests done in batch from
128 to say