2

I am having a problem with a Linux server that it is writing to much to disk, thus I am having a slow response time due to a big IO wait. I already check smart values for the disks and are OK. This is a two disk setup in RAID1 softwareRaid, ext4 filesystem.

Since I cannot upgrade hardware for the moment nor get rid of the intense I/O applications, I was planning to configure Linux vm setting in an attempt to easy the I/O wait time.

I am thinking in tuning swapping but mainly dirty_background_ratio and dirty_ratio.

Question:

How can I estimate the tuning of this values based on my current system load and memory usage?

Improve this question
3

You want few things. First you want to reduce swappiness

sysctl -w vm.swappiness=10

This will save some disk IO; because last thing you need is additional writes to disk when kernel tries to page out some stuff from mem. Goal is to tune things so as little swapping is needed. However don't turn off swappiness by setting it to 0 or disabling. I would recommend extreme action to go as far as setting swappiness to 1. If you observe dstat output for a while you will notice quick how much data actually gets written and read from swap.

Now there is mechanism in newer kernels (3.2+) called writeback throttling. To be able to use it as you said you need to tune dirty ratios. Check for more details this link. Quote from there that interests you is

Once dirty_ratio (resp. dirty_bytes) limit is hit then the process which 
writes gets throttled.

So by defaults dirties are rather high especially if you have a lot of memory and slow disk subsystem. So you need to tune them down; as low as possible not to affect normal use* and yet value will dictate volume of data that will exist in memory before kernel spawns up processes to write that out to disk, when your disk IO bottleneck situation starts to occur. At that point you want that process to be throttled, which kernel does by injecting sleeps in it.

*to figure out what is normal use; I'd recommend to install atop and observe what is going on there; you want to check figures of dirty there and see the D overview where disk read/writes are tracked. There is column WCANCL; these are actually writes that were handled in memory and never were needed to be written to disk (dirty pages) but for some temporary data. Mysql has those when it does complex queries, compiler when making bunch of small obj files which wont be needed for long etc...

Other than that it may help to switch to deadline disk scheduler and to adjust affinity of reads vs writes to suite your environment better. e.g. if you do 10x more reads than writes you may want to set

/sys/block/<device>/queue/iosched/writes_starved

to 5 rather than default 2. Setting higher

/sys/block/<device>/queue/iosched/write_expire

will also help. Additionally you can gain some latency if you decrease number of requests done in batch from 128 to say 32

/sys/block/<device>/queue/nr_requests

Improve this answer
2
  • Interesting. I was testing with higher dirty_background and dirty_ratio values and for some time I saw what it looked like a tiny improvement in performance. My system is mainly writing, I think reads are not an issue. Regarding /sys/block//queue, since I am using SoftwareRAID, should I apply the setting via md devices or point to the real hardware? Is it possible that ext4 + SoftwareRAID are causing some trouble? I am seeing [jdb2/mdx] and [mdx_raid1] at iotop output displaying 70-90% IO. – Matías Nov 3 '14 at 17:37
  • 1
    You saw tiny improvements because with higher dirty thresholds everything was handled in RAM, but if operations size excedes dirty tresholds; and it wasn't some temporary write it will eventually need to be written to disk. At this point kernel starts offloading dirty to disk and your disk becomes bottleneck. To avoid it you need to tune dirty ratio down to make use of kernel throttling I wrote about. Without that you lose all the latency. Regarding changes to block queue, applying settings to md should be fine. Also consider removing barrier if you're using ext4 (mount option nobarrier) – Hrvoje Špoljar Nov 3 '14 at 18:09
2

If you have heavy writes, you'll be constrained by not having a write caching layer available to your system. Two disks and software RAID make that difficult. Usually, this is a feature of hardware RAID. What you have now is not the right hardware configuration for your workload.

In order to provide better answers, we would need specifics about what your application is doing, the OS, whether write barriers are enabled on your filesystem, etc.

Edit: You can only tune so far if your foundation is bad. Maybe you should consider SSDs instead of spinning disks for this purpose.

Improve this answer
3
  • For the moment I cannot upgrade hardware. This is a modern Linux box, and the I/O is produced by intense mysqld, mongodb and jenkins stuff. – Matías Nov 3 '14 at 15:08
  • But if you have a high write load, you should be using some form of write caching and/or more disks (spindles) or SSDs... right? – ewwhite Nov 23 '14 at 13:06
  • Yes, I just wanted to improve performance until reaching the hardware upgrade :) – Matías Nov 24 '14 at 15:10
0

I wrote about dirty_background_ratio and dirty_ratio etc in my blog recently:

http://models.street-artists.org/2016/10/09/nfs-syncasync-some-of-the-issue-solved-or-how-to-set-vm-dirty_bytes-and-vm-dirty_background_bytes/

The short version is to not use the *_ratio variables, but instead use the *_bytes version, and estimate the number of bytes by taking the bandwidth (or data generation rate) and multiplying by a maximum latency you are willing to have before large writes start to hit the disk.

By setting the dirty_background_bytes value relatively low (less than a second of delay at full data receive/generate rate) you'll ensure that the buffer doesn't build up while no-one's doing anything. Setting the dirty_bytes a factor of 2 or 3 higher (at least, maybe up to 10x or more, depending on your amount of RAM) will ensure that a bursty process doesn't get throttled. You could estimate the dirty_bytes value by considering the difference between data generation rate and the disk write speed. This is a buffer fill rate, and you can then multiply by a maximum buffering time before the fill gets throttled. So for example if you generate data at rate Rg and write to disk at rate Rd, and Rg is bigger than Rd, you can set dirty_background_bytes to Rg*(0.5 seconds) so that your disk starts writing about 0.5 seconds after you start slamming data into the buffers, and then set dirty_bytes to max(2*Rg*0.5, (Rg-Rd)*(2 seconds)) for example. Bursty processes will be able to write for up to 2 seconds before the buffers get big enough that they get throttled.

Improve this answer
1
  • actually, set dirty_bytes to something like Rg*0.5 + (Rg-Rd)*2, so in other words, it will allow the buffer to fill to the point where background kicks in, and then another 2 seconds after that until it throttles your write process. – dlakelan Oct 11 '16 at 17:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.