Determine what's causing high disk IO

Question

I'm having a problem with my VPS and disk IO. My server is running nginx + PHP-FPM + APC. The database is located on another dedicated VPS. I have several Wordpress MU sites living on the web server. The average IO rate is 6k block/sec.

I'm trying to understand what's causing the high IO.

Output of 'free -m':

            total   used   free   shared   buffers   cached
Mem:         1005    973     31        0        96      568
-/+ buffers/cache:   307    697
Swap:         255      8    247

Output of 'iotop':

Total DISK READ: 0.00 B/s | Total DISK WRITE: 3.90 M/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 2150 be/4 root        0.00 B/s    0.00 B/s  0.00 % 65.25 % [flush-202:0]
 6694 be/4 www-data    0.00 B/s   19.64 K/s  0.00 %  0.00 % php-fpm: pool www
 6700 be/4 www-data    0.00 B/s   23.56 K/s  0.00 %  0.00 % php-fpm: pool www
 8646 be/4 www-data    0.00 B/s  424.12 K/s  0.00 %  0.00 % php-fpm: pool www
10974 be/4 www-data    0.00 B/s   19.64 K/s  0.00 %  0.00 % php-fpm: pool www

The 'flush-202:0' process sometimes hits IO of 99%. I've read this is the disk cache flushing process but don't really understand what causes it to run/how to fix it.

Any help would be appreciated.

Thanks

Answer 1

I'm not sure that iotop sample shows something unusual. It's not a problem for the flush process to be a high percentage of your i/o at any point in time if there isn't much i/o going on at that time. I would install atop, which can present real-time data like iotop but has the advantage of also logging samples throughout the day. A day after installing it, I would open the logged data with atop -r log_filename, then go through the samples with t until I found times when the i/o reported in the system-level output is high. Then I would switch the per- process output to disk with d to see what processes were generating the i/o activity.

Answer 2

It sounds like your answer can be found here:

limit linux background flush (dirty pages)

Answer 3

You can acheive this with the pidstat program. Some distros dont come with it installed. But you can download the sysstate package from here and compile it. Dont install it but copy out the pidstat it compiles (or just run it in the current directory). You can pass the '-d' flag to get the output you want.

Answer 4

Ok, so what filesystem are you using?

I think this is caused by using too much data in buffers, looks like [flush] is initiated by "sync" command. Try running "sync" manually - does it hit 99% of I/O?

What hardware are you running on? Is this Hardware RAID card with it's own cache?

Ahh.. VPS.. You probably don't know :( Let's do some search: http://linux.livejournal.com/1707762.html?style=mine

Issue vmstat -d: what is it saying?

Answer 5

Why Your swap is completly full? It's causing heavy I/O load, shold be no swap used. And low memory usage? How this is possible with so high swap usage.. I'm searching answer to my problem when high I/O causes timeouts of the server. "Total DISK WRITE: 3.90 M/s" - it's saying low disk usage, but it's probably RANDOM write.