High disk I/O activity in CentOS server

Question

I have about 16 websites in a CentOS dedicated, and I am having some problems on high traffic hours, it seems to be a high disk I/O activity causing a general slowdown.

I've installed atop and this is what I see on the bottom (the server has been restarted thats why the values are so low):

*** system and process activity since boot ***

PID         RDDSK       WRDSK       WCANCL      DSK     CMD        1/18
2176        1.7G        7.3G        854.4M      39      mysqld
671         1248K       3.0G        0K          13      flush-8:0
566         0K          1.1G        0K          5       jbd2/sda2-8
2401        124.2M      529.1M      22408K      3       crond
2032        2.2G        502.0M      0K          12      nginx
2360        425.8M      115.3M      4188K       2       httpd

flush-8:0 and jbd2/sda2-8 are the processes I see with iotop using 99% on the IO> column, and they are the processes that write the most on the hdd (after mysql). From what I saw in google this could be caused by some ext4 related bug, the current kernel is:

Linux srvr.com 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011 x86_64 x86_64 x86_64 GNU/Linux

I asked the hosting support to update the kernel and they tried but they now say that the server wont boot with the new installed kernel and they had to go back to the previous, they are not helping very much.

Does someone has any idea how could I solve the high disk usage caused by flush-8:0 and jbd2/sda2-8 processes?

Answer 1

You haven't given us enough info for me to tell, but my first hunch would be that you are overusing RAM, and churning swap. with atop, show us the memory listing (press m), including the header section.

Don't blame your support. Your kernel and ext4 are very unlikely to be the problem.

Answer 2

flush-8:0 is kernel thread, works on flushing dirty RAM buffer and/or swap to disk, jbd2/sda2-8 is kernel thread, that is working with journaling on filesystems - they're intensive IO are results, not reasons of your problem.

more looks like your problem is heavy usage of disk by mysql. while JBD is in use - looks like multiple write operations are in progress.

check which files are changed by mysql or check it's logs, DB usage, table optimizations and if mysql daemons are configured to use as much RAM as possible for caching.

you can use mysqltuner for generic recommendations.

if writes will remain, probably, you need to increase disk speeds for more IOPS.

check which files are more often written. not only MB/s rate matters, you might write 20 MB but consume more IO operations than others get at 200MB.

Answer 3

Apparently mysql doesn't like ext4. Upgrading the kernel or changing mysql data directory to ext3 partition should solve this issue. http://ubuntuforums.org/showthread.php?t=1313834&page=2