Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've seen this question: How to identify heavy write to disk?

And I've used dstat and atop before...but they don't seem to pinpoint what process is causing disk I/O. For example, from dstat:

dstat -ta --top-bio
----system---- ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- ----most-expensive----
     time     |usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw |  block i/o process
14-12 16:16:25| 22   3  49  26   0   0|2324k    0 |  17k 6144B|   0     0 |1324     0 |
14-12 16:16:26| 24   3  30  43   0   0|4960k 8192B|1498B 4322B|   0     0 |1494     0 |wget          0  4096B
14-12 16:16:27| 25   4  38  33   0   0|4612k  548k|5011B   27k|   0     0 |1582     0 |kjournald     0    24k
14-12 16:16:28| 23   3  42  32   0   0|5072k    0 |  24k 4368B|   0     0 |1495     0 |

Notice how high dsk/total is -- between 2 and 5 MB/sec. But then look at 'most expensive' column -- it's only a couple bytes here, a couple KB there, and sometimes even nothing. It's the same sort of thing with 'atop'. Shows high overall disk usage, but low usage from individual processes. I'm running CentOS 5, kernel 2.6.18-53.

Do I need a newer kernel version? Maybe some system config setting somewhere? The 'atop' homepage recommends installing some kernel patches, but I'd rather not go through the hassle of configuring & compiling my own kernel.

share|improve this question
add comment

1 Answer

up vote 21 down vote accepted

iotop (link) for starter ;) I haven't seen you posting an output of it.

1: I have experienced almost the same situation with a logging filesystem and atime - however with more writes.

Try to remount with noatime and turn off filesystem logging (later for testing only) in order to see if it's filesystem based and as said, iotop if it's process based.

2: I guess this partition is not part of a just-rebuilding raid array, is it?

3: If you are having a lot of very small files (a lot smaller than the actual block device blocksize and/or the filesystem blocksize), and you are reading those small files, you end up reading entire blocks from the system, and most of those blocks will be read for nothing.

4: If nothing helps above, you can always get the list of files accessed by executing

echo 1 > /proc/sys/vm/block_dump

please note that it degrades system performance by a lot. Instructions are available in my previous post here

share|improve this answer
1  
Beat me with about 5 seconds ;) +1 –  pehrs Dec 14 '10 at 22:35
    
I just got lucky, but almost immediately extended the answer, as it was not exactly an answer for the question ;) –  asdmin Dec 14 '10 at 22:37
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.