Linux IO monitoring per file?

Question

I am interested in a utility or process for monitoring disk IO per file on CentOS.

On Win2008, the resmon utility allows this type of drilldown, but none of the Linux utilities I have found do this (iostat, iotop, dstat, nmon).

My interest in monitoring IO bottlenecks on database servers. With MSSQL, I have found it an informative diagnostic to know which files / filespaces are getting hit the hardest.

Answer 1

Systemtap is probably your best option:

http://sourceware.org/systemtap/

Here is an example here:

http://sourceware.org/systemtap/SystemTap_Beginners_Guide/iotimesect.html

825946 3364 (NetworkManager) access /sys/class/net/eth0/carrier read: 8190 write: 0
825955 3364 (NetworkManager) iotime /sys/class/net/eth0/carrier time: 9
[...]
117061 2460 (pcscd) access /dev/bus/usb/003/001 read: 43 write: 0
117065 2460 (pcscd) iotime /dev/bus/usb/003/001 time: 7
[...]
3973737 2886 (sendmail) access /proc/loadavg read: 4096 write: 0
3973744 2886 (sendmail) iotime /proc/loadavg time: 11

The disadvantage (aside from the learning curve) is that you will need to install kernel-debug, which may not be possible on a production. However, you can setup cross-instrumentation where you compile a module from the host system and use that instead with your system tap scripts:

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/SystemTap_Beginners_Guide/cross-compiling.html

Or if you are impatient, here are the system tap scripts you can use:

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/SystemTap_Beginners_Guide/useful-systemtap-scripts.html

Answer 2

The only tool I know of that can monitor I/O activity by file is inotifywatch. It's part of the inotify-tools package. Unfortunately, it only gives you operation counts.

Answer 3

You can monitor i/o per block device (via /proc/diskstats) and per process (io accounting via /proc/$PID/io or taskstats), but I don't know of a way to do it per-file.

Answer 4

I'd argue you might have asked the wrong question. if you're looking for i/o bottlenecks, it may be just as important to see what's happening on your disk. db's are notorious for doing random i/o which can significantly reduce throughput, especially if you only have a few spindles.

what may be more interesting is to see if you're having long wait times on the disks themselves. you can do this with collectl via the command "collectl -sD", which will show individual disk performance stats. Are --home to turn it into a top-like utility. If there are lots of disks involved, run it via colmux: colmux -command "-sD" and it will let you sort by a column of your choice, even across multiple systems.

Answer 5

I think that iotop is one of the best tool on Linux for identify bottlenecks on IO.

Answer 6

I recently was tinkering with collectl , it looks a great tool, and pretty straigforward to install. The most interesting is that you can find out which is the responsible process for IO bottlenecks. I would recommend you to read Using Collectl , it might be useful.

Answer 7

I would recommend you to check http://dag.wieers.com/home-made/dstat/. This great tool allows to check a lot of stats.