Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am interested in a utility or process for monitoring disk IO per file on CentOS.

On Win2008, the resmon utility allows this type of drilldown, but none of the Linux utilities I have found do this (iostat, iotop, dstat, nmon).

My interest in monitoring IO bottlenecks on database servers. With MSSQL, I have found it an informative diagnostic to know which files / filespaces are getting hit the hardest.

share|improve this question
2  
Maybe Linux Performance Analysis and Tools: Brendan Gregg's Talk at SCaLE 11x can help you; see slide 72/115. –  Cristian Ciupitu Oct 3 '13 at 2:14
1  
If this is possible, note most files are mapped into pagecache so your numbers could be all over the place depending on what bytes are in pagecache and what are on disk. –  Matthew Ife Oct 10 '13 at 18:07
    
@Matt But with a working answer! –  ewwhite Oct 17 '13 at 22:29
add comment

13 Answers

up vote 16 down vote accepted

Systemtap is probably your best option:

http://sourceware.org/systemtap/

Here is an example here:

http://sourceware.org/systemtap/SystemTap_Beginners_Guide/iotimesect.html

825946 3364 (NetworkManager) access /sys/class/net/eth0/carrier read: 8190 write: 0
825955 3364 (NetworkManager) iotime /sys/class/net/eth0/carrier time: 9
[...]
117061 2460 (pcscd) access /dev/bus/usb/003/001 read: 43 write: 0
117065 2460 (pcscd) iotime /dev/bus/usb/003/001 time: 7
[...]
3973737 2886 (sendmail) access /proc/loadavg read: 4096 write: 0
3973744 2886 (sendmail) iotime /proc/loadavg time: 11

The disadvantage (aside from the learning curve) is that you will need to install kernel-debug, which may not be possible on a production. However, you can setup cross-instrumentation where you compile a module from the host system and use that instead with your system tap scripts:

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/SystemTap_Beginners_Guide/cross-compiling.html

Or if you are impatient, here are the system tap scripts you can use:

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/SystemTap_Beginners_Guide/useful-systemtap-scripts.html

share|improve this answer
add comment

SystemTap script:

#!/usr/bin/env stap
#
# Monitor the I/O of a program.
#
# Usage:
#   ./monitor-io.stp name-of-the-program

@define program_name %( @1 %)

probe begin {
  printf("%5s %1s %6s %7s %s\n",
         "PID", "D", "BYTES", "us", "FILE")
}

probe vfs.read.return, vfs.write.return {
  # skip other programs
  if (execname() != @program_name)
    next

  if (devname=="N/A")
    next

  time_delta = gettimeofday_us() - @entry(gettimeofday_us())
  direction = name == "vfs.read" ? "R" : "W"

  // XXX task_dentry_path doesn't work with systemtap-2.3-1.fc19.x86_64
  // and the Fedora 19 kernel, but it works with systemtap-1.8-7.el6.x86_64
  // and the RHEL6 kernel (on the other hand @define is unsupported, so the
  // script needs to be changed a bit)
  //
  // if you want the full path, use
  // filename = task_dentry_path(task_current(),
  //                             $file->f_path->dentry,
  //                             $file->f_path->mnt)
  // if you only want the filename, use
  // filename = kernel_string($file->f_path->dentry->d_name->name)
  // if you only want the path from the mountpoint, use
  filename = devname . "," . reverse_path_walk($file->f_path->dentry)

  printf("%5d %1s %6d %7d %s\n",
         pid(), direction, $return, time_delta, filename)
}

The output looks like this:

[root@f19 sf-question543457]# ./monitor-io.stp cat
  PID D  BYTES      us FILE
26207 R    392       4 vda3,usr/lib64/ld-2.17.so
26207 R    832       3 vda3,usr/lib64/libc-2.17.so
26207 R   1758       4 vda3,etc/passwd
26207 R      0       1 vda3,etc/passwd
26208 R    392       3 vda3,usr/lib64/ld-2.17.so
26208 R    832       3 vda3,usr/lib64/libc-2.17.so
26208 R  35147      16 vdb7,ciupicri/doc/grub2-2.00/COPYING
26208 R      0       2 vdb7,ciupicri/doc/grub2-2.00/COPYING

[root@f19 sf-question543457]# mount | grep -E '(vda3|vdb7)'
/dev/vda3 on / type ext4 (rw,relatime,seclabel,data=ordered)
/dev/vdb7 on /mnt/mnt1/mnt11/data type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

Limitations/bugs:

  • full file paths can not be printed yet, but there's a workaround
  • mmap based I/O doesn't show up (because devname is "N/A")
  • files on tmpfs don't show up (because devname is "N/A")
  • it doesn't matter if the reads are from the cache or the writes are to the buffer

P.S. Here are the results for MIfe's programs:

  • for mmaptest private:

     PID D  BYTES      us FILE
    3140 R    392       5 vda3,usr/lib64/ld-2.17.so
    3140 R    832       5 vda3,usr/lib64/libc-2.17.so
    3140 W     23       9 N/A,3
    3140 W     23       4 N/A,3
    3140 W     17       3 N/A,3
    3140 W     17     118 N/A,3
    3140 W     17     125 N/A,3
    
  • for mmaptest shared:

     PID D  BYTES      us FILE
    3168 R    392       3 vda3,usr/lib64/ld-2.17.so
    3168 R    832       3 vda3,usr/lib64/libc-2.17.so
    3168 W     23       7 N/A,3
    3168 W     23       2 N/A,3
    3168 W     17       2 N/A,3
    3168 W     17      98 N/A,3
    
  • for diotest:

     PID D  BYTES      us FILE
    3178 R    392       2 vda3,usr/lib64/ld-2.17.so
    3178 R    832       3 vda3,usr/lib64/libc-2.17.so
    3178 W     16       6 N/A,3
    3178 W 1048576     509 vda3,var/tmp/test_dio.dat
    3178 R 1048576     244 vda3,var/tmp/test_dio.dat
    3178 W     16      25 N/A,3
    
share|improve this answer
    
Thats some pretty impressive systemtap right there. An excellent demonstration of its versatility. –  Matthew Ife Oct 16 '13 at 19:34
    
Does this measure direct I/O and asyncronous I/O? (using io_submit, not posix) –  Matthew Ife Oct 16 '13 at 19:49
    
@Mlfe: thanks! As a side note, while writing the script I managed to discover a tiny bug in pv and another one in SystemTap (task_dentry_path) :-) I have no idea about that I/O, but I can test it if you give me a command or a sample program. For example I used Python to test mmap. dd iflag=direct oflag=direct shows up. –  Cristian Ciupitu Oct 16 '13 at 19:58
2  
Try this for mmap: gist.github.com/anonymous/7014284 I'm betting private mappings aren't measured but shared ones are.. –  Matthew Ife Oct 16 '13 at 20:29
2  
Heres a direct IO test: gist.github.com/anonymous/7014604 –  Matthew Ife Oct 16 '13 at 20:50
show 2 more comments

You'd actually want to use blktrace for this. See this: http://burtonator.wordpress.com/2010/04/27/2009/

I'll see if I can post one of my examples soon.


Edit:

You don't mention distribution of Linux, but maybe this is a good case for a dtrace script on Linux or even System Tap, if you're using a RHEL-like system.

share|improve this answer
2  
Thanks, good thing and very close to the point, but it provides detailed block-layer info, i need something that works on VFS abstraction layer. –  GioMac Oct 3 '13 at 1:41
    
I started to attempt some systemtap scripts to get this running. I failed because the server crashed. I can get this info on Dtrace on Solaris. I'll attempt with Linux today. –  ewwhite Oct 12 '13 at 15:02
add comment

The only tool I know of that can monitor I/O activity by file is inotifywatch. It's part of the inotify-tools package. Unfortunately, it only gives you operation counts.

share|improve this answer
add comment

You mean something like iotop?

share|improve this answer
2  
iotop shows utilization of IO per process/thread, I need to see utilization per file. –  GioMac Oct 3 '13 at 0:46
add comment

use iotop to get the PIDs of processes that contribute high IO

run strace against the PID you generated, you will see which files are being accessed by a particular process

strace -f -p $PID -e trace=open,read,write
share|improve this answer
    
strace will provide information on syscalls and accessed files, it will be very hard to parse and get stats on the usage... –  GioMac Oct 3 '13 at 14:35
    
Thought I'd try this. It generates ALOT of data. And can crash the process when you ctrl+c. It seems to be rather dangerous. –  Matt Oct 10 '13 at 3:26
add comment

You can monitor i/o per block device (via /proc/diskstats) and per process (io accounting via /proc/$PID/io or taskstats), but I don't know of a way to do it per-file.

share|improve this answer
add comment

I'd argue you might have asked the wrong question. if you're looking for i/o bottlenecks, it may be just as important to see what's happening on your disk. db's are notorious for doing random i/o which can significantly reduce throughput, especially if you only have a few spindles.

what may be more interesting is to see if you're having long wait times on the disks themselves. you can do this with collectl via the command "collectl -sD", which will show individual disk performance stats. Are --home to turn it into a top-like utility. If there are lots of disks involved, run it via colmux: colmux -command "-sD" and it will let you sort by a column of your choice, even across multiple systems.

share|improve this answer
    
I don't disagree with you from a disk perspective. Where I can get some insight is when a database filespaces are used to partition data, indexes, logs, etc., but mounted on shared disks when resources are limited - development servers for example. Ideally, each of these filespaces would be on separate volumes, thus looking at the IO from the disk perspective would be adequate - which is likely why all the monitoring utilities are disk, not file based. –  MattK Nov 5 '11 at 16:18
    
It's the right question; the goal is trying to figure out "which table is all this I/O happening to?", and in most databases a table is one or more files. Any disk is going to end up with many files on it, and determining which of those are the hotspots is a useful database tuning input. –  Greg Smith Aug 1 '12 at 0:53
add comment

I recently was tinkering with collectl , it looks a great tool, and pretty straigforward to install. The most interesting is that you can find out which is the responsible process for IO bottlenecks. I would recommend you to read Using Collectl , it might be useful.

share|improve this answer
1  
collectl does not monitor per file, it works per process –  Greg Smith Aug 1 '12 at 0:49
add comment

I would recommend you to check http://dag.wieers.com/home-made/dstat/. This great tool allows to check a lot of stats.

share|improve this answer
1  
dstat does not monitor per file, it summarizes per process –  Greg Smith Aug 1 '12 at 0:50
add comment

May be inotify will resolve solve this.

The inotify API provides a mechanism for monitoring filesystem events.Inotify can be used to monitor individual files, or to monitor directories. When a directory is monitored, inotify will return events for the directory itself, and for files inside the directory.

Monitor File System Activity with inotify

inotify Reference

share|improve this answer
    
This may provide the calls done on the file, but offers little to help discover what pid did it, how big the write was, where and how long it took. –  Matthew Ife Oct 12 '13 at 10:45
add comment

While there's lots of good info in the answers here, I'm wondering if it's actually applicable?

If you are talking about files in the 10s of gigabytes, constantly being written to, then unless they are log files or similar which are constantly appended to (in which case just monitor file sizes), it's most likely that the files are mmap'd. If that's the case, then the best answer may be that you should stop looking at most solutions. The first thing you should then ask of any other proposed solution is "does it work with mmap", because mostly they wont.However, you may be able to turn the problem into monitoring a block device rather than monitoring a file.

When a program asks for a page from a mmap'd file, it's just referencing a location in virtual memory. That page might or might not already be in memory. IF it's not, then that generates a page fault, which triggers the page being loaded in from disk, but that happens within the virtual memory system, which is not easily tied to a specific application process or to a specific file. Similarly, when your app updates a mmap'd page, depending on flags, that might not trigger an immediate write to disk, and in some cases may not go to disk at all (though presumably those last aren't the cases you are interested in).

The best I can think of for mmap'd files, if it's viable for you, is to put each of the files of interest onto a separate device, and use the device statistics to collect your usage info. You could use lvm partitions for this. A bind mount won't work though as it doesn't create a new block device.

Once you have your files on separate block devices you can get stats from /sys/block/*, or /proc/diskstats

It might be too disruptive to introduce this to a production server, but maybe you can make use of it.

IF the files are not mmapped, then yes, you can choose one of the other solutions here.

share|improve this answer
    
Read carefully please, I don't need block-level stats :) –  GioMac Oct 18 '13 at 9:12
    
Right, but the kind of stats you are asking for are not possible for mmapped files, so if you're running up against that, then this gives you a possible way to get stats about files by using one file per device, and reading the device stats. –  mc0e Oct 23 '13 at 18:13
add comment

I think that iotop is one of the best tool on Linux for identify bottlenecks on IO.

share|improve this answer
3  
-1 iotop does not monitor per file, it works per process –  dyasny Mar 7 '12 at 13:42
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.