35

I am interested in a utility or process for monitoring disk IO per file on CentOS.

On Win2008, the resmon utility allows this type of drilldown, but none of the Linux utilities I have found do this (iostat, iotop, dstat, nmon).

My interest in monitoring IO bottlenecks on database servers. With MSSQL, I have found it an informative diagnostic to know which files / filespaces are getting hit the hardest.

3

13 Answers 13

21

SystemTap is probably your best option.

Here is how the output from the iotime.stp example looks like:

825946 3364 (NetworkManager) access /sys/class/net/eth0/carrier read: 8190 write: 0
825955 3364 (NetworkManager) iotime /sys/class/net/eth0/carrier time: 9
[...]
117061 2460 (pcscd) access /dev/bus/usb/003/001 read: 43 write: 0
117065 2460 (pcscd) iotime /dev/bus/usb/003/001 time: 7
[...]
3973737 2886 (sendmail) access /proc/loadavg read: 4096 write: 0
3973744 2886 (sendmail) iotime /proc/loadavg time: 11

The disadvantage (aside from the learning curve) is that you will need to install kernel-debug, which may not be possible on a production system. However, you can resort to cross-instrumentation where you compile a module on a development system, and run that .ko on the production system.

Or if you are impatient, look at Chapter 4. Useful SystemTap Scripts from the beginners guide.

19
+500

SystemTap* script:

#!/usr/bin/env stap
#
# Monitor the I/O of a program.
#
# Usage:
#   ./monitor-io.stp name-of-the-program

global program_name = @1

probe begin {
  printf("%5s %1s %6s %7s %s\n",
         "PID", "D", "BYTES", "us", "FILE")
}

probe vfs.read.return, vfs.write.return {
  # skip other programs
  if (execname() != program_name)
    next

  if (devname=="N/A")
    next

  time_delta = gettimeofday_us() - @entry(gettimeofday_us())
  direction = name == "vfs.read" ? "R" : "W"

  # If you want only the filename, use
  // filename = kernel_string($file->f_path->dentry->d_name->name)
  # If you want only the path from the mountpoint, use
  // filename = devname . "," . reverse_path_walk($file->f_path->dentry)
  # If you want the full path, use
  filename = task_dentry_path(task_current(),
                              $file->f_path->dentry,
                              $file->f_path->mnt)

  printf("%5d %1s %6d %7d %s\n",
         pid(), direction, $return, time_delta, filename)
}

The output looks like this:

[root@sl6 ~]# ./monitor-io.stp cat
PID D  BYTES      us FILE
3117 R    224       2 /lib/ld-2.12.so
3117 R    512       3 /lib/libc-2.12.so
3117 R  17989     700 /usr/share/doc/grub-0.97/COPYING
3117 R      0       3 /usr/share/doc/grub-0.97/COPYING

Or if you choose to display only the path from the mountpoint:

[root@f19 ~]# ./monitor-io.stp cat
  PID D  BYTES      us FILE
26207 R    392       4 vda3,usr/lib64/ld-2.17.so
26207 R    832       3 vda3,usr/lib64/libc-2.17.so
26207 R   1758       4 vda3,etc/passwd
26207 R      0       1 vda3,etc/passwd
26208 R    392       3 vda3,usr/lib64/ld-2.17.so
26208 R    832       3 vda3,usr/lib64/libc-2.17.so
26208 R  35147      16 vdb7,ciupicri/doc/grub2-2.00/COPYING
26208 R      0       2 vdb7,ciupicri/doc/grub2-2.00/COPYING

[root@f19 ~]# mount | grep -E '(vda3|vdb7)'
/dev/vda3 on / type ext4 (rw,relatime,seclabel,data=ordered)
/dev/vdb7 on /mnt/mnt1/mnt11/data type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

Limitations/bugs:

  • mmap based I/O doesn't show up because devname is "N/A"
  • files on tmpfs don't show up because devname is "N/A"
  • it doesn't matter if the reads are from the cache or the writes are to the buffer

The results for Matthew Ife's programs:

  • for mmaptest private:

     PID D  BYTES      us FILE
    3140 R    392       5 vda3,usr/lib64/ld-2.17.so
    3140 R    832       5 vda3,usr/lib64/libc-2.17.so
    3140 W     23       9 N/A,3
    3140 W     23       4 N/A,3
    3140 W     17       3 N/A,3
    3140 W     17     118 N/A,3
    3140 W     17     125 N/A,3
    
  • for mmaptest shared:

     PID D  BYTES      us FILE
    3168 R    392       3 vda3,usr/lib64/ld-2.17.so
    3168 R    832       3 vda3,usr/lib64/libc-2.17.so
    3168 W     23       7 N/A,3
    3168 W     23       2 N/A,3
    3168 W     17       2 N/A,3
    3168 W     17      98 N/A,3
    
  • for diotest (direct I/O):

     PID D  BYTES      us FILE
    3178 R    392       2 vda3,usr/lib64/ld-2.17.so
    3178 R    832       3 vda3,usr/lib64/libc-2.17.so
    3178 W     16       6 N/A,3
    3178 W 1048576     509 vda3,var/tmp/test_dio.dat
    3178 R 1048576     244 vda3,var/tmp/test_dio.dat
    3178 W     16      25 N/A,3
    

*Quick setup instructions for RHEL 6 or equivalent: yum install systemtap and debuginfo-install kernel

6
  • Thats some pretty impressive systemtap right there. An excellent demonstration of its versatility. Oct 16, 2013 at 19:34
  • Does this measure direct I/O and asyncronous I/O? (using io_submit, not posix) Oct 16, 2013 at 19:49
  • @Mlfe: thanks! As a side note, while writing the script I managed to discover a tiny bug in pv and another one in SystemTap (task_dentry_path) :-) I have no idea about that I/O, but I can test it if you give me a command or a sample program. For example I used Python to test mmap. dd iflag=direct oflag=direct shows up. Oct 16, 2013 at 19:58
  • 2
    Try this for mmap: gist.github.com/anonymous/7014284 I'm betting private mappings aren't measured but shared ones are.. Oct 16, 2013 at 20:29
  • 2
    Heres a direct IO test: gist.github.com/anonymous/7014604 Oct 16, 2013 at 20:50
9

You'd actually want to use blktrace for this. See Visualizing Linux IO with Seekwatcher and blktrace.

I'll see if I can post one of my examples soon.


Edit:

You don't mention distribution of Linux, but maybe this is a good case for a dtrace script on Linux or even System Tap, if you're using a RHEL-like system.

2
  • 2
    Thanks, good thing and very close to the point, but it provides detailed block-layer info, i need something that works on VFS abstraction layer.
    – GioMac
    Oct 3, 2013 at 1:41
  • I started to attempt some systemtap scripts to get this running. I failed because the server crashed. I can get this info on Dtrace on Solaris. I'll attempt with Linux today.
    – ewwhite
    Oct 12, 2013 at 15:02
5

use iotop to get the PIDs of processes that contribute high IO

run strace against the PID you generated, you will see which files are being accessed by a particular process

strace -f -p $PID -e trace=open,read,write
2
  • strace will provide information on syscalls and accessed files, it will be very hard to parse and get stats on the usage...
    – GioMac
    Oct 3, 2013 at 14:35
  • 1
    Thought I'd try this. It generates ALOT of data. And can crash the process when you ctrl+c. It seems to be rather dangerous.
    – hookenz
    Oct 10, 2013 at 3:26
4

The only tool I know of that can monitor I/O activity by file is inotifywatch. It's part of the inotify-tools package. Unfortunately, it only gives you operation counts.

4

I eventually used Sysdig for this

2
  • How-to: Install sysdig, run csysdig, press F2, and select Files view. You will see top of accessed files by OPS column (can be changed pressing F9).
    – catpnosis
    Jul 29, 2017 at 18:48
  • csysdig -v files goes directly to the "Files" view Jan 21, 2019 at 9:14
2

I'd argue you might have asked the wrong question. if you're looking for i/o bottlenecks, it may be just as important to see what's happening on your disk. db's are notorious for doing random i/o which can significantly reduce throughput, especially if you only have a few spindles.

what may be more interesting is to see if you're having long wait times on the disks themselves. you can do this with collectl via the command "collectl -sD", which will show individual disk performance stats. Are --home to turn it into a top-like utility. If there are lots of disks involved, run it via colmux: colmux -command "-sD" and it will let you sort by a column of your choice, even across multiple systems.

2
  • I don't disagree with you from a disk perspective. Where I can get some insight is when a database filespaces are used to partition data, indexes, logs, etc., but mounted on shared disks when resources are limited - development servers for example. Ideally, each of these filespaces would be on separate volumes, thus looking at the IO from the disk perspective would be adequate - which is likely why all the monitoring utilities are disk, not file based.
    – kermatt
    Nov 5, 2011 at 16:18
  • It's the right question; the goal is trying to figure out "which table is all this I/O happening to?", and in most databases a table is one or more files. Any disk is going to end up with many files on it, and determining which of those are the hotspots is a useful database tuning input.
    – Greg Smith
    Aug 1, 2012 at 0:53
2

You can monitor i/o per block device (via /proc/diskstats) and per process (io accounting via /proc/$PID/io or taskstats), but I don't know of a way to do it per-file.

0

May be inotify will resolve solve this.

The inotify API provides a mechanism for monitoring filesystem events.Inotify can be used to monitor individual files, or to monitor directories. When a directory is monitored, inotify will return events for the directory itself, and for files inside the directory.

Monitor File System Activity with inotify

inotify Reference

3
  • This may provide the calls done on the file, but offers little to help discover what pid did it, how big the write was, where and how long it took. Oct 12, 2013 at 10:45
  • The question does not ask about the process (that can possibly be discovered by other means, like lsof) Jan 21, 2019 at 9:20
  • As it's mostly to be expected, the IBM link is no longer valid.
    – PoC
    Jul 4, 2022 at 10:03
0

While there's lots of good info in the answers here, I'm wondering if it's actually applicable?

If you are talking about files in the 10s of gigabytes, constantly being written to, then unless they are log files or similar which are constantly appended to (in which case just monitor file sizes), it's most likely that the files are mmap'd. If that's the case, then the best answer may be that you should stop looking at most solutions. The first thing you should then ask of any other proposed solution is "does it work with mmap", because mostly they wont.However, you may be able to turn the problem into monitoring a block device rather than monitoring a file.

When a program asks for a page from a mmap'd file, it's just referencing a location in virtual memory. That page might or might not already be in memory. IF it's not, then that generates a page fault, which triggers the page being loaded in from disk, but that happens within the virtual memory system, which is not easily tied to a specific application process or to a specific file. Similarly, when your app updates a mmap'd page, depending on flags, that might not trigger an immediate write to disk, and in some cases may not go to disk at all (though presumably those last aren't the cases you are interested in).

The best I can think of for mmap'd files, if it's viable for you, is to put each of the files of interest onto a separate device, and use the device statistics to collect your usage info. You could use lvm partitions for this. A bind mount won't work though as it doesn't create a new block device.

Once you have your files on separate block devices you can get stats from /sys/block/*, or /proc/diskstats

It might be too disruptive to introduce this to a production server, but maybe you can make use of it.

IF the files are not mmapped, then yes, you can choose one of the other solutions here.

2
  • Read carefully please, I don't need block-level stats :)
    – GioMac
    Oct 18, 2013 at 9:12
  • Right, but the kind of stats you are asking for are not possible for mmapped files, so if you're running up against that, then this gives you a possible way to get stats about files by using one file per device, and reading the device stats.
    – mc0e
    Oct 23, 2013 at 18:13
-1

I recently was tinkering with collectl , it looks a great tool, and pretty straigforward to install. The most interesting is that you can find out which is the responsible process for IO bottlenecks. I would recommend you to read Using Collectl , it might be useful.

1
  • 1
    collectl does not monitor per file, it works per process
    – Greg Smith
    Aug 1, 2012 at 0:49
-1

I would recommend you to check http://dag.wieers.com/home-made/dstat/. This great tool allows to check a lot of stats.

1
  • 1
    dstat does not monitor per file, it summarizes per process
    – Greg Smith
    Aug 1, 2012 at 0:50
-2

I think that iotop is one of the best tool on Linux for identify bottlenecks on IO.

1
  • 3
    -1 iotop does not monitor per file, it works per process
    – dyasny
    Mar 7, 2012 at 13:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .