Find processes generating large IO load on a specific drive

Question

One of our debian-based Proxmox hypervisors is having issues with high IO load on a Raid1 HDD array, used primarily for the root and home filesystems of the Linux VMs running on top.

We have CollectD-based graphs available and can clearly see a large change in the number of Read operations a few days back. These graphs aren't, however, a way to see what process causes the load.

Is there a tool, similar to iotop or dstat, that could display processes generating the largest load, but only on a specific drive? As the server also has an SSD array under high load, all results from those tools are completely useless.

Answer 1

There is a tool that will show processes generating block device accesses for (a) given block device(s). It would be then up to you to exploit the resulting data to figure out what is causing the most I/O load: blktrace and its companion blkparse. The man pages have a few examples.

For example:

% blktrace -d /dev/sda -o - | blkparse -i -

or simply using the btrace wrapper:

% btrace /dev/sda

will show in realtime activity on /dev/sda, including user processes with PID and name, and kernel threads (eg: a dmcrypt thread), with some stats (that won't include any process) when stopped.

The standard mode is to have blktrace record and later blkparse interpret.

By altering the filter mask in blktrace and the output format in blkparse suitable information should be then made available for a further tool in the pipeline in charge of sorting relevant results and associating them with other sources (like process accounting or other logs).