How can I tell what's spinning up my drive?

Question

I have enable "spindown" on one of my Linux server's drives, which is only accessed by a BitTorrent client - meaning, when I don't have torrents running, nothing else should be accessing the disk and waking it up from spindown (right?).

But something is making the disk spin up every 15 minutes or so. How can I track the rogue culprit?

Thanks for any feedback!

Answer 1

You can check out http://samwel.tk/laptop_mode/ . Although this page is not primarly about your problem, but gives a good overall picture how IO works and ideas, how to silent your disks by shutting them off.

The short answer for your problem:

echo 1 > /proc/sys/vm/block_dump

it will produce messages in syslog like:

Jul 23 10:04:05 locutus kernel: ls(5224): READ block 7340136 on dm-4
Jul 23 10:04:05 locutus kernel: ls(5224): READ block 4335136 on dm-4
Jul 23 10:04:05 locutus kernel: ls(5224): READ block 15505568 on dm-4
Jul 23 10:04:05 locutus kernel: ls(5224): READ block 7340160 on dm-4
Jul 23 10:04:05 locutus kernel: ls(5224): READ block 7340168 on dm-4

I've used a simple ls command

NOTE: be sure your system logging is shut off or written to a ramdisk (type: tmpfs) before doing so, because (quote): "your system may get into a feedback loop, where syslogd causes disk activity, this causes kernel output, and this causes syslogd to do more disk activity"

I recommend mounting /tmp as a ramdisk then setting up logging to /tmp/syslog. This in /etc/fstab (and a reboot) will do:

none /tmp tmpfs defaults,size=256M 0 0

Answer 2

In Linux, the recent-ish fanotify syscall enabled a nice utility called fatrace that informs on which process is hitting which file either for the whole system or a particular partition. Definitely my favourite tool for this sort of thing:

deluged(6542): R /usr/share/GeoIP/GeoIP.dat
rsyslogd(1737): W /var/log/syslog
rsyslogd(1737): W /var/log/kern.log

Answer 3

I have been using iotop to figure out what is using the drive. From the website you must have "Linux kernel ≥ 2.6.20 with the TASK_DELAY_ACCT and TASK_IO_ACCOUNTING options enabled".

It sounds like you are just interested in one particular drive. You might consider writing a script that routinely check with lsof or fuser for access to that drive/partition.

Answer 4

If a process is starting to write data to the drive then you could find out about it with checking regularly the open files: lsof.

Else you could look with "free" if the swap file is used or not.

Answer 5

You should be able to use Linux auditing to add a watch on the drive's mountpoint, then monitor the logs with ausearch after the event to find out what file was read from or written to (if any), and what process did it, audit can watch for reads, writes, executes and attribute changes on any filesystem (as long as it's supported on your system – it should be).

Answer 6

Maybe you could use incron which uses inotify to launch commands based on file system activity. You could then possibly watch the mount point and use incron to trigger lsof to look for which process is accessing it.

Answer 7

Easy thing to check what is spinning up your disk is:

go to the directory of the mounted disk, for me was /media/disk0/transmission

then you execute: sudo fatrace -c

you will have the output of anyting that hits that folder, for example you can login with ssh and do an ls.

df(54956): CO /media/disk1 bash(55025): O /media/disk1 bash(55025): RC /media/disk1 bash(55026): O /media/disk1 bash(55026): RC /media/disk1