Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Is it possible to see in Linux how many read and write requests from user space end up causing cache hits and misses for block devices?

share|improve this question

3 Answers 3

up vote 8 down vote accepted

You can develop your own SystemTap script. You need to account the following two subsystems:

  • VFS: this represents all I/O requests before the Buffer cache (i.e. absolutely every I/O request); review the "vfs.read", "vfs.write" and the "kernel.function("vfs_*")" probes; you need to filter out the block devices you want to monitor by their respective major+minor numbers.
  • Block: this represents all I/O requests sent to the block devices before the I/O scheduler (which also does merge + reorder of the I/O requests); here we know which requests were missed by the Buffer cache; review the "ioblock.request" probe.

SystemTap development takes some time to learn. If you are a moderate developer and have good knowledge in Linux, you should be done in 3-4 days. Yes, it takes time to learn, but you'll be very happy with the results - SystemTap gives you the opportunity to (safely) put probes in almost any place in the Linux kernel.

Note that your kernel must have support for loading and unloading of kernel modules. Most stock kernels nowadays support this. You'll also need to install the debug symbols for your kernel. For my Ubuntu system, this was as easy as downloading a several hundred MB .deb file, which the Ubuntu kernel development team compiled for me. This is explained at the SystemtapOnUbuntu Wiki page, for example.

P.S. Take the SystemTap approach only if you have no other solution, because it's a totally new framework which you have to learn, and that costs time/money and sometimes frustration.

share|improve this answer
1  
+1 nice & clean explanation. thank you, i'll checkout systemtap too. –  risyasin Jul 6 '10 at 7:15
    
I am really excited to play with this! Thanks! –  Kyle Brandt Jul 6 '10 at 13:32
    
A SystemTap script for this is in their wiki. –  Michael Hampton Mar 13 at 3:08

I went ahead and wrote an stap script for this. There is one on the systemtap wiki, but it doesn't appear to be correct. In basic testing, this appears pretty accurate but YMMV.

#! /usr/bin/env stap
global total_bytes, disk_bytes, counter

probe vfs.read.return {
  if (bytes_read>0) {
    if (devname=="N/A") {
    } else {
      total_bytes += bytes_read
    }
  }
}
probe ioblock.request
{
    if (rw == 0 && size > 0)
    {
        if (devname=="N/A") { 
        } else {
          disk_bytes += size
        }
    }

}

# print VFS hits and misses every 5 second, plus the hit rate in %
probe timer.s(5) {
    if (counter%15 == 0) {
        printf ("\n%18s %18s %10s %10s\n", 
            "Cache Reads (KB)", "Disk Reads (KB)", "Miss Rate", "Hit Rate")
    }
    cache_bytes = total_bytes - disk_bytes
    if (cache_bytes < 0)
      cache_bytes = 0
    counter++
    hitrate =  10000 * cache_bytes / (cache_bytes+disk_bytes)
    missrate = 10000 * disk_bytes / (cache_bytes+disk_bytes)
    printf ("%18d %18d %6d.%02d%% %6d.%02d%%\n",
        cache_bytes/1024, disk_bytes/1024,
        missrate/100, missrate%100, hitrate/100, hitrate%100)
    total_bytes = 0
    disk_bytes = 0
}
share|improve this answer
    
Awesome! I added a little average cache use stat to print out when you close it: pastie.org/1845683 –  entropo Apr 29 '11 at 1:21

/proc/slabinfo is a good start, but doesn't give you quite the information you're looking for (don't be fooled by the hit/miss percentages on systems with multiple cores and stats enabled; those are something else). As far as I know, there's not a way to pull that particular information out of the kernel, though it shouldn't be terribly difficult to write up a bit of code to do.

Edit: http://www.kernel.org/doc/man-pages/online/pages/man5/slabinfo.5.html

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.