Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm noticing intermittently slow disk read/write speeds on my CenTOS 5.x VM.

Sometimes hdparm will report:

/dev/sda3:
 Timing buffered disk reads:    6 MB in  3.03 seconds =   2.04 MB/sec

Other times, it will report:

/dev/sda3:
 Timing buffered disk reads:   80 MB in  3.53 seconds =  22.34 MB/sec

I'm inclined to suspect that the VMWare host system is overtaxed but before I raise this with the VMWare admin, I'd like to rule out anything else specific to the OS that could cause this behavior.

Are there any other areas or tests I can run? Would any sort of VM/OS corruption cause this type of behavior? Would rebuilding/replacing the VM help?

share|improve this question
add comment

1 Answer

up vote 3 down vote accepted

According to the man page:

   -t     Perform timings of device reads  for  benchmark  and  comparison
          purposes.   For  meaningful  results,  this  operation should be
          repeated 2-3 times on an otherwise  inactive  system  (no  other
          active  processes)  with  at least a couple of megabytes of free
          memory.  

So, yes other processes can interfere with these results.

Are there any other areas or tests I can run?

One way you can look to see if other processes are utilizing the disk(s) is to download sysstat from the main webpage here. Sysstat is of course available from the repo already but, unfortunately it does not include the pidstat command which is needed to check this out.

EL5 kernels backported disk accounting into the kernel since EL5.4 without providing an interface to utilize it, but pidstat will work once you've done this.

Then run the pidstat -d command to generate useful metrics for disk I/O, in particular what other processes are doing with the disk. You can also use pidstat -d <interval> <count> to get a more realtime contention metric of the disk being used.

Would any sort of VM/OS corruption cause this type of behavior?

OS corruption is quite unlikely (system calls being screwed somehow I guess). hdparm does not utilize the filesystem to do its tests which eliminates any slowdown from that area, including degragmentation issues. If your using LVM, theres a risk that you happen to be reading from extents that are fragmented though. Your examples dont indicate this however.

VM corruption, well anythings game I guess and thats down to a number of factors I can think of off hand, but probably would include more:

  • What image format the disk is in.
  • How that data is allocated to the underlying storage mechanism (is it a logical volume with extents itself?)
  • If there is a raid rebuild or some other block level maintenance going on inside of the disk controller. I.E if the controller is actually a SAN doing some form of raid rebalancing.
  • Disk failure in a raid forcing parity calculations.
  • Whether the data is being read in reality by a 'fast' cache like FusionIO, data cache present in SANs or battery backed raid cards.
  • Whether the underlying media is tiering your storage somehow.
  • Disk contention by the hypervisor which causes your I/O requests to be queued for longer.
  • Disk contention by the SAN which causes I/O requests to be queued for longer (its very common for VMs to reside on shared storage).
  • Thinly provisioned volumes pose a high risk of fragmentation. How this affects your test is completely dictated by how many blocks you get allocated for a thinly provisioned extent. For example in LVM thin provisioning in linux the extent size is by default 32M, so passing a 32M marker on your disk could impose an extra seek that will mess up your timings.

Would rebuilding/replacing the VM help?

Pot luck I guess if its better or worse. See the above environmental factors that might help/hinder.

At the end of the day, if you are using volume management software or thin provisioning software at any point in the flow of data being written to media (on your host, at your VM level, at the SAN/Controller level) you can have no reliable expectation that the sequential read you are doing is really sequential or that the media you are writing to is consistent (if its a fast disk or data was moved to a slow disk).

Virtualization is so powerful because it adds a layer of logical abstraction to a host. But it can also be potentially horrendous to perform any reliable degree of capacity management on them because of that layer of abstraction.

share|improve this answer
    
Wow. Thank you very much the brilliant and detailed write up. Much appreciated. –  Mike B Jul 27 '12 at 23:43
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.