According to the man page:
-t Perform timings of device reads for benchmark and comparison
purposes. For meaningful results, this operation should be
repeated 2-3 times on an otherwise inactive system (no other
active processes) with at least a couple of megabytes of free
So, yes other processes can interfere with these results.
Are there any other areas or tests I can run?
One way you can look to see if other processes are utilizing the disk(s) is to download
sysstat from the main webpage here.
Sysstat is of course available from the repo already but, unfortunately it does not include the
pidstat command which is needed to check this out.
EL5 kernels backported disk accounting into the kernel since EL5.4 without providing an interface to utilize it, but pidstat will work once you've done this.
Then run the
pidstat -d command to generate useful metrics for disk I/O, in particular what other processes are doing with the disk. You can also use
pidstat -d <interval> <count> to get a more realtime contention metric of the disk being used.
Would any sort of VM/OS corruption cause this type of behavior?
OS corruption is quite unlikely (system calls being screwed somehow I guess). hdparm does not utilize the filesystem to do its tests which eliminates any slowdown from that area, including degragmentation issues.
If your using LVM, theres a risk that you happen to be reading from extents that are fragmented though. Your examples dont indicate this however.
VM corruption, well anythings game I guess and thats down to a number of factors I can think of off hand, but probably would include more:
- What image format the disk is in.
- How that data is allocated to the underlying storage mechanism (is it a logical volume with extents itself?)
- If there is a raid rebuild or some other block level maintenance going on inside of the disk controller. I.E if the controller is actually a SAN doing some form of raid rebalancing.
- Disk failure in a raid forcing parity calculations.
- Whether the data is being read in reality by a 'fast' cache like FusionIO, data cache present in SANs or battery backed raid cards.
- Whether the underlying media is tiering your storage somehow.
- Disk contention by the hypervisor which causes your I/O requests to be queued for longer.
- Disk contention by the SAN which causes I/O requests to be queued for longer (its very common for VMs to reside on shared storage).
- Thinly provisioned volumes pose a high risk of fragmentation. How this affects your test is completely dictated by how many blocks you get allocated for a thinly provisioned extent. For example in LVM thin provisioning in linux the extent size is by default 32M, so passing a 32M marker on your disk could impose an extra seek that will mess up your timings.
Would rebuilding/replacing the VM help?
Pot luck I guess if its better or worse. See the above environmental factors that might help/hinder.
At the end of the day, if you are using volume management software or thin provisioning software at any point in the flow of data being written to media (on your host, at your VM level, at the SAN/Controller level) you can have no reliable expectation that the sequential read you are doing is really sequential or that the media you are writing to is consistent (if its a fast disk or data was moved to a slow disk).
Virtualization is so powerful because it adds a layer of logical abstraction to a host. But it can also be potentially horrendous to perform any reliable degree of capacity management on them because of that layer of abstraction.