Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have a poor man's SAN setup in a 1U Ubuntu server running iSCSI-Target with two 300GB drives in RAID-0. We then are using it for block level storage for virtual machines. The hypervisor is connected to the SAN via gigabit on a dedicated VLAN and interfaces.

We only have a single virtual machine setup and doing some benchmarks. If we run hdparm -t /dev/sda1 from the virtual machine, we get 'ok' performance of 75MB/s from the virtual machine to the SAN. Then we basically compile a package with ./configure and make. Things start ok, but then all the sudden the load average on the SAN grows to 7+ and things slow down to a crawl. When we SSH into the SAN and run top, sure the load is 7+, but the CPU usage is basically nothing, also the server has 1.5GB of memory available. When we kill the compile on the virtual machine, slowly the LOAD on the SAN goes back to sub 1 figures.

What in the world is causing this? How can we diagnosis this further?

Here are two screenshot from the SAN during high load.

1> Output of iotop on the SAN:

http://imgur.com/2doVP

2> Output of top on the SAN:

enter image description here

share|improve this question
    
Can you please provide more details about other specs? What model/brand of server? What type of disks? what type of raid controller (battery backed up?)?. –  3molo May 20 '11 at 7:39
add comment

3 Answers

This looks very much like a typical case of underspeced storage. Hypervisors (especially ESXi / vSphere) would issue synchronous writes significantly more often than you would see with a bare-metal installation of an OS like Linux - where the vast majority of write requests would be asynchronous (unless you've screwed up your filesystem settings). Synchronous writes again would need the storage to confirm that an operation has completed and has been committed to a permanent storage. If all you have is 2 disks, it is going to be a tough game - you are seeing the results.

Your options:

  1. use a RAID controller with an own, battery- or flash-backed cache so it could report completion as soon as data has been written to the cache
  2. lie to your Hypervisor that data has been committed to permanent storage while in fact it has not by enabling IOMode=wb for your LUN definition in ietd.conf

Note that the latter is not recommended as it could lead to corruption of your Hypervisor's data store, the guests' file systems and transactional databases upon a power outage or the crash of your storage server (and IET might crash occasionally indeed), but it is quite suitable as a quick check if sync writes are what is causing your load and abysmal performance numbers when compiling.

share|improve this answer
add comment

bottleneck. could be on the initiator side, network on both sides, target software or target disk subsystem. by the description, I'd start with networking, making sure offloads are turned off (ethtool -K {tso, gso, lro} off)

share|improve this answer
    
You recommend turning offload off on SAN right. –  Justin May 20 '11 at 7:53
    
on the storage and on the hypervisor –  dyasny May 20 '11 at 8:09
    
See above additions of screnshots. Don't think its network, see iotop and top screenshot. –  Justin May 20 '11 at 8:19
add comment

hdparm is a very poor tool for benchmarking IO performance. You should consider using bonnie++, or one of the more application-specific tools.

When doing your ./configure; make process, you'll end up doing a range of reads and writes, with variable sizes, more likely than not spread over the entire disk rather than in a contiguous area.

Once you have a better understanding of your IO system performance, you can look at identifying the root cause.

Is performance OK on the iSCSI target when writing directly to the disk, but not OK when you're talking over iSCSI? If so, probably networking related (offloads, mtu, duplex/speed mismatch etc). If not, probably controller/disk related (write cache etc)

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.