2

We have a poor mans SAN setup in a 1U Ubuntu server running iSCSI-Target with two 300GB drives in RAID-0. We then are using it for block level storage for virtual machines. The hypervisor is connected to the SAN via gigabit on a dedicated VLAN and interfaces.

We only have a single virtual machine setup and doing some benchmarks. If we run hdparm -t /dev/sda1 from the virtual machine, we get "ok" performance of 75MB/s from the virtual machine to the SAN. Then we basically compile a package with ./configure and make. Things start ok, but then all the sudden the load average on the SAN grows to 7+ and things slow down to a crawl. When we SSH into the SAN and run top, sure the load is 7+, but the CPU usage is basically nothing, also the server has 1.5GB of memory available. When we kill the compile on the virtual machine, slowly the LOAD on the SAN goes back to sub 1 figures.

What in the world is causing this? How can we diagnosis this further?

Here are two screenshot from the SAN during high load.

1> Output of iotop on the SAN:

https://i.stack.imgur.com/u0xt9.jpg

2> Output of top on the SAN:

https://i.stack.imgur.com/UK0f8.png

6
  • what/kind how fast are the 300GB drives? Software RAID? Hardware RAID? What kind of numbers are you getting from the SAN Ubuntu Server with hdparm? I'd really want to do some benchmarking of the "SAN" first to see what kind of I/O you're getting before introducing iSCSI, virtualization, etc.
    – gravyface
    May 21, 2011 at 3:08
  • Standard 7200rpm drives, software RAID, hdparm from SAN is: 75MB/Sec.
    – Justin
    May 21, 2011 at 3:13
  • And you were expecting to get more out of your virtual machine than 75MB/s when the "SAN" is only getting that locally?
    – gravyface
    May 21, 2011 at 3:49
  • We aren't even getting close to 75MB/s from the virtual machine to the SAN. Look at iotop, the load is 6+, but only pushing about 2/MBs of writes. The SAN slows to a crawl and load skyrockets under low I/O.
    – Justin
    May 21, 2011 at 3:57
  • you said from the virtual machine to the SAN, you were getting 75MB/s with hdparm.
    – gravyface
    May 21, 2011 at 4:05

3 Answers 3

2

You should see a significant increase of performance after enabling write caching on the target (details depends on the implementation - what are you using, tgt?) and your disks

hdparm -W 1 /dev/sda
hdparm -W 1 /dev/sdb

There is a price however: this will endanger data integrity (especially if you run databases) in the case of a power outage or a system hang of the SAN, as data which is thought to have been permanently written to disk, only resided in volatile DRAM. To mitigate this risk, you should use a controller with BBWC (battery-backed write cache) where data would survive a power outage for a while (typically 1-2 days).

The main "problem" with ESXi is that it is constantly sync()ing the disks. The need to write metadata to VMFS (if you have it) makes it even worse. The vmware community forums are full of "my disks are slow" posts whenever people are using controllers without write caches.

2
  • Oh my god, making that small change, turning on write cache reduced the compile time from the virtual machine from 20 minutes to 2 minutes. Is that command hdparm -W 1 /dev/sda permanent, i.e. still enabled after reboot.
    – Justin
    May 22, 2011 at 0:50
  • No. Depending on your implementation, there might a /etc/hdparm.conf to automatically set hdparm settings on boot. Check the docs to this file. Of course, if it is not there, you always could introduce your own startup script running hdparm with the appropriate parameters.
    – the-wabbit
    May 22, 2011 at 7:03
1

Run iometer in your virtual machine.

With just two 7.2k rpm drives random access is going to hurt you. You can get only so many iops from them.

Try running two scenarios with iometer:

1) sequential read/write -- this should give nice, fat numbers. 2) random access to the drive -- here you should be in for a land of hurt.

Setup a file for tests large enough to force it to be pushed out of cache of the virtual machine.

2
  • Can't get IO meter installed. When I run make getting error: make -C /tmp/tmpwork/linux-2.6.0.xscale/ SUBDIRS=pwd modules make: *** /tmp/tmpwork/linux-2.6.0.xscale/: No such file or directory. Stop. make: *** [standalone] Error 2
    – Justin
    May 21, 2011 at 5:28
  • Maybe try iozone then? The idea is to use some synthetic benchmark, where you can control the file system access pattern. May 23, 2011 at 9:14
1

I would recommend trying a few things:

  1. Try doing some throughput tests with non-iSCSI traffic (e.g. dd if=/dev/zero bs=1M | nc ...), and see if loads spike the same way (compare both loads and CPU %s). You should probably try testing with only one connection, as well as running about 8 of those tests concurrently. And try with sending and receiving data in both directions.
  2. Try using different target software (e.g. the tgt Ubuntu package)
  3. Update your kernel on the SAN to a more recent version, just in case you ran into a kernel bug.
  4. Report this issue at the iSCSI-Target mailing list, or if no luck there, maybe the linux-kernel mailing list?
  5. If all else fails, and if this an option for you, try NexentaOS or NexentaStor and see if you get better results.

I also stumbled across some iSCSI performance tuning guidelines recently, which you may find helpful, even though those recommendations may not address the particular issue you're experiencing.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .