2

I have 2 new servers with the exact same hardware running FreeNAS. There are 30 4TB drives in each enclosure. Each has the exact same zpool configuration (5x 5 4TB drive RAIDZ volumes, 3 spares, mirrored log drives, 72TB pool).

One server is benchmarking write at 1.25Gbps , and the other at around .7Gbps.

The reporting graphs show a different trend for CPU usage (10% more for the better performer) and for System Load (the worse performer spiking out and having higher load generally) , but approximately the same trend for Memory Utilization.

Both volumes show a healthy state, and everything about the configuration (hw & sw) is identical as it appears to me.

What could be causing this strange behavior, and how can I go about troubleshooting?

Improve this question
3
  • How are these benchmarks done? Include all details.
    – 3molo
    Feb 3, 2013 at 11:04
  • What is the specific hardware? Server, RAID/SAS controllers, disk models, etc.?
    – ewwhite
    Feb 3, 2013 at 16:04
  • backblaze 2.0 storage pod (i3, 9 backplanes with 5 drive slots connected to 3 controllers, 8GB RAM) with a mix of hitachi and wd 7200 SATA drives. one difference is how the drives are arranged on the backplanes. (the slower one has 15 drives on each of 2 controllers, and I believe the faster one has 10 drives attached to each of the three controllers. benchmarking by dding 20 GB from /dev/zero.
    – Brandon Frohbieter
    Feb 3, 2013 at 16:14

1 Answer 1

Reset to default
3

Run diskinfo -c -t /dev/drive against each drive on both. With that many drives it'd probably be easier to script, and there will be tons of output (script might be appropriate).

You'll probably see differences in the performance. If you don't see any differences then you're bottlenecked at a bus (ie, the way the drives are plugged into the SATA controllers).

Side note: The Backblaze enclosures are not meant to be performance units. The SATA and Multiplier Controllers they use are inappropriate for most professional use. SAS controllers and Expander would provide a variety of improvements, though it sounds like your deployment values CapEx over TCO.

Improve this answer
4
  • Yes, my organization is using a combination of enterprise-level storage solutions, but they are taking a look at something like this for archival purposes. I am pleasantly surprised though that it is able to push over a Gb/s write :) I'll take a look at the diskinfo output, thanks for the suggestion. Is it possible that the 3 controllers in parallel are that much faster than 2 controllers (see comment above)? Thanks for your help with this.
    – Brandon Frohbieter
    Feb 3, 2013 at 16:47
  • Considering the "quality" of those controllers I'd strongly suspect that using two vs three would make a noticeable difference.
    – Chris S
    Feb 3, 2013 at 17:01
  • Awesome, thanks for your help. So far the diskinfo is pretty consistent, looks like I'll have to pop it open and try with the same drive configuration as the faster one.
    – Brandon Frohbieter
    Feb 3, 2013 at 17:09
  • Thanks for the tip. After configuring the drives to utilize all 3 controllers, performance is consistent between machines - 179007182 Bytes/sec 179141933 Bytes/sec
    – Brandon Frohbieter
    Feb 4, 2013 at 2:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .