7

We are using an LSI 9211-8i raid controller (flashed in IT mode) in a PCI-e 2.0 x8 slot. It's connected to a supermicro SAS 836EL1 backplane with 1 SFF-8087 cable. We have 16 drives installed.

When we do a 'badblocks' test of one disk performance is about 140 mbyte/s. When doing badblocks tests concurrently on more drives all drives keep getting about 140mbyte/s until we exceed 8 drives.

With 16 drives active we only get 60mbyte/s on each disk.

Is this normal behavior for an expander backplane? If not, what could be the cause?

Improve this question
2
  • 1
    What models of disk are you using? What OS is this?
    – ewwhite
    Oct 20, 2014 at 18:27
  • 1
    WD2003FYYS disks. FreeBSD (FreeNAS) OS.
    – Jorne
    Oct 20, 2014 at 18:31

3 Answers 3

Reset to default
10

This is how SAS expanders work. You're technically oversubscribed, but in reality, you shouldn't be.

  • Your WD2003FYYS disks link at 3.0Gbps, but are only capable of 1.2Gbps/disk.
  • Your Supermicro drive backplane/expander has 16 ports with 6.0Gbps downlinks to the drives, but a single SFF-8087 uplink to the controller.
  • From what I can tell, this backplane only supports a single SAS SFF-8087 connection to an HBA. The 836EL2 model supports multiple connections.

enter image description here

The SAS SFF-8087 connection is capable of 4 x 6.0Gbps == 24Gbps throughput.

140 Megbytes/second is ~1.2Gbps. 8 disks at that rate is about 9Gbps. You can probably get a few more disks going before rates drop. Account for some overhead.

You're topping out at 3.0Gbps speeds (12Gbps ceiling on a 4-lane SAS cable).

There may be a link negotiation issue between your LSI HBA and the expander backplane. Check the LSI controller and see if it is showing 3.0Gbps link on the SFF-8087 port or 6.0Gbps. You're only getting 3.0Gbps speeds, but all of the devices in the chain (except for the disks) are 6.0Gbps.

Have you brought the firmware of the expander up-to-date?

Edit:

I think I ran into this at work... There was an issue with this expander type and LSI controllers that caused SATA drives to link at 1.5Gbps. Take a look here. So the negotiation issues may be on the drive side. Can you also check firmware on your LSI controller.

Improve this answer
1
  • Good explanation and good mathematics.
    – TomTom
    Nov 11, 2014 at 16:33
3

SAS expanders do not (as of this writing) "store and forward" like an Ethernet switch would. They can only go as fast as the slowest signaling device in the array.

In this case, your drives only support 3.0 Gbps, and the single SFF-8087 cable carries 4 ports to the controller, for a total of 12.0 Gbps. With encoding overhead, that leaves you with roughly 1200 MB/s to share between the 16 drives, roughly 75 MB/s per drive. Throw in some SATA protocol overhead and we're down to the 60 MB/s you are seeing.

If you add another SFF-8087 cable for a total of 8x 3.0 Gbps channels , you will double the available bus bandwidth, realistically achieving ~120 MB/s per drive. Conversely, if the drives supported 6.0 Gbps instead of 3.0, you could achieve that same 120 MB/s over a single cable, or hit each individual drive's peak throughput over two cables. (TL;DR: use two SFF-8087 uplinks whenever possible)

This is one of the reasons why SATA and SAS drives now ship with 6.0 or even 12.0 Gbps signaling rates. The individual disks can only deliver a small fraction of that throughput, but by running the entire SAS bus at a faster rate, it can handle more drives before bottlenecking.

Improve this answer
1
0

I think the bottle neck is on the single SFF-8087 cable which gives you around 12GB/S throughout.

You can confirm this by dropping 4 disks and see if the performance is 80mb/s per disk (16 ^ 60 / 12), or try using 2 cables to connect to LSI 9211-8i.

Bottom line read the manual.http://www.supermicro.com/manuals/other/BPN-SAS-836EL.pdf

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .