Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

In an effort to circumvent compatibility and cost barriers to using SSD drives with newer HP ProLiant Gen8 servers, I'm working to validate PCIe-based SSDs on the platform. I've been experimenting with an interesting product from Other World Computing called the Accelsior E2.

enter image description here

This is a basic design; a PCIe card with a Marvell 6Gbps SATA RAID controller and two SSD "blades" connected to the card. These can be passed-through to the OS for software RAID (ZFS, for instance) or leveraged as a hardware RAID0 stripe or RAID1 mirrored pair. Nifty. It's really just compacting a controller and disks into a really small form-factor.

The problem:

Look at that PCIe connector. That's a PCie x2 interface. Physical PCIe slot/lane sizes are typically x1, x4, x8 and x16, with electrical connections usually being x1, x4, x8 and x16. That's fine. I've used x1 cards in servers before.

I started testing the performance of this card on a booted system and discovered that read/write speeds where throttled to ~410 MB/s, regardless of server/slot/BIOS configuration. The servers in use were HP ProLiant G6, G7 and Gen8 (Nehalem, Westmere and Sandy Bridge) systems with x4 and x8 PCIe slots. Looking at the card's BIOS showed that the device negotiated: PCIe 2.0 5.0Gbps x1 - So it's only using one PCIe lane instead of two, thus only half the advertised bandwidth is available.

Is there any way to force a PCIe device to run at a different speed?

My research shows that PCIe x2 is a bit of an oddball lane width... The PCI Express standard apparently does not require compatibility with x2 lane widths, so my guess is that the controllers on my servers are falling back to x1... Do I have any recourse?


Abbreviated lspci -vvv output. Note the difference between the LnkSta and LnkCap lines.

05:00.0 SATA controller: Marvell Technology Group Ltd. Device 9230 (rev 10) (prog-if 01 [AHCI 1.0])
        Subsystem: Marvell Technology Group Ltd. Device 9230
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s L1, Latency L0 <512ns, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
        Kernel driver in use: ahci
        Kernel modules: ahci
share|improve this question
    
I'm assuming you are putting the card into a x4 slot, right? In any case, I'm not sure how the HP will handle the x2 card. In an x4 slot, it should be able to use both lanes but, as you mentioned, there is no guarantee that it has to use it. All that being said, I'm not sure how much of a performance bump you might see as the 6gbs SATA is still generally viewed as being limited to a max of 4.8 gb/s with the encoding overhead with a single PCIe lane having a throughput max of 5gbs. –  Rex Nov 20 '13 at 14:51
    
@Rex "x4 and x8 PCIe slots" were tried. The card has two discrete 6Gbps SATA SSDs onboard, so there is a speed bump available once/if PCIe x2 is negotiated. –  ewwhite Nov 20 '13 at 14:55
    
Ah.. missed the x4/x8 slots in the question, but you do say "PCIe card with a Marvell 6Gbps SATA RAID controller" implying a single controller :) If it has two discrete controllers, is each SSD connected to it's own controller? Then how can it do hardware RAID across the two different controllers? –  Rex Nov 20 '13 at 15:00
    
@Rex There are two SSDs connected to one controller. A 6Gbps RAID controller has 6Gbps bandwidth per SAS/SATA lane. –  ewwhite Nov 20 '13 at 15:07
    
The PCIe spec does not require that slots support anything except 1x. When a card is inserted the host queries it for lane support, and whatever the highest power of two that both support is used (so 1x, 2x, 4x, 8x, 16x, or 32x). Also, each lane on an endpoint (host or card) may support initial negotiation, but typically only the first lane does (when additional lanes support negotiation the slot will be capable of bifurcation, plugging more than one card in through the use of a splitter). As noted below, the servers in question do not support 2x lane mode. –  Chris S Dec 4 '13 at 19:16

3 Answers 3

That's the official answer from OWS, now another answer if it is possible to force hp controller to up it to x2 width, that will require some more research. -)

enter image description here

share|improve this answer
    
But the card is in a G6 ProLiant at the moment - Manufacturer: HP Product Name: ProLiant DL180 G6 –  ewwhite Nov 20 '13 at 16:02
    
Right, likely all Proliant DL pci controllers will have issue with the card. –  Danila Ladner Nov 20 '13 at 16:13
    
Look at you, going to the source!! –  ewwhite Nov 20 '13 at 16:38
    
Also on Gen7, Gen8 for first slot on pcie riser will always get 1x: 1 PCIe 2.0 X1 X4 Half Length/Full Height from here: h18000.www1.hp.com/products/quickspecs/14339_na/14339_na.pdf I assume you tried different slots. –  Danila Ladner Nov 20 '13 at 19:10

I have on hand an IBM server with a Broadcom 4×1GbE card in it… negotiated down from 2.0x4 to 2.0x2:

0b:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 
Subsystem: IBM Device 03a9 
Capabilities: [ac] Express (v2) Endpoint, MSI 00
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Latency L0 <2us, L1 <2us 
LnkSta: Speed 5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

It had a x4 connector which I'm guessing is only fully used when put into a PCIe v1 slot, so it seems that it's negotiating down to x2 when used in a PCIe v2 slot (since 5GT/s bidirectional is enough for 4×1GbE).

Could the same thing be happening with your card?

(If not, my answer is: use IBM instead of HP :P)

share|improve this answer
1  
No IBM!!! Never ;) –  ewwhite Nov 20 '13 at 16:24
up vote 0 down vote accepted

I tried this again on a slightly different HP platform, the 2U HP ProLiant DL380p Gen8 versus the 1U DL360p Gen8. I received the proper results using this combination of hardware.

Current Gen8 host firmware now allows the PCIe links to negotiate at the proper speeds, so these devices ARE compatible with ProLiant DL380p Gen8 servers.

Speed 5GT/s, Width x2, ASPM L0s L1, Latency L0 <512ns, L1 <64us
LnkSta: Speed 5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.