15

In an effort to circumvent compatibility and cost barriers to using SSD drives with newer HP ProLiant Gen8 servers, I'm working to validate PCIe-based SSDs on the platform. I've been experimenting with an interesting product from Other World Computing called the Accelsior E2.

enter image description here

This is a basic design; a PCIe card with a Marvell 6Gbps SATA RAID controller and two SSD "blades" connected to the card. These can be passed-through to the OS for software RAID (ZFS, for instance) or leveraged as a hardware RAID0 stripe or RAID1 mirrored pair. Nifty. It's really just compacting a controller and disks into a really small form-factor.

The problem:

Look at that PCIe connector. That's a PCie x2 interface. Physical PCIe slot/lane sizes are typically x1, x4, x8 and x16, with electrical connections usually being x1, x4, x8 and x16. That's fine. I've used x1 cards in servers before.

I started testing the performance of this card on a booted system and discovered that read/write speeds where throttled to ~410 MB/s, regardless of server/slot/BIOS configuration. The servers in use were HP ProLiant G6, G7 and Gen8 (Nehalem, Westmere and Sandy Bridge) systems with x4 and x8 PCIe slots. Looking at the card's BIOS showed that the device negotiated: PCIe 2.0 5.0Gbps x1 - So it's only using one PCIe lane instead of two, thus only half the advertised bandwidth is available.

Is there any way to force a PCIe device to run at a different speed?

My research shows that PCIe x2 is a bit of an oddball lane width... The PCI Express standard apparently does not require compatibility with x2 lane widths, so my guess is that the controllers on my servers are falling back to x1... Do I have any recourse?


Abbreviated lspci -vvv output. Note the difference between the LnkSta and LnkCap lines.

05:00.0 SATA controller: Marvell Technology Group Ltd. Device 9230 (rev 10) (prog-if 01 [AHCI 1.0])
        Subsystem: Marvell Technology Group Ltd. Device 9230
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s L1, Latency L0 <512ns, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
        Kernel driver in use: ahci
        Kernel modules: ahci
6
  • I'm assuming you are putting the card into a x4 slot, right? In any case, I'm not sure how the HP will handle the x2 card. In an x4 slot, it should be able to use both lanes but, as you mentioned, there is no guarantee that it has to use it. All that being said, I'm not sure how much of a performance bump you might see as the 6gbs SATA is still generally viewed as being limited to a max of 4.8 gb/s with the encoding overhead with a single PCIe lane having a throughput max of 5gbs.
    – Rex
    Nov 20, 2013 at 14:51
  • @Rex "x4 and x8 PCIe slots" were tried. The card has two discrete 6Gbps SATA SSDs onboard, so there is a speed bump available once/if PCIe x2 is negotiated.
    – ewwhite
    Nov 20, 2013 at 14:55
  • Ah.. missed the x4/x8 slots in the question, but you do say "PCIe card with a Marvell 6Gbps SATA RAID controller" implying a single controller :) If it has two discrete controllers, is each SSD connected to it's own controller? Then how can it do hardware RAID across the two different controllers?
    – Rex
    Nov 20, 2013 at 15:00
  • @Rex There are two SSDs connected to one controller. A 6Gbps RAID controller has 6Gbps bandwidth per SAS/SATA lane.
    – ewwhite
    Nov 20, 2013 at 15:07
  • The PCIe spec does not require that slots support anything except 1x. When a card is inserted the host queries it for lane support, and whatever the highest power of two that both support is used (so 1x, 2x, 4x, 8x, 16x, or 32x). Also, each lane on an endpoint (host or card) may support initial negotiation, but typically only the first lane does (when additional lanes support negotiation the slot will be capable of bifurcation, plugging more than one card in through the use of a splitter). As noted below, the servers in question do not support 2x lane mode.
    – Chris S
    Dec 4, 2013 at 19:16

3 Answers 3

6

That's the official answer from OWS, now another answer if it is possible to force hp controller to up it to x2 width, that will require some more research. -)

enter image description here

4
  • But the card is in a G6 ProLiant at the moment - Manufacturer: HP Product Name: ProLiant DL180 G6
    – ewwhite
    Nov 20, 2013 at 16:02
  • Right, likely all Proliant DL pci controllers will have issue with the card. Nov 20, 2013 at 16:13
  • Look at you, going to the source!!
    – ewwhite
    Nov 20, 2013 at 16:38
  • Also on Gen7, Gen8 for first slot on pcie riser will always get 1x: 1 PCIe 2.0 X1 X4 Half Length/Full Height from here: h18000.www1.hp.com/products/quickspecs/14339_na/14339_na.pdf I assume you tried different slots. Nov 20, 2013 at 19:10
3

I tried this again on a slightly different HP platform, the 2U HP ProLiant DL380p Gen8 versus the 1U DL360p Gen8. I received the proper results using this combination of hardware.

Current Gen8 host firmware now allows the PCIe links to negotiate at the proper speeds, so these devices ARE compatible with ProLiant DL380p Gen8 servers.

Speed 5GT/s, Width x2, ASPM L0s L1, Latency L0 <512ns, L1 <64us
LnkSta: Speed 5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
2
  • Could you tell me what firmware is that? I guess firmwares from that version onwards will support it properly... I'm waiting for a DL380e gen8 so I dont know what fw it has yet but want to be prepared
    – Dredok
    Apr 29, 2020 at 16:38
  • 1
    No idea. That was almost 7 years ago. But just make sure your DL380e is on current firmware.
    – ewwhite
    Apr 29, 2020 at 20:02
0

I have on hand an IBM server with a Broadcom 4×1GbE card in it… negotiated down from 2.0x4 to 2.0x2:

0b:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20) 
Subsystem: IBM Device 03a9 
Capabilities: [ac] Express (v2) Endpoint, MSI 00
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Latency L0 <2us, L1 <2us 
LnkSta: Speed 5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

It had a x4 connector which I'm guessing is only fully used when put into a PCIe v1 slot, so it seems that it's negotiating down to x2 when used in a PCIe v2 slot (since 5GT/s bidirectional is enough for 4×1GbE).

Could the same thing be happening with your card?

(If not, my answer is: use IBM instead of HP :P)

1
  • 1
    No IBM!!! Never ;)
    – ewwhite
    Nov 20, 2013 at 16:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .