Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have installed three Intel X540-T1's in servers running Ubuntu Server 12.04, but all are non-functional, no link lights, no packets sent or received, and no connection on ip4 or ip6 whether set up as dhcp or static. Also, dmesg doesn't detect cable connection or disconnection.

I updated the default ixgbe driver to Intel's latest version (3.11.33) with no change. The ethernet controller is being reported as X540-AT2 (which might be a problem that I can't figure out how to fix), but the subsystem is X540-T1 so I believe that might be intended.

Does anyone have any experience with this that could assist?

ifconfig eth2
    eth2      Link encap:Ethernet  HWaddr a0:36:9f:14:5f:ea
    inet addr:192.168.101.1  Bcast:192.168.101.255  Mask:255.255.255.0
    UP BROADCAST MULTICAST  MTU:1500  Metric:1<br>
    RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000 
    RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)


ethtool -i eth2
    driver: ixgbe
    version: 3.11.33
    firmware-version: 0x8000037c
    bus-info: 0000:08:00.0
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: yes
    supports-register-dump: yes


lspci -vvnns 08:00.0
    08:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2 [8086:1528] (rev 01)
    Subsystem: Intel Corporation Ethernet Converged Network Adapter X540-T1 [8086:0002]
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 32 bytes
    Interrupt: pin A routed to IRQ 16
    Region 0: Memory at e8000000 (64-bit, prefetchable) [size=2M]
    Region 4: Memory at e8200000 (64-bit, prefetchable) [size=16K]
    [virtual] Expansion ROM at e8280000 [disabled] [size=512K]
    Capabilities: <access denied>
    Kernel driver in use: ixgbe
    Kernel modules: ixgbe

Edit output of dmesg:

dmesg | grep ixgbe
    [    1.684782] ixgbe 0000:08:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
    [    1.684800] ixgbe 0000:08:00.0: setting latency timer to 64
    [    1.685138] ixgbe: 0000:08:00.0: ixgbe_check_options: FCoE Offload feature enabled
    [    2.068069] ixgbe 0000:08:00.0: irq 45 for MSI/MSI-X
    [    2.068075] ixgbe 0000:08:00.0: irq 46 for MSI/MSI-X
    [    2.068079] ixgbe 0000:08:00.0: irq 47 for MSI/MSI-X
    [    2.429029] ixgbe 0000:08:00.0: (PCI Express:2.5GT/s:Width x8) a0:36:9f:14:5f:ea
    [    2.620029] ixgbe 0000:08:00.0: eth2: MAC: 3, PHY: 3, PBA No: G54042-004
    [    2.620033] ixgbe 0000:08:00.0: eth2: Enabled Features: RxQ: 2 TxQ: 2 FdirHash RSC 
    [    2.652184] ixgbe 0000:08:00.0: eth2: Intel(R) 10 Gigabit Network Connection
    [    2.652237] ixgbe 0000:10:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
    [    2.652254] ixgbe 0000:10:00.0: setting latency timer to 64
    [    2.652566] ixgbe: 0000:10:00.0: ixgbe_check_options: FCoE Offload feature enabled
    [    3.032057] ixgbe 0000:10:00.0: irq 48 for MSI/MSI-X
    [    3.032062] ixgbe 0000:10:00.0: irq 49 for MSI/MSI-X
    [    3.032067] ixgbe 0000:10:00.0: irq 50 for MSI/MSI-X
    [    3.393034] ixgbe 0000:10:00.0: (PCI Express:2.5GT/s:Width x8) a0:36:9f:10:a8:f2
    [    3.584019] ixgbe 0000:10:00.0: eth3: MAC: 3, PHY: 3, PBA No: G54042-004
    [    3.584023] ixgbe 0000:10:00.0: eth3: Enabled Features: RxQ: 2 TxQ: 2 FdirHash RSC 
    [    3.616178] ixgbe 0000:10:00.0: eth3: Intel(R) 10 Gigabit Network Connection

Edit output of sudo lspci

sudo lspci -vvnns 08:00.0
    08:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10 Gigabit X540-AT2 [8086:1528] (rev 01)
Subsystem: Intel Corporation Ethernet Converged Network Adapter X540-T1 [8086:0002]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 16
Region 0: Memory at e8000000 (64-bit, prefetchable) [size=2M]
Region 4: Memory at e8200000 (64-bit, prefetchable) [size=16K]
[virtual] Expansion ROM at e8280000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
    Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
    Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
    Address: 0000000000000000  Data: 0000
    Masking: 00000000  Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
    Vector table: BAR=4 offset=00000000
    PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
    DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
    DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
        MaxPayload 128 bytes, MaxReadReq 512 bytes
    DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
    LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Latency L0 <512ns, L1 <4us
        ClockPM- Surprise- LLActRep- BwNot-
    LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
    LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
    DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
    DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
    LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
         Compliance De-emphasis: -6dB
    LnkSta2: Current De-emphasis Level: -6dB
Capabilities: [100 v2] Advanced Error Reporting
    UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
    UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
    UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
    CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
    CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
    AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Device Serial Number a0-36-9f-ff-ff-14-5f-ea
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
    ARICap: MFVC- ACS-, Next Function: 0
    ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
    IOVCap: Migration-, Interrupt Message Number: 000
    IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
    IOVSta: Migration-
    Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 00
    VF offset: 384, stride: 2, Device ID: 1515
    Supported Page Size: 00000553, System Page Size: 00000001
    Region 0: Memory at 00000000c0e00000 (64-bit, non-prefetchable)
    Region 3: Memory at 00000000c0d00000 (64-bit, non-prefetchable)
    VF Migration: offset: 00000000, BIR: 0
Capabilities: [1d0 v1] Access Control Services
    ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
    ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: ixgbe
Kernel modules: ixgbe
share|improve this question
    
Maybe not helpful, but a serious question: why would you pay for that hardware and run in on Ubuntu, instead of an OS for which you can get vendor support? (SLES, RHLE, even Windows...) –  HopelessN00b Nov 9 '12 at 19:05
    
This is one of a number of test servers that will only be up for a couple of days. It has only a very short life span and a single purpose. –  Jeff Carr Nov 9 '12 at 20:01
    
Oh, well sorry about your luck. Sounds like the test results are in, and Ubuntu doesn't play well with those cards. Although, since it's listed as a supported OS, I'd get Intel tech support on the phone and make it their problem. Or try to, at least. –  HopelessN00b Nov 9 '12 at 20:27
    
Can you add the output of " dmesg | grep ixgbe "? If you do the lspci as root or with sudo, then Capabilities: <access denied> seems problem. –  John Siu Nov 10 '12 at 2:18
    
Did you check the cables? After all you don't have a link according to your link lights. And those should be on even if you don't have any driver installed at all. –  lsmooth Nov 10 '12 at 10:33
show 6 more comments

1 Answer 1

up vote 2 down vote accepted

It turns out that the prototype NICs won't talk directly to any non-prototype NICs. I'm not sure why exactly. The x540-T1 in Ubuntu work just fine to talk to each other, at least after installing the latest Intel drivers (and possibly before). Thanks for the help everyone.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.