How can I tell which interface my Supemicro IPMI is piggybacking on?

Question

I've used IPMI before, but only on servers where the IPMI interface had a dedicated ethernet port. I've got an Ubuntu 10.04 server with two ethernet cards, which is supposed to have an IPMI interface on it (the motherboard is a Supermicro H8DMR-I2). From what I understand, the IPMI interface is piggybacking on one of the two NICs.

Is there any way I can tell which NIC the IPMI interface is piggybacking on? Using ipmitool I've tried to set the IP address on the IPMI interface for the subnet for eth0, and then the subnet for eth1, and it's never reachable.

(Can you even reach an IPMI interface from the same NIC it's piggybacking off of, or do you need to try connecting from a different machine on the network?)

Also, is there anything special I need to do to enable it? I can access the IPMI interface locally using "ipmitool".

Here are the firmware details:

# ipmitool mc info
Device ID                 : 34
Device Revision           : 1
Firmware Revision         : 1.50
IPMI Version              : 2.0
Manufacturer ID           : 10437
Manufacturer Name         : Peppercon AG
Product ID                : 4 (0x0004)
Product Name              : Unknown (0x4)
Device Available          : yes
Provides Device SDRs      : no
Additional Device Support :
    Sensor Device
    SDR Repository Device
    SEL Device
    FRU Inventory Device
    Chassis Device
Aux Firmware Rev Info     : 
    0x00
    0x00
    0x54
    0x20

Answer 1

You may be running in to an unfortunate effect of the Supermicro BMC firmware. When power is applied to the power supply, the BMC powers on immediately. During the boot process the BMC (via Uboot which is booting Linux on the BMC) checks to see if the dedicated IPMI NIC port sees a link state. If not, the shared NIC port will be used. The NIC port selected at BMC boot time will be the NIC port used until the BMC is power cycled, either through a direct BMC reboot or when power is removed from the power supply. Rebooting the system itself will do nothing to the BMC.

This creates a cabling time race condition between plugging in the dedicated IPMI NIC and the power cable which is very obnoxious. Or, for example, if you have a power outtage and the BMC comes up before the switch does, the BMC will select the shared NIC in spite of the dedicated NIC being wired and LAN IPMI access will, in the case of VLANed ports, will be on the wrong network. We experience this more often than we like and find it quite frustrating.

If you were able (which, you won't be able to if the BMC comes up on the "wrong" NIC) to connect over the LAN you could SSH to the BMC using the ADMIN account (default password "ADMIN). When logged on to the BMC via SSH you can see the effect of the Uboot time decision in the command line as shown by the usencsi= option at the end of the command line:

# cat /proc/cmdline 
root=/dev/ramdisk ro ip=none ramdisk_blocksize=4096 console=ttyS0,38400 rootfstype=cramfs bigphysarea=1025 usencsi=0

On my system (X8DTi-LN4F) usencsi=0 means "use the dedicated IPMI NIC."

Of course, this requires that you connect to the BMC via the LAN. I've looked pretty hard with the r1.05 firmware and can find no way to discern the selected NIC accessing IPMI from the host. I've just started to look at the r1.32 firmware for this system. In any case, I don't see your motherboard model listed on the SuperMicro IPMI firmware page here:

• http://www.supermicro.com/support/bios/firmware.aspx

What's most frustrating about this is I know what two bytes I would like to hardwire in the BMC firmware letting us set the IPMI interface either to the dedicated NIC or shared NIC but as far as I can discern there is no setting allowing this.

Answer 2

As of the r1.38 revision of the IPMI firmware for the X8 series Supermicro motherboards, you can set the IPMI LAN port to be auto, dedicated or shared via the web UI. It is also accessible via ipmitool (albeit using raw netfn commands):

http://www.supermicro.com/support/faqs/faq.cfm?faq=9848

This doesn't allow you to see which interface IPMI is on in default (failover) mode but does allow you to (A) fix the IPMI to a single interface and (B) read which interface that is.

Hope that helps!

Answer 3

If the host is up you can also run: ipmitool mc reset cold to reboot the BMC which will then do the port test again. If you use the dedicated port, it will then start using it.

Answer 4

you should be able to access it over both NIC's once configured or just via the software ipmitool without network settings. NIC's are only used when you are configuring it remotely.

Is ipmi actually enabled in the BIOS?

Answer 5

Most commonly the ports are labeled on the IO-Panel, and IPMI is almost always on the first one, or both; depending on the chipset.

Also, a few common settings: It's common for IPMI to be on a vlan too, check the settings. Some boards have "remote" IPMI turned off by default, it just has to be turned on. You should be able to access the "remote" features from the local machine as well (though it's possible that particular implementation doesn't support it, that'd be weird).

Answer 6

The shared port is always the first network interface on the motherboard (lowest MAC on the main board) if dedicated is not enabled. This is listed in the motherboard manual, but is true of all Supermicro Chassis.