Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a Supermicro 2027TR-HTFRF server that I'm using as a xen server.

What I'm wanting to do is get SR-IOV operating for the Connectx-3 card so that I can expose Infiniband to virtual machines.

Following this recipe I managed to get the I350 Ethernet to expose the VF's.

Adding /etc/modprobe.d/mlx4_core.conf containing

options max_vfs=4

Didn't create the virtual functions for the mellanox device despite the fact that this is a valid parameter to the mlx4_core module. i.e.

The error I get is

mlx4_core: `4' invalid for parameter `max_vfs'

I also tried a value of 1 to which I got a similar error

mlx4_core: `1' invalid for parameter `max_vfs'

Output from lspci -v

06:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
    Subsystem: Mellanox Technologies Device 0051
    Flags: bus master, fast devsel, latency 0, IRQ 11
    Memory at df900000 (64-bit, non-prefetchable) [size=1M]
    Memory at de000000 (64-bit, prefetchable) [size=8M]
    Capabilities: [40] Power Management version 3
    Capabilities: [48] Vital Product Data
    Capabilities: [9c] MSI-X: Enable- Count=128 Masked-
    Capabilities: [60] Express Endpoint, MSI 00
    Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [148] Device Serial Number 00-25-90-ff-ff-17-57-24
    Capabilities: [18c] #19
    Kernel modules: mlx4_core

So it would seem that SR-IOV is not available.

I then discovered this thread: http://community.mellanox.com/thread/1073

But I'm using ubuntu 12.04.

Next I installed mstflint using apt-get.

Then I identified what type of card it is.

mstflint -d 06:00.0 q

Unfortunately mstflint crashes.

Warning: memory access to device 06:00.0 failed: Input/output error.
Warning: Fallback on IO: much slower, and unsafe if device in use.
*** buffer overflow detected ***: mstflint terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f66d91ac817]

Some of the strace output:

strace mstflint -d 06:00.0 q
...
open("/proc/bus/pci/06/00.0", O_RDWR|O_DSYNC) = 3
ioctl(3, IIOCNETDIF, 0xdf900000)        = 0
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0xdf900000) = 0x7f0e0a200000
munmap(0x7f0e0a200000, 65536)           = 0
close(3)                                = 0
open("/dev/mem", O_RDWR|O_DSYNC)        = 3
mmap(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0xdf900000) = 0x7f0e092f2000
munmap(0x7f0e092f2000, 65536)           = 0
close(3)                                = 0
write(2, "Warning: memory access to device"..., 136Warning: memory access to device 06:00.0 failed: Input/output error.
Warning: Fallback on IO: much slower, and unsafe if device in use.
) = 136
open("/sys/bus/pci/devices/0000:06:00.0/config", O_RDWR|O_DSYNC) = 3
pwrite(3, "\375\374\373\372", 4, 88)    = 4
pread(3, "\375\374\373\372", 4, 88)     = 4
close(3)                                = 0
open("/dev/tty", O_RDWR|O_NOCTTY|O_NONBLOCK) = 3
writev(3, [{"*** ", 4}, {"buffer overflow detected", 24}, {" ***: ", 6}, {"mstflint", 8}, {" terminated\n", 12}], 5*** buffer overflow detected ***: mstflint terminated
) = 54
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0e0a3f9000
write(3, "======= Backtrace: =========\n", 29======= Backtrace: =========
) = 29
writev(3, [{"/lib/x86_64-linux-gnu/libc.so.6", 31}, {"(", 1}, {"__fortify_fail", 14}, {"+0x", 3}, {"37", 2}, {")", 1}, {"[0x", 3}, {"7f0e094fc817", 12}, {"]\n", 2}], 9/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f0e094fc817]
) = 69
writev(3, [{"/lib/x86_64-linux-gnu/libc.so.6", 31}, {"(", 1}, {"+0x", 3}, {"109710", 6}, {")", 1}, {"[0x", 3}, {"7f0e094fb710", 12}, {"]\n", 2}], 8/lib/x86_64-linux-gnu/libc.so.6(+0x109710)[0x7f0e094fb710]
) = 59
writev(3, [{"/lib/x86_64-linux-gnu/libc.so.6", 31}, {"(", 1}, {"+0x", 3}, {"108b79", 6}, {")", 1}, {"[0x", 3}, {"7f0e094fab79", 12}, {"]\n", 2}], 8/lib/x86_64-linux-gnu/libc.so.6(+0x108b79)[0x7f0e094fab79]
) = 59
writev(3, [{"/lib/x86_64-linux-gnu/libc.so.6", 31}, {"(", 1}, {"_IO_default_xsputn", 18}, {"+0x", 3}, {"dd", 2}, {")", 1}, {"[0x", 3}, {"7f0e0946e13d", 12}, {"]\n", 2}], 9/lib/x86_64-linux-gnu/libc.so.6(_IO_default_xsputn+0xdd)[0x7f0e0946e13d]
) = 73
writev(3, [{"/lib/x86_64-linux-gnu/libc.so.6", 31}, {"(", 1}, {"_IO_vfprintf", 12}, {"+0x", 3}, {"1ae7", 4}, {")", 1}, {"[0x", 3}, {"7f0e0943c4a7", 12}, {"]\n", 2}], 9/lib/x86_64-linux-gnu/libc.so.6(_IO_vfprintf+0x1ae7)[0x7f0e0943c4a7]
) = 69
writev(3, [{"/lib/x86_64-linux-gnu/libc.so.6", 31}, {"(", 1}, {"__vsprintf_chk", 14}, {"+0x", 3}, {"94", 2}, {")", 1}, {"[0x", 3}, {"7f0e094fac14", 12}, {"]\n", 2}], 9/lib/x86_64-linux-gnu/libc.so.6(__vsprintf_chk+0x94)[0x7f0e094fac14]
) = 69
writev(3, [{"/lib/x86_64-linux-gnu/libc.so.6", 31}, {"(", 1}, {"__sprintf_chk", 13}, {"+0x", 3}, {"7d", 2}, {")", 1}, {"[0x", 3}, {"7f0e094fab5d", 12}, {"]\n", 2}], 9/lib/x86_64-linux-gnu/libc.so.6(__sprintf_chk+0x7d)[0x7f0e094fab5d]
) = 68
writev(3, [{"mstflint", 8}, {"[0x", 3}, {"40de46", 6}, {"]\n", 2}], 4mstflint[0x40de46]
) = 19
writev(3, [{"mstflint", 8}, {"[0x", 3}, {"40ee56", 6}, {"]\n", 2}], 4mstflint[0x40ee56]
) = 19
writev(3, [{"mstflint", 8}, {"[0x", 3}, {"406475", 6}, {"]\n", 2}], 4mstflint[0x406475]
) = 19
writev(3, [{"mstflint", 8}, {"[0x", 3}, {"408d00", 6}, {"]\n", 2}], 4mstflint[0x408d00]
) = 19
writev(3, [{"/lib/x86_64-linux-gnu/libc.so.6", 31}, {"(", 1}, {"__libc_start_main", 17}, {"+0x", 3}, {"ed", 2}, {")", 1}, {"[0x", 3}, {"7f0e0941376d", 12}, {"]\n", 2}], 9/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f0e0941376d]
) = 72
writev(3, [{"mstflint", 8}, {"[0x", 3}, {"401af9", 6}, {"]\n", 2}], 4mstflint[0x401af9]
) = 19
write(3, "======= Memory map: ========\n", 29======= Memory map: ========
) = 29
open("/proc/self/maps", O_RDONLY)       = 4
read(4, "00400000-0041a000 r-xp 00000000 "..., 1024) = 1024
write(3, "00400000-0041a000 r-xp 00000000 "..., 102400400000-0041a000 r-xp 00000000 08:01 220                                /usr/bin/mstflint
00619000-0061a000 r--p 00019000 08:01 220                                /usr/bin/mstflint
0061a000-0061b000 rw-p 0001a000 08:01 220                                /usr/bin/mstflint
024a3000-024c4000 rw-p 00000000 00:00 0                                  [heap]
7f0e09302000-7f0e093f2000 rw-s df910000 00:05 1028                       /dev/mem
7f0e093f2000-7f0e095a7000 r-xp 00000000 08:01 135350                     /lib/x86_64-linux-gnu/libc-2.15.so
7f0e095a7000-7f0e097a6000 ---p 001b5000 08:01 135350                     /lib/x86_64-linux-gnu/libc-2.15.so
7f0e097a6000-7f0e097aa000 r--p 001b4000 08:01 135350                     /lib/x86_64-linux-gnu/libc-2.15.so
7f0e097aa000-7f0e097ac000 rw-p 001b8000 08:01 135350                     /lib/x86_64-linux-gnu/libc-2.15.so
7f0e097ac000-7f0e097b1000 rw-p 00000000 00:00 0 
7f0e097b1000-7f0e097c6000 r-xp 00000000 08:01 131116                     /lib/x86_64-linux-gnu/libgcc_s.so.1) = 1024
read(4, "\n7f0e097c6000-7f0e099c5000 ---p "..., 1024) = 1024
write(3, "\n7f0e097c6000-7f0e099c5000 ---p "..., 1024
7f0e097c6000-7f0e099c5000 ---p 00015000 08:01 131116                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f0e099c5000-7f0e099c6000 r--p 00014000 08:01 131116                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f0e099c6000-7f0e099c7000 rw-p 00015000 08:01 131116                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f0e099c7000-7f0e09ac2000 r-xp 00000000 08:01 135361                     /lib/x86_64-linux-gnu/libm-2.15.so
7f0e09ac2000-7f0e09cc1000 ---p 000fb000 08:01 135361                     /lib/x86_64-linux-gnu/libm-2.15.so
7f0e09cc1000-7f0e09cc2000 r--p 000fa000 08:01 135361                     /lib/x86_64-linux-gnu/libm-2.15.so
7f0e09cc2000-7f0e09cc3000 rw-p 000fb000 08:01 135361                     /lib/x86_64-linux-gnu/libm-2.15.so
7f0e09cc3000-7f0e09da5000 r-xp 00000000 08:01 30579                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7f0e09da5000-7f0e09fa4000 ---p 000e2000 08:01 30579                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7f0e09fa4000-7f0e09fac000 ) = 1024
read(4, "r--p 000e1000 08:01 30579       "..., 1024) = 1024
write(3, "r--p 000e1000 08:01 30579       "..., 1024r--p 000e1000 08:01 30579                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7f0e09fac000-7f0e09fae000 rw-p 000e9000 08:01 30579                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7f0e09fae000-7f0e09fc3000 rw-p 00000000 00:00 0 
7f0e09fc3000-7f0e09fd9000 r-xp 00000000 08:01 131300                     /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f0e09fd9000-7f0e0a1d8000 ---p 00016000 08:01 131300                     /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f0e0a1d8000-7f0e0a1d9000 r--p 00015000 08:01 131300                     /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f0e0a1d9000-7f0e0a1da000 rw-p 00016000 08:01 131300                     /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f0e0a1da000-7f0e0a1fc000 r-xp 00000000 08:01 135367                     /lib/x86_64-linux-gnu/ld-2.15.so
7f0e0a210000-7f0e0a300000 rw-s df910000 00:03 4026533088                 /proc/bus/pci/06/00.0
7f0e0a300000-7f0e0a3f0000 rw-s df910000 00:0e 19851                      /sys/devices/pci0000:00/0000:00:03.2/0000:06:) = 1024
read(4, "00.0/resource0\n7f0e0a3f0000-7f0e"..., 1024) = 570
write(3, "00.0/resource0\n7f0e0a3f0000-7f0e"..., 57000.0/resource0
7f0e0a3f0000-7f0e0a3f5000 rw-p 00000000 00:00 0 
7f0e0a3f9000-7f0e0a3fc000 rw-p 00000000 00:00 0 
7f0e0a3fc000-7f0e0a3fd000 r--p 00022000 08:01 135367                     /lib/x86_64-linux-gnu/ld-2.15.so
7f0e0a3fd000-7f0e0a3ff000 rw-p 00023000 08:01 135367                     /lib/x86_64-linux-gnu/ld-2.15.so
7fffdd53f000-7fffdd560000 rw-p 00000000 00:00 0                          [stack]
7fffdd5ff000-7fffdd600000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
) = 570
read(4, "", 1024)                       = 0
close(4)                                = 0
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
gettid()                                = 4122
tgkill(4122, 4122, SIGABRT)             = 0
--- SIGABRT (Aborted) @ 0 (0) ---
+++ killed by SIGABRT (core dumped) +++
Aborted (core dumped)

At this point I'm somewhat stuck.


Update: I managed to obtain a firmware update from SuperMicro which has SR-IOV enabled.

Also I needed to set options max_vfs= to options num_vfs=. It seems the named parameter is slightly different. This was enough to get it working under Ubuntu.

Installing the firmware I used this method: http://www.panticz.de/mellanox But with the addition that I had to call insmod on each module before calling mst start. Modprobe didn't work.

share|improve this question
    
where is "Capabilities: [108] Single Root I/O Virtualization (SR-IOV)" in lspci -v? I do not think it is enabled by default. –  Danila Ladner Sep 26 '13 at 4:22
    
as for the error on mstflint run "mst start" (requires MFT - Mellanox Firmware Tools package) and then re-run mstflint. –  Danila Ladner Sep 26 '13 at 4:28
    
Correct, it seems that the SR-IOV isn't enabled by default in the firmware. mst isn't available on ubuntu. However, I've managed somehow to burn firmware on Ubuntu before and can't recall how I did it. –  Matt Sep 26 '13 at 5:45
    
Ok, I've come to the conclusion that I need newer firmware. But the card has the latest from Supermicro. So this leads me on to a new question: serverfault.com/questions/542031/… –  Matt Sep 27 '13 at 2:14
    
In the end I got it by obtaining a new firmware from SuperMicro. Also you can modify the ini file that comes with oem firmware as well. I did that on another branded box with onboard infiniband. That got me working with SRIOV. –  Matt Oct 22 '13 at 22:56
show 1 more comment

1 Answer

Check the obvious stuff first:

  • In your BIOS settings, ensure that Intel VT-d and Intel I/OAT are both Enabled.
  • Try the card in a different PCI slot.
  • Contact the package maintainer to report a bug in the tool.
share|improve this answer
    
They Intel VT-D & I/O AT are enabled. As mentioned I got the i350 to work but not the infiniband card. –  Matt Sep 26 '13 at 5:43
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.