I recently configured SRP target/initiator pair on Linux, and had ~100% performance increase (580MB/s on 10Gbps SDR) over traditional iSCSI-over-IPoIB configuration (300MB/s on SDR).
- Distribution: Debian sid
- Linux kernel: 3.4.0-rc1 (3.3 or above is required for in-kernel SRP)
- Infiniband stack: OFED-1.4 (which comes with Debian)
- SRP/iSCSI target: Linux-iSCSI with in-kernel ib_srpt.ko
- SRP initiator: in-kernel ib_srp.ko
NOTE: AFAIK, SCST is now obsolete as Linux (kernel) is going with Linux-iSCSI (LIO), obsoleting STGT (previous in-kernel implementation) as well. Plan is to merge SCST features into LIO.
- set IB card to "connected" mode (echo connected > /sys/class/net/ib0/mode)
- configure sysctl parameters (same as above post)
- set MTU to maximum (ip link set dev ib0 mtu 65520)
This one is somewhat confusing to figure out, so I'll just paste from my worklog.
=== SRP target configuration ===
// NOTE: This is GUID of your IB interface on target-side. You can check it with ibstatus(1)
/> cd /ib_srpt
/ib_srpt> create 0xfe800000000000000008f1040399d85a
Created target 0xfe800000000000000008f1040399d85a.
Entering new node /ib_srpt/0xfe800000000000000008f1040399d85a
/ib_srpt/0xfe...8f1040399d85a> cd luns
// This is just a dm-zero mapped "/dev/zero"-like block device
/ib_srpt/0xfe...0399d85a/luns> create /backstores/iblock/zero
/ib_srpt/0xfe...85a/luns/lun0> cd ../../acls
// This is GUID of your IB interface on initiator-side
/ib_srpt/0xfe...0399d85a/acls> create 0x00000000000000000008f1040399d832
In above (actual) example, GUID varies between 0xfe80... style and
0x0000xxx style, but I think both can be used interchangeably. You can
configure canonicalization rule by editing /var/target/fabric/ib_srpt.spec (or
whereever Python rtslib library (which Linux-iSCSI tool uses) is installed).
=== SRP initiator configuration ===
// uMAD device must match with IB interface being used
# ibsrpdm -c -d /dev/infiniband/umad1
// Supply above string to ib_srp.ko, in order to setup SRP connection
# for i in $(ibsrpdm -c -d /dev/infiniband/umad1); \
do echo $i > /sys/class/infiniband_srp/srp-mlx4_0-2/add_target; done
If everything went successfully, you will see a message similar to below in your dmesg:
[10713.616495] scsi host9: ib_srp: new target: id_ext 0008f1040399d858 ioc_guid 0008f1040399d858 pkey ffff service_id 0008f1040399d858 dgid fe80:0000:0000:0000:0008:f104:0399:d85a
[10713.815843] scsi9 : SRP.T10:0008F1040399D858
[10713.891557] scsi 9:0:0:0: Direct-Access LIO-ORG IBLOCK 4.0 PQ: 0 ANSI: 5
[10713.988846] sd 9:0:0:0: [sde] 2147483648 512-byte logical blocks: (1.09 TB/1.00 TiB)
As a final note, both ib_srp.ko/ib_srpt.ko is still somewhat immature.
They both work fine, but feature like disconnection seems unimplemented.
So once SCSI block device is attached, there is no way to detach it.
However, their performance is excellent.