I have just configured a server-to-server 4x SDR InfiniBand link to increase bandwidth for the backend storage of my virtual machines, specifically a database server. I am using iSCSI with TGTD on CentOS 6 and I am able to achieve about 6.45Gbps using IPoIB when testing with iperf.

I have managed to tune the iSCSI initiator and target to the point where I can achieve about 260MB/s write to an SSD. However, this is still not an optimal write speed given I can write at about 500-600MB/s locally.

The parameters I am currently using are:

  • MaxRecvDataSegmentLength 4194304
  • MaxXmitDataSegmentLength 524288
  • MaxBurstLength 4194304
  • FirstBurstLength 262144
  • MaxOutstandingUnexpectedPDUs=0
  • MaxOutstandingR2T=1
  • MaxCommands=256
  • QueueDepth=128

Has anyone had any experience in tuning this? Is it possible to get any higher than this with configuration changes? Even by doing a RAID 0 on two devices, I can still only achieve the same bandwidth, which I would have expected higher.


The results you got are reasonably expectable with current setup. You can get additional perfomance gain with iser transport mode.

Configuration tips from README.iser


    ./tgtadm --lld iser --mode target \
             --op new --tid 1 --targetname "iqn.$(hostname).t1"
    ./tgtadm --lld iser --mode target \
             --op bind --tid 1 --initiator-address ALL
    ./tgtadm --lld iser --mode logicalunit \
             --op new --tid 1 --lun 1 \ --backing-store /dev/sde
             --bstype rdwr


    iscsiadm -m discovery -t sendtargets -p $targe
    iscsiadm -m node -p $targetip -T $targetname --op update \
        -n node.transport_name -v iser
    iscsiadm -m node -p $targetip -T $targetname --login

Related usefull SF question: Set up simple Infiniband Block Storage (SRP or iSER).

  • I have changed the transport mode to 'iser' but do not seem to notice much difference, however there may be a few added MB/s which I'm just not noticing due to the inconsistent results. From some further testing, it almost seems like iSCSI is ignoring my MTU of 65520 (for Infiniband). – jaseeey Dec 25 '13 at 14:34
  • I checked out the article you posted... I'm thinking I might need to re-implement this using SRP to get better performance instead of IPoIB. Thank you for your advice! :) – jaseeey Dec 25 '13 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.