Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have just configured a server-to-server 4x SDR InfiniBand link to increase bandwidth for the backend storage of my virtual machines, specifically a database server. I am using iSCSI with TGTD on CentOS 6 and I am able to achieve about 6.45Gbps using IPoIB when testing with iperf.

I have managed to tune the iSCSI initiator and target to the point where I can achieve about 260MB/s write to an SSD. However, this is still not an optimal write speed given I can write at about 500-600MB/s locally.

The parameters I am currently using are:

  • MaxRecvDataSegmentLength 4194304
  • MaxXmitDataSegmentLength 524288
  • MaxBurstLength 4194304
  • FirstBurstLength 262144
  • MaxOutstandingUnexpectedPDUs=0
  • MaxOutstandingR2T=1
  • MaxCommands=256
  • QueueDepth=128

Has anyone had any experience in tuning this? Is it possible to get any higher than this with configuration changes? Even by doing a RAID 0 on two devices, I can still only achieve the same bandwidth, which I would have expected higher.

share|improve this question

1 Answer 1

up vote 2 down vote accepted

The results you got are reasonably expectable with current setup. You can get additional perfomance gain with iser transport mode.

Configuration tips from README.iser

Target:

    ./tgtadm --lld iser --mode target \
             --op new --tid 1 --targetname "iqn.$(hostname).t1"
    ./tgtadm --lld iser --mode target \
             --op bind --tid 1 --initiator-address ALL
    ./tgtadm --lld iser --mode logicalunit \
             --op new --tid 1 --lun 1 \ --backing-store /dev/sde
             --bstype rdwr

Initiator:

    iscsiadm -m discovery -t sendtargets -p $targe
    iscsiadm -m node -p $targetip -T $targetname --op update \
        -n node.transport_name -v iser
    iscsiadm -m node -p $targetip -T $targetname --login

Related usefull SF question: Set up simple Infiniband Block Storage (SRP or iSER).

share|improve this answer
    
I have changed the transport mode to 'iser' but do not seem to notice much difference, however there may be a few added MB/s which I'm just not noticing due to the inconsistent results. From some further testing, it almost seems like iSCSI is ignoring my MTU of 65520 (for Infiniband). –  Jason Ilicic Dec 25 '13 at 14:34
    
I checked out the article you posted... I'm thinking I might need to re-implement this using SRP to get better performance instead of IPoIB. Thank you for your advice! :) –  Jason Ilicic Dec 25 '13 at 15:07

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.