We are looking for a resonable speed storage. Because of low budget we decided to use software iSCSI or AoE targets. Before we change our production infrastucture, we are doing some tests to choose the best technology.
For testing we use:
- Fujitsu Siemens RX200 S4 as target
- Fujitsu Siemens RX200 S4 as initiator
- NetGear managed 1GBit switch
- onboard NICs (Broadcom w/TOE), EdiMax NICs, Broadcom NICs w/TOE - all 1GBit
- target server is using a QLogic controller with 6 2TB WD Blue SATA drives.
- both target and initiator operating systems are Ubuntu 16.04 LTS with all updates. Switch is dedicated for storage purpose. We test bonds and multipathing.
Our problem is low read speed. For testing we use
dd and a 40-100GB file.
- local read and write on a target server is over 300MB/s.
- writing to server by iSCSI or AoE is over 200MB/s which satisfies us.
- reading from server is always 95-99MB/s.
We have tried ietd, aoetools, LIO. We have used bonds of 2 NICs: balance-rr and LACP, multipathing with r-r. Used normal and jumbo frames. Finally we even did direct ethernet connection between target and host (no switch).
All test give more less the same results (of course using common NICs without TOE and iSCSI gave 20-30% worse results).
Testing network with iperf showed transfers about 200MB/s (2GBit). Watching NICs usage on target with bmon showed equal utilization of both devices (each about 50MB/s for reading, about 100MB/s for writing).
As we had no luck, we decided to use a third NIC (both sides of course). The results were strange:
- 2 NICs - 50MB/s each
- 3 NICs - 33MB/s each
Is there any limit on target software that disables output higher than 1GBit/s?
What do we do wrong?