Ubuntu 14.04 (Trusty Tahr) with 16 GB RAM and 16 core CPU as LVM backed iSCSI target using three Samsung SSD disks, each capable of doing 65k IOPS using an LSI 6 Gbit/s controller with on board cache.
Benchmark on SSD disk in target:
fio --filename=/dev/sdd --direct=1 --sync=1 --rw=write --bs=4k --numjobs=10 --iodepth=1 --runtime=60 --time_based --group_reporting --name=ssd-max iops=65514
sdd is configured in hardware RAID 0 using three Samsung 850 EVO SSDs.
I exported a 500G LUN on an Ubuntu 14.04 client with 32 GB RAM and 8 core CPUs.
Benchmark on exported LUN
fio --filename=/dev/sdg --direct=1 --sync=1 --rw=write --bs=4k --numjobs=10 --iodepth=1 --runtime=60 --time_based --group_reporting --name=client-max iops=2400
There is significant performance drop when doing DAS and over network, I was expecting at least 10k IOPS.
Communication between target and initiator is less than 1 ms and iperf shows a network throughput of 9.2 Gbit/s.
I understand that there will be a performance impact for 4k writes as each data has to go through the network stack of both the initiator and target before getting written to disk, but this is an unacceptable drop from 65k to 2k.
Where can the problem be? I have a 10 Gbit/s Ethernet NIC between the target and initiator. Any ideas?