We have a disk exported via iSCSI. When writing to this particular disk, following issue manifests:
Initial speed is decent, then after a few gigabytes of writes
i) disk becomes 100% busy
ii) disk write speed drops to 10 MB/s
iii) Disk IOPS drop to 80 transfers / sec
Some (alarming) points to consider:
1) All other disks in our network function properly, that is their speeds when exported as iSCSI targets are normal and sustained.
2) This specific disk functions properly when used locally, and not as iSCSI target.
3) Disk's behaviour was duplicated in a virtual environment. That is, iSCSI target and iSCSI initiator are two virtual machines in the same bare metal machine.
If it wasn't for point 2) we would come to the conclusion that disk is faulty. But current behaviour (problems only for 1 specific disk, only for 1 specific environment) cannot be explained.
Top left is iSCSI target (hostname "san"). Exported disk is xvdb.
Top right is iSCSI initiator (hostname "sandbox"). Mounted disk is sdb.
Bottom right is a transfer operation (as rsync) in initiator system, from a local disk to aforementioned problematic drive.
Note: Initial speed of ~ 25 MB/s was justifiable due to initiator's CPU bottleneck. But depicted utilization differs from CPU utilization in proper functioning ( 30% user, 70% sys, 0% wait, 0% idle).