I had some performance issues with iSCSI in a production setup I was unable to debug (due to the production nature) and I'm trying to recreate setup to test, benchmark and eventually replace the prior installation.
My initial guess is there is low performance and huge latency spikes due to inefficient stacking of technologies and improper alignment. Also, an software upgrade of the iSCSI server (Synology NAS) from 5.x to 6.x could contribute to performance issues.
The setup consists of:
An iSCSI server on a dedicated switch to dedicated interfaces on the connected hosts, with (tested) jumbo frames.
Created LUN's and targets based on application purpose (examples: database volume, archival storage, etc). To move them around if needed.
Multiple hosts running KVM virtualisation on CentOS servers.
iSCSI initiator on the KVM hosts to connect to the iSCSI server to present the targets as block devices.
LVM on the KVM hosts using the raw iSCSI block devices (no partition table) to create volume groups and logical volumes to present as raw storage devices to virtual machines.
A GPT partion table with boot partion and LVM on the virtual hosts (as by default CentOS installations) with ext4 formatted filesystems.
My typical 'grow' scenario for hosts would be to: add a new LUN or grow an existing LUN, rescan the SCSI device on the KVM host, pvresize the block device, create a new partition or grow a partition on the VM, and resize the filesystem on the VM.
I'm now evaluating the current setup and I'm able to redefine the stack and check and stress-test if the setup is working correctly. I would like the communities input on the best practices of this kind of setup, the specifics in choices of including KVM and/or partition tables in specific layers, the alignment of 'sectors', and proper blocks sizes for KVM and filesystems.
If there are any questions about the working of the current setup and requirements for the new setup I will update this post based on your comments!