I am about to buy a new server to run VWware ESXi (Dell T320 w/25GB + RAID10). It will mainly be running a Linux host running web applications and a Windows file/database server. I plan to use AWS Storage Gateway to provide their file systems over iSCSI. Does SR-IOV offer material benefits for such a set up? The majority of network communications will be between the guest OSs (iSCSI). External communication will be at most 100mbit.
For VMware and your use-case, probably not. SR-IOV is handy for external networking, but the scenario you describe sounds like using the VMXNET3 virtual adapters and a separate local virtual switch would do the job.
Something like the following, where vSwitch1 is the storage network between virtual machines: