Your question needs to be broken down into two parts: internal vs external storage, and iSCSI vs the alternatives (FC, FCoE, NFS). My experience is with large corporate clients running mainly Oracle, so this may not apply to smaller environments or other databases. In my opinion, there is a lot of value in external storage; iSCSI is one option for providing this but it is not as mature as some of the alternatives.
Internal vs External Storage
External storage, generally provisioned in a disk array and presented through a storage area network (SAN), provides a number of advantages:
- Fast performance, often with hardware-accelerated RAID and large battery-backed caches.
- Easy to scale up volumes to use more disks for performance and capacity.
- Centralized access to storage resources to avoid silos of wasted resources.
- Able to share storage for high-performance or high-availability clusters (e.g. Oracle RAC).
- May come with features for snapshotting and replicating data to remote sites.
- May come with decent analytics tools to track performance.
The main disadvantages of external storage are the complexity and cost of setting up and maintaining the storage area network and the storage arrays.
iSCSI vs Alternatives
Fibre Channel is currently the standard mechanism for accessing external storage for databases. It is common to see SAS (Serial-Attached-SCSI) used for less critical data, more as an extension to internal disk than as disk in a SAN. The key point about these technologies is that they run on dedicated storage networks.
Newer alternatives such as FCoE and iSCSI provide effectively the same protocol as FC and SAS, except they run over Ethernet and therefore can use the same infrastructure used for host-to-host networking. The idea is that by converging on Ethernet, companies can reduce the cost and complexity of having external storage. However, there are still questions around whether Ethernet as a transport provides the speed and reliability of the dedicated technologies.
NFS is a file-level protocol which also runs over Ethernet. In the past it has been considered to have too much overhead for databases, but with hardware offloading in newer network adapters, better OS network stacks, and direct support by the database (e.g. the Direct NFS feature in Oracle) it is also a viable option for some companies. NFS is particularly nice because it reduces the administration involved in resizing volumes, and also fits the model of virtualized storage such as EMC VNX and Oracle ZFS Storage.