Generally SAN refers to a dedicated network for storage, block level
NFS/CIFS uses TCP/UDP and offers file level access
iSCSI is an TCP protocol (routeable but probably shouldn't be routed)
FC doesn't follow OSI model and needs FC networking devices (FC switch, FC HBA)
FCoE uses lower layers of Ethernet but doesn't use IP (FCoE switches like Cisco Nexus and CNA(HBA+NIC))
In order to share access on block level storage, a cluster aware file system has to be used (to handle conflicts, locking)
With SAN dedicated hardware handles I/O requests (to some degree) so it's less pressure on the CPU.
You can use iSCSI with almost no investment in hardware. Be aware that NIC iSCSI offload may require a licence (Broadcom).
SAN supports multipathing. Or you can probably use layer 2 link aggregation (NIC and switch support required). 100Mbps link is far from enough for storage. I would suggest at least 2x1Gbps links. Congested links can greatly degrade storage performance. Use a dedicated VLAN or (dumb) switch for iSCSI to ensure performance and ease troubleshooting.