updates: from the below answers from "Chopper3", its possible to connect to HP MDS600 JBOD using HP DL server with a RAID controller card. Multiple servers can connect to MDS600 array; create volumes, and distribute those volumes over to others as NFS. MDS600 has two drawers of disks.
the new question is how to tune this for best performance? are there any advantages of moving over to 6Gbps SAS, 10GB ethernet?
although this question is posted by me here:
http://serverfault.com/questions/134155/hardware-for-bulding-unified-storage-solution-using-nexentastor had few trailing questions and wanted to consolidate my understanding:
there is a new software: apart from Nexenta, there is a new Gluster Storage platform. Unlike Nexenta this needs hardware RAID, but does not seem to suffer from any perf issues.
hardware: "Chopper" suggested HP DL370 + MDS600 array. Since MDS600 is just a JBOD enclosure, could this be provisioned using DL370+OS (Nexenta or Gluster) with SAS (external) connectivity. If TRUE are there any perf downsides of using only 1 out of 4 ports on MDS600.