I have ~200 compute nodes that I run my programs on. When these programs are run, these nodes will need access to a shared location where my data is stored. I need a storage that can house ~3TB of data but should be zippy when accessed from all the nodes at the same time.

The usage pattern is reading and writing ~10 files each around 2MB for each program that runs. Each node will run 8 instances of such program. Thus, I have 1600 instances of the same program running and trying to access the data.

There is no concurrent write to the same file.

What storage system should I use?

Currently, I am thinking of buying a couple of Dell servers running Linux + 1 4U RAID enclosure. The Dell servers will serve the files on from the enclosure. I don't know what software will allow me to do load-balancing file serving yet.

  • 1
    I would be talking to vendors at that scale, You will likely be looking at either an fibre channel SAN or an Infiniband connected SAN – Geraint Jones Oct 7 '10 at 3:40
  • that will be looking to read 31.25gb when initially launched. two dell servers will not cut the mustard - unless your happy with ~100 second inital load time – Geraint Jones Oct 7 '10 at 3:42

The most important question in this is what existing I/O do your 200 nodes already have?

The reason I say this is that many compute intensive rigs use infiniband and if you've already got that infrastructure in place I'd strongly advise you to simply buy an FC-to-IB gateway and 'sweat' your existing investment - it'll be fast and resilient.

In the absence of IB the next best performant SAN would be an 8/4Gbps FC network - this will be super fast but very costly for 200 nodes as FC HBAs/SFP+s/FC-switches etc aren't cheap at that volume. That said this is a great solution that would let you sleep well at night. Oh and I agree with Adrian above that HP EVAs are super easy to setup.

If your nodes only have 1 or 10GB ethernet then I'd be tempted to go for either one of these NAS-like distributed filesystems such as gluster or just a plain old mid/high-end NAS such as a NetApp (smaller ones will really struggle with the load). This'll work but you may see issues at peak load times, of course it means your costs will be far lower than the other solutions. If you go the 'big NAS' route I'd be tempted to hook them up with 10Gbps NICs as their price have come down a lot and they'll be much easier to deal with than lots of teamed 1Gbps NICs.

What I would absolutely avoid is iSCSI, the reason is that if you wish to have any shared storage, even just two nodes seeing the same volume, you'll need a cluster file system - these are fine for a few nodes but won't work well for 200 nodes - if at all - the arbitration alone would cripple the environment. This is less of an issue with FC by the way as the arbitration is designed into FC more.

Hope this helps, feel free to come back with follow up questions. Oh and good question by the way :)

  • All nodes have Infiniband cards them. Can you elaborate more on the FC-to-IB gateway solution? – Dat Chu Oct 7 '10 at 22:04
  • Cisco just 'end-of-life'd their excellent SFS3500-series switches but the Mellanox boxes are great too (mellanox.com/content/…). They also seem to have a IB-to-FCoE solution too if that would be of interest. – Chopper3 Oct 8 '10 at 7:22

If you want to roll your own, this has been brought up in a past post/question and it looks very interesting...



ummm, ok if you are interested in dell products, you can ask for information about Dell Equallogic. It's a SAN iSCSI solution. It can provides 4 (ports)*10Gb (i think is the fastests port configuration allowed for equallogic) bandwidth.

You can go for a fiber channel solution too.. HP MSA or EVA...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.