Say I'd like to store 40 terabyte of data for frequent access, mainly media files, through a web interface. Should I build 4 computers with 10 terabyte each? Any limitations I should know about? What would be appropriate?

I'm talking about Linux rack servers.

migrated from superuser.com Feb 22 '10 at 20:16

This question came from our site for computer enthusiasts and power users.


You may want to look into a SAN solution. I worked with a client that had large data needs. They were adding TB's of new user-supplied data every month. Due to budget, we opted for cheap SATA disks in RAID 5 and crammed as many as we could into a 2U server. This worked for a while but as they kept growing performance issues, scalability and management issues arose.

Solutions such as EqualLogic, NetApp and others provide very high scalability and good management tools -- they have price tags to match.

You've not specified how much bandwidth you expect to use, but looking into cloud based solutions that also provide delivery services could be another option, especially if you looking for a pay as you go solution.

We have access to Internap's CDN via one of our partners. We can store data on their systems and then have it delivered via their CDN. I've never inquired as to 40 TB of storage but would not think it to be an issue.

If you do build 4 10TB systems, you will need to flush out these questions: 1) How many front end server nodes will connect to them? 2) How often will the files be accessed? 3) How do you plan to backup the data? 4) What connection method will you use NFS, GFS, iSCSI? 5) How will traffic/storage requirements change over time?

All of these are very important considerations into picking a technology. While building 4 10TB systems may be easy and relatively cheap, if they fail to handle the service loads in 3-5 months, then your money will have been wasted.

Tip: Often with media sharing sites (if that's what this is), some files are more popular than others. 10-20% of your total content may account for >80% of your total traffic. In this case, you may want to consider "fast" and "slow" storage systems. Keep the popular stuff on the fast system and archive off the slower system.

  • Thank you for a good answer. My service will not use streaming so load balancing and bandwith wont be a problem. I'd rather focus on how to optimize the data storage to maximum. With SAN I suppose I would RAID 5 those disks. Should I use 10x1 TB or 5x2 TB or some other combination? – user35692 Feb 22 '10 at 21:01
  • Avoid RAID5...RAID5 was great in the 90's when you had small drives but it's not longer really relevant. RAID6 would be the place to start with larger disks. So to get 10TB of usable space you'll need 12 1TB or 7 2TB drives. If speed and data protection are a high priority then RAID10 would be the better option. It would take 10 2TB drives to get 10TB of usable storage. – 3dinfluence Feb 23 '10 at 0:58
  • When you've got more than 8 disks in an array you should really use RAID 6. There's no fast and hard rule to when RAID 6 is necessary, but when you're using so many drive the chances of double failure starts becoming a number worth thinking about. For instance you might want to consider using 24 2TB drives, you'd get 22 storage drives, and about 39.6GB usable. – Chris S Feb 23 '10 at 14:46
  • RAID 6 it is then. Because speed are not that important since the bottleneck will be the network download speed. Great data protection isn't necessary either. Storage is the most important, and RAID 6 seems perfect for that. So my solution will be 2-6 RAID 6 rack servers with a front end server that stores only the location of every file and then serves it to the client by redirecting the request to one of the back end servers. Thank you. – user35692 Feb 23 '10 at 19:47

It will depend on how critical the data is, what kind of throughput you need, you budget, what technologies you comfortable administering (or what your comfortable outsourcing, or learning), and other factors.

For example, you could grab a relatively cheap server, pop a SAS HBA with extral port(s) into it, then rack up a few drive enclosures. Load an OS with iSCSI Target software on it, and you have a realtively cheap SAN.

Alternatively you can buy prebuilt SAN soltuions such as HP's Lefthand boxes. These make setup and administration very easy, but that comes at a cost.

I have worked with CDW's storage experts on several projects in the past, as long as they know what you're looking for they have always come through for me. Having well defined goals and knowing your limits is a key factor in deploying a successful SAN.

  • Indeed with the well defined goals and knowing my limits. I will consider your tips, also, my limitations is to only use rack servers. So is it possible to pump in 40 TB in a server, or should I split them up? – user35692 Feb 22 '10 at 21:08
  • It's certainly possible. Dell NX4 maxes out at 72 TB, if I read the specs correctly. HP StorageWorks X1000 at over 100TB. IBM and Oracle/Sun/StorageTek have similar products as well. NetApp and EMC make storage solutions that start at 40 TB. Your choices for vendor and configuration depend on budget, performance, features, and amount of external support you want to pay for. – mpez0 Feb 23 '10 at 0:52
  • Yes I have looked at the Dell servers and it will probably be of those I purchase. – user35692 Feb 23 '10 at 19:48
  • Be sure to verify storage controller, backplane, and drive compatibity. Most hardware is compatible these days, but certainly far from all. – Chris S Feb 23 '10 at 21:39

Do yourself a great bit favour and speak to Nexenta. I've just build a 40TB solution with 2TB's of SSD for fast SQL/Data builds all for under £95K - that includes a high spec Citrix Xenserver Setup in DR configuration with 10Gbe!


If you need this storage for a web application, and not something like a database, look at mogilefs:


It can easily scale to 40TB, without costing $100,000. It's pretty much the exact opposite of a SAN.

  • Yeah, if you write your app to use it, there are bunch of nifty solutions, but it only works with stuff you write for it. – Ronald Pottol Feb 24 '10 at 4:11
  • well, the question did say "mainly media files, through a web interface" which is exactly the sort of thing mogilefs was written for. integrating your app with mogilefs isn't all that hard anyway. – Justin Feb 24 '10 at 4:57
  • Yeah, I reread the question, you are right. – Ronald Pottol Feb 24 '10 at 6:28

Have you looked at gluster? it's not super fast (though it is for some people), but it does do much of what you are looking for, and it can be fault tolerant and highly available. From my looking, it is just about the only free HA and FT cluster FS that works today. A server goes down, your clients don't notice. Uses FUSE for the clients, so Linux, many other Unixes, some degree of win (I think) support.

And it's free, with a paid option. It can also speak NFS or smb/cifs (no HA stuff though, need to mount another server if the one you are talking to goes down). For the mailing list archives: http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

http://www.gluster.org/ for the free part http://www.gluster.com/ for the pay part

Then again, as others have said, given that this is for your app, mogilefs and several others may well be best.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.