If you were setting up a site for a file archive of 10TB mainly ISO images (600MB-5GB) what kind of architecture and storage would you use?

I'd like to be able to grow delivery capacity from say 100 concurrent users to 1000 users without wasting the initial hardware (if that's realistic.)

I'd also like to be able to grow storage capacity from 10TB to 100TB in the same way.

I can't assume anything about the active fileset at this stage. It could end up being just one file which could be cached in RAM, but it could also be 95% of the archive.

Would you use single public facing server with a DAS?

Would you use tiered storage like Sun 7410 and expand delivery capacity with its SSD cache modules?

What about commodity hardware with a distributed filesystem like MogileFS and GlusterFS?

What's the best way to set something like this up so it's easy to grow?


For a similar size and expandibility we use 3U supermicro servers. Each server can hold 14 drives (there are 14 integrated sata ports on the motherboard, so we didn't have to buy a raid card, and we have 1 slot left empty in the cases). Each drive is shared via AOE for the front-end server. We have RAID-10 blocks of 14 drives (with linux md), and they are put together with LVM.

If you are careful to not put any two mirrored drives of the raid-10 blocks into the same backend server, then this system is redundant, it can even withstand a loss a complete backend server.

It is easy to grow: just buy another server, build the raid, expand the lvm onto it, and you are ready to go. We use XFS which supports online resizing.

Currently we use 1TB and 1.5TB drives, and we are planning to use 2TB ones in the future. The drives do not need to be all the same size, only if they are in the same raid block.

For your needs, one server filled with 14x 1.5TB or 2TB drives would be a good start imo. You do not even need a separate frontend server for the start.

Our setup can easily produce 1Gb/s throughput, I think that would be more than enough for a web service.

I can help you with more detailed info if you like the idea.


A few more things:

  • This is some kinda homebrew poor-mans-storage solution. If you have the budget, do not want to waste your time, and want to go for top-notch enterprise-class reliability, then you should probably buy some real SAN product. The price can be easily 10-30x as high tough, especially if you want to go for 100TB or more. This does not mean that the above solution does not work - it does, and I think for a lot of purposes the extra price of a SAN product does not worth it.

  • about the backend speed: unfortunately I cannot give exact numbers or test measurements. But I can tell you, that even 1 raid-10 block of 14 drives is fast. If you will need more space, you will use more backend servers and the overall backend speed will increase even more. What's more, if any of the backends would suffer from too many concurrent users (which I doubt), you could go from 2-way mirrorring to 3-way or even more, improving speed, seek time and even reliability.

  • about the frontend speed: the frontend does not have to do much more than forward packets. A modern server should easily keep up at even 10gigabits/s, so it cannot be a bottleneck anytime soon.

| improve this answer | |
  • Hi, thanks for the answer! I like the sound of this, it sounds easy to expand. I'm curious to know what type of access pattern you can produce 1Gb/s throughput with? e.g. how large is your active/hot fileset and the number of concurrent clients? Obviously 90% clients requesting the same file and 10% requesting different files, most of the load can be delivered from the server's RAM. What if 100 users are all requesting different files? If you did become disk-bound with such configuration, can you add front-end servers (to grow overall RAM availble for caching) ? – olivert Jul 20 '09 at 0:41
  • I have edited my answer, hope that helps. ps: we only have a few clients, and our active fileset is about 100% :) – Balázs Pozsár Jul 20 '09 at 19:55
  • One more thing: it is possible to use more than one frontend server, but it gets much more complicated that way. You would have to use a shared filesystem (like gfs or ocfs2) and a shared volume management like drbd or clvm. – Balázs Pozsár Jul 20 '09 at 20:01
  • Thank you for that edit! What is a few clients BTW, 3 or 300? :-) I have to plan for 100 - 1000 (even when library may not grow) so it seems workload would become more random than sequential especially since they are all clients of varying speeds. In that case I think 14 x SATA disks may not have the random IOPS to keep up? Obviously I have to benchmark this to know for sure but if you have any comments on that I'm interested to hear them :-) – olivert Jul 20 '09 at 23:29
  • 1
    Few clients means less than 10 in this case :) I think that a raid-10 setup of 14 sata disk would handle a few hundred clients, if the overall bandwidth is below 1Gb/s and you are serving big files. A real-life read bandwidth of a sata drive today is about 1Gb/s, which means that on average, every drive has to produce only 1/14 of its peak throughput, iow, it has 13/14 of its time to do the seeks. Bechmarks will tell you the final truth, but beware, sometimes they can give you false results. I recommend setting readahead to a high value. See also linux-raid.osdl.org/index.php/Overview – Balázs Pozsár Jul 21 '09 at 13:32

If you don't want local storage then I can safely recommend Nexsan SATABeast. It can hold 42 disks in a 4U enclose, and can thereby give you maximum 40TB raw storage (RAID5 with 1 global spare). It has both Fibre Channel and iSCSI, with the option of dual controllers with failover (active/active).

| improve this answer | |
  • So I could grow the Nexsan SATABeast to 40TB to accomodate the storage requirements, but what about delivery capacity? I think that is the part I'm most confused on. With something like the Sun 7410 I can see options to add SSD cache modules to improve performance. How would I do this in this architecture? Can I have an array of webservers all mounting the same SATABeast, and just load them up with RAM, and keep adding webservers until the disk IO load goes down? – olivert Jul 20 '09 at 0:52
  • Caching in memory and caching on disk is two different things. You could set up Varnish or similar and use local SSD disks for its cache, and a SAN with for example SATABeast for data storage. If you want all servers to have all the storage mounted then you'll need a clustered file system, like Red Hat's GFS or IBM's GPFS. NOTE: Balazs' suggestion above is much cheaper than going for a SAN. – pauska Jul 20 '09 at 11:18
  • Thanks for the answer pauska. varnish/squid plus some application level switching should solve my problem. – olivert Jul 20 '09 at 23:35
  • Happy to help! Note: You said that this would be a site holding ISO images, wich in my head translates to alot of static data that doesnt move (wich again translates to mostly read of data). RAID5 will probably be your best fit to optimize capacity (or perhaps RAID6 for better redundancy). – pauska Jul 21 '09 at 0:38
  • You're right about the access pattern it's almost all reads. I was thinking RAID6 too, although in light of AoE using standard SATA disks perhaps RAID10 is afforable? I have some options to weigh up. Thanks again :-) – olivert Jul 21 '09 at 0:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.