For a actual project I must setup a high-available storage server who can store and publish (http) 150 million photos in 7 sizes means total 1050 million files. For every photo we need total 200 KB space to store them in all 7 sizes in total 28 TB.

Actually I have two servers available (2x E5620, 12GB Ram, Raid Controller 1 GB NV Cache, 2x160 GB Disk for OS), both have attached a storage array (DAS) with 12x3TB SAS Disks.

I'm not sure if my planed setup is really the best solution:


Disk Array: Raid 6, ext4/rsync or gfs2

HTTP Server: Apache Traffic Server 3 or nginx

This way both server store and publish the photos.

Any advice for me? I can add more servers if needed. What file system is the right way to go? Is Raid 6 ok?

  • Why was this downvoted? It's a common question that needs some thinking (and balls to demand the money if it's the first time you are tasked with it). I even use cases like this in interviews to get an idea how candidates take problems...
    – Martin M.
    Sep 18, 2011 at 15:17
  • @server horror I had nothing to do with the voting or the closure but I'd suggest it could have gone due to being subjective; "design me a solution" questions like this arguably don't fit into the server fault format too well.
    – Rob Moir
    Sep 18, 2011 at 15:43
  • @Robert: My question was not really "Design me a solution"-style, i told what's my idea and ask for advice. As I never had to fight with such amount of files I have think this is the right place to get a feedback from people who has this done. Now I really must ask what's the idea of this site?
    – nenad007
    Sep 18, 2011 at 20:20
  • Well it's not really about advice, but rather about specific solutions to specific problems. For example, a series of questions asking how to optimise storage for files like the ones you are storing, best practice for a web server handling that thpenof content serving, etc. Would be just fine.
    – Rob Moir
    Sep 18, 2011 at 20:46

2 Answers 2


EDIT: Misread the storage requirments!

I'd use at least 2+k+n servers.

  • 2 Servers being load balancers with keepalived, running in pure failover (or whatever floats your boat) -- I'm guessing that 1GigE-Connections are available and that can handle a damn lot of simple GET requests if you use direct return for your IPVS configuration
  • k Servers being Frontend HTTP Servers, the HTTP server would be nginx with some extra partition for local cache. k depends on the amout of traffic you expected to be served out (see OPEN QUESTIONS below)
  • n Servers set up with glusterfs to store the data. This way you can start with 2 GlusterFs servers and test your setup. Since you only store rather small files theres no need to stripe a single file across multiple servers, GlusterFS should do fine. The local cache on the fronteds should be able to overcome any speed problems since the amount of files accesses is usually less than 5% (but I don't know your use case -- this is just wild guessing). n is easily calculated. And yes that's just some example, I don't write that because I think you can't do it but I find myself to often forget about the obvious parts...
    • Take one storage server with 8 500GB disks. Gives you about 6 * 500GB storage (RAID6) 3 TB per server,
    • 10 Servers is 30TB of storage (2TB reserved for initial growth). You do have no redundancy by now,
    • so add in another 10 servers and you with GlusterFS you can configure it to keep 2 copies of each file so any of the storage servers can fail at any time and nothing bad will happen.
    • this is easily expandable by just adding more servers, just get yourself warm with GlusterFS and all should be fine.
  • mount storage servers on the frontends: start happily serving content

OPEN QUESTIONS (and the cover-your-behind questions): (don't know if the requirements are clear to you yet)

  • How much traffic do you expect (necessary to get a sizing for the number of frontends and upstream bandwith)
  • peak times and how many requests per second -- average traffic/day is nice and all but what if all the traffic happens within 6hours of the day
  • expected growth (outgoing traffic and amount of total data)
  • where do the logfiles go? -- sounds like someone would like to run numbers on where all the files go, you'll need to have space for those too.
  • Is your management willing to spend a few bucks on a lab setup? If not ask them how much downtime they can afford if you have to try new configurations on the live hardware. Ask them how much money one minute of downtime will cost. If the don't know or don't give you the budget they can easily find out

In any case I'd stay away from solutions that involve syncing the files around since it sounds like you want to be able to place a file somewhere and have it available immediately. Having a file only available 15minutes after being placed on the servers is not quite optimal and will cost real money in most cases.

NOTE I assume you do have a backup system that will take the 28TB, if not throw in another storage system with the required redundancy to handle worst case options. Add in some off-site backup to handle what will happen if you forgot some worst case scenarion

Doesn't sound too complicated after all. The interesting question is: Is your management willing to spend the money?

  • Moneysaver here: you could always kill the frontend servers and use the storage servers as frontend servers, just give them dedicated network links for storage traffic, and http traffic and you should be fine. Just don't tell your management in the first place you have that option. Also getting a "No to expensive" is not a bad thing, it's just a business decision.
    – Martin M.
    Sep 18, 2011 at 15:15
  • thank you for your answer, for some reason my topic is closed. Regarding your questions: I think it's enough if we can deliver 1gbps (we have a 10G Uplink, every server has 2x1G bonded to switch) and I have think i can reach this with my idea of 2 sotrage servers and every with 30TB, why you suggest to have only around 3TB storage/srv? Regarding req/sec i think 10'000 is fine, nginx and ApacheTS can provide this, I'm not sure about storage access. The except growth is hard to say, this setup will fit for 6 months, after we must react on the new facts. Will continue...
    – nenad007
    Sep 18, 2011 at 20:29
  • ... i plan to hold the logfiles on the frontend server and to move daily to a online space. I don't have planed a backup system, I have think is enough if we store every file two times on our storage server and have them in raid 6. We don't have unlimited budget, but we can buy what's really needed. Please tell me if we really need to split the storage servers down to 3TB per server and why (concurrent access?)? RHEL has builtin-support for GFS2 what advance we get with GlusterFS, i take a look on the web but see some small advantage for both of them.
    – nenad007
    Sep 18, 2011 at 20:39
  • I have now read the Whitepaper Performance in a Gluster System, regarding that the file size for our photos is depending size somewhere between 2kb and 100kb glusterfs can deliver around 20-40 MB/s and some 200-400 operations/sec I'm now not sure if this performs well enough if we use 2-4 node, this would be the reason you suggest 20 nodes I think, exist there any other better performing solution?
    – nenad007
    Sep 18, 2011 at 21:47
  • I don't suggest anything regarding the storage server size. It's just a sample of one possible way. You can always get a server attach a disk array to it and be fine. but with 30TB of storage you need at least 15 disk in that array, still quite an amount of data. Regarding performance: I suggested having a local cache on the frontend servers. To get some real indiciation of numbers don't rely on someone elses benchmarks. Do your own benchmark with your own access behaviour...
    – Martin M.
    Sep 19, 2011 at 16:59

Why not store one large file and have the server convert them to requested sizes on demand then store them in the cache? Consider also running several frontend servers (Through load balancer) to serve the requests, then maybe using NAS or several other servers to serve the static content. The number of frontends you need depends on how much traffic you'll get (youtube capacity or just storing the content for occasional accesses).

  • I think the convert on demand would slow down the server as it need cpu time. If I'm not wrong, if I have this 2 storage server with http server on it, I don't will get much benefit if I put let's say 10 front end servers in front they need anyway to wait until my storage server serve the file (except for local cached files, but the content is hard to cache). Both storage server could serve probably around 500MB/s per server, OK i can put one proxy server in front and connect with bonded 2x1Gbps NIC but I' not sure if it really has a advantage,
    – nenad007
    Sep 18, 2011 at 1:23
  • I can't think of any other way to do this except run servers that are all round robin'ed together and serve the front and backends on themselves, but that would make the problem of having the same content on all of the servers at the same time. I'm trying to think of a way to do this efficiently: not crash your budget and still be able to handle the tasks.
    – U4iK_HaZe
    Sep 18, 2011 at 1:29
  • @U4iK_HaZe it a simple decision the business has to take. Cache on write: probably faster responses but more storage needed. Cache on read: first response slow for an uncached item, only the storage needed that is dedicated to cache. No cache: probably all the responses are slower. Since we don't know the requirements we can not possibly tell what's more important. My guess is the budget to buy this is there. It's not usual quick setup you do on you free time but rather something that sounds like it's making real money. A few 10K could well be possible for hardware budget only...
    – Martin M.
    Sep 18, 2011 at 15:20

Not the answer you're looking for? Browse other questions tagged or ask your own question.