Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I need to store 25M Photos in 4 sizes = total 100M Files, the filesize will vary between 3Kb and 200 kb per file and the used storage at beginning is about 14-15 TB.

Our goal is to have the data on 2-4 Server available and to serve them with a local fast Webserver (nginx or lighthttpd), we need to server as much possible req/sec.

My plan is to use 2U Servercase from Intel with 12x2TB (WD RE4) with Raid 6 (or FS with redunancy??) for the data and 2x60GB SSD for the OS, is that a good way? Now: I found the Adaptec 5805ZQ who can use SSD SLC Drives for cache of most used files, any suggestions for it?

What Read cache size I need to choose?

What will be the best way for redunancy and load balancing, if I plan to have 2-4 of such Server?

Whats pro/con between Cluster and distributed FS regarding our goal?

share|improve this question
That's many questions in one. You could consider to revise it into a couple of more targeted questions, in order to get better answers... – Jesper Mortensen Sep 10 '10 at 14:15
@Jesper - I saw but it was too late, but I get really good inputs from the community and hope I will get a few more and to be able to mix it together and to get the best solution for my case :) – Nenad Sep 10 '10 at 19:34

If this is greenfield development, then I would absolutely use the cloud for this. 100 M files is a lot of data; it would be a major improvement to offload the redundant storage of that to fx Amazon S3.

Given that we're talking of 100 M files I believe we can safely say some parts of the data set will be 'hot' (frequently requested) and most parts will be cold. Hence we really want caching.

A overview of how this could be done on Amazon Web Services:

  • First layer: Amazon-managed Elastic Load Balancing and Amazon CloudWatch monitoring to a couple of small EC2 instances with nginx or Apache. These servers are just dumb load balancers with static config files, so Cloudwatch can monitor them for us and automatically spawn new instances if one of them crashes.
  • From the first layer: Consistent hasting based on request URL (filename) to a layer of cache servers. You want hashing based on file name to ensure that each file isn't cached many times (reducing your cache hit rate), but rather with N cache servers each server handles 1/N of the address space.
  • Second layer: Cache server(s). Your cache servers are EC2 instances with more memory, and Squid or Varnish or Apache Traffic Server cache installed.
  • From the second layer: Plain old HTTP to Amazon S3 file storage.

Since this setup is loosely coupled, scaling it out horizontally is easy (as scaling problems go).

How fast it is will depend greatly on the ratio between hot and cold data. If your workload is mostly hot, then I wouldn't be surprised to see well above 10.000 req/s from just 2 small load balancer EC2s and 2 high-mem cache EC2 instances.

share|improve this answer
Yes tht will be a really good setup, but I forget to mention that Amazon is not a solution for us. It's above our budget, we will deliver around 80-90 TB monthly and amazon ask much money for this, next is the store of 14 TB which is expensive too and the biggest problem the handling of them for such amount of data. So we want have our own storage server and use 2x 1 Gbps flat Uplink to handle the traffic. But your proposal I think I will give a try for static files as CSS, JS, Icon-Img's. Thank you. – Nenad Sep 10 '10 at 19:19
@Nenad: If you're required to be much much cheaper than Amazon is, then I would worry greatly. Amazon's prices are pretty close to market rate for these things; if you're much cheaper, then there is a high risk that you're cutting corners or being short-changed youself. I don't think 2 x 1Gbps global IP transit of decent quality plus capital & operation cost of 4 large physical servers comes out much cheaper than Amazon's pricing... Regarding CSS, JS just use a CDN, they can be had from 50 - 100 USD/month now. – Jesper Mortensen Sep 10 '10 at 20:22
we pay $1000/1Gbps (HE, fine for http traffic) monthly, 4 storage server will cost $1400/monthly (based on 3 yr lifetime calc), when i put my numbers in the Amazacon Calc I'm between 8'800 (light) and 11'000 (proper) this would be yearly +$100K and above our budget. – Nenad Sep 10 '10 at 20:31
Did you add in your time (and time of other developers/sysadmins) into your cost estimates to make a fair comparison? Setting up and managing a distributed redundant file system that handles 100M objects well is a non-trivial exercise. The cloud providers have already solved this problem, and I suspect they can do it more cheaply that you can. You did not mention how many requests per second, bandwidth, etc. you actually expect. But the cloud solution may be way cheaper. Also look at Rackspace, The Planet, and others, as well as the cheaper CDN providers (SimpleCDN is very inexpensive). – rmalayter Sep 21 '10 at 16:32

SSD's for the OS itself is overkill, unless you're really really interested in booting 30s faster. Just get a pair of small SAS drives and it should be more than enough.

Wrt. the cache, the usefulness of the cache depends on the working set. I.e. are requests for the images expected to be spread evenly around all the images, or do you expect that most requests will be for a small subset? In the latter case, a cache might be useful, in the former, not so much. Note that cache on the disk controller is useful mostly for caching writes (if the cache is non-volatile), which is helpful for fsync()-intensive applications such as databases. For image serving I suspect the benefit won't be that big.

A problem with cluster and distributed FS's is that they are more complicated to set up, and especially distributed FS's are less mature than "normal" single node FS's. A cluster FS typically means shared storage, which means a relatively expensive SAN if you want to avoid single points of failure.

An alternative would be to set up a cluster running some kind of Amazon S3-lookalike that provides a HTTP accessible distributed and replicated key-value storage. E.g. openstack storage.

share|improve this answer
Of this 25M photos, I would say 1-2M has much Views and all the others will be viewed very rare, means a Cache will help me. The special on the Adaptec Controller is that you can use 1-n SLC SSD who the controller self use as cache, they write with 4 SSD Drives on a Webserver scenario you can handle 800% more rquests. OpenSTack sounds very good, but it's at the moment a Developer Preview only, exist there some smiliar who can be used in a productive enviroment? Thank you. – Nenad Sep 10 '10 at 19:32

A lot depends on how frequently those items will be used. If you can expect a small percentage of them to be very active at a time, then you might want to consider Varnish to do your front-end handling, load balanced off your nginx/lighttpd backends. Since frequently used images would be cached, disk speed is a little less important.

However, if the images are not requested repeatedly and caching wouldn't provide a huge boost, nginx/lighttpd on a server or two would do it. You also need to consider the amount of bandwidth that you're going to be delivering. 800mb/sec of a small subset of your dataset would easily be cached by the OS. 800mb/sec of a huge subset of your dataset will likely run into an IO bottleneck in that you can't get the data off the disk fast enough to be served in which case you need to split your system into enough parts to have the IO bandwidth.

Even though you're running raid-6, that is still no substitute for backups, so, budget a similar machine to do backups, or possibly to act as a failover storage server.

share|improve this answer
I understand what you mean, so a bettwer way would be to go with 4 Server with Raid0 and a load balancer in front. But can anyone tell me how to take care that i have on all 4 server all files all the time available, is the right way to use some distributed file system or clustering? – Nenad Sep 10 '10 at 19:26

I'd choose a custom cluster instead of a distributed FS, because it is simpler to understand and troubleshoot, while still working. I.e., the reliability tradeoffs of your own cluster are obvious, while it is a task on its own to figure out how a distributed FS reacts to a dead server or failed switch.

A possible solution to your type of problem is to split the whole photo archive into parts (say, 2 parts) and make the part id explicit in the URL (e.g., make it a subdomain or a GET parameter that is easy to extract with regular expressions). Then, you'll have 4 storage servers with photos (2 servers for each part). Use the fifth server as a reverse proxy that distributes and balances the load. All five servers can run lighttpd. I.e., I propose a very dumb, but working (for the company I worked in - with the total load of ~5000 requests per second, files with 3-10 KB in size, 8 TB of unique files total, server from 24 backends that, however, run a custom HTTP daemon instead of lighttpd) solution.

As for the disks and RAM: we used a software RAID-0 made of four fast but cheap SATA disks on each server (if a disk fails, all data can be copied anyway from a replica on a different server), plus a custom solution to take the whole server offline after a single read error. RAID-5 and RAID-6 are very bad speed-wise even if one disk fails, please don't use them. On the content servers, a lot of RAM is essential (as a disk cache), look for 24 GB or more. Even then, be prepared for a 30-minutes warmup time. On the reverse proxy, if you use lighttpd, please take into account that it buffers the whole upstream response into RAM as fast as possible, and can spend a lot of time pushing the cached photo to someone on dialup or GPRS (and during that time, needs that buffer in RAM). We also took 24 GB just to have identical configurations, but I am not sure if this is overkill. Memory-based HTTP cache on the reverse proxy is not essential (even if there are hot images!), because the OS-provided disk cache on the backends works just as well.

For making sure that all backends that serve the same part of your archive have the same data: this is easy. When publishing photos, just copy them to all servers. Then use rsync on old parts of the archive to correct for any discrepancies, thus making one copy the master.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.