Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I was wondering how online services which have a high demand for storage capacity - scale physically? (hardware)

Let's assume a service needs to store 1000TB of data, at one geographic location, with the ability to scale. Is there a common way to approach it?

share|improve this question
up vote 0 down vote accepted

There are many different ways, depending on the infrastructure of the application. Network Attached Storage (NAS) solutions from companies like NetApp are common. For other applications, enterprise SAN arrays may be appropriate using iSCSI or FibreChannel.

There are even more creative folks on the hardware side, such as that detailed in this blog post from Backblaze about their Storage Pods, 67TB of NAS for less than $8k (probably even less now with falling drive prices).

Other considerations include redundancy and backup. Performance considerations may affect whether one uses enterprise arrays with SAS drives, or commodity SATA hardware.

Obviously any additional hardware comes with considerations for power, space, and connectivity. These may be handled by the company if they run their own datacenter, or they may be handled by another company via colocation or a fully managed environment.

share|improve this answer
Those pods are awesome phoebus, thanks for the link share! – Nick O'Neil Oct 29 '10 at 20:48
Thanks for the fast response - I actually thought that an online service implies using a service of an existing datacenter? - Regards performance, most of the operation is reading - serving clients. Let's assume there are lots of cache-servers for clients, to take the load off from the primary storage, so I guess we might as well assume minimum performance needs. Let's take a popular service such as Youtube for example? – Doori Bar Oct 29 '10 at 20:58
Some online services have their own datacenters. Youtube certainly does (well, probably its parent company, Google). I just meant that considerations are split up if the datacenter isn't the responsibility of the company looking for storage. – phoebus Oct 29 '10 at 20:59
You'd have to ask Google what they use, but knowing them, they probably have a custom solution based on massive amounts of commodity hardware, since that's how they run their search/ad business. – phoebus Oct 29 '10 at 20:59
@phoebus: So if Google chooses commodity hardware for such requirement, I suppose it's a common decision for similar demanding services? (I assume the link you gave for 'backblaze' is an example of a commodity solution) – Doori Bar Oct 29 '10 at 21:04

There is a always room for growth when design is properly placed I would focus on 2 primary factors in consideration.

The facility at which this content resides, primary power and cooling since we have not spoke yet about the performance needs or bandwidth, this storage will need ample power and air conditioning at proper temperatures.

Secondary would be the technology solution used to store this data. An enterprise SAN device? Commodity white boxes or a combination of both and some DAS enclosures.

share|improve this answer
Thanks for the fast response - I just commented on the following question – Doori Bar Oct 29 '10 at 20:58
your welcome, anytime. feel free to reach out to me if you need any help. – Nick O'Neil Oct 30 '10 at 5:25

besides the neat hardware boxes mentioned by phoebus, you need some software to tie it up. One way is to run iSCSI on each box and mount them all as a big filesystem (ZFS or XFS would be happy to handle all the PetaBytes), but it quickly becomes a management nightmare and a bottleneck.

A much more scalable and failure resistant is to use a 'object store' system. There are several ones available, both proprietary and Open Source, but the ones i'm leaning to are mogileFS and GlusterFS, the first one for simplicity and the latter for maturity (and POSIX compliancy).

share|improve this answer
Thanks for the information, but I was wondering if there's anything wrong with treating each box as a whole, which being identified by it's internally assigned IP? which means, each database record will have an identifier of the IP, and that way it will access a specific storage-box for the relevant path/file? – Doori Bar Oct 30 '10 at 14:05
@Doory Bar: absolutely. in fact, that's the start of those object-store systems i mentioned. of course, there's a lot more than that: redundancy, management, load balance, etc. i did write my own store a long time ago, then networked block storage became affordable and replaced most of it with AoE+LVM+XFS, and currently plan to replace the rest with GlusterFS. – Javier Oct 31 '10 at 1:14
I see, Thanks once again :) – Doori Bar Oct 31 '10 at 8:58

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.