Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Amazon, Facebook and Google must take in TBs if not PBs per day, so does that mean they have people continuously installing new hard disks and cables in new racks in the same way you might build a regular server, or is there a different technology used for connecting drives en masse?

share|improve this question
2  
    
Another facet to think of is drive failure rates with that much storage and how they keep up. Some people have estimated Google might have a new drive failure somewhere every minute. –  mtm Sep 26 '13 at 14:04
add comment

5 Answers

up vote 16 down vote accepted

I don't know if any of them are actually installing hardware one server at a time. Back in 2008 MS started building its data centers by getting servers delivered in sealed and pre-wired shipping containers of servers that they just needed to unload from a truck and plug power/network connections into. While the 08 build was a mix of containers and traditional for their most recent datacenter they've since gone to a custom prefab design that's weatherproof and doesn't need to be housed inside separate buildings.

Both HP and IBM sell similar packages with prebuilt containers full of servers that just need power/data connections to deploy.

Microsoft's Boyden Virginia Data Center with prefab modules exposed to the weather.

share|improve this answer
    
This is, to me, the only answer really addressing the initial question. +1 –  Kwaio Aug 30 '13 at 14:00
1  
BTW, here's the source for that picture. It's a Microsoft Azure datacenter in Virginia. –  tedder42 Sep 4 '13 at 21:56
    
Someone still has to build/connect/test the shipping container server at a time and google/facebook tend to do their hardware in house. –  mtm Sep 26 '13 at 14:04
    
@mindthemonkey that's done in the same factory using the same cheap Chinese labor that ordinary rack mounted servers/blades used by the rest of us are assembled in. –  Dan Neely Sep 26 '13 at 14:23
    
Not all of them –  mtm Sep 26 '13 at 16:29
add comment

Google has several technologies which they developed internally to store these huge masses of data. Using these technologies they can actually add truck loads of hard disks into their cluster without any downtime, but yes, they do still need people doing that.

As far as I know from the Google Blog the two main parts are the Google File System, which is a distributed file system that can scale up to really big scale: Google File System

And on top of the Google File System they have Big Table which is some kind of Key Value database and also scales up into huge scales: Big Table

For guaranteeing high availability everything is redundant many times, more than 3 times in most cases.

share|improve this answer
1  
I think the question is more hardware-oriented, and your answer a bit off-topic, but this is a good-to-know information. –  Kwaio Aug 30 '13 at 13:58
add comment

That's precisely correct. I remember that at one time, Facebook datacenters were adding three tractor-trailers full of hard drives and rack-mount servers in the average day. Of course, they have complicated schemes to make storage scalable and redundant. Google, for example, has GFS. Facebook has three data centers just for their equipment, each larger than two Wal-Marts and a new one planned four times larger than their existing centers.

share|improve this answer
add comment

With new-generation Open Compute storage solutions able to fit 180 TB of disk in 4 rack units, a PB is not really a huge amount of space: adding 1PB a day would mean racking 5 such servers a day, not a big problem. Better yet, these can be had for about $10K, which means you are paying less than $60 per TB.

So yeah, the technology is available to all of us to do the same, at a price that doesn't break the bank.

But you have to keep in mind that larger companies always get a better deal, and sign large contracts to procure vast amounts of storage. They may get small shipments spread out over the year, but it's not like they order just 1PB at a time.

share|improve this answer
1  
The open vault specs are online. With 4TB drives they are up to 120TB in 2U. –  mtm Sep 26 '13 at 14:03
add comment

Here is an interesting blog post from BackBlaze on how they do this (they proably don't need as much disks as Google or Facebook, but still a whole lt), and what they needed to do when there were floods in Thailand that made harddisk expensive and harder to get by:

http://blog.backblaze.com/2012/10/09/backblaze_drive_farming/

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.