0

Just wanted to know what hosting configurations famous sites use.. for example:

  1. Facebook.com uses shared PHP/mysql hosting

  2. php.net uses private Apache/PHP/Mysql hosting running 3 private 3Ghz/8 GB ram servers. Mysql is used as a key/value database with no joins to improve speed.

(I know those are not the real specs, they're just an example)

This can help us build a better understanding of what top sites use and apply that knowledge to our own sites.

0

3 Answers 3

1

@TomTom, your logic is quite wrong.

Average salary at Facebook is $98k/annum, taking into account taxes and benefits, we'll use a base cost of $117k per employee, not including office space, infrastructure, workstation, etc. Facebook uses COTS machines, lets say $2k per including switch ports, power etc. Lets assign a lifetime of 2 years to each machine, therefore, you could buy 117 machines every two years per programmer hired.

The fact that HipHip reduced computing requirements on each server doesn't add mean that their reduced hardware budget has any material effect on their ability to hire. HipHop merely helped scaling and allowed them 35% more capacity from their existing infrastructure. By Facebook's claims regarding HipHop, your claim would be that they could hire an additional 300 employees. The numbers get decidedly worse if Facebook doesn't replace servers every two years -- and we know they don't. HipHip was not developed to allow them to hire more employees, it was developed to prevent massive server requirements which add a host of other scaling problems.

Google has 470k servers based on their last public mention in May of this year. This is prior to their most recent data center opening which typically contain 80k-100k servers. Google's current scaling problem is that map/reduce jobs are constrained to a single data center, meaning that any problem they need to solve is only able to access 80k-100k servers at a time. Having this as a limitation speaks volumes about their data mining requirements.

highscalability.com, as 3molo mentions, contains articles, breakdowns and educated guesses about the infrastructure of many high volume sites.

First rule of scaling, design something that needs scaling, then fix that problem. I read a quote on the web somewhere that 'Going from one server to two servers is exponentially harder than going from two servers to three servers.'

When your application lives on a single server, you haven't hit network bottlenecks, failover issues, stonith problems, vlan and private lan stumbling blocks, worked around tcp congestion issues on your local network, writing a mysql plugin to do key/value or swapping over to udp for lossless session tracking/best effort. Until you have to start mining 750 million records a day for business intelligence or analyzing live streaming data for trends, you don't know what stumbling blocks you'll run into.

Every client interaction needs to reduce the number of clock cycles required to serve that page - this is the basic key to scaling. The fewer clock cycles required, the further your app will go on the hardware. As you start adding decisions to the page generation, you need to add other processes to continue scaling and keep your application online. Google's front end talks to a number of servers when it hands back the results of a search. Speculation is that at least 30 servers are contacted the instant you hit enter on a search query to build portions of the page which are assembled by the frontend.

While the question is quite open ended, the answer is caching. Caching is easy, cache invalidation is hard. Making sure your business process does the minimal work involved to hand back a response, making sure your datastore backends are appropriate for the problem you're solving, making sure your storage system is adequate for each step in the process goes a long way towards success.

As for some examples:

twitter: varnish, nginx, memcached, tons of servers, single mysql master that keeps track of everything as an archival purpose. Their infrastructure leverages memcached and everything is supposed to be served from ram, referring to mysql only when restoring a memcache server. Traversal of the responses is done through a list set in memcached with each timeline stored per user and updated for each tweet. (This may have been modified somewhat recently)

php.net: multiple server mirrors placed at various data centers. Their application is rather simple displaying static pages with a search function.

amazon: oracle, c++, perl (mason templating), they've recently migrated to java/servlets, etc

facebook: php, cassandra, hbase (which is used for their messaging), mysql, memcached, varnish

google: a distributed computer built on linux. If you've worked with the app engine, you understand how a sandboxed version of what google's internal systems are able to use looks like. Google invented their own system to solve problems in a unique manner. While most people use expensive computers for reliable systems, google used COTS hardware. 6 COTS machines cost the same as two really reliable computers, but, they have 3x the available CPU. This allows them to do things that are computationally expensive that other companies can't do. There is a benefit to doing things 'the cheap way'.

wordpress details their architecture on their system. Nginx, Varnish, Apache, two data centers, load balancers, etc.

0

I would say most of them use the LAMP stack or a CMS based solution, Drupal, Joomla, Wordpress.

http://builtwith.com/default.aspx is an interesting site which has some capability of finding out what technologies may currently be in use.

1
  • I'd say you're a bit wrong. Most sites might use LAMP but will then use reverse proxies in front such as Nginx, HAProxy etc. As for Drupal, Joomla and Wordpress. I doubt any really big sites would run anything but custom code. All of the CMS out there are made for the general public which means they are bloated and too slow for huge sites. Nov 28, 2010 at 22:36
-1

This can help us build a better understanding of what top sites use and apply that knowledge to our own sites.

Ah - how? if you ahve to ask this question you are not in the top site league, you know.

Facebook runs on a "number" of servers.- That is in excess of 100.000. WAY in excess. So much in excess they wrote thir own php accelerator and IT PAID because the investment into top programemrs was cheaper than the additional servers they did not ahve to buy.

This is a typical scheme here. All top sites use a custom setup that costs so much you dont even deam of being in the same league if you have to ask this question.

Want another example? Google. They dont have many servers. They have MANY DATA CENTERS that they own and operate. All with a LOT of servers (again, IIRC, in excess - way in excess - of 100.000 PER DATA CENTER).

Basically, if you get into this area you do what people who know what they are doing want you do to. It starts with HIRING someone who has done that before or has a lot of in depth knowledge on how the internet works. For example how to o redundant routing between data centers.

Specific answers to this are impossible. Sorry.

1
  • Easy tiger. Is the question vague? Absolutely. But I'm not sure that warranted a long winded "You are an idiot if you ask this b/c..." I don't believe they ever claimed to be in the same league. If you think a question is stupid and you are too lazy to ask for some clarification from the poster, flag the post; don't get on some soapbox. Most of us didn't come out of the womb w/the wealth of scalability knowledge/experience that you did and so we need to ask questions to learn. If you don't want to help...don't.
    – malonso
    Nov 28, 2010 at 23:36