Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote -6 down vote favorite

how many machines should we approximately have in order to reach Google kind of index in 1 year?also the internet speed & machine configuration We developing the Crawler in JAVA

share|improve this question
I have no idea, but I hope you have a big budget... – Justin Ethier Aug 15 '10 at 17:00
Why a negative vote? – Alex Mathew Aug 15 '10 at 17:02
13  
I can imagine the Dilbert boss asking exactly that question. – Iznogood Aug 15 '10 at 17:07
6  
-1 because of the combination of over ambitiousness and under specification. Or to quote a famous trope: "if you have to ask, you can't afford it." – John Kugelman Aug 15 '10 at 17:11
2  
@radius the question is not whether or not it can be done, but what would it take. It's completely unanswerable... – squillman Aug 15 '10 at 17:43
show 18 more comments

migrated from stackoverflow.com Aug 15 '10 at 17:02

closed as not a real question by SvW, squillman, radius, Ward, Zoredache Aug 15 '10 at 19:40

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, see the FAQ.

5 Answers

up vote 1 down vote accepted

Trusting http://www.wisegeek.com/how-big-is-the-internet.htm it seems that google has indexed 200 terabytes. If this is exact, with a 100 Mbps connexion you should be able to get 200 TB in a year. But you will have an index with some data that are totally outdated... Server will totally depend of your crawler performance, and of the way you store data ! (Database used, ...)
Anyway the bigger problem will be the storage I think.

Finally, no one here will be able to give you a correct answer without a complete set of information. You'd better hire specialist in this very specific domain.

share|improve this answer
thks for this valuable info......your answer is what i need....thks very much – Alex Mathew Aug 15 '10 at 17:29
up vote 4 down vote

Look at the current hardware section on wiki page.

The combined processing power of these servers might reach from 20 to 100 petaflops.

share|improve this answer
This include client request, while he only want to build a crawler. He did not say that he will have billions of client doing queries. – radius Aug 15 '10 at 17:42
@Radius Look at the question God:How many machines would we need to beat google – Iznogood Aug 15 '10 at 19:10
up vote 2 down vote

Based on your list of questions on StackOverflow the last weeks I would strongly suspect that you do not have the skills and experience that Google has - not to mention the 10 year headstart - and that is much, much more important than the raw data.

Please notice that even Microsoft cannot catch up, so you will need more than them.

share|improve this answer
up vote 1 down vote

From the same wiki page as jethro linked:

Upwards of 15,000 servers ranging from 533 MHz Intel Celeron to dual 1.4 GHz Intel Pentium III (as of 2003). A 2005 estimate by Paul Strassmann has 200,000 servers, while unspecified sources claimed this number to be upwards of 450,000 in 2006.

One or more 80 GB hard disks per server (2003)

2–4 GB of memory per machine (2004)

share|improve this answer
While interesting, I wonder how much this has changed recently. The WP article claims that Google is super secretive about their datacenters, but their staff seem to publish alot of public whitepapers about the hardware. IIRC, some whitepapers refute the Wikipedia claims above (Many nodes don't have local drives, etc.). – Stefan Lasiewski Aug 15 '10 at 17:31
This include search engine part, while he only want to build a crawler. – radius Aug 15 '10 at 17:45
up vote 0 down vote

A couple of thousand, I would assume. Good luck with that.

share|improve this answer
Thks.... – Alex Mathew Aug 15 '10 at 17:08
3  
way too low.... – Joris Aug 15 '10 at 17:29
@Joris, may be not to only build a crawler. Thousands are probably enough – radius Aug 15 '10 at 17:44
Yeah. do not forget that MOST of the load with google is answering queries, not crawling. 1000 decent lower end servers (dual socket etc.) should easily handle a LOT of pages. – TomTom Aug 15 '10 at 18:17

Not the answer you're looking for? Browse other questions tagged or ask your own question.