2

I just finished developing a web crawler that essentially allows users to crawl their site and it informs them on errors through out their site. However my app is very server resource intensive. it is a ruby app using rails, mongo, a handful of gems and runs the crawl in the background using delayed job. Testing the crawler on my Mac which has an Intel Core 2 Duo @ 2.4GHz and 4gb of ram I can only crawl 5 websites at time with an average speed of crawling 500 pages an hour. If I try to crawl anymore then 5 at a time the process just gets killed and the crawl fails. When crawling 5 websites at once my CPU usage is at 95% - 100% the whole time, and I only use about 600mb of ram.

I am in the market for a server and my budget is under $1,000. so my question is; Do you think it will be better to get a server with more cores with a lower GHz or a server with less cores and higher GHz?

I have explored a lot of options such as amazon aws, dedicated servers, multiple small VPS, etc. And I feel it would be most cost efficient for me to purchase a server or two and have it racked at a data center. I have access to half a cabinet that I can get for well below market price through a friend. I just don't know how to best spend my money, any insight would be greatly appreciated.

3
  • How many processes does your program start? If more than one, I'd lean towards more cores. Have you profiled your app to see what's using so much CPU?
    – Starfish
    Jan 11, 2012 at 20:03
  • 2
    It completely depends on the design of your application. Do it use threading or lots of processes to take advantage of many cores, or not? If you wrote it, and you don't know, how can you expect us to know? If you can't get equipment to test on, then you probably just have to guess, or flip a coin.
    – Zoredache
    Jan 11, 2012 at 20:04
  • 1
    Are you sure the network connection is not the bottleneck of your setup? How is the system usage distributed across user, system, iowait? What was your load-average on the mac?
    – Nils
    Jan 11, 2012 at 21:07

2 Answers 2

4

It sounds like your application needs better design and lacks appropriate parallelism unless you're actually rendering those pages in some way, and even then I think you should be able to surpass 8 per second if you uncouple the downloading from the final render -- interactivity is not a priority.

You are best served by ensuring proper parallelism in your application and getting the equipment that will provide you with the most operations per second. That almost always means more processor cores. The basic calculation should be cores * gHz = score.

That said, certain types of processor may make a difference as well.

0

Sorry I'm not a mac user so I'm thinking of windows task manager that shows utilisation per core - if you have something like this available to you - check if you are seeing both cores maxed out, then yes you are better off getting more cores.

If you are seeing one core maxed out and the other pretty much idling, then GHz is more important to you.

Not the answer you're looking for? Browse other questions tagged .