I just finished developing a web crawler that essentially allows users to crawl their site and it informs them on errors through out their site. However my app is very server resource intensive. it is a ruby app using rails, mongo, a handful of gems and runs the crawl in the background using delayed job. Testing the crawler on my Mac which has an Intel Core 2 Duo @ 2.4GHz and 4gb of ram I can only crawl 5 websites at time with an average speed of crawling 500 pages an hour. If I try to crawl anymore then 5 at a time the process just gets killed and the crawl fails. When crawling 5 websites at once my CPU usage is at 95% - 100% the whole time, and I only use about 600mb of ram.
I am in the market for a server and my budget is under $1,000. so my question is; Do you think it will be better to get a server with more cores with a lower GHz or a server with less cores and higher GHz?
I have explored a lot of options such as amazon aws, dedicated servers, multiple small VPS, etc. And I feel it would be most cost efficient for me to purchase a server or two and have it racked at a data center. I have access to half a cabinet that I can get for well below market price through a friend. I just don't know how to best spend my money, any insight would be greatly appreciated.