I am currently thinking of how to best set up my server architecture for a high traffic website which scrapes data from the internet and provides a REST API.
One idea is renting root servers in different data centers, another idea is creating my own datacenter. I do have a few technical things that are uncertain and maybe you guys can enlighten me.
We have fiber connections that allow 622 Mbit/s connections. So I have this coming into my data center, so far so good!
Since there is Gigabit switches I also think we don't have a big problem so far, the network infrastructure is still faster than my 622 Mbit/s backbone.
I am considering buying a bundle of 10 used Dell PowerEdge with XEON QuadCore L5335 2.0GHz and 4GB Ram (to deal with all the scraping), I put them in a rack and hook them up to the switches, cooling and redundant power supply. I will have one more powerful machine as my MySQL datacenter. Now comes the first question, how do I most efficiently administer the 10 web servers? I basically only need them to run Tomcat on a basic linux box which to talk to my sql server. Is there some sort of central administration tool I can use to deploy the preconfigured operating systems to the server?
Now if somebody calls my website the a record points to my IP address, how do I now balance the load so the 10 web servers are stressed equally? Also how do I ensure that these 10 webservers all have the same share of my bandwidth, is that done with managed switches?