we are designing a cluster of 2xnginx, 2xapache, 2xmemcache, 2xmysql servers for a high traffic web site.
Current web site runs on 1 dedicated web server and 1 dedicated db server. the current web server sees a peak of ~60Mbps incoming traffic and 25Mbps outgoing traffic. the current db server sees a peak of ~60Mbps outgoing traffic and a peak of ~20Mbps incoming traffic.
For our new setup we were thinking of having 1Gbit connections between the various nodes but the provider supports 1Gbit only if we dont use KVM cards as they only run at 100Mbit.
Based on the figures above I have the feeling that 1Gbit connectivity will not give us any benefit and the above limitation is not a show stopper.
Can any seasoned web cluster admins confirm or advise against this?