This is very little info for a Server Stack:
- are you asking for a hardware or software recommendation (assuming software)
- why are the upstream servers failing? (CPU, Network, Disk IO, is the upstram of the upstream failing -- read the database or whatever kind of persistence you have)
- you seem to be able to fail gracefully. You are doing it anyway if you timeout within a second and return something. You should have numbers on:
- How much failure is acceptable?
- When (date/time and No. of requests) did you pass the acceptable fail rate?
I am wondering if there are good tutorials or benchmarking reports I can refer and if there are any other configurations I could possible give a shot
Please, please don't rely on benchmarks that somebody else did on another use case. Please do the benchmarks for your data, your client behaviour, your system
May I assume that:
- collecting data == POST (or something that stores data on the server)
- sending data == GET (answering to the clients)
It doesn't actually matter but have you tried to seperate "collecting" servers from "sending" servers?
Client caching doesn't seem to help in your case since the collecting part can't possibly be cached.
Wild guesses on application architecture (software wise):
Are you using something like APC?
If it's possible for your application I'd recommend switching to some async model.
- Just submit the "collect" jobs to a queue and
- send back a HTTP 202 immediately with the content-location set set to where you expect the content to come up (or a 302/303 with location in case the clients are browsers)
- As long as you don't have word from the workers just return a 200 with some default content and
- if it's finished 200 with the actual content
It's hard to tell but it seems that just throwing more Apache servers at it could solve the problem. You already have 6 why don't just add more and be done with it - doing some calculation wether developer time or hardware is cheaper shouldn't be too hard. In any case: I wouldn't throw away the "old" server right away, just assign them a lower priority in your load balancer.
Also you mention a "reverse proxy" (singular) have you tried scaling at that layer?
Overall I'd say: You seem to have the ability to scale out by throwing more LAMP servers at it. Why don't you just do so?