Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We do around 1000 req/second on our webserver and there are layers of caching and balancing before it reaches the final LAMP stack. We have 6 Apache (Prefork MPM; 2.2.12) + PHP (5.3.2) servers with common memcache to serve the final request in case its not a hit on varnish. Now the problem we are facing is with scaling. We are super critical about the response times and our reverse proxy disconnects from backend if backend dont respond within a second. With more requests/second, we are facing more and more upstream errors and our LAMP machines are unable to take the load.

We are rethinking of a new server architecture and now looking for some good alternatives.

There are no static images / no static content involved. The only 4 request that reaches these servers are (3 collecting data, 1 sedning data - max 2KB dynamic file)

We are looking into Apache (worker MPM) + PHP CLI nginx + PHP FPM

I am wondering if there are good tutorials or benchmarking reports I can refer and if there are any other configurations I could possible give a shot

Thanks Sparsh Gupta

share|improve this question
add comment

1 Answer

up vote 4 down vote accepted

This is very little info for a Server Stack:

  • are you asking for a hardware or software recommendation (assuming software)
  • why are the upstream servers failing? (CPU, Network, Disk IO, is the upstram of the upstream failing -- read the database or whatever kind of persistence you have)
  • you seem to be able to fail gracefully. You are doing it anyway if you timeout within a second and return something. You should have numbers on:
    • How much failure is acceptable?
    • When (date/time and No. of requests) did you pass the acceptable fail rate?

I am wondering if there are good tutorials or benchmarking reports I can refer and if there are any other configurations I could possible give a shot

Please, please don't rely on benchmarks that somebody else did on another use case. Please do the benchmarks for your data, your client behaviour, your system

May I assume that:

  • collecting data == POST (or something that stores data on the server)
  • sending data == GET (answering to the clients)

It doesn't actually matter but have you tried to seperate "collecting" servers from "sending" servers?

Client caching doesn't seem to help in your case since the collecting part can't possibly be cached.

Wild guesses on application architecture (software wise):

Are you using something like APC?

If it's possible for your application I'd recommend switching to some async model.

Very rough:

  1. Just submit the "collect" jobs to a queue and
  2. send back a HTTP 202 immediately with the content-location set set to where you expect the content to come up (or a 302/303 with location in case the clients are browsers)
  3. As long as you don't have word from the workers just return a 200 with some default content and
  4. if it's finished 200 with the actual content

Hardware Wise:

It's hard to tell but it seems that just throwing more Apache servers at it could solve the problem. You already have 6 why don't just add more and be done with it - doing some calculation wether developer time or hardware is cheaper shouldn't be too hard. In any case: I wouldn't throw away the "old" server right away, just assign them a lower priority in your load balancer.

Also you mention a "reverse proxy" (singular) have you tried scaling at that layer?

Overall I'd say: You seem to have the ability to scale out by throwing more LAMP servers at it. Why don't you just do so?

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.