Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a LAMP server that can be serving ~400 requests per second at times, the server configuration is:

CPU: Intel Quad Core Xeon X3430 (4 x 2.40 GHz, 8MB Cache)
RAM: 16 GB REG ECC DDR3
HD: 500 GB Enterprise Grade SATA II
OS: CentOS 64 Bit (Latest Stable)

Apache2 prefork config:

StartServers      128
MinSpareServers   16
MaxSpareServers   64
ServerLimit     1500
MaxClients      1500
MaxRequestsPerChild  10000

At peak times the server load is very high, cpu utilization around 90% and load average around 130

I'm not sure it's server hardware limit or my prefork config have something wrong - What could be the problem?

share|improve this question
    
Investigate a CDN, move the database onto a dedicated server, implement caching, etc. Lots of things you can do. –  ceejayoz Jan 13 '12 at 16:59
1  
It's dedicated server, Caching & CDN have been used. –  Roy Jan 13 '12 at 17:02
1  
Have you looked at server memory usage and IO load? –  Khaled Jan 13 '12 at 17:25
2  
@Roy If MySQL is already on its own dedicated server, you don't have a LAMP server, you have a LAP server. –  ceejayoz Jan 13 '12 at 17:27
    
Yes, my server us runing MySQL also, do you mean I should separate the database server? but the server do not have many MySQL query cos many page is cached. –  Roy Jan 13 '12 at 23:53

3 Answers 3

What is the average size of your httpd processes?

Run this command when the server is under load:

ps -ylC httpd --sort:rss | awk '{sum+=$8; ++n} END {print "Tot="sum"("n")";print "Avg="sum"/"n"="sum/n/1024"MB"}'

That will tell you approximately the average size of an Apache process.

Your MaxClients is probably way too high.

share|improve this answer

You are allowing Apache to spawn up to 1500 children to serve requests (ServerLimit / MaxClients) -- It's no wonder your server load (the number of processes waiting in the run queue) is getting enormous!

My first suggestions at 400 requests per second with the numbers you're quoting in your question would be "Move the MySQL server to its own box" or perhaps a better choice: "Add another web server and load balance your requests".
To help you figure out how to size the additional server follow the advice HDDP500 gave in their answer to figure the average size of an Apache process. Figure out how many apache processes you want to run on the new server and determine how much RAM you'll need to handle them. Remember to add a safety margin (a gig or two).

Figuring out CPU requirements (speed/number of cores) is a little more difficult - You will need to take into account how much "work" the server has to do to generate each page. A tool like XDebug can help you out here by telling you how long it takes to generate a page (you should perform this test on an unloaded server as well as one under load, but you should also not use a tool like XDebug on a production server in most cases.)

share|improve this answer
1  
+1 for this. Move MySQL. Get 2+ Apache nodes. Use HAProxy/Varnish to load balance. –  Tom O'Connor Jan 14 '12 at 11:07
    
you could probably even get away with DNS round-robin for a little while, but a real load balancing solution (I'll throw pf or pfsense into the ring) is definitely preferable –  voretaq7 Jan 15 '12 at 21:04

Take a look at your CPU interrupts and see where the bottleneck starts (via top); as mentioned before, It'd be a good idea to dial down your ServerLimit/MaxClients.

Depending on your consumption, this may (should) speed things up as you lessen the ability of the box to starve itself of CPU.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.