Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have an Apache webserver running on CentOS 5.6 (installed via yum). Today we have had an expected influx of new users -- however unexpectedly we seem to have hit a limit of 300 accesses per second -- I have a graph (which Stack overflow is not leting me upload), but it shows that three times today (when our site almost ground to a halt) the "accesses / second" hit 300 and stayed for a while. As I type the current figure is 292.23 and the max today is 300.41.

We are running PHP inside the apache process, and that is connecting to a MySQL server (different physical box). However we're happy that MySQL is responding fine.

We've upped our limits in the httpd.conf from the default 256, however this hasn't had an effect on the magic 300 figure. We've searched both the php.ini and httpd.conf file for 300 and cannot find it. Below is the prefork config we are now using:

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      400
MaxClients       400
MaxRequestsPerChild  4000
</IfModule>

If anybody has any clue to what is causing this 300 limit I'd greatly appreciate it.

share|improve this question

2 Answers 2

There are all sorts of system-performance-related possibilities, but the fact that the limit is such a nice, round number is rather suspicious -- you wouldn't expect such a neat limit if the bottleneck were, say, CPU or disk I/O (or even network). What's even more suspicious is that it's per second -- if it were a limit to Apache or MySQL, you'd expect a limit on concurrent connections or requests, but a limit of 300 per second sounds like something else.

What is sitting upstream from you, network wise? If there's any firewalls, load balancers, DDoS mitigation systems, or other L3+ devices up there, I'd take a poke at them to see if there's anything in there that could be causing it.

Do you know what anyone getting above your 300/second limit sees? (You might have to hammer the site yourself a bit to see what happens) Are there log messages anywhere that might give a clue?

share|improve this answer
    
We think we just have a DDoS mitigation system upstream - it's a rented server from fast hosts so we've not entirely sure. –  Dom Jul 18 '11 at 15:00
1  
Time to see what their support's made of and ask them. –  womble Jul 18 '11 at 15:05

After making the configuration changes did you do a graceful restart or shutdown/start? Changes to ServerLimit will not take affect if you do a graceful restart. A complete shutdown and start is required.

share|improve this answer
    
Yes we preformed a graceful restart, now have done a stop, followed by a start. However we our outside of our peek access time now, so I think we're going to have to find out tomorrow at 8am BST when our next peek time occurs. –  Dom Jul 18 '11 at 15:00
    
@Dom In that case, this would be a good time to install and use apachebench (ab which is in apache2-utils on Debian, probably a similar package on Centos). If you use it on the server locally, the load from ab itself might slow down the server, but it will help you rule out upstream network problems that womble suggest might cause it. Also, run it against both static content (like an image file) as well as against a PHP script to see if PHP (or MySQL) is causing the issue. –  DerfK Jul 18 '11 at 15:20

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.