Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Unfortunately we had to move to using KeepAlive after a series of ddos attacks on our server. I run a web game so pages are generally small and largely text, so having KeepAlive off worked well for many years. Things didn't seem much different (apart from being able to handle the ddos attacks better) after some quick configuration changes and server load stays fairly low at on average about 0.50 with memory usage on about 10% (24GB total).

About once a day however apache just hangs and won't respond for about 15 minutes. Even throughout the day, sometimes pages take 5 seconds to load and I notice in the apache status the only thing on the scoreboard are keepalive (read) requests, no waiting for connection, reading reply, closing connection etc, just about 100+ read requests. Then it sorts itself out. It just seems random when this happens, although it is normally during a busy time of the day neither the server load nor the memory is high nor is apache running out of available request slots.

My configuration changes are as follows:

ServerLimit 1024
MaxClients 1024
Timeout 2
KeepAliveTimeout 5
KeepAlive On
MaxKeepAliveRequests 100
MinSpareServers 100
MaxSpareServers 200

I hope someone has some insight into this or can suggest what logs / processes I can check to see why this is happening.

Thanks in advance.

EDIT: Incase this helps...

56 requests/sec - 116.2 kB/second - 2126 B/request
105 requests currently being processed, 154 idle workers

Later in the day I'd expect double that.

EDIT 2: KeepAliveTimeout changed from 5 to 2

EDIT 3: It just happened again. This time I was around to see it. Apache was unresponsive and wasn't taking up any memory. There were 250 connections from 127.0.0.1 and that never happens. After an Apache restart all was fine. Very odd!

share|improve this question
    
Add your KeepAliveTimeout setting to this info –  troyengel Nov 25 '10 at 15:16
1  
You mentioned you're running CPanel only offhandedly in a comment to one of the answers. CPanel changes things quite a bit. You should mention that you're running it in the question. –  BMDan May 21 '11 at 19:07
    
What are your load numbers, and what's the division of load (e.g. user vs. iowait, f. ex.), when things lock up? If you're running top, try hitting 1 to get per-CPU stats. –  BMDan May 21 '11 at 19:10
add comment

1 Answer

A few things you can do or check:

  1. We had a similar issue where Apache 1.3 didn't properly close connections which left them lingering forever and eventually would fill up all the client slots. To get around this we just restart Apache once a day in a cron script. Your issue sounds different as you mention it clears up eventually so daily restarting may not help anything.
  2. I assume you are never reaching the "MaxClients" in concurrent connections?
  3. What is your "KeepAliveTimeout" setting? The default is 15 secs which is large for most sites with 1 or 2 seconds typically working better (especially since your "Timeout" is set to 2 anyways).
  4. Check "netstat -an" for obvious signs of DOS from one client. I've found that often when there are non-obvious site issues it is due to one client hammering the site continually with 20 requests/sec. Check to make sure you are not capping the server's bandwidth (which is easy if you only have a 10 Mbps connection). I use my provider's on-line bandwidth monitoring graphs but there should be command line ways to do this as well (ifconfig might let you). There are various Apache or server level solutions to automatically preventing DOS attacks (whether innocent or not).
  5. Check "top" during the issue and look at the CPU, IO, and memory usage. Make sure you are not swapping memory (which you shouldn't if you are only at 10% memory usage).
  6. Speaking of memory....10% memory usage of even a 24GB RAM server seems low. I have several servers with 1-4GB of RAM and they are all in the 75-90% memory usage (much of that is in the OS cache however). I suppose this would depend on your Apache setup and server usage.
  7. Make sure there are not other dead-locks in your application layer that are causing the issue (like a database). For example, check your Apache "server-status" page and if that loads quickly but your regular site's page loads slowly then the issue is likely not the Apache server.
  8. Check logs in "/var/log", particularly the Apache error logs and the "message" log for relevant messages. If you don't have these, or other application logs enabled, enable them at least temporarily.
  9. Check the system limits of things like the number of open files allowed at a time (ulimit -n). The default settings of many servers/OSes are not necessarily configured for a high-volume server.
  10. If all else fails challenge your assumptions about what the issue could be and double-check items that "can't" fail or that you've already checked.
share|improve this answer
    
Wow, amazing response. I've changed KeepAliveTimeout to 2 (it was on 5) and ulimit -n 10000 and I'll see how it pans out. I'll be sure to check the logs and top if / when it happens again. Thank you. –  Ant Nov 25 '10 at 15:23
    
It just happened again. This time I was around to see it. Apache was unresponsive and wasn't taking up any memory. There were 250 connections from 127.0.0.1 and that never happens. After an Apache restart all was fine. Very odd! –  Ant Nov 26 '10 at 19:10
    
Are there any other services/apps running on the server that access Apache (like a cache/proxy) or do you have any forwarding/proxy rules setup in Apache? These may be the cause of all the localhost connections. –  uesp Nov 27 '10 at 2:54
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.