Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm running into trouble on a Centos 5 webserver every night when yahoo crawls the site and takes it down in the process. I've monitored the server while it's under the yahoo flood and the memory is never exhausted so it never hits the swap partition. The load never seems to exceed 1.5. I wouldn't think that there was a problem unless apache just stopped accepting incoming connections for awhile.

I suspect that the problem has something to do with the server not accepting any new connections, but this is murky territory for me.

Can someone help me figure out what's going on here or direct me to some material that I can learn from.

EDIT:

I ran the netstat command provided by Glenn Kelley in a while loop that ran every 20 seconds. When I noticed that I started to get more than 5 ESTABLISHED connections at a time (my norm) the CLOSE_WAIT number would start climbing to around 40 - 60. At this point, I added 'mysqladmin proc stat' to the while loop and noticed that I had a backup script that was running mysqldump at that time.

Apparently, mysqldump was locking all tables in the database which prevented writes from happening. Because of this and the coincidentally timed yahoo crawl (40 pages per minute) the server would eventually stop accepting incoming requests to Apache after MaxClients was reached.

I'm now looking into using MySQL replication to allow for non-locking backups or finding a better time to run my backups. Using mysqldump with --skip-lock-tables isn't a option for me and I can't use mysqlhotcopy as I have a couple of InnoDB tables.

share|improve this question
add comment

4 Answers

up vote 1 down vote accepted

There are a few things you can do here.

  1. Have you fine-tuned the server to allow for x amount of connections from 1 IP ?
  2. Installing mod_evasive should help a great deal.
  3. Run netstat to see if any other traffic is hitting the system:

netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

That command will list every ip hitting the server and how many connections it has.

  1. Edit using your favorite editor (FIRST BACKUP PLEASE) this file /usr/local/apache/conf/httpd.conf

Change the major options: (tune to your liking and needs of course)

MaxKeepAliveRequests 50 (100) KeepAliveTimeout 60 (30)

Additional edits that may help would be to each of the following variables Timeout KeepAliv MinSpareServers MaxSpareServers MaxClients

CHANGE 1 AT A TIME - this way you know what your changing and ... if one or another hurts or helps.

Adding additional Anti-DDOS tools may help as well. The scripts from InetBase may help - depending upon your usage.

http://www.inetbase.com/scripts/ddos/

And finally - a simple reverse proxy may really make a difference. One such would be NGINX

Are you using a control panel by any chance? if so - knowing which one - may help the community help you a bit more...

share|improve this answer
add comment

Ive seen something like this before..

My guess is that you have a bad link that never finishes.. so the process waits.. and waits and waits.. The next query hits the same link.. and waits and waits and waits..

Basically it keeps doing this until all processes are in use.. and you can accept anymore :(

Check the scoreboard ( to see what urls are stuck before restarting the server )

share|improve this answer
add comment

Also make sure mysql logging is enabled and check that max_connections isn't being reached (assuming your web app is using mysql).

share|improve this answer
add comment

It'd be helpful if you provided details of how your webserver is configured, the number of connections, the number of yahho connections, the average and max duration.

Sounds like you've got a bad combination set for maxClients and MaxKeepAliveRequests / KeepAliveTimeout

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.