I'm running into trouble on a Centos 5 webserver every night when yahoo crawls the site and takes it down in the process. I've monitored the server while it's under the yahoo flood and the memory is never exhausted so it never hits the swap partition. The load never seems to exceed 1.5. I wouldn't think that there was a problem unless apache just stopped accepting incoming connections for awhile.
I suspect that the problem has something to do with the server not accepting any new connections, but this is murky territory for me.
Can someone help me figure out what's going on here or direct me to some material that I can learn from.
I ran the netstat command provided by Glenn Kelley in a while loop that ran every 20 seconds. When I noticed that I started to get more than 5 ESTABLISHED connections at a time (my norm) the CLOSE_WAIT number would start climbing to around 40 - 60. At this point, I added 'mysqladmin proc stat' to the while loop and noticed that I had a backup script that was running mysqldump at that time.
Apparently, mysqldump was locking all tables in the database which prevented writes from happening. Because of this and the coincidentally timed yahoo crawl (40 pages per minute) the server would eventually stop accepting incoming requests to Apache after MaxClients was reached.
I'm now looking into using MySQL replication to allow for non-locking backups or finding a better time to run my backups. Using mysqldump with --skip-lock-tables isn't a option for me and I can't use mysqlhotcopy as I have a couple of InnoDB tables.