Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

So, I have a several APP servers running Apache which handle the frontend traffic of my web applications.

It's a high traffic environment, and my infrastructure at Amazon can auto-scale these APP servers and add more to the rotation as needed.

These web servers connect to a hefty dedicated MySQL server.

Once in a while, particular if we get hit with a large burst of traffic, things will lock up.

When this happens, the app servers get locked up, and the max connections on the MySQL database soars from about 10-15 to 1024 (the limit)

Things will continue to be jammed up til I reboot the MySQL server. Then, everything returns to normal on the DB and the web servers eventually settle down.

My very broad question is, what is going on here?

My more specific questions are:

  1. Why do these connections remain in-use rather than timing out?
  2. Here's more of a "what came first, the chicken or the egg" kind of question: do you think the app servers are messing up, or do you think the database has errors then makes the app servers lock up?

Thanks guys!

Dave

share|improve this question
    
Are you really using "APP" as an acronym for something? Or just for inappropriate emphasis? –  womble Jul 30 '11 at 10:22
    
Dpeends what you mean by 'once in a while'. Are you certain that the high traffic comes first, or could you be reading the latency spike as a cause instead of a symptom? If you've got CloudWatch running, see if there's a solid frequency to these occurrences. If so, then possibly some in-app scheduled task is failing to unlock rows or tables. What does the slow query log on your RDS instance say is happening around that time? –  SmallClanger Jul 30 '11 at 11:18

1 Answer 1

There are a number of possible causes for this. It's likely you have some sort of locking issue. This is especially probable if you're using the MyISAM storage engine, which does table-level locking. When the problem occurs, type "show processlist" or "show full processlist" at a command line to see all active queries and their state. Once you know what's causing the issue, you can start to figure out the best way to address it, e.g. change storage engines, reschedule a batch job, move your drupal cache out of the database server to memcached, etc.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.