So, I have a several APP servers running Apache which handle the frontend traffic of my web applications.
It's a high traffic environment, and my infrastructure at Amazon can auto-scale these APP servers and add more to the rotation as needed.
These web servers connect to a hefty dedicated MySQL server.
Once in a while, particular if we get hit with a large burst of traffic, things will lock up.
When this happens, the app servers get locked up, and the max connections on the MySQL database soars from about 10-15 to 1024 (the limit)
Things will continue to be jammed up til I reboot the MySQL server. Then, everything returns to normal on the DB and the web servers eventually settle down.
My very broad question is, what is going on here?
My more specific questions are:
- Why do these connections remain in-use rather than timing out?
- Here's more of a "what came first, the chicken or the egg" kind of question: do you think the app servers are messing up, or do you think the database has errors then makes the app servers lock up?