So I'm pretty confused here, figured I'd throw this out here and see if anyone else has any good ideas. We suffered a pretty big outage two nights ago, and I'm still searching for the root cause. I believe I know WHAT happened and basically how it happened, but I haven't figured out the absolute specific yet.
Basically our app servers connect to a db server (db01). db01 had a stuck query that was stuck in the state of "copying to tmp table", then db01 went through it's normal backup procedure, tried to flush tables with read lock, only the read lock was never released as the flush tables were stuck because of the stuck query. So all queries coming in just got stuck in the processlist as waiting for readlock. Then the connections maxed out.
At the moment the connections maxed out ("Too many connections"), the site went down. That's expected. However, 2 minutes later, this seemed to have affected more services, primarily LDAP. We could not log in to any servers because the ldap server could not be reached. However, LDAP was fine. LDAP never showed any errors, the process never died, etc.
So I came to the conclusion that somehow the mysql/connections issue caused a major network slowdown. We use nssldap and authldap for vpn - they have timeouts set. So they were timing out because of the network being slow. There were other minor issues that were not normal (nagios passive checks not being sent, etc).
The only thing the LDAP server has in common with the app/db servers is that they both live on the same private network. The app and db servers communicate via private network. So it's as if the connections got maxed and this massive loop was created, bringing everything to a slowdown.
The only thing that fixed the issue was unplugging one of the app servers from the private network. I made the decision to have this server unplugged because it's swap usage was abnormally high, and our hosting provider did not a significant spike (public) on that server about 20 minutes before the issue started. 9 minutes after this server was unplugged, services seemed to return to normal.
Sorry, this was very long winded. I guess I'm just wondering if anyone has any idea what could have caused such behavior? Unfortunately there is no bandwidth monitoring on the private network switch, so I cannot determine if traffic was abnormal during this time.