Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

So I'm pretty confused here, figured I'd throw this out here and see if anyone else has any good ideas. We suffered a pretty big outage two nights ago, and I'm still searching for the root cause. I believe I know WHAT happened and basically how it happened, but I haven't figured out the absolute specific yet.

Basically our app servers connect to a db server (db01). db01 had a stuck query that was stuck in the state of "copying to tmp table", then db01 went through it's normal backup procedure, tried to flush tables with read lock, only the read lock was never released as the flush tables were stuck because of the stuck query. So all queries coming in just got stuck in the processlist as waiting for readlock. Then the connections maxed out.

At the moment the connections maxed out ("Too many connections"), the site went down. That's expected. However, 2 minutes later, this seemed to have affected more services, primarily LDAP. We could not log in to any servers because the ldap server could not be reached. However, LDAP was fine. LDAP never showed any errors, the process never died, etc.

So I came to the conclusion that somehow the mysql/connections issue caused a major network slowdown. We use nssldap and authldap for vpn - they have timeouts set. So they were timing out because of the network being slow. There were other minor issues that were not normal (nagios passive checks not being sent, etc).

The only thing the LDAP server has in common with the app/db servers is that they both live on the same private network. The app and db servers communicate via private network. So it's as if the connections got maxed and this massive loop was created, bringing everything to a slowdown.

The only thing that fixed the issue was unplugging one of the app servers from the private network. I made the decision to have this server unplugged because it's swap usage was abnormally high, and our hosting provider did not a significant spike (public) on that server about 20 minutes before the issue started. 9 minutes after this server was unplugged, services seemed to return to normal.

Sorry, this was very long winded. I guess I'm just wondering if anyone has any idea what could have caused such behavior? Unfortunately there is no bandwidth monitoring on the private network switch, so I cannot determine if traffic was abnormal during this time.

share|improve this question

migrated from stackoverflow.com Jan 26 '11 at 18:49

This question came from our site for professional and enthusiast programmers.

4  
That's one big wall of text. –  shamittomar Jan 26 '11 at 18:34
    
What is in the logs of the crazy app server? –  sunn0 Jan 26 '11 at 18:42
    
It looks to me like your internal network was indeed flooded with mysql requests or somehow your mysql backup interacts with the LDAP server, could you throw some RRD graphs into the info to have a clearer view of the situation? –  lynxman Jan 26 '11 at 19:13
    
Couple questions. First, you say you couldn't log into any machines, but then you say LDAP was fine. Were you able to log into that machine? If so, did you do any ping tests to your app or db server to check any latency issues? –  slillibri Feb 1 '11 at 18:19
    
@slillibri - No - I was not able to log into any machine, including the LDAP machine. Sorry if that was confusing. We only have one machine that you can log into without a VPN, using pam_abl. I was occasionally able to reach a login prompt on this machine (after a few minutes of waiting), but once I entered my password, it would just timeout and I'd get dropped. This machine is NOT the ldap machine. –  user68356 Feb 1 '11 at 22:21

3 Answers 3

Private network, eh? Would I be remiss in wondering whether this was behind a SOHO-class switch device? I could believe that those persistent TCP connections piling up might have tripped a threshold in the switch for max number of connections tracked and caused it to enter a less-than-ideal state. If your switch keeps logs, I'd start there. Hope that helps!

share|improve this answer

Your issue seems to be with the ldap server. Is there only one ldap server?.

Check the logging for slapd.

share|improve this answer
    
Yes - only one ldap server (now there's two :). There was nothing of note in the slapd logs. –  user68356 Feb 1 '11 at 22:22

For bandwidth monitoring you can install iftop on every server.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.