Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 7 down vote favorite

So I'm pretty confused here, figured I'd throw this out here and see if anyone else has any good ideas. We suffered a pretty big outage two nights ago, and I'm still searching for the root cause. I believe I know WHAT happened and basically how it happened, but I haven't figured out the absolute specific yet.

Basically our app servers connect to a db server (db01). db01 had a stuck query that was stuck in the state of "copying to tmp table", then db01 went through it's normal backup procedure, tried to flush tables with read lock, only the read lock was never released as the flush tables were stuck because of the stuck query. So all queries coming in just got stuck in the processlist as waiting for readlock. Then the connections maxed out.

At the moment the connections maxed out ("Too many connections"), the site went down. That's expected. However, 2 minutes later, this seemed to have affected more services, primarily LDAP. We could not log in to any servers because the ldap server could not be reached. However, LDAP was fine. LDAP never showed any errors, the process never died, etc.

So I came to the conclusion that somehow the mysql/connections issue caused a major network slowdown. We use nssldap and authldap for vpn - they have timeouts set. So they were timing out because of the network being slow. There were other minor issues that were not normal (nagios passive checks not being sent, etc).

The only thing the LDAP server has in common with the app/db servers is that they both live on the same private network. The app and db servers communicate via private network. So it's as if the connections got maxed and this massive loop was created, bringing everything to a slowdown.

The only thing that fixed the issue was unplugging one of the app servers from the private network. I made the decision to have this server unplugged because it's swap usage was abnormally high, and our hosting provider did not a significant spike (public) on that server about 20 minutes before the issue started. 9 minutes after this server was unplugged, services seemed to return to normal.

Sorry, this was very long winded. I guess I'm just wondering if anyone has any idea what could have caused such behavior? Unfortunately there is no bandwidth monitoring on the private network switch, so I cannot determine if traffic was abnormal during this time.

share|improve this question
4  
That's one big wall of text. – shamittomar Jan 26 '11 at 18:34
What is in the logs of the crazy app server? – sunn0 Jan 26 '11 at 18:42
It looks to me like your internal network was indeed flooded with mysql requests or somehow your mysql backup interacts with the LDAP server, could you throw some RRD graphs into the info to have a clearer view of the situation? – lynxman Jan 26 '11 at 19:13
Couple questions. First, you say you couldn't log into any machines, but then you say LDAP was fine. Were you able to log into that machine? If so, did you do any ping tests to your app or db server to check any latency issues? – slillibri Feb 1 '11 at 18:19
@slillibri - No - I was not able to log into any machine, including the LDAP machine. Sorry if that was confusing. We only have one machine that you can log into without a VPN, using pam_abl. I was occasionally able to reach a login prompt on this machine (after a few minutes of waiting), but once I entered my password, it would just timeout and I'd get dropped. This machine is NOT the ldap machine. – user68356 Feb 1 '11 at 22:21

migrated from stackoverflow.com Jan 26 '11 at 18:49

4 Answers

up vote 1 down vote

Private network, eh? Would I be remiss in wondering whether this was behind a SOHO-class switch device? I could believe that those persistent TCP connections piling up might have tripped a threshold in the switch for max number of connections tracked and caused it to enter a less-than-ideal state. If your switch keeps logs, I'd start there. Hope that helps!

share|improve this answer
up vote 0 down vote

Your issue seems to be with the ldap server. Is there only one ldap server?.

Check the logging for slapd.

share|improve this answer
Yes - only one ldap server (now there's two :). There was nothing of note in the slapd logs. – user68356 Feb 1 '11 at 22:22
up vote 0 down vote

For bandwidth monitoring you can install iftop on every server.

share|improve this answer
up vote -1 down vote

Mysql apparently ate all of your CPU and not network , mysql and database are known of eating up CPU resources , try migrating to something like NOSQL if you're having large amounts of records/data , or try caching mysql queries results if no frequent updates are happening !

share|improve this answer
1  
Yes, but that doesn't explain how LDAP, which runs on an entirely different server, became unavailable. – user68356 Jan 26 '11 at 19:49
LDAP is used for reading and editing directories over an IP network , if the destination server is the one with the suffering CPU , LDAP will just wait for that CPU to process what it asked for over the network ... – Ronan Dejhero Jan 26 '11 at 19:58
Very interesting theory. So you're suggesting that one server, lets say the app server I am questioning, attempted an LDAP lookup. The app server was so heavily loaded that it began the request but could not finish it. Therefore, the LDAP server sat waiting for the request to finish, and was not accepting new requests. Once I had the server in question unplugged from the private network, the LDAP server was released. Is this essentially what you're saying? This makes a lot of sense, but has a few flaws. – user68356 Jan 26 '11 at 20:14
Flaw 1: During the outage (~2 hours), all servers were disconnected from the private network on two occasions. This would essentially sever any connection to LDAP, thus releasing it, yes? So this would mean the very same server would have to be the first to request when it came up again. – user68356 Jan 26 '11 at 20:16
Flaw 2: The outage lasted 2 hours. It seems unlikely that an ldap query could be stuck for this long. Perhaps I am wrong. – user68356 Jan 26 '11 at 20:16

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.