Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a small network with about 8 linux servers, a Cisco 2600 router, and a Cisco 3500XL switch. The router and switch have been configured and working properly for years.

About 6 hours ago the time to establish a connection via certain protocols skyrocketed. Connecting to a server via SSH can take a couple minutes to establish. But once the connection is made, it works normally. Copying files via scp is fast as well, but making the initial connection takes forever. Same with telnet.

However, connection via HTTP or HTTPS are perfectly fine. They cruise along like normal. Also SFTP seems to be fine as well.

SNMP connections also seem to be affected. My Cacti monitoring server has stopped working properly with timeout errors in the logs. PHPSVR: Poller[0] Maximum runtime of 292 seconds exceeded for the Script Server. Exiting. It gets intermittent, but mostly failed results from the hosts, however it is fairly reliable in reporting the router and switch cpu and memory.

The strange thing about the snmp monitoring of the switch is that some switch ports continue to be reported and show up in the cacti graphs, and others stop about 6 hours ago. But logging into the switch CLI shows activity on those ports.

My colo facility claims they have no networking issues at this time. I do think it is something in my network, but can't figure out what. I'm not seeing any significant traffic spikes and I can perform DNS queries without problems (dig queries taking around 34 msec). The delay happens even when connecting between hosts inside the network.

Any pointers on how to proceed troubleshooting this? I've reloaded both the switch and router and the problem continues. The following shows the output of ssh -vv with some comments that indicate how many seconds of a delay there were at different points in the login process. https://gist.github.com/963682

share|improve this question
add comment

1 Answer

up vote 6 down vote accepted

Your name resolution has broken.

share|improve this answer
    
As I indicated in my question, dig queries are returning in milliseconds. –  Tauren May 10 '11 at 5:13
2  
Even reverse resolution? –  Ignacio Vazquez-Abrams May 10 '11 at 5:14
    
OMG -- I think you're right. It looks like one of my DNS servers is down, while the others are still running. Hopefully this will be a quick fix! –  Tauren May 10 '11 at 5:17
    
+1 Clarvoyant. Not really, but good call Ignacio –  Mike Pennington May 10 '11 at 7:21
1  
DNS and name resolution is always one of the first things to carefully check when things start to fail suddenly and there is no apparent reason. –  SvW May 10 '11 at 7:34
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.