3

I have a small network with about 8 linux servers, a Cisco 2600 router, and a Cisco 3500XL switch. The router and switch have been configured and working properly for years.

About 6 hours ago the time to establish a connection via certain protocols skyrocketed. Connecting to a server via SSH can take a couple minutes to establish. But once the connection is made, it works normally. Copying files via scp is fast as well, but making the initial connection takes forever. Same with telnet.

However, connection via HTTP or HTTPS are perfectly fine. They cruise along like normal. Also SFTP seems to be fine as well.

SNMP connections also seem to be affected. My Cacti monitoring server has stopped working properly with timeout errors in the logs. PHPSVR: Poller[0] Maximum runtime of 292 seconds exceeded for the Script Server. Exiting. It gets intermittent, but mostly failed results from the hosts, however it is fairly reliable in reporting the router and switch cpu and memory.

The strange thing about the snmp monitoring of the switch is that some switch ports continue to be reported and show up in the cacti graphs, and others stop about 6 hours ago. But logging into the switch CLI shows activity on those ports.

My colo facility claims they have no networking issues at this time. I do think it is something in my network, but can't figure out what. I'm not seeing any significant traffic spikes and I can perform DNS queries without problems (dig queries taking around 34 msec). The delay happens even when connecting between hosts inside the network.

Any pointers on how to proceed troubleshooting this? I've reloaded both the switch and router and the problem continues. The following shows the output of ssh -vv with some comments that indicate how many seconds of a delay there were at different points in the login process. https://gist.github.com/963682

1 Answer 1

6

Your name resolution has broken.

6
  • As I indicated in my question, dig queries are returning in milliseconds.
    – Tauren
    May 10, 2011 at 5:13
  • 3
    Even reverse resolution? May 10, 2011 at 5:14
  • OMG -- I think you're right. It looks like one of my DNS servers is down, while the others are still running. Hopefully this will be a quick fix!
    – Tauren
    May 10, 2011 at 5:17
  • +1 Clarvoyant. Not really, but good call Ignacio May 10, 2011 at 7:21
  • 1
    DNS and name resolution is always one of the first things to carefully check when things start to fail suddenly and there is no apparent reason.
    – Sven
    May 10, 2011 at 7:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .