Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I put up a diagram of our network and equipment here: http://imgur.com/bp7l0

Symptoms

  • Twice in 3 weeks, we have experienced intermittent network failures. These usually manifest as a timeout on a web page, or sometimes missing site content (stylesheets don't load, e.g.). The problem has occurred on all floors in our building. Normally a forced refresh of the page will fix.
  • Tracert to the web page will work every time I've tried it, even when I am consistently getting page load errors on every second or third new URL. Sometimes the second hop fails, although this may simply reflect that ICMP is blocked by that IP address.
  • Some users have experienced slow network performance.
  • Meanwhile, overall network usage appears to be normal, well below the limit of the 10 MB pipe.
  • Doing a speedtest at speedtest.net gives normal results--a little below the limit, as expected due to not being the sole user on the network.
  • Once when I was out and received an emergency call, I suggested to our IT staff to restart either the router or the firewall. They restarted the firewall, which apparently cleared up the problem for a few weeks.

Overview of the network See diagram here: http://imgur.com/bp7l0.

We have two network connections, a primary and a failover connection. Both network connections are plugged in directly to the firewall. From the firewall to our primary switch, the connection is copper, cat5e. The port is configured to full duplex 100 megabit. Some users are plugged in directly to this switch via an IDF, other users on different floors have a separate switch, connected to the primary switch via fiber, and go from there to an IDF.

During the window when I was able to observe the firewall, the failover connection does not appear to have been engaged. The way it works is when a bandwidth threshold is reached (10 MB) the secondary connection kicks on. It also is used if the primary connection dies completely.

Troubleshooting already performed

  • Connected to managed switch, looked at statistics for port with copper link. Everything seems normal, but I don't know 100% what to look for. I looked for drops and collisions; both were low on this particular port. Not sure of the time range for the data collection without an external logging server.
  • Watched statistics on the firewall for a while. Observed bandwidth utilization, error reports. No unusual flood of connections.

My question

What should I investigate next, and what steps should I take? Any guesses as to what type of problem I'm encountering here--cable, switch, firewall, or ISP? What are some tools that can help me test the various components involved here? The problem is tough because it is intermittent. I think I can use SNMP to collect data from the switch for a longer time period, as well as for the firewall, but that would be a big project with a lot to learn for me. Are there any configuration changes worth making? Adjusting a timeout that I can easily do globally?

Any help would be much appreciated. Thanks!

share|improve this question

3 Answers 3

up vote 2 down vote accepted

Without getting into a lot of very specific guidance, which I'm sure others will offer:

  1. Don't make any changes without knowing that the specific component being changed is the cause of the problem and that the change will resolve the problem. Making random changes in the hopes that something will work is analogous to driving a car blindfolded. You may fix the problem but it will only be due to sheer luck and you'll never know what the real cause was.

  2. You've already hit on something: the firewall. If it's the case that rebooting the firewall resolved the issue last time then that's where I'd start. Take a look, if possible, at whatever counters are available on the firewall such as CPU and memory usage, traffic loads, dropped packets, etc. Put a network sniffer on the inside and outside of the firewall and run some tests from a client machine. Do you see packets dropped on the inside? On the outside? What's the timing of the traffic look like as it enters and exits the firewall? Is there a marked delay?

From there I'd move to the switch or the router and perform the same tests, depending on the results of testing the firewall.

share|improve this answer
    
That's a good point. I should add that I did check the load on the firewall, and it was normal when the problem recurred. I checked and I can set up a log on the switch itself, so I will do that for a while. Thanks for the ideas! –  Quinten Oct 26 '11 at 15:09

Isolation of an intermittent problem can be troublesome. It would be a good idea to install some kind of monitoring to frequently check availability and timing at several layers.

Run frequent ICMP pings to

  • an internal machine separated by switches
  • the internal interface of your firewall appliance
  • the routers of your ISPs (take the first router outside the organisation that you see along the way when using traceroute)
  • some infrastructure servers of your ISPs (e.g. the name servers or the mail servers)
  • a public website with a high availability (e.g. google.com)

Also add some frequent HTTP requests using wget with recursive retrieval and --delete-after statements - write datetime-stamps, execution timings and errors to a log.

Combining this information should help you encircle the problem area.

share|improve this answer

What version of firmware are you using on the Firebox? I have seem this issues with these recently. We upgraded to V12 and this resolved the issue.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.