I put up a diagram of our network and equipment here: https://i.stack.imgur.com/BWeMo.jpg
Symptoms
- Twice in 3 weeks, we have experienced intermittent network failures. These usually manifest as a timeout on a web page, or sometimes missing site content (stylesheets don't load, e.g.). The problem has occurred on all floors in our building. Normally a forced refresh of the page will fix.
- Tracert to the web page will work every time I've tried it, even when I am consistently getting page load errors on every second or third new URL. Sometimes the second hop fails, although this may simply reflect that ICMP is blocked by that IP address.
- Some users have experienced slow network performance.
- Meanwhile, overall network usage appears to be normal, well below the limit of the 10 MB pipe.
- Doing a speedtest at speedtest.net gives normal results--a little below the limit, as expected due to not being the sole user on the network.
- Once when I was out and received an emergency call, I suggested to our IT staff to restart either the router or the firewall. They restarted the firewall, which apparently cleared up the problem for a few weeks.
Overview of the network See diagram here: https://i.stack.imgur.com/BWeMo.jpg.
We have two network connections, a primary and a failover connection. Both network connections are plugged in directly to the firewall. From the firewall to our primary switch, the connection is copper, cat5e. The port is configured to full duplex 100 megabit. Some users are plugged in directly to this switch via an IDF, other users on different floors have a separate switch, connected to the primary switch via fiber, and go from there to an IDF.
During the window when I was able to observe the firewall, the failover connection does not appear to have been engaged. The way it works is when a bandwidth threshold is reached (10 MB) the secondary connection kicks on. It also is used if the primary connection dies completely.
Troubleshooting already performed
- Connected to managed switch, looked at statistics for port with copper link. Everything seems normal, but I don't know 100% what to look for. I looked for drops and collisions; both were low on this particular port. Not sure of the time range for the data collection without an external logging server.
- Watched statistics on the firewall for a while. Observed bandwidth utilization, error reports. No unusual flood of connections.
My question
What should I investigate next, and what steps should I take? Any guesses as to what type of problem I'm encountering here--cable, switch, firewall, or ISP? What are some tools that can help me test the various components involved here? The problem is tough because it is intermittent. I think I can use SNMP to collect data from the switch for a longer time period, as well as for the firewall, but that would be a big project with a lot to learn for me. Are there any configuration changes worth making? Adjusting a timeout that I can easily do globally?
Any help would be much appreciated. Thanks!