Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Scenario: small business with around 40 users behind a Watchguard XTM 3.0 firewall and 20Mb leased line internet connection.

Problem: Users experience occasional dropouts in internet connectivity. These are particularly annoying during VOIP calls eg Skype as connection breaks. Browsing to internet sites is also affected when the dropouts occur. The dropouts are regular enough to be a business problem, though most of the time everything is fine.

Comments: We think the issue is at our end since calls to the same Skype recipients from elsewhere eg home broadband seem to work fine. The problem has also persisted through an upgrade from ADSL to leased line. However we would like to know definitively if the problem is on the LAN or the WAN. Switches are currently unmanaged but shortly to be replaced with new managed switches. The dropouts occur for users anywhere on the LAN as far as we can tell.

Any idea on how to trace the cause of the dropouts? I've wondered if there is a way to test continuity of connection within the XTM? You can see easily that there are no long dropouts but how can we test for short dropouts (but long enough to break a Skype call)?

The more likely reason is something on the LAN - how do we narrow this down without leaving people disconnected for long periods?

Tim

share|improve this question
    
It could be a wrong MTU value on your router. Skype works but occasionally it drops calls. –  Sergius Feb 13 at 16:16

3 Answers 3

Finding the source of this kind of problem can be extremely frustrating, especially if they are rare. However, this is how I approach intermittent network problems

  1. Map the network to the best of your ability
  2. Identify potentially problematic systems
  3. Create a (preferably automated) monitoring solution to identify where the problem is located
  4. Handle the problem.

Step 1 and 2 should be relatively straight forward. A drawing on a whiteboard with the complete path and the involved systems is helpful. For step 3 I tend to use Nagios or other longterm monitoring solutions. There are many plugins for nagios which may be useful, and you can configure it to monitor many properties of the systems with a very high resolution from your NOC. The monitoring has two purposes. One is to gather information for later debugging, but it also informs you about problems which lets you correlate them more easily to sources. When it comes to intermittent network connectivity issues I make sure to configure the routing monitoring and connectivity tests to all systems along the path.

Once I have found a solution to the problem I deploy it, and leave the monitoring in place until I am confident that the problem has been resolved.

By the way, unmanaged equipment has no place in a production network, as you have probably figured out by now. Debugging problems in a LAN without access to at least SNMP on the switches is a huge headache. And if you are unlucky a single patch between two network ports somewhere in the network is enough to make your network crash and burn...

share|improve this answer
1  
I would go as far as leaving monitoring on indefinitely, especially if it does not strain the resources too badly. This way, you are likely to be able to use this information for debugging recurrences and related / similar problems. –  the-wabbit Feb 13 at 14:28
1  
BTW, it is easy enough to diagnose a network loop, if you ever seen one. It obviously will be significantly more difficult to find the looping cable in a complex network full of unmanaged switches. Also, if you have your managed switches configured correctly, users would not be able to disrupt network connectivity by patching loops in the first place. I can't say unmanaged switches "have no place" in a production network - often enough there is a number of them around. But as long as it does not exceed a critical mass, they usually do not pose a major problem. –  the-wabbit Feb 13 at 14:53

I suppose you could do simple ping tests to your switches, and record/track where and when the dropouts occur (and which switches they happen on), then correlate that data with latency and dropped pings from your ping tests. That's not going to be especially accurate, granted, but it's the best you'll do with unmanaged switches. It should also be enough to make an educated assessment of whether it's a limitation on any one specific switch, or a bigger issue like network saturation or insufficient bandwidth at some point in your LAN.

Ultimately, the solution, and the only way to really narrow this down much is to get managed switches so you can get a detailed map of network usage (this could well be an issue of network saturation, or insufficient bandwidth causing packets to be dropped somewhere along the line), and set up QoS. If you use VOIP you, need QoS.

share|improve this answer

If you have something which is actually breaking a Skype call with only a short outage for actual connectivity (<15s), this probably would be something actively tearing down the connection.

For diagnosis, you could take an analytic approach by running a full packet trace (using Wireshark or Network Monitor) on one of the affected stations until the issue occurs and looking at the trace for possible clues for why the UDP packet exchange of the Skype connection has been interrupted (as the Skype call is likely to be the only heavily-used UDP-based protocol at the time of the call, you should be able to identify the stream easily). You might see something like an ICMP-destination-unreachable packet from one of the routers in the path which would give you a hint about where to look further or just the absence of any response packets for any requests indicating that it is a connectivity problem between the client and the rest of the network.

You also might want to browse the Watchguard's logs to see if any entries would correlate to the reported connection teardowns. The same goes for the client's logs to see if it might have lost the connection and/or reconfigured the IP interface.

Also, consider possible points of failure and try recording connectivity data from behind these points, e.g.

  • a continuous ping to an internal server to check if it is something with the switching infrastructure
  • a continuous ping to the first hop on the net behind your Watchguard to see if it might be related to the Watchguard or the line
  • a continuous ping to a well-known Internet host with good availability (like 8.8.8.8) to check for general Internet connectivity
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.