Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have a simple 2 office location network. This was previuosly connected via kilostream connection. This has now been replaced by an MPLS connection using Cisco routers.

When we connect the MPLS to the network of the main office location, the network grinds to a halt due to a loop. We have identified that the Spanning Tree Protocol from the cisco router as the cause. We have switched this protocol off and everything works fine.

Unfortunately the company providing the MPLS will not support the connection unless this protocol is switched on. They say the fault is with our network and we have reached stalemate.

Can someone please suggest any tools or services we could try to identify the problem?

share|improve this question

8 Answers 8

How have you identified that STP on the router is the culprit? If you take the output of a "show STP" on each of your routers/switches, you should be able to see where the spanning tree's root is.

Running a packet trace with Wireshark or something similar would show the broadcast storm in progress, if that's what's happening, and hopefully point you in the direction of the box responsible.

share|improve this answer

Is the rest of your network running STP? Perhaps this is an issue with root election.

BTW - I havent seen much replies on serverfault from Cisco experts, maybe you should try #ciscohelp on EFNet (IRC).

share|improve this answer

Some troubleshooting suggestions. You should apply these universally to all network bridges within the same L2 ethernet domain, which will likely include routers and switches alike. You may find it useful to sketch a quick diagram of all devices involved.

  1. Are you sure that the loss of connectivity is due to a loop and not, conversely, STP attempting to prevent a loop? You can confirm this by checking whether the STP port state is "blocking" before and after the outage.

  2. Are you using the same STP type and hardware vendor at both sites? There are a number of varieties - STP, RSTP, PVST, RPVST and MSTP. Not all of which are cross compatible, especially so when inter-mixing vendors.

  3. Have you ever configured any of the STP devices beyond their default settings? Such as to elect a root bridge, change timings, or create STP groups.

  4. You may not consider this last item to be the most helpful piece of advice immediately, but I'd advise to read a basic primer on STP. In small networks it will often "just work". As soon as you start to grow it's really important to have a good picture as to how it operates and how you should best influence it. It will soon become simple and less black magic.

share|improve this answer

How many switches/hubs are present at each site? Are there any other links between the switching networks at each site?

What process did you follow to identify that Spanning Tree was the cause of the problem? Did you see spanning tree logs on the Cisco device? If so please post them, we may be able to suggest an alternative interpretation of them (a lot of Cisco logs can by quite cryptic). Spanning tree is there to make sure there are no loops in your network. It cannot introduce loops; however it can fail to see them if it is mis-configured.

Start by drawing your network topology. Make sure you know where the root of your spanning tree should be. Identify any loops in the network; if your network has any loops then disabling spanning tree is not an option.

share|improve this answer

Could you provide a bit more detail?

  1. Was your kilostream connection routed, previously? Or did you have a flat layer 2 network spanning both your sites?
  2. Is your new provider giving you Layer 2 MPLS, or Layer 3 MPLS?
  3. Would it be possible for you to link to a topology diagram, pre and post change?

Regarding disabling spanning tree; if you have spanning tree disabled, then it is possible that layer 2 loops will form in your network. This will cause issues with (for example) broadcast traffic, as Ethernet does not have any concept of 'Time to live'; broadcast frames will loop around the network forever, slowly eating up resources. As mentioned by Dan C, it is definitely worth reading a primer on STP. This question provides a short summary, and links to an article which discusses the issue of bridging loops in more detail.

share|improve this answer

You almost always want to be using Rapid Per-VLAN STP, rather than simple STP. I say almost only because you can't prove a negative.

Cisco Docs are available.

STP has to elect the root bridge for L2 segments, and that election can take upwards of a minute or more. If you haven't manually configured who the root bridge should be, it's possible you can have a sub-optimal piece of equipment doing the bridging (i.e. you attach 10-year old switch to a 6500 and the older one has a good likelyhood of winning the election because of the rules of STP).

RSTP decreases the timers on STP dramatically so you should converge much quicker.

PVST has separate elections for each segment, so if your MPLS link is it's own segment (and if it's not then you're either doing something extremely interesting or extremely bad), then that election should only affect that link.

Rapid-PVST combines the advantages of RSTP and PVST.

share|improve this answer
1  
I would disagree with chosing rapid PVST as the default for all environments. Most cisco architectures have a limit on the maximum number of spanning tree instances. I have seen a number of situations where the number of VLANS has exceeded this and switches configured for PVST/PVST+ have been silently discarding BPDUs and created loops in the network. PVST is great for all cisco small networks, but if you are likely to need several hundred VLANS or need vendor interoperability MST is a better choice. –  Russell Heilling Jun 12 '09 at 14:19

I think we'll need more information to give you an kind of useful answer. For example, you say that "have identified that the Spanning Tree Protocol from the cisco router as the cause", but normally routers don't participate in spanning tree, only switches do. When you said router, did you actually mean a L3 switch (like a 3750)?

In general, Spanning Tree performs poorly across high latency networks, highly congested networks, or networks with even fairly minor reliability/loss issues.

It's hard to believe you'd actually be having a loop, since from your description you only have 1 connection between two offices. More likely you've got a misconfiguration, or you've got problems with your MPLS circuit that are causing BPDU's to get dropped, thereby throwing spanning tree into a tizzy.

I think to troubleshoot this, you need to figure out which of your switches is the spanning tree root bridge and document your spanning tree topology. My guess is that when you do you'll find a clear issue such as two switches that are both trying to be the root bridge, a spanning tree depth greater than 7, etc.

Can you post a diagram of your network?

share|improve this answer

I think I understand what you are saying. On the switch port(s) connecting to your MPLS provider, you should be able to block STP messages (called BPDU filtering).

On HP Procurves, the command to enable BPDU filter is:

spanning-tree <port-list> bpdu-filter

On Cisco switchis the command should be something similar to:

set spantree portfast bpdu-filter 6/1 enable
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.