Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Here's the issue...we have a network with a lot of Cisco switches.

Someone plugged in a hub on the network, and then we started seeing "weird" behavior; errors in communication between clients and servers, or network timeouts, dropping network connections, etc. It seemed that somehow that hub (or SOHO switch) was particularly freaking out our Cisco 3700 series switches.

Disconnect that hub or netgear-type SOHO switch and things settled down again.

We're in the process of trying to get a centralized logging server for SNMP and management, etc., to see if we can trap errors or narrow down when someone does this sort of thing without our knowledge because things seem to work, for the most part, without issue, we just get freaky oddball incidents on particular switches that don't seem to have any explanation until we find out someone decided to take matters into their own hands to expand available ports in their room.

Without getting into procedure changes or locking down ports or "in our organization they'd be fired" answers, can someone explain why adding a small switch or hub, not necessarily a SOHO router (even a dumb hub apparently caused the 3700's to freak out) sending DHCP request out, will cause issues? The boss said it's because the Cisco's are getting confused because that rogue hub/switch is bridging multiple MAC's/IP's into one port on the Cisco switches and they just choke on that, but I thought their routing tables should be able to handle multiple machines coming into the port. Anyone see that behavior before and have a clearer explanation of what's happening?

I'd like to know for future troubleshooting and better understanding that just waving my hand and saying "you just can't".

Here's a show run

Current configuration : 25591 bytes

!

version 12.2

no service pad

service timestamps debug datetime msec

service timestamps log datetime msec

service password-encryption

!

hostname ###########

!

boot-start-marker

boot-end-marker

!

enable secret 5 ############

!

!

!

no aaa new-model

switch 1 provision ws-c3750g-24ps

switch 2 provision ws-c3750-48ts

switch 3 provision ws-c3750-48ts

switch 4 provision ws-c3750-48ts

switch 5 provision ws-c3750-48ts

system mtu routing 1500

authentication mac-move permit

ip subnet-zero

ip routing

!

!

!

mls qos map policed-dscp 24 26 46 to 0

mls qos map cos-dscp 0 8 16 24 32 46 48 56

mls qos srr-queue input bandwidth 90 10

mls qos srr-queue input threshold 1 8 16

mls qos srr-queue input threshold 2 34 66

mls qos srr-queue input buffers 67 33

mls qos srr-queue input cos-map queue 1 threshold 2 1

mls qos srr-queue input cos-map queue 1 threshold 3 0

mls qos srr-queue input cos-map queue 2 threshold 1 2

mls qos srr-queue input cos-map queue 2 threshold 2 4 6 7

mls qos srr-queue input cos-map queue 2 threshold 3 3 5

mls qos srr-queue input dscp-map queue 1 threshold 2 9 10 11 12 13 14 15

mls qos srr-queue input dscp-map queue 1 threshold 3 0 1 2 3 4 5 6 7

mls qos srr-queue input dscp-map queue 1 threshold 3 32

mls qos srr-queue input dscp-map queue 2 threshold 1 16 17 18 19 20 21 22 23

mls qos srr-queue input dscp-map queue 2 threshold 2 33 34 35 36 37 38 39 48

mls qos srr-queue input dscp-map queue 2 threshold 2 49 50 51 52 53 54 55 56

mls qos srr-queue input dscp-map queue 2 threshold 2 57 58 59 60 61 62 63

mls qos srr-queue input dscp-map queue 2 threshold 3 24 25 26 27 28 29 30 31

mls qos srr-queue input dscp-map queue 2 threshold 3 40 41 42 43 44 45 46 47

mls qos srr-queue output cos-map queue 1 threshold 3 5

mls qos srr-queue output cos-map queue 2 threshold 3 3 6 7

mls qos srr-queue output cos-map queue 3 threshold 3 2 4

mls qos srr-queue output cos-map queue 4 threshold 2 1

mls qos srr-queue output cos-map queue 4 threshold 3 0

mls qos srr-queue output dscp-map queue 1 threshold 3 40 41 42 43 44 45 46 47

mls qos srr-queue output dscp-map queue 2 threshold 3 24 25 26 27 28 29 30 31

mls qos srr-queue output dscp-map queue 2 threshold 3 48 49 50 51 52 53 54 55

mls qos srr-queue output dscp-map queue 2 threshold 3 56 57 58 59 60 61 62 63

mls qos srr-queue output dscp-map queue 3 threshold 3 16 17 18 19 20 21 22 23

mls qos srr-queue output dscp-map queue 3 threshold 3 32 33 34 35 36 37 38 39

mls qos srr-queue output dscp-map queue 4 threshold 1 8

mls qos srr-queue output dscp-map queue 4 threshold 2 9 10 11 12 13 14 15

mls qos srr-queue output dscp-map queue 4 threshold 3 0 1 2 3 4 5 6 7

mls qos queue-set output 1 threshold 1 138 138 92 138

mls qos queue-set output 1 threshold 2 138 138 92 400

mls qos queue-set output 1 threshold 3 36 77 100 318

mls qos queue-set output 1 threshold 4 20 50 67 400

mls qos queue-set output 2 threshold 1 149 149 100 149

mls qos queue-set output 2 threshold 2 118 118 100 235

mls qos queue-set output 2 threshold 3 41 68 100 272

mls qos queue-set output 2 threshold 4 42 72 100 242

mls qos queue-set output 1 buffers 10 10 26 54

mls qos queue-set output 2 buffers 16 6 17 61

mls qos

!

!

!

!

!

spanning-tree mode pvst

spanning-tree etherchannel guard misconfig

spanning-tree extend system-id

!

vlan internal allocation policy ascending

!

!

class-map match-all AutoQoS-VoIP-RTP-Trust

match ip dscp ef

class-map match-all AutoQoS-VoIP-Control-Trust

match ip dscp cs3 af31

!

!

policy-map AutoQoS-Police-CiscoPhone

class AutoQoS-VoIP-RTP-Trust

set dscp ef

police 320000 8000 exceed-action policed-dscp-transmit

class AutoQoS-VoIP-Control-Trust

set dscp cs3

police 32000 8000 exceed-action policed-dscp-transmit

!

!

!

!

interface GigabitEthernet1/0/1

switchport trunk encapsulation dot1q

switchport trunk native vlan 11

switchport mode trunk

spanning-tree portfast

!

!

!

interface GigabitEthernet5/0/1

!

interface GigabitEthernet5/0/2

!

interface GigabitEthernet5/0/3

!

interface GigabitEthernet5/0/4

!

interface Vlan1

ip address ############## 255.255.0.0

!

ip classless

ip route 0.0.0.0 0.0.0.0 ##############

no ip http server

no ip http secure-server

!

!

ip sla enable reaction-alerts

!

!

!

line con 0

line vty 0 4

password 7 ############

login

line vty 5 15

password 7 ############

login

!

end

share|improve this question
    
what is meant by "dumb hub apparently caused the 3700's to freak out?" What does freak out mean? What, specifically, is happening? All these other comments about broadcast storms and loops are erroneous in nature. The IEEE Spanning Tree protocol has protections against such things. It might help if you dumped a copy of your "show run" for the switch. –  Kilo May 10 '10 at 14:37
    
By freak out I mean that the switch starts showing seemingly random issues with the fact that the hub or switch was connected to another switch somewhere else in the network that caused seemingly random errors with clients, like spurious dropping of network traffic. In the first case we tracked it down someone connected a hub to switch A's port, and switch B in another part of the building started causing a lab to have communication trouble on the network. Removed the hub from A, everything worked again on switch B. –  Bart Silverstrim May 10 '10 at 15:43
    
I'll add a show run for our switch here shortly... –  Bart Silverstrim May 10 '10 at 15:43
add comment

5 Answers

up vote 10 down vote accepted

We operate a couple network implementations where third-party connections are linked up to a centralized Cisco backbone (i.e. multi-tenant setup). I can say I've seen a bunch of diverse (okay, ghetto) devices connected up to the Catalyst platform, and if there's one thing I've learned, it's that the Cisco platform is remarkably resilient to these kinds of things.

There is one achilles heel, though - A cheap hub in the right configuration can easily bring down an entire network with a broadcast storm, and it's not even the Cisco platform's fault. I discovered this in a production configuration, and the only real research I did was finding the closest trash can for that hub, but here's how it happened:

  1. Connect hub to Cisco switch as normal, with uplink port
  2. Connect a workstation to a hub port (in our case, running Windows XP OS, but shouldn't matter)
  3. Connect two other ports together on the hub (either directly, with a single CAT5 or indirectly through another hub).

Everything runs smoothly until that workstation sends out a broadcast announcement. While the hub and the Cisco are smart enough to prevent a broadcast storm on other broadcast packets, the hub isn't smart enough to detect that two of its ports are connected to each other, and will use up almost 100% of its processing power to broadcast that packet in an infinite loop back and forth, as well as out all the other ports (i.e. the uplink to your Cisco).

If this is the case in your configuration, you will notice that across your network, all of the ports on that broadcast VLAN will go nuts, up until the hub can't sustain the capacity and drops the magical looping packet (could be a couple minutes depending on the competing traffic), and then all is back to normal.

In this situation SNMP won't help you since all the ports on that VLAN go crazy with traffic. However, Wireshark is your friend here, since it's easy to capture which IP (and sometimes machine name if it's a broadcast packet) caused the loop, and quickly locate the offending device.

May not be the exact case you're experiencing, but this one bit us hard and might give you some ideas to research with your situation.

share|improve this answer
    
This is the case for turning on storm-control towards customers. –  pehrs Jan 27 at 9:29
add comment

If you're using port security on the ports, you'd end up with problems whenever there's more than the expected number of MACs on a given port. The main problems with sticking hubs into a switched network is that you end up with larger collision domains (instead of "switch + end unit", you end up with "switch + everything downstream on the hub"), you can cause switching loops (if a hub is ever connected to two switches) and you may end up with a network that's large enough that the broadcast domain is "too wide" (there's an absolute requirement for a LAN to be small enough that packets from one extreme to the other will be seen within the Ethernet pre-amble; as hubs tend towards being worse at letting packets through at pace (being cheaper and all), that can end up being violated).

If you have a larger collision domain, anything that tries to talk to/from that collision domain will end up having lowqer throughput.

If you have switching loops, you risk broadcast storms and ports randomly not forwarding traffic as Spanning Tree shuts down forwarding ports.

If you have a too-wide broadcast domain, you'll end up having spurious re-transmits as you get late collisions.

share|improve this answer
add comment

Trivial points, but I haven't seen them mentioned yet:

Make sure your switch ports aren't forced to full duplex because they won't work with hubs if they are. Come to think of it - if their duplex or their speeds are forced they likely won't connect correctly to a Netgear type switch which wants to auto-negotiate

Make sure your users don't connect the switch to two network outlets "to double their bandwidth".

Cisco switches used to (and maybe still do - I'm out of that business now) have a feature called "Faststart". For any port with Faststart enabled the switch would not do a full spanning tree analysis and by enabling the feature the admin "promised" not to connect a switch or hub to that port. The reason for doing this was to avoid the client PC's DHCP request timing out before the Cisco decided it was safe to enable the port. You may want to look at that also. (If I have totally mis-remembered all this I apologise in advance and hope someone with more up-to-date knowledge will correct me)

share|improve this answer
    
+1 excellent point with Faststart, this can also bite you quickly since its usually enabled to silence complaining users and forwarding analysis does not take place –  Brandon May 10 '10 at 15:18
    
If you have "switchport fast" configured, also make sure to configure "switchport bdpu-guard", this will shut the port down if any spanning-tree packet is seen. –  Vatine May 11 '10 at 10:52
add comment

this is a classic switching problem.
if you do the cisco ccnp or just study spanning tree protocol the answer becomes obvious.

spanning tree prevents loops-> it does this by initially determining a "root bridge"->
this is determined by the lowest switch priority-> if there is a tie then mac addresses of the switches are compared-> often old switch equates to lower mac address-> all traffic on the network goes through the old switch!!!! = crash fixed by setting your fastest/and most centrally connected switch with a hard-coded priority that is lower than the default.

by default cisco use per-vlan-spanning-tree (pvst+) command: spanning-tree vlan 1-666 priority 24576

i recommend using rapid-spanning-tree "spanning-tree mode rapid-pvst" which basically speeds things up a lot

from wikipedia (http://en.wikipedia.org/wiki/Spanning_tree_protocol)

The root bridge of the spanning tree is the bridge with the smallest (lowest) bridge ID. Each bridge has a unique identifier (ID) and a configurable priority number; the bridge ID contains both numbers. To compare two bridge IDs, the priority is compared first. If two bridges have equal priority, then the MAC addresses are compared. For example, if switches A (MAC=0200.0000.1111) and B (MAC=0200.0000.2222) both have a priority of 10, then switch A will be selected as the root bridge. If the network administrators would like switch B to become the root bridge, they must set its priority to be less than 10.

share|improve this answer
add comment

I had this happening in a non Cisco Environment. A hub was plugged into a network port which then had three computers connected to it. One user, no issues. When everyone else came in to play it started killing RDP connections to the Terminal Server. Pull the hub out and problem goes away.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.