2

Switches are supposed to only send traffic to a system if it's intended for that system. Ours appears not to: if I run "tcpdump not host myhostname", I can see lots of packets that are clearly non-broadcast (ssh, nfs) travelling between other hosts. How can I prevent this? I think it may be causing poor performance (we have heavy NFS use; if it's coming out of all the switch ports, that can't be good).

The switch is a stack of three Netgear GS748TS managed switches. The switch activity lights all flash in sync continuously, which seems wrong too.

6 Answers 6

3

You may not be seeing what I'm going to describe, the there's a chance that you are. I've seen the symptoms you're talking about on several low-end and middle-of-the-road Netgear and Linksys switches. The switches I've seen this "crazy" behaviour happen with have been in place for awhile, working fine, but begin to flood frames out all ports. The call I normally get is "the network is slow", and subsequent bandwidth monitoring usually locates a single switch that has "gone crazy", often with its activity lights on solid, pumping out large quantities of "bogus" traffic (sometimes made up of legitimate traffic, other times seemingly random garbage).

I like Zypher's suggestion about checking on CPU utilization, but I'd also consider breaking up the "stack" (on these particular switches I don't know if a "stack" operates as a single logical unit or not) and testing each switch individually.

It seems like this problem has gotten worse in the last few years w/ no-name Ethernet switches, Linksys, and Netgear switches. I don't know if there's some silicon in common that has an issue amongst the problem switches I've seen or not. (We're recommending our Customers purchase Dell PowerConnect switches and haven't seen these kinds of problems with their switches-- yet.)

2
  • I've ticked your answer because breaking up the stack and factory resetting the switched seems to have made the situation a lot better.
    – pjc50
    Jul 15, 2009 at 14:24
  • Keep a sharp eye on them. If your experience turns out like some on mine have one or more of the switches will continue its descent into madness. Jul 15, 2009 at 14:37
3

Normally, a switch should not fail back into 'hub' mode unless you have overloaded its MAC address table (normally around 8,000+ MAC addresses, but check the make/model detail). You can check the number of MAC addresses on a given Cisco switch with this command:

sh mac-address-table | inc Total

Switches make all of their decisions based on the vlan tag (if you're using them) and mac address. If you are getting flooding then you need to look at why the switch isn't building an accurate mac-address-table.

You can get this behavior however if you get what I call 'vlan bleed'. If you have two switch ports connected back-to-back, but on different vlans you could see this behavior.

A device with multiple network cards could also be bridging vlans together. Microsoft had a 'feature' to bridge network cards so your wireless and hard-wired networks would be bridged for instanance.

Look at your mac table for ports with more than one address that are not uplinks to another switch. You can also follow the MAC address of one of the devices that should be unicast through the MAC tables of your switches.

On Cisco switchs these commands may be helpful:

sh mac-address-table | ex CPU
sh mac-address-table | ex CPU|<uplink port name>
1

The first thing i would check would be the load on your switch. If you are running it at such a high load that it can't properly switch the packets (on most switchest this starts happening around the 80-90% cpu load area) it will fail back to effectively being a hub, as well as the possibility of dropping packets.

1

Although the other responses sound more likely from what you have described, be sure that the port you are plugged into is not set up as a mirror port.

1

This sort of behaviour is normally indicative of the switch not having an entry in it's bridging table (a.k.a. CAM table or MAC address table) for the destination MAC of a frame. Cisco refer to it as unicast flooding.

Can you provide a little more detail regarding your topology? Is all of the 'unexpected' traffic destined for the same host, or for multiple hosts? Do any of your hosts use bonded/teamed NICs? It's possible that inbound traffic is being sent to one MAC address, but that outbound traffic from the host is being sourced from another NIC.

EDIT: As the spurious traffic is limited to some MACs, this may be a good place to start. Can you interrogate the switch's bridging table (show mac-address-table on Cisco devices) and find if there is an entry for the offending destination MAC addresses? Also, are you able to confirm what the timeout value for the switch's bridging table is?

One case where I have seen this behaviour before is on subnets where the ARP timeout value is set higher than the MAC address table timeout. Normally, when an ARP entry ages out, an ARP probe is sent. The response to the probe gets noted by the switch, which stores the source MAC of the host in its bridging table. When these timers are reversed, the bridging entry ages out before the ARP entry. This means that routers/hosts on a subnet know what MAC address owns a given IP, but the switch does not know which port that MAC is connected to. In this case, the switch will flood traffic to all ports in that VLAN.

4
  • The topology is almost flat: the only strange things are a hub under a desk with some workstations on, and an OpenVPN host that bridges its clients onto the LAN. There is a Cisco WAN router as well that emits occasional 802.1d (STP) packets, but STP is turned off on the switch stack. There are no VLANs and no bonded NICs.
    – pjc50
    Jul 15, 2009 at 9:01
  • What about the spurious traffic itself? Is it all destined to one particular IP/MAC address, or is all traffic being flooded out of all ports? Jul 15, 2009 at 9:36
  • Spurious traffic seemed to be limited to some MACs.
    – pjc50
    Jul 15, 2009 at 14:26
  • pjc: see revised answer. Jul 15, 2009 at 20:54
0

Switch is suppose to send directed traffic, but not always.
Every switch maintain an ARP table, like every host on the network.
A simple attack--
If you have to sniff a packet that does not belong to you in a switched LAN. What you do is generate lots of fake arp requests, and overload the ARP table in the switch. Once the ARP table is overloaded, switch operates in a default HUB mode.
CPU over-utilization can be responsible for poor performance, but i almost certainly think, not for packet storms like this.

2
  • 3
    Non-routing switches do not maintain ARP tables (except for their own, management type use). They have MAC address tables only. What you are doing with this type of attack is overloading the MAC table, forcing the switch to flood all traffic.
    – Peter
    Jul 14, 2009 at 17:18
  • @Peter: +1 to you if I could. I get so sick of hearing bridge/MAC tables called "ARP tables". Jul 14, 2009 at 17:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .