5

I am sure many of us are used to (or have experienced) the routine (or at least occasional need) of having to reboot (or power cycle) a cable modem, dsl modem, router or hub/switch.

However, I decided to post here in response to a recent widespread issue I just experienced. We have a number of network closets across several buildings.

Most of them have managed, high quality switches - gigabit, fiber between buildings and sometimes between closets.

Over the weekend we had some sort of power glitch. However the glitch only hit one building, not all the locations.

After that, there were lots of network issues. Issues with printers, loss of connectivity and more. Across all the buildings.

It seems like all of the 'high quality' managed network equipment is fine. However, in some areas we have some consumer grade - non managed - switches. For example, a large office that has only 1 network drop, but requires several connections. We have now been gradually making our way around to all of these switches (due to users calling with issues) and power cycling them. That fixes the issue for the user. The switch usually looks normal. Some of them all the lights are on (when they shouldn't be).

So why would all these switches start malfunctioning? Some kind of bogus routing data being pushed out from a switch hit with the power glitch?

4 Answers 4

7

I'm going to invoke Occam's Razor on this. While I suppose it's possible that some specific malformed packet(s) could cause your lower cost switches to fall into the failure mode you're describing I'd consider that a very unlikely cause. The switches that you're describing as having problems (small, unmanaged switches) aren't likely to have spanning tree implementations, let alone support for layer 3 switching and dynamic routing protocols. That type of switch should be "blind" to the actual content of the frames its switching, beyond using the source and destination MAC addresses to make switching decisions.

This makes me believe that you had a power issue more widely than you realize.

Going with a power issue assumption, I'd say you're having problems with the low cost switches because they're likely low quality switches. I know this sounds trite, but that's been my experience with networking gear over my entire career (with very few exceptions). You generally get what you pay for (and, though something may be priced incorrectly the market sorts that out pretty quickly).

A higher cost switch is typically going to have a better power supply that is more likely to run within tolerances when exposed to "glitchy" utility power. I suspect that the power supplies in your lower cost switches probably started putting out bad power when the utility power went out of spec. At that point, some part of the "brains" of the switch ended up in a "this should never happen" scenario because one or more of the power rails drifted too far out of tolerance.

An Ethernet switch isn't typically a single ASIC running the whole show but rather are typically groups of systems of ASICs that do different jobs connected to each other. Without knowing about the architecture of the switch in question it's hard to say anything definite. I've had experiences with a model of switch, many years ago, that used a single ASIC to run a group of 4 ports. Certain types of failures would cause groups of 4 ports on the switch to "flake out" while the rest of the switch kept running fine. A partial failure of a switch isn't abnormal in my experience.

In the case of your failure, the parts of the switch that handled keeping the lights on, for example, kept running fine. The physical interface hardware (the PHYs) probably kept running just fine (since you were probably seeing "lights" out on the far ends of the connections). Something else, however, didn't keep working right and you ended up seeing a lack of connectivity. In the cases where I've been "fortunate" enough to catch a switch "in the act" of failing like this I've plugged my laptop into a "problem" port and observed (using Wireshark) a totally "dark" network without any broadcast packets or the other "noise" commonly associated with a typical "working network". Packets transmitted into these ports never showed up elsewhere in the network-- they just fell into a "black hole". I bet you'd see something similar in your situation.

5
  • A "switch", by definition, must integrate spanning-tree. If it doesn't, then it's a hub. However, this doesn't mean that it's a crappy implementation of it.
    – Aaron
    Oct 26, 2011 at 17:09
  • @Aaron: Can't say I've heard that definition before. Switches are just multi-port bridges. I'm not aware of any Ethernet standard specifying that bridges must implement spanning tree protocol (STP). Certainly, a bridge that doesn't implement STP must be transparent to bridge protocol data units (BDPUs), but BPDUs will be transparent to any Ethernet switch as a matter of the design of STP and the BPDU frames (since they're just Ethernet frames with a specific destination address). A lot of switches, to your definition, must be "hubs" then, because most "cheapie" switches don't implement STP. Oct 26, 2011 at 17:29
  • I guess I think of a multi-port bridge as a hub. I think of a switch as a device that builds a MAC table and then only forwards certain packets out certain ports - i.e, it has memory as to which stations are where.
    – Aaron
    Oct 26, 2011 at 17:32
  • 2
    @Aaron: A multi-port bridge isn't a hub, though-- it's a switch. Any bridge, even a two port bridge, can be considered an Ethernet switch (since it forwards frames basd on MAC / port adjacency). The basic act of switching-- using a MAC / port table to intelligently transmit frames only to the appropriate port-- doesn't at all rely on STP. STP is just a fancy way of deterministically selecting ports on which to block transmissions to prevent switch loops across a group of switches. It doesn't change the basic operation of an Ethernet switch. Oct 26, 2011 at 17:52
  • @Aaron: To follow up on Evan's reply. All ports on a hub share a collision domain, and are either passive devices, or may amplify signals back to full strength as part of the process. A repeater is a device that acts much like a hub, any packet that comes in will be rebroadcast on all other ports. However each such port has a seperate collision domain and repeaters will rebroadcast in the event of a collision. Many so-called Hubs are actually repeaters. Oct 26, 2011 at 18:07
1

Cisco 1900 series switches where notorious for this some years ago.

These switches used 2 power-feeds internally: 5 Volt for CPU/backplane, 12 Volts for CAM memory. On a short power-spike the 5 Volt remained stable enough for the switch to continue running but the 12 Volt dropped enough for the CAM memory tables to get corrupted. Unfortunately there was no way the switch CPU could detect the memory corruption which caused all sorts of havoc with L2 switching and ARP.

That's why we put a small UPS in each patch-cabinet. That was a lot cheaper than resetting every switch by hand. (And dealing with the pissed users.)

Cisco fixed this in later models. I have heard of the same issues with older HP switches as well.

I'm certain there is a lot of hardware around, especially in the consumer/soho segment, that has similar issues. A good quality power-supply is still one of the more expensive components of the device so it's usually the first item that gets down-graded when the PHBs decide that the margins on the product are too low.

0

Possibly some CRC / Jabber / STP / broadcast error that the managed switches could "handle". The consumer grade ones couldn't handle the data so crashed.

It's posisble that it has nothing to do with this, and a cheeky user has found a way to crash your network!

0

Most of the network problems you'll see with non-managed switches are arp table related. There's not really much else to go wrong at that level of the network. And it should be relatively easy to test using arpping from various locations on your network. If it's arp related you'll find some areas of your network where the arp ping doesn't resolve.

Now if you're using STP on the managed portions of your network then there's a possibility that is setting a link down that should be up. But you should be able to figure that out from the managed switches. Your non-managed switches won't support STP so they won't be participating.

3
  • 2
    An unmanaged layer 2 switch doesn't have an "arp table". It's got a bridge / CAM table that it uses to make switching decisions, but that has nothing to do with ARP. All you're doing with arping is making ARP requests and looking for responses, which an unmanaged switch isn't going to respond to directly. Oct 26, 2011 at 16:27
  • Thanks for enlightening me. Reading up on the CAM table it looks like it's populated by ARP request. It also looks like most switches will fail back to broadcasting packets out of every port if the CAM table is full or otherwise unable to determine where to send the packet. So this probably isn't the cause of the problem. I like your theory on the ASIC going AWOL for whatever reason. Oct 26, 2011 at 17:44
  • It so happens that the first traffic a switch will see moving between two hosts in an IP network will likely be ARP traffic. That doesn't mean that switching is at all related to ARP. Imagine a network using the NetBEUI protocol (a layer 2 protocol that doesn't use IP at all). The switches would still populate their bridge tables but no ARP packets would be involved. It's just an artifact of how IP maps onto Ethernet that the first frames in any IP packet exchange will be ARP. Oct 26, 2011 at 17:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .