1

I just had a weird experience on my home network. Our ethernet went down; pinging an adjacent host was impossible. I checked the switch; all lights were on and flickering, although they were flickering in synch which was a bit worrying. Then I noticed that my Linux box had crashed (unresponsive to mouse and keyboard). I hit the reset button, and at that moment the network cleared.

This would be of only academic interest, except that my employer happens to be in a business where service continuity is of great importance. Critical data is sent over dual independent ethernet LANs. Our reliability models assume that the only thing that could take down an entire LAN is a failed switch. So the idea that a single malfunctioning host could do it is ... worrying.

This message on a Cisco forum says its impossible so don't worry.

This report about an outage in the US Customs sounds similar: a malfunctioning Ethernet card bought down their network. That was a single network and it sounds like a hardware fault, so it wouldn't bring down both of our dual networks. But I'm wondering: could a device driver bug wedge a card into a state where it was jamming the network? If so then if it was driving two bonded channels it might wedge both in the same way.

Does anyone know more about the potential failure modes of Ethernet?

Edit

What I'm trying to understand is: what could a single node do in software (e.g. in a device driver) that could bring down the entire network. Lets assume that its not malware, so obscure bugs of specific switches are probably not an issue. Sending frames to a single specific host won't do it. Would sending lots of broadcast frames (destination FF:FF:FF:FF:FF:FF) have this effect? What about jabber? Is that a thing still?

closed as off-topic by Greg Askew, ThoriumBR, Nathan C, Jenny D, Dave M Nov 4 '17 at 10:56

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions on Server Fault must be about managing information technology systems in a business environment. Home and end-user computing questions may be asked on Super User, and questions about development, testing and development tools may be asked on Stack Overflow." – Greg Askew, ThoriumBR, Nathan C
If this question can be reworded to fit the rules in the help center, please edit the question.

  • Duplicate MAC-Address? – ansi_lumen Oct 24 '17 at 11:15
  • @ansi_lumen: no, it just failed for no readily apparent reason while under way. The MAC address was not changed. – Paul Johnson Oct 24 '17 at 12:40
4

Here are just a few things that can cause the behavior you witnessed:

  1. A switch loop.

  2. Malware.

  3. A bad/defective NIC.

  4. A buggy/misbehaving NIC driver.

  5. A broadcast storm (usually associated with a switch loop).

To address your edit: A broadcast storm or switch flooding (which are two different things) could cause this problem. Make note that there are two broadcast addresses at work here: FF-FF-FF-FF-FF-FF (255.255.255.255), which is the Layer 2 broadcast address, AND the Layer 3 subnet broadcast address (for example 192.168.1.255 is the Layer 3 subnet broadcast address for the 192.168.1.0/24 subnet). A broadcast storm at either Layer 2 or Layer 3 could cause this problem.

  • It wasn't a switch loop. Possibly a misbehaving NIC or driver, as the problem went away when the crashed host was rebooted. But how can the driver on one host cause the entire network to lock up like that? – Paul Johnson Oct 23 '17 at 17:55
  • If the misbehaving host floods the switch with traffic it could cause the CPU, memory, and backplane on the switch to become saturated, effectively causing the switch to overload. I've seen this happen with just a few host machines infected with malware bring down an entire network with a broadcast storm. – joeqwerty Oct 23 '17 at 18:18
  • @PaulJohnson - since you didn't tell us what your home switch is, I'll assume that it's unmanaged and you have no way of diagnosing problems on it or checking the logs? If that's the case, you're asking for a lot of guesses. If you do have actual logs, SHARE THEM WITH THE REST OF THE CLASS please. – mfinni Oct 24 '17 at 13:10
  • @mfinni, as you say my home switch is unmanaged. The Linux box logs show nothing remarkable on the network, but at the time of the crash (as near as I can figure it) there are a lot of errors about the Radeon GPU locking up. My concern is not actually to diagnose this specific home network failure, its about assessing the risks of losing both ethernets at work. – Paul Johnson Oct 24 '17 at 13:45
  • Paul - above, you did ask " But how can the driver on one host cause the entire network to lock up like that?" - which was your home network. I'm answering that question - without the logs from a managed switch, we can only throw guesses at the wall, which is not what this site is for. If you have concerns about your work network, hire a network consultant to review your design. – mfinni Oct 25 '17 at 14:59
2

Switches run code, in their firmware. Sometimes that code is buggy, and unexpected input can crash the switch. So yes, a misbehaving host can crash a switch. It's not really likely, but it can happen.

Years ago (2003 maybe?) I had unmanaged Netgear switches that would fall down 2-4 times a week, as if they were undergoing a broadcast storm - like your description above. Rebooting the stack was the only fix. Netgear support said they had a known issue with running IP and IPX on these, and since they were unmanaged, there was nothing to troubleshoot. They had been EoL with no further firmware upgrades, so they replaced them with newer managed switches, under warranty.

As far as "please list all potential failure modes of Ethernet" - no, that's a silly request. For your own education, though, read up on spanning-tree loops, that's a common user-induced failure mode.

0

Since the Linux box appears to have two LAN interfaces: can you rule out that it wasn't temporarily bridging these two interfaces, creating a bridge loop?

Just using two switches isn't high availability. You should have indicators on the switches signalling a broadcast storm and appropriate monitoring software. For this, configure a management VLAN with a higher priority so it won't get interrupted by a broadcast storm. Alternatively, run the management functions over physically separate network links or out-of-band.

PS to your edit: On a switched network, the only things able to bring down all ports are broadcast storms or severe congestion. Oversized frames (jabber), fragments, or similar anomalies are simply dropped by a switch. A broadcast storm from an ingress port can flood the network with the bandwidth of that port - a 100M port doesn't do much harm to a 1G network but a 1G port can easily drown all 100M egress ports. Similarly, sending more data through an uplink that it can handle will drop most other traffic in that direction.

Broadcast storms are usually caused by bridge loops. Spanning tree is a good remedy for that, also allowing you to add redundant links to your network. Other storms can be handled by broadcast limiting on edge ports.

Congestion is a tougher beast. The hardware approach is to make sure all up/download ports are faster than any edge port. On a gigabit switch with a 10GE uplink you'd need at least 10 edge ports to saturate the uplink. Another approach is to limit edge port bandwidth so they can't overfeed the uplink.

  • My home box has a single LAN. At work, yes we do all of that, but we can't afford the downtime to diagnose the fault if it happens on both LANs simultaneously. – Paul Johnson Oct 24 '17 at 12:44
  • Are you asking about your home network (off-topic here) or your work network? There's a multitude of safeguards against network outage but that wasn't your question. – Zac67 Oct 24 '17 at 17:54
  • I'm actually concerned about the work network, but the odd behavior of my home network is what started me worrying. I need to understand the failure modes of Ethernet better. I thought that what I saw happen on my home network (one node bringing the entire network down) wasn't possible. If it sometimes happens at home then thats not important, but if it happens to both redundant networks at work then that is important. – Paul Johnson Oct 25 '17 at 18:54
  • So essentially, you want to know what could go wrong in an Ethernet network? This is a very broad question... – Zac67 Oct 25 '17 at 18:59
  • see my edit to the original question, where I try to narrow it down a bit. Maybe its still too broad; I don't know. Thanks for your patience. – Paul Johnson Oct 25 '17 at 19:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.