2

Let me preface this with the fact that I'm just a web developer at my company with little networking knowledge.

Earlier today there was a department that lost all of their network connections so I popped open Wireshark and observed the influx of packets to my machine.

There was normal traffic (ARP requests, etc.) coming in at ~50 packets every second. Then all of a sudden the log was flooded with packets arriving ~5000 a second. It looks as though they all contain the same data, just a looped sequence.

We have someone here looking at it but I thought I would ask if anyone had seen anything like this before.

Here's a selection from one of the captures in Wireshark (click on the image to open the Cloudshark capture): capture hosted at Cloudshark

Improve this question
5
  • 6
    Looks like someone connected a switch to another switch in the same broadcast domain.
    – David Schwartz
    Nov 9, 2012 at 19:37
  • 3
    Those aren't packets, they're Ethernet frames. Specifically they're LLC frames. What they are exactly I don't know but they're coming from a Xerox MAC address so I would look at whatever Xerox devices you have.
    – joeqwerty
    Nov 9, 2012 at 20:21
  • @DavidSchwartz no way - the destination address of the frames is not a broadcast address (and there surely would be a broadcast storm in case of a loop). And the frames are not identical - they differ slightly.
    – the-wabbit
    Nov 9, 2012 at 21:00
  • Adam, did you mangle the MAC addresses within your capture or are those truly the ones from the wire?
    – the-wabbit
    Nov 9, 2012 at 21:03
  • 6
    I have never heard of CloudShark before - thank you.
    – the-wabbit
    Nov 9, 2012 at 21:36

3 Answers 3

Reset to default
10

First, neither the number of frames nor the amount of data should as such significantly impact network connections even with just Fast Ethernet in place - 5,000 frames of 500 bytes amount to a bit less than 2,5 MB / second of data. They might trigger broadcast storm detection mechanisms on your swithes though, leading to broadcast frame drops of legitimate traffic - especially ARP requests - which could adversely impact IP connectivity (although typically not interrupt it completely, you are likely to see packet losses due to untimely ARP resolution).

The LLC frames in your submitted capture look weird. Neither the source address nor the multicast destination address look like valid, real-world addresses. Also, the LLC frame format violates the standard - NULL addresses are used in conjunction with a UI frame type - which should never happen:

The null address is only valid for use in the address fields of XID and TEST PDUs. The use of the null address (DSAP and SSAP) is specified in ISO/IEC 8802-2.

(Source: IEEE LLC tutorial)

I would suspect some device (presumably not a Xerox albeit the source address resolves to Xerox's MAC address space - I would expect them to know about and abide by basic rules) is violating the protocol. Try hunting it down by looking into your managed switches' FDB / address tables: start at an arbitrary managed switch, find the 00:00:03:20:00:00 address in the table which presumably would be associated with an upstream port to another switch, follow to the next switch repeating the procedure unless you find the address associated with an edge port (i.e. a port with a single connected host).

Improve this answer
2
  • 1
    All this assumes the frames aren't just junk transmitted by broken hardware or buggy software; see the other answer to this question, which notes that the frame appears to consist of 0x03 0x20 0x00 0x00 repeated over and over again.
    – user137177
    May 18, 2014 at 15:14
  • @GuyHarris it does not matter much whether the device transmitting the frames is broken. My answer is primarily meant to state that a) the traffic is unlikely the reason for the network failure and b) there is a way to systematically identify the component if your switching infrastructure consists of managed devices.
    – the-wabbit
    May 20, 2014 at 13:05
2

When I look at the hexdump I notice that it is just the same sequence of four bytes repeating over and over. That sequence of four bytes is not valid data, and trying to decode it will not yield any useful information.

That repeating sequence covers both source and destination MAC address as well as the Ether type. This means neither source nor destination MAC address means anything, they have been corrupted. This is also not an LLC frame, it just happens that this sequence of bytes got decoded as LLC.

If you see something like that again, I'd first try to capture the traffic using more than one brand of Ethernet adapter to be sure those corrupt frames are truly being sent on the wire, and is not just an artifact introduced on the machine doing the capture.

If you have established this traffic is really being sent on the wire, then you'll have to start looking for what device is producing it. This looks like a low level problem at the physical layer beneath the MAC layer. You won't have MAC addresses to go by, so the only option for finding the source may be to look at flashing link status, and unplug wires until you find the one producing those packets.

It is not unlikely that the root cause was defective hardware.

Improve this answer
1
  • Or defective firmware or software running on that hardware. This answer is probably correct.
    – user137177
    May 18, 2014 at 15:15
2

Faulty NIC can do this and I have no idea why. Only very rarely do I see this but it's always one NIC that's gone nuts. Swap it out and you will likely never see this again.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .