3

I'm not our normal network guy... I've just been drafted to help with this issue, so please bear with me.

We have a fairly large (~4,000 devices?) network mostly comprised of HP Procurve gear. From time to time over the last two weeks, we've been getting some wicked broadcast storms that pretty much take down everything. I set up Wireshark to do 3MB dumps, and I caught some of this in the act this morning.

There are thousands of ping requests. They appear to come from the MAC addresses of our switches and APs, and are sent to an IPv4 multicast MAC. The source IP addresses do not match those of the switches and APs... they are IP addresses assigned to a few clients on the network. The destination IP address is always that of the gateway.

My read on this is that the firmware on these Procurve devices is either compromised or gone crazy... or someone is spoofing the addresses and causing this mess. Neither of which seem likely here, so I am asking if you have any ideas of other things to consider.

On another note, our network is not subnetted. (Yes, I know, I know... not my call, unfortunately.) Everything is flat.

Thank you for your time.

Edit: Below are two sequential packets from my capture. I posted the full Wireshark summary. I apologize that it is a bit messy, but it better explains what I am seeing.

No.     Time        Source                Destination           Protocol Info
  16885 2.094869    1.2.41.194        1.2.32.250        ICMP     Echo (ping) request

Frame 16885 (98 bytes on wire, 98 bytes captured)
    Arrival Time: Aug 31, 2010 07:59:11.185552000
    [Time delta from previous captured frame: 0.000123000 seconds]
    [Time delta from previous displayed frame: 0.000123000 seconds]
    [Time since reference or first frame: 2.094869000 seconds]
    Frame Number: 16885
    Frame Length: 98 bytes
    Capture Length: 98 bytes
    [Frame is marked: True]
    [Protocols in frame: eth:ip:icmp:data]
    [Coloring Rule Name: ICMP]
    [Coloring Rule String: icmp || icmpv6]
Ethernet II, Src: Procurve_44:0f:26 (00:1f:fe:44:0f:26), Dst: IPv4mcast_62:20:fa (01:00:5e:62:20:fa)
    Destination: IPv4mcast_62:20:fa (01:00:5e:62:20:fa)
        Address: IPv4mcast_62:20:fa (01:00:5e:62:20:fa)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: Procurve_44:0f:26 (00:1f:fe:44:0f:26)
        Address: Procurve_44:0f:26 (00:1f:fe:44:0f:26)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IP (0x0800)
Internet Protocol, Src: 1.2.41.194 (1.2.41.194), Dst: 1.2.32.250 (1.2.32.250)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 84
    Identification: 0x7718 (30488)
    Flags: 0x00
        0.. = Reserved bit: Not Set
        .0. = Don't fragment: Not Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 2
        [Expert Info (Note/Sequence): "Time To Live" only 2]
            [Message: "Time To Live" only 2]
            [Severity level: Note]
            [Group: Sequence]
    Protocol: ICMP (0x01)
    Header checksum: 0xd710 [correct]
        [Good: True]
        [Bad : False]
    Source: 1.2.41.194 (1.2.41.194)
    Destination: 1.2.32.250 (1.2.32.250)
Internet Control Message Protocol
    Type: 8 (Echo (ping) request)
    Code: 0 ()
    Checksum: 0xf112 [correct]
    Identifier: 0xdffd
    Sequence number: 0 (0x0000)
    Data (56 bytes)

0000  f8 17 44 fa 62 8d 00 00 80 11 42 da 8f e2 27 66   ..D.b.....B...'f
0010  8f e2 4e 0d 00 89 00 89 00 3a 2d df 00 00 00 00   ..N......:-.....
0020  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0030  00 00 00 00 00 00 00 00                           ........
        Data: F81744FA628D0000801142DA8FE227668FE24E0D00890089...
        [Length: 56]

No.     Time        Source                Destination           Protocol Info
  16886 2.094991    1.2.44.101        1.2.32.250        ICMP     Echo (ping) request

Frame 16886 (98 bytes on wire, 98 bytes captured)
    Arrival Time: Aug 31, 2010 07:59:11.185674000
    [Time delta from previous captured frame: 0.000122000 seconds]
    [Time delta from previous displayed frame: 0.000122000 seconds]
    [Time since reference or first frame: 2.094991000 seconds]
    Frame Number: 16886
    Frame Length: 98 bytes
    Capture Length: 98 bytes
    [Frame is marked: True]
    [Protocols in frame: eth:ip:icmp:data]
    [Coloring Rule Name: ICMP]
    [Coloring Rule String: icmp || icmpv6]
Ethernet II, Src: HewlettP_05:5e:70 (00:17:a4:05:5e:70), Dst: IPv4mcast_62:20:fa (01:00:5e:62:20:fa)
    Destination: IPv4mcast_62:20:fa (01:00:5e:62:20:fa)
        Address: IPv4mcast_62:20:fa (01:00:5e:62:20:fa)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: HewlettP_05:5e:70 (00:17:a4:05:5e:70)
        Address: HewlettP_05:5e:70 (00:17:a4:05:5e:70)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IP (0x0800)
Internet Protocol, Src: 1.2.44.101 (1.2.44.101), Dst: 1.2.32.250 (1.2.32.250)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 84
    Identification: 0x7718 (30488)
    Flags: 0x00
        0.. = Reserved bit: Not Set
        .0. = Don't fragment: Not Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 2
        [Expert Info (Note/Sequence): "Time To Live" only 2]
            [Message: "Time To Live" only 2]
            [Severity level: Note]
            [Group: Sequence]
    Protocol: ICMP (0x01)
    Header checksum: 0xd46d [correct]
        [Good: True]
        [Bad : False]
    Source: 1.2.44.101 (1.2.44.101)
    Destination: 1.2.32.250 (1.2.32.250)
Internet Control Message Protocol
    Type: 8 (Echo (ping) request)
    Code: 0 ()
    Checksum: 0xf112 [correct]
    Identifier: 0xdffd
    Sequence number: 0 (0x0000)
    Data (56 bytes)

0000  f8 17 44 fa 62 8d 00 00 80 11 42 da 8f e2 27 66   ..D.b.....B...'f
0010  8f e2 4e 0d 00 89 00 89 00 3a 2d df 00 00 00 00   ..N......:-.....
0020  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0030  00 00 00 00 00 00 00 00                           ........
        Data: F81744FA628D0000801142DA8FE227668FE24E0D00890089...
        [Length: 56]
4
  • I forgot to mention... I have replaced our actual class B block with 1.2.x.x above.
    – Brad
    Aug 31, 2010 at 16:56
  • Thanks again for everyone's suggestions. I apologize for my slow responses... I am working through several people on our end on this problem. This is also difficult to debug because currently, the problem went away on its own. This is a college campus network, and our network admin requested that students unplug all of their own personal wireless routers. This was to fix a separate issue with several DHCP servers on the network, but as a side effect, our broadcast storms have disappeared for now.
    – Brad
    Sep 6, 2010 at 3:26
  • This is an intriguing problem. I hope you'll keep us updated as you make progress.
    – Spiff
    Sep 7, 2010 at 23:46
  • Just an update... they called in some consultants who have set up sFlow to try to track down this, and a few other issues on our network. My hope is that the consultants will suggest properly segmenting our network, and that they will figure out where these packets were coming from. In the mean time, I am out of the loop. I am going to accept an answer for now, but I will definitely post back once I know what is going on. Thanks again to everyone for your suggestions. I greatly appreciate it.
    – Brad
    Sep 9, 2010 at 14:28

5 Answers 5

1

I'm going to dump a lot of disjoint thoughts and observations on you, in case any of them are able to stimulate your thinking or someone else's.

  1. The multicast destination MAC address corresponds to unicast IPv4 address of your firewall.
    It's as if some software took your firewall's IP address, pretended it was a multicast address, then followed the formula for generating a corresponding Ethernet multicast MAC address. That is, it took the last 23 bits of the IP address, and appended them to the end of the 01:00:5e OUI.

    I have a vague recollection that there may be protocols that do this (use a multicast address based on a unicast address), but my vague memories tell me this is something more likely to be done in IPv6 than IPv4. I'll have to research it some more later.

    Update: I was thinking of IPv6 Neighbor Discovery's (IPv6's ARP equivalent's) use of a "solicited-node multicast address". I'm not sure if this is much of a lead, though, because I can't see why anyone would want to do that for IPv4 pings.

  2. The payloads of those multicasted ping packets you captured don't look like typical ping payloads; they have meaningful data in them that might be a clue.

    Normal ping payloads usually contain every byte value from 0x00 to 0xff in order, so in the ASCII version of the dump, you'll see every character in order. These pings you captured seem to contain meaningful data. I see some definite IP addresses, and some possible port numbers:

    0000 f8 17 44 fa 62 8d 00 00 80 11 42 da 8f e2 27 66 ..D.b.....B...'f
    0010 8f e2 4e 0d 00 89 00 89 00 3a 2d df 00 00 00 00 ..N......:-.....

    At offset 0x000c, I see 8f e2 27 66, which is IP address [your class B].39.102. Reverse-DNS lookup on that reveals justin.[your school].edu. Is that hostname meaningful to you? Tracking down what that host is, and what software and services it's running (and possibly whether it's infected or pwned) might yield clues as to what this traffic is.

    Directly following that, at offset 0x0010, I see 8f e2 4e 0d, which is IP address [your class B].78.13, which I can't get a reverse-DNS result on, but maybe you can figure out what it is from where you are.

    Directly after those IP addresses, I see what I would guess to be two port numbers, 00 89 ( == 137, NetBIOS Name Service), and 00 89 again.

    Could this be a netbios-ns message that somehow got written out as ICMP instead of UDP? Seems unlikely. Too many fields in the wrong places, it seems to me.

    Was this supposed to be an ICMP destination unreachable message in response to a netbios-ns message, but the ICMP header got written wrong (as "echo request", instead of "destination unreachable")? Seems unlikely too. Too many fields in the wrong places again.

    Is this some sort of malware message, coordinating which hosts are infected/pwned and which ones to attack next? Hanlon's razor would seem to discount that, but I think it's plausible.

    Update: Come to think of it, perhaps the most likely possibility is that something is just re-using a buffer to fill the body of this ping request. Thus the contents of it might be a red herring.

  3. Are the TTLs of these frames always just 2, or do you see bursts of descending TTLs?
    Same TTLs always would indicate that the packet storm is something of a pure layer-2 bridging loop. Decrementing TTLs would indicate that layer 3, the IP layer, is getting involved; one of those student's personal wireless router's NAT gateway code could be buggy and forwarding certain frames it has no business forwarding, potentially creating a loop.

    That is, I'm suggesting that there might be two separate problems interacting here. One is whatever's sending these weird pings with meaningful payloads to a multicast MAC address but unicast IPv4 address. The second problem could be a separate buggy device that sees those frames it wouldn't normally see, and forwards them back onto the network an extra time, creating a loop.

2
  • Thank you for your comments. As for point #1, I don't know much about IPv6, so I don't know where to begin there. For point #2, I was suspicious of the ping data as well, but missed the IP addresses there. I too am leaning towards a malware issue. Looking up those IPs on our network... one was a UPS, and another was a student's computer. I am going back through our broadcast packet captures over the last couple days to see if anything is from these two.
    – Brad
    Sep 8, 2010 at 14:23
  • Point #3... The TTL levels are always the same. In prior broadcast storms I have seen where the TTL levels will differ by 1 or 2, but never the countdown that I had expected.
    – Brad
    Sep 8, 2010 at 14:24
2

9 times in 10 when I see something like that, it is a loop somewhere. If it is infrequent, someone is plugging something in, then getting baffled why it isn't working, then unplugging it again. This is murder to find in a lab or development environment, which is why I always try to insist on separating the two physically and with a firewall.

The next most frequent issue is some kind of viral activity. Less frequently it is a spanning-tree storm, and less frequently than that is an actually broken network device.

Loops have always been the low-hanging fruit for me. If spanning-tree isn't enabled you might look into that; some HP switches even have loop-protect features that will shut down a port if they detect a loop on that port. Very cool to see in action.

7
  • Thanks David. Any suggestions for tracking this stuff down?
    – Brad
    Aug 31, 2010 at 15:28
  • Draw out the switch to switch connections on a piece of paper, then check the spanning tree port settings on each switch. Depending on how the switches are linked together and the use of STP on the switches, you'll want to find at least one switch to switch link in the blocking\discarding state. Verify that there is only one path from any host to any other host. If you find more than one path (discount any STP ports in the blocking\discarding state) then you have a switch loop.
    – joeqwerty
    Aug 31, 2010 at 16:08
  • Thanks Joe for that information. We are digging around now. If there was a loop, what would cause this packet to appear in the first place?
    – Brad
    Aug 31, 2010 at 16:55
  • @Brad: I don't do packet analysis very often so I'm not very good at it. After looking at your Wireshark output I don't think you have a switch loop (although you should check it out to make sure) I think the switches have some default multicast settings enabled and they're sending some type of multicast discovery packets (maybe looking for a multicast router) so I'm of the opinion that you should look at the multicast configuration on each switch and disable the multicast settings if you're not using any type of multicasting in your network.
    – joeqwerty
    Aug 31, 2010 at 17:30
  • @Brad: In addition, what percentage of the total traffic is this multicast traffic? How many multicast packets are there out of how many total packets captured? Also, save your capture and open it in ColaSoft Capsa free edition. It has a nice visual display of packets captured, a diagnostic page, and a plethora of ways to look at the traffic captured. I find it easier to use than Wireshark in trying to "visualize" the problem.
    – joeqwerty
    Aug 31, 2010 at 17:40
1

check for the default advertisement timings for your routing protocols.

and what spanning tree setup are you using on these hp switches?

1
  • There are no routing protocols. We are on one big subnet. Unless... I'm not understanding what you are referring to. We are not using any spanning tree configuration... STP is disabled. We do not have redundant links, and some of our HP switches do not support it.
    – Brad
    Sep 8, 2010 at 15:10
1

Please can you provide more information about the destination IP address? From your capture, it appears to be a unicast address. Can you confirm that it comes from the same /16 (Class B) as the rest of your gear?

  • Does the address actually exist on your LAN?
  • If so, what type of device is it?

Edit based on comment:

Could you provide a topology diagram? If I understand you correctly, you have something like the following physical topology, with your switches plugged in to the firewall:

                       {
Router --- Firewall ---{ [multiple switches]
                       {

Some further followup questions:

  • Do you have switches uplinked into other switches?
  • What netmask are you using on all your hosts? Are you using /16 (255.255.0.0) on all devices, or do you have multiple smaller VLANs/subnets?
  • What default gateway is configured on your switches?
  • Does the destination MAC address in the captured traffic match the MAC address of the firewall interface which has the destination IP configured on it?

If you have a valid support contract, I'd consider opening a case with HP and asking them what would case the switches to use a multicast MAC address when sending traffic to a known unicast destination. This seems like a bug to me.

2
  • Murali, the destination IP address (1.2.32.250) is actually our gateway... our firewall, just before the router. It definitely exists on the network and I would guess that it receives the most traffic out of any device.
    – Brad
    Sep 6, 2010 at 3:22
  • Your network diagram is correct. When you get to the switches, there is a core switch with many switches attached, and then in the buildings, many more switches attached. Basically, a tree of switches. All straight-up Ethernet, with fiber in between buildings. No subnets... 255.255.0.0 is the netmask on everything. (Yeah, yeah, I know. Not my call...) The MAC address of the firewall does not appear in these packet captures at all.
    – Brad
    Sep 8, 2010 at 14:29
0

I'm not making sense of your question. You say that you have ping requests going to multicast addresses? I've never seen that and I'm wondering how exactly you determined that, are they actually ICMP echo request packets going to multicast addresses?. You also stated that the ping requests appear to be coming from your switches, but then you go on to say that the source MAC addresses are not from your switches so I'm confused as to what you're actually seeing. Can you give us more detail on what exactly you're seeing in the capture? Maybe even post a line or two of the capture showing the packets you're referring to. Thanks.

2
  • I'm sorry, it is a bit confusing. I have edited my post to include two samples of packets. The ping requests appear to be coming from the MAC addresses of our switches, but not the IP addresses of our switches. The destination MAC address is for multicast, but the destination IP address is that of our gateway.
    – Brad
    Aug 31, 2010 at 16:55
  • OK. I'm not very well versed in multicast traffic but it appears from the TTL that it's probably some type of multicast discovery packet based on the fact that it's ICMP traffic rather than IGMP traffic.
    – joeqwerty
    Aug 31, 2010 at 17:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .