I'm not our normal network guy... I've just been drafted to help with this issue, so please bear with me.
We have a fairly large (~4,000 devices?) network mostly comprised of HP Procurve gear. From time to time over the last two weeks, we've been getting some wicked broadcast storms that pretty much take down everything. I set up Wireshark to do 3MB dumps, and I caught some of this in the act this morning.
There are thousands of ping requests. They appear to come from the MAC addresses of our switches and APs, and are sent to an IPv4 multicast MAC. The source IP addresses do not match those of the switches and APs... they are IP addresses assigned to a few clients on the network. The destination IP address is always that of the gateway.
My read on this is that the firmware on these Procurve devices is either compromised or gone crazy... or someone is spoofing the addresses and causing this mess. Neither of which seem likely here, so I am asking if you have any ideas of other things to consider.
On another note, our network is not subnetted. (Yes, I know, I know... not my call, unfortunately.) Everything is flat.
Thank you for your time.
Edit: Below are two sequential packets from my capture. I posted the full Wireshark summary. I apologize that it is a bit messy, but it better explains what I am seeing.
No. Time Source Destination Protocol Info
16885 2.094869 1.2.41.194 1.2.32.250 ICMP Echo (ping) request
Frame 16885 (98 bytes on wire, 98 bytes captured)
Arrival Time: Aug 31, 2010 07:59:11.185552000
[Time delta from previous captured frame: 0.000123000 seconds]
[Time delta from previous displayed frame: 0.000123000 seconds]
[Time since reference or first frame: 2.094869000 seconds]
Frame Number: 16885
Frame Length: 98 bytes
Capture Length: 98 bytes
[Frame is marked: True]
[Protocols in frame: eth:ip:icmp:data]
[Coloring Rule Name: ICMP]
[Coloring Rule String: icmp || icmpv6]
Ethernet II, Src: Procurve_44:0f:26 (00:1f:fe:44:0f:26), Dst: IPv4mcast_62:20:fa (01:00:5e:62:20:fa)
Destination: IPv4mcast_62:20:fa (01:00:5e:62:20:fa)
Address: IPv4mcast_62:20:fa (01:00:5e:62:20:fa)
.... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Source: Procurve_44:0f:26 (00:1f:fe:44:0f:26)
Address: Procurve_44:0f:26 (00:1f:fe:44:0f:26)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Type: IP (0x0800)
Internet Protocol, Src: 1.2.41.194 (1.2.41.194), Dst: 1.2.32.250 (1.2.32.250)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 84
Identification: 0x7718 (30488)
Flags: 0x00
0.. = Reserved bit: Not Set
.0. = Don't fragment: Not Set
..0 = More fragments: Not Set
Fragment offset: 0
Time to live: 2
[Expert Info (Note/Sequence): "Time To Live" only 2]
[Message: "Time To Live" only 2]
[Severity level: Note]
[Group: Sequence]
Protocol: ICMP (0x01)
Header checksum: 0xd710 [correct]
[Good: True]
[Bad : False]
Source: 1.2.41.194 (1.2.41.194)
Destination: 1.2.32.250 (1.2.32.250)
Internet Control Message Protocol
Type: 8 (Echo (ping) request)
Code: 0 ()
Checksum: 0xf112 [correct]
Identifier: 0xdffd
Sequence number: 0 (0x0000)
Data (56 bytes)
0000 f8 17 44 fa 62 8d 00 00 80 11 42 da 8f e2 27 66 ..D.b.....B...'f
0010 8f e2 4e 0d 00 89 00 89 00 3a 2d df 00 00 00 00 ..N......:-.....
0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0030 00 00 00 00 00 00 00 00 ........
Data: F81744FA628D0000801142DA8FE227668FE24E0D00890089...
[Length: 56]
No. Time Source Destination Protocol Info
16886 2.094991 1.2.44.101 1.2.32.250 ICMP Echo (ping) request
Frame 16886 (98 bytes on wire, 98 bytes captured)
Arrival Time: Aug 31, 2010 07:59:11.185674000
[Time delta from previous captured frame: 0.000122000 seconds]
[Time delta from previous displayed frame: 0.000122000 seconds]
[Time since reference or first frame: 2.094991000 seconds]
Frame Number: 16886
Frame Length: 98 bytes
Capture Length: 98 bytes
[Frame is marked: True]
[Protocols in frame: eth:ip:icmp:data]
[Coloring Rule Name: ICMP]
[Coloring Rule String: icmp || icmpv6]
Ethernet II, Src: HewlettP_05:5e:70 (00:17:a4:05:5e:70), Dst: IPv4mcast_62:20:fa (01:00:5e:62:20:fa)
Destination: IPv4mcast_62:20:fa (01:00:5e:62:20:fa)
Address: IPv4mcast_62:20:fa (01:00:5e:62:20:fa)
.... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Source: HewlettP_05:5e:70 (00:17:a4:05:5e:70)
Address: HewlettP_05:5e:70 (00:17:a4:05:5e:70)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Type: IP (0x0800)
Internet Protocol, Src: 1.2.44.101 (1.2.44.101), Dst: 1.2.32.250 (1.2.32.250)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 84
Identification: 0x7718 (30488)
Flags: 0x00
0.. = Reserved bit: Not Set
.0. = Don't fragment: Not Set
..0 = More fragments: Not Set
Fragment offset: 0
Time to live: 2
[Expert Info (Note/Sequence): "Time To Live" only 2]
[Message: "Time To Live" only 2]
[Severity level: Note]
[Group: Sequence]
Protocol: ICMP (0x01)
Header checksum: 0xd46d [correct]
[Good: True]
[Bad : False]
Source: 1.2.44.101 (1.2.44.101)
Destination: 1.2.32.250 (1.2.32.250)
Internet Control Message Protocol
Type: 8 (Echo (ping) request)
Code: 0 ()
Checksum: 0xf112 [correct]
Identifier: 0xdffd
Sequence number: 0 (0x0000)
Data (56 bytes)
0000 f8 17 44 fa 62 8d 00 00 80 11 42 da 8f e2 27 66 ..D.b.....B...'f
0010 8f e2 4e 0d 00 89 00 89 00 3a 2d df 00 00 00 00 ..N......:-.....
0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0030 00 00 00 00 00 00 00 00 ........
Data: F81744FA628D0000801142DA8FE227668FE24E0D00890089...
[Length: 56]