5

We are having an intermittent fault which is effecting one of our control systems on one of our HP Procurve switches.
For some reason, this PLC (10mbit port - 192.168.6.56) which is attached directly to the HP Switch intermittantly start's receiving data which is not destined for it. The data is being sent from a Thecus NAS with latest firmware (192.168.6.218) to a physical IBM Server running Win2003R2 and SAP (192.168.6.225). The problem does not just send to this server, it has been to other physical servers in the past too, but always from the Thecus NAS.
I am using a monitor port to wireshark what is going in/out of the PLC - normally there would be about 1mb in/out per 2 or 3 minutes - only a server asking the state of the coils. When the problem occurs, there is a flood of data being put onto the PLC line - in this captured instance, about 67mb in less than a minute.
Due to this, there is no way that the PLC can be queried as the port is effectively DOSed, in turn killing part of our factory. I know that having Production on the same vlan as IT is not a good idea - I agree, however it cannot be changed at the moment (will have to wait 3 months), as well as the problem has only started happening in the last 3 months. Here is a screen cap of one of the packets being sent from the Thecus NAS which was captured from the PLC port on the HP Switch:
Screen Grab of stray packet
And there are over 700 of these in this one 1024kb file.
If anyone has any idea on what could be going on, some help would be greatly appreciated. If you need to know anything more, let me know!
Cheers!

7
  • 1
    It could be unicast flooding. This could occur due to the switch's forwarding table aging out too quickly or it could be the switch's forwarding table overflowing. Do you have the HP equivalent of portfast enabled on all host connected ports on the switch?
    – joeqwerty
    Oct 22, 2012 at 0:11
  • STP is enabled - 802.1t Oct 22, 2012 at 1:04
  • It does look like unicast flooding, and I have extended the mac age timeout from 300ms to 1200ms to test overnight. Will see how that goes. Cheers Oct 22, 2012 at 5:12
  • Based on this question and your previous one, your PLC is clearly cursed. You should drive a wooden stake though it or shoot it with a silver bullet or something... More seriously, instead of doing Modbus over Ethernet, is there any way you could just use "classic" Modbus, with RS-485 wiring (or whatever it is, it's been 10+ years since I worked on a Modicon PLC) Oct 22, 2012 at 5:39
  • 1
    300ms? That seems way too short. 1200ms seems way to short. I'm pretty certain that Cisco switches have a default aging-time of 300 seconds for the mac address table. Are you sure that your switch uses milliseconds and not seconds?
    – joeqwerty
    Oct 22, 2012 at 13:59

3 Answers 3

1

Is the CAM (MAC address) table on your switch overloading? If so, it will send traffic out all ports because it doesn't know which port it's supposed to use--this essentially turns the switch into a hub. A common attack is to flood the CAM table of a router with invalid MAC addresses until the CAM table falls over, then sniff all the traffic coming into the attacking host.

http://en.wikipedia.org/wiki/MAC_flooding

This can also happen with misconfigured equipment. Did you add anything new to your network around the time this started happening?

You can configure port security on most HP switches, which will limit the number of MAC addresses each port can learn, and mitigate the attack:

http://www.hp.com/rnd/device_help/help/hpwnd/webhelp/HPJ4121A/security_perports.htm

0

Make sure that the traffic is really coming from Thecus NAS. I think that somebody is spoofing the mac address.

0

Looks like there is a possible fault on the Thecus, however HP support will look at it again when the fault occurs - they are wanting to see if it is the switch that is causing the flood, or the nas.
They want to see two different captures when the fault occurs: a capture of the NAS using a mirrored port, as well as a capture of the switch (just plug laptop into switch, and don't mirror anything) while the fault happens.
Thank you all for the assistance!

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .