2

We are having an issue with our Catalyst 6500 switch where we suspect that the ARP cache is being corrupted. This presents itself with the following symptoms:

  1. When you attempt to ping a system which has not been resolved before, the first ping response times out, and each proceeding one succeeds: Pinging foo.network.com [xxx.xx.xx.xx] with 32 bytes of data: Request timed out. Reply from xxx.xx.xx.xx: bytes=32 time=5ms TTL=55 Reply from xxx.xx.xx.xx: bytes=32 time=3ms TTL=55 Reply from xxx.xx.xx.xx: bytes=32 time=3ms TTL=55

  2. When the corruption issues occurs, every other ping times out: Pinging foo.network.com [xxx.xx.xx.xx] with 32 bytes of data: Reply from xxx.xx.xx.xx: bytes=32 time=5ms TTL=55 Request timed out. Reply from xxx.xx.xx.xx: bytes=32 time=5ms TTL=55 Request timed out.

  3. Clearing the ARP cache temporarily resolves the issue. To clear the ARP cache we use the commands: clear arp cache clear ip cache This fixes it, but it is sure to happen again.

Details on the switch:

IOS (tm) s72033_rp Software (s72033_rp-PK9SV-M), Version 12.2(17d)SXB8, RELEASE SOFTWARE (fc2)

cisco WS-C6509-E (R7000) processor (revision 1.1)

Any help appreciated, Thanks

CLARIFICATION: We have the network that we manage, and then we are plugged into the corporate network. All requests to machines inside of the network that we manage work fine. We are only having problems with machines on the other network.

5
  • What does the mac-address-table show for the MAC address of the device having problems?
    – Joseph
    Jul 9, 2009 at 21:19
  • Do keep us updated, if you would. I'm still lobbying for my guess that some device is "spoofing" its source address (either because of a bug in the device or maliciously) and that you'll find the errant device thru "show mac-" or "show arp". Jul 10, 2009 at 12:14
  • I will keep you updated. Although I have access to the switch, I really need to have the CCNA do this work, and that requires a service request utilizing form 44-A and management approval if it needs to be prioritized above all other work. I am going through some of the above and seeing what I can find. As for the 'show mac-address-table' I am not sure that I can post it here, it is literally pages and pages long.
    – esac
    Jul 10, 2009 at 19:13
  • We are only interested in the entries related to foo.network.com. So find the MAC address and type 'show mac-address-table address xxxx.xxxx.xxxx'
    – Joseph
    Jul 10, 2009 at 20:49
  • @Joseph: You're assuming that there isn't a router between foo.network.com and the 6509. If there is, the next-hop-router's MAC address is what we're interested in. Jul 10, 2009 at 20:55

5 Answers 5

1

I would suggest you to open a case to Cisco.
They will be able to check for know bugs on your IOS version and will ask you configuration details that you may don't want to publish here. (but if you want you can put the result of a sh tech somewhere it could help us)
Also doest it append after a reboot or did it start to get corrupt after a long uptime ?

4
  • Aww... you're giving up too easily! Jul 9, 2009 at 21:24
  • There is too much possibility. Troubleshooting this would require access to the switch. I already this kind of problem on 2960, the problem was occurring even after reboot and a rma was done. I also see something like this on 3570 after unstacking/stacking and a full reboot of the stack has fixed the problem. Of course the problem could also come from a network device... but you asked him good question ;)
    – radius
    Jul 9, 2009 at 21:44
  • We already had help from Cisco, and it got to the point where they couldn't give anymore 'free' help and it isn't enough of an issue to warrant a support contract. We have an onsite CCNA who can handle 99% of the issues, just not this one. As for reboot, it generally happens fairly soon after a reboot, although we haven't done a test specifically for that.
    – esac
    Jul 9, 2009 at 22:02
  • When corruption occur, does it appends for only one host or for all host at the same time ? I mean is the problem start to be global for everybody at the same or does the problem appends for one IP then another IP and so on ?
    – radius
    Jul 9, 2009 at 22:09
1
  • You're seeing this problem with PINGs from the switch's CLI, or from a PC connected to the switch?

  • Is this switch providing layer 3 (routing) functions?

  • Are these PINGs your showing having problems between two devices on the same subnet, or across subnets?

  • Does the log on the switch ("show log hist", I believe) show anything amiss?

  • Is the issue affecting packet delivery to only the a couple of device(s), or are you seeing it affecting a number of devices?

I had a similiar issue to this at a Customer site a few years back. I captured the output of a "show mac-" prior to the issue occurring, and then during the issue occurring, and compared looking for devices that appeared to be on different ports prior to the outage starting and after.

I found that there was an embedded device on the LAN (a clock, in this case) that would periodically transmit a batch of frames with a "spoofed" source address, confusing the switch's bridging table and causing the switch to send frames out the wrong port for awhile. I was able to see it in the "show mac-" output by noticing that devices that should not have been changing ports appeared to be doing so.

Sounds like fun to troubleshoot! Wish I were there... >smile<

Edit:

Thanks for the comments.

"show log hist" shows a persistent log. As long as you're not clearing the log, any messages reported there will still be there after you clear the arp cache on the switch.

Is there any other router between your 6509 and the corporate datacenter where the problem-devices live?

Are you using any dynamic routing protocols?

Here's what my gut says:

I'm going to strongly recommend that you save a copy of "show mac-" and "show arp" before a failure occurs and again when a failure is occurring (it should only take a moment to capture them with something like PuTTY, so you can get on with clearing the arp cache quickly).

I realize you can't easily post these captures here, but I'd recommend that you throw them into a spreadsheet or database and match up MAC address against ports in one report, and MAC addresses against IP address in another. If you compare "before" and "during", I predict you're going to see some differences.

Assuming there's a router between your 6509 and the corporate data center, I predict that you're going to find that router's MAC address to be "moving" between ports, or its IP address moving between MAC addresses.

If there's no router and the corporate data-center machines are talking to this 6509 at layer 2 I'll predict that the devices themselves might show some "moving" between ports, or moving IP addresses between MAC addresses.

6
  • You're seeing this problem with PINGs from the switch's CLI, or from a PC connected to the switch? ...... both
    – esac
    Jul 10, 2009 at 19:54
  • • Is this switch providing layer 3 (routing) functions? ....... Yes, it’s the core switch that we have in our network, and performs all routing
    – esac
    Jul 10, 2009 at 19:55
  • • Are these PINGs your showing having problems between two devices on the same subnet, or across subnets? ...... It only happens when you try to hit things outside of our network, so they’re on a different subnet. Specifically, we think we see these problems only with machines housed in a remote datacenter.
    – esac
    Jul 10, 2009 at 20:03
  • • Does the log on the switch ("show log hist", I believe) show anything amiss? .... Not to my knowledge. However everytime we hit this issue, somebody needs it fixed right away so we just flush the arp cache so they can get their job done, and we don't really have time to investigate. We can see if we can get this really quickly next time.
    – esac
    Jul 10, 2009 at 20:05
  • Is the issue affecting packet delivery to only the a couple of device(s), or are you seeing it affecting a number of devices? ..... If by devices you mean machines on the network that we manage, then yes it affects multiple devices. Please see my clarification on the original post.
    – esac
    Jul 10, 2009 at 20:07
0

If you run a sniffer on the client being ping'd do you see all of the pings or only half of them?

What happens if you source the pings from different interfaces on the 6500? Does it happen for hosts that the 6500 is the default gateway for?

What does the mac address table look like? How about a traceroute? And a 'ping -r9 '?

Don't rule out an IOS bug, but it could also be a lot of other things...

0

It can be case of ARP spoofing. If some one is trying to spoof all the address on the network including gateway the spoofing machine will get too much traffic and hence may not be able to transfer all data to correct addresses after reading it. Or spoofing machine can intentionally drop additional packets.

Run wireshark. Then use "arp -d" to delete arp entries of all IP address on your subnet. Then try to ping few IPs on your subnet. Then stop capturing packets from wireshark and just analyze ARP traffic. If you see multiple ARP responses for each IP you pinged like IP 172.16.1.1 is at xx:xx:xx:xx:xx:xx0 followed by IP address 172.16.1,1 is at yy:yy:yy:yy:yy:yy. Then it is definitely case of ARP spoofing and there is nothing wrong with the switch.

In case this does not work. Try upgrading switch IOS to latest version.

0

I have to agree with Peter and Evan. This sounds more like a bouncing route/port than a cache attack. Especially on a 65xx. To amplify Evan's comment, be sure to get the (working) arp table, but the only entry you'll really need is the next-hop router. Have you ruled out multi-path problems? I saw someone ask if you were running a dynamic routing protocol (or multiple gateways w/floating static routes) but I haven't seen your answer. Good luck!

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .