Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

This afternoon, with no changes to the network, a switch suddenly started dropping off lots of connections. These connections would come back up a few minutes later, then another area connected to the switch would drop off. This is an older 4006 chassis switch which could in and of itself be a problem but I'm looking to see what else you all would look for in trying to find a root cause.

Switch is connected via ports 1/1 and 1/2 in an etherchannel to a VSS core 1/1/42 and 2/1/42. Both sides are up and working however the CPU on the switch will spike up to 99% and that's when CRC errors start to hit the VSS core on one of those interfaces and end points start dropping off. We tried new transceivers and SFP's on each side of the link, same result. When we tried swapping the fiber patch cables on the access switch the CRC errors did not follow the fiber cables they stayed with port 1/2 on the access switch. So port 1/2 on the supervisor module looks like the culprit. We actually tried to create a new member of the ethernet channel by taking a fiber media converter to cat5 and make that a member of the port-channel but when we plugged it in you couldn't even reach the switch. I'm guessing that's unrelated and a problem with the media converter.

As of right now we have left it in a state of only one fiber cable running to one side of the VSS core (1/1 Access Switch --> 2/1/42). I've sent some info into TAC and they are looking into the situation but does anyone else have any commands I could run or some troubleshooting I could look into in the meantime?

share|improve this question
    
Have you done a show etherchannel of the gig1/1 and gig1/2 channel group? I'm wondering if someone might have accidently added some other port to the group. –  Bad Dos Sep 17 '12 at 20:03

1 Answer 1

CPU spikes of that magnitude are usually either some kind of spanning tree event (read: loop) or excessive traffic hitting the control plane (broadcast/multicast storms most commonly).

Is the 4006 connected to any other downstream switches? Is there any possibility of that 4K being accidentally cross-connected to another switch?

Are you graphing traffic on these switch ports? Look for radically higher traffic levels. TAC is going to also going to want to figure out what process is consuming all that CPU - what are the major consumers from show proc cpu? Are you also seeing high CPU utilization on the 4K?

Similarly, are there unusual syslogs around the times where you're seeing high CPU? Any changes (to the network or hosts) around the time this all began? At a simpler level, what's the port channel configurations look like on either side LACP? Static? Misconfiguration in this area might cause some problem behaviors.

share|improve this answer
    
At the time of one of the events the Cat4k Mgmt HiPri and the Cat4k Mgmt LoPri were starting to peg out at 14% and 70% respectively. These are normally going to be involved in spanning-tree unfortunately it looks as if there is no spanning-tree convergence going on. –  user135458 Sep 11 '12 at 13:15
    
What about any traffic spikes? –  rnxrx Sep 11 '12 at 18:38

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.