4

Short Version: We've had a problem with a particular server receiving massive duplicates of packets. After much fault finding we found the problem was caused by a switch sending my packet out every port (ie acting like a hub) and 3 routers were reflecting the packet back. When the routers reflected the packet back the switch would again send the packet out every other port and they would again get reflected back by the other routers. This would last until the TTL decremented to 0. What I don't understand is why the routers were sending the packet back. Shouldn't the router drop the packet as the dest mac address doesn't match its mac address?

Long Version: Say there are 3 routers (A, B and C) attached to a switch and my VM. A udp packet would arrive from router A destined for my VM. At this stage the TTL is 60. The switch doesn't have my VM's mac address so it floods the packet out every other port. Routers B and C receive the packet and see the IP address is on the same port so decrement the TTL and send the packet back out the same port. We now have 2 packets with TTL of 59. The switch again floods the 2 packets out and it is reflected off the other 2 routers, so we end up with 4 packets with TTL of 58. We then get 8 at 57, 16 at 56 etc until packets start to get dropped. All the while my VM is receiving a copy of all these packets. Because the VM only ever receives it never manages to get itself into the switch's MAC table. ARP helps here but it's timeout was larger than the switches timeout on the mac table. Lowering that solved the problem but the question remains as to why the routers were reflecting the packet back. From what I understand this isn't standard behavior.

EDIT: Details from the wireshark capture (showing last 2 digits of mac only):

MAC    TTL
95     60

99     59
EC     59

95     58
EC     58
95     58
99     58

then 8 packets at 57 etc

See how it looks like it comes in from MAC address ending with 95, then bounces off the other 2 routers, then both those packets bounce off the other 2 again to make 4 packets?

  • Most managed switches have a way to add static entries to the Ethernet switching tables (MAC address to port mapping). – Brian Aug 26 '14 at 2:29
  • 1
    Are you certain the routers are sending the packet back? It sounds like you have a L2 or L3 loop. What is the source and destination MAC addresses of the troublesome packet, and is the IP Address unicast or broadcast? I have very recently witnessed very similar behavior when I accidentally created an L3 loop using proxy-arp. This meant that the router accepted the UDP packet as the destination MAC matched an address that it was acting as a proxy ARP for. Is it possible that one of your routers has a second link back to the switch? Are the routers connected to each other as well as the switch? – blacklight Aug 26 '14 at 5:51
  • That is a good question as I'm not 100% certain of what is happening. The reason I think we don't have an L2 loop is because the TTL decremented. I don't think we have an L3 loop because the TTL decremented by 1 (surely a loop would require it going through another router hence decrement by 2 at least). I will update my question with the wireshark capture details. – MikeKulls Aug 27 '14 at 6:36
  • Does all of the packets have the same destination MAC address? Is it a globally unique unicast MAC address? What is the first byte of the MAC address? – kasperd Aug 27 '14 at 16:59
  • I can only see the packets when they arrive at my VM and they all have the dest mac of my VM. It is possible they have a different mac at other times although I suspect it is not likely. The full address of my VM is 00:0C:29:A2:29:5B – MikeKulls Aug 27 '14 at 21:52
4

What you are describing is what happens when a switch's CAM table is full, where it can no longer learn MAC address and it forwards packets out every port. It might be hard to figure out if this is the problem with an unmanaged switch, but with a managed switch you should be able to display the CAM table.

What also would help in this question is a diagram of the network.

EDIT:

From diagram added, you can see you have two problems. The first still remains where the switch is going to forward the packet out every port if it doesn't know where to go. Check the logs to see why your MAC address is being removed from the table.

The second problem is that the routers recieves the packet and send it back to the switch, which in turn, sends it to every port (except its own). Is there a particular port set up with a default route, 0.0.0.0?

  • 1
    We checked and the MAC table isn't filling up. What happens is my MAC is being removed from the table after the 5 minute timeout. Once it is removed the only thing that puts it back in is ARP (as my server is silent otherwise). Because the ARP timer was longer than 5 minutes this caused the switch to not have my mac. – MikeKulls Aug 27 '14 at 6:33
  • I should add I agree that this looks like the likely cause and this was the first thing I suggested but apparently the MAC table doesn't have that many entries in it. – MikeKulls Aug 27 '14 at 8:52
  • From my understanding the switch flooding the packet is explained in that the switch discards entries after 5 mins and it has no mechanism to get back in there until the next ARP. That shouldn't be a problem as the network should function even if we put an old fashioned hub in there. What is a mystery is why the routers wouldn't just discard the packet. As someone at work suggested the routers should discard the packet because the dest mac doesn't belong to it. I can check if there is a route 0.0.0.0 but should that make a difference? – MikeKulls Aug 27 '14 at 21:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.