0

I'm running an EKS cluster on AWS and I've been having periodic outages recently. I tracked down the issue to duplicate network interfaces on one of my nodes, which I believe are created by the AWS CNI plugin.

When the node is created, it has 2 NICs. But at some point later in the node's life, a 3rd NIC is created (not by me) which causes strange behavior. It looks like the rp_filter is rejecting packets from the ELB as the SYN packet shows up in the messages log as a martian.

I should add that this only seems to happen for services that are annotated as service.beta.kubernetes.io/external-traffic: OnlyLocal

When I detach the 3rd NIC, everything returns to normal.

My questions are

  1. Why might the 3rd NIC cause this problem? Is this a bug in CNI?
  2. How can I workaround this issue?
    1. Is there some way I can prevent the 3rd NIC from being attached?
    2. Is there some way to configure CNI so that it attaches the 3rd NIC correctly? Do I have EKS configured incorrectly somehow?

Some evidence:

netstat -s is showing IPReversePathFilter: 666179 and increments by one for every SYN from the ELB exactly.

tail /var/log/messages shows the martian packets like so

Nov 22 18:37:17 ip-192-168-90-83 kernel: IPv4: martian source 192.168.102.150 from 192.168.236.246, on dev eth0
Nov 22 18:37:17 ip-192-168-90-83 kernel: ll header: 00000000: 0e 76 b3 cc 10 f4 0e 6f 51 60 fa be 08 00        .v.....oQ`....

When I set rp_filter=0, then the SYN from the ELB isn't dropped anymore and I see the SYN-ACKs head toward the ELB, but they never make it there.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.