Extra ip rule table breaking my connectivity with container

Question

I have host with one interface that has the ip 10.0.10.5/28. The host has one container with the interface cali02ad7e68ce1 and ip 10.42.1.2/26. This is the main routing table of the host:

$> ip r list table main
default via 10.0.10.1 dev eth0 proto dhcp metric 100 
10.0.10.0/28 dev eth0 proto kernel scope link src 10.0.10.5 metric 100 
10.42.1.2 dev cali02ad7e68ce1 scope link 

This is the list of ip rules:

$> ip rule
0:  from all lookup local
30400:  from 10.0.10.5 lookup 30400
32766:  from all lookup main
32767:  from all lookup default

And this is the routing table 30400:

$> ip r list table 30400
default via 10.0.10.1 dev eth0 proto static metric 10 
10.0.10.1 dev eth0 proto static scope link metric 10 

When I try to ping the container ping 10.42.1.2, I receive no packets. However, if I tcpdump on the container's interface, I can see both echo request and echo reply.

$> sudo tcpdump -eni cali02ad7e68ce1
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on cali02ad7e68ce1, link-type EN10MB (Ethernet), capture size 262144 bytes
15:57:24.589384 ee:ee:ee:ee:ee:ee > 4e:b1:cd:f0:62:82, ethertype IPv4 (0x0800), length 98: 10.0.10.5 > 10.42.1.2: ICMP echo request, id 34667, seq 1, length 64
15:57:24.589405 4e:b1:cd:f0:62:82 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 98: 10.42.1.2 > 10.0.10.5: ICMP echo reply, id 34667, seq 1, length 64
15:57:25.637186 ee:ee:ee:ee:ee:ee > 4e:b1:cd:f0:62:82, ethertype IPv4 (0x0800), length 98: 10.0.10.5 > 10.42.1.2: ICMP echo request, id 34667, seq 2, length 64
15:57:25.637216 4e:b1:cd:f0:62:82 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 98: 10.42.1.2 > 10.0.10.5: ICMP echo reply, id 34667, seq 2, length 64

As soon as I delete the rule 30400, ping works. I am confused because I don't understand how that rule makes the echo reply to never reach the ping process. AFAIK, that rule should only apply when 10.0.10.5 is source ip. Any help or guess will be appreciated!

UPDATE

Adding bridge info as requested in the comments:

$> ip -br link; ip -br address`
lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP> 
eth0             UP             06:e4:85:e5:1b:94 <BROADCAST,MULTICAST,UP,LOWER_UP> 
cali02ad7e68ce1@if3 UP             ee:ee:ee:ee:ee:ee <BROADCAST,MULTICAST,UP,LOWER_UP> 
lo               UNKNOWN        127.0.0.1/8 ::1/128 
eth0             UP             10.0.10.5/28 
cali02ad7e68ce1@if3 UP             fe80::ecee:eeff:feee:eeee/64 

$> ip -br link show type bridge
$> ip -br link show type bridge_slave
$> 

UPDATE 2

$> ping 10.42.1.2 &
[1] 1645713
$> PING 10.42.1.2 (10.42.1.2) 56(84) bytes of data.

$> sudo nstat -az TcpExtIPReversePathFilter
#kernel
TcpExtIPReversePathFilter       0                  0.0
$> sudo nstat -az TcpExtIPReversePathFilter
#kernel
TcpExtIPReversePathFilter       0                  0.0
$> ip route get 10.0.10.5 from 10.42.1.2 iif cali02ad7e68ce1
local 10.0.10.5 from 10.42.1.2 dev lo table local 
    cache <local> iif cali02ad7e68ce1 

Answer 1

It seems like the reverse path checking issue. Please check the counter with nstat -az TcpExtIPReversePathFilter. If the counter is being incremented, then check the route to source of these packets with command

ip route get 10.0.10.5 from 10.42.1.2 iif cali02ad7e68ce1

Likely you will get the error.

The reverse path filter checks the route to the source of incoming packets and can drop the packets received on unexpected interface. The action depends on the value of sysctl variable net.ipv4.conf.<iface>.rp_filter.

The solutions:

• Disable the rp_filter or set it into the loose mode (value 2 of net.ipv4.conf.<iface>.rp_filter) - see the sysctl documentation.
• Set the additional route to container in the 30400 routing table.

Update:

The output of commands shows the reverse path filter doesn't drop any packets (SNMP counter is zero, route is valid). So the cause of issue is in something other. Check the firewall rules with iptables-save -c command. It shows the full rule set with hit counters. May be some rule blocks the reply packets.