Failed to ping through gateway which is not default gateway

Question

I have the following network interface configuration,

• eth0: 192.168.78.1/24
• eth1: 192.168.254.76/24 (Gateway 192.168.254.1)
• eth2: 192.168.253.131/24 (Gateway 192.168.253.1)

where both gateways have access to internet.

In this scenario ping -I eth1 4.2.2.2 is not working. Below are outputs of commands:

$ ip route show
192.168.78.0/24 dev eth0  proto kernel  scope link  src 192.168.78.1 
192.168.253.0/24 dev eth2  proto kernel  scope link  src 192.168.253.131 
192.168.254.0/24 dev eth1  proto kernel  scope link  src 192.168.254.65 
169.254.0.0/16 dev eth0  scope link  metric 1002 
169.254.0.0/16 dev eth1  scope link  metric 1003 
169.254.0.0/16 dev eth2  scope link  metric 1004 
default via 192.168.253.1 dev eth2 

$ cat /etc/iproute2/rt_tables
255 local
254 main
253 default
0   unspec
1    eth1
2    eth2

$ ip rule ls
0:  from all lookup local 
32765:  from 192.168.254.65 lookup eth1 
32766:  from all lookup main 
32767:  from all lookup default 


$ ip route show table eth1
default via 192.168.254.1 dev eth1

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 100
    link/ether 00:0b:ab:65:10:97 brd ff:ff:ff:ff:ff:ff
    inet 192.168.78.1/24 brd 192.168.78.255 scope global eth0
    inet6 fe80::20b:abff:fe65:1097/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 100
    link/ether 00:0b:ab:65:10:98 brd ff:ff:ff:ff:ff:ff
    inet 192.168.254.65/24 brd 192.168.254.255 scope global eth1
    inet6 fe80::20b:abff:fe65:1098/64 scope link
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 100
    link/ether 00:0b:ab:65:10:99 brd ff:ff:ff:ff:ff:ff
    inet 192.168.253.131/24 brd 192.168.253.255 scope global eth2
    inet6 fe80::20b:abff:fe65:1099/64 scope link
       valid_lft forever preferred_lft forever
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether 00:0b:ab:65:10:9a brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether 00:0b:ab:65:10:9b brd ff:ff:ff:ff:ff:ff
7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether 00:0b:ab:65:10:9c brd ff:ff:ff:ff:ff:ff
8: sit0: <NOARP> mtu 1480 qdisc noop state DOWN
    link/sit 0.0.0.0 brd 0.0.0.0

All rp_filter values are set to 0.

Note: traceroute -i eth1 4.2.2.2 is working as expected, but ping through interface eth1 is unsuccessful if it is not default route.

Please let me know if any more setting is required or if you have any pointers.

Answer 1

As far I understand it shouldn't work the way you think. Packets will go through the default gateway unless you add a static route for a concrete destination IP or use policy routing. So try adding some policy routing rules:

echo  300 eth2_gate >> /etc/iproute2/rt_tables
ip route add default via 192.168.254.1 table eth2_gate
ip rule add from 192.168.253.131 table eth2_gate

After that ping -I should work. I hope it helps.

I am also confused why does the traceroute -s work though.

Answer 2

Maybe it is related with rp_filter. See this response: https://superuser.com/a/1504004/162485