Linux: force source address for packets to locally bound IPs

Question

I have 10.1.1.1 and 10.2.2.2 bound to eth0.

# ip address show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 04:01:05:ff:42:01 brd ff:ff:ff:ff:ff:ff
    inet 10.2.2.2/24 brd 10.2.2.255 scope global eth0
    inet 10.1.1.1/32 scope global eth0
    inet6 fe80::601:5ff:feff:4201/64 scope link 
       valid_lft forever preferred_lft forever
# ip route get 10.1.1.1
local 10.1.1.1 dev lo  src 10.1.1.1 

Question A: why does it use "dev lo" when the IP is bound to eth0?

I want to force local connections to 10.1.1.1 to have a source address of 10.2.2.2. So I try:

# ip route add 10.1.1.1/32 dev eth0 src 10.2.2.2
# ip route show
default via 10.2.2.1 dev eth0 
10.1.1.1 dev eth0  scope link  src 10.2.2.2 
10.2.2.0/24 dev eth0  proto kernel  scope link  src 10.2.2.2 
# ip route get 10.1.1.1
local 10.1.1.1 dev lo  src 10.1.1.1 

Question B: How do I override this strange "local" route?

Background: the packets are intercepted by the IPVS kernel subsystem. Before it gets there, I need the correct source address, otherwise IPVS doesn't know what to do with it. SNATting with iptables doesn't work, as that happens in the POSTROUTING phase, and IPVS skips that. See also my related question at Linux IPVS in DR mode: VIP unreachable for director

Thanks!

Answer 1

The local table (table 255) is consulted before main, and contains all local routes (hence the name). It is maintained by the kernel (hence proto kernel). On your machine, it likely looks something like this.

local 10.1.1.1 dev eth0 proto kernel scope host src 10.1.1.1
broadcast 10.2.2.0 dev eth0 proto kernel scope link src 10.2.2.2
local 10.2.2.2 dev eth0 proto kernel scope host src 10.2.2.2
broadcast 10.2.2.255 dev eth0 proto kernel scope link src 10.2.2.2
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope host src 127.0.0.1

The source addresses are specified therein.

Linux supports more than one routing table, and routing involves consulting the use of the routing policy database (RPDB) to decide when and which table to consult. If a table doesn't contain the answer, or a route is of type throw, the next RPDB rule is consulted.

$ ip ru
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

You could try removing the rule for the local table lookup, and place it later (ip rule del pref 0; ip rule add from all lookup local pref 1), which isn't recommended. You'd then prepend a rule to consult a custom routing table (pick any number of your fancy above 255), in which there would be a sole route for 10.1.1.1 with the source address set to 10.2.2.2. The better way to do this is to just call bind() if you are the author of the programs in question that'll be talking to 10.1.1.1.