0

Our CentOS 8 server at Hetzner has a primary IP aquired through DHCP, and an additional /29 subnet. No matter what outgoing IP I choose, packets are always sent with the primary IP. I would like to choose different outgoing IPs for different services, i.e. DNS and SMTP.

Anonymized setup:

  • Primary IP: 1.0.0.213/27
  • Secondary subnet: 2.0.0.88/29
  • Test IP: 2.0.0.89
  • NIC: enp3s0

Add IP secondary subnet:

$ nmcli con modify enp3s0 ipv4.addresses "2.0.0.89/32"
$ reboot

Configuration:

$ ip addr
1: lo: ...
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff
    inet 2.0.0.89/32 scope global noprefixroute enp3s0
       valid_lft forever preferred_lft forever
    inet 1.0.0.213/27 brd 1.0.0.223 scope global dynamic noprefixroute enp3s0
       valid_lft 43105sec preferred_lft 43105sec
    inet6 ...
    inet6 ...

$ ip route
default via 1.0.0.193 dev enp3s0 proto dhcp metric 100 
1.0.0.192/27 dev enp3s0 proto kernel scope link src 1.0.0.213 metric 100 

Test with our own what's-my-ip service hosted elsewhere:

$ curl -v -v -v -v -v --silent --interface 2.0.0.89 ip.mycompany.tld 
* Rebuilt URL to: ip.mycompany.tld/
*   Trying XX.XX.XX.XX...
* TCP_NODELAY set
* Name '2.0.0.89' family 2 resolved to '2.0.0.89' family 2 <-- correctly bound to 2.0.0.89?
* Local port: 0
* Connected to ip.mycompany.tld (XX.XX.XX.XX) port 80 (#0)
> GET / HTTP/1.1
...
1.0.0.213 <-- Expected 2.0.0.89 here?!?

Anonymized tcpdump confirms we're using the primary IP:

16:29:57.850153 IP (tos 0x0, ttl 64, id 4961, offset 0, flags [DF], proto TCP (6), length 60)                                                                   
    1.0.0.213.47307 > XX.XX.XX.XX.http: Flags [S], cksum 0xcae8 (incorrect -> 0x3f22), seq 1180758295, win 29200, options [mss 1460,sackOK,TS val 4199670061 ecr
 0,nop,wscale 7], length 0                                                                                                                                      
16:29:57.861321 IP (tos 0x0, ttl 57, id 0, offset 0, flags [DF], proto TCP (6), length 60)                                                                      
    XX.XX.XX.XX.http > 1.0.0.213.47307: Flags [S.], cksum 0xe167 (correct), seq 1203046991, ack 1180758296, win 28960, options [mss 1460,sackOK,TS val 255416431
1 ecr 4199670061,nop,wscale 7], length 0                                                                                                                        
16:29:57.861374 IP (tos 0x0, ttl 64, id 4962, offset 0, flags [DF], proto TCP (6), length 52)                                                                   
    1.0.0.213.47307 > XX.XX.XX.XX.http: Flags [.], cksum 0xcae0 (incorrect -> 0x8064), seq 1, ack 1, win 229, options [nop,nop,TS val 4199670072 ecr 2554164311]
, length 0                                                                                                                                                      
16:29:57.861468 IP (tos 0x0, ttl 64, id 4963, offset 0, flags [DF], proto TCP (6), length 133)                                                                  
    1.0.0.213.47307 > XX.XX.XX.XX.http: Flags [P.], cksum 0xcb31 (incorrect -> 0xf6dc), seq 1:82, ack 1, win 229, options [nop,nop,TS val 4199670072 ecr 2554164
311], length 81: HTTP, length: 81                                                                                                                               

Hetzner's support page on additional IPs: https://docs.hetzner.com/robot/dedicated-server/ip/additional-ip-adresses/

What am I missing here? :-)

  • CentOS 8.2.2004
  • Firewalld + NetworkManager
  • SELinux enabled
  • While NOT the proper soluton, I - out of pure desperation - tried to add a SNAT rule for outgoing HTTP requests, and that worked - so now I at least know that outgoing traffic on one of the alternate IPs is possible. What I tested: iptables -t nat -A POSTROUTING -p tcp --dport 80 -o enp3s0 -j SNAT --to-source 2.0.0.89 => HTTP request to our IP service returned 2.0.0.89 as the remote IP. – Saustrup Sep 11 at 15:55
0

You need to change your default route to the IP address you want to use. Your output shows 1.0.0.213 is the current default. Something like:

ip route replace default via 1.0.0.193 dev eth0 src 2.0.0.89
| improve this answer | |
  • I've tried to remove all other routes except the one you suggested, "default via 1.0.0.193 dev enp3s0 src 2.0.0.89 metric 100", but the detected outgoing address is still 1.0.0.213 - it's mindboggling. I've added Hetzner's documentation on this to the question, but their advice doesn't seem to work either. – Saustrup Sep 11 at 14:30
0

Apparently this happened because I enabled masquerading on the public zone, and forgotten all about it. The reason I did this, was to allow the Docker containers to reach the internet. Apparently this will cause all non-primary IPs to masquerade as the primary IP, even if they're listed on the interface.

Turning off masquerading and rebooting fixed the issue:

$ firewall-cmd --remove-masquerade --zone=public --permanent
$ reboot

This does leave me with a container connectivity issue, but that could probably be fixed with a more precise SNAT/masquerade rule.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.