5

I have a CentOS server, configured with 4 consecutive IPs:
eth0 5.x.x.251
eth0:0 5.x.x.252
eth0:1 5.x.x.253
eth0:2 5.x.x.254

The problem is that all traffic goes out to the internet with eth0:0 (5.x.x.252) as the source IP, instead of eth0.
# curl ifconfig.me
5.x.x.252

How can I fix this, so that all traffic goes out via eth0, ie my main IP?

PS: My server is VPS running on a Xen dom0, the latter being configured in routed mode networking.

Thanks in advance!

Server configuration

# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:x:x:x:x:AE  
          inet addr:5.x.x.251  Bcast:5.x.x.255  Mask:255.255.255.255
          inet6 addr: fe80::x:x:x:x/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:14675569 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9463227 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:4122016502 (3.8 GiB)  TX bytes:25959110751 (24.1 GiB)
          Interrupt:23 

eth0:0    Link encap:Ethernet  HWaddr 00:x:x:x:x:AE   
          inet addr:5.x.x.252  Bcast:5.x.x.255  Mask:255.255.255.224
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:23 

eth0:1    Link encap:Ethernet  HWaddr 00:x:x:x:x:AE  
          inet addr:5.x.x.253  Bcast:5.x.x.255  Mask:255.255.255.224
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:23 

eth0:2    Link encap:Ethernet  HWaddr 00:x:x:x:x:AE  
          inet addr:5.x.x.254  Bcast:5.x.x.255  Mask:255.255.255.224
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:23  

# cat /etc/hosts
127.0.0.1       localhost.localdomain localhost
5.x.x.251       [fqdn] [hostname]

# cat ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
IPADDR=5.x.x.251
NETMASK=255.255.255.224
SCOPE="peer 5.x.y.82"

# cat ifcfg-eth0:0
DEVICE=eth0:0
BOOTPROTO=static
ONBOOT=yes
IPADDR=5.x.x.252
NETMASK=255.255.255.224

# cat route-eth0 
ADDRESS0=0.0.0.0
NETMASK0=0.0.0.0
GATEWAY0=5.x.y.82

# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
5.x.y.82        0.0.0.0         255.255.255.255 UH        0 0          0 eth0
5.x.x.224       0.0.0.0         255.255.255.224 U         0 0          0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
0.0.0.0         5.x.y.82        0.0.0.0         UG        0 0          0 eth0

Update 29 August 2012
When I run /etc/init.d/network restart I get an RTNETLINK answers: File exists error, which leads me to believe that there's another route to another network using the default gateway, as seen here.

# /etc/init.d/network restart
Shutting down interface eth0:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0:  RTNETLINK answers: File exists
                                                           [  OK  ]

The sure thing is that when I first delete ip route del default via 5.x.y.82 dev eth0 and then add it again with the .251 IP, everything works as expected.
Is there any way to add a route del command either to the ifcfg-eth0 or the route-eth0 files? I can see that this can be done with pre-up ip addr del ... on Debian (more info), but what about CentOS?

4
  • Try ip ro li and see what is your src Aug 28, 2012 at 22:45
  • Here's the output of ip ro li: pastebin.com/A1KRiWTz I think the last line is what I'm looking for... how can I change src 5.x.x.252 to scr 5.x.x.251? Aug 28, 2012 at 22:59
  • Are you using an official CentOS kernel? Try "uname -a". I've seen similarly weird behavior (e.g., arp flux) with OpenVZ kernels.
    – joechip
    Sep 7, 2012 at 5:35
  • I'm giving the correct answer & bounty to Zoredache who was the one to find the source of the problem in the first place. With his help I managed to at least fix it on the live system, and upon restart (via /etc/rc.local). Only thing left to fix is when I run /etc/init.d/network restart but ok I can live without it :) Sep 7, 2012 at 9:19

7 Answers 7

6
+100

From this document it appears that at least on CentOS 5 and I suspect above you can specify your routes by simply giving the correct iproute2 command arguments. (see the section on the IP Command Arguments Format)

So instead of writing something like:

# route-eth0 
ADDRESS0=0.0.0.0
NETMASK0=0.0.0.0
GATEWAY0=5.x.y.82

You could just have a file like this:

default via 5.x.y.82 dev eth0  src 5.x.x.251

I don't have a Redhat/Redhat derived box handy to test though.

7
  • I just tried it, unfortunately even when defining the .251 IP and restarting networking, ip ro li still reports the .252 as source. Aug 28, 2012 at 23:19
  • 1
    How annoying. If you remove the default route and add it back manually do things work properly? ip route del default via 5.x.y.82 dev eth0; ip route add default via 5.x.y.82 dev eth0 src 5.x.x.251?
    – Zoredache
    Aug 28, 2012 at 23:22
  • Let me login via console from the dom0 to try this out and i'll get back to you in a second. Aug 28, 2012 at 23:23
  • Yay! deleting the ip default route and adding it back works, BUT if I restart the VM or do a /etc/init.d/network restart I'm back to .252 :( I guess could add the above command to /etc/rc.local to cover the VM restart part... Aug 28, 2012 at 23:30
  • Hrm, well it sure seems from the documentation that the route-interface file should support any valid iproute command. You may need to either go with the rc.local fix, or hope someone else can give you an answer.
    – Zoredache
    Aug 29, 2012 at 0:37
2

If the aliased IP addresses are not to be used as source addresses to non-local destinations, they should not be in the same subnet as the target of your default route. So change their netmasks to 255.255.255.255 and remove their broadcast addresses.

2

First clear your 0.0.0.0 default gateway

route del -net default

Then declare default eth0 as default gateway device

route add -net default gw 5.x.y.82 dev eth0

It should work. Verify with

ip ro li

To save that as a rule try this messy thing:

Open /etc/sysconfig/network-scripts/network-functions, locate function add_default_route () and in that function, after the call to find_gateway_dev add a line with GATEWAYDEV="eth0" .

What happens is that the network-functions script decides which device gets to be the default gateway. It examines the route to the gateway IP with ip get route to GATEWAY and, using sed, matches the a device. It seems that eth0:0 gets to be the one each time, so by hardcoding GATEWAYDEV="eth0" you make sure that eth0 is always selected as the gateway device.

In older versions of RH you could just edit /etc/sysconfig/static-routes and add default via 5.x.y.82 dev eth0

As a last resort /etc/rc.local is always your friend!

8
  • Kalispera :) Zoredache already proposed this as a solution and it partially works. What I mean is that it will work fine until I either reboot the VM or run /etc/init.d/network restart. Then I'm back to the .252 IP Aug 28, 2012 at 23:57
  • On older RH versions you could add at /etc/syconfig/static-routes a default gw 5.x.y.82 dev eth0 On newer versions edit /etc/sysconfig/network-scripts/ifcfg-eth0 and /etc/sysconfig/network-scripts/ifcfg-eth0:0, remove from eth0:0 the DEFROUTE=yes and add it to eth0, I am also editing this on my reply above. Aug 29, 2012 at 0:24
  • I tried the DEFROUTE=yes thing on ifcfg-eth0 (also tried adding DEFROUTE=no to the other ifcfg-eth0:? files, but it didn't work. PS: I have updated my question with some extra information Aug 29, 2012 at 21:07
  • What I am about to suggest is a bit messy but does the trick. Open /etc/sysconfig/network-scripts/network-functions, locate function add_default_route () and in that function, after the call to find_gateway_dev add a line with GATEWAYDEV="eth0" Placing that in my answer. Aug 29, 2012 at 21:15
  • Unfortunatelly this didn't do the trick either... :( Here's a screenshot. I got the RTNETLINK answers: File exists error again. Until now the only thing that worked is manually deleting the default route and adding it again. This File exists error occurs because of two default routes. Does my route -n help? Aug 29, 2012 at 21:33
2

Sounds like you might be a victim of a setting in newer kernels. Enabling alias promotion will get past that.

Set this in your /etc/sysctl.conf

net.ipv4.conf.all.promote_secondaries = 1

Incidentally, do you have any funky SNAT/MASQUERADE iptables rules that are altering the outbound source IP?

1

Since they are all on the same subnet the kernel will pick one to use.. You can verify this by running

 ip addr list dev eth0

You have to do some iptables magic to get it to work correctly. See the following

When IP aliasing how does the OS determine which IP address will be used as source for outbound TCP/IP connections?

1
  • Here's the output of ip addr list dev eth0: pastebin.com/mFv4XCb9 I understand that it can be fixed with iptables, but my main concern first of all is WHY this is happening? I just noticed that the eth0 subnet mask from ifconfig compared to ifcfg-eth0 is different... Aug 28, 2012 at 22:56
0

route and ifconfig use old, deprecated, network interfaces in kernel that don't know about policy based routing. You should be using the ip command, especially for routing-related information.

That being said, ip ro li or the full command ip route list table main, outputs the actual routing table used by the kernel. ip rule list will show you all the routing tables in use (there should at least be local, main and default).

As you've posted (from http://pastebin.com/A1KRiWTz):

# ip ro li
5.x.y.82 dev eth0  proto kernel  scope link  src 5.x.x.251 
5.x.x.224/27 dev eth0  proto kernel  scope link  src 5.x.x.252 
169.254.0.0/16 dev eth0  scope link  metric 1002 
default via 5.x.y.82 dev eth0  src 5.x.x.252

you need to remove the default route and recreate it with correct source address:

ip route del default 
ip route add default via 2.x.y.82 dev eth0 src 5.x.x.251 scope global

I don't know how to do it the RedHat way though (where to put it in RedHat-specific config files), using /etc/rc.local should work, albeit it's somewhat hacky solution. You could try using system-config-network

-3

You need to write a PREROUTING rule for inbound and outbound traffic in iptable, you need to say any tcp or UDP traffic or port 80 or 8080 (for default internet only) should go to desired eth.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .