Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have an issue with private network traffic not being masqueraded in very specific circumstances.

The network is a group of VMware guests using the 10.1.0.0/18 network.

The problematic host is 10.1.4.20 255.255.192.0 and the only gateway it is configured to use is 10.1.63.254. The gateway server 37.59.245.59 should be masquerading all outbound traffic and forwarding it through 37.59.245.62, but for some reason, 10.1.4.20 ends up occasionally having 37.59.245.62 in its routing cache.

ip -s route show cache 199.16.156.40
199.16.156.40 from 10.1.4.20 via 37.59.245.62 dev eth0
    cache  used 149 age 17sec ipid 0x9e49
199.16.156.40 via 37.59.245.62 dev eth0  src 10.1.4.20
    cache  used 119 age 11sec ipid 0x9e49

ip route flush cache 199.16.156.40

ping api.twitter.com
PING api.twitter.com (199.16.156.40) 56(84) bytes of data.
64 bytes from 199.16.156.40: icmp_req=1 ttl=247 time=93.4 ms

ip -s route show cache 199.16.156.40
199.16.156.40 from 10.1.4.20 via 10.1.63.254 dev eth0
    cache  age 3sec
199.16.156.40 via 10.1.63.254 dev eth0  src 10.1.4.20
    cache  used 2 age 2sec

The question is, why am I seeing a public IP address in my routing cache on a private network?

Network information for the app server (without lo) :

ip a

eth0      Link encap:Ethernet  HWaddr 00:50:56:a4:48:20
          inet addr:10.1.4.20  Bcast:10.1.63.255  Mask:255.255.192.0
          inet6 addr: fe80::250:56ff:fea4:4820/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1523222895 errors:0 dropped:407 overruns:0 frame:0
          TX packets:1444207934 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1524116772058 (1.5 TB)  TX bytes:565691877505 (565.6 GB)

Network information for the VPN gateway (without lo too) :

 eth0      Link encap:Ethernet  HWaddr 00:50:56:a4:56:e9
           inet addr:37.59.245.59  Bcast:37.59.245.63  Mask:255.255.255.192
           inet6 addr: fe80::250:56ff:fea4:56e9/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:7030472688 errors:0 dropped:1802 overruns:0 frame:0
           TX packets:6959026084 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:7777330931859 (7.7 TB)  TX bytes:7482143729162 (7.4 TB)

 eth0:0    Link encap:Ethernet  HWaddr 00:50:56:a4:56:e9
           inet addr:10.1.63.254  Bcast:10.1.63.255  Mask:255.255.192.0
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

 eth0:1    Link encap:Ethernet  HWaddr 00:50:56:a4:56:e9
           inet addr:10.1.127.254  Bcast:10.1.127.255  Mask:255.255.192.0
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

 tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
           inet addr:10.8.1.1  P-t-P:10.8.1.2  Mask:255.255.255.255
           UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
           RX packets:477047415 errors:0 dropped:0 overruns:0 frame:0
           TX packets:833650386 errors:0 dropped:101834 overruns:0 carrier:0
           collisions:0 txqueuelen:100
           RX bytes:89948688258 (89.9 GB)  TX bytes:1050533566879 (1.0 TB)

eth0 leads to the outside world, and tun0 to an openvpn network of VMs on which sits the app server.

ip r for the VPN gateway :

default via 37.59.245.62 dev eth0  metric 100
10.1.0.0/18 dev eth0  proto kernel  scope link  src 10.1.63.254
10.1.64.0/18 dev eth0  proto kernel  scope link  src 10.1.127.254
10.8.1.0/24 via 10.8.1.2 dev tun0
10.8.1.2 dev tun0  proto kernel  scope link  src 10.8.1.1
10.9.0.0/28 via 10.8.1.2 dev tun0
37.59.245.0/26 dev eth0  proto kernel  scope link  src 37.59.245.59

ip r on the app server :

default via 10.1.63.254 dev eth0  metric 100
10.1.0.0/18 dev eth0  proto kernel  scope link  src 10.1.4.20

Firewall rules:

Chain PREROUTING (policy ACCEPT 380M packets, 400G bytes) 
pkts bytes target prot opt in out source destination 

Chain INPUT (policy ACCEPT 127M packets, 9401M bytes) 
pkts bytes target prot opt in out source destination 

Chain OUTPUT (policy ACCEPT 1876K packets, 137M bytes) 
pkts bytes target prot opt in out source destination 

Chain POSTROUTING (policy ACCEPT 223M packets, 389G bytes) 
pkts bytes target prot opt in out source destination 

32M 1921M MASQUERADE all -- * eth0 10.1.0.0/17 0.0.0.0/0
share|improve this question
    
might be useful to add the --no-dns (or just -n) flag to your mtr. I suspect that one of the routes in the middle is your 37.59.245.62. –  Dan Pritts Mar 7 at 15:29
    
Looks like you have a lot of flapping routes. –  NickW Mar 7 at 15:29
    
@DanPritts : updated my answer –  greg0ire Mar 7 at 15:33
    
@NickW: how do you see that ? –  greg0ire Mar 7 at 15:39
    
Not you specifically, but the route used by your machine when you ping changes often, either the network admin is trying some failing form of load balancing, or the routes offered to get to twitter are changing quite often.. –  NickW Mar 7 at 15:45

3 Answers 3

Unfortunately, most of what you're seeing is due to routing issues between external routers, they obtain and update their routing info dynamically, to help route traffic around problematic areas, but when those routes get changed often (normally due to availability) it is called route flapping. That is getting reflected down to you, normally end users don't see any of this..

You could attempt to disable your route cache, as explained here (note the caveats, it's not something that seems to offer much on the upside), but I think you'd be better off just talking to the network admin(s) locally as it seems it's their routing which is really unstable.

I am of course going with the assumption that it isn't you responsible for network administration.

share|improve this answer
    
Does the 37.59.245.62 that show up in the route cache indicate that 37.59.245.62 is having problems ? I'm not the main responsible for network administration (I'm a developer), but I have the power to make changes on it. –  greg0ire Mar 7 at 16:22
    
No, basically it means that you are receiving dynamic updates to your routes, and those routes may be becoming unavailable (for many reasons), leaving you with a bad cached route –  NickW Mar 7 at 16:37
    
37.59.245.62 could be having issues, but I wouldn't point a finger at it immediately, it could just have issues with external routes changing frequently, what you really need is to be connected to a router which will keep you and your routing cache away from those updates. That or the network admins need to ensure that routes with higher priorities are not unstable. –  NickW Mar 7 at 16:41
    
Regarding your second comment, 37.59.245.62, is not available to the app server, never has been, never will be. How can this entry even exist ? That's what I don't understand... As you explained to me before, that would be because of ICMP messages –  greg0ire Mar 10 at 10:42
    
Well, it could be ICMP, but now I see a bit more information on your network, it looks like you are receiving dynamic routing updates directly on your machine, which is usually not a good idea if the network is as unstable as yours is (because of the routing cache). Proper routers have methods for invalidating caches when they receive updates, I've not used linux in those sort of situations, so I can't say if that is the case. I'd personally ask your network admins to keep those updates to themselves. –  NickW Mar 10 at 10:54

Have someone, or you, take a look at the router/L3 device at 10.1.4.20. It looks like it might be receiving bad routes from an upstream peer that are then being withdrawn and then re-advertised.

share|improve this answer
    
I have access to 10.1.4.20, it is the gateway of my VPN. What do you want me to do ? –  greg0ire Mar 7 at 15:56
    
Ooops 10.1.4.20 is not the gateway, it is the app server –  greg0ire Mar 7 at 15:59
    
Is this server multihomed? eg. are the there multiple NICs, and are they all plugged in? what about on the network upstream? I'm taking a closer look at the traces, and it looks like you're receiving routes from 10.1.63.254 and 37.59.245.62. Are those both on the same network? If so, you should have routes directing traffic. –  Couradical Mar 7 at 18:32
    
PS - what's your NW config look like? IP/NetMask/GW would help. –  Couradical Mar 7 at 18:35
    
I added ifconfig output for both the app server and the gateway. There is one NIC for the app server, butmany NICs for the gateway. How can I know if 10.1.63.254 and 37.59.245.62 are on the same network ? Sorry for the n00bery, I'm a developer, not a sysadmin. –  greg0ire Mar 7 at 20:48
up vote 0 down vote accepted

I asked this somewhere else, and it turns out the solution was to turn off ICMP redirects.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.