Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm trying to understand what this output from traceroute means. I changed the IP addresses for privacy but retained the public/private IP range distinction.

traceroute.db -e -n 10.1.1.9
traceroute to (10.1.1.9), 30 hops max, 60 byte packets
 1  10.0.0.1  0.596 ms  0.588 ms  0.577 ms
 2  10.0.0.2  1.032 ms  1.029 ms  1.084 ms
 3  10.0.0.3  3.360 ms  3.355 ms  3.338 ms
 4  23.0.0.4  3.974 ms  4.592 ms  4.584 ms
 5  23.0.0.5  13.442 ms  13.445 ms  13.434 ms
 6  45.0.0.6  13.195 ms  12.924 ms  12.913 ms
 7  67.0.0.7  52.088 ms  51.683 ms  52.040 ms
 8  10.1.1.8  46.878 ms  44.575 ms  44.815 ms
 9  10.1.1.9  45.932 ms  45.603 ms  45.593 ms

The first 10.0.* range is inside my organisation. The last 10.1.* range is another site of my organisation. The intermediate addresses belong to various ISPs. I expect that there is some kind of VPN between the two sites, but I don't know much about our network topology.

What I don't understand is how the route can go from a private address through public addresses back into private addresses. Searching led me to Public IPs on MPLS Traceroute, which gives a possible explanation: MPLS. Is MPLS the only possible or most likely explanation? Otherwise what does this tell me about our network infrastructure?

Bonus question for my edification: in this scenario, who is generating the ICMP TTL exceeded packets and if relevant mangling their source and destination addresses?

share|improve this question
    
I don't necessarily have an answer for you, but I get similar output when I traceroute do our remote office, which is connected to us through a T1 line. It goes from my network to the T1 gateway provided to us by the ISP inside our network, then to one of their routers with a private IP, then one of their public IP address, then to the internal IP on my remote side. –  Safado Jun 28 '11 at 18:53
    
@Gilles: Just because the addresses are routable doesn't make them public. The term should really be "routable" and "non-routable", meaning the address is routable on the internet or it is not routable on the internet. Just because it's routable doesn't actually mean "it's on the internet". Conversley, a non-routable address in the middle of your tracert is not on the internet, it's on a dedicated private circuit/network such as a T1, Frame Relay, or MPLS. I use routable addresses on my internal network, but none of them are "on the internet". –  joeqwerty Jun 28 '11 at 22:28
    
@joeqwerty They're public in the sense that I can ping them from outside my org (unlike, obviously, the 10.* addresses). –  Gilles Jun 28 '11 at 22:38
    
@Gilles: Gotcha. Thanks for the clarification. –  joeqwerty Jun 28 '11 at 22:42
    
What is the default gateway address for each of the internal systems traversed? –  John Gardeniers Jun 29 '11 at 0:22
show 1 more comment

1 Answer

There are servers at both ends which are doing Network Address Translation (NAT). As the address passes though these servers the header address on the data packet gets rewritten to that servers Internet address. The server keeps track of which connections belong to which internal host.

Traceroute displays data from inside an ICMP packet indicating whether or not the host was reached in a given number of hops. The NAT routers do not alter this data. As a result you see the address that each host received the packet on.

Normally the servername on the far end in this case has been routed using DNAT (Destination NAT) to a host on the private network.

It is likely that the address is being passed over a VPN tunnel between two sites. The VPN be encapsulating the source and final addresses inside the packets being sent between hops 3 and 7. The effect is the same, although the mechanism is different. The routers at hops 3 and 7 would know the addresses ranges supported by the remote routers, and route the packets accordingly. Leaving hop 7 the IP destination would be 67.0.0.7 with a public address belonging to hop 3. This is invisible due to the way route tracing works. Depending on the VPN protocol some hops after hop 7 may not be traceable.

In some cases you may see an ISP routing over private addresses to a public address. This will appear as one ore more private addresses between two public addresses. If the intermediate routers with public addresses belong to the same organization, it is possible they have routing rules allowing end to end communication without translation.

share|improve this answer
    
+1. The only exception I'd make to this answer is that we know that the connections between the local and remote sites are either via VPN or a dedicated, private circuit as it's not possible to tracert to an RFC1918 address through the internet. The fact that the OP can tracert from one private ip address to another at a remote site tells me there's a dedicated connection of some sort between the two. –  joeqwerty Jun 28 '11 at 19:27
    
I don't think 10.1.1.9 has a public address. So I don't understand what the packets between 2 and 8 look like — I expected that part to be encapsulated inside some VPN protocol and therefore transparent. What's the destination of the packet when it reaches 4, and on the recieving side how does 8 figure that it's coming from inside our org? –  Gilles Jun 28 '11 at 19:48
    
@Gilles: I hadn't noticed you where tracing to a private address. It is most probable that you do have a VPN protocol involved. I have updated my answer accordingly. I am not sure which tunnel protocol is being used. –  BillThor Jun 28 '11 at 22:50
    
@BillThor: It does seem curious to me as to why the routers at hops 4 through 7 show up at all if it's a VPN connection. I originally commented (and then deleted) that I thought it was a VPN connection, but now I'm not sure. I don't have a VPN to test with so I can't say if that's normal or not. Is that normal from your experience? –  joeqwerty Jun 28 '11 at 23:35
    
@Gilles: It is also curious to me as well. It appears the routing is keeping the TTL values on the outer protocol header. I am not sure which VPN protocols would do so. It does enable using traceroute to locate problems. I don't recall tracing routes over a VPN, and don't recall any problems which would have caused me to do so. –  BillThor Jun 29 '11 at 0:32
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.