5

I'm trying to understand what this output from traceroute means. I changed the IP addresses for privacy but retained the public/private IP range distinction.

traceroute.db -e -n 10.1.1.9
traceroute to (10.1.1.9), 30 hops max, 60 byte packets
 1  10.0.0.1  0.596 ms  0.588 ms  0.577 ms
 2  10.0.0.2  1.032 ms  1.029 ms  1.084 ms
 3  10.0.0.3  3.360 ms  3.355 ms  3.338 ms
 4  23.0.0.4  3.974 ms  4.592 ms  4.584 ms
 5  23.0.0.5  13.442 ms  13.445 ms  13.434 ms
 6  45.0.0.6  13.195 ms  12.924 ms  12.913 ms
 7  67.0.0.7  52.088 ms  51.683 ms  52.040 ms
 8  10.1.1.8  46.878 ms  44.575 ms  44.815 ms
 9  10.1.1.9  45.932 ms  45.603 ms  45.593 ms

The first 10.0.* range is inside my organisation. The last 10.1.* range is another site of my organisation. The intermediate addresses belong to various ISPs. I expect that there is some kind of VPN between the two sites, but I don't know much about our network topology.

What I don't understand is how the route can go from a private address through public addresses back into private addresses. Searching led me to Public IPs on MPLS Traceroute, which gives a possible explanation: MPLS. Is MPLS the only possible or most likely explanation? Otherwise what does this tell me about our network infrastructure?

Bonus question for my edification: in this scenario, who is generating the ICMP TTL exceeded packets and if relevant mangling their source and destination addresses?

5
  • I don't necessarily have an answer for you, but I get similar output when I traceroute do our remote office, which is connected to us through a T1 line. It goes from my network to the T1 gateway provided to us by the ISP inside our network, then to one of their routers with a private IP, then one of their public IP address, then to the internal IP on my remote side.
    – Safado
    Jun 28, 2011 at 18:53
  • @Gilles: Just because the addresses are routable doesn't make them public. The term should really be "routable" and "non-routable", meaning the address is routable on the internet or it is not routable on the internet. Just because it's routable doesn't actually mean "it's on the internet". Conversley, a non-routable address in the middle of your tracert is not on the internet, it's on a dedicated private circuit/network such as a T1, Frame Relay, or MPLS. I use routable addresses on my internal network, but none of them are "on the internet".
    – joeqwerty
    Jun 28, 2011 at 22:28
  • @joeqwerty They're public in the sense that I can ping them from outside my org (unlike, obviously, the 10.* addresses). Jun 28, 2011 at 22:38
  • What is the default gateway address for each of the internal systems traversed? Jun 29, 2011 at 0:22
  • @JohnGardeniers I only have shell access on .1. The rest is me trying to figure out what's going on and how to cope with various network problems. Jun 29, 2011 at 0:30

1 Answer 1

5

There are servers at both ends which are doing Network Address Translation (NAT). As the address passes though these servers the header address on the data packet gets rewritten to that servers Internet address. The server keeps track of which connections belong to which internal host.

Traceroute displays data from inside an ICMP packet indicating whether or not the host was reached in a given number of hops. The NAT routers do not alter this data. As a result you see the address that each host received the packet on.

Normally the servername on the far end in this case has been routed using DNAT (Destination NAT) to a host on the private network.

It is likely that the address is being passed over a VPN tunnel between two sites. The VPN be encapsulating the source and final addresses inside the packets being sent between hops 3 and 7. The effect is the same, although the mechanism is different. The routers at hops 3 and 7 would know the addresses ranges supported by the remote routers, and route the packets accordingly. Leaving hop 7 the IP destination would be 67.0.0.7 with a public address belonging to hop 3. This is invisible due to the way route tracing works. Depending on the VPN protocol some hops after hop 7 may not be traceable.

In some cases you may see an ISP routing over private addresses to a public address. This will appear as one ore more private addresses between two public addresses. If the intermediate routers with public addresses belong to the same organization, it is possible they have routing rules allowing end to end communication without translation.

6
  • +1. The only exception I'd make to this answer is that we know that the connections between the local and remote sites are either via VPN or a dedicated, private circuit as it's not possible to tracert to an RFC1918 address through the internet. The fact that the OP can tracert from one private ip address to another at a remote site tells me there's a dedicated connection of some sort between the two.
    – joeqwerty
    Jun 28, 2011 at 19:27
  • I don't think 10.1.1.9 has a public address. So I don't understand what the packets between 2 and 8 look like — I expected that part to be encapsulated inside some VPN protocol and therefore transparent. What's the destination of the packet when it reaches 4, and on the recieving side how does 8 figure that it's coming from inside our org? Jun 28, 2011 at 19:48
  • @Gilles: I hadn't noticed you where tracing to a private address. It is most probable that you do have a VPN protocol involved. I have updated my answer accordingly. I am not sure which tunnel protocol is being used.
    – BillThor
    Jun 28, 2011 at 22:50
  • @BillThor: It does seem curious to me as to why the routers at hops 4 through 7 show up at all if it's a VPN connection. I originally commented (and then deleted) that I thought it was a VPN connection, but now I'm not sure. I don't have a VPN to test with so I can't say if that's normal or not. Is that normal from your experience?
    – joeqwerty
    Jun 28, 2011 at 23:35
  • @Gilles: It is also curious to me as well. It appears the routing is keeping the TTL values on the outer protocol header. I am not sure which VPN protocols would do so. It does enable using traceroute to locate problems. I don't recall tracing routes over a VPN, and don't recall any problems which would have caused me to do so.
    – BillThor
    Jun 29, 2011 at 0:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .