Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

By inspecting the payload of ICMP time-exceeded packets, I noticed that sometimes it is the last but one router (when ttl=2 in the returned packet) or even a previous one(up to 5 hops before, ttl=5) that drops the packet and generates an ICMP message.

How so? Any reason behind this?

How do you set this in a CISCO router?

Edit:

please note that ALL these packets are ICMP type 11 code 0, which means:

type = time-exceeded, code = ttl-zero-during-transit

Edit2: Here are two examples of such ICMP packets.

###[ IP ]###
  version   = 4L
  ihl       = 5L
  tos       = 0x0
  len       = 168
  id        = 9969
  flags     = 
  frag      = 0L
  ttl       = 243
  proto     = icmp
  chksum    = 0x19ea
  src       = 193.51.189.25
  dst       = 134.59.129.241
  \options   \
###[ ICMP ]###
     type      = time-exceeded
     code      = ttl-zero-during-transit
     chksum    = 0xbf6e
     unused    = 0
###[ IP in ICMP ]###
        version   = 4L
        ihl       = 5L
        tos       = 0x0
        len       = 52
        id        = 57161
        flags     = DF
        frag      = 0L
        ttl       = 2
        proto     = tcp
        chksum    = 0xcf32
        src       = 134.59.129.241
        dst       = 173.194.20.89
        \options   \
###[ TCP in ICMP ]###
           sport     = 43843
           dport     = http
           seq       = 3927922380L
           ack       = 3188073609L
           dataofs   = 8L
           reserved  = 0L
           flags     = A
           window    = 14165
           chksum    = 0x51f9
           urgptr    = 0
           options   = [('NOP', None), ('NOP', None), ('Timestamp', (5088093, 1579045454))]
###[ Padding ]###
              load      = '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00\x9d\xeb\x00\x08\x01\x01\x00\nA\x01'





    ###[ IP ]###
  version   = 4L
  ihl       = 5L
  tos       = 0x0
  len       = 168
  id        = 37758
  flags     = 
  frag      = 0L
  ttl       = 246
  proto     = icmp
  chksum    = 0xaa73
  src       = 193.51.189.2
  dst       = 134.59.129.241
  \options   \
###[ ICMP ]###
     type      = time-exceeded
     code      = ttl-zero-during-transit
     chksum    = 0x2e1c
     unused    = 4
###[ IP in ICMP ]###
        version   = 4L
        ihl       = 5L
        tos       = 0x0
        len       = 60
        id        = 53079
        flags     = DF
        frag      = 0L
        ttl       = 5
        proto     = tcp
        chksum    = 0x6d73
        src       = 134.59.129.241
        dst       = 74.125.230.71
        \options   \
###[ TCP in ICMP ]###
           sport     = 45799
           dport     = http
           seq       = 2382327024L
           ack       = 0
           dataofs   = 10L
           reserved  = 0L
           flags     = S
           window    = 14600
           chksum    = 0x83ed
           urgptr    = 0
           options   = [('MSS', 1460), ('SAckOK', ''), ('Timestamp', (5088167, 0)), ('NOP', None), ('WScale', 4)]
###[ Padding ]###
              load      = '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 \x00X\xf6\x00\x08\x01\x01\x04\x01\x81\xff'
share|improve this question
    
To clarify are you saying the you are pinging from R1 to R4 where the topology is R1->-R2->-R3->-R4 and R3 is generating the time-exceeded message, or R2 is generating the message? –  jwbensley Mar 19 '13 at 11:45
    
Also, would the last hop be on the same physical router, just another loopback interface nearer to the sender? –  Alexander Janssen Mar 19 '13 at 11:55
2  
No no, I'm not talking about echo-request or echo-reply packets. Only time-exceeded. I found ICMP time exceeded packets whose payload contained an IP header with TTL=2 instead of TTL=1, which means that it wasn't when TTL=1 that the packet was dropped and an ICMP message was generated, but one hop before. Anyway, I don't have control over those routers, so I don't know if I'm going through multiple interfaces over the same physical router. Is there any way to check this? –  Ricky Robinson Mar 19 '13 at 13:02
    
Since you mention you are 'forging' low TTL values in your packets in another comment, can you provide the returned packets you are asking about? Or you can check nanog.org/meetings/nanog45/abstracts.php?pt=MTE4NSZuYW5vZzQ1 for how to interpret a traceroute. –  cpt_fink Mar 27 '13 at 1:12
1  
Updated my answer... you're running into MPLS TTL expiration. I was able to capture a similar packet (TTL of 2 in a TTL-expired return). –  cpt_fink Mar 28 '13 at 3:33

5 Answers 5

up vote 6 down vote accepted
+50

http://packetlife.net/blog/2008/dec/22/disabling-mpls-ttl-propagation/

http://www.ciscopress.com/articles/article.asp?p=680824&seqNum=4

Your packets are MPLS-encapsulated when the outer label's TTL is decremented to 0, but the inner packet TTL is not updated, so the TTL-expired labeled packet is forwarded on and the internal IP packet (with an apparently valid TTL) is returned to you as expired by the final MPLS router.

============================

When a labeled packet TTL expires the packet is actually forwarded on until the end of the 'tunnel' it's on, since the router that decremented the TTL field to 0 might not have a valid route back to the original sender. So the MPLS label is edited to indicate TTL expiration and eventually the final tunnel router decapsulates the 'valid but label-expired' packet and sends it back with a TTL failure message.

Disclaimer: I read through TTL-relevant sections of several RFC's but nothing was definite on this handling, so I would say that this behavior could vary from vendor to vendor.

Evidence from a captured packet:

Internet Control Message Protocol
Type: 11 (Time-to-live exceeded)
Code: 0 (Time to live exceeded in transit)
Checksum: 0xf4df [correct]
Internet Protocol, Src: 192.168.1.x (192.168.1.x), Dst: 8.8.8.8 (8.8.8.8)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x80 (DSCP 0x20: Class Selector 4; ECN: 0x00)
    Total Length: 92
    Identification: 0x6b56 (27478)
    Flags: 0x00
    Fragment offset: 0
    Time to live: 2  <===== payload of packet entering MPLS tunnel
    Protocol: ICMP (1)
    Header checksum: 0x7abb [correct]
    Source: 192.168.1.x (192.168.1.x)
    Destination: 8.8.8.8 (8.8.8.8)
Internet Control Message Protocol
    Type: 8 (Echo (ping) request)
    Code: 0
    Checksum: 0xf78f [correct]
    Identifier: 0x0001
    Sequence number: 111 (0x006f)
    Sequence number (LE): 28416 (0x6f00)
    Data (64 bytes)
MPLS Extensions
    Version: 2
    Reserved: 0x000
    Checksum: 0x5581 [correct]
    MPLS Stack Entry
        Length: 0x0008
        Class: 1
        C-Type: 1
        Label: 1864, Exp: 4, S: 1, TTL: 1
            0000 0000 0111 0100 1000 .... = Label: 1864
            .... .... .... .... .... 100. = Experimental: 4
            .... .... .... .... .... ...1 = Stack bit: Set
            Time to live: 1    <========== MPLS TTL 
share|improve this answer
    
THanks a lot! Why are you saying that my packets are "mpls-encapsulated" when the IP ttl is 0? If I got it right, when my data packet enters an MPLS tunnel, its IP TTL is copied onto the newly created label TTL. If this label TTL expires inside the MPLS tunnel, then an ICMP packet is generated. This ICMP message, as soon as it gets out of the MPLS tunnel, shows in its payload the TTL that my data packet had before entering the MPLS tunnel. Is that correct? –  Ricky Robinson Mar 28 '13 at 15:27
    
Or are you saying that the ingress MPLS router doesn't drop an incoming packet with IP ttl=1, but it encapsulates it with a label and sends it through the MPLS tunnel? But in that case, if # no mpls ip propagate-ttl isn't active, that is in the default case, the IP TTL is copied onto the label TTL, so it expires anyway. When instead # no mpls ip propagate-ttl is on, egress router will see an IP packet with TTL=1 and generate an ICMP packet containing an IP header of ttl=1, not higher, –  Ricky Robinson Mar 28 '13 at 15:35
    
@RickyRobinson At any point, when a packet travels down an MPLS tunnel and drops out because the TTL reaches 0, weather in the middle or at the end, the ICMP generating router will perform an IP lookup on the source and generate a packet with a TTL that is high enough to reach that source. So if the packet gets dropped at the end of the tunnel it won't sent it with a TTL=1, because it wouldn't make back to the source. –  jwbensley Mar 28 '13 at 21:43
    
Yeah, of course. I was referring to the TTL inside the ICMP payload, that is to say the TTL of the packet that triggered the generation of that ICMP message. I'm not talking about the the TTL of the IP hearder that carries the whole ICMP message. –  Ricky Robinson Mar 29 '13 at 1:23
    
Editing my answer... this is too long for a comment. –  cpt_fink Mar 29 '13 at 3:28

Routers are supposed to decrement the time to live field by 1 for each second spent processing the packet, but in no case should it decrement by less than 1.

So if a router spends more than a second processing a packet, it should decrement the TTL by more than one. However, it's exceedingly rare that a router would spend more than a second processing a packet, unless it was terribly bogged down.

Barring router implementation bugs, that's about the only thing I can think of that would explain this.

share|improve this answer
    
No, I'm also checking the RTT for each (packet, corresponding ICMP message) pair that I have, and it's never greater than 0.01s, often much smaller. It looks like some routers, when they receive a packet with a low TTL value (actually as high as 5, I've noticed), don't let the packet through, but they drop it and they generate an ICMP time-exceeded packet. Otherwise, why would the payload of some packets contain an IP header with ttl=5? –  Ricky Robinson Mar 20 '13 at 13:31
    
Have you tried running debug ip icmp on the router that is dropping packets and checking the logs as it is happening? –  GerryEgan Mar 21 '13 at 19:28
    
I don't have control over that router. –  Ricky Robinson Mar 22 '13 at 9:41
    
In any case, I was dealing with an MPLS router (the ICMP message I reported ends with 12 Bytes representing ICMP extension for MPLS) –  Ricky Robinson Aug 14 '13 at 14:04

While I can't say exactly what it is going on here, the Time Exceeded packet is sent on one of two conditions: TTL is exceeded, or fragment reassembly time is exceeded. What is the code sent back (second bite of the payload). If it is 1, the reason for sending the packet is reassembly time exceeded. This should normally be set at 60 to 120 seconds, not at 0.01s. Is it allways the same router(s) sending these packets back? Can you post the complete packet you are getting back and? Can you post any info on the routers in questions? Make? Model?

share|improve this answer
    
The ICMP code is always 0, which means it's not a Reassembly Timeout. I don't have control over the router, so I don't know model or brand. It looks like this one router handles all the packets with a low ttl value and generates an ICMP packet on behalf of other routers, perhaps in the same network. –  Ricky Robinson Mar 26 '13 at 18:58
    
My previous question remains: Can you post the entire packet you are getting back? –  JelmerS Mar 27 '13 at 10:53
    
Yes, sorry. Now you can see two of those packets above. –  Ricky Robinson Mar 27 '13 at 15:18

While possible, a bug at this rudimentary level is unlikely ; plus considering the (fast) time it takes to process packets, I would rather direct the likeness of a problem towards some kind of loop or bouncing. In the trivial / usual case, the router decrements the TTL and directs the packet to the right interface based on its routing table.

There are some more complex cases where the TTL may depend upon the router implementation. From the top of my head, comes NAT masquerading where the packet may "reenter" the router after the address translation has been performed - this is typically the case when one wants to test a router WAN IP from the inside (and this doesn't work on all routers)

For instance,

source:
local  10.1.2.3/24 
destination:
public: 198.252.206.16/32 (an example..)

the address 198.252.206.16 being one's own address as seen from the Internet (the WAN side of the router) which is bound internally to 10.1.2.3 ie is actually the same host as the sender

The router receives the packet and realizes the WAN address is its own address, the packet "reenters" the WAN interface (implementation dependent) and is driven to the LAN address 10.1.2.3.

How is the TTL is treated in that case (which is not working on all routers) is implementation dependent.

share|improve this answer

The answer to "Why are you receiving time-exceeded packets" (your original question of "How so? Any reason behind this?") is easy to answer:

Look at the time exceeded packet you caputured, what is the code value inside. If it is 0, it was a TTL issue at the generating router, if it was 1 it was a fragmentation issue at the generating router.

The question "How do you set this in a CISCO router?" doesn't make an sense, set what? No out of the ordinary behaviour has been displayed according to what you have said.

"Why is the router having TTL or fragmentation issues?" is a good question here I believe. If the router (assuming its always the same one) is out of your control then we can't say for sure. But we can speculate a little bit. It could be an MTU mismatch between interfaces, it could be a buffering problem I suppose also. This is assuming its a fragmentation issues. If it's a TTL issue, it could be a misconfiguration of an MPLS LSP or an MPLS LSR which is giving a conflicting ICMP reading due to PHP/UHP etc (although unlikely).

When you are receiving these time exceeded messages, are the current UDP/TCP flows having issues, do any of them drop? The time exceeded message should contain part of the data packet which caused the generation of the ICMP packet, is the original data unit a jumbo frame or large TCP packet, does it have the DF bit set?

You haven't given much to go on in terms of the context in which the ICMP packets are being generated in the first place.

share|improve this answer
    
Thank you for your answer. The only thing is that I'm not dealing with ICMP type 11 code 1 packets. Sorry for not making it too clear in my initial post. These packets are always code 0, so no it's not a reassembly timeout problem. –  Ricky Robinson Mar 26 '13 at 18:59
    
As for how I generate ICMP packets, I just replay a packet trace where I forged the TTL field with a low value. I'm trying to measure round trip times to each hop to destination. –  Ricky Robinson Mar 26 '13 at 19:01
    
On a side note: the TTL value inside an ICMP payload is never 0, it's always 1. –  Ricky Robinson Mar 27 '13 at 15:10
1  
Given the above info in your comments I still maintain that no one can really tell you what's wrong here as you don't have access to the routers. It could be anything like MPLS (as I mentioned) or GRE tunnels causing issues, (un-)ECMP routing, or a problem with how the router's are handling the ICMP packets in their code (software bug), or they could be using a CoPP like system that is braking the ICMP flow. If you can't access or control them it will be very difficult to resolve this. I would like to know should you find an answer. I am going to think on this more, it is annoying me now :) –  jwbensley Mar 27 '13 at 17:27
    
Thanks a lot! I'll look into those things you mentioned. What baffles me is that out of three locations in which I tested this, two showed this behaviour, so it can't be just an unfortunate router configuration. At least two! :) –  Ricky Robinson Mar 27 '13 at 20:06

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.