4

Investigating a slow VPN connection (Cisco ASA IPSec) to a remote office, I noticed on our firewall a lot of access rule matches:

Denied ICMP type=3, code=4 from *ip_address* on interface outside

I noticed that a traceroute to the remote site included the same IP address, somewhere between our ISP and the ISP the remote site uses.

I'm also seeing a message immediately after before saying

No matching connection for ICMP error mesage: icmp src outside *ip_address* dst identity:*firewall_outside_ip_address* (type 3, code 4) on outside interface.  Original IP payload: protocol 50 src  *firewall_outside_ip_address* dst *remote_site_ip_address*

Cisco suggest that this may be symptoms of an attack, but I don't think so.

Protocol 50 is ESP, which is part of IPSec. The remote site is connected to HQ via IPSec VPN using Cisco ASA 5505 at the remote end and ASA 5510 at HQ.

ICMP type=3, code=4 means Fragmentation Needed and Don't Fragment was Set.

Setting Don't Fragment is normal for IPSec ESP packets.

I think what is happening is that packets are leaving our ASA 5510 with the default MTU of 1500. When it hits the router with ip_address that router is unable to pass the traffic to the next hop that uses a smaller MTU, thus requiring fragmentation. The router is sending an ICMP packet back as DF is set, but our Firewall is blocking this, not because of an access rule, but because for some reason our ASA 5510 wasn't expecting this ICMP message.

I am trying to figure out whether the problem is with the configuration on our HQ ASA 5510 (although we have another 36 sites all working fine), the remote ASA 5505 (which is configured uniformly with our other remote ASA 5505s) or something in between the two.

What should I do next?

Update As requested here are the ICMP lines from the HQ ASA 5510:

icmp unreachable rate-limit 1 burst-size 1
icmp permit any echo-reply outside
icmp permit any time-exceeded outside
icmp permit any unreachable outside
Improve this question
2
  • 2
    IMHO, denying any kind of ICMP packet is just security through obfuscation.. There might have been a point when a ICMP flood could terminate your 28k modem connection, but not in these days..
    – pauska
    Apr 19, 2012 at 18:19
  • Absolutely - I am fed up of having to try and convince people of this who were at my level during the "ping of death" days and think ICMP = evil!
    – dunxd
    Apr 20, 2012 at 9:16

3 Answers 3

Reset to default
3

Try setting crypto ipsec df-bit clear-df outside. This won't fix the direct issue here, but may work around it.

As far as the immediate issue - it seems like the ASA isn't realizing that the ICMP packet needs to be used as Path MTU discovery for its tunnel. Check if there's anything in the PMTUD counters displayed by show crypto ipsec sa?

Improve this answer
8
  • I'm not seeing PMTUD in the output of show crypto ipsec sa peer *remote_ip_address*. I am seeing #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0` and path mtu 1500, ipsec overhead 74, media mtu 1500. MTU is set at 1500 (default setting) on ASAs at both ends.
    – dunxd
    Apr 20, 2012 at 9:39
  • Before I set crypto ipsec df-bit clear-df outside can you tell me whether this is going to have a detrimental affect on the majority of IPSec tunnels that aren't having a problem? As far as I can tell this is a global setting.
    – dunxd
    Apr 20, 2012 at 9:42
  • Yeah, that's the counter you're looking for - 0 received PMTUs means that no discovery is occurring. That configuration only cause problems for other tunnels if they're successfully processing PMTU packets. Check the counters on the existing tunnels.
    – Shane Madden
    Apr 20, 2012 at 15:11
  • I am seeing the same PMTUs sent/received ais 0 for all SAs. If I run sh crypto ipsec stats I can see PMTUs sent: 149154 and PMTUs rcvd: 297. I am seeing some pre-fragmentation but I think this is only happening when packets arrive that are larger than the default MTU of 1500. I guess that means I am safe to switch on the clear-df bit, but have I now discovered an additional issue of PMTU not working correctly on any of my tunnels? Is that something I can resolve, or am I having bad luck with the paths my tunnels are taking?
    – dunxd
    Apr 23, 2012 at 12:44
  • Ok - have applied crypto ipsec df-bit clear-df outside and the messages about that IP address stopped immediately. Happily, all the other tunnels remain up, and appear to be working fine too. So my only remaining concern is that I am now running a sub-optimal configuration, and there may be a better way to resolve the issue with PMTUs not working correctly. Am I now more or less robust than I was with df-bit copied?
    – dunxd
    Apr 23, 2012 at 13:10
2

ICMP type 3 code 4 messages are "fragmentation needed but don't fragment set". This means your device sent a packet larger than the MTU of the device sending the ICMP message to you. Normally, the packet could be fragmented, but the DF bit was set. Since you're denying the inbound ICMP message, the ASA doesn't get notified that its packet wasn't delivered. Dropping these ICMP messages is generally bad for performance because it essentially results in packet loss.

Cisco's ASA configuration guide recommends always permitting ICMP type 3 messages, and it specifically mentions that problems can arise with IPsec if these messages are blocked. You can configure the ASA reporting this error to allow them with the following command:

icmp permit any unreachable outside

This only affects ICMP unreachables destined for the ASA itself. If you also need to permit them through the ASA to internal hosts, you'll need to do so with an access-list on your outside interface.

Improve this answer
6
  • Annoyingly, that line is in the ASA 5510 config, and the destination is to the ASA 5510. So I think the denial is explained by the second log entry above. The question is - what should I do next? +1 for supplying the rule the absence of which would normally cause this.
    – dunxd
    Apr 19, 2012 at 15:24
  • Could you update your question with a sanitized version of the icmp statements in the ASA reporting these errors?
    – James Sneeringer
    Apr 19, 2012 at 17:08
  • Done - see question.
    – dunxd
    Apr 20, 2012 at 9:26
  • What version is the 5510 running? Cisco bug id CSCsk68658 matches what you're seeing, but it's pretty old, so I don't know if it's what you're actually running into.
    – James Sneeringer
    Apr 20, 2012 at 14:20
  • Running ASA 8.3(2) so not the latest (8.4), but that bug was apparently resolved with 8.0. So this probably isn't the result of a bug?
    – dunxd
    Apr 23, 2012 at 12:26
1

I had some input from a Cisco engineer who said I had the following options:

  • Setting clear-df as suggested by Shane Madden
  • Setting the MTU on our servers to 1300, so the packets they send are less likely to need fragmentation in the first place
  • Setting the Maximum Segment Size for TCP on the ASA using sysopt connection tcp-mss 1300

I have applied the tcp-mss change, and removed the clear-df command on our HQ ASA and the site with the problems is able to work ok.

This may be a better solution than clearing the df-bit, as that will lead to fragmentation which is not desirable. It is equivalent to setting the MTU on the ASA to 1340 (1300 + 20 bytes for the IP header and 20 bytes for the TCP header) but only affects TCP traffic. It also allows PMTUD to work, which isn't the case when clearing the Don't Fragment bit.

Cisco discuss this option in detail in Resolve IP Fragmentation, MTU, MSS, and PMTUD Issues with GRE and IPSEC.

Improve this answer
5
  • Related to Cisco's "suggestions" and why they a bad idea (which you probably already knew): What exactly is the MTU/MRU issue, what it is caused by and how to fix it?
    – Chris S
    Apr 26, 2012 at 12:38
  • 1
    I should add that the engineer was just outlining the different things I could do myself to mitigate against the problem of broken PMTUD. He was saying that if I can't do anything about fragmentation on route, that either of last two options would result in the packets not needing to be fragmented after they had been encrypted. The first option would result in large packets getting fragmented on route, which is not good for encrypted packets - see the article linked above.
    – dunxd
    Apr 26, 2012 at 12:53
  • @ChrisS have read your very good answer to that other question. It doesn't look like anything on route was actually doing anything wrong according to the RFCs, but for some reason yet to be established, my ASA didn't like the ICMP Destination Unreachable packets it was getting - but it certainly wasn't caused by the ASA config specifying not to allow ICMP packets. Any input on what other diagnostics or actions I might take would be very gratefully received.
    – dunxd
    Apr 26, 2012 at 13:20
  • @dunxd Regarding the root cause question - what device is throwing the fragmentation needed packets at you, anyway? Is it an internet router in between you and that remote site, or something more odd?
    – Shane Madden
    Apr 26, 2012 at 15:15
  • Internet router in between me and remote site. It's not a router owned by either of the ISPs from which we buy services. It is intermediate (I'm assuming a backbone provider).
    – dunxd
    Apr 27, 2012 at 10:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .