Cisco ASA VPN: Where can I find information on what the detailed logs mean?

Question

I'm working on troubleshooting a Cisco ASA VPN connection and I'm after more information on what some of the log messages mean, specifically this one at the moment but a good general resource on how to read these logs would be ideal:

At log level 7 I can see

Group = 1.1.1.1, IP = 1.1.1.1, IKE AM Responder FSM error history (struct &0x3f76598)  <state>, <event>:  AM_DONE, EV_ERROR-->AM_WAIT_MSG3, EV_PROB_AUTH_FAIL-->AM_WAIT_MSG3, EV_TIMEOUT-->AM_WAIT_MSG3, NullEvent-->AM_SND_MSG2, EV_CRYPTO_ACTIVE-->AM_SND_MSG2, EV_SND_MSG-->AM_SND_MSG2, EV_START_TMR-->AM_SND_MSG2, EV_RESEND_MSG

Formatted for readability:

Group = 1.1.1.1, IP = 1.1.1.1, 
IKE AM Responder FSM error history (struct &0x3f76598)  
<state>, <event>:  
AM_DONE, 
EV_ERROR-->AM_WAIT_MSG3, 
EV_PROB_AUTH_FAIL-->AM_WAIT_MSG3, 
EV_TIMEOUT-->AM_WAIT_MSG3, 
NullEvent-->AM_SND_MSG2, 
EV_CRYPTO_ACTIVE-->AM_SND_MSG2, 
EV_SND_MSG-->AM_SND_MSG2, 
EV_START_TMR-->AM_SND_MSG2, 
EV_RESEND_MSG

Is this a history of the events that have taken place to establish and VPN, and is it meant to be read as EVENT->RESULT, bottom to top? Are there any good documents detailing how to interpret theses logs for troubleshooting connections? Almost everything I can find in my searches is of the level "post your config and we'll figure out what is wrong", which isn't helping when eveyone who looks at our configs says "it should work" :-)

Answer 1

It seems that you have something dropping packet between the 2 VPN Peer.

FSM means finite state machine.
So you got an error in the state machine of IKE AM. Why ? You just need to see the FSM state as described after the error message (from bottom to top as you correctly tell).

You sent AM_SND_MSG2 (EV_SND_MSG), nothing was comming so you send it again (the NullEvent) and finally the FSM timeout by waiting AM_WAIT_MSG3 from the remote peer (EV_TIMEOUT). So this trigger an authentification problem (EV_PROB_AUTH_FAIL), so an error (EV_ERROR) and finally the FSM reach its end (AM_DONE ).

So this mean start you started to communicate with the remote peer but never get any answer. You have to double check configuration at both end, and double check that all necessary port and protocole are open. (UDP port 500, ESP, HA, UDP port 4500, ... depending of your configuration).

You may also want to check the debug on the remote peer to see if AM_SND_MSG2 is received.

Unfortunately I don't think that Cisco publish their how their FSM are working and you have to understand the log yourself or ask the TAC or SF ;)