Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am trying to troubleshoot communication between two servers on a Windows network where IPSEC is encrypting everything. I installed wireshark on the source server and captured traffic at the point that the communication is failing, but other than a few ARPs and DNS packets, everything else that is captured is an ESP (Encapsultating Security Payload) encrypted packet.

I would understand this if I were doing a man-in-the-middle capture, but I am on the source machine. Is there a way to specify that Wireshark capture farther up the stack (after the decryption is complete)? Source machine is W2K8R2 running as a Hyper-V VM if it matters.

share|improve this question
up vote 2 down vote accepted

If you want to inspect and analyse ESP traffic directly your version of Wireshark needs to be linked with libcrypt. More details here.

share|improve this answer
Thanks Helvick- That makes sense. Is that doable on the Windows version of Wireshark? (everything on that page looked like it was for *nix) – Sean Earp Feb 18 '10 at 5:06

To answer my own question (or at least to mention my solution), Netmon is able to capture and parse the same traffic with no problems. I saved the Netmon capture and opened it in Wireshark, and everything still shows up as ESP packets. Apparently Wireshark doesn't like unencrypting the packets. Maybe Netmon uses the local key to do so? In any case, the answer was to use Netmon. It is not nearly as good for analyzing traffic, but it does open ESP packets if you capture them from an endpoint.

share|improve this answer

You probably just need to tell Wireshark to capture on the virtual interface provided by the IPSec VPN service, rather than on the actual interface. Go to capture->interfaces or to capture->options and select the interface from the dropdown.

share|improve this answer

In Wireshark, go to Edit/Preferences and expand the Protocol list. Find ESP in the list and enter your key information.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.