My current mental model

  1. An application interacts with the operating system to produce an OSI model stack-based packet process - subsequent layers of encapsulation
  2. Somewhere along the line before the packet is encapsulated in an ethernet frame, but assuming after it has been given an IP header, the VPN software hijacks it and adds another encapsulation layer to it (AH header and encrypts payload) for VPN tunnel routing
  3. new packet with (probably) ESP and AH headers and an encrypted payload is passed to the network interface driver which gives it an ethernet frame and transmits on the wire

What I don't understand is actually how the VPN software hijacks the interprocess communication pipes like this - how it knows which applications to hijack, and how it can do this selectively without modifying all network communication from the host.

Please explain how a VPN process will intercept IPC destined for the network interface driver

  • If someone could go so far as to explain which kernel operations govern this type of process I would also appreciate it - I hadn't found this information anywhere else. – gal Apr 16 '13 at 14:36

VPN software usually works by adding a 'virtual' network interface to your system.

It uses plain old routing to direct traffic via the tunnel interface, (beware, very generalized) save for the endpoint host which it will communicate with the regular way.

  • Oh, so it sets the default interface to the virtual interface and redirects traffic from the virtual interface to the real interface, and then uses the virtual interface to apply IPSec? Okay that works, thank you =) – gal Apr 16 '13 at 15:05
  • A bit oversimplified. Routing tables play a rather key component in whether packets are routed over the VPN or not. – Magellan Apr 16 '13 at 15:09
  • It doesn't necessarily set the default interface to the virtual interface. The main thing is the Routing table. In some implementations, yes, the route will be set to the virtual interface, this is in use to keep most of the traffic going through the tunnel. Otherwise it will just add routes to the internal network with the virtual interface set for those routes. – Snowburnt Apr 16 '13 at 15:27

It doesn't "hijack" anything.

When your application needs to reach a system that is behind the other end of a VPN, it will send the packets in the tunnel.

If you are using a client to connect to a VPN then your PC is doing the encrypting part (and decrypting when receiving packets). It will encrypt and encapsulate the packets at the creation and send it to the other peer in the tunnel.

That peer will then decrypt the packets.

If you are going in a B2B tunnel, then the peer on your side of the tunnel is encrypting/decrypting the packets.

  • Right, but when "application X" creates a packet and passes it to the network interface card driver, the driver transmits it - if VPN software is going to encapsulate and encrypt this packet, it needs to be able to modify the communications channel to support the VPN configuration - EG: firefox doesn't just start encrypting packets for a VPN without some additional plug-in, whereas there are system-wide VPN software clients that provide services to all the applications that would use network services on a given machine. – gal Apr 16 '13 at 15:04
  • @Andrew I'd suggest doing a little research on how routing works. Essentially, you're asking how VPNs route packets which cannot be properly explained without knowledge of routing. – Magellan Apr 16 '13 at 15:10
  • 1
    @Andrew - applications don't pass packets to network drivers. Applications tell the OS, "Hey I have this data, please ensure it gets sent out properly.", then the OS takes the data, looks at its own routing table, and does what it needs to do to send the packet. The application typically has zero knowledge of what happens to the data after it hands it off to the OS. If the default route points through the VPN tunnel, then everything gets encrypted. If only a subnet or two point through the tunnel, then only that traffic will go through the VPN. – EEAA Apr 16 '13 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.