IPsec VPNs connect and route traffic but don't actually work

Question

Setup

All 23 branch offices have:

1. A mix of ADSL and VDSL Internet connections.
2. Cyberoam CR10iNGs.
3. A site-to-site IPsec VPN to the head office Cyberoam CR35iNG.
4. A site-to-site IPsec VPN to the data centre Cyberoam CR50iNG.

 

Changes

3 different branch offices had their Internet connections upgraded from ADSL to FTTC and, as such, their modems upgraded from ZyXEL P-660R-D1s to ZyXEL VMG1312-B10As (a BT- / OpenReach-approved VDSL modem as BT no longer provide VDSL modems) and the Cyberoam CR10iNGs' WAN interfaces were reconfigured to use PPPoE.

 

Symptoms

Ever since:

• All IPsec VPNs connect successfully.
• Attempting to access resources (RDWeb via HTTPS, RDSH server via RDP, etc) across the IPsec VPN loads a tiny amount of data (blue background and web page title for RDWeb, certificate for RDSH server, etc) but eventually times out.
• Attempting to access the same resources across the Internet (advantages of the AD DS domain being a subdomain of a web domain) loads perfectly.
• All IPsec VPN-bound traffic (IKE, ESP, HTTPS, RDP, etc) is routed and allowed both ends correctly.

Answer 1

We theorised two root causes:

1. The ISP was throttling VPN traffic. We contacted the ISP who clarified that the Internet connection is a business one and, as such, applies no throttling.
2. A VDSL modem bug. We found http://support.aa.net.uk/VMG1312-B10A:_Bugs#PPPoE_Session-ID_caching_bug_.28In_Bridge_mode.29 which says, among other things, that there is indeed a bug which affects PPPoE and IPsec traffic so we replaced the ZyXEL VMG1312-B10A with a DrayTek Vigor 130 (another BT- / OpenReach-approved VDSL modem) which resolved the problem.