Setup
2x Cisco ASA 5520's
ASA 8.3(1), ASDM 6.3(1)
IPSec Crypto Settings:
ESP-3DES-SHA, bidirectional, perfect forwarding secrecy group 1, 8 HR lifetime or 4,608,000 Kilobytes, NAT-T enabled
Network A, Chicago 10.110.1.0/24 100 MBPS Cogent connection
Network B, Denver 10.110.2.0/24 Into datacenter fiber network at 60 MBPS
Issue
Any traffic between Chicago and Devner (IPSec, HTTP, FTP, etc) hits a wall at 7 MBPS Denver -> Chicago and 1.5 MBPS Chicago -> Denver.
Tests
- Microsoft updates download at 60+MBPS from either network so I know that the WAN connection at either end is fast.
- Rebooted both firewalls just for good measure.
- Transfers from the Chicago ASA to another Chicago ASA ran at the expected 45-50 MBPS. So it doesn't look like my ASA are going to be rate limiting or IPSec is having trouble with the through output. CPU sits at 3-4% during the day and 10% during the 7/1.5 MBPS transfers.
- Traceroute between Chicago and Denver comes to 16 hops, no hop higher than 60ms
- Have transferred files over the IPSec tunnel in Windows, outside the IPSec tunnel as HTTP or FTP traffic and still see the 7/1.5 MBPS limits.
- Have connected a desktop to our Cogent switch in Chicago, assigned a WAN IP, transferred HTTP and FTP traffic and still see a 7/1.5 MBPS limit.
- Looked at TCP Window size on transfers while I was seeing a limit between Chicago and Denver, and while just downloading a file. Both were at 65,535.
- Asked Datacenter to look at the line to see if there were any lost packets, CRC errors, etc. They saw nothing.
- 24 hour ping shows no packet loss and average response time of 85ms.
- Double-checked Cisco ASA setting to make sure there was no "police output" commands issued. Our ASA is pretty vanilla as we have some NATs, ACL for access, and a IPSec VPN back home to Chicago.
Questions
- I can't figure out what OSI layer to deal with when troubleshooting.
- Do I engage the ISPs and tell them my issues? Who should I point the finger at?