2

Setup

2x Cisco ASA 5520's

ASA 8.3(1), ASDM 6.3(1)

IPSec Crypto Settings:

ESP-3DES-SHA, bidirectional, perfect forwarding secrecy group 1, 8 HR lifetime or 4,608,000 Kilobytes, NAT-T enabled

Network A, Chicago 10.110.1.0/24 100 MBPS Cogent connection

Network B, Denver 10.110.2.0/24 Into datacenter fiber network at 60 MBPS

Issue

Any traffic between Chicago and Devner (IPSec, HTTP, FTP, etc) hits a wall at 7 MBPS Denver -> Chicago and 1.5 MBPS Chicago -> Denver.

Tests

  1. Microsoft updates download at 60+MBPS from either network so I know that the WAN connection at either end is fast.
  2. Rebooted both firewalls just for good measure.
  3. Transfers from the Chicago ASA to another Chicago ASA ran at the expected 45-50 MBPS. So it doesn't look like my ASA are going to be rate limiting or IPSec is having trouble with the through output. CPU sits at 3-4% during the day and 10% during the 7/1.5 MBPS transfers.
  4. Traceroute between Chicago and Denver comes to 16 hops, no hop higher than 60ms
  5. Have transferred files over the IPSec tunnel in Windows, outside the IPSec tunnel as HTTP or FTP traffic and still see the 7/1.5 MBPS limits.
  6. Have connected a desktop to our Cogent switch in Chicago, assigned a WAN IP, transferred HTTP and FTP traffic and still see a 7/1.5 MBPS limit.
  7. Looked at TCP Window size on transfers while I was seeing a limit between Chicago and Denver, and while just downloading a file. Both were at 65,535.
  8. Asked Datacenter to look at the line to see if there were any lost packets, CRC errors, etc. They saw nothing.
  9. 24 hour ping shows no packet loss and average response time of 85ms.
  10. Double-checked Cisco ASA setting to make sure there was no "police output" commands issued. Our ASA is pretty vanilla as we have some NATs, ACL for access, and a IPSec VPN back home to Chicago.

Questions

  1. I can't figure out what OSI layer to deal with when troubleshooting.
  2. Do I engage the ISPs and tell them my issues? Who should I point the finger at?
2
  • Update: 1. Both TCP and UDP tests come out the same. 2. I'm only able to send a 1472 byte sized ping over the Internet with the no fragment flag set.
    – Brent
    Aug 17, 2010 at 14:18
  • Update: I was playing with our DoubleTake SQL replication between Denver and Chicago today and restarted the sync. When it was working on the MDF, I saw the 7/1.5 MBPS limits. It seemed that when it hit the LDF, the traffic spiked up to 40MBPS for a few minutes. How weird is that?!
    – Brent
    Aug 17, 2010 at 15:47

2 Answers 2

3

It looks like it is time to run wireshark on both ends. My guess is it will help you find the traffic that is causing the problem. We recently had a similar (it was causing our iscsi san not to sync) it ended up being a bad data center switch. I used the wireshark logs to show them the problem and they replaced the switch.

3

my thoughts on this would be first, to try running multiple concurrent connections to see if you achieve your allotted speeds through multiple simultaneous connections, then you have pretty much ruled out any layer 1, 2 or 3 difficulties

possibly try a UDP throughput test, if you can send 70mbps in UDP traffic but you are limited to 7mbps on your TCP throughput, then I would strongly lean towards it being some sort of TCP sliding window issue

we ran into something very similar to this scenario a few months ago, and it was resolved by a switch that did not have the proper MTU setting on it, which I guess was limiting the TCP sliding window from hitting it's peak performance. I would verify you can send 1500 byte packets (the standard size for internet packets) from end to end with the do not fragment bit set on the ping.

hope this helps

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .