Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Having an issue with any large file transfer that crosses our Cisco ASA unit come to an eventual pause.

Setup

Test1: Server A, FileZilla Client <- 1GBPS -> Cisco ASA <- 1 GBPS -> Server B, FileZilla Server

TCP Window size on large transfers will drop to 0 after around 30 seconds of a large file transfer. RDP session then becomes unresponsive for a minute or two and then is sporadic. After a minute or two, the FTP transfer resumes, but at 1-2 MB/s.

When the FTP transfer is over, the responsiveness of the RDP session returns to normal.

Test2: Server C in same network as Server B, FileZilla Client <- local network -> Server B, FileZilla Server

File will transfer at 30+ MB/s.

Details

ASA: 5520 running 8.3(1) with ASDM 6.3(1)

Windows: Server 2003 R2 SP2 with latest patches

Server: VMs running on HP C3000 blade chasis

FileZilla: 3.3.5.1, latest stable build

Transfer: 20 GB SQL .BAK file

Protocol: Active FTP over tcp/20, tcp/21

Switches: Cisco Small Business 2048 Gigabit running latest 2.0.0.8

VMware: 4.1

HP: Flex-10 3.15, latest version

Notes

All servers are VMs.

Thoughts

Pretty sure the ASA is at fault since a transfer between VMs on the same network will not show a shrinking Window size.

Our ASA is pretty vanilla. No major changes made to any of the settings. It has a bunch of NAT and ACLs.

Wireshark Sample

No.     Time        Source           Destination       Protocol  Info
 234905 73.916986   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=131981791 Win=65535 Len=0
 234906 73.917220   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234907 73.917224   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234908 73.917231   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=131984551 Win=64155 Len=0
 234909 73.917463   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234910 73.917467   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234911 73.917469   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234912 73.917476   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=131988691 Win=60015 Len=0
 234913 73.917706   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234914 73.917710   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234915 73.917715   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=131991451 Win=57255 Len=0
 234916 73.917949   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234917 73.917953   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234918 73.917958   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=131994211 Win=54495 Len=0
 234919 73.918193   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234920 73.918197   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234921 73.918202   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=131996971 Win=51735 Len=0
 234922 73.918435   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234923 73.918440   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234924 73.918445   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=131999731 Win=48975 Len=0
 234925 73.918679   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234926 73.918684   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234927 73.918689   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=132002491 Win=46215 Len=0
 234928 73.918922   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234929 73.918927   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234930 73.918932   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=132005251 Win=43455 Len=0
 234931 73.919165   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234932 73.919169   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234933 73.919174   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=132008011 Win=40695 Len=0
 234934 73.919408   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234935 73.919413   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234936 73.919418   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=132010771 Win=37935 Len=0
 234937 73.919652   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234938 73.919656   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234939 73.919661   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=132013531 Win=35175 Len=0
 234940 73.919895   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234941 73.919899   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234942 73.919904   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=132016291 Win=32415 Len=0
 234943 73.920138   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234944 73.920142   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234945 73.920147   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=132019051 Win=29655 Len=0
 234946 73.920381   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234947 73.920386   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234948 73.920391   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=132021811 Win=26895 Len=0
 234949 73.920625   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234950 73.920629   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234951 73.920632   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234952 73.920638   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=132025951 Win=22755 Len=0
 234953 73.920868   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234954 73.920871   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234955 73.920876   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=132028711 Win=19995 Len=0
 234956 73.921111   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234957 73.921115   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234958 73.921120   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=132031471 Win=17235 Len=0
 234959 73.921356   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234960 73.921362   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234961 73.921370   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=132034231 Win=14475 Len=0
 234962 73.921598   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234963 73.921606   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234964 73.921613   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=132036991 Win=11715 Len=0
 234965 73.921841   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234966 73.921848   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234967 73.921855   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=132039751 Win=8955 Len=0
 234968 73.922085   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234969 73.922092   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234970 73.922099   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=132042511 Win=6195 Len=0
 234971 73.922328   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234972 73.922335   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234973 73.922342   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=132045271 Win=3435 Len=0
 234974 73.922571   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234975 73.922579   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 1380 bytes
 234976 73.922586   1.1.1.1           2.2.2.2          TCP      ftp-data > ivecon-port [ACK] Seq=1 Ack=132048031 Win=675 Len=0
 234981 75.866453   2.2.2.2          1.1.1.1           FTP-DATA FTP Data: 675 bytes
 234985 76.020168   1.1.1.1           2.2.2.2          TCP      [TCP ZeroWindow] ftp-data > ivecon-port [ACK] Seq=1 Ack=132048706 Win=0 Len=0
 234989 76.771633   2.2.2.2          1.1.1.1           TCP      [TCP ZeroWindowProbe] ivecon-port > ftp-data [ACK] Seq=132048706 Ack=1 Win=65535 Len=1
 234990 76.771648   1.1.1.1           2.2.2.2          TCP      [TCP ZeroWindowProbeAck] [TCP ZeroWindow] ftp-data > ivecon-port [ACK] Seq=1 Ack=132048706 Win=0 Len=0
 234997 78.279701   2.2.2.2          1.1.1.1           TCP      [TCP ZeroWindowProbe] ivecon-port > ftp-data [ACK] Seq=132048706 Ack=1 Win=65535 Len=1
 234998 78.279714   1.1.1.1           2.2.2.2          TCP      [TCP ZeroWindowProbeAck] [TCP ZeroWindow] ftp-data > ivecon-port [ACK] Seq=1 Ack=132048706 Win=0 Len=0

January 18 Addition

I have found that I can replicate the issue between two server that are on the same subnet.

http://imgur.com/jiURb

The above graph is from the VM running the FTP server. 1. FTP transfer is started and the Disk Queue builds up. 2. I stopped the FTP transfer yet the Disk Queue is still very, very long 3. Stop recording. The high disk queue continues for around 1-2 minutes and then the VM becomes responsive.

January 19 Addition

It seems like the Disk Queuing has lead me down a RAID I/O path.

From what I understand, FTP delivers data is small amounts, rather than a modern protocol that can do fancy jumbo work.

I played around withe the buffer in FileZilla.

FileZilla Internal Buffer  Avg Disk Queue  MB/s
65,535                          200.0      10.0 (VM RDP becomes unresponsive)
131,072                      92.0        8.0 (VM RDP becomes unresponsive)
262,144                      9.8         4.1
524,288                      5.6         4.3
786,432                      7.2         5.2

There seems to be a relationship between the Disk Queuing and the buffer size. I cannot seem to get the throughput to go higher than ~4.2 MB/s

share|improve this question
add comment

2 Answers

This looks like an ASA configuration issue, I had an issue like this in the past

Make sure that the tcp-options window-scale clear option is not configured in the config, if it is remove it.

This is an old ASA bug so also trying to upgrade your ASA version although disruptive is not a bad idea.

share|improve this answer
    
I do not see the "tcp-options window-scale clear" line in my config –  Brent Jan 13 '11 at 19:41
    
Then I would definitely go for an ASA version upgrade, this is as said an old ASA bug. –  lynxman Jan 13 '11 at 19:41
    
Looks like I can update ASA 8.3(1) -> 8.3(2) and ASDM 6.3(1) -> 6.3(5). I'll get some downtime scheduled this week and proceed with the update. –  Brent Jan 13 '11 at 19:48
    
Update to 8.3(2) and 6.3(5) did not resolve the issue. –  Brent Jan 13 '11 at 21:36
    
Hmm... alright then try to simplify your problem, if possible take the ASA out of the equation just for testing and put a normal linux/freebsd box inbetween with a network sniffer and see what happens, if that does the same then you know the TCP stack of one of those servers is at fault. Otherwise posting your ASA configuration here might also help diagnose. –  lynxman Jan 14 '11 at 9:59
add comment

You're not testing the same thing. In test 1 serverA is the client while in test 2 serverC is the client. Windows scaling is a funtion of the recieving host. I'm not saying that the ASA is not at fault but how do you know that serverA is not the problem if you're not testing it both through the ASA and on the same LAN as when you're testing serverC? If you move serverA to the LAN and serverC to the WAN do you get the same results for both tests or do the results "swap" as in serverC on the WAN now has the problem and serverA on the LAN is fine?

share|improve this answer
    
Good point. I'll move them around and retest. –  Brent Jan 14 '11 at 17:11
    
Curious to see what you find out... –  joeqwerty Jan 14 '11 at 18:19
    
Further testing has shown that I can replicate the Shrinking Window issue between two VMs on the same subnet. So that means that the ASA is not at fault. –  Brent Jan 18 '11 at 18:28
    
Well that's progress then. Keep us posted on what else you find. Were these tests done with a FileZilla server and client? If so, maybe one or the other is the problem. –  joeqwerty Jan 18 '11 at 20:19
    
If you're testing with different servers internally than your original test then I'd say that rules any of the servers out as the problem, seeing as they all exhibit the same behavior. I think that leaves Filezilla as the culprit. It can't process the data as fast as the TCP\IP stack is sending it up and it's telling TCP to slow down (by reducing the windows size). –  joeqwerty Jan 21 '11 at 1:54
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.