Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I am trying to transfer a number of large (10's of GB up to 100's of GB) files to an HP DL380 host, running vsftpd on a 10 GigE network / LAN. The transfer initially starts out fast, i.e. 45-60 MB/sec or roughly the speed of the disks. After several minutes (usually about 10-12) the transfer rate drops to a trickle (250 KB/sec or so). We do this sort of thing all the time to other servers, so it seems there is something with the server itself or perhaps a firewall between the client / server.

I am trying to troubleshoot this issue, and why our rates would suddenly and dramatically change. It does not seem to have anything to do with load on the network or the client/server. We suspected it had something to do with PASSIVE/ACTIVE mode, but that hasn't played out (transfers are in PASSIVE mode).

I'm hoping someone out there has seen something similar, i.e. a dramatic drop off in FTP performance. Some network parameter perhaps? A config item in vsftpd?

Anyway, any suggestions appreciated.

share|improve this question
    
serverfault.com/questions/275430/… is an exact dupe –  anthonysomerset May 31 '11 at 8:27
    
Yes, sorry about that, I plead newbie. The other one is closed now. –  Mr. Ed May 31 '11 at 10:37

3 Answers 3

Before zeroing in on the FTP aspect, you should perform some synthetic tests to rule out other causes. (I've assumed both the client and the server are Linux since you are running vsftpd.)

  • Network packet drops: Check the output of cat /proc/net/dev on the client and server to see if there are any errors or dropped packets.
  • Network load testing: Use a throughput testing tool such as iperf with a long test time to see if you can reproduce the issue. Monitor throughput with cat /proc/net/dev.
  • MTU: If you're using 10GbE, you may have jumbo frames enabled (that is, MTU size greater than 1500). If you do, make sure that all the hosts and network devices support jumbo frames and that you aren't filtering ICMP packets anywhere (ICMP packets are used for MTU path discovery).
  • Server disk: On the server, use dd to read from /dev/zero and write to a local file on the target filesystem. Monitor throughput with iostat.
  • Client disk: May not need to test this given that you can FTP successfully to other servers from this client.

If any of the above tests show problems, you should have a clue about where to look next. Otherwise, look into the FTP side:

  • If a transfer has slowed down, what happens if you disconnect and then resume the transfer? Is it slow straight away, or does it go fast again before slowing down?
  • Do you get different behaviour when initiating the transfer from the client side vs the server side (i.e. connecting to an FTP server on the client)?
  • Check the memory usage of the FTP server and client using top. If there's a memory leak something may be swapping and slowing everything down.
  • Check the CPU usage of the FTP server and client for any strange behaviour.
share|improve this answer

In addition to Tom's answer, you may want to check your switches to make sure your running at the correct speeds and at full duplex.

share|improve this answer

I would check your disk IO... Perhaps the reason why the transfers seem fast at first and then slow down is that at first the OS is loading the data into the RAM and disk buffers.

Most modern Operating Systems will do this to make things look faster...

Once your RAM and Disk Buffers are full, which might take several minutes the true Disk IO speed is revealed and the file transfer slows down...

We recently published a blog about disk IO for 10GBPS file transfers, please take a look it might help you...

http://www.filecatalyst.com/analysis-on-improving-throughput-part-1-disk-io/

Full Disclosure: This user is an employee of FileCatalyst.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.