Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've got a remote server that I can access via SFTP. (Not FTP over SSL. SFTP.)

Periodically, I need to push a few thousand new files to that machine.

How can I:

  1. Upload upload these in parallel?
  2. Gather statistics, so I can judge the ideal level of parallelization given my particular server/network configuration?

A command line client would be ideal, but a GUI could work as well.

Thank you

share|improve this question
add comment

2 Answers

up vote 2 down vote accepted
  1. Use a client like Filezilla or WinSCP that will allow multiple streams. Or script it in a parallel fashion, using either something multi-threaded, or divide the files into chunks that you can run multiple scripts on.
  2. WinSCP will keep detailed logs, so you can go back through those logs and generate your own stats. Filezilla may, I'm not sure. Your own script would generate the logs that you tell it to.
share|improve this answer
    
I've tried Filezilla, but I can't seem to find a way to get it to report times. I could use a physical clock, I guess. (WinSCP apparently has had some adware issues recently. So I'm trying to avoid it.) –  nonot1 Mar 16 '11 at 17:26
    
I use WinSCP and don't know what adware issue you might be referring to. For filezilla's logging, go to Settings->Logging and enable logging. with timestamps. Bam - logs with times. –  mfinni Mar 16 '11 at 17:28
    
Filezilla->Settings-Logging did the trick. –  nonot1 Mar 17 '11 at 3:31
add comment

mfinni has given you the best answers I can think of using sftp.

My suggestion to you is to investigate rsync and see if it meets your needs.
It can be used to push, pull, synchronize, replace, etc. with appropriate command line options, implements good compression and sends deltas rather than whole files when possible. It also has decent statistics which may meet your needs.

share|improve this answer
    
Dunno if rsync is the best option for his needs. These are new files, so the only-sync-diffs won't help. AFAIK, rsync isn't multithreaded. Also, the default option of checking hash values, while it improves reliability, will not speed things up. Rsync is great for a lot of things, but I don't know that this is one of those situations. –  mfinni Mar 16 '11 at 15:56
    
@mfinni agreed - rsync may not be the ideal solution here. Just another option to consider when evaluating tools. –  voretaq7 Mar 16 '11 at 16:08
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.