1

I've got a remote server that I can access via SFTP. (Not FTP over SSL. SFTP.)

Periodically, I need to push a few thousand new files to that machine.

How can I:

  1. Upload upload these in parallel?
  2. Gather statistics, so I can judge the ideal level of parallelization given my particular server/network configuration?

A command line client would be ideal, but a GUI could work as well.

Thank you

Improve this question

2 Answers 2

Reset to default
2
  1. Use a client like Filezilla or WinSCP that will allow multiple streams. Or script it in a parallel fashion, using either something multi-threaded, or divide the files into chunks that you can run multiple scripts on.
  2. WinSCP will keep detailed logs, so you can go back through those logs and generate your own stats. Filezilla may, I'm not sure. Your own script would generate the logs that you tell it to.
Improve this answer
3
  • I've tried Filezilla, but I can't seem to find a way to get it to report times. I could use a physical clock, I guess. (WinSCP apparently has had some adware issues recently. So I'm trying to avoid it.)
    – nonot1
    Mar 16, 2011 at 17:26
  • I use WinSCP and don't know what adware issue you might be referring to. For filezilla's logging, go to Settings->Logging and enable logging. with timestamps. Bam - logs with times.
    – mfinni
    Mar 16, 2011 at 17:28
  • Filezilla->Settings-Logging did the trick.
    – nonot1
    Mar 17, 2011 at 3:31
0

mfinni has given you the best answers I can think of using sftp.

My suggestion to you is to investigate rsync and see if it meets your needs.
It can be used to push, pull, synchronize, replace, etc. with appropriate command line options, implements good compression and sends deltas rather than whole files when possible. It also has decent statistics which may meet your needs.

Improve this answer
2
  • Dunno if rsync is the best option for his needs. These are new files, so the only-sync-diffs won't help. AFAIK, rsync isn't multithreaded. Also, the default option of checking hash values, while it improves reliability, will not speed things up. Rsync is great for a lot of things, but I don't know that this is one of those situations.
    – mfinni
    Mar 16, 2011 at 15:56
  • @mfinni agreed - rsync may not be the ideal solution here. Just another option to consider when evaluating tools.
    – voretaq7
    Mar 16, 2011 at 16:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.