Scheduled File Transfer Between Servers

Question

I am interested in scheduling the copying and sending of large files over the internet from one server to another. The servers are not on the same network.

I do not know about all of my available options for solving this issue, but a couple protocols/specifications with which I am vaguely familiar which may be useful for this task are (1) FTP and (2) bittorrent.

I have used Filezilla before for manually sending large files between groups of servers. There are no scheduling capabilities of that particular FTP application, so it will not suffice for meeting my scheduling requirement. Ideally, I would like to accomplish this task using something more programmatic (non-GUI-based), and not proprietary (like a for-pay application).

I am already using Python scripts which run on a scheduled basis which perform other operations on a group of servers. The servers are running Windows and the scripts are automated via the Windows Task Scheduler. If I could send files via FTP using Python, that would be ideal for me specifically because I could just throw the FTP code into what I have already got working and scheduled. However, I briefly tried toying with Python's “ftplib” and could not make heads or tails of it. So, before I attempt to go down that path, I wanted to post here first.

I am uncertain as to whether I should pursue a FTP-based approach, so I would like some recommendations on that. Also, I do not really care what programming language I use for this task, although I would slightly prefer Python, as it seems like a natural choice since I am dealing with server-side scripting. I would like recommendations on that too.

As this is my first serverfault post, you can probably imagine I am mostly clueless about many of the topics mentioned above. Thanks in advance.

Answer 1

I would start by saying that you would want to avoid FTP and Bit Torrent. Bit Torrent is a great technology when many people have the file that you'd like and the download can be distributed amongst all of them (fast download of a linux ISO for instance). I would not recommend it for a simple site A to site B transfer.

FTP is largely regarded as out-dated because it's insecure. All of the information is transmitted in plain text and easily intercepted by malicious parties who may be listening to your traffic. You will want to do complete this transfer over a secure channel.

For that, you have a couple of options to encrypt your transmission.

• SSH tunnels (scp / sftp)
• VPN

I would recommend the following

• Download and install WinSCP on both nodes
• Review the Scripting Documentation and write a script to securely copy the files from point A to point B. You'll notice that this script can also easily be scheduled with the Windows Scheduler.

That is probably the easiest way I could recommend. Your other option would be to establish a VPN with the remote machine and simply write a one line script that would copy the file from your local server to the remote server.

Welcome to SF.

Answer 2

Just use any scriptable Windows FTP (or better FTPS or SFTP) client.

For example, as the accepted answer already suggests, WinSCP supports both scripting and FTP/FTPS/SFTP.

See automating file transfers to FTP server.

A simple batch file to upload files with WinSCP looks like:

winscp.com /log=c:\path\ftp.log /command ^
    "open ftp://user:password@example.com/" ^
    "put c:\local\path\* /remote/path/" ^
    "exit"

You can have the batch file generated by WinSCP GUI for you.

---

For scheduling, simply use the Windows Scheduler.
For details see scheduling file transfers to FTP server.

(I'm the author of WinSCP)

Answer 3

For a Windows based solution I recommend looking into TeraCopy (http://codesector.com/teracopy/). It works significantly faster than Windows copy, support pause/resume, and retry if the connection drops.

There are command-line parameters available so you can script out the transfers via batch file or Powershell and then schedule via Windows Task Scheduler.

I don't recommend FTP either. It isn't firewall friendly, resuming an upload may require special configuration on the FTP server end.