Big FTP transfer from one server to the other

Question

Greetings, I am using ncftpput to transfer a lot of files from on server to the other

ncftpput -f server.txt -vRdb /public_html /var/www/site.com

What happens is, the connection gets cut after about 100 transfers. There are thousands of files that have to be transferred. Problem is after the connection is cut and I reenter the command, it starts from the beginning. Replacing all existing files, making the previous transfer redundant. Is there a way to skip existing files? It's not in the man page so I'm assuming no. So does anybody suggest an alternative command line solution?

Answer 1

lftp is a great tool for this, it's scriptable using the mirror command:

 lftp has builtin mirror which can download or update a whole
 directory  tree.  There  is  also reverse mirror (mirror -R)
 which uploads or updates a directory tree on server.  Mirror
 can also synchronize directories between two remote servers,
 using FXP if available.

$ lftp -f

script_file:

open ftp://<user>:<pass>@ftp.blah.org/~
set net:timeout 10
set net:max-retries 2
set net:reconnect-interval-base 5
set net:reconnect-interval-multiplier 1
mirror dir1 dir1
exit

Answer 2

You could try to work with ncftpput's -DD option. This will delete the local file after it was successfully transfered. If you don't want to delete files in your /public_html directory, you could copy them to a temporary directory first.

Answer 3

#!/bin/bash

FILES=`cat server.txt`
for FILE in $FILES
do
  if [ ! -d "$FILE" ]; then
    echo "Uploading $FILE..."
    ncftpput -DD -u $FTPUSER -p $FTPPASS $FTPHOST /destination/ftp/dir/$FILE
  fi
done

Don't forget to set the FTPUSER, $FTPPASS, $FTPHOST The -DD option will delete the file after a successful upload. The "! -d "$FILE"" in line 4 will skip directories within server.txt. Don't forget to make the script executable with "chmod 755 yourscriptfile".

Answer 4

Have you tried this option?

-A Append to remote files, instead of overwriting them.

(from ncftp site)

Answer 5

Try wput

Answer 6

Is there any reason why you can't use rsync rather than ftp?

IMO rsync is the right tool for this job. It will only transfer new and changed files, which means that if the transfer dies and has to be restarted, it will pick up from where it left off rather than start from the beginning again.