Automate the FTP process

Question

What is the best way to automate the ftp process. I mean can a script be written which when executed would ftp to a remote system and fetch files with out human intervention.

Please suggest.

Answer 1

NcFTP Client

Answer 2

You've gotten a lot of recommendations already to use something different than FTP. That's fair, because scripting FTP is a little annoying, but it's not necessarily portable, nor is it teaching you anything new.

It's actually quite simple to script by directing commands to ftp from STDIN. This sort of thing works for many kinds of tools.

cd /target
ftp -n host <<EOF
USER falken
PASS joshua
cd place
binary
get file
bye
EOF
echo 'hi! um... what'\''s this all about again? I am done.'

The login part could be automated by adding information to the script owner's ~/.netrc file. In fact, if you man netrc you'll learn a lot about automating ftp.

Answer 3

I typically just use ftp with an authentication file .netrc and then cat everything into it. For example

cat >> EOF | ftp
open ftpserver
mget *
bye
EOF

Answer 4

The Solaris ftp command can be automated with a netrc(4) file. The macdef init is the key. sample:

cat ~/.netrc
machine remote-host1.domain.com login my-ftp-account
password password4host1
macdef init
cd
/pub/DOWNLOAD get readme

default login anonymous password my-email@mail.domain.com

Notes:

1. The two blank lines after the init macro definition are important,
2. This also automates an interactive ftp session to remote-host1.

Your script can create the netrc file before it calls ftp.

Answer 5

I like curl the best:

curl is a command line tool for transferring files with URL syntax, supporting FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS and FILE. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, kerberos...), file transfer resume, proxy tunneling and a busload of other useful tricks.

Very handy for all your URL-fetching needs.

Answer 6

ncftp usually includes ncftpget and ncftpput, which would be suitable for inclusion in scripts. They can usually be built from source, or check csw or sunfreeware.

Answer 7

There are Perl packages for this purpose. For instance CPAN modules Net::Lite::FTP and Net::FTP.

It will allow your script to make decisions based on information from the server. E.g. if the file name is not constant (may contain a date or a version number) then the script can process a directory listing before deciding which file to download.

I have succesfully used it for automating downloading/updating of large molecular sequence databases.

Answer 8

Hi I've written one once for our needs,

#!/bin/sh

hostname="IP-ADRESS"

port="21"

username="serverfault"

password="serverfalut"

storage_path="DESTINATION"

current_date="$(date +%Y%m%d)"

$remotefile="REMOTE-FILE"

ftp -in $hostname $port << EOF

quote user $username

quote pass $password

binary

get $remotefile $storage_path/

delete $output_dir/output$current_date.tar.gz

quit

close

EOF

exit 0

Answer 9

On Solaris you can also use expect automation tool. Of course you should install it first. For FTP you can find an example in Wikipedia article on Expect.

Answer 10

I think you can use wget to do such a thing:

WGET is free so you can grab that and check the help file

Something like:

wget -user=username -pass=password ftp://ftp.moose.com/download.zip

Obviously usable with variables in a script etc.

Just noticed this is for solaris, so not sure if this would work.

Answer 11

I personally use lftp, which is highly scriptable (see this tutorial).

Of course, you can also write an adapted FTP client using perl/ruby/python/your-prefered-scripting-language.