What is the best way to automate the ftp process. I mean can a script be written which when executed would ftp to a remote system and fetch files with out human intervention.
Please suggest.
I personally use lftp, which is highly scriptable (see this tutorial).
Of course, you can also write an adapted FTP client using perl/ruby/python/your-prefered-scripting-language.
I think you can use wget to do such a thing:
WGET is free so you can grab that and check the help file
Something like:
wget -user=username -pass=password ftp://ftp.moose.com/download.zip
Obviously usable with variables in a script etc.
Just noticed this is for solaris, so not sure if this would work.
There are Perl packages for this purpose. For instance CPAN modules Net::Lite::FTP and Net::FTP.
It will allow your script to make decisions based on information from the server. E.g. if the file name is not constant (may contain a date or a version number) then the script can process a directory listing before deciding which file to download.
I have succesfully used it for automating downloading/updating of large molecular sequence databases.
I like curl the best:
curl is a command line tool for transferring files with URL syntax, supporting FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS and FILE. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, kerberos...), file transfer resume, proxy tunneling and a busload of other useful tricks.
Very handy for all your URL-fetching needs.
The Solaris ftp
command can be automated with a netrc(4) file. The macdef init is the key. sample:
cat ~/.netrc
machine remote-host1.domain.com login my-ftp-account
password password4host1
macdef init
cd
/pub/DOWNLOAD get readmedefault login anonymous password [email protected]
Your script can create the netrc file before it calls ftp.
You've gotten a lot of recommendations already to use something different than FTP. That's fair, because scripting FTP is a little annoying, but it's not necessarily portable, nor is it teaching you anything new.
It's actually quite simple to script by directing commands to ftp from STDIN. This sort of thing works for many kinds of tools.
cd /target
ftp -n host <<EOF
USER falken
PASS joshua
cd place
binary
get file
bye
EOF
echo 'hi! um... what'\''s this all about again? I am done.'
The login part could be automated by adding information to the script owner's ~/.netrc
file. In fact, if you man netrc
you'll learn a lot about automating ftp
.
On Solaris you can also use expect automation tool. Of course you should install it first. For FTP you can find an example in Wikipedia article on Expect.
I've written one once for our needs,
#!/bin/sh
hostname="IP-ADRESS"
port="21"
username="serverfault"
password="serverfalut"
storage_path="DESTINATION"
current_date="$(date +%Y%m%d)"
$remotefile="REMOTE-FILE"
ftp -in $hostname $port << EOF
quote user $username
quote pass $password
binary
get $remotefile $storage_path/
delete $output_dir/output$current_date.tar.gz
quit
close
EOF
exit 0
ncftp usually includes ncftpget and ncftpput, which would be suitable for inclusion in scripts. They can usually be built from source, or check csw or sunfreeware.
I typically just use ftp with an authentication file .netrc and then cat everything into it. For example
cat >> EOF | ftp
open ftpserver
mget *
bye
EOF