4

Is it possible to get all files in a directory using curl? Here's my string so far:

curl --ftp-ssl -k ftp://user:pass@IP

This will LIST the files in the user FTP directory, how do I adjust this string to get (RETR) all files in the remote directory?

  • Is there a strict requirement to use curl, or can you install lftp? – Aaron Jul 6 '17 at 13:02
4

AFAIK, there is no such option to download a directory with curl, so you must get the listing first and pipe it to curl to download file by file, something like this:

$ curl -s ftp://user:pass@IP/path/to/folder/ | \
    grep -e '^-' | awk '{ print $9 }' | \
        while read f; do \
            curl -O ftp://user:pass@IP/path/to/folder/$f; \
        done
2

You can also parallelise the process via xargs:

curl -s "ftp://user:pass@IP/path/to/folder/" \
    | grep -e '^-' \
    | awk '{ print $9 }' \
    | xargs -I {} -P ${#procs} curl -O "ftp://user:pass@IP/path/to/folder/{}"
0

Here's more reusable script that accomplishes quanta's curl|pipe|while solution:

#!/bin/bash
#
# Use cURL to download the files in a given FTPS directory (FTP over SSL)
# eg:
# curl_get_ftp_dir -u myUserName -p myPassword -d ftps://ftpserver.com/dir1/dir2
# Optionally, use:
#    -k - to ignore bad SSL certificates
#    --config userPassFile - to use the contents of a file including myUserName:myPassword in place of command line arguments
#    --silent - to silently download (otherwise cURL will display stats per file)

while [[ $# -gt 1 ]]
do
key="$1"

case $key in
    -u|--username)
    user="$2"
    shift 
    ;;
    -p|--password)
    pass="$2"
    shift 
    ;;
    -d|--dir|--directory)
    ftpDir="$2"
    shift 
    ;;
    -k|--ignoreSSL|--ignoreBadSSLCertificate)
    ignoreBadSSLCertificate="-k"
    ;;
    -s|--silent)
    silent="-s"
    ;;
    -K|--config|--configFile)
    config="$2"
    shift 
    ;;
    *)
        echo "Unknown Option!"
    ;;
esac
shift 
done

if [ -z "${config+x}" ]; then 
    CURL_OPTS="$silent --ftp-ssl $ignoreBadSSLCertificate -u $user:$pass"
else
    #CURL_OPTS="$silent --ftp-ssl $ignoreBadSSLCertificate --config $config"
    #Originally intended to be a curl config file - for simplicity, now just a file with username:password in it
    #Be sure to chmod this file to either 700, 400, or 500
    CURL_OPTS="$silent --ftp-ssl $ignoreBadSSLCertificate -u $(cat $config)"
fi

curl $CURL_OPTS $ftpDir | \
grep -e '^-' | awk '{ print $9 }' | \
    while read f; do \
        curl -O $CURL_OPTS $ftpDir/$f; \
    done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.