I have a file that has all the urls from which I need to download. However I need to limit one download at a time. i.e. the next download should begin only once previous one is finished. Is this possible using curl? Or should I use anything else.
closed as off-topic by Jenny D, Andrew Schulman, MadHatter, Sirch, DanBig Jun 6 '18 at 14:25
This question appears to be off-topic. The users who voted to close gave this specific reason:
- "Questions on Server Fault must be about managing information technology systems in a business environment. Home and end-user computing questions may be asked on Super User, and questions about development, testing and development tools may be asked on Stack Overflow." – Jenny D, Andrew Schulman, Sirch, DanBig
wget(1) works sequentally by default, and has this option built in:
-i file --input-file=file Read URLs from a local or external file. If - is specified as file, URLs are read from the standard input. (Use ./- to read from a file literally named -.) If this function is used, no URLs need be present on the command line. If there are URLs both on the command line and in an input file, those on the command lines will be the first ones to be retrieved. If --force-html is not specified, then file should consist of a series of URLs, one per line. However, if you specify --force-html, the document will be regarded as html. In that case you may have problems with relative links, which you can solve either by adding "<base href="url">" to the documents or by specifying --base=url on the command line. If the file is an external one, the document will be automatically treated as html if the Content-Type matches text/html. Furthermore, the file's location will be implicitly used as base href if none was specified.
xargs -n 1 curl -O < your_files.txt
This is possible using curl within a shell script, something like this but you'll need to research appropriate options for curl etc for yourself
while read URL curl some options $URL if required check exit status take appropriate action done <fileontainingurls
Based on @iain answer, but using proper shell scripting -
while read url; do echo "== $url ==" curl -sL -O "$url" done < list_of_urls.txt
Will also work with weird characters like ampersands, etc...
Can replace the
-O with a redirect into a file instead, or whatever is suitable.