Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

What is the best way to execute 5 curl requests in parallel from a bash script? I can't run them in serial for performance reasons.

share|improve this question
    
Have you tried searching for parts of your solution? Another SF question seems to be exactly what you're asking for: serverfault.com/questions/248143/… –  Theuni Dec 9 '12 at 9:47

3 Answers 3

Use '&' after a command to background a process, and 'wait' to wait for them to finish. Use '()' around the commands if you need to create a sub-shell.

#!/bin/bash

curl -s -o foo http://example.com/file1 && echo "done1" &
curl -s -o bar http://example.com/file2 && echo "done2" & 
curl -s -o baz http://example.com/file3 && echo "done3" &

wait
share|improve this answer

xargs has a "-P" parameter to run processes in parallel. For example:

wget -nv http://en.wikipedia.org/wiki/Linux -O- | egrep -o "http://[^[:space:]]*.jpg" | xargs -P 10 -r -n 1 wget -nv

Reference: http://www.commandlinefu.com/commands/view/3269/parallel-file-downloading-with-wget

share|improve this answer

I use gnu parallel for tasks like this.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.