18

What is the best way to execute 5 curl requests in parallel from a bash script? I can't run them in serial for performance reasons.

29

Use '&' after a command to background a process, and 'wait' to wait for them to finish. Use '()' around the commands if you need to create a sub-shell.

#!/bin/bash

curl -s -o foo http://example.com/file1 && echo "done1" &
curl -s -o bar http://example.com/file2 && echo "done2" & 
curl -s -o baz http://example.com/file3 && echo "done3" &

wait
  • Simple, but effective for a first step. Gets hacky quickly when things need to change, like hostname or number of repetitions. Thanks. – Chris Oct 16 '18 at 13:07
9

xargs has a "-P" parameter to run processes in parallel. For example:

wget -nv http://en.wikipedia.org/wiki/Linux -O- | egrep -o "http://[^[:space:]]*.jpg" | xargs -P 10 -r -n 1 wget -nv

Reference: http://www.commandlinefu.com/commands/view/3269/parallel-file-downloading-with-wget

5

I use gnu parallel for tasks like this.

  • 4
    Could you provide an example for calling curl with gnu parallel? – m13r Jan 18 '18 at 12:34
  • Yes, parallel seems very good and it's easy to send the same request 100 times. But an example on how to use parallel with sending 100 different curl requests would make this answer better. – рüффп Mar 27 at 9:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.