17

I'm using xargs to execute a command on a set of input parameters something like this:

cat <someinput> | xargs -n 1 -P 5 <somecmd>

The input file is really long and take a long time to run. So I'm just waiting for the command prompt to show up. Is there a way to display a progress bar for the number of input arguments that have completed?

I tried using 'bar' but I always got an 'infinite' throughput. It seems like xargs reads the entire input before executing commands.

https://i.stack.imgur.com/5Wsgx.gif

Improve this question

3 Answers 3

Reset to default
19

If you have GNU Parallel you can run:

cat <someinput> | parallel --bar -P 5 <somecmd>

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

Simple scheduling

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

GNU Parallel scheduling

Installation

A personal installation does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel

Improve this answer
1
  • 1
    thank you very much (for the answer, and for writing the software!), it works great!
    – Greg Sadetsky
    May 4, 2018 at 15:22
11

You can use pv :

cat <someinput> | pv -p -s sizeof_someimput | xargs -n 1 -P 5 <somecmd>

With this you will know where the reading of someimput is, so you will know approximately where the treatment of someimput is.

Improve this answer
3
  • 2
    Another frequent use-case might be the combination with find: FIND=( find "$DIR" -type f ) && ${FIND[@]} -print0 | pv -0lps $(${FIND[@]} | wc -l) | xargs -0 -I {} -P 5 <somecmd>
    – mxmlnkn
    May 14, 2016 at 0:09
  • 2
    The issue I have when using this approach is that the pipe buffer consumes the entire input of the cat command, showing a "complete" progress bar, while the xargs command has just started
    – Max Bileschi
    Feb 26, 2021 at 13:43
  • The solution to @MaxBileschi’s problem for me was to put pv after xargs: xargs -n 1 -P 5 --verbose ‹somecmd› < ‹someinput› 2>&1 | pv -l -s $(wc -l < ‹someinput›) > /dev/null
    – hfs
    Apr 9, 2022 at 23:16
7

If you're looking for just a general indication of progress the simplest method is to just echo prior to doing the command you'd like to do.

Example: cat <someinput> | xargs -I{} sh -c 'echo {}; <somecmd>;'

-I{} sets {} to the current string being processed

sh -c will allow you to execute multiple commands (note: semi-colon after every command is required including the last one.

Improve this answer
1
  • you can use xargs's -t, --verbose instead of echo {};
    – danilopopeye
    Dec 14, 2021 at 14:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .