7

I have a deep directory structure, with a large number of files (about 1M). I'd like to execute a command against each. (In my case the files are pngs which I'd like to run optipng against.)

I've tried this:

find . -name *.png | xargs sudo optipng -o2

but am getting the error argument list too long.

I'm assuming that xargs should be able to handle this, and that there must be a problem with my syntax.

  • 2
    Caution: If your current directory contains PNG files, the shell will expand *.png to their names - thus doubly defeating the purpose of the command (not matching all .png files, and possibly causing an argument list too long for find itself. Always quote things you want to pass as-is: find . -iname '*.png' | ... – muru Jun 25 '15 at 13:49
7

Do the exec directly with find rather than via xargs.

sudo find . -name '*.png' -exec optipng -o {} +
  • Thanks. Will that execute the optipng as sudo though? I've tried sudo find with xargs, and it didn't (I had to put the sudo next to xargs) [not my downvote btw] – UpTheCreek Jun 25 '15 at 7:59
  • 2
    Yes the find gets run as root and it spawns optipng which because it is being spawned by a process owned by root will also run as root. – Geraint Jones Jun 25 '15 at 8:19
  • 1
    Do you know if there is a way to get it to spawn n parallel processes? (e.g. with xargs you can do this with the -P switch) – UpTheCreek Jun 26 '15 at 8:08
  • 1
    You should quote *.png. Otherwise, if there are any PNG files in the current directory, it will be expanded to them. – Barmar Jun 30 '15 at 21:43
4

Using sudo as command to execute by xargs isn't working, as you see. Try it the other way around:

find . -name '*.png' | sudo xargs optipng -o2

The difference is that your version creates a command like

 sudo optipng -o2 file1 file2 file3 file4 ......

and sudo can't handle so many parameters.

Doing it the other way around executes sudo once, which in turn starts xargs which then generates a command like

optipng -o2 file1 file2 file3 file4 ......

and if optipng can handle many files as parameters, it should work.

If that still doesn't work, you will have to use -exec instead like in Geraint's answer, but this should be your last resort.

  • BTW - why would -exec be a last resort? (not familiar with it, so just curious) – UpTheCreek Jun 25 '15 at 8:02
  • 2
    He says 1M files, bash is the guy thats saying argument list too long not optipng or sudo. Your only option if you really have 1 million files is to use -exec – Geraint Jones Jun 25 '15 at 8:17
  • 4
    @GeraintJones: No, because it's precisely the job of xargs to make sure this doesn't happen. – Sven Jun 25 '15 at 9:00
  • 6
    If the -exec command is terminated with a + instead of a ; then find will group the files into groups small enough to avoid the "Argument list too long" error. (You must also provide a {} argument before the + which will be replaced by the filename list.) This feature wasn't in the original unix find, but it's been around long enough (in POSIX for over 10 years)... it's time to unlearn the use of xargs. – user193597 Jun 25 '15 at 12:15
  • 1
    You need to quote *.png in case there are any matching files in the current directory. – Barmar Jun 30 '15 at 21:43
2

Yet another shot at this:

find . -name '*.png' | sudo xargs -n x -P y optipng -o2

Which will start one optipng for x files and y such processes in parallel (might help you get things done faster if you have some cores to spare). Having x greater than 10 probably won't have great impact on performance.

That will help keep the argument list short. So you could probably move sudo to where you had it, but that's just another command that would have to be run for every batch.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.