11

I have a directory with a couple hundred thousand files in it.

I want to delete all the files, but rm * -f reports: "argument list too long"

What is the best way to clear out this directory?

1
11

In the event you cannot remove the directory, you can always use find.

find . -maxdepth 1 -type f -exec rm -f {} \;

That will delete all files in the current directory, and only the current directory (not subdirectories).

4
  • almost perfect (see my post too) – asdmin Jul 24 '09 at 17:52
  • 2
    On linux you can do it even faster by using "+" instead of "\;". That will cause it to remove more files per "rm" in one go. – Thomas Jul 28 '09 at 13:07
  • 5
    Even faster: find dir/to/delete -delete (Does not spawn a rm process per file to delete). – Morten Siebuhr Jul 28 '09 at 14:42
  • Morten: He never mentioned wanting to delete the directory itself, eh said onky the files. find dir/to/delete -type f -delete – richo Dec 9 '10 at 6:07
21
find . -maxdepth 1 -type f -exec rm -f {} \;

it simply takes too long (one exec of rm per file).

this one is much more efficient:

find . -maxdepth 1 -type f -print0 | xargs -r0 rm -f

as it takes as much filenames as argument to rm as much it's possible, then runs rm with the next load of filenames... it may happen that rm is only called 2 or 3 times.

4
  • 5
    Glad to see someone know how find works... Also consider with new gnu find's the -delete action or -exec command {} + which acts like xargs. See the Actions section of 'man find'. – Kyle Brandt Jul 24 '09 at 18:21
  • I didn't know about -exec cmd {} +. That looks rather useful. Slightly more generic than -delete :) – David Pashley Jul 24 '09 at 21:16
  • -delete may be fine, but I personally prefer commands which can be used on variety of machines. Solaris systems I work with are older, so -delete would not work on them – asdmin Jul 24 '09 at 21:33
  • 2
    Adding ionice just before rm is a good idea if the computer is used by other people at the same time. – Hubert Kario Apr 26 '12 at 15:16
6

Both these will get round the problem. There is an analysis of the respective performance of each technique over here.

find . -name WHATEVER -exec rm -rf {} \;

or

ls WHATEVER | xargs rm -rf

The problem stems from bash expanding "*" with everysingle item in the directory. Both these solutions work through each file in turn instead.

6
  • 2
    The find solution is slower with many files because it invokes the rm command for every single file. The xargs solution is faster but it only works when the filenames do not contain spaces (otherwise you need GNU find and find . -print0 | xargs -0 rm). – robcast Jul 24 '09 at 16:13
  • "find . -name WHATEVER -print0 | xargs -0 rm -rf" will be more efficient. find will fork an rm for every file, where as xargs will reduce it to the minimum. You need the -print0 and -0 to cope with files with spaces in. You probably also want to use -depth on find, so it does a depth first search. – David Pashley Jul 24 '09 at 16:13
  • The ls command will not work, as that too returns "argument list too long" – Brent Jul 24 '09 at 16:19
  • 2
    You're doing "ls *", and the "*" expands to the argument list that your shell complains about being too long. Do "ls ." instead (or go up one directory level and do "ls [dirname]"). – James Sneeringer Jul 24 '09 at 18:03
  • the second command line (ls WHATEVER | xargs rm -rf) would not work for the same reason as the question did not work: WHATEVER would be substituded by shell to an exceeding length – asdmin Jul 24 '09 at 21:31
3

I was able to do this by backing up one level:

cd ..

And running:

rm directory name -rf

And then re-creating the directory.

1
  • 1
    it works only if you don't have to have the dir permanently and all the files and directories shall be recursively deleted. most of the cases in my life, this way would have not worked. – asdmin Jul 26 '09 at 8:09
2

All these find invocations are very nice but I seldom remember exactly the nomenclature needed when I'm in a hurry: instead I use ls. As someone mentions, ls . would work but I prefer ls -1 as in:

ls -1 | xargs -n 100 rm -rf

The -n xxx figure is pretty safe to play around with as exceeding the maximum will either be auto-corrected ( if size-max is exceeded; see -s ) or if the args-max for an app is exceeded it will usually be rather obvious.

It should be noted grep is handy to insert in the middle of this chain when you only want to delete a subset of files in a large directory, and don't for whatever reason want to use find.

This answer assumes you are using Gnu core utilities for your ls, xargs & etc.

2
  • Will this work with a directory that is too large for "ls" to work? (same error - arguement list too long) – Brent Aug 5 '09 at 13:58
  • Yes Brent: just be sure not to use a filespec when invoking ls. Use just the -1 (dash One) as shown above. And as explained above, use grep after the ls if you need to pair down the file list. If you are getting the "too long" error after making sure you're not using any wildcards with ls, then maybe xargs is complaining. Use 'xargs -n 5' instead of 'xargs -n 100' and you should definitely be safe, if a little slower. – rixtertech Dec 14 '09 at 18:03
1

You can use the -exec + option to find which which will try to run rm as few times as possible, which might be faster.

find . -type f -exec rm '{}' +
0

Here's a version for deleting large number of files when the system needs to remain responsive.

It works by issuing work in small batches (100 files by default) and waiting a bit for other jobs to finish.

Worked brilliantly for deleting over half a million files from single directory on ext3. It prints percentage done as a little bonus

noOfFiles=`ls -U | wc -l`
n=0
while (ls -U | tail -n 100 | xargs rm -f); do 
   n=$((n+100));
   sync;
   echo -en "$((n*100/noOfFiles))%, ($n of $noOfFiles)\r";
   sleep 5;
done
0

Solves "argument too long" or "cannot allocate memory" errors

This did the trick on 220,000+ files in session folder....

Advantage: instantly starts removing files

cd path/to/folder
ls -f | xargs rm -f -v

CLICK for screenshot of files being removed - (removed all files in ~ 15min)

-f (after ls) keeps from presorting

-v (after rm) displays each file as being removed

-f (after rm) forces through with no prompt on write-protected files

Tip: Rename folder (ex session to session_old) first to keep additional autogenerated files from being added while you are trying to delete files. You can remake original directory manually if it doesn't automatically as in my case

1
  • What’s new in your answer that was missing from the previous ones? – user2233709 Mar 14 '17 at 21:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.