2

Possible Duplicate:
Doing an rm -rf on a massive directory tree takes hours

I'm running a simulation program on a computing cluster (Scientific Linux) that generates hundreds of thousands of atomic coordinate files. But I'm having a problem deleting the files because rm -rf never completes and neither does

find . -name * | xargs r

Isn't there a way to just unlink this directory from the directory tree? The storage unit is used by hundreds of other people, so reformatting is not an option.

Thanks

1
  • If you have lots of files it just takes time. For example I have a test system for this with a directory containing 900k files. The solutions in @ewwhite's answer (I recognise those commands Ed!) take ~80 minutes to complete.
    – user9517
    Nov 11, 2012 at 8:36

4 Answers 4

3

Method 1 Assuming those file are meant to be created, just need to be removed after use.

If possible, have all those files, and only those files, create in a standalone partition or disk. When it is time to deleted them, umount the partition and format it. EXT4 (not EXT2) format only takes a few seconds.

Make sure you are not saving information/report/etc in the same location.

You can mount a new partition or a new disk to the original location, either directly or with the -o bind option.

Method 2

Thinking a bit out of the box, instead of individual file, put all those data into a database table. Then drop the whole table after use.

3
  • Making a new ext file system doesn't just take a few seconds: for any decently-sized disc it takes many minutes, and for large RAID arrays can take over an hour.
    – MadHatter
    Nov 11, 2012 at 8:16
  • More arguments for the use of XFS :)
    – ewwhite
    Nov 11, 2012 at 8:27
  • It is correct if you are referring to ext2, but not ext4. By the way, how much disk space are we talking here, regarding those atomic files?
    – John Siu
    Nov 11, 2012 at 8:28
2

I typically use something like:

find ./directoryname -type f -name '*file-pattern*' -exec rm {} +

It is also possible to use the -delete flag to the find command.

find ./directoryname -type f -name '*file-pattern*' -delete

Is the generation of these files a problem/bug? Is there anything at the application level that can help?

2

Just unlinking the directory would be perfectly possible if you didn't mind not getting the free space back, and all the files reappearing in /lost+found at the next fsck.

Removing the files isn't the time-consuming bit, it's all the file system maintenance code that tidies up behind the scenes that is time-consuming, and it takes an extra-long time to do millions of small files. It takes even longer if they are in a flat, wide file structure, instead of a deep, thin one (ie many files in few directories instead of many files in may nested directories). As you've noticed, in some cases it can longer to do this than simply to recreate the file system.

If this were my issue, I'd make a custom partition to keep those files in, and in addition, I'd probably use tmpfs, which is better-designed for the storage of temporary files anyway, and will cut down the file system re-creation time.

2

My guess is you're running across a strange filetype that's blocking rm from completing. Try something like

find . -type d -o -type f -print0 | xargs -0 rm -rf --

Not the answer you're looking for? Browse other questions tagged or ask your own question.