How to make `rm` faster on ext3/linux?

Question

I have ext3 filesystem mounted with default options. On it I have some ~ 100GB files.

Removal of any of such files takes long time (8 minutes) and causes a lot of io traffic, which increases load on server.

Is there any way to make the rm not as disruptive?

Answer 1

You can give ionice a try. It won't make it faster but it might make it less disruptive.

Answer 2

Upgrade to ext4 or some other modern filesystem that uses extents. Since ext3 uses the indirect blocks scheme rather than extents, deleting large files inevitably entails lots of work.

Answer 3

In terms of efficiency, using one rm per file is not optimal, as it requires a fork and exec for each rm.

Assuming you have a list.txt containing the files you want to remove this would be more efficient but it's still gonna be slow:

xargs -i rm {} < list.txt

Another approach would be to : nice -20 xargs -i rm {} < list.txt
(this will take less time but will affect your system greatly :)

or

I don't know how fast this would be but:

mv <file-name> /dev/null 

or

Create a special mount point with a fast filesystem (using a loop device ?) , use that to store and delete your Huge files.
(maybe move the files there before you delete them, maybe it's faster or maybe just unmount it when you want files gone)

or

cat /dev/null > /file/to/be/deleted (so it's zero-sized now) and if you want it to disappear just rm -rf <file> now

or even better

drop the cat and just do # > /file/to/be/emptied

Answer 4

Since the most interesting answer is buried in a comment on the question, here it is as a first class answer for your upvoting pleasure:

Basically no method from here worked, so we developed our own. Described it in here: http://www.depesz.com/index.php/2010/04/04/how-to-remove-backups/ – depesz Apr 6 '10 at 15:15

That link is an incredibly thorough analysis of the exploration for and discovery of a workable solution.

Answer 5

Also note that the answer by Dennis Williamson, who suggests ionice as a workaround for the load, will work only if your block device uses the CFQ io scheduler.

Answer 6

mv /dev/null

/dev/null is a file not a directory. Can't move a file, to a file, or you risk overwriting it.

Create a special mount point with a fast filesystem (using a loop device ?) , use that to store and delete your Huge files. (maybe move the files there before you delete them, maybe it's faster or maybe just unmount it when you want files gone)

I don't think this is practical. It would use unnecessarily more I/O than the OP would like.

Answer 7

/dev/null is a file not a directory. Can't move a file, to a file, or you risk overwriting it.

Actually it's a device and all data written to it gets discarded so mv <file> /dev/null makes sense

From Wikipedia, the free encyclopedia
In Unix-like operating systems, /dev/null or the null device is a special file that discards all data written to it (but reports that the write operation succeeded), and provides no data to any process that reads from it (yielding EOF immediately).[1]

Answer 8

You could try creating a loop file system to store your backups on.

# dd if=/dev/zero of=/path/to/virtualfs bs=100M count=1024 # 100 MB * 1024 = 100 GB
# mke2fs /path/to/virtualfs
# mount -t ext2 /path/to/virtualfs /mnt/backups -o loop

Then, when you want to clear out the backups:

# umount /mnt/backups
# mke2fs /path/to/virtualfs
# mount -t ext2 /path/to/virtualfs /mnt/backups -o loop

Presto! The entire virtual file system is cleared out in a matter of moments.

Answer 9

You can use multitheading whith xargs

find . -type f | xargs -P 30 rm -rf

where 30 is the number of threads that you want to create. If you are using zero, the system creates maximum threads available to the user executing the task.

BR