Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have ext3 filesystem mounted with default options. On it I have some ~ 100GB files.

Removal of any of such files takes long time (8 minutes) and causes a lot of io traffic, which increases load on server.

Is there any way to make the rm not as disruptive?

share|improve this question
4  
Basically no method from here worked, so we developed our own. Described it in here: depesz.com/index.php/2010/04/04/how-to-remove-backups –  user13185 Apr 6 '10 at 15:15
add comment

10 Answers

up vote 11 down vote accepted

The most interesting answer was originally buried in a comment on the question. Here it is as a first class answer to make it more visible:

Basically no method from here worked, so we developed our own. Described it in here: http://www.depesz.com/index.php/2010/04/04/how-to-remove-backups/ – depesz Apr 6 '10 at 15:15

That link is an incredibly thorough analysis of the exploration for and discovery of a workable solution.

Note also:

The article says:

As you can see, I used -c2 -n7 options to ionice, which seem sane.

which is true, but user TafT says if you want no disruption then -c3 'idle' would be a better choice than -c2 'best-effort'. He has used -c3 to build in the background and has found it to work well without causing the build to wait for ever. If you really do have 100% io usage then -c3 will not let the delete ever complete but he doesn't expect that is what you have based on the worked test.

share|improve this answer
add comment

I had problems getting the directory to delete at a reasonable pace, turns out the process was locking the disk and creating a pileup of processes trying to access the disk. ionice didn't work, it just continued to use 99% of the disk IO and locked all the other processes out.

Here's the Python code that worked for me. It deletes 500 files at a time, then takes a 2 second break to let the other processes do their work, then continues. Works great.

import os, os.path
import time

for root, dirs, files in os.walk('/dir/to/delete/files'):
    file_num = 0
    for f in files:
        fullpath = os.path.join(root, f)
        os.remove(fullpath)
        if file_num%500 == 1:
            time.sleep(2)
            print "Deleted %i files" % file_num
        file_num = file_num + 1
share|improve this answer
1  
Try it on 100G+ files on ext3 filesystem. The problem is in size of single file, not number of files. –  user13185 Dec 23 '12 at 18:51
    
In your case it sounds like it wouldn't work. But I had a ton of small files. Thanks for the feedback. –  NickWoodhams Dec 23 '12 at 23:03
add comment

You can use multitheading whith xargs

find . -type f | xargs -P 30 rm -rf 

where 30 is the number of threads that you want to create. If you are using zero, the system creates maximum threads available to the user executing the task.

share|improve this answer
    
find has a -delete option which is a much better alternative. –  Ariel Oct 8 '13 at 7:05
add comment

You could try creating a loop file system to store your backups on.

# dd if=/dev/zero of=/path/to/virtualfs bs=100M count=1024 # 100 MB * 1024 = 100 GB
# mke2fs /path/to/virtualfs
# mount -t ext2 /path/to/virtualfs /mnt/backups -o loop

Then, when you want to clear out the backups:

# umount /mnt/backups
# mke2fs /path/to/virtualfs
# mount -t ext2 /path/to/virtualfs /mnt/backups -o loop

Presto! The entire virtual file system is cleared out in a matter of moments.

share|improve this answer
    
doesn't solve the problem, as it would work only if i'd want to remove all backups on given filesystem. –  user13185 Feb 22 '11 at 9:48
add comment

/dev/null is a file not a directory. Can't move a file, to a file, or you risk overwriting it.

Actually it's a device and all data written to it gets discarded so mv <file> /dev/null makes sense

From Wikipedia, the free encyclopedia
In Unix-like operating systems, /dev/null or the null device is a special file that discards all data written to it (but reports that the write operation succeeded), and provides no data to any process that reads from it (yielding EOF immediately).[1]

share|improve this answer
add comment

mv /dev/null

/dev/null is a file not a directory. Can't move a file, to a file, or you risk overwriting it.

Create a special mount point with a fast filesystem (using a loop device ?) , use that to store and delete your Huge files. (maybe move the files there before you delete them, maybe it's faster or maybe just unmount it when you want files gone)

I don't think this is practical. It would use unnecessarily more I/O than the OP would like.

share|improve this answer
add comment

Also note that the answer by Dennis Williamson, who suggests ionice as a workaround for the load, will work only if your block device uses the CFQ io scheduler.

share|improve this answer
add comment

In terms of efficiency, using one rm per file is not optimal, as it requires a fork and exec for each rm.

Assuming you have a list.txt containing the files you want to remove this would be more efficient but it's still gonna be slow:

xargs -i rm {} < list.txt

Another approach would be to : nice -20 xargs -i rm {} < list.txt
(this will take less time but will affect your system greatly :)

or

I don't know how fast this would be but:

mv <file-name> /dev/null 

or

Create a special mount point with a fast filesystem (using a loop device ?) , use that to store and delete your Huge files.
(maybe move the files there before you delete them, maybe it's faster or maybe just unmount it when you want files gone)

or

cat /dev/null > /file/to/be/deleted (so it's zero-sized now) and if you want it to disappear just rm -rf <file> now

or even better

drop the cat and just do # > /file/to/be/emptied

share|improve this answer
    
well, i'm removing 1 file, so there is no overhead. –  user13185 Mar 31 '10 at 9:38
    
stackoverflow.com/questions/1795370/… - check this too –  Jimmy Mar 31 '10 at 9:50
add comment

Upgrade to ext4 or some other modern filesystem that uses extents. Since ext3 uses the indirect blocks scheme rather than extents, deleting large files inevitably entails lots of work.

share|improve this answer
add comment

You can give ionice a try. It won't make it faster but it might make it less disruptive.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.