Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I generate more than 50GB of cache files on my RHEL server (and typical file size is 200kb so no of files is huge). When I try to delete these files it takes 8-10 hours.

However, the bigger issue is that the system load goes to critical for these 8-10 hours. Is there anyway where I can keep the system load under control during the deletion.

I tried using

nice -n19 rm -rf *

but that doesn't help in system load.

P.S. I asked the same question on superuser.com but didn't get a good enough answer so trying here.

share|improve this question
    
What kind of disk subsystem, what other load is on this filesystem at the same time? –  James Mar 29 '10 at 14:26
    
Its and ext3 filesystem. Lots of disk reads happen on it as it contains a file based cache. –  Sanjay Mar 30 '10 at 10:21
add comment

9 Answers 9

up vote 7 down vote accepted

Here are some benchmarks for various operations and filesystems for your reference. (On a busy system of course you would have different results, but hopefully this will give you an idea of what to expect).

If I would be in your chair I would try to get a baseline benchmark of the scenario:

  • establish how long the operation would take on bare hardware isolated of everything else (and yes, it should take much, much less then 7-8 hours even on pretty old hardware).
  • try to add other operations that typically occur in a controlled manner and see what actually makes it run so long

Some numbers.

On 5 year old notebook, ext3 mounted rw, noatime, running top and nothing much more create 10k directories with shell script create10kdirs.sh

#!/bin/bash
for i in $(seq 10000)
do
    mkdir $i
done

sudo time ./create10kdirs.sh
24.59user
20.70system
0:47.04elapsed
96%CPU (0avgtext+0avgdata 0maxresident)k80inputs+8outputs (1major+2735150minor)pagefaults 0swaps

delete 10k directories with sudo time rm -rf
0.10user
19.75system
0:20.71elapsed
95%CPU (0avgtext+0avgdata 0maxresident)k0inputs+8outputs (0major+222minor)pagefaults 0swaps

same hardware, ext4 mounted rw, noatime create 10k directories with shell script sudo time create10kdirs.sh
23.96user
22.31system
0:49.26elapsed
93%CPU (0avgtext+0avgdata0maxresident)k1896inputs+8outputs(20major+2715174minor)pagefaults 0swaps

delete 10k directories with sudo time rm -rf
0.13user
16.96system
0:28.21elapsed
60%CPU (0avgtext+0avgdata0maxresident)k10160inputs+0outputs(1major+219minor)pagefaults0swaps

4 year old notebook, xfs mounted rw,relatime,nobarrier on USB sudo time create10kdirs.sh
14.19user
13.86system
0:29.75elapsed
94%CPU (0avgtext+0avgdata0maxresident)k432inputs+0outputs(1major+2735243minor)pagefaults 0swaps

delete 10k directories with
sudo time rm -rf
0.13user
2.65system
0:08.20elapsed
33%CPU (0avgtext+0avgdata 0maxresident)k120inputs+0outputs (1major+222minor)pagefaults 0swaps

Conclusion: This old hardware would erase 400k small files+folders on ext3 in approx 21s * 40 = 12m40s. On xfs (with nobarriers) it would do it in approx 5m20s. Granted in both test cases the test machine was not under heavy load, but to me it seems that your problems are not strictly related to your choice of filesystem.

EDIT2 Also, after running above benchmarks I went to try the delete with find . -mindepth 1 -maxdepth 1 -delete

and the results!:

ext3 delete 10k directories with sudo time find . -mindepth 1 -maxdepth 1 -delete
0.04user
0.44system
0:00.88elapsed
55%CPU (0avgtext+0avgdata 0maxresident)k516inputs+8outputs(1major+688minor)pagefaults0swaps

ext4 delete 10k directories with
sudo time find . -mindepth 1 -maxdepth 1 -delete
0.05user
0.66system
0:01.02elapsed
70%CPU (0avgtext+0avgdata 0maxresident)k568inputs+0outputs (1major+689minor)pagefaults swaps

xfs delete 10k directories with
sudo time find . -mindepth 1 -maxdepth 1 -delete
0.06user
0.84system
0:04.55elapsed
19%CPU (0avgtext+0avgdata 0maxresident)k416inputs+0outputs (3major+685minor)pagefaults 0swaps

Real conclusion is that rm -rf is not very clever and that it'll under-perform for big trees. (providing that my test case is really representative).

Note: I also tested xargs variant and it is fast, but not as fast as the above.

share|improve this answer
    
As per the benchmarks link you mentioned, it seems like ext3 takes a real performance hit when deleting lots of directories. My cache has lots of files as well as directories. That may be one of the reasons why it is so slow. –  Sanjay Mar 29 '10 at 13:10
    
+1 explains the question with supporting data. –  Warner Mar 29 '10 at 15:29
add comment

As you mentioned in a comment, you are using ext3.

It is well known that rm performance for large files on ext3 is poor; it is one of the things which were fixed in ext4. See for instance this post, or kernelnewbies (which mentions extents improve delete and truncate speeds for large files).

I do not know how much that applies to your typical file sizes. I would it expect it to apply at least a little, since with around 200kB you would already be using indirect blocks on ext3, versus possibly a single extent on ext4.


As a workaround (since you probably will not upgrade to ext4 just for that), delete only a few files each time and add a sleep between the deletions. It is not pretty, but should help reducing the load.

Also, if losing the files on power loss is not a problem (since it is a cache of some sort), you could put them in a separate partition which you mkfs again on boot, and use ext3 without a journal or even ext2. The cause for the high load is probably the journal being flushed to disk conflicting with the reads (you mentioned in another post that you have lots of concurrent reads).

share|improve this answer
add comment

Maybe the shell is the cause of the problem. You should use directly find : find /dir -mindepth 1 -maxdepth 1 -delete

share|improve this answer
    
how did you know that this could be a reason for such high gain? –  Unreason Apr 8 '10 at 15:59
add comment

This may or may not be related: but I have had occasions where rm could not handle the number of files I provided it on the command line (through the star operator). Instead I would use the following command from the shell:

for i in *; do rm -rf $i; done

In this case you could be deleting trees in which case the above may not do what you need. You might have to split up the delete operation into parts, e.g.

for i in [a-mA-M]*; do rm -rf $i; done
for i in [n-zN-Z]*; do rm -rf $i; done
share|improve this answer
    
I do rm -rf <dir> and it works. My issue is with the time and CPU it takes. I need something which is less CPU intensive. –  Sanjay Mar 29 '10 at 10:03
    
My point is that rm might work quicker when it doesn't have to juggle so many files in memory. Dom's answer might be most appropriate. –  PP. Mar 29 '10 at 10:21
add comment

That's only 250,000 files or so, shouldn't be a problem really - what file system are you using and is this volume used for anything else?

share|improve this answer
    
Its an ext3 filesystem. Lots of disk reads are happening on this disk and it contains a file based cache. Are you saying the file deletions of this number should not take 7-8 hours? –  Sanjay Mar 29 '10 at 10:02
    
I am saying exactly that, ext3 should be much quicker than this, have you considered putting these files onto a dedicated file system and retesting? –  Chopper3 Mar 29 '10 at 10:58
    
Considering the data is taking so much time to delete, the time to tar and transfer to another disk would be huge too (Its a remote server). I don't think I have the resources to copy this data into another disk and check disk operations on that. –  Sanjay Mar 29 '10 at 14:04
    
Sorry, that's not what I meant, I meant creating a new FS and creating your new files on that new FS in the first place, not moving existing ones. –  Chopper3 Mar 29 '10 at 16:09
add comment

When you have a lot of files like you described you are invoking the command each time. Also you have to keep in mind on journaled FS you're dealing with buffer hits and metadata which can greatly impact process times.

Your best bet is to use the find command like mentioned above only with a less than obvious feature.

find / -name filename.* -exec /bin/rm -f '{}' \+

Basically the "+" is your friend. What this does is create the filenames in sets and invokes rm command once per set. That's pretty much the same as what 'xargs' does but you don't have to worry about the correct flags if on BSD/Linux.

Very curious how much that speeds it up for you. So reply back if you are still around. Good luck !

share|improve this answer
add comment

After studying filesystem benchmarks I've choosen JFS as file store for my mythtv video files because file deletes are fast (and mythtv waits for the delete to finish, making the IO sluggish).

You could also invoke 'rm' through 'find' and 'xargs' instead of rm -rf. This might be faster:

find <dir> | xargs rm
share|improve this answer
add comment

How about piping a list of those files to Perl and using its unlink function?

find <dir> | perl -nle 'unlink;'
share|improve this answer
add comment

I agree that it shouldn't take that long, but depending on the underlying storage being used, it may be expected with the intensive reads. I think ultimately your best solution is to add additional disks and split up your activities amongst them. RAID could help in some scenarios if you go down that route. What does iostat tell you during these times? You could also use a for loop and wrap the 'rm' commands in 'time' to get some additional information.

Another possibility, depending on your setup and the application of course, but maybe you could make these cache files on a different partition and just format the drive periodically rather than remove the files? I think running mkfs would cut the time down considerably, but your application is going to be unavailable while that happens so it won't be ideal.

I also like the idea of cleaning them up more often. Say in cron you schedule something like this every hour:

find ./ -maxdepth 1 -type f -name "some pattern" -ctime +1 -exec rm -f {} \;

This would delete every file that is older than 24 hours, rather then trying to do them all at once.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.