11

I'd like to delete an nginx cache directory, which I quickly purged by:

mv cache cache.bak
mkdir cache
service nginx restart

Now I have a cache.bak folder which has 2 million files. I'd like to delete it, without disturbing the server.

A simple rm -rf cache.bak trashes the server, even the simplest HTTP response takes 16 seconds while rm is running, so I cannot do that.

I tried ionice -c3 rm -rf cache.bak, but it didn't help. The server has an HDD, not an SSD, probably on an SSD these might not be a problem.

I believe the best solution would be some kind of throttling, like how nginx's built in cache manager does.

How would you solve this? Is there any tool which can do exactly this?

ext4 on Ubuntu 16.04

  • 1
    How did you recover from "rm -rf cache.bak"? It appears that nginx was running when you did the rename, so it might have maintained file descriptors and even switched to the new directory. I think you need to shut nginx down completely, delete the cache, then re-start. – Jan Steinman Dec 24 '16 at 2:29
  • 6
    In the future, please stick your cache on a separate filesystem. That way you can simply nuke that filesystem, which is much faster than trying to delete millions of files. Learned this the hard way a few years ago with a hylafax spool directory containing zillions of files. – Dennis Kaarsemaker Dec 24 '16 at 8:54
  • Have you tried to run rm using nice? – Vladislav Rastrusny Dec 27 '16 at 10:08
  • Try rsync to fast delete - answers to a similar case - unix.stackexchange.com/questions/37329/… – kawu Dec 29 '16 at 23:17
  • Thanks for all the comments, I've summarised my findings to writeup answer. – hyperknot Dec 30 '16 at 0:16
9

Make a bash script like this:

#!/bin/bash
rm -- "$*"
sleep 0.5

Save it with name deleter.sh for example. Run chmod u+x deleter.sh to make it executable.

This script deletes all files passed to it as arguments, and then sleeps 0.5 seconds.

Then, you can run

find cache.bak -print0 | xargs -0 -n 5 deleter.sh

This command retrieves a list of all files in cache.bak and passes the five filenames at a time to the delete script.

So, you can adjust how many files are deleted at a time, and how long a delay is between each delete operation.

  • Thanks for this solution, I've included it in my overall writeup. One question though, how is this handling large n-s? I usually had problems with * character in large directories giving errors, isn't it the case here? – hyperknot Dec 30 '16 at 0:15
  • xargs understands the maximum size of a command line and tries not to exceed it by default. This one has additional limits of no more than 5 paths at a time. – BowlOfRed Dec 30 '16 at 0:23
  • 1
    Just be aware that at the rate of 10 files per second, it will take 55 hours to delete 2 million files. – Andrew Henle Dec 30 '16 at 0:43
4

You should consider saving your cache on a separate filesystem that you can mount/unmount as someone stated in comments. Until you do, you can use this one liner /usr/bin/find /path/to/files/ -type f -print0 -exec sleep 0.2 \; -exec echo \; -delete assuming your find binary is located under /usr/bin and you want to see the progress on screen. Adjust the sleep accordingly, so you don't over stress your HDD.

  • One doesn't need -print0 here, since you are not piping the output of find anywhere. – Tero Kilkanen Dec 30 '16 at 10:46
  • You just might be interested in what's rm-ing. Call it paranoia, but I always want to be sure I'm deleting the right files. – Alex Dec 30 '16 at 10:52
  • Ah true, I wasn't decoding the command correctly, my bad. – Tero Kilkanen Dec 30 '16 at 11:20
3

You may want to try ionice on a script consuming a the output of a find command. Something like the following:

ionice -c3 $(
for file in find cache.bak -type f; do
    rm $file
done
for dir in find cache.bak -depthe -type d -empty; do
    rmdir $dir
done
)

Depending on the filesystem each file delete may result in rewriting that entire directory. For large directories that can be quite a hit. There are additional updates required to the inode table, and possibly a free space list.

If the file system has a journal, changes are written to the journal; applied; and removed from the journal. This increases I/O requirements for write intensive activity.

You may want to use a filesystem without a journal for the cache.

Instead of ionice, you can use a sleep command to rate limit the actions. This will work even if ionice does not, but it will take a long time to delete all your files.

2

I got many useful answers / comments here, which I'd like to conclude as well as show my solution as well.

  1. Yes, the best way to prevent such thing happening is to keep the cache dir on a separate filesystem. Nuking / quick formatting a file system always takes a few seconds (maybe minutes) at most, unrelated to how many files / dirs were present on it.

  2. The ionice / nice solutions didn't do anything, because the deleting process actually caused almost no I/O. What caused the I/O was I believe kernel / filesystem level queues / buffers filling up when files were deleted too quickly by the delete process.

  3. The way I solved it is similar to Tero Kilkanen's solution, but didn't require calling a shell script. I used rsync's built in --bwlimit switch to limit the speed of deleting.

Full command was:

mkdir empty_dir
rsync -v -a --delete --bwlimit=1 empty_dir/ cache.bak/

Now bwlimit specifies bandwidth in kilobyes, which in this case applied to the filename or path of the files. By setting it to 1 KBps, it was deleting around 100,000 files per hour, or 27 files per second. Files had relative paths like cache.bak/e/c1/db98339573acc5c76bdac4a601f9ec1e, which is 47 characters long, so it would give 1000/47 ~= 21 files per second, so kind of similar to my guess of 100,000 files per hour.

Now why --bwlimit=1? I tried various values:

  • 10000, 1000, 100 -> system slowing down like before
  • 10 -> system working quite well for a while, but produces partial slowdowns once a minute or so. HTTP response times still < 1 sec.
  • 1 -> no system slowdown at all. I'm not in a hurry and 2 million files can be deleted in < 1 day this way, so I choose it.

I like the simplicity of rsync's built in method, but this solution depends on the relative path's length. Not a big problem as most people would find the right value via trial and error.

  • And now I'm curious as to what the disk effect would be if you did something like "mv cache.dir-old /dev/null" – ivanivan Dec 30 '16 at 3:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.