Background: physical server, about two years old, 7200-RPM SATA drives connected to a 3Ware RAID card, ext3 FS mounted noatime and data=ordered, not under crazy load, kernel 2.6.18-92.1.22.el5, uptime 545 days. Directory doesn't contain any subdirectories, just millions of small (~100 byte) files, with some larger (a few KB) ones.

We have a server that has gone a bit cuckoo over the course of the last few months, but we only noticed it the other day when it started being unable to write to a directory due to it containing too many files. Specifically, it started throwing this error in /var/log/messages:

ext3_dx_add_entry: Directory index full!

The disk in question has plenty of inodes remaining:

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda3            60719104 3465660 57253444    6% /

So I'm guessing that means we hit the limit of how many entries can be in the directory file itself. No idea how many files that would be, but it can't be more, as you can see, than three million or so. Not that that's good, mind you! But that's part one of my question: exactly what is that upper limit? Is it tunable? Before I get yelled at—I want to tune it down; this enormous directory caused all sorts of issues.

Anyway, we tracked down the issue in the code that was generating all of those files, and we've corrected it. Now I'm stuck with deleting the directory.

A few options here:

  1. rm -rf (dir)

    I tried this first. I gave up and killed it after it had run for a day and a half without any discernible impact.

  2. unlink(2) on the directory: Definitely worth consideration, but the question is whether it'd be faster to delete the files inside the directory via fsck than to delete via unlink(2). That is, one way or another, I've got to mark those inodes as unused. This assumes, of course, that I can tell fsck not to drop entries to the files in /lost+found; otherwise, I've just moved my problem. In addition to all the other concerns, after reading about this a bit more, it turns out I'd probably have to call some internal FS functions, as none of the unlink(2) variants I can find would allow me to just blithely delete a directory with entries in it. Pooh.
  3. while [ true ]; do ls -Uf | head -n 10000 | xargs rm -f 2>/dev/null; done )

    This is actually the shortened version; the real one I'm running, which just adds some progress-reporting and a clean stop when we run out of files to delete, is:

    export i=0;
    time ( while [ true ]; do
      ls -Uf | head -n 3 | grep -qF '.png' || break;
      ls -Uf | head -n 10000 | xargs rm -f 2>/dev/null;
      export i=$(($i+10000));
      echo "$i...";
    done )

    This seems to be working rather well. As I write this, it has deleted 260,000 files in the past thirty minutes or so.

Now, for the questions:
  1. As mentioned above, is the per-directory entry limit tunable?
  2. Why did it take "real 7m9.561s / user 0m0.001s / sys 0m0.001s" to delete a single file which was the first one in the list returned by ls -U, and it took perhaps ten minutes to delete the first 10,000 entries with the command in #3, but now it's hauling along quite happily? For that matter, it deleted 260,000 in about thirty minutes, but it's now taken another fifteen minutes to delete 60,000 more. Why the huge swings in speed?
  3. Is there a better way to do this sort of thing? Not store millions of files in a directory; I know that's silly, and it wouldn't have happened on my watch. Googling the problem and looking through SF and SO offers a lot of variations on find that are not going to be significantly faster than my approach for several self-evident reasons. But does the delete-via-fsck idea have any legs? Or something else entirely? I'm eager to hear out-of-the-box (or inside-the-not-well-known-box) thinking.
Thanks for reading the small novel; feel free to ask questions and I'll be sure to respond. I'll also update the question with the final number of files and how long the delete script ran once I have that.

Final script output!:


real    253m59.331s
user    0m6.061s
sys     5m4.019s

So, three million files deleted in a bit over four hours.

  • 1
    rm (GNU coreutils) 8.4 has this option: "-v, --verbose explain what is being done". It will display all the files that are being deleted. – Cristian Ciupitu Sep 27 '10 at 18:34
  • 2
    Actually, that'd be a neat way to do a progress bar: since each file would be thirty-seven characters long (36 + a '\n'), I could easily write a parser for that, and since printf() is cheap and the rm command already has the name of the file loaded, there's no especial performance penalty. Seems like a non-starter for doing the whole shebang, since I could never get "rm" to do anything like that, anyway. But it could work quite well as an intra-10,000 progress bar; perhaps a "." for every hundred files? – BMDan Sep 27 '10 at 21:40
  • 8
    rm -rfv | pv -l >/dev/null. pv should be available in the EPEL repository. – Cristian Ciupitu Sep 27 '10 at 23:04
  • 5
    pv is overwhelmingly awesome. I leave a trail of pv installations in my wake. – BMDan Sep 28 '10 at 1:54
  • I had this exact same issue recently. Thankyou! – richo Dec 23 '10 at 23:26

24 Answers 24


The data=writeback mount option deserves to be tried, in order to prevent journaling of the file system. This should be done only during the deletion time, there is a risk however if the server is being shutdown or rebooted during the delete operation.

According to this page,

Some applications show very significant speed improvement when it is used. For example, speed improvements can be seen (...) when applications create and delete large volumes of small files.

The option is set either in fstab or during the mount operation, replacing data=ordered with data=writeback. The file system containing the files to be deleted has to be remounted.

  • 1
    He could also increase the time from the commit option: "This default value (or any low value) will hurt performance, but it's good for data-safety. Setting it to 0 will have the same effect as leaving it at the default (5 seconds). Setting it to very large values will improve performance". – Cristian Ciupitu Sep 26 '10 at 19:14
  • 1
    Writeback looks stellar, except the documentation I was looking at (gentoo.org/doc/en/articles/l-afig-p8.xml#doc_chap4) explicitly mentions that it still journals metadata, which I presume includes all the data I'm changing (I'm certainly not changing any data in the files themselves). Is my understanding of the option incorrect? – BMDan Sep 27 '10 at 1:20
  • Lastly, FYI, not mentioned in that link is that fact that data=writeback can be a huge security hole, since data pointed to by a given entry may not have the data that was written there by the app, meaning that a crash could result in the old, possibly-sensitive/private data being exposed. Not a concern here, since we're only turning it on temporarily, but I wanted to alert everyone to that caveat in case either you or others who run across that suggestion weren't aware. – BMDan Sep 27 '10 at 1:23
  • commit: that's pretty slick! Thanks for the pointer. – BMDan Sep 27 '10 at 1:26
  • 2
    data=writeback still journals metadata before writing it into the main filesystem. As I understand it, it just doesn't enforce ordering between things like writing an extent map and writing data into those extents. Maybe there are other ordering constraints it relaxes, too, if you saw a perf gain from this. Of course, mounting without the journal at all could be even higher performance. (It might let the metadata changes just happen in RAM, without needing to have anything on disk before the unlink op completes). – Peter Cordes Mar 17 '15 at 11:30

Whilst a major cause of this problem is ext3 performance with millions of files, the actual root cause of this problem is different.

When a directory needs to be listed readdir() is called on the directory which yields a list of files. readdir is a posix call, but the real Linux system call being used here is called 'getdents'. Getdents list directory entries by filling a buffer with entries.

The problem is mainly down to the fact that that readdir() uses a fixed buffer size of 32Kb to fetch files. As a directory gets larger and larger (the size increases as files are added) ext3 gets slower and slower to fetch entries and additional readdir's 32Kb buffer size is only sufficient to include a fraction of the entries in the directory. This causes readdir to loop over and over and invoke the expensive system call over and over.

For example, on a test directory I created with over 2.6 million files inside, running "ls -1|wc-l" shows a large strace output of many getdent system calls.

$ strace ls -1 | wc -l
brk(0x4949000)                          = 0x4949000
getdents(3, /* 1025 entries */, 32768)  = 32752
getdents(3, /* 1024 entries */, 32768)  = 32752
getdents(3, /* 1025 entries */, 32768)  = 32760
getdents(3, /* 1025 entries */, 32768)  = 32768
brk(0)                                  = 0x4949000
brk(0x496a000)                          = 0x496a000
getdents(3, /* 1024 entries */, 32768)  = 32752
getdents(3, /* 1026 entries */, 32768)  = 32760

Additionally the time spent in this directory was significant.

$ time ls -1 | wc -l

real    0m20.609s
user    0m16.241s
sys 0m3.639s

The method to make this a more efficient process is to call getdents manually with a much larger buffer. This improves performance significantly.

Now, you're not supposed to call getdents yourself manually so no interface exists to use it normally (check the man page for getdents to see!), however you can call it manually and make your system call invocation way more efficient.

This drastically reduces the time it takes to fetch these files. I wrote a program that does this.

/* I can be compiled with the command "gcc -o dentls dentls.c" */

#define _GNU_SOURCE

#include <dirent.h>     /* Defines DT_* constants */
#include <err.h>
#include <fcntl.h>
#include <getopt.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <unistd.h>

struct linux_dirent {
        long           d_ino;
        off_t          d_off;
        unsigned short d_reclen;
        char           d_name[256];
        char           d_type;

static int delete = 0;
char *path = NULL;

static void parse_config(
        int argc,
        char **argv)
    int option_idx = 0;
    static struct option loptions[] = {
      { "delete", no_argument, &delete, 1 },
      { "help", no_argument, NULL, 'h' },
      { 0, 0, 0, 0 }

    while (1) {
        int c = getopt_long(argc, argv, "h", loptions, &option_idx);
        if (c < 0)

        switch(c) {
          case 0: {

          case 'h': {
              printf("Usage: %s [--delete] DIRECTORY\n"
                     "List/Delete files in DIRECTORY.\n"
                     "Example %s --delete /var/spool/postfix/deferred\n",
                     argv[0], argv[0]);


    if (optind >= argc)
      errx(EXIT_FAILURE, "Must supply a valid directory\n");

    path = argv[optind];

int main(
    int argc,
    char** argv)

    parse_config(argc, argv);

    int totalfiles = 0;
    int dirfd = -1;
    int offset = 0;
    int bufcount = 0;
    void *buffer = NULL;
    char *d_type;
    struct linux_dirent *dent = NULL;
    struct stat dstat;

    /* Standard sanity checking stuff */
    if (access(path, R_OK) < 0) 
        err(EXIT_FAILURE, "Could not access directory");

    if (lstat(path, &dstat) < 0) 
        err(EXIT_FAILURE, "Unable to lstat path");

    if (!S_ISDIR(dstat.st_mode))
        errx(EXIT_FAILURE, "The path %s is not a directory.\n", path);

    /* Allocate a buffer of equal size to the directory to store dents */
    if ((buffer = calloc(dstat.st_size*3, 1)) == NULL)
        err(EXIT_FAILURE, "Buffer allocation failure");

    /* Open the directory */
    if ((dirfd = open(path, O_RDONLY)) < 0) 
        err(EXIT_FAILURE, "Open error");

    /* Switch directories */

    if (delete) {
        printf("Deleting files in ");
        for (int i=5; i > 0; i--) {
            printf("%u. . . ", i);

    while (bufcount = syscall(SYS_getdents, dirfd, buffer, dstat.st_size*3)) {
        offset = 0;
        dent = buffer;
        while (offset < bufcount) {
            /* Don't print thisdir and parent dir */
            if (!((strcmp(".",dent->d_name) == 0) || (strcmp("..",dent->d_name) == 0))) {
                d_type = (char *)dent + dent->d_reclen-1;
                /* Only print files */
                if (*d_type == DT_REG) {
                    printf ("%s\n", dent->d_name);
                    if (delete) {
                        if (unlink(dent->d_name) < 0)
                            warn("Cannot delete file \"%s\"", dent->d_name);
            offset += dent->d_reclen;
            dent = buffer + offset;
    fprintf(stderr, "Total files: %d\n", totalfiles);


Whilst this does not combat the underlying fundamental problem (lots of files, in a filesystem that performs poorly at it). It's likely to be much, much faster than many of the alternatives being posted.

As a forethought, one should remove the affected directory and remake it after. Directories only ever increase in size and can remain poorly performing even with a few files inside due to the size of the directory.

Edit: I've cleaned this up quite a bit. Added an option to allow you to delete on the command line at runtime and removed a bunch of the treewalk stuff which, honestly looking back was questionable at best. Also was shown to produce memory corruption.

You can now do dentls --delete /my/path

New results. Based off of a directory with 1.82 million files.

## Ideal ls Uncached
$ time ls -u1 data >/dev/null

real    0m44.948s
user    0m1.737s
sys 0m22.000s

## Ideal ls Cached
$ time ls -u1 data >/dev/null

real    0m46.012s
user    0m1.746s
sys 0m21.805s

### dentls uncached
$ time ./dentls data >/dev/null
Total files: 1819292

real    0m1.608s
user    0m0.059s
sys 0m0.791s

## dentls cached
$ time ./dentls data >/dev/null
Total files: 1819292

real    0m0.771s
user    0m0.057s
sys 0m0.711s

Was kind of surprised this still works so well!

  • 1
    Two minor concerns: one, [256] should probably be [FILENAME_MAX], and two, my Linux (2.6.18==CentOS 5.x) doesn't seem to include a d_type entry in dirent (at least according to getdents(2)). – BMDan Nov 7 '11 at 5:01
  • 1
    Could you please elaborate a bit on btree rebalancing and why deletion in order helps preventing it? I tried Googling for it, unfortunately to no avail. – ovgolovin Jul 28 '13 at 23:28
  • 1
    Because now it seems to me if we are deleting in-order, we force rebalancing, as we remove leaves on one side and leave on the other: en.wikipedia.org/wiki/B-tree#Rebalancing_after_deletion – ovgolovin Jul 28 '13 at 23:37
  • 1
    I hope I don't bother you with this matters. But still I started a question about deleting files in-order stackoverflow.com/q/17955459/862380, which seems not to receive an answer which will explain the issue with the example, which will be understandable for ordinary programmers. If you have time and feel like so, could you look into it? Maybe you could write a better explanation. – ovgolovin Aug 8 '13 at 18:35
  • 4
    This is an amazing piece of code. It was the only tool I could find able to list and delete some 11,000,000 (eleven million) session files that had built up in a directory, probably over some years. The Plesk process that was supposed to keep them under control using find and other tricks in other answers here, was unable to complete a run, so the files just kept building up. It is a tribute to the binary tree that the filesystem uses to store the directory, that the sessions were able to work at all - you could create a file and retrieve it with no delay. Just listings were unusable. – Jason Feb 12 '16 at 15:10

Would it be possible to backup all of the other files from this file system to a temporary storage location, reformat the partition, and then restore the files?

  • 3
    I really like this answer, actually. As a practical matter, in this case, no, but it's not one I would have thought of. Bravo! – BMDan Sep 23 '10 at 10:23
  • Exactly what I was thinking too. This is an answer for question 3. Ideal if you ask me :) – Joshua Oct 1 '10 at 15:31

There is no per directory file limit in ext3 just the filesystem inode limit (i think there is a limit on the number of subdirectories though).

You may still have problems after removing the files.

When a directory has millions of files, the directory entry itself becomes very large. The directory entry has to be scanned for every remove operation, and that takes various amounts of time for each file, depending on where its entry is located. Unfortunately even after all the files have been removed the directory entry retains its size. So further operations that require scanning the directory entry will still take a long time even if the directory is now empty. The only way to solve that problem is to rename the directory, create a new one with the old name, and transfer any remaining files to the new one. Then delete the renamed one.

  • Indeed, I noticed just this behavior after deleting everything. Luckily, we had already mv'd the directory out of the "line of fire", as it were, so I could just rmdir it. – BMDan Sep 23 '10 at 10:24
  • 2
    That said, if there's no per-directory file limit, why did I get "ext3_dx_add_entry: Directory index full!" when there were still inodes available on that partition? There were no subdirectories inside this directory. – BMDan Sep 23 '10 at 10:26
  • 3
    hmm i did a little more research and it seems there's a limit of number of blocks a directory can take up. The exact number of files is dependent on a few things eg filename length. This gossamer-threads.com/lists/linux/kernel/921942 seems to indicate that with 4k blocks you should be able to have more than 8 million files in a directory. Were they particularly long filenames? – Alex J. Roberts Sep 23 '10 at 12:04
  • Each filename was exactly 36 characters long. – BMDan Sep 23 '10 at 15:34
  • well that's me out of ideas :) – Alex J. Roberts Sep 24 '10 at 0:13

I haven't benchmarked it, but this guy did:

rsync -a --delete ./emptyDirectoty/ ./hugeDirectory/

TLDR: use rsync -a --delete emptyfolder/ x.

This question has 50k views, and quite a few answers, but nobody seems to have benchmarked all the different replies. There's one link to an external benchmark, but that one's over 7 years old and didn't look at the program provided in this answer: https://serverfault.com/a/328305/565293

Part of the difficulty here is that the time it takes to remove a file depends heavily on the disks in use and the file system. In my case, I tested both with a consumer SSD running BTRFS on Arch Linux (updated as of 2020-03), but I got the same ordering of the results on a different distribution (Ubuntu 18.04), filesystem (ZFS), and drive type (HDD in a RAID10 configuration).

Test setup was identical for each run:

# setup
mkdir test && cd test && mkdir empty
# create 800000 files in a folder called x
mkdir x && cd x
seq 800000 | xargs touch
cd ..

Test results:

rm -rf x: 30.43s

find x/ -type f -delete: 29.79

perl -e 'for(<*>){((stat)[9]<(unlink))}': 37.97s

rsync -a --delete empty/ x: 25.11s

(The following is the program from this answer, but modified to not print anything or wait before it deletes files.)

./dentls --delete x: 29.74

The rsync version proved to be the winner every time I repeated the test, although by a pretty low margin. The perl command was slower than any other option on my systems.

Somewhat shockingly, the program from the top answer to this question proved to be no faster on my systems than a simple rm -rf. Let's dig into why that is.

First of all, the answer claims that the problem is that rm is using readdir with a fixed buffer size of 32Kb with getdents. This proved not to be the case on my Ubuntu 18.04 system, which used a buffer four times larger. On the Arch Linux system, it was using getdents64.

In addition, the answer misleadingly provides statistics giving its speed at listing the files in a large directory, but not removing them (which is what the question was about). It compares dentls to ls -u1, but a simple strace reveals that getdents is not the reason why ls -u1 is slow, at least not on my system (Ubuntu 18.04 with 1000000 files in a directory):

strace -c ls -u1 x >/dev/null
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 94.00    7.177356           7   1000000           lstat
  5.96    0.454913        1857       245           getdents

This ls command makes a million calls to lstat, which slows the program way down. The getdents calls only add up to 0.455 seconds. How long do the getdents calls take in dentls on the same folder?

strace -c ./dentls x >/dev/null
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 99.91    0.489895       40825        12           getdents

That's right! Even though dentls only makes 12 calls instead of 245, it actually takes the system longer to run these calls. So the explanation given in that answer is actually incorrect - at least for the two systems I've been able to test this on.

The same applies to rm and dentls --delete. Whereas rm takes 0.42s calling getdents, dentls takes 0.53s. In either case, the vast majority of the time is spent calling unlink!

So in short, don't expect to see massive speedups running dentls, unless your system is like the author's and has a lot of overhead on individual getdents. Maybe the glibc folks have considerably sped it up in the years since the answer was written, and it now takes a linear about of time to respond for different buffer sizes. Or maybe the response time of getdents depends on the system architecture in some way that isn't obvious.

  • 1
    I noticed the use of -u, and I suspect that should be -U (which is the opposite, -U means to not sort on per-file properties, so that lstat can be avoided). I've asked on the other answer just now whether @Matthew Ife used -u accidentally. – nh2 May 26 '20 at 22:37
  • Could you strace the rsync approach and check which syscalls it makes, to figure out why it's faster than dentls here? – nh2 May 26 '20 at 22:39
  • Never saw this but quite honestly, really good work. My original answer is nearly 10 years old. Thats an EON in computing terms. At the VERY LEAST ext4 and extents really REALLY speeds up unlink() for large files and I'm sure stuff like block preallocation helps unlink run faster on larger directories. As for the getdents(), there is a overhead but on modern day block devices its negligable. In fact I'm not even sure on modern filesystem the problem of 'millions of files in a directory' has the same impact as it used to have. – Matthew Ife May 27 '20 at 15:19
  • 1
    As @MatthewIfe said, the circumstances have changed quite a bit from when I originally posed this question. I also had the opportunity to spend some time with a coreutils maintainer, who had some very cogent points about ways in which they had dramatically improved the performance of rm -r in the time since the release of the downright Paleolithic version included with CentOS 5.x. Moral of the story is: if you're on a modern system, you can probably just run rm -rf. Otherwise, feel free to run the updated version of dentls on Github, but @nh2 should totally accept my PR on it. :D – BMDan Nov 13 '20 at 22:26

find simply did not work for me, even after changing the ext3 fs's parameters as suggested by the users above. Consumed way too much memory. This PHP script did the trick - fast, insignificant CPU usage, insignificant memory usage:

$dir = '/directory/in/question';
$dh = opendir($dir)) { 
while (($file = readdir($dh)) !== false) { 
    unlink($dir . '/' . $file); 

I posted a bug report regarding this trouble with find: http://savannah.gnu.org/bugs/?31961

  • This saved me!! – jestro Mar 20 '11 at 10:12

I recently faced a similar issue and was unable to get ring0's data=writeback suggestion to work (possibly due to the fact that the files are on my main partition). While researching workarounds I stumbled upon this:

tune2fs -O ^has_journal <device>

This will turn off journaling completely, regardless of the data option give to mount. I combined this with noatime and the volume had dir_index set, and it seemed to work pretty well. The delete actually finished without me needing to kill it, my system remained responsive, and it's now back up and running (with journaling back on) with no issues.

  • I was going to suggest mounting it as ext2 instead of ext3, to avoid journaling the metadata ops. This should do the same. – Peter Cordes Mar 17 '15 at 9:26

Make sure you do:

mount -o remount,rw,noatime,nodiratime /mountpoint

which should speed things up a bit as well.

  • 4
    Good call, but it's already mounted noatime, as I mentioned in the header to the question. And nodiratime is redundant; see lwn.net/Articles/245002 . – BMDan Sep 27 '10 at 21:19
  • 1
    ppl repeat this mantra "noatime,nodiratime,nodevatime,noreadingdocsatime" – poige Jun 12 '16 at 6:11

ls very slow command. Try:

find /dir_to_delete ! -iname "*.png" -type f -delete
  • rm -rf ran for a day and a half, and I finally killed it, without ever knowing if it had actually accomplished anything. I needed a progress bar. – BMDan Sep 23 '10 at 10:18
  • 4
    As to rm being very slow, "time find . -delete" on 30k files: 0m0.357s / 0m0.019s / 0m0.337s real/user/sys. "time ( ls -1U | xargs rm -f )" on those same files: 0m0.366s / 0m0.025s / 0m0.340s. Which is basically margin-of-error territory. – BMDan Sep 23 '10 at 10:20
  • 1
    You could have just run strace -r -p <pid of rm> to attach to the already-running rm process. Then you can see how fast unlink system calls are scrolling past. (-r puts the time since the previous system call at the start of every line.) – Peter Cordes Mar 17 '15 at 9:14

Is dir_index set for the filesystem? (tune2fs -l | grep dir_index) If not, enable it. It's usually on for new RHEL.

  • 1
    Yes, it is enabled, but awesome suggestion! – BMDan Sep 27 '10 at 21:23

A couple of years back, I found a directory with 16 million XML files in the / filesystem. Due the criticity of the server, we used the following command that took about 30 hours to finish:

perl -e 'for(<*>){((stat)[9]<(unlink))}'

It was an old 7200 rpm hdd, and despite the IO bottleneck and CPU spikes, the old webserver continued its service.


My preferred option is the newfs approach, already suggested. The basic problem is, again as already noted, the linear scan to handle deletion is problematic.

rm -rf should be near optimal for a local filesystem (NFS would be different). But at millions of files, 36 bytes per filename and 4 per inode (a guess, not checking value for ext3), that's 40 * millions, to be kept in RAM just for the directory.

At a guess, you're thrashing the filesystem metadata cache memory in Linux, so that blocks for one page of the directory file are being expunged while you're still using another part, only to hit that page of the cache again when the next file is deleted. Linux performance tuning isn't my area, but /proc/sys/{vm,fs}/ probably contain something relevant.

If you can afford downtime, you might consider turning on the dir_index feature. It switches the directory index from linear to something far more optimal for deletion in large directories (hashed b-trees). tune2fs -O dir_index ... followed by e2fsck -D would work. However, while I'm confident this would help before there are problems, I don't know how the conversion (e2fsck with the -D) performs when dealing with an existing v.large directory. Backups + suck-it-and-see.

  • 2
    pubbs.net/201008/squid/… suggests that /proc/sys/fs/vfs_cache_pressure might be the value to use, but I don't know whether the directory itself counts towards page cache (because that's what it is) or the inode cache (because, despite not being an inode, it's FS metadata and bundled into there for that reason). As I say, Linux VM tuning is not my area. Play and see what helps. – Phil P Sep 26 '10 at 12:10

Obviously not apples to apples here, but I setup a little test and did the following:

Created 100,000 512-byte files in a directory (dd and /dev/urandom in a loop); forgot to time it, but it took roughly 15 minutes to create those files.

Ran the following to delete said files:

ls -1 | wc -l && time find . -type f -delete


real    0m4.208s
user    0m0.270s
sys     0m3.930s 

This is a Pentium 4 2.8GHz box (couple hundred GB IDE 7200 RPM I think; EXT3). Kernel 2.6.27.

  • Interesting, so perhaps the fact that the files were being created over a long period of time is relevant? But that shouldn't matter; the block cache should have all of the pertinent metadata blocks in RAM. Maybe it's because unlink(2) is transactional? In your estimation, would turning off journaling for the duration of the rm be a potential (albeit admittedly somewhat dangerous) solution? It doesn't look like you can just turn off journaling entirely on a mounted filesystem without a tune2fs/fsck/reboot, which somewhat defeats the purpose. – BMDan Sep 27 '10 at 1:11
  • I can't comment on that, but anecdotally (in various NIX discussions over the years), I've always heard that rm is horribly slow on a large number of files, hence the find -delete option. With a wildcard on the shell, it'll expand each filename matched, and I'm assuming there's a limited memory buffer for that, so you could see how that would get inefficient. – gravyface Sep 28 '10 at 12:58
  • 1
    rm would be slow because it's looking for a file by name, which means iterating through the directory entries one by one until it finds it. In this case, however, since each entry it's being handed is (at that point) the first one in the list (ls -U/ls -f), it should be nearly as fast. That said, rm -rf <dir>, which should have run like a champ, was slow as could be. Perhaps it's time to write a patch to coreutils to speed massive deletes? Maybe it's secretly globbing/sorting in some recursive way in order to implement rm -rf? Uncertainties like this are why I asked the question. ;) – BMDan Sep 30 '10 at 3:11
  • 1
    Reboot the machine after you run the creation step. You should get a noticeably slower delete. – hookenz Apr 23 '12 at 23:09

Sometimes Perl can work wonders in cases like this. Have you already tried if a small script such as this could outperform bash and the basic shell commands?

@files = grep("/*\.png/", readdir(ANNOYINGDIR));

for (@files) {
    printf "Deleting %s\n",$_;
    unlink $_;

Or another, perhaps even faster, Perl approach:

unlink(glob("/path/to/your/directory/*.png")) or die("Could not delete files, this happened: $!");

EDIT: I just gave my Perl scripts a try. The more verbose one does something right. In my case I tried this with a virtual server with 256 MB RAM and half a million files.

time find /test/directory | xargs rm results:

real    2m27.631s
user    0m1.088s
sys     0m13.229s

compared to

time perl -e 'opendir(FOO,"./"); @files = readdir(FOO); closedir(FOO); for (@files) { unlink $_; }'

real    0m59.042s
user    0m0.888s
sys     0m18.737s
  • I hesitate to imagine what that glob() call would do; I assume it does a scandir(). If so, that's going to take FOREVER to return. A modification of the first suggestion that doesn't pre-read all the dir entries might have some legs; however, in its current form, it, too, would use an unholy amount of CPU on just reading all the directory entries at once. Part of the goal here is to divide-and-conquer; this code isn't fundamentally different from 'rm -f *.png', notwithstanding issues with shell expansion. If it helps, there's nothing in the directory that I didn't want to delete. – BMDan Sep 27 '10 at 21:30
  • I gotta try more as soon as I get to work. I just tried to create 100 000 files in a single directory and find + xargs + rm combination took 7.3 seconds, Perl + unlink(glob)... combination finished in 2.7 seconds. Tried that couple of times, the result was always the same. At work I'll try this with more files. – Janne Pikkarainen Sep 28 '10 at 5:57
  • I learnt something new while testing this. At least with ext3 and ext4 the directory entry itself remains huge even after deleting all the files from there. After couple of tests my /tmp/test directory was taking 15 MB of disk space. Is there other way to clean that up other than removing the directory and recreating it? – Janne Pikkarainen Sep 28 '10 at 12:53
  • 2
    No, you need to recreate it. I hit this when dealing with a mail-system and folder-per-recipient and cleanups after significant issues: there's no way, other than creating a new directory and shuffling the directories about, then nuking the old one. So you can reduce the time window when there's no directory, but not eliminate it. – Phil P Sep 28 '10 at 23:49
  • Note that glob() will sort the results, much as shell globbing normally does, so because you only have 100k files, everything fits easily and the sort is fast. With a much larger directory, you'd want to opendir()/readdir()/closedir() just to avoid the sort. [I say normally for shell, since zsh has a glob modifier to make the sort order unsorted, which is useful when dealing with large numbers of files; *(oN) ] – Phil P Sep 28 '10 at 23:50

From what I remember the deletion of inodes in ext filesystems is O(n^2), so the more files you delete the faster the rest will go.

There was a one time I was faced with similar problem (though my estimates looked at ~7h deletion time), in the end went the jftuga suggested route in first comment.


Well, this is not a real answer, but...

Would it be possible to convert the filesystem to ext4 and see if things change?

  • It appears that doing this "live" requires an fsck on a mounted filesystem, which is... alarming. Got a better way? – BMDan Sep 27 '10 at 21:25
  • The filesystem has to be unmounted before the conversion, i.e. before the necessary tunefs commands. – marcoc Sep 28 '10 at 7:00

Alright this has been covered in various ways in the rest of the thread but I thought I would throw in my two cents. The performance culprit in your case is probably readdir. You are getting back a list of files that are not necessarily in any way sequential on disk which is causing disk access all over the place when you unlink. The files are small enough that the unlink operation probably doesn't jump around too much zeroing out the space. If you readdir and then sort by ascending inode you would probably get better performance. So readdir into ram (sort by inode) -> unlink -> profit.

Inode is a rough approximation here I think .. but basing on your use case it might be fairly accurate...

  • 1
    Correct me if I'm wrong, but unlink(2) doesn't zero the inode, it just removes the reference to it from the directory. I like the chutzpah of this approach, though. Care to run some time-trials and see if it holds true? – BMDan Sep 30 '10 at 3:07

I would probably have whipped out a C compiler and done the moral equivalent of your script. That is, use opendir(3) to get a directory handle, then use readdir(3) to get the name of files, then tally up files as I unlink them and once in a while print "%d files deleted" (and possibly elapsed time or current time stamp).

I don't expect it to be noticeably faster than the shell script version, it's just that I'm used to have to rip out the compiler now and again, either because there's no clean way of doing what I want from the shell or because while doable in shell, it's unproductively slow that way.


You are likely running into rewrite issues with the directory. Try deleting the newest files first. Look at mount options that will defer writeback to disk.

For a progress bar try running something like rm -rv /mystuff 2>&1 | pv -brtl > /dev/null

  • In terms of deleting the newest files first: ls -Ur? I'm pretty sure that'd load the dir entries, then reverse them; I don't believe ls is smart enough to start at the end of the dir entry list and rewind its way back to the beginning. "ls -1" also probably isn't a great idea, since it would probably take 50+ MB of core and several minutes to run; you'd want "ls -U" or "ls -f". – BMDan Sep 27 '10 at 1:28
  • It is likely only practical if the file names increase in predicable pattern. However you my try ls -1 piped to reverse, and piped to xargs. Use files instead of pipes if you want to see your intermediate results. You haven't provided any information on file nameing. You would generate the patter in reverse and delete the files using the pattern. You may need to handle missing file entries. Given your comment on memory required, you have an idea of the I/O require to rewrite the directory. – BillThor Sep 28 '10 at 4:12

Here is how I delete the millions of trace files that can sometimes gather on a large Oracle database server:

for i in /u*/app/*/diag/*/*/*/trace/*.tr? ; do rm $i; echo -n . ;  done

I find that this results in a fairly slow deletion that has low impact on server performance, usually something along the lines of an hour per million files on a "typical" 10,000 IOPS setup.

It will often take several minutes before the directories have been scanned, the initial file list generated and the first file is deleted. From there and on, a . is echoed for every file deleted.

The delay caused by echoing to the terminal has proven enough of a delay to prevent any significant load while deletion is progressing.

  • You're being eaten alive by globbing. How about something more like: find /u* -maxdepth 3 -mindepth 3 -type d -path '*/app/*' -name diag -print0 | xargs -0I = find = -mindepth 4 -maxdepth 4 -type d -name 'trace' -print0 | xargs -0I = find = -mindepth 1 -maxdepth 1 -name '*.tr' ? Add -delete to the last one to actually delete things; as written, it just lists what it would delete. Note that this is optimized for circumstances where you have a lot of uninteresting things in nearby directories; if that's not the case, you can simplify the logic a great deal. – BMDan Oct 10 '14 at 14:21
  • find -delete tends to cause too much I/O and easily impacts production performance. Perhaps with ionice. – Roy Oct 19 '14 at 8:15
  • It's causing all that I/O simply by being more efficient, though! The globbing is all front-loaded for your example (that is, the full list of files is generated before the first rm happens), so you have relatively efficient I/O at startup from that, followed by painful, out-of-order rms that probably don't cause much I/O, but involve scandir walking the directory repeatedly (not causing I/O because that's already been loaded into block cache; see also vfs_cache_pressure). If you'd like to slow things down, ionice is an option, but I'd probably use fractional-second sleeps. – BMDan Oct 28 '14 at 17:59
  • find /u*/app/*/diag -path '*/trace/*.tr' -execdir rm {} + would run one rm per directory, so you'd have less CPU overhead. As long as you have tons of CPU time to spare, throttling disk IO by forking a whole rm process for every unlink works, I guess, but it's ugly. perl with a sleep per unlink would be nicer if sleeping between rm of whole directories at a time is too bursty. (-execdir sh -c ... maybe) – Peter Cordes Mar 17 '15 at 9:42

You could use 'xargs' parallelization features:

ls -1|xargs -P nb_concurrent_jobs -n nb_files_by_job rm -rf
  • 1
    This won't help. The bottleneck is the poor random I/O on the drive. Doing parallel deletes could make it even worse and just increase CPU load. – Wim Kerkhoff Aug 20 '11 at 21:16
ls|cut -c -4|sort|uniq|awk '{ print "rm -rf " $1 }' | sh -x
  • 1
    Wow. I guess that falls pretty firmly in the "more than one way to skin a cat" camp. Seriously, though, with the sort and the uniq? "ls" sorts by default, anyway, and I'd sure hope that filenames are unique. :/ – BMDan Sep 30 '10 at 14:39

actually, this one is a little better if the shell you use does command line expansion:

ls|cut -c -4|sort|uniq|awk '{ print "echo " $1 ";rm -rf " $1 "*"}' |sh

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.