Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Background: physical server, about two years old, 7200-RPM SATA drives connected to a 3Ware RAID card, ext3 FS mounted noatime and data=ordered, not under crazy load, kernel 2.6.18-92.1.22.el5, uptime 545 days. Directory doesn't contain any subdirectories, just millions of small (~100 byte) files, with some larger (a few KB) ones.

We have a server that has gone a bit cuckoo over the course of the last few months, but we only noticed it the other day when it started being unable to write to a directory due to it containing too many files. Specifically, it started throwing this error in /var/log/messages:

ext3_dx_add_entry: Directory index full!

The disk in question has plenty of inodes remaining:

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda3            60719104 3465660 57253444    6% /

So I'm guessing that means we hit the limit of how many entries can be in the directory file itself. No idea how many files that would be, but it can't be more, as you can see, than three million or so. Not that that's good, mind you! But that's part one of my question: exactly what is that upper limit? Is it tunable? Before I get yelled at--I want to tune it down; this enormous directory caused all sorts of issues.

Anyway, we tracked down the issue in the code that was generating all of those files, and we've corrected it. Now I'm stuck with deleting the directory.

A few options here:

  1. rm -rf (dir)
    I tried this first. I gave up and killed it after it had run for a day and a half without any discernible impact.
  2. unlink(2) on the directory: Definitely worth consideration, but the question is whether it'd be faster to delete the files inside the directory via fsck than to delete via unlink(2). That is, one way or another, I've got to mark those inodes as unused. This assumes, of course, that I can tell fsck not to drop entries to the files in /lost+found; otherwise, I've just moved my problem. In addition to all the other concerns, after reading about this a bit more, it turns out I'd probably have to call some internal FS functions, as none of the unlink(2) variants I can find would allow me to just blithely delete a directory with entries in it. Pooh.
  3. while [ true ]; do ls -Uf | head -n 10000 | xargs rm -f 2>/dev/null; done )
    This is actually the shortened version; the real one I'm running, which just adds some progress-reporting and a clean stop when we run out of files to delete, is:

    export i=0; time ( while [ true ]; do ls -Uf | head -n 3 | grep -qF '.png' || break; ls -Uf | head -n 10000 | xargs rm -f 2>/dev/null; export i=$(($i+10000)); echo "$i..."; done )

    This seems to be working rather well. As I write this, it's deleted 260,000 files in the past thirty minutes or so.

Now, for the questions:
  1. As mentioned above, is the per-directory entry limit tunable?
  2. Why did it take "real 7m9.561s / user 0m0.001s / sys 0m0.001s" to delete a single file which was the first one in the list returned by "ls -U", and it took perhaps ten minutes to delete the first 10,000 entries with the command in #3, but now it's hauling along quite happily? For that matter, it deleted 260,000 in about thirty minutes, but it's now taken another fifteen minutes to delete 60,000 more. Why the huge swings in speed?
  3. Is there a better way to do this sort of thing? Not store millions of files in a directory; I know that's silly, and it wouldn't have happened on my watch. Googling the problem and looking through SF and SO offers a lot of variations on "find" that are not going to be significantly faster than my approach for several self-evident reasons. But does the delete-via-fsck idea have any legs? Or something else entirely? I'm eager to hear out-of-the-box (or inside-the-not-well-known-box) thinking.
Thanks for reading the small novel; feel free to ask questions and I'll be sure to respond. I'll also update the question with the final number of files and how long the delete script ran once I have that.

Final script output!:

2970000...
2980000...
2990000...
3000000...
3010000...

real    253m59.331s
user    0m6.061s
sys     5m4.019s

So, three million files deleted in a bit over four hours.

share|improve this question
1  
rm (GNU coreutils) 8.4 has this option: "-v, --verbose explain what is being done". It will display all the files that are being deleted. –  Cristian Ciupitu Sep 27 '10 at 18:34
2  
Actually, that'd be a neat way to do a progress bar: since each file would be thirty-seven characters long (36 + a '\n'), I could easily write a parser for that, and since printf() is cheap and the rm command already has the name of the file loaded, there's no especial performance penalty. Seems like a non-starter for doing the whole shebang, since I could never get "rm" to do anything like that, anyway. But it could work quite well as an intra-10,000 progress bar; perhaps a "." for every hundred files? –  BMDan Sep 27 '10 at 21:40
6  
rm -rfv | pv -l >/dev/null. pv should be available in the EPEL repository. –  Cristian Ciupitu Sep 27 '10 at 23:04
5  
pv is overwhelmingly awesome. I leave a trail of pv installations in my wake. –  BMDan Sep 28 '10 at 1:54
    
I had this exact same issue recently. Thankyou! –  richo Dec 23 '10 at 23:26
show 1 more comment

21 Answers

up vote 16 down vote accepted
+500

The data=writeback mount option deserves to be tried, in order to prevent journaling of the file system. This should be done only during the deletion time, there is a risk however if the server is being shutdown or rebooted during the delete operation.

According to this page,

Some applications show very significant speed improvement when it is used. For example, speed improvements can be seen (...) when applications create and delete large volumes of small files.

The option is set either in fstab or during the mount operation, replacing data=ordered with data=writeback. The file system containing the files to be deleted has to be remounted.

share|improve this answer
    
He could also increase the time from the commit option: "This default value (or any low value) will hurt performance, but it's good for data-safety. Setting it to 0 will have the same effect as leaving it at the default (5 seconds). Setting it to very large values will improve performance". –  Cristian Ciupitu Sep 26 '10 at 19:14
1  
Writeback looks stellar, except the documentation I was looking at (gentoo.org/doc/en/articles/l-afig-p8.xml#doc_chap4) explicitly mentions that it still journals metadata, which I presume includes all the data I'm changing (I'm certainly not changing any data in the files themselves). Is my understanding of the option incorrect? –  BMDan Sep 27 '10 at 1:20
    
Lastly, FYI, not mentioned in that link is that fact that data=writeback can be a huge security hole, since data pointed to by a given entry may not have the data that was written there by the app, meaning that a crash could result in the old, possibly-sensitive/private data being exposed. Not a concern here, since we're only turning it on temporarily, but I wanted to alert everyone to that caveat in case either you or others who run across that suggestion weren't aware. –  BMDan Sep 27 '10 at 1:23
    
commit: that's pretty slick! Thanks for the pointer. –  BMDan Sep 27 '10 at 1:26
    
The data=writeback doesn't perform any kind of journaling. This is the reason why a reboot shouldn't happen during the delete operation. I didn't have the occasion to use it on a million files FS, but it should be faster. –  ring0 Sep 27 '10 at 22:29
show 6 more comments

Would it be possible to backup all of the other files from this file system to a temporary storage location, reformat the partition, and then restore the files?

share|improve this answer
1  
I really like this answer, actually. As a practical matter, in this case, no, but it's not one I would have thought of. Bravo! –  BMDan Sep 23 '10 at 10:23
    
Exactly what I was thinking too. This is an answer for question 3. Ideal if you ask me :) –  Joshua Oct 1 '10 at 15:31
add comment

Whilst a major cause of this problem is ext3 performance with millions of files, the actual root cause of this problem is different.

When a directory needs to be listed readdir() is called on the directory which yields a list of files. readdir is a posix call, but the real linux system call being used here is called 'getdents'. Getdents list directory entries by filling a buffer is entries.

The problem is mainly down to the fact that that readdir() uses a fixed buffer size of 32Kb to fetch files. As a directory gets larger and larger (the size increases as files are added) ext3 gets slower and slower to fetch entries and additional readdir's 32Kb buffer size is only sufficient to include a fraction of the entries in the directory. This causes readdir to loop over and over and invoke the expensive system call over and over.

For example, on a test directory I created with over 2.6 million files inside, running "ls -1|wc-l" shows a large strace output of many getdent system calls.

$ strace ls -1 | wc -l
brk(0x4949000)                          = 0x4949000
getdents(3, /* 1025 entries */, 32768)  = 32752
getdents(3, /* 1024 entries */, 32768)  = 32752
getdents(3, /* 1025 entries */, 32768)  = 32760
getdents(3, /* 1025 entries */, 32768)  = 32768
brk(0)                                  = 0x4949000
brk(0x496a000)                          = 0x496a000
getdents(3, /* 1024 entries */, 32768)  = 32752
getdents(3, /* 1026 entries */, 32768)  = 32760
...

Additionally the time spent in this directory was significant.

$ time ls -1 | wc -l
2616044

real    0m20.609s
user    0m16.241s
sys 0m3.639s

The method to make this a more efficient process is to call getdents manually with a much larger buffer. This improves performance significantly.

Now, your not supposed to call getdents yourself manually so no interface exists to use it normally (check the man page for getdents to see!), however you can call it manually and make your system call invocation way more efficient.

This drastically reduces the time it takes to fetch these files. I wrote a program that does this.

$ time ./dentls bigfolder >out.txt

real    0m2.355s
user    0m0.326s
sys 0m1.995s

Almost ten times more efficient! I suspect that the larger the directory the more efficient this would end up being.

I've provided the source to this program below. If you want to delete, uncomment the unlink line. This will drastically slow down performance I imagine. It also avoids printing/unlinking anything that is not a file.

It will spit out file names to stdout. You should probably redirect this output. You can use that to delete the files outside of the program afterwards if you wanted.

/* I can be compiled with the command "gcc -o dentls dentls.c" */

#define _GNU_SOURCE
#include <search.h>     /* Defines tree functions */
#include <dirent.h>     /* Defines DT_* constants */
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <string.h>

/* Because most filesystems use btree to store dents
 * its very important to perform an in-order removal
 * of the file contents. Performing an 'as-is read' of
 * the contents causes lots of btree rebalancing
 * that has significantly negative effect on unlink performance
 */

/* Tests indicate that performing a ascending order traversal
 * is about 1/3 faster than a descending order traversal */
int compare_fnames(const void *key1, const void *key2) {
  return strcmp((char *)key1, (char *)key2);
}

void walk_tree(const void *node, VISIT val, int lvl) {
  int rc = 0;
  switch(val) {
  case leaf:
    rc = unlink(*(char **)node);
    break;
  /* End order is deliberate here as it offers the best btree
   * rebalancing avoidance.
   */
  case endorder:
    rc = unlink(*(char **)node);
  break;
  default:
    return;
    break;
  }

  if (rc < 0) {
    perror("unlink problem");
    exit(1);
  }

}

void dummy_destroy(void *nil) {
  return;
}

void *tree = NULL;

struct linux_dirent {
        long           d_ino;
        off_t          d_off;
        unsigned short d_reclen;
        char           d_name[256];
        char           d_type;
};

int main(const int argc, const char** argv) {
    int totalfiles = 0;
    int dirfd = -1;
    int offset = 0;
    int bufcount = 0;
    void *buffer = NULL;
    char *d_type;
    struct linux_dirent *dent = NULL;
    struct stat dstat;

    /* Test we have a directory path */
    if (argc < 2) {
        fprintf(stderr, "You must supply a valid directory path.\n");
        exit(1);
    }

    const char *path = argv[1];

    /* Standard sanity checking stuff */
    if (access(path, R_OK) < 0) {
        perror("Could not access directory");
        exit(1);
    }

    if (lstat(path, &dstat) < 0) {
        perror("Unable to lstat path");
        exit(1);
    }

    if (!S_ISDIR(dstat.st_mode)) {
        fprintf(stderr, "The path %s is not a directory.\n", path);
        exit(1);
    }

    /* Allocate a buffer of equal size to the directory to store dents */
    if ((buffer = malloc(dstat.st_size+10240)) == NULL) {
        perror("malloc failed");
        exit(1);
    }

    /* Open the directory */
    if ((dirfd = open(path, O_RDONLY)) < 0) {
        perror("Open error");
        exit(1);
    }

    /* Switch directories */
    fchdir(dirfd);

    while (bufcount = syscall(SYS_getdents, dirfd, buffer, dstat.st_size+10240)) {
        offset = 0;
        dent = buffer;
        while (offset < bufcount) {
            /* Dont print thisdir and parent dir */
            if (!((strcmp(".",dent->d_name) == 0) || (strcmp("..",dent->d_name) == 0))) {
                d_type = (char *)dent + dent->d_reclen-1;
                /* Only print files */
                if (*d_type == DT_REG) {
                    /* Sort all our files into a binary tree */
            if (!tsearch(dent->d_name, &tree, compare_fnames)) {
                      fprintf(stderr, "Cannot acquire resources for tree!\n");
                      exit(1);
                    }
                    totalfiles++;
                }
            }
            offset += dent->d_reclen;
            dent = buffer + offset;
        }
    }
    fprintf(stderr, "Total files: %d\n", totalfiles);
    printf("Performing delete..\n");

    twalk(tree, walk_tree);
    printf("Done\n");
    close(dirfd);
    free(buffer);
    tdestroy(tree, dummy_destroy);
}

Whilst this does not combat the underlying fundamental problem (lots of files, in a filesystem that performs poorly at it). Its likely to be much, much faster than many of the alternatives being posted.

As a forethought, one should remove the affected directory and remake it after. Directories only ever increase in size and can remain poorly performing even with a few files inside due to the size of the directory.

Edit: I revisited this today, because most filesystems store their directory structures in a btree format, the order of which you delete files is also important. One needs to avoid rebalancing the btree when you perform the unlink. As such I added a sort before deletes occur.

The program will now (on my system) delete 1000000 files in 43 seconds. The closest program to this was rsync -a --delete which took 60 seconds (which also does deletions in-order, too but does not perform an efficient directory lookup).

share|improve this answer
1  
Two minor concerns: one, [256] should probably be [FILENAME_MAX], and two, my Linux (2.6.18==CentOS 5.x) doesn't seem to include a d_type entry in dirent (at least according to getdents(2)). –  BMDan Nov 7 '11 at 5:01
    
indeed, Noted on point one. The struct can actually have d_type removed as according to the man page the struct it utilizes doesnt physically represent the struct described. Basically d_type is reclen-1, and thats been in the kernel since 2.6.4 –  Matthew Ife Nov 7 '11 at 7:51
    
Could you please elaborate a bit on btree rebalancing and why deletion in order helps preventing it? I tried Googling for it, unfortunately to no avail. –  ovgolovin Jul 28 '13 at 23:28
1  
Because now it seems to me if we are deleting in-order, we force rebalancing, as we remove leaves on one side and leave on the other: en.wikipedia.org/wiki/B-tree#Rebalancing_after_deletion –  ovgolovin Jul 28 '13 at 23:37
    
I hope I don't bother you with this matters. But still I started a question about deleting files in-order stackoverflow.com/q/17955459/862380, which seems not to receive an answer which will explain the issue with the example, which will be understandable for ordinary programmers. If you have time and feel like so, could you look into it? Maybe you could write a better explanation. –  ovgolovin Aug 8 '13 at 18:35
show 2 more comments

There is no per directory file limit in ext3 just the filesystem inode limit (i think there is a limit on the number of subdirectories though).

You may still have problems after removing the files.

When a directory has millions of files, the directory entry itself becomes very large. The directory entry has to be scanned for every remove operation, and that takes various amounts of time for each file, depending on where its entry is located. Unfortunately even after all the files have been removed the directory entry retains its size. So further operations that require scanning the directory entry will still take a long time even if the directory is now empty. The only way to solve that problem is to rename the directory, create a new one with the old name, and transfer any remaining files to the new one. Then delete the renamed one.

share|improve this answer
    
Indeed, I noticed just this behavior after deleting everything. Luckily, we had already mv'd the directory out of the "line of fire", as it were, so I could just rmdir it. –  BMDan Sep 23 '10 at 10:24
2  
That said, if there's no per-directory file limit, why did I get "ext3_dx_add_entry: Directory index full!" when there were still inodes available on that partition? There were no subdirectories inside this directory. –  BMDan Sep 23 '10 at 10:26
3  
hmm i did a little more research and it seems there's a limit of number of blocks a directory can take up. The exact number of files is dependent on a few things eg filename length. This gossamer-threads.com/lists/linux/kernel/921942 seems to indicate that with 4k blocks you should be able to have more than 8 million files in a directory. Were they particularly long filenames? –  Alex J. Roberts Sep 23 '10 at 12:04
    
Each filename was exactly 36 characters long. –  BMDan Sep 23 '10 at 15:34
    
well that's me out of ideas :) –  Alex J. Roberts Sep 24 '10 at 0:13
add comment

make sure you do:

mount -o remount,rw,noatime,nodiratime /mountpoint

which should speed things up a bit as well.

share|improve this answer
3  
Good call, but it's already mounted noatime, as I mentioned in the header to the question. And nodiratime is redundant; see lwn.net/Articles/245002 . –  BMDan Sep 27 '10 at 21:19
add comment

ls very slow command. Try:

find /dir_to_delete ! -iname "*.png" -type f -delete
share|improve this answer
    
rm -rf ran for a day and a half, and I finally killed it, without ever knowing if it had actually accomplished anything. I needed a progress bar. –  BMDan Sep 23 '10 at 10:18
4  
As to rm being very slow, "time find . -delete" on 30k files: 0m0.357s / 0m0.019s / 0m0.337s real/user/sys. "time ( ls -1U | xargs rm -f )" on those same files: 0m0.366s / 0m0.025s / 0m0.340s. Which is basically margin-of-error territory. –  BMDan Sep 23 '10 at 10:20
add comment

Is dir_index set for the f/s ? tune2fs -l | grep dir_index, if not enabling it ... it's usually on for new RH

share|improve this answer
    
Yes, it is enabled, but awesome suggestion! –  BMDan Sep 27 '10 at 21:23
add comment

find simply did not work for me, even after changing the ext3 fs's parameters as suggested by the users above. Consumed way too much memory. This PHP script did the trick - fast, insignificant CPU usage, insignificant memory usage:

<?php 
$dir = '/directory/in/question';
$dh = opendir($dir)) { 
while (($file = readdir($dh)) !== false) { 
    unlink($dir . '/' . $file); 
} 
closedir($dh); 
?>

I posted a bug report regarding this trouble with find: http://savannah.gnu.org/bugs/?31961

share|improve this answer
    
This saved me!! –  jestro Mar 20 '11 at 10:12
add comment

My preferred option is the newfs approach, already suggested. The basic problem is, again as already noted, the linear scan to handle deletion is problematic.

rm -rf should be near optimal for a local filesystem (NFS would be different). But at millions of files, 36 bytes per filename and 4 per inode (a guess, not checking value for ext3), that's 40 * millions, to be kept in RAM just for the directory.

At a guess, you're thrashing the filesystem metadata cache memory in Linux, so that blocks for one page of the directory file are being expunged while you're still using another part, only to hit that page of the cache again when the next file is deleted. Linux performance tuning isn't my area, but /proc/sys/{vm,fs}/ probably contain something relevant.

If you can afford downtime, you might consider turning on the dir_index feature. It switches the directory index from linear to something far more optimal for deletion in large directories (hashed b-trees). tune2fs -O dir_index ... followed by e2fsck -D would work. However, while I'm confident this would help before there are problems, I don't know how the conversion (e2fsck with the -D) performs when dealing with an existing v.large directory. Backups + suck-it-and-see.

share|improve this answer
1  
pubbs.net/201008/squid/… suggests that /proc/sys/fs/vfs_cache_pressure might be the value to use, but I don't know whether the directory itself counts towards page cache (because that's what it is) or the inode cache (because, despite not being an inode, it's FS metadata and bundled into there for that reason). As I say, Linux VM tuning is not my area. Play and see what helps. –  Phil P Sep 26 '10 at 12:10
add comment

Obviously not apples to apples here, but I setup a little test and did the following:

Created 100,000 512-byte files in a directory (dd and /dev/urandom in a loop); forgot to time it, but it took roughly 15 minutes to create those files.

Ran the following to delete said files:

ls -1 | wc -l && time find . -type f -delete

100000

real    0m4.208s
user    0m0.270s
sys     0m3.930s 

This is a Pentium 4 2.8GHz box (couple hundred GB IDE 7200 RPM I think; EXT3). Kernel 2.6.27.

share|improve this answer
    
Interesting, so perhaps the fact that the files were being created over a long period of time is relevant? But that shouldn't matter; the block cache should have all of the pertinent metadata blocks in RAM. Maybe it's because unlink(2) is transactional? In your estimation, would turning off journaling for the duration of the rm be a potential (albeit admittedly somewhat dangerous) solution? It doesn't look like you can just turn off journaling entirely on a mounted filesystem without a tune2fs/fsck/reboot, which somewhat defeats the purpose. –  BMDan Sep 27 '10 at 1:11
    
I can't comment on that, but anecdotally (in various NIX discussions over the years), I've always heard that rm is horribly slow on a large number of files, hence the find -delete option. With a wildcard on the shell, it'll expand each filename matched, and I'm assuming there's a limited memory buffer for that, so you could see how that would get inefficient. –  gravyface Sep 28 '10 at 12:58
    
rm would be slow because it's looking for a file by name, which means iterating through the directory entries one by one until it finds it. In this case, however, since each entry it's being handed is (at that point) the first one in the list (ls -U/ls -f), it should be nearly as fast. That said, rm -rf <dir>, which should have run like a champ, was slow as could be. Perhaps it's time to write a patch to coreutils to speed massive deletes? Maybe it's secretly globbing/sorting in some recursive way in order to implement rm -rf? Uncertainties like this are why I asked the question. ;) –  BMDan Sep 30 '10 at 3:11
1  
Reboot the machine after you run the creation step. You should get a noticeably slower delete. –  Matt Apr 23 '12 at 23:09
add comment

Sometimes Perl can work wonders in cases like this. Have you already tried if a small script such as this could outperform bash and the basic shell commands?

#!/usr/bin/perl 
open(ANNOYINGDIR,"/path/to/your/directory");
@files = grep("/*\.png/", readdir(ANNOYINGDIR));
close(ANNOYINGDIR);

for (@files) {
    printf "Deleting %s\n",$_;
    unlink $_;
}

Or another, perhaps even faster, Perl approach:

#!/usr/bin/perl
unlink(glob("/path/to/your/directory/*.png")) or die("Could not delete files, this happened: $!");

EDIT: I just gave my Perl scripts a try. The more verbose one does something right. In my case I tried this with a virtual server with 256 MB RAM and half a million files.

time find /test/directory | xargs rm results:

real    2m27.631s
user    0m1.088s
sys     0m13.229s

compared to

time perl -e 'opendir(FOO,"./"); @files = readdir(FOO); closedir(FOO); for (@files) { unlink $_; }'

real    0m59.042s
user    0m0.888s
sys     0m18.737s
share|improve this answer
    
I hesitate to imagine what that glob() call would do; I assume it does a scandir(). If so, that's going to take FOREVER to return. A modification of the first suggestion that doesn't pre-read all the dir entries might have some legs; however, in its current form, it, too, would use an unholy amount of CPU on just reading all the directory entries at once. Part of the goal here is to divide-and-conquer; this code isn't fundamentally different from 'rm -f *.png', notwithstanding issues with shell expansion. If it helps, there's nothing in the directory that I didn't want to delete. –  BMDan Sep 27 '10 at 21:30
    
I gotta try more as soon as I get to work. I just tried to create 100 000 files in a single directory and find + xargs + rm combination took 7.3 seconds, Perl + unlink(glob)... combination finished in 2.7 seconds. Tried that couple of times, the result was always the same. At work I'll try this with more files. –  Janne Pikkarainen Sep 28 '10 at 5:57
    
I learnt something new while testing this. At least with ext3 and ext4 the directory entry itself remains huge even after deleting all the files from there. After couple of tests my /tmp/test directory was taking 15 MB of disk space. Is there other way to clean that up other than removing the directory and recreating it? –  Janne Pikkarainen Sep 28 '10 at 12:53
2  
No, you need to recreate it. I hit this when dealing with a mail-system and folder-per-recipient and cleanups after significant issues: there's no way, other than creating a new directory and shuffling the directories about, then nuking the old one. So you can reduce the time window when there's no directory, but not eliminate it. –  Phil P Sep 28 '10 at 23:49
    
Note that glob() will sort the results, much as shell globbing normally does, so because you only have 100k files, everything fits easily and the sort is fast. With a much larger directory, you'd want to opendir()/readdir()/closedir() just to avoid the sort. [I say normally for shell, since zsh has a glob modifier to make the sort order unsorted, which is useful when dealing with large numbers of files; *(oN) ] –  Phil P Sep 28 '10 at 23:50
show 1 more comment

I recently faced a similar issue and was unable to get ring0's data=writeback suggestion to work (possibly due to the fact that the files are on my main partition). While researching workarounds I stumbled upon this:

tune2fs -O ^has_journal <device>

This will turn off journaling completely, regardless of the data option give to mount. I combined this with noatime and the volume had dir_index set, and it seemed to work pretty well. The delete actually finished without me needing to kill it, my system remained responsive, and it's now back up and running (with journaling back on) with no issues.

share|improve this answer
add comment

Well, this is not a real answer, but...

Would it be possible to convert the filesystem to ext4 and see if things change?

share|improve this answer
    
It appears that doing this "live" requires an fsck on a mounted filesystem, which is... alarming. Got a better way? –  BMDan Sep 27 '10 at 21:25
    
The filesystem has to be unmounted before the conversion, i.e. before the necessary tunefs commands. –  marcoc Sep 28 '10 at 7:00
add comment

Alright this has been covered in various ways in the rest of the thread but I thought I would throw in my two cents. The performance culprit in your case is probably readdir. You are getting back a list of files that are not necessarily in any way sequential on disk which is causing disk access all over the place when you unlink. The files are small enough that the unlink operation probably doesn't jump around too much zeroing out the space. If you readdir and then sort by ascending inode you would probably get better performance. So readdir into ram (sort by inode) -> unlink -> profit.

Inode is a rough approximation here I think .. but basing on your use case it might be fairly accurate...

share|improve this answer
1  
Correct me if I'm wrong, but unlink(2) doesn't zero the inode, it just removes the reference to it from the directory. I like the chutzpah of this approach, though. Care to run some time-trials and see if it holds true? –  BMDan Sep 30 '10 at 3:07
add comment

From what I remember the deletion of inodes in ext filesystems is O(n^2), so the more files you delete the faster the rest will go.

There was a one time I was faced with similar problem (though my estimates looked at ~7h deletion time), in the end went the jftuga suggested route in first comment.

share|improve this answer
add comment

I would probably have whipped out a C compiler and done the moral equivalent of your script. That is, use opendir(3) to get a directory handle, then use readdir(3) to get the name of files, then tally up files as I unlink them and once in a while print "%d files deleted" (and possibly elapsed time or current time stamp).

I don't expect it to be noticeably faster than the shell script version, it's just that I'm used to have to rip out the compiler now and again, either because there's no clean way of doing what I want from the shell or because while doable in shell, it's unproductively slow that way.

share|improve this answer
    
He could at least start by modifying the source code of rm from coreutils. –  Cristian Ciupitu Sep 26 '10 at 19:51
add comment

You are likely running into rewrite issues with the directory. Try deleting the newest files first. Look at mount options that will defer writeback to disk.

For a progress bar try running something like 'rm -rv /mystuff 2>&1 | pv -brtl > /dev/null' .

share|improve this answer
    
In terms of deleting the newest files first: ls -Ur? I'm pretty sure that'd load the dir entries, then reverse them; I don't believe ls is smart enough to start at the end of the dir entry list and rewind its way back to the beginning. "ls -1" also probably isn't a great idea, since it would probably take 50+ MB of core and several minutes to run; you'd want "ls -U" or "ls -f". –  BMDan Sep 27 '10 at 1:28
    
It is likely only practical if the file names increase in predicable pattern. However you my try ls -1 piped to reverse, and piped to xargs. Use files instead of pipes if you want to see your intermediate results. You haven't provided any information on file nameing. You would generate the patter in reverse and delete the files using the pattern. You may need to handle missing file entries. Given your comment on memory required, you have an idea of the I/O require to rewrite the directory. –  BillThor Sep 28 '10 at 4:12
add comment

I haven't benchmarked it, but this guy did:

rsync -a --delete ./emptyDirectoty/ ./hugeDirectory/
share|improve this answer
add comment

You could use 'xargs' parallelization features:

ls -1|xargs -P nb_concurrent_jobs -n nb_files_by_job rm -rf
share|improve this answer
    
This won't help. The bottleneck is the poor random I/O on the drive. Doing parallel deletes could make it even worse and just increase CPU load. –  Wim Kerkhoff Aug 20 '11 at 21:16
add comment
ls|cut -c -4|sort|uniq|awk '{ print "rm -rf " $1 }' | sh -x
share|improve this answer
1  
Wow. I guess that falls pretty firmly in the "more than one way to skin a cat" camp. Seriously, though, with the sort and the uniq? "ls" sorts by default, anyway, and I'd sure hope that filenames are unique. :/ –  BMDan Sep 30 '10 at 14:39
add comment

actually, this one is a little better if the shell you use does command line expansion:

ls|cut -c -4|sort|uniq|awk '{ print "echo " $1 ";rm -rf " $1 "*"}' |sh
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.