Reputation
5,111
Top tag
Next privilege 10,000 Rep.
Access moderator tools
Badges
1 10 28
Newest
 Enlightened
Impact
~329k people reached

Sep
27
comment rm on a directory with millions of files
I hesitate to imagine what that glob() call would do; I assume it does a scandir(). If so, that's going to take FOREVER to return. A modification of the first suggestion that doesn't pre-read all the dir entries might have some legs; however, in its current form, it, too, would use an unholy amount of CPU on just reading all the directory entries at once. Part of the goal here is to divide-and-conquer; this code isn't fundamentally different from 'rm -f *.png', notwithstanding issues with shell expansion. If it helps, there's nothing in the directory that I didn't want to delete.
Sep
27
comment rm on a directory with millions of files
It appears that doing this "live" requires an fsck on a mounted filesystem, which is... alarming. Got a better way?
Sep
27
comment rm on a directory with millions of files
Yes, it is enabled, but awesome suggestion!
Sep
27
comment rm on a directory with millions of files
Good call, but it's already mounted noatime, as I mentioned in the header to the question. And nodiratime is redundant; see lwn.net/Articles/245002 .
Sep
27
comment rm on a directory with millions of files
In terms of deleting the newest files first: ls -Ur? I'm pretty sure that'd load the dir entries, then reverse them; I don't believe ls is smart enough to start at the end of the dir entry list and rewind its way back to the beginning. "ls -1" also probably isn't a great idea, since it would probably take 50+ MB of core and several minutes to run; you'd want "ls -U" or "ls -f".
Sep
27
comment rm on a directory with millions of files
commit: that's pretty slick! Thanks for the pointer.
Sep
27
comment rm on a directory with millions of files
Lastly, FYI, not mentioned in that link is that fact that data=writeback can be a huge security hole, since data pointed to by a given entry may not have the data that was written there by the app, meaning that a crash could result in the old, possibly-sensitive/private data being exposed. Not a concern here, since we're only turning it on temporarily, but I wanted to alert everyone to that caveat in case either you or others who run across that suggestion weren't aware.
Sep
27
comment rm on a directory with millions of files
Writeback looks stellar, except the documentation I was looking at (gentoo.org/doc/en/articles/l-afig-p8.xml#doc_chap4) explicitly mentions that it still journals metadata, which I presume includes all the data I'm changing (I'm certainly not changing any data in the files themselves). Is my understanding of the option incorrect?
Sep
27
comment rm on a directory with millions of files
Interesting, so perhaps the fact that the files were being created over a long period of time is relevant? But that shouldn't matter; the block cache should have all of the pertinent metadata blocks in RAM. Maybe it's because unlink(2) is transactional? In your estimation, would turning off journaling for the duration of the rm be a potential (albeit admittedly somewhat dangerous) solution? It doesn't look like you can just turn off journaling entirely on a mounted filesystem without a tune2fs/fsck/reboot, which somewhat defeats the purpose.
Sep
23
comment rm on a directory with millions of files
Each filename was exactly 36 characters long.
Sep
23
comment rm on a directory with millions of files
That said, if there's no per-directory file limit, why did I get "ext3_dx_add_entry: Directory index full!" when there were still inodes available on that partition? There were no subdirectories inside this directory.
Sep
23
comment rm on a directory with millions of files
Indeed, I noticed just this behavior after deleting everything. Luckily, we had already mv'd the directory out of the "line of fire", as it were, so I could just rmdir it.
Sep
23
comment rm on a directory with millions of files
I really like this answer, actually. As a practical matter, in this case, no, but it's not one I would have thought of. Bravo!
Sep
23
comment rm on a directory with millions of files
As to rm being very slow, "time find . -delete" on 30k files: 0m0.357s / 0m0.019s / 0m0.337s real/user/sys. "time ( ls -1U | xargs rm -f )" on those same files: 0m0.366s / 0m0.025s / 0m0.340s. Which is basically margin-of-error territory.
Sep
23
comment rm on a directory with millions of files
rm -rf ran for a day and a half, and I finally killed it, without ever knowing if it had actually accomplished anything. I needed a progress bar.
Sep
22
comment Installing Zend Server via Puppet, on RHEL and Ubuntu?
Indeed, that's how we do it.
Aug
24
comment mysql dump of talble limit 100
That's hideous and hilarious. Well-played. And here I was writing a script!
Aug
24
comment How can I figure out if port 25 is blocked on my network?
John: "filtered" means that no response was received from the server. "Closed" means a TCP RST was received. The former usually indicates that a firewall has blocked the connection (as they rarely issue RSTs for their "deny" rules), while the latter typically indicates that there really is nothing listening on that port, albeit with the caveat that, with a bit of effort, a crafty net.admin can completely mess up that set of assumptions.
Aug
24
comment How can I figure out if port 25 is blocked on my network?
Slick! You can even specify a port with "-p", so it'd be "tcptraceroute -Tp 25 mydomain.tld" in this case. Thanks for the pointer.
Aug
17
comment Remotely view or logon as another user
First off, this is just patently wrong-headed. If you're the administrator, you can already impersonate someone with impunity, either by logging a message of any kind, or, if that's not sneaky enough, installing a driver that can do even scarier (and even harder-to-trace) things. Nothing the OQ is asking for is impossible at a technical level. Whether it's advisable, or violates ethics standards, or is more difficult than other potential solutions--those are separate questions that deserve proper airing. Simply saying, "sorry, nope, it's impossible," is neither helpful, nor even correct.