Reputation
4,643
Top tag
Next privilege 5,000 Rep.
Approve tag wiki edits
Badges
1 7 26
Newest
 Custodian
Impact
~220k people reached

Sep
29
comment rm on a directory with millions of files
ring0: I believe that's incorrect; see batleth.sapienti-sat.org/projects/FAQs/ext3-faq.html (or just google "ext3 writeback"). Writeback mode does still journal metadata changes.
Sep
28
comment What metaphors have you used to convey complex technical terms to the layman or beginner?
Glad you like it. To help with the visualization of an individual packet on that one, I sometimes suggest dropping a gummi bear/ping-pong ball/food coloring in on one end. You can also explain why TCP setup takes so long by explaining how you have to reverse the flow of the pipe and completely flush out all the water in it, two or three times (depending on how you're explaining it), in order to complete the three-way handshake.
Sep
28
comment rm on a directory with millions of files
pv is overwhelmingly awesome. I leave a trail of pv installations in my wake.
Sep
28
answered What metaphors have you used to convey complex technical terms to the layman or beginner?
Sep
27
revised rm on a directory with millions of files
formatting
Sep
27
comment rm on a directory with millions of files
Actually, that'd be a neat way to do a progress bar: since each file would be thirty-seven characters long (36 + a '\n'), I could easily write a parser for that, and since printf() is cheap and the rm command already has the name of the file loaded, there's no especial performance penalty. Seems like a non-starter for doing the whole shebang, since I could never get "rm" to do anything like that, anyway. But it could work quite well as an intra-10,000 progress bar; perhaps a "." for every hundred files?
Sep
27
comment rm on a directory with millions of files
I hesitate to imagine what that glob() call would do; I assume it does a scandir(). If so, that's going to take FOREVER to return. A modification of the first suggestion that doesn't pre-read all the dir entries might have some legs; however, in its current form, it, too, would use an unholy amount of CPU on just reading all the directory entries at once. Part of the goal here is to divide-and-conquer; this code isn't fundamentally different from 'rm -f *.png', notwithstanding issues with shell expansion. If it helps, there's nothing in the directory that I didn't want to delete.
Sep
27
comment rm on a directory with millions of files
It appears that doing this "live" requires an fsck on a mounted filesystem, which is... alarming. Got a better way?
Sep
27
comment rm on a directory with millions of files
Yes, it is enabled, but awesome suggestion!
Sep
27
comment rm on a directory with millions of files
Good call, but it's already mounted noatime, as I mentioned in the header to the question. And nodiratime is redundant; see lwn.net/Articles/245002 .
Sep
27
comment rm on a directory with millions of files
In terms of deleting the newest files first: ls -Ur? I'm pretty sure that'd load the dir entries, then reverse them; I don't believe ls is smart enough to start at the end of the dir entry list and rewind its way back to the beginning. "ls -1" also probably isn't a great idea, since it would probably take 50+ MB of core and several minutes to run; you'd want "ls -U" or "ls -f".
Sep
27
comment rm on a directory with millions of files
commit: that's pretty slick! Thanks for the pointer.
Sep
27
comment rm on a directory with millions of files
Lastly, FYI, not mentioned in that link is that fact that data=writeback can be a huge security hole, since data pointed to by a given entry may not have the data that was written there by the app, meaning that a crash could result in the old, possibly-sensitive/private data being exposed. Not a concern here, since we're only turning it on temporarily, but I wanted to alert everyone to that caveat in case either you or others who run across that suggestion weren't aware.
Sep
27
comment rm on a directory with millions of files
Writeback looks stellar, except the documentation I was looking at (gentoo.org/doc/en/articles/l-afig-p8.xml#doc_chap4) explicitly mentions that it still journals metadata, which I presume includes all the data I'm changing (I'm certainly not changing any data in the files themselves). Is my understanding of the option incorrect?
Sep
27
comment rm on a directory with millions of files
Interesting, so perhaps the fact that the files were being created over a long period of time is relevant? But that shouldn't matter; the block cache should have all of the pertinent metadata blocks in RAM. Maybe it's because unlink(2) is transactional? In your estimation, would turning off journaling for the duration of the rm be a potential (albeit admittedly somewhat dangerous) solution? It doesn't look like you can just turn off journaling entirely on a mounted filesystem without a tune2fs/fsck/reboot, which somewhat defeats the purpose.
Sep
26
awarded  Nice Question
Sep
26
awarded  Promoter
Sep
23
comment rm on a directory with millions of files
Each filename was exactly 36 characters long.
Sep
23
revised rm on a directory with millions of files
added 1 characters in body; edited tags
Sep
23
revised rm on a directory with millions of files
added 230 characters in body; added 94 characters in body