Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a 3-level directory structure defined by 2 hex digits as such:

0A/FF/2B/someimagefile.gif

I have 300M small files in 1.5TB of compressed files that will populate these directories (we will have more files to come in the future, so I chose the directory structure to keep the mass of files from crashing a typical extX filesystem).

Unpacking these files moves at 1MB per second (or ~18 days to unpack). Ouchie!

I guess it was slow because I was creating the directory structure and then the files (done from Java APIs). So I set out to just create the directory structure alone in a bash loop.

The directories alone is about a 5 day task at current rate.

Any ideas on improving the speed that this moves?


UPDATE

One part of the puzzle is solved, using perl, rather than bash, creates the directories over 200 times faster, now it's an operation that give you a coffee break, not an extended weekend off.

But file creation is still extremely slow, even without needing to create the directories.

share|improve this question
1  
SSD, reiserfs, mkdir -p –  dmourati Mar 15 '13 at 5:34
1  
Very useful comment, thank you, it's spurred a lot of thought, though none of these options seem to be a solution - I don't have an SSD, wish I did, some benchmarks on reiserfs suggest it's only incrementally better than ext4 which I'm on anyway, and mkdir -p isn't moving any faster for me (I pre-created the first 2 levels of directories pretty quickly, it's that last one that's slow), so the -p isn't buying me so much. –  davidparks21 Mar 15 '13 at 6:42
    
What problem is this the solution to? Are you recreating google-maps or something? –  Grizly Mar 16 '13 at 23:32
    
Yeh, sort of, we aggregate a product catalogs from many thousands of sellers and provide the user a unified shopping platform. We can't trust the 1000's of other webservers to always be up or have performance we can count on, so we host their images from our own servers. Since posting this I've moved to a new approach (used by facebook) in which I dump the binary data to a single large file and store an index into the file location where the image binary data is stored. I'll post on it's successfullness later. –  davidparks21 Mar 17 '13 at 3:09

2 Answers 2

up vote 1 down vote accepted

My final answer to this: "Don't do it".

I could not find a way to improve the speed beyond about 2Mbytes/sec when creating many small files. For terrabyte data volumes this is just too much inertia to work against.

We are following in the footsteps of facebook and dumping the files to a binary data store (or using a massive mysql/myisam table with BLOBs, experimenting now...).

It's a bit more complex, but eliminates the random seek problem associated with small files, and I can work with terrabyte volumes of data in a matter of hours, or a day, rather than weeks.

MongoDB has come in as another good option to investigate.

share|improve this answer

remount the filesystem with the options of noatime, nodiratime

share|improve this answer
    
Had those two settings in fstab since the beginning, with those it's still 2MB/sec on a single disk with 2 processes running. –  davidparks21 Mar 30 '13 at 4:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.