2

I have a 3-level directory structure defined by 2 hex digits as such:

0A/FF/2B/someimagefile.gif

I have 300M small files in 1.5TB of compressed files that will populate these directories (we will have more files to come in the future, so I chose the directory structure to keep the mass of files from crashing a typical extX filesystem).

Unpacking these files moves at 1MB per second (or ~18 days to unpack). Ouchie!

I guess it was slow because I was creating the directory structure and then the files (done from Java APIs). So I set out to just create the directory structure alone in a bash loop.

The directories alone is about a 5 day task at current rate.

Any ideas on improving the speed that this moves?


UPDATE

One part of the puzzle is solved, using perl, rather than bash, creates the directories over 200 times faster, now it's an operation that give you a coffee break, not an extended weekend off.

But file creation is still extremely slow, even without needing to create the directories.

5
  • 1
    SSD, reiserfs, mkdir -p
    – dmourati
    Mar 15, 2013 at 5:34
  • 1
    Very useful comment, thank you, it's spurred a lot of thought, though none of these options seem to be a solution - I don't have an SSD, wish I did, some benchmarks on reiserfs suggest it's only incrementally better than ext4 which I'm on anyway, and mkdir -p isn't moving any faster for me (I pre-created the first 2 levels of directories pretty quickly, it's that last one that's slow), so the -p isn't buying me so much. Mar 15, 2013 at 6:42
  • What problem is this the solution to? Are you recreating google-maps or something?
    – Grizly
    Mar 16, 2013 at 23:32
  • Yeh, sort of, we aggregate a product catalogs from many thousands of sellers and provide the user a unified shopping platform. We can't trust the 1000's of other webservers to always be up or have performance we can count on, so we host their images from our own servers. Since posting this I've moved to a new approach (used by facebook) in which I dump the binary data to a single large file and store an index into the file location where the image binary data is stored. I'll post on it's successfullness later. Mar 17, 2013 at 3:09
  • The choice of file system becomes important when needing to do things like this. Dec 4, 2014 at 7:20

2 Answers 2

1

My final answer to this: "Don't do it".

I could not find a way to improve the speed beyond about 2Mbytes/sec when creating many small files. For terrabyte data volumes this is just too much inertia to work against.

We are following in the footsteps of facebook and dumping the files to a binary data store (or using a massive mysql/myisam table with BLOBs, experimenting now...).

It's a bit more complex, but eliminates the random seek problem associated with small files, and I can work with terrabyte volumes of data in a matter of hours, or a day, rather than weeks.

MongoDB has come in as another good option to investigate.

1
  • A MySQL/MyISAM table worked beautifully. Loads at disk speeds, doesn't cost much in index space, and serves images easily and consistently. Dec 4, 2014 at 21:40
0

remount the filesystem with the options of noatime, nodiratime

1
  • Had those two settings in fstab since the beginning, with those it's still 2MB/sec on a single disk with 2 processes running. Mar 30, 2013 at 4:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .