I have a 3-level directory structure defined by 2 hex digits as such:
I have 300M small files in 1.5TB of compressed files that will populate these directories (we will have more files to come in the future, so I chose the directory structure to keep the mass of files from crashing a typical extX filesystem).
Unpacking these files moves at 1MB per second (or ~18 days to unpack). Ouchie!
I guess it was slow because I was creating the directory structure and then the files (done from Java APIs). So I set out to just create the directory structure alone in a bash loop.
The directories alone is about a 5 day task at current rate.
Any ideas on improving the speed that this moves?
One part of the puzzle is solved, using perl, rather than bash, creates the directories over 200 times faster, now it's an operation that give you a coffee break, not an extended weekend off.
But file creation is still extremely slow, even without needing to create the directories.