What is the best filesystem for storing thousands of files in one dictionary-like id-blob structure?

Question

What filesystem best suits my needs?

1. Thousands or even millions of files in one directory.
2. Good (ext4 & ntfs level or close) reliability (incl. fault tolerance) and access speed.
3. No directories actually needed, as well as descriptive names, just a dictionary-like structure of id-blob pairs is all I need.
4. No links, attributes, and access control features needed.
5. Having storage-level closed-key data encryption (so that if the hard drive is lost or stolen, no one can make any use of its content) would be cool.

The purpose is a file storage where all the metadata (data describing all the facts about what the file actually contains and who can access it) is stored in a MySQL database.

As far as I know common filesystems like NTFS and ext3/4 can go dead-slow if there are too many files placed in one directory - that's why I ask.

Answer 1

• DO NOT ;)

• The problem is not so much the file system (NTFS actually is ok with some hundred thousands) - it is all the tools around. Don't even DARE opening the directory in windows explorer. Shell scripts will take ages if a dir returns 2 million etc.

Better in a folder hierarchy.

Give every file a 16 byte hex code ;)

• Make folders / File names in 4 char segments,

So a file would maybe be in affc/2548/2224.... etc.

Keeps the directories shorter ;)

AND you may be able to implement mount points here (though a 4 symbol filest level is too wide for that).

DO not forget, too, you need to possibly backup/restore that ;)

Answer 2

XFS: http://en.wikipedia.org/wiki/XFS

Do not use ext3 since it use a linked list for the content of a directory --> very slow directory with large set of file. Further more ext3 doesn't clean well this structure when removing files.

On linux, you can have an idea of the size of this list using ls -lahd directory/.