What filesystem best suits my needs?

  1. Thousands or even millions of files in one directory.
  2. Good (ext4 & ntfs level or close) reliability (incl. fault tolerance) and access speed.
  3. No directories actually needed, as well as descriptive names, just a dictionary-like structure of id-blob pairs is all I need.
  4. No links, attributes, and access control features needed.
  5. Having storage-level closed-key data encryption (so that if the hard drive is lost or stolen, no one can make any use of its content) would be cool.

The purpose is a file storage where all the metadata (data describing all the facts about what the file actually contains and who can access it) is stored in a MySQL database.

As far as I know common filesystems like NTFS and ext3/4 can go dead-slow if there are too many files placed in one directory - that's why I ask.

  • Have you considered using Cloud storage like Amazon Web Services S3 or Microsoft Azure BLOB Storage? – Michael Brown Apr 24 '17 at 8:45
  • The problem is not so much the file system (NTFS actually is ok with some hundred thousands) - it is all the tools around. Don't even DARE opening the directory in windows explorer. Shell scripts will take ages if a dir returns 2 million etc.

Better in a folder hierarchy.

Give every file a 16 byte hex code

  • Make folders / File names in 4 char segments,

So a file would maybe be in affc/2548/2224.... etc.

Keeps the directories shorter

AND you may be able to implement mount points here (though a 4 symbol file level is too wide for that).

DO not forget, too, you need to possibly backup/restore that


XFS: http://en.wikipedia.org/wiki/XFS

Do not use ext3 since it use a linked list for the content of a directory --> very slow directory with large set of file. Further more ext3 doesn't clean well this structure when removing files.

On linux, you can have an idea of the size of this list using ls -lahd directory/.

  • 2
    Unless you use dir_index. – Ignacio Vazquez-Abrams May 11 '10 at 18:48
  • right but you will still be limited by the 32K sub-directory limitation of ext3. Anyway I do not know if it matters here... – aligot May 11 '10 at 19:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.