Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

What filesystem best suits my needs?

  1. Thousands or even millions of files in one directory.
  2. Good (ext4 & ntfs level or close) reliability (incl. fault tolerance) and access speed.
  3. No directories actually needed, as well as descriptive names, just a dictionary-like structure of id-blob pairs is all I need.
  4. No links, attributes, and access control features needed.
  5. Having storage-level closed-key data encryption (so that if the hard drive is lost or stolen, no one can make any use of its content) would be cool.

The purpose is a file storage where all the metadata (data describing all the facts about what the file actually contains and who can access it) is stored in a MySQL database.

As far as I know common filesystems like NTFS and ext3/4 can go dead-slow if there are too many files placed in one directory - that's why I ask.

share|improve this question

2 Answers 2

  • DO NOT ;)

  • The problem is not so much the file system (NTFS actually is ok with some hundred thousands) - it is all the tools around. Don't even DARE opening the directory in windows explorer. Shell scripts will take ages if a dir returns 2 million etc.

Better in a folder hierarchy.

Give every file a 16 byte hex code ;)

  • Make folders / File names in 4 char segments,

So a file would maybe be in affc/2548/2224.... etc.

Keeps the directories shorter ;)

AND you may be able to implement mount points here (though a 4 symbol filest level is too wide for that).

DO not forget, too, you need to possibly backup/restore that ;)

share|improve this answer

XFS: http://en.wikipedia.org/wiki/XFS

Do not use ext3 since it use a linked list for the content of a directory --> very slow directory with large set of file. Further more ext3 doesn't clean well this structure when removing files.

On linux, you can have an idea of the size of this list using ls -lahd directory/.

share|improve this answer
1  
Unless you use dir_index. –  Ignacio Vazquez-Abrams May 11 '10 at 18:48
    
right but you will still be limited by the 32K sub-directory limitation of ext3. Anyway I do not know if it matters here... –  aligot May 11 '10 at 19:06

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.