Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Please tell me solution for this scenario:

  • several millions files, located in one directory ("img/8898f6152a0ecd7997a68631768fb72e9ac2efe1_1.jpg")
  • ~80k file size in average
  • 90% random read access
  • backup (replication) to other servers (every 5 minutes or immediately)
  • metadata of images saves into database

When number of files have become greater than 2 millions, we got a problem with slow random access time. File system is ext3 with noatime and *dir_index* options, but there is no need to use commands like 'ls' or 'find'.

Solutions which I consider possible:

  1. stay with ext3 and simply convert directory tree structure to "img/889/8f6/152/a0ecd7997a68631768fb72e9ac2efe1_1.jpg"
  2. migrate to other file system (ReiserFS, XFS, EXT4 etc.)
  3. setup storage engine with distributed filesystem (give examples)
  4. or maybe other...

If we choose 1 or 2, how are we to replication? rsync can not handle with such a lot of datas on ext3 file system.

The best solution for us is to use Amazon S3, but this is too expensive with our traffic... Maybe you recommend some analogs (cheap CDN or open-source project)

share|improve this question
    
With those requirements I suspect you need to look at block level replication (eg drbd) and give up on any replication at the filesystem level. Though I don't really have direct experience with that kind of setup. –  Zoredache Sep 4 '11 at 10:07
    
where do the files come from? can you make a log of every file you change/add? Then replication can be limited to "send me everything since time X" –  Flexo Sep 4 '11 at 11:29
    
Files come from php script, metadata saves to database. Yes, "send me everything since time X" is possible solution for replication. –  Roman S Sep 4 '11 at 11:36

2 Answers 2

up vote 4 down vote accepted

Millions of files in one directory is bad design and will be slow. Subdivide them into directories with smaller number of entries.

Take a look at http://unix.stackexchange.com/questions/3733/number-of-files-per-directory

Use RAID and /or SSDs. This will not in itself solve the slow access times, but if you introduce multiple directories and reduce the number of files per directory, say by an order of magnitude or two, it will help to prevent hotspots.

Consider XFS, especially when using multiple drives and multiple directories, it may give you nice gains (see e.g. this thread for options to use. It give some tips for XFS on md RAID).

share|improve this answer

Personally I would:

  1. Stick with your current FS. Split them into directories like you suggested, if you want you can still present it as a single directory, e.g. with mod_rewrite (guessing this is a CDN type application)
  2. Log changes that will need replicating, e.g. daily/hourly etc. such that every time you need to sync working out what files need to be copied can be as simple as running diff on the logs (i.e. you always sync the logs and sync them first but do a diff before replacing them to compute what else needs copying).
share|improve this answer
    
Do I have enough performance ext3? –  Roman S Sep 4 '11 at 12:01
    
AFAIK if you keep the depth/breath ratio sensible it should be fine –  Flexo Sep 4 '11 at 12:26

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.