4

Please tell me solution for this scenario:

  • several millions files, located in one directory ("img/8898f6152a0ecd7997a68631768fb72e9ac2efe1_1.jpg")
  • ~80k file size in average
  • 90% random read access
  • backup (replication) to other servers (every 5 minutes or immediately)
  • metadata of images saves into database

When number of files have become greater than 2 millions, we got a problem with slow random access time. File system is ext3 with noatime and dir_index options, but there is no need to use commands like 'ls' or 'find'.

Solutions which I consider possible:

  1. stay with ext3 and simply convert directory tree structure to "img/889/8f6/152/a0ecd7997a68631768fb72e9ac2efe1_1.jpg"
  2. migrate to other file system (ReiserFS, XFS, EXT4 etc.)
  3. setup storage engine with distributed filesystem (give examples)
  4. or maybe other...

If we choose 1 or 2, how are we to replication? rsync can not handle with such a lot of datas on ext3 file system.

The best solution for us is to use Amazon S3, but this is too expensive with our traffic... Maybe you recommend some analogs (cheap CDN or open-source project)

  • With those requirements I suspect you need to look at block level replication (eg drbd) and give up on any replication at the filesystem level. Though I don't really have direct experience with that kind of setup. – Zoredache Sep 4 '11 at 10:07
  • where do the files come from? can you make a log of every file you change/add? Then replication can be limited to "send me everything since time X" – Flexo Sep 4 '11 at 11:29
  • Files come from php script, metadata saves to database. Yes, "send me everything since time X" is possible solution for replication. – Roman S Sep 4 '11 at 11:36
4

Millions of files in one directory is bad design and will be slow. Subdivide them into directories with smaller number of entries.

Take a look at https://unix.stackexchange.com/questions/3733/number-of-files-per-directory

Use RAID and /or SSDs. This will not in itself solve the slow access times, but if you introduce multiple directories and reduce the number of files per directory, say by an order of magnitude or two, it will help to prevent hotspots.

Consider XFS, especially when using multiple drives and multiple directories, it may give you nice gains (see e.g. this thread for options to use. It give some tips for XFS on md RAID).

2

Personally I would:

  1. Stick with your current FS. Split them into directories like you suggested, if you want you can still present it as a single directory, e.g. with mod_rewrite (guessing this is a CDN type application)
  2. Log changes that will need replicating, e.g. daily/hourly etc. such that every time you need to sync working out what files need to be copied can be as simple as running diff on the logs (i.e. you always sync the logs and sync them first but do a diff before replacing them to compute what else needs copying).
  • Do I have enough performance ext3? – Roman S Sep 4 '11 at 12:01
  • AFAIK if you keep the depth/breath ratio sensible it should be fine – Flexo Sep 4 '11 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.