# Linux filesystem or CDN for millions of files with replication

Please tell me solution for this scenario:

• several millions files, located in one directory ("img/8898f6152a0ecd7997a68631768fb72e9ac2efe1_1.jpg")
• ~80k file size in average
• backup (replication) to other servers (every 5 minutes or immediately)
• metadata of images saves into database

When number of files have become greater than 2 millions, we got a problem with slow random access time. File system is ext3 with noatime and dir_index options, but there is no need to use commands like 'ls' or 'find'.

Solutions which I consider possible:

1. stay with ext3 and simply convert directory tree structure to "img/889/8f6/152/a0ecd7997a68631768fb72e9ac2efe1_1.jpg"
2. migrate to other file system (ReiserFS, XFS, EXT4 etc.)
3. setup storage engine with distributed filesystem (give examples)
4. or maybe other...

If we choose 1 or 2, how are we to replication? rsync can not handle with such a lot of datas on ext3 file system.

The best solution for us is to use Amazon S3, but this is too expensive with our traffic... Maybe you recommend some analogs (cheap CDN or open-source project)

• With those requirements I suspect you need to look at block level replication (eg drbd) and give up on any replication at the filesystem level. Though I don't really have direct experience with that kind of setup. – Zoredache Sep 4 '11 at 10:07
• where do the files come from? can you make a log of every file you change/add? Then replication can be limited to "send me everything since time X" – Flexo Sep 4 '11 at 11:29
• Files come from php script, metadata saves to database. Yes, "send me everything since time X" is possible solution for replication. – Roman S Sep 4 '11 at 11:36

Consider XFS, especially when using multiple drives and multiple directories, it may give you nice gains (see e.g. this thread for options to use. It give some tips for XFS on md RAID).
1. Stick with your current FS. Split them into directories like you suggested, if you want you can still present it as a single directory, e.g. with mod_rewrite (guessing this is a CDN type application)
2. Log changes that will need replicating, e.g. daily/hourly etc. such that every time you need to sync working out what files need to be copied can be as simple as running diff on the logs (i.e. you always sync the logs and sync them first but do a diff before replacing them to compute what else needs copying).