Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm going to be generating about 50 million HTML files that I'd like to serve. Each file has a unique id (e.g., thingy) and I'd like to serve them up as if they were all in one directory (e.g., example.com/thingy).

I suspect that putting 50 million files in one directory is asking the gods to strike me down, so I'm inclined to do it with nested directories (e.g., thingy is in /t/h/i/thingy). I think I could do this with Apache and mod_rewrite without too much pain, but I'm wondering if there are other options that make more sense.

If it matters, I intend to do this on Linux.

share|improve this question
    
Why do you need 50 million html files ?? –  Arenstar Nov 17 '10 at 0:14
    
Because I have 50 million things I want to index and display, and the options for searching and displaying that many HTML files are free, easy, and well tested. –  William Pietri Nov 17 '10 at 0:39
    
I've done this before with scientific data, and while mod_rewrite will work, this will be a pain to deal with for future generations. Why not use a database to hold the 50M widgets with a built-in fulltext index? A simple PHP/python/perl/ruby CGI can serve them up dynamically. –  hurfdurf Nov 17 '10 at 1:09
    
The source data for this is about 2TB of complicated JSON. For this activity, a few people want to be able to perform fast arbitrary text searches and view the related data in an interface less hideous than JSON globs. If the requirements change to something more complicated, I'll certainly do something more complicated. –  William Pietri Nov 17 '10 at 1:25
    
I see somebody changed "50m" to "50 million" on the grounds that "m" doesn't mean million. That's fine, and I appreciate the help. However, for the record, "m" does also mean million: economist.com/research/styleGuide/index.cfm?page=738514 –  William Pietri Nov 17 '10 at 1:38

3 Answers 3

up vote 5 down vote accepted

Are you certain that all (or a majority of) the 50M files will be requested? If not, and if your problem domain allows it, you could consider taking a "lazy computation" approach. That is, only generate (and then cache) those files that are actually requested.

Still, yes, you will want to use a nested directory structure (say 3+ levels deep), so that no single directory gets more than a few thousand files in it. Then, use mod_rewrite to convert requests to the actual physical file names, something like the following (but probably with more checks and logic):

RewriteRule (\w)(\w)(\w)(\w+) /path/to/cache/$1/$2/$3/$1$2$3$4 [L]

Finally, some filesystems are better at efficient handling of large numbers of files than others, so you may want to do some testing and benchmarking with a few candidates (e.g. ext4, xfs, jfs, reiserfs) before going into production.

share|improve this answer
    
Good question! In my case, I need to index them all, and an unknown fraction will be viewed. I have to write the HTML renderer regardless, so it seemed easier to just do a rendering pass and then turn a standard HTML indexer loose rather than writing a custom indexer that works from the same raw data. –  William Pietri Nov 17 '10 at 1:01

No, mod_rewrite is how you would do this.

share|improve this answer

For this kind of thing I would use a database, and serve the files from the database. You may be able to template the page so you don't have the full page in the database.

Depending on how you are generating the pages, consider putting the source you generate the page from in the database and generate the pages as needed. There are caching techniques that can be used to prevent the need to generate the page each time it is requested.

share|improve this answer
    
I appreciate your good intentions, but I'm not seeing how that's an improvement on taking my 50m JSON files, transforming them to 50m HTML files, and then feeding them to a web indexer. Involving a database and dynamic page generation plus a cache would require more code, more resources, and more time. –  William Pietri Nov 17 '10 at 1:33
    
You specified you were generating HTML files. Even if you can't generate the pages I would still consider putting them in a database. For JSON I believe a NoSQL database may be appropriate. –  BillThor Nov 17 '10 at 1:52
    
This makes no sense to me. Involving a database would only slow things down. Apache serving static files is incredibly fast and totally bulletproof, and the code is much simpler to write. –  William Pietri Nov 17 '10 at 2:08
    
Apache may serve the data up quickly, but the database may do the search quickly without having to reindex the website on each change. However, a rewrite scheme such as proposed by Steven is the way to handle putting the files on the file system. –  BillThor Nov 17 '10 at 2:32
    
Vote++. I would not be wanting to rely on a file system to index 50M files. MySQL/MSSQL/Oracle will eat up this workload, especially as its textual. I also do not believe involving a DB will slow things down; an indexed DB, pulling compressed textual data from a handful of large binary files will be rapid. A file system index dealing with 50M objects may well be slower. And then there's the reliabilty aspect, backup and recovery, etc. –  Simon Catlin Nov 17 '10 at 20:45

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.