Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

What do you do when your site scales so much that your uploaded_media (in Django, where all uploaded media files from users is stored) can't fit on one machine?

My personal case, I'm using Django and have one really large uploaded_media server with over 600GB of storage (but that space is running out.)

Possible solutions I've brainstormed:
1) Try to build a bigger storage machine.
2) Move to a more abstracted system where the location and paths to uploaded_media are stored in a Database (each file is given a DB id and that database record stores where the file is stored across multiple uploaded_media servers.)

share|improve this question

closed as not constructive by John Gardeniers, Ward, Magellan, HopelessN00b, DerfK Nov 18 '12 at 4:41

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.

    
... Acquire a SAN? :-) –  voretaq7 Dec 17 '11 at 0:19

2 Answers 2

Move your static media off to another storage subsystem. That could either take the form of a large NAS, with NFS exports to your webserver or preferably, use an object store like Amazon S3 for your static media. With the right incantations, you can actually have your users' browsers upload files directly to S3. If that's not in the cards, though, you can store uploaded files locally for a short time and then have a job that runs periodically and uploads items to S3 (also updating your database as to their correct location).

share|improve this answer
1  
Can't say this better. S3 is really the way to go here, S3 also doubles in that it does pretty well as a lightweight CDN (even without CloudFront). Check out how CarrierWave and Paperclip work for Ruby on Rails, should give a nice starting point. –  Xorlev Dec 16 '11 at 3:36

A few options come to mind:

  1. Add more storage (i.e. a second hard drive) (and, possibly set it up as JBOD so it appears contiguous to the server).
  2. Consider a networked file system such as GlusterFS. This will let you have data stored on multiple machines, all of which is transparently accessible to the local machine.
  3. Consider using an external storage service such as Amazon's S3. You can mount this as local storage (e.g. using S3fuse), and it will scale with your needs (other than cost, there is a bit of a performance limitation in this scenario - it definitely isn't suitable for something such as a database, but for media files should be quite adequate).
  4. A bad option (at least in my opinion) would be to use a compressed file system (fusecompress, btrfs) to gain a bit of extra space - depending on the type of files, this could result in substantial space savings (e.g. for documents) or nearly none at all (for images/music/video). There is, of course, a significant performance hit that would be incurred.

It may be possible to combine some of the above (e.g. use your existing storage and S3 on top of GlusterFS for a seamless result - but the more layers you add, the poorer performance will be. Then again, for a setup that is primarily storing files, this may not be a major consideration).

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.