2

What do you do when your site scales so much that your uploaded_media (in Django, where all uploaded media files from users is stored) can't fit on one machine?

My personal case, I'm using Django and have one really large uploaded_media server with over 600GB of storage (but that space is running out.)

Possible solutions I've brainstormed:
1) Try to build a bigger storage machine.
2) Move to a more abstracted system where the location and paths to uploaded_media are stored in a Database (each file is given a DB id and that database record stores where the file is stored across multiple uploaded_media servers.)

1
  • ... Acquire a SAN? :-)
    – voretaq7
    Dec 17, 2011 at 0:19

2 Answers 2

5

Move your static media off to another storage subsystem. That could either take the form of a large NAS, with NFS exports to your webserver or preferably, use an object store like Amazon S3 for your static media. With the right incantations, you can actually have your users' browsers upload files directly to S3. If that's not in the cards, though, you can store uploaded files locally for a short time and then have a job that runs periodically and uploads items to S3 (also updating your database as to their correct location).

1
  • 1
    Can't say this better. S3 is really the way to go here, S3 also doubles in that it does pretty well as a lightweight CDN (even without CloudFront). Check out how CarrierWave and Paperclip work for Ruby on Rails, should give a nice starting point.
    – Xorlev
    Dec 16, 2011 at 3:36
2

A few options come to mind:

  1. Add more storage (i.e. a second hard drive) (and, possibly set it up as JBOD so it appears contiguous to the server).
  2. Consider a networked file system such as GlusterFS. This will let you have data stored on multiple machines, all of which is transparently accessible to the local machine.
  3. Consider using an external storage service such as Amazon's S3. You can mount this as local storage (e.g. using S3fuse), and it will scale with your needs (other than cost, there is a bit of a performance limitation in this scenario - it definitely isn't suitable for something such as a database, but for media files should be quite adequate).
  4. A bad option (at least in my opinion) would be to use a compressed file system (fusecompress, btrfs) to gain a bit of extra space - depending on the type of files, this could result in substantial space savings (e.g. for documents) or nearly none at all (for images/music/video). There is, of course, a significant performance hit that would be incurred.

It may be possible to combine some of the above (e.g. use your existing storage and S3 on top of GlusterFS for a seamless result - but the more layers you add, the poorer performance will be. Then again, for a setup that is primarily storing files, this may not be a major consideration).

Not the answer you're looking for? Browse other questions tagged .