I've a big application with user uploaded images. The storage is increased by ~1gb every day.

I need to backup all images somehow. Backing up all images every day could become a big problem (big amount, network traffic etc).

What are best ways to backup this sort of data? Which storages are preferred (FTP, S3, local server, etc)? Which backup strategies are best?

migrated from stackoverflow.com Jan 17 '10 at 8:52

This question came from our site for professional and enthusiast programmers.


Couple pointers:

First, snapshots. Any backup operation (even an incremental backup, probably) is going to take a significant amount of time to complete. Unless you can lock your store while the backup is taking place, you will probably have a problem with inconsistent data. I suggest you investigate your fs and system possibilities for taking snapshots. You lock the store, take the snapshot (which should be very, very quick) and unlock the store. Your snapshot now contains a consistent replica of your store which can be backed at any pace you need. Look for volume shadow copies in Windows land, LVM snapshots in Linux, etc.

Second, replication. Again, any backup job is probably going to take a decent chunk of IO and CPU performance away from the store. Either your main app's performance will be affected or your backup will take ages to complete. If you can keep your storage system continuously replicated to a second system, you'll be able to back up without worries of affecting your live, production system. This replica might also be useful to prevent downtime (if the main storage fails, switch to the replica). When using replication, you take the snapshots in the replicated system- needing only to pause the replication while the backup is taken.

Third, once you have replication and snapshots, you just need to find a backup method which makes sense wrt. to bandwidth, storage and your requirements. First of all, figure out how back you want to go when recovering. Do you only need a copy of your storage as it was yesterday? 8 hours ago? Do you need to go back and recover files from last week? Last month? Last year?

How long does a full copy of your storage take? How long does a copy of a daily increment take? This limits how often can you take full backups and incremental backups. If you are not deleting data from your store, a replica might be enough.

Last, if you are moving huge volumes of data, you are probably using a SAN. And have a service contract. Your SAN probably has something builtin to handle backups, or at least the guys that support it should give you more specific ideas...


Do you want to programatically remove older versions of the assets?

If so you could build a mechanism that archives older images based on specific parameters. These files will possibly be compressed to archives that are removed from machines. Older versions could be marked as archived with some reference to where they will be stored remotely/physically on Disc/Tape etc for later retrieval.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.