I need a better storage and archive system for my small business's files. Specifically the files are completed video projects. Beyond time and cost limitations what is holding me back is I don't believe in any of the solutions I have pondered. Therefore I am laying outthe problem and my thoughts. I would appreciate any opinions.
Budget: I believe in spending what it takes. That being said, we are a small business. I am hoping I can get out of this for <5k and more around 1-3k. That might be a pipe dream. Just tell me so.
- Raw video files are huge in filesize. We have accumulated probably 10+tb so far and that is growing fast.
- Video editing require fast read/write access to files so a central or cloud based file server will not be fast enough. Therefore we probably need an achieve solution for old projects and current projects will have to stay local.
- We want some sort of redundancy and offsite solution.
What we currently do:
- We use large, high quality, external hard drives.
- We always buy in pairs and manually duplicate content. In other words, we work off of one, and duplicate the files to the other which serves as a backup/fall back.
- These HDs are fast enough with firewire800 or USB3 to directly work off of.
- Once filled, we set the pair aside.
What's wrong with the current solution:
- Although the data is duplicated across two drives, these drives are not "backed-up" or stored offsite.
- Organization across these many external HDs is hard. What project is on what drive? etc.
- Eventually we are going to have a ridiculous amount of hard drives.
- Duplication is not RAID.
A Local Server
- Buy a rack mount server and a rack mounted hard drive array enclosure, like a Norco, (SAS) (20 bays).
- All video files would be stored on this server. We could install and pay a cloud service to back up this one computer/server. CrashPlan works on Linux and has no limits on how much data. The har ddrives would be physical drives connected to the server so we get around the "no NAS" rules companies like CrashPlan have. It is not a personal computer so the syncing can run 24/7/365. This would solve the offsite issue.
- Instead of using an online backup service like CrashPlan we could write a script to sync these files to an Amazon Glacier account.
- A policy that video peeps work off of external hard drives for current projects but must put the project on this new computer when complete. In other words, continue using external hard drives for current projects and store archived projects on this server.
Cloud based backup services (CrashPlan.com, BackBlaze.com, Carbonite.com)
- Typically only let you backup an external harddrive that is physically connected to a computer. (no NAS or network drives).
- Typically they expect a backed up external drive to stay connected to your computer and all data to remain on the drive. If you don't hook up an external harddrive for months, what happens to the backups? If you clean up space by deleting old projects, they will be deleted from the online service too.
- Requires our users to leave the external harddrives connected to their computer until all data is in the cloud. This can take weeks for a big project.
- Restoring a project would be very slow due to internet transfer speeds.
- These cloud backup accounts are usually specific to one user/one computer. So if a harddrive is backed up by one user. Then a second user works on the project, what does that mean?
A Big NAS
- A NAS is "Network Area Storage". You stick in as many hard drives as it will hold. It will raid them. You can access this via the network connection or maybe USB3/Firewire.
- Most have an Operating System baked into it. So you can't run other software like cloud based backup services. Nor can you do any customization or run your own software. You get what you buy.
- Big NASs are pretty expensive and not really that big. You don't find many with more than 4 bays. Currently a big HD is 3tb. So 4bays might be somewhere around <12tb of storage. Not super comfy for the future.
Other ideas are:
- Tape Backups.
- Just archive the older projects directly to Amazon Glacier, Skip building a local server to store them.
Thanks for any advice!!! Jed