24

We have 72 hard drives that contain our webcast inventory. The number is increasing. We're located in a frame building and we are afraid of not only fire, but catastrophic fire.

I've priced fireproof safes that hold to the required 125F for hard drives. Their price is through the roof.

Seems to me if we made backups of each of the hard drives and stored them off-site somewhere, or contracted with an online backup storage company, we might run up a bill buying backup drives that would approach the $7,000 cost of the safe!

What's the best way to protect our data from the risk of fire?

9
  • 37
    You don't have off site backups? You hvae bigger problems than the risk of fire... Dec 7, 2012 at 0:28
  • 2
    Now with that out of the way, exactly how much data do you have here? Hard drives vary in size, you know, so "72 hard drives" doesn't give us a good sense of the scope of the problem. Dec 7, 2012 at 0:36
  • 18
    only fire? what about a water pipe bursting and flooding the safe? a tornado carrying off the safe? There is no substitute for offsite backup. Comparing the cost of disks to the cost of a safe is invalid, you need to compare it to the cost of losing your webcast inventory.
    – jqa
    Dec 7, 2012 at 0:38
  • 26
    Your data isn't worth $7,000?
    – psusi
    Dec 7, 2012 at 3:05
  • 2
    Would a content delivery network be more appropriate and possibly as cost effective as self hosting and off-siting; especially in this case of webcast archives?
    – JustinC
    Dec 7, 2012 at 10:13

8 Answers 8

29

Well, first of all, you should always have an off-site copy of your data. Not just because there might be a fire, but what would you do in the event of a natural disaster, or similar event making your building inaccessible? Without a copy of your data off site, you're just hosed.

So, that said, there are two ways to have an off site copy of your data.

  1. Offsite Backups.

    • Typically you contract with a company like Cintas (the big name in that area I'm familiar with) to take your backup tapes to one of their facilities, but even something like having an employee take the daily backups home with him at the end of the day is better than nothing.

  2. A co-location facility of some sort.

    • Somewhere else that you have a server or servers to house your data, either a physical site, like a data center or a cloud service.

The advantage of #2 is that you can use it for #1 as well, in addition to the ability to have a business continuity plan in the case of a disaster. You can put in a backup system (hopefully in addition to the one you have at your main site), so you have off site backups, and if your main site goes down, you can run your services from the co-location site (often in a somewhat decreased capacity, if cost-savings are an issue).

Neither is particularly cheap, and may well exceed the price of a fireproof safe, but the advantages are that they provide something you can't get from a fireproof safe, which is protection against all types of data loss, whereas a fireproof safe only protects against fire.

The business case/cot justification for this is basically asking the question "what happens if we lose our data?" The answer is almost always that you go out of business, which makes it fairly easy to convince the people in charge of the need for a proper backup scheme and/or co-location facility. (And most places I've worked have had both, even if only because the techs pushed for them until they happened.)

7
  • 4
    Or a real world event that almost happened to us: Someone stealing the server. (It appears they ran out of space to put things when the server was the next thing they would have taken.) Dec 7, 2012 at 3:54
  • 1
    @LorenPechtel Yeah, latest real-world event we had was a freak 30 minute storm that knocked out power to most of the city. Our generator didn't work, so we ran out of our colo facility for 4 days or so while power was being restored to our building. A lot of good things to be said for having a fail-over site, and it comes in handy a lot more often than you'd think. Dec 7, 2012 at 4:09
  • 5
    Absolutely. And if you want to hear a thrilling story how something like this can play out, then listen to this SE podcast episode. Dec 7, 2012 at 8:27
  • 4
    Fireproof safes won't save you from a hard drive failure. You should have the data replicated to a remote server location specially designed for backups.
    – aglassman
    Dec 7, 2012 at 15:51
  • 1
    just to be clear, if you do have people take backups offsite you need more than one offsite backup and a rota. Else all day every day your only "offsite" backup is actually in the office until fred goes home. I've seen companies do this, it's scary.
    – Sirex
    Dec 10, 2012 at 23:16
20

Back them up on Amazon glacier. Cheap storage for things that you don't need to access often. It will cost you 1cent per GB per month. As a downside, it might be pricey to fetch your data back in case of failure.

12
  • 2
    -1. This may not work due to the TIME it takes to reconstruct.
    – TomTom
    Dec 7, 2012 at 14:09
  • 2
    +1 If there's a fire, you need to rebuild your datacenter anyway.
    – mgjk
    Dec 7, 2012 at 14:56
  • @mgjk Really? This is small stuff.
    – TomTom
    Dec 7, 2012 at 15:42
  • 6
    Before using Amazon Glacier, you should always calculate the bandwidth charges to recover the data. It's not such a great deal if actually recovering your backups costs more than you'd spend on a physical backup solution. With large amounts of data on Glacier, that very well could be the case. Dec 7, 2012 at 15:54
  • 1
    @TomTom ok, you need to rebuild your office with your data closet. If your company is only big enough for a data closet, it's still going to take time to rebuild.
    – mgjk
    Dec 7, 2012 at 17:11
9

If the data's critical, you should be backing it up. Your backup scheme should include backups to tape and offsite storage of some sets of tapes.

Beyond that, it's all details... how much data? how often does it change? what sort of recovery time do you need?

If "webcast inventory" means large files that don't change frequently, then a periodic backup to tape seems pretty straightforward.

3
  • 3
    Tapes are a tried and true method of managing offsite data retention requirements, but there are a variety of other methods if tapes don't work for a particular situation.
    – Chris S
    Dec 7, 2012 at 0:45
  • 2
    Hardly. Tapes are still most sconomical, lowest cost and safest. Not aware of any alterantive that is widely usable - well, except maybe a microwave link to another building or something "high bandwidth wan". I once knewa company building 2 data centers, wownwning all the land bewtween them and having a direct fiber connection for this reason.
    – TomTom
    Dec 11, 2012 at 5:40
  • Tape is certainly going to have the lowest media cost.
    – mc0e
    Feb 21, 2017 at 3:37
4

You don't really mention the SIZE of the 72 drives, but if it's really critical then you should at least have a backup at a remote location. Media rated firesafes are great for protecting the data on-site, but what if your building is a total loss. As in, everything is gone, including the safe. If you do start utilizing off-site or online backups, make sure that their building/ data center isn't close enough to yours, that they could both be destroyed at once.

3

You need to imagine: what happens if I can't access part of my data? Usually this could mean you go out of business! (You can't just rely on the hope that that small part that fails will not be the core data necessary for your company's operation, for example.)

You'll probably need:

  • redundancy of data (ie, have your data on at least 2 separate piles of disks, or a version on disk and another on tapes, or a combination thereof)
  • redundancy of location (ie, have a good off-site backup, in case you experience fire/theft/sabotage/flood/? on your primary location)

You can combine both, but off-site usually takes some time to retrieve (unless you also have off-site office, or access to it via some means, check with your off-site backup providers)

About the need for redundancy :

Hard Drives (actually, any storage medium) ages, and some have defects, and some gets bumped (for example, a little knock with a screwdriver or against another drive while storing them on a shelf could yeld enough G to scratch a disk's surface, and this will cause failure either now or within a short-to-not-long-enough timeframe). Which means: your backup media will fail, some within days, others within years, with no means to know when.

You can't know when something will go wrong, but you can assess the risks (not to have precise infos, but to get a sense of what can go wrong and why it's good to prevent it with redundancy and with regular transfer to other mediums before the prior mediums fails):

Imagine that you have similar disks and that you can say that the probability of 1 failing within 2 years is 5%

P(one disk fails within 2 years)=5%=0.05

P(that disk won't fail within 2 years)=0.95
P(you have n disk, 0 disk fails within 2 years)=(0.95)^n
P(at least 1 disk amongst n fails within 2 years)=1-(0.95)^n
 if n=72 (you have 72 disks), P=1-0.95^72=1-0,024894281=0,975...=more than 97% 

In english : if you have 72 disks, and if you can say that a single disk have a 5% probability of failing within 2 years, then you'll have more than 97% chance you have a disk failure within 2 years. This is not at all precise, but it should show you that things can go wrong, and faster than usually expected...

To "scare" you even more, you could say now that P(one disk fails within 20 years)=0.99 (20 years is a long time for a drive. And even if it works, the devices to read them may not be able to handle them)

P(that disk won't fail within 20 years)=0.01
P(no disk amongst n fails within 20 years)=(0.01)^n=0
P(at least one disk amongst n fails within 20 years)=1-(0.01)^n=1=100% (or so close to 100% that it doesn't matter)

And in addition to all this: if you buy all the drives at the same vendor, P(one of those disk fails after another failed) is very high, much higher than the P(one disk fails) : disks are built similarly, so if one fails, chances that the others in the same batch will fail around the same time is high (or very high) !

So I recommend:

  • backup on at least 2 separate things (2 x 72 hd, from 2 separate vendors ? or hd and tapes? ).
  • Have at least one (if possible, yet another one) backup stored in another town. Update this every n days (weekly at least?) so you can go back to the backup time in case the primary storage all fails (ie: you'll be able to go back to the data of last week in case the primary storage fails this week)
  • check backups regularly (I mean, do check them, reading the whole thing byte per byte at least once a month. A good time to also copy to yet-another-backup media too.)
  • ensure at least 2 people know about this and can help retrieving the data when the external backup is needed.
  • etc.
1

Similar to @longneck's suggestion, but slightly different: backup to Amazon S3, or Google Cloud Storage, etc. (I recommend NOT using Amazon's Glacier if you are looking for a warm backup for your data, since you will be paying through your nose for fast data restoration from Glacier. For more on Glacier's retrieval fee, there's great discussion on Hacker News.)

Amazon S3 costs you $0.095 per GB for starters (it used to cost $0.125 per GB, but Amazon just dropped it a week ago). Google's Cloud Storage and other services are probably in the same price range.

In addition to the cost and "infinite growing storage", you also get fine-grained ACL and permissioning for your data. Amazon offers IAM for ACL control, and I suspect other providers offer similar control.

Finally, these cloud storage services give you really powerful APIs to read/write and do other things with your data. You may find this useful for automating some backup tasks.

0

If tapes don't row your boat, try backup to disk.

Buy two more disk arrays and have them fitted into shock mounted travel cases. connect them via something with lots of bandwidth like fiber.

You rotate them on-site, mirror to them and have them stored off-site at a secure storage facility.

Similar to tape backup but may cost out better depending on your data volumes and data production volume per week.

0

I would like to suggest another alternative to your question. By all means you need to have an offsite backup, thus giving you redundancy. I found a NAS type drive bay by a company called Synology they have many versions of drive bays, you just need to look in to what a good fit for you would be.

The best part is that these drives can RSYNC each other, even to an offsite location and they can do a block level backup so that only the changes to a file get backed up, not the whole file if it isn't needed. So your data is essentially duplicated to another location and the backups take lass time and bandwidth. I like the DS1812+ but they have better faster drive bays then that for larger organizations or faster needs. I will paste a link to the company site below:

http://www.synology.com/us/index.php

I hope this helps you out, good luck.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .