5

How we back up

We currently have a file server that is backed up to tape. The backups are incremental which means the files that have changed since the last backup get written to tape. Over time the tape sets get larger and larger. Once they reach a certain size our backup software grooms the sets back down by removing old versions of files.

The problem

Under that scheme if a file gets deleted from the server by accident we have a limited amount of time to find out and restore it. Otherwise all versions of that file will get groomed out of the backup tapes after enough time. That's not an acceptable risk.

Possible solutions?

The only solution I see is to stop grooming our tape sets. The sets will continue to grow in size and once they become unwieldy perhaps we just stow them somewhere off-site and start a new set. Am I missing a better solution? Is such an indefinite file retention policy just unreasonable?

2 Answers 2

5

Backup to tape is great for long term retention, but for recovering from mistakes, we used backup to disk.

Depending on your total storage, this can be a cheap way of keeping at least one copy of every file cheaply and quickly.

Say you have a total of 4TB of storage to backup. It doesn't cost much to build a 12TB backup volume and backup to there. Then keep your tapes for long term archival, doing full backups and shipping them off site or into a fire proof safe, etc.

2
  • So, have a huge backup RAID onsite with incremental backups for going "back in time" and save the off-site tapes for regular backups for disaster recovery? Sounds like kind of a brute force solution but I guess keeping every incremental backup ever is a pretty tall order.
    – James M
    Jan 24, 2013 at 21:28
  • The other advantage of disk-based backups is de-duplication. Whether you dedupe with the filesystem (ZFS) or inside the backup utility (almost all of them), your additional copies of the files can take up almost no space. A 10Mb file that has 500kb changed 10 times might only take 15mb instead of 50mb. You can get some serious retention that way. Jan 24, 2013 at 22:56
7

Typically, if you care about recovering your data after a disaster in an acceptable amount of time, full backups are done regularly (weekly, bi-weekly, whatever) and incrementals or differentials are done daily. Differentials will allow you do do a full recovery in three steps:

  1. Restore the latest full (which will only be a week or two old).

  2. Restore the latest differential.

  3. Go have a snack.

Incrementals will require you to restore every single tape since the last full backup was taken. This is not a good thing if you just have never-ending incrementals.

Also, keep in mind that backups stored on-site are subject to destruction in a disaster just like your servers are (flood, fire, angry ex-employee). Send them off-site. Send them off-site regularly. Tape rotations are quite common.

I have about 200 tapes in my rotation right now. We use about 12 for a full every weekend, and between 3-5 per day for the differentials. We have 5 tape "sequences" that are basically a week long. Every day a courier picks up the day's tapes and brings back a box from 5 weeks ago. The only tapes that we keep on hand are what we need for the week's backups. Everything else is off-site in a hardened facility, in a fireproof case. This is the only way to do it if you want to recover from a disaster.

When the tapes return from the DR facility and are fed to the hungry tape robot, they're added to the scratch pool and are overwritten. At most, I will only ever lose a day's worth of data and I can go back as far as 5 weeks if necessary. There are other policies for certain things that legally need longer retention, but that's another story and they're the exception not the rule.

Of course disk-to-disk is good too for quick restores, but offite archives are a must for DR.


tl;dr - Most people don't just use the same few tapes over and over. The best way to do things if you care about archival recovery in the event of a disaster is to have multiple tape sequences that are cycled off-site, and just overwrite the oldest when they come back unless you have a legal reason not to.

4
  • Sorry, I omitted a few details from our situation. We do have daily incremental backups to disk and well as our weekly and monthly tape sets which live off-site. The problem I'm struggling with is that those tape sets keep getting bigger and bigger because we do incremental backups to them. To keep the sizes manageable we refresh them from time to time but then we lose old versions of files we may need.
    – James M
    Jan 24, 2013 at 21:21
  • How far back do you need to go? If this is for legal reasons, you just need to eat the cost of more tapes and storage. If its a business reason, how often are you going to have to go to tape backups that are older than a month or two? Especially if you're doing something like an incremental-forever dedupe pool on your D2D backups.
    – MDMarra
    Jan 24, 2013 at 21:53
  • Technically we need to be able to go back to pretty much the beginning. If someone accidentally deletes a file and we don't catch it for two years we still have to be able to restore that file from somewhere. It would be rare that we ever have to go that far back but we still need the option. I'm warming up to Mark Henderson's answer below about building a large cheap backup volume/server/NAS to store all our incremental backups.
    – James M
    Jan 24, 2013 at 22:08
  • Yup. That sounds like the way to go. Just make an incremental-forever deduped pool and call it a day
    – MDMarra
    Jan 24, 2013 at 22:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .