Good backup strategy for heterogeneous data consisting of images/databases/office files/svn repositories/

Question

I'm looking for a simple on-site (i.e. not online) backup solution for our small company. Right now we have approximately 4TB of data in total, maybe adding ~500GB per year. The amount of data changing per day is much less tough - I guess much less than 1GB on average.

All data is only accessed from the intranet and most machines are running Windows and some are running under MacOS if it matters.

The data in detail:

(a) Large parts of the data are images/videos/documentation(pdf) and alike, I would guess 2.5TB.

(b) Frequently accessed are our CAD-data files, but they only take up 10-20GB. These are controlled/accessed by a centralized CAD vcs called GAIN (I think it keeps it's data in a binary database). Currently this is dumped on the evening and then backed up.

(c) Some primarily source-code data are already under version control (SVN, GIT) taking up less than 2GB.

(d) Some programs only have binary source codes and are "archived" as zip-files. New versions are added and some old versions are restored sometimes but old versions never get changed. These programs take up roughly 80GB.

(e) Some personal backups (emails etc.) and other stuff takes roughly 1TB I guess.

(f) We also have a small amount of data on a single Microsoft SQL server. This should make up less than 1GB.

Right now we do a full backup every Monday to Friday evening from network disks to local server disk to tape drive on the server. We alternate the Friday tape, i.e. we have tapes labelled mo,tue,wed,thu,fri1,fri2. This implies we cannot go back in time more than 2 weeks in the worst case.

What's a good solution to this heterogeneous system consisting of

(a) large seldom accessed, seldom changed, seldom added data,

(b) frequent accessed rather small data delivered by a program internally using a database,

(c) frequent accessed rather small data under "common" version control,

(d) large binary files (~100MB) which are mostly added, seldom read, never changed (should optionally be disposable) and

(e) miscellaneous data like office files, data logs, mail folders which are seldom added/changed

(f) data on Microsoft SQL server

I'm firm with programming, version control and computers in general but new to backup strategies. So it would be good if the solution is quite simple to maintain.

If possible a versioning like that offered by SVN/Git would be nice, so the last successful backup allows restoration of every single file ever backed up (and not manually removed).

Problems with the strategy so far:

• backing up takes a long time (15 hours)

  => There's not enough time to test the backup

  => Hard to tell if the backup is really working

  => What to do if the backup time reaches 24 hours?

• restoring a backup is quite a pain
• restoring something I deleted/modified/overwrote a month ago is not possible

A solution should address all these problems.

Time usage in detail:

• Gathering of data from other servers over network to backup server: 02:15

• Copy data on backup-server (which acts as a "regular" server as well) to another drive on the backup server: 09:00

• Copy all data from internal drive on backup server to tape attached to backup server: 03:45

Answer 1

1. backing up takes a long time (17 hours) - Perform a Full backup on the weekend and perform Incremental backups during the week. This will reduce the backup window during the week and will also reduce the amount of storage needed for your backup sets.

2. There's not enough time to test the backup - What are you testing exactly? You should perform a test restore of small data sets from your backup sets every week or every month. You don't need to test restore the entire backup set. Restore a handful of files and a database or two.

3. Hard to tell if the backup is really working - See number 2. You need to test restore data from the backups to know if they're working. You should do this often enough that you're confident that the backups and backup process is reliable from week to week.

4. What to do if the backup time reaches 24 hours? - See number 1.

5. restoring a backup is quite a pain - How so? Is it the process? The backup software? Etc., etc.

6. restoring something I deleted/modified/overwrote a month ago is not possible - Acquire enough backup media to meet your recovery needs. Determine how much backup media is needed per week and how many weeks you need to be able to go back and recover from. Then multiply the two. This will give you a rough idea of how much backup media you need and will help you determine your backup media rotation schedule.

EDIT

To address your comment:

As far as restoring data is concerned, it depends on the backup software and what type of backup media you use. BackupExec uses an on tape and on disk catalog of backups. Finding data that needs to be restored doesn't require "reading" the tapes until you find the data. It only requires finding the data in the Restore selections window in BackupExec. Once you've found the media that the data resides on it's a simple matter of providing that media to BackupExec. To further this point, BackupExec recommends performing backups to disk and then duplicating (copying) those backups to tape. If you provide enough disk space to run a full week of backups then all of the data that you could possibly need to be restore during the entire week will be on disk and won't require you to swap tapes around at all. You'll simply select the data to be restored and BackupExec will find it in the on disk media.

As for the type of backup to perform, that's up to you. I recommend a weekly Full and a daily Incremental because the daily Incremental backups will run faster and will be smaller than a daily Differential backup, saving you time and money (in terms of backup window and backup media). The case where a Differential backup would be needed to restore data is far and few between and I've never actually encountered that scenario in 13 years in the IT profession.

Answer 2

This smells fishy, sorry.

Right now we have approximately 4TB of data in total, maybe adding ~500GB per year

4000gb is not a large backup by any means and should not take 17 hours. How you do that - 1gbit network? Maybe time to put in a decent infrastructure. 10g backbone for the backup servr, something like MIcrosoft DPM with local change agents and the functionality to allow single file restores for users, 10-12tb of disc space in the backup server to keep backups some time on disc (for fast restore by users).

This is all well known and documented stuff - it looks to me like this is mostly your definition of how to make backups that is bad. from not enough hardware to not enough software. You should reevaluate your setup.

Answer 3

I'm trying to summarize what advice has been given:

• get an dedicated backup server so that production servers are not free to change once all data is on the backup server
• if the time for backup is too large switch from full backup every day to full backup on Friday and incremental/differential backups from Monday to Thursday
• Backup tests will suffice when done on Friday backup - or when we have a dedicated backup server maybe everyday (as long as it's automated and therefore doesn't take my time)
• Get enough tapes so we can restore data from long ago
• To speed up recovery one might even consider making a backup to disk in addition to backups on tape
• Incremental backups during week are to be preferred since they are faster and the advantage of the differential backups are seldom ever used

There is no separate treatment of repositories/databases and "plain" data in respect to backup (other than that the database must not be used when being backed up).