I'm looking for a simple on-site (i.e. not online) backup solution for our small company. Right now we have approximately 4TB of data in total, maybe adding ~500GB per year. The amount of data changing per day is much less tough - I guess much less than 1GB on average.

All data is only accessed from the intranet and most machines are running Windows and some are running under MacOS if it matters.

The data in detail:

(a) Large parts of the data are images/videos/documentation(pdf) and alike, I would guess 2.5TB.

(b) Frequently accessed are our CAD-data files, but they only take up 10-20GB. These are controlled/accessed by a centralized CAD vcs called GAIN (I think it keeps it's data in a binary database). Currently this is dumped on the evening and then backed up.

(c) Some primarily source-code data are already under version control (SVN, GIT) taking up less than 2GB.

(d) Some programs only have binary source codes and are "archived" as zip-files. New versions are added and some old versions are restored sometimes but old versions never get changed. These programs take up roughly 80GB.

(e) Some personal backups (emails etc.) and other stuff takes roughly 1TB I guess.

(f) We also have a small amount of data on a single Microsoft SQL server. This should make up less than 1GB.

Right now we do a full backup every Monday to Friday evening from network disks to local server disk to tape drive on the server. We alternate the Friday tape, i.e. we have tapes labelled mo,tue,wed,thu,fri1,fri2. This implies we cannot go back in time more than 2 weeks in the worst case.

What's a good solution to this heterogeneous system consisting of

(a) large seldom accessed, seldom changed, seldom added data,

(b) frequent accessed rather small data delivered by a program internally using a database,

(c) frequent accessed rather small data under "common" version control,

(d) large binary files (~100MB) which are mostly added, seldom read, never changed (should optionally be disposable) and

(e) miscellaneous data like office files, data logs, mail folders which are seldom added/changed

(f) data on Microsoft SQL server

I'm firm with programming, version control and computers in general but new to backup strategies. So it would be good if the solution is quite simple to maintain.

If possible a versioning like that offered by SVN/Git would be nice, so the last successful backup allows restoration of every single file ever backed up (and not manually removed).

Problems with the strategy so far:

  • backing up takes a long time (15 hours)

    => There's not enough time to test the backup

    => Hard to tell if the backup is really working

    => What to do if the backup time reaches 24 hours?

  • restoring a backup is quite a pain
  • restoring something I deleted/modified/overwrote a month ago is not possible

A solution should address all these problems.

Time usage in detail:

  • Gathering of data from other servers over network to backup server: 02:15

  • Copy data on backup-server (which acts as a "regular" server as well) to another drive on the backup server: 09:00

  • Copy all data from internal drive on backup server to tape attached to backup server: 03:45

  • 1
    What's a good solution? What's the problem? What are your needs and requirements? I'd like to make a sandwich. I have some lunch meat and some bread and cheese. What's the best sandwich to make?
    – joeqwerty
    Commented Jan 8, 2013 at 14:58
  • @joeqwerty Thanks for your comment. I added the problems so far at the end of the question. A good solution solves all these problems ;-).
    – Onur
    Commented Jan 8, 2013 at 16:44
  • OK, now we've got something to work with. Answers forthcoming.
    – joeqwerty
    Commented Jan 8, 2013 at 17:10

3 Answers 3

  1. backing up takes a long time (17 hours) - Perform a Full backup on the weekend and perform Incremental backups during the week. This will reduce the backup window during the week and will also reduce the amount of storage needed for your backup sets.

  2. There's not enough time to test the backup - What are you testing exactly? You should perform a test restore of small data sets from your backup sets every week or every month. You don't need to test restore the entire backup set. Restore a handful of files and a database or two.

  3. Hard to tell if the backup is really working - See number 2. You need to test restore data from the backups to know if they're working. You should do this often enough that you're confident that the backups and backup process is reliable from week to week.

  4. What to do if the backup time reaches 24 hours? - See number 1.

  5. restoring a backup is quite a pain - How so? Is it the process? The backup software? Etc., etc.

  6. restoring something I deleted/modified/overwrote a month ago is not possible - Acquire enough backup media to meet your recovery needs. Determine how much backup media is needed per week and how many weeks you need to be able to go back and recover from. Then multiply the two. This will give you a rough idea of how much backup media you need and will help you determine your backup media rotation schedule.


To address your comment:

As far as restoring data is concerned, it depends on the backup software and what type of backup media you use. BackupExec uses an on tape and on disk catalog of backups. Finding data that needs to be restored doesn't require "reading" the tapes until you find the data. It only requires finding the data in the Restore selections window in BackupExec. Once you've found the media that the data resides on it's a simple matter of providing that media to BackupExec. To further this point, BackupExec recommends performing backups to disk and then duplicating (copying) those backups to tape. If you provide enough disk space to run a full week of backups then all of the data that you could possibly need to be restore during the entire week will be on disk and won't require you to swap tapes around at all. You'll simply select the data to be restored and BackupExec will find it in the on disk media.

As for the type of backup to perform, that's up to you. I recommend a weekly Full and a daily Incremental because the daily Incremental backups will run faster and will be smaller than a daily Differential backup, saving you time and money (in terms of backup window and backup media). The case where a Differential backup would be needed to restore data is far and few between and I've never actually encountered that scenario in 13 years in the IT profession.

  • ad 2. The backup software (Symantec Backup Exec 2012) allows for a read test of the tape. I was told (i.e. don't know for sure) that it rereads the whole tape and compares with the data on disk. ad 1. If I do incremental updates Monday to Thursday, if I need to recover data on Friday I may have to search all 4 tapes, right? Or is there an index on each tape so that looking at the latest tape tells me, where the file is? I suppose I also could make differential backups based on the last full backup to speed things up. Is this right?
    – Onur
    Commented Jan 9, 2013 at 7:45
  • The number of times we need to restore data is less than once a year on average. I suppose even it would take me 2 hours to recover a file it would be acceptable as long as it's secure and verifiable!
    – Onur
    Commented Jan 9, 2013 at 13:45

This smells fishy, sorry.

Right now we have approximately 4TB of data in total, maybe adding ~500GB per year

4000gb is not a large backup by any means and should not take 17 hours. How you do that - 1gbit network? Maybe time to put in a decent infrastructure. 10g backbone for the backup servr, something like MIcrosoft DPM with local change agents and the functionality to allow single file restores for users, 10-12tb of disc space in the backup server to keep backups some time on disc (for fast restore by users).

This is all well known and documented stuff - it looks to me like this is mostly your definition of how to make backups that is bad. from not enough hardware to not enough software. You should reevaluate your setup.

  • We don't need to recover files often but the last time something needed to be recovered it was not possible because it has not been backed up ;-). Before that incident our backup was managed by the IT of our mother company. That's why it's up to me now... So I inherited this system and am not quite happy with the current solution. The data is copied over the company network (1gbit is suppose). They also have some TB they copy each evening over the same network (we share the same building). That's what is slowing things down I suppose.
    – Onur
    Commented Jan 9, 2013 at 15:55
  • Starts the problem. I got mysel some nice 1gbit switch recently - 48 ports, and 4 x 10g. Now i move the core elements over to 10g, cheap and the file servers have dons more capacity. Also single item restore is a big item in DPM - and I dont really want to user other software on top (been problematic in the past, been burned). Seems your network lacks a proper backbone - 1g is not a lot these days. It is PERFECT for a ROOM of workstations etc., but as a main backbone - no way. Not anymore.
    – TomTom
    Commented Jan 9, 2013 at 15:58
  • I checked the logs and gathering of data from the several servers to the backup server take up only 2:15. But we need 9 hours to copy all gathered data to another internal drive which feeds the tape! The writing to tape accounts for 3:45, giving 15h in total (not 17h as I wrote first). So it doesn't look like it's the network that's slow but the copy of an internal drive to another internal drive... I'll further investigate this. I also added time usage to the question.
    – Onur
    Commented Jan 10, 2013 at 7:50
  • Ok, but let's get real - your actual time for backing up is 2:15 - then it is on the backup Server. DPM would consider the dat backed up at this Point. The copy is WAY too Long. Given you take 2:15 to copy the data over a 1gbit Network, disc copy should be faster. Copy to take can happen a day later - that is how DPM does it, Keep the current backup on discs, then move out older data to tape as needed.
    – TomTom
    Commented Jan 10, 2013 at 8:43
  • It's not just a plain copy. It also kind of compresses the data (there are a lot of 50GB "bkf"(Symantec backup to disk files) around). Maybe it even checks the data which should nearly double the time. I'll have to investigate this further...
    – Onur
    Commented Jan 10, 2013 at 9:31

I'm trying to summarize what advice has been given:

  • get an dedicated backup server so that production servers are not free to change once all data is on the backup server
  • if the time for backup is too large switch from full backup every day to full backup on Friday and incremental/differential backups from Monday to Thursday
  • Backup tests will suffice when done on Friday backup - or when we have a dedicated backup server maybe everyday (as long as it's automated and therefore doesn't take my time)
  • Get enough tapes so we can restore data from long ago
  • To speed up recovery one might even consider making a backup to disk in addition to backups on tape
  • Incremental backups during week are to be preferred since they are faster and the advantage of the differential backups are seldom ever used

There is no separate treatment of repositories/databases and "plain" data in respect to backup (other than that the database must not be used when being backed up).

  • @TomTom: Thanks for your advice which I have summed up in this answer.
    – Onur
    Commented Jan 11, 2013 at 8:51
  • @joeqwerty: Thanks for your advice which I have summed up in this answer.
    – Onur
    Commented Jan 11, 2013 at 8:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .