3

My company recently wasted a lot of time restoring data from corrupted USB disk backups. Everyone here is now keen to prevent the same happening again - but before buying the $5K tape drive now recommended I wanted to investigate a bit further.

The data recovery company that helped restore our data told our engineer that backing up to USB disk is a stupid idea. I wanted to find out why this might be, but have so far failed to find any likely reason. The only issues that I can think of that might affect these backups are

  • Physical damage to the disks. Although disks are less robust than tapes, it seems unlikely as the disks themselves are readable and were not badly treated.
  • Failing to eject the device before unplugging (also seems unlikely, as the disks would in any case typically be unplugged long after the backup finished)

Does anybody know of any fundamental problems either with backing up via USB, and/or to disk? Is NAS any better? My suspicion is that it was the fact that backups were never checked (no test restores ever done) that was the problem, not the fact that the backups were on disk or performed via USB.

For background... :

The fault we experienced: One day the server crashed due to a motherboard fault, resulting in corrupted data on a RAID disk pair. When it came to restoring the server, we found that the USB backup disks (which had apparently verified ok) were mostly unreadable. On the one BKF backup that I checked it seems that it became unreadable was the MS Exchange database folder. We managed to restore data using the rather clunky but usable 'systools BKF repair'.

Our precautions against data loss consisted of

  • RAID disk pairs, with windows sever set to record 'previous versions' of files
  • Weekly backup/verify of complete SBS 2003 server (~200GB) to 500GB USB disks
  • Daily incremental internet backup of user data (~60GB)

5 Answers 5

6

What kind of RAID? Motherboard-based? (couldn't tell from the description)? That is a problem if it is.

As for using USB disks, I don't think it's a bad idea necessarily, as long as you are verifying the backups and the speed/capacity are adequate for your purposes.

Our current tape backup solution has a dedicated server (using Microsoft DPM) that saves backups to a local hard drive, then rotates out the backups after a period of time to tape for longer term storage.

USB can be slow, but again, if it's meeting your needs then there's nothing fundamentally wrong with it. The thing to do is sit down and analyze your options and disaster recovery plans. How far back are you able to recover data? Are you able to recover from bare-metal failure? How long of a downtime would it be? Are your backups taking too much time to perform or hindering server performance while users are trying to get work done?

Also, how available is the data? Tapes are high-capacity and easier to store for a period of time, but if you need data on a tape that your drive no longer reads or can't get new tapes for your device because it's too old, what then? Hard drive backup tends not to have this issue, as there are tapes that you can have trouble reading from ten years ago but USB will be around and usable for quite some time in all likelihood.

And is your solution proprietary? I.e., the old unix tarballs from way back when dragons were real and pirates didn't have law degrees can still be decompressed and read. If your backup is in a format specific to a product, do you think you can recover data if the program is no longer supported or has other issues?

What it boils down to is analyzing your situation and coming up with recovery scenarios and making sure you're covered in the most likely events. USB drives are fine, as long as they meet your needs. But they also can be prone to any other hardware failure, so you should probably make sure to cycle them out periodically for redundancy.

6
  • Thanks for the response. It's a smartarray 6400 and we use raid 5.
    – ttt
    Oct 14, 2010 at 15:34
  • 1
    I'm wary of RAID 5 anymore. You might someday want to consider RAID 10. As drives up in capacity, tolerances for Unrecoverable Read Errors creep up. We lost a volume because a drive died, replaced it, and the controller hit a previously unrecorded URE spot on another drive and couldn't fix it. Had to do a full backup recovery. Oct 14, 2010 at 15:44
  • Get a better controller. Mine checks all discs every week to make sure there are no unrecorded URE. ;) And then there is Raid 6. Raid 10 is fastest, but again.... Raid 6 is cheaper.
    – TomTom
    Oct 28, 2010 at 6:44
  • Agreed. With the drive sizes and data patterns of today, RAID 5 is a liability. Go Raid 6 or 10 or higher
    – Posipiet
    Oct 28, 2010 at 6:45
  • @TomTom: I have read of other people having the same issue I did, and assume that PERC controllers are in rather wide use. The drive simply didn't reveal the error until it was in the middle of a rebuild...even the "repair routine" would pick up an error and claimed to have fixed it, re-run the rebuild, and halt at the same spot again. Oct 28, 2010 at 9:31
5

Case shows: Any backup that isnt tested, doesnt work. Corollary: Tested Backup on USB disks is better than untested backup on tape.

Testing means: at least once a month, have an audit, consisting of a list like the following

  • Check the logs. If your Backup doesnt have logs, it is not a backup.
  • Restore some random files.
  • Make a file for the sole purpose of seeing if it lands in the backup. Check if it is handled correctly. Put it at a different location each time.
  • Several more steps are possible.

A backup operator not doing that, or a management disaproving that are negligent.

Backups are a hard thing to do. Most people dont get it right on the first try. So revisiting it regularly gives you a chance to find the mistakes made.

Also regularly checking the backup gives you proficiency with the tools. You will need the proficiency when the day comes.

2
  • +1 - Testing is crucial because it's the only way to know that you can do a restore. Just to add to that, the parts that can be automated should be automated. Don't manually check the logs, because after a surprisingly short time that will get skipped (boring!). Create a system that will send you an email advising of the outcome, with any warnings or errors clearly shown. Oct 29, 2010 at 21:19
  • agreed, but still check the logs once a month, because the emailer demon may just be wrong and not see certain entries
    – Posipiet
    Oct 30, 2010 at 7:22
1

Fundamentally, USB-based disks should work OK but the USB data channel is very slow so performance is often cited as a reason for not using them. Personally, I won't use them as backing up to removeable disks is a manual process and imples that the disks have to be disconnected and taken somewhere else by somebody, with risks of damage during movement, and physical loss or theft.

NAS would generally be quicker, but where is the NAS box? if the backup box is in the same building or room you are clobbered in the event of a fire, flood or building collapse.

1
  • Try USB 3.0. I know a guy who got a USB 3.0 adapter for his computer just to be able to use a USB 3.0 file transfer channel (i.e. USB disc) and he tells me it FLIES.
    – TomTom
    Oct 28, 2010 at 6:45
1

I'm inclined to agree with the advice you say you already have. Backups to USB disk are crazy in all but the smallest business - USB disks can do fail and can still be sensitive to being unplugged at the wrong moment.

Backups to disk aren't inherently unreliable as such, but disks tend to have a finite life (as does tape), they can be sensitive to being constantly disconnected and reconnected, and in my experience people doing backup to disk don't have enough disks to generate a good backup cycle. That would make me worry. I think tapes are more robust, but if USB disks fit your budget and requirements then go for it.

Do you have a problem with tape besides the cost? You've been pointed in that direction and it is still, in my opinion, where backups should ultimately be stored. I think NAS makes a lot of sense as an intermediate step, e.g. the backups go from servers --> backup NAS --> tape. I appreciate $5k might sound like a lot of money, but I'm betting that the disruption you're facing from events now is going to cost more than that.

I think Bart makes a good point about analysing your needs. I'd start there myself and think about the cost of a reliable backup solution (that includes the costs of someone's time to do the occasional test restore to check both the data itself AND your DR process). You just need to be clear on the costs and risks of having a "better" solution vs. having a "cheaper" solution and how much risk and delay the business is prepared to take.

0

You could go for a pair of eSATA drives, to have both speed and simplicity. For simple backups, just alternate two (or more) external eSATA drives, having the non-present one outside the building for catastrophes.

If you like to store generations of data, a tape drive might be better in the long run.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .