Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Recently, one of the main file servers at our company failed. It was using a 4 disk RAID array, but apparently 3 of the disks died, and all the data on the server has been lost.

Speaking to the sys admin, he says that he has been warning the upper management about the backup situation for months. He had been trying to get approval to buy an enterprise-level backup solution, but he never got the budget approved for it - because management thought it was over the top.

The sys admin is a dedicated properly certified sys admin, whereas his managers are not IT-oriented.

His manager is asking why he didn't buy a cheap external drive and use this to backup the file server. The sys admin thinks that this is just a mickey-mouse solution that's suitable for use at home, but not a professional IT company - which is why he did not do it.

It seems to me that the sys admin wants proper, text-book IT strategy that costs a lot more money, whereas the management (without a deep IT understanding) wants cheaper solutions that they think are adequate.

I'm wondering what is the opinion of other sys admins? Was this sys admin correct in his actions? Or should he always make sure there is a backup of the important data, even if he believes that the cheaper way is not good enough?


Edit: based upon the answers, I'll add that the sys admin has an IT manager who would've known of the situation. He reports to the ultimate boss. I don't know if the manager ever reported the full situation to the boss. I think it is quite tough for the manager, as he is stuck in the middle, and he wants to be diplomatic with both sides.

share|improve this question
1  
Sounds like the admin cut his nose off to spite his face. –  Wesley Feb 5 '10 at 0:47
    
Wow. Middle management FTL. The great thing about management is that really, it's his fault. The buck stops there. He's the guy who's supposed to be keeping track of the big picture. He's the guy who needs to be setting out budget allocations, and demanding emergency capital for things like backups. Must be a small company if they're even dealing with the admin directly...Where I work, I'd no more be talking to management (other than my boss) about backup problems than I'd be flying to the moon. –  Satanicpuppy Feb 5 '10 at 6:56
1  
I'm a hopelessly undiplomatic manager. I'm fortunate that I work in a company where people appreciate it when you speak frankly. And for an added bonus, something that anyone who works in management should read: joelonsoftware.com/articles/TwoStories.html (it applies to I feel any kind of manager, maybe the manager in question should read it) –  Mark Henderson Feb 5 '10 at 7:33
    
Great article, Farseeker. –  Wesley Feb 5 '10 at 22:09

13 Answers 13

up vote 11 down vote accepted

I would agree that doing it right is the preferred method. But, to stand by and do nothing is unprofessional. Was management informed that there was no backup in place? It is the admins job to present the options, including costs and risks, to management. He presented his preferred option, and when it was denied, he did nothing. Not cool.

share|improve this answer
5  
If this guy didn't put out at least 3 options for "enterprise backup" I'd call that unprofessional. At the same time if he presented "Lamborghini, Toyota & Yugo" solutions and they all got shot down that's a management screw-up. –  voretaq7 Feb 4 '10 at 22:58
3  
Agreed, as long as they were properly, formally, informed that there was no backup implemented. –  Craig Feb 4 '10 at 23:15
2  
In this sort of case asking your boss to sign something to the effect of "I understand I'm screwing us" is sometimes effective (plus as a consultant I found they usually read it when it's something they had to sign). –  voretaq7 Feb 4 '10 at 23:22
1  
I think "not cool" might be just slightly understating it. –  John Gardeniers Feb 5 '10 at 1:17

i would honestly say it is a failure on both parts.

The logistics of the situation might mean that he would have to take away time that he should have been spending doing other, immediate, important tasks.

However, ultimately, yes, he should have done something. A bunch of hard drives from here and there would have been better than nothing as has been said repeatedly.

On the other hand, the entire purpose of management is to make sure that the people beneath you can do their jobs, and do. and thus from a leadership point of view, the managers failed miserably and can be held equally responsible, if not moreso.

share|improve this answer
2  
+1 for the failure on both parts; I still can't accept that the admin thought it better to hold out and do nothing though. –  Darth Satan Feb 5 '10 at 0:22
    
+1 I agree it was a failure on both sides. –  iPaulo Feb 5 '10 at 1:49
1  
+1 for the double failure, and +1 for mh's comment: Unless the company was totally unreasonable ("you want to do backups? Then YOU buy everything needed to do them!") the admin really screwed the pooch. –  voretaq7 Feb 5 '10 at 4:08

If there are no backups, as far as I'm concerned it's the sysadmin's responsibility to:

1) Explicitly tell the higher ups that there is NO backups, in no uncertain terms, so that they are aware of it

2) Back up the data anyway, any way he can

Frankly I would expect to be fired if this happened, because even if management are making my life hard, that's not an excuse, especially if they're still under the impression that they have something rather than nothing.

share|improve this answer
    
I don't necessarally agree with point 2, but point 1 is the heart of what an admin should be doing. Something like "WE HAVE NO BACKUPS" should be repeated to every excutive you can find at least monthly until the problem is fixed... –  voretaq7 Feb 4 '10 at 22:54
    
And then this happens and when heads roll it's yours on the block... –  Mark Henderson Feb 4 '10 at 23:02
2  
To be clear it's not the "back it up anyway" part I disagree with, it's the "any way he can": Should I come in every night after hours & manually cut backups to my own personal external hard drive? (Some companies won't even authorize you to buy a 500GB USB drive, and IMHO there's no helping those companies...) –  voretaq7 Feb 4 '10 at 23:16

It's damned if you do, damned if you don't. Frankly, if there was no money spent by management on a backup solution, then it's their fault. On the other hand, the admin should have been active in trying to work out a stopgap solution, rather than just sitting on his ass waiting for something to break (I don't think any sort of external drive solution is acceptable. You're never going to get a decent backup with that.) You can't just say, "Well I don't have what I want, so I'm not responsible" but you can say, "I've repeatedly tried to get you to do something and you've given me nothing and this is not my problem."

I was actually in a situation once--I wasn't even an ADMIN at this job--where I was working on a database, and made a backup before I changed it (which is s.o.p), and I (as I usually do, whenever I can) saved it to my own local machine. Two days later they lost the raid array, and ooops, turned out there was no backup solution. They'd been backing up the database to the raid array.

So I come in late on this, and I say, "Oh, I backed it up myself day before yesterday."

You know what the outcome was? I was censured for my bad backup solution. For a machine that I was in no way responsible for. And it wasn't because the backup I had was too old, it's because I'd only backed up the database I was working on, not every database.

So the problem is this: if you do a mickey mouse solution, if you do anything, and it's not quite good enough, you're going to get just as much hell as if you do nothing at all. If backups are your responsibilty, explicitly, and there is no budget, you should try to cobble something together, but you better make damn sure it works, and you need to raise hell about it. Repeatedly. At every opportunity.

If it's not your responsibility, point out that there exists a problem, and absolutely, categorically, refuse to take responsibility for an unfunded mandate when they try to assign it to you. No one makes disaster recovery a priority until there is a disaster, and then they scapegoat everyone to try and make up for their own shortsightedness.

share|improve this answer

Unfortunately, companies skimping on backups is all too common. Most never change until they get burned and lose everything.

BUT

If you are employed to be the sysadmin you have to work with the tools that you have including your brain. No matter what management or anyone else says on good days, when the poop hits the fan everyone gets selective memory.

A mickey mouse backup is better than no backup at all.

share|improve this answer
    
Bravo and encore! =) –  Wesley Feb 4 '10 at 22:58
1  
Sometimes a mickey mouse backup is worse: Everyone assumes their data is safe, but when the poop hits the fan the files they want are nowhere to be found -- "Oh, you saved that on drive X? Our backups only included drive Y!" & because "there are backups" they didn't keep any copies of their own... –  voretaq7 Feb 4 '10 at 23:04
    
Well now that can be true enough!! I should have said "TESTED" mickey mouse backups. Anyway, ALL backups should be tested. I had someone tell me once "Any Idiot can make a backup, but restores take brains.." or something like that. –  BhmJeep Feb 4 '10 at 23:41
2  
The way I've always heard it is: "Backups aren't important, Restores are IMPORTANT!" –  BillN Feb 5 '10 at 0:17
    
@BillN - Absolutely! The storage guy at the last place I worked used to drill it into people: "If you haven't had a successful restore test, you have NO BACKUP" –  voretaq7 Feb 5 '10 at 4:03

To me it sounds like the sysadmin wanted all or nothing. It's nice to get all, but if you can't have it should you accept nothing?

In my experience, the thing to do is evaluate all the possible options, (not in too much depth), and draw up a few bullet points for each indicating the pros and cons, costs (both inital and ongoing). Include in this the "do nothing" option.

Then you allow the managers to decide what solution they choose. It would seem to me that there was probably more than one possible option for your sysadmin. Perhaps he only saw the one he really wanted though?

share|improve this answer
1  
Indeed. I am the king of the crappy hand-built 10 terabyte "backup server". Salt a few of those around, and dump data to them on a regular basis. Not as good as a nice tape rotation, but one of those costs less than a box of decent sized tapes. –  Satanicpuppy Feb 5 '10 at 6:44
1  
Giant multi-TB "backup" RAIDs, treating USB hard drives as "tapes", backing up over the network to rsync.net, using tarsnap.com -- There's always something as long as the budget isn't $0 (and sometimes you can even make do with $0 :-) –  voretaq7 Feb 5 '10 at 15:44

I'll add my voice to those saying that the admin should have implemented something here. He's badly at fault for not having done so. There's a part of me that would like to sympathise with his position, but in an ideal world backup and restore would take no time, always work, and never be needed. This isn't that world and even the best backup solution is going to have flaws that you'll need to accept and learn to work with.

Half-assed is better than no-assed, and even using an el-cheapo USB HD would have gotten him out of the woods, and would have given weight to his position when management are told that they can't get data more than day or two old back. But it would have still saved his neck in this case.

share|improve this answer
    
And in an extreme case perhaps even save the company and everyone's jobs. More than one company has gone under as a result of data loss. –  John Gardeniers Feb 5 '10 at 1:28

Should a sys admin backup important data in anyway he can.

I don't know that I would say a you should make a backup under any conditions. There are some things you might be tempted to do that would possibly be illegal. For example I would not backup health records over the network to my personal computer. I would not do something illegal just to have a backup.

OTOH to have at least some backup system in place I would accept a lot of compromises. Then whenever a compromise was made I would make a point to make sure my objections are clear and documented about why it was a bad compromise that will cause problems, be inadequate, or become less useful in the future.

share|improve this answer

As a sysadmin I believe it is my responsibility to ensure the systems under my care are as secure and reliable as I can possibly make them. Backups fall under the reliability tags. Frustrating as it may be to have to argue with non-understanding senior staff (I think we've all been there at some time or other), we still should be doing our jobs as best we can.

When the backup system I inherited in my current position failed and management hesitated about spending the money on the system I wanted I didn't leave the system without backups. Instead, I brought my personal external drive in and used that for a week or so. Despite having an absolute abhorrence for using hard drives for backups the fact remains that it was vastly preferable to having none at all.

share|improve this answer
    
Wow. Props for bringing your own stuff in. I don't know if I would have gone that far. Sometimes I see crossing the line between personal and business as The Last Great Evil. However, maybe it's the Second To Last Great Evil, next to Standing In The Unemployment Line Updating Your Resume Without Any References From Your Last Employer. –  Wesley Feb 5 '10 at 0:51
    
When you're a one man IT "department" you tend to be somewhat more flexible than you might be working for a larger organisation. As for that line between business and personal, I find it's more like a really wide grey smudge, rather than a clear dividing line. –  John Gardeniers Feb 5 '10 at 1:21
1  
I'm a one man IT shop for a non profit that just spent more money in 2009 than they took in... and I /still/ wouldn't bring my own stuff in. Maybe I'm a jerk? :-/ –  Wesley Feb 5 '10 at 2:28
    
I'll admit I use some of my own equipment (e.g. I have copies of our backup encryption keys in an (encrypted) file on the thumb drive on my keys). For me the line is "If this is absolutely necessary for the business to function, the company needs to provide it" –  voretaq7 Feb 5 '10 at 4:06
1  
Believe me, if the external drive had been necessary for more than a few days I would not have continued to use my own. That was purely a stop-gap measure. Letting management know I was using it probably helped to get the approval to buy what was required. –  John Gardeniers Feb 5 '10 at 4:56

If the sysadmin was unable to convince management of the importance of a good backup solution the only way they will ever be convinced is via catastrophic data loss, but as a sysadmin it is your responsibility to educate management and users about the importance of things like backup, and to make sure they thoroughly understand the current state (in this case "no backups") and potential consequences ("We lose a disk and your precious data is gone forever").

My personal opinion is that the admin kinda screwed up here: Ad-Hoc backups are a bad idea (you'll miss stuff, important data will be lost, if you're not around backups don't happen), but at the same time they should have been able to find a reliable "enterprise" backup solution within the company's budget.
Software like Bacula and Amanda is available for free, and both of those can work with removable USB media and CDs securely and reliably. Including the cost of media and server hardware you could have a good system for less than $2000 US - even cheaper if you recycle hardware for the server.

Now if management is also opposed to the admin spending TIME on getting backups working there's just no helping this company: As I said above sometimes the only way to teach people is catastrophic data loss, and if that's the case it sucks for the poor admin who has to take the blame for institutional stupidity.

share|improve this answer
    
I think the sys admin was thinking along these lines beforehand, that it would take a catastrophic loss to teach them the need for proper backup strategy. Now that it has happened, I know he's been pretty stressed out. –  Rick Feb 4 '10 at 23:00
    
Problem is as an admin you can't rely on pain to teach: Yes it will probably get him a backup system once the fire is out, but everyone will remember that they lost data on his watch, and it hurts the company (at least in the short-term). –  voretaq7 Feb 4 '10 at 23:08

My personal opinion is that it's my job as a sysadmin to inform and impress upon management the need for and the importance of having an adequate, appropriate backup solution and requesting the neccessary budget for such, and to explain the risks associated with not doing so. It's not my responsibility to go "outside" of the mandate of the management and just do whatever I think is right regardless of how poor those management decisions are. It's not my responsibility to cobble together some half-baked, half-assed solution.

If I was an insurance agent and I told you it was important to have fire coverage in your home owner's policy, and if I adequately explained the risk of not having fire coverage, and you declined said fire coverage, and your house burned down, who's responsibility is it? Should I have given you fire coverage anyway?

My opinion is that the sysadmin exercised due dilligence in performing the duties of his job by bringing the matter to the attention of management, explaining the importance of having an appropriate backup solution, explaining the risks of not having it, and requesting the neccessary budget for such. If he was rebuffed in his efforts then the responsibility lies squarely on the shoulders of the management.

People make poor decisions all the time and bad things happen because of those poor decisions, that's a fact of life. I can't be responsible for every bad decision my boss makes, regardless of the risks associated with those decisions.

share|improve this answer

Did the same situation happen with the RAID array? As soon as one disk dies, you are in a situation where one more means data loss.. you better replace that drive immediately.

If I was in the sys admin's shoes, the instant the first drive went:

  1. Email manager with formal request to replace the drive, reminding that no backup system has been approved so this is a critical situation. Cite the prior requests for the backup system by attaching that email, or even better, the manager's response denying the request.
  2. If no response, re-send message, this time CC'ing your manager's manager.
  3. If still no response, well.. not much more you can do. Polish the resume and start looking for a better job.

If you get denied along the way, at least you have it in writing for when the shit hits the fan (Get it in writing/email, do not accept a verbal response. You need a paper trail here. If your manager refuses to write it, then go over his/her head, because that's just shady -- there is no legit reason not to write it down.)

The same process should have been followed for getting a backup system, though perhaps without escalation as quickly (or going over your manager's head at all). If none of the requests are in writing, well.. shit rolls downhill. At least it's a good life lesson.

If you don't lose your job over the situation, well, start making that request again, citing the disaster it caused last time your request was denied. If it's still denied, then you need to decide if that's an environment you want to work in, and it's worth the stress. If every morning you expect to walk into work finding a panic because data was lost, well, that's no way to live.

share|improve this answer
2  
I once had a 12 disk array, and I got a warning that one drive was going south. So I put in a new drive, and told the array to rebuild it. During the rebuild, another drive started failing, and then another. By the end I'd had 10 of 12 drives start to fail, and I'd replaced all 12 out of paranoia. It is NOT uncommon to lose multiple drives in a raid. It is NOT impossible to lose the raid itself, without any drives failing. RAID is not a backup solution. –  Satanicpuppy Feb 5 '10 at 6:41
    
Very often the act of kicking off a RAID rebuild (especially on old arrays) will cause enough soft errors on one or more of the remaining drives for the RAID controller to declare the drive "failed", and BAM! There goes your data. –  voretaq7 Feb 5 '10 at 6:44

The company is clearly looking for a scape goat in this, the sys admin is quite right not to backup critical data to a removable device.

1) They are not reliable 2) They are not secure

Ultimately it lies with the managers for not ensuring a proper DR (Disaster recovery) solution was put in place.

Look at it this way, how much has this data loss cost the company? Suddenly I'm sure the "over the top" solution doesn't look so expensive.

edit: yes I concede the fact any backup is more reliable than none, but my original point remains if this person has managers, the managers should of ensured the backup was in place, I'm not pardoning the sys admin of all blame here, but this this what the manager should be checking.

And what if the server failed and the data on the removable drives was irrecoverable for whatever reason? having had this occur myself in the past USB drives are far from reliable, but to some they can be used in a "pinch" the problem is as it appears in this case the management would of allowed removable drive backup to be used in the long run.

share|improve this answer
6  
An absence of a backup is even more unreliable than an unreliable backup. See answers by Farseeker and Craig for more info. –  Wesley Feb 4 '10 at 22:55
    
+1 to Wesley -- unreliable/ad-hoc backups will bite you (and Murphy says you'll lose the most important 1% of data when they do) but at least you still have the other 99%... –  voretaq7 Feb 4 '10 at 23:11
    
Not sure I understand your second point. Almost no backup media is secure on its own. If you enable encryption in the backup software, the backup will be secure. A simple truecrypt volume on a portable drive would give you adequate security. Your first point is equally weak. No backup media is 100% reliable, they all can fail. A backup on a portable hard drive can't be any worse the no backup at all if the proper security is put in place. –  Zoredache Feb 4 '10 at 23:58
    
@Oneiroi - this is wrong on so many levels. They may not be as reliable and secure as a proper backup is what I hope you mean, but even so, when the alternative is no backup at all I don't think taking a high and mighty stance for the sake of making a point is going to cut it in the Real World. –  Darth Satan Feb 5 '10 at 0:14

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.