Disaster recovery and preparedness is an unfortunate aspect of systems administration. This tag should be used for help with planning, implementation and best-practices related to recovering from a catastrophic event on a server or in a datacenter environment.

learn more… | top users | synonyms

114
votes
13answers
6k views

Engineers are using explosives to remove hard rock outside our office building. What countermeasures should we take?

Our building is located approx. 100 meters from the explosive charges. They happen several times per day, and really shake the entire building a lot. This is going to go on for many days and the ...
40
votes
20answers
5k views

What's your checklist for when everything blows up?

Users can't get to their e-mail, the CEO can't get to the company's home page, and your pager just went off with a "911" code. What do you do when everything blows up?
33
votes
7answers
1k views

What things do you look for when picking a server hosting company?

We are going through an RFP process of changing hosting companies for most of our servers (~10 fairly powerful workhorses and database servers). When the existing company was picked I wasn't at the ...
30
votes
4answers
2k views

My server room has flooded

We recently went through a hurricane and our server room became flooded. Hooray for insurance. Anyway, I need to save as much data off one of the hard drives as possible. Yes, it was submerged for ...
27
votes
11answers
2k views

Disaster recovery plan development best practicies or resources? [closed]

I have been tasked with leading a project regarding updating a old and somewhat onesided disaster recovery plan. For now we're just looking at getting the IT side of DR sorted out. The last time ...
25
votes
2answers
875 views

Retrieving an RSA key from a running instance of Apache?

I created an RSA keypair for an SSL certificate and stored the private key in /etc/ssl/private/server.key. Unfortunately this was the only copy of the private key that I had. Then I accidentally ...
23
votes
9answers
29k views

Unmount a nfs mount where the nfs server has disappeared

Server A used to be a NFS server. Server B was mounting an export of that. Everything was fine. Then A died. Just switched off. Gone. Vanished. However that folder is still mounted on B. I obviously ...
22
votes
5answers
4k views

BBWC: in theory a good idea but has one ever saved your data?

I'm familiar with what a BBWC (Battery-backed write cache) is intended to do - and previously used them in my servers even with good UPS. There are obvously failures it does not provide protection for....
18
votes
9answers
5k views

Architecture for highly available MySQL with automatic failover in physically diverse locations

I have been researching high availability (HA) solutions for MySQL between data centers. For servers located in the same physical environment, I have preferred dual master with heartbeat (floating ...
17
votes
9answers
1k views

Documentation As-A-Manual vs. Documentation As-A-Checklist

I've had discussions in the past with other people in my department about documentation, specifically, level-of-detail and requirements. In their view, documentation is a simple checklist of Y things ...
14
votes
4answers
758 views

IT lead does not have a backup, DR plan in writing [closed]

This is a general management question to IT managers out there. We are a small firm with about 4 servers in our colo cabinent. No full time IT manager. But we do have one person on monthly contract ...
13
votes
6answers
17k views

How to recover from a drive failure in a RAID 5 configuration?

This morning a drive failed on our database server. The drive array (3 disks) is setup in a RAID 5 configuration. While we wait for a drive replacement we are preparing for a recovery strategy. Users ...
13
votes
7answers
3k views

Setting up a new backup scheme

I'm in the process of designing my first ever backup scheme. I'm completely new to managing data backup, and there are some concepts that I don't totally understand. Here's what I've got so far, and ...
12
votes
4answers
4k views

How do I backup my TRAC installations?

We use separate TRAC instances as our ticket system for many projects and need to have them moved off site several times a day for disaster recovery. What is the best way to make this happen? Is ...
10
votes
12answers
592 views

What's the first thing you check when an untouched unix server starts going berserk?

So you have this neatly setup unix server and it's super fast and works swell and everything is great for months, and suddenly all kinds of weird errors start showing up for a variety of different ...
10
votes
3answers
11k views

Battery Backed Write Cache

I recently got some U server price quotes and some of them include BBWC: What exactly does it do? Is it just for RAID configurations? If there is a power malfunction, isn't the data loss ...
10
votes
5answers
494 views

High server availabilty for a small business

After having a bit of scare with a server that wouldn't come up one morning, the higher ups have decided that the business needs a high availability / fail over setup. We have 5 main servers (4x ...
8
votes
3answers
5k views

Active Directory disaster recovery with DPM

I have a sort of catch-22 question here. Suppose I'm using Microsoft System Center Data Protection Manager (2010 or 2012, it works the same way) to backup, amongst various other things, my Active ...
8
votes
1answer
2k views

Recover data from SCSI hard disk

We've got an old server with SCSI hard disk. The server crashed last week and it isn't exactly known what hardware component is damaged. Since the server is due to be retired anyway we don't want to ...
8
votes
3answers
2k views

Backing up VirtualBox VMs

Does anyone have a good complete strategy for backing up a bunch of virtual machines running under VirtualBox? I intend to run a handful of virtual machines on a single hardware platform and back ...
7
votes
2answers
833 views

How to recover data from an Exchange 2013 database after a complete Active Directory loss?

Scenario: a single Exchange 2013 server in a Windows Server 2003 AD domain; one DC malfunctioned months ago and was dismissed (without proper demotion, no less); the other DC died yesterday and there ...
7
votes
5answers
2k views

Local to Remote Webserver Failover

Short and sweet, I don't suppose you'll need more detail than this: We host our website on an in-house webserver. A catastrophe has and will happen again where communication from the web into/out ...
7
votes
1answer
922 views

Recovery strategy for Master-Master replication

I have implemented a HA solution for mysql based on master-master replication. There is a mechanism on the front end part which guarantees that only one db will be read/written to at a given time (i.e....
6
votes
5answers
6k views

What's the difference between a Disaster Recovery Plan and a Business Continuity Plan?

I used to think both terms referred to the exact same thing, but one of my clients just requested to have a look at both documents. The request emanates from the security department of a very big ...
6
votes
3answers
211 views

Disaster Recovery Planning, tower or rack?

I'm working on this project to develop a system with centralized information regarding emergencies delivered via open Wi-Fi on a small city. I'm from Chile, so we thought of this system to work ...
6
votes
5answers
380 views

Fault tolerant server structure for the smallest of businesses

I'm trying to figure out what to do for a small business that has been plagued by ridiculous hardware problems. Right now, this business runs on five or six desktop machines; no server infrastructure ...
6
votes
2answers
415 views

WHEN to put the contingency plan into action in case of a main server failure?

We have a production SQL Server database server shipping transactional log backups to two standby servers. The disaster recovery plan is already finished: we have a well documented procedure and ...
6
votes
2answers
974 views

Best practice for IIS 6.0 (Windows Server 2003) backups?

What is the best backup strategy for saving IIS 6.0 data: web metadata, files, logs etc. for disaster recovery?
6
votes
3answers
1k views

Reconstructing .bashrc from running session

I accidentally deleted my .bashrc. I still have the terminal running. What settings can I recover? I already have the aliases (from the alias command). I assume that all ifs and cases are gone, but I ...
6
votes
2answers
261 views

Green System Administrator looking for helpful tips [closed]

I have just been promoted to Systems Administrator for our product. We are designing a application that communicates with the cloud(Amazon EC2). I will be in charge of maintaining all Instances and ...
6
votes
2answers
950 views

Active Directory Disaster Recovery in a Small Business

This is hypothetical question, but one I’m sure that someone must have encountered and/or given some thought to before. Situation: Consider this, a small business is running an Active Directory ...
6
votes
2answers
2k views

MySQL replication issues after a power outage

After a power outage at our data centre, the slave MySQL databases are struggling. This is in the logs for one of the slaves: 100118 10:05:56 [Note] Slave I/O thread: connected to master 'repl@db1:...
5
votes
4answers
299 views

Virtualization for hardware resiliency?

Can anyone tell me if it is possible to pool several physical servers to run a resilient virtualization environment. Our servers are getting more and more critical to our clients and we want to do ...
5
votes
3answers
5k views

If DNS Failover is not recommended, what is?

As a followup question to his very popular question: Why is DNS failover not recommended?, I think it was agreed that DNS failover is not 100% reliable due to caching. However the highest voted ...
5
votes
4answers
464 views

Disaster Recovery/Sabotage Protection for a small business

I've been contacted by two partners in a small professional firm. They are concerned about their other partner and want to take some steps to be absolutely sure that the company's data and systems ...
5
votes
6answers
396 views

Can A Virtual Machine Be Converted to a Virtual Server e.g VMWare?

The Simple Question Can I convert an existing VM to a Virtual Server (e.g. VMWare)? I'm using Oracle's one and only awesome product, VirtualBox, and I'm trying to setup a SharePoint Farm to ...
5
votes
3answers
1k views

Is DFSR designed for use for Disaster Recovery?

We are currently working on implementing a DR strategy. Instead of SAN-SAN replication, it has been decided to have 2 live file servers replicating via DFSR. However, I don't know whether or not this ...
5
votes
3answers
665 views

How can you recover a SQL Server database if the ldf file has been deleted

We had a drive die and lost the ldf file, but the mdf file is in tact. Is there a process for re-connecting to the mdf file, considering the ldf lost? I have searched without much luck.
5
votes
2answers
659 views

Automated bare-metal recovery practices for small network

I have several machines which are on a small network with one DC and 3 to 5 workstations on the network at any given time. These are all setup with DNS and AD on the same server. I want the ability ...
5
votes
1answer
822 views

DRBD as DR: syncing datastores of 2 ESXI hosts, vmdk consistency?

does anyone have experience with using DRBD (protocol C) to sync parts of the datastores of 2 esxi hosts for disaster recovery of selected guests? I have 2-3 guests that should be able to recover ...
5
votes
1answer
498 views

Mysql disaster recovery

We just got a major disaster: somebody made an uncontrolled update on the production database, and obviously, the backup process is not working since a long time, so we got a major data loss. A 40 ...
5
votes
4answers
320 views

Hard drive data rescue services

Inspired by the question Crashed Hard Drive Data Retrieval which is currently concentrating on software tools and self-help methods of data rescue, I am curious what the community's experiences are ...
5
votes
2answers
4k views

Hadoop HDFS Backup & DR Strategy

We are preparing to implement our first Hadoop cluster. As such we are starting out small with a four node setup. (1 master node, and 3 worker nodes) Each node will have 6TB of storage. (6 x 1TB disks)...
5
votes
1answer
2k views

How to get cheap disaster recovery for a 124 TB Isilon filesystem?

On our Isilon cluster, we have a 124 TB file system. It is currently 38 percent full, with 31 million files. About half the data are image files, and the mean file size is 1.5 MB. We use snapshots ...
4
votes
10answers
378 views

How do you recover when your hosting provider loses everything?

You've probably seen the messages at the stackoverflow blog and on codinghorror: blog.stackoverflow.com experienced 100% data loss at our hosting provider, CrystalTech. We're working to ...
4
votes
5answers
24k views

Edit Hard Disk Serial Number with VMware

I'm virtualizing a Rockwell AssetCentre Server and I'm looking at Disaster Recovery scenarios. This server contains a lot of other Rockwell Software like RSLinx, Logix 5000, Logix 500, and more... ...
4
votes
3answers
10k views

How to actually use mysql slave as soon the master is failover or got burnt

I have MySQL master-slave replication that works fine; I googled the whole net and MySQL site to find the standard procedure to make use of the replication but found nothing. It is as if admins are ...
4
votes
6answers
4k views

Need help with estimating required bandwidth for SAN array to SAN array replication over WAN

I have a long-term goal of setting up a DR site in a colo somewhere and part of that plan includes replicating some volumes of my EqualLogic SAN. I'm having a bit of a difficult time doing this ...
4
votes
2answers
147 views

Business Continuity and Disaster Recovery Plans

My company is looking for outside assistance in (re)putting together our BC and DR plans. How do you judge companies from one another in this arena?
4
votes
4answers
2k views

How do you properly do Disaster Recovery for a file server?

We are currently working on implementing a DR strategy for a windows file server. We have ruled out Storage Replication because it is a preview feature, and Failover Clustering is designed for high-...