Disaster recovery and preparedness is an unfortunate aspect of systems administration. This tag should be used for help with planning, implementation and best-practices related to recovering from a catastrophic event on a server or in a datacenter environment.

learn more… | top users | synonyms

113
votes
13answers
6k views

Engineers are using explosives to remove hard rock outside our office building. What countermeasures should we take?

Our building is located approx. 100 meters from the explosive charges. They happen several times per day, and really shake the entire building a lot. This is going to go on for many days and the ...
32
votes
7answers
1k views

What things do you look for when picking a server hosting company?

We are going through an RFP process of changing hosting companies for most of our servers (~10 fairly powerful workhorses and database servers). When the existing company was picked I wasn't at the ...
29
votes
4answers
1k views

My server room has flooded

We recently went through a hurricane and our server room became flooded. Hooray for insurance. Anyway, I need to save as much data off one of the hard drives as possible. Yes, it was submerged for ...
27
votes
11answers
2k views

Disaster recovery plan development best practicies or resources? [closed]

I have been tasked with leading a project regarding updating a old and somewhat onesided disaster recovery plan. For now we're just looking at getting the IT side of DR sorted out. The last time ...
24
votes
2answers
704 views

Retrieving an RSA key from a running instance of Apache?

I created an RSA keypair for an SSL certificate and stored the private key in /etc/ssl/private/server.key. Unfortunately this was the only copy of the private key that I had. Then I accidentally ...
21
votes
5answers
3k views

BBWC: in theory a good idea but has one ever saved your data?

I'm familiar with what a BBWC (Battery-backed write cache) is intended to do - and previously used them in my servers even with good UPS. There are obvously failures it does not provide protection ...
18
votes
7answers
21k views

Unmount a nfs mount where the nfs server has disappeared

Server A used to be a NFS server. Server B was mounting an export of that. Everything was fine. Then A died. Just switched off. Gone. Vanished. However that folder is still mounted on B. I obviously ...
18
votes
9answers
5k views

Architecture for highly available MySQL with automatic failover in physically diverse locations

I have been researching high availability (HA) solutions for MySQL between data centers. For servers located in the same physical environment, I have preferred dual master with heartbeat (floating ...
17
votes
9answers
1k views

Documentation As-A-Manual vs. Documentation As-A-Checklist

I've had discussions in the past with other people in my department about documentation, specifically, level-of-detail and requirements. In their view, documentation is a simple checklist of Y things ...
13
votes
6answers
14k views

How to recover from a drive failure in a RAID 5 configuration?

This morning a drive failed on our database server. The drive array (3 disks) is setup in a RAID 5 configuration. While we wait for a drive replacement we are preparing for a recovery strategy. Users ...
13
votes
7answers
3k views

Setting up a new backup scheme

I'm in the process of designing my first ever backup scheme. I'm completely new to managing data backup, and there are some concepts that I don't totally understand. Here's what I've got so far, and ...
11
votes
4answers
4k views

How do I backup my TRAC installations?

We use separate TRAC instances as our ticket system for many projects and need to have them moved off site several times a day for disaster recovery. What is the best way to make this happen? Is ...
10
votes
12answers
581 views

What's the first thing you check when an untouched unix server starts going berserk?

So you have this neatly setup unix server and it's super fast and works swell and everything is great for months, and suddenly all kinds of weird errors start showing up for a variety of different ...
10
votes
3answers
9k views

Battery Backed Write Cache

I recently got some U server price quotes and some of them include BBWC: What exactly does it do? Is it just for RAID configurations? If there is a power malfunction, isn't the data loss ...
10
votes
5answers
477 views

High server availabilty for a small business

After having a bit of scare with a server that wouldn't come up one morning, the higher ups have decided that the business needs a high availability / fail over setup. We have 5 main servers (4x ...
8
votes
1answer
2k views

Recover data from SCSI hard disk

We've got an old server with SCSI hard disk. The server crashed last week and it isn't exactly known what hardware component is damaged. Since the server is due to be retired anyway we don't want to ...
7
votes
3answers
4k views

Active Directory disaster recovery with DPM

I have a sort of catch-22 question here. Suppose I'm using Microsoft System Center Data Protection Manager (2010 or 2012, it works the same way) to backup, amongst various other things, my Active ...
7
votes
2answers
189 views

How to recover data from an Exchange 2013 database after a complete Active Directory loss?

Scenario: a single Exchange 2013 server in a Windows Server 2003 AD domain; one DC malfunctioned months ago and was dismissed (without proper demotion, no less); the other DC died yesterday and there ...
7
votes
3answers
2k views

Backing up VirtualBox VMs

Does anyone have a good complete strategy for backing up a bunch of virtual machines running under VirtualBox? I intend to run a handful of virtual machines on a single hardware platform and back ...
7
votes
5answers
2k views

Local to Remote Webserver Failover

Short and sweet, I don't suppose you'll need more detail than this: We host our website on an in-house webserver. A catastrophe has and will happen again where communication from the web into/out ...
7
votes
1answer
735 views

Recovery strategy for Master-Master replication

I have implemented a HA solution for mysql based on master-master replication. There is a mechanism on the front end part which guarantees that only one db will be read/written to at a given time ...
6
votes
5answers
6k views

What's the difference between a Disaster Recovery Plan and a Business Continuity Plan?

I used to think both terms referred to the exact same thing, but one of my clients just requested to have a look at both documents. The request emanates from the security department of a very big ...
6
votes
3answers
204 views

Disaster Recovery Planning, tower or rack?

I'm working on this project to develop a system with centralized information regarding emergencies delivered via open Wi-Fi on a small city. I'm from Chile, so we thought of this system to work ...
6
votes
5answers
374 views

Fault tolerant server structure for the smallest of businesses

I'm trying to figure out what to do for a small business that has been plagued by ridiculous hardware problems. Right now, this business runs on five or six desktop machines; no server infrastructure ...
6
votes
2answers
391 views

WHEN to put the contingency plan into action in case of a main server failure?

We have a production SQL Server database server shipping transactional log backups to two standby servers. The disaster recovery plan is already finished: we have a well documented procedure and ...
6
votes
2answers
896 views

Best practice for IIS 6.0 (Windows Server 2003) backups?

What is the best backup strategy for saving IIS 6.0 data: web metadata, files, logs etc. for disaster recovery?
6
votes
3answers
845 views

Reconstructing .bashrc from running session

I accidentally deleted my .bashrc. I still have the terminal running. What settings can I recover? I already have the aliases (from the alias command). I assume that all ifs and cases are gone, but I ...
6
votes
2answers
260 views

Green System Administrator looking for helpful tips [closed]

I have just been promoted to Systems Administrator for our product. We are designing a application that communicates with the cloud(Amazon EC2). I will be in charge of maintaining all Instances and ...
6
votes
2answers
796 views

Active Directory Disaster Recovery in a Small Business

This is hypothetical question, but one I’m sure that someone must have encountered and/or given some thought to before. Situation: Consider this, a small business is running an Active Directory ...
6
votes
2answers
2k views

MySQL replication issues after a power outage

After a power outage at our data centre, the slave MySQL databases are struggling. This is in the logs for one of the slaves: 100118 10:05:56 [Note] Slave I/O thread: connected to master ...
5
votes
4answers
291 views

Virtualization for hardware resiliency?

Can anyone tell me if it is possible to pool several physical servers to run a resilient virtualization environment. Our servers are getting more and more critical to our clients and we want to do ...
5
votes
3answers
4k views

If DNS Failover is not recommended, what is?

As a followup question to his very popular question: Why is DNS failover not recommended?, I think it was agreed that DNS failover is not 100% reliable due to caching. However the highest voted ...
5
votes
4answers
453 views

Disaster Recovery/Sabotage Protection for a small business

I've been contacted by two partners in a small professional firm. They are concerned about their other partner and want to take some steps to be absolutely sure that the company's data and systems ...
5
votes
6answers
390 views

Can A Virtual Machine Be Converted to a Virtual Server e.g VMWare?

The Simple Question Can I convert an existing VM to a Virtual Server (e.g. VMWare)? I'm using Oracle's one and only awesome product, VirtualBox, and I'm trying to setup a SharePoint Farm to ...
5
votes
3answers
713 views

Is DFSR designed for use for Disaster Recovery?

We are currently working on implementing a DR strategy. Instead of SAN-SAN replication, it has been decided to have 2 live file servers replicating via DFSR. However, I don't know whether or not this ...
5
votes
3answers
500 views

How can you recover a SQL Server database if the ldf file has been deleted

We had a drive die and lost the ldf file, but the mdf file is in tact. Is there a process for re-connecting to the mdf file, considering the ldf lost? I have searched without much luck.
5
votes
2answers
650 views

Automated bare-metal recovery practices for small network

I have several machines which are on a small network with one DC and 3 to 5 workstations on the network at any given time. These are all setup with DNS and AD on the same server. I want the ability ...
5
votes
1answer
484 views

Mysql disaster recovery

We just got a major disaster: somebody made an uncontrolled update on the production database, and obviously, the backup process is not working since a long time, so we got a major data loss. A 40 ...
5
votes
1answer
1k views

How to get cheap disaster recovery for a 124 TB Isilon filesystem?

On our Isilon cluster, we have a 124 TB file system. It is currently 38 percent full, with 31 million files. About half the data are image files, and the mean file size is 1.5 MB. We use snapshots ...
4
votes
5answers
19k views

Edit Hard Disk Serial Number with VMware

I'm virtualizing a Rockwell AssetCentre Server and I'm looking at Disaster Recovery scenarios. This server contains a lot of other Rockwell Software like RSLinx, Logix 5000, Logix 500, and more... ...
4
votes
6answers
4k views

Need help with estimating required bandwidth for SAN array to SAN array replication over WAN

I have a long-term goal of setting up a DR site in a colo somewhere and part of that plan includes replicating some volumes of my EqualLogic SAN. I'm having a bit of a difficult time doing this ...
4
votes
2answers
143 views

Business Continuity and Disaster Recovery Plans

My company is looking for outside assistance in (re)putting together our BC and DR plans. How do you judge companies from one another in this arena?
4
votes
4answers
833 views

How do you properly do Disaster Recovery for a file server?

We are currently working on implementing a DR strategy for a windows file server. We have ruled out Storage Replication because it is a preview feature, and Failover Clustering is designed for ...
4
votes
3answers
1k views

lvm+ext3 vs ext3 recovery

I would like to hear from people who have personal experience recovering crashed systems or data from crashed drives using lvm+ext3 and ext3. Which option is harder to recover from backup or to ...
4
votes
3answers
205 views

Virtualisation for Disaster Recovery

Can anyone give me any ideas/links so that I can better get an idea of how virtualisation can help me from a disaster recovery point of view? We have a server sitting in a datacentre, it basically ...
4
votes
3answers
382 views

Is there any way to recover a 2003 Exchange database without the domain controller?

I have access to the Exchange Server but the domain controller RAID card died and the restored backup causes the new server to reboot without even a blue screen. I have rebuilt a new domain controller ...
4
votes
4answers
1k views

DPM Bare Metal Restore

We currently use Microsoft's Data Protection Manager (DPM) for backups in a relatively small environment, but I'm still worried about our main active directory server. I've been thinking that it'd be ...
4
votes
1answer
410 views

DRBD as DR: syncing datastores of 2 ESXI hosts, vmdk consistency?

does anyone have experience with using DRBD (protocol C) to sync parts of the datastores of 2 esxi hosts for disaster recovery of selected guests? I have 2-3 guests that should be able to recover ...
4
votes
3answers
896 views

Recovering with DDRescue Cannot Complete (write error: Read-only file system)

I'm trying to recover a corrupt VDI using vdfuse to mount the VDI and using dd_rescue to rescue the borked partition. dd_rescue seems to be working fine but once it reached about half of the ...
4
votes
1answer
71 views

Blackberry Outage: How can I PIN a large number of users?

We lost power due to the hurricane and need to notify BES users that services will be restored soon. How can I extract all BES users and send them a notification message? Also how do I alter the ...