Tag Info

Hot answers tagged

121

If I were you, I'd call the company's insurance company, and have them place out an accelerometer. This way - the insurance company will know that you're not the one shaking the disks, and the insurance company will know for sure when every blast has gone off, in case your SAN dies at the exact same moment. (We did this.)


88

Now would be the time to make sure you've verified your backup solution. All the replacement hardware in the world won't save you if your backups are corrupt or have otherwise been rendered useless.


47

In addition to all the other excellent suggestions (particularly off-site backups) you should consider dust proofiing your room to the extent practical: Weather stripping around the doors, tape around the windows, etc. If you have external air intakes plan on changing the filters when the blasting is over. All that said, I wouldn't waste time/resources on ...


40

It's a good thing that you're thinking about what questions to ask your hosting company, but I think you're approaching it backwards. First figure out your requirements, and then ask each company how their infrastructure will meet them. When they're explaining how their infrastructure meets your needs don't be afraid to ask questions, and if you aren't ...


37

SUCCESS! I was able to retrieve the private key. But it wasn't easy. Here's what you need to do: Make sure you do not restart the server or Apache. The game is over at that point. That also means making sure that no monitoring services restart Apache. Grab this file - source code for a tool named passe-partout. Extract the source code and adjust line 9 of ...


28

Sure. I've had battery-backed cache (BBWC) and later flash-backed write cache (FBWC) protect in-flight data following crashes and sudden power loss. On HP ProLiant servers, the typical message is: POST Error: 1792-Drive Array Reports Valid Data Found in Array Accelerator Which means, "Hey, there's data in the write cache that survived the ...


25

Assuming Linux: umount -f -l /mnt/myfolder Will sort of fix the problem: -f Force unmount (in case of an unreachable NFS system). (Requires kernel 2.1.116 or later.) -l Lazy unmount. Detach the filesystem from the filesystem hierarchy now, and cleanup all references to the filesystem as soon as it is not busy anymore. (Requires kernel ...


25

Mount your server racks on ISO-Base platforms. This will isolate your server rack from the ground shocks and allow your systems to function even in the event of movement. It's not easy, but is effective. Especially in earthquake zones. To see the ISO-Base in action, check out this video showing the effects in a server rack during an earthquake simulation. ...


21

http://www.storagesearch.com/disklabs-art3-floods.html Do NOT attempt to recover the data yourself. This will do more damage to your data and makes it more difficult to recover when it eventually gets to a data recovery specialist. When hard disk drives get wet, the 'heads' can get stuck to the platters. When the hard drive is powered ...


19

First Order: Is it responsive? If you can't log in, there's bigger problems afoot. This generally comes in two flavors: hardware failure, and software failure. Both are potentially catastrophic. To prevent DFA errors, check the general hardware health first - a simple glance-over usually will suffice. Second Order: Are the system's underlying structures ...


17

Power: If you lost power, will all of the servers, storage, san (etc) in your rack reboot on their own? Or will they come up only after you press the 'On' button? This for two reasons: If the power is unstable for a couple of hours, you may want to leave your servers down until things stablize - less chance of a bouncing server being taken out by a ...


15

Please be aware that in PostgreSQL, your cluster data directory is a self-contained unit that cannot reliably be restored in parts, as per the documentation here. Your best hope is to use the full data directory of the old server and to start a server on it, then restore from a dump of the database you get from there: Get a postgresql.conf that fit the ...


14

What exactly does it do? The excerpt from this Compaq document explains it well: Power interruptions, even for brief moments, result in the loss of data which was being written to or read from storage... Power interruptions can have terminal effects on data which is in the process of being written and is temporarily residing in cache. This data does not yet ...


13

Actually neither, we use Documentation As-a-Testcase That being said we have written documentation that goes with Documentation As-a-Manual. We had checklists in place but when growing we found them to be error prone and really failing on the system as a whole. We do however have kind of "Documentation As-a-Checklist" installed, that is - as mentioned ...


13

The system is running very slowly because it has to reconstruct the missing data which involves additional CPU and I/O. If you have a missing disk in a RAID-5 configuration you have no recovery strategy. If another disk goes down you will lose your data. Run, don't walk, to the nearest vendor from which you can get a compatible part covered by ...


12

Most people talk about backup - not many talk about the restore. Make sure that restoring from backup is as easy as possible. In particular if time is critical. If you have a co-location replication, switch those to be the masters (if timing permits) since the ones in your building are much more vulnerable. They are also more likely to go down during work ...


11

Play heavy metal to replace the hard rock as it explodes. Hard rock is so passé anyway. ;) More seriously, maybe there's a DJ gear shop around the corner where you can pick up some shock absorpbtion/isolation stuff. Many (dance music) DJ's have a similarly hard time when they're playing records (i.e. keeping a tiny needle in a tiny spinning groove, somewhat ...


10

An excellent source of information is Disaster Recovery Journal (about). Community resources available include the current draft of their Generally Accepted Practices (GAP) document, which provides an excellent outline of the process and deliverables that constitute a solid business continuity plan and process. Also available are several white papers ...


10

Guilt and plausible disaster scenarios are a good start, but nothing teaches like a real disaster barely averted through heroic efforts. It took us a couple of years to convince the powers that be that our tape libraries needed replacing. It took far too much effort and we didn't get what we needed (we had to settle for SDLT320, couldn't afford LTO), but at ...


9

When writing mine I've always devolved into writing two three sets. The get-er-done checklist, with a MUCH LONGER appendix about the architecture of the system including why things are done the way they are, probable sticking points when coming online, and abstract design assumptions. followed by a list of probable problems and their resolutions, followed by ...


9

Make sure you have a emergency contact roster. aka a Recall Roster It should look like a tree, and show who contacts who. At the end of a branch, the last person should call the first and report anyone who could not be contacted. (This can be co-ordinated through HR, and used for any type of disaster)


8

Question 1: You're overthinking it. SQL 2005's database mirroring specifications are built into the connection string. You specify both the primary server and the mirror right in the connection string, and the driver tries the servers in order. That's one of the beauties of database mirroring - no client app changes are required. More info here: ...


8

When it comes to recovering data from a hosed disk or system, straight ext3 will be slightly easier than LVM+ext3, simply because it adds a bit of additional complexity and your recovery tools need to be LVM aware. However, LVM gives you significantly increased flexibility and is nearly always justified in my opinion. It can also make backups a lot easier ...


8

Elaborating upon the hint given by David Pashley, unless "umount -l" solves your problem, you can set up a fake server with the same address as the one that has gone away - but you don't actually have to set up a new sever or anything. The easiest way out of the blocking/hung umount situation is to set up a local alias IP interface, as follows: ifconfig ...


8

Disaster Recovery Plan is the proceedures the administrators do to restore normal business workflow. Business Continuity Plan is what end users do to remain productive when normal business workflow is interrupted.


8

You will face the "CAP" theorem problem. You cannot have consistency, availability and partition-tolerance at the same time. DRBD / MySQL HA relies on synchronous replication at the block device level. This is fine while both nodes are available, or if one suffers a temporary fault, is rebooted etc, then comes back. The problems start when you get a network ...


8

This type of consulting work is hard to evaluate. Building recommendations is a very, very political process as the consultant has to work across competing demands from different departments, and still stay within a budget. Or if it's the type of engagement where "money is no object," balancing just how high the sky can go before the client is likely to ...


8

I'm surprised no-one has mentioned the process side of things. This is the perfect opportunity to go over what you have planned for business continuity. What is the plan if you have to move offices for a day or a week? Do you have up-to-date plans on drawings and which systems have priority for restoration? Is management briefed that you do have a plan and ...


8

I used to run computers on a mine site that was being reclaimed. As the reclamation got closer and closer to the office, the data room would shake almost daily at the end of the day. My counter measures included moving the racks on top of some hard rubber feet designed for vibration dampening. They were made by Manson and I just estimated the weight of my ...


8

Most likely it is storing the key in memory, which it does because it needs to keep a copy after it drops privileges and/or decrypts the key using a supplied passphrase. In theory, you could get it out of the process image if you attached a debugger, though if they are following best practices it will be encrypted against something in memory. That said, if ...



Only top voted, non community-wiki answers of a minimum length are eligible