Tag Info

Hot answers tagged

124

If I were you, I'd call the company's insurance company, and have them place out an accelerometer. This way - the insurance company will know that you're not the one shaking the disks, and the insurance company will know for sure when every blast has gone off, in case your SAN dies at the exact same moment. (We did this.)


91

Now would be the time to make sure you've verified your backup solution. All the replacement hardware in the world won't save you if your backups are corrupt or have otherwise been rendered useless.


59

Stay Calm Don't freak out. Breathe! (From the diaphragm, it helps.) If you've studied meditation, that can help too. When faced with extreme stress your body will go into a flight-or-fight mode, because your body thinks it's in a life-or-death situation. At this time your body will actually pump less blood to some parts of your brain, lessening functions ...


49

In addition to all the other excellent suggestions (particularly off-site backups) you should consider dust proofiing your room to the extent practical: Weather stripping around the doors, tape around the windows, etc. If you have external air intakes plan on changing the filters when the blasting is over. All that said, I wouldn't waste time/resources on ...


40

It's a good thing that you're thinking about what questions to ask your hosting company, but I think you're approaching it backwards. First figure out your requirements, and then ask each company how their infrastructure will meet them. When they're explaining how their infrastructure meets your needs don't be afraid to ask questions, and if you aren't ...


37

SUCCESS! I was able to retrieve the private key. But it wasn't easy. Here's what you need to do: Make sure you do not restart the server or Apache. The game is over at that point. That also means making sure that no monitoring services restart Apache. Grab this file - source code for a tool named passe-partout. Extract the source code and adjust line 9 of ...


35

The first answer is stay calm! I learned that the hard way that panicking often just makes things worse. Once thats achieved the next thing is to actually ascertain what the problem is. Complaints from users and managers will be coming at you from all angles, telling you what THEY cannot do, but not what the problem is. Once you know the problem you can ...


30

Sure. I've had battery-backed cache (BBWC) and later flash-backed write cache (FBWC) protect in-flight data following crashes and sudden power loss. On HP ProLiant servers, the typical message is: POST Error: 1792-Drive Array Reports Valid Data Found in Array Accelerator Which means, "Hey, there's data in the write cache that survived the ...


26

Assuming Linux: umount -f -l /mnt/myfolder Will sort of fix the problem: -f Force unmount (in case of an unreachable NFS system). (Requires kernel 2.1.116 or later.) -l Lazy unmount. Detach the filesystem from the filesystem hierarchy now, and cleanup all references to the filesystem as soon as it is not busy anymore. (Requires kernel ...


25

Mount your server racks on ISO-Base platforms. This will isolate your server rack from the ground shocks and allow your systems to function even in the event of movement. It's not easy, but is effective. Especially in earthquake zones. To see the ISO-Base in action, check out this video showing the effects in a server rack during an earthquake simulation. ...


24

Don't Panic.


23

http://www.storagesearch.com/disklabs-art3-floods.html Do NOT attempt to recover the data yourself. This will do more damage to your data and makes it more difficult to recover when it eventually gets to a data recovery specialist. When hard disk drives get wet, the 'heads' can get stuck to the platters. When the hard drive is powered ...


22

Step 0. Check that it's not your monitoring system that is at fault


19

First Order: Is it responsive? If you can't log in, there's bigger problems afoot. This generally comes in two flavors: hardware failure, and software failure. Both are potentially catastrophic. To prevent DFA errors, check the general hardware health first - a simple glance-over usually will suffice. Second Order: Are the system's underlying structures ...


18

First, it's YOUR business and the first step is for YOU to determine what your business continuity and disaster recovery needs and objectives are. Have you defined and documented those? If not, do so. A BC/DR is NOT just about the technology and the data. Once you've done that you can present them to this person and tell him you need him to provide ...


17

Power: If you lost power, will all of the servers, storage, san (etc) in your rack reboot on their own? Or will they come up only after you press the 'On' button? This for two reasons: If the power is unstable for a couple of hours, you may want to leave your servers down until things stablize - less chance of a bouncing server being taken out by a ...


15

Please be aware that in PostgreSQL, your cluster data directory is a self-contained unit that cannot reliably be restored in parts, as per the documentation here. Your best hope is to use the full data directory of the old server and to start a server on it, then restore from a dump of the database you get from there: Get a postgresql.conf that fit the ...


15

What exactly does it do? The excerpt from this Compaq document explains it well: Power interruptions, even for brief moments, result in the loss of data which was being written to or read from storage... Power interruptions can have terminal effects on data which is in the process of being written and is temporarily residing in cache. This data does not yet ...


13

Actually neither, we use Documentation As-a-Testcase That being said we have written documentation that goes with Documentation As-a-Manual. We had checklists in place but when growing we found them to be error prone and really failing on the system as a whole. We do however have kind of "Documentation As-a-Checklist" installed, that is - as mentioned ...


13

The system is running very slowly because it has to reconstruct the missing data which involves additional CPU and I/O. If you have a missing disk in a RAID-5 configuration you have no recovery strategy. If another disk goes down you will lose your data. Run, don't walk, to the nearest vendor from which you can get a compatible part covered by ...


13

I'd ask him why he doesn't have one in writing now, what he needs to get that done... and then give it to him and hold him accountable for delivering. I've been in a number of small shops as the lead or only IT guy, and didn't have a written DR or backup plan because I never had time to write one up, or more than 5 minutes between firefighting excursions. ...


12

Most people talk about backup - not many talk about the restore. Make sure that restoring from backup is as easy as possible. In particular if time is critical. If you have a co-location replication, switch those to be the masters (if timing permits) since the ones in your building are much more vulnerable. They are also more likely to go down during work ...


12

log in to serverfault


11

Immediately book a flight to a non-extradition country


11

Play heavy metal to replace the hard rock as it explodes. Hard rock is so passé anyway. ;) More seriously, maybe there's a DJ gear shop around the corner where you can pick up some shock absorpbtion/isolation stuff. Many (dance music) DJ's have a similarly hard time when they're playing records (i.e. keeping a tiny needle in a tiny spinning groove, somewhat ...


10

When writing mine I've always devolved into writing two three sets. The get-er-done checklist, with a MUCH LONGER appendix about the architecture of the system including why things are done the way they are, probable sticking points when coming online, and abstract design assumptions. followed by a list of probable problems and their resolutions, followed by ...


10

An excellent source of information is Disaster Recovery Journal (about). Community resources available include the current draft of their Generally Accepted Practices (GAP) document, which provides an excellent outline of the process and deliverables that constitute a solid business continuity plan and process. Also available are several white papers ...


10

Guilt and plausible disaster scenarios are a good start, but nothing teaches like a real disaster barely averted through heroic efforts. It took us a couple of years to convince the powers that be that our tape libraries needed replacing. It took far too much effort and we didn't get what we needed (we had to settle for SDLT320, couldn't afford LTO), but at ...


9

You will face the "CAP" theorem problem. You cannot have consistency, availability and partition-tolerance at the same time. DRBD / MySQL HA relies on synchronous replication at the block device level. This is fine while both nodes are available, or if one suffers a temporary fault, is rebooted etc, then comes back. The problems start when you get a network ...


9

Make sure you have a emergency contact roster. aka a Recall Roster It should look like a tree, and show who contacts who. At the end of a branch, the last person should call the first and report anyone who could not be contacted. (This can be co-ordinated through HR, and used for any type of disaster)



Only top voted, non community-wiki answers of a minimum length are eligible