Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

You've probably seen the messages at the stackoverflow blog and on codinghorror:

blog.stackoverflow.com experienced 100% data loss at our hosting provider, CrystalTech.

We're working to restore it from backups ASAP!

Some of the stuff Jeff's doing is on Twitter. What would you be doing in a similar situation?

share|improve this question
    
This isn't really any different than "What do you do when your server dies and you have no backup?" –  MikeyB Dec 13 '09 at 17:14
    
Here is a link to the question that Jeff posted to the superuser community: superuser.com/questions/82036/… –  faultyserver Dec 14 '09 at 20:03
add comment

10 Answers

up vote 5 down vote accepted

Firstly, make your own damn offsite backups. And test them. There are dirt cheap services that handle this. Or if you insist your expert hosting company should do this for you, make them restore an image from backup monthly just to keep them on their toes. But seriously, even just adding your own RSS feeds and setting the client to cache forever is a good first step.

Secondly, the Wayback Machine and Google cache can help. So can your own browser cache, so make copies of that before you thrash it. I hope you haven't been following the "clear your cache" step helpdesk likes to give.

share|improve this answer
    
+1 for the internet archive. –  Matt Simmons Dec 12 '09 at 0:41
add comment

Worst case (which they're probably doing now): reconstruct everything from sources as:

share|improve this answer
add comment

I would double up on my efforts to design and build a time machine. Once that project was complete I would go back to late last week and smack myself around the head until I:

  1. had a backup plan that involved more than one host (i.e. backing up to one or more machines held by one or more other providers
  2. had a backup testing plan, so the backups could be verified often enough
  3. had an offline backup arrangement of my own as well as any provided by the main hosting provider, i.e. backing up to a local machine and making tape/CD/DVD backups of that data regularly and storing them in a safe place.
  4. had implemented tested and retested all the above plans

In the absense of success in the TimeMachine project, all I would be able to do was wait patiently for the host to do what they could and hope that their backup arrangements were sufficient such that the data (or at least a recent copy there-of) can be restored in a short amount of time. I would then make sure that the above mentioned plans were made, implemented and regularly tested.

share|improve this answer
    
While you're travelling back in time, would you tell the 18 year old me to get off the sofa and stop watching so much TV? –  Wesley Dec 12 '09 at 0:05
1  
Use of my time machine would be more selfish. I might go back to the 18 year old me and try knock some sense into the idiot though. –  David Spillett Dec 12 '09 at 1:07
add comment

There isn't much to do. Find a new host, restore from backups.

If the host really wants to play ball and be nice they'd immediately freeze any usage of the disks in question and get them to a data recovery specialist... but in reality thats never gonna happen.

share|improve this answer
add comment

I tar and mysqldump all my hosting data to disk, then move it to a bigger RAID-type disk. I have never deleted a single copy of a backup. Yes, it takes up tons of disk space, but I value my user's data more than my own life! (Little exaggerated there, but I see it as my responsibility to make sure there is data integrity.)

I don't know why other companies don't do something similar - but I'll leave opinions to myself :).

If I were in your shoes, I would run a Linux (Ubuntu preferred) box and run programs like:

  • sshfs
  • curlftpfs

And mount my site as a local directory and use my own backup scripts to back up the data.

share|improve this answer
add comment

You select a new host (or stay if they have a really, really good reason for losing your data), and restore from your offsite backup ;) !

share|improve this answer
add comment

The first thing I would do is find out what 100% loss means. Even with disk failures, data can be extracted. Unless the whole server was melted in a fire, I'm sure something can be recovered.

share|improve this answer
add comment

You can modify your website, so when you create some post in blog it will automatically create this post on backup site (complete site mirror on other hosting, free hosting such as Google AppEngine, or your developer machine — but inaccessible for ordinal internet users). When your main hosting lose all data you can just copy it from backup hosting (and no changes in DB or site structure needed!). When your hosting is down you can just modify DNS record for your domain name with IP of backup site (and eventually change some access permissions on backup site).

And, ofcourse if you cannot afford backup hosting, you can use file system that have proved tools for recovery. I suggest NTFS for valuable "backups".

share|improve this answer
add comment

I use rsnapshot to back up my remote servers. It login in every night from the home office and backups everything that change during that day. It light on bandwidth and has a small foot print. On problem is it only works on UNIX-like systems.

share|improve this answer
add comment

I would think how wonderfully faster and better it would be to rebuild the second (or more) time around ...

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.