Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Here's a an entertaining tragedy. This morning I was doing a bit of maintenance on my production server, when I mistakenly executed the following command:

sudo rm -rf --no-preserve-root /mnt/hetznerbackup /

I didn't spot the last space before / and a few seconds later, when warnings was flooding my command line, I realised that I had just hit the self-destruct button. Here's a bit of what burned into my eyes:

rm: cannot remove `/mnt/hetznerbackup': Is a directory
rm: cannot remove `/sys/fs/ecryptfs/version': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/inode_readahead_blks': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_max_to_scan': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/delayed_allocation_blocks': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/max_writeback_mb_bump': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_stream_req': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_min_to_scan': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_stats': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/trigger_fs_error': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/session_write_kbytes': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/lifetime_write_kbytes': Operation not permitted
# and so on..

I stopped the task and was relieved when I discovered that the production service was still running. Sadly, the server no longer accept my public key or password for any user via SSH.

How would you move forward from here? I'll swim an ocean of barbed wire to get that SSH-access back.

The server is running Ubuntu-12.04 and hosted at Hetzner.

share|improve this question
32  
Restore from backups. Honestly, this is one of those no-easy-way-back scenarios. –  MadHatter Apr 7 at 6:45
108  
How do you even type --no-preserve-root accidentally?! :-o –  ThatGraemeGuy Apr 7 at 6:46
48  
Greame, the keys are like right next to each other. –  MadHatter Apr 7 at 6:47
12  
Tuesday work: Look for new job ;) Take it as a lesson why backups are needed. –  TomTom Apr 7 at 7:00
6  
This sure seems like trolling to me. You can't accidentally type --i-really-mean-delete-my-whole-root. –  psusi Apr 8 at 1:35

5 Answers 5

up vote 32 down vote accepted

Boot into the rescue system provided by Hetzner and check what damage you have done.
Transfer out any files to a safe location and redeploy the server afterwards.

I'm afraid that is the best solution in your case.

share|improve this answer
22  
look on the bright side, at least he has no problems with heartbleed! –  metacom Apr 8 at 20:04

When you delete stuff with rm -rf --no-preserve-root, its nigh impossible to recover. It's very likely you've lost all the important files.

As @faker said in his answer, the best course of action is to transfer the files to a safe location and redeploy the server afterwards.

To avoid similar situations in future, I'd suggest you:

  • Take backups weekly, or at least fortnightly. This would help you in getting the affected service back up with the least possible MTTR.

  • Don't work as root when not needed. And always think twice before doing anything. I'd suggest you also install safe-rm.

  • Don't type options that you don't intend to invoke, such as --no-preserve-root or --permission-to-kill-kittens-explicitly-granted, for that matter.

share|improve this answer
50  
The most sensible bit of advise is missing: "don't type options that you don't intend to invoke, such as --no-preserve-root or --permission-to-kill-kittens-explicitly-granted, for that matter" –  sehe Apr 7 at 9:57
1  
@user22394: Do you mind if I edit that into the answer? ;) –  Amal Murali Apr 7 at 10:01
    
Of course not :) –  sehe Apr 7 at 10:02
1  
@OlivierDulac isn't that obvious as it is? Also, it's not that witty :) (yes I just changed my nick /just/ to confuse you :)) –  sehe Apr 7 at 11:48
4  
Similarly, unless you REALLY MEAN IT, don't add the --please-destroy-my-drive parameter to hdparm. –  MikeyB Apr 8 at 6:17

The best way to fix a problem like this is to not have it in the first place.

Do not manually enter an "rm -rf" command that has a slash in the argument list. (Putting such commands in a shell script with really good validation/sanity routines to protect you from doing something stupid is different.)

Just don't do it.
Ever. If you think you need to do it, you aren't thinking hard enough.

Instead, change your working directory to the parent of the directory from which you intend to start the removal, so that the target of the rm command does not require a slash:

cd /mnt

sudo rm -rf hetznerbackup

share|improve this answer
3  
I always put the -rf at the end of the argument list, so rm /bla/foo/bar -rf. At least that way I'm not into a lot of trouble when I accendentily press return after typing the rm / part. –  Jens Timmerman Apr 14 at 16:37
    
Similarly, when removing "*~" files, I type the tilde first, then add in the asterisk. –  tekknolagi Apr 24 at 1:59

As mentioned in another answer, Hetzner has a rescue system. It includes both a netboot option with ssh access as well as a java applet to give you screen and keyboard on your vserver.

If you want to recover as much as possible, reboot the server into the netboot system and then log in and download an image of the filesystem by reading from the appropriate device inode.

I think something like this should work:

ssh root@host cat /dev/sda > server.img

Of course the redirection is done by the shell before the ssh command is invoked, so server.img is a local file. If you want just the root file system and not the full disk, replace sda by sda3 assuming you are using the same image as me.

share|improve this answer
    
could maybe be : ssh root@host cat /dev/sda | gzip -c - > /path/to/dir_on_huge_partition/server.img.gz (the on-the-fly gzip will or won't help depending on what the content of the filesystem is...) –  Olivier Dulac Apr 7 at 11:06
    
@OlivierDulac Using gzip that way would send the data uncompressed over the network and then compress it on the receiving side. I assume the result you intended to achieve was to compress the data while being transferred. The local image could be stored compressed or not, but tools you'd like to apply to that image later will not work with the compressed version. If all you want to achieve is compression of data while in transit, you can make use of the compression feature in ssh. It can be enabled with -C if it is not already enabled in your configuration. –  kasperd Apr 7 at 11:16
1  
I was more trying to reduce the size of the file. But if you want to save bandwidth (good idea) : just add quotes: ssh root@host "cat /dev/sda | gzip -c - " > /path/to/dir_on_huge_partition/server.img.gz (the -c option of ssh is usually good too, but you'd still need to compress at the end, as ssh will only compress at entrance of its tunnel and uncompress before sending to stdout) –  Olivier Dulac Apr 7 at 11:20

I don't know if this helps, but this story is entertaining as well. In this story, "rm -rf /" was allowed to run accidentally for a bit before being interrupted. The author tells of some very creative engineering using tools in unusual ways to reconstruct "/etc" and then "/etc/passwd", etc.

Unix Recovery Legend

share|improve this answer
    
Remember you should include a summary about the page you link to. If the page ceases to exist your answer becomes empty. –  gnp Apr 10 at 15:08

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.