Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Please note: The answers and comments to this question contains content from another, similar question that has received a lot of attention from outside media but turned out to be hoax question in some kind of viral marketing scheme. As we don't allow ServerFault to be abused in such a way, the original question has been deleted and the answers merged with this question.


Here's a an entertaining tragedy. This morning I was doing a bit of maintenance on my production server, when I mistakenly executed the following command:

sudo rm -rf --no-preserve-root /mnt/hetznerbackup /

I didn't spot the last space before / and a few seconds later, when warnings was flooding my command line, I realised that I had just hit the self-destruct button. Here's a bit of what burned into my eyes:

rm: cannot remove `/mnt/hetznerbackup': Is a directory
rm: cannot remove `/sys/fs/ecryptfs/version': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/inode_readahead_blks': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_max_to_scan': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/delayed_allocation_blocks': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/max_writeback_mb_bump': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_stream_req': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_min_to_scan': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_stats': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/trigger_fs_error': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/session_write_kbytes': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/lifetime_write_kbytes': Operation not permitted
# and so on..

I stopped the task and was relieved when I discovered that the production service was still running. Sadly, the server no longer accept my public key or password for any user via SSH.

How would you move forward from here? I'll swim an ocean of barbed wire to get that SSH-access back.

The server is running Ubuntu-12.04 and hosted at Hetzner.

share|improve this question
47  
Restore from backups. Honestly, this is one of those no-easy-way-back scenarios. – MadHatter Apr 7 '14 at 6:45
268  
How do you even type --no-preserve-root accidentally?! :-o – ThatGraemeGuy Apr 7 '14 at 6:46
122  
Greame, the keys are like right next to each other. – MadHatter Apr 7 '14 at 6:47
31  
Tuesday work: Look for new job ;) Take it as a lesson why backups are needed. – TomTom Apr 7 '14 at 7:00
34  
This sure seems like trolling to me. You can't accidentally type --i-really-mean-delete-my-whole-root. – psusi Apr 8 '14 at 1:35

10 Answers 10

up vote 82 down vote accepted

Boot into the rescue system provided by Hetzner and check what damage you have done.
Transfer out any files to a safe location and redeploy the server afterwards.

I'm afraid that is the best solution in your case.

share|improve this answer
85  
look on the bright side, at least he has no problems with heartbleed! – metacom Apr 8 '14 at 20:04

Fact is? At this point, there's no simple/easy automatic fix for this. Data recovery is a science and even the basic, common tools need someone to sit down and ensure the data is there. If you're expecting to recover from this without massive amounts of downtime, you're going to be disappointed.

I'd suggest using testdisk or some file system specific recovery tool. Try one system, see if it works, and so on. There's no real way to automate the process but you can probably carefully do it in batches.

That said, there is a few very scary things in the questions and comments that ought to be part of your after action reports.

Firstly, you ran the command everywhere without checking it first. Run a command on one box. Then a few, then more. Basically if something goes wrong, its better to have it affect a few rather than all your systems.

Secondly

@Tim how to do a backup without mounting a remote drive on the server?

Scares me. File level one way backups are a solved problem. Rsync can be used to preserve permissions and copy over files one way to a backup site. Accidentally something? Reinstall (preferably automatically) rsync back, and things work. In future, you might use file system level snapshots with btrfs or zfs snapshots and shipping those for system level backups. I'd actually toy with separating application servers, databases and storage and introduce the principle of least privilege so you would split up the risk of something like this..

I know there is anything I can do. I now need to think how to protect myself

After something has happened is the worst time to consider this.

What can we learn from this?

  1. Backups save data. Possibly careers.
  2. If you have a tool and arn't aware if what it can do, its dangerous. A jedi can do amazing things with a lightsaber. A roomful of chimps with lightsabers... would get messy.
  3. Never run a command everywhere at once. Seperate out test and production machines, and preferably do production machines in stages. Its better to fix 1 or 10 machines rather than 100 or 1000.

  4. Double and triple check commands. There's no shame in asking a co worker to double check "hey, I'm about to dd a drive, could you sanity check this so I don't end up wiping a drive?". A wrapper might help as well, but nothing beats a less tired set of eyes.

What can you do now? Get an email out to customers. Let them know there's downtime and there's catastrophic failures. Talk to your higher ups, legal, sales and such and see how you can mitigate the damage. Start planning for recovery, and if needed you're going to have to, at best, hire extra hands. At worst, plan for spending a lot of money on recovery. At this stage, you're going to work at mitigating the fall out as well as technical fixes.

share|improve this answer
58  
I'd add to this excellent answer: Step away from the computer. Do not try to fix anything until you've calmed down. You're already looking at some serious downtime; taking the time to think things through instead of wrecking your systems even more (as in the dd issue above) is not going to make it worse. – Jenny D Apr 11 at 8:44
21  
Any idea why the command actually ran? If $foo and $bar were both undefined, rm -rf / should have errored out with the --no-preserve-root message. The only way I can think of that this would have actually worked on a CentOS7 machine is if $bar evaluated to *, so what was run was rm -rf /*. – terdon Apr 11 at 9:23
19  
@MarcoMarsala well at least you're famous now independent.co.uk/life-style/gadgets-and-tech/news/… – Martin Smith Apr 14 at 15:06
27  
@MartinSmith luckily we recovered almost all data! – bleemboy Apr 14 at 16:04
25  
@MarcoMarsala Can you tell us the steps you took to recover the data? – kkaosninja Apr 14 at 17:25

When you delete stuff with rm -rf --no-preserve-root, its nigh impossible to recover. It's very likely you've lost all the important files.

As @faker said in his answer, the best course of action is to transfer the files to a safe location and redeploy the server afterwards.

To avoid similar situations in future, I'd suggest you:

  • Take backups weekly, or at least fortnightly. This would help you in getting the affected service back up with the least possible MTTR.

  • Don't work as root when not needed. And always think twice before doing anything. I'd suggest you also install safe-rm.

  • Don't type options that you don't intend to invoke, such as --no-preserve-root or --permission-to-kill-kittens-explicitly-granted, for that matter.

share|improve this answer
112  
The most sensible bit of advise is missing: "don't type options that you don't intend to invoke, such as --no-preserve-root or --permission-to-kill-kittens-explicitly-granted, for that matter" – sehe Apr 7 '14 at 9:57
2  
@user22394: Do you mind if I edit that into the answer? ;) – Amal Murali Apr 7 '14 at 10:01
2  
@OlivierDulac isn't that obvious as it is? Also, it's not that witty :) (yes I just changed my nick /just/ to confuse you :)) – sehe Apr 7 '14 at 11:48
13  
Similarly, unless you REALLY MEAN IT, don't add the --please-destroy-my-drive parameter to hdparm. – MikeyB Apr 8 '14 at 6:17
2  
I'd like to add; "Triple-check your arguments (and options) when working as root", "Check your CurrentWorkingDirectory (before doing something like rm -rf *)", and "Use full-paths to commands (don't relay on $PATH). – Baard Kopperud Apr 8 '14 at 10:41

I've had the same issue but just testing with a harddrive, I've lost everything. I don't know if it'll be useful but don't install anything, don't overwrite your data, you need to mount your hard drives and launch some forensics tools such us autopsy, photorec, Testdisk.

I strongly recommend Testdisk, with some basics command you can recover your data if you didn't overwrite it.

share|improve this answer
7  
I would definitely recommend takign the storage offline if at all possible and re-mounting as 'read only' if you can at all. Whether with a livedisk or another server instance. – mhouston100 Apr 12 at 0:21
2  
I'd even consider doing a dd bitcopy of the original disk to a new disk from a read-only mount of the original disk just to be safe. – Jim Apr 14 at 19:49
1  
these tools won't recover the file name and path. they're good if you are recovering your wedding' photos, your films, your mp3 songs, where the file name is not so important... but if you're recovering .php files that made up a web site? These softwares rely on the fact binary files have start and end markers. Text files not always haven't it (yes for php you can look ad <?php ?> but not so robust as idea) – bleemboy Apr 15 at 8:11
3  
«these tools won't recover the file name and path» Yes, they do. Of the 3 mentioned tools, only one (Photorec) performs carving. – Andrea Lazzarotto Apr 16 at 16:34

The best way to fix a problem like this is to not have it in the first place.

Do not manually enter an "rm -rf" command that has a slash in the argument list. (Putting such commands in a shell script with really good validation/sanity routines to protect you from doing something stupid is different.)

Just don't do it.
Ever. If you think you need to do it, you aren't thinking hard enough.

Instead, change your working directory to the parent of the directory from which you intend to start the removal, so that the target of the rm command does not require a slash:

cd /mnt

sudo rm -rf hetznerbackup

share|improve this answer
29  
I always put the -rf at the end of the argument list, so rm /bla/foo/bar -rf. At least that way I'm not into a lot of trouble when I accendentily press return after typing the rm / part. – Jens Timmerman Apr 14 '14 at 16:37
5  
Similarly, when removing "*~" files, I type the tilde first, then add in the asterisk. – tekknolagi Apr 24 '14 at 1:59
2  
So you'd rather delete your home than everything in the current directory ?!? – greg0ire Apr 17 at 13:50
    
@greg0ire No, i think he wanted to say, that within /mnt/hetznerbackup, he had to use "/" to mark everything inside that folder.. but from parent, only hetznerbackup is enough, without slashes. – tazo todua May 16 at 19:41
    
@tazotodua : I was referring to tekknolagi's comment – greg0ire May 17 at 14:06

I would try to recover backup machine, where all copies were stored:

  • 1st step - Make a backup of this erased "backup machine" drives with dd comand.
  • 2nd step - Use testdisk to recover files.

So lets say you want to recover 1TB, You will need extra 2TB, 1TB for backup (1st step) plus 1TB for recovery (2nd step).

I did similar mistake with alias rm -fr [phone rang] and cd to precious directory. Now i always think twice and recheck couple times before i use rm or dd command.

share|improve this answer
7  
I swapped if and of while doing dd. What to do now? – bleemboy Apr 11 at 7:02
6  
Pretty much zeroed your disk by doing that. That seriously makes it a lot harder to recover. There's a good reason the OP suggested you tried using testdisk, and recovering first, and while dd's syntax can be a little odd, that's a good reason to double and triple check before you run the command. You only wiped one server, right? – Journeyman Geek Apr 11 at 7:16
128  
sorry to say that, but i feel huge troll in this question... – tymik Apr 11 at 22:19
3  
hope u feel small troll in the answer :) – Abc Xyz Apr 11 at 22:27
5  
To be honest. I'm not sure you're real. If you are, you're probably in the wrong job... – leftcase Apr 14 at 19:05

As mentioned in another answer, Hetzner has a rescue system. It includes both a netboot option with ssh access as well as a java applet to give you screen and keyboard on your vserver.

If you want to recover as much as possible, reboot the server into the netboot system and then log in and download an image of the filesystem by reading from the appropriate device inode.

I think something like this should work:

ssh root@host cat /dev/sda > server.img

Of course the redirection is done by the shell before the ssh command is invoked, so server.img is a local file. If you want just the root file system and not the full disk, replace sda by sda3 assuming you are using the same image as me.

share|improve this answer
    
could maybe be : ssh root@host cat /dev/sda | gzip -c - > /path/to/dir_on_huge_partition/server.img.gz (the on-the-fly gzip will or won't help depending on what the content of the filesystem is...) – Olivier Dulac Apr 7 '14 at 11:06
    
@OlivierDulac Using gzip that way would send the data uncompressed over the network and then compress it on the receiving side. I assume the result you intended to achieve was to compress the data while being transferred. The local image could be stored compressed or not, but tools you'd like to apply to that image later will not work with the compressed version. If all you want to achieve is compression of data while in transit, you can make use of the compression feature in ssh. It can be enabled with -C if it is not already enabled in your configuration. – kasperd Apr 7 '14 at 11:16
1  
I was more trying to reduce the size of the file. But if you want to save bandwidth (good idea) : just add quotes: ssh root@host "cat /dev/sda | gzip -c - " > /path/to/dir_on_huge_partition/server.img.gz (the -c option of ssh is usually good too, but you'd still need to compress at the end, as ssh will only compress at entrance of its tunnel and uncompress before sending to stdout) – Olivier Dulac Apr 7 '14 at 11:20

I would advice in such case is unmount and use debugfs, and with help of lsdel you can list all recently removed files, which where not cleaned up from journals and then dump needed files. Fast search link for the same: http://www.linuxvoodoo.com/resources/howtos/debugfs

hope it will help someone. ;)

And yes, once of suggestions is to make script, which moved ream rm to real.rm and symlinc mv to rm ;)

share|improve this answer

How would you move forward from here?

I would swear off using rm for the rest of my life and think that it's madness that trash-cli isn't the default removal command on nix systems.

https://github.com/andreafrancia/trash-cli

I would make sure it is the first thing I install on a brand new system and alias rm to something that tells people to use trash-cli instead. It would also include a note about another alias that actually runs /bin/rm but tells them to avoid using it in most cases.

:( True story

share|improve this answer
1  
In my experience, these kind of tools are more like a nuisance than an actual help - sooner or later, and after some swearing, you'll remove it. It might be ok for a workstation, but in many if not most situations when you are doing administrative work on a server, you really need to delete the data, not just move it somewhere else (and if that were the case, just use mv instead). Besides, automatically moving data to a trash folder might lead to serious issues by itself (e.g. trash not on the same filesystem, security). – maetthu Apr 17 at 10:01
    
@maetthu Oh of course things are removed after they have been in the trash for a certain number of days. Ubuntu desktop does this to items which have been in the trash more than 30 days. On a server you might want something shorter, eg. trash-empty 5 in a cron. The point is to allow you some grace period because humans make mistakes. – Gerry Apr 17 at 13:29
    
Isn't it better to have a working desaster-recovery plan instead of banning essential system tools? – user292812 Apr 17 at 15:56
    
@user292812 I did not suggest banning /bin/rm, just that it shouldn't be the first option in most cases (note the /bin/rm alias). Your question also suggests a false choice between disaster recovery and a human friendly deletion option. You should have both. – Gerry Apr 17 at 17:56
1  
A two-step remove process can save a lot of trouble: 1. move to trash (verbosely), 2. empty trash. I alias such a script to "rm" and it has saved me from accidentally deleting important things many times. – Sam Watkins Apr 18 at 13:54

Stop all server processess and everything which may cause disk i/o ... then run testdisk, it should be in your software stack. If you have physical access, use a livecd with testdisk.

share|improve this answer
1  
I don't quite get why you think three answers providing the exact same suggestion was not enough? – kasperd Apr 17 at 18:24

protected by Michael Hampton Apr 14 at 15:02

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.