150

Please note: The answers and comments to this question contains content from another, similar question that has received a lot of attention from outside media but turned out to be hoax question in some kind of viral marketing scheme. As we don't allow ServerFault to be abused in such a way, the original question has been deleted and the answers merged with this question.

Here's a an entertaining tragedy. This morning I was doing a bit of maintenance on my production server, when I mistakenly executed the following command:

sudo rm -rf --no-preserve-root /mnt/hetznerbackup /

I didn't spot the last space before / and a few seconds later, when warnings was flooding my command line, I realised that I had just hit the self-destruct button. Here's a bit of what burned into my eyes:

rm: cannot remove `/mnt/hetznerbackup': Is a directory
rm: cannot remove `/sys/fs/ecryptfs/version': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/inode_readahead_blks': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_max_to_scan': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/delayed_allocation_blocks': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/max_writeback_mb_bump': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_stream_req': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_min_to_scan': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_stats': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/trigger_fs_error': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/session_write_kbytes': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/lifetime_write_kbytes': Operation not permitted
# and so on..

I stopped the task and was relieved when I discovered that the production service was still running. Sadly, the server no longer accept my public key or password for any user via SSH.

How would you move forward from here? I'll swim an ocean of barbed wire to get that SSH-access back.

The server is running Ubuntu-12.04 and hosted at Hetzner.

Improve this question
33
  • 49
    Restore from backups. Honestly, this is one of those no-easy-way-back scenarios.
    – MadHatter
    Commented Apr 7, 2014 at 6:45
  • 322
    How do you even type --no-preserve-root accidentally?! :-o
    – ThatGraemeGuy
    Commented Apr 7, 2014 at 6:46
  • 152
    Greame, the keys are like right next to each other.
    – MadHatter
    Commented Apr 7, 2014 at 6:47
  • 41
    Tuesday work: Look for new job ;) Take it as a lesson why backups are needed.
    – TomTom
    Commented Apr 7, 2014 at 7:00
  • 46
    This sure seems like trolling to me. You can't accidentally type --i-really-mean-delete-my-whole-root.
    – psusi
    Commented Apr 8, 2014 at 1:35

9 Answers 9

Reset to default
97

Boot into the rescue system provided by Hetzner and check what damage you have done.
Transfer out any files to a safe location and redeploy the server afterwards.

I'm afraid that is the best solution in your case.

Improve this answer
1
  • 107
    look on the bright side, at least he has no problems with heartbleed!
    – metacom
    Commented Apr 8, 2014 at 20:04
224

Fact is? At this point, there's no simple/easy automatic fix for this. Data recovery is a science and even the basic, common tools need someone to sit down and ensure the data is there. If you're expecting to recover from this without massive amounts of downtime, you're going to be disappointed.

I'd suggest using testdisk or some file system specific recovery tool. Try one system, see if it works, and so on. There's no real way to automate the process but you can probably carefully do it in batches.

That said, there is a few very scary things in the questions and comments that ought to be part of your after action reports.

Firstly, you ran the command everywhere without checking it first. Run a command on one box. Then a few, then more. Basically if something goes wrong, its better to have it affect a few rather than all your systems.

Secondly

@Tim how to do a backup without mounting a remote drive on the server?

Scares me. File level one way backups are a solved problem. Rsync can be used to preserve permissions and copy over files one way to a backup site. Accidentally something? Reinstall (preferably automatically) rsync back, and things work. In future, you might use file system level snapshots with btrfs or zfs snapshots and shipping those for system level backups. I'd actually toy with separating application servers, databases and storage and introduce the principle of least privilege so you would split up the risk of something like this..

I know there is anything I can do. I now need to think how to protect myself

After something has happened is the worst time to consider this.

What can we learn from this?

  1. Backups save data. Possibly careers.
  2. If you have a tool and arn't aware if what it can do, its dangerous. A jedi can do amazing things with a lightsaber. A roomful of chimps with lightsabers... would get messy.

  3. Never run a command everywhere at once. Seperate out test and production machines, and preferably do production machines in stages. Its better to fix 1 or 10 machines rather than 100 or 1000.

  4. Double and triple check commands. There's no shame in asking a co worker to double check "hey, I'm about to dd a drive, could you sanity check this so I don't end up wiping a drive?". A wrapper might help as well, but nothing beats a less tired set of eyes.

What can you do now? Get an email out to customers. Let them know there's downtime and there's catastrophic failures. Talk to your higher ups, legal, sales and such and see how you can mitigate the damage. Start planning for recovery, and if needed you're going to have to, at best, hire extra hands. At worst, plan for spending a lot of money on recovery. At this stage, you're going to work at mitigating the fall out as well as technical fixes.

Improve this answer
33
  • 9
    @MarcoMarsala If you mounted anything before using rsync, you weren't doing it correctly. You should be using rsync over ssh.
    – Michael Hampton
    Commented Apr 11, 2016 at 8:19
  • 69
    I'd add to this excellent answer: Step away from the computer. Do not try to fix anything until you've calmed down. You're already looking at some serious downtime; taking the time to think things through instead of wrecking your systems even more (as in the dd issue above) is not going to make it worse.
    – Jenny D
    Commented Apr 11, 2016 at 8:44
  • 22
    Any idea why the command actually ran? If $foo and $bar were both undefined, rm -rf / should have errored out with the --no-preserve-root message. The only way I can think of that this would have actually worked on a CentOS7 machine is if $bar evaluated to *, so what was run was rm -rf /*.
    – terdon
    Commented Apr 11, 2016 at 9:23
  • 9
    I love the stylism in "Accidentally something?". That must mean the word "removed" was "deleted" or "dropped" accidentally.
    – sehe
    Commented Apr 11, 2016 at 20:17
  • 20
    @MarcoMarsala well at least you're famous now independent.co.uk/life-style/gadgets-and-tech/news/…
    – Martin Smith
    Commented Apr 14, 2016 at 15:06
93

When you delete stuff with rm -rf --no-preserve-root, its nigh impossible to recover. It's very likely you've lost all the important files.

As @faker said in his answer, the best course of action is to transfer the files to a safe location and redeploy the server afterwards.

To avoid similar situations in future, I'd suggest you:

  • Take backups weekly, or at least fortnightly. This would help you in getting the affected service back up with the least possible MTTR.

  • Don't work as root when not needed. And always think twice before doing anything. I'd suggest you also install safe-rm.

  • Don't type options that you don't intend to invoke, such as --no-preserve-root or --permission-to-kill-kittens-explicitly-granted, for that matter.

Improve this answer
2
  • 23
    Similarly, unless you REALLY MEAN IT, don't add the --please-destroy-my-drive parameter to hdparm.
    – MikeyB
    Commented Apr 8, 2014 at 6:17
  • 3
    I'd like to add; "Triple-check your arguments (and options) when working as root", "Check your CurrentWorkingDirectory (before doing something like rm -rf *)", and "Use full-paths to commands (don't relay on $PATH).
    – Baard Kopperud
    Commented Apr 8, 2014 at 10:41
48

I've had the same issue but just testing with a harddrive, I've lost everything. I don't know if it'll be useful but don't install anything, don't overwrite your data, you need to mount your hard drives and launch some forensics tools such us autopsy, photorec, Testdisk.

I strongly recommend Testdisk, with some basics command you can recover your data if you didn't overwrite it.

Improve this answer
3
  • 9
    I would definitely recommend takign the storage offline if at all possible and re-mounting as 'read only' if you can at all. Whether with a livedisk or another server instance.
    – mhouston100
    Commented Apr 12, 2016 at 0:21
  • 2
    I'd even consider doing a dd bitcopy of the original disk to a new disk from a read-only mount of the original disk just to be safe.
    – Jim
    Commented Apr 14, 2016 at 19:49
  • 3
    «these tools won't recover the file name and path» Yes, they do. Of the 3 mentioned tools, only one (Photorec) performs carving.
    – Andrea Lazzarotto
    Commented Apr 16, 2016 at 16:34
34

The best way to fix a problem like this is to not have it in the first place.

Do not manually enter an "rm -rf" command that has a slash in the argument list. (Putting such commands in a shell script with really good validation/sanity routines to protect you from doing something stupid is different.)

Just don't do it.
Ever. If you think you need to do it, you aren't thinking hard enough.

Instead, change your working directory to the parent of the directory from which you intend to start the removal, so that the target of the rm command does not require a slash:

cd /mnt

sudo rm -rf hetznerbackup

Improve this answer
6
  • 33
    I always put the -rf at the end of the argument list, so rm /bla/foo/bar -rf. At least that way I'm not into a lot of trouble when I accendentily press return after typing the rm / part.
    – Jens Timmerman
    Commented Apr 14, 2014 at 16:37
  • 5
    Similarly, when removing "*~" files, I type the tilde first, then add in the asterisk.
    – tekknolagi
    Commented Apr 24, 2014 at 1:59
  • 4
    So you'd rather delete your home than everything in the current directory ?!?
    – greg0ire
    Commented Apr 17, 2016 at 13:50
  • @greg0ire No, i think he wanted to say, that within /mnt/hetznerbackup, he had to use "/" to mark everything inside that folder.. but from parent, only hetznerbackup is enough, without slashes.
    – T.Todua
    Commented May 16, 2016 at 19:41
  • 1
    @tazotodua : I was referring to tekknolagi's comment
    – greg0ire
    Commented May 17, 2016 at 14:06
15

I would try to recover backup machine, where all copies were stored:

  • 1st step - Make a backup of this erased "backup machine" drives with dd comand.
  • 2nd step - Use testdisk to recover files.

So lets say you want to recover 1TB, You will need extra 2TB, 1TB for backup (1st step) plus 1TB for recovery (2nd step).

I did similar mistake with alias rm -fr [phone rang] and cd to precious directory. Now i always think twice and recheck couple times before i use rm or dd command.

Improve this answer
5
  • 6
    Pretty much zeroed your disk by doing that. That seriously makes it a lot harder to recover. There's a good reason the OP suggested you tried using testdisk, and recovering first, and while dd's syntax can be a little odd, that's a good reason to double and triple check before you run the command. You only wiped one server, right?
    – Journeyman Geek
    Commented Apr 11, 2016 at 7:16
  • 1
    You can still recover, depends how long you allowed dd to erase your last chance.
    – Abc Xyz
    Commented Apr 11, 2016 at 13:53
  • 129
    sorry to say that, but i feel huge troll in this question...
    – tymik
    Commented Apr 11, 2016 at 22:19
  • 3
    hope u feel small troll in the answer :)
    – Abc Xyz
    Commented Apr 11, 2016 at 22:27
  • 5
    To be honest. I'm not sure you're real. If you are, you're probably in the wrong job...
    – leftcase
    Commented Apr 14, 2016 at 19:05
7

As mentioned in another answer, Hetzner has a rescue system. It includes both a netboot option with ssh access as well as a java applet to give you screen and keyboard on your vserver.

If you want to recover as much as possible, reboot the server into the netboot system and then log in and download an image of the filesystem by reading from the appropriate device inode.

I think something like this should work:

ssh root@host cat /dev/sda > server.img

Of course the redirection is done by the shell before the ssh command is invoked, so server.img is a local file. If you want just the root file system and not the full disk, replace sda by sda3 assuming you are using the same image as me.

Improve this answer
3
  • could maybe be : ssh root@host cat /dev/sda | gzip -c - > /path/to/dir_on_huge_partition/server.img.gz (the on-the-fly gzip will or won't help depending on what the content of the filesystem is...)
    – Olivier Dulac
    Commented Apr 7, 2014 at 11:06
  • @OlivierDulac Using gzip that way would send the data uncompressed over the network and then compress it on the receiving side. I assume the result you intended to achieve was to compress the data while being transferred. The local image could be stored compressed or not, but tools you'd like to apply to that image later will not work with the compressed version. If all you want to achieve is compression of data while in transit, you can make use of the compression feature in ssh. It can be enabled with -C if it is not already enabled in your configuration.
    – kasperd
    Commented Apr 7, 2014 at 11:16
  • 2
    I was more trying to reduce the size of the file. But if you want to save bandwidth (good idea) : just add quotes: ssh root@host "cat /dev/sda | gzip -c - " > /path/to/dir_on_huge_partition/server.img.gz (the -c option of ssh is usually good too, but you'd still need to compress at the end, as ssh will only compress at entrance of its tunnel and uncompress before sending to stdout)
    – Olivier Dulac
    Commented Apr 7, 2014 at 11:20
3

How would you move forward from here?

I would swear off using rm for the rest of my life and think that it's madness that trash-cli isn't the default removal command on nix systems.

https://github.com/andreafrancia/trash-cli

I would make sure it is the first thing I install on a brand new system and alias rm to something that tells people to use trash-cli instead. It would also include a note about another alias that actually runs /bin/rm but tells them to avoid using it in most cases.

:( True story

Improve this answer
9
  • 3
    In my experience, these kind of tools are more like a nuisance than an actual help - sooner or later, and after some swearing, you'll remove it. It might be ok for a workstation, but in many if not most situations when you are doing administrative work on a server, you really need to delete the data, not just move it somewhere else (and if that were the case, just use mv instead). Besides, automatically moving data to a trash folder might lead to serious issues by itself (e.g. trash not on the same filesystem, security).
    – maetthu
    Commented Apr 17, 2016 at 10:01
  • @maetthu Oh of course things are removed after they have been in the trash for a certain number of days. Ubuntu desktop does this to items which have been in the trash more than 30 days. On a server you might want something shorter, eg. trash-empty 5 in a cron. The point is to allow you some grace period because humans make mistakes.
    – Gerry
    Commented Apr 17, 2016 at 13:29
  • Isn't it better to have a working desaster-recovery plan instead of banning essential system tools?
    – user292812
    Commented Apr 17, 2016 at 15:56
  • @user292812 I did not suggest banning /bin/rm, just that it shouldn't be the first option in most cases (note the /bin/rm alias). Your question also suggests a false choice between disaster recovery and a human friendly deletion option. You should have both.
    – Gerry
    Commented Apr 17, 2016 at 17:56
  • 1
    A two-step remove process can save a lot of trouble: 1. move to trash (verbosely), 2. empty trash. I alias such a script to "rm" and it has saved me from accidentally deleting important things many times.
    – Sam Watkins
    Commented Apr 18, 2016 at 13:54
2

I would advice in such case is unmount and use debugfs, and with help of lsdel you can list all recently removed files, which where not cleaned up from journals and then dump needed files. Fast search link for the same: http://www.linuxvoodoo.com/resources/howtos/debugfs

hope it will help someone. ;)

And yes, once of suggestions is to make script, which moved ream rm to real.rm and symlinc mv to rm ;)

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .