We need to recover from overwriting the authorized_keys files on a set of servers. Remote, embedded Linux devices were configured to ssh into Ubuntu 14.04 servers autonomously. We do not have record of the ssh public keys for those devices, and we cannot physically get to these devices. We need to allow them to connect to the server again via ssh without a password. They do not always connect using the same IP address. They do always connect using an IP address associated with T-mobile in the USA. The devices will continue to attempt to log in. We just need to allow them in. Once they're connected, reverse ssh is already configured, so we can log in and get the ssh-keys. Is there a way to fix this?


If you have no backup you may need to resort to data recovery techniques. Your millage will vary depending on the filesystem in use, if compression or encryption are being used, what tool was used to overwrite the file, how long it has been since the file has been overwritten ect...

Ideally the moment you realize you need to perform data recovery from a hard disk you should power off the machine and make a block level image of the hard using dd. If the partition is just being used for your home directories and not by software that needs to write to it you can get away with just remounting it as read only.

It depends on what tools are being used but its likely that when the file was overwritten that the new contents where written to a new location on the hard drive and previous locations marked as freed but not immediately overwritten.

This would be like changing the page number in the table of contents of a book to point at a new page but leaving the old pages in the book with no reference too them.

The longer you wait though the more likely new data will be written to the old location which is now marked as free and usable space for any new data. Once that happens no software tools can recover the data.

I cant go over every possible data recovery technique here but the most basic way to start might be with a command such as this. Replacing /dev/sda1 with whatever device/partition the authorized_keys files lived on or better yet a image of that partition.

sudo grep -z -i -a '^ssh-rsa AAAAB3NzaC1yc2E' /dev/sda1

You will likely a high noise to signal ratio and need to pick through the output looking for any valid ssh-key. Again your millage may vary.

| improve this answer | |
  • Data recovery yielded 100% of the lost ssh keys. Thanks for the speedy advice! Backups were turned on for most, though not all of our servers. Expect not to make that same mistake again...but I'm sure there will be others. – Ken Sills Apr 8 '16 at 14:49

Hmmmm.....If you have the private key, I believe it is possible to regenerate the public key from that.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.