Smoothest workflow to handle SSH host verification errors?

Question

This is a simple issue that we all face and probably resolve manually without giving much thought.

As servers change, are re-provisioned, or IP addresses reallocated, we receive the SSH host verification message below. I'm interested in streamlining the workflow to resolve these ssh identification errors.

Given the following message, I typically vi /root/.ssh/known_hosts +434 and remove (dd) the offending line.

I've seen developers/users in other organizations delete their entire known_hosts file out of frustration in seeing this message. While I don't go that far, I know there's a more elegant way to handle this.

Tips?

[root@xt ~]# ssh las-db1

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
ed:86:a2:c4:cd:9b:c5:7a:b1:2b:cc:42:15:76:8c:56.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending key in /root/.ssh/known_hosts:434
RSA host key for las-db1 has changed and you have requested strict checking.
Host key verification failed.

Answer 1

You can use the ssh-keygen command to remove specific entries by host:

ssh-keygen -R las-db1

If you don't have that command, you could always use sed:

sed -i '/las-db1/d' /root/.ssh/known_hosts

Answer 2

As a puppet user, my method for resolving this is to actually have my puppet server collect my SSH host keys and publish them to all my systems that make SSH connections.

This way I don't have to worry about removing them. Ninety-nine percent of the time puppet has run and updated the keys for me since I have my agents running every thirty minutes. The exceptions for me are very rare, and so I don't mind a quick edit of the system wide known_hosts if I am not willing to wait.

class ssh::hostkeys {

  @@sshkey { "${::clientcert}_rsa":
    type => rsa,
    key => $sshrsakey,
    tag => 'rsa_key',
  }

  if 'true' == $common::params::sshclient {
    Sshkey <<| tag == 'rsa_key' |>> {
      ensure => present
    }
  }

  file {'/etc/ssh/ssh_known_hosts':
    ensure => present,
    owner => 'root',
    group => 'root',
    mode => 0644,
  }

}

Answer 3

I would like to add a suggestion that can help you in very specific cases where security is of less concern.

I have a lab environment with machines that get reinstalled often. Every time that happens, new host keys get generated (I could probably save the host key somewhere and set it in the post-install script).

Since security is not an issue for me in this lab environment, and the keys change so often, I have the following in my .ssh/config file:

Host lab-*
  User kenny
  IdentityFile ~/.ssh/lab_id_rsa
  StrictHostKeyChecking no
  UserKnownHostsFile=/dev/null

This makes sure that connecting to my lab machines will never cause that error again and my ssh client will just connect without checking the host key.

This is something you should only do if security is of no concern to you at all, because this puts you in a vulnerable position for a man-in-the-middle attack.

Answer 4

According to openssh 5.6 release note , you don't need to use hosts keys anymore :

Hostbased authentication may now use certificate host keys. CA keys must be specified in a known_hosts file using the @cert-authority marker as described in sshd(8).

Warning : I've never heard of anyone using this feature in production so far. Use at your own risk (but I believe openssh developpers are paranoid enough not to release such a killer feature without a lot of testing and code auditing).

Answer 5

SSHFP DNS ResourceRecord may help depending on how much your ssh client takes advantage of it. Store all your ssh public key fingerprints up there with the host name.

Implement DNSSEC or DNS over SSL beforehand.
http://www.ietf.org/rfc/rfc4255.txt

FreeIPA.org handles host and user key management as well as PKI Certificates. It also automatically uploads SSHFP DNS records when new keys are created.

The System Security Services Daemon (SSSD) can be configured to cache and retrieve host SSH keys so that applications and services only have to look in one location for host keys. Because SSSD can use FreeIPA as one of its identity information providers, FreeIPA provides a universal and centralized repository of keys. Administrators do not need to worry about distributing, updating, or verifying host SSH keys.

http://docs.fedoraproject.org/en-US/Fedora/17/html/FreeIPA_Guide/host-keys.html