Google cloud server ssh host key changed unexpectedly?

Question

I have a Google cloud server that currently runs Ubuntu 16.04. I have not done much to it in years and it always ran great.

Somehow or other, for the first time, it experienced downtime today of about 15 minutes.

After I restarted it, ssh gives me a warning "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!". I am certain that I, the sole administrator, did not initiate any such changes. Last time I logged in, was yesterday.

While I can, obviously, change the known hosts file and adjust to the new host key, I am extremely disturbed by this change and I am suspecting that something is amiss. Would anyone have any clue as to why this count happen?

I did log on to it from a google cloud ssh window and, on the first look, it runs as usual running my website. However, right now I am overwhelmed with paranoia and what it could be.

Thanks!

Answer 1

Same thing happened to me today (Google Compute Cloud VM, running CentOS 7.x). After trying to log in from computers on different networks, vpns etc.. (e.g. to eliminate the possibility of a man in the middle on my end) I eventually bit the bullet and logged into one of my less important machines (that uses a different private key than my prod server, but was also affected by the same issue).

Long story short, yes, it does look like some Google VMs automatically change their own config. On CentOS you can check in /etc/yum/yum-cron.conf and you'll probably see: apply_updates=yes

Then also checking ls -lh /etc/ssh you might see something like this:

Then finally, checking /var/log/messages you might also see evidence of an automatic update affecting the ssh configuration:

Answer 2

Managing SSH keys changing has been a consideration for about as long as SSH has been around. Here's one of several Server Fault questions on the topic: SSH host key seems to be changing unexpectedly. In short, the key changed, configuration changed, or someone is doing something nasty.

Look through logs and your backups. Compare the keys before to the key now, including if the key type changed. Check if any SSH packages were upgraded on the server or client. Look for log file entries such as cloud-init which may have rekeyed if VM identification changed.

A cautious option is to rebuild a clean OS image, with new keys and credentials. Only restore the application data from backups. This work may be overkill. However, it also can be a useful business continuity exercise to prove you can abandon the instance but keep your data.