3

I want to update my ~/.ssh/known_hosts with the host key information for a newly created GCE instance. But I'm not sure how to securely retrieve that information.

I thought something like

gcloud compute ssh <GCEUSER>@<GCEHOST> --command='ssh-keyscan 127.0.0.1'

might work. But that (per the gcloud compute ssh documentation) appears to just be a wrapper for ssh (and, based on seeing StrictHostKeyChecking=no in the parameters listed in the associated log file under $HOME/.config/gcloud/logs/, apparently isn't doing any sort of checking on the host's identity).

There does seem to be a way to use the web console to launch a browser-based ssh session (and interactively/manually run ssh-keyscan), but 1) I can't see the internals to know if it really is as secure as it should be and 2) isn't an effective API for script integration.

Is there an API/gcloud mechanism for securely retrieving the GCE instance's host key?

0

No, there is no straightforward way to do this (there is an open issue to remedy this security risk).

  • A proposed workaround is to have the base image (on initial instance spin-up) create a cert (e.g., via let's encrypt/certbot), enable an HTTP server, and then post its host keys via a well-defined web page. I've not actually tried this so I'm not offering it as a solution. – jhfrontz Dec 14 '18 at 0:37
-1

Host keys are already retrieved for you to the box you are running gcloud on, in ~/.ssh/google_compute_known_hosts.

Per --help, the behavior of host key checking is as follows:

 --strict-host-key-checking=STRICT_HOST_KEY_CHECKING
    Override the default behavior of StrictHostKeyChecking for the
    connection. By default, StrictHostKeyChecking is set to 'no' the first
    time you connect to an instance, and will be set to 'yes' for all
    subsequent connections. STRICT_HOST_KEY_CHECKING must be one of: yes,
    no, ask.

Which is the behavior in the log file if you look at subsequent connections.

2018-08-04 14:48:29,222 DEBUG    root            Running command [/usr/bin/ssh -t -i /home/john/.ssh/google_compute_engine -o CheckHostIP=no -o HostKeyAlias=compute.1520573357386694112 -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/home/john/.ssh/google_compute_known_hosts john@35.188.181.248].

2018-08-04 14:51:06,249 DEBUG    root            Executing command: [u'/usr/bin/ssh', u'-t', u'-i', u'/home/john/.ssh/google_compute_engine', u'-o', u'CheckHostIP=no', u'-o', u'HostKeyAlias=compute.1520573357386694112', u'-o', u'IdentitiesOnly=yes', u'-o', u'StrictHostKeyChecking=yes', u'-o', u'UserKnownHostsFile=/home/john/.ssh/google_compute_known_hosts', u'john@35.188.181.248']

This is a convenience so the first connection will not ask about the host, but future ones will check.

Effectively it is the same thing as a user seeing an unknown host key from their client and accepting it without reading the key. With some confidence in the host name of the connection this is reasonable to do in most scenarios.

In this case, your trust in the system should be how much you trust GCE metadata service to route you to the correct host, and push your ssh key to it.

  • 1
    Thanks but this isn’t a secure method—it’s vulnerable to a “man in the middle” attack. – jhfrontz Aug 6 '18 at 12:21
-1

"gcloud command" or "API" will only retrieve information related to the specific project or Google products, such an instance metadata from a GCE instance. It is not possible to retrieve instance's configuration such as host keys via the gcloud command or API.

Update

SSH host key pair, A private and public key are kept on the SSH server. The public key is shared with the SSH client on each connection. This is used to authenticate the SSH server to the client. After the first connection, the “known host” key is typically stored in ~/.ssh/known_hosts on the SSH client (your computer, for example), though gcloud stores it in a different place.

When you use gcloud compute ssh, your SSH client behaves slightly differently: the server’s public host key is automatically accepted for you the first time you connect, and it’s written to ~/.ssh/google_compute_known_hosts instead.

Every time you connect to a SSH server, it sends its public host key. In most circumstances, the server’s public host key won’t change. After you’ve connected once and saved the public key from your first connection, your SSH client can check to see if the server’s public host key is different. If different, you’ll see a message saying:

Host key verification failed.

When you see this message, one of the following is the case:

  • You’re connecting to a different server. This isn’t always malicious. For example, if you replace a server with a new one, you’ll see this message.
  • You’re connecting to the same server, but its SSH host keys have been re-generated. Again, this isn’t automatically an indicator of malicious behavior. For example, an administrator could have used ssh-keygen to create a new host key pair.

By default, gcloud compute ssh does not perform strict host key checking. There are reasons for this; however, you can enable strict host key checking by adding the --oStrictHostKeyChecking=yes option to gcloud compute ssh. Here’s an example:

gcloud compute ssh [INSTANCE_NAME] --zone [ZONE] -- -oStrictHostKeyChecking=yes

This is a good example of how you can add arbitrary SSH arguments to gcloud compute ssh.

Skipping host key checking isn’t a problem. Suppose you have created a new instance to which you’ve never connected. How would you know that you’re connecting to that instance and not somewhere else?

Consider the following example. Suppose you have no SSH user key pair for gcloud to use. (If you move both ~/.ssh/google_compute_engine and ~/.ssh/google_compute_engine.pub aside, you can test this.) Suppose you’re not logged into gcloud. You can log out with gcloud auth revoke.

Now try to SSH into an instance using gcloud compute ssh. You’ll have to authenticate to gcloud first. So, run gcloud auth login and sign in. Now your system has an access token stored in ~/.config/gcloud/. This token is used for authentication each time you make an API call using gcloud.

Now try to SSH into the same instance using gcloud compute ssh again. Without a SSH user key pair, it will call ssh-keygen on your behalf to create a new SSH user key pair. The private (user) key will be stored in ~/.ssh/google_compute_engine, and the public (user) key will be stored in ~/.ssh/google_compute_engine.pub. The private key remains local to your computer, and you’re responsible for securing it. The contents of the public key are sent via an API call to projects.setCommonInstanceMetadata or instances.setMetadata. By default, gcloud will add your public key to the project metadata (for all instances), unless the instance to which you’re connecting has been configured to block project wide SSH keys; in that case, it will add the public key to the instance’s metadata. Both API calls are authenticated with the token you received when you did gcloud auth login. The public key is encrypted in transit, because gcloud commands are executed via API calls over HTTPS.

At this point, you have a brand new SSH user private key. Unless you’ve copied it elsewhere, it’s local to your computer. The same is true for its corresponding public key, except that you have shared it with Google by sending it via an authenticated gcloud command over HTTPS. So the only systems that should have access to the public key are the one(s) you’re running in your GCP project.

Only the bearer of the SSH user private key can successfully authenticate to the SSH server(s) running in GCP. Only the SSH server(s) running in GCP have a copy of your SSH user public key. Thus, it’s safe to accept the host public key automatically. If you were connecting to a rogue server instead of your GCP SSH server, the authentication process would fail because that other server would not have your SSH user public key.

There are certain situations in GCP automatically re-creates a SSH server’s host key pair. For example, if you use an instance to create a template from which other instances in a group would be generated. In each case, there’s a need to update the SSH host key. For this reason, performing strict SSH host key checking can generate misleading error messages.

update Sep 5, 2018

It is possible to connect to an instance without "glcoud" and GCP team is aware of and working on it. A public bug is filed here. Any progress can be found there.

  • 1
    So there is no way to securely retrieve the host keys? – jhfrontz Aug 14 '18 at 18:50
  • I have updated the answer. I am not aware of any available methods to retrieve host key securely. However, I explained how GCP secured SSH connection even for a newly created instance. Shared where host key is kept and skipping host key checking isn’t a problem for the first attempt. Provided a detailed answer for larger group benefit. I believe this explanation will help you to understand a situation like you have created a new instance to which you’ve never connected and how would you know that you’re connecting to that instance and not somewhere else. – Rahi Aug 24 '18 at 22:23
  • 1
    Skipping host-key checking on the first attempt means there is no protection against MITM attack (i.e., you can not know that you're connected to that instance). – jhfrontz Aug 25 '18 at 1:20
  • @Jhfrontz, updated the answer. – Rahi Sep 6 '18 at 3:56
  • More information on why a client public key can't be used for authenticating a server: security.stackexchange.com/questions/189445/… – jhfrontz Sep 6 '18 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.