31

I am going to introduce Ansible into my data center, and I'm looking for some security best practice on where to locate the control machine and how to manage the SSH keys.

Question 1: the control machine

We of course need a control machine. The control machine has public SSH keys saved on it. If an attacker has access to the control machine, it potentially has access to the whole data center (or to the servers managed by Ansible). So is it better to have a dedicated control machine in data center or a remote control machine (like my laptop remotely connected to the data center)?

If the best practice is to use my laptop (which could be stolen, of course, but I could have my public keys securely saved online in the cloud or offline on a portable crypted device), what if I need to use some web interfaces with Ansible, like Ansible Tower, Semaphore, Rundeck or Foreman which needs to be installed on a centralised machine into the datacenter? How to secure it and avoid it to become a "single point of attack"?

Question 2: the SSH keys

Assume that I need to use Ansible to make some tasks which require to be executed by root (like installing software packages or something like this). I think the best practice is not to use the root user on controlled servers, but to add a normal user for Ansible with sudo permissions. But, if Ansible needs to make almost every task, it needs to have access to every commands through sudo. So, what is the best choice:

  • let Ansible use the root user (with its public key saved in ~/.ssh/authorized_keys
  • create a unprivileged user dedicated for Ansible with sudo access
  • let the Ansible user to run every commands through sudo specifying a password (which is unique needs to be known by every sysadmin which uses Ansible to control that servers)
  • let the Ansible user to run every commands through sudo without specifying any password
  • any other hints?
  • you want a dedicated management subnet (or VLAN). your ansible control computer is on that subnet. If you need to communicate w the control computer, you VPN into that subnet. Don't let ansible be root – Neil McGuigan Jan 3 '17 at 22:27
  • 1
    The Ansible control host will be in the internal LAN and will not be accessible from outside, but I think this is not enough. – Mat Jan 3 '17 at 22:33
12

The bastion host (the ansible control center) belongs to a separate subnet. It shouldn't be directly accessible from outside, it shouldn't be directly accessible from the managed servers!

Your laptop is the least secure device of all. One stupid mail, one stupid flash vulnerability, one stupid guest Wifi and it gets pwned.

For servers, don't allow root access via ssh at all. Many audits scoff at this.

For ansible, let every admin use their own personal account on each target server, and let them sudo with passwords. This way no password is shared between two people. You can check who did what on each server. It's up to you if personal accounts allow login on password, ssh key only, or require both.

To clarify ansible doesn't require to use a single target login name. Each admin could and should have personal target login name.

A side note: Try to never create an account called some word (like "ansible" or "admin" or "cluster" or "management" or "operator") if it has a password. The only good name for account that has a password is a name of a human being, like "jkowalski". Only a human being can be responsible for the actions done via the account and responsible for improperly securing their password, "ansible" cannot.

  • Thanks, you convinced me about using a centralised bastion host in a separate subnet in data center. I am also aware that it's best to use one personal user per every sysadmin, but now I have another question (perhaps this would be better on a separate Serverfault question): how to centralise users and SSH keys on Linux hosts without use NIS, NFS shares or something like this? Please note that I already have Active Directory for Windows servers. – Mat Jan 4 '17 at 21:07
  • Ok, better thinking to my question I have two possible answers: use LDAP key storage or Userify (see two answers below). But... Do I really need it? No, if I use my own user to deploy new users and keys through Ansible itself! :-) – Mat Jan 4 '17 at 21:12
  • @Mat, agreed completely - as I said below, you really don't need Userify or any similar tool until your team gets larger. (It does make it a lot nicer though.) LDAP is kind of a pain to install (research pam_ldap and nss_ldap), but we have built in tools to integrate in seconds with Ansible, Chef, Puppet, etc. Also, it's free for <20 servers. – Jamieson Becker Jan 11 '17 at 16:00
15

> Question 1: the control machine

At Userify (full disclosure: we actually offer software to manage ssh keys), we deal with this all the time, since we also run the largest SSH key warehouse. We generally recommend local installation rather than using cloud, since you have increased control, reduce your surface area, you can really lock it down to just known trusted networks.

The important thing to remember is that, in a properly built system like this, there really shouldn't be any significant secrets that can be leaked to an attacker. If someone drives a forklift into your datacenter and walks away with your server, they won't get a whole lot except for some heavily hashed passwords, probably some heavily encrypted files, and some public keys without their corresponding private keys. In other words, not that much.

As you point out, the real threat vectors here are what happens if an attacker gains control of that machine and uses it to deploy their own user accounts and (public) keys. This is a risk for virtually every cloud platform (ex:Linode). You should be most strongly focused on preventing access to the control plane, which means minimizing attack surface (only exposing a few ports, and locking down those ports as much as possible) and preferably using software that is hardened against privilege escalation and various attacks (SQL injection, XSS, CSRF, etc.) Enable 2FA/MFA access to the control plane and focus on locking down that control plane as much as possible.

So is it better to have a dedicated control machine in data center or a remote control machine (like my laptop remotely connected to the data center)?

It's definitely better to have a dedicated control machine in a secure datacenter, because you can isolate it and lock it down to prevent/minimize risk of theft or unauthorized access.

If the best practice is to use my laptop (which could be stolen, of course, but I could have my public keys securely saved online in the cloud or offline on a portable crypted device), what if I need to use some web interfaces with Ansible, like Ansible Tower, Semaphore, Rundeck or Foreman which needs to be installed on a centralised machine into the datacenter?

You don't need to run ANY web interface or secondary control plane to manage your keys (even Userify) until you get large enough to start getting into management issues due to a larger number of users and different levels of authorization across servers or need extra hand-holding for your users who may not have knowledge or access to Ansible to update keys. Userify at first was not much more than a bunch of shell scripts (today they'd be Ansible, probably!) and there's nothing wrong with that at all, until you start needing additional management control and easy ways for people to manage/rotate their own keys. (Of course, please take a look at Userify if you get to that point!)

How to secure it and avoid it to become a "single point of attack"?

Well, of course check out all of the resources on the net for locking things down, but most importantly start with a secure foundation:

1. Architect your solution with security in mind from the very beginning. Choose technology (i.e., database, or languages) that have traditionally had fewer problems, and then code with security at front-of-mind. Sanitize all incoming data, even from trusted users. Paranoia is a virtue.

2. Eventually, everything gets broken. Minimize the damage when that occurs: as you pointed out already, try to minimize the handling of secret material.

3. Keep it simple. Don't do the latest exotic stuff unless you're certain it will measurably and provably increase your security. For example, we selected X25519/NaCl (libsodium) over AES for our encryption layer (we encrypt everything, at rest and in motion), because it was originally designed and written by someone we trusted (DJB et al) and was reviewed by world-renowned researchers like Schneier and Google's security team. Use things that tend toward simplicity if they are newer, since simplicity makes it harder to conceal deep bugs.

4. Meet security standards. Even if you don't fall into a security regime like PCI or the HIPAA Security Rule, read through those standards and figure out how to meet them or at least very strong compensating controls. This will help ensure that you are truly meeting 'best practices'.

5. Bring in outside/independent penetration testing and run bug bounties to make sure you are following those best practices on an on-going basis. Everything looks great until you get some smart and highly motivated people banging on it... once that's finished, you'll have a great deal of confidence in your solution.


Question 2: the SSH keys What is the best choice: let Ansible use the root user (with its public key saved in ~/.ssh/authorized_keys / let the Ansible user to run every commands through sudo specifying a password (which is unique needs to be known by every sysadmin which uses Ansible to control that servers)

Try to avoid ever using passwords on servers, even for sudo. That is dealing with secrets and ultimately will undermine your security (you can't really vary that sudo password between machines very easily, you have to store it somewhere, the password means you can't really do server-to-server automation which is exactly what it is all about. Also, if you leave SSH at its defaults, those passwords can be brute forced, which makes the keys somewhat meaningless. Also, avoid use of root user for any purpose, and especially remote login.

Create a unprivileged user dedicated for Ansible with sudo access / let the Ansible user to run every commands through sudo without specifying any password

Exactly. An unprivileged user that you can audit back to ansible, with sudo roles. Ideally, create a standard user dedicated to server-to-server/ansible communications with sudo access (without password).

... N.B., if you were using Userify, the way I would suggest doing it would be to create a Userify user for ansible (you can also break this up by project or even server group if you have multiple ansible control machines), generate an SSH key on the control server, and provide its public key in its Userify profile page. (This textbox essentially becomes /home/ansible/.ssh/authorized_keys). You should keep the ansible system account separate from other server-to-server system accounts such as a remote backup account, secret management, etc. Then invite your humans and they can create and manage their own keys as well and everything stays separated. But, just like with locking down an Ansible control server, try to lock down your Userify server (or whatever solution you deploy) in the same way.

any other hints?

I think you're definitely going about this the right way and asking the right questions. If you'd like to discuss this sort of thing, email me (first dot last name at userify) and I'd be glad to have a chat no matter what direction you ultimately pursue. Good luck!

4

Answer 1: the control machine

A little of both, you can use your laptop to connect to servers VIA a bastion host. something like:

Host private1
  IdentityFile ~/.ssh/rsa_private_key
  ProxyCommand ssh user@bastion -W %h:%p

Host bastion
  IdentityFile ~/.ssh/bastion_rsa_key

More on bastion Hosts

Where you have a key for the bastion server, and then a separate key for the host behind it. (personally I would use gpg-agent/ssh-agent)

Answer 2: Authentication

I'm not sure how "ansible" specific best practices differ from any other ssh connection best practices But no, you want to run ansible as yourself, not a service account and not a root account.

A Combination of the following authentications:

Other Thoughts:

  • Always Store secrets/private information in ansible-vault.
  • Ansible does not require SUDO/Root to run, unless what you are doing requires sudo/root.
  • Ansible can elevate permissions many different ways

Lastly, you mentioned nothing about windows. So I can only assume you are not using this. However in this case I would use the delegate option to have your laptop use the bastion host (delegate_to: bastion.hostname.fqdn) and kerberos/winrm https with kerberos tickets.

In case you missed it, best practice for computing, never do anything as root, always use named accounts

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.