7

See also: https://stackoverflow.com/questions/29003420/reload-ansibles-dynamic-inventory.

My question: is there a better way of doing what's below?

I have an ansible role that provisions AWS machines, and runs correctly (notice the provision tag):

- name: AWS provision
  hosts: localhost
  gather_facts: no
  vars_files:
    - vars/dev.yml
  user: ec2-user
  roles:
    - provision
  tags:
    - provision

I then have a base role, that I want to be able to run independently (for example during development, so I don't have to wait for re-provisioning (notice the base tag). I run a play find running instances that filters and stores the hosts in the group started:

- name: find running instances
  hosts: localhost
  vars_files:
    - vars/dev.yml
  gather_facts: no
  tags:
    - base
  tasks:
    - name: gather remote facts
      ec2_remote_facts:
        region: "{{ target_aws_region }}"
        filters:
          instance-state-name: running
          "tag:Name": "{{ instance_name }}"
      register: ec2_facts

    - debug: var=ec2_facts

    - name: add hosts to groups
      add_host:
        name: "{{ item.id }}"
        ansible_ssh_host: "{{ item.public_dns_name }}"
        groups: started
      changed_when: false
      with_items: ec2_facts.instances

- name: base setup
  hosts: started
  gather_facts: no
  vars_files:
    - vars/dev.yml
  user: ec2-user
  roles:
    - base
  tags:
    - base

My question: the plays are working, but is there a better way of doing this? For example I'm got gather_facts: no followed by ec2_remote_facts and the filters - it all seems rather convoluted.

A clarification: thanks for the comment about ec2.py -- I'm already using it in my first play (when I call the provision role).

But for testing purposes I want to jump into subsequent plays without re-doing the (slow) provisioning. So how do I re-populate my hosts data? Is ec2_remote_facts followed by add_host the right way? Or can I somehow use gather_facts: yes?

3 Answers 3

6

I'd probably use the EC2 dynamic inventory script instead, which you can employ by configuring ec2.ini and passing -i ec2.py to ansible-playbook.

See http://docs.ansible.com/ansible/intro_dynamic_inventory.html#example-aws-ec2-external-inventory-script for more info.

Note that there are plenty of options in ec2.ini. Be sure to have a look at those, e.g. cache_max_age. You can also make the inventory generation faster by filtering unnecessary resources (e.g. set rds = False if you are only interested in EC2 instances).

UPDATE: With Ansible 2.x+ you can also use - meta: refresh_inventory mid-play.

5
  • Why do you run a play to find the instances "programmatically" when you can just use the ec2 inventory to find the instances e.g. by tags? Note the cache property I mentioned. You can set it to zero to have a fresh inventory for each run (if the first provision run leaves you with an empty inventory since the instances have not been provisioned yet at the start of the run).
    – Jukka
    Dec 20, 2015 at 18:24
  • That's what I'm trying to understand :-) How do I get ec2.py to populate the inventory when I jump in halfway through a play, without setting cache to zero? Use "gather_facts: true" and just filter on the tags? Dec 20, 2015 at 23:33
  • 1
    Either run the second play separately and empty cache in-between or use Ansible 2.x's - meta: refresh_inventory. If you choose to empty the ec2.py cache, you might want to tweak its settings to filter as much as possible (i.e. everything but the ec2 instances you are interested in) to make it faster.
    – Jukka
    Dec 22, 2015 at 7:34
  • 1
    Fantastic, thanks @jukka. I didn't know about meta: refresh_inventory. Dec 23, 2015 at 1:39
  • 1
    Glad to help, I updated my answer for future readers as well.
    – Jukka
    Dec 23, 2015 at 6:57
2

While the meta: refresh_inventory is the "preferred method", I tend to like OP's proposal of using ec2_remote_facts in conjunction with add_host. I've setup such a playbook and it has the strength to be 100% dynamic without caching glitches.

Assuming your ASG fired up instances with the env: cool_asg_instance tag, just add the following under the ec2_asg playbook call:

- ec2_remote_facts:
    filters:
      "tag:env": "cool_asg_instance"
  register: instance_facts

You'll then gather a full JSON dataset containing all the needed informations, from there you can use Jinja2 capabilities within the playbook to extract newly created IP addresses, i.e.:

- name: group hosts
  add_host: hostname={{ item }} groups=launched
  with_items: "{{ instance_facts.instances|selectattr('state', 'equalto', 'running')|map(attribute='private_ip_address')|list }}"

Filter is courtesy of this wonerfull blog post: https://bonovoxly.github.io/2016-02-11-ansible-stuffs-ec2_remote_facts_instead_of_ec2_py

From now on you can use the launched group on your parent deployment YAML file like this:

- hosts: launched
  gather_facts: no

  tasks: 
    - name: wait for SSH
      wait_for: port=22 host="{{ inventory_hostname }}" search_regex=OpenSSH delay=5

Some may ask why the headache, well imagine that instead of having an hideous userdata which will git clone both Ansible and a playbook from the Internet, you can trigger the instance setup from your own deployment center by setting up a simple SNS topic which will publish to a SQS queue, watched by a 10 lines python code (https://github.com/alexandregama/python-sqs-consumer/blob/master/sqs-message-consumer-polling.py) which will trigger Ansible when a new instance comes out.

2
  • Just confirming - When you say "ASG", you're referring to an AWS Auto-Scaling Group, correct?
    – blong
    Mar 28, 2017 at 13:39
  • 1
    Yes, ASG is a shorthand for Auto Scaling Group
    – iMil
    Apr 8, 2017 at 18:36
0

I found this was easier than I expected. The following code allows you to run playbooks against existing AWS inventory, which is what I originally want to do. Based on [1] and [2].

(my solution is a bit more flexible eg custom ansible.cfg & ssh_config)

Setup ansible config file and use it:

% cat ./ansible.cfg.foo
[defaults]
hostfile = inventory
host_key_checking = false
private_key_file = bar.pem
remote_user = ec2-user
ssh_args = -F ./ssh_config

export ANSIBLE_CONFIG=ansible.cfg.foo

Optionally setup custom ssh config, so your laptop's settings don't interfere. ssh config could also be done in ansible.cfg.foo, but I like having it separate so an existing ~/.ssh/config can be used by third parties.

% cat ./ssh_config
StrictHostKeyChecking no
...

Run your playbook:

ansible-playbook config.yml --tags base

Where playbook might look like the following. aws tags (eg tag_env_test) are used to select hosts.

% cat config.yml
- name: base setup
  hosts:
    - tag_env_test
  become: true
  vars_files:
    - vars/qux.yml
  roles:
    - base
  tags:
    - base

- name:  java setup
  hosts:
  ...

I use ./ec2.py --list > ../ec2_output.txt to investigate hosts I'm interested in selecting.

I've marked Jukka's answer as the solution, to thank him for helping :-)

[1] http://www.slideshare.net/bfschott/using-ansible-dynamic-inventory-with-amazon-ec2

[2] https://aws.amazon.com/blogs/apn/getting-started-with-ansible-and-dynamic-amazon-ec2-inventory-management/

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .