How to wait for user_data script to run when starting EC2 instances with ansible?

Question

I am using Ansible to start and configure EC2 instances that are currently based in an AMI (ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20161020 (ami-40d28157)) that doesn't have Python installed from start.

Since most ansible modules need python on the remote machine to work, rather than choosing to use a different AMI I tried to install python on the starting instances using an user_data script.

It kinda worked (Python is installed correctly), but the ansible ec2 task returns before the user_data script finishes running, and so the tasks on the following play (those that would actually configure the instance) fail with the error '/bin/sh: 1: /usr/bin/python: not found\r\n'

Since without python I can't even run most tasks on the remote host I can't simply check if /usr/bin/python or a marker file is there, or check the syslog output, for example.

I thought about adding a nc -l $SOME_PORT at the end of the user_data script and checking for it with the wait_for module, but it sounds like too much of a hack for me.

Is there a way to use the ec2 module to start an instance so that it waits until the user_data script finishes running?

For reference:

A playbook similar to what I'm using:

- name: Create the EC2 instance
  hosts: localhost
  gather_facts: false
  vars: 
    name: myinstance
    ami: ami-40d28157
    #Other vars ommited for brevity
  tasks:
    - name: Launch Instance
      ec2:
        key_name: "{{ keypair }}"
        group: "{{ security_group }}"
        instance_type: "{{ instance_type }}"
        image: "{{ ami }}"
        user_data: "{{ lookup('file', 'user_data.sh') }}"
        wait: true
        region: "{{ region }}"
        vpc_subnet_id: "{{ subnet }}"
        assign_public_ip: no
        zone: "{{ zone }}"
        instance_tags: "{{ tag_map }}"
        exact_count: 1
        count_tag:
          Name: "{{ name }}"
      register: ec2

    - name: Add new Instance to host group
      add_host: name={{ item.private_ip }} groups={{ name }}
      with_items: "{{ ec2.instances }}"

    - name: Wait for SSH to come up
      wait_for: host={{ item.private_ip }} port=22 delay=60 timeout=320 state=started
      with_items: "{{ ec2.instances }}"

- name: Configure instance(s)
  hosts: myinstance
  tasks:
    - name: Example task
      command: touch a_file

The user_data.sh script:

#!/bin/bash
set -e -x
export DEBIAN_FRONTEND=noninteractive
sudo locale-gen pt_BR.UTF-8
sudo apt-get update && apt-get upgrade -y
sudo apt-get install -y python-minimal

Answer 1

Here is the answer to your question "How to wait for user data script to run when starting EC2 instances with ansible?":

- raw: test -f /var/lib/cloud/instance/boot-finished
  retries: 20
  register: cmd_res
  changed_when: false
  until: cmd_res | success

This will connect via ssh and succeed when boot-finished is present, which is created after cloud-init module on the host completes all modules (including user_data script).

But I'd suggest to create AMI with Python preinstalled, or use raw or script Ansible module to check/install Python (these modules don't require Python).

Answer 2

It turns out you can use cloud-init command for that, namely:

cloud-init status --wait

So you should add the following snipped to your playbook:

- name: Wait for cloud-init / user-data to finish
  command: cloud-init status --wait
  changed_when: false

It looks a bit cleaner compared to the first answer.