0

Putting the EC2 instance back into the ELB used to work 90% of the time. Unfortunately lately the deployment regulary fails with the next error:

15:51:59 TASK: [Start the app] ********************************************************* 
15:51:59 changed: [app-01a] => {"changed": true, "enabled": true, "name": "app", "state": "started"}
15:51:59 
15:51:59 TASK: [Wait for the app to be ready] ****************************************** 
15:52:17 ok: [app-01a] => {"changed": false, "elapsed": 17, "path": null, "port": 8080, "search_regex": null, "state": "started"}
15:52:17 
15:52:17 TASK: [Check health check on localhost] *************************************** 
15:52:22 ok: [app-01a] => {"cache_control": "must-revalidate,no-cache,no-..."status": 200,...
15:52:22 
15:52:22 TASK: [Exit if health check fails] ******************************************** 
15:52:22 skipping: [app-01a]
15:52:22 
15:52:22 TASK: [Register restapp instance back into load balancer] ********************* 
15:52:39 failed: [app-01a -> 127.0.0.1] => (item=app-ELB) => {"failed": true, "item": "app-ELB"}
15:52:39 msg: The instance i-b1234567 could not be put in service on LoadBalancer:app-ELB. Reason: Instance has not passed the configured HealthyThreshold number of health checks consecutively.
15:52:39 
15:52:39 FATAL: all hosts have already failed -- aborting

This is the Ansible code:

- name: Start the app
  service: name={{ app_name }} state=started enabled=yes

- name: Wait for the app to be ready
  wait_for: port={{ app_port }} state=started timeout=120

- name: Check health check on localhost
  action: uri url=http://localhost:8081/healthcheck
  register: webpage

- name: Exit if health check fails
  command: /bin/false
  when: webpage.status != 200

- name: Register restapp instance back into load balancer
  sudo: false
  local_action:
    module: ec2_elb
    instance_id: "{{ appInstanceId }}"
    ec2_elbs: "{{ item }}"
    state: 'present'
    region: "eu-west-1"
  with_items: appLoadBalancer

ELB Settings:

http://i.imgur.com/RIM8L5q.png

1

You probably need to change the ping target from HTTP:8081/pin(clipped) to HTTP:8081/healthcheck like it is in your Ansible play.

0

This is expected AWS behaviour. If you fix your app, the ELB will re-enable traffic to your endpoint. Trying to force an offline host into service goes against what the health checks are trying to accomplish.

You can't do this in the ELB console, and even removing and re-adding the host would require you to wait for the health check to pass before it was passed traffic.

If you want the host to return to service quicker, change your health-check tolerances.

1
  • The application is back up however when re-registering it back into the load balancer. It's even waiting until the application comes online and doing a health check. – ujjain Jun 15 '15 at 9:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.