0

I have an ansible playbook as shown below and it works fine most of the times. But recently what I am noticing is it is getting stuck on some of the servers from ALL group and just sits there. It doesn't even move forward to other servers in the ALL list.

# This will copy files
---
- hosts: ALL
  serial: "{{ num_serial }}"
  tasks:
      - name: copy files
        shell: "(ssh -o StrictHostKeyChecking=no abc.com 'ls -1 /var/lib/jenkins/workspace/copy/stuff/*' | parallel -j20 'scp -o StrictHostKeyChecking=no abc.com:{} /data/records/')"

      - name: sleep for 5 sec
        pause: seconds=5

So when I started debugging, I noticed on the actual server it is getting stuck - I can ssh (login) fine but when I run ps command then it just hangs and I don't get my cursor back so that means ansible is also getting stuck executing above scp command on that server.

So my question is even if I have some server in that state, why not just Ansible times out and move to other server? IS there anything we can do here so that ansible doesnt pause everything just waiting for that server to respond.

Note server is up and running and I can ssh fine but when we run ps command it just hangs and because of that ansible is also hanging.

Is there any way to run this command ps aux | grep app on all the servers in ALL group and make a list of all the servers which executed this command fine (and if gets hang on some server then time out and move to other server in ALL list) and then pass on that list to work with my above ansible playbook? Can we do all this in one playbook?

Update:-

I am getting an error like this:

ERROR! The 'pause' module bypasses the host loop, which is currently not supported in the free strategy and would instead execute for every host in the inventory list.

The error appears to have been in '/var/lib/jenkins/workspace/process/check.yml': line 10, column 9, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


      - name: sleep for 5 sec
        ^ here
Build step 'Execute shell' marked build as failure
1

(Ansible is getting stuck executing scp command on that server). If I have some server in that state, why not just Ansible times out and move to other servers?

Very probably the plays run with linear strategy

By default, plays run with a linear strategy, in which all hosts will run each task before any host starts the next task

Use free strategy

which allows each host to run until the end of the play as fast as it can

- hosts: all
  serial: "{{ num_serial }}"
  strategy: free
  tasks:

To try async the playbook below with 3 hosts in the group 'test' asynchronously let the hosts sleep random number (1-10) of seconds and wait 5 seconds for the hosts to finish. Then retries once to collect the async_status and exit the play. Next play at localhost will run when at least one host succeeds and will print the status of each host.

- hosts: test
  strategy: free
  vars:
    max_sleep_time: 10
    max_wait_time: 5
  tasks:
    - set_fact:
        my_time: "{{ max_sleep_time|random(start=1) }}"
    - debug:
        msg: "Sleep {{ my_time }} seconds"
    - command: "sleep {{ my_time }}"
      register: play_status
      async: "{{ max_wait_time }}"
      poll: 0
    - async_status:
        jid: "{{ play_status.ansible_job_id }}"
      register: play_status
      until: play_status.finished
      retries: 1

- hosts: localhost
  tasks:
    - debug:
        msg: "{{ item }} finished: {{ hostvars[item].play_status.finished }}"
      loop: "{{ groups['test'] }}"

The output (abridged) shows hosts test_01 and test_03 slept 9 seconds, did not finish in time (max_wait_time: 5) and failed.

TASK [debug]
ok: [test_01] => {
    "msg": "Sleep 9 seconds"
}
ok: [test_02] => {
    "msg": "Sleep 1 seconds"
}
ok: [test_03] => {
    "msg": "Sleep 9 seconds"
}
TASK [async_status]
changed: [test_02]
fatal: [test_01]: FAILED! => {"ansible_job_id": "10701665445.1564", "attempts": 1, "changed": false, "finished": 0, "started": 1}
fatal: [test_03]: FAILED! => {"ansible_job_id": "752000555573.1558", "attempts": 1, "changed": false, "finished": 0, "started": 1}
...
TASK [debug]
ok: [localhost] => (item=test_01) => {
    "msg": "test_01 finished: 0"
}
ok: [localhost] => (item=test_02) => {
    "msg": "test_02 finished: 1"
}
ok: [localhost] => (item=test_03) => {
    "msg": "test_03 finished: 0"
}
...
PLAY RECAP
localhost                  : ok=2    changed=0    unreachable=0    failed=0   
test_01                    : ok=3    changed=1    unreachable=0    failed=1   
test_02                    : ok=4    changed=2    unreachable=0    failed=0   
test_03                    : ok=3    changed=1    unreachable=0    failed=1 

Set max_wait_time > max_sleep_time to see all hosts finished.

  • I am getting this error when I run with free strategy - ERROR! The 'pause' module bypasses the host loop, which is currently not supported in the free strategy and would instead execute for every host in the inventory list.. I updated my question with full error message as well. So in short I want to run that particular task on x number of machines in parallel and then sleep for some time and then do the same thing. I am not sure whether this is possible. – user5447339 Jun 24 at 18:24
  • The error is weird. It would take to review the source to understand it, I think. It might be worth to try async instead, or synchronise outside the playbook. See ansible-runner. – Vladimir Botka Jun 24 at 19:02
  • Do you think you can provide an example on how to use these with my example? That would help me understand better. – user5447339 Jun 24 at 19:19
  • Sure. I've updated the answer. Feel free to experiment. – Vladimir Botka Jun 24 at 22:25
1

You could gather facts.

---
hosts: all
gather_facts: True
tasks:

By gathering facts explicitly, you force Ansible to try to connect to every host (and update your fact cache). If a host is unreachable, it will be skipped for the rest of the playbook. By default the timeout for gathering facts is 10 seconds, so this should reduce the amount of time you have to wait.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.