3

I'm currently building an infrastructure management tool that provisions bare metal and VM's etc. We have a worker VM that runs commands(via ansible) on the remote nodes over SSH.

One of the steps requires the rebooting of nodes to apply some configurations. The worker process has to run more commands on the nodes after the reboot is complete(must be done synchronously).

My question is, how can I check to see whether the reboot has completed?

I could add a sleep timer(to wait until the reboot completes), but I feel like that is a bad solution for a number of reasons.

Another option is to try to SSH to the remote node, from my worker process, every 5 seconds or so, if it fails, keep trying again until I get a successful connection.

Is there another way of doing this?

  • SSH attempts is how packer and vagrant handles waiting on instances to boot. – jordanm Mar 6 '17 at 14:56
  • A working SSH connection doesn't necessarily mean that the reboot cycle is complete, just that it has reached the point where the SSH daemon is started. – Sven Mar 6 '17 at 14:58
  • You could also get a positive SSH before the host goes down, if it is slow or wedged on something on shutdown. There are options for a host to run commands at startup, maybe use those? – thrig Mar 6 '17 at 15:23
  • I am currently working on a similar problem to automate the installation of vm's using ansible. After the minimal system installation, the post-install script creates systemd service startup script that will be run after sshd comes up (sufficient for my case). The startup script makes a simple http get on the host system with the hostname/ip as parameter, where a webapp is listening. The webapp will initiate the next steps. I am also thinking about using ssh to ping back from the guest. – rda Mar 6 '17 at 18:55
  • figure out what starts last, and make a hook after that. check for existence of that "hook", done. – Dennis Nolte Mar 9 '17 at 12:38
1

As you mentioned that you are running commands via ansible, here is what I use for reboots in a playbook (I'm managing Ubuntu 14/16.04 machines):

---
# execute like:
# ansible-playbook reboot.yaml --inventory hosts --extra-vars "hosts=all user=admin"
# or
# ansible-playbook reboot.yaml -i hosts -e "hosts=all user=admin"
- hosts: "{{ hosts }}"
  remote_user: "{{ user }}"
  become: yes
  tasks:
    # add this to to guard you from yourself ;)
    #- name: "ask for verification"
    #  pause:
    #    prompt: "Are you sure you want to restart all specified hosts?"

    # here comes the juicy part
    - name: "reboot hosts"
      shell: "sleep 2 && shutdown -r now 'Reboot triggered by Ansible'" # sleep 2 is needed, else this task might fail
      async: "1" # run asynchronously
      poll: "0" # don't ask for the status of the command, just fire and forget
      ignore_errors: yes # this command will get cut off by the reboot, so ignore errors
    - name: "wait for hosts to come up again"
      wait_for:
        host: "{{ inventory_hostname }}"
        port: "22" # wait for ssh as this is what is needed for ansible
        state: "started"
        delay: "120" # start checking after this amount of time
        timeout: "360" # give up after this amount of time
      delegate_to: "localhost" # check from the machine executing the playbook
...

Update

Ansible 2.7 now has a reboot module, so you don't need to create commands on your own. The playbook from above would translate into this:

---
# execute like:
# ansible-playbook reboot.yaml --inventory hosts --extra-vars "hosts=all user=admin"
# or
# ansible-playbook reboot.yaml -i hosts -e "hosts=all user=admin"
- hosts: "{{ hosts }}"
  remote_user: "{{ user }}"
  become: yes
  tasks:
    # add this to to guard you from yourself ;)
    #- name: "ask for verification"
    #  pause:
    #    prompt: "Are you sure you want to restart all specified hosts?"

    - name: "reboot hosts"
      reboot:
        msg: "Reboot triggered by Ansible"
        reboot_timeout: 360
...
  • Thanks, I didn't realize that Ansible had this. I will give this a try. – grizzthedj Mar 7 '17 at 13:27
  • this is actually not correct. as Sven already wrote, SSH connection doesn't mean a full reboot. – Dennis Nolte Mar 9 '17 at 12:39
-2

If you want to check status of hosts, time when they was rebooted and many many other parameters then you should use monitoring software like Zabbix, Nagios and etc.

Reboot time may be checked by uptime system parameter. It shows time since last boot. You can get it by command uptime on Linux/UNIX host or by SNMP protocol remotely when snmpd service is running on the host:

snmpget -v2c -c public host_name_or_ip_address sysUpTime.0
  • this doesn't answer the question(at all) in my opinion. You sure get the time when the kernel starts logging, the device is reachable again, ssh was started.. but not exactly when the boot process is finished. – Dennis Nolte Mar 9 '17 at 12:41
  • The server is up if you can get uptime (via SSH or SNMP). If uptime returns new value which is less then old then the system was rebooted. – Mikhail Khirgiy Mar 9 '17 at 13:21
  • yes but it is not a definition of a fully started server.. you can only define a reboot had happend (or power cycle for that reason) you cannot(!) define that the server is fully booted because you can reach ssh and uptime.. imagine a DB taking f.e. 60min to recover after a power cycle. you reach ssh after some seconds, perhaps 1min. so for 59 minutes you think the server is finished rebooting while it still performes tasks you think are already finished. – Dennis Nolte Mar 10 '17 at 23:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.