Update SLURM node state prior/after playbook execution

Question

I would like to automatically set the state of a node in a SLURM cluster before/after running my Ansible playbook (from idle to drained and after applying the playbook back to idle). The scontrol command that is required for this, is only available on the head node of the cluster. The Ansible playbook, however, is applied to the compute nodes of the cluster. Is there any way to run a remote command on another host than the one currently connected to? I could think of just using the built-in shell module and then SSH to the head node. But maybe there's a nicer way of doing it?

I already looked for ready-to-go Ansible modules but couldn't find any for my use case. The existing ones are all focused on installing/configuring a SLURM cluster.

My idea is then to use a do-until loop that sets the new cluster node state and then repeatedly checks whether the node already switched to the new state (as there still could be running jobs).

Answer 1

I finally managed it in the following way:

• Note that I'm running the playbook from my local client (not on the head node itself).

• I added a strategy: free command to allow execution on each node as fast as it can because it is less likely that all of the nodes are available at the same time to apply the configuration on.

• I added a pre_task that is executed before and drains a node:

  pre_tasks:
  - name:
    tags:
    - always
    delegate_to: 127.0.0.1
    shell: ssh -t user@slurm-head "srun -w {{ slurm_node_name}} sleep 3 && echo {{ ldap_passwd }} | sudo -S scontrol update NodeName={{ slurm_node_name }} State=Drain Reason=Maintenance"
    any_errors_fatal: true

• I added a block that bundles all my tasks and ends with a always section to ensure that even if some task fails, the node is moved back into the Idle status. My assumption here is that failing to apply a configuration doesn't break the node and it's better to make it available again than to block it until I find the time to look into the issue.

  tasks:
    - name: Configure node for Hardware Security course
      block:
      - name: Tasks 1
      ...
      - name: Task 2
      ...

      any_errors_fatal: true
      always:
      - name:
        tags:
        - always
        delegate_to: 127.0.0.1
        shell: ssh -t user@slurm-head "echo {{ ldap_password }} | sudo -S scontrol update NodeName={{ slurm_node_name }} State=RESUME"