Rolling update with puppet, ansible or fabric

Question

I have some jetty servers with a loadbalancer before them. Now I want to update my application without downtime. When one jetty is going down and no longer reachable the loadbalancer automatically removes it from the list, so this is not the issue.

The main problem is to avoid down time: so, I have to make sure only one jetty is restarting at a time - or make sure that at least N jettys are online!

At the moment I'm using a simple bash script where I need to wait manually for one jetty to came back online again before restarting the next jetty and so on.

Now bash is not very optimal for this kind of stuff and I hope there are tools better suited to make the whole task automated. E.g. I could use a simple ping URL http://jetty-number-n.com/ping which responds OK (200) if the n-th jetty is online.

How could I solve this task and with which tool?

Thanks to @ceejayoz I found rolling update for ansible. But it is still suboptimal as I would need to set a fixed timeout.

Answer 1

This is fairly easy with Ansible. Rough pseudo-ansible playbook:

---
  - hosts: your_server_group
    sudo: yes
    serial: 1
    tasks:
      - shell: reboot now
      - wait_for: port=22

Answer 2

Salt has a convenient batch option.

salt -G 'os:RedHat' --batch-size 25% service.restart jetty

Restart the jetty service on 25% of RedHat servers at a time.

salt -N group1 -b 2 system.restart

Restart servers in the pre-defined "group1", 2 at a time.

Answer 3

Puppet has the concepts of Services and Ordering. But puppet runs on on a single server. So it doesn't have insight into other systems that it is running on.

If you wanted to use puppet, you could have it so one master control server has a init script for managing the services of each server. So the init script would run the restart on each server and return back the status code. You could do this and chain each service restart from this one server and an ordering, chaining or notify and subscribe to move on to the next one.

You could also write a custom library for puppet, that will treat one server as the master control server, but instead of using the init scripts. It can use a custom ruby library to manage the services of each server. It's a bit more complex to set up, but could work.

Answer 4

I use Puppet at work right now, but as the first poster replied, this doesn't sound like an ideal first task for Puppet. I can't speak to ansible (though I want to start trying it out).

Puppet is geared more toward rolling out changes/restarting systems simultaneously--a Puppet daemon runs on each client, and checks against a central server at a scheduled interval (default every half hour). If you want to buck the default scheduling, you usually use a different mechanism to call puppet.

If you're ok with a simultaneous restart, then restarting a service is a matter of defining a manifest with

file { 'your_war_file.war':
    ensure  => file,
    source  => "puppet:///...",
}
service { 'jetty':
    subscribe  => File['your_war_file.war'],
    hasrestart => True,
}

resource directives, then pushing out the manifest and letting Puppet do its thing. This assumes that you're ok with calling

/etc/init.d/jetty restart

which may or may not be acceptable or even possible.

You also could define your own custom resource types for this, which is definitely beyond the scope of a simple first-time puppet task.

Check out http://forge.puppetlabs.com/, however, to see if anyone else has solved a similar problem.

Answer 5

The task you want to do can be done using ansible, and it definitely could be done using Puppet. I'm going to focus on how to accomplish it using puppet.

You can deploy your WAR files to jetty, using puppet, by using a normal "file" resource and a defined resource that you initialize for each application.

Puppet will downloaded from the URL you specify (on your Puppet Master). Jetty is able to redeploy WAR files without restarting, if the WAR file modification time changes. So if your manifest creates a copy in $JETTY_HOME/webapps it should be automatically picked up by Jetty. This way (or by touching the context.xml) you don't need to manually restart Jetty.

In order to download the WAR file from your Puppet Master to your Application Directory you will need the following manifest (replacing $JETTY_HOME):

define jetty::deployment($path) {

  notice("Deploying ${name} to http://$hostname:${appserver-jetty::port}/"),
  include jetty,

  file { "$JETTY_HOME/webapps/${name}.war":
    owner => 'root',
    source => $path,
  }
}

and you need to instruct your nodes to get a specific version of your application one after another (after you're sure that it has successfully started on one of the nodes). This can be done using an ENC (external node classifier) or, if you are not using ENC, by modifying your site.pp section for each node (or group of nodes):

node jetty-node1 {
   jetty::deployment { "servlet":
     path => '/srv/application/Servlet-1.2.3.war'
   }
},
node jetty-node2 {
   jetty::deployment { "servlet":
     path => '/srv/application/Servlet-2.0.1.war'
   }
}

The include jetty line is necessary only if you want to manage your Jetty configuration through puppet using eg. https://github.com/maestrodev/puppet-jetty and this defined resource needs an appserver-jetty::$port variable. This example shows the simplest way you could control exactly which version of the Application is served by which node. This way you can script the deployment, based on your health-checks and deploy on node2 only after node1 is successfully running the new version.

This example is only to demonstrate the concept.

You might want to check these additional resources for more information and ideas: On reloading the contexts: https://stackoverflow.com/questions/13965643/auto-reloading-war-in-jetty-standalone

This is for deployment and management of tomcat, but the ideas (and the manifests) are similar: http://www.tomcatexpert.com/blog/2010/04/29/deploying-tomcat-applications-puppet

This is for a, IMO, better alternative to directly copying the WAR files - using native packages (eg. RPM) for deploying your application: http://www.slideshare.net/actionjackx/automated-java-deployments-with-rpm