cherrypy fails to stop when puppet tries to ensure running and refresh it at the same time

Question

I am trying to manage a cherrypy service with puppet. However, when the configuration is applied, cherryd ends up with no PID file although the process is up and running. Since the PID file is lost I can no longer stop the process with /etc/init.d/mycherryd stop (unless I modify the handmade init script to lookup the PID with ps or something.)

$ /etc/init.d/mycherryd status
cherryd dead but subsys locked

The problem seems to be that puppet is trying to refresh/restart cherryd (triggered by changes in configuration files) immediately after ensuring it's running (as specified in the manifest), and cherrypy fails to stop and start (restart) itself while still executing its startup process.

Is there a clear cut solution to this problem? Is this a bug on the cherrypy side, or can I write a puppet manifest so refresh is called only after the service is up and running? Any suggestion welcome.

cherrypy log

See how cherrypy catches SIGTERM midway through startup and still starts to listen.

:cherrypy.error[18666] 2010-02-12 13:10:23,551 INFO: ENGINE Listening for SIGHUP.
:cherrypy.error[18666] 2010-02-12 13:10:23,552 INFO: ENGINE Listening for SIGTERM.
:cherrypy.error[18666] 2010-02-12 13:10:23,552 INFO: ENGINE Listening for SIGUSR1.
:cherrypy.error[18666] 2010-02-12 13:10:23,552 INFO: ENGINE Bus STARTING
:cherrypy.error[18671] 2010-02-12 13:10:23,554 INFO: ENGINE Daemonized to PID: 18671
:cherrypy.error[18671] 2010-02-12 13:10:23,554 INFO: ENGINE PID 18671 written to '/var/mycherryd/cherry.pid'.
:cherrypy.error[18671] 2010-02-12 13:10:23,555 INFO: ENGINE Started monitor thread '_TimeoutMonitor'.
:cherrypy.error[18670] 2010-02-12 13:10:23,556 INFO: ENGINE Forking twice.
:cherrypy.error[18666] 2010-02-12 13:10:23,557 INFO: ENGINE Forking once.

:cherrypy.error[18671] 2010-02-12 13:10:23,716 INFO: ENGINE Caught signal SIGTERM.
:cherrypy.error[18671] 2010-02-12 13:10:23,716 INFO: ENGINE Bus STOPPING
:cherrypy.error[18671] 2010-02-12 13:10:23,716 INFO: ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('0.0.0.0', 12380)) already shut down
:cherrypy.error[18671] 2010-02-12 13:10:23,717 INFO: ENGINE Stopped thread '_TimeoutMonitor'.
:cherrypy.error[18671] 2010-02-12 13:10:23,717 INFO: ENGINE Bus STOPPED
:cherrypy.error[18671] 2010-02-12 13:10:23,732 INFO: ENGINE Bus EXITING
:cherrypy.error[18671] 2010-02-12 13:10:23,759 INFO: ENGINE PID file removed: '/var/mycherryd/cherry.pid'.
:cherrypy.error[18671] 2010-02-12 13:10:23,782 INFO: ENGINE Bus EXITED

:cherrypy.error[18671] 2010-02-12 13:10:23,792 INFO: ENGINE Serving on 0.0.0.0:12380
:cherrypy.error[18671] 2010-02-12 13:10:23,826 INFO: ENGINE Bus STARTED

puppet log

puppet tries to refresh the service immediately after ensuring it to be 'running'.

Feb 12 13:10:22 localhost puppetd[8159]: (//mycherrypy/File[conffiles]) Scheduling refresh of Service[cherryd]
Feb 12 13:10:22 localhost last message repeated 12 times
Feb 12 13:10:23 localhost puppetd[8159]: (//mycherrypy/Service[mycherryd]/ensure) ensure changed 'stopped' to 'running'
Feb 12 13:10:23 localhost puppetd[8159]: (//mycherrypy/Service[mycherryd]) Triggering 'refresh' from 13 dependencies
Feb 12 13:11:23 localhost puppetd[8159]: (//mycherrypy/Service[mycherryd]) Failed to call refresh on Service[mycherryd]: Could not stop Service[mycherryd]: Execution of '/sbin/service mycherryd stop' returned 1:  at /etc/puppet/manifests/mycherrypy.pp:161
Feb 12 13:11:24 localhost puppetd[8159]: Value of 'preferred_serialization_format' (pson) is invalid for report, using default (marshal)
Feb 12 13:11:24 localhost puppetd[8159]: Finished catalog run in 99.25 seconds

puppet manifest excerpt

class mycherrypy {
  file {
    'rpm':
      path   => "/tmp/${apiserver}.i386.rpm",
      source => "${fileserver}/${apiserver}.i386.rpm";
    'conffiles':
      require => Package["${apiserver}"],
      path    => "${service_home}/config/",
      ensure  => present,
      source  => "${fileserver}/config/",
      notify  => Service["mycherryd"];
  }

  package {
    "$apiserver":
      provider => 'rpm',
      source   => "/tmp/${apiserver}.i386.rpm",
      ensure   => latest;
  }

  service {
    "mycherryd":
      require => [File["conffiles"], Package["${apiserver}"]],
      ensure    => running,
      provider  => redhat,
      hasstatus => true;
  }
}

Answer 1

Try adding the attribute hasrestart => true to your mycherryd service resource; this will tell puppet to run 'service mycherryd restart' instead of 'service mycherryd stop+service mycherryd start`.

Also make sure the status codes of the init script itself are sane; its often the case that init script authors do not follow the LSB spec for exit codes -- i.e. run service mycherryd stop; echo $? and it should say 0 for a successful exit. If not, puppet's going to have a hard time managing the service and the mycherryd maintainers should fix their script.