1

I have a situation where a RHEL 6.4 server is sending TERM and KILL signals too quickly, before applications and databases have had a chance to gracefully stop. It seems that Upstart is handing control to the sysv-rc scripts prematurely.

To troubleshoot this, I've tried adding sleep and logger commands into the Upstart config. The pre-script stanza is writing to syslog, but the sleep never completes because the system reboots within 10 seconds. I've also added a kill timeout which is ignored.

# cleanup at system shutdown
# Does stop all on apps, stops monit instances, then does a clean.
start on runlevel [016]

console output
kill timeout 120

task

pre-start script
    logger -s -t "arcsight-services-stopall" "Running pre-start..."
    /etc/init.d/arcsight_services stop
    sleep 60
end script

script
    logger -s -t "arcsight-services-stopall" "Running script..."
    /etc/init.d/arcsight_services shutdown monit
    /etc/init.d/arcsight_services clean all
end script

I know Upstart was suppose to parallelize the boot/halt process, but it has only paralyzed my debugging attempts.

Under RHEL 6, what is the definitive order of scripts that get executed after issuing: shutdown -r now?

  1. shutdown -r now
  2. ?????
  3. /etc/init/rc.conf (Upstart)
  4. /etc/rc.d/rc (sysv-rc)
  5. ?????
  6. /etc/rc3.d/K* (sysv-rc)
  7. /etc/rc6.d/S* (sysv-rc)
  8. ?????

Where do the other /etc/init/*.conf scripts get called?

UPDATE: In dissecting /etc/rc.d/rc, I found that if I touch /var/run/confirm, the process enters an interactive mode and then my sleep & logger commands seem to execute. This baffles me, because I thought, at that point, Upstart has passed control to sysv-rc.

  • Don't write upstart scripts on RHEL 6. They are not supported (even though it actually uses upstart). Use a SysV style init script instead. – Michael Hampton Dec 23 '13 at 21:29
  • HP ArcSight Developers, are you reading this? "Don't write upstart scripts on RHEL 6." Thanks @MichaelHampton. – skohrs Dec 26 '13 at 13:38
  • 1
    It's really just an unfortunate artifact of timing that RHEL ended up with Upstart at all. It was tried in Fedora and quickly abandoned since it was crap, but RHEL 6 went into feature freeze at just that time. Virtually everyone else has abandoned Upstart, too, and RHEL 7 uses systemd. That will last a little longer... – Michael Hampton Dec 26 '13 at 16:27
1

While this answer doesn't address the order of scripts that get executed during a RHEL 6 shutdown, it does solve the issue of the system killing processes before they are gracefully stopped.

/etc/init/arcsight-services-stopall.conf:

# cleanup at system shutdown
# Does stop all on apps, stops monit instances, then does a clean.

start on starting rc RUNLEVEL=[016]

task
kill timeout 330

pre-start script
    logger -s -t "ArcSight" "ArcSight ESM shutdown initiated..."
    /etc/init.d/arcsight_services shutdown all
    /etc/init.d/arcsight_services shutdown monit
    /etc/init.d/arcsight_services clean all
    sleep 300
end script

script
    logger -s -t "ArcSight" "ArcSight ESM shutdown complete."
end script

The key was modifying start on starting rc RUNLEVEL=[016]. By starting this script while /etc/init/rc.conf is just starting, it blocks for 5 minutes, before the sysv-rc scripts are executed. Hopefully, during that 5 mintues, all the ArcSight databases and applications are gracefully stopped. Testing indicated that everything was stopped within 3 minutes, so 5 minutes should be a safe delay.

It's always nice to see a multi-millon dollar product require a customer hack.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.