1

I have a server running nginx for static content, with apache behind it, running php and python code.

About once a day Apache just quits, leaving Nginx serving a 502 Bad Gateway error. All I can find in the logs is Apache received SIGTERM, no other details.

[mpm_prefork:notice] [pid 11811] AH00169: caught SIGTERM, shutting down

It has been running properly for some time now, with no maintenance other than running out of disk space. I did some SSL proxy configuration a few months ago. Running Debian Jessie

3

Use auditd to log all of the kill syscalls. -S kill in an audit.rules file. See How to use audit to monitor a specific SYSCALL?. The log entries will contain the pid, uid, and exe of every kill, find the one right before httpd shuts down.

I will guess this is a log rotate script that isn't restarting correctly.

  • it's at a random time every day, or multiple times per day, and started happening with no changes to logging as far as I know. – JAL Feb 19 '18 at 12:48
  • thanks, I installed auditd and I'm pretty sure that will give me the answer. – JAL Feb 19 '18 at 13:09
  • Yep, it was something running not on cron, but systemd. – JAL Feb 21 '18 at 19:37
  • type=SERVICE_START msg=audit(1519111948.453:2257): pid=1 uid=0 auid=4233968205 ses=4233968205 msg=' comm="certbot" exe="/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' is present around when the server stops. From the certbot log itheir python script is actually crashing and exiting abnormally, so I should file a bug report with them. – JAL Feb 21 '18 at 19:42
1

caught SIGTERM

That means a systems is sending a terminate command to Apache. I'd start looking for Rogue cron jobs that stop/restart Apache, or start looking over your sar logs to see if you're having any resource issues. When you're having memory issues, the server will automatically kill off processes to keep itself alive. Apache is typically one of the things that will get killed off like this in memory problems.

  • The server has 4GB and rather low traffic, so I'm not sure. I don't see anything unsual in any logs. – JAL Feb 18 '18 at 21:12
  • Do you have sysstat installed? What do the logs show your usage is for the last few days sar -r -f /var/log/sysstat/sa[DAY # GOES HERE] – BoomShadow Feb 19 '18 at 16:45
  • i don't have sysstat running, but do have other things such as longview. – JAL Feb 19 '18 at 22:20
  • Ah, I've not used longview but that looks like a sufficient tool. What logs have you checked so far? Aside from the standard Apache error logs, be sure to check the main syslog of your server to find larger server issues. Be sure to check all the previous days' logs too. It can be a simple 1 entry that gives you your answer, so check timestamps well to match up to times you know Apache died. Also grep for words like: resource, memory, core, kill, & error. If you're running a machine with systemd read up on all the available journald commands. – BoomShadow Feb 20 '18 at 16:09
  • I checked all web server logs, syslog, etc there were no entries, hence my confusion. It looks like it was certbot 's systemd timer, failing and then not restarting apache. I am new to systemd - thanks for the info. – JAL Feb 21 '18 at 19:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.