0

I've run into an issue where enabling the PHP driver for MS SQL Server [1] causes Apache httpd to timeout after three minutes when started with systemd (systemctl start httpd). Apache functions fine if started directly (httpd -X). The systemctl error is:

Job for httpd.service failed because a fatal signal was delivered to the control process. See "systemctl status httpd.service" and "journalctl -xe" for details.

If I disable the PHP/MSSQL driver, the problem goes away.

This is RedHat 7.5 and SELinux is in "permissive" mode.

The linke below has relevant log files from running echo $(date) && systemctl restart httpd. Date echoed was Mon Jun 25 11:46:25 CDT 2018, so logs start at that point. Estimated failure time at 11:49:25 since it's been timing out 3 minutes after running restart httpd. Note that there were no entries in the PHP error log, so that's not included. Apache LogLevel set to debug.

Log files: https://gist.github.com/jamesmontalvo3/50cfaa2b2cee966917fc96e3c6784887

Since the logs above don't appear to show anything useful, I also ran strace -ff -o /tmp/st2 -s 5000 systemctl start httpd. The output of the the six stack traced processes can be found here: https://gist.github.com/jamesmontalvo3/89f5e3897c1e2b71b68818215f220477. However, I also don't see any errors here.

How can I determine how to fix this without any useful error messages? Or is there something that I'm just missing?

I've posted this issue on the PHP/MSSQL driver's GitHub [2] and on the Apache mailing list [3].

Refs:

[1] https://github.com/Microsoft/msphpsql

[2] https://github.com/Microsoft/msphpsql/issues/805

[3] http://mail-archives.apache.org/mod_mbox/httpd-users/201807.mbox/%3CCANkjp2rF2beFWWaEpLSouCOc+txzdGh0rihLYz4s4-uWQmOoOg@mail.gmail.com%3E

  • [Mon Jun 25 11:48:31.921182 2018] [core:warn] [pid 11956] AH00098: pid file /run/httpd/httpd.pid overwritten -- Unclean shutdown of previous Apache run? Have you checked no istance of httpd is already running? – Guido Vaccarella Jul 3 '18 at 15:12
  • 1
    Enabling MSSQL driver makes the problem show up. Disabling it makes the problem go away. So I don't think the presence of the httpd.pid file is relevant. That said, I just shut down httpd, verified the httpd.pid file was not present, enabled MSSQL driver, and ran systemctl start httpd. The result was the same (error occurred), though I don't see the "...httpd.pid overwritten" line in httpd error_log. After the failure the httpd.pid file is present, though. So in the log files I linked, that line is probably there because the httpd.pid file gets left un-removed after the previous failed run. – Jamesmontalvo3 Jul 3 '18 at 15:42
1

I believe the important message is this one:

Jun 25 11:47:56 server-that-fails systemd[1]: httpd.service start operation timed out. Terminating.

By default, systemd gives a service 90 seconds (DefaultTimeoutStartSec in system.conf) to start up – if it takes longer than that, the service is killed (first with SIGTERM, then after another 90 s with SIGKILL, see systemd.kill(5) – that’s where your three minutes are coming from). Presumably, enabling the PHP/MSSQL driver makes Apache slow enough to start that it takes longer than 90 seconds.

Try increasing the TimeoutStartSec of apache.service, e. g. TimeoutStartSec=5min or even TimeoutStartSec=infinity. You can do this with the following command:

systemctl set-property apache.service TimeoutStartSec=5min

(By the way, straceing systemctl isn’t very useful, since systemctl doesn’t start the service itself – it just talks to PID1 and tells it to do stuff. To find out what is making Apache so slow, you could add strace to the ExecStart= directive of apache.service.)

  • I couldn't do the systemctl set-property ... command due to an error, but I manually edited /usr/lib/systemd/system/httpd.service and added TimeoutStartSec=600. This allowed httpd to start! Thanks! With the command echo $(date) && sudo systemctl start httpd && echo $(date) I measured it taking just over 2 minutes to start. Any idea how to determine what is causing so much lag? – Jamesmontalvo3 Jul 5 '18 at 18:06
  • 1
    Editing the file in /usr/lib/ is a bad idea, the next package update will overwrite it. Instead, create a drop-in file in /etc/systemd/system/httpd.service.d/timeout.conf and configure TimeoutStartSec= there (don’t forget the section as well). (That’s also what systemctl set-property should do, by the way – perhaps it needs sudo?) – Lucas Werkmeister Jul 5 '18 at 19:36
  • 1
    To find out why Apache is so slow to start, my approach would be strace, either by modifying the ExecStart= directive (also in a drop-in file!), or by attaching to the process after it was started. – Lucas Werkmeister Jul 5 '18 at 19:37
  • 1
    I discovered that when using systemctl it's checking for a user config file at /home/.odbcinst.ini. On most systems it'd just see that file doesn't exist and move on. However, the failing system has /home configured with autofs and when it checks for /home/.odbcinst.ini it tries to mount that file and lags for about 2 minutes before timing out. I added HOME=/usr/share/httpd to /etc/sysconfig/httpd and the problem went away since it now looks for the user config file in /usr/share/httpd/.odbcinst.ini and quickly moves on. Is /etc/sysconfig/httpd the right place to put this info? – Jamesmontalvo3 Jul 6 '18 at 15:45
  • You could also add it to the unit file (or override file) with Environment=HOME=/usr/share/httpd, but I don’t know about RHEL – perhaps /etc/sysconfig/httpd is the best place for it. – Lucas Werkmeister Jul 6 '18 at 19:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.