1

My server sometimes gets filled up with connections in apache stuck in a "Sending Reply" state, requiring me to restart apache. Most of the time this works, but sometimes I will get this error when trying to restart apache instead:

Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

Running systemctl status httpd.service or journalctl -xe as suggested then brings back the following relevant information:

Nov 15 06:24:06 hostname.biologyreporter.com systemd-logind[874]: Failed to remove runtime directory /run/user/1067: Device or resource busy

Nov 15 06:24:27 hostname.biologyreporter.com restartsrv_httpd[29484]: [Fri Nov 15 06:24:27.255594 2019] [core:emerg] [pid 29509:tid 47498001208384] (28)No space left on device: AH00023: Couldn't create the mpm-accept mutex

Nov 15 06:24:27 hostname.biologyreporter.com restartsrv_httpd[29484]: (28)No space left on device: could not create accept mutex

Nov 15 06:24:27 hostname.biologyreporter.com restartsrv_httpd[29484]: AH00015: Unable to open logs Nov 15 06:24:27 hostname.biologyreporter.com systemd[1]: httpd.service: control process exited, code=exited status=1

Nov 15 06:24:27 hostname.biologyreporter.com systemd[1]: Failed to start Apache web server managed by cPanel EasyApache.

-- Subject: Unit httpd.service has failed

It may then take ~10 minutes or so for apache to actually restart and the website to be functional again. I don't know why it is saying "No space left on device" as running df, for example, returns the following (nowhere is 100% of space taken):

Filesystem     1K-blocks      Used Available Use% Mounted on
devtmpfs         3973024         0   3973024   0% /dev
tmpfs            3983400         0   3983400   0% /dev/shm
tmpfs            3983400    255296   3728104   7% /run
tmpfs            3983400         0   3983400   0% /sys/fs/cgroup
/dev/sda2      952008348 137586024 766039760  16% /
/dev/sda1         999320    134892    795616  15% /boot
/dev/loop0       3997376      8856   3778808   1% /tmp
tmpfs             796684         0    796684   0% /run/user/0
tmpfs             796684         0    796684   0% /run/user/1022

I've been unable to find any reliable information on the cause and solution of the specific error(s) as listed above ("No space left on device: AH00023: Couldn't create the mpm-accept mutex" and "Failed to remove runtime directory /run/user/1067: Device or resource busy"). What should I do to solve this so that these errors don't show up and apache always restarts without issues?

  • what if you do "df -i" instead? – Orphans Nov 15 '19 at 15:20
  • Loads of free space for everything – user548654 Nov 19 '19 at 3:18
2

This is most likely caused by the semaphore limit of the OS, and Apache not properly cleaning up after itself.

See more details about semaphore limits https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_for_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guide-setting_semaphores-setting_semaphore_parameters

Those errors mean that there is a lack of inner-process communication resources in the system, such as semaphores or shared memory segments. Try running next to see if the semaphore limit is reached and see results:

ipcs -s | wc -l

cat /proc/sys/kernel/msgmni

cat /proc/sys/kernel/sem

Execute the following command via SSH or console and see if it helps :

ipcs -s | awk -v user=apache '$3==user {system("ipcrm -s "$2)}'
  • what does the command ipcs -s | awk -v user=apache '$3==user {system("ipcrm -s "$2)}' do? – user548654 Nov 19 '19 at 3:16
  • 1
    @user548654, ipcs -s lists currently used semaphores. awk -v user=apache filters out the ones, owned by apache user and next part executes icprm -s "id" for each of it which removes that semaphore. I.e. whole command removes all semaphores owned by apache user. – NStorm Nov 19 '19 at 7:50
  • @NStorm thank you so much for the explanation! – dzup4uk Nov 19 '19 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.