0

I've got my Apache2.4 (mpm_worker) instance exiting with following log:

[Mon May 27 11:27:33.196177 2019] [core:warn] [pid 567365:tid 139765793668032] AH00045: child process 567368 still did not exit, sending a SIGTERM
[Mon May 27 11:27:35.198179 2019] [core:warn] [pid 567365:tid 139765793668032] AH00045: child process 567368 still did not exit, sending a SIGTERM
[Mon May 27 11:27:37.200177 2019] [core:error] [pid 567365:tid 139765793668032] AH00046: child process 567368 still did not exit, sending a SIGKILL
[Mon May 27 11:27:38.297736 2019] [core:warn] [pid 569972:tid 140126180117440] AH00098: pid file /var/run/apache2/apache2.pid overwritten -- Unclean shutdown of previous Apache run?
[Mon May 27 11:27:38.300264 2019] [mpm_worker:notice] [pid 569972:tid 140126180117440] AH00292: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.0g configured -- resuming normal operations
[Mon May 27 11:27:38.300296 2019] [core:notice] [pid 569972:tid 140126180117440] AH00094: Command line: '/usr/sbin/apache2'
[Mon May 27 11:27:38.303713 2019] [mpm_worker:alert] [pid 569975:tid 140126052628224] (11)Resource temporarily unavailable: AH03142: apr_thread_create: unable to create worker thread
[Mon May 27 11:27:38.359830 2019] [mpm_worker:alert] [pid 569973:tid 140126180117440] (11)Resource temporarily unavailable: AH00282: apr_thread_create: unable to create worker thread
[Mon May 27 11:27:38.371350 2019] [mpm_worker:alert] [pid 569974:tid 140126060156672] (11)Resource temporarily unavailable: AH03142: apr_thread_create: unable to create worker thread
[Mon May 27 11:27:40.302219 2019] [mpm_worker:alert] [pid 569972:tid 140126180117440] AH02325: A resource shortage or other unrecoverable failure was encountered before any child process initialized successfully... httpd is exiting!

Reading the last line in the log, i was assuming apache would be dead now. Instead top gives me a 320% CPU usage for Apache. This runs forever (at least many days). Reading the first lines in the log, its clear that after apache is pretending to exit, its unable to close or kill some child process for any reason.

Googling 'unable to create worker thread' lead me to server ressource limits, and adjusting correct apache2 settings. So here is the Ulimit for www-data:

Ulimit for Apache

root@xxx:~# sudo -u www-data bash -c 'ulimit -a'
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1546431
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 105000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 10000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

mpm_worker.conf

<IfModule mpm_worker_module>
    StartServers             2
    MinSpareThreads      5
    MaxSpareThreads      10
    ThreadLimit          64
    ThreadsPerChild      5
    MaxRequestWorkers     100
    MaxConnectionsPerChild   50
</IfModule>

Server info:

VPS: 6core 2Ghz
RAM: 8GB
Usage: Only one domain/apache instance for Nextcloud.

#1

How can i debug whats going on with apache, after its saying its exited, but its causing high cpu usage? (fixing would be even better)

#2

How can i adjust the worker settings to the use case? (i guess thats whats the root cause)

Any help would be appreciated! Thanks

1
+50

Review your mpm_worker_module configuration to determine if you have enough memory to create the configured amount of threads.

( ThreadLimit X ServerLimit ) X ThreadStackSize = Memory_Max (in bytes) <= System_Memory

Note: The default value of ThreadStackSize varies on other operating systems. You can check it with ulimit -s in RHEL.

Review your /etc/security/limits.conf and/or /etc/security/limits.d/90-nproc.conf to see if the process limit needs to be increased.

The following will show you how many processes httpd worker has running.

ps -elf | grep httpd | grep -v grep | wc -l 

If it exceeds the nproc setting in your etc/security/limits.conf then you can not create enough process (not enough resources) to continue. See ulimit notes on how to increase the setting.

Most common reason for such issue are:

Not enough memory to create a new thread.
Not enough resources to create a new thread (files, process limits)

Algorithm for seeing Apache's Max Memory (for Threads) ThreadStackSize is in bytes.

( ThreadLimit X ServerLimit ) X ThreadStackSize = Memory_Max in bytes

Lowering ServerLimit and/or ThreadsPerChild will reduce the overall number of threads required, so should avoid the error(based on available resource)

  • i guess ThreadStack Size was the keypoint. i reduced from 8192 to 4096 and for now.... it seems to work. Thank you very much. – Sunchezz Jul 1 at 15:17
  • Sorry, had to revert the accept, because this did not help. Reducing the stack size did not change anything. I now realized, that the apache sometimes start with low cpu load and after some time it increases... so, still something very wrong. – Sunchezz Jul 2 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.