0

I would like to find bottleneck for web server (running on 32GB RAM/24 cores HT enabled, apache/mod_php).

Here are my observations,

  1. memory does not seem to be a problem (only 10MB of swap)
  2. very slow in serving static content
  3. php performance looks decent (page served under second)
  4. high cpu usage, occasionally sky rocketing with SYS percentage (as shown below)

I guess first step should be ngnix in front of apache for serving static content?

Or how about MPM-worker apache with php-fpm as alternative? Please suggest and help me improve this question.


top - 11:22:54 up 170 days,  2:35,  6 users,  load average: 324.55, 210.85, 105.87
Tasks: 1351 total, 353 running, 990 sleeping,   7 stopped,   1 zombie
Cpu0  :  0.9%us, 98.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  1.1%si,  0.0%st
Cpu1  :  0.0%us,100.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  3.7%us, 96.3%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  2.8%us, 97.2%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  3.4%us, 96.6%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  :  0.0%us,100.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  :  0.0%us, 99.1%sy,  0.9%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  :  0.0%us,100.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu8  :  0.0%us,100.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu9  :  6.6%us, 93.4%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu10 :  0.6%us, 99.4%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu11 :  0.0%us,100.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu12 :  6.6%us, 92.9%sy,  0.6%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu13 :  0.0%us,100.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu14 :  0.6%us, 99.4%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu15 :  0.3%us, 99.7%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu16 :  8.5%us, 91.5%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu17 :  0.0%us,100.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu18 :  0.0%us,100.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu19 :  0.0%us,100.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu20 :  6.3%us, 93.7%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu21 :  0.0%us,100.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu22 :  4.5%us, 95.5%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu23 :  0.0%us,100.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  32869868k total, 22865280k used, 10004588k free,   177048k buffers
Swap:  4194296k total,   104016k used,  4090280k free,  3578260k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 6736 apache    20   0  371m  59m 3724 R  7.6  0.2   0:28.97 httpd
 6868 apache    20   0  362m  45m 3756 R  7.6  0.1   0:23.53 httpd
  384 apache    20   0  350m  39m 3692 R  7.3  0.1   0:13.44 httpd
  488 apache    20   0  340m  32m 3848 R  7.3  0.1   0:15.58 httpd
  508 apache    20   0  328m  22m 4156 R  7.3  0.1   0:47.30 httpd
  731 apache    20   0  330m  22m 3852 R  7.3  0.1   0:42.05 httpd
 1028 apache    20   0  360m  48m 4168 R  7.3  0.2   0:45.40 httpd
 1113 apache    20   0  365m  58m 4168 R  7.3  0.2   0:28.10 httpd
 1197 apache    20   0  340m  34m 4156 R  7.3  0.1   0:38.98 httpd
 1373 apache    20   0  364m  59m 4860 R  7.3  0.2   0:36.98 httpd

[Sat Feb 22 09:40:25 2014] [notice] child pid 23674 exit signal Segmentation fault (11)
[Sat Feb 22 11:15:01 2014] [notice] child pid 24974 exit signal Segmentation fault (11)
[Sat Feb 22 11:15:03 2014] [notice] child pid 5107 exit signal Segmentation fault (11)
[Sat Feb 22 11:15:06 2014] [notice] child pid 23932 exit signal Segmentation fault (11)
[Sat Feb 22 11:15:06 2014] [notice] child pid 486 exit signal Segmentation fault (11)
[Sat Feb 22 11:15:07 2014] [notice] child pid 5269 exit signal Segmentation fault (11)
[Sat Feb 22 11:15:10 2014] [notice] child pid 2391 exit signal Segmentation fault (11)
[Sat Feb 22 11:15:10 2014] [notice] child pid 5295 exit signal Segmentation fault (11)
[Sat Feb 22 11:15:15 2014] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Sat Feb 22 11:23:46 2014] [warn] child process 27887 still did not exit, sending a SIGTERM
[Sat Feb 22 11:23:46 2014] [warn] child process 5260 still did not exit, sending a SIGTERM
[Sat Feb 22 11:23:46 2014] [warn] child process 6713 still did not exit, sending a SIGTERM
[Sat Feb 22 11:23:48 2014] [warn] child process 27887 still did not exit, sending a SIGTERM
[Sat Feb 22 11:23:48 2014] [warn] child process 5260 still did not exit, sending a SIGTERM
[Sat Feb 22 11:23:48 2014] [warn] child process 6713 still did not exit, sending a SIGTERM
[Sat Feb 22 11:23:50 2014] [warn] child process 27887 still did not exit, sending a SIGTERM
[Sat Feb 22 11:23:50 2014] [warn] child process 5260 still did not exit, sending a SIGTERM
[Sat Feb 22 11:23:50 2014] [warn] child process 6713 still did not exit, sending a SIGTERM
[Sat Feb 22 11:23:52 2014] [error] child process 27887 still did not exit, sending a SIGKILL
[Sat Feb 22 11:23:52 2014] [error] child process 5260 still did not exit, sending a SIGKILL
[Sat Feb 22 11:23:52 2014] [error] child process 6713 still did not exit, sending a SIGKILL
9
  • One classic issue is when apache's forked process are crashing when php module ends. There are several buggy modules, such as imagick, php_pgsql with notifications enabled, and probably many more. Can you check your logs and check for segfaults in apache processes ?
    – Olivier S
    Feb 22 '14 at 19:51
  • 1
    @OlivierS you're right, there are segfaults in error.log, and pgsql is installed but phpinfo says that notifications are disabled? => pgsql.log_notice Off Off
    – mpapec
    Feb 22 '14 at 22:15
  • 1
    ok, the issue is because apache processes are crashing, probably when apache is doing the cleanup of its process. Can you give more info about these crashes? You should have infos in the logs. Eventually try to have a core dump and show us a backtrace.
    – Olivier S
    Feb 23 '14 at 7:26
  • @OlivierS I've added error log, and I'll try core dump later.
    – mpapec
    Feb 23 '14 at 7:49
  • in your system event log ( what os are you running by the way ? ), you should have some infos about the segfault
    – Olivier S
    Feb 23 '14 at 7:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.