0

I have strange problem with Apache (MPM Prefork) and mod_php. Server is running for some time and after random time few process is hang and using 100% CPU.

 1741 fileu  20   0  840m  65m  41m R 100.0  0.3 441:49.97 httpd
12200 fileu  20   0  836m  61m  41m R 100.0  0.3 191:42.99 httpd
14023 fileu  20   0  837m  66m  45m R 100.0  0.3 101:49.21 httpd
16530 fileu  20   0  838m  64m  42m R 100.0  0.3 136:46.84 httpd
17094 fileu  20   0  837m  73m  51m R 100.0  0.3 176:53.57 httpd
27652 fileu  20   0  835m  54m  35m R 100.0  0.2 306:45.53 httpd
28681 fileu  20   0  834m  75m  57m R 100.0  0.3 121:55.73 httpd
32203 fileu  20   0  836m  67m  47m R 100.0  0.3  77:00.48 httpd

CPU utils is going up also with load average. Other Apache process are working well. I was trying to use strace -p 32203 -s 80 -o /root/httpd.log but file is empty. Also in error_log nothing special. After restart, everything is OK.

What can I do to check what is going with this process and why it is using 100% CPU?

Some details about server:

  • 2.6.32-504.8.1.el6.x86_64 #1 SMP Fri Dec 19 12:09:25 EST 2014 x86_64 x86_64 x86_64 GNU/Linux
  • Red Hat Enterprise Linux Server release 6.6 (Santiago)
  • PHP 5.6.6 with Zend OPcache v7.0.4-dev
  • Apache/2.2.15
  • Are you useing APC? We had some problems with APC not releasing a semaphore and it caused apache processes to hang. – Christopher Perrin Mar 31 '15 at 13:56
  • No. I have repleaced APC with Opcache, but I don't think that this is and issue, because it start hanging few days ago, and OpCache I have since few months. – paszczak000 Mar 31 '15 at 17:52
  • Could you make a stack trace of one of the processes? – Christopher Perrin Mar 31 '15 at 18:10
0

In case the issue is PHP-based, you could use xdebug profiling to capture some data and then analyze it using wincachegrind or kcachegrind (see http://www.xdebug.org/docs/profiler). Be very watching if doing this on a production server though as this will add extra load and eat up disk space rapidly.

Or you could use xdebug tracing to see the last line of code that's being executed before this happens. You can specify the tracing should be written in such a way that the process ID is part of the filename so you can match it up.

  • if they hang randomly, how are you going to know when to see the last line of code? – Scott May 22 '18 at 14:31
  • @Scott The xdebug trace file logs code as it executes. Just scroll to the end of the file to see the last line. – g491 May 23 '18 at 15:16
  • what is an xdebug trace file logs? and I'm suspicious part of the problem is server load issues. Adding xdebug will just increase server load creating still more of the hung processes if correct. – Scott May 30 '18 at 10:42
  • @Scott Yes, it will increase the load. You can set it to just capture when a particular cookie is set though so it's not doing it for all sessions. Here is what a trace log is xdebug.org/docs/execution_trace – g491 May 31 '18 at 16:01
  • wow, no kidding. And fills up the hard drive fast. But ok, what if you are trying to figure out what is actually causing the hung processes and don't know where to look? As in where/when are you going to set the cookie? Isn't there some way to examine a thread to see what the url and/or script is that's hanging? – Scott Jun 1 '18 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.