0

I have strange problem with Apache (MPM Prefork) and mod_php. Server is running for some time and after random time few process is hang and using 100% CPU.

 1741 fileu  20   0  840m  65m  41m R 100.0  0.3 441:49.97 httpd
12200 fileu  20   0  836m  61m  41m R 100.0  0.3 191:42.99 httpd
14023 fileu  20   0  837m  66m  45m R 100.0  0.3 101:49.21 httpd
16530 fileu  20   0  838m  64m  42m R 100.0  0.3 136:46.84 httpd
17094 fileu  20   0  837m  73m  51m R 100.0  0.3 176:53.57 httpd
27652 fileu  20   0  835m  54m  35m R 100.0  0.2 306:45.53 httpd
28681 fileu  20   0  834m  75m  57m R 100.0  0.3 121:55.73 httpd
32203 fileu  20   0  836m  67m  47m R 100.0  0.3  77:00.48 httpd

CPU utils is going up also with load average. Other Apache process are working well. I was trying to use strace -p 32203 -s 80 -o /root/httpd.log but file is empty. Also in error_log nothing special. After restart, everything is OK.

What can I do to check what is going with this process and why it is using 100% CPU?

Some details about server:

  • 2.6.32-504.8.1.el6.x86_64 #1 SMP Fri Dec 19 12:09:25 EST 2014 x86_64 x86_64 x86_64 GNU/Linux
  • Red Hat Enterprise Linux Server release 6.6 (Santiago)
  • PHP 5.6.6 with Zend OPcache v7.0.4-dev
  • Apache/2.2.15
3
  • Are you useing APC? We had some problems with APC not releasing a semaphore and it caused apache processes to hang. Commented Mar 31, 2015 at 13:56
  • No. I have repleaced APC with Opcache, but I don't think that this is and issue, because it start hanging few days ago, and OpCache I have since few months. Commented Mar 31, 2015 at 17:52
  • Could you make a stack trace of one of the processes? Commented Mar 31, 2015 at 18:10

1 Answer 1

0

In case the issue is PHP-based, you could use xdebug profiling to capture some data and then analyze it using wincachegrind or kcachegrind (see http://www.xdebug.org/docs/profiler). Be very watching if doing this on a production server though as this will add extra load and eat up disk space rapidly.

Or you could use xdebug tracing to see the last line of code that's being executed before this happens. You can specify the tracing should be written in such a way that the process ID is part of the filename so you can match it up.

8
  • if they hang randomly, how are you going to know when to see the last line of code?
    – Scott
    Commented May 22, 2018 at 14:31
  • @Scott The xdebug trace file logs code as it executes. Just scroll to the end of the file to see the last line.
    – g491
    Commented May 23, 2018 at 15:16
  • what is an xdebug trace file logs? and I'm suspicious part of the problem is server load issues. Adding xdebug will just increase server load creating still more of the hung processes if correct.
    – Scott
    Commented May 30, 2018 at 10:42
  • @Scott Yes, it will increase the load. You can set it to just capture when a particular cookie is set though so it's not doing it for all sessions. Here is what a trace log is xdebug.org/docs/execution_trace
    – g491
    Commented May 31, 2018 at 16:01
  • wow, no kidding. And fills up the hard drive fast. But ok, what if you are trying to figure out what is actually causing the hung processes and don't know where to look? As in where/when are you going to set the cookie? Isn't there some way to examine a thread to see what the url and/or script is that's hanging?
    – Scott
    Commented Jun 1, 2018 at 16:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .