Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have this recurring issue where an httpd processes will randomly start running at 100% CPU. Often other httpd processes will join in, and it will continue until I restart Apache. Oddly, the thing it's running 100% CPU on is "OPTIONS * HTTP/1.0". Here's the output from one of these:

9-0     38787   1/9/391 C   103.14  1323    7   0.0 0.08    4.11    ::1 www.mysite.com  OPTIONS * HTTP/1.0

CPU is at 103.14% and it's been 1323 seconds since its last request. It's also stuck in 'C' - closing connection state.

Here's another case where other processes join in running 100% CPU:

0-0     12792   0/33/64 W   95.73   1097    0   0.0 0.10    0.39    66.68.237.216   www.mysite.com      POST /page_a.php HTTP/1.1
9-0     12795   1/6/15  C   94.42   1174    0   0.0 0.03    0.07    ::1             www.myserver.com    OPTIONS * HTTP/1.0 
19-0    12986   0/4/41  W   95.67   1011    0   0.0 0.03    0.24    81.237.216.111  www.mysite.com      POST /page_b.php HTTP/1.1 
20-0    12720   0/10/10 W   94.32   1220    0   0.0 0.03    0.03    187.184.103.218 www.mysite.com      POST /page_a.php HTTP/1.1

My setup is that this is on OS X Lion 10.4.7. I'm running Apache2 with PHP 5.3. Server Version: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 PHP/5.3.10 with Suhosin-Patch

Some potentially relevant http.conf settings:

MaxRequestsPerChild 100000
Timeout 300
KeepAlive On
KeepAliveTimeout 8
MaxKeepAliveRequests 100

When I sample the runaway processes, I see each of them show _spin_lock and kevent like so:

   Sort by top of stack, same collapsed (when >= 5):
    _spin_lock$VARIANT$mp  (in libsystem_c.dylib)        4358
    kevent  (in libsystem_kernel.dylib)        2179

Here is some of the non-repeating Apple code I see:

Call graph:
2413 Thread_1080146   DispatchQueue_1: com.apple.main-thread  (serial)
+ 2413 start  (in httpd) + 52  [0x10f364794]
+   2413 main  (in httpd) + 4045  [0x10f37048d]
+     2413 ap_mpm_run  (in httpd) + 1740  [0x10f3aaabc]
+       2413 perform_idle_server_maintenance  (in httpd) + 703  [0x10f3aa38f]
+         2413 make_child  (in httpd) + 435  [0x10f3aa003]
+           2413 child_main  (in httpd) + 1831  [0x10f3a9e47]
+             2413 clean_child_exit  (in httpd) + 49  [0x10f3a8d31]
+               2413 apr_pool_destroy  (in libapr-1.0.dylib) + 52  [0x10f45761b]
+                 2413 ???  (in libapr-1.0.dylib)  load address 0x10f44b000 + 0xb7b4  [0x10f4567b4]
+                   2413 php_apache_child_shutdown  (in libphp5.so) + 17  [0x10fa3842f]
+                     2413 php_module_shutdown_wrapper  (in libphp5.so) + 9  [0x10f97021d]
+                       2413 php_module_shutdown  (in libphp5.so) + 35  [0x10f970167]
+                         2413 zend_shutdown  (in libphp5.so) + 57  [0x10f9c0692]
+                           2413 zend_hash_destroy  (in libphp5.so) + 53  [0x10f9cb19a]
+                             2413 destroy_op_array  (in libphp5.so) + 271  [0x10f9b91c0]
+                               2413 _efree  (in libphp5.so) + 52  [0x10f9a6312]
+                                 2413 _zend_mm_free_canary_int  (in libphp5.so) + 473  [0x10f9db899]
+                                   2413 free  (in libsystem_c.dylib) + 71  [0x7fff8f86170e]
+                                     2413 szone_size_try_large  (in libsystem_c.dylib) + 37  [0x7fff8f8240f9]
+                                       2413 _spin_lock$VARIANT$mp  (in libsystem_c.dylib) + 30,25,...  [0x7fff8f86336e,0x7fff8f863369,...]
2413 Thread_1080153   DispatchQueue_2: com.apple.libdispatch-manager  (serial)
  2413 _dispatch_mgr_thread  (in libdispatch.dylib) + 54  [0x7fff908fc31a]
    2413 _dispatch_mgr_invoke  (in libdispatch.dylib) + 923  [0x7fff908fd78a]
      2413 kevent  (in libsystem_kernel.dylib) + 10  [0x7fff8ed047e6]

And then stuff like this:

       0x10f4ce000 -        0x10f4d0ff7  mod_reqtimeout.so (??? - ???) <035F872B-8196-3CCE-A4D0-AA8D5C1550EC> /usr/libexec/apache2/mod_reqtimeout.so
   0x10f4d4000 -        0x10f4d8ff7  mod_ext_filter.so (??? - ???) <F33A3409-BFBA-3C8F-9FC9-C1BBC17DBE3F> /usr/libexec/apache2/mod_ext_filter.so
   0x10f4dd000 -        0x10f4eaff7  mod_include.so (??? - ???) <70E541B9-A864-3FE1-AB85-EBF632FFD376> /usr/libexec/apache2/mod_include.so
   0x10f4ef000 -        0x10f4f2ff7  mod_filter.so (??? - ???) <2093EE45-E335-3B36-A6BA-6EA4EB7E483C> /usr/libexec/apache2/mod_filter.so
   0x10f4f6000 -        0x10f4f8ff7  mod_substitute.so (??? - ???) <9ED1AB37-EE13-39DC-AB97-98A2B39555B0> /usr/libexec/apache2/mod_substitute.so
   0x10f4fc000 -        0x10f501ff7  mod_deflate.so (??? - ???) <E334FEA5-BB4E-36BE-A364-DCE248793345> /usr/libexec/apache2/mod_deflate.so
   0x10f506000 -        0x10f50bfff  mod_log_config.so (??? - ???) <61EA3051-8D4A-3A00-B7BC-C68E18CA9479> /usr/libexec/apache2/mod_log_config.so
   0x10f511000 -        0x10f512fef  mod_log_forensic.so (??? - ???) <06654BB4-CA2A-3D70-B759-12191119E5C7> /usr/libexec/apache2/mod_log_forensic.so
   0x10f516000 -        0x10f516ff7  mod_logio.so (??? - ???) <A3CE3AED-D5FE-3947-8CEB-0A8F4C534E86> /usr/libexec/apache2/mod_logio.so
   0x10f51a000 -        0x10f51aff7  mod_env.so (??? - ???) <DE58A08A-C6C9-307B-A077-03A9F1AA14C5> /usr/libexec/apache2/mod_env.so
   0x10f51e000 -        0x10f524fff  mod_mime_magic.so (??? - ???) <1737F398-6315-31B4-B8B4-57F590F07268> /usr/libexec/apache2/mod_mime_magic.so
   0x10f529000 -        0x10f52aff7  mod_cern_meta.so (??? - ???) <B0D4FF96-C96D-3F3A-AECD-431F22A998FA> /usr/libexec/apache2/mod_cern_meta.so
   0x10f52e000 -        0x10f52ffff  mod_expires.so (??? - ???) <FE97E6F0-3B84-3CAC-8873-3B1525142041> /usr/libexec/apache2/mod_expires.so
   0x10f533000 -        0x10f536ff7  mod_headers.so (??? - ???) <5701D330-D777-3AAF-AEEF-F02D067F851E> /usr/libexec/apache2/mod_headers.so
   0x10f53b000 -        0x10f53cfff  mod_ident.so (??? - ???) <5FDFBB79-3A0C-3439-BF71-74E6A3A4B7AC> /usr/libexec/apache2/mod_ident.so
   0x10f540000 -        0x10f542ff7  mod_usertrack.so (??? - ???) <09F36BB5-4F8D-339B-AA52-5FA23A946837> /usr/libexec/apache2/mod_usertrack.so
   0x10f546000 -        0x10f547fff  mod_setenvif.so (??? - ???) <D312073B-0320-3810-9535-3AB8EBF83659> /usr/libexec/apache2/mod_setenvif.so
   0x10f54b000 -        0x10f54cff7  mod_version.so (??? - ???) <4BF2E21C-E452-340E-A6B4-196DBD731CA8> /usr/libexec/apache2/mod_version.so
   0x10f550000 -        0x10f566fff  mod_proxy.so (??? - ???) <0169F3B2-A81A-3E23-92FE-8E9B92C38795> /usr/libexec/apache2/mod_proxy.so
   0x10f56e000 -        0x10f576ff7  mod_proxy_http.so (??? - ???) <F5B55152-D3D1-34AE-A95A-029F01476A54> /usr/libexec/apache2/mod_proxy_http.so
   0x10f57c000 -        0x10f57efff  mod_proxy_scgi.so (??? - ???) <DFD46BF1-F69D-383B-8B33-5273DD8037E4> /usr/libexec/apache2/mod_proxy_scgi.so
   0x10f583000 -        0x10f589ff7  mod_proxy_balancer.so (??? - ???) <BC9C283D-8852-3016-8142-96A5675956FA> /usr/libexec/apache2/mod_proxy_balancer.so

I can post more details as requested.

share|improve this question
    
I ran into a similar problem once on FreeBSD. See /server-status shows over 240 requests like “OPTIONS * HTTP/1.0” 200 - “-” “Apache (internal dummy connection)” . The trigger was filesystem snapshots, which momentarily suspended writes to the filesystem so that the system could take a filesystem snapshot. Is your filesystem healthy? Anything unusual on your filesystem? –  Stefan Lasiewski May 27 '12 at 18:13

2 Answers 2

I am not totally familiar with the macosx process command line tools, but if those values in the 2nd column are httpd child process pids, then I would try and attach to the process using the dtrace tool and see what they are up to while hung like that.

dtrace -p 12345

http://www.troeger.eu/cms/?p=255
man pages

share|improve this answer
    
I can sample the process from the activity monitor, and it appears to be endless non-repeating Apple code. I'll post some in my original question –  Thomas Adams May 31 '12 at 20:07

So I've solved this problem by rewriting some of the code. It appears this odd behavior was due to something in my PHP code. I don't know what it was exactly, but I removed a number of global variables and broke several parts into smaller functions, and the issue stopped. It's odd that PHP's timeout wouldn't be triggered, but there you go.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.