5

I have a very odd intermittent issue. It occurs on a shared hosting server running CentOS and Apache/2.2.8, serving mostly PHP based websites at a rate of between 10 - 30 requests/sec. 3.5 GB RAM, kernel 2.6.9-67.0.7ELsmp in case that matters.

What happens is, about once a week (but seemingly randomly with no correlation) apache will get "clogged" handling requests for only ONE domain, and normally most of the retests are all for the same URL. I know the problem occurs because load goes through the roof. (Load reaches 50 or 100, if I don't stop it load hits 200!) Only fix I've found is to: killall -9 httpd; /etc/init.d/httpd restart.

If I check /server-status when this is happening, the output is similar to the following. Note it doesn't always happen on this site but about 75% of the time this site's the culprit: EDIT: This is now happening on a completely different server, so it's not specific to a given script.

UPDATE: We have moved all the sites on this server to a brand new virtual machine on brand new hardware. Faster disks, more RAM. The issue persists, in fact it's even worse! Any ideas? I'd gladly offer a gigantic bounty to the correct answer, but I am reluctant to actually offer one as this question has gone years with no correct answer in sight.

Apache Server Status for localhost

Server Version: Apache/2.2.8 (Unix) mod_ssl/2.2.8 OpenSSL/0.9.7a DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 PHP/5.2.6 mod_perl/2.0.4 Perl/v5.8.8
Server Built: Jun 7 2008 09:35:37
Current Time: Friday, 17-Jul-2009 10:15:40 EDT
Restart Time: Friday, 03-Jul-2009 12:36:59 EDT
Parent Server Generation: 3
Server uptime: 13 days 21 hours 38 minutes 40 seconds
Total accesses: 10343186 - Total Traffic: 111.6 GB
CPU Usage: u1710.63 s2129.19 cu.41 cs0 - .32% CPU load
8.61 requests/sec - 97.4 kB/second - 11.3 kB/request
250 requests currently being processed, 0 idle workers
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWKWWWWWWWWWWWWWWWWWW......
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Srv PID Acc M   CPU SS  Req Conn    Child   Slot    Client  VHost   Request
0-3 3282    0/52/316180 W   1.47    41802   0   0.0 0.54    3849.21 209.249.53.104  zachdotsey.com  GET /wordpress/2007/08/18/the-body-exhibition/ HTTP/1.0
1-3 30061   0/191/313974    W   34.46   44051   0   0.0 2.62    3400.19 65.184.161.115  zachdotsey.com  POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
2-3 2720    1/23/307812 W   0.13    42298   0   0.5 0.01    3479.29 68.118.8.157    zachdotsey.com  GET /wordpress/ HTTP/1.1
3-3 1733    0/382/306392    W   3.59    41744   0   0.0 3.67    3383.63 75.101.200.204  zachdotsey.com  GET /wordpress/2007/08/18/the-body-exhibition/ HTTP/1.0
4-3 3141    0/132/305071    W   3.52    41776   0   0.0 2.20    3335.79 209.249.53.236  zachdotsey.com  GET /wordpress/2007/08/18/the-body-exhibition/ HTTP/1.0
5-3 31819   0/73/300070 W   0.79    43932   0   0.0 2.05    3349.65 65.184.161.115  zachdotsey.com  POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
6-3 6799    0/13/294060 W   0.16    40354   0   0.0 0.25    3261.88 76.23.230.184   zachdotsey.com  GET /wordpress/?p=111 HTTP/1.1
7-3 4334    0/92/289873 W   0.42    40277   0   0.0 0.36    3211.94 72.233.56.144   zachdotsey.com  GET /wordpress/feed/ HTTP/1.1
8-3 18326   0/11/294687 W   0.14    32476   0   0.0 0.06    3217.41 66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
9-3 2451    0/56/291730 W   0.23    42381   0   0.0 0.32    3113.19 68.118.8.157    zachdotsey.com  GET /wordpress/index.php HTTP/1.1
10-3    31211   0/59/280899 W   6.04    44051   0   0.0 0.77    3128.00 72.52.238.128   zachdotsey.com  POST /wordpress/wp-cron.php?doing_wp_cron HTTP/1.0
11-3    3823    0/4/277229  W   0.17    41746   0   0.0 0.04    3014.66 68.118.8.157    zachdotsey.com  GET /wordpress/index.php HTTP/1.1
12-3    4473    0/9/277074  W   0.08    38453   0   0.0 0.00    3313.94 66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
13-3    4242    1/14/272166 W   0.10    41659   0   0.5 0.05    2977.05 68.118.8.157    zachdotsey.com  GET /wordpress/ HTTP/1.1
14-3    2323    0/234/269513    W   0.84    42136   0   0.0 1.02    2946.48 65.184.161.115  zachdotsey.com  POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
15-3    32051   0/6/257740  W   0.08    43872   0   0.0 0.58    2866.34 65.184.161.115  zachdotsey.com  POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
16-3    16986   0/271/259463    W   18.13   16761   0   0.0 4.12    2896.05 74.6.18.233 zachdotsey.com  GET /wordpress/tag/xbox/ HTTP/1.0
17-3    2297    1/8/252501  W   0.54    42502   0   0.5 0.18    2797.73 68.118.8.157    zachdotsey.com  GET /wordpress/ HTTP/1.1
18-3    9476    1/145/247116    W   0.61    40095   0   0.4 1.53    2703.11 64.237.119.18   zachdotsey.com  GET /wordpress/ HTTP/1.1
19-3    2494    0/214/244830    W   3.16    42017   0   0.0 2.40    2618.41 65.184.161.115  zachdotsey.com  POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
20-3    5526    0/118/238426    W   3.50    37656   0   0.0 1.73    2544.06 68.10.212.178   zachdotsey.com  GET /wordpress/2009/07/15/andre-dawkins-to-duke-a-year-early/ H
21-3    31901   0/28/228757 W   0.14    43992   0   0.0 0.04    2522.36 65.184.161.115  zachdotsey.com  POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
22-3    12398   0/25/216141 W   0.40    39998   0   0.0 4.72    2398.82 38.117.108.131  zachdotsey.com  GET /wordpress/ HTTP/1.1
23-3    2300    0/187/222141    W   0.68    42315   0   0.0 0.67    2503.15 65.184.161.115  zachdotsey.com  GET /wordpress/ HTTP/1.1
24-3    31969   0/25/211848 W   0.26    43810   0   0.0 0.38    2326.01 65.184.161.115  zachdotsey.com  POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
25-3    31582   0/25/198495 W   0.17    69010   0   0.0 0.25    2139.13 70.63.195.121   myvenue.venuecom.com    POST /fastmail/ HTTP/1.1
26-3    28115   1/1/197089  W   0.04    40155   0   0.4 0.00    2126.75 64.237.119.18   zachdotsey.com  GET /wordpress/ HTTP/1.1
27-3    12988   1/42/186898 W   0.55    35055   0   2.1 0.36    2037.54 71.193.156.250  zachdotsey.com  GET /wordpress/ HTTP/1.1
28-3    3780    1/26/181659 W   0.34    41662   0   0.5 0.05    1918.00 68.118.8.157    zachdotsey.com  GET /wordpress/ HTTP/1.1
29-3    1002    0/22/169331 W   0.18    39885   0   0.0 0.03    1888.59 64.40.121.190   zachdotsey.com  GET /robots.txt HTTP/1.0
30-3    4493    0/6/163101  W   0.06    41465   0   0.0 0.18    1732.58 209.131.41.49   zachdotsey.com  GET /wordpress/feed/ HTTP/1.0
31-3    32458   0/1/162852  W   0.04    43750   0   0.0 0.01    1758.80 65.184.161.115  zachdotsey.com  POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
32-3    14198   0/571/157504    W   47.36   32680   0   0.0 8.64    1733.04 74.6.18.233 zachdotsey.com  GET /wordpress/2008/12/ HTTP/1.0
33-3    2213    0/9/152830  W   0.32    42624   0   0.0 0.00    1782.36 66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
34-3    2308    0/1/137395  W   0.06    39398   0   0.0 0.00    1433.25 64.40.121.190   zachdotsey.com  GET /wordpress/ HTTP/1.0
35-3    6984    0/123/127498    W   1.88    40241   0   0.0 0.18    1350.38 206.196.125.113 zachdotsey.com  GET /wordpress/?feed=rss2 HTTP/1.1
36-3    1003    0/21/116316 W   0.81    39874   0   0.0 1.15    1216.35 67.202.60.246   zachdotsey.com  GET /wordpress/ HTTP/1.1
37-3    3025    0/19/106816 W   0.16    42196   0   0.0 0.28    1096.76 65.184.161.115  zachdotsey.com  POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
38-3    4250    1/15/101047 W   0.09    41660   0   0.5 0.01    1198.01 68.118.8.157    zachdotsey.com  GET /wordpress/ HTTP/1.1
39-3    3073    0/73/91871  W   2.32    41891   0   0.0 1.81    924.03  68.118.8.157    zachdotsey.com  GET /wordpress/index.php HTTP/1.1
40-3    2133    0/15/86368  W   0.20    39445   0   0.0 0.09    888.41  66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
41-3    2803    0/93/79858  W   0.67    39020   0   0.0 0.69    877.38  64.40.121.190   zachdotsey.com  GET /wordpress/ HTTP/1.0
42-3    3030    0/165/72532 W   35.08   41532   0   0.0 2.17    777.91  65.184.161.115  zachdotsey.com  GET /wordpress/?feed=rss2 HTTP/1.1
43-3    13566   0/427/68292 W   14.81   33263   0   0.0 6.28    682.48  174.129.193.94  zachdotsey.com  GET /wordpress/feed/ HTTP/1.1
44-3    31951   0/6/65523   W   0.39    43980   0   0.0 0.15    624.37  78.31.110.126   zachdotsey.com  GET /wordpress/feed HTTP/1.1
45-3    17775   0/4/64507   W   0.11    32768   0   0.0 0.00    666.08  66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
46-3    17485   0/280/61533 W   30.17   16250   0   0.0 6.34    655.87  77.88.42.26 zachdotsey.com  GET /robots.txt HTTP/1.1
47-3    1379    0/305/51058 W   2.50    42256   0   0.0 5.02    691.25  65.184.161.115  zachdotsey.com  POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
48-3    15264   0/28/49621  W   0.11    33958   0   0.0 0.02    527.41  66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
49-3    20544   1/28/48765  W   1.55    40125   0   0.4 0.17    516.42  64.237.119.18   zachdotsey.com  GET /wordpress/ HTTP/1.1
50-3    14318   0/408/50772 W   26.77   32722   0   0.0 7.28    509.18  74.6.18.233 zachdotsey.com  GET /wordpress/2008/12/ HTTP/1.0
51-3    2258    0/18/45699  W   0.08    39388   0   0.0 0.01    456.81  64.40.121.190   zachdotsey.com  GET /robots.txt HTTP/1.0
52-3    3039    0/21/37675  W   0.38    42077   0   0.0 1.58    442.82  65.184.161.115  zachdotsey.com  POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
53-3    2747    0/3/35053   W   0.06    39268   0   0.0 0.22    371.79  64.40.121.190   zachdotsey.com  GET /wordpress/ HTTP/1.0
54-3    17408   0/22/35784  W   0.26    32871   0   0.0 0.42    367.04  74.6.18.233 zachdotsey.com  GET /robots.txt HTTP/1.0
55-3    4511    0/6/29200   W   0.40    41444   0   0.0 0.23    375.89  66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
56-3    5534    0/54/27466  W   2.63    37695   0   0.0 0.38    305.27  68.10.212.178   zachdotsey.com  GET /wordpress/2009/07/15/andre-dawkins-to-duke-a-year-early/ H
57-3    23760   0/41/20806  W   0.23    40553   0   0.0 0.33    219.16  66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
58-3    5638    0/247/17586 W   2.79    40174   0   0.0 2.66    182.34  64.237.119.18   zachdotsey.com  GET /wordpress/feed/ HTTP/1.1
59-3    31783   0/5/18112   W   0.05    40067   0   0.0 0.06    182.11  65.113.57.7 zachdotsey.com  GET /wordpress/feed/ HTTP/1.0
60-3    2319    0/1/13920   W   0.03    39391   0   0.0 0.00    145.61  38.117.108.131  zachdotsey.com  GET /wordpress/ HTTP/1.1
61-3    23730   0/155/12935 W   29.86   28181   0   0.0 3.25    153.35  72.30.161.222   zachdotsey.com  GET /wordpress/2008/10/26/tiring-surf-sessions/ HTTP/1.0
62-3    12586   0/25/12506  W   0.14    40271   0   0.0 0.09    135.23  72.233.56.156   zachdotsey.com  GET /wordpress/ HTTP/1.1
63-3    13567   0/15/10439  W   0.54    34975   0   0.0 0.10    137.24  84.217.7.221    zachdotsey.com  GET /wordpress/2009/06/06/emergency-baby-call-tests/ HTTP/1.1
64-3    17776   1/161/10534 W   1.65    31758   0   0.4 2.14    128.33  208.92.218.250  zachdotsey.com  GET /wordpress/ HTTP/1.1
65-3    3053    0/159/7939  W   7.25    41485   0   0.0 2.64    74.76   209.131.41.48   zachdotsey.com  GET /robots.txt HTTP/1.0
66-3    5056    0/115/6934  W   1.56    37848   0   0.0 2.43    81.76   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
67-3    16535   0/160/6199  W   19.57   39641   0   0.0 2.57    67.22   64.40.121.190   zachdotsey.com  GET /wordpress/ HTTP/1.0
68-3    17777   0/37/7722   W   0.24    32569   0   0.0 0.15    93.77   74.6.18.233 zachdotsey.com  GET /wordpress/2008/12/ HTTP/1.0
69-3    2355    0/2/5737    W   0.15    39346   0   0.0 0.03    73.54   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
70-3    10432   0/24/5080   W   0.33    36392   0   0.0 0.47    45.89   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
71-3    10433   0/12/4099   W   0.93    36365   0   0.0 0.65    41.80   68.7.141.144    zachdotsey.com  GET /wordpress/2006/12/18/the-plastic-coating-on-the-end-of-a-s
72-3    6145    0/114/4835  W   12.61   37285   0   0.0 1.49    57.55   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
73-3    5539    0/71/4717   W   5.37    37680   0   0.0 1.63    62.39   68.10.212.178   zachdotsey.com  GET /wordpress/2009/07/15/andre-dawkins-to-duke-a-year-early/ H
74-3    17690   0/38/5004   W   0.12    32810   0   0.0 0.25    46.34   74.6.18.233 zachdotsey.com  GET /wordpress/2008/12/ HTTP/1.0
75-3    1528    0/38/3991   W   0.14    39764   0   0.0 0.08    41.91   64.40.121.190   zachdotsey.com  GET /wordpress/ HTTP/1.0
76-3    2751    0/26/2886   W   0.22    39144   0   0.0 0.63    25.56   64.40.121.190   zachdotsey.com  GET /wordpress/ HTTP/1.0
77-3    5453    0/95/6939   W   1.95    21796   0   0.0 1.09    87.49   69.147.79.37    zachdotsey.com  GET /robots.txt HTTP/1.0
78-3    5061    0/192/3238  W   9.87    37110   0   0.0 2.13    39.70   78.31.110.126   zachdotsey.com  GET /wordpress/feed HTTP/1.1
79-3    9955    0/143/4659  W   11.55   35985   0   0.0 7.35    64.53   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
80-3    17779   0/29/4181   W   0.33    32548   0   0.0 0.47    48.11   74.6.18.233 zachdotsey.com  GET /wordpress/2008/12/ HTTP/1.0
81-3    1192    0/74/2861   W   0.54    39651   0   0.0 0.42    27.09   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
82-3    13108   0/42/4116   W   0.13    35144   0   0.0 0.10    37.39   71.193.156.250  zachdotsey.com  GET /wordpress/wp-login.php?loggedout=true HTTP/1.1
83-3    32180   0/0/6535    W   0.98    10753   0   0.0 0.00    105.04  142.166.3.122   zachdotsey.com  GET /robots.txt HTTP/1.1
84-3    24479   0/148/4080  W   98.02   28139   0   0.0 3.46    60.62   72.30.161.222   zachdotsey.com  GET /wordpress/2008/10/26/tiring-surf-sessions/ HTTP/1.0
85-3    776 1/204/8349  W   40.83   9214    0   0.4 3.57    99.11   64.237.119.18   zachdotsey.com  GET /wordpress/ HTTP/1.1
86-3    13569   0/25/3218   W   10.53   34863   0   0.0 0.29    28.96   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
87-3    13110   0/2/2535    W   0.09    35187   0   0.0 0.00    26.16   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
88-3    4716    0/157/2237  W   3.75    37444   0   0.0 3.58    25.28   206.196.125.113 zachdotsey.com  GET /wordpress/?feed=rss2 HTTP/1.1
89-3    19878   0/110/2928  W   10.39   31357   0   0.0 0.54    26.66   58.61.164.39    zachdotsey.com  GET /robots.txt HTTP/1.1
90-3    2003    0/28/1847   W   1.81    39519   0   0.0 0.39    19.10   64.40.121.190   zachdotsey.com  GET /wordpress/ HTTP/1.0
91-3    13570   0/0/1928    W   10.59   35116   0   0.0 0.00    23.59   71.193.156.250  zachdotsey.com  GET /wordpress/ HTTP/1.1
92-3    16358   0/282/1910  W   33.11   32280   0   0.0 5.57    26.53   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
93-3    16367   0/50/1664   W   10.03   33243   0   0.0 0.88    17.62   174.129.193.94  zachdotsey.com  GET /wordpress/feed/ HTTP/1.1
94-3    25642   0/82/2391   W   19.24   28268   0   0.0 2.07    34.95   72.30.161.222   zachdotsey.com  GET /wordpress/2008/10/26/tiring-surf-sessions/ HTTP/1.0
95-3    24480   0/261/2711  W   31.22   27528   0   0.0 3.14    43.29   75.219.249.154  zachdotsey.com  GET /wordpress/tag/missy-warkentien/ HTTP/1.1
96-3    17692   0/514/1024  W   46.55   30965   0   0.0 6.50    14.37   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
97-3    10442   0/126/1041  W   1.25    35891   0   0.0 1.20    8.47    203.171.91.144  zachdotsey.com  GET /wordpress/2009/07/15/andre-dawkins-to-duke-a-year-early/ H
98-3    25643   0/103/1705  W   0.67    28008   0   0.0 2.83    20.89   74.6.18.233 zachdotsey.com  GET /wordpress/2008/10/26/tiring-surf-sessions/ HTTP/1.0
99-3    22619   0/338/1657  W   27.38   28329   0   0.0 4.00    27.55   72.30.161.222   zachdotsey.com  GET /robots.txt HTTP/1.0
100-3   19714   0/0/1845    W   12.66   31778   0   0.0 0.00    22.14   208.92.218.250  zachdotsey.com  GET /wordpress/feed/ HTTP/1.1
101-3   19367   1/155/1983  W   10.76   31728   0   0.4 0.15    23.66   208.92.218.250  zachdotsey.com  GET /wordpress/ HTTP/1.1
102-3   26652   0/122/1187  W   41.42   26872   0   0.0 3.00    14.95   75.219.249.154  zachdotsey.com  GET /wordpress/tag/missy-warkentien/ HTTP/1.1
103-3   7280    3/361/1639  W   12.44   35165   0   8.9 4.80    17.06   71.193.156.250  zachdotsey.com  GET /wordpress/ HTTP/1.1
104-3   20843   0/507/949   W   99.96   28160   0   0.0 7.70    11.64   72.30.161.222   zachdotsey.com  GET /wordpress/2008/10/26/tiring-surf-sessions/ HTTP/1.0
105-3   20844   0/308/1868  W   23.03   30024   0   0.0 3.00    28.11   78.31.110.126   zachdotsey.com  GET /wordpress/feed HTTP/1.1
106-3   17791   1/278/1026  W   26.89   31698   0   0.4 4.18    13.29   208.92.218.250  zachdotsey.com  GET /wordpress/ HTTP/1.1
107-3   17792   0/304/642   W   12.27   31388   0   0.0 2.72    8.79    58.61.164.39    zachdotsey.com  GET /robots.txt HTTP/1.1
108-3   17697   0/57/515    W   0.30    32611   0   0.0 1.78    8.41    74.6.18.233 zachdotsey.com  GET /wordpress/2008/12/ HTTP/1.0
109-3   19715   0/611/783   W   60.52   29153   0   0.0 5.94    7.20    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
110-3   13952   0/345/614   W   5.95    33656   0   0.0 3.47    8.18    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
111-3   32009   0/75/1959   W   5.25    24778   0   0.0 1.06    22.60   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
112-3   17793   0/35/178    W   11.11   32590   0   0.0 0.44    1.92    74.6.18.233 zachdotsey.com  GET /wordpress/2008/12/ HTTP/1.0
113-3   19533   0/8/2004    W   0.16    16705   0   0.0 1.32    34.66   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
114-3   22622   0/304/925   W   181.63  28201   0   0.0 5.05    12.30   72.30.161.222   zachdotsey.com  GET /wordpress/2008/10/26/tiring-surf-sessions/ HTTP/1.0
115-3   26656   0/38/763    W   0.55    27724   0   0.0 0.40    6.22    75.219.249.154  zachdotsey.com  GET /wordpress/tag/missy-warkentien/ HTTP/1.1
116-3   31804   0/57/3039   W   2.17    10748   0   0.0 1.27    44.70   142.166.3.122   zachdotsey.com  GET /wordpress/2009/07/13/millys-first-time-in-a-restaurant/ HT
117-3   26658   0/0/866 W   14.05   28069   0   0.0 0.00    11.50   74.6.18.233 zachdotsey.com  GET /robots.txt HTTP/1.0
118-3   26659   0/46/646    W   4.59    27608   0   0.0 0.47    4.90    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
119-3   5457    0/86/998    W   21.35   21916   0   0.0 1.59    14.79   69.147.79.37    zachdotsey.com  GET /robots.txt HTTP/1.0
120-3   9855    0/486/1470  W   58.03   18535   0   0.0 8.31    25.07   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
121-3   17802   0/8/206 W   0.05    32743   0   0.0 0.00    1.67    74.6.18.233 zachdotsey.com  GET /wordpress/2008/12/ HTTP/1.0
122-3   17803   0/10/141    W   0.19    32701   0   0.0 0.00    1.08    74.6.18.233 zachdotsey.com  GET /wordpress/2008/12/ HTTP/1.0
123-3   19888   0/69/120    W   0.37    31389   0   0.0 0.97    1.10    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
124-3   25184   0/160/689   W   33.97   27701   0   0.0 5.24    9.16    75.219.249.154  zachdotsey.com  GET /wordpress/tag/missy-warkentien/ HTTP/1.1
125-3   27208   0/615/2460  W   28.73   10736   0   0.0 7.88    44.54   142.166.3.122   zachdotsey.com  GET /wordpress/2009/07/14/millys-smile/ HTTP/1.1
126-3   21945   0/281/410   W   4.56    29359   0   0.0 3.82    7.16    74.6.18.233 zachdotsey.com  GET /robots.txt HTTP/1.0
127-3   25187   0/392/429   W   58.81   25928   0   0.0 7.26    7.47    75.219.249.154  zachdotsey.com  GET /wordpress/ HTTP/1.1
128-3   21948   0/223/299   W   2.32    30044   0   0.0 1.29    1.77    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
129-3   27517   0/211/551   W   60.71   25918   0   0.0 4.73    10.25   75.219.249.154  zachdotsey.com  GET /wordpress/2009/07/15/andre-dawkins-to-duke-a-year-early/ H
130-3   9078    1/150/753   W   13.68   21327   0   0.4 2.78    11.88   208.92.218.250  zachdotsey.com  GET /wordpress/ HTTP/1.1
131-3   20742   0/97/2004   W   1.96    15475   0   0.0 2.14    35.30   78.31.110.126   zachdotsey.com  GET /wordpress/feed HTTP/1.1
132-3   12265   0/658/1664  W   62.77   16822   0   0.0 11.80   27.61   74.6.18.233 zachdotsey.com  GET /robots.txt HTTP/1.0
133-3   31859   0/72/2354   W   12.54   10604   0   0.0 1.14    54.86   174.129.193.99  zachdotsey.com  GET /wordpress/feed/ HTTP/1.1
134-3   4105    0/60/658    W   12.23   22622   0   0.0 1.46    9.19    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
135-3   4106    0/195/679   W   15.18   21676   0   0.0 3.77    14.09   69.147.79.37    zachdotsey.com  GET /robots.txt HTTP/1.0
136-3   25192   0/8/8   W   0.21    28835   0   0.0 0.01    0.01    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
137-3   25193   0/103/103   W   15.55   27940   0   0.0 1.20    1.20    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
138-3   25194   0/37/37 W   11.09   28465   0   0.0 1.07    1.07    38.117.108.35   zachdotsey.com  GET /wordpress/feed HTTP/1.1
139-3   28509   0/64/106    W   1.12    26708   0   0.0 0.82    1.54    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
140-3   28955   0/68/68 W   2.49    26135   0   0.0 1.37    1.37    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
141-3   28957   0/59/59 W   8.77    26104   0   0.0 1.60    1.60    75.219.249.154  zachdotsey.com  GET /wordpress/2009/07/15/andre-dawkins-to-duke-a-year-early/ H
142-3   28958   0/126/126   W   4.49    26058   0   0.0 1.75    1.75    75.219.249.154  zachdotsey.com  GET /wordpress/2009/07/15/andre-dawkins-to-duke-a-year-early/ H
143-3   3101    0/26/1994   W   1.88    9152    0   0.0 0.16    32.99   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
144-3   21433   0/29/1004   W   1.06    15819   0   0.0 0.21    16.72   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
145-3   12273   0/336/786   W   29.77   18507   0   0.0 7.08    15.99   58.61.164.141   zachdotsey.com  GET /wordpress/2008/11/01/early-voting/ HTTP/1.1
146-3   4109    0/65/232    W   1.64    22700   0   0.0 0.52    2.96    78.31.110.126   zachdotsey.com  GET /wordpress/feed HTTP/1.1
147-3   9081    0/250/582   W   61.84   20036   0   0.0 4.35    10.27   125.244.77.2    zachdotsey.com  POST /_vti_bin/_vti_aut/author.dll HTTP/1.1
148-3   21639   0/216/1433  W   8.19    14597   0   0.0 1.97    25.39   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
149-3   31137   0/1/1657    W   0.08    11318   0   0.0 0.00    29.24   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
150-3   31210   0/227/227   W   86.27   23567   0   0.0 8.50    8.50    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
151-3   4071    0/42/2081   W   12.87   8452    0   0.0 0.74    32.38   76.13.20.61 zachdotsey.com  GET /robots.txt HTTP/1.0
152-3   9904    0/31/445    W   1.53    21377   0   0.0 0.71    10.45   208.92.218.250  zachdotsey.com  GET /wordpress/feed/ HTTP/1.1
153-3   25865   0/8/1236    W   0.20    13614   0   0.0 0.13    14.50   76.13.20.61 zachdotsey.com  GET /robots.txt HTTP/1.0
154-3   9905    0/78/294    W   15.40   20614   0   0.0 1.30    4.53    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
155-3   9085    0/75/78 W   11.31   21562   0   0.0 0.88    0.88    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
156-3   18546   0/51/192    W   3.91    17077   0   0.0 0.22    4.83    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
157-3   32217   0/0/1740    W   0.24    10744   0   0.0 0.00    28.44   142.166.3.122   zachdotsey.com  GET /wordpress/2009/07/11/millys-anticipation/ HTTP/1.1
158-3   14856   0/264/641   W   3.65    17963   0   0.0 2.75    9.95    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
159-3   18153   0/92/3545   W   4.82    2279    0   0.0 1.50    45.82   74.6.18.233 zachdotsey.com  GET /wordpress/2008/10/24/ HTTP/1.0
160-3   31679   0/56/56 W   0.54    25195   0   0.0 0.61    0.61    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
161-3   32316   0/1/1628    W   0.05    10733   0   0.0 0.00    27.14   142.166.3.122   zachdotsey.com  GET /wordpress/2009/07/12/millys-first-poop-on-daddy/ HTTP/1.1
162-3   25433   0/89/590    W   1.86    13495   0   0.0 1.50    9.58    76.13.20.61 zachdotsey.com  GET /robots.txt HTTP/1.0
163-3   19960   0/278/338   W   26.51   14965   0   0.0 3.34    3.96    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
164-3   20271   0/111/598   W   3.85    15553   0   0.0 3.49    10.91   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
165-3   13282   0/113/3121  W   3.01    4585    0   0.0 0.52    37.39   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
166-3   25641   0/227/658   W   27.64   12512   0   0.0 2.51    8.38    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
167-3   26850   0/303/816   W   6.27    12241   0   0.0 2.71    12.95   77.88.42.26 zachdotsey.com  GET /robots.txt HTTP/1.1
168-3   25966   0/413/898   W   16.14   12226   0   0.0 3.66    10.86   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
169-3   25967   0/35/276    W   0.84    13409   0   0.0 1.19    3.09    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
170-3   28752   0/426/4119  W   31.27   134 0   0.0 4.26    65.53   71.127.124.128  northwoodcinemagrill.com    GET / HTTP/1.1
171-3   10540   0/16/1765   W   0.22    5732    0   0.0 0.23    20.56   92.51.157.168   zachdotsey.com  GET /robots.txt HTTP/1.1
172-3   4297    0/26/1208   W   0.43    8572    0   0.0 0.02    22.46   76.13.20.61 zachdotsey.com  GET /robots.txt HTTP/1.0
173-3   10221   0/140/140   W   63.93   19817   0   0.0 4.69    4.69    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
174-3   26697   0/529/845   W   24.04   11002   0   0.0 6.17    13.25   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
175-3   32336   0/224/1243  W   15.09   9411    0   0.0 2.93    20.51   207.179.181.130 zachdotsey.com  GET /wordpress/feed/ HTTP/1.1
176-3   25871   0/1/189 W   0.08    13705   0   0.0 0.01    1.38    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
177-3   25648   0/57/166    W   3.24    13460   0   0.0 0.38    3.02    58.61.164.39    zachdotsey.com  GET /wordpress/2008/10/25/big-day-for-duke-sports/ HTTP/1.1
178-3   31175   0/93/975    W   26.19   10624   0   0.0 3.61    13.03   174.129.193.99  zachdotsey.com  GET /wordpress/feed/ HTTP/1.1
179-3   10228   1/34/34 W   0.26    21297   0   0.4 1.60    1.60    208.92.218.250  zachdotsey.com  GET /wordpress/ HTTP/1.1
180-3   10229   0/235/235   W   11.14   19431   0   0.0 6.96    6.96    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
181-3   14002   0/180/2169  W   10.99   4030    0   0.0 1.11    18.94   65.184.161.115  zachdotsey.com  GET /wordpress/ HTTP/1.1
182-3   31180   0/307/460   W   38.99   9753    0   0.0 6.36    7.88    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
183-3   10232   1/28/28 W   1.84    21357   0   0.4 0.26    0.26    208.92.218.250  zachdotsey.com  GET /wordpress/ HTTP/1.1
184-3   31929   0/46/714    W   0.38    10749   0   0.0 0.27    10.75   142.166.3.122   zachdotsey.com  GET /wordpress/2009/07/15/andre-dawkins-to-duke-a-year-early/ H
185-3   8930    0/201/1413  W   11.72   6142    0   0.0 0.90    18.43   92.51.157.168   zachdotsey.com  GET /robots.txt HTTP/1.1
186-3   31931   0/314/471   W   47.33   9256    0   0.0 9.23    13.30   76.13.20.61 zachdotsey.com  GET /robots.txt HTTP/1.0
187-3   1832    0/78/850    W   21.91   9234    0   0.0 1.14    8.29    64.237.119.18   zachdotsey.com  GET /wordpress/feed/ HTTP/1.1
188-3   31181   0/121/556   W   5.97    10756   0   0.0 2.10    13.38   142.166.3.122   zachdotsey.com  GET /robots.txt HTTP/1.1
189-3   9444    0/251/1730  W   5.98    5671    0   0.0 2.12    28.18   65.184.161.115  zachdotsey.com  POST /wordpress/wp-admin/admin-ajax.php HTTP/1.1
190-3   31932   0/25/169    W   2.85    10769   0   0.0 0.73    2.87    142.166.3.122   zachdotsey.com  GET /robots.txt HTTP/1.1
191-3   25658   0/6/6   W   0.10    13734   0   0.0 0.04    0.04    76.13.20.61 zachdotsey.com  GET /robots.txt HTTP/1.0
192-3   9884    0/136/1248  W   1.37    5630    0   0.0 1.32    14.36   65.184.161.115  zachdotsey.com  GET /wordpress/ HTTP/1.1
193-3   324 0/37/116    W   5.95    10106   0   0.0 1.27    2.86    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
194-3   19087   0/756/2584  W   67.99   1279    0   0.0 9.26    35.86   78.31.110.126   zachdotsey.com  GET /wordpress/feed HTTP/1.1
195-3   20058   0/722/2684  W   62.21   1123    0   0.0 7.96    30.71   206.196.125.113 zachdotsey.com  GET /wordpress/?feed=rss2 HTTP/1.1
196-3   2425    1/38/56 W   2.18    9154    0   0.4 0.47    0.59    64.237.119.18   zachdotsey.com  GET /wordpress/ HTTP/1.1
197-3   28767   0/341/3091  W   12.60   64  0   0.0 2.89    33.12   65.184.161.115  myvenue.venuecom.com    GET / HTTP/1.1
198-3   4816    0/260/942   W   26.04   7324    0   0.0 2.25    10.97   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
199-3   21139   0/567/2438  W   121.57  18  0   0.0 10.32   43.01   208.59.151.137  webdav.venuecom.com GET /calendar/dnewcorn/calendars/Web%20Demos.ics HTTP/1.1
200-3   4311    0/55/188    W   4.84    8571    0   0.0 1.31    3.77    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
201-3   5891    0/15/205    W   0.11    7957    0   0.0 1.61    6.07    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
202-3   30984   0/0/2581    W   42.72   40  0   0.0 0.00    31.17   207.210.148.62  marketplace-mail.com    GET /checklist.php HTTP/1.1
203-3   2435    1/40/42 W   19.37   9184    0   0.4 2.43    2.44    64.237.119.18   zachdotsey.com  GET /wordpress/ HTTP/1.1
204-3   29989   0/26/2777   W   30.30   78  0   0.0 1.51    38.95   198.178.147.1   raleighwoodmovies.com   GET / HTTP/1.1
205-3   19694   0/0/1370    W   0.14    2633    0   0.0 0.00    19.70   66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
206-3   19406   0/202/1756  W   10.98   2238    0   0.0 1.83    20.94   74.6.18.233 zachdotsey.com  GET /wordpress/2008/10/24/ HTTP/1.0
207-3   19695   0/937/2326  W   42.78   482 0   0.0 8.58    23.44   76.182.6.152    raleighwoodmovies.com   GET / HTTP/1.1
208-3   11763   0/129/802   W   14.71   4766    0   0.0 2.16    9.27    65.184.161.115  zachdotsey.com  GET /wordpress/?feed=rss2 HTTP/1.1
209-3   2442    0/26/27 W   9.27    9236    0   0.0 0.54    0.54    77.88.42.26 zachdotsey.com  GET /robots.txt HTTP/1.1
210-3   2447    0/94/95 W   7.65    9136    0   0.0 0.73    0.73    76.13.20.61 zachdotsey.com  GET /robots.txt HTTP/1.0
211-3   3914    0/80/100    W   6.41    8330    0   0.0 0.48    0.61    78.31.110.126   zachdotsey.com  GET /wordpress/feed HTTP/1.1
212-3   11674   0/0/519 W   14.19   5522    0   0.0 0.00    7.40    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
213-3   7627    0/70/211    W   2.47    6989    0   0.0 1.77    3.11    77.88.42.26 zachdotsey.com  GET /robots.txt HTTP/1.1
214-3   29724   0/557/557   W   80.97   9376    0   0.0 7.93    7.93    76.13.20.61 zachdotsey.com  GET /robots.txt HTTP/1.0
215-3   19410   0/980/2490  W   162.44  138 0   0.0 7.87    26.41   198.178.147.1   raleighwoodmovies.com   GET /printable.php HTTP/1.1
216-3   9451    0/9/405 W   0.16    6415    0   0.0 0.06    2.35    66.249.66.2 zachdotsey.com  GET /robots.txt HTTP/1.1
217-3   11764   0/9/695 W   0.13    5407    0   0.0 0.08    9.91    98.122.166.13   zachdotsey.com  GET /wordpress/ HTTP/1.1
218-3   20889   0/46/1701   W   0.18    2217    0   0.0 0.37    26.16   74.6.18.233 zachdotsey.com  GET /wordpress/2008/10/24/ HTTP/1.0
219-3   17642   0/511/1352  W   8.05    2259    0   0.0 2.89    13.17   74.6.18.233 zachdotsey.com  GET /wordpress/2008/10/24/ HTTP/1.0
220-3   19411   0/757/1910  W   29.09   1126    0   0.0 5.47    18.72   65.184.161.115  zachdotsey.com  GET /wordpress/ HTTP/1.1
221-3   19412   0/284/1153  W   5.34    2085    0   0.0 3.74    22.78   74.6.18.233 zachdotsey.com  GET /wordpress/2008/10/24/ HTTP/1.0
222-3   19696   0/885/1794  W   54.08   237 0   0.0 15.96   25.04   69.86.60.67 webdav.venuecom.com GET /calendar/lswuser/private/calendars/Prospecting.ics HTTP/1.
223-3   19697   0/57/1272   W   0.58    2409    0   0.0 0.38    11.08   74.6.18.233 zachdotsey.com  GET /robots.txt HTTP/1.0
224-3   20500   0/226/1395  W   0.82    2147    0   0.0 2.12    11.41   74.6.18.233 zachdotsey.com  GET /wordpress/2008/10/24/ HTTP/1.0
225-3   21060   0/776/1869  W   36.25   475 0   0.0 10.92   24.86   78.31.110.127   allnews.allchoiceinsurance.com  GET /feed/ HTTP/1.1
226-3   15942   8/360/708   W   10.60   2428    0   29.1    3.08    6.61    65.184.161.115  zachdotsey.com  GET /wordpress/ HTTP/1.1
227-3   18059   1/9/402 W   2.48    3120    0   419.1   0.83    4.29    24.199.245.190  zachdotsey.com  GET /wordpress/ HTTP/1.1
228-3   3929    0/102/102   W   16.70   8332    0   0.0 1.24    1.24    76.13.20.61 zachdotsey.com  GET /robots.txt HTTP/1.0
229-3   7814    0/343/396   W   2.93    6380    0   0.0 1.71    1.86    98.122.166.13   zachdotsey.com  GET /wordpress/ HTTP/1.1
230-3   20958   0/746/2021  W   86.23   121 0   0.0 11.92   25.76   72.30.87.105    baucomsbest.com GET /index.php HTTP/1.0
231-3   20897   1/765/1673  K   43.84   4   15  10.6    7.17    17.99   24.163.58.115   myvenue.venuecom.com    GET /editor/images/progress-bar-back.gif HTTP/1.1
...
Improve this question
5
  • Also I would love to add tag 'intermittent' but don't have the rep. required to do so.
    – Josh
    Jul 21, 2009 at 20:14
  • 2
    Not enough information for an answer, but usually it's a script (php, perl, whatever) going berserk Also check the log to see if there's any hint on which request is stalling the server (search for return codes 5xx)
    – Luca Tettamanti
    Jul 21, 2009 at 20:31
  • @Luca: Thanks. Nothing in the logs. Everytime it happens on this site I do see a request for [ POST /wordpress/wp-admin/admin-ajax.php ]. But that script is used successfully 99% of the time...
    – Josh
    Jul 21, 2009 at 20:34
  • anything in dmesg?
    – Tarnay Kálmán
    Dec 9, 2009 at 3:43
  • No, nothing useful... :-(
    – Josh
    Dec 9, 2009 at 16:48

8 Answers 8

Reset to default
6
+500

After you saying what modules you had, I noticed you said EAccelerator, which has a bug (Bug tracker link here) that causes Apache to lockup in never ending loop, and bring the server to a near standstill, try disabling that and see what happens.

Improve this answer
1
  • Thanks for the link. We do use eAccelerator, and it's clearly a PHP resource contention issue as others have said. I have been unable to find the cause, but this seems like a very likely culprit. If this works, the bounty is yours. Thanks!
    – Josh
    Apr 11, 2011 at 20:24
5

Do you see many requests for POST /wordpress/wp-admin/admin-ajax.php at the problem times? If so then it could be caused by either a passing spammer trying to brute force something, by a deliberate DoS attempt, or just because the site is getting posted somewhere busy every now and again (do your logs show many requests coming from the same off-site referrer?).

It could also be a database locking issue, i.e. one request has asked a long operation to be performed with full deny-read-until-done locks on a frequently accessed table and the other requests are just stuck waiting for the lock to be released (which may happen naturally, or as a result of you killing Apache and therefore the connection that is holding the lock(s). If the site is busy enough then this can create significant load as lots of Apache processes get created which then sit consuming memory until they can continue past the lock - if your Apache config is such that a relatively unconstrained number of child processes can be created then the system could easily start thrashing swap at this point.

To see if the load is being caused by CPU use, IO due to DB access and such, or IO due to swap thrashing, you'll need to investigate the output of htop (or just plain top), free, and the other usual system load monitoring utils. This may give more clues as to where the root of the issue might be found.

Improve this answer
1
  • David, thanks for your answers. I normally only see one or two requests for the admin-ajax.php script. But this sometimes happens on requests that have no database access at all... so it's not specific to wordpress and not related to a database. CPU load isn't relatively high, it's just like Apache deadlocks and starts not serving requests, eventually reaching MaxChildren and stopping completely.
    – Josh
    Jul 30, 2009 at 19:10
2

That definitely looks like a slowloris DoS attack against one of the sites you host.

Find more details, as well as Debian and Red-Hat solutions here

One of the other symptoms of that type of attack will be the appearance of dozens/hundreds of 400 errors in your Apache logs (if/after the attack ends - read more at ha.ckers.org/slowloris/)

Improve this answer
0

You could try setting the TimeOut value in apache to something quite draconian like 20 seconds. This should finish the connection if certain actions take longer to complete. This is sort of papering over the problem, rather than solving it. It should probably be used to debug if the problem is due to something taking too long, rather than as a final solution.

Improve this answer
1
  • Thanks for the suggestion... I had a server that was acting up yesterday and I tried this, and for about 10 minutes I thought it was working, but then it happened again.
    – Josh
    Dec 9, 2009 at 16:48
0

Are your PHP scripts making any database requests? If so, verify that the database is responding in a timely manner.

Improve this answer
2
  • This particular script does. But the server has gotten hung up on scripts that don't access any databases.
    – Josh
    Jul 21, 2009 at 22:29
  • In fact I can verify that this has happened on PHP scripts which simply read from /proc/loadavg
    – Josh
    Dec 9, 2009 at 16:49
0

David covers most of the essentials in his response.

Also, similar to the DB locking issue, could be a call on the backend to an external resource that's not responding or responding slowly. For example, if you had a CGI that did a LWP to example.net, Apache would wait until timeout or response. This can cause a resource exhaustion as well.

Improve this answer
0

How often do you ssh into the box? I had a problem, as weird as it sounds, that ssh would actually kill the apache service. I would ssh in and everything would just lock up. I haven't vpn'ed into the box in months and there has been no problem at all. It didn't happen everytime. But I know there was a correlation.

Improve this answer
1
  • I'm SSHed into this box almost all day long, and I'm SSHed into almost all our apache boxes often, so I don't think this is it...
    – Josh
    Apr 11, 2011 at 20:23
0

When dealing with any performance problem, you have to know the input. Configure your switch to mirror the webserver port to a sniffer server where you record all HTTP traffic. Try to correlate the traffic with the problem and try to reproduce the problem on a test machine.

Next step is to see what is happening with those processes. Use strace and see in which system call they are waiting. Use lsof to find out the file descriptors involved in the strace. ls -l /proc/<PID>/fd is useful too.

As chronos said, it could be a slowloris atack.

Improve this answer
2
  • It's happened to my own scripts, called from my Nagios server. I can verify that this has happened on PHP scripts which simply read from /proc/loadavg, so it's not a slowloris attack. The input for this script would be a simple GET /url/to/script HTTP/1.1
    – Josh
    Apr 11, 2011 at 20:50
  • And what are the processes doing when you run strace? Do you see the open system call for /proc/loadavg file? Do you have any errors in apache error log? Are the requests logged in access log?
    – Mircea Vutcovici
    Apr 11, 2011 at 20:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .