Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have experienced some really high server loads:

1:49pm up 2 days 1:51, 1 user, load average: 79.05, 101.35, 111.53

then the site crashes. I mean, the web don't load, I can't access by ssh nor ftp and we have to do a hard reset.

It happens randomly.

The crash hour error logs (something like this but repeated hundreds of times):

[Sat Jul 09 13:02:54 2011] [notice] child pid 1966 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:57 2011] [notice] child pid 1967 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:57 2011] [notice] child pid 1969 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:57 2011] [notice] child pid 1970 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:57 2011] [notice] child pid 1971 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:57 2011] [notice] child pid 1972 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:57 2011] [notice] child pid 1973 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:57 2011] [notice] child pid 1974 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:58 2011] [notice] child pid 1976 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:58 2011] [notice] child pid 1977 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:58 2011] [notice] child pid 1978 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:58 2011] [notice] child pid 1979 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:58 2011] [notice] child pid 1980 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:58 2011] [notice] child pid 1981 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:58 2011] [notice] child pid 1982 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:58 2011] [notice] child pid 1983 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:58 2011] [notice] child pid 1984 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:58 2011] [notice] child pid 1985 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:58 2011] [notice] child pid 1986 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:58 2011] [notice] child pid 1987 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:58 2011] [notice] child pid 1988 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:58 2011] [notice] child pid 1989 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:58 2011] [notice] child pid 1990 exit signal Segmentation fault (11)
[Sat Jul 09 13:02:58 2011] [notice] child pid 1991 exit signal Segmentation fault (11)
[Sat Jul 09 13:03:16 2011] [notice] child pid 1992 exit signal Segmentation fault (11)
[Sat Jul 09 13:03:17 2011] [notice] child pid 1993 exit signal Segmentation fault (11)
[Sat Jul 09 13:03:21 2011] [notice] child pid 1994 exit signal Segmentation fault (11)
[Sat Jul 09 13:03:21 2011] [notice] child pid 1995 exit signal Segmentation fault (11)

The crash hour access log:

::1 - - [09/Jul/2011:12:54:07 +0200] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.10 (Linux/SUSE) (internal dummy connection)"
::1 - - [09/Jul/2011:13:38:48 +0200] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.10 (Linux/SUSE) (internal dummy connection)"

We don't a high amount of users so this is a little bit extrange. Where can I find some info about the crash?, We have a LAMP architecture.

Thanks in advance

share|improve this question
    
Run "top" and report what the value of wa% is in the top summary area. –  EEAA Jul 9 '11 at 14:51
    
You've got two separate problems here and are conflating the two. The child segfaults are almost certainly not the cause of your high load and server crash. Work on the two problems separately. –  womble Jul 9 '11 at 22:21
    
@ErikA now if I run top I get 0.0% at wa%. I'll try to run it next time the server increases it's load –  LuisClemente Jul 10 '11 at 9:49
add comment

2 Answers

Is it happening at a regular time? Or is it random?

Did you check the access and error logs on the web server?

Did you look at iostat to see if the load is produced by disk access or network access or...?

Does Top tell you if a particular process is eating resources?

Does the system log tell you anything?

When you say crash, what do you mean? Kernel panic? Application has a fault? The server just stops responding? The console works but the network doesn't?

We need a lot more information to even hazard a guess. You need to narrow down when it's happening, if there's a pattern to when it happens, and what subsystem it is occurring from.

share|improve this answer
    
Thanks! I edited with some new data. We don't have any special on our system log and I've just installed iostat to have more info next time. –  LuisClemente Jul 9 '11 at 14:10
add comment

Configure Apache to make it generate a core dump on segfault. After that, try to debug with gdb to see what causes.

share|improve this answer
    
Ok, good idea. I'll do it –  LuisClemente Jul 10 '11 at 9:51
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.