Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

My setup: i have 3 nearly identical webserver machines serving the same high loaded dynamic website with simple load balancing over dns. The service has been working for over two ears with the same apache config. apache2, php5, ubuntu 8.04 linux 2.6.24-29-server

My problem: since about two weeks i'm experiencing problems with this config. Nearly every day i have one small moment about 5 minutes, in which the website is unreachable. I'm still able to login to the servers over ssh. If i run htop, i see the machine simply doing nothing. i have about 1000 apache processes running, but no cpu activity.

i've used the apache mod_status to debug this situation. the process scoreboard looks like this:

_C.___K_______________________R._______.__K_K____K___C_______.__
_______C__________.___________________________________.________C
_.____K__________K___K_WK_____._K_____________________________._
W______K__________K________.____________________._______C_______
_C_.__K__K____.._.._____________________________________C_______
_R___________K___.______C________.C_________.______._____C______
____________KKC____K_____K__WC_________________C_____.__.____.__
_____________________C_________K______.____C______._____________
_.___C____.___.___________________________.K______.____K________
W__.___________________C.__.____K________K_______R_._.__._______
__C__C_.__________C__C_______._____W______________C_.___C_______
____.______C_____________C________.____C____________.________._K
__.__________.K_____________K_________._____C____.K__________KW_
__K.W________R_________._______.___W___________.____.__K_____W__
W___.___..________W____K

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

So the most of the processes are just waiting for connection. after about 5 minutes the situation will return to normal: i have lot least processes on every machine, the most workers have the "."-status (meaing they are open to process a request) and of course the website is reachable!

so i'm trying to find something in the logs, but there is simply nothing... the apache access log is silent for about 4 minutes, the same is for the error log. i also can not figure out anything wrong in other system logs.

the situation is the same on all 3 webservers (all of them have this load peak and unresposibility at the same time), so i do not thing this is hardware related. but i think, this might be related to some network (tcp) issue.

any ideas?

EDIT: some more information, that i have just discovered:

it has just happened again. and i was able to verify that i'm also not able to connect locally when this problem occurs. i have made some connection statistics with the following command after it happend netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c

  • 109 CLOSE_WAIT
  • 2652 ESTABLISHED
  • 2 FIN_WAIT1
  • 11 LAST_ACK
  • 12 LISTEN
  • 91 SYN_RECV
  • 1 SYN_SENT
  • 16 TIME_WAIT

If i execute the same command some time later, i have something like this:

  • 4 CLOSING
  • 108 ESTABLISHED
  • 18 FIN_WAIT1
  • 182 FIN_WAIT2
  • 37 LAST_ACK
  • 12 LISTEN
  • 50 SYN_RECV
  • 11276 TIME_WAIT

So in the normal situation i have only 100-200 open connections by clients beeing handled by apache in this moment. when i have this "crash", i have a lot more connections. what is the best way to analyse this?

EDIT2: the important lines in apache2.conf are:

KeepAlive On
MaxKeepAliveRequests 20
KeepAliveTimeout 1
<IfModule mpm_prefork_module>
ServerLimit           920
StartServers          30
MinSpareServers       80
MaxSpareServers      120
MaxClients          920
MaxRequestsPerChild   700
</IfModule>

it is an apache2 prefork with php_mod.

the server has 8GB ram and a 4gb swap partition.

share|improve this question
    
Does the web site show the same symptoms when you run a wget or curl from the local host or between servers (if they're on the same network)? –  Alex Forbes Jan 31 '12 at 14:16
    
Maybe a traffic dump (tcpdump) will help you get to the root of the problem ... btw what's your memory usage and firewall policies ? –  drcelus Jan 31 '12 at 14:34
    
@al4 the last time this happend i've been able to connect to the server-status page from the local host, while i was unable to connect to the webpage from outside. i'm not quite sure, as it could also be a random thing, while some of the workers became available. i will test this more the next time the problem occurs. what would be your suggestion, if i could confirm any difference between outside and local connections? –  Jeff Jan 31 '12 at 14:57
    
If you can confirm that it works locally but not from outside it strengthens the case for the network being the issue - meaning you should test with tcpdumps and wireshark at both ends to see what's getting through, rather than strace'ing the apache processes. I'd also test from a host on the same LAN if possible. And check dmesg to see if there are any messages that could be related but sounds like you've already done that. –  Alex Forbes Jan 31 '12 at 16:30
    
it has just happened again. and i was able to verify that i'm also not able to connect locally when this problem occures. i have also made some connection statistics with netstat: see question text –  Jeff Feb 1 '12 at 14:16

2 Answers 2

Also, bear in mind that in the prefork MPM, each process will have PHP in its memory space (what's its memory limit setting?). You may like to try changing to the worker MPM, which may require a slightly different PHP module.

Also worth remote earring to trim your Apache config of extraneous modules

In my experience, such things are triggered by things like a search engine crawler, or things like ARP conflicts. Or traffic levels in some related part of the network.

You may well find 'sar' useful... not the most friendly, but certainly useful.

Possibly also io related. Sar can tell you (if you configure it to record disk activity), what the average io wait time is. You can also look at the IO Wait time in top (which is a percentage, read up on what it actually means). This can be significant if you are using a SAN or virtual environment.

share|improve this answer

Show your apache MPM settings, and keepalive settings.

It is probably a bad combination of these.

EDIT: I just saw you mentioned php.

If this is mod_php you're using, that machine better have 64GB of memory, or you will never sustain 2500 connections.

share|improve this answer
    
i have added the apache config to the question. i do not know how it could affect the accepting of connections, but the MaxClients setting is really set to high. i have checked this: my apache workers take usually 15-30 MB RAM. Do you think, the higher setting could have such effect? –  Jeff Feb 1 '12 at 16:55
    
i have reduced the ServerLimit and MaxClients to 500. since then the problem occured two times again. for about 4 minutes i have no activity in my apache log on all three servers and the website is not reachable! so the ServerLimit was not the point... –  Jeff Feb 3 '12 at 13:49

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.