Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a needle in a haystack type problem I'm trying to track down. Have not been able to recreate it on test servers.

The symptom is TCP connections left in an ESTABLISHED state so don't seen to be closing.

The environment is PHP / Apache / Postgresql on CentOS

Under normal operations ESTAB DB connections is under 20 while active HTTPD process can climb into the 400 range.

In the failure scenario the ESTAB DB connections spike to the configured limit of 1000 while the HTTPD proc count doesn't change dramatically

To confirm that the TCP connections are related to HTTPD I have used netstat -nop and then count the number established to the DB.

At the same time I look at the number of active HTTPD with ps -ef

I haven't been able to catch the failure yet to try to strace one of the HTTPD.

Seems to me the TCP connections should be FIN'd down with the exit of the HTTPD. I've researched TPC orphan but that seems to only apply if the connection has been attempted to be closed.

Log files don't appear to have any activity related to this issue. I am working on setting the log level higher to see if more verbosity will give me some clues.

Has anybody else run into TCP connections not closing down when the owning process goes away?

share|improve this question
    
Is it apache or php that is connecting to the database? Is apache/php exiting cleanly or crashing? –  David Feb 17 '12 at 0:51
1  
Are you sure that the httpd that owns the connection is no longer running? how are you determining this? when you run netstat -plant as root, what is shown in the last column? –  stew Feb 17 '12 at 1:06
    
@David httpd owns the connection. I believe they are exiting cleanly. When it pegs the available connections it hovers around the limit. –  andy.morgan Feb 17 '12 at 3:53
    
@stew I have some netstat and PS output I'm going to look at where to confirm. I setup a script to grab output when the counts go high. I'll update after I confirm. Right now I'm assuming based on stats that I have gathered. I have been watching two numbers at 5 second intervals. One is "netstat -n | grep -i estab | grep -c 5439" to count database connections in ESTABLISHED state 5439 is the DB port. And also PS -EF | grep -ic httpd to count running HTTPD process. I've assumed when DB number is higher then HTTP number that HTTPD abandonded the connection and closed. –  andy.morgan Feb 17 '12 at 3:59
    
When the spike occures, what is happening on the DB side? Is one query/operation causing the others to block, causing the pile up to occur? If you sniff the network traffic of one of these phantom established sockets, does anything show up? –  David Mar 3 '12 at 3:16
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.