I have a needle in a haystack type problem I'm trying to track down. Have not been able to recreate it on test servers.
The symptom is TCP connections left in an ESTABLISHED state so don't seen to be closing.
The environment is PHP / Apache / Postgresql on CentOS
Under normal operations ESTAB DB connections is under 20 while active HTTPD process can climb into the 400 range.
In the failure scenario the ESTAB DB connections spike to the configured limit of 1000 while the HTTPD proc count doesn't change dramatically
To confirm that the TCP connections are related to HTTPD I have used netstat -nop and then count the number established to the DB.
At the same time I look at the number of active HTTPD with ps -ef
I haven't been able to catch the failure yet to try to strace one of the HTTPD.
Seems to me the TCP connections should be FIN'd down with the exit of the HTTPD. I've researched TPC orphan but that seems to only apply if the connection has been attempted to be closed.
Log files don't appear to have any activity related to this issue. I am working on setting the log level higher to see if more verbosity will give me some clues.
Has anybody else run into TCP connections not closing down when the owning process goes away?