we have apache 2.2 with mod_perl2 connecting to an oracle SE 11g db, the problem is that on apache restart we sometimes get a lot of W states on all the children and for a long time (from 1 minute to 20 minutes or more) ... these seem to be cause due to the oracle db connection. We traced the W states problem to the oracle db connection by using strace and lsof ... strace shows that the apache child process hangs at a read syscall like so: "read(18," checking with lsof on those apache processes, the number 18 handle for that process is an ESTABLISHED connection to the oracle db server.
also, while checking with tcpdump (filtered by remote port and oracle db port) there seems to be no communication through that channel.
we're using DBD::Oracle latest version from cpan and also Apache::DBI.
Also, we can connect from the webservers through sqlplus successfully while this is happening.
This seems to only happen on apache restarts and happens from time to time on all webservers (we have Gentoo and centos webservers). If for example I stop the apache daemon and wait a while ( I also check that all connections to the oracle db server are gone from netstat -anp ) and then start the server up again it seems to start fine.
At some point we figured it's because of the TIME_WAIT states of the connections to oracle db after the apache restart but we tried setting the /proc/sys/net/ipv4/tcp_tw_recycle and tcp_tw_reuse parameters and it did not fix the problem.
Does anyone have any other ideas of how we should proceed with debugging or solving this problem?
