I run a number of tomcat instances and occasionally some just stop reponding to requests - timeout on every connection.
I'm using AJP with mod_proxy in Apache 2.2.
I get a timeout via Apache/AJP through Tomcat's AJP connector, but also via the direct HTTP connector on 8080.
/server-status configured within Apache and it shows 16 requests currently being processed with
W, 4 idle requests and 200+ open slots with no connection. My AJP connector is configured as:
<Connector port="8009" address="localhost" maxThreads="250" minSpareThreads="5" maxSpareThreads="15" connectionTimeout="1000" packetSize="16384" maxHttpHeaderSize="16384" enableLookups="false" redirectPort="8443" emptySessionPath="true" URIEncoding="UTF-8" protocol="AJP/1.3"/>
so it should have plenty of threads to accept new connections.
top I see CPU and wait both under 1% and the java process has 80% memory. There is 60M free mem and 200M free swap.
I set up a special
threads.jsp page using
SystemThreadList stl = new SystemThreadList(); Thread allThreads = stl.getAllThreads();
which gives useful information, but in this state - it doesn't load either.
In catalina.log I see:
Mar 07, 2014 11:53:09 AM org.apache.jk.common.ChannelSocket processConnection WARNING: processCallbacks status 2
and occassional activity from other web requests, but not mine.
Is there a way from the command line, or using a profiler to get a list of threads and stack traces to find out what is getting stuck?