mod_jk problem: Tomcat is probably not started or is listening on the wrong port

Question

I am running some application on Tomcat 6.0.26. There is Apache in front of web server talking to it over mod_jk. Every few hours when I try to access application browser simply spins, and no content is retrieved.

No error is reported in Tomcat logs, but I fond such errors in mod_jk log:

[Sun Jul 04 21:19:13 2010][error] ajp_service::jk_ajp_common.c (1758): Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. worker=***** failed
[Sun Jul 04 21:19:13 2010][info]  jk_handler::mod_jk.c (1985): Service error=0 for worker==*****
[Sun Jul 04 21:19:13 2010][info]  ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 46
[Sun Jul 04 21:19:13 2010][info]  ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 46
[Sun Jul 04 21:19:13 2010][info]  ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 46
[Sun Jul 04 21:19:13 2010][error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Jul 04 21:19:13 2010][error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Jul 04 21:19:13 2010][info]  ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 46
[Sun Jul 04 21:19:13 2010][error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Sun Jul 04 21:19:13 2010][info]  ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 45
[Sun Jul 04 21:19:13 2010][info]  ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 46
[Sun Jul 04 21:19:13 2010][info]  ajp_service::jk_ajp_common.c (1721): Receiving from tomcat failed, recoverable operation attempt=0

my worker is configured in following way:

worker.admanagonode.port=8009
worker.admanagonode.host=*****.com
worker.admanagonode.type=ajp13
worker.admanagonode.ping_mode=A
worker.admanagonode.socket_timeout=60
worker.admanagonode.prepost_timeout=10000
worker.admanagonode.connect_timeout=10000
worker.admanagonode.connection_pool_size=200
worker.admanagonode.connection_pool_timeout=300
worker.admanagonode.retries=20
worker.admanagonode.socket_keepalive=1
worker.admanagonode.cachesize=10
worker.admanagonode.cache_timeout=600

Tomcat has same port number in Connector configuration:

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" address="*********" />

Does any of you has any ideas what i am missing? What can cause such problems?

Cheers Konrad

Answer 1

It appears that no response is being sent Tomcat before the timeout period. This could be an application issue. What I would do in this situation is have some sort of monitoring set up (be it Nagios, or something far simpler) to make sure on a frequent interval that:

• Port 8009 on the application server is accepting incoming connections.
• Port 8009 on the application server is accepting AJP/1.3 requests at all.

If you graph the historical of these tests you might find an issue with your application and/or application server having periodic outages, which will eventually lead you to a root cause. Best of luck... intermittent issues are the worst.

Answer 2

Is this a busy site? Maybe you are just reaching maxThreads.

Answer 3

thanks for all your support, I really appreciate it!

It was actually, as some of you suggested - application issue. This was caused by incorrect configuration of MySQL connection pooling (typo in timeout - so value was having one 0 to much). So application was running out of Connections to DB, as those were never timing out.

Fixing pooling configuration (in my case c3p0) resolved issue!

Thanks once again!

Konrad

Answer 4

worker.admanagonode.host=*****.com

Is your machine on a NAT, and that hostname resolves to an external IP? Typically you leave that to 'localhost' (default) unless you're specifically backending to an off-machine Tomcat running in the same network (so there's an /etc/hosts for it) and so forth.

Same goes for your Connector address from server.xml = do you have that set to something other than 127.0.0.1? And why?

Answer 5

Try a simple test, comment all the timeouts, retries and other optionally stuff:

worker.admanagonode.port=8009
worker.admanagonode.host=*****.com
worker.admanagonode.type=ajp13

And test this config. I think one of this timeout options should be responsible for the problem.