I'm running some benchmarks using apache bench for a java app that is running on tomcat.
Say I run a test like:
ab -c 10 -n 10000 http://localhost:8080/hello/world
It will run just fine. If I follow it with:
ab -c 50 -n 50000 http://localhost:8080/hello/world
Again it will run fine, but if I try again it starts to slow down after maybe 3500 completed requests.
I need help in terms of trying to debug the root cause of this.
I ran top, and I have a few gigs of memory that is unused so memory doesn't seem to be the issue.
The tomcat6 process does go to 70-80 or even 107%.
It seems restarting tomcat solves the issue, but at times a server reboot is required.
This is on a default tomcat install that has 200 threads allocated to it.
Tomcat logs are empty.
So I changed both tcp_tw_recycle/reuse to 1, and running netstat shows a very low count now.
Previous to changing tcp_tw_recycle/reuse, I noticed things slowing down and ran netstat and I had 32400 tcp TIME_WAIT connections.
So an update on running the benchmarks now, with the -k switch I'm seeing MUCH more throughput. BUT, at some point things again start to slow down, but restarting tomcat now brings things back to normal. Before, even if I restarted tomcat, response times running ab would be very very slow. Now after changing tcp_tw_recycle/reuse, restarting tomcat brings things back to normal. Running top shows tomcat at only around 20% of cpu, so it seems the problem is with tomcat now, but how can I figure out what?