Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a weird issue. Every time I stop Tomcat, SSH becomes very slow, almost totally unresponsive. It takes a minute or even more for SSH to accept any command. When I finally get Tomcat started again everything goes back to normal.

This is happening on a very busy server. Problem is that I need to stop the Tomcat often because of the application upgrades. It would normally take few seconds to upgrade, but here it takes almost 10 minutes and because of that we are experiencing unwanted downtimes.

One thing that I see is when I stop Tomcat top shows a lot of ksoftirqd/X processes at 100% CPU. Could this be the issue?

Kernel version is: 2.6.18-308.11.1.el5

Red Hat version is: Red Hat Enterprise Linux Server release 5.9 (Tikanga)

Any idea why this is happening?

share|improve this question
    
Not an idea per se, but what you could try to investigate further is to check why the ksoftirqd kernel-threads are that busy. I suggest to install atop, run as atop -l and filter for ksoftirq (press "P" and give "ksoftirq" as filter). Then observe RDDSK and WRDSK, as well the state of the process (S) - is it non-interuptable - D - for instance? Also you might as well have a look at your io-subsystem using iostat to check for possible IO-bottlenecks. –  Alexander Janssen Mar 20 '13 at 13:48
3  
This is a shot in the dark here, but maybe set the firewall to drop all requests to the Tomcat port before you stop Tomcat. –  user1936123 Mar 20 '13 at 21:10
    
Did you know when this "slow ssh" issue starts? Is something new in server environment? –  dsznajder Mar 24 '13 at 11:59
1  
Solution from @user1936123 worked. I haven't tried any other solution because we can not afford downtimes and it is very hard to test anything when console is almost unresponsive. –  Paxxil Mar 28 '13 at 8:37
    
Smells like your app has a leak. You can get what all the interrupts are for by doing cat /proc/interrupts –  user May 10 '13 at 16:31
add comment

2 Answers

Slow ssh connection / ssh lag is a symptom of a high load. High load is often caused by io blocking, which is often caused by swapping.

to check your load, run uptime or top you will probably see load numbers over 10 when ssh is not responding. They will probably hover under 2 during normal use.

run free or top to see your memory usage, you will probably see a lot of swap.

Once you find out the root symptoms, you can then search for why is "tomcat swapping on shutdown" or "tomcat high load" which is probably because it's trying to write some things stored in cache or swap to disk. Is your tomcat jvm max heap size larger than the amount of memory you have?

Maybe something connecting to your webapp retries constantly when it goes down creating DoS scenario.

This could all be specific to your webapp so use general terms.

share|improve this answer
add comment

I know that this isn't the "best practice", but, i would suggest you to reset tomcat remotely through ssh and set the outputs to null:

ssh your_server '/etc/init.d/tomcat restart > /dev/null 2>&1'

(You could replace the command above with the equivalent you use to restart tomcat).

This is a workaround, not a solution. Could you try this while connected on other ssh session and check if the problem still occurrs and affects all the sessions?

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.