Restart tomcat on two different servers at same time

Question

We are running the Apache solr on two servers. Unfortunately on heavy load of indexing (guess) servers going down. We are manually restarting the tomcat each time.

When I google I find the way two restart tomcat automatically using vm options

-XX:OnOutOfMemoryError="<cmd args>;
<cmd args>" 

So I believe this below command will restart the tomcat when every we getting the out of memory

-XX:OnOutOfMemoryError="/etc/init.d/tomcat restart"

But how can I sync this process to another server. both servers are in sharding.

Can anyone help me on this?

Answer 1

Instead of directly running tomcat restart from OnOutOfMemoryError trigger, you may run a simple script that will do it on local server and ssh to the remote to do the same.

Answer 2

i would add a trigger to some monitoring checks, that just runs a script that restarts the tomcat on both systems. And maybe doing some sanity checks before and after the restart.

You can write those scripts for example with Fabric (http://www.fabfile.org/) or Rex (http://www.rexify.org/).

Also i would try to figure out why the tomcat freeze. If it runs out of memory, maybe you can give the jvm more memory?

If you need to monitor the internal memory usage of the jvm you can use http://www.jolokia.org/ to integrate it into nagios or other monitoring tools.