1

I have a Linux server which I have installed Java on.

Sometimes, and only sometimes when a large number of visitors visit my website, the site hangs.

When I open the terminal and enter the "top" command to see whats going on, I can see that "Java" process is eating CPU! Like 400%.

I have also tried ps aux command, and can see that the command is from usr/bin/java

I have little experience in troubleshooting this kind of things, so I turn to you guys for help.

I have a java container installed (Jetty) which I must have in order to use SOLR (search engine) which is integrated into my website.

I can start and stop SOLR by:

  etc/init.d/solr stop

But this didn't remove the java process from the "Top" command. Still java was eating 400% CPU.

Is there other methods to restart java only?

This has happened twice to me, and each time I have now restarted my entire servers and everything is fine.

If you need more input let me know!

3
  • Java eats CPU. That's a given. As for why it is so, just because a process is eating up 100%, doesn't make it a bad application. It's using all it has available. That said, perhaps there's a broken configuration somewhere.
    – user3914
    Jan 4, 2011 at 18:58
  • Broken configuration? I am asking how to troubleshoot here. Could you explain some tips? Jan 4, 2011 at 19:02
  • See @Tom Purl's answer, which details what I'm referring to: "reconfiguring solr and jetty at the very least".
    – user3914
    Jan 5, 2011 at 0:01

3 Answers 3

3

If you stop the solr process and a java process is still running, then there's another java process on your server. The first step is to document all of the Java processes that are running. A good tool for this on Unix is the ps tool. Try this:

$ ps auxwww | grep java

That output should show you all of the java processes running, and the commands that are being executed. Try this before and after you stop solr.

The second question is "which of these java processes is eating up so much CPU"? You may need to stop the jetty process in addition to the solr process. Also, you should never really need to restart a Linux server just because a single process is misbehaving. You can also use the kill command to stop a process if you now it's process id, which you can get from either top or ps.

In the short term it may be a good idea to install some sort of "watchdog" script on your machine to help with these situations. For example, monit can be used to automatically restart a service or process when it consumes a certain amount of CPU resources.

In the long term, I'm sorry to say that you have a performance issue. You need to look at reconfiguring solr and jetty at the very least. You may also need to look into garbage collection tuning and possibly adding more hardware. There's lots of information on these topics online, and I'm sorry to say that this process can be somewhat difficult.

Good luck!

Tom Purl

1
  • Nice answer. +1 for you.
    – user3914
    Jan 5, 2011 at 0:02
2

If the CPU is getting maxed out while response times are dropping, then the first place I'd be looking is the heap management within the java - it's probably doing lots of major collections - and the quickest way to solve the problem is to just give the container more memory to run in.

-1

This isn't really the right place to ask program debugging question. If Solr is the only Java application running you should definitely check how often is your Lucene index being updated and how many search queries are you running per second when the server starts gets overloaded.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .