Two questions:

First, what "java skills" do you require for a sysadmin administering servers-running-java-apps? e.g.: take stack traces, take heap dumps, monitor jvm, etc.

Second, what tools do such sysadmins find helpful. (For example, an app to take a stack trace of all running jvm's might be helpful).

[Edit: on windows, at least, its sometimes hard to find out which java.exe is which. Thus a stack trace for 'all jvms' would simplify the sysadmins task, rather than hunting them down. ]

Our sysadmins have windows backgrounds. We would like to shrink the gap between the developers and the admins, so sysadmins can act as first lines of defense should the java apps hang/use resources/etc.

What works for you?

  • Might be useful to add "Windows" as a tag to this as I was about to answer but am limited to experience on *NIX systems. – DaveG Oct 14 '10 at 20:23
  • fwiw, In response to closing this question as 'off-topic': our organization has faced this problem as sysadmins trained on MS-Windows push back on learning "jvm admin skills" because "they're Windows Admins" and "they're not programmers". This question seems a better fit on serverfault. than the other stackxchange sites. (e.g. superuser or stackoverflow). – user50460 Feb 27 '12 at 16:09

I would say a JVM admin needs to know how to at least:

  1. Able to install any given (sub)version of the runtime, startup JVM with recommended parms,
  2. Adjust runtime parms and directives for necessary app logging, (i.e. enable GC logging)
  3. Know filesystem-level security for multiple runtimes on single OS instance.
  4. Workload management (CP affinity, JVM ergonomics)
  5. Virtual network interfaces,
  6. Normalize process utilization against system utilization.
  7. Enable/disable profiling
  8. jstat/jmap &c

Someone else has already mentioned knowing how to generate heap dumps.

| improve this answer | |

Couple of useful tools:

Taking thread dumps (kill -HUP on Linux systems ... not sure what the equivalent mechanism on Windows is) will give you a snapshot of the state of the application - similar to Heisenberg's uncertaintity principle (no really :-)) it will tell you the state of the system right now but won't tell you what it's about to do or had previously just done.

Thread dumps are good because they just print the output to STDERR (or appropriate log) for the application and don't cause it to lock up at all.

Taking heap dumps is useful to find out the state of the data within the application - you can use a tool such as jmap (part JDK/JRE distributions) to take a heap dump. The key issue with this though is that in order to take a heap dump it needs to pause the JVM while it's doing so. This causes the application to essentially lock up to the end-user. So you probably don't want to do this unless things are really screwed up.

Thirdly, you can use Java Management Extensions to be able to monitor the internal running state of your application's CPU, memory, threads and classes states. This can also allow the developer of the Java application to expose internal statistics about the application which can also be monitored - for example using Nagios' check_jmx utility.

Lastly, you obviously have the standard system tools which you can get to monitor CPU, disk I/O, memory utilisation, etc. etc. which can all give you an insight into the overall health of your system and whether or not it's in a "nice" state or not.

| improve this answer | |
  • fwiw, there ain't no equivalent of "kill -HUP" on windows. Nor 'ps ax | grep java'. Nor storing pids in files for easy access later. But the UI demos well. – user50460 Oct 14 '10 at 21:55
  • Get process explorer from sysinternals to give you a decent process listing as well. – DaveG Oct 15 '10 at 11:50

I think the other answers cover most of the bases, but in terms of specific skills and tools to use these stand out as a starting point.

For immediate inspection of running threads, and selecting Java process from all those available ones, the java.net VisualVM tool is pretty excellent.

There is an intuitive tree view local and remote (if you enable remote debugging on your app) of the running Java threads and processes;

And you can use it to save a snapshot of a running process which you can then analyse at your leisure;

enter image description here

and you can examine what your thread state it, or take a profile and see what methods your app spends all its time in;

enter image description here

In addition there are a bunch of plugins for specific integrations and there is an eclipse plug-in.

Check out some more screen shots for cool stuff - http://visualvm.java.net/

understand the classloader

I think I caused myself a lot of pain in the early days by not putting the effort into understand how the JVM works, particularly the classloader and the container lifecycle. And a lot of the weird things with Generics and parametized classes makes a lots more sense if you what strategy the the JVM is using.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.