5

I just completed a migration from Windows server 2008 R2 to Ubuntu 10.04. I manage a java application (Java 6, Tomcat) that is having some performance issues. I'd like to use JMX to try and troubleshoot, but I can't seem to get jvisualvm to connect.

If I do a ps -ef | grep "java", I see the following parameters.

-Dcom.sun.management.jmxremote.port=8084 
-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=false

Netstat shows that port 8084 is listening on 0.0.0.0. In my config, JMX is setup to bind to the FQDN of the server (we use a private DNS server). My firewall (IPTABLES/UFW) is setup to allow all outgoing traffic, and to allow incoming traffic on port 8084.

The server itself is virtual with two NICs, a public and a private. The public NIC's gateway is disabled so that connections can only come in on the private side.

When I try to connect jvisualvm to my app server using JMX, I get the following error in jvisualvm.

Cannot connect to [FQDN OMITTED]:8084 using server:jmx:rmi:///jndi/rmi://[FQDN OMITTED]:8084/jmxrmi

If I look in the jvisualvm log, I see the following trace.

NFO [com.sun.tools.visualvm.jmx.impl.JmxModelImpl]: connect(service:jmx:rmi:///jndi/rmi://[FQDN OMITTED]:8084/jmxrmi)
java.io.EOFException: SSL peer shut down incorrectly
                at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:333)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
Caused: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:808)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1120)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:623)
                at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
                at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
                at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
                at java.io.DataOutputStream.flush(DataOutputStream.java:106)
                at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:211)
Caused: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
                javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
                at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:286)
                at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
                at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
                at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
                at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
Caused: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
                javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]
                at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
                at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
                at javax.naming.InitialContext.lookup(InitialContext.java:392)
                at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
                at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
                at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257)
Caused: java.io.IOException: Failed to retrieve RMIServer stub
                at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
                at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.tryConnect(JmxModelImpl.java:451)
[catch] at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.connect(JmxModelImpl.java:395)
                at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.connect(JmxModelImpl.java:216)
                at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.<init>(JmxModelImpl.java:205)
                at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:61)
                at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:42)
                at com.sun.tools.visualvm.core.model.ModelFactory.getModel(ModelFactory.java:111)
                at com.sun.tools.visualvm.tools.jmx.JmxModelFactory.getJmxModelFor(JmxModelFactory.java:69)
                at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.addJmxApplication(JmxApplicationProvider.java:267)
                at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.createJmxApplication(JmxApplicationProvider.java:185)
                at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationImpl(JmxApplicationsSupport.java:283)
                at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationInteractive(JmxApplicationsSupport.java:261)
                at com.sun.tools.visualvm.jmx.impl.AddJMXConnectionAction$1.run(AddJMXConnectionAction.java:80)
                at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:577)
                at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1030)

Does anyone have any ideas?

6

You can specify same port for both JMX and RMI conversation:

-Dcom.sun.management.jmxremote.port=8084 
-Dcom.sun.management.jmxremote.rmi.port=8084 
3

The problem is that even if JMX is configured to listen on port 8084, after the initial connection is made, the JMX host will dynamically pick a different port for the rest of the "conversation". My firewall allowed traffic through 8084, but was blocking the subsequent traffic.

There are two options.

  1. Allow inbound traffic on all ephemeral ports (yikes!).
  2. Develop a JMX wrapper that uses a single port for communication.
  • 1
    If you are on Java 7 update 4 or later there is hope with the flag `` -Dcom.sun.management.jmxremote.rmi.port=7091`` See this blog post: hirt.se/blog/?p=289 – davey Oct 9 '13 at 21:55
1

I struggled with this same issue trying to connect to our new Tomcat8 VMs over JMX (using VisualVM). Our Linux team had to open a ticket with Redhat to help us troubleshoot. But I stumbled on this article before they could resolve our issue.

The solution as mentioned above was to add

-Dcom.sun.management.jmxremote.rmi.port=X

and X should match whatever your setting is for

-Dcom.sun.management.jmxremote.port 

Complete List of Args:

-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port=8091
-Dcom.sun.management.jmxremote.rmi.port=8091
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false 
-Djava.rmi.server.hostname=MYHOST

Hope this helps someone else having the same issue. Good luck!

0

I have same problem too on tomcat 9.0.9. on path/to/tomcat/bin/setenv.sh, following configuration worked for me:

CATALINA_OPTS="$CATALINA_OPTS 
-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port=1099 
-Dcom.sun.management.jmxremote.rmi.port=1099 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false 
-Djava.rmi.server.hostname=192.168.1.20 
-Djmx.rmi.registry.port=1099 
-Djmx.rmi.port=1099 
" 

export CATALINA_OPTS

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.