I have a Cassandra cluster set up on 5 EC2 nodes. The nodes can communicate using the private IP addresses just fine, but get connection failures when using the private internal DNS name. Localhost also fails.
I am trying to use the cassandra-stress tool but I get the Failed to connect over JMX; not collecting these stats
, and when turning on verbose logging I get the following stack:
java.lang.RuntimeException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: ip-172-1-2-2.eu-west-1.compute.internal; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]
at org.apache.cassandra.stress.util.JmxCollector.connect(JmxCollector.java:99)
at org.apache.cassandra.stress.util.JmxCollector.<init>(JmxCollector.java:85)
at org.apache.cassandra.stress.StressMetrics.<init>(StressMetrics.java:62)
at org.apache.cassandra.stress.StressAction.run(StressAction.java:198)
at org.apache.cassandra.stress.StressAction.runMulti(StressAction.java:126)
at org.apache.cassandra.stress.StressAction.run(StressAction.java:72)
at org.apache.cassandra.stress.Stress.main(Stress.java:109)
When I run cqlsh
I have to specify the private IP address (./cqlsh 172.1.2.3
) as using the default localhost fails.
Is there some sort of additional configuration that is required to map this internal name to localhost, or to get it to resolve the private IP addresses on each node?
EDIT:
Running host ip-172-1-2-3.eu-west-1.compute.internal
gives the correct output:
ip-172-1-2-3.eu-west-1.compute.internal has address 172.1.2.3
So it looks like DNS is working correctly.
For my cassandra.yaml settings, I have listen_address
commented out, but am using:
listen_interface: eth0
broadcast_address: 172.1.2.3
rpc_address: 172.1.2.3
Replication works fine with these settings. I'm not sure how to configure JMX to listen explicitly on the private IP address.
host <hostname>
command, to see if DNS resolution is working properly. Localhost is a different issue, as it maps to a different address, and server processes can choose to listen on specific addresses even if they are configured on the same host.listen_address
andrpc_address
are configured. Also, have you configured JMX explicitly to listen on private IP? JMX &cqlsh
are very different settings...