0

I have a Cassandra cluster set up on 5 EC2 nodes. The nodes can communicate using the private IP addresses just fine, but get connection failures when using the private internal DNS name. Localhost also fails.

I am trying to use the cassandra-stress tool but I get the Failed to connect over JMX; not collecting these stats, and when turning on verbose logging I get the following stack:

java.lang.RuntimeException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: ip-172-1-2-2.eu-west-1.compute.internal; nested exception is:
    java.net.ConnectException: Connection refused (Connection refused)]
    at org.apache.cassandra.stress.util.JmxCollector.connect(JmxCollector.java:99)
    at org.apache.cassandra.stress.util.JmxCollector.<init>(JmxCollector.java:85)
    at org.apache.cassandra.stress.StressMetrics.<init>(StressMetrics.java:62)
    at org.apache.cassandra.stress.StressAction.run(StressAction.java:198)
    at org.apache.cassandra.stress.StressAction.runMulti(StressAction.java:126)
    at org.apache.cassandra.stress.StressAction.run(StressAction.java:72)
    at org.apache.cassandra.stress.Stress.main(Stress.java:109)

When I run cqlsh I have to specify the private IP address (./cqlsh 172.1.2.3) as using the default localhost fails.

Is there some sort of additional configuration that is required to map this internal name to localhost, or to get it to resolve the private IP addresses on each node?

EDIT:

Running host ip-172-1-2-3.eu-west-1.compute.internal gives the correct output:

ip-172-1-2-3.eu-west-1.compute.internal has address 172.1.2.3

So it looks like DNS is working correctly.

For my cassandra.yaml settings, I have listen_address commented out, but am using:

listen_interface: eth0
broadcast_address: 172.1.2.3
rpc_address: 172.1.2.3

Replication works fine with these settings. I'm not sure how to configure JMX to listen explicitly on the private IP address.

5
  • Please check the output of the host <hostname> command, to see if DNS resolution is working properly. Localhost is a different issue, as it maps to a different address, and server processes can choose to listen on specific addresses even if they are configured on the same host.
    – b0fh
    Mar 22, 2019 at 13:59
  • please add how the listen_address and rpc_address are configured. Also, have you configured JMX explicitly to listen on private IP? JMX & cqlsh are very different settings...
    – Alex Ott
    Mar 25, 2019 at 9:13
  • @b0fh I've added the information you asked for into my question. Mar 25, 2019 at 12:11
  • @AlexOtt I've added my cassandra.yaml settings to my question. Mar 25, 2019 at 12:12
  • Those addresses are wrong. It is probably either a typo or you have deliberately put fake information unnecessarily. Try using the correct addresses. Mar 25, 2019 at 13:45

1 Answer 1

0

Your command cqlsh 172.1.2.3 fails because you configured rpc_address to listen only on the given address. If you want to listen for CQL traffic everywhere, you need to set rpc_address to 0.0.0.0, and set broadcast_rpc_address to 172.1.2.3 - it's all described in the cassandra.yaml itself.

Regarding JMX - it's not really required that you had it enabled - it collects some data about garbage collection, etc., that maybe not necessary yet if you're just starting. I recommend to concentrate on testing itself, like, described in this article, for example.

But if you really want to expose JMX to outside, you need to look into cassandra-env.sh - it has detailed comments on what to do. You only need to make sure that you secured JMX, otherwise anybody can do almost anything with your cluster - wipe data, configure it, etc.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .