0

I have a standard 3 node MongoDB replica set:

  • 10.0.2.35 - Primary
  • 10.0.3.169 - Secondary
  • 10.0.1.48 - Secondary

I'm currently not able to connect to them as a replica set, I can only connect through the primary. If I run rs.status() on the primary, I repeatedly get:

{
        "set" : "ecReplica",
        "date" : ISODate("2018-04-23T19:12:10.014Z"),
        "myState" : 1,
        "term" : NumberLong(-1),
        "heartbeatIntervalMillis" : NumberLong(2000),
        "members" : [
                {
                        "_id" : 0,
                        "name" : "ip-10-0-3-169:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 10717677,
                        "optime" : Timestamp(1524510722, 15),
                        "optimeDate" : ISODate("2018-04-23T19:12:02Z"),
                        "lastHeartbeat" : ISODate("2018-04-23T19:12:08.186Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-23T19:12:09.656Z"),
                        "pingMs" : NumberLong(1),
                        "syncingTo" : "ip-10-0-2-35:27017",
                        "configVersion" : 405240
                },
                {
                        "_id" : 1,
                        "name" : "ip-10-0-1-48:27017",
                        "health" : 0,
                        "state" : 6,
                        "stateStr" : "(not reachable/healthy)",
                        "uptime" : 0,
                        "optime" : Timestamp(0, 0),
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2018-04-23T19:12:09.116Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-23T19:12:08.404Z"),
                        "pingMs" : NumberLong(0),
                        "authenticated" : false,
                        "configVersion" : -1
                },
                {
                        "_id" : 2,
                        "name" : "ip-10-0-2-35:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 10717680,
                        "optime" : Timestamp(1524510722, 15),
                        "optimeDate" : ISODate("2018-04-23T19:12:02Z"),
                        "electionTime" : Timestamp(1524486537, 1),
                        "electionDate" : ISODate("2018-04-23T12:28:57Z"),
                        "configVersion" : 405240,
                        "self" : true
                }
        ],
        "ok" : 1
}

When I ssh into the primary, I see the following error in /var/log/mongodb/mongod.log:

2018-04-23T19:23:54.326+0000 I REPL [ReplicationExecutor] Error in heartbeat request to ip-10-0-1-48:27017; Unauthorized not authorized on admin to execute command { replSetHeartbeat: "ecReplica", pv: 1, v: 405240, from: "ip-10-0-2-35:27017", fromId: 2, checkEmpty: false }

Additional info

Connectivity

I can connect to all 3 nodes individually using Mongo Shell and Robo3T using SSH tunneling, but I can't connect to the 3 as a replica set.

Production servers apparently connects successfully to the replica set.

telnet

telnet 10.0.1.48 27017 from 10.0.2.35 works.

/etc/mongod.conf

The config files are almost exactly equal, the only difference is in the net section:

Node 10.0.1.48:

# network interfaces
net:
  port: 27017
  bindIp: [127.0.0.1,10.0.3.169,10.0.2.35]

Node 10.0.3.169:

# network interfaces
net:
  port: 27017
  bindIp: [10.0.1.48,10.0.2.35,127.0.0.1]

Node 10.0.2.35:

# network interfaces
net:
  port: 27017
  bindIp: [127.0.0.1,10.0.3.169,10.0.1.48]

NOTICE: the security section is empty, so this is not a key-file issue.

db.version()

3.2.0

Infrastructure

All nodes are running in the same AWS VPC, while they're in different Availability Zones, they belong to the same Security Group and use the same Network ACLs and Route Tables.


This is an inherited setup, it has been live for more than 2 years.

What am I missing?

  • If you want a replica set connection (with member discovery and failover) as opposed to a direct connection to a single member of a replica set, your remote client needs to be able to connect to the replica set using the host names and ports as configured in rs.conf(). Since ssh tunnelling works I suspect you are trying to connect via forwarded ports. Can you confirm how your remote client is connecting to the replica set? – Stennie Apr 24 '18 at 2:10
  • NOTE: the bindIp values determine which network interfaces a mongod listens to. I notice you have some overlap in your IPs, which is likely incorrect (two members shouldn't have the same IP and port). – Stennie Apr 24 '18 at 2:10
  • Hi @Stennie... Yes, I'm trying to do it through forwarded ports. I installed mongo shell on the bastion server to test connectivity and it worked as expected, using the following connection string: mongodb://<user>:<password>@10.0.1.48:27017,10.0.3.169:27017,10.0.2.35:27017/<db-name>?replicaSet=<rs-name> – Henrique Barcelos Apr 24 '18 at 15:05
  • When tuneling, I forward each of the host:ports pairs to 9005, 9006 and 9007 and try to connect to them as localhost. – Henrique Barcelos Apr 24 '18 at 15:07
  • 1
    There are several aspects to your question. Port forwarding replica set connections will not work as expected because Clients use the hostnames listed in the replica set config, not the seed list. After connecting to a member of the provided seed list, clients are expected to discover the current replica set configuration (as per rs.conf()) and use those canonical hostnames and ports. – Stennie Apr 24 '18 at 23:10
0

I was able to find the root of the problem thanks to the inputs from @Stennie.

I've changed the replica set configuration to use the actual IP addresses instead of the AWS internal DNS.

Also, I fixed the bindIp option for the hosts:

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
  bindIp: [127.0.0.1,**INSTANCE PRIVATE IP**]


#processManagement:

#security:
#  authorization: enabled

#operationProfiling:

replication:
  replSetName: ecReplica
#sharding:

## Enterprise-Only Options:

#auditLog:

#snmp:

Still, running rs.status() gave me:

{
    "set" : "ecReplica",
    "date" : ISODate("2018-04-25T16:03:27.277Z"),
    "myState" : 1,
    "term" : NumberLong(-1),
    "heartbeatIntervalMillis" : NumberLong(2000),
    "members" : [
        {
            "_id" : 0,
            "name" : "10.0.3.169:27017",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 6153,
            "optime" : Timestamp(1524671340, 1),
            "optimeDate" : ISODate("2018-04-25T15:49:00Z"),
            "electionTime" : Timestamp(1524672170, 1),
            "electionDate" : ISODate("2018-04-25T16:02:50Z"),
            "configVersion" : 405245,
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "10.0.1.48:27017",
            "health" : 0,
            "state" : 6,
            "stateStr" : "(not reachable/healthy)",
            "uptime" : 0,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("2018-04-25T16:03:26.532Z"),
            "lastHeartbeatRecv" : ISODate("2018-04-25T16:03:25.651Z"),
            "pingMs" : NumberLong(0),
            "authenticated" : false,
            "configVersion" : -1
        },
        {
            "_id" : 2,
            "name" : "10.0.2.35:27017",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 5916,
            "optime" : Timestamp(1524671340, 1),
            "optimeDate" : ISODate("2018-04-25T15:49:00Z"),
            "lastHeartbeat" : ISODate("2018-04-25T16:03:26.531Z"),
            "lastHeartbeatRecv" : ISODate("2018-04-25T16:03:25.760Z"),
            "pingMs" : NumberLong(1),
            "lastHeartbeatMessage" : "could not find member to sync from",
            "configVersion" : 405245
        }
    ],
    "ok" : 1
}

Also, I noticed that the following line was uncommented in /etc/mongod.conf at 10.0.1.48:

security:
  authorization: enabled

In the other 2 config files, such line was commented.

When I commented out the authorization line and restarted the service, the rogue node could finally sync'ing:

{
    "set" : "ecReplica",
    "date" : ISODate("2018-04-25T16:14:18.489Z"),
    "myState" : 1,
    "term" : NumberLong(-1),
    "heartbeatIntervalMillis" : NumberLong(2000),
    "members" : [
        {
            "_id" : 0,
            "name" : "10.0.3.169:27017",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 70,
            "optime" : Timestamp(1524672835, 2),
            "optimeDate" : ISODate("2018-04-25T16:13:55Z"),
            "lastHeartbeat" : ISODate("2018-04-25T16:14:18.248Z"),
            "lastHeartbeatRecv" : ISODate("2018-04-25T16:14:16.605Z"),
            "pingMs" : NumberLong(1),
            "lastHeartbeatMessage" : "syncing from: 10.0.2.35:27017",
            "syncingTo" : "10.0.2.35:27017",
            "configVersion" : 405245
        },
        {
            "_id" : 1,
            "name" : "10.0.1.48:27017",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 186,
            "optime" : Timestamp(1524672835, 2),
            "optimeDate" : ISODate("2018-04-25T16:13:55Z"),
            "lastHeartbeat" : ISODate("2018-04-25T16:14:18.313Z"),
            "lastHeartbeatRecv" : ISODate("2018-04-25T16:14:17.533Z"),
            "pingMs" : NumberLong(3),
            "lastHeartbeatMessage" : "syncing from: 10.0.3.169:27017",
            "syncingTo" : "10.0.3.169:27017",
            "configVersion" : 405245
        },
        {
            "_id" : 2,
            "name" : "10.0.2.35:27017",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 186,
            "optime" : Timestamp(1524672835, 2),
            "optimeDate" : ISODate("2018-04-25T16:13:55Z"),
            "infoMessage" : "could not find member to sync from",
            "electionTime" : Timestamp(1524672748, 1),
            "electionDate" : ISODate("2018-04-25T16:12:28Z"),
            "configVersion" : 405245,
            "self" : true
        }
    ],
    "ok" : 1
}
  • I still need to find a way to enable authorization and keep this working... When I figure that out, I'll edit this answer. – Henrique Barcelos Apr 25 '18 at 16:20
  • The MongoDB manual has tutorials for enabling authentication. I don't think it's necessary to update the answer with that information: the original problem was incorrect configuration. – Stennie Apr 25 '18 at 19:43
0

It seems that your settings is still auth disabled in ReplicaSet.

To enable it, simply add a security.keyFile in settings or use --keyFile in command line option.

Here is an example showing how to generate such file:

openssl rand -base64 756 > <path-to-keyfile>
chmod 400 <path-to-keyfile>

then add to mongod.conf the path of the genreated keyfile:

security:
   authorization: enabled
   keyFile: /path/to/keyfile

restart the mongod service, right now your mongo should be auth enabled.

for more information about keyFile, refer to Enforce Keyfile Access Control in a Replica Set

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.