3

Has anyone successfully managed to monitor a MongoDB 3 cluster (or standalone db) with StackDriver in GCE ?

I've setup a MongoDB 3.0.6 cluster in GCE (replica set with 2 replica and 1 arbiter)

I'm trying to monitor it through StackDriver provided by Google.

I've followed all instructions to install monitoring agent and mongodb plugin found here : https://cloud.google.com/monitoring/agent/plugins/mongodb

When I start the agent on the replica where it is configured :

sudo /etc/init.d/stackdriver-agent restart

I get the following errors in /var/log/syslog :

collectd[6013]: tcpconns plugin: Reading from netlink succeeded. Will use the netlink method from now on.
collectd[6013]: mongo plugin: Authenticating to localhost:27017 failed:
collectd[6013]: mongo plugin: Connecting to localhost:27017 failed:
collectd[6013]: read-function of plugin `mongodb' failed. Will suspend it for 120.000 seconds.

I suspect the StackDriver agent is not compatible with MongoDB 3, because :

  • In the past, I've used the Click-to-deploy feature of GCE to create a cluster and was able to monitor it with StackDriver. At that time, it was with MongoDB 2.6.x.

  • I've quickly setup again a standalone installation of MongoDB 2.6.x, configured the StackDriver agent in the same way, and ... it works :-(

Any help will be really appreciated.

Details of configuration :

Mongodb :

  • auth=true

  • in database admin, user with roles : dbAdminAnyDatabase, clusterAdmin and readAnyDatabase

Stackdriver Mongodb plugin :

  • this user and password used in /opt/stackdriver/collectd/etc/collectd.d/mongodb.conf

Additional information :

No Auth : the plugin initializes successfully

/etc/mongod.conf :

# Turn on/off security.  Off is currently the default
#noauth = true
#auth = true

/opt/stackdriver/collectd/etc/collectd.d/mongodb.conf :

LoadPlugin mongodb

<Plugin "mongodb">
    Host "localhost"
    Port "27017"

   #  If you restricted access to the database, you can
   #  set the username and password here
   #  User "user_name"
   #  Password "user_password"

   # For performance/eventually consistent trade-offs you may add this line
   # PreferSecondaryQuery true
</Plugin>

Auth mode : authentication errors when restarting the agent

/etc/mongod.conf :

# Turn on/off security.  Off is currently the default
#noauth = true
auth = true

3 users configured

use admin
db.createUser(
  {
    user: "siteUserAdmin",
    pwd: "xxx",
    roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
  }
)

db.auth("siteUserAdmin", "xxx");

db.createUser( {
    user: "siteRootAdmin",
    pwd: "xxx",
    roles: [ { role: "root", db: "admin" } ]
  });

db.createUser(
  {
    user: "monitoring",
    pwd: "xxx",
    roles: [ 
    { role: "dbAdminAnyDatabase", db: "admin" },
    { role: "clusterAdmin", db: "admin" },
    { role: "readAnyDatabase", db: "admin" } ]
  }
)

/opt/stackdriver/collectd/etc/collectd.d/mongodb.conf :

LoadPlugin mongodb

<Plugin "mongodb">
    Host "localhost"
    Port "27017"

   #  If you restricted access to the database, you can
   #  set the username and password here
   User "monitoring"
   Password "xxx"

   # For performance/eventually consistent trade-offs you may add this line
   # PreferSecondaryQuery true
</Plugin>

Same errors when I use siteRootAdmin in the plugin configuration.

Explanation and solution

The culprit was actually the auth schema used by StackDriver agent

I adapted the solution suggested by Adam C, because I already have users created with the SCRAM-SHA-1 schema.

In fact, I only need that the user for the monitoring uses MONGODB-CR.

To do so :

Restart MongoDB 3.0 with auth disabled

Connect to the instance

Change temporarily the auth schema to MONGODB-CR

use admin
var schema = db.system.version.findOne({"_id" : "authSchema"});
schema.currentVersion = 3;
db.system.version.save(schema);

Create the user for the StackDriver plugin

db.createUser(
      {
        user: "monitoring",
        pwd: "xxx",
        roles: [ 
        { role: "dbAdminAnyDatabase", db: "admin" },
        { role: "clusterAdmin", db: "admin" },
        { role: "readAnyDatabase", db: "admin" } ]
      }
    )

Check it has the correct auth schema : MONGODB-CR

> db.system.users.find({"user":"monitoring"})
{ "_id" : "admin.monitoring", "user" : "monitoring", "db" : "admin", "credentials" : { "MONGODB-CR" ...

Set back the auth schema to SCRAM-SHA-1

var schema = db.system.version.findOne({"_id" : "authSchema"});
schema.currentVersion = 5;
db.system.version.save(schema);

Restart MongoDB 3.0 with auth enabled

Restart the StackDriver agent

When StackDriver will support SCRAM-SHA-1, it will be useful to upgrade the auth schema for this user

db.adminCommand({authSchemaUpgrade: 1});
3
  • Did you try this in noauth mode to see if the issue is related to the authentication?
    – Kamran
    Sep 19 '15 at 3:12
  • @Kamran Yes I tried, see the edited post for configuration details.
    – Arnaud J
    Sep 21 '15 at 8:46
  • This plugin has been deprecated. Google is directing ppl to use other commercial products
    – Rondo
    Oct 21 '19 at 20:58
1

I suspect that the Stack Driver does not (yet) support the new SCRAM-SHA-1 Authentication mechanism. This new mechanism was added in 3.0 to replace MONGODB-CR and is the default in 3.0+, however it requires that the driver support the new mechanism.

To get MongoDB 3.0 to use the older mechanism, you can either start with 2.6 and create your users there, then upgrade or you can do the following (based on this comment):

  • Start MongoDB 3.0 with auth disabled (make sure no users have been added yet)
  • Connect to the instance and run the following:

    var schema = db.system.version.findOne({"_id" : "authSchema"}); schema.currentVersion = 3; db.system.version.save(schema);

  • Start MongoDB with auth enabled and create your users

These uses should now be created with MONGODB-CR. The StackDriver is using libmongoc which supports SCRAM-SHA-1 as of version 1.1 but their version looks a lot older based on some Github browsing. Once they update their driver, this issue should go away - for now you will have to work around it.

1
  • Thanks a lot, I adapted your suggestion. See details in the question.
    – Arnaud J
    Sep 22 '15 at 9:51
3

I work at Google on the Stackdriver agent. Adam C's answer is correct. I've been working on this lately, and now we have a version of this in beta that has been working well in our own testing. This new version addresses not only the SCRAM-SHA-1 issue but also has some performance improvements as well. Before releasing it more broadly, we would love the opportunity to test it in a couple of customer environments.

If anyone is willing to be a beta tester for the new version of our agent, I could arrange to deliver a .deb or .rpm file to you appropriate to your platform. As always there is a small risk that something can go wrong, so I think this only makes sense if you have a non-production environment you can try it in.

EDIT: As of July 12, 2016, this change is now available in production!

3
  • This change is now available in production. :-) Jul 12 '16 at 19:35
  • I'm also having problem monitoring MongoDB 3.2 with Stackdriver as of 28 Aug 2016. There is no mention of mongo whatsoever in /var/log/syslog but I made a configuration error on the .conf file, it complains so I know it's loading the file correctly... So no errors, but no mention of mongo in /var/log/syslog and app.google.stackdriver.com/services/mongodb claims I haven't installed the agent. Aug 28 '16 at 6:31
  • Continued in serverfault.com/questions/799571/… Aug 28 '16 at 7:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.