1

On Thursday afternoon our project (maybe all of them) on Google Cloud was migrated to a newer version of Stackdriver. At that point all of our Process Health alerts were triggered. It was a bit scary, but we realized what is going on, so no serious harm done. However since then:

  • any process health alerts that have been triggered got stuck in an alerting state
  • Any new alerts can be triggered once (by manually stopping a monitored process), then these also get stuck in that alerting state.

Attempted fixes so far:

  • upgraded stackdriver-agent
  • recreated some alerts
  • verified that on app.google.stackdriver.com I can see all the running processes on each instance.

Upon restarting the stackdriver-agent I do see a warning:

Not authorized to talk to the GCM API, falling back on legacy ... (warning)

Anybody else is experiencing the same issue?

1 Answer 1

2

After the migration to the new Stackdriver service, you might have experienced false positive health alerts from some instances that weren't originally configured with the needed API scopes, or didn't have the monitoring agent installed with the '--write-gcm' flag enabled.

To verify the instance scopes are correct, see the documentation section 'Verifying Compute Engine credentials' for details about how to do this in the Cloud Console. If you do not have Write Only or Full permissions for the Cloud Monitoring API, the documentation will direct you in the last step to set up private-key service account credentials in the section 'Adding credentials'.

Alternatively, if you recreate the instance without deleting the boot disk, the correct scopes will be added to the Compute Engine credentials by default.

You should also ensure the monitoring agent is installed with the '--write-gcm' flag, as per 'Installing on Linux'. Finally, make sure that the Cloud Monitoring API is enabled on your project.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.