On Thursday afternoon our project (maybe all of them) on Google Cloud was migrated to a newer version of Stackdriver. At that point all of our Process Health alerts were triggered. It was a bit scary, but we realized what is going on, so no serious harm done. However since then:
- any process health alerts that have been triggered got stuck in an alerting state
- Any new alerts can be triggered once (by manually stopping a monitored process), then these also get stuck in that alerting state.
Attempted fixes so far:
- upgraded stackdriver-agent
- recreated some alerts
- verified that on app.google.stackdriver.com I can see all the running processes on each instance.
Upon restarting the stackdriver-agent I do see a warning:
Not authorized to talk to the GCM API, falling back on legacy ... (warning)
Anybody else is experiencing the same issue?