Stackdriver Extractor fails

Question

I'm trying to get stackdriver working with GCE. I went through the standard instructions to install (executing stack-install.sh --write-gcm), however it doesn't seem to work at all. When I add monitoring alerts to watch processes, I get:

Process monitoring not available for this instance

In /var/log/messages I see:

Mar 28 19:47:59 myhost stackdriver-agent: which: no java in (/root/.gvm/bin:/usr/local/sbin:/sbin
:/usr/sbin:/bin:/usr/bin:/go/bin:/bin)
Mar 28 19:47:59 myhost stackdriver-agent: Starting stackdriver-collectd: option = PIDFile; value 
= /var/run/stackdriver-agent.pid;
Mar 28 19:47:59 myhost stackdriver-agent: option = Interval; value = 60.000000;
Mar 28 19:47:59 myhost stackdriver-agent: Created new plugin context.
Mar 28 19:47:59 myhost stackdriver-agent: [  OK  ]
Mar 28 19:47:59 myhost collectd[1934]: Initialization complete, entering read-loop.
Mar 28 19:47:59 myhost systemd: Started LSB: start and stop Stackdriver Agent.
Mar 28 19:47:59 myhost collectd[1934]: match_throttle_metadata_keys: 1 history entries, 1 distinc
t keys, 71 bytes server memory.
Mar 28 19:47:59 myhost collectd[1934]: tcpconns plugin: Reading from netlink succeeded. Will use 
the netlink method from now on.
Mar 28 19:47:59 myhost collectd[1934]: write_gcm: Asking metadata server for auth token

I have verified that the service is enabled. When I check the permissions, it says I don't need to need special credentials since I am using GCE. However, when I restart the stackdriver-extractor service, I get:

Please set STACKDRIVER_API_KEY in /etc/sysconfig/stackdriver

And so if I set that... I get:

stackdriver-extractor[3020]: Starting stackdriver-extractor: Unable to determine extractor endpoint! 
stackdriver-extractor[3020]: Unable to generate config file

I feel like something is wrong at the project level where this service is not getting the key information automatically like it is supposed to.

I dug into the scripts and found where it gets it...

curl -f -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/project/attributes/stackdriver-agent-key

But this is getting a 404.

Also, when I check the scopes:

# curl --silent -f -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/scopes
https://www.googleapis.com/auth/cloud.useraccounts.readonly
https://www.googleapis.com/auth/devstorage.read_only
https://www.googleapis.com/auth/logging.write
https://www.googleapis.com/auth/monitoring.write

Any ideas? My ultimate problem is that I can't create alerts on the process counts, but I think this is where Stackdriver gets it from.

Answer 1

To fix this problem, I fixed the API key. I accidentally left out the double quotes when I edited the file.

That brings me to the next point. Why did I edit the file? They provide a nice easy utility for adding an API key. We should use it.

/opt/stackdriver/stack-config --api-key DEADBEEFDEADBEEFDEADBEEFDEADBEEF...

It will even restart the services for you.

As for how I got to this place to begin with... I accidentally copy/pasted a partial API key. It wasn't obvious because I missed a little bit at the end. Little bits matter.

As for why I could not get this working by changing the scope to all API access, I don't know. I thought that would work as I really don't want to have to mess around with any more statefulness in my VM disk than I have to. But I'll take a working solution at this point.

One final bit of advice... if you are here, you might need to also find your API key. You can find it here:

https://app.google.stackdriver.com/settings/accounts/agent/

If one isn't there, you can create it.

Good luck!

Answer 2

Just FYI the use of API keys is kind of depreciated/Legacy. You should enable the Google Monitoring API and just install the agent.