1

I have a single node cluster in container engine that sits at ~40% cpu when idle as seen by the monitoring dashboard.

When I click through the monitoring, all the pods are at 0% cpu.

When I ssh into the instance, I can see that docker, kubelet, and heapster are the main culprits, but I don't understand what work they're doing.

If I look in the logs, I see lines like the following, repeated over and over many times per second. I'm guessing this is related.

gke-rogue-dev-7248e467-node-9hvh 2015-10-13 19:50:55.000 time="2015-10-13T23:50:55Z" level=info msg="-job containers() = OK (0)"
gke-rogue-dev-7248e467-node-9hvh 2015-10-13 19:50:55.000 time="2015-10-13T23:50:55Z" level=info msg="+job containers()"
gke-rogue-dev-7248e467-node-9hvh 2015-10-13 19:50:55.000 time="2015-10-13T23:50:55Z" level=info msg="GET /containers/json"

What should be my next step to figure out why this is happening?

  • What size nodes are you using? And how large is your cluster? – Robert Bailey Oct 14 '15 at 2:51
  • It's a single node n1-standard-1 instance. – Marc Hughes Oct 14 '15 at 12:16
2

I had the same question recently: https://serverfault.com/q/728211/310585

The answer to "what work they're doing" is "logging and monitoring".

To prevent this overhead you just need to unselect those features when creating the cluster. In the developer console there are check boxes for them. In the CLI add the options --no-enable-cloud-logging --no-enable-cloud-monitoring to the gcloud container clusters create command.

  • Unfortunately, your other question was removed so I can't see any discussion. 40% would seem awfully excessive for monitoring and logging! Do you know of any way to disable it on a cluster after creation? Ideally, we'd be able to slow down the sample rate or something to keep monitoring on, but less aggressive. – Marc Hughes Oct 22 '15 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.