0

I have a 3-6 f1-micro instance GKE (k8) cluster that is plagued by the f1-micro instances constantly re-creating.

Looking at the cluster now it is currently scaled to 3 instances with the following uptimes: 10hrs, 3hrs, 1hr.

Why are my instances constantly churning? How can I debug the 'why' of instances constantly being added and removed from the instance group?

The instances are NOT preemptable. I notice in GCP they do have automatic restart set in the "availability" section.

Any help much appreciated.

Additional info:

My suspicion is the reason I'm seeing this is trying to run GKE on f1-micro instances. I've switched to a g-small instance instead and it already seems more stable.

I notice that in the stackdriver Monitoring Overview (http://app.google.stackdriver.com/) I have a lot of "gke-my-instance-xzy in cluster X is not ready" in the Events box. This is the first place in logs I've managed to find such a message. So I'd conclude that the instances are reporting unhealthy at some layer and eventually getting killed. I see recreateInstance (or something similar) in the logs frequently.

Which logs to look in to find the right health check I couldn't determine. I did notice this in one set of logs --eviction-hard=memory.available<100Mi if that means instances hard shutdown when they have less than 100MB of memory then I imagine I was hitting that. I still can't see and 'healthcheck failed' type messages in any logs.

Additional info:

I have confirmed that moving up to one small sized instance all the instability goes away. It would seem running GKE on f1-micro instances isn't currently a good idea.

I'm leaving the question open because it is about how I could debug why my f1-micro instances were being recreated so frequently and Sunny's answer didn't lead me to a 'why' message anywhere in the logs.

Solution

In a practical sense moving to a larger node size solved the problem as noted above. In the OP comments @Daniel has provided a link to a page that provides the command needed to view the logs

gcloud container operations list

I can see in the output of this command all the auto-repair events that were killing my nodes.

  • Did you define an autoscaling policy, if yes. It could be that autoscaler is performing an automatic scaling based on the measured load, which is an expected behavior? – arp-sunny. Jul 20 '18 at 14:50
  • The number of machines goes below the cluster minimum so it can't be that. I'm about to add some more info to the OP. – Ryan Jul 20 '18 at 19:21
  • What version of Kubernetes is your Cluster Master running at? This is because, beginning with Kubernetes version 1.7, you can specify a minimum size of zero for your node pool. This allows your node pool to scale down completely if the instances within aren't required to run your workloads. However, while a node pool can scale to a zero size, the overall cluster size does not scale down to zero nodes (as at least one node is always required to run system Pods). – arp-sunny. Jul 20 '18 at 19:27
  • Do you have GKE node auto repair set by any chance? f1-micros are the smallest, could be running out of resource and failing node health check and thus being constantly repaired - cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair – Daniel t. Jul 27 '18 at 18:36
  • 1
    @Daniel - yes, I did. Following that link and running the gcloud container operations list command it suggests I can see the autorepair events that were killing my nodes. This is the answer I was looking for - If you felt like writing an answer I'll happily tag it as the solution. I'll also add this command in the OP as an addendum. – Ryan Jul 28 '18 at 7:40
1

Automatic restart is always an optional availability policy you can set on the instance template used on the instance group. When set, this will allow compute engine to automatically restart VM instances if they are terminated for non-user-initiated reasons such as (maintenance event, hardware failure, software failure, etc.) so is unlikely to be that is a result of this policy that instances are getting added and removed from your instance group.

As described in the following documentation, when your applications require additional compute resources, managed instance groups can automatically scale the number of instances in the group.

In addition, Managed instance groups can automatically identify and recreate unhealthy instances in a group to ensure that all of the instances are running optimally.

Finally;

If an instance in the group stops, crashes, or is deleted by an action other than the instance groups commands, the managed instance group automatically recreates the instance so it can resume its processing tasks. The recreated instance uses the same name and the same instance template as the previous instance, even if the group references a different instance template.

You may also view the autoscaler logs as described in this public documentation to confirm if autoscaler is behind this behavior.

OR

Convert the "Filter by label field in Stackdriver to advanced filter and define the following filters

resource.type="gce_autoscaler"
protoPayload.methodName="v1.compute.autoscalers.insert"

For list of all instance created by autoscaler

AND

resource.type="gce_autoscaler"
protoPayload.methodName="v1.compute.autoscalers.delete"

For deleted instances.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.