I'm testing GKE with their default monitoring charts.
Cluster summary say each node (n2-custom-4-8192) have 6.36 GB Memory allocatable.
In the node details page, I can see the peak "Used" is 4.19 when the pod get killed.

I'm missing something? Or this is an issue with the chart? 6.36 GB Memory allocatable

Pod killed when using 4.19 GB


In this case everything working as intended. I guess you missed information about Eviction threshold.

Allocatable is the value of Capacity minus Reserved and Eviction Threshold.

In GKE documentation Node allocatable resources you can find information about resource allocation.

Some of a node's resources are required to run the GKE and Kubernetes node components necessary to make that node function as part of your cluster. As such, you may notice a disparity between your node's total resources (as specified in the machine type documentation) and the node's allocatable resources in GKE. As larger machine types tend to run more containers (and by extension, more Pods), the amount of resources that GKE reserves for Kubernetes components scales upward for larger machines. Windows Server nodes also require more resources than a typical Linux node. The nodes need the extra resources to account for running the Windows OS and for the Windows Server components that can't run in containers.

To inspect the node allocatable resources available in a cluster, run the following command:

$ kubectl describe node ${NodeName} | grep Allocatable -B 7 -A 6

The returned output contains Capacity and Allocatable fields with measurements for ephemeral storage, memory, and CPU.

If you will scroll a bit lower to Allocatable memory and CPU resources you will read that allocatable resources are calculated in the following way:

Allocatable = Capacity - Reserved - Eviction Threshold

For memory resources, GKE reserves the following:

  • 255 MiB of memory for machines with less than 1 GB of memory
  • 25% of the first 4GB of memory
  • 20% of the next 4GB of memory (up to 8GB)
  • 10% of the next 8GB of memory (up to 16GB)
  • 6% of the next 112GB of memory (up to 128GB)
  • 2% of any memory above 128GB
  • thanks you, I indeed missing k8s Eviction Threshold – nvcnvn Aug 27 '20 at 9:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.