1

I have a one a one node Kubernetes cluster and the memory usage reported by the metrics server does not seem to be the same as the memory usage shown with the free command

# kubectl top nodes
NAME          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
<node_ip>   1631m        10%    13477Mi         43%

# free -m
              total        used        free      shared  buff/cache   available
Mem:          32010       10794         488          81       20727       19133
Swap:         16127        1735       14392

And the difference is significant ~ 3 GB.

I have also tested this on a 3 node cluster, and the issue is present there too:

# kubectl top nodes
NAME          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
<node_ip1>   1254m        8%     26211Mi         84%
<node_ip2>   221m         1%     5021Mi          16%
<node_ip3>   363m         2%     8731Mi          28%
<node_ip4>   1860m        11%    20399Mi         66%

# free -m (this is on node 1)
              total        used        free      shared  buff/cache   available
Mem:          32010        5787         369        1676       25853       24128
Swap:         16127           0       16127

Why is there a difference?

  • Kubernetes and swap? – uav Sep 24 '19 at 20:49
2

Let's start from understanding how both commands works.

Free - Display amount of free and used memory in the system

Kubectl top - Allows you to see the resource consumption for nodes or pods.

There is an article explaining why you might see the differences:

Most container specific metrics are available at the cgroup filesystem via /path/to/cgroup/memory.stat, /path/to/cgroup/memory.usage_in_bytes, /path/to/cgroup/memory.limit_in_bytes and others. Most of the Linux tools providing system resource metrics were created before cgroups even existed. They usually read memory metrics from the proc filesystem: /proc/meminfo, /proc/vmstat, /proc/PID/smaps and others.

Also, the swap part (as mentioned by @uav) should trigger a red light as you must disable it in order for the kubelet to work properly. It is one of the prerequisites for installing kubeadm.

I hope it helps.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.