I work on a Kubernetes cluster where, right now, about 95% of the CPUs and 90% of the memory have been allocated to pods. However, according to the Kubernetes Dashboard, the overall instantaneous CPU load on the cluster is only about 5% of the total number of cores in the cluster, and the total used memory is only about 33% of the total memory in the cluster. So clearly some and possibly most pods running on the cluster are dramatically overprovisioned; most of the requested CPU and memory is not actually in use at any given time.
How do I find out which pods are most to blame for this? The Dashboard will show me the how much of each node's resources are allocated, and how what resources are actually in use in each running pod. But to see a pod's requests I have to
kubectl describe it; I can't find the requests anywhere in the dashboard. Moreover, when a pod finishes and gets cleaned up it disappears, and I don't know of any way to ask questions like "What portion of the requested memory did this completed pod use at its peak?", or "How many core-hours did this pod request but not use over its lifetime?".
What tools exist for finding and diagnosing wasted, requested-but-not-consumed resources in Kubernetes clusters? And what best practices should be employed for right-sizing pods to workloads? I think we got into this situation by letting all the users just double their resource requests until their pods stopped being evicted.