We use kubernetes (specifically: openshift) to run our infrastructure in our team.
One of the daemonsets, fluentd, is currently causing a lot of trouble, frequently taking full nodes down with huge CPU, memory and disk I/O requirements (like really, it's absolutely stupid!).
We've set the following resource limits and requests on pod in the DaemonSet definition:
resources: limits: memory: 512Mi requests: cpu: 100m memory: 512Mi
I was expecting k8s to kill the pod when the memory consumption exceeds 512Mi. Yet, those pods are freely allowed to consume 1000+% of CPU and all of the available RAM on the machine (far above 512Mi).
I've done some research, and it seems the world is quite divided on what happens when the memory limits are exceeded.
- Some say the pod will keep running until the system is OOM and decides to kill something.
- Others say the pod will instantly be killed when it exceeds the allotted memory.
We would very much like to have the 2nd option happening to that pesky pod! What are we missing?