I have my own physical kubernetes nodes, hosted on physical machines. (12 cores, 16GB ram) I have noticed that during heavy load I have failing pods and OOM events (out-of memory kill) in the logs.

How could I force kubernetes not to schedule any job on a node if RAM is exhausted? I am running kubernetes v1.16.11 and CentOS Linux 7 3.10.0-1127.10.1.el7.x86_64

P.S I add a snippet of offending pod code, I am using requests and limit

  - name: jnlp
        cpu: 500m
        memory: 1024Mi
        memory: 2048Mi

 - name: build
        cpu: 3000m
        memory: 4096Mi
  • "in the logs" for the kernel, meaning the machine is generating OOM events, or in the Pod logs, meaning kubernetes is killing those containers? Because if it's the latter, the system is doing exactly as it was instructed via the resources: descriptors – mdaniel Jun 29 '20 at 16:09
  • Build process running inside the pod (container) cc1plus is causing OOM in Linux kernel logs. And that causes whole node to go to Not Ready state. My guess due to heavy building and lack of RAM. I would like kubernetes to NOT SCHEDULE new pod if there is no RAM available on node, but all it cares is available vCPU? – Wojtas.Zet Jun 30 '20 at 22:13
  • Right, but you overlooked my point: does the Pod have a declaration of the maximum memory the scheduler will allow the Pod to consume? Because if there are no resource constraints on the Pod, yes, it can absolutely take over the whole Node and do what you're describing. If there are resource constraints, kubernetes will kill the Pod but will keep any one Pod from taking down the Node – mdaniel Jul 1 '20 at 5:35
  • Yes, I have both requests and limits set for memory in every pod. There is certainly several pod scheduled on the node concurrently. – Wojtas.Zet Jul 1 '20 at 11:17
  • I have added snippet of pod code to the question. I have noticed that for container build I have only requests and no limit. Does it mean container build can eat more than 4096Mi and cause this kernel OOM? – Wojtas.Zet Jul 1 '20 at 11:26

As others have noted, you are most likely lacking requests and limits for your Pods. In your case, since even your kernel reports OOM events, your Pods seem to consume a lot of memory. You can read more about Resource Management here: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

So review your Pods and try to enforce container resources. Here is an example for a container that "reserves" 300Mi of memory (resources that the container is guaranteed to get), but can use up to 600Mi of memory:

apiVersion: v1
kind: Pod
  name: example-pod
  - name: example-container
    image: example-image:1.0
        memory: "300Mi"
        cpu: "1000m"
        memory: "600Mi"
        cpu: "2000m"

If the container tries to use more than 600Mi of memory, it will be OOMKilled.

  • I have added snippet of code of my pod to the question above, please take a look. – Wojtas.Zet Jul 1 '20 at 11:22
  • As others have noted above, you did not set a limit for your build container, most likely leading to the OOM situation. I recommend that you set a limit for that container. – Simon Jul 2 '20 at 15:46
  • So actually its not an option cause it causes build to fail, and build at peak eats up 13GB. (node has 16) I need to find out a smart way to schedule different memory size builds at a node. I suppose one way would be to have separate container for large memory build and separate one for medium memory build. Anyway I consider initial question resolved. – Wojtas.Zet Jul 14 '20 at 10:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.