1

In have scaled my pods to 20 in my cluster and when I see the next day the few of the scaled pods were recreated.

When I say POD recreated, it is getting deleted and created freshly and the timestamp of the recreated pod and the scaled pods vary.

I was unable to find the reasons for the recreate of the PODs.

I could not find which POD went for a recreate as the POD is deleted and gone away. There are no logs in the journalctl regarding which POD got recreated. Is there any way I can debug further to find the reason for the POD recreate. or What might be the reason for the PODs getting deleted.

Note: I have readiness and liveness probes defined, but these probes would act on container and would not lead to POD recreate is my understanding.

6
  • Those liveness probes are designed to kill off bogus Pods, that's their job in life, but you are correct that a liveness fail will cause a restart event, and will not change the .metadata.name of the Pod, only its restart count. It is also supremely likely that if the Pod names are changing, it's due to a ReplicaSet change, not a Pod event. Either way, you have a lot of words that are not kubectl describe, kubectl get events and similar investigations and their outcomes. No one can guess what is wrong with your setup without knowing what concrete experiments you have already tried
    – mdaniel
    May 16 at 18:11
  • Add the commands you have tried so far to debug and their outputs. Also include the outputs from commands mentioned by mdaniel
    – Matt
    May 17 at 8:09
  • Hi @Matt, the POD is gone... I cannot do describe or events using kubectl and events does not give the details of all the pods.
    – Nikhil
    May 17 at 8:16
  • @mdaniel, the Pods were scaled to 20 and there was a load executed for these pods. After sometime few of the Pods recreated. This is the only operation done on the cluster.
    – Nikhil
    May 17 at 8:20
  • "When I say POD restart, it is getting deleted and created freshly" - So its not being restarted, it's being recreated. Can you change every occurance of a word restarted with a word recreated in your question, where this is what you actually mean. Because this is misleading. restarted != recreated - these are two different things.
    – Matt
    May 17 at 9:35
2

Basically you need to check pod's events (here is a tip how to do it).

Keep in mind, that events retention period is too short (approximately 1h), and you may need to store them somewhere else. Here is a good article how to do it with EFK stack

The most common reason of PODs recreation - node unavailability. If pod's node is not available, Kubernetes recreates pods at other nodes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.