We have a Kubernetes CronJob resource that runs a Job every minute. I need the container that results from the Job to know when it was scheduled to run. Is this possible?
Background info just in case it's useful
The hierarchy of Kubernetes resources is this: CronJob -> Job -> Pod -> initContainer -> Container -> PHP CLI command.
I have noticed the container will start between 20 seconds and 10 minutes after the minute which we wanted it to run. There are 2 reasons for the significant startup time:
init-containerdoes quite a few things including pulling docker images, and it also does a
cpof the >300MB app sourecode into a volume.
containerresource limits are rather high, e.g 2G memory, meaning there sometimes isn't enough capacity in the cluster so the cluster-autoscaler has to provision a new node to run the Job Pod, and the bootstrapping process can take a while before the new node joins the cluster.
This start up delay can have interesting effects such as a Job scheduled later can beat a Job scheduled earlier to run because it happens to have the capacity it needs to run whereas the previous one didn't so is waiting for a new node to start.
Some things I've looked at so far to solve my problem
- I looked at the DownWards API so the pod can look in
/etc/labelsto know what labels it has but unfortunately it does not provide pod startup time, only pod name.
- I looked at using a dynamic value in a Pod label, i.e. the current timestamp. But as far as I know, this isn't possible?
Kubernetes v1.10, running in AWS EKS. Job is a PHP 7.2 CLI command.
Steps to reproduce
- Set up a Kubernetes cluster with a cluster-autoscaler installed and enabled.
- Create a
CronJob. Put the following yaml in a file called
apiVersion: batch/v1beta1 kind: CronJob metadata: name: tomtest labels: app: test tier: test tester: tom spec: schedule: "* * * * *" jobTemplate: metadata: name: tomtest-crons spec: template: metadata: labels: app: test tier: test tester: tom build_id: tom3 spec: containers: - name: cron image: giantswarm/tiny-tools imagePullPolicy: IfNotPresent env: - name: TOMTEST value: "3" args: - /bin/sh - -c - date;echo hi;sleep 600;echo bye;date restartPolicy: Never resources: requests: cpu: "1" memory: "2G" limits: cpu: "1" memory: "2G"
- Start the CronJob on your cluster:
kubectl create -f cron_test.yaml
- This will launch a container each minute which does nothing but sleep for 10 minutes.
- Wait a few minutes, the containers will start to stack up and because they have high resource limits it's likely the cluster-autoscaler will kick in and add a new node or 2. If not, increase the resource limits further.
kubectl get podsto find Pods that seem late - i.e. that have a startup time that isn't exactly a minute from the previous one.
- Inspect the Pod info:
kubectl get pod tomtest-123-456 -o=yaml- notice there is a
creationTimestampfield and a
startTimebut these are not when the Pod was actually scheduled.
- When finished, clean up:
kubectl delete CronJob tomtest(This removes all Jobs and Pods as well)