Is it possible to rerun kubernetes job?

Question

I have the following Kubernetes Job configuration:

---
apiVersion: batch/v1
kind: Job
metadata:
  name: dbload
  creationTimestamp: 
spec:
  template:
    metadata:
      name: dbload
    spec:
      containers:
      - name: dbload
        image: sdvl3prox001:7001/pbench/tdload
        command: ["/opt/pbench/loadTpcdsData.sh",  "qas0063", "dbc", "dbc", "1"]
      restartPolicy: Never
      imagePullSecrets: 
        - name: pbenchregkey
status: {}

When I do kubectl create -f dbload-deployment.yml --record the job and a pod are created, Docker container runs to completion and I get this status:

$ kubectl get job dbload
NAME      DESIRED   SUCCESSFUL   AGE
dbload    1         1            1h
$ kubectl get pods -a
NAME           READY     STATUS      RESTARTS   AGE
dbload-0mk0d   0/1       Completed   0          1h

This job is one time deal and I need to be able to rerun it. If I attempt to rerun it with kubectl create command I get this error

$ kubectl create -f dbload-deployment.yml --record
Error from server: error when creating "dbload-deployment.yml": jobs.batch "dbload" already exists

Of course I can do kubectl delete job dbload and then run kubectl create but I'm wondering if I can somehow re-awaken the job that already exists?

Answer 1

No. There is definitely no way to rerun a kubernetes job. You need to delete it first.

Answer 2

Simulate a rerun by replacing the job with itself:

1. Backup your job:

• kubectl get job "your-job" -o json > your-job.json

2. Replace the job in place:

• kubectl get job "your-job" -o json | kubectl replace --force -f -

If you get errors due to auto-generated labels or selectors, you can delete or edit them with jq:

• kubectl get job "your-job" -o json | jq 'del(.spec.selector)' | jq 'del(.spec.template.metadata.labels)' | kubectl replace --force -f -

UPDATED with Jeremy Huiskamp's suggestion

Answer 3

You can also avoid the error you mentioned by specifying

  generateName: dbload

instead of simply name

In that case, each job you submit with this yaml file will have a unique name that will look something like dbloada1b2c. Then you can decide whether you need to delete the old jobs, but you won't have to do it.

Here is a working yaml example:

apiVersion: batch/v1
kind: Job
metadata:
  generateName: netutils-
spec:
  parallelism: 1
  template:
    spec:
      containers:
      - image: amouat/network-utils 
        name: netutil
      restartPolicy: Never

This is the output from kubectl get job after two kubectl create -f example.yaml commands:

NAME             COMPLETIONS   DURATION   AGE
netutils-5bs2s   0/1           14s        14s
netutils-dsvfk   0/1           10s        10s

Answer 4

As an improvement on @F. Santiago's idea, you can simply use the value stored at the annotation "kubectl.kubernetes.io/last-applied-configuration" that holds the initial applied configuration without any auto generated field:

kubectl get job <jobname> -o json | \
jq -r '.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration"' | \
kubectl replace --save-config --force -f -

Note: for kubectl replace, remember to pass --save-config so it updates the annotation field with the last config applied.

Answer 5

There is no way to run a job that completed but you can simulat a rerun by doing the following

1. Get yaml file of the existing job:

   kubectl get job <job_name> -o yaml > <job_name>.yaml
   

2. Delete the existing job:

   kubectl delete job <job_name>
   

3. Run the job again:

   kubectl apply -f <job_name>.yaml
   

Answer 6

Based on @Marcelo's idea I made it work with the following, without any processing of the template:

kubectl get job <job-name> -o custom-columns=:metadata.annotations.kubectl\.kubernetes\.io/last-applied-configuration > job.json
kubectl delete -f job.json
kubectl apply -f job.json

Please note the escaped dots (\.) in the annotation name: kubectl\.kubernetes\.io/last-applied-configuration. Without it, it returns <none>.

Answer 7

I've implemented F. Santiago's method using a kubectl plugin. I have these two files in my PATH [1] so that kubectl picks it up.

Now issuing kubectl replacejob [job name] looks like this:

[w] eddie@eddie ~ $ kubectl replacejob my_job_name
Writing out backup file: my_job_name.bak.json
job.batch "my_job_name" deleted
job.batch/my_job_name replaced

[1] Files to make the plugin work:

kubectl-replacejob.cmd: Simple wrapper to call python with the same args

@echo off

pushd . 
cd %~dp0
python kubectl-replacejob.py %*
popd 

kubectl-replacejob.py: Does the 'hard' work of replacement.

import sys
import subprocess
import json
import io

if len( sys.argv ) < 2:
    print("Error: please specify the job you wish to replace.")
    sys.exit(-1)

job_name = sys.argv[1]

# Fetch the job as json
job_as_json_text = subprocess.check_output(f'kubectl get job {job_name} -o json', shell=True).decode()
job_as_json = json.loads(job_as_json_text)

# Save out a backup
backup_file = f'{job_name}.bak.json'
print(f"Writing out backup file: {backup_file}")
with open(backup_file, 'w') as f:
    f.write(job_as_json_text)

# Remove references to controller uid that borks replace
def remove_key_if_present(obj, *keys):
    for i in range(len(keys)):
        key = keys[i]
        if key in obj:
            if i == len(keys)-1:
                del obj[key]
            else:
                obj = obj[key]
        else:
            print(f"WARNING: Failed to remove {'.'.join(keys)}: failed finding key at {key}!")
            return 


remove_key_if_present(job_as_json, 'spec', 'selector', 'matchLabels', 'controller-uid')
remove_key_if_present(job_as_json, 'spec', 'template', 'metadata', 'labels', 'controller-uid')
job_as_json_text = json.dumps(job_as_json)

# Pretty print for testing
#print(json.dumps(job_as_json, indent=4, sort_keys=True))

# Issue the replace
subprocess.run(f'kubectl replace --force -f -', shell=True, input=job_as_json_text.encode())

Answer 8

On the subject of deleting a Job before you can run it again, as the accepted answer states, note that you can now also set up a Job to automatically clean itself after completion with .spec.ttlSecondsAfterFinished