104

I have the following Kubernetes Job configuration:

---
apiVersion: batch/v1
kind: Job
metadata:
  name: dbload
  creationTimestamp: 
spec:
  template:
    metadata:
      name: dbload
    spec:
      containers:
      - name: dbload
        image: sdvl3prox001:7001/pbench/tdload
        command: ["/opt/pbench/loadTpcdsData.sh",  "qas0063", "dbc", "dbc", "1"]
      restartPolicy: Never
      imagePullSecrets: 
        - name: pbenchregkey
status: {}

When I do kubectl create -f dbload-deployment.yml --record the job and a pod are created, Docker container runs to completion and I get this status:

$ kubectl get job dbload
NAME      DESIRED   SUCCESSFUL   AGE
dbload    1         1            1h
$ kubectl get pods -a
NAME           READY     STATUS      RESTARTS   AGE
dbload-0mk0d   0/1       Completed   0          1h

This job is one time deal and I need to be able to rerun it. If I attempt to rerun it with kubectl create command I get this error

$ kubectl create -f dbload-deployment.yml --record
Error from server: error when creating "dbload-deployment.yml": jobs.batch "dbload" already exists

Of course I can do kubectl delete job dbload and then run kubectl create but I'm wondering if I can somehow re-awaken the job that already exists?

7 Answers 7

58

No. There is definitely no way to rerun a kubernetes job. You need to delete it first.

4
  • 6
    Those like me who need more details - this is two a step process. First delete your job with kubectl delete job <job_name> and then kubectl apply -f <job_yml>
    – Niks
    Jun 12, 2020 at 14:58
  • 5
    It can be done in one step with kubectl replace --force - which will delete the job if it exists, and then (re-)create it unconditionally. See below.
    – Caesar
    Mar 31, 2021 at 0:58
  • 1
    @Caesar An example? Because I get error: must specify one of -f and -k
    – DimiDak
    Dec 17, 2021 at 14:55
  • don't use kubectl replace --force by default. If you do and it fails (because it includes autogenerated fields), the original job will have been already deleted and you will not be able to rerun the command. Use kubectl get -o yaml first and at that point using kubectl apply is easier anyway.
    – Ordoshsen
    Apr 28 at 9:26
75

Simulate a rerun by replacing the job with itself:

  1. Backup your job:
  • kubectl get job "your-job" -o json > your-job.json
  1. Replace the job in place:
  • kubectl get job "your-job" -o json | kubectl replace --force -f -

If you get errors due to auto-generated labels or selectors, you can delete or edit them with jq:

  • kubectl get job "your-job" -o json | jq 'del(.spec.selector)' | jq 'del(.spec.template.metadata.labels)' | kubectl replace --force -f -

UPDATED with Jeremy Huiskamp's suggestion

4
  • 8
    Would strongly recommend saving a copy of the job json to a file first. kubectl replace deletes the job before running into the errors recreating it. Apr 4, 2019 at 12:40
  • 2
    Save the json first and then recreate!!
    – deepdive
    Nov 14, 2019 at 23:48
  • No need to save the json first if he still has the original job definition file?
    – Caesar
    Mar 31, 2021 at 0:56
  • F Santiago can you update the answer to incorporate @JeremyHuiskamp. edit queue is full Jun 17, 2021 at 15:18
44

You can also avoid the error you mentioned by specifying

  generateName: dbload

instead of simply name

In that case, each job you submit with this yaml file will have a unique name that will look something like dbloada1b2c. Then you can decide whether you need to delete the old jobs, but you won't have to do it.

Here is a working yaml example:

apiVersion: batch/v1
kind: Job
metadata:
  generateName: netutils-
spec:
  parallelism: 1
  template:
    spec:
      containers:
      - image: amouat/network-utils 
        name: netutil
      restartPolicy: Never

This is the output from kubectl get job after two kubectl create -f example.yaml commands:

NAME             COMPLETIONS   DURATION   AGE
netutils-5bs2s   0/1           14s        14s
netutils-dsvfk   0/1           10s        10s

10
  • I believe generateName only applies to kind=pod and NOT job.
    – user518066
    Aug 30, 2017 at 14:28
  • 5
    No, it's a standard part of ObjectMeta and applies to both pod and job: k8s reference. I've been using it all the time, it's core to what I'm doing.
    – vp124
    Sep 1, 2017 at 14:52
  • 6
    Thank you very much for this dodge. Just for documentation this does only work with kubectl create
    – Ohmen
    Nov 29, 2017 at 14:27
  • disagree. Tried to do this right now and it causes error "resource name may not be empty". Feb 8, 2020 at 0:48
  • 2
    Are you creating your job with kubectl create or kubectl apply? The latter will not work, as per this reference: github.com/kubernetes/kubernetes/issues/44501. So you have to do kubectl create. I have also just tried on Azure, it works there as well. kubectl version shows v1.15.10 on AKS and v1.14.10-gke.27 ok GKE, but like I said, I've been doing this for quite a while, so it worked on earlier versions, too.
    – vp124
    May 4, 2020 at 16:06
5

As an improvement on @F. Santiago's idea, you can simply use the value stored at the annotation "kubectl.kubernetes.io/last-applied-configuration" that holds the initial applied configuration without any auto generated field:

kubectl get job <jobname> -o json | \
jq -r '.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration"' | \
kubectl replace --save-config --force -f -

Note: for kubectl replace, remember to pass --save-config so it updates the annotation field with the last config applied.

3

There is no way to run a job that completed but you can simulat a rerun by doing the following

  1. Get yaml file of the existing job:

    kubectl get job <job_name> -o yaml > <job_name>.yaml
    
  2. Delete the existing job:

    kubectl delete job <job_name>
    
  3. Run the job again:

    kubectl apply -f <job_name>.yaml
    
2

Based on @Marcelo's idea I made it work with the following, without any processing of the template:

kubectl get job <job-name> -o custom-columns=:metadata.annotations.kubectl\.kubernetes\.io/last-applied-configuration > job.json
kubectl delete -f job.json
kubectl apply -f job.json

Please note the escaped dots (\.) in the annotation name: kubectl\.kubernetes\.io/last-applied-configuration. Without it, it returns <none>.

0

I've implemented F. Santiago's method using a kubectl plugin. I have these two files in my PATH [1] so that kubectl picks it up.

Now issuing kubectl replacejob [job name] looks like this:

[w] eddie@eddie ~ $ kubectl replacejob my_job_name
Writing out backup file: my_job_name.bak.json
job.batch "my_job_name" deleted
job.batch/my_job_name replaced

[1] Files to make the plugin work:

kubectl-replacejob.cmd: Simple wrapper to call python with the same args

@echo off

pushd . 
cd %~dp0
python kubectl-replacejob.py %*
popd 

kubectl-replacejob.py: Does the 'hard' work of replacement.

import sys
import subprocess
import json
import io

if len( sys.argv ) < 2:
    print("Error: please specify the job you wish to replace.")
    sys.exit(-1)

job_name = sys.argv[1]

# Fetch the job as json
job_as_json_text = subprocess.check_output(f'kubectl get job {job_name} -o json', shell=True).decode()
job_as_json = json.loads(job_as_json_text)

# Save out a backup
backup_file = f'{job_name}.bak.json'
print(f"Writing out backup file: {backup_file}")
with open(backup_file, 'w') as f:
    f.write(job_as_json_text)

# Remove references to controller uid that borks replace
def remove_key_if_present(obj, *keys):
    for i in range(len(keys)):
        key = keys[i]
        if key in obj:
            if i == len(keys)-1:
                del obj[key]
            else:
                obj = obj[key]
        else:
            print(f"WARNING: Failed to remove {'.'.join(keys)}: failed finding key at {key}!")
            return 


remove_key_if_present(job_as_json, 'spec', 'selector', 'matchLabels', 'controller-uid')
remove_key_if_present(job_as_json, 'spec', 'template', 'metadata', 'labels', 'controller-uid')
job_as_json_text = json.dumps(job_as_json)

# Pretty print for testing
#print(json.dumps(job_as_json, indent=4, sort_keys=True))

# Issue the replace
subprocess.run(f'kubectl replace --force -f -', shell=True, input=job_as_json_text.encode())

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.