Now this is a weird one, but I hope you will bare with me, and have suggestions as to what to do.
I have two things, which are not immediate compatible:
- The old stuff: Several Jenkins Build-server jobs that must be performed, once or several times daily, currently configured with the cron to trigger at specific times. These jobs sometimes run for many hours, and their runtime is not predictable.
- The new stuff: A global IT department who insists that they must be able to patch and reboot any jenkins server and/or agent as they wish - although within some known maintenance windows.
- The window can be described in cron syntax as "30 15 19-25 * 6" to "00 20 19-25 * 6"
Currently the addition of the maintenance breaks things in at least these ways:
- Jenkins may not be running when a job is supposed to start.
- Jenkins jobs that are running are terminated unexpectedly.
In both cases whats needed is a method to detect if a given job was started at a specific time and allowed to "naturally" complete its course. In either case the job should be scheduled as soon as possible thereafter.
I've naturally asked the IT-Department to allow us to do this in a way that's compatible to our systems, but that's not going to happen in this decade. :-(
I'm now considering how to deal with it. My thoughts currently are:
- To minimise damage
- Use Windows cron to shut down the Jenkins service well before the maintenance window - ensuring it can not start during the maintenance window, and to start up the Jenkins service following the maintenance period. (I'm here thinking to trigger two scripts, which will -indirectly- modify the service startup setting in the registry.)
- To fix damage, have a separate Jenkins job run on first boot following maintenance, performing the following using a groovy script
- Scan a set of known jobs for jobs that ended prematurely at the time the service window started.
- Scan a set of known jobs for jobs that ought to have started during the maintenance window,
- For each job found in step 2, rebuild those one by one, prioritized based on their original start time.
But this looks like a complicated task, and potentially a hell to administer. :-(
What other options do I have?