Cronjob · XB Field Notes

Kubernetes CronJobs need a history policy

Fri, 25 Feb 2028 00:00:00 +0000

A CronJob does not run your work; it creates a Job, and the Job creates a pod. When the pod finishes, neither the Job nor the pod disappears. The pod is kept on purpose so you can still read its logs after the fact, and the Job is kept so you can still read its status.

That default is right for one run and wrong for ten thousand. Every finished Job is an object in etcd, a row in every list response, and a watch event for every controller that cares about Jobs. Two separate mechanisms bound the pile, and they are independent: count-based history limits applied by the CronJob controller, and age-based TTL applied by the TTL controller.

A finished Job is not free

A completed Job keeps its pod, and a pod keeps its container logs for as long as the pod exists. Retention is the only thing between you and a namespace full of tombstones. Thirty days of an hourly schedule is 720 runs: 720 Jobs and 720 pods, each with a name, a status, an owner reference and a line in every kubectl get jobs. A finished Job is not inert either: its status conditions stay in the API for every watcher to re-read. The cost per object is trivial, but it is linear in time, so it stays invisible until it is a problem.

The visibility cost arrives before the resource cost. Once the namespace holds hundreds of finished Jobs, kubectl get jobs no longer fits on a screen and a genuine failure is one more line among the successes. API server memory pressure is real but gradual; the degradation of your ability to debug is immediate, and it is what makes people reach for a cleanup policy. The arithmetic compounds quietly: an hourly job accumulates a year of tombstones, and a per-minute job gets there in a week.

Two mechanisms, two controllers

The CronJob controller trims by count. successfulJobsHistoryLimit defaults to 3 and failedJobsHistoryLimit defaults to 1, and setting either to 0 keeps none of that class. The trim is per class and oldest-first, so a chronically failing job keeps only its newest failure. Neither limit is enforced by the API; an absent policy is a valid policy, and a leaking one.

The TTL controller trims by age. ttlSecondsAfterFinished lives on the Job, and the controller deletes the Job that many seconds after it finishes, cascading to its pods. 0 makes the Job eligible immediately after it finishes; unset means the TTL controller never touches it. The feature is stable since v1.23 and reads the Job rather than the CronJob, which matters below. The two do not coordinate: whichever fires first wins, so a TTL shorter than your history window silently overrides the count you configured.

Owner references decide who cleans up

Jobs created by a CronJob carry an owner reference to that CronJob, so deleting the CronJob cascades to its Jobs and their pods. Cascading deletion is why deleting a CronJob looks like it cleaned everything up: the Jobs and pods go with it, and the history limits never enter into it. The count-based trim, however, is performed by the CronJob controller, which needs the CronJob to exist in order to know the limits. kubectl delete cronjob --cascade=orphan therefore leaves Jobs with a dangling owner reference and nothing left to trim them, while TTL still applies because it is evaluated per Job.

For Jobs you create directly rather than through a CronJob, the default deletion policy is orphanDependents: deleting the Job can leave its pods behind. The control plane garbage collects them eventually, but the docs warn that lingering pods can degrade a cluster and recommend ttlSecondsAfterFinished for exactly that reason. A cleanup policy is a retention decision, not a hygiene preference.

When it breaks

Setting both history limits to 0 to “keep the namespace clean” deletes every Job the moment it finishes, and the pod and its logs go with it. The one failure you needed to read is gone, with no recovery path. Keep a few successes and enough failures to cover a weekend; evidence you cannot read is not evidence.

TTL only counts from completion, so a Job that never finishes never ages out. A pod stuck in ImagePullBackOff leaves its Job active forever: not a success, not a failure, so neither history limit counts it. Give the job template an activeDeadlineSeconds so a stuck run fails, and let TTL collect it.

Check it quickly

Compare what is configured against what is actually accumulating:

kubectl get cronjob hourly-rollup -o jsonpath='{.spec.successfulJobsHistoryLimit} {.spec.failedJobsHistoryLimit} {.spec.jobTemplate.spec.ttlSecondsAfterFinished}{"\n"}'
kubectl get jobs --sort-by=.metadata.creationTimestamp --no-headers | wc -l
kubectl get pods --field-selector=status.phase=Succeeded --no-headers | wc -l
kubectl get jobs -o jsonpath='{range .items[*]}{.metadata.name} ttl={.spec.ttlSecondsAfterFinished} owner={.metadata.ownerReferences[0].name}{"\n"}{end}'

Two empty values on the first line mean nothing bounds retention. A steady Job count near the sum of the two limits means the CronJob controller is trimming; a count that grows with the clock means it is not. In the last command, a Job with an empty ttl and an empty owner is the orphan case, and neither controller will ever remove it.

Sources

Kubernetes Job, “Clean up finished jobs automatically” and the TTL mechanism for finished Jobs, checked 2026-09-16.
Kubernetes CronJob, default history limits and the job template, checked 2026-09-16.
Kubernetes garbage collection, owner references and cascading deletion, checked 2026-09-16.

A CronJob turns a schedule into a pod

Fri, 17 Dec 2027 00:00:00 +0000

A CronJob is a Job template plus a cron expression. Every time the expression matches, the controller creates a new Job, and that Job runs a pod to completion. Nothing stays resident between runs: a batch job inherits the cluster’s retry and scheduling machinery without a daemon kept alive to trigger it.

The tension is that a schedule is only approximately honoured: Kubernetes promises a Job “approximately once per execution time”, covering both failure directions — two Jobs for one tick, or none at all. Anything you schedule has to be safe to run twice, because the guarantee is at-least-once, not exactly-once.

A schedule makes a job

schedule takes the familiar five fields — minute, hour, day of month, month, day of week — with Vixie-cron ranges and steps, so */10 * * * * fires every ten minutes and */15 9-17 * * mon-fri stays inside business hours. ? means *, and @daily and @hourly expand to their equivalents. A CronJob with no timeZone follows the controller manager’s clock, so moving the control plane changes when it runs. A missing field is a validation error, one of the few CronJob mistakes that fails loudly.

Each match produces a separate Job named after the CronJob, which is why the CronJob name is capped at 52 characters: the controller appends eleven characters and the Job name must still fit in 63. A CronJob never runs the work itself; it only creates Jobs, which own the pods. Since Kubernetes v1.32 the Job also carries a batch.kubernetes.io/cronjob-scheduled-timestamp annotation holding the time it was meant to run, the only way to tell a late run from an on-time one after a controller restart.

Concurrency is a policy, not a guess

concurrencyPolicy decides what happens when a tick arrives while the previous Job is still running. Allow, the default, starts a second copy. Forbid skips the tick. Replace cancels the running Job and starts a fresh one. The policy applies only to Jobs from the same CronJob, so two CronJobs never block each other.

Replace is the dangerous one: if the Job reliably outruns its interval, every tick kills the run that was about to finish and the work never completes. Forbid is usually what people mean by “don’t overlap”, with two caveats — skipped runs are lost, and once the running Job finishes, startingDeadlineSeconds is still evaluated, so a long-overdue run may start anyway.

Missed runs have a budget of a hundred

startingDeadlineSeconds bounds how late a missed run is still worth starting; past that deadline the run is skipped and counted as failed, and skipping one occurrence does not stop the schedule, because the next matching time still fires. Left unset, missed runs have no deadline of their own, but they are still subject to the controller’s hard cap. It counts the schedules missed since the last scheduled time, and above one hundred it starts nothing and logs too many missed start times. Set or decrease .spec.startingDeadlineSeconds or check clock skew.

The documented example is concrete. A per-minute CronJob whose controller is down from 08:29 to 10:21 has missed 112 schedules, so nothing catches up on recovery. Add startingDeadlineSeconds: 200 and the controller instead counts only the schedules missed inside the last 200 seconds — three of them — and the Job does start. The same field hides a trap: values below ten seconds are unreliable, because the controller only re-evaluates every ten seconds. suspend: true parks a CronJob without touching Jobs already running, but suspended executions count as missed, so unsuspending without a deadline fires the whole backlog at once.

When it breaks

The classic failure is a Job that overlaps itself. A twenty-five minute report on a ten-minute schedule, with the default Allow, peaks at three concurrent copies reading the same input and writing the same output. It never presents as an error: duplicated rows, doubled API spend, or a lock timeout in one copy out of three. Either move to Forbid and accept skipped runs, or make the Job idempotent with a lease so the late copy exits early.

The second failure is silent catch-up loss. A control-plane outage longer than a hundred schedule intervals means the whole backlog is discarded on recovery, and the only trace is a line in the controller manager’s log. That is right for a per-minute poll and wrong for a nightly backup, which is what startingDeadlineSeconds is for: setting it to the real tolerance — eight hours for a twice-daily backup — both bounds how stale a run may be and keeps the hundred-interval cap permanently out of reach.

Check it quickly

Read the schedule, the policy and the last firing time together:

kubectl get cronjob nightly-report -o jsonpath='{.spec.schedule} {.spec.concurrencyPolicy} {.spec.startingDeadlineSeconds}{"\n"}'
kubectl get cronjob nightly-report -o jsonpath='{.status.lastScheduleTime}{"\n"}'
kubectl get jobs --sort-by=.metadata.creationTimestamp -o custom-columns=NAME:.metadata.name,START:.status.startTime,SUCCEEDED:.status.succeeded
kubectl get events --field-selector involvedObject.kind=CronJob --sort-by=.lastTimestamp | tail -5

lastScheduleTime should sit within one schedule interval of now; a gap of hours means the controller was down or the expression never matched. Overlapping START values with SUCCEEDED empty mean Allow plus a Job that outruns its interval, and a too many missed start times event names the missed-run rule directly.

Sources

Kubernetes CronJob, schedule syntax, concurrency policy, the 100-missed-schedule rule and history limits, checked 2026-09-16.
CronJob v1 API reference, field-level definition of the CronJob spec, checked 2026-09-16.