Scheduler · XB Field Notes

The scheduler accounts for requests, not momentary utilization

Sun, 02 Apr 2028 00:00:00 +0000

Kubernetes scheduling answers whether a Pod can fit a node under the applicable resource requests and placement constraints. It does not simply look at a current CPU utilization chart and place work wherever the line is low. Requests provide accounting inputs for future demand; actual usage is a separate observation. Diagnose both before deciding that an idle-looking cluster has capacity for a particular Pod.

Account for the admitted specification

For ordinary application containers, the scheduler accounts for their requested resources when testing node fit. Initialization requirements, Pod overhead and supported Pod-level resource configuration can also affect the effective request. The final admitted Pod is the relevant input, including defaults and mutations, rather than just the source manifest that an operator remembers submitting.

Node allocatable is also different from the hardware’s total capacity. It represents the resource amount available to Pods under the node’s configuration. Other admitted workloads consume scheduling headroom through their requests even when their processes happen to be idle. A request does not mean that physical memory has already been allocated to the application or that CPU cores are exclusively assigned.

Limits constrain running workloads through the execution stack; they are not generally substituted for an explicitly smaller request in scheduler fit checks. A container can therefore have a CPU limit larger than one node’s CPU capacity and still schedule with a small request. That accepted limit does not manufacture additional processors or promise the process can reach the declared ceiling.

When it breaks

A Pending Pod may be blocked by insufficient requested resources, but also by taints, affinity, storage, topology or other constraints. Read the PodScheduled condition and scheduling events before changing resource values. Lowering requests cannot repair an incompatible node selector or unavailable volume, and blindly lowering an honest request can create runtime contention instead of a sound placement.

Defaulting makes omission easy to misread. When a limit is specified without a request, and no admission-time mechanism supplies another request, Kubernetes copies that resource’s limit into its request. Setting only a large limit can therefore produce a large scheduling demand. Inspect both maps on the stored object, including any LimitRange effects, before interpreting a failed placement.

The API also rejects a request greater than its corresponding limit. That is an admission failure, not a scheduler decision about cluster capacity. Keeping those failure stages separate helps explain why one manifest never creates a Pod while another produces an accepted but unscheduled Pod that can be inspected.

QoS is not an immunity promise. Under conventional per-container rules, Guaranteed requires CPU and memory requests and limits to be present and equal for every relevant container. Matching only memory is insufficient. Even a Guaranteed Pod can encounter resource failure; the class does not replace priority, capacity planning, application correctness or actual node-pressure analysis.

Check it quickly

Use a disposable Kubernetes cluster with at least one Ready, untainted worker, kubectl and python:3.11-slim-bookworm on its nodes. Save this as request-placement.py and run python3 request-placement.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. The account needs namespace creation and deletion permissions. The recorded server was Kubernetes 1.34.11, and the selected worker reported ten allocatable CPUs.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

from decimal import Decimal, ROUND_CEILING

def cpu_quantity(value):
 return Decimal(value[:-1]) / 1000 if value.endswith('m') else Decimal(value)

namespace = 'notes-requests-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 workers = [n for n in obj(namespace, 'nodes')['items'] if
 not n['spec'].get('unschedulable') and not n['spec'].get('taints')
 and any(c['type'] == 'Ready' and c['status'] == 'True' for c in n['status']['conditions'])]
 assert workers, 'a Ready worker without taints is required'
 worker = workers[0]
 allocatable = cpu_quantity(worker['status']['allocatable']['cpu'])
 too_large = str(int(allocatable.to_integral_value(rounding=ROUND_CEILING)) + 1)
 for name, request in [('burst', '10m'), ('oversized', too_large)]:
 podspec = spec()
 podspec['nodeSelector'] = {'kubernetes.io/hostname': worker['metadata']['labels']['kubernetes.io/hostname']}
 podspec['containers'][0]['resources'] = {
 'requests': {'cpu': request, 'memory': '24Mi'}, 'limits': {'cpu': too_large, 'memory': '96Mi'}}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': name}, 'spec': podspec})
 burst = wait_for(lambda: obj(namespace, 'pod', 'burst'), pod_ready)
 assert burst['spec']['nodeName'] == worker['metadata']['name']
 def insufficient(p):
 return any(c['type'] == 'PodScheduled' and c['status'] == 'False'
 and c.get('reason') == 'Unschedulable' and 'Insufficient cpu' in c.get('message', '')
 for c in p.get('status', {}).get('conditions', []))
 pending = wait_for(lambda: obj(namespace, 'pod', 'oversized'), insufficient)
 assert pending['status']['phase'] == 'Pending' and not pending['spec'].get('nodeName')
 only_limit = spec()
 only_limit['containers'][0]['resources'] = {'limits': {'cpu': '200m', 'memory': '64Mi'}}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'defaulted'}, 'spec': only_limit})
 defaulted = wait_for(lambda: obj(namespace, 'pod', 'defaulted'), pod_ready)
 resources = defaulted['spec']['containers'][0]['resources']
 assert resources['requests'] == resources['limits'] == {'cpu': '200m', 'memory': '64Mi'}
 assert defaulted['status']['qosClass'] == 'Guaranteed'
 invalid = spec()
 invalid['containers'][0]['resources'] = {'requests': {'cpu': '200m'}, 'limits': {'cpu': '100m'}}
 denied = k('create', '-n', namespace, '-f', '-', payload={
 'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'invalid'}, 'spec': invalid}, expected=None)
 assert denied.returncode != 0 and 'must be less than or equal to cpu limit' in denied.stderr
 print(json.dumps({'node_allocatable_cpu': str(allocatable), 'limit_cpu': too_large,
 'small_request_with_large_limit_scheduled': True,
 'larger_than_node_request_pending': True,
 'limit_only_defaulted_resources': resources,
 'defaulted_qos': defaulted['status']['qosClass'],
 'request_above_limit_rejected': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The script uses nodeSelector to target that worker while leaving the scheduler responsible for assignment. Both comparison Pods have an eleven-CPU limit. The first requests only ten millicpu and becomes Ready on the selected node. The second requests eleven CPUs, remains Pending without nodeName, and reports Unschedulable with Insufficient cpu. Neither Pod tries to consume eleven CPUs; their processes are idle.

A third Pod specifies only 200m CPU and 64Mi memory limits. The stored object contains identical requests, and its observed QoS class is Guaranteed. A fourth attempted Pod requests 200m while limiting CPU to 100m; the API rejects it. All assertions passed, and the namespace was removed.

This separates successful placement, scheduler rejection, defaulting and API validation with real objects. It does not test actual peak capacity, resource-manager exclusivity or every scheduling plugin. Size requests from workload evidence and review actual admitted demand alongside usage, instead of turning an idle chart or a familiar ratio into a placement guarantee.

Sources

Kubernetes: Resource Management, request-based placement, limits and admission defaulting, checked 2026-09-13.
Kubernetes: Pod Quality of Service Classes, Guaranteed requirements and the limits of QoS protection, checked 2026-09-13.

Kubernetes CronJobs need a history policy

Fri, 25 Feb 2028 00:00:00 +0000

A CronJob does not run your work; it creates a Job, and the Job creates a pod. When the pod finishes, neither the Job nor the pod disappears. The pod is kept on purpose so you can still read its logs after the fact, and the Job is kept so you can still read its status.

That default is right for one run and wrong for ten thousand. Every finished Job is an object in etcd, a row in every list response, and a watch event for every controller that cares about Jobs. Two separate mechanisms bound the pile, and they are independent: count-based history limits applied by the CronJob controller, and age-based TTL applied by the TTL controller.

A finished Job is not free

A completed Job keeps its pod, and a pod keeps its container logs for as long as the pod exists. Retention is the only thing between you and a namespace full of tombstones. Thirty days of an hourly schedule is 720 runs: 720 Jobs and 720 pods, each with a name, a status, an owner reference and a line in every kubectl get jobs. A finished Job is not inert either: its status conditions stay in the API for every watcher to re-read. The cost per object is trivial, but it is linear in time, so it stays invisible until it is a problem.

The visibility cost arrives before the resource cost. Once the namespace holds hundreds of finished Jobs, kubectl get jobs no longer fits on a screen and a genuine failure is one more line among the successes. API server memory pressure is real but gradual; the degradation of your ability to debug is immediate, and it is what makes people reach for a cleanup policy. The arithmetic compounds quietly: an hourly job accumulates a year of tombstones, and a per-minute job gets there in a week.

Two mechanisms, two controllers

The CronJob controller trims by count. successfulJobsHistoryLimit defaults to 3 and failedJobsHistoryLimit defaults to 1, and setting either to 0 keeps none of that class. The trim is per class and oldest-first, so a chronically failing job keeps only its newest failure. Neither limit is enforced by the API; an absent policy is a valid policy, and a leaking one.

The TTL controller trims by age. ttlSecondsAfterFinished lives on the Job, and the controller deletes the Job that many seconds after it finishes, cascading to its pods. 0 makes the Job eligible immediately after it finishes; unset means the TTL controller never touches it. The feature is stable since v1.23 and reads the Job rather than the CronJob, which matters below. The two do not coordinate: whichever fires first wins, so a TTL shorter than your history window silently overrides the count you configured.

Owner references decide who cleans up

Jobs created by a CronJob carry an owner reference to that CronJob, so deleting the CronJob cascades to its Jobs and their pods. Cascading deletion is why deleting a CronJob looks like it cleaned everything up: the Jobs and pods go with it, and the history limits never enter into it. The count-based trim, however, is performed by the CronJob controller, which needs the CronJob to exist in order to know the limits. kubectl delete cronjob --cascade=orphan therefore leaves Jobs with a dangling owner reference and nothing left to trim them, while TTL still applies because it is evaluated per Job.

For Jobs you create directly rather than through a CronJob, the default deletion policy is orphanDependents: deleting the Job can leave its pods behind. The control plane garbage collects them eventually, but the docs warn that lingering pods can degrade a cluster and recommend ttlSecondsAfterFinished for exactly that reason. A cleanup policy is a retention decision, not a hygiene preference.

When it breaks

Setting both history limits to 0 to “keep the namespace clean” deletes every Job the moment it finishes, and the pod and its logs go with it. The one failure you needed to read is gone, with no recovery path. Keep a few successes and enough failures to cover a weekend; evidence you cannot read is not evidence.

TTL only counts from completion, so a Job that never finishes never ages out. A pod stuck in ImagePullBackOff leaves its Job active forever: not a success, not a failure, so neither history limit counts it. Give the job template an activeDeadlineSeconds so a stuck run fails, and let TTL collect it.

Check it quickly

Compare what is configured against what is actually accumulating:

kubectl get cronjob hourly-rollup -o jsonpath='{.spec.successfulJobsHistoryLimit} {.spec.failedJobsHistoryLimit} {.spec.jobTemplate.spec.ttlSecondsAfterFinished}{"\n"}'
kubectl get jobs --sort-by=.metadata.creationTimestamp --no-headers | wc -l
kubectl get pods --field-selector=status.phase=Succeeded --no-headers | wc -l
kubectl get jobs -o jsonpath='{range .items[*]}{.metadata.name} ttl={.spec.ttlSecondsAfterFinished} owner={.metadata.ownerReferences[0].name}{"\n"}{end}'

Two empty values on the first line mean nothing bounds retention. A steady Job count near the sum of the two limits means the CronJob controller is trimming; a count that grows with the clock means it is not. In the last command, a Job with an empty ttl and an empty owner is the orphan case, and neither controller will ever remove it.

Sources

Kubernetes Job, “Clean up finished jobs automatically” and the TTL mechanism for finished Jobs, checked 2026-09-16.
Kubernetes CronJob, default history limits and the job template, checked 2026-09-16.
Kubernetes garbage collection, owner references and cascading deletion, checked 2026-09-16.

Cron system and user tables have different contracts

Wed, 12 Jan 2028 00:00:00 +0000

Cron accepts schedules from more than one kind of table. A user crontab belongs to one account, while system tables can name the account for each entry. Copying a line between them without adjusting its fields changes what cron interprets as the command. Identify the table format before debugging an apparently correct five-field calendar expression.

The account column changes the parser boundary

A conventional user entry contains minute, hour, day of month, month, day of week, then command text. The crontab command installs it for the selected user. Debian’s /etc/crontab and files under /etc/cron.d add a username after the five schedule fields. That username belongs to the system-table format; it is not an optional annotation in a user’s command.

The month constraint remains required. When both day-of-month and day-of-week fields are restricted, the Debian/Vixie day rule permits either day match; month and weekday are not alternatives. Check the scheduler’s documented dialect before transferring expressions to products with seconds fields or different day rules. Familiar-looking spacing does not establish compatible semantics.

System drop-ins have ownership, permission, filename, and final-newline requirements. On Debian, use root-owned files that are not group- or other-writable and accepted names rather than arbitrary backup suffixes. Periodic directories such as /etc/cron.daily are invoked through the distribution’s scheduling arrangement, commonly using run-parts; every script is not independently parsed as a crontab. Inspect the installed arrangement before deciding when it runs.

When it breaks

Putting root after the five fields in root’s user crontab asks the shell to execute a command named root. Installation success cannot prove that this command exists. Conversely, omitting the account from a system entry moves command text into the username position. Review both the parser boundary and the effective identity, especially when the job writes files or accesses protected data.

The job environment is also a contract. Do not assume interactive aliases, shell startup files, or variables from the daemon’s parent shell are available. Set required values explicitly and choose an intentional PATH and interpreter. Environment settings in a cron.d file should not be assumed to inherit those in /etc/crontab. Test permissions and outputs as the actual execution account.

Percent signs have special handling before the shell sees the complete command. Unescaped percent characters delimit standard input, with subsequent percents becoming newlines. Shell quoting alone does not remove cron’s interpretation. Keep complex commands in scripts and escape literal percent signs when they must appear directly in a table entry. Separate scheduling syntax checks from command execution and application outcome checks.

Check it quickly

Use a fresh disposable Debian 12 Docker container as root with python3, cron, procps, and passwd installed. Ensure it has no active cron daemon, root crontab, or account named xbtime. Save this as cron-tables.py and run python3 cron-tables.py. It creates only its temporary account, task table, and files, waits up to seventy-five seconds for a real scheduled minute, then removes them.

from pathlib import Path
from tempfile import TemporaryDirectory
import json
import os
import pwd
import shutil
import subprocess
import time

assert os.geteuid() == 0 and Path('/.dockerenv').exists(), 'use a disposable root Linux container'
assert subprocess.run(['pgrep', '-x', 'cron'], capture_output=True).returncode == 1
assert subprocess.run(['crontab', '-l'], capture_output=True).returncode != 0
assert shutil.which('root') is None
username = 'xbtime'
try:
 pwd.getpwnam(username)
except KeyError:
 pass
else:
 raise AssertionError('test account already exists')
system_table = Path('/etc/cron.d/xbcatm_time_test')
assert not system_table.exists()
subprocess.run(['useradd', '--create-home', '--shell', '/bin/sh', username], check=True)
account = pwd.getpwnam(username)
daemon = None
try:
 with TemporaryDirectory(prefix='cron-tables-') as directory:
 root = Path(directory)
 os.chown(root, account.pw_uid, account.pw_gid)
 worker = root / 'worker.py'
 worker.write_text('''from pathlib import Path
import json,os,sys
root=Path(__file__).parent
mode=sys.argv[1]
data={'uid':os.getuid(),'table':os.environ.get('LAB_TABLE'),
 'path':os.environ.get('PATH'),'shell':os.environ.get('SHELL'),
 'parent_only':os.environ.get('PARENT_ONLY')}
if mode=='percent': data['stdin']=sys.stdin.read()
(root/(mode+'.json')).write_text(json.dumps(data))
''')
 user_table = ('SHELL=/bin/sh\nPATH=/usr/bin:/bin\nLAB_TABLE=user\n'
 f'* * * * * /usr/bin/python3 {worker} user\n'
 f'* * * * * /usr/bin/python3 {worker} percent%first%second\n'
 f'* * * * * root /usr/bin/id >{root}/wrong-column.err 2>&1\n')
 subprocess.run(['crontab', '-'], input=user_table, text=True, check=True)
 invalid = subprocess.run(['crontab', '-'], input='* * * * * /bin/true',
 text=True, capture_output=True)
 assert invalid.returncode != 0
 assert subprocess.check_output(['crontab', '-l'], text=True) == user_table
 system_table.write_text('SHELL=/bin/sh\nPATH=/usr/bin:/bin\nLAB_TABLE=system\n'
 f'* * * * * {username} /usr/bin/python3 {worker} system\n')
 system_table.chmod(0o644)
 daemon = subprocess.Popen(['cron', '-f'], env={**os.environ, 'PARENT_ONLY': 'parent-demo'},
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
 try:
 deadline = time.monotonic() + 75
 names = ['user.json', 'system.json', 'percent.json', 'wrong-column.err']
 while time.monotonic() < deadline:
 assert daemon.poll() is None
 if all((root/name).exists() and (root/name).stat().st_size for name in names):
 break
 time.sleep(0.2)
 else:
 raise AssertionError('scheduled table outputs were not produced')
 user = json.loads((root/'user.json').read_text())
 system = json.loads((root/'system.json').read_text())
 percent = json.loads((root/'percent.json').read_text())
 wrong = (root/'wrong-column.err').read_text()
 assert user['uid'] == 0 and user['table'] == 'user'
 assert system['uid'] == account.pw_uid and system['table'] == 'system'
 assert all(value['parent_only'] is None for value in (user, system))
 assert all(value['path'] == '/usr/bin:/bin' for value in (user, system))
 assert percent['stdin'] == 'first\nsecond\n'
 assert 'root: not found' in wrong
 print(json.dumps({'user_table': user, 'system_table': system,
 'missing_final_newline_rejected': invalid.returncode,
 'previous_table_preserved': True, 'percent_stdin': percent['stdin'],
 'username_in_user_table_fails': wrong.strip()}), flush=True)
 finally:
 daemon.terminate()
 daemon.communicate(timeout=5)
 daemon = None
finally:
 if daemon is not None:
 daemon.terminate()
 daemon.communicate(timeout=5)
 system_table.unlink(missing_ok=True)
 subprocess.run(['crontab', '-r'], capture_output=True)
 subprocess.run(['userdel', '--remove', username], capture_output=True, check=True)

The user-table job runs as UID zero. The system-table job runs as the newly created account, UID 1000 in this run. Each receives its own LAB_TABLE value and explicit PATH; neither receives the daemon parent’s PARENT_ONLY variable. A percent-bearing entry supplies first\nsecond\n on standard input. The wrongly inserted username produces root: not found, despite its table installing successfully.

An attempted replacement missing its final newline is rejected with exit one, and the previous table remains intact. All assertions passed on cron 3.0pl1-162. Separate probes found that this package accepted malformed minute text even with its dry-run option, so no universal field-validation guarantee is inferred from a zero exit status. The exercise verifies observed account, environment, parsing, and installation behavior; it does not test mail delivery, daylight-saving handling, or business success. Keep those additional requirements visible when deploying a real scheduled task.

Sources

Debian 12: crontab(5), user/system table syntax, environment and percent handling, checked 2026-09-13.
Debian 12: cron(8), system drop-ins, ownership and execution environment, checked 2026-09-13.
Debian 12: crontab(1), installation, dry-run behavior and newline requirements, checked 2026-09-13.