Kubernetes · XB Field Notes

The scheduler accounts for requests, not momentary utilization

Sun, 02 Apr 2028 00:00:00 +0000

Kubernetes scheduling answers whether a Pod can fit a node under the applicable resource requests and placement constraints. It does not simply look at a current CPU utilization chart and place work wherever the line is low. Requests provide accounting inputs for future demand; actual usage is a separate observation. Diagnose both before deciding that an idle-looking cluster has capacity for a particular Pod.

Account for the admitted specification

For ordinary application containers, the scheduler accounts for their requested resources when testing node fit. Initialization requirements, Pod overhead and supported Pod-level resource configuration can also affect the effective request. The final admitted Pod is the relevant input, including defaults and mutations, rather than just the source manifest that an operator remembers submitting.

Node allocatable is also different from the hardware’s total capacity. It represents the resource amount available to Pods under the node’s configuration. Other admitted workloads consume scheduling headroom through their requests even when their processes happen to be idle. A request does not mean that physical memory has already been allocated to the application or that CPU cores are exclusively assigned.

Limits constrain running workloads through the execution stack; they are not generally substituted for an explicitly smaller request in scheduler fit checks. A container can therefore have a CPU limit larger than one node’s CPU capacity and still schedule with a small request. That accepted limit does not manufacture additional processors or promise the process can reach the declared ceiling.

When it breaks

A Pending Pod may be blocked by insufficient requested resources, but also by taints, affinity, storage, topology or other constraints. Read the PodScheduled condition and scheduling events before changing resource values. Lowering requests cannot repair an incompatible node selector or unavailable volume, and blindly lowering an honest request can create runtime contention instead of a sound placement.

Defaulting makes omission easy to misread. When a limit is specified without a request, and no admission-time mechanism supplies another request, Kubernetes copies that resource’s limit into its request. Setting only a large limit can therefore produce a large scheduling demand. Inspect both maps on the stored object, including any LimitRange effects, before interpreting a failed placement.

The API also rejects a request greater than its corresponding limit. That is an admission failure, not a scheduler decision about cluster capacity. Keeping those failure stages separate helps explain why one manifest never creates a Pod while another produces an accepted but unscheduled Pod that can be inspected.

QoS is not an immunity promise. Under conventional per-container rules, Guaranteed requires CPU and memory requests and limits to be present and equal for every relevant container. Matching only memory is insufficient. Even a Guaranteed Pod can encounter resource failure; the class does not replace priority, capacity planning, application correctness or actual node-pressure analysis.

Check it quickly

Use a disposable Kubernetes cluster with at least one Ready, untainted worker, kubectl and python:3.11-slim-bookworm on its nodes. Save this as request-placement.py and run python3 request-placement.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. The account needs namespace creation and deletion permissions. The recorded server was Kubernetes 1.34.11, and the selected worker reported ten allocatable CPUs.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

from decimal import Decimal, ROUND_CEILING

def cpu_quantity(value):
 return Decimal(value[:-1]) / 1000 if value.endswith('m') else Decimal(value)

namespace = 'notes-requests-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 workers = [n for n in obj(namespace, 'nodes')['items'] if
 not n['spec'].get('unschedulable') and not n['spec'].get('taints')
 and any(c['type'] == 'Ready' and c['status'] == 'True' for c in n['status']['conditions'])]
 assert workers, 'a Ready worker without taints is required'
 worker = workers[0]
 allocatable = cpu_quantity(worker['status']['allocatable']['cpu'])
 too_large = str(int(allocatable.to_integral_value(rounding=ROUND_CEILING)) + 1)
 for name, request in [('burst', '10m'), ('oversized', too_large)]:
 podspec = spec()
 podspec['nodeSelector'] = {'kubernetes.io/hostname': worker['metadata']['labels']['kubernetes.io/hostname']}
 podspec['containers'][0]['resources'] = {
 'requests': {'cpu': request, 'memory': '24Mi'}, 'limits': {'cpu': too_large, 'memory': '96Mi'}}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': name}, 'spec': podspec})
 burst = wait_for(lambda: obj(namespace, 'pod', 'burst'), pod_ready)
 assert burst['spec']['nodeName'] == worker['metadata']['name']
 def insufficient(p):
 return any(c['type'] == 'PodScheduled' and c['status'] == 'False'
 and c.get('reason') == 'Unschedulable' and 'Insufficient cpu' in c.get('message', '')
 for c in p.get('status', {}).get('conditions', []))
 pending = wait_for(lambda: obj(namespace, 'pod', 'oversized'), insufficient)
 assert pending['status']['phase'] == 'Pending' and not pending['spec'].get('nodeName')
 only_limit = spec()
 only_limit['containers'][0]['resources'] = {'limits': {'cpu': '200m', 'memory': '64Mi'}}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'defaulted'}, 'spec': only_limit})
 defaulted = wait_for(lambda: obj(namespace, 'pod', 'defaulted'), pod_ready)
 resources = defaulted['spec']['containers'][0]['resources']
 assert resources['requests'] == resources['limits'] == {'cpu': '200m', 'memory': '64Mi'}
 assert defaulted['status']['qosClass'] == 'Guaranteed'
 invalid = spec()
 invalid['containers'][0]['resources'] = {'requests': {'cpu': '200m'}, 'limits': {'cpu': '100m'}}
 denied = k('create', '-n', namespace, '-f', '-', payload={
 'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'invalid'}, 'spec': invalid}, expected=None)
 assert denied.returncode != 0 and 'must be less than or equal to cpu limit' in denied.stderr
 print(json.dumps({'node_allocatable_cpu': str(allocatable), 'limit_cpu': too_large,
 'small_request_with_large_limit_scheduled': True,
 'larger_than_node_request_pending': True,
 'limit_only_defaulted_resources': resources,
 'defaulted_qos': defaulted['status']['qosClass'],
 'request_above_limit_rejected': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The script uses nodeSelector to target that worker while leaving the scheduler responsible for assignment. Both comparison Pods have an eleven-CPU limit. The first requests only ten millicpu and becomes Ready on the selected node. The second requests eleven CPUs, remains Pending without nodeName, and reports Unschedulable with Insufficient cpu. Neither Pod tries to consume eleven CPUs; their processes are idle.

A third Pod specifies only 200m CPU and 64Mi memory limits. The stored object contains identical requests, and its observed QoS class is Guaranteed. A fourth attempted Pod requests 200m while limiting CPU to 100m; the API rejects it. All assertions passed, and the namespace was removed.

This separates successful placement, scheduler rejection, defaulting and API validation with real objects. It does not test actual peak capacity, resource-manager exclusivity or every scheduling plugin. Size requests from workload evidence and review actual admitted demand alongside usage, instead of turning an idle chart or a familiar ratio into a placement guarantee.

Sources

Kubernetes: Resource Management, request-based placement, limits and admission defaulting, checked 2026-09-13.
Kubernetes: Pod Quality of Service Classes, Guaranteed requirements and the limits of QoS protection, checked 2026-09-13.

Kubernetes CronJobs need a history policy

Fri, 25 Feb 2028 00:00:00 +0000

A CronJob does not run your work; it creates a Job, and the Job creates a pod. When the pod finishes, neither the Job nor the pod disappears. The pod is kept on purpose so you can still read its logs after the fact, and the Job is kept so you can still read its status.

That default is right for one run and wrong for ten thousand. Every finished Job is an object in etcd, a row in every list response, and a watch event for every controller that cares about Jobs. Two separate mechanisms bound the pile, and they are independent: count-based history limits applied by the CronJob controller, and age-based TTL applied by the TTL controller.

A finished Job is not free

A completed Job keeps its pod, and a pod keeps its container logs for as long as the pod exists. Retention is the only thing between you and a namespace full of tombstones. Thirty days of an hourly schedule is 720 runs: 720 Jobs and 720 pods, each with a name, a status, an owner reference and a line in every kubectl get jobs. A finished Job is not inert either: its status conditions stay in the API for every watcher to re-read. The cost per object is trivial, but it is linear in time, so it stays invisible until it is a problem.

The visibility cost arrives before the resource cost. Once the namespace holds hundreds of finished Jobs, kubectl get jobs no longer fits on a screen and a genuine failure is one more line among the successes. API server memory pressure is real but gradual; the degradation of your ability to debug is immediate, and it is what makes people reach for a cleanup policy. The arithmetic compounds quietly: an hourly job accumulates a year of tombstones, and a per-minute job gets there in a week.

Two mechanisms, two controllers

The CronJob controller trims by count. successfulJobsHistoryLimit defaults to 3 and failedJobsHistoryLimit defaults to 1, and setting either to 0 keeps none of that class. The trim is per class and oldest-first, so a chronically failing job keeps only its newest failure. Neither limit is enforced by the API; an absent policy is a valid policy, and a leaking one.

The TTL controller trims by age. ttlSecondsAfterFinished lives on the Job, and the controller deletes the Job that many seconds after it finishes, cascading to its pods. 0 makes the Job eligible immediately after it finishes; unset means the TTL controller never touches it. The feature is stable since v1.23 and reads the Job rather than the CronJob, which matters below. The two do not coordinate: whichever fires first wins, so a TTL shorter than your history window silently overrides the count you configured.

Owner references decide who cleans up

Jobs created by a CronJob carry an owner reference to that CronJob, so deleting the CronJob cascades to its Jobs and their pods. Cascading deletion is why deleting a CronJob looks like it cleaned everything up: the Jobs and pods go with it, and the history limits never enter into it. The count-based trim, however, is performed by the CronJob controller, which needs the CronJob to exist in order to know the limits. kubectl delete cronjob --cascade=orphan therefore leaves Jobs with a dangling owner reference and nothing left to trim them, while TTL still applies because it is evaluated per Job.

For Jobs you create directly rather than through a CronJob, the default deletion policy is orphanDependents: deleting the Job can leave its pods behind. The control plane garbage collects them eventually, but the docs warn that lingering pods can degrade a cluster and recommend ttlSecondsAfterFinished for exactly that reason. A cleanup policy is a retention decision, not a hygiene preference.

When it breaks

Setting both history limits to 0 to “keep the namespace clean” deletes every Job the moment it finishes, and the pod and its logs go with it. The one failure you needed to read is gone, with no recovery path. Keep a few successes and enough failures to cover a weekend; evidence you cannot read is not evidence.

TTL only counts from completion, so a Job that never finishes never ages out. A pod stuck in ImagePullBackOff leaves its Job active forever: not a success, not a failure, so neither history limit counts it. Give the job template an activeDeadlineSeconds so a stuck run fails, and let TTL collect it.

Check it quickly

Compare what is configured against what is actually accumulating:

kubectl get cronjob hourly-rollup -o jsonpath='{.spec.successfulJobsHistoryLimit} {.spec.failedJobsHistoryLimit} {.spec.jobTemplate.spec.ttlSecondsAfterFinished}{"\n"}'
kubectl get jobs --sort-by=.metadata.creationTimestamp --no-headers | wc -l
kubectl get pods --field-selector=status.phase=Succeeded --no-headers | wc -l
kubectl get jobs -o jsonpath='{range .items[*]}{.metadata.name} ttl={.spec.ttlSecondsAfterFinished} owner={.metadata.ownerReferences[0].name}{"\n"}{end}'

Two empty values on the first line mean nothing bounds retention. A steady Job count near the sum of the two limits means the CronJob controller is trimming; a count that grows with the clock means it is not. In the last command, a Job with an empty ttl and an empty owner is the orphan case, and neither controller will ever remove it.

Sources

Kubernetes Job, “Clean up finished jobs automatically” and the TTL mechanism for finished Jobs, checked 2026-09-16.
Kubernetes CronJob, default history limits and the job template, checked 2026-09-16.
Kubernetes garbage collection, owner references and cascading deletion, checked 2026-09-16.

cgroups enforce container budgets within a hierarchy

Fri, 11 Feb 2028 00:00:00 +0000

Kubernetes resource fields become useful constraints only when the node’s execution stack applies them. On Linux, kubelet and the container runtime use cgroups to organize processes and configure resource controllers. The result is a hierarchy, not merely one number attached to a Pod. Check the relevant container group and its inherited constraints before deciding what budget a process can actually consume.

Read the controller that enforces the resource

In cgroup v2, cpu.max expresses CPU bandwidth as quota and period in microseconds. A value of 10000 100000 permits one tenth of a CPU’s time per period for that group. It does not pin the process to one core or reduce the processor’s clock speed. Several runnable threads can consume a group’s allowance faster within the same period.

CPU weight serves a different purpose: it influences relative distribution under contention among eligible groups at the applicable hierarchy level. A Kubernetes CPU request can inform that weight, as well as scheduling placement. A request is not itself a hard runtime ceiling, and a weight does not guarantee an application’s wall-clock latency or exclusive ownership of a CPU.

The memory controller exposes limits and usage separately. memory.max describes the group’s hard memory boundary, while memory.current reports accounted use. Reclaim and OOM behavior are memory mechanisms, not CPU-style periodic throttling. Application heap size alone does not explain all accounted memory, so compare runtime memory metrics with the controller’s view when investigating a limit.

When it breaks

Hierarchical constraints can be tighter than one local file suggests. A child group with no local CPU cap can still be constrained by an ancestor. Container cgroup namespaces may hide parts of that hierarchy from the container’s view. Do not infer unrestricted node capacity from a local max value without checking the node-level context and effective parent constraints.

A multi-container Pod can have separate container limits. One container’s unused allowance is not automatically a transferable promise that another container may exceed its own cap. Pod-level resource features introduce additional configuration and version considerations; they should not be silently substituted for per-container settings. The experiment below deliberately uses only container resources and confirms that distinction in the admitted Pod.

High throttling can explain latency even when a host-wide CPU average looks modest. Conversely, one throttling counter cannot identify every source of delay. Read changes over a relevant interval, correlate workload demand and request latency, and remember that ancestor contention and other resources can also matter. A short synthetic loop establishes enforcement, not an application capacity plan.

Controller files also depend on the environment. cgroup v1 uses different interfaces, and CPU-manager policies or runtime configuration can alter placement and enforcement details. Verify the actual node setup rather than copying a pathname and assuming its meaning. The fixture requires the cgroup v2 interface and reads settings without changing node controller files.

Check it quickly

Use a disposable Linux Kubernetes cluster with cgroup v2, kubectl and python:3.11-slim-bookworm available on the nodes. Save the script as cgroup-enforcement.py and run python3 cgroup-enforcement.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. Namespace creation and deletion permissions are required. The recorded server was Kubernetes 1.34.11 using the kind container runtime setup.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-cgroup-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 podspec = spec()
 podspec['containers'] = []
 for name, request, limit, memory in [('small', '25m', '100m', '64Mi'), ('large', '50m', '200m', '96Mi')]:
 container = spec()['containers'][0]
 container['name'] = name
 container['resources'] = {'requests': {'cpu': request, 'memory': '24Mi'},
 'limits': {'cpu': limit, 'memory': memory}}
 podspec['containers'].append(container)
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'budgets'}, 'spec': podspec})
 wait_for(lambda: obj(namespace, 'pod', 'budgets'), pod_ready)
 snapshot = '''from pathlib import Path
import json,os
p=Path('/sys/fs/cgroup')
assert (p/'cgroup.controllers').exists(), 'this example requires cgroup v2'
print(json.dumps({'cpu_max':(p/'cpu.max').read_text().strip(),
 'cpu_weight':int((p/'cpu.weight').read_text()),
 'memory_max':int((p/'memory.max').read_text()),
 'group_inode':os.stat(p).st_ino}))
'''
 observations = {}
 for name, expected_cpu, expected_memory in [('small', 0.1, 64), ('large', 0.2, 96)]:
 data = json.loads(k('exec', '-n', namespace, 'budgets', '-c', name, '--', 'python3', '-c', snapshot).stdout)
 quota, period = map(int, data['cpu_max'].split())
 assert quota / period == expected_cpu
 assert data['memory_max'] == expected_memory * 1024 * 1024
 observations[name] = data
 assert observations['small']['group_inode'] != observations['large']['group_inode']
 load = '''from pathlib import Path
import json,time
p=Path('/sys/fs/cgroup/cpu.stat')
def counters(): return {k:int(v) for k,v in (line.split() for line in p.read_text().splitlines())}
before=counters(); begin=time.monotonic(); cpu=time.process_time(); value=0
while time.monotonic()-begin < 3: value=(value+1)%1000003
after=counters()
print(json.dumps({'wall_seconds':time.monotonic()-begin,'process_cpu_seconds':time.process_time()-cpu,
 'nr_throttled_delta':after['nr_throttled']-before['nr_throttled'],
 'throttled_usec_delta':after['throttled_usec']-before['throttled_usec']}))
'''
 throttling = json.loads(k('exec', '-n', namespace, 'budgets', '-c', 'small', '--', 'python3', '-c', load).stdout)
 assert throttling['nr_throttled_delta'] > 0 and throttling['throttled_usec_delta'] > 0
 final = obj(namespace, 'pod', 'budgets')
 assert pod_ready(final) and all(c['restartCount'] == 0 for c in final['status']['containerStatuses'])
 assert 'resources' not in final['spec']
 print(json.dumps({'container_groups': observations, 'measured_throttling': throttling,
 'container_restarts': 0, 'pod_level_resource_field_used': False}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

Two containers share one Pod but expose different cgroup directory identities. The smaller container reports cpu.max of 10000 100000 and memory.max of 67108864. The larger reports 20000 100000 and 100663296. Observed CPU weights are six and eleven; those values are runtime mappings, not portable constants that every installation must reproduce.

A three-second busy loop in the smaller container records thirty additional throttled periods and about 2.707 seconds of throttled time. Its process CPU time is about 0.298 seconds over 3.007 seconds elapsed. Both containers remain Ready with zero restarts. All assertions passed, and cleanup removed the namespace.

The test verifies distinct container settings and actual CPU throttling. It does not measure fairness between contending workloads, expose every ancestor setting, or exercise Pod-level resource allocation. Keep those boundaries explicit when turning low-level controller observations into operational conclusions.

Sources

Kubernetes: About cgroup v2, Linux resource-controller integration and prerequisites, checked 2026-09-13.
Kubernetes: Resource Management, container requests and limits passed to the runtime, checked 2026-09-13.
Linux: Control Group v2, hierarchy, CPU bandwidth, weights and memory accounting, checked 2026-09-13.

Cordon and drain take a node out safely

Thu, 03 Feb 2028 00:00:00 +0000

Cordon and drain are two actions people usually say in one breath. Cordon sets spec.unschedulable on the node so the scheduler stops choosing it. Drain evicts what is already running there. Cordon alone is instant and reversible; drain is a loop that talks to the eviction API once per pod.

The tension is that drain is a client-side procedure with a safety rail: it refuses before it deletes anything, and every flag that gets past a refusal trades some of that safety away. Knowing which flag covers which refusal is the difference between a maintenance window and an outage.

Cordon closes the door, drain clears the room

kubectl cordon node-1 patches spec.unschedulable: true, and the scheduler filters that node out for new pods. Pods already running keep serving, which is why you cordon first: stop accepting new work before removing existing work. Cordoning is also what makes the drain meaningful: without it the scheduler would place replacements straight back onto the node you are emptying. kubectl drain sets the same flag itself, so a separate cordon is a courtesy rather than a requirement. kubectl uncordon is the same patch in reverse, and the scheduler starts using the node again immediately — confirm the kubelet is Ready first.

Drain then evicts pods one at a time through the eviction API, falling back to a plain delete where the server does not. Eviction is a graceful termination: terminationGracePeriodSeconds is honoured and drain waits for it. Grace is a window, not a guarantee: a pod that ignores SIGTERM is killed when it expires, so the pod’s own shutdown logic protects the work. Mirror pods are the exception — the kubelet manages them from a static manifest and they cannot be deleted through the API server at all, so drain skips them rather than failing on them.

The refusal list is all or nothing

Drain refuses before it deletes anything, and one bad pod blocks every eviction on the node. The refusals map to three flags, and the binary names them: cannot delete Pods that declare no controller (use --force to override) for a bare pod or one whose managing resource has gone missing; Pods with local storage (use --delete-emptydir-data to override) for anything using emptyDir; and DaemonSet-managed Pods (use --ignore-daemonsets to ignore).

The DaemonSet case is the one where the flag does not change the outcome. Drain never deletes DaemonSet pods, with or without the flag, because the DaemonSet controller ignores the unschedulable taint and would recreate them immediately; --ignore-daemonsets only silences the refusal. Two other kinds of pod reach a drained node regardless: anything tolerating node.kubernetes.io/unschedulable, and anything that sets nodeName directly, bypassing the scheduler. Pods tolerating that taint are usually DaemonSets, which is why the docs say not to tolerate it for anything else.

The budget, not the flags, decides the rest

Once the refusals are cleared, drain starts submitting evictions and the PodDisruptionBudget has the last word. A PDB with disruptionsAllowed at zero answers 429, and drain retries with a backoff until --timeout, whose default of zero means forever. That retry loop is why a stalled drain looks like a hang, not a failure: drain is doing exactly what it was told. No error, no progress, just an evicting pod line repeating.

--disable-eviction is the flag that bypasses budgets, by deleting pods directly instead of asking the eviction API. --force does not do that; it only covers pods with no controller, the opposite of what people assume. The two flags are not interchangeable. The default unhealthy-pod policy, IfHealthyBudget, also lets one CrashLoopBackOff pod stall an entire drain until it becomes Ready.

When it breaks

The drain that never returns. A PDB with minAvailable equal to the replica count, or maxUnavailable: 0, resolves to zero allowed disruptions, so the eviction is rejected forever and the terminal just sits there. Draining several nodes in parallel does not help either; they share the same budget. Diagnose with kubectl get pdb -A: a PDB whose pods live on the node you are draining, showing ALLOWED DISRUPTIONS 0, is the culprit; the fix is a budget that permits one eviction, not a longer timeout.

The drain that takes the workload down. Faced with a stalled drain, someone reaches for --force, and the node empties — including the pods the budget was protecting. If that workload held quorum, the maintenance becomes the outage. The safe order is to fix the budget and wait, reserving --force for a pod you have verified is genuinely stateless.

Check it quickly

Confirm the node is closed and see what drain would refuse:

kubectl get node node-1 -o jsonpath='{.spec.unschedulable}{"\n"}'
kubectl drain node-1 --dry-run=client --ignore-daemonsets --delete-emptydir-data
kubectl get pods --field-selector spec.nodeName=node-1 -o custom-columns=NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind
kubectl get pdb -A -o custom-columns=NAME:.metadata.name,MIN:.spec.minAvailable,MAX:.spec.maxUnavailable,ALLOWED:.status.disruptionsAllowed

The first prints true once the node is cordoned. The dry run lists the pods drain would evict and the ones it refuses by name. In the third command an empty OWNER column means --force is required for that pod, and in the fourth an ALLOWED of 0 on a PDB whose pods are on this node means the drain will wait rather than fail.

Sources

Safely Drain a Node, the drain workflow and its interaction with PodDisruptionBudgets, checked 2026-09-16.
kubectl drain reference, the exact refusal conditions and flags, checked 2026-09-16.
Pod disruptions, voluntary versus involuntary disruption and eviction retries, checked 2026-09-16.

The Horizontal Pod Autoscaler scales by a target utilization

Wed, 22 Dec 2027 00:00:00 +0000

Because it can act on the mean of a series instead of a single spike, you need to pair it with a max-rc that keeps the maths from running forever.

A smoothed average decides

The HPA does not react to a single instantaneous read; it averages the observed value over a window and smooths it before comparing against the target. That is deliberate: a one-second spike in a metric should not repaint the replica count, so the controller aggregates first, then only changes the count when the smoothed value moves past the target. To make that work you choose the metric source (CPU, custom metrics API, or external) and the target, because the smoothing window and the aggregation method, per-pod average versus cluster level, together define what ’the load’ actually is. A metric read over too short a window reacts to noise; too long and it lags demand. Reading the document’s sync-period and aggregation paragraph is the first step in tuning, not the last.

Pick the metric you can act on

The real design choice is the metric, because scaling is only as useful as the target you give it. CPU is easy on any cluster and works well for CPU-bound workloads, but it is a proxy: a service waiting on queues or a database is often standing idle on CPU while its latency explodes. Demand-driven metrics such as request rate or a custom kafka_consumer_lag are harder to wire but report the thing users feel. Whatever you pick, choose one you can act on, a target that, when crossed, really changes the experience, and use the same definition the app uses, so the HPA is scaling the number that matters rather than a number that happens to be cheap to read.

Automation is orthogonal to defaults

Set the initial replicas in the Deployment because that is the starting point, not because the HPA reads it, once the HPA exists it owns the count and the initial value is only a floor or a first estimate. The two are orthogonal in the good way: you can disable the controller and still have a sane static stated count, and you can run a steady HPA without a fancy initial guess. Keep the minimum realistic (that is the real floor), pick a maximum that bounds spend, and remember that the initial value is what makes the very first seconds sane before the controller’s first reconcile lands.

When it breaks

A missing custom metric does not fall back to CPU. When the custom-metrics API returns an error or no series, the HPA marks ScalingActive False and holds replicas instead of guessing, so a workload with only a queue metric freezes exactly when it needed scaling. List CPU alongside the custom metric and let the maximum win, keeping a degraded but live input.

A Prometheus Adapter pipeline also adds delay. Scrape intervals plus adapter caching can serve queue depth from minutes ago, causing scale-up against drained work and overshoot. Check the series timestamp and adapter latency before retuning the target, and expose one pre-aggregated series per workload.

Cardinality breaks the same path differently: per-user or per-request labels multiply one signal into thousands of series, slow the API, and flip readings to Unknown. Aggregate in Prometheus first. A target-type mismatch does the same: custom metrics accept only Value or AverageValue, and CPU utilization without pod requests stays Unknown. Describe the HPA and align the type with what the pipeline serves.

Check it quickly

Save the following as hpa_metric_sim.py and run python3 hpa_metric_sim.py. It uses only the standard library, touches no cluster, and models the rules above: multiple metrics combine by maximum, missing metrics are skipped, and an all-missing poll holds replicas.

import math
import sys


def desired_for(current, value, target):
 """Replicas wanted by one metric; None means unavailable."""
 if value is None or target is None or target <= 0:
 return None
 return math.ceil(current * value / target)


def decide(current, metrics, min_replicas, max_replicas):
 """Combine metrics by maximum; hold when every metric is missing."""
 wants = []
 for _, value, target in metrics:
 want = desired_for(current, value, target)
 if want is not None:
 wants.append(want)
 if not wants:
 return current
 return max(min_replicas, min(max_replicas, max(wants)))


def check(actual, expected, message):
 if actual != expected:
 raise AssertionError(f"{message}: got {actual}, want {expected}")
 print(f"OK: {message} -> {actual}")


def main():
 current = 4
 cpu = ("cpu", 80, 50) # 4 * 80/50 = 6.4 -> 7
 queue = ("queue", 900, 300) # 4 * 900/300 = 12
 check(decide(current, [cpu, queue], 1, 20), 12, "max wins across metrics")
 check(decide(current, [cpu, ("queue", None, 300)], 1, 20), 7, "one missing metric still scales on CPU")
 check(decide(current, [("cpu", None, 50), ("queue", None, 300)], 1, 20), 4, "all missing holds replicas")
 check(decide(current, [cpu, queue], 1, 10), 10, "maxReplicas still clamps metric math")
 print("PASS: 4 scenarios; max-wins and hold-on-missing verified")
 print(sys.version.split()[0])


if __name__ == "__main__":
 main()

The four scenarios prove the maximum wins when CPU asks for seven and queue depth asks for twelve, a single missing series still leaves CPU in charge, all-missing holds instead of scaling to zero, and the maximum bound still clamps the result. Expected output ends with PASS: 4 scenarios; max-wins and hold-on-missing verified.

The assertions passed with Python 3.12.6 on macOS. The script checks replica math only, not scrape latency, adapter caching, or cardinality. Before changing production, verify timestamps and HPA events against the real custom-metrics API.

Sources

Kubernetes Horizontal Pod Autoscaling, documents algorithm and metric sources, checked 2026-06-01.
Resource Metrics Pipeline, documents metrics-server and the resource metrics API, checked 2026-06-01.

A CronJob turns a schedule into a pod

Fri, 17 Dec 2027 00:00:00 +0000

A CronJob is a Job template plus a cron expression. Every time the expression matches, the controller creates a new Job, and that Job runs a pod to completion. Nothing stays resident between runs: a batch job inherits the cluster’s retry and scheduling machinery without a daemon kept alive to trigger it.

The tension is that a schedule is only approximately honoured: Kubernetes promises a Job “approximately once per execution time”, covering both failure directions — two Jobs for one tick, or none at all. Anything you schedule has to be safe to run twice, because the guarantee is at-least-once, not exactly-once.

A schedule makes a job

schedule takes the familiar five fields — minute, hour, day of month, month, day of week — with Vixie-cron ranges and steps, so */10 * * * * fires every ten minutes and */15 9-17 * * mon-fri stays inside business hours. ? means *, and @daily and @hourly expand to their equivalents. A CronJob with no timeZone follows the controller manager’s clock, so moving the control plane changes when it runs. A missing field is a validation error, one of the few CronJob mistakes that fails loudly.

Each match produces a separate Job named after the CronJob, which is why the CronJob name is capped at 52 characters: the controller appends eleven characters and the Job name must still fit in 63. A CronJob never runs the work itself; it only creates Jobs, which own the pods. Since Kubernetes v1.32 the Job also carries a batch.kubernetes.io/cronjob-scheduled-timestamp annotation holding the time it was meant to run, the only way to tell a late run from an on-time one after a controller restart.

Concurrency is a policy, not a guess

concurrencyPolicy decides what happens when a tick arrives while the previous Job is still running. Allow, the default, starts a second copy. Forbid skips the tick. Replace cancels the running Job and starts a fresh one. The policy applies only to Jobs from the same CronJob, so two CronJobs never block each other.

Replace is the dangerous one: if the Job reliably outruns its interval, every tick kills the run that was about to finish and the work never completes. Forbid is usually what people mean by “don’t overlap”, with two caveats — skipped runs are lost, and once the running Job finishes, startingDeadlineSeconds is still evaluated, so a long-overdue run may start anyway.

Missed runs have a budget of a hundred

startingDeadlineSeconds bounds how late a missed run is still worth starting; past that deadline the run is skipped and counted as failed, and skipping one occurrence does not stop the schedule, because the next matching time still fires. Left unset, missed runs have no deadline of their own, but they are still subject to the controller’s hard cap. It counts the schedules missed since the last scheduled time, and above one hundred it starts nothing and logs too many missed start times. Set or decrease .spec.startingDeadlineSeconds or check clock skew.

The documented example is concrete. A per-minute CronJob whose controller is down from 08:29 to 10:21 has missed 112 schedules, so nothing catches up on recovery. Add startingDeadlineSeconds: 200 and the controller instead counts only the schedules missed inside the last 200 seconds — three of them — and the Job does start. The same field hides a trap: values below ten seconds are unreliable, because the controller only re-evaluates every ten seconds. suspend: true parks a CronJob without touching Jobs already running, but suspended executions count as missed, so unsuspending without a deadline fires the whole backlog at once.

When it breaks

The classic failure is a Job that overlaps itself. A twenty-five minute report on a ten-minute schedule, with the default Allow, peaks at three concurrent copies reading the same input and writing the same output. It never presents as an error: duplicated rows, doubled API spend, or a lock timeout in one copy out of three. Either move to Forbid and accept skipped runs, or make the Job idempotent with a lease so the late copy exits early.

The second failure is silent catch-up loss. A control-plane outage longer than a hundred schedule intervals means the whole backlog is discarded on recovery, and the only trace is a line in the controller manager’s log. That is right for a per-minute poll and wrong for a nightly backup, which is what startingDeadlineSeconds is for: setting it to the real tolerance — eight hours for a twice-daily backup — both bounds how stale a run may be and keeps the hundred-interval cap permanently out of reach.

Check it quickly

Read the schedule, the policy and the last firing time together:

kubectl get cronjob nightly-report -o jsonpath='{.spec.schedule} {.spec.concurrencyPolicy} {.spec.startingDeadlineSeconds}{"\n"}'
kubectl get cronjob nightly-report -o jsonpath='{.status.lastScheduleTime}{"\n"}'
kubectl get jobs --sort-by=.metadata.creationTimestamp -o custom-columns=NAME:.metadata.name,START:.status.startTime,SUCCEEDED:.status.succeeded
kubectl get events --field-selector involvedObject.kind=CronJob --sort-by=.lastTimestamp | tail -5

lastScheduleTime should sit within one schedule interval of now; a gap of hours means the controller was down or the expression never matched. Overlapping START values with SUCCEEDED empty mean Allow plus a Job that outruns its interval, and a too many missed start times event names the missed-run rule directly.

Sources

Kubernetes CronJob, schedule syntax, concurrency policy, the 100-missed-schedule rule and history limits, checked 2026-09-16.
CronJob v1 API reference, field-level definition of the CronJob spec, checked 2026-09-16.

Node affinity places workloads on matching nodes

Thu, 09 Dec 2027 00:00:00 +0000

Affinity is placement matching: prefer a label or force it, and use anti-affinity to keep replicas spread apart for resilience. Hardware is never uniform for long — a GPU pool here, fast local NVMe there, an isolated zone for regulated data — and workloads quietly depend on landing in the right place. Without explicit rules, the scheduler spreads pods by free capacity alone, and the pod that needs a GPU lands on the node that has none, failing at runtime instead of at schedule time.

Affinity moves that failure earlier, where it belongs. Stated as data on nodes (labels) matched against rules on pods, placement becomes reviewable configuration rather than tribal knowledge about “which nodes are the special ones.”

Match nodes by label

Affinity rules read node labels, so you can state that a pod should, or must, run only on nodes carrying a particular label. Put a stable label on your nodes once, then let scheduling match against it. Labels like accelerator=nvidia-a100, storage=nvme, or zone=regulated describe durable properties of the machine, and because they live on the node object, every pod rule referencing them stays valid across node replacements — new hardware gets the same label, old rules keep working.

Choose label names with the same care as API fields, because renaming a widely referenced label is a migration across every manifest that matches it. Validate labels in CI the way you validate image tags — a pod requiring a label no node carries is a typo that should fail the pipeline, not the 3 a.m. deploy.

Prefer or require

Preferred rules are best-effort and balance load when possible, while required rules hard-block a pod that cannot satisfy the label. Choose required only when placement genuinely matters; otherwise preference keeps the scheduler flexible. The cost model is straightforward: required rules trade availability for correctness — if no matching node has room, the pod stays Pending rather than running in the wrong place — while preferred rules trade correctness for availability, landing somewhere suboptimal but alive.

Most estates want required for the few hard constraints (GPU present, compliance zone, architecture) and preferred for everything soft (spread across racks, favor the newer instance family). Review required rules quarterly, because hardware that was scarce last year may be plentiful now, and a required rule nobody remembers is how a cluster with free capacity stops scheduling.

Anti-affinity splits replicas

Anti-affinity looks at other pods, spreading replicas of one workload across different nodes so a single node failure stings less. Pair it with replicas and topology keys so the split is meaningful rather than cosmetic. topologyKey: kubernetes.io/hostname spreads across machines; topologyKey: topology.zone spreads across failure domains.

Required anti-affinity deserves the same caution as required affinity: with three replicas and a required hostname spread, the fourth node going down is fine but the workload can never exceed the node count, and a two-node cluster cannot host it at all. For most services, preferred anti-affinity with a high weight gives nearly all the resilience with none of the scheduling fragility. Reserve the required form for the handful of workloads — quorum members, single-leader pairs — where co-location is genuinely worse than Pending.

When it breaks

The signature failure is the permanently Pending pod: events show didn't match pod affinity rules while nodes sit half empty. Causes are almost always label-side — the node label was never applied, was renamed, or carries a different value than the rule expects (nvme vs nvme-ssd). kubectl get nodes -L hardware,workload-class shows the actual label surface in one table; diff it against the pod’s requirements and the mismatch usually confesses immediately.

The subtler failure is the satisfied-but-wrong placement: preferred rules silently lose to capacity pressure, and over weeks the GPU workloads drift onto shared nodes while the GPU pool idles. Preferred means the scheduler may ignore you, and under pressure it will. Audit placement periodically against intent — a query grouping pods by workload and node label — and promote the rules that keep drifting from preferred to required, or fix the capacity shortage that forces the compromise.

Check it quickly

See labels, requirements, and actual placement together:

kubectl get nodes -L accelerator,workload-class --no-headers
kubectl describe pod api-0 | grep -A 8 "Node-Selectors\|Affinity"
kubectl get pods -o wide -l app=api | awk '{print $1, $7}' | sort | uniq -c | sort -rn | head

The first shows the label surface, the second shows what one pod demands, and the third proves whether replicas actually spread or quietly piled onto one node.

Sources

Kubernetes node affinity docs, node affinity semantics, checked 2026-06-10.
Kubernetes taints and tolerations, documents repelling pods from specialized nodes, checked 2026-09-05.

Kubernetes rolling updates replace pods gradually

Sat, 27 Nov 2027 00:00:00 +0000

Two windows, maxSurge and maxUnavailable, bound how many extra or missing pods a rollout may have, so a release can proceed under capacity control.

Replace a few at a time

A rolling update does not swap the workload in one cut; it brings up new pods and removes old ones gradually, respecting readiness the whole way. The controller waits for a new pod to become ready before it considers the update moving, so a pod that never becomes ready stalls the rollout rather than silently serving traffic. That is the safety property: at no point does the fleet jump from the old version to the new one with nothing in between. Because readiness gates the transition, you can catch a broken image at the first new pod and stop the rollout with far less blast radius than a full swap.

Surge and drain windows

Two settings bound the shape of the rollout. maxSurge is how many extra pods above the desired count you allow while old ones are still draining; maxUnavailable is how far below the desired count you may fall during the swap. Together they trade speed against headroom: a large surge runs fast but needs spare capacity to land the extra pods, while maxUnavailable: 1 with maxSurge: 0 prefers a steady capacity but proceeds more slowly and can present a maintenance window. Choose them to describe the risk you accept on a release, not a value that looks balanced, because they literally set the ceiling and floor of what the rollout may touch.

Roll back is a redeploy

Because a rollout is just another desired state, rolling back is not a special operation: you apply the image the cluster was running before, and the same bounded, readiness-gated churn happens in reverse. Keep the prior revision available and restartable, and prefer an old known-good image over an ad-hoc patch, so the rollback path is itself a tested path. The audit trail is the same, the pod churn is bounded by the same two windows, and the team knows exactly what ‘rollback’ means, it is a redeploy of the last-known-good spec, nothing more.

Recreate stops everything first

strategy.type has only two values, and the default hides the choice. RollingUpdate overlaps old and new pods inside the surge and unavailable windows; Recreate deletes every old pod first and only then creates the new ones, so minimum availability is zero by construction and the two windows are ignored. Pick Recreate when two versions must never coexist: a single-writer migration the old code cannot read back, an exclusive file lock or singleton worker, or a protocol where old and new disagree. The companion post tunes pace; this one picks shape. Default to RollingUpdate for stateless tiers, and when you choose Recreate, schedule the outage instead of discovering it: measure a cold start, pre-pull the image, and announce the window.

When it breaks

A stateful workload on RollingUpdate is the classic split-brain. Old and new pods overlap for one wave, so two leaders briefly coexist, each writing in its own schema, and the merge afterward is manual. Where versions cannot coexist, overlap is the bug: choose Recreate or add coordination.

The mirror failure is a Recreate whose stop outlasts its window. The team budgets thirty seconds, but the new image pulls slowly and readiness takes minutes, and with minimum availability at zero every extra minute is a full outage. Time a cold start before choosing it.

Third, a RollingUpdate configured into a full stop: four replicas with maxUnavailable: 50% shed two pods at once, one replacement CrashLoops, and the lone survivor melts under full load. Keep maxUnavailable at zero or one on small fleets.

Check it quickly

Save the following as strategy_window.py and run python3 strategy_window.py. It uses only the standard library, deriving each strategy’s floor from replica and window arithmetic.

"""Compare the availability floor of Recreate vs RollingUpdate (stdlib only)."""

def rolling_window(replicas, max_surge, max_unavailable):
 if replicas <= 0:
 raise ValueError("replicas must be positive")
 if max_unavailable >= replicas:
 raise ValueError("maxUnavailable covers the whole fleet: configured full stop")
 floor = replicas - max_unavailable
 ceiling = replicas + max_surge
 return floor, ceiling


def recreate_window():
 return 0, 0


def main():
 replicas, surge, unavailable = 6, 1, 1
 rolling_floor, rolling_ceiling = rolling_window(replicas, surge, unavailable)
 stopped, _ = recreate_window()
 assert rolling_floor == 5, rolling_floor
 assert rolling_ceiling == 7, rolling_ceiling
 assert rolling_floor > 0, "rolling update must keep serving"
 assert stopped == 0, "recreate stops everything first"
 try:
 rolling_window(4, 1, 4)
 except ValueError as error:
 print("caught expected misconfiguration:", error)
 else:
 raise AssertionError("maxUnavailable >= replicas should fail")
 print(f"replicas={replicas} surge={surge} unavailable={unavailable}")
 print(f"rolling: min available={rolling_floor}, max total={rolling_ceiling}")
 print("recreate: min available=0 (full stop, then rebuild)")
 print("PASS: rolling keeps serving; recreate stops everything first")


if __name__ == "__main__":
 main()

Six replicas, a surge of one, and one unavailable keep five serving with at most seven total; Recreate floors at zero. The assertions encode that contract, and a maxUnavailable covering all four replicas must raise instead of returning a floor.

The assertions passed with Python 3.14.4 on macOS. The model checks window arithmetic, not pulls, probes, or scheduling. Before release day, confirm the live strategy.type and time one cold start.

Sources

Kubernetes Deployment rolling update, rolling update semantics, checked 2026-06-12.
Kubernetes updating a Deployment, documents rollout behavior and rollback, checked 2026-09-05.

Security context defines constraints that need runtime evidence

Sat, 20 Nov 2027 00:00:00 +0000

A Kubernetes securityContext configures process identity and selected operating-system constraints. It can reduce what a compromised application is permitted to do, but it does not establish a complete security boundary by itself. Start with the operations the application actually needs, configure the appropriate fields, and test both a permitted operation and a forbidden one in the resulting container.

Put each setting at the right scope

Pod-level settings can supply user and group identity, shared-volume group ownership behavior and a seccomp profile. Container-level settings can override applicable defaults and configure capabilities, privilege escalation and a read-only root filesystem. Check the field’s actual API scope; placing a familiar name under the wrong securityContext does not create an equivalent constraint.

runAsUser selects a numeric user, while runAsNonRoot requires the container to satisfy a non-root policy. These settings complement image metadata rather than trusting a username’s appearance. Volume permissions must also permit the selected identity to perform required work. For an emptyDir used below, fsGroup supplies group access so the application can write its designated working directory.

Linux capabilities divide privileges that would otherwise be associated broadly with root. Dropping ALL removes those capabilities from the container’s configured set. Standard containers do not inherit every Linux capability by default, and a container running as root is not automatically proof of a successful host escape. Review the actual effective privileges rather than using either shortcut.

When it breaks

allowPrivilegeEscalation false sets the no_new_privs constraint for the process. It restricts gaining privileges through executing programs, such as set-user-ID or file-capability transitions. It does not revoke every privilege the process already has or replace application authorization. Inspect the process flag and effective capabilities; do not attribute every denied operation to this one field.

readOnlyRootFilesystem makes the container’s root filesystem read-only. Other mounts can remain writable, including explicitly supplied working volumes. This does not imply a blanket no-execution policy: an interpreter can read and run code stored in a writable volume. It also does not stop network requests or remove access to already mounted data. Read-only storage is one constrained operation, not proof that a compromise cannot progress.

Seccomp filters system calls, but RuntimeDefault depends on the runtime’s supplied profile. A process reporting seccomp filter mode confirms an active filtering mechanism, not that every undesirable syscall or attack path is blocked. Ordinary Linux runtimes can enforce these fields; a special sandbox is not required merely for the settings to have an effect, though stronger isolation has separate design considerations.

Conflicting identity settings can pass API admission but prevent container startup. An image may also expect to write caches or temporary files under its root filesystem. Diagnose startup state and denied paths, then provide only the working storage the application requires. Avoid solving every permission failure by restoring root, privilege escalation or a writable root filesystem.

Check it quickly

Use a disposable Linux Kubernetes cluster, kubectl and python:3.11-slim-bookworm available on its nodes. Save the script as security-context.py and run python3 security-context.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. The account needs namespace creation and deletion permissions. The recorded server was Kubernetes 1.34.11 using kind’s standard runtime configuration.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-security-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 probe = '''from pathlib import Path
import errno,json,os,subprocess,time
status=dict(line.split(':',1) for line in Path('/proc/self/status').read_text().splitlines() if ':' in line)
try:
 Path('/tmp/forbidden-write').write_text('x')
 raise AssertionError('root filesystem unexpectedly writable')
except OSError as error:
 assert error.errno==errno.EROFS
try:
 os.setuid(0)
 raise AssertionError('unexpected UID escalation')
except OSError as error:
 assert error.errno==errno.EPERM
work=Path('/work'); (work/'data').write_text('writable')
(work/'example.py').write_text("print('interpreted from the writable volume')")
ran=subprocess.check_output(['python3',str(work/'example.py')],text=True).strip()
result={'uid':os.getuid(),'gid':os.getgid(),'cap_eff':int(status['CapEff'].strip(),16),
 'no_new_privs':int(status['NoNewPrivs']),'seccomp':int(status['Seccomp']),
 'root_write_errno':errno.EROFS,'uid_zero_denied':True,
 'volume_value':(work/'data').read_text(),'writable_volume_code':ran}
assert result['uid']==result['gid']==10001
assert result['cap_eff']==0 and result['no_new_privs']==1 and result['seccomp']==2
print(json.dumps(result),flush=True); time.sleep(600)
'''
 podspec = spec(probe)
 podspec['securityContext'] = {'runAsNonRoot': True, 'runAsUser': 10001,
 'runAsGroup': 10001, 'fsGroup': 10001, 'seccompProfile': {'type': 'RuntimeDefault'}}
 podspec['volumes'] = [{'name': 'work', 'emptyDir': {}}]
 c = podspec['containers'][0]
 c['securityContext'] = {'allowPrivilegeEscalation': False, 'readOnlyRootFilesystem': True,
 'capabilities': {'drop': ['ALL']}}
 c['volumeMounts'] = [{'name': 'work', 'mountPath': '/work'}]
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'restricted'}, 'spec': podspec})
 wait_for(lambda: obj(namespace, 'pod', 'restricted'), pod_ready)
 observed = json.loads(wait_for(lambda: k('logs', '-n', namespace, 'restricted').stdout.strip(), bool))
 conflicting = spec()
 conflicting['securityContext'] = {'runAsNonRoot': True, 'runAsUser': 0}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'conflict'}, 'spec': conflicting})
 refused = wait_for(lambda: obj(namespace, 'pod', 'conflict'), lambda p:
 any(c.get('state', {}).get('waiting', {}).get('reason') == 'CreateContainerConfigError'
 for c in p.get('status', {}).get('containerStatuses', [])))
 waiting = refused['status']['containerStatuses'][0]
 assert 'runAsUser breaks non-root policy' in waiting['state']['waiting']['message']
 assert waiting['restartCount'] == 0
 print(json.dumps({'runtime_observation': observed,
 'conflicting_uid_accepted_by_api_but_container_start_refused': True,
 'container_escape_resistance_tested': False}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The restricted Pod reports UID and GID 10001, effective capabilities zero, NoNewPrivs one and seccomp mode two. Writing under the root filesystem’s /tmp fails with EROFS, and changing to UID zero fails with EPERM. Those results reflect the combined identity and privilege settings, not an isolated test of each field’s individual effect.

The process successfully writes a value under /work and invokes Python on a file it wrote there. This deliberately demonstrates the allowed working-volume behavior and the limit of the read-only-root claim. A second Pod combines runAsNonRoot with runAsUser zero: the API accepts it, but kubelet reports CreateContainerConfigError before the container starts.

All assertions passed, and the namespace was removed. The fixture verifies these concrete runtime constraints and startup failure. It does not test container escape resistance, every seccomp rule, hostile multi-tenancy or the application’s authorization model.

Sources

Kubernetes: Security Context, Pod and container security fields and runtime effects, checked 2026-09-13.
Linux: No New Privileges Flag, restrictions on privilege gains through exec, checked 2026-09-13.
Linux: capabilities(7), separate capability privileges and process capability sets, checked 2026-09-13.

The horizontal autoscaler scales on demand

Thu, 04 Nov 2027 00:00:00 +0000

The autoscaler derives desired replicas from current usage and a target, so the metric you choose matters more than the replica count you set.

Metrics drive the count

The HPA does not watch load directly; it polls a configured metric on a schedule, reads a current value against the target you set, and derives a desired replica count from the ratio. If the observed value runs twice the target it scales up, if half it scales down, and the result is recomputed on each reconciliation. That is why the metric choice, its aggregation (per pod or cluster-level), and the window matter, they define what the load even is. The desired count is then clamped by maxReplicas and minReplicas, and reconciled on the default --horizontal-pod-autoscaler-sync-period. To make scaling predictable you reason about the input metric, not about a single constant.

Scale up eagerly, down gently

Scale-down is deliberately asymmetric. The controller does not remove a pod the instant utilization dips, because a transient spike would otherwise ping-pong the replica count and thrash the cluster. It waits out a stabilization window set by --horizontal-pod-autoscaler-downscale-stabilization, keeps a floor via minReplicas, and only then shrinks. Scaling up is fast, because queued latency hurts right now; scaling down is slow, because idle capacity is cheaper than a flapping fleet. A dependable setup states a floor generous enough for the spikes you actually see, a target it can digest, and treats the stabilization window as the amount of load noise you are willing to ride out, so the group does not bounce every time the graph blips.

The metric is the weak link

The whole controller inherits the quality of your metric. If the metric lags real demand, the controller is always chasing a stale number; if it watches a proxy that correlates poorly with the actual bottleneck, the controller scales the wrong thing. So the hard part of an HPA is rarely the YAML, it is the metric and the target. Putting CPU on a latency-bound service, or memory on an I-O-bound worker, often points scaling in the wrong direction. Begin from what actually saturates, requests in flight, queue depth, or connections, choose an average over a window you can act on, and prove that a rise in that metric really predicts worse latency. Then the HPA becomes a tool that scales your workload, not a timer that wastes a guess.

When it breaks

Stale or missing metrics make the controller flap. When the metrics pipeline lags, HPA scales against last minute’s load: it adds replicas after the spike already passed, then holds them through the downscale stabilization window, so dashboards show oscillation rather than tracking. If metrics go missing entirely, the controller refuses to scale and surfaces ScalingActive=False; treating “no data” as “no load” and deleting capacity is the classic outage version of this bug.

Bounds freeze the loop in the other direction. A maxReplicas set from cost anxiety caps the fleet during a real surge and reports ScalingLimited=True, while a generous minReplicas hides an overestimated target because the floor never lets the controller prove demand fell. Finally, do not let two controllers steer the same knob: HPA fighting VPA (one changes count, the other changes requests, which moves utilization) or an aggressive rolling update (new pods report zero usage while old pods still serve) produces sawtooth replicas that no target tuning will fix.

Check it quickly

Save the following as hpa_loop.py in a scratch directory and run python3 hpa_loop.py. It models one HPA control loop with pure standard library: desired replicas are the ceiling of current average utilization over target, clamped to min and max, with missing metrics holding the last decision and a downscale stabilization window blocking fast shrink. The script asserts scale-up on a hot sample, cooldown-blocked scale-down on the next cool sample, eventual shrink after the window, and clamping at both bounds.

import math

TARGET = 50.0
MIN_REPLICAS = 2
MAX_REPLICAS = 10
STABILIZATION_S = 300


def desired(current_replicas, current_util):
 raw = math.ceil(current_replicas * current_util / TARGET)
 return max(MIN_REPLICAS, min(MAX_REPLICAS, raw))


def step(state, now, current_util):
 replicas, last_scale = state
 if current_util is None:
 print(f"t={now}s metric=missing hold={replicas}")
 return state
 want = desired(replicas, current_util)
 if want < replicas and now - last_scale < STABILIZATION_S:
 print(f"t={now}s metric={current_util} want={want} hold={replicas} (cooldown)")
 return state
 print(f"t={now}s metric={current_util} {replicas}->{want}")
 if want != replicas:
 return (want, now)
 return state


def main():
 state = (3, -10**9)
 state = step(state, 0, 100)
 assert state[0] == 6, state
 state = step(state, 60, 15)
 assert state[0] == 6, state
 state = step(state, 400, 15)
 assert state[0] == 2, state
 state = step(state, 460, None)
 assert state[0] == 2, state
 assert desired(2, 500) == 10
 assert desired(2, 5) == 2
 print("PASS: scale-up 3->6, cooldown holds 6, then 6->2; clamps at 10 and 2")


if __name__ == "__main__":
 main()

The run prints each decision and ends with PASS. The hot sample grows 3 replicas to 6, the immediate cool sample stays at 6 while the stabilization window is fresh, the same cool value shrinks to 2 after the window expires, and out-of-range calculations stick at max 10 and min 2. The fixture checks control-loop math, not metrics-server latency or startup time. Validate cluster tuning against kubectl describe hpa conditions.

Sources

Kubernetes horizontal pod autoscaling, how HPA derives replica counts, checked 2026-05-06.
HorizontalPodAutoscaler walkthrough, worked example with multiple metrics and status conditions, checked 2026-05-06.

Kubernetes labels define selection, not authorization

Thu, 09 Sep 2027 00:00:00 +0000

Labels are key-value metadata used to select Kubernetes objects. A Service selector, an operational query, and a workload controller can all rely on them, but each consumer has its own API and behavior. A label change can therefore alter group membership while leaving the underlying process healthy. Review labels as inputs to actual consumers rather than as purely descriptive stickers.

Know which selector language is in use

Label keys use a qualified-name syntax, and values have format and length constraints. A value may be empty; otherwise its permitted characters and endpoints are restricted, with a maximum of sixty-three characters. Labels are not a place for arbitrary paragraphs or unconstrained JSON. Use annotations for descriptive metadata, while still respecting annotation validation and consumer contracts.

Equality requirements can test that a key has a value, while set-based requirements support membership and existence tests. Multiple requirements in a selector are combined as conditions that must all hold. An important detail is inequality: track!=canary also matches an object with no track label. If absence is unacceptable, require the key to exist as well instead of assuming inequality implies its presence.

Selector forms differ between resources. A normal selector-based Service uses a map of equality requirements; it does not accept the complete set-expression language as arbitrary map values. Other APIs expose LabelSelector structures, and kubectl list queries accept selector expressions. Check the exact field’s schema before translating a convenient command-line query into a manifest.

When it breaks

A Service’s selector identifies matching Pods in its namespace, and the endpoint machinery tracks addresses and readiness. Matching labels alone do not prove that a server is ready or that its port is correct. With normal readiness behavior, available routing targets depend on endpoint conditions too. Inspect the relevant EndpointSlices and application checks rather than stopping at a successful label query.

The publishNotReadyAddresses setting changes how readiness is treated for Service discovery. Do not assume an endpoint’s ready flag always proves the Pod passed its application probe under that setting. The fixture below uses the ordinary default and does not exercise this override.

Relabelling a healthy Pod can remove it from a Service without terminating it. Conversely, an overly broad selector can include a workload that was never intended as a backend. Changing labels on controller-managed Pods can also interact with ownership and reconciliation. Update the intended desired state through the responsible controller, and check all important consumers of a shared label before changing its meaning.

Selectors are not network authorization. A Pod excluded from one Service can remain reachable through its own address or another Service unless other controls prevent that access. NetworkPolicy, application authorization, and routing each solve different parts of the problem. Do not describe a selector as a guest list that denies all direct traffic to unselected Pods.

Check it quickly

Use a disposable Kubernetes cluster with kubectl and python:3.11-slim-bookworm available on the nodes. Save this as label-selectors.py and run python3 label-selectors.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. The script creates a unique namespace containing three standalone HTTP Pods and a Service, then removes it. The recorded server version was Kubernetes 1.34.11.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-labels-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 for name, role, track in [('a', 'reader', 'stable'), ('b', 'reader', 'canary'), ('c', 'writer', None)]:
 script = '''from http.server import BaseHTTPRequestHandler,HTTPServer
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 data=LABEL.encode();self.send_response(200);self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)
 def log_message(self,*args): pass
HTTPServer(('0.0.0.0',8080),Handler).serve_forever()
'''.replace('LABEL', repr(name))
 podspec = spec(script)
 podspec['containers'][0]['readinessProbe'] = {'httpGet': {'path': '/', 'port': 8080}, 'periodSeconds': 1}
 labels = {'app': 'catalog', 'role': role}
 if track:
 labels['track'] = track
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': name, 'labels': labels}, 'spec': podspec})
 for name in ('a', 'b', 'c'):
 wait_for(lambda name=name: obj(namespace, 'pod', name), pod_ready)
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'readers'},
 'spec': {'selector': {'app': 'catalog', 'role': 'reader'}, 'ports': [{'port': 80, 'targetPort': 8080}]}})
 def selected():
 slices = obj(namespace, 'endpointslices', selector='kubernetes.io/service-name=readers')['items']
 return {e['targetRef']['name'] for item in slices for e in item.get('endpoints', [])
 if e.get('conditions', {}).get('ready') and e.get('targetRef')}
 before = wait_for(selected, lambda names: names == {'a', 'b'})
 inequality = {p['metadata']['name'] for p in obj(namespace, 'pods', selector='app=catalog,track!=canary')['items']}
 assert inequality == {'a', 'c'}
 expression = {p['metadata']['name'] for p in obj(namespace, 'pods', selector='app in (catalog),role=reader')['items']}
 assert expression == {'a', 'b'}
 writer_ip = obj(namespace, 'pod', 'c')['status']['podIP']
 probe = 'from urllib.request import urlopen; print(urlopen(' + repr('http://'+writer_ip+':8080/') + ',timeout=3).read().decode())'
 assert k('exec', '-n', namespace, 'a', '--', 'python3', '-c', probe).stdout.strip() == 'c'
 k('label', 'pod', 'b', '-n', namespace, 'role=writer', '--overwrite')
 after = wait_for(selected, lambda names: names == {'a'})
 assert pod_ready(obj(namespace, 'pod', 'b'))
 invalid = k('patch', 'pod', 'a', '-n', namespace, '--type=merge',
 '-p', json.dumps({'metadata': {'labels': {'notes.example/value': 'has spaces'}}}), expected=None)
 assert invalid.returncode != 0 and 'Invalid' in invalid.stderr
 print(json.dumps({'service_targets_before': sorted(before), 'service_targets_after': sorted(after),
 'inequality_also_matches_missing_key': sorted(inequality),
 'set_selector_matches': sorted(expression), 'excluded_pod_directly_reachable': True,
 'invalid_label_value_rejected': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

Pods a and b have role=reader; c has role=writer. All share app=catalog and become ready through HTTP probes. Pod a has a stable track, b has canary, and c lacks track. The reader Service initially has ready targets a and b. A set-based query finds those readers, while track!=canary finds a and c, confirming the missing-key behavior.

An actual request from a to c’s Pod address returns c, even though c is excluded from the reader Service. Changing b’s role to writer then leaves only a in the Service’s ready endpoint set. Pod b remains ready. A separate patch containing spaces in a label value is rejected by the API. All assertions passed.

This verifies API selection, EndpointSlice changes, and direct reachability in the isolated test network. It does not test every controller’s adoption rules, enforce a NetworkPolicy, or benchmark Service traffic. Keep selection intent, endpoint readiness, and access policy independently observable when reviewing routing changes.

Sources

Kubernetes: Labels and Selectors, valid values, equality and set-based requirements, checked 2026-09-13.
Kubernetes: Services, selector-based backend discovery, checked 2026-09-13.
Kubernetes: EndpointSlices, endpoint identity and readiness conditions, checked 2026-09-13.

Namespaces scope objects; policies establish other boundaries

Wed, 25 Aug 2027 00:00:00 +0000

A namespace scopes many Kubernetes objects within one cluster. Two teams can each create a ConfigMap named settings because those objects have different namespace-qualified identities. This organizational boundary is useful, but it does not automatically provide network isolation, a dedicated kernel, reserved node capacity, or complete tenant separation. Identify which additional policies enforce the boundaries your workload actually requires.

Names are scoped by resource type

Pods, Services, ConfigMaps, and many workload controllers are namespaced. Nodes, namespaces themselves, and other cluster-level resources are not divided into separate copies merely because a client specifies a namespace. Check the resource’s API scope rather than assuming every object lives inside a room. Namespaces also do not form a nested hierarchy of independent child clusters.

Within a Pod, a short Service name normally resolves in the Pod’s namespace. A qualified name can target a Service elsewhere, such as api.other.svc.cluster.local in a cluster using the default domain. This naming convention locates a destination; it does not authorize or prohibit the connection. The cluster domain can be configured differently, so do not mistake the example suffix for a universal constant.

Namespace choice must remain explicit in automation. A correct object name combined with the wrong active namespace still addresses the wrong resource. Use deliberate kubeconfig, context, and namespace inputs, especially for mutation. Namespace scoping makes identity clearer, but an administrator with broad permissions can still modify objects across those scopes.

When it breaks

A namespace without an applicable NetworkPolicy remains connected according to the cluster network’s behavior. Merely creating a namespace or qualifying a DNS name does not deny other Pods. Network isolation requires suitable policies and a network implementation that enforces them. Application authentication remains another layer; successful name resolution says nothing about who should be allowed to read a response.

RBAC controls API operations separately. A RoleBinding can grant an account access to ConfigMaps in one namespace without granting the same access elsewhere. Permissions are additive, so another broad binding can still grant additional access. Test actual allowed and denied operations for the intended identity rather than assuming that adding a narrow Role cancels wider grants.

ResourceQuota primarily enforces admission limits, including object counts and aggregate resource requests or limits. It does not itself throttle a running process or reserve physical capacity. LimitRange can supply defaults and bounds for admitted workloads, while runtime resource enforcement has other mechanisms. These policies must be configured; the namespace alone does not make a noisy neighbor harmless.

Check it quickly

Use a disposable Kubernetes cluster with the default cluster.local DNS domain, kubectl administrator access for impersonation, and python:3.11-slim-bookworm available on its nodes. Save this as namespaces.py and run python3 namespaces.py KUBECONFIG_PATH CONTEXT with explicit test configuration. The recorded cluster ran Kubernetes 1.34.11 with kind networking. The script creates two unique namespaces and removes both afterward.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

suffix = uuid4().hex[:8]
first, second = 'notes-a-' + suffix, 'notes-b-' + suffix
created = []
try:
 for namespace, value in [(first, 'A'), (second, 'B')]:
 k('create', 'namespace', namespace)
 created.append(namespace)
 apply(namespace, {'apiVersion': 'v1', 'kind': 'ConfigMap', 'metadata': {'name': 'settings'}, 'data': {'scope': value}})
 script = '''from http.server import BaseHTTPRequestHandler,HTTPServer
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 data=VALUE.encode();self.send_response(200);self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)
 def log_message(self,*args): pass
HTTPServer(('0.0.0.0',8080),Handler).serve_forever()
'''.replace('VALUE', repr(value))
 podspec = spec(script)
 podspec['containers'][0]['readinessProbe'] = {'httpGet': {'path': '/', 'port': 8080}, 'periodSeconds': 1}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'web', 'labels': {'app': 'web'}}, 'spec': podspec})
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'api'},
 'spec': {'selector': {'app': 'web'}, 'ports': [{'port': 80, 'targetPort': 8080}]}})
 wait_for(lambda namespace=namespace: obj(namespace, 'pod', 'web'), pod_ready)
 wait_for(lambda namespace=namespace: obj(namespace, 'endpointslices', selector='kubernetes.io/service-name=api')['items'],
 lambda slices: any(e.get('conditions', {}).get('ready') for s in slices for e in s.get('endpoints', [])))
 assert obj(first, 'configmap', 'settings')['data']['scope'] == 'A'
 assert obj(second, 'configmap', 'settings')['data']['scope'] == 'B'
 nodes_a = {n['metadata']['uid'] for n in obj(first, 'nodes')['items']}
 nodes_b = {n['metadata']['uid'] for n in obj(second, 'nodes')['items']}
 assert nodes_a == nodes_b and nodes_a
 apply(first, {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'name': 'reader'}, 'automountServiceAccountToken': False})
 apply(first, {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'Role', 'metadata': {'name': 'read-config'},
 'rules': [{'apiGroups': [''], 'resources': ['configmaps'], 'verbs': ['get']}]})
 apply(first, {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'RoleBinding', 'metadata': {'name': 'read-config'},
 'subjects': [{'kind': 'ServiceAccount', 'name': 'reader', 'namespace': first}],
 'roleRef': {'apiGroup': 'rbac.authorization.k8s.io', 'kind': 'Role', 'name': 'read-config'}})
 identity = 'system:serviceaccount:' + first + ':reader'
 assert k('--as', identity, 'get', 'configmap', 'settings', '-n', first).returncode == 0
 denied = k('--as', identity, 'get', 'configmap', 'settings', '-n', second, expected=None)
 assert denied.returncode != 0 and 'Forbidden' in denied.stderr
 apply(first, {'apiVersion': 'v1', 'kind': 'ResourceQuota', 'metadata': {'name': 'one-service'},
 'spec': {'hard': {'services': '1'}}})
 wait_for(lambda: obj(first, 'resourcequota', 'one-service'), lambda q: q.get('status', {}).get('used', {}).get('services') == '1')
 extra = {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'extra', 'namespace': first},
 'spec': {'ports': [{'port': 80}]}}
 quota_denial = k('create', '-f', '-', payload=extra, expected=None)
 assert quota_denial.returncode != 0 and 'exceeded quota' in quota_denial.stderr
 assert not obj(first, 'networkpolicies')['items'] and not obj(second, 'networkpolicies')['items']
 remote = 'api.' + second + '.svc.cluster.local'
 probe = '''import json,socket
from urllib.request import urlopen
print(json.dumps({'short_name':socket.gethostbyname('api'),
 'qualified_name':socket.gethostbyname(REMOTE),
 'cross_namespace_body':urlopen('http://'+REMOTE+'/',timeout=3).read().decode()}))'''.replace('REMOTE', repr(remote))
 network = json.loads(k('exec', '-n', first, 'web', '--', 'python3', '-c', probe).stdout)
 assert network['short_name'] == obj(first, 'service', 'api')['spec']['clusterIP']
 assert network['qualified_name'] == obj(second, 'service', 'api')['spec']['clusterIP']
 assert network['cross_namespace_body'] == 'B'
 print(json.dumps({'same_names_different_configmaps': True, 'nodes_are_cluster_scoped': True,
 'rbac_allows_local_and_denies_other_namespace': True,
 'quota_rejects_second_service': True, 'short_and_qualified_dns_match_services': True,
 'cross_namespace_http_succeeds_without_network_policy': True}))
finally:
 for namespace in reversed(created):
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

Each namespace receives a ConfigMap named settings, a Pod named web, and a Service named api. The ConfigMaps retain different A and B values. Node queries made with either namespace return the same cluster-scoped node identities. This verifies object-name scope without inventing separate node pools.

A test service account is granted ConfigMap reads only in the first namespace. Requests impersonating it succeed there and return Forbidden in the second. A ResourceQuota then permits one Service in the first namespace; creating another is rejected after the existing usage is observed. This exercises a real admission decision rather than only checking that a quota object exists.

Finally, a request from the first Pod resolves api to its local Service, resolves the qualified second Service to that Service’s address, and receives B over HTTP. Neither namespace has a NetworkPolicy. All assertions passed. The experiment verifies naming, API authorization, one quota dimension, and network reachability. It does not demonstrate NetworkPolicy enforcement, CPU fairness, or full hostile-tenant isolation. Review and test each required boundary explicitly instead of making namespace membership stand in for all of them.

Sources

Kubernetes: Namespaces, object-name scope and cluster-scoped resources, checked 2026-09-13.
Kubernetes: Resource Quotas, namespace admission limits and object counts, checked 2026-09-13.
Kubernetes: RBAC authorization, namespace bindings and additive permissions, checked 2026-09-13.
Kubernetes: Network Policies, network isolation requires policy and an enforcing implementation, checked 2026-09-13.
Kubernetes: DNS for Services and Pods, short names, namespace-qualified Service names and cluster domains, checked 2026-09-13.

Kubernetes Services give pods a stable home

Mon, 23 Aug 2027 00:00:00 +0000

You cannot memorise a Pod that will be recreated. A Service fixes an address in front of them, keeping callers stable while pods churn.

One address in front of many

A Deployment gives you multiple, interchangeable pods, and each one has its own IP that will soon be recreated. A Service is the stable home in front of them: you declare a name and a selector, and the platform hands the Service a virtual cluster IP plus a DNS name that stay fixed. Cluster DNS resolves that name, so components (other pods, an Ingress, a load balancer) refer to the Service, never to an individual pod address. That is the point of the abstraction, the address is durable even though everything standing behind it is disposable.

Selection drives which pods respond

A pod does not join a Service by being listed; it joins by carrying the labels the Service’s selector asks for, and it leaves when it no longer matches or is deleted. The platform maintains the endpoint set from the selector, adding a pod once it is running and ready, removing it on scale-down or failure. So the ‘backend list’ is a derived, always-changing view rather than a hand-edited roster. Because membership is derived from labels plus readiness, a pod that becomes unhealthy is dropped automatically and traffic is never aimed at a pod that should not serve.

Traffic goes to whatever matches

All of this gives you indirection that survives churn. Clients hold one name; the set behind it can be scaled, replaced, or rolled without them changing anything. A rolling update swaps pods one by one, readiness-gated, and the Service keeps forwarding to the current healthy matching set, so callers see no interruption. The guarantee you rely on is: as long as the selector is precise and the pods match it, the Service routes to the right set at every moment. Keep the selector tight, gate readiness, and this stable home stays stable even when every backer is new.

The stable name is a promise to callers

A companion article explains how the virtual address itself is allocated; this one is about what that stability means to whoever calls it. A well-behaved client resolves the Service name once, keeps using it for the life of the process, and survives every backend being replaced without a config push. That promise holds only while three habits hold: the selector keeps matching exactly the intended pods, readiness keeps unready pods out of the set, and no caller cheats by caching a pod IP it observed once. Treat the Service name as the only supported handle, and let scaling, crashes, and rollouts happen behind it.

When it breaks

The quietest failure is a selector that matches nothing. A typo such as app: webs instead of app: web, or a Deployment that stopped setting the expected label, leaves the Service with zero endpoints. The cluster IP and DNS name still resolve, so the outage looks like refused connections rather than a missing name. When a Service answers lookups but carries no traffic, inspect its endpoint set first; an empty set points at labels, not at the network.

A headless Service chosen by habit is the next trap. Setting clusterIP: None removes the virtual IP, so DNS returns the changing pod addresses and each client must balance across them. Code written for one stable endpoint suddenly faces many shifting ones, with uneven retries and broken connection reuse. Choose headless only when the caller explicitly wants discovery.

Ports break the promise from the other side. A Service port pointed at the wrong container targetPort accepts traffic it can never deliver, and a hand-picked NodePort can collide with another Service on the same nodes. Finally, leaving sessionAffinity: ClientIP on a stateless pool pins each client to one backend, so one dying pod looks like a total outage. Keep affinity off unless stickiness is the requirement.

Check it quickly

Save the following as service_endpoints.py in a scratch directory and run python3 service_endpoints.py. It models endpoint convergence with plain dictionaries and changes nothing outside the process: one function derives the endpoint set from a selector plus pod labels and readiness, and four scenarios exercise it, including a deliberate empty selector as the negative control.

"""Simulate Service endpoint convergence from selector plus readiness."""

PODS = [
 {"name": "web-1", "labels": {"app": "web"}, "ready": True},
 {"name": "web-2", "labels": {"app": "web"}, "ready": True},
 {"name": "web-3", "labels": {"app": "web"}, "ready": False},
 {"name": "worker-1", "labels": {"app": "worker"}, "ready": True},
]


def select_endpoints(pods, selector):
 """Derive the endpoint set; an empty selector warns instead of matching."""
 if not selector:
 return [], "warning: empty selector matches nothing"
 matched = sorted(
 pod["name"]
 for pod in pods
 if pod.get("ready")
 and all(pod.get("labels", {}).get(key) == value for key, value in selector.items())
 )
 return matched, ""


def check(condition, message):
 if not condition:
 raise AssertionError(message)


def main():
 endpoints, warning = select_endpoints(PODS, {"app": "web"})
 check(endpoints == ["web-1", "web-2"], "ready web pods should form the set")
 check(warning == "", "a precise selector must not warn")
 print("steady state:", endpoints)

 rolling = [dict(pod, ready=True) if pod["name"] == "web-3" else pod for pod in PODS]
 rolling = [pod for pod in rolling if pod["name"] != "web-1"]
 endpoints, _ = select_endpoints(rolling, {"app": "web"})
 check(endpoints == ["web-2", "web-3"], "rollover must swap one ready pod for another")
 print("after rollover:", endpoints)

 endpoints, _ = select_endpoints(PODS, {"app": "web", "track": "stable"})
 check(endpoints == [], "an over-narrow selector should converge to empty, not error")
 print("over-narrow selector:", endpoints)

 endpoints, warning = select_endpoints(PODS, {})
 check(endpoints == [] and "empty selector" in warning, "empty selector must warn loudly")
 print("empty selector:", repr(warning))

 print("PASS: four convergences; readiness-gated selection, rollover, and empty warning hold")


if __name__ == "__main__":
 main()

The four scenarios converge as a healthy Service should: the steady state holds the two ready web pods while the unready one stays out, and the rollover swaps one ready pod for another without changing the selector. The empty selector is the negative control: it warns loudly instead of silently matching everything or nothing. All assertions passed with Python 3.14.4 on macOS.

The simulation checks label-plus-readiness convergence only. It does not run kube-proxy, cluster DNS, or real probe timing. Before deploying, inspect the live endpoint set alongside pod labels and readiness, and repeat the lookup against the real Service name.

Sources

Kubernetes: Services, the Service abstraction, checked 2027-08-15.
Kubernetes: Pods, the pods a Service selects, checked 2027-08-15.
Kubernetes: Deployments, managing the pod set, checked 2027-08-15.

Startup probes delay other probes until initialization succeeds

Sun, 22 Aug 2027 00:00:00 +0000

A startup probe gives a container a separate initialization contract. While that configured probe has not succeeded, Kubernetes does not execute the container’s liveness or readiness probes. This is stronger and more specific than ignoring some liveness failures during a grace period: those other checks have not begun. Once startup succeeds, ordinary ongoing health and readiness behavior takes over.

Give each stage an observable condition

The application process starts before its startup probe can succeed. A Running container may therefore still have started reported as false in container status and may not make its Pod Ready. Running describes execution; successful startup describes completion of the configured startup check. An HTTP listener that opens early can expose progress without yet claiming that initialization is complete.

Choose a startup condition that represents necessary local preparation, such as loading an index or validating required initial state. A port opening is sufficient only if that is the actual readiness prerequisite you intend to observe. If the listener appears before expensive setup finishes, checking only that listener can release liveness protection too early and recreate the original restart loop.

After a successful startup check, kubelet stops that startup probe for the current container instance. It runs liveness and readiness according to their separate configuration. Startup success is not a permanent health certificate: the application can later become unready or require a restart. A new container instance must complete its own startup check again.

When it breaks

A startup probe that never succeeds eventually reaches its failure threshold. Kubelet then terminates the container and applies its restart policy. Startup protection is therefore not permission to initialize forever. Set period, timeout and failure threshold from measured initialization behavior, allowing for the environment’s realistic resource contention and expected recovery time.

The product of failureThreshold and periodSeconds is useful when planning the startup allowance, but do not present it as an exact stopwatch for every observed restart. Initial delay, probe execution and scheduling, termination grace and restart backoff also affect the timeline. Record the actual attempts and state changes when investigating a slow deployment.

Do not make successful readiness a prerequisite for liveness unless that dependency is explicitly represented by startup configuration. Readiness and liveness otherwise answer separate ongoing questions. Equally, passing startup does not force readiness to succeed; a later readiness check can still reject traffic eligibility for a different reason, while the process remains alive.

Changing checks requires application support. An endpoint that always returns success cannot verify a missing initialization step merely because it is named startup. Keep its state transition deliberate, expose enough diagnostic information to explain waiting, and avoid placing credentials or sensitive initialization details in responses readable by unintended clients. The fixture below uses only counters and synthetic state.

Check it quickly

Use a disposable Kubernetes cluster, kubectl and python:3.11-slim-bookworm available to its nodes. Save the script as startup-gate.py and run python3 startup-gate.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. Namespace creation and deletion permissions are required. The recorded server was Kubernetes 1.34.11. The startup probe allows sixty failed attempts at one-second periods for this short controlled exercise.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-startup-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 server = '''from http.server import BaseHTTPRequestHandler,HTTPServer
from pathlib import Path
import json
counts={'startup':0,'live':0,'ready':0}
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 key=self.path.removeprefix('/'); code=200
 if key in counts:
 counts[key]+=1
 if key=='startup' and not Path('/tmp/start').exists(): code=503
 body=json.dumps(counts).encode(); self.send_response(code)
 self.send_header('Content-Length',str(len(body))); self.end_headers(); self.wfile.write(body)
 def log_message(self,*args): pass
HTTPServer(('0.0.0.0',8080),Handler).serve_forever()
'''
 podspec = spec(server)
 c = podspec['containers'][0]
 c['startupProbe'] = {'httpGet': {'path': '/startup', 'port': 8080}, 'periodSeconds': 1, 'failureThreshold': 60}
 for field, path in [('livenessProbe', '/live'), ('readinessProbe', '/ready')]:
 c[field] = {'httpGet': {'path': path, 'port': 8080}, 'periodSeconds': 1, 'failureThreshold': 2}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'slow'}, 'spec': podspec})
 wait_for(lambda: obj(namespace, 'pod', 'slow'), lambda p:
 any('running' in c['state'] for c in p.get('status', {}).get('containerStatuses', [])))
 read = 'from urllib.request import urlopen; print(urlopen("http://127.0.0.1:8080/stats",timeout=3).read().decode())'
 def counts():
 result = k('exec', '-n', namespace, 'slow', '--', 'python3', '-c', read, expected=None)
 return json.loads(result.stdout) if result.returncode == 0 else None
 before = wait_for(counts, lambda d: d and d['startup'] >= 3)
 assert before['live'] == before['ready'] == 0
 starting = obj(namespace, 'pod', 'slow')
 assert not pod_ready(starting)
 assert starting['status']['containerStatuses'][0]['started'] is False
 assert starting['status']['containerStatuses'][0]['restartCount'] == 0
 k('exec', '-n', namespace, 'slow', '--', 'python3', '-c',
 'from pathlib import Path; Path("/tmp/start").touch()')
 ready = wait_for(lambda: obj(namespace, 'pod', 'slow'), pod_ready)
 assert ready['status']['containerStatuses'][0]['started'] is True
 enabled = wait_for(counts, lambda d: d and d['live'] > 0 and d['ready'] > 0)
 time.sleep(3)
 after = counts()
 assert after['startup'] == enabled['startup']
 assert after['live'] > enabled['live'] and after['ready'] > enabled['ready']
 assert obj(namespace, 'pod', 'slow')['status']['containerStatuses'][0]['restartCount'] == 0
 print(json.dumps({'before_startup_success': before, 'after_startup_success': after,
 'other_probes_suppressed_before_startup': True,
 'startup_probe_stops_after_success': True, 'container_restarts': 0}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The HTTP server counts requests separately for startup, live and ready paths. Its statistics path is observational and does not increment those counters. Startup initially returns 503 until a local release file appears. After three observed startup requests, both other counters are zero, the container has not restarted, and its started field is false.

Creating the release file makes the next startup request succeed. The Pod becomes Ready, started becomes true, and the live and ready counters begin increasing. After another three seconds, the startup counter stays fixed while both ongoing counters increase. The recorded final counts were four, four and five; exact ongoing counts can vary with timing.

All assertions passed without a restart, and the namespace was removed. This verifies suppression and transition using actual HTTP calls. It does not measure a production boot-time percentile or exercise startup-budget exhaustion. Use those additional measurements when choosing real thresholds, while keeping startup completion distinct from continuing service health.

Sources

Kubernetes: Configure Probes, startup failure budgets and probe configuration, checked 2026-09-13.
Kubernetes: Pod Lifecycle, suppression of liveness and readiness until startup succeeds, checked 2026-09-13.

Kubernetes Secrets require explicit access boundaries

Sat, 21 Aug 2027 00:00:00 +0000

A Kubernetes Secret is an API object for confidential values that workloads need. It gives operators a dedicated object type, consumption paths, and access-control targets. It does not make data unreadable to every other actor. Design the boundary around who can retrieve the object, create consumers, inspect their output, and administer the nodes that run them.

Encoding and protection are different layers

In the JSON API representation, Secret data values use base64; stringData is a write convenience for supplying strings. Base64 can be reversed without a key and is not encryption. The representation does not prove how a particular cluster protects its backing storage. Kubernetes documentation describes Secrets as unencrypted at rest by default unless encryption is configured. Verify the actual API-server and storage setup rather than inferring it from encoded output.

Storage encryption also does not prevent an authorized API reader from obtaining the data. RBAC rules for get, list, and watch can expose Secret contents, so a broad listing permission is not merely access to object names. Protect backups and diagnostic output too. A working application eventually receives usable values, which means a compromised authorized consumer remains inside the data’s trust boundary.

Secrets are intended for small values, with a one-MiB object limit. Mount only required keys and restrict access to the relevant containers. Unix file modes can narrow access within a workload, but do not replace API authorization or control of workload creation. Environment injection also has an exposure and lifecycle tradeoff; choose it from the application’s needs rather than assuming it is automatically safer than a file.

When it breaks

Denying get on Secrets does not necessarily deny indirect access. A subject allowed to create Pods in a namespace can normally name a Secret for a volume in its Pod specification. Kubelet then supplies the data to that workload. Permissions to create suitable controllers can have similar consequences. Review workload-creation rights and applicable admission controls alongside direct data-reading rules.

Broadly mounting every key expands the set of values a process can read. Selecting one key through volume items limits that particular projection, but a privileged creator may be able to request a different projection. Do not mistake a narrow manifest for proof that the actor writing it has equally narrow authority. The intended consumer and the authority to define consumers are separate parts of the boundary.

Rotation requires more than editing an API object. Delivery, application reload, credential issuance, and revocation have their own timing. Environment snapshots and subPath mounts do not automatically refresh like a normal Secret projection. An updated file also does not force an application to discard a value it already cached. Test the issuing system and consumers together before claiming that old credentials are no longer accepted.

Check it quickly

Use a disposable Kubernetes cluster with administrator access for test impersonation and python:3.11-slim-bookworm available on its nodes. Save this as secret-access.py and run python3 secret-access.py KUBECONFIG_PATH CONTEXT against its dedicated configuration. Kubernetes 1.34.11 was used for the recorded run. Every value in this fixture is deliberately synthetic, and output reports checks without printing the stored value.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

import base64
namespace = 'notes-secrets-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 demo = 'DEMO_ONLY_NOT_A_CREDENTIAL'
 k('create', '-n', namespace, '-f', '-', payload={'apiVersion': 'v1', 'kind': 'Secret', 'metadata': {'name': 'example'},
 'type': 'Opaque', 'stringData': {'needed': demo, 'unused': 'UNUSED_DEMO'}})
 stored = obj(namespace, 'secret', 'example')
 assert base64.b64decode(stored['data']['needed']).decode() == demo
 apply(namespace, {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'name': 'creator'},
 'automountServiceAccountToken': False})
 apply(namespace, {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'Role',
 'metadata': {'name': 'pod-creator'},
 'rules': [{'apiGroups': [''], 'resources': ['pods'], 'verbs': ['create', 'get']},
 {'apiGroups': [''], 'resources': ['pods/log'], 'verbs': ['get']}]})
 apply(namespace, {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'RoleBinding',
 'metadata': {'name': 'pod-creator'},
 'subjects': [{'kind': 'ServiceAccount', 'name': 'creator', 'namespace': namespace}],
 'roleRef': {'apiGroup': 'rbac.authorization.k8s.io', 'kind': 'Role', 'name': 'pod-creator'}})
 identity = 'system:serviceaccount:' + namespace + ':creator'
 denied = k('--as', identity, 'get', 'secret', 'example', '-n', namespace, expected=None)
 assert denied.returncode != 0 and 'Forbidden' in denied.stderr
 probe = "from pathlib import Path\nimport json,time\nroot=Path('/example')\nprint(json.dumps({'synthetic_value_matches':(root/'one').read_text()=='DEMO_ONLY_NOT_A_CREDENTIAL',\n 'unused_key_absent':not (root/'unused').exists(),\n 'service_account_token_absent':not Path('/var/run/secrets/kubernetes.io/serviceaccount/token').exists()}),flush=True)\ntime.sleep(600)"
 podspec = spec(probe)
 podspec['serviceAccountName'] = 'creator'
 podspec['volumes'] = [{'name': 'example', 'secret': {'secretName': 'example',
 'items': [{'key': 'needed', 'path': 'one'}], 'defaultMode': 0o400}}]
 podspec['containers'][0]['volumeMounts'] = [{'name': 'example', 'mountPath': '/example', 'readOnly': True}]
 definition = {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'reader', 'namespace': namespace},
 'spec': podspec}
 k('--as', identity, 'create', '-f', '-', payload=definition)
 wait_for(lambda: obj(namespace, 'pod', 'reader'), pod_ready)
 output = wait_for(lambda: k('--as', identity, 'logs', '-n', namespace, 'reader').stdout.strip(), bool)
 observed = json.loads(output)
 assert all(observed.values())
 print(json.dumps({'api_base64_decodes_to_synthetic_value': True,
 'direct_secret_get_forbidden': True, 'same_identity_can_create_consuming_pod': True,
 'mount_observation': observed, 'etcd_encryption_tested': False}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The script creates a Secret with two dummy keys and confirms that its API base64 representation decodes to the supplied synthetic text. It binds a service account to a Role permitting Pod creation and inspection of Pod logs, without permitting Secret reads. A request impersonating that account to get the Secret returns Forbidden.

The same impersonated subject then creates a Pod that mounts only the needed key. Its own program verifies the expected dummy value, absence of the unused key, and absence of an automatically mounted service-account token. The subject can read those boolean results through its permitted log access. All checks pass, demonstrating the indirect workload path despite the direct denial.

The namespace is removed afterward. No real credential, issued service-account token, backing-store encryption, or rotation process was tested. The experiment establishes authorization and projection behavior, not a complete secret-management system. Assess both direct readers and actors who can create or alter readers before describing a Secret as narrowly scoped.

Sources

Kubernetes: Secrets, API representation, consumption and access implications, checked 2026-09-13.
Kubernetes: Good practices for Secrets, encryption, least privilege and workload access, checked 2026-09-13.
Kubernetes: RBAC authorization, resource verbs, role bindings and namespace scope, checked 2026-09-13.

ConfigMaps separate configuration from the image

Fri, 20 Aug 2027 00:00:00 +0000

A ConfigMap stores non-confidential configuration separately from an image. That lets a release change an environment-specific value without rebuilding the application artifact. The useful boundary is explicit: the image supplies code, while the workload definition identifies which configuration that code should consume. A release is therefore more than an image tag; its configuration reference is part of the behavior being deployed.

Give configuration a reviewable identity

ConfigMap data values are strings, with binaryData available for binary values represented through the API’s encoding. The object is intended for small configuration, with a one-MiB data limit. It is not a general file repository or a secret-protection mechanism. The API can validate object structure without understanding whether a particular application setting is meaningful or safe for the business.

A Pod can consume values through environment references or mounted files. Ordinary Pod references resolve the ConfigMap in that Pod’s namespace. Choose names and required keys deliberately, then validate the application’s interpretation. An accepted Deployment object does not prove that every referenced key exists or that a new container can start successfully with the selected values.

Define units, accepted values, and defaults in the application’s configuration schema. A stored string resembling a boolean or number does not choose its parsing rules. Review those rules with the consumer before promoting a configuration between environments.

Versioned immutable ConfigMaps make release identity easier to inspect. A name such as settings-v2 can be retained alongside settings-v1, with immutable preventing data changes to the existing object. This protection does not prohibit an authorized actor from deleting and recreating a name. Keep object lifecycle permissions and release records aligned with the intended immutability promise.

When it breaks

Editing configuration and replacing processes are different actions. Environment values already injected into a running container are not rewritten by a ConfigMap update. Normal projected files can update eventually, but the application must read the new contents; subPath and previously opened files introduce additional lifetime distinctions. Do not report a configuration rollout merely because the API now shows a new value.

Changing a Deployment’s Pod-template reference to a new ConfigMap is a concrete rollout trigger. Changing the ConfigMap itself does not automatically rewrite that template. Likewise, a checksum annotation is useful only when a tool actually updates it in the Pod template. Kubernetes does not discover arbitrary external configuration changes and generate that checksum on the application’s behalf.

Rollback depends on retained inputs. Repointing a workload to settings-v1 fails if the old object has been deleted, and restoring an image alone cannot restore a changed configuration. Keep needed versions available and check consumers before cleanup. Configuration changes also need business validation; a syntactically valid threshold can still disable a service or change its behavior incorrectly.

Check it quickly

Use a disposable Kubernetes cluster with kubectl access and python:3.11-slim-bookworm available to its nodes. Save this as config-releases.py and run python3 config-releases.py KUBECONFIG_PATH CONTEXT, substituting that cluster’s dedicated configuration path and context name. The test needs permission to create and delete its own namespace. The recorded server version was Kubernetes 1.34.11, with kubectl 1.34.1.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}


def deployment(namespace, name, podspec):
 return apply(namespace, {'apiVersion': 'apps/v1', 'kind': 'Deployment', 'metadata': {'name': name},
 'spec': {'replicas': 1, 'selector': {'matchLabels': {'app': name}},
 'template': {'metadata': {'labels': {'app': name}}, 'spec': podspec}}})


def deployment_pod(namespace, name):
 wait_for(lambda: obj(namespace, 'deployment', name), lambda d:
 d.get('status', {}).get('observedGeneration', 0) >= d['metadata']['generation']
 and d.get('status', {}).get('updatedReplicas') == 1
 and d.get('status', {}).get('availableReplicas') == 1
 and d.get('status', {}).get('replicas') == 1)
 def pods():
 return [p for p in obj(namespace, 'pods', selector='app=' + name)['items']
 if not p['metadata'].get('deletionTimestamp')]
 return wait_for(pods, lambda p: len(p) == 1 and pod_ready(p[0]))[0]

namespace = 'notes-config-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 for version, mode in [('v1', 'blue'), ('v2', 'green')]:
 apply(namespace, {'apiVersion': 'v1', 'kind': 'ConfigMap',
 'metadata': {'name': 'settings-' + version}, 'immutable': True, 'data': {'mode': mode}})
 podspec = spec()
 podspec['containers'][0]['env'] = [{'name': 'MODE', 'valueFrom':
 {'configMapKeyRef': {'name': 'settings-v1', 'key': 'mode'}}}]
 deployment(namespace, 'web', podspec)
 first = deployment_pod(namespace, 'web')
 def mode(pod):
 return k('exec', '-n', namespace, pod['metadata']['name'], '--', 'python3', '-c',
 'import os; print(os.environ["MODE"])').stdout.strip()
 assert mode(first) == 'blue'
 rejected = k('patch', 'configmap', 'settings-v1', '-n', namespace, '--type=merge',
 '-p', json.dumps({'data': {'mode': 'changed'}}), expected=None)
 assert rejected.returncode != 0 and 'immutable' in rejected.stderr
 assert obj(namespace, 'configmap', 'settings-v1')['data']['mode'] == 'blue'
 def select(version):
 patch = [{'op': 'replace',
 'path': '/spec/template/spec/containers/0/env/0/valueFrom/configMapKeyRef/name',
 'value': 'settings-' + version}]
 k('patch', 'deployment', 'web', '-n', namespace, '--type=json', '-p', json.dumps(patch))
 return deployment_pod(namespace, 'web')
 second = select('v2')
 assert mode(second) == 'green' and second['metadata']['uid'] != first['metadata']['uid']
 third = select('v1')
 assert mode(third) == 'blue' and third['metadata']['uid'] != second['metadata']['uid']
 image_ids = [p['status']['containerStatuses'][0]['imageID'] for p in (first, second, third)]
 assert len(set(image_ids)) == 1
 assert len({p['metadata']['uid'] for p in (first, second, third)}) == 3
 print(json.dumps({'modes': ['blue', 'green', 'blue'], 'distinct_pod_uids': 3,
 'same_image_id': image_ids[0], 'immutable_update_rejected': True,
 'retained_config_rollback_verified': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The script creates two immutable ConfigMaps and a single-replica Deployment referencing settings-v1. The first Pod reads blue from its environment. Attempting to overwrite that ConfigMap’s data is rejected, and reading the object confirms the original value remains blue. This checks actual server enforcement rather than just the manifest’s immutable field.

Next, a Pod-template patch selects settings-v2. After controller reconciliation and readiness, a different Pod reads green. Selecting the retained first version causes another replacement that reads blue again. The three Pod UIDs are distinct, while all three observed image identities match. All assertions passed, and the temporary namespace was removed.

This verifies configuration selection, replacement, and a retained-input rollback using a simple worker. It does not establish application compatibility, production availability during rollout, or rollback of external side effects. Carry the image, configuration version, workload definition, and observed application result together in the release evidence so each part of the change can be reviewed.

Sources

Kubernetes: ConfigMaps, non-confidential data, consumption and immutability, checked 2026-09-13.
Kubernetes: Deployments, Pod-template changes and rolling replacement, checked 2026-09-13.

Kubernetes Deployments describe desired state

Thu, 19 Aug 2027 00:00:00 +0000

You describe the end state, not the steps. That idea underlies replicas, updates, and rollback: the cluster keeps working to match your description.

Spec beats imperative steps

A Deployment is a statement of the end state, not a list of steps: how many replicas, which image, what update strategy, which labels. The operator publishes the what, and the controller, a reconcile loop, is responsible for the how. The difference from running a command is structural: a one-off docker run or kubectl run orders something to happen now, while a Deployment says ’this is the state I want to remain true.’ If the work is interrupted, the command fails and the state may be left half-done, whereas the controller keeps trying to converge, which is exactly what turns startups and fixes into things the system will keep pursuing.

A controller converges the gap

The reconcile loop is what makes a Deployment trustworthy. Whenever the controller observes that reality differs from the spec, fewer replicas than declared, a pod evicted, a node lost, it does the smallest corrective step and repeats until the gap is closed. Nobody runs ‘create three more’ by hand; the controller re-converges every few seconds. That is the essential difference from running imperative commands, drift is not something you notice and fix, it is a condition the system continuously returns to spec on its own. It is why a crashed pod is replaced and an evicted workload comes back, without an operator watching.

Updates happen through new Revision

Change itself is described as state. A rolling update is a new desired image; the controller stages a new controller-generation and swaps traffic gradually, respecting readiness so the fleet moves forward under control. Because each change creates a new Revision with its own record, a rollout is auditable, and roll back is simply applying a prior Revision rather than writing a new, unplanned change. That keeps the cluster maintainable at scale: operators express intent once, the controller enforces it forever, and history of every change is a versioned object rather than a notebook of commands.

When it breaks

An immutable selector turns a small label edit into lasting drift. spec.selector cannot change after creation, so editing only spec.template.labels leaves new pods outside the selector: the ReplicaSet reports missing replicas while stray pods run unowned nearby. The mirror mistake is two Deployments with overlapping selectors, where each controller counts the other’s pods as observed state and they fight over scale. Treat the selector plus template labels as one atomic contract; when the selector must change, create a new Deployment rather than patching the old one.

Manual edits lose to the loop by design. Scaling the ReplicaSet directly, deleting a pod to save capacity, or running kubectl scale without updating the declared spec works for seconds, then the Deployment controller restores spec.replicas on the next sync. The symptom looks like a flaky API, but kubectl describe shows the controller closing the same gap every time. If an emergency scale is real, change the declared desired state in Git, not the observed objects.

History can outgrow its usefulness. Every spec change keeps one ReplicaSet while revisionHistoryLimit allows, so a pipeline that applies on every commit with a limit of fifty fills etcd with dead revisions and slows rollout history. A limit of zero has the opposite cost: no rollback target remains. Keep the default of ten without a measured reason, and keep the audit in version control rather than in the cluster.

Check it quickly

Save the following as desired_state_loop.py and run python3 desired_state_loop.py. It uses only the standard library and models the same loop the controller runs: compare desired against observed, emit one corrective action per tick, and repeat until the gap closes.

"""Simulate a Deployment-style desired-state reconcile loop (stdlib only)."""

def reconcile_step(desired, observed):
 if observed < desired:
 return "create-one-pod"
 if observed > desired:
 return "delete-one-pod"
 return "in-sync"


def converge(desired, observed, log):
 while observed != desired:
 action = reconcile_step(desired, observed)
 log.append(action)
 if action == "create-one-pod":
 observed += 1
 else:
 observed -= 1
 log.append("in-sync")
 return observed


def main():
 desired = 3
 observed = 1 # one survivor after a node loss
 actions = []
 observed = converge(desired, observed, actions)
 assert observed == desired
 assert actions == ["create-one-pod", "create-one-pod", "in-sync"], actions

 # Manual drift: someone adds two pods without changing desired.
 observed = 5
 drift = []
 observed = converge(desired, observed, drift)
 assert observed == desired
 assert drift == ["delete-one-pod", "delete-one-pod", "in-sync"], drift

 # Selector drift is not auto-healed: template labels outside selector never become observed.
 selector = {"app": "api"}
 template_labels = {"app": "api-v2"} # deliberate mismatch, a negative control
 matched = all(template_labels.get(k) == v for k, v in selector.items())
 assert not matched, "mismatched labels must not count as observed"

 print(f"desired={desired} final-observed={observed}")
 print(f"recovery-actions={actions}")
 print(f"manual-scale-reverted={drift}")
 print("PASS: controller closes the desired-vs-observed gap; label mismatch stays visible")


if __name__ == "__main__":
 main()

The two recoveries both end in sync, while the mismatched labels are the negative control: pods outside the selector never count as observed, so the loop exposes the contract error instead of healing it.

The assertions passed with Python 3.14.4 on macOS. The model checks convergence arithmetic, not scheduling or storage. Before release day, confirm the live selector, replicas, and revision history.

Sources

Kubernetes: Deployments, the Deployment resource explained, checked 2027-08-15.
Wikipedia: Kubernetes, the platform and its model, checked 2027-08-15.

A PVC gives a Pod durable storage beyond the container

Fri, 13 Aug 2027 00:00:00 +0000

A container is ephemeral by design, so a PersistentVolumeClaim is the handle that pins real storage to it even when the Pod is recreated. The split is deliberate: a PersistentVolume declares capacity that exists, a PersistentVolumeClaim requests capacity that is needed, and a controller matches one to the other. With a StorageClass the claim can create its own volume through a provisioner, so nobody has to pre-build the storage.

The part that bites is that claim and volume have separate lifetimes governed by separate rules. A finalizer decides when the claim is allowed to disappear, and a reclaim policy decides what happens to the volume afterwards. Get the deletion order wrong and you get a claim stuck in Terminating; get the policy wrong and you get either data you meant to keep or a bill for storage nobody is using.

The claim, the volume, and the binding

Binding is a conjunction, not a best match. A volume is a candidate only if its capacity is at least the request, its access modes include the requested mode, its storageClassName matches, and no other claim owns it. A claim that satisfies nothing stays Pending with an event naming the requirement that failed, which is more useful than a generic failure; the usual causes are an access mode the class cannot deliver and a class name that does not exist.

accessModes has no default; you must choose one, and ReadWriteOnce is what most single-node block provisioners actually deliver. It is also single-node, so a StatefulSet with replicas on different nodes cannot share one claim. When a class supports dynamic provisioning, volumeBindingMode sets the timing: Immediate provisions the volume the moment the claim appears, which can place a topology-constrained volume in a zone with no node to use it, while WaitForFirstConsumer keeps the claim Pending until a pod is scheduled and then provisions in that pod’s zone.

The deletion order and the finalizer

The order is pod, then claim, then volume, and the protection is implemented with finalizers. Deleting a claim while a pod still mounts it does not delete anything: the claim gets a deletion timestamp, flips to Terminating, and stops there because kubernetes.io/pvc-protection is still in its finalizer list. The controller removes that finalizer only once no pod references the claim, so the “Terminating forever” PVC is not a stuck controller but a claim something still uses.

The same protection exists in the other direction. A PersistentVolume carries kubernetes.io/pv-protection, so deleting a volume that is bound to a claim is postponed until it is no longer bound. That ordering makes an accidental kubectl delete pv recoverable, and explains why a volume can seem to ignore a delete for as long as its claim exists.

Reclaim policy: Delete or Retain

When the claim is finally gone the volume is released, and the reclaim policy takes over. Delete removes the PV object and the external storage asset with it. Retain keeps the PV object, marks it Released, and deliberately does not make it available for a new claim, because the previous claimant’s data is still on it — reclamation becomes a manual sequence: delete the PV, clean the storage asset, delete the asset.

Dynamically provisioned volumes inherit the policy from their StorageClass, whose default is Delete. That default is right for scratch and wrong for a database, and it is invisible: nothing in the PVC or pod spec mentions the policy, so the decision lives in a cluster-scoped object nobody reviewing the workload ever sees.

When it breaks

The classic failure is the claim that will not die. Someone deletes a PVC, sees Terminating, and waits — while a pod somewhere still mounts it, usually a StatefulSet that was never scaled down or a pod stuck on an unreachable node. Nothing errors, nothing times out, and the PVC blocks the storage behind it. Find the pod rather than deleting the finalizer by hand; removing it while a pod still writes is how you detach a volume under a running process.

The second failure is the Retain orphan. The claim is gone, the pods are gone, the namespace may be gone, and a Released PV plus its storage asset sit there indefinitely. Nothing is in use and nothing is in error, so no alert fires and the volume does not appear in any workload’s list. The inverse mistake is just as quiet: a class whose default Delete policy was never overridden, discovered only when someone tries to recover data that the reclaim already removed.

Check it quickly

Look at both sides of the binding, and at what holds the claim:

kubectl get pvc -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,STATUS:.status.phase,VOLUME:.spec.volumeName,CLASS:.spec.storageClassName
kubectl get pv -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,POLICY:.spec.persistentVolumeReclaimPolicy,CLAIM:.spec.claimRef.name
kubectl get pvc data -o jsonpath='{.metadata.finalizers}{"\n"}'
kubectl get pods -A -o json | jq -r '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName=="data") | "\(.metadata.namespace)/\(.metadata.name)"'

A claim in Terminating whose finalizers still contain kubernetes.io/pvc-protection is being held by a pod, and the fourth command names it. On the volume side, a PV stuck in Released with a Retain policy is an orphan: nothing references it, nothing will reuse it, and only manual reclamation clears it.

Sources

Persistent Volumes, binding, access modes, reclaim policies, Released state and the pvc/pv protection finalizers, checked 2026-09-16.
Storage Classes, dynamic provisioning, reclaimPolicy, and volumeBindingMode, checked 2026-09-16.

The HPA grows and shrinks a Deployment by load

Wed, 11 Aug 2027 00:00:00 +0000

A HorizontalPodAutoscaler is a feedback controller, not a capacity planner. Once every sync period — 15 seconds by default — it asks the resource metrics API for per-pod CPU, divides the observed value by the target to get a ratio, and writes the result into the target workload’s scale subresource. The formula is one line: desiredReplicas = ceil(currentReplicas * currentMetricValue / desiredMetricValue).

Everything interesting is in the qualifiers. The controller deliberately ignores small ratios, deliberately delays shrinking, and refuses to act at all when it cannot read a metric or cannot compute a denominator. Those behaviours make it stable, and they are also why a correctly configured-looking HPA can sit at a fixed replica count while the service burns.

The ratio and the dead band

For a utilization target the denominator is not a percentage but a number of millicores: a container with a 200m CPU request and a 60% target produces a desired metric of 120m. The ratio is then compared against 1.0 with a configurable tolerance, 0.1 by default, and the controller skips scaling entirely when the ratio is inside that band. The band is not a rounding detail; it is the anti-thrash mechanism.

The band means the observable behaviour is coarse. With a 120m desired metric, average usage of 110m, 120m and 132m all leave the replica count exactly where it was; 133m is the first value that moves it. The step itself is a ceiling, so 160m against 120m rounds three replicas up to four rather than to 3.99 — a fractional pod cannot exist, so the controller always errs on the side of capacity.

Missing metrics and the veto

Pods whose metrics have not arrived are set aside rather than counted as zero, and the controller then re-adds them conservatively: a missing pod is assumed to consume the full target when the decision is to scale down, and nothing when the decision is to scale up. That asymmetry shrinks the magnitude of any move in either direction. A pod with no CPU request has no denominator, so its utilization is undefined and the HPA takes no action for that metric — the commonest cause of an HPA that appears to do nothing.

Multiple metrics make this stricter rather than looser. The controller computes a count per metric and takes the largest, so a scale-up on any one metric still goes through. But if a metric cannot be converted into a count at all while the metrics that do resolve suggest scaling down, the whole scale-down is skipped. A broken metrics source therefore freezes shrinkage while leaving growth possible.

Stabilisation and rate limits

Scale-down is filtered through a 300-second stabilisation window. Every recommendation is recorded and the controller applies the highest one inside the window, approximating a rolling maximum. A brief spike that produced a recommendation of ten replicas keeps the applied count at ten until that recommendation ages out, so the first real shrink happens a full five minutes after load dropped. Scale-up has no stabilisation window at all.

Rate limiting is asymmetric too. The default scale-up policy allows the larger of 100% of current replicas or four pods to be added every 15 seconds, and scale-down allows 100% of replicas to be removed. A cold HPA starting from one replica can therefore go to five, then ten, then twenty, doubling before the metrics have had time to react — a feature during a spike and a hazard when the metric is wrong.

When it breaks

The classic failure is a target with no denominator. The HPA reports unknown utilization, logs a condition such as FailedGetResourceMetric or ScalingActive=False, and never changes the replica count — so the symptom is not an error page, it is a service that keeps timing out at a fixed size. The cause is usually a container without a CPU request, or a missing metrics-server; kubectl create deployment emits resources: {} and has no flag to set one, so this is the default state of a hand-made manifest.

The second failure is a target that cannot be reached. Because maxReplicas is applied after the ratio, a workload needing more than the cap keeps computing a larger number every tick and keeps being truncated: utilization stays pegged, the status reports that it cannot scale further, and no event says the target itself is impossible. Set the cap from measured capacity, not from hope.

Check it quickly

Confirm that a metric exists, that a denominator exists, and what the controller decided:

kubectl get hpa web -o wide
kubectl top pods -l app=web
kubectl get deploy web -o jsonpath='{.spec.template.spec.containers[0].resources.requests.cpu}{"\n"}'
kubectl get hpa web -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
kubectl describe hpa web | sed -n '/Events/,$p'

The first line shows current against target utilization; <unknown> means the metric is missing. The third prints nothing when no CPU request is set, which is the usual reason the second command works while the HPA still refuses to scale. The last two name the condition reason and any FailedGetResourceMetric event, which is where the real explanation lives.

Sources

Horizontal Pod Autoscaling, the algorithm, tolerance, missing-metric handling, stabilisation window and default policies, checked 2026-09-16.
HPA walkthrough, the metrics-server dependency and the requests prerequisite, checked 2026-09-16.
metrics-server, the resource metrics API implementation the HPA reads, checked 2026-09-16.

Liveness and readiness tell Kubernetes how the Pod feels

Wed, 14 Jul 2027 00:00:00 +0000

A slow start is not a dead process: readiness shields traffic while liveness only cares about keeping the process alive. Teams wire a single HTTP endpoint into both probes, and then every slow dependency becomes a restart loop — the app needs time, liveness refuses to grant it. Two probes exist because “not ready” and “dead” demand opposite responses: one asks for time, the other for a restart.

The mental model that prevents the confusion is a hospital triage. Readiness is the waiting-room nurse deciding who sees a doctor right now; liveness is the crash cart, rolled out only when the patient codes. You would never defibrillate someone for sitting in the waiting room. Keep that image and the probe configuration almost writes itself.

Readiness gates traffic

A readiness probe failure removes the Pod from the Service’s endpoints but leaves it running, so the process can finish warming up or recover on its own. Traffic flows again as soon as the probe passes. This makes readiness the safe probe to fail: caches can warm, migrations can finish, downstream dependencies can come back, all without losing the Pod’s in-memory state or paying a restart penalty.

Design the readiness endpoint around genuine servability. Check what makes this Pod able to answer correctly right now: migrations applied, required caches populated, critical downstream reachable. A probe that only checks “the HTTP server responds” admits traffic to Pods that will fail every request. But resist stuffing every dependency into it: if a non-critical recommendation service is down, failing readiness sheds real traffic over a decoration.

Liveness guards the process

A failing liveness probe triggers a restart, a last resort that ends a hopeless wedge. Use it for what it is a liveness issue—not for containers that are merely slow to become ready. The legitimate targets are narrow: deadlocks where no request will ever succeed again, runaway goroutine or thread leaks that only a fresh process clears, corrupted in-memory state that no retry can heal. In each case the defining property is irreversibility — without a restart, recovery is impossible, not merely slow.

That bar means many services should ship a minimal liveness probe or none at all. A liveness check against an external database turns every database blip into a fleet-wide restart storm. Prefer a cheap local check — a watchdog timestamp refreshed by the main loop — that detects “this process is wedged” without consulting anything outside the process. Kubernetes already restarts crashed containers; liveness exists only for the living dead.

Give the probes grace

Set the initialDelaySeconds high enough that a cold start is not counted as a crash, and keep timeoutSeconds consistent with real work. Separate the two probes so a slow pod is protected by readiness, not mistakenly recycled by liveness. Better still, add a startup probe for slow-booting containers: while it runs, liveness stays disabled, so version upgrades that warm large caches get minutes instead of seconds without weakening steady-state liveness at all.

Tune thresholds from observed behavior, not round numbers copied between services. periodSeconds times failureThreshold is the real reaction time — 10 seconds times 3 means 30 seconds of badness before action. Measure cold-start p99, dependency recovery time, and worst-case pauses, then set each number to react faster than the alert but slower than the noise.

When it breaks

The signature failure is the restart loop during dependency outages: the database slows, readiness would have sufficed, but liveness probes share the same failing endpoint and the whole fleet restarts in waves. Each restart wipes warm caches, making the next wave slower to recover, until the system oscillates indefinitely. Diagnose by correlating restart timestamps with the dependency incident; the fix is splitting the endpoints — readiness checks the dependency, liveness checks only the process — so a sick downstream sheds traffic instead of killing Pods.

The mirror failure is the zombie Pod: liveness passes because the HTTP server responds, but every request hangs behind a deadlocked worker pool. Traffic keeps flowing to a Pod that answers nothing. The fix is a liveness endpoint that exercises the real path — acquire a worker slot with a short timeout, verify the last successful request timestamp is fresh — so wedged state fails the probe while mere slowness does not.

Check it quickly

Inspect probe wiring and recent kills:

kubectl get pod web-0 -o jsonpath='{.spec.containers[*].livenessProbe}{"\n"}{.spec.containers[*].readinessProbe}{"\n"}'
kubectl get events --field-selector reason=Unhealthy --sort-by=.lastTimestamp | tail -n 10
kubectl describe pod web-0 | grep -A3 -i 'liveness\|readiness'

Shared endpoints across both probes, or Unhealthy events coinciding with downstream incidents, mark the split that still needs doing.

Sources

Kubernetes probes, Official docs on liveness and readiness, checked 2026-06-01.
Configure liveness and readiness probes, task guide for probe configuration, checked 2026-09-05.

Kubernetes · XB Field Notes

The scheduler accounts for requests, not momentary utilization

Account for the admitted specification

When it breaks

Check it quickly

Sources

Kubernetes CronJobs need a history policy

A finished Job is not free

Two mechanisms, two controllers

Owner references decide who cleans up

When it breaks

Check it quickly

Sources

cgroups enforce container budgets within a hierarchy

Read the controller that enforces the resource

When it breaks

Check it quickly

Sources

Cordon and drain take a node out safely

Cordon closes the door, drain clears the room

The refusal list is all or nothing

The budget, not the flags, decides the rest

When it breaks

Check it quickly

Sources

The Horizontal Pod Autoscaler scales by a target utilization

A smoothed average decides

Pick the metric you can act on

Automation is orthogonal to defaults

When it breaks

Check it quickly

Further reading

Sources

A CronJob turns a schedule into a pod

A schedule makes a job

Concurrency is a policy, not a guess

Missed runs have a budget of a hundred

When it breaks

Check it quickly

Sources

Node affinity places workloads on matching nodes

Match nodes by label

Prefer or require

Anti-affinity splits replicas

When it breaks

Check it quickly

Sources

Kubernetes rolling updates replace pods gradually

Replace a few at a time

Surge and drain windows

Roll back is a redeploy

Recreate stops everything first

When it breaks

Check it quickly

Further reading

Sources

Security context defines constraints that need runtime evidence

Put each setting at the right scope

When it breaks

Check it quickly

Sources

The horizontal autoscaler scales on demand

Metrics drive the count

Scale up eagerly, down gently

The metric is the weak link

When it breaks

Check it quickly

Further reading

Sources

Kubernetes labels define selection, not authorization

Know which selector language is in use

When it breaks

Check it quickly

Sources

Namespaces scope objects; policies establish other boundaries

Names are scoped by resource type

When it breaks

Check it quickly

Sources

Kubernetes Services give pods a stable home