Workloads · XB Field Notes

ConfigMaps separate configuration from the image

Fri, 20 Aug 2027 00:00:00 +0000

A ConfigMap stores non-confidential configuration separately from an image. That lets a release change an environment-specific value without rebuilding the application artifact. The useful boundary is explicit: the image supplies code, while the workload definition identifies which configuration that code should consume. A release is therefore more than an image tag; its configuration reference is part of the behavior being deployed.

Give configuration a reviewable identity

ConfigMap data values are strings, with binaryData available for binary values represented through the API’s encoding. The object is intended for small configuration, with a one-MiB data limit. It is not a general file repository or a secret-protection mechanism. The API can validate object structure without understanding whether a particular application setting is meaningful or safe for the business.

A Pod can consume values through environment references or mounted files. Ordinary Pod references resolve the ConfigMap in that Pod’s namespace. Choose names and required keys deliberately, then validate the application’s interpretation. An accepted Deployment object does not prove that every referenced key exists or that a new container can start successfully with the selected values.

Define units, accepted values, and defaults in the application’s configuration schema. A stored string resembling a boolean or number does not choose its parsing rules. Review those rules with the consumer before promoting a configuration between environments.

Versioned immutable ConfigMaps make release identity easier to inspect. A name such as settings-v2 can be retained alongside settings-v1, with immutable preventing data changes to the existing object. This protection does not prohibit an authorized actor from deleting and recreating a name. Keep object lifecycle permissions and release records aligned with the intended immutability promise.

When it breaks

Editing configuration and replacing processes are different actions. Environment values already injected into a running container are not rewritten by a ConfigMap update. Normal projected files can update eventually, but the application must read the new contents; subPath and previously opened files introduce additional lifetime distinctions. Do not report a configuration rollout merely because the API now shows a new value.

Changing a Deployment’s Pod-template reference to a new ConfigMap is a concrete rollout trigger. Changing the ConfigMap itself does not automatically rewrite that template. Likewise, a checksum annotation is useful only when a tool actually updates it in the Pod template. Kubernetes does not discover arbitrary external configuration changes and generate that checksum on the application’s behalf.

Rollback depends on retained inputs. Repointing a workload to settings-v1 fails if the old object has been deleted, and restoring an image alone cannot restore a changed configuration. Keep needed versions available and check consumers before cleanup. Configuration changes also need business validation; a syntactically valid threshold can still disable a service or change its behavior incorrectly.

Check it quickly

Use a disposable Kubernetes cluster with kubectl access and python:3.11-slim-bookworm available to its nodes. Save this as config-releases.py and run python3 config-releases.py KUBECONFIG_PATH CONTEXT, substituting that cluster’s dedicated configuration path and context name. The test needs permission to create and delete its own namespace. The recorded server version was Kubernetes 1.34.11, with kubectl 1.34.1.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}


def deployment(namespace, name, podspec):
 return apply(namespace, {'apiVersion': 'apps/v1', 'kind': 'Deployment', 'metadata': {'name': name},
 'spec': {'replicas': 1, 'selector': {'matchLabels': {'app': name}},
 'template': {'metadata': {'labels': {'app': name}}, 'spec': podspec}}})


def deployment_pod(namespace, name):
 wait_for(lambda: obj(namespace, 'deployment', name), lambda d:
 d.get('status', {}).get('observedGeneration', 0) >= d['metadata']['generation']
 and d.get('status', {}).get('updatedReplicas') == 1
 and d.get('status', {}).get('availableReplicas') == 1
 and d.get('status', {}).get('replicas') == 1)
 def pods():
 return [p for p in obj(namespace, 'pods', selector='app=' + name)['items']
 if not p['metadata'].get('deletionTimestamp')]
 return wait_for(pods, lambda p: len(p) == 1 and pod_ready(p[0]))[0]

namespace = 'notes-config-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 for version, mode in [('v1', 'blue'), ('v2', 'green')]:
 apply(namespace, {'apiVersion': 'v1', 'kind': 'ConfigMap',
 'metadata': {'name': 'settings-' + version}, 'immutable': True, 'data': {'mode': mode}})
 podspec = spec()
 podspec['containers'][0]['env'] = [{'name': 'MODE', 'valueFrom':
 {'configMapKeyRef': {'name': 'settings-v1', 'key': 'mode'}}}]
 deployment(namespace, 'web', podspec)
 first = deployment_pod(namespace, 'web')
 def mode(pod):
 return k('exec', '-n', namespace, pod['metadata']['name'], '--', 'python3', '-c',
 'import os; print(os.environ["MODE"])').stdout.strip()
 assert mode(first) == 'blue'
 rejected = k('patch', 'configmap', 'settings-v1', '-n', namespace, '--type=merge',
 '-p', json.dumps({'data': {'mode': 'changed'}}), expected=None)
 assert rejected.returncode != 0 and 'immutable' in rejected.stderr
 assert obj(namespace, 'configmap', 'settings-v1')['data']['mode'] == 'blue'
 def select(version):
 patch = [{'op': 'replace',
 'path': '/spec/template/spec/containers/0/env/0/valueFrom/configMapKeyRef/name',
 'value': 'settings-' + version}]
 k('patch', 'deployment', 'web', '-n', namespace, '--type=json', '-p', json.dumps(patch))
 return deployment_pod(namespace, 'web')
 second = select('v2')
 assert mode(second) == 'green' and second['metadata']['uid'] != first['metadata']['uid']
 third = select('v1')
 assert mode(third) == 'blue' and third['metadata']['uid'] != second['metadata']['uid']
 image_ids = [p['status']['containerStatuses'][0]['imageID'] for p in (first, second, third)]
 assert len(set(image_ids)) == 1
 assert len({p['metadata']['uid'] for p in (first, second, third)}) == 3
 print(json.dumps({'modes': ['blue', 'green', 'blue'], 'distinct_pod_uids': 3,
 'same_image_id': image_ids[0], 'immutable_update_rejected': True,
 'retained_config_rollback_verified': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The script creates two immutable ConfigMaps and a single-replica Deployment referencing settings-v1. The first Pod reads blue from its environment. Attempting to overwrite that ConfigMap’s data is rejected, and reading the object confirms the original value remains blue. This checks actual server enforcement rather than just the manifest’s immutable field.

Next, a Pod-template patch selects settings-v2. After controller reconciliation and readiness, a different Pod reads green. Selecting the retained first version causes another replacement that reads blue again. The three Pod UIDs are distinct, while all three observed image identities match. All assertions passed, and the temporary namespace was removed.

This verifies configuration selection, replacement, and a retained-input rollback using a simple worker. It does not establish application compatibility, production availability during rollout, or rollback of external side effects. Carry the image, configuration version, workload definition, and observed application result together in the release evidence so each part of the change can be reviewed.

Sources

Kubernetes: ConfigMaps, non-confidential data, consumption and immutability, checked 2026-09-13.
Kubernetes: Deployments, Pod-template changes and rolling replacement, checked 2026-09-13.

Kubernetes Deployments describe desired state

Thu, 19 Aug 2027 00:00:00 +0000

You describe the end state, not the steps. That idea underlies replicas, updates, and rollback: the cluster keeps working to match your description.

Spec beats imperative steps

A Deployment is a statement of the end state, not a list of steps: how many replicas, which image, what update strategy, which labels. The operator publishes the what, and the controller, a reconcile loop, is responsible for the how. The difference from running a command is structural: a one-off docker run or kubectl run orders something to happen now, while a Deployment says ’this is the state I want to remain true.’ If the work is interrupted, the command fails and the state may be left half-done, whereas the controller keeps trying to converge, which is exactly what turns startups and fixes into things the system will keep pursuing.

A controller converges the gap

The reconcile loop is what makes a Deployment trustworthy. Whenever the controller observes that reality differs from the spec, fewer replicas than declared, a pod evicted, a node lost, it does the smallest corrective step and repeats until the gap is closed. Nobody runs ‘create three more’ by hand; the controller re-converges every few seconds. That is the essential difference from running imperative commands, drift is not something you notice and fix, it is a condition the system continuously returns to spec on its own. It is why a crashed pod is replaced and an evicted workload comes back, without an operator watching.

Updates happen through new Revision

Change itself is described as state. A rolling update is a new desired image; the controller stages a new controller-generation and swaps traffic gradually, respecting readiness so the fleet moves forward under control. Because each change creates a new Revision with its own record, a rollout is auditable, and roll back is simply applying a prior Revision rather than writing a new, unplanned change. That keeps the cluster maintainable at scale: operators express intent once, the controller enforces it forever, and history of every change is a versioned object rather than a notebook of commands.

When it breaks

An immutable selector turns a small label edit into lasting drift. spec.selector cannot change after creation, so editing only spec.template.labels leaves new pods outside the selector: the ReplicaSet reports missing replicas while stray pods run unowned nearby. The mirror mistake is two Deployments with overlapping selectors, where each controller counts the other’s pods as observed state and they fight over scale. Treat the selector plus template labels as one atomic contract; when the selector must change, create a new Deployment rather than patching the old one.

Manual edits lose to the loop by design. Scaling the ReplicaSet directly, deleting a pod to save capacity, or running kubectl scale without updating the declared spec works for seconds, then the Deployment controller restores spec.replicas on the next sync. The symptom looks like a flaky API, but kubectl describe shows the controller closing the same gap every time. If an emergency scale is real, change the declared desired state in Git, not the observed objects.

History can outgrow its usefulness. Every spec change keeps one ReplicaSet while revisionHistoryLimit allows, so a pipeline that applies on every commit with a limit of fifty fills etcd with dead revisions and slows rollout history. A limit of zero has the opposite cost: no rollback target remains. Keep the default of ten without a measured reason, and keep the audit in version control rather than in the cluster.

Check it quickly

Save the following as desired_state_loop.py and run python3 desired_state_loop.py. It uses only the standard library and models the same loop the controller runs: compare desired against observed, emit one corrective action per tick, and repeat until the gap closes.

"""Simulate a Deployment-style desired-state reconcile loop (stdlib only)."""

def reconcile_step(desired, observed):
 if observed < desired:
 return "create-one-pod"
 if observed > desired:
 return "delete-one-pod"
 return "in-sync"


def converge(desired, observed, log):
 while observed != desired:
 action = reconcile_step(desired, observed)
 log.append(action)
 if action == "create-one-pod":
 observed += 1
 else:
 observed -= 1
 log.append("in-sync")
 return observed


def main():
 desired = 3
 observed = 1 # one survivor after a node loss
 actions = []
 observed = converge(desired, observed, actions)
 assert observed == desired
 assert actions == ["create-one-pod", "create-one-pod", "in-sync"], actions

 # Manual drift: someone adds two pods without changing desired.
 observed = 5
 drift = []
 observed = converge(desired, observed, drift)
 assert observed == desired
 assert drift == ["delete-one-pod", "delete-one-pod", "in-sync"], drift

 # Selector drift is not auto-healed: template labels outside selector never become observed.
 selector = {"app": "api"}
 template_labels = {"app": "api-v2"} # deliberate mismatch, a negative control
 matched = all(template_labels.get(k) == v for k, v in selector.items())
 assert not matched, "mismatched labels must not count as observed"

 print(f"desired={desired} final-observed={observed}")
 print(f"recovery-actions={actions}")
 print(f"manual-scale-reverted={drift}")
 print("PASS: controller closes the desired-vs-observed gap; label mismatch stays visible")


if __name__ == "__main__":
 main()

The two recoveries both end in sync, while the mismatched labels are the negative control: pods outside the selector never count as observed, so the loop exposes the contract error instead of healing it.

The assertions passed with Python 3.14.4 on macOS. The model checks convergence arithmetic, not scheduling or storage. Before release day, confirm the live selector, replicas, and revision history.

Sources

Kubernetes: Deployments, the Deployment resource explained, checked 2027-08-15.
Wikipedia: Kubernetes, the platform and its model, checked 2027-08-15.

A DaemonSet puts one Pod on every eligible node

Mon, 29 Mar 2027 00:00:00 +0000

Where a Deployment cares about total count, a DaemonSet cares about coverage: a new node immediately gains the same service. Node-level concerns — shipping logs, exposing hardware metrics, running the network fabric, enforcing policy — must exist everywhere or they exist nowhere useful. A per-node script or a sidecar in every pod approximates this and rots immediately: new nodes miss it, version skew creeps in, and nobody can answer “which nodes run the agent” without an audit.

The DaemonSet controller closes that gap by making coverage declarative. You describe the daemon once, the controller reconciles reality toward one pod per eligible node forever, and scaling the cluster automatically scales the daemon fleet. Coverage stops being a runbook item and becomes an invariant the system maintains.

One per node, by design

You describe the daemon once and the controller attaches a copy to every node, which is the natural home for node-level logging, metrics agents, and kube-proxy-style traffic. There is no scaling knob to size by hand. That absence is a feature: replica counts drift from intent the moment someone edits them under pressure, while “every eligible node” cannot drift — it is recomputed continuously from the node list. Joining nodes gain the pod during registration; leaving nodes release it; cordoned nodes keep theirs unless drained.

Because placement is per-node rather than per-capacity, DaemonSet pods typically need priority and preemption headroom to survive on full nodes. Give them a high PriorityClass and modest, honest resource requests: they must schedule even when the node is otherwise saturated, because a monitoring agent missing from an overloaded node blinds you exactly when visibility matters most.

Coverage over count

Because the desired number of replicas is implicit, a rollout touches each member automatically: add a node to the pool and the daemon appears there without extra work. That is the guarantee no per-node script can match. Updates roll across the fleet with maxUnavailable bounding disruption — set it to 1 and the daemon fleet refreshes node by node, never leaving the fleet uncovered. Version skew, the chronic disease of node scripts, becomes structurally impossible: the controller converges every node toward the same template, and kubectl rollout status ds/agent tells you when convergence completes.

This guarantee extends to heterogeneous fleets through node selectors and affinity: one DaemonSet per node class, each converging its own subset. GPU nodes get the device plugin daemon, edge nodes get the lightweight forwarder, and each controller minds only its own coverage.

Rides the node lifecycle

Tolerations let a DaemonSet reach specialized nodes where normal taints keep generic workloads off, preserving fleet-wide consistency. The canonical examples are the masters’ control-plane taints and GPU or spot-instance taints: without tolerations, precisely the most interesting nodes lack the agent, and your fleet view has holes shaped exactly like your special hardware. Use nodeSelectors to pick which subsets get the daemon while keeping every eligible node in sync — selection by label keeps the subsets declarative, so relabeling a node moves it between daemon fleets without touching any controller.

Respect eviction and drain semantics in return. DaemonSet pods ignore ordinary scheduling pressure by design, but they must still terminate gracefully on drain: configure terminationGracePeriodSeconds honestly and handle SIGTERM, or node maintenance turns into log gaps and metric holes.

When it breaks

The classic failure is the silently uncovered node: a new node pool with a taint nobody told the DaemonSet about, so agents run everywhere except the forty new machines — which are, of course, the ones under investigation. Diagnose by comparing node count against daemon pod count per pool: desiredNumberScheduled versus numberReady per DaemonSet, grouped by node labels. Any pool where the numbers diverge names the missing toleration or selector immediately.

The second failure is the bad daemon update rolling fleet-wide: a broken agent image with maxUnavailable too high takes down logging on many nodes at once, blinding the very investigation the logs would serve. Keep maxUnavailable at 1 for observability daemons, pause and verify on a canary pool first (a nodeSelector-scoped DaemonSet makes a natural canary), and remember that rolling back a DaemonSet is one command — use it fast, because every minute without agents is a minute of missing evidence.

Check it quickly

Compare desired coverage against reality:

kubectl get ds -A -o custom-columns=NAME:.metadata.name,DESIRED:.status.desiredNumberScheduled,READY:.status.numberReady
kubectl get pods -o wide -l app=log-agent | awk '{print $7}' | sort | uniq -c | wc -l
kubectl get nodes --no-headers | wc -l

The first shows per-daemon coverage gaps, and the pod-node count versus node count proves every machine actually runs its copy.

Sources

Kubernetes DaemonSet, documents daemonset behavior, checked 2026-06-01.
Kubernetes taints and tolerations, documents reaching tainted nodes, checked 2026-09-05.

Regular init containers establish startup prerequisites

Sun, 03 Jan 2027 00:00:00 +0000

A regular init container performs setup before a Pod’s application containers start. It has its own command, image and filesystem view, and can mount a volume that the application later reads. This separates a prerequisite from the main process, but does not make arbitrary setup safe or automatically bounded. Design the step around its actual retry behavior and the lifetime of its output.

Sequence is local to one Pod

Kubelet runs ordinary init containers in their declared order. Each must finish successfully before the next begins, and the application containers wait for that sequence. Inspect initContainerStatuses to distinguish a waiting step, a running step and a terminated attempt. A Pod’s phase alone does not explain which command failed or whether another attempt is underway.

Shared storage requires explicit mounts. An emptyDir can carry a generated file between init and application containers, and it survives a container restart within that Pod. It disappears with the Pod. The separate containers do not share their image root filesystems, and a file written only inside an init container’s writable layer is not automatically delivered to the application.

Ordinary init containers must complete. Kubernetes also supports restartable sidecars represented in initContainers with container-level restartPolicy set to Always; those have different lifecycle rules. Do not apply the ordinary run-to-completion description to every entry merely because it appears in that field. This article’s example uses only regular init containers.

When it breaks

A nonzero exit does not always fail the whole Pod immediately. With an ordinary application Pod’s Always policy, a failed regular init is retried until it succeeds; successful init steps do not restart merely because the application remains running. Backoff slows repeated attempts, but is not a fixed retry-count limit. With Pod restartPolicy Never, a failed initialization instead makes that Pod fail.

Write setup so another attempt can encounter existing output safely. An interrupted download may leave a partial file, and an emptyDir marker may survive the failed process. Validate the completed artifact and publish it deliberately. The counter used below is diagnostic state; it is not a production strategy for deciding that an external prerequisite must have become valid.

Ordering inside one Pod also does not serialize a fleet. Putting a database migration in every replica’s init sequence can run several migrations concurrently. Use the migration system’s actual locking and compatibility contract. Waiting for a dependency once cannot prove that dependency will remain available after initialization; continuing request handling and readiness need their own design.

Initialization affects resource placement too. For ordinary init containers, scheduling accounts for peak initialization requirements alongside the summed application requirements, with overhead where applicable. A brief but large setup request can prevent scheduling even when the steady application is small. Sidecars and Pod-level resource features require checking their additional accounting rules.

Check it quickly

Use a disposable Kubernetes cluster, kubectl access and python:3.11-slim-bookworm on its nodes. Save the code as init-order.py and run python3 init-order.py KUBECONFIG_PATH CONTEXT, replacing the arguments with explicit test-cluster details. The account must create and delete namespaces. The recorded server was Kubernetes 1.34.11; allow time for one failed initialization and its retry.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-init-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 prepare = r'''from pathlib import Path
import sys,time
root=Path('/work'); counter=root/'attempts'
attempt=int(counter.read_text())+1 if counter.exists() else 1
counter.write_text(str(attempt))
with (root/'order').open('a') as f: f.write('prepare'+str(attempt)+'\n')
if attempt == 1: sys.exit(7)
print('waiting for release',flush=True)
while not (root/'go').exists(): time.sleep(0.1)
(root/'input').write_text('ready')
'''
 transform = r'''from pathlib import Path
root=Path('/work'); assert (root/'input').read_text()=='ready'
(root/'output').write_text('READY')
with (root/'order').open('a') as f: f.write('transform\n')
'''
 main = r'''from pathlib import Path
import json,time
root=Path('/work'); assert (root/'output').read_text()=='READY'
with (root/'order').open('a') as f: f.write('app\n')
print(json.dumps({'order':(root/'order').read_text().splitlines(),
 'artifact':(root/'output').read_text()}),flush=True)
time.sleep(600)
'''
 podspec = spec(main)
 mount = [{'name': 'work', 'mountPath': '/work'}]
 podspec['volumes'] = [{'name': 'work', 'emptyDir': {}}]
 podspec['containers'][0]['volumeMounts'] = mount
 podspec['initContainers'] = []
 for name, script in [('prepare', prepare), ('transform', transform)]:
 container = spec(script)['containers'][0]
 container['name'] = name
 container['volumeMounts'] = mount
 podspec['initContainers'].append(container)
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'ordered'}, 'spec': podspec})
 def retry_running(p):
 statuses = p.get('status', {}).get('initContainerStatuses', [])
 return statuses and statuses[0]['restartCount'] >= 1 and 'running' in statuses[0]['state']
 waiting = wait_for(lambda: obj(namespace, 'pod', 'ordered'), retry_running)
 first = waiting['status']['initContainerStatuses'][0]
 assert first['lastState']['terminated']['exitCode'] == 7
 assert waiting['status']['phase'] == 'Pending'
 assert all('running' not in c['state'] for c in waiting['status'].get('containerStatuses', []))
 assert 'running' not in waiting['status']['initContainerStatuses'][1]['state']
 k('exec', '-n', namespace, 'ordered', '-c', 'prepare', '--', 'python3', '-c',
 'from pathlib import Path; Path("/work/go").touch()')
 ready = wait_for(lambda: obj(namespace, 'pod', 'ordered'), pod_ready)
 assert ready['metadata']['uid'] == waiting['metadata']['uid']
 output = wait_for(lambda: k('logs', '-n', namespace, 'ordered', '-c', 'app').stdout.strip(), bool)
 observed = json.loads(output)
 assert observed == {'order': ['prepare1', 'prepare2', 'transform', 'app'], 'artifact': 'READY'}
 assert ready['status']['initContainerStatuses'][0]['restartCount'] == 1
 assert all(c['state']['terminated']['exitCode'] == 0 for c in ready['status']['initContainerStatuses'])
 never = spec()
 never['restartPolicy'] = 'Never'
 failed_init = spec('raise SystemExit(7)')['containers'][0]
 failed_init['name'] = 'fails'
 never['initContainers'] = [failed_init]
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'never'}, 'spec': never})
 failed = wait_for(lambda: obj(namespace, 'pod', 'never'), lambda p: p.get('status', {}).get('phase') == 'Failed')
 status = failed['status']['initContainerStatuses'][0]
 assert status['restartCount'] == 0 and status['state']['terminated']['exitCode'] == 7
 assert not any('running' in c['state'] for c in failed['status'].get('containerStatuses', []))
 print(json.dumps({'ordered_artifact': observed, 'retried_init_count': 1,
 'app_waited_for_both_regular_init_containers': True,
 'never_policy_failed_pod_without_retry': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The preparation container records its first attempt and exits seven. During its second attempt it waits for a release file. The script confirms the Pod is Pending, the first failure is recorded, and neither the next init step nor the application is running. It then releases preparation through that container’s shared volume.

The second init transforms ready into READY, and the application validates the artifact. Its log records prepare1, prepare2, transform and app in that order; the first init’s restart count is one. A separate Pod with Never policy exits initialization unsuccessfully and reaches Failed with no init retry. All assertions passed, and both Pods were removed with their namespace. The experiment verifies local sequencing and failure policy, not distributed migration safety or durable storage.

Sources

Kubernetes: Init Containers, regular init ordering, retries, resource accounting and repeatable setup, checked 2026-09-13.
Kubernetes: Pod Lifecycle, Pod phases, container states and restart policies, checked 2026-09-13.

Workloads · XB Field Notes

ConfigMaps separate configuration from the image

Give configuration a reviewable identity

When it breaks

Check it quickly

Sources

Kubernetes Deployments describe desired state

Spec beats imperative steps

A controller converges the gap

Updates happen through new Revision

When it breaks

Check it quickly

Further reading

Sources

A DaemonSet puts one Pod on every eligible node

One per node, by design

Coverage over count

Rides the node lifecycle

When it breaks

Check it quickly

Sources

Regular init containers establish startup prerequisites

Sequence is local to one Pod

When it breaks

Check it quickly

Sources