Probes · XB Field Notes

Startup probes delay other probes until initialization succeeds

Sun, 22 Aug 2027 00:00:00 +0000

A startup probe gives a container a separate initialization contract. While that configured probe has not succeeded, Kubernetes does not execute the container’s liveness or readiness probes. This is stronger and more specific than ignoring some liveness failures during a grace period: those other checks have not begun. Once startup succeeds, ordinary ongoing health and readiness behavior takes over.

Give each stage an observable condition

The application process starts before its startup probe can succeed. A Running container may therefore still have started reported as false in container status and may not make its Pod Ready. Running describes execution; successful startup describes completion of the configured startup check. An HTTP listener that opens early can expose progress without yet claiming that initialization is complete.

Choose a startup condition that represents necessary local preparation, such as loading an index or validating required initial state. A port opening is sufficient only if that is the actual readiness prerequisite you intend to observe. If the listener appears before expensive setup finishes, checking only that listener can release liveness protection too early and recreate the original restart loop.

After a successful startup check, kubelet stops that startup probe for the current container instance. It runs liveness and readiness according to their separate configuration. Startup success is not a permanent health certificate: the application can later become unready or require a restart. A new container instance must complete its own startup check again.

When it breaks

A startup probe that never succeeds eventually reaches its failure threshold. Kubelet then terminates the container and applies its restart policy. Startup protection is therefore not permission to initialize forever. Set period, timeout and failure threshold from measured initialization behavior, allowing for the environment’s realistic resource contention and expected recovery time.

The product of failureThreshold and periodSeconds is useful when planning the startup allowance, but do not present it as an exact stopwatch for every observed restart. Initial delay, probe execution and scheduling, termination grace and restart backoff also affect the timeline. Record the actual attempts and state changes when investigating a slow deployment.

Do not make successful readiness a prerequisite for liveness unless that dependency is explicitly represented by startup configuration. Readiness and liveness otherwise answer separate ongoing questions. Equally, passing startup does not force readiness to succeed; a later readiness check can still reject traffic eligibility for a different reason, while the process remains alive.

Changing checks requires application support. An endpoint that always returns success cannot verify a missing initialization step merely because it is named startup. Keep its state transition deliberate, expose enough diagnostic information to explain waiting, and avoid placing credentials or sensitive initialization details in responses readable by unintended clients. The fixture below uses only counters and synthetic state.

Check it quickly

Use a disposable Kubernetes cluster, kubectl and python:3.11-slim-bookworm available to its nodes. Save the script as startup-gate.py and run python3 startup-gate.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. Namespace creation and deletion permissions are required. The recorded server was Kubernetes 1.34.11. The startup probe allows sixty failed attempts at one-second periods for this short controlled exercise.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-startup-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 server = '''from http.server import BaseHTTPRequestHandler,HTTPServer
from pathlib import Path
import json
counts={'startup':0,'live':0,'ready':0}
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 key=self.path.removeprefix('/'); code=200
 if key in counts:
 counts[key]+=1
 if key=='startup' and not Path('/tmp/start').exists(): code=503
 body=json.dumps(counts).encode(); self.send_response(code)
 self.send_header('Content-Length',str(len(body))); self.end_headers(); self.wfile.write(body)
 def log_message(self,*args): pass
HTTPServer(('0.0.0.0',8080),Handler).serve_forever()
'''
 podspec = spec(server)
 c = podspec['containers'][0]
 c['startupProbe'] = {'httpGet': {'path': '/startup', 'port': 8080}, 'periodSeconds': 1, 'failureThreshold': 60}
 for field, path in [('livenessProbe', '/live'), ('readinessProbe', '/ready')]:
 c[field] = {'httpGet': {'path': path, 'port': 8080}, 'periodSeconds': 1, 'failureThreshold': 2}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'slow'}, 'spec': podspec})
 wait_for(lambda: obj(namespace, 'pod', 'slow'), lambda p:
 any('running' in c['state'] for c in p.get('status', {}).get('containerStatuses', [])))
 read = 'from urllib.request import urlopen; print(urlopen("http://127.0.0.1:8080/stats",timeout=3).read().decode())'
 def counts():
 result = k('exec', '-n', namespace, 'slow', '--', 'python3', '-c', read, expected=None)
 return json.loads(result.stdout) if result.returncode == 0 else None
 before = wait_for(counts, lambda d: d and d['startup'] >= 3)
 assert before['live'] == before['ready'] == 0
 starting = obj(namespace, 'pod', 'slow')
 assert not pod_ready(starting)
 assert starting['status']['containerStatuses'][0]['started'] is False
 assert starting['status']['containerStatuses'][0]['restartCount'] == 0
 k('exec', '-n', namespace, 'slow', '--', 'python3', '-c',
 'from pathlib import Path; Path("/tmp/start").touch()')
 ready = wait_for(lambda: obj(namespace, 'pod', 'slow'), pod_ready)
 assert ready['status']['containerStatuses'][0]['started'] is True
 enabled = wait_for(counts, lambda d: d and d['live'] > 0 and d['ready'] > 0)
 time.sleep(3)
 after = counts()
 assert after['startup'] == enabled['startup']
 assert after['live'] > enabled['live'] and after['ready'] > enabled['ready']
 assert obj(namespace, 'pod', 'slow')['status']['containerStatuses'][0]['restartCount'] == 0
 print(json.dumps({'before_startup_success': before, 'after_startup_success': after,
 'other_probes_suppressed_before_startup': True,
 'startup_probe_stops_after_success': True, 'container_restarts': 0}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The HTTP server counts requests separately for startup, live and ready paths. Its statistics path is observational and does not increment those counters. Startup initially returns 503 until a local release file appears. After three observed startup requests, both other counters are zero, the container has not restarted, and its started field is false.

Creating the release file makes the next startup request succeed. The Pod becomes Ready, started becomes true, and the live and ready counters begin increasing. After another three seconds, the startup counter stays fixed while both ongoing counters increase. The recorded final counts were four, four and five; exact ongoing counts can vary with timing.

All assertions passed without a restart, and the namespace was removed. This verifies suppression and transition using actual HTTP calls. It does not measure a production boot-time percentile or exercise startup-budget exhaustion. Use those additional measurements when choosing real thresholds, while keeping startup completion distinct from continuing service health.

Sources

Kubernetes: Configure Probes, startup failure budgets and probe configuration, checked 2026-09-13.
Kubernetes: Pod Lifecycle, suppression of liveness and readiness until startup succeeds, checked 2026-09-13.

Readiness changes routing eligibility; liveness restarts a container

Wed, 13 Jan 2027 00:00:00 +0000

Readiness and liveness answer different operational questions. Readiness asks whether this container should contribute to a Pod that accepts ordinary Service traffic now. Liveness asks whether restarting this container is the configured recovery action. Both can use HTTP, but sharing a protocol does not make their meanings interchangeable. Define the action you want before choosing the condition that triggers it.

Observe the action as well as the response

A readiness failure makes the container unready and affects Pod readiness. For a normal selector-based Service, endpoint reconciliation then changes which backends are eligible for ordinary routing. The Pod can remain Running while it is not Ready, and the application process may continue executing. Restart count and Pod UID answer different questions from the Ready condition.

Liveness failures reaching the configured threshold cause kubelet to terminate the container and apply its restart behavior. In a usual Always-policy application Pod this produces a replacement container inside the same Pod. It does not inherently create a new Pod object or deploy another application version. Compare container restart count and termination state alongside the Pod identity.

Use inexpensive checks with meaningful success criteria. A TCP connection proves that a listener accepts connections, not that an application can complete useful work. An HTTP endpoint can answer a richer question, but can also create load if it recursively checks expensive dependencies. Keep the probe’s own execution cost within the workload’s capacity and timeout budget.

When it breaks

Treating a shared database outage as a liveness failure can repeatedly restart every healthy application process without repairing the database. Adding that dependency to readiness also needs judgment: withdrawing every replica may remove useful fallback responses or overload the remaining replicas during a partial failure. Define readiness from the service contract, including any deliberately supported degraded behavior.

Readiness is not a network access control. A Pod address can remain reachable when the Pod has no ready Service target. Existing connections, other routing layers, and Service options such as publishNotReadyAddresses also require their own analysis. Do not describe a failed readiness probe as an instantaneous guarantee that no request can ever reach the process.

Thresholds and timing control sensitivity. A short timeout can turn CPU contention into apparent failure; several consecutive failures can avoid reacting to one brief delay. Startup protection is separate: a configured startup probe suppresses readiness and liveness checks until it succeeds. These settings are sampling and recovery policy, not a proof of zero lost requests during termination.

Check it quickly

Use a disposable Kubernetes cluster with kubectl access and python:3.11-slim-bookworm available on its nodes. Save the code as probe-actions.py, then run python3 probe-actions.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. The account needs namespace creation and deletion permissions. The recorded server was Kubernetes 1.34.11. The script creates one HTTP Pod and an ordinary Service with default endpoint readiness behavior.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-probes-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 server = '''from http.server import BaseHTTPRequestHandler,HTTPServer
from pathlib import Path
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 flag={'/live':'/tmp/live-fail','/ready':'/tmp/ready-fail'}.get(self.path)
 code=503 if flag and Path(flag).exists() else 200
 body=b'worker'; self.send_response(code); self.send_header('Content-Length',str(len(body)))
 self.end_headers(); self.wfile.write(body)
 def log_message(self,*args): pass
HTTPServer(('0.0.0.0',8080),Handler).serve_forever()
'''
 podspec = spec(server)
 c = podspec['containers'][0]
 c['readinessProbe'] = {'httpGet': {'path': '/ready', 'port': 8080}, 'periodSeconds': 1, 'failureThreshold': 2}
 c['livenessProbe'] = {'httpGet': {'path': '/live', 'port': 8080}, 'periodSeconds': 1, 'failureThreshold': 3}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'worker', 'labels': {'app': 'worker'}}, 'spec': podspec})
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'web'},
 'spec': {'selector': {'app': 'worker'}, 'ports': [{'port': 80, 'targetPort': 8080}]}})
 def targets():
 return {e['targetRef']['uid'] for s in obj(namespace, 'endpointslices', selector='kubernetes.io/service-name=web')['items']
 for e in s.get('endpoints', []) if e.get('conditions', {}).get('ready') and e.get('targetRef')}
 initial = wait_for(lambda: obj(namespace, 'pod', 'worker'), pod_ready)
 uid = initial['metadata']['uid']
 wait_for(targets, lambda value: value == {uid})
 def change(path, create):
 method = 'touch' if create else 'unlink'
 k('exec', '-n', namespace, 'worker', '--', 'python3', '-c',
 'from pathlib import Path; Path('+repr(path)+').'+method+'()')
 change('/tmp/ready-fail', True)
 wait_for(lambda: obj(namespace, 'pod', 'worker'), lambda p: not pod_ready(p))
 wait_for(targets, lambda value: not value)
 time.sleep(4)
 unready = obj(namespace, 'pod', 'worker')
 assert unready['metadata']['uid'] == uid and unready['status']['containerStatuses'][0]['restartCount'] == 0
 address = 'http://' + unready['status']['podIP'] + ':8080/api'
 probe = 'from urllib.request import urlopen; print(urlopen('+repr(address)+',timeout=3).read().decode())'
 assert k('exec', '-n', namespace, 'worker', '--', 'python3', '-c', probe).stdout.strip() == 'worker'
 change('/tmp/ready-fail', False)
 wait_for(targets, lambda value: value == {uid})
 change('/tmp/live-fail', True)
 restarted = wait_for(lambda: obj(namespace, 'pod', 'worker'), lambda p:
 pod_ready(p) and p['status']['containerStatuses'][0]['restartCount'] >= 1)
 assert restarted['metadata']['uid'] == uid
 assert restarted['status']['containerStatuses'][0]['restartCount'] == 1
 wait_for(targets, lambda value: value == {uid})
 print(json.dumps({'readiness_removed_ready_target_without_restart': True,
 'unready_pod_direct_http_still_succeeds': True,
 'readiness_recovery_restores_target': True,
 'liveness_container_restarts': 1, 'same_pod_uid': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The initial Pod is Ready and its UID appears in the Service’s ready EndpointSlice targets. Creating a local flag makes only the readiness endpoint return 503. After the failure threshold and reconciliation, the ready-target set is empty. Four seconds later the Pod UID is unchanged and the container restart count remains zero.

A direct HTTP request to that unready Pod’s address still returns worker. Removing the readiness flag restores its ready target. The script then creates a different flag that makes the liveness endpoint fail. Kubelet restarts the container once, its fresh writable layer removes the flag, and the ready target returns. The Pod UID remains the same throughout.

All assertions passed and the namespace was removed. This demonstrates distinct controller actions and direct reachability, not graceful draining of existing connections or production availability. For a real rollout, verify endpoint propagation, application shutdown, outstanding work and client behavior together; a green probe alone cannot establish the whole request lifecycle.

Sources

Kubernetes: Configure Probes, failure thresholds and distinct readiness and liveness actions, checked 2026-09-13.
Kubernetes: Pod Lifecycle, container restarts and Pod readiness, checked 2026-09-13.
Kubernetes: EndpointSlices, backend readiness conditions and discovery behavior, checked 2026-09-13.