Health · XB Field Notes

Consul keeps a live map of healthy services

Thu, 06 Apr 2028 00:00:00 +0000

A service registry is only as useful as its honesty. Consul keeps a catalog of registered instances and attaches health checks to them, so a caller asking where a healthy instance of a service lives gets an answer that changes as reality changes. Registration is trivial; the check is the whole product, because a registry that lists dead instances is worse than a static list — it actively routes traffic into a hole.

The freshness of that answer is a function of two choices nobody revisits after the first week: how often the check runs, and what it actually tests. A check that runs every 30 seconds is blind for 30 seconds, and a check that only proves the process exists is blind forever.

Registration, and two levels of checks

An instance registers a name, an address, a port and one or more checks, through a config file or the registration API. Checks start in a critical state by default, deliberately, so a service cannot enter the pool before its health has been verified at least once — an initial passing status is something you opt into explicitly.

Two levels of check sit on the same node. A node-level check such as serfHealth carries an empty service name and describes the machine; a service-level check such as service:web-1 is bound to one registration. They fail independently, but the healthy-only filter requires every check on the node to pass, so a critical node-level check removes all of that machine’s services at once while a critical service check removes exactly one instance.

What a caller actually receives

The catalog answers two different questions. Unfiltered, /v1/health/service/web returns every registration with each check’s status; with passing=true it returns only the instances where all checks pass. DNS and SRV always answer the second way, so a resolver never sees a name it cannot use, and the passing set is the load-balancing pool.

The catalog is also eventually consistent, and the mechanism is anti-entropy. Registration is immediate — the agent tells the leader — but repair is periodic: each agent reconciles its local state with the catalog on an interval of one minute for clusters up to 128 nodes, growing with cluster size, and each agent staggers its start so they do not all sync at once. A sync that fails is logged and retried next period, so a wrong catalog entry can survive quietly for up to one interval. The read path has its own trade: by default the agent forwards catalog reads to the leader using a leader lease that assumes leadership is briefly stable, which is fast and can occasionally return a stale result, while the consistent mode makes the leader verify with a quorum first and stale lets any server answer without checking.

When it breaks

The first failure is the interval itself. A check is only as fresh as its period, so a service that dies right after a passing probe stays in rotation until the next probe — 30 seconds for a 30-second check, 60 for a 60-second one. A TTL check is worse than a slow interval because it is passive: the instance reports in on its own schedule, so a dead instance with a 120-second TTL keeps its passing status for the full 120 seconds.

The second failure is a check that answers the wrong question. A script that runs pgrep against the process reports passing for the entire duration of a deadlock — the process exists, the port is bound, and every request routed to it hangs. The catalog cannot tell that instance from a healthy one, so the failure is silent and total for every caller in the pool. A check that exercises a dependency, even imperfectly, converges on the truth within one interval; a liveness-only check never does. The direction of the error is what makes it expensive: a check that fails when it should pass removes a healthy instance and costs capacity, while a check that passes when it should fail keeps feeding traffic to something that cannot serve it, and the caller only finds out through its own error rate.

Check it quickly

Compare the filtered and unfiltered views, and see what DNS hands out:

consul catalog services
consul health service web
curl -s 'http://127.0.0.1:8500/v1/health/service/web' | jq -r '.[] | "\(.Service.ID) \(.Checks[] | select(.Status!="passing") | .CheckID + "=" + .Status)"'
curl -s 'http://127.0.0.1:8500/v1/health/service/web?passing=true' | jq -r '.[] | "\(.Service.ID) \(.Node.Address):\(.Service.Port)"'
dig +short SRV web.service.consul

An instance that shows up unfiltered with a failing CheckID must be absent from the passing=true list and from the SRV answer; if it is still there, the check is not being evaluated at all. A node whose node-level check is critical drops out of both lists entirely, taking every service on that machine with it.

Sources

Service discovery, the catalog, registration and the role of health checks, checked 2026-09-16.
Health checks, check types, intervals and the default critical initial state, checked 2026-09-16.
Health HTTP API, /v1/health/service and the passing filter, checked 2026-09-16.
Anti-entropy, the periodic sync interval and its failure behaviour, checked 2026-09-16.

Liveness and readiness tell Kubernetes how the Pod feels

Wed, 14 Jul 2027 00:00:00 +0000

A slow start is not a dead process: readiness shields traffic while liveness only cares about keeping the process alive. Teams wire a single HTTP endpoint into both probes, and then every slow dependency becomes a restart loop — the app needs time, liveness refuses to grant it. Two probes exist because “not ready” and “dead” demand opposite responses: one asks for time, the other for a restart.

The mental model that prevents the confusion is a hospital triage. Readiness is the waiting-room nurse deciding who sees a doctor right now; liveness is the crash cart, rolled out only when the patient codes. You would never defibrillate someone for sitting in the waiting room. Keep that image and the probe configuration almost writes itself.

Readiness gates traffic

A readiness probe failure removes the Pod from the Service’s endpoints but leaves it running, so the process can finish warming up or recover on its own. Traffic flows again as soon as the probe passes. This makes readiness the safe probe to fail: caches can warm, migrations can finish, downstream dependencies can come back, all without losing the Pod’s in-memory state or paying a restart penalty.

Design the readiness endpoint around genuine servability. Check what makes this Pod able to answer correctly right now: migrations applied, required caches populated, critical downstream reachable. A probe that only checks “the HTTP server responds” admits traffic to Pods that will fail every request. But resist stuffing every dependency into it: if a non-critical recommendation service is down, failing readiness sheds real traffic over a decoration.

Liveness guards the process

A failing liveness probe triggers a restart, a last resort that ends a hopeless wedge. Use it for what it is a liveness issue—not for containers that are merely slow to become ready. The legitimate targets are narrow: deadlocks where no request will ever succeed again, runaway goroutine or thread leaks that only a fresh process clears, corrupted in-memory state that no retry can heal. In each case the defining property is irreversibility — without a restart, recovery is impossible, not merely slow.

That bar means many services should ship a minimal liveness probe or none at all. A liveness check against an external database turns every database blip into a fleet-wide restart storm. Prefer a cheap local check — a watchdog timestamp refreshed by the main loop — that detects “this process is wedged” without consulting anything outside the process. Kubernetes already restarts crashed containers; liveness exists only for the living dead.

Give the probes grace

Set the initialDelaySeconds high enough that a cold start is not counted as a crash, and keep timeoutSeconds consistent with real work. Separate the two probes so a slow pod is protected by readiness, not mistakenly recycled by liveness. Better still, add a startup probe for slow-booting containers: while it runs, liveness stays disabled, so version upgrades that warm large caches get minutes instead of seconds without weakening steady-state liveness at all.

Tune thresholds from observed behavior, not round numbers copied between services. periodSeconds times failureThreshold is the real reaction time — 10 seconds times 3 means 30 seconds of badness before action. Measure cold-start p99, dependency recovery time, and worst-case pauses, then set each number to react faster than the alert but slower than the noise.

When it breaks

The signature failure is the restart loop during dependency outages: the database slows, readiness would have sufficed, but liveness probes share the same failing endpoint and the whole fleet restarts in waves. Each restart wipes warm caches, making the next wave slower to recover, until the system oscillates indefinitely. Diagnose by correlating restart timestamps with the dependency incident; the fix is splitting the endpoints — readiness checks the dependency, liveness checks only the process — so a sick downstream sheds traffic instead of killing Pods.

The mirror failure is the zombie Pod: liveness passes because the HTTP server responds, but every request hangs behind a deadlocked worker pool. Traffic keeps flowing to a Pod that answers nothing. The fix is a liveness endpoint that exercises the real path — acquire a worker slot with a short timeout, verify the last successful request timestamp is fresh — so wedged state fails the probe while mere slowness does not.

Check it quickly

Inspect probe wiring and recent kills:

kubectl get pod web-0 -o jsonpath='{.spec.containers[*].livenessProbe}{"\n"}{.spec.containers[*].readinessProbe}{"\n"}'
kubectl get events --field-selector reason=Unhealthy --sort-by=.lastTimestamp | tail -n 10
kubectl describe pod web-0 | grep -A3 -i 'liveness\|readiness'

Shared endpoints across both probes, or Unhealthy events coinciding with downstream incidents, mark the split that still needs doing.

Sources

Kubernetes probes, Official docs on liveness and readiness, checked 2026-06-01.
Configure liveness and readiness probes, task guide for probe configuration, checked 2026-09-05.

Probes tell the platform when to act

Fri, 18 Jun 2027 00:00:00 +0000

A probe result is evidence consumed by a control loop. Kubernetes does not infer your application’s recovery strategy from a route named healthz. The probe type, result, configured thresholds, and container lifecycle determine what happens next. Treating every failure as “restart the Pod” erases distinctions that directly affect availability.

Three questions, different actions

A startup probe gives initialization its own detection policy. When configured, it prevents liveness and readiness probes from running until it succeeds. It stops checking after success for that container lifetime; a newly started container needs to pass startup again.

Liveness is intended to identify conditions for which restarting the affected container can help. After the failure threshold, the kubelet kills that container and applies its restart policy. This is not a promise to replace the whole Pod, change its identity, or repair an unavailable external database.

Readiness controls whether the container is considered ready. A failed readiness check leaves the container running and continues checking for recovery. It affects ordinary readiness-aware Service routing, but is not a firewall and does not cancel requests already using a connection.

Thresholds operate on consecutive observations

A failureThreshold of three is not a three-failures-ever counter. A success breaks the run. Readiness can require multiple consecutive successes to recover; successThreshold must be one for startup and liveness. Know both directions of a transition before interpreting a short trace.

Timeouts, initial delays, periods, and termination grace also matter. Threshold multiplied by period is useful for planning, not an exact end-to-end failover stopwatch. Scheduling, probe execution, and routing propagation intervene. Readiness may be checked more frequently than its configured period while a container is not ready.

The experiment uses four startup failures, three liveness failures, and two readiness failures, with two readiness successes required for recovery. These settings make counters observable; they are not production recommendations.

Match the mechanism to the evidence

An HTTP check uses the documented 200–399 success range. A 200 response containing an error message is not a failed HTTP probe merely because a human dislikes its body. Verify redirects and authentication on the actual probe route rather than assuming the same behavior as an interactive browser.

A TCP check shows that a connection can be opened, not that a query or transaction succeeds. An exec check needs a command present in the container and a meaningful exit status; frequent process creation also costs resources. Select the cheapest mechanism that answers the required question.

Readiness is not a prerequisite for liveness. Without startup gating, a long initialization period can repeatedly fail liveness before the application has time to become ready. Conversely, an excessively generous startup allowance can postpone detection of a genuinely broken launch.

When it breaks

Using external dependency availability as liveness can restart healthy processes throughout a shared outage. An always-green handler on an unrelated thread can miss a deadlocked business worker. Restarts are not automatically cheap: they can lose in-flight work and repeat expensive initialization.

A readiness gate covering every optional dependency can withdraw useful capacity. Rapid state flapping can transfer load between replicas and worsen saturation. Check false positives and recovery behavior, not just whether a deliberately broken endpoint turns red.

Compare probe outcomes with container restart counts, events, readiness transitions, and actual client traffic. Confirm the deployed configuration and routing consumer. A successful local policy model cannot establish that a real kubelet, network path, or load balancer applies it correctly.

Check it quickly

Save this block as probe_policy_check.py and run python3 probe_policy_check.py with Python 3.10 or newer. It ran on Python 3.14.4. This deterministic model consumes binary observations; it does not deploy a Pod, call Kubernetes, schedule probes, or restart an operating-system process.

class ProbePolicy:
 # (consecutive successes, consecutive failures), chosen for this experiment.
 thresholds = {"startup": (1, 4), "liveness": (1, 3), "readiness": (2, 2)}

 def __init__(self):
 self.restart_requests = 0
 self.start_container()

 def start_container(self):
 self.started = self.ready = self.restart_requested = False
 self.successes = dict.fromkeys(self.thresholds, 0)
 self.failures = dict.fromkeys(self.thresholds, 0)

 def sample(self, kind, passed):
 if self.restart_requested:
 return "awaiting-restart"
 if kind != "startup" and not self.started:
 return "suppressed"
 if kind == "startup" and self.started:
 return "disabled"
 success_limit, failure_limit = self.thresholds[kind]
 self.successes[kind] = self.successes[kind] + 1 if passed else 0
 self.failures[kind] = 0 if passed else self.failures[kind] + 1
 if passed and self.successes[kind] >= success_limit:
 if kind == "startup":
 self.started = True
 elif kind == "readiness":
 self.ready = True
 if not passed and self.failures[kind] >= failure_limit:
 self.ready = False
 if kind != "readiness":
 self.restart_requested = True
 self.restart_requests += 1
 return "restart-requested" # A model output, not a kubelet or OS action.
 return "sampled"


policy = ProbePolicy()
for _ in range(3):
 assert policy.sample("startup", False) == "sampled"
 assert policy.sample("liveness", False) == "suppressed"
 assert policy.sample("readiness", True) == "suppressed"
assert policy.restart_requests == 0 and not policy.ready
assert policy.failures["liveness"] == 0 and policy.successes["readiness"] == 0
policy.sample("startup", True)
assert policy.started and not policy.ready
assert policy.sample("startup", False) == "disabled"
print("three startup failures do not trigger liveness; startup success enables the other probes")

policy.sample("readiness", True)
assert not policy.ready
policy.sample("readiness", True)
assert policy.ready
policy.sample("readiness", False)
assert policy.ready
policy.sample("readiness", True) # Breaks the consecutive-failure run.
policy.sample("readiness", False)
assert policy.ready
policy.sample("readiness", False)
assert not policy.ready and policy.restart_requests == 0
for result in (True, False, True):
 policy.sample("readiness", result)
assert not policy.ready
policy.sample("readiness", True)
assert policy.ready
print("two readiness failures remove readiness without restart; two consecutive successes restore it")

for result in (False, False, True, False, False):
 policy.sample("liveness", result)
assert policy.restart_requests == 0
assert policy.sample("liveness", False) == "restart-requested"
assert policy.restart_requests == 1 and not policy.ready
assert policy.sample("liveness", False) == "awaiting-restart"
assert policy.restart_requests == 1
policy.start_container() # Explicitly model the beginning of another container lifetime.
assert not policy.started and not policy.ready
for _ in range(4):
 policy.sample("startup", False)
assert policy.restart_requests == 2
print("three consecutive liveness failures request a restart; a fresh lifetime runs startup checks again")
print("four startup failures request another restart; no real container, clock, or traffic route was changed")

Three startup failures remain below the fixture’s threshold, and attempted liveness/readiness observations are suppressed. Startup success enables those checks without automatically declaring readiness.

Two consecutive readiness failures remove readiness without requesting a restart; recovery needs two consecutive successes. Intervening opposite outcomes reset the relevant streak. Three consecutive liveness failures produce one restart request, not repeated requests while awaiting restart.

An explicitly modeled new container lifetime clears startup state. Four startup failures then request another restart. Unknown results, transport error classification, real timing, restart policy execution, termination grace, and endpoint propagation remain untested; validate those on the intended cluster before relying on the full operational behavior.

Sources

Kubernetes: Liveness, readiness, and startup probes, probe actions, mechanisms, thresholds, and startup gating, checked 2026-09-09.
Kubernetes: Configure probes, executable probe configuration and observed container restarts, checked 2026-09-09.
Kubernetes: Pod lifecycle, container restart policy and termination versus traffic lifecycle, checked 2026-09-09.

Health · XB Field Notes

Consul keeps a live map of healthy services

Registration, and two levels of checks

What a caller actually receives

When it breaks

Check it quickly

Sources

Liveness and readiness tell Kubernetes how the Pod feels

Readiness gates traffic

Liveness guards the process

Give the probes grace

When it breaks

Check it quickly

Sources

Probes tell the platform when to act

Three questions, different actions

Thresholds operate on consecutive observations

Match the mechanism to the evidence

When it breaks

Check it quickly

Further reading

Sources