A health endpoint is an input to an operational decision, not a universal certificate that a service works. A small endpoint returning 200 can be useful when its consumer needs exactly that limited evidence. Problems begin when operators interpret “this handler answered” as “every customer request will succeed.”

Name the question and its consumer

Liveness asks whether the application has reached a state for which restarting is appropriate. Readiness asks whether this instance should currently receive the traffic covered by its contract. A diagnostic dashboard can ask broader questions without giving a scheduler authority to restart anything.

The consumer matters. Returning 503 does not remove an instance from a router that never checks the endpoint. Likewise, changing readiness does not cancel existing requests. Document who polls, what counts as failure, which traffic changes, and how quickly the decision propagates.

Required does not mean everything

A strict dependency can legitimately gate readiness: without required input, this instance might only return errors. But an optional annotation service need not remove a report endpoint whose product contract permits reports without annotations. Decide from the supported response, not from the length of the dependency inventory.

A shared dependency outage can make every replica fail the same readiness test. Removing all replicas does not repair that dependency and can also remove routes that remain useful. Evaluate the service boundary and degraded-response contract before linking every backend check to a single traffic gate.

Keep remote dependency failures out of liveness unless there is evidence that restarting this process remedies the condition. Restarting every instance during a database outage usually adds initialization work without restoring the database.

Keep the signal cheap and current

Frequent liveness and readiness checks should not repeatedly perform expensive business work. Repeatedly generating reports, scanning tables, or allocating scarce connections can overload a dependency and distort the signal. Bound check duration and concurrency separately from the normal request workload.

An endpoint may expose recently observed dependency state instead of contacting everything on every poll. That changes the contract: observation age and expiration need an explicit policy. An indefinitely cached green flag can hide an outage, while an overaggressive expiry can unnecessarily drain healthy capacity.

Return a small, unambiguous status and avoid exposing credentials, topology, or exception traces. Cache-Control: no-store tells compliant HTTP caches not to store the response; it does not provide authentication or establish that the health observation itself is fresh.

When it breaks

A shallow HTTP handler can answer while a business worker is deadlocked. Conversely, a deep liveness check can turn a transient external problem into a restart storm. Neither the endpoint name nor a green status proves that the selected question matches the desired recovery action.

Wrong paths, authentication redirects, stale intermediaries, and inaccessible probe networks can produce misleading results. Test the actual consumer route, not just a request from inside the process. Excessively tight timeouts and thresholds can also make healthy-but-busy instances flap out of service.

Google SRE distinguishes internal instrumentation from black-box checks of user-visible behavior. Use both appropriately: an external report check can detect a broken customer path that an internal readiness flag misses. It still should not automatically become a restart trigger.

Check it quickly

Save this block as health_check.py and run python3 health_check.py with Python 3.10 or newer. It ran on Python 3.14.4 using only the standard library. A real HTTP server binds an assigned loopback port. Dependencies are injected flags, not live databases; the declared report contract allows omission of an optional annotation.

from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Lock, Thread


state = {"initialized": False, "required_ok": True, "optional_ok": True,
         "draining": False, "reports": 0}
lock = Lock()


def change(**values):
    with lock:
        state.update(values)


class Handler(BaseHTTPRequestHandler):
    def log_message(self, *args):
        pass

    def do_GET(self):
        with lock:
            ready = state["initialized"] and state["required_ok"] and not state["draining"]
            if self.path == "/livez":
                status, body = 200, "live"  # Only proves this HTTP handler can answer.
            elif self.path == "/readyz":
                status, body = (200, "ready") if ready else (503, "not-ready")
            elif self.path == "/report":
                if ready:
                    state["reports"] += 1
                    status = 200
                    body = "report+annotation" if state["optional_ok"] else "report"
                else:
                    status, body = 503, "not-ready"
            else:
                status, body = 404, "not-found"
        payload = body.encode("ascii")
        self.send_response(status)
        self.send_header("Content-Type", "text/plain; charset=utf-8")
        self.send_header("Cache-Control", "no-store")
        self.send_header("Content-Length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)


server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
server.daemon_threads = False
thread = Thread(target=server.serve_forever, kwargs={"poll_interval": 0.01})
thread.start()


def get(route):
    connection = HTTPConnection("127.0.0.1", server.server_port, timeout=2)
    try:
        connection.request("GET", route)
        response = connection.getresponse()
        body = response.read().decode("ascii")
        assert response.getheader("Cache-Control") == "no-store"
        return response.status, body
    finally:
        connection.close()


try:
    assert get("/livez") == (200, "live")
    assert get("/readyz") == (503, "not-ready")
    assert get("/report") == (503, "not-ready")
    change(initialized=True)
    assert get("/readyz") == (200, "ready")
    assert get("/report") == (200, "report+annotation")
    print("warm-up: live 200, ready 503; initialized: ready 200 and a report is served")

    change(optional_ok=False)
    assert get("/readyz") == (200, "ready")
    assert get("/report") == (200, "report")
    change(required_ok=False)
    assert get("/livez") == (200, "live")
    assert get("/readyz") == (503, "not-ready")
    assert get("/report") == (503, "not-ready")
    print("optional annotation loss keeps readiness; required input loss fails readiness, not liveness")

    for _ in range(20):
        assert get("/livez")[0] == 200
        assert get("/readyz")[0] == 503
    with lock:
        assert state["reports"] == 2  # Probes did not execute the business report.
    change(required_ok=True)
    assert get("/readyz")[0] == 200
    change(draining=True)
    assert get("/readyz")[0] == 503 and get("/report")[0] == 503
    assert get("/livez")[0] == 200 and get("/unknown")[0] == 404
    print("40 probes add no reports; recovery restores readiness, and drain removes it without declaring death")
finally:
    server.shutdown()  # serve_forever runs in the other thread.
    thread.join(timeout=2)
    server.server_close()
    assert not thread.is_alive()

During warm-up, liveness returns 200 while readiness and reports return 503. Initialization enables reports. Losing annotations preserves readiness and returns the supported basic report; losing required input fails readiness without changing liveness.

Forty subsequent probes create no additional reports. Restoring required input restores readiness; entering drain removes readiness and refuses reports while liveness still answers. Each reply includes no-store, and an unknown route returns 404.

This verifies HTTP responses and the declared decision policy, not dependency detection, observation freshness, real deadlocks, router propagation, or production isolation. Python’s http.server is a local test fixture, not a recommended production health server.

Further reading

Sources