A health endpoint is a contract with a consumer that will take action. Returning 200 because a handler ran does not prove that an instance can serve useful traffic. Before writing the endpoint, identify the decision it supports: allow initialization, admit requests, or restart a process that cannot recover. Different questions need different evidence.

Separate process recovery from traffic admission

Liveness should represent a local condition for which restarting is a useful remedy. A failed shared database is usually not repaired by restarting every caller. Readiness answers whether this instance should receive its intended traffic now. Include dependencies essential to that traffic, but do not make an optional recommendation service remove an otherwise useful checkout instance from rotation.

Startup protects slow initialization from premature recovery actions. With a Kubernetes startup probe configured, liveness and readiness probes wait until startup succeeds. Readiness failure affects traffic admission; repeated liveness or startup failures can cause restarts according to the probe configuration and restart policy. A single 503 response does not itself execute either action.

Document the status contract for the actual consumer. Kubernetes HTTP probes accept statuses from 200 through 399, so a login redirect can accidentally appear healthy. This example deliberately returns only 200 or 503 for its three GET probe routes. It does not depend on parsing a reassuring response body or following a redirect.

Bound the work the probe introduces

Use a cheap, read-only operation that represents the essential dependency. Acquiring a connection or making a small dependency request is evidence about that path, not proof of every business transaction. Specify a deadline shorter than the caller’s timeout, pass cancellation into downstream work, release responses, and cancel timers. Returning early while abandoned operations continue accumulating is not a bounded probe.

Estimate the load from instances, independent observers, and polling frequency. Avoid a deep dependency fan-out on every health request. If an existing background checker supplies a result, its age and failure meaning must be explicit; an indefinitely cached green value is not current health. Probe caching should not let an intermediary keep a failed instance looking ready.

When it breaks

An always-green endpoint remains successful during a broken database connection, exhausted worker pool, or incomplete initialization. The opposite mistake makes a shared-dependency outage trigger fleet-wide restart loops. Readiness checks can themselves consume the scarce connections needed for recovery, while overly aggressive thresholds remove too much capacity during a short pause.

Test startup, a negative dependency response, a stalled dependency, deliberate draining, and recovery. Liveness must remain independent of the shared dependency in this design. Draining should stop new admission before in-flight work is completed; this endpoint alone does not wait for load-balancer propagation or drain existing requests. Keep public responses small and avoid exposing dependency addresses, credentials, or stack traces. Detailed causes belong in protected operational telemetry.

Check it quickly

Save this program as health_probe.go and run go run health_probe.go. Two loopback servers model an application and its one required dependency. The dependency’s healthy contract is 204. A 100-millisecond child context bounds the dependency request; the outer test client allows two seconds. Flags model startup and draining without modifying any real service.

package main

import (
	"context"
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"
	"runtime"
	"sync/atomic"
	"time"
)

func must(err error) {
	if err != nil {
		panic(err)
	}
}

func check(ok bool, message string) {
	if !ok {
		panic(message)
	}
}

func main() {
	var mode, dependencyCalls atomic.Int64
	var started, draining atomic.Bool
	canceled := make(chan struct{}, 1)
	dependency := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		dependencyCalls.Add(1)
		switch mode.Load() {
		case 1:
			w.WriteHeader(503)
		case 2:
			<-r.Context().Done()
			canceled <- struct{}{}
		default:
			w.WriteHeader(204)
		}
	}))
	defer dependency.Close()
	transport := &http.Transport{}
	defer transport.CloseIdleConnections()
	client := &http.Client{Transport: transport, Timeout: 2 * time.Second,
		CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse },
	}
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "text/plain; charset=utf-8")
		w.Header().Set("Cache-Control", "no-store")
		if r.Method != "GET" {
			w.Header().Set("Allow", "GET")
			w.WriteHeader(405)
			return
		}
		ready := false
		switch r.URL.Path {
		case "/livez":
			ready = true // No shared-dependency restart policy.
		case "/startupz":
			ready = started.Load()
		case "/readyz":
			if started.Load() && !draining.Load() {
				ctx, cancel := context.WithTimeout(r.Context(), 100*time.Millisecond)
				defer cancel()
				request, err := http.NewRequestWithContext(ctx, "GET", dependency.URL, nil)
				must(err)
				response, err := client.Do(request)
				if err == nil {
					ready = response.StatusCode == 204
					must(response.Body.Close())
				}
			}
		default:
			w.WriteHeader(404)
			return
		}
		if !ready {
			w.WriteHeader(503)
			return
		}
		w.WriteHeader(200)
	}))
	defer server.Close()
	probe := func(method, route string, status int, calls int64) {
		before := dependencyCalls.Load()
		request, err := http.NewRequest(method, server.URL+route, nil)
		must(err)
		response, err := client.Do(request)
		must(err)
		_, readError := io.Copy(io.Discard, response.Body)
		closeError := response.Body.Close()
		must(readError)
		must(closeError)
		check(response.Proto == "HTTP/1.1" && response.StatusCode == status, "wrong probe status")
		check(response.Header.Get("Cache-Control") == "no-store", "probe response must not be cached")
		check(dependencyCalls.Load() == before+calls, "unexpected dependency work")
		if status == 405 {
			check(response.Header.Get("Allow") == "GET", "method contract missing")
		}
		fmt.Printf("%s: %d, dependency calls=%d\n", route, status, dependencyCalls.Load())
	}
	probe("GET", "/livez", 200, 0)
	probe("GET", "/startupz", 503, 0)
	probe("GET", "/readyz", 503, 0)
	started.Store(true)
	probe("GET", "/startupz", 200, 0)
	probe("GET", "/readyz", 200, 1)
	mode.Store(1)
	probe("GET", "/livez", 200, 0)
	probe("GET", "/readyz", 503, 1)
	mode.Store(2)
	probe("GET", "/readyz", 503, 1)
	select {
	case <-canceled:
		fmt.Println("deadline canceled the stalled dependency request")
	case <-time.After(time.Second):
		panic("dependency did not observe cancellation")
	}
	mode.Store(0)
	probe("GET", "/readyz", 200, 1)
	draining.Store(true)
	probe("GET", "/readyz", 503, 0)
	probe("GET", "/livez", 200, 0)
	draining.Store(false)
	probe("GET", "/readyz", 200, 1)
	probe("POST", "/readyz", 405, 0)
	probe("GET", "/unknown", 404, 0)
	check(dependencyCalls.Load() == 5, "only five readiness checks should query the dependency")
	fmt.Println("PASS: fourteen probes; live, starting, unready, stalled, draining, and recovered")
	fmt.Println(runtime.Version())
}

Fourteen probes produce five dependency requests. Before startup, readiness and startup return 503 while liveness stays 200. Once initialized, readiness succeeds. A dependency 503 changes readiness but not liveness. A stalled request reaches the child deadline, yields 503, and the dependency observes cancellation; the next healthy check succeeds. Draining returns 503 without contacting the dependency, while liveness remains successful. Clearing the fixture’s drain flag restores readiness. POST and an unknown route produce 405 and 404 independently.

The assertions passed with Go 1.27.0 on macOS. They verify status decisions, dependency-call counts, cancellation, and recovery, not Kubernetes restart thresholds, EndpointSlice propagation, real connection-pool exhaustion, or production load. The live endpoint here proves only that its HTTP handler responds; a real application’s unrecoverable local failure needs its own truthful liveness condition. Repeat the scenario against the configured consumer before relying on its actions.

Further reading

Sources