A Docker HEALTHCHECK runs a configured command inside a container and records the result as health state. It adds an observation beyond whether the main process is running. The meaning of that observation depends on what the command tests: a process check, an HTTP response, or an application-specific readiness condition. Docker does not infer a universal readiness contract from the presence of the instruction.

When it breaks

A command that always exits zero produces reassuring health reports even when the application cannot serve useful work. A shell sequence can accidentally hide an earlier failure by ending with a successful command. Conversely, a probe that depends on an optional external service can mark a usable application unhealthy and encourage unnecessary recovery actions by whatever system consumes the result.

Standalone Docker restart policies react to container termination, not merely to an unhealthy health status. Adding restart=always does not make Docker restart a still-running application when its probe fails. Likewise, a healthcheck does not automatically remove a container from every proxy’s routing table. A controller or proxy must explicitly consume that state and implement the intended action.

Probe timing is another source of incorrect expectations. The retry threshold counts consecutive failures, while startup grace changes which early failures count. A successful probe during startup can end that initial grace behavior. Detection time depends on command duration, interval, timeout, and configuration; it is not simply a promise that one failed request instantly stops traffic.

Check it quickly

Run this script using host Python and Docker with the pinned base already available. The recorded environment used Docker 29.3.1 and Linux arm64 Python 3.11.16. It creates one temporary image and container, disables networking outside the container, and uses its own loopback HTTP server. The only state changed is a harmless readiness file inside the experiment:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,time,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
    result=subprocess.run(['docker',*args],text=True,capture_output=True,timeout=120)
    if check:assert result.returncode==0,result.stderr
    return result
def inspect(container):return json.loads(docker('inspect',container).stdout)[0]
def wait_for(container,wanted):
    deadline=time.monotonic()+25
    while time.monotonic()<deadline:
        state=inspect(container)
        if state['State']['Health']['Status']==wanted:return state
        time.sleep(.1)
    raise AssertionError('health did not reach '+wanted)

name='xbcatm-health-'+uuid.uuid4().hex[:10];tag=name+':probe'
try:
    with TemporaryDirectory(prefix=name+'-') as directory:
        root=Path(directory)
        (root/'server.py').write_text('from http.server import BaseHTTPRequestHandler,HTTPServer\nfrom pathlib import Path\nclass Handler(BaseHTTPRequestHandler):\n def do_GET(self):\n  code=200 if Path("/ready").exists() else 503\n  self.send_response(code);self.end_headers();self.wfile.write(str(code).encode())\n def log_message(self,*args):pass\nHTTPServer(("127.0.0.1",8080),Handler).serve_forever()\n')
        probe='import urllib.request;urllib.request.urlopen("http://127.0.0.1:8080/health",timeout=.5).read()'
        (root/'Dockerfile').write_text(f'FROM {BASE}\nCOPY server.py /server.py\nHEALTHCHECK --interval=1s --timeout=1s --retries=2 --start-period=5s CMD ["python", "-c", {json.dumps(probe)}]\nCMD ["python", "/server.py"]\n')
        docker('build','--network','none','-t',tag,str(root))
        began=time.monotonic()
        docker('run','-d','--name',name,'--network','none','--restart','always',tag)
        initial=inspect(name);assert initial['State']['Health']['Status']=='starting'
        first=wait_for(name,'unhealthy');assert first['State']['Running'] and first['RestartCount']==0
        assert first['State']['StartedAt']==initial['State']['StartedAt']
        assert first['State']['Health']['FailingStreak']>=2
        assert any(log['ExitCode']!=0 and '503' in log['Output'] for log in first['State']['Health']['Log'])
        time.sleep(max(0,12-(time.monotonic()-began)))
        stable=inspect(name);assert stable['State']['Running'] and stable['RestartCount']==0
        assert stable['State']['StartedAt']==initial['State']['StartedAt'] and stable['State']['Health']['Status']=='unhealthy'
        docker('exec',name,'python','-c','from pathlib import Path;Path("/ready").touch()')
        healthy=wait_for(name,'healthy');assert healthy['RestartCount']==0 and healthy['State']['Health']['FailingStreak']==0
        docker('exec',name,'python','-c','from pathlib import Path;Path("/ready").unlink()')
        unhealthy=wait_for(name,'unhealthy');assert unhealthy['RestartCount']==0 and unhealthy['State']['Running']
        assert unhealthy['State']['StartedAt']==initial['State']['StartedAt']
        print('starting -> unhealthy -> healthy -> unhealthy follows the real HTTP probe; failure logs contain HTTP 503')
        print('restart=always does not restart a still-running process solely because its health is unhealthy')
finally:
    docker('rm','-f',name,check=False);docker('image','rm',tag,check=False)

The server returns HTTP 503 while /ready is absent. Its Dockerfile healthcheck uses Python’s HTTP client, which raises on that status and exits unsuccessfully. The probe has a one-second interval and timeout, two retries, and a five-second start period. The container starts with restart=always, and initial inspection records health as starting.

After the counted failures, Docker reports unhealthy; the health log contains the real HTTP 503 error and the failing streak is at least two. The fixture retains the initial start timestamp, waits until at least twelve seconds after launch, and confirms the main process is still running with no restart. This avoids confusing initial startup behavior with recovery from an established unhealthy state.

Creating /ready makes the HTTP endpoint return 200. A later probe changes health to healthy and resets the failing streak to zero. The same container process remains running. That transition demonstrates actual application recovery as observed by the command, without requiring the controller to manufacture a restart simply to clear the status.

Removing the file makes the endpoint fail again, and health eventually returns to unhealthy. The start timestamp remains the same and RestartCount remains zero. The test verifies a full failure, recovery, and failure sequence, rather than observing one initial boot failure and assuming all later health behavior will be identical.

Probe exit status is the interface Docker consumes: zero means success, one indicates failure, and two is reserved by the Dockerfile contract. The Python client naturally returns a failing process status for the HTTP error. The script also inspects the diagnostic output, which helps distinguish a meaningful readiness failure from a missing probe executable or a malformed command.

Define the reaction separately

Choose a cheap, bounded check that represents the state its consumer needs. An internal HTTP check can verify application handling, but does not establish public DNS, TLS termination, proxy routing, or end-user reachability. Use external observations when those paths matter. Keep probes free of destructive actions and avoid logging credentials into retained health output.

For Compose startup dependencies, explicitly use a health-based condition when another service must wait for readiness. For runtime routing or remediation, verify the actual controller’s policy and test the action it performs. Health observation and action should be connected deliberately, with enough grace to avoid turning transient dependency trouble into restart storms.

During an incident, inspect main-process state, health state, probe logs, and restart history separately. A running unhealthy process, an exited process, and a repeatedly restarting process are different failures. Clear evidence for each state makes the recovery choice more precise.

Sources