Healthcheck · XB Field Notes

Docker health and Prometheus up answer different questions

Mon, 21 Feb 2028 00:00:00 +0000

Docker health state and Prometheus’s up series are observations from different checks. A Compose healthcheck runs a command inside the application container. Prometheus independently scrapes a configured metrics endpoint and records whether that scrape succeeded. The two observations can legitimately disagree because an application may refuse useful work while still serving its metrics, or may serve work while its metrics endpoint is broken.

When it breaks

Interpreting up=1 as proof of application readiness can hide an outage. It means the target’s metrics scrape succeeded, not that login, database writes, or another business operation worked. The application needs appropriate behavioral metrics or an external probe for those questions. A custom readiness gauge can be useful, but its meaning must come from the application contract.

The inverse mistake is assuming Docker health is automatically exported to Prometheus. Defining healthcheck in Compose does not create a Prometheus metric or configure a scrape target. An exporter or application instrumentation must explicitly expose the required observation. Even then, its collection path and freshness differ from the engine’s own health state and need to be understood.

Neither observation automatically defines a restart or routing action. Docker’s restart: always handles container termination; an unhealthy but running container is not restarted solely because of its health state. A monitoring system can alert on evidence, and an explicitly configured controller can act on it, but those connections do not appear simply because both tools are present in one Compose project.

Check it quickly

Run this script with host Python, Docker Compose, and the two pinned images available. The recorded environment used Compose 5.1.1, Docker 29.3.1, Python 3.11.16, and Prometheus 3.5.1 on Linux arm64. It builds a temporary Prometheus configuration image, creates a unique project, publishes the Prometheus API only on host loopback, and removes its own project, data volume, and tag:

from pathlib import Path
from tempfile import TemporaryDirectory
from urllib.parse import urlencode
from urllib.request import urlopen
import json,subprocess,time,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
PROM='prom/prometheus@sha256:38c3b05c3bc744ff1b0b7b4eb82196026442845e62a1e2073795565da506d7a2'
def run(args,check=True):
 result=subprocess.run(args,text=True,capture_output=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def docker(*args,check=True):return run(['docker',*args],check)
def inspect(identity):return json.loads(docker('inspect',identity).stdout)[0]

name='xbcatm-prom-health-'+uuid.uuid4().hex[:10];tag=name+':prometheus'
with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);file=root/'compose.yaml'
 def compose(*args,check=True):return run(['docker','compose','-p',name,'-f',str(file),*args],check)
 def wait(predicate):
 deadline=time.monotonic()+25
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except (OSError,ValueError):pass
 time.sleep(.2)
 raise AssertionError('observation did not become true')
 server='''from http.server import BaseHTTPRequestHandler,HTTPServer
from pathlib import Path
Path('/ready').touch()
class H(BaseHTTPRequestHandler):
 def do_GET(self):
 ready=Path('/ready').exists()
 code=(503 if Path('/broken-metrics').exists() else 200) if self.path=='/metrics' else (200 if ready else 503)
 body=('# TYPE app_ready gauge\\napp_ready '+str(int(ready))+'\\n').encode()
 self.send_response(code);self.send_header('Content-Type','text/plain; version=0.0.4');self.end_headers();self.wfile.write(body)
 def log_message(self,*args):pass
HTTPServer(('0.0.0.0',8080),H).serve_forever()
'''
 (root/'prometheus.yml').write_text('global:\n scrape_interval: 1s\n scrape_timeout: 500ms\nscrape_configs:\n - job_name: app\n static_configs:\n - targets: [app:8080]\n')
 (root/'Dockerfile').write_text(f'FROM {PROM}\nCOPY prometheus.yml /etc/prometheus/prometheus.yml\n')
 health=['CMD','python','-c','import urllib.request;urllib.request.urlopen("http://127.0.0.1:8080/health",timeout=.5).read()']
 file.write_text(json.dumps({'services':{'app':{'image':BASE,'command':['python','-c',server],'restart':'always','healthcheck':{'test':health,'interval':'1s','timeout':'1s','retries':2,'start_period':'3s'}},'prometheus':{'image':tag,'ports':[{'target':9090,'published':'0','host_ip':'127.0.0.1'}],'volumes':['metrics:/prometheus']}},'volumes':{'metrics':{}}},indent=2))
 try:
 docker('build','--network','none','-t',tag,str(root));compose('up','-d')
 app=compose('ps','-q','app').stdout.strip();prometheus=compose('ps','-q','prometheus').stdout.strip()
 port=inspect(prometheus)['NetworkSettings']['Ports']['9090/tcp'][0];assert port['HostIp']=='127.0.0.1'
 endpoint='http://127.0.0.1:'+port['HostPort']
 def value(expression):
 with urlopen(endpoint+'/api/v1/query?'+urlencode({'query':expression}),timeout=2) as response:data=json.load(response)
 result=data['data']['result'];return float(result[0]['value'][1]) if result else None
 wait(lambda:value('up{job="app"}')==1 and value('app_ready')==1 and inspect(app)['State']['Health']['Status']=='healthy')
 time.sleep(11)
 original=inspect(app)
 docker('exec',app,'python','-c','from pathlib import Path;Path("/ready").unlink()')
 wait(lambda:value('up{job="app"}')==1 and value('app_ready')==0 and inspect(app)['State']['Health']['Status']=='unhealthy')
 current=inspect(app);assert current['RestartCount']==0 and current['State']['StartedAt']==original['State']['StartedAt']
 print('Docker unhealthy and app_ready=0 coexist with Prometheus up=1 because the metrics endpoint is still scrapeable')
 docker('exec',app,'python','-c','from pathlib import Path;Path("/broken-metrics").touch()')
 wait(lambda:value('up{job="app"}')==0)
 assert inspect(app)['State']['Running'] and inspect(app)['RestartCount']==0
 print('returning HTTP 503 from /metrics changes up to 0 without stopping or restarting the application')
 finally:
 compose('down','--volumes','--remove-orphans',check=False);docker('image','rm',tag,check=False)

The application serves /health and /metrics through the same small HTTP server. Its readiness file controls the health response and the explicitly defined app_ready gauge. A different file controls whether the metrics request itself returns an error. Those independent switches let the experiment separate application state from scrape availability without replacing Prometheus with a simulated calculation.

Prometheus scrapes app:8080 once per second, using the default metrics path and a half-second scrape timeout. Compose separately runs the application’s internal health command. The controller waits until health is healthy, up is one, and app_ready is one, then allows the application to remain running beyond its initial startup period before changing the condition.

Removing /ready causes /health to return HTTP 503 and the gauge to become zero. After two failed health probes, Docker reports unhealthy. Prometheus still records up=1 because /metrics continues returning a successful response containing valid metric data. The fixture checks all three observations through the actual Docker inspection and Prometheus query APIs.

The application has restart: always, but its start timestamp remains unchanged and its restart count remains zero. This directly verifies that the unhealthy status has not caused an engine restart. The result is useful for incident diagnosis: a running process and a functioning metrics handler can coexist with a failed readiness contract.

Creating /broken-metrics makes the metrics endpoint return HTTP 503 too. A subsequent scrape records up=0, while the application process remains running. The test verifies the failed-scrape case independently of process termination. A metrics path error, timeout, or malformed exposition can similarly make scraping fail without proving that the whole process has exited.

The controller polls for observed states instead of assuming the two probe schedules align exactly. The short intervals belong to this disposable experiment and are not a universal production recommendation. Cleanup explicitly deletes only its temporary metrics volume, whose retained samples are not needed after the assertions.

Alert on the contract and the observation path

Use separate signals for scrape availability, application readiness, and user-visible success. Give custom metrics precise definitions and bounded label sets. If exporting Docker health, document the exporter, target identity, and update delay, then verify that the metric actually reflects the engine state rather than relying on a coincidentally similar name.

Distinguish a failed scrape from a missing series or removed target. An empty query result is not automatically the same as a zero sample; discovery and staleness rules affect what is present. Alert rules should account for the intended target inventory and acceptable observation gaps.

Finally, test the action separately from the signal. Verify alert routing, traffic removal, or recovery behavior in the system that owns it. Good monitoring preserves these distinctions so operators can tell whether the service failed, the measurement path failed, or both did.

Sources

Compose healthcheck and restart settings, container probe configuration and service restart policy, checked 2026-09-12.
Prometheus jobs and instances, the up series describing scrape success, checked 2026-09-12.
Prometheus scrape configuration, scrape targets, paths, intervals, and timeouts, checked 2026-09-12.
Docker restart policies, restart behavior for container termination, checked 2026-09-12.

Docker healthchecks report a state; recovery needs a consumer

Mon, 05 Jul 2027 00:00:00 +0000

A Docker HEALTHCHECK runs a configured command inside a container and records the result as health state. It adds an observation beyond whether the main process is running. The meaning of that observation depends on what the command tests: a process check, an HTTP response, or an application-specific readiness condition. Docker does not infer a universal readiness contract from the presence of the instruction.

When it breaks

A command that always exits zero produces reassuring health reports even when the application cannot serve useful work. A shell sequence can accidentally hide an earlier failure by ending with a successful command. Conversely, a probe that depends on an optional external service can mark a usable application unhealthy and encourage unnecessary recovery actions by whatever system consumes the result.

Standalone Docker restart policies react to container termination, not merely to an unhealthy health status. Adding restart=always does not make Docker restart a still-running application when its probe fails. Likewise, a healthcheck does not automatically remove a container from every proxy’s routing table. A controller or proxy must explicitly consume that state and implement the intended action.

Probe timing is another source of incorrect expectations. The retry threshold counts consecutive failures, while startup grace changes which early failures count. A successful probe during startup can end that initial grace behavior. Detection time depends on command duration, interval, timeout, and configuration; it is not simply a promise that one failed request instantly stops traffic.

Check it quickly

Run this script using host Python and Docker with the pinned base already available. The recorded environment used Docker 29.3.1 and Linux arm64 Python 3.11.16. It creates one temporary image and container, disables networking outside the container, and uses its own loopback HTTP server. The only state changed is a harmless readiness file inside the experiment:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,time,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],text=True,capture_output=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def inspect(container):return json.loads(docker('inspect',container).stdout)[0]
def wait_for(container,wanted):
 deadline=time.monotonic()+25
 while time.monotonic()<deadline:
 state=inspect(container)
 if state['State']['Health']['Status']==wanted:return state
 time.sleep(.1)
 raise AssertionError('health did not reach '+wanted)

name='xbcatm-health-'+uuid.uuid4().hex[:10];tag=name+':probe'
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory)
 (root/'server.py').write_text('from http.server import BaseHTTPRequestHandler,HTTPServer\nfrom pathlib import Path\nclass Handler(BaseHTTPRequestHandler):\n def do_GET(self):\n code=200 if Path("/ready").exists() else 503\n self.send_response(code);self.end_headers();self.wfile.write(str(code).encode())\n def log_message(self,*args):pass\nHTTPServer(("127.0.0.1",8080),Handler).serve_forever()\n')
 probe='import urllib.request;urllib.request.urlopen("http://127.0.0.1:8080/health",timeout=.5).read()'
 (root/'Dockerfile').write_text(f'FROM {BASE}\nCOPY server.py /server.py\nHEALTHCHECK --interval=1s --timeout=1s --retries=2 --start-period=5s CMD ["python", "-c", {json.dumps(probe)}]\nCMD ["python", "/server.py"]\n')
 docker('build','--network','none','-t',tag,str(root))
 began=time.monotonic()
 docker('run','-d','--name',name,'--network','none','--restart','always',tag)
 initial=inspect(name);assert initial['State']['Health']['Status']=='starting'
 first=wait_for(name,'unhealthy');assert first['State']['Running'] and first['RestartCount']==0
 assert first['State']['StartedAt']==initial['State']['StartedAt']
 assert first['State']['Health']['FailingStreak']>=2
 assert any(log['ExitCode']!=0 and '503' in log['Output'] for log in first['State']['Health']['Log'])
 time.sleep(max(0,12-(time.monotonic()-began)))
 stable=inspect(name);assert stable['State']['Running'] and stable['RestartCount']==0
 assert stable['State']['StartedAt']==initial['State']['StartedAt'] and stable['State']['Health']['Status']=='unhealthy'
 docker('exec',name,'python','-c','from pathlib import Path;Path("/ready").touch()')
 healthy=wait_for(name,'healthy');assert healthy['RestartCount']==0 and healthy['State']['Health']['FailingStreak']==0
 docker('exec',name,'python','-c','from pathlib import Path;Path("/ready").unlink()')
 unhealthy=wait_for(name,'unhealthy');assert unhealthy['RestartCount']==0 and unhealthy['State']['Running']
 assert unhealthy['State']['StartedAt']==initial['State']['StartedAt']
 print('starting -> unhealthy -> healthy -> unhealthy follows the real HTTP probe; failure logs contain HTTP 503')
 print('restart=always does not restart a still-running process solely because its health is unhealthy')
finally:
 docker('rm','-f',name,check=False);docker('image','rm',tag,check=False)

The server returns HTTP 503 while /ready is absent. Its Dockerfile healthcheck uses Python’s HTTP client, which raises on that status and exits unsuccessfully. The probe has a one-second interval and timeout, two retries, and a five-second start period. The container starts with restart=always, and initial inspection records health as starting.

After the counted failures, Docker reports unhealthy; the health log contains the real HTTP 503 error and the failing streak is at least two. The fixture retains the initial start timestamp, waits until at least twelve seconds after launch, and confirms the main process is still running with no restart. This avoids confusing initial startup behavior with recovery from an established unhealthy state.

Creating /ready makes the HTTP endpoint return 200. A later probe changes health to healthy and resets the failing streak to zero. The same container process remains running. That transition demonstrates actual application recovery as observed by the command, without requiring the controller to manufacture a restart simply to clear the status.

Removing the file makes the endpoint fail again, and health eventually returns to unhealthy. The start timestamp remains the same and RestartCount remains zero. The test verifies a full failure, recovery, and failure sequence, rather than observing one initial boot failure and assuming all later health behavior will be identical.

Probe exit status is the interface Docker consumes: zero means success, one indicates failure, and two is reserved by the Dockerfile contract. The Python client naturally returns a failing process status for the HTTP error. The script also inspects the diagnostic output, which helps distinguish a meaningful readiness failure from a missing probe executable or a malformed command.

Define the reaction separately

Choose a cheap, bounded check that represents the state its consumer needs. An internal HTTP check can verify application handling, but does not establish public DNS, TLS termination, proxy routing, or end-user reachability. Use external observations when those paths matter. Keep probes free of destructive actions and avoid logging credentials into retained health output.

For Compose startup dependencies, explicitly use a health-based condition when another service must wait for readiness. For runtime routing or remediation, verify the actual controller’s policy and test the action it performs. Health observation and action should be connected deliberately, with enough grace to avoid turning transient dependency trouble into restart storms.

During an incident, inspect main-process state, health state, probe logs, and restart history separately. A running unhealthy process, an exited process, and a repeatedly restarting process are different failures. Clear evidence for each state makes the recovery choice more precise.

Sources

Dockerfile HEALTHCHECK reference, probe exit codes, intervals, retries, startup grace, and health states, checked 2026-09-12.
Start containers automatically, restart policies responding to container termination, checked 2026-09-12.
Control startup order in Compose, explicit dependency conditions that consume health state, checked 2026-09-12.