Docker health and Prometheus up answer different questions

Mon, 21 Feb 2028 00:00:00 +0000

Docker health state and Prometheus’s up series are observations from different checks. A Compose healthcheck runs a command inside the application container. Prometheus independently scrapes a configured metrics endpoint and records whether that scrape succeeded. The two observations can legitimately disagree because an application may refuse useful work while still serving its metrics, or may serve work while its metrics endpoint is broken.

When it breaks

Interpreting up=1 as proof of application readiness can hide an outage. It means the target’s metrics scrape succeeded, not that login, database writes, or another business operation worked. The application needs appropriate behavioral metrics or an external probe for those questions. A custom readiness gauge can be useful, but its meaning must come from the application contract.

The inverse mistake is assuming Docker health is automatically exported to Prometheus. Defining healthcheck in Compose does not create a Prometheus metric or configure a scrape target. An exporter or application instrumentation must explicitly expose the required observation. Even then, its collection path and freshness differ from the engine’s own health state and need to be understood.

Neither observation automatically defines a restart or routing action. Docker’s restart: always handles container termination; an unhealthy but running container is not restarted solely because of its health state. A monitoring system can alert on evidence, and an explicitly configured controller can act on it, but those connections do not appear simply because both tools are present in one Compose project.

Check it quickly

Run this script with host Python, Docker Compose, and the two pinned images available. The recorded environment used Compose 5.1.1, Docker 29.3.1, Python 3.11.16, and Prometheus 3.5.1 on Linux arm64. It builds a temporary Prometheus configuration image, creates a unique project, publishes the Prometheus API only on host loopback, and removes its own project, data volume, and tag:

from pathlib import Path
from tempfile import TemporaryDirectory
from urllib.parse import urlencode
from urllib.request import urlopen
import json,subprocess,time,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
PROM='prom/prometheus@sha256:38c3b05c3bc744ff1b0b7b4eb82196026442845e62a1e2073795565da506d7a2'
def run(args,check=True):
 result=subprocess.run(args,text=True,capture_output=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def docker(*args,check=True):return run(['docker',*args],check)
def inspect(identity):return json.loads(docker('inspect',identity).stdout)[0]

name='xbcatm-prom-health-'+uuid.uuid4().hex[:10];tag=name+':prometheus'
with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);file=root/'compose.yaml'
 def compose(*args,check=True):return run(['docker','compose','-p',name,'-f',str(file),*args],check)
 def wait(predicate):
 deadline=time.monotonic()+25
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except (OSError,ValueError):pass
 time.sleep(.2)
 raise AssertionError('observation did not become true')
 server='''from http.server import BaseHTTPRequestHandler,HTTPServer
from pathlib import Path
Path('/ready').touch()
class H(BaseHTTPRequestHandler):
 def do_GET(self):
 ready=Path('/ready').exists()
 code=(503 if Path('/broken-metrics').exists() else 200) if self.path=='/metrics' else (200 if ready else 503)
 body=('# TYPE app_ready gauge\\napp_ready '+str(int(ready))+'\\n').encode()
 self.send_response(code);self.send_header('Content-Type','text/plain; version=0.0.4');self.end_headers();self.wfile.write(body)
 def log_message(self,*args):pass
HTTPServer(('0.0.0.0',8080),H).serve_forever()
'''
 (root/'prometheus.yml').write_text('global:\n scrape_interval: 1s\n scrape_timeout: 500ms\nscrape_configs:\n - job_name: app\n static_configs:\n - targets: [app:8080]\n')
 (root/'Dockerfile').write_text(f'FROM {PROM}\nCOPY prometheus.yml /etc/prometheus/prometheus.yml\n')
 health=['CMD','python','-c','import urllib.request;urllib.request.urlopen("http://127.0.0.1:8080/health",timeout=.5).read()']
 file.write_text(json.dumps({'services':{'app':{'image':BASE,'command':['python','-c',server],'restart':'always','healthcheck':{'test':health,'interval':'1s','timeout':'1s','retries':2,'start_period':'3s'}},'prometheus':{'image':tag,'ports':[{'target':9090,'published':'0','host_ip':'127.0.0.1'}],'volumes':['metrics:/prometheus']}},'volumes':{'metrics':{}}},indent=2))
 try:
 docker('build','--network','none','-t',tag,str(root));compose('up','-d')
 app=compose('ps','-q','app').stdout.strip();prometheus=compose('ps','-q','prometheus').stdout.strip()
 port=inspect(prometheus)['NetworkSettings']['Ports']['9090/tcp'][0];assert port['HostIp']=='127.0.0.1'
 endpoint='http://127.0.0.1:'+port['HostPort']
 def value(expression):
 with urlopen(endpoint+'/api/v1/query?'+urlencode({'query':expression}),timeout=2) as response:data=json.load(response)
 result=data['data']['result'];return float(result[0]['value'][1]) if result else None
 wait(lambda:value('up{job="app"}')==1 and value('app_ready')==1 and inspect(app)['State']['Health']['Status']=='healthy')
 time.sleep(11)
 original=inspect(app)
 docker('exec',app,'python','-c','from pathlib import Path;Path("/ready").unlink()')
 wait(lambda:value('up{job="app"}')==1 and value('app_ready')==0 and inspect(app)['State']['Health']['Status']=='unhealthy')
 current=inspect(app);assert current['RestartCount']==0 and current['State']['StartedAt']==original['State']['StartedAt']
 print('Docker unhealthy and app_ready=0 coexist with Prometheus up=1 because the metrics endpoint is still scrapeable')
 docker('exec',app,'python','-c','from pathlib import Path;Path("/broken-metrics").touch()')
 wait(lambda:value('up{job="app"}')==0)
 assert inspect(app)['State']['Running'] and inspect(app)['RestartCount']==0
 print('returning HTTP 503 from /metrics changes up to 0 without stopping or restarting the application')
 finally:
 compose('down','--volumes','--remove-orphans',check=False);docker('image','rm',tag,check=False)

The application serves /health and /metrics through the same small HTTP server. Its readiness file controls the health response and the explicitly defined app_ready gauge. A different file controls whether the metrics request itself returns an error. Those independent switches let the experiment separate application state from scrape availability without replacing Prometheus with a simulated calculation.

Prometheus scrapes app:8080 once per second, using the default metrics path and a half-second scrape timeout. Compose separately runs the application’s internal health command. The controller waits until health is healthy, up is one, and app_ready is one, then allows the application to remain running beyond its initial startup period before changing the condition.

Removing /ready causes /health to return HTTP 503 and the gauge to become zero. After two failed health probes, Docker reports unhealthy. Prometheus still records up=1 because /metrics continues returning a successful response containing valid metric data. The fixture checks all three observations through the actual Docker inspection and Prometheus query APIs.

The application has restart: always, but its start timestamp remains unchanged and its restart count remains zero. This directly verifies that the unhealthy status has not caused an engine restart. The result is useful for incident diagnosis: a running process and a functioning metrics handler can coexist with a failed readiness contract.

Creating /broken-metrics makes the metrics endpoint return HTTP 503 too. A subsequent scrape records up=0, while the application process remains running. The test verifies the failed-scrape case independently of process termination. A metrics path error, timeout, or malformed exposition can similarly make scraping fail without proving that the whole process has exited.

The controller polls for observed states instead of assuming the two probe schedules align exactly. The short intervals belong to this disposable experiment and are not a universal production recommendation. Cleanup explicitly deletes only its temporary metrics volume, whose retained samples are not needed after the assertions.

Alert on the contract and the observation path

Use separate signals for scrape availability, application readiness, and user-visible success. Give custom metrics precise definitions and bounded label sets. If exporting Docker health, document the exporter, target identity, and update delay, then verify that the metric actually reflects the engine state rather than relying on a coincidentally similar name.

Distinguish a failed scrape from a missing series or removed target. An empty query result is not automatically the same as a zero sample; discovery and staleness rules affect what is present. Alert rules should account for the intended target inventory and acceptable observation gaps.

Finally, test the action separately from the signal. Verify alert routing, traffic removal, or recovery behavior in the system that owns it. Good monitoring preserves these distinctions so operators can tell whether the service failed, the measurement path failed, or both did.

Sources

Compose healthcheck and restart settings, container probe configuration and service restart policy, checked 2026-09-12.
Prometheus jobs and instances, the up series describing scrape success, checked 2026-09-12.
Prometheus scrape configuration, scrape targets, paths, intervals, and timeouts, checked 2026-09-12.
Docker restart policies, restart behavior for container termination, checked 2026-09-12.

Orchestrator · XB Field Notes

Docker health and Prometheus up answer different questions

When it breaks

Check it quickly

Alert on the contract and the observation path

Sources