Compose · XB Field Notes

Docker health and Prometheus up answer different questions

Mon, 21 Feb 2028 00:00:00 +0000

Docker health state and Prometheus’s up series are observations from different checks. A Compose healthcheck runs a command inside the application container. Prometheus independently scrapes a configured metrics endpoint and records whether that scrape succeeded. The two observations can legitimately disagree because an application may refuse useful work while still serving its metrics, or may serve work while its metrics endpoint is broken.

When it breaks

Interpreting up=1 as proof of application readiness can hide an outage. It means the target’s metrics scrape succeeded, not that login, database writes, or another business operation worked. The application needs appropriate behavioral metrics or an external probe for those questions. A custom readiness gauge can be useful, but its meaning must come from the application contract.

The inverse mistake is assuming Docker health is automatically exported to Prometheus. Defining healthcheck in Compose does not create a Prometheus metric or configure a scrape target. An exporter or application instrumentation must explicitly expose the required observation. Even then, its collection path and freshness differ from the engine’s own health state and need to be understood.

Neither observation automatically defines a restart or routing action. Docker’s restart: always handles container termination; an unhealthy but running container is not restarted solely because of its health state. A monitoring system can alert on evidence, and an explicitly configured controller can act on it, but those connections do not appear simply because both tools are present in one Compose project.

Check it quickly

Run this script with host Python, Docker Compose, and the two pinned images available. The recorded environment used Compose 5.1.1, Docker 29.3.1, Python 3.11.16, and Prometheus 3.5.1 on Linux arm64. It builds a temporary Prometheus configuration image, creates a unique project, publishes the Prometheus API only on host loopback, and removes its own project, data volume, and tag:

from pathlib import Path
from tempfile import TemporaryDirectory
from urllib.parse import urlencode
from urllib.request import urlopen
import json,subprocess,time,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
PROM='prom/prometheus@sha256:38c3b05c3bc744ff1b0b7b4eb82196026442845e62a1e2073795565da506d7a2'
def run(args,check=True):
 result=subprocess.run(args,text=True,capture_output=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def docker(*args,check=True):return run(['docker',*args],check)
def inspect(identity):return json.loads(docker('inspect',identity).stdout)[0]

name='xbcatm-prom-health-'+uuid.uuid4().hex[:10];tag=name+':prometheus'
with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);file=root/'compose.yaml'
 def compose(*args,check=True):return run(['docker','compose','-p',name,'-f',str(file),*args],check)
 def wait(predicate):
 deadline=time.monotonic()+25
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except (OSError,ValueError):pass
 time.sleep(.2)
 raise AssertionError('observation did not become true')
 server='''from http.server import BaseHTTPRequestHandler,HTTPServer
from pathlib import Path
Path('/ready').touch()
class H(BaseHTTPRequestHandler):
 def do_GET(self):
 ready=Path('/ready').exists()
 code=(503 if Path('/broken-metrics').exists() else 200) if self.path=='/metrics' else (200 if ready else 503)
 body=('# TYPE app_ready gauge\\napp_ready '+str(int(ready))+'\\n').encode()
 self.send_response(code);self.send_header('Content-Type','text/plain; version=0.0.4');self.end_headers();self.wfile.write(body)
 def log_message(self,*args):pass
HTTPServer(('0.0.0.0',8080),H).serve_forever()
'''
 (root/'prometheus.yml').write_text('global:\n scrape_interval: 1s\n scrape_timeout: 500ms\nscrape_configs:\n - job_name: app\n static_configs:\n - targets: [app:8080]\n')
 (root/'Dockerfile').write_text(f'FROM {PROM}\nCOPY prometheus.yml /etc/prometheus/prometheus.yml\n')
 health=['CMD','python','-c','import urllib.request;urllib.request.urlopen("http://127.0.0.1:8080/health",timeout=.5).read()']
 file.write_text(json.dumps({'services':{'app':{'image':BASE,'command':['python','-c',server],'restart':'always','healthcheck':{'test':health,'interval':'1s','timeout':'1s','retries':2,'start_period':'3s'}},'prometheus':{'image':tag,'ports':[{'target':9090,'published':'0','host_ip':'127.0.0.1'}],'volumes':['metrics:/prometheus']}},'volumes':{'metrics':{}}},indent=2))
 try:
 docker('build','--network','none','-t',tag,str(root));compose('up','-d')
 app=compose('ps','-q','app').stdout.strip();prometheus=compose('ps','-q','prometheus').stdout.strip()
 port=inspect(prometheus)['NetworkSettings']['Ports']['9090/tcp'][0];assert port['HostIp']=='127.0.0.1'
 endpoint='http://127.0.0.1:'+port['HostPort']
 def value(expression):
 with urlopen(endpoint+'/api/v1/query?'+urlencode({'query':expression}),timeout=2) as response:data=json.load(response)
 result=data['data']['result'];return float(result[0]['value'][1]) if result else None
 wait(lambda:value('up{job="app"}')==1 and value('app_ready')==1 and inspect(app)['State']['Health']['Status']=='healthy')
 time.sleep(11)
 original=inspect(app)
 docker('exec',app,'python','-c','from pathlib import Path;Path("/ready").unlink()')
 wait(lambda:value('up{job="app"}')==1 and value('app_ready')==0 and inspect(app)['State']['Health']['Status']=='unhealthy')
 current=inspect(app);assert current['RestartCount']==0 and current['State']['StartedAt']==original['State']['StartedAt']
 print('Docker unhealthy and app_ready=0 coexist with Prometheus up=1 because the metrics endpoint is still scrapeable')
 docker('exec',app,'python','-c','from pathlib import Path;Path("/broken-metrics").touch()')
 wait(lambda:value('up{job="app"}')==0)
 assert inspect(app)['State']['Running'] and inspect(app)['RestartCount']==0
 print('returning HTTP 503 from /metrics changes up to 0 without stopping or restarting the application')
 finally:
 compose('down','--volumes','--remove-orphans',check=False);docker('image','rm',tag,check=False)

The application serves /health and /metrics through the same small HTTP server. Its readiness file controls the health response and the explicitly defined app_ready gauge. A different file controls whether the metrics request itself returns an error. Those independent switches let the experiment separate application state from scrape availability without replacing Prometheus with a simulated calculation.

Prometheus scrapes app:8080 once per second, using the default metrics path and a half-second scrape timeout. Compose separately runs the application’s internal health command. The controller waits until health is healthy, up is one, and app_ready is one, then allows the application to remain running beyond its initial startup period before changing the condition.

Removing /ready causes /health to return HTTP 503 and the gauge to become zero. After two failed health probes, Docker reports unhealthy. Prometheus still records up=1 because /metrics continues returning a successful response containing valid metric data. The fixture checks all three observations through the actual Docker inspection and Prometheus query APIs.

The application has restart: always, but its start timestamp remains unchanged and its restart count remains zero. This directly verifies that the unhealthy status has not caused an engine restart. The result is useful for incident diagnosis: a running process and a functioning metrics handler can coexist with a failed readiness contract.

Creating /broken-metrics makes the metrics endpoint return HTTP 503 too. A subsequent scrape records up=0, while the application process remains running. The test verifies the failed-scrape case independently of process termination. A metrics path error, timeout, or malformed exposition can similarly make scraping fail without proving that the whole process has exited.

The controller polls for observed states instead of assuming the two probe schedules align exactly. The short intervals belong to this disposable experiment and are not a universal production recommendation. Cleanup explicitly deletes only its temporary metrics volume, whose retained samples are not needed after the assertions.

Alert on the contract and the observation path

Use separate signals for scrape availability, application readiness, and user-visible success. Give custom metrics precise definitions and bounded label sets. If exporting Docker health, document the exporter, target identity, and update delay, then verify that the metric actually reflects the engine state rather than relying on a coincidentally similar name.

Distinguish a failed scrape from a missing series or removed target. An empty query result is not automatically the same as a zero sample; discovery and staleness rules affect what is present. Alert rules should account for the intended target inventory and acceptable observation gaps.

Finally, test the action separately from the signal. Verify alert routing, traffic removal, or recovery behavior in the system that owns it. Good monitoring preserves these distinctions so operators can tell whether the service failed, the measurement path failed, or both did.

Sources

Compose healthcheck and restart settings, container probe configuration and service restart policy, checked 2026-09-12.
Prometheus jobs and instances, the up series describing scrape success, checked 2026-09-12.
Prometheus scrape configuration, scrape targets, paths, intervals, and timeouts, checked 2026-09-12.
Docker restart policies, restart behavior for container termination, checked 2026-09-12.

A Compose project combines declarations with external inputs

Wed, 18 Aug 2027 00:00:00 +0000

Docker Compose turns service declarations into containers, networks, and storage attachments managed as a project. The file describes useful relationships: which image runs a service, how configuration reaches it, and which storage or network it uses. Reproducing a stack still depends on external inputs such as image availability, variable values, host capabilities, and existing data. A versioned file makes those inputs reviewable only when they are identified.

When it breaks

The same Compose text can resolve differently when the invoking shell supplies different variables. An env file participates in interpolation, but it is not automatically equivalent to the environment eventually injected into every container. Service-level environment declarations and env_file settings have their own roles. Inspect the rendered configuration and the running service rather than assuming a value in one file must be the final value.

Network addresses are another trap. Services on a Compose network can resolve one another by service name and use the container port. Published host ports serve a different path. Pointing a client container at localhost usually sends the request back to that client, not to the sibling API. Hardcoding a container IP also makes replacement more fragile than using the stable service name.

Project lifecycle does not give every resource the same lifetime. A normal down operation removes the project’s containers and networks, while named volumes remain unless volume removal is requested. Recreating the network can change its identity and assigned addresses. Retained data does not mean the old network survived, and a newly created container does not imply an empty data volume.

Check it quickly

Run the script with host Python, Docker Compose, and the pinned Python image. It was executed with Compose 5.1.1 and Docker 29.3.1 on Linux arm64. The generated Compose document uses JSON syntax accepted by YAML, requires no extra Python parser, and has a unique project name. The API is published only on host loopback at an automatically assigned port:

from pathlib import Path
from tempfile import TemporaryDirectory
from urllib.request import urlopen
import json,os,subprocess,time,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def run(args,check=True,env=None):
 result=subprocess.run(args,text=True,capture_output=True,timeout=120,env=env)
 if check:assert result.returncode==0,result.stderr
 return result
def docker(*args,check=True):return run(['docker',*args],check)

name='xbcatm-project-'+uuid.uuid4().hex[:10];environment=os.environ.copy();environment.pop('TASK_COMPOSE_GREETING',None)
with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);file=root/'compose.yaml'
 def compose(*args,check=True,env=None):return run(['docker','compose','-p',name,'--env-file',str(root/'.env'),'-f',str(file),*args],check,env or environment)
 def container(service):return compose('ps','-q',service).stdout.strip()
 def inspect(identity):return json.loads(docker('inspect',identity).stdout)[0]
 def request():
 port=inspect(container('api'))['NetworkSettings']['Ports']['8080/tcp'][0]
 assert port['HostIp']=='127.0.0.1'
 deadline=time.monotonic()+10
 while time.monotonic()<deadline:
 try:
 with urlopen('http://127.0.0.1:'+port['HostPort'],timeout=1) as response:return response.read().decode()
 except OSError:time.sleep(.1)
 raise AssertionError('published endpoint unavailable')
 server='from http.server import BaseHTTPRequestHandler,HTTPServer\nimport os\nclass H(BaseHTTPRequestHandler):\n def do_GET(self):self.send_response(200);self.end_headers();self.wfile.write(os.environ["GREETING"].encode())\n def log_message(self,*args):pass\nHTTPServer(("0.0.0.0",8080),H).serve_forever()\n'
 spec={'services':{'api':{'image':BASE,'command':['python','-c',server],'environment':{'GREETING':'${TASK_COMPOSE_GREETING:-default}'},'ports':[{'target':8080,'published':'0','host_ip':'127.0.0.1'}],'volumes':['state:/state']},'client':{'image':BASE,'command':['python','-c','import time;time.sleep(180)']}},'volumes':{'state':{}}}
 file.write_text(json.dumps(spec,indent=2));(root/'.env').write_text('TASK_COMPOSE_GREETING=from-file\n')
 try:
 rendered=json.loads(compose('config','--format','json').stdout);assert rendered['services']['api']['environment']['GREETING']=='from-file'
 override=environment|{'TASK_COMPOSE_GREETING':'from-shell'}
 rendered=json.loads(compose('config','--format','json',env=override).stdout);assert rendered['services']['api']['environment']['GREETING']=='from-shell'
 compose('up','-d',env=override);assert request()=='from-shell'
 api=container('api');client=container('client');first_network=next(iter(inspect(api)['NetworkSettings']['Networks'].values()))['NetworkID']
 result=docker('exec',client,'python','-c','import urllib.request;print(urllib.request.urlopen("http://api:8080",timeout=2).read().decode())')
 assert result.stdout.strip()=='from-shell'
 docker('exec',client,'python','-c','import socket;s=socket.socket();assert s.connect_ex(("127.0.0.1",8080))!=0;s.close()')
 docker('exec',api,'python','-c','from pathlib import Path;Path("/state/record").write_text("preserved")')
 volume=next(m['Name'] for m in inspect(api)['Mounts'] if m['Destination']=='/state')
 assert volume==name+'_state'
 compose('down');assert docker('network','inspect',first_network,check=False).returncode!=0
 assert docker('volume','inspect',volume,check=False).returncode==0
 compose('up','-d',env=override);assert request()=='from-shell'
 api=container('api');assert next(iter(inspect(api)['NetworkSettings']['Networks'].values()))['NetworkID']!=first_network
 docker('exec',api,'python','-c','from pathlib import Path;assert Path("/state/record").read_text()=="preserved"')
 assert docker('exec',container('client'),'python','-c','import urllib.request;print(urllib.request.urlopen("http://api:8080",timeout=2).read().decode())').stdout.strip()=='from-shell'
 print('shell interpolation overrides .env; internal clients resolve api:8080 while localhost remains their own container')
 print('the host endpoint is published only on loopback; down/up recreates the project network and retains the named data volume')
 finally:compose('down','--volumes','--remove-orphans',check=False)

The private env file supplies TASK_COMPOSE_GREETING=from-file. The script explicitly selects that file and removes the same variable from its initial shell environment. compose config --format json resolves the service’s GREETING value to from-file. Providing from-shell in the invoking environment then changes the rendered value, establishing the tested interpolation precedence.

Starting the project with the override makes the API return from-shell. The host-side request uses the published port discovered through container inspection, and the binding is asserted to be 127.0.0.1. This verifies both the effective configuration and the intended exposure. Merely rendering a valid document would not prove the container was started with that value or port binding.

The client service successfully requests http://api:8080 through Compose’s network and gets the same response. A connection attempt to its own loopback port 8080 fails. These two observations distinguish service-name discovery from host publishing and container-local loopback. No fixed container IP is embedded in the application request.

The API writes a marker to its named state volume. After ordinary compose down, inspection confirms the old network no longer exists while the named volume still does. Bringing the project up again creates a different network identity. The marker remains readable, and the client can still reach the API using the unchanged service name.

The fixture’s final cleanup includes --volumes because all of its data is disposable. That option has a different destructive scope from the earlier ordinary down used to demonstrate retention. In a real environment, apply it only when removal of those volumes is intended; it should not be appended casually to every restart or troubleshooting command.

Record the project inputs that matter

Keep required variable names, image digests, storage ownership, and exposed ports explicit. Render configuration in the intended execution environment before starting the stack, but avoid publishing rendered output containing credentials. Pinning an image reduces one source of drift; it does not freeze host architecture, mounted files, runtime configuration, or persistent database contents.

Use separate project names for independent local or CI runs when resources should be isolated. Explicitly named external networks or volumes can intentionally cross that boundary, so review those references rather than assuming project naming isolates every attachment. Port bindings can also conflict across otherwise separate projects.

Compose can support a managed single-host deployment as well as local testing. Its suitability depends on operational requirements, not a blanket ban on production use. Host redundancy, rollout coordination, backup, readiness, and monitoring still need deliberate solutions. The value of the declaration is a clear, testable resource model with known limits.

Sources

Compose services reference, service configuration, environment, ports, and volume mounts, checked 2026-09-12.
Networking in Compose, project networks and service-name discovery, checked 2026-09-12.
Compose variable interpolation, shell and env-file interpolation precedence, checked 2026-09-12.
Docker compose down, network removal and optional volume removal, checked 2026-09-12.

Compose startup ordering needs an explicit readiness condition

Thu, 08 Apr 2027 00:00:00 +0000

Docker Compose describes a graph of services and can start them according to declared dependencies. Process startup is only one possible condition in that graph. A database or API may be running before it can accept the operation another service needs. An explicit service_healthy condition lets Compose wait for a dependency’s configured healthcheck before starting the dependent service, provided that check represents meaningful readiness.

When it breaks

The short depends_on form establishes startup ordering without establishing application readiness. Replacing it with an arbitrary sleep is fragile because initialization time changes with storage, cache state, migrations, or machine load. A sleep can be unnecessarily long on one run and too short on another. The condition should come from observable dependency behavior rather than an estimated elapsed time.

A health-based dependency condition is also not a continuous availability guarantee. After the dependent starts, the dependency can fail or restart. Compose does not automatically stop or restart an already-running dependent merely because the dependency’s health later changes. Applications still need a runtime policy for connection loss, bounded retries, and operations that may have partially completed.

The restart: true option nested under a dependency has a separate meaning from a service’s restart policy. It concerns dependent restarts after explicit Compose operations that update or restart the dependency. It should not be read as a general subscription to every health transition or every engine-level crash. These similarly named settings solve different lifecycle problems.

Check it quickly

Run the script using host Python, Docker Compose, and the pinned Python image. The recorded run used Compose 5.1.1, Docker 29.3.1, and Linux arm64 Python 3.11.16. The script writes a Compose document using JSON syntax, which is accepted as YAML, creates a unique project, and keeps all HTTP traffic on its project network. Cleanup removes only that project:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,time,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def run(args,check=True):
 result=subprocess.run(args,text=True,capture_output=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def docker(*args,check=True):return run(['docker',*args],check)
def inspect(name):return json.loads(docker('inspect',name).stdout)[0]

name='xbcatm-depends-'+uuid.uuid4().hex[:10]
with TemporaryDirectory(prefix=name+'-') as directory:
 file=Path(directory)/'compose.yaml';base=['docker','compose','-p',name,'-f',str(file)]
 def compose(*args,check=True):return run(base+list(args),check)
 def service_id(service):
 return docker('ps','-aq','--filter','label=com.docker.compose.project='+name,'--filter','label=com.docker.compose.service='+service).stdout.strip()
 def wait(predicate):
 deadline=time.monotonic()+20
 while time.monotonic()<deadline:
 result=predicate()
 if result:return result
 time.sleep(.1)
 raise AssertionError('condition did not become true')
 server='from http.server import BaseHTTPRequestHandler,HTTPServer\nfrom pathlib import Path\nclass H(BaseHTTPRequestHandler):\n def do_GET(self):self.send_response(200 if Path("/ready").exists() else 503);self.end_headers()\n def log_message(self,*args):pass\nHTTPServer(("0.0.0.0",8080),H).serve_forever()\n'
 health=['CMD','python','-c','import urllib.request;urllib.request.urlopen("http://127.0.0.1:8080",timeout=.5).read()']
 application='import urllib.request,time;urllib.request.urlopen("http://dependency:8080",timeout=2).read();print("started-after-ready",flush=True);time.sleep(180)'
 file.write_text(json.dumps({'services':{'dependency':{'image':BASE,'command':['python','-c',server],'healthcheck':{'test':health,'interval':'1s','timeout':'1s','retries':2,'start_period':'30s'}},'application':{'image':BASE,'command':['python','-c',application],'depends_on':{'dependency':{'condition':'service_healthy'}}}}},indent=2))
 process=None
 try:
 process=subprocess.Popen(base+['up','-d'],text=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
 dependency=wait(lambda:service_id('dependency'))
 wait(lambda:inspect(dependency)['State']['Running'])
 assert inspect(dependency)['State']['Health']['Status']=='starting'
 application_id=service_id('application')
 assert not application_id or not inspect(application_id)['State']['Running']
 assert process.poll() is None
 docker('exec',dependency,'python','-c','from pathlib import Path;Path("/ready").touch()')
 stdout,stderr=process.communicate(timeout=30);assert process.returncode==0,stderr
 application_id=service_id('application');wait(lambda:'started-after-ready' in docker('logs',application_id).stdout)
 original=inspect(application_id);assert original['State']['Running']
 assert inspect(dependency)['State']['Health']['Status']=='healthy'
 docker('exec',dependency,'python','-c','from pathlib import Path;Path("/ready").unlink()')
 wait(lambda:inspect(dependency)['State']['Health']['Status']=='unhealthy')
 after=inspect(application_id);assert after['State']['Running'] and after['State']['StartedAt']==original['State']['StartedAt'] and after['RestartCount']==0
 print('service_healthy keeps the dependent application stopped until the dependency probe succeeds')
 print('later dependency health failure does not stop or restart the already-running dependent application')
 finally:
 if process is not None and process.poll() is None:process.terminate();process.communicate(timeout=10)
 compose('down','--volumes','--remove-orphans',check=False)

The dependency runs an HTTP server whose response remains 503 until /ready exists. Its healthcheck tests that real endpoint. A thirty-second startup grace period gives the controller time to inspect the initial state without early failures immediately exhausting the two-failure threshold. The application declares condition: service_healthy and performs its own HTTP request when it eventually starts.

The controller launches compose up -d asynchronously and observes the dependency running with health starting. At that point the application is either not yet created or not running, and the Compose command is still waiting. The test allows either creation detail because the required contract is the dependent’s startup gate, not a particular intermediate container-allocation sequence.

Creating /ready changes the dependency’s HTTP response to 200. Compose completes startup successfully, and the application prints started-after-ready only after its own request succeeds. This connects the declared condition to real reachable behavior. A green health label alone would be weaker evidence if the dependent could not actually access the dependency over its network path.

Next, the controller removes /ready. After enough failed probes, the dependency becomes unhealthy. The application remains running with the same start timestamp and zero restarts. This is the central negative case: a satisfied startup condition is not a promise that Compose will continuously synchronize the dependent’s lifecycle with later dependency health.

The application intentionally sleeps after its initial request so that its process lifetime can be observed independently. It is not a production client implementation, and the fixture makes no claim that its sleeping state proves successful business operations during the outage. The observed result is precisely that Compose leaves the existing dependent process alone.

Design startup and runtime handling together

Use service_started when process startup is the actual requirement, service_healthy when a meaningful healthcheck defines readiness, and a successful-completion condition for an appropriate one-shot prerequisite. A migration job’s successful exit and a long-running database’s readiness answer different questions. Choose the condition that matches the prerequisite, and retain useful failure output when it is not satisfied.

Make readiness checks cheap and specific. A TCP listener may exist before authentication, schema initialization, or required data is ready. Conversely, checking every optional downstream system can prevent otherwise useful services from starting. The check should represent the minimum dependency contract needed by the service being gated.

Once running, handle dependency changes in the application and its operational controls. Bound retries, avoid duplicating non-idempotent work, and test reconnection after replacement. Compose can coordinate a reproducible startup sequence, but ongoing correctness still depends on the services’ actual communication and failure semantics.

Sources

Control startup order in Compose, started, healthy, and completed dependency conditions, checked 2026-09-12.
Compose services reference, depends_on conditions and explicit Compose restart behavior, checked 2026-09-12.
Dockerfile HEALTHCHECK, probe state, startup grace, and consecutive failure counting, checked 2026-09-12.