Docker · XB Field Notes

Docker health and Prometheus up answer different questions

Mon, 21 Feb 2028 00:00:00 +0000

Docker health state and Prometheus’s up series are observations from different checks. A Compose healthcheck runs a command inside the application container. Prometheus independently scrapes a configured metrics endpoint and records whether that scrape succeeded. The two observations can legitimately disagree because an application may refuse useful work while still serving its metrics, or may serve work while its metrics endpoint is broken.

When it breaks

Interpreting up=1 as proof of application readiness can hide an outage. It means the target’s metrics scrape succeeded, not that login, database writes, or another business operation worked. The application needs appropriate behavioral metrics or an external probe for those questions. A custom readiness gauge can be useful, but its meaning must come from the application contract.

The inverse mistake is assuming Docker health is automatically exported to Prometheus. Defining healthcheck in Compose does not create a Prometheus metric or configure a scrape target. An exporter or application instrumentation must explicitly expose the required observation. Even then, its collection path and freshness differ from the engine’s own health state and need to be understood.

Neither observation automatically defines a restart or routing action. Docker’s restart: always handles container termination; an unhealthy but running container is not restarted solely because of its health state. A monitoring system can alert on evidence, and an explicitly configured controller can act on it, but those connections do not appear simply because both tools are present in one Compose project.

Check it quickly

Run this script with host Python, Docker Compose, and the two pinned images available. The recorded environment used Compose 5.1.1, Docker 29.3.1, Python 3.11.16, and Prometheus 3.5.1 on Linux arm64. It builds a temporary Prometheus configuration image, creates a unique project, publishes the Prometheus API only on host loopback, and removes its own project, data volume, and tag:

from pathlib import Path
from tempfile import TemporaryDirectory
from urllib.parse import urlencode
from urllib.request import urlopen
import json,subprocess,time,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
PROM='prom/prometheus@sha256:38c3b05c3bc744ff1b0b7b4eb82196026442845e62a1e2073795565da506d7a2'
def run(args,check=True):
 result=subprocess.run(args,text=True,capture_output=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def docker(*args,check=True):return run(['docker',*args],check)
def inspect(identity):return json.loads(docker('inspect',identity).stdout)[0]

name='xbcatm-prom-health-'+uuid.uuid4().hex[:10];tag=name+':prometheus'
with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);file=root/'compose.yaml'
 def compose(*args,check=True):return run(['docker','compose','-p',name,'-f',str(file),*args],check)
 def wait(predicate):
 deadline=time.monotonic()+25
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except (OSError,ValueError):pass
 time.sleep(.2)
 raise AssertionError('observation did not become true')
 server='''from http.server import BaseHTTPRequestHandler,HTTPServer
from pathlib import Path
Path('/ready').touch()
class H(BaseHTTPRequestHandler):
 def do_GET(self):
 ready=Path('/ready').exists()
 code=(503 if Path('/broken-metrics').exists() else 200) if self.path=='/metrics' else (200 if ready else 503)
 body=('# TYPE app_ready gauge\\napp_ready '+str(int(ready))+'\\n').encode()
 self.send_response(code);self.send_header('Content-Type','text/plain; version=0.0.4');self.end_headers();self.wfile.write(body)
 def log_message(self,*args):pass
HTTPServer(('0.0.0.0',8080),H).serve_forever()
'''
 (root/'prometheus.yml').write_text('global:\n scrape_interval: 1s\n scrape_timeout: 500ms\nscrape_configs:\n - job_name: app\n static_configs:\n - targets: [app:8080]\n')
 (root/'Dockerfile').write_text(f'FROM {PROM}\nCOPY prometheus.yml /etc/prometheus/prometheus.yml\n')
 health=['CMD','python','-c','import urllib.request;urllib.request.urlopen("http://127.0.0.1:8080/health",timeout=.5).read()']
 file.write_text(json.dumps({'services':{'app':{'image':BASE,'command':['python','-c',server],'restart':'always','healthcheck':{'test':health,'interval':'1s','timeout':'1s','retries':2,'start_period':'3s'}},'prometheus':{'image':tag,'ports':[{'target':9090,'published':'0','host_ip':'127.0.0.1'}],'volumes':['metrics:/prometheus']}},'volumes':{'metrics':{}}},indent=2))
 try:
 docker('build','--network','none','-t',tag,str(root));compose('up','-d')
 app=compose('ps','-q','app').stdout.strip();prometheus=compose('ps','-q','prometheus').stdout.strip()
 port=inspect(prometheus)['NetworkSettings']['Ports']['9090/tcp'][0];assert port['HostIp']=='127.0.0.1'
 endpoint='http://127.0.0.1:'+port['HostPort']
 def value(expression):
 with urlopen(endpoint+'/api/v1/query?'+urlencode({'query':expression}),timeout=2) as response:data=json.load(response)
 result=data['data']['result'];return float(result[0]['value'][1]) if result else None
 wait(lambda:value('up{job="app"}')==1 and value('app_ready')==1 and inspect(app)['State']['Health']['Status']=='healthy')
 time.sleep(11)
 original=inspect(app)
 docker('exec',app,'python','-c','from pathlib import Path;Path("/ready").unlink()')
 wait(lambda:value('up{job="app"}')==1 and value('app_ready')==0 and inspect(app)['State']['Health']['Status']=='unhealthy')
 current=inspect(app);assert current['RestartCount']==0 and current['State']['StartedAt']==original['State']['StartedAt']
 print('Docker unhealthy and app_ready=0 coexist with Prometheus up=1 because the metrics endpoint is still scrapeable')
 docker('exec',app,'python','-c','from pathlib import Path;Path("/broken-metrics").touch()')
 wait(lambda:value('up{job="app"}')==0)
 assert inspect(app)['State']['Running'] and inspect(app)['RestartCount']==0
 print('returning HTTP 503 from /metrics changes up to 0 without stopping or restarting the application')
 finally:
 compose('down','--volumes','--remove-orphans',check=False);docker('image','rm',tag,check=False)

The application serves /health and /metrics through the same small HTTP server. Its readiness file controls the health response and the explicitly defined app_ready gauge. A different file controls whether the metrics request itself returns an error. Those independent switches let the experiment separate application state from scrape availability without replacing Prometheus with a simulated calculation.

Prometheus scrapes app:8080 once per second, using the default metrics path and a half-second scrape timeout. Compose separately runs the application’s internal health command. The controller waits until health is healthy, up is one, and app_ready is one, then allows the application to remain running beyond its initial startup period before changing the condition.

Removing /ready causes /health to return HTTP 503 and the gauge to become zero. After two failed health probes, Docker reports unhealthy. Prometheus still records up=1 because /metrics continues returning a successful response containing valid metric data. The fixture checks all three observations through the actual Docker inspection and Prometheus query APIs.

The application has restart: always, but its start timestamp remains unchanged and its restart count remains zero. This directly verifies that the unhealthy status has not caused an engine restart. The result is useful for incident diagnosis: a running process and a functioning metrics handler can coexist with a failed readiness contract.

Creating /broken-metrics makes the metrics endpoint return HTTP 503 too. A subsequent scrape records up=0, while the application process remains running. The test verifies the failed-scrape case independently of process termination. A metrics path error, timeout, or malformed exposition can similarly make scraping fail without proving that the whole process has exited.

The controller polls for observed states instead of assuming the two probe schedules align exactly. The short intervals belong to this disposable experiment and are not a universal production recommendation. Cleanup explicitly deletes only its temporary metrics volume, whose retained samples are not needed after the assertions.

Alert on the contract and the observation path

Use separate signals for scrape availability, application readiness, and user-visible success. Give custom metrics precise definitions and bounded label sets. If exporting Docker health, document the exporter, target identity, and update delay, then verify that the metric actually reflects the engine state rather than relying on a coincidentally similar name.

Distinguish a failed scrape from a missing series or removed target. An empty query result is not automatically the same as a zero sample; discovery and staleness rules affect what is present. Alert rules should account for the intended target inventory and acceptable observation gaps.

Finally, test the action separately from the signal. Verify alert routing, traffic removal, or recovery behavior in the system that owns it. Good monitoring preserves these distinctions so operators can tell whether the service failed, the measurement path failed, or both did.

Sources

Compose healthcheck and restart settings, container probe configuration and service restart policy, checked 2026-09-12.
Prometheus jobs and instances, the up series describing scrape success, checked 2026-09-12.
Prometheus scrape configuration, scrape targets, paths, intervals, and timeouts, checked 2026-09-12.
Docker restart policies, restart behavior for container termination, checked 2026-09-12.

Docker rebuilds reuse recorded work, including stale work

Sun, 20 Feb 2028 00:00:00 +0000

Docker’s build cache stores results of build operations so a later build can reuse work whose relevant inputs still match. A cache hit does not mean the command ran again and produced identical output. It means the builder accepted an existing result. That distinction matters for package installation, generated timestamps, remote downloads, and any operation whose output could change without a corresponding change to its declared inputs.

When it breaks

An unchanged RUN apt-get update can reuse yesterday’s result without contacting a repository. The builder does not routinely inspect that remote repository to decide whether a cached RUN remains fresh. A successful cached build therefore provides no independent evidence that packages received the newest available security updates. Refresh policy must be deliberate and compatible with the project’s reproducibility requirements.

COPY has different inputs from a plain command string. File contents and relevant metadata participate in its cache calculation, but source modification time alone does not. A deployment script that merely touches a file can leave the dependent RUN cached. Conversely, changing an executable permission can invalidate a COPY even when the text remains identical, causing dependent work to execute again.

The base image, build arguments used by a step, and preceding dependencies also matter. Cache invalidation follows dependencies; treating a complex multistage build as one unavoidable linear rebuild can be misleading. In the simple single-stage chain below, changing the copied file invalidates the following timestamp-producing RUN, while the fixed base filesystem layers remain reusable.

Check it quickly

Run the script with host Python and access to a disposable Docker builder. Pull the named digest first if it is unavailable locally. The recorded run used Docker 29.3.1, Buildx 0.32.1, and a Linux arm64 Python 3.11.16 base. Builds and containers disable networking. Temporary tags are removed, but the script deliberately leaves unrelated images and builder caches alone:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,shlex,subprocess,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(["docker",*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def inspect(image):return json.loads(docker("image","inspect",image).stdout)[0]

name='xbcatm-cache-'+uuid.uuid4().hex[:10];tag=name+':probe'
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);app=root/'app.txt';app.write_text('initial '+name);app.chmod(0o644)
 build_code="from pathlib import Path;import time;Path('/built-at').write_text(str(time.time_ns()))"
 (root/'Dockerfile').write_text(f'FROM {BASE}\nCOPY app.txt /app.txt\nRUN python -c {shlex.quote(build_code)}\n')
 def build(*options):return docker('build','--network','none','--progress','plain',*options,'-t',tag,str(root))
 def stamp():return docker('run','--rm','--network','none',tag,'python','-c',"from pathlib import Path;print(Path('/built-at').read_text())").stdout.strip()
 build();initial=stamp();base_layers=inspect(BASE)['RootFS']['Layers']
 result=build();assert stamp()==initial and 'CACHED' in result.stderr
 timestamp=app.stat().st_mtime_ns+10_000_000_000;os.utime(app,ns=(timestamp,timestamp))
 build();assert stamp()==initial
 print('unchanged inputs reuse the RUN result; changing only source mtime also leaves that result cached')
 app.chmod(0o755);build();mode_changed=stamp();assert mode_changed!=initial
 app.write_text('changed '+name);build();content_changed=stamp();assert content_changed!=mode_changed
 print('changing copied-file permissions or content invalidates COPY and reruns the dependent RUN')
 build('--no-cache');assert stamp()!=content_changed
 assert inspect(tag)['RootFS']['Layers'][:len(base_layers)]==base_layers
 print('no-cache reruns the instruction while this pinned base retains the same filesystem layer prefix')
finally:docker('image','rm',tag,check=False)

The first build writes a nanosecond timestamp into /built-at. A second identical build reports a cached operation and returns exactly the same timestamp. This is stronger evidence than comparing elapsed build times: a quick build could still have executed the command, while the retained timestamp identifies the already-recorded result that was reused.

The script then advances only the source file’s modification time. The timestamp inside the image remains unchanged. This demonstrates the documented mtime exclusion directly; it does not imply that all filesystem metadata is ignored. The following change from mode 0644 to 0755 causes a new timestamp, establishing that the executable-bit difference affects this COPY dependency.

Changing the file’s text produces another timestamp. The test uses a uniquely named input so an old experiment cannot accidentally supply the initial application result. Its assertions compare actual image contents after each build, rather than treating a generic CACHED message somewhere in the build log as proof that the particular operation under discussion was reused.

Finally, --no-cache makes the RUN execute again with unchanged application inputs. Inspection still shows the same pinned base-layer prefix. This option disables reuse for build instructions; it does not replace an explicitly selected base digest with a newer one. Obtaining a changed base requires choosing the intended artifact, and floating-tag refresh has its own pull policy.

Make freshness an explicit input

Place stable dependency inputs before frequently edited source, and use narrow COPY instructions so unrelated files do not become dependencies. Pin package versions or verified artifacts when repeatability is required. For intentional dependency refreshes, update those declarations and review the resulting artifact instead of assuming that a cache miss is itself a security policy.

Secret mounts need separate attention: changing a secret’s contents does not automatically invalidate the build cache. If the secret controls a fetch whose output must be refreshed, use a documented nonsecret revision input or explicit invalidation. Never embed the credential in an argument merely to force a new cache key.

Treat cache-hit logs as performance evidence and executed tests as behavioral evidence. Record the final artifact identity, test it, and retain enough build inputs to explain how it was produced. A fast rebuild is useful only when the reused work is still the work the release intends to ship.

Sources

Docker build cache invalidation, cache inputs, modification times, RUN reuse, and explicit invalidation, checked 2026-09-12.
Dockerfile reference, COPY metadata, RUN instructions, and build arguments, checked 2026-09-12.

ENTRYPOINT and CMD define defaults that operators can override

Sun, 03 Oct 2027 00:00:00 +0000

An image’s ENTRYPOINT and CMD contribute to the command used when a container is created. With an exec-form ENTRYPOINT, CMD commonly supplies default arguments appended to that entrypoint. Without an entrypoint, CMD can supply the executable and its arguments. These are startup defaults, not an enforcement boundary: an authorized operator can replace the entrypoint or arguments when creating a container.

When it breaks

Trailing arguments in docker run image ... replace CMD; they are not automatically added after CMD’s existing defaults. The exec-form entrypoint remains unless explicitly overridden. A user expecting to append one flag can accidentally discard required defaults, so command-line tools should document which values are optional, which must be repeated, and how their own argument parser treats overrides.

Exec form uses a JSON array and does not implicitly run a shell. A string such as $DEMO reaches the program literally instead of receiving shell variable expansion. Pipes, redirects, and wildcards likewise need an actual interpreter if that is the intended behavior. Shell form changes argument and process semantics, so switching forms is more than a cosmetic syntax choice.

A startup wrapper can also hide the service behind a shell that never forwards the container’s stop signal. Docker normally signals the main container process, waits for the configured timeout, then forces termination if necessary. A service’s correct SIGTERM handler is ineffective if the signal is delivered only to another process that does not pass it on.

Check it quickly

Run this script with host Python and access to a Docker builder after obtaining the pinned base. It was executed with Docker 29.3.1 and Linux arm64 Python 3.11.16. The fixture creates isolated temporary containers with networking disabled, waits for each service’s readiness output, tests stopping, and removes its containers and image tag:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,time,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def inspect(name):return json.loads(docker('inspect',name).stdout)[0]

name='xbcatm-entrypoint-'+uuid.uuid4().hex[:10];tag=name+':probe';containers=[name+'-exec',name+'-child']
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory)
 (root/'args.py').write_text('import json,sys\nprint(json.dumps(sys.argv[1:]))\n')
 (root/'service.py').write_text('import json,os,signal\ndef stop(signum,frame):\n print("stopped",flush=True)\n raise SystemExit(0)\nsignal.signal(signal.SIGTERM,stop)\nprint(json.dumps({"pid":os.getpid()}),flush=True)\nwhile True:signal.pause()\n')
 (root/'exec.sh').write_text('#!/bin/sh\nexec python /service.py "$@"\n')
 (root/'child.sh').write_text('#!/bin/sh\npython /service.py "$@" &\nwait $!\n')
 (root/'Dockerfile').write_text(f'FROM {BASE}\nCOPY args.py service.py exec.sh child.sh /\nRUN chmod +x /exec.sh /child.sh\nENV DEMO=expanded\nENTRYPOINT ["python", "/args.py"]\nCMD ["default", "$DEMO"]\n')
 docker('build','--network','none','-t',tag,str(root))
 assert json.loads(docker('run','--rm','--network','none',tag).stdout)==['default','$DEMO']
 assert json.loads(docker('run','--rm','--network','none',tag,'changed').stdout)==['changed']
 assert docker('run','--rm','--network','none','--entrypoint','python',tag,'-c','print("override")').stdout.strip()=='override'
 assert docker('run','--rm','--network','none','--entrypoint','/bin/sh',tag,'-c','printf "%s" "$DEMO"').stdout=='expanded'
 print('CMD supplies literal default arguments; trailing arguments replace it; --entrypoint replaces the executable')
 for container,wrapper in zip(containers,['/exec.sh','/child.sh']):
 docker('run','-d','--name',container,'--network','none','--entrypoint',wrapper,tag)
 deadline=time.monotonic()+10
 while time.monotonic()<deadline:
 logs=docker('logs',container).stdout
 if logs.strip():break
 time.sleep(.05)
 else:raise AssertionError('service did not become ready')
 pid=json.loads(logs.splitlines()[0])['pid']
 assert (pid==1)==(wrapper=='/exec.sh')
 docker('stop','--time','1',container)
 state=inspect(container)['State'];logs=docker('logs',container).stdout
 if wrapper=='/exec.sh':assert state['ExitCode']==0 and 'stopped' in logs
 else:assert state['ExitCode']==137 and 'stopped' not in logs
 print('exec makes the handler PID 1 and stops cleanly; the non-forwarding shell wrapper leaves its child unnotified until forced termination')
finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',tag,check=False)

The first image command prints its received argument list. With no override it receives default and the literal $DEMO, even though the image defines DEMO as expanded. Supplying changed after the image name replaces both default arguments, and the program receives exactly one argument. These assertions inspect the final argument vector rather than guessing from Dockerfile syntax.

Using --entrypoint python and explicit Python arguments runs an entirely different command that prints override. Using /bin/sh -c explicitly expands $DEMO and prints expanded. Together, these cases show both that ENTRYPOINT is overridable and that expansion belongs to the invoked shell, not to a hidden expansion step in exec-form command construction.

The second part compares two wrapper scripts launching the same service. The exec wrapper replaces itself with Python, which reports PID 1 inside the container. Its installed SIGTERM handler prints stopped and exits with status zero when Docker stops the container. The fixture verifies the message and the recorded exit status.

The other wrapper starts Python as a child and waits, without forwarding signals. The service reports a PID other than one. Stopping that container with a one-second timeout produces exit status 137 in the recorded environment, and the service never prints stopped. The forced termination is observed directly; no conclusion is inferred merely from the presence of a shell script.

PID 1 has special signal and child-reaping responsibilities in a Linux PID namespace. An exec wrapper makes the service the signal recipient, but it does not automatically implement application cleanup, drain in-flight work, or reap arbitrary descendants. The service still needs appropriate behavior, or a deliberately chosen init/supervision arrangement when its process model requires one.

Treat startup and shutdown as one interface

Use valid JSON arrays with double-quoted strings for exec-form instructions. Keep the executable and stable arguments clear, and document how callers replace defaults. If initialization requires a shell wrapper, finish with an appropriate exec when the wrapper’s job is to hand control to one foreground service. Quote forwarded arguments so their boundaries remain intact.

Test the actual deployed launch path, including any orchestration overrides. A correct image entrypoint can be replaced by a configuration that introduces another shell, and an overridden command may omit necessary initialization. Inspect the resulting process tree and exercise termination with representative active work.

Choose the stop signal and grace period according to the service’s contract. Record readiness, successful cleanup, and exit status separately. A container that eventually disappears after SIGKILL has stopped, but that observation alone does not prove requests completed or buffered state was safely handled.

Sources

Dockerfile ENTRYPOINT and CMD, exec and shell forms, argument combination, and entrypoint overrides, checked 2026-09-12.
Docker container stop, stop signals, timeout, and forced termination, checked 2026-09-12.
pid_namespaces(7), namespace-init signal behavior and termination of namespace members, checked 2026-09-12.

Image tags can move while existing containers keep their image

Tue, 24 Aug 2027 00:00:00 +0000

An image tag is a readable name associated with an image reference. Unless a registry policy prevents reassignment, the same tag can later identify another artifact. Tags such as v1.4 and latest differ in naming convention, not in an inherent guarantee of immutability. Reproducing a deployment requires recording the artifact actually selected, together with the platform and runtime configuration that determined its behavior.

When it breaks

The word latest does not ask a registry to find its newest image or highest semantic version. It is the conventional default tag when a reference omits one. Publishing or tagging another version does not automatically move latest; someone or some automation must update that specific reference. Treating every push as an implicit latest update gives an incorrect picture of release behavior.

A tag’s local resolution can also differ from its current registry resolution. Pull policy, local image availability, and orchestration behavior determine when the registry is consulted. Two machines running a readable tag can therefore use different artifacts. Recording only the command text leaves uncertainty about what each machine actually fetched and which cached reference it retained.

Moving a tag does not rewrite the image selected by an already-created container. Restarting that container retains its configured image and writable state rather than recreating it from the tag’s new target. An update procedure that retags an image and merely restarts existing containers can therefore appear successful while continuing to run the old application.

Check it quickly

Run this Python script with a Docker builder and the pinned base available. The recorded environment used Docker 29.3.1 on Linux arm64. All tags are unique local names; nothing is pushed to a registry. Containers use no network, and cleanup removes only the experiment’s container and tags:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def inspect(name,kind='image'):return json.loads(docker(kind,'inspect',name).stdout)[0]

name='xbcatm-tags-'+uuid.uuid4().hex[:10];tags=[name+':one',name+':two',name+':release',name+':latest'];container=name+'-old'
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory)
 (root/'Dockerfile').write_text(f'FROM {BASE}\nCOPY version.txt /version.txt\nCMD ["cat", "/version.txt"]\n')
 for tag,version in zip(tags,['one','two']):
 (root/'version.txt').write_text(version+'\n');docker('build','--network','none','-t',tag,str(root))
 one=inspect(tags[0])['Id'];two=inspect(tags[1])['Id'];assert one!=two
 docker('image','tag',tags[0],tags[2]);docker('image','tag',tags[0],name)
 assert inspect(tags[2])['Id']==inspect(tags[3])['Id']==one
 docker('create','--name',container,'--network','none',tags[2])
 docker('image','tag',tags[1],tags[2])
 assert inspect(tags[2])['Id']==two and inspect(tags[3])['Id']==one
 assert inspect(container,'container')['Image']==one
 assert docker('start','--attach',container).stdout.strip()=='one'
 assert docker('run','--rm','--network','none',tags[2]).stdout.strip()=='two'
 assert docker('run','--rm','--network','none',tags[3]).stdout.strip()=='one'
 print('moving release changes newly created containers; the existing container still runs its original image')
 print('omitting a tag selected latest, but updating release did not update latest')
 docker('image','rm',tags[2]);assert inspect(tags[1])['Id']==two
 print('removing one tag leaves another reference to the same image intact; all operations stayed local')
finally:
 docker('rm','-f',container,check=False);docker('image','rm',*tags,check=False)

The script builds two images whose /version.txt contents are one and two. Their image identities differ. It points a release tag at the first image and tags that same image using a repository name with no explicit tag. Inspection confirms that the omitted tag became latest, and both references initially select the first image.

Before moving release, the script creates a container from it without starting the container. It then moves release to the second image. The release tag now identifies the second artifact, while latest still identifies the first. This isolates tag reassignment from the independent default-tag convention and demonstrates that related names do not automatically move together.

Starting the existing container prints one. Creating a new container from release prints two, while creating one from latest prints one. The container inspection also retains the original selected image identity. These assertions compare actual execution results, so the distinction does not depend solely on interpreting a table of tag names.

Removing the release tag leaves the second image available through its two tag. A tag is a reference, and removing one reference does not necessarily delete shared content or every other name. Image reclamation depends on remaining references and the engine’s storage rules; this experiment does not prune caches or measure physical space recovery.

The fixture compares identities returned by the current engine without assuming that every inspect Id universally means an OCI configuration digest. Modern image stores can expose different object levels. Registry indexes, platform manifests, image configurations, and layer blobs each have their own digests, so the object type and reference context must accompany an identity claim.

Promote the artifact that was tested

For registry deployments, record a repository-qualified digest reference from the publication workflow. A digest can identify an image index containing multiple platform-specific manifests; selecting the intended platform still matters. Retain the corresponding artifact and required blobs so a later rollback can actually retrieve them. A recorded hash cannot recover content that is no longer available.

Use tags as human navigation when useful, and enforce registry immutability rules if the release process promises that version tags never move. Verify the policy rather than inferring it from a version-shaped name. Digest pinning also does not prove who built an artifact or whether it is safe; provenance and behavioral validation answer those questions.

After deployment, compare the running workload’s selected artifact and expected behavior with the release record. An updated tag, a successful pull, and a running container are separate observations. The release is established by connecting them to the exact artifact that passed validation.

Sources

Docker image tag, local tag creation and the default latest tag, checked 2026-09-12.
Docker image pull, pulling repository artifacts by tag or digest, checked 2026-09-12.
OCI image index 1.1.1, an index referencing platform-specific image manifests, checked 2026-09-12.
OCI image manifest 1.1.1, manifest references to configuration and layer objects, checked 2026-09-12.

A Compose project combines declarations with external inputs

Wed, 18 Aug 2027 00:00:00 +0000

Docker Compose turns service declarations into containers, networks, and storage attachments managed as a project. The file describes useful relationships: which image runs a service, how configuration reaches it, and which storage or network it uses. Reproducing a stack still depends on external inputs such as image availability, variable values, host capabilities, and existing data. A versioned file makes those inputs reviewable only when they are identified.

When it breaks

The same Compose text can resolve differently when the invoking shell supplies different variables. An env file participates in interpolation, but it is not automatically equivalent to the environment eventually injected into every container. Service-level environment declarations and env_file settings have their own roles. Inspect the rendered configuration and the running service rather than assuming a value in one file must be the final value.

Network addresses are another trap. Services on a Compose network can resolve one another by service name and use the container port. Published host ports serve a different path. Pointing a client container at localhost usually sends the request back to that client, not to the sibling API. Hardcoding a container IP also makes replacement more fragile than using the stable service name.

Project lifecycle does not give every resource the same lifetime. A normal down operation removes the project’s containers and networks, while named volumes remain unless volume removal is requested. Recreating the network can change its identity and assigned addresses. Retained data does not mean the old network survived, and a newly created container does not imply an empty data volume.

Check it quickly

Run the script with host Python, Docker Compose, and the pinned Python image. It was executed with Compose 5.1.1 and Docker 29.3.1 on Linux arm64. The generated Compose document uses JSON syntax accepted by YAML, requires no extra Python parser, and has a unique project name. The API is published only on host loopback at an automatically assigned port:

from pathlib import Path
from tempfile import TemporaryDirectory
from urllib.request import urlopen
import json,os,subprocess,time,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def run(args,check=True,env=None):
 result=subprocess.run(args,text=True,capture_output=True,timeout=120,env=env)
 if check:assert result.returncode==0,result.stderr
 return result
def docker(*args,check=True):return run(['docker',*args],check)

name='xbcatm-project-'+uuid.uuid4().hex[:10];environment=os.environ.copy();environment.pop('TASK_COMPOSE_GREETING',None)
with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);file=root/'compose.yaml'
 def compose(*args,check=True,env=None):return run(['docker','compose','-p',name,'--env-file',str(root/'.env'),'-f',str(file),*args],check,env or environment)
 def container(service):return compose('ps','-q',service).stdout.strip()
 def inspect(identity):return json.loads(docker('inspect',identity).stdout)[0]
 def request():
 port=inspect(container('api'))['NetworkSettings']['Ports']['8080/tcp'][0]
 assert port['HostIp']=='127.0.0.1'
 deadline=time.monotonic()+10
 while time.monotonic()<deadline:
 try:
 with urlopen('http://127.0.0.1:'+port['HostPort'],timeout=1) as response:return response.read().decode()
 except OSError:time.sleep(.1)
 raise AssertionError('published endpoint unavailable')
 server='from http.server import BaseHTTPRequestHandler,HTTPServer\nimport os\nclass H(BaseHTTPRequestHandler):\n def do_GET(self):self.send_response(200);self.end_headers();self.wfile.write(os.environ["GREETING"].encode())\n def log_message(self,*args):pass\nHTTPServer(("0.0.0.0",8080),H).serve_forever()\n'
 spec={'services':{'api':{'image':BASE,'command':['python','-c',server],'environment':{'GREETING':'${TASK_COMPOSE_GREETING:-default}'},'ports':[{'target':8080,'published':'0','host_ip':'127.0.0.1'}],'volumes':['state:/state']},'client':{'image':BASE,'command':['python','-c','import time;time.sleep(180)']}},'volumes':{'state':{}}}
 file.write_text(json.dumps(spec,indent=2));(root/'.env').write_text('TASK_COMPOSE_GREETING=from-file\n')
 try:
 rendered=json.loads(compose('config','--format','json').stdout);assert rendered['services']['api']['environment']['GREETING']=='from-file'
 override=environment|{'TASK_COMPOSE_GREETING':'from-shell'}
 rendered=json.loads(compose('config','--format','json',env=override).stdout);assert rendered['services']['api']['environment']['GREETING']=='from-shell'
 compose('up','-d',env=override);assert request()=='from-shell'
 api=container('api');client=container('client');first_network=next(iter(inspect(api)['NetworkSettings']['Networks'].values()))['NetworkID']
 result=docker('exec',client,'python','-c','import urllib.request;print(urllib.request.urlopen("http://api:8080",timeout=2).read().decode())')
 assert result.stdout.strip()=='from-shell'
 docker('exec',client,'python','-c','import socket;s=socket.socket();assert s.connect_ex(("127.0.0.1",8080))!=0;s.close()')
 docker('exec',api,'python','-c','from pathlib import Path;Path("/state/record").write_text("preserved")')
 volume=next(m['Name'] for m in inspect(api)['Mounts'] if m['Destination']=='/state')
 assert volume==name+'_state'
 compose('down');assert docker('network','inspect',first_network,check=False).returncode!=0
 assert docker('volume','inspect',volume,check=False).returncode==0
 compose('up','-d',env=override);assert request()=='from-shell'
 api=container('api');assert next(iter(inspect(api)['NetworkSettings']['Networks'].values()))['NetworkID']!=first_network
 docker('exec',api,'python','-c','from pathlib import Path;assert Path("/state/record").read_text()=="preserved"')
 assert docker('exec',container('client'),'python','-c','import urllib.request;print(urllib.request.urlopen("http://api:8080",timeout=2).read().decode())').stdout.strip()=='from-shell'
 print('shell interpolation overrides .env; internal clients resolve api:8080 while localhost remains their own container')
 print('the host endpoint is published only on loopback; down/up recreates the project network and retains the named data volume')
 finally:compose('down','--volumes','--remove-orphans',check=False)

The private env file supplies TASK_COMPOSE_GREETING=from-file. The script explicitly selects that file and removes the same variable from its initial shell environment. compose config --format json resolves the service’s GREETING value to from-file. Providing from-shell in the invoking environment then changes the rendered value, establishing the tested interpolation precedence.

Starting the project with the override makes the API return from-shell. The host-side request uses the published port discovered through container inspection, and the binding is asserted to be 127.0.0.1. This verifies both the effective configuration and the intended exposure. Merely rendering a valid document would not prove the container was started with that value or port binding.

The client service successfully requests http://api:8080 through Compose’s network and gets the same response. A connection attempt to its own loopback port 8080 fails. These two observations distinguish service-name discovery from host publishing and container-local loopback. No fixed container IP is embedded in the application request.

The API writes a marker to its named state volume. After ordinary compose down, inspection confirms the old network no longer exists while the named volume still does. Bringing the project up again creates a different network identity. The marker remains readable, and the client can still reach the API using the unchanged service name.

The fixture’s final cleanup includes --volumes because all of its data is disposable. That option has a different destructive scope from the earlier ordinary down used to demonstrate retention. In a real environment, apply it only when removal of those volumes is intended; it should not be appended casually to every restart or troubleshooting command.

Record the project inputs that matter

Keep required variable names, image digests, storage ownership, and exposed ports explicit. Render configuration in the intended execution environment before starting the stack, but avoid publishing rendered output containing credentials. Pinning an image reduces one source of drift; it does not freeze host architecture, mounted files, runtime configuration, or persistent database contents.

Use separate project names for independent local or CI runs when resources should be isolated. Explicitly named external networks or volumes can intentionally cross that boundary, so review those references rather than assuming project naming isolates every attachment. Port bindings can also conflict across otherwise separate projects.

Compose can support a managed single-host deployment as well as local testing. Its suitability depends on operational requirements, not a blanket ban on production use. Host redundancy, rollout coordination, backup, readiness, and monitoring still need deliberate solutions. The value of the declaration is a clear, testable resource model with known limits.

Sources

Compose services reference, service configuration, environment, ports, and volume mounts, checked 2026-09-12.
Networking in Compose, project networks and service-name discovery, checked 2026-09-12.
Compose variable interpolation, shell and env-file interpolation precedence, checked 2026-09-12.
Docker compose down, network removal and optional volume removal, checked 2026-09-12.

Copy dependency inputs before application code

Tue, 17 Aug 2027 00:00:00 +0000

Dockerfile order determines which earlier inputs an expensive build operation depends on. If application source is copied before dependency installation, changing that source can make the installation run again even when the dependency declaration is unchanged. Copying the dependency inputs first, installing them, and copying application code afterward creates a smaller dependency boundary that matches the work the installer actually needs.

When it breaks

A broad COPY . . before package installation makes documentation, generated output, or unrelated source edits potential reasons to repeat expensive work. The problem is not that COPY is inherently slow. It is that the later operation depends on filesystem state containing changes that were unnecessary for resolving and installing the selected dependencies.

Copying only a manifest is insufficient when the install also consumes a lockfile, local package source, patches, workspace metadata, or a package-manager configuration file. Omitting a real dependency can preserve a cached result that no longer matches the intended build. The correct boundary includes every actual installer input, not simply whichever single filename appears in a common Dockerfile example.

Cache ordering also does not fix mutable dependencies. An unchanged install command may retain an old result while a clean build resolves different remote content. Pinning versions and verifying hashes addresses artifact selection; controlling COPY order addresses reuse. Both concerns matter, and making a build faster does not establish that its package graph is reproducible or current.

Check it quickly

Run this host Python script with a Docker builder and the pinned base available. It downloads two public idna wheels over HTTPS and verifies their fixed SHA-256 values before any build. Container builds use no network, and pip installs with required hashes from the local wheel directory. The recorded environment used Docker 29.3.1 and Python 3.11.16 inside the Linux arm64 base:

from pathlib import Path
from tempfile import TemporaryDirectory
from urllib.request import urlopen
import hashlib,json,subprocess,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
WHEELS={
 '3.10':('https://files.pythonhosted.org/packages/76/c6/c88e154df9c4e1a2a66ccf0005a88dfb2650c1dffb6f5ce603dfbd452ce3/idna-3.10-py3-none-any.whl','946d195a0d259cbba61165e88e65941f16e9b36ea6ddb97f00452bae8b1287d3'),
 '3.11':('https://files.pythonhosted.org/packages/0e/61/66938bbb5fc52dbdf84594873d5b51fb1f7c7794e9c0f5bd885f30bc507b/idna-3.11-py3-none-any.whl','771a87f49d9defaf64091e6e6fe9c18d4833f140bd19464795bc32d966ca37ea')}
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result

name='xbcatm-order-'+uuid.uuid4().hex[:10];tags=[name+':early',name+':late']
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'wheels').mkdir()
 for version,(url,digest) in WHEELS.items():
 with urlopen(url,timeout=40) as response:data=response.read()
 assert hashlib.sha256(data).hexdigest()==digest
 (root/'wheels'/url.rsplit('/',1)[1]).write_bytes(data)
 install='RUN pip install --no-index --no-deps --require-hashes --find-links=/wheels -r /app/requirements.txt && python -c "import time;open(\'/installed\',\'w\').write(str(time.time_ns()))"\n'
 common=f'FROM {BASE}\nLABEL example.run="{name}"\nCOPY wheels /wheels\n'
 (root/'early.Dockerfile').write_text(common+'COPY requirements.txt app.py /app/\n'+install+'CMD ["python", "/app/app.py"]\n')
 (root/'late.Dockerfile').write_text(common+'COPY requirements.txt /app/\n'+install+'COPY app.py /app/\nCMD ["python", "/app/app.py"]\n')
 def requirement(version):
 (root/'requirements.txt').write_text(f'idna=={version} --hash=sha256:{WHEELS[version][1]}\n')
 def application(message):
 (root/'app.py').write_text(f'import idna\nprint({message!r},idna.__version__,idna.encode("bücher.example").decode())\n')
 def build():
 for tag,kind in zip(tags,['early','late']):docker('build','--network','none','-f',str(root/(kind+'.Dockerfile')),'-t',tag,str(root))
 def marks():
 return [docker('run','--rm','--network','none',tag,'cat','/installed').stdout for tag in tags]
 requirement('3.10');application('first');build();first=marks()
 application('second');build();second=marks()
 assert first[0]!=second[0] and first[1]==second[1]
 for tag in tags:assert docker('run','--rm','--network','none',tag).stdout.strip()=='second 3.10 xn--bcher-kva.example'
 print('editing only application code reruns the early-copy dependency install but reuses the late-copy install')
 requirement('3.11');build();third=marks()
 assert all(a!=b for a,b in zip(second,third))
 for tag in tags:assert docker('run','--rm','--network','none',tag).stdout.strip()=='second 3.11 xn--bcher-kva.example'
 print('changing the hashed requirement reruns both installs and actually changes idna from 3.10 to 3.11')
finally:docker('image','rm',*tags,check=False)

The early-copy Dockerfile copies both requirements.txt and app.py before pip runs. The late-copy version copies the requirement first, runs the same installation command, and copies the application afterward. Both include the verified wheel directory before installation because those wheel files are genuine inputs. A marker written immediately after pip records whether that installation operation executed.

The initial application uses idna 3.10 to encode bücher.example. Both images run the actual installed library and produce the expected ASCII domain xn--bcher-kva.example. This exercises a real third-party dependency, rather than replacing dependency installation with a sleep or treating a successful image build as proof that the application can import its package.

After only the application message changes from first to second, both images print the new message with version 3.10. The early-copy image gets a different installation marker, while the late-copy image keeps its original marker. The output proves the code update reached both images; the marker comparison identifies which dependency work was unnecessarily repeated.

Next, the requirement changes to idna 3.11 with its matching hash. Both installation markers change, and both applications report version 3.11 with the same expected domain conversion. This is the necessary opposite case: a genuine dependency declaration change must invalidate the install in both layouts. A layout that ignored this input would be fast but incorrect.

The experiment supplies both wheels from the beginning so changing the requirement does not simultaneously alter the wheel directory. That keeps the cause of the second invalidation explicit. Temporary images and files are cleaned up without pruning unrelated builder state. Installation timestamps are test instrumentation, not a recommended input for reproducible production artifacts.

Match the cache boundary to the package boundary

Review what the installer reads, then group those inputs before the install operation. Keep unrelated files outside that COPY scope and use .dockerignore to exclude material the build never needs. An ignored file cannot later satisfy a required COPY, so check actual build-context contents when a supposedly present dependency disappears.

Package-manager cache mounts can reduce downloads when an install genuinely must rerun, but they do not eliminate the need for correct invalidation or verified outputs. Dependency installation and download-cache reuse are different layers of optimization. Test the resulting application after changes to either one.

Measure improvement on representative builds: unchanged inputs, source-only edits, and dependency upgrades. Retain correctness assertions for all three cases. A useful Dockerfile makes the necessary work repeat and the unnecessary work reusable, with enough evidence to tell those cases apart.

Sources

Optimize Docker cache usage, stable dependency layers and narrow COPY operations, checked 2026-09-12.
Docker cache invalidation, how changed inputs invalidate dependent work, checked 2026-09-12.
pip install reference, offline wheel installation and required hashes, checked 2026-09-12.

Deleting a file in a later image layer does not erase its bytes

Mon, 16 Aug 2027 00:00:00 +0000

Container image layers describe filesystem changes applied in order. A layer can add or replace a file, or record that an earlier path should disappear from the merged filesystem. That final view is what a normal container process sees, but it is not the complete history of bytes distributed with the image. Inspecting only the running container can therefore miss data retained in earlier layers.

When it breaks

Copying a credential into an image and deleting it in a later RUN does not remove the credential from the earlier immutable layer. Anyone who can obtain the relevant layer may recover the original bytes. The same mechanism explains why deleting a large build archive in a later instruction can leave the image’s distributed content unnecessarily large even though the final path is absent.

Layer changes are filesystem changesets, not necessarily compact binary differences. Replacing a large file can place its replacement content in another layer. Deletion uses a whiteout marker to suppress an earlier path when layers are applied. OCI also defines opaque-directory behavior for hiding lower directory contents, so implementing layer application as ordinary repeated tar extraction is insufficient.

Not every Dockerfile instruction creates a filesystem layer. ENV, LABEL, and CMD change image configuration and can add history entries marked empty_layer. Counting Dockerfile lines or history records therefore does not reliably count root filesystem layers. Configuration changes still matter to execution and artifact identity even when they add no application file bytes.

Check it quickly

Use Python with access to a Docker builder and the pinned base digest. The recorded run used Docker 29.3.1 with its containerd image store and a Linux arm64 Python 3.11.16 base. The marker is deliberately public. The script builds and saves one temporary image, inspects archive members without extracting them into the host filesystem, and removes its own tag:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,tarfile,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(["docker",*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def inspect(image):return json.loads(docker("image","inspect",image).stdout)[0]

name='xbcatm-whiteout-'+uuid.uuid4().hex[:10];tag=name+':probe';marker=b'PUBLIC_DEMONSTRATION_BYTES\n'
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'original.txt').write_bytes(marker)
 (root/'Dockerfile').write_text(f'FROM {BASE}\nCOPY original.txt /demo/original.txt\nRUN rm /demo/original.txt\nENV EXAMPLE_MODE=demo\nLABEL example.purpose="public fixture"\nCMD ["python", "-c", "from pathlib import Path;assert not Path(\'/demo/original.txt\').exists();print(\'absent\')"]\n')
 docker('build','--network','none','-t',tag,str(root))
 assert docker('run','--rm','--network','none',tag).stdout.strip()=='absent'
 archive=root/'image.tar';docker('image','save','-o',str(archive),tag)
 with tarfile.open(archive) as outer:
 manifests=json.load(outer.extractfile('manifest.json'));assert len(manifests)==1
 manifest=manifests[0];config=json.load(outer.extractfile(manifest['Config']))
 assert len(manifest['Layers'])==len(config['rootfs']['diff_ids'])
 assert len(manifest['Layers'])==len(inspect(BASE)['RootFS']['Layers'])+2
 found_data=False;found_whiteout=False
 for layer in manifest['Layers'][-2:]:
 with tarfile.open(fileobj=outer.extractfile(layer),mode='r:*') as contents:
 for entry in contents:
 path=entry.name.removeprefix('./')
 if path=='demo/original.txt':found_data=contents.extractfile(entry).read()==marker
 if path=='demo/.wh.original.txt':found_whiteout=True
 assert found_data and found_whiteout
 assert all(item.get('empty_layer') for item in config['history'][-3:])
 print('the final container hides the removed file, but the older layer still contains the exact public marker')
 print('the deletion layer contains a whiteout; ENV, LABEL and CMD add metadata history without filesystem layers')
finally:docker('image','rm',tag,check=False)

The Dockerfile copies original.txt into /demo, then removes it in a later RUN. The configured container command asserts the path is absent and prints absent. That confirms the merged runtime view. It does not establish that the bytes are absent from the image archive, so the next part examines the saved representation separately.

The archive’s manifest.json identifies its configuration and ordered layer files. The script reads the configuration’s rootfs.diff_ids and checks that the number of filesystem layers matches the manifest. It then inspects the final two layer archives, which correspond to the COPY addition and the subsequent deletion in this particular build.

The addition layer still contains the exact PUBLIC_DEMONSTRATION_BYTES marker. The deletion layer contains demo/.wh.original.txt. That whiteout instructs layer application to hide the lower file; it does not edit the original layer in place. Both facts are asserted directly, so an absent file in the container cannot be mistaken for evidence of secure erasure.

The final three configuration-history entries describe ENV, LABEL, and CMD and are marked as having no filesystem layer. The image has two additional filesystem layers above its base, not five. This is a concrete counterexample to the claim that each Dockerfile instruction necessarily appends another filesystem diff.

Archive representation and runtime storage are different interfaces. The experiment inspects the image-save format produced by the recorded Docker version and OCI changeset semantics. It does not assume that a snapshotter must store its internal deletion markers as the same visible .wh. filenames, nor does it measure how a particular host allocates physical blocks.

Exclude unnecessary bytes before they enter an image

Use an appropriate build context and ignore rules to keep unrelated files out of COPY inputs. Pass build credentials through the builder’s secret mechanism, and ensure commands do not write those credentials into retained outputs. If a real credential entered a published layer, rotate it and handle retained copies; a later deletion layer is not remediation.

Create and remove disposable package indexes or temporary archives within the same build operation when they are needed only during that operation. A separate final stage can copy the required runtime artifacts without inheriting the builder’s filesystem layers. Still inspect exported caches and copied outputs, because earlier build material can remain in the builder or escape through another export.

Judge the result by the actual final artifact and its intended runtime dependencies. Fewer layers alone does not guarantee smaller downloads, faster starts, or safer content. The useful question is which bytes are distributed, which paths become visible, and whether every retained file has a justified role.

Sources

OCI filesystem layer changesets 1.1.1, file additions, modifications, removals, and whiteout application, checked 2026-09-12.
OCI image configuration 1.1.1, rootfs DiffIDs and history entries without filesystem layers, checked 2026-09-12.
Dockerfile reference, COPY, RUN, ENV, LABEL, and CMD behavior, checked 2026-09-12.

Docker healthchecks report a state; recovery needs a consumer

Mon, 05 Jul 2027 00:00:00 +0000

A Docker HEALTHCHECK runs a configured command inside a container and records the result as health state. It adds an observation beyond whether the main process is running. The meaning of that observation depends on what the command tests: a process check, an HTTP response, or an application-specific readiness condition. Docker does not infer a universal readiness contract from the presence of the instruction.

When it breaks

A command that always exits zero produces reassuring health reports even when the application cannot serve useful work. A shell sequence can accidentally hide an earlier failure by ending with a successful command. Conversely, a probe that depends on an optional external service can mark a usable application unhealthy and encourage unnecessary recovery actions by whatever system consumes the result.

Standalone Docker restart policies react to container termination, not merely to an unhealthy health status. Adding restart=always does not make Docker restart a still-running application when its probe fails. Likewise, a healthcheck does not automatically remove a container from every proxy’s routing table. A controller or proxy must explicitly consume that state and implement the intended action.

Probe timing is another source of incorrect expectations. The retry threshold counts consecutive failures, while startup grace changes which early failures count. A successful probe during startup can end that initial grace behavior. Detection time depends on command duration, interval, timeout, and configuration; it is not simply a promise that one failed request instantly stops traffic.

Check it quickly

Run this script using host Python and Docker with the pinned base already available. The recorded environment used Docker 29.3.1 and Linux arm64 Python 3.11.16. It creates one temporary image and container, disables networking outside the container, and uses its own loopback HTTP server. The only state changed is a harmless readiness file inside the experiment:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,time,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],text=True,capture_output=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def inspect(container):return json.loads(docker('inspect',container).stdout)[0]
def wait_for(container,wanted):
 deadline=time.monotonic()+25
 while time.monotonic()<deadline:
 state=inspect(container)
 if state['State']['Health']['Status']==wanted:return state
 time.sleep(.1)
 raise AssertionError('health did not reach '+wanted)

name='xbcatm-health-'+uuid.uuid4().hex[:10];tag=name+':probe'
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory)
 (root/'server.py').write_text('from http.server import BaseHTTPRequestHandler,HTTPServer\nfrom pathlib import Path\nclass Handler(BaseHTTPRequestHandler):\n def do_GET(self):\n code=200 if Path("/ready").exists() else 503\n self.send_response(code);self.end_headers();self.wfile.write(str(code).encode())\n def log_message(self,*args):pass\nHTTPServer(("127.0.0.1",8080),Handler).serve_forever()\n')
 probe='import urllib.request;urllib.request.urlopen("http://127.0.0.1:8080/health",timeout=.5).read()'
 (root/'Dockerfile').write_text(f'FROM {BASE}\nCOPY server.py /server.py\nHEALTHCHECK --interval=1s --timeout=1s --retries=2 --start-period=5s CMD ["python", "-c", {json.dumps(probe)}]\nCMD ["python", "/server.py"]\n')
 docker('build','--network','none','-t',tag,str(root))
 began=time.monotonic()
 docker('run','-d','--name',name,'--network','none','--restart','always',tag)
 initial=inspect(name);assert initial['State']['Health']['Status']=='starting'
 first=wait_for(name,'unhealthy');assert first['State']['Running'] and first['RestartCount']==0
 assert first['State']['StartedAt']==initial['State']['StartedAt']
 assert first['State']['Health']['FailingStreak']>=2
 assert any(log['ExitCode']!=0 and '503' in log['Output'] for log in first['State']['Health']['Log'])
 time.sleep(max(0,12-(time.monotonic()-began)))
 stable=inspect(name);assert stable['State']['Running'] and stable['RestartCount']==0
 assert stable['State']['StartedAt']==initial['State']['StartedAt'] and stable['State']['Health']['Status']=='unhealthy'
 docker('exec',name,'python','-c','from pathlib import Path;Path("/ready").touch()')
 healthy=wait_for(name,'healthy');assert healthy['RestartCount']==0 and healthy['State']['Health']['FailingStreak']==0
 docker('exec',name,'python','-c','from pathlib import Path;Path("/ready").unlink()')
 unhealthy=wait_for(name,'unhealthy');assert unhealthy['RestartCount']==0 and unhealthy['State']['Running']
 assert unhealthy['State']['StartedAt']==initial['State']['StartedAt']
 print('starting -> unhealthy -> healthy -> unhealthy follows the real HTTP probe; failure logs contain HTTP 503')
 print('restart=always does not restart a still-running process solely because its health is unhealthy')
finally:
 docker('rm','-f',name,check=False);docker('image','rm',tag,check=False)

The server returns HTTP 503 while /ready is absent. Its Dockerfile healthcheck uses Python’s HTTP client, which raises on that status and exits unsuccessfully. The probe has a one-second interval and timeout, two retries, and a five-second start period. The container starts with restart=always, and initial inspection records health as starting.

After the counted failures, Docker reports unhealthy; the health log contains the real HTTP 503 error and the failing streak is at least two. The fixture retains the initial start timestamp, waits until at least twelve seconds after launch, and confirms the main process is still running with no restart. This avoids confusing initial startup behavior with recovery from an established unhealthy state.

Creating /ready makes the HTTP endpoint return 200. A later probe changes health to healthy and resets the failing streak to zero. The same container process remains running. That transition demonstrates actual application recovery as observed by the command, without requiring the controller to manufacture a restart simply to clear the status.

Removing the file makes the endpoint fail again, and health eventually returns to unhealthy. The start timestamp remains the same and RestartCount remains zero. The test verifies a full failure, recovery, and failure sequence, rather than observing one initial boot failure and assuming all later health behavior will be identical.

Probe exit status is the interface Docker consumes: zero means success, one indicates failure, and two is reserved by the Dockerfile contract. The Python client naturally returns a failing process status for the HTTP error. The script also inspects the diagnostic output, which helps distinguish a meaningful readiness failure from a missing probe executable or a malformed command.

Define the reaction separately

Choose a cheap, bounded check that represents the state its consumer needs. An internal HTTP check can verify application handling, but does not establish public DNS, TLS termination, proxy routing, or end-user reachability. Use external observations when those paths matter. Keep probes free of destructive actions and avoid logging credentials into retained health output.

For Compose startup dependencies, explicitly use a health-based condition when another service must wait for readiness. For runtime routing or remediation, verify the actual controller’s policy and test the action it performs. Health observation and action should be connected deliberately, with enough grace to avoid turning transient dependency trouble into restart storms.

During an incident, inspect main-process state, health state, probe logs, and restart history separately. A running unhealthy process, an exited process, and a repeatedly restarting process are different failures. Clear evidence for each state makes the recovery choice more precise.

Sources

Dockerfile HEALTHCHECK reference, probe exit codes, intervals, retries, startup grace, and health states, checked 2026-09-12.
Start containers automatically, restart policies responding to container termination, checked 2026-09-12.
Control startup order in Compose, explicit dependency conditions that consume health state, checked 2026-09-12.

Compose startup ordering needs an explicit readiness condition

Thu, 08 Apr 2027 00:00:00 +0000

Docker Compose describes a graph of services and can start them according to declared dependencies. Process startup is only one possible condition in that graph. A database or API may be running before it can accept the operation another service needs. An explicit service_healthy condition lets Compose wait for a dependency’s configured healthcheck before starting the dependent service, provided that check represents meaningful readiness.

When it breaks

The short depends_on form establishes startup ordering without establishing application readiness. Replacing it with an arbitrary sleep is fragile because initialization time changes with storage, cache state, migrations, or machine load. A sleep can be unnecessarily long on one run and too short on another. The condition should come from observable dependency behavior rather than an estimated elapsed time.

A health-based dependency condition is also not a continuous availability guarantee. After the dependent starts, the dependency can fail or restart. Compose does not automatically stop or restart an already-running dependent merely because the dependency’s health later changes. Applications still need a runtime policy for connection loss, bounded retries, and operations that may have partially completed.

The restart: true option nested under a dependency has a separate meaning from a service’s restart policy. It concerns dependent restarts after explicit Compose operations that update or restart the dependency. It should not be read as a general subscription to every health transition or every engine-level crash. These similarly named settings solve different lifecycle problems.

Check it quickly

Run the script using host Python, Docker Compose, and the pinned Python image. The recorded run used Compose 5.1.1, Docker 29.3.1, and Linux arm64 Python 3.11.16. The script writes a Compose document using JSON syntax, which is accepted as YAML, creates a unique project, and keeps all HTTP traffic on its project network. Cleanup removes only that project:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,time,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def run(args,check=True):
 result=subprocess.run(args,text=True,capture_output=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def docker(*args,check=True):return run(['docker',*args],check)
def inspect(name):return json.loads(docker('inspect',name).stdout)[0]

name='xbcatm-depends-'+uuid.uuid4().hex[:10]
with TemporaryDirectory(prefix=name+'-') as directory:
 file=Path(directory)/'compose.yaml';base=['docker','compose','-p',name,'-f',str(file)]
 def compose(*args,check=True):return run(base+list(args),check)
 def service_id(service):
 return docker('ps','-aq','--filter','label=com.docker.compose.project='+name,'--filter','label=com.docker.compose.service='+service).stdout.strip()
 def wait(predicate):
 deadline=time.monotonic()+20
 while time.monotonic()<deadline:
 result=predicate()
 if result:return result
 time.sleep(.1)
 raise AssertionError('condition did not become true')
 server='from http.server import BaseHTTPRequestHandler,HTTPServer\nfrom pathlib import Path\nclass H(BaseHTTPRequestHandler):\n def do_GET(self):self.send_response(200 if Path("/ready").exists() else 503);self.end_headers()\n def log_message(self,*args):pass\nHTTPServer(("0.0.0.0",8080),H).serve_forever()\n'
 health=['CMD','python','-c','import urllib.request;urllib.request.urlopen("http://127.0.0.1:8080",timeout=.5).read()']
 application='import urllib.request,time;urllib.request.urlopen("http://dependency:8080",timeout=2).read();print("started-after-ready",flush=True);time.sleep(180)'
 file.write_text(json.dumps({'services':{'dependency':{'image':BASE,'command':['python','-c',server],'healthcheck':{'test':health,'interval':'1s','timeout':'1s','retries':2,'start_period':'30s'}},'application':{'image':BASE,'command':['python','-c',application],'depends_on':{'dependency':{'condition':'service_healthy'}}}}},indent=2))
 process=None
 try:
 process=subprocess.Popen(base+['up','-d'],text=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
 dependency=wait(lambda:service_id('dependency'))
 wait(lambda:inspect(dependency)['State']['Running'])
 assert inspect(dependency)['State']['Health']['Status']=='starting'
 application_id=service_id('application')
 assert not application_id or not inspect(application_id)['State']['Running']
 assert process.poll() is None
 docker('exec',dependency,'python','-c','from pathlib import Path;Path("/ready").touch()')
 stdout,stderr=process.communicate(timeout=30);assert process.returncode==0,stderr
 application_id=service_id('application');wait(lambda:'started-after-ready' in docker('logs',application_id).stdout)
 original=inspect(application_id);assert original['State']['Running']
 assert inspect(dependency)['State']['Health']['Status']=='healthy'
 docker('exec',dependency,'python','-c','from pathlib import Path;Path("/ready").unlink()')
 wait(lambda:inspect(dependency)['State']['Health']['Status']=='unhealthy')
 after=inspect(application_id);assert after['State']['Running'] and after['State']['StartedAt']==original['State']['StartedAt'] and after['RestartCount']==0
 print('service_healthy keeps the dependent application stopped until the dependency probe succeeds')
 print('later dependency health failure does not stop or restart the already-running dependent application')
 finally:
 if process is not None and process.poll() is None:process.terminate();process.communicate(timeout=10)
 compose('down','--volumes','--remove-orphans',check=False)

The dependency runs an HTTP server whose response remains 503 until /ready exists. Its healthcheck tests that real endpoint. A thirty-second startup grace period gives the controller time to inspect the initial state without early failures immediately exhausting the two-failure threshold. The application declares condition: service_healthy and performs its own HTTP request when it eventually starts.

The controller launches compose up -d asynchronously and observes the dependency running with health starting. At that point the application is either not yet created or not running, and the Compose command is still waiting. The test allows either creation detail because the required contract is the dependent’s startup gate, not a particular intermediate container-allocation sequence.

Creating /ready changes the dependency’s HTTP response to 200. Compose completes startup successfully, and the application prints started-after-ready only after its own request succeeds. This connects the declared condition to real reachable behavior. A green health label alone would be weaker evidence if the dependent could not actually access the dependency over its network path.

Next, the controller removes /ready. After enough failed probes, the dependency becomes unhealthy. The application remains running with the same start timestamp and zero restarts. This is the central negative case: a satisfied startup condition is not a promise that Compose will continuously synchronize the dependent’s lifecycle with later dependency health.

The application intentionally sleeps after its initial request so that its process lifetime can be observed independently. It is not a production client implementation, and the fixture makes no claim that its sleeping state proves successful business operations during the outage. The observed result is precisely that Compose leaves the existing dependent process alone.

Design startup and runtime handling together

Use service_started when process startup is the actual requirement, service_healthy when a meaningful healthcheck defines readiness, and a successful-completion condition for an appropriate one-shot prerequisite. A migration job’s successful exit and a long-running database’s readiness answer different questions. Choose the condition that matches the prerequisite, and retain useful failure output when it is not satisfied.

Make readiness checks cheap and specific. A TCP listener may exist before authentication, schema initialization, or required data is ready. Conversely, checking every optional downstream system can prevent otherwise useful services from starting. The check should represent the minimum dependency contract needed by the service being gated.

Once running, handle dependency changes in the application and its operational controls. Bound retries, avoid duplicating non-idempotent work, and test reconnection after replacement. Compose can coordinate a reproducible startup sequence, but ongoing correctness still depends on the services’ actual communication and failure semantics.

Sources

Control startup order in Compose, started, healthy, and completed dependency conditions, checked 2026-09-12.
Compose services reference, depends_on conditions and explicit Compose restart behavior, checked 2026-09-12.
Dockerfile HEALTHCHECK, probe state, startup grace, and consecutive failure counting, checked 2026-09-12.

Related images share layers; running containers keep separate changes

Mon, 22 Mar 2027 00:00:00 +0000

An image describes an ordered filesystem assembled from immutable layer changesets, together with configuration such as its default command. Related images can reference the same base layers while adding different application files. A running container gets its own writable layer above that image. Distinguishing those three things explains how image reuse saves work without making every container’s runtime edits visible to its neighbors.

When it breaks

Summing the displayed sizes of several related images can overstate their combined local storage use because shared content may be counted repeatedly. Transfer size, unpacked filesystem size, and actual storage allocated by a snapshotter are also different measurements. Layer sharing supports reuse, but it does not justify claiming an exact disk saving without measuring the relevant storage implementation and accounting method.

The opposite mistake is treating container changes as edits to the image. Installing a package or changing a file inside one ordinary container modifies that container’s writable filesystem state. A new container from the original image does not automatically inherit the repair. An operator who patches a running service and then replaces it can therefore see the original failure return.

Sharing does not make an image layer a universal application compatibility boundary. Two applications might use equal base layers yet depend on different configuration, mounted data, architectures, or library versions introduced later. Compare the ordered filesystem identifiers and the actual behavior under test; a matching readable image tag alone says little about which bytes or settings are in use.

Check it quickly

Run this script with Python and a Docker builder after obtaining the pinned Python base digest. The recorded environment used Docker 29.3.1 on Linux arm64, with Python 3.11.16 inside the image. It creates two related images and two temporary containers, uses no container networking, and removes only its own named containers and tags when finished:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(["docker",*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def inspect(image):return json.loads(docker("image","inspect",image).stdout)[0]

name='xbcatm-layers-'+uuid.uuid4().hex[:10];tags=[name+':left',name+':right'];containers=[name+'-one',name+'-two']
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'shared.py').write_text('value=10\n# '+name+'\n')
 (root/'Dockerfile').write_text(f'FROM {BASE}\nCOPY shared.py /app/shared.py\nCOPY main.py /app/main.py\nCMD ["python", "/app/main.py"]\n')
 for tag,increment in zip(tags,[1,2]):
 (root/'main.py').write_text(f'from shared import value\nprint(value+{increment})\n')
 docker('build','--network','none','-t',tag,str(root))
 left=inspect(tags[0])['RootFS']['Layers'];right=inspect(tags[1])['RootFS']['Layers']
 assert left[:-1]==right[:-1] and left[-1]!=right[-1]
 assert docker('run','--rm','--network','none',tags[0]).stdout.strip()=='11'
 assert docker('run','--rm','--network','none',tags[1]).stdout.strip()=='12'
 print('two images share',len(left)-1,'filesystem layer identities and differ in the final application layer')
 for container in containers:
 docker('run','-d','--name',container,'--network','none',tags[0],'python','-c','import time;time.sleep(60)')
 docker('exec',containers[0],'python','-c',"from pathlib import Path;Path('/app/shared.py').write_text('value=100\\n')")
 assert docker('exec',containers[0],'python','/app/main.py').stdout.strip()=='101'
 assert docker('exec',containers[1],'python','/app/main.py').stdout.strip()=='11'
 assert '/app/shared.py' in docker('diff',containers[0]).stdout
 assert inspect(tags[0])['RootFS']['Layers']==left
 assert docker('run','--rm','--network','none',tags[0]).stdout.strip()=='11'
 print('a write in one container changes its writable layer while the sibling container and original image retain the old code')
finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False)

Both builds copy the same shared.py, which defines a value of ten. Their final main.py differs: one adds one and the other adds two. The resulting programs print 11 and 12. Image inspection shows an identical ordered prefix of five filesystem layers and a different final application layer in this recorded base and build arrangement.

The five-layer count is an observation, not a requirement for every Python image. The meaningful assertion compares all layers before the final COPY and confirms that the last identifiers differ. If the publisher changes the base structure, a new digest may have another number of layers while the same sharing principle continues to apply.

Next, two long-running containers start from the first image. A command changes /app/shared.py in only the first container to define one hundred. Executing its application now prints 101. Executing the sibling’s application still prints 11, and docker diff reports the changed path in the edited container’s writable state.

The original image retains its layer identifiers. A fresh container created from that image also prints 11, proving that the runtime edit was not written back into the image. The experiment uses ordinary image-backed paths with no shared volume or host bind mount. A shared writable mount would have a different visibility contract and must be examined separately.

The inspected RootFS.Layers list identifies uncompressed filesystem changesets through DiffIDs. Those values should not be casually substituted for registry manifest digests, compressed layer-blob digests, or image configuration digests. All are content identifiers, but they identify different serialized objects and serve different purposes in storage, retrieval, and artifact selection.

Build and replace the intended artifact

Keep common dependency inputs stable when related applications can actually share them, then copy application-specific code later. This improves the opportunity for cache and content reuse without requiring that unrelated applications use an artificial common dependency set. Correct dependency boundaries come from the software, not from maximizing the number of matching hash strings.

Turn a necessary runtime repair into a reviewed image change and test a replacement container. Persist durable application data through an explicit storage contract; do not rely on the writable layer surviving replacement. Conversely, do not assume deleting one container removes data held in an independently managed volume.

For operational accounting, measure shared and unique storage with tools appropriate to the builder and image store. For release identity, record the artifact reference and runtime configuration. For debugging, inspect both the image and container differences. These observations answer related questions, but no single image-size column or tag answers all of them.

Sources

Docker image layers, shared read-only layers and container writable layers, checked 2026-09-12.
OCI image configuration 1.1.1, ordered filesystem DiffIDs and their distinction from other digests, checked 2026-09-12.
Optimize Docker cache usage, dependency ordering and narrow build inputs, checked 2026-09-12.