Containers · XB Field Notes

A veth pair carries frames between two endpoints; routing supplies their meaning

Sun, 21 Nov 2027 00:00:00 +0000

A veth pair consists of two interconnected virtual Ethernet devices. A frame transmitted through one is delivered to the other, where the receiving network stack or bridge handles it. The two ends can reside in separate Linux network namespaces, making the pair useful for container networking. Creating the pair supplies a link; addresses, routes, bridge membership, forwarding, and access policy are additional configuration.

When it breaks

An interface marked administratively UP can still lack carrier because its veth peer is down. Reading only the UP flag can therefore make an incomplete setup look ready. Inspect both endpoints and distinguish administrative state from LOWER_UP. A route may already name the interface even while the paired link cannot carry useful traffic, so a route listing alone is not a connectivity check.

The pair does not restrict IP payloads to the peer interface’s own address. Routing chooses which link should carry a packet, and Ethernet forwarding or the receiving stack decides what happens next. Traffic destined beyond a container’s immediate neighbor can traverse the pair toward a router or bridge. Saying that every packet not addressed to the peer stays local confuses link delivery with IP destination selection.

Endpoint lifetime is coupled too. Deleting one veth endpoint removes its peer, even if the peer is managed from another namespace. A cleanup action in the host-side context can therefore remove a container’s interface. Stopping traffic, removing routes, and deleting the pair need an intentional order when they affect a live workload; namespace separation does not make the two devices independent resources.

Check it quickly

Save this as check-veth.py and run python3 check-veth.py with Docker. The pinned Linux container uses Python 3.11.16 and iproute2 6.1.0, with NET_ADMIN and SYS_ADMIN for nested namespaces. It creates only disposable virtual devices. The left and right namespaces receive one endpoint each, with 192.0.2.1/30 and 192.0.2.2/30 respectively:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,'child exited without a response'
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found

def server(stack,name,address,port=18080,label='server'):
 code=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import json,socket,sys
address,port,label=sys.argv[1],int(sys.argv[2]),sys.argv[3]
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 body=json.dumps({'server':label,'peer':self.client_address,'path':self.path}).encode()
 self.send_response(200);self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
 def log_message(self,*args):pass
class Server(ThreadingHTTPServer):
 address_family=socket.AF_INET6 if ':' in address else socket.AF_INET
 def server_bind(self):
 if self.address_family==socket.AF_INET6:self.socket.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
 super().server_bind()
service=Server((address,port),Handler);print('ready',flush=True);service.serve_forever()
'''
 process=spawn(stack,['ip','netns','exec',name,'python3','-u','-c',code,address,str(port),label])
 assert line(process).strip()=='ready';return process
def request(name,address,port=18080):
 code=r'''
import http.client,json,sys
client=http.client.HTTPConnection(sys.argv[1],int(sys.argv[2]),timeout=.7)
try:
 client.request('GET','/lab-visible-message');response=client.getresponse();body=json.loads(response.read())
 assert response.status==200;print(json.dumps({'ok':True,'status':response.status,'body':body}))
except OSError as error:print(json.dumps({'ok':False,'type':type(error).__name__,'errno':error.errno}))
finally:client.close()
'''
 return json.loads(ns(name,'python3','-c',code,address,str(port)).stdout)

with lab(['left','right']) as (root,stack,setting):
 run('ip','link','add','left0','type','veth','peer','name','right0')
 run('ip','link','set','left0','netns','left');run('ip','link','set','right0','netns','right')
 ip('left','addr','add','192.0.2.1/30','dev','left0');ip('right','addr','add','192.0.2.2/30','dev','right0')
 ip('left','link','set','left0','up')
 before=json.loads(ip('left','-j','link','show','left0').stdout)[0]
 assert 'LOWER_UP' not in before['flags']
 failed=ns('left','ping','-n','-c','1','-W','1','192.0.2.2',check=False);assert failed.returncode!=0
 ip('right','link','set','right0','up')
 after={name:json.loads(ip(name,'-j','link','show',interface).stdout)[0] for name,interface in [('left','left0'),('right','right0')]}
 assert all('LOWER_UP' in v['flags'] for v in after.values())
 ip('left','neigh','flush','dev','left0');ip('right','neigh','flush','dev','right0')
 successful=ns('left','ping','-n','-c','1','-W','1','192.0.2.2');assert successful.returncode==0
 daemon=server(stack,'right','192.0.2.2',label='right');response=request('left','192.0.2.2');assert response['ok']
 ip('right','addr','add','203.0.113.9/32','dev','lo');ip('left','route','add','203.0.113.9/32','via','192.0.2.2')
 server(stack,'right','203.0.113.9',label='beyond-link-address');routed_response=request('left','203.0.113.9');assert routed_response['ok']
 ip('left','link','delete','left0');gone=ip('right','link','show','right0',check=False);assert gone.returncode!=0 and 'does not exist' in gone.stderr
 print(json.dumps({'one_end_up_flags':before['flags'],'one_end_up_ping_exit':failed.returncode,
 'both_up_flags':{n:v['flags'] for n,v in after.items()},'both_up_ping_exit':successful.returncode,'http':response,'other_ip_destination':routed_response,
 'deleting_one_removes_peer':True,'peer_absent_diagnostic':gone.stderr.strip()}))
"""
name='xbcatm-netpath-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

Initially only left0 is brought up. Its flags include UP and NO-CARRIER, with no LOWER_UP, because right0 remains down. A one-packet ping fails. This gives both a link-state observation and an actual unsuccessful exchange; neither is replaced by assuming that creation of the interface means the other endpoint has been configured correctly.

After bringing right0 up, both endpoints report LOWER_UP. The script clears neighbor entries from the failed attempt before making the healthy comparison, avoiding residual neighbor-resolution timers. A new ping succeeds, and a real HTTP request from left to 192.0.2.2 returns the right service’s identity. The cleared state is explicit; the exercise does not promise instantaneous recovery for every already-pending application request.

Next, the right namespace adds 203.0.113.9/32 on its loopback interface. The left namespace installs a host route to that address through 192.0.2.2. A second HTTP listener bound to 203.0.113.9 responds successfully across the same veth pair. The packet’s IP destination is not the peer’s link address, yet its selected path still uses that virtual Ethernet link.

This second destination is local to the right namespace, so the fixture does not claim to forward onward through another router. It isolates the distinction between a link’s neighboring address and a packet’s final IP destination. A larger topology can connect the receiving endpoint to a bridge or routed network, but those additional components need their own configuration and verification.

Finally, deleting left0 makes ip link show right0 fail in the right namespace with a device-not-found diagnostic. The peer has been removed, not merely placed administratively down. Both namespaces themselves still exist until cleanup. This separates the paired device lifecycle from the lifetime of the namespace that happened to contain one endpoint.

Diagnose the layers in order

Confirm which namespace owns each end and which names the runtime assigned. Then inspect administrative and carrier state, address configuration, neighbor resolution, and the route for the failing destination. A stale neighbor entry can delay recovery after a link change, while an absent route can fail before any frame is transmitted. These conditions require different repairs even when a high-level client reports only a connection timeout.

A pair is not a multiport switch. A bridge can join several such links at Ethernet level, while a router forwards between IP networks. Enabling an endpoint does not automatically add either behavior or an internet route. Document the actual attachment and filtering points, especially when a container runtime creates the devices and another component manages routes or firewall policy.

Treat deletion as a change to both endpoints and verify the affected workload afterward. If throughput or latency motivates tuning MTU, queues, or offload behavior, measure the relevant path rather than inferring performance from the link being virtual. This exercise verifies paired delivery, link readiness, destination routing, and coupled removal.

Sources

Linux veth(4), paired transmission, carrier state and namespace use, checked 2026-09-12.
Linux ip-link(8), veth creation, link state, movement and deletion, checked 2026-09-12.
Linux network_namespaces(7), network boundaries and virtual-device lifetime, checked 2026-09-12.

Security context defines constraints that need runtime evidence

Sat, 20 Nov 2027 00:00:00 +0000

A Kubernetes securityContext configures process identity and selected operating-system constraints. It can reduce what a compromised application is permitted to do, but it does not establish a complete security boundary by itself. Start with the operations the application actually needs, configure the appropriate fields, and test both a permitted operation and a forbidden one in the resulting container.

Put each setting at the right scope

Pod-level settings can supply user and group identity, shared-volume group ownership behavior and a seccomp profile. Container-level settings can override applicable defaults and configure capabilities, privilege escalation and a read-only root filesystem. Check the field’s actual API scope; placing a familiar name under the wrong securityContext does not create an equivalent constraint.

runAsUser selects a numeric user, while runAsNonRoot requires the container to satisfy a non-root policy. These settings complement image metadata rather than trusting a username’s appearance. Volume permissions must also permit the selected identity to perform required work. For an emptyDir used below, fsGroup supplies group access so the application can write its designated working directory.

Linux capabilities divide privileges that would otherwise be associated broadly with root. Dropping ALL removes those capabilities from the container’s configured set. Standard containers do not inherit every Linux capability by default, and a container running as root is not automatically proof of a successful host escape. Review the actual effective privileges rather than using either shortcut.

When it breaks

allowPrivilegeEscalation false sets the no_new_privs constraint for the process. It restricts gaining privileges through executing programs, such as set-user-ID or file-capability transitions. It does not revoke every privilege the process already has or replace application authorization. Inspect the process flag and effective capabilities; do not attribute every denied operation to this one field.

readOnlyRootFilesystem makes the container’s root filesystem read-only. Other mounts can remain writable, including explicitly supplied working volumes. This does not imply a blanket no-execution policy: an interpreter can read and run code stored in a writable volume. It also does not stop network requests or remove access to already mounted data. Read-only storage is one constrained operation, not proof that a compromise cannot progress.

Seccomp filters system calls, but RuntimeDefault depends on the runtime’s supplied profile. A process reporting seccomp filter mode confirms an active filtering mechanism, not that every undesirable syscall or attack path is blocked. Ordinary Linux runtimes can enforce these fields; a special sandbox is not required merely for the settings to have an effect, though stronger isolation has separate design considerations.

Conflicting identity settings can pass API admission but prevent container startup. An image may also expect to write caches or temporary files under its root filesystem. Diagnose startup state and denied paths, then provide only the working storage the application requires. Avoid solving every permission failure by restoring root, privilege escalation or a writable root filesystem.

Check it quickly

Use a disposable Linux Kubernetes cluster, kubectl and python:3.11-slim-bookworm available on its nodes. Save the script as security-context.py and run python3 security-context.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. The account needs namespace creation and deletion permissions. The recorded server was Kubernetes 1.34.11 using kind’s standard runtime configuration.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-security-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 probe = '''from pathlib import Path
import errno,json,os,subprocess,time
status=dict(line.split(':',1) for line in Path('/proc/self/status').read_text().splitlines() if ':' in line)
try:
 Path('/tmp/forbidden-write').write_text('x')
 raise AssertionError('root filesystem unexpectedly writable')
except OSError as error:
 assert error.errno==errno.EROFS
try:
 os.setuid(0)
 raise AssertionError('unexpected UID escalation')
except OSError as error:
 assert error.errno==errno.EPERM
work=Path('/work'); (work/'data').write_text('writable')
(work/'example.py').write_text("print('interpreted from the writable volume')")
ran=subprocess.check_output(['python3',str(work/'example.py')],text=True).strip()
result={'uid':os.getuid(),'gid':os.getgid(),'cap_eff':int(status['CapEff'].strip(),16),
 'no_new_privs':int(status['NoNewPrivs']),'seccomp':int(status['Seccomp']),
 'root_write_errno':errno.EROFS,'uid_zero_denied':True,
 'volume_value':(work/'data').read_text(),'writable_volume_code':ran}
assert result['uid']==result['gid']==10001
assert result['cap_eff']==0 and result['no_new_privs']==1 and result['seccomp']==2
print(json.dumps(result),flush=True); time.sleep(600)
'''
 podspec = spec(probe)
 podspec['securityContext'] = {'runAsNonRoot': True, 'runAsUser': 10001,
 'runAsGroup': 10001, 'fsGroup': 10001, 'seccompProfile': {'type': 'RuntimeDefault'}}
 podspec['volumes'] = [{'name': 'work', 'emptyDir': {}}]
 c = podspec['containers'][0]
 c['securityContext'] = {'allowPrivilegeEscalation': False, 'readOnlyRootFilesystem': True,
 'capabilities': {'drop': ['ALL']}}
 c['volumeMounts'] = [{'name': 'work', 'mountPath': '/work'}]
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'restricted'}, 'spec': podspec})
 wait_for(lambda: obj(namespace, 'pod', 'restricted'), pod_ready)
 observed = json.loads(wait_for(lambda: k('logs', '-n', namespace, 'restricted').stdout.strip(), bool))
 conflicting = spec()
 conflicting['securityContext'] = {'runAsNonRoot': True, 'runAsUser': 0}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'conflict'}, 'spec': conflicting})
 refused = wait_for(lambda: obj(namespace, 'pod', 'conflict'), lambda p:
 any(c.get('state', {}).get('waiting', {}).get('reason') == 'CreateContainerConfigError'
 for c in p.get('status', {}).get('containerStatuses', [])))
 waiting = refused['status']['containerStatuses'][0]
 assert 'runAsUser breaks non-root policy' in waiting['state']['waiting']['message']
 assert waiting['restartCount'] == 0
 print(json.dumps({'runtime_observation': observed,
 'conflicting_uid_accepted_by_api_but_container_start_refused': True,
 'container_escape_resistance_tested': False}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The restricted Pod reports UID and GID 10001, effective capabilities zero, NoNewPrivs one and seccomp mode two. Writing under the root filesystem’s /tmp fails with EROFS, and changing to UID zero fails with EPERM. Those results reflect the combined identity and privilege settings, not an isolated test of each field’s individual effect.

The process successfully writes a value under /work and invokes Python on a file it wrote there. This deliberately demonstrates the allowed working-volume behavior and the limit of the read-only-root claim. A second Pod combines runAsNonRoot with runAsUser zero: the API accepts it, but kubelet reports CreateContainerConfigError before the container starts.

All assertions passed, and the namespace was removed. The fixture verifies these concrete runtime constraints and startup failure. It does not test container escape resistance, every seccomp rule, hostile multi-tenancy or the application’s authorization model.

Sources

Kubernetes: Security Context, Pod and container security fields and runtime effects, checked 2026-09-13.
Linux: No New Privileges Flag, restrictions on privilege gains through exec, checked 2026-09-13.
Linux: capabilities(7), separate capability privileges and process capability sets, checked 2026-09-13.

Signed images prove who built them

Thu, 21 Oct 2027 00:00:00 +0000

A container image reference is a mutable pointer; a digest is immutable. Signing ties a digest to a key, so a verifier can ask who produced these exact bytes before anything runs them. That is the question registries and manifests cannot answer, because they tell you where bytes came from, not who made them.

The tension is that a signature is only a control when something verifies it, and verification is only meaningful when the trust root is managed. Signing is a few lines in a pipeline; deciding which keys count, and what happens when one is retired, is the actual engineering.

Digests are the identity, tags are pointers

The digest is a hash over the exact manifest bytes, so any change to the image produces a different digest. Signing that digest binds the signature to content rather than to a name. With an EC P-256 keypair, a real SHA-256 digest and a cosign-shaped payload, openssl dgst -verify reports success; flipping a single byte of the artifact changes the digest, and the same verification returns failure.

Tags behave differently. Three tags pointing at the signed digest all verify, because the tag was never part of the signed payload. That is the design: tags are pointers to digests, and re-pointing one is a normal registry operation. Move a tag to a digest that was never signed and verification against the resolved digest fails — but only if the verifier resolves the tag first.

Signing a tag binds nothing

Sign the string app:v2 instead of a digest and verification succeeds forever, because the string never changes. Point that tag at a completely different image and the signature still verifies: it is a true statement about a string that says nothing about bytes. This is why a signature must be over the digest, and why verification has to resolve the tag to a digest before checking anything.

The trust root is the other half. A signature proves that whoever holds the private key produced it; the public key, or a Fulcio-issued certificate paired with a Rekor log entry, is what turns that into “this identity built it”. Distributing and rotating that anchor is the security boundary, and it lives outside the signature entirely.

Verification is the control, not the signature

A pipeline that signs and then deploys without verifying has bought decoration. The measured contrast is blunt: a tampered artifact is blocked when the deploy step verifies, and the same artifact deploys cleanly when the step only records provenance — with the provenance record now naming the tampered digest. Enforcement belongs at admission, or in a gate before deploy, so an unsigned image never reaches a node.

Revocation is the part with no answer inside the signature. A signature carries no expiry and no revocation list: it verifies under its key forever. Rotating keys does not revoke anything — the old signature is rejected under the new key and still accepted under the old one — so anyone holding a retired key keeps producing artifacts that pass. The fix is a trust root with short-lived certificates and a transparency log, plus policy that pins identity rather than merely checking that some signature exists.

When it breaks

The classic failure is an unsigned image that a policy engine never checked. The pipeline signs, the deploy step pulls by tag and runs, and the gap stays invisible until someone pushes different bytes under the same tag. The audit trail then claims provenance for an image nobody verified, which is worse than having no provenance at all.

The second failure is a signature that verifies but was made by a key you no longer trust. Because there is no revocation inside the signature, a leaked or retired key keeps signing acceptable artifacts, and the blast radius is every artifact that key ever signed, retroactively. Pin the signing identity in policy, keep certificates short-lived, and record signatures in a transparency log so an unexpected key appears as a log entry rather than as a silent pass.

Check it quickly

Verify the digest, then confirm the tag still resolves to it:

cosign verify --key cosign.pub registry.example.com/app@sha256:9ab5a134...
cosign tree registry.example.com/app@sha256:9ab5a134...
skopeo inspect --format '{{.Digest}}' docker://registry.example.com/app:latest

cosign verify must print the verified identity and exit 0, and cosign tree should list the signatures and attestations attached to that digest. skopeo inspect must print the same digest you verified. A wrong result is verify failing with no matching signatures, or verify succeeding while skopeo reports a different digest — meaning the tag now points at bytes that were never signed.

Sources

Sigstore: verifying signatures with cosign, key-based and keyless verification over a digest, checked 2026-09-16.
SLSA v1.0 specification, provenance, digests and what a signature is claimed to attest, checked 2026-09-16.
Sigstore policy-controller, enforcing signature and identity policy at admission, checked 2026-09-16.

A pinned base fixes one input; the release needs the rest

Fri, 30 Apr 2027 00:00:00 +0000

Pinning a container base by digest fixes the referenced image object instead of relying on a movable tag. That removes one source of unexpected build changes, but it does not make every derived image or running instance identical. Application source, build arguments, fetched dependencies, target platform, runtime configuration, and persistent data can still change what gets built and how it behaves.

When it breaks

Two Dockerfiles can use the same FROM digest and produce different applications. Even the same Dockerfile can produce different output when a build argument changes or an unpinned download resolves new content. Declaring the base immutable therefore does not prove that the complete build is deterministic. The full set of inputs and the transformations applied to them determine that property.

Running the exact same application image with different environment values can select different endpoints, feature behavior, or storage. A writable bind mount can also replace files that were tested inside the image. Keeping configuration outside the image is often useful, but the configuration still needs versioning and review when it contributes to the release’s intended behavior.

An ordinary container also has a writable filesystem layer. An emergency edit can change one instance while leaving the image and other instances untouched. Replacement from the unchanged image then loses the edit. This is why immutable deployment practice combines artifact replacement with controlled runtime state; naming a base digest alone does not enforce that operational discipline.

Check it quickly

Run the script using host Python and a Docker builder with the pinned base available. It was executed with Docker 29.3.1 and Linux arm64 Python 3.11.16. Builds and containers disable networking, so remote package changes cannot explain the results. The fixture removes its temporary containers and tags and never edits an existing service:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def inspect(image):return json.loads(docker('image','inspect',image).stdout)[0]

name='xbcatm-pinning-'+uuid.uuid4().hex[:10];tags=[name+':one',name+':two'];container=name+'-running'
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory)
 (root/'Dockerfile').write_text(f'FROM {BASE}\nARG MESSAGE\nRUN printf "%s" "$MESSAGE" > /built-message\nENV MODE=default\nCOPY app.py /app.py\nCMD ["python", "/app.py"]\n')
 (root/'app.py').write_text('from pathlib import Path\nimport os\nprint(Path("/built-message").read_text(),os.environ["MODE"])\n')
 layers=inspect(BASE)['RootFS']['Layers']
 for tag,message in zip(tags,['one','two']):
 docker('build','--network','none','--build-arg','MESSAGE='+message,'-t',tag,str(root))
 assert inspect(tag)['RootFS']['Layers'][:len(layers)]==layers
 assert inspect(tags[0])['Id']!=inspect(tags[1])['Id']
 assert docker('run','--rm','--network','none',tags[0]).stdout.strip()=='one default'
 assert docker('run','--rm','--network','none',tags[1]).stdout.strip()=='two default'
 assert docker('run','--rm','--network','none','-e','MODE=changed',tags[0]).stdout.strip()=='one changed'
 print('both builds use the same pinned base layers; different build arguments produce different artifacts')
 print('the identical application image behaves differently when runtime configuration changes')
 docker('run','-d','--name',container,'--network','none',tags[0],'python','-c','import time;time.sleep(60)')
 docker('exec',container,'python','-c','from pathlib import Path;Path("/built-message").write_text("edited")')
 assert docker('exec',container,'python','/app.py').stdout.strip()=='edited default'
 assert docker('run','--rm','--network','none',tags[0]).stdout.strip()=='one default'
 failed=docker('run','--rm','--network','none','--read-only',tags[0],'python','-c','from pathlib import Path;Path("/built-message").write_text("edited")',check=False)
 assert failed.returncode!=0 and 'Read-only file system' in failed.stderr
 print('a writable container can drift while its image stays unchanged; a read-only root rejects that particular write')
finally:
 docker('rm','-f',container,check=False);docker('image','rm',*tags,check=False)

Both builds use the same base reference and application source. The only selected build difference is MESSAGE, which the Dockerfile writes to /built-message. One image prints one default; the other prints two default. Their complete image identities differ, while inspection confirms that their ordered base filesystem-layer prefix is identical.

That pair establishes a bounded claim: the base remained fixed, but the derived artifact changed because another explicit input changed. It does not attempt to prove bit-for-bit reproducibility of repeated builds with every input held constant. Such a test would also need to control timestamps, dependency retrieval, toolchain behavior, and metadata emitted by the builder.

Next, the first application image runs with MODE overridden to changed. It prints one changed without rebuilding the image. This separates artifact identity from release configuration. A registry digest can identify application bytes, but it cannot by itself tell an incident investigator which environment values, credentials, mounted configuration, or external services the process actually used.

The script then starts a long-running container from the first image and edits its message file. Running the application inside that container prints edited default, while a fresh container from the unchanged image still prints one default. The repair exists only in the modified container’s writable state and will not automatically propagate into replacements.

A final run enables a read-only root filesystem and attempts the same file edit. It fails with a read-only filesystem error. This demonstrates a useful enforcement mechanism for that image-backed path, not a guarantee that every possible runtime state is frozen. Explicit writable mounts, external databases, environment settings, and the host kernel remain separate parts of the execution environment.

Record a release, then replace its instances

Keep a release record connecting the tested application digest, target platform, configuration revision, and required data compatibility. An OCI index can contain manifests for several platforms, so pinning the index still allows the runtime to select a platform-specific image. Record the selection when architecture affects reproducibility or diagnosis.

Build required changes into a new artifact, validate it, and replace instances through the deployment mechanism. Put durable data in storage with its own lifecycle and recovery plan. Avoid treating the container’s writable layer as the only durable copy, and avoid silently changing shared configuration underneath instances that supposedly belong to one fixed release.

Digest pinning also means new base fixes do not arrive automatically. Track available updates, review a changed digest, rebuild dependent applications, and test before promotion. Retaining a known input is valuable, but retaining an obsolete input forever is not the same as maintaining a secure release.

When investigating drift, compare artifact, configuration, mounts, and runtime changes separately. Each has a different owner and evidence trail. Replacing ad hoc instance edits with reviewed releases makes those differences explainable, while acknowledging that identical image bytes do not imply identical external state or identical behavior under every workload.

Sources

Docker build best practices, base-image digest pinning and deliberate update workflows, checked 2026-09-12.
Docker container run, runtime environment overrides and writable/read-only filesystems, checked 2026-09-12.
The Twelve-Factor App: build, release, run, separating a build artifact from release configuration and execution, checked 2026-09-12.
OCI image index 1.1.1, platform selection within an image index, checked 2026-09-12.

Regular init containers establish startup prerequisites

Sun, 03 Jan 2027 00:00:00 +0000

A regular init container performs setup before a Pod’s application containers start. It has its own command, image and filesystem view, and can mount a volume that the application later reads. This separates a prerequisite from the main process, but does not make arbitrary setup safe or automatically bounded. Design the step around its actual retry behavior and the lifetime of its output.

Sequence is local to one Pod

Kubelet runs ordinary init containers in their declared order. Each must finish successfully before the next begins, and the application containers wait for that sequence. Inspect initContainerStatuses to distinguish a waiting step, a running step and a terminated attempt. A Pod’s phase alone does not explain which command failed or whether another attempt is underway.

Shared storage requires explicit mounts. An emptyDir can carry a generated file between init and application containers, and it survives a container restart within that Pod. It disappears with the Pod. The separate containers do not share their image root filesystems, and a file written only inside an init container’s writable layer is not automatically delivered to the application.

Ordinary init containers must complete. Kubernetes also supports restartable sidecars represented in initContainers with container-level restartPolicy set to Always; those have different lifecycle rules. Do not apply the ordinary run-to-completion description to every entry merely because it appears in that field. This article’s example uses only regular init containers.

When it breaks

A nonzero exit does not always fail the whole Pod immediately. With an ordinary application Pod’s Always policy, a failed regular init is retried until it succeeds; successful init steps do not restart merely because the application remains running. Backoff slows repeated attempts, but is not a fixed retry-count limit. With Pod restartPolicy Never, a failed initialization instead makes that Pod fail.

Write setup so another attempt can encounter existing output safely. An interrupted download may leave a partial file, and an emptyDir marker may survive the failed process. Validate the completed artifact and publish it deliberately. The counter used below is diagnostic state; it is not a production strategy for deciding that an external prerequisite must have become valid.

Ordering inside one Pod also does not serialize a fleet. Putting a database migration in every replica’s init sequence can run several migrations concurrently. Use the migration system’s actual locking and compatibility contract. Waiting for a dependency once cannot prove that dependency will remain available after initialization; continuing request handling and readiness need their own design.

Initialization affects resource placement too. For ordinary init containers, scheduling accounts for peak initialization requirements alongside the summed application requirements, with overhead where applicable. A brief but large setup request can prevent scheduling even when the steady application is small. Sidecars and Pod-level resource features require checking their additional accounting rules.

Check it quickly

Use a disposable Kubernetes cluster, kubectl access and python:3.11-slim-bookworm on its nodes. Save the code as init-order.py and run python3 init-order.py KUBECONFIG_PATH CONTEXT, replacing the arguments with explicit test-cluster details. The account must create and delete namespaces. The recorded server was Kubernetes 1.34.11; allow time for one failed initialization and its retry.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-init-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 prepare = r'''from pathlib import Path
import sys,time
root=Path('/work'); counter=root/'attempts'
attempt=int(counter.read_text())+1 if counter.exists() else 1
counter.write_text(str(attempt))
with (root/'order').open('a') as f: f.write('prepare'+str(attempt)+'\n')
if attempt == 1: sys.exit(7)
print('waiting for release',flush=True)
while not (root/'go').exists(): time.sleep(0.1)
(root/'input').write_text('ready')
'''
 transform = r'''from pathlib import Path
root=Path('/work'); assert (root/'input').read_text()=='ready'
(root/'output').write_text('READY')
with (root/'order').open('a') as f: f.write('transform\n')
'''
 main = r'''from pathlib import Path
import json,time
root=Path('/work'); assert (root/'output').read_text()=='READY'
with (root/'order').open('a') as f: f.write('app\n')
print(json.dumps({'order':(root/'order').read_text().splitlines(),
 'artifact':(root/'output').read_text()}),flush=True)
time.sleep(600)
'''
 podspec = spec(main)
 mount = [{'name': 'work', 'mountPath': '/work'}]
 podspec['volumes'] = [{'name': 'work', 'emptyDir': {}}]
 podspec['containers'][0]['volumeMounts'] = mount
 podspec['initContainers'] = []
 for name, script in [('prepare', prepare), ('transform', transform)]:
 container = spec(script)['containers'][0]
 container['name'] = name
 container['volumeMounts'] = mount
 podspec['initContainers'].append(container)
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'ordered'}, 'spec': podspec})
 def retry_running(p):
 statuses = p.get('status', {}).get('initContainerStatuses', [])
 return statuses and statuses[0]['restartCount'] >= 1 and 'running' in statuses[0]['state']
 waiting = wait_for(lambda: obj(namespace, 'pod', 'ordered'), retry_running)
 first = waiting['status']['initContainerStatuses'][0]
 assert first['lastState']['terminated']['exitCode'] == 7
 assert waiting['status']['phase'] == 'Pending'
 assert all('running' not in c['state'] for c in waiting['status'].get('containerStatuses', []))
 assert 'running' not in waiting['status']['initContainerStatuses'][1]['state']
 k('exec', '-n', namespace, 'ordered', '-c', 'prepare', '--', 'python3', '-c',
 'from pathlib import Path; Path("/work/go").touch()')
 ready = wait_for(lambda: obj(namespace, 'pod', 'ordered'), pod_ready)
 assert ready['metadata']['uid'] == waiting['metadata']['uid']
 output = wait_for(lambda: k('logs', '-n', namespace, 'ordered', '-c', 'app').stdout.strip(), bool)
 observed = json.loads(output)
 assert observed == {'order': ['prepare1', 'prepare2', 'transform', 'app'], 'artifact': 'READY'}
 assert ready['status']['initContainerStatuses'][0]['restartCount'] == 1
 assert all(c['state']['terminated']['exitCode'] == 0 for c in ready['status']['initContainerStatuses'])
 never = spec()
 never['restartPolicy'] = 'Never'
 failed_init = spec('raise SystemExit(7)')['containers'][0]
 failed_init['name'] = 'fails'
 never['initContainers'] = [failed_init]
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'never'}, 'spec': never})
 failed = wait_for(lambda: obj(namespace, 'pod', 'never'), lambda p: p.get('status', {}).get('phase') == 'Failed')
 status = failed['status']['initContainerStatuses'][0]
 assert status['restartCount'] == 0 and status['state']['terminated']['exitCode'] == 7
 assert not any('running' in c['state'] for c in failed['status'].get('containerStatuses', []))
 print(json.dumps({'ordered_artifact': observed, 'retried_init_count': 1,
 'app_waited_for_both_regular_init_containers': True,
 'never_policy_failed_pod_without_retry': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The preparation container records its first attempt and exits seven. During its second attempt it waits for a release file. The script confirms the Pod is Pending, the first failure is recorded, and neither the next init step nor the application is running. It then releases preparation through that container’s shared volume.

The second init transforms ready into READY, and the application validates the artifact. Its log records prepare1, prepare2, transform and app in that order; the first init’s restart count is one. A separate Pod with Never policy exits initialization unsuccessfully and reaches Failed with no init retry. All assertions passed, and both Pods were removed with their namespace. The experiment verifies local sequencing and failure policy, not distributed migration safety or durable storage.

Sources

Kubernetes: Init Containers, regular init ordering, retries, resource accounting and repeatable setup, checked 2026-09-13.
Kubernetes: Pod Lifecycle, Pod phases, container states and restart policies, checked 2026-09-13.

A container combines process isolation with a shared Linux kernel

Mon, 07 Dec 2026 00:00:00 +0000

A conventional Linux container runs processes using the daemon host’s Linux kernel while providing selected isolated views through namespaces. Image files supply a userspace environment; they do not boot a separate kernel for each container. On Docker Desktop for macOS, the daemon and Linux containers run inside a Linux virtual machine, so the shared kernel is that VM’s kernel, not the macOS kernel.

When it breaks

Assuming that every container has an independent operating-system kernel leads to wrong diagnoses. A kernel feature missing from the daemon’s Linux environment does not appear merely because the image includes another distribution’s userspace files. Conversely, changing an image’s packages does not upgrade the host kernel. The distinction matters for system calls, kernel modules, security boundaries, and resource accounting.

Namespace isolation is configurable rather than an identical fixed bundle in every deployment. A container can join another network namespace, use host networking, share a volume, or receive different user-namespace mappings. Seeing UID zero inside a container does not by itself describe its authority outside the container. Capabilities, user mappings, security policies, and exposed resources must be considered together.

Localhost is another common source of confusion. With separate network namespaces, 127.0.0.1 reaches the calling container’s own loopback space. It does not automatically reach a sibling application or the developer’s laptop. Likewise, a file created in one ordinary container’s writable layer does not automatically become visible to another container from the same image.

Check it quickly

Run this host Python script with a Docker daemon and the pinned Python image available. The recorded Docker 29.3.1 environment used Linux 6.12.76 in Docker Desktop’s VM and arm64 Python 3.11.16. Both containers use the same image, receive separate hostnames, and have networking set to none. The script removes only its two temporary containers:

import json,subprocess,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],text=True,capture_output=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def inspect(name):return json.loads(docker('inspect',name).stdout)[0]
def execute(name,code):return docker('exec',name,'python','-c',code).stdout.strip()

name='xbcatm-isolation-'+uuid.uuid4().hex[:10];containers=[name+'-a',name+'-b']
try:
 listener='import socket,time;s=socket.socket();s.bind(("127.0.0.1",18080));s.listen();print("ready",flush=True);time.sleep(120)'
 for container,command in zip(containers,[listener,'import time;time.sleep(120)']):
 docker('run','-d','--name',container,'--hostname',container,'--network','none',BASE,'python','-c',command)
 probe='import json,os;print(json.dumps({"kernel":os.uname().release,"hostname":os.uname().nodename,"uid":os.getuid(),"namespaces":{name:os.readlink("/proc/self/ns/"+name) for name in ["pid","mnt","net","uts","ipc","user"]}}))'
 left,right=[json.loads(execute(container,probe)) for container in containers]
 assert left['kernel']==right['kernel']
 assert left['hostname']!=right['hostname']
 for namespace in ['pid','mnt','net','uts','ipc']:assert left['namespaces'][namespace]!=right['namespaces'][namespace]
 assert inspect(containers[0])['State']['Pid']!=inspect(containers[1])['State']['Pid']
 for container in containers:assert 'python' in execute(container,'from pathlib import Path;print(Path("/proc/1/cmdline").read_bytes())')
 execute(containers[0],'from pathlib import Path;Path("/private-marker").write_text("only-a")')
 assert execute(containers[1],'from pathlib import Path;print(Path("/private-marker").exists())')=='False'
 execute(containers[0],'import socket;s=socket.create_connection(("127.0.0.1",18080),timeout=1);s.close()')
 execute(containers[1],'import socket;s=socket.socket();assert s.connect_ex(("127.0.0.1",18080))!=0;s.close();s=socket.socket();s.bind(("127.0.0.1",18080));s.close()')
 print('same daemon-side Linux kernel:',left['kernel'],'with different PID, mount, network, UTS, and IPC namespace identities')
 print('each container sees its own PID 1, hostname, writable file state, and loopback port space')
 print('observed uid values:',left['uid'],right['uid'],'same user namespace:',left['namespaces']['user']==right['namespaces']['user'])
finally:docker('rm','-f',*containers,check=False)

Both containers report the same Linux kernel release. Their PID, mount, network, UTS, and IPC namespace identifiers differ, and each sees its own Python process at PID 1. Docker’s inspected daemon-side process IDs differ as well. These observations connect the container-local view with separate processes managed by the same daemon; the reported host IDs belong to the Linux environment.

The assigned hostnames also differ. UTS namespaces control that view, while PID namespaces control process-number visibility and mount namespaces control mount views. These mechanisms have distinct jobs. Linux has additional namespace types beyond those compared here, so the example should not be read as a complete inventory of all namespace features.

The first container writes /private-marker, and the second reports that the path is absent. The example does not mount a shared data directory, so the difference demonstrates ordinary per-container writable state. If both containers were given the same writable volume, visibility through that mount would follow the shared volume’s contract instead.

A listener in the first container accepts a connection to its own loopback port 18080. The second container cannot connect to that address and port, but can bind the same port number in its own network namespace. Disabling external networking has not removed loopback; it has left each container with a separate local network space.

Both recorded processes run as UID zero, and their user-namespace identities are equal in this daemon’s configuration. That is a deliberate observation of the actual defaults, not an assertion that Docker always creates a separate user namespace. Other deployments can enable user-namespace remapping or use a different runtime model, and those choices change the security analysis.

Inspect the boundary that was actually configured

Read namespace, mount, network, capability, and user settings when diagnosing access or isolation. Avoid treating a container name as proof of a particular boundary. Resource ceilings are another dimension: cgroups can limit aggregate CPU, memory, or task usage, while namespaces determine views. An isolated process namespace does not itself impose a memory budget.

Use one clear primary service role when it fits the application, but do not confuse that convention with a kernel rule requiring exactly one process. A service can start workers, and PID 1 still needs appropriate signal and child-reaping behavior. Process supervision should follow the actual application model.

For durable state, define storage whose lifecycle matches the data. For stronger tenant separation, assess the runtime and host trust model explicitly. Container isolation is real, but its guarantees come from the configured mechanisms and maintained kernel, not from imagining a small independent machine inside every image.

Sources

namespaces(7), namespace types, isolated views, and namespace identities, checked 2026-09-12.
Docker container run, process, network, user, and privilege settings, checked 2026-09-12.
pid_namespaces(7), PID numbering and namespace-init behavior, checked 2026-09-12.

cgroup v2 enforces different budgets with different failure modes

Tue, 10 Nov 2026 00:00:00 +0000

cgroup v2 organizes resource control in a unified hierarchy. A workload’s processes can share aggregate budgets, with limits inherited through the hierarchy rather than applied only to one particular PID. CPU bandwidth, memory usage, and task count are separate controllers with different enforcement behavior. Reading the actual cgroup files and failure counters is more useful than treating every configured limit as the same kind of hard stop.

When it breaks

cpu.max limits CPU bandwidth over a period. It can throttle runnable work after the group consumes its quota, but it does not reserve a dedicated processor or guarantee a particular request latency. Multiple threads can consume a shared quota quickly. CPU weights instead influence relative sharing under contention; confusing weights, quotas, and CPU affinity leads to incorrect capacity expectations.

Memory limits do not universally turn a large allocation into a clean application-level error. At memory.max, the kernel can reclaim memory and may invoke the cgroup OOM mechanism if usage cannot be reduced. Some allocation paths have other failure behavior. memory.high applies reclaim pressure and throttling rather than itself invoking OOM, so the two settings should not be described interchangeably.

Task limits also cover threads, not just the number of application processes a dashboard happens to display. Reaching pids.max can make fork or clone fail even when CPU and memory still look available. Hierarchical restrictions mean a generous child value cannot simply override a tighter ancestor budget. The relevant path and controller delegation matter when interpreting visible files.

Check it quickly

Run the script with Python and a Docker daemon using cgroup v2, after obtaining the pinned image. The recorded environment used Docker 29.3.1, Linux 6.12.76, and Python 3.11.16. Every limit applies only to a disposable experiment container. The script performs a bounded memory exhaustion test, removes its containers, and never changes host-wide cgroup settings:

import json,subprocess,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],text=True,capture_output=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def inspect(name):return json.loads(docker('inspect',name).stdout)[0]

name='xbcatm-budgets-'+uuid.uuid4().hex[:10];containers=[name+'-limits',name+'-oom',name+'-killed']
limits_code='''from pathlib import Path
import errno,json,subprocess,time
root=Path('/sys/fs/cgroup')
assert (root/'cgroup.controllers').exists()
quota,period=(root/'cpu.max').read_text().split();assert int(quota)*2==int(period)
assert int((root/'memory.max').read_text())==64*1024*1024
assert int((root/'pids.max').read_text())==12
def stats():return dict(line.split() for line in (root/'cpu.stat').read_text().splitlines())
before=stats();end=time.monotonic()+2
while time.monotonic()<end:pass
after=stats();assert int(after['nr_throttled'])>int(before['nr_throttled'])
children=[]
try:
 while True:
 try:children.append(subprocess.Popen(['sleep','30']))
 except OSError as error:
 assert error.errno==errno.EAGAIN;break
 assert children and int((root/'pids.current').read_text())==12
 assert int(dict(line.split() for line in (root/'pids.events').read_text().splitlines())['max'])>0
 print(json.dumps({'cpu.max':[quota,period],'memory.max':(root/'memory.max').read_text().strip(),'pids.max':12,'children':len(children),'throttled_periods':int(after['nr_throttled'])-int(before['nr_throttled'])}),flush=True)
finally:
 for child in children:child.terminate()
 for child in children:child.wait()
'''
try:
 result=docker('run','--name',containers[0],'--network','none','--cpus','.5','--memory','64m','--memory-swap','64m','--pids-limit','12',BASE,'python','-c',limits_code)
 print(result.stdout.strip())
 oom=docker('run','--name',containers[1],'--network','none','--memory','64m','--memory-swap','64m',BASE,'python','-c','print("allocating",flush=True);data=bytearray(256*1024*1024)',check=False)
 state=inspect(containers[1])['State'];assert oom.returncode==137 and state['OOMKilled'] and state['ExitCode']==137
 docker('run','-d','--name',containers[2],'--network','none',BASE,'python','-c','import time;time.sleep(60)')
 docker('kill','--signal','KILL',containers[2]);docker('wait',containers[2])
 state=inspect(containers[2])['State'];assert state['ExitCode']==137 and not state['OOMKilled']
 print('a 64 MiB memory budget caused actual OOM termination; an explicit SIGKILL also returned 137 but OOMKilled was false')
finally:docker('rm','-f',*containers,check=False)

The first container reads cpu.max as 50000 100000, memory.max as 67,108,864 bytes, and pids.max as twelve. The CPU values represent fifty milliseconds of bandwidth per hundred-millisecond period. Setting Docker’s memory-plus-swap limit equal to its memory limit prevents this fixture from extending its budget through swap; the memory test remains deliberately small.

A two-second busy loop increases cpu.stat’s nr_throttled counter. The recorded increase was twenty periods, while the assertion requires only a positive increase. This confirms actual enforcement rather than merely checking that a Docker flag was accepted. It is not a benchmark proving an exact wall-clock slowdown for every workload or scheduler environment.

The process then starts sleeping children until task creation raises EAGAIN. In the recorded run, eleven children plus the controlling Python process reach the limit of twelve, and the pids controller’s rejection counter increases. The children are terminated and reaped. The refusal comes from the group’s task budget even though the children are mostly idle.

Another container receives a 64 MiB memory limit and attempts to initialize a 256 MiB bytearray. It exits with status 137, and Docker records OOMKilled as true. This validates an actual memory-limit failure for the tested allocation, rather than asserting that every possible allocation must produce the same exception or termination pattern.

A control container is explicitly sent SIGKILL. It also exits with status 137, but OOMKilled is false. That comparison is essential: the exit status alone cannot establish an OOM diagnosis. Correlate runtime state with memory events and system evidence when investigating production incidents, especially if several supervisors can terminate a workload.

Size and observe the whole workload

Budget for the application’s workers and relevant charged memory, not just a single process’s heap estimate. Page cache and other accounted memory can contribute to pressure. Leave room for startup, dependency changes, and recovery operations, then measure the workload under representative concurrency. A tight limit can change latency or failure rates before it produces an obvious crash.

Monitor current usage, pressure signals, and controller events alongside application behavior. Pressure indicators can reveal sustained contention, but a sudden allocation spike may outrun a periodic alert. Monitoring should explain failures and guide sizing without promising that every limit breach will be predicted in advance.

Change resource policy through the manager that owns the workload, such as Docker or systemd, and verify the effective controller values afterward. Container-local cgroup paths may reflect a namespace view rather than the host’s full hierarchy. Correct diagnosis connects that view, inherited constraints, and the specific enforcement mechanism that actually acted.

Sources

Linux 6.12 cgroup v2 documentation, hierarchical CPU, memory, and task controllers and their counters, checked 2026-09-12.
Docker resource constraints, Docker CPU, memory, and swap configuration, checked 2026-09-12.
cgroups(7), control-group membership and resource-accounting semantics, checked 2026-09-12.

Multi-stage builds need a tested runtime boundary

Fri, 25 Sep 2026 00:00:00 +0000

A multistage Dockerfile can compile software in one stage and copy selected outputs into another stage that becomes the runtime image. The final stage inherits its own base and the files explicitly copied into it, rather than automatically inheriting the builder’s compiler, source tree, and package caches. This creates a useful boundary, but the author must still identify everything the program needs when it executes.

When it breaks

COPY can successfully place a dynamically linked executable into a scratch image even when its ELF interpreter and shared libraries are missing. The image build may finish normally because copying does not execute the program or validate its entire runtime dependency graph. Startup can then report no such file or directory despite the executable itself being present at the configured path.

Scripts need their interpreters, applications may need certificate roots or timezone data, and programs can load plugins or assets only on particular request paths. A simple launch test might miss those later requirements. Selecting a small runtime base is useful only when it provides the actual contract the application relies on, including supported architecture and necessary filesystem paths.

The earlier stage is also not securely erased just because it is absent from the final image’s ancestry. Builder caches and explicitly exported intermediate images can retain its contents. Sensitive build inputs require appropriate secret handling, and copied outputs must be checked for accidental inclusion. Multistage structure controls final filesystem composition; it is not an automatic confidentiality guarantee.

Check it quickly

Run this script with Python, Docker, the pinned Python base, and network access for the builder’s Debian package installation. Runtime containers disable networking. The recorded run used Docker 29.3.1 on Linux arm64 and GCC 12.2.0 from Debian’s 12.2.0-14+deb12u1 package. The package repository is live, so the exercise demonstrates runtime composition rather than promising identical compiler bytes forever:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,tarfile,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=360)
 if check:assert result.returncode==0,result.stderr
 return result
def inspect(image):return json.loads(docker('image','inspect',image).stdout)[0]

name='xbcatm-stages-'+uuid.uuid4().hex[:10];tags=[name+':builder',name+':good',name+':bad'];container=name+'-inspect'
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory)
 (root/'square.c').write_text('#include <stdio.h>\n#include <stdlib.h>\n#include <errno.h>\nint main(int argc,char **argv){\n char *end;long value;\n if(argc!=2){fprintf(stderr,"one integer required\\n");return 2;}\n errno=0;value=strtol(argv[1],&end,10);\n if(errno||end==argv[1]||*end||value<0||value>100){fprintf(stderr,"range 0..100\\n");return 2;}\n printf("%ld\\n",value*value);return 0;\n}\n')
 (root/'Dockerfile').write_text(f'''FROM {BASE} AS builder
RUN apt-get update && apt-get install -y --no-install-recommends gcc libc6-dev && rm -rf /var/lib/apt/lists/*
WORKDIR /source
COPY square.c .
RUN gcc -O2 -Wall -Wextra -Werror -static -o /square-static square.c && gcc -O2 -Wall -Wextra -Werror -o /square-dynamic square.c
FROM scratch AS good
COPY --from=builder /square-static /square
USER 65532:65532
ENTRYPOINT ["/square"]
FROM scratch AS bad
COPY --from=builder /square-dynamic /square
USER 65532:65532
ENTRYPOINT ["/square"]
''')
 for tag,target in zip(tags,['builder','good','bad']):docker('build','--target',target,'-t',tag,str(root))
 print('compiler:',docker('run','--rm','--network','none',tags[0],'gcc','--version').stdout.splitlines()[0])
 for argument,expected in [('0','0'),('7','49'),('100','10000')]:
 assert docker('run','--rm','--network','none',tags[1],argument).stdout.strip()==expected
 for argument in ['7x','-1','101','','999999999999999999999999999']:
 invalid=docker('run','--rm','--network','none',tags[1],argument,check=False)
 assert invalid.returncode==2 and 'range 0..100' in invalid.stderr
 invalid=docker('run','--rm','--network','none',tags[1],check=False)
 assert invalid.returncode==2 and 'one integer required' in invalid.stderr
 invalid=docker('run','--rm','--network','none',tags[1],'7','8',check=False)
 assert invalid.returncode==2 and 'one integer required' in invalid.stderr
 good=inspect(tags[1]);builder=inspect(tags[0]);assert good['Size']<builder['Size'] and len(good['RootFS']['Layers'])==1
 assert good['Config']['User']=='65532:65532'
 docker('create','--name',container,tags[1],'7');archive=root/'rootfs.tar';docker('export','-o',str(archive),container)
 with tarfile.open(archive) as contents:
 names={entry.name.removeprefix('./') for entry in contents}
 assert 'square' in names and 'usr/bin/gcc' not in names and 'source/square.c' not in names
 bad=docker('run','--rm','--network','none',tags[2],'7',check=False)
 assert bad.returncode!=0 and 'no such file or directory' in bad.stderr
 print('the static final image runs valid input, rejects invalid input, and contains one application layer without compiler or source')
 print('recorded image sizes:',builder['Size'],'builder bytes;',good['Size'],'final bytes')
 print('the dynamic scratch image builds successfully but cannot start because its interpreter/runtime was not copied')
finally:
 docker('rm','-f',container,check=False);docker('image','rm',*tags,check=False)

The builder compiles the same small C program twice, once statically and once dynamically. The program accepts one integer in the range zero through one hundred and prints its square. Compilation enables warnings and treats them as errors. Named final targets copy only the corresponding executable into scratch and select numeric user and group 65532.

The static final image prints 0, 49, and 10000 for inputs zero, seven, and one hundred. It returns status two for trailing text, negative or out-of-range values, an empty argument, and an integer too large for strtol. Missing or extra arguments also return status two. These checks exercise valid boundaries and rejected inputs in the actual final image.

Inspection confirms one filesystem layer in that final image and the intended nonroot numeric user. Exporting a created container shows /square, with no compiler or source file at the builder’s paths. The export is inspected as an archive; it is not extracted over the host filesystem. This verifies final composition independently of whether the Dockerfile merely looks minimal.

Docker reported 109,490,331 bytes for the builder image and 312,508 bytes for the static final image in the recorded run. These are the engine’s image-size fields, not a measurement of unique physical disk allocation or network transfer time. The useful invariant is that the final image excludes the toolchain and is smaller under the same reported accounting method.

The dynamic final target also builds successfully, but starting it fails with a missing-file error because its interpreter/runtime was not copied. This is the critical negative test. The failure is expected and asserted; it disproves the idea that a successful final COPY automatically tells the builder which runtime libraries should be added.

Validate what the last stage actually runs

Choose an appropriate runtime image for the application. Static linking fits this small calculation, but does not prove that every statically linked service needs no external files or runtime-loaded components. For a dynamic application, supply compatible runtime libraries or a suitable runtime base and execute representative functionality inside the final artifact.

Use named stages and narrow COPY paths so reviewers can see the handoff. Avoid copying a broad builder directory that silently includes tests, credentials, caches, or unused tools. Set working directories, writable paths, user identity, and startup arguments according to the runtime’s real requirements.

Test the final stage after dependency or compiler updates, including errors and less common paths that load extra resources. Keep the selected artifact identity with the results. A smaller image is a useful outcome when it still fulfills the service contract; build success alone cannot establish that contract.

Sources

Docker multi-stage builds, named stages, target selection, and copying selected artifacts, checked 2026-09-12.
Dockerfile reference, COPY –from, scratch, USER, and exec-form entrypoints, checked 2026-09-12.
execve(2), missing executable interpreters and execution errors, checked 2026-09-12.

Containers · XB Field Notes

A veth pair carries frames between two endpoints; routing supplies their meaning

When it breaks

Check it quickly

Diagnose the layers in order

Sources

Security context defines constraints that need runtime evidence

Put each setting at the right scope

When it breaks

Check it quickly

Sources

Signed images prove who built them

Digests are the identity, tags are pointers

Signing a tag binds nothing

Verification is the control, not the signature

When it breaks

Check it quickly

Sources

A pinned base fixes one input; the release needs the rest

When it breaks

Check it quickly

Record a release, then replace its instances

Further reading

Sources

Regular init containers establish startup prerequisites

Sequence is local to one Pod

When it breaks

Check it quickly

Sources

A container combines process isolation with a shared Linux kernel

When it breaks

Check it quickly

Inspect the boundary that was actually configured

Sources

cgroup v2 enforces different budgets with different failure modes

When it breaks

Check it quickly

Size and observe the whole workload

Sources

Multi-stage builds need a tested runtime boundary

When it breaks

Check it quickly

Validate what the last stage actually runs

Sources