Cache · XB Field Notes

dnsmasq separates forwarded cache entries from configured local names

Sat, 15 Apr 2028 00:00:00 +0000

dnsmasq can forward DNS questions upstream, retain supported answers in a bounded cache, and serve names configured locally. These are different sources of data. A repeated answer may be a cache hit or a local configuration result. Establishing which source supplied it is essential when a hostname disagrees with the authority or continues resolving after a cache flush.

When it breaks

Starting dnsmasq does not automatically make every application use it. Clients must direct their DNS queries through the intended listener, and some applications use other resolvers. Likewise, a correct local listener can forward to the wrong upstream or read unexpected host configuration. Diagnose the actual query path before enlarging the cache or restarting a daemon that never received the failing query.

A local name can deliberately override upstream data. Clearing the forwarding cache does not delete that configured name, so repeated flushing cannot make the upstream address win. Treat a local record as configuration with an owner and purpose. An answer with local TTL zero can still be returned repeatedly because dnsmasq generates it from configuration rather than retaining a forwarded response.

Cache capacity and freshness solve different problems. A larger cache can retain more eligible entries, but it does not extend their valid TTLs or make every possible query answerable during an upstream outage. New names and expired entries still need the appropriate resolution path. Cache-size tuning also does not directly specify the maximum number of simultaneous upstream queries or reserve application-level availability.

Check it quickly

Save the script as check-dnsmasq.py and run python3 check-dnsmasq.py with Docker. The pinned environment uses dnsmasq 2.90 and BIND 9.18.49. The test forwarders ignore ambient resolver and hosts files, bind explicit loopback listeners, and forward only to the disposable authority. They publish no host ports and use no DHCP configuration:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends bind9=1:9.18.49-1~deb12u2 bind9-dnsutils=1:9.18.49-1~deb12u2 dnsmasq-base=2.90-4~deb12u2 python3-dnspython=2.3.0-1 && rm -rf /var/lib/apt/lists/*\nCMD ["/usr/bin/python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,re,signal,subprocess,time
import dns.exception,dns.message,dns.query,dns.rcode,dns.rdatatype

def run(*args,check=True):
 result=subprocess.run(args,capture_output=True,text=True,timeout=15)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def query(name,kind='A',port=1053):
 return dns.query.udp(dns.message.make_query(name,kind),'127.0.0.1',port=port,timeout=1)
def values(response,kind='A'):
 wanted=dns.rdatatype.from_text(kind)
 return [item.to_text() for rrset in response.answer if rrset.rdtype==wanted for item in rrset]
def show(response):
 def section(records):return [{'name':str(rr.name),'type':dns.rdatatype.to_text(rr.rdtype),'ttl':rr.ttl,'data':[item.to_text() for item in rr]} for rr in records]
 return {'rcode':dns.rcode.to_text(response.rcode()),'answer':section(response.answer),'authority':section(response.authority)}
def wait(predicate,timeout=15):
 deadline=time.monotonic()+timeout
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except (OSError,dns.exception.Timeout):pass
 time.sleep(.05)
 raise AssertionError('DNS fixture condition did not become true')
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=5)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=5)

@contextmanager
def lab(records,soa_ttl=8,minimum=4,extra=''):
 with TemporaryDirectory(prefix='xbcatm-dns-') as directory,ExitStack() as stack:
 root=Path(directory);zone=root/'zone';conf=root/'named.conf';log=root/'named.log';serial=1
 def save(body):
 zone.write_text(f'$ORIGIN lab.test.\n$TTL 30\n@ {soa_ttl} IN SOA ns.lab.test. hostmaster.lab.test. ({serial} 60 60 3600 {minimum})\n@ IN NS ns.lab.test.\nns IN A 127.0.0.1\n'+body+'\n')
 run('named-checkzone','lab.test',str(zone))
 save(records)
 conf.write_text(f'''options {{ directory "{root}"; listen-on port 1053 {{127.0.0.1;}}; listen-on-v6 {{none;}};
recursion no; dnssec-validation no; notify no; querylog yes; max-cache-size 8m; pid-file "{root}/named.pid"; session-keyfile none;
{extra} }};
controls {{}};
zone "lab.test" {{ type primary; file "{zone}"; }};
''')
 run('named-checkconf',str(conf))
 output=stack.enter_context(log.open('w'))
 authority=subprocess.Popen(['named','-g','-c',str(conf),'-n','1','-U','1'],stdout=output,stderr=subprocess.STDOUT)
 stack.callback(stop,authority)
 def ready():
 assert authority.poll() is None,log.read_text()
 response=query('lab.test.','SOA');return response.rcode()==0 and response.answer[0][0].serial==serial
 wait(ready)
 def update(body):
 nonlocal serial
 serial+=1;save(body);authority.send_signal(signal.SIGHUP);wait(ready)
 def cache(port=1054,upstream=1053,size=150,options=()):
 logfile=root/('cache-'+str(port)+'.log');stream=stack.enter_context(logfile.open('w'))
 command=['dnsmasq','--no-daemon','--conf-file=/dev/null','--no-resolv','--no-hosts','--bind-interfaces',
 '--listen-address=127.0.0.1','--port='+str(port),'--server=127.0.0.1#'+str(upstream),
 '--cache-size='+str(size),'--log-queries','--log-facility=-',*options]
 process=subprocess.Popen(command,stdout=stream,stderr=subprocess.STDOUT);stack.callback(stop,process)
 def cache_ready():
 assert process.poll() is None,logfile.read_text()
 return query('lab.test.','SOA',port).rcode()==0
 wait(cache_ready);return process,logfile
 def queries(name,kind='A'):
 return sum(bool(re.search(r'query: '+re.escape(name.rstrip('.'))+r' IN '+kind+r'\b',line)) for line in log.read_text().splitlines())
 yield root,update,cache,queries
with lab('cached 20 IN A 127.0.0.2\nuncached 20 IN A 127.0.0.3\nlocal 20 IN A 127.0.0.4') as (_,update,cache,queries):
 daemon,log=cache(options=('--host-record=local.lab.test,127.0.0.9',))
 for _ in range(3):assert values(query('cached.lab.test.',port=1054))==['127.0.0.2']
 wait(lambda: queries('cached.lab.test')==1)
 cache(port=1055,size=0)
 for _ in range(3):assert values(query('uncached.lab.test.',port=1055))==['127.0.0.3']
 wait(lambda: queries('uncached.lab.test')==3)
 local=query('local.lab.test.',port=1054);assert values(local)==['127.0.0.9'] and local.answer[0].ttl==0
 assert queries('local.lab.test')==0
 authoritative=query('local.lab.test.');assert values(authoritative)==['127.0.0.4']
 wait(lambda: queries('local.lab.test')==1)
 update('cached 20 IN A 127.0.0.8\nuncached 20 IN A 127.0.0.3\nlocal 20 IN A 127.0.0.4')
 stale=query('cached.lab.test.',port=1054);assert values(stale)==['127.0.0.2'] and queries('cached.lab.test')==1
 previous=log.read_text().count('cleared cache');daemon.send_signal(signal.SIGHUP)
 wait(lambda: log.read_text().count('cleared cache')>previous)
 refreshed=query('cached.lab.test.',port=1054);assert values(refreshed)==['127.0.0.8']
 wait(lambda: queries('cached.lab.test')==2)
 retained=query('local.lab.test.',port=1054);assert values(retained)==['127.0.0.9']
 print(json.dumps({'three_lookups_upstream_counts':{'cache_enabled':1,'cache_disabled':3},'local_record':show(local),
 'same_name_authority':show(authoritative),'before_SIGHUP':show(stale),'after_SIGHUP':show(refreshed),
 'configured_local_record_retained':show(retained)},sort_keys=True))
"""
name='xbcatm-dns-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 built=docker('build','-t',name,str(root));assert built.returncode==0,built.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--memory','384m','--pids-limit','128',name,
 '/usr/bin/python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

Three queries for cached.lab.test through the cache-enabled listener all return 127.0.0.2, while the authoritative log records one query. A second listener uses cache-size=0. Three queries for a separate name through it generate three upstream queries. This directly verifies the cache’s effect on forwarding work; the experiment does not infer hits from response speed or assert a particular latency improvement.

The first listener also has host-record=local.lab.test,127.0.0.9. Its answer is 127.0.0.9 with TTL zero, and no upstream query is logged for that lookup. A direct authoritative query for the same name returns 127.0.0.4 and increments the authority’s counter. The differing results show an explicit local override, with a positive control confirming that the query counter detects real upstream traffic.

Next, the authority changes cached.lab.test to 127.0.0.8. The cache still returns its earlier 127.0.0.2 answer within the remaining TTL. The script sends SIGHUP to this specific dnsmasq process and waits for its cache-cleared log entry. The following query reaches the authority and returns 127.0.0.8, increasing the corresponding authoritative count from one to two.

After that same signal, local.lab.test continues returning the configured 127.0.0.9 value. The signal clears forwarded cache state while leaving the explicitly configured local record available. This is why “flush DNS” is incomplete troubleshooting advice: the expected result depends on which process owns which data, and whether that data is cached, configured, or stored somewhere else entirely.

Operate the listener and its data sources deliberately

Inspect the effective configuration, listener address, upstream destinations, and client settings together. Avoid creating a forwarding loop between two local resolvers. The example removes ambient inputs to isolate behavior; a real installation may intentionally use hosts files, DHCP leases, or resolver-management software. Record those dependencies so an automatic rewrite or a local override is visible during troubleshooting.

For dnsmasq, SIGHUP clears its cache and reloads certain data sources described by the manual, but it does not generally reread the main configuration file. Use the documented service procedure for the kind of change being made. A successful cache clear cannot be treated as proof that edited listener, forwarding, or other startup configuration has been activated.

Limit listener exposure to intended clients, especially when forwarding recursion on a reachable network. DHCP, DNSSEC validation, and encrypted upstream transport require their own configuration and verification; none is demonstrated by this cache experiment. Query logging is useful during diagnosis but can reveal client naming activity, so select its scope and retention according to the environment rather than leaving detailed logs indefinitely by habit.

Finally, compare cached answers with the authority when it is the expected source. For a configured local name, compare against its configuration. Track upstream failures, hits, and entry churn separately. Identify whether the discrepancy concerns capacity, lifetime, the upstream service, or a local naming rule.

Sources

dnsmasq manual, forwarding, cache-size, host-record, local TTL, and SIGHUP behavior, checked 2026-09-12.
RFC 1035 domain names, DNS resource records and cached lifetimes, checked 2026-09-12.

A microcache serves fresh entries briefly and coalesces misses explicitly

Sun, 12 Mar 2028 00:00:00 +0000

A microcache keeps reusable responses for a short freshness lifetime, often a few seconds. Requests for the same selected representation can reuse a fresh entry during that window. The response might lag a new origin update, but it is not HTTP-stale merely because it came from a cache. Expiration, permission to serve stale content, and concurrent cache filling are separate policies.

When it breaks

A short TTL does not make a shared response safe for every user. If personalized requests read or populate a generic entry, one user’s representation can be served in another context. proxy_cache_bypass prevents reading from cache under its conditions; proxy_no_cache prevents saving the response. Using only one side can leave the other path inconsistent with the intended privacy boundary.

The cache key also defines which requests share a response. Ignoring a meaningful query or host dimension can merge different representations. Including arbitrary varying data can instead destroy reuse. The fixture below uses one backend authority and preserves the complete request URI, including arguments. A service varying by tenant, language, or other inputs must establish its own representation identity.

Caching alone does not guarantee one origin request during a cold burst. Several requests can miss the same entry before any response is stored. proxy_cache_lock coordinates filling a new entry, with finite timeout and age controls. That lock is not the TTL, and a brief freshness lifetime does not implicitly turn lock behavior on.

Check it quickly

Save the script as check-microcache.py and run it with Python, Docker, and the pinned images. The recorded environment used nginx 1.28.0 on Linux arm64. A disposable backend reports per-path request counts, and nginx uses a two-second validity setting with explicit cookie and authorization bypass conditions. The listener is restricted to host loopback:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True)
 target.write_bytes(content if isinstance(content,bytes) else content.encode())
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 docker('run','-d','--name',containers[0],'--network',name,'-p','127.0.0.1::8080',tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 request.port=port
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'
from concurrent.futures import ThreadPoolExecutor
import threading
backend=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
from urllib.parse import urlsplit,parse_qs
import json,threading
lock=threading.Lock();counts={};gates={'/lock':threading.Event(),'/nolock':threading.Event()}
class Handler(BaseHTTPRequestHandler):
 protocol_version='HTTP/1.1'
 def log_message(self,*args):pass
 def do_GET(self):
 parsed=urlsplit(self.path)
 if parsed.path=='/control/state':
 with lock:data=dict(counts)
 elif parsed.path=='/control/release':
 gates[parse_qs(parsed.query)['path'][0]].set();data={'released':True}
 else:
 with lock:counts[self.path]=counts.get(self.path,0)+1;count=counts[self.path]
 if parsed.path in gates:assert gates[parsed.path].wait(15)
 data={'path':self.path,'count':count,'cookie':self.headers.get('Cookie'),'authorization_present':bool(self.headers.get('Authorization'))}
 body=json.dumps(data,sort_keys=True).encode();self.send_response(200)
 if parsed.path=='/cookie':self.send_header('Set-Cookie','fixture=public-example; Path=/')
 if parsed.path=='/private':self.send_header('Cache-Control','private')
 if parsed.path=='/no-store':self.send_header('Cache-Control','no-store')
 self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
ThreadingHTTPServer(('0.0.0.0',8081),Handler).serve_forever()
'''
extra='''
proxy_cache_path /tmp/cache levels=1:2 keys_zone=pages:1m inactive=30s max_size=10m;
map $http_cookie $skip_cookie { default 1; "" 0; }
map $http_authorization $skip_auth { default 1; "" 0; }
proxy_cache pages; proxy_cache_key "$scheme|$proxy_host|$request_uri"; proxy_cache_valid 200 2s;
proxy_cache_bypass $skip_cookie $skip_auth; proxy_no_cache $skip_cookie $skip_auth;
proxy_cache_lock on; proxy_cache_lock_timeout 5s; proxy_cache_lock_age 5s;
add_header X-Cache $upstream_cache_status always;
'''
locations='''
location /control/ { proxy_cache off; proxy_pass http://backend:8081; }
location = /nolock { proxy_cache_lock off; proxy_pass http://backend:8081; }
location / { proxy_pass http://backend:8081; }
'''
with lab(config(locations,extra),backend=backend) as (_,request):
 def get(path,headers=None):
 status,h,b=request(path,headers=headers);assert status==200;return h.get('x-cache'),json.loads(b)
 results={}
 first=get('/item?x=1');second=get('/item?x=1');assert first[0]=='MISS' and second==('HIT',first[1])
 other=get('/item?x=2');assert other[0]=='MISS' and other[1]['path']!=first[1]['path']
 results['fresh']=[first[0],second[0]];results['different_query']=other[0]
 for name,headers in [('cookie',{'Cookie':'session=public-fixture'}),('authorization',{'Authorization':'Bearer public-fixture'})]:
 private=get('/item?x=1',headers);assert private[0]=='BYPASS' and private[1]!=first[1]
 assert get('/item?x=1')==('HIT',first[1])
 new='/new-'+name;assert get(new,headers)[0]=='BYPASS'
 public=get(new);assert public[0]=='MISS' and public[1]['cookie'] is None and not public[1]['authorization_present']
 results[name]={'warm_key':'BYPASS','public_after_private_miss':public[0]}
 for path in ['/cookie','private','no-store']:
 path='/'+path.lstrip('/');a=get(path);b=get(path)
 assert a[0]==b[0]=='MISS' and a[1]['count']==1 and b[1]['count']==2
 results[path]=[a[0],b[0]]
 deadline=time.monotonic()+6
 while time.monotonic()<deadline:
 value=get('/item?x=2')
 if value[0]=='EXPIRED':break
 assert value[0]=='HIT';time.sleep(.15)
 else:raise AssertionError('TTL did not expire')
 assert value[1]['count']==2;results['after_ttl']={'status':value[0],'count':value[1]['count']}
 for path,number,origin_count in [('/lock',6,1),('/nolock',4,4)]:
 barrier=threading.Barrier(number+1)
 def client():barrier.wait();return get(path)
 with ThreadPoolExecutor(max_workers=number) as pool:
 futures=[pool.submit(client) for _ in range(number)];barrier.wait()
 wait(lambda:get('/control/state')[1].get(path)==origin_count)
 time.sleep(.3);before=get('/control/state')[1][path]
 assert before==origin_count and not any(f.done() for f in futures)
 get('/control/release?path='+path);responses=[f.result(timeout=10) for f in futures]
 assert get('/control/state')[1][path]==origin_count
 statuses=sorted(r[0] for r in responses)
 if path=='/lock':assert statuses==['HIT']*5+['MISS'] and all(r[1]==responses[0][1] for r in responses)
 else:assert statuses==['MISS']*4
 results[path]={'origin_requests_while_held':before,'cache_statuses':statuses}
 print(json.dumps(results,sort_keys=True))

The first public request to /item?x=1 reports MISS, and the next reports HIT with the same backend result. /item?x=2 has its own MISS and distinct path in the response. Polling that second entry eventually produces EXPIRED and a backend count of two. This observes actual expiration using the server’s clock; it does not assume an exact subsecond transition time.

Requests carrying the example session cookie or authorization marker report BYPASS even when a public entry is already warm. A later public request still returns the original public HIT. A second control first visits an unused key with those headers, then visits it publicly: the public request is MISS, proving that the preceding private-context response did not populate a reusable entry.

Backend responses containing Set-Cookie, Cache-Control: private, or Cache-Control: no-store each produce two successive MISS results and increasing origin counts. The configuration does not ignore those response controls. This distinguishes client-side bypass policy from origin-provided storage restrictions; the short TTL is not used as an excuse to override either privacy signal.

For a cold locked entry, six clients start together while the backend deliberately withholds the response. The backend sees one request before release; afterward, one client reports MISS and five report HIT with the same result. For a separate entry with the lock disabled, four held clients create four origin requests and all report MISS. The gate makes overlapping misses directly observable.

Choose freshness and overload behavior separately

The two-second setting applies to eligible 200 responses in this experiment. Relevant origin cache headers can take priority over configured validity. Decide which responses are shareable before selecting the lifetime, and keep the cache key aligned with the inputs that actually change their content. A high hit ratio is not evidence that the selected representation was correct.

Lock waiters exchange extra waiting for reduced origin work. This fixture releases the origin within the configured lock limits; it does not establish a permanent single-request guarantee when those limits expire or a backend fails. Review lock timeout, lock age, and any stale-serving policy together with the service’s acceptable latency and failure behavior.

Observe cache status alongside origin counts and response identity. MISS means the requested representation was not served as a cache hit, while BYPASS identifies the explicit read exclusion shown here. Fresh HIT, EXPIRED refill, and intentionally stale service should not be collapsed into one “cached” result. Their different effects on correctness and backend load are the reason to measure them separately.

Sources

nginx proxy module, cache validity, cache locks, bypass, no-cache, and response header controls, checked 2026-09-12.
nginx content caching guide, cache keys, status observation, and caching behavior, checked 2026-09-12.
RFC 9111 HTTP Caching, fresh versus stale responses and private/no-store directives, checked 2026-09-12.

DNS TTL trades repeated upstream queries against retained answers

Wed, 01 Mar 2028 00:00:00 +0000

A DNS TTL is a lifetime attached to cached data, not a refresh job scheduled for every second. A cache can reuse a valid answer while it retains that entry. After expiry, a subsequent request usually requires fresh resolution unless another configured behavior applies. An idle record need not produce an upstream query merely because its TTL has reached zero.

When it breaks

Shortening TTL increases potential upstream work when clients keep asking for the same data. It does not multiply all authoritative traffic by one universal factor: query frequency, resolver sharing, cache capacity, prefetching, and record diversity affect the result. A short TTL on one popular name behaves differently from a large stream of names that are each queried only once.

A long TTL permits continued use of an answer after the authority changes its data. That cached answer may still be valid under the lifetime originally received even though it no longer matches the newest zone content. Calling every mismatch an expired or broken cache obscures the actual contract. TTL expresses permitted retention; it does not promise continuous synchronization with the authority.

Multiple forwarding layers do not ordinarily receive a fresh copy of the original TTL at every hop. A cache returns remaining lifetime, which the next cache then ages. Adding full TTLs per layer invents a propagation delay that ordinary compliant forwarding does not create. Local TTL overrides and serve-stale policies are separate behaviors that should be identified explicitly when present.

Check it quickly

Save this as check-ttl-cost.py and run python3 check-ttl-cost.py with Docker. The pinned container uses BIND 9.18.49 and dnsmasq 2.90, with no external runtime network. The authoritative server logs actual queries. One forwarder initially serves two names, and the script later starts a second forwarder behind the first to inspect the TTL handed downstream:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends bind9=1:9.18.49-1~deb12u2 bind9-dnsutils=1:9.18.49-1~deb12u2 dnsmasq-base=2.90-4~deb12u2 python3-dnspython=2.3.0-1 && rm -rf /var/lib/apt/lists/*\nCMD ["/usr/bin/python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,re,signal,subprocess,time
import dns.exception,dns.message,dns.query,dns.rcode,dns.rdatatype

def run(*args,check=True):
 result=subprocess.run(args,capture_output=True,text=True,timeout=15)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def query(name,kind='A',port=1053):
 return dns.query.udp(dns.message.make_query(name,kind),'127.0.0.1',port=port,timeout=1)
def values(response,kind='A'):
 wanted=dns.rdatatype.from_text(kind)
 return [item.to_text() for rrset in response.answer if rrset.rdtype==wanted for item in rrset]
def show(response):
 def section(records):return [{'name':str(rr.name),'type':dns.rdatatype.to_text(rr.rdtype),'ttl':rr.ttl,'data':[item.to_text() for item in rr]} for rr in records]
 return {'rcode':dns.rcode.to_text(response.rcode()),'answer':section(response.answer),'authority':section(response.authority)}
def wait(predicate,timeout=15):
 deadline=time.monotonic()+timeout
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except (OSError,dns.exception.Timeout):pass
 time.sleep(.05)
 raise AssertionError('DNS fixture condition did not become true')
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=5)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=5)

@contextmanager
def lab(records,soa_ttl=8,minimum=4,extra=''):
 with TemporaryDirectory(prefix='xbcatm-dns-') as directory,ExitStack() as stack:
 root=Path(directory);zone=root/'zone';conf=root/'named.conf';log=root/'named.log';serial=1
 def save(body):
 zone.write_text(f'$ORIGIN lab.test.\n$TTL 30\n@ {soa_ttl} IN SOA ns.lab.test. hostmaster.lab.test. ({serial} 60 60 3600 {minimum})\n@ IN NS ns.lab.test.\nns IN A 127.0.0.1\n'+body+'\n')
 run('named-checkzone','lab.test',str(zone))
 save(records)
 conf.write_text(f'''options {{ directory "{root}"; listen-on port 1053 {{127.0.0.1;}}; listen-on-v6 {{none;}};
recursion no; dnssec-validation no; notify no; querylog yes; max-cache-size 8m; pid-file "{root}/named.pid"; session-keyfile none;
{extra} }};
controls {{}};
zone "lab.test" {{ type primary; file "{zone}"; }};
''')
 run('named-checkconf',str(conf))
 output=stack.enter_context(log.open('w'))
 authority=subprocess.Popen(['named','-g','-c',str(conf),'-n','1','-U','1'],stdout=output,stderr=subprocess.STDOUT)
 stack.callback(stop,authority)
 def ready():
 assert authority.poll() is None,log.read_text()
 response=query('lab.test.','SOA');return response.rcode()==0 and response.answer[0][0].serial==serial
 wait(ready)
 def update(body):
 nonlocal serial
 serial+=1;save(body);authority.send_signal(signal.SIGHUP);wait(ready)
 def cache(port=1054,upstream=1053,size=150,options=()):
 logfile=root/('cache-'+str(port)+'.log');stream=stack.enter_context(logfile.open('w'))
 command=['dnsmasq','--no-daemon','--conf-file=/dev/null','--no-resolv','--no-hosts','--bind-interfaces',
 '--listen-address=127.0.0.1','--port='+str(port),'--server=127.0.0.1#'+str(upstream),
 '--cache-size='+str(size),'--log-queries','--log-facility=-',*options]
 process=subprocess.Popen(command,stdout=stream,stderr=subprocess.STDOUT);stack.callback(stop,process)
 def cache_ready():
 assert process.poll() is None,logfile.read_text()
 return query('lab.test.','SOA',port).rcode()==0
 wait(cache_ready);return process,logfile
 def queries(name,kind='A'):
 return sum(bool(re.search(r'query: '+re.escape(name.rstrip('.'))+r' IN '+kind+r'\b',line)) for line in log.read_text().splitlines())
 yield root,update,cache,queries
with lab('short 1 IN A 127.0.0.2\nlong 20 IN A 127.0.0.3\nchain 6 IN A 127.0.0.4') as (_,update,cache,queries):
 cache();samples=[]
 for i in range(3):
 pair={name:show(query(name+'.lab.test.',port=1054)) for name in ('short','long')};samples.append(pair)
 if i<2:time.sleep(1.2)
 wait(lambda: queries('short.lab.test')==3 and queries('long.lab.test')==1)
 first=query('chain.lab.test.',port=1054);assert first.answer[0].ttl==6
 wait(lambda: queries('chain.lab.test')==1)
 time.sleep(2.2);cache(port=1055,upstream=1054)
 downstream=query('chain.lab.test.',port=1055);upstream=query('chain.lab.test.',port=1054)
 assert values(downstream)==['127.0.0.4'] and 0<downstream.answer[0].ttl<6
 assert abs(downstream.answer[0].ttl-upstream.answer[0].ttl)<=1
 assert queries('chain.lab.test')==1
 update('short 1 IN A 127.0.0.2\nlong 20 IN A 127.0.0.3\nchain 6 IN A 127.0.0.5')
 wait(lambda: values(query('chain.lab.test.',port=1055))==['127.0.0.5'],timeout=8)
 wait(lambda: queries('chain.lab.test')==2)
 print(json.dumps({'three_lookups_each':samples,'authority_query_counts':{'short':queries('short.lab.test'),'long':queries('long.lab.test')},
 'cache_chain':{'original':show(first),'downstream_first':show(downstream),'upstream_remaining':show(upstream),'authority_queries_after_expiry':queries('chain.lab.test')}},sort_keys=True))
"""
name='xbcatm-dns-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 built=docker('build','-t',name,str(root));assert built.returncode==0,built.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--memory','384m','--pids-limit','128',name,
 '/usr/bin/python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The short.lab.test record has TTL 1 and long.lab.test has TTL 20. The program queries each three times, waiting 1.2 seconds between rounds. The authoritative log records three queries for the short-lived record and one for the long-lived record. Both produce successful answers on every round. The difference is upstream work under this controlled request schedule, not a measured application-latency improvement.

The returned TTL for the long-lived record decreases across rounds, while each refreshed short-lived answer starts at 1 again. The script checks the server log rather than assuming a fast response means a cache hit. On loopback, even an uncached query can be quick, so response time alone would not establish whether the authority was contacted or the answer came from memory.

For the layered case, the first forwarder caches chain.lab.test with TTL 6. After 2.2 seconds, a newly started second forwarder queries through it. The recorded downstream answer has TTL 3, matching the upstream remaining value within integer-second timing. It does not regain TTL 6. The authoritative count remains one, confirming that this downstream fill reused the first forwarder’s cached answer.

The authority then changes the chain record’s address. Queries through the second forwarder eventually receive the new address, and the authoritative count rises to two. This checks successful refresh through both layers after the retained data expires. The exact expiry fraction is implementation-dependent; the asserted properties are preserved remaining lifetime, no premature extra authoritative query, and eventual acquisition of the changed value.

Choose TTL from observed demand and change needs

Use expected change frequency and acceptable old-answer overlap alongside query volume. A busy shared resolver can amortize one answer across many clients, whereas isolated caches duplicate fills. Eviction can make an entry disappear before its TTL, so a long configured lifetime does not guarantee a hit. Measure hits, misses, upstream requests, and available capacity instead of treating TTL as a cache-size control.

For a planned move, lower TTL early enough for earlier longer-lived copies to expire. Keep the destination overlap aligned with the actual retained data. Raising TTL after stability reduces repeated resolution for busy records, but it also extends the next change’s ordinary waiting window. The relevant unit is the RRset: records of the same name, class, and type should have a consistent TTL.

Finally, compare answer data and remaining lifetime at each resolver actually used by the application. A direct authoritative query and a cached recursive answer reveal different states. Application caches and existing connections may add their own behavior outside this DNS experiment. Separate those lifetimes before estimating how quickly users will discover a changed endpoint or how much authoritative capacity a TTL adjustment requires.

Sources

RFC 1035 domain names, TTL as a cache lifetime and treatment of cached records, checked 2026-09-12.
RFC 2181 DNS clarifications, TTL limits and consistent lifetime within an RRset, checked 2026-09-12.
dnsmasq manual, cache sizing, forwarding, and TTL options, checked 2026-09-12.

Docker rebuilds reuse recorded work, including stale work

Sun, 20 Feb 2028 00:00:00 +0000

Docker’s build cache stores results of build operations so a later build can reuse work whose relevant inputs still match. A cache hit does not mean the command ran again and produced identical output. It means the builder accepted an existing result. That distinction matters for package installation, generated timestamps, remote downloads, and any operation whose output could change without a corresponding change to its declared inputs.

When it breaks

An unchanged RUN apt-get update can reuse yesterday’s result without contacting a repository. The builder does not routinely inspect that remote repository to decide whether a cached RUN remains fresh. A successful cached build therefore provides no independent evidence that packages received the newest available security updates. Refresh policy must be deliberate and compatible with the project’s reproducibility requirements.

COPY has different inputs from a plain command string. File contents and relevant metadata participate in its cache calculation, but source modification time alone does not. A deployment script that merely touches a file can leave the dependent RUN cached. Conversely, changing an executable permission can invalidate a COPY even when the text remains identical, causing dependent work to execute again.

The base image, build arguments used by a step, and preceding dependencies also matter. Cache invalidation follows dependencies; treating a complex multistage build as one unavoidable linear rebuild can be misleading. In the simple single-stage chain below, changing the copied file invalidates the following timestamp-producing RUN, while the fixed base filesystem layers remain reusable.

Check it quickly

Run the script with host Python and access to a disposable Docker builder. Pull the named digest first if it is unavailable locally. The recorded run used Docker 29.3.1, Buildx 0.32.1, and a Linux arm64 Python 3.11.16 base. Builds and containers disable networking. Temporary tags are removed, but the script deliberately leaves unrelated images and builder caches alone:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,shlex,subprocess,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(["docker",*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def inspect(image):return json.loads(docker("image","inspect",image).stdout)[0]

name='xbcatm-cache-'+uuid.uuid4().hex[:10];tag=name+':probe'
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);app=root/'app.txt';app.write_text('initial '+name);app.chmod(0o644)
 build_code="from pathlib import Path;import time;Path('/built-at').write_text(str(time.time_ns()))"
 (root/'Dockerfile').write_text(f'FROM {BASE}\nCOPY app.txt /app.txt\nRUN python -c {shlex.quote(build_code)}\n')
 def build(*options):return docker('build','--network','none','--progress','plain',*options,'-t',tag,str(root))
 def stamp():return docker('run','--rm','--network','none',tag,'python','-c',"from pathlib import Path;print(Path('/built-at').read_text())").stdout.strip()
 build();initial=stamp();base_layers=inspect(BASE)['RootFS']['Layers']
 result=build();assert stamp()==initial and 'CACHED' in result.stderr
 timestamp=app.stat().st_mtime_ns+10_000_000_000;os.utime(app,ns=(timestamp,timestamp))
 build();assert stamp()==initial
 print('unchanged inputs reuse the RUN result; changing only source mtime also leaves that result cached')
 app.chmod(0o755);build();mode_changed=stamp();assert mode_changed!=initial
 app.write_text('changed '+name);build();content_changed=stamp();assert content_changed!=mode_changed
 print('changing copied-file permissions or content invalidates COPY and reruns the dependent RUN')
 build('--no-cache');assert stamp()!=content_changed
 assert inspect(tag)['RootFS']['Layers'][:len(base_layers)]==base_layers
 print('no-cache reruns the instruction while this pinned base retains the same filesystem layer prefix')
finally:docker('image','rm',tag,check=False)

The first build writes a nanosecond timestamp into /built-at. A second identical build reports a cached operation and returns exactly the same timestamp. This is stronger evidence than comparing elapsed build times: a quick build could still have executed the command, while the retained timestamp identifies the already-recorded result that was reused.

The script then advances only the source file’s modification time. The timestamp inside the image remains unchanged. This demonstrates the documented mtime exclusion directly; it does not imply that all filesystem metadata is ignored. The following change from mode 0644 to 0755 causes a new timestamp, establishing that the executable-bit difference affects this COPY dependency.

Changing the file’s text produces another timestamp. The test uses a uniquely named input so an old experiment cannot accidentally supply the initial application result. Its assertions compare actual image contents after each build, rather than treating a generic CACHED message somewhere in the build log as proof that the particular operation under discussion was reused.

Finally, --no-cache makes the RUN execute again with unchanged application inputs. Inspection still shows the same pinned base-layer prefix. This option disables reuse for build instructions; it does not replace an explicitly selected base digest with a newer one. Obtaining a changed base requires choosing the intended artifact, and floating-tag refresh has its own pull policy.

Make freshness an explicit input

Place stable dependency inputs before frequently edited source, and use narrow COPY instructions so unrelated files do not become dependencies. Pin package versions or verified artifacts when repeatability is required. For intentional dependency refreshes, update those declarations and review the resulting artifact instead of assuming that a cache miss is itself a security policy.

Secret mounts need separate attention: changing a secret’s contents does not automatically invalidate the build cache. If the secret controls a fetch whose output must be refreshed, use a documented nonsecret revision input or explicit invalidation. Never embed the credential in an argument merely to force a new cache key.

Treat cache-hit logs as performance evidence and executed tests as behavioral evidence. Record the final artifact identity, test it, and retain enough build inputs to explain how it was produced. A fast rebuild is useful only when the reused work is still the work the release intends to ship.

Sources

Docker build cache invalidation, cache inputs, modification times, RUN reuse, and explicit invalidation, checked 2026-09-12.
Dockerfile reference, COPY metadata, RUN instructions, and build arguments, checked 2026-09-12.

nginx validators are only as sound as their representation identity

Thu, 17 Feb 2028 00:00:00 +0000

A validator lets a recipient ask whether its stored representation still matches the selected one. A successful conditional GET can return 304 without retransmitting the body. Freshness is a separate decision: it determines when a stored response can be reused without that request. An ETag does not itself specify a cache lifetime, and a 304 is not a replacement body.

When it breaks

Calling every ETag a body hash hides an important nginx detail. In the tested 1.28.0 static-file implementation, the generated tag combines modification time and content length. It does not hash file contents. Replacing bytes while preserving both inputs can retain the same tag, allowing a client holding different bytes to receive an apparently successful validation.

Weak and strong validators also serve different comparisons. A strong validator must distinguish representation data changes for uses requiring byte identity. A weak validator can group representations considered semantically equivalent, even when their bytes differ. The W/ prefix does not mean “byte-equivalent changes,” nor does it make the tag suitable for every precondition or resumable byte-range operation.

Conditional requests do not guarantee that application work was avoided. A server might compute the representation before deciding that its validator matches. Static nginx handling has no application renderer in this example. A smaller response proves saved body transfer here; it does not establish a universal reduction in database queries, rendering cost, or origin request count.

Check it quickly

Save the script as check-validators.py, obtain the pinned nginx image, and run it with Python and Docker. It serves a disposable four-byte static file through host loopback using nginx 1.28.0. File times are deliberately controlled inside the experiment container; the host and Docker VM clocks are not changed:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True)
 target.write_bytes(content if isinstance(content,bytes) else content.encode())
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 docker('run','-d','--name',containers[0],'--network',name,'-p','127.0.0.1::8080',tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 request.port=port
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'
files={'asset.txt':'AAAA','noetag.txt':'AAAA'}
with lab(config('location = /noetag.txt { etag off; }'),files) as (proxy,request):
 def timestamp(value):docker('exec',proxy,'touch','-m','-d','@'+str(value),'/srv/asset.txt')
 timestamp(1577836800)
 status,headers,body=request('/asset.txt');assert status==200 and body==b'AAAA'
 tag=headers['etag'];modified=headers['last-modified'];results={'etag':tag,'last_modified':modified}
 cases={
 'matching_tag':({'If-None-Match':tag},304),
 'weak_comparison':({'If-None-Match':'W/'+tag},304),
 'matching_date':({'If-Modified-Since':modified},304),
 'tag_takes_precedence':({'If-None-Match':'"different"','If-Modified-Since':modified},200),
 'weak_if_match_rejected':({'If-Match':'W/'+tag},412),
 }
 for name,(conditions,expected) in cases.items():
 status,_,body=request('/asset.txt',headers=conditions);assert status==expected,(name,status)
 if status==304:assert body==b''
 results[name]=status
 docker('exec',proxy,'sh','-c','printf BBBB > /srv/asset.txt');timestamp(1577836800)
 status,current,body=request('/asset.txt');assert status==200 and body==b'BBBB' and current['etag']==tag
 status,_,body=request('/asset.txt',headers={'If-None-Match':tag});assert status==304 and body==b''
 results['changed_bytes_preserved_metadata']=status
 timestamp(1577836802)
 status,current,body=request('/asset.txt',headers={'If-None-Match':tag})
 assert status==200 and body==b'BBBB' and current['etag']!=tag and current['last-modified']!=modified
 results['changed_metadata']={'status':status,'etag':current['etag']}
 status,current,_=request('/noetag.txt');assert status==200 and 'etag' not in current
 print(json.dumps(results,sort_keys=True))

The initial file contains AAAA, has modification time 2020-01-01, and produces ETag "5e0be100-4". A GET carrying that tag in If-None-Match returns 304 with an empty body. Adding W/ to the request tag still returns 304 because this condition uses weak comparison. An equal If-Modified-Since date also returns 304 in the tested configuration.

When a nonmatching If-None-Match accompanies the matching date, the response is 200. The entity-tag condition takes precedence over the date condition; a client cannot make the mismatching tag disappear by also supplying a favorable timestamp. The separate If-Match check rejects the weak request tag with 412. That test is a GET precondition check, not an executed write workflow.

Next, the script changes AAAA to BBBB and restores the original modification time. An unconditional GET proves the bytes changed, while the tag remains identical. Sending the old tag now yields 304 again. This deliberate counterexample demonstrates why metadata preservation during deployment can violate the intended identity contract, even though nginx follows its configured tag-generation algorithm.

Advancing the file time by two seconds changes the generated tag and Last-Modified value. The old entity tag then receives 200 with BBBB. The final control disables etag on another resource and confirms that no ETag header is generated there. Disabling tags removes that validator; it does not automatically provide a stronger replacement or disable every form of caching.

Make deployments preserve the validator contract

For assets whose bytes must remain addressable independently, content-derived filenames can separate versions at the URL level. For stable URLs, ensure deployment metadata changes when representation bytes change, or deliberately supply an appropriate content-derived validator through the serving application. Review rapid same-length updates as well as tools that preserve timestamps while copying new content.

Treat Last-Modified with similar care. It represents a timestamp with limited precision, not a cryptographic identity. Do not assume every date is a strong validator for range resumption. The relevant protocol conditions and the server’s update behavior matter, especially when multiple revisions can occur within the timestamp’s effective resolution.

Finally, test the representation actually delivered through intermediaries and content codings. A validator for one selected representation must not falsely identify another byte sequence where strong comparison is required. Keep cache lifetime, validation, privacy, and byte integrity as explicit decisions, and verify both the unchanged and changed cases during deployment checks.

Sources

RFC 9110 HTTP Semantics, strong and weak validators, conditional precedence, and 304/412 responses, checked 2026-09-12.
RFC 9111 HTTP Caching, freshness, stored responses, and successful validation, checked 2026-09-12.
nginx 1.28.0 core source, the static ETag calculation from modification time and content length, checked 2026-09-12.
nginx HTTP core module, etag and if_modified_since configuration, checked 2026-09-12.

Postgres uses two cache layers

Thu, 27 Jan 2028 00:00:00 +0000

For ordinary buffered relation I/O, PostgreSQL’s shared buffers sit above the operating system’s file cache. A shared-buffer hit avoids another read through that lower layer; a shared-buffer miss can still be served from memory by the operating system. The same file data can occupy both caches, so this is not a promise that PostgreSQL and the kernel share one physical copy.

When it breaks

An EXPLAIN buffer count labelled shared read means the block was brought into PostgreSQL’s buffer pool. It does not establish that a storage device performed a physical read. Treating every shared read as a disk miss can make a warm operating-system cache look cold and lead to the wrong memory-sizing decision. Likewise, a faster second query does not identify which cache changed by itself.

Cache residency is temporary. Other queries can evict pages, and a scan larger than the available cache cannot keep every page warm indefinitely. A relation may use a scan strategy that changes buffer reuse. Test with a known working set, inspect the access path, and keep timing claims separate from the cache-layer facts actually observed.

Check it quickly

Use a disposable PostgreSQL 18 instance as a superuser, with the supplied pg_buffercache and pg_prewarm extensions available but not already installed. The tested instance used 128 MB of shared buffers. The relation below is small enough to fit comfortably. Run blocks in one fresh psql -X -v ON_ERROR_STOP=1 session. The eviction function is for developer testing; restrict it to this isolated fixture.

CREATE SCHEMA cache_layer_demo;
CREATE EXTENSION pg_buffercache WITH SCHEMA cache_layer_demo;
CREATE EXTENSION pg_prewarm WITH SCHEMA cache_layer_demo;
CREATE TABLE cache_layer_demo.items (id integer PRIMARY KEY,payload text)
 WITH (autovacuum_enabled=false);
INSERT INTO cache_layer_demo.items
SELECT g,repeat(md5(g::text),10) FROM generate_series(1,12000) AS g;
VACUUM (ANALYZE) cache_layer_demo.items;
CREATE VIEW cache_layer_demo.residency AS
SELECT count(*) AS heap_buffers FROM cache_layer_demo.pg_buffercache
WHERE reldatabase=(SELECT oid FROM pg_database WHERE datname=current_database())
 AND relfilenode=pg_relation_filenode('cache_layer_demo.items')
 AND relforknumber=0;
SET max_parallel_workers_per_gather=0;
SELECT pg_relation_size('cache_layer_demo.items')/current_setting('block_size')::int AS heap_blocks;
SELECT heap_buffers>0 AS initially_buffered FROM cache_layer_demo.residency;

The heap contains 546 blocks in the tested build, and the initial residency comparison is true. The view filters by database, current file identifier, and main fork, so it does not count another relation’s buffers or the fixture’s auxiliary forks. Next, evict only this relation’s shared buffers:

SELECT * FROM cache_layer_demo.pg_buffercache_evict_relation('cache_layer_demo.items'::regclass);
SELECT heap_buffers=0 AS no_shared_heap_buffers FROM cache_layer_demo.residency;

The function reports evicted, flushed, and skipped buffers. This run returned 550, 550, and zero; the total includes auxiliary forks, explaining why it exceeds the heap-only count. The following comparison confirms no heap buffers remain. It says nothing about whether the operating system still caches those file pages.

Read the heap through the operating system without loading PostgreSQL shared buffers:

SELECT cache_layer_demo.pg_prewarm('cache_layer_demo.items','read','main');
SELECT heap_buffers=0 AS still_no_shared_heap_buffers FROM cache_layer_demo.residency;

The prewarm call processes 546 blocks, and the shared-buffer count remains zero. Its read mode performs synchronous reads; buffer mode would instead load the database buffer pool. Now execute a query that needs every payload:

EXPLAIN (ANALYZE,BUFFERS,COSTS OFF,TIMING OFF,SUMMARY OFF,FORMAT JSON)
SELECT sum(length(payload)) FROM cache_layer_demo.items;

The sequential scan reports 546 shared reads and zero shared hits. Those reads are compatible with an already warm operating-system cache; they are not evidence of 546 device operations. Repeat the same query:

EXPLAIN (ANALYZE,BUFFERS,COSTS OFF,TIMING OFF,SUMMARY OFF,FORMAT JSON)
SELECT sum(length(payload)) FROM cache_layer_demo.items;

The second scan reports zero shared reads and 546 shared hits. Both scans process twelve thousand rows, so the change in buffer accounting is not a reduction in the logical input. Finally, confirm residency and the aggregate result, then remove the fixture and its extensions:

SELECT heap_buffers>0 AS now_shared FROM cache_layer_demo.residency;
SELECT sum(length(payload)) FROM cache_layer_demo.items;
RESET max_parallel_workers_per_gather;
DROP SCHEMA cache_layer_demo CASCADE;

The result is 3,840,000 payload characters and a true shared-residency comparison. Exact block counts depend on row layout and build parameters. The meaningful observations are the empty shared pool after read-mode prewarming, the subsequent shared reads, and the repeated scan’s hits. They do not measure physical-device latency or guarantee a particular speedup.

Budget memory for the complete server

The PostgreSQL resource guide suggests roughly a quarter of system memory as an initial shared-buffer setting for a dedicated server with sufficient RAM. That is a starting point, not a universal optimum. Connection memory, concurrent sorts and hashes, maintenance, and the operating-system cache all consume the same host budget.

Measure representative working sets and concurrent workloads before changing the allocation. Warm-cache microbenchmarks can hide I/O pressure, while cold-cache tests can exaggerate a workload that normally reuses pages. Keep cache-layer counters, actual device observations, and query results together so a configuration change is credited for the effect it actually produced.

Sources

PostgreSQL 18 resource consumption, shared buffers and operating-system cache budgeting, checked 2026-09-10.
PostgreSQL 18 pg_prewarm, read versus buffer prewarming modes, checked 2026-09-10.
PostgreSQL 18 pg_buffercache, buffer identity and development-only relation eviction, checked 2026-09-10.

Redis expiry is a deadline, eviction is a capacity policy

Sun, 26 Dec 2027 00:00:00 +0000

Expiration and eviction can both turn a successful lookup into a miss, but they answer different questions. Expiration follows a key’s deadline. Eviction removes eligible keys to respond to memory pressure. Extending a TTL cannot prevent capacity-driven eviction, and increasing memory cannot correct a deadline that the application set incorrectly.

Time and capacity are separate axes

A volatile key has an expiration attached to it. Redis checks expired keys during access and through active expiration work. This does not promise that the key survives until its deadline: explicit deletion, replacement, and an eligible eviction policy can change the outcome earlier. A cache client must handle misses regardless of how far away the original expiration appeared.

Maxmemory selects an accounting limit used by Redis’s memory-management decisions. It is not an operating-system RSS ceiling. Replication and append-only buffers can include memory excluded from eviction accounting; process overhead, allocator behavior, and persistence work also need capacity headroom. A large command can temporarily exceed the configured limit. Size the process environment as well as the cache budget.

Select candidates before debating ranking

Allkeys policies consider keys regardless of whether they have TTLs. Volatile policies restrict candidates to keys with expiration. Within those sets, LRU favors keys that have gone longer without access, LFU uses frequency information, and random policies select randomly. Redis approximates LRU and LFU rather than promising one globally exact victim order. Volatile-ttl favors shorter remaining lifetimes, not keys that have necessarily already expired.

Noeviction declines memory-growing commands when the limit cannot be accommodated instead of discarding keys for capacity. Reads can still succeed. It is not a durability guarantee: ordinary expiration, explicit deletion, process failure, and persistence configuration remain separate. Inspect the actual configured policy instead of assuming a universal deployment or cluster default.

A volatile policy without expiring candidates behaves like noeviction for this purpose. Merely choosing volatile-lru does not cause permanent keys to become disposable. Conversely, putting session or authoritative records beside rebuildable cache entries under an allkeys policy makes those records eligible too. A key prefix is a naming convention, not an eviction boundary.

When it breaks

A sudden miss spike can reflect expiration, eviction, invalidation, or changed key construction. Compare expired_keys and evicted_keys deltas with the request workload and command outcomes. These are cumulative counters, so compare a bounded interval and account for restarts or resets. A rising eviction counter alone does not reveal which application feature lost data.

Watch refused writes as well as reads. A warm cache can keep returning plausible values while its writers receive memory errors. Ignoring those errors turns stale data into an application consistency defect. Increasing TTL may then make existing values older without fixing the inability to write replacements. Retry policy must not turn memory pressure into an unlimited request storm.

Separating authoritative state from a disposable cache is an architectural decision, not a command to migrate data during an incident. Verify recoverability, persistence, and ownership before changing an existing instance’s policy. The exercise below changes configuration only inside processes it creates and tests three candidate policies without changing any production keyspace.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. Build core Redis with libc and install the pinned client locally. The source and package downloads need the network; no system installation or optional bundled-module build is performed.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Execute the Python block with "$redis_lab/venv/bin/python". Each policy gets its own temporary Unix socket, empty dataset, and process with TCP and persistence disabled. The memory limit is the measured initial allocation plus two mebibytes. Bounded writes use 64-KiB values; TTLs are one hour where enabled. Every process and its runtime directory are cleaned up afterward.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance(policy="noeviction"):
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", policy,
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=True,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


for policy, expiring in [("allkeys-lru", True), ("noeviction", True),
 ("volatile-lru", False)]:
 with instance(policy) as client:
 baseline = client.info("memory")["used_memory"]
 client.config_set("maxmemory", baseline + 2 * 1024 * 1024)
 payload = "x" * 65536
 written, rejected = 0, False
 for index in range(128):
 try:
 client.set(f"item:{index}", payload, ex=3600 if expiring else None)
 except redis.exceptions.OutOfMemoryError:
 rejected = True
 break
 written += 1
 stats = client.info("stats")
 remaining = client.dbsize()
 assert stats["expired_keys"] == 0
 if policy == "allkeys-lru":
 assert written == 128 and not rejected
 assert stats["evicted_keys"] > 0 and remaining < written
 assert client.ttl("item:127") > 0
 else:
 assert rejected and 0 < written < 128
 assert stats["evicted_keys"] == 0 and remaining == written
 assert client.get("item:0") == payload
 if not expiring:
 assert client.ttl("item:0") == -1
 print(policy, {"written": written, "remaining": remaining,
 "evicted": stats["evicted_keys"],
 "expired": stats["expired_keys"], "rejected": rejected})

In this run, allkeys-lru accepted 128 writes, retained 23 keys, and reported 105 evictions with zero expirations. The newest surviving key still had a positive TTL. This establishes capacity-driven loss before expiration without claiming an exact LRU victim sequence.

Noeviction accepted 23 writes before rejecting the next allocation; existing reads still worked and no keys were evicted. Volatile-lru with no expiring keys produced the same refusal behavior. Exact counts depend on allocation overhead and the build, so assertions check relationships rather than those particular counts. This is not a throughput benchmark or evidence about replication, failover, persistence peaks, or an operating-system memory limit.

Sources

Redis key eviction, maxmemory accounting, candidate policies, and approximate ranking, checked 2026-09-08.
Redis EXPIRE, per-key deadlines and expiration behavior, checked 2026-09-08.
Redis INFO, memory accounting and eviction/expiration counters, checked 2026-09-08.
Redis 8.10.1 build instructions, isolated core build and allocator selection, checked 2026-09-08.

Cache-Control is a contract between cache and origin

Mon, 20 Dec 2027 00:00:00 +0000

A cache policy needs a behavioral test, not just a header-presence check. Two responses can both contain max-age yet behave differently because one also requires validation or prohibits storage. The request matters too: a browser can bypass an otherwise reusable copy without changing the origin’s response policy or deleting an older cached entry.

Both sides must agree

Cache-Control directives are directional. A request directive is not automatically copied into the response, and RFC 9111 describes request directives as advisory. Response directives impose requirements on conforming caches. Intermediaries must forward applicable cache directives even when they do not implement a cache themselves. Deployment-specific overrides need inspection; they are not evidence that the origin’s declared policy was followed.

Separate three questions. May this response be stored? Is its current age within the applicable freshness lifetime? Does this request permit reuse without validation? No-store prohibits HTTP cache storage; no-cache permits storage but requires successful validation before reuse. Max-age sets freshness, while must-revalidate prohibits unvalidated reuse after the response becomes stale. Must-revalidate does not require a round trip while the response is still fresh.

Private restricts storage to single-user caches; public explicitly permits caching subject to other requirements and is not mandatory on every cacheable response. S-maxage overrides freshness for shared caches, not browser caches. Vary supplies request-matching dimensions. None of these mechanisms identifies the current application account or makes personal data safe merely because its lifetime is short.

When it breaks

Testing with a new query parameter on every request changes the cache key and can hide reuse entirely. Disabling the browser cache in developer tools can create the same misleading result. Some automation interception mechanisms also disable caching. A useful test repeats the same URL in one isolated context and counts requests at the origin, while asserting the returned body as well.

Fetch’s cache option is a browser API, not a response header. Its no-store mode bypasses lookup and avoids updating the HTTP cache; reload bypasses lookup but can update it. Neither should be confused with erasing all previous copies. Likewise, changing origin headers cannot force an already-fresh entry to contact the origin immediately. History restoration and service-worker storage introduce additional behavior outside this test.

No-cache does not promise a 304. A validator and an unchanged representation are needed for that optimization. Without ETag or Last-Modified, repeated requests can transfer complete 200 responses. Count both origin visits and transferred bodies before concluding that validation is saving work.

Check it quickly

This was tested on macOS with Node 26.0.0, Playwright 1.63.0, and its Chromium 153.0.8010.12 headless shell. Install into a temporary directory; Linux also needs Playwright’s documented browser system dependencies. The download needs network access. Run the JavaScript block using NODE_PATH="$cache_browser_lab/node_modules" PLAYWRIGHT_BROWSERS_PATH="$cache_browser_lab/browsers" node. Test requests use only an automatically allocated loopback port, with no existing browser profile.

cache_browser_lab=$(mktemp -d)
npm install --prefix "$cache_browser_lab" --ignore-scripts --no-audit --no-fund --package-lock=false playwright@1.63.0
PLAYWRIGHT_BROWSERS_PATH="$cache_browser_lab/browsers" node "$cache_browser_lab/node_modules/playwright/cli.js" install chromium --only-shell --no-remove

const assert = require("node:assert/strict");
const http = require("node:http");
const { once } = require("node:events");
const { chromium } = require("playwright");

const policies = new Map([
 ["/fresh", "max-age=3600"],
 ["/must", "max-age=3600, must-revalidate"],
 ["/validate", "no-cache, max-age=3600"],
 ["/discard", "no-store, max-age=3600"],
]);
const hits = new Map();
const server = http.createServer((request, response) => {
 const route = new URL(request.url, "http://127.0.0.1").pathname;
 if (route === "/") {
 response.writeHead(200, { "Content-Type": "text/html", "Cache-Control": "no-store" });
 response.end("<!doctype html><title>Cache experiment</title>");
 } else if (policies.has(route)) {
 const count = (hits.get(route) || 0) + 1;
 hits.set(route, count);
 response.writeHead(200, {
 "Content-Type": "text/plain", "Cache-Control": policies.get(route),
 });
 response.end(`${route}:${count}`);
 } else {
 response.writeHead(404, { "Cache-Control": "no-store" });
 response.end();
 }
});

async function main() {
 server.listen(0, "127.0.0.1");
 await once(server, "listening");
 let browser;
 try {
 browser = await chromium.launch();
 console.log("Chromium", browser.version());
 const context = await browser.newContext({ serviceWorkers: "block" });
 try {
 const page = await context.newPage();
 await page.goto(`http://127.0.0.1:${server.address().port}/`);
 const read = (route, cache = "default") => page.evaluate(async ({ route, cache }) => {
 const response = await fetch(route, { cache });
 if (!response.ok) throw new Error(`HTTP ${response.status}`);
 return response.text();
 }, { route, cache });
 for (const [route, expectedHits] of [
 ["/fresh", 1], ["/must", 1], ["/validate", 2], ["/discard", 2],
 ]) {
 const first = await read(route);
 const second = await read(route);
 assert.equal(first, `${route}:1`);
 assert.equal(second, `${route}:${expectedHits}`);
 assert.equal(hits.get(route), expectedHits);
 console.log(route, "origin requests:", hits.get(route));
 }
 assert.equal(await read("/fresh", "no-store"), "/fresh:2");
 assert.equal(await read("/fresh"), "/fresh:1");
 assert.equal(hits.get("/fresh"), 2);
 assert.equal(await read("/fresh", "reload"), "/fresh:3");
 assert.equal(await read("/fresh"), "/fresh:3");
 assert.equal(hits.get("/fresh"), 3);
 console.log("no-store bypass retained version 1; reload stored version 3");
 } finally {
 await context.close();
 }
 } finally {
 if (browser) await browser.close();
 await new Promise((resolve, reject) => server.close(error => error ? reject(error) : resolve()));
 }
}

main().catch(error => { console.error(error); process.exitCode = 1; });

The initial origin counts are one, one, two, and two. Both fresh policies reuse the first body. The no-cache and no-store responses reach the origin again despite their large max-age. This fixture intentionally has no validators, so those visits return full bodies; it does not distinguish their storage eligibility by counts alone.

The later no-store fetch obtains version two, but an ordinary fetch still returns the previously cached version one. Reload obtains and stores version three, which the next ordinary fetch reuses. Origin counters ensure these are actual cache decisions, not coincidentally identical response strings. The server and isolated browser context are closed afterward.

These observations cover this Chromium HTTP cache and these exact requests. They do not verify a shared CDN, persistence across browser restarts, stale delivery, or user-account isolation. Keep those separate acceptance tests rather than extending a successful local cache experiment into a broader claim.

Sources

RFC 9111: Cache-Control, directional directives and storage/reuse requirements, checked 2026-09-07.
MDN: Request.cache, default, no-store, and reload behavior, checked 2026-09-07.
Playwright browser contexts, isolated non-persistent contexts, checked 2026-09-07.
Playwright browser installation, version-matched headless shell and system dependencies, checked 2026-09-07.

nginx proxy cache balances stale and fresh

Thu, 24 Jun 2027 00:00:00 +0000

An nginx proxy cache has several independent decisions: identifying the stored response, deciding whether it may be reused, deciding whether a new response may be stored, and handling an expired copy. Treating those decisions as one TTL setting makes failures difficult to explain. Verify the resulting request path and bytes, not merely that nginx accepts the configuration.

Separate storage from freshness

Proxy_cache_path names the disk storage and a shared-memory keys_zone for metadata; proxy_cache enables that zone for requests. The metadata size does not bound response-body storage. Max_size is managed by cache maintenance rather than an instantaneous disk quota, while inactive retention is not the response’s freshness lifetime. An expired response can remain stored and useful for revalidation.

The default key is $scheme$proxy_host$request_uri. Request_uri already includes the query string, so appending $is_args$args repeats it. Do not casually substitute a normalized URI or omit representation-changing inputs. GET and HEAD share the default cache behavior because HEAD conversion is enabled; disabling that conversion requires accounting for the request method in the key.

Verify the active response policy

Proxy_cache_valid supplies configured lifetimes by response status, but processed origin freshness headers have higher priority. X-Accel-Expires has its own precedence; Cache-Control and Expires apply when it is absent. Ignoring those headers changes the contract. In the exercise, origin max-age one overrides configured validity of sixty seconds, while no-store prevents response storage.

Proxy_cache_revalidate enables conditional requests for expired items using validators such as ETag and Last-Modified. A matching 304 can preserve the body and refresh metadata; a changed 200 supplies replacement bytes. This does not imply asynchronous refresh. Background updating is a separate directive and also needs permission to serve stale content during updating.

When it breaks

Proxy_cache_bypass prevents reading from cache; proxy_no_cache prevents saving the new response. A route requiring both behaviors needs both decisions expressed. Bypassing once also does not purge an older entry. The fixture pairs the directives with a laboratory header and proves that the original cached response remains available afterward. That header is a test switch, not an authentication policy.

Proxy_cache_use_stale http_500 permits an existing stale response after an upstream 500. It cannot help a completely cold key, and the directive does not itself specify a maximum stale duration. An origin stale-if-error window has lower priority than these directive parameters, so adding that header does not cap this setting. Decide whether old content is acceptable before enabling it.

Otherwise a successful-looking response can conceal an origin failure or keep obsolete data visible. Record the upstream outcome, nginx cache status, and delivered body together. A syntax check cannot establish account isolation, and a single warm hit cannot prove behavior during concurrent cold loads. Cache locking, retention under pressure, and multi-worker behavior require their own verification.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.11 or newer. This ran on macOS with clang 21.0.0, Python 3.14.4, and nginx 1.30.4. The commands build an isolated HTTP-only test binary without installation or sudo. They omit rewrite and gzip modules to avoid their external dependencies; this is not a TLS deployment recipe.

nginx_lab=$(mktemp -d)
curl --fail --location --output "$nginx_lab/nginx.tar.gz" https://nginx.org/download/nginx-1.30.4.tar.gz
tar -xzf "$nginx_lab/nginx.tar.gz" -C "$nginx_lab"
cd "$nginx_lab/nginx-1.30.4"
./configure --prefix="$nginx_lab/install" --without-http_rewrite_module --without-http_gzip_module
make -j2
export NGINX_BIN="$nginx_lab/nginx-1.30.4/objs/nginx"

Run the Python block with python3 in that shell. It creates its own configuration and cache directory, checks the configuration, and starts only loopback services on selected ports. Two short waits allow real cache expiry. It shuts down its own nginx process and origin, then removes only the temporary runtime directory. Source downloads need the network; requests in the experiment remain local.

import hashlib
import os
from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, HTTPServer
from pathlib import Path
import signal
import socket
import subprocess
from tempfile import TemporaryDirectory
from threading import Thread
import time

state = {"body": b"A", "fail": False}
seen = []


class Origin(BaseHTTPRequestHandler):
 def log_message(self, *_):
 pass

 def do_GET(self):
 tag = '"' + hashlib.sha256(state["body"]).hexdigest() + '"'
 conditional = self.headers.get("If-None-Match")
 if self.path == "/short" and state["fail"]:
 status, body = 500, b"failure"
 elif conditional == tag:
 status, body = 304, b""
 else:
 status, body = 200, state["body"]
 directive = "max-age=1" if self.path == "/short" else "max-age=60"
 if self.path == "/no-store" or status == 500:
 directive = "no-store"
 seen.append((self.path, status, conditional))
 self.send_response(status)
 self.send_header("Cache-Control", directive)
 if status != 500:
 self.send_header("ETag", tag)
 if status != 304:
 self.send_header("Content-Length", str(len(body)))
 self.end_headers()
 self.wfile.write(body)


origin = HTTPServer(("127.0.0.1", 0), Origin)
worker = Thread(target=origin.serve_forever)
worker.start()
try:
 with TemporaryDirectory(prefix="nginx-cache-lab-") as directory:
 root = Path(directory)
 with socket.socket() as reservation:
 reservation.bind(("127.0.0.1", 0))
 port = reservation.getsockname()[1]
 config = root / "nginx.conf"
 config.write_text(f"""
pid nginx.pid;
error_log stderr error;
events {{ worker_connections 64; }}
http {{
 access_log off;
 proxy_cache_path "{root}/cache" keys_zone=lab:1m max_size=10m;
 server {{
 listen 127.0.0.1:{port};
 location / {{
 proxy_pass http://127.0.0.1:{origin.server_port};
 proxy_cache lab;
 proxy_cache_valid 200 60s;
 proxy_cache_revalidate on;
 proxy_cache_use_stale http_500;
 proxy_cache_bypass $http_x_lab_bypass;
 proxy_no_cache $http_x_lab_bypass;
 proxy_connect_timeout 2s;
 proxy_read_timeout 2s;
 add_header X-Lab-Cache $upstream_cache_status always;
 }}
 }}
}}
""", encoding="utf-8")
 command = [os.environ["NGINX_BIN"], "-p", str(root) + "/",
 "-c", str(config), "-e", "stderr"]
 checked = subprocess.run(command + ["-t"], capture_output=True, text=True)
 assert checked.returncode == 0, checked.stderr
 process = subprocess.Popen(command + ["-g", "daemon off; master_process off;"],
 stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, text=True)
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "nginx exited during startup"
 try:
 with socket.create_connection(("127.0.0.1", port), timeout=0.1):
 break
 except OSError:
 if time.monotonic() >= deadline:
 raise TimeoutError("nginx did not become ready")
 time.sleep(0.05)

 def get(path, expected, body, bypass=False):
 connection = HTTPConnection("127.0.0.1", port, timeout=3)
 try:
 connection.request("GET", path, headers={"X-Lab-Bypass": "1"} if bypass else {})
 response = connection.getresponse()
 actual = (response.status, response.getheader("X-Lab-Cache"), response.read())
 assert actual == (200, expected, body), actual
 print(path, expected, body.decode())
 finally:
 connection.close()

 get("/long", "MISS", b"A")
 get("/long", "HIT", b"A")
 state["body"] = b"B"
 get("/long", "BYPASS", b"B", bypass=True)
 get("/long", "HIT", b"A")
 assert sum(path == "/long" for path, _, _ in seen) == 2

 get("/short", "MISS", b"B")
 time.sleep(2.1)
 get("/short", "REVALIDATED", b"B")
 assert seen[-1][1:] == (304, '"' + hashlib.sha256(b"B").hexdigest() + '"')
 state["fail"] = True
 time.sleep(2.1)
 get("/short", "STALE", b"B")
 assert seen[-1][1] == 500
 state.update(body=b"C", fail=False)
 get("/short", "EXPIRED", b"C")
 get("/short", "HIT", b"C")
 get("/no-store", "MISS", b"C")
 get("/no-store", "MISS", b"C")
 assert sum(path == "/no-store" for path, _, _ in seen) == 2
 finally:
 if process.poll() is None:
 process.send_signal(signal.SIGQUIT)
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors
finally:
 origin.shutdown()
 worker.join()
 origin.server_close()

The long-lived route reports MISS A, HIT A, BYPASS B, then HIT A. Only two requests reached that origin path, and B was not saved by the bypass request. The short-lived route reports MISS B and, after expiry, REVALIDATED B; the origin log confirms a 304 and the correct conditional ETag.

With an injected origin 500 it serves STALE B. Once the origin recovers with C, EXPIRED C replaces the body and the following request is HIT C. Two no-store requests both remain MISS. These are real nginx decisions for this single-process fixture, not evidence for background refresh, distributed invalidation, production authorization, disk-pressure handling, or every stale-policy combination.

Sources

nginx proxy module, keys, header precedence, revalidation, bypass, storage, and stale directives, checked 2026-09-07.
nginx build configuration, build prefix and optional module dependencies, checked 2026-09-07.
nginx content caching guide, disk storage, shared metadata, and cache management, checked 2026-09-07.
RFC 9111: stale responses, freshness and permitted stale reuse, checked 2026-09-07.

Cache · XB Field Notes

dnsmasq separates forwarded cache entries from configured local names

When it breaks

Check it quickly

Operate the listener and its data sources deliberately

Sources

A microcache serves fresh entries briefly and coalesces misses explicitly

When it breaks

Check it quickly

Choose freshness and overload behavior separately

Sources

DNS TTL trades repeated upstream queries against retained answers

When it breaks

Check it quickly

Choose TTL from observed demand and change needs

Sources

Docker rebuilds reuse recorded work, including stale work

When it breaks

Check it quickly

Make freshness an explicit input

Sources

nginx validators are only as sound as their representation identity

When it breaks

Check it quickly

Make deployments preserve the validator contract

Sources

Postgres uses two cache layers

When it breaks

Check it quickly

Budget memory for the complete server

Sources

Redis expiry is a deadline, eviction is a capacity policy

Time and capacity are separate axes

Select candidates before debating ranking

When it breaks

Check it quickly

Further reading

Sources

Cache-Control is a contract between cache and origin

Both sides must agree

When it breaks

Check it quickly

Sources

nginx proxy cache balances stale and fresh

Separate storage from freshness

Verify the active response policy

When it breaks

Check it quickly

Further reading

Sources