Scaling · XB Field Notes

The Horizontal Pod Autoscaler scales by a target utilization

Wed, 22 Dec 2027 00:00:00 +0000

Because it can act on the mean of a series instead of a single spike, you need to pair it with a max-rc that keeps the maths from running forever.

A smoothed average decides

The HPA does not react to a single instantaneous read; it averages the observed value over a window and smooths it before comparing against the target. That is deliberate: a one-second spike in a metric should not repaint the replica count, so the controller aggregates first, then only changes the count when the smoothed value moves past the target. To make that work you choose the metric source (CPU, custom metrics API, or external) and the target, because the smoothing window and the aggregation method, per-pod average versus cluster level, together define what ’the load’ actually is. A metric read over too short a window reacts to noise; too long and it lags demand. Reading the document’s sync-period and aggregation paragraph is the first step in tuning, not the last.

Pick the metric you can act on

The real design choice is the metric, because scaling is only as useful as the target you give it. CPU is easy on any cluster and works well for CPU-bound workloads, but it is a proxy: a service waiting on queues or a database is often standing idle on CPU while its latency explodes. Demand-driven metrics such as request rate or a custom kafka_consumer_lag are harder to wire but report the thing users feel. Whatever you pick, choose one you can act on, a target that, when crossed, really changes the experience, and use the same definition the app uses, so the HPA is scaling the number that matters rather than a number that happens to be cheap to read.

Automation is orthogonal to defaults

Set the initial replicas in the Deployment because that is the starting point, not because the HPA reads it, once the HPA exists it owns the count and the initial value is only a floor or a first estimate. The two are orthogonal in the good way: you can disable the controller and still have a sane static stated count, and you can run a steady HPA without a fancy initial guess. Keep the minimum realistic (that is the real floor), pick a maximum that bounds spend, and remember that the initial value is what makes the very first seconds sane before the controller’s first reconcile lands.

When it breaks

A missing custom metric does not fall back to CPU. When the custom-metrics API returns an error or no series, the HPA marks ScalingActive False and holds replicas instead of guessing, so a workload with only a queue metric freezes exactly when it needed scaling. List CPU alongside the custom metric and let the maximum win, keeping a degraded but live input.

A Prometheus Adapter pipeline also adds delay. Scrape intervals plus adapter caching can serve queue depth from minutes ago, causing scale-up against drained work and overshoot. Check the series timestamp and adapter latency before retuning the target, and expose one pre-aggregated series per workload.

Cardinality breaks the same path differently: per-user or per-request labels multiply one signal into thousands of series, slow the API, and flip readings to Unknown. Aggregate in Prometheus first. A target-type mismatch does the same: custom metrics accept only Value or AverageValue, and CPU utilization without pod requests stays Unknown. Describe the HPA and align the type with what the pipeline serves.

Check it quickly

Save the following as hpa_metric_sim.py and run python3 hpa_metric_sim.py. It uses only the standard library, touches no cluster, and models the rules above: multiple metrics combine by maximum, missing metrics are skipped, and an all-missing poll holds replicas.

import math
import sys


def desired_for(current, value, target):
 """Replicas wanted by one metric; None means unavailable."""
 if value is None or target is None or target <= 0:
 return None
 return math.ceil(current * value / target)


def decide(current, metrics, min_replicas, max_replicas):
 """Combine metrics by maximum; hold when every metric is missing."""
 wants = []
 for _, value, target in metrics:
 want = desired_for(current, value, target)
 if want is not None:
 wants.append(want)
 if not wants:
 return current
 return max(min_replicas, min(max_replicas, max(wants)))


def check(actual, expected, message):
 if actual != expected:
 raise AssertionError(f"{message}: got {actual}, want {expected}")
 print(f"OK: {message} -> {actual}")


def main():
 current = 4
 cpu = ("cpu", 80, 50) # 4 * 80/50 = 6.4 -> 7
 queue = ("queue", 900, 300) # 4 * 900/300 = 12
 check(decide(current, [cpu, queue], 1, 20), 12, "max wins across metrics")
 check(decide(current, [cpu, ("queue", None, 300)], 1, 20), 7, "one missing metric still scales on CPU")
 check(decide(current, [("cpu", None, 50), ("queue", None, 300)], 1, 20), 4, "all missing holds replicas")
 check(decide(current, [cpu, queue], 1, 10), 10, "maxReplicas still clamps metric math")
 print("PASS: 4 scenarios; max-wins and hold-on-missing verified")
 print(sys.version.split()[0])


if __name__ == "__main__":
 main()

The four scenarios prove the maximum wins when CPU asks for seven and queue depth asks for twelve, a single missing series still leaves CPU in charge, all-missing holds instead of scaling to zero, and the maximum bound still clamps the result. Expected output ends with PASS: 4 scenarios; max-wins and hold-on-missing verified.

The assertions passed with Python 3.12.6 on macOS. The script checks replica math only, not scrape latency, adapter caching, or cardinality. Before changing production, verify timestamps and HPA events against the real custom-metrics API.

Sources

Kubernetes Horizontal Pod Autoscaling, documents algorithm and metric sources, checked 2026-06-01.
Resource Metrics Pipeline, documents metrics-server and the resource metrics API, checked 2026-06-01.

TCP window scaling enlarges receive-window encoding in each direction

Thu, 02 Dec 2027 00:00:00 +0000

TCP’s header has a 16-bit receive-window field. Window scaling lets established connections interpret that field using a shift negotiated during setup, allowing a receiver to advertise more than 65535 bytes of space. The two directions can use different shifts. The extension enlarges the encoding of receive-window information; it does not directly enlarge every buffer or guarantee that a sender will keep that much data in flight.

When it breaks

A high-bandwidth path with substantial RTT may need much more outstanding data than an unscaled receive window can describe. For example, 100 megabits per second over an 80-millisecond round trip corresponds to roughly one million bytes in flight. A 65535-byte window would impose a much lower idealized rate. This bandwidth-delay calculation is a sizing estimate, not a throughput measurement or a guarantee that no other limit applies.

The sender also has a congestion window and needs application data to send. Enlarging receive-window encoding cannot fix a small congestion budget, an application that stops reading, packet loss, or a blocked route. A scaled zero window is still zero. Diagnose which resource is actually constraining progress instead of treating the presence of a window-scale option as proof that the connection should saturate the link.

Negotiation errors can make captures misleading. Both endpoints must offer the option for scaling to operate, and each scale applies to windows advertised by the endpoint that supplied it. The window field in SYN and SYN-ACK is never scaled. Applying the wrong direction’s shift, or multiplying the SYN window, produces an incorrect capacity calculation even when the packet bytes are perfectly valid.

Check it quickly

Save this as check-window-scaling.py and run python3 check-window-scaling.py with Docker. The disposable environment uses Python 3.11.16, iproute2 6.1.0, and tcpdump 4.99.3, tested on Linux 6.12.76. NET_ADMIN and SYS_ADMIN enable nested namespaces and their temporary proc mount. The endpoints use a new veth link, and settings affect only that laboratory:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,('child exited without a response',process.stderr.read())
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found


def capture(stack,root,name,interface,expression,label):
 path=root/(label+'.pcap')
 process=subprocess.Popen(['ip','netns','exec',name,'tcpdump','--immediate-mode','-Z','root','-U','-n','-i',interface,'-w',str(path),expression],
 stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process)
 deadline=time.monotonic()+3
 while time.monotonic()<deadline:
 assert process.poll() is None,'tcpdump exited before listening'
 if select.select([process.stderr],[],[],.1)[0] and 'listening on' in process.stderr.readline():return process,path
 raise AssertionError('tcpdump did not become ready')
def packets(process,path):
 import struct,socket
 process.send_signal(signal.SIGINT);_,diagnostic=process.communicate(timeout=3)
 assert process.returncode==0,diagnostic
 raw=path.read_bytes();assert raw[:4]==bytes.fromhex('d4c3b2a1') and struct.unpack_from('<I',raw,20)[0]==1
 offset=24;result=[]
 while offset<len(raw):
 seconds,micros,size,original=struct.unpack_from('<IIII',raw,offset);offset+=16
 frame=raw[offset:offset+size];offset+=size
 if frame[12:14]!=b'\x08\x00' or frame[23]!=6:continue
 ihl=(frame[14]&15)*4;length=struct.unpack_from('!H',frame,16)[0];tcp=14+ihl
 sport,dport,seq,ack=struct.unpack_from('!HHII',frame,tcp);header=(frame[tcp+12]>>4)*4
 opts={};cursor=tcp+20
 while cursor<tcp+header:
 kind=frame[cursor]
 if kind==0:break
 if kind==1:cursor+=1;continue
 option_length=frame[cursor+1];data=frame[cursor+2:cursor+option_length];assert option_length>=2
 if kind==2:opts['mss']=struct.unpack('!H',data)[0]
 elif kind==3:opts['wscale']=data[0]
 elif kind==4:opts['sack_ok']=True
 elif kind==5:opts['sack']=[list(pair) for pair in struct.iter_unpack('!II',data)]
 cursor+=option_length
 body=frame[tcp+header:14+length]
 result.append({'t':seconds+micros/1e6,'src':socket.inet_ntoa(frame[26:30]),'dst':socket.inet_ntoa(frame[30:34]),
 'sport':sport,'dport':dport,'seq':seq,'ack':ack,'flags':frame[tcp+13],'window':struct.unpack_from('!H',frame,tcp+14)[0],
 'length':len(body),'options':opts,'preview':body[:24].hex()})
 return result
with lab(['client','target']) as (root,stack,setting):
 wire('client','c0','target','t0',['192.0.2.1/30'],['192.0.2.2/30'])
 server_code=r'''
import socket,json
def read_exact(stream,length):
 data=b''
 while len(data)<length:
 chunk=stream.recv(length-len(data));assert chunk,'premature EOF';data+=chunk
 return data
listener=socket.socket();listener.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
listener.setsockopt(socket.SOL_SOCKET,socket.SO_RCVBUF,262144)
listener.bind(('192.0.2.2',18080));listener.listen();print('listening',flush=True)
with listener.accept()[0] as connection:
 connection.settimeout(4);print(json.dumps({'rcvbuf':connection.getsockopt(socket.SOL_SOCKET,socket.SO_RCVBUF)}),flush=True)
 assert read_exact(connection,4)==b'ping';connection.sendall(b'pong')
 assert connection.recv(1)==b''
'''
 client_code=r'''
import socket,json,sys
def read_exact(stream,length):
 data=b''
 while len(data)<length:
 chunk=stream.recv(length-len(data));assert chunk,'premature EOF';data+=chunk
 return data
s=socket.socket();s.settimeout(4);s.setsockopt(socket.SOL_SOCKET,socket.SO_RCVBUF,65536)
s.bind(('192.0.2.1',40400+int(sys.argv[1])));s.connect(('192.0.2.2',18080));s.sendall(b'ping')
assert read_exact(s,4)==b'pong'
print(json.dumps({'rcvbuf':s.getsockopt(socket.SOL_SOCKET,socket.SO_RCVBUF),'reply':'pong'}),flush=True)
assert sys.stdin.readline().strip()=='close';s.close()
'''
 outcomes={}
 for enabled in [1,0]:
 setting('client','net.ipv4.tcp_window_scaling',enabled);setting('target','net.ipv4.tcp_window_scaling',enabled)
 service=spawn(stack,['ip','netns','exec','target','python3','-u','-c',server_code]);assert line(service).strip()=='listening'
 tap,path=capture(stack,root,'client','c0','tcp port 18080','window'+str(enabled))
 client=spawn(stack,['ip','netns','exec','client','python3','-u','-c',client_code,str(enabled)])
 client_result=json.loads(line(client));server_result=json.loads(line(service))
 diagnostics=ns('client','ss','-tin','state','established').stdout
 client.stdin.write('close\n');client.stdin.flush();client.wait(timeout=2);service.wait(timeout=2)
 assert client.returncode==service.returncode==0
 observed=packets(tap,path);syn=next(p for p in observed if p['flags']&0x12==0x02)
 synack=next(p for p in observed if p['flags']&0x12==0x12)
 advertised=next(p for p in observed if p['sport']==18080 and p['length']==4)
 if enabled:
 assert 'wscale' in syn['options'] and 'wscale' in synack['options']
 effective=advertised['window']<<synack['options']['wscale'];assert effective>65535
 else:
 assert 'wscale' not in syn['options'] and 'wscale' not in synack['options'];effective=advertised['window'];assert effective<=65535
 outcomes[enabled]={'client':client_result,'server':server_result,'syn':syn,'synack':synack,
 'server_post_handshake_window_field':advertised['window'],'effective_receive_window':effective,'ss':diagnostics}
 print(json.dumps(outcomes))
"""
name='xbcatm-tcpflow-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

Before opening connections, the client requests a receive buffer of 65536 bytes and the server listener requests 262144 bytes. The script records the effective socket-option values rather than assuming the requested sizes are used literally. Linux buffer accounting and limits matter: this run reported 131072 on the client and 425984 on the accepted server socket. Neither number should simply be equated with an advertised TCP window.

With scaling enabled in both namespaces, the captured client SYN offers shift one and the server SYN-ACK offers shift two. Their initial window fields remain unscaled. After setup, a server data segment advertises window field 53213. Applying the server’s shift gives 53213 times four, or 212852 bytes. The client socket diagnostic independently reports the corresponding send-window value.

The next fresh connection disables scaling in both namespaces while keeping the buffer requests and application exchange the same. Neither SYN carries a scale option, and the observed server receive window is 65535 bytes. Both trials send ping and receive pong, checking actual communication as well as negotiation. The comparison demonstrates the encoding limit without pretending that a four-byte exchange measures maximum throughput.

The fixture captures setup because the scale is fixed for that connection. Changing a system default later does not renegotiate an existing stream. Configuring appropriate receive-buffer behavior before connect or listen matters when it influences the scale selected during setup. A packet trace starting after the handshake needs the previously negotiated state before its raw window fields can be interpreted correctly.

Separate buffer allocation, advertised capacity, and observed performance

Record the relevant socket settings, negotiated shifts, current advertised window, congestion window, RTT, and application activity. Those describe different constraints. A large allocation can coexist with a smaller advertised window because the stack accounts for buffering and current occupancy. The experiment’s reported allocation and wire-level value intentionally make that distinction visible rather than hiding it behind one ambiguous “window size.”

If a transfer underperforms, check whether the receiver keeps consuming data and whether the sender has enough work to use its permitted flight size. Review losses and congestion behavior before changing buffer budgets. Larger allocations across many concurrent sockets also consume resources, so tune with workload evidence rather than maximizing every available limit.

The protocol limits the shift to fourteen, giving a representable window below one gibibyte. That encoding ceiling is not a requirement to allocate that much memory. Validate the deployed kernel and representative traffic after a change. This example verifies negotiation, direction, arithmetic, and a successful exchange; sustained capacity and application behavior need measurements suited to the real path.

Sources

RFC 7323 TCP Extensions, directional scale factors, SYN exceptions and receive-window calculation, checked 2026-09-12.
Linux tcp(7), receive buffers, window scaling and socket options before connection setup, checked 2026-09-12.
RFC 5681 Congestion Control, the separate limits imposed by congestion and receive windows, checked 2026-09-12.

Cardinality is a metric budget

Wed, 21 Jul 2027 00:00:00 +0000

A metric costs more than its short name suggests. Prometheus identifies a time series by its metric name and complete label set. A new observed label combination creates a different series, which must be ingested, indexed, retained, and queried. Cardinality budgeting asks which distinctions are worth paying for and whether their number stays bounded as the service grows.

Count combinations that actually exist

Label dimensions multiply the possible combinations, but the product of their distinct values is an upper bound unless every combination occurs. Two routes, two methods, and two status categories allow eight combinations. A service that only accepts one method on each route may emit fewer. Count observed combinations as well as theoretical limits, and include target labels such as instance when estimating what the server will ingest.

The value set matters more than the label’s friendly name. A normalized route such as /orders/{order_id} groups operations; a raw path containing each order identifier creates a growing dimension. Request IDs, timestamps, user identifiers, and arbitrary query strings can grow with event volume. A label named route is not bounded merely because the developer intended it to describe a route. Check the values produced on real traffic, including errors and uncommon paths.

Include the metric representation

A counter usually contributes one series for each observed label combination, before optional companion metrics. A classic histogram expands the same combination into its bucket, sum, and count series. With six finite bucket boundaries, one positive-infinity bucket, a sum, and a count, that is nine core series per combination. Some clients expose an additional creation-time series; include it in a real inventory rather than assuming this simplified count is complete.

Replicas add their own instance dimension. Scrape interval changes samples per series, while retention changes how long historical samples remain; neither is the same as the number of distinct series. Rapidly changing identifiers also create churn: yesterday’s short-lived combinations may remain in storage even after their producers disappear. Monitor active series, creation rate, ingestion volume, memory, and query behavior instead of using one fixed bytes-per-series estimate for every deployment.

When it breaks

An innocent-looking diagnostic label can turn an aggregate metric into a per-request event store. Removing that label from a dashboard query does not undo the cost already paid to ingest and retain it. Hashing each identifier still preserves its distinctness, and truncating a random identifier generally only postpones the problem while introducing collisions. Choose a deliberately bounded category when aggregation is the actual requirement.

Do not collapse away distinctions needed to detect customer impact. Keep useful bounded outcome or service dimensions, and move request-level investigation to appropriately controlled logs or traces. If unwanted labels are removed during collection, check for duplicate resulting series within a scrape and for lost diagnostic meaning. Prefer fixing the producer where possible, then verify the new label set and plan how historical series age out.

Check it quickly

Save this as cardinality.py and run Python 3. The synthetic events exercise every route, method, and status combination fifty times. One version retains only those bounded dimensions; another also includes a unique request ID. The calculation assumes twenty replicas each expose the same combinations as classic histograms with six finite boundaries.

from itertools import product

routes = ["/orders/{order_id}", "/products/{product_id}"]
methods = ["GET", "POST"]
statuses = ["200", "500"]
events = list(product(routes, methods, statuses, range(50)))
bounded = {(route, method, status) for route, method, status, _ in events}
with_request_id = {(route, method, status, f"request-{i}")
 for i, (route, method, status, _) in enumerate(events)}

finite_buckets = 6
# Core classic-histogram series: finite buckets, +Inf, _sum, _count.
# Optional _created series are deliberately excluded from this calculation.
series_per_combination = finite_buckets + 3
replicas = 20
bounded_series = len(bounded) * series_per_combination * replicas
request_id_series = len(with_request_id) * series_per_combination * replicas

assert len(events) == 400 and len(bounded) == 8
assert len(with_request_id) == 400
assert bounded_series == 1440 and request_id_series == 72000
print(f"observed label sets: bounded={len(bounded)}, request_id={len(with_request_id)}")
print(f"core classic-histogram series: bounded={bounded_series}, request_id={request_id_series}")
print(f"multiplier={request_id_series // bounded_series}x")

Four hundred events collapse into eight bounded combinations. Adding request IDs keeps four hundred distinct combinations, giving 1,440 versus 72,000 core histogram series across the twenty replicas: a fiftyfold increase. Optional creation-time series and other metric families are excluded explicitly, so these are modeled series counts rather than a measurement of Prometheus memory or storage use.

The comparison demonstrates why aggregating events at instrumentation time changes the cost profile. Before approving another label, state the operational question it answers, its maximum expected value set, the combinations actually emitted, and the effect on each metric family. Then validate that estimate against the collector and server as the workload changes.

Sources

Prometheus: Data Model, series identity and changes in label sets, checked 2026-09-13.
Prometheus: Metric and Label Naming, bounded labels and high-cardinality dimensions, checked 2026-09-13.
Prometheus: Instrumentation, cardinality costs and instrumentation choices, checked 2026-09-13.

Read replicas serve demand copies

Mon, 12 Jul 2027 00:00:00 +0000

A read replica can serve queries while another database handles writes. That can move reporting, search, or other eligible reads away from the writer, but routing a connection is only part of the design. The application must specify which committed changes a result needs to include. This article uses a PostgreSQL physical hot standby, whose normal recovery mode permits reads but does not accept ordinary writes.

When it breaks

With asynchronous replication, a successful write can be followed by a stale replica read. A user changes an account and refreshes the page, yet the previous value reappears. The replica may be healthy and continuously receiving WAL; the required commit still has to be replayed. A freshness check based on receipt rather than replay tests the wrong boundary.

Even replay progress is insufficient when the query uses an older transaction snapshot. A repeatable-read transaction intentionally preserves its view as recovery advances. Waiting until a replica has applied a write does not refresh that transaction. A correct read-after-write path needs both adequate replay progress and a snapshot taken after the progress condition was satisfied.

The freshness promise must also be limited. Including this user’s earlier write does not guarantee that no newer transaction exists elsewhere. A balance used to authorize another debit still needs a transactional check on the write path. Replica read capacity does not replace the concurrency control that protects the business operation.

Check it quickly

Run this script as a non-root Linux account with PostgreSQL 18 server and client binaries in PATH. The recorded run used PostgreSQL 18.6. It builds private primary and standby clusters connected through Unix sockets, with no TCP listener. A named pipe keeps one reader session open so the snapshot boundary is controlled without installing a driver.

#!/bin/sh
set -eu
umask 077
lab=$(mktemp -d /tmp/xbcatm-read-replica.XXXXXX)
mkdir "$lab/socket"
primary=$lab/primary
standby=$lab/standby
reader_pid=''
reader_open=0
cleanup() {
 if test "$reader_open" = 1; then exec 3>&-; fi
 if test -n "$reader_pid"; then wait "$reader_pid" || :; fi
 for cluster in "$standby" "$primary"; do
 if test -f "$cluster/postmaster.pid"; then
 pg_ctl -D "$cluster" -m fast -w stop >/dev/null
 fi
 done
}
trap cleanup EXIT
initdb -D "$primary" --no-locale --encoding=UTF8 --auth-local=trust --auth-host=reject >/dev/null
cat >> "$primary/postgresql.conf" <<CONF
listen_addresses = ''
unix_socket_directories = '$lab/socket'
port = 56432
wal_level = replica
wal_keep_size = '64MB'
CONF
pg_ctl -D "$primary" -l "$lab/primary.log" -w -t 15 start >/dev/null
export PGHOST=$lab/socket PGPORT=56432 PGDATABASE=postgres PGUSER=$(id -un)
export PGCONNECT_TIMEOUT=2
psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
CREATE TABLE account (id integer PRIMARY KEY,balance integer);
INSERT INTO account VALUES (1,100);
SQL
pg_basebackup -D "$standby" -R -X stream -c fast -w
cat >> "$standby/postgresql.conf" <<CONF
port = 56433
hot_standby = on
CONF
pg_ctl -D "$standby" -l "$lab/standby.log" -w -t 15 start >/dev/null
mkfifo "$lab/reader.sql"
PGPORT=56433 PGAPPNAME=replica_snapshot psql -X -v ON_ERROR_STOP=1 -qAt <"$lab/reader.sql" >"$lab/reader.log" 2>&1 &
reader_pid=$!
exec 3>"$lab/reader.sql"
reader_open=1
printf '%s\n' 'BEGIN ISOLATION LEVEL REPEATABLE READ;' 'SELECT balance FROM account WHERE id=1;' >&3
attempt=0
while test "$(PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT EXISTS (SELECT FROM pg_stat_activity WHERE application_name='replica_snapshot' AND state='idle in transaction' AND backend_xmin IS NOT NULL)")" != t; do
 attempt=$((attempt+1))
 test "$attempt" -lt 100 || exit 1
 sleep 0.1
done
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT pg_wal_replay_pause()' >/dev/null
attempt=0
while test "$(PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT pg_get_wal_replay_pause_state()='paused'")" != t; do
 attempt=$((attempt+1))
 test "$attempt" -lt 100 || exit 1
 sleep 0.1
done
psql -X -v ON_ERROR_STOP=1 -qAt -c 'UPDATE account SET balance=80 WHERE id=1'
target=$(psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT pg_current_wal_lsn()')
printf 'primary_balance '
psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT balance FROM account WHERE id=1'
printf 'paused_replica_balance '
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT balance FROM account WHERE id=1'
for attempt in 1 2 3 4 5; do
 test "$(PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT pg_last_wal_replay_lsn()>='$target'::pg_lsn")" = f
 sleep 0.1
done
echo fresh_read_gate_timed_out
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT pg_wal_replay_resume()' >/dev/null
attempt=0
while test "$(PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT pg_last_wal_replay_lsn()>='$target'::pg_lsn")" != t; do
 attempt=$((attempt+1))
 test "$attempt" -lt 150 || exit 1
 sleep 0.1
done
test "$(PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT balance FROM account WHERE id=1')" = 80
echo fresh_snapshot_after_replay_80
printf '%s\n' 'SELECT balance FROM account WHERE id=1;' 'COMMIT;' >&3
exec 3>&-
reader_open=0
wait "$reader_pid"
reader_pid=''
test "$(cat "$lab/reader.log")" = "$(printf '100\n100')"
echo old_repeatable_read_snapshot_still_100
printf 'lab_directory %s\n' "$lab"

The fixture begins with a balance of one hundred. On the standby, a repeatable-read transaction performs an initial query and remains open. The script confirms its backend is idle inside a transaction with a snapshot, then pauses actual WAL replay. The primary commits a change to eighty and captures a post-commit WAL position as the read’s progress target.

Expected output shows primary_balance 80 and paused_replica_balance 100. Five replay checks separated by short waits remain false, producing fresh_read_gate_timed_out. That result refuses to certify freshness while the required change is unapplied. The exercise does not silently treat the stale result as acceptable or pretend those polling intervals form a precise end-to-end latency guarantee.

Replay then resumes. After its LSN reaches the target, a new standby query must read eighty. The original repeatable-read session queries again and still reads one hundred. Its saved output must contain one hundred twice, proving the old snapshot survived the change in replay progress. Both facts can hold simultaneously: the server has caught up, and that particular transaction still sees older data.

The script commits the held read transaction, closes its pipe, stops both servers, and prints the retained log directory. Its assertions inspect database values and actual replay state. No production routing or application connection pool is changed.

Make the routing promise explicit

A progress token must refer to the same database history as the destination replica. An LSN is not a globally unique business version across unrelated clusters or diverged timelines. Bind any production gating design to the relevant topology and handle failover explicitly. The fixture uses one fixed primary and its direct standby, so that identity is established by construction.

If freshness cannot be established within the request budget, choose an explicit application response or an authorized read from the writer with capacity available. Do not let a deadline quietly weaken the meaning of the result. Acquire a fresh snapshot after the gate succeeds, and keep transactions short enough that snapshot retention does not defeat later checks.

Long standby queries can also conflict with recovery. PostgreSQL may cancel them to let replay proceed; feedback settings trade such cancellations against retention pressure on the primary. Monitor replay delay, query cancellations, and transaction age together. Measure the read workload actually moved off the writer, alongside the replication and operational costs of maintaining the copy.

Sources

PostgreSQL 18 hot standby, read-only recovery, transaction snapshots, and recovery conflicts, checked 2026-09-12.
PostgreSQL 18 administrative functions, replay position and pause/resume semantics, checked 2026-09-10.
PostgreSQL 18 replication configuration, standby delay and feedback tradeoffs, checked 2026-09-12.

Scaling · XB Field Notes

The Horizontal Pod Autoscaler scales by a target utilization

A smoothed average decides

Pick the metric you can act on

Automation is orthogonal to defaults

When it breaks

Check it quickly

Further reading

Sources

TCP window scaling enlarges receive-window encoding in each direction

When it breaks

Check it quickly

Separate buffer allocation, advertised capacity, and observed performance

Sources

Cardinality is a metric budget

Count combinations that actually exist

Include the metric representation

When it breaks

Check it quickly

Sources

Read replicas serve demand copies

When it breaks

Check it quickly

Make the routing promise explicit

Sources