Performance · XB Field Notes

Huge pages cut TLB misses at the cost of flexibility

Tue, 21 Dec 2027 00:00:00 +0000

Databases and high-throughput servers often win, but they pay by needing contiguous memory and a pool sized in advance, so the trade is architectural, not free.

Why 4 KB hurts

On most architectures a standard page is 4 KiB, and the TLB can hold only a limited number of translations. A database or high-throughput server that touches gigabytes typically walks a working set of millions of pages, so it keeps missing the TLB, the hardware then has to walk the page table to find each mapping. The cost is not the page itself but the constant page-table walks, and at million-page working sets the misses outrun how fast the TLB can be refilled. Bigger pages mean each entry covers more memory, so the same working set needs far fewer entries, which is exactly the relief a memory-heavy workload feels.

hugetlb vs transparent

There are two ways to get bigger pages. hugetlb gives you a pool of explicitly sized pages, 2 MiB or more, that you allocate from a reserved area and manage yourself, which is deterministic and predictable; transparent huge pages (THP) try to promote mappings automatically in the background, which is convenient but can add latency spikes, fragmentation, and copy-on-write surprises on some workloads. For a database or a latency-sensitive service, hugetlb lets you own the pool and the trade, whereas THP leaves those decisions to a general policy you do not fully control. The choice is about who owns the sizing decision, you or a generic heuristics.

Budget the pool

The win of a large page depends on it being fully used and resident in memory. A 2 MiB page that is mostly unused still occupies a whole huge entry, so the memory it locks is effectively wasted, which makes pool sizing the whole skill. Reserve a pool only for the memory the application genuinely and predictably uses, rather than sizing every large allocation by feel, and account for the extra memory that a sparse page set costs. When the pages are fully resident and the working set trims into a small number of entries, huge pages deliver their whole benefit; when they are drafted for code that rarely fills them, they just burn RAM.

When it breaks

An under-sized pool fails loudly at startup. A database configured for more shared buffers than the reserved pages allow exits with Cannot allocate memory, even though free RAM looks plentiful, because hugetlb mappings cannot fall back to 4 KiB pages. Size the pool from the documented requirement plus margin, then confirm HugePages_Total and HugePages_Rsvd in /proc/meminfo after a restart.

A pool that worked at boot can fail later through fragmentation. Huge pages need contiguous 2 MiB ranges, so a long-running machine with free memory may still refuse a new mapping when no contiguous range remains. The symptom is a runtime error with HugePages_Free above zero plus a kernel message about a failed huge-page allocation; early reservation or boot parameters fix it, not retrying the same mmap.

Placement and automation cause quieter damage. Pages reserved on the wrong NUMA node force remote access on every hit, trading TLB wins for latency, so check per-node counts before pinning a database. Transparent huge pages add another risk: background compaction and copy-on-write splitting can turn a steady p99 into spikes, visible as thp_collapse_alloc stalls and khugepaged CPU. When latency matters, set THP to madvise and opt in only where measured.

Check it quickly

Save the following as hugepage_check.py and run python3 hugepage_check.py. It only reads /proc/meminfo and os.sysconf, allocates nothing, and needs no privileges, so it is safe in an unprivileged container. On a host without huge-page support or without /proc, it prints SKIP and shows the page-table arithmetic instead of raising.

#!/usr/bin/env python3
"""Report hugepage coverage and estimated TLB entries saved (read-only)."""
import os
import re
from pathlib import Path

WORKING_SET_BYTES = 8 * 1024**3 # 8 GiB example working set

def parse_meminfo(text):
 vals = {}
 for key in ("HugePages_Total", "HugePages_Free", "HugePages_Rsvd",
 "HugePages_Surp", "Hugepagesize"):
 m = re.search(r"^" + key + r":\s+(\d+)", text, re.M)
 if m:
 vals[key] = int(m.group(1))
 return vals

def main():
 vals = {}
 try:
 text = Path("/proc/meminfo").read_text(encoding="utf-8")
 except OSError as exc:
 print(f"SKIP: /proc/meminfo unavailable ({exc}); showing page-table math only")
 text = ""
 if text:
 try:
 vals = parse_meminfo(text)
 if "HugePages_Total" not in vals or "Hugepagesize" not in vals:
 print("SKIP: no HugePages fields in /proc/meminfo (hugepages unsupported or disabled)")
 else:
 size_kb = vals["Hugepagesize"]
 total = vals["HugePages_Total"]
 free = vals.get("HugePages_Free", 0)
 rsvd = vals.get("HugePages_Rsvd", 0)
 surp = vals.get("HugePages_Surp", 0)
 cov_mb = total * size_kb / 1024
 print(f"Hugepagesize: {size_kb} kB")
 print(f"HugePages total/free/rsvd/surp: {total}/{free}/{rsvd}/{surp}")
 print(f"Reserved coverage: {cov_mb:.1f} MiB ({cov_mb/1024:.2f} GiB)")
 except Exception as exc:
 print(f"SKIP: failed to parse /proc/meminfo ({exc})")
 try:
 base = os.sysconf("SC_PAGE_SIZE")
 except (AttributeError, ValueError, OSError):
 base = 4096
 huge = vals.get("Hugepagesize", 2048) * 1024
 base_entries = WORKING_SET_BYTES // base
 huge_entries = WORKING_SET_BYTES // huge
 print(f"Base page size (sysconf): {base} bytes")
 print(f"8 GiB needs ~{base_entries} base-page entries vs ~{huge_entries} huge-page entries")
 print(f"Estimated TLB entries saved: ~{base_entries - huge_entries}")

if __name__ == "__main__":
 main()

On Linux it names the huge-page size, the total, free, reserved, and surplus counters, and the reserved coverage in MiB and GiB, plus the estimated entries for an 8 GiB working set with base versus huge pages. On macOS or a minimal container you see SKIP plus the entry math with the local SC_PAGE_SIZE. It passed on macOS without /proc; it measures reservation and potential savings, not TLB misses, fragmentation, or NUMA distance, so confirm those with dmesg, per-node counts, and latency percentiles before resizing a pool.

Sources

Kernel hugepages documentation, defines hugetlb settings and behavior, checked 2026-06-01.
Kernel transparent hugepages documentation, defines THP modes and compaction behavior, checked 2026-06-01.

Cache-Control is a contract between cache and origin

Mon, 20 Dec 2027 00:00:00 +0000

A cache policy needs a behavioral test, not just a header-presence check. Two responses can both contain max-age yet behave differently because one also requires validation or prohibits storage. The request matters too: a browser can bypass an otherwise reusable copy without changing the origin’s response policy or deleting an older cached entry.

Both sides must agree

Cache-Control directives are directional. A request directive is not automatically copied into the response, and RFC 9111 describes request directives as advisory. Response directives impose requirements on conforming caches. Intermediaries must forward applicable cache directives even when they do not implement a cache themselves. Deployment-specific overrides need inspection; they are not evidence that the origin’s declared policy was followed.

Separate three questions. May this response be stored? Is its current age within the applicable freshness lifetime? Does this request permit reuse without validation? No-store prohibits HTTP cache storage; no-cache permits storage but requires successful validation before reuse. Max-age sets freshness, while must-revalidate prohibits unvalidated reuse after the response becomes stale. Must-revalidate does not require a round trip while the response is still fresh.

Private restricts storage to single-user caches; public explicitly permits caching subject to other requirements and is not mandatory on every cacheable response. S-maxage overrides freshness for shared caches, not browser caches. Vary supplies request-matching dimensions. None of these mechanisms identifies the current application account or makes personal data safe merely because its lifetime is short.

When it breaks

Testing with a new query parameter on every request changes the cache key and can hide reuse entirely. Disabling the browser cache in developer tools can create the same misleading result. Some automation interception mechanisms also disable caching. A useful test repeats the same URL in one isolated context and counts requests at the origin, while asserting the returned body as well.

Fetch’s cache option is a browser API, not a response header. Its no-store mode bypasses lookup and avoids updating the HTTP cache; reload bypasses lookup but can update it. Neither should be confused with erasing all previous copies. Likewise, changing origin headers cannot force an already-fresh entry to contact the origin immediately. History restoration and service-worker storage introduce additional behavior outside this test.

No-cache does not promise a 304. A validator and an unchanged representation are needed for that optimization. Without ETag or Last-Modified, repeated requests can transfer complete 200 responses. Count both origin visits and transferred bodies before concluding that validation is saving work.

Check it quickly

This was tested on macOS with Node 26.0.0, Playwright 1.63.0, and its Chromium 153.0.8010.12 headless shell. Install into a temporary directory; Linux also needs Playwright’s documented browser system dependencies. The download needs network access. Run the JavaScript block using NODE_PATH="$cache_browser_lab/node_modules" PLAYWRIGHT_BROWSERS_PATH="$cache_browser_lab/browsers" node. Test requests use only an automatically allocated loopback port, with no existing browser profile.

cache_browser_lab=$(mktemp -d)
npm install --prefix "$cache_browser_lab" --ignore-scripts --no-audit --no-fund --package-lock=false playwright@1.63.0
PLAYWRIGHT_BROWSERS_PATH="$cache_browser_lab/browsers" node "$cache_browser_lab/node_modules/playwright/cli.js" install chromium --only-shell --no-remove

const assert = require("node:assert/strict");
const http = require("node:http");
const { once } = require("node:events");
const { chromium } = require("playwright");

const policies = new Map([
 ["/fresh", "max-age=3600"],
 ["/must", "max-age=3600, must-revalidate"],
 ["/validate", "no-cache, max-age=3600"],
 ["/discard", "no-store, max-age=3600"],
]);
const hits = new Map();
const server = http.createServer((request, response) => {
 const route = new URL(request.url, "http://127.0.0.1").pathname;
 if (route === "/") {
 response.writeHead(200, { "Content-Type": "text/html", "Cache-Control": "no-store" });
 response.end("<!doctype html><title>Cache experiment</title>");
 } else if (policies.has(route)) {
 const count = (hits.get(route) || 0) + 1;
 hits.set(route, count);
 response.writeHead(200, {
 "Content-Type": "text/plain", "Cache-Control": policies.get(route),
 });
 response.end(`${route}:${count}`);
 } else {
 response.writeHead(404, { "Cache-Control": "no-store" });
 response.end();
 }
});

async function main() {
 server.listen(0, "127.0.0.1");
 await once(server, "listening");
 let browser;
 try {
 browser = await chromium.launch();
 console.log("Chromium", browser.version());
 const context = await browser.newContext({ serviceWorkers: "block" });
 try {
 const page = await context.newPage();
 await page.goto(`http://127.0.0.1:${server.address().port}/`);
 const read = (route, cache = "default") => page.evaluate(async ({ route, cache }) => {
 const response = await fetch(route, { cache });
 if (!response.ok) throw new Error(`HTTP ${response.status}`);
 return response.text();
 }, { route, cache });
 for (const [route, expectedHits] of [
 ["/fresh", 1], ["/must", 1], ["/validate", 2], ["/discard", 2],
 ]) {
 const first = await read(route);
 const second = await read(route);
 assert.equal(first, `${route}:1`);
 assert.equal(second, `${route}:${expectedHits}`);
 assert.equal(hits.get(route), expectedHits);
 console.log(route, "origin requests:", hits.get(route));
 }
 assert.equal(await read("/fresh", "no-store"), "/fresh:2");
 assert.equal(await read("/fresh"), "/fresh:1");
 assert.equal(hits.get("/fresh"), 2);
 assert.equal(await read("/fresh", "reload"), "/fresh:3");
 assert.equal(await read("/fresh"), "/fresh:3");
 assert.equal(hits.get("/fresh"), 3);
 console.log("no-store bypass retained version 1; reload stored version 3");
 } finally {
 await context.close();
 }
 } finally {
 if (browser) await browser.close();
 await new Promise((resolve, reject) => server.close(error => error ? reject(error) : resolve()));
 }
}

main().catch(error => { console.error(error); process.exitCode = 1; });

The initial origin counts are one, one, two, and two. Both fresh policies reuse the first body. The no-cache and no-store responses reach the origin again despite their large max-age. This fixture intentionally has no validators, so those visits return full bodies; it does not distinguish their storage eligibility by counts alone.

The later no-store fetch obtains version two, but an ordinary fetch still returns the previously cached version one. Reload obtains and stores version three, which the next ordinary fetch reuses. Origin counters ensure these are actual cache decisions, not coincidentally identical response strings. The server and isolated browser context are closed afterward.

These observations cover this Chromium HTTP cache and these exact requests. They do not verify a shared CDN, persistence across browser restarts, stale delivery, or user-account isolation. Keep those separate acceptance tests rather than extending a successful local cache experiment into a broader claim.

Sources

RFC 9111: Cache-Control, directional directives and storage/reuse requirements, checked 2026-09-07.
MDN: Request.cache, default, no-store, and reload behavior, checked 2026-09-07.
Playwright browser contexts, isolated non-persistent contexts, checked 2026-09-07.
Playwright browser installation, version-matched headless shell and system dependencies, checked 2026-09-07.

EXPLAIN ANALYZE runs the query to show real cost

Thu, 16 Dec 2027 00:00:00 +0000

EXPLAIN describes a PostgreSQL execution plan; EXPLAIN ANALYZE also executes the statement and measures its work. The useful comparison is between estimated rows and observed rows at the same node. Neither number is a verdict by itself: an estimate is a model, while a measurement describes one execution under particular data, cache, concurrency, and instrumentation conditions.

Read comparable quantities

Planner costs are relative units, not milliseconds. The two cost values describe startup and completion, and the preferred plan can change when LIMIT makes early rows especially valuable. Actual timings use milliseconds. Comparing a cost of 100 with an elapsed time of 100 is therefore meaningless even when the printed values happen to match.

For a node executed repeatedly, actual rows and node timings are averages per loop. Multiplying by loops helps expose repeated work, but summing every node’s time double-counts work included in parent nodes. Parallel execution makes naive totals still less useful. Read the tree, its loop counts, filters, and row flow before assigning blame to the visually largest number.

BUFFERS separates shared, local, and temporary block activity. Shared hits mean PostgreSQL found blocks in its shared buffer cache. A shared read does not prove a physical disk access: the operating system may satisfy it from another cache. Temporary tables use local buffers, while sort or hash spills can use temporary files. Parent counters include children, so adding all rows of counters inflates the apparent I/O.

Investigate the first meaningful discrepancy

A selective predicate estimated at ten rows but returning ten thousand deserves investigation before rewriting a join above it. Stale statistics, skew, correlated columns, parameter values, and unsupported selectivity models can all contribute. ANALYZE refreshes statistics, but it cannot guarantee exact estimates or repair every plan. Inspect representative data and consider targeted statistics changes only after identifying the missing information.

An accurate row estimate does not prove the query is efficient. Missing access paths, expensive expressions, spills, lock waits, and workload contention can remain. Conversely, a sequential scan can be appropriate for a small table or a query returning much of it. Measure the application’s actual query shape and parameters instead of treating every index-free plan as a defect.

When it breaks

The dangerous mistake is treating ANALYZE as a dry run. An UPDATE changes rows, triggers execute, locks are acquired, and a SELECT can call a function with side effects. Wrapping the statement in a transaction and rolling back restores transactional table changes, but sequence advances and external effects are not generally reversible. Use a disposable dataset when the statement’s effects are uncertain.

Timing also misleads when the first execution warms caches or instrumentation dominates a cheap operation. TIMING OFF retains actual row counts and total execution time while avoiding per-node clock reads. EXPLAIN discards ordinary result delivery, so its duration does not include sending the application’s full response over the network. Repeated representative measurements and application latency answer different parts of the performance question.

Check it quickly

Use psql against a disposable PostgreSQL 18 database. Save this block as explain.sql and run psql -X -v ON_ERROR_STOP=1 -f explain.sql. Connection options must name that database. Everything created here is temporary, and the final rollback removes it.

\set ON_ERROR_STOP on
BEGIN;
SET LOCAL statement_timeout = '5s';
CREATE TEMP TABLE orders (id integer, status text);
INSERT INTO orders SELECT g, CASE WHEN g <= 100 THEN 'paid' ELSE 'new' END
FROM generate_series(1, 10000) AS g;
ANALYZE orders;
EXPLAIN (COSTS ON) SELECT * FROM orders WHERE status = 'paid';
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT * FROM orders WHERE status = 'paid';
CREATE TEMP SEQUENCE measured_calls;
SAVEPOINT before_measurement;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
UPDATE orders SET id = id + 10000 WHERE id <= 3;
EXPLAIN (ANALYZE, TIMING OFF) SELECT nextval('measured_calls');
ROLLBACK TO SAVEPOINT before_measurement;
SELECT count(*) AS updated_rows_after_rollback FROM orders WHERE id > 10000;
SELECT nextval('measured_calls') AS sequence_after_rollback;
DO $$ BEGIN
 IF (SELECT count(*) FROM orders WHERE status = 'paid') <> 100 THEN
 RAISE EXCEPTION 'unexpected selected row count';
 END IF;
 IF EXISTS (SELECT FROM orders WHERE id > 10000) THEN
 RAISE EXCEPTION 'UPDATE was not rolled back';
 END IF;
 IF (SELECT last_value FROM measured_calls) <> 2 THEN
 RAISE EXCEPTION 'sequence advance unexpectedly rolled back';
 END IF;
END $$;
ROLLBACK;

The fixture contains exactly 100 paid orders. The measured SELECT reports 100 actual rows and 9,900 removed by its filter. The UPDATE’s scan reports three rows even though its top node returns zero without RETURNING; zero there does not mean nothing changed. After the savepoint rollback, the changed-row count is zero, but the next sequence value is two because the measured SELECT already consumed one.

On PostgreSQL 18.6 these assertions pass. Buffer counts, timings, formatting, and chosen plans can vary; the deterministic results are the row counts, rollback behavior, and sequence advance. The five-second statement timeout bounds individual statements in this exercise. It does not make arbitrary production writes safe to inspect.

Sources

PostgreSQL EXPLAIN, execution effects and instrumentation options, checked 2026-09-10.
PostgreSQL using EXPLAIN, costs, loops and buffer interpretation, checked 2026-09-10.
PostgreSQL planner statistics, estimates and statistics limits, checked 2026-09-10.
PostgreSQL sequence functions, sequence advances outside rollback, checked 2026-09-10.

pg_stat_statements finds the queries that cost the most

Sun, 12 Dec 2027 00:00:00 +0000

Because pg_stat_statements aggregates every execution of one query shape, it turns a vague “the page is slow” into a ranked table. The setup is a config line and a restart; the payoff is knowing which of your query shapes actually owns the clock.

The one decision that matters is which column you sort by. Sorting by mean time reproduces what the slow log already told you; sorting by total time is what finds the query that has been quietly consuming a quarter of your database.

Normalize, then aggregate

The extension does not store your SQL verbatim. Constants are folded into parameter symbols, so WHERE tenant_id = 42 and WHERE tenant_id = 8712 collapse into one entry, displayed as the text of the first query that produced that queryid. The numbering rule is positional — symbols start at the next number after the highest $n already present, which is why a PL/pgSQL statement like SELECT i + 1 INTO j shows up as SELECT i + $2. Lists behave the same way: two queries differing only in the number of IN elements get squashed into IN ($1 /*, ... */) with a call count of two.

The effect is dramatic: a modelled workload of 1,420,052 executions, each with different literals, collapsed into 6 rows — 236,675 raw statements per row. Two caveats come with it: queryid is a hash of the post-parse-analysis tree, so identical text can land in different entries when search_path differs, unrelated queries can collide into one entry, and the value is not promised to be stable across major versions. Treat queryid as a better identifier than the text, not as a permanent key.

Total time, not mean time

The columns that matter are calls, total_exec_time, mean_exec_time, rows, and the block counters. min/max/stddev_exec_time tell you whether a query is uniformly slow or occasionally long, which points at bad plans versus lock contention respectively.

Ranking by mean answers a different question from ranking by total, and the gap is large. In the same workload the query with the worst mean (9,689 ms, 12 calls) is 22.4% of measured time; the query with the worst total is 25.9% but averages 0.749 ms per call. The worst-by-total query never appears in a slow log at any sane log_min_duration_statement setting, and the three worst by total together are 65.5% of all measured time. A 0.099 ms lookup called 900,000 times ranks sixth by mean and third by total, at 89,217 ms.

Counters accumulate, so compare windows

Counters run from the moment statistics started, so a query that used to be slow never fades out of the top of the table on its own. The fix is either a scoped reset — pg_stat_statements_reset(userid, dbid, queryid, minmax_only) takes all four, so you can clear one query, one database, or just the min/max fields — or a delta between two snapshots.

The block counters deserve their own reading. shared_blks_hit and shared_blks_read are always tracked; track_io_timing adds the *_blk_read_time and *_blk_write_time columns, not the counts. Hit ratio is a diagnostic, not a cost: the query with the best hit ratio in the workload (98.89%) was also the largest block consumer at 81,000,000 blocks, while the query with the worst ratio (62.22%) moved 37x fewer.

When it breaks

The first failure is that the extension is simply not there. It must be listed in shared_preload_libraries because it needs shared memory, which means adding or removing it requires a restart, compute_query_id must be auto or on, and the views only exist in databases where CREATE EXTENSION pg_stat_statements was run. Miss any of those and every query against it fails, so the monitoring you thought you had has been empty for months.

The second failure is a reset that destroys the comparison. Reset the counters mid-window and every delta comes out negative: in the fixture, 6 of 6 shapes reported negative time. The arithmetic is fine; the window is not. pg_stat_statements_info.stats_reset exists precisely for this — check it before trusting any subtraction, and if the reset was intentional, start the window after it rather than trying to reconcile across it.

Check it quickly

Run both rankings and see how little they agree:

psql -c "SELECT calls, round(total_exec_time) AS total_ms, round(mean_exec_time,3) AS mean_ms, rows FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 10;"
psql -c "SELECT calls, round(mean_exec_time,2) AS mean_ms, round(total_exec_time) AS total_ms FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"
psql -c "SELECT stats_reset FROM pg_stat_statements_info;"
psql -c "SELECT sum(calls) AS calls, round(sum(total_exec_time)/1000) AS total_s, round(sum(shared_blks_read)*8.0/1024/1024,1) AS read_gib FROM pg_stat_statements;"
psql -c "SELECT pg_stat_statements_reset(0, 0, queryid) FROM pg_stat_statements WHERE query LIKE 'SELECT%';"

The first two lists should disagree, and the first is the one to work from. stats_reset is the timestamp that decides whether a delta window is valid. The totals line is the sanity check on magnitude — if total_s is larger than the wall-clock age of your statistics, you are reading across a restart. The last command resets selectively; run it deliberately, because it is exactly what makes the next comparison wrong.

Sources

PostgreSQL: pg_stat_statements, normalization, the statistics columns, and pg_stat_statements_reset, checked 2026-09-16.
PostgreSQL: cumulative statistics system, where stats_reset and the counters live, checked 2026-09-16.
PostgreSQL: statistics monitoring configuration, track_io_timing and the block-time columns it enables, checked 2026-09-16.

TCP window scaling enlarges receive-window encoding in each direction

Thu, 02 Dec 2027 00:00:00 +0000

TCP’s header has a 16-bit receive-window field. Window scaling lets established connections interpret that field using a shift negotiated during setup, allowing a receiver to advertise more than 65535 bytes of space. The two directions can use different shifts. The extension enlarges the encoding of receive-window information; it does not directly enlarge every buffer or guarantee that a sender will keep that much data in flight.

When it breaks

A high-bandwidth path with substantial RTT may need much more outstanding data than an unscaled receive window can describe. For example, 100 megabits per second over an 80-millisecond round trip corresponds to roughly one million bytes in flight. A 65535-byte window would impose a much lower idealized rate. This bandwidth-delay calculation is a sizing estimate, not a throughput measurement or a guarantee that no other limit applies.

The sender also has a congestion window and needs application data to send. Enlarging receive-window encoding cannot fix a small congestion budget, an application that stops reading, packet loss, or a blocked route. A scaled zero window is still zero. Diagnose which resource is actually constraining progress instead of treating the presence of a window-scale option as proof that the connection should saturate the link.

Negotiation errors can make captures misleading. Both endpoints must offer the option for scaling to operate, and each scale applies to windows advertised by the endpoint that supplied it. The window field in SYN and SYN-ACK is never scaled. Applying the wrong direction’s shift, or multiplying the SYN window, produces an incorrect capacity calculation even when the packet bytes are perfectly valid.

Check it quickly

Save this as check-window-scaling.py and run python3 check-window-scaling.py with Docker. The disposable environment uses Python 3.11.16, iproute2 6.1.0, and tcpdump 4.99.3, tested on Linux 6.12.76. NET_ADMIN and SYS_ADMIN enable nested namespaces and their temporary proc mount. The endpoints use a new veth link, and settings affect only that laboratory:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,('child exited without a response',process.stderr.read())
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found


def capture(stack,root,name,interface,expression,label):
 path=root/(label+'.pcap')
 process=subprocess.Popen(['ip','netns','exec',name,'tcpdump','--immediate-mode','-Z','root','-U','-n','-i',interface,'-w',str(path),expression],
 stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process)
 deadline=time.monotonic()+3
 while time.monotonic()<deadline:
 assert process.poll() is None,'tcpdump exited before listening'
 if select.select([process.stderr],[],[],.1)[0] and 'listening on' in process.stderr.readline():return process,path
 raise AssertionError('tcpdump did not become ready')
def packets(process,path):
 import struct,socket
 process.send_signal(signal.SIGINT);_,diagnostic=process.communicate(timeout=3)
 assert process.returncode==0,diagnostic
 raw=path.read_bytes();assert raw[:4]==bytes.fromhex('d4c3b2a1') and struct.unpack_from('<I',raw,20)[0]==1
 offset=24;result=[]
 while offset<len(raw):
 seconds,micros,size,original=struct.unpack_from('<IIII',raw,offset);offset+=16
 frame=raw[offset:offset+size];offset+=size
 if frame[12:14]!=b'\x08\x00' or frame[23]!=6:continue
 ihl=(frame[14]&15)*4;length=struct.unpack_from('!H',frame,16)[0];tcp=14+ihl
 sport,dport,seq,ack=struct.unpack_from('!HHII',frame,tcp);header=(frame[tcp+12]>>4)*4
 opts={};cursor=tcp+20
 while cursor<tcp+header:
 kind=frame[cursor]
 if kind==0:break
 if kind==1:cursor+=1;continue
 option_length=frame[cursor+1];data=frame[cursor+2:cursor+option_length];assert option_length>=2
 if kind==2:opts['mss']=struct.unpack('!H',data)[0]
 elif kind==3:opts['wscale']=data[0]
 elif kind==4:opts['sack_ok']=True
 elif kind==5:opts['sack']=[list(pair) for pair in struct.iter_unpack('!II',data)]
 cursor+=option_length
 body=frame[tcp+header:14+length]
 result.append({'t':seconds+micros/1e6,'src':socket.inet_ntoa(frame[26:30]),'dst':socket.inet_ntoa(frame[30:34]),
 'sport':sport,'dport':dport,'seq':seq,'ack':ack,'flags':frame[tcp+13],'window':struct.unpack_from('!H',frame,tcp+14)[0],
 'length':len(body),'options':opts,'preview':body[:24].hex()})
 return result
with lab(['client','target']) as (root,stack,setting):
 wire('client','c0','target','t0',['192.0.2.1/30'],['192.0.2.2/30'])
 server_code=r'''
import socket,json
def read_exact(stream,length):
 data=b''
 while len(data)<length:
 chunk=stream.recv(length-len(data));assert chunk,'premature EOF';data+=chunk
 return data
listener=socket.socket();listener.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
listener.setsockopt(socket.SOL_SOCKET,socket.SO_RCVBUF,262144)
listener.bind(('192.0.2.2',18080));listener.listen();print('listening',flush=True)
with listener.accept()[0] as connection:
 connection.settimeout(4);print(json.dumps({'rcvbuf':connection.getsockopt(socket.SOL_SOCKET,socket.SO_RCVBUF)}),flush=True)
 assert read_exact(connection,4)==b'ping';connection.sendall(b'pong')
 assert connection.recv(1)==b''
'''
 client_code=r'''
import socket,json,sys
def read_exact(stream,length):
 data=b''
 while len(data)<length:
 chunk=stream.recv(length-len(data));assert chunk,'premature EOF';data+=chunk
 return data
s=socket.socket();s.settimeout(4);s.setsockopt(socket.SOL_SOCKET,socket.SO_RCVBUF,65536)
s.bind(('192.0.2.1',40400+int(sys.argv[1])));s.connect(('192.0.2.2',18080));s.sendall(b'ping')
assert read_exact(s,4)==b'pong'
print(json.dumps({'rcvbuf':s.getsockopt(socket.SOL_SOCKET,socket.SO_RCVBUF),'reply':'pong'}),flush=True)
assert sys.stdin.readline().strip()=='close';s.close()
'''
 outcomes={}
 for enabled in [1,0]:
 setting('client','net.ipv4.tcp_window_scaling',enabled);setting('target','net.ipv4.tcp_window_scaling',enabled)
 service=spawn(stack,['ip','netns','exec','target','python3','-u','-c',server_code]);assert line(service).strip()=='listening'
 tap,path=capture(stack,root,'client','c0','tcp port 18080','window'+str(enabled))
 client=spawn(stack,['ip','netns','exec','client','python3','-u','-c',client_code,str(enabled)])
 client_result=json.loads(line(client));server_result=json.loads(line(service))
 diagnostics=ns('client','ss','-tin','state','established').stdout
 client.stdin.write('close\n');client.stdin.flush();client.wait(timeout=2);service.wait(timeout=2)
 assert client.returncode==service.returncode==0
 observed=packets(tap,path);syn=next(p for p in observed if p['flags']&0x12==0x02)
 synack=next(p for p in observed if p['flags']&0x12==0x12)
 advertised=next(p for p in observed if p['sport']==18080 and p['length']==4)
 if enabled:
 assert 'wscale' in syn['options'] and 'wscale' in synack['options']
 effective=advertised['window']<<synack['options']['wscale'];assert effective>65535
 else:
 assert 'wscale' not in syn['options'] and 'wscale' not in synack['options'];effective=advertised['window'];assert effective<=65535
 outcomes[enabled]={'client':client_result,'server':server_result,'syn':syn,'synack':synack,
 'server_post_handshake_window_field':advertised['window'],'effective_receive_window':effective,'ss':diagnostics}
 print(json.dumps(outcomes))
"""
name='xbcatm-tcpflow-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

Before opening connections, the client requests a receive buffer of 65536 bytes and the server listener requests 262144 bytes. The script records the effective socket-option values rather than assuming the requested sizes are used literally. Linux buffer accounting and limits matter: this run reported 131072 on the client and 425984 on the accepted server socket. Neither number should simply be equated with an advertised TCP window.

With scaling enabled in both namespaces, the captured client SYN offers shift one and the server SYN-ACK offers shift two. Their initial window fields remain unscaled. After setup, a server data segment advertises window field 53213. Applying the server’s shift gives 53213 times four, or 212852 bytes. The client socket diagnostic independently reports the corresponding send-window value.

The next fresh connection disables scaling in both namespaces while keeping the buffer requests and application exchange the same. Neither SYN carries a scale option, and the observed server receive window is 65535 bytes. Both trials send ping and receive pong, checking actual communication as well as negotiation. The comparison demonstrates the encoding limit without pretending that a four-byte exchange measures maximum throughput.

The fixture captures setup because the scale is fixed for that connection. Changing a system default later does not renegotiate an existing stream. Configuring appropriate receive-buffer behavior before connect or listen matters when it influences the scale selected during setup. A packet trace starting after the handshake needs the previously negotiated state before its raw window fields can be interpreted correctly.

Separate buffer allocation, advertised capacity, and observed performance

Record the relevant socket settings, negotiated shifts, current advertised window, congestion window, RTT, and application activity. Those describe different constraints. A large allocation can coexist with a smaller advertised window because the stack accounts for buffering and current occupancy. The experiment’s reported allocation and wire-level value intentionally make that distinction visible rather than hiding it behind one ambiguous “window size.”

If a transfer underperforms, check whether the receiver keeps consuming data and whether the sender has enough work to use its permitted flight size. Review losses and congestion behavior before changing buffer budgets. Larger allocations across many concurrent sockets also consume resources, so tune with workload evidence rather than maximizing every available limit.

The protocol limits the shift to fourteen, giving a representable window below one gibibyte. That encoding ceiling is not a requirement to allocate that much memory. Validate the deployed kernel and representative traffic after a change. This example verifies negotiation, direction, arithmetic, and a successful exchange; sustained capacity and application behavior need measurements suited to the real path.

Sources

RFC 7323 TCP Extensions, directional scale factors, SYN exceptions and receive-window calculation, checked 2026-09-12.
Linux tcp(7), receive buffers, window scaling and socket options before connection setup, checked 2026-09-12.
RFC 5681 Congestion Control, the separate limits imposed by congestion and receive windows, checked 2026-09-12.

A cache stampede hits when expiry syncs

Mon, 22 Nov 2027 00:00:00 +0000

A cache stampede occurs when concurrent misses cause redundant work against the same dependency. Synchronized expiration is one trigger, but eviction, a cold deployment, or cache unavailability can create the same burst. A single popular key can be enough. The useful question is how many rebuilds reach the source while one result is still being computed.

Coalesce work within its actual boundary

Go’s singleflight Group gives one key at most one in-flight function execution within that Group. Duplicate callers receive the same result, including the same error. Separate Groups do not coordinate, even within one process, and separate service instances normally have separate Groups. It is neither a distributed lock nor a cache of completed values.

Keep the Group shared across requests, and make its key identify equivalent work. Mixing tenants, permissions, currencies, or representations under one key can share the wrong result. In a cache-backed loader, check the cache again inside the shared function, fetch only if still necessary, and publish a successful value before returning. Singleflight itself does none of those storage operations.

Spread deadlines without changing the promise

Jitter spreads expiration across different entries or independently maintained copies. It does not prevent all callers of one expired hot key from colliding. If a maximum TTL is a freshness requirement, randomly shortening lifetimes within a chosen bound avoids silently exceeding it; adding arbitrary positive jitter changes that requirement. There is no universally correct percentage or number of seconds.

Early refresh and permitted stale responses can keep callers from waiting, but they require explicit application rules. A last-known public catalog may be acceptable temporarily; a revoked permission or current balance may not be. Coalescing reduces duplicate work without deciding whether old data is safe, and a cold cache still needs a source response.

When it breaks

A stuck shared loader can hold many waiters. Bound the loader’s dependency calls and give callers their own waiting deadlines. DoChan returns a result channel, not a cancellation mechanism: one caller abandoning its wait does not automatically stop the underlying work. Forget allows a new execution for the key; it does not cancel the old execution, so it can reintroduce overlapping loads.

Errors are shared only while that execution is in flight. Immediate retries after it finishes can create successive waves against a failing dependency. Use a retry and admission policy matched to that dependency; cacheable absence and transient failure need different treatment. Likewise, a fleet of one hundred local Groups can still create one hundred rebuilds. Do not describe one-process suppression as a global origin-load bound.

Check it quickly

Use Go 1.25 or newer; this exercise was run with Go 1.27.0 and x/sync v0.22.0. Create an isolated module below, put the Go block in main.go there, and run go run -race main.go. Dependency installation requires the network. The experiment starts at the cold-miss boundary and invokes counted loader functions, not a real database, cache server, or timed TTL implementation.

stampede_lab=$(mktemp -d)
cd "$stampede_lab"
go mod init example.invalid/stampede-lab
go get golang.org/x/sync@v0.22.0

package main

import (
 "errors"
 "fmt"
 "sync"
 "sync/atomic"

 "golang.org/x/sync/singleflight"
)

const callers = 12

func require(ok bool) {
 if !ok {
 panic("unexpected result")
 }
}

func wave(groups []*singleflight.Group, failure error) int32 {
 var calls atomic.Int32
 gate := make(chan struct{})
 results := make([]<-chan singleflight.Result, callers)
 load := func() (any, error) {
 calls.Add(1)
 <-gate
 if failure != nil {
 return nil, failure
 }
 return "catalog-v1", nil
 }
 for i := range results {
 results[i] = groups[i%len(groups)].DoChan("catalog", load)
 }
 close(gate)
 for _, result := range results {
 r := <-result
 require(r.Shared && r.Err == failure)
 if failure == nil {
 require(r.Val == "catalog-v1")
 } else {
 require(r.Val == nil)
 }
 }
 return calls.Load()
}

func main() {
 var plain atomic.Int32
 var workers sync.WaitGroup
 gate := make(chan struct{})
 for range callers {
 workers.Add(1)
 go func() {
 defer workers.Done()
 plain.Add(1)
 <-gate
 }()
 }
 close(gate)
 workers.Wait()
 require(plain.Load() == callers)
 fmt.Println("without coalescing: 12 loader calls")

 group := new(singleflight.Group)
 require(wave([]*singleflight.Group{group}, nil) == 1)
 require(wave([]*singleflight.Group{group}, nil) == 1)
 fmt.Println("one Group: 1 call per wave; completed results are not cached")
 other := new(singleflight.Group)
 require(wave([]*singleflight.Group{group, other}, nil) == 2)
 fmt.Println("two independent Groups: 2 loader calls")

 failure := errors.New("origin unavailable")
 require(wave([]*singleflight.Group{group}, failure) == 1)
 require(wave([]*singleflight.Group{group}, nil) == 1)
 fmt.Println("one error reaches 12 callers; next wave starts a new load")
}

The baseline executes twelve loaders. Each single-Group wave executes one, and running a second wave on the same Group executes another: finished results are not retained. Splitting the twelve callers between two Groups produces two executions. A failing wave sends the same error to all twelve callers; the next successful wave starts fresh rather than retrieving a cached failure.

The gate keeps the shared load pending until every DoChan call has registered, so the test does not depend on sleeps or favorable scheduling. Shared is true for all results in these joined waves, including the initiating caller’s result. Receive exactly one result from each channel; DoChan does not close those channels, so ranging over one would wait indefinitely.

For a service-level follow-up, measure loader executions, waiting callers, latency, and failures during a deliberately cold test in an isolated environment. Include multiple instances and source timeouts, not just a high warm-cache hit ratio. The race-detector run validates this program’s observed execution, not every possible race in a production cache implementation.

Sources

Go x/sync v0.22.0: singleflight, Group scope, DoChan, shared errors, and Forget, checked 2026-09-07.
AWS Builders’ Library: caching challenges and strategies, cold caches, request coalescing, failure behavior, and empirical expiration choices, checked 2026-09-07.

Nagle and delayed ACK can stall the second part of a small request

Sun, 31 Oct 2027 00:00:00 +0000

Nagle’s algorithm reduces small TCP segments by holding additional short data while earlier data remains unacknowledged, unless enough data accumulates for a full segment. Delayed acknowledgment is a separate receiver behavior that can postpone an ACK briefly. The interaction matters when an application sends a request in multiple small writes and the peer needs the later bytes before producing a response.

When it breaks

An isolated small request is not automatically delayed by Nagle. If no earlier data is outstanding, the first small segment can be sent. The troublesome case has a first piece already in flight and a second short piece waiting behind it. If the receiver delays acknowledgment while its application waits for the complete request, progress can pause until the acknowledgment timer expires.

This is a bounded delay, not an indefinite protocol deadlock. An ACK can release the queued data, and accumulating enough data can also change the sender’s decision. The observed delay is not universally one additional RTT or a fixed forty milliseconds. Receiver policy, existing acknowledgment state, request framing, and network conditions determine the actual packet sequence and timing.

TCP_NODELAY disables Nagle on the selected socket; it does not disable every form of buffering, congestion control, receive-window flow control, or scheduler delay. TCP_QUICKACK on Linux requests a change in acknowledgment mode, but that mode is not permanent. Treating either option as a universal latency switch can hide application buffering or a different transport constraint that the option never addressed.

Check it quickly

Save this as check-nagle.py and run python3 check-nagle.py with Docker. The disposable environment uses Python 3.11.16, iproute2 6.1.0, and tcpdump 4.99.3, tested on Linux 6.12.76. NET_ADMIN and SYS_ADMIN enable nested namespaces. Client and target communicate over a new veth link; all settings stay inside that laboratory rather than changing host networking:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,('child exited without a response',process.stderr.read())
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found


def capture(stack,root,name,interface,expression,label):
 path=root/(label+'.pcap')
 process=subprocess.Popen(['ip','netns','exec',name,'tcpdump','--immediate-mode','-Z','root','-U','-n','-i',interface,'-w',str(path),expression],
 stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process)
 deadline=time.monotonic()+3
 while time.monotonic()<deadline:
 assert process.poll() is None,'tcpdump exited before listening'
 if select.select([process.stderr],[],[],.1)[0] and 'listening on' in process.stderr.readline():return process,path
 raise AssertionError('tcpdump did not become ready')
def packets(process,path):
 import struct,socket
 process.send_signal(signal.SIGINT);_,diagnostic=process.communicate(timeout=3)
 assert process.returncode==0,diagnostic
 raw=path.read_bytes();assert raw[:4]==bytes.fromhex('d4c3b2a1') and struct.unpack_from('<I',raw,20)[0]==1
 offset=24;result=[]
 while offset<len(raw):
 seconds,micros,size,original=struct.unpack_from('<IIII',raw,offset);offset+=16
 frame=raw[offset:offset+size];offset+=size
 if frame[12:14]!=b'\x08\x00' or frame[23]!=6:continue
 ihl=(frame[14]&15)*4;length=struct.unpack_from('!H',frame,16)[0];tcp=14+ihl
 sport,dport,seq,ack=struct.unpack_from('!HHII',frame,tcp);header=(frame[tcp+12]>>4)*4
 opts={};cursor=tcp+20
 while cursor<tcp+header:
 kind=frame[cursor]
 if kind==0:break
 if kind==1:cursor+=1;continue
 option_length=frame[cursor+1];data=frame[cursor+2:cursor+option_length];assert option_length>=2
 if kind==2:opts['mss']=struct.unpack('!H',data)[0]
 elif kind==3:opts['wscale']=data[0]
 elif kind==4:opts['sack_ok']=True
 elif kind==5:opts['sack']=[list(pair) for pair in struct.iter_unpack('!II',data)]
 cursor+=option_length
 body=frame[tcp+header:14+length]
 result.append({'t':seconds+micros/1e6,'src':socket.inet_ntoa(frame[26:30]),'dst':socket.inet_ntoa(frame[30:34]),
 'sport':sport,'dport':dport,'seq':seq,'ack':ack,'flags':frame[tcp+13],'window':struct.unpack_from('!H',frame,tcp+14)[0],
 'length':len(body),'options':opts,'preview':body[:24].hex()})
 return result
with lab(['client','target']) as (root,stack,setting):
 wire('client','c0','target','t0',['192.0.2.1/30'],['192.0.2.2/30'])
 setting('client','net.ipv4.tcp_autocorking',0)
 server_code=r'''
import socket,json
def read_exact(stream,length):
 data=b''
 while len(data)<length:
 chunk=stream.recv(length-len(data));assert chunk,'premature EOF';data+=chunk
 return data
listener=socket.socket();listener.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
listener.bind(('192.0.2.2',18080));listener.listen();print('listening',flush=True)
for trial in range(2):
 with listener.accept()[0] as connection:
 connection.settimeout(3);connection.setsockopt(socket.IPPROTO_TCP,socket.TCP_QUICKACK,0)
 print('configured',flush=True)
 request=read_exact(connection,2);assert request==b'ab',request
 connection.sendall(b'OK');print(json.dumps({'request':request.decode()}),flush=True)
'''
 service=spawn(stack,['ip','netns','exec','target','python3','-u','-c',server_code]);assert line(service).strip()=='listening'
 tap,path=capture(stack,root,'client','c0','tcp port 18080','nagle')
 client_code=r'''
import json,socket,sys,time
def read_exact(stream,length):
 data=b''
 while len(data)<length:
 chunk=stream.recv(length-len(data));assert chunk,'premature EOF';data+=chunk
 return data
nodelay=int(sys.argv[1]);s=socket.socket();s.settimeout(3)
s.setsockopt(socket.IPPROTO_TCP,socket.TCP_NODELAY,nodelay);s.bind(('192.0.2.1',40300+nodelay));s.connect(('192.0.2.2',18080))
print('connected',flush=True);assert sys.stdin.readline().strip()=='go'
start=time.monotonic();s.sendall(b'a');time.sleep(.002);s.sendall(b'b');reply=read_exact(s,2)
assert reply==b'OK';print(json.dumps({'nodelay':nodelay,'reply':reply.decode(),'elapsed_ms':(time.monotonic()-start)*1000}),flush=True);s.close()
'''
 outcomes=[]
 for nodelay in [0,1]:
 client=spawn(stack,['ip','netns','exec','client','python3','-u','-c',client_code,str(nodelay)])
 assert line(client).strip()=='connected' and line(service).strip()=='configured'
 client.stdin.write('go\n');client.stdin.flush();outcomes.append(json.loads(line(client)))
 assert json.loads(line(service))['request']=='ab';client.wait(timeout=2);assert client.returncode==0
 observed=packets(tap,path);timelines={}
 for nodelay in [0,1]:
 port=40300+nodelay;flow=[p for p in observed if port in (p['sport'],p['dport'])]
 first=next(p for p in flow if p['sport']==port and p['preview']=='61')
 second=next(p for p in flow if p['sport']==port and p['preview']=='62')
 assert (second['seq']-first['seq'])%2**32==1
 acks=[p for p in flow if p['dport']==port and p['flags']&0x10 and 0<(p['ack']-first['seq'])%2**32<=2]
 assert acks
 if not nodelay:
 intermediate=next(p for p in acks if (p['ack']-first['seq'])%2**32==1)
 assert first['t']<intermediate['t']<=second['t'] and second['t']-first['t']>.01
 else:assert second['t']<=acks[0]['t']
 timelines[nodelay]={'first_byte':first,'second_byte':second,'first_ack_covering_data':acks[0],
 'data_segment_gap_ms':(second['t']-first['t'])*1000}
 print(json.dumps({'trials':outcomes,'packet_timelines':timelines,'receiver_quickack_requested':0,'client_autocorking':0}))
"""
name='xbcatm-tcpflow-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The server waits for the complete request ab before replying OK. Before each trial, it requests non-quick acknowledgment mode on the accepted connection. The client disables namespace-local automatic corking to remove that additional coalescing mechanism from the comparison. Parent-child synchronization ensures the receiver has configured its socket before the client starts sending application bytes.

In the first trial, TCP_NODELAY is zero. The client sends a, pauses for two milliseconds, and sends b. Those writes are application actions, not a guarantee of immediate separate packets. The capture shows the first byte leaving, an ACK covering that byte arriving, and only then the second byte leaving. The program checks this ordering instead of attributing any slow response to Nagle without packet evidence.

In the recorded run, the two data segments were approximately forty-one milliseconds apart, and the complete exchange took roughly the same time. The second trial enables TCP_NODELAY and repeats the same framing and deliberate pause. The second byte leaves before the first data acknowledgment, with about two milliseconds between segments. The server receives the complete request and returns the same valid reply in both cases.

The timing values describe this controlled run, not a production latency guarantee. The fixture expects a visible delayed-ACK interaction under its stated settings and checks the causal packet order. A different kernel or receiver policy can require fresh investigation. Stream reads explicitly accumulate the expected number of bytes because recv, even with a requested length, does not generally promise an entire application message at once.

Choose framing and socket policy together

For a small logical request already available in memory, consider writing its complete representation together. That can avoid waiting for a later fragment while preserving useful transport batching. For an interactive stream that must deliver independent small updates promptly, TCP_NODELAY may be appropriate. Choose using the workload’s framing and measured behavior rather than applying one setting to every connection.

Capture both data and acknowledgments when a request periodically stalls. Determine whether the second piece is still in application buffering, queued in the sender, or already received by the peer. A timeout in application code alone cannot distinguish those stages. Correlate the capture with socket configuration and the point at which the receiver considers a request complete.

Also account for packet-rate and bandwidth costs when sending many tiny updates. Reducing an unnecessary batching delay can improve interaction while increasing segments per operation. Measure the relevant response latency and resource cost together. The useful result is a justified policy for a particular exchange, supported here by two successful requests and different, directly observed transmission ordering.

Sources

RFC 896 Congestion Control, coalescing small writes while earlier data remains unacknowledged, checked 2026-09-12.
RFC 1122 Host Requirements, delayed acknowledgment behavior and bounded acknowledgment delay, checked 2026-09-12.
Linux tcp(7), TCP_NODELAY and the temporary nature of TCP_QUICKACK, checked 2026-09-12.

Full table scans win on small tables

Thu, 16 Sep 2027 00:00:00 +0000

An index is a bet: you pay write and storage cost up front so reads can skip most of the table. When a query genuinely needs most of the table, the bet loses: the engine descends a B-tree per row instead of reading the pages once in order.

Planners price both routes, so the real question is where the crossover sits. A small table is not a small amount of work, and the plan you see at a thousand rows is not the plan to expect at ten million.

The currency is pages, not rows

A sequential scan reads the table’s pages in physical order, many rows per page. Postgres prices that with seq_page_cost (1.0 by default) against random_page_cost (4.0 by default): a non-sequentially fetched page is priced four times a sequential one, and lowering random_page_cost relative to seq_page_cost is precisely what makes the planner start preferring index scans. When the whole database fits in cache, the documentation suggests the two can reasonably be equal.

Because pages are the currency, row count is a poor proxy for size. Two tables of 1,000 rows: a 4-byte payload is 12 pages, a 4,000-byte payload is 1,010. On sqlite 3.53 with a 2 MB page cache the broad query over the narrow table finished in 0.02 ms and over the wide one in 1.30 ms — a 65x spread at identical row counts. Only one of those tables is small.

Where the index stops paying

Forcing each access path makes the crossover visible. The table is events(id, tenant, payload), 1,000 distinct tenants, an index on tenant, predicate matching roughly 90% of rows:

1,000 rows in 21 pages: scan 0.04 ms, index 0.05 ms. A tie. The index buys nothing at this size and still has to be maintained on every write.
50,000 rows in 767 pages: scan 1.79 ms, index 6.61 ms. The scan wins by 3.7x.
1,000,000 rows in 15,474 pages (63 MB): scan 44.37 ms, index 531.18 ms. The scan wins by 12x, and across seven samples the ratio never dropped below 11.3x.

The same index on the same million-row table is a 29x win for a selective predicate: tenant = 7 covers about 0.1% of rows and took 0.61 ms through the index against 17.88 ms for the scan. The index is not bad; it is bad for that predicate. Selectivity decides, and it decides harder as the table grows.

Let the planner decide, but keep its inputs honest

sqlite chose the scan for the broad predicate at every size and the index for the selective one. That is the correct default, and the lever when a plan looks wrong is the statistics, not a hint.

Statistics decay and the plan decays with them. ANALYZE the million-row table, then move every row to tenant = 5 without re-analyzing. sqlite_stat1 still reports 1,000 rows per key, so the planner keeps picking SEARCH ... USING INDEX for a predicate matching 100% of the table. Run ANALYZE and the statistic becomes 1,000,000 rows per key and the plan flips to SCAN. The wrong plan cost only 1.2x here, because sqlite’s table b-tree is keyed by rowid so the index walk still fetches pages in near-physical order; in Postgres each index hit is a separate heap fetch priced at random_page_cost.

When it breaks

The classic failure is the plan that flips when statistics go stale. A nightly job loads a tenant that comes to dominate the table; the histogram still describes last week; the dashboard queries that used to be index lookups now read a fifth of the table through the index. Nothing errors. Latency simply steps up, on exactly the queries nobody was watching.

The second failure is scale, not shape. The scan that costs 0.03 ms over 21 pages costs 1.88 ms over 767 and 41.34 ms over 15,474, because cost is linear in pages. A table that is fine today becomes an unaffordable full scan after a year of growth. The fix is an index the planner will actually choose, or a query that returns fewer rows.

Check it quickly

Confirm the table is small in pages, then see whether the planner agrees with reality:

sqlite3 events.db "PRAGMA page_count; PRAGMA page_size;"
sqlite3 events.db "EXPLAIN QUERY PLAN SELECT sum(length(payload)) FROM events WHERE tenant < 900;"
sqlite3 events.db "EXPLAIN QUERY PLAN SELECT sum(length(payload)) FROM events NOT INDEXED WHERE tenant < 900;"
sqlite3 events.db "ANALYZE; SELECT stat FROM sqlite_stat1 WHERE idx='events_tenant';"
printf '.timer on\nSELECT sum(length(payload)) FROM events NOT INDEXED WHERE tenant < 900;\n' | sqlite3 events.db

The first line gives you pages, the number that matters. The next two should name the same path for a broad predicate: if the planner says SEARCH ... USING INDEX while the forced SCAN is measurably faster, the statistics are the suspect. sqlite_stat1 shows the average rows per key — when it still says 1000 while one key owns the table, re-run ANALYZE and read the plan again.

Sources

PostgreSQL: planner cost constants, seq_page_cost and random_page_cost, checked 2026-09-16.
PostgreSQL: using EXPLAIN, reading plans and row estimates, checked 2026-09-16.
SQLite: EXPLAIN QUERY PLAN, how scan and search steps are reported, checked 2026-09-16.

SACK reports received ranges beyond a gap so recovery can target missing data

Sat, 11 Sep 2027 00:00:00 +0000

A TCP cumulative acknowledgment identifies the next byte expected in order. If a segment is missing but later bytes arrive, that acknowledgment cannot advance across the hole. Selective Acknowledgment, or SACK, adds information about received byte ranges beyond the cumulative boundary. It helps a sender distinguish missing data from data already queued at the receiver, improving the information available to its recovery algorithm.

When it breaks

Without SACK, a cumulative ACK does not force the sender to retransmit every later segment. A sender can use duplicate ACKs, timers, and recovery algorithms to infer loss, but the feedback is less explicit, particularly with multiple gaps. Describing ordinary TCP as mandatory retransmission of the entire suffix confuses a limitation in information with a required recovery policy.

SACK blocks describe received ranges, not a direct list of missing ranges. Each block has an inclusive left edge and an exclusive right edge in sequence space. A hole is inferred by comparing those blocks with the cumulative acknowledgment and outstanding data. SACK does not change what the ordinary ACK field means, and the application still receives an ordered byte stream rather than out-of-order blocks.

The capability must also be permitted for the connection. SACK-Permitted in a SYN lets the peer send SACK options back to the SYN sender; it does not prove any SACK block was needed during a clean transfer. Conversely, missing SACK information is not automatically evidence that a firewall stripped an option. Inspect endpoint configuration and the actual handshake before selecting a cause.

Check it quickly

Save this as check-sack.py and run python3 check-sack.py with Docker. The disposable environment uses Python 3.11.16, iproute2 6.1.0, nftables 1.0.6, and tcpdump 4.99.3, tested on Linux 6.12.76. NET_ADMIN and SYS_ADMIN support nested network namespaces and their temporary proc mount. The example uses only a new veth link and namespace-local settings:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,'child exited without a response'
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found


def capture(stack,root,name,interface,expression,label):
 path=root/(label+'.pcap')
 process=subprocess.Popen(['ip','netns','exec',name,'tcpdump','--immediate-mode','-Z','root','-U','-n','-i',interface,'-w',str(path),expression],
 stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process)
 deadline=time.monotonic()+3
 while time.monotonic()<deadline:
 assert process.poll() is None,'tcpdump exited before listening'
 if select.select([process.stderr],[],[],.1)[0] and 'listening on' in process.stderr.readline():return process,path
 raise AssertionError('tcpdump did not become ready')
def packets(process,path):
 import struct,socket
 process.send_signal(signal.SIGINT);_,diagnostic=process.communicate(timeout=3)
 assert process.returncode==0,diagnostic
 raw=path.read_bytes();assert raw[:4]==bytes.fromhex('d4c3b2a1') and struct.unpack_from('<I',raw,20)[0]==1
 offset=24;result=[]
 while offset<len(raw):
 seconds,micros,size,original=struct.unpack_from('<IIII',raw,offset);offset+=16
 frame=raw[offset:offset+size];offset+=size
 if frame[12:14]!=b'\x08\x00' or frame[23]!=6:continue
 ihl=(frame[14]&15)*4;length=struct.unpack_from('!H',frame,16)[0];tcp=14+ihl
 sport,dport,seq,ack=struct.unpack_from('!HHII',frame,tcp);header=(frame[tcp+12]>>4)*4
 opts={};cursor=tcp+20
 while cursor<tcp+header:
 kind=frame[cursor]
 if kind==0:break
 if kind==1:cursor+=1;continue
 option_length=frame[cursor+1];data=frame[cursor+2:cursor+option_length];assert option_length>=2
 if kind==2:opts['mss']=struct.unpack('!H',data)[0]
 elif kind==3:opts['wscale']=data[0]
 elif kind==4:opts['sack_ok']=True
 elif kind==5:opts['sack']=[list(pair) for pair in struct.iter_unpack('!II',data)]
 cursor+=option_length
 body=frame[tcp+header:14+length]
 result.append({'t':seconds+micros/1e6,'src':socket.inet_ntoa(frame[26:30]),'dst':socket.inet_ntoa(frame[30:34]),
 'sport':sport,'dport':dport,'seq':seq,'ack':ack,'flags':frame[tcp+13],'window':struct.unpack_from('!H',frame,tcp+14)[0],
 'length':len(body),'options':opts,'preview':body[:24].hex()})
 return result
with lab(['client','target']) as (root,stack,setting):
 wire('client','c0','target','t0',['192.0.2.1/30'],['192.0.2.2/30'])
 setting('client','net.ipv4.tcp_sack',1);setting('target','net.ipv4.tcp_sack',1)
 setting('client','net.ipv4.tcp_autocorking',0)
 server_code=r'''
import socket,json,hashlib
listener=socket.socket();listener.bind(('192.0.2.2',18080));listener.listen();print('listening',flush=True)
with listener.accept()[0] as connection:
 connection.settimeout(6);data=b''
 while len(data)<12*256:
 chunk=connection.recv(4096);assert chunk;data+=chunk
 expected=b''.join(bytes([i])*256 for i in range(12));assert data==expected
 result={'bytes':len(data),'sha256':hashlib.sha256(data).hexdigest()}
 connection.sendall(json.dumps(result).encode()+b'\n');print(json.dumps(result),flush=True)
'''
 service=spawn(stack,['ip','netns','exec','target','python3','-u','-c',server_code]);assert line(service).strip()=='listening'
 nft('target','''table ip one_gap {
 chain input { type filter hook input priority 0; policy accept;
 tcp dport 18080 ip length > 100 numgen inc mod 2147483647 == 1 counter drop comment "second_data_packet"
 }
}
''')
 tap,path=capture(stack,root,'client','c0','tcp port 18080','sack')
 client_code=r'''
import socket,json,time,hashlib
s=socket.socket();s.settimeout(6);s.setsockopt(socket.IPPROTO_TCP,socket.TCP_NODELAY,1)
s.bind(('192.0.2.1',40500));s.connect(('192.0.2.2',18080))
for i in range(12):s.sendall(bytes([i])*256);time.sleep(.003)
with s.makefile('rb') as stream:result=json.loads(stream.readline())
expected=b''.join(bytes([i])*256 for i in range(12))
assert result=={'bytes':len(expected),'sha256':hashlib.sha256(expected).hexdigest()}
print(json.dumps(result));s.close()
'''
 result=json.loads(ns('client','python3','-c',client_code).stdout);assert json.loads(line(service))==result
 dropped=counters('target');assert dropped['second_data_packet']==1,dropped
 observed=packets(tap,path);syn=next(p for p in observed if p['flags']&0x12==0x02);synack=next(p for p in observed if p['flags']&0x12==0x12)
 assert syn['options'].get('sack_ok') and synack['options'].get('sack_ok')
 reports=[p for p in observed if p['sport']==18080 and p['options'].get('sack')]
 assert reports,observed
 data_start=(syn['seq']+1)%2**32
 def relative(sequence):return (sequence-data_start)%2**32
 hole=(data_start+256)%2**32
 assert any(relative(p['ack'])==256 and any(256<relative(left)<relative(right)<=3072 for left,right in p['options']['sack']) for p in reports),reports
 retransmissions=[p for p in observed if p['sport']==40500 and p['length'] and p['seq']==hole]
 assert len(retransmissions)>=2,retransmissions
 print(json.dumps({'sack_negotiated_both_directions':True,'missing_byte_sequence':hole,'sack_reports':reports,
 'original_and_retransmitted_hole':retransmissions,'drop_counter':dropped,'verified_delivery':result}))
"""
name='xbcatm-tcpflow-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

Both endpoints enable SACK. The client enables TCP_NODELAY and disables automatic corking in its own namespace, then sends twelve distinguishable chunks of 256 bytes with short pauses. The target input chain uses an incrementing rule expression to drop only the second matching data packet. The very large modulus makes that a single controlled loss within this short transfer; the counter must report exactly one drop.

The capture confirms SACK-Permitted in both SYN messages. After the first chunk arrives, the missing second chunk creates a hole. The receiver’s cumulative acknowledgment stays at the first byte of that hole, while a SACK block describes a later received range. The script checks the relationship using the actual initial sequence number rather than assuming a fixed packet sequence across runs.

The missing range appears in the capture first as the original transmitted segment and again as a retransmission with the same sequence number. In this run, it was a 256-byte segment. The server eventually receives all 3072 bytes in their expected order and checks their content. It returns a SHA-256 digest, which the client independently checks against the intended payload.

Those observations establish a real gap, informative acknowledgment, retransmission, and successful delivery. They do not benchmark SACK against a disabled configuration or establish a universal reduction in retransmission count. The capture also does not, by itself, identify every detail of Linux’s loss-detection decision. Negotiating SACK and choosing when to retransmit are related but distinct parts of the transport implementation.

Interpret feedback without promising perfect retransmission decisions

RFC 2018 makes SACK advisory: a receiver may later discard data it previously reported. The sender therefore cannot treat SACK as a permanent application-level receipt or discard all retransmission responsibility merely because a range appeared in an option. The cumulative acknowledgment remains central to determining which bytes have been accepted in order by TCP.

Reordering can also produce gaps and selective reports without permanent packet loss. Correlate sequence ranges, retransmission timing, cumulative progress, and capture location before classifying every SACK as a damaged network. Here, the known input filter and its exact drop counter provide evidence of deliberate loss that would be absent from an unexplained production trace.

For an incident, retain handshake options and enough of the data exchange to reconstruct the relevant ranges. Compare application delivery and transport recovery separately. Correct reassembly may hide a transient network loss from application correctness while still affecting latency or throughput. Use representative workload measurements for those performance questions; this small fixture verifies the protocol information and resulting byte delivery.

Sources

RFC 2018 TCP Selective Acknowledgment, received-byte ranges, direction of permission and the advisory nature of SACK, checked 2026-09-12.
RFC 9293 TCP, cumulative acknowledgments, sequence space and reliable byte delivery, checked 2026-09-12.
Linux tcp(7), Linux TCP SACK configuration and socket diagnostics, checked 2026-09-12.

TLS session resumption saves a round trip

Sun, 08 Aug 2027 00:00:00 +0000

Saving a round trip depends on the protocol and early-data mode. In TLS 1.3, an ordinary full handshake and a resumed handshake without early data normally both take one TLS round trip. Resumption can still reduce certificate transmission and authentication work. The title is not a promise that enabling tickets makes every HTTPS request start one RTT sooner.

New connection, fresh traffic keys

TLS resumption establishes a new connection using a PSK derived from an earlier handshake. A NewSessionTicket supplies an opaque identity associated with that state; it is not permission to reuse old traffic keys unchanged. The new handshake derives traffic keys for its own context. Servers may protect ticket state cryptographically or keep state behind an opaque identifier; applications should not parse tickets as portable credentials.

When the server accepts PSK authentication, it omits its Certificate and CertificateVerify messages. Resumption can include a fresh ephemeral Diffie-Hellman exchange for forward secrecy. TLS 1.3 also permits PSK-only operation, which lacks that protection. Do not infer either the selected mode or a latency saving from the existence of a ticket.

Optional 0-RTT lets eligible application data accompany ClientHello. That is separate from ordinary resumption. Early data is not forward secret and can be replayed across connections; even completing one handshake does not undo a replay elsewhere. HTTP deployments must decide which operations tolerate replay and handle rejection appropriately, including 425 Too Early where specified.

When it breaks

A client may never cache tickets, may close before processing a post-handshake ticket, or may lose cached state between requests. A server may decline an expired or unrecognized ticket after a restart or key rotation. A full handshake after an attempted resume is not automatically a TLS outage; measure the resume rate separately from connection success.

HTTP connection reuse can hide the distinction entirely. Repeating requests on one established socket performs no new handshake. To test resumption, deliberately create a new TCP connection and inspect both peers’ negotiated state. Request duration alone mixes scheduling, transport setup, server work, and cache effects.

Ticket and cache state are sensitive. Limit their scope and lifetime, coordinate server key rotation, and revisit cached authentication when policy changes. In Go, VerifyPeerCertificate is not called on resumed connections; VerifyConnection runs on all connections. Custom checks that must apply to resumptions belong on the latter path. This does not justify disabling normal certificate verification.

Check it quickly

Save this as resumption_probe.go and run go run resumption_probe.go with Go 1.23 or newer. The local HTTPS fixture and client both require TLS 1.3. Its certificate is trusted only inside the fixture. HTTP/1.1 keep-alive reuse is disabled, while two separate transports share the same four-entry TLS session cache. A third transport deliberately has no cache.

package main

import (
 "context"
 "crypto/tls"
 "fmt"
 "io"
 "net"
 "net/http"
 "net/http/httptest"
 "runtime"
 "strconv"
 "sync/atomic"
 "time"
)

type connectionKey struct{}

func check(ok bool, message string) {
 if !ok {
 panic(message)
 }
}

func main() {
 var serial, verified atomic.Int64
 server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 id := r.Context().Value(connectionKey{}).(int64)
 w.Header().Set("X-Connection-ID", strconv.FormatInt(id, 10))
 w.Header().Set("X-Resumed", strconv.FormatBool(r.TLS.DidResume))
 fmt.Fprintln(w, "ok")
 }))
 server.Config.ConnContext = func(ctx context.Context, connection net.Conn) context.Context {
 return context.WithValue(ctx, connectionKey{}, serial.Add(1))
 }
 server.TLS = &tls.Config{MinVersion: tls.VersionTLS13, MaxVersion: tls.VersionTLS13}
 server.StartTLS()
 defer server.Close()

 base := server.Client().Transport.(*http.Transport).TLSClientConfig.Clone()
 base.MinVersion, base.MaxVersion = tls.VersionTLS13, tls.VersionTLS13
 check(!base.InsecureSkipVerify, "certificate verification must stay enabled")
 base.VerifyConnection = func(state tls.ConnectionState) error {
 if len(state.VerifiedChains) == 0 {
 return fmt.Errorf("verified certificate chain required")
 }
 verified.Add(1)
 return nil
 }
 request := func(cache tls.ClientSessionCache) (int64, bool) {
 config := base.Clone()
 config.ClientSessionCache = cache
 transport := &http.Transport{
 TLSClientConfig: config,
 DisableKeepAlives: true,
 }
 defer transport.CloseIdleConnections()
 client := &http.Client{Transport: transport, Timeout: 2 * time.Second}
 response, err := client.Get(server.URL)
 check(err == nil, fmt.Sprint(err))
 defer response.Body.Close()
 body, err := io.ReadAll(response.Body)
 check(err == nil, fmt.Sprint(err))
 check(response.StatusCode == 200 && string(body) == "ok\n", "unexpected HTTP reply")
 check(response.ProtoMajor == 1 && response.ProtoMinor == 1, "HTTP/1.1 required")
 check(response.TLS.HandshakeComplete && response.TLS.Version == tls.VersionTLS13, "TLS 1.3 required")
 id, err := strconv.ParseInt(response.Header.Get("X-Connection-ID"), 10, 64)
 check(err == nil, fmt.Sprint(err))
 resumed := response.TLS.DidResume
 check(response.Header.Get("X-Resumed") == strconv.FormatBool(resumed), "peers disagree")
 return id, resumed
 }

 cache := tls.NewLRUClientSessionCache(4)
 firstID, first := request(cache)
 secondID, second := request(cache)
 thirdID, third := request(nil)
 check(firstID == 1 && secondID == 2 && thirdID == 3, "three new TCP connections required")
 check(!first && second && !third, "expected full, resumed, full")
 check(verified.Load() == 3, "VerifyConnection must run for all three handshakes")
 fmt.Printf("new TCP connections: %d %d %d; DidResume: %t %t %t\n",
 firstID, secondID, thirdID, first, second, third)
 fmt.Println("both peers agree; VerifyConnection ran three times; no early data was sent")
 fmt.Println(runtime.Version())
}

Reading each response to completion lets this fixture’s post-handshake ticket be processed before the next connection. Each accepted TCP connection gets an independent server-side number. The expected sequence is connections 1, 2, 3 with DidResume false, true, false. The response also carries the server’s own resume flag, which must agree with the client’s value. VerifyConnection runs three times, including the resumed handshake, while the standard certificate checks remain enabled.

All assertions passed on macOS with Go 1.27.0. The resumed connection’s verified-chain state does not mean a new certificate message was transmitted; it is associated with the authenticated session being resumed. Session state stays in the in-memory cache; no ticket, private key, or traffic secret is printed or written to disk.

This experiment proves acceptance of a cached session on a new transport and full negotiation without the cache. It does not send early data, count packets, benchmark handshake CPU, or validate ticket sharing across a load-balanced fleet. Test those separately before predicting a production speedup. A faster local second request alone would prove none of them.

Sources

RFC 8446: TLS 1.3, sections 2.2, 2.3, 4.6.1, and 8: PSKs, tickets, early data, and replay, checked 2026-09-09.
RFC 8470: Using Early Data in HTTP, replay-sensitive processing and 425 responses, checked 2026-09-09.
Go crypto/tls, ClientSessionCache, DidResume, and verification callbacks, checked 2026-09-09.
Go net/http/httptest, isolated loopback HTTPS fixtures, checked 2026-09-09.

TLS handshakes add latency you can shrink

Mon, 28 Jun 2027 00:00:00 +0000

HTTPS latency combines several stages, and a warm request may avoid some without making cryptography faster. DNS, connection acquisition, TCP setup, TLS authentication, server work, and response transfer have different clocks. First determine which stages actually ran. Request duration alone cannot distinguish an existing connection from a new connection that resumed a TLS session.

Messages travel in flights

A TLS handshake contains several messages, but each message does not require its own round trip. Messages can travel together in a flight. A usual TLS 1.3 full handshake needs one TLS round trip before normal client application data; HelloRetryRequest adds another. TCP establishment and the later application request/response remain separate costs.

Ordinary TLS 1.3 resumption without early data also normally takes one TLS round trip. It can reduce certificate transmission and authentication work, but may still perform fresh ephemeral key exchange. Resumption derives new traffic keys rather than reusing old encryption keys unchanged. Optional 0-RTT is a separate facility with replay-related security limits, not a synonym for a fast repeat request.

HTTP connection reuse avoids establishing another connection entirely, so that request performs no new TLS handshake. HTTP/2 multiplexing can similarly share one established connection among streams. In contrast, closing an idle connection and opening another one still requires transport setup, even if a cached ticket lets TLS resume.

When it breaks

Creating a new transport for every normal request discards the connection pool. Leaving response bodies unread or unclosed can also prevent reuse. Idle timeouts, endpoint changes, or concurrent demand can create additional connections. Classify requests by observed reuse and handshake events before attributing a latency change to certificate algorithms.

A full handshake after attempted resumption is not necessarily an outage. Tickets can expire or be declined, and a client may lack usable cached state. Preserve normal certificate verification and measure the resulting behavior. Reducing authentication checks to win a benchmark would change the security contract, not simply improve the same operation.

Timing names need definitions. Time to first response byte includes more than TLS, while total request time also includes consuming the response. Trace hooks can run on different goroutines, so collectors need synchronization. Compare representative distributions and connection categories; a few loopback microsecond values cannot predict an Internet speedup.

Check it quickly

Save this as latency_probe.go and run go run latency_probe.go with Go 1.23 or newer. The fixture uses verified TLS 1.3 and HTTP/1.1. A server-side counter identifies accepted TCP connections. The first transport retains both an HTTP connection pool and a TLS session cache; a separate transport deliberately has no session cache.

package main

import (
 "context"
 "crypto/tls"
 "fmt"
 "io"
 "net"
 "net/http"
 "net/http/httptest"
 "net/http/httptrace"
 "runtime"
 "strconv"
 "sync"
 "sync/atomic"
 "time"
)

type connectionKey struct{}

func must(err error) {
 if err != nil {
 panic(err)
 }
}

func check(ok bool, message string) {
 if !ok {
 panic(message)
 }
}

func main() {
 var connections atomic.Int64
 server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 w.Header().Set("X-Connection-ID", fmt.Sprint(r.Context().Value(connectionKey{})))
 w.Header().Set("X-Resumed", strconv.FormatBool(r.TLS.DidResume))
 fmt.Fprint(w, "ok")
 }))
 server.Config.ConnContext = func(ctx context.Context, _ net.Conn) context.Context {
 return context.WithValue(ctx, connectionKey{}, connections.Add(1))
 }
 server.TLS = &tls.Config{MinVersion: tls.VersionTLS13, MaxVersion: tls.VersionTLS13}
 server.StartTLS()
 defer server.Close()
 base := server.Client().Transport.(*http.Transport).TLSClientConfig.Clone()
 base.MinVersion, base.MaxVersion = tls.VersionTLS13, tls.VersionTLS13
 check(!base.InsecureSkipVerify, "certificate verification must stay enabled")
 cached := base.Clone()
 cached.ClientSessionCache = tls.NewLRUClientSessionCache(4)
 transport := &http.Transport{TLSClientConfig: cached}
 defer transport.CloseIdleConnections()
 fresh := &http.Transport{TLSClientConfig: base.Clone()}
 defer fresh.CloseIdleConnections()
 probe := func(label string, transport *http.Transport, wantID int64, wantReuse, wantResume bool) {
 started := time.Now()
 var mu sync.Mutex
 var handshakeStarted time.Time
 var handshakeError error
 var observation struct {
 got, starts, done, first int
 reused bool
 tlsTime, ttfb time.Duration
 }
 trace := &httptrace.ClientTrace{
 GotConn: func(info httptrace.GotConnInfo) {
 mu.Lock()
 defer mu.Unlock()
 observation.got++
 observation.reused = info.Reused
 },
 TLSHandshakeStart: func() {
 mu.Lock()
 defer mu.Unlock()
 handshakeStarted = time.Now()
 observation.starts++
 },
 TLSHandshakeDone: func(_ tls.ConnectionState, err error) {
 mu.Lock()
 defer mu.Unlock()
 observation.done++
 observation.tlsTime = time.Since(handshakeStarted)
 handshakeError = err
 },
 GotFirstResponseByte: func() {
 mu.Lock()
 defer mu.Unlock()
 observation.first++
 observation.ttfb = time.Since(started)
 },
 }
 request, err := http.NewRequest(http.MethodGet, server.URL, nil)
 must(err)
 request = request.WithContext(httptrace.WithClientTrace(request.Context(), trace))
 client := &http.Client{Transport: transport, Timeout: 2 * time.Second}
 response, err := client.Do(request)
 must(err)
 body, err := io.ReadAll(response.Body)
 closeError := response.Body.Close()
 must(err)
 must(closeError)
 total := time.Since(started)
 check(response.StatusCode == 200 && string(body) == "ok", "wrong response")
 check(response.Proto == "HTTP/1.1", "fixture requires HTTP/1.1")
 check(response.TLS.Version == tls.VersionTLS13 && len(response.TLS.VerifiedChains) > 0, "verified TLS 1.3 required")
 id, err := strconv.ParseInt(response.Header.Get("X-Connection-ID"), 10, 64)
 must(err)
 check(id == wantID && response.TLS.DidResume == wantResume, "wrong connection or resumption")
 check(response.Header.Get("X-Resumed") == strconv.FormatBool(wantResume), "peers disagree on resumption")
 mu.Lock()
 observed, tlsError := observation, handshakeError
 mu.Unlock()
 must(tlsError)
 wantHandshakes := 1
 if wantReuse {
 wantHandshakes = 0
 }
 check(observed.got == 1 && observed.first == 1 && observed.reused == wantReuse, "wrong request trace")
 check(observed.starts == wantHandshakes && observed.done == wantHandshakes, "wrong handshake count")
 check(total >= observed.ttfb && observed.ttfb > 0, "invalid response timing")
 fmt.Printf("%s: conn=%d reused=%t handshakes=%d connection_resumed=%t tls_us=%d ttfb_us=%d total_us=%d\n",
 label, id, observed.reused, observed.done, response.TLS.DidResume,
 observed.tlsTime.Microseconds(), observed.ttfb.Microseconds(), total.Microseconds())
 }
 probe("cold", transport, 1, false, false)
 probe("reuse", transport, 1, true, false)
 transport.CloseIdleConnections()
 probe("resume", transport, 2, false, true)
 probe("no cache", fresh, 3, false, false)
 check(connections.Load() == 3, "four requests must use three TCP connections")
 fmt.Println("timings are observations, not a speedup or wire-RTT assertion")
 fmt.Println(runtime.Version())
}

The invariant results are:

Request	TCP ID	Reused	New handshakes	DidResume
cold	1	false	1	false
reuse	1	true	0	false
resume	2	false	1	true
no cache	3	false	1	false

The second request keeps connection one. Closing idle connections before the third forces another TCP connection without discarding the TLS cache, so it resumes. The fourth request uses a transport with no cached session and performs a full handshake. Both peers’ resumption flags agree, and four HTTP requests use exactly three accepted TCP connections.

DidResume describes how that connection was established, not whether the current HTTP request triggered resumption. Reusing a connection originally established by resumption can still report true without another handshake. Interpret this connection-level flag alongside the request’s reuse and handshake events, rather than counting every true response as a new resumed connection.

TLS timing spans TLSHandshakeStart through TLSHandshakeDone. TTFB starts before the request and ends when the first response-header byte becomes available; total ends after reading and closing the body. The reused row’s zero TLS duration means no new handshake event was observed. It does not mean an extraordinarily fast handshake or TLS 0-RTT.

All assertions passed on macOS with Go 1.27.0. Bodies are consumed and closed, trace state is mutex-protected, and each request has a two-second timeout. Session state stays in memory; no ticket, private key, or traffic secret is printed. On resumed connections, verified-chain state belongs to the authenticated session, not evidence of a newly transmitted certificate.

The timing numbers intentionally have no speedup assertion and will vary. This local experiment does not count packets or measure wire RTT, DNS, HTTP/2 or HTTP/3, early data, fleet ticket rotation, handshake CPU in isolation, or production latency distributions. Use the trace categories to structure those later measurements without treating this fixture as their result.

Sources

RFC 8446: TLS 1.3, handshake flights, resumption, and early-data limits, checked 2026-09-09.
Go net/http/httptrace, request hooks, handshake events, timing boundaries, and concurrency, checked 2026-09-09.
Go net/http, transport reuse and response-body lifecycle, checked 2026-09-09.
Go crypto/tls, session caching and negotiated resumption state, checked 2026-09-09.
Go net/http/httptest, isolated TLS servers and trusted test clients, checked 2026-09-09.

HTTP keep-alive reuses one TCP connection

Fri, 28 May 2027 00:00:00 +0000

HTTP persistence lets several request and response exchanges use one connection. That can avoid repeated TCP setup and, for HTTPS, repeated TLS negotiation. The benefit depends on latency, request size, and whether another usable connection already exists. Persistence does not make application work disappear, nor does it promise that a socket will remain open indefinitely.

Reuse requires complete messages

HTTP/1.1 connections are persistent by default unless a participant signals Connection: close or closes the transport. Adding Connection: keep-alive is normally unnecessary. A client connection pool can retain an idle connection for another compatible request, subject to the client’s origin, proxy, certificate, and pool policies. Two separate command-line client processes generally do not share such a pool.

Both sides must know where each message ends. A response body can be delimited by an accurate Content-Length, chunked transfer coding, or the rules for responses without bodies. A response whose end is defined only by connection closure cannot leave that connection reusable. The client must consume the response body before reuse; the server must consume the request body or close. Otherwise leftover bytes could be mistaken for the following message.

HTTP persistence is also distinct from TCP keepalive. TCP keepalive probes a sufficiently idle transport to help detect an unreachable peer. It neither delimits HTTP messages nor overrides an application server’s idle timeout. Confusing the two can leave a client repeatedly trying connections that a proxy has already retired.

Sequential reuse also differs from pipelining: a client can wait for each response before sending the next request and still reuse the connection. HTTP/1.1 pipelining sends requests ahead but requires responses in order. Persistence alone therefore does not provide the independent response streams associated with HTTP/2 multiplexing.

Each hop owns its timeout

An idle timeout measures inactivity according to a particular implementation; a request deadline bounds an individual operation. Treat them as separate controls. A reverse proxy has a client-side connection and an upstream connection, with potentially different protocols and timeout policies. Reuse observed between a browser and the edge proves nothing about reuse between that edge and the origin.

An idle pool should retire connections with the server’s known limits in mind, but timing races still exist: the server may close just as the client sends again. Monitor connection creation, successful reuse, resets, and pool occupancy alongside latency. Longer retention saves handshakes while reserving descriptors and memory. There is no universal timeout that follows from a site being busy.

When it breaks

A response with an incorrect length can hang the reader or corrupt the boundary of the next exchange. Diagnose framing before increasing timeouts. A client that abandons streamed responses without consuming or closing them can also exhaust its pool; queued requests then wait even though the server has spare capacity.

A stale connection may fail after the request reached the server. Reconnecting and replaying a payment automatically can duplicate it. Retry according to method semantics and the application’s idempotency contract, since a transport error alone does not establish whether work committed. HTTP/2 multiplexes streams but forbids HTTP/1.x connection-specific headers such as Connection and Keep-Alive; copying these headers into an HTTP/2 configuration is not a reuse optimization.

Check it quickly

Run this complete Python 3 experiment. It binds an available loopback port, sends four sequential requests, and shuts down its test server. No external service is required; the HTTP server is a local fixture, not a deployment example.

from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread

class Handler(BaseHTTPRequestHandler):
 protocol_version = "HTTP/1.1"

 def do_GET(self):
 self.server.connections.append(self.connection)
 self.send_response(200)
 self.send_header("Content-Length", "2")
 if self.path == "/close":
 self.send_header("Connection", "close")
 self.end_headers()
 self.wfile.write(b"ok")

 def log_message(self, *args):
 pass

with ThreadingHTTPServer(("127.0.0.1", 0), Handler) as server:
 server.connections = []
 worker = Thread(target=server.serve_forever, daemon=True)
 worker.start()
 client = HTTPConnection(*server.server_address, timeout=3)
 try:
 for path in ("/one", "/two", "/close", "/four"):
 client.request("GET", path)
 response = client.getresponse()
 assert response.status == 200
 assert response.read() == b"ok"
 sockets = server.connections
 print("reused:", sockets[0] is sockets[1] is sockets[2])
 print("new after close:", sockets[2] is not sockets[3])
 finally:
 client.close()
 server.shutdown()
 worker.join()

Expect reused: True and new after close: True. The complete body reads permit the first three exchanges to share a socket; the explicit close makes the fourth reconnect. These observations establish local HTTP/1.1 reuse, not TLS savings, browser pool behavior, or production proxy settings. Inspect connection identifiers on each real hop before extending that conclusion to a deployed service.

Sources

RFC 9112: HTTP/1.1 connection management, framing, persistence, retries, and closure, checked 2026-09-06.
MDN: Connection management in HTTP/1.x, persistent connections and their costs, checked 2026-09-06.
RFC 9113: HTTP/2 connection-specific fields, fields prohibited in HTTP/2 messages, checked 2026-09-06.

EXPLAIN plans reveal the route of a query

Fri, 21 May 2027 00:00:00 +0000

A slow query is a route problem before it is a tuning problem. EXPLAIN shows the planned route node by node, so the fix targets the step that actually costs time rather than a guess.

Read the plan as a route map

EXPLAIN prints a tree, and the root node is the answer while the leaves do the real work. Each node names an operation, a row estimate and an estimated cost, so instead of guessing you can look for the outlier: a scan whose row estimate dwarfs the number it eventually returns, or an estimate that is wildly off what the query actually touches. The arrow of blame is worth reading the way a map is: follow the node that accounts for most of the work, not the first one listed in the output. Comparing EXPLAIN alongside EXPLAIN ANALYZE shows the gap between what the planner predicted and what actually ran, which is where slow queries usually hide.

Statistics steer the chosen route

An index is a candidate, not a guarantee: the planner chooses the route from table statistics such as row counts, nulls and column value distribution, and the same query can change routes after an auto-vacuum, a bulk load or a changed cache ratio. That means a hot query deserves a re-check when the data profile shifts, not just when it is new. When statistics are stale, ANALYZE (or the database’s auto-analysis) can change the plan more than most manual rewrites. Treat the plan as evidence to re-verify against real EXPLAIN ANALYZE numbers rather than a snapshot to memorise.

Confirm the route moved

Make the plan part of the change loop: capture a baseline EXPLAIN ANALYZE, apply an index or a rewritten query, and confirm the route actually shifted before you declare victory. If the cost node you were targeting still shows the same large step after a rewrite, the change is cosmetic rather than structural, no matter how reasonable it sounds. IQ and experience are a poor substitute for comparing before and after plans; the same holds for measured timing and rows fetched. When the new plan wins, keep the before and after as evidence in the merge description so the gain is attributable and repeatable.

When it breaks

Stale statistics pick the wrong road. After a bulk load or a long stretch without ANALYZE, the planner may still believe a table holds thousands of rows when it holds millions, and it keeps favoring an index or nested-loop path that is only cheap at small scale. EXPLAIN ANALYZE gives the tell: estimated rows differ from actual rows by orders of magnitude. Re-run ANALYZE and compare those two numbers before adding any index.

A prepared statement can lock in the wrong plan for the wrong value. A plan cached around a highly selective first parameter looks fast for that case and slow for a value matching half the table, yet the literal tested with EXPLAIN is rarely the value production binds. Probe both extremes and check whether a custom or generic plan was chosen.

Implicit conversion silently retires the index. Comparing a varchar column to a bare number, mixing collations, or wrapping the column in LOWER() or a date truncation forces a scan plus filter even though the index exists. The symptom is a Seq Scan whose rows removed by filter dwarf the rows returned. Fix the type at the call site, or add a matching expression index.

Cost alone is the last trap. Cost is a unitless planning estimate, not time: the cheapest plan can still pile up shared-buffer hits or spill sorts to temp files. Confirm with EXPLAIN (ANALYZE, BUFFERS), reading actual time, buffers, and rows removed by filter next to the cost before calling a route healthy.

Check it quickly

Save the following as plan_route_check.py in a scratch directory and run python3 plan_route_check.py. It uses only the standard library to model two candidate routes — a sequential scan and an index scan — under a stale row estimate, then re-scores the chosen route at the actual row count. The script prints estimated and true costs, the estimate error, and a stale-statistics warning, and asserts that the warning fires:

#!/usr/bin/env python3
"""Toy planner model: estimated cost picks a route, actual rows judge it.

Stdlib only. It simulates stale statistics: the planner believes 500 rows
match, while 600,000 really do, and checks that the estimate-error alarm fires.
"""

from __future__ import annotations


def seq_scan_cost(pages: float, rows: float) -> float:
 return pages * 1.0 + rows * 0.01


def index_scan_cost(selectivity_est: float, pages: float, rows: float) -> float:
 heap_pages = selectivity_est * pages
 return 4.0 + heap_pages * 4.0 + selectivity_est * rows * 0.01


def main() -> None:
 pages, rows = 8_000.0, 1_000_000.0
 stale_selectivity = 0.0005 # planner belief: 500 rows
 true_selectivity = 0.60 # reality: 600,000 rows
 est_rows = stale_selectivity * rows
 actual_rows = true_selectivity * rows

 seq = seq_scan_cost(pages, rows)
 idx_est = index_scan_cost(stale_selectivity, pages, rows)
 idx_true = index_scan_cost(true_selectivity, pages, rows)
 chosen = "index scan" if idx_est < seq else "seq scan"
 print(f"estimated rows: {est_rows:.0f}; actual rows: {actual_rows:.0f}")
 print(f"estimated cost: seq={seq:.1f} index={idx_est:.1f} -> planner picks {chosen}")
 print(f"true cost at actual selectivity: seq={seq:.1f} index={idx_true:.1f}")

 error = abs(actual_rows - est_rows) / actual_rows
 print(f"estimate error: {error:.1%}")
 warned = False
 if error > 0.50:
 warned = True
 print("WARNING: row estimate off by >50% - statistics are stale, re-run ANALYZE.")
 if chosen == "index scan" and idx_true > seq:
 warned = True
 print("WARNING: chosen route loses at actual selectivity - check EXPLAIN ANALYZE.")
 assert warned, "expected a stale-estimate warning for this fixture"
 assert est_rows < actual_rows / 10, "fixture must keep the estimate an order of magnitude low"
 print("PASS: planner picked the stale route and the estimate-error alarm fired")


if __name__ == "__main__":
 main()

Expect the planner to pick the index scan on the stale estimate, an estimate error near 100%, and a final PASS line. The numbers are a toy model of one failure mode, not a real optimizer: page costs are constants and only two paths compete. They train the habit that matters — estimate first, then confront the estimate with actual rows — before reading a production plan. The fixture touches no database and needs no packages; re-check any plan change against real EXPLAIN (ANALYZE, BUFFERS) output before shipping it.

Sources

PostgreSQL: EXPLAIN, generating the plan, checked 2027-05-19.
PostgreSQL: using EXPLAIN, interpreting the output, checked 2027-05-19.
PostgreSQL: planner/optimizer, how routes are chosen, checked 2027-05-19.
MySQL: EXPLAIN statement, plan format and output columns, checked 2027-05-19.

Load shedding protects the system

Sun, 16 May 2027 00:00:00 +0000

Load shedding deliberately refuses some work when accepting everything would threaten useful service. The aim is to preserve capacity for work that can still complete, not to promise that every accepted request will be fast or important. Rejection is itself work, so the admission point and its cost matter as much as the threshold.

Protect useful capacity

An overloaded queue consumes memory and adds waiting time. Clients can abandon requests before workers start them, leaving the server to spend resources on answers nobody needs. Longer queues can absorb some bursts, but cannot create processing capacity under sustained excess demand.

Google SRE describes queue limits, resource-pressure signals, selective rejection, and reduced-quality responses. Shedding and degradation are related but different: refusing a request is not the same as completing a cheaper version of it. The latter needs an explicit product contract about the result that remains useful.

Protection can also target different components. Envoy’s overload manager protects the proxy’s own resources, while its circuit-breaking mechanisms primarily protect upstream services. A dependency breaker does not automatically prevent local memory, connection, or file-descriptor exhaustion.

Choose a justified policy

Priority-aware shedding can protect critical business paths, but priority is not determined by HTTP method or retryability alone. A read can be essential, while a write can be expendable telemetry. Use trusted application classification rather than letting every caller declare its own request critical.

Uniform admission limits or proportion-based rejection can also be legitimate policies. They are not inherently incorrect because they lack business priority. State the fairness and capacity tradeoffs, including whether lower-priority work can starve and whether different tenants share the same limit.

Count, bytes, execution time, and CPU cost are different budgets. A cap on three tasks does not bound memory when each task can retain an arbitrarily large payload. Select signals that correspond to the resource being protected and measure their behavior under realistic request mixes.

Bound admission, not only workers

A thread-pool worker limit bounds simultaneous execution, not necessarily accepted queued work. A semaphore acquired only inside a worker leaves the queue in front of it unbounded. Make a bounded admission decision before submitting expensive work, and return that capacity when its lifecycle actually ends.

Even similarly named proxy limits need precise interpretation. NGINX limit_conn counts requests under its documented connection rules; with HTTP/2 and HTTP/3, each concurrent request counts separately. Do not assume a configured connection cap always means a count of TCP sockets.

The fixture below bounds running plus queued tasks. Refused submissions create no future. A completion callback returns permits after success, failure, or cancellation before execution; a rejected submission to a closed executor also returns its unused permit.

When it breaks

Rejecting after parsing huge bodies, opening database connections, or performing a business write may be too late to protect the intended resource. Expensive logging or response generation can itself overload the rejection path. Observe rejection cost alongside accepted throughput, queue age, active work, and resource pressure.

HTTP 503 can signal temporary service unavailability, while 429 concerns excessive request rates under a caller policy. Retry-After is a wait signal, not proof that replaying a write is safe or that capacity will exist later. Immediate retries can restore the very load that was shed.

Do not reclaim a running task’s permit merely because its caller stopped waiting. A future-result timeout does not terminate the worker. Premature release allows replacement work to overlap the supposedly removed work and breaks the capacity bound.

Check it quickly

Save the block as shedding_check.py and run python3 shedding_check.py with Python 3.10 or newer. It ran with Python 3.14.4 using only the standard library. Real worker threads wait on test coordination events; no external service is called, and no production load is generated. The fixture compares identical twelve-task bursts.

from concurrent.futures import ThreadPoolExecutor
from threading import Barrier, BoundedSemaphore, Event, Lock


def exercise(gated):
 slots = BoundedSemaphore(3) # Bounds queued plus running work, not just threads.
 started, release, lock = Barrier(3), Event(), Lock()
 running, began = set(), []

 def held(job):
 with lock:
 running.add(job)
 began.append(job)
 try:
 if job < 2:
 started.wait(timeout=5)
 assert release.wait(timeout=5), "test coordinator did not release workers"
 return job
 finally:
 with lock:
 running.remove(job)

 with ThreadPoolExecutor(max_workers=2) as pool:
 def submit(function, *args):
 if not gated:
 return pool.submit(function, *args)
 if not slots.acquire(blocking=False):
 return None # Explicit local admission refusal; no future was enqueued.
 try:
 future = pool.submit(function, *args)
 except RuntimeError:
 slots.release()
 raise
 # Completion, failure, and cancellation before execution all return the slot.
 future.add_done_callback(lambda _: slots.release())
 return future

 futures = [submit(held, job) for job in range(12)]
 accepted = [future for future in futures if future is not None]
 replacement = None
 try:
 started.wait(timeout=5)
 with lock:
 assert running == {0, 1} and set(began) == {0, 1}
 assert all(not future.done() for future in accepted)
 if gated:
 assert len(accepted) == 3 and futures[3:] == [None] * 9
 assert futures[2].cancel() # Both workers are held, so job 2 has not started.
 replacement = submit(held, 12)
 assert replacement is not None
 assert submit(held, 13) is None
 print("gated burst: 2 running + 1 queued, 9 initial refusals; cancellation returns a slot")
 else:
 assert len(accepted) == 12
 print("ungated burst: 2 workers accept 12 unfinished tasks; the other 10 still wait")
 finally:
 release.set()
 completed = [future.result(timeout=5) for future in accepted if not future.cancelled()]
 if replacement is not None:
 completed.append(replacement.result(timeout=5))
 assert sorted(completed) == ([0, 1, 12] if gated else list(range(12)))

 if gated:
 def fail():
 raise ValueError("injected task failure")

 failed = submit(fail)
 assert failed is not None
 try:
 failed.result(timeout=5)
 except ValueError as error:
 assert str(error) == "injected task failure"
 else:
 raise AssertionError("the failed task must preserve its failure")

 if gated:
 # The executor is now closed; a failed submission must not consume a slot.
 try:
 submit(lambda: None)
 except RuntimeError:
 pass
 else:
 raise AssertionError("submission to a closed executor must fail")
 acquired = [slots.acquire(blocking=False) for _ in range(3)]
 try:
 assert acquired == [True, True, True]
 assert not slots.acquire(blocking=False)
 finally:
 for success in acquired:
 if success:
 slots.release()
 print("completion, task failure, cancellation, and closed-executor submission leave all 3 slots reusable")


exercise(False)
exercise(True)

With two workers but no admission gate, twelve tasks remain unfinished: two running and ten waiting. With three permits, only three are accepted and nine are refused before submission. The admitted set contains two running tasks and one queued task, not three simultaneous workers.

Canceling the queued task returns a permit for one replacement, while another submission is refused. After coordination releases the workers, the expected tasks complete. An injected task failure and a submission after executor shutdown both preserve their errors. Public semaphore operations finally confirm that all three permits are reusable.

This tests admission accounting and task lifecycle, not HTTP responses, byte limits, priority scheduling, production latency, or an automatic resource-pressure controller. NGINX and Envoy are cited for their contracts, not claimed to be deployed or exercised here.

Sources

Google SRE: Handling overload, criticality, resource signals, rejection costs, and client behavior, checked 2026-09-08.
Google SRE: Addressing cascading failures, queue limits, load shedding, deadlines, and degradation, checked 2026-09-08.
Envoy overload manager, local resource protection versus upstream circuit breaking, checked 2026-09-08.
NGINX connection limiting, counted requests, HTTP/2–HTTP/3 semantics, and rejection status, checked 2026-09-08.
Python futures, executor submission, result timeouts, cancellation, and completion callbacks, checked 2026-09-08.
Python threading, nonblocking bounded-semaphore admission and coordination primitives, checked 2026-09-08.
RFC 9110, service-unavailability, Retry-After, and retry safety semantics, checked 2026-09-08.
RFC 6585, section 4’s separate definition of 429 and server-selected counting policy, checked 2026-09-08.

A materialized view stores a query's result for reuse

Mon, 26 Apr 2027 00:00:00 +0000

A materialized view stores the result of a defining query so later reads can access those stored rows. It moves computation from every read into refresh operations, which can benefit repeated reporting queries. The tradeoff is an explicit freshness boundary and another relation to maintain; neither instant reads nor automatic synchronization with the source tables is guaranteed.

Stored results have their own lifecycle

A normal view stores a query definition, while a materialized view also stores its result. PostgreSQL can index the materialized result and plan reads against it without rerunning the original joins and aggregates. Ordinary source-table inserts and updates do not automatically update that stored result. Applications must query the materialized view intentionally; creating one does not transparently rewrite every equivalent source query.

REFRESH MATERIALIZED VIEW reruns the defining query and replaces the stored contents. Standard PostgreSQL refresh is not automatic incremental maintenance: even when only a few source rows changed, the defining query still has to produce the new result. Account for refresh computation, temporary work, storage, and index maintenance when comparing the total cost with repeated direct queries.

The stored rows can still be large or poorly indexed, so materialization does not make arbitrary reads free. Choose the aggregation grain and indexes according to the report’s filters and projections. Also use an ORDER BY in the reading query when ordering matters; refresh does not promise to preserve the physical order from the defining query.

Concurrent refresh changes reader access

An ordinary refresh can block readers while replacing the materialization. CONCURRENTLY permits concurrent selects to continue accessing the existing contents during refresh, with the new contents becoming visible according to transaction and snapshot rules. It still takes locks, uses resources, and can wait for other operations. Only one refresh at a time may run against a given materialized view.

Concurrent refresh requires an already populated materialized view and at least one eligible unique index using only column names and covering all rows. A partial unique index or expression index does not satisfy that requirement. The unique index lets PostgreSQL identify result rows during refresh; it is not a declaration that the source tables themselves have become immutable or unique at the same grain.

WITH NO DATA leaves a materialized view unpopulated and unscannable. It does not mean an ordinary empty result that SELECT can read, and it does not preserve the old cached answer. Populate it with a normal refresh before requesting a concurrent one. CONCURRENTLY and WITH NO DATA cannot be combined into a shortcut around these conditions.

When it breaks

A refresh schedule alone does not bound staleness. A job can start late, take a long time, fail, or wait behind another refresh; the source snapshot also precedes completion of the work. Track successful refreshes and the source coverage that the report needs, rather than assuming that a configured interval proves current data. Failed jobs must not advance a displayed freshness marker as if new results were published.

Missing or unsuitable unique indexes can make CONCURRENTLY fail when deployment expected an uninterrupted read path. Test that path before relying on it. In PostgreSQL 18, refresh also requires the applicable MAINTAIN privilege, so the role executing the scheduled job must actually be able to refresh the relation. Successful manual execution as its owner does not establish the job role’s permissions.

Check it quickly

Save this as materialized.sql and run psql -X -v ON_ERROR_STOP=1 -f materialized.sql in a disposable PostgreSQL 18 database. The role needs permission to create the test schema. The script creates matview_demo inside a transaction and rolls it back; if that name already exists, choose an unused name rather than deleting an existing schema.

\set ON_ERROR_STOP on
BEGIN;
CREATE SCHEMA matview_demo;
SET LOCAL search_path=matview_demo,pg_catalog;
CREATE TABLE sales (region text NOT NULL,amount integer NOT NULL);
INSERT INTO sales VALUES ('east',10),('west',5);
CREATE MATERIALIZED VIEW totals AS SELECT region,sum(amount) AS total FROM sales GROUP BY region;
UPDATE sales SET amount=17 WHERE region='east';
SELECT * FROM totals ORDER BY region;
DO $$ BEGIN
 REFRESH MATERIALIZED VIEW CONCURRENTLY totals;
 RAISE EXCEPTION 'concurrent refresh accepted without unique index';
EXCEPTION WHEN object_not_in_prerequisite_state THEN
 RAISE NOTICE 'concurrent refresh requires an eligible unique index';
END $$;
CREATE UNIQUE INDEX totals_region ON totals (region);
REFRESH MATERIALIZED VIEW CONCURRENTLY totals;
SELECT * FROM totals ORDER BY region;
REFRESH MATERIALIZED VIEW totals WITH NO DATA;
DO $$ BEGIN
 PERFORM * FROM totals;
 RAISE EXCEPTION 'unpopulated view was readable';
EXCEPTION WHEN object_not_in_prerequisite_state THEN RAISE NOTICE 'unpopulated view cannot be read';
END $$;
DO $$ BEGIN
 REFRESH MATERIALIZED VIEW CONCURRENTLY totals;
 RAISE EXCEPTION 'concurrent refresh populated an unreadable view';
EXCEPTION WHEN feature_not_supported THEN RAISE NOTICE 'first population needs a normal refresh';
END $$;
REFRESH MATERIALIZED VIEW totals;
DO $$ BEGIN
 IF (SELECT total FROM totals WHERE region='east') IS DISTINCT FROM 17::bigint OR
 (SELECT count(*) FROM totals) <> 2 THEN RAISE EXCEPTION 'refresh result wrong'; END IF;
END $$;
ROLLBACK;

PostgreSQL 18.6 continued showing east total 10 after the source changed to 17. Concurrent refresh without an eligible unique index failed with the expected prerequisite error. After creating the index, refresh changed the stored east total to 17 while west remained five.

WITH NO DATA then made reading fail, and a concurrent refresh could not repopulate the unscannable view. A normal refresh restored readable results and satisfied the final assertions. This fixture verifies state transitions and prerequisites, not refresh throughput. For operational sizing, measure the actual report query and test reader behavior during a representative refresh, including the effect of transaction boundaries and failed jobs.

Sources

PostgreSQL materialized views, stored results, indexing, and refresh use cases, checked 2026-09-10.
PostgreSQL REFRESH MATERIALIZED VIEW, locking, concurrent prerequisites, privileges, and unpopulated state, checked 2026-09-10.

Request body limits set a boundary

Sun, 18 Apr 2027 00:00:00 +0000

A request-body limit bounds accepted input, not merely the number written in Content-Length. Enforce it while consuming the body, before committing application effects. Compressed input needs separate limits for encoded and decoded bytes. A 413 response is useful only if the expensive allocation or irreversible write has not already happened.

Define which bytes the budget counts

Content-Length describes the framed message body when that framing is used. An HTTP/1.1 chunked request can arrive without a known length. A large declared length permits early rejection, but a missing length does not authorize unlimited reading. Let the HTTP implementation handle message framing, then apply the application’s body budget to its reader.

The body reader excludes HTTP headers and chunk framing overhead. Its limit therefore is not a total network-byte quota. For Content-Encoding: gzip, those body bytes are compressed; the decoded representation can be much larger. Check the encoded stream before decompression and the decoded stream afterward. A small compressed file can exceed a legitimate decoded limit.

Set limits from an endpoint’s contract, including units and supported content codings. A total-body limit also differs from an individual multipart-file or field limit. Streaming avoids buffering everything at once, but does not automatically limit total storage, parser work, or how long a slow sender holds a request.

Validate before making effects visible

Go’s MaxBytesReader reports a MaxBytesError when a read exceeds the configured input limit. By contrast, a plain LimitReader stops with EOF at its boundary; without another check, an application might accept a truncated prefix as complete input. Reading at most the decoded limit plus one byte makes exact-limit acceptance distinguishable from oversize input.

For gzip, successfully reading a prefix does not establish checksum validity. Read through the end before accepting an in-budget representation; closing the decoder alone does not verify its checksum. Keep file publication, database changes, and other effects behind successful complete validation, or stage work with a defined abort path.

When it breaks

Checking only Content-Length misses unknown-length input. Applying the same number only before decompression misses expansion. Parsing first and rejecting afterward spends the resources the limit was meant to protect. Ignoring an error because some bytes were returned can turn a damaged gzip stream into accepted application data.

Size limits do not repair conflicting HTTP framing, authenticate callers, validate file types, or bound aggregate concurrent memory. They also do not stop a sender from dripping a small body forever. Use separate time and concurrency budgets; do not call a byte cap an idle timeout or claim that one rejected request proves resistance to load.

Check it quickly

Save this as body_probe.go and run go run body_probe.go with Go 1.23 or newer. The loopback fixture permits 64 encoded bytes and eight decoded bytes, with no content coding or one gzip coding. These deliberately tiny constants are test contracts, not production recommendations. A counter represents the application’s commit point; no database or uploaded file is written.

package main

import (
 "bytes"
 "compress/gzip"
 "errors"
 "fmt"
 "io"
 "net/http"
 "net/http/httptest"
 "runtime"
 "strings"
 "sync/atomic"
 "time"
)

const encodedLimit int64 = 64
const decodedLimit int64 = 8

func must(err error) {
 if err != nil {
 panic(err)
 }
}

func check(ok bool, message string) {
 if !ok {
 panic(message)
 }
}

func main() {
 var committed, handled atomic.Int64
 server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 handled.Add(1)
 defer r.Body.Close()
 w.Header().Set("X-Received-Length", fmt.Sprint(r.ContentLength))
 fail := func(err error) {
 var tooLarge *http.MaxBytesError
 if errors.As(err, &tooLarge) {
 http.Error(w, "encoded limit", http.StatusRequestEntityTooLarge)
 } else {
 http.Error(w, "invalid body", http.StatusBadRequest)
 }
 }
 if r.ContentLength > encodedLimit {
 http.Error(w, "encoded limit", http.StatusRequestEntityTooLarge)
 return
 }
 limited := http.MaxBytesReader(w, r.Body, encodedLimit)
 var reader io.Reader = limited
 coding := strings.TrimSpace(strings.ToLower(strings.Join(r.Header.Values("Content-Encoding"), ",")))
 switch coding {
 case "":
 case "gzip":
 decoded, err := gzip.NewReader(limited)
 if err != nil {
 fail(err)
 return
 }
 defer decoded.Close()
 reader = decoded
 default:
 http.Error(w, "unsupported coding", http.StatusUnsupportedMediaType)
 return
 }
 body, err := io.ReadAll(io.LimitReader(reader, decodedLimit+1))
 if err != nil {
 fail(err)
 return
 }
 if int64(len(body)) > decodedLimit {
 http.Error(w, "decoded limit", http.StatusRequestEntityTooLarge)
 return
 }
 committed.Add(1) // No side effect before the complete body passes validation.
 fmt.Fprint(w, "accepted")
 }))
 server.Config.ReadTimeout = 2 * time.Second
 server.Config.ReadHeaderTimeout = time.Second
 server.Start()
 defer server.Close()
 transport := &http.Transport{}
 defer transport.CloseIdleConnections()
 client := &http.Client{Transport: transport, Timeout: 3 * time.Second}
 probe := func(label string, payload []byte, coding string, chunked bool, want int) {
 request, err := http.NewRequest(http.MethodPost, server.URL, bytes.NewReader(payload))
 must(err)
 if chunked {
 request.ContentLength = -1
 request.TransferEncoding = []string{"chunked"}
 }
 if coding != "" {
 request.Header.Set("Content-Encoding", coding)
 }
 before := committed.Load()
 response, err := client.Do(request)
 must(err)
 body, readError := io.ReadAll(response.Body)
 closeError := response.Body.Close()
 must(readError)
 must(closeError)
 check(response.Proto == "HTTP/1.1" && response.StatusCode == want, label+": wrong status/protocol")
 wantLength := len(payload)
 if chunked {
 wantLength = -1
 }
 check(response.Header.Get("X-Received-Length") == fmt.Sprint(wantLength), "wrong wire framing")
 delta := int64(0)
 if want == 200 {
 delta = 1
 check(string(body) == "accepted", "wrong successful body")
 }
 check(committed.Load() == before+delta, label+": rejected input caused a side effect")
 fmt.Printf("%s: status=%d committed=%d\n", label, response.StatusCode, committed.Load())
 }
 compress := func(value, name string) []byte {
 var buffer bytes.Buffer
 writer := gzip.NewWriter(&buffer)
 writer.Name = name
 _, err := writer.Write([]byte(value))
 must(err)
 must(writer.Close())
 return buffer.Bytes()
 }
 smallGzip := compress("12345678", "")
 expansion := compress(strings.Repeat("a", 1024), "")
 largeHeader := compress("12345678", strings.Repeat("h", 80))
 check(int64(len(expansion)) <= encodedLimit && int64(len(largeHeader)) > encodedLimit, "gzip fixtures must cross different limits")
 corrupt := append([]byte(nil), smallGzip...)
 corrupt[len(corrupt)-8] ^= 0xff

 probe("exact decoded limit", []byte("12345678"), "", false, 200)
 probe("known oversized", []byte("123456789"), "", false, 413)
 probe("chunked oversized", []byte("123456789"), "", true, 413)
 probe("known encoded limit", bytes.Repeat([]byte("x"), 65), "", false, 413)
 probe("compressed expansion", expansion, "gzip", false, 413)
 probe("exact gzip body", smallGzip, "GZip", false, 200)
 probe("chunked gzip header", largeHeader, "gzip", true, 413)
 probe("bad gzip checksum", corrupt, "gzip", false, 400)
 probe("unsupported coding", []byte("12345678"), "br", false, 415)
 probe("stacked coding", smallGzip, "gzip, gzip", false, 415)
 probe("recovery", []byte("small"), "", false, 200)
 check(handled.Load() == 11 && committed.Load() == 3, "eleven requests must yield three commits")
 fmt.Println(runtime.Version())
}

Eleven requests exercise exact-size plain input, known and chunked oversize input, declared encoded overflow, gzip expansion, exact-size gzip, oversized gzip headers with unknown length, a bad checksum, unsupported coding, stacked coding, and recovery. Only the exact plain body, exact gzip body, and final small body increment the commit counter: three commits total.

Oversize cases return 413, the damaged checksum returns 400, and unsupported or stacked codings return 415. Mixed-case GZip is accepted because coding names are case-insensitive. The server echoes its received ContentLength; unknown-length cases must show -1, so the exercise verifies actual request framing rather than trusting the client-side flag.

The large gzip-header fixture exceeds the encoded limit even though its decoded payload is small. The expansion fixture fits the encoded limit but exceeds the decoded limit. This separates the two budgets instead of treating every 413 as evidence of the same branch.

All assertions passed on macOS with Go 1.27.0. Reads and requests are time-bounded, every response body is closed, and rejection leaves the commit count unchanged. The fixture does not benchmark allocations, inject slow clients, parse multipart or JSON structures, test a reverse proxy, or prove durable transaction behavior.

Sources

RFC 9110: HTTP semantics, Content-Length, content codings, and 413/415 semantics, checked 2026-09-09.
Go net/http, MaxBytesReader, request-body lifecycle, framing, and timeouts, checked 2026-09-09.
Go compress/gzip, decoded streams and checksum verification at EOF, checked 2026-09-09.
OWASP: File Upload Cheat Sheet, upload budgets, decompression limits, and additional validation boundaries, checked 2026-09-09.

Request buffering controls when the backend receives an upload

Tue, 13 Apr 2027 00:00:00 +0000

An upload has two separate connections: client to nginx and nginx to the application. proxy_request_buffering on reads the complete client request body before sending that request upstream. Turning it off lets nginx forward body bytes as they arrive. This changes when the application becomes occupied; it does not create unlimited capacity or make a slow sender disappear.

When it breaks

Disabling the wrong directive is a common mistake. proxy_buffering controls responses, while proxy_request_buffering controls uploads. A streaming upload can still wait at the proxy when only response buffering was disabled. Conversely, an application that should receive complete small submissions can become tied to slow clients when request buffering is disabled globally without considering backend concurrency.

Buffering transfers resource use to nginx. A request body that exceeds the configured memory buffer can use client body temporary storage. Many simultaneous uploads therefore consume proxy connections and potentially disk space or I/O. Increasing a buffer is not a substitute for choosing acceptable body sizes, upload timeouts, concurrency limits, and a temporary filesystem with enough capacity.

Streaming also changes failure handling. Once nginx has begun sending the request body upstream, it cannot simply pass that streamed request to the next server. An application may already have processed a prefix or produced side effects. Buffering can preserve body bytes for some retry paths, but it cannot make an unsafe application operation safe to replay.

Check it quickly

Save this as check-upload.py and run it with host Python and Docker after obtaining the pinned images. The recorded setup used nginx 1.28.0 and Python 3.11.16 containers on Linux arm64. The script creates a private backend network, publishes nginx only on host loopback, and cleans up its own containers, image tags, and network:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True)
 target.write_bytes(content if isinstance(content,bytes) else content.encode())
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 docker('run','-d','--name',containers[0],'--network',name,'-p','127.0.0.1::8080',tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 request.port=port
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'
import hashlib
backend=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import hashlib,json,threading
state={};lock=threading.Lock()
class Handler(BaseHTTPRequestHandler):
 protocol_version='HTTP/1.1'
 def log_message(self,*args):pass
 def reply(self,data):
 data=json.dumps(data).encode();self.send_response(200)
 self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)
 def do_GET(self):
 with lock:self.reply(dict(state))
 def do_POST(self):
 length=int(self.headers['Content-Length'])
 with lock:state[self.path]={'entered':True,'bytes':0}
 first=self.rfile.read(1)
 with lock:state[self.path]['bytes']=len(first)
 body=first+self.rfile.read(length-1)
 with lock:state[self.path]['bytes']=len(body)
 self.reply({'length':len(body),'sha256':hashlib.sha256(body).hexdigest()})
ThreadingHTTPServer(('0.0.0.0',8081),Handler).serve_forever()
'''
locations='''
location /on { proxy_request_buffering on; proxy_pass http://backend:8081; }
location /off { proxy_request_buffering off; proxy_pass http://backend:8081; }
location /state { proxy_pass http://backend:8081; }
'''
with lab(config(locations,'client_max_body_size 1m; proxy_http_version 1.1;'),backend=backend) as (_,request):
 payload=b'x'*65536;results={}
 def state():return json.loads(request('/state')[2])
 for mode in ('on','off'):
 connection=http.client.HTTPConnection('127.0.0.1',request.port,timeout=10)
 connection.putrequest('POST','/'+mode);connection.putheader('Content-Length',str(len(payload)))
 connection.endheaders();connection.send(payload[:1024])
 if mode=='off':wait(lambda:state().get('/off',{}).get('bytes')==1)
 else:
 until=time.monotonic()+1
 while time.monotonic()<until:
 assert '/on' not in state();time.sleep(.05)
 before=state().get('/'+mode)
 connection.send(payload[1024:]);response=connection.getresponse();data=json.loads(response.read());connection.close()
 assert response.status==200 and data=={'length':65536,'sha256':hashlib.sha256(payload).hexdigest()}
 results[mode]={'before_completion':before,'result':data}
 connection=http.client.HTTPConnection('127.0.0.1',request.port,timeout=5)
 connection.putrequest('POST','/off-too-large');connection.putheader('Content-Length',str(2*1024*1024))
 connection.endheaders();response=connection.getresponse();response.read();connection.close()
 assert response.status==413 and '/off-too-large' not in state()
 results['oversize_status']=response.status
 print(json.dumps(results,sort_keys=True))

The client declares a 65,536-byte body but initially sends only 1,024 bytes. With buffering enabled, repeated backend state checks show that the upload handler has not been entered during the controlled pause. The client has started transmitting, yet the application has not been asked to consume the incomplete body. Observing backend state makes this distinction clearer than comparing total request durations.

With buffering disabled, the backend enters its handler and records reading the first byte before the client sends the remainder. It then waits for the rest of the declared body. This demonstrates actual forwarding during the upload, rather than merely confirming that nginx accepts the directive. The backend’s deliberately simple read sequence makes that intermediate state visible.

After the remaining bytes arrive, both modes return length 65,536 and the same expected SHA-256 digest. The buffered and streamed paths therefore preserve this request body despite different timing. These assertions check received bytes, not just a successful status. They do not claim that every application parser, storage operation, or disconnect recovery policy has been exercised.

A separate request announces a two-megabyte body against a one-megabyte limit. nginx returns 413 before the client sends that body, and no matching backend handler appears. This is a size policy applied at the proxy; disabling request buffering does not disable it. The example deliberately uses known Content-Length framing so the admission decision is directly observable.

Choose the upload contract before tuning buffers

Use buffered uploads when keeping slow senders away from scarce application workers is useful and proxy storage is acceptable. Use streaming where the application must process incoming data progressively, while accounting for occupied backend resources and incomplete submissions. Neither mode promises that writers can never be slowed by downstream capacity.

Keep request and response policies independent. An endpoint might accept a buffered upload and stream progress afterward, or stream an upload and return a small buffered result. X-Accel-Buffering is an upstream response control; it does not replace the request-body policy demonstrated here.

Finally, inspect framing and the upstream protocol when extending the experiment. In nginx 1.28.0, an HTTP/1.1 chunked client upload still requires HTTP/1.1 proxying to avoid mandatory buffering in that path. This fixture explicitly selects upstream HTTP/1.1 but tests fixed-length bodies. Validate chunked bodies and disconnect behavior separately when those are part of the endpoint’s real contract.

Sources

nginx proxy module, request buffering, streamed request replay limits, and upstream HTTP version, checked 2026-09-12.
nginx HTTP core module, client body size limits, memory buffers, and temporary storage, checked 2026-09-12.
RFC 9110 HTTP Semantics, Content-Length and the meaning of a 413 response, checked 2026-09-12.

HTTP/3 and QUIC recover per stream

Sun, 21 Mar 2027 00:00:00 +0000

The next version of HTTP moves from TCP to QUIC, a transport with its own reliability built on UDP. The payoff is that packet loss on one stream no longer stalls the whole connection the way it can on TCP with HTTP/2.

Why TCP was the ceiling

HTTP/2 multiplexes many streams over one TCP connection, but TCP itself only knows a single ordered byte stream. When a packet is lost, TCP retransmits and requires in-order delivery, so every stream waiting on that connection halts until recovery completes. That is the head-of-line block HTTP/2 inherits from its transport. The fix is not more TCP tuning but a different transport: give each stream its own reliable, independently-ordered delivery, so a loss in one stream does not stall its neighbors. This is the specific ceiling HTTP/3 is built to remove.

QUIC changes the transport, not the semantics

HTTP/3 keeps the ideas HTTP/2 introduced, multiplexing, header compression, and stream priority, but runs them over QUIC, which lives on UDP and owns its own reliability. QUIC implements per-stream flow control and loss recovery, carries its own encryption, and can migrate a connection when the network path changes, all features TCP ties to the connection. From an application’s point of view the semantics barely change: same methods, headers, and status codes. What moves is the way the bytes are protected, and where connection repair can happen, which is what lets one lossy stream stop dragging the whole page.

When it matters most

Per-stream recovery pays for itself on lossy paths: poor mobile coverage, congested last-mile links, and lightly packet-lossy networks, exactly the places where a single stalling stream used to hold an entire page hostage. It is also a negotiated upgrade, enabled when both the client and server agree, and falling back to HTTP/2 without any action from the site operator where QUIC is unavailable. Rolling forward is therefore gradual and compatible; the deployment risk is low, and the benefit concentrates exactly where users feel it, on unreliable connections.

When it breaks

QUIC still rides on UDP, and some networks treat that as suspicious. Enterprise firewalls and older middleboxes may throttle or drop UDP, or pass small DNS-sized datagrams while discarding larger QUIC packets. The symptom is rarely a clean error. Alt-Svc advertises HTTP/3, the client attempts QUIC, and the handshake times out before falling back to HTTP/2 over TCP. If the fallback timer is too generous, users feel the upgrade as a regression, so track handshake success by network type and keep TCP ready.

0-RTT resumption is a second trap. Replaying the ClientHello saves a round trip, but anything accepted as early data can be replayed by an observer. RFC 9001 therefore limits early data to replay-safe operations. Sending a non-idempotent POST in 0-RTT risks executing the same order twice. Servers must reject risky early data or add anti-replay handling.

Connection migration has its own edge. QUIC identifies a connection by ID rather than the four-tuple, so a phone moving from Wi-Fi to cellular can continue without a new handshake. A NAT rebinding looks similar, however, and a stale NAT binding or a load balancer that hashes on addresses can send return packets nowhere. Endpoints must validate the new path before trusting it.

Finally, do not promise that QUIC removes every queue. It removes transport-level head-of-line blocking between streams, but connection-wide congestion control still applies. A saturated bottleneck delays all streams together. Per-stream recovery isolates loss, not overload.

Check it quickly

Save the following as quic_streams.py in a scratch directory and run python3 quic_streams.py. It models four streams of ten packets each with one packet lost on stream zero, then compares an idealized QUIC scheduler against an idealized TCP scheduler. No sockets are opened; the numbers are a scheduling calculation, not a wire capture.

STREAMS = 4
PACKETS = 10
LOST_STREAM = 0
RETRANSMIT_PENALTY = 3


def quic_completion():
 """Each stream recovers on its own: only the lossy stream pays."""
 completions = []
 for stream in range(STREAMS):
 if stream == LOST_STREAM:
 completions.append(PACKETS + RETRANSMIT_PENALTY)
 else:
 completions.append(PACKETS)
 return completions


def tcp_completion():
 """One ordered byte stream: a single loss blocks every stream."""
 blocked_at = PACKETS + RETRANSMIT_PENALTY
 return [blocked_at] * STREAMS


def main():
 quic = quic_completion()
 tcp = tcp_completion()
 # Only the stream that lost a packet waits for retransmission.
 assert quic[LOST_STREAM] == PACKETS + RETRANSMIT_PENALTY, quic
 assert all(t == PACKETS for i, t in enumerate(quic) if i != LOST_STREAM), quic
 # TCP cannot release later bytes until the loss is repaired.
 assert all(t == PACKETS + RETRANSMIT_PENALTY for t in tcp), tcp
 quic_blocked = sum(1 for t in quic if t > PACKETS)
 tcp_blocked = sum(1 for t in tcp if t > PACKETS)
 assert quic_blocked == 1, quic
 assert tcp_blocked == STREAMS, tcp
 print(f"QUIC completion: {quic} (blocked streams: {quic_blocked})")
 print(f"TCP completion: {tcp} (blocked streams: {tcp_blocked})")
 print("PASS: loss stalls one QUIC stream but all TCP streams")


if __name__ == "__main__":
 main()

The QUIC schedule completes the three healthy streams at time ten while only the lossy stream waits until time thirteen. The TCP schedule completes every stream at time thirteen because the single ordered byte stream cannot release later bytes until the loss is repaired. The script asserts that isolation, then prints both vectors. The model ignores congestion and reordering; it isolates loss recovery so the difference is visible in one run.

Sources

Cloudflare: HTTP/3 and QUIC, the protocol rationale, checked 2027-03-19.
RFC 9000: QUIC, the transport protocol, checked 2027-03-19.
RFC 9114: HTTP/3, HTTP over QUIC, checked 2027-03-19.

EXPLAIN estimates need execution context

Fri, 19 Mar 2027 00:00:00 +0000

EXPLAIN describes the plan PostgreSQL intends to execute. EXPLAIN ANALYZE executes that plan and adds measurements. Comparing them is useful only after interpreting what each number counts. An apparent mismatch can reflect stale estimates, repeated execution, or a parent that stopped requesting rows. Matching cardinalities strengthen one part of the model; they do not prove the chosen plan is the fastest possible plan.

Read a node in its context

Estimated rows describes the output of a node, not necessarily all tuples it examines. A scan may examine thousands of tuples and emit ten after filtering. Startup cost and total cost are planner units, not milliseconds. Parent costs include child work, so adding every node’s cost double-counts the tree. Actual parent timing is also generally inclusive rather than a list of independent charges.

For repeatedly executed nodes, actual row counts and timings are reported per execution, averaged across loops. A node returning two rows with three loops supplied six rows in total. Filter-removal counts are likewise per-loop averages, so rounding can matter. Start at the leaves and follow where the observed population first differs materially from the estimate, while respecting those execution boundaries.

A LIMIT, EXISTS, or another early-stopping parent may leave a child only partly consumed. The child’s estimate can describe its full potential output while the actual count describes only rows requested. That is not automatically a statistics error. Sorts can also read their entire input while delivering only the requested top rows.

When it breaks

Running EXPLAIN ANALYZE on a write actually performs the write. A rollback can protect transactional table changes, but does not reverse every possible side effect, such as sequence advancement or an external action invoked by a function. Choose a disposable fixture or understand the complete statement before measuring it.

Another mistake is diagnosing storage from BUFFERS alone. A shared-buffer hit avoids a PostgreSQL buffer miss; a reported read can still be served by the operating system cache. Buffer counts describe block accesses, not distinct blocks or guaranteed physical-device reads. Likewise, accurate rows do not exclude lock waits, memory spills, configuration problems, or an access path that the search did not select.

Check it quickly

Use PostgreSQL 18 and psql. The transaction creates three parents and thirty children, then compares a repeated lateral lookup with a simple top-three query. All data is temporary and the final rollback removes the fixture.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE parents (id integer PRIMARY KEY);
CREATE TEMP TABLE children (parent_id integer,value integer);
INSERT INTO parents VALUES (1),(2),(3);
INSERT INTO children SELECT 1+(g-1)%3,g FROM generate_series(1,30) AS g;
ANALYZE parents;
ANALYZE children;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT p.id,c.value FROM parents p CROSS JOIN LATERAL (
 SELECT value FROM children WHERE parent_id=p.id ORDER BY value DESC LIMIT 2
) c;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT value FROM children ORDER BY value DESC LIMIT 3;
SELECT p.id,c.value FROM parents p CROSS JOIN LATERAL (
 SELECT value FROM children WHERE parent_id=p.id ORDER BY value DESC LIMIT 2
) c ORDER BY p.id,c.value DESC;
DO $$ BEGIN
 IF (SELECT array_agg(value ORDER BY parent_id,value DESC) FROM (
 SELECT p.id AS parent_id,c.value FROM parents p CROSS JOIN LATERAL (
 SELECT value FROM children WHERE parent_id=p.id ORDER BY value DESC LIMIT 2
 ) c
 ) s) IS DISTINCT FROM ARRAY[28,25,29,26,30,27] THEN RAISE EXCEPTION 'wrong lateral result'; END IF;
END $$;
ROLLBACK;

The PostgreSQL 18.6 lateral plan used a nested loop. Its inner Limit reported actual rows of two and loops=3, producing six joined rows. The child scan returned ten matching rows per loop and removed twenty, so it examined all thirty children on each of three passes. Read those numbers together instead of concluding that the whole query inspected only ten rows.

The final ordered results were parent one with values 28 and 25, parent two with 29 and 26, and parent three with 30 and 27. The assertion checks those memberships independently of the plan’s appearance. Correct answers remain required even if another PostgreSQL version chooses different access nodes.

For the separate top-three query, the sort estimated thirty output rows but emitted three because its parent stopped there. Its scan still read thirty rows. The fixture had just been analyzed, and this mismatch demonstrates consumption semantics rather than evidence that refreshing statistics would fix anything.

Measure the question you need answered

TIMING OFF keeps actual row counts and overall execution time while avoiding repeated per-node clock reads. BUFFERS adds useful context about page access, and JSON output can support systematic comparison. Neither option makes the instrumented query identical to application latency: client transmission, rendering, connection setup, and concurrent load need their own observations. Preserve the parameters and relevant session settings when comparing plans, then explain a change in terms of rows processed, loops, spills, or waits rather than declaring every new node name a regression.

Sources

PostgreSQL 18: EXPLAIN, execution effects and measurement options, checked 2026-09-10.
PostgreSQL 18: Using EXPLAIN, row counts and loops and plan interpretation, checked 2026-09-10.

Upstream keepalive caches idle connections, not active capacity

Mon, 15 Mar 2027 00:00:00 +0000

An nginx upstream keepalive cache preserves eligible idle backend connections for later requests. In the tested configuration, keepalive 2 permits two idle connections in that upstream cache for each worker. It does not restrict the worker to two active backend connections. Reuse reduces repeated connection setup; it does not itself impose an application concurrency budget or guarantee lower latency for every request.

When it breaks

An idle-cache size mistaken for a connection cap leaves the backend exposed to more simultaneous work than expected. A busy worker can open additional active connections while its retained idle cache remains small. With several workers, each worker’s idle cache also matters. Count active and idle resources separately when comparing nginx settings with a backend’s actual connection capacity.

Connection reuse requires a reusable protocol exchange. For this nginx 1.28.0 HTTP example, upstream HTTP/1.1 and clearing the default Connection: close header are explicit configuration choices. Merely setting Host does not establish persistence. A backend can still close connections, and incomplete or incorrectly framed messages cannot be treated as clean reusable exchanges just because a cache is configured.

Client-side persistence is a different connection relationship. A fresh client connection can be served using a previously cached upstream connection, while a long-lived client can trigger several backend connections. Assuming those lifetimes match leads to misleading measurements. Connection setup savings also differ between a plain HTTP upstream and a TLS upstream; this experiment uses plain HTTP behind nginx.

Check it quickly

Save the script as check-upstream-keepalive.py and run it with Python, Docker, and the pinned images. It uses one nginx 1.28.0 worker and a Python HTTP/1.1 backend on a private network. Each accepted backend connection receives an identity, and each response has an explicit length. Client requests enter through host loopback:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True)
 target.write_bytes(content if isinstance(content,bytes) else content.encode())
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 docker('run','-d','--name',containers[0],'--network',name,'-p','127.0.0.1::8080',tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 request.port=port
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'
from concurrent.futures import ThreadPoolExecutor
backend=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import itertools,json,threading
ids=itertools.count(1);lock=threading.Lock();opened=set();active=set();release=threading.Event()
class Handler(BaseHTTPRequestHandler):
 protocol_version='HTTP/1.1'
 def setup(self):
 super().setup();self.ident=next(ids)
 with lock:opened.add(self.ident)
 def finish(self):
 try:super().finish()
 finally:
 with lock:opened.discard(self.ident)
 def log_message(self,*args):pass
 def do_GET(self):
 if self.path=='/hold':
 with lock:active.add(self.ident)
 assert release.wait(15)
 with lock:active.remove(self.ident)
 elif self.path=='/release':release.set()
 with lock:data=json.dumps({'connection':self.ident,'active':sorted(active),'other_open':sorted(opened-{self.ident}),
 'host':self.headers.get('Host'),'connection_header':self.headers.get('Connection')}).encode()
 self.send_response(200);self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)
ThreadingHTTPServer(('0.0.0.0',8081),Handler).serve_forever()
'''
locations='''
location /reuse/ { proxy_http_version 1.1; proxy_set_header Connection ""; proxy_pass http://pool/; }
location /close/ { proxy_http_version 1.1; proxy_set_header Connection close; proxy_pass http://pool/; }
location /host-only/ { proxy_http_version 1.1; proxy_set_header Host service.example; proxy_pass http://pool/; }
location /control/ { proxy_set_header Connection close; proxy_pass http://backend:8081/; }
'''
extra='upstream pool { server backend:8081; keepalive 2; keepalive_timeout 30s; }'
with lab(config(locations,extra),backend=backend) as (_,request):
 def get(path):
 status,_,body=request(path);assert status==200;return json.loads(body)
 results={}
 for mode in ['reuse','close','host-only']:
 first=get('/'+mode+'/id');second=get('/'+mode+'/id')
 assert (first['connection']==second['connection'])==(mode=='reuse')
 if mode=='host-only':assert first['host']=='service.example' and first['connection_header']=='close'
 results[mode]={'connections':[first['connection'],second['connection']],'host':first['host'],'connection_header':first['connection_header']}
 with ThreadPoolExecutor(max_workers=4) as pool:
 futures=[pool.submit(get,'/reuse/hold') for _ in range(4)]
 wait(lambda:len(get('/control/state')['active'])==4)
 active=get('/control/state')['active'];assert len(set(active))==4
 get('/control/release');replies=[f.result(timeout=10) for f in futures]
 assert len({r['connection'] for r in replies})==4
 wait(lambda:len(get('/control/state')['other_open'])==2)
 results['concurrent_active']=active;results['idle_after_burst']=get('/control/state')['other_open']
 print(json.dumps(results,sort_keys=True))

Two sequential /reuse/id requests return the same backend connection identity. The client helper opens and closes its own HTTP connection for each request, so this is upstream reuse across distinct downstream connections. The location explicitly uses upstream HTTP/1.1 and sends no Connection header. Reading each complete response leaves the backend exchange eligible to become idle.

The /close/id control sends Connection: close and returns different backend identities for successive requests. The /host-only/id control changes Host but leaves the closing behavior intact; its responses also use different connections. These comparisons show why a routing header is not a substitute for correct persistence settings, even when all paths contact the same application.

Next, four clients request responses that the backend deliberately holds open. The controller waits until the backend reports four active connection identities before releasing any response. That directly demonstrates four simultaneous upstream connections under keepalive 2. It does not infer concurrency from timing or from a count of completed requests that might have been processed sequentially.

After release, all four replies complete. The fixture observes exactly two remaining idle backend connections, excluding the short-lived control connection used to inspect state. Which identities survive can vary with completion order; their count is the assertion. This distinguishes the configured idle retention limit from the larger set of active connections used during the burst.

Match reuse settings to backend behavior

Size idle retention using connection setup cost, expected reuse, backend limits, and the number of nginx workers. A retained connection consumes resources even while it carries no request. A very large cache can therefore compete with new useful connections, while a very small one may repeatedly discard connections that would soon have been reused.

Request-count, idle-time, and total connection-lifetime controls can recycle persistent connections for different reasons. Do not respond to every reconnection by increasing all of them. Observe backend closes, configuration, completed response framing, and actual connection identities first. The tested idle timeout is thirty seconds; the short burst does not benchmark long-running connection aging or TLS session behavior.

Finally, use an actual concurrency or admission policy when limiting backend work is the objective. Keepalive is reuse policy. The fixture chooses its protocol and cache settings explicitly because rolling documentation includes later nginx defaults and features. Apply the semantics of the deployed version, and verify both reuse and a concurrent burst instead of assuming one successful repeated request proves the capacity boundary.

Sources

nginx upstream module, idle connection caches, worker scope, and keepalive limits, checked 2026-09-12.
nginx 1.28.0 upstream keepalive source, the tested version’s idle-cache and connection reuse implementation, checked 2026-09-12.
RFC 9112 HTTP/1.1, persistent connections, message framing, and Connection close, checked 2026-09-12.

Performance · XB Field Notes

Huge pages cut TLB misses at the cost of flexibility

Why 4 KB hurts

hugetlb vs transparent

Budget the pool

When it breaks

Check it quickly

Further reading

Sources

Cache-Control is a contract between cache and origin

Both sides must agree

When it breaks

Check it quickly

Sources

EXPLAIN ANALYZE runs the query to show real cost

Read comparable quantities

Investigate the first meaningful discrepancy

When it breaks

Check it quickly

Sources

pg_stat_statements finds the queries that cost the most

Normalize, then aggregate

Total time, not mean time

Counters accumulate, so compare windows

When it breaks

Check it quickly

Sources

TCP window scaling enlarges receive-window encoding in each direction

When it breaks

Check it quickly

Separate buffer allocation, advertised capacity, and observed performance

Sources

A cache stampede hits when expiry syncs

Coalesce work within its actual boundary

Spread deadlines without changing the promise

When it breaks

Check it quickly

Further reading

Sources

Nagle and delayed ACK can stall the second part of a small request

When it breaks

Check it quickly

Choose framing and socket policy together

Sources

Full table scans win on small tables

The currency is pages, not rows

Where the index stops paying

Let the planner decide, but keep its inputs honest

When it breaks

Check it quickly

Sources

SACK reports received ranges beyond a gap so recovery can target missing data

When it breaks

Check it quickly

Interpret feedback without promising perfect retransmission decisions

Sources

TLS session resumption saves a round trip

New connection, fresh traffic keys

When it breaks

Check it quickly

Further reading

Sources

TLS handshakes add latency you can shrink

Messages travel in flights

When it breaks

Check it quickly

Further reading

Sources

HTTP keep-alive reuses one TCP connection

Reuse requires complete messages

Each hop owns its timeout

When it breaks

Check it quickly

Sources

EXPLAIN plans reveal the route of a query

Read the plan as a route map

Statistics steer the chosen route

Confirm the route moved

When it breaks

Check it quickly