Postgres · XB Field Notes

Locks wait politely when a transaction grabs them

Wed, 16 Feb 2028 00:00:00 +0000

A lock wait can be healthy coordination or an expensive queue. PostgreSQL’s lock_timeout lets an operation stop waiting, but it does not diagnose why the holder is slow or decide whether retrying is appropriate. Choose the timeout from the operation’s latency budget, and inspect the blocking transaction before increasing the limit or multiplying attempts.

When it breaks

lock_timeout applies separately to each lock acquisition attempt, including implicit locks. It measures time waiting for that lock, not all time spent by the statement. A query already doing useful work can still hit the limit when it later requests another lock. Several shorter waits can also accumulate beyond one configured interval. Treating this as a whole-query deadline leaves that accumulation uncontrolled.

statement_timeout bounds statement execution, including lock waiting. If it is shorter, it can cancel the statement before the lock timeout fires. idle_in_transaction_session_timeout addresses a different problem: a session waiting for its next client command while keeping a transaction open. A backend blocked inside an update is ordinarily active, with a lock wait event; calling every blocked connection idle in transaction directs diagnosis at the wrong sessions.

Check it quickly

Open two fresh psql -X -v ON_ERROR_STOP=1 sessions, A and B, against the same disposable PostgreSQL 18 database. Start with the default lock_timeout of zero and no conflicting timeout settings. The short durations below make the demonstration fast; they are not production recommendations. A creates the fixture:

CREATE SCHEMA timeout_demo;
CREATE TABLE timeout_demo.counter (id integer PRIMARY KEY,n integer);
INSERT INTO timeout_demo.counter VALUES (1,0);

A updates the row inside an open transaction and keeps its lock:

BEGIN;
UPDATE timeout_demo.counter SET n=10 WHERE id=1;

B waits at most one hundred milliseconds for its conflicting update. Temporarily allowing expected errors also exposes the transaction state before rollback:

BEGIN;
SET LOCAL lock_timeout='100ms';
SET LOCAL statement_timeout='1s';
\set ON_ERROR_STOP off
UPDATE timeout_demo.counter SET n=n+1 WHERE id=1;
\echo lock_state :SQLSTATE
SELECT n FROM timeout_demo.counter WHERE id=1;
\echo transaction_state :SQLSTATE
\set ON_ERROR_STOP on
ROLLBACK;

Expect lock_state 55P03, then transaction_state 25P02. The first error cancels this statement and leaves the explicit transaction failed. B’s rollback clears that failed state; it does not release A’s lock or undo A’s pending update. Repeating work inside B’s failed transaction would only generate another transaction-state error.

Leave A open. B now gives the whole statement a shorter budget than the lock acquisition:

BEGIN;
SET LOCAL lock_timeout='100ms';
SET LOCAL statement_timeout='50ms';
\set ON_ERROR_STOP off
UPDATE timeout_demo.counter SET n=n+1 WHERE id=1;
\echo statement_state :SQLSTATE
\set ON_ERROR_STOP on
ROLLBACK;

This returns statement_state 57014. The operation is still waiting for the same row, but the statement deadline wins. These error classes are evidence to interpret with context: 55P03 can also arise from NOWAIT, and 57014 also covers other query cancellations. Neither code by itself proves a deadlock or names the blocking application.

In A, discard the pending value ten and release the row:

ROLLBACK;

B starts a fresh, bounded transaction and succeeds:

BEGIN;
SET LOCAL lock_timeout='100ms';
SET LOCAL statement_timeout='1s';
UPDATE timeout_demo.counter SET n=n+1 WHERE id=1 RETURNING n;
COMMIT;

The returned value is one. Neither failed B attempt committed an increment, and A rolled back its ten. Next, B demonstrates that ordinary elapsed execution does not consume the lock-wait budget:

BEGIN;
SET LOCAL lock_timeout='50ms';
SET LOCAL statement_timeout='1s';
SELECT pg_sleep(0.2);
\echo sleep_state :SQLSTATE
COMMIT;
SHOW lock_timeout;

The two-hundred-millisecond sleep succeeds with sleep_state 00000 despite a fifty-millisecond lock timeout. After commit, SHOW returns the session’s original zero because SET LOCAL ended with its transaction. A verifies the counter remains one and cleans up:

SELECT n FROM timeout_demo.counter WHERE id=1;
DROP SCHEMA timeout_demo CASCADE;

Recover without amplifying contention

Select retry policy per operation. A lock timeout may justify a new transaction after rollback, provided the request still has time and replay is valid. A small attempt cap, jitter, and an overall deadline bound the extra load; they cannot make a permanently blocked resource available. Keep the original error and report exhausted attempts instead of silently dropping the request.

A true deadlock is a cycle of dependencies. PostgreSQL detects it and reports 40P01; a lock timeout can fire before that detection, so a timed-out run does not establish that no cycle existed. Inspect blocker identities, transaction age, and lock acquisition order. Shortening a transaction that waits for external input often removes the cause more effectively than raising every caller’s timeout. Do not copy these laboratory millisecond limits into an unrelated workload without measuring normal hold times.

Sources

PostgreSQL 18 client connection defaults, lock, statement, and idle transaction timeout scopes, checked 2026-09-10.
PostgreSQL 18 explicit locking, conflicting locks and deadlock detection, checked 2026-09-10.

Partial indexes trim a column

Tue, 15 Feb 2028 00:00:00 +0000

Not every row deserves an index slot; a partial index keeps the tree thin exactly where the query never looks. Most tables have a skew the schema pretends not to see: ninety percent of orders are closed, nearly all events are processed, the flag column is false almost everywhere. A full index dutifully records all of it, charging maintenance on every write to protect lookups that only ever touch the small live slice. The partial index stops paying for the rows nobody queries.

The bargain is precision for size. The index becomes smaller, faster to scan, and cheaper to maintain — but only queries whose predicates match the index condition may use it. Everything else falls back to competing plans. That makes partial indexes a deliberate pairing of access pattern with storage, not a general speedup. Choose them where the hot subset is stable and the queries agree on its definition.

Index only what you query

A partial index carries a WHERE clause, so rows that fail the predicate never enter the tree. When most rows are rarely read, this keeps the index small and the lookups fast on the ones that matter. An index on open orders rather than all orders can be an order of magnitude smaller, which means more of it fits in cache, fewer pages per lookup, and plans that stay index-driven as the cold history grows unboundedly.

Pick the predicate from the queries. The right condition is the one every important query already repeats: status equals open, deleted_at is null. If queries slice the table differently, either the dominant slice wins or the table needs two partial indexes, each still thinner than a full one.

Fewer rows, cheaper maintenance

Fewer entries mean inserts, updates, and VACUUM touch a lighter structure, and disk usage drops visibly. The savings are most obvious on write-heavy tables where the full set would be mostly noise. Every insert that fails the predicate skips the index entirely — no page split risk, no WAL for that entry — and updates that leave the predicate untouched skip index maintenance too. On tables absorbing thousands of cold writes per second, the avoided work dwarfs the lookup gains.

Smaller indexes also vacuum faster with fewer pages for bloat to hide in. But monitor size against the full-index equivalent: if the partial index approaches full size, the predicate no longer selects a minority and the bargain has expired — drop it or redefine the slice.

Shape the statement to match

The planner only honors the index for queries whose predicate matches the partial condition exactly, or implies it. Write the filter the same way in the query and the index so the planner can actually use it. status = ‘open’ matches an index predicated on status = ‘open’; status IN (‘open’, ‘pending’) does not, even though a human sees the overlap. The matcher is logical implication, not textual similarity, and it is conservative by design.

This strictness dictates a discipline: canonicalize the filter expression across the codebase. If the index says deleted_at IS NULL, no query may phrase it as deleted_at = NULL or COALESCE(deleted_at, …) IS NULL and expect the index. Centralize the hot-slice condition — a view, a scope, a query builder fragment — so every statement inherits the exact predicate. EXPLAIN is the arbiter: if the plan ignores the index, the predicates differ in a way the matcher cannot bridge.

When it breaks

The classic failure is the drifting predicate: the business redefines “open” to include a new status, application queries update, and the partial index silently stops matching half the workload. Nothing errors — plans just degrade to sequential scans one query at a time, and latency climbs without a deploy to blame. Guard with a periodic EXPLAIN on the canonical queries in CI or monitoring, asserting the index name appears in the plan. A plan assertion catches predicate drift the day it ships, not the quarter it gets noticed.

The second failure is the over-sliced table: five partial indexes, each serving one report, whose combined maintenance exceeds the full index they replaced. Audit with pg_stat_user_indexes — near-zero-scan indexes are removal candidates, and overlapping predicates should merge into one wider partial index.

Check it quickly

Prove the index matches and pays:

CREATE INDEX ON orders (created_at) WHERE status = 'open';
EXPLAIN SELECT * FROM orders WHERE status = 'open' ORDER BY created_at LIMIT 20;
SELECT pg_size_pretty(pg_relation_size('orders_created_at_idx'));

The plan must show an index scan on the partial index, and the size should be a small fraction of the full-index equivalent.

Sources

PostgreSQL partial indexes, Official partial index documentation, checked 2026-06-01.
PostgreSQL EXPLAIN, plan inspection for verifying index usage, checked 2026-09-05.

Partition pruning skips whole partitions a query cannot need

Sun, 13 Feb 2028 00:00:00 +0000

Partition pruning excludes entire partitions that cannot contain matching rows. It is different from using an index to find rows within a remaining partition, and it is not a page-level cache trick. PostgreSQL can perform this exclusion while planning a query or later when parameter values become available during execution.

Prove which partitions cannot match

Declarative partitions have bounds derived from their partition key. A predicate restricting a date key to October can prove that September and November are irrelevant. Pruning uses those bounds, not an estimate that another partition probably contains few matches. Statistics still influence other planning decisions, but an inaccurate row estimate does not make partition membership a guess.

Indexes are not required for pruning. After excluding irrelevant partitions, PostgreSQL may choose a sequential scan of a remaining partition because most of its rows are needed. An Index Scan label is therefore not the test for successful pruning. Inspect the relation names and plan structure, then separately evaluate how each remaining relation is accessed.

A restriction must be usable with the partition key and its comparison semantics. Converting a date column to text can hide the direct bound relationship even when a particular string comparison happens to select the same rows. Use typed parameters and predicates that describe the intended interval directly. Do not assume that arbitrary expressions around a partition key preserve pruning or even preserve the original date semantics.

Parameters can prune during execution

A constant can allow planning-time pruning, leaving only the relevant child relation in the plan. A prepared statement’s generic plan cannot know every future parameter value in advance. That does not mean the statement must scan every partition: execution-time pruning can use values known when the plan initializes.

EXPLAIN ANALYZE reports Subplans Removed for partitions discarded during initialization. Other cases, such as parameters changing on the inner side of a nested-loop join, can prune during execution itself. Inspect loop counts and whether subplans actually ran when reasoning about those cases. A never-executed node alone can have other causes, such as an empty outer input, so interpret it in its full plan context.

Avoid turning these observations into a blanket rule that every nonconstant expression disables pruning. Parameter availability and expression support determine when exclusion is possible. Custom and generic prepared plans can differ while returning the same answer. The useful question is which partitions were accessed for the actual values used by the workload.

When it breaks

A filter on an unrelated column may leave every partition relevant. Casting the key into an incompatible expression can do the same. Check the exact predicate and enable_partition_pruning setting before changing the partition layout. Manually inventing a contradictory CHECK constraint is not a repair; ordinary local declarative partitions enforce their bounds, and constraint exclusion is a separate optimization using CHECK constraints.

Pruning also is not a promise of no locking or negligible planning cost. Partitions removed during execution initialization are still locked at the start of execution. Large partition hierarchies can consume planning time and per-session metadata memory even when a query eventually scans only a few children. Choose the hierarchy for realistic query and maintenance patterns rather than maximizing partition count.

Finally, pruning two partitions can still leave thousands of irrelevant rows inside the third. A matching local index may help that second stage. Returning only a hundred rows does not prove that only a hundred rows were examined, and buffer hits are not the same as physical disk reads.

Check it quickly

Save this as partition-pruning.sql and run psql -X -v ON_ERROR_STOP=1 -f partition-pruning.sql in a disposable PostgreSQL 18 database. The script enables pruning within its transaction and uses a generic prepared plan to expose initialization-time pruning. It creates no indexes on the temporary partitions.

\set ON_ERROR_STOP on
BEGIN;
SET LOCAL datestyle='ISO, YMD';
SET LOCAL enable_partition_pruning=on;
CREATE TEMP TABLE observations (day date,id integer) PARTITION BY RANGE(day);
CREATE TEMP TABLE obs_sep PARTITION OF observations FOR VALUES FROM ('2026-09-01') TO ('2026-10-01');
CREATE TEMP TABLE obs_oct PARTITION OF observations FOR VALUES FROM ('2026-10-01') TO ('2026-11-01');
CREATE TEMP TABLE obs_nov PARTITION OF observations FOR VALUES FROM ('2026-11-01') TO ('2026-12-01');
INSERT INTO observations SELECT DATE '2026-09-01'+g%90,g FROM generate_series(0,8999) AS g;
ANALYZE observations;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT count(*) FROM observations WHERE day>=DATE '2026-10-01' AND day<DATE '2026-11-01';
SET LOCAL plan_cache_mode=force_generic_plan;
PREPARE by_day(date) AS SELECT count(*) FROM observations WHERE day=$1;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) EXECUTE by_day(DATE '2026-10-10');
EXECUTE by_day(DATE '2026-10-10');
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT count(*) FROM observations WHERE day::text='2026-10-10';
DO $$ BEGIN
 IF (SELECT count(*) FROM observations WHERE day=DATE '2026-10-10') <> 100 OR
 (SELECT count(*) FROM observations WHERE day>=DATE '2026-10-01' AND day<DATE '2026-11-01') <> 3100 THEN
 RAISE EXCEPTION 'unexpected pruned result';
 END IF;
END $$;
DEALLOCATE by_day;
ROLLBACK;

PostgreSQL 18.6 planned the constant October range as a scan of obs_oct alone, reading 3,100 matching rows. The generic date lookup reported Subplans Removed: 2 and returned 100 rows, but still filtered 3,000 other rows within October. This demonstrates both successful partition exclusion and remaining local work.

The text-cast predicate scanned all three children and returned the same 100 rows under the explicitly chosen ISO DateStyle. Assertions verify the typed query’s counts, and DEALLOCATE plus rollback clean up the prepared statement and temporary tables. These results establish the predicate and pruning relationship for this fixture; measure representative plans separately before choosing an index or changing production partition granularity.

Sources

PostgreSQL table partitioning, planning and execution pruning, bounds, and locking, checked 2026-09-10.
PostgreSQL PREPARE, generic and custom plans and parameter handling, checked 2026-09-10.

Table partitioning splits data by range or key

Thu, 10 Feb 2028 00:00:00 +0000

Range partitioning turns a boundary definition into a write-routing rule. PostgreSQL places a row in the partition whose bounds accept its key, whether the row arrives on time or months late. Correct boundaries, compatible timestamp types, and a plan for missing ranges matter before any query can benefit from pruning.

Use adjacent half-open ranges

A range partition includes its lower bound and excludes its upper bound. September can cover the interval from September first through, but not including, October first. October then starts at that exact boundary without overlap. Do not model a month by guessing its final representable timestamp; adjacent half-open ranges work regardless of the timestamp’s fractional precision.

Queries should express the intended interval just as carefully. A lower-inclusive, upper-exclusive predicate matches a monthly slice. BETWEEN includes both endpoints, so using the next month’s first instant as its upper endpoint can include data from that next month. Partition routing does not correct a predicate that asks for the wrong interval.

The key’s type and time zone are part of the definition. A timestamptz value represents an instant, while a timestamp without time zone represents a local date and time without that instant’s zone interpretation. The example uses explicit UTC offsets for every boundary. If business months follow another time zone, derive boundaries using that zone’s rules instead of assuming every local day has the same elapsed duration.

Missing ranges are an operational state

An insert with no matching partition raises an error unless a DEFAULT partition accepts it. PostgreSQL does not automatically create the next monthly table when the calendar advances. Create needed ranges ahead of demand and account for late data and corrections to old dates. A key need not arrive monotonically for routing to work; its destination simply must still exist.

DEFAULT can absorb values outside the explicitly declared ranges. That can reduce immediate insertion failures, but it also makes missing partition management less visible unless the default table is inspected. A growing default partition can contain several unrelated time ranges and become harder to reorganize. Keep the difference between an intentional exception area and forgotten routine data visible.

Adding a new explicit partition changes which values the default partition may contain. If rows belonging to the new range remain in DEFAULT, PostgreSQL rejects the change rather than leaving overlapping ownership. Move those rows to an appropriate staging table or otherwise reconcile them before attaching the new partition. Bounds are enforced data rules, not labels that can simply be made inaccurate.

When it breaks

A deployment can fail exactly at a month boundary because the next partition was never created. Another can fail while adding that partition because DEFAULT already accumulated its rows. Both failures concern routing and data placement, not stale planner statistics. Inspect the rejected key and existing bounds before running ANALYZE or changing indexes.

ATTACH may need to scan the candidate table to validate its bounds and also check the default partition for conflicting rows. Suitable validated constraints can avoid some scans, but the operation still requires locks and compatible table structure. On active tables, coordinate writers and lock acquisition while relocating data; an isolated transactional demonstration is not a complete online migration procedure.

Nullability is separate from range selection. The example requires a nonnull timestamp so a missing event time cannot quietly become ordinary exception data. Also verify row counts and identifiers after relocation: a successful attach is not proof that an earlier copy step neither lost nor duplicated rows.

Check it quickly

Save this as partition-ranges.sql and run psql -X -v ON_ERROR_STOP=1 -f partition-ranges.sql in a disposable PostgreSQL 18 database. All tables are temporary and the whole experiment is rolled back.

\set ON_ERROR_STOP on
BEGIN;
SET LOCAL timezone='UTC';
CREATE TEMP TABLE samples (at timestamptz NOT NULL,id integer) PARTITION BY RANGE(at);
CREATE TEMP TABLE samples_sep PARTITION OF samples
FOR VALUES FROM ('2026-09-01 00:00:00+00') TO ('2026-10-01 00:00:00+00');
INSERT INTO samples VALUES ('2026-09-01 00:00:00+00',1),('2026-09-30 23:59:59.999999+00',2);
DO $$ BEGIN
 INSERT INTO samples VALUES ('2026-10-01 00:00:00+00',3);
 RAISE EXCEPTION 'upper boundary entered September';
EXCEPTION WHEN check_violation THEN RAISE NOTICE 'upper boundary needs another partition';
END $$;
CREATE TEMP TABLE samples_default PARTITION OF samples DEFAULT;
INSERT INTO samples VALUES ('2026-10-01 00:00:00+00',3);
DO $$ BEGIN
 CREATE TEMP TABLE samples_oct_bad PARTITION OF samples
 FOR VALUES FROM ('2026-10-01 00:00:00+00') TO ('2026-11-01 00:00:00+00');
 RAISE EXCEPTION 'new partition overlapped populated default';
EXCEPTION WHEN check_violation THEN RAISE NOTICE 'matching default rows must be moved first';
END $$;
CREATE TEMP TABLE samples_oct (LIKE samples INCLUDING ALL);
WITH moved AS (
 DELETE FROM samples_default
 WHERE at >= TIMESTAMPTZ '2026-10-01 00:00:00+00' AND at < TIMESTAMPTZ '2026-11-01 00:00:00+00'
 RETURNING *
) INSERT INTO samples_oct SELECT * FROM moved;
ALTER TABLE samples ATTACH PARTITION samples_oct
FOR VALUES FROM ('2026-10-01 00:00:00+00') TO ('2026-11-01 00:00:00+00');
SELECT id,at,tableoid::regclass AS physical_table FROM samples ORDER BY id;
DO $$ BEGIN
 IF (SELECT array_agg(id ORDER BY id) FROM samples_sep) IS DISTINCT FROM ARRAY[1,2] OR
 (SELECT array_agg(id ORDER BY id) FROM samples_oct) IS DISTINCT FROM ARRAY[3] OR
 (SELECT count(*) FROM samples_default) <> 0 THEN
 RAISE EXCEPTION 'range migration lost or duplicated data';
 END IF;
END $$;
ROLLBACK;

PostgreSQL 18.6 accepted the first instant of September and the final tested instant before October. October first was rejected before a destination existed, then entered DEFAULT after that partition was created. Attempting to add an overlapping October partition failed while the matching default row remained there.

The script moved that row using DELETE RETURNING, attached its staging table as October, and displayed each row’s actual tableoid. Ids one and two remained in September, id three appeared in October, and DEFAULT was empty. Assertions verify the final membership and count. This tests both boundary inclusion and a complete relocation path, without treating a quiet default partition as evidence that future range management can be omitted.

Sources

PostgreSQL table partitioning, range boundaries and routing behavior, checked 2026-09-10.
PostgreSQL ALTER TABLE, attach validation, default partitions, and locks, checked 2026-09-10.

Window functions rank rows without collapsing groups

Tue, 08 Feb 2028 00:00:00 +0000

Ranking functions attach a position to each input row within a partition. They do not collapse a team, department, or customer into one aggregate row. The important choice is what a tie means: a unique position, a shared competition rank with gaps, or a shared rank counting distinct score groups.

Three questions, three rankings

row_number() assigns successive integers starting at one. If two rows have the same ordering values, their relative positions are unspecified unless the window order includes a tie-breaker. For an exactly-two-winners rule, order by score descending and then a stable unique identifier. The identifier decides which equally scored entrant takes the remaining place, so that policy should be intentional.

rank() gives peers the position of the first row in their peer group. Scores 100, 90, 90, and 80 therefore produce ranks 1, 2, 2, and 4. The two second-place rows consume two positions, leaving a gap before the next score. dense_rank() instead counts peer groups, producing 1, 2, 2, and 3 for the same scores.

Peers are defined by all expressions in the window’s ORDER BY. Adding a unique identifier to a rank window eliminates score ties, even if the developer only intended to make display order deterministic. Use separate window definitions when row_number needs an identifier but rank should retain shared places. Apply an outer ORDER BY to present the result consistently without redefining the ranking peers.

Partition and filter deliberately

PARTITION BY restarts numbering for each group. Omitting it creates one partition over the rows visible to the window. This is distinct from a window frame: ranking is based on partition ordering and peers, not on a moving aggregate frame. A ROWS boundary does not turn rank into a count of rows inside that boundary.

Windows see the rows left after the query’s FROM, WHERE, grouping, and HAVING stages. Filtering in the same query can therefore change the competitors before they are ranked. To rank everyone and then keep the first positions, calculate the window values in a subquery or CTE and filter in the outer query. PostgreSQL does not permit a window function directly in WHERE.

“Top two” needs a business definition. Position at most two returns at most two rows per partition. Rank at most two includes everyone tied at second place and can return more rows. Dense rank at most two selects the first two distinct score groups, which can differ from competition rank when the leading score itself has several peers. These are different result sets, not interchangeable optimizations.

When it breaks

A leaderboard may fluctuate among equal scores when row_number lacks a deterministic tie-breaker. Conversely, adding an identifier everywhere can silently remove the ties that rank was meant to preserve. Test tied values at the cutoff, not only unique scores. If scores can be NULL, decide their placement explicitly; PostgreSQL descending order puts nulls first unless NULLS LAST is specified.

Deterministic ordering also does not freeze a changing dataset. New scores can move existing positions between requests, so offset pages or stored row numbers are not durable identities. Large rankings can require sorting and temporary files; multiple window orders may require different processing. Read representative plans rather than assuming that every window query performs one inexpensive pass.

Check it quickly

Save this as ranking.sql and run psql -X -v ON_ERROR_STOP=1 -f ranking.sql in a disposable PostgreSQL 18 database. The fixture contains a tie in red and an independent blue partition, and rolls back its temporary table and view.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE scores (id integer, team text, score integer);
INSERT INTO scores VALUES (1,'red',100),(2,'red',90),(3,'red',90),(4,'red',80),(5,'blue',50);
CREATE TEMP VIEW ranked AS
SELECT *, row_number() OVER (PARTITION BY team ORDER BY score DESC,id) AS position,
 rank() OVER w AS rank_with_gaps, dense_rank() OVER w AS rank_without_gaps
FROM scores WINDOW w AS (PARTITION BY team ORDER BY score DESC);
SELECT * FROM ranked ORDER BY team,id;
SELECT count(*) AS top_two_positions FROM ranked WHERE team='red' AND position<=2;
SELECT count(*) AS top_two_ranks FROM ranked WHERE team='red' AND rank_with_gaps<=2;
DO $$ BEGIN
 IF (SELECT array_agg(rank_with_gaps ORDER BY id) FROM ranked WHERE team='red')
 IS DISTINCT FROM ARRAY[1,2,2,4]::bigint[] THEN RAISE EXCEPTION 'wrong rank'; END IF;
 IF (SELECT array_agg(rank_without_gaps ORDER BY id) FROM ranked WHERE team='red')
 IS DISTINCT FROM ARRAY[1,2,2,3]::bigint[] THEN RAISE EXCEPTION 'wrong dense rank'; END IF;
 IF (SELECT array_agg(id ORDER BY position) FROM ranked WHERE team='red' AND position<=2)
 IS DISTINCT FROM ARRAY[1,2] THEN RAISE EXCEPTION 'unstable top two'; END IF;
 IF (SELECT count(*) FROM ranked WHERE team='red' AND rank_with_gaps<=2) <> 3 THEN
 RAISE EXCEPTION 'tied second place was dropped';
 END IF;
END $$;
ROLLBACK;

On PostgreSQL 18.6, red received positions 1 through 4, competition ranks 1, 2, 2, 4, and dense ranks 1, 2, 2, 3. Blue started at one independently. Filtering red by position returned ids 1 and 2; filtering by rank at most two returned three rows, preserving both second-place entrants.

The assertions check rank arrays, the chosen tie-breaker, and the cutoff count. Changing the shared rank window to include id should make these expectations fail, demonstrating that display stability and peer semantics cannot be mixed casually. All observations concern one fixed dataset; use a separate snapshot or pagination design when a report must remain stable while scores change.

Sources

PostgreSQL window functions, ranking functions and peer groups, checked 2026-09-10.
PostgreSQL window tutorial, partitions, ordering, and filtering after ranking, checked 2026-09-10.

Autovacuum keeps write-heavy tables lean

Fri, 04 Feb 2028 00:00:00 +0000

A busy table produces obsolete versions at a rate its maintenance must accommodate. Lowering an autovacuum trigger can make that table eligible sooner, but eligibility is not completion. A worker must still reach the table, acquire the required locks, and finish enough useful cleanup before the backlog grows again. Measure that whole cycle instead of treating one configuration value as a throughput guarantee.

When it breaks

The normal update/delete trigger combines a fixed threshold with a scale factor multiplied by the estimated table population. PostgreSQL 18 also applies a maximum threshold when enabled. Insert-triggered vacuum and age-driven freezing have their own reasons to run. Looking only at one dead-tuple ratio misses those paths, and interpreting every vacuum as proof that the update/delete threshold fired can misdiagnose a table.

Reducing the threshold does not remove old snapshot retention or create more I/O capacity. More workers and less cost delay can increase resource competition with foreground requests. A dead-tuple estimate near zero at every instant is neither a realistic universal target nor sufficient evidence of good performance. The useful target is sustainable cleanup with acceptable growth, latency, and maintenance cost.

Check it quickly

Use a disposable PostgreSQL 18 instance with autovacuum and statistics tracking enabled, available workers, and its normal one-minute database inspection interval. Run the script through a fresh psql -X -v ON_ERROR_STOP=1 connection. It waits up to 130 seconds for an actual automatic run; a timeout is a failed observation, not permission to claim success from a manual vacuum instead.

CREATE SCHEMA churn_demo;
CREATE TABLE churn_demo.jobs (id integer PRIMARY KEY,n integer)
 WITH (autovacuum_enabled=false,
 autovacuum_vacuum_threshold=50,
 autovacuum_vacuum_scale_factor=0.2);
INSERT INTO churn_demo.jobs SELECT g,0 FROM generate_series(1,10000) AS g;
SELECT pg_stat_force_next_flush();
VACUUM (ANALYZE) churn_demo.jobs;
UPDATE churn_demo.jobs SET n=1 WHERE id<=1000;
SELECT pg_stat_force_next_flush();
SELECT n_live_tup,n_dead_tup,vacuum_count,autovacuum_count
FROM pg_stat_user_tables WHERE relid='churn_demo.jobs'::regclass;
SELECT 50+0.2*reltuples AS old_threshold,50+0.01*reltuples AS new_threshold
FROM pg_class WHERE oid='churn_demo.jobs'::regclass;
ALTER TABLE churn_demo.jobs SET
 (autovacuum_enabled=true,autovacuum_vacuum_scale_factor=0.01);
DO $$
DECLARE deadline timestamptz := clock_timestamp()+interval '130 seconds';
BEGIN
 LOOP
 PERFORM pg_stat_clear_snapshot();
 EXIT WHEN (SELECT autovacuum_count>0 FROM pg_stat_user_tables
 WHERE relid='churn_demo.jobs'::regclass);
 IF clock_timestamp()>deadline THEN
 RAISE EXCEPTION 'no automatic vacuum observed within the experiment budget';
 END IF;
 PERFORM pg_sleep(0.25);
 END LOOP;
END $$;
SELECT pg_stat_clear_snapshot();
SELECT n_dead_tup,last_autovacuum IS NOT NULL AS ran,autovacuum_count>0 AS counted
FROM pg_stat_user_tables WHERE relid='churn_demo.jobs'::regclass;
SELECT count(*),count(*) FILTER (WHERE n=1) FROM churn_demo.jobs;
DO $$ BEGIN
 IF (SELECT count(*) FROM churn_demo.jobs)<>10000
 OR (SELECT count(*) FROM churn_demo.jobs WHERE n=1)<>1000 THEN
 RAISE EXCEPTION 'maintenance changed application data';
 END IF;
END $$;
DROP SCHEMA churn_demo CASCADE;

The table initially disables only its own routine autovacuum so setup and the controlled update complete first. Manual VACUUM ANALYZE establishes statistics and resets the insertion backlog before one thousand rows are updated. The explicit statistics-flush calls keep pending write counters from appearing after the manual maintenance estimate; they are laboratory synchronization, not a production tuning recommendation.

Before enabling the worker, expect estimated live and dead counts of ten thousand and one thousand, a manual-vacuum count of one, and an automatic-vacuum count of zero. Exact application counts are checked separately. With the specified fixed threshold of fifty, the old scale factor gives 2,050 and the smaller factor gives 150. The maximum threshold does not bind for this small table.

One thousand obsolete versions exceed the new trigger. The script enables routine autovacuum and lowers only that table’s factor, then waits for its automatic counter to advance. The polling block clears cached statistics before checking, so it does not keep reading one old statistics snapshot. It performs no manual vacuum after enabling the worker.

The final statistics report zero estimated dead tuples, a populated automatic timestamp, and a positive automatic counter. Application data remain ten thousand rows, with exactly one thousand values updated to one. In the tested instance, the observed background run completed within the budget. The exact delay depends on scheduler timing and worker availability, so the script asserts the event and data rather than one elapsed duration.

Diagnose the limiting stage

If no automatic run appears, check effective table options, server autovacuum settings, statistics tracking, worker activity, and conflicting locks. A manual maintenance timestamp cannot stand in for last_autovacuum. If runs complete but obsolete versions remain, investigate long transactions and other visibility horizons. If removable versions accumulate faster than completed work, then examine capacity, cost throttling, and write patterns.

Changing the scale factor for a large hot table is often more targeted than changing every table’s settings. Record the old options, compare several representative workload periods, and watch file growth and application latency along with maintenance counters. Reset or revise the override when the workload changes. The demonstration deletes its entire fixture at the end; a real table needs its chosen settings maintained deliberately, and continued freezing protection even when its update traffic falls quiet.

Sources

PostgreSQL 18 autovacuum configuration, thresholds, scale factors, worker limits, and inspection intervals, checked 2026-09-10.
PostgreSQL 18 routine vacuuming, trigger formulas and distinct maintenance purposes, checked 2026-09-10.
PostgreSQL 18 cumulative statistics, manual versus automatic counters and cached statistics, checked 2026-09-10.
PostgreSQL 18 statistics function implementation, the flush helper used to synchronize fixture observations, checked 2026-09-10.

A surrogate key keeps identity apart from meaning

Wed, 02 Feb 2028 00:00:00 +0000

A surrogate primary key separates row identity from a business value that can change. A customer’s email can be corrected while orders continue referencing the same customer identifier. This is useful when the business name is mutable or wide, but stability comes from schema and application policy. A generated number is not automatically immutable, secret, globally unique, or an authorization decision.

Generate identity and enforce uniqueness

PostgreSQL identity columns obtain default values from an associated sequence. GENERATED ALWAYS AS IDENTITY rejects ordinary explicit values, while GENERATED BY DEFAULT allows them. ALWAYS still permits an explicit OVERRIDING SYSTEM VALUE import. Neither form alone guarantees uniqueness; declare a PRIMARY KEY or UNIQUE constraint when duplicate identifiers must be rejected.

The primary key combines uniqueness with non-nullability and provides an index usable by references. Identity controls generation, while the constraint controls which stored values are legal. Keeping these responsibilities distinct matters during imports: a manually supplied value can collide with future generated values if the sequence is not reconciled. The sequence is not automatically advanced merely because a large identifier was inserted explicitly.

Preserve the business rule separately

A surrogate id does not stop duplicate business entities. If email is the unique login identifier, retain an explicit uniqueness constraint and decide the normalization rule. A plain text UNIQUE constraint follows the column’s comparison semantics; it does not silently implement the application’s desired case folding, whitespace handling, or mailbox rules. Make those decisions before migrating existing data.

Likewise, child tables need actual foreign keys if dangling references must be prevented. Renaming a non-key email leaves those references unchanged because they point to the id. This reduces the scope of that edit, but does not promise that every business rename becomes trivial: search indexes, audit records, external integrations, and cache invalidation may still depend on the changed value.

When it breaks

Treating sequence values as creation or commit order produces incorrect results under concurrency. Transactions can allocate numbers and commit in a different order, and aborted statements can leave gaps. Use an explicit timestamp and a deterministic tie-breaker for display ordering. A sequential identifier is also guessable; authorization must check the requesting principal and resource, even when a separate opaque public identifier is used.

Imports create another trap. Overriding identity generation can insert an id far ahead of the sequence without moving its current position. Eventually a generated insert may hit an existing primary key. Reconcile sequence state as part of a controlled import procedure, with concurrent writers excluded or otherwise coordinated. max(id) in an unrelated session is not a concurrency-safe allocation algorithm.

A migration can also remove the old natural-key constraint by accident. Audit duplicates and nulls before changing keys, retain required business uniqueness, and update every real reference. Do not add a surrogate solely because it is fashionable: a stable natural key may already express the intended contract without another column and index.

Check it quickly

Save this as identity.sql and run psql -X -v ON_ERROR_STOP=1 -f identity.sql in a disposable PostgreSQL 18 database. The experiment uses temporary tables and a final rollback, with no dependency on an existing users table.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE users (
 id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
 email text NOT NULL UNIQUE
);
CREATE TEMP TABLE orders (
 id integer PRIMARY KEY,
 user_id bigint NOT NULL REFERENCES users(id)
);
INSERT INTO users(email) VALUES ('a@example.test') RETURNING id AS owner_id \gset
INSERT INTO orders VALUES (1, :owner_id);
UPDATE users SET email='renamed@example.test' WHERE id=:owner_id;
SELECT u.id, u.email FROM orders o JOIN users u ON u.id=o.user_id;
DO $$ BEGIN
 INSERT INTO users(email) VALUES ('renamed@example.test');
 RAISE EXCEPTION 'duplicate email accepted';
EXCEPTION WHEN unique_violation THEN RAISE NOTICE 'duplicate email rejected';
END $$;
DO $$ BEGIN
 INSERT INTO users(id,email) VALUES (99,'manual@example.test');
 RAISE EXCEPTION 'explicit identity accepted without override';
EXCEPTION WHEN generated_always THEN RAISE NOTICE 'explicit identity rejected';
END $$;
INSERT INTO users(id,email) OVERRIDING SYSTEM VALUE VALUES (100,'import@example.test');
INSERT INTO users(email) VALUES ('next@example.test') RETURNING id;
DO $$ BEGIN
 IF (SELECT email FROM users u JOIN orders o ON u.id=o.user_id)
 IS DISTINCT FROM 'renamed@example.test' THEN
 RAISE EXCEPTION 'rename broke the reference';
 END IF;
 IF (SELECT id FROM users WHERE email='next@example.test') IS DISTINCT FROM 3 THEN
 RAISE EXCEPTION 'unexpected identity sequence position';
 END IF;
END $$;
ROLLBACK;

The order still resolves to user one after the email changes. A duplicate email is rejected, and that failed insert consumes sequence value two. An explicit id without an override is rejected. The controlled override inserts id one hundred, but the following generated row receives id three, demonstrating that the import did not reposition the sequence. Assertions check the surviving reference and exact sequence result for this isolated fixture.

These results were verified on PostgreSQL 18.6. The final rollback removes the test objects; it does not establish gap-free sequence behavior for ordinary application transactions. The example demonstrates why generation, primary-key enforcement, foreign-key integrity, business uniqueness, and access control need separate reasoning even when they all involve one identifier.

Sources

PostgreSQL identity columns, generation modes and uniqueness limits, checked 2026-09-10.
PostgreSQL constraints, primary keys and referential integrity, checked 2026-09-10.
PostgreSQL sequence functions, nontransactional allocation and gaps, checked 2026-09-10.

A slower query needs both a plan and a wait check

Mon, 31 Jan 2028 00:00:00 +0000

A formerly fast query can become slow with the same data, statistics, and access plan. Another transaction may hold a lock, the storage cache may be colder, or the application may wait before PostgreSQL even receives the statement. Rechecking the plan is useful, but it belongs inside a broader comparison of where the elapsed time went. Begin by distinguishing time spent doing work from time spent waiting to do it.

Keep a comparable baseline

Record the statement shape, representative parameter values, returned row count, relevant session settings, and whether the application uses a prepared statement. Different parameters can produce very different populations. A manual query with a literal may not reproduce a prepared statement’s generic plan. A different role or search path can also change which objects and permissions apply.

Inspect current activity while the slowdown is happening. pg_stat_activity can show an active query with a lock wait; active does not mean continuously consuming CPU. pg_blocking_pids identifies sessions currently blocking that backend. Visibility into other roles’ activity requires appropriate privileges, so missing details are not evidence that nothing is running. Use fresh autocommit observations rather than repeatedly reading a cached monitoring snapshot inside one long transaction.

When it breaks

Blindly running ANALYZE treats every slowdown as a stale-statistics problem. That can add maintenance work while leaving the actual blocker untouched. Likewise, building an index on a one-row table cannot release another transaction’s row lock. Accurate estimated rows and a tiny sequential scan can coexist with arbitrarily long application waits.

EXPLAIN ANALYZE is an execution, including for writes, and can itself block. A broad update copied from production is not a harmless inspection command. First inspect waiting and lock ownership; measure a safe fixture or a statement whose effects are understood. Do not terminate an unfamiliar transaction merely because its identifier appears in the blocker list.

Check it quickly

Open three fresh psql sessions, A, B, and C, using the same role and disposable PostgreSQL 18 database with normal autocommit enabled. The named schema must not already exist. Keep the session roles separate; pasting every block into one connection cannot demonstrate blocking between transactions.

In A, create the fixture, then begin a transaction that changes its only row without committing.

\set ON_ERROR_STOP on
CREATE SCHEMA query_recheck_demo;
CREATE TABLE query_recheck_demo.items (id integer PRIMARY KEY,value integer);
INSERT INTO query_recheck_demo.items VALUES (1,0);
ANALYZE query_recheck_demo.items;

BEGIN;
UPDATE query_recheck_demo.items SET value=10 WHERE id=1;
SELECT pg_backend_pid() AS blocker_pid;

In B, start the measured update. It waits until A ends its transaction or the sixty-second statement timeout expires.

SET application_name='xbcatm_recheck_waiter';
SET statement_timeout='60s';
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
UPDATE query_recheck_demo.items SET value=value+1 WHERE id=1;

While B is waiting, run this in C, before the timeout. The test application name identifies B without exposing its complete query text.

SELECT pid,state,wait_event_type,wait_event,pg_blocking_pids(pid) AS blockers
FROM pg_stat_activity WHERE application_name='xbcatm_recheck_waiter';

In A, undo the uncommitted change and release its lock. Then collect B’s completed plan.

ROLLBACK;

After B finishes successfully, use C to verify the stored value and remove the fixture.

SELECT value FROM query_recheck_demo.items WHERE id=1;
DROP SCHEMA query_recheck_demo CASCADE;

The PostgreSQL 18.6 observation showed B as active with wait_event_type=Lock, wait_event=transactionid, and A’s process identifier in blockers. The wait-event detail can depend on the exact contention; the blocking relationship is the central evidence. After A rolled back, B incremented the original zero to one. The final query returned one, and cleanup removed the schema.

B’s scan estimated and read one row. Its top Update node returned zero rows because there was no RETURNING clause, not because it changed nothing. Execution time includes the induced wait and depends on how quickly A releases it. If the timeout occurs first, roll back A and use C’s final block to remove the fixture before restarting. The canceled update did not complete its increment.

Follow the observed cause

For a lock wait, investigate transaction lifetime and ownership before changing access paths. For excessive processed rows, compare selectivity and current statistics. For sorts or hashes that spill, inspect their memory and temporary I/O. Finally, compare database execution with application latency to expose connection or client delays. A new plan is meaningful only when it explains the measured regression and preserves the required result.

Sources

PostgreSQL 18: Monitoring database activity, active sessions and wait observations, checked 2026-09-10.
PostgreSQL 18: Using EXPLAIN, row counts and loops and plan interpretation, checked 2026-09-10.

Postgres uses two cache layers

Thu, 27 Jan 2028 00:00:00 +0000

For ordinary buffered relation I/O, PostgreSQL’s shared buffers sit above the operating system’s file cache. A shared-buffer hit avoids another read through that lower layer; a shared-buffer miss can still be served from memory by the operating system. The same file data can occupy both caches, so this is not a promise that PostgreSQL and the kernel share one physical copy.

When it breaks

An EXPLAIN buffer count labelled shared read means the block was brought into PostgreSQL’s buffer pool. It does not establish that a storage device performed a physical read. Treating every shared read as a disk miss can make a warm operating-system cache look cold and lead to the wrong memory-sizing decision. Likewise, a faster second query does not identify which cache changed by itself.

Cache residency is temporary. Other queries can evict pages, and a scan larger than the available cache cannot keep every page warm indefinitely. A relation may use a scan strategy that changes buffer reuse. Test with a known working set, inspect the access path, and keep timing claims separate from the cache-layer facts actually observed.

Check it quickly

Use a disposable PostgreSQL 18 instance as a superuser, with the supplied pg_buffercache and pg_prewarm extensions available but not already installed. The tested instance used 128 MB of shared buffers. The relation below is small enough to fit comfortably. Run blocks in one fresh psql -X -v ON_ERROR_STOP=1 session. The eviction function is for developer testing; restrict it to this isolated fixture.

CREATE SCHEMA cache_layer_demo;
CREATE EXTENSION pg_buffercache WITH SCHEMA cache_layer_demo;
CREATE EXTENSION pg_prewarm WITH SCHEMA cache_layer_demo;
CREATE TABLE cache_layer_demo.items (id integer PRIMARY KEY,payload text)
 WITH (autovacuum_enabled=false);
INSERT INTO cache_layer_demo.items
SELECT g,repeat(md5(g::text),10) FROM generate_series(1,12000) AS g;
VACUUM (ANALYZE) cache_layer_demo.items;
CREATE VIEW cache_layer_demo.residency AS
SELECT count(*) AS heap_buffers FROM cache_layer_demo.pg_buffercache
WHERE reldatabase=(SELECT oid FROM pg_database WHERE datname=current_database())
 AND relfilenode=pg_relation_filenode('cache_layer_demo.items')
 AND relforknumber=0;
SET max_parallel_workers_per_gather=0;
SELECT pg_relation_size('cache_layer_demo.items')/current_setting('block_size')::int AS heap_blocks;
SELECT heap_buffers>0 AS initially_buffered FROM cache_layer_demo.residency;

The heap contains 546 blocks in the tested build, and the initial residency comparison is true. The view filters by database, current file identifier, and main fork, so it does not count another relation’s buffers or the fixture’s auxiliary forks. Next, evict only this relation’s shared buffers:

SELECT * FROM cache_layer_demo.pg_buffercache_evict_relation('cache_layer_demo.items'::regclass);
SELECT heap_buffers=0 AS no_shared_heap_buffers FROM cache_layer_demo.residency;

The function reports evicted, flushed, and skipped buffers. This run returned 550, 550, and zero; the total includes auxiliary forks, explaining why it exceeds the heap-only count. The following comparison confirms no heap buffers remain. It says nothing about whether the operating system still caches those file pages.

Read the heap through the operating system without loading PostgreSQL shared buffers:

SELECT cache_layer_demo.pg_prewarm('cache_layer_demo.items','read','main');
SELECT heap_buffers=0 AS still_no_shared_heap_buffers FROM cache_layer_demo.residency;

The prewarm call processes 546 blocks, and the shared-buffer count remains zero. Its read mode performs synchronous reads; buffer mode would instead load the database buffer pool. Now execute a query that needs every payload:

EXPLAIN (ANALYZE,BUFFERS,COSTS OFF,TIMING OFF,SUMMARY OFF,FORMAT JSON)
SELECT sum(length(payload)) FROM cache_layer_demo.items;

The sequential scan reports 546 shared reads and zero shared hits. Those reads are compatible with an already warm operating-system cache; they are not evidence of 546 device operations. Repeat the same query:

EXPLAIN (ANALYZE,BUFFERS,COSTS OFF,TIMING OFF,SUMMARY OFF,FORMAT JSON)
SELECT sum(length(payload)) FROM cache_layer_demo.items;

The second scan reports zero shared reads and 546 shared hits. Both scans process twelve thousand rows, so the change in buffer accounting is not a reduction in the logical input. Finally, confirm residency and the aggregate result, then remove the fixture and its extensions:

SELECT heap_buffers>0 AS now_shared FROM cache_layer_demo.residency;
SELECT sum(length(payload)) FROM cache_layer_demo.items;
RESET max_parallel_workers_per_gather;
DROP SCHEMA cache_layer_demo CASCADE;

The result is 3,840,000 payload characters and a true shared-residency comparison. Exact block counts depend on row layout and build parameters. The meaningful observations are the empty shared pool after read-mode prewarming, the subsequent shared reads, and the repeated scan’s hits. They do not measure physical-device latency or guarantee a particular speedup.

Budget memory for the complete server

The PostgreSQL resource guide suggests roughly a quarter of system memory as an initial shared-buffer setting for a dedicated server with sufficient RAM. That is a starting point, not a universal optimum. Connection memory, concurrent sorts and hashes, maintenance, and the operating-system cache all consume the same host budget.

Measure representative working sets and concurrent workloads before changing the allocation. Warm-cache microbenchmarks can hide I/O pressure, while cold-cache tests can exaggerate a workload that normally reuses pages. Keep cache-layer counters, actual device observations, and query results together so a configuration change is credited for the effect it actually produced.

Sources

PostgreSQL 18 resource consumption, shared buffers and operating-system cache budgeting, checked 2026-09-10.
PostgreSQL 18 pg_prewarm, read versus buffer prewarming modes, checked 2026-09-10.
PostgreSQL 18 pg_buffercache, buffer identity and development-only relation eviction, checked 2026-09-10.

Hash indexes recheck collisions and maintain changed keys

Mon, 24 Jan 2028 00:00:00 +0000

PostgreSQL hash indexes narrow an equality search using a hash code, then verify the original value in the table. A hash match is a candidate, not proof of equality. Two different values can have the same code, and different codes can also map to the same bucket. Neither collisions nor shared buckets imply incorrect query results when the normal recheck is performed.

What lives in a bucket

An index tuple contains a four-byte hash value and a reference to the table tuple; it does not contain the full indexed value. The bucket mapping directs a lookup to relevant index pages. Buckets are shared storage, not one separately allocated bucket for every distinct key. An overflowing bucket can acquire additional pages that its scans must visit.

PostgreSQL can sort hash codes within an individual index page to accelerate local searching, but that does not provide ordering by the original column. A hash index cannot satisfy a key range or an ordered result. It supports single-column equality access, cannot enforce uniqueness, and does not support IS NULL. A B-tree is still a candidate for equality as well as the broader operations it supports.

The absence of original values makes hash scans lossy: matching index entries must be checked against table values. A collision can therefore appear as Rows Removed by Index Recheck in an execution plan. That is evidence that candidate verification worked, not an indication that the database returned both colliding values as equal.

When it breaks

Describing an indexed update as touching only one bucket hides important work. Changing the indexed key changes the tuple’s indexed value and may send its new entry to another bucket. PostgreSQL’s versioned rows and deferred index cleanup mean obsolete references do not necessarily disappear immediately. Growth can also split a bucket, and overflow pages add I/O.

Consequently, a workload with heavily repeated values or rapid growth can behave differently from a nearly unique, steady equality workload. Hash lookup is not an unconditional constant-time latency guarantee. Modern PostgreSQL hash indexes are crash recoverable, but durability does not eliminate storage maintenance or turn the structure into a uniqueness constraint.

Check it quickly

Run this with psql on a disposable PostgreSQL 18 database. It creates two hundred thousand temporary text keys using C collation, discovers a collision with the server’s own hashtext, and tests lookup and update behavior. It does not assume hash values are portable identifiers. Rollback removes the fixture.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE hash_items (key text COLLATE "C");
INSERT INTO hash_items SELECT 'key-'||g FROM generate_series(1,200000) AS g;
CREATE TEMP TABLE collision_pair AS
SELECT min(key) AS a,max(key) AS b FROM hash_items
GROUP BY hashtext(key) HAVING count(*)>1 ORDER BY hashtext(key) LIMIT 1;
SELECT a AS collision_a,b AS collision_b FROM collision_pair \gset
SELECT a,b,hashtext(a),hashtext(b) FROM collision_pair;
CREATE INDEX hash_items_key ON hash_items USING hash(key);
ANALYZE hash_items;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT key FROM hash_items WHERE key=:'collision_a';
UPDATE hash_items SET key='changed-'||key WHERE key=:'collision_a';
SELECT count(*) AS old_key_rows FROM hash_items WHERE key=:'collision_a';
SELECT count(*) AS new_key_rows FROM hash_items WHERE key='changed-'||:'collision_a';
DO $$ BEGIN
 IF (SELECT hashtext(a)=hashtext(b) AND a<>b FROM collision_pair) IS DISTINCT FROM true THEN
 RAISE EXCEPTION 'fixture did not establish a collision';
 END IF;
 IF (SELECT count(*) FROM hash_items h JOIN collision_pair p ON h.key=p.a) <> 0 OR
 (SELECT count(*) FROM hash_items h JOIN collision_pair p ON h.key='changed-'||p.a) <> 1 OR
 (SELECT count(*) FROM hash_items h JOIN collision_pair p ON h.key=p.b) <> 1 THEN
 RAISE EXCEPTION 'hash lookup or indexed update returned wrong keys';
 END IF;
END $$;
ROLLBACK;

On PostgreSQL 18.6, the discovered keys were key-129539 and key-7295, both with hash code -1960436116. The equality plan used hash_items_key, returned one row, and reported one row removed by index recheck. The two strings remained distinct despite their identical hash code. This is a concrete negative control against treating hashes as collision-free keys.

Updating the first key to its changed- form left zero rows under its old value and one under its new value. The other colliding key still had one row. The final assertions verify all three facts and verify that the original pair really collided. A successful update alone would not establish that unrelated colliding entries remained searchable.

The particular pair and plan are observations from this version. If a different environment does not discover a pair, the script must stop rather than claiming a collision was tested. Even when the pair changes, the required semantics stay the same: exact equality returns only matching original values, and updating one value does not change another merely because their hashes coincide.

Evaluate maintenance with lookup performance

Measure page accesses, equality latency, update behavior, and index size using representative distributions. VACUUM can remove dead references and recycle overflow pages, but it does not shrink the hash index’s bucket count or return its allocation to the operating system. Rebuilding is a separate maintenance operation with its own cost. A smaller hash entry can be useful, yet only a full workload comparison establishes whether that benefit exceeds recheck, overflow, and write overhead.

Sources

PostgreSQL 18: Hash indexes, collision rechecks and bucket maintenance, checked 2026-09-10.
PostgreSQL 18: Index types, supported predicates for B-tree and hash indexes, checked 2026-09-10.

LATERAL lets each row open its own subquery

Wed, 19 Jan 2028 00:00:00 +0000

LATERAL lets a FROM item refer to columns from preceding items. It is useful when each parent needs a small, differently parameterized result, such as its two newest orders. The query remains one SQL statement and one database round trip, while the lateral expression states which outer values determine each result. It does not promise one particular physical execution strategy.

Keep the correlation visible

A subquery in FROM normally cannot reference a sibling table’s columns. LATERAL permits that reference on the appropriate side of the join. A condition such as o.customer_id = c.id is the connection between the current customer and its orders; the keyword alone does not create that condition. Without correlation, the same global result can be repeated for every parent.

For top-N lookups, place the parent’s filter, ORDER BY, and LIMIT inside the lateral subquery. A LIMIT outside the join restricts the final combined result, not the number of orders per customer. Add a unique tie-breaker to the ordering: two orders can share the same timestamp, and ordering only by that timestamp leaves their relative choice unspecified.

The subquery’s ORDER BY chooses its top rows, but does not guarantee final presentation order. Apply an outer ORDER BY when the client requires a stable sequence of customers and orders. This distinction matters for exported reports and tests, where relying on today’s apparent plan order creates intermittent changes after an upgrade or statistics refresh.

Decide what an empty result means

CROSS JOIN LATERAL drops a parent when its subquery returns no rows. LEFT JOIN LATERAL with ON true preserves that parent, filling the right-hand columns with nulls. This is often the required behavior for customers who have not placed an order. A right-side condition moved into the outer WHERE can reject that null-extended row and undo the intended preservation.

Aggregation adds another distinction. COUNT(r.id) counts non-null matched order identifiers; COUNT(*) also counts a left join’s synthetic row. A correlated aggregate without GROUP BY can itself return one row for an empty input, with COUNT zero or SUM null. Read the returned row shape rather than treating every lateral expression as an identical list lookup.

When it breaks

Removing the outer reference to make a syntax error disappear changes the query’s meaning. Every customer may receive the same newest orders, and inflated totals can still look plausible. Test parents with no matches, one match, several matches, and tied ordering values. An assertion about exact identifiers is stronger than checking that the query ran successfully.

Performance can also deteriorate when an expensive inner operation repeats for many outer rows. An index beginning with the correlation key and continuing with the requested ordering can support selective top-N lookups. It is not guaranteed to be chosen, especially for tiny tables. Read EXPLAIN’s actual loops and rows, and compare a window-function formulation when processing a large fraction of the child table. LATERAL removes client round trips; it does not erase database work.

Check it quickly

Save the following as lateral.sql and run psql -X -v ON_ERROR_STOP=1 -f lateral.sql against a disposable PostgreSQL 18 database. It creates three customers, four orders, a matching index, and a temporary view inside one transaction.

\set ON_ERROR_STOP on
BEGIN;
SET LOCAL statement_timeout = '5s';
CREATE TEMP TABLE customers (id integer PRIMARY KEY, name text);
CREATE TEMP TABLE orders (id integer PRIMARY KEY, customer_id integer, placed_at date);
INSERT INTO customers VALUES (1,'Ada'), (2,'Lin'), (3,'Sam');
INSERT INTO orders VALUES
 (10,1,'2026-09-01'), (11,1,'2026-09-02'), (12,1,'2026-09-02'),
 (20,2,'2026-09-01');
CREATE INDEX orders_recent ON orders (customer_id, placed_at DESC, id DESC);
ANALYZE customers;
ANALYZE orders;
CREATE TEMP VIEW recent AS
SELECT c.id AS customer_id, r.id AS order_id
FROM customers c LEFT JOIN LATERAL (
 SELECT o.id FROM orders o WHERE o.customer_id=c.id
 ORDER BY o.placed_at DESC, o.id DESC LIMIT 2
) r ON true;
SELECT * FROM recent ORDER BY customer_id, order_id DESC;
SELECT c.id, count(r.id) AS selected_orders
FROM customers c LEFT JOIN LATERAL (
 SELECT o.id FROM orders o WHERE o.customer_id=c.id
 ORDER BY o.placed_at DESC, o.id DESC LIMIT 2
) r ON true GROUP BY c.id ORDER BY c.id;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT * FROM recent;
DO $$ BEGIN
 IF (SELECT array_agg(order_id ORDER BY order_id) FROM recent WHERE customer_id=1)
 IS DISTINCT FROM ARRAY[11,12] THEN
 RAISE EXCEPTION 'top two or tie-breaker is wrong';
 END IF;
 IF NOT EXISTS (SELECT FROM recent WHERE customer_id=3 AND order_id IS NULL) THEN
 RAISE EXCEPTION 'LEFT JOIN lost the customer without orders';
 END IF;
 IF (SELECT count(*) FROM recent) <> 4 THEN RAISE EXCEPTION 'wrong row count'; END IF;
END $$;
ROLLBACK;

The result contains (1,12), (1,11), (2,20), and customer three with a null order. Counts of matched orders are two, one, and zero. The tied date for orders eleven and twelve exercises the identifier tie-breaker; customer three exercises empty-result preservation. Assertions verify the identifiers and total row count, and the final rollback removes all objects.

PostgreSQL 18.6 used repeated small scans for this fixture, which is reasonable for four orders. That observation does not prove the proposed index is ineffective at scale. Test production-shaped parent counts and child distributions before drawing a performance conclusion. The five-second statement timeout bounds the experiment, while correctness depends on correlation, join type, and deterministic ordering.

Sources

PostgreSQL table expressions, lateral references and outer join semantics, checked 2026-09-10.
PostgreSQL SELECT, ordering and limit behavior, checked 2026-09-10.

pg_dump captures a consistent snapshot

Fri, 14 Jan 2028 00:00:00 +0000

pg_dump exports one PostgreSQL database using a consistent view of transactional data, even while other sessions commit changes. This prevents the ordinary table-by-table copy problem in which related tables reflect different committed states. The snapshot is only part of the backup contract: the archive must also be complete for its intended scope, readable by compatible tools, and successfully restored into a usable database.

Know what the snapshot covers

Rows visible to the dump’s snapshot remain its view throughout the export. Later commits do not gradually enter the dump just because one table is read after another. Parallel dumps synchronize workers’ snapshots rather than independently observing each table. They require directory format; custom format supports parallel restoration, but not parallel dumping.

A database snapshot does not make every business workflow atomic. If an application commits a batch header and its detail rows in separate transactions, a consistent snapshot can legitimately fall between those commits. Sequence state is also not ordinary MVCC row data, and external files or services are outside the database snapshot. Define the recovery invariant around actual transaction boundaries and all required data stores.

Account for load and dependencies

Dumping normally allows ordinary readers and writers to continue, but it still takes ACCESS SHARE locks and consumes CPU, I/O, connections, and snapshot lifetime. Conflicting schema changes can wait or make the dump fail. Long snapshots can prevent cleanup of old row versions. Parallel workers increase load; they do not make backup overhead disappear.

A standby can offload some reads, but recovery conflicts and lag need consideration, and unlogged table contents are excluded when dumping from a standby. One pg_dump does not include cluster-wide roles and tablespaces. Extensions, ownership, permissions, collation behavior, and target configuration must be included in the restore plan through their appropriate mechanisms. Splitting an export into unrelated table dumps can lose the common snapshot and dependency closure.

When it breaks

An archive listing can look healthy while restoration fails on a missing role or extension. Restore into an empty, isolated target with explicit error handling, then validate application invariants and representative queries. pg_restore normally continues after SQL errors and reports an error count; checking only the last printed query is inadequate. A single-transaction restore can provide atomicity for a small rehearsal, but cannot be combined with parallel jobs.

Tool versions are another boundary. A pg_dump client refuses a server newer than its own major version. Output is intended for the same or newer target versions; loading it into an older major version is not guaranteed. Verify the actual client, source, target, extensions, and restore options rather than relying on the generic claim that “Postgres backups are portable.”

Logical exports can meet an explicitly measured recovery objective, but they do not provide arbitrary point-in-time replay between dumps. Physical base backups plus a complete retained WAL chain serve that requirement. The acceptable interval follows the required data-loss window and restore time, not a universal rule that anything shorter than a day requires one specific schedule.

Check it quickly

Save the Bash block as dump.sh. It creates its own PostgreSQL cluster and removes it on exit. Run it with PostgreSQL 18 tools as a non-root user, or use this isolated container with no published port or host-data mount:

docker run --rm --network none --user postgres --entrypoint bash -i \
 postgres:18-alpine < dump.sh

#!/usr/bin/env bash
set -euo pipefail
dump_lab=$(mktemp -d /tmp/pgdump-lab.XXXXXX)
cleanup() {
 if [ -f "$dump_lab/data/postmaster.pid" ]; then
 pg_ctl -D "$dump_lab/data" -m fast -w stop >/dev/null
 fi
 rm -rf -- "$dump_lab"
}
trap cleanup EXIT
initdb -D "$dump_lab/data" --no-locale --encoding=UTF8 --auth=trust >/dev/null
pg_ctl -D "$dump_lab/data" -l "$dump_lab/server.log" \
 -o "-k $dump_lab -p 55432 -c listen_addresses=''" -w start >/dev/null
export PGHOST="$dump_lab" PGPORT=55432 PGDATABASE=source_db
export DUMP_FILE="$dump_lab/source.dump" DUMP_OK="$dump_lab/dump-ok"
createdb source_db
createdb restored_db
psql -X -v ON_ERROR_STOP=1 <<'SQL'
CREATE TABLE customers (id integer PRIMARY KEY, name text);
CREATE TABLE orders (id integer PRIMARY KEY, customer_id integer REFERENCES customers(id));
INSERT INTO customers VALUES (1,'before');
INSERT INTO orders VALUES (10,1);
BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY;
SELECT pg_export_snapshot() AS snapshot_id \gset
\setenv DUMP_SNAPSHOT :snapshot_id
\! psql -X -v ON_ERROR_STOP=1 -c "BEGIN; INSERT INTO customers VALUES (2,'after'); INSERT INTO orders VALUES (20,2); COMMIT;" && pg_dump --format=custom --snapshot="$DUMP_SNAPSHOT" --file="$DUMP_FILE" && touch "$DUMP_OK"
COMMIT;
SQL
test -f "$DUMP_OK"
pg_restore --list "$DUMP_FILE" > "$dump_lab/archive-list"
pg_restore --exit-on-error --single-transaction --dbname=restored_db "$DUMP_FILE"
counts='SELECT (SELECT count(*) FROM customers), (SELECT count(*) FROM orders)'
test "$(psql -X -Atc "$counts")" = '2|2'
test "$(psql -X -d restored_db -Atc "$counts")" = '1|1'
test "$(psql -X -d restored_db -Atc 'SELECT count(*) FROM orders o LEFT JOIN customers c ON c.id=o.customer_id WHERE c.id IS NULL')" = 0
psql -X -d restored_db -v ON_ERROR_STOP=1 -c \
 'SELECT c.name,o.id FROM customers c JOIN orders o ON o.customer_id=c.id'
pg_dump --version
printf 'PASS: source=2|2 restored=1|1 orphan_orders=0\n'

The first session exports a snapshot containing one customer and one order. While that transaction remains open, a second session commits another related pair. The dump explicitly imports the earlier snapshot; restoration therefore produces 1|1, while the live source reports 2|2. The join returns customer “before” with order ten, and an orphan check returns zero.

On PostgreSQL 18.6, the complete restore and all assertions pass. The success marker ensures a failed nested shell dump cannot masquerade as success merely because psql continued. This demonstrates snapshot selection and restoration for the fixture, not a measured recovery time for the real application. Keep actual restore rehearsals, artifact checksums, retention, and dependency records with the production backup procedure.

Sources

PostgreSQL pg_dump, snapshots, formats, scope and version compatibility, checked 2026-09-10.
PostgreSQL pg_restore, restore error handling and transaction options, checked 2026-09-10.

NOW and clock_timestamp differ by when they freeze

Thu, 13 Jan 2028 00:00:00 +0000

PostgreSQL offers several meanings of “current time.” A transaction timestamp identifies the beginning of a transaction, a statement timestamp identifies a command’s arrival, and a wall-clock reading samples the current clock during evaluation. Choosing among them changes the meaning of audit fields, expiration checks, and observations made during long-running work.

A transaction has one start time

now(), transaction_timestamp(), and CURRENT_TIMESTAMP without a precision argument return the same transaction-start instant. They remain fixed throughout an explicit transaction, even if later statements run seconds or minutes afterward. CURRENT_TIMESTAMP can take a precision argument that rounds its result; that formatting precision is separate from when the time is sampled.

This stability is useful when related rows should share one logical transaction time. It does not record the exact moment each row was inserted, nor the eventual commit time. A transaction can wait for locks or perform substantial work before an insert, and it can still roll back. Calling a column created_at does not resolve which of these events its default actually represents.

In ordinary autocommit use, successive statements normally run in separate transactions, so now can appear to advance on every statement. That observation does not change the function’s contract. Introducing an explicit transaction later can expose an assumption that a supposedly live timestamp was actually fixed much earlier.

Statement markers and live readings

statement_timestamp() reports the receipt time of the latest command message from the client. It remains stable during that command’s execution and can differ from the transaction timestamp in later commands. Multiple SQL commands sent in one protocol message can share this marker; it is not necessarily the instant each internal operation starts after waiting.

clock_timestamp() reads the current wall clock when evaluated and can change within a single statement. Separate calls are not guaranteed to be distinct, because clocks have finite precision. The value also is not a monotonic timer: clock adjustments can affect differences between readings. Use a suitable monotonic clock for elapsed-duration guarantees rather than treating database wall time as an unconditionally increasing counter.

None of these values alone establishes commit order or a unique event identity across concurrent transactions. Choose a transaction marker for consistent related fields, a statement marker for command timing, or a wall-clock observation when that observation is what the application needs. Keep identity and ordering requirements explicit instead of expecting a timestamp to provide them automatically.

When it breaks

An expiration test using now inside a long transaction continues comparing against that transaction’s start. A condition that was true at the beginning can remain true after wall time has passed the deadline. Conversely, replacing every default with clock_timestamp can give related rows different values when they were meant to share a logical creation time. Decide which event the column represents before changing functions.

The special literal TIMESTAMPTZ 'now' is another trap in a column default. PostgreSQL converts it when the constant is parsed, so later inserts can keep receiving the table-creation time. DEFAULT now() is a function call evaluated when the default is used. Similar-looking spelling does not imply the same evaluation lifetime.

These functions return timestamptz values. Changing the session time zone changes how an instant is displayed, not the stored instant itself. Timestamptz does not preserve the originally supplied zone name. If the application must remember that original zone, model it separately rather than inferring it from a later display offset.

Check it quickly

Save this as timestamps.sql and run psql -X -v ON_ERROR_STOP=1 -f timestamps.sql in a disposable PostgreSQL 18 database. Keep psql’s normal autocommit behavior outside the explicit transaction. The second experiment needs separate insert transactions, and all tables are temporary.

\set ON_ERROR_STOP on
BEGIN;
SET LOCAL timezone='UTC';
CREATE TEMP TABLE moments (step integer,tx timestamptz,statement timestamptz,wall timestamptz);
INSERT INTO moments VALUES (1,now(),statement_timestamp(),clock_timestamp());
SELECT pg_sleep(0.03);
INSERT INTO moments VALUES (2,CURRENT_TIMESTAMP,statement_timestamp(),clock_timestamp());
SELECT step,tx,statement,wall FROM moments ORDER BY step;
SELECT min(tx)=max(tx) AS one_transaction_time,
 count(DISTINCT statement)=2 AS two_statement_times FROM moments;
SELECT now()=transaction_timestamp() AND now()=CURRENT_TIMESTAMP AS aliases;
SELECT extract(epoch FROM now()) AS instant_utc;
SET LOCAL timezone='Asia/Shanghai';
SELECT now() AS same_instant_different_display,extract(epoch FROM now()) AS instant_shanghai;
DO $$ BEGIN
 IF (SELECT min(tx)=max(tx) FROM moments) IS DISTINCT FROM true THEN
 RAISE EXCEPTION 'transaction time changed';
 END IF;
 IF now() IS DISTINCT FROM transaction_timestamp() THEN RAISE EXCEPTION 'aliases differ'; END IF;
 IF (SELECT tx FROM moments WHERE step=1) IS DISTINCT FROM now() THEN
 RAISE EXCEPTION 'time zone changed the instant';
 END IF;
END $$;
ROLLBACK;
CREATE TEMP TABLE defaults_demo (
 id integer,
 parsed timestamptz DEFAULT TIMESTAMPTZ 'now',
 called timestamptz DEFAULT now()
);
INSERT INTO defaults_demo (id) VALUES (1);
SELECT pg_sleep(0.03);
INSERT INTO defaults_demo (id) VALUES (2);
SELECT id,parsed,called FROM defaults_demo ORDER BY id;
SELECT min(parsed)=max(parsed) AS one_parsed_constant,
 count(DISTINCT called)=2 AS two_insert_transaction_times FROM defaults_demo;
DO $$ BEGIN
 IF (SELECT min(parsed)=max(parsed) FROM defaults_demo) IS DISTINCT FROM true THEN
 RAISE EXCEPTION 'parsed default was evaluated again';
 END IF;
END $$;
DROP TABLE defaults_demo;

On PostgreSQL 18.6, the first two rows shared tx while their statement and wall readings differed. Alias comparison returned true. Switching from UTC to Asia/Shanghai changed the visible offset to plus eight hours, while the two epoch values remained identical. Assertions check transaction stability and the unchanged instant.

After rollback, the default experiment inserted two rows in separate transactions. Both retained the same parsed literal, while called advanced between inserts. The short sleeps make the difference observable in this run, not a clock-resolution or scheduling guarantee. The final DROP removes the second temporary table; session termination also removes temporary objects if an assertion stops execution early.

Sources

PostgreSQL date/time functions, transaction, statement, wall-clock, and default evaluation semantics, checked 2026-09-10.
PostgreSQL date/time types, timestamptz storage, display, and special input literals, checked 2026-09-10.

A checkpoint flushes dirty buffers to disk

Thu, 06 Jan 2028 00:00:00 +0000

A checkpoint establishes a starting point for PostgreSQL crash recovery. It synchronizes the data-file changes required for that checkpoint and records the corresponding recovery information. WAL must reach durable storage before dependent data pages are written. This ordering allows committed work to survive without forcing every modified table page to disk at each commit, while recovery can replay later changes when necessary.

When it breaks

A checkpoint is not a transaction commit. Data pages can contain versions written by transactions that remain uncommitted; visibility and transaction status still determine which rows readers can see. Checkpoint completion does not make those writes visible or turn a raw copy of changing database files into a valid base backup. Concurrent activity can also dirty pages again, so completion does not mean the buffer pool stays completely clean.

WAL retention has other constraints. max_wal_size is a soft target associated with automatic checkpoint scheduling, not a hard disk-usage limit. Archive failure and replication-slot retention can keep WAL beyond it. min_wal_size concerns recycling a reserve of WAL files, not a second threshold that forces a checkpoint. Misreading those settings can hide a growing archive backlog behind apparently reasonable checkpoint intervals.

Check it quickly

Use two fresh psql -X -v ON_ERROR_STOP=1 sessions, A and B, on one quiet, disposable PostgreSQL 18 instance. Use an administrative role for the explicit checkpoint and control-file information. Keep the normal fsync and synchronous-commit settings enabled. In B, create a counter and save the current checkpoint location:

SHOW fsync;
SHOW synchronous_commit;
CREATE SCHEMA checkpoint_demo;
CREATE TABLE checkpoint_demo.items (id integer PRIMARY KEY,n integer);
INSERT INTO checkpoint_demo.items VALUES (1,10);
CREATE TABLE checkpoint_demo.baseline AS
SELECT checkpoint_lsn FROM pg_control_checkpoint();

Both displayed settings should be on. The initial committed counter is ten. In A, change it inside a transaction that remains open:

BEGIN;
UPDATE checkpoint_demo.items SET n=99 WHERE id=1 RETURNING n;

A receives ninety-nine from its own update. Do not commit it. In B, request a checkpoint and inspect what an ordinary reader can see:

CHECKPOINT;
SELECT c.checkpoint_lsn>b.checkpoint_lsn AS checkpoint_advanced,
 (SELECT n FROM checkpoint_demo.items WHERE id=1) AS visible_value
FROM checkpoint_demo.baseline AS b CROSS JOIN pg_control_checkpoint() AS c;

The checkpoint location advances, but B still reads ten. The checkpoint has completed while A’s transaction remains undecided. This separates maintenance progress from business commit status. In A, verify its own version is still visible, then discard it:

SELECT n FROM checkpoint_demo.items WHERE id=1;
ROLLBACK;

A reads ninety-nine before rollback. B can now confirm the original ten and perform a normal committed update, with checkpoint locations captured immediately around that operation:

SELECT n FROM checkpoint_demo.items WHERE id=1;
UPDATE checkpoint_demo.baseline
SET checkpoint_lsn=(SELECT checkpoint_lsn FROM pg_control_checkpoint());
UPDATE checkpoint_demo.items SET n=11 WHERE id=1;
SELECT c.checkpoint_lsn=b.checkpoint_lsn AS no_new_checkpoint,
 (SELECT n FROM checkpoint_demo.items WHERE id=1) AS committed_value
FROM checkpoint_demo.baseline AS b CROSS JOIN pg_control_checkpoint() AS c;
DO $$ BEGIN
 IF (SELECT n FROM checkpoint_demo.items WHERE id=1) IS DISTINCT FROM 11 THEN
 RAISE EXCEPTION 'checkpoint/transaction boundary result mismatch';
 END IF;
END $$;
DROP SCHEMA checkpoint_demo CASCADE;

The results are ten, followed by a true unchanged-checkpoint comparison and committed value eleven in the tested short schedule. The commit succeeded without requesting another checkpoint. An independently scheduled checkpoint can change that comparison in a busy instance; it does not change the meaning of commit. The assertion checks the final row value, and the fixture is removed afterward.

This drill verifies transaction visibility and checkpoint boundaries. It does not measure recovery duration or simulate a storage device losing acknowledged writes. The durability contract still depends on enabled durability settings and storage that honors synchronization requests. Use separate crash-recovery rehearsals to validate operational recovery time against the workload you actually retain.

Tune against recovery and foreground work

Longer checkpoint intervals can reduce repeated flushing and, with full-page writes enabled, avoid some repeated full-page-image logging. They can also leave more work to replay after a crash. Frequent checkpoints can increase write traffic and latency. Fast local storage alone is not evidence that a larger interval meets a particular recovery objective.

checkpoint_completion_target spreads ordinary checkpoint work across a target fraction of the interval; its usual 0.9 value is a pacing target, not a guarantee against every latency spike. Measure checkpoint duration, requested versus timed checkpoints, WAL growth, and foreground latency together. In PostgreSQL 18, checkpoint counters and write/sync times are in pg_stat_checkpointer; the background-writer view describes a different component. Investigate repeated WAL-driven requests and retention causes before changing limits, and compare recovery tests before crediting a tuning change as successful.

Sources

PostgreSQL 18 WAL introduction, write-ahead ordering and commit durability without immediate data-file flushing, checked 2026-09-11.
PostgreSQL 18 CHECKPOINT, explicit checkpoint behavior and required privileges, checked 2026-09-10.
PostgreSQL 18 WAL configuration, checkpoint scheduling, pacing, and WAL size semantics, checked 2026-09-10.
PostgreSQL 18 cumulative statistics, the dedicated checkpointer statistics view, checked 2026-09-10.

VACUUM FULL rebuilds a table instead of pruning it

Mon, 03 Jan 2028 00:00:00 +0000

After a large deletion, a PostgreSQL table may retain a large file even when only a small fraction of its rows remain. Ordinary vacuum makes obsolete space reusable; VACUUM FULL rewrites the surviving contents into a new file and rebuilds indexes. That stronger reclamation has a different lock and capacity budget, so it belongs to planned maintenance rather than routine reaction to a large size number.

When it breaks

The rewrite takes an ACCESS EXCLUSIVE table lock. It conflicts with ordinary readers as well as writers, including a transaction that already finished its query but remains open while holding a table lock. Calling it an exclusive row lock, or assuming read traffic can continue, understates the outage risk. A queued rewrite can also affect later lock requests while it waits.

The old storage remains until replacement succeeds, so the operation needs additional disk space. Starting only when the filesystem is already full can remove the room needed to recover. Estimate the replacement table and indexes, I/O and WAL impact, and acceptable lock duration. Regular autovacuum does not automatically perform this rewrite.

Check it quickly

Open two fresh psql -X -v ON_ERROR_STOP=1 sessions, A and B, in the same disposable PostgreSQL 18 database. Run vacuum commands outside explicit transaction blocks. B creates five thousand rows, deletes the first 4,500, and performs ordinary vacuum:

CREATE SCHEMA compact_demo;
CREATE TABLE compact_demo.items (id integer PRIMARY KEY,payload text)
 WITH (autovacuum_enabled=false);
INSERT INTO compact_demo.items
SELECT g,repeat(md5(g::text),10) FROM generate_series(1,5000) AS g;
CREATE TABLE compact_demo.baseline AS
SELECT pg_relation_size('compact_demo.items') AS heap_bytes,
 pg_relation_filenode('compact_demo.items') AS file_id;
DELETE FROM compact_demo.items WHERE id<=4500;
VACUUM (ANALYZE) compact_demo.items;
SELECT pg_relation_size('compact_demo.items')=heap_bytes AS same_heap_size,
 pg_relation_filenode('compact_demo.items')=file_id AS same_file
FROM compact_demo.baseline;

The two comparisons return true: heap bytes and the relation’s file identifier remain unchanged. The surviving rows occupy the tail, so ordinary vacuum cannot shrink this particular file by truncating an empty end. This does not establish that ordinary vacuum never shrinks files; empty trailing pages can be returned when truncation is possible.

In A, read the remaining five hundred rows and keep the transaction open:

BEGIN;
SELECT count(*) FROM compact_demo.items;

A has made no updates, but still holds its table lock. In B, give the rewrite a short acquisition budget so the demonstration does not wait indefinitely:

SET lock_timeout='100ms';
\set ON_ERROR_STOP off
VACUUM (FULL) compact_demo.items;
\echo full_state :SQLSTATE
\set ON_ERROR_STOP on
RESET lock_timeout;

Expect full_state 55P03. The existing reader prevents the stronger lock from being acquired. Temporarily allowing the expected error shows the result without skipping cleanup, and resetting the timeout restores B’s original session setting. This timeout is a laboratory value, not a suggested production maintenance budget.

End the reader transaction in A:

COMMIT;

Now B can perform the rewrite, inspect storage identity, and verify every retained payload:

VACUUM (FULL,ANALYZE) compact_demo.items;
SELECT pg_relation_size('compact_demo.items')<heap_bytes AS smaller_heap,
 pg_relation_filenode('compact_demo.items')<>file_id AS different_file
FROM compact_demo.baseline;
SELECT count(*),min(id),max(id),sum(id),
 bool_and(payload=repeat(md5(id::text),10)) AS payloads_preserved
FROM compact_demo.items;
DO $$ BEGIN
 IF (SELECT sum(id) FROM compact_demo.items)<>2375250
 OR (SELECT count(*) FROM compact_demo.items)<>500
 OR EXISTS (SELECT FROM compact_demo.items
 WHERE payload<>repeat(md5(id::text),10)) THEN
 RAISE EXCEPTION 'compaction changed application data';
 END IF;
END $$;
DROP SCHEMA compact_demo CASCADE;

Both storage comparisons are true: the heap is smaller and its file identifier has changed. The data check returns five hundred rows, identifiers 4501 through 5000, an identifier sum of 2,375,250, and true payload preservation. The assertion checks the row population and content; a smaller file alone would not prove correct compaction. Finally, the fixture schema is removed.

Decide whether compaction pays for itself

If future inserts will reuse the free space, keeping it inside the relation may be preferable to paying for a rewrite and later growing the file again. Measure persistent excess space and access cost after the workload settles. Distinguish the main heap from indexes and auxiliary storage, because a single table-size number may combine several objects with different causes of growth.

Schedule the operation where its exclusive lock is acceptable, and check for long-running readers before requesting it. A low-traffic period does not guarantee no open transaction exists. Set an intentional acquisition budget, observe failure rather than endlessly retrying, and verify data and storage afterward. Keep ordinary vacuum healthy so severe accumulation does not become the normal maintenance cycle.

The experiment demonstrates lock conflict, file replacement, and preserved rows on a small table. It does not predict rewrite duration or spare-space needs for a production table with large indexes, external values, or replication traffic. Those quantities require measurements from the actual object and a maintenance plan that can tolerate the resulting lock window.

Sources

PostgreSQL 18 VACUUM, rewrite behavior, extra space, truncation, and transaction restrictions, checked 2026-09-10.
PostgreSQL 18 explicit locking, ACCESS EXCLUSIVE conflicts with readers and writers, checked 2026-09-10.
PostgreSQL 18 routine vacuuming, routine reuse versus exceptional full compaction, checked 2026-09-10.

Partitioning splits one table into many pieces that look like one

Thu, 23 Dec 2027 00:00:00 +0000

Declarative partitioning lets PostgreSQL expose one logical table while storing its rows in separate partitions. The parent routes writes and provides a common query target; the physical partitions hold the data. Its strongest benefits often concern lifecycle management and selective access, but the choice also affects keys, query shape, and operational locking.

A logical parent with physical children

A partitioned parent has no row storage of its own. Each attached partition holds rows permitted by its bounds and can have its own storage parameters, indexes, and maintenance activity. A partitioned index similarly represents a collection of indexes on the physical partitions. Splitting a table does not copy every row into both a parent store and a child store.

Range, list, and hash partitioning divide data according to different rules. Choose the key from real access and retention requirements: monthly event data may fit range partitioning, while other workloads need a different distribution. Declarative partitioning does not by itself distribute those partitions across independent database servers. Local partitions still share the resources and transactional environment of their database.

Keeping the parent name stable can simplify application access, but it does not prove that application code needs no changes. Queries must expose useful partition-key restrictions to benefit from pruning. Keys and foreign-key references may need redesign, and operations that previously addressed a globally unique identifier must retain their intended identity semantics.

The key affects more than routing

A unique or primary-key constraint on a partitioned table must include all partition-key columns, and the partition key must not contain expressions for these constraints. PostgreSQL’s individual partition indexes cannot independently prove arbitrary uniqueness across the other partitions. A composite key on day and id therefore does not promise that id alone is unique across all days.

That distinction matters during a migration from a single table with a global id key. Merely adding the date to the primary key can weaken the original identity contract. Review callers, references, and duplicate-prevention requirements before changing it. The partition key is not just a physical tuning option that can be changed without examining the public data model.

Updating a partition key can move a row to another partition. The destination must exist and accept the row, and indexes and constraints still apply. Such updates are real data changes, not free metadata edits. If retention has already removed the destination range, a late correction may fail even though the logical parent table still exists.

When it breaks

DETACH removes a table from the partition hierarchy while retaining that table and its rows. Queries through the parent stop seeing it, but the detached relation remains available for separate inspection or export. DROP removes the partition’s data as well; detaching is not deletion, and neither operation automatically creates a verified archive or backup.

Partition maintenance takes locks. A plain detach needs a strong lock on the parent, so it can wait behind long-running work and affect concurrent access. PostgreSQL provides a concurrent detach form with different locking and additional restrictions, including that it cannot run inside an explicit transaction block. Choose the operation using its actual contract rather than assuming that “metadata operation” means “no blocking.”

Too many partitions can also increase planning and memory costs. There is no universal row-count threshold that makes partitioning worthwhile. Measure representative reads and maintenance operations, create future partitions before writes need them, and define how late-arriving data interacts with retention. A small well-indexed table may remain simpler than a large collection of tiny partitions.

Check it quickly

Save this as partition-lifecycle.sql and run psql -X -v ON_ERROR_STOP=1 -f partition-lifecycle.sql in a disposable PostgreSQL 18 database. The experiment uses temporary partitions and ordinary transactional detach; final rollback removes the entire fixture.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE events (day date,id integer,PRIMARY KEY(day,id)) PARTITION BY RANGE(day);
CREATE TEMP TABLE events_sep PARTITION OF events FOR VALUES FROM ('2026-09-01') TO ('2026-10-01');
CREATE TEMP TABLE events_oct PARTITION OF events FOR VALUES FROM ('2026-10-01') TO ('2026-11-01');
INSERT INTO events VALUES ('2026-09-15',1),('2026-09-30',2),('2026-10-01',3);
SELECT id,day,tableoid::regclass AS physical_table FROM events ORDER BY id;
DO $$ BEGIN
 CREATE UNIQUE INDEX events_id_only ON events(id);
 RAISE EXCEPTION 'global id uniqueness accepted without partition key';
EXCEPTION WHEN feature_not_supported THEN RAISE NOTICE 'unique key must include partition key';
END $$;
SAVEPOINT id_scope;
INSERT INTO events VALUES ('2026-10-03',1);
SELECT id,day FROM events WHERE id=1 ORDER BY day;
ROLLBACK TO id_scope;
RELEASE id_scope;
UPDATE events SET day='2026-10-02' WHERE id=2;
SELECT id,day,tableoid::regclass AS physical_table FROM events ORDER BY id;
ALTER TABLE events DETACH PARTITION events_sep;
SELECT (SELECT count(*) FROM events) AS attached_rows,(SELECT count(*) FROM events_sep) AS detached_rows;
DO $$ BEGIN
 IF (SELECT array_agg(id ORDER BY id) FROM events) IS DISTINCT FROM ARRAY[2,3] OR
 (SELECT array_agg(id ORDER BY id) FROM events_sep) IS DISTINCT FROM ARRAY[1] THEN
 RAISE EXCEPTION 'detach changed stored rows';
 END IF;
END $$;
ROLLBACK;

PostgreSQL 18.6 routed ids one and two to September and id three to October. It rejected a parent unique index on id alone. Within a savepoint, a second id one on another day was accepted under the composite primary key; rolling back that savepoint restores the original fixture.

Updating id two’s day moved it to October, as tableoid showed. Detaching September then left two rows visible through the parent and one row in the detached table. Assertions verify the exact identifiers on both sides, proving that detach changed membership without discarding stored data. This validates the lifecycle and key semantics, not production lock duration or a complete archival procedure.

Sources

PostgreSQL table partitioning, physical storage, keys, routing, and partition management, checked 2026-09-10.
PostgreSQL ALTER TABLE, attach and detach behavior and locking restrictions, checked 2026-09-10.

A Common Table Expression names a query you can reuse

Sun, 19 Dec 2027 00:00:00 +0000

A common table expression gives a query result a name within one SQL statement. It can make a complex report readable by naming populations and intermediate calculations before the final SELECT. That name is not a persistent table, and it does not by itself promise a cached result, a particular scan count, or a sequence of procedural steps.

Scope first, execution second

A WITH clause belongs to its statement. A later statement cannot reuse its CTE name, and there is no schema object to index or query after execution. Use a temporary table, view, or another appropriate database object if results or definitions must outlive that statement. A CTE is useful when its scope and its named result match the work being expressed.

In modern PostgreSQL, a nonrecursive SELECT CTE without volatile functions can be folded into its parent query. By default, this happens when it is referenced once. Folding lets outer restrictions participate in optimization, so an outer primary-key lookup can reach an index on the underlying table instead of filtering a separately produced intermediate result.

When the same eligible CTE is referenced more than once, PostgreSQL normally materializes it instead. That can avoid repeating expensive calculations, but it can also prevent a restrictive condition at one use from being pushed into the shared computation. This distinction has applied since PostgreSQL 12; assuming that every WITH is always an optimization fence describes older behavior, not the current default.

Materialization is a tradeoff

MATERIALIZED requests separate computation. NOT MATERIALIZED permits folding even with repeated references, potentially repeating work while allowing each use to access only the rows it needs. PostgreSQL ignores NOT MATERIALIZED for recursive or non-side-effect-free CTEs; it does not authorize duplicating arbitrary writes or volatile operations.

A materialized SELECT CTE is not necessarily evaluated to exhaustion. Its consumers may demand only part of its result, though sorts, joins, or other operations can require more. Materialization also does not create a reusable indexed table for later statements. Inspect the plan and measure the actual workload before forcing a boundary merely because the word sounds like a cache.

The useful comparison is total work: repeated selective index probes may be cheaper than building and scanning a large shared result, while repeating an expensive calculation may be worse. A readable name and an efficient execution plan are compatible, but neither implies the other. Keep the business predicate in one authoritative definition and choose execution controls only when the evidence supports them.

When it breaks

Data-modifying CTEs follow different rules. INSERT, UPDATE, or DELETE inside WITH executes once and to completion even if its output is not consumed. The named relation is the RETURNING output, not a freshly rescanned target table. A modifying CTE without RETURNING still performs its modification but cannot supply rows to a later reference.

The modifying substatements and main statement use the same snapshot, and their actual modification order is not a procedural top-to-bottom guarantee. Pass changed rows through RETURNING when another part of the statement needs them. Avoid attempting to modify the same row twice in one statement and expecting textual order to decide the winner. A plain SELECT-looking outer query does not make a data-modifying WITH read-only.

Check it quickly

Save this as cte.sql and run psql -X -v ON_ERROR_STOP=1 -f cte.sql in a disposable PostgreSQL 18 database. It compares four read plans, then moves two temporary rows using DELETE RETURNING and INSERT. The final rollback removes the fixture and reverses its changes.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE items (id integer PRIMARY KEY, value text);
INSERT INTO items SELECT g,'item-'||g FROM generate_series(1,10000) AS g;
ANALYZE items;
EXPLAIN (COSTS OFF)
WITH source AS (SELECT * FROM items) SELECT * FROM source WHERE id=7;
EXPLAIN (COSTS OFF)
WITH source AS MATERIALIZED (SELECT * FROM items) SELECT * FROM source WHERE id=7;
EXPLAIN (COSTS OFF)
WITH source AS (SELECT * FROM items)
SELECT a.id FROM source a JOIN source b USING(id) WHERE a.id=7;
EXPLAIN (COSTS OFF)
WITH source AS NOT MATERIALIZED (SELECT * FROM items)
SELECT a.id FROM source a JOIN source b USING(id) WHERE a.id=7;
CREATE TEMP TABLE moved (id integer,value text);
WITH removed AS (DELETE FROM items WHERE id<=2 RETURNING *)
INSERT INTO moved SELECT * FROM removed;
SELECT (SELECT count(*) FROM items) AS remaining,(SELECT count(*) FROM moved) AS moved;
DO $$ BEGIN
 IF (SELECT array_agg(id ORDER BY id) FROM moved) IS DISTINCT FROM ARRAY[1,2] THEN
 RAISE EXCEPTION 'RETURNING handoff failed';
 END IF;
 IF (SELECT count(*) FROM items) <> 9998 THEN RAISE EXCEPTION 'DELETE did not execute'; END IF;
END $$;
ROLLBACK;

On PostgreSQL 18.6, the single-reference default plan used items_pkey directly. Explicit MATERIALIZED introduced a CTE Scan over a planned sequential scan. The default double reference also shared a materialized CTE, while NOT MATERIALIZED produced two index-based accesses. These plain EXPLAIN statements show planned structure; they do not measure execution time or prove a production performance improvement.

The modifying statement actually executed and left 9,998 rows in items and two in moved. Assertions verify that the transferred identifiers are exactly one and two and that the source rows were deleted. RETURNING provides the handoff between operations. Changing the outer statement’s demand is not a safe method for suppressing a modifying CTE; keep such experiments transactional and inspect all WITH bodies before executing unfamiliar SQL.

Sources

PostgreSQL WITH queries, folding, materialization, and data-modifying CTEs, checked 2026-09-10.
PostgreSQL SELECT reference, WITH scope and MATERIALIZED controls, checked 2026-09-10.

EXPLAIN ANALYZE runs the query to show real cost

Thu, 16 Dec 2027 00:00:00 +0000

EXPLAIN describes a PostgreSQL execution plan; EXPLAIN ANALYZE also executes the statement and measures its work. The useful comparison is between estimated rows and observed rows at the same node. Neither number is a verdict by itself: an estimate is a model, while a measurement describes one execution under particular data, cache, concurrency, and instrumentation conditions.

Read comparable quantities

Planner costs are relative units, not milliseconds. The two cost values describe startup and completion, and the preferred plan can change when LIMIT makes early rows especially valuable. Actual timings use milliseconds. Comparing a cost of 100 with an elapsed time of 100 is therefore meaningless even when the printed values happen to match.

For a node executed repeatedly, actual rows and node timings are averages per loop. Multiplying by loops helps expose repeated work, but summing every node’s time double-counts work included in parent nodes. Parallel execution makes naive totals still less useful. Read the tree, its loop counts, filters, and row flow before assigning blame to the visually largest number.

BUFFERS separates shared, local, and temporary block activity. Shared hits mean PostgreSQL found blocks in its shared buffer cache. A shared read does not prove a physical disk access: the operating system may satisfy it from another cache. Temporary tables use local buffers, while sort or hash spills can use temporary files. Parent counters include children, so adding all rows of counters inflates the apparent I/O.

Investigate the first meaningful discrepancy

A selective predicate estimated at ten rows but returning ten thousand deserves investigation before rewriting a join above it. Stale statistics, skew, correlated columns, parameter values, and unsupported selectivity models can all contribute. ANALYZE refreshes statistics, but it cannot guarantee exact estimates or repair every plan. Inspect representative data and consider targeted statistics changes only after identifying the missing information.

An accurate row estimate does not prove the query is efficient. Missing access paths, expensive expressions, spills, lock waits, and workload contention can remain. Conversely, a sequential scan can be appropriate for a small table or a query returning much of it. Measure the application’s actual query shape and parameters instead of treating every index-free plan as a defect.

When it breaks

The dangerous mistake is treating ANALYZE as a dry run. An UPDATE changes rows, triggers execute, locks are acquired, and a SELECT can call a function with side effects. Wrapping the statement in a transaction and rolling back restores transactional table changes, but sequence advances and external effects are not generally reversible. Use a disposable dataset when the statement’s effects are uncertain.

Timing also misleads when the first execution warms caches or instrumentation dominates a cheap operation. TIMING OFF retains actual row counts and total execution time while avoiding per-node clock reads. EXPLAIN discards ordinary result delivery, so its duration does not include sending the application’s full response over the network. Repeated representative measurements and application latency answer different parts of the performance question.

Check it quickly

Use psql against a disposable PostgreSQL 18 database. Save this block as explain.sql and run psql -X -v ON_ERROR_STOP=1 -f explain.sql. Connection options must name that database. Everything created here is temporary, and the final rollback removes it.

\set ON_ERROR_STOP on
BEGIN;
SET LOCAL statement_timeout = '5s';
CREATE TEMP TABLE orders (id integer, status text);
INSERT INTO orders SELECT g, CASE WHEN g <= 100 THEN 'paid' ELSE 'new' END
FROM generate_series(1, 10000) AS g;
ANALYZE orders;
EXPLAIN (COSTS ON) SELECT * FROM orders WHERE status = 'paid';
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT * FROM orders WHERE status = 'paid';
CREATE TEMP SEQUENCE measured_calls;
SAVEPOINT before_measurement;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
UPDATE orders SET id = id + 10000 WHERE id <= 3;
EXPLAIN (ANALYZE, TIMING OFF) SELECT nextval('measured_calls');
ROLLBACK TO SAVEPOINT before_measurement;
SELECT count(*) AS updated_rows_after_rollback FROM orders WHERE id > 10000;
SELECT nextval('measured_calls') AS sequence_after_rollback;
DO $$ BEGIN
 IF (SELECT count(*) FROM orders WHERE status = 'paid') <> 100 THEN
 RAISE EXCEPTION 'unexpected selected row count';
 END IF;
 IF EXISTS (SELECT FROM orders WHERE id > 10000) THEN
 RAISE EXCEPTION 'UPDATE was not rolled back';
 END IF;
 IF (SELECT last_value FROM measured_calls) <> 2 THEN
 RAISE EXCEPTION 'sequence advance unexpectedly rolled back';
 END IF;
END $$;
ROLLBACK;

The fixture contains exactly 100 paid orders. The measured SELECT reports 100 actual rows and 9,900 removed by its filter. The UPDATE’s scan reports three rows even though its top node returns zero without RETURNING; zero there does not mean nothing changed. After the savepoint rollback, the changed-row count is zero, but the next sequence value is two because the measured SELECT already consumed one.

On PostgreSQL 18.6 these assertions pass. Buffer counts, timings, formatting, and chosen plans can vary; the deterministic results are the row counts, rollback behavior, and sequence advance. The five-second statement timeout bounds individual statements in this exercise. It does not make arbitrary production writes safe to inspect.

Sources

PostgreSQL EXPLAIN, execution effects and instrumentation options, checked 2026-09-10.
PostgreSQL using EXPLAIN, costs, loops and buffer interpretation, checked 2026-09-10.
PostgreSQL planner statistics, estimates and statistics limits, checked 2026-09-10.
PostgreSQL sequence functions, sequence advances outside rollback, checked 2026-09-10.

pg_stat_statements finds the queries that cost the most

Sun, 12 Dec 2027 00:00:00 +0000

Because pg_stat_statements aggregates every execution of one query shape, it turns a vague “the page is slow” into a ranked table. The setup is a config line and a restart; the payoff is knowing which of your query shapes actually owns the clock.

The one decision that matters is which column you sort by. Sorting by mean time reproduces what the slow log already told you; sorting by total time is what finds the query that has been quietly consuming a quarter of your database.

Normalize, then aggregate

The extension does not store your SQL verbatim. Constants are folded into parameter symbols, so WHERE tenant_id = 42 and WHERE tenant_id = 8712 collapse into one entry, displayed as the text of the first query that produced that queryid. The numbering rule is positional — symbols start at the next number after the highest $n already present, which is why a PL/pgSQL statement like SELECT i + 1 INTO j shows up as SELECT i + $2. Lists behave the same way: two queries differing only in the number of IN elements get squashed into IN ($1 /*, ... */) with a call count of two.

The effect is dramatic: a modelled workload of 1,420,052 executions, each with different literals, collapsed into 6 rows — 236,675 raw statements per row. Two caveats come with it: queryid is a hash of the post-parse-analysis tree, so identical text can land in different entries when search_path differs, unrelated queries can collide into one entry, and the value is not promised to be stable across major versions. Treat queryid as a better identifier than the text, not as a permanent key.

Total time, not mean time

The columns that matter are calls, total_exec_time, mean_exec_time, rows, and the block counters. min/max/stddev_exec_time tell you whether a query is uniformly slow or occasionally long, which points at bad plans versus lock contention respectively.

Ranking by mean answers a different question from ranking by total, and the gap is large. In the same workload the query with the worst mean (9,689 ms, 12 calls) is 22.4% of measured time; the query with the worst total is 25.9% but averages 0.749 ms per call. The worst-by-total query never appears in a slow log at any sane log_min_duration_statement setting, and the three worst by total together are 65.5% of all measured time. A 0.099 ms lookup called 900,000 times ranks sixth by mean and third by total, at 89,217 ms.

Counters accumulate, so compare windows

Counters run from the moment statistics started, so a query that used to be slow never fades out of the top of the table on its own. The fix is either a scoped reset — pg_stat_statements_reset(userid, dbid, queryid, minmax_only) takes all four, so you can clear one query, one database, or just the min/max fields — or a delta between two snapshots.

The block counters deserve their own reading. shared_blks_hit and shared_blks_read are always tracked; track_io_timing adds the *_blk_read_time and *_blk_write_time columns, not the counts. Hit ratio is a diagnostic, not a cost: the query with the best hit ratio in the workload (98.89%) was also the largest block consumer at 81,000,000 blocks, while the query with the worst ratio (62.22%) moved 37x fewer.

When it breaks

The first failure is that the extension is simply not there. It must be listed in shared_preload_libraries because it needs shared memory, which means adding or removing it requires a restart, compute_query_id must be auto or on, and the views only exist in databases where CREATE EXTENSION pg_stat_statements was run. Miss any of those and every query against it fails, so the monitoring you thought you had has been empty for months.

The second failure is a reset that destroys the comparison. Reset the counters mid-window and every delta comes out negative: in the fixture, 6 of 6 shapes reported negative time. The arithmetic is fine; the window is not. pg_stat_statements_info.stats_reset exists precisely for this — check it before trusting any subtraction, and if the reset was intentional, start the window after it rather than trying to reconcile across it.

Check it quickly

Run both rankings and see how little they agree:

psql -c "SELECT calls, round(total_exec_time) AS total_ms, round(mean_exec_time,3) AS mean_ms, rows FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 10;"
psql -c "SELECT calls, round(mean_exec_time,2) AS mean_ms, round(total_exec_time) AS total_ms FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"
psql -c "SELECT stats_reset FROM pg_stat_statements_info;"
psql -c "SELECT sum(calls) AS calls, round(sum(total_exec_time)/1000) AS total_s, round(sum(shared_blks_read)*8.0/1024/1024,1) AS read_gib FROM pg_stat_statements;"
psql -c "SELECT pg_stat_statements_reset(0, 0, queryid) FROM pg_stat_statements WHERE query LIKE 'SELECT%';"

The first two lists should disagree, and the first is the one to work from. stats_reset is the timestamp that decides whether a delta window is valid. The totals line is the sanity check on magnitude — if total_s is larger than the wall-clock age of your statistics, you are reading across a restart. The last command resets selectively; run it deliberately, because it is exactly what makes the next comparison wrong.

Sources

PostgreSQL: pg_stat_statements, normalization, the statistics columns, and pg_stat_statements_reset, checked 2026-09-16.
PostgreSQL: cumulative statistics system, where stats_reset and the counters live, checked 2026-09-16.
PostgreSQL: statistics monitoring configuration, track_io_timing and the block-time columns it enables, checked 2026-09-16.

Postgres jsonb stores JSON for querying

Sat, 04 Dec 2027 00:00:00 +0000

PostgreSQL offers two JSON storage types with different representation costs. json validates and retains the input text, while jsonb converts it into a decomposed binary value. Both support extracting fields and paths. Choosing between them is primarily a decision about preserving representation, processing documents repeatedly, and supporting the operators your queries need.

Preserve text or preserve a value

A json value retains insignificant whitespace, object key order, and repeated object keys. Its processing functions still need to parse that text, and when duplicate keys occur they treat the last value as operative. Retaining duplicates therefore does not mean every consumer will interpret them the same way. Avoid using duplicate keys as a meaningful part of an application contract.

A jsonb value discards insignificant whitespace and object key order, retaining only the last occurrence of each object key. This statement concerns objects: arrays retain their element order and repeated elements. Array containment has its own matching rules, so containment results must not be mistaken for evidence that the stored array was reordered or deduplicated.

The binary format belongs to PostgreSQL; it is not BSON and is not a portable signing format. The textual rendering of jsonb is not a promise of a particular external canonicalization algorithm. If a signature covers original request bytes, preserve those bytes separately, for example in bytea, and verify against that representation. A json column preserves JSON input text, but client encoding conversion and protocol processing mean it is not a general archive of the original network bytes.

Extraction is available in both types

Both types support -> for a JSON value, ->> for text, and #> or #>> for a nested path. Extracting a field from json does not require first casting the whole document to jsonb. The performance distinction is reparsing stored text versus working with the decomposed representation, not an absence of extraction operators.

Jsonb additionally supports structural containment, key existence, and index strategies such as GIN. A scalar expression extracted from either type can also be a candidate for an appropriate expression index. Index selection still depends on the exact expression, operator, and workload; selecting jsonb alone does not make every nested condition fast. Frequently joined, constrained, or ordered attributes may belong in ordinary typed columns.

Input and processing costs also matter. Jsonb does extra work during conversion and can use more space for some documents. It accepts almost, but not exactly, the same inputs as json: PostgreSQL numeric limits and stricter Unicode handling can reject values accepted as json text. Test representative payloads before converting an existing column, especially unusually large numbers and escaped characters.

When it breaks

Changing json to jsonb can destroy representation that an exporter expected to replay. Changing types also does not enforce an application schema: a valid JSON string, number, or array is still JSON even when the application expects an object. Add the actual structural and business constraints rather than assuming that JSON validation checks required fields or their meaning.

A subtler failure is treating all parsers as interchangeable around duplicate keys. The database’s last-value behavior does not establish what an upstream signer or downstream parser used. Reject ambiguous input at the appropriate boundary when agreement matters. For query results, distinguish missing paths, JSON null, and SQL NULL rather than translating all three into an unexplained empty string.

Check it quickly

Save this as json-types.sql and run psql -X -v ON_ERROR_STOP=1 -f json-types.sql in a disposable PostgreSQL 18 database. It uses one temporary table inside a transaction and rolls everything back.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE sample (raw json, parsed jsonb);
INSERT INTO sample VALUES ('{ "b": 2, "a": 1, "a": 3 }','{ "b": 2, "a": 1, "a": 3 }');
SELECT raw::text, parsed::text, raw->>'a' AS json_a, parsed->>'a' AS jsonb_a FROM sample;
SELECT '{"user":{"name":"Ada"}}'::json #>> '{user,name}' AS json_path;
DO $$ BEGIN
 IF (SELECT raw::text FROM sample) IS DISTINCT FROM '{ "b": 2, "a": 1, "a": 3 }' THEN
 RAISE EXCEPTION 'json text was changed';
 END IF;
 IF (SELECT parsed FROM sample) IS DISTINCT FROM '{"a":3,"b":2}'::jsonb THEN
 RAISE EXCEPTION 'unexpected jsonb value';
 END IF;
 IF (SELECT raw->>'a' FROM sample) IS DISTINCT FROM '3' THEN
 RAISE EXCEPTION 'json extraction did not use the last duplicate key';
 END IF;
 IF '[2,1]'::jsonb = '[1,2]'::jsonb THEN
 RAISE EXCEPTION 'jsonb array order was ignored';
 END IF;
 IF '[2,1,2]'::jsonb = '[2,1]'::jsonb THEN
 RAISE EXCEPTION 'jsonb duplicate array elements were ignored';
 END IF;
END $$;
DO $$ BEGIN
 PERFORM '{broken'::json;
 RAISE EXCEPTION 'invalid JSON accepted';
EXCEPTION WHEN invalid_text_representation THEN RAISE NOTICE 'invalid JSON rejected';
END $$;
ROLLBACK;

PostgreSQL 18.6 returned the original spaced text for raw, a normalized object containing a=3 and b=2 for parsed, and 3 from both extraction expressions. The nested path on json returned Ada directly. The assertions check these semantics and separately check that array equality respects order and repeated elements; the handled invalid-input error prints an “invalid JSON rejected” notice.

This exercise establishes representation and extraction behavior, not a speed or storage benchmark. Compare actual document sizes, write rates, and query plans separately before migrating a production column. Keep original data available until conversion failures and any intentional loss of representation have been accounted for.

Sources

PostgreSQL JSON types, representation, input restrictions, and indexing, checked 2026-09-10.
PostgreSQL JSON functions and operators, extraction operators for both types and null behavior, checked 2026-09-10.