Tuning · XB Field Notes

Partial indexes trim a column

Tue, 15 Feb 2028 00:00:00 +0000

Not every row deserves an index slot; a partial index keeps the tree thin exactly where the query never looks. Most tables have a skew the schema pretends not to see: ninety percent of orders are closed, nearly all events are processed, the flag column is false almost everywhere. A full index dutifully records all of it, charging maintenance on every write to protect lookups that only ever touch the small live slice. The partial index stops paying for the rows nobody queries.

The bargain is precision for size. The index becomes smaller, faster to scan, and cheaper to maintain — but only queries whose predicates match the index condition may use it. Everything else falls back to competing plans. That makes partial indexes a deliberate pairing of access pattern with storage, not a general speedup. Choose them where the hot subset is stable and the queries agree on its definition.

Index only what you query

A partial index carries a WHERE clause, so rows that fail the predicate never enter the tree. When most rows are rarely read, this keeps the index small and the lookups fast on the ones that matter. An index on open orders rather than all orders can be an order of magnitude smaller, which means more of it fits in cache, fewer pages per lookup, and plans that stay index-driven as the cold history grows unboundedly.

Pick the predicate from the queries. The right condition is the one every important query already repeats: status equals open, deleted_at is null. If queries slice the table differently, either the dominant slice wins or the table needs two partial indexes, each still thinner than a full one.

Fewer rows, cheaper maintenance

Fewer entries mean inserts, updates, and VACUUM touch a lighter structure, and disk usage drops visibly. The savings are most obvious on write-heavy tables where the full set would be mostly noise. Every insert that fails the predicate skips the index entirely — no page split risk, no WAL for that entry — and updates that leave the predicate untouched skip index maintenance too. On tables absorbing thousands of cold writes per second, the avoided work dwarfs the lookup gains.

Smaller indexes also vacuum faster with fewer pages for bloat to hide in. But monitor size against the full-index equivalent: if the partial index approaches full size, the predicate no longer selects a minority and the bargain has expired — drop it or redefine the slice.

Shape the statement to match

The planner only honors the index for queries whose predicate matches the partial condition exactly, or implies it. Write the filter the same way in the query and the index so the planner can actually use it. status = ‘open’ matches an index predicated on status = ‘open’; status IN (‘open’, ‘pending’) does not, even though a human sees the overlap. The matcher is logical implication, not textual similarity, and it is conservative by design.

This strictness dictates a discipline: canonicalize the filter expression across the codebase. If the index says deleted_at IS NULL, no query may phrase it as deleted_at = NULL or COALESCE(deleted_at, …) IS NULL and expect the index. Centralize the hot-slice condition — a view, a scope, a query builder fragment — so every statement inherits the exact predicate. EXPLAIN is the arbiter: if the plan ignores the index, the predicates differ in a way the matcher cannot bridge.

When it breaks

The classic failure is the drifting predicate: the business redefines “open” to include a new status, application queries update, and the partial index silently stops matching half the workload. Nothing errors — plans just degrade to sequential scans one query at a time, and latency climbs without a deploy to blame. Guard with a periodic EXPLAIN on the canonical queries in CI or monitoring, asserting the index name appears in the plan. A plan assertion catches predicate drift the day it ships, not the quarter it gets noticed.

The second failure is the over-sliced table: five partial indexes, each serving one report, whose combined maintenance exceeds the full index they replaced. Audit with pg_stat_user_indexes — near-zero-scan indexes are removal candidates, and overlapping predicates should merge into one wider partial index.

Check it quickly

Prove the index matches and pays:

CREATE INDEX ON orders (created_at) WHERE status = 'open';
EXPLAIN SELECT * FROM orders WHERE status = 'open' ORDER BY created_at LIMIT 20;
SELECT pg_size_pretty(pg_relation_size('orders_created_at_idx'));

The plan must show an index scan on the partial index, and the size should be a small fraction of the full-index equivalent.

Sources

PostgreSQL partial indexes, Official partial index documentation, checked 2026-06-01.
PostgreSQL EXPLAIN, plan inspection for verifying index usage, checked 2026-09-05.

Autovacuum keeps write-heavy tables lean

Fri, 04 Feb 2028 00:00:00 +0000

A busy table produces obsolete versions at a rate its maintenance must accommodate. Lowering an autovacuum trigger can make that table eligible sooner, but eligibility is not completion. A worker must still reach the table, acquire the required locks, and finish enough useful cleanup before the backlog grows again. Measure that whole cycle instead of treating one configuration value as a throughput guarantee.

When it breaks

The normal update/delete trigger combines a fixed threshold with a scale factor multiplied by the estimated table population. PostgreSQL 18 also applies a maximum threshold when enabled. Insert-triggered vacuum and age-driven freezing have their own reasons to run. Looking only at one dead-tuple ratio misses those paths, and interpreting every vacuum as proof that the update/delete threshold fired can misdiagnose a table.

Reducing the threshold does not remove old snapshot retention or create more I/O capacity. More workers and less cost delay can increase resource competition with foreground requests. A dead-tuple estimate near zero at every instant is neither a realistic universal target nor sufficient evidence of good performance. The useful target is sustainable cleanup with acceptable growth, latency, and maintenance cost.

Check it quickly

Use a disposable PostgreSQL 18 instance with autovacuum and statistics tracking enabled, available workers, and its normal one-minute database inspection interval. Run the script through a fresh psql -X -v ON_ERROR_STOP=1 connection. It waits up to 130 seconds for an actual automatic run; a timeout is a failed observation, not permission to claim success from a manual vacuum instead.

CREATE SCHEMA churn_demo;
CREATE TABLE churn_demo.jobs (id integer PRIMARY KEY,n integer)
 WITH (autovacuum_enabled=false,
 autovacuum_vacuum_threshold=50,
 autovacuum_vacuum_scale_factor=0.2);
INSERT INTO churn_demo.jobs SELECT g,0 FROM generate_series(1,10000) AS g;
SELECT pg_stat_force_next_flush();
VACUUM (ANALYZE) churn_demo.jobs;
UPDATE churn_demo.jobs SET n=1 WHERE id<=1000;
SELECT pg_stat_force_next_flush();
SELECT n_live_tup,n_dead_tup,vacuum_count,autovacuum_count
FROM pg_stat_user_tables WHERE relid='churn_demo.jobs'::regclass;
SELECT 50+0.2*reltuples AS old_threshold,50+0.01*reltuples AS new_threshold
FROM pg_class WHERE oid='churn_demo.jobs'::regclass;
ALTER TABLE churn_demo.jobs SET
 (autovacuum_enabled=true,autovacuum_vacuum_scale_factor=0.01);
DO $$
DECLARE deadline timestamptz := clock_timestamp()+interval '130 seconds';
BEGIN
 LOOP
 PERFORM pg_stat_clear_snapshot();
 EXIT WHEN (SELECT autovacuum_count>0 FROM pg_stat_user_tables
 WHERE relid='churn_demo.jobs'::regclass);
 IF clock_timestamp()>deadline THEN
 RAISE EXCEPTION 'no automatic vacuum observed within the experiment budget';
 END IF;
 PERFORM pg_sleep(0.25);
 END LOOP;
END $$;
SELECT pg_stat_clear_snapshot();
SELECT n_dead_tup,last_autovacuum IS NOT NULL AS ran,autovacuum_count>0 AS counted
FROM pg_stat_user_tables WHERE relid='churn_demo.jobs'::regclass;
SELECT count(*),count(*) FILTER (WHERE n=1) FROM churn_demo.jobs;
DO $$ BEGIN
 IF (SELECT count(*) FROM churn_demo.jobs)<>10000
 OR (SELECT count(*) FROM churn_demo.jobs WHERE n=1)<>1000 THEN
 RAISE EXCEPTION 'maintenance changed application data';
 END IF;
END $$;
DROP SCHEMA churn_demo CASCADE;

The table initially disables only its own routine autovacuum so setup and the controlled update complete first. Manual VACUUM ANALYZE establishes statistics and resets the insertion backlog before one thousand rows are updated. The explicit statistics-flush calls keep pending write counters from appearing after the manual maintenance estimate; they are laboratory synchronization, not a production tuning recommendation.

Before enabling the worker, expect estimated live and dead counts of ten thousand and one thousand, a manual-vacuum count of one, and an automatic-vacuum count of zero. Exact application counts are checked separately. With the specified fixed threshold of fifty, the old scale factor gives 2,050 and the smaller factor gives 150. The maximum threshold does not bind for this small table.

One thousand obsolete versions exceed the new trigger. The script enables routine autovacuum and lowers only that table’s factor, then waits for its automatic counter to advance. The polling block clears cached statistics before checking, so it does not keep reading one old statistics snapshot. It performs no manual vacuum after enabling the worker.

The final statistics report zero estimated dead tuples, a populated automatic timestamp, and a positive automatic counter. Application data remain ten thousand rows, with exactly one thousand values updated to one. In the tested instance, the observed background run completed within the budget. The exact delay depends on scheduler timing and worker availability, so the script asserts the event and data rather than one elapsed duration.

Diagnose the limiting stage

If no automatic run appears, check effective table options, server autovacuum settings, statistics tracking, worker activity, and conflicting locks. A manual maintenance timestamp cannot stand in for last_autovacuum. If runs complete but obsolete versions remain, investigate long transactions and other visibility horizons. If removable versions accumulate faster than completed work, then examine capacity, cost throttling, and write patterns.

Changing the scale factor for a large hot table is often more targeted than changing every table’s settings. Record the old options, compare several representative workload periods, and watch file growth and application latency along with maintenance counters. Reset or revise the override when the workload changes. The demonstration deletes its entire fixture at the end; a real table needs its chosen settings maintained deliberately, and continued freezing protection even when its update traffic falls quiet.

Sources

PostgreSQL 18 autovacuum configuration, thresholds, scale factors, worker limits, and inspection intervals, checked 2026-09-10.
PostgreSQL 18 routine vacuuming, trigger formulas and distinct maintenance purposes, checked 2026-09-10.
PostgreSQL 18 cumulative statistics, manual versus automatic counters and cached statistics, checked 2026-09-10.
PostgreSQL 18 statistics function implementation, the flush helper used to synchronize fixture observations, checked 2026-09-10.

A slower query needs both a plan and a wait check

Mon, 31 Jan 2028 00:00:00 +0000

A formerly fast query can become slow with the same data, statistics, and access plan. Another transaction may hold a lock, the storage cache may be colder, or the application may wait before PostgreSQL even receives the statement. Rechecking the plan is useful, but it belongs inside a broader comparison of where the elapsed time went. Begin by distinguishing time spent doing work from time spent waiting to do it.

Keep a comparable baseline

Record the statement shape, representative parameter values, returned row count, relevant session settings, and whether the application uses a prepared statement. Different parameters can produce very different populations. A manual query with a literal may not reproduce a prepared statement’s generic plan. A different role or search path can also change which objects and permissions apply.

Inspect current activity while the slowdown is happening. pg_stat_activity can show an active query with a lock wait; active does not mean continuously consuming CPU. pg_blocking_pids identifies sessions currently blocking that backend. Visibility into other roles’ activity requires appropriate privileges, so missing details are not evidence that nothing is running. Use fresh autocommit observations rather than repeatedly reading a cached monitoring snapshot inside one long transaction.

When it breaks

Blindly running ANALYZE treats every slowdown as a stale-statistics problem. That can add maintenance work while leaving the actual blocker untouched. Likewise, building an index on a one-row table cannot release another transaction’s row lock. Accurate estimated rows and a tiny sequential scan can coexist with arbitrarily long application waits.

EXPLAIN ANALYZE is an execution, including for writes, and can itself block. A broad update copied from production is not a harmless inspection command. First inspect waiting and lock ownership; measure a safe fixture or a statement whose effects are understood. Do not terminate an unfamiliar transaction merely because its identifier appears in the blocker list.

Check it quickly

Open three fresh psql sessions, A, B, and C, using the same role and disposable PostgreSQL 18 database with normal autocommit enabled. The named schema must not already exist. Keep the session roles separate; pasting every block into one connection cannot demonstrate blocking between transactions.

In A, create the fixture, then begin a transaction that changes its only row without committing.

\set ON_ERROR_STOP on
CREATE SCHEMA query_recheck_demo;
CREATE TABLE query_recheck_demo.items (id integer PRIMARY KEY,value integer);
INSERT INTO query_recheck_demo.items VALUES (1,0);
ANALYZE query_recheck_demo.items;

BEGIN;
UPDATE query_recheck_demo.items SET value=10 WHERE id=1;
SELECT pg_backend_pid() AS blocker_pid;

In B, start the measured update. It waits until A ends its transaction or the sixty-second statement timeout expires.

SET application_name='xbcatm_recheck_waiter';
SET statement_timeout='60s';
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
UPDATE query_recheck_demo.items SET value=value+1 WHERE id=1;

While B is waiting, run this in C, before the timeout. The test application name identifies B without exposing its complete query text.

SELECT pid,state,wait_event_type,wait_event,pg_blocking_pids(pid) AS blockers
FROM pg_stat_activity WHERE application_name='xbcatm_recheck_waiter';

In A, undo the uncommitted change and release its lock. Then collect B’s completed plan.

ROLLBACK;

After B finishes successfully, use C to verify the stored value and remove the fixture.

SELECT value FROM query_recheck_demo.items WHERE id=1;
DROP SCHEMA query_recheck_demo CASCADE;

The PostgreSQL 18.6 observation showed B as active with wait_event_type=Lock, wait_event=transactionid, and A’s process identifier in blockers. The wait-event detail can depend on the exact contention; the blocking relationship is the central evidence. After A rolled back, B incremented the original zero to one. The final query returned one, and cleanup removed the schema.

B’s scan estimated and read one row. Its top Update node returned zero rows because there was no RETURNING clause, not because it changed nothing. Execution time includes the induced wait and depends on how quickly A releases it. If the timeout occurs first, roll back A and use C’s final block to remove the fixture before restarting. The canceled update did not complete its increment.

Follow the observed cause

For a lock wait, investigate transaction lifetime and ownership before changing access paths. For excessive processed rows, compare selectivity and current statistics. For sorts or hashes that spill, inspect their memory and temporary I/O. Finally, compare database execution with application latency to expose connection or client delays. A new plan is meaningful only when it explains the measured regression and preserves the required result.

Sources

PostgreSQL 18: Monitoring database activity, active sessions and wait observations, checked 2026-09-10.
PostgreSQL 18: Using EXPLAIN, row counts and loops and plan interpretation, checked 2026-09-10.

VACUUM FULL rebuilds a table instead of pruning it

Mon, 03 Jan 2028 00:00:00 +0000

After a large deletion, a PostgreSQL table may retain a large file even when only a small fraction of its rows remain. Ordinary vacuum makes obsolete space reusable; VACUUM FULL rewrites the surviving contents into a new file and rebuilds indexes. That stronger reclamation has a different lock and capacity budget, so it belongs to planned maintenance rather than routine reaction to a large size number.

When it breaks

The rewrite takes an ACCESS EXCLUSIVE table lock. It conflicts with ordinary readers as well as writers, including a transaction that already finished its query but remains open while holding a table lock. Calling it an exclusive row lock, or assuming read traffic can continue, understates the outage risk. A queued rewrite can also affect later lock requests while it waits.

The old storage remains until replacement succeeds, so the operation needs additional disk space. Starting only when the filesystem is already full can remove the room needed to recover. Estimate the replacement table and indexes, I/O and WAL impact, and acceptable lock duration. Regular autovacuum does not automatically perform this rewrite.

Check it quickly

Open two fresh psql -X -v ON_ERROR_STOP=1 sessions, A and B, in the same disposable PostgreSQL 18 database. Run vacuum commands outside explicit transaction blocks. B creates five thousand rows, deletes the first 4,500, and performs ordinary vacuum:

CREATE SCHEMA compact_demo;
CREATE TABLE compact_demo.items (id integer PRIMARY KEY,payload text)
 WITH (autovacuum_enabled=false);
INSERT INTO compact_demo.items
SELECT g,repeat(md5(g::text),10) FROM generate_series(1,5000) AS g;
CREATE TABLE compact_demo.baseline AS
SELECT pg_relation_size('compact_demo.items') AS heap_bytes,
 pg_relation_filenode('compact_demo.items') AS file_id;
DELETE FROM compact_demo.items WHERE id<=4500;
VACUUM (ANALYZE) compact_demo.items;
SELECT pg_relation_size('compact_demo.items')=heap_bytes AS same_heap_size,
 pg_relation_filenode('compact_demo.items')=file_id AS same_file
FROM compact_demo.baseline;

The two comparisons return true: heap bytes and the relation’s file identifier remain unchanged. The surviving rows occupy the tail, so ordinary vacuum cannot shrink this particular file by truncating an empty end. This does not establish that ordinary vacuum never shrinks files; empty trailing pages can be returned when truncation is possible.

In A, read the remaining five hundred rows and keep the transaction open:

BEGIN;
SELECT count(*) FROM compact_demo.items;

A has made no updates, but still holds its table lock. In B, give the rewrite a short acquisition budget so the demonstration does not wait indefinitely:

SET lock_timeout='100ms';
\set ON_ERROR_STOP off
VACUUM (FULL) compact_demo.items;
\echo full_state :SQLSTATE
\set ON_ERROR_STOP on
RESET lock_timeout;

Expect full_state 55P03. The existing reader prevents the stronger lock from being acquired. Temporarily allowing the expected error shows the result without skipping cleanup, and resetting the timeout restores B’s original session setting. This timeout is a laboratory value, not a suggested production maintenance budget.

End the reader transaction in A:

COMMIT;

Now B can perform the rewrite, inspect storage identity, and verify every retained payload:

VACUUM (FULL,ANALYZE) compact_demo.items;
SELECT pg_relation_size('compact_demo.items')<heap_bytes AS smaller_heap,
 pg_relation_filenode('compact_demo.items')<>file_id AS different_file
FROM compact_demo.baseline;
SELECT count(*),min(id),max(id),sum(id),
 bool_and(payload=repeat(md5(id::text),10)) AS payloads_preserved
FROM compact_demo.items;
DO $$ BEGIN
 IF (SELECT sum(id) FROM compact_demo.items)<>2375250
 OR (SELECT count(*) FROM compact_demo.items)<>500
 OR EXISTS (SELECT FROM compact_demo.items
 WHERE payload<>repeat(md5(id::text),10)) THEN
 RAISE EXCEPTION 'compaction changed application data';
 END IF;
END $$;
DROP SCHEMA compact_demo CASCADE;

Both storage comparisons are true: the heap is smaller and its file identifier has changed. The data check returns five hundred rows, identifiers 4501 through 5000, an identifier sum of 2,375,250, and true payload preservation. The assertion checks the row population and content; a smaller file alone would not prove correct compaction. Finally, the fixture schema is removed.

Decide whether compaction pays for itself

If future inserts will reuse the free space, keeping it inside the relation may be preferable to paying for a rewrite and later growing the file again. Measure persistent excess space and access cost after the workload settles. Distinguish the main heap from indexes and auxiliary storage, because a single table-size number may combine several objects with different causes of growth.

Schedule the operation where its exclusive lock is acceptable, and check for long-running readers before requesting it. A low-traffic period does not guarantee no open transaction exists. Set an intentional acquisition budget, observe failure rather than endlessly retrying, and verify data and storage afterward. Keep ordinary vacuum healthy so severe accumulation does not become the normal maintenance cycle.

The experiment demonstrates lock conflict, file replacement, and preserved rows on a small table. It does not predict rewrite duration or spare-space needs for a production table with large indexes, external values, or replication traffic. Those quantities require measurements from the actual object and a maintenance plan that can tolerate the resulting lock window.

Sources

PostgreSQL 18 VACUUM, rewrite behavior, extra space, truncation, and transaction restrictions, checked 2026-09-10.
PostgreSQL 18 explicit locking, ACCESS EXCLUSIVE conflicts with readers and writers, checked 2026-09-10.
PostgreSQL 18 routine vacuuming, routine reuse versus exceptional full compaction, checked 2026-09-10.