Planner · XB Field Notes

Query planners bet on cost, not certainty

Fri, 17 Sep 2027 00:00:00 +0000

A query planner chooses an execution path using estimated work. It does not run every candidate, time them, and keep the winner. PostgreSQL compares possible scans, joins, sorts, and other operations using statistics and a cost model. The chosen plan is therefore a prediction under current assumptions. Understanding a wrong prediction requires identifying which assumption failed, rather than treating a sequential scan or a nested loop as inherently suspicious.

Costs describe a model

PostgreSQL costs combine estimates of page access, tuple processing, operator work, and related activity. These units are not milliseconds. Startup cost matters when a consumer needs only a few rows, while total cost matters when the full result must be produced. Useful ordering can avoid a later sort, so the cheapest isolated scan is not necessarily the best input to the complete query.

The search also has limits. PostgreSQL considers alternative paths and join orders rather than executing them all. Large join problems can use the genetic query optimizer to control planning effort. Available indexes, legal transformations, configuration, and search choices constrain the candidates. Calling the selected plan mathematically optimal over every imaginable implementation overstates what the optimizer guarantees.

When it breaks

Fresh statistics can still produce a bad estimate. Ordinary statistics describe columns individually, so combining two predicates may assume independence even when the columns always move together. An underestimated intermediate result can make repeated lookups appear cheaper than they become, or distort later join choices. Repeating the same analysis without changing the statistical model cannot teach it a missing relationship.

Extended statistics address particular questions, not every estimation error. Functional dependencies help compatible equality conditions against constants, but do not establish the actual mapping between values. They are not general proofs for range filters or cross-column comparisons. An incompatible pair of constants can remain badly estimated even after a dependency is collected.

Check it quickly

Run this temporary-table experiment with PostgreSQL 18 and psql. Both columns contain the same one hundred distinct values, each repeated one hundred times. The final rollback removes the table and its statistics object.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE dimensions (a integer,b integer);
INSERT INTO dimensions SELECT g%100,g%100 FROM generate_series(1,10000) AS g;
ANALYZE dimensions;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT * FROM dimensions WHERE a=7 AND b=7;
CREATE STATISTICS pg_temp.dimensions_dependency (dependencies) ON a,b FROM dimensions;
ANALYZE dimensions;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT * FROM dimensions WHERE a=7 AND b=7;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT * FROM dimensions WHERE a=7 AND b=8;
DO $$ BEGIN
 IF (SELECT count(*) FROM dimensions WHERE a=7 AND b=7) <> 100 OR
 (SELECT count(*) FROM dimensions WHERE a=7 AND b=8) <> 0 THEN RAISE EXCEPTION 'wrong correlated population'; END IF;
END $$;
ROLLBACK;

Before the extended statistics, PostgreSQL 18.6 estimated one row for a=7 AND b=7, although the query returned one hundred. Each condition individually selects one percent; multiplying those fractions predicts one hundredth of a percent of ten thousand rows. The calculation is reasonable only under an independence assumption that this fixture deliberately violates.

Creating the dependencies object tells the next ANALYZE which relationship to collect. The second plan estimated one hundred rows and returned one hundred. Merely defining the object would not have populated it. The scan remained sequential because this example has no index and still examines the same small table. A better cardinality estimate is demonstrated; a faster application is not.

The final predicate asks for a=7 AND b=8. It returns zero rows, but the dependency estimate remains one hundred. This is the important negative case: knowing that one column determines another does not tell this statistics kind that seven must map specifically to seven. The assertions distinguish correct query results from inaccurate predictions; estimation errors do not change SQL semantics.

Choose the next experiment from the discrepancy

A multivariate common-value list can represent frequent value combinations when that is the missing information. Other workloads may need different statistics, an appropriate index, or a rewritten expression that exposes a usable condition. Add only the object justified by the observed predicates; unnecessary statistics consume analysis and planning resources too.

When comparing alternatives, keep the dataset, parameters, relevant settings, and result requirements consistent. Include rows processed, buffer activity, memory spills, and measured latency. A locally cheaper-looking node is not sufficient evidence if it creates more work upstream. Conversely, accurate row estimates do not rule out waiting, costly I/O, or an unsuitable cost model. Follow the specific difference between predicted and observed work before changing global planner settings.

Sources

PostgreSQL 18: Planner and optimizer, candidate access paths and join strategies, checked 2026-09-10.
PostgreSQL 18: Planner statistics, column distributions and extended-statistics limits, checked 2026-09-10.
PostgreSQL 18: Multivariate statistics examples, correlated predicates and dependency estimates, checked 2026-09-10.

Joins pick a strategy from the sizes both sides bring

Sat, 01 May 2027 00:00:00 +0000

A join strategy connects two input streams; it is not a permanent property of the SQL text. Restricting one side to three customers can make indexed repeated lookups attractive, while joining every purchase may favor scanning once and building a hash table. PostgreSQL chooses among legal alternatives using estimated work, available access paths, ordering, and memory expectations. Table size matters, but the size after filtering matters more than a table’s name or reputation.

Repeated lookups and a reusable lookup table

A nested loop takes an outer row and executes an inner path using that row. The inner path need not scan the entire table: it can be an index or bitmap lookup. A small outer population and selective inner access can make repeated work economical. Read the inner node’s loops alongside its per-loop rows to see the total amount requested.

A hash join builds a temporary hash table from one input and probes it with the other. This execution structure is unrelated to a persistent index created with USING hash; creating such an index is not a prerequisite. The build side is the subtree below the plan’s Hash node. Inspect it instead of assuming that SQL’s left or right spelling, or the smaller base table, always determines the build.

Merge joins provide another alternative when suitable ordered inputs are available or worth sorting. Consequently, comparing nested loops with hashes does not exhaust the planner’s choices. Different predicates, output requirements, and legal join conditions can change which alternatives make sense.

When it breaks

An underestimated outer population can turn an inexpensive inner lookup into excessive repeated work. A hash table can also exceed its memory budget and use multiple batches with temporary I/O; it is not guaranteed to remain entirely in memory. Check batch counts and temporary-buffer activity instead of assuming the word Hash promises linear, constant-cost performance.

Large numbers of duplicate join keys can produce a huge result regardless of the lookup strategy. No algorithm avoids producing the rows that the query actually requests. Fixing an accidental many-to-many join requires correcting the relationship or query semantics, not simply selecting a different node.

Plain PostgreSQL does not interpret optimizer-hint comments such as /*+ HashJoin */. Extensions can add hint behavior, but that is a separate environment contract. Planner switches can support controlled comparisons; keeping a globally disabled join method after one small experiment can damage unrelated workloads.

Check it quickly

Run this transaction in PostgreSQL 18 with psql. It creates five hundred customers and fifty thousand purchases, one hundred purchases per customer, with a B-tree index on the reference column. Both statements compute sums over the same relationship, with different filtering.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE customers (id integer PRIMARY KEY);
CREATE TEMP TABLE purchases (id integer PRIMARY KEY,customer_id integer,amount integer);
INSERT INTO customers SELECT g FROM generate_series(1,500) AS g;
INSERT INTO purchases SELECT g,1+(g-1)%500,10 FROM generate_series(1,50000) AS g;
CREATE INDEX purchases_customer ON purchases(customer_id);
ANALYZE customers;
ANALYZE purchases;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT sum(p.amount) FROM customers c JOIN purchases p ON p.customer_id=c.id WHERE c.id<=3;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT sum(p.amount) FROM customers c JOIN purchases p ON p.customer_id=c.id;
DO $$ BEGIN
 IF (SELECT sum(p.amount) FROM customers c JOIN purchases p ON p.customer_id=c.id WHERE c.id<=3)
 IS DISTINCT FROM 3000::bigint THEN RAISE EXCEPTION 'wrong selective join'; END IF;
 IF (SELECT sum(p.amount) FROM customers c JOIN purchases p ON p.customer_id=c.id)
 IS DISTINCT FROM 500000::bigint THEN RAISE EXCEPTION 'wrong broad join'; END IF;
END $$;
ROLLBACK;

On PostgreSQL 18.6, filtering customers to identifiers at most three produced a nested loop. Its inner bitmap heap scan returned one hundred purchases per loop and ran three times. The join emitted three hundred rows, yielding a sum of 3,000. Seeing a bitmap scan inside a nested loop is expected: the node describes how the inner rows are obtained, while the join describes repeated combination.

Without the customer filter, the plan used a hash join. It built from five hundred customers and scanned fifty thousand purchases, yielding 500,000. The measured hash used one batch; that observation applies to this fixture, not all hash joins. The SQL assertions verify both sums, so a different plan must still return the same answers.

Compare equivalent work before choosing

These two statements request different result populations. Their elapsed times cannot establish that one algorithm is universally faster. To compare strategies for one workload, keep its parameters, output, dataset, and concurrency constant, then examine total loops, rows, buffers, memory, and latency. Refresh stale statistics when the divergence supports that diagnosis, and inspect the useful inner access path when repeated work dominates. A correct response may be a targeted index or a better estimate; it need not be a permanent preference for either join algorithm.

Sources

PostgreSQL 18: Planner and optimizer, candidate access paths and join strategies, checked 2026-09-10.
PostgreSQL 18: Using EXPLAIN, row counts and loops and plan interpretation, checked 2026-09-10.
PostgreSQL 18: Query planning configuration, planner method switches and their scope, checked 2026-09-10.

A partial index stores only the rows that matter

Tue, 27 Apr 2027 00:00:00 +0000

A partial index can keep a frequently queried subset of rows in a smaller structure. An index on job id WHERE status is open is a useful candidate when open jobs are rare and most requests target them. Its availability depends on what the planner can prove about each query, not merely on whether the application happens to pass “open” during one execution.

Prove that excluded rows are unnecessary

The index has no entries for closed jobs, so it cannot by itself answer a query that might need them. PostgreSQL must recognize at planning time that the query condition implies the index predicate. An explicit status = 'open' supplies that fact. A generic parameter status = $1 does not, because the same plan must remain valid when the parameter is closed.

Implication is not always textual identity. PostgreSQL recognizes some simple logical relationships, but it is not a general theorem prover. Standardizing the intended predicate helps prevent refactors from hiding an otherwise usable index. A condition on the indexed id alone cannot establish the status restriction, even if current data or application conventions make the row usually open.

Prepared plans change the evidence available

Prepared statements can use custom plans based on supplied parameter values or generic plans reusable across values. A custom plan for status open can know the needed predicate and consider the partial index. A generic plan with an unknown status cannot safely use that index as the complete answer path. The application’s SQL text has not changed, but the information available during planning has.

This does not mean parameterized queries are incompatible with partial indexes. A query with literal status = 'open' and a parameter only for id retains the predicate in a generic plan. Nor should parameters be replaced by string concatenation: parameter binding remains the correct way to carry untrusted values. Design the query around its real invariant rather than weakening data handling to chase an index.

The default plan policy chooses between custom and generic plans using its own cost considerations. For diagnosis, plan_cache_mode can force either behavior in a session. Such forcing is an experiment, not an instruction to disable generic plans globally. Replanning costs, parameter skew, and execution frequency determine the appropriate tradeoff.

When it breaks

A query can become slower after repeated prepared execution when a reusable plan replaces a parameter-specific one. The symptom may look like random index disappearance. Capture the actual prepared SQL and inspect EXPLAIN EXECUTE with representative values; an ad hoc literal query does not reproduce every prepared-plan condition. Confirm the predicate, plan mode, estimated rows, and runtime work together.

The subset can also stop being small. A backlog that leaves most jobs open reduces the expected size advantage, and status transitions still require index maintenance. Statistics and production-like distributions are needed before claiming lower write cost or better cache residency. A partial index is a workload decision, not a guarantee that the table’s future state will stay favorable.

Check it quickly

Save this as partial-plan.sql and run psql -X -v ON_ERROR_STOP=1 -f partial-plan.sql in a disposable PostgreSQL 18 database. The temporary table has one hundred open jobs among ten thousand rows, and the index includes only that open subset.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE jobs (id integer, status text NOT NULL);
INSERT INTO jobs SELECT g, CASE WHEN g<=100 THEN 'open' ELSE 'closed' END
FROM generate_series(1,10000) AS g;
CREATE INDEX jobs_open_id ON jobs (id) WHERE status='open';
ANALYZE jobs;
PREPARE chosen(text,integer) AS SELECT id FROM jobs WHERE status=$1 AND id=$2;
SET LOCAL plan_cache_mode=force_generic_plan;
EXPLAIN (COSTS OFF) EXECUTE chosen('open',7);
EXECUTE chosen('open',7);
SET LOCAL plan_cache_mode=force_custom_plan;
EXPLAIN (COSTS OFF) EXECUTE chosen('open',7);
EXECUTE chosen('open',7);
PREPARE always_open(integer) AS SELECT id FROM jobs WHERE status='open' AND id=$1;
SET LOCAL plan_cache_mode=force_generic_plan;
EXPLAIN (COSTS OFF) EXECUTE always_open(7);
EXECUTE always_open(7);
DEALLOCATE chosen;
DEALLOCATE always_open;
ROLLBACK;

On PostgreSQL 18.6, the generic plan with unknown status used a sequential scan. Forcing a custom plan for the same prepared statement and open value used jobs_open_id. The second statement, whose status predicate is a literal and only its id is parameterized, also used that partial index under a generic plan. All three executions returned id seven.

These plans demonstrate why a known predicate matters, not a promise that every machine selects the same access path. The explicit SET LOCAL changes end with rollback, and DEALLOCATE removes the prepared statements. The test does not reproduce automatic plan switching under a real connection pool; that requires observing the application’s actual sessions and repeated execution history.

Sources

PostgreSQL partial indexes, predicate implication at planning time, checked 2026-09-10.
PostgreSQL PREPARE, custom and generic execution plans, checked 2026-09-10.

EXPLAIN estimates need execution context

Fri, 19 Mar 2027 00:00:00 +0000

EXPLAIN describes the plan PostgreSQL intends to execute. EXPLAIN ANALYZE executes that plan and adds measurements. Comparing them is useful only after interpreting what each number counts. An apparent mismatch can reflect stale estimates, repeated execution, or a parent that stopped requesting rows. Matching cardinalities strengthen one part of the model; they do not prove the chosen plan is the fastest possible plan.

Read a node in its context

Estimated rows describes the output of a node, not necessarily all tuples it examines. A scan may examine thousands of tuples and emit ten after filtering. Startup cost and total cost are planner units, not milliseconds. Parent costs include child work, so adding every node’s cost double-counts the tree. Actual parent timing is also generally inclusive rather than a list of independent charges.

For repeatedly executed nodes, actual row counts and timings are reported per execution, averaged across loops. A node returning two rows with three loops supplied six rows in total. Filter-removal counts are likewise per-loop averages, so rounding can matter. Start at the leaves and follow where the observed population first differs materially from the estimate, while respecting those execution boundaries.

A LIMIT, EXISTS, or another early-stopping parent may leave a child only partly consumed. The child’s estimate can describe its full potential output while the actual count describes only rows requested. That is not automatically a statistics error. Sorts can also read their entire input while delivering only the requested top rows.

When it breaks

Running EXPLAIN ANALYZE on a write actually performs the write. A rollback can protect transactional table changes, but does not reverse every possible side effect, such as sequence advancement or an external action invoked by a function. Choose a disposable fixture or understand the complete statement before measuring it.

Another mistake is diagnosing storage from BUFFERS alone. A shared-buffer hit avoids a PostgreSQL buffer miss; a reported read can still be served by the operating system cache. Buffer counts describe block accesses, not distinct blocks or guaranteed physical-device reads. Likewise, accurate rows do not exclude lock waits, memory spills, configuration problems, or an access path that the search did not select.

Check it quickly

Use PostgreSQL 18 and psql. The transaction creates three parents and thirty children, then compares a repeated lateral lookup with a simple top-three query. All data is temporary and the final rollback removes the fixture.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE parents (id integer PRIMARY KEY);
CREATE TEMP TABLE children (parent_id integer,value integer);
INSERT INTO parents VALUES (1),(2),(3);
INSERT INTO children SELECT 1+(g-1)%3,g FROM generate_series(1,30) AS g;
ANALYZE parents;
ANALYZE children;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT p.id,c.value FROM parents p CROSS JOIN LATERAL (
 SELECT value FROM children WHERE parent_id=p.id ORDER BY value DESC LIMIT 2
) c;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT value FROM children ORDER BY value DESC LIMIT 3;
SELECT p.id,c.value FROM parents p CROSS JOIN LATERAL (
 SELECT value FROM children WHERE parent_id=p.id ORDER BY value DESC LIMIT 2
) c ORDER BY p.id,c.value DESC;
DO $$ BEGIN
 IF (SELECT array_agg(value ORDER BY parent_id,value DESC) FROM (
 SELECT p.id AS parent_id,c.value FROM parents p CROSS JOIN LATERAL (
 SELECT value FROM children WHERE parent_id=p.id ORDER BY value DESC LIMIT 2
 ) c
 ) s) IS DISTINCT FROM ARRAY[28,25,29,26,30,27] THEN RAISE EXCEPTION 'wrong lateral result'; END IF;
END $$;
ROLLBACK;

The PostgreSQL 18.6 lateral plan used a nested loop. Its inner Limit reported actual rows of two and loops=3, producing six joined rows. The child scan returned ten matching rows per loop and removed twenty, so it examined all thirty children on each of three passes. Read those numbers together instead of concluding that the whole query inspected only ten rows.

The final ordered results were parent one with values 28 and 25, parent two with 29 and 26, and parent three with 30 and 27. The assertion checks those memberships independently of the plan’s appearance. Correct answers remain required even if another PostgreSQL version chooses different access nodes.

For the separate top-three query, the sort estimated thirty output rows but emitted three because its parent stopped there. Its scan still read thirty rows. The fixture had just been analyzed, and this mismatch demonstrates consumption semantics rather than evidence that refreshing statistics would fix anything.

Measure the question you need answered

TIMING OFF keeps actual row counts and overall execution time while avoiding repeated per-node clock reads. BUFFERS adds useful context about page access, and JSON output can support systematic comparison. Neither option makes the instrumented query identical to application latency: client transmission, rendering, connection setup, and concurrent load need their own observations. Preserve the parameters and relevant session settings when comparing plans, then explain a change in terms of rows processed, loops, spills, or waits rather than declaring every new node name a regression.

Sources

PostgreSQL 18: EXPLAIN, execution effects and measurement options, checked 2026-09-10.
PostgreSQL 18: Using EXPLAIN, row counts and loops and plan interpretation, checked 2026-09-10.

ANALYZE updates the estimates the planner relies on

Fri, 12 Mar 2027 00:00:00 +0000

ANALYZE updates the statistical model PostgreSQL uses to estimate query results. It does not rewrite the query, rebuild its indexes, remove dead tuples, or promise a faster plan. Its useful question is specific: does the planner understand the current distribution well enough to compare access paths? A bulk status change can invalidate that understanding even when the table still contains exactly the same number of live rows.

What the statistics describe

For ordinary columns, pg_stats exposes information such as null fractions, common values and their frequencies, distinct-value estimates, and histograms. The planner combines that information with table size and cost settings. Knowing that one percent of jobs are ready suggests a different amount of work from knowing that ninety percent are ready. An index can help the former while reading the table sequentially can suit the latter.

On large tables, analysis samples rows. More detailed statistics require a larger sample and additional work; sampling does not mean every analysis is cheap. Approximate statistics can change slightly between runs without any application change. Raising a particular column’s statistics target can improve a poorly represented distribution, but it cannot teach ordinary single-column statistics every relationship between columns.

When it breaks

After a bulk update, an old common-value frequency may badly understate a filter’s output. A nested loop can consequently repeat far more work than expected, or an index scan can fetch most of the table. However, an unused index alone is not evidence of failure: a broad query may legitimately prefer a sequential scan.

Repeatedly analyzing a table while a query waits on another transaction will not release that transaction’s lock. Statistics do not create missing indexes or resolve connection waits. Required ordering still needs a sort or an already ordered access path. Compare the first meaningful cardinality divergence with the observed bottleneck before deciding that statistics are the cause.

Check it quickly

Run this in psql against a disposable PostgreSQL 18 database. It uses a temporary table inside a transaction and finishes with rollback. Temporary tables are deliberately useful here because autovacuum cannot analyze them between the controlled measurements.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE jobs (id integer,ready boolean,payload text);
INSERT INTO jobs SELECT g,g<=100,repeat('x',100) FROM generate_series(1,10000) AS g;
CREATE INDEX jobs_ready ON jobs(ready);
ANALYZE jobs;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT id FROM jobs WHERE ready;
UPDATE jobs SET ready=true WHERE id<=9000;
SELECT most_common_vals,most_common_freqs FROM pg_stats
WHERE schemaname=(SELECT nspname FROM pg_namespace WHERE oid=pg_my_temp_schema())
 AND tablename='jobs' AND attname='ready';
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT id FROM jobs WHERE ready;
ANALYZE jobs;
SELECT most_common_vals,most_common_freqs FROM pg_stats
WHERE schemaname=(SELECT nspname FROM pg_namespace WHERE oid=pg_my_temp_schema())
 AND tablename='jobs' AND attname='ready';
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT id FROM jobs WHERE ready;
DO $$ BEGIN
 IF (SELECT count(*) FROM jobs WHERE ready) <> 9000 THEN RAISE EXCEPTION 'data changed during analysis'; END IF;
END $$;
ROLLBACK;

On PostgreSQL 18.6, the first query estimated and returned 100 ready jobs through jobs_ready. The update changed the ready population to 9,000. Before the second analysis, pg_stats still displayed {f,t} with frequencies {0.99,0.01}: the distribution model had not learned the change.

The intermediate plan estimated 190 rows and returned 9,000. Do not demand that the estimate remain exactly 100: PostgreSQL can scale relation estimates using current physical size, which changed after the update. The stale one-percent frequency, not one frozen row-count number, is the decisive observation.

After ANALYZE, the frequencies became {0.9,0.1} for {t,f}, and this environment selected a sequential scan estimating and returning 9,000 rows. The assertion confirms that analysis did not modify the qualifying population. Plan shapes, costs, and timings can vary elsewhere; the distribution change and returned counts explain the result without claiming a universal speedup.

Keep the refresh tied to evidence

Autovacuum normally schedules automatic analysis for ordinary tables after sufficient changes, so permanent blanket reductions in its thresholds are not a default remedy. Check whether analysis actually lagged the workload. Bulk imports and known distribution changes can justify an explicit analysis before latency-sensitive traffic resumes.

Partitioned parents need separate attention: automatic analysis of their leaf tables does not automatically maintain the parent’s hierarchy statistics. Temporary tables also require manual analysis in their owning session. In PostgreSQL 18, table maintenance privileges govern who can analyze a table; examine skipped-table messages instead of treating a completed command as proof that every intended table was refreshed. Recheck the affected plan and application latency after the refresh, then investigate other causes if the evidence no longer points to statistics.

Sources

PostgreSQL 18: ANALYZE, sampling and statistics refresh boundaries, checked 2026-09-10.
PostgreSQL 18: Statistics used by the planner, column distributions and extended-statistics limits, checked 2026-09-10.