Query · XB Field Notes

InnoDB covering indexes carry the query's columns

Fri, 03 Mar 2028 00:00:00 +0000

A covering index contains the columns needed to evaluate a particular query and return its requested values. In InnoDB, secondary-index records also carry the primary-key columns, so a query can be covered without explicitly listing the primary key again in the secondary-index definition. Coverage is a relationship between one query and an index, not a permanent property that applies to every query against the table.

When it breaks

Adding a projected column can remove coverage. An index that supplies an identifier and timestamp may not contain the large payload that a later application change begins selecting. The optimizer can still use that same index to locate matching rows, then obtain missing values from the clustered records. Index use and covering access are different facts.

Coverage also does not guarantee a selective access path. A wide scan can be fully covered while examining many entries. Column order, predicates, sorting requirements, and row estimates still matter. Do not interpret Using index as evidence that a query is fast, or confuse it with Using index condition, which describes index condition pushdown rather than complete projection coverage.

InnoDB visibility checks qualify the usual index-only explanation. Secondary records do not contain the same hidden transaction metadata as clustered records. Delete-marked secondary entries or pages changed by newer transactions can require clustered-record lookups to obtain the correct visible version. A covering plan is therefore not a universal promise of zero clustered access under concurrent MVCC activity.

Check it quickly

Use a fresh MySQL 8.4 instance and a client session with permission to create a disposable schema. The recorded run used MySQL 8.4.11. Ensure xbcatm_index_lab is unused, stop on unexpected SQL errors, and keep the same session so its selected database remains active. Create ten thousand rows, with one hundred paid orders and a deliberately wider payload:

CREATE DATABASE xbcatm_index_lab;
USE xbcatm_index_lab;
CREATE TABLE orders (
 id INT PRIMARY KEY,
 status VARCHAR(10) NOT NULL,
 created_at DATETIME NOT NULL,
 payload VARCHAR(1000) NOT NULL,
 INDEX by_status_created (status,created_at)
) ENGINE=InnoDB;
INSERT INTO orders
WITH digits(n) AS (
 SELECT 0 UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3
 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6
 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9
)
SELECT number+1,IF(MOD(number+1,100)=0,'paid','open'),
 TIMESTAMP '2026-01-01 00:00:00' + INTERVAL (number+1) SECOND,
 REPEAT('x',640)
FROM (SELECT a.n+10*b.n+100*c.n+1000*d.n AS number
 FROM digits a CROSS JOIN digits b CROSS JOIN digits c CROSS JOIN digits d) AS numbered;
ANALYZE TABLE orders;
SELECT COUNT(*),SUM(status='paid') FROM orders;

The final count must be ten thousand and one hundred. The secondary index contains status and creation time explicitly; InnoDB supplies the primary identifier implicitly. Inspect and run a query that needs only those values:

EXPLAIN SELECT id,created_at FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;
EXPLAIN ANALYZE SELECT id,created_at FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;
SELECT id,created_at FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;

The recorded tabular plan reports by_status_created and Using index. The analyzed plan identifies a covering index lookup and returns twenty rows. Result identifiers are one hundred, two hundred, and so on through two thousand, ordered by the timestamp and identifier. These are checked results on populated data, not a plan inferred from an empty schema.

Now request the payload as well:

EXPLAIN SELECT id,created_at,payload FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;
EXPLAIN ANALYZE SELECT id,created_at,payload FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;

The recorded plan still uses by_status_created, but loses Using index; the analyzed plan describes an index lookup without the covering qualification. This isolates the effect of changing the requested columns while keeping the filter, ordering, and index unchanged. The diagnostic execution does run the SELECT, so account for that when applying EXPLAIN ANALYZE to expensive real queries.

Remove the schema created for this exercise:

DROP DATABASE xbcatm_index_lab;

The timings from this tiny warm experiment are not a throughput or latency benchmark. Scheduling, caching, and instrumentation can outweigh the difference between two twenty-row reads. The verified conclusion concerns access shape and returned values, not a promised percentage reduction in physical I/O.

Pay for coverage deliberately

Appending columns to a secondary index consumes storage and buffer-pool capacity and increases maintenance when those indexed values change. Long primary keys enlarge secondary entries too. Cover the important narrow query rather than automatically copying a large payload into every index. Keep projected columns explicit so later application changes make their access requirements visible in review.

Compare representative plans, examined rows, foreground latency, and write cost before retaining a new index. Test with realistic selectivity and concurrent activity; a quiet fixture does not measure MVCC fallback frequency or replica maintenance cost. An index should earn its place through the workload it serves, not through the appearance of a desirable label in one explanation.

Sources

MySQL 8.4 clustered and secondary indexes, primary-key columns carried by secondary records, checked 2026-09-12.
MySQL 8.4 EXPLAIN output, index-use and covering-plan indicators, checked 2026-09-12.
MySQL 8.4 InnoDB multiversioning, clustered lookups required for secondary-index visibility checks, checked 2026-09-12.

Partition pruning skips whole partitions a query cannot need

Sun, 13 Feb 2028 00:00:00 +0000

Partition pruning excludes entire partitions that cannot contain matching rows. It is different from using an index to find rows within a remaining partition, and it is not a page-level cache trick. PostgreSQL can perform this exclusion while planning a query or later when parameter values become available during execution.

Prove which partitions cannot match

Declarative partitions have bounds derived from their partition key. A predicate restricting a date key to October can prove that September and November are irrelevant. Pruning uses those bounds, not an estimate that another partition probably contains few matches. Statistics still influence other planning decisions, but an inaccurate row estimate does not make partition membership a guess.

Indexes are not required for pruning. After excluding irrelevant partitions, PostgreSQL may choose a sequential scan of a remaining partition because most of its rows are needed. An Index Scan label is therefore not the test for successful pruning. Inspect the relation names and plan structure, then separately evaluate how each remaining relation is accessed.

A restriction must be usable with the partition key and its comparison semantics. Converting a date column to text can hide the direct bound relationship even when a particular string comparison happens to select the same rows. Use typed parameters and predicates that describe the intended interval directly. Do not assume that arbitrary expressions around a partition key preserve pruning or even preserve the original date semantics.

Parameters can prune during execution

A constant can allow planning-time pruning, leaving only the relevant child relation in the plan. A prepared statement’s generic plan cannot know every future parameter value in advance. That does not mean the statement must scan every partition: execution-time pruning can use values known when the plan initializes.

EXPLAIN ANALYZE reports Subplans Removed for partitions discarded during initialization. Other cases, such as parameters changing on the inner side of a nested-loop join, can prune during execution itself. Inspect loop counts and whether subplans actually ran when reasoning about those cases. A never-executed node alone can have other causes, such as an empty outer input, so interpret it in its full plan context.

Avoid turning these observations into a blanket rule that every nonconstant expression disables pruning. Parameter availability and expression support determine when exclusion is possible. Custom and generic prepared plans can differ while returning the same answer. The useful question is which partitions were accessed for the actual values used by the workload.

When it breaks

A filter on an unrelated column may leave every partition relevant. Casting the key into an incompatible expression can do the same. Check the exact predicate and enable_partition_pruning setting before changing the partition layout. Manually inventing a contradictory CHECK constraint is not a repair; ordinary local declarative partitions enforce their bounds, and constraint exclusion is a separate optimization using CHECK constraints.

Pruning also is not a promise of no locking or negligible planning cost. Partitions removed during execution initialization are still locked at the start of execution. Large partition hierarchies can consume planning time and per-session metadata memory even when a query eventually scans only a few children. Choose the hierarchy for realistic query and maintenance patterns rather than maximizing partition count.

Finally, pruning two partitions can still leave thousands of irrelevant rows inside the third. A matching local index may help that second stage. Returning only a hundred rows does not prove that only a hundred rows were examined, and buffer hits are not the same as physical disk reads.

Check it quickly

Save this as partition-pruning.sql and run psql -X -v ON_ERROR_STOP=1 -f partition-pruning.sql in a disposable PostgreSQL 18 database. The script enables pruning within its transaction and uses a generic prepared plan to expose initialization-time pruning. It creates no indexes on the temporary partitions.

\set ON_ERROR_STOP on
BEGIN;
SET LOCAL datestyle='ISO, YMD';
SET LOCAL enable_partition_pruning=on;
CREATE TEMP TABLE observations (day date,id integer) PARTITION BY RANGE(day);
CREATE TEMP TABLE obs_sep PARTITION OF observations FOR VALUES FROM ('2026-09-01') TO ('2026-10-01');
CREATE TEMP TABLE obs_oct PARTITION OF observations FOR VALUES FROM ('2026-10-01') TO ('2026-11-01');
CREATE TEMP TABLE obs_nov PARTITION OF observations FOR VALUES FROM ('2026-11-01') TO ('2026-12-01');
INSERT INTO observations SELECT DATE '2026-09-01'+g%90,g FROM generate_series(0,8999) AS g;
ANALYZE observations;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT count(*) FROM observations WHERE day>=DATE '2026-10-01' AND day<DATE '2026-11-01';
SET LOCAL plan_cache_mode=force_generic_plan;
PREPARE by_day(date) AS SELECT count(*) FROM observations WHERE day=$1;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) EXECUTE by_day(DATE '2026-10-10');
EXECUTE by_day(DATE '2026-10-10');
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT count(*) FROM observations WHERE day::text='2026-10-10';
DO $$ BEGIN
 IF (SELECT count(*) FROM observations WHERE day=DATE '2026-10-10') <> 100 OR
 (SELECT count(*) FROM observations WHERE day>=DATE '2026-10-01' AND day<DATE '2026-11-01') <> 3100 THEN
 RAISE EXCEPTION 'unexpected pruned result';
 END IF;
END $$;
DEALLOCATE by_day;
ROLLBACK;

PostgreSQL 18.6 planned the constant October range as a scan of obs_oct alone, reading 3,100 matching rows. The generic date lookup reported Subplans Removed: 2 and returned 100 rows, but still filtered 3,000 other rows within October. This demonstrates both successful partition exclusion and remaining local work.

The text-cast predicate scanned all three children and returned the same 100 rows under the explicitly chosen ISO DateStyle. Assertions verify the typed query’s counts, and DEALLOCATE plus rollback clean up the prepared statement and temporary tables. These results establish the predicate and pruning relationship for this fixture; measure representative plans separately before choosing an index or changing production partition granularity.

Sources

PostgreSQL table partitioning, planning and execution pruning, bounds, and locking, checked 2026-09-10.
PostgreSQL PREPARE, generic and custom plans and parameter handling, checked 2026-09-10.

A Common Table Expression names a query you can reuse

Sun, 19 Dec 2027 00:00:00 +0000

A common table expression gives a query result a name within one SQL statement. It can make a complex report readable by naming populations and intermediate calculations before the final SELECT. That name is not a persistent table, and it does not by itself promise a cached result, a particular scan count, or a sequence of procedural steps.

Scope first, execution second

A WITH clause belongs to its statement. A later statement cannot reuse its CTE name, and there is no schema object to index or query after execution. Use a temporary table, view, or another appropriate database object if results or definitions must outlive that statement. A CTE is useful when its scope and its named result match the work being expressed.

In modern PostgreSQL, a nonrecursive SELECT CTE without volatile functions can be folded into its parent query. By default, this happens when it is referenced once. Folding lets outer restrictions participate in optimization, so an outer primary-key lookup can reach an index on the underlying table instead of filtering a separately produced intermediate result.

When the same eligible CTE is referenced more than once, PostgreSQL normally materializes it instead. That can avoid repeating expensive calculations, but it can also prevent a restrictive condition at one use from being pushed into the shared computation. This distinction has applied since PostgreSQL 12; assuming that every WITH is always an optimization fence describes older behavior, not the current default.

Materialization is a tradeoff

MATERIALIZED requests separate computation. NOT MATERIALIZED permits folding even with repeated references, potentially repeating work while allowing each use to access only the rows it needs. PostgreSQL ignores NOT MATERIALIZED for recursive or non-side-effect-free CTEs; it does not authorize duplicating arbitrary writes or volatile operations.

A materialized SELECT CTE is not necessarily evaluated to exhaustion. Its consumers may demand only part of its result, though sorts, joins, or other operations can require more. Materialization also does not create a reusable indexed table for later statements. Inspect the plan and measure the actual workload before forcing a boundary merely because the word sounds like a cache.

The useful comparison is total work: repeated selective index probes may be cheaper than building and scanning a large shared result, while repeating an expensive calculation may be worse. A readable name and an efficient execution plan are compatible, but neither implies the other. Keep the business predicate in one authoritative definition and choose execution controls only when the evidence supports them.

When it breaks

Data-modifying CTEs follow different rules. INSERT, UPDATE, or DELETE inside WITH executes once and to completion even if its output is not consumed. The named relation is the RETURNING output, not a freshly rescanned target table. A modifying CTE without RETURNING still performs its modification but cannot supply rows to a later reference.

The modifying substatements and main statement use the same snapshot, and their actual modification order is not a procedural top-to-bottom guarantee. Pass changed rows through RETURNING when another part of the statement needs them. Avoid attempting to modify the same row twice in one statement and expecting textual order to decide the winner. A plain SELECT-looking outer query does not make a data-modifying WITH read-only.

Check it quickly

Save this as cte.sql and run psql -X -v ON_ERROR_STOP=1 -f cte.sql in a disposable PostgreSQL 18 database. It compares four read plans, then moves two temporary rows using DELETE RETURNING and INSERT. The final rollback removes the fixture and reverses its changes.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE items (id integer PRIMARY KEY, value text);
INSERT INTO items SELECT g,'item-'||g FROM generate_series(1,10000) AS g;
ANALYZE items;
EXPLAIN (COSTS OFF)
WITH source AS (SELECT * FROM items) SELECT * FROM source WHERE id=7;
EXPLAIN (COSTS OFF)
WITH source AS MATERIALIZED (SELECT * FROM items) SELECT * FROM source WHERE id=7;
EXPLAIN (COSTS OFF)
WITH source AS (SELECT * FROM items)
SELECT a.id FROM source a JOIN source b USING(id) WHERE a.id=7;
EXPLAIN (COSTS OFF)
WITH source AS NOT MATERIALIZED (SELECT * FROM items)
SELECT a.id FROM source a JOIN source b USING(id) WHERE a.id=7;
CREATE TEMP TABLE moved (id integer,value text);
WITH removed AS (DELETE FROM items WHERE id<=2 RETURNING *)
INSERT INTO moved SELECT * FROM removed;
SELECT (SELECT count(*) FROM items) AS remaining,(SELECT count(*) FROM moved) AS moved;
DO $$ BEGIN
 IF (SELECT array_agg(id ORDER BY id) FROM moved) IS DISTINCT FROM ARRAY[1,2] THEN
 RAISE EXCEPTION 'RETURNING handoff failed';
 END IF;
 IF (SELECT count(*) FROM items) <> 9998 THEN RAISE EXCEPTION 'DELETE did not execute'; END IF;
END $$;
ROLLBACK;

On PostgreSQL 18.6, the single-reference default plan used items_pkey directly. Explicit MATERIALIZED introduced a CTE Scan over a planned sequential scan. The default double reference also shared a materialized CTE, while NOT MATERIALIZED produced two index-based accesses. These plain EXPLAIN statements show planned structure; they do not measure execution time or prove a production performance improvement.

The modifying statement actually executed and left 9,998 rows in items and two in moved. Assertions verify that the transferred identifiers are exactly one and two and that the source rows were deleted. RETURNING provides the handoff between operations. Changing the outer statement’s demand is not a safe method for suppressing a modifying CTE; keep such experiments transactional and inspect all WITH bodies before executing unfamiliar SQL.

Sources

PostgreSQL WITH queries, folding, materialization, and data-modifying CTEs, checked 2026-09-10.
PostgreSQL SELECT reference, WITH scope and MATERIALIZED controls, checked 2026-09-10.

EXPLAIN plans reveal the route of a query

Fri, 21 May 2027 00:00:00 +0000

A slow query is a route problem before it is a tuning problem. EXPLAIN shows the planned route node by node, so the fix targets the step that actually costs time rather than a guess.

Read the plan as a route map

EXPLAIN prints a tree, and the root node is the answer while the leaves do the real work. Each node names an operation, a row estimate and an estimated cost, so instead of guessing you can look for the outlier: a scan whose row estimate dwarfs the number it eventually returns, or an estimate that is wildly off what the query actually touches. The arrow of blame is worth reading the way a map is: follow the node that accounts for most of the work, not the first one listed in the output. Comparing EXPLAIN alongside EXPLAIN ANALYZE shows the gap between what the planner predicted and what actually ran, which is where slow queries usually hide.

Statistics steer the chosen route

An index is a candidate, not a guarantee: the planner chooses the route from table statistics such as row counts, nulls and column value distribution, and the same query can change routes after an auto-vacuum, a bulk load or a changed cache ratio. That means a hot query deserves a re-check when the data profile shifts, not just when it is new. When statistics are stale, ANALYZE (or the database’s auto-analysis) can change the plan more than most manual rewrites. Treat the plan as evidence to re-verify against real EXPLAIN ANALYZE numbers rather than a snapshot to memorise.

Confirm the route moved

Make the plan part of the change loop: capture a baseline EXPLAIN ANALYZE, apply an index or a rewritten query, and confirm the route actually shifted before you declare victory. If the cost node you were targeting still shows the same large step after a rewrite, the change is cosmetic rather than structural, no matter how reasonable it sounds. IQ and experience are a poor substitute for comparing before and after plans; the same holds for measured timing and rows fetched. When the new plan wins, keep the before and after as evidence in the merge description so the gain is attributable and repeatable.

When it breaks

Stale statistics pick the wrong road. After a bulk load or a long stretch without ANALYZE, the planner may still believe a table holds thousands of rows when it holds millions, and it keeps favoring an index or nested-loop path that is only cheap at small scale. EXPLAIN ANALYZE gives the tell: estimated rows differ from actual rows by orders of magnitude. Re-run ANALYZE and compare those two numbers before adding any index.

A prepared statement can lock in the wrong plan for the wrong value. A plan cached around a highly selective first parameter looks fast for that case and slow for a value matching half the table, yet the literal tested with EXPLAIN is rarely the value production binds. Probe both extremes and check whether a custom or generic plan was chosen.

Implicit conversion silently retires the index. Comparing a varchar column to a bare number, mixing collations, or wrapping the column in LOWER() or a date truncation forces a scan plus filter even though the index exists. The symptom is a Seq Scan whose rows removed by filter dwarf the rows returned. Fix the type at the call site, or add a matching expression index.

Cost alone is the last trap. Cost is a unitless planning estimate, not time: the cheapest plan can still pile up shared-buffer hits or spill sorts to temp files. Confirm with EXPLAIN (ANALYZE, BUFFERS), reading actual time, buffers, and rows removed by filter next to the cost before calling a route healthy.

Check it quickly

Save the following as plan_route_check.py in a scratch directory and run python3 plan_route_check.py. It uses only the standard library to model two candidate routes — a sequential scan and an index scan — under a stale row estimate, then re-scores the chosen route at the actual row count. The script prints estimated and true costs, the estimate error, and a stale-statistics warning, and asserts that the warning fires:

#!/usr/bin/env python3
"""Toy planner model: estimated cost picks a route, actual rows judge it.

Stdlib only. It simulates stale statistics: the planner believes 500 rows
match, while 600,000 really do, and checks that the estimate-error alarm fires.
"""

from __future__ import annotations


def seq_scan_cost(pages: float, rows: float) -> float:
 return pages * 1.0 + rows * 0.01


def index_scan_cost(selectivity_est: float, pages: float, rows: float) -> float:
 heap_pages = selectivity_est * pages
 return 4.0 + heap_pages * 4.0 + selectivity_est * rows * 0.01


def main() -> None:
 pages, rows = 8_000.0, 1_000_000.0
 stale_selectivity = 0.0005 # planner belief: 500 rows
 true_selectivity = 0.60 # reality: 600,000 rows
 est_rows = stale_selectivity * rows
 actual_rows = true_selectivity * rows

 seq = seq_scan_cost(pages, rows)
 idx_est = index_scan_cost(stale_selectivity, pages, rows)
 idx_true = index_scan_cost(true_selectivity, pages, rows)
 chosen = "index scan" if idx_est < seq else "seq scan"
 print(f"estimated rows: {est_rows:.0f}; actual rows: {actual_rows:.0f}")
 print(f"estimated cost: seq={seq:.1f} index={idx_est:.1f} -> planner picks {chosen}")
 print(f"true cost at actual selectivity: seq={seq:.1f} index={idx_true:.1f}")

 error = abs(actual_rows - est_rows) / actual_rows
 print(f"estimate error: {error:.1%}")
 warned = False
 if error > 0.50:
 warned = True
 print("WARNING: row estimate off by >50% - statistics are stale, re-run ANALYZE.")
 if chosen == "index scan" and idx_true > seq:
 warned = True
 print("WARNING: chosen route loses at actual selectivity - check EXPLAIN ANALYZE.")
 assert warned, "expected a stale-estimate warning for this fixture"
 assert est_rows < actual_rows / 10, "fixture must keep the estimate an order of magnitude low"
 print("PASS: planner picked the stale route and the estimate-error alarm fired")


if __name__ == "__main__":
 main()

Expect the planner to pick the index scan on the stale estimate, an estimate error near 100%, and a final PASS line. The numbers are a toy model of one failure mode, not a real optimizer: page costs are constants and only two paths compete. They train the habit that matters — estimate first, then confront the estimate with actual rows — before reading a production plan. The fixture touches no database and needs no packages; re-check any plan change against real EXPLAIN (ANALYZE, BUFFERS) output before shipping it.

Sources

PostgreSQL: EXPLAIN, generating the plan, checked 2027-05-19.
PostgreSQL: using EXPLAIN, interpreting the output, checked 2027-05-19.
PostgreSQL: planner/optimizer, how routes are chosen, checked 2027-05-19.
MySQL: EXPLAIN statement, plan format and output columns, checked 2027-05-19.

Query · XB Field Notes

InnoDB covering indexes carry the query's columns

When it breaks

Check it quickly

Pay for coverage deliberately

Sources

Partition pruning skips whole partitions a query cannot need

Prove which partitions cannot match

Parameters can prune during execution

When it breaks

Check it quickly

Sources

A Common Table Expression names a query you can reuse

Scope first, execution second

Materialization is a tradeoff

When it breaks

Check it quickly

Sources

EXPLAIN plans reveal the route of a query

Read the plan as a route map

Statistics steer the chosen route

Confirm the route moved

When it breaks

Check it quickly

Further reading

Sources