Index · XB Field Notes

InnoDB covering indexes carry the query's columns

Fri, 03 Mar 2028 00:00:00 +0000

A covering index contains the columns needed to evaluate a particular query and return its requested values. In InnoDB, secondary-index records also carry the primary-key columns, so a query can be covered without explicitly listing the primary key again in the secondary-index definition. Coverage is a relationship between one query and an index, not a permanent property that applies to every query against the table.

When it breaks

Adding a projected column can remove coverage. An index that supplies an identifier and timestamp may not contain the large payload that a later application change begins selecting. The optimizer can still use that same index to locate matching rows, then obtain missing values from the clustered records. Index use and covering access are different facts.

Coverage also does not guarantee a selective access path. A wide scan can be fully covered while examining many entries. Column order, predicates, sorting requirements, and row estimates still matter. Do not interpret Using index as evidence that a query is fast, or confuse it with Using index condition, which describes index condition pushdown rather than complete projection coverage.

InnoDB visibility checks qualify the usual index-only explanation. Secondary records do not contain the same hidden transaction metadata as clustered records. Delete-marked secondary entries or pages changed by newer transactions can require clustered-record lookups to obtain the correct visible version. A covering plan is therefore not a universal promise of zero clustered access under concurrent MVCC activity.

Check it quickly

Use a fresh MySQL 8.4 instance and a client session with permission to create a disposable schema. The recorded run used MySQL 8.4.11. Ensure xbcatm_index_lab is unused, stop on unexpected SQL errors, and keep the same session so its selected database remains active. Create ten thousand rows, with one hundred paid orders and a deliberately wider payload:

CREATE DATABASE xbcatm_index_lab;
USE xbcatm_index_lab;
CREATE TABLE orders (
 id INT PRIMARY KEY,
 status VARCHAR(10) NOT NULL,
 created_at DATETIME NOT NULL,
 payload VARCHAR(1000) NOT NULL,
 INDEX by_status_created (status,created_at)
) ENGINE=InnoDB;
INSERT INTO orders
WITH digits(n) AS (
 SELECT 0 UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3
 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6
 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9
)
SELECT number+1,IF(MOD(number+1,100)=0,'paid','open'),
 TIMESTAMP '2026-01-01 00:00:00' + INTERVAL (number+1) SECOND,
 REPEAT('x',640)
FROM (SELECT a.n+10*b.n+100*c.n+1000*d.n AS number
 FROM digits a CROSS JOIN digits b CROSS JOIN digits c CROSS JOIN digits d) AS numbered;
ANALYZE TABLE orders;
SELECT COUNT(*),SUM(status='paid') FROM orders;

The final count must be ten thousand and one hundred. The secondary index contains status and creation time explicitly; InnoDB supplies the primary identifier implicitly. Inspect and run a query that needs only those values:

EXPLAIN SELECT id,created_at FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;
EXPLAIN ANALYZE SELECT id,created_at FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;
SELECT id,created_at FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;

The recorded tabular plan reports by_status_created and Using index. The analyzed plan identifies a covering index lookup and returns twenty rows. Result identifiers are one hundred, two hundred, and so on through two thousand, ordered by the timestamp and identifier. These are checked results on populated data, not a plan inferred from an empty schema.

Now request the payload as well:

EXPLAIN SELECT id,created_at,payload FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;
EXPLAIN ANALYZE SELECT id,created_at,payload FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;

The recorded plan still uses by_status_created, but loses Using index; the analyzed plan describes an index lookup without the covering qualification. This isolates the effect of changing the requested columns while keeping the filter, ordering, and index unchanged. The diagnostic execution does run the SELECT, so account for that when applying EXPLAIN ANALYZE to expensive real queries.

Remove the schema created for this exercise:

DROP DATABASE xbcatm_index_lab;

The timings from this tiny warm experiment are not a throughput or latency benchmark. Scheduling, caching, and instrumentation can outweigh the difference between two twenty-row reads. The verified conclusion concerns access shape and returned values, not a promised percentage reduction in physical I/O.

Pay for coverage deliberately

Appending columns to a secondary index consumes storage and buffer-pool capacity and increases maintenance when those indexed values change. Long primary keys enlarge secondary entries too. Cover the important narrow query rather than automatically copying a large payload into every index. Keep projected columns explicit so later application changes make their access requirements visible in review.

Compare representative plans, examined rows, foreground latency, and write cost before retaining a new index. Test with realistic selectivity and concurrent activity; a quiet fixture does not measure MVCC fallback frequency or replica maintenance cost. An index should earn its place through the workload it serves, not through the appearance of a desirable label in one explanation.

Sources

MySQL 8.4 clustered and secondary indexes, primary-key columns carried by secondary records, checked 2026-09-12.
MySQL 8.4 EXPLAIN output, index-use and covering-plan indicators, checked 2026-09-12.
MySQL 8.4 InnoDB multiversioning, clustered lookups required for secondary-index visibility checks, checked 2026-09-12.

Partial indexes trim a column

Tue, 15 Feb 2028 00:00:00 +0000

Not every row deserves an index slot; a partial index keeps the tree thin exactly where the query never looks. Most tables have a skew the schema pretends not to see: ninety percent of orders are closed, nearly all events are processed, the flag column is false almost everywhere. A full index dutifully records all of it, charging maintenance on every write to protect lookups that only ever touch the small live slice. The partial index stops paying for the rows nobody queries.

The bargain is precision for size. The index becomes smaller, faster to scan, and cheaper to maintain — but only queries whose predicates match the index condition may use it. Everything else falls back to competing plans. That makes partial indexes a deliberate pairing of access pattern with storage, not a general speedup. Choose them where the hot subset is stable and the queries agree on its definition.

Index only what you query

A partial index carries a WHERE clause, so rows that fail the predicate never enter the tree. When most rows are rarely read, this keeps the index small and the lookups fast on the ones that matter. An index on open orders rather than all orders can be an order of magnitude smaller, which means more of it fits in cache, fewer pages per lookup, and plans that stay index-driven as the cold history grows unboundedly.

Pick the predicate from the queries. The right condition is the one every important query already repeats: status equals open, deleted_at is null. If queries slice the table differently, either the dominant slice wins or the table needs two partial indexes, each still thinner than a full one.

Fewer rows, cheaper maintenance

Fewer entries mean inserts, updates, and VACUUM touch a lighter structure, and disk usage drops visibly. The savings are most obvious on write-heavy tables where the full set would be mostly noise. Every insert that fails the predicate skips the index entirely — no page split risk, no WAL for that entry — and updates that leave the predicate untouched skip index maintenance too. On tables absorbing thousands of cold writes per second, the avoided work dwarfs the lookup gains.

Smaller indexes also vacuum faster with fewer pages for bloat to hide in. But monitor size against the full-index equivalent: if the partial index approaches full size, the predicate no longer selects a minority and the bargain has expired — drop it or redefine the slice.

Shape the statement to match

The planner only honors the index for queries whose predicate matches the partial condition exactly, or implies it. Write the filter the same way in the query and the index so the planner can actually use it. status = ‘open’ matches an index predicated on status = ‘open’; status IN (‘open’, ‘pending’) does not, even though a human sees the overlap. The matcher is logical implication, not textual similarity, and it is conservative by design.

This strictness dictates a discipline: canonicalize the filter expression across the codebase. If the index says deleted_at IS NULL, no query may phrase it as deleted_at = NULL or COALESCE(deleted_at, …) IS NULL and expect the index. Centralize the hot-slice condition — a view, a scope, a query builder fragment — so every statement inherits the exact predicate. EXPLAIN is the arbiter: if the plan ignores the index, the predicates differ in a way the matcher cannot bridge.

When it breaks

The classic failure is the drifting predicate: the business redefines “open” to include a new status, application queries update, and the partial index silently stops matching half the workload. Nothing errors — plans just degrade to sequential scans one query at a time, and latency climbs without a deploy to blame. Guard with a periodic EXPLAIN on the canonical queries in CI or monitoring, asserting the index name appears in the plan. A plan assertion catches predicate drift the day it ships, not the quarter it gets noticed.

The second failure is the over-sliced table: five partial indexes, each serving one report, whose combined maintenance exceeds the full index they replaced. Audit with pg_stat_user_indexes — near-zero-scan indexes are removal candidates, and overlapping predicates should merge into one wider partial index.

Check it quickly

Prove the index matches and pays:

CREATE INDEX ON orders (created_at) WHERE status = 'open';
EXPLAIN SELECT * FROM orders WHERE status = 'open' ORDER BY created_at LIMIT 20;
SELECT pg_size_pretty(pg_relation_size('orders_created_at_idx'));

The plan must show an index scan on the partial index, and the size should be a small fraction of the full-index equivalent.

Sources

PostgreSQL partial indexes, Official partial index documentation, checked 2026-06-01.
PostgreSQL EXPLAIN, plan inspection for verifying index usage, checked 2026-09-05.

Hash indexes recheck collisions and maintain changed keys

Mon, 24 Jan 2028 00:00:00 +0000

PostgreSQL hash indexes narrow an equality search using a hash code, then verify the original value in the table. A hash match is a candidate, not proof of equality. Two different values can have the same code, and different codes can also map to the same bucket. Neither collisions nor shared buckets imply incorrect query results when the normal recheck is performed.

What lives in a bucket

An index tuple contains a four-byte hash value and a reference to the table tuple; it does not contain the full indexed value. The bucket mapping directs a lookup to relevant index pages. Buckets are shared storage, not one separately allocated bucket for every distinct key. An overflowing bucket can acquire additional pages that its scans must visit.

PostgreSQL can sort hash codes within an individual index page to accelerate local searching, but that does not provide ordering by the original column. A hash index cannot satisfy a key range or an ordered result. It supports single-column equality access, cannot enforce uniqueness, and does not support IS NULL. A B-tree is still a candidate for equality as well as the broader operations it supports.

The absence of original values makes hash scans lossy: matching index entries must be checked against table values. A collision can therefore appear as Rows Removed by Index Recheck in an execution plan. That is evidence that candidate verification worked, not an indication that the database returned both colliding values as equal.

When it breaks

Describing an indexed update as touching only one bucket hides important work. Changing the indexed key changes the tuple’s indexed value and may send its new entry to another bucket. PostgreSQL’s versioned rows and deferred index cleanup mean obsolete references do not necessarily disappear immediately. Growth can also split a bucket, and overflow pages add I/O.

Consequently, a workload with heavily repeated values or rapid growth can behave differently from a nearly unique, steady equality workload. Hash lookup is not an unconditional constant-time latency guarantee. Modern PostgreSQL hash indexes are crash recoverable, but durability does not eliminate storage maintenance or turn the structure into a uniqueness constraint.

Check it quickly

Run this with psql on a disposable PostgreSQL 18 database. It creates two hundred thousand temporary text keys using C collation, discovers a collision with the server’s own hashtext, and tests lookup and update behavior. It does not assume hash values are portable identifiers. Rollback removes the fixture.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE hash_items (key text COLLATE "C");
INSERT INTO hash_items SELECT 'key-'||g FROM generate_series(1,200000) AS g;
CREATE TEMP TABLE collision_pair AS
SELECT min(key) AS a,max(key) AS b FROM hash_items
GROUP BY hashtext(key) HAVING count(*)>1 ORDER BY hashtext(key) LIMIT 1;
SELECT a AS collision_a,b AS collision_b FROM collision_pair \gset
SELECT a,b,hashtext(a),hashtext(b) FROM collision_pair;
CREATE INDEX hash_items_key ON hash_items USING hash(key);
ANALYZE hash_items;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT key FROM hash_items WHERE key=:'collision_a';
UPDATE hash_items SET key='changed-'||key WHERE key=:'collision_a';
SELECT count(*) AS old_key_rows FROM hash_items WHERE key=:'collision_a';
SELECT count(*) AS new_key_rows FROM hash_items WHERE key='changed-'||:'collision_a';
DO $$ BEGIN
 IF (SELECT hashtext(a)=hashtext(b) AND a<>b FROM collision_pair) IS DISTINCT FROM true THEN
 RAISE EXCEPTION 'fixture did not establish a collision';
 END IF;
 IF (SELECT count(*) FROM hash_items h JOIN collision_pair p ON h.key=p.a) <> 0 OR
 (SELECT count(*) FROM hash_items h JOIN collision_pair p ON h.key='changed-'||p.a) <> 1 OR
 (SELECT count(*) FROM hash_items h JOIN collision_pair p ON h.key=p.b) <> 1 THEN
 RAISE EXCEPTION 'hash lookup or indexed update returned wrong keys';
 END IF;
END $$;
ROLLBACK;

On PostgreSQL 18.6, the discovered keys were key-129539 and key-7295, both with hash code -1960436116. The equality plan used hash_items_key, returned one row, and reported one row removed by index recheck. The two strings remained distinct despite their identical hash code. This is a concrete negative control against treating hashes as collision-free keys.

Updating the first key to its changed- form left zero rows under its old value and one under its new value. The other colliding key still had one row. The final assertions verify all three facts and verify that the original pair really collided. A successful update alone would not establish that unrelated colliding entries remained searchable.

The particular pair and plan are observations from this version. If a different environment does not discover a pair, the script must stop rather than claiming a collision was tested. Even when the pair changes, the required semantics stay the same: exact equality returns only matching original values, and updating one value does not change another merely because their hashes coincide.

Evaluate maintenance with lookup performance

Measure page accesses, equality latency, update behavior, and index size using representative distributions. VACUUM can remove dead references and recycle overflow pages, but it does not shrink the hash index’s bucket count or return its allocation to the operating system. Rebuilding is a separate maintenance operation with its own cost. A smaller hash entry can be useful, yet only a full workload comparison establishes whether that benefit exceeds recheck, overflow, and write overhead.

Sources

PostgreSQL 18: Hash indexes, collision rechecks and bucket maintenance, checked 2026-09-10.
PostgreSQL 18: Index types, supported predicates for B-tree and hash indexes, checked 2026-09-10.

Postgres JSONB pairs with GIN for flexible queries

Wed, 28 Jul 2027 00:00:00 +0000

A PostgreSQL JSONB column can hold documents with different shapes, but efficient querying still requires a precise contract between the predicate and the index. A GIN index on the whole document supports particular JSONB operators. It does not automatically accelerate every expression that extracts a field from that document.

Match the operator to the index

The containment predicate payload @> '{"status":"pending"}' asks whether the document contains that key and JSON string value. With the default jsonb_ops operator class, a whole-column GIN index can support containment and top-level key existence, together with supported JSON path searches. The index stores searchable components; a bitmap heap scan may still visit the table and recheck candidate rows.

By contrast, payload->>'status' = 'pending' compares extracted text. A GIN index on payload is not an index on that text expression. A B-tree expression index on (payload->>'status') matches the scalar equality query. If many queries sort, join, or constrain status, an ordinary typed column may express the contract more clearly and avoid repeatedly deriving it from flexible documents.

Do not rewrite every text predicate to containment without checking types. The text extraction of a JSON number and a JSON string can produce the same text, while containment distinguishes their JSON types. A rewrite is equivalent only under the relevant data assumptions. Parameterize values through the database driver instead of constructing JSON or SQL by concatenating untrusted input.

An operator class is also a capability choice. jsonb_path_ops supports containment and supported path-match operators, but not the key-existence operators supplied by jsonb_ops. It can be smaller and more selective for suitable data, yet it is not a universally better replacement. Likewise, an index on a nested expression must match the expression actually queried; indexing the complete document does not imply every extracted subtree has its own index.

Missing and null are different questions

An absent key, a key whose value is JSON null, and a key containing the string “null” are distinct states. The existence operator detects the latter two as present. Extracting JSON with -> returns SQL NULL for a missing key, but a JSON null value for an explicit null. Extracting text with ->> maps both missing and JSON null to SQL NULL; the string remains text.

This affects filters, uniqueness assumptions, and application defaults. Use a key-existence check when presence matters and inspect JSON types when a field must be a string or number. SQL equality with NULL does not mean “is absent.” Also remember that the ? operator searches the top level; it does not recursively find a key nested anywhere in the document.

When it breaks

A common performance failure is installing GIN while leaving an incompatible extraction predicate unchanged. Another is assuming that an eligible index must be selected. Statistics, selectivity, table size, and estimated costs influence the plan; reading much of the table can make a sequential scan appropriate. Compare estimated and actual row counts before blaming the access method.

Large flexible documents also increase write work. GIN maintains many entries, and updating a document still updates a row with associated locking and storage costs. Index only the questions the workload actually asks. A valid JSONB value is not evidence that required business fields exist or have acceptable types.

Check it quickly

Save this as json-query.sql and run psql -X -v ON_ERROR_STOP=1 -f json-query.sql against a disposable PostgreSQL 18 database. The transaction creates temporary data and removes it on rollback. Only one percent of rows have pending status.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE documents (id integer, payload jsonb NOT NULL);
INSERT INTO documents SELECT g, jsonb_build_object('status',
 CASE WHEN g%100=0 THEN 'pending' ELSE 'done' END)
FROM generate_series(1,10000) AS g;
CREATE INDEX documents_payload ON documents USING gin(payload);
ANALYZE documents;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT id FROM documents WHERE payload @> '{"status":"pending"}';
EXPLAIN (COSTS OFF)
SELECT id FROM documents WHERE payload->>'status'='pending';
CREATE INDEX documents_status ON documents ((payload->>'status'));
ANALYZE documents;
EXPLAIN (COSTS OFF)
SELECT id FROM documents WHERE payload->>'status'='pending';
SELECT value, value ? 'x' AS has_key,
 value->'x' IS NULL AS sql_null_json, value->>'x' IS NULL AS sql_null_text
FROM (VALUES ('{}'::jsonb),('{"x":null}'::jsonb),('{"x":"null"}'::jsonb)) v(value);
DO $$ BEGIN
 IF (SELECT count(*) FROM documents WHERE payload @> '{"status":"pending"}') <> 100 THEN
 RAISE EXCEPTION 'wrong containment count';
 END IF;
 IF '{"n":1}'::jsonb @> '{"n":"1"}'::jsonb THEN
 RAISE EXCEPTION 'number and string were conflated';
 END IF;
 IF '{"outer":{"x":1}}'::jsonb ? 'x' THEN
 RAISE EXCEPTION 'top-level existence searched recursively';
 END IF;
 IF ('{"x":null}'::jsonb->'x') IS NULL THEN
 RAISE EXCEPTION 'JSON null confused with missing key';
 END IF;
END $$;
ROLLBACK;

On PostgreSQL 18.6, containment used a bitmap index scan through documents_payload and returned 100 rows. The text extraction predicate first used a sequential scan, then used documents_status after its expression index was created. These are observed choices for this distribution, not fixed plan guarantees or production latency estimates.

The final three rows show missing, explicit JSON null, and the text string separately. Their existence flags are false, true, true; their text-is-SQL-null flags are true, true, false. Assertions also check typed containment and nonrecursive existence. Keep these semantic checks when changing query shape: a faster plan is useful only if it still returns the intended documents.

Sources

PostgreSQL JSON types and indexing, GIN operator classes and indexed expressions, checked 2026-09-10.
PostgreSQL JSON functions and operators, extraction, existence, containment, and null semantics, checked 2026-09-10.

A Postgres index lives to answer a query

Tue, 20 Jul 2027 00:00:00 +0000

An index is valuable because it changes the work needed by a real query. Counting indexed columns is not a useful success measure. A narrow B-tree that finds one customer’s newest three orders can outperform a wider structure that cannot supply the needed ordering, while a query reading almost every row may correctly ignore both. Evaluate access, ordering, and result volume together.

Start from the requested result

Suppose the application asks for the newest three orders for one customer. Equality on customer narrows the population, and descending placed_at chooses the newest records within that customer. A final id key gives deterministic ordering when timestamps tie. The index (customer, placed_at DESC, id DESC) expresses that useful sequence of keys.

The benefit is not just avoiding a table scan. An ordered index path can stop after producing the limited result instead of collecting and sorting every matching order. Without LIMIT, or when most rows are needed, the balance can change. The same index must be judged against the full statement and its common parameter values, not an isolated WHERE fragment.

Textual condition order does not set index order. PostgreSQL plans logical expressions, so moving customer equality before or after another AND condition does not redefine the tree. Index direction, null ordering, leading-key constraints, and the requested output order are relevant structural properties. Verify them explicitly when adapting a query rather than applying a blanket column-order slogan.

Read the chosen plan honestly

EXPLAIN shows the planner’s proposed work, and EXPLAIN ANALYZE executes it to show actual rows and loops. A sequential scan can be the right answer when a query returns most of a table, especially when it also needs wide columns absent from the index. Such a plan is not proof that column order is wrong or that index creation failed.

Conversely, seeing an index name does not prove a query is cheap. An index scan can visit many entries and heap tuples, a bitmap path can need a separate sort, and an index-only plan can still perform heap visibility checks. Read actual rows, Heap Fetches where applicable, buffer activity, and sorting together. A LIMIT near the top is useful only if lower nodes avoid doing excessive work first.

When it breaks

A production parameter can select thousands of times more rows than a development value. The index that looked excellent for a small customer may be less beneficial for an exceptionally large one. Data distribution, current statistics, caching, and concurrency all affect the plan and observed latency. Retain representative parameter cases when validating a change.

Indexes also impose persistent costs: storage, cache pressure, insert/update maintenance, and build time. Adding note as payload merely because one diagnostic query selected it can make a hot write path more expensive. Separate the narrow application lookup from occasional broad reporting, and require evidence that each extra key or included column serves a recurring need.

A final failure is using unspecified ordering in pagination. The first few rows can change across plans or tied timestamps unless ORDER BY defines a deterministic choice. An index does not independently establish an API ordering contract. Keep that order explicit in SQL and validate exact identifiers, not just the number of returned rows.

Check it quickly

Save the following as topn.sql and run psql -X -v ON_ERROR_STOP=1 -f topn.sql against a disposable PostgreSQL 18 database. It creates twenty thousand orders spread over one hundred customers, including a wide note column for the broad-query comparison.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE orders (id integer, customer integer, placed_at integer, note text);
INSERT INTO orders SELECT g, g%100, g, repeat('x',100) FROM generate_series(1,20000) AS g;
CREATE INDEX orders_customer_recent ON orders (customer, placed_at DESC, id DESC);
ANALYZE orders;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT id,placed_at FROM orders WHERE customer=7 ORDER BY placed_at DESC,id DESC LIMIT 3;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT * FROM orders WHERE customer>=0;
SELECT array_agg(id ORDER BY placed_at DESC,id DESC) AS newest
FROM (SELECT id,placed_at FROM orders WHERE customer=7
 ORDER BY placed_at DESC,id DESC LIMIT 3) s;
DO $$ BEGIN
 IF (SELECT array_agg(id ORDER BY placed_at DESC,id DESC)
 FROM (SELECT id,placed_at FROM orders WHERE customer=7
 ORDER BY placed_at DESC,id DESC LIMIT 3) s)
 IS DISTINCT FROM ARRAY[19907,19807,19707] THEN
 RAISE EXCEPTION 'wrong newest orders';
 END IF;
END $$;
ROLLBACK;

The customer-seven query returns ids 19907,19807,19707 in that order. PostgreSQL 18.6 chose a Limit over an index-only scan and stopped after three matching rows, with heap checks because the new table was not vacuumed. The all-customers query selected a sequential scan and returned twenty thousand rows including their notes. Both choices are consistent with their different tasks.

The assertion checks the deterministic top-three result; timing and buffer numbers are deliberately not fixed expectations. Rollback removes the fixture. This demonstrates workload-shaped index use, not a universal speed ratio or proof that the same plan will remain optimal as production data and query requirements evolve.

Sources

PostgreSQL multicolumn indexes, key constraints and scan scope, checked 2026-09-10.
PostgreSQL indexes and ORDER BY, ordered retrieval and LIMIT, checked 2026-09-10.
PostgreSQL using EXPLAIN, interpreting actual plan work, checked 2026-09-10.

A function index precomputes a value at write time

Wed, 05 May 2027 00:00:00 +0000

An expression index stores the result of a computation as an index key. A query comparing lower(email) can search those precomputed keys instead of evaluating lower over every table row. This moves work into index creation and maintenance. It does not change the stored email, guarantee that every query uses the index, or make every textual variation a different expression.

Index the computed meaning

The index on lower(email) contains normalized values associated with rows. An equality predicate using that expression can become an index condition. Ordinary equality on the original email has different semantics and does not automatically gain the same search path. Likewise, applying trim before lower adds another operation; the existing index cannot assume trimming never changes data.

PostgreSQL matches parsed expressions and performs some simplification. Extra parentheses do not inherently defeat matching, so queries need not use identical source text. Conversely, mathematically equivalent expressions are not always recognized as interchangeable. Inspect the planned condition and the actual expression definition rather than relying on a cosmetic string comparison or a universal equivalence promise.

A unique expression index can also enforce a rule such as case-normalized uniqueness. That is a change to allowed data, not just an optimization. Existing conflicting rows will prevent its creation. Lowercasing alone is not a complete definition of email equivalence, and locale or collation behavior can matter for other text. Choose a business normalization rule intentionally before making it a stored constraint.

Keep indexed values stable

Functions used in index expressions must satisfy PostgreSQL’s immutability requirement: their results must depend on their arguments rather than changing external state or the current time. now() cannot be indexed as if it were a stable value forever. Marking a function IMMUTABLE merely to bypass the check does not make the claim true and can leave stored keys inconsistent with later evaluations.

The expression is computed during insertion and applicable updates. Its storage and maintenance cost remain even if the planner never chooses the index. Changing a function’s behavior or normalization policy also requires considering already stored index values; replacing code alone does not recompute them. Plan the rebuild and data validation as part of the semantic change.

When it breaks

An ORM refactor can introduce lower(trim(email)) while the index still stores lower(email). Queries remain correct but may scan more rows. A team can also change collation or a custom function and accidentally rely on old stored keys. Compare the final generated SQL, index definition, relevant function behavior, and target environment when diagnosing the missing path.

Another failure is treating a unique expression index as a harmless performance tweak. Duplicate normalized values can make index creation fail, and future inserts now face a new constraint. Audit and resolve real duplicates without silently deleting business records. For production builds, choose an appropriate index creation procedure and verify its completion; an index name existing in the catalog is not sufficient evidence of a usable finished index.

Check it quickly

Save this as expression.sql and execute psql -X -v ON_ERROR_STOP=1 -f expression.sql in a disposable PostgreSQL 18 database. The ten thousand addresses use the reserved example.test domain, and the table is temporary.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE accounts (id integer, email text NOT NULL);
INSERT INTO accounts SELECT g, 'User'||g||'@example.test' FROM generate_series(1,10000) AS g;
CREATE UNIQUE INDEX account_email_folded ON accounts (lower(email));
ANALYZE accounts;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT id FROM accounts WHERE ((lower(email)))='user42@example.test';
EXPLAIN (COSTS OFF)
SELECT id FROM accounts WHERE lower(trim(email))='user42@example.test';
DO $$ BEGIN
 INSERT INTO accounts VALUES (10001,'USER42@example.test');
 RAISE EXCEPTION 'case-folded duplicate accepted';
EXCEPTION WHEN unique_violation THEN RAISE NOTICE 'folded duplicate rejected';
END $$;
DO $$ BEGIN
 EXECUTE 'CREATE INDEX invalid_clock ON accounts ((now()))';
 RAISE EXCEPTION 'non-immutable index expression accepted';
EXCEPTION WHEN invalid_object_definition THEN RAISE NOTICE 'non-immutable expression rejected';
END $$;
DO $$ BEGIN
 IF (SELECT id FROM accounts WHERE lower(email)='user42@example.test') IS DISTINCT FROM 42 THEN
 RAISE EXCEPTION 'expression lookup failed';
 END IF;
END $$;
ROLLBACK;

PostgreSQL 18.6 used account_email_folded even with redundant parentheses around lower(email). The trim variant used a sequential scan. Inserting an uppercase duplicate of user forty-two failed the unique expression index, and indexing now() failed the immutability requirement. The final assertion confirms the supported lookup still returns id forty-two.

The exception handlers catch only the expected database errors; unexpected success fails the script. Rollback removes all test objects. These results distinguish harmless syntax from changed meaning without claiming the fixture covers Unicode normalization, every collation, production build locking, or arbitrary expression equivalence. Those require their own concrete data and operations.

Sources

PostgreSQL indexes on expressions, stored computations and unique expressions, checked 2026-09-10.
PostgreSQL CREATE INDEX, immutability and build requirements, checked 2026-09-10.

A partial index stores only the rows that matter

Tue, 27 Apr 2027 00:00:00 +0000

A partial index can keep a frequently queried subset of rows in a smaller structure. An index on job id WHERE status is open is a useful candidate when open jobs are rare and most requests target them. Its availability depends on what the planner can prove about each query, not merely on whether the application happens to pass “open” during one execution.

Prove that excluded rows are unnecessary

The index has no entries for closed jobs, so it cannot by itself answer a query that might need them. PostgreSQL must recognize at planning time that the query condition implies the index predicate. An explicit status = 'open' supplies that fact. A generic parameter status = $1 does not, because the same plan must remain valid when the parameter is closed.

Implication is not always textual identity. PostgreSQL recognizes some simple logical relationships, but it is not a general theorem prover. Standardizing the intended predicate helps prevent refactors from hiding an otherwise usable index. A condition on the indexed id alone cannot establish the status restriction, even if current data or application conventions make the row usually open.

Prepared plans change the evidence available

Prepared statements can use custom plans based on supplied parameter values or generic plans reusable across values. A custom plan for status open can know the needed predicate and consider the partial index. A generic plan with an unknown status cannot safely use that index as the complete answer path. The application’s SQL text has not changed, but the information available during planning has.

This does not mean parameterized queries are incompatible with partial indexes. A query with literal status = 'open' and a parameter only for id retains the predicate in a generic plan. Nor should parameters be replaced by string concatenation: parameter binding remains the correct way to carry untrusted values. Design the query around its real invariant rather than weakening data handling to chase an index.

The default plan policy chooses between custom and generic plans using its own cost considerations. For diagnosis, plan_cache_mode can force either behavior in a session. Such forcing is an experiment, not an instruction to disable generic plans globally. Replanning costs, parameter skew, and execution frequency determine the appropriate tradeoff.

When it breaks

A query can become slower after repeated prepared execution when a reusable plan replaces a parameter-specific one. The symptom may look like random index disappearance. Capture the actual prepared SQL and inspect EXPLAIN EXECUTE with representative values; an ad hoc literal query does not reproduce every prepared-plan condition. Confirm the predicate, plan mode, estimated rows, and runtime work together.

The subset can also stop being small. A backlog that leaves most jobs open reduces the expected size advantage, and status transitions still require index maintenance. Statistics and production-like distributions are needed before claiming lower write cost or better cache residency. A partial index is a workload decision, not a guarantee that the table’s future state will stay favorable.

Check it quickly

Save this as partial-plan.sql and run psql -X -v ON_ERROR_STOP=1 -f partial-plan.sql in a disposable PostgreSQL 18 database. The temporary table has one hundred open jobs among ten thousand rows, and the index includes only that open subset.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE jobs (id integer, status text NOT NULL);
INSERT INTO jobs SELECT g, CASE WHEN g<=100 THEN 'open' ELSE 'closed' END
FROM generate_series(1,10000) AS g;
CREATE INDEX jobs_open_id ON jobs (id) WHERE status='open';
ANALYZE jobs;
PREPARE chosen(text,integer) AS SELECT id FROM jobs WHERE status=$1 AND id=$2;
SET LOCAL plan_cache_mode=force_generic_plan;
EXPLAIN (COSTS OFF) EXECUTE chosen('open',7);
EXECUTE chosen('open',7);
SET LOCAL plan_cache_mode=force_custom_plan;
EXPLAIN (COSTS OFF) EXECUTE chosen('open',7);
EXECUTE chosen('open',7);
PREPARE always_open(integer) AS SELECT id FROM jobs WHERE status='open' AND id=$1;
SET LOCAL plan_cache_mode=force_generic_plan;
EXPLAIN (COSTS OFF) EXECUTE always_open(7);
EXECUTE always_open(7);
DEALLOCATE chosen;
DEALLOCATE always_open;
ROLLBACK;

On PostgreSQL 18.6, the generic plan with unknown status used a sequential scan. Forcing a custom plan for the same prepared statement and open value used jobs_open_id. The second statement, whose status predicate is a literal and only its id is parameterized, also used that partial index under a generic plan. All three executions returned id seven.

These plans demonstrate why a known predicate matters, not a promise that every machine selects the same access path. The explicit SET LOCAL changes end with rollback, and DEALLOCATE removes the prepared statements. The test does not reproduce automatic plan switching under a real connection pool; that requires observing the application’s actual sessions and repeated execution history.

Sources

PostgreSQL partial indexes, predicate implication at planning time, checked 2026-09-10.
PostgreSQL PREPARE, custom and generic execution plans, checked 2026-09-10.

Choose B-tree or hash from the required operations

Sun, 07 Feb 2027 00:00:00 +0000

PostgreSQL’s default B-tree index supports several common query requirements: equality, ranges, ordered access, and uniqueness enforcement. A hash index supports a narrower equality access path. Choosing between them starts with the required operators and constraints, then proceeds to measurement. Saying that hashes are always faster for equality, or that PostgreSQL almost never chooses them, replaces a workload question with an unsupported generalization.

Capabilities before timings

A B-tree retains ordering under its operator class and collation. That can support a bounded range and an ORDER BY without a separate sort, as well as a single-key equality lookup. B-trees can also support IS NULL and IS NOT NULL. An index’s existence still does not force its use: reading a large fraction of a table may favor another plan.

Text patterns have additional requirements. A constant anchored prefix such as LIKE 'abc%' can use an appropriate B-tree access path, but collation and operator class matter. Outside the C locale, a pattern-specific operator class may be necessary. A leading wildcard is not made into a selective ordered range merely by creating a normal B-tree index.

A PostgreSQL hash index stores a four-byte hash code rather than the complete indexed value. It supports equality, lacks key ordering, supports only a single indexed column, and cannot enforce uniqueness. Its smaller entries can help some equality workloads with large values, but heap verification and bucket behavior still contribute to cost. The data structure alone does not establish the winner.

When it breaks

Replacing a B-tree with a hash index can remove useful range or ordered access. It also removes an access path for IS NULL: PostgreSQL hash indexes do not support that predicate. Replacing the index behind a uniqueness requirement is a more serious mistake because a hash index cannot supply the required contract at all.

Null semantics deserve a separate check. A default unique B-tree permits multiple nulls because they are treated as distinct for uniqueness. If the application requires at most one null, that must be expressed with the appropriate constraint semantics, not inferred from the word unique. Supported access operations and enforced data rules are related but different decisions.

Check it quickly

Run this experiment with PostgreSQL 18 and psql. The temporary table contains twenty thousand integer keys plus two null-key rows. It first has a hash index, then replaces that index with a unique B-tree. Rollback removes the experiment.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE points (key integer,value text);
INSERT INTO points SELECT g,'value-'||g FROM generate_series(1,20000) AS g;
INSERT INTO points VALUES (NULL,'no-key-a'),(NULL,'no-key-b');
CREATE INDEX points_hash ON points USING hash(key);
ANALYZE points;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT value FROM points WHERE key=12345;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT key,value FROM points WHERE key>=12340 AND key<12350 ORDER BY key;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT value FROM points WHERE key IS NULL;
DO $$ BEGIN
 CREATE UNIQUE INDEX points_hash_unique ON points USING hash(key);
 RAISE EXCEPTION 'hash uniqueness unexpectedly supported';
EXCEPTION WHEN feature_not_supported THEN RAISE NOTICE 'hash cannot enforce uniqueness';
END $$;
DROP INDEX points_hash;
CREATE UNIQUE INDEX points_btree ON points(key);
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT value FROM points WHERE key=12345;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT key,value FROM points WHERE key>=12340 AND key<12350 ORDER BY key;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT value FROM points WHERE key IS NULL;
DO $$ BEGIN
 IF (SELECT value FROM points WHERE key=12345) IS DISTINCT FROM 'value-12345' OR
 (SELECT count(*) FROM points WHERE key>=12340 AND key<12350) <> 10 OR
 (SELECT count(*) FROM points WHERE key IS NULL) <> 2 THEN RAISE EXCEPTION 'index changed answers'; END IF;
END $$;
ROLLBACK;

On PostgreSQL 18.6, the hash equality query used points_hash and returned value-12345. The ten-key range used a sequential scan plus sort, while the null query used a sequential scan returning two rows. Attempting a unique hash index raised the expected unsupported-feature exception, which the block catches deliberately; any different failure still stops the script.

After replacement, the B-tree supported the equality, bounded ordered range, and null lookup in this fixture. The range returned ten rows and the null lookup still returned two. Building the unique index succeeded despite those nulls, matching PostgreSQL’s default null-distinct uniqueness behavior. Assertions verify that changing access paths did not change query answers.

Plan choice and page counts can vary with relation size, cache state, statistics, and configuration. This experiment establishes supported operations and an observed selection, not a benchmark proving one method faster on every machine. The unsupported uniqueness capability does not become supported when costs or data sizes change.

Measure the complete workload

For a real candidate, compare representative equality lookups, range queries, ordering requirements, write activity, storage, and maintenance. Test alternatives in a controlled environment instead of building competing large indexes casually on production. A hash index may earn its storage on a measured equality workload, while the B-tree may still be required for uniqueness or other queries. Keep every retained index tied to an actual access or integrity requirement, and account for its write cost as well as its read benefit.

Sources

PostgreSQL 18: Index types, supported predicates for B-tree and hash indexes, checked 2026-09-10.
PostgreSQL 18: Hash indexes, collision rechecks and bucket maintenance, checked 2026-09-10.
PostgreSQL 18: Constraints, uniqueness and null semantics, checked 2026-09-10.