Indexes · XB Field Notes

Partial indexes cover only the rows that matter

Fri, 15 Oct 2027 00:00:00 +0000

A partial index stores entries only for rows satisfying its predicate. This can reduce index size when the useful subset is small, but it can also enforce a rule that applies only to that subset. For example, a seat may have many historical reservations while allowing at most one active reservation. A partial unique index expresses that rule at the database boundary.

Membership follows the predicate

CREATE INDEX ... WHERE active includes rows for which active evaluates to true. False and null do not satisfy the predicate. Whether null should be allowed is a separate schema decision; the experiment makes active non-null so each reservation is explicitly active or inactive. The indexed key need not be the same column used by the predicate.

Membership changes with ordinary writes. An inactive row becoming active must enter the index and pass its checks; an active row becoming inactive leaves the active subset. This is not an immutable archive boundary. Predicate evaluation and changing membership still cost work, so a smaller index does not mean every update is free or automatically faster.

Uniqueness can apply to a subset

A unique index on seat WHERE active rejects two active rows for the same seat, while permitting any number of inactive historical rows. The index enforces that rule even if no SELECT ever chooses it. Query acceleration and integrity enforcement are different reasons to retain an index, so usage counters alone cannot justify deleting this one.

An application-level “check first, insert later” is weaker because another transaction can insert between the check and the write. A unique index coordinates conflicting writes in PostgreSQL; the loser must handle a uniqueness error or another chosen conflict policy. The fixture exercises rejection sequentially, while concurrency behavior remains part of the database’s constraint contract rather than a claim about an application booking workflow.

Plan queries with the same boundary

To use a partial index as an access path, the planner must recognize that the query’s conditions imply the index predicate. Merely querying the seat does not prove the caller wants only active rows. A query that explicitly asks for active reservations can qualify, although a tiny table may still be cheaper to scan. A logically similar expression is useful only if the planner can establish the implication.

Do not create hundreds of overlapping partial indexes to simulate partitioning. Planning and maintenance costs grow, and the optimizer does not automatically understand them as one coordinated partition scheme. Choose a predicate from a stable workload or integrity requirement, then verify the actual query and transition behavior.

When it breaks

Forgetting the predicate in a query can return old reservations alongside the current one, even though active uniqueness is enforced perfectly. Conversely, treating the index as globally unique can reject valid history in application logic. Document whether the rule concerns all rows or only the current subset, and test both populations.

A failed reactivation is another important boundary. Changing an old inactive reservation back to active can conflict with the current holder. The application must keep the old row inactive when that update fails and report the conflict, not silently replace the active booking. Bulk imports and status migrations need the same checks as ordinary insertion.

Check it quickly

Save this block as partial-unique.sql and run psql -X -v ON_ERROR_STOP=1 -f partial-unique.sql in a disposable PostgreSQL 18 database. It creates a temporary reservation table, so no real seat inventory is touched.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE reservations (id integer PRIMARY KEY, seat text NOT NULL, active boolean NOT NULL);
CREATE UNIQUE INDEX one_active_seat ON reservations (seat) WHERE active;
INSERT INTO reservations VALUES (1,'A1',true),(2,'A1',false),(3,'A1',false);
DO $$ BEGIN
 INSERT INTO reservations VALUES (4,'A1',true);
 RAISE EXCEPTION 'two active reservations accepted';
EXCEPTION WHEN unique_violation THEN RAISE NOTICE 'active duplicate rejected';
END $$;
UPDATE reservations SET active=false WHERE id=1;
INSERT INTO reservations VALUES (4,'A1',true);
DO $$ BEGIN
 UPDATE reservations SET active=true WHERE id=2;
 RAISE EXCEPTION 'reactivation bypassed uniqueness';
EXCEPTION WHEN unique_violation THEN RAISE NOTICE 'conflicting reactivation rejected';
END $$;
SELECT active,count(*) FROM reservations GROUP BY active ORDER BY active;
SELECT id FROM reservations WHERE active AND seat='A1';
DO $$ BEGIN
 IF (SELECT id FROM reservations WHERE active AND seat='A1') IS DISTINCT FROM 4 THEN
 RAISE EXCEPTION 'wrong active reservation';
 END IF;
 IF (SELECT count(*) FROM reservations WHERE NOT active) <> 3 THEN
 RAISE EXCEPTION 'history was not preserved';
 END IF;
END $$;
ROLLBACK;

The first active duplicate is rejected while two historical duplicates are accepted. Deactivating reservation one releases the seat for reservation four. Reactivating reservation two then fails because four is active. The final counts are three inactive rows and one active row, whose identifier is four. Assertions check both the current holder and retained history.

PostgreSQL 18.6 passes these checks. The exception blocks catch only the expected uniqueness violation; an unexpected success raises a different error and fails the script. Rollback removes the fixture. This verifies subset membership and uniqueness transitions, not payment atomicity, distributed reservations, or the performance of a large historical table.

Sources

PostgreSQL partial indexes, predicate membership and partial uniqueness, checked 2026-09-10.
PostgreSQL CREATE INDEX, unique index enforcement and predicates, checked 2026-09-10.

Covering indexes answer without the table

Thu, 14 Oct 2027 00:00:00 +0000

A covering index contains the values needed by a particular query. PostgreSQL can sometimes return those values through an index-only scan, avoiding heap visits for qualifying rows. The word “sometimes” matters: having the columns is a structural requirement, while visibility information determines whether execution can actually skip the heap. An Index Only Scan label alone does not prove zero table access.

Cover the actual projection

For a lookup by sku that returns price, a B-tree on sku with INCLUDE (price) stores price as payload. The sku key guides the search; the included price does not become another search key or establish an ordering by price. If the index is unique, uniqueness applies to its key columns, not to the included payload.

A later query change can invalidate the coverage assumption. Selecting a description column absent from the index requires another source for that value, usually the heap. Selecting every column casually can therefore turn a carefully covered lookup into a different access pattern. Start from the real predicate and projected columns, including expressions and visibility needs, rather than adding all table columns to a speculative index.

Not every access method can reconstruct complete values for index-only execution. B-tree supports it, while GIN generally stores extracted keys rather than the complete original value and does not support index-only scans. A matching index must support the needed operation as well as contain or reconstruct the requested data. Similar terminology across database engines does not imply identical implementation details.

The visibility map decides heap visits

PostgreSQL needs to know whether a candidate tuple is visible to the query’s MVCC snapshot. Index entries do not carry all the required visibility information. An index-only scan checks the heap page’s all-visible bit in the visibility map; when that bit is set, the value can be returned without fetching the heap tuple.

A write can clear the page’s all-visible status, making later scans visit the heap again even when their plan name remains unchanged. Vacuum can establish all-visible status when the page’s tuples meet the visibility requirements. Long-running transactions and ongoing changes affect that process. It is a page-level property, so an update to one row can influence the visibility check for another row on that page.

When it breaks

The common mistake is declaring success from EXPLAIN’s plan label. Read Heap Fetches and relevant buffers in EXPLAIN ANALYZE, and confirm the actual returned values. A write-heavy table may require frequent heap checks, reducing the read benefit while still paying for duplicated index payload. Repeating VACUUM after every application write is not a substitute for evaluating the workload.

Wide payload columns also consume space and cache, increase maintenance work, and can exceed index tuple limits. Updating an included column must maintain the index and can prevent a heap-only update that might otherwise have been possible. Include only values whose read benefit justifies their write and storage cost; a covering index is not a free materialized copy of the table.

Check it quickly

Save this as covering.sql and run psql -X -v ON_ERROR_STOP=1 -f covering.sql against a disposable PostgreSQL 18 database. Keep psql’s normal autocommit behavior: VACUUM cannot run inside an explicit transaction block. The temporary table remains confined to this session.

\set ON_ERROR_STOP on
CREATE TEMP TABLE products (sku integer, price integer);
INSERT INTO products SELECT g, g*10 FROM generate_series(1,10000) AS g;
CREATE INDEX products_price_lookup ON products (sku) INCLUDE (price);
VACUUM (ANALYZE) products;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT price FROM products WHERE sku=4321;
UPDATE products SET price=price+5 WHERE sku=4321;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT price FROM products WHERE sku=4321;
VACUUM (ANALYZE) products;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT price FROM products WHERE sku=4321;
DO $$ BEGIN
 IF (SELECT price FROM products WHERE sku=4321) IS DISTINCT FROM 43215 THEN
 RAISE EXCEPTION 'covering lookup returned an obsolete price';
 END IF;
END $$;
DROP TABLE products;

The first vacuum makes this quiescent fixture eligible for heap-free reads. PostgreSQL 18.6 used Index Only Scan in all three plans, with Heap Fetches zero, then two after the price update, then zero after vacuum. The assertion confirms the new price is 43215 rather than an obsolete indexed value. Temporary objects are removed explicitly and also disappear when the session ends.

The exact intermediate count can vary with implementation and page state; its significance is that heap access can occur under the same plan label. Heap Fetches counts heap visits, not necessarily physical disk reads: the pages may already be cached. This is a controlled visibility demonstration, not a guarantee that production vacuum will immediately make every page all-visible. Compare representative reads before and after real update activity when deciding whether coverage pays off.

Sources

PostgreSQL index-only scans, visibility maps and included payload, checked 2026-09-10.
PostgreSQL CREATE INDEX, INCLUDE semantics and index restrictions, checked 2026-09-10.

Full table scans win on small tables

Thu, 16 Sep 2027 00:00:00 +0000

An index is a bet: you pay write and storage cost up front so reads can skip most of the table. When a query genuinely needs most of the table, the bet loses: the engine descends a B-tree per row instead of reading the pages once in order.

Planners price both routes, so the real question is where the crossover sits. A small table is not a small amount of work, and the plan you see at a thousand rows is not the plan to expect at ten million.

The currency is pages, not rows

A sequential scan reads the table’s pages in physical order, many rows per page. Postgres prices that with seq_page_cost (1.0 by default) against random_page_cost (4.0 by default): a non-sequentially fetched page is priced four times a sequential one, and lowering random_page_cost relative to seq_page_cost is precisely what makes the planner start preferring index scans. When the whole database fits in cache, the documentation suggests the two can reasonably be equal.

Because pages are the currency, row count is a poor proxy for size. Two tables of 1,000 rows: a 4-byte payload is 12 pages, a 4,000-byte payload is 1,010. On sqlite 3.53 with a 2 MB page cache the broad query over the narrow table finished in 0.02 ms and over the wide one in 1.30 ms — a 65x spread at identical row counts. Only one of those tables is small.

Where the index stops paying

Forcing each access path makes the crossover visible. The table is events(id, tenant, payload), 1,000 distinct tenants, an index on tenant, predicate matching roughly 90% of rows:

1,000 rows in 21 pages: scan 0.04 ms, index 0.05 ms. A tie. The index buys nothing at this size and still has to be maintained on every write.
50,000 rows in 767 pages: scan 1.79 ms, index 6.61 ms. The scan wins by 3.7x.
1,000,000 rows in 15,474 pages (63 MB): scan 44.37 ms, index 531.18 ms. The scan wins by 12x, and across seven samples the ratio never dropped below 11.3x.

The same index on the same million-row table is a 29x win for a selective predicate: tenant = 7 covers about 0.1% of rows and took 0.61 ms through the index against 17.88 ms for the scan. The index is not bad; it is bad for that predicate. Selectivity decides, and it decides harder as the table grows.

Let the planner decide, but keep its inputs honest

sqlite chose the scan for the broad predicate at every size and the index for the selective one. That is the correct default, and the lever when a plan looks wrong is the statistics, not a hint.

Statistics decay and the plan decays with them. ANALYZE the million-row table, then move every row to tenant = 5 without re-analyzing. sqlite_stat1 still reports 1,000 rows per key, so the planner keeps picking SEARCH ... USING INDEX for a predicate matching 100% of the table. Run ANALYZE and the statistic becomes 1,000,000 rows per key and the plan flips to SCAN. The wrong plan cost only 1.2x here, because sqlite’s table b-tree is keyed by rowid so the index walk still fetches pages in near-physical order; in Postgres each index hit is a separate heap fetch priced at random_page_cost.

When it breaks

The classic failure is the plan that flips when statistics go stale. A nightly job loads a tenant that comes to dominate the table; the histogram still describes last week; the dashboard queries that used to be index lookups now read a fifth of the table through the index. Nothing errors. Latency simply steps up, on exactly the queries nobody was watching.

The second failure is scale, not shape. The scan that costs 0.03 ms over 21 pages costs 1.88 ms over 767 and 41.34 ms over 15,474, because cost is linear in pages. A table that is fine today becomes an unaffordable full scan after a year of growth. The fix is an index the planner will actually choose, or a query that returns fewer rows.

Check it quickly

Confirm the table is small in pages, then see whether the planner agrees with reality:

sqlite3 events.db "PRAGMA page_count; PRAGMA page_size;"
sqlite3 events.db "EXPLAIN QUERY PLAN SELECT sum(length(payload)) FROM events WHERE tenant < 900;"
sqlite3 events.db "EXPLAIN QUERY PLAN SELECT sum(length(payload)) FROM events NOT INDEXED WHERE tenant < 900;"
sqlite3 events.db "ANALYZE; SELECT stat FROM sqlite_stat1 WHERE idx='events_tenant';"
printf '.timer on\nSELECT sum(length(payload)) FROM events NOT INDEXED WHERE tenant < 900;\n' | sqlite3 events.db

The first line gives you pages, the number that matters. The next two should name the same path for a broad predicate: if the planner says SEARCH ... USING INDEX while the forced SCAN is measurably faster, the statistics are the suspect. sqlite_stat1 shows the average rows per key — when it still says 1000 while one key owns the table, re-run ANALYZE and read the plan again.

Sources

PostgreSQL: planner cost constants, seq_page_cost and random_page_cost, checked 2026-09-16.
PostgreSQL: using EXPLAIN, reading plans and row estimates, checked 2026-09-16.
SQLite: EXPLAIN QUERY PLAN, how scan and search steps are reported, checked 2026-09-16.

Composite indexes match multi-column queries

Wed, 15 Sep 2027 00:00:00 +0000

A composite B-tree index orders tuples by its first key, then by the next key within each first-key group. This structure can make a tenant-specific time range cheap to locate. The useful rule is about how much index space a predicate narrows, not a prohibition against mentioning later columns or writing WHERE conditions in a different textual order.

Order the keys, not the SQL clauses

For an index on (tenant, happened), equality on tenant and a range on happened describe a compact portion of the tree. PostgreSQL can search that range rather than inspecting all tenants. Writing the time condition before the tenant condition in SQL does not change the logical conjunction or the index’s key order. The schema’s column order and the query’s presentation order are different things.

Leading equality conditions and the first non-equality key generally determine the range that must be scanned. Conditions on later keys can still be checked within the index, saving heap visits even when they do not further shrink a simple contiguous range. Reversing key columns changes these opportunities; it does not make an index universally unusable.

Later keys can still help

PostgreSQL 18 can use B-tree skip scans when repeated searches over missing leading-key values are cheaper than reading most of the index. With only a few tenants, a predicate on happened alone can search within each tenant group. With many distinct leading values, that strategy may be too expensive, and the planner can prefer a sequential scan or another index.

Even without a profitable skip scan, a composite index can sometimes be scanned for a condition involving a later key. The question is cost, coverage, and selectivity, not whether such SQL is legal or whether the index must be ignored. Do not use the shorthand “leftmost prefix” as an absolute claim that contradicts the target engine’s actual access methods.

Include ordering and the workload

Fixing the tenant key can make the following happened key useful for ordered results. Across several tenants, however, the index groups by tenant first; it does not provide one globally time-sorted stream. A bitmap scan also loses index ordering and may need a separate sort. Look at the complete plan instead of inferring final order from an index declaration.

Equality-first and range-next is a useful starting point for one query shape, but there is no universal “most selective column first” recipe. Shared prefixes, joins, sort direction, LIMIT, parameter distributions, and write costs all affect the choice. Compare actual recurring queries, then keep the smallest useful set of indexes. Redundant wide indexes consume cache and maintenance work even when nobody benefits from them.

When it breaks

A team may build (tenant, happened) for per-tenant queries, then add a global timeline and assume it is equally efficient. The timeline’s ordering and selectivity differ. Another regression comes from testing one tiny tenant while another owns most rows: the same parameterized SQL can have very different costs. Statistics and representative values matter more than a screenshot of one favorable plan.

Incorrect rules also cause unnecessary index churn. Adding one index per suffix because “the existing index cannot be used” can duplicate useful storage. Conversely, assuming skip scan always rescues a missing prefix can hide a costly path. Measure the chosen plan, actual rows, buffers, and sort work before changing the schema.

Check it quickly

Save this as composite.sql and run psql -X -v ON_ERROR_STOP=1 -f composite.sql on a disposable PostgreSQL 18 database. The temporary fixture contains four tenant values and twenty thousand ordered event numbers.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE events (tenant integer, happened integer, body text);
INSERT INTO events SELECT g % 4, g, repeat('x',40) FROM generate_series(1,20000) AS g;
CREATE INDEX events_tenant_time ON events (tenant, happened);
ANALYZE events;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT happened FROM events WHERE tenant=2 AND happened BETWEEN 100 AND 120
ORDER BY happened;
SELECT array_agg(happened ORDER BY happened) AS matching_times
FROM events WHERE happened BETWEEN 100 AND 120 AND tenant=2;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT tenant FROM events WHERE happened=12346;
DO $$ BEGIN
 IF (SELECT array_agg(happened ORDER BY happened) FROM events
 WHERE happened BETWEEN 100 AND 120 AND tenant=2)
 IS DISTINCT FROM ARRAY[102,106,110,114,118] THEN
 RAISE EXCEPTION 'wrong composite range';
 END IF;
 IF (SELECT tenant FROM events WHERE happened=12346) IS DISTINCT FROM 2 THEN
 RAISE EXCEPTION 'suffix predicate lost the row';
 END IF;
END $$;
ROLLBACK;

The tenant-range query returns exactly 102,106,110,114,118, including when the WHERE conditions are written in the opposite order. On PostgreSQL 18.6 it used a bitmap index scan followed by a sort. The suffix-only lookup returned tenant two through an index-only scan with repeated index searches, demonstrating a usable path without tenant equality.

Plan names, buffer counts, and search counts are observations for this dataset, not guaranteed outputs across configurations. The assertions verify result semantics, and rollback removes all objects. Repeat the measurement with more leading-key values to assess whether the suffix path remains worthwhile for a real workload.

Sources

PostgreSQL multicolumn indexes, leading constraints and skip scans, checked 2026-09-10.
PostgreSQL indexes and ORDER BY, ordered scans and sort choices, checked 2026-09-10.

Partial indexes cover only the rows that matter

Sat, 17 Oct 2026 00:00:00 +0000

A queue with a hundred ready jobs and thousands of completed jobs can benefit from an index containing only ready work. PostgreSQL calls this a partial index: its predicate determines which table rows have entries. The benefit depends on the actual query population and write pattern, not simply on making every index smaller.

When it breaks

The direction of implication matters. The query conditions must imply the index predicate, so every row the query might need is represented. A ready-only index cannot answer an unrestricted query about all jobs. Merely having compatible-looking conditions is insufficient, and PostgreSQL does not prove every mathematical equivalence. A predicate written close to the application’s filter is easier for the planner and reviewers to recognize.

Planning also determines what values are known. A custom prepared plan can use the supplied state value to choose a partial index. A generic plan must be valid for every possible parameter value; it cannot assume that a state parameter always means ready. This explains why testing a literal query can look good while an application’s prepared query uses a different access path.

Check it quickly

Run the blocks in one fresh psql -X -v ON_ERROR_STOP=1 session on a disposable PostgreSQL 18 database. The fixture has no full-table index, making the eligible access paths easy to inspect. Create ten thousand jobs, of which exactly one hundred are ready:

CREATE SCHEMA partial_demo;
CREATE TABLE partial_demo.jobs (id integer,state text);
INSERT INTO partial_demo.jobs
SELECT g,CASE WHEN g%100=0 THEN 'ready' ELSE 'done' END
FROM generate_series(1,10000) AS g;
CREATE INDEX ready_jobs ON partial_demo.jobs(id) WHERE state='ready';
VACUUM (ANALYZE) partial_demo.jobs;
SELECT state,count(*) FROM partial_demo.jobs GROUP BY state ORDER BY state;

The grouped counts are 9,900 done and 100 ready. The indexed column is the job identifier, while the predicate uses state; those columns need not be the same. Inspect a literal query:

EXPLAIN (ANALYZE,COSTS OFF,TIMING OFF,SUMMARY OFF,FORMAT JSON)
SELECT id FROM partial_demo.jobs WHERE state='ready' ORDER BY id LIMIT 5;

The tested plan contains an Index Only Scan using ready_jobs beneath Limit, returning five rows. Vacuum established the visibility information needed to avoid heap fetches in this fixture. Partial indexing alone does not promise an index-only scan: selected columns and visibility still matter. The important eligibility evidence is the index name and correct result.

Prepare the same request with a state parameter and force a custom plan for comparison:

PREPARE next_jobs(text) AS
SELECT id FROM partial_demo.jobs WHERE state=$1 ORDER BY id LIMIT 5;
SET plan_cache_mode=force_custom_plan;

EXPLAIN (ANALYZE,COSTS OFF,TIMING OFF,SUMMARY OFF,FORMAT JSON)
EXECUTE next_jobs('ready');

This plan also uses ready_jobs, because the supplied ready value is available when planning. Now request a generic plan for the same prepared statement:

SET plan_cache_mode=force_generic_plan;

EXPLAIN (ANALYZE,COSTS OFF,TIMING OFF,SUMMARY OFF,FORMAT JSON)
EXECUTE next_jobs('ready');

The generic plan scans the table, filters out 9,900 rows, and sorts the matching identifiers before taking five. Its state expression remains a parameter, so the ready-only index cannot cover every possible execution. These forced modes isolate the distinction; they are diagnostic controls, not a recommendation to force one mode globally.

Execute both parameter values, then move one ready job across the predicate boundary:

EXECUTE next_jobs('ready');
EXECUTE next_jobs('done');
DEALLOCATE next_jobs;
RESET plan_cache_mode;
UPDATE partial_demo.jobs SET state='done' WHERE id=100;
SELECT id FROM partial_demo.jobs WHERE state='ready' ORDER BY id LIMIT 5;
SELECT count(*) FROM partial_demo.jobs WHERE state='ready';
DO $$ BEGIN
 IF (SELECT array_agg(id ORDER BY id) FROM
 (SELECT id FROM partial_demo.jobs WHERE state='ready' ORDER BY id LIMIT 5) AS q)
 IS DISTINCT FROM ARRAY[200,300,400,500,600] THEN
 RAISE EXCEPTION 'partial-index membership result mismatch';
 END IF;
END $$;
DROP SCHEMA partial_demo CASCADE;

The first request returns identifiers 100, 200, 300, 400, and 500. The done request returns one through five. After job 100 becomes done, the first five ready identifiers are 200 through 600 in steps of one hundred, and 99 ready jobs remain. The assertion checks that maintenance follows the updated predicate membership. Removing the prepared statement and schema leaves no experimental data behind.

Evaluate the whole workload

Updates that enter or leave the subset change index membership. Even updates outside it still perform ordinary table work; a partial index does not eliminate all write cost. Measure index size, query plans, and maintenance under representative state transitions. If most jobs become ready, the expected advantage may disappear, and a sequential scan can be the right choice.

A partial unique index enforces uniqueness only among rows satisfying its predicate. Rows outside that subset remain unconstrained by that index, which can be useful for historical records but is wrong for a global identifier rule. Also avoid predicates based on a moving current time: index expressions and predicates require immutable functions, and yesterday’s membership would not automatically change as the clock advances. Prefer an explicitly maintained business state when that state defines the real query contract.

Sources

PostgreSQL 18 partial indexes, predicate implication and subset uniqueness, checked 2026-09-10.
PostgreSQL 18 CREATE INDEX, index predicates and immutable-expression requirements, checked 2026-09-10.
PostgreSQL 18 PREPARE, generic and custom prepared plans and their controls, checked 2026-09-10.