Sql · XB Field Notes

Table partitioning splits data by range or key

Thu, 10 Feb 2028 00:00:00 +0000

Range partitioning turns a boundary definition into a write-routing rule. PostgreSQL places a row in the partition whose bounds accept its key, whether the row arrives on time or months late. Correct boundaries, compatible timestamp types, and a plan for missing ranges matter before any query can benefit from pruning.

Use adjacent half-open ranges

A range partition includes its lower bound and excludes its upper bound. September can cover the interval from September first through, but not including, October first. October then starts at that exact boundary without overlap. Do not model a month by guessing its final representable timestamp; adjacent half-open ranges work regardless of the timestamp’s fractional precision.

Queries should express the intended interval just as carefully. A lower-inclusive, upper-exclusive predicate matches a monthly slice. BETWEEN includes both endpoints, so using the next month’s first instant as its upper endpoint can include data from that next month. Partition routing does not correct a predicate that asks for the wrong interval.

The key’s type and time zone are part of the definition. A timestamptz value represents an instant, while a timestamp without time zone represents a local date and time without that instant’s zone interpretation. The example uses explicit UTC offsets for every boundary. If business months follow another time zone, derive boundaries using that zone’s rules instead of assuming every local day has the same elapsed duration.

Missing ranges are an operational state

An insert with no matching partition raises an error unless a DEFAULT partition accepts it. PostgreSQL does not automatically create the next monthly table when the calendar advances. Create needed ranges ahead of demand and account for late data and corrections to old dates. A key need not arrive monotonically for routing to work; its destination simply must still exist.

DEFAULT can absorb values outside the explicitly declared ranges. That can reduce immediate insertion failures, but it also makes missing partition management less visible unless the default table is inspected. A growing default partition can contain several unrelated time ranges and become harder to reorganize. Keep the difference between an intentional exception area and forgotten routine data visible.

Adding a new explicit partition changes which values the default partition may contain. If rows belonging to the new range remain in DEFAULT, PostgreSQL rejects the change rather than leaving overlapping ownership. Move those rows to an appropriate staging table or otherwise reconcile them before attaching the new partition. Bounds are enforced data rules, not labels that can simply be made inaccurate.

When it breaks

A deployment can fail exactly at a month boundary because the next partition was never created. Another can fail while adding that partition because DEFAULT already accumulated its rows. Both failures concern routing and data placement, not stale planner statistics. Inspect the rejected key and existing bounds before running ANALYZE or changing indexes.

ATTACH may need to scan the candidate table to validate its bounds and also check the default partition for conflicting rows. Suitable validated constraints can avoid some scans, but the operation still requires locks and compatible table structure. On active tables, coordinate writers and lock acquisition while relocating data; an isolated transactional demonstration is not a complete online migration procedure.

Nullability is separate from range selection. The example requires a nonnull timestamp so a missing event time cannot quietly become ordinary exception data. Also verify row counts and identifiers after relocation: a successful attach is not proof that an earlier copy step neither lost nor duplicated rows.

Check it quickly

Save this as partition-ranges.sql and run psql -X -v ON_ERROR_STOP=1 -f partition-ranges.sql in a disposable PostgreSQL 18 database. All tables are temporary and the whole experiment is rolled back.

\set ON_ERROR_STOP on
BEGIN;
SET LOCAL timezone='UTC';
CREATE TEMP TABLE samples (at timestamptz NOT NULL,id integer) PARTITION BY RANGE(at);
CREATE TEMP TABLE samples_sep PARTITION OF samples
FOR VALUES FROM ('2026-09-01 00:00:00+00') TO ('2026-10-01 00:00:00+00');
INSERT INTO samples VALUES ('2026-09-01 00:00:00+00',1),('2026-09-30 23:59:59.999999+00',2);
DO $$ BEGIN
 INSERT INTO samples VALUES ('2026-10-01 00:00:00+00',3);
 RAISE EXCEPTION 'upper boundary entered September';
EXCEPTION WHEN check_violation THEN RAISE NOTICE 'upper boundary needs another partition';
END $$;
CREATE TEMP TABLE samples_default PARTITION OF samples DEFAULT;
INSERT INTO samples VALUES ('2026-10-01 00:00:00+00',3);
DO $$ BEGIN
 CREATE TEMP TABLE samples_oct_bad PARTITION OF samples
 FOR VALUES FROM ('2026-10-01 00:00:00+00') TO ('2026-11-01 00:00:00+00');
 RAISE EXCEPTION 'new partition overlapped populated default';
EXCEPTION WHEN check_violation THEN RAISE NOTICE 'matching default rows must be moved first';
END $$;
CREATE TEMP TABLE samples_oct (LIKE samples INCLUDING ALL);
WITH moved AS (
 DELETE FROM samples_default
 WHERE at >= TIMESTAMPTZ '2026-10-01 00:00:00+00' AND at < TIMESTAMPTZ '2026-11-01 00:00:00+00'
 RETURNING *
) INSERT INTO samples_oct SELECT * FROM moved;
ALTER TABLE samples ATTACH PARTITION samples_oct
FOR VALUES FROM ('2026-10-01 00:00:00+00') TO ('2026-11-01 00:00:00+00');
SELECT id,at,tableoid::regclass AS physical_table FROM samples ORDER BY id;
DO $$ BEGIN
 IF (SELECT array_agg(id ORDER BY id) FROM samples_sep) IS DISTINCT FROM ARRAY[1,2] OR
 (SELECT array_agg(id ORDER BY id) FROM samples_oct) IS DISTINCT FROM ARRAY[3] OR
 (SELECT count(*) FROM samples_default) <> 0 THEN
 RAISE EXCEPTION 'range migration lost or duplicated data';
 END IF;
END $$;
ROLLBACK;

PostgreSQL 18.6 accepted the first instant of September and the final tested instant before October. October first was rejected before a destination existed, then entered DEFAULT after that partition was created. Attempting to add an overlapping October partition failed while the matching default row remained there.

The script moved that row using DELETE RETURNING, attached its staging table as October, and displayed each row’s actual tableoid. Ids one and two remained in September, id three appeared in October, and DEFAULT was empty. Assertions verify the final membership and count. This tests both boundary inclusion and a complete relocation path, without treating a quiet default partition as evidence that future range management can be omitted.

Sources

PostgreSQL table partitioning, range boundaries and routing behavior, checked 2026-09-10.
PostgreSQL ALTER TABLE, attach validation, default partitions, and locks, checked 2026-09-10.

Window functions rank rows without collapsing groups

Tue, 08 Feb 2028 00:00:00 +0000

Ranking functions attach a position to each input row within a partition. They do not collapse a team, department, or customer into one aggregate row. The important choice is what a tie means: a unique position, a shared competition rank with gaps, or a shared rank counting distinct score groups.

Three questions, three rankings

row_number() assigns successive integers starting at one. If two rows have the same ordering values, their relative positions are unspecified unless the window order includes a tie-breaker. For an exactly-two-winners rule, order by score descending and then a stable unique identifier. The identifier decides which equally scored entrant takes the remaining place, so that policy should be intentional.

rank() gives peers the position of the first row in their peer group. Scores 100, 90, 90, and 80 therefore produce ranks 1, 2, 2, and 4. The two second-place rows consume two positions, leaving a gap before the next score. dense_rank() instead counts peer groups, producing 1, 2, 2, and 3 for the same scores.

Peers are defined by all expressions in the window’s ORDER BY. Adding a unique identifier to a rank window eliminates score ties, even if the developer only intended to make display order deterministic. Use separate window definitions when row_number needs an identifier but rank should retain shared places. Apply an outer ORDER BY to present the result consistently without redefining the ranking peers.

Partition and filter deliberately

PARTITION BY restarts numbering for each group. Omitting it creates one partition over the rows visible to the window. This is distinct from a window frame: ranking is based on partition ordering and peers, not on a moving aggregate frame. A ROWS boundary does not turn rank into a count of rows inside that boundary.

Windows see the rows left after the query’s FROM, WHERE, grouping, and HAVING stages. Filtering in the same query can therefore change the competitors before they are ranked. To rank everyone and then keep the first positions, calculate the window values in a subquery or CTE and filter in the outer query. PostgreSQL does not permit a window function directly in WHERE.

“Top two” needs a business definition. Position at most two returns at most two rows per partition. Rank at most two includes everyone tied at second place and can return more rows. Dense rank at most two selects the first two distinct score groups, which can differ from competition rank when the leading score itself has several peers. These are different result sets, not interchangeable optimizations.

When it breaks

A leaderboard may fluctuate among equal scores when row_number lacks a deterministic tie-breaker. Conversely, adding an identifier everywhere can silently remove the ties that rank was meant to preserve. Test tied values at the cutoff, not only unique scores. If scores can be NULL, decide their placement explicitly; PostgreSQL descending order puts nulls first unless NULLS LAST is specified.

Deterministic ordering also does not freeze a changing dataset. New scores can move existing positions between requests, so offset pages or stored row numbers are not durable identities. Large rankings can require sorting and temporary files; multiple window orders may require different processing. Read representative plans rather than assuming that every window query performs one inexpensive pass.

Check it quickly

Save this as ranking.sql and run psql -X -v ON_ERROR_STOP=1 -f ranking.sql in a disposable PostgreSQL 18 database. The fixture contains a tie in red and an independent blue partition, and rolls back its temporary table and view.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE scores (id integer, team text, score integer);
INSERT INTO scores VALUES (1,'red',100),(2,'red',90),(3,'red',90),(4,'red',80),(5,'blue',50);
CREATE TEMP VIEW ranked AS
SELECT *, row_number() OVER (PARTITION BY team ORDER BY score DESC,id) AS position,
 rank() OVER w AS rank_with_gaps, dense_rank() OVER w AS rank_without_gaps
FROM scores WINDOW w AS (PARTITION BY team ORDER BY score DESC);
SELECT * FROM ranked ORDER BY team,id;
SELECT count(*) AS top_two_positions FROM ranked WHERE team='red' AND position<=2;
SELECT count(*) AS top_two_ranks FROM ranked WHERE team='red' AND rank_with_gaps<=2;
DO $$ BEGIN
 IF (SELECT array_agg(rank_with_gaps ORDER BY id) FROM ranked WHERE team='red')
 IS DISTINCT FROM ARRAY[1,2,2,4]::bigint[] THEN RAISE EXCEPTION 'wrong rank'; END IF;
 IF (SELECT array_agg(rank_without_gaps ORDER BY id) FROM ranked WHERE team='red')
 IS DISTINCT FROM ARRAY[1,2,2,3]::bigint[] THEN RAISE EXCEPTION 'wrong dense rank'; END IF;
 IF (SELECT array_agg(id ORDER BY position) FROM ranked WHERE team='red' AND position<=2)
 IS DISTINCT FROM ARRAY[1,2] THEN RAISE EXCEPTION 'unstable top two'; END IF;
 IF (SELECT count(*) FROM ranked WHERE team='red' AND rank_with_gaps<=2) <> 3 THEN
 RAISE EXCEPTION 'tied second place was dropped';
 END IF;
END $$;
ROLLBACK;

On PostgreSQL 18.6, red received positions 1 through 4, competition ranks 1, 2, 2, 4, and dense ranks 1, 2, 2, 3. Blue started at one independently. Filtering red by position returned ids 1 and 2; filtering by rank at most two returned three rows, preserving both second-place entrants.

The assertions check rank arrays, the chosen tie-breaker, and the cutoff count. Changing the shared rank window to include id should make these expectations fail, demonstrating that display stability and peer semantics cannot be mixed casually. All observations concern one fixed dataset; use a separate snapshot or pagination design when a report must remain stable while scores change.

Sources

PostgreSQL window functions, ranking functions and peer groups, checked 2026-09-10.
PostgreSQL window tutorial, partitions, ordering, and filtering after ranking, checked 2026-09-10.

LATERAL lets each row open its own subquery

Wed, 19 Jan 2028 00:00:00 +0000

LATERAL lets a FROM item refer to columns from preceding items. It is useful when each parent needs a small, differently parameterized result, such as its two newest orders. The query remains one SQL statement and one database round trip, while the lateral expression states which outer values determine each result. It does not promise one particular physical execution strategy.

Keep the correlation visible

A subquery in FROM normally cannot reference a sibling table’s columns. LATERAL permits that reference on the appropriate side of the join. A condition such as o.customer_id = c.id is the connection between the current customer and its orders; the keyword alone does not create that condition. Without correlation, the same global result can be repeated for every parent.

For top-N lookups, place the parent’s filter, ORDER BY, and LIMIT inside the lateral subquery. A LIMIT outside the join restricts the final combined result, not the number of orders per customer. Add a unique tie-breaker to the ordering: two orders can share the same timestamp, and ordering only by that timestamp leaves their relative choice unspecified.

The subquery’s ORDER BY chooses its top rows, but does not guarantee final presentation order. Apply an outer ORDER BY when the client requires a stable sequence of customers and orders. This distinction matters for exported reports and tests, where relying on today’s apparent plan order creates intermittent changes after an upgrade or statistics refresh.

Decide what an empty result means

CROSS JOIN LATERAL drops a parent when its subquery returns no rows. LEFT JOIN LATERAL with ON true preserves that parent, filling the right-hand columns with nulls. This is often the required behavior for customers who have not placed an order. A right-side condition moved into the outer WHERE can reject that null-extended row and undo the intended preservation.

Aggregation adds another distinction. COUNT(r.id) counts non-null matched order identifiers; COUNT(*) also counts a left join’s synthetic row. A correlated aggregate without GROUP BY can itself return one row for an empty input, with COUNT zero or SUM null. Read the returned row shape rather than treating every lateral expression as an identical list lookup.

When it breaks

Removing the outer reference to make a syntax error disappear changes the query’s meaning. Every customer may receive the same newest orders, and inflated totals can still look plausible. Test parents with no matches, one match, several matches, and tied ordering values. An assertion about exact identifiers is stronger than checking that the query ran successfully.

Performance can also deteriorate when an expensive inner operation repeats for many outer rows. An index beginning with the correlation key and continuing with the requested ordering can support selective top-N lookups. It is not guaranteed to be chosen, especially for tiny tables. Read EXPLAIN’s actual loops and rows, and compare a window-function formulation when processing a large fraction of the child table. LATERAL removes client round trips; it does not erase database work.

Check it quickly

Save the following as lateral.sql and run psql -X -v ON_ERROR_STOP=1 -f lateral.sql against a disposable PostgreSQL 18 database. It creates three customers, four orders, a matching index, and a temporary view inside one transaction.

\set ON_ERROR_STOP on
BEGIN;
SET LOCAL statement_timeout = '5s';
CREATE TEMP TABLE customers (id integer PRIMARY KEY, name text);
CREATE TEMP TABLE orders (id integer PRIMARY KEY, customer_id integer, placed_at date);
INSERT INTO customers VALUES (1,'Ada'), (2,'Lin'), (3,'Sam');
INSERT INTO orders VALUES
 (10,1,'2026-09-01'), (11,1,'2026-09-02'), (12,1,'2026-09-02'),
 (20,2,'2026-09-01');
CREATE INDEX orders_recent ON orders (customer_id, placed_at DESC, id DESC);
ANALYZE customers;
ANALYZE orders;
CREATE TEMP VIEW recent AS
SELECT c.id AS customer_id, r.id AS order_id
FROM customers c LEFT JOIN LATERAL (
 SELECT o.id FROM orders o WHERE o.customer_id=c.id
 ORDER BY o.placed_at DESC, o.id DESC LIMIT 2
) r ON true;
SELECT * FROM recent ORDER BY customer_id, order_id DESC;
SELECT c.id, count(r.id) AS selected_orders
FROM customers c LEFT JOIN LATERAL (
 SELECT o.id FROM orders o WHERE o.customer_id=c.id
 ORDER BY o.placed_at DESC, o.id DESC LIMIT 2
) r ON true GROUP BY c.id ORDER BY c.id;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT * FROM recent;
DO $$ BEGIN
 IF (SELECT array_agg(order_id ORDER BY order_id) FROM recent WHERE customer_id=1)
 IS DISTINCT FROM ARRAY[11,12] THEN
 RAISE EXCEPTION 'top two or tie-breaker is wrong';
 END IF;
 IF NOT EXISTS (SELECT FROM recent WHERE customer_id=3 AND order_id IS NULL) THEN
 RAISE EXCEPTION 'LEFT JOIN lost the customer without orders';
 END IF;
 IF (SELECT count(*) FROM recent) <> 4 THEN RAISE EXCEPTION 'wrong row count'; END IF;
END $$;
ROLLBACK;

The result contains (1,12), (1,11), (2,20), and customer three with a null order. Counts of matched orders are two, one, and zero. The tied date for orders eleven and twelve exercises the identifier tie-breaker; customer three exercises empty-result preservation. Assertions verify the identifiers and total row count, and the final rollback removes all objects.

PostgreSQL 18.6 used repeated small scans for this fixture, which is reasonable for four orders. That observation does not prove the proposed index is ineffective at scale. Test production-shaped parent counts and child distributions before drawing a performance conclusion. The five-second statement timeout bounds the experiment, while correctness depends on correlation, join type, and deterministic ordering.

Sources

PostgreSQL table expressions, lateral references and outer join semantics, checked 2026-09-10.
PostgreSQL SELECT, ordering and limit behavior, checked 2026-09-10.

Partitioning splits one table into many pieces that look like one

Thu, 23 Dec 2027 00:00:00 +0000

Declarative partitioning lets PostgreSQL expose one logical table while storing its rows in separate partitions. The parent routes writes and provides a common query target; the physical partitions hold the data. Its strongest benefits often concern lifecycle management and selective access, but the choice also affects keys, query shape, and operational locking.

A logical parent with physical children

A partitioned parent has no row storage of its own. Each attached partition holds rows permitted by its bounds and can have its own storage parameters, indexes, and maintenance activity. A partitioned index similarly represents a collection of indexes on the physical partitions. Splitting a table does not copy every row into both a parent store and a child store.

Range, list, and hash partitioning divide data according to different rules. Choose the key from real access and retention requirements: monthly event data may fit range partitioning, while other workloads need a different distribution. Declarative partitioning does not by itself distribute those partitions across independent database servers. Local partitions still share the resources and transactional environment of their database.

Keeping the parent name stable can simplify application access, but it does not prove that application code needs no changes. Queries must expose useful partition-key restrictions to benefit from pruning. Keys and foreign-key references may need redesign, and operations that previously addressed a globally unique identifier must retain their intended identity semantics.

The key affects more than routing

A unique or primary-key constraint on a partitioned table must include all partition-key columns, and the partition key must not contain expressions for these constraints. PostgreSQL’s individual partition indexes cannot independently prove arbitrary uniqueness across the other partitions. A composite key on day and id therefore does not promise that id alone is unique across all days.

That distinction matters during a migration from a single table with a global id key. Merely adding the date to the primary key can weaken the original identity contract. Review callers, references, and duplicate-prevention requirements before changing it. The partition key is not just a physical tuning option that can be changed without examining the public data model.

Updating a partition key can move a row to another partition. The destination must exist and accept the row, and indexes and constraints still apply. Such updates are real data changes, not free metadata edits. If retention has already removed the destination range, a late correction may fail even though the logical parent table still exists.

When it breaks

DETACH removes a table from the partition hierarchy while retaining that table and its rows. Queries through the parent stop seeing it, but the detached relation remains available for separate inspection or export. DROP removes the partition’s data as well; detaching is not deletion, and neither operation automatically creates a verified archive or backup.

Partition maintenance takes locks. A plain detach needs a strong lock on the parent, so it can wait behind long-running work and affect concurrent access. PostgreSQL provides a concurrent detach form with different locking and additional restrictions, including that it cannot run inside an explicit transaction block. Choose the operation using its actual contract rather than assuming that “metadata operation” means “no blocking.”

Too many partitions can also increase planning and memory costs. There is no universal row-count threshold that makes partitioning worthwhile. Measure representative reads and maintenance operations, create future partitions before writes need them, and define how late-arriving data interacts with retention. A small well-indexed table may remain simpler than a large collection of tiny partitions.

Check it quickly

Save this as partition-lifecycle.sql and run psql -X -v ON_ERROR_STOP=1 -f partition-lifecycle.sql in a disposable PostgreSQL 18 database. The experiment uses temporary partitions and ordinary transactional detach; final rollback removes the entire fixture.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE events (day date,id integer,PRIMARY KEY(day,id)) PARTITION BY RANGE(day);
CREATE TEMP TABLE events_sep PARTITION OF events FOR VALUES FROM ('2026-09-01') TO ('2026-10-01');
CREATE TEMP TABLE events_oct PARTITION OF events FOR VALUES FROM ('2026-10-01') TO ('2026-11-01');
INSERT INTO events VALUES ('2026-09-15',1),('2026-09-30',2),('2026-10-01',3);
SELECT id,day,tableoid::regclass AS physical_table FROM events ORDER BY id;
DO $$ BEGIN
 CREATE UNIQUE INDEX events_id_only ON events(id);
 RAISE EXCEPTION 'global id uniqueness accepted without partition key';
EXCEPTION WHEN feature_not_supported THEN RAISE NOTICE 'unique key must include partition key';
END $$;
SAVEPOINT id_scope;
INSERT INTO events VALUES ('2026-10-03',1);
SELECT id,day FROM events WHERE id=1 ORDER BY day;
ROLLBACK TO id_scope;
RELEASE id_scope;
UPDATE events SET day='2026-10-02' WHERE id=2;
SELECT id,day,tableoid::regclass AS physical_table FROM events ORDER BY id;
ALTER TABLE events DETACH PARTITION events_sep;
SELECT (SELECT count(*) FROM events) AS attached_rows,(SELECT count(*) FROM events_sep) AS detached_rows;
DO $$ BEGIN
 IF (SELECT array_agg(id ORDER BY id) FROM events) IS DISTINCT FROM ARRAY[2,3] OR
 (SELECT array_agg(id ORDER BY id) FROM events_sep) IS DISTINCT FROM ARRAY[1] THEN
 RAISE EXCEPTION 'detach changed stored rows';
 END IF;
END $$;
ROLLBACK;

PostgreSQL 18.6 routed ids one and two to September and id three to October. It rejected a parent unique index on id alone. Within a savepoint, a second id one on another day was accepted under the composite primary key; rolling back that savepoint restores the original fixture.

Updating id two’s day moved it to October, as tableoid showed. Detaching September then left two rows visible through the parent and one row in the detached table. Assertions verify the exact identifiers on both sides, proving that detach changed membership without discarding stored data. This validates the lifecycle and key semantics, not production lock duration or a complete archival procedure.

Sources

PostgreSQL table partitioning, physical storage, keys, routing, and partition management, checked 2026-09-10.
PostgreSQL ALTER TABLE, attach and detach behavior and locking restrictions, checked 2026-09-10.

A Common Table Expression names a query you can reuse

Sun, 19 Dec 2027 00:00:00 +0000

A common table expression gives a query result a name within one SQL statement. It can make a complex report readable by naming populations and intermediate calculations before the final SELECT. That name is not a persistent table, and it does not by itself promise a cached result, a particular scan count, or a sequence of procedural steps.

Scope first, execution second

A WITH clause belongs to its statement. A later statement cannot reuse its CTE name, and there is no schema object to index or query after execution. Use a temporary table, view, or another appropriate database object if results or definitions must outlive that statement. A CTE is useful when its scope and its named result match the work being expressed.

In modern PostgreSQL, a nonrecursive SELECT CTE without volatile functions can be folded into its parent query. By default, this happens when it is referenced once. Folding lets outer restrictions participate in optimization, so an outer primary-key lookup can reach an index on the underlying table instead of filtering a separately produced intermediate result.

When the same eligible CTE is referenced more than once, PostgreSQL normally materializes it instead. That can avoid repeating expensive calculations, but it can also prevent a restrictive condition at one use from being pushed into the shared computation. This distinction has applied since PostgreSQL 12; assuming that every WITH is always an optimization fence describes older behavior, not the current default.

Materialization is a tradeoff

MATERIALIZED requests separate computation. NOT MATERIALIZED permits folding even with repeated references, potentially repeating work while allowing each use to access only the rows it needs. PostgreSQL ignores NOT MATERIALIZED for recursive or non-side-effect-free CTEs; it does not authorize duplicating arbitrary writes or volatile operations.

A materialized SELECT CTE is not necessarily evaluated to exhaustion. Its consumers may demand only part of its result, though sorts, joins, or other operations can require more. Materialization also does not create a reusable indexed table for later statements. Inspect the plan and measure the actual workload before forcing a boundary merely because the word sounds like a cache.

The useful comparison is total work: repeated selective index probes may be cheaper than building and scanning a large shared result, while repeating an expensive calculation may be worse. A readable name and an efficient execution plan are compatible, but neither implies the other. Keep the business predicate in one authoritative definition and choose execution controls only when the evidence supports them.

When it breaks

Data-modifying CTEs follow different rules. INSERT, UPDATE, or DELETE inside WITH executes once and to completion even if its output is not consumed. The named relation is the RETURNING output, not a freshly rescanned target table. A modifying CTE without RETURNING still performs its modification but cannot supply rows to a later reference.

The modifying substatements and main statement use the same snapshot, and their actual modification order is not a procedural top-to-bottom guarantee. Pass changed rows through RETURNING when another part of the statement needs them. Avoid attempting to modify the same row twice in one statement and expecting textual order to decide the winner. A plain SELECT-looking outer query does not make a data-modifying WITH read-only.

Check it quickly

Save this as cte.sql and run psql -X -v ON_ERROR_STOP=1 -f cte.sql in a disposable PostgreSQL 18 database. It compares four read plans, then moves two temporary rows using DELETE RETURNING and INSERT. The final rollback removes the fixture and reverses its changes.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE items (id integer PRIMARY KEY, value text);
INSERT INTO items SELECT g,'item-'||g FROM generate_series(1,10000) AS g;
ANALYZE items;
EXPLAIN (COSTS OFF)
WITH source AS (SELECT * FROM items) SELECT * FROM source WHERE id=7;
EXPLAIN (COSTS OFF)
WITH source AS MATERIALIZED (SELECT * FROM items) SELECT * FROM source WHERE id=7;
EXPLAIN (COSTS OFF)
WITH source AS (SELECT * FROM items)
SELECT a.id FROM source a JOIN source b USING(id) WHERE a.id=7;
EXPLAIN (COSTS OFF)
WITH source AS NOT MATERIALIZED (SELECT * FROM items)
SELECT a.id FROM source a JOIN source b USING(id) WHERE a.id=7;
CREATE TEMP TABLE moved (id integer,value text);
WITH removed AS (DELETE FROM items WHERE id<=2 RETURNING *)
INSERT INTO moved SELECT * FROM removed;
SELECT (SELECT count(*) FROM items) AS remaining,(SELECT count(*) FROM moved) AS moved;
DO $$ BEGIN
 IF (SELECT array_agg(id ORDER BY id) FROM moved) IS DISTINCT FROM ARRAY[1,2] THEN
 RAISE EXCEPTION 'RETURNING handoff failed';
 END IF;
 IF (SELECT count(*) FROM items) <> 9998 THEN RAISE EXCEPTION 'DELETE did not execute'; END IF;
END $$;
ROLLBACK;

On PostgreSQL 18.6, the single-reference default plan used items_pkey directly. Explicit MATERIALIZED introduced a CTE Scan over a planned sequential scan. The default double reference also shared a materialized CTE, while NOT MATERIALIZED produced two index-based accesses. These plain EXPLAIN statements show planned structure; they do not measure execution time or prove a production performance improvement.

The modifying statement actually executed and left 9,998 rows in items and two in moved. Assertions verify that the transferred identifiers are exactly one and two and that the source rows were deleted. RETURNING provides the handoff between operations. Changing the outer statement’s demand is not a safe method for suppressing a modifying CTE; keep such experiments transactional and inspect all WITH bodies before executing unfamiliar SQL.

Sources

PostgreSQL WITH queries, folding, materialization, and data-modifying CTEs, checked 2026-09-10.
PostgreSQL SELECT reference, WITH scope and MATERIALIZED controls, checked 2026-09-10.

Prepared statements separate values from SQL structure

Sat, 16 Oct 2027 00:00:00 +0000

Parameter binding gives a database value a designated position in a SQL statement. A name containing a quote or a semicolon remains a value instead of becoming another clause. Reusing a named prepared statement adds a separate optimization: the server can retain statement information across executions in one session. Safe binding does not require the application to create a named statement for every query.

When it breaks

In Psycopg, %s is a driver placeholder, not an invitation to use Python string formatting. Pass the values as the second argument to execute, leave the placeholder unquoted, and use a one-element tuple for one positional value. Interpolating a value first and then calling a parameter-capable API has already crossed the boundary the API was supposed to preserve.

A placeholder represents a value, not a table, column, or keyword. Dynamic identifiers require a deliberate structural choice. Use an allow-list to restrict the supported columns, then the driver’s identifier composition facility to quote that identifier correctly. Keep direction keywords under application control. Numeric LIMIT and OFFSET values can be bound, while still requiring product-specific range validation.

Preparation does not freeze the first execution plan forever. PostgreSQL distinguishes custom plans, built using current parameter values, from generic plans reused across values. Automatic selection considers planning overhead and estimated execution costs. Schema or statistics changes can also trigger replanning. A statement can remain parameterized while its access path changes; plan reuse and injection resistance are different properties.

Check it quickly

Use Python with Psycopg 3 installed and set PG_DSN to a disposable PostgreSQL database. Supply connection configuration through your existing secure mechanism. The recorded run used Psycopg 3.3.5 and PostgreSQL 18.6. Save the following as a Python file and execute it with that environment variable set. The table is temporary and disappears when the connection closes:

import os
import psycopg
from psycopg import sql

with psycopg.connect(os.environ['PG_DSN'], autocommit=True,
 prepare_threshold=5) as conn:
 conn.execute('CREATE TEMP TABLE people (id integer PRIMARY KEY, label text NOT NULL)')
 conn.execute("INSERT INTO people VALUES (1,'alice'),(2,'bob')")
 query = 'SELECT id FROM people WHERE label=%s ORDER BY id LIMIT %s'
 hostile = "' OR true --"
 assert conn.execute(query, (hostile, 10), prepare=False).fetchall() == []
 count = conn.execute('SELECT count(*) FROM pg_prepared_statements').fetchone()[0]
 assert count == 0
 print('bound without named preparation:', count, 'named statements; no match')
 for value in ("O'Reilly", r'a\b', 'x; DROP TABLE people; --'):
 assert conn.execute('SELECT %s::text', (value,), prepare=False).fetchone() == (value,)
 for label, expected in (('alice', [(1,)]), ('bob', [(2,)])):
 assert conn.execute(query, (label, 10), prepare=True).fetchall() == expected
 assert conn.execute('SELECT count(*) FROM pg_prepared_statements').fetchone() == (1,)
 print('named preparation reused: alice=1, bob=2')

 def ordered(column, limit):
 if column not in {'id', 'label'}:
 raise ValueError('unsupported sort column')
 if type(limit) is not int or not 1 <= limit <= 100:
 raise ValueError('limit must be an integer from 1 through 100')
 statement = sql.SQL('SELECT id FROM people ORDER BY {} LIMIT %s').format(
 sql.Identifier(column))
 return conn.execute(statement, (limit,), prepare=False).fetchall()

 assert ordered('label', 1) == [(1,)]
 for column, limit in [('label; DROP TABLE people', 1), ('id', 0), ('id', 101)]:
 try:
 ordered(column, limit)
 except ValueError:
 pass
 else:
 raise AssertionError('invalid request accepted')
 print('identifier allow-list and bound LIMIT checked')

 conn.execute('PREPARE plan_lab(text) AS SELECT id FROM people WHERE label=$1')
 conn.execute('SET plan_cache_mode=force_custom_plan')
 assert conn.execute("EXECUTE plan_lab('alice')").fetchall() == [(1,)]
 counters = "SELECT generic_plans,custom_plans FROM pg_prepared_statements WHERE name='plan_lab'"
 assert conn.execute(counters).fetchone() == (0, 1)
 conn.execute('SET plan_cache_mode=force_generic_plan')
 assert conn.execute("EXECUTE plan_lab('bob')").fetchall() == [(2,)]
 assert conn.execute(counters).fetchone() == (1, 1)
 print('same prepared statement: custom=1, generic=1')
 conn.execute('RESET plan_cache_mode')
 conn.execute('DEALLOCATE plan_lab')

The first result reports zero named statements and no match for the hostile value. That call explicitly disables named preparation while retaining binding, so it demonstrates that the safety boundary exists before statement reuse. Quotes, backslashes, and semicolons then round-trip unchanged; the table remains available for subsequent queries.

Two calls with prepare=True reuse one named statement and return Alice’s identifier one and Bob’s identifier two. The configured threshold permits explicit preparation. Setting prepare_threshold=None instead disables named preparation altogether in the tested driver, including explicit requests. Do not confuse that switch with disabling value binding.

The ordering function returns one row for a supported column and bound limit. It rejects an unsupported column and limits outside one through one hundred before issuing SQL. Identifier quoting protects syntax; the allow-list defines which valid syntax the endpoint actually supports. Neither decision should be delegated to arbitrary request text.

Finally, the same SQL prepared statement executes once under forced custom planning and once under forced generic planning. Its counters change from zero generic and one custom plan to one of each. Both return the expected row. These forced settings make the distinction observable; they do not measure what automatic planning would choose for a realistic skewed workload or recommend forcing either mode globally.

Inspect reuse in the real deployment

PostgreSQL’s default automatic policy starts parameterized prepared statements with custom plans and later compares a generic alternative. Look at representative parameter distributions and actual execution plans before deciding whether reuse helps. This two-row exercise makes no latency or network-round-trip claim. Driver protocol batching, connection setup, and query cost affect those measurements independently.

Named statements belong to server sessions. Connection pools and transaction-pooling proxies must manage that lifetime correctly, using explicitly supported prepared-statement behavior or disabling named reuse where necessary. Binding remains the default even for a query executed only once. Continue to enforce tenant predicates, object permissions, and result-size limits: a safely parameterized query can still return data the caller was never meant to see.

Sources

Psycopg parameter API, binding syntax and identifier composition, checked 2026-09-12.
Psycopg prepared statements, thresholds, explicit preparation, and session lifetime, checked 2026-09-12.
PostgreSQL 18 PREPARE, custom and generic plans and automatic selection, checked 2026-09-10.

Query planners bet on cost, not certainty

Fri, 17 Sep 2027 00:00:00 +0000

A query planner chooses an execution path using estimated work. It does not run every candidate, time them, and keep the winner. PostgreSQL compares possible scans, joins, sorts, and other operations using statistics and a cost model. The chosen plan is therefore a prediction under current assumptions. Understanding a wrong prediction requires identifying which assumption failed, rather than treating a sequential scan or a nested loop as inherently suspicious.

Costs describe a model

PostgreSQL costs combine estimates of page access, tuple processing, operator work, and related activity. These units are not milliseconds. Startup cost matters when a consumer needs only a few rows, while total cost matters when the full result must be produced. Useful ordering can avoid a later sort, so the cheapest isolated scan is not necessarily the best input to the complete query.

The search also has limits. PostgreSQL considers alternative paths and join orders rather than executing them all. Large join problems can use the genetic query optimizer to control planning effort. Available indexes, legal transformations, configuration, and search choices constrain the candidates. Calling the selected plan mathematically optimal over every imaginable implementation overstates what the optimizer guarantees.

When it breaks

Fresh statistics can still produce a bad estimate. Ordinary statistics describe columns individually, so combining two predicates may assume independence even when the columns always move together. An underestimated intermediate result can make repeated lookups appear cheaper than they become, or distort later join choices. Repeating the same analysis without changing the statistical model cannot teach it a missing relationship.

Extended statistics address particular questions, not every estimation error. Functional dependencies help compatible equality conditions against constants, but do not establish the actual mapping between values. They are not general proofs for range filters or cross-column comparisons. An incompatible pair of constants can remain badly estimated even after a dependency is collected.

Check it quickly

Run this temporary-table experiment with PostgreSQL 18 and psql. Both columns contain the same one hundred distinct values, each repeated one hundred times. The final rollback removes the table and its statistics object.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE dimensions (a integer,b integer);
INSERT INTO dimensions SELECT g%100,g%100 FROM generate_series(1,10000) AS g;
ANALYZE dimensions;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT * FROM dimensions WHERE a=7 AND b=7;
CREATE STATISTICS pg_temp.dimensions_dependency (dependencies) ON a,b FROM dimensions;
ANALYZE dimensions;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT * FROM dimensions WHERE a=7 AND b=7;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) SELECT * FROM dimensions WHERE a=7 AND b=8;
DO $$ BEGIN
 IF (SELECT count(*) FROM dimensions WHERE a=7 AND b=7) <> 100 OR
 (SELECT count(*) FROM dimensions WHERE a=7 AND b=8) <> 0 THEN RAISE EXCEPTION 'wrong correlated population'; END IF;
END $$;
ROLLBACK;

Before the extended statistics, PostgreSQL 18.6 estimated one row for a=7 AND b=7, although the query returned one hundred. Each condition individually selects one percent; multiplying those fractions predicts one hundredth of a percent of ten thousand rows. The calculation is reasonable only under an independence assumption that this fixture deliberately violates.

Creating the dependencies object tells the next ANALYZE which relationship to collect. The second plan estimated one hundred rows and returned one hundred. Merely defining the object would not have populated it. The scan remained sequential because this example has no index and still examines the same small table. A better cardinality estimate is demonstrated; a faster application is not.

The final predicate asks for a=7 AND b=8. It returns zero rows, but the dependency estimate remains one hundred. This is the important negative case: knowing that one column determines another does not tell this statistics kind that seven must map specifically to seven. The assertions distinguish correct query results from inaccurate predictions; estimation errors do not change SQL semantics.

Choose the next experiment from the discrepancy

A multivariate common-value list can represent frequent value combinations when that is the missing information. Other workloads may need different statistics, an appropriate index, or a rewritten expression that exposes a usable condition. Add only the object justified by the observed predicates; unnecessary statistics consume analysis and planning resources too.

When comparing alternatives, keep the dataset, parameters, relevant settings, and result requirements consistent. Include rows processed, buffer activity, memory spills, and measured latency. A locally cheaper-looking node is not sufficient evidence if it creates more work upstream. Conversely, accurate row estimates do not rule out waiting, costly I/O, or an unsuitable cost model. Follow the specific difference between predicted and observed work before changing global planner settings.

Sources

PostgreSQL 18: Planner and optimizer, candidate access paths and join strategies, checked 2026-09-10.
PostgreSQL 18: Planner statistics, column distributions and extended-statistics limits, checked 2026-09-10.
PostgreSQL 18: Multivariate statistics examples, correlated predicates and dependency estimates, checked 2026-09-10.

Composite indexes match multi-column queries

Wed, 15 Sep 2027 00:00:00 +0000

A composite B-tree index orders tuples by its first key, then by the next key within each first-key group. This structure can make a tenant-specific time range cheap to locate. The useful rule is about how much index space a predicate narrows, not a prohibition against mentioning later columns or writing WHERE conditions in a different textual order.

Order the keys, not the SQL clauses

For an index on (tenant, happened), equality on tenant and a range on happened describe a compact portion of the tree. PostgreSQL can search that range rather than inspecting all tenants. Writing the time condition before the tenant condition in SQL does not change the logical conjunction or the index’s key order. The schema’s column order and the query’s presentation order are different things.

Leading equality conditions and the first non-equality key generally determine the range that must be scanned. Conditions on later keys can still be checked within the index, saving heap visits even when they do not further shrink a simple contiguous range. Reversing key columns changes these opportunities; it does not make an index universally unusable.

Later keys can still help

PostgreSQL 18 can use B-tree skip scans when repeated searches over missing leading-key values are cheaper than reading most of the index. With only a few tenants, a predicate on happened alone can search within each tenant group. With many distinct leading values, that strategy may be too expensive, and the planner can prefer a sequential scan or another index.

Even without a profitable skip scan, a composite index can sometimes be scanned for a condition involving a later key. The question is cost, coverage, and selectivity, not whether such SQL is legal or whether the index must be ignored. Do not use the shorthand “leftmost prefix” as an absolute claim that contradicts the target engine’s actual access methods.

Include ordering and the workload

Fixing the tenant key can make the following happened key useful for ordered results. Across several tenants, however, the index groups by tenant first; it does not provide one globally time-sorted stream. A bitmap scan also loses index ordering and may need a separate sort. Look at the complete plan instead of inferring final order from an index declaration.

Equality-first and range-next is a useful starting point for one query shape, but there is no universal “most selective column first” recipe. Shared prefixes, joins, sort direction, LIMIT, parameter distributions, and write costs all affect the choice. Compare actual recurring queries, then keep the smallest useful set of indexes. Redundant wide indexes consume cache and maintenance work even when nobody benefits from them.

When it breaks

A team may build (tenant, happened) for per-tenant queries, then add a global timeline and assume it is equally efficient. The timeline’s ordering and selectivity differ. Another regression comes from testing one tiny tenant while another owns most rows: the same parameterized SQL can have very different costs. Statistics and representative values matter more than a screenshot of one favorable plan.

Incorrect rules also cause unnecessary index churn. Adding one index per suffix because “the existing index cannot be used” can duplicate useful storage. Conversely, assuming skip scan always rescues a missing prefix can hide a costly path. Measure the chosen plan, actual rows, buffers, and sort work before changing the schema.

Check it quickly

Save this as composite.sql and run psql -X -v ON_ERROR_STOP=1 -f composite.sql on a disposable PostgreSQL 18 database. The temporary fixture contains four tenant values and twenty thousand ordered event numbers.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE events (tenant integer, happened integer, body text);
INSERT INTO events SELECT g % 4, g, repeat('x',40) FROM generate_series(1,20000) AS g;
CREATE INDEX events_tenant_time ON events (tenant, happened);
ANALYZE events;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT happened FROM events WHERE tenant=2 AND happened BETWEEN 100 AND 120
ORDER BY happened;
SELECT array_agg(happened ORDER BY happened) AS matching_times
FROM events WHERE happened BETWEEN 100 AND 120 AND tenant=2;
EXPLAIN (ANALYZE, BUFFERS, TIMING OFF)
SELECT tenant FROM events WHERE happened=12346;
DO $$ BEGIN
 IF (SELECT array_agg(happened ORDER BY happened) FROM events
 WHERE happened BETWEEN 100 AND 120 AND tenant=2)
 IS DISTINCT FROM ARRAY[102,106,110,114,118] THEN
 RAISE EXCEPTION 'wrong composite range';
 END IF;
 IF (SELECT tenant FROM events WHERE happened=12346) IS DISTINCT FROM 2 THEN
 RAISE EXCEPTION 'suffix predicate lost the row';
 END IF;
END $$;
ROLLBACK;

The tenant-range query returns exactly 102,106,110,114,118, including when the WHERE conditions are written in the opposite order. On PostgreSQL 18.6 it used a bitmap index scan followed by a sort. The suffix-only lookup returned tenant two through an index-only scan with repeated index searches, demonstrating a usable path without tenant equality.

Plan names, buffer counts, and search counts are observations for this dataset, not guaranteed outputs across configurations. The assertions verify result semantics, and rollback removes all objects. Repeat the measurement with more leading-key values to assess whether the suffix path remains worthwhile for a real workload.

Sources

PostgreSQL multicolumn indexes, leading constraints and skip scans, checked 2026-09-10.
PostgreSQL indexes and ORDER BY, ordered scans and sort choices, checked 2026-09-10.

EXPLAIN shows how Postgres will do the work

Thu, 08 Jul 2027 00:00:00 +0000

EXPLAIN prints the planner’s proposed execution plan; EXPLAIN ANALYZE executes the statement and adds measured rows and times. A slow query needs both a hypothesis and a measurement. Save the query parameters, PostgreSQL version, and original plan before changing an index or rewriting a join. Those details make the next plan comparable and prevent an improvement on one input from hiding a regression on another.

Two habits separate effective plan readers from collectors of EXPLAIN output. First, always compare estimated and actual rows before touching anything; the gap between them is the diagnosis. Second, change one thing at a time and re-check the plan, because query tuning without a before-and-after plan is indistinguishable from luck.

Read the shape top-down or leaves-first

Text output prints a parent before its indented children. Read downwards to understand which inputs a node consumes, then follow rows upwards to understand the result. Execution can be demand-driven: a LIMIT may stop requesting rows early, while a sort often needs its entire input before returning anything. Indentation describes dependencies, not a universal chronological execution order.

Costs use planner units, not milliseconds. A node’s total cost includes its children, so adding every displayed cost double-counts work. Startup cost estimates the work before the first row; total cost estimates completing the node. Actual times and rows are averages per execution when loops exceeds one. Multiply by loops when judging repeated work, and remember that parent timing also includes descendant work.

Learn the small vocabulary of node types and each plan becomes skimmable. Sequential scan, index scan, bitmap heap scan; nested loop, hash join, merge join; sort, aggregate, limit — a dozen shapes cover nearly every plan. For each, know its scaling story: nested loops love small outer inputs and punish large ones, hash joins need memory for the inner table, merge joins demand sorted inputs. The plan names the strategy; your job is knowing what each strategy costs under your data shape.

Focus on the widest node

The largest row count is a clue, not proof of the bottleneck. A cheap sequential scan can outperform many random index reads; a small node repeated thousands of times can dominate a query. Find the first substantial estimated-versus-actual row mismatch, then trace its effect on joins, sorting, and memory use. Check buffers and temporary reads or writes before deciding whether storage access, repeated execution, or a spill explains the delay.

Ask why the planner chose that shape rather than assuming stupidity. It picks the cheapest plan under its statistics; if the statistics lie, the choice looks foolish but the reasoning was sound. Check pg_stats for the misestimated column: stale stats, skewed distributions, or correlated columns the planner treats as independent. Sometimes the right fix is ANALYZE or extended statistics rather than a rewritten query — cheaper, and it fixes every query sharing the misestimate, not just this one.

Bind rows to cost with ANALYZE

EXPLAIN ANALYZE runs the real query. Large row-estimation errors can come from stale statistics, skew, or correlations between columns. Refreshing statistics may help, but does not guarantee a fix. Extended statistics can describe some correlations; a representative test must still confirm the chosen plan and result. An index is justified by the access pattern and measured cost, not just by seeing many filtered rows.

Use an isolated database for write experiments. Rolling back table changes does not undo sequence allocation or external effects from functions and triggers. Even a SELECT can call a function with side effects. Set a statement timeout, understand the statement, and use plain EXPLAIN first when execution cost is uncertain. Buffer reads also do not prove physical disk I/O: the operating system may satisfy them from its cache.

When it breaks

The classic failure is tuning the wrong query: the slow page fires five statements, the engineer optimizes the familiar one, and p99 never moves because the expensive statement was the obscure background refresh. Profile at the workload level first — pg_stat_statements ranked by total time — and let the ranking pick the target. Optimizing without ranking is how teams shave milliseconds off the innocent while the guilty runs unexamined.

Another failure is parameter sensitivity: one tenant is tiny and another holds most rows, while a prepared statement’s generic plan treats both alike. Compare representative parameters through the same prepared-statement path used by the application. PostgreSQL core has no general SQL hint syntax; investigate statistics, query shape, and generic versus custom plans before changing planner settings globally.

Check it quickly

In a writable scratch database, create data owned only by this session:

BEGIN;
SET LOCAL statement_timeout = '10s';
CREATE TEMP TABLE explain_demo AS
SELECT n AS id, n % 100 AS tenant_id FROM generate_series(1, 20000) AS g(n);
ANALYZE explain_demo;
EXPLAIN (ANALYZE, BUFFERS)
SELECT id FROM explain_demo WHERE tenant_id = 17 ORDER BY id LIMIT 10;
SELECT id FROM explain_demo WHERE tenant_id = 17 ORDER BY id LIMIT 10;
CREATE INDEX explain_demo_lookup ON explain_demo (tenant_id, id);
EXPLAIN (ANALYZE, BUFFERS)
SELECT id FROM explain_demo WHERE tenant_id = 17 ORDER BY id LIMIT 10;
SELECT id FROM explain_demo WHERE tenant_id = 17 ORDER BY id LIMIT 10;
ROLLBACK;

Both queries must return the same ten IDs. Compare scan type, sort presence, rows filtered, and buffers; the second plan will commonly use the ordered index. Do not require an exact time or node shape across versions. Repeat on representative data before applying the index to a real workload; rollback removes this experiment.

Sources

PostgreSQL EXPLAIN documentation, EXPLAIN options and execution side effects, checked 2026-09-05.
PostgreSQL using EXPLAIN, plan trees, row estimates, loops and measured costs, checked 2026-09-05.

Query parameters preserve the boundary between values and SQL

Mon, 17 May 2027 00:00:00 +0000

A parameterized query sends the database instruction and its values through distinct API arguments. The database interprets the instruction as SQL and uses each parameter where a value belongs. This boundary works in embedded engines such as SQLite as well as client-server databases; it does not depend on a network exchange or on an application explicitly managing named server statements.

When it breaks

String concatenation removes that boundary. A quote supplied as part of a name can end the intended literal, and following text can introduce a condition or comment. Blocking semicolons does not fix the problem: an attacker may change the meaning of one SELECT without adding a second statement. Similarly, an API that refuses multiple statements still needs proper parameter binding.

Binding prevents a value from becoming SQL grammar, but the chosen operator still interprets the value according to its documented rules. With LIKE, percent matches a sequence of characters and underscore matches one character. A bound search string can therefore match more records than a literal-text search intended. That is a pattern-semantics issue, not evidence that parameter binding failed.

Other mistakes produce incorrect results without injection. A comma-separated string bound to one IN placeholder remains one scalar value; it is not expanded into a list. A bound column name remains a text value rather than a column reference. Type checks, allowed structural choices, and application permissions remain necessary alongside binding.

Check it quickly

Run this complete Python script with the standard sqlite3 module. It creates only an in-memory fixture and closes it after the assertions. The recorded environment was Python 3.14.4 with SQLite 3.53.0. The deliberately vulnerable query is included solely to compare behavior against three invented users:

import sqlite3

with sqlite3.connect(':memory:') as db:
 db.execute('CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT NOT NULL)')
 db.executemany('INSERT INTO users VALUES (?, ?)', [(1, 'Alice'), (2, 'A_1'), (3, 'Ab1')])
 payload = "' OR 1=1 --"
 # Deliberately vulnerable comparison, only against this in-memory fixture.
 unsafe = "SELECT id FROM users WHERE name='" + payload + "' ORDER BY id"
 assert sorted(db.execute(unsafe).fetchall()) == [(1,), (2,), (3,)]
 assert db.execute('SELECT id FROM users WHERE name=?', (payload,)).fetchall() == []
 print('concatenated: 3 rows; bound: 0 rows')
 for value in ("O'Reilly", r'a\b', 'x; DROP TABLE users; --'):
 assert db.execute('SELECT ?', (value,)).fetchone() == (value,)
 assert db.execute('SELECT count(*) FROM users').fetchone() == (3,)
 print('quote, backslash, semicolon: literal round trips')
 assert db.execute('SELECT id FROM users WHERE name LIKE ? ORDER BY id', ('A_1',)).fetchall() == [(2,), (3,)]
 literal = 'A_1'
 escaped = literal.replace('!', '!!').replace('%', '!%').replace('_', '!_')
 assert db.execute("SELECT id FROM users WHERE name LIKE ? ESCAPE '!' ORDER BY id", (escaped,)).fetchall() == [(2,)]
 assert db.execute('SELECT id FROM users WHERE name=?', (literal,)).fetchall() == [(2,)]
 print('bound LIKE: 2 rows; escaped LIKE and equality: 1 row')
 assert db.execute('SELECT id FROM users WHERE id IN (?)', ('1,3',)).fetchall() == []
 assert db.execute('SELECT id FROM users WHERE id IN (?,?) ORDER BY id', (1, 3)).fetchall() == [(1,), (3,)]
 assert db.execute('SELECT ? FROM users LIMIT 1', ('name',)).fetchone() == ('name',)
 assert db.execute('SELECT id FROM users ORDER BY id LIMIT ? OFFSET ?', (1, 1)).fetchall() == [(2,)]
 print('IN needs separate values; bound column is a literal; LIMIT/OFFSET bind')
db.close()

The concatenated query returns all three identifiers, while the bound version returns no matches for the same input. The injected comment also removes the appended ordering clause, so the assertion sorts the vulnerable result before comparing identifiers. The test does not depend on an unordered query coincidentally returning insertion order.

The next assertions round-trip a quote, a backslash, and a semicolon-containing string exactly. The three-row count remains unchanged. These are actual calls to the SQLite driver, not a string-escaping simulation or a check that a query happens to contain a question mark. Save the SQL text and parameter assignment together when reviewing a real call site.

The bound LIKE pattern A_1 matches both A_1 and Ab1. Escaping the underscore using an explicit exclamation-mark escape reduces the result to the literal name. Escaping the escape character first is necessary when arbitrary search text can contain it. For exact matching, equality is simpler and also returns only the literal name here.

A single bound value containing 1,3 returns no identifiers. Two placeholders with two integers return identifiers one and three. The next SELECT returns the literal text name, demonstrating why a value placeholder cannot choose a projected column. Finally, bound numeric limit and offset values return identifier two. No numeric string interpolation is required for pagination.

Review the complete data path

Choose placeholders according to the actual driver. Python’s SQLite interface accepts question marks and named parameters; Psycopg uses different binding notation. Pass the parameter collection separately rather than applying language-level formatting first. A value loaded from your own database is still a value when reused in another query; storing it once does not make later interpolation safe.

For variable-length lists, construct only the required placeholder structure and bind each element, using the driver’s supported collection facilities where available. Define empty-list behavior and respect parameter-count limits. Dynamic identifiers and ordering keywords need an allowed set of query structures; filtering punctuation out of arbitrary SQL text is not a reliable substitute.

Set appropriate page-size bounds and enforce object authorization even when every value is bound. Parameterization protects the SQL grammar boundary, not the entire business operation. Tests should check ordinary results as well as adversarial strings and operator-specific edge cases, so a secure rewrite does not silently broaden searches, change ordering, or expose another user’s rows.

Sources

Python sqlite3, driver placeholders and binding API, checked 2026-09-12.
SQLite expressions, parameters, LIKE escaping, and IN semantics, checked 2026-09-12.
OWASP SQL injection prevention, parameterized values and allowed structural choices, checked 2026-09-12.

INSERT ... ON CONFLICT turns a race into one statement

Thu, 25 Mar 2027 00:00:00 +0000

INSERT … ON CONFLICT lets PostgreSQL resolve an insert collision inside the statement that performs the write. It replaces a fragile “check whether the key exists, then insert” sequence with an explicit conflict policy backed by an eligible index or constraint. It does not turn every insertion error into a successful update or define the application’s merge policy for you.

The conflict target defines the race

A target such as (name) asks PostgreSQL to infer a matching unique index. The table must actually have an eligible uniqueness rule; writing ON CONFLICT does not create one. Index expressions and partial-index predicates matter when the key is more complicated than a simple column. An unmatched target raises an error instead of silently falling back to another strategy.

DO UPDATE requires a target, and its arbiter must be a supported nondeferrable unique constraint or unique index. Exclusion constraints are not supported as arbiters for DO UPDATE. DO NOTHING has broader conflict-handling options, including supported exclusion constraints, and may omit a target to handle conflicts with usable arbiters. None of these rules makes foreign-key, not-null, or check violations disappear.

The atomic insert-or-update behavior protects against the concurrent key race. It can still wait for another transaction holding a relevant row or index conflict, and normal transaction errors can still occur. Atomicity describes the outcome’s coordination, not a promise that the statement is lock-free, never blocks, or cannot need transaction-level retry handling.

Write the merge policy explicitly

The target table name or alias refers to the existing row. EXCLUDED represents the proposed insertion, including relevant effects of BEFORE INSERT triggers. Assigning EXCLUDED.total replaces the old value; assigning the existing total plus EXCLUDED.total adds a contribution. These policies answer different business questions, even though both use the same SQL syntax.

For a counter, an update expression using the existing row avoids calculating a replacement value from an earlier client-side read. For a versioned document, a WHERE condition can reject an older proposed version. Such a condition is evaluated after identifying and locking the conflicting row; a rejected update can therefore still contend with other writers.

RETURNING emits only rows actually inserted or updated. DO NOTHING returns no row for the existing record, and DO UPDATE with a false WHERE condition also returns no row. An empty result is therefore not uniquely diagnostic of DO NOTHING. Define the caller’s interpretation of skipped work instead of treating every empty result as a missing database record.

When it breaks

A batch can contain two proposed rows with the same conflict key. DO UPDATE cannot affect one existing row twice in the same statement and raises a cardinality violation in that case. Combine or reject duplicate inputs according to the intended policy before submitting the batch. Arbitrarily keeping the last input can silently change an additive operation into replacement semantics.

ON CONFLICT also does not imply idempotency. Retrying an additive upsert after an uncertain response can add the same contribution again. A separate request identity and its uniqueness contract are needed when one logical request must be applied once. Likewise, an unconditional replacement can overwrite a newer value if an older request arrives later.

Errors from other constraints remain important. A conflicting key does not make invalid new values harmless: PostgreSQL can reject a CHECK violation before taking DO NOTHING. Sequence defaults may also be evaluated before conflict resolution, so skipped insertions can consume sequence numbers. Neither behavior means the conflict handler malfunctioned.

Check it quickly

Save this as upsert.sql and run psql -X -v ON_ERROR_STOP=1 -f upsert.sql in a disposable PostgreSQL 18 database. The temporary counter table exists only within the test transaction, which is rolled back at the end.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE tally (
 name text PRIMARY KEY,total integer NOT NULL CHECK (total>=0),version integer NOT NULL
);
INSERT INTO tally VALUES ('jobs',5,1) RETURNING *;
INSERT INTO tally VALUES ('jobs',3,2)
ON CONFLICT (name) DO UPDATE
SET total=tally.total+EXCLUDED.total,version=EXCLUDED.version
RETURNING *;
WITH changed AS (
 INSERT INTO tally VALUES ('jobs',100,3) ON CONFLICT (name) DO NOTHING RETURNING *
) SELECT count(*) AS do_nothing_returned FROM changed;
WITH changed AS (
 INSERT INTO tally VALUES ('jobs',100,1)
 ON CONFLICT (name) DO UPDATE SET total=EXCLUDED.total,version=EXCLUDED.version
 WHERE tally.version<EXCLUDED.version RETURNING *
) SELECT count(*) AS stale_update_returned FROM changed;
DO $$ BEGIN
 INSERT INTO tally VALUES ('jobs',-1,4) ON CONFLICT (name) DO NOTHING;
 RAISE EXCEPTION 'check violation was swallowed';
EXCEPTION WHEN check_violation THEN RAISE NOTICE 'CHECK still rejects invalid input';
END $$;
DO $$ BEGIN
 INSERT INTO tally VALUES ('jobs',1,4),('jobs',2,5)
 ON CONFLICT (name) DO UPDATE SET total=EXCLUDED.total;
 RAISE EXCEPTION 'same row updated twice';
EXCEPTION WHEN cardinality_violation THEN RAISE NOTICE 'duplicate input keys rejected for DO UPDATE';
END $$;
SELECT * FROM tally;
DO $$ BEGIN
 IF (SELECT total FROM tally WHERE name='jobs') IS DISTINCT FROM 8 OR
 (SELECT version FROM tally WHERE name='jobs') IS DISTINCT FROM 2 THEN
 RAISE EXCEPTION 'conflict action changed the wrong state';
 END IF;
END $$;
ROLLBACK;

PostgreSQL 18.6 inserted total five at version one, then added three and produced total eight at version two. Both the DO NOTHING case and the stale conditional update returned zero rows. The negative CHECK input and duplicate-key batch raised their expected errors, which the fixture catches specifically rather than swallowing arbitrary failures.

The final assertion confirms that rejected work did not replace the total or version. This example exercises the distinct conflict paths; it is not a latency benchmark or a guarantee that arbitrary merge expressions are safe. Review the update expression, the target uniqueness rule, and the caller’s retry semantics together before using the statement in a concurrent workflow.

Sources

PostgreSQL INSERT, arbiters, atomic conflict handling, EXCLUDED, and RETURNING, checked 2026-09-10.
PostgreSQL constraints, uniqueness, checks, nullability, and referential integrity, checked 2026-09-10.

A window frame decides which rows one aggregate sees

Tue, 23 Feb 2027 00:00:00 +0000

A window partition determines which rows belong together; a window frame determines which of those rows feed a frame-sensitive calculation for the current row. Confusing these boundaries can turn a running total into a repeated subtotal or make last_value return the current value when the report expected the final value in the partition.

Defaults depend on ordering

In PostgreSQL, the default frame is RANGE from the partition’s beginning through the current row’s last peer. Peers have equal values for every expression in the window ORDER BY. With an order, a default running sum therefore includes all rows tied with the current row, including peers that appear later in the displayed output.

Without a window ORDER BY, all rows are peers and the default frame covers the whole partition. Thus sum(amount) OVER (PARTITION BY account) repeats an account total, while adding an order changes the calculation into an ordered cumulative result. Neither “the default is always the whole group” nor “the default adds exactly one row” describes both cases correctly.

The window order is separate from the final result order. An outer ORDER BY can arrange rows for display without changing the frame. Conversely, adding a unique identifier inside a RANGE window can split a peer group, altering the calculation rather than merely making the output prettier. Treat that change as a semantic decision.

Rows, value ranges, and peer groups

ROWS counts positions in the window ordering. A frame from two preceding rows through the current row contains at most three rows, fewer near the partition start. It does not mean three days or three minutes. If multiple events share a timestamp, include a suitable unique tie-breaker when their positional order must be deterministic.

RANGE uses ordering values and peer boundaries. With an offset, PostgreSQL requires one ordering expression and a compatible offset type; a timestamp order can use an interval. A value-based time range includes events by time difference, so its row count can grow during busy periods and shrink during quiet ones. GROUPS instead counts peer groups, allowing a frame to move by distinct ordering groups while retaining all their tied rows.

First_value, last_value, nth_value, and aggregates used as windows depend on the frame. Ranking and lag or lead have different semantics based on partition order; shrinking a frame does not make lag forget the preceding partition row. Check the function’s definition before applying one generic explanation to all window functions.

When it breaks

Last_value with an ordered default frame usually means the end of the current frame, not the end of the entire partition. With unique ordering, that is the current row. To repeat the partition’s final value, specify an ending boundary of UNBOUNDED FOLLOWING and provide an order that defines what “final” means. A chronological maximum and the value of the last chronological row are different questions.

Running totals can also surprise when tied dates receive the same cumulative number. This is expected peer behavior under the default RANGE frame. If the business wants each individual event to advance the total, use an explicit ROWS frame and deterministic event ordering. Do not eliminate genuine ties from a date-level report just to obtain a more familiar-looking curve.

Check it quickly

Save this as frames.sql and run psql -X -v ON_ERROR_STOP=1 -f frames.sql in a disposable PostgreSQL 18 database. Two events share day one, and a third belongs to day two. Temporary objects and inserted rows disappear on rollback.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE sales (id integer, day integer, amount integer);
INSERT INTO sales VALUES (1,1,10),(2,1,20),(3,2,5);
CREATE TEMP VIEW framed AS
SELECT *,
 sum(amount) OVER (ORDER BY day) AS default_sum,
 sum(amount) OVER (ORDER BY day,id ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS row_sum,
 last_value(amount) OVER (ORDER BY day,id) AS default_last,
 last_value(amount) OVER (ORDER BY day,id ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS partition_last,
 sum(amount) OVER () AS whole_sum
FROM sales;
SELECT * FROM framed ORDER BY id;
DO $$ BEGIN
 IF (SELECT array_agg(default_sum ORDER BY id) FROM framed)
 IS DISTINCT FROM ARRAY[30,30,35]::bigint[] THEN RAISE EXCEPTION 'wrong peer frame'; END IF;
 IF (SELECT array_agg(row_sum ORDER BY id) FROM framed)
 IS DISTINCT FROM ARRAY[10,30,35]::bigint[] THEN RAISE EXCEPTION 'wrong row frame'; END IF;
 IF (SELECT array_agg(default_last ORDER BY id) FROM framed)
 IS DISTINCT FROM ARRAY[10,20,5] THEN RAISE EXCEPTION 'wrong current-frame end'; END IF;
 IF EXISTS (SELECT FROM framed WHERE partition_last IS DISTINCT FROM 5 OR whole_sum IS DISTINCT FROM 35) THEN
 RAISE EXCEPTION 'whole partition values wrong';
 END IF;
END $$;
ROLLBACK;

PostgreSQL 18.6 returned default sums 30, 30, 35: both first-day rows see the complete first-day peer group. The explicit ROWS sum, ordered by day and id, returned 10, 30, 35. The id deliberately defines the order between individual events; it is not included in the date-based default window.

The default last_value column returned 10, 20, 5 because its day-and-id order makes every row its own peer group. The full-partition frame returned 5 everywhere, while the unordered sum returned 35 everywhere. The assertions check these separate results and will fail if a frame is silently changed.

This small fixture establishes answer semantics, not resource costs. Frames that need future rows and large partitions can require substantial buffering or sorting. Inspect realistic execution plans and data distributions separately, preserving the tested tie and boundary behavior while tuning performance.

Sources

PostgreSQL window expressions, frame modes, offsets, and defaults, checked 2026-09-10.
PostgreSQL window functions, frame-sensitive functions and partition semantics, checked 2026-09-10.

GROUPING SETS build several aggregate views

Fri, 15 Jan 2027 00:00:00 +0000

GROUPING SETS asks one SQL statement to produce several aggregation levels. A sales report can return totals by region and product, region subtotals, and a grand total together. The result contains different kinds of rows, so identifying each row’s level is as important as calculating its amount correctly.

Declare the report’s levels

GROUPING SETS ((region,product),(region),()) specifies three groupings. The first yields one row for each observed region-product combination, the second one per region, and the empty set aggregates all selected input rows. This is a declaration of result semantics; the optimizer chooses how to execute it. It can share work across levels, but the syntax does not guarantee a particular number of physical scans or eliminate sorting and memory costs.

The shorthand ROLLUP(region,product) expands to these same prefixes, including the empty set. Reversing the columns changes the subtotal dimension. CUBE(region,product) also includes product-only totals, covering all subsets of its dimensions. Six independent dimensions mean 64 grouping sets, not merely 64 output rows: each set can contain many distinct groups, so output size may grow much more than the number of levels suggests.

All levels aggregate the population selected by FROM and WHERE. A filter restricting sales to one month also restricts the grand total to that month. If the report needs different populations at different levels, a single shared input with these sets may not express that requirement. Define the population before comparing totals with another report.

Label levels with GROUPING

Columns omitted from a grouping set appear as SQL NULL in its output rows. Real input values may also be NULL, so inspecting null columns cannot reliably identify a subtotal. GROUPING(region,product) supplies a bitmask that describes which grouping expressions are absent from the current set. The rightmost argument contributes the least significant bit.

For this report, zero means both expressions participate, one means product is omitted, and three means both are omitted. A region subtotal for an actual null region and the grand total can therefore have identical null dimension columns but different masks. Use that mask in the data contract and derive labels such as “all regions” only when the level indicates aggregation.

A grouping expression’s presence is different from its value being nonnull. GROUPING reports the former. Its arguments must match grouping expressions from the query; it is not an ordinary function for testing arbitrary values. Give downstream consumers the numeric level or another unambiguous discriminator instead of replacing every null with a subtotal label.

When it breaks

Adding every returned total together counts the same sales multiple times, once at each included level. Select a single level when computing another summary, or use the already calculated grand total. Averages and distinct counts need even more care: averaging region averages without their weights does not generally reproduce the overall average, and overlapping distinct sets cannot simply be added.

The empty grouping set still emits a row when its input is empty. Count returns zero, while sum returns NULL, so a blank amount is not evidence that the query failed to return a grand-total row. Decide how the report represents an empty population. Also avoid accidentally repeating grouping sets; duplicate level specifications can produce repeated aggregate rows rather than extra independent evidence.

Check it quickly

Save this as grouping.sql and run psql -X -v ON_ERROR_STOP=1 -f grouping.sql in a disposable PostgreSQL 18 database. The fixture deliberately includes a real null region and uses a transaction to remove its temporary objects afterward.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE sales (region text, product text, amount integer);
INSERT INTO sales VALUES ('east','A',10),('east','B',20),(NULL,'A',5);
CREATE TEMP VIEW levels AS
SELECT region,product,sum(amount) AS total,GROUPING(region,product) AS level
FROM sales GROUP BY GROUPING SETS ((region,product),(region),());
SELECT * FROM levels ORDER BY level,region NULLS LAST,product NULLS LAST;
SELECT sum(total) AS incorrect_retotal,
 sum(total) FILTER (WHERE level=0) AS detail_total FROM levels;
CREATE TEMP VIEW empty_level AS
SELECT count(*) AS n,sum(amount) AS total FROM sales WHERE false GROUP BY GROUPING SETS (());
SELECT * FROM empty_level;
DO $$ BEGIN
 IF (SELECT count(*) FROM levels) <> 6 THEN RAISE EXCEPTION 'wrong level count'; END IF;
 IF (SELECT total FROM levels WHERE level=3) IS DISTINCT FROM 35 THEN
 RAISE EXCEPTION 'wrong grand total';
 END IF;
 IF (SELECT total FROM levels WHERE level=1 AND region IS NULL) IS DISTINCT FROM 5 THEN
 RAISE EXCEPTION 'real null region subtotal lost';
 END IF;
 IF (SELECT count(*) FROM levels WHERE region IS NULL AND product IS NULL) <> 2 THEN
 RAISE EXCEPTION 'test did not include ambiguous NULL labels';
 END IF;
 IF (SELECT n FROM empty_level) IS DISTINCT FROM 0::bigint OR
 (SELECT total FROM empty_level) IS NOT NULL THEN
 RAISE EXCEPTION 'wrong empty-input aggregate';
 END IF;
END $$;
ROLLBACK;

PostgreSQL 18.6 returned six rows: three detailed groups, two region subtotals, and one grand total. The east subtotal was 30. The null-region subtotal was 5 with mask one, while the grand total was 35 with mask three. Both latter rows had null region and product columns, proving why null-based labeling is ambiguous.

The incorrect re-total was 105, whereas summing just the detailed groups returned 35. The empty-input query returned one row with count zero and a null sum. Assertions check the level count, grand total, real-null subtotal, and empty-input behavior. Final ORDER BY controls presentation explicitly; GROUPING SETS itself does not promise that subtotals arrive next to their detail rows.

Sources

PostgreSQL table expressions, grouping sets, rollup, cube, and omitted dimensions, checked 2026-09-10.
PostgreSQL aggregate functions, GROUPING bitmasks and empty-input aggregates, checked 2026-09-10.

A foreign key refuses to let a reference dangle

Sat, 02 Jan 2027 00:00:00 +0000

Adding a PostgreSQL foreign key to an existing table is both a schema change and a statement about old data. New writes can be checked immediately while historical rows still contain broken references. Separate the constraint’s enforcement, its validation state, and its checking time when planning a migration or diagnosing an unexpected integrity failure.

Enforcement is not historical validation

An ordinary enforced foreign key requires a matching referenced key for each applicable nonnull reference. The referenced columns need eligible uniqueness, such as a suitable nondeferrable primary or unique constraint or a nonpartial unique index. The child column remains nullable unless separately declared NOT NULL; a foreign key does not make every child relationship mandatory.

ADD CONSTRAINT … NOT VALID skips the initial scan of existing rows. It still checks new or changed references, so it can stop additional orphans from accumulating while old ones are investigated. NOT VALID does not mean that the rule is disabled. The database also does not yet assume that every historical row satisfies it.

VALIDATE CONSTRAINT scans the existing data and marks the constraint validated if the check succeeds. If a historical orphan remains, validation fails and the constraint stays unvalidated. Fix the data using the authoritative business relationship, then validate again. Inventing a parent or deleting a child merely to make the check green can corrupt the business record even when the resulting database is structurally consistent.

Separate the deployment phases

Adding a foreign key takes locks on both the referencing and referenced tables, even with NOT VALID. Avoid describing that option as zero-lock deployment. Its benefit is avoiding a potentially long initial validation scan while the add operation holds its locks. VALIDATE uses a weaker lock that permits ordinary data modifications, but still consumes resources and coordinates with other schema operations.

For a migration intended to shorten the first lock window, commit the short add phase before performing validation as a later operation. Wrapping both phases in one long transaction keeps earlier locks until the transaction ends and can defeat that purpose. The example below deliberately uses one transaction only because its entire test schema must be removed on rollback.

PostgreSQL does not automatically index the referencing columns. The parent’s uniqueness already has an index, but parent updates and deletions may need to find child references. Inspect existing child indexes and add a suitable access path when the workload needs it. A declared foreign key is not a guarantee of join elimination or fast cascades.

When to defer a check

A DEFERRABLE foreign key can postpone checking until transaction end. INITIALLY DEFERRED makes that the starting mode; SET CONSTRAINTS can switch eligible constraints back to immediate checking and retroactively check outstanding work. This supports inserting a child before its parent when both are established by the same transaction.

Deferral does not permit committing unresolved references. It changes the checking boundary, not the final invariant. NO ACTION can participate in deferred checking, whereas the RESTRICT delete action cannot be deferred. Cascading actions also are not postponed merely because the constraint is deferrable. Choose the action and checking mode together instead of treating DEFERRABLE as permission to ignore integrity temporarily without a completion condition.

When it breaks

A migration can appear successful because the add statement succeeded while convalidated remains false. Track successful validation as a separate completion condition. Conversely, a catalog flag alone is not proof of data integrity after privileged operations bypass enforcement. Historical imports and replication procedures must be evaluated under the checks they actually performed, not the constraint name they left behind.

Constraint errors can also reveal a valid policy working as designed. Diagnose the missing referenced key and the transaction’s ordering before weakening the rule. Parent and child operations can wait on concurrent transactions; bypassing the check to avoid waiting trades a coordination issue for possible data corruption.

Check it quickly

Save this as foreign-key-validation.sql and run psql -X -v ON_ERROR_STOP=1 -f foreign-key-validation.sql in a disposable PostgreSQL 18 database. It uses temporary tables and rolls back all changes.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE customers (id integer PRIMARY KEY);
CREATE TEMP TABLE invoices (id integer PRIMARY KEY,customer_id integer);
INSERT INTO customers VALUES (1);
INSERT INTO invoices VALUES (10,1),(11,99);
ALTER TABLE invoices ADD CONSTRAINT invoices_customer_fk
FOREIGN KEY (customer_id) REFERENCES customers NOT VALID;
SELECT convalidated FROM pg_constraint
WHERE conrelid='invoices'::regclass AND conname='invoices_customer_fk';
DO $$ BEGIN
 INSERT INTO invoices VALUES (12,99);
 RAISE EXCEPTION 'new orphan accepted';
EXCEPTION WHEN foreign_key_violation THEN RAISE NOTICE 'NOT VALID still checks new references';
END $$;
DO $$ BEGIN
 ALTER TABLE invoices VALIDATE CONSTRAINT invoices_customer_fk;
 RAISE EXCEPTION 'old orphan was not detected';
EXCEPTION WHEN foreign_key_violation THEN RAISE NOTICE 'validation found the old orphan';
END $$;
INSERT INTO customers VALUES (99);
ALTER TABLE invoices VALIDATE CONSTRAINT invoices_customer_fk;
SELECT convalidated FROM pg_constraint
WHERE conrelid='invoices'::regclass AND conname='invoices_customer_fk';
SELECT indexrelid::regclass AS child_indexes FROM pg_index WHERE indrelid='invoices'::regclass;
CREATE INDEX invoices_customer_lookup ON invoices (customer_id);
CREATE TEMP TABLE deferred_child (
 id integer PRIMARY KEY,
 customer_id integer REFERENCES customers DEFERRABLE INITIALLY DEFERRED
);
INSERT INTO deferred_child VALUES (20,7);
INSERT INTO customers VALUES (7);
SET CONSTRAINTS ALL IMMEDIATE;
DO $$ BEGIN
 IF (SELECT convalidated FROM pg_constraint
 WHERE conrelid='invoices'::regclass AND conname='invoices_customer_fk') IS DISTINCT FROM true THEN
 RAISE EXCEPTION 'constraint not validated';
 END IF;
 IF EXISTS (SELECT FROM invoices i LEFT JOIN customers c ON c.id=i.customer_id
 WHERE i.customer_id IS NOT NULL AND c.id IS NULL) THEN
 RAISE EXCEPTION 'orphan remains';
 END IF;
 IF (SELECT customer_id FROM deferred_child WHERE id=20) IS DISTINCT FROM 7 THEN
 RAISE EXCEPTION 'deferred insertion failed';
 END IF;
END $$;
ROLLBACK;

PostgreSQL 18.6 reported convalidated false after the add. A new reference to missing customer 99 failed, and validation independently detected the existing orphan. Restoring the fixture’s known missing parent allowed validation to report true. The index listing initially showed only invoices_pkey; the reference index was added explicitly.

The deferred example inserted child 20 before customer seven, then supplied that parent and successfully switched to immediate checking. Final assertions verify validation, absence of remaining invoice orphans, and the deferred relationship. These checks demonstrate a complete data repair and validation path, not just successful execution of a constraint declaration.

Sources

PostgreSQL ALTER TABLE, NOT VALID, validation, and lock behavior, checked 2026-09-10.
PostgreSQL constraints, referenced uniqueness, child indexes, and referential actions, checked 2026-09-10.

Window functions give a row its neighbourhood

Sun, 18 Oct 2026 00:00:00 +0000

Window functions let a report retain individual readings while adding information from related rows. A sensor reading can show its predecessor, its change, and its share of the sensor’s total without reducing the sensor to one output row. Correctness depends on defining the population, the partition, and the ordering before interpreting those extra columns.

Define a neighbourhood with meaning

PARTITION BY identifies independent sequences. A lag over readings partitioned by sensor compares values from the same sensor; omitting that partition can compare the first reading of one sensor with the last reading of another. The ORDER BY inside OVER defines which row is previous. Use a stable sequence key, or a timestamp plus a unique tie-breaker when timestamps alone can repeat.

Lag refers to a preceding row in this ordered population, not necessarily to the preceding minute, day, or scheduled observation. Gaps in collection do not create missing rows automatically. If a calculation requires the previous calendar day, define that calendar relationship explicitly or construct the required time series before applying the calculation. A neighbour in a table is not proof of a regular sampling interval.

At the beginning of a partition, lag returns its default, normally SQL NULL. That accurately represents the absence of a predecessor. Replacing it with zero changes the meaning of the first delta and should follow a business rule. PostgreSQL also respects null values in lag and lead; a null-valued preceding reading is not silently skipped in search of an earlier nonnull value.

Totals and shares retain detail

An aggregate used with OVER preserves the input rows while adding a result. sum(value) OVER (PARTITION BY sensor) has no window ordering, so it repeats the whole sensor total. Adding an order without considering the default frame can change that total into a cumulative amount and therefore change every calculated share.

Cast to numeric before division if fractional shares are required. Dividing integers can truncate the result, making small contributions appear as zero. The example uses NULLIF to turn a zero denominator into SQL NULL: an undefined share remains undefined. That policy is explicit; substituting zero, rejecting the dataset, or presenting a separate status would be different business choices.

Not every window calculation uses a moving frame. Aggregates and functions such as last_value depend on the frame, while lag and lead navigate partition rows. A narrow ROWS frame does not, by itself, redefine which preceding partition row lag returns. Choose each function and its window according to the question rather than applying one frame formula to all of them.

When it breaks

WHERE filters the window’s input before the window values are calculated. Selecting only the newest reading in the same query leaves lag with no older row to inspect. To display one reading with its existing history, compute the window in a subquery or view, then filter the result outside. Conversely, filtering first is correct when excluded readings should not participate in the comparison.

Joins can also multiply rows before the window runs. A reading joined to several labels can appear several times, inflating totals and changing neighbouring positions. Verify the input’s intended grain before adding window calculations. The syntax does not repair duplicated events or guarantee a single cheap scan; different orders, large partitions, and sorting can dominate execution cost.

Check it quickly

Save this as row-context.sql and run psql -X -v ON_ERROR_STOP=1 -f row-context.sql in a disposable PostgreSQL 18 database. The script creates temporary readings and a view inside a transaction, compares both filtering orders, and rolls back the fixture.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE readings (id integer, sensor text, value integer);
INSERT INTO readings VALUES (1,'A',10),(2,'A',15),(3,'A',12),(4,'B',0);
CREATE TEMP VIEW contextual AS
SELECT *, lag(value) OVER (PARTITION BY sensor ORDER BY id) AS previous_value,
 sum(value) OVER (PARTITION BY sensor) AS total,
 value::numeric / NULLIF(sum(value) OVER (PARTITION BY sensor),0) AS share
FROM readings;
SELECT id,sensor,value,previous_value,value-previous_value AS delta,total,round(share,3) AS share
FROM contextual ORDER BY id;
SELECT id,previous_value FROM contextual WHERE id=3;
SELECT id,lag(value) OVER (PARTITION BY sensor ORDER BY id) AS previous_value
FROM readings WHERE id=3;
DO $$ BEGIN
 IF (SELECT previous_value FROM contextual WHERE id=3) IS DISTINCT FROM 15 THEN
 RAISE EXCEPTION 'previous row was lost';
 END IF;
 IF (SELECT share FROM contextual WHERE sensor='B') IS NOT NULL THEN
 RAISE EXCEPTION 'zero denominator policy not preserved';
 END IF;
 IF (SELECT count(*) FROM contextual) <> 4 THEN RAISE EXCEPTION 'rows collapsed'; END IF;
 IF (SELECT array_agg(previous_value ORDER BY id) FROM contextual WHERE sensor='A')
 IS DISTINCT FROM ARRAY[NULL,10,15]::integer[] THEN RAISE EXCEPTION 'wrong lag sequence'; END IF;
 IF EXISTS (SELECT FROM contextual WHERE sensor='A' AND total IS DISTINCT FROM 37) THEN
 RAISE EXCEPTION 'wrong partition total';
 END IF;
 IF (SELECT previous_value FROM (
 SELECT lag(value) OVER (PARTITION BY sensor ORDER BY id) AS previous_value
 FROM readings WHERE id=3
 ) AS filtered) IS NOT NULL THEN RAISE EXCEPTION 'WHERE did not narrow the window input'; END IF;
END $$;
ROLLBACK;

PostgreSQL 18.6 retained all four readings. Sensor A’s previous values were NULL, 10, 15, its deltas were NULL, 5, minus 3, and its total was 37 on every row. Numeric shares rounded to three decimals were 0.270, 0.405, and 0.324; rounding individual rows need not produce an exactly one-valued displayed sum.

Filtering the completed view to id 3 preserved its predecessor of 15. Filtering the source to id 3 before lag produced NULL instead. Sensor B’s first predecessor and zero-total share were both NULL for different reasons. Assertions check row preservation, the predecessor sequence, totals, and filtering semantics, so a refactor that changes the window population cannot pass merely because it still returns a row.

Sources

PostgreSQL window tutorial, virtual input tables and post-window filtering, checked 2026-09-10.
PostgreSQL window functions, lag, lead, nulls, and aggregate frames, checked 2026-09-10.