Schema · XB Field Notes

Protobuf lets old and new services talk

Sat, 26 Feb 2028 00:00:00 +0000

A gRPC service and its clients ship on their own schedules, so the schema has to change while both old and new binaries are running. Protobuf is built for exactly that: the compatibility contract lives in field numbers and wire types rather than in names, and a reader that does not recognise a field skips it instead of failing.

The tension is that this safety is narrow and unenforced. A small set of edits is compatible; the rest corrupt data silently, and the compiler only refuses the edits you told it to refuse. Knowing which is which is the whole skill.

The wire speaks numbers, not names

Every field on the wire is a tag — the field number shifted left three bits and OR’d with a wire type — followed by the payload. For a message with id=7, name=“alpha”, code=1234 the bytes are 08 07 12 05 61 6c 70 68 61 18 d2 09, where 0x18 marks field 3 as a varint and d2 09 is 1234. protoc --encode produces byte-identical output, so this is the format rather than a description of it.

Because the name never appears, renaming is free: the same bytes decoded against a schema that calls field 1 event_id and field 3 status produce the same values under the new labels. Adding a field with a fresh number is safe too — an old reader surfaces the unknown field number and preserves its bytes, and re-encoding reproduces the newer message exactly. That preservation is what lets an old proxy forward a message it does not understand.

Never reuse a field number

The corrupting edit is deleting a field and reusing its number. There are two variants and both are bad. If the wire type changes — an int32 varint becomes a string — the new reader sees the old value stranded as an unrecognised field number while the declared string stays empty: silent loss, with nothing logged. If the wire type stays compatible it is worse, because the value decodes cleanly into the wrong meaning: the same bytes read as int32 give -1 and as uint32 give 4294967295, and 1 read as a bool is true.

The tool for this is reserved. Declaring reserved 3; reserved "code"; makes the compiler reject any field using number 3 or the name code, and suggest the next free number. Without that declaration the same schema compiles without complaint, which is how a corrupting change reaches production.

Presence is a separate decision

proto3 fields have implicit presence: an absent field and a field set to zero encode to identical bytes, so a reader cannot distinguish “the client did not send it” from “the client sent zero”. Adding optional changes that — a zero now emits its tag — and the generated accessor reads that bit. Use it wherever zero is a meaningful value rather than a synonym for missing.

The related failure is assuming a field is required. There is no required on the wire, and a new field that old producers never send will simply be absent, so treating absence as an error breaks every client that has not been upgraded yet. Reserve numbers when you delete, add optional fields rather than changing types, and let absence mean what it says.

When it breaks

The classic failure is a reused tag number silently mis-decoding. A producer writes the new schema, an older consumer reads the same bytes into a field that now means something else, and nothing is logged because nothing failed: the values are simply wrong. It presents as one service computing nonsense from correct input while every other service agrees, and the way to diagnose it is to dump the wire bytes and walk the tags by hand.

The second failure is a required-field-style assumption breaking old clients. A consumer that treats a missing field as a protocol error rejects every producer that predates it, and a consumer that treats a missing field as zero takes the wrong branch. Both are the same mistake — reading semantics into absence — and both are avoided by reserving retired numbers and adding optional fields.

Check it quickly

Look at the numbers on the wire, then make the compiler enforce the retirement:

protoc --decode_raw < captured.bin
protoc -I. --descriptor_set_out=/dev/null api.proto
grep -rn 'reserved' proto/

--decode_raw prints field numbers and values with no names at all, which is what a peer actually receives. The compile must succeed; if it fails with a reserved-number error, someone tried to reuse a retired tag. The grep must show a reserved statement for every number you have removed. A wrong result is a number you thought was retired still carrying a value, or a schema that compiles only because the number was never reserved.

Sources

Protobuf: encoding, tag layout, varints and the wire types behind int32, uint32 and bool, checked 2026-09-16.
Protobuf: proto3 language guide, field numbers, reserved, optional and presence semantics, checked 2026-09-16.
Protobuf: dos and don’ts, the rules for retiring and reusing field numbers, checked 2026-09-16.

Foreign keys keep references honest

Tue, 25 May 2027 00:00:00 +0000

A foreign key makes a relationship enforceable during ordinary database writes. A task can reference an existing project, and the database can reject an unknown project identifier even when an import or a new application code path forgets to check it. The constraint defines referential integrity; it does not decide whether the selected project is the correct one for the business operation.

State the relationship precisely

A referencing key points to eligible unique columns in the parent table, commonly its primary key. Several child rows may reference the same parent, so a foreign key alone describes neither one-to-one cardinality nor a requirement that every parent have children. Add the separate uniqueness or other rules that the actual relationship requires instead of inferring them from the reference.

Nullability is an independent decision. A single-column foreign key normally permits SQL NULL, representing no reference. Add NOT NULL when every child must have a parent. For composite keys, PostgreSQL’s default MATCH SIMPLE allows a row to avoid matching when any referencing component is null; MATCH FULL instead requires all components to be null or all to match. A partially filled composite reference is therefore not automatically invalid under the default.

Scope belongs in the key too. If identifiers are only unique within a tenant, reference a suitable tenant-and-identifier key rather than assuming a matching identifier in another tenant is acceptable. Even a correctly scoped foreign key is not an authorization check: an existing project can still be the wrong project for the current user or workflow.

Parent deletion needs a policy

NO ACTION is PostgreSQL’s default delete action. With the normal immediate checking mode, deleting a referenced parent fails unless the resulting statement satisfies the constraint. A deferrable NO ACTION constraint can postpone the check; RESTRICT does not permit that deferral of the delete restriction. The two names should not be treated as unconditional synonyms.

CASCADE removes referencing child rows when the parent is deleted. It fits a component that has no independent lifecycle, but its reach can extend through several relationships. SET NULL retains the child while clearing its reference, which requires compatible nullability and other constraints. SET DEFAULT also must leave a valid reference; choosing a default does not waive referential integrity.

ON UPDATE separately describes what happens when referenced key values change. An update cascade copies the new key into referencing rows; it is not a request to overwrite unrelated child attributes. Choose actions according to ownership and retention requirements. Replacing a restrictive rule with cascade merely to silence an error can turn a useful integrity failure into unintended data deletion.

When it breaks

An application-side existence query can race with another transaction deleting the parent. The database constraint coordinates the actual write, so keep it as the authoritative relationship check rather than relying only on the earlier read. Privileged operations that disable enforcement, or migrations that have not validated old rows, are separate states and cannot be assumed to satisfy a fully enforced, validated constraint.

The write cost is also workload-dependent. Parent deletion can require finding many children, taking locks, and executing cascades. PostgreSQL indexes the eligible referenced key but does not automatically create an index on the child reference columns. Review those access paths and the size of the deletion graph. A declared relationship does not guarantee join elimination or make every join inexpensive.

Check it quickly

Save this as foreign-key-actions.sql and run psql -X -v ON_ERROR_STOP=1 -f foreign-key-actions.sql in a disposable PostgreSQL 18 database. The fixture uses temporary tables for tasks, project-owned files, and retained history, with different delete actions. Rollback removes all test objects and changes.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE projects (id integer PRIMARY KEY);
CREATE TEMP TABLE tasks (id integer PRIMARY KEY,project_id integer REFERENCES projects ON DELETE NO ACTION);
CREATE TEMP TABLE files (id integer PRIMARY KEY,project_id integer REFERENCES projects ON DELETE CASCADE);
CREATE TEMP TABLE audit_history (id integer PRIMARY KEY,project_id integer REFERENCES projects ON DELETE SET NULL);
INSERT INTO projects VALUES (1),(2);
INSERT INTO tasks VALUES (10,1),(11,NULL);
INSERT INTO files VALUES (20,1);
INSERT INTO audit_history VALUES (30,1);
DO $$ BEGIN
 INSERT INTO tasks VALUES (12,999);
 RAISE EXCEPTION 'dangling reference accepted';
EXCEPTION WHEN foreign_key_violation THEN RAISE NOTICE 'missing project rejected';
END $$;
DO $$ BEGIN
 DELETE FROM projects WHERE id=1;
 RAISE EXCEPTION 'referenced project deleted';
EXCEPTION WHEN foreign_key_violation THEN RAISE NOTICE 'NO ACTION blocked parent deletion';
END $$;
DO $$ BEGIN
 IF (SELECT count(*) FROM files) <> 1 OR
 (SELECT project_id FROM audit_history WHERE id=30) IS DISTINCT FROM 1 THEN
 RAISE EXCEPTION 'failed deletion left partial side effects';
 END IF;
END $$;
DELETE FROM tasks WHERE id=10;
DELETE FROM projects WHERE id=1;
SELECT * FROM tasks;
SELECT * FROM audit_history;
SELECT count(*) AS remaining_files FROM files;
DO $$ BEGIN
 IF (SELECT array_agg(id ORDER BY id) FROM projects) IS DISTINCT FROM ARRAY[2] THEN
 RAISE EXCEPTION 'wrong surviving project';
 END IF;
 IF (SELECT count(*) FROM files) <> 0 OR
 (SELECT count(*) FROM audit_history WHERE id=30 AND project_id IS NULL) <> 1 OR
 (SELECT count(*) FROM tasks WHERE id=11 AND project_id IS NULL) <> 1 THEN
 RAISE EXCEPTION 'wrong delete action or null handling';
 END IF;
END $$;
ROLLBACK;

PostgreSQL 18.6 rejected the task pointing to project 999, while accepting task 11 with a null reference. The first deletion of project one failed because task 10 still referenced it. That failed statement did not partially remove the other dependent data.

After explicitly removing task 10, the parent deletion succeeded. Its file disappeared through CASCADE, history row 30 remained with a null project, and the unrelated null-reference task remained. Project two survived without any children. Assertions verify these distinct outcomes, showing that “references stay valid” and “children must be deleted” are different contracts. Use the same lifecycle examples when reviewing schema changes with application owners.

Sources

PostgreSQL constraints, referential integrity, nulls, actions, and indexing, checked 2026-09-10.
PostgreSQL CREATE TABLE, foreign-key matching and deferrability syntax, checked 2026-09-10.

Zero-downtime migrations change schema while old code runs

Tue, 03 Nov 2026 00:00:00 +0000

A migration that rewrites a busy table in one committing statement is long and blocking. A sequence of small, backward-compatible changes moves the same schema forward with no maintenance window, because each step is separately deployable and separately reversible.

What is being managed is a lock, and the question is how long it is held — which depends on how much data the statement touches.

Expand, backfill, contract

Three phases: expand adds the new column while the old shape still works, backfill populates it in batches, and contract retires the old shape once no code reads it.

Between them sits the dual-write phase: the application writes both columns for a while, so a rollback loses nothing, and reads switch once the new column is trustworthy.

Old code needs the old shape, so the ordering rules follow: adding a nullable column is safe, dropping a column is not, renaming is a two-step affair, and a NOT NULL constraint comes after the backfill, never before. On a 500,000-row sqlite table, ADD COLUMN note TEXT took 2.0 ms and left the page count at 13,926 — metadata only. Making that column NOT NULL required a rewrite: 115.4 ms and 13,926 pages became 28,532. Same schema change, two costs: the difference is whether the engine must touch every row.

Batching matters more than phase order. Backfilling 500,000 rows in one transaction held the write lock for 671.2 ms; in batches of 20,000 the same work ran 25 transactions with a median hold of 7.3 ms and a p90 of 7.8 ms. Total work is identical — only the longest single lock differs, and that is the number the application experiences.

The lock is the whole problem

PostgreSQL’s ACCESS EXCLUSIVE conflicts with every other lock mode, including the ACCESS SHARE a bare SELECT takes. It is acquired by TRUNCATE, VACUUM FULL, REFRESH MATERIALIZED VIEW without CONCURRENTLY, and many ALTER TABLE forms. Everyone waits.

A migration can also be blocked, and then it becomes the blocker. In rollback-journal mode an ALTER TABLE issued while another session held a read transaction open waited 1,512 ms for it to finish, and four application writes failed in that window; the same migration in WAL mode finished in 0.3 ms with zero failures. The lesson is not about journal modes: a forgotten BEGIN turns a 3 ms schema edit into a 1.5 second outage that queues everyone behind it.

Validate outside, enforce inside

SET NOT NULL is where the split is explicit in the docs: it normally scans the whole table to prove no NULL exists, unless a valid CHECK constraint already proves it or it was added NOT VALID. ADD CONSTRAINT ... NOT VALID records the constraint without checking existing rows, and VALIDATE CONSTRAINT does the scan later under a weaker lock.

That is the right division. The validation scan over 500,000 rows took 16.3 ms and needed no write lock — zero application writes failed while it ran — while the enforcement step, which rewrites the table, took 210.4 ms. The expensive part is the scan, and the scan needs no lock.

When it breaks

The first failure is the backfill in one transaction: it locks and bloats. A single UPDATE over a large table holds the write lock for the whole run, and in PostgreSQL writes a new row version for every row touched, which then needs vacuuming. The measured version is 671 ms of exclusive access instead of a p90 of 7.8 ms, and the blast radius is every write in the application, all of which fail at once if the pool timeout is shorter than the lock.

The second failure is a migration that holds a lock long enough to trip the application’s own timeout. A plain CREATE INDEX takes SHARE — blocks writes, allows reads — for the entire build; the fixture’s three builds held the write lock for 67, 83, and 69 ms on 500,000 rows, and production tables are bigger. CREATE INDEX CONCURRENTLY exists for this, at a price: SHARE UPDATE EXCLUSIVE, two table scans, a wait for any transaction that could modify the table, no transaction block, and a failed build that leaves an invalid index updates still pay for.

Check it quickly

Find the lock holder, then run DDL with a timeout so it fails instead of blocking:

psql -c "SELECT pid, state, now()-xact_start AS age, left(query,50) FROM pg_stat_activity WHERE xact_start IS NOT NULL ORDER BY xact_start;"
psql -c "SELECT pid, mode, granted, left(relation::regclass::text,40) FROM pg_locks WHERE NOT granted;"
psql -c "SELECT indexrelid::regclass, indisvalid FROM pg_index WHERE NOT indisvalid;"
psql -c "SET lock_timeout='3s'; ALTER TABLE events ADD COLUMN note text;"
psql -c "SELECT conname, convalidated FROM pg_constraint WHERE conrelid='events'::regclass;"

The first query names the transaction your migration will queue behind: a large age on a non-null xact_start. The second shows a migration waiting for a lock; readers queueing behind it are the blast radius. The third catches the invalid index a failed CONCURRENTLY build leaves. The fourth is how to run every DDL statement — lock_timeout converts “blocks everything for forty seconds” into “fails fast and you retry”. The last shows convalidated, proving the CHECK constraint before you rely on it for SET NOT NULL.

Sources

PostgreSQL: ALTER TABLE, SET NOT NULL, NOT VALID, and VALIDATE CONSTRAINT, checked 2026-09-16.
PostgreSQL: CREATE INDEX, building indexes concurrently and the invalid-index failure mode, checked 2026-09-16.
PostgreSQL: explicit locking, which commands take ACCESS EXCLUSIVE and SHARE, checked 2026-09-16.
Martin Fowler: parallel change, the expand/contract pattern under another name, checked 2026-09-16.