Protobuf lets schemas evolve without breaking encoders

Fri, 19 Nov 2027 00:00:00 +0000

Backward compatibility comes from never reusing a field number and from defaulting to new, unknown fields being skipped rather than fatal.

Numbers pin the contract

A protobuf field is committed to a wire identity the moment it is given a number, and that number is the contract, not the name. Renaming a field is safe, changing its wire number is a breaking change. So the mapping of tag numbers to meanings must be adjudicated carefully and never reused for a different type, because old binaries holding the old meaning will decode it wrongly the moment the number is reassigned.

The declaration order in .proto is only a convenience for reading; what travels over the wire is a sequence of (field_number, wire_type, value) pairs. Since both sides agree on those numbers, a client can safely ignore fields it does not know. That is what gives protobuf its forward-compatibility: reserving numbers for fields you plan to grow into keeps a stable namespace even before the field ships.

Forward and backward compatibility come from a few rules. Backward compatible means a newer binary can read data written by an older one, so old fields stay, and when a field goes away you mark it reserved. Forward compatible means an older binary tolerates data written by a newer schema, which works because unknown fields are preserved and re-serialized rather than dropped. Adding a new field with a new number is compatible both ways; changing the wire type of an existing field is not.

The real trap is not the field numbers but reusing them. Deleting and repurposing a number breaks every binary that still has the old definition, because the same tag now means a different thing to older readers. The norms are to reserve numbers, never delete-and-reuse, keep wire types stable for a number, and add validation for values at the boundary where data enters the system.

Unknown fields skip, not fail

The behaviours that make evolution painless are hiding in the encoding. Unknown fields are stored and passed through unchanged by default, so an old process can relay a message it does not fully understand without corrupting the parts it does. That means a gateway or a log shipper is no longer a reason to upgrade every consumer whenever the schema changes; a consumer can ignore what it does not need.

Good hygiene on top of the wire format keeps this true at scale: generate code from the same .proto, keep a schema registry, run cross-version tests between the oldest and newest clients, store field numbers in a table for review, and treat a field whose type must change as a new field plus a deprecated marker, so the incompatible rebuild stays visible instead of silently biting the next deploy.

When it breaks

Reusing a deleted field number is the classic silent corruption: an old writer stores an int32 count under tag 5 while a new reader expects a string name there, so the bytes parse as the wrong type or fail the wire-type check. Adding a required field is a different outage: old messages lack it, and strict parsers reject them, which breaks rolling upgrades. Changing int32 to string, or singular to repeated without a plan, splits the fleet into writers and readers that disagree on the same tag. Implicit-presence defaults add a quieter trap: a missing field and an explicit zero look identical, so a consumer cannot tell “not sent” from “set to false”.

Check it quickly

Save the following as proto_compat_check.py and run python3 proto_compat_check.py. It models an old and a new schema as plain dictionaries, so it needs only the standard library and changes nothing on disk. The bad pair reuses a number with a new type and adds a required field; the good pair adds an optional field with a fresh number.

"""Minimal compatibility checker: pure stdlib, no protoc required."""

def check(old, new):
 errors = []
 for num, field in new.items():
 if num in old and field["type"] != old[num]["type"]:
 errors.append(
 f"wire-type change at field {num}: "
 f"{old[num]['type']} -> {field['type']}"
 )
 if num not in old and field.get("label") == "required":
 errors.append(f"required field added at field {num}: {field['name']}")
 return errors


old = {
 1: {"name": "display_name", "type": "string", "label": "optional"},
 2: {"name": "login_count", "type": "int32", "label": "optional"},
}
bad = {
 1: {"name": "display_name", "type": "string", "label": "optional"},
 2: {"name": "login_count", "type": "string", "label": "optional"},
 3: {"name": "email", "type": "string", "label": "required"},
}
good = {
 1: {"name": "display_name", "type": "string", "label": "optional"},
 2: {"name": "login_count", "type": "int32", "label": "optional"},
 4: {"name": "email", "type": "string", "label": "optional"},
}

bad_errors = check(old, bad)
good_errors = check(old, good)
print(f"REJECT bad: {'; '.join(bad_errors)}")
print(f"ACCEPT good: 1 field added, {len(good_errors)} broken rules")
assert len(bad_errors) == 2, bad_errors
assert good_errors == [], good_errors
print("PASS: bad rejected, good accepted")

The run prints REJECT for the bad pair with two rule names, then ACCEPT for the good pair. If you see that output, the checker distinguishes number reuse and required additions from a safe additive change.

REJECT bad: wire-type change at field 2: int32 -> string; required field added at field 3: email
ACCEPT good: 1 field added, 0 broken rules
PASS: bad rejected, good accepted

The script encodes numbering and presence rules, not a full protoc compile; run protoc --version separately if you need code generation.

Sources

Protocol Buffers language guide, field numbers and evolvability, checked 2026-06-12.
Protocol Buffers overview, updating definitions without rewriting code, checked 2026-06-12.