Protobuf · XB Field Notes

Protobuf lets old and new services talk

Sat, 26 Feb 2028 00:00:00 +0000

A gRPC service and its clients ship on their own schedules, so the schema has to change while both old and new binaries are running. Protobuf is built for exactly that: the compatibility contract lives in field numbers and wire types rather than in names, and a reader that does not recognise a field skips it instead of failing.

The tension is that this safety is narrow and unenforced. A small set of edits is compatible; the rest corrupt data silently, and the compiler only refuses the edits you told it to refuse. Knowing which is which is the whole skill.

The wire speaks numbers, not names

Every field on the wire is a tag — the field number shifted left three bits and OR’d with a wire type — followed by the payload. For a message with id=7, name=“alpha”, code=1234 the bytes are 08 07 12 05 61 6c 70 68 61 18 d2 09, where 0x18 marks field 3 as a varint and d2 09 is 1234. protoc --encode produces byte-identical output, so this is the format rather than a description of it.

Because the name never appears, renaming is free: the same bytes decoded against a schema that calls field 1 event_id and field 3 status produce the same values under the new labels. Adding a field with a fresh number is safe too — an old reader surfaces the unknown field number and preserves its bytes, and re-encoding reproduces the newer message exactly. That preservation is what lets an old proxy forward a message it does not understand.

Never reuse a field number

The corrupting edit is deleting a field and reusing its number. There are two variants and both are bad. If the wire type changes — an int32 varint becomes a string — the new reader sees the old value stranded as an unrecognised field number while the declared string stays empty: silent loss, with nothing logged. If the wire type stays compatible it is worse, because the value decodes cleanly into the wrong meaning: the same bytes read as int32 give -1 and as uint32 give 4294967295, and 1 read as a bool is true.

The tool for this is reserved. Declaring reserved 3; reserved "code"; makes the compiler reject any field using number 3 or the name code, and suggest the next free number. Without that declaration the same schema compiles without complaint, which is how a corrupting change reaches production.

Presence is a separate decision

proto3 fields have implicit presence: an absent field and a field set to zero encode to identical bytes, so a reader cannot distinguish “the client did not send it” from “the client sent zero”. Adding optional changes that — a zero now emits its tag — and the generated accessor reads that bit. Use it wherever zero is a meaningful value rather than a synonym for missing.

The related failure is assuming a field is required. There is no required on the wire, and a new field that old producers never send will simply be absent, so treating absence as an error breaks every client that has not been upgraded yet. Reserve numbers when you delete, add optional fields rather than changing types, and let absence mean what it says.

When it breaks

The classic failure is a reused tag number silently mis-decoding. A producer writes the new schema, an older consumer reads the same bytes into a field that now means something else, and nothing is logged because nothing failed: the values are simply wrong. It presents as one service computing nonsense from correct input while every other service agrees, and the way to diagnose it is to dump the wire bytes and walk the tags by hand.

The second failure is a required-field-style assumption breaking old clients. A consumer that treats a missing field as a protocol error rejects every producer that predates it, and a consumer that treats a missing field as zero takes the wrong branch. Both are the same mistake — reading semantics into absence — and both are avoided by reserving retired numbers and adding optional fields.

Check it quickly

Look at the numbers on the wire, then make the compiler enforce the retirement:

protoc --decode_raw < captured.bin
protoc -I. --descriptor_set_out=/dev/null api.proto
grep -rn 'reserved' proto/

--decode_raw prints field numbers and values with no names at all, which is what a peer actually receives. The compile must succeed; if it fails with a reserved-number error, someone tried to reuse a retired tag. The grep must show a reserved statement for every number you have removed. A wrong result is a number you thought was retired still carrying a value, or a schema that compiles only because the number was never reserved.

Sources

Protobuf: encoding, tag layout, varints and the wire types behind int32, uint32 and bool, checked 2026-09-16.
Protobuf: proto3 language guide, field numbers, reserved, optional and presence semantics, checked 2026-09-16.
Protobuf: dos and don’ts, the rules for retiring and reusing field numbers, checked 2026-09-16.

Protobuf lets schemas evolve without breaking encoders

Fri, 19 Nov 2027 00:00:00 +0000

Backward compatibility comes from never reusing a field number and from defaulting to new, unknown fields being skipped rather than fatal.

Numbers pin the contract

A protobuf field is committed to a wire identity the moment it is given a number, and that number is the contract, not the name. Renaming a field is safe, changing its wire number is a breaking change. So the mapping of tag numbers to meanings must be adjudicated carefully and never reused for a different type, because old binaries holding the old meaning will decode it wrongly the moment the number is reassigned.

The declaration order in .proto is only a convenience for reading; what travels over the wire is a sequence of (field_number, wire_type, value) pairs. Since both sides agree on those numbers, a client can safely ignore fields it does not know. That is what gives protobuf its forward-compatibility: reserving numbers for fields you plan to grow into keeps a stable namespace even before the field ships.

Forward and backward compatibility come from a few rules. Backward compatible means a newer binary can read data written by an older one, so old fields stay, and when a field goes away you mark it reserved. Forward compatible means an older binary tolerates data written by a newer schema, which works because unknown fields are preserved and re-serialized rather than dropped. Adding a new field with a new number is compatible both ways; changing the wire type of an existing field is not.

The real trap is not the field numbers but reusing them. Deleting and repurposing a number breaks every binary that still has the old definition, because the same tag now means a different thing to older readers. The norms are to reserve numbers, never delete-and-reuse, keep wire types stable for a number, and add validation for values at the boundary where data enters the system.

Unknown fields skip, not fail

The behaviours that make evolution painless are hiding in the encoding. Unknown fields are stored and passed through unchanged by default, so an old process can relay a message it does not fully understand without corrupting the parts it does. That means a gateway or a log shipper is no longer a reason to upgrade every consumer whenever the schema changes; a consumer can ignore what it does not need.

Good hygiene on top of the wire format keeps this true at scale: generate code from the same .proto, keep a schema registry, run cross-version tests between the oldest and newest clients, store field numbers in a table for review, and treat a field whose type must change as a new field plus a deprecated marker, so the incompatible rebuild stays visible instead of silently biting the next deploy.

When it breaks

Reusing a deleted field number is the classic silent corruption: an old writer stores an int32 count under tag 5 while a new reader expects a string name there, so the bytes parse as the wrong type or fail the wire-type check. Adding a required field is a different outage: old messages lack it, and strict parsers reject them, which breaks rolling upgrades. Changing int32 to string, or singular to repeated without a plan, splits the fleet into writers and readers that disagree on the same tag. Implicit-presence defaults add a quieter trap: a missing field and an explicit zero look identical, so a consumer cannot tell “not sent” from “set to false”.

Check it quickly

Save the following as proto_compat_check.py and run python3 proto_compat_check.py. It models an old and a new schema as plain dictionaries, so it needs only the standard library and changes nothing on disk. The bad pair reuses a number with a new type and adds a required field; the good pair adds an optional field with a fresh number.

"""Minimal compatibility checker: pure stdlib, no protoc required."""

def check(old, new):
 errors = []
 for num, field in new.items():
 if num in old and field["type"] != old[num]["type"]:
 errors.append(
 f"wire-type change at field {num}: "
 f"{old[num]['type']} -> {field['type']}"
 )
 if num not in old and field.get("label") == "required":
 errors.append(f"required field added at field {num}: {field['name']}")
 return errors


old = {
 1: {"name": "display_name", "type": "string", "label": "optional"},
 2: {"name": "login_count", "type": "int32", "label": "optional"},
}
bad = {
 1: {"name": "display_name", "type": "string", "label": "optional"},
 2: {"name": "login_count", "type": "string", "label": "optional"},
 3: {"name": "email", "type": "string", "label": "required"},
}
good = {
 1: {"name": "display_name", "type": "string", "label": "optional"},
 2: {"name": "login_count", "type": "int32", "label": "optional"},
 4: {"name": "email", "type": "string", "label": "optional"},
}

bad_errors = check(old, bad)
good_errors = check(old, good)
print(f"REJECT bad: {'; '.join(bad_errors)}")
print(f"ACCEPT good: 1 field added, {len(good_errors)} broken rules")
assert len(bad_errors) == 2, bad_errors
assert good_errors == [], good_errors
print("PASS: bad rejected, good accepted")

The run prints REJECT for the bad pair with two rule names, then ACCEPT for the good pair. If you see that output, the checker distinguishes number reuse and required additions from a safe additive change.

REJECT bad: wire-type change at field 2: int32 -> string; required field added at field 3: email
ACCEPT good: 1 field added, 0 broken rules
PASS: bad rejected, good accepted

The script encodes numbering and presence rules, not a full protoc compile; run protoc --version separately if you need code generation.

Sources

Protocol Buffers language guide, field numbers and evolvability, checked 2026-06-12.
Protocol Buffers overview, updating definitions without rewriting code, checked 2026-06-12.

Protobuf · XB Field Notes

Protobuf lets old and new services talk

The wire speaks numbers, not names

Never reuse a field number

Presence is a separate decision

When it breaks

Check it quickly

Sources

Protobuf lets schemas evolve without breaking encoders

Numbers pin the contract

Old and new share the wire

Unknown fields skip, not fail

When it breaks

Check it quickly

Sources