A gRPC service and its clients ship on their own schedules, so the schema has to change while both old and new binaries are running. Protobuf is built for exactly that: the compatibility contract lives in field numbers and wire types rather than in names, and a reader that does not recognise a field skips it instead of failing.

The tension is that this safety is narrow and unenforced. A small set of edits is compatible; the rest corrupt data silently, and the compiler only refuses the edits you told it to refuse. Knowing which is which is the whole skill.

The wire speaks numbers, not names

Every field on the wire is a tag — the field number shifted left three bits and OR’d with a wire type — followed by the payload. For a message with id=7, name=“alpha”, code=1234 the bytes are 08 07 12 05 61 6c 70 68 61 18 d2 09, where 0x18 marks field 3 as a varint and d2 09 is 1234. protoc --encode produces byte-identical output, so this is the format rather than a description of it.

Because the name never appears, renaming is free: the same bytes decoded against a schema that calls field 1 event_id and field 3 status produce the same values under the new labels. Adding a field with a fresh number is safe too — an old reader surfaces the unknown field number and preserves its bytes, and re-encoding reproduces the newer message exactly. That preservation is what lets an old proxy forward a message it does not understand.

Never reuse a field number

The corrupting edit is deleting a field and reusing its number. There are two variants and both are bad. If the wire type changes — an int32 varint becomes a string — the new reader sees the old value stranded as an unrecognised field number while the declared string stays empty: silent loss, with nothing logged. If the wire type stays compatible it is worse, because the value decodes cleanly into the wrong meaning: the same bytes read as int32 give -1 and as uint32 give 4294967295, and 1 read as a bool is true.

The tool for this is reserved. Declaring reserved 3; reserved "code"; makes the compiler reject any field using number 3 or the name code, and suggest the next free number. Without that declaration the same schema compiles without complaint, which is how a corrupting change reaches production.

Presence is a separate decision

proto3 fields have implicit presence: an absent field and a field set to zero encode to identical bytes, so a reader cannot distinguish “the client did not send it” from “the client sent zero”. Adding optional changes that — a zero now emits its tag — and the generated accessor reads that bit. Use it wherever zero is a meaningful value rather than a synonym for missing.

The related failure is assuming a field is required. There is no required on the wire, and a new field that old producers never send will simply be absent, so treating absence as an error breaks every client that has not been upgraded yet. Reserve numbers when you delete, add optional fields rather than changing types, and let absence mean what it says.

When it breaks

The classic failure is a reused tag number silently mis-decoding. A producer writes the new schema, an older consumer reads the same bytes into a field that now means something else, and nothing is logged because nothing failed: the values are simply wrong. It presents as one service computing nonsense from correct input while every other service agrees, and the way to diagnose it is to dump the wire bytes and walk the tags by hand.

The second failure is a required-field-style assumption breaking old clients. A consumer that treats a missing field as a protocol error rejects every producer that predates it, and a consumer that treats a missing field as zero takes the wrong branch. Both are the same mistake — reading semantics into absence — and both are avoided by reserving retired numbers and adding optional fields.

Check it quickly

Look at the numbers on the wire, then make the compiler enforce the retirement:

protoc --decode_raw < captured.bin
protoc -I. --descriptor_set_out=/dev/null api.proto
grep -rn 'reserved' proto/

--decode_raw prints field numbers and values with no names at all, which is what a peer actually receives. The compile must succeed; if it fails with a reserved-number error, someone tried to reuse a retired tag. The grep must show a reserved statement for every number you have removed. A wrong result is a number you thought was retired still carrying a value, or a schema that compiles only because the number was never reserved.

Sources