Grpc · XB Field Notes

Protobuf lets old and new services talk

Sat, 26 Feb 2028 00:00:00 +0000

A gRPC service and its clients ship on their own schedules, so the schema has to change while both old and new binaries are running. Protobuf is built for exactly that: the compatibility contract lives in field numbers and wire types rather than in names, and a reader that does not recognise a field skips it instead of failing.

The tension is that this safety is narrow and unenforced. A small set of edits is compatible; the rest corrupt data silently, and the compiler only refuses the edits you told it to refuse. Knowing which is which is the whole skill.

The wire speaks numbers, not names

Every field on the wire is a tag — the field number shifted left three bits and OR’d with a wire type — followed by the payload. For a message with id=7, name=“alpha”, code=1234 the bytes are 08 07 12 05 61 6c 70 68 61 18 d2 09, where 0x18 marks field 3 as a varint and d2 09 is 1234. protoc --encode produces byte-identical output, so this is the format rather than a description of it.

Because the name never appears, renaming is free: the same bytes decoded against a schema that calls field 1 event_id and field 3 status produce the same values under the new labels. Adding a field with a fresh number is safe too — an old reader surfaces the unknown field number and preserves its bytes, and re-encoding reproduces the newer message exactly. That preservation is what lets an old proxy forward a message it does not understand.

Never reuse a field number

The corrupting edit is deleting a field and reusing its number. There are two variants and both are bad. If the wire type changes — an int32 varint becomes a string — the new reader sees the old value stranded as an unrecognised field number while the declared string stays empty: silent loss, with nothing logged. If the wire type stays compatible it is worse, because the value decodes cleanly into the wrong meaning: the same bytes read as int32 give -1 and as uint32 give 4294967295, and 1 read as a bool is true.

The tool for this is reserved. Declaring reserved 3; reserved "code"; makes the compiler reject any field using number 3 or the name code, and suggest the next free number. Without that declaration the same schema compiles without complaint, which is how a corrupting change reaches production.

Presence is a separate decision

proto3 fields have implicit presence: an absent field and a field set to zero encode to identical bytes, so a reader cannot distinguish “the client did not send it” from “the client sent zero”. Adding optional changes that — a zero now emits its tag — and the generated accessor reads that bit. Use it wherever zero is a meaningful value rather than a synonym for missing.

The related failure is assuming a field is required. There is no required on the wire, and a new field that old producers never send will simply be absent, so treating absence as an error breaks every client that has not been upgraded yet. Reserve numbers when you delete, add optional fields rather than changing types, and let absence mean what it says.

When it breaks

The classic failure is a reused tag number silently mis-decoding. A producer writes the new schema, an older consumer reads the same bytes into a field that now means something else, and nothing is logged because nothing failed: the values are simply wrong. It presents as one service computing nonsense from correct input while every other service agrees, and the way to diagnose it is to dump the wire bytes and walk the tags by hand.

The second failure is a required-field-style assumption breaking old clients. A consumer that treats a missing field as a protocol error rejects every producer that predates it, and a consumer that treats a missing field as zero takes the wrong branch. Both are the same mistake — reading semantics into absence — and both are avoided by reserving retired numbers and adding optional fields.

Check it quickly

Look at the numbers on the wire, then make the compiler enforce the retirement:

protoc --decode_raw < captured.bin
protoc -I. --descriptor_set_out=/dev/null api.proto
grep -rn 'reserved' proto/

--decode_raw prints field numbers and values with no names at all, which is what a peer actually receives. The compile must succeed; if it fails with a reserved-number error, someone tried to reuse a retired tag. The grep must show a reserved statement for every number you have removed. A wrong result is a number you thought was retired still carrying a value, or a schema that compiles only because the number was never reserved.

Sources

Protobuf: encoding, tag layout, varints and the wire types behind int32, uint32 and bool, checked 2026-09-16.
Protobuf: proto3 language guide, field numbers, reserved, optional and presence semantics, checked 2026-09-16.
Protobuf: dos and don’ts, the rules for retiring and reusing field numbers, checked 2026-09-16.

gRPC multiplexes many streams over one HTTP/2 connection

Sat, 16 Jan 2027 00:00:00 +0000

Multiplexing shares the transport and keeps ordering within each stream, which is why long-lived pushes fit this model better than request-reply websockets.

One connection, many streams

HTTP/2 multiplexes many logical streams over a single TCP connection, and gRPC makes each call a stream, so a chatty microservice stops reopening sockets per request. That one shared connection means connection setup, TLS handshake, and idle bookkeeping happen once instead of per call. Streams are independent in ordering and cancellation, so a slow, long-lived stream does not block a short one on the same pipe. The practical effect is a low army of connection state, with per-call overhead shrinking to a frame header.

Streaming beats HTTP/1.1 fan-out

Because gRPC is streaming in both directions, a server can begin sending a partial result before the client finishes sending the request, and a client can keep pushing frames over the same call. That fits search, large-file streaming, log shipping, and interactive feeds where a single reply would force the whole payload to buffer first. It also makes fan-out cheaper than HTTP/1.1, where a burst of calls would each need its own connection and its own handshake. Everything that needs a back-and-forth loop maps naturally onto one long-lived stream instead of a series of separate requests.

The elegance of sharing one connection is real, but it is still bounded by the transport. A retry requires a new stream, not a replay on the same one, and backpressure still matters, since a slow consumer can stall the shared connection for everyone else. Deadlines therefore cut across the whole pipe, so one hair-loss deadline on a stream can surface as latency for every stream on that connection. The practical habit is to give each call a real deadline and a way to cancel, rather than letting the shared pipe silently donate time to someone else’s stuck call.

When it breaks

A stalled flow-control window looks like a healthy connection with zero throughput. HTTP/2 keeps a stream-level and a connection-level window, and only the receiver advances them with WINDOW_UPDATE. A slow consumer first parks its own stream; once the connection window fills, every stream behind it stalls too. Separate the two signals in dashboards, and give each RPC a bounded buffer and a consumer timeout.

Oversize metadata fails fast and retries cannot help. gRPC sends initial metadata as HTTP/2 headers with caps near eight kilobytes. An auth token plus tracing baggage plus tenant tags can cross the limit, and the call dies with RESOURCE_EXHAUSTED before business logic runs. Keep metadata small, move correlation blobs out of band, and remember binary metadata pays base64 inflation.

Long-lived connections defeat naive balancing. Clients that each hold one multiplexed connection pin all streams to the backend chosen at dial time, leaving new pods idle while old ones saturate. Fix it with client-side balancing or proxyless xDS, periodic MaxConnAge reconnects, and a cap on concurrent streams per connection.

Lost deadline propagation turns one slow leaf into whole-tree latency. gRPC carries the budget as grpc-timeout, but a gateway that fans out with a fresh timeout or a background context silently drops it. Derive downstream calls from the incoming context, keep each attempt shorter than the end-to-end deadline, and cancel siblings after the first failure.

Check it quickly

Save the following as mux_sim.py and run python3 mux_sim.py. It uses only the standard library to model one connection carrying three logical streams as interleaved frames. Stream 2 sends one frame and then stalls to simulate an exhausted window, while the scheduler must finish streams 1 and 3. Assertions check per-stream order and wire interleaving.

from collections import deque

# One shared connection, three logical streams. Each stream is an
# ordered queue of application messages; the wire is a single log of
# interleaved (stream_id, payload) frames.
streams = {
 1: deque(["a1", "a2", "a3"]),
 2: deque(["b1", "b2", "b3"]),
 3: deque(["c1", "c2"]),
}
stalled = set() # streams whose flow-control window is exhausted
wire = [] # shared transport log
delivered = {sid: [] for sid in streams}


def pump():
 """One round-robin pass: each ready stream emits at most one frame."""
 progressed = False
 for sid in sorted(streams):
 if sid in stalled:
 continue
 if streams[sid]:
 payload = streams[sid].popleft()
 wire.append((sid, payload))
 delivered[sid].append(payload)
 progressed = True
 return progressed


def drain():
 while pump():
 pass


# Round one: every stream places its opening frame on the wire.
assert pump() is True
assert wire == [(1, "a1"), (2, "b1"), (3, "c1")], wire

# Stream 2 exhausts its window and stops sending WINDOW_UPDATE.
stalled.add(2)
drain()

# Siblings must complete while stream 2 is stuck on one frame.
assert delivered[1] == ["a1", "a2", "a3"], delivered[1]
assert delivered[3] == ["c1", "c2"], delivered[3]
assert delivered[2] == ["b1"], delivered[2]
assert len(wire) == 6, wire
assert list(streams[2]) == ["b2", "b3"]

# The window reopens: stream 2 resumes on the same connection.
stalled.discard(2)
drain()

assert delivered[2] == ["b1", "b2", "b3"], delivered[2]
for sid, items in delivered.items():
 assert items == sorted(items), (sid, items) # per-stream order preserved
assert len(wire) == 8, wire
assert all(not q for q in streams.values())

print("wire:", wire)
print("PASS: 3 streams, 8 frames, one shared wire; stall of stream 2 did not block 1 and 3")

The script pumps one round so each stream places its opening frame, then stalls stream 2. The next drain must complete streams 1 and 3 while stream 2 stays at one frame, proving one blocked stream does not block its siblings. Reopening the window drains the rest, and the final check confirms in-order arrival on one log. This models scheduling only, without HPACK, TLS, or real flow-control bytes, so confirm window and deadline behavior on a gRPC stack before tuning production.

Sources

gRPC core concepts, defines streams over HTTP/2, checked 2026-06-01.
RFC 9113: HTTP/2, streams, multiplexing, and flow control, checked 2026-09-09.

Grpc · XB Field Notes

Protobuf lets old and new services talk

The wire speaks numbers, not names

Never reuse a field number

Presence is a separate decision

When it breaks

Check it quickly

Sources

gRPC multiplexes many streams over one HTTP/2 connection

One connection, many streams

Streaming beats HTTP/1.1 fan-out

Deadlines share the same pipe

When it breaks

Check it quickly

Further reading

Sources