Queues · XB Field Notes

Backpressure works when bounded admission changes the producer’s behavior

Thu, 12 Aug 2027 00:00:00 +0000

Backpressure connects a consumer’s limited capacity to the behavior of its producers. A full channel may make a producer wait, reject new work, or apply a deliberately chosen shedding policy. The important property is that the limit changes what upstream components do. A growing queue or a dashboard showing lag is evidence of imbalance, but neither is automatically a mechanism that controls admission.

When it breaks

An unbounded queue can absorb a temporary burst while hiding sustained overload. If work arrives faster than it finishes, outstanding work and waiting time continue growing. Moving the same backlog into pending threads, retry tasks, or another unbounded buffer does not remove that pressure. The capacity boundary must cover the place where work is actually retained, not merely one convenient queue object.

A bounded queue also does not define a complete overload policy by itself. Blocking forever can consume all producer threads or leave shutdown stuck. Immediate rejection can lose work if callers treat it as success. Retrying every rejection immediately can increase load. Decide whether work should wait within a deadline, be rejected for later retry, or be discarded under an explicit application contract.

Queue capacity and processing completion are separate. Removing an item can free a queue slot while the consumer is still working on that item. A capacity of two waiting items may therefore coexist with additional in-progress work. Likewise, a producer’s successful enqueue says only that the queue accepted the item; it does not establish execution, durable storage, or completion of its downstream side effects.

Check it quickly

Save this as check-backpressure.py and run python3 check-backpressure.py. It uses only Python’s standard library and was executed with Python 3.14.4. Threads coordinate through explicit events and bounded waits, so the example does not depend on a remote broker, database, or an artificially slow external service. All queues and work items exist only inside the process:

from queue import Queue,Full
from threading import Event,Thread
import json

unbounded=Queue()
for item in range(1000):unbounded.put_nowait(item)
assert unbounded.qsize()==1000
channel=Queue(maxsize=2)
channel.put('a');channel.put('b')
try:
 channel.put('deadline-rejected',timeout=.02)
 raise AssertionError('a full bounded queue accepted another item')
except Full:rejected=True
assert channel.qsize()==2
entered,delivered=Event(),Event()
def produce():
 entered.set();channel.put('c',timeout=2);delivered.set()
producer=Thread(target=produce,daemon=True);producer.start();assert entered.wait(1)
assert not delivered.wait(.05) and channel.qsize()==2
first=channel.get(timeout=1);assert first=='a'
assert delivered.wait(1);producer.join(timeout=1);assert not producer.is_alive()
assert channel.qsize()==2
rest=[channel.get(timeout=1),channel.get(timeout=1)];assert rest==['b','c']
assert channel.empty()
completed=Event()
def join():channel.join();completed.set()
waiter=Thread(target=join,daemon=True);waiter.start();assert not completed.wait(.05)
for item in [first,*rest]:channel.task_done()
assert completed.wait(1);waiter.join(timeout=1);assert not waiter.is_alive()
print(json.dumps({'unbounded_backlog':unbounded.qsize(),'bounded_capacity':2,'deadline_rejected':rejected,
 'blocked_producer_resumed_after_get':delivered.is_set(),'delivered_items':[first,*rest],
 'empty_queue_did_not_mean_processing_complete':True,'join_finished_after_task_done':completed.is_set()}))

The first queue has the default unbounded capacity. With no consumer running, a thousand put_nowait calls all succeed and leave a backlog of a thousand items. This is a small behavioral demonstration, not a memory-exhaustion benchmark. It shows that successful admission alone cannot establish downstream progress or that the arrival rate is sustainable.

The second queue has capacity two and starts with items a and b. Another put uses a short timeout and raises Full, leaving the queued items unchanged. The failed item is not secretly enqueued for later execution. This gives the caller a definite admission outcome on which to base retry or rejection handling instead of presenting an unlimited wait as successful delivery.

A producer thread then tries to put c. The test confirms that it remains blocked while both slots are occupied. Removing a opens one slot, the producer completes its put, and b and c remain queued. The program verifies that all three accepted items are retrieved in order. This demonstrates pressure reaching the producer and being released by a concrete capacity change.

After retrieving the remaining items, the queue is empty, but a separate thread waiting in join still has not completed. Only after task_done is called for all three accepted items does join return. Dequeueing freed capacity earlier; task acknowledgment marks the tracked work complete later. The test separates these events so an empty queue cannot be mistaken for completed business processing.

Preserve the admission contract across the pipeline

Use the queue operation’s actual result rather than a check-then-act decision based on qsize or full. Those observations are approximate when other threads can change the queue. The fixture’s events and paused consumers make its assertions controlled observations; a production caller should handle Full or a successful put atomically through the provided operation.

Choose a bound appropriate to retained resources. An item-count limit does not cap memory when items can be arbitrarily large, and a waiting-queue limit does not include every active worker or external request. Include processing concurrency and upstream retry behavior when evaluating overload. A bounded component surrounded by unlimited pending work can still participate in an unbounded system.

Finally, define what completion means. Python task_done updates queue accounting; it is not a durable transaction or proof that an external side effect succeeded. A worker failure, process crash, or retry after an ambiguous response needs the application’s corresponding delivery and idempotency contract. Backpressure controls admission and pace; correctness still depends on handling accepted, rejected, in-progress, and completed work explicitly.

Sources

Python queue documentation, bounded capacity, Full, blocking puts and task completion, checked 2026-09-12.
Google SRE: Handling Overload, bounded queues, admission control and overload behavior, checked 2026-09-12.

Queues choose delivery guarantees

Fri, 05 Feb 2027 00:00:00 +0000

A queue is a promise about delivery between a producer and a consumer. That promise can be “at least once” (never lose) or “at most once” (never duplicate), but never both for free. Choosing the guarantee is choosing which failure you can tolerate.

The trade between losing and duplicating

A queue is a promise about what happens to a message, and the promise you choose decides the failure modes you live with. At-most-once tries hard to send a message exactly once and, on failure, prefers to drop it rather than risk a duplicate; at-least-once guarantees a message is not lost but hands out duplicates when retries happen; exactly-once is the ideal that requires an idempotent consumer or a transactional handshake to actually hold. Most people do not set this directly; they inherit it from the broker, the consumer, and how they acknowledge.

The real spot the guarantee shows is the acknowledgment. A consumer that acknowledges after writing makes a small window where a crash can redeliver a message, which is how an “at-least-once” queue is born. A consumer that drops a message silently on processing failure turns the queue into “at most once”. Understanding where the acknowledgment sits is the difference between the guarantee your queue documents and the one it actually delivers.

Idempotency makes at-least-once safe

At-least-once reads scary in theory and is common in practice, because its risk is manageable when the consumer is idempotent. The reason a duplicate is harmless is a consumer that can apply the same work twice and still produce the same result: a payment that is keyed by a stable id, an insert that is naturally deduplicated by a unique constraint. Building idempotency into the consumer turns the duplicate that at-least-once can cause into a non-event.

Idempotency is not a magic property; it is an explicit design choice at the function level. The classic trick is to store the message id alongside a unique key, so a retry hits a constraint instead of re-processing. When a job has expensive side effects, wrap the whole unit in a transaction that includes the dedup marker, and ensure the retry, not the process, is what decides the final outcome. Judged across a real failure, the cost of an occasional duplicate is cheaper than the cost of a lost message.

Pick by what the job allows

The right pick depends on what the job tolerates, not on what sounds most reliable. A command that charges a user wants the message to never be lost, so at-least-once plus idempotency fits; a telemetry counter where a rare drop is acceptable may prefer at-most-once to avoid unbounded duplicates and storage growth. Exactly-once is not usually real without extra machinery, so treat it as a behavior you assemble from at-least-once plus an idempotent consumer plus a durable dedup store.

The checklist to settle the choice: does a lost message cost more than a duplicated one, can the consumer be made idempotent cheaply, and what does the broker actually preserve on restart? Write the chosen guarantee into the code docs, and add a test that replays the safe duplicate to prove the consumer tolerates it. The guarantee is a contract with the system, not a label you can assume.

When it breaks

A double charge bills one order twice while the broker reports no error. The cause is a crash between applying the message and sending the acknowledgment, so the broker redelivers work already finished. Join the ledger against message ids: identical ids on two charges mean redelivery, distinct ids mean the producer sent twice.

A second pattern is a stuck consumer where throughput falls to zero and one offset retries forever. That is a poison message that always raises, paired with unbounded retry and no dead-letter queue. It differs from a redelivery storm, where throughput stays high but the same batch reappears because acknowledgments are lost or the visibility timeout is shorter than processing. One message with a huge redelivery count means poison; many messages with climbing counts mean lost acknowledgments.

The slower failure is dedup expiry: duplicates take effect again after months of quiet because the window aged out or the table was truncated.

Check it quickly

Save the following as delivery_guarantee_probe.py and run python3 delivery_guarantee_probe.py.

def main():
 inbox = [("pay-1", 100), ("pay-2", 50)]
 deliveries = inbox + [inbox[0]] # lost ack redelivers pay-1 once
 naive = list(deliveries) # non-idempotent: bills every delivery
 seen = set()
 ledger = []
 for msg_id, amount in deliveries:
 if msg_id not in seen:
 seen.add(msg_id)
 ledger.append((msg_id, amount))
 assert len(naive) == 3, naive # duplicate was billed twice
 assert sum(a for _, a in ledger) == 150, ledger # idempotent bills pay-1 once
 assert seen == {"pay-1", "pay-2"}, seen # dedup holds one entry per id
 print("PASS: naive=3 idempotent=2 dedup=2")


if __name__ == "__main__":
 main()

The asserts prove the naive consumer bills the redelivery twice while the idempotent consumer bills it once and the dedup set holds two ids. Expect PASS: naive=3 idempotent=2 dedup=2.

Sources

Cloud Pub/Sub: delivery semantics, at-least-once delivery described, checked 2027-02-03.
Amazon SQS: at-least-once, standard queue behavior, checked 2027-02-03.
Redis Streams, consumer groups and acknowledgment, checked 2027-02-03.

Queues · XB Field Notes

Backpressure works when bounded admission changes the producer’s behavior

When it breaks

Check it quickly

Preserve the admission contract across the pipeline

Sources

Queues choose delivery guarantees

The trade between losing and duplicating

Idempotency makes at-least-once safe

Pick by what the job allows

When it breaks

Check it quickly

Further reading

Sources