Systems · XB Field Notes

Backpressure works when bounded admission changes the producer’s behavior

Thu, 12 Aug 2027 00:00:00 +0000

Backpressure connects a consumer’s limited capacity to the behavior of its producers. A full channel may make a producer wait, reject new work, or apply a deliberately chosen shedding policy. The important property is that the limit changes what upstream components do. A growing queue or a dashboard showing lag is evidence of imbalance, but neither is automatically a mechanism that controls admission.

When it breaks

An unbounded queue can absorb a temporary burst while hiding sustained overload. If work arrives faster than it finishes, outstanding work and waiting time continue growing. Moving the same backlog into pending threads, retry tasks, or another unbounded buffer does not remove that pressure. The capacity boundary must cover the place where work is actually retained, not merely one convenient queue object.

A bounded queue also does not define a complete overload policy by itself. Blocking forever can consume all producer threads or leave shutdown stuck. Immediate rejection can lose work if callers treat it as success. Retrying every rejection immediately can increase load. Decide whether work should wait within a deadline, be rejected for later retry, or be discarded under an explicit application contract.

Queue capacity and processing completion are separate. Removing an item can free a queue slot while the consumer is still working on that item. A capacity of two waiting items may therefore coexist with additional in-progress work. Likewise, a producer’s successful enqueue says only that the queue accepted the item; it does not establish execution, durable storage, or completion of its downstream side effects.

Check it quickly

Save this as check-backpressure.py and run python3 check-backpressure.py. It uses only Python’s standard library and was executed with Python 3.14.4. Threads coordinate through explicit events and bounded waits, so the example does not depend on a remote broker, database, or an artificially slow external service. All queues and work items exist only inside the process:

from queue import Queue,Full
from threading import Event,Thread
import json

unbounded=Queue()
for item in range(1000):unbounded.put_nowait(item)
assert unbounded.qsize()==1000
channel=Queue(maxsize=2)
channel.put('a');channel.put('b')
try:
 channel.put('deadline-rejected',timeout=.02)
 raise AssertionError('a full bounded queue accepted another item')
except Full:rejected=True
assert channel.qsize()==2
entered,delivered=Event(),Event()
def produce():
 entered.set();channel.put('c',timeout=2);delivered.set()
producer=Thread(target=produce,daemon=True);producer.start();assert entered.wait(1)
assert not delivered.wait(.05) and channel.qsize()==2
first=channel.get(timeout=1);assert first=='a'
assert delivered.wait(1);producer.join(timeout=1);assert not producer.is_alive()
assert channel.qsize()==2
rest=[channel.get(timeout=1),channel.get(timeout=1)];assert rest==['b','c']
assert channel.empty()
completed=Event()
def join():channel.join();completed.set()
waiter=Thread(target=join,daemon=True);waiter.start();assert not completed.wait(.05)
for item in [first,*rest]:channel.task_done()
assert completed.wait(1);waiter.join(timeout=1);assert not waiter.is_alive()
print(json.dumps({'unbounded_backlog':unbounded.qsize(),'bounded_capacity':2,'deadline_rejected':rejected,
 'blocked_producer_resumed_after_get':delivered.is_set(),'delivered_items':[first,*rest],
 'empty_queue_did_not_mean_processing_complete':True,'join_finished_after_task_done':completed.is_set()}))

The first queue has the default unbounded capacity. With no consumer running, a thousand put_nowait calls all succeed and leave a backlog of a thousand items. This is a small behavioral demonstration, not a memory-exhaustion benchmark. It shows that successful admission alone cannot establish downstream progress or that the arrival rate is sustainable.

The second queue has capacity two and starts with items a and b. Another put uses a short timeout and raises Full, leaving the queued items unchanged. The failed item is not secretly enqueued for later execution. This gives the caller a definite admission outcome on which to base retry or rejection handling instead of presenting an unlimited wait as successful delivery.

A producer thread then tries to put c. The test confirms that it remains blocked while both slots are occupied. Removing a opens one slot, the producer completes its put, and b and c remain queued. The program verifies that all three accepted items are retrieved in order. This demonstrates pressure reaching the producer and being released by a concrete capacity change.

After retrieving the remaining items, the queue is empty, but a separate thread waiting in join still has not completed. Only after task_done is called for all three accepted items does join return. Dequeueing freed capacity earlier; task acknowledgment marks the tracked work complete later. The test separates these events so an empty queue cannot be mistaken for completed business processing.

Preserve the admission contract across the pipeline

Use the queue operation’s actual result rather than a check-then-act decision based on qsize or full. Those observations are approximate when other threads can change the queue. The fixture’s events and paused consumers make its assertions controlled observations; a production caller should handle Full or a successful put atomically through the provided operation.

Choose a bound appropriate to retained resources. An item-count limit does not cap memory when items can be arbitrarily large, and a waiting-queue limit does not include every active worker or external request. Include processing concurrency and upstream retry behavior when evaluating overload. A bounded component surrounded by unlimited pending work can still participate in an unbounded system.

Finally, define what completion means. Python task_done updates queue accounting; it is not a durable transaction or proof that an external side effect succeeded. A worker failure, process crash, or retry after an ambiguous response needs the application’s corresponding delivery and idempotency contract. Backpressure controls admission and pace; correctness still depends on handling accepted, rejected, in-progress, and completed work explicitly.

Sources

Python queue documentation, bounded capacity, Full, blocking puts and task completion, checked 2026-09-12.
Google SRE: Handling Overload, bounded queues, admission control and overload behavior, checked 2026-09-12.

The OOM killer makes the final memory call

Mon, 26 Oct 2026 00:00:00 +0000

When memory is exhausted, someone has to give. The kernel picks a victim by a policy you can read and influence, turning a crash into an order you can reason about.

Exhaustion forces a choice

Physical memory runs out, the allocator’s reclaim path has already been walked, and the kernel still cannot satisfy a grant — that is when the out-of-memory killer acts. It is not random; it computes which process to kill using a numeric score and terminates the least valuable candidate under pressure. The score weighs resident memory against a process’s share of the host, so a big, idle service that contributes little can be picked off while a small but essential one is spared. Understanding that the OOM killer runs on a score, not on who arrived first, is the whole basis for predicting its next move.

The score picks the victim

The score is steerable. oom_score reflects how costly the footprint is; oom_score_adj shifts a process up or down the league, and a negative value deliberately protects a critical daemon so it is not the victim. Setting a oom_score_adj of -500 on the process that must survive (your container controller, your agent) and leaving the bulk of the memory hog neutral is how you make the killer’s ’least important’ align with your actual priorities. Do this as part of the container/supervisor config, because by the time memory is exhausted it is far too late to decide the victim. Overcommit decides the grant; this score decides who dies when granted memory runs out.

Steer the policy before the crash

The reliable way to make OOM a non-event is to never let the system reach the threshold: cap the workload (ulimit, per-container limits), watch RSS and swap so you see the trend, and provision headroom you can afford. When you do hit it, treat it as a signal in the alerting, not a mystery condition. The OOM killer deserves to be boring — a last-resort breaker, explicitly configured, with a clear accounting of who it took and why, rather than the plan itself.

When it breaks

The classic surprise is that the most important process dies first. A database holding a large buffer pool carries the biggest resident set, so the score picks it while the swarm of small workers that created the pressure survives. The outage looks backwards, but the Out of memory: Killed process line shows the scoring did exactly what it was configured to do.

Misconfigured adjustments fail in the opposite direction. An oom_score_adj of -1000 on every service leaves the killer no acceptable victim, so it reaches past the protected set and takes the supervisor, the SSH session, or the monitoring agent instead. The sign is another trap: positive means more likely to be killed, so copying +500 from a priority example onto a critical daemon paints a target on it. Adjustments set on an interactive shell instead of the supervised unit vanish with the session.

Container scope adds a third confusion. A cgroup that hits its own memory.max triggers a memcg OOM inside that group while the host still shows free memory, so node dashboards report a shortage that never happened. The mirror image is an uncapped container that grows until a host-wide OOM takes down a neighbour. Finally, a slow leak mimics a capacity problem: the same process name holds more memory before each kill, and adding RAM only stretches the interval between restarts.

Check it quickly

Save the following as oom_drill.py in a scratch directory and run python3 oom_drill.py. It uses only the standard library, reads /proc/[pid]/oom_score, oom_score_adj, and comm read-only, and needs no privileges. Without those proc files it prints SKIP and ranks a small simulated table with the same sorting rule instead.

"""Read-only OOM victim ranking drill: live /proc scan with simulated fallback."""

from pathlib import Path


def read_text(path):
 try:
 return Path(path).read_text(encoding="utf-8").strip()
 except (FileNotFoundError, NotADirectoryError):
 return None
 except OSError as exc:
 return "UNREADABLE: %s" % exc


def live_rows():
 rows = []
 try:
 entries = list(Path("/proc").iterdir())
 except OSError:
 return None
 for entry in entries:
 if not entry.name.isdigit():
 continue
 score_text = read_text(entry / "oom_score")
 adj_text = read_text(entry / "oom_score_adj")
 if score_text is None or adj_text is None:
 continue
 if score_text.startswith("UNREADABLE") or adj_text.startswith("UNREADABLE"):
 continue
 try:
 score = int(score_text.split()[0])
 adj = int(adj_text.split()[0])
 except (ValueError, IndexError):
 continue
 name = read_text(entry / "comm") or "?"
 if name.startswith("UNREADABLE"):
 name = "?"
 short = name.split()[0] if name.split() else "?"
 rows.append((entry.name, short, score, adj))
 return rows


def report(title, rows):
 print(title)
 for pid, name, score, adj in sorted(rows, key=lambda row: (-row[2], row[0]))[:5]:
 print("pid=%s name=%s oom_score=%d oom_score_adj=%d" % (pid, name, score, adj))


def main():
 rows = live_rows()
 if rows is None:
 print("SKIP: /proc unavailable (non-Linux?) -- ranking simulated table instead")
 elif rows:
 report("live top candidates by oom_score:", rows)
 print("PASS: live ranking uses kernel scores; higher oom_score is killed first")
 return
 else:
 print("SKIP: no readable /proc/[pid]/oom_score entries -- ranking simulated table instead")
 simulated = [
 ("4242", "batch-worker", 860, 0),
 ("1001", "postgres", 640, -500),
 ("9001", "cache", 500, 0),
 ("7", "agent", 120, -800),
 ]
 report("simulated ranking (same sort rule):", simulated)
 print("PASS: unprotected worker outranks protected postgres and agent; adj sign matches kernel rule")


if __name__ == "__main__":
 main()

On Linux it prints the five highest live scores with names and adjustments, so you can confirm the ordering before pressure arrives. On macOS or containers with hidden PIDs it prints SKIP and then the simulated ranking, where the large unprotected worker outscores the protected controller. What is under test is the ranking logic: re-run it after changing a unit’s OOMScoreAdjust to confirm the intended victim moved.

Sources

Kernel: OOM killer concepts, when the kernel invokes the OOM killer and what it sacrifices, checked 2026-10-25.
man7: proc_pid_oom_score, how the kernel scores each process for OOM selection, checked 2026-10-25.
man7: proc_pid_oom_score_adj, the -1000 to +1000 adjustment range and its meaning, checked 2026-10-25.

Clock synchronization bounds time disagreement

Wed, 09 Sep 2026 00:00:00 +0000

Clock synchronization reduces disagreement between machines’ civil-time readings. It supports certificate validity checks, timestamp interpretation, and many expiration policies, but it does not establish a perfect global clock. A host can emit well-formatted UTC while measuring the wrong instant. Applications still need to distinguish elapsed durations, absolute deadlines, and causal ordering rather than making them all depend on raw wall-clock comparisons.

Four timestamps estimate two quantities

An NTP exchange records client send time T1, server receive time T2, server transmit time T3, and client receive time T4. The usual offset estimate is ((T2-T1)+(T3-T4))/2; a positive result means the server appears ahead. The round-trip delay estimate is (T4-T1)-(T3-T2), subtracting time spent processing at the server. These are estimates from observations, not direct measurements of every one-way link delay.

If forward and reverse paths have unequal delay, the offset estimate inherits an error. A small round-trip measurement can be useful without proving symmetry. A synchronization daemon uses repeated samples, selection, filtering, and a clock model rather than treating one packet as unquestionable truth. Source reachability, uncertainty, and source trust all matter; a low stratum number alone is not an accuracy certificate.

Timestamp representation also differs from measurement accuracy. A fractional field can encode very fine increments while the underlying clock or network observation remains much less certain. Printing additional decimal places does not improve that evidence.

Clock control is another stage. A daemon can gradually slew corrections, and configured conditions can permit stepping the clock. Do not promise that NTP never produces a wall-clock discontinuity or always reaches sub-millisecond accuracy. The network, oscillator, platform, and configuration determine achievable behavior. Choose monitoring bounds from application needs and measured conditions instead of adopting one universal offset threshold.

When it breaks

A wall clock can move backward or forward, making naive elapsed-time subtraction negative or misleading. Use a suitable monotonic clock for a local duration or timeout. Its reference point is not a portable timestamp for another host or a later boot. Certificate dates and business deadlines still need civil-time semantics, so monotonic time does not replace all timestamp handling.

Malformed, unrelated, unsynchronized, or policy-denied NTP responses must not be accepted as ordinary samples. Matching a response’s origin timestamp to the request helps reject unrelated packets, but it is not cryptographic authentication. Production clients need the protocol’s fuller validation and source-selection behavior. Synchronization also does not turn a timestamp-based lease into a safe distributed lock or guarantee that close events sort in causal order.

Check it quickly

Use a fresh disposable Debian 12 Docker container with Python 3 and chrony installed; the package setup is apt-get update followed by apt-get install -y python3 chrony. Save the code as ntp-exchange.py and run it as root with python3 ntp-exchange.py. The recorded environment used chrony 4.3 and Python 3.11.2. Drop SYS_TIME capability and avoid host networking when creating the container.

from pathlib import Path
from tempfile import TemporaryDirectory
import json
import os
import socket
import struct
import subprocess
import time

assert os.geteuid() == 0 and Path('/.dockerenv').exists(), 'use a disposable root Linux container'
NTP_EPOCH = 2208988800
ERA = 2**32


def stamp(unix_time):
 value = unix_time + NTP_EPOCH
 whole = int(value)
 return struct.pack('!II', whole % ERA, int((value - whole) * ERA))


def seconds(data, reference):
 whole, fraction = struct.unpack('!II', data)
 era = round((reference + NTP_EPOCH - whole) / ERA)
 return whole + era * ERA - NTP_EPOCH + fraction / ERA


def estimate(t1, t2, t3, t4):
 return ((t2 - t1) + (t3 - t4)) / 2, (t4 - t1) - (t3 - t2)


def validate(data, sent):
 if len(data) < 48:
 raise ValueError('short packet')
 if data[0] >> 6 == 3:
 raise ValueError('unsynchronised source')
 if (data[0] & 7) != 4 or ((data[0] >> 3) & 7) != 4:
 raise ValueError('unexpected mode or version')
 if not 1 <= data[1] <= 15:
 raise ValueError('unusable stratum')
 if data[24:32] != sent:
 raise ValueError('origin timestamp mismatch')


with TemporaryDirectory(prefix='ntp-exchange-') as directory:
 root = Path(directory)
 with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as reservation:
 reservation.bind(('127.0.0.1', 0))
 port = reservation.getsockname()[1]
 config = root / 'chrony.conf'
 config.write_text(f'pidfile {root}/pid\ncmdport 0\nbindcmdaddress {root}/command.sock\n'
 f'port {port}\nbindaddress 127.0.0.1\nlocal stratum 8\nallow 127.0.0.1\n')
 process = subprocess.Popen(['chronyd', '-x', '-n', '-u', 'root', '-f', str(config)],
 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 try:
 deadline = time.monotonic() + 5
 while not (root/'command.sock').exists():
 assert process.poll() is None
 if time.monotonic() > deadline:
 raise AssertionError('chronyd did not start')
 time.sleep(0.05)
 with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as client:
 client.settimeout(5)
 client.connect(('127.0.0.1', port))
 request = bytearray(48)
 request[0] = 0x23
 t1, start = time.time(), time.monotonic()
 sent = stamp(t1)
 request[40:48] = sent
 client.send(request)
 data = client.recv(512)
 t4, finish = time.time(), time.monotonic()
 validate(data, sent)
 t2, t3 = seconds(data[32:40], t1), seconds(data[40:48], t1)
 offset, delay = estimate(t1, t2, t3, t4)
 assert data[1] == 8 and finish >= start
 bad_origin = bytearray(data); bad_origin[24:32] = bytes(8)
 alarm = bytearray(data); alarm[0] |= 0xC0
 denied = bytearray(data); denied[1] = 0
 rejected = []
 for candidate in (data[:40], bad_origin, alarm, denied):
 try:
 validate(candidate, sent)
 except ValueError as error:
 rejected.append(str(error))
 else:
 raise AssertionError('invalid packet accepted')
 symmetric = estimate(100, 105.020, 105.022, 100.042)
 asymmetric = estimate(100, 105.010, 105.012, 100.042)
 assert abs(symmetric[0] - 5) < 1e-9 and abs(symmetric[1] - 0.040) < 1e-9
 assert abs(asymmetric[0] - 4.990) < 1e-9 and abs(asymmetric[1] - 0.040) < 1e-9
 print(json.dumps({'actual_loopback_sample': {'stratum': data[1],
 'offset_seconds': offset, 'delay_seconds': delay,
 'monotonic_elapsed_seconds': finish-start},
 'rejected_packets': rejected,
 'synthetic_symmetric_estimate': list(symmetric),
 'synthetic_asymmetric_estimate': list(asymmetric),
 'known_synthetic_offset': 5,
 'system_clock_adjustment_disabled': True,
 'external_UTC_accuracy_tested': False}))
 finally:
 process.terminate()
 process.communicate(timeout=5)

The fixture starts chronyd with -x, which disables system-clock adjustment, and a loopback-only source using local stratum 8. It sends a real version-four client packet and checks the returned mode, stratum, origin timestamp, and alarm state. Expect a stratum-eight sample with measured offset and delay values that vary between runs. Four altered or truncated packets are rejected. Both sockets and the daemon are closed afterward.

Two separate synthetic calculations hold the true server lead at five seconds. With twenty milliseconds in each direction, the estimate is five seconds and the round-trip network delay forty milliseconds. With ten milliseconds outbound and thirty inbound, the same forty-millisecond delay accompanies an estimated lead of 4.990 seconds. The ten-millisecond error demonstrates the symmetry assumption; those values are not measurements of the loopback path.

All assertions passed. The local source shares the test host’s clock and has no external UTC reference. This experiment neither disciplines that clock nor proves external accuracy, NTS authentication, or full NTP client conformance. Keep the distinction between a valid observed exchange, a bounded estimate, and trustworthy operational time visible in diagnostics.

Sources

RFC 5905: Network Time Protocol Version 4, timestamp exchange, offset estimation and delay, checked 2026-09-13.
chronyd 4.3 manual, clock control and the no-adjustment option, checked 2026-09-13.
Python: time, wall-clock and monotonic-clock behavior, checked 2026-09-13.

Systems · XB Field Notes

Backpressure works when bounded admission changes the producer’s behavior

When it breaks

Check it quickly

Preserve the admission contract across the pipeline

Sources

The OOM killer makes the final memory call

Exhaustion forces a choice

The score picks the victim

Steer the policy before the crash

When it breaks

Check it quickly

Further reading

Sources

Clock synchronization bounds time disagreement

Four timestamps estimate two quantities

When it breaks

Check it quickly

Sources