Capacity · XB Field Notes

A token bucket smooths the request rate

Wed, 01 Dec 2027 00:00:00 +0000

A token bucket bounds admitted work while allowing accumulated credit to be spent in a burst. It does not automatically space requests evenly, guarantee completed throughput, or cap the number of operations still running. Those are separate scheduling and capacity questions. The bucket’s two parameters describe a budget, not a promise that the service can perform it.

Refill credit, then charge work

Let capacity B be the maximum stored tokens and rate r be tokens added per second. At each arrival, update credit to min(B, previous_credit + r × elapsed). Admit work only if its cost fits, then subtract that cost. Under the strict rule used here, rejected work does not consume credit.

The bucket can refill lazily when requests arrive; a background timer is unnecessary for this calculation. Idle time cannot accumulate unlimited credit because capacity clips the result. Initial credit is a policy choice: the example starts full, allowing an immediate burst, while an empty initialization would behave differently.

A burst allowance is not a peak rate

For this ideal strict bucket, admitted cost over an interval of duration t is at most B + r × t. As the interval grows, the burst contribution becomes relatively smaller. That bound does not mean every one-second window admits at most r requests, nor does it establish a separate instantaneous peak-rate ceiling.

For capacity three and rate two, three unit-cost requests can pass at the same timestamp. A fourth is denied. Half a second later one token is available. A real scheduler, bounded queue, or additional pacing rule is needed if permitted work must depart at evenly spaced times.

Rate limiting admission also does not bound concurrent execution: slow requests can remain active while more tokens refill. Protect scarce worker or database capacity with an appropriate concurrency limit and timeout policy. Likewise, an accepted request may still fail downstream, so refill rate is not guaranteed successful throughput.

Match the budget to its scope

A token need not mean one request. Weighted costs can represent bytes or estimated work, provided refill rate and capacity use the same units. A request costing more than B can never fit this strict bucket, even after a long idle period. Handle that case explicitly instead of promising an eventual retry time.

Choose the identity and ownership of the budget carefully. Four independent full buckets each holding three tokens can admit twelve requests at once. They are not one global three-token bucket. A shared limiter must make refill, comparison, and deduction an atomic decision; merely sharing a counter without coordinating those steps is insufficient.

Use an elapsed-time clock for local refill calculations. Wall-clock adjustments must not mint credit or move the accounting backward. Python’s monotonic clock is designed for elapsed differences; its reference point is not a portable timestamp to exchange between unrelated machines. Distributed clock and state ownership need their own contract.

When it breaks

A full bucket can release a burst that overwhelms a fragile dependency even when the long-run allowance looks conservative. Unbounded waiting queues turn excess demand into memory growth and latency. Denial and delayed shaping are different outcomes, so document which one callers receive.

Observe admitted and denied cost, queueing, concurrency, and downstream latency together. Retries should respect the limiting policy instead of immediately recreating rejected load. Charging attempted work versus completed work is another explicit choice; refunding failures changes the model and requires careful accounting.

Managed implementations can provide weaker ceilings than the ideal algorithm. Amazon API Gateway documents throttling and quotas as best-effort targets that can be exceeded. Do not present the mathematical bound as a verified guarantee for that service, a billing cap, or any other untested deployment.

Check it quickly

Save the block as bucket_check.py and run python3 bucket_check.py with Python 3.10 or newer. It ran with Python 3.14.4. The standard-library Fraction type and injected nondecreasing timestamps make boundaries exact without sleeping. This is a single-threaded admission model, not a network limiter or distributed store.

from fractions import Fraction


class Bucket:
 def __init__(self, capacity, rate):
 assert capacity > 0 and rate > 0
 self.capacity, self.rate = capacity, rate
 self.tokens, self.updated = Fraction(capacity), Fraction(0)

 def take(self, now, cost=1):
 assert now >= self.updated and cost > 0
 self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
 self.updated = now
 if self.tokens < cost:
 return False
 self.tokens -= cost
 return True


bucket = Bucket(capacity=3, rate=2)
events = ([(Fraction(0), 1)] * 4 + [(Fraction(1, 4), 1), (Fraction(1, 2), 1),
 (Fraction(3, 2), 2)] + [(Fraction(100), 1)] * 4)
results = [(now, cost, bucket.take(now, cost)) for now, cost in events]
assert [allowed for _, _, allowed in results] == [
 True, True, True, False, False, True, True, True, True, True, False,
]
for now, cost, allowed in results:
 print(f"t={str(now):>3} cost={cost} {'allow' if allowed else 'deny'}")
assert bucket.tokens == 0
marks = sorted({now for now, _ in events})
for start in marks:
 for end in marks:
 if end >= start:
 consumed = sum(cost for now, cost, allowed in results if allowed and start <= now <= end)
 assert consumed <= bucket.capacity + bucket.rate * (end - start)
print("every event-bounded interval obeys admitted cost <= B + r * elapsed")

assert not bucket.take(Fraction(200), cost=4)
assert bucket.tokens == 3
assert bucket.take(Fraction(200), cost=3)
print("cost above capacity never fits; rejection does not consume tokens")

workers = [Bucket(capacity=3, rate=2) for _ in range(4)]
local_admitted = sum(worker.take(Fraction(0)) for worker in workers for _ in range(4))
shared = Bucket(capacity=3, rate=2)
shared_admitted = sum(shared.take(Fraction(0)) for _ in range(16))
assert local_admitted == 12 and shared_admitted == 3
print("four independent full buckets admit", local_admitted, "at once; one shared bucket admits", shared_admitted)

The trace allows three simultaneous requests, rejects the fourth and the quarter-second request, then allows one at half a second and cost two at one-and-a-half seconds. After long idle time it permits only another three-token burst. Assertions check every interval bounded by event timestamps against the cost envelope.

An oversized request is denied without spending credit. Finally, four independent buckets admit twelve simultaneous requests while one shared sequential model admits three. No concurrency races, real clock adjustments, queue scheduling, cloud throttling, network partitions, or production performance are tested.

Sources

RFC 3290, Appendix A’s token-bucket envelope and strict conformance calculation, checked 2026-09-08.
Amazon API Gateway throttling, request budgets, bursts, and best-effort limits, checked 2026-09-08.
Python time, monotonic elapsed-time clocks and their reference-point limits, checked 2026-09-08.

Least connections measures occupied connections, not every kind of load

Thu, 11 Nov 2027 00:00:00 +0000

Least-connections balancing chooses a peer using active connection counts, with the implementation’s weighting and tie rules. In nginx, least_conn accounts for server weights and uses weighted round robin among equally suitable peers. This can help when requests occupy connections for different durations. It is still a connection-based measurement, not direct knowledge of CPU, memory pressure, queued jobs, or database contention.

When it breaks

Calling the selected peer the “emptiest server” overstates the signal. One connection may carry an expensive operation while another carries a cheap wait. Background work can continue after an HTTP response closes, and multiplexed protocols can place several operations on one connection. A low connection count therefore does not prove that the backend has spare capacity for the next costly request.

The count is also the balancer’s observed upstream state, not an omniscient census of every backend client. Work arriving through another proxy or a direct internal path may be outside that view. In nginx, worker scope and upstream shared-zone configuration affect which runtime state is shared. Independent balancers do not automatically merge their observations into one global least-connections decision.

Weights make raw-count comparisons insufficient. A peer with weight two can appropriately carry more active connections than one with weight one. Changing a weight expresses a capacity assumption; it does not measure whether that assumption is true. An idle keepalive cache is another different quantity and should not be confused with the active connections used for this selection.

Check it quickly

Save this as check-least-connections.py and run it with Python, Docker, and the pinned images. The recorded setup uses one nginx 1.28.0 worker with no upstream keepalive cache and two HTTP peers on separate ports. The backend provides controlled response gates and active counters, making held work visible without depending on a guessed sleep duration:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None,extra_ports=()):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True)
 target.write_bytes(content if isinstance(content,bytes) else content.encode())
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 published=[arg for value in (8080,*extra_ports) for arg in ('-p','127.0.0.1::'+str(value))]
 docker('run','-d','--name',containers[0],'--network',name,*published,tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 request.port=port;request.network=name
 request.ports={int(key.split('/')[0]):int(value[0]['HostPort']) for key,value in details['NetworkSettings']['Ports'].items() if value}
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'
backend=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
from urllib.parse import urlsplit,parse_qs
import itertools,json,threading
lock=threading.Lock();ids=itertools.count(1);counts={'A':0,'B':0};active={'A':0,'B':0};faults={'A':200,'B':200};gates={}
class Handler(BaseHTTPRequestHandler):
 protocol_version='HTTP/1.1'
 def setup(self):super().setup();self.ident=next(ids)
 def log_message(self,*args):pass
 def do_GET(self):
 parsed=urlsplit(self.path);label=self.server.label;code=200
 if label=='control':
 query=parse_qs(parsed.query)
 with lock:
 if parsed.path=='/fail':faults[query['peer'][0]]=int(query['status'][0])
 if parsed.path=='/release':gates[query['path'][0]].set()
 data={'counts':dict(counts),'active':dict(active),'faults':dict(faults),'held':list(gates)}
 else:
 with lock:
 counts[label]+=1;active[label]+=1;code=faults[label]
 if parsed.path.startswith('/hold/'):gates[parsed.path]=threading.Event()
 if parsed.path.startswith('/hold/'):assert gates[parsed.path].wait(20)
 data={'peer':label,'path':self.path,'connection':self.ident}
 body=json.dumps(data,sort_keys=True).encode();self.send_response(code)
 self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(body)))
 self.end_headers();self.wfile.write(body)
 if label!='control':
 with lock:active[label]-=1
for port,label in [(8081,'A'),(8082,'B'),(8083,'control')]:
 server=ThreadingHTTPServer(('0.0.0.0',port),Handler);server.label=label
 threading.Thread(target=server.serve_forever,daemon=True).start()
threading.Event().wait()
'''
from concurrent.futures import ThreadPoolExecutor
extra='''
upstream least { least_conn; server backend:8081; server backend:8082; }
upstream weighted_least { least_conn; server backend:8081 weight=2; server backend:8082; }
'''
locations='''
location /least/ { proxy_pass http://least/; }
location /weighted/ { proxy_pass http://weighted_least/; }
location /control/ { proxy_pass http://backend:8083/; }
'''
with lab(config(locations,extra),backend=backend) as (_,request):
 def get(path):
 status,_,body=request(path);assert status==200;return json.loads(body)
 def state():return get('/control/state')
 results={}
 with ThreadPoolExecutor(max_workers=3) as pool:
 held=pool.submit(get,'/least/hold/first');wait(lambda:sum(state()['active'].values())==1)
 occupied=next(peer for peer,value in state()['active'].items() if value==1)
 quick=[get('/least/quick')['peer'] for _ in range(6)]
 assert all(peer!=occupied for peer in quick),quick
 get('/control/release?path=/hold/first');assert held.result()['peer']==occupied
 wait(lambda:sum(state()['active'].values())==0)
 results['held_peer']=occupied;results['quick_while_held']=quick
 futures=[]
 for index in range(3):
 futures.append(pool.submit(get,'/weighted/hold/w'+str(index)))
 wait(lambda:sum(state()['active'].values())==index+1)
 snapshot=state()['active'];assert snapshot=={'A':2,'B':1},snapshot
 for index in range(3):get('/control/release?path=/hold/w'+str(index))
 results['weighted_active']=snapshot;results['weighted_completed']=[f.result()['peer'] for f in futures]
 wait(lambda:sum(state()['active'].values())==0)
 print(json.dumps(results,sort_keys=True))

The first request enters the equally weighted least-connections group and waits inside one backend handler. The controller observes exactly one active request and identifies the occupied peer before sending anything else. Which peer wins the initial tie is not the important assertion; the fixture discovers that identity from the backend’s actual state.

Six short requests then complete through the same upstream group. All choose the other peer while the first request remains held. Only afterward does the controller release the original response and verify that it completes on its original peer. This demonstrates selection based on ongoing occupancy, rather than merely counting how many requests each server has answered over its lifetime.

The second group assigns weight two to A and weight one to B. Three requests are started one at a time, with each left unfinished and its arrival observed before the next starts. The active snapshot becomes A=2 and B=1. All three are then released and complete successfully, showing how weights alter the effective comparison instead of always preferring the smallest raw number.

The experiment’s HTTP exchanges keep one active request on each selected upstream connection. The two peer endpoints share a disposable container, so their names do not represent separate hardware capacity. The gates demonstrate occupancy and placement; they do not create a CPU benchmark or validate a particular weight for production machines.

Compare the observed metric with the resource you need to protect

Ask what keeps an upstream connection active in the real path. A slow client may hold it longer when response buffering is disabled, while a buffered proxy can sometimes finish reading upstream before that client finishes downloading. Changing buffering can therefore change the count seen by least connections without changing the application’s intrinsic computational cost.

Measure per-peer latency, errors, queueing, and resource use alongside connection counts. If one peer is slow because a dependency is failing, directing fewer new connections there can reduce pressure but does not repair it. Health eligibility and retry policy remain separate decisions. Least connections is not an active health probe or a gradual recovery ramp for a newly returned peer.

Finally, validate the scope and protocol of the deployed implementation. A method named least connections in another product can have different weighting, persistence, and state-sharing rules. Test a held request, release, and weighted competition through the actual balancer, and use the results to justify the selected metric rather than treating the algorithm’s name as a general capacity guarantee.

Sources

nginx upstream module, least_conn weighting, ties, shared zones, and active connection state, checked 2026-09-12.
nginx HTTP load balancing guide, least-connected selection and its workload rationale, checked 2026-09-12.

A leaked connection drains the pool

Fri, 05 Nov 2027 00:00:00 +0000

A pool leak is an ownership failure: application work finishes, but its borrowed connection remains checked out. The database may still answer other clients normally while this application has no leases left. Treat release as part of the operation’s contract, including exceptions and early returns, rather than a cleanup courtesy after success.

A lease has an owner and an end

A SQLAlchemy Connection represents a borrow from its Engine’s pool. Closing that Connection normally returns the underlying DBAPI connection; it does not necessarily disconnect it. Returning the lease and committing a transaction are separate decisions. A pool can therefore be healthy while business changes were rolled back, or exhausted while its physical database connections remain alive.

Use the smallest ownership scope consistent with the transaction’s atomicity. with engine.begin() commits a successful block, rolls back a failed block, and releases its connection. with engine.connect() scopes the connection, but does not promise to commit writes: an outstanding transaction is rolled back when the connection closes. Do not replace one with the other without checking the intended persistence behavior.

Keep unrelated network waits and user interaction outside a database transaction. That does not justify splitting changes that must commit together. If a caller owns the connection, a helper should not secretly close it; if the helper borrows it, its interface must make any transfer of ownership explicit. Counting opens without identifying the responsible owner misses the real boundary.

When it breaks

A forgotten release reduces usable capacity until another checkout waits or times out. With a finite QueuePool and acquisition timeout, new callers need not hang forever: they receive a pool timeout before executing SQL. Raising the limit can delay recurrence while increasing pressure on the database; it does not repair the missing release.

Exhaustion alone does not prove a leak. Legitimate concurrency, a long transaction, a database lock, or an application deadlock can hold every lease. Flat database CPU is not decisive, because an application can hold a connection while waiting elsewhere. Correlate checked-out counts, checkout wait, lease age, transaction state, and the work that actually remains active.

A useful leak reproduction shows that completed work still owns leases and that returning those exact leases restores capacity. Do not depend on garbage collection to establish timely release. Likewise, Engine.dispose() handles idle connections in its pool, not forced reclamation of connections still checked out. Restarting a process may hide the ownership bug without correcting its next occurrence.

Check it quickly

In an empty macOS or Linux directory, run python3 -m venv .venv and .venv/bin/python -m pip install SQLAlchemy==2.0.52. Save this as pool_probe.py, then run .venv/bin/python pool_probe.py. The checked environment was Python 3.14.4 with SQLite 3.53.0. The temporary database is local and automatically removed.

from pathlib import Path
from tempfile import TemporaryDirectory

import sqlalchemy
from sqlalchemy import create_engine
from sqlalchemy.exc import TimeoutError as PoolTimeout
from sqlalchemy.pool import QueuePool


class WorkFailed(Exception):
 pass


assert sqlalchemy.__version__ == "2.0.52"
with TemporaryDirectory(prefix="pool-leak-") as directory:
 engine = create_engine(
 "sqlite:///" + str(Path(directory) / "probe.db"),
 poolclass=QueuePool, pool_size=2, max_overflow=0, pool_timeout=0.05,
 )
 borrowed = []
 try:
 with engine.begin() as connection:
 connection.exec_driver_sql("CREATE TABLE ledger(id INTEGER PRIMARY KEY)")
 for _ in range(2):
 borrowed.append(engine.connect()) # Deliberately retain both leases.
 assert engine.pool.checkedout() == 2
 try:
 unexpected = engine.connect()
 except PoolTimeout:
 pass
 else:
 unexpected.close()
 raise AssertionError("retained leases must exhaust the two-slot pool")
 assert engine.pool.checkedout() == 2
 print("two retained leases exhaust the pool; the next checkout times out")

 borrowed[0].close()
 with engine.begin() as connection:
 connection.exec_driver_sql("INSERT INTO ledger VALUES (1)")
 assert engine.pool.checkedout() == 1
 print("returning one lease restores capacity without enlarging the pool")

 try:
 with engine.begin() as connection:
 connection.exec_driver_sql("INSERT INTO ledger VALUES (2)")
 raise WorkFailed("injected after the insert")
 except WorkFailed:
 pass
 assert engine.pool.checkedout() == 1
 with engine.connect() as connection:
 assert connection.exec_driver_sql("SELECT id FROM ledger").scalars().all() == [1]
 connection.exec_driver_sql("INSERT INTO ledger VALUES (3)")
 # No commit: this connection context rolls back when it closes.
 with engine.connect() as connection:
 assert connection.exec_driver_sql("SELECT id FROM ledger").scalars().all() == [1]
 print("an exception rolls back row 2 and returns its lease; closing without commit also removes row 3")

 def early_return():
 with engine.connect() as connection:
 return connection.exec_driver_sql("SELECT 1").scalar_one()

 assert early_return() == 1 and engine.pool.checkedout() == 1
 with engine.begin() as connection:
 connection.exec_driver_sql("INSERT INTO ledger VALUES (4)")
 with engine.connect() as connection:
 rows = connection.exec_driver_sql("SELECT id FROM ledger ORDER BY id").scalars().all()
 assert rows == [1, 4] and engine.pool.checkedout() == 1
 borrowed[1].close()
 assert engine.pool.checkedout() == 0 and engine.pool.checkedin() == 2
 print("early return releases its lease; successful transaction commits row 4; final rows", rows)
 finally:
 for connection in borrowed:
 connection.close()
 engine.dispose()

The list deliberately holds strong references to both borrowed connections, making the simulated leak independent of garbage collection. A third checkout raises PoolTimeout. Returning one retained lease lets a transaction insert row 1 without enlarging the pool. One deliberate leak remains, so the expected checked-out count between properly scoped operations is one, not zero.

The injected WorkFailed exception occurs after inserting row 2. Exiting the transaction context rolls that insert back and returns the lease. A separate connection context inserts row 3 without committing; closing it also removes that row. An early return still exits its connection context. Finally, a successful transaction commits row 4, and the ordered result must be exactly [1, 4].

Closing the last intentionally retained lease brings checked-out count to zero while two physical connections remain idle for reuse. The final cleanup returns all retained handles before disposing the Engine. These assertions test local pool accounting and transaction outcomes, not production lock contention, cancellation, or distributed side effects. Returning a connection does not undo an HTTP call already sent elsewhere. Repeat the ownership test around the actual failure path before changing pool capacity.

Sources

SQLAlchemy connection pooling, checkouts, return and finite QueuePool limits, checked 2026-09-09.
SQLAlchemy Engine and Connection, transaction contexts, close and disposal semantics, checked 2026-09-09.
SQLAlchemy pool exhaustion errors, leaks, long transactions, concurrency and deadlocks, checked 2026-09-09.

Resource limits constrain execution; requests guide placement

Mon, 14 Jun 2027 00:00:00 +0000

A Kubernetes resource request and a resource limit belong to different decisions. Requests participate in scheduling and resource sharing; limits configure execution constraints supported by the runtime and operating system. Treating a limit as the scheduler’s reservation obscures both mechanisms. A useful resource policy identifies placement needs, acceptable contention and the failure behavior when a running workload reaches its boundary.

CPU and memory fail differently

On a typical Linux setup, a CPU limit becomes a bandwidth constraint: a busy group can be throttled while its processes remain alive. A memory limit uses memory accounting and reclaim, with cgroup OOM handling when the constrained allocation cannot be satisfied. The kernel does not gently slow every memory allocation in the same way that CPU bandwidth can be delayed.

Memory enforcement is reactive. Do not describe it as an application callback fired at an exact byte boundary or as a guarantee that usage can never transiently exceed a configured number. The application might receive an allocation failure, or OOM handling can terminate a process. Observe the actual termination reason, container state and available memory events before naming the cause.

The memory limit covers more than the application’s intended payload. Runtime overhead, additional processes and charged cache or other memory can consume the group’s allowance. A program with a sixty-four-MiB limit cannot assume it can safely allocate a sixty-four-MiB data structure on top of everything else. Memory-backed volumes can contribute to memory consumption too; their storage appearance does not make that cost disappear.

When it breaks

An exit code of 137 indicates a SIGKILL-style termination convention; it is not sufficient evidence of OOM by itself. Correlate it with the container’s OOMKilled reason and relevant runtime or kernel evidence. A node-pressure eviction is another mechanism, with Pod-level status and eviction decisions that should not be conflated with one container exceeding its cgroup budget.

Restart policy determines what follows termination. A repeatedly leaking application under Always policy can enter a restart cycle while the underlying defect remains. With Never policy, the terminated container stays terminated and the Pod can fail. The limit contains one resource dimension; it neither repairs the application nor guarantees that neighboring workloads experience no contention before the failure.

Defaulting also matters. If a resource has a limit but no request and no admission mechanism supplies a different default, Kubernetes copies that limit into the request. Review the admitted object instead of assuming omitted requests mean zero. Matching memory requests and limits alone is not enough to establish the conventional Guaranteed QoS requirements; CPU settings and all relevant containers matter as well.

Use explicit units such as Mi and Gi for memory. Lowercase m denotes a milli-unit, not megabytes. Base limits on measured working sets and bursts, accounting for startup, concurrency and runtime overhead. A copied ratio or fixed percentile cannot choose acceptable OOM behavior for every application.

Check it quickly

Use a disposable Linux Kubernetes cluster with cgroup v2, kubectl access and python:3.11-slim-bookworm on its nodes. Save this as memory-limit.py and run python3 memory-limit.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. The account needs namespace creation and deletion permissions. The recorded server was Kubernetes 1.34.11. The fixture bounds its attempted payload to 128 MiB and places a 64 MiB limit on the allocating container.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-memory-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 allocator = '''from pathlib import Path
import json,time
group=Path('/sys/fs/cgroup')
print(json.dumps({'memory_max':(group/'memory.max').read_text().strip(),
 'initial_memory_current':int((group/'memory.current').read_text())}),flush=True)
while not Path('/tmp/go').exists(): time.sleep(0.1)
chunks=[]
for i in range(16):
 chunks.append(bytearray(8*1024*1024))
 print('allocated_chunks='+str(len(chunks)),flush=True)
 time.sleep(0.05)
raise RuntimeError('bounded allocation unexpectedly escaped the expected limit')
'''
 constrained = spec(allocator)
 constrained['restartPolicy'] = 'Never'
 constrained['containers'][0]['resources'] = {
 'requests': {'cpu': '10m', 'memory': '32Mi'}, 'limits': {'cpu': '200m', 'memory': '64Mi'}}
 for name, podspec in [('limited', constrained), ('neighbor', spec())]:
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': name}, 'spec': podspec})
 wait_for(lambda name=name: obj(namespace, 'pod', name), pod_ready)
 first_log = wait_for(lambda: k('logs', '-n', namespace, 'limited').stdout.strip(), bool)
 before = json.loads(first_log)
 assert before['memory_max'] == str(64 * 1024 * 1024)
 k('exec', '-n', namespace, 'limited', '--', 'python3', '-c',
 'from pathlib import Path; Path("/tmp/go").touch()')
 failed = wait_for(lambda: obj(namespace, 'pod', 'limited'), lambda p: p.get('status', {}).get('phase') == 'Failed')
 status = failed['status']['containerStatuses'][0]
 terminated = status['state']['terminated']
 assert terminated['reason'] == 'OOMKilled' and terminated['exitCode'] == 137
 assert status['restartCount'] == 0
 assert failed['status'].get('reason') != 'Evicted'
 neighbor = obj(namespace, 'pod', 'neighbor')
 assert pod_ready(neighbor) and neighbor['status']['containerStatuses'][0]['restartCount'] == 0
 output = k('logs', '-n', namespace, 'limited').stdout.splitlines()
 chunks = [int(line.split('=')[1]) for line in output if line.startswith('allocated_chunks=')]
 assert chunks and max(chunks) < 16
 print(json.dumps({'before': before, 'last_reported_allocated_chunks': max(chunks),
 'termination_reason': terminated['reason'], 'exit_code': terminated['exitCode'],
 'container_restarts_with_never_policy': 0,
 'pod_was_not_evicted': True, 'neighbor_still_ready': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The first log records memory.max as 67108864 bytes. After a release file appears, the original container process allocates eight-MiB chunks. The recorded run reported six completed chunks before termination. Its status shows OOMKilled and exit 137, while the Pod is Failed rather than Evicted. Never policy leaves its restart count at zero.

A separate idle test Pod remains Ready with no restart at the final observation. All assertions passed and the namespace was removed. This verifies one constrained OOM event and its lifecycle result; it does not benchmark neighbor latency or establish full node isolation. Use the failure evidence to size and diagnose the workload, rather than treating a configured limit as a complete availability guarantee.

Sources

Kubernetes: Resource Management, requests, limits, defaulting and Linux enforcement, checked 2026-09-13.
Kubernetes: Assign Memory Resources, memory limits and observed OOM termination, checked 2026-09-13.
Linux: Control Group v2, memory.max, reclaim and cgroup OOM behavior, checked 2026-09-13.

Capacity planning is a number, not a guess

Sun, 21 Feb 2027 00:00:00 +0000

Capacity planning turns a demand forecast into a provisioned operating envelope. The useful question is not how many requests a machine can accept before crashing, but what workload the service can sustain while meeting its latency and correctness requirements. That answer depends on request cost, dependencies, and failure assumptions. A spreadsheet becomes useful when each of those assumptions has a source and an owner.

Measure an acceptable operating point

Load tests should use a representative request mix, data size, cache state, concurrency, and software configuration. Increase demand while recording latency, errors, CPU, memory, queue depth, storage, and connection usage. The highest tested point meeting the chosen requirements is evidence for that workload, not proof that every intermediate point or future release behaves identically. A throughput result without outcome quality can reward a server that rejects work quickly.

Track different bottlenecks separately. More application replicas may increase CPU capacity while leaving a shared database connection limit unchanged. Expensive requests can also change the relationship between requests per second and resource consumption. A request-rate model is reasonable only while that cost distribution is comparable; monitor the underlying resources and revise the model when the workload changes. Do not project a quiet average into a peak-capacity promise.

Forecast demand and remove capacity deliberately

Start with an observed peak, then state the growth or seasonal scenario being planned. Add provisioning lead time and uncertainty explicitly. The operating reserve should cover a named reason, such as load imbalance, forecast error, or time needed to add instances, rather than being a mysterious percentage inherited from an old spreadsheet. Revisit both the measurement and the demand assumption after meaningful changes.

Next remove the capacity that the reliability design says may fail. A service spread across three zones needs a different instance count if it must serve the same peak with one zone unavailable. Account for which dependencies survive, how traffic moves, and whether enough instances can start before queues overflow. Balanced placement does not by itself guarantee balanced demand, and autoscaling cannot supply capacity instantaneously.

When it breaks

A common estimate divides forecast traffic by a single-machine maximum, rounds up, and stops. It may satisfy a healthy-cluster calculation while failing immediately after a zone loss. Counting reserved capacity twice is another error: distinguish a tested throughput point from the lower rate chosen for normal planning. State which boundary a reported headroom number uses so that spare capacity is not confused with physical maximum throughput.

Linear scaling can fail at a shared lock, database, network link, or external quota. Cold caches and synchronized retries can make a failover workload more expensive than the original benchmark. Validate the failure scenario under realistic conditions and record where the model stops applying. If a forecast cannot be served safely, admission control or a tested degraded mode may be necessary while provisioning catches up; adding an optimistic cell to the plan will not protect users.

Check it quickly

Save this as capacity.py and run Python 3. The inputs are synthetic load-test results for a fixed workload. This exercise verifies the sizing arithmetic and failure assumption, not the performance of your hardware. A planning rate of 70% of the accepted test point is an explicit example choice, not a universal utilization recommendation.

from math import ceil

# Synthetic load-test results for one fixed request mix and machine size.
samples = [
 {"rps": 150, "p99_seconds": 0.11, "error_ratio": 0.0001},
 {"rps": 200, "p99_seconds": 0.22, "error_ratio": 0.0003},
 {"rps": 250, "p99_seconds": 0.50, "error_ratio": 0.0020},
]
acceptable = [s["rps"] for s in samples
 if s["p99_seconds"] <= 0.30 and s["error_ratio"] <= 0.001]
tested_rps = max(acceptable)
forecast_rps = ceil(1600 * 1.25)
zones = 3
surviving_zones = zones - 1
planning_rps = tested_rps * 0.70

naive_per_zone = ceil(ceil(forecast_rps / tested_rps) / zones)
planned_per_zone = ceil(forecast_rps / (planning_rps * surviving_zones))
naive_survivor_rps = naive_per_zone * surviving_zones * tested_rps
planned_survivor_rps = planned_per_zone * surviving_zones * planning_rps

assert tested_rps == 200 and forecast_rps == 2000
assert naive_per_zone * zones == 12 and naive_survivor_rps == 1600
assert planned_per_zone * zones == 24 and planned_survivor_rps == 2240
assert naive_survivor_rps < forecast_rps <= planned_survivor_rps
print(f"tested={tested_rps} rps/node; forecast={forecast_rps} rps")
print(f"naive={naive_per_zone * zones} nodes; zone-loss capacity={naive_survivor_rps} rps")
print(f"planned={planned_per_zone * zones} nodes; zone-loss planning capacity={planned_survivor_rps:.0f} rps")
print(f"zone-loss spare={planned_survivor_rps - forecast_rps:.0f} rps")

The accepted listed point is 200 requests per second per node; the 250-request sample fails the quality requirements. Forecast demand becomes 2,000 requests per second. The naive balanced allocation uses twelve nodes but leaves only eight after a zone loss, providing 1,600 requests per second even at the accepted test rate.

The reserve-aware plan allocates eight nodes per zone, twenty-four total. Sixteen survivors at the lower planning rate provide 2,240 requests per second, leaving 240 below that operating limit. This result depends on the fixture’s fixed request mix, independent node scaling, and surviving dependencies. Replace those assumptions with measured evidence before turning the number into a purchase or a reliability commitment.

Sources

Google SRE: Handling Overload, resource capacity, request cost, and overload behavior, checked 2026-09-13.
Google SRE Workbook: Managing Load, capacity-aware routing and failure scenarios, checked 2026-09-13.
Google SRE: Monitoring Distributed Systems, saturation and limits of static load-test estimates, checked 2026-09-13.

Connection caps guard a shared database

Sat, 05 Sep 2026 00:00:00 +0000

A connection cap limits one owner’s demand; it does not automatically cap every application sharing a database. The useful budget is the sum of physical connections that can reach each backend, including idle pooled connections. Begin with the backend’s usable slots, then allocate them across all callers and deployment states.

Count pools before choosing their size

For a finite SQLAlchemy QueuePool, simultaneous connections are bounded by pool_size + max_overflow. That is a per-pool limit. Engines with separately created QueuePools have independent limits; worker processes and replicas can multiply them again. Count actual pools, since derived Engine views can share one. For identical owners, multiply replicas, processes per replica, pools per process, and the per-pool cap. For different owners, sum their individual caps instead.

Count the maximum overlapping deployment, not merely the number of ready replicas after rollout. New replicas can open connections while old replicas still drain work or retain idle connections. Include background jobs, migration tools, monitoring, and other services in the inventory. Connections to different database servers require separate budgets rather than one number that hides the constrained target.

PostgreSQL’s max_connections is the server ceiling, not the application’s allocation. Ordinary roles without reserved-slot privileges must leave both superuser_reserved_connections and reserved_connections untouched. The latter is for roles with pg_use_reserved_connections privileges. Explicitly subtract other ordinary clients and operating margin as well; do not borrow emergency capacity for routine traffic.

When it breaks

A service can respect every local pool cap and still exhaust the database because there are more pools than the worksheet counted. Unlimited overflow defeats that worksheet altogether. Conversely, an excessively small cap can cause acquisition waits even when the backend has capacity. Neither observation establishes that changing the database ceiling is the right fix.

More connections do not create more CPU, faster storage, or fewer conflicting transactions. PostgreSQL allocates some resources according to its configured ceiling, so raising it also has resource consequences. Examine acquisition waits, physical connections, active work, and transaction duration together. A cap constrains connection count; it does not guarantee throughput, fairness between services, or short lock waits.

An external pooler adds another accounting boundary. PgBouncer session pooling retains a server connection for the client session, whereas transaction pooling returns it after each transaction. Client connections therefore are not interchangeable with backend-session counts. Inventory the pooler’s server-side allocation and pooling mode; transaction mode changes assumptions about session features, so adopting it is not just a capacity-number edit.

Check it quickly

Use an empty directory on macOS or Linux. Run python3 -m venv .venv, then .venv/bin/python -m pip install SQLAlchemy==2.0.52. Save this as pool_probe.py and run .venv/bin/python pool_probe.py. It was checked with Python 3.14.4 and SQLite 3.53.0. Its three real pools live in one process and share a temporary local database file.

from pathlib import Path
from tempfile import TemporaryDirectory

import sqlalchemy
from sqlalchemy import create_engine, event
from sqlalchemy.exc import TimeoutError as PoolTimeout
from sqlalchemy.pool import QueuePool


assert sqlalchemy.__version__ == "2.0.52"
with TemporaryDirectory(prefix="pool-caps-") as directory:
 engines, held, opened, closed = [], [], [0, 0, 0], [0, 0, 0]
 try:
 for owner in range(3):
 engine = create_engine(
 "sqlite:///" + str(Path(directory) / "probe.db"),
 poolclass=QueuePool, pool_size=2, max_overflow=1, pool_timeout=0.05,
 )
 engines.append(engine)

 def on_connect(dbapi_connection, entry, index=owner):
 opened[index] += 1

 def on_close(dbapi_connection, entry, index=owner):
 closed[index] += 1

 event.listen(engine, "connect", on_connect)
 event.listen(engine, "close", on_close)
 for _ in range(3):
 held.append(engine.connect())
 try:
 unexpected = engine.connect()
 except PoolTimeout:
 pass
 else:
 unexpected.close()
 raise AssertionError("each pool must reject a fourth simultaneous lease")
 assert len({id(engine.pool) for engine in engines}) == 3
 view = engines[0].execution_options(logging_token="same-pool")
 assert view.pool is engines[0].pool
 print("an Engine execution-options view shares its parent pool; it adds no connection slots")
 assert opened == [3, 3, 3]
 assert [engine.pool.checkedout() for engine in engines] == [3, 3, 3]
 print("three independent pools each cap at 3, yet together hold 9 physical SQLite connections")

 for connection in held:
 connection.close()
 assert [engine.pool.checkedin() for engine in engines] == [2, 2, 2]
 assert closed == [1, 1, 1]
 print("after return, 6 connections remain idle; the 3 overflow connections are closed")
 finally:
 for connection in held:
 connection.close()
 for engine in engines:
 engine.dispose()
 assert opened == closed == [3, 3, 3]


def demand(replicas, processes, pools_per_process, size, overflow):
 return replicas * processes * pools_per_process * (size + overflow)


# Illustrative PostgreSQL worksheet, not a live server configuration or benchmark.
maximum, superuser_reserve, role_reserve = 100, 3, 5
other_clients, operating_margin = 24, 8
budget = maximum - superuser_reserve - role_reserve - other_clients - operating_margin
assert budget == 60
assert demand(6, 1, 1, 8, 2) == 60
assert demand(8, 1, 1, 8, 2) == 80 > budget # Two extra replicas during rollout.
assert demand(8, 1, 1, 6, 1) == 56 <= budget
assert demand(8, 2, 1, 6, 1) == 112 > budget # A second process creates another pool.
assert demand(8, 1, 2, 6, 1) == 112 > budget # A second independent pool per process.
print("ordinary-client allocation: 60; steady demand 60; rollout demand 80; revised rollout demand 56")
print("doubling either processes or pools raises that revised demand to 112; no PostgreSQL server was run")

Each pool holds three physical SQLite connections and rejects a fourth checkout with PoolTimeout. Together they hold nine, proving that a local cap of three is not a database-wide cap of three. An execution-options Engine view shares the first pool and adds no capacity. Returning the leases retains six idle connections and closes three excess connections. Disposing all Engines after return makes the observed connect and close event counts agree.

The second half is explicitly an arithmetic model, not a running PostgreSQL configuration. Its illustrative ceiling of 100 loses three superuser slots, five reserved-role slots, 24 other ordinary-client slots, and eight margin slots, leaving 60. Five reserved-role slots are a chosen example; PostgreSQL’s documented default for that setting is zero.

Six replicas with caps of ten fit exactly, but eight overlapping replicas require 80. Revising each cap to six retained plus one overflow gives 56 during overlap. Adding a second worker process or a second pool per process doubles that to 112, exceeding the same allocation.

These checks establish local pool behavior and budget arithmetic, not production performance or a safe universal pool size. No PostgreSQL server, PgBouncer instance, or real rollout was exercised. Before changing production limits, verify the actual pool inventory and maximum overlap, then measure waiting and backend saturation under the intended workload.

Sources

SQLAlchemy connection pooling, finite QueuePool capacity and idle retention, checked 2026-09-09.
SQLAlchemy Engine and Connection, derived Engine views can share a connection pool, checked 2026-09-09.
SQLAlchemy pool events, observed connection creation and close events, checked 2026-09-09.
PostgreSQL connection settings, server ceiling and reserved-slot privileges, checked 2026-09-09.
PgBouncer features, session and transaction pooling boundaries, checked 2026-09-09.

Capacity · XB Field Notes

A token bucket smooths the request rate

Refill credit, then charge work

A burst allowance is not a peak rate

Match the budget to its scope

When it breaks

Check it quickly

Further reading

Sources

Least connections measures occupied connections, not every kind of load

When it breaks

Check it quickly

Compare the observed metric with the resource you need to protect

Sources

A leaked connection drains the pool

A lease has an owner and an end

When it breaks

Check it quickly

Sources

Resource limits constrain execution; requests guide placement

CPU and memory fail differently

When it breaks

Check it quickly

Sources

Capacity planning is a number, not a guess

Measure an acceptable operating point

Forecast demand and remove capacity deliberately

When it breaks

Check it quickly

Further reading

Sources

Connection caps guard a shared database

Count pools before choosing their size

When it breaks

Check it quickly

Sources