Load shedding protects the system

Load shedding deliberately refuses some work when accepting everything would threaten useful service. The aim is to preserve capacity for work that can still complete, not to promise that every accepted request will be fast or important. Rejection is itself work, so the admission point and its cost matter as much as the threshold.

Protect useful capacity

An overloaded queue consumes memory and adds waiting time. Clients can abandon requests before workers start them, leaving the server to spend resources on answers nobody needs. Longer queues can absorb some bursts, but cannot create processing capacity under sustained excess demand.

Google SRE describes queue limits, resource-pressure signals, selective rejection, and reduced-quality responses. Shedding and degradation are related but different: refusing a request is not the same as completing a cheaper version of it. The latter needs an explicit product contract about the result that remains useful.

Protection can also target different components. Envoy’s overload manager protects the proxy’s own resources, while its circuit-breaking mechanisms primarily protect upstream services. A dependency breaker does not automatically prevent local memory, connection, or file-descriptor exhaustion.

Choose a justified policy

Priority-aware shedding can protect critical business paths, but priority is not determined by HTTP method or retryability alone. A read can be essential, while a write can be expendable telemetry. Use trusted application classification rather than letting every caller declare its own request critical.

Uniform admission limits or proportion-based rejection can also be legitimate policies. They are not inherently incorrect because they lack business priority. State the fairness and capacity tradeoffs, including whether lower-priority work can starve and whether different tenants share the same limit.

Count, bytes, execution time, and CPU cost are different budgets. A cap on three tasks does not bound memory when each task can retain an arbitrarily large payload. Select signals that correspond to the resource being protected and measure their behavior under realistic request mixes.

Bound admission, not only workers

A thread-pool worker limit bounds simultaneous execution, not necessarily accepted queued work. A semaphore acquired only inside a worker leaves the queue in front of it unbounded. Make a bounded admission decision before submitting expensive work, and return that capacity when its lifecycle actually ends.

Even similarly named proxy limits need precise interpretation. NGINX limit_conn counts requests under its documented connection rules; with HTTP/2 and HTTP/3, each concurrent request counts separately. Do not assume a configured connection cap always means a count of TCP sockets.

The fixture below bounds running plus queued tasks. Refused submissions create no future. A completion callback returns permits after success, failure, or cancellation before execution; a rejected submission to a closed executor also returns its unused permit.

When it breaks

Rejecting after parsing huge bodies, opening database connections, or performing a business write may be too late to protect the intended resource. Expensive logging or response generation can itself overload the rejection path. Observe rejection cost alongside accepted throughput, queue age, active work, and resource pressure.

HTTP 503 can signal temporary service unavailability, while 429 concerns excessive request rates under a caller policy. Retry-After is a wait signal, not proof that replaying a write is safe or that capacity will exist later. Immediate retries can restore the very load that was shed.

Do not reclaim a running task’s permit merely because its caller stopped waiting. A future-result timeout does not terminate the worker. Premature release allows replacement work to overlap the supposedly removed work and breaks the capacity bound.

Check it quickly

Save the block as shedding_check.py and run python3 shedding_check.py with Python 3.10 or newer. It ran with Python 3.14.4 using only the standard library. Real worker threads wait on test coordination events; no external service is called, and no production load is generated. The fixture compares identical twelve-task bursts.

from concurrent.futures import ThreadPoolExecutor
from threading import Barrier, BoundedSemaphore, Event, Lock


def exercise(gated):
    slots = BoundedSemaphore(3)  # Bounds queued plus running work, not just threads.
    started, release, lock = Barrier(3), Event(), Lock()
    running, began = set(), []

    def held(job):
        with lock:
            running.add(job)
            began.append(job)
        try:
            if job < 2:
                started.wait(timeout=5)
            assert release.wait(timeout=5), "test coordinator did not release workers"
            return job
        finally:
            with lock:
                running.remove(job)

    with ThreadPoolExecutor(max_workers=2) as pool:
        def submit(function, *args):
            if not gated:
                return pool.submit(function, *args)
            if not slots.acquire(blocking=False):
                return None  # Explicit local admission refusal; no future was enqueued.
            try:
                future = pool.submit(function, *args)
            except RuntimeError:
                slots.release()
                raise
            # Completion, failure, and cancellation before execution all return the slot.
            future.add_done_callback(lambda _: slots.release())
            return future

        futures = [submit(held, job) for job in range(12)]
        accepted = [future for future in futures if future is not None]
        replacement = None
        try:
            started.wait(timeout=5)
            with lock:
                assert running == {0, 1} and set(began) == {0, 1}
            assert all(not future.done() for future in accepted)
            if gated:
                assert len(accepted) == 3 and futures[3:] == [None] * 9
                assert futures[2].cancel()  # Both workers are held, so job 2 has not started.
                replacement = submit(held, 12)
                assert replacement is not None
                assert submit(held, 13) is None
                print("gated burst: 2 running + 1 queued, 9 initial refusals; cancellation returns a slot")
            else:
                assert len(accepted) == 12
                print("ungated burst: 2 workers accept 12 unfinished tasks; the other 10 still wait")
        finally:
            release.set()
        completed = [future.result(timeout=5) for future in accepted if not future.cancelled()]
        if replacement is not None:
            completed.append(replacement.result(timeout=5))
        assert sorted(completed) == ([0, 1, 12] if gated else list(range(12)))

        if gated:
            def fail():
                raise ValueError("injected task failure")

            failed = submit(fail)
            assert failed is not None
            try:
                failed.result(timeout=5)
            except ValueError as error:
                assert str(error) == "injected task failure"
            else:
                raise AssertionError("the failed task must preserve its failure")

    if gated:
        # The executor is now closed; a failed submission must not consume a slot.
        try:
            submit(lambda: None)
        except RuntimeError:
            pass
        else:
            raise AssertionError("submission to a closed executor must fail")
        acquired = [slots.acquire(blocking=False) for _ in range(3)]
        try:
            assert acquired == [True, True, True]
            assert not slots.acquire(blocking=False)
        finally:
            for success in acquired:
                if success:
                    slots.release()
        print("completion, task failure, cancellation, and closed-executor submission leave all 3 slots reusable")


exercise(False)
exercise(True)

With two workers but no admission gate, twelve tasks remain unfinished: two running and ten waiting. With three permits, only three are accepted and nine are refused before submission. The admitted set contains two running tasks and one queued task, not three simultaneous workers.

Canceling the queued task returns a permit for one replacement, while another submission is refused. After coordination releases the workers, the expected tasks complete. An injected task failure and a submission after executor shutdown both preserve their errors. Public semaphore operations finally confirm that all three permits are reusable.

This tests admission accounting and task lifecycle, not HTTP responses, byte limits, priority scheduling, production latency, or an automatic resource-pressure controller. NGINX and Envoy are cited for their contracts, not claimed to be deployed or exercised here.

Sources

Google SRE: Handling overload, criticality, resource signals, rejection costs, and client behavior, checked 2026-09-08.
Google SRE: Addressing cascading failures, queue limits, load shedding, deadlines, and degradation, checked 2026-09-08.
Envoy overload manager, local resource protection versus upstream circuit breaking, checked 2026-09-08.
NGINX connection limiting, counted requests, HTTP/2–HTTP/3 semantics, and rejection status, checked 2026-09-08.
Python futures, executor submission, result timeouts, cancellation, and completion callbacks, checked 2026-09-08.
Python threading, nonblocking bounded-semaphore admission and coordination primitives, checked 2026-09-08.
RFC 9110, service-unavailability, Retry-After, and retry safety semantics, checked 2026-09-08.
RFC 6585, section 4’s separate definition of 429 and server-selected counting policy, checked 2026-09-08.

Protect useful capacity#

Choose a justified policy#

Bound admission, not only workers#

When it breaks#

Check it quickly#

Further reading#

Sources#