Resilience · XB Field Notes

A cache stampede hits when expiry syncs

Mon, 22 Nov 2027 00:00:00 +0000

A cache stampede occurs when concurrent misses cause redundant work against the same dependency. Synchronized expiration is one trigger, but eviction, a cold deployment, or cache unavailability can create the same burst. A single popular key can be enough. The useful question is how many rebuilds reach the source while one result is still being computed.

Coalesce work within its actual boundary

Go’s singleflight Group gives one key at most one in-flight function execution within that Group. Duplicate callers receive the same result, including the same error. Separate Groups do not coordinate, even within one process, and separate service instances normally have separate Groups. It is neither a distributed lock nor a cache of completed values.

Keep the Group shared across requests, and make its key identify equivalent work. Mixing tenants, permissions, currencies, or representations under one key can share the wrong result. In a cache-backed loader, check the cache again inside the shared function, fetch only if still necessary, and publish a successful value before returning. Singleflight itself does none of those storage operations.

Spread deadlines without changing the promise

Jitter spreads expiration across different entries or independently maintained copies. It does not prevent all callers of one expired hot key from colliding. If a maximum TTL is a freshness requirement, randomly shortening lifetimes within a chosen bound avoids silently exceeding it; adding arbitrary positive jitter changes that requirement. There is no universally correct percentage or number of seconds.

Early refresh and permitted stale responses can keep callers from waiting, but they require explicit application rules. A last-known public catalog may be acceptable temporarily; a revoked permission or current balance may not be. Coalescing reduces duplicate work without deciding whether old data is safe, and a cold cache still needs a source response.

When it breaks

A stuck shared loader can hold many waiters. Bound the loader’s dependency calls and give callers their own waiting deadlines. DoChan returns a result channel, not a cancellation mechanism: one caller abandoning its wait does not automatically stop the underlying work. Forget allows a new execution for the key; it does not cancel the old execution, so it can reintroduce overlapping loads.

Errors are shared only while that execution is in flight. Immediate retries after it finishes can create successive waves against a failing dependency. Use a retry and admission policy matched to that dependency; cacheable absence and transient failure need different treatment. Likewise, a fleet of one hundred local Groups can still create one hundred rebuilds. Do not describe one-process suppression as a global origin-load bound.

Check it quickly

Use Go 1.25 or newer; this exercise was run with Go 1.27.0 and x/sync v0.22.0. Create an isolated module below, put the Go block in main.go there, and run go run -race main.go. Dependency installation requires the network. The experiment starts at the cold-miss boundary and invokes counted loader functions, not a real database, cache server, or timed TTL implementation.

stampede_lab=$(mktemp -d)
cd "$stampede_lab"
go mod init example.invalid/stampede-lab
go get golang.org/x/sync@v0.22.0

package main

import (
 "errors"
 "fmt"
 "sync"
 "sync/atomic"

 "golang.org/x/sync/singleflight"
)

const callers = 12

func require(ok bool) {
 if !ok {
 panic("unexpected result")
 }
}

func wave(groups []*singleflight.Group, failure error) int32 {
 var calls atomic.Int32
 gate := make(chan struct{})
 results := make([]<-chan singleflight.Result, callers)
 load := func() (any, error) {
 calls.Add(1)
 <-gate
 if failure != nil {
 return nil, failure
 }
 return "catalog-v1", nil
 }
 for i := range results {
 results[i] = groups[i%len(groups)].DoChan("catalog", load)
 }
 close(gate)
 for _, result := range results {
 r := <-result
 require(r.Shared && r.Err == failure)
 if failure == nil {
 require(r.Val == "catalog-v1")
 } else {
 require(r.Val == nil)
 }
 }
 return calls.Load()
}

func main() {
 var plain atomic.Int32
 var workers sync.WaitGroup
 gate := make(chan struct{})
 for range callers {
 workers.Add(1)
 go func() {
 defer workers.Done()
 plain.Add(1)
 <-gate
 }()
 }
 close(gate)
 workers.Wait()
 require(plain.Load() == callers)
 fmt.Println("without coalescing: 12 loader calls")

 group := new(singleflight.Group)
 require(wave([]*singleflight.Group{group}, nil) == 1)
 require(wave([]*singleflight.Group{group}, nil) == 1)
 fmt.Println("one Group: 1 call per wave; completed results are not cached")
 other := new(singleflight.Group)
 require(wave([]*singleflight.Group{group, other}, nil) == 2)
 fmt.Println("two independent Groups: 2 loader calls")

 failure := errors.New("origin unavailable")
 require(wave([]*singleflight.Group{group}, failure) == 1)
 require(wave([]*singleflight.Group{group}, nil) == 1)
 fmt.Println("one error reaches 12 callers; next wave starts a new load")
}

The baseline executes twelve loaders. Each single-Group wave executes one, and running a second wave on the same Group executes another: finished results are not retained. Splitting the twelve callers between two Groups produces two executions. A failing wave sends the same error to all twelve callers; the next successful wave starts fresh rather than retrieving a cached failure.

The gate keeps the shared load pending until every DoChan call has registered, so the test does not depend on sleeps or favorable scheduling. Shared is true for all results in these joined waves, including the initiating caller’s result. Receive exactly one result from each channel; DoChan does not close those channels, so ranging over one would wait indefinitely.

For a service-level follow-up, measure loader executions, waiting callers, latency, and failures during a deliberately cold test in an isolated environment. Include multiple instances and source timeouts, not just a high warm-cache hit ratio. The race-detector run validates this program’s observed execution, not every possible race in a production cache implementation.

Sources

Go x/sync v0.22.0: singleflight, Group scope, DoChan, shared errors, and Forget, checked 2026-09-07.
AWS Builders’ Library: caching challenges and strategies, cold caches, request coalescing, failure behavior, and empirical expiration choices, checked 2026-09-07.

Anycast serves the nearest replica

Mon, 01 Nov 2027 00:00:00 +0000

Anycast gives an address several homes. Multiple sites announce the same prefix, and each network selects one usable path according to its BGP policy. A client usually lands at a topologically close replica without first contacting a global load balancer. When one site withdraws its announcement, the control plane can converge on another home.

One address, many announcements

An anycast service assigns the same unicast address or prefix to independent instances. Each edge router advertises reachability with BGP. Neighbors compare paths and install a best route in their local routing information base; packets then follow ordinary forwarding toward that selected next hop. The application sees one destination even though several sites can answer it.

“Nearest” means nearest according to routing policy and topology, not a guaranteed geographic distance. A higher local preference can beat a physically shorter path. If local preference ties, a shorter AS path is generally preferred, followed by more vendor- and policy-specific tie breakers. Operators can steer traffic by changing these attributes, prepending an AS path, or withdrawing a prefix. The important contract is reachability, not a promise that every client chooses the same site.

Routing performs the failover

If a site loses its service or upstream, its route should be withdrawn, often by a health-controlled session. Upstream routers propagate the withdrawal and select a surviving announcement. The forwarding path does not need a DNS answer to change, so a whole edge location can disappear while the address remains valid. BGP convergence is not instantaneous: packets can be lost during the transition, and different networks may converge at different times. Clients still need retries and sensible connection timeouts.

Anycast is especially useful for stateless or easily re-established protocols such as DNS, UDP-based discovery, and CDN edges. It also spreads an attack surface across sites, although capacity and filtering must be coordinated. A route that remains advertised while the process is dead is worse than no route: traffic is sent to a black hole. Tie route announcements to a real readiness signal and monitor both control-plane reachability and application responses.

Stateful sessions mind the move

An established TCP connection can reach a different site after a route change. Without the old socket state, that instance may reset or drop it. A shared backend preserves application state, not the TCP connection; clients still need reconnect-and-rebuild behavior. Idempotency keys can prevent repeated side effects after a reconnect. Do not infer stickiness from one successful trace: topology changes can select another replica even during a flow.

Observability must include the serving site, route age, and withdrawal reason. A single global success rate can hide one region black-holing traffic while other regions look healthy. Sample from several networks, record the anycast address separately from the site identity, and alert when a site is still advertised but its service checks fail.

When it breaks

A route leak or an accidental high local preference can attract traffic far from the intended site. A partial withdrawal can leave some peers on a stale path, producing asymmetric loss rather than a clean failover. MTU, firewall, and ACL differences between replicas create failures that look like random routing. If the service has local state, users may see resets only after the control plane has already converged.

Avoid advertising a prefix from a site until the process, dependencies, and return path are ready. Test withdrawal from a staging address, measure convergence from more than one vantage point, and keep a non-anycast management path for recovery. Routing is the selection mechanism; it is not an application health check unless the announcement is explicitly coupled to one.

Check it quickly

This local Go model captures only local preference and AS-path length for one prefix. It tests path tie-breaking, input-order independence, policy override, withdrawal, an empty candidate set, and backup recovery. No packets or BGP session leave the process.

package main

import "fmt"

type route struct {
 name string
 localPref int
 asPath int
 advertised bool
}

// This fixture models only local preference and AS-path length for one prefix.
func best(routes []route) (route, bool) {
 var chosen route
 for _, candidate := range routes {
 if !candidate.advertised {
 continue
 }
 if chosen.name == "" || candidate.localPref > chosen.localPref ||
 (candidate.localPref == chosen.localPref && candidate.asPath < chosen.asPath) {
 chosen = candidate
 }
 }
 return chosen, chosen.name != ""
}

func want(ok bool, message string) {
 if !ok {
 panic(message)
 }
}

func main() {
 routes := []route{
 {name: "tokyo", localPref: 100, asPath: 2, advertised: true},
 {name: "singapore", localPref: 100, asPath: 3, advertised: true},
 }
 chosen, ok := best(routes)
 want(ok && chosen.name == "tokyo", "shorter AS path should win the tie")
 chosen, ok = best([]route{routes[1], routes[0]})
 want(ok && chosen.name == "tokyo", "input order must not beat path length")
 routes[1].localPref = 200
 chosen, ok = best(routes)
 want(ok && chosen.name == "singapore", "local policy should override path length")
 routes[1].advertised = false
 chosen, ok = best(routes)
 want(ok && chosen.name == "tokyo", "withdrawal should select the surviving site")
 routes[0].advertised = false
 _, ok = best(routes)
 want(!ok, "with every announcement withdrawn there is no route")
 _, ok = best(nil)
 want(!ok, "an empty candidate set must not invent a route")
 fallback := route{name: "sydney", localPref: 50, asPath: 4, advertised: true}
 chosen, ok = best(append(routes, fallback))
 want(ok && chosen.name == "sydney", "a backup announcement should recover reachability")
 fmt.Println("PASS: path length, policy priority, withdrawal, empty set, and backup")
}

Save this as anycast_model.go and run go run anycast_model.go. The expected output is PASS: path length, policy priority, withdrawal, empty set, and backup. Check formatting with gofmt -d, static diagnostics with go vet, and execution with go run -race. The model omits MED, eBGP/iBGP rules, communities, topology, and convergence timing. It proves selection among given candidates, not convergence on the Internet.

Sources

RFC 4271: A Border Gateway Protocol 4, route selection and path attributes, checked 2026-09-09.
RFC 4786: Operation of Anycast Services, deployment, routing, and failure considerations, checked 2026-09-09.

Circuit breakers stop storming a down service

Tue, 06 Apr 2027 00:00:00 +0000

A circuit breaker decides whether a new call should reach a dependency that appears unhealthy. It can reject promptly instead of spending resources on another likely failure. It does not repair that dependency, cancel calls already running, or prove that rejected work would have failed. Its value depends on the failure signals and recovery policy it actually implements.

Name the policy, not just the states

Closed means calls may proceed, not that every call will succeed. Open means new calls are refused under the breaker’s policy. Half-open admits limited trial work to gather evidence before deciding whether ordinary traffic can resume.

The detection rule is not universally a sliding-window failure percentage. Fowler’s example counts failures and resets on success; other implementations use time windows, failure ratios, slow-call thresholds, or minimum sample counts. Specify the actual rule rather than borrowing a diagram and assuming its thresholds.

The fixture uses two consecutive classified failures in completion order, a five-second cooldown, and one half-open probe. One success resets the consecutive counter. These numbers make transitions easy to verify; they are not suggested production settings.

Classify and scope failures

A malformed caller request does not necessarily mean a dependency is down. Decide which exceptions, statuses, and slow outcomes should influence the breaker. A locally refused call is also different from a request that reached the dependency and failed. Mixing those observations can make the breaker reinforce its own refusal state.

Scope state to the failure domain you intend to protect. One unavailable shard or operation need not justify blocking a healthy independent provider. Conversely, thousands of independent process-local breakers can each allow a probe, so “one half-open request” may mean one per instance, not one globally.

Timeouts remain necessary. A call that never finishes might hold a connection or worker without ever producing the failure observation needed to open the breaker. The admission decision does not replace a deadline, a concurrency limit, or resource cleanup.

Recovery is a controlled experiment

After cooldown, reserve probe admission atomically. If several threads all observe that trial work is allowed before any claims the permit, the supposed single probe becomes a burst. The example protects its small state transition with a lock; it does not hold that lock while a dependency operation runs.

An old in-flight result also needs an explicit interpretation. Here, opening creates a new local generation, and results from older permits cannot close that newer open state. The generation is bookkeeping inside this model, not a distributed fencing token or a way to undo remote work.

A failed probe starts another cooldown. A successful probe closes this fixture’s breaker, but one successful trial does not prove capacity for all returning traffic. If gradual recovery is required, validate the actual traffic-admission policy instead of assuming the word half-open guarantees a ramp.

When it breaks

Aggressive thresholds can deny healthy requests; long cooldowns can keep refusing after recovery. Overly broad scopes hide partial health. A retry loop that repeatedly catches an open-circuit refusal can waste resources even though it sends no new dependency requests.

Observe forwarded calls, classified outcomes, local refusals, state transitions, and probe results separately. Test sustained failures, occasional successes, concurrent trial admission, and late completions. A green unit test for a counter is not evidence that a live dependency recovers safely.

Check it quickly

Save the block as breaker_check.py and run python3 breaker_check.py with Python 3.10 or newer. It ran with Python 3.14.4 using only the standard library. The script models outcomes and injects monotonic-time values without sleeping or contacting a service. Eight real threads contend for a half-open permit.

from concurrent.futures import ThreadPoolExecutor
from threading import Barrier, Lock


class OpenCircuit(Exception):
 pass


class Breaker:
 # Explicit fixture policy: two consecutive classified failures, five-second cooldown.
 def __init__(self):
 self.state, self.failures, self.until = "closed", 0, 0
 self.epoch, self.probe_busy = 0, False
 self.lock = Lock()

 def acquire(self, now):
 with self.lock:
 if self.state == "open":
 if now < self.until:
 raise OpenCircuit
 self.state = "half-open"
 if self.state == "half-open":
 if self.probe_busy:
 raise OpenCircuit
 self.probe_busy = True
 return self.epoch, True
 return self.epoch, False

 def complete(self, permit, succeeded, now):
 epoch, probe = permit
 with self.lock:
 if epoch != self.epoch:
 return # This outcome belongs to an earlier breaker generation.
 if probe:
 self.probe_busy = False
 self.epoch += 1
 self.failures = 0
 self.state = "closed" if succeeded else "open"
 if not succeeded:
 self.until = now + 5
 elif self.state == "closed":
 self.failures = 0 if succeeded else self.failures + 1
 if self.failures >= 2:
 self.state, self.until = "open", now + 5
 self.epoch += 1


breaker = Breaker()
other_dependency = Breaker()
forwarded = []


def invoke(succeeded, now):
 permit = breaker.acquire(now)
 forwarded.append(now) # A modeled dependency invocation, not a network call.
 breaker.complete(permit, succeeded, now)


invoke(False, 0)
assert breaker.failures == 1
invoke(True, 1)
assert breaker.failures == 0
late = breaker.acquire(1)
invoke(False, 2)
invoke(False, 3)
assert breaker.state == "open" and breaker.until == 8
other_permit = other_dependency.acquire(4)
other_dependency.complete(other_permit, True, 4)
assert other_dependency.state == "closed"
before = len(forwarded)
try:
 invoke(True, 7.99)
except OpenCircuit:
 pass
else:
 raise AssertionError("open circuit must reject before dispatch")
assert len(forwarded) == before
breaker.complete(late, True, 7.99)
assert breaker.state == "open"
print("success resets consecutive failures; two later failures open until time 8")
print("open calls are not dispatched; an older in-flight success does not close a newer open generation")

gate = Barrier(8)


def probe_attempt(_):
 gate.wait(timeout=5)
 try:
 return breaker.acquire(8)
 except OpenCircuit:
 return None


with ThreadPoolExecutor(max_workers=8) as pool:
 permits = list(pool.map(probe_attempt, range(8)))
admitted = [permit for permit in permits if permit is not None]
assert len(admitted) == 1 and admitted[0][1] is True
assert breaker.state == "half-open" and breaker.probe_busy
breaker.complete(admitted[0], False, 8)
assert breaker.state == "open" and breaker.until == 13
print("eight simultaneous half-open candidates receive one probe permit and seven refusals")
print("failed probe opens a new cooldown ending at time 13")

try:
 breaker.acquire(12.99)
except OpenCircuit:
 pass
else:
 raise AssertionError("second cooldown must still apply")
probe = breaker.acquire(13)
breaker.complete(probe, True, 13)
assert breaker.state == "closed" and not breaker.probe_busy
invoke(True, 14)
assert other_dependency.state == "closed" and other_dependency.failures == 0
print("a successful later probe closes this breaker; another dependency has separate state")

An initial success resets the failure count. Two later failures open the breaker until time eight, and an attempted call before that boundary adds no modeled dispatch. A previously admitted success cannot close the newer generation. A second dependency’s already existing breaker remains usable.

At time eight, eight simultaneous candidates obtain one permit and seven refusals. Failure of that probe opens another interval ending at thirteen. A later successful probe closes the breaker and permits ordinary work again.

Each granted permit in this model receives one completion with an already classified outcome. Real transport timeouts, cancellation, automatic failure classification, distributed probe coordination, library integration, and recovery throughput are not tested. Use an established implementation where appropriate and verify its precise contracts against the real workload.

Sources

Martin Fowler: Circuit breaker, failure counting, success reset, and trial recovery, checked 2026-09-08.
Azure architecture: Circuit breaker, state semantics, exception classification, concurrency, and failure-domain scope, checked 2026-09-08.
Python threading, locks and barriers used by the concurrent permit experiment, checked 2026-09-08.

Serving stale cache beats downtime

Mon, 01 Feb 2027 00:00:00 +0000

A stale response can preserve access to a public manual during an origin outage. It can also preserve an obsolete price, revoked permission, or misleading status. Availability is not automatically more important than correctness. Choose the information that may remain usable, the conditions that permit reuse, and the maximum tolerable age before enabling this behavior.

The trade is availability over freshness

Expiration and eviction are different events. An expired response can remain stored for conditional validation or permitted stale delivery. But a cold or evicted cache has no old body to return. Stale directives cannot create one, and retaining bytes does not by itself authorize reuse. The selected response must still match the request, including relevant Vary dimensions and any access boundary.

How it is signaled

Stale-while-revalidate permits a bounded stale response while validation is attempted without blocking that response. Stale-if-error instead permits using a stale response when the attempted origin interaction encounters an eligible error. These are different triggers. Neither extension mandates storage or guarantees that every cache implements the feature. A response arriving after the allowed window cannot rely on the earlier permission indefinitely.

For max-age=30, stale-if-error=60, the extra sixty seconds begin after freshness ends, not after each failed attempt. The response can be a candidate for error-time reuse for roughly ninety seconds of total age. RFC 5861 identifies errors that would produce 500, 502, 503, or 504. Authentication failures and missing resources must not casually be relabeled as availability failures to hide them behind an old success response.

When it breaks

Resetting Age after every failed revalidation turns a bounded exception into unlimited stale delivery. So does replacing the entry’s stored timestamp while keeping the old body. Preserve age on failure; only a legitimate new response or successful validation supplies the basis for updating freshness. Background validation is usually demand-triggered, so an idle cache does not necessarily refresh itself before traffic returns.

Read-only or idempotent does not mean safe to serve stale. A GET can reveal permissions or prices whose age matters. RFC 9111 prohibits stale reuse when an applicable directive such as no-cache or must-revalidate forbids it. Avoid contradictory policies and test the deployed cache’s treatment of restrictions. The old Warning-field advice in RFC 5861 is superseded by RFC 9111, which obsoletes that field.

Observe stale responses separately from fresh hits. Record total age, the failure category, and whether validation succeeded without logging secrets. A healthy HTTP success rate can conceal a failing origin if every response is old. Recovery requires verifying actual replacement or successful validation, not merely noticing that the origin process has started again.

Check it quickly

Use Node.js and npm; this was run with Node 26.0.0 and http-cache-semantics 4.2.0. Install the pinned package, then execute the JavaScript with NODE_PATH="$stale_lab/node_modules" node. This is a deterministic policy-and-body-selection experiment with synthetic responses and a controlled clock. It makes no requests and causes no outage; only installation needs network access.

stale_lab=$(mktemp -d)
npm install --prefix "$stale_lab" --ignore-scripts --no-audit --no-fund --package-lock=false http-cache-semantics@4.2.0

const assert = require("node:assert/strict");
const CachePolicy = require("http-cache-semantics");

const start = Date.parse("2026-09-07T00:00:00Z");
let clock = start;
class TestPolicy extends CachePolicy {
 now() { return clock; }
}
const request = { url: "https://example.test/manual", method: "GET", headers: {} };
const policy = new TestPolicy(request, { status: 200, headers: {
 "cache-control": "public, max-age=30, stale-if-error=60",
 etag: '"revision-a"', age: "0",
} });
const oldBody = Buffer.from("manual revision A");
clock = start + 35000;
const decision = policy.evaluateRequest(request);
assert.equal(decision.response, undefined);
assert.equal(decision.revalidation.synchronous, true);
assert.equal(decision.revalidation.headers["if-none-match"], '"revision-a"');
console.log("age 35: validate before using stale-if-error");

for (const status of [500, 502, 503, 504]) {
 const result = policy.revalidatedPolicy(request, { status, headers: { "cache-control": "no-store" } });
 assert.equal(result.modified, false);
 assert.equal(result.policy, policy);
 assert.equal(result.policy.responseHeaders().age, "35");
 const body = result.modified ? Buffer.from("origin error") : oldBody;
 assert.equal(body.toString(), "manual revision A");
 console.log(status, "uses old body without resetting Age");
}
const denied = policy.revalidatedPolicy(request, { status: 403, headers: { "cache-control": "no-store" } });
assert.equal(denied.modified, true);
assert.equal(denied.policy.storable(), false);
console.log("403 does not use the stale success response");

clock = start + 89000;
assert.equal(policy.revalidatedPolicy(request, { status: 503, headers: {} }).modified, false);
clock = start + 91000;
const expired = policy.revalidatedPolicy(request, { status: 503, headers: { "cache-control": "no-store" } });
assert.equal(expired.modified, true);
assert.equal(expired.policy.storable(), false);
console.log("age 89: stale allowed; age 91: error must be surfaced");

const recovered = policy.revalidatedPolicy(request, { status: 200, headers: {
 "cache-control": "public, max-age=30, stale-if-error=60",
 etag: '"revision-b"', age: "0",
} });
assert.equal(recovered.modified, true);
assert.equal(recovered.policy.age(), 0);
assert.equal(recovered.policy.satisfiesWithoutRevalidation(request), true);
const newBody = Buffer.from("manual revision B");
const delivered = recovered.modified ? newBody : oldBody;
assert.equal(delivered.toString(), "manual revision B");
console.log("recovery 200: replace the body and start the new policy");

At age thirty-five the normal decision requires synchronous validation; stale-if-error alone does not authorize an immediate stale hit. Each eligible simulated error then keeps revision A and its original policy. Outgoing Age remains thirty-five rather than resetting to zero. The 403 branch does not reuse that success body and its no-store response is ineligible for storage.

The same old entry survives an eligible error at age eighty-nine but not ninety-one. Repeated failures have not extended the window. A subsequent successful 200 with revision B produces a fresh policy and selects the new body. The fixture verifies both metadata decisions and which bytes the caller retains or replaces.

The library does not fetch, retry, schedule background work, or retain bodies for the caller. These assertions cover one matching warm entry and this nonconflicting policy, not network timeout handling, all directive combinations, or a complete CDN. Deployment acceptance still needs cold-cache, exhausted-window, and recovery tests through the real path, alongside evidence that stale delivery stays within the resource’s correctness requirements.

Sources

RFC 5861: stale controls, extension windows and eligible error statuses, checked 2026-09-07.
RFC 9111: serving stale responses, restrictions on stale reuse, checked 2026-09-07.
http-cache-semantics 4.2.0, revalidation decisions and caller-owned body storage, checked 2026-09-07.

Resilience · XB Field Notes

A cache stampede hits when expiry syncs

Coalesce work within its actual boundary

Spread deadlines without changing the promise

When it breaks

Check it quickly

Further reading

Sources

Anycast serves the nearest replica

One address, many announcements

Routing performs the failover

Stateful sessions mind the move

When it breaks

Check it quickly

Further reading

Sources

Circuit breakers stop storming a down service

Name the policy, not just the states

Classify and scope failures

Recovery is a controlled experiment

When it breaks

Check it quickly

Further reading

Sources

Serving stale cache beats downtime

The trade is availability over freshness

How it is signaled

When it breaks

Check it quickly

Further reading

Sources