Availability · XB Field Notes

A PodDisruptionBudget caps how many replicas can drift

Sat, 10 Apr 2027 00:00:00 +0000

A PodDisruptionBudget is a floor, not a ceiling: it names how many replicas of a workload must stay available while something voluntarily takes the rest away. The thing doing the taking is usually kubectl drain, a node autoscaler, or an operator compacting nodes.

The distinction it encodes is between a crash and a decision. A node failing is an involuntary disruption and no budget can stop it; a drain is voluntary and the Eviction API refuses it once the budget is spent. Confusing the two produces the two classic outcomes: a budget that blocks maintenance forever, or a maintenance that takes quorum down. A budget is a statement about a controlled operation, and it is only as good as the health of the replicas behind it.

Only voluntary disruptions are gated

Voluntary disruptions are the ones someone chose: draining a node for repair, scaling a node group down, removing a pod to make room for a bigger one. Involuntary disruptions — hardware failure, a kernel panic, a node lost to a network partition, the kubelet evicting pods under memory pressure — are not gated by a budget at all. The docs are explicit that they still count against it, which is how a crash can silently consume the room your next drain needed.

The gate is the Eviction API, and it is the only gate. Deleting a pod or a Deployment directly bypasses every budget in the namespace, and so do rolling updates. expectedPods comes from the .spec.replicas of the workload that owns the pods, discovered through their ownerReferences, rather than from a count of matching pods — which is why a selector that matches pods from two controllers behaves strangely. A budget also cannot see pods that are already unhealthy: the default IfHealthyBudget policy refuses to evict a running-but-not-Ready pod while the application is disrupted, so a CrashLoopBackOff pod can pin a drain in place. AlwaysAllow exists precisely to break that deadlock.

The arithmetic is one subtraction

minAvailable sets desiredHealthy directly; maxUnavailable sets it to expectedPods minus the allowance. Then disruptionsAllowed is currentHealthy minus desiredHealthy, floored at zero. With three healthy replicas and minAvailable: 2, one eviction is allowed. Change it to minAvailable: 3 and none ever is: a valid manifest that permanently forbids maintenance. Two numbers in the status tell you which case you are in: expectedPods is the denominator and currentHealthy is the numerator, so a wrong denominator is a selector bug rather than a budget bug.

Percentages round up in both directions. minAvailable: "50%" on seven replicas leaves four healthy; maxUnavailable: "30%" on a single replica still allows that replica to go, for 100% unavailability. Only one of the two fields may be set, and the docs recommend maxUnavailable because it tracks replica-count changes: minAvailable: 3 is right at five replicas and wrong at two.

Quorum is a number you can compute

A three-member quorum service tolerates exactly one loss, so minAvailable: 2 or maxUnavailable: 1 encodes that and nothing more. maxUnavailable: 1 keeps working as the cluster grows; minAvailable: 3 stops making sense the moment the StatefulSet scales. A budget of one also means a drain can only ever move one replica at a time, and each move waits for the replacement to become Ready before the next eviction is admitted.

Rolling updates are the awkward neighbour. Pods unavailable during a rolling upgrade count against the budget, but Deployments and StatefulSets are not limited by budgets while rolling. A rolling update can therefore spend the whole allowance and leave a subsequent drain stalled, with the relationship between the two events visible nowhere except the PDB status.

When it breaks

minAvailable equal to the replica count. kubectl get pdb shows ALLOWED DISRUPTIONS 0 forever, every drain of a node holding those pods hangs, and the numbers look right so nobody suspects the budget. It is the single most common PDB misconfiguration, and it is indistinguishable from a healthy one until you try to do maintenance. The tell is that it never moves: a healthy budget’s ALLOWED count changes as pods restart, while a permanent 0 is a budget with no room.

The budget that guards nothing. A selector with a typo, or one matching labels no pod carries, yields expectedPods: 0 and disruptionsAllowed: 0 — it protects nothing and blocks nothing, and you find out when a drain succeeds and takes the workload below quorum. Check that expectedPods equals the replica count you believe you have. The failure is asymmetric: a too-strict budget blocks maintenance loudly, while a budget that matches nothing fails silently at the worst possible moment.

Check it quickly

Compare each budget against the workload it claims to guard:

kubectl get pdb -A -o custom-columns=NAME:.metadata.name,MIN:.spec.minAvailable,MAX:.spec.maxUnavailable,EXPECTED:.status.expectedPods,HEALTHY:.status.currentHealthy,ALLOWED:.status.disruptionsAllowed
kubectl get pdb zk-pdb -o jsonpath='{.spec.selector.matchLabels}{"\n"}'
kubectl get statefulset zookeeper -o jsonpath='{.spec.replicas}{"\n"}'
kubectl get pods -l app=zookeeper --field-selector=status.phase=Running --no-headers | wc -l

EXPECTED must equal the controller’s replica count; if it is 0 the selector matches nothing and the budget is decoration. ALLOWED at 0 while HEALTHY equals EXPECTED means the budget forbids every eviction, and ALLOWED at 0 while HEALTHY is lower means a crash already spent it.

Sources

Configure a PodDisruptionBudget, minAvailable, maxUnavailable, percentage rounding and the zero-eviction case, checked 2026-09-16.
Pod disruptions, voluntary versus involuntary disruption, eviction retries and rolling-update behaviour, checked 2026-09-16.

Pod anti-affinity spreads peers away from each other

Thu, 18 Mar 2027 00:00:00 +0000

Anti-affinity is the scheduler rule that keeps replicas apart. Where affinity says “run me next to these pods”, anti-affinity says “keep me away from them”, and the reason is almost always failure domains: if all three replicas land on one node, one node failure is a full outage.

The tension lives in a single word. required makes the rule a filter, so a pod with nowhere legal to go stays Pending forever; preferred makes it a score, so the same rule degrades gracefully. One YAML shape produces either a deadlock or a soft preference depending on which you typed.

A rule is a selector plus a topology

The rule has two parts. A labelSelector chooses which pods count as peers, and a topologyKey names the node label that defines a domain. kubernetes.io/hostname makes every node its own domain; topology.kubernetes.io/zone groups nodes into zones, so “keep apart” comes to mean “keep to different zones”. The namespaces field defaults to the pod’s own namespace, so anti-affinity never accidentally spans tenants.

Nodes that lack the topologyKey label are not part of any domain. The docs warn that anti-affinity requires consistently labelled nodes and that a missing key leads to unintended behaviour. In practice an unlabelled node is invisible to the rule: neither filtered out nor counted, so it can collect exactly the replicas the rule was meant to spread. One more wrinkle: preferred rules attached to pods that already exist are ignored when a new pod is scheduled. Only the incoming pod’s own rules count, so retrofitting a spread rule onto a running workload does not reshuffle what is already placed.

Required is a cap, not a guarantee

Required anti-affinity is a node filter, so it caps a workload at one replica per domain. Nine replicas across six nodes leaves three Pending forever, no matter how much spare capacity exists. Across three zones, four replicas leave one Pending. Every rejected pod reports FailedScheduling, the autoscaler keeps reporting that it cannot help, and nothing ever errors.

That is why the docs warn against required anti-affinity except where the hard constraint is real, such as one instance per node for a hostPort or a local SSD. They also note that inter-pod affinity and anti-affinity need substantial processing and are not recommended above several hundred nodes, which is a second reason to reach for something cheaper. A required rule is also not retroactive: it filters placement at admission time, so a node label that changes later does not evict pods already running, which is the IgnoredDuringExecution half of the name.

Spread constraints say the same thing more cheaply

topologySpreadConstraints expresses the same intent with an explicit tolerance. maxSkew is the permitted difference between the fullest domain and the global minimum; topologyKey names the domain; whenUnsatisfiable decides whether an unsatisfiable skew is fatal (DoNotSchedule, the default) or merely a scoring penalty (ScheduleAnyway). Nine replicas over three zones with maxSkew: 1 land as three, three, three.

The docs’ worked example is worth internalising: two zones holding two and one matching pods, with maxSkew: 1, forces the next pod into the emptier zone, because placing it in the fuller one makes the skew two. minDomains raises the global minimum so a cluster with fewer zones than expected cannot call itself balanced, and it is only valid with DoNotSchedule. The reason to prefer constraints over anti-affinity is cost as much as clarity: spread counts pods per domain, while anti-affinity evaluates a term against every node. Constraints also compose, so one per topology key can spread across zones and nodes at the same time.

When it breaks

Required anti-affinity with more replicas than domains. The workload scales up to the number of nodes and stops there, every further pod sits Pending, and the horizontal autoscaler keeps asking for more. The same trap appears in a small cluster: a two-node test cluster with three replicas and a required hostname rule can never run the third. Switch to preferred or to a spread constraint with ScheduleAnyway, and keep a hard rule only where one pod per node is genuinely required.

Nodes without the topology label. A node pool that never received topology.kubernetes.io/zone is outside every domain, so it is invisible to the constraint: replicas pile onto the labelled nodes while a third of the cluster sits unused. Verify with kubectl get nodes -L topology.kubernetes.io/zone that every node carries a value.

Check it quickly

Confirm the topology is real before trusting the spread:

kubectl get nodes -L topology.kubernetes.io/zone -L kubernetes.io/hostname
kubectl get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,PHASE:.status.phase
kubectl get events --field-selector reason=FailedScheduling --sort-by=.lastTimestamp | tail -5
kubectl get pods -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort | uniq -c

A blank cell in the first command means that node belongs to no domain. FailedScheduling events naming anti-affinity rules mean the required rule has no eligible node left. The last command shows the real distribution: a count of one per node means required anti-affinity is capping the workload, while uneven counts mean a spread constraint is not being honoured.

Sources

Assigning Pods to Nodes, inter-pod affinity and anti-affinity rules, topology keys and scheduling behaviour, checked 2026-09-16.
Pod Topology Spread Constraints, maxSkew, whenUnsatisfiable, minDomains and the worked skew example, checked 2026-09-16.

Readiness changes routing eligibility; liveness restarts a container

Wed, 13 Jan 2027 00:00:00 +0000

Readiness and liveness answer different operational questions. Readiness asks whether this container should contribute to a Pod that accepts ordinary Service traffic now. Liveness asks whether restarting this container is the configured recovery action. Both can use HTTP, but sharing a protocol does not make their meanings interchangeable. Define the action you want before choosing the condition that triggers it.

Observe the action as well as the response

A readiness failure makes the container unready and affects Pod readiness. For a normal selector-based Service, endpoint reconciliation then changes which backends are eligible for ordinary routing. The Pod can remain Running while it is not Ready, and the application process may continue executing. Restart count and Pod UID answer different questions from the Ready condition.

Liveness failures reaching the configured threshold cause kubelet to terminate the container and apply its restart behavior. In a usual Always-policy application Pod this produces a replacement container inside the same Pod. It does not inherently create a new Pod object or deploy another application version. Compare container restart count and termination state alongside the Pod identity.

Use inexpensive checks with meaningful success criteria. A TCP connection proves that a listener accepts connections, not that an application can complete useful work. An HTTP endpoint can answer a richer question, but can also create load if it recursively checks expensive dependencies. Keep the probe’s own execution cost within the workload’s capacity and timeout budget.

When it breaks

Treating a shared database outage as a liveness failure can repeatedly restart every healthy application process without repairing the database. Adding that dependency to readiness also needs judgment: withdrawing every replica may remove useful fallback responses or overload the remaining replicas during a partial failure. Define readiness from the service contract, including any deliberately supported degraded behavior.

Readiness is not a network access control. A Pod address can remain reachable when the Pod has no ready Service target. Existing connections, other routing layers, and Service options such as publishNotReadyAddresses also require their own analysis. Do not describe a failed readiness probe as an instantaneous guarantee that no request can ever reach the process.

Thresholds and timing control sensitivity. A short timeout can turn CPU contention into apparent failure; several consecutive failures can avoid reacting to one brief delay. Startup protection is separate: a configured startup probe suppresses readiness and liveness checks until it succeeds. These settings are sampling and recovery policy, not a proof of zero lost requests during termination.

Check it quickly

Use a disposable Kubernetes cluster with kubectl access and python:3.11-slim-bookworm available on its nodes. Save the code as probe-actions.py, then run python3 probe-actions.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. The account needs namespace creation and deletion permissions. The recorded server was Kubernetes 1.34.11. The script creates one HTTP Pod and an ordinary Service with default endpoint readiness behavior.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

namespace = 'notes-probes-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 server = '''from http.server import BaseHTTPRequestHandler,HTTPServer
from pathlib import Path
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 flag={'/live':'/tmp/live-fail','/ready':'/tmp/ready-fail'}.get(self.path)
 code=503 if flag and Path(flag).exists() else 200
 body=b'worker'; self.send_response(code); self.send_header('Content-Length',str(len(body)))
 self.end_headers(); self.wfile.write(body)
 def log_message(self,*args): pass
HTTPServer(('0.0.0.0',8080),Handler).serve_forever()
'''
 podspec = spec(server)
 c = podspec['containers'][0]
 c['readinessProbe'] = {'httpGet': {'path': '/ready', 'port': 8080}, 'periodSeconds': 1, 'failureThreshold': 2}
 c['livenessProbe'] = {'httpGet': {'path': '/live', 'port': 8080}, 'periodSeconds': 1, 'failureThreshold': 3}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'worker', 'labels': {'app': 'worker'}}, 'spec': podspec})
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'web'},
 'spec': {'selector': {'app': 'worker'}, 'ports': [{'port': 80, 'targetPort': 8080}]}})
 def targets():
 return {e['targetRef']['uid'] for s in obj(namespace, 'endpointslices', selector='kubernetes.io/service-name=web')['items']
 for e in s.get('endpoints', []) if e.get('conditions', {}).get('ready') and e.get('targetRef')}
 initial = wait_for(lambda: obj(namespace, 'pod', 'worker'), pod_ready)
 uid = initial['metadata']['uid']
 wait_for(targets, lambda value: value == {uid})
 def change(path, create):
 method = 'touch' if create else 'unlink'
 k('exec', '-n', namespace, 'worker', '--', 'python3', '-c',
 'from pathlib import Path; Path('+repr(path)+').'+method+'()')
 change('/tmp/ready-fail', True)
 wait_for(lambda: obj(namespace, 'pod', 'worker'), lambda p: not pod_ready(p))
 wait_for(targets, lambda value: not value)
 time.sleep(4)
 unready = obj(namespace, 'pod', 'worker')
 assert unready['metadata']['uid'] == uid and unready['status']['containerStatuses'][0]['restartCount'] == 0
 address = 'http://' + unready['status']['podIP'] + ':8080/api'
 probe = 'from urllib.request import urlopen; print(urlopen('+repr(address)+',timeout=3).read().decode())'
 assert k('exec', '-n', namespace, 'worker', '--', 'python3', '-c', probe).stdout.strip() == 'worker'
 change('/tmp/ready-fail', False)
 wait_for(targets, lambda value: value == {uid})
 change('/tmp/live-fail', True)
 restarted = wait_for(lambda: obj(namespace, 'pod', 'worker'), lambda p:
 pod_ready(p) and p['status']['containerStatuses'][0]['restartCount'] >= 1)
 assert restarted['metadata']['uid'] == uid
 assert restarted['status']['containerStatuses'][0]['restartCount'] == 1
 wait_for(targets, lambda value: value == {uid})
 print(json.dumps({'readiness_removed_ready_target_without_restart': True,
 'unready_pod_direct_http_still_succeeds': True,
 'readiness_recovery_restores_target': True,
 'liveness_container_restarts': 1, 'same_pod_uid': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The initial Pod is Ready and its UID appears in the Service’s ready EndpointSlice targets. Creating a local flag makes only the readiness endpoint return 503. After the failure threshold and reconciliation, the ready-target set is empty. Four seconds later the Pod UID is unchanged and the container restart count remains zero.

A direct HTTP request to that unready Pod’s address still returns worker. Removing the readiness flag restores its ready target. The script then creates a different flag that makes the liveness endpoint fail. Kubelet restarts the container once, its fresh writable layer removes the flag, and the ready target returns. The Pod UID remains the same throughout.

All assertions passed and the namespace was removed. This demonstrates distinct controller actions and direct reachability, not graceful draining of existing connections or production availability. For a real rollout, verify endpoint propagation, application shutdown, outstanding work and client behavior together; a green probe alone cannot establish the whole request lifecycle.

Sources

Kubernetes: Configure Probes, failure thresholds and distinct readiness and liveness actions, checked 2026-09-13.
Kubernetes: Pod Lifecycle, container restarts and Pod readiness, checked 2026-09-13.
Kubernetes: EndpointSlices, backend readiness conditions and discovery behavior, checked 2026-09-13.

Passive upstream health follows failed traffic, not a probe timer

Tue, 05 Jan 2027 00:00:00 +0000

nginx passive upstream health uses outcomes of real requests to decide whether a peer should temporarily stop receiving traffic. max_fails sets an unsuccessful-attempt threshold, while fail_timeout supplies both the counting interval and the unavailability period. These settings do not schedule periodic application probes. Active checks require a supported checking capability and configuration beyond this passive request-outcome policy.

When it breaks

Not every error-looking HTTP response counts as an unsuccessful upstream attempt. The proxy retry configuration helps define that classification. In the tested setup, HTTP 503 only contributes to passive failure handling when http_503 is included. Treating every 5xx as an automatic health failure can leave an operator expecting exclusion while nginx continues selecting the peer according to its actual rules.

Another mistake is treating the timeout window as proof of recovery. When the unavailable period ends, a peer can become eligible for real traffic again without a successful independent health probe first. A still-broken backend can make that request fail. least_conn changes selection based on active connections; it is not a gradual recovery ramp or a replacement for a supported slow-start mechanism.

Retries have boundaries as well. Switching peers cannot transparently repair an error after response data has already been sent to the client. Replaying an operation with side effects can create duplicates. The following fixture uses GET requests and failures returned before a successful body, with an explicit attempt limit; it does not establish safe retry semantics for every application method.

Check it quickly

Save this as check-passive-health.py and run it with Python, Docker, and the pinned images. The recorded server is nginx 1.28.0 with one worker. Two HTTP endpoints in a disposable container report their identities and can be instructed to return 503. A separate control listener exposes counts without generating requests through the balanced groups:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None,extra_ports=()):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True)
 target.write_bytes(content if isinstance(content,bytes) else content.encode())
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 published=[arg for value in (8080,*extra_ports) for arg in ('-p','127.0.0.1::'+str(value))]
 docker('run','-d','--name',containers[0],'--network',name,*published,tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 request.port=port;request.network=name
 request.ports={int(key.split('/')[0]):int(value[0]['HostPort']) for key,value in details['NetworkSettings']['Ports'].items() if value}
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'
backend=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
from urllib.parse import urlsplit,parse_qs
import itertools,json,threading
lock=threading.Lock();ids=itertools.count(1);counts={'A':0,'B':0};active={'A':0,'B':0};faults={'A':200,'B':200};gates={}
class Handler(BaseHTTPRequestHandler):
 protocol_version='HTTP/1.1'
 def setup(self):super().setup();self.ident=next(ids)
 def log_message(self,*args):pass
 def do_GET(self):
 parsed=urlsplit(self.path);label=self.server.label;code=200
 if label=='control':
 query=parse_qs(parsed.query)
 with lock:
 if parsed.path=='/fail':faults[query['peer'][0]]=int(query['status'][0])
 if parsed.path=='/release':gates[query['path'][0]].set()
 data={'counts':dict(counts),'active':dict(active),'faults':dict(faults),'held':list(gates)}
 else:
 with lock:
 counts[label]+=1;active[label]+=1;code=faults[label]
 if parsed.path.startswith('/hold/'):gates[parsed.path]=threading.Event()
 if parsed.path.startswith('/hold/'):assert gates[parsed.path].wait(20)
 data={'peer':label,'path':self.path,'connection':self.ident}
 body=json.dumps(data,sort_keys=True).encode();self.send_response(code)
 self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(body)))
 self.end_headers();self.wfile.write(body)
 if label!='control':
 with lock:active[label]-=1
for port,label in [(8081,'A'),(8082,'B'),(8083,'control')]:
 server=ThreadingHTTPServer(('0.0.0.0',port),Handler);server.label=label
 threading.Thread(target=server.serve_forever,daemon=True).start()
threading.Event().wait()
'''
extra='''
upstream passive { server backend:8081 max_fails=1 fail_timeout=2s; server backend:8082; }
upstream plain { server backend:8081 max_fails=1 fail_timeout=30s; server backend:8082; }
upstream single { server backend:8081 max_fails=1 fail_timeout=30s; }
add_header X-Upstream $upstream_addr always; add_header X-Attempts $upstream_status always;
'''
locations='''
location /passive { proxy_next_upstream error timeout http_503; proxy_next_upstream_tries 2; proxy_pass http://passive; }
location /plain { proxy_next_upstream error timeout; proxy_pass http://plain; }
location /single { proxy_next_upstream error timeout http_503; proxy_pass http://single; }
location /control/ { proxy_pass http://backend:8083/; }
'''
active_config=config('location / { proxy_pass http://peers; health_check; }',
 'upstream peers { zone peers 64k; server backend:8081; server backend:8082; }')
with lab(config(locations,extra),{'active.conf':active_config},backend) as (proxy,request):
 def control(path):
 status,_,body=request('/control/'+path);assert status==200;return json.loads(body)
 def call(path):
 status,h,b=request(path);return {'status':status,'attempts':h['x-attempts'],'upstream':h['x-upstream'],'peer':json.loads(b)['peer']}
 control('fail?peer=A&status=503');results={}
 plain=[call('/plain') for _ in range(6)]
 assert [r['status'] for r in plain].count(503)==3 and all(',' not in r['attempts'] for r in plain)
 results['unlisted_http_503']=plain
 for _ in range(2):
 first=call('/passive')
 if first['attempts']=='503, 200':break
 assert first['status']==200 and first['attempts']=='503, 200' and first['peer']=='B',first
 results['failed_attempt_then_success']=first
 after_failure=control('state')['counts']['A']
 excluded=[call('/passive') for _ in range(4)]
 assert all(r['peer']=='B' and r['attempts']=='200' for r in excluded)
 assert control('state')['counts']['A']==after_failure
 time.sleep(.4);assert control('state')['counts']['A']==after_failure
 results['during_exclusion']=excluded;results['background_probe_count']=0
 control('fail?peer=A&status=200');deadline=time.monotonic()+6;recovered=None
 while time.monotonic()<deadline:
 observed=call('/passive')
 if observed['peer']=='A':recovered=observed;break
 time.sleep(.1)
 assert recovered and recovered['status']==200;results['live_request_after_window']=recovered
 control('fail?peer=A&status=503');before=control('state')['counts']['A']
 single=[call('/single') for _ in range(2)]
 assert all(r['status']==503 and r['attempts']=='503' for r in single)
 assert control('state')['counts']['A']==before+2;results['single_peer_still_attempted']=single
 probe=docker('exec',proxy,'nginx','-t','-c','/srv/active.conf',check=False)
 assert probe.returncode!=0 and 'unknown directive "health_check"' in probe.stderr
 results['active_health_check_directive_supported']=False
 print(json.dumps(results,sort_keys=True))

First, peer A returns actual HTTP 503 while B returns 200. A comparison group retries only connection errors and timeouts. Six requests still select A three times and B three times, with no multi-attempt response. A’s max_fails=1 does not remove it for that unlisted HTTP outcome. The fixture inspects statuses and selected upstream addresses, not only the final response body.

The passive group includes http_503, allows two tries, and uses max_fails=1 fail_timeout=2s for A. A request encounters A’s 503, then succeeds on B with final 200. nginx reports upstream statuses 503, 200 and both addresses. Those values are real backend HTTP responses in this experiment, not synthetic labels inferred from a connection refusal.

Four subsequent requests use only B during A’s exclusion period. A’s request count remains unchanged, including through a controlled pause, demonstrating that this configuration is not sending background health probes. The backend is then made healthy. After the configured window, a live request reaches A and succeeds; the test waits for that observed selection instead of assuming an exact clock boundary.

A separate group contains only A. Even with max_fails=1 fail_timeout=30s, two requests still reach that failing peer and return 503. nginx ignores those passive-unavailability parameters for a single-server group. There is no alternative peer to select, so the multi-peer exclusion behavior cannot be assumed to create availability where the topology provides none.

Finally, a disposable candidate configuration includes health_check. The tested binary rejects the directive as unknown. This verifies the installed capability boundary directly: the passive settings did not enable active checking. The valid running configuration is retained; no unverified module or commercial capability is represented as available merely because it appears in general documentation.

Design failure detection with the service’s response contract

Choose which outcomes indicate an unusable peer and which are legitimate application responses. An authorization failure or missing object need not mean the server is unhealthy. Passive detection also depends on receiving traffic, so a quiet but broken peer may remain untested until a user request arrives. Active checking, where available, answers a different timing question.

Define a meaningful health endpoint if deploying active checks, and understand which dependencies it exercises. A TCP accept alone does not prove a useful application response. Conversely, an overly broad dependency check can remove every peer for a shared downstream problem. Keep the check’s success condition aligned with the traffic that removing a peer would actually protect.

Observe final client status together with attempt statuses, peer addresses, latency, and exclusion state. A final 200 can conceal a failed first attempt and extra delay. Verify the deployed version’s capabilities, retry limits, and recovery path so the availability claim matches what clients experience rather than what the word “health” suggests.

Sources

nginx upstream module, max_fails, both fail_timeout meanings, single-peer behavior, and slow start, checked 2026-09-12.
nginx proxy module, unsuccessful-attempt classification, retry conditions, and retry limits, checked 2026-09-12.
nginx HTTP health check guide, passive versus active checks and product capabilities, checked 2026-09-12.