nginx passive upstream health uses outcomes of real requests to decide whether a peer should temporarily stop receiving traffic. max_fails sets an unsuccessful-attempt threshold, while fail_timeout supplies both the counting interval and the unavailability period. These settings do not schedule periodic application probes. Active checks require a supported checking capability and configuration beyond this passive request-outcome policy.

When it breaks

Not every error-looking HTTP response counts as an unsuccessful upstream attempt. The proxy retry configuration helps define that classification. In the tested setup, HTTP 503 only contributes to passive failure handling when http_503 is included. Treating every 5xx as an automatic health failure can leave an operator expecting exclusion while nginx continues selecting the peer according to its actual rules.

Another mistake is treating the timeout window as proof of recovery. When the unavailable period ends, a peer can become eligible for real traffic again without a successful independent health probe first. A still-broken backend can make that request fail. least_conn changes selection based on active connections; it is not a gradual recovery ramp or a replacement for a supported slow-start mechanism.

Retries have boundaries as well. Switching peers cannot transparently repair an error after response data has already been sent to the client. Replaying an operation with side effects can create duplicates. The following fixture uses GET requests and failures returned before a successful body, with an explicit attempt limit; it does not establish safe retry semantics for every application method.

Check it quickly

Save this as check-passive-health.py and run it with Python, Docker, and the pinned images. The recorded server is nginx 1.28.0 with one worker. Two HTTP endpoints in a disposable container report their identities and can be instructed to return 503. A separate control listener exposes counts without generating requests through the balanced groups:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
    result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
    if check:assert result.returncode==0,result.stderr
    return result
def wait(predicate):
    deadline=time.monotonic()+15
    while time.monotonic()<deadline:
        try:
            if predicate():return
        except OSError:pass
        time.sleep(.1)
    raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None,extra_ports=()):
    name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
    try:
        with TemporaryDirectory(prefix=name+'-') as directory:
            root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
            for path,content in (files or {}).items():
                target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True)
                target.write_bytes(content if isinstance(content,bytes) else content.encode())
            docker('network','create',name)
            if backend is not None:
                (root/'backend.py').write_text(backend)
                (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
                docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
                docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
                wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
            (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
            docker('build','--network','none','-t',tags[0],str(root))
            docker('run','--rm','--network',name,tags[0],'nginx','-t')
            published=[arg for value in (8080,*extra_ports) for arg in ('-p','127.0.0.1::'+str(value))]
            docker('run','-d','--name',containers[0],'--network',name,*published,tags[0])
            details=json.loads(docker('inspect',containers[0]).stdout)[0]
            binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
            def ready():
                connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
            wait(ready)
            def request(path,method='GET',headers=None,body=None):
                connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
                try:
                    connection.request(method,path,body=body,headers=headers or {})
                    response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
                finally:connection.close()
            request.port=port;request.network=name
            request.ports={int(key.split('/')[0]):int(value[0]['HostPort']) for key,value in details['NetworkSettings']['Ports'].items() if value}
            yield containers[0],request
    finally:
        docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
    return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'
backend=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
from urllib.parse import urlsplit,parse_qs
import itertools,json,threading
lock=threading.Lock();ids=itertools.count(1);counts={'A':0,'B':0};active={'A':0,'B':0};faults={'A':200,'B':200};gates={}
class Handler(BaseHTTPRequestHandler):
    protocol_version='HTTP/1.1'
    def setup(self):super().setup();self.ident=next(ids)
    def log_message(self,*args):pass
    def do_GET(self):
        parsed=urlsplit(self.path);label=self.server.label;code=200
        if label=='control':
            query=parse_qs(parsed.query)
            with lock:
                if parsed.path=='/fail':faults[query['peer'][0]]=int(query['status'][0])
                if parsed.path=='/release':gates[query['path'][0]].set()
                data={'counts':dict(counts),'active':dict(active),'faults':dict(faults),'held':list(gates)}
        else:
            with lock:
                counts[label]+=1;active[label]+=1;code=faults[label]
                if parsed.path.startswith('/hold/'):gates[parsed.path]=threading.Event()
            if parsed.path.startswith('/hold/'):assert gates[parsed.path].wait(20)
            data={'peer':label,'path':self.path,'connection':self.ident}
        body=json.dumps(data,sort_keys=True).encode();self.send_response(code)
        self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(body)))
        self.end_headers();self.wfile.write(body)
        if label!='control':
            with lock:active[label]-=1
for port,label in [(8081,'A'),(8082,'B'),(8083,'control')]:
    server=ThreadingHTTPServer(('0.0.0.0',port),Handler);server.label=label
    threading.Thread(target=server.serve_forever,daemon=True).start()
threading.Event().wait()
'''
extra='''
upstream passive { server backend:8081 max_fails=1 fail_timeout=2s; server backend:8082; }
upstream plain { server backend:8081 max_fails=1 fail_timeout=30s; server backend:8082; }
upstream single { server backend:8081 max_fails=1 fail_timeout=30s; }
add_header X-Upstream $upstream_addr always; add_header X-Attempts $upstream_status always;
'''
locations='''
location /passive { proxy_next_upstream error timeout http_503; proxy_next_upstream_tries 2; proxy_pass http://passive; }
location /plain { proxy_next_upstream error timeout; proxy_pass http://plain; }
location /single { proxy_next_upstream error timeout http_503; proxy_pass http://single; }
location /control/ { proxy_pass http://backend:8083/; }
'''
active_config=config('location / { proxy_pass http://peers; health_check; }',
    'upstream peers { zone peers 64k; server backend:8081; server backend:8082; }')
with lab(config(locations,extra),{'active.conf':active_config},backend) as (proxy,request):
    def control(path):
        status,_,body=request('/control/'+path);assert status==200;return json.loads(body)
    def call(path):
        status,h,b=request(path);return {'status':status,'attempts':h['x-attempts'],'upstream':h['x-upstream'],'peer':json.loads(b)['peer']}
    control('fail?peer=A&status=503');results={}
    plain=[call('/plain') for _ in range(6)]
    assert [r['status'] for r in plain].count(503)==3 and all(',' not in r['attempts'] for r in plain)
    results['unlisted_http_503']=plain
    for _ in range(2):
        first=call('/passive')
        if first['attempts']=='503, 200':break
    assert first['status']==200 and first['attempts']=='503, 200' and first['peer']=='B',first
    results['failed_attempt_then_success']=first
    after_failure=control('state')['counts']['A']
    excluded=[call('/passive') for _ in range(4)]
    assert all(r['peer']=='B' and r['attempts']=='200' for r in excluded)
    assert control('state')['counts']['A']==after_failure
    time.sleep(.4);assert control('state')['counts']['A']==after_failure
    results['during_exclusion']=excluded;results['background_probe_count']=0
    control('fail?peer=A&status=200');deadline=time.monotonic()+6;recovered=None
    while time.monotonic()<deadline:
        observed=call('/passive')
        if observed['peer']=='A':recovered=observed;break
        time.sleep(.1)
    assert recovered and recovered['status']==200;results['live_request_after_window']=recovered
    control('fail?peer=A&status=503');before=control('state')['counts']['A']
    single=[call('/single') for _ in range(2)]
    assert all(r['status']==503 and r['attempts']=='503' for r in single)
    assert control('state')['counts']['A']==before+2;results['single_peer_still_attempted']=single
    probe=docker('exec',proxy,'nginx','-t','-c','/srv/active.conf',check=False)
    assert probe.returncode!=0 and 'unknown directive "health_check"' in probe.stderr
    results['active_health_check_directive_supported']=False
    print(json.dumps(results,sort_keys=True))

First, peer A returns actual HTTP 503 while B returns 200. A comparison group retries only connection errors and timeouts. Six requests still select A three times and B three times, with no multi-attempt response. A’s max_fails=1 does not remove it for that unlisted HTTP outcome. The fixture inspects statuses and selected upstream addresses, not only the final response body.

The passive group includes http_503, allows two tries, and uses max_fails=1 fail_timeout=2s for A. A request encounters A’s 503, then succeeds on B with final 200. nginx reports upstream statuses 503, 200 and both addresses. Those values are real backend HTTP responses in this experiment, not synthetic labels inferred from a connection refusal.

Four subsequent requests use only B during A’s exclusion period. A’s request count remains unchanged, including through a controlled pause, demonstrating that this configuration is not sending background health probes. The backend is then made healthy. After the configured window, a live request reaches A and succeeds; the test waits for that observed selection instead of assuming an exact clock boundary.

A separate group contains only A. Even with max_fails=1 fail_timeout=30s, two requests still reach that failing peer and return 503. nginx ignores those passive-unavailability parameters for a single-server group. There is no alternative peer to select, so the multi-peer exclusion behavior cannot be assumed to create availability where the topology provides none.

Finally, a disposable candidate configuration includes health_check. The tested binary rejects the directive as unknown. This verifies the installed capability boundary directly: the passive settings did not enable active checking. The valid running configuration is retained; no unverified module or commercial capability is represented as available merely because it appears in general documentation.

Design failure detection with the service’s response contract

Choose which outcomes indicate an unusable peer and which are legitimate application responses. An authorization failure or missing object need not mean the server is unhealthy. Passive detection also depends on receiving traffic, so a quiet but broken peer may remain untested until a user request arrives. Active checking, where available, answers a different timing question.

Define a meaningful health endpoint if deploying active checks, and understand which dependencies it exercises. A TCP accept alone does not prove a useful application response. Conversely, an overly broad dependency check can remove every peer for a shared downstream problem. Keep the check’s success condition aligned with the traffic that removing a peer would actually protect.

Observe final client status together with attempt statuses, peer addresses, latency, and exclusion state. A final 200 can conceal a failed first attempt and extra delay. Verify the deployed version’s capabilities, retry limits, and recovery path so the availability claim matches what clients experience rather than what the word “health” suggests.

Sources