Transport · XB Field Notes

TIME-WAIT protects a connection incarnation, not exclusive ownership of a port

Fri, 13 Nov 2026 00:00:00 +0000

TIME-WAIT is a TCP state retained after certain orderly connection closures. It gives the closing exchange time to complete if the final acknowledgment must be repeated and helps prevent delayed segments from an earlier connection incarnation being confused with a later one. The relevant identity includes both endpoint addresses and ports. A TIME-WAIT entry is not simply a universal lock on one local port number.

When it breaks

In an ordinary close, the endpoint initiating the close commonly enters TIME-WAIT after the peer’s FIN is acknowledged. That endpoint can be a client or a server; simultaneous close can involve both. Assuming TIME-WAIT always belongs on the server hides the workload actually creating connection churn. Inspect the state and endpoint tuple instead of inferring ownership from application roles.

High churn can constrain new connections to a destination, but a large TIME-WAIT count alone does not prove exhaustion or a leaked application file descriptor. The operating system can retain protocol state after the application closes its socket. Source-address choices, available ports, destination diversity, explicit binding, reuse options, and kernel behavior affect the practical limit. Measure connection failures alongside counts before diagnosing resource exhaustion.

Confusing TIME-WAIT with FIN-WAIT-2 leads to ineffective tuning. Linux’s tcp_fin_timeout controls a different orphaned-socket situation; it is not a general TIME-WAIT duration knob. Old advice about tcp_tw_recycle is also obsolete on current Linux. Reuse behavior has protocol-safety conditions, including timestamp-related checks, so changing a setting without understanding the actual socket path can produce misleading expectations.

Check it quickly

Save this as check-time-wait.py and run python3 check-time-wait.py with Docker. The disposable environment uses Python 3.11.16 and iproute2 6.1.0, tested on Linux 6.12.76. NET_ADMIN and SYS_ADMIN permit nested namespaces and their temporary proc mount. A new veth joins the client and target; the test changes only these namespaces, with no host service or interface involved:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,('child exited without a response',process.stderr.read())
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found


with lab(['client','target']) as (root,stack,setting):
 wire('client','c0','target','t0',['192.0.2.1/30'],['192.0.2.2/30'])
 setting('client','net.ipv4.tcp_tw_reuse',0)
 setting('client','net.ipv4.tcp_timestamps',0);setting('target','net.ipv4.tcp_timestamps',0)
 server_code=r'''
import socketserver,threading
class Handler(socketserver.BaseRequestHandler):
 def handle(self):
 while self.request.recv(4096):pass
class Server(socketserver.ThreadingTCPServer):allow_reuse_address=True;daemon_threads=True
servers=[Server(('192.0.2.2',port),Handler) for port in [18080,18081]]
for server in servers:threading.Thread(target=server.serve_forever,daemon=True).start()
print('ready',flush=True);threading.Event().wait()
'''
 service=spawn(stack,['ip','netns','exec','target','python3','-u','-c',server_code]);assert line(service).strip()=='ready'
 client_code=r'''
import json,socket
def exchange(destination):
 with socket.socket() as s:
 s.settimeout(2);s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1);s.bind(('192.0.2.1',40200))
 s.connect(('192.0.2.2',destination));s.sendall(b'closed cleanly');s.shutdown(socket.SHUT_WR)
 assert s.recv(4096)==b''
exchange(18080)
try:exchange(18080);same={'ok':True}
except OSError as error:same={'ok':False,'errno':error.errno,'type':type(error).__name__}
assert not same['ok'] and same['errno']==99,same
exchange(18081)
print(json.dumps({'same_tuple':same,'same_source_port_new_destination_port':True}))
'''
 result=json.loads(ns('client','python3','-c',client_code).stdout)
 rows=ns('client','ss','-nHto','state','time-wait').stdout
 assert '192.0.2.1:40200' in rows and '192.0.2.2:18080' in rows and '192.0.2.2:18081' in rows,rows
 target=ns('target','ss','-nHto','state','time-wait').stdout;assert ':40200' not in target,target
 print(json.dumps({'client_active_close':result,'client_time_wait':rows.strip(),'target_time_wait':target.strip(),
 'timestamps_disabled_in_both_namespaces':True,'client_tcp_tw_reuse':ns('client','sysctl','-n','net.ipv4.tcp_tw_reuse').stdout.strip()}))
"""
name='xbcatm-tcpflow-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

To make an immediate duplicate-tuple rejection reproducible, the fixture disables timestamps in both namespaces and sets client tcp_tw_reuse to zero. These are explicit experiment controls, not a proposed production tuning profile. Timestamp-enabled sockets can take different safe-reuse paths. The matching kernel source contains timestamp-dependent checks, which is why the test does not claim that one reuse setting determines every possible outcome.

The target listens on ports 18080 and 18081. The client sets SO_REUSEADDR before binding its source port to 40200, connects to port 18080, sends data, and shuts down its writing direction first. The target drains the stream and closes, and the client reads EOF before closing its descriptor. This establishes the client as the active closer through real protocol traffic.

Immediately reconnecting from that same source address and port to the same destination address and port fails with Linux EADDRNOTAVAIL, errno 99, under the configured controls. Changing only the destination port to 18081 succeeds while retaining source port 40200. Both sockets use SO_REUSEADDR; the comparison therefore tests distinct connection tuples without changing the binding option between attempts.

The client’s socket listing then shows two TIME-WAIT entries sharing local port 40200, one for each target port. The target has no corresponding TIME-WAIT entries in this run. Each client timer is approximately sixty seconds when observed. The fixture records those timers; it does not wait for expiry or claim that this measured Linux duration is a universal TCP constant.

Reduce unnecessary churn before changing reuse policy

Where the application protocol allows it, connection reuse can reduce repeated setup and teardown. A pool still needs bounded size, healthy connection handling, and appropriate deadlines; retaining a dead connection is not a remedy for churn. Check whether the application actively closes after every small operation or whether a proxy timeout or server policy forces repeated reconnects.

When investigating a limit, record the failing system call and error, source and destination tuples, active connection counts, and TIME-WAIT distribution. A bind failure, a connect failure, and an application timeout are different observations. This example deliberately binds a source port; an application that lets the kernel choose an ephemeral port follows a different allocation path and needs its own representative test.

Keep the protocol’s delayed-segment and closing-handshake protections in the diagnosis. The useful objective is reliable new connections at the required workload, not eliminating a state that TCP intentionally retains. Validate any proposed change against the deployed kernel, socket options, and traffic pattern, and verify successful application exchanges as well as the resulting connection-state counts.

Sources

RFC 9293 TCP, connection termination, TIME-WAIT and delayed segments, checked 2026-09-12.
Linux tcp(7), socket reuse controls and the separate FIN-WAIT-2 timeout, checked 2026-09-12.
Linux 6.12.76 TCP IPv4 implementation, timestamp-dependent checks when reusing a TIME-WAIT connection, checked 2026-09-12.

TCP congestion control belongs to each sender, and peers can use different algorithms

Thu, 12 Nov 2026 00:00:00 +0000

TCP congestion control limits how a sender injects data into a network that has finite shared capacity. Its congestion window constrains outstanding data, while pacing, when used, controls how transmissions are distributed over time. The receiver’s advertised window imposes a separate flow-control limit. These mechanisms interact, but they describe different constraints and should not be collapsed into one ambiguous sending-speed setting.

When it breaks

The two endpoints do not need to select an identical congestion-control algorithm. Each endpoint controls the traffic it sends, so a Reno client can communicate with a CUBIC server. TCP setup negotiates transport options, not a requirement that the two sender algorithms have matching names. Diagnosing different controller names as a protocol mismatch can distract from a real route, buffering, or application failure.

Algorithm availability is also an actual runtime property. A configuration string naming a controller does not prove the kernel supports it or permits the calling process to select it. On Linux, TCP_CONGESTION sets a per-socket choice, subject to available and allowed algorithms. Check the result and read the selected name back instead of silently assuming that a requested label became effective.

Loss recovery is not one universal curve. A retransmission timeout, fast recovery, explicit congestion notification, and model-based estimation can produce different responses depending on the controller. Loss may come from congestion or another cause; the sender still has to respond under its algorithm’s rules. Increasing buffers cannot automatically fix a sender that is repeatedly losing packets or waiting for acknowledgment.

Check it quickly

Save this as check-congestion-control.py and run python3 check-congestion-control.py with Docker. The disposable environment uses Python 3.11.16, iproute2 6.1.0, and nftables 1.0.6, tested on Linux 6.12.76. NET_ADMIN and SYS_ADMIN support nested namespaces. A new veth link receives ten milliseconds of added egress delay in each direction; no host interface or production flow is modified:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,('child exited without a response',process.stderr.read())
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found


info_code=r'''
import struct
def tcp_info(stream):
 raw=stream.getsockopt(socket.IPPROTO_TCP,socket.TCP_INFO,104);assert len(raw)==104
 offsets={'snd_mss':16,'unacked':24,'rtt_us':68,'snd_ssthresh':76,'snd_cwnd':80,'total_retrans':100}
 return {'ca_state':raw[1],**{key:struct.unpack_from('=I',raw,offset)[0] for key,offset in offsets.items()}}
'''
with lab(['client','target']) as (root,stack,setting):
 wire('client','c0','target','t0',['192.0.2.1/30'],['192.0.2.2/30'])
 available=ns('client','sysctl','-n','net.ipv4.tcp_available_congestion_control').stdout.split();assert {'reno','cubic'}<=set(available),available
 for name,interface in [('client','c0'),('target','t0')]:
 ns(name,'tc','qdisc','add','dev',interface,'root','netem','delay','10ms','limit','10000')
 server_code=r'''
import socket,json,hashlib
listener=socket.socket();listener.setsockopt(socket.IPPROTO_TCP,socket.TCP_CONGESTION,b'cubic')
listener.bind(('192.0.2.2',18080));listener.listen();print('listening',flush=True)
with listener.accept()[0] as connection:
 connection.settimeout(8)
 print(json.dumps({'algorithm':connection.getsockopt(socket.IPPROTO_TCP,socket.TCP_CONGESTION,16).split(b'\0')[0].decode()}),flush=True)
 for size in [65536,16384]:
 data=b''
 while len(data)<size:
 chunk=connection.recv(size-len(data));assert chunk;data+=chunk
 assert data==b'x'*size
 connection.sendall((hashlib.sha256(data).hexdigest()+'\n').encode())
'''
 service=spawn(stack,['ip','netns','exec','target','python3','-u','-c',server_code]);assert line(service).strip()=='listening'
 client_code=info_code+r'''
import socket,json,sys,time,hashlib
with socket.socket() as unsupported:
 try:unsupported.setsockopt(socket.IPPROTO_TCP,socket.TCP_CONGESTION,b'xbcatm_absent');raise AssertionError('unexpected available algorithm')
 except OSError as error:selection_failure={'errno':error.errno,'type':type(error).__name__}
assert selection_failure['errno']==2,selection_failure
stream=socket.socket();stream.settimeout(8);stream.setsockopt(socket.IPPROTO_TCP,socket.TCP_CONGESTION,b'reno')
stream.bind(('192.0.2.1',40600));stream.connect(('192.0.2.2',18080))
with stream.makefile('rb') as incoming:
 stream.sendall(b'x'*65536);assert incoming.readline().decode().strip()==hashlib.sha256(b'x'*65536).hexdigest()
 baseline=tcp_info(stream);assert baseline['snd_cwnd']>1
 algorithm=stream.getsockopt(socket.IPPROTO_TCP,socket.TCP_CONGESTION,16).split(b'\0')[0].decode()
 print(json.dumps({'baseline':baseline,'algorithm':algorithm,'unavailable_algorithm':selection_failure}),flush=True)
 assert sys.stdin.readline().strip()=='drop';stream.sendall(b'x'*16384)
 deadline=time.monotonic()+4;samples=[]
 while time.monotonic()<deadline:
 state=tcp_info(stream);samples.append(state)
 if state['total_retrans']>baseline['total_retrans'] and state['snd_cwnd']==1 and state['ca_state']==4:break
 time.sleep(.01)
 else:raise AssertionError(('no timeout recovery observed',samples))
 print(json.dumps({'during_blackout':state}),flush=True)
 assert sys.stdin.readline().strip()=='restore'
 assert incoming.readline().decode().strip()==hashlib.sha256(b'x'*16384).hexdigest()
 print(json.dumps({'after_restore':tcp_info(stream),'verified_bytes':65536+16384}),flush=True)
stream.close()
'''
 client=spawn(stack,['ip','netns','exec','client','python3','-u','-c',client_code])
 baseline=json.loads(line(client));server_result=json.loads(line(service));assert baseline['algorithm']=='reno' and server_result['algorithm']=='cubic'
 nft('target','''table ip blackout {
 chain input { type filter hook input priority 0; policy accept;
 tcp sport 40600 tcp dport 18080 ip length > 60 counter drop comment "data_dropped"
 }
}
''')
 client.stdin.write('drop\n');client.stdin.flush();loss=json.loads(line(client,5))
 count=counters('target');assert count['data_dropped']>0
 nft('target','delete table ip blackout');client.stdin.write('restore\n');client.stdin.flush()
 recovery=json.loads(line(client,8));client.wait(timeout=2);service.wait(timeout=2);assert client.returncode==service.returncode==0
 print(json.dumps({'available':available,'client':baseline,'server':server_result,'loss':loss,'drop_counter':count,'recovery':recovery}))
"""
name='xbcatm-tcpflow-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The script first verifies that Reno and CUBIC are available. A deliberate request for an absent controller fails with ENOENT, errno two, surfaced by Python as FileNotFoundError. That exception refers to the unavailable controller selection, not a missing application data file. The real client explicitly selects Reno, while the server listener selects CUBIC and its accepted socket reports that inherited choice.

A first transfer sends 65536 bytes successfully and verifies the server’s digest. Both endpoints report their different algorithms while communication works. The client’s recorded baseline congestion window is fifty-six segments, with no retransmissions and no outstanding data at the observation point. This demonstrates that matching controller names are unnecessary and establishes a healthy baseline before inducing loss.

The target then installs an input rule dropping data for this connection. The client sends another 16384 bytes and samples TCP_INFO while acknowledgments fail to arrive. The kernel enters its loss state after a retransmission timeout: this run shows the congestion window reduced to one segment, the slow-start threshold set to twenty-eight, and the retransmission counter increasing. The target rule counter confirms actual packet drops.

Removing the rule allows the same connection to recover. The client verifies the second payload’s digest, for 81920 bytes delivered across both phases. The final window has grown again, and the retransmission total reflects recovery work. These are observations of this Reno timeout scenario, not a promise that every controller halves its threshold or returns to one segment for every kind of loss.

Measure the constraint that matters to the workload

The experiment samples Linux’s documented TCP_INFO layout and records MSS, congestion window, threshold, RTT, outstanding segments, and retransmissions. Window values are segment counts in these fields. A snapshot alone does not establish throughput: the application may have nothing to send, the receiver may limit progress, or queued data may still await transmission. Keep these distinctions when interpreting socket diagnostics.

For operational changes, record the effective controller on representative sockets and compare latency, throughput, loss, and fairness under the relevant workload. A single successful connection cannot certify behavior with competing senders or a different bottleneck. Likewise, this isolated loss rule is a controlled fault, not an estimate of ordinary internet packet-loss rates.

Use transport recovery evidence alongside application correctness. Retransmission can preserve the byte stream while increasing completion time, so a valid response does not mean the network was loss-free. Conversely, a lower window during recovery is not itself a broken connection. The useful outcome is verified delivery and an explanation of the sender’s measured response to the conditions it encountered.

Sources

RFC 5681 Congestion Control, congestion windows and retransmission-timeout recovery, checked 2026-09-12.
RFC 9438 CUBIC, CUBIC behavior and its relationship to congestion signals, checked 2026-09-12.
Linux tcp(7), per-socket congestion-controller selection, checked 2026-09-12.
Linux 6.12.76 TCP_INFO fields, congestion states, window, threshold and retransmission counters, checked 2026-09-12.