Tcp · XB Field Notes

The TCP backlog holds connections that are still syncing

Fri, 24 Mar 2028 00:00:00 +0000

A listening TCP socket has two queues, not one, and the sysctl this article is named after governs the queue that almost never drops anything. The backlog passed to listen(2) is the accept queue, holding connections that finished the handshake and await the application; the SYN queue holds half-open requests. Confusing them produces the wrong fix.

That matters because the famous advice, raise tcp_max_syn_backlog, is largely obsolete: tcp_syncookies defaults to 1, and as listen(2) puts it, with syncookies enabled there is no logical maximum length and that setting is ignored. What actually bites is the accept queue and how fast the application calls accept().

The backlog argument is the accept queue

listen(2) is explicit that the behaviour changed with Linux 2.2: backlog now specifies the queue length for completely established sockets waiting to be accepted, not the number of incomplete requests. The incomplete queue is bounded separately by /proc/sys/net/ipv4/tcp_max_syn_backlog. And if the backlog argument exceeds /proc/sys/net/core/somaxconn it is silently capped to that value, 4096 since Linux 5.4 and 128 before. So listen(8192) on a default host gives you 4096, and nothing logs it.

tcp(7) describes tcp_max_syn_backlog as the maximum number of queued requests that have not received an acknowledgement, defaulting to 256, raised to 1024 when memory is at least 128 MB, cut to 128 below 32 MB. ip-sysctl adds the qualifications: it is a per-listener limit, each half-open request costs about 304 bytes, and tcp_syncookies defaults to 1, sending a cookie on overflow instead of dropping.

With syncookies on, the SYN queue stops mattering

ip-sysctl documents tcp_syncookies as a fallback that must not be used to help a loaded server stand against legal connection throughput, and says syncookies seriously violate the protocol, disallow TCP extensions, and can degrade services. listen(2) goes further: with syncookies enabled there is no logical maximum length and tcp_max_syn_backlog is ignored. tcp(7) still describes the same knob as dropping requests once exceeded, which is the syncookies-off behaviour. Both pages are right about different configurations.

The honest reading is therefore: leave syncookies at 1 and stop tuning the SYN queue. The cost is that the fallback loses TCP options such as window scaling and SACK on the connections it rescues, which is a throughput problem rather than a connectivity one. If you see SYN-flood warnings while the traffic is legitimate, ip-sysctl’s own advice is to tune the other knobs, not to disable syncookies.

The accept queue is where the pressure lands

With the SYN queue neutralised, the queue that fills is the accept queue, sized by min(listen(backlog), somaxconn). Overflow behaviour depends on tcp_abort_on_overflow, default FALSE: ip-sysctl says the default lets a connection recover if the overflow was a burst, and warns that enabling the abort option can harm clients. The mechanism is the surprise: by default the kernel silently discards the final ACK, so connect() succeeds and the failure only surfaces when the first request stalls or is reset.

A bigger backlog is a delay line, not capacity. In the model, listen(128) and listen(65535) accept the same 20,000 connections in a second; the larger backlog only postpones the overflow, because the accept rate is set by the worker count.

When it breaks

The first failure is the silent one, and it is the default. A service whose accept loop is too slow overflows the accept queue, the kernel drops the final ACKs, and clients see connections that establish and then hang. Nothing is refused, so a client connection pool fills with dead sockets and the error surfaces as timeouts somewhere unrelated. Watch TcpExtListenOverflows and TcpExtListenDrops: those counters climbing while the accept rate is flat means the handler is the limit.

The second failure is the dropped SYN, which needs syncookies off to happen. A full SYN queue then drops the request and the client retransmits on a schedule whose documented endpoints are 67 seconds to the last retransmission and 131 seconds to the final timeout, so the user-visible symptom is a connect that hangs for over two minutes and then fails. Raising tcp_max_syn_backlog is a real fix in that configuration, and turning syncookies back on is a better one, if you accept the loss of TCP extensions.

Check it quickly

ss -lnt 'sport = :8443'
nstat -az TcpExtListenOverflows TcpExtListenDrops TcpExtTCPReqQFullDrop TcpExtTCPReqQFullDoCookies
sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog net.ipv4.tcp_syncookies net.ipv4.tcp_abort_on_overflow

In ss -lnt, Send-Q is the backlog the socket was created with and Recv-Q is how many established connections are queued and unaccepted, so Recv-Q sitting at Send-Q is the queue being full. The nstat counters should be zero or near it: TcpExtListenOverflows and TcpExtListenDrops are accept-queue events, while TcpExtTCPReqQFullDrop is a dropped SYN and TcpExtTCPReqQFullDoCookies is the fallback doing its job. The sysctl line shows what the kernel will honour, and if somaxconn is smaller than your listen() argument then that smaller number is your real backlog.

Sources

Kernel ip-sysctl documentation, somaxconn, tcp_max_syn_backlog, tcp_abort_on_overflow and the syncookies warning, checked 2026-09-16.
Linux listen(2), the accept queue since Linux 2.2, the silent somaxconn cap, and syncookies removing the SYN-queue bound, checked 2026-09-16.
Linux tcp(7), tcp_max_syn_backlog defaults by memory size and the syncookies caveats, checked 2026-09-16.

TCP slow start grows with acknowledged data, not merely with elapsed time

Tue, 28 Dec 2027 00:00:00 +0000

Slow start grows a TCP sender’s congestion window while it learns how much outstanding data the path can support. With enough data and suitable acknowledgments, the classic behavior is approximately exponential growth across round trips. “Doubles every RTT” is a useful simplified pattern, not an unconditional timer rule. The window changes in response to transport feedback and the implementation’s growth conditions.

When it breaks

Waiting after connect does not automatically warm up the congestion window. If the application sends nothing, there are no data acknowledgments to support that growth. A very small exchange may also finish without using the existing flight budget. Expecting every short request to display a large doubling sequence mistakes an application-limited transfer for a continuously backlogged sender.

The initial window is not universally one or two segments on modern systems. RFC 6928 describes an experimental extension allowing a larger initial window, and the tested Linux sender begins with ten segments. The relevant unit matters too: Linux TCP_INFO reports snd_cwnd in segments, whereas many protocol explanations express congestion windows in bytes. Compare it with the sender’s MSS before reasoning about outstanding data volume.

Slow start does not always continue until a first loss, followed by a universal collapse. The slow-start threshold helps select between startup growth and congestion avoidance, and implementations may use additional startup detection. Recovery from a retransmission timeout differs from fast recovery. Applying one cartoon curve to every algorithm, loss signal, and workload can produce an incorrect diagnosis from otherwise valid measurements.

Check it quickly

Save this as check-slow-start.py and run python3 check-slow-start.py with Docker. The disposable environment uses Python 3.11.16 and iproute2 6.1.0, tested on Linux 6.12.76. NET_ADMIN and SYS_ADMIN support nested namespaces. A new veth joins client and target; netem adds ten milliseconds of egress delay in each direction, with all changes confined to those virtual endpoints:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,('child exited without a response',process.stderr.read())
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found


info_code=r'''
import struct
def tcp_info(stream):
 raw=stream.getsockopt(socket.IPPROTO_TCP,socket.TCP_INFO,104);assert len(raw)==104
 offsets={'snd_mss':16,'unacked':24,'rtt_us':68,'snd_ssthresh':76,'snd_cwnd':80,'total_retrans':100}
 return {'ca_state':raw[1],**{key:struct.unpack_from('=I',raw,offset)[0] for key,offset in offsets.items()}}
'''
with lab(['client','target']) as (root,stack,setting):
 wire('client','c0','target','t0',['192.0.2.1/30'],['192.0.2.2/30'])
 for name,interface in [('client','c0'),('target','t0')]:
 ns(name,'tc','qdisc','add','dev',interface,'root','netem','delay','10ms','limit','10000')
 server_code=r'''
import socket,json,hashlib,sys
size=int(sys.argv[1]);listener=socket.socket();listener.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
listener.bind(('192.0.2.2',18080));listener.listen();print('listening',flush=True)
with listener.accept()[0] as connection:
 connection.settimeout(6);remaining=size;digest=hashlib.sha256()
 while remaining:
 chunk=connection.recv(min(65536,remaining));assert chunk;remaining-=len(chunk);digest.update(chunk)
 assert digest.hexdigest()==hashlib.sha256(b'x'*size).hexdigest()
 connection.sendall((digest.hexdigest()+'\n').encode())
'''
 client_code=info_code+r'''
import socket,json,time,threading,sys,hashlib
size=int(sys.argv[1]);stream=socket.socket();stream.settimeout(6)
stream.setsockopt(socket.IPPROTO_TCP,socket.TCP_CONGESTION,b'reno');stream.connect(('192.0.2.2',18080))
initial=tcp_info(stream);time.sleep(.06);idle=tcp_info(stream);assert idle['snd_cwnd']==initial['snd_cwnd']
done=threading.Event();errors=[];samples=[initial,idle];reply=[]
def transfer():
 try:
 stream.sendall(b'x'*size)
 with stream.makefile('rb') as incoming:reply.append(incoming.readline().decode().strip())
 except BaseException as error:errors.append(repr(error))
 finally:done.set()
worker=threading.Thread(target=transfer);worker.start();deadline=time.monotonic()+7
while not done.wait(.005):
 assert time.monotonic()<deadline,'transfer deadline exceeded'
 snapshot=tcp_info(stream)
 if snapshot!=samples[-1]:samples.append(snapshot)
worker.join(timeout=1);assert not worker.is_alive() and not errors,errors
samples.append(tcp_info(stream));assert reply==[hashlib.sha256(b'x'*size).hexdigest()]
assert all(row['total_retrans']==0 for row in samples),samples
if size>1:assert max(row['snd_cwnd'] for row in samples)>initial['snd_cwnd']*2,samples
else:assert max(row['snd_cwnd'] for row in samples)==initial['snd_cwnd'],samples
print(json.dumps({'bytes':size,'algorithm':stream.getsockopt(socket.IPPROTO_TCP,socket.TCP_CONGESTION,16).split(b'\0')[0].decode(),
 'before_data':initial,'after_idle_without_data':idle,'samples':samples,'sha256':reply[0]}));stream.close()
'''
 results={}
 for label,size in [('short',1),('bulk',1048576)]:
 service=spawn(stack,['ip','netns','exec','target','python3','-u','-c',server_code,str(size)]);assert line(service).strip()=='listening'
 results[label]=json.loads(ns('client','python3','-c',client_code,str(size),timeout=10).stdout)
 service.wait(timeout=2);assert service.returncode==0
 print(json.dumps({'egress_delay_each_direction_ms':10,'transfers':results}))
"""
name='xbcatm-tcpflow-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The client explicitly selects Reno so the experiment is not silently comparing different default congestion controllers. It records TCP_INFO before sending, waits sixty milliseconds without data, and records it again. The window stays at ten segments in both observations. That directly tests the mistaken expectation that simply leaving a new connection open makes its congestion budget grow on a clock.

The first transfer sends one byte and receives a digest from the server. Samples include one segment outstanding and then none, while the congestion window remains ten. Successful delivery therefore does not require visible exponential growth for this workload. There was already enough permitted flight size for the entire request, so its limited application data does not exercise a bulk startup ramp.

The second transfer sends one mebibyte. A separate thread performs the blocking exchange while the main thread samples socket state, avoiding loss of observations while sendall waits. The captured progression includes ten, twenty, forty, and eighty segments, followed by less tidy values as acknowledgment grouping and transfer progress interact. The test requires substantial growth, not one exact sample sequence or one sample per RTT.

Both server and client verify the payload through SHA-256, and all sampled retransmission totals remain zero. The threshold is much larger than the observed window in this run. These observations support an ACK-driven startup demonstration without deliberately testing loss recovery or measuring a shared bottleneck. The artificial delay makes progress observable; it does not establish a real WAN’s capacity or fairness.

Interpret a growth trace in its workload context

Record algorithm, MSS, congestion window, threshold, outstanding data, RTT, and retransmission evidence together. A window larger than the current unacknowledged count is possible because the application or receiver can impose another limit. Treat a sampled window as the sender’s current allowance, not as a promise that exactly that many segments are continuously occupying the path.

Sampling also affects the picture. A userspace observer can miss intermediate changes, and its five-millisecond interval is not synchronized with ACK arrivals. Delayed acknowledgments, offload behavior, and scheduling can alter the visible progression. The fixture checks relationships supported by the data rather than declaring that an omitted intermediate value proves a broken congestion controller.

For short operations, reducing unnecessary new connections may avoid repeatedly starting with a fresh congestion budget, where the application protocol permits reuse. A reused connection can still restart with a reduced window after an idle period, depending on its restart policy. For bulk performance, test representative RTT, loss, receive behavior, and competing traffic before changing startup settings. This experiment separates idle time, a tiny successful exchange, and sustained data, providing a concrete basis for deciding whether slow start is relevant to the observed delay.

Sources

RFC 5681 Congestion Control, ACK-driven slow start, congestion avoidance, threshold transitions and idle restart, checked 2026-09-12.
RFC 6928 Initial Window, the experimental extension allowing a larger initial congestion window, checked 2026-09-12.
Linux 6.12.76 TCP_INFO fields, congestion window, threshold, unacknowledged segments and retransmissions, checked 2026-09-12.

TCP window scaling enlarges receive-window encoding in each direction

Thu, 02 Dec 2027 00:00:00 +0000

TCP’s header has a 16-bit receive-window field. Window scaling lets established connections interpret that field using a shift negotiated during setup, allowing a receiver to advertise more than 65535 bytes of space. The two directions can use different shifts. The extension enlarges the encoding of receive-window information; it does not directly enlarge every buffer or guarantee that a sender will keep that much data in flight.

When it breaks

A high-bandwidth path with substantial RTT may need much more outstanding data than an unscaled receive window can describe. For example, 100 megabits per second over an 80-millisecond round trip corresponds to roughly one million bytes in flight. A 65535-byte window would impose a much lower idealized rate. This bandwidth-delay calculation is a sizing estimate, not a throughput measurement or a guarantee that no other limit applies.

The sender also has a congestion window and needs application data to send. Enlarging receive-window encoding cannot fix a small congestion budget, an application that stops reading, packet loss, or a blocked route. A scaled zero window is still zero. Diagnose which resource is actually constraining progress instead of treating the presence of a window-scale option as proof that the connection should saturate the link.

Negotiation errors can make captures misleading. Both endpoints must offer the option for scaling to operate, and each scale applies to windows advertised by the endpoint that supplied it. The window field in SYN and SYN-ACK is never scaled. Applying the wrong direction’s shift, or multiplying the SYN window, produces an incorrect capacity calculation even when the packet bytes are perfectly valid.

Check it quickly

Save this as check-window-scaling.py and run python3 check-window-scaling.py with Docker. The disposable environment uses Python 3.11.16, iproute2 6.1.0, and tcpdump 4.99.3, tested on Linux 6.12.76. NET_ADMIN and SYS_ADMIN enable nested namespaces and their temporary proc mount. The endpoints use a new veth link, and settings affect only that laboratory:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,('child exited without a response',process.stderr.read())
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found


def capture(stack,root,name,interface,expression,label):
 path=root/(label+'.pcap')
 process=subprocess.Popen(['ip','netns','exec',name,'tcpdump','--immediate-mode','-Z','root','-U','-n','-i',interface,'-w',str(path),expression],
 stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process)
 deadline=time.monotonic()+3
 while time.monotonic()<deadline:
 assert process.poll() is None,'tcpdump exited before listening'
 if select.select([process.stderr],[],[],.1)[0] and 'listening on' in process.stderr.readline():return process,path
 raise AssertionError('tcpdump did not become ready')
def packets(process,path):
 import struct,socket
 process.send_signal(signal.SIGINT);_,diagnostic=process.communicate(timeout=3)
 assert process.returncode==0,diagnostic
 raw=path.read_bytes();assert raw[:4]==bytes.fromhex('d4c3b2a1') and struct.unpack_from('<I',raw,20)[0]==1
 offset=24;result=[]
 while offset<len(raw):
 seconds,micros,size,original=struct.unpack_from('<IIII',raw,offset);offset+=16
 frame=raw[offset:offset+size];offset+=size
 if frame[12:14]!=b'\x08\x00' or frame[23]!=6:continue
 ihl=(frame[14]&15)*4;length=struct.unpack_from('!H',frame,16)[0];tcp=14+ihl
 sport,dport,seq,ack=struct.unpack_from('!HHII',frame,tcp);header=(frame[tcp+12]>>4)*4
 opts={};cursor=tcp+20
 while cursor<tcp+header:
 kind=frame[cursor]
 if kind==0:break
 if kind==1:cursor+=1;continue
 option_length=frame[cursor+1];data=frame[cursor+2:cursor+option_length];assert option_length>=2
 if kind==2:opts['mss']=struct.unpack('!H',data)[0]
 elif kind==3:opts['wscale']=data[0]
 elif kind==4:opts['sack_ok']=True
 elif kind==5:opts['sack']=[list(pair) for pair in struct.iter_unpack('!II',data)]
 cursor+=option_length
 body=frame[tcp+header:14+length]
 result.append({'t':seconds+micros/1e6,'src':socket.inet_ntoa(frame[26:30]),'dst':socket.inet_ntoa(frame[30:34]),
 'sport':sport,'dport':dport,'seq':seq,'ack':ack,'flags':frame[tcp+13],'window':struct.unpack_from('!H',frame,tcp+14)[0],
 'length':len(body),'options':opts,'preview':body[:24].hex()})
 return result
with lab(['client','target']) as (root,stack,setting):
 wire('client','c0','target','t0',['192.0.2.1/30'],['192.0.2.2/30'])
 server_code=r'''
import socket,json
def read_exact(stream,length):
 data=b''
 while len(data)<length:
 chunk=stream.recv(length-len(data));assert chunk,'premature EOF';data+=chunk
 return data
listener=socket.socket();listener.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
listener.setsockopt(socket.SOL_SOCKET,socket.SO_RCVBUF,262144)
listener.bind(('192.0.2.2',18080));listener.listen();print('listening',flush=True)
with listener.accept()[0] as connection:
 connection.settimeout(4);print(json.dumps({'rcvbuf':connection.getsockopt(socket.SOL_SOCKET,socket.SO_RCVBUF)}),flush=True)
 assert read_exact(connection,4)==b'ping';connection.sendall(b'pong')
 assert connection.recv(1)==b''
'''
 client_code=r'''
import socket,json,sys
def read_exact(stream,length):
 data=b''
 while len(data)<length:
 chunk=stream.recv(length-len(data));assert chunk,'premature EOF';data+=chunk
 return data
s=socket.socket();s.settimeout(4);s.setsockopt(socket.SOL_SOCKET,socket.SO_RCVBUF,65536)
s.bind(('192.0.2.1',40400+int(sys.argv[1])));s.connect(('192.0.2.2',18080));s.sendall(b'ping')
assert read_exact(s,4)==b'pong'
print(json.dumps({'rcvbuf':s.getsockopt(socket.SOL_SOCKET,socket.SO_RCVBUF),'reply':'pong'}),flush=True)
assert sys.stdin.readline().strip()=='close';s.close()
'''
 outcomes={}
 for enabled in [1,0]:
 setting('client','net.ipv4.tcp_window_scaling',enabled);setting('target','net.ipv4.tcp_window_scaling',enabled)
 service=spawn(stack,['ip','netns','exec','target','python3','-u','-c',server_code]);assert line(service).strip()=='listening'
 tap,path=capture(stack,root,'client','c0','tcp port 18080','window'+str(enabled))
 client=spawn(stack,['ip','netns','exec','client','python3','-u','-c',client_code,str(enabled)])
 client_result=json.loads(line(client));server_result=json.loads(line(service))
 diagnostics=ns('client','ss','-tin','state','established').stdout
 client.stdin.write('close\n');client.stdin.flush();client.wait(timeout=2);service.wait(timeout=2)
 assert client.returncode==service.returncode==0
 observed=packets(tap,path);syn=next(p for p in observed if p['flags']&0x12==0x02)
 synack=next(p for p in observed if p['flags']&0x12==0x12)
 advertised=next(p for p in observed if p['sport']==18080 and p['length']==4)
 if enabled:
 assert 'wscale' in syn['options'] and 'wscale' in synack['options']
 effective=advertised['window']<<synack['options']['wscale'];assert effective>65535
 else:
 assert 'wscale' not in syn['options'] and 'wscale' not in synack['options'];effective=advertised['window'];assert effective<=65535
 outcomes[enabled]={'client':client_result,'server':server_result,'syn':syn,'synack':synack,
 'server_post_handshake_window_field':advertised['window'],'effective_receive_window':effective,'ss':diagnostics}
 print(json.dumps(outcomes))
"""
name='xbcatm-tcpflow-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

Before opening connections, the client requests a receive buffer of 65536 bytes and the server listener requests 262144 bytes. The script records the effective socket-option values rather than assuming the requested sizes are used literally. Linux buffer accounting and limits matter: this run reported 131072 on the client and 425984 on the accepted server socket. Neither number should simply be equated with an advertised TCP window.

With scaling enabled in both namespaces, the captured client SYN offers shift one and the server SYN-ACK offers shift two. Their initial window fields remain unscaled. After setup, a server data segment advertises window field 53213. Applying the server’s shift gives 53213 times four, or 212852 bytes. The client socket diagnostic independently reports the corresponding send-window value.

The next fresh connection disables scaling in both namespaces while keeping the buffer requests and application exchange the same. Neither SYN carries a scale option, and the observed server receive window is 65535 bytes. Both trials send ping and receive pong, checking actual communication as well as negotiation. The comparison demonstrates the encoding limit without pretending that a four-byte exchange measures maximum throughput.

The fixture captures setup because the scale is fixed for that connection. Changing a system default later does not renegotiate an existing stream. Configuring appropriate receive-buffer behavior before connect or listen matters when it influences the scale selected during setup. A packet trace starting after the handshake needs the previously negotiated state before its raw window fields can be interpreted correctly.

Separate buffer allocation, advertised capacity, and observed performance

Record the relevant socket settings, negotiated shifts, current advertised window, congestion window, RTT, and application activity. Those describe different constraints. A large allocation can coexist with a smaller advertised window because the stack accounts for buffering and current occupancy. The experiment’s reported allocation and wire-level value intentionally make that distinction visible rather than hiding it behind one ambiguous “window size.”

If a transfer underperforms, check whether the receiver keeps consuming data and whether the sender has enough work to use its permitted flight size. Review losses and congestion behavior before changing buffer budgets. Larger allocations across many concurrent sockets also consume resources, so tune with workload evidence rather than maximizing every available limit.

The protocol limits the shift to fourteen, giving a representable window below one gibibyte. That encoding ceiling is not a requirement to allocate that much memory. Validate the deployed kernel and representative traffic after a change. This example verifies negotiation, direction, arithmetic, and a successful exchange; sustained capacity and application behavior need measurements suited to the real path.

Sources

RFC 7323 TCP Extensions, directional scale factors, SYN exceptions and receive-window calculation, checked 2026-09-12.
Linux tcp(7), receive buffers, window scaling and socket options before connection setup, checked 2026-09-12.
RFC 5681 Congestion Control, the separate limits imposed by congestion and receive windows, checked 2026-09-12.

Nagle and delayed ACK can stall the second part of a small request

Sun, 31 Oct 2027 00:00:00 +0000

Nagle’s algorithm reduces small TCP segments by holding additional short data while earlier data remains unacknowledged, unless enough data accumulates for a full segment. Delayed acknowledgment is a separate receiver behavior that can postpone an ACK briefly. The interaction matters when an application sends a request in multiple small writes and the peer needs the later bytes before producing a response.

When it breaks

An isolated small request is not automatically delayed by Nagle. If no earlier data is outstanding, the first small segment can be sent. The troublesome case has a first piece already in flight and a second short piece waiting behind it. If the receiver delays acknowledgment while its application waits for the complete request, progress can pause until the acknowledgment timer expires.

This is a bounded delay, not an indefinite protocol deadlock. An ACK can release the queued data, and accumulating enough data can also change the sender’s decision. The observed delay is not universally one additional RTT or a fixed forty milliseconds. Receiver policy, existing acknowledgment state, request framing, and network conditions determine the actual packet sequence and timing.

TCP_NODELAY disables Nagle on the selected socket; it does not disable every form of buffering, congestion control, receive-window flow control, or scheduler delay. TCP_QUICKACK on Linux requests a change in acknowledgment mode, but that mode is not permanent. Treating either option as a universal latency switch can hide application buffering or a different transport constraint that the option never addressed.

Check it quickly

Save this as check-nagle.py and run python3 check-nagle.py with Docker. The disposable environment uses Python 3.11.16, iproute2 6.1.0, and tcpdump 4.99.3, tested on Linux 6.12.76. NET_ADMIN and SYS_ADMIN enable nested namespaces. Client and target communicate over a new veth link; all settings stay inside that laboratory rather than changing host networking:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,('child exited without a response',process.stderr.read())
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found


def capture(stack,root,name,interface,expression,label):
 path=root/(label+'.pcap')
 process=subprocess.Popen(['ip','netns','exec',name,'tcpdump','--immediate-mode','-Z','root','-U','-n','-i',interface,'-w',str(path),expression],
 stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process)
 deadline=time.monotonic()+3
 while time.monotonic()<deadline:
 assert process.poll() is None,'tcpdump exited before listening'
 if select.select([process.stderr],[],[],.1)[0] and 'listening on' in process.stderr.readline():return process,path
 raise AssertionError('tcpdump did not become ready')
def packets(process,path):
 import struct,socket
 process.send_signal(signal.SIGINT);_,diagnostic=process.communicate(timeout=3)
 assert process.returncode==0,diagnostic
 raw=path.read_bytes();assert raw[:4]==bytes.fromhex('d4c3b2a1') and struct.unpack_from('<I',raw,20)[0]==1
 offset=24;result=[]
 while offset<len(raw):
 seconds,micros,size,original=struct.unpack_from('<IIII',raw,offset);offset+=16
 frame=raw[offset:offset+size];offset+=size
 if frame[12:14]!=b'\x08\x00' or frame[23]!=6:continue
 ihl=(frame[14]&15)*4;length=struct.unpack_from('!H',frame,16)[0];tcp=14+ihl
 sport,dport,seq,ack=struct.unpack_from('!HHII',frame,tcp);header=(frame[tcp+12]>>4)*4
 opts={};cursor=tcp+20
 while cursor<tcp+header:
 kind=frame[cursor]
 if kind==0:break
 if kind==1:cursor+=1;continue
 option_length=frame[cursor+1];data=frame[cursor+2:cursor+option_length];assert option_length>=2
 if kind==2:opts['mss']=struct.unpack('!H',data)[0]
 elif kind==3:opts['wscale']=data[0]
 elif kind==4:opts['sack_ok']=True
 elif kind==5:opts['sack']=[list(pair) for pair in struct.iter_unpack('!II',data)]
 cursor+=option_length
 body=frame[tcp+header:14+length]
 result.append({'t':seconds+micros/1e6,'src':socket.inet_ntoa(frame[26:30]),'dst':socket.inet_ntoa(frame[30:34]),
 'sport':sport,'dport':dport,'seq':seq,'ack':ack,'flags':frame[tcp+13],'window':struct.unpack_from('!H',frame,tcp+14)[0],
 'length':len(body),'options':opts,'preview':body[:24].hex()})
 return result
with lab(['client','target']) as (root,stack,setting):
 wire('client','c0','target','t0',['192.0.2.1/30'],['192.0.2.2/30'])
 setting('client','net.ipv4.tcp_autocorking',0)
 server_code=r'''
import socket,json
def read_exact(stream,length):
 data=b''
 while len(data)<length:
 chunk=stream.recv(length-len(data));assert chunk,'premature EOF';data+=chunk
 return data
listener=socket.socket();listener.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
listener.bind(('192.0.2.2',18080));listener.listen();print('listening',flush=True)
for trial in range(2):
 with listener.accept()[0] as connection:
 connection.settimeout(3);connection.setsockopt(socket.IPPROTO_TCP,socket.TCP_QUICKACK,0)
 print('configured',flush=True)
 request=read_exact(connection,2);assert request==b'ab',request
 connection.sendall(b'OK');print(json.dumps({'request':request.decode()}),flush=True)
'''
 service=spawn(stack,['ip','netns','exec','target','python3','-u','-c',server_code]);assert line(service).strip()=='listening'
 tap,path=capture(stack,root,'client','c0','tcp port 18080','nagle')
 client_code=r'''
import json,socket,sys,time
def read_exact(stream,length):
 data=b''
 while len(data)<length:
 chunk=stream.recv(length-len(data));assert chunk,'premature EOF';data+=chunk
 return data
nodelay=int(sys.argv[1]);s=socket.socket();s.settimeout(3)
s.setsockopt(socket.IPPROTO_TCP,socket.TCP_NODELAY,nodelay);s.bind(('192.0.2.1',40300+nodelay));s.connect(('192.0.2.2',18080))
print('connected',flush=True);assert sys.stdin.readline().strip()=='go'
start=time.monotonic();s.sendall(b'a');time.sleep(.002);s.sendall(b'b');reply=read_exact(s,2)
assert reply==b'OK';print(json.dumps({'nodelay':nodelay,'reply':reply.decode(),'elapsed_ms':(time.monotonic()-start)*1000}),flush=True);s.close()
'''
 outcomes=[]
 for nodelay in [0,1]:
 client=spawn(stack,['ip','netns','exec','client','python3','-u','-c',client_code,str(nodelay)])
 assert line(client).strip()=='connected' and line(service).strip()=='configured'
 client.stdin.write('go\n');client.stdin.flush();outcomes.append(json.loads(line(client)))
 assert json.loads(line(service))['request']=='ab';client.wait(timeout=2);assert client.returncode==0
 observed=packets(tap,path);timelines={}
 for nodelay in [0,1]:
 port=40300+nodelay;flow=[p for p in observed if port in (p['sport'],p['dport'])]
 first=next(p for p in flow if p['sport']==port and p['preview']=='61')
 second=next(p for p in flow if p['sport']==port and p['preview']=='62')
 assert (second['seq']-first['seq'])%2**32==1
 acks=[p for p in flow if p['dport']==port and p['flags']&0x10 and 0<(p['ack']-first['seq'])%2**32<=2]
 assert acks
 if not nodelay:
 intermediate=next(p for p in acks if (p['ack']-first['seq'])%2**32==1)
 assert first['t']<intermediate['t']<=second['t'] and second['t']-first['t']>.01
 else:assert second['t']<=acks[0]['t']
 timelines[nodelay]={'first_byte':first,'second_byte':second,'first_ack_covering_data':acks[0],
 'data_segment_gap_ms':(second['t']-first['t'])*1000}
 print(json.dumps({'trials':outcomes,'packet_timelines':timelines,'receiver_quickack_requested':0,'client_autocorking':0}))
"""
name='xbcatm-tcpflow-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The server waits for the complete request ab before replying OK. Before each trial, it requests non-quick acknowledgment mode on the accepted connection. The client disables namespace-local automatic corking to remove that additional coalescing mechanism from the comparison. Parent-child synchronization ensures the receiver has configured its socket before the client starts sending application bytes.

In the first trial, TCP_NODELAY is zero. The client sends a, pauses for two milliseconds, and sends b. Those writes are application actions, not a guarantee of immediate separate packets. The capture shows the first byte leaving, an ACK covering that byte arriving, and only then the second byte leaving. The program checks this ordering instead of attributing any slow response to Nagle without packet evidence.

In the recorded run, the two data segments were approximately forty-one milliseconds apart, and the complete exchange took roughly the same time. The second trial enables TCP_NODELAY and repeats the same framing and deliberate pause. The second byte leaves before the first data acknowledgment, with about two milliseconds between segments. The server receives the complete request and returns the same valid reply in both cases.

The timing values describe this controlled run, not a production latency guarantee. The fixture expects a visible delayed-ACK interaction under its stated settings and checks the causal packet order. A different kernel or receiver policy can require fresh investigation. Stream reads explicitly accumulate the expected number of bytes because recv, even with a requested length, does not generally promise an entire application message at once.

Choose framing and socket policy together

For a small logical request already available in memory, consider writing its complete representation together. That can avoid waiting for a later fragment while preserving useful transport batching. For an interactive stream that must deliver independent small updates promptly, TCP_NODELAY may be appropriate. Choose using the workload’s framing and measured behavior rather than applying one setting to every connection.

Capture both data and acknowledgments when a request periodically stalls. Determine whether the second piece is still in application buffering, queued in the sender, or already received by the peer. A timeout in application code alone cannot distinguish those stages. Correlate the capture with socket configuration and the point at which the receiver considers a request complete.

Also account for packet-rate and bandwidth costs when sending many tiny updates. Reducing an unnecessary batching delay can improve interaction while increasing segments per operation. Measure the relevant response latency and resource cost together. The useful result is a justified policy for a particular exchange, supported here by two successful requests and different, directly observed transmission ordering.

Sources

RFC 896 Congestion Control, coalescing small writes while earlier data remains unacknowledged, checked 2026-09-12.
RFC 1122 Host Requirements, delayed acknowledgment behavior and bounded acknowledgment delay, checked 2026-09-12.
Linux tcp(7), TCP_NODELAY and the temporary nature of TCP_QUICKACK, checked 2026-09-12.

SYN cookies answer when the queue overflows

Sat, 30 Oct 2027 00:00:00 +0000

A listening socket keeps two queues, not one. Half-open connections live in the SYN queue until the final ACK arrives; established ones then wait in the accept queue until the application calls accept(). Both have a limit, and which one overflows decides what the client experiences.

SYN cookies exist for the first queue. When it is full the server stops allocating state and encodes what it needs into the sequence number, so the handshake completes with no memory held. That is a real rescue and a degradation: the connection it saves is worse than the one it would have built normally.

Two queues, two different symptoms

A SYN allocates a request socket, moves the connection to SYN_RECV and sends a SYN-ACK. The client’s ACK then moves it to the accept queue as ESTABLISHED. The SYN queue is capped per listener by tcp_max_syn_backlog — minimum 128, scaling with memory, about 304 bytes per entry. The accept queue is capped by somaxconn (4096 by default on current kernels, 128 before Linux 5.4) combined with the application’s listen() backlog.

The two overflows look nothing alike. SYN queue overflow is what syncookies were built for: the connection still completes. Accept queue overflow is worse and quieter — with tcp_abort_on_overflow at its default of 0 the server ignores the final ACK, so connect() returns success, the client sends its request, and nothing answers. Measured with backlog 1 against 400 clients: all 400 believed they connected, the server accepted 3, and the other 397 were lost with no error on either side.

With tcp_syncookies at its default of 1, the kernel sends a cookie when the SYN backlog overflows. Instead of allocating a request socket it computes the initial sequence number as a counter in the top 8 bits and a truncated hash in the low 24, the hash covering both addresses, both ports, the counter and a server secret. The counter advances every 64 seconds, so the same 4-tuple yields a different cookie each tick.

Verification reverses it. The ACK’s acknowledgement number carries the counter in its top 8 bits; the server checks it against its current and previous tick, recomputes the hash and compares the low 24 bits. Reproducing that construction: a fresh ACK verifies, one replayed 70 seconds later — a single tick — still verifies, and ACKs 200 and 600 seconds later are rejected as stale. Changing the source port or address by one is rejected outright, so the cookie cannot be lifted onto another connection. Blind forgery probability is one in 16.7 million, which is the entire security argument.

A cookie SYN-ACK carries no options. The kernel documentation is blunt: syncookies “seriously violate TCP protocol, do not allow to use TCP extensions, can result in serious degradation of some services”. The expensive omission is window scaling — without it the receive window caps at 65535 bytes instead of 8388480 at scale factor 7. On a 100 ms path that is 5.2 Mbit/s instead of 671.1 Mbit/s, a 128-fold difference from one missing option. SACK is gone too, so a sender losing several segments in one window recovers one cumulative ACK at a time.

The MSS is partly recovered: the cookie path steals three hash bits to index a small MSS table, and that is the only option that survives. Timestamps, SACK and window scale are simply absent. A server in cookie mode is not just admitting fewer connections; the ones it admits are slow, and that looks like a network problem rather than a handshake one.

When it breaks

The first failure is cookies hiding a real flood from monitoring. SyncookiesSent climbing is the signal, but by design the flood is invisible at the application layer: no half-open backlog to alert on, no failed connections, no error log, just a counter. A dashboard built on error rates or SYN_RECV depth shows a healthy server while it is under sustained attack. Watch the cookie counters and their ratio to new connections, not the application.

The second failure is the accept queue overflowing anyway, which cookies do nothing about. ListenOverflows climbs, clients see timeouts on their first request, and the bottleneck is accept(), not the SYN queue — measured here as 397 of 400 lost with every client believing it connected. The fix is to accept faster, add workers and raise somaxconn and the listen backlog, not to tune syncookies.

Check it quickly

Compare both queues and the counters:

sysctl net.ipv4.tcp_syncookies net.ipv4.tcp_max_syn_backlog net.core.somaxconn
nstat -az | grep -E 'Syncookies|ListenOverflow|ListenDrops'
ss -lnt

On a healthy server all three counters read 0, and ss -lnt shows Send-Q matching the configured backlog with Recv-Q at 0. A rising SyncookiesSent means the SYN queue overflowed; a rising ListenOverflows means the accept queue did, and cookies will not fix that one. If Recv-Q on a listening socket stays above zero, established connections are waiting for an application not calling accept() fast enough.

Sources

RFC 4987 TCP SYN flood mitigation, the cookie construction and its trade-offs, checked 2026-09-16.
RFC 9293 TCP, the three-way handshake and the notion of a backlog, checked 2026-09-16.
Linux kernel ip-sysctl, tcp_syncookies, tcp_max_syn_backlog, somaxconn and tcp_abort_on_overflow, checked 2026-09-16.

The listen backlog holds connections waiting for accept

Mon, 25 Oct 2027 00:00:00 +0000

On Linux TCP sockets, the listen() backlog concerns established connections waiting for the application to accept them. The incomplete-handshake queue is a separate concept. Calling accept() removes a pending connection and returns a connected socket for application use. That accepted connection no longer occupies the listener’s accept queue, even if it remains open and busy for a long time.

When it breaks

If connections arrive faster than the application accepts them, the pending queue can fill. New clients may then encounter retransmission delays or timeouts, depending on kernel behavior and settings. Existing accepted requests can still look healthy. A process being alive, or some requests succeeding, does not prove that its listener is accepting new connections quickly enough.

The requested backlog is also constrained by net.core.somaxconn; increasing only a larger application value cannot exceed that cap. The exact observable queue boundary is implementation-dependent, however, so treating the integer as a portable promise of precisely that many successful simultaneous connects is unsafe. Measure the deployed kernel rather than converting one small experiment into a universal off-by-one rule.

Queue counters need context. ListenOverflows and ListenDrops are network-namespace counters, not automatically per-service explanations. Other listeners can contribute to them, and a client timeout can have causes outside the accept queue. Correlate queue occupancy, acceptance progress, and counters before attributing production failures to a particular backlog setting or declaring that a larger queue will fix them.

Check it quickly

Save this as check-listen-backlog.py and run it with host Python, Docker, and the pinned Python image available. The actual socket operations run inside a disposable Linux container with networking disabled except loopback. The recorded kernel was 6.12.76-linuxkit. The script reads relevant settings without modifying them and requires tcp_abort_on_overflow=0 for its timeout observation:

import json,subprocess
image='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
program=r'''
from contextlib import ExitStack
from pathlib import Path
import json,platform,socket
def counters():
 lines=Path('/proc/net/netstat').read_text().splitlines()
 for index,line in enumerate(lines):
 if line.startswith('TcpExt:'):
 return dict(zip(line.split()[1:],map(int,lines[index+1].split()[1:])))
def queued(port):
 for line in Path('/proc/net/tcp').read_text().splitlines()[1:]:
 fields=line.split()
 if fields[3]=='0A' and int(fields[1].split(':')[1],16)==port:
 return int(fields[4].split(':')[1],16)
 raise AssertionError('listener missing')
with ExitStack() as stack:
 def fresh():
 s=stack.enter_context(socket.socket());s.settimeout(.25);return s
 def listener(backlog):
 s=fresh();s.bind(('127.0.0.1',0));s.listen(backlog);return s
 somaxconn=int(Path('/proc/sys/net/core/somaxconn').read_text())
 abort=int(Path('/proc/sys/net/ipv4/tcp_abort_on_overflow').read_text());assert abort==0
 small=listener(1);address=small.getsockname();before=counters();clients=[];timed_out=False
 for _ in range(8):
 client=fresh()
 try:client.connect(address)
 except TimeoutError:timed_out=True;client.close();break
 clients.append(client)
 depth=queued(address[1]);after=counters()
 assert timed_out and depth==len(clients) and 1<=depth<8,(depth,len(clients))
 changes={name:after[name]-before[name] for name in ['ListenOverflows','ListenDrops']}
 assert changes['ListenOverflows']>0 and changes['ListenDrops']>0,changes
 accepted=[]
 for client in clients:
 server,_=small.accept();stack.enter_context(server);accepted.append(server);server.sendall(b'ready')
 reader=stack.enter_context(client.makefile('rb'));assert reader.read(5)==b'ready'
 assert queued(address[1])==0
 for _ in range(4):
 client=fresh();client.connect(address);server,_=small.accept();stack.enter_context(server)
 server.sendall(b'ready');reader=stack.enter_context(client.makefile('rb'));assert reader.read(5)==b'ready'
 accepted.append(server);clients.append(client)
 assert len(accepted)>1 and all(s.fileno()>=0 for s in accepted)
 larger=listener(16)
 for _ in range(6):client=fresh();client.connect(larger.getsockname())
 assert queued(larger.getsockname()[1])==6
 print(json.dumps({'kernel':platform.release(),'somaxconn':somaxconn,'tcp_abort_on_overflow':abort,
 'requested_small_backlog':1,'queued_before_accept':depth,'next_connect_timed_out':timed_out,
 'counter_changes':changes,'accepted_connections_still_open':len(accepted),'larger_backlog_queued':6},sort_keys=True))
'''
result=subprocess.run(['docker','run','--rm','-i','--network','none',image,'python','-'],
 input=program,capture_output=True,text=True,timeout=30)
assert result.returncode==0,result.stderr
record=json.loads(result.stdout);print(json.dumps(record,sort_keys=True))

The first listener requests backlog one and deliberately does not accept connections. In the recorded environment, two client connects complete and /proc/net/tcp reports two queued connections. The next connect exceeds the client’s 250-millisecond timeout. The script records the observed depth rather than asserting that every Linux version must admit exactly two under this setting.

During that controlled overflow, both ListenOverflows and ListenDrops increase by one. Because the experiment has its own network namespace and only these loopback listeners, the observation can be associated with its deliberately stalled listener. No global kernel setting is changed, and the example does not manufacture failure by altering the shared Docker VM’s networking policy.

The application then accepts the queued connections and sends a short acknowledgement over each returned socket. The clients read the complete acknowledgement, and the listener queue falls to zero. Four more connections are created and promptly accepted while the previous accepted sockets remain open. Six accepted connections coexist despite the original requested backlog of one.

A second listener requests backlog sixteen. Six clients connect before any acceptance, and its queue reports six pending connections. This demonstrates that a larger queue can absorb this particular arrival burst while the application is paused. It does not show higher processing throughput, prove those clients received application responses, or establish that increasing a backlog solves sustained overload.

Tune acceptance and waiting capacity together

The recorded somaxconn value is 4096, comfortably above both requested backlogs. That value is an observation of the test environment, not a recommendation to copy into another host. Inspect the application’s actual listen argument, its accept loop, and the kernel cap together. A high system limit cannot enlarge a small backlog the application deliberately requests.

Separate incomplete handshakes from established connections awaiting acceptance. SYN-queue controls and SYN-cookie behavior answer different questions from the accept queue shown here. Likewise, tcp_abort_on_overflow changes overflow handling; enabling resets is not an increase in service capacity. This experiment leaves it disabled and demonstrates the client-visible timeout under that existing policy.

Finally, investigate why acceptance falls behind. Event-loop stalls, descriptor exhaustion, dispatch bottlenecks, and deliberate pauses can affect it differently from slow work on already accepted sockets. A larger queue buys waiting room, which must fit client deadlines and memory budgets. If sustained arrivals exceed the rate the service can accept and handle, additional waiting only postpones the capacity failure.

Sources

Linux listen(2), established connections waiting for accept and the somaxconn cap, checked 2026-09-12.
Linux accept(2), removing a pending connection and returning a connected socket, checked 2026-09-12.
Linux IP sysctl documentation, somaxconn, SYN backlog, and tcp_abort_on_overflow, checked 2026-09-12.

TCP keepalive probes idle connections

Tue, 05 Oct 2027 00:00:00 +0000

TCP keepalive asks whether the other TCP endpoint still responds after a quiet period. It does not ask whether an HTTP handler, database query, or worker is making progress. A peer’s kernel can acknowledge traffic while its application is stalled. Treat transport reachability and useful application responses as different observations.

A transport probe, not an application heartbeat

Keepalive can expose a connection whose remote state disappeared or whose path silently stopped delivering traffic. The implementation sends probes after sufficient inactivity and eventually reports failure after its configured unanswered-probe policy. A single lost probe is not proof of a dead peer; transient loss must not be mistaken for certainty.

These probes are handled below the application’s byte stream. They are not HTTP requests, database pings, or WebSocket heartbeats. If a proxy terminates the TCP connection, the peer being probed is that proxy, not the origin behind it. Probes may affect some middlebox tracking, but cannot be assumed to refresh an application’s idle timer or preserve every NAT mapping.

Already outstanding data has its own retransmission and failure rules. Do not translate a keepalive setting into a universal deadline for busy connections. A FIN, reset, application timeout, and silent network loss are different events and may require different recovery decisions.

Configure the socket you actually use

RFC 1122 requires an optional keepalive mechanism to default off, with a configurable default idle interval of at least two hours. Applications and libraries can explicitly enable it and choose shorter intervals; Go’s dialer is one example. Therefore kernel defaults alone do not establish an application’s effective behavior.

On Linux, tcp_keepalive_time, tcp_keepalive_intvl, and tcp_keepalive_probes describe idle seconds, probe spacing, and the unanswered-probe limit. The documented defaults are 7200, 75, and 9. Per-socket TCP_KEEPIDLE, TCP_KEEPINTVL, and TCP_KEEPCNT can specialize the policy when SO_KEEPALIVE is enabled. These names and interfaces are not universal macOS or Windows settings.

Choose intervals against detection needs, connection count, transient-loss tolerance, and intermediary policies. An idle-plus-probe arithmetic estimate is not a hard real-time guarantee. Prefer scoped socket configuration over changing machine-wide defaults for unrelated services, and check setter errors and the actual enabled state.

When it breaks

A supposedly protected request may hang because only keepalive was configured, with no application read deadline. Conversely, an application deadline can expire while the TCP connection remains usable. Neither result proves that the peer process died. Retrying a write after a local timeout can duplicate work that the peer has already received.

A lower idle timeout at a proxy can still close a connection regardless of the endpoint’s keepalive policy. Excessively frequent probes also add traffic and power cost across large fleets. Diagnose the actual terminating hop and failure signal before changing all three timers together.

Check it quickly

Save this as keepalive_probe.go and run go run keepalive_probe.go with Go 1.23 or newer on macOS or Linux. The SO_KEEPALIVE inspection uses a Unix socket API; this exact program is not a Windows recipe. It creates only loopback sockets and changes no sysctl, firewall, or shared service.

package main

import (
 "errors"
 "fmt"
 "io"
 "net"
 "os"
 "runtime"
 "syscall"
 "time"
)

func must(err error) {
 if err != nil {
 panic(err)
 }
}

func check(ok bool, message string) {
 if !ok {
 panic(message)
 }
}

func keepalive(connection *net.TCPConn) int {
 raw, err := connection.SyscallConn()
 must(err)
 var value int
 var optionError error
 must(raw.Control(func(fd uintptr) {
 value, optionError = syscall.GetsockoptInt(int(fd), syscall.SOL_SOCKET, syscall.SO_KEEPALIVE)
 }))
 must(optionError)
 return value
}

func main() {
 listener, err := net.ListenTCP("tcp4", &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1)})
 must(err)
 defer listener.Close()
 must(listener.SetDeadline(time.Now().Add(2 * time.Second)))
 dialer := net.Dialer{Timeout: 2 * time.Second, KeepAlive: -1}
 connection, err := dialer.Dial("tcp4", listener.Addr().String())
 must(err)
 client := connection.(*net.TCPConn)
 defer client.Close()
 peer, err := listener.AcceptTCP()
 must(err)
 defer peer.Close()

 check(keepalive(client) == 0, "keepalive must start disabled")
 must(client.SetKeepAliveConfig(net.KeepAliveConfig{
 Enable: true, Idle: 30 * time.Second,
 Interval: 10 * time.Second, Count: 3,
 }))
 enabled := keepalive(client)
 check(enabled != 0, "SO_KEEPALIVE must be enabled")
 fmt.Printf("SO_KEEPALIVE: 0 -> %d (enabled); per-socket configuration accepted\n", enabled)

 must(client.SetWriteDeadline(time.Now().Add(2 * time.Second)))
 n, err := client.Write([]byte("ping"))
 must(err)
 check(n == 4, "request must be fully written")
 must(client.SetReadDeadline(time.Now().Add(200 * time.Millisecond)))
 reply := make([]byte, 4)
 n, err = client.Read(reply)
 check(n == 0 && errors.Is(err, os.ErrDeadlineExceeded), "silent application must reach read deadline")
 fmt.Println("read deadline fired while the peer application was silent")

 must(peer.SetDeadline(time.Now().Add(2 * time.Second)))
 request := make([]byte, 4)
 _, err = io.ReadFull(peer, request)
 must(err)
 check(string(request) == "ping", "the original request is still queued")
 n, err = peer.Write([]byte("pong"))
 must(err)
 check(n == 4, "reply must be fully written")
 must(client.SetReadDeadline(time.Now().Add(2 * time.Second)))
 _, err = io.ReadFull(client, reply)
 must(err)
 check(string(reply) == "pong", "same connection must deliver the reply")
 fmt.Println("after resetting the read deadline, the original connection returns pong")
 fmt.Println(runtime.Version(), runtime.GOOS)
}

The dialer first disables keepalive. Explicit socket configuration then enables it, and getsockopt must return a nonzero value; enabled need not equal one. The checked macOS Go 1.27.0 run returned eight. The peer application deliberately leaves the four-byte request unread while the client’s 200-millisecond read deadline expires.

Afterward, the peer reads the original ping and writes pong. Resetting the client’s read deadline permits that reply on the same connection, without reconnecting or resending. The timeout bounded a local wait; it did not withdraw queued bytes or cancel remote work.

All assertions passed. This short experiment checks socket configuration and application-deadline behavior, not actual keepalive probes: the configured first-probe idle interval is thirty seconds. It neither drops packets nor measures failure detection through a NAT. Those require a separately controlled network test with packet evidence and recovery checks.

Sources

RFC 1122: Requirements for Internet Hosts, section 4.2.3.6: optional keepalive, inactivity, and lost probes, checked 2026-09-09.
Linux tcp(7), keepalive defaults, socket options, and shorter intermediary timeouts, checked 2026-09-09.
Go net, KeepAliveConfig, socket configuration, and deadline semantics, checked 2026-09-09.

SACK reports received ranges beyond a gap so recovery can target missing data

Sat, 11 Sep 2027 00:00:00 +0000

A TCP cumulative acknowledgment identifies the next byte expected in order. If a segment is missing but later bytes arrive, that acknowledgment cannot advance across the hole. Selective Acknowledgment, or SACK, adds information about received byte ranges beyond the cumulative boundary. It helps a sender distinguish missing data from data already queued at the receiver, improving the information available to its recovery algorithm.

When it breaks

Without SACK, a cumulative ACK does not force the sender to retransmit every later segment. A sender can use duplicate ACKs, timers, and recovery algorithms to infer loss, but the feedback is less explicit, particularly with multiple gaps. Describing ordinary TCP as mandatory retransmission of the entire suffix confuses a limitation in information with a required recovery policy.

SACK blocks describe received ranges, not a direct list of missing ranges. Each block has an inclusive left edge and an exclusive right edge in sequence space. A hole is inferred by comparing those blocks with the cumulative acknowledgment and outstanding data. SACK does not change what the ordinary ACK field means, and the application still receives an ordered byte stream rather than out-of-order blocks.

The capability must also be permitted for the connection. SACK-Permitted in a SYN lets the peer send SACK options back to the SYN sender; it does not prove any SACK block was needed during a clean transfer. Conversely, missing SACK information is not automatically evidence that a firewall stripped an option. Inspect endpoint configuration and the actual handshake before selecting a cause.

Check it quickly

Save this as check-sack.py and run python3 check-sack.py with Docker. The disposable environment uses Python 3.11.16, iproute2 6.1.0, nftables 1.0.6, and tcpdump 4.99.3, tested on Linux 6.12.76. NET_ADMIN and SYS_ADMIN support nested network namespaces and their temporary proc mount. The example uses only a new veth link and namespace-local settings:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,'child exited without a response'
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found


def capture(stack,root,name,interface,expression,label):
 path=root/(label+'.pcap')
 process=subprocess.Popen(['ip','netns','exec',name,'tcpdump','--immediate-mode','-Z','root','-U','-n','-i',interface,'-w',str(path),expression],
 stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process)
 deadline=time.monotonic()+3
 while time.monotonic()<deadline:
 assert process.poll() is None,'tcpdump exited before listening'
 if select.select([process.stderr],[],[],.1)[0] and 'listening on' in process.stderr.readline():return process,path
 raise AssertionError('tcpdump did not become ready')
def packets(process,path):
 import struct,socket
 process.send_signal(signal.SIGINT);_,diagnostic=process.communicate(timeout=3)
 assert process.returncode==0,diagnostic
 raw=path.read_bytes();assert raw[:4]==bytes.fromhex('d4c3b2a1') and struct.unpack_from('<I',raw,20)[0]==1
 offset=24;result=[]
 while offset<len(raw):
 seconds,micros,size,original=struct.unpack_from('<IIII',raw,offset);offset+=16
 frame=raw[offset:offset+size];offset+=size
 if frame[12:14]!=b'\x08\x00' or frame[23]!=6:continue
 ihl=(frame[14]&15)*4;length=struct.unpack_from('!H',frame,16)[0];tcp=14+ihl
 sport,dport,seq,ack=struct.unpack_from('!HHII',frame,tcp);header=(frame[tcp+12]>>4)*4
 opts={};cursor=tcp+20
 while cursor<tcp+header:
 kind=frame[cursor]
 if kind==0:break
 if kind==1:cursor+=1;continue
 option_length=frame[cursor+1];data=frame[cursor+2:cursor+option_length];assert option_length>=2
 if kind==2:opts['mss']=struct.unpack('!H',data)[0]
 elif kind==3:opts['wscale']=data[0]
 elif kind==4:opts['sack_ok']=True
 elif kind==5:opts['sack']=[list(pair) for pair in struct.iter_unpack('!II',data)]
 cursor+=option_length
 body=frame[tcp+header:14+length]
 result.append({'t':seconds+micros/1e6,'src':socket.inet_ntoa(frame[26:30]),'dst':socket.inet_ntoa(frame[30:34]),
 'sport':sport,'dport':dport,'seq':seq,'ack':ack,'flags':frame[tcp+13],'window':struct.unpack_from('!H',frame,tcp+14)[0],
 'length':len(body),'options':opts,'preview':body[:24].hex()})
 return result
with lab(['client','target']) as (root,stack,setting):
 wire('client','c0','target','t0',['192.0.2.1/30'],['192.0.2.2/30'])
 setting('client','net.ipv4.tcp_sack',1);setting('target','net.ipv4.tcp_sack',1)
 setting('client','net.ipv4.tcp_autocorking',0)
 server_code=r'''
import socket,json,hashlib
listener=socket.socket();listener.bind(('192.0.2.2',18080));listener.listen();print('listening',flush=True)
with listener.accept()[0] as connection:
 connection.settimeout(6);data=b''
 while len(data)<12*256:
 chunk=connection.recv(4096);assert chunk;data+=chunk
 expected=b''.join(bytes([i])*256 for i in range(12));assert data==expected
 result={'bytes':len(data),'sha256':hashlib.sha256(data).hexdigest()}
 connection.sendall(json.dumps(result).encode()+b'\n');print(json.dumps(result),flush=True)
'''
 service=spawn(stack,['ip','netns','exec','target','python3','-u','-c',server_code]);assert line(service).strip()=='listening'
 nft('target','''table ip one_gap {
 chain input { type filter hook input priority 0; policy accept;
 tcp dport 18080 ip length > 100 numgen inc mod 2147483647 == 1 counter drop comment "second_data_packet"
 }
}
''')
 tap,path=capture(stack,root,'client','c0','tcp port 18080','sack')
 client_code=r'''
import socket,json,time,hashlib
s=socket.socket();s.settimeout(6);s.setsockopt(socket.IPPROTO_TCP,socket.TCP_NODELAY,1)
s.bind(('192.0.2.1',40500));s.connect(('192.0.2.2',18080))
for i in range(12):s.sendall(bytes([i])*256);time.sleep(.003)
with s.makefile('rb') as stream:result=json.loads(stream.readline())
expected=b''.join(bytes([i])*256 for i in range(12))
assert result=={'bytes':len(expected),'sha256':hashlib.sha256(expected).hexdigest()}
print(json.dumps(result));s.close()
'''
 result=json.loads(ns('client','python3','-c',client_code).stdout);assert json.loads(line(service))==result
 dropped=counters('target');assert dropped['second_data_packet']==1,dropped
 observed=packets(tap,path);syn=next(p for p in observed if p['flags']&0x12==0x02);synack=next(p for p in observed if p['flags']&0x12==0x12)
 assert syn['options'].get('sack_ok') and synack['options'].get('sack_ok')
 reports=[p for p in observed if p['sport']==18080 and p['options'].get('sack')]
 assert reports,observed
 data_start=(syn['seq']+1)%2**32
 def relative(sequence):return (sequence-data_start)%2**32
 hole=(data_start+256)%2**32
 assert any(relative(p['ack'])==256 and any(256<relative(left)<relative(right)<=3072 for left,right in p['options']['sack']) for p in reports),reports
 retransmissions=[p for p in observed if p['sport']==40500 and p['length'] and p['seq']==hole]
 assert len(retransmissions)>=2,retransmissions
 print(json.dumps({'sack_negotiated_both_directions':True,'missing_byte_sequence':hole,'sack_reports':reports,
 'original_and_retransmitted_hole':retransmissions,'drop_counter':dropped,'verified_delivery':result}))
"""
name='xbcatm-tcpflow-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

Both endpoints enable SACK. The client enables TCP_NODELAY and disables automatic corking in its own namespace, then sends twelve distinguishable chunks of 256 bytes with short pauses. The target input chain uses an incrementing rule expression to drop only the second matching data packet. The very large modulus makes that a single controlled loss within this short transfer; the counter must report exactly one drop.

The capture confirms SACK-Permitted in both SYN messages. After the first chunk arrives, the missing second chunk creates a hole. The receiver’s cumulative acknowledgment stays at the first byte of that hole, while a SACK block describes a later received range. The script checks the relationship using the actual initial sequence number rather than assuming a fixed packet sequence across runs.

The missing range appears in the capture first as the original transmitted segment and again as a retransmission with the same sequence number. In this run, it was a 256-byte segment. The server eventually receives all 3072 bytes in their expected order and checks their content. It returns a SHA-256 digest, which the client independently checks against the intended payload.

Those observations establish a real gap, informative acknowledgment, retransmission, and successful delivery. They do not benchmark SACK against a disabled configuration or establish a universal reduction in retransmission count. The capture also does not, by itself, identify every detail of Linux’s loss-detection decision. Negotiating SACK and choosing when to retransmit are related but distinct parts of the transport implementation.

Interpret feedback without promising perfect retransmission decisions

RFC 2018 makes SACK advisory: a receiver may later discard data it previously reported. The sender therefore cannot treat SACK as a permanent application-level receipt or discard all retransmission responsibility merely because a range appeared in an option. The cumulative acknowledgment remains central to determining which bytes have been accepted in order by TCP.

Reordering can also produce gaps and selective reports without permanent packet loss. Correlate sequence ranges, retransmission timing, cumulative progress, and capture location before classifying every SACK as a damaged network. Here, the known input filter and its exact drop counter provide evidence of deliberate loss that would be absent from an unexplained production trace.

For an incident, retain handshake options and enough of the data exchange to reconstruct the relevant ranges. Compare application delivery and transport recovery separately. Correct reassembly may hide a transient network loss from application correctness while still affecting latency or throughput. Use representative workload measurements for those performance questions; this small fixture verifies the protocol information and resulting byte delivery.

Sources

RFC 2018 TCP Selective Acknowledgment, received-byte ranges, direction of permission and the advisory nature of SACK, checked 2026-09-12.
RFC 9293 TCP, cumulative acknowledgments, sequence space and reliable byte delivery, checked 2026-09-12.
Linux tcp(7), Linux TCP SACK configuration and socket diagnostics, checked 2026-09-12.

The TCP handshake establishes sequence spaces, not synchronized clocks

Fri, 10 Sep 2027 00:00:00 +0000

The TCP three-way handshake establishes the two directions of a byte stream. Each endpoint supplies an initial sequence number and acknowledges the other endpoint’s number. Those are independent sequence spaces, not a shared wall clock. Applications do not need synchronized machine clocks to connect. Later sequence numbers identify bytes and support ordering, duplicate handling, acknowledgments, and retransmission within that connection.

When it breaks

A client sends SYN with its initial sequence number. The server returns SYN-ACK containing its own initial number and acknowledging the client’s SYN. The client acknowledges that response. SYN consumes one position in sequence space, so its acknowledgment advances by one even when the segment contains no application data. Treating these values as timestamps produces incorrect packet interpretations and can send an investigation toward unrelated clock synchronization.

The two endpoints do not change state at exactly the same instant. A client’s connect call can succeed after receiving SYN-ACK while its final acknowledgment is lost before reaching the server. The server can remain in SYN-RECEIVED, displayed as SYN-RECV by Linux tools. Subsequent data carrying an appropriate acknowledgment can also complete the exchange, so a missing standalone third packet does not inevitably cause permanent failure.

Successful TCP setup establishes neither application health nor peer authentication. A service might accept connections but never produce a valid response. A proxy may complete one TCP connection while failing to establish its separate upstream connection. Likewise, sequence-number selection helps protect connection incarnations and resist prediction attacks, but it is not an authentication protocol and does not encrypt the stream.

Check it quickly

Save this as check-tcp-handshake.py and run python3 check-tcp-handshake.py with Docker. The disposable container uses Python 3.11.16, iproute2 6.1.0, nftables 1.0.6, and tcpdump 4.99.3, tested with Linux 6.12.76. NET_ADMIN and SYS_ADMIN support its nested network namespaces. A newly created veth joins the client and target, with no host interfaces or published ports involved:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,('child exited without a response',process.stderr.read())
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found


def capture(stack,root,name,interface,expression,label):
 path=root/(label+'.pcap')
 process=subprocess.Popen(['ip','netns','exec',name,'tcpdump','--immediate-mode','-Z','root','-U','-n','-i',interface,'-w',str(path),expression],
 stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process)
 deadline=time.monotonic()+3
 while time.monotonic()<deadline:
 assert process.poll() is None,'tcpdump exited before listening'
 if select.select([process.stderr],[],[],.1)[0] and 'listening on' in process.stderr.readline():return process,path
 raise AssertionError('tcpdump did not become ready')
def packets(process,path):
 import struct,socket
 process.send_signal(signal.SIGINT);_,diagnostic=process.communicate(timeout=3)
 assert process.returncode==0,diagnostic
 raw=path.read_bytes();assert raw[:4]==bytes.fromhex('d4c3b2a1') and struct.unpack_from('<I',raw,20)[0]==1
 offset=24;result=[]
 while offset<len(raw):
 seconds,micros,size,original=struct.unpack_from('<IIII',raw,offset);offset+=16
 frame=raw[offset:offset+size];offset+=size
 if frame[12:14]!=b'\x08\x00' or frame[23]!=6:continue
 ihl=(frame[14]&15)*4;length=struct.unpack_from('!H',frame,16)[0];tcp=14+ihl
 sport,dport,seq,ack=struct.unpack_from('!HHII',frame,tcp);header=(frame[tcp+12]>>4)*4
 opts={};cursor=tcp+20
 while cursor<tcp+header:
 kind=frame[cursor]
 if kind==0:break
 if kind==1:cursor+=1;continue
 option_length=frame[cursor+1];data=frame[cursor+2:cursor+option_length];assert option_length>=2
 if kind==2:opts['mss']=struct.unpack('!H',data)[0]
 elif kind==3:opts['wscale']=data[0]
 elif kind==4:opts['sack_ok']=True
 elif kind==5:opts['sack']=[list(pair) for pair in struct.iter_unpack('!II',data)]
 cursor+=option_length
 body=frame[tcp+header:14+length]
 result.append({'t':seconds+micros/1e6,'src':socket.inet_ntoa(frame[26:30]),'dst':socket.inet_ntoa(frame[30:34]),
 'sport':sport,'dport':dport,'seq':seq,'ack':ack,'flags':frame[tcp+13],'window':struct.unpack_from('!H',frame,tcp+14)[0],
 'length':len(body),'options':opts,'preview':body[:24].hex()})
 return result
with lab(['client','target']) as (root,stack,setting):
 wire('client','c0','target','t0',['192.0.2.1/30'],['192.0.2.2/30'])
 server_code=r'''
import socketserver
class Handler(socketserver.StreamRequestHandler):
 def handle(self):
 data=self.rfile.readline();self.wfile.write(b'echo:'+data);self.wfile.flush()
class Server(socketserver.ThreadingTCPServer):allow_reuse_address=True;daemon_threads=True
server=Server(('192.0.2.2',18080),Handler);print('ready',flush=True);server.serve_forever()
'''
 service=spawn(stack,['ip','netns','exec','target','python3','-u','-c',server_code]);assert line(service).strip()=='ready'
 tap,path=capture(stack,root,'client','c0','tcp port 18080','handshake')
 normal=ns('client','python3','-c',
 'import socket; s=socket.socket();s.settimeout(2);s.bind(("192.0.2.1",40100));s.connect(("192.0.2.2",18080));s.sendall(b"normal\\n");print(s.makefile("rb").readline().decode().strip());s.close()')
 assert normal.stdout.strip()=='echo:normal'
 nft('client','''table ip missing_ack {
 chain output { type filter hook output priority 0; policy accept;
 tcp dport 18080 tcp flags & (syn|ack|rst|fin) == ack counter drop comment "missing_final_ack"
 }
}
''')
 client_code=r'''
import json,socket,sys
s=socket.socket();s.settimeout(.25);s.bind(('192.0.2.1',40101));s.connect(('192.0.2.2',18080))
print(json.dumps({'connect_returned':True}),flush=True);s.sendall(b'deferred\n')
try:s.recv(64);raise AssertionError('unexpected application response')
except TimeoutError:print(json.dumps({'application_timeout':True}),flush=True)
assert sys.stdin.readline().strip()=='restore';s.settimeout(5)
received=b''
while not received.endswith(b'\n'):
 chunk=s.recv(64);assert chunk,'premature EOF';received+=chunk
assert received==b'echo:deferred\n',received
print(json.dumps({'recovered':received.decode().strip()}),flush=True);s.close()
'''
 client=spawn(stack,['ip','netns','exec','client','python3','-u','-c',client_code])
 connected=json.loads(line(client));failure=json.loads(line(client));assert connected['connect_returned'] and failure['application_timeout']
 pending=ns('target','ss','-nHt','state','syn-recv').stdout;assert '192.0.2.1:40101' in pending,pending
 blocked=counters('client');assert blocked['missing_final_ack']>0
 nft('client','delete table ip missing_ack');client.stdin.write('restore\n');client.stdin.flush()
 recovery=json.loads(line(client,6));assert recovery['recovered']=='echo:deferred'
 client.wait(timeout=2);assert client.returncode==0
 observed=packets(tap,path);flow=[p for p in observed if 40100 in (p['sport'],p['dport'])]
 syn=next(p for p in flow if p['flags']&0x12==0x02)
 synack=next(p for p in flow if p['flags']&0x12==0x12)
 third=next(p for p in flow if p['sport']==40100 and p['flags']&0x12==0x10)
 assert synack['ack']==(syn['seq']+1)%2**32
 assert third['seq']==(syn['seq']+1)%2**32 and third['ack']==(synack['seq']+1)%2**32
 print(json.dumps({'three_messages':[syn,synack,third],'normal_reply':normal.stdout.strip(),
 'client_connect_returned':connected['connect_returned'],'application_timeout':failure['application_timeout'],
 'server_still_syn_recv':pending.strip(),'drop_counters':blocked,'after_ack_filter_removed':recovery}))
"""
name='xbcatm-tcpflow-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The first connection uses client port 40100 and target port 18080. It sends a newline-terminated request and receives the expected echo. A packet capture supplies the actual SYN, SYN-ACK, and final ACK values. Assertions check that SYN-ACK acknowledges the client’s sequence plus one and that the third message acknowledges the server’s sequence plus one, using modulo arithmetic for the 32-bit sequence space.

The second connection uses client port 40101. An output filter in the client namespace drops ACK-bearing packets without SYN, FIN, or RST, including the final acknowledgment and subsequent data. Blocking that data matters: otherwise a later data segment could acknowledge the SYN-ACK and repair the intentionally missing third message. The example therefore makes the intended asymmetric state observable instead of relying on a narrow timing accident.

The client reports that connect returned successfully, but its application read reaches a deadline. At the same point, the server’s socket listing shows that exact client tuple in SYN-RECV. A rule counter confirms dropped packets. These independent observations establish the distinction between the client’s local API result, the server’s protocol state, and the failed application exchange.

Removing the filter lets the same connection finish the exchange and deliver the queued request. The client receives echo:deferred; it does not create a replacement socket to hide the original failure. Framed reads accumulate the response rather than assuming one recv call always returns the entire application message. The test verifies both the temporary failure and recovery after its controlled cause is removed.

Diagnose setup separately from useful service

Record the four-tuple, namespace, capture point, and state on each endpoint. A reset, repeated SYN, repeated SYN-ACK, and a completed handshake followed by an application timeout suggest different next checks. Capture both directions when possible: an outgoing acknowledgment visible locally is not proof that the remote endpoint received it. The fixture’s counter and remote socket state supply that missing context.

For service validation, follow the transport connection with a meaningful request and inspect its response. For identity and confidentiality, use the appropriate authenticated protocol above TCP. This exercise covers an ordinary connection without Fast Open or simultaneous open, and does not claim every valid connection begins with exactly three separately visible packets. Its central check is the sequence and state relationship, not a simplistic packet-count rule.

Sources

RFC 9293 TCP, sequence numbers, connection states and the three-way handshake, checked 2026-09-12.
Linux tcp(7), TCP socket behavior and Linux-specific controls, checked 2026-09-12.
Linux ss(8), examining established and SYN-RECV sockets, checked 2026-09-12.

HTTP keep-alive reuses one TCP connection

Fri, 28 May 2027 00:00:00 +0000

HTTP persistence lets several request and response exchanges use one connection. That can avoid repeated TCP setup and, for HTTPS, repeated TLS negotiation. The benefit depends on latency, request size, and whether another usable connection already exists. Persistence does not make application work disappear, nor does it promise that a socket will remain open indefinitely.

Reuse requires complete messages

HTTP/1.1 connections are persistent by default unless a participant signals Connection: close or closes the transport. Adding Connection: keep-alive is normally unnecessary. A client connection pool can retain an idle connection for another compatible request, subject to the client’s origin, proxy, certificate, and pool policies. Two separate command-line client processes generally do not share such a pool.

Both sides must know where each message ends. A response body can be delimited by an accurate Content-Length, chunked transfer coding, or the rules for responses without bodies. A response whose end is defined only by connection closure cannot leave that connection reusable. The client must consume the response body before reuse; the server must consume the request body or close. Otherwise leftover bytes could be mistaken for the following message.

HTTP persistence is also distinct from TCP keepalive. TCP keepalive probes a sufficiently idle transport to help detect an unreachable peer. It neither delimits HTTP messages nor overrides an application server’s idle timeout. Confusing the two can leave a client repeatedly trying connections that a proxy has already retired.

Sequential reuse also differs from pipelining: a client can wait for each response before sending the next request and still reuse the connection. HTTP/1.1 pipelining sends requests ahead but requires responses in order. Persistence alone therefore does not provide the independent response streams associated with HTTP/2 multiplexing.

Each hop owns its timeout

An idle timeout measures inactivity according to a particular implementation; a request deadline bounds an individual operation. Treat them as separate controls. A reverse proxy has a client-side connection and an upstream connection, with potentially different protocols and timeout policies. Reuse observed between a browser and the edge proves nothing about reuse between that edge and the origin.

An idle pool should retire connections with the server’s known limits in mind, but timing races still exist: the server may close just as the client sends again. Monitor connection creation, successful reuse, resets, and pool occupancy alongside latency. Longer retention saves handshakes while reserving descriptors and memory. There is no universal timeout that follows from a site being busy.

When it breaks

A response with an incorrect length can hang the reader or corrupt the boundary of the next exchange. Diagnose framing before increasing timeouts. A client that abandons streamed responses without consuming or closing them can also exhaust its pool; queued requests then wait even though the server has spare capacity.

A stale connection may fail after the request reached the server. Reconnecting and replaying a payment automatically can duplicate it. Retry according to method semantics and the application’s idempotency contract, since a transport error alone does not establish whether work committed. HTTP/2 multiplexes streams but forbids HTTP/1.x connection-specific headers such as Connection and Keep-Alive; copying these headers into an HTTP/2 configuration is not a reuse optimization.

Check it quickly

Run this complete Python 3 experiment. It binds an available loopback port, sends four sequential requests, and shuts down its test server. No external service is required; the HTTP server is a local fixture, not a deployment example.

from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread

class Handler(BaseHTTPRequestHandler):
 protocol_version = "HTTP/1.1"

 def do_GET(self):
 self.server.connections.append(self.connection)
 self.send_response(200)
 self.send_header("Content-Length", "2")
 if self.path == "/close":
 self.send_header("Connection", "close")
 self.end_headers()
 self.wfile.write(b"ok")

 def log_message(self, *args):
 pass

with ThreadingHTTPServer(("127.0.0.1", 0), Handler) as server:
 server.connections = []
 worker = Thread(target=server.serve_forever, daemon=True)
 worker.start()
 client = HTTPConnection(*server.server_address, timeout=3)
 try:
 for path in ("/one", "/two", "/close", "/four"):
 client.request("GET", path)
 response = client.getresponse()
 assert response.status == 200
 assert response.read() == b"ok"
 sockets = server.connections
 print("reused:", sockets[0] is sockets[1] is sockets[2])
 print("new after close:", sockets[2] is not sockets[3])
 finally:
 client.close()
 server.shutdown()
 worker.join()

Expect reused: True and new after close: True. The complete body reads permit the first three exchanges to share a socket; the explicit close makes the fourth reconnect. These observations establish local HTTP/1.1 reuse, not TLS savings, browser pool behavior, or production proxy settings. Inspect connection identifiers on each real hop before extending that conclusion to a deployed service.

Sources

RFC 9112: HTTP/1.1 connection management, framing, persistence, retries, and closure, checked 2026-09-06.
MDN: Connection management in HTTP/1.x, persistent connections and their costs, checked 2026-09-06.
RFC 9113: HTTP/2 connection-specific fields, fields prohibited in HTTP/2 messages, checked 2026-09-06.

MSS clamping can repair one TCP path without repairing its MTU feedback

Sun, 24 Jan 2027 00:00:00 +0000

TCP’s Maximum Segment Size option advertises the receiving endpoint’s maximum TCP data size under the option’s defined accounting rules. Each direction has its own advertisement; the two values need not match. A router can reduce those advertisements while forwarding SYN messages so later data segments fit a constrained path. That changes TCP behavior for the affected connections, not the underlying link MTU itself.

When it breaks

An intermediate bottleneck can be smaller than either endpoint interface. If classic path-MTU feedback is filtered, a connection may establish successfully and then stall on larger data. Reducing MSS can help that TCP flow avoid the oversized packets, but the missing feedback remains missing. Treating the workaround as a complete network repair leaves other protocols and traffic patterns exposed to the original limitation.

MSS is not the full IP packet size. For an IPv4 path MTU of 1280 with ordinary twenty-byte IPv4 and TCP base headers, an advertised MSS of 1240 follows the basic calculation. Actual IP or TCP options require additional allowance when forming data segments. Confusing the advertisement with a final wire-packet length can make a seemingly conservative value too large.

Direction and timing matter too. Editing only one SYN direction leaves the other endpoint’s advertisement unchanged. Updating a rule does not rewrite options already exchanged by existing connections. A clamp should reduce an excessive advertisement, not raise a peer’s already smaller limit. Scope such policy to the real bottleneck and relevant flow instead of treating one hard-coded value as suitable for every network.

Check it quickly

Save this as check-mss-clamp.py and run python3 check-mss-clamp.py with Docker. The disposable environment uses Python 3.11.16, iproute2 6.1.0, nftables 1.0.6, and tcpdump 4.99.3, tested on Linux 6.12.76. NET_ADMIN and SYS_ADMIN enable four nested namespaces. Endpoint links have MTU 1500; the link between two routers has MTU 1280:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess,json,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time
def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,process.stderr.read();return value
@contextmanager
def lab():
 with TemporaryDirectory(prefix='mtu-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc));stack.callback(run,'umount',str(proc))
 for name in ['client','r1','r2','target']:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False);ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in ['client','r1','r2','target']
 path=proc/'sys'/key.replace('.','/')
 ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r});p.write_text({str(value)!r});assert p.read_text().strip()=={str(value)!r}')
 yield root,stack,setting
def topology(setting,ipv6=False):
 links=[('client','c0','r1','a0',1,2,1,1500),('r1','a1','r2','b0',2,1,2,1280),('r2','b1','target','t0',3,1,2,1500)]
 for left,left_if,right,right_if,network,left_host,right_host,mtu in links:
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,host in [(left,left_if,left_host),(right,right_if,right_host)]:
 run('ip','link','set',interface,'netns',name);ip(name,'link','set',interface,'mtu',str(mtu))
 address=f'2001:db8:{network}::{host}/64' if ipv6 else f'10.210.{network}.{host}/30'
 ip(name,'addr','add',address,'dev',interface,*(['nodad'] if ipv6 else []));ip(name,'link','set',interface,'up')
 family='-6' if ipv6 else '-4'
 for name in ['r1','r2']:setting(name,'net.ipv6.conf.all.forwarding' if ipv6 else 'net.ipv4.ip_forward',1)
 routes=[('client','default','2001:db8:1::1' if ipv6 else '10.210.1.1'),('target','default','2001:db8:3::1' if ipv6 else '10.210.3.1'),
 ('r1','2001:db8:3::/64' if ipv6 else '10.210.3.0/30','2001:db8:2::2' if ipv6 else '10.210.2.2'),
 ('r2','2001:db8:1::/64' if ipv6 else '10.210.1.0/30','2001:db8:2::1' if ipv6 else '10.210.2.1')]
 for name,destination,gateway in routes:ip(name,family,'route','add',destination,'via',gateway)
 if ipv6:
 deadline=time.monotonic()+4
 while any('tentative' in ip(name,'-6','address','show').stdout for name in ['client','r1','r2','target']):
 assert time.monotonic()<deadline,'IPv6 link-local DAD did not finish';time.sleep(.02)
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 result={}
 for item in json.loads(ns(name,'nft','-j','list','ruleset').stdout)['nftables']:
 rule=item.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:result[rule['comment']]=expression['counter']['packets']
 return result
def capture(stack,root,name,interface,expression,label):
 path=root/(label+'.pcap')
 process=subprocess.Popen(['ip','netns','exec',name,'tcpdump','--immediate-mode','-Z','root','-U','-n','-i',interface,'-w',str(path),expression],stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);deadline=time.monotonic()+3
 while time.monotonic()<deadline:
 assert process.poll() is None,'capture exited'
 if select.select([process.stderr],[],[],.1)[0] and 'listening on' in process.stderr.readline():return process,path
 raise AssertionError('capture did not become ready')
def captured(process,path):
 process.send_signal(signal.SIGINT);_,diagnostic=process.communicate(timeout=3);assert process.returncode==0,diagnostic
 return run('tcpdump','-nn','-vv','-r',str(path)).stdout


def server(stack):
 code=r'''
import socketserver,hashlib
class Handler(socketserver.StreamRequestHandler):
 def handle(self):
 self.request.settimeout(3)
 header=self.rfile.readline()
 if not header:return
 length=int(header);data=self.rfile.read(length)
 if len(data)!=length:return
 self.wfile.write(hashlib.sha256(data).hexdigest().encode()+b'\n');self.wfile.flush()
class Server(socketserver.ThreadingTCPServer):allow_reuse_address=True;daemon_threads=True
service=Server(('10.210.3.2',18080),Handler);print('ready',flush=True);service.serve_forever()
'''
 process=spawn(stack,['ip','netns','exec','target','python3','-u','-c',code]);assert line(process).strip()=='ready'
def transfer(size):
 code=r'''
import socket,json,hashlib,sys,struct
size=int(sys.argv[1]);stream=socket.socket();stream.settimeout(1.5);connected=False
try:
 stream.connect(('10.210.3.2',18080));connected=True
 stream.sendall(str(size).encode()+b'\n'+b'x'*size)
 with stream.makefile('rb') as incoming:reply=incoming.readline().decode().strip()
 assert reply==hashlib.sha256(b'x'*size).hexdigest()
 print(json.dumps({'ok':True,'connected':True,'bytes':size,'sha256':reply,'snd_mss':stream.getsockopt(socket.IPPROTO_TCP,socket.TCP_MAXSEG)}))
except OSError as error:
 stream.setsockopt(socket.SOL_SOCKET,socket.SO_LINGER,struct.pack('ii',1,0))
 print(json.dumps({'ok':False,'connected':connected,'type':type(error).__name__,'errno':error.errno}))
finally:stream.close()
'''
 return json.loads(ns('client','python3','-c',code,str(size)).stdout)
def hide_fragmentation_feedback():
 nft('r1','''table ip hidden_mtu {
 chain output { type filter hook output priority 0; policy accept;
 icmp type destination-unreachable icmp code frag-needed counter drop comment "fragmentation_feedback_dropped"
 }
}
''')
with lab() as (root,stack,setting):
 topology(setting);setting('client','net.ipv4.tcp_mtu_probing',0);server(stack);hide_fragmentation_feedback()
 failed=transfer(65536);assert failed['connected'] and not failed['ok'] and failed['type']=='TimeoutError'
 initial_counter=counters('r1')['fragmentation_feedback_dropped'];assert initial_counter>0
 nft('r1','''table ip clamp {
 chain forward { type filter hook forward priority 0; policy accept;
 tcp flags & syn == syn tcp option maxseg size > 1240 tcp option maxseg size set 1240 counter comment "mss_rewritten"
 }
}
''')
 tap,path=capture(stack,root,'client','c0','tcp port 18080','mss')
 recovered=transfer(65536);assert recovered['ok'] and recovered['snd_mss']<=1240,recovered
 trace=captured(tap,path);assert 'mss 1240' in trace,trace
 after_transfer=counters('r1');assert after_transfer['mss_rewritten']>=2 and after_transfer['fragmentation_feedback_dropped']==initial_counter,after_transfer
 large_ping=ns('client','ping','-n','-M','do','-s','1400','-c','1','-W','1','10.210.3.2',check=False)
 assert large_ping.returncode!=0 and counters('r1')['fragmentation_feedback_dropped']>initial_counter
 print(json.dumps({'before_clamp':failed,'after_clamp':recovered,'counters_after_tcp':after_transfer,
 'icmp_echo_still_fails':{'returncode':large_ping.returncode,'stdout':large_ping.stdout,'stderr':large_ping.stderr},'capture':trace}))
"""
name='xbcatm-mtupath-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN','--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The first router suppresses outgoing ICMP fragmentation-needed messages. The client disables its separate TCP MTU probing feature, making this a controlled classic-discovery failure. A 65536-byte transfer establishes TCP and then reaches its application deadline. The router counter confirms feedback was suppressed. The failed socket is aborted so its remaining retransmissions cannot be mistaken for packets from the later comparison.

The script adds a forwarding rule matching SYN-bearing TCP packets whose advertised MSS exceeds 1240. It sets that option to 1240 and counts the changes. SYN-ACK also has SYN set, so the rule covers the corresponding return advertisement. The comparison uses a fresh connection because an already established stream would retain the options it received during its own setup.

The new 65536-byte transfer succeeds and verifies the returned SHA-256 digest. The capture shows the reduced MSS, and the rule counter records changes in both handshake directions. The effective sending MSS is 1228 in this run because the active TCP options consume additional space. The fragmentation-feedback drop counter does not increase during this successful transfer, showing that the previous feedback path was not needed to complete it.

An IPv4 echo probe with a 1400-byte payload and Don’t Fragment set still fails afterward. The suppressed-feedback counter increases again. This larger ICMP packet is outside the TCP MSS mechanism: reducing an option in a TCP handshake did not shrink it or repair the router’s feedback policy. The paired outcomes define the workaround’s actual scope through traffic, not just a rule listing.

Keep the workaround tied to observed packet constraints

The local capture contains offload representations that can be larger than the resulting TCP segments and can carry apparent partial checksums. Do not conclude from those records alone that MSS was ignored or that giant packets crossed the 1280-byte link. The option, socket diagnostic, counters, and verified application response provide complementary evidence at different processing stages.

Derive a suitable bound from the actual encapsulation and IP version. An IPv6 outer header or additional tunnel overhead changes the packet budget, and path changes can invalidate a previously adequate fixed value. The rule in this isolated topology uses a known bottleneck; it is not a discovery algorithm that learns every later route change automatically.

After applying a production policy, test both new connections and relevant non-TCP traffic. Preserve peer advertisements that are already smaller, and inspect both directions when transfers differ by upload versus download. Prefer correcting required feedback and path configuration where possible. MSS adjustment is useful when its constrained role is explicit and the intended TCP exchange is actually verified.

Sources

RFC 6691 TCP Options and MSS, MSS advertisement and accounting for actual header options, checked 2026-09-12.
nftables packet-header mangling, editing MSS on SYN and SYN-ACK and directional advertisements, checked 2026-09-12.
RFC 1191 Path MTU Discovery, the underlying IPv4 discovery problem that clamping does not repair, checked 2026-09-12.
Linux segmentation offloads, local offload representations versus final TCP segments, checked 2026-09-12.

Path MTU discovery can fail while handshakes and small requests still work

Sat, 14 Nov 2026 00:00:00 +0000

The path MTU is the smallest link MTU along a particular route. A sender cannot determine it solely from its own interface’s MTU because an intermediate link may carry smaller packets. Classic IPv4 Path MTU Discovery sends packets with Don’t Fragment set and uses ICMP fragmentation-needed feedback to reduce the estimated packet size. A working route and a working size estimate are separate requirements.

When it breaks

A tunnel or another constrained intermediate link can make large transfers fail while a TCP handshake and small request succeed. Their packets fit through the bottleneck, so those successes do not validate full-sized data. Repeatedly testing only a short ping or a tiny health response may therefore miss a path-MTU problem that affects uploads, larger responses, or sustained transfers.

Filtering the required ICMP feedback can create a black hole for oversized packets. The sender sees missing progress but lacks the classic signal identifying the smaller MTU. Other probing mechanisms may help, depending on transport and configuration; this is not proof that every TCP stack always hangs under such filtering. Identify which discovery or black-hole recovery mechanism is actually enabled before drawing conclusions.

Path information can also be cached. A test conducted after a successful discovery may behave differently from a fresh client, even after a firewall rule changes. Reusing that state can hide a broken feedback path. A route change or asymmetric return path complicates the picture further, so record direction, source, destination, and the conditions under which the estimate was learned.

Check it quickly

Save this as check-path-mtu.py and run python3 check-path-mtu.py with Docker. The disposable environment uses Python 3.11.16, iproute2 6.1.0, nftables 1.0.6, and tcpdump 4.99.3, tested on Linux 6.12.76. NET_ADMIN and SYS_ADMIN support four nested namespaces. Two routers connect the endpoints through newly created veth links; no host route or interface is changed:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess,json,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time
def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,process.stderr.read();return value
@contextmanager
def lab():
 with TemporaryDirectory(prefix='mtu-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc));stack.callback(run,'umount',str(proc))
 for name in ['client','r1','r2','target']:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False);ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in ['client','r1','r2','target']
 path=proc/'sys'/key.replace('.','/')
 ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r});p.write_text({str(value)!r});assert p.read_text().strip()=={str(value)!r}')
 yield root,stack,setting
def topology(setting,ipv6=False):
 links=[('client','c0','r1','a0',1,2,1,1500),('r1','a1','r2','b0',2,1,2,1280),('r2','b1','target','t0',3,1,2,1500)]
 for left,left_if,right,right_if,network,left_host,right_host,mtu in links:
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,host in [(left,left_if,left_host),(right,right_if,right_host)]:
 run('ip','link','set',interface,'netns',name);ip(name,'link','set',interface,'mtu',str(mtu))
 address=f'2001:db8:{network}::{host}/64' if ipv6 else f'10.210.{network}.{host}/30'
 ip(name,'addr','add',address,'dev',interface,*(['nodad'] if ipv6 else []));ip(name,'link','set',interface,'up')
 family='-6' if ipv6 else '-4'
 for name in ['r1','r2']:setting(name,'net.ipv6.conf.all.forwarding' if ipv6 else 'net.ipv4.ip_forward',1)
 routes=[('client','default','2001:db8:1::1' if ipv6 else '10.210.1.1'),('target','default','2001:db8:3::1' if ipv6 else '10.210.3.1'),
 ('r1','2001:db8:3::/64' if ipv6 else '10.210.3.0/30','2001:db8:2::2' if ipv6 else '10.210.2.2'),
 ('r2','2001:db8:1::/64' if ipv6 else '10.210.1.0/30','2001:db8:2::1' if ipv6 else '10.210.2.1')]
 for name,destination,gateway in routes:ip(name,family,'route','add',destination,'via',gateway)
 if ipv6:
 deadline=time.monotonic()+4
 while any('tentative' in ip(name,'-6','address','show').stdout for name in ['client','r1','r2','target']):
 assert time.monotonic()<deadline,'IPv6 link-local DAD did not finish';time.sleep(.02)
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 result={}
 for item in json.loads(ns(name,'nft','-j','list','ruleset').stdout)['nftables']:
 rule=item.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:result[rule['comment']]=expression['counter']['packets']
 return result
def capture(stack,root,name,interface,expression,label):
 path=root/(label+'.pcap')
 process=subprocess.Popen(['ip','netns','exec',name,'tcpdump','--immediate-mode','-Z','root','-U','-n','-i',interface,'-w',str(path),expression],stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);deadline=time.monotonic()+3
 while time.monotonic()<deadline:
 assert process.poll() is None,'capture exited'
 if select.select([process.stderr],[],[],.1)[0] and 'listening on' in process.stderr.readline():return process,path
 raise AssertionError('capture did not become ready')
def captured(process,path):
 process.send_signal(signal.SIGINT);_,diagnostic=process.communicate(timeout=3);assert process.returncode==0,diagnostic
 return run('tcpdump','-nn','-vv','-r',str(path)).stdout


def server(stack):
 code=r'''
import socketserver,hashlib
class Handler(socketserver.StreamRequestHandler):
 def handle(self):
 self.request.settimeout(3)
 header=self.rfile.readline()
 if not header:return
 length=int(header);data=self.rfile.read(length)
 if len(data)!=length:return
 self.wfile.write(hashlib.sha256(data).hexdigest().encode()+b'\n');self.wfile.flush()
class Server(socketserver.ThreadingTCPServer):allow_reuse_address=True;daemon_threads=True
service=Server(('10.210.3.2',18080),Handler);print('ready',flush=True);service.serve_forever()
'''
 process=spawn(stack,['ip','netns','exec','target','python3','-u','-c',code]);assert line(process).strip()=='ready'
def transfer(size):
 code=r'''
import socket,json,hashlib,sys,struct
size=int(sys.argv[1]);stream=socket.socket();stream.settimeout(1.5);connected=False
try:
 stream.connect(('10.210.3.2',18080));connected=True
 stream.sendall(str(size).encode()+b'\n'+b'x'*size)
 with stream.makefile('rb') as incoming:reply=incoming.readline().decode().strip()
 assert reply==hashlib.sha256(b'x'*size).hexdigest()
 print(json.dumps({'ok':True,'connected':True,'bytes':size,'sha256':reply,'snd_mss':stream.getsockopt(socket.IPPROTO_TCP,socket.TCP_MAXSEG)}))
except OSError as error:
 stream.setsockopt(socket.SOL_SOCKET,socket.SO_LINGER,struct.pack('ii',1,0))
 print(json.dumps({'ok':False,'connected':connected,'type':type(error).__name__,'errno':error.errno}))
finally:stream.close()
'''
 return json.loads(ns('client','python3','-c',code,str(size)).stdout)
def hide_fragmentation_feedback():
 nft('r1','''table ip hidden_mtu {
 chain output { type filter hook output priority 0; policy accept;
 icmp type destination-unreachable icmp code frag-needed counter drop comment "fragmentation_feedback_dropped"
 }
}
''')
results={}
for hidden in [False,True]:
 with lab() as (root,stack,setting):
 topology(setting);setting('client','net.ipv4.tcp_mtu_probing',0);server(stack)
 if hidden:hide_fragmentation_feedback()
 small=transfer(128);assert small['ok']
 before=ip('client','route','get','10.210.3.2').stdout
 tap,path=capture(stack,root,'client','c0','icmp or tcp port 18080','pmtu')
 large=transfer(65536);trace=captured(tap,path);after=ip('client','route','get','10.210.3.2').stdout
 if hidden:
 assert not large['ok'] and large['connected'] and large['type']=='TimeoutError',large
 count=counters('r1');assert count['fragmentation_feedback_dropped']>0
 else:
 assert large['ok'] and 'mtu 1280' in after,(large,after)
 assert 'unreachable - need to frag (mtu 1280)' in trace,trace;count={}
 results[str(hidden)]={'small_transfer':small,'large_transfer':large,'route_before':before,'route_after':after,
 'icmp_feedback_filtered':hidden,'counter':count,'capture':trace}
print(json.dumps(results))
"""
name='xbcatm-mtupath-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN','--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

Both endpoint links have MTU 1500, while the router-to-router link has MTU 1280. This placement matters: neither endpoint simply advertises a smaller local-interface MSS during setup. The client explicitly disables TCP’s separate MTU probing feature so the comparison isolates classic ICMP-based discovery. Each trial creates fresh namespaces, avoiding a learned destination estimate leaking from the healthy case into the filtered case.

With feedback permitted, a 128-byte transfer succeeds first. A new connection then sends 65536 bytes and verifies the server’s SHA-256 response. The capture contains an ICMP message reporting that fragmentation is needed with MTU 1280. The client’s route lookup subsequently shows a cached MTU of 1280, and the socket reports an effective sending MSS of 1228 in this timestamp-enabled exchange.

The second topology suppresses only the first router’s outgoing fragmentation-needed messages. Its small transfer still succeeds, and the larger transfer completes TCP setup before timing out. The rule counter records actual suppressed feedback, while the client’s route lookup has not learned the smaller MTU. These observations connect size-dependent failure to the deliberately missing discovery signal instead of blaming a stopped listener.

Failed test sockets are explicitly aborted so their queued retransmissions do not contaminate subsequent observations. The 1.5-second application deadline bounds this exercise; it is not the operating system’s complete retry lifetime. Successful cases validate exact payload content, while failed cases preserve both the local connection result and timeout type. That distinction keeps a successful handshake from being reported as successful data delivery.

Interpret captures at the correct processing point

Local captures may show large offload representations or apparent partial checksums. Such records are not automatically the final segments placed on a physical link, so a large captured length does not prove that an oversized wire packet crossed the bottleneck. Correlate route state, ICMP feedback, effective MSS, and application results rather than interpreting one capture field without its processing context.

Use several meaningful payload sizes when diagnosing a path, and account for IP, transport, option, and tunnel overhead. A ping payload size is not the full IP packet size, and an application write is not one TCP segment. Preserve the original failing traffic direction because success with a small response in the reverse direction may not exercise the same constraint.

Restore required feedback or correct the actual path configuration, then repeat a representative transfer from a fresh state. A transport-specific workaround may relieve one application without repairing discovery for other protocols. This laboratory demonstrates classic IPv4 discovery and its deliberate failure; it does not certify every alternate probing algorithm or establish a universal timeout for an arbitrary internet path.

Sources

RFC 1191 Path MTU Discovery, IPv4 DF, fragmentation-needed feedback and path estimates, checked 2026-09-12.
Linux ip(7), path MTU discovery and cached destination information, checked 2026-09-12.
Linux IP sysctl documentation, TCP MTU probing as a separate configurable mechanism, checked 2026-09-12.
Linux segmentation offloads, the distinction between local packet representations and transmitted segments, checked 2026-09-12.
Linux checksum offloads, partial checksum interpretation in local captures, checked 2026-09-12.

TIME-WAIT protects a connection incarnation, not exclusive ownership of a port

Fri, 13 Nov 2026 00:00:00 +0000

TIME-WAIT is a TCP state retained after certain orderly connection closures. It gives the closing exchange time to complete if the final acknowledgment must be repeated and helps prevent delayed segments from an earlier connection incarnation being confused with a later one. The relevant identity includes both endpoint addresses and ports. A TIME-WAIT entry is not simply a universal lock on one local port number.

When it breaks

In an ordinary close, the endpoint initiating the close commonly enters TIME-WAIT after the peer’s FIN is acknowledged. That endpoint can be a client or a server; simultaneous close can involve both. Assuming TIME-WAIT always belongs on the server hides the workload actually creating connection churn. Inspect the state and endpoint tuple instead of inferring ownership from application roles.

High churn can constrain new connections to a destination, but a large TIME-WAIT count alone does not prove exhaustion or a leaked application file descriptor. The operating system can retain protocol state after the application closes its socket. Source-address choices, available ports, destination diversity, explicit binding, reuse options, and kernel behavior affect the practical limit. Measure connection failures alongside counts before diagnosing resource exhaustion.

Confusing TIME-WAIT with FIN-WAIT-2 leads to ineffective tuning. Linux’s tcp_fin_timeout controls a different orphaned-socket situation; it is not a general TIME-WAIT duration knob. Old advice about tcp_tw_recycle is also obsolete on current Linux. Reuse behavior has protocol-safety conditions, including timestamp-related checks, so changing a setting without understanding the actual socket path can produce misleading expectations.

Check it quickly

Save this as check-time-wait.py and run python3 check-time-wait.py with Docker. The disposable environment uses Python 3.11.16 and iproute2 6.1.0, tested on Linux 6.12.76. NET_ADMIN and SYS_ADMIN permit nested namespaces and their temporary proc mount. A new veth joins the client and target; the test changes only these namespaces, with no host service or interface involved:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,('child exited without a response',process.stderr.read())
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found


with lab(['client','target']) as (root,stack,setting):
 wire('client','c0','target','t0',['192.0.2.1/30'],['192.0.2.2/30'])
 setting('client','net.ipv4.tcp_tw_reuse',0)
 setting('client','net.ipv4.tcp_timestamps',0);setting('target','net.ipv4.tcp_timestamps',0)
 server_code=r'''
import socketserver,threading
class Handler(socketserver.BaseRequestHandler):
 def handle(self):
 while self.request.recv(4096):pass
class Server(socketserver.ThreadingTCPServer):allow_reuse_address=True;daemon_threads=True
servers=[Server(('192.0.2.2',port),Handler) for port in [18080,18081]]
for server in servers:threading.Thread(target=server.serve_forever,daemon=True).start()
print('ready',flush=True);threading.Event().wait()
'''
 service=spawn(stack,['ip','netns','exec','target','python3','-u','-c',server_code]);assert line(service).strip()=='ready'
 client_code=r'''
import json,socket
def exchange(destination):
 with socket.socket() as s:
 s.settimeout(2);s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1);s.bind(('192.0.2.1',40200))
 s.connect(('192.0.2.2',destination));s.sendall(b'closed cleanly');s.shutdown(socket.SHUT_WR)
 assert s.recv(4096)==b''
exchange(18080)
try:exchange(18080);same={'ok':True}
except OSError as error:same={'ok':False,'errno':error.errno,'type':type(error).__name__}
assert not same['ok'] and same['errno']==99,same
exchange(18081)
print(json.dumps({'same_tuple':same,'same_source_port_new_destination_port':True}))
'''
 result=json.loads(ns('client','python3','-c',client_code).stdout)
 rows=ns('client','ss','-nHto','state','time-wait').stdout
 assert '192.0.2.1:40200' in rows and '192.0.2.2:18080' in rows and '192.0.2.2:18081' in rows,rows
 target=ns('target','ss','-nHto','state','time-wait').stdout;assert ':40200' not in target,target
 print(json.dumps({'client_active_close':result,'client_time_wait':rows.strip(),'target_time_wait':target.strip(),
 'timestamps_disabled_in_both_namespaces':True,'client_tcp_tw_reuse':ns('client','sysctl','-n','net.ipv4.tcp_tw_reuse').stdout.strip()}))
"""
name='xbcatm-tcpflow-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

To make an immediate duplicate-tuple rejection reproducible, the fixture disables timestamps in both namespaces and sets client tcp_tw_reuse to zero. These are explicit experiment controls, not a proposed production tuning profile. Timestamp-enabled sockets can take different safe-reuse paths. The matching kernel source contains timestamp-dependent checks, which is why the test does not claim that one reuse setting determines every possible outcome.

The target listens on ports 18080 and 18081. The client sets SO_REUSEADDR before binding its source port to 40200, connects to port 18080, sends data, and shuts down its writing direction first. The target drains the stream and closes, and the client reads EOF before closing its descriptor. This establishes the client as the active closer through real protocol traffic.

Immediately reconnecting from that same source address and port to the same destination address and port fails with Linux EADDRNOTAVAIL, errno 99, under the configured controls. Changing only the destination port to 18081 succeeds while retaining source port 40200. Both sockets use SO_REUSEADDR; the comparison therefore tests distinct connection tuples without changing the binding option between attempts.

The client’s socket listing then shows two TIME-WAIT entries sharing local port 40200, one for each target port. The target has no corresponding TIME-WAIT entries in this run. Each client timer is approximately sixty seconds when observed. The fixture records those timers; it does not wait for expiry or claim that this measured Linux duration is a universal TCP constant.

Reduce unnecessary churn before changing reuse policy

Where the application protocol allows it, connection reuse can reduce repeated setup and teardown. A pool still needs bounded size, healthy connection handling, and appropriate deadlines; retaining a dead connection is not a remedy for churn. Check whether the application actively closes after every small operation or whether a proxy timeout or server policy forces repeated reconnects.

When investigating a limit, record the failing system call and error, source and destination tuples, active connection counts, and TIME-WAIT distribution. A bind failure, a connect failure, and an application timeout are different observations. This example deliberately binds a source port; an application that lets the kernel choose an ephemeral port follows a different allocation path and needs its own representative test.

Keep the protocol’s delayed-segment and closing-handshake protections in the diagnosis. The useful objective is reliable new connections at the required workload, not eliminating a state that TCP intentionally retains. Validate any proposed change against the deployed kernel, socket options, and traffic pattern, and verify successful application exchanges as well as the resulting connection-state counts.

Sources

RFC 9293 TCP, connection termination, TIME-WAIT and delayed segments, checked 2026-09-12.
Linux tcp(7), socket reuse controls and the separate FIN-WAIT-2 timeout, checked 2026-09-12.
Linux 6.12.76 TCP IPv4 implementation, timestamp-dependent checks when reusing a TIME-WAIT connection, checked 2026-09-12.

TCP congestion control belongs to each sender, and peers can use different algorithms

Thu, 12 Nov 2026 00:00:00 +0000

TCP congestion control limits how a sender injects data into a network that has finite shared capacity. Its congestion window constrains outstanding data, while pacing, when used, controls how transmissions are distributed over time. The receiver’s advertised window imposes a separate flow-control limit. These mechanisms interact, but they describe different constraints and should not be collapsed into one ambiguous sending-speed setting.

When it breaks

The two endpoints do not need to select an identical congestion-control algorithm. Each endpoint controls the traffic it sends, so a Reno client can communicate with a CUBIC server. TCP setup negotiates transport options, not a requirement that the two sender algorithms have matching names. Diagnosing different controller names as a protocol mismatch can distract from a real route, buffering, or application failure.

Algorithm availability is also an actual runtime property. A configuration string naming a controller does not prove the kernel supports it or permits the calling process to select it. On Linux, TCP_CONGESTION sets a per-socket choice, subject to available and allowed algorithms. Check the result and read the selected name back instead of silently assuming that a requested label became effective.

Loss recovery is not one universal curve. A retransmission timeout, fast recovery, explicit congestion notification, and model-based estimation can produce different responses depending on the controller. Loss may come from congestion or another cause; the sender still has to respond under its algorithm’s rules. Increasing buffers cannot automatically fix a sender that is repeatedly losing packets or waiting for acknowledgment.

Check it quickly

Save this as check-congestion-control.py and run python3 check-congestion-control.py with Docker. The disposable environment uses Python 3.11.16, iproute2 6.1.0, and nftables 1.0.6, tested on Linux 6.12.76. NET_ADMIN and SYS_ADMIN support nested namespaces. A new veth link receives ten milliseconds of added egress delay in each direction; no host interface or production flow is modified:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,('child exited without a response',process.stderr.read())
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found


info_code=r'''
import struct
def tcp_info(stream):
 raw=stream.getsockopt(socket.IPPROTO_TCP,socket.TCP_INFO,104);assert len(raw)==104
 offsets={'snd_mss':16,'unacked':24,'rtt_us':68,'snd_ssthresh':76,'snd_cwnd':80,'total_retrans':100}
 return {'ca_state':raw[1],**{key:struct.unpack_from('=I',raw,offset)[0] for key,offset in offsets.items()}}
'''
with lab(['client','target']) as (root,stack,setting):
 wire('client','c0','target','t0',['192.0.2.1/30'],['192.0.2.2/30'])
 available=ns('client','sysctl','-n','net.ipv4.tcp_available_congestion_control').stdout.split();assert {'reno','cubic'}<=set(available),available
 for name,interface in [('client','c0'),('target','t0')]:
 ns(name,'tc','qdisc','add','dev',interface,'root','netem','delay','10ms','limit','10000')
 server_code=r'''
import socket,json,hashlib
listener=socket.socket();listener.setsockopt(socket.IPPROTO_TCP,socket.TCP_CONGESTION,b'cubic')
listener.bind(('192.0.2.2',18080));listener.listen();print('listening',flush=True)
with listener.accept()[0] as connection:
 connection.settimeout(8)
 print(json.dumps({'algorithm':connection.getsockopt(socket.IPPROTO_TCP,socket.TCP_CONGESTION,16).split(b'\0')[0].decode()}),flush=True)
 for size in [65536,16384]:
 data=b''
 while len(data)<size:
 chunk=connection.recv(size-len(data));assert chunk;data+=chunk
 assert data==b'x'*size
 connection.sendall((hashlib.sha256(data).hexdigest()+'\n').encode())
'''
 service=spawn(stack,['ip','netns','exec','target','python3','-u','-c',server_code]);assert line(service).strip()=='listening'
 client_code=info_code+r'''
import socket,json,sys,time,hashlib
with socket.socket() as unsupported:
 try:unsupported.setsockopt(socket.IPPROTO_TCP,socket.TCP_CONGESTION,b'xbcatm_absent');raise AssertionError('unexpected available algorithm')
 except OSError as error:selection_failure={'errno':error.errno,'type':type(error).__name__}
assert selection_failure['errno']==2,selection_failure
stream=socket.socket();stream.settimeout(8);stream.setsockopt(socket.IPPROTO_TCP,socket.TCP_CONGESTION,b'reno')
stream.bind(('192.0.2.1',40600));stream.connect(('192.0.2.2',18080))
with stream.makefile('rb') as incoming:
 stream.sendall(b'x'*65536);assert incoming.readline().decode().strip()==hashlib.sha256(b'x'*65536).hexdigest()
 baseline=tcp_info(stream);assert baseline['snd_cwnd']>1
 algorithm=stream.getsockopt(socket.IPPROTO_TCP,socket.TCP_CONGESTION,16).split(b'\0')[0].decode()
 print(json.dumps({'baseline':baseline,'algorithm':algorithm,'unavailable_algorithm':selection_failure}),flush=True)
 assert sys.stdin.readline().strip()=='drop';stream.sendall(b'x'*16384)
 deadline=time.monotonic()+4;samples=[]
 while time.monotonic()<deadline:
 state=tcp_info(stream);samples.append(state)
 if state['total_retrans']>baseline['total_retrans'] and state['snd_cwnd']==1 and state['ca_state']==4:break
 time.sleep(.01)
 else:raise AssertionError(('no timeout recovery observed',samples))
 print(json.dumps({'during_blackout':state}),flush=True)
 assert sys.stdin.readline().strip()=='restore'
 assert incoming.readline().decode().strip()==hashlib.sha256(b'x'*16384).hexdigest()
 print(json.dumps({'after_restore':tcp_info(stream),'verified_bytes':65536+16384}),flush=True)
stream.close()
'''
 client=spawn(stack,['ip','netns','exec','client','python3','-u','-c',client_code])
 baseline=json.loads(line(client));server_result=json.loads(line(service));assert baseline['algorithm']=='reno' and server_result['algorithm']=='cubic'
 nft('target','''table ip blackout {
 chain input { type filter hook input priority 0; policy accept;
 tcp sport 40600 tcp dport 18080 ip length > 60 counter drop comment "data_dropped"
 }
}
''')
 client.stdin.write('drop\n');client.stdin.flush();loss=json.loads(line(client,5))
 count=counters('target');assert count['data_dropped']>0
 nft('target','delete table ip blackout');client.stdin.write('restore\n');client.stdin.flush()
 recovery=json.loads(line(client,8));client.wait(timeout=2);service.wait(timeout=2);assert client.returncode==service.returncode==0
 print(json.dumps({'available':available,'client':baseline,'server':server_result,'loss':loss,'drop_counter':count,'recovery':recovery}))
"""
name='xbcatm-tcpflow-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The script first verifies that Reno and CUBIC are available. A deliberate request for an absent controller fails with ENOENT, errno two, surfaced by Python as FileNotFoundError. That exception refers to the unavailable controller selection, not a missing application data file. The real client explicitly selects Reno, while the server listener selects CUBIC and its accepted socket reports that inherited choice.

A first transfer sends 65536 bytes successfully and verifies the server’s digest. Both endpoints report their different algorithms while communication works. The client’s recorded baseline congestion window is fifty-six segments, with no retransmissions and no outstanding data at the observation point. This demonstrates that matching controller names are unnecessary and establishes a healthy baseline before inducing loss.

The target then installs an input rule dropping data for this connection. The client sends another 16384 bytes and samples TCP_INFO while acknowledgments fail to arrive. The kernel enters its loss state after a retransmission timeout: this run shows the congestion window reduced to one segment, the slow-start threshold set to twenty-eight, and the retransmission counter increasing. The target rule counter confirms actual packet drops.

Removing the rule allows the same connection to recover. The client verifies the second payload’s digest, for 81920 bytes delivered across both phases. The final window has grown again, and the retransmission total reflects recovery work. These are observations of this Reno timeout scenario, not a promise that every controller halves its threshold or returns to one segment for every kind of loss.

Measure the constraint that matters to the workload

The experiment samples Linux’s documented TCP_INFO layout and records MSS, congestion window, threshold, RTT, outstanding segments, and retransmissions. Window values are segment counts in these fields. A snapshot alone does not establish throughput: the application may have nothing to send, the receiver may limit progress, or queued data may still await transmission. Keep these distinctions when interpreting socket diagnostics.

For operational changes, record the effective controller on representative sockets and compare latency, throughput, loss, and fairness under the relevant workload. A single successful connection cannot certify behavior with competing senders or a different bottleneck. Likewise, this isolated loss rule is a controlled fault, not an estimate of ordinary internet packet-loss rates.

Use transport recovery evidence alongside application correctness. Retransmission can preserve the byte stream while increasing completion time, so a valid response does not mean the network was loss-free. Conversely, a lower window during recovery is not itself a broken connection. The useful outcome is verified delivery and an explanation of the sender’s measured response to the conditions it encountered.

Sources

RFC 5681 Congestion Control, congestion windows and retransmission-timeout recovery, checked 2026-09-12.
RFC 9438 CUBIC, CUBIC behavior and its relationship to congestion signals, checked 2026-09-12.
Linux tcp(7), per-socket congestion-controller selection, checked 2026-09-12.
Linux 6.12.76 TCP_INFO fields, congestion states, window, threshold and retransmission counters, checked 2026-09-12.

Persistent connections are the default

Sat, 24 Oct 2026 00:00:00 +0000

HTTP persistence allows multiple exchanges to share a connection, so repeated requests may avoid TCP setup and TLS negotiation. It is a protocol capability, not a promise that a particular socket stays open. Whether another request actually reuses it also depends on complete message framing, client pool policy, server behavior, and the connection’s current condition.

Default does not mean immortal

HTTP/1.1 permits persistence by default. A client does not normally need to send Connection: keep-alive to enable it. Connection: close instead announces that the connection will not continue after the current exchange. HTTP/1.0 used different persistence rules; do not copy that negotiation model into every modern request.

Reuse needs an unambiguous message boundary. An accurate Content-Length, chunked framing, or a no-body response rule can provide it. A response delimited by closing the connection cannot leave that connection reusable. The receiving side must finish consuming the message before interpreting subsequent bytes; simply seeing HTTP 200 does not establish that the response body is complete.

Reusing an established HTTPS connection performs no new TLS handshake. Opening a new connection with a cached TLS session is different: that connection still handshakes. TCP keepalive is different again; its idle probes do not renew HTTP message boundaries or override an application’s decision to close. HTTP/2 also has connection reuse, but not HTTP/1.x’s Connection header semantics.

Budget the pool and the operation separately

Idle sockets retain descriptors, buffers, and implementation state. In Go, MaxIdleConnsPerHost controls retained idle connections; MaxConnsPerHost limits dialing, active, and idle connections together. The former is not a total concurrency cap. A shared Transport owns the reusable pool, so constructing unrelated transports for successive calls discards opportunities for reuse.

IdleConnTimeout limits an idle connection’s lifetime; Client.Timeout limits a request, including connection acquisition and response-body reading. Neither name means the other. A reverse proxy also has separate downstream and upstream connections. Reuse and timeout observations on the browser-facing hop do not automatically describe its origin-facing pool.

When it breaks

A server can close while a client is about to reuse an idle connection. Such races require handling, but a reset does not prove the server performed no work. Retrying a non-idempotent operation without an application-level contract can duplicate effects. Increasing idle retention cannot remove this uncertainty.

Unconsumed bodies can reserve resources, delay reuse, or force closure. Complete or explicitly close each response according to the client’s documented behavior. Libraries may drain some abandoned bodies themselves, with implementation-specific limits; do not assume every early close immediately frees a reusable socket. Conversely, one failed reuse does not justify disabling persistence everywhere.

Check it quickly

Save this as persistence_probe.go and run go run persistence_probe.go with Go 1.23 or newer. The local fixture serves HTTP/1.1 over verified TLS 1.3, assigns identifiers to accepted TCP connections, and returns two-byte, length-delimited bodies. It changes neither public services nor the system trust store. One transport permits reuse; its separate clone disables it.

package main

import (
 "context"
 "crypto/tls"
 "fmt"
 "io"
 "net"
 "net/http"
 "net/http/httptest"
 "net/http/httptrace"
 "runtime"
 "strconv"
 "sync/atomic"
 "time"
)

type connectionKey struct{}

func must(err error) {
 if err != nil {
 panic(err)
 }
}

func check(ok bool, message string) {
 if !ok {
 panic(message)
 }
}

func main() {
 var accepted, handled atomic.Int64
 server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 handled.Add(1)
 w.Header().Set("X-Connection-ID", fmt.Sprint(r.Context().Value(connectionKey{})))
 w.Header().Set("X-Request-Connection", r.Header.Get("Connection"))
 w.Header().Set("Content-Length", "2")
 if r.URL.Path == "/close" {
 w.Header().Set("Connection", "close")
 }
 fmt.Fprint(w, "ok")
 }))
 server.Config.ConnContext = func(ctx context.Context, _ net.Conn) context.Context {
 return context.WithValue(ctx, connectionKey{}, accepted.Add(1))
 }
 server.TLS = &tls.Config{MinVersion: tls.VersionTLS13, MaxVersion: tls.VersionTLS13}
 server.StartTLS()
 defer server.Close()
 config := server.Client().Transport.(*http.Transport).TLSClientConfig.Clone()
 check(!config.InsecureSkipVerify, "certificate verification must stay enabled")
 transport := &http.Transport{
 TLSClientConfig: config, MaxConnsPerHost: 1, IdleConnTimeout: 10 * time.Second,
 }
 defer transport.CloseIdleConnections()
 noReuse := transport.Clone()
 noReuse.DisableKeepAlives = true
 defer noReuse.CloseIdleConnections()
 probe := func(label, route string, tr *http.Transport, wantID int64, wantReuse bool) {
 var got atomic.Int64
 var reused atomic.Bool
 trace := &httptrace.ClientTrace{GotConn: func(info httptrace.GotConnInfo) {
 got.Add(1)
 reused.Store(info.Reused)
 }}
 request, err := http.NewRequest(http.MethodGet, server.URL+route, nil)
 must(err)
 request = request.WithContext(httptrace.WithClientTrace(request.Context(), trace))
 client := &http.Client{Transport: tr, Timeout: 2 * time.Second}
 response, err := client.Do(request)
 must(err)
 body, readError := io.ReadAll(response.Body)
 closeError := response.Body.Close()
 must(readError)
 must(closeError)
 check(response.StatusCode == 200 && string(body) == "ok", "wrong response")
 check(response.Proto == "HTTP/1.1", "fixture requires HTTP/1.1")
 check(response.TLS.Version == tls.VersionTLS13 && len(response.TLS.VerifiedChains) > 0, "verified TLS 1.3 required")
 id, err := strconv.ParseInt(response.Header.Get("X-Connection-ID"), 10, 64)
 must(err)
 check(id == wantID && got.Load() == 1 && reused.Load() == wantReuse, "wrong connection reuse")
 check(response.Close == (route == "/close" || tr.DisableKeepAlives), "wrong close indication")
 wantHeader := ""
 if tr.DisableKeepAlives {
 wantHeader = "close"
 }
 check(response.Header.Get("X-Request-Connection") == wantHeader, "unexpected request Connection header")
 fmt.Printf("%s: conn=%d reused=%t closing=%t\n", label, id, reused.Load(), response.Close)
 }
 probe("cold", "/ok", transport, 1, false)
 probe("reuse", "/ok", transport, 1, true)
 probe("server close", "/close", transport, 1, true)
 probe("after server close", "/ok", transport, 2, false)
 transport.CloseIdleConnections()
 probe("after idle retirement", "/ok", transport, 3, false)
 probe("reuse disabled one", "/ok", noReuse, 4, false)
 probe("reuse disabled two", "/ok", noReuse, 5, false)
 check(accepted.Load() == 5 && handled.Load() == 7, "seven requests must use five TCP connections")
 fmt.Println(runtime.Version())
}

The first three requests use connection one, including the response announcing server closure. The next request therefore uses connection two. Explicitly retiring idle connections then makes the following request use connection three. The disabled-reuse transport uses connections four and five for its two requests. Seven requests reach five accepted TCP connections.

GotConn confirms the expected reuse flag for every request. The server echoes the received Connection field: ordinary requests have no such field, yet still reuse the socket; disabled-reuse requests carry close. This tests actual connections, protocol selection, certificate validation, body completion, and closure indications, not just a configuration object’s values.

All assertions passed on macOS with Go 1.27.0. Every body is read and closed, each request has a two-second timeout, and both transports are cleaned up. IdleConnTimeout is configured, but expiration is not tested: explicit retirement makes that branch deterministic. The fixture does not measure speedup, TLS resumption, TCP keepalive, pipelining, production resource limits, or proxy behavior. Apply those conclusions only after separate measurements.

Sources

RFC 9112: HTTP/1.1, authoritative persistence, message-boundary, closure, and retry rules, checked 2026-09-09.
MDN: Connection management in HTTP/1.x, hop-by-hop ownership and distinctions between persistence and pipelining, checked 2026-09-09.
Go net/http, transport pooling, connection limits, response lifecycle, and timeout semantics, checked 2026-09-09.

Tcp · XB Field Notes

The TCP backlog holds connections that are still syncing

The backlog argument is the accept queue

With syncookies on, the SYN queue stops mattering

The accept queue is where the pressure lands

When it breaks

Check it quickly

Sources

TCP slow start grows with acknowledged data, not merely with elapsed time

When it breaks

Check it quickly

Interpret a growth trace in its workload context

Sources

TCP window scaling enlarges receive-window encoding in each direction

When it breaks

Check it quickly

Separate buffer allocation, advertised capacity, and observed performance

Sources

Nagle and delayed ACK can stall the second part of a small request

When it breaks

Check it quickly

Choose framing and socket policy together

Sources

SYN cookies answer when the queue overflows

Two queues, two different symptoms

The cookie is the state you refused to store

What a cookie cannot carry

When it breaks

Check it quickly

Sources

The listen backlog holds connections waiting for accept

When it breaks

Check it quickly

Tune acceptance and waiting capacity together

Sources

TCP keepalive probes idle connections

A transport probe, not an application heartbeat

Configure the socket you actually use

When it breaks

Check it quickly

Further reading

Sources

SACK reports received ranges beyond a gap so recovery can target missing data

When it breaks

Check it quickly

Interpret feedback without promising perfect retransmission decisions

Sources

The TCP handshake establishes sequence spaces, not synchronized clocks

When it breaks

Check it quickly

Diagnose setup separately from useful service

Sources

HTTP keep-alive reuses one TCP connection

Reuse requires complete messages

Each hop owns its timeout

When it breaks

Check it quickly

Sources

MSS clamping can repair one TCP path without repairing its MTU feedback

When it breaks

Check it quickly

Keep the workaround tied to observed packet constraints

Sources

Path MTU discovery can fail while handshakes and small requests still work

When it breaks

Check it quickly

Interpret captures at the correct processing point

Sources

TIME-WAIT protects a connection incarnation, not exclusive ownership of a port

When it breaks

Check it quickly

Reduce unnecessary churn before changing reuse policy

Sources

TCP congestion control belongs to each sender, and peers can use different algorithms

When it breaks

Check it quickly

Measure the constraint that matters to the workload

Sources

Persistent connections are the default

Default does not mean immortal