Nagle and delayed ACK can stall the second part of a small request

Sun, 31 Oct 2027 00:00:00 +0000

Nagle’s algorithm reduces small TCP segments by holding additional short data while earlier data remains unacknowledged, unless enough data accumulates for a full segment. Delayed acknowledgment is a separate receiver behavior that can postpone an ACK briefly. The interaction matters when an application sends a request in multiple small writes and the peer needs the later bytes before producing a response.

When it breaks

An isolated small request is not automatically delayed by Nagle. If no earlier data is outstanding, the first small segment can be sent. The troublesome case has a first piece already in flight and a second short piece waiting behind it. If the receiver delays acknowledgment while its application waits for the complete request, progress can pause until the acknowledgment timer expires.

This is a bounded delay, not an indefinite protocol deadlock. An ACK can release the queued data, and accumulating enough data can also change the sender’s decision. The observed delay is not universally one additional RTT or a fixed forty milliseconds. Receiver policy, existing acknowledgment state, request framing, and network conditions determine the actual packet sequence and timing.

TCP_NODELAY disables Nagle on the selected socket; it does not disable every form of buffering, congestion control, receive-window flow control, or scheduler delay. TCP_QUICKACK on Linux requests a change in acknowledgment mode, but that mode is not permanent. Treating either option as a universal latency switch can hide application buffering or a different transport constraint that the option never addressed.

Check it quickly

Save this as check-nagle.py and run python3 check-nagle.py with Docker. The disposable environment uses Python 3.11.16, iproute2 6.1.0, and tcpdump 4.99.3, tested on Linux 6.12.76. NET_ADMIN and SYS_ADMIN enable nested namespaces. Client and target communicate over a new veth link; all settings stay inside that laboratory rather than changing host networking:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,('child exited without a response',process.stderr.read())
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found


def capture(stack,root,name,interface,expression,label):
 path=root/(label+'.pcap')
 process=subprocess.Popen(['ip','netns','exec',name,'tcpdump','--immediate-mode','-Z','root','-U','-n','-i',interface,'-w',str(path),expression],
 stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process)
 deadline=time.monotonic()+3
 while time.monotonic()<deadline:
 assert process.poll() is None,'tcpdump exited before listening'
 if select.select([process.stderr],[],[],.1)[0] and 'listening on' in process.stderr.readline():return process,path
 raise AssertionError('tcpdump did not become ready')
def packets(process,path):
 import struct,socket
 process.send_signal(signal.SIGINT);_,diagnostic=process.communicate(timeout=3)
 assert process.returncode==0,diagnostic
 raw=path.read_bytes();assert raw[:4]==bytes.fromhex('d4c3b2a1') and struct.unpack_from('<I',raw,20)[0]==1
 offset=24;result=[]
 while offset<len(raw):
 seconds,micros,size,original=struct.unpack_from('<IIII',raw,offset);offset+=16
 frame=raw[offset:offset+size];offset+=size
 if frame[12:14]!=b'\x08\x00' or frame[23]!=6:continue
 ihl=(frame[14]&15)*4;length=struct.unpack_from('!H',frame,16)[0];tcp=14+ihl
 sport,dport,seq,ack=struct.unpack_from('!HHII',frame,tcp);header=(frame[tcp+12]>>4)*4
 opts={};cursor=tcp+20
 while cursor<tcp+header:
 kind=frame[cursor]
 if kind==0:break
 if kind==1:cursor+=1;continue
 option_length=frame[cursor+1];data=frame[cursor+2:cursor+option_length];assert option_length>=2
 if kind==2:opts['mss']=struct.unpack('!H',data)[0]
 elif kind==3:opts['wscale']=data[0]
 elif kind==4:opts['sack_ok']=True
 elif kind==5:opts['sack']=[list(pair) for pair in struct.iter_unpack('!II',data)]
 cursor+=option_length
 body=frame[tcp+header:14+length]
 result.append({'t':seconds+micros/1e6,'src':socket.inet_ntoa(frame[26:30]),'dst':socket.inet_ntoa(frame[30:34]),
 'sport':sport,'dport':dport,'seq':seq,'ack':ack,'flags':frame[tcp+13],'window':struct.unpack_from('!H',frame,tcp+14)[0],
 'length':len(body),'options':opts,'preview':body[:24].hex()})
 return result
with lab(['client','target']) as (root,stack,setting):
 wire('client','c0','target','t0',['192.0.2.1/30'],['192.0.2.2/30'])
 setting('client','net.ipv4.tcp_autocorking',0)
 server_code=r'''
import socket,json
def read_exact(stream,length):
 data=b''
 while len(data)<length:
 chunk=stream.recv(length-len(data));assert chunk,'premature EOF';data+=chunk
 return data
listener=socket.socket();listener.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
listener.bind(('192.0.2.2',18080));listener.listen();print('listening',flush=True)
for trial in range(2):
 with listener.accept()[0] as connection:
 connection.settimeout(3);connection.setsockopt(socket.IPPROTO_TCP,socket.TCP_QUICKACK,0)
 print('configured',flush=True)
 request=read_exact(connection,2);assert request==b'ab',request
 connection.sendall(b'OK');print(json.dumps({'request':request.decode()}),flush=True)
'''
 service=spawn(stack,['ip','netns','exec','target','python3','-u','-c',server_code]);assert line(service).strip()=='listening'
 tap,path=capture(stack,root,'client','c0','tcp port 18080','nagle')
 client_code=r'''
import json,socket,sys,time
def read_exact(stream,length):
 data=b''
 while len(data)<length:
 chunk=stream.recv(length-len(data));assert chunk,'premature EOF';data+=chunk
 return data
nodelay=int(sys.argv[1]);s=socket.socket();s.settimeout(3)
s.setsockopt(socket.IPPROTO_TCP,socket.TCP_NODELAY,nodelay);s.bind(('192.0.2.1',40300+nodelay));s.connect(('192.0.2.2',18080))
print('connected',flush=True);assert sys.stdin.readline().strip()=='go'
start=time.monotonic();s.sendall(b'a');time.sleep(.002);s.sendall(b'b');reply=read_exact(s,2)
assert reply==b'OK';print(json.dumps({'nodelay':nodelay,'reply':reply.decode(),'elapsed_ms':(time.monotonic()-start)*1000}),flush=True);s.close()
'''
 outcomes=[]
 for nodelay in [0,1]:
 client=spawn(stack,['ip','netns','exec','client','python3','-u','-c',client_code,str(nodelay)])
 assert line(client).strip()=='connected' and line(service).strip()=='configured'
 client.stdin.write('go\n');client.stdin.flush();outcomes.append(json.loads(line(client)))
 assert json.loads(line(service))['request']=='ab';client.wait(timeout=2);assert client.returncode==0
 observed=packets(tap,path);timelines={}
 for nodelay in [0,1]:
 port=40300+nodelay;flow=[p for p in observed if port in (p['sport'],p['dport'])]
 first=next(p for p in flow if p['sport']==port and p['preview']=='61')
 second=next(p for p in flow if p['sport']==port and p['preview']=='62')
 assert (second['seq']-first['seq'])%2**32==1
 acks=[p for p in flow if p['dport']==port and p['flags']&0x10 and 0<(p['ack']-first['seq'])%2**32<=2]
 assert acks
 if not nodelay:
 intermediate=next(p for p in acks if (p['ack']-first['seq'])%2**32==1)
 assert first['t']<intermediate['t']<=second['t'] and second['t']-first['t']>.01
 else:assert second['t']<=acks[0]['t']
 timelines[nodelay]={'first_byte':first,'second_byte':second,'first_ack_covering_data':acks[0],
 'data_segment_gap_ms':(second['t']-first['t'])*1000}
 print(json.dumps({'trials':outcomes,'packet_timelines':timelines,'receiver_quickack_requested':0,'client_autocorking':0}))
"""
name='xbcatm-tcpflow-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The server waits for the complete request ab before replying OK. Before each trial, it requests non-quick acknowledgment mode on the accepted connection. The client disables namespace-local automatic corking to remove that additional coalescing mechanism from the comparison. Parent-child synchronization ensures the receiver has configured its socket before the client starts sending application bytes.

In the first trial, TCP_NODELAY is zero. The client sends a, pauses for two milliseconds, and sends b. Those writes are application actions, not a guarantee of immediate separate packets. The capture shows the first byte leaving, an ACK covering that byte arriving, and only then the second byte leaving. The program checks this ordering instead of attributing any slow response to Nagle without packet evidence.

In the recorded run, the two data segments were approximately forty-one milliseconds apart, and the complete exchange took roughly the same time. The second trial enables TCP_NODELAY and repeats the same framing and deliberate pause. The second byte leaves before the first data acknowledgment, with about two milliseconds between segments. The server receives the complete request and returns the same valid reply in both cases.

The timing values describe this controlled run, not a production latency guarantee. The fixture expects a visible delayed-ACK interaction under its stated settings and checks the causal packet order. A different kernel or receiver policy can require fresh investigation. Stream reads explicitly accumulate the expected number of bytes because recv, even with a requested length, does not generally promise an entire application message at once.

Choose framing and socket policy together

For a small logical request already available in memory, consider writing its complete representation together. That can avoid waiting for a later fragment while preserving useful transport batching. For an interactive stream that must deliver independent small updates promptly, TCP_NODELAY may be appropriate. Choose using the workload’s framing and measured behavior rather than applying one setting to every connection.

Capture both data and acknowledgments when a request periodically stalls. Determine whether the second piece is still in application buffering, queued in the sender, or already received by the peer. A timeout in application code alone cannot distinguish those stages. Correlate the capture with socket configuration and the point at which the receiver considers a request complete.

Also account for packet-rate and bandwidth costs when sending many tiny updates. Reducing an unnecessary batching delay can improve interaction while increasing segments per operation. Measure the relevant response latency and resource cost together. The useful result is a justified policy for a particular exchange, supported here by two successful requests and different, directly observed transmission ordering.

Sources

RFC 896 Congestion Control, coalescing small writes while earlier data remains unacknowledged, checked 2026-09-12.
RFC 1122 Host Requirements, delayed acknowledgment behavior and bounded acknowledgment delay, checked 2026-09-12.
Linux tcp(7), TCP_NODELAY and the temporary nature of TCP_QUICKACK, checked 2026-09-12.

Timing · XB Field Notes

Nagle and delayed ACK can stall the second part of a small request

When it breaks

Check it quickly

Choose framing and socket policy together

Sources