Networking · XB Field Notes

A veth pair carries frames between two endpoints; routing supplies their meaning

Sun, 21 Nov 2027 00:00:00 +0000

A veth pair consists of two interconnected virtual Ethernet devices. A frame transmitted through one is delivered to the other, where the receiving network stack or bridge handles it. The two ends can reside in separate Linux network namespaces, making the pair useful for container networking. Creating the pair supplies a link; addresses, routes, bridge membership, forwarding, and access policy are additional configuration.

When it breaks

An interface marked administratively UP can still lack carrier because its veth peer is down. Reading only the UP flag can therefore make an incomplete setup look ready. Inspect both endpoints and distinguish administrative state from LOWER_UP. A route may already name the interface even while the paired link cannot carry useful traffic, so a route listing alone is not a connectivity check.

The pair does not restrict IP payloads to the peer interface’s own address. Routing chooses which link should carry a packet, and Ethernet forwarding or the receiving stack decides what happens next. Traffic destined beyond a container’s immediate neighbor can traverse the pair toward a router or bridge. Saying that every packet not addressed to the peer stays local confuses link delivery with IP destination selection.

Endpoint lifetime is coupled too. Deleting one veth endpoint removes its peer, even if the peer is managed from another namespace. A cleanup action in the host-side context can therefore remove a container’s interface. Stopping traffic, removing routes, and deleting the pair need an intentional order when they affect a live workload; namespace separation does not make the two devices independent resources.

Check it quickly

Save this as check-veth.py and run python3 check-veth.py with Docker. The pinned Linux container uses Python 3.11.16 and iproute2 6.1.0, with NET_ADMIN and SYS_ADMIN for nested namespaces. It creates only disposable virtual devices. The left and right namespaces receive one endpoint each, with 192.0.2.1/30 and 192.0.2.2/30 respectively:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,'child exited without a response'
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found

def server(stack,name,address,port=18080,label='server'):
 code=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import json,socket,sys
address,port,label=sys.argv[1],int(sys.argv[2]),sys.argv[3]
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 body=json.dumps({'server':label,'peer':self.client_address,'path':self.path}).encode()
 self.send_response(200);self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
 def log_message(self,*args):pass
class Server(ThreadingHTTPServer):
 address_family=socket.AF_INET6 if ':' in address else socket.AF_INET
 def server_bind(self):
 if self.address_family==socket.AF_INET6:self.socket.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
 super().server_bind()
service=Server((address,port),Handler);print('ready',flush=True);service.serve_forever()
'''
 process=spawn(stack,['ip','netns','exec',name,'python3','-u','-c',code,address,str(port),label])
 assert line(process).strip()=='ready';return process
def request(name,address,port=18080):
 code=r'''
import http.client,json,sys
client=http.client.HTTPConnection(sys.argv[1],int(sys.argv[2]),timeout=.7)
try:
 client.request('GET','/lab-visible-message');response=client.getresponse();body=json.loads(response.read())
 assert response.status==200;print(json.dumps({'ok':True,'status':response.status,'body':body}))
except OSError as error:print(json.dumps({'ok':False,'type':type(error).__name__,'errno':error.errno}))
finally:client.close()
'''
 return json.loads(ns(name,'python3','-c',code,address,str(port)).stdout)

with lab(['left','right']) as (root,stack,setting):
 run('ip','link','add','left0','type','veth','peer','name','right0')
 run('ip','link','set','left0','netns','left');run('ip','link','set','right0','netns','right')
 ip('left','addr','add','192.0.2.1/30','dev','left0');ip('right','addr','add','192.0.2.2/30','dev','right0')
 ip('left','link','set','left0','up')
 before=json.loads(ip('left','-j','link','show','left0').stdout)[0]
 assert 'LOWER_UP' not in before['flags']
 failed=ns('left','ping','-n','-c','1','-W','1','192.0.2.2',check=False);assert failed.returncode!=0
 ip('right','link','set','right0','up')
 after={name:json.loads(ip(name,'-j','link','show',interface).stdout)[0] for name,interface in [('left','left0'),('right','right0')]}
 assert all('LOWER_UP' in v['flags'] for v in after.values())
 ip('left','neigh','flush','dev','left0');ip('right','neigh','flush','dev','right0')
 successful=ns('left','ping','-n','-c','1','-W','1','192.0.2.2');assert successful.returncode==0
 daemon=server(stack,'right','192.0.2.2',label='right');response=request('left','192.0.2.2');assert response['ok']
 ip('right','addr','add','203.0.113.9/32','dev','lo');ip('left','route','add','203.0.113.9/32','via','192.0.2.2')
 server(stack,'right','203.0.113.9',label='beyond-link-address');routed_response=request('left','203.0.113.9');assert routed_response['ok']
 ip('left','link','delete','left0');gone=ip('right','link','show','right0',check=False);assert gone.returncode!=0 and 'does not exist' in gone.stderr
 print(json.dumps({'one_end_up_flags':before['flags'],'one_end_up_ping_exit':failed.returncode,
 'both_up_flags':{n:v['flags'] for n,v in after.items()},'both_up_ping_exit':successful.returncode,'http':response,'other_ip_destination':routed_response,
 'deleting_one_removes_peer':True,'peer_absent_diagnostic':gone.stderr.strip()}))
"""
name='xbcatm-netpath-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

Initially only left0 is brought up. Its flags include UP and NO-CARRIER, with no LOWER_UP, because right0 remains down. A one-packet ping fails. This gives both a link-state observation and an actual unsuccessful exchange; neither is replaced by assuming that creation of the interface means the other endpoint has been configured correctly.

After bringing right0 up, both endpoints report LOWER_UP. The script clears neighbor entries from the failed attempt before making the healthy comparison, avoiding residual neighbor-resolution timers. A new ping succeeds, and a real HTTP request from left to 192.0.2.2 returns the right service’s identity. The cleared state is explicit; the exercise does not promise instantaneous recovery for every already-pending application request.

Next, the right namespace adds 203.0.113.9/32 on its loopback interface. The left namespace installs a host route to that address through 192.0.2.2. A second HTTP listener bound to 203.0.113.9 responds successfully across the same veth pair. The packet’s IP destination is not the peer’s link address, yet its selected path still uses that virtual Ethernet link.

This second destination is local to the right namespace, so the fixture does not claim to forward onward through another router. It isolates the distinction between a link’s neighboring address and a packet’s final IP destination. A larger topology can connect the receiving endpoint to a bridge or routed network, but those additional components need their own configuration and verification.

Finally, deleting left0 makes ip link show right0 fail in the right namespace with a device-not-found diagnostic. The peer has been removed, not merely placed administratively down. Both namespaces themselves still exist until cleanup. This separates the paired device lifecycle from the lifetime of the namespace that happened to contain one endpoint.

Diagnose the layers in order

Confirm which namespace owns each end and which names the runtime assigned. Then inspect administrative and carrier state, address configuration, neighbor resolution, and the route for the failing destination. A stale neighbor entry can delay recovery after a link change, while an absent route can fail before any frame is transmitted. These conditions require different repairs even when a high-level client reports only a connection timeout.

A pair is not a multiport switch. A bridge can join several such links at Ethernet level, while a router forwards between IP networks. Enabling an endpoint does not automatically add either behavior or an internet route. Document the actual attachment and filtering points, especially when a container runtime creates the devices and another component manages routes or firewall policy.

Treat deletion as a change to both endpoints and verify the affected workload afterward. If throughput or latency motivates tuning MTU, queues, or offload behavior, measure the relevant path rather than inferring performance from the link being virtual. This exercise verifies paired delivery, link readiness, destination routing, and coupled removal.

Sources

Linux veth(4), paired transmission, carrier state and namespace use, checked 2026-09-12.
Linux ip-link(8), veth creation, link state, movement and deletion, checked 2026-09-12.
Linux network_namespaces(7), network boundaries and virtual-device lifetime, checked 2026-09-12.

VXLAN needs matching tunnel parameters as well as a working underlay

Thu, 28 Oct 2027 00:00:00 +0000

VXLAN carries an Ethernet frame inside a UDP packet so an overlay network can cross an IP underlay. Tunnel endpoints encapsulate and decapsulate frames, while a 24-bit VXLAN Network Identifier, or VNI, identifies the virtual segment. The outer IP addresses route between endpoints; the inner Ethernet and IP addresses serve the overlay. These are separate layers with separate reachability and configuration requirements.

When it breaks

Pinging a tunnel endpoint proves little about a particular overlay segment. Both endpoints may be reachable while using different VNIs, UDP destination ports, or remote-peer configuration. An underlay firewall can also admit ICMP while dropping encapsulated UDP. Diagnose the actual transport and virtual segment instead of assuming that one successful underlay ping validates all tenant traffic above it.

Encapsulation consumes the underlay’s packet budget. For an untagged inner Ethernet frame over ordinary IPv4 and UDP, an overlay IP MTU of 1450 fits a 1500-byte underlay IP MTU: inner Ethernet, VXLAN, UDP, and outer IPv4 headers account for fifty bytes. IPv6 outer headers, VLAN tags, or other encapsulations change the calculation. A successful small request alone does not validate large packets or path MTU discovery.

A VNI also is not an encryption key or authenticated tenant identity. Anyone permitted to observe the encapsulated traffic may be able to read its contents, and trust in tunnel endpoints depends on the surrounding network controls. VXLAN’s segmentation role does not establish confidentiality. Protect the underlay and use appropriate authenticated encryption where the threat model requires it.

Check it quickly

Save this as check-vxlan.py and run python3 check-vxlan.py with Docker. The disposable Linux container uses Python 3.11.16, iproute2 6.1.0, and tcpdump 4.99.3. NET_ADMIN and SYS_ADMIN enable nested network namespaces and their temporary proc mount. The two endpoints use only a new veth link, with no host route changes, host mounts, or published service ports:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,'child exited without a response'
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found

def server(stack,name,address,port=18080,label='server'):
 code=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import json,socket,sys
address,port,label=sys.argv[1],int(sys.argv[2]),sys.argv[3]
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 body=json.dumps({'server':label,'peer':self.client_address,'path':self.path}).encode()
 self.send_response(200);self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
 def log_message(self,*args):pass
class Server(ThreadingHTTPServer):
 address_family=socket.AF_INET6 if ':' in address else socket.AF_INET
 def server_bind(self):
 if self.address_family==socket.AF_INET6:self.socket.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
 super().server_bind()
service=Server((address,port),Handler);print('ready',flush=True);service.serve_forever()
'''
 process=spawn(stack,['ip','netns','exec',name,'python3','-u','-c',code,address,str(port),label])
 assert line(process).strip()=='ready';return process
def request(name,address,port=18080):
 code=r'''
import http.client,json,sys
client=http.client.HTTPConnection(sys.argv[1],int(sys.argv[2]),timeout=.7)
try:
 client.request('GET','/lab-visible-message');response=client.getresponse();body=json.loads(response.read())
 assert response.status==200;print(json.dumps({'ok':True,'status':response.status,'body':body}))
except OSError as error:print(json.dumps({'ok':False,'type':type(error).__name__,'errno':error.errno}))
finally:client.close()
'''
 return json.loads(ns(name,'python3','-c',code,address,str(port)).stdout)

def capture(stack,root,name,interface,expression,label):
 path=root/(label+'.pcap')
 process=subprocess.Popen(['ip','netns','exec',name,'tcpdump','--immediate-mode','-Z','root','-U','-n','-i',interface,'-w',str(path),expression],
 stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process)
 deadline=time.monotonic()+3
 while time.monotonic()<deadline:
 assert process.poll() is None,'tcpdump exited before listening'
 if select.select([process.stderr],[],[],.1)[0] and 'listening on' in process.stderr.readline():return process,path
 raise AssertionError('tcpdump did not become ready')
def captured(process,path,ascii=False):
 process.send_signal(signal.SIGINT);_,diagnostic=process.communicate(timeout=3)
 assert process.returncode==0,diagnostic
 result=run('tcpdump','-nn','-vv',*(['-A'] if ascii else []),'-r',str(path))
 return result.stdout

with lab(['left','right']) as (root,stack,setting):
 wire('left','ul','right','ur',['192.0.2.1/30'],['192.0.2.2/30'])
 for name,interface in [('left','ul'),('right','ur')]:ip(name,'link','set',interface,'mtu','1500')
 def overlay(name,vni):
 local,remote,dev,address,mac=('192.0.2.1','192.0.2.2','ul','10.42.0.1/24','02:00:00:00:42:01') if name=='left' else ('192.0.2.2','192.0.2.1','ur','10.42.0.2/24','02:00:00:00:42:02')
 ip(name,'link','add','vx0','type','vxlan','id',str(vni),'local',local,'remote',remote,'dev',dev,'dstport','4789','nolearning')
 ip(name,'link','set','vx0','address',mac,'mtu','1450')
 ip(name,'addr','add',address,'dev','vx0');ip(name,'link','set','vx0','up')
 overlay('left',42);overlay('right',42);server(stack,'right','10.42.0.2',label='overlay')
 tap,path=capture(stack,root,'left','ul','udp port 4789','vxlan')
 successful=request('left','10.42.0.2');assert successful['ok'] and successful['body']['server']=='overlay'
 packets=captured(tap,path,ascii=True)
 assert 'vni 42' in packets.lower() and 'lab-visible-message' in packets and '.4789:' in packets,packets
 details=json.loads(ip('left','-d','-j','link','show','vx0').stdout)[0]
 assert details['linkinfo']['info_data']['id']==42 and details['mtu']==1450
 ip('right','link','delete','vx0');overlay('right',43)
 underlay=ns('left','ping','-n','-c','1','-W','1','192.0.2.2');assert underlay.returncode==0
 mismatch=request('left','10.42.0.2');assert not mismatch['ok'] and mismatch['type']=='TimeoutError'
 ip('right','link','delete','vx0');overlay('right',42)
 recovered=request('left','10.42.0.2');assert recovered['ok']
 print(json.dumps({'healthy_overlay':successful,'underlay_packet_capture':packets,'vni':details['linkinfo']['info_data']['id'],
 'underlay_mtu':1500,'overlay_mtu':details['mtu'],'mismatched_vni_underlay_ping_exit':underlay.returncode,
 'mismatched_vni_overlay':mismatch,'matching_vni_restored':recovered}))
"""
name='xbcatm-netpath-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The underlay addresses are 192.0.2.1 and 192.0.2.2. Each endpoint has an explicitly configured remote peer and UDP destination port 4789. Setting the port explicitly avoids depending on historical implementation defaults. Both overlay devices initially use VNI 42, fixed MAC addresses, and addresses 10.42.0.1 and 10.42.0.2. This is a small unicast tunnel, not a complete multipoint fabric or EVPN deployment.

An HTTP request across the overlay returns the expected server identity. Simultaneously, tcpdump observes the underlay carrying UDP packets marked VNI 42, including inner ARP and TCP traffic. The capture contains the literal request path /lab-visible-message. That is direct evidence that this configuration transports the application bytes without encrypting them; the VNI scopes delivery but does not conceal the payload.

The local veth capture also reports apparently incorrect UDP and TCP checksums while the HTTP exchange succeeds. Linux can represent checksums as partial work to be completed at another processing stage, including offloaded encapsulated traffic. This capture is not evidence of wire corruption. The fixture does not disable offloads or identify every checksum-processing stage, so it does not claim a measured hardware fault or a complete checksum diagnosis.

Next, the right overlay device is recreated with VNI 43 while retaining its address and MAC. The underlay ping still succeeds, but the overlay HTTP request times out. Recreating it with VNI 42 restores HTTP success. Holding the other overlay properties fixed makes the identifier mismatch the controlled change, instead of accidentally attributing a stale-neighbor problem to the VNI.

Match evidence to the layer being changed

Inspect the endpoint’s underlay route, selected interface, configured UDP port, VNI, peer mapping, and overlay address. Capture on both the tunnel device and its transport interface when possible: each exposes a different representation of the same communication. Record where the capture was taken, particularly when interpreting packet sizes or checksum warnings that may reflect kernel offload processing.

This example deliberately reserves encapsulation space with MTUs of 1500 and 1450, but tests only a short application exchange. Validate the real path’s packet-size requirements separately before using it for larger workloads. In a production fabric, also review how remote MAC destinations are learned or distributed and how unknown traffic is handled. Those control-plane choices cannot be inferred from this two-endpoint success.

Sources

RFC 7348 VXLAN, VNI, UDP encapsulation, MTU and security considerations, checked 2026-09-12.
Linux VXLAN documentation, VXLAN devices, destination ports and tunnel offloads, checked 2026-09-12.
Linux checksum offloads, partial and encapsulated checksums at different processing stages, checked 2026-09-12.

Kubernetes Services give pods a stable home

Mon, 23 Aug 2027 00:00:00 +0000

You cannot memorise a Pod that will be recreated. A Service fixes an address in front of them, keeping callers stable while pods churn.

One address in front of many

A Deployment gives you multiple, interchangeable pods, and each one has its own IP that will soon be recreated. A Service is the stable home in front of them: you declare a name and a selector, and the platform hands the Service a virtual cluster IP plus a DNS name that stay fixed. Cluster DNS resolves that name, so components (other pods, an Ingress, a load balancer) refer to the Service, never to an individual pod address. That is the point of the abstraction, the address is durable even though everything standing behind it is disposable.

Selection drives which pods respond

A pod does not join a Service by being listed; it joins by carrying the labels the Service’s selector asks for, and it leaves when it no longer matches or is deleted. The platform maintains the endpoint set from the selector, adding a pod once it is running and ready, removing it on scale-down or failure. So the ‘backend list’ is a derived, always-changing view rather than a hand-edited roster. Because membership is derived from labels plus readiness, a pod that becomes unhealthy is dropped automatically and traffic is never aimed at a pod that should not serve.

Traffic goes to whatever matches

All of this gives you indirection that survives churn. Clients hold one name; the set behind it can be scaled, replaced, or rolled without them changing anything. A rolling update swaps pods one by one, readiness-gated, and the Service keeps forwarding to the current healthy matching set, so callers see no interruption. The guarantee you rely on is: as long as the selector is precise and the pods match it, the Service routes to the right set at every moment. Keep the selector tight, gate readiness, and this stable home stays stable even when every backer is new.

The stable name is a promise to callers

A companion article explains how the virtual address itself is allocated; this one is about what that stability means to whoever calls it. A well-behaved client resolves the Service name once, keeps using it for the life of the process, and survives every backend being replaced without a config push. That promise holds only while three habits hold: the selector keeps matching exactly the intended pods, readiness keeps unready pods out of the set, and no caller cheats by caching a pod IP it observed once. Treat the Service name as the only supported handle, and let scaling, crashes, and rollouts happen behind it.

When it breaks

The quietest failure is a selector that matches nothing. A typo such as app: webs instead of app: web, or a Deployment that stopped setting the expected label, leaves the Service with zero endpoints. The cluster IP and DNS name still resolve, so the outage looks like refused connections rather than a missing name. When a Service answers lookups but carries no traffic, inspect its endpoint set first; an empty set points at labels, not at the network.

A headless Service chosen by habit is the next trap. Setting clusterIP: None removes the virtual IP, so DNS returns the changing pod addresses and each client must balance across them. Code written for one stable endpoint suddenly faces many shifting ones, with uneven retries and broken connection reuse. Choose headless only when the caller explicitly wants discovery.

Ports break the promise from the other side. A Service port pointed at the wrong container targetPort accepts traffic it can never deliver, and a hand-picked NodePort can collide with another Service on the same nodes. Finally, leaving sessionAffinity: ClientIP on a stateless pool pins each client to one backend, so one dying pod looks like a total outage. Keep affinity off unless stickiness is the requirement.

Check it quickly

Save the following as service_endpoints.py in a scratch directory and run python3 service_endpoints.py. It models endpoint convergence with plain dictionaries and changes nothing outside the process: one function derives the endpoint set from a selector plus pod labels and readiness, and four scenarios exercise it, including a deliberate empty selector as the negative control.

"""Simulate Service endpoint convergence from selector plus readiness."""

PODS = [
 {"name": "web-1", "labels": {"app": "web"}, "ready": True},
 {"name": "web-2", "labels": {"app": "web"}, "ready": True},
 {"name": "web-3", "labels": {"app": "web"}, "ready": False},
 {"name": "worker-1", "labels": {"app": "worker"}, "ready": True},
]


def select_endpoints(pods, selector):
 """Derive the endpoint set; an empty selector warns instead of matching."""
 if not selector:
 return [], "warning: empty selector matches nothing"
 matched = sorted(
 pod["name"]
 for pod in pods
 if pod.get("ready")
 and all(pod.get("labels", {}).get(key) == value for key, value in selector.items())
 )
 return matched, ""


def check(condition, message):
 if not condition:
 raise AssertionError(message)


def main():
 endpoints, warning = select_endpoints(PODS, {"app": "web"})
 check(endpoints == ["web-1", "web-2"], "ready web pods should form the set")
 check(warning == "", "a precise selector must not warn")
 print("steady state:", endpoints)

 rolling = [dict(pod, ready=True) if pod["name"] == "web-3" else pod for pod in PODS]
 rolling = [pod for pod in rolling if pod["name"] != "web-1"]
 endpoints, _ = select_endpoints(rolling, {"app": "web"})
 check(endpoints == ["web-2", "web-3"], "rollover must swap one ready pod for another")
 print("after rollover:", endpoints)

 endpoints, _ = select_endpoints(PODS, {"app": "web", "track": "stable"})
 check(endpoints == [], "an over-narrow selector should converge to empty, not error")
 print("over-narrow selector:", endpoints)

 endpoints, warning = select_endpoints(PODS, {})
 check(endpoints == [] and "empty selector" in warning, "empty selector must warn loudly")
 print("empty selector:", repr(warning))

 print("PASS: four convergences; readiness-gated selection, rollover, and empty warning hold")


if __name__ == "__main__":
 main()

The four scenarios converge as a healthy Service should: the steady state holds the two ready web pods while the unready one stays out, and the rollover swaps one ready pod for another without changing the selector. The empty selector is the negative control: it warns loudly instead of silently matching everything or nothing. All assertions passed with Python 3.14.4 on macOS.

The simulation checks label-plus-readiness convergence only. It does not run kube-proxy, cluster DNS, or real probe timing. Before deploying, inspect the live endpoint set alongside pod labels and readiness, and repeat the lookup against the real Service name.

Sources

Kubernetes: Services, the Service abstraction, checked 2027-08-15.
Kubernetes: Pods, the pods a Service selects, checked 2027-08-15.
Kubernetes: Deployments, managing the pod set, checked 2027-08-15.

Connection timeouts prevent silent hangs

Sun, 01 Aug 2027 00:00:00 +0000

A timeout names how long a particular operation may wait. It is not automatically a deadline for the whole request, and it does not prove that the remote operation stopped. Separate connection establishment, individual reads, and the caller’s overall budget before choosing numbers or interpreting a timeout error.

Successful connection does not mean a useful reply

A TCP connection can complete while the application behind it never produces a response. That failure belongs to the read stage, not to an unfinished TCP handshake. Name the stage in diagnostics; otherwise a message such as “database connection timed out” can hide whether the caller was acquiring a pool lease, opening a socket, or waiting for query results.

Python’s socket timeout applies to blocking socket operations. Repeated successful recv() calls can each consume a fresh allowance, so a peer that keeps sending small pieces can extend the whole read loop well beyond that number. In the exercise, each read asks for one byte to make this distinction visible.

For an overall receive budget, create one deadline using time.monotonic(). Before the next operation, compute its remaining duration and choose the smaller of that duration and the operation limit. Do not start another wait after the budget is exhausted. The monotonic clock avoids wall-clock corrections changing a local elapsed-time budget; it does not make thread scheduling instantaneous.

Read the actual client contract before extending this pattern. Name resolution, address retries, TLS negotiation, request writing, and connection-pool acquisition may have separate behavior. This example uses one numeric loopback address and plain TCP. It does not demonstrate bounded DNS resolution or a complete HTTPS client deadline.

When it breaks

A silent accepted peer holds a reader until its read timeout. A trickling peer can keep individual reads successful while making the full response too slow. Setting only the connection timeout handles neither problem after the connection has already succeeded. Conversely, a timeout that is too small can reject useful work during normal scheduling or network variation.

The failure path must close owned resources and preserve the difference between partial and complete data. An empty socket read means EOF, not proof that the application message is complete; this fixture knows the expected six-byte frame and rejects premature closure. A partial response must not silently become a successful result.

Also separate abandoning a wait from canceling remote work. The peer might already have performed a side effect when the caller times out. Retrying needs an appropriate replay contract and a remaining budget, not a fresh unlimited sequence of waits. The local deadline remains a cooperative boundary between operations, with scheduling and cleanup overhead, rather than a hard real-time guarantee.

Check it quickly

Save this as connection_probe.py and run python3 connection_probe.py. It uses only the standard library and was checked with Python 3.14.4 on macOS. Each case creates a task-owned loopback listener and peer thread, then closes the sockets and joins that thread. No external host, firewall rule, or production service is involved.

import socket
import threading
import time


PAYLOAD = b"abcdef"
PER_OPERATION = 0.5


def exercise(trickle, total=None, peer_payload=PAYLOAD):
 stop = threading.Event()
 failures = []
 with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as listener:
 listener.bind(("127.0.0.1", 0))
 listener.listen(1)
 listener.settimeout(2.0)

 def peer():
 try:
 connection, _ = listener.accept()
 with connection:
 connection.settimeout(0.5)
 if not trickle:
 stop.wait(2.0)
 return
 for byte in peer_payload:
 if stop.wait(0.1):
 return
 connection.sendall(bytes([byte]))
 except (BrokenPipeError, ConnectionResetError):
 pass # The deadline case deliberately closes the client.
 except Exception as error:
 failures.append(error)

 worker = threading.Thread(target=peer)
 worker.start()
 started = time.monotonic()
 deadline = None if total is None else started + total
 received = bytearray()
 phase, outcome = "connect", "complete"

 def allowance():
 if deadline is None:
 return PER_OPERATION
 remaining = deadline - time.monotonic()
 if remaining <= 0:
 raise TimeoutError("total budget expired")
 return min(PER_OPERATION, remaining)

 try:
 with socket.create_connection(listener.getsockname(), timeout=allowance()) as client:
 phase = "read"
 while len(received) < len(PAYLOAD):
 client.settimeout(allowance())
 chunk = client.recv(1)
 if not chunk:
 raise EOFError(f"short frame: {len(received)}/{len(PAYLOAD)} bytes")
 received.extend(chunk)
 except TimeoutError:
 outcome = "total" if deadline is not None and time.monotonic() >= deadline else phase
 finally:
 elapsed = time.monotonic() - started
 stop.set()
 worker.join(2.0)
 assert not worker.is_alive(), "fixture peer failed to stop"
 assert not failures, failures
 return outcome, bytes(received), elapsed


quiet = exercise(trickle=False)
assert quiet[0] == "read" and quiet[1] == b""
trickle = exercise(trickle=True)
assert trickle[0] == "complete" and trickle[1] == PAYLOAD
assert trickle[2] >= 0.5 # Each successful recv got a fresh operation timeout.
bounded = exercise(trickle=True, total=0.25)
assert bounded[0] == "total" and bounded[1] != PAYLOAD
try:
 exercise(trickle=True, peer_payload=b"ab")
except EOFError as error:
 assert str(error) == "short frame: 2/6 bytes"
else:
 raise AssertionError("a truncated frame must not become success")
print("early EOF rejects a two-byte fragment of the six-byte frame")
for name, result in (("quiet", quiet), ("trickle", trickle), ("bounded", bounded)):
 outcome, data, elapsed = result
 print(name, outcome, repr(data), "elapsed", round(elapsed, 3))

The quiet case connects successfully but receives nothing, then reports a read timeout around half a second. The trickle case sends six bytes with a pause before each one. All reads finish within their individual allowance, yet the complete frame takes longer than the configured half-second operation timeout. “Complete” here means all six expected bytes arrived, not merely that one read succeeded.

With the quarter-second overall budget, the same trickle cannot complete the frame. The output reports total expiration and partial bytes; the exact byte count and elapsed values depend on scheduling. Assertions deliberately check the outcome and completeness instead of promising exact millisecond timing. A fourth peer sends two bytes and closes; the reader raises EOFError instead of accepting that fragment as a complete frame.

The stop event is explicit test coordination that releases the local peer during cleanup. It is not cancellation transmitted by TCP to an arbitrary remote operation. These results validate the difference between per-operation waits and a shared receive budget, not connection blackholes, TLS failures, retries, or production tail latency.

Sources

Python socket, blocking-operation timeouts, connection establishment and socket reads, checked 2026-09-09.
Python time, monotonic elapsed-time measurement, checked 2026-09-09.

A CNAME changes DNS resolution while the requested hostname stays the same

Fri, 09 Jul 2027 00:00:00 +0000

A CNAME record associates an alias name with another DNS name. When resolving an address through that alias, a resolver follows the target until it obtains the requested data or encounters a failure. This lets a public hostname remain stable while its DNS dependency changes. It does not redirect a browser to another URL or replace the application’s requested hostname.

When it breaks

An alias can resolve perfectly while the destination serves the wrong website. HTTP still identifies the original hostname, and HTTPS certificate verification is tied to the requested service identity. Configuring only the target’s virtual host or certificate is insufficient. DNS supplies an address path; the selected server must also accept the hostname that users and applications actually request.

A CNAME cannot coexist with ordinary A, AAAA, TXT, or other data at the same owner name. DNSSEC-related metadata has specific exceptions, but those do not make a CNAME plus an A record valid. The ordinary zone apex already requires SOA and NS data, so placing a conventional CNAME there conflicts with those requirements. Provider alias-flattening features need their own documented semantics.

Chains can terminate at missing data or form loops. Each dependency also has its own lifetime and administrative boundary. Shortening an alias TTL does not shorten a target address already cached under a different RRset. Nor does every extra alias require another network round trip: a server may include several chain records in one answer, or the resolver may already have them cached.

Check it quickly

Save this as check-cname.py and run python3 check-cname.py with Docker. The pinned environment uses BIND 9.18.49, dnsmasq 2.90, and a small HTTP server. All DNS names belong to an isolated test zone. Resolver configuration changes and candidate zone files stay inside the disposable container; no public domain or host resolver is modified:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends bind9=1:9.18.49-1~deb12u2 bind9-dnsutils=1:9.18.49-1~deb12u2 dnsmasq-base=2.90-4~deb12u2 python3-dnspython=2.3.0-1 && rm -rf /var/lib/apt/lists/*\nCMD ["/usr/bin/python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,re,signal,subprocess,time
import dns.exception,dns.message,dns.query,dns.rcode,dns.rdatatype

def run(*args,check=True):
 result=subprocess.run(args,capture_output=True,text=True,timeout=15)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def query(name,kind='A',port=1053):
 return dns.query.udp(dns.message.make_query(name,kind),'127.0.0.1',port=port,timeout=1)
def values(response,kind='A'):
 wanted=dns.rdatatype.from_text(kind)
 return [item.to_text() for rrset in response.answer if rrset.rdtype==wanted for item in rrset]
def show(response):
 def section(records):return [{'name':str(rr.name),'type':dns.rdatatype.to_text(rr.rdtype),'ttl':rr.ttl,'data':[item.to_text() for item in rr]} for rr in records]
 return {'rcode':dns.rcode.to_text(response.rcode()),'answer':section(response.answer),'authority':section(response.authority)}
def wait(predicate,timeout=15):
 deadline=time.monotonic()+timeout
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except (OSError,dns.exception.Timeout):pass
 time.sleep(.05)
 raise AssertionError('DNS fixture condition did not become true')
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=5)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=5)

@contextmanager
def lab(records,soa_ttl=8,minimum=4,extra=''):
 with TemporaryDirectory(prefix='xbcatm-dns-') as directory,ExitStack() as stack:
 root=Path(directory);zone=root/'zone';conf=root/'named.conf';log=root/'named.log';serial=1
 def save(body):
 zone.write_text(f'$ORIGIN lab.test.\n$TTL 30\n@ {soa_ttl} IN SOA ns.lab.test. hostmaster.lab.test. ({serial} 60 60 3600 {minimum})\n@ IN NS ns.lab.test.\nns IN A 127.0.0.1\n'+body+'\n')
 run('named-checkzone','lab.test',str(zone))
 save(records)
 conf.write_text(f'''options {{ directory "{root}"; listen-on port 1053 {{127.0.0.1;}}; listen-on-v6 {{none;}};
recursion no; dnssec-validation no; notify no; querylog yes; max-cache-size 8m; pid-file "{root}/named.pid"; session-keyfile none;
{extra} }};
controls {{}};
zone "lab.test" {{ type primary; file "{zone}"; }};
''')
 run('named-checkconf',str(conf))
 output=stack.enter_context(log.open('w'))
 authority=subprocess.Popen(['named','-g','-c',str(conf),'-n','1','-U','1'],stdout=output,stderr=subprocess.STDOUT)
 stack.callback(stop,authority)
 def ready():
 assert authority.poll() is None,log.read_text()
 response=query('lab.test.','SOA');return response.rcode()==0 and response.answer[0][0].serial==serial
 wait(ready)
 def update(body):
 nonlocal serial
 serial+=1;save(body);authority.send_signal(signal.SIGHUP);wait(ready)
 def cache(port=1054,upstream=1053,size=150,options=()):
 logfile=root/('cache-'+str(port)+'.log');stream=stack.enter_context(logfile.open('w'))
 command=['dnsmasq','--no-daemon','--conf-file=/dev/null','--no-resolv','--no-hosts','--bind-interfaces',
 '--listen-address=127.0.0.1','--port='+str(port),'--server=127.0.0.1#'+str(upstream),
 '--cache-size='+str(size),'--log-queries','--log-facility=-',*options]
 process=subprocess.Popen(command,stdout=stream,stderr=subprocess.STDOUT);stack.callback(stop,process)
 def cache_ready():
 assert process.poll() is None,logfile.read_text()
 return query('lab.test.','SOA',port).rcode()==0
 wait(cache_ready);return process,logfile
 def queries(name,kind='A'):
 return sum(bool(re.search(r'query: '+re.escape(name.rstrip('.'))+r' IN '+kind+r'\b',line)) for line in log.read_text().splitlines())
 yield root,update,cache,queries
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import http.client,threading
records='alias 9 IN CNAME middle.lab.test.\nmiddle 12 IN CNAME target.lab.test.\ntarget 20 IN A 127.0.0.1\nloop-a 9 IN CNAME loop-b.lab.test.\nloop-b 9 IN CNAME loop-a.lab.test.'
class Handler(BaseHTTPRequestHandler):
 def log_message(self,*args):pass
 def do_GET(self):
 body=json.dumps({'host':self.headers['Host'],'path':self.path}).encode()
 self.send_response(200);self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
with lab(records) as (root,update,cache,queries),ThreadingHTTPServer(('127.0.0.1',0),Handler) as server:
 chain=query('alias.lab.test.');assert values(chain)==['127.0.0.1']
 assert values(chain,'CNAME')==['middle.lab.test.','target.lab.test.']
 assert [r.ttl for r in chain.answer]==[9,12,20]
 invalid={}
 for case,addition in [('same_owner','alias IN A 127.0.0.2'),('apex','@ IN CNAME target.lab.test.')]:
 candidate=root/(case+'.zone');candidate.write_text((root/'zone').read_text()+addition+'\n')
 checked=run('named-checkzone','lab.test',str(candidate),check=False)
 assert checked.returncode!=0 and 'CNAME and other data' in checked.stdout+checked.stderr
 invalid[case]={'returncode':checked.returncode,'diagnostic':'CNAME and other data'}
 seen=set();name='loop-a.lab.test.';hops=[]
 while name not in seen:
 seen.add(name);answer=query(name,'CNAME');target=values(answer,'CNAME')[0];hops.append([name,target]);name=target
 assert len(hops)==2 and name=='loop-a.lab.test.'
 cache(port=53)
 Path('/etc/resolv.conf').write_text('nameserver 127.0.0.1\noptions timeout:1 attempts:1\n')
 nss=Path('/etc/nsswitch.conf');nss.write_text(re.sub(r'^hosts:.*$','hosts: dns',nss.read_text(),flags=re.M))
 thread=threading.Thread(target=server.serve_forever,daemon=True);thread.start()
 try:
 client=http.client.HTTPConnection('alias.lab.test',server.server_port,timeout=2)
 client.request('GET','/original');response=client.getresponse();body=json.loads(response.read());client.close()
 assert response.status==200 and body=={'host':'alias.lab.test:'+str(server.server_port),'path':'/original'}
 print(json.dumps({'chain':show(chain),'invalid_candidates':invalid,'loop_detected_by_client':hops,'HTTP':body,'HTTP_status':response.status},sort_keys=True))
 finally:server.shutdown();thread.join(timeout=2)
"""
name='xbcatm-dns-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 built=docker('build','-t',name,str(root));assert built.returncode==0,built.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--memory','384m','--pids-limit','128',name,
 '/usr/bin/python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The address query for alias.lab.test returns two CNAME records followed by an A record: alias points to middle, middle points to target, and target has 127.0.0.1. Their respective TTLs are 9, 12, and 20 seconds. This single authoritative response contains the complete local chain, so the example does not claim that its two aliases necessarily cost two additional DNS exchanges.

Two separate candidate zones demonstrate invalid combinations. One adds an A record at the alias owner; the other adds a CNAME at the zone apex. named-checkzone rejects both with “CNAME and other data.” These candidates never replace the running valid zone. Syntax and data validation therefore establish the configuration restriction without making a failed candidate look like a successfully activated change.

A second pair of aliases points in a circle. The example client queries each CNAME explicitly, remembers visited names, and detects the repeat after two hops. This is a bounded demonstration of the loop itself. It is not a measurement of every recursive resolver’s timeout, retry count, or chosen error response when presented with that same dependency cycle.

Finally, Python’s HTTP client requests alias.lab.test through the container’s normal name-resolution path. The server reports Host alias.lab.test with its temporary port and path /original, returning status 200. The client is not manually told to substitute the target hostname. The result demonstrates address resolution through the chain while preserving the original HTTP service identity and request path.

Move the dependency and preserve the service contract

Before switching an alias, provision the destination to serve the original hostname, including routing and certificates where applicable. Verify both address families and any provider-specific domain ownership checks. An alias into an external service also depends on that service retaining the intended resource; a dangling target is an ownership and availability concern, not a harmless unused name.

Plan the change around all relevant cached RRsets. Updating the alias target and updating the old target’s address are different operations. Observe authoritative answers, recursive answers, and actual application traffic so a cached link in the chain is distinguishable from a misconfigured destination. Retain the old endpoint for the overlap your migration requires rather than assuming the stable public spelling makes the transition instantaneous.

Keep the chain understandable, with a named owner for each dependency and explicit checks for missing targets and cycles. Use an HTTP redirect when changing the URL presented to the client is the intended behavior. Use CNAME when the DNS alias relationship is the intended behavior, and validate the application separately after that relationship successfully resolves.

Sources

RFC 1034 domain names, alias processing, canonical names, and loop handling, checked 2026-09-12.
RFC 2181 DNS clarifications, CNAME coexistence constraints and RRset TTL rules, checked 2026-09-12.
BIND 9.18.49 tools, validating candidate zone data with named-checkzone, checked 2026-09-12.

Multiple DNS addresses do not guarantee balanced or healthy traffic

Tue, 06 Jul 2027 00:00:00 +0000

Several A or AAAA records at one name provide an address set. A DNS server may rotate the order it returns, but the receiving resolver and application decide how to use that set. Ordinary address records contain no request-load measurement or application-health result. Their presence alone does not promise equal traffic, automatic failover, or even a particular first address across implementations.

When it breaks

DNS answer distribution and application request distribution have different denominators. One recursive resolver can serve many clients from a cached answer, and one application can issue thousands of requests after a single lookup. Connection pooling makes that separation even stronger. Even perfectly alternating first addresses in authoritative replies can coexist with highly unequal request counts at the actual servers.

A failed peer remains in an ordinary static RRset until DNS data changes or another system changes it. The DNS server does not learn application health simply by returning the address. Some clients try an alternate address after connection failure; others exhaust their deadline or expose the failure immediately. Retry behavior is a property to verify for the specific client and operation.

Describing the mechanism as a guaranteed coin flip is also inaccurate. Ordering can be configured, randomized, retained by a cache, or changed by address-selection policy. Clients may select the first address, race attempts, or reuse a prior choice. An observed distribution therefore needs the DNS configuration, resolver behavior, connection reuse, and workload alongside the server-side request counts.

Check it quickly

Save this as check-dns-addresses.py and run python3 check-dns-addresses.py with Docker. The pinned environment uses BIND 9.18.49 with rrset-order explicitly set to cyclic. Two real HTTP listeners bind separate loopback addresses on the same temporary port inside an isolated container. Their response bodies identify the listener that actually served each request:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends bind9=1:9.18.49-1~deb12u2 bind9-dnsutils=1:9.18.49-1~deb12u2 dnsmasq-base=2.90-4~deb12u2 python3-dnspython=2.3.0-1 && rm -rf /var/lib/apt/lists/*\nCMD ["/usr/bin/python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,re,signal,subprocess,time
import dns.exception,dns.message,dns.query,dns.rcode,dns.rdatatype

def run(*args,check=True):
 result=subprocess.run(args,capture_output=True,text=True,timeout=15)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def query(name,kind='A',port=1053):
 return dns.query.udp(dns.message.make_query(name,kind),'127.0.0.1',port=port,timeout=1)
def values(response,kind='A'):
 wanted=dns.rdatatype.from_text(kind)
 return [item.to_text() for rrset in response.answer if rrset.rdtype==wanted for item in rrset]
def show(response):
 def section(records):return [{'name':str(rr.name),'type':dns.rdatatype.to_text(rr.rdtype),'ttl':rr.ttl,'data':[item.to_text() for item in rr]} for rr in records]
 return {'rcode':dns.rcode.to_text(response.rcode()),'answer':section(response.answer),'authority':section(response.authority)}
def wait(predicate,timeout=15):
 deadline=time.monotonic()+timeout
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except (OSError,dns.exception.Timeout):pass
 time.sleep(.05)
 raise AssertionError('DNS fixture condition did not become true')
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=5)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=5)

@contextmanager
def lab(records,soa_ttl=8,minimum=4,extra=''):
 with TemporaryDirectory(prefix='xbcatm-dns-') as directory,ExitStack() as stack:
 root=Path(directory);zone=root/'zone';conf=root/'named.conf';log=root/'named.log';serial=1
 def save(body):
 zone.write_text(f'$ORIGIN lab.test.\n$TTL 30\n@ {soa_ttl} IN SOA ns.lab.test. hostmaster.lab.test. ({serial} 60 60 3600 {minimum})\n@ IN NS ns.lab.test.\nns IN A 127.0.0.1\n'+body+'\n')
 run('named-checkzone','lab.test',str(zone))
 save(records)
 conf.write_text(f'''options {{ directory "{root}"; listen-on port 1053 {{127.0.0.1;}}; listen-on-v6 {{none;}};
recursion no; dnssec-validation no; notify no; querylog yes; max-cache-size 8m; pid-file "{root}/named.pid"; session-keyfile none;
{extra} }};
controls {{}};
zone "lab.test" {{ type primary; file "{zone}"; }};
''')
 run('named-checkconf',str(conf))
 output=stack.enter_context(log.open('w'))
 authority=subprocess.Popen(['named','-g','-c',str(conf),'-n','1','-U','1'],stdout=output,stderr=subprocess.STDOUT)
 stack.callback(stop,authority)
 def ready():
 assert authority.poll() is None,log.read_text()
 response=query('lab.test.','SOA');return response.rcode()==0 and response.answer[0][0].serial==serial
 wait(ready)
 def update(body):
 nonlocal serial
 serial+=1;save(body);authority.send_signal(signal.SIGHUP);wait(ready)
 def cache(port=1054,upstream=1053,size=150,options=()):
 logfile=root/('cache-'+str(port)+'.log');stream=stack.enter_context(logfile.open('w'))
 command=['dnsmasq','--no-daemon','--conf-file=/dev/null','--no-resolv','--no-hosts','--bind-interfaces',
 '--listen-address=127.0.0.1','--port='+str(port),'--server=127.0.0.1#'+str(upstream),
 '--cache-size='+str(size),'--log-queries','--log-facility=-',*options]
 process=subprocess.Popen(command,stdout=stream,stderr=subprocess.STDOUT);stack.callback(stop,process)
 def cache_ready():
 assert process.poll() is None,logfile.read_text()
 return query('lab.test.','SOA',port).rcode()==0
 wait(cache_ready);return process,logfile
 def queries(name,kind='A'):
 return sum(bool(re.search(r'query: '+re.escape(name.rstrip('.'))+r' IN '+kind+r'\b',line)) for line in log.read_text().splitlines())
 yield root,update,cache,queries
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import http.client,threading
class Handler(BaseHTTPRequestHandler):
 def log_message(self,*args):pass
 def do_GET(self):
 body=self.server.server_address[0].encode();self.server.hits+=1
 self.send_response(200);self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
def request(address,port):
 connection=http.client.HTTPConnection(address,port,timeout=.5)
 try:
 connection.request('GET','/');response=connection.getresponse();assert response.status==200;return response.read().decode()
 finally:connection.close()
with lab('pool 30 IN A 127.0.0.2\npool 30 IN A 127.0.0.3',extra='rrset-order { order cyclic; };') as (_,update,cache,queries):
 servers=[];threads=[]
 try:
 for address in ['127.0.0.2','127.0.0.3']:
 server=ThreadingHTTPServer((address,0 if not servers else servers[0].server_port),Handler);server.hits=0
 thread=threading.Thread(target=server.serve_forever,daemon=True);thread.start();servers.append(server);threads.append(thread)
 port=servers[0].server_port;orders=[values(query('pool.lab.test.')) for _ in range(6)]
 assert all(set(order)=={'127.0.0.2','127.0.0.3'} for order in orders)
 assert {tuple(order) for order in orders}=={('127.0.0.2','127.0.0.3'),('127.0.0.3','127.0.0.2')}
 saved=values(query('pool.lab.test.'));selected=saved[0]
 hits=[request(selected,port) for _ in range(12)];assert hits==[selected]*12
 distribution={s.server_address[0]:s.hits for s in servers};assert sorted(distribution.values())==[0,12]
 victim=next(s for s in servers if s.server_address[0]==selected);victim.shutdown();victim.server_close()
 still_published=values(query('pool.lab.test.'));assert selected in still_published
 try:request(selected,port);raise AssertionError('stopped HTTP server accepted a connection')
 except OSError as error:failure=type(error).__name__
 alternate=next(a for a in saved if a!=selected);assert request(alternate,port)==alternate
 print(json.dumps({'cyclic_answer_orders':orders,'one_lookup_twelve_requests':distribution,'after_peer_stopped':still_published,
 'selected_address_error':failure,'explicit_alternate_succeeded':alternate},sort_keys=True))
 finally:
 for server in servers:server.shutdown();server.server_close()
 for thread in threads:thread.join(timeout=2)
"""
name='xbcatm-dns-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 built=docker('build','-t',name,str(root));assert built.returncode==0,built.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--memory','384m','--pids-limit','128',name,
 '/usr/bin/python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

Six direct queries for pool.lab.test return the same two addresses, 127.0.0.2 and 127.0.0.3, in alternating order in the recorded run. Every answer includes the complete set. The assertion checks both membership and observed order changes. This establishes the configured behavior of this authoritative server; it does not claim that all DNS servers or recursive caches must preserve that order.

The example then performs one additional lookup, saves the returned list, and uses its first address for twelve HTTP requests. All twelve reach that selected listener, producing a 12:0 distribution. Each request establishes its own HTTP connection, so this imbalance does not require connection pooling. Retaining one DNS selection across multiple connections is enough to separate answer rotation from request balancing.

Next, the selected listener is stopped and its socket closed. A fresh authoritative DNS query still includes its address because the zone has not changed. A connection to that peer fails with ConnectionRefusedError in the recorded environment. The program explicitly selects the other saved address and receives a successful response, demonstrating an available alternate without attributing that choice to DNS itself.

Choose the balancing and failure contract explicitly

If simple address distribution is sufficient, keep all advertised peers capable of serving the hostname and the expected workload. Monitor each peer directly as well as the aggregate name. A successful request through one address cannot establish the health of every other advertised address, especially when the client silently tries alternatives and hides the first failure from the caller.

If a DNS provider removes unhealthy addresses, account for the separate detection delay and previously cached answers. Health-driven DNS changes improve future discovery but do not revoke old answers or move established connections. Keep a defined client retry policy and migration overlap where required. Lowering TTL changes the caching tradeoff; it does not turn DNS into an application-level scheduler.

When per-request routing, capacity weights, or bounded failover behavior is required, evaluate an appropriate load balancer and its actual health and retry settings. That choice brings its own operational responsibilities. The useful comparison is between observable contracts, not between a supposedly random DNS mechanism and a supposedly infallible proxy. Measure which peer served requests and what users experienced when one peer stopped.

Finally, remember that same-name, same-type address records form an RRset and should share one TTL. Assigning different lifetimes to individual members is not a sound way to express weights. Use a supported policy with defined semantics, and validate distribution under realistic resolver sharing and connection reuse before making promises about how evenly traffic will spread.

Sources

RFC 2181 DNS clarifications, complete RRsets and equal TTLs within a set, checked 2026-09-12.
BIND 9.18.49 reference, rrset-order and configured cyclic answer order, checked 2026-09-12.
RFC 1034 domain names, resolver behavior and cached resource records, checked 2026-09-12.

NodePort publishes a service on every node's fixed port

Tue, 11 May 2027 00:00:00 +0000

A Service of type NodePort does one blunt thing: it reserves a port in the 30000-32767 range and makes every node in the cluster listen on it. kube-proxy programs the same forwarding rule into the iptables or IPVS tables of every machine, so curl <any-node-ip>:<nodePort> reaches the Service no matter which node answers. That is the whole appeal — external reachability with no cloud load balancer, no DNS record and no ingress controller.

The cost is hidden inside the same design. Because every node answers, a packet frequently lands on a machine that is not running the pod, and the cluster then has to decide whose address the backend sees. externalTrafficPolicy is that decision: Cluster spreads load but rewrites the client address, Local preserves it but drops traffic on any node without a local endpoint. Choosing one is unavoidable, and each choice breaks something different.

A port every node owns

kubectl create service nodeport without --node-port emits no nodePort field at all; the API server allocates one from the range. The allocation is cluster-wide rather than per node, so a given number belongs to exactly one Service across the whole fleet. That is why two Services pinning 31000 cannot coexist, and also why a single firewall rule for one number is portable to every machine.

Every node receives the rule whether or not it runs a backend. That is what makes the type useful — there is no node to avoid and no membership list for the caller to maintain — and it is also what makes it expensive, because the port is open on machines that may have no business accepting that traffic. When the endpoint set is empty the rule still exists and the connection is refused rather than silently dropped, which is at least honest.

The extra hop and the client address

A request arriving on node B is forwarded to node A when that is where the pod lives, so the datapath is one leg longer than hitting node A directly. With the default externalTrafficPolicy: Cluster, kube-proxy also source-NATs the packet to node B’s address, so the pod sees a cluster-internal address rather than the client. Anything that depends on the client address — rate limiting, geo rules, audit logs, an allow list — is then reading the wrong number.

externalTrafficPolicy: Local keeps the original source address by forwarding only to endpoints on the node that received the packet. The price is availability: a node with no local replica has no working node port at all, so a three-node cluster running one replica answers on one node and drops on two. That is the trade — a correct client address and one less hop, in exchange for a node port that only works on some of the nodes.

A side door around the ingress

A node port terminates nothing and inspects nothing. A request sent straight to node:31000 skips the load balancer, TLS termination, the WAF and every host or path routing rule that would have applied, then lands on the pod with whatever the caller chose to send. The pod is left making security decisions it was never written to make.

Treat it as a stopgap with an owner and an expiry date. Reserve the range, pin a number only when a firewall or DNS record forces you to, and restrict who can reach the range at the network layer instead of assuming the ingress is the only way in.

When it breaks

The classic failure is the collision. A nodePort is a single cluster-wide resource, so the second Service that pins a number already in use is rejected with provided port is already allocated, and a Service asking for an automatic port can never get one once the 2768-port range is full. Both present as a Service that exists but has no usable external port, which reads like a networking fault and is really a capacity one.

The second failure is quieter. A node port is a real socket bind on every host, so anything else already listening on that number makes it dead on that machine while kubectl get svc still reports a healthy Service — health checks, dashboards and the Service object all lie together. Because the same port bypasses the ingress layer, the rule that makes debugging easy also makes the cluster reachable in a way nobody is filtering. The blast radius is every node in the fleet, not the one you were looking at.

Check it quickly

Confirm the port, the endpoints, and who else owns the port:

kubectl get svc web -o jsonpath='{.spec.type} {.spec.ports[0].nodePort}{"\n"}'
kubectl get endpoints web
kubectl get svc -A -o json | jq -r '.items[] | select(.spec.type=="NodePort") | "\(.spec.ports[0].nodePort) \(.metadata.namespace)/\(.metadata.name)"' | sort -n | uniq -D -w5
ss -lnt | grep -E ':3[0-9]{4}\b'
curl -sS -o /dev/null -w '%{http_code}\n' "http://$(hostname -I | awk '{print $1}'):$(kubectl get svc web -o jsonpath='{.spec.ports[0].nodePort}')/"

The third line prints any nodePort claimed twice, and the fourth shows whether a process other than kube-proxy already owns that port on this host. The curl should print 200 from every node; a refusal on exactly one node means something local took the port, and a refusal everywhere means the Service has no ready endpoints.

Sources

Kubernetes Service, NodePort type, the 30000-32767 default range and cluster-wide port allocation, checked 2026-09-16.
Virtual IPs and Service Proxies, kube-proxy iptables/IPVS rule programming and externalTrafficPolicy, checked 2026-09-16.
Using Source IP, SNAT on NodePort and the client-IP consequence, checked 2026-09-16.

Idle timeouts recycle edge connections

Mon, 26 Apr 2027 00:00:00 +0000

An idle timeout closes a connection after a defined period without qualifying activity. It is a resource policy, not proof that the client disappeared. Quiet connections can belong to healthy clients; the useful question is which activity resets which timer, and what the protocol allows after closure.

Idle is defined at a particular boundary

A TCP connection consumes file descriptors, kernel buffers, and application or proxy bookkeeping. It does not normally require a distinct routing-table entry per connection. Reclaiming unused connections can bound retained resources, but legitimate reuse also avoids setup. Choose the policy for the workload instead of assuming that every quiet connection is abandoned.

Go’s HTTP Server.IdleTimeout bounds waiting for the next request on a keep-alive connection. It does not time out a handler merely because that handler is working without sending bytes. Request-header reading, body transfer, response writing, and application execution need their own relevant controls. A limit with the word “idle” should not be treated as a complete request deadline.

Nginx also distinguishes boundaries: its client keepalive_timeout controls retained client connections, while proxy_read_timeout limits gaps between successive reads from an upstream response, not the whole response duration. These settings observe different activity on different sides of the proxy. Matching their numeric values does not make their meanings identical.

Zero is not portable either. Nginx’s zero keepalive_timeout disables client keep-alive connections. Go’s zero IdleTimeout uses ReadTimeout; when both provide no positive limit, no idle timeout is established. Check the implementation rather than interpreting zero as universally meaning “off” or “immediately close.”

When it breaks

A threshold shorter than legitimate quiet periods forces reconnections or interrupts the relevant flow. An excessively generous one can retain more sockets than the service can afford. Record connection counts, close reasons, reuse, and workload patterns; silence alone does not identify a broken client or establish the best threshold.

HTTP keep-alive means connection reuse, not a periodic application heartbeat. TCP keepalive probes are another mechanism and are not application request bytes. An application heartbeat only affects a timer if that particular endpoint counts it as activity. A proxy, upstream, and client need not observe the same events, especially with streaming or multiplexed protocols.

Closure can race a client starting another request. A stale pooled connection therefore needs a recovery path, but reconnecting does not make every request safe to replay. HTTP explicitly permits asynchronous connection closure; retry decisions still depend on request semantics and the uncertainty about prior effects. “The server closes slightly early” is not automatically harmless for a mutation.

Check it quickly

Save this as idle_probe.go and run go run idle_probe.go. It uses only Go’s standard library and was checked with Go 1.27.0 on macOS. The test server listens on a local loopback address. Requests are plain HTTP/1.1; no public edge, Nginx instance, certificate, or production configuration is involved.

package main

import (
 "context"
 "fmt"
 "io"
 "net"
 "net/http"
 "net/http/httptest"
 "runtime"
 "strconv"
 "sync/atomic"
 "time"
)

type connectionKey struct{}

func check(ok bool, message string) {
 if !ok {
 panic(message)
 }
}

func main() {
 const idle = 200 * time.Millisecond
 const headers = 500 * time.Millisecond
 const work = 350 * time.Millisecond
 var serial, handled atomic.Int64
 closed := make(chan struct{}, 8)
 server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 handled.Add(1)
 if r.URL.Path == "/slow" {
 time.Sleep(work)
 }
 w.Header().Set("X-Connection-ID", strconv.FormatInt(r.Context().Value(connectionKey{}).(int64), 10))
 w.Header().Set("Content-Type", "text/plain")
 fmt.Fprintln(w, "ok")
 }))
 server.Config.IdleTimeout = idle
 server.Config.ReadHeaderTimeout = headers
 server.Config.ConnContext = func(ctx context.Context, _ net.Conn) context.Context {
 return context.WithValue(ctx, connectionKey{}, serial.Add(1))
 }
 server.Config.ConnState = func(_ net.Conn, state http.ConnState) {
 if state == http.StateClosed {
 closed <- struct{}{}
 }
 }
 server.Start()
 defer server.Close()
 transport := &http.Transport{MaxIdleConns: 1, MaxIdleConnsPerHost: 1}
 defer transport.CloseIdleConnections()
 client := &http.Client{Transport: transport, Timeout: 2 * time.Second}

 get := func(path string) int64 {
 response, err := client.Get(server.URL + path)
 check(err == nil, fmt.Sprint(err))
 body, readErr := io.ReadAll(response.Body)
 closeErr := response.Body.Close()
 check(readErr == nil && closeErr == nil, "response body did not finish cleanly")
 check(response.Proto == "HTTP/1.1" && response.StatusCode == 200, "unexpected protocol or status")
 check(string(body) == "ok\n", "unexpected response body")
 id, err := strconv.ParseInt(response.Header.Get("X-Connection-ID"), 10, 64)
 check(err == nil, "missing connection identity")
 return id
 }

 first := get("/ping")
 check(get("/ping") == first, "the fully consumed responses should allow reuse")
 started := time.Now()
 check(get("/slow") == first, "the active request should keep its connection")
 check(time.Since(started) >= work, "the handler did not complete its work")
 fmt.Println("two requests reuse one HTTP/1.1 connection; active work outlasts IdleTimeout")

 select {
 case <-closed:
 case <-time.After(2 * time.Second):
 panic("the idle connection was not closed")
 }
 check(get("/ping") != first, "the request after idle closure needs a new connection")
 fmt.Println("server reports idle closure; a later GET succeeds on a new connection")

 before := handled.Load()
 connection, err := net.DialTimeout("tcp", server.Listener.Addr().String(), time.Second)
 check(err == nil, fmt.Sprint(err))
 defer connection.Close()
 check(connection.SetDeadline(time.Now().Add(2*time.Second)) == nil, "could not bound raw probe")
 _, err = io.WriteString(connection, "GET /ping HTTP/1.1\r\nHost: probe\r\n")
 check(err == nil, fmt.Sprint(err))
 reply, err := io.ReadAll(connection)
 check(err == nil && len(reply) == 0, "partial headers should end with EOF in this server")
 check(handled.Load() == before, "incomplete headers must not invoke the handler")
 fmt.Println("incomplete request headers close without reaching the handler; ReadHeaderTimeout is separate")
 fmt.Println(runtime.Version())
}

The client reads each response completely and closes its body, allowing reuse. Two successive requests report the same server-assigned connection identity. A third request runs its handler for 350 milliseconds, longer than the 200-millisecond idle allowance, and still succeeds on that connection. Active handler time is not the between-request idle interval.

After the response finishes, the client sends nothing. The server reports the connection closed, and a later GET succeeds with a different identity. The example observes that transition directly instead of sleeping an assumed amount and declaring that cleanup must have occurred.

A separate raw connection sends an incomplete request header and waits for closure. The tested server returns EOF without invoking the handler; its 500-millisecond ReadHeaderTimeout is distinct from idle reuse policy. Client-side deadlines bound the test’s own waits, and deferred cleanup closes the owned transport and server.

The short durations make the exercise observable, not suitable as universal production values. These checks cover Go’s HTTP/1.1 behavior, not HTTP/2 streams, WebSockets, kernel keepalive timing, or every intermediary. Validate each real hop’s definition of activity before coordinating its idle policy.

Sources

Go net/http, idle and header timeouts, connection states and response-body reuse, checked 2026-09-09.
Go net/http/httptest, a task-owned loopback test server and its lifecycle, checked 2026-09-09.
Nginx HTTP core module, client keep-alive timeout semantics, checked 2026-09-09.
Nginx proxy module, per-read upstream response timeout, checked 2026-09-09.
RFC 9112: HTTP/1.1, persistent connections and asynchronous closure, checked 2026-09-09.

Default ports are conventions, not protocol detection or security

Wed, 14 Apr 2027 00:00:00 +0000

A default port lets a client omit a number when a protocol or configuration already supplies it. HTTP commonly uses TCP 80, HTTPS TCP 443, and SSH TCP 22. These assignments reduce configuration, but they do not inspect a listener’s bytes or establish its identity. An endpoint includes an address and transport as well as a port, and its application protocol still has to match the caller.

When it breaks

Moving a service to another port does not make its clients discover the change. URLs, database connection settings, probes, proxies, and access rules may each carry their own destination. Updating only the server can leave the health check aimed at an old listener or a client failing against an unused port. Record the intended mapping and test the actual callers affected by the change.

A familiar number can also create false confidence. A process can speak cleartext HTTP on TCP 443, or a different protocol on TCP 80. Selecting HTTPS means performing TLS, including the intended certificate checks; selecting port 443 alone does not add encryption. Likewise, a successful TCP connection proves neither that the expected application speaks next nor that the peer is authorized to serve its identity.

Service-name databases are descriptions, not service managers. Linux /etc/services maps names to numbers and transport protocols, but adding an entry does not start a daemon or open a firewall. Its contents can differ from another host’s database and from the current IANA registry. Name resolution for a service number and hostname resolution for an address are separate inputs to a connection.

Check it quickly

Save this as check-default-ports.py and run python3 check-default-ports.py with Docker. Python 3.11.16 runs in a pinned Linux container with no published ports. The fixture deliberately starts a plain HTTP server on TCP 443 and later binds a UDP socket to the same number. These listeners are disposable and cannot replace a host’s production HTTPS server:

import json,subprocess,uuid
image="python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84"
program=r"""
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import http.client,json,socket,ssl,threading

class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 body=b'plain HTTP on TCP 443\n';self.send_response(200);self.send_header('Content-Length',str(len(body)))
 self.end_headers();self.wfile.write(body)
 def log_message(self,*args):pass
server=ThreadingHTTPServer(('127.0.0.1',443),Handler)
thread=threading.Thread(target=server.serve_forever,daemon=True);thread.start()
try:
 plain=http.client.HTTPConnection('127.0.0.1',443,timeout=2)
 try:
 plain.request('GET','/');response=plain.getresponse();body=response.read()
 assert response.status==200 and body==b'plain HTTP on TCP 443\n'
 finally:plain.close()
 encrypted=http.client.HTTPSConnection('127.0.0.1',443,timeout=2)
 try:
 encrypted.request('GET','/');raise AssertionError('plain server spoke TLS')
 except ssl.SSLError as error:tls_error=error.reason
 finally:encrypted.close()
 with socket.socket(socket.AF_INET,socket.SOCK_DGRAM) as udp,socket.socket(socket.AF_INET,socket.SOCK_DGRAM) as client:
 udp.bind(('127.0.0.1',443));udp.settimeout(1);client.settimeout(1)
 client.sendto(b'UDP 443',udp.getsockname());data,peer=udp.recvfrom(64);assert data==b'UDP 443'
 udp.sendto(b'UDP reply',peer);assert client.recvfrom(64)[0]==b'UDP reply'
 concurrent=udp.getsockname()[1]==server.server_port
 defaults={'http':http.client.HTTPConnection('127.0.0.1').port,'https':http.client.HTTPSConnection('127.0.0.1').port}
 registry={'http_tcp':socket.getservbyname('http','tcp'),'https_tcp':socket.getservbyname('https','tcp'),
 'domain_tcp':socket.getservbyname('domain','tcp'),'domain_udp':socket.getservbyname('domain','udp')}
 assert defaults=={'http':80,'https':443} and registry=={'http_tcp':80,'https_tcp':443,'domain_tcp':53,'domain_udp':53}
 print(json.dumps({'client_defaults':defaults,'local_services':registry,'explicit_http_status':response.status,
 'explicit_http_body':body.decode().strip(),'https_failure':tls_error,'tcp_udp_same_number':concurrent}))
finally:
 server.shutdown();server.server_close();thread.join(timeout=2);assert not thread.is_alive()
"""
name='xbcatm-socket-'+uuid.uuid4().hex[:10]
try:
 result=subprocess.run(['docker','run','--rm','-i','--name',name,'--network','none',
 '--memory','128m','--pids-limit','64',image,'python3','-'],input=program,
 capture_output=True,text=True,timeout=60)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),ensure_ascii=False,sort_keys=True))
finally:
 subprocess.run(['docker','rm','-f',name],capture_output=True,text=True,timeout=10)

The client’s unconnected HTTPConnection and HTTPSConnection objects select 80 and 443 when no port is supplied. Local service lookups also return HTTP/TCP 80 and HTTPS/TCP 443. For the domain service, the database returns 53 for both TCP and UDP. These observations show the configured defaults and registry entries; they do not establish that any service is listening on the listed ports.

An HTTPConnection explicitly pointed at TCP 443 sends a normal HTTP request and receives status 200 with plain HTTP on TCP 443. Changing the port argument did not change that client’s protocol. The server handles exactly what it implements, regardless of the number traditionally associated with a different transport security expectation.

An HTTPSConnection to that same endpoint fails during TLS negotiation. In the recorded environment, OpenSSL reports WRONG_VERSION_NUMBER because the peer is a plain HTTP server. This is not an expired-certificate test: no valid TLS session was established to authenticate a certificate. Diagnose the protocol mismatch before trying to repair a trust chain that the server never supplied.

While the TCP listener remains active, another socket binds UDP 443 and completes a datagram request and reply. There is no bind conflict because TCP and UDP have separate port namespaces. This does not demonstrate HTTP/3 or QUIC; the UDP payload is explicitly an arbitrary test message. A firewall rule or probe for TCP 443 does not by itself cover UDP 443.

Keep the complete endpoint in configuration

Use defaults where they simplify a deployment, and use explicit alternatives where topology or multiple instances require them. Nonstandard ports can reduce some indiscriminate traffic, but they do not provide meaningful authentication or confidentiality. Base those properties on verified identities, encrypted transport where required, application permissions, and an access policy appropriate to the service’s legitimate callers.

Include the scheme or protocol, address, port, and expected server identity in connection documentation. A reverse proxy may terminate public HTTPS on 443 and speak HTTP to a private backend on another port. That is a deliberate pair of connections, not a contradiction in the port registry. Test each hop according to its own protocol and trust boundary.

When a connection fails, first distinguish no listener, rejected network access, incompatible protocol, certificate failure, and application rejection. They can occur at the same numeric port while requiring different fixes. Keep the response or specific exception alongside the destination and transport. Port conventions help find the right starting point; successful protocol and identity checks establish whether the endpoint fulfills its contract.

Sources

IANA service and port registry, registered service names, numbers and transport protocols, checked 2026-09-12.
Linux services(5), the local service-name database and its lack of service activation, checked 2026-09-12.
Python 3.11 socket, service-name lookups and separate transport sockets, checked 2026-09-12.

Retries with jitter and backoff

Sat, 10 Apr 2027 00:00:00 +0000

A retry is another unit of work, not an automatic improvement in reliability. It can recover from a transient failure, but can also repeat an unsafe operation or add demand to an overloaded dependency. Decide whether the operation is eligible, how much total effort is allowed, and when that effort should happen before choosing a delay formula.

Bound attempts before spreading them

Distinguish attempts from retries: three total attempts means one initial call and at most two retries. Also choose which layer owns recovery. If three nested layers each allow three attempts, one outer action can cause twenty-seven calls to a persistently failing leaf. Adding jitter changes their timing, not that multiplication.

Google SRE describes both per-request and wider retry budgets. A per-request cap alone still allows a large population of failing clients to generate substantial extra work. Measure retry traffic separately from original demand and coordinate with existing SDK, proxy, and application policies.

Exponential backoff sets a ceiling

A capped exponential schedule increases its ceiling until reaching a maximum. Without jitter, clients that fail together can calculate the same next time, including the same capped interval. Waiting longer does not necessarily break their synchronization.

Full jitter samples between zero and the current ceiling. Because the draw is random, the next realized wait can be shorter than the previous one even while the ceiling grows. It is inaccurate to say every individual wait must increase.

AWS’s comparison distinguishes full, equal, and decorrelated jitter and reports results from a particular contention simulation. Those results are useful evidence, not a promise that one policy minimizes every workload’s latency or prevents all bursts. Correlated random seeds, timer granularity, and many clients can still create clusters.

Use one end-to-end deadline

The budget must include execution, waiting, and subsequent attempts. Creating a fresh full timeout each time silently expands the original caller’s allowed lifetime. Propagate the remaining budget to downstream operations and recheck it before doing more work.

Use a monotonic clock for elapsed-time accounting within a process. A client timeout is not proof that the server stopped or never committed. Cancellation and remote deadlines need real transport support, and the application still needs its retry or idempotency contract.

A server-directed wait such as Retry-After adds another constraint. Do not sample an earlier retry than that boundary merely because a local jitter formula permits it. If the required wait cannot fit, abandon that retry under the current deadline instead of shortening the requirement.

When it breaks

Permanent validation errors do not become valid through repetition. A nonidempotent operation with an uncertain result can duplicate its effect. Select retriable failure classes deliberately; a blanket catch-and-retry loop hides programming errors and may convert a small incident into an overload cycle.

A backoff cap without an attempt or elapsed-time bound still permits endless work. Conversely, a deadline that is too short can reject every expensive legitimate operation. Observe completed work, retry counts, total latency, and abandonment reasons together; a falling error rate alone can hide rising retry cost.

Check it quickly

Save the block as retry_check.py and run python3 retry_check.py with Python 3.10 or newer. It ran with Python 3.14.4 using only the standard library. The clock and downstream calls are deterministic models; the script does not sleep, start servers, or contact a dependency. Its retry policy handles timeouts and 503 only for GET or HEAD.

from collections import Counter
import random


class Clock:
 def __init__(self):
 self.now = 0.0

 def advance(self, seconds):
 self.now += seconds


class ScriptedCall:
 def __init__(self, clock, steps):
 self.clock, self.steps, self.timeouts = clock, iter(steps), []

 def __call__(self, timeout):
 self.timeouts.append(timeout)
 status, duration = next(self.steps)
 self.clock.advance(min(duration, timeout))
 if duration >= timeout:
 raise TimeoutError("the cooperative fixture used the remaining budget")
 return status


def retry(call, clock, budget, sample, method="GET", max_attempts=3):
 deadline = clock.now + budget
 for attempt in range(1, max_attempts + 1):
 remaining = deadline - clock.now
 if remaining <= 0:
 return "deadline", attempt - 1
 try:
 status = call(remaining)
 except TimeoutError:
 status = None
 if clock.now >= deadline:
 return "deadline", attempt
 if status == 200:
 return "success", attempt
 if status not in (None, 503) or method not in ("GET", "HEAD"):
 return "not-retryable", attempt
 if attempt == max_attempts:
 return "attempt-limit", attempt
 ceiling = min(8.0, 2.0 ** (attempt - 1))
 delay = ceiling * sample()
 assert 0 <= delay <= ceiling
 if delay >= deadline - clock.now:
 return "wait-exceeds-budget", attempt
 clock.advance(delay)
 raise AssertionError("positive attempt limit required")


clock = Clock()
call = ScriptedCall(clock, [(503, 0.25), (503, 0.25), (200, 0.25)])
assert retry(call, clock, 5, lambda: 0.5) == ("success", 3)
assert clock.now == 2.25 and call.timeouts == [5, 4.25, 3]
print("three attempts finish at 2.25; propagated timeouts shrink from 5 to 4.25 to 3")

for method, status in (("POST", 503), ("GET", 400)):
 clock = Clock()
 call = ScriptedCall(clock, [(status, 0.25)])
 assert retry(call, clock, 5, lambda: 0.5, method) == ("not-retryable", 1)
clock = Clock()
call = ScriptedCall(clock, [(503, 0.25)] * 4)
assert retry(call, clock, 5, lambda: 0.5) == ("attempt-limit", 3)
assert len(call.timeouts) == 3
print("POST, permanent failure, and the total-attempt cap stop further work")

clock = Clock()
call = ScriptedCall(clock, [(503, 0.25)] * 7)
assert retry(call, clock, 100, lambda: 1.0, max_attempts=7) == ("attempt-limit", 7)
assert clock.now == 32.75 # Seven 0.25 calls plus waits 1, 2, 4, 8, 8, 8.
print("the exponential ceiling stops at 8 rather than growing without bound")

clock = Clock()
call = ScriptedCall(clock, [(503, 0.1)])
assert retry(call, clock, 0.5, lambda: 1.0) == ("wait-exceeds-budget", 1)
assert clock.now == 0.1 and len(call.timeouts) == 1
clock = Clock()
call = ScriptedCall(clock, [(503, 99)])
assert retry(call, clock, 0.5, lambda: 0.5) == ("deadline", 1)
assert clock.now == 0.5
clock = Clock()
call = ScriptedCall(clock, [])
assert retry(call, clock, 0, lambda: 0.5) == ("deadline", 0) and call.timeouts == []
print("a wait that cannot fit is abandoned; cooperative execution cannot consume a fresh budget per attempt")

rng = random.Random(7) # Repeatable samples for this experiment, not a production seed.
delays = [rng.random() for _ in range(100)]
bins = Counter(int(delay * 10) for delay in delays)
fixed_bins = Counter(int(1.0 * 10) for _ in range(100))
assert all(0 <= delay < 1 for delay in delays) and sum(bins.values()) == 100
assert max(bins.values()) < 100
assert max(fixed_bins.values()) == 100
print("first-retry 0.1-second-bin peak: fixed delay", max(fixed_bins.values()), "; full-jitter sample", max(bins.values()))

leaf_calls = 0


class Unavailable(Exception):
 pass


def nested(depth):
 global leaf_calls
 if depth == 0:
 leaf_calls += 1
 raise Unavailable
 for _ in range(3):
 try:
 return nested(depth - 1)
 except Unavailable:
 pass
 raise Unavailable


try:
 nested(3)
except Unavailable:
 pass
assert leaf_calls == 27
print("three retrying layers, each allowing three total attempts, invoke the failing leaf 27 times")

Two failures followed by success finish at modeled time 2.25. The propagated timeouts shrink from five to 4.25 to three seconds; execution does not receive a fresh budget. Separate assertions cover the total-attempt limit, an eight-second ceiling, permanent errors, POST exclusion, a wait that cannot fit, and an already exhausted deadline.

One hundred fixed first-retry delays land in one 0.1-second bin. The reproducible full-jitter sample has a peak bin count of thirteen. This compares calculated schedules, not actual network arrival times or successful throughput. The fixed random seed exists only to reproduce the experiment, not to synchronize production clients.

Finally, three nested retry loops produce twenty-seven leaf invocations before failure. The example deliberately shows that amplification rather than recommending it. Wider retry quotas, server-directed delay parsing, real timers, transport cancellation, distributed clocks, and production latency remain outside this model’s tests.

Sources

AWS: Exponential backoff and jitter, capped backoff, full-jitter formula, and workload-specific comparisons, checked 2026-09-08.
Google SRE: Handling overload, request/client retry budgets and retry ownership, checked 2026-09-08.
Google SRE: Addressing cascading failures, retry multiplication, deadline propagation, and cancellation, checked 2026-09-08.
RFC 9110, idempotent-method retries and Retry-After semantics, checked 2026-09-08.

A bind address selects local destinations; reachability needs the whole path

Fri, 02 Apr 2027 00:00:00 +0000

A server’s bind address selects which local destination addresses its socket can accept. Binding an IPv4 listener to 127.0.0.1 differs from binding it to 0.0.0.0, the wildcard address. Neither setting alone describes the complete access policy. Routing, network namespaces, packet filtering, port publication, proxies, and application authentication determine how a request reaches the listener and what it may do afterward.

When it breaks

A wildcard listener is not proof that the internet can reach a service. It may run inside an isolated namespace without an external interface or have a restrictive firewall. Conversely, a loopback listener can be reached indirectly through a reverse proxy or tunnel that accepts remote requests. Investigate those forwarding paths before describing the service as accessible only to a particular group of people.

The IPv6 wildcard :: adds another choice. On Linux, IPV6_V6ONLY controls whether an IPv6 socket also accepts IPv4 connections as mapped addresses. Defaults depend on the environment and application. Seeing one IPv6 entry in a listener listing therefore does not establish that IPv4 is unavailable. Set the option deliberately when the service contract requires a particular family boundary, then test each family.

Binding to a specific address can also fail during startup. An address already occupied by a conflicting socket produces a different error from an address not assigned locally. Reuse options and nonlocal-binding facilities alter some cases, but they are not enabled here. Treating every bind failure as a busy port can send an operator toward killing an unrelated process when the actual fault is missing address configuration.

Check it quickly

Save the script as check-bind-address.py and run python3 check-bind-address.py with Docker. The pinned Python 3.11.16 image runs Linux sockets inside a disposable container with no published ports. The experiment compares two IPv4 loopback addresses, then explicitly changes IPV6_V6ONLY. It does not connect to a public host or change the host firewall:

import json,subprocess,uuid
image="python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84"
program=r"""
from contextlib import ExitStack
import errno,json,socket

def read_exact(stream,length):
 result=b''
 while len(result)<length:
 part=stream.recv(length-len(result));assert part,'unexpected end of stream'
 result+=part
 return result
def listener(stack,address,family=socket.AF_INET,v6only=None):
 stream=stack.enter_context(socket.socket(family,socket.SOCK_STREAM))
 if v6only is not None: stream.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,v6only)
 stream.settimeout(1);stream.bind((address,0));stream.listen(4)
 return stream
def exchange(server,address):
 with socket.create_connection((address,server.getsockname()[1]),timeout=1) as client:
 connection,peer=server.accept()
 with connection:
 connection.settimeout(1);client.sendall(b'ping');assert read_exact(connection,4)==b'ping'
 connection.sendall(b'pong');assert read_exact(client,4)==b'pong'
 return {'listener':server.getsockname(),'accepted_local':connection.getsockname(),'peer':peer}
def refused(server,address):
 try:
 with socket.create_connection((address,server.getsockname()[1]),timeout=1):pass
 except ConnectionRefusedError:return 'ECONNREFUSED'
 raise AssertionError('unexpectedly connected')
with ExitStack() as stack:
 narrow=listener(stack,'127.0.0.1')
 one=exchange(narrow,'127.0.0.1');other=refused(narrow,'127.0.0.2')
 duplicate=stack.enter_context(socket.socket())
 try:duplicate.bind(narrow.getsockname());raise AssertionError('duplicate bind succeeded')
 except OSError as error:assert error.errno==errno.EADDRINUSE
 unassigned=stack.enter_context(socket.socket())
 try:unassigned.bind(('192.0.2.99',0));raise AssertionError('unassigned bind succeeded')
 except OSError as error:assert error.errno==errno.EADDRNOTAVAIL
 wildcard=listener(stack,'0.0.0.0')
 both=[exchange(wildcard,a) for a in ['127.0.0.1','127.0.0.2']]
 narrow.close();wildcard.close()
 only6=listener(stack,'::',socket.AF_INET6,1)
 six=exchange(only6,'::1');four_rejected=refused(only6,'127.0.0.1')
 dual=listener(stack,'::',socket.AF_INET6,0)
 mapped=exchange(dual,'127.0.0.1');assert mapped['peer'][0].startswith('::ffff:')
 print(json.dumps({'narrow':one,'other_loopback':other,'duplicate':'EADDRINUSE','unassigned':'EADDRNOTAVAIL',
 'wildcard':both,'ipv6_only':six,'ipv6_only_v4':four_rejected,'dual_stack':mapped}))
"""
name='xbcatm-socket-'+uuid.uuid4().hex[:10]
try:
 result=subprocess.run(['docker','run','--rm','-i','--name',name,'--network','none',
 '--memory','128m','--pids-limit','64',image,'python3','-'],input=program,
 capture_output=True,text=True,timeout=60)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),ensure_ascii=False,sort_keys=True))
finally:
 subprocess.run(['docker','rm','-f',name],capture_output=True,text=True,timeout=10)

The exact listener accepts a request to 127.0.0.1 and returns pong. Connecting to 127.0.0.2 on the same port is refused. Both are loopback destinations, so this comparison isolates address matching without claiming to test external filtering. Each listener binds port zero, allowing the kernel to choose an available port; the program reads that actual port before connecting.

The wildcard listener successfully serves both destinations. Its listening socket still reports 0.0.0.0, while each accepted connection reports the concrete local address used by that client. This difference matters when interpreting diagnostics: a wildcard describes the listener’s matching rule, whereas an accepted connection has a specific local endpoint. The example reads the complete four-byte response instead of assuming that stream reads preserve message boundaries.

Two deliberate setup errors are separately asserted. A second ordinary socket cannot bind the exact listener’s occupied address and port, yielding EADDRINUSE. Binding 192.0.2.99, which is not assigned in this namespace, yields EADDRNOTAVAIL. These negative controls verify the expected causes; an arbitrary exception or timeout would fail the exercise instead of being counted as equivalent evidence.

With IPV6_V6ONLY set to one, the IPv6 wildcard accepts ::1 and refuses IPv4 on the same port. With the option set to zero, a separate IPv6 listener accepts an IPv4 connection and reports ::ffff:127.0.0.1. The observed mapping follows the explicit socket option. It is not a prediction of another operating system’s default or a recommendation to disable IPv6.

Verify the listener and the permitted caller

Record address family, bind address, selected port, process, and network namespace together. A listener inside a container and one on its host can use identical address text without referring to the same socket. Check the application’s effective configuration after startup, because a deployment variable or library default may differ from the configuration file an operator first inspected.

Choose the smallest address scope compatible with legitimate callers, then enforce the required source and application policy. A proxy may need to reach a backend through a container interface rather than its own loopback. Binding the backend narrowly should follow that actual topology. Authentication still matters for local callers and for remote requests forwarded by an authorized intermediary.

Finally, test both permitted and rejected paths from the namespaces that represent real clients. This exercise establishes local binding semantics, duplicate-address errors, and explicitly configured address-family behavior. It does not prove a production firewall, tunnel, or load balancer is configured correctly. Those components must be checked along the real request path before publishing an access guarantee.

Sources

Linux bind(2), binding errors and local socket addresses, checked 2026-09-12.
Linux ip(7), IPv4 wildcard and loopback binding, checked 2026-09-12.
Linux IPV6_V6ONLY, explicit IPv6-only and mapped IPv4 behavior, checked 2026-09-12.
Python 3.11 socket, socket creation, options and endpoint inspection, checked 2026-09-12.

SRV discovery gives clients targets and ports that they must interpret

Sat, 13 Mar 2027 00:00:00 +0000

An SRV record describes a service target using priority, weight, port, and a target hostname. Its owner normally combines a service label, transport label, and domain, such as _echo._tcp.lab.test. A client that understands that service can query SRV, obtain target addresses through A or AAAA, and connect to the advertised port. Ordinary hostname lookup does not automatically perform this application-specific discovery sequence.

When it breaks

Publishing SRV does not make every existing client use it. The application needs a defined service label and supported discovery behavior. A browser or library expecting an ordinary hostname and configured port may never ask for SRV. A successful DNS response to a manual SRV query therefore cannot establish that the application’s endpoint selection has changed or that redeploying configuration is now unnecessary.

Priority and weight are not interchangeable. Lower numeric priority is preferred among reachable targets. Weight influences selection among records with the same priority; a large weight on a higher-priority-number backup does not override an available lower-numbered target. Simply sorting every record by weight or selecting the first wire-order entry can reverse the service owner’s intended primary and backup relationship.

Discovery also does not prove health. A target can remain published after its listener stops, and cached SRV and address RRsets have independent lifetimes. A client must apply connection deadlines and appropriate retry behavior. Retrying a read after connection refusal is different from repeating an operation whose side effects might already have occurred, so endpoint failover must remain consistent with the application protocol.

Check it quickly

Save this as check-srv-discovery.py and run python3 check-srv-discovery.py with Docker. The pinned BIND 9.18.49 environment publishes a test _echo._tcp service. Two real HTTP listeners provide identifiable read-only responses on different temporary ports. HTTP is used as the fixture’s observable payload; the example is a custom SRV-aware client, not a claim that generic HTTP clients implement this service label:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends bind9=1:9.18.49-1~deb12u2 bind9-dnsutils=1:9.18.49-1~deb12u2 dnsmasq-base=2.90-4~deb12u2 python3-dnspython=2.3.0-1 && rm -rf /var/lib/apt/lists/*\nCMD ["/usr/bin/python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,re,signal,subprocess,time
import dns.exception,dns.message,dns.query,dns.rcode,dns.rdatatype

def run(*args,check=True):
 result=subprocess.run(args,capture_output=True,text=True,timeout=15)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def query(name,kind='A',port=1053):
 return dns.query.udp(dns.message.make_query(name,kind),'127.0.0.1',port=port,timeout=1)
def values(response,kind='A'):
 wanted=dns.rdatatype.from_text(kind)
 return [item.to_text() for rrset in response.answer if rrset.rdtype==wanted for item in rrset]
def show(response):
 def section(records):return [{'name':str(rr.name),'type':dns.rdatatype.to_text(rr.rdtype),'ttl':rr.ttl,'data':[item.to_text() for item in rr]} for rr in records]
 return {'rcode':dns.rcode.to_text(response.rcode()),'answer':section(response.answer),'authority':section(response.authority)}
def wait(predicate,timeout=15):
 deadline=time.monotonic()+timeout
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except (OSError,dns.exception.Timeout):pass
 time.sleep(.05)
 raise AssertionError('DNS fixture condition did not become true')
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=5)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=5)

@contextmanager
def lab(records,soa_ttl=8,minimum=4,extra=''):
 with TemporaryDirectory(prefix='xbcatm-dns-') as directory,ExitStack() as stack:
 root=Path(directory);zone=root/'zone';conf=root/'named.conf';log=root/'named.log';serial=1
 def save(body):
 zone.write_text(f'$ORIGIN lab.test.\n$TTL 30\n@ {soa_ttl} IN SOA ns.lab.test. hostmaster.lab.test. ({serial} 60 60 3600 {minimum})\n@ IN NS ns.lab.test.\nns IN A 127.0.0.1\n'+body+'\n')
 run('named-checkzone','lab.test',str(zone))
 save(records)
 conf.write_text(f'''options {{ directory "{root}"; listen-on port 1053 {{127.0.0.1;}}; listen-on-v6 {{none;}};
recursion no; dnssec-validation no; notify no; querylog yes; max-cache-size 8m; pid-file "{root}/named.pid"; session-keyfile none;
{extra} }};
controls {{}};
zone "lab.test" {{ type primary; file "{zone}"; }};
''')
 run('named-checkconf',str(conf))
 output=stack.enter_context(log.open('w'))
 authority=subprocess.Popen(['named','-g','-c',str(conf),'-n','1','-U','1'],stdout=output,stderr=subprocess.STDOUT)
 stack.callback(stop,authority)
 def ready():
 assert authority.poll() is None,log.read_text()
 response=query('lab.test.','SOA');return response.rcode()==0 and response.answer[0][0].serial==serial
 wait(ready)
 def update(body):
 nonlocal serial
 serial+=1;save(body);authority.send_signal(signal.SIGHUP);wait(ready)
 def cache(port=1054,upstream=1053,size=150,options=()):
 logfile=root/('cache-'+str(port)+'.log');stream=stack.enter_context(logfile.open('w'))
 command=['dnsmasq','--no-daemon','--conf-file=/dev/null','--no-resolv','--no-hosts','--bind-interfaces',
 '--listen-address=127.0.0.1','--port='+str(port),'--server=127.0.0.1#'+str(upstream),
 '--cache-size='+str(size),'--log-queries','--log-facility=-',*options]
 process=subprocess.Popen(command,stdout=stream,stderr=subprocess.STDOUT);stack.callback(stop,process)
 def cache_ready():
 assert process.poll() is None,logfile.read_text()
 return query('lab.test.','SOA',port).rcode()==0
 wait(cache_ready);return process,logfile
 def queries(name,kind='A'):
 return sum(bool(re.search(r'query: '+re.escape(name.rstrip('.'))+r' IN '+kind+r'\b',line)) for line in log.read_text().splitlines())
 yield root,update,cache,queries

@contextmanager
def named_process(root,config):
 root.mkdir(exist_ok=True);conf=root/'named.conf';log=root/'named.log';conf.write_text(config)
 run('named-checkconf',str(conf))
 with log.open('w') as output:
 process=subprocess.Popen(['named','-g','-c',str(conf),'-n','1','-U','1'],stdout=output,stderr=subprocess.STDOUT)
 try:
 def ready():
 assert process.poll() is None,log.read_text()
 return re.search(r'\brunning\s*$',log.read_text(),re.M) is not None
 wait(ready);yield process,log
 finally:stop(process)
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import http.client,threading
class Handler(BaseHTTPRequestHandler):
 def log_message(self,*args):pass
 def do_GET(self):
 body=self.server.identity.encode();self.send_response(200);self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
servers=[];threads=[]
try:
 for identity in ('primary','backup'):
 server=ThreadingHTTPServer(('127.0.0.1',0),Handler);server.identity=identity
 thread=threading.Thread(target=server.serve_forever,daemon=True);thread.start();servers.append(server);threads.append(thread)
 records=f'''_echo._tcp 30 IN SRV 10 1 {servers[0].server_port} first.lab.test.
_echo._tcp 30 IN SRV 20 65535 {servers[1].server_port} second.lab.test.
_absent._tcp 30 IN SRV 0 0 0 .
first 30 IN A 127.0.0.1
second 30 IN A 127.0.0.1'''
 with lab(records) as (_,update,cache,queries):
 srv=query('_echo._tcp.lab.test.','SRV');entries=sorted(srv.answer[0],key=lambda r:r.priority)
 assert [r.priority for r in entries]==[10,20] and [r.weight for r in entries]==[1,65535]
 def discover():
 attempts=[]
 for record in entries:
 addresses=values(query(str(record.target)))
 for address in addresses:
 client=http.client.HTTPConnection(address,record.port,timeout=.5)
 try:
 attempts.append(str(record.target));client.request('GET','/');response=client.getresponse();body=response.read().decode()
 assert response.status==200;return {'identity':body,'attempted_targets':attempts,'selected_port':record.port}
 except OSError:continue
 finally:client.close()
 raise AssertionError('all service endpoints failed')
 first=discover();assert first['identity']=='primary' and first['attempted_targets']==['first.lab.test.']
 servers[0].shutdown();servers[0].server_close()
 assert len(query('_echo._tcp.lab.test.','SRV').answer[0])==2
 failed_over=discover();assert failed_over['identity']=='backup' and len(failed_over['attempted_targets'])==2
 absent=query('_absent._tcp.lab.test.','SRV');assert str(absent.answer[0][0].target)=='.'
 ordinary=query('lab.test.','A');assert ordinary.rcode()==0 and not ordinary.answer
 print(json.dumps({'SRV':show(srv),'healthy_selection':first,'after_primary_stopped':failed_over,
 'ordinary_A_lookup':show(ordinary),'unavailable_service':show(absent)},sort_keys=True))
finally:
 for server in servers:server.shutdown();server.server_close()
 for thread in threads:thread.join(timeout=2)
"""
name='xbcatm-dns-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 built=docker('build','-t',name,str(root));assert built.returncode==0,built.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--memory','384m','--pids-limit','128',name,
 '/usr/bin/python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The first target has priority 10 and weight 1; the backup has priority 20 and weight 65535. Each advertises its actual listener port, and both target names resolve to loopback. The client reads SRV, orders these distinct priority groups, resolves each target, and connects using the record’s port. With both listeners healthy, it reaches only the first target and receives the primary identity.

The program stops the primary listener while leaving DNS unchanged. A fresh SRV query still contains both records. Reusing the previously discovered candidate set, the client attempts the primary, encounters connection failure, and then successfully contacts the backup. The result records both attempted names and the selected port. DNS did not remove the failed target; the client’s connection handling performed the transition.

A separate service has one SRV record whose target is the root name, .. RFC 2782 defines that form as explicit service unavailability at the domain, so it is not an instruction to resolve the root and connect there. Meanwhile, an ordinary A query for lab.test returns no address despite the _echo._tcp records being present. The query type and discovery name are part of the contract.

Match the client algorithm to the published record set

This fixture intentionally has one target per priority group. Its priority ordering is sufficient for those records and does not implement weighted ordering among equal-priority peers. A general client needs the complete selection behavior appropriate to its protocol, including equal-priority weights and the unavailable-service case. Test those branches with the actual library rather than assuming an SRV-capable label implies full conformance.

SRV targets should name hosts with address records and must not be aliases under RFC 2782’s rules. Check the target’s A and AAAA data, port listener, and required service identity. A returned address alone does not verify TLS certificates, authentication, or application readiness. Some protocols define additional discovery records or identity rules; their documented behavior takes precedence over a generic example’s convenient layout.

Refresh discovery according to the relevant cache lifetimes and deployment policy. Changing SRV does not withdraw a cached target, and changing a target address does not modify an already established connection. During planned removal, account for retained candidates and drain existing work. Observe authoritative publication, resolver answers, and actual connection attempts to distinguish stale discovery from a reachable server rejecting the application operation.

Keep ownership of service labels and update permissions explicit. DNS-based discovery can reduce hard-coded endpoint configuration, but it shifts responsibility into records and client selection logic. Verify a normal request, a failed preferred endpoint, and the resulting alternate connection. Those observations demonstrate the useful behavior more directly than merely confirming that a service record exists in a zone file.

Sources

RFC 2782 DNS SRV records, priority, weight, port, target, and explicit service unavailability, checked 2026-09-12.
RFC 1035 domain names, address records and DNS caching, checked 2026-09-12.
BIND 9.18.49 reference, authoritative zone data and service records, checked 2026-09-12.

A Service gives a pod a stable virtual address

Tue, 09 Feb 2027 00:00:00 +0000

The selector turns the Service from a fixed Endpoint into a live view: refresh happens in the background, and a client never edits its config when pods restart.

Declared, then given an address

A Service turns the constant churn of pods into a stable name. You declare a selector (which pods), a port, and a protocol, and the control plane assigns a virtual IP (cluster IP) and a DNS name that stay fixed while everything behind them keeps changing. The stable name is the contract for every other resource: a Deployment, an Ingress, or another pod can address a service by name without caring which concrete pod exists right now. That indirection is what lets the runtime rearrange the membership without a client ever noticing.

Selector is the live membership

The backend list a Service routes to is not a fixed set, it is rebuilt continuously from the selector the moment pods match or stop matching it. A pod that fails its readiness probe is dropped from the endpoints; a new pod that passes joins; the update happens in the background. Because membership is derived, you never hand-edit the list and you never drift: as long as the selector and the pod labels line up, the Service automatically reflects the current, healthy backends. That readme is what makes a Service ahead of static lists you would have to maintain by hand.

Stable even as pods churn

Clients talk only to the virtual address, never to an individual pod IP, which is exactly what decouples the caller from the fleet. You can scale up, scale down, replace, or kill a pod and the Service keeps answering through the survivors; the client sees no change because its target never changes. That stability is the whole reason Services allow rolling restarts and horizontal scaling without a coordinated announcement. Keep the selector precise and probe-gated, and the Service gives you a stable facade over an inherently unstable set of workers.

When it breaks

The pool is finite, and exhaustion is the most literal failure. Every ordinary Service takes one address from --service-cluster-ip-range, so a small CIDR or leftover Services eventually reject creation with an allocation error while the pods look fine. Treat failed to allocate IP as capacity accounting: delete orphaned Services before widening the range, or widening only postpones the outage.

Mixing headless and ordinary Services is the next confusion. Setting clusterIP: None allocates no virtual address, so DNS returns pod IPs instead of one stable VIP, and a client written for a single endpoint must suddenly balance across shifting addresses. The reverse also hurts: discovery code expecting per-pod DNS keeps hitting one VIP after someone recreates the Service with a ClusterIP. Because clusterIP is effectively immutable, switching modes means recreating the Service.

The last two failures live below the API. Iptables and IPVS modes program the same VIP differently, and a stale or restarting kube-proxy can leave an allocated ClusterIP resolving but blackholing traffic until rules resync. Finally, copying a ClusterIP into configs freezes a value the control plane may reassign: the address is stable only while the Service object lives. Callers must use the DNS name, and remember the VIP routes only inside the cluster.

Check it quickly

Save the following as clusterip_pool.py in a scratch directory and run python3 clusterip_pool.py. It models a miniature allocator over 10.96.0.0/29 with the first address reserved, using only the standard library and changing nothing outside the process: one class hands out addresses, refuses duplicates and out-of-range requests, lets headless Services bypass the pool, and reuses released addresses, with five scenarios asserting each rule.

"""Simulate ClusterIP allocation from a small service CIDR."""

import ipaddress


class ClusterIPPool:
 """Hand out virtual addresses; headless Services bypass the pool."""

 def __init__(self, cidr, reserved=1):
 network = ipaddress.ip_network(cidr)
 hosts = list(network.hosts())
 self._all = set(hosts)
 self._reserved = set(hosts[:reserved])
 self.free = sorted(set(hosts) - self._reserved)
 self.allocated = {}

 def allocate(self, service, requested=None, headless=False):
 if headless:
 return None
 if service in self.allocated:
 raise ValueError(f"{service} already holds {self.allocated[service]}")
 if requested is not None:
 ip = ipaddress.ip_address(requested)
 if ip in self._reserved or ip not in self._all:
 raise ValueError(f"{requested} is outside the allocatable range")
 if ip not in self.free:
 raise ValueError(f"{requested} is already taken")
 self.free.remove(ip)
 self.allocated[service] = ip
 return str(ip)
 if not self.free:
 raise RuntimeError("service CIDR exhausted: no ClusterIP left")
 ip = self.free.pop(0)
 self.allocated[service] = ip
 return str(ip)

 def release(self, service):
 ip = self.allocated.pop(service, None)
 if ip is None:
 raise ValueError(f"{service} holds nothing to release")
 self.free.append(ip)
 self.free.sort()
 return str(ip)


def check(condition, message):
 if not condition:
 raise AssertionError(message)


def main():
 pool = ClusterIPPool("10.96.0.0/29", reserved=1)
 web = pool.allocate("web")
 api = pool.allocate("api", requested="10.96.0.5")
 check(web != api, "sequential Services must get distinct addresses")
 print("allocated:", web, api)

 try:
 pool.allocate("worker", requested="10.96.0.5")
 raise SystemExit("conflict was silently shared")
 except ValueError as exc:
 print("conflict rejected:", exc)

 try:
 pool.allocate("evil", requested="10.99.0.1")
 raise SystemExit("out-of-range request was accepted")
 except ValueError as exc:
 print("out-of-range rejected:", exc)

 headless = pool.allocate("db-headless", headless=True)
 check(headless is None, "headless Service must bypass the pool")
 check(len(pool.allocated) == 2, "headless must not consume the pool")
 print("headless bypass: None")

 freed = pool.release("web")
 worker = pool.allocate("worker")
 check(worker == freed, "released address must be reusable")
 print("reused:", freed, "-> worker")

 pool.allocate("extra-1")
 pool.allocate("extra-2")
 pool.allocate("extra-3")
 try:
 pool.allocate("overflow")
 raise SystemExit("exhaustion was not reported")
 except RuntimeError as exc:
 print("exhaustion reported:", exc)

 print("PASS: allocation, conflict rejection, release reuse, and exhaustion hold")


if __name__ == "__main__":
 main()

The run behaves like a healthy allocator: sequential Services get distinct addresses, a pinned request holds its address, and the deliberate double allocation is rejected instead of shared. Releasing one Service makes its address reusable, and filling the last slot proves exhaustion errors instead of recycling a live address. All assertions passed with Python 3.14.4 on macOS.

The simulation checks bookkeeping only. It does not run the API server, etcd, or kube-proxy, and its tiny /29 stands in for a real service CIDR. Before deploying, inspect the live service CIDR, allocated ClusterIPs, and proxy rules on a node.

Sources

Kubernetes Service, documents service semantics, checked 2026-06-01.
Virtual IPs and Service Proxies, documents ClusterIP allocation and kube-proxy modes, checked 2026-06-01.

gRPC multiplexes many streams over one HTTP/2 connection

Sat, 16 Jan 2027 00:00:00 +0000

Multiplexing shares the transport and keeps ordering within each stream, which is why long-lived pushes fit this model better than request-reply websockets.

One connection, many streams

HTTP/2 multiplexes many logical streams over a single TCP connection, and gRPC makes each call a stream, so a chatty microservice stops reopening sockets per request. That one shared connection means connection setup, TLS handshake, and idle bookkeeping happen once instead of per call. Streams are independent in ordering and cancellation, so a slow, long-lived stream does not block a short one on the same pipe. The practical effect is a low army of connection state, with per-call overhead shrinking to a frame header.

Streaming beats HTTP/1.1 fan-out

Because gRPC is streaming in both directions, a server can begin sending a partial result before the client finishes sending the request, and a client can keep pushing frames over the same call. That fits search, large-file streaming, log shipping, and interactive feeds where a single reply would force the whole payload to buffer first. It also makes fan-out cheaper than HTTP/1.1, where a burst of calls would each need its own connection and its own handshake. Everything that needs a back-and-forth loop maps naturally onto one long-lived stream instead of a series of separate requests.

The elegance of sharing one connection is real, but it is still bounded by the transport. A retry requires a new stream, not a replay on the same one, and backpressure still matters, since a slow consumer can stall the shared connection for everyone else. Deadlines therefore cut across the whole pipe, so one hair-loss deadline on a stream can surface as latency for every stream on that connection. The practical habit is to give each call a real deadline and a way to cancel, rather than letting the shared pipe silently donate time to someone else’s stuck call.

When it breaks

A stalled flow-control window looks like a healthy connection with zero throughput. HTTP/2 keeps a stream-level and a connection-level window, and only the receiver advances them with WINDOW_UPDATE. A slow consumer first parks its own stream; once the connection window fills, every stream behind it stalls too. Separate the two signals in dashboards, and give each RPC a bounded buffer and a consumer timeout.

Oversize metadata fails fast and retries cannot help. gRPC sends initial metadata as HTTP/2 headers with caps near eight kilobytes. An auth token plus tracing baggage plus tenant tags can cross the limit, and the call dies with RESOURCE_EXHAUSTED before business logic runs. Keep metadata small, move correlation blobs out of band, and remember binary metadata pays base64 inflation.

Long-lived connections defeat naive balancing. Clients that each hold one multiplexed connection pin all streams to the backend chosen at dial time, leaving new pods idle while old ones saturate. Fix it with client-side balancing or proxyless xDS, periodic MaxConnAge reconnects, and a cap on concurrent streams per connection.

Lost deadline propagation turns one slow leaf into whole-tree latency. gRPC carries the budget as grpc-timeout, but a gateway that fans out with a fresh timeout or a background context silently drops it. Derive downstream calls from the incoming context, keep each attempt shorter than the end-to-end deadline, and cancel siblings after the first failure.

Check it quickly

Save the following as mux_sim.py and run python3 mux_sim.py. It uses only the standard library to model one connection carrying three logical streams as interleaved frames. Stream 2 sends one frame and then stalls to simulate an exhausted window, while the scheduler must finish streams 1 and 3. Assertions check per-stream order and wire interleaving.

from collections import deque

# One shared connection, three logical streams. Each stream is an
# ordered queue of application messages; the wire is a single log of
# interleaved (stream_id, payload) frames.
streams = {
 1: deque(["a1", "a2", "a3"]),
 2: deque(["b1", "b2", "b3"]),
 3: deque(["c1", "c2"]),
}
stalled = set() # streams whose flow-control window is exhausted
wire = [] # shared transport log
delivered = {sid: [] for sid in streams}


def pump():
 """One round-robin pass: each ready stream emits at most one frame."""
 progressed = False
 for sid in sorted(streams):
 if sid in stalled:
 continue
 if streams[sid]:
 payload = streams[sid].popleft()
 wire.append((sid, payload))
 delivered[sid].append(payload)
 progressed = True
 return progressed


def drain():
 while pump():
 pass


# Round one: every stream places its opening frame on the wire.
assert pump() is True
assert wire == [(1, "a1"), (2, "b1"), (3, "c1")], wire

# Stream 2 exhausts its window and stops sending WINDOW_UPDATE.
stalled.add(2)
drain()

# Siblings must complete while stream 2 is stuck on one frame.
assert delivered[1] == ["a1", "a2", "a3"], delivered[1]
assert delivered[3] == ["c1", "c2"], delivered[3]
assert delivered[2] == ["b1"], delivered[2]
assert len(wire) == 6, wire
assert list(streams[2]) == ["b2", "b3"]

# The window reopens: stream 2 resumes on the same connection.
stalled.discard(2)
drain()

assert delivered[2] == ["b1", "b2", "b3"], delivered[2]
for sid, items in delivered.items():
 assert items == sorted(items), (sid, items) # per-stream order preserved
assert len(wire) == 8, wire
assert all(not q for q in streams.values())

print("wire:", wire)
print("PASS: 3 streams, 8 frames, one shared wire; stall of stream 2 did not block 1 and 3")

The script pumps one round so each stream places its opening frame, then stalls stream 2. The next drain must complete streams 1 and 3 while stream 2 stays at one frame, proving one blocked stream does not block its siblings. Reopening the window drains the rest, and the final check confirms in-order arrival on one log. This models scheduling only, without HPACK, TLS, or real flow-control bytes, so confirm window and deadline behavior on a gRPC stack before tuning production.

Sources

gRPC core concepts, defines streams over HTTP/2, checked 2026-06-01.
RFC 9113: HTTP/2, streams, multiplexing, and flow control, checked 2026-09-09.

Useful hostnames distinguish local identity, service names, and lookup scope

Tue, 01 Dec 2026 00:00:00 +0000

A useful name gives people and software a reference they can carry across an address change. That reference needs a defined scope and owner. A machine’s local hostname, a service’s DNS name, and an instance identifier may intentionally differ. Naming them clearly helps operators understand logs and deployments without pretending that one label automatically establishes every other identity in the system.

When it breaks

Setting a local hostname does not, by itself, publish an A or AAAA record in an authoritative zone. Registration may be performed by separate provisioning or network-management software, but it must actually exist. Conversely, changing a DNS address does not rename the local kernel hostname. Confusing these operations produces servers whose local label looks correct while clients cannot discover the intended service.

Short names depend on lookup context. service can resolve differently under two search domains even when both machines appear to use normal DNS settings. A copied runbook command may therefore reach a different environment from the one its author tested. Use the complete intended DNS name in configuration when ambiguity matters, and preserve the context that explains short names in interactive use.

Human meaning also has limits. A name such as app-prod-01 can suggest role, environment, and instance, but it is not authoritative evidence of permissions, ownership, or current workload. Reusing an old instance name can confuse historical logs. Keep stable service names separate from replacement-machine identity when their lifetimes differ, and attach operational metadata instead of trying to encode every changing fact in the hostname.

Check it quickly

Save this as check-hostnames.py and run python3 check-hostnames.py with Docker. The pinned environment uses BIND 9.18.49, dnsmasq 2.90, and Python 3.11. The container’s resolver and NSS files are disposable. Forwarding cache is disabled for this exercise so the search-domain and address changes can be observed directly without waiting for a previous cached generation:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends bind9=1:9.18.49-1~deb12u2 bind9-dnsutils=1:9.18.49-1~deb12u2 dnsmasq-base=2.90-4~deb12u2 python3-dnspython=2.3.0-1 && rm -rf /var/lib/apt/lists/*\nCMD ["/usr/bin/python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,re,signal,subprocess,time
import dns.exception,dns.message,dns.query,dns.rcode,dns.rdatatype

def run(*args,check=True):
 result=subprocess.run(args,capture_output=True,text=True,timeout=15)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def query(name,kind='A',port=1053):
 return dns.query.udp(dns.message.make_query(name,kind),'127.0.0.1',port=port,timeout=1)
def values(response,kind='A'):
 wanted=dns.rdatatype.from_text(kind)
 return [item.to_text() for rrset in response.answer if rrset.rdtype==wanted for item in rrset]
def show(response):
 def section(records):return [{'name':str(rr.name),'type':dns.rdatatype.to_text(rr.rdtype),'ttl':rr.ttl,'data':[item.to_text() for item in rr]} for rr in records]
 return {'rcode':dns.rcode.to_text(response.rcode()),'answer':section(response.answer),'authority':section(response.authority)}
def wait(predicate,timeout=15):
 deadline=time.monotonic()+timeout
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except (OSError,dns.exception.Timeout):pass
 time.sleep(.05)
 raise AssertionError('DNS fixture condition did not become true')
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=5)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=5)

@contextmanager
def lab(records,soa_ttl=8,minimum=4,extra=''):
 with TemporaryDirectory(prefix='xbcatm-dns-') as directory,ExitStack() as stack:
 root=Path(directory);zone=root/'zone';conf=root/'named.conf';log=root/'named.log';serial=1
 def save(body):
 zone.write_text(f'$ORIGIN lab.test.\n$TTL 30\n@ {soa_ttl} IN SOA ns.lab.test. hostmaster.lab.test. ({serial} 60 60 3600 {minimum})\n@ IN NS ns.lab.test.\nns IN A 127.0.0.1\n'+body+'\n')
 run('named-checkzone','lab.test',str(zone))
 save(records)
 conf.write_text(f'''options {{ directory "{root}"; listen-on port 1053 {{127.0.0.1;}}; listen-on-v6 {{none;}};
recursion no; dnssec-validation no; notify no; querylog yes; max-cache-size 8m; pid-file "{root}/named.pid"; session-keyfile none;
{extra} }};
controls {{}};
zone "lab.test" {{ type primary; file "{zone}"; }};
''')
 run('named-checkconf',str(conf))
 output=stack.enter_context(log.open('w'))
 authority=subprocess.Popen(['named','-g','-c',str(conf),'-n','1','-U','1'],stdout=output,stderr=subprocess.STDOUT)
 stack.callback(stop,authority)
 def ready():
 assert authority.poll() is None,log.read_text()
 response=query('lab.test.','SOA');return response.rcode()==0 and response.answer[0][0].serial==serial
 wait(ready)
 def update(body):
 nonlocal serial
 serial+=1;save(body);authority.send_signal(signal.SIGHUP);wait(ready)
 def cache(port=1054,upstream=1053,size=150,options=()):
 logfile=root/('cache-'+str(port)+'.log');stream=stack.enter_context(logfile.open('w'))
 command=['dnsmasq','--no-daemon','--conf-file=/dev/null','--no-resolv','--no-hosts','--bind-interfaces',
 '--listen-address=127.0.0.1','--port='+str(port),'--server=127.0.0.1#'+str(upstream),
 '--cache-size='+str(size),'--log-queries','--log-facility=-',*options]
 process=subprocess.Popen(command,stdout=stream,stderr=subprocess.STDOUT);stack.callback(stop,process)
 def cache_ready():
 assert process.poll() is None,logfile.read_text()
 return query('lab.test.','SOA',port).rcode()==0
 wait(cache_ready);return process,logfile
 def queries(name,kind='A'):
 return sum(bool(re.search(r'query: '+re.escape(name.rstrip('.'))+r' IN '+kind+r'\b',line)) for line in log.read_text().splitlines())
 yield root,update,cache,queries
import socket
records='service 30 IN A 127.0.0.2\nservice.branch 30 IN A 127.0.0.3\n_echo._tcp 30 IN SRV 0 0 8080 service.lab.test.'
with lab(records) as (root,update,cache,queries):
 cache(port=53,size=0)
 nss=Path('/etc/nsswitch.conf');nss.write_text(re.sub(r'^hosts:.*$','hosts: dns',nss.read_text(),flags=re.M))
 def search(domain):Path('/etc/resolv.conf').write_text('nameserver 127.0.0.1\nsearch '+domain+'\noptions timeout:1 attempts:1\n')
 def address(name):
 return json.loads(run('/usr/bin/python3','-c','import socket,json,sys;print(json.dumps(sorted({r[4][0] for r in socket.getaddrinfo(sys.argv[1],80,socket.AF_INET,socket.SOCK_STREAM)})))',name).stdout)
 local_name=socket.gethostname();unpublished=query(local_name+'.lab.test.');assert unpublished.rcode()==dns.rcode.NXDOMAIN
 lower=query('service.lab.test.');mixed=query('SeRvIcE.LaB.TeSt.');assert values(lower)==values(mixed)==['127.0.0.2']
 search('lab.test');first=address('service');absolute_first=address('service.lab.test.');assert first==absolute_first==['127.0.0.2']
 search('branch.lab.test');second=address('service');absolute_second=address('service.lab.test.');assert second==['127.0.0.3'] and absolute_second==first
 update(records.replace('service 30 IN A 127.0.0.2','service 30 IN A 127.0.0.4'));changed=address('service.lab.test.')
 assert changed==['127.0.0.4'] and socket.gethostname()==local_name
 invalid=root/'invalid.zone';invalid.write_text((root/'zone').read_text()+'bad_name IN A 127.0.0.1\n')
 check=run('named-checkzone','-k','fail','lab.test',str(invalid),check=False)
 assert check.returncode!=0 and 'bad owner name' in check.stdout+check.stderr
 assert values(query('_echo._tcp.lab.test.','SRV'),'SRV')==['0 0 8080 service.lab.test.']
 print(json.dumps({'local_hostname_published_in_zone':False,'case_insensitive_A':values(mixed),
 'short_name_by_search_domain':{'lab.test':first,'branch.lab.test':second},'absolute_name_unchanged_by_search':[absolute_first,absolute_second],
 'DNS_address_after_change':changed,'local_hostname_unchanged':True,'invalid_host_owner_rejected':check.returncode,'underscore_SRV_owner_accepted':True},sort_keys=True))
"""
name='xbcatm-dns-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 built=docker('build','-t',name,str(root));assert built.returncode==0,built.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--memory','384m','--pids-limit','128',name,
 '/usr/bin/python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The script reads the container’s local hostname, then queries that label under lab.test. The authority returns NXDOMAIN because no corresponding zone record was published. The result illustrates the missing registration in this setup; it does not assert that every installation lacks hostname registration. Lowercase and mixed-case queries for service.lab.test both return 127.0.0.2, demonstrating ASCII DNS case-insensitive comparison.

With search domain lab.test, a fresh application’s lookup of service returns 127.0.0.2. Changing the search domain to branch.lab.test makes the same short input return 127.0.0.3. The absolute name service.lab.test. continues returning 127.0.0.2 in both contexts. The trailing root dot makes that test’s absolute DNS name explicit rather than relying on search-list behavior.

Next, the authority changes service.lab.test to 127.0.0.4. A fresh lookup sees that address while the local hostname remains unchanged. This verifies independence between the DNS mapping and local machine label. It does not prove that existing connections move or that other caches refresh immediately; those behaviors have separate lifetimes and are intentionally excluded from this cache-disabled comparison.

Finally, a candidate A owner named bad_name fails named-checkzone -k fail with a bad-owner-name diagnostic. The valid _echo._tcp SRV owner remains accepted and queryable. This separates strict host-address owner checks from the broader DNS namespace: underscores used for service labels do not imply that ordinary hostnames should follow the same naming rules or that every DNS label is a host label.

Choose names for the lifetime and audience they serve

Use a documented, predictable convention with enough meaning to help distinguish environments and instances. Keep sensitive or rapidly changing details out of publicly visible names. When replacing a machine behind a stable service name, update discovery and service configuration deliberately, retaining any required address overlap. The stable spelling helps callers, but working TLS, routing, and application state still need their own preparation.

Record fully qualified names alongside local hostnames in inventory and incident evidence. Preserve case for display if useful, but do not treat ASCII DNS names differing only by case as separate services. When joining datasets, normalize according to the identifier’s actual rules; an arbitrary application ID or a URL path may have different case semantics from its DNS hostname.

Check generated names before deployment against the rules of the systems that consume them. DNS labels, certificate identities, operating-system hostnames, and service-discovery labels are related but not interchangeable validation domains. A name that a kernel accepts is not automatically suitable as an ordinary internet hostname. Meaningful names work best when both their human interpretation and their technical scope remain explicit.

Sources

RFC 1123 Internet host requirements, preferred hostname syntax and host naming requirements, checked 2026-09-12.
RFC 4343 DNS case insensitivity, ASCII case-insensitive DNS comparison, checked 2026-09-12.
Python 3.11 socket documentation, gethostname and application address lookup, checked 2026-09-12.

IPv6 text normalization preserves addresses, but endpoint context still matters

Mon, 30 Nov 2026 00:00:00 +0000

An IPv6 address contains 128 bits, usually written as eight hexadecimal groups. Text may omit leading zeros and replace one consecutive run of zero groups with ::. These rules preserve the address while shortening its spelling. RFC 4291 defines valid forms; RFC 5952 recommends a consistent representation for display and exchange. Valid syntax and the preferred spelling are related but different questions.

When it breaks

Raw string equality can treat one address as several identities. Uppercase letters, leading zeros, and expanded zero groups can all change the text without changing the 128 bits. That can split inventory entries or defeat an application’s intended address comparison. Parse first with a library appropriate to the input, then compare address values instead of assuming lowercasing alone performs full normalization.

Compression also has constraints. :: can appear only once, because two omitted runs would leave their lengths ambiguous. For the recommended representation, compress the longest zero run, choosing the first when lengths tie, and do not compress a single zero field. General IPv6 syntax permits some spellings that these display rules deliberately avoid. A normalizer should not confuse an accepted nonpreferred form with malformed input.

An address is not always a complete endpoint. A URL needs brackets around an IPv6 literal to separate its colons from the port delimiter. A link-local destination may also need a zone identifier to select the intended interface context. Stripping a zone or flattening an IPv4-mapped address without an explicit comparison policy can discard information the caller needs to make the right connection.

Check it quickly

Save this as check-ipv6-text.py and run python3 check-ipv6-text.py with Docker. The pinned Python 3.11.16 image runs standard-library parsing without network traffic. It compares valid variants, rejects malformed inputs, and inspects endpoint components. The test does not assign the example addresses, send IPv6 packets, or establish that any route to them exists:

import json,subprocess,uuid
image="python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84"
program=r"""
from ipaddress import IPv6Address
from urllib.parse import urlsplit
import json

variants=['2001:0DB8:0000:0000:0000:ff00:0042:8329','2001:db8:0:0:0:ff00:42:8329','2001:db8::ff00:42:8329']
parsed=[IPv6Address(v) for v in variants];assert len(set(parsed))==1 and len(set(variants))==3
tie=IPv6Address('2001:0:0:1:0:0:1:1').compressed;assert tie=='2001::1:0:0:1:1'
single=IPv6Address('2001:db8::1:2:3:4:5').compressed;assert single=='2001:db8:0:1:2:3:4:5'
bad=[]
for text in ['2001::db8::1','2001:db8:1:2:3:4:5:6:7','2001:db8::gg','2001:db8::1/64']:
 try:IPv6Address(text);raise AssertionError('invalid address accepted')
 except ValueError:bad.append(text)
url=urlsplit('https://[2001:db8::1]:8443/health');assert IPv6Address(url.hostname)==IPv6Address('2001:db8::1') and url.port==8443
try:urlsplit('https://2001:db8::1:8443/health').port;raise AssertionError('unbracketed endpoint accepted')
except ValueError:bad_url=True
mapped=IPv6Address('::ffff:192.0.2.1');assert str(mapped.ipv4_mapped)=='192.0.2.1'
zone_a=IPv6Address('fe80::1%eth0');zone_b=IPv6Address('fe80::1%eth1')
assert zone_a!=zone_b and zone_a.packed==zone_b.packed
print(json.dumps({'variants':variants,'same_128_bits':len(set(parsed))==1,'normalized':str(parsed[0]),
 'leftmost_tie':tie,'single_zero_retained':single,'rejected':bad,'url_host':url.hostname,'url_port':url.port,
 'unbracketed_url_port_rejected':bad_url,'mapped_python_text':mapped.compressed,'mapped_ipv4':str(mapped.ipv4_mapped),
 'zone_scope_ids':[zone_a.scope_id,zone_b.scope_id],'same_bits_different_scopes':True}))
"""
name='xbcatm-socket-'+uuid.uuid4().hex[:10]
try:
 result=subprocess.run(['docker','run','--rm','-i','--name',name,'--network','none',
 '--memory','128m','--pids-limit','64',image,'python3','-'],input=program,
 capture_output=True,text=True,timeout=60)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),ensure_ascii=False,sort_keys=True))
finally:
 subprocess.run(['docker','rm','-f',name],capture_output=True,text=True,timeout=10)

Three strings represent the same address, including 2001:0DB8:0000:0000:0000:ff00:0042:8329 and 2001:db8::ff00:42:8329. Their string set contains three entries, while their parsed-address set contains one. The normalized output is 2001:db8::ff00:42:8329. This directly exposes why textual spelling is an unreliable key when the application’s identity is meant to be the underlying address.

For 2001:0:0:1:0:0:1:1, the first of two equal zero runs is shortened, producing 2001::1:0:0:1:1. The valid spelling 2001:db8::1:2:3:4:5 contains only one omitted zero group, so its normalized output restores that group as 2001:db8:0:1:2:3:4:5. The exercise explicitly verifies the single-zero rule rather than recommending compression wherever it makes a string shorter.

Inputs with two :: markers, too many groups, or nonhexadecimal digits fail address parsing. A string ending in /64 also fails this address-only constructor because it describes a different kind of input. Select an interface or network parser when a prefix is part of the contract; do not silently discard the suffix merely to make an address parser accept it.

The bracketed URL https://[2001:db8::1]:8443/health separates hostname and port correctly. Accessing the port of the intentionally unbracketed counterpart raises ValueError. The fixture also recognizes the IPv4 component of ::ffff:192.0.2.1; this pinned runtime prints that mapped address in mixed notation. Those results describe the tested parser, not a claim that every runtime formats every special address identically.

Two scoped values, fe80::1%eth0 and fe80::1%eth1, have identical packed address bytes but compare differently with their scopes retained. The zone is additional context rather than part of the 128-bit address. No such interfaces are contacted here. The test checks that the representation can preserve the distinction, not that either scoped destination is configured or reachable.

Choose the comparison contract before storing text

Decide whether a field stores an address, network prefix, interface assignment, URL, or socket endpoint. Use that type’s parser and preserve its required components. Keeping raw input separately can help explain a configuration mistake, while a normalized representation supports consistent display and indexing. Neither normalization nor parsing authenticates who controls the address or whether the requested service is safe.

Document treatment of mapped IPv4 and scope identifiers when correlating logs or applying address rules. Test equivalent valid forms, malformed inputs, and context-bearing cases against that policy. The goal is a stable interpretation of the intended endpoint, not merely the shortest possible string. Address formatting becomes dependable when the binary value and its surrounding connection context remain distinct and explicit.

Sources

RFC 4291 IPv6 addressing, valid IPv6 syntax, compressed forms and embedded IPv4, checked 2026-09-12.
RFC 5952 IPv6 text representation, recommended zero compression, tie handling and lowercase, checked 2026-09-12.
RFC 3986 URI syntax, bracketed IP literals and separation from a port, checked 2026-09-12.
Python 3.11 ipaddress, parsed address equality, mapped addresses and scope IDs, checked 2026-09-12.

DNS needs a working TCP path when a UDP answer is truncated

Thu, 19 Nov 2026 00:00:00 +0000

DNS supports both UDP and TCP. A server sets the TC flag when a response is truncated for the available message space. A client receiving that flag must handle the incomplete result, commonly by retrying over TCP. This is about the DNS response limit in use, not proof that the physical network cannot carry any larger UDP datagram.

When it breaks

A network that permits UDP port 53 but blocks TCP port 53 can pass small lookups and fail larger ones. DNSSEC data and larger record sets make the difference visible, but TCP is not reserved for those cases. General-purpose DNS implementations need TCP support, and clients may choose it directly. A successful small UDP query therefore does not establish that the DNS service’s complete transport path works.

A truncated answer can still contain records. Ignoring TC and using those records as the complete set can omit valid destinations or other required data. NOERROR does not cancel the truncation flag. Capture the flags and complete question alongside answer counts, especially when an application appears to see fewer records than a direct authoritative query through a different transport.

TCP also does not remove all DNS message-size limits. Each DNS message on a TCP stream is preceded by a two-byte length, which limits that message to 65,535 bytes. A zone transfer can use multiple DNS messages. Stream reads may split either the length prefix or the message, so code must read the required number of bytes rather than assuming one recv call returns a whole answer.

Check it quickly

Save this as check-dns-tcp.py and run python3 check-dns-tcp.py with Docker. The pinned BIND 9.18.49 environment serves a four-record TXT set inside an isolated container. The initial query explicitly disables EDNS to exercise the traditional 512-byte UDP limit. Later stages use a real TCP connection and a deliberately UDP-only relay:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends bind9=1:9.18.49-1~deb12u2 bind9-dnsutils=1:9.18.49-1~deb12u2 dnsmasq-base=2.90-4~deb12u2 python3-dnspython=2.3.0-1 && rm -rf /var/lib/apt/lists/*\nCMD ["/usr/bin/python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,re,signal,subprocess,time
import dns.exception,dns.message,dns.query,dns.rcode,dns.rdatatype

def run(*args,check=True):
 result=subprocess.run(args,capture_output=True,text=True,timeout=15)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def query(name,kind='A',port=1053):
 return dns.query.udp(dns.message.make_query(name,kind),'127.0.0.1',port=port,timeout=1)
def values(response,kind='A'):
 wanted=dns.rdatatype.from_text(kind)
 return [item.to_text() for rrset in response.answer if rrset.rdtype==wanted for item in rrset]
def show(response):
 def section(records):return [{'name':str(rr.name),'type':dns.rdatatype.to_text(rr.rdtype),'ttl':rr.ttl,'data':[item.to_text() for item in rr]} for rr in records]
 return {'rcode':dns.rcode.to_text(response.rcode()),'answer':section(response.answer),'authority':section(response.authority)}
def wait(predicate,timeout=15):
 deadline=time.monotonic()+timeout
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except (OSError,dns.exception.Timeout):pass
 time.sleep(.05)
 raise AssertionError('DNS fixture condition did not become true')
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=5)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=5)

@contextmanager
def lab(records,soa_ttl=8,minimum=4,extra=''):
 with TemporaryDirectory(prefix='xbcatm-dns-') as directory,ExitStack() as stack:
 root=Path(directory);zone=root/'zone';conf=root/'named.conf';log=root/'named.log';serial=1
 def save(body):
 zone.write_text(f'$ORIGIN lab.test.\n$TTL 30\n@ {soa_ttl} IN SOA ns.lab.test. hostmaster.lab.test. ({serial} 60 60 3600 {minimum})\n@ IN NS ns.lab.test.\nns IN A 127.0.0.1\n'+body+'\n')
 run('named-checkzone','lab.test',str(zone))
 save(records)
 conf.write_text(f'''options {{ directory "{root}"; listen-on port 1053 {{127.0.0.1;}}; listen-on-v6 {{none;}};
recursion no; dnssec-validation no; notify no; querylog yes; max-cache-size 8m; pid-file "{root}/named.pid"; session-keyfile none;
{extra} }};
controls {{}};
zone "lab.test" {{ type primary; file "{zone}"; }};
''')
 run('named-checkconf',str(conf))
 output=stack.enter_context(log.open('w'))
 authority=subprocess.Popen(['named','-g','-c',str(conf),'-n','1','-U','1'],stdout=output,stderr=subprocess.STDOUT)
 stack.callback(stop,authority)
 def ready():
 assert authority.poll() is None,log.read_text()
 response=query('lab.test.','SOA');return response.rcode()==0 and response.answer[0][0].serial==serial
 wait(ready)
 def update(body):
 nonlocal serial
 serial+=1;save(body);authority.send_signal(signal.SIGHUP);wait(ready)
 def cache(port=1054,upstream=1053,size=150,options=()):
 logfile=root/('cache-'+str(port)+'.log');stream=stack.enter_context(logfile.open('w'))
 command=['dnsmasq','--no-daemon','--conf-file=/dev/null','--no-resolv','--no-hosts','--bind-interfaces',
 '--listen-address=127.0.0.1','--port='+str(port),'--server=127.0.0.1#'+str(upstream),
 '--cache-size='+str(size),'--log-queries','--log-facility=-',*options]
 process=subprocess.Popen(command,stdout=stream,stderr=subprocess.STDOUT);stack.callback(stop,process)
 def cache_ready():
 assert process.poll() is None,logfile.read_text()
 return query('lab.test.','SOA',port).rcode()==0
 wait(cache_ready);return process,logfile
 def queries(name,kind='A'):
 return sum(bool(re.search(r'query: '+re.escape(name.rstrip('.'))+r' IN '+kind+r'\b',line)) for line in log.read_text().splitlines())
 yield root,update,cache,queries
import dns.flags,socket,struct,threading
texts=[f'{i}-'+('x'*200) for i in range(4)]
records='small 30 IN A 127.0.0.2\n'+'\n'.join('large 30 IN TXT "'+s+'"' for s in texts)
def exact(stream,size):
 data=b''
 while len(data)<size:
 part=stream.recv(size-len(data));assert part,'unexpected TCP EOF';data+=part
 return data
def description(response):return {'wire_bytes':len(response.to_wire()),'TC':bool(response.flags&dns.flags.TC),'answer_records':sum(len(rr) for rr in response.answer)}
with lab(records) as (_,update,cache,queries):
 q=dns.message.make_query('large.lab.test.','TXT',use_edns=False)
 udp=dns.query.udp(q,'127.0.0.1',port=1053,timeout=2)
 assert udp.flags&dns.flags.TC and len(udp.to_wire())<=512
 tcp=dns.query.tcp(q,'127.0.0.1',port=1053,timeout=2)
 assert not tcp.flags&dns.flags.TC and set(values(tcp,'TXT'))=={'"'+s+'"' for s in texts}
 assert len(tcp.to_wire())>512
 with socket.create_connection(('127.0.0.1',1053),timeout=2) as stream:
 lengths=[]
 for name,kind in [('large.lab.test.','TXT'),('small.lab.test.','A')]:
 request=dns.message.make_query(name,kind);wire=request.to_wire();stream.sendall(struct.pack('!H',len(wire))+wire)
 size=struct.unpack('!H',exact(stream,2))[0];reply=dns.message.from_wire(exact(stream,size));assert reply.id==request.id and reply.rcode()==0
 assert values(reply,kind);lengths.append(size)
 with socket.socket() as reservation,socket.socket(socket.AF_INET,socket.SOCK_DGRAM) as relay:
 reservation.bind(('127.0.0.1',0));port=reservation.getsockname()[1];relay.bind(('127.0.0.1',port));relay.settimeout(3)
 errors=[]
 def one_datagram():
 try:
 data,peer=relay.recvfrom(4096);answer=dns.query.udp(dns.message.from_wire(data),'127.0.0.1',port=1053,timeout=2);relay.sendto(answer.to_wire(),peer)
 except Exception as error:errors.append(type(error).__name__)
 thread=threading.Thread(target=one_datagram);thread.start()
 limited=dns.query.udp(q,'127.0.0.1',port=port,timeout=2);assert limited.flags&dns.flags.TC
 thread.join(timeout=4);assert not thread.is_alive() and not errors
 try:dns.query.tcp(q,'127.0.0.1',port=port,timeout=1);raise AssertionError('UDP-only endpoint unexpectedly accepted TCP')
 except OSError as error:failure=type(error).__name__
 print(json.dumps({'UDP':description(udp),'TCP':description(tcp),'two_messages_one_connection_lengths':lengths,
 'UDP_only_path':{'TC_received':True,'TCP_error':failure}},sort_keys=True))
"""
name='xbcatm-dns-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 built=docker('build','-t',name,str(root));assert built.returncode==0,built.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--memory','384m','--pids-limit','128',name,
 '/usr/bin/python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The recorded UDP response is 462 bytes, has TC set, and contains only two of the four TXT records. The TCP response is 892 bytes, has TC clear, and contains all four expected values. The script compares the complete record contents, rather than accepting a larger byte count as proof that the missing data was recovered correctly.

The program next opens one TCP connection and sends two DNS queries through it, first for the large TXT answer and then for a small A answer. It reads each two-byte length prefix and the corresponding message body, obtaining message lengths 892 and 48 in this run. It also checks response IDs. The same stream serves both exchanges, demonstrating that a fresh handshake is not required for every query.

A separate endpoint forwards one UDP question to the real authority and returns its truncated response. On that same endpoint number, a reserved but non-listening TCP socket prevents a TCP listener from existing. The UDP query succeeds with TC, while the TCP attempt fails with ConnectionRefusedError. This reproduces the transport mismatch without changing firewall rules or relying on an unrelated public resolver.

Diagnose truncation and connection failure separately

Check the response flags, advertised EDNS capacity if present, TCP connection result, and final DNS contents. A TC response followed by a refused connection has a different cause from an unanswered UDP packet or an authoritative NXDOMAIN. The relay is intentionally limited to one exchange; it is a controlled failure fixture, not a general DNS proxy implementation intended for deployment.

Permit the required TCP path through the client, resolver, server listener, and intervening access rules. Keep connection deadlines and resource limits appropriate to the resolver workload. Reuse can reduce repeated setup work, but idle connections, connection limits, and server behavior still matter. Do not convert the extra handshake observed for one new connection into a fixed latency surcharge for every TCP lookup.

EDNS can allow a larger UDP response and avoid truncation in many cases, but it does not eliminate the need for TCP. Larger datagrams can encounter path-MTU and fragmentation problems before the client ever receives a useful TC signal. Test the configured UDP size alongside a complete TCP exchange, and avoid treating an unusually large advertised payload as a substitute for a working second transport.

Finally, TCP provides a reliable byte stream, not DNS confidentiality or authenticated record data. Plain DNS over TCP remains plaintext. TLS-based DNS transport and DNSSEC address separate security properties. Keep those claims separate from this experiment, whose evidence establishes truncation handling, complete record recovery, stream framing, connection reuse, and a concrete failure when only UDP is available.

Sources

RFC 7766 DNS over TCP, TCP support, connection reuse, and DNS transport requirements, checked 2026-09-12.
RFC 1035 domain names, TC, traditional UDP size, and two-byte TCP framing, checked 2026-09-12.
RFC 5936 AXFR, zone transfers spanning multiple DNS messages, checked 2026-09-12.

NAT shares an address by keeping distinct connection mappings

Wed, 18 Nov 2026 00:00:00 +0000

Network Address Translation changes addressing information as packets cross a boundary. The familiar many-clients-to-one-address arrangement uses network address and port translation: the router keeps mappings that distinguish simultaneous flows. A remote TCP server may see one source IP while seeing different source ports. That shared address identifies a translation boundary, not necessarily one computer, account, or person.

When it breaks

Translation does not replace routing. A client still needs a route toward the gateway, the gateway must forward packets, and the translated address must support a return path. Conversely, private addressing does not inherently require NAT inside a network that routes those prefixes. A missing return route and a missing translation rule can produce similar timeouts but require different configuration decisions.

Stateful NAT also makes existing and new connections behave differently during rule changes. nftables consults a NAT chain to establish a flow’s mapping; later packets use the stored binding. Removing an SNAT rule therefore does not necessarily revoke established translated connections. This depends on connection tracking remaining active and the binding remaining valid, so treating a ruleset edit as an unconditional session purge is unsafe.

Port mappings and connection-tracking capacity are finite resources. Exhaustion, expiry, or a failover device lacking the relevant state can interrupt communication even when the public address remains unchanged. A usable translated source port is considered within the relevant protocol and connection tuple; avoid estimating capacity as one universal port count for every destination and workload. NAT is also separate from the firewall policy deciding permitted traffic.

Check it quickly

Save this as check-nat.py and run python3 check-nat.py with Docker. It builds a disposable environment with Python 3.11.16 and nftables 1.0.6. NET_ADMIN and SYS_ADMIN enable four nested network namespaces and their temporary proc mount. The example was executed on Linux 6.12.76; Docker shares its host kernel. No host interface or existing firewall is changed:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,'child exited without a response'
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found

with lab(['a','b','router','target']) as (root,stack,setting):
 wire('a','a0','router','ra',['10.203.1.2/30'],['10.203.1.1/30'])
 wire('b','b0','router','rb',['10.203.2.2/30'],['10.203.2.1/30'])
 wire('router','wan','target','t0',['192.0.2.1/30'],['192.0.2.2/30'])
 setting('router','net.ipv4.ip_forward',1)
 ip('a','route','add','default','via','10.203.1.1');ip('b','route','add','default','via','10.203.2.1')
 server_code=r'''
import json,socketserver
class Handler(socketserver.StreamRequestHandler):
 def handle(self):
 for raw in self.rfile:
 data=json.loads(raw);data['peer']=self.client_address
 self.wfile.write(json.dumps(data).encode()+b'\n');self.wfile.flush()
class Server(socketserver.ThreadingTCPServer):allow_reuse_address=True;daemon_threads=True
server=Server(('192.0.2.2',18080),Handler);print('ready',flush=True);server.serve_forever()
'''
 daemon=spawn(stack,['ip','netns','exec','target','python3','-u','-c',server_code]);assert line(daemon).strip()=='ready'
 client_code=r'''
import json,socket,sys
label,port=sys.argv[1],int(sys.argv[2])
try:
 with socket.socket() as stream:
 stream.settimeout(.7);stream.bind(('0.0.0.0',port));stream.connect(('192.0.2.2',18080))
 with stream.makefile('rb') as incoming:
 def exchange(phase):
 stream.sendall(json.dumps({'label':label,'phase':phase}).encode()+b'\n')
 print(json.dumps({'ok':True,'reply':json.loads(incoming.readline())}),flush=True)
 exchange('initial')
 for command in sys.stdin:exchange(command.strip())
except OSError as error:print(json.dumps({'ok':False,'type':type(error).__name__,'errno':error.errno}),flush=True)
'''
 def once(name,port):return json.loads(ns(name,'python3','-u','-c',client_code,name,str(port),input='').stdout)
 before=once('a',39000);assert not before['ok'] and before['type']=='TimeoutError'
 nft('router','''table ip translation {
 chain postrouting { type nat hook postrouting priority srcnat; policy accept;
 ip saddr 10.203.0.0/16 oifname "wan" counter snat to 192.0.2.1 comment "new_mapping"
 }
}
table inet tracking {
 chain forward { type filter hook forward priority filter; policy accept;
 ct state established,related counter accept comment "established"
 }
}
''')
 clients={};initial={}
 for name in ['a','b']:
 clients[name]=spawn(stack,['ip','netns','exec',name,'python3','-u','-c',client_code,name,'40000'])
 initial[name]=json.loads(line(clients[name]));assert initial[name]['ok'] and initial[name]['reply']['label']==name
 peers=[value['reply']['peer'] for value in initial.values()]
 assert all(peer[0]=='192.0.2.1' for peer in peers) and peers[0][1]!=peers[1][1]
 mapped=counters('router');assert mapped['new_mapping']==2
 nft('router','flush chain ip translation postrouting')
 retained={}
 for name,process in clients.items():
 process.stdin.write('after-rule-removal\n');process.stdin.flush();retained[name]=json.loads(line(process))
 assert retained[name]['ok'] and retained[name]['reply']['peer']==initial[name]['reply']['peer'],(name,retained[name],initial[name])
 new_flow=once('a',40001);assert not new_flow['ok'] and new_flow['type']=='TimeoutError'
 ip('target','route','add','10.203.1.0/30','via','192.0.2.1')
 routed_without_nat=once('a',40003);assert routed_without_nat['ok'] and routed_without_nat['reply']['peer'][0]=='10.203.1.2'
 nft('router','delete table inet tracking')
 clients['a'].stdin.write('after-tracking-removal\n');clients['a'].stdin.flush()
 tracking_removed=json.loads(line(clients['a']));assert not tracking_removed['ok'],tracking_removed
 print(json.dumps({'without_nat_or_return_route':before,'both_clients_bind_port':40000,'translated_connections':initial,
 'nat_first_packet_counter':mapped,'existing_after_rule_removal':retained,'new_after_rule_removal':new_flow,
 'explicit_return_route_without_nat':routed_without_nat,'last_tracking_reference_removed':tracking_removed}))
"""
name='xbcatm-netpath-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

Clients a and b occupy separate private subnets behind a router. The target has only its connected route to the router’s outward interface, 192.0.2.1. These documentation addresses remain inside the laboratory. Without translation or an explicit return route to the client subnet, the first TCP exchange times out even though the router has forwarding enabled.

The script installs an SNAT rule and an established-state tracking rule. Both clients bind source port 40000 and contact the same target address and port. The target reports source address 192.0.2.1 for both, with different translated ports. The script checks that replies retain the correct client label; merely observing two accepted sockets would not establish that responses were delivered to the intended clients.

The NAT rule counter records two newly mapped connections. The script then flushes that rule while leaving the NAT base chain and separate tracking rule in place. Both existing sockets exchange another message using their original mappings, while a new connection times out. This directly demonstrates why checking only a previously opened connection can miss a broken configuration for new clients.

Next, adding a return route on the target allows another new connection without SNAT. Its reported peer is the actual private address, 10.203.1.2. Finally, removing the last tracking expression breaks an existing translated socket; this run reported ConnectionResetError. The matching kernel source unregisters tracking hooks when the final namespace reference is released. That is a different operation from deleting one NAT rule, and does not justify claiming that every stored entry was erased.

Diagnose the mapping and the return path together

Record original and translated source addresses and ports, destination tuple, protocol, and timestamps when correlating traffic across a gateway. Server logs containing only the shared IP cannot distinguish the clients in this example. Authentication and application identities remain necessary where attribution matters; a translation log provides network correlation rather than proof of a user’s identity.

For a change or incident, test both an established connection and a newly opened one. Check routing in both directions and inspect the applicable translation and filtering rules. Preserve state intentionally during maintenance when continuity is required. If sessions must be terminated, use an explicit, scoped mechanism whose behavior is verified for the deployed system; do not infer termination from a removed rule alone.

Sources

RFC 3022 Traditional NAT, address and port translation, mappings and session limitations, checked 2026-09-12.
nftables NAT documentation, first-packet rule lookup and retained connection mappings, checked 2026-09-12.
Linux 6.12.76 connection-tracking hooks, unregistering protocol hooks when the last namespace reference is released, checked 2026-09-12.
Linux 6.12.76 nftables ct expressions, connection-tracking references held by state expressions, checked 2026-09-12.

Loopback returns to the caller’s network namespace

Tue, 17 Nov 2026 00:00:00 +0000

Loopback addresses let a process communicate through its local IP stack without sending the exchange over a physical network link. IPv4 reserves 127.0.0.0/8 for loopback; 127.0.0.1 is the familiar member. IPv6 uses ::1. On Linux, the relevant local networking context is a network namespace. Two processes on the same physical machine can therefore have different answers to what counts as their own loopback service.

When it breaks

A client inside a container often fails when configured with 127.0.0.1 for a database running on the host or in another container. The client has asked for its own namespace’s listener. The spelling is correct but the topology is wrong. A published host port, a container interface address, and another process’s loopback are different destinations that must be selected according to the deployment.

Loopback is not a guarantee that every probe succeeds. The target port still needs a listener, the intended address family must be available, and local networking policy can affect delivery. A successful IPv4 request does not prove that a service also listens on ::1. Conversely, a refused connection can simply mean there is no listener at that address and port in the caller’s namespace.

Local delivery also does not eliminate the IP stack. Loopback traffic still uses sockets, protocol processing, and local routing decisions; it merely avoids a physical trip through the external network. Timing a request to loopback cannot validate switch configuration, remote routing, an internet firewall, or the public service name’s certificate. Keep those paths separate when interpreting a health check.

Check it quickly

Save this as check-loopback.py and run python3 check-loopback.py with Docker. It creates two disposable containers using the same pinned Python 3.11.16 image, each with Docker networking disabled and no host port publication. Container A starts IPv4 and IPv6 listeners on port 19091. Container B initially runs without a service on that port:

import json,subprocess,time,uuid

image='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
names=['xbcatm-loop-'+uuid.uuid4().hex[:10] for _ in range(2)]
server=r'''
import json,os,selectors,socket,sys
selector=selectors.DefaultSelector()
for family,address in [(socket.AF_INET,'127.0.0.1'),(socket.AF_INET6,'::1')]:
 listener=socket.socket(family,socket.SOCK_STREAM)
 if family==socket.AF_INET6:listener.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
 listener.bind((address,19091));listener.listen(8);selector.register(listener,selectors.EVENT_READ)
print('ready',flush=True)
while True:
 for key,events in selector.select():
 connection,peer=key.fileobj.accept()
 with connection:
 connection.settimeout(1)
 connection.sendall(json.dumps({'service':sys.argv[1],'netns':os.readlink('/proc/self/ns/net')}).encode())
'''
probe=r'''
import json,os,socket
answers={}
for address in ['127.0.0.1','::1']:
 try:
 with socket.create_connection((address,19091),timeout=1) as stream:
 pieces=[]
 while True:
 piece=stream.recv(1024)
 if not piece:break
 pieces.append(piece)
 answers[address]=json.loads(b''.join(pieces))
 except ConnectionRefusedError:answers[address]={'error':'ECONNREFUSED'}
print(json.dumps({'netns':os.readlink('/proc/self/ns/net'),'interfaces':socket.if_nameindex(),'answers':answers}))
'''
def docker(*args):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=30)
 assert result.returncode==0,result.stderr
 return result.stdout
def inspect(name):return json.loads(docker('exec',name,'python3','-c',probe))
def wait_for(name,identity):
 deadline=time.monotonic()+5
 while time.monotonic()<deadline:
 result=inspect(name)
 if all(answer.get('service')==identity for answer in result['answers'].values()):return result
 time.sleep(.05)
 raise AssertionError('listener did not become ready')
try:
 for name,command in [(names[0],['python3','-u','-c',server,'A']),
 (names[1],['python3','-c','import time;time.sleep(120)'])]:
 docker('run','-d','--rm','--name',name,'--network','none','--memory','128m','--pids-limit','64',image,*command)
 a=wait_for(names[0],'A');b_before=inspect(names[1])
 assert a['netns']!=b_before['netns']
 assert all(v=={'error':'ECONNREFUSED'} for v in b_before['answers'].values())
 docker('exec','-d',names[1],'python3','-u','-c',server,'B')
 b_after=wait_for(names[1],'B');a_after=inspect(names[0])
 assert all(v.get('service')=='A' for v in a_after['answers'].values())
 for result in [a,b_after,a_after]:
 assert all(answer['netns']==result['netns'] for answer in result['answers'].values())
 assert all(any(interface[1]=='lo' for interface in result['interfaces']) for result in [a,b_before,b_after])
 print(json.dumps({'A':a,'B_before_listener':b_before,'B_after_listener':b_after,'A_still_A':a_after},sort_keys=True))
finally:
 for name in names:subprocess.run(['docker','rm','-f',name],capture_output=True,text=True,timeout=10)

A probe executed inside A reaches both 127.0.0.1 and ::1 and reads the service identity A. Each response also carries the server’s network namespace identifier. The probe’s own identifier matches it. This links the successful application response to the networking context in which the client ran, rather than assuming that all local-looking addresses on the host share a listener.

The same probe inside B receives ECONNREFUSED for both addresses, even while A continues serving. B’s namespace identifier differs from A’s. The program uses literal addresses, so DNS search rules and a host’s interpretation of localhost cannot explain the difference. The failure comes from probing B’s local endpoints, where no matching service has yet been started.

The script next starts a second server inside B using the identical address strings and port. B now receives identity B through both families, while a new probe inside A still receives A. Both servers coexist without a port conflict. The test directly demonstrates namespace-specific loopback and socket port spaces, not merely two configuration files containing different container names.

The interface inventory is included for context, but the result does not assume that lo is the only interface name. This kernel also creates inactive tunnel devices in fresh namespaces. Their presence does not change which loopback listener answered. The assertions depend on distinct namespace identities and actual IPv4/IPv6 exchanges, rather than a fragile expected count of interface names.

Put probes in the context they claim to test

Run an application-local probe in the application’s namespace when the question is whether its listener responds. Run a separate probe from a representative client network when the question is whether users can reach the public endpoint. An internal success and an external failure can both be correct observations because the requests traverse different routing, forwarding, and authentication paths.

Processes sharing a network namespace also share loopback, even if they otherwise have separate process or filesystem environments. Host networking and intentionally shared container networking change the boundary. Before replacing localhost in a connection string, identify where the client executes and which namespace owns the server. Avoid treating container names or machine ownership as substitutes for that information.

A loopback listener can also be exposed indirectly by a proxy or tunnel that forwards requests to it. Local address scope is therefore different from user authorization. Authenticate the service as its contract requires, including requests originating from another local process. Use loopback to express the intended local transport path, and verify the complete forwarding chain when remote access is deliberately provided.

Sources

RFC 1122 Internet host requirements, the reserved IPv4 loopback address block, checked 2026-09-12.
RFC 4291 IPv6 addressing, the IPv6 loopback address and its scope, checked 2026-09-12.
Linux network_namespaces(7), isolation of interfaces, routes and socket port numbers, checked 2026-09-12.
Linux ip(7), local IPv4 delivery and socket addressing, checked 2026-09-12.

IPv6 resolves neighbors with ICMPv6 before the TCP connection can work

Mon, 16 Nov 2026 00:00:00 +0000

An IPv6 host on an Ethernet link needs a link-layer destination before it can send a frame to a neighbor. Neighbor Discovery performs that address-resolution work using ICMPv6 Neighbor Solicitation and Neighbor Advertisement messages. It also covers functions beyond IPv4 ARP, including router discovery and neighbor reachability detection. Calling it merely “ARP with longer addresses” hides dependencies that matter when configuring an IPv6 network.

When it breaks

A valid IPv6 address and route do not prove that the next hop’s Ethernet address can be resolved. If the required solicitation or advertisement is filtered, a TCP service can be listening correctly while clients fail before exchanging application data. Opening only the application’s TCP port does not repair this lower-layer dependency. Inspect the neighbor table and relevant ICMPv6 traffic alongside the service listener.

Cached state can conceal the error. A host with a usable neighbor entry may continue communicating after a damaging rule is installed, while a fresh client or an expired entry exposes the failure. Conversely, a STALE entry is not automatically broken: Neighbor Discovery can use existing link-layer information and subsequently check reachability. Interpret the state machine rather than requiring every healthy entry to remain REACHABLE forever.

Another common confusion is resolving the remote application’s address on the local Ethernet segment. For an off-link destination, the host normally resolves its selected next-hop router. Confirm routing before deciding which neighbor should appear in the table. A capture on the wrong interface, or a cached entry requiring no new solicitation, can also explain an empty trace without proving that discovery is absent.

Check it quickly

Save this as check-neighbor-discovery.py and run python3 check-neighbor-discovery.py with Docker. The disposable Linux environment uses Python 3.11.16, iproute2 6.1.0, nftables 1.0.6, and tcpdump 4.99.3. NET_ADMIN and SYS_ADMIN support its nested namespaces and temporary proc mount. Two endpoints share a newly created veth link; no host neighbor table or firewall rule is changed:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,'child exited without a response'
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found

def server(stack,name,address,port=18080,label='server'):
 code=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import json,socket,sys
address,port,label=sys.argv[1],int(sys.argv[2]),sys.argv[3]
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 body=json.dumps({'server':label,'peer':self.client_address,'path':self.path}).encode()
 self.send_response(200);self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
 def log_message(self,*args):pass
class Server(ThreadingHTTPServer):
 address_family=socket.AF_INET6 if ':' in address else socket.AF_INET
 def server_bind(self):
 if self.address_family==socket.AF_INET6:self.socket.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
 super().server_bind()
service=Server((address,port),Handler);print('ready',flush=True);service.serve_forever()
'''
 process=spawn(stack,['ip','netns','exec',name,'python3','-u','-c',code,address,str(port),label])
 assert line(process).strip()=='ready';return process
def request(name,address,port=18080):
 code=r'''
import http.client,json,sys
client=http.client.HTTPConnection(sys.argv[1],int(sys.argv[2]),timeout=.7)
try:
 client.request('GET','/lab-visible-message');response=client.getresponse();body=json.loads(response.read())
 assert response.status==200;print(json.dumps({'ok':True,'status':response.status,'body':body}))
except OSError as error:print(json.dumps({'ok':False,'type':type(error).__name__,'errno':error.errno}))
finally:client.close()
'''
 return json.loads(ns(name,'python3','-c',code,address,str(port)).stdout)

def capture(stack,root,name,interface,expression,label):
 path=root/(label+'.pcap')
 process=subprocess.Popen(['ip','netns','exec',name,'tcpdump','--immediate-mode','-Z','root','-U','-n','-i',interface,'-w',str(path),expression],
 stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process)
 deadline=time.monotonic()+3
 while time.monotonic()<deadline:
 assert process.poll() is None,'tcpdump exited before listening'
 if select.select([process.stderr],[],[],.1)[0] and 'listening on' in process.stderr.readline():return process,path
 raise AssertionError('tcpdump did not become ready')
def captured(process,path,ascii=False):
 process.send_signal(signal.SIGINT);_,diagnostic=process.communicate(timeout=3)
 assert process.returncode==0,diagnostic
 result=run('tcpdump','-nn','-vv',*(['-A'] if ascii else []),'-r',str(path))
 return result.stdout

with lab(['client','target']) as (root,stack,setting):
 wire('client','c0','target','t0',['192.0.2.1/30','2001:db8:1::1/64'],['192.0.2.2/30','2001:db8:1::2/64'])
 server(stack,'target','2001:db8:1::2')
 ip('client','-4','neigh','flush','dev','c0');ip('target','-4','neigh','flush','dev','t0')
 arp_process,arp_file=capture(stack,root,'client','c0','arp','arp')
 ns('client','ping','-4','-n','-c','1','-W','1','192.0.2.2')
 arp=captured(arp_process,arp_file);assert 'ARP,' in arp and 'Request who-has' in arp and 'Reply' in arp,arp
 ip('client','-6','neigh','flush','dev','c0');ip('target','-6','neigh','flush','dev','t0')
 nd_process,nd_file=capture(stack,root,'client','c0','icmp6 or arp','nd')
 ns('client','ping','-6','-n','-c','1','-W','1','2001:db8:1::2')
 nd=captured(nd_process,nd_file)
 assert 'neighbor solicitation' in nd and 'neighbor advertisement' in nd and 'ARP,' not in nd,nd
 mac=json.loads(ip('target','-j','link','show','t0').stdout)[0]['address']
 neighbors=json.loads(ip('client','-6','-j','neigh','show','2001:db8:1::2').stdout)
 assert len(neighbors)==1 and neighbors[0]['lladdr']==mac
 healthy=request('client','2001:db8:1::2');assert healthy['ok']
 setting('client','net.ipv6.neigh.c0.retrans_time_ms',200);setting('client','net.ipv6.neigh.c0.mcast_solicit',3)
 nft('client','''table inet block_nd {
 chain input { type filter hook input priority 0; policy accept;
 icmpv6 type nd-neighbor-advert counter drop comment "blocked_advertisements"
 }
}
''')
 ip('client','-6','neigh','flush','dev','c0')
 failed=ns('client','ping','-6','-n','-c','1','-W','1','2001:db8:1::2',check=False);assert failed.returncode!=0
 broken=request('client','2001:db8:1::2');assert not broken['ok']
 blocked=counters('client');assert blocked['blocked_advertisements']>0
 failed_neighbors=json.loads(ip('client','-6','-j','neigh','show','2001:db8:1::2').stdout)
 assert failed_neighbors and all('lladdr' not in entry for entry in failed_neighbors)
 nft('client','delete table inet block_nd');ip('client','-6','neigh','flush','dev','c0')
 recovered=request('client','2001:db8:1::2');assert recovered['ok']
 print(json.dumps({'ipv4_arp_capture':arp,'ipv6_capture':nd,'resolved_neighbor':neighbors,'target_mac':mac,
 'healthy_http':healthy,'blocked_ping_exit':failed.returncode,'blocked_http':broken,'blocked_counters':blocked,
 'unresolved_neighbor':failed_neighbors,'recovered_http':recovered}))
"""
name='xbcatm-netpath-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The script assigns IPv4 and IPv6 documentation addresses to both endpoints. It first captures a real IPv4 ARP request and reply as a positive control for the capture setup. It then clears the client’s IPv6 neighbor cache and sends an IPv6 echo request. The trace contains Neighbor Solicitation, Neighbor Advertisement, and echo traffic, with no ARP frames in this IPv6 exchange.

The solicitation for 2001:db8:1::2 is sent to solicited-node multicast address ff02::1:ff00:2. The reply provides the target’s link-layer address, which the script compares with the target interface’s actual MAC address and the client’s neighbor entry. An HTTP request then returns the expected server identity. This joins the control exchange, learned information, and application outcome instead of relying on configuration text alone.

Next, the client drops incoming neighbor advertisements. Its cache is cleared so an older mapping cannot hide the change. Discovery retransmission settings are shortened only inside the client namespace to keep the test bounded. Ping fails; an HTTP attempt reports Linux EHOSTUNREACH, errno 113; and the neighbor entry becomes FAILED without a resolved link-layer address. The filter counter confirms that advertisements reached the drop rule.

Removing that rule and clearing the failed entry restores HTTP success. The target service and address stay unchanged throughout the comparison. This isolates address-resolution failure from a stopped application or a missing TCP allow rule. The exact timing and packet count depend on the configured discovery retries, so the example checks the failure state and recovery rather than prescribing one universal timeout.

Keep discovery, addressing, and authentication separate

The fixture uses static IPv6 addresses with duplicate-address detection disabled for setup. It demonstrates neighbor resolution, not SLAAC, Router Advertisement processing, or Duplicate Address Detection. Those functions require their own observations and policy review. Passing this small test cannot certify every ICMPv6 dependency in a production network; RFC 4890 discusses filtering requirements in broader operational contexts.

An advertisement also should not be treated as authenticated proof of ownership merely because it supplies a MAC address. The protocol’s local-link checks limit where messages can originate but do not make every node on that link trustworthy. When debugging, record interface, selected next hop, neighbor state, filters, and the corresponding application result. That evidence distinguishes an unresolved neighbor from an unrelated service failure.

Sources

RFC 4861 Neighbor Discovery, solicitation, advertisement, address resolution and reachability states, checked 2026-09-12.
Linux ip-neighbour(8), inspecting and flushing IPv6 neighbor entries, checked 2026-09-12.
RFC 4890 ICMPv6 filtering, preserving ICMPv6 functions required by IPv6 networks, checked 2026-09-12.

Networking · XB Field Notes

A veth pair carries frames between two endpoints; routing supplies their meaning

When it breaks

Check it quickly

Diagnose the layers in order

Sources

VXLAN needs matching tunnel parameters as well as a working underlay

When it breaks

Check it quickly

Match evidence to the layer being changed

Sources

Kubernetes Services give pods a stable home

One address in front of many

Selection drives which pods respond

Traffic goes to whatever matches

The stable name is a promise to callers

When it breaks

Check it quickly

Further reading

Sources

Connection timeouts prevent silent hangs

Successful connection does not mean a useful reply

When it breaks

Check it quickly

Sources

A CNAME changes DNS resolution while the requested hostname stays the same

When it breaks

Check it quickly

Move the dependency and preserve the service contract

Sources

Multiple DNS addresses do not guarantee balanced or healthy traffic

When it breaks

Check it quickly

Choose the balancing and failure contract explicitly

Sources

NodePort publishes a service on every node's fixed port

A port every node owns

The extra hop and the client address

A side door around the ingress

When it breaks

Check it quickly

Sources

Idle timeouts recycle edge connections

Idle is defined at a particular boundary

When it breaks

Check it quickly

Sources

Default ports are conventions, not protocol detection or security

When it breaks

Check it quickly

Keep the complete endpoint in configuration

Further reading

Sources

Retries with jitter and backoff

Bound attempts before spreading them

Exponential backoff sets a ceiling

Use one end-to-end deadline

When it breaks

Check it quickly

Further reading

Sources

A bind address selects local destinations; reachability needs the whole path

When it breaks

Check it quickly

Verify the listener and the permitted caller

Further reading

Sources

SRV discovery gives clients targets and ports that they must interpret

When it breaks

Check it quickly

Match the client algorithm to the published record set

Further reading

Sources

A Service gives a pod a stable virtual address

Declared, then given an address

Selector is the live membership

Stable even as pods churn

When it breaks

Check it quickly

Further reading