VXLAN needs matching tunnel parameters as well as a working underlay

Thu, 28 Oct 2027 00:00:00 +0000

VXLAN carries an Ethernet frame inside a UDP packet so an overlay network can cross an IP underlay. Tunnel endpoints encapsulate and decapsulate frames, while a 24-bit VXLAN Network Identifier, or VNI, identifies the virtual segment. The outer IP addresses route between endpoints; the inner Ethernet and IP addresses serve the overlay. These are separate layers with separate reachability and configuration requirements.

When it breaks

Pinging a tunnel endpoint proves little about a particular overlay segment. Both endpoints may be reachable while using different VNIs, UDP destination ports, or remote-peer configuration. An underlay firewall can also admit ICMP while dropping encapsulated UDP. Diagnose the actual transport and virtual segment instead of assuming that one successful underlay ping validates all tenant traffic above it.

Encapsulation consumes the underlay’s packet budget. For an untagged inner Ethernet frame over ordinary IPv4 and UDP, an overlay IP MTU of 1450 fits a 1500-byte underlay IP MTU: inner Ethernet, VXLAN, UDP, and outer IPv4 headers account for fifty bytes. IPv6 outer headers, VLAN tags, or other encapsulations change the calculation. A successful small request alone does not validate large packets or path MTU discovery.

A VNI also is not an encryption key or authenticated tenant identity. Anyone permitted to observe the encapsulated traffic may be able to read its contents, and trust in tunnel endpoints depends on the surrounding network controls. VXLAN’s segmentation role does not establish confidentiality. Protect the underlay and use appropriate authenticated encryption where the threat model requires it.

Check it quickly

Save this as check-vxlan.py and run python3 check-vxlan.py with Docker. The disposable Linux container uses Python 3.11.16, iproute2 6.1.0, and tcpdump 4.99.3. NET_ADMIN and SYS_ADMIN enable nested network namespaces and their temporary proc mount. The two endpoints use only a new veth link, with no host route changes, host mounts, or published service ports:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,'child exited without a response'
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found

def server(stack,name,address,port=18080,label='server'):
 code=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import json,socket,sys
address,port,label=sys.argv[1],int(sys.argv[2]),sys.argv[3]
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 body=json.dumps({'server':label,'peer':self.client_address,'path':self.path}).encode()
 self.send_response(200);self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
 def log_message(self,*args):pass
class Server(ThreadingHTTPServer):
 address_family=socket.AF_INET6 if ':' in address else socket.AF_INET
 def server_bind(self):
 if self.address_family==socket.AF_INET6:self.socket.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
 super().server_bind()
service=Server((address,port),Handler);print('ready',flush=True);service.serve_forever()
'''
 process=spawn(stack,['ip','netns','exec',name,'python3','-u','-c',code,address,str(port),label])
 assert line(process).strip()=='ready';return process
def request(name,address,port=18080):
 code=r'''
import http.client,json,sys
client=http.client.HTTPConnection(sys.argv[1],int(sys.argv[2]),timeout=.7)
try:
 client.request('GET','/lab-visible-message');response=client.getresponse();body=json.loads(response.read())
 assert response.status==200;print(json.dumps({'ok':True,'status':response.status,'body':body}))
except OSError as error:print(json.dumps({'ok':False,'type':type(error).__name__,'errno':error.errno}))
finally:client.close()
'''
 return json.loads(ns(name,'python3','-c',code,address,str(port)).stdout)

def capture(stack,root,name,interface,expression,label):
 path=root/(label+'.pcap')
 process=subprocess.Popen(['ip','netns','exec',name,'tcpdump','--immediate-mode','-Z','root','-U','-n','-i',interface,'-w',str(path),expression],
 stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process)
 deadline=time.monotonic()+3
 while time.monotonic()<deadline:
 assert process.poll() is None,'tcpdump exited before listening'
 if select.select([process.stderr],[],[],.1)[0] and 'listening on' in process.stderr.readline():return process,path
 raise AssertionError('tcpdump did not become ready')
def captured(process,path,ascii=False):
 process.send_signal(signal.SIGINT);_,diagnostic=process.communicate(timeout=3)
 assert process.returncode==0,diagnostic
 result=run('tcpdump','-nn','-vv',*(['-A'] if ascii else []),'-r',str(path))
 return result.stdout

with lab(['left','right']) as (root,stack,setting):
 wire('left','ul','right','ur',['192.0.2.1/30'],['192.0.2.2/30'])
 for name,interface in [('left','ul'),('right','ur')]:ip(name,'link','set',interface,'mtu','1500')
 def overlay(name,vni):
 local,remote,dev,address,mac=('192.0.2.1','192.0.2.2','ul','10.42.0.1/24','02:00:00:00:42:01') if name=='left' else ('192.0.2.2','192.0.2.1','ur','10.42.0.2/24','02:00:00:00:42:02')
 ip(name,'link','add','vx0','type','vxlan','id',str(vni),'local',local,'remote',remote,'dev',dev,'dstport','4789','nolearning')
 ip(name,'link','set','vx0','address',mac,'mtu','1450')
 ip(name,'addr','add',address,'dev','vx0');ip(name,'link','set','vx0','up')
 overlay('left',42);overlay('right',42);server(stack,'right','10.42.0.2',label='overlay')
 tap,path=capture(stack,root,'left','ul','udp port 4789','vxlan')
 successful=request('left','10.42.0.2');assert successful['ok'] and successful['body']['server']=='overlay'
 packets=captured(tap,path,ascii=True)
 assert 'vni 42' in packets.lower() and 'lab-visible-message' in packets and '.4789:' in packets,packets
 details=json.loads(ip('left','-d','-j','link','show','vx0').stdout)[0]
 assert details['linkinfo']['info_data']['id']==42 and details['mtu']==1450
 ip('right','link','delete','vx0');overlay('right',43)
 underlay=ns('left','ping','-n','-c','1','-W','1','192.0.2.2');assert underlay.returncode==0
 mismatch=request('left','10.42.0.2');assert not mismatch['ok'] and mismatch['type']=='TimeoutError'
 ip('right','link','delete','vx0');overlay('right',42)
 recovered=request('left','10.42.0.2');assert recovered['ok']
 print(json.dumps({'healthy_overlay':successful,'underlay_packet_capture':packets,'vni':details['linkinfo']['info_data']['id'],
 'underlay_mtu':1500,'overlay_mtu':details['mtu'],'mismatched_vni_underlay_ping_exit':underlay.returncode,
 'mismatched_vni_overlay':mismatch,'matching_vni_restored':recovered}))
"""
name='xbcatm-netpath-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The underlay addresses are 192.0.2.1 and 192.0.2.2. Each endpoint has an explicitly configured remote peer and UDP destination port 4789. Setting the port explicitly avoids depending on historical implementation defaults. Both overlay devices initially use VNI 42, fixed MAC addresses, and addresses 10.42.0.1 and 10.42.0.2. This is a small unicast tunnel, not a complete multipoint fabric or EVPN deployment.

An HTTP request across the overlay returns the expected server identity. Simultaneously, tcpdump observes the underlay carrying UDP packets marked VNI 42, including inner ARP and TCP traffic. The capture contains the literal request path /lab-visible-message. That is direct evidence that this configuration transports the application bytes without encrypting them; the VNI scopes delivery but does not conceal the payload.

The local veth capture also reports apparently incorrect UDP and TCP checksums while the HTTP exchange succeeds. Linux can represent checksums as partial work to be completed at another processing stage, including offloaded encapsulated traffic. This capture is not evidence of wire corruption. The fixture does not disable offloads or identify every checksum-processing stage, so it does not claim a measured hardware fault or a complete checksum diagnosis.

Next, the right overlay device is recreated with VNI 43 while retaining its address and MAC. The underlay ping still succeeds, but the overlay HTTP request times out. Recreating it with VNI 42 restores HTTP success. Holding the other overlay properties fixed makes the identifier mismatch the controlled change, instead of accidentally attributing a stale-neighbor problem to the VNI.

Match evidence to the layer being changed

Inspect the endpoint’s underlay route, selected interface, configured UDP port, VNI, peer mapping, and overlay address. Capture on both the tunnel device and its transport interface when possible: each exposes a different representation of the same communication. Record where the capture was taken, particularly when interpreting packet sizes or checksum warnings that may reflect kernel offload processing.

This example deliberately reserves encapsulation space with MTUs of 1500 and 1450, but tests only a short application exchange. Validate the real path’s packet-size requirements separately before using it for larger workloads. In a production fabric, also review how remote MAC destinations are learned or distributed and how unknown traffic is handled. Those control-plane choices cannot be inferred from this two-endpoint success.

Sources

RFC 7348 VXLAN, VNI, UDP encapsulation, MTU and security considerations, checked 2026-09-12.
Linux VXLAN documentation, VXLAN devices, destination ports and tunnel offloads, checked 2026-09-12.
Linux checksum offloads, partial and encapsulated checksums at different processing stages, checked 2026-09-12.

Overlay · XB Field Notes

VXLAN needs matching tunnel parameters as well as a working underlay

When it breaks

Check it quickly

Match evidence to the layer being changed

Sources