Traceroute sends probes with increasing TTL values to discover where packets expire and which systems report that expiration. An IPv4 router forwarding a packet decrements its TTL; when it expires, an ICMP Time Exceeded response normally identifies a responding hop. A traditional UDP trace finishes when the destination reports an unused port. The output is a series of observations about probes and replies, not a guaranteed inventory of every forwarding device.

When it breaks

An asterisk means a probe did not receive an expected response within its waiting period. The router might still forward ordinary traffic while filtering, suppressing, or rate-limiting its own ICMP responses. The return path for those responses can also fail. A silent intermediate hop therefore does not, by itself, establish that application packets stop there; later replies may directly contradict that interpretation.

The probing protocol matters. Traditional high-port UDP traffic may encounter a different policy from TCP traffic to an allowed service port. TCP traceroute can make the probe resemble that permitted transport, but still does not perform the application’s complete exchange. A SYN response or a reset can reveal destination reachability without proving that HTTP, authentication, or a particular business operation works.

Per-hop timings are round trips from the observer, including response generation and the return path. Subtracting adjacent values does not reliably measure the one-way delay of the link between those routers. Load balancing, asymmetric return routes, scheduling, and ICMP handling can all change the observations. Keep the probe method and measurement conditions when comparing traces instead of treating one printed sequence as a timeless path.

Check it quickly

Save this as check-traceroute.py and run python3 check-traceroute.py with Docker. The disposable environment uses traceroute 2.1.2, nftables 1.0.6, and Python 3.11.16. NET_ADMIN and SYS_ADMIN support nested namespaces and a temporary proc mount. A client, router, and target communicate over newly created veth links; all routing and filtering changes stay within the laboratory:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
    result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
    if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
    return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
    if process.poll() is None:
        process.terminate()
        try:process.wait(timeout=2)
        except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
    assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
    value=process.stdout.readline();assert value,'child exited without a response'
    return value
def spawn(stack,command):
    process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
    stack.callback(stop,process);return process
@contextmanager
def lab(names):
    with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
        root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
        stack.callback(run,'umount',str(proc))
        for name in names:
            run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
            ip(name,'link','set','lo','up')
        def setting(name,key,value):
            assert name in names
            path=proc/'sys'/key.replace('.','/')
            return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
        yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
    run('ip','link','add',left_if,'type','veth','peer','name',right_if)
    for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
        run('ip','link','set',interface,'netns',name)
        for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
        ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
    data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
    for entry in data['nftables']:
        rule=entry.get('rule',{})
        if 'comment' in rule:
            for expression in rule['expr']:
                if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
    return found

def server(stack,name,address,port=18080,label='server'):
    code=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import json,socket,sys
address,port,label=sys.argv[1],int(sys.argv[2]),sys.argv[3]
class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        body=json.dumps({'server':label,'peer':self.client_address,'path':self.path}).encode()
        self.send_response(200);self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
    def log_message(self,*args):pass
class Server(ThreadingHTTPServer):
    address_family=socket.AF_INET6 if ':' in address else socket.AF_INET
    def server_bind(self):
        if self.address_family==socket.AF_INET6:self.socket.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
        super().server_bind()
service=Server((address,port),Handler);print('ready',flush=True);service.serve_forever()
'''
    process=spawn(stack,['ip','netns','exec',name,'python3','-u','-c',code,address,str(port),label])
    assert line(process).strip()=='ready';return process
def request(name,address,port=18080):
    code=r'''
import http.client,json,sys
client=http.client.HTTPConnection(sys.argv[1],int(sys.argv[2]),timeout=.7)
try:
    client.request('GET','/lab-visible-message');response=client.getresponse();body=json.loads(response.read())
    assert response.status==200;print(json.dumps({'ok':True,'status':response.status,'body':body}))
except OSError as error:print(json.dumps({'ok':False,'type':type(error).__name__,'errno':error.errno}))
finally:client.close()
'''
    return json.loads(ns(name,'python3','-c',code,address,str(port)).stdout)

def routed(setting):
    wire('client','c0','router','r0',['10.201.1.2/30'],['10.201.1.1/30'])
    wire('router','r1','target','t0',['10.201.2.1/30'],['10.201.2.2/30'])
    setting('router','net.ipv4.ip_forward',1)
    ip('client','route','add','default','via','10.201.1.1')
    ip('target','route','add','default','via','10.201.2.1')
with lab(['client','router','target']) as (root,stack,setting):
    routed(setting);server(stack,'target','10.201.2.2')
    def trace(*options):return ns('client','traceroute','-n','-q','1','-w','0.3','-m','4',*options,'10.201.2.2')
    initial=trace();assert '10.201.1.1' in initial.stdout and '10.201.2.2' in initial.stdout.splitlines()[-1]
    nft('router','''table ip quiet {
 chain output { type filter hook output priority 0; policy accept;
  icmp type time-exceeded counter drop comment "hidden_hop"
 }
}
''')
    silent=trace();rows=silent.stdout.splitlines();assert '*' in rows[1] and '10.201.2.2' in rows[2]
    still_works=request('client','10.201.2.2');assert still_works['ok'] and counters('router')['hidden_hop']>0
    nft('router','delete table ip quiet')
    nft('target','''table ip udp_filter {
 chain input { type filter hook input priority 0; policy accept;
  udp dport 33434-33500 counter drop comment "udp_probe_drop"
 }
}
''')
    udp_blocked=trace();assert '10.201.1.1' in udp_blocked.stdout and all('10.201.2.2' not in row for row in udp_blocked.stdout.splitlines()[1:])
    tcp=trace('-T','-p','18080');assert '10.201.2.2' in tcp.stdout.splitlines()[-1]
    assert counters('target')['udp_probe_drop']>0 and request('client','10.201.2.2')['ok']
    print(json.dumps({'ordinary_udp_trace':initial.stdout,'router_time_exceeded_filtered':silent.stdout,
        'http_works_with_silent_hop':still_works,'destination_udp_filtered':udp_blocked.stdout,
        'udp_trace_exit_despite_missing_destination':udp_blocked.returncode,'tcp_trace_to_service':tcp.stdout,
        'target_filter_counters':counters('target')}))
"""
name='xbcatm-netpath-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
    with TemporaryDirectory(prefix=name+'-') as directory:
        root=Path(directory);(root/'Dockerfile').write_text(recipe)
        build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
        result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
            '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
        assert result.returncode==0,result.stderr
        print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
    docker('rm','-f',name);docker('image','rm',name)

The target listens for real HTTP requests on TCP port 18080. With ordinary forwarding and no added filters, a numeric UDP trace reports router 10.201.1.1 at hop one and target 10.201.2.2 at hop two. The script uses one probe per hop, a short reply deadline, and a four-hop limit. These deliberately bounded settings are suitable for this local topology, not recommended timeout values for every WAN.

Next, the router drops only its locally generated ICMP Time Exceeded messages in an output chain. The first hop becomes an asterisk, while the destination still replies at hop two. An actual HTTP request succeeds, and the router’s filter counter increases. This isolates lost diagnostic feedback from lost forwarding: application traffic can cross the router despite that missing trace response.

The router filter is removed, and the target then drops the high UDP ports used by the default tracing method. A new trace shows the router followed by stars, with no destination reply. In the tested traceroute version, the process nevertheless exits with status zero. A successful command exit therefore cannot serve as an assertion that the intended destination was observed.

With the target’s UDP filter still installed, the script switches to TCP probes using -T -p 18080. The trace reaches the destination, and an independent HTTP request also succeeds. The UDP drop counter confirms the earlier probes traversed the intended filter. Changing the probe method explains the different trace results without requiring any change to the actual application’s listening port or service behavior.

Use a trace to narrow a hypothesis

Start with the application’s real source, destination, address family, and transport where possible. Run the application probe alongside the trace and compare repeated observations from relevant locations. A result obtained from a laptop may differ from a container namespace or another network with different routing and policies. The script’s isolated namespaces make the chosen source and packet path explicit.

When a hop is silent, inspect later responses and end-to-end behavior before assigning a fault to that router. When only one tracing method fails, check policy for that method rather than assuming all protocols share its fate. For automation, parse the relevant destination result and combine it with a meaningful service check. Preserve raw trace output and timing context so a later reviewer can distinguish missing evidence from demonstrated failure.

Sources