Strace observes system calls and signals at the boundary between a selected Linux process and the kernel. It can reveal attempted paths, arguments, return values, and errors without rebuilding the program. That evidence is useful when application logs omit an operation, but it does not expose every user-space decision or automatically identify which failed call caused the application’s final outcome.
When it breaks
A parent can launch a child that performs the relevant work. Tracing only the parent may show waiting and exit behavior while missing the child’s failed file open. Following created processes with -f expands coverage, but does not automatically trace every preexisting sibling or external service involved in the request. The selected process set is part of the evidence.
An error return is not necessarily the root cause. Programs routinely probe optional files, search several locations, and recover from expected failures. Finding the first ENOENT, or the last unsuccessful call before exit, cannot replace understanding application control flow. The meaningful question is how that result affected the operation being investigated.
Filters and timing also shape interpretation. Restricting output to file opens excludes network and other activity by design. Tracing can change scheduling and latency, especially for workloads making many system calls; there is no universal slowdown factor. A clean filtered trace does not prove that excluded operations succeeded or that the same timing behavior occurs without a tracer.
Check it quickly
Save this as straceprobe.py and run python3 straceprobe.py in a disposable Linux environment with strace installed. The recorded run used Linux 6.12.76, Python 3.11.2, and strace 6.1. The fixture starts its own harmless program, uses temporary configuration paths, and requires permission to trace the children it launches:
from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess,sys
child=r'''
from pathlib import Path
import sys
try:Path(sys.argv[1]).read_bytes()
except FileNotFoundError:pass
try:content=Path(sys.argv[2]).read_bytes()
except FileNotFoundError:sys.exit(7)
assert content==b'configured'
'''
parent='import subprocess,sys\nchild='+repr(child)+'\nsys.exit(subprocess.run([sys.executable,"-c",child,*sys.argv[1:]]).returncode)'
with TemporaryDirectory(prefix='xbcatm-strace-') as directory:
root=Path(directory);optional=root/'optional.conf';required=root/'required.conf'
command=[sys.executable,'-c',parent,str(optional),str(required)]
def trace(name,follow):
output=root/(name+'.trace')
args=['strace']+(['-f'] if follow else [])+['-tt','-T','-yy','-s','200','-e','trace=openat,close','-o',str(output),'--',*command]
result=subprocess.run(args,capture_output=True,text=True,timeout=10)
return result.returncode,output.read_text(),result.stderr
code,unfollowed,error=trace('parent-only',False)
assert code==7 and str(required) not in unfollowed,(code,error)
code,followed,error=trace('with-children',True)
assert code==7,(code,error)
def calls(text,path):return [line for line in text.splitlines() if 'openat(' in line and '"'+str(path)+'"' in line]
for path in [optional,required]:
lines=calls(followed,path);assert len(lines)==1 and 'ENOENT' in lines[0],lines
print('failing run:',lines[0])
print('without -f the parent trace misses the child opens; following the child exposes both missing paths')
required.write_bytes(b'configured')
code,recovered,error=trace('recovered',True);assert code==0,(code,error)
assert 'ENOENT' in calls(recovered,optional)[0]
assert ' = -1 ' not in calls(recovered,required)[0]
print('successful run:',calls(recovered,optional)[0])
print('successful run:',calls(recovered,required)[0])
print('a successful application can contain ENOENT; the fixture distinguishes optional lookup from required configuration')
summary=root/'summary'
result=subprocess.run(['strace','-f','-c','-e','trace=openat','-o',str(summary),'--',*command],capture_output=True,text=True,timeout=10)
assert result.returncode==0,result.stderr
text=summary.read_text();assert 'openat' in text and 'total' in text
print('syscall summary:\n'+text)
The application child tries an optional configuration file and tolerates its absence. It then requires a second configuration file, exiting with status seven when that file is missing. The parent returns the child’s status. This produces a complete failure whose cause is known from the fixture code, rather than relying on an unavailable placeholder application.
The first trace omits -f. The command returns seven, but the required path does not appear among the traced parent opens. Repeating with -f records the child’s opens for both optional and required paths, each returning ENOENT. The comparison demonstrates why process selection can hide the relevant operation even when the overall command’s failure is visible.
The fixture then creates only the required file with the expected content. The next run exits zero, and its required open succeeds. The optional lookup still returns ENOENT. This successful run is a deliberate counterexample to the claim that any error line proves application failure; the optional and required results have different roles in the program’s logic.
The trace includes -tt timestamps, -T elapsed durations for traced calls, and -yy descriptor context. A bounded string length keeps the controlled paths visible, while -o separates trace output from application output. The script asserts actual return codes and targeted open results instead of treating the existence of a trace file as sufficient verification.
A final -c run summarizes the selected openat calls, including their errors, for the recovered application. Its count includes runtime startup as well as fixture work. These totals are not the number of business requests, and default summary timing should not be equated with total application wall time. Exact counts and durations may vary with the runtime and environment.
Follow a hypothesis through to an outcome
Start from a concrete operation, identify the processes performing it, and choose a filter that preserves the evidence needed to test the hypothesis. File, network, and process tracing answer different questions. Expand the selection when a dependency falls outside the current view, while keeping the captured workload small enough to interpret.
Attaching with -p depends on process identity and tracing permissions. Credentials, security policy, namespace relationships, and container configuration can prevent access even when the PID exists. This fixture traces newly launched children without extra container capabilities; that result does not establish permission to attach to arbitrary existing services on another system.
Keep captured arguments and data appropriate to the investigation. Trace output can contain paths, request data, or credentials, so use controlled inputs and limit what is retained. Correlate the trace with application behavior, reproduce the proposed fix, and check the final outcome. Here, successful recovery with the optional error still present provides that causal comparison.
