Linux · XB Field Notes

LVM snapshots capture a block device in time

Mon, 10 Apr 2028 00:00:00 +0000

An LVM snapshot is neither a copy nor a reflink: it is a fixed-size copy-on-write exception store attached to an origin logical volume. lvcreate -s -L 5G -n snap vg/data allocates 5 GiB from the volume group, and the device-mapper snapshot target records the original contents of each origin chunk the first time it is written. Creation is instant whatever the origin’s size, because nothing is copied.

That size is the whole safety margin, and it is the parameter people get wrong. A store sized for a quiet week is exhausted in an hour by a database checkpoint or an rsync, and when it fills the snapshot does not degrade — it becomes invalid and returns I/O errors. The recovery point disappears exactly when you needed it, which is why the store’s free space belongs on a dashboard, not in a runbook.

An exception store, not a copy

A classic LVM snapshot uses two device-mapper targets: the origin is re-mapped as snapshot-origin, and a new snapshot device combines it with a COW device. Reads of unchanged chunks go straight to the origin; the first write to a chunk copies the old chunk into the COW device, then writes the new data. So consumption tracks the number of distinct chunks that changed, not the number of writes: the simulation writes a million times to sixteen hot chunks and consumes fifteen exceptions, about 60 KiB.

Watch it with lvs -a -o lv_name,lv_attr,origin,data_percent,lv_size. A classic snapshot’s lv_attr reads swi-a-s--- — s for snapshot, w writable, and a state character that becomes I when the store is exhausted. data_percent is the store’s fill, and that is the number to alert on, at 70 or 80 percent; 100 percent is not a warning, it is an obituary. lvextend grows the COW device while the snapshot is still valid; lvchange --monitor y lets dmeventd do it automatically.

Exhaustion is invalidation

The kernel documentation is blunt: the COW device will often be smaller than the origin, and if it fills up the snapshot becomes useless and is disabled, returning errors. LVM’s guidance is to plan for expected writes — often 20 percent of the origin — and extend before it fills. A small slice of the store holds the COW block map, so usable data capacity is a little under the -L value.

The blast radius is narrow but the timing is cruel: the origin keeps running normally, so nothing looks wrong until the backup that reads the snapshot fails or the rollback is unavailable. There is no partial mode: at 100 percent, the snapshot never existed.

Merging, and why thin changes the failure

lvconvert --merge vg/snap rolls the snapshot’s changed chunks back into the origin, and it runs in the background while the origin stays usable — writes to chunks not yet merged are deferred. The constraint is on the origin: it must be inactive when the merge is set up, so unmount it, or schedule the merge and activate the origin afterwards. Merging consumes the snapshot, so copy out anything you need first.

Thin snapshots change the failure mode rather than removing it. A thin snapshot is created when the origin is a thin LV and no size is given; it shares blocks in the thin pool instead of owning a store, so its own divergence cannot invalidate it. The pool is the shared resource now: when it runs out, the damage is not one dead snapshot but every thin volume in the pool, which LVM warns may be damaged and hard to repair. thin_pool_autoextend_threshold and dmeventd exist precisely because pool exhaustion is a pool-wide outage.

When it breaks

The classic failure is a backup that silently stops being a backup. The job mounts the snapshot, copies for longer than the store can cover, and the store reaches 100 percent; the snapshot goes invalid and the copy fails — or worse, monitoring watches only the exit code while the job reads a stale mount. Check data_percent on every snapshot LV, alert at 70 percent, and size for the busiest hour in the retention window.

The second failure is the merge that cannot happen when you need it. lvconvert --merge requires the origin closed, and teams discover that during the incident. Rehearse the merge on a disposable volume, keep a real backup independent of the snapshot, and remember that a snapshot whose store is full cannot be merged.

Check it quickly

Look at the store’s fill before trusting the snapshot:

lvs -a -o lv_name,vg_name,lv_attr,origin,data_percent,lv_size
lvs -a -o lv_name,data_percent,metadata_percent vg/thinpool
dmsetup status vg-snap
lvextend -L +2G vg/snap

data_percent is the COW store’s fill and should sit well under 100; a state character of I in lv_attr means the snapshot is already invalid. For a thin snapshot, data_percent and metadata_percent on the pool are the numbers that matter; dmsetup status shows allocated sectors against the total so you can watch a merge drain.

Sources

lvcreate(8), documents -s/--snapshot, COW versus thin snapshots and the sizing guidance, checked 2026-09-16.
lvs(8), defines the lv_attr state characters including invalid snapshot, checked 2026-09-16.
Device-mapper snapshot support, the exception store, invalidation and lvconvert --merge behaviour, checked 2026-09-16.
lvmthin(7), thin snapshots, pool exhaustion and automatic extension, checked 2026-09-16.

The TCP backlog holds connections that are still syncing

Fri, 24 Mar 2028 00:00:00 +0000

A listening TCP socket has two queues, not one, and the sysctl this article is named after governs the queue that almost never drops anything. The backlog passed to listen(2) is the accept queue, holding connections that finished the handshake and await the application; the SYN queue holds half-open requests. Confusing them produces the wrong fix.

That matters because the famous advice, raise tcp_max_syn_backlog, is largely obsolete: tcp_syncookies defaults to 1, and as listen(2) puts it, with syncookies enabled there is no logical maximum length and that setting is ignored. What actually bites is the accept queue and how fast the application calls accept().

The backlog argument is the accept queue

listen(2) is explicit that the behaviour changed with Linux 2.2: backlog now specifies the queue length for completely established sockets waiting to be accepted, not the number of incomplete requests. The incomplete queue is bounded separately by /proc/sys/net/ipv4/tcp_max_syn_backlog. And if the backlog argument exceeds /proc/sys/net/core/somaxconn it is silently capped to that value, 4096 since Linux 5.4 and 128 before. So listen(8192) on a default host gives you 4096, and nothing logs it.

tcp(7) describes tcp_max_syn_backlog as the maximum number of queued requests that have not received an acknowledgement, defaulting to 256, raised to 1024 when memory is at least 128 MB, cut to 128 below 32 MB. ip-sysctl adds the qualifications: it is a per-listener limit, each half-open request costs about 304 bytes, and tcp_syncookies defaults to 1, sending a cookie on overflow instead of dropping.

With syncookies on, the SYN queue stops mattering

ip-sysctl documents tcp_syncookies as a fallback that must not be used to help a loaded server stand against legal connection throughput, and says syncookies seriously violate the protocol, disallow TCP extensions, and can degrade services. listen(2) goes further: with syncookies enabled there is no logical maximum length and tcp_max_syn_backlog is ignored. tcp(7) still describes the same knob as dropping requests once exceeded, which is the syncookies-off behaviour. Both pages are right about different configurations.

The honest reading is therefore: leave syncookies at 1 and stop tuning the SYN queue. The cost is that the fallback loses TCP options such as window scaling and SACK on the connections it rescues, which is a throughput problem rather than a connectivity one. If you see SYN-flood warnings while the traffic is legitimate, ip-sysctl’s own advice is to tune the other knobs, not to disable syncookies.

The accept queue is where the pressure lands

With the SYN queue neutralised, the queue that fills is the accept queue, sized by min(listen(backlog), somaxconn). Overflow behaviour depends on tcp_abort_on_overflow, default FALSE: ip-sysctl says the default lets a connection recover if the overflow was a burst, and warns that enabling the abort option can harm clients. The mechanism is the surprise: by default the kernel silently discards the final ACK, so connect() succeeds and the failure only surfaces when the first request stalls or is reset.

A bigger backlog is a delay line, not capacity. In the model, listen(128) and listen(65535) accept the same 20,000 connections in a second; the larger backlog only postpones the overflow, because the accept rate is set by the worker count.

When it breaks

The first failure is the silent one, and it is the default. A service whose accept loop is too slow overflows the accept queue, the kernel drops the final ACKs, and clients see connections that establish and then hang. Nothing is refused, so a client connection pool fills with dead sockets and the error surfaces as timeouts somewhere unrelated. Watch TcpExtListenOverflows and TcpExtListenDrops: those counters climbing while the accept rate is flat means the handler is the limit.

The second failure is the dropped SYN, which needs syncookies off to happen. A full SYN queue then drops the request and the client retransmits on a schedule whose documented endpoints are 67 seconds to the last retransmission and 131 seconds to the final timeout, so the user-visible symptom is a connect that hangs for over two minutes and then fails. Raising tcp_max_syn_backlog is a real fix in that configuration, and turning syncookies back on is a better one, if you accept the loss of TCP extensions.

Check it quickly

ss -lnt 'sport = :8443'
nstat -az TcpExtListenOverflows TcpExtListenDrops TcpExtTCPReqQFullDrop TcpExtTCPReqQFullDoCookies
sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog net.ipv4.tcp_syncookies net.ipv4.tcp_abort_on_overflow

In ss -lnt, Send-Q is the backlog the socket was created with and Recv-Q is how many established connections are queued and unaccepted, so Recv-Q sitting at Send-Q is the queue being full. The nstat counters should be zero or near it: TcpExtListenOverflows and TcpExtListenDrops are accept-queue events, while TcpExtTCPReqQFullDrop is a dropped SYN and TcpExtTCPReqQFullDoCookies is the fallback doing its job. The sysctl line shows what the kernel will honour, and if somaxconn is smaller than your listen() argument then that smaller number is your real backlog.

Sources

Kernel ip-sysctl documentation, somaxconn, tcp_max_syn_backlog, tcp_abort_on_overflow and the syncookies warning, checked 2026-09-16.
Linux listen(2), the accept queue since Linux 2.2, the silent somaxconn cap, and syncookies removing the SYN-queue bound, checked 2026-09-16.
Linux tcp(7), tcp_max_syn_backlog defaults by memory size and the syncookies caveats, checked 2026-09-16.

iotop shows which process is burning the disks

Wed, 15 Mar 2028 00:00:00 +0000

A disk that spikes with no obvious writer is hard to chase; iotop maps throughput straight back to the task doing the damage. System-wide tools show the symptom — high await, saturated utilization — but never the suspect: fifty processes share the disk and the average blames all of them equally. The question that ends the investigation is always per-process — who, how much, since when — and iotop answers exactly that, live, from kernel counters no application can hide from.

Keep it beside the system-wide view rather than instead of it. iostat or vmstat says the disk is the bottleneck; iotop names the process making it so. Together they compress a storage mystery into a two-command diagnosis: confirm the resource, identify the consumer.

Per-process counters

iotop reads per-process IO from the kernel, so live read and write rates map to a pid rather than a nameless load average. Run it with -o to show only processes that are actually doing IO. The default view splits reads from writes and shows swap-in percentage alongside throughput, which separates the genuinely disk-hungry from the merely present — a process with high DISK READ but zero writes is scanning, one with writes is producing, and the distinction decides whether you throttle, relocate, or kill.

Note what the counters actually measure: bytes passing through the IO layer per process, including buffered writes not yet on disk. A process can show huge write rates while the disk looks idle (page cache absorbing) or modest rates while the disk saturates (many small synchronous writes). Read iotop together with device utilization: process-side blame plus device-side saturation is the complete picture, and either half alone misleads.

See whatever sweeps the disk

Sort by bandwidth and the offender stands out, whether it is a log hoarder, an indexer, or a runaway backup. Follow the pid to the exact process (and often the file) before acting. The usual lineup: an unrotated log written at debug level, a reindex job scheduled against peak hours, a backup writing to the same spindle it reads from, a container layer unpack looping on a corrupt image. Each has a different fix — rotate, reschedule, separate, prune — and killing the process treats the symptom while the cause reschedules itself for tomorrow.

Resolve the pid to its full context before touching anything: command line, working directory, open files via lsof, parent process. A java process burning writes might be ten applications behind one PID namespace; the fix belongs to the right one. Attribution first, action second — every storage war story that ends in regret skipped the first step.

Watch before you act

iotop observes rather than changes anything, so you can investigate the culprit before deciding what to throttle, move, or kill. Pair it with strace or lsof on the pid to pin the file and the syscall. The combination is decisive: iotop says process 18412 writes 400 MB/s, lsof says the writes go to one growing log file, strace shows O_APPEND writes of 4 KB — diagnosis complete, and the fix (log level, rotation, async appender) is obvious instead of debated.

Record the baseline while healthy so the incident comparison means something. A five-minute iotop batch capture on a normal day — -b -n 60 to a file — gives you “normal for this host” in numbers: which processes usually lead, what rates are ordinary. Without it, every spike looks unprecedented; with it, most incidents classify in seconds as “the indexer, again, an hour early.”

When it breaks

The classic meta-failure is permission blindness: run without privileges and iotop shows question marks instead of process details, or hides other users’ processes entirely — and the actual offender, running as another user, stays invisible. Run with the necessary privileges (root or the right capabilities) from the start, or the tool exonerates exactly the processes you cannot see. An investigation that “found nothing” with unprivileged iotop found nothing about your own processes only.

The second failure is sampling illusion on bursty IO: a process doing violent one-second bursts every minute looks innocent in a 5-second average and guilty in a 1-second one, or vice versa. Vary the delay interval when the numbers and the symptoms disagree, and prefer accumulated totals (-a) over instantaneous rates for “who hurt the disk over the last hour” questions. The right aggregation answers the question asked; the default answers whichever question its interval assumes.

Check it quickly

Capture who does IO, with context, non-interactively:

iotop -b -n 5 -o -d 2 | head -n 30
pid=$(iotop -b -n 1 -o | awk 'NR>5 {print $1}' | head -n 1); ls -l /proc/$pid/exe; cat /proc/$pid/cmdline | tr '\0' ' '
iostat -x 1 3 | grep -E "Device|sd|nvme"

The batch capture ranks offenders without a terminal UI, the pid resolution names the top writer’s binary, and iostat confirms the device side is actually saturated.

Sources

iotop man page, Official iotop reference, checked 2026-06-01.
vmstat man page, documents system-wide CPU and IO-wait reporting, checked 2026-09-05.

Network namespaces isolate networking; their names do not own every process

Sat, 19 Feb 2028 00:00:00 +0000

A Linux network namespace separates networking resources such as interfaces, routes, neighbor tables, socket ports, and packet-filter rules. It shares the kernel with other namespaces; it is not a separate operating system. A process normally inherits its parent’s network namespace, and tools can execute a process in another one. The resulting isolation covers networking resources, not every aspect of process identity or filesystem access.

When it breaks

Running ip netns exec does not automatically provide the full isolation of a container. PID namespaces, user namespaces, and hostname isolation are separate mechanisms. An application may see different routes while still sharing ordinary files and a hostname with its parent environment. Treating one network namespace as a complete sandbox can therefore assign it guarantees that were never configured.

A new namespace also does not automatically receive useful external connectivity. Interfaces need deliberate placement and configuration, and local loopback may need to be brought up. Moving an interface changes which namespace manages it; the old context will no longer list that device as its own. A configuration command executed from the wrong namespace can report a missing interface even though the intended device still exists elsewhere.

Namespace names have their own lifecycle. ip netns conventionally exposes handles beneath /run/netns, commonly also reachable through /var/run/netns. Deleting a name removes that handle, but a process or another reference can keep the namespace alive. Cleanup that equates disappearance from the named list with termination of every service can leave networking resources and a running process behind.

Check it quickly

Save this as check-network-namespace.py and run python3 check-network-namespace.py with Docker. The pinned environment uses Python 3.11.16 and iproute2 6.1.0. NET_ADMIN and SYS_ADMIN permit nested namespace operations inside a disposable container. A temporary proc mount supports namespace-local network settings; no host interface is moved and no host port is published:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,'child exited without a response'
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found

def server(stack,name,address,port=18080,label='server'):
 code=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import json,socket,sys
address,port,label=sys.argv[1],int(sys.argv[2]),sys.argv[3]
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 body=json.dumps({'server':label,'peer':self.client_address,'path':self.path}).encode()
 self.send_response(200);self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
 def log_message(self,*args):pass
class Server(ThreadingHTTPServer):
 address_family=socket.AF_INET6 if ':' in address else socket.AF_INET
 def server_bind(self):
 if self.address_family==socket.AF_INET6:self.socket.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
 super().server_bind()
service=Server((address,port),Handler);print('ready',flush=True);service.serve_forever()
'''
 process=spawn(stack,['ip','netns','exec',name,'python3','-u','-c',code,address,str(port),label])
 assert line(process).strip()=='ready';return process
def request(name,address,port=18080):
 code=r'''
import http.client,json,sys
client=http.client.HTTPConnection(sys.argv[1],int(sys.argv[2]),timeout=.7)
try:
 client.request('GET','/lab-visible-message');response=client.getresponse();body=json.loads(response.read())
 assert response.status==200;print(json.dumps({'ok':True,'status':response.status,'body':body}))
except OSError as error:print(json.dumps({'ok':False,'type':type(error).__name__,'errno':error.errno}))
finally:client.close()
'''
 return json.loads(ns(name,'python3','-c',code,address,str(port)).stdout)

with lab(['blue','green']) as (root,stack,setting):
 probe='import json,os,socket;print(json.dumps({"net":os.readlink("/proc/self/ns/net"),"pid":os.readlink("/proc/self/ns/pid"),"hostname":socket.gethostname()}))'
 parent=json.loads(run('python3','-c',probe).stdout)
 views={name:json.loads(ns(name,'python3','-c',probe).stdout) for name in ['blue','green']}
 assert len({parent['net'],views['blue']['net'],views['green']['net']})==3
 assert all(v['pid']==parent['pid'] and v['hostname']==parent['hostname'] for v in views.values())
 for name in ['blue','green']:
 ip(name,'link','add','dummy0','type','dummy');ip(name,'addr','add','192.0.2.10/24','dev','dummy0');ip(name,'link','set','dummy0','up')
 ip('blue','route','add','203.0.113.0/24','dev','dummy0')
 assert json.loads(ip('blue','-j','route','show','203.0.113.0/24').stdout)
 assert not json.loads(ip('green','-j','route','show','203.0.113.0/24').stdout)
 setting('blue','net.ipv4.ip_forward',0);setting('green','net.ipv4.ip_forward',1)
 forwarding={name:ns(name,'sysctl','-n','net.ipv4.ip_forward').stdout.strip() for name in ['blue','green']};assert forwarding=={'blue':'0','green':'1'}
 marker=root/'shared.txt';marker.write_text('shared filesystem')
 assert ns('blue','cat',str(marker)).stdout==ns('green','cat',str(marker)).stdout=='shared filesystem'
 daemon=server(stack,'blue','192.0.2.10',label='blue')
 named=request('blue','192.0.2.10');assert named['ok'] and named['body']['server']=='blue'
 run('ip','netns','delete','blue');assert not Path('/run/netns/blue').exists()
 assert daemon.poll() is None and os.readlink(f'/proc/{daemon.pid}/ns/net')==views['blue']['net']
 by_pid=run('nsenter','-t',str(daemon.pid),'-n','python3','-c',
 'import http.client,json; c=http.client.HTTPConnection("192.0.2.10",18080,timeout=1);c.request("GET","/");r=c.getresponse();print(json.dumps({"status":r.status,"body":json.loads(r.read())}));c.close()')
 surviving=json.loads(by_pid.stdout);assert surviving['status']==200 and surviving['body']['server']=='blue'
 print(json.dumps({'parent':parent,'namespace_views':views,'blue_only_route':True,'forwarding':forwarding,
 'same_file_visible':True,'before_name_deleted':named,'name_deleted_process_alive':True,'after_name_deleted':surviving}))
"""
name='xbcatm-netpath-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The parent, blue, and green report three different network namespace identifiers. Their PID namespace identifiers and hostnames remain equal. A marker file created by the parent is also readable in both children. Those positive observations establish the intended scope: networking has changed, while these particular process and filesystem properties remain shared in the fixture.

Each child creates its own dummy0 with address 192.0.2.10/24. The identical names and addresses coexist because they belong to different networking contexts. Only blue receives a route for 203.0.113.0/24. Querying green’s routes confirms that it has no corresponding entry. The example verifies separate effective tables rather than merely printing the two namespace names.

The program explicitly sets IPv4 forwarding to zero in blue and one in green, then reads the values back through each namespace. This is a setting-isolation check, not a packet-forwarding demonstration. The local dummy interfaces have no physical peer. Configured addresses and routes can be inspected without implying that either namespace can reach an external service.

A real HTTP server starts in blue on 192.0.2.10. A normal named-namespace request receives status 200 and identity blue. The script then deletes the blue name and confirms its handle is absent, while the server process and its network namespace identifier remain. The process has not been killed by removing the naming reference.

Using that known server PID, nsenter executes another client in the surviving network namespace. It again receives status 200 and identity blue. This establishes continued application operation after name deletion, not just the presence of a proc entry. The fixture finally stops its own process and removes remaining named resources through its cleanup stack and container lifetime.

Keep naming, process ownership, and connectivity separate

Record both a service’s process ownership and the networking context used for management. Namespace identifiers help compare currently live contexts; do not turn an ephemeral identifier into a permanent inventory identity. If a name disappears unexpectedly, inspect the relevant owned process before concluding that its networking state has vanished or creating a replacement with assumed empty state.

For ordinary configuration files, ip netns exec can create a mount namespace and bind namespace-specific files into their conventional locations. This supports arrangements such as per-namespace resolver configuration; it does not establish a private copy of every filesystem object. Document those additional choices when reproducing a client’s environment so that DNS settings and route isolation are not confused.

Build connectivity through explicit virtual links, routes, forwarding, and filtering as required. A network namespace supplies a boundary for those resources, while their configuration determines communication. During teardown, stop the processes and release references owned by the service before expecting the namespace to disappear completely. Name deletion is one management operation within that lifecycle, not a universal process-termination command.

Sources

Linux network_namespaces(7), network resources isolated by a namespace, checked 2026-09-12.
Linux ip-netns(8), named handles, execution and deletion while processes remain, checked 2026-09-12.
Linux ip-link(8), interface creation and namespace placement, checked 2026-09-12.

Strace explains selected system-call boundaries

Mon, 14 Feb 2028 00:00:00 +0000

Strace observes system calls and signals at the boundary between a selected Linux process and the kernel. It can reveal attempted paths, arguments, return values, and errors without rebuilding the program. That evidence is useful when application logs omit an operation, but it does not expose every user-space decision or automatically identify which failed call caused the application’s final outcome.

When it breaks

A parent can launch a child that performs the relevant work. Tracing only the parent may show waiting and exit behavior while missing the child’s failed file open. Following created processes with -f expands coverage, but does not automatically trace every preexisting sibling or external service involved in the request. The selected process set is part of the evidence.

An error return is not necessarily the root cause. Programs routinely probe optional files, search several locations, and recover from expected failures. Finding the first ENOENT, or the last unsuccessful call before exit, cannot replace understanding application control flow. The meaningful question is how that result affected the operation being investigated.

Filters and timing also shape interpretation. Restricting output to file opens excludes network and other activity by design. Tracing can change scheduling and latency, especially for workloads making many system calls; there is no universal slowdown factor. A clean filtered trace does not prove that excluded operations succeeded or that the same timing behavior occurs without a tracer.

Check it quickly

Save this as straceprobe.py and run python3 straceprobe.py in a disposable Linux environment with strace installed. The recorded run used Linux 6.12.76, Python 3.11.2, and strace 6.1. The fixture starts its own harmless program, uses temporary configuration paths, and requires permission to trace the children it launches:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess,sys

child=r'''
from pathlib import Path
import sys
try:Path(sys.argv[1]).read_bytes()
except FileNotFoundError:pass
try:content=Path(sys.argv[2]).read_bytes()
except FileNotFoundError:sys.exit(7)
assert content==b'configured'
'''
parent='import subprocess,sys\nchild='+repr(child)+'\nsys.exit(subprocess.run([sys.executable,"-c",child,*sys.argv[1:]]).returncode)'
with TemporaryDirectory(prefix='xbcatm-strace-') as directory:
 root=Path(directory);optional=root/'optional.conf';required=root/'required.conf'
 command=[sys.executable,'-c',parent,str(optional),str(required)]
 def trace(name,follow):
 output=root/(name+'.trace')
 args=['strace']+(['-f'] if follow else [])+['-tt','-T','-yy','-s','200','-e','trace=openat,close','-o',str(output),'--',*command]
 result=subprocess.run(args,capture_output=True,text=True,timeout=10)
 return result.returncode,output.read_text(),result.stderr
 code,unfollowed,error=trace('parent-only',False)
 assert code==7 and str(required) not in unfollowed,(code,error)
 code,followed,error=trace('with-children',True)
 assert code==7,(code,error)
 def calls(text,path):return [line for line in text.splitlines() if 'openat(' in line and '"'+str(path)+'"' in line]
 for path in [optional,required]:
 lines=calls(followed,path);assert len(lines)==1 and 'ENOENT' in lines[0],lines
 print('failing run:',lines[0])
 print('without -f the parent trace misses the child opens; following the child exposes both missing paths')
 required.write_bytes(b'configured')
 code,recovered,error=trace('recovered',True);assert code==0,(code,error)
 assert 'ENOENT' in calls(recovered,optional)[0]
 assert ' = -1 ' not in calls(recovered,required)[0]
 print('successful run:',calls(recovered,optional)[0])
 print('successful run:',calls(recovered,required)[0])
 print('a successful application can contain ENOENT; the fixture distinguishes optional lookup from required configuration')
 summary=root/'summary'
 result=subprocess.run(['strace','-f','-c','-e','trace=openat','-o',str(summary),'--',*command],capture_output=True,text=True,timeout=10)
 assert result.returncode==0,result.stderr
 text=summary.read_text();assert 'openat' in text and 'total' in text
 print('syscall summary:\n'+text)

The application child tries an optional configuration file and tolerates its absence. It then requires a second configuration file, exiting with status seven when that file is missing. The parent returns the child’s status. This produces a complete failure whose cause is known from the fixture code, rather than relying on an unavailable placeholder application.

The first trace omits -f. The command returns seven, but the required path does not appear among the traced parent opens. Repeating with -f records the child’s opens for both optional and required paths, each returning ENOENT. The comparison demonstrates why process selection can hide the relevant operation even when the overall command’s failure is visible.

The fixture then creates only the required file with the expected content. The next run exits zero, and its required open succeeds. The optional lookup still returns ENOENT. This successful run is a deliberate counterexample to the claim that any error line proves application failure; the optional and required results have different roles in the program’s logic.

The trace includes -tt timestamps, -T elapsed durations for traced calls, and -yy descriptor context. A bounded string length keeps the controlled paths visible, while -o separates trace output from application output. The script asserts actual return codes and targeted open results instead of treating the existence of a trace file as sufficient verification.

A final -c run summarizes the selected openat calls, including their errors, for the recovered application. Its count includes runtime startup as well as fixture work. These totals are not the number of business requests, and default summary timing should not be equated with total application wall time. Exact counts and durations may vary with the runtime and environment.

Follow a hypothesis through to an outcome

Start from a concrete operation, identify the processes performing it, and choose a filter that preserves the evidence needed to test the hypothesis. File, network, and process tracing answer different questions. Expand the selection when a dependency falls outside the current view, while keeping the captured workload small enough to interpret.

Attaching with -p depends on process identity and tracing permissions. Credentials, security policy, namespace relationships, and container configuration can prevent access even when the PID exists. This fixture traces newly launched children without extra container capabilities; that result does not establish permission to attach to arbitrary existing services on another system.

Keep captured arguments and data appropriate to the investigation. Trace output can contain paths, request data, or credentials, so use controlled inputs and limit what is retained. Correlate the trace with application behavior, reproduce the proposed fix, and check the final outcome. Here, successful recovery with the optional error still present provides that causal comparison.

Sources

strace(1), follow and filter options, return values, timing, summaries, and decoding, checked 2026-09-12.
ptrace(2), tracing relationships, stops, and access restrictions, checked 2026-09-12.

Cron system and user tables have different contracts

Wed, 12 Jan 2028 00:00:00 +0000

Cron accepts schedules from more than one kind of table. A user crontab belongs to one account, while system tables can name the account for each entry. Copying a line between them without adjusting its fields changes what cron interprets as the command. Identify the table format before debugging an apparently correct five-field calendar expression.

The account column changes the parser boundary

A conventional user entry contains minute, hour, day of month, month, day of week, then command text. The crontab command installs it for the selected user. Debian’s /etc/crontab and files under /etc/cron.d add a username after the five schedule fields. That username belongs to the system-table format; it is not an optional annotation in a user’s command.

The month constraint remains required. When both day-of-month and day-of-week fields are restricted, the Debian/Vixie day rule permits either day match; month and weekday are not alternatives. Check the scheduler’s documented dialect before transferring expressions to products with seconds fields or different day rules. Familiar-looking spacing does not establish compatible semantics.

System drop-ins have ownership, permission, filename, and final-newline requirements. On Debian, use root-owned files that are not group- or other-writable and accepted names rather than arbitrary backup suffixes. Periodic directories such as /etc/cron.daily are invoked through the distribution’s scheduling arrangement, commonly using run-parts; every script is not independently parsed as a crontab. Inspect the installed arrangement before deciding when it runs.

When it breaks

Putting root after the five fields in root’s user crontab asks the shell to execute a command named root. Installation success cannot prove that this command exists. Conversely, omitting the account from a system entry moves command text into the username position. Review both the parser boundary and the effective identity, especially when the job writes files or accesses protected data.

The job environment is also a contract. Do not assume interactive aliases, shell startup files, or variables from the daemon’s parent shell are available. Set required values explicitly and choose an intentional PATH and interpreter. Environment settings in a cron.d file should not be assumed to inherit those in /etc/crontab. Test permissions and outputs as the actual execution account.

Percent signs have special handling before the shell sees the complete command. Unescaped percent characters delimit standard input, with subsequent percents becoming newlines. Shell quoting alone does not remove cron’s interpretation. Keep complex commands in scripts and escape literal percent signs when they must appear directly in a table entry. Separate scheduling syntax checks from command execution and application outcome checks.

Check it quickly

Use a fresh disposable Debian 12 Docker container as root with python3, cron, procps, and passwd installed. Ensure it has no active cron daemon, root crontab, or account named xbtime. Save this as cron-tables.py and run python3 cron-tables.py. It creates only its temporary account, task table, and files, waits up to seventy-five seconds for a real scheduled minute, then removes them.

from pathlib import Path
from tempfile import TemporaryDirectory
import json
import os
import pwd
import shutil
import subprocess
import time

assert os.geteuid() == 0 and Path('/.dockerenv').exists(), 'use a disposable root Linux container'
assert subprocess.run(['pgrep', '-x', 'cron'], capture_output=True).returncode == 1
assert subprocess.run(['crontab', '-l'], capture_output=True).returncode != 0
assert shutil.which('root') is None
username = 'xbtime'
try:
 pwd.getpwnam(username)
except KeyError:
 pass
else:
 raise AssertionError('test account already exists')
system_table = Path('/etc/cron.d/xbcatm_time_test')
assert not system_table.exists()
subprocess.run(['useradd', '--create-home', '--shell', '/bin/sh', username], check=True)
account = pwd.getpwnam(username)
daemon = None
try:
 with TemporaryDirectory(prefix='cron-tables-') as directory:
 root = Path(directory)
 os.chown(root, account.pw_uid, account.pw_gid)
 worker = root / 'worker.py'
 worker.write_text('''from pathlib import Path
import json,os,sys
root=Path(__file__).parent
mode=sys.argv[1]
data={'uid':os.getuid(),'table':os.environ.get('LAB_TABLE'),
 'path':os.environ.get('PATH'),'shell':os.environ.get('SHELL'),
 'parent_only':os.environ.get('PARENT_ONLY')}
if mode=='percent': data['stdin']=sys.stdin.read()
(root/(mode+'.json')).write_text(json.dumps(data))
''')
 user_table = ('SHELL=/bin/sh\nPATH=/usr/bin:/bin\nLAB_TABLE=user\n'
 f'* * * * * /usr/bin/python3 {worker} user\n'
 f'* * * * * /usr/bin/python3 {worker} percent%first%second\n'
 f'* * * * * root /usr/bin/id >{root}/wrong-column.err 2>&1\n')
 subprocess.run(['crontab', '-'], input=user_table, text=True, check=True)
 invalid = subprocess.run(['crontab', '-'], input='* * * * * /bin/true',
 text=True, capture_output=True)
 assert invalid.returncode != 0
 assert subprocess.check_output(['crontab', '-l'], text=True) == user_table
 system_table.write_text('SHELL=/bin/sh\nPATH=/usr/bin:/bin\nLAB_TABLE=system\n'
 f'* * * * * {username} /usr/bin/python3 {worker} system\n')
 system_table.chmod(0o644)
 daemon = subprocess.Popen(['cron', '-f'], env={**os.environ, 'PARENT_ONLY': 'parent-demo'},
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
 try:
 deadline = time.monotonic() + 75
 names = ['user.json', 'system.json', 'percent.json', 'wrong-column.err']
 while time.monotonic() < deadline:
 assert daemon.poll() is None
 if all((root/name).exists() and (root/name).stat().st_size for name in names):
 break
 time.sleep(0.2)
 else:
 raise AssertionError('scheduled table outputs were not produced')
 user = json.loads((root/'user.json').read_text())
 system = json.loads((root/'system.json').read_text())
 percent = json.loads((root/'percent.json').read_text())
 wrong = (root/'wrong-column.err').read_text()
 assert user['uid'] == 0 and user['table'] == 'user'
 assert system['uid'] == account.pw_uid and system['table'] == 'system'
 assert all(value['parent_only'] is None for value in (user, system))
 assert all(value['path'] == '/usr/bin:/bin' for value in (user, system))
 assert percent['stdin'] == 'first\nsecond\n'
 assert 'root: not found' in wrong
 print(json.dumps({'user_table': user, 'system_table': system,
 'missing_final_newline_rejected': invalid.returncode,
 'previous_table_preserved': True, 'percent_stdin': percent['stdin'],
 'username_in_user_table_fails': wrong.strip()}), flush=True)
 finally:
 daemon.terminate()
 daemon.communicate(timeout=5)
 daemon = None
finally:
 if daemon is not None:
 daemon.terminate()
 daemon.communicate(timeout=5)
 system_table.unlink(missing_ok=True)
 subprocess.run(['crontab', '-r'], capture_output=True)
 subprocess.run(['userdel', '--remove', username], capture_output=True, check=True)

The user-table job runs as UID zero. The system-table job runs as the newly created account, UID 1000 in this run. Each receives its own LAB_TABLE value and explicit PATH; neither receives the daemon parent’s PARENT_ONLY variable. A percent-bearing entry supplies first\nsecond\n on standard input. The wrongly inserted username produces root: not found, despite its table installing successfully.

An attempted replacement missing its final newline is rejected with exit one, and the previous table remains intact. All assertions passed on cron 3.0pl1-162. Separate probes found that this package accepted malformed minute text even with its dry-run option, so no universal field-validation guarantee is inferred from a zero exit status. The exercise verifies observed account, environment, parsing, and installation behavior; it does not test mail delivery, daylight-saving handling, or business success. Keep those additional requirements visible when deploying a real scheduled task.

Sources

Debian 12: crontab(5), user/system table syntax, environment and percent handling, checked 2026-09-13.
Debian 12: cron(8), system drop-ins, ownership and execution environment, checked 2026-09-13.
Debian 12: crontab(1), installation, dry-run behavior and newline requirements, checked 2026-09-13.

Shell word splitting turns unquoted variables into several arguments

Fri, 31 Dec 2027 00:00:00 +0000

A shell variable holds a string, but an unquoted expansion can turn that string into several command arguments. In Bash, word splitting examines expansion results using IFS as the delimiter set. This is why a pathname that works when typed with quotes can fail after being stored in a variable. Debug the arguments the receiving command actually gets, including empty ones, before blaming the filesystem.

Splitting depends on the context

With the default IFS, runs of spaces, tabs, and newlines separate fields. Leading and trailing IFS whitespace is discarded. An unquoted value containing two words therefore becomes two arguments; an empty unquoted value normally contributes none. In contrast, "$value" supplies one argument even when the string is empty. Empty and absent arguments can have completely different meanings to a program.

Splitting applies to results of unquoted parameter expansion, command substitution, and arithmetic expansion in suitable contexts. It is not a second parse of the entire command line. An ordinary assignment such as copy=$value stores the string without splitting or filename expansion. Later passing $copy unquoted to a command reintroduces both hazards. Bash conditional expressions using double brackets also have different expansion rules from ordinary command arguments.

IFS is a set of delimiter characters, not a multi-character separator or a regular expression. Non-whitespace delimiters behave differently from IFS whitespace: adjacent colons, for example, can produce an empty field. Setting IFS to an empty string disables splitting, but it does not turn an empty unquoted expansion into an explicit empty argument. Use a parser suited to the format when quoting, escaping, or structured records matter.

When it breaks

Disabling splitting does not disable the filename expansion that follows it. A variable containing a star can still become matching filenames when IFS is empty. The result can change merely because a file appeared in the working directory. Quoting the variable prevents both transformations; changing IFS alone only addresses one stage. Shell options controlling globbing are independent as well.

A scalar string is also a poor representation of an argument list. Adding quote characters inside the string does not make Bash treat those characters as shell quotation on expansion. The quotation that controls parsing must be in the command’s source. Avoid eval as a repair: it introduces another shell parse and can execute text that was previously data. For Bash commands assembled incrementally, use an array and expand it as "${args[@]}".

Some data is deliberately delimited, so splitting is not always a mistake. Make the format and delimiter explicit, keep changes to IFS local to the operation, and decide what empty fields mean. Do not use whitespace splitting to process arbitrary filenames or CSV records. These examples specifically target Bash; other shells, including zsh in its normal configuration, do not necessarily split scalar expansions the same way.

For example, a semicolon obtained from a variable does not by itself become a command separator during ordinary expansion. It remains argument data, even though splitting can still damage the surrounding value. This distinction separates incorrect argument construction from deliberately reparsing a string as code.

Check it quickly

This self-contained Bash experiment checks argument counts rather than using echo, which hides the distinction between one spaced argument and several arguments. It creates its own two matching files, tests an empty value, and checks the difference between whitespace and colon delimiters. The temporary directory is removed at exit.

bash --noprofile --norc <<'BASH'
set -e
set +f
unset GLOBIGNORE
shopt -u nullglob failglob dotglob
work=$(mktemp -d)
trap 'rm -rf -- "$work"' EXIT
cd "$work"
IFS=$' \t\n'
value='report final.md'
set -- $value
[[ $# -eq 2 ]]
set -- "$value"
[[ $# -eq 1 && $1 == 'report final.md' ]]
value=''
set -- $value
[[ $# -eq 0 ]]
set -- "$value"
[[ $# -eq 1 && $1 == '' ]]
touch a.txt 'two words.txt'
IFS=''
pattern='*.txt'
set -- $pattern
[[ $# -eq 2 ]]
set -- "$pattern"
[[ $# -eq 1 && $1 == '*.txt' ]]
IFS=:
record='red::blue'
set -- $record
[[ $# -eq 3 && $2 == '' ]]
printf 'colon field <%s>\n' "$@"
args=('report final.md' '' '*.txt')
set -- "${args[@]}"
[[ $# -eq 3 && $2 == '' && $3 == '*.txt' ]]
printf 'PASS: boundaries, empty values, globbing, delimiters\n'
BASH

Expect three colon-field lines, with an empty middle field, followed by PASS. The assertions before those lines distinguish zero arguments from one empty argument and prove that an empty IFS still permits globbing. In a failing script, put a small argument-counting function at the call boundary and print each argument separately with Bash printf’s %q format. That diagnostic shows boundaries and escaping; it does not turn a human-readable log into a safe command to execute later.

Sources

GNU Bash: Word Splitting, IFS and empty-field rules, checked 2026-09-06.
GNU Bash: Shell Expansions, splitting before filename expansion, checked 2026-09-06.
GNU Bash: Arrays, preserving a list of arguments, checked 2026-09-06.

Huge pages cut TLB misses at the cost of flexibility

Tue, 21 Dec 2027 00:00:00 +0000

Databases and high-throughput servers often win, but they pay by needing contiguous memory and a pool sized in advance, so the trade is architectural, not free.

Why 4 KB hurts

On most architectures a standard page is 4 KiB, and the TLB can hold only a limited number of translations. A database or high-throughput server that touches gigabytes typically walks a working set of millions of pages, so it keeps missing the TLB, the hardware then has to walk the page table to find each mapping. The cost is not the page itself but the constant page-table walks, and at million-page working sets the misses outrun how fast the TLB can be refilled. Bigger pages mean each entry covers more memory, so the same working set needs far fewer entries, which is exactly the relief a memory-heavy workload feels.

hugetlb vs transparent

There are two ways to get bigger pages. hugetlb gives you a pool of explicitly sized pages, 2 MiB or more, that you allocate from a reserved area and manage yourself, which is deterministic and predictable; transparent huge pages (THP) try to promote mappings automatically in the background, which is convenient but can add latency spikes, fragmentation, and copy-on-write surprises on some workloads. For a database or a latency-sensitive service, hugetlb lets you own the pool and the trade, whereas THP leaves those decisions to a general policy you do not fully control. The choice is about who owns the sizing decision, you or a generic heuristics.

Budget the pool

The win of a large page depends on it being fully used and resident in memory. A 2 MiB page that is mostly unused still occupies a whole huge entry, so the memory it locks is effectively wasted, which makes pool sizing the whole skill. Reserve a pool only for the memory the application genuinely and predictably uses, rather than sizing every large allocation by feel, and account for the extra memory that a sparse page set costs. When the pages are fully resident and the working set trims into a small number of entries, huge pages deliver their whole benefit; when they are drafted for code that rarely fills them, they just burn RAM.

When it breaks

An under-sized pool fails loudly at startup. A database configured for more shared buffers than the reserved pages allow exits with Cannot allocate memory, even though free RAM looks plentiful, because hugetlb mappings cannot fall back to 4 KiB pages. Size the pool from the documented requirement plus margin, then confirm HugePages_Total and HugePages_Rsvd in /proc/meminfo after a restart.

A pool that worked at boot can fail later through fragmentation. Huge pages need contiguous 2 MiB ranges, so a long-running machine with free memory may still refuse a new mapping when no contiguous range remains. The symptom is a runtime error with HugePages_Free above zero plus a kernel message about a failed huge-page allocation; early reservation or boot parameters fix it, not retrying the same mmap.

Placement and automation cause quieter damage. Pages reserved on the wrong NUMA node force remote access on every hit, trading TLB wins for latency, so check per-node counts before pinning a database. Transparent huge pages add another risk: background compaction and copy-on-write splitting can turn a steady p99 into spikes, visible as thp_collapse_alloc stalls and khugepaged CPU. When latency matters, set THP to madvise and opt in only where measured.

Check it quickly

Save the following as hugepage_check.py and run python3 hugepage_check.py. It only reads /proc/meminfo and os.sysconf, allocates nothing, and needs no privileges, so it is safe in an unprivileged container. On a host without huge-page support or without /proc, it prints SKIP and shows the page-table arithmetic instead of raising.

#!/usr/bin/env python3
"""Report hugepage coverage and estimated TLB entries saved (read-only)."""
import os
import re
from pathlib import Path

WORKING_SET_BYTES = 8 * 1024**3 # 8 GiB example working set

def parse_meminfo(text):
 vals = {}
 for key in ("HugePages_Total", "HugePages_Free", "HugePages_Rsvd",
 "HugePages_Surp", "Hugepagesize"):
 m = re.search(r"^" + key + r":\s+(\d+)", text, re.M)
 if m:
 vals[key] = int(m.group(1))
 return vals

def main():
 vals = {}
 try:
 text = Path("/proc/meminfo").read_text(encoding="utf-8")
 except OSError as exc:
 print(f"SKIP: /proc/meminfo unavailable ({exc}); showing page-table math only")
 text = ""
 if text:
 try:
 vals = parse_meminfo(text)
 if "HugePages_Total" not in vals or "Hugepagesize" not in vals:
 print("SKIP: no HugePages fields in /proc/meminfo (hugepages unsupported or disabled)")
 else:
 size_kb = vals["Hugepagesize"]
 total = vals["HugePages_Total"]
 free = vals.get("HugePages_Free", 0)
 rsvd = vals.get("HugePages_Rsvd", 0)
 surp = vals.get("HugePages_Surp", 0)
 cov_mb = total * size_kb / 1024
 print(f"Hugepagesize: {size_kb} kB")
 print(f"HugePages total/free/rsvd/surp: {total}/{free}/{rsvd}/{surp}")
 print(f"Reserved coverage: {cov_mb:.1f} MiB ({cov_mb/1024:.2f} GiB)")
 except Exception as exc:
 print(f"SKIP: failed to parse /proc/meminfo ({exc})")
 try:
 base = os.sysconf("SC_PAGE_SIZE")
 except (AttributeError, ValueError, OSError):
 base = 4096
 huge = vals.get("Hugepagesize", 2048) * 1024
 base_entries = WORKING_SET_BYTES // base
 huge_entries = WORKING_SET_BYTES // huge
 print(f"Base page size (sysconf): {base} bytes")
 print(f"8 GiB needs ~{base_entries} base-page entries vs ~{huge_entries} huge-page entries")
 print(f"Estimated TLB entries saved: ~{base_entries - huge_entries}")

if __name__ == "__main__":
 main()

On Linux it names the huge-page size, the total, free, reserved, and surplus counters, and the reserved coverage in MiB and GiB, plus the estimated entries for an 8 GiB working set with base versus huge pages. On macOS or a minimal container you see SKIP plus the entry math with the local SC_PAGE_SIZE. It passed on macOS without /proc; it measures reservation and potential savings, not TLB misses, fragmentation, or NUMA distance, so confirm those with dmesg, per-node counts, and latency percentiles before resizing a pool.

Sources

Kernel hugepages documentation, defines hugetlb settings and behavior, checked 2026-06-01.
Kernel transparent hugepages documentation, defines THP modes and compaction behavior, checked 2026-06-01.

Linux overcommit is a policy the kernel bet on

Sat, 18 Dec 2027 00:00:00 +0000

Tuning vm.overcommit_memory shifts the failure point from a blind heuristic back to your own budget, which changes exactly who eats an OOM kill.

Why malloc rarely fails

vm.overcommit_memory defaults to 0, the heuristic mode that says yes to almost every mmap and malloc while it can plausibly honor them, and only refuses when the system is genuinely short on memory. Committing virtual memory costs almost nothing up front because pages are handed out lazily, so most allocations succeed and most normal applications stay alive at near-zero cost. The price of the generosity is that the kernel can over-commit far beyond RAM, betting that you will not touch every page; when many bytes actually get written, someone has to pay with an OOM kill.

The knob

vm.overcommit_memory lets you choose who holds the budget. The default 0 is heuristic overcommit: the kernel refuses only obviously impossible requests and otherwise says yes, trusting most processes never touch every byte they reserve. Setting it to 1 means always overcommit: almost any allocation is granted, which suits huge sparse mappings that stay mostly untouched, but every failure is pushed downstream to reclaim or the OOM killer. Setting it to 2 means strict accounting: any allocation beyond CommitLimit, roughly swap plus overcommit_ratio percent of RAM (overcommit_kbytes takes over when nonzero), fails with ENOMEM at request time. Neither extreme is free: always-overcommit hides pressure until the machine stalls, and strict mode breaks programs that reserve generously but touch little.

Heuristic versus strict

The real choice is who pays when the reckoning arrives. In heuristic mode the kernel keeps over-committing and, under pressure, picks a victim to kill; in strict mode the boundary is drawn in advance, so the failure surfaces at malloc time and the operator, not the scheduler, decides which process eats an OOM kill. Containers and VMs typically set a controlled value because a host over-committing on behalf of many guests can collapse when their aggregate demand spikes. A laptop can be generous; a production box that must not silently lose a worker should be explicit about its budget.

When it breaks

Under the default heuristic mode, the most confusing failure is a large malloc or mmap that returns success and then dies later. No pages were touched yet, so the allocation looked fine; once the process writes into them and memory runs short, the OOM killer picks a victim, often not the over-allocator but the largest nearby process. The log shows Out of memory: Killed process with no failed allocation in sight, so teams chase the wrong binary while the real over-reservation stays hidden.

Mode 1, always overcommit, fails slower. Absurd reservations are granted without protest, pressure builds silently, and a machine with swap slides into a swap storm where everything is alive but nothing progresses. Latency collapses before any kill frees memory, sessions stall, and the eventual OOM kill looks random because too many processes hold promises the system could never keep together.

Mode 2, strict accounting, fails at request time instead. A JVM reserving a large heap up front, or a fork-heavy service briefly needing twice its resident set accounted, gets ENOMEM at startup even though it would never touch every page. Operators who copy a strict overcommit_ratio from a swapless laptop onto a database host watch healthy programs refuse to start; size the ratio against the gap between CommitLimit and Committed_AS in /proc/meminfo instead of flipping the mode blindly.

Check it quickly

Save the following as overcommit_probe.py and run python3 overcommit_probe.py. It uses only the standard library, opens the overcommit_* knobs and /proc/meminfo read-only, changes nothing, and needs no privileges. Without those proc files it prints SKIP instead of failing.

"""Read-only overcommit audit: report mode, budget, and pressure."""

from pathlib import Path


def read_value(path):
 try:
 return Path(path).read_text(encoding="utf-8").strip()
 except FileNotFoundError:
 return None
 except OSError as exc:
 return "UNREADABLE: %s" % exc


def meminfo_kb(text):
 fields = {}
 for line in text.splitlines():
 name, _, rest = line.partition(":")
 digits = "".join(ch for ch in rest if ch.isdigit())
 if name and digits:
 fields[name.strip()] = int(digits)
 return fields


def main():
 mode = read_value("/proc/sys/vm/overcommit_memory")
 ratio = read_value("/proc/sys/vm/overcommit_ratio")
 kbytes = read_value("/proc/sys/vm/overcommit_kbytes")
 meminfo = read_value("/proc/meminfo")
 if mode is None or meminfo is None:
 print("SKIP: /proc/sys/vm/overcommit_memory or /proc/meminfo missing (non-Linux?)")
 return
 info = meminfo_kb(meminfo)
 print("overcommit_memory=%s overcommit_ratio=%s overcommit_kbytes=%s" % (mode, ratio, kbytes))
 print("MemTotal=%s kB SwapTotal=%s kB CommitLimit=%s kB Committed_AS=%s kB" % (
 info.get("MemTotal", "?"), info.get("SwapTotal", "?"),
 info.get("CommitLimit", "?"), info.get("Committed_AS", "?"),
 ))
 names = {"0": "heuristic", "1": "always overcommit", "2": "strict accounting"}
 print("mode means: %s" % names.get(mode, "unknown value"))
 limit = info.get("CommitLimit")
 committed = info.get("Committed_AS")
 if mode == "0":
 print("suggestion: heuristic default; watch Committed_AS growth, prefer mode 2 for workers that must not be OOM-killed")
 elif mode == "1":
 print("suggestion: always overcommit; expect swap storms under pressure, prefer 0 or 2 unless sparse mappings need it")
 elif mode == "2" and limit and committed:
 headroom = limit - committed
 print("commit headroom=%s kB; suggestion: enlarge ratio/kbytes if headroom is small, shrink them if headroom is huge" % headroom)
 else:
 print("suggestion: unexpected mode value; compare Committed_AS against CommitLimit before changing anything")


if __name__ == "__main__":
 main()

On a default workstation the script reports overcommit_memory=0 and names it heuristic. Under mode 2, compare the headroom: Committed_AS near CommitLimit explains startup ENOMEM, while huge headroom means the ratio is merely conservative. Under mode 1 the suggestion warns about swap storms, matching the slow-then-random failure above.

Sources

Kernel vm sysctl documentation, defines the overcommit_memory modes and defaults, checked 2026-06-01.
proc manual page, documents /proc/meminfo CommitLimit and Committed_AS accounting, checked 2026-06-01.

systemd-tmpfiles applies creation and age policies

Fri, 10 Dec 2027 00:00:00 +0000

systemd-tmpfiles describes filesystem objects and housekeeping rules in configuration files. A rule can create a directory with a chosen mode, while an age rule can make old entries eligible for cleanup. The command must actually run the relevant operation for either effect to occur. This is useful lifecycle policy, but it is not a continuous watcher or a hard storage quota.

When it breaks

An age is not simply “time since the file was created” or “time since its contents last changed.” The documented default considers several timestamps, including access and modification, status change for files, and creation time where available. A recently changed qualifying timestamp can preserve an entry even when its mtime looks old. Rewriting timestamps for a test can therefore change the result in an unexpected direction.

--create and --clean perform different operations. Creating the configured directory does not prove that old contents have been removed. The d rule used below keeps its directory while making eligible contents subject to age cleanup. Other types have different removal semantics; do not substitute a recursive removal rule simply because both actions sound like cleaning.

An x exclusion preserves a matching directory and its contents during age-based cleanup, whereas X excludes the directory itself without the same recursive protection. These exclusions do not override explicit r or R removal rules. Read the effective configuration and operation together before deciding which paths a maintenance run can delete.

Check it quickly

Run this Python script as an administrator in a disposable Linux environment with systemd-tmpfiles. The recorded run used Debian 12 with systemd 252.39. It passes one explicit configuration file and a temporary --root, so all created objects and deletion targets belong to the exercise. The live system’s ordinary tmpfiles configuration is not used:

from pathlib import Path
from tempfile import TemporaryDirectory
import os,stat,subprocess,time

with TemporaryDirectory(prefix='xbcatm-tmpfiles-') as directory:
 root=Path(directory);config=root/'policy.conf'
 definition='d /work 0750 - - 1s -\nx /work/keep - - - - -\n'
 config.write_text(definition)
 def apply(action):
 result=subprocess.run(['systemd-tmpfiles','--root='+str(root),action,str(config)],capture_output=True,text=True,timeout=10)
 assert result.returncode==0,result.stderr
 apply('--create');work=root/'work'
 assert stat.S_IMODE(work.stat().st_mode)==0o750
 old=work/'old';old.write_text('fixture');past=time.time()-60;os.utime(old,(past,past))
 apply('--clean');assert old.exists()
 print('old atime and mtime alone do not satisfy the default age rule')
 config.write_text(definition.replace('1s','m:1s'));apply('--clean')
 assert not old.exists()
 print('explicit mtime-only age removes the same recently created file')
 config.write_text(definition);old.write_text('aged fixture')
 keep=work/'keep';keep.mkdir();(keep/'preserved').write_text('excluded fixture')
 outside=root/'outside';outside.write_text('outside cleanup tree')
 link=work/'link';link.symlink_to(outside)
 with old.open('rb') as retained:
 time.sleep(1.2)
 fresh=work/'fresh';fresh.write_text('recent fixture')
 apply('--clean')
 assert not old.exists() and not link.is_symlink()
 assert retained.read()==b'aged fixture'
 print('unlinked file remains readable through its already open descriptor')
 assert fresh.read_text()=='recent fixture'
 assert (keep/'preserved').read_text()=='excluded fixture'
 assert outside.read_text()=='outside cleanup tree'
 assert work.is_dir()
 print('aged entries removed; recent file and recursively excluded subtree remain')
 print('symlink removed without following it; target outside the cleaned tree survives')
print('all creation and cleanup occurred in a temporary alternate root')

The create operation produces /work inside the temporary root with mode 0750. A new file then receives deliberately old access and modification timestamps. Default cleanup leaves it present because other relevant timestamps are still recent. Merely backdating mtime and atime is not enough to make this file old under the complete default rule.

Changing only the age specification to m:1s makes modification time the selected file timestamp. Cleanup now removes that same recently created file. This proves that timestamp selection changes policy, not just display formatting. The script restores the default rule afterward; mtime-only aging is an intentional alternative to assess against the application’s access pattern, not an automatic improvement.

The next phase creates an ordinary file, an excluded subtree, and a symbolic link to another harmless file outside the cleaned tree. After waiting long enough for their timestamps to age, it creates a fresh file and runs cleanup. The aged ordinary entry and symlink disappear, while the recent file and excluded subtree remain. The symlink’s target is still intact, and the configured /work directory still exists.

The ordinary file is held open across cleanup. Although its pathname is gone, its existing descriptor can still read the original bytes until it is closed. Removing a directory entry and releasing the storage backing every open reference are different events. A process holding deleted files can consequently explain why a cleanup removed names without reclaiming all the expected space.

Assertions compare actual modes, path existence, and file contents. No test treats a zero exit status alone as proof that the intended deletion happened. The entire alternate root is removed afterward, leaving the host’s existing temporary files and normal policy untouched.

Match housekeeping to the producer

Inspect the distribution’s tmpfiles setup and cleanup units to establish when these rules are invoked. Eligibility at one second does not mean deletion occurs exactly one second later; the cleanup cadence and current workload matter. An inactive timer or failed cleanup command can leave eligible files in place indefinitely.

Age policies also cannot bound how many bytes a producer creates between cleanup runs. High write volume, recent accesses, exclusions, and open descriptors all affect retained space. Use appropriate storage limits and disk monitoring when capacity must be bounded, and diagnose the producing application instead of shortening retention until useful evidence disappears.

Keep service-owned runtime, cache, and persistent state conceptually separate. Data needed after restart should not accidentally inherit a temporary-file deletion policy. When managed service directories already express the required ownership and lifetime, reuse that mechanism. Add tmpfiles rules for a concrete lifecycle requirement and verify both eligible deletion and required preservation.

Sources

systemd 252 tmpfiles.d, creation types, timestamp-based aging, exclusions, and symbolic links, checked 2026-09-12.
systemd 252 systemd-tmpfiles, separate create/clean operations, alternate roots, and scheduled integration, checked 2026-09-12.

A systemd oneshot completes one activation

Sun, 28 Nov 2027 00:00:00 +0000

A Type=oneshot service represents finite work whose startup completes when its start commands finish successfully. That makes it useful for setup steps and scheduled jobs whose completion matters to a dependent unit. The word oneshot describes one activation’s process model. It does not create a permanent record that prevents the task from running again after a restart, another activation, or a reboot.

When it breaks

A successful oneshot normally becomes inactive when its commands finish. That is expected behavior for a completed task, not evidence that a daemon crashed. With RemainAfterExit=yes, the manager instead keeps the unit active after the commands exit. An active (exited) state can therefore have no main process at all. Monitoring that assumes every active service needs a live PID will misclassify this service type.

Multiple ExecStart entries run in sequence for a oneshot. Under ordinary success rules, a failing command prevents later start commands from running and marks the activation failed. Earlier commands may already have changed files or databases. systemd does not roll those effects back, so retrying the activation can repeat work that succeeded before the failure.

RemainAfterExit is manager state, not a durable exactly-once guarantee. Starting an already active unit does not rerun it, but restarting it does. A timer targeting an active oneshot also does not turn each elapsed interval into another execution. Choose the retained state only when it represents a real ongoing condition, and keep repeatable scheduled work inactive between runs.

Check it quickly

Run this Python script as an administrator in a disposable Linux environment managed by systemd. It creates one runtime unit and a temporary step program. The recorded run used Debian 12 with systemd 252.39. Ensure the example unit filename is unused. Each step records its name before the second step deliberately fails:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess

unit='xbcatm-oneshot-lab.service';path=Path('/run/systemd/system')/unit
assert not path.exists()
def ctl(*args,success=True):
 result=subprocess.run(['systemctl',*args],text=True,capture_output=True,timeout=10)
 assert (result.returncode==0)==success,(args,result.stderr)
 return result.stdout.strip()
def value(key):return ctl('show',unit,'-p',key,'--value')
with TemporaryDirectory(prefix='xbcatm-oneshot-',dir='/run') as directory:
 root=Path(directory);worker=root/'step.py';calls=root/'calls'
 worker.write_text("""from pathlib import Path
import sys
root=Path(__file__).parent
with (root/'calls').open('a') as output: output.write(sys.argv[1]+'\\n')
if sys.argv[1]=='second' and not (root/'healthy').exists(): sys.exit(7)
""")
 definition=f'''[Service]
Type=oneshot
RemainAfterExit=yes
TimeoutStartSec=3s
ExecStart=/usr/bin/python3 {worker} first
ExecStart=/usr/bin/python3 {worker} second
ExecStart=/usr/bin/python3 {worker} third
'''
 path.write_text(definition)
 try:
 subprocess.run(['systemd-analyze','verify',str(path)],check=True)
 ctl('daemon-reload');ctl('start',unit,success=False)
 assert calls.read_text().splitlines()==['first','second']
 assert value('Result')=='exit-code' and value('ExecMainStatus')=='7'
 print('second command fails; third command does not run')
 (root/'healthy').touch();ctl('reset-failed',unit);ctl('start',unit)
 assert calls.read_text().splitlines()==['first','second','first','second','third']
 assert (value('ActiveState'),value('SubState'),value('MainPID'))==('active','exited','0')
 before=calls.read_text();ctl('start',unit);assert calls.read_text()==before
 print('active/exited with PID 0; repeated start does not rerun')
 ctl('restart',unit)
 assert calls.read_text()==before+'first\nsecond\nthird\n'
 print('restart executes all commands again')
 ctl('stop',unit);path.write_text(definition.replace('RemainAfterExit=yes','RemainAfterExit=no'))
 ctl('daemon-reload');before=calls.read_text()
 for _ in range(2):
 ctl('start',unit)
 assert value('ActiveState')=='inactive' and value('Result')=='success'
 assert calls.read_text()==before+'first\nsecond\nthird\n'*2
 print('without RemainAfterExit: two starts, two successful executions')
 finally:
 ctl('stop',unit)
 if value('ActiveState')=='failed':ctl('reset-failed',unit)
 path.unlink();ctl('daemon-reload')

The initial activation records only first and second, returns a failing start command, and exposes exit status seven. The third command does not run. The existing log entries remain, which makes the partial effects visible instead of pretending that a failed activation left no work behind.

After the healthy marker is created, the retry records all three steps. The complete file now contains the two earlier entries followed by a fresh sequence of three. State becomes active and exited, with main PID zero. That combination means the manager retains a successful activation, not that the program remains running in the background.

A second start while active leaves the file unchanged. Restarting the same unit appends another complete sequence. This is the direct counterexample to interpreting oneshot as “execute this migration once for the lifetime of the machine.” The manager is doing exactly what the new activation requests.

The script then stops the unit, changes RemainAfterExit to no, reloads the manager, and starts it twice. Both executions succeed and both end inactive. The log gains two full sequences. This behavior fits a job that should run whenever triggered and complete before another trigger can start it again.

The unit has an explicit three-second startup timeout. Oneshot startup timeouts are disabled by default in the documented version, so a stuck command otherwise may hold an activation open indefinitely. Choose a realistic bound for actual work rather than copying this small laboratory timeout into a long migration.

Give the task its own correctness guarantees

Keep each start command finite and in the foreground. If the task launches a long-running worker, describe that worker with an appropriate service lifecycle instead of inferring continued operation from the parent’s successful exit. Unit type, command completion, and the business result should describe the same event.

For a migration, use the database’s migration ledger, transaction behavior, and concurrency controls to decide whether a change has already been applied. A retained active state in one manager cannot coordinate multiple hosts or survive every operational transition. Make retries safe at the layer where side effects occur, and preserve the result needed to diagnose partial completion.

Inspect the service’s recorded result and application output, not just whether a timer or dependency triggered it. A timer can be active while its last job failed. Separate installation, scheduling, and execution guarantees, and verify a failed path as well as the successful one before depending on the task during boot or release.

Sources

systemd 252 service types, oneshot command ordering, retained state, and startup timeouts, checked 2026-09-12.
systemd 252 systemctl, start, restart, and manager reload behavior, checked 2026-09-12.
systemd 252 timers, activation of an already active target service, checked 2026-09-12.

A veth pair carries frames between two endpoints; routing supplies their meaning

Sun, 21 Nov 2027 00:00:00 +0000

A veth pair consists of two interconnected virtual Ethernet devices. A frame transmitted through one is delivered to the other, where the receiving network stack or bridge handles it. The two ends can reside in separate Linux network namespaces, making the pair useful for container networking. Creating the pair supplies a link; addresses, routes, bridge membership, forwarding, and access policy are additional configuration.

When it breaks

An interface marked administratively UP can still lack carrier because its veth peer is down. Reading only the UP flag can therefore make an incomplete setup look ready. Inspect both endpoints and distinguish administrative state from LOWER_UP. A route may already name the interface even while the paired link cannot carry useful traffic, so a route listing alone is not a connectivity check.

The pair does not restrict IP payloads to the peer interface’s own address. Routing chooses which link should carry a packet, and Ethernet forwarding or the receiving stack decides what happens next. Traffic destined beyond a container’s immediate neighbor can traverse the pair toward a router or bridge. Saying that every packet not addressed to the peer stays local confuses link delivery with IP destination selection.

Endpoint lifetime is coupled too. Deleting one veth endpoint removes its peer, even if the peer is managed from another namespace. A cleanup action in the host-side context can therefore remove a container’s interface. Stopping traffic, removing routes, and deleting the pair need an intentional order when they affect a live workload; namespace separation does not make the two devices independent resources.

Check it quickly

Save this as check-veth.py and run python3 check-veth.py with Docker. The pinned Linux container uses Python 3.11.16 and iproute2 6.1.0, with NET_ADMIN and SYS_ADMIN for nested namespaces. It creates only disposable virtual devices. The left and right namespaces receive one endpoint each, with 192.0.2.1/30 and 192.0.2.2/30 respectively:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,'child exited without a response'
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found

def server(stack,name,address,port=18080,label='server'):
 code=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import json,socket,sys
address,port,label=sys.argv[1],int(sys.argv[2]),sys.argv[3]
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 body=json.dumps({'server':label,'peer':self.client_address,'path':self.path}).encode()
 self.send_response(200);self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
 def log_message(self,*args):pass
class Server(ThreadingHTTPServer):
 address_family=socket.AF_INET6 if ':' in address else socket.AF_INET
 def server_bind(self):
 if self.address_family==socket.AF_INET6:self.socket.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
 super().server_bind()
service=Server((address,port),Handler);print('ready',flush=True);service.serve_forever()
'''
 process=spawn(stack,['ip','netns','exec',name,'python3','-u','-c',code,address,str(port),label])
 assert line(process).strip()=='ready';return process
def request(name,address,port=18080):
 code=r'''
import http.client,json,sys
client=http.client.HTTPConnection(sys.argv[1],int(sys.argv[2]),timeout=.7)
try:
 client.request('GET','/lab-visible-message');response=client.getresponse();body=json.loads(response.read())
 assert response.status==200;print(json.dumps({'ok':True,'status':response.status,'body':body}))
except OSError as error:print(json.dumps({'ok':False,'type':type(error).__name__,'errno':error.errno}))
finally:client.close()
'''
 return json.loads(ns(name,'python3','-c',code,address,str(port)).stdout)

with lab(['left','right']) as (root,stack,setting):
 run('ip','link','add','left0','type','veth','peer','name','right0')
 run('ip','link','set','left0','netns','left');run('ip','link','set','right0','netns','right')
 ip('left','addr','add','192.0.2.1/30','dev','left0');ip('right','addr','add','192.0.2.2/30','dev','right0')
 ip('left','link','set','left0','up')
 before=json.loads(ip('left','-j','link','show','left0').stdout)[0]
 assert 'LOWER_UP' not in before['flags']
 failed=ns('left','ping','-n','-c','1','-W','1','192.0.2.2',check=False);assert failed.returncode!=0
 ip('right','link','set','right0','up')
 after={name:json.loads(ip(name,'-j','link','show',interface).stdout)[0] for name,interface in [('left','left0'),('right','right0')]}
 assert all('LOWER_UP' in v['flags'] for v in after.values())
 ip('left','neigh','flush','dev','left0');ip('right','neigh','flush','dev','right0')
 successful=ns('left','ping','-n','-c','1','-W','1','192.0.2.2');assert successful.returncode==0
 daemon=server(stack,'right','192.0.2.2',label='right');response=request('left','192.0.2.2');assert response['ok']
 ip('right','addr','add','203.0.113.9/32','dev','lo');ip('left','route','add','203.0.113.9/32','via','192.0.2.2')
 server(stack,'right','203.0.113.9',label='beyond-link-address');routed_response=request('left','203.0.113.9');assert routed_response['ok']
 ip('left','link','delete','left0');gone=ip('right','link','show','right0',check=False);assert gone.returncode!=0 and 'does not exist' in gone.stderr
 print(json.dumps({'one_end_up_flags':before['flags'],'one_end_up_ping_exit':failed.returncode,
 'both_up_flags':{n:v['flags'] for n,v in after.items()},'both_up_ping_exit':successful.returncode,'http':response,'other_ip_destination':routed_response,
 'deleting_one_removes_peer':True,'peer_absent_diagnostic':gone.stderr.strip()}))
"""
name='xbcatm-netpath-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

Initially only left0 is brought up. Its flags include UP and NO-CARRIER, with no LOWER_UP, because right0 remains down. A one-packet ping fails. This gives both a link-state observation and an actual unsuccessful exchange; neither is replaced by assuming that creation of the interface means the other endpoint has been configured correctly.

After bringing right0 up, both endpoints report LOWER_UP. The script clears neighbor entries from the failed attempt before making the healthy comparison, avoiding residual neighbor-resolution timers. A new ping succeeds, and a real HTTP request from left to 192.0.2.2 returns the right service’s identity. The cleared state is explicit; the exercise does not promise instantaneous recovery for every already-pending application request.

Next, the right namespace adds 203.0.113.9/32 on its loopback interface. The left namespace installs a host route to that address through 192.0.2.2. A second HTTP listener bound to 203.0.113.9 responds successfully across the same veth pair. The packet’s IP destination is not the peer’s link address, yet its selected path still uses that virtual Ethernet link.

This second destination is local to the right namespace, so the fixture does not claim to forward onward through another router. It isolates the distinction between a link’s neighboring address and a packet’s final IP destination. A larger topology can connect the receiving endpoint to a bridge or routed network, but those additional components need their own configuration and verification.

Finally, deleting left0 makes ip link show right0 fail in the right namespace with a device-not-found diagnostic. The peer has been removed, not merely placed administratively down. Both namespaces themselves still exist until cleanup. This separates the paired device lifecycle from the lifetime of the namespace that happened to contain one endpoint.

Diagnose the layers in order

Confirm which namespace owns each end and which names the runtime assigned. Then inspect administrative and carrier state, address configuration, neighbor resolution, and the route for the failing destination. A stale neighbor entry can delay recovery after a link change, while an absent route can fail before any frame is transmitted. These conditions require different repairs even when a high-level client reports only a connection timeout.

A pair is not a multiport switch. A bridge can join several such links at Ethernet level, while a router forwards between IP networks. Enabling an endpoint does not automatically add either behavior or an internet route. Document the actual attachment and filtering points, especially when a container runtime creates the devices and another component manages routes or firewall policy.

Treat deletion as a change to both endpoints and verify the affected workload afterward. If throughput or latency motivates tuning MTU, queues, or offload behavior, measure the relevant path rather than inferring performance from the link being virtual. This exercise verifies paired delivery, link readiness, destination routing, and coupled removal.

Sources

Linux veth(4), paired transmission, carrier state and namespace use, checked 2026-09-12.
Linux ip-link(8), veth creation, link state, movement and deletion, checked 2026-09-12.
Linux network_namespaces(7), network boundaries and virtual-device lifetime, checked 2026-09-12.

A page fault is not always a disk read

Sat, 13 Nov 2027 00:00:00 +0000

Treating all faults as slow conflates the cheap and the expensive, which is why the split between minor and major matters when you tune memory-heavy services.

Minor faults map a page

A minor page fault happens when the page is already resident in RAM but the process does not yet have it mapped. The kernel only has to wire up a page-table entry — pointing the virtual address at physical memory that is already there. Common cases: first touching a freshly allocated zeroed region, mapping a shared library that another process already loaded, or a page that was swapped in once and sits in memory. Minor faults are cheap: a few hundred nanoseconds, no disk, no IO. They happen constantly in normal-running processes and are not, by themselves, a performance problem.

Major faults read the disk

A major fault means the data has to come from disk (or a page-in from swap) before the mapping can complete, which is orders of magnitude more expensive — commonly interpreted as reading a few hundred microseconds to several milliseconds. Majors are the ones that dominate cold-start costs: warming a process, reading a big binary for the first time, or evicting from swap all surface as a burst of major faults. The distinction that matters is that a major fault proves the working set did not locate your reference, so it also scores as what your prefetch and/or cache should have made unnecessary.

Count them separately

Tools split the two for a reason, because telling them apart changes the diagnosis. ps, sar, and /proc/self/stat (minor/major) report them separately; a healthy server has a constant background of minors and few majors. Several things give away the real story: a workload with many majors is suffering from cache misses and slow startup, while a workload with drastic minor spikes but few majors is usually bad page-table churn, not disk pressure at all. Look at them separately, and when majors rise, the fix is about cache/pread, not the kernel.

When it breaks

A major-fault storm looks like latency but behaves like cold cache. After a deploy, a container restart, or a cold node, every worker touches its binary, libraries, and data for the first time at once. perf stat -e minor-faults,major-faults shows majors jumping from near zero to thousands per second while CPU stays oddly idle, and p99 stretches into hundreds of milliseconds. The fix is warming, not more CPU: pre-touch the working set or stagger restarts so processes do not fault the same pages at once.

Swap makes the same signal resonate. When the working set no longer fits in RAM, reclaimed pages fault back from disk on next touch. Majors rise with swap-in, load climbs, and every request pays disk time though the code did not change. Adding threads only grows the working set and deepens the thrash. Fit the hot set in RAM or move cold data off the request path.

A third failure is misreading mmap. Scanning a large file-backed mapping still faults each page in on first touch, so a dashboard full of majors does not mean random I/O is broken. Check residency with mincore before tuning readahead or blaming storage.

Check it quickly

Save the following as fault_probe.py and run python3 fault_probe.py. It uses only the Python standard library, needs no privileges, and runs on Linux or macOS. It snapshots resource.getrusage minor and major counters around two passes over a fresh anonymous mapping, timing each pass with time.perf_counter.

import mmap
import resource
import time

PAGE = 4096
PAGES = 20000 # about 80 MiB: large enough to see, small enough for CI


def counters():
 usage = resource.getrusage(resource.RUSAGE_SELF)
 return usage.ru_minflt, usage.ru_majflt


def touch(mapping):
 for offset in range(0, len(mapping), PAGE):
 mapping[offset] = 1


def main():
 mapping = mmap.mmap(-1, PAGES * PAGE)
 base_min, base_maj = counters()
 start = time.perf_counter()
 touch(mapping)
 first_time = time.perf_counter() - start
 mid_min, mid_maj = counters()

 start = time.perf_counter()
 touch(mapping) # pages are already mapped here
 second_time = time.perf_counter() - start
 end_min, end_maj = counters()
 mapping.close()

 first_min = mid_min - base_min
 first_maj = mid_maj - base_maj
 second_min = end_min - mid_min
 second_maj = end_maj - mid_maj
 print(f"first touch: minflt={first_min} majflt={first_maj} time={first_time:.4f}s")
 print(f"second touch: minflt={second_min} majflt={second_maj} time={second_time:.4f}s")
 assert first_min > 1000, f"expected thousands of minor faults, got {first_min}"
 assert first_maj == 0, f"anonymous first touch should not cause majors, got {first_maj}"
 assert second_min < first_min // 10, f"mapped pages should barely fault: {second_min} vs {first_min}"
 assert second_maj == 0, f"unexpected major faults on second pass: {second_maj}"
 print("PASS: first touch faults as minors without disk reads; second touch is almost fault-free")


if __name__ == "__main__":
 main()

The first pass touches every page once, so expect a large minflt jump of several thousand, almost no majflt change, and a slower elapsed time. The second pass revisits mapped pages, so expect near-zero new faults and a shorter time. That contrast is the point: anonymous first touch costs minor faults, not disk reads.

The assertions passed with Python 3.12 on macOS and Linux. The probe covers anonymous memory only, so it cannot locate a file-backed major, nor replace perf stat -e minor-faults,major-faults on production. Enlarge the mapping if the time gap is too small to see.

Sources

Linux man page proc(5), per-process minor and major fault counters, checked 2026-06-25.
Linux man page mmap(2), demand paging with MAP_PRIVATE and MAP_ANONYMOUS, checked 2026-09-12.
Brendan Gregg: perf examples, counting minor-faults and major-faults with perf stat, checked 2026-09-12.
Kernel documentation: concepts overview, demand paging, page cache, and reclaim, checked 2026-09-12.

Socket activation passes a listener to a compatible service

Mon, 08 Nov 2027 00:00:00 +0000

Socket activation lets systemd open a listening endpoint before the application process starts. Activity on that endpoint can then trigger a compatible service, which receives descriptors for sockets the manager already owns. The useful boundary is listener ownership versus application lifetime. An open endpoint is not yet proof that the application has initialized, accepted a request, or produced a correct response.

When it breaks

The application must understand the activation protocol. A daemon that unconditionally binds the same endpoint again can fail because the manager already occupies it. Activation moves binding responsibility into the socket unit; it does not eliminate address conflicts, missing interfaces, filesystem permission errors, or other reasons that initial bind can fail.

With Accept=no, the service receives the listening socket and handles acceptance itself. Accept=yes instead supports a per-connection service-instance model for suitable stream sockets. These are different application contracts. Neither mode implies that an arbitrary daemon will automatically become idle, exit after work, or restart only when doing so is safe.

Queued connections also have limits. Clients can establish transport state before application startup finishes, then time out waiting for a response. Socket activation does not make database dependencies ready or create an unlimited request queue. Keep explicit service dependencies, readiness behavior, and client timeouts appropriate to the real workload.

Check it quickly

Run this Python script as an administrator in a disposable Linux system whose PID 1 is systemd. The recorded environment was Debian 12 with systemd 252.39. It creates a unique socket/service pair using a Unix stream socket in a temporary directory. No public TCP port is exposed. The worker intentionally serves one newline-terminated message and exits:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,socket,subprocess,time,uuid

name='xbcatm-socket-'+uuid.uuid4().hex[:8];service=name+'.service';unit=name+'.socket'
paths={n:Path('/run/systemd/system')/n for n in (service,unit)}
assert all(not p.exists() for p in paths.values())
def run(*args):
 result=subprocess.run(args,capture_output=True,text=True,timeout=10)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
def value(key):return run('systemctl','show',service,'-p',key,'--value')
def wait_inactive():
 deadline=time.monotonic()+5
 while value('ActiveState')!='inactive':
 assert time.monotonic()<deadline,'service did not exit';time.sleep(0.02)
with TemporaryDirectory(prefix=name+'-',dir='/run') as directory:
 root=Path(directory);endpoint=root/'listener.sock';worker=root/'worker.py'
 worker.write_text(r'''from pathlib import Path
import errno,json,os,socket
root=Path(__file__).parent
assert int(os.environ['LISTEN_PID'])==os.getpid()
assert int(os.environ['LISTEN_FDS'])==1
with socket.socket(fileno=3) as listener:
 assert listener.getsockopt(socket.SOL_SOCKET,socket.SO_ACCEPTCONN)==1
 with socket.socket(socket.AF_UNIX,socket.SOCK_STREAM) as duplicate:
 try:duplicate.bind(str(root/'listener.sock'))
 except OSError as error:assert error.errno==errno.EADDRINUSE
 else:raise AssertionError('unexpected duplicate bind success')
 connection,_=listener.accept()
 with connection,connection.makefile('rb') as incoming:
 message=incoming.readline(256);assert message.endswith(b'\n')
 connection.sendall(b'ack '+message)
 (root/(str(os.getpid())+'.json')).write_text(json.dumps({'pid':os.getpid(),'fd':3,'message':message.decode().strip()}))
''')
 paths[unit].write_text(f'[Socket]\nListenStream={endpoint}\nAccept=no\nSocketMode=0600\nRemoveOnStop=yes\n')
 paths[service].write_text(f'[Service]\nType=exec\nExecStart=/usr/bin/python3 {worker}\n')
 try:
 run('systemd-analyze','verify',*[str(p) for p in paths.values()]);run('systemctl','daemon-reload')
 run('systemctl','start',unit)
 assert endpoint.is_socket() and value('ActiveState')=='inactive' and value('MainPID')=='0'
 print('socket listening while service has no process')
 for message in ('first','second'):
 with socket.socket(socket.AF_UNIX,socket.SOCK_STREAM) as client:
 client.settimeout(5);client.connect(str(endpoint));client.sendall((message+'\n').encode())
 with client.makefile('rb') as response:assert response.readline()==('ack '+message+'\n').encode()
 wait_inactive();assert endpoint.is_socket() and value('Result')=='success'
 records=[json.loads(p.read_text()) for p in root.glob('*.json')]
 assert len(records)==2 and len({row['pid'] for row in records})==2
 assert {row['message'] for row in records}=={'first','second'}
 assert all(row['fd']==3 for row in records)
 print('two requests: two deliberate one-request workers; each uses inherited listening fd 3')
 print('rebinding the occupied socket fails; manager retains listener between workers')
 run('systemctl','stop',unit);assert not endpoint.exists()
 print('stopping socket removes its endpoint with RemoveOnStop=yes')
 finally:
 run('systemctl','stop',unit,service)
 for name,path in paths.items():
 if run('systemctl','show',name,'-p','ActiveState','--value')=='failed':run('systemctl','reset-failed',name)
 path.unlink()
 run('systemctl','daemon-reload')

After starting only the socket unit, the endpoint exists while the corresponding service is inactive and has no main PID. A first client connection then causes the worker to execute. The client sends a complete framed message and checks an application reply, rather than treating a successful connect call as proof of usable service.

The worker verifies that LISTEN_PID names its own process and LISTEN_FDS reports one inherited descriptor. It wraps descriptor three and checks that it is already listening. The numbering and environment checks match the native activation contract. A production program handling multiple descriptors should also identify and validate the sockets it actually expects.

Attempting another bind to the same Unix socket path produces EADDRINUSE. The worker continues using the inherited listener and returns the expected reply. This controlled failure demonstrates why an application cannot simply retain its ordinary bind path while assuming activation will somehow avoid the conflict.

After the worker exits successfully, the manager still holds the listening endpoint. A second connection launches another worker, which handles the second message and exits. The recorded PIDs are different and both invocations used descriptor three. Two processes appear because this specific worker deliberately exits after one request; Accept=no did not itself require one process per connection.

Stopping the socket unit removes the endpoint because the fixture explicitly sets RemoveOnStop=yes. The script then stops any remaining test units and removes their runtime definitions and temporary files. This cleanup behavior is configured and observed, not a universal promise that every socket path disappears whenever any associated daemon exits.

Design the handoff and shutdown together

For a long-running daemon, accept and serve the inherited sockets using its documented integration, and let its normal shutdown behavior decide when descriptors close. For a daemon intended to leave when idle, verify the idle-exit logic and what happens to in-flight requests. systemd cannot infer when an application’s work is safely complete.

Stopping future activation and terminating current work are separate operational decisions. Inspect both socket and service units during maintenance. A service may be started explicitly through another dependency or by an operator, and a still-active listener may trigger it again. Test the deployed unit relationships rather than assuming that stopping one visible process permanently disables the endpoint.

Set appropriate endpoint permissions and service identity. Passing an already-open descriptor allows listener setup and application privileges to differ, but it does not make the service a sandbox. Verify real request handling, restart behavior, queued-client outcomes, and failure diagnostics. The practical advantage is a deliberate, testable handoff between listener and process, not automatic readiness or zero resource cost.

Sources

systemd 252 systemd.socket, listener ownership, Accept modes, activation, and endpoint removal, checked 2026-09-12.
systemd 252 sd_listen_fds, inherited descriptor numbering, count, and process-identity checks, checked 2026-09-12.
systemd 252 systemd.service, service startup types and process lifecycle, checked 2026-09-12.

systemd restart policies need explicit failure and rate rules

Sun, 07 Nov 2027 00:00:00 +0000

A restart policy decides whether a service manager should try another process after the current one ends. It can recover from a transient failure, but it cannot correct a missing configuration or a deterministic bug. The useful contract has three parts: which endings qualify, how long to wait, and how many starts are permitted within a time window. Those settings answer different questions.

When it breaks

Restart=on-failure does not mean restart after every signal. For service types other than oneshot, ordinary SIGHUP, SIGINT, SIGTERM, and SIGPIPE terminations are normally considered clean. A nonzero exit or an unclean signal such as SIGKILL qualifies under the default success rules. SuccessExitStatus and the restart-specific exception directives can change classification, so inspect the effective unit.

on-abnormal is not simply a stricter synonym for on-failure: it excludes ordinary nonzero process exit codes while covering abnormal signals and certain timeout or watchdog failures. Choose according to how the program reports recoverable and intentional endings. A process that remains alive but stops doing useful work needs a separately configured detection mechanism; exit-based restart logic cannot infer that condition.

RestartSec supplies a delay before restarting. In the systemd 252 exercise below, it is a fixed delay, not an exponentially increasing backoff. StartLimitIntervalSec and StartLimitBurst limit starts within an interval, including manual starts. They are not a lifetime retry counter, and reaching the limit does not schedule an automatic recovery attempt at the next interval boundary.

Check it quickly

Run this Python script as an administrator in a disposable Linux systemd environment. It creates a uniquely named runtime unit and a temporary worker, then removes them. The recorded run used Debian 12 with systemd 252.39. The worker initially exits with status seven and records every actual invocation:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess,time,uuid

unit='xbcatm-restart-lab-'+uuid.uuid4().hex[:8]+'.service'
path=Path('/run/systemd/system')/unit
assert not path.exists()
def ctl(*args):
 result=subprocess.run(['systemctl',*args],text=True,capture_output=True,timeout=10)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
def value(key):return ctl('show',unit,'-p',key,'--value')
def wait_for(check):
 deadline=time.monotonic()+5
 while not check():
 assert time.monotonic()<deadline,'state transition timed out'
 time.sleep(0.02)
with TemporaryDirectory(prefix='xbcatm-restart-',dir='/run') as directory:
 root=Path(directory);worker=root/'worker.py';calls=root/'calls';healthy=root/'healthy'
 worker.write_text("""from pathlib import Path
import sys,time
root=Path(__file__).parent
with (root/'calls').open('a') as output: output.write('started\\n')
if not (root/'healthy').exists(): sys.exit(7)
time.sleep(60)
""")
 path.write_text(f'''[Unit]
StartLimitIntervalSec=10s
StartLimitBurst=3
[Service]
Type=exec
ExecStart=/usr/bin/python3 {worker}
Restart=on-failure
RestartSec=100ms
''')
 try:
 ctl('daemon-reload');ctl('start',unit)
 wait_for(lambda: value('ActiveState')=='failed')
 assert calls.read_text().splitlines()==['started']*3
 assert value('Result')=='exit-code'
 subprocess.run(['journalctl','--sync'],check=True)
 logs=subprocess.run(['journalctl','-u',unit,'--no-pager','-o','cat'],capture_output=True,text=True,check=True).stdout
 assert 'Start request repeated too quickly' in logs
 print('three starts; failed/exit-code with start-rate-limit journal message')
 healthy.touch();ctl('reset-failed',unit);ctl('start',unit)
 wait_for(lambda: len(calls.read_text().splitlines())==4)
 first=value('MainPID');assert first!='0'
 ctl('kill','--kill-who=main','--signal=TERM',unit)
 wait_for(lambda: value('ActiveState')=='inactive')
 assert value('NRestarts')=='0'
 print('on-failure treats ordinary SIGTERM as clean; no restart')
 ctl('start',unit);first=value('MainPID')
 ctl('kill','--kill-who=main','--signal=KILL',unit)
 wait_for(lambda: value('MainPID') not in ('0',first))
 assert value('NRestarts')=='1'
 print('SIGKILL triggers one restart')
 ctl('stop',unit);time.sleep(0.2)
 assert value('ActiveState')=='inactive'
 print('explicit stop remains stopped')
 finally:
 ctl('stop',unit)
 if value('ActiveState')=='failed':ctl('reset-failed',unit)
 path.unlink();ctl('daemon-reload')

The first phase permits three starts in ten seconds with a short fixed restart delay. The worker’s own file contains exactly three entries, so the count represents actual process launches rather than a counter whose meaning was assumed. The manager then leaves the unit failed and records that the start request was repeated too quickly.

On the tested version, the terminal Result remains exit-code, reflecting the preceding process failure, even though the journal identifies start-rate limiting. Requiring a literal start-limit-hit result would miss this observed case. Use state, application invocation evidence, and the corresponding journal message together rather than relying on one expected diagnostic label across versions and failure paths.

The script repairs the fixture by creating its healthy marker, clears the failed state and rate counter, and starts it again. This time the process remains alive. Sending an ordinary SIGTERM produces an inactive unit with no automatic restart under on-failure. A later SIGKILL does cause one replacement process.

Finally, an explicit systemctl stop leaves the service inactive after the restart delay. Manager-requested shutdown is not treated as a crash that policy should immediately reverse. The cleanup handles failed state only when present, since an inactive unreferenced unit can be unloaded by the manager.

Recover the service, not just its counter

Read the failure before resetting its evidence. Inspect the configured command, exit status, recent invocation logs, and any dependency errors. Correct the cause or deploy a known working version, then reset the limiter if an immediate retry is justified. Repeatedly clearing the counter without changing the failure simply converts a bounded crash loop into an operator-driven one.

Alert on sustained unavailability and excessive restarts using the monitoring system actually responsible for the service. The unit settings shown here do not create an alert destination. Retain enough diagnostic context to distinguish failed execution, an application exit, resource exhaustion, and a deliberate stop without logging sensitive configuration.

Select the delay and burst to match startup cost and dependency behavior. A rapid retry may recover quickly from one fault but amplify load during a shared outage. Verify both recovery and termination in the deployed configuration, including any drop-ins. A policy is useful when the application resumes useful work and remains controllable, not merely when a PID repeatedly changes.

Sources

systemd 252 service restart rules, exit classification, fixed delay, and restart exceptions, checked 2026-09-12.
systemd 252 unit start limits, interval limits, manual starts, and counter reset, checked 2026-09-12.
systemd 252 systemctl, stop, state inspection, and reset-failed operations, checked 2026-09-12.

Read procfs as documented views of a changing process

Fri, 29 Oct 2027 00:00:00 +0000

Linux procfs exposes process and kernel information through filesystem interfaces. Many entries generate their contents when read instead of storing ordinary disk-file bytes. Each entry has a documented meaning, permission model, and format. Reading several entries does not freeze the process or create a transactionally consistent snapshot, so useful observations must be tied to the right process instance and interpretation.

When it breaks

Ordinary file assumptions can fail immediately. A proc entry can report size zero yet return useful content when read. A per-process directory can disappear as the process exits, and a numerical PID can eventually identify another instance. Treating a failed read as proof of a broken installation, or assuming a PID forever names one workload, can misdiagnose normal lifecycle changes.

Formats require entry-specific parsing. status contains labeled fields, while stat uses a positional layout with a parenthesized command name that can contain spaces and closing parentheses. Splitting the whole stat line on whitespace can shift the expected fields. cmdline and environ use NUL separators, so ordinary line or space splitting loses argument and value boundaries.

The views also have semantic limits. environ describes the environment region established at program execution, not a guaranteed live dictionary of every later environment change. Command-line contents can be changed by the process. A descriptor link ending in (deleted) does not mean the open object has vanished. These entries are evidence about specific state, not interchangeable proofs of identity or access.

Check it quickly

Save this as procview.py and run python3 procview.py in a disposable Linux environment. The recorded run used Linux 6.12.76 and Python 3.11.2. The script creates one controlled child with a small, explicitly supplied environment, reads only its own and that child’s proc entries, and performs no writes to kernel tuning interfaces:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,subprocess,sys

worker=r'''
from pathlib import Path
import ctypes,json,os,sys
assert ctypes.CDLL(None).prctl(15,b'worker ) sample',0,0,0)==0
os.environ['XB_PROC_TEST']='changed-after-start'
path=Path(sys.argv[1]);path.write_bytes(b'fixture-data')
with path.open('rb') as handle:
 info=os.fstat(handle.fileno());path.unlink()
 print(json.dumps({'fd':handle.fileno(),'device':info.st_dev,'inode':info.st_ino,'current_environment':os.environ['XB_PROC_TEST']}),flush=True)
 assert sys.stdin.readline()=='done\n'
'''
with TemporaryDirectory(prefix='xbcatm-proc-') as directory:
 path=Path(directory)/'data'
 process=subprocess.Popen([sys.executable,'-c',worker,str(path),'argument with spaces'],stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True,env={'PATH':'/usr/bin:/bin','XB_PROC_TEST':'fixture-value'})
 try:
 metadata=json.loads(process.stdout.readline());root=Path('/proc')/str(process.pid)
 status_path=root/'status';status_text=status_path.read_text()
 assert status_path.stat().st_size==0 and len(status_text)>0
 status=dict(line.split(':',1) for line in status_text.splitlines() if ':' in line)
 assert int(status['Pid'])==process.pid and int(status['PPid'])==os.getpid()
 raw=(root/'stat').read_text();pid_text,rest=raw.split(' (',1);comm,fields=rest.rsplit(') ',1)
 fields=fields.split();assert int(pid_text)==process.pid and comm=='worker ) sample'
 assert int(fields[1])==os.getpid() and int(fields[19])>0
 assert raw.split()[2] not in ('R','S','D','T','Z')
 print('proc files can have size zero but readable data; stat comm can contain spaces and a closing parenthesis')
 args=(root/'cmdline').read_bytes().rstrip(b'\0').split(b'\0')
 environment=(root/'environ').read_bytes().rstrip(b'\0').split(b'\0')
 assert args[-1]==b'argument with spaces' and b'XB_PROC_TEST=fixture-value' in environment
 assert metadata['current_environment']=='changed-after-start'
 assert os.readlink('/proc/self')==str(os.getpid())
 print('cmdline and environ use NUL separators; environ retains the initial value here; self resolves for the reader')
 entry=root/'fd'/str(metadata['fd']);target=os.readlink(entry);actual=entry.stat()
 assert target==str(path)+' (deleted)' and not path.exists()
 assert (actual.st_dev,actual.st_ino)==(metadata['device'],metadata['inode'])
 assert entry.read_bytes()==b'fixture-data'
 print('a deleted pathname remains inspectable through an open descriptor subject to access checks')
 output,error=process.communicate('done\n',timeout=5)
 assert process.returncode==0,error
 assert not root.exists()
 print('after the controlled child exits and is reaped, its proc directory disappears')
 finally:
 if process.poll() is None:process.kill();process.wait()

The child sets its short process name to worker ) sample, opens a temporary file, and unlinks its pathname while keeping the descriptor open. It then reports only fixture metadata and waits. This gives the controller a known live process whose name deliberately exercises the stat-format edge case.

The controller finds that status has reported size zero but nonempty readable contents. It checks labeled PID and parent PID fields. For stat, it separates the PID and parenthesized name before interpreting the remaining positional fields, confirming the parent PID and a positive start-time field. Naive whitespace splitting would mistake part of the name for a later field.

The final command-line argument contains spaces and remains one argument after NUL-based parsing. The initial environment contains XB_PROC_TEST=fixture-value, even though the child has changed its application environment value after startup. Both observations use harmless fixture data. They demonstrate why whitespace parsing and assuming a live environment map can produce incorrect conclusions.

Reading /proc/self in the controller resolves to the controller’s PID. Reading the child’s descriptor link returns the removed pathname with the deleted suffix, but following that proc descriptor entry still exposes the expected device/inode pair and fixture bytes under the experiment’s permissions. The missing ordinary pathname and the surviving open object are distinct facts.

After receiving the controller’s message, the child closes its handle and exits. The controller waits for completion and confirms that this child’s proc directory disappears. The example establishes a controlled before-and-after lifecycle observation; it does not promise atomic consistency among arbitrary proc reads while a busy application changes its state.

Match the view to the question

Use the relevant entry’s documented parser and units. A stat field number should be interpreted only after handling the command-name field correctly; a status label should not be assigned an assumed line number. If correlating samples, include process start time and boot context as appropriate so a reused numeric PID is not silently treated as the same instance.

Access and namespace context matter. Proc mount options, credentials, and ptrace-related checks can restrict visibility. /proc/net reflects the relevant network namespace, and a container’s visible proc data is not automatically its configured resource budget. Confirm which namespace and resource interface answer the operational question before labeling an observation as host-wide or container-specific.

Some proc entries are writable controls, but their scope differs. /proc/sys exposes tunables, while /proc/PID/oom_score_adj adjusts a particular process’s OOM preference; writing integers to status is not the interface for that task. Keep observation separate from a deliberate configuration change, and inspect the actual target and documented permissions before applying any write.

Sources

proc(5), procfs structure, per-process visibility, and mount options, checked 2026-09-12.
proc_pid_stat(5), field layout, parenthesized command name, and start time, checked 2026-09-12.
proc_pid_environ(5), initial environment region and NUL-separated entries, checked 2026-09-12.
proc_pid_fd(5), descriptor links, referenced objects, and permission checks, checked 2026-09-12.
proc_pid_cmdline(5), argument separators and process-controlled command-line contents, checked 2026-09-12.

The systemd journal needs an explicit retention contract

Tue, 26 Oct 2027 00:00:00 +0000

The systemd journal stores messages with structured metadata in journal files. That makes service output searchable without first inventing a parser for every application’s text format. Keeping those messages for a useful period is a separate requirement. Storage location, size policy, rotation, and forwarding determine which evidence remains available when an incident is investigated later.

When it breaks

The default namespace normally uses Storage=auto: persistent storage is selected when /var/log/journal exists, otherwise runtime storage is used. Storage=volatile writes beneath /run/log/journal; this is not simply the kernel’s text ring buffer. Those runtime files normally disappear across a machine reboot, but restarting the journald service is a different event and need not erase them.

Storage=persistent prefers /var/log/journal, with runtime storage used during early boot or when persistent storage cannot be written. A flush operation transfers runtime data under the documented conditions. Inspect actual files and effective configuration instead of inferring location solely from the presence of an option in one configuration fragment.

journald has default size limits and rotation behavior. It does not grow without bounds merely because no administrator wrote a rotation job. However, SystemMaxUse is not an exact hard quota that guarantees the filesystem can never fill. Runtime storage has separate settings, free-space constraints also apply, other programs consume space, and only archived journal files can be deleted during vacuuming.

Check it quickly

Run this Python script as an administrator in a disposable Linux system managed by systemd. The recorded environment was Debian 12 with systemd 252.39. It creates a unique journal namespace, gives it an explicit persistent-storage configuration, emits two harmless messages through transient services, and removes only that namespace’s data afterward. The vacuum operations deliberately delete the experiment’s old message:

from pathlib import Path
import json,shutil,subprocess,uuid

name='xbcatm-journal-'+uuid.uuid4().hex[:8]
config=Path('/etc/systemd')/('journald@'+name+'.conf')
directory=Path('/var/log/journal')/(Path('/etc/machine-id').read_text().strip()+'.'+name)
assert not config.exists() and not directory.exists()
def run(*args):
 result=subprocess.run(args,text=True,capture_output=True,timeout=15)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout
def journal(*args):return run('journalctl','--namespace='+name,*args)
def emit(message):
 run('systemd-run','--quiet','--wait','--collect','--unit='+name+'-'+message,
 '--property=LogNamespace='+name,'--property=StandardOutput=journal',
 '--property=SyslogIdentifier='+name,'/usr/bin/printf',message+'\n')
 journal('--sync')
def messages():
 rows=journal('--no-pager','-o','json','SYSLOG_IDENTIFIER='+name)
 return [json.loads(line)['MESSAGE'] for line in rows.splitlines()]
config.write_text('[Journal]\nStorage=persistent\nSystemMaxUse=16M\nRuntimeMaxUse=8M\n')
try:
 emit('old');journal('--flush');journal('--sync')
 assert messages()==['old'] and list(directory.glob('*.journal'))
 run('systemctl','restart','systemd-journald@'+name+'.service')
 assert messages()==['old']
 print('persistent namespace: message survives journald service restart')
 journal('--vacuum-size=1')
 assert messages()==['old']
 print('vacuum cannot remove the active journal file to satisfy a one-byte target')
 journal('--rotate');emit('new')
 assert messages()==['old','new']
 assert list(directory.glob('*@*.journal'))
 journal('--vacuum-size=1')
 assert messages()==['new'] and not list(directory.glob('*@*.journal'))
 print('after rotation: archived old message removed; active new message remains')
 print('persistent files observed under /var/log/journal; no reboot durability claim')
finally:
 run('systemctl','stop','systemd-journald@'+name+'.service',
 'systemd-journald@'+name+'.socket','systemd-journald-varlink@'+name+'.socket')
 config.unlink(missing_ok=True)
 for path in (directory,Path('/run/log/journal')/directory.name,Path('/run/systemd')/('journal.'+name)):
 if path.exists():shutil.rmtree(path)

The first message is present in journal files beneath the namespace’s /var/log/journal directory. After restarting that namespace’s journald service, the message is still queryable. This verifies the selected storage path and survival across a daemon restart. It does not simulate a power failure, prove storage-device durability, or test a complete machine reboot.

Vacuuming to a one-byte target initially leaves the message visible because its file is still active. The requested size is too small, but vacuum is not permitted to delete that active file just to meet the number. This provides a concrete counterexample to treating a vacuum target as a guaranteed final disk-usage figure.

The script then rotates the journal and writes a new message. Rotation closes the earlier file as an archive and opens a replacement. Both messages are readable before the second vacuum. Afterward, the archived old message is gone while the new message in the active file remains. Rotation and deletion are separate operations with different effects on evidence.

All journal commands select the unique namespace, so they do not vacuum the system’s default journal. Namespace service and socket instances are stopped before their test directories are removed. A normal query without the namespace selector would look in the default namespace and could miss these records entirely, even though they were successfully stored.

Size the evidence window

Choose retention from the incident-response window and observed message volume. A size limit cannot promise a fixed number of days when traffic changes. A retention-age limit likewise does not reserve enough bytes to preserve every message until that age. Monitor actual disk use and the earliest available records, especially after an unusually noisy failure.

Log collection can also lose messages before retention becomes relevant. Rate limits, application output routing, permissions, and unavailable storage affect what reaches the journal. Preserve the distinction between “no matching stored record” and “the event never occurred.” Avoid putting credentials into messages merely because local journal access is restricted.

If records must survive loss of the machine, provide and verify an independent collection path. Forwarding configuration alone is not evidence that a remote destination received or retained the message. Check delivery, access control, and the remote retention window using an identifiable harmless event. Local searchable logs and durable external evidence answer related but different operational needs.

Sources

systemd 252 journald.conf, storage selection, size defaults, free-space limits, and retention, checked 2026-09-12.
systemd 252 journalctl, namespace selection, rotation, synchronization, and archive vacuuming, checked 2026-09-12.
systemd 252 systemd-journald.service, journal namespaces, transport, and persistent file locations, checked 2026-09-12.

Inotify reports changes, but it is not a complete history

Wed, 13 Oct 2027 00:00:00 +0000

Inotify delivers Linux filesystem notifications through an instance with registered watches and a readable event queue. Each record identifies a watch, event mask, and possibly a name and move cookie. This can avoid repeatedly checking unchanged paths, but the stream is not an unlimited audit log. Correct consumers interpret notifications as evidence that relevant state may need to be inspected again.

When it breaks

Watching a directory does not recursively watch its descendants. It reports relevant activity on immediate entries, so a new subdirectory needs its own watch if deeper changes matter. A tree watcher must handle initial enumeration and directories arriving later. Assuming one root watch covers the entire tree silently misses changes below that root.

A file watch follows the watched object, not a pathname forever. An editor can finish writing a temporary file and rename it over the target. The parent directory reports the replacement, while the old file watch does not transfer to the new inode. Waiting for a later close-write event on the destination name can fail because the replacement was already closed before publication.

Identical unread events can coalesce, and a full queue drops events while reporting IN_Q_OVERFLOW. Event counts therefore do not reliably count writes. After overflow, a consumer cannot reconstruct every missing operation from the remaining records. It must reconcile the current watched state instead of continuing with an assumption that its event-derived model is complete.

Check it quickly

Save the script as inotify-events.py and run python3 inotify-events.py in a disposable Linux environment. The recorded run used Linux 6.12.76, Python 3.11.2, and ext4. It creates ASCII-named temporary entries and intentionally fills its own queue. The configured queue limit must be at most 32768; the script reads that limit without changing any sysctl:

from pathlib import Path
from tempfile import TemporaryDirectory
import ctypes,os,struct

MODIFY=0x2;ATTRIB=0x4;CLOSE_WRITE=0x8;MOVED_FROM=0x40;MOVED_TO=0x80
CREATE=0x100;DELETE=0x200;DELETE_SELF=0x400;MOVE_SELF=0x800;OVERFLOW=0x4000;IGNORED=0x8000
libc=ctypes.CDLL(None,use_errno=True)
libc.inotify_init1.argtypes=[ctypes.c_int];libc.inotify_init1.restype=ctypes.c_int
libc.inotify_add_watch.argtypes=[ctypes.c_int,ctypes.c_char_p,ctypes.c_uint32];libc.inotify_add_watch.restype=ctypes.c_int
header=struct.Struct('iIII')
with TemporaryDirectory(prefix='xbcatm-inotify-') as directory:
 root=Path(directory);fd=libc.inotify_init1(os.O_NONBLOCK|os.O_CLOEXEC);assert fd>=0,os.strerror(ctypes.get_errno())
 def watch(path,mask):
 wd=libc.inotify_add_watch(fd,os.fsencode(path),mask);assert wd>=0,os.strerror(ctypes.get_errno());return wd
 def drain():
 events=[]
 while True:
 try:data=os.read(fd,65536)
 except BlockingIOError:return events
 offset=0
 while offset<len(data):
 wd,mask,cookie,length=header.unpack_from(data,offset);offset+=header.size
 name=data[offset:offset+length].split(b'\0',1)[0].decode();offset+=length
 events.append((wd,mask,cookie,name))
 try:
 parent=watch(root,MODIFY|CLOSE_WRITE|CREATE|DELETE|MOVED_FROM|MOVED_TO)
 sub=root/'sub';sub.mkdir();drain();(sub/'unwatched').touch();assert drain()==[]
 nested=watch(sub,CREATE);(sub/'watched').touch();assert any(wd==nested and mask&CREATE and name=='watched' for wd,mask,_,name in drain())
 print('a directory watch is not recursive; adding a child-directory watch observes its new entries')
 counter=root/'counter';stream=os.open(counter,os.O_CREAT|os.O_WRONLY,0o600);drain()
 try:
 for _ in range(3):os.write(stream,b'x')
 finally:os.close(stream)
 events=drain();assert sum(wd==parent and bool(mask&MODIFY) and name=='counter' for wd,mask,_,name in events)==1
 moved=root/'moved';counter.rename(moved);events=drain()
 before=next(e for e in events if e[1]&MOVED_FROM and e[3]=='counter');after=next(e for e in events if e[1]&MOVED_TO and e[3]=='moved')
 assert before[2]==after[2]!=0
 print('three unread identical modifications coalesce; rename FROM/TO events share a nonzero cookie')
 target=root/'target';target.write_bytes(b'old');file_watch=watch(target,MODIFY|ATTRIB|DELETE_SELF|MOVE_SELF);drain()
 replacement=root/'replacement';replacement.write_bytes(b'new');drain();replacement.replace(target);events=drain()
 assert any(wd==file_watch and mask&IGNORED for wd,mask,_,_ in events)
 assert any(wd==parent and mask&MOVED_TO and name=='target' for wd,mask,_,name in events)
 assert not any(wd==parent and mask&CLOSE_WRITE and name=='target' for wd,mask,_,name in events)
 target.write_bytes(b'newer');events=drain()
 assert any(wd==parent and mask&MODIFY and name=='target' for wd,mask,_,name in events)
 assert all(wd!=file_watch for wd,_,_,_ in events)
 print('replacement invalidates the old file watch; the parent-directory watch continues to observe the pathname')
 burst=root/'burst';burst.mkdir();drain();watch(burst,CREATE)
 limit=int(Path('/proc/sys/fs/inotify/max_queued_events').read_text());assert 0<limit<=32768,'use a disposable environment with a bounded queue limit'
 for number in range(limit+8):(burst/str(number)).touch()
 events=drain();assert any(wd==-1 and mask&OVERFLOW for wd,mask,_,_ in events)
 current=len(list(burst.iterdir()));assert current==limit+8
 print('actual queue overflow:',len(events),'events retained for',current,'creates; rescan sees the current entries')
 finally:os.close(fd)

The first case watches the parent directory, creates a subdirectory, and drains the parent event. Creating a file inside that subdirectory produces no event for the parent watch. Adding an explicit subdirectory watch makes the next creation visible. This distinguishes direct-entry coverage from recursive monitoring using the same inotify instance.

Three writes to one file occur before the queue is read. The fixture observes one matching modify event, demonstrating coalescing rather than one notification per write. Renaming the file then produces FROM and TO events with the same nonzero cookie. The cookie associates these two records; it is not a durable identifier for the file’s entire future lifetime.

Next, the script watches a file and replaces its pathname with a previously written and closed file. The old watch receives an ignored event when its object is deleted, while the parent watch reports MOVED_TO for the target name. No destination close-write event accompanies that publication. A subsequent write to the replacement is visible through the parent watch, not the removed file watch.

For overflow, the fixture creates 16392 distinct entries with a configured limit of 16384 queued events. It receives 16385 records, including the overflow notification with watch descriptor minus one. A directory rescan finds all 16392 current entries. Those results show both missing event history and a concrete way to recover present-state knowledge.

All watches share one owned instance, which closes before temporary-tree cleanup. The experiment does not induce a system-wide watch shortage or modify other applications’ queues. Its deliberately bounded burst validates overflow behavior without requiring an application to remain permanently overloaded or relying on an assumed loss signal that was never observed.

Reconcile names and object lifetimes

Pair move cookies when both sides are visible, but allow for moves into or out of the observed set and records separated by other events. Recheck the relevant paths before acting. A name in an event describes the reported change; it does not guarantee that the same object will still occupy that name when the consumer later opens it.

For trees, establish watches and reconcile directory contents while accounting for activity during enumeration. On overflow, rebuild the affected state from the filesystem and restore the necessary watch coverage. An overflow record does not identify one missing filename, so handling it as an ordinary per-watch event is insufficient.

Use notifications to trigger work that tolerates repeated or combined changes, such as reloading a complete configuration after validating its current contents. If every transition must be preserved, use an application journal or another interface with the required history contract. Inotify is useful precisely when its coverage and loss rules match the consumer’s actual needs.

Sources

inotify(7), event formats, coalescing, overflow, renames, and nonrecursive watches, checked 2026-09-12.
inotify_add_watch(2), watch registration and object identity, checked 2026-09-12.
Linux 6.12 inotify UAPI, event structure and flag values used by the executable fixture, checked 2026-09-12.

Hard links share an inode until a name is replaced

Sun, 10 Oct 2027 00:00:00 +0000

A hard link adds another directory entry referring to an existing file. Neither name is intrinsically the original: both identify the same inode within its filesystem. File content, ownership, permissions, and link count belong to that shared object. A filename is a way to reach it, so changes to directory entries must be distinguished from changes to the file itself.

When it breaks

Treating a hard link as an independent backup creates a false recovery boundary. Writing or truncating the shared inode through either name changes what the other name reads. Changing its mode likewise changes the same object’s permissions. A second pathname can survive removal of the first, but it does not preserve an independent historical version against ordinary in-place modifications.

Many editors and deployment tools update differently: they create a replacement file and rename it over one pathname. That directory entry now reaches a new inode, while another hard link still reaches the old one. Whether both names show the update depends on the update method. The mere fact that they were linked yesterday does not establish their identity today.

Hard links also stay within the applicable filesystem mount boundary. link cannot join arbitrary files from different mounts, even when directories appear next to each other in one pathname tree. Ordinary directory hard links are prohibited on Linux. These restrictions prevent using hard links as a general replacement for symbolic links across a directory hierarchy.

Check it quickly

Save the script as hardlinks.py and run python3 hardlinks.py as root in a disposable Linux container. It requires writable /tmp and /run on different filesystems and checks that condition. The recorded environment used separate tmpfs mounts, Linux 6.12.76, and Python 3.11.2. All created entries are confined to temporary directories:

from pathlib import Path
from tempfile import TemporaryDirectory
import errno,os

with TemporaryDirectory(prefix='xbcatm-hardlinks-',dir='/tmp') as directory, TemporaryDirectory(prefix='xbcatm-crosslink-',dir='/run') as other:
 root=Path(directory);source=root/'source';alias=root/'alias'
 source.write_bytes(b'original');os.link(source,alias)
 before=source.stat();linked=alias.stat()
 assert (before.st_dev,before.st_ino)==(linked.st_dev,linked.st_ino)
 assert before.st_nlink==linked.st_nlink==2 and before.st_size==linked.st_size==8
 alias.write_bytes(b'changed')
 assert source.read_bytes()==b'changed'
 print('two names share device/inode, size, and link count; in-place write changes both views')
 replacement=root/'replacement';replacement.write_bytes(b'new file');replacement.replace(source)
 assert source.stat().st_ino!=alias.stat().st_ino
 assert source.read_bytes()==b'new file' and alias.read_bytes()==b'changed'
 assert source.stat().st_nlink==alias.stat().st_nlink==1
 print('atomic replacement changes one name to a new inode; the other name retains the old file')
 with alias.open('rb') as opened:
 alias.unlink()
 assert os.fstat(opened.fileno()).st_nlink==0 and opened.read()==b'changed'
 print('last unlink removes the final name but an existing open descriptor still reads the old data')
 assert root.stat().st_dev!=Path(other).stat().st_dev,'exercise requires two filesystem mounts'
 try:os.link(source,Path(other)/'cross-filesystem')
 except OSError as error:assert error.errno==errno.EXDEV,error
 else:raise AssertionError('cross-filesystem hard link unexpectedly succeeded')
 child=root/'directory';child.mkdir()
 try:os.link(child,root/'directory-link')
 except OSError as error:assert error.errno==errno.EPERM,error
 else:raise AssertionError('directory hard link unexpectedly succeeded')
 print('cross-filesystem link fails with EXDEV; directory hard link fails with EPERM on Linux')

The fixture creates source, writes eight bytes, and adds alias with os.link. Both names report identical device/inode pairs, size eight, and link count two. Comparing the device as well as the inode matters because inode numbers are not globally unique across filesystems. Matching file contents alone would also be insufficient evidence of shared identity.

Writing changed through alias updates the existing inode, so reading source returns the same changed bytes. The operation uses an ordinary truncating write, not replacement. This establishes why the second name cannot serve as a separate copy protecting the prior contents from a writer with access to the shared file.

Next, a separate file containing new file replaces source by rename. The two names now have different inode numbers and each has link count one. source reads the replacement, while alias retains changed. Replacement affects the selected directory entry; it neither searches for other names nor rewrites every hard link to the old inode.

The fixture opens alias and removes its final directory entry. File status through the open descriptor reports link count zero, yet that descriptor still reads changed. The open reference preserves access until it is closed. A zero link count means no hard-link names remain, not that every process has immediately lost its existing reference.

Finally, linking into the second filesystem fails with EXDEV, and attempting to hard-link a directory fails with EPERM. The script checks the specific errors. If its mount precondition is absent, it fails explicitly rather than reporting that a cross-filesystem operation was tested when both directories actually used the same filesystem.

Inspect identity and update behavior

Before relying on shared storage, inspect the device/inode pair and link count of the actual paths. Link count tells how many hard-link directory entries refer to the inode; it does not list where those names are. Traversing directories to find matching inodes is a separate operation, affected by permissions and changes happening during the search.

For snapshot-like directory trees built with hard links, ensure updates replace files instead of modifying shared inodes, and control all writers that can reach them. Filesystem snapshots and independent copies have different contracts. A space-saving tree of links should be described by its actual update and retention behavior rather than simply being called a backup.

Unlinking and renaming concern namespace visibility. This experiment verifies identity and open-reference behavior, not persistence across power loss. If durable publication matters, separately verify the filesystem’s synchronization requirements and the application’s recovery procedure. Keeping another name or observing an atomic rename does not by itself prove the intended data will survive a crash.

Sources

link(2), shared inode identity, cross-filesystem restrictions, and directory-link errors, checked 2026-09-12.
unlink(2), link-count reduction and lifetime of an unlinked open file, checked 2026-09-12.
rename(2), replacement of one pathname without changing other hard links, checked 2026-09-12.

Cron repeats on the calendar

Thu, 30 Sep 2027 00:00:00 +0000

Cron matches calendar fields and launches commands at matching times. It does not inherently wait for an earlier invocation of the same command to finish. If a minute-level task takes seventy seconds, the next minute can launch another copy. Treat trigger policy, concurrency policy, and successful completion as separate responsibilities when operating recurring work.

Read fields as calendar selections

A conventional user crontab entry contains minute, hour, day of month, month, day of week, then the command. Asterisks select all values in their fields. A step such as */15 in minutes selects minute values within each hour; it is not a timer measuring fifteen minutes since the previous completion. A step that does not divide the field range evenly can produce unequal gaps across boundaries.

Day matching has a special rule in the Debian/Vixie format: when day of month and day of week are both restricted, either matching day can permit execution, while the minute, hour, and month constraints still apply. This is not a disjunction between month and weekday. Write representative dates down and verify the intended calendar before installing a terse expression.

Timezone, daylight-saving changes, and downtime affect calendar scheduling. Implementations differ in how they handle skipped or repeated times. Debian cron documents special handling for certain small clock changes and fixed-time jobs, so rules copied from another scheduler may be wrong here. This experiment does not change the clock or test those transition cases. A calendar trigger is also not a durable queue that automatically guarantees every missed run will be replayed.

When it breaks

Overlapping jobs can duplicate side effects, compete for resources, or overwrite shared files. If one invocation must run at a time, implement that policy explicitly. A nonblocking advisory lock can let a later invocation record a skip; a blocking lock can instead leave waiting processes. Neither choice provides exactly-once business effects or repairs a partially completed operation.

A local file lock only coordinates participants using the same relevant locking facility. It does not automatically serialize jobs on unrelated hosts, and network filesystem semantics require separate examination. Keep the lock held for the work’s lifetime, and make skipped work visible. Silently dropping every later trigger while one worker hangs can look like a healthy schedule unless completion age is monitored.

Cron is not universally log-free. Debian logs job starts to syslog by default and offers additional logging controls. Command output may be mailed when a working mail path exists, but a minimal container often lacks that path. Record useful start, completion, failure, and skip outcomes through an intentional destination. A start message proves dispatch, not completion of the application’s task.

Check it quickly

Use a fresh disposable Debian 12 Docker container as root, with no existing root crontab or cron daemon. Install python3, cron, and procps using apt, save the following as cron-overlap.py, and run python3 cron-overlap.py. Allow roughly two minutes: the script waits for actual calendar ticks rather than replacing cron with a simulated scheduler.

from pathlib import Path
from tempfile import TemporaryDirectory
import json
import os
import subprocess
import time

assert os.geteuid() == 0 and Path('/.dockerenv').exists(), 'use a disposable root Linux container'
assert subprocess.run(['pgrep', '-x', 'cron'], capture_output=True).returncode == 1
assert subprocess.run(['crontab', '-l'], capture_output=True).returncode != 0

with TemporaryDirectory(prefix='cron-overlap-') as directory:
 root = Path(directory)
 events_path = root / 'events.jsonl'
 stop = root / 'stop'
 worker = root / 'worker.py'
 worker.write_text('''from pathlib import Path
import fcntl,json,os,sys,time
root=Path(__file__).parent
kind=sys.argv[1]
def record(event):
 data=json.dumps({'event':event,'kind':kind,'pid':os.getpid(),'monotonic':time.monotonic()})+'\\n'
 fd=os.open(root/'events.jsonl',os.O_WRONLY|os.O_CREAT|os.O_APPEND,0o600)
 try: os.write(fd,data.encode())
 finally: os.close(fd)
lock=None
if kind=='locked':
 lock=open(root/'job.lock','a')
 try: fcntl.flock(lock,fcntl.LOCK_EX|fcntl.LOCK_NB)
 except BlockingIOError:
 record('skip')
 raise SystemExit(0)
record('start')
try:
 until=time.monotonic()+70
 while time.monotonic()<until and not (root/'stop').exists(): time.sleep(0.1)
finally:
 record('end')
 if lock: lock.close()
''')
 table = ''.join(f'* * * * * /usr/bin/python3 {worker} {kind} >>{root}/worker.log 2>&1\n'
 for kind in ('plain', 'locked'))
 subprocess.run(['crontab', '-'], input=table, text=True, check=True)
 daemon = subprocess.Popen(['cron', '-f'], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
 def read_events():
 if not events_path.exists():
 return []
 return [json.loads(line) for line in events_path.read_text().splitlines()]
 try:
 deadline = time.monotonic() + 140
 while time.monotonic() < deadline:
 assert daemon.poll() is None, 'cron exited'
 events = read_events()
 plain = [e for e in events if (e['kind'], e['event']) == ('plain', 'start')]
 locked = [e for e in events if (e['kind'], e['event']) == ('locked', 'start')]
 skipped = [e for e in events if (e['kind'], e['event']) == ('locked', 'skip')]
 if len(plain) >= 2 and skipped:
 break
 time.sleep(0.2)
 else:
 raise AssertionError('two real cron ticks were not observed')
 assert len(plain) == 2 and len(locked) == 1 and len(skipped) == 1
 assert plain[1]['monotonic'] - plain[0]['monotonic'] < 70
 assert not any(e['event'] == 'end' and e['pid'] == plain[0]['pid'] for e in events)
 print(json.dumps({'actual_cron_starts': len(plain), 'overlap_observed': True,
 'seconds_between_starts': round(plain[1]['monotonic']-plain[0]['monotonic'], 3),
 'locked_starts': len(locked), 'locked_skips': len(skipped)}), flush=True)
 finally:
 daemon.terminate()
 stop.touch()
 daemon.communicate(timeout=5)
 subprocess.run(['crontab', '-r'], check=True)
 deadline = time.monotonic() + 5
 while time.monotonic() < deadline:
 events = read_events()
 started = {e['pid'] for e in events if e['event'] == 'start'}
 ended = {e['pid'] for e in events if e['event'] == 'end'}
 if started <= ended:
 break
 time.sleep(0.1)
 else:
 raise AssertionError('test workers did not finish')

Two every-minute entries run workers that would last seventy seconds. The plain worker has no lock. The locked worker uses a shared file and a nonblocking exclusive flock, recording a skip when acquisition fails. Each event includes the worker PID and a monotonic timestamp, so distinct invocations and their overlap can be checked directly.

On Debian cron 3.0pl1-162, the plain worker started twice, 60.003 seconds apart, before its first invocation ended. The guarded worker started once and recorded one skip. All assertions passed. Cleanup stopped cron, signaled the test workers through their stop file, waited for their end records, and removed the temporary crontab. The experiment verifies local overlap and explicit exclusion; it does not implement durable retries, distributed locking, recovery after reboot, or exactly-once effects. Select those policies from the job’s actual business contract instead of assuming the calendar expression supplies them.

Sources

Debian 12: crontab(5), calendar fields, steps and the two day fields, checked 2026-09-13.
Debian 12: cron(8), daemon execution, logging and clock-change behavior, checked 2026-09-13.
Linux: flock(2), advisory locks and nonblocking acquisition, checked 2026-09-13.

Linux · XB Field Notes

LVM snapshots capture a block device in time

An exception store, not a copy

Exhaustion is invalidation

Merging, and why thin changes the failure

When it breaks

Check it quickly

Sources

The TCP backlog holds connections that are still syncing

The backlog argument is the accept queue

With syncookies on, the SYN queue stops mattering

The accept queue is where the pressure lands

When it breaks

Check it quickly

Sources

iotop shows which process is burning the disks

Per-process counters

See whatever sweeps the disk

Watch before you act

When it breaks

Check it quickly

Sources

Network namespaces isolate networking; their names do not own every process

When it breaks

Check it quickly

Keep naming, process ownership, and connectivity separate

Sources

Strace explains selected system-call boundaries

When it breaks

Check it quickly

Follow a hypothesis through to an outcome

Sources

Cron system and user tables have different contracts

The account column changes the parser boundary

When it breaks

Check it quickly

Sources

Shell word splitting turns unquoted variables into several arguments

Splitting depends on the context

When it breaks

Check it quickly

Sources

Huge pages cut TLB misses at the cost of flexibility

Why 4 KB hurts

hugetlb vs transparent

Budget the pool

When it breaks

Check it quickly

Further reading

Sources

Linux overcommit is a policy the kernel bet on

Why malloc rarely fails

The knob

Heuristic versus strict

When it breaks

Check it quickly

Further reading

Sources

systemd-tmpfiles applies creation and age policies

When it breaks

Check it quickly

Match housekeeping to the producer

Further reading

Sources

A systemd oneshot completes one activation

When it breaks

Check it quickly

Give the task its own correctness guarantees

Further reading

Sources

A veth pair carries frames between two endpoints; routing supplies their meaning

When it breaks

Check it quickly

Diagnose the layers in order

Sources

A page fault is not always a disk read

Minor faults map a page

Major faults read the disk

Count them separately

When it breaks