Filesystem · XB Field Notes

Extended attributes belong to the inode and to a namespace

Tue, 14 Dec 2027 00:00:00 +0000

Extended attributes attach named byte values to filesystem objects outside the ordinary file-content stream. Applications can use the user namespace for small metadata, while other namespaces have kernel, security-module, or privileged semantics. Their usefulness depends on the filesystem, access rules, and transfer tools. A metadata tag is reliable only when the workflow preserves and interprets it according to an explicit contract.

When it breaks

An attribute value is not necessarily text. It can contain NUL bytes or arbitrary binary data, so treating it as a terminated string or assuming a particular encoding can corrupt the value. An empty attribute and a missing attribute are also different states. Returning an empty string for both can hide whether the metadata was deliberately set or lost during transfer.

Same-filesystem rename retains the object and its attributes, but copying content usually creates another inode. The copy tool decides which metadata it attempts to preserve. Equal file hashes therefore do not prove equal extended attributes. A pipeline can successfully deliver every content byte while omitting a classification, origin, or application version tag that downstream processing expects.

Namespace names do not create application authorization rules. Setting user.classification=private does not make a readable file confidential. User attributes follow applicable file-access permissions; security attributes have their own documented semantics and policy. Confusing an application label with enforced access control can expose content while the label appears reassuringly intact.

Check it quickly

Save this as xattrs.py and run python3 xattrs.py as root in a disposable Linux container whose temporary directory supports user extended attributes. The recorded run used Linux 6.12.76, Python 3.11.2, and an ext4 volume. Root prepares the fixture and starts one child under an unprivileged numeric identity; no account is created:

from pathlib import Path
from tempfile import TemporaryDirectory
import errno,os,shutil,subprocess,sys

def fails(code,operation):
 try:operation()
 except OSError as error:assert error.errno==code,error
 else:raise AssertionError('expected attribute error')
with TemporaryDirectory(prefix='xbcatm-xattr-') as directory:
 root=Path(directory);root.chmod(0o755);path=root/'data';path.write_bytes(b'public fixture');path.chmod(0o644)
 os.setxattr(path,'user.review',b'binary\0value\xff',os.XATTR_CREATE)
 assert os.getxattr(path,'user.review')==b'binary\0value\xff'
 fails(errno.EEXIST,lambda:os.setxattr(path,'user.review',b'other',os.XATTR_CREATE))
 fails(errno.ENODATA,lambda:os.setxattr(path,'user.missing',b'value',os.XATTR_REPLACE))
 os.setxattr(path,'user.empty',b'');assert os.getxattr(path,'user.empty')==b''
 os.removexattr(path,'user.empty');fails(errno.ENODATA,lambda:os.getxattr(path,'user.empty'))
 print('attribute values are bytes; empty differs from missing; CREATE and REPLACE enforce their preconditions')
 alias=root/'alias';os.link(path,alias)
 os.setxattr(alias,'user.review',b'shared',os.XATTR_REPLACE)
 assert os.getxattr(path,'user.review')==b'shared'
 moved=root/'moved';path.rename(moved);assert os.getxattr(moved,'user.review')==b'shared'
 plain=root/'plain';preserved=root/'preserved';shutil.copyfile(moved,plain);shutil.copy2(moved,preserved)
 fails(errno.ENODATA,lambda:os.getxattr(plain,'user.review'))
 assert os.getxattr(preserved,'user.review')==b'shared'
 assert plain.read_bytes()==preserved.read_bytes()==moved.read_bytes()
 print('hard links and same-filesystem rename retain inode attributes; copyfile omits them, copy2 preserves this supported user attribute')
 os.setxattr(moved,'user.classification',b'private')
 probe=r'''
from pathlib import Path
import errno,os,sys
path=Path(sys.argv[1]);assert path.read_bytes()==b'public fixture'
assert os.getxattr(path,'user.classification')==b'private'
try:os.setxattr(path,'user.classification',b'changed')
except OSError as error:assert error.errno==errno.EACCES
else:raise AssertionError('another identity changed a read-only user attribute')
'''
 result=subprocess.run([sys.executable,'-c',probe,str(moved)],user=61001,group=61001,extra_groups=[],capture_output=True,text=True,timeout=5)
 assert result.returncode==0,result.stderr
 fails(errno.E2BIG,lambda:os.setxattr(moved,'user.too_large',b'x'*65537))
 print('a user.classification label does not enforce confidentiality; file permissions block the other identity from changing it; oversized value returns E2BIG')

The fixture stores a value containing both NUL and a non-UTF-8 byte and retrieves the exact bytes. Creating the same key again with XATTR_CREATE fails with EEXIST, while replacing a missing key with XATTR_REPLACE fails with ENODATA. These flags express preconditions instead of silently turning every write into an unconditional update.

An explicitly empty value reads back as empty bytes. After removal, reading that key returns ENODATA. This provides a direct distinction between absence and an intentionally empty attribute. A caller using metadata to drive behavior needs to decide how each state affects the operation rather than collapsing both into one convenient default.

A hard link shares the same inode, so updating the attribute through that name changes the value seen through the original name. Renaming the original within the filesystem preserves the attribute. The experiment then copies the file twice: copyfile preserves the content but omits the user attribute, while copy2 preserves this supported attribute in the tested environment.

All three content streams compare equal despite the metadata difference. Python documents metadata copying as dependent on platform support and permissions, so this successful copy2 result is not a universal guarantee for every attribute namespace or destination. Verification should inspect the attributes actually needed after the specific transfer, not merely the copy command’s exit status.

The file remains mode 0644 when tagged private. Another ordinary identity reads both its content and the tag, proving that the label did not enforce confidentiality. That identity cannot change the attribute because it lacks the relevant write permission. Finally, a value larger than 65536 bytes is rejected with E2BIG, demonstrating a real size boundary.

Preserve meaning across the workflow

Choose application-specific key names to avoid collisions within the shared user namespace, and define the value encoding and version. Keep values small enough for the target filesystems; their practical limits can be tighter than the Linux interface maximum and can depend on other metadata already attached to the inode.

Test backup, archive, restore, and cross-filesystem copy paths using both content and attribute assertions. Some destinations do not support the required namespaces, and some tools preserve them only with explicit options. Decide whether loss should fail the workflow or be represented through a documented alternative, based on the actual metadata contract.

Updating file bytes and updating a separate attribute are not automatically one application transaction. A checksum tag can become stale after a later write, and a writer able to change both data and metadata is not an independent integrity authority. Validate the relationship when consuming it, and use a stronger storage or signing contract when that relationship must survive concurrent changes or untrusted writers.

Sources

xattr(7), attribute namespaces, permission rules, and storage limits, checked 2026-09-12.
setxattr(2), CREATE/REPLACE behavior and attribute error conditions, checked 2026-09-12.
getxattr(2), binary values and missing-attribute errors, checked 2026-09-12.
Python 3.11 shutil documentation, copyfile and best-effort metadata preservation by copy2, checked 2026-09-12.

Btrfs snapshots are copy-on-write, not copies

Sun, 14 Nov 2027 00:00:00 +0000

A btrfs snapshot is not a copy of anything. btrfs subvolume snapshot creates a new subvolume whose root points at the same extent tree as the source, so every extent the two share is referenced twice and stored once. Creation is a metadata operation measured in milliseconds whether the subvolume holds a gigabyte or ten terabytes.

That makes snapshots the cheapest rollback primitive on the box, and it is also why they are the most commonly over-trusted. Space grows with divergence, not with the size of the data; the cost arrives later, on somebody else’s write, on the same device that holds the original. A snapshot you never prune is a slow-motion disk-full incident, and a snapshot on the same disk is not a backup no matter how many of them you keep.

Shared extents, not copied blocks

Snapshots are subvolumes, and a subvolume is a subtree with its own root, so the snapshot machinery is the reference-counting machinery that makes reflinks work. Take a read-only snapshot with btrfs subvolume snapshot -r /data /snap/data-2026-09-16; the -r matters, because a writable snapshot starts diverging the moment anything writes to it, and a backup target should not be writable in the first place.

The first write after a snapshot is what costs. When either side overwrites a shared extent, the filesystem allocates a new extent for the writer and leaves the old one to the other side, so the price of a snapshot is paid by the next write, in units of the extents that write touches — not by the snapshot command. APFS clonefiles share extents under the same rule, and the fixture measures the shape: a 64 MiB clone costs 0 KiB, a plain 64 MiB copy costs 65536 KiB, and overwriting the first 8 MiB of the clone costs 8192 KiB. Rewriting that same 8 MiB a second time costs 0 KiB, because those extents already diverged.

Where the space actually goes

btrfs filesystem du is the honest accounting: it prints Total, Exclusive and Set shared for each argument, where Set shared is the space it shares with the others named on the command line. Right after a snapshot, Exclusive should be near zero and Set shared close to the origin’s Total. compsize gives the same split per file, and btrfs qgroup show -reF tracks exclusive and referenced bytes per qgroup when quotas are enabled.

Deleting the origin does not free the shared extents — they become the snapshot’s exclusive data, and the space changes owner. This is the most common support question about btrfs: a file is deleted and df does not move. Only when the last reference goes away does the space return, which is why retention policy and snapshot cadence must be designed together.

A restore point is not a replica

Snapshots live in the same filesystem on the same device as their source, so a controller failure, a dead disk, or a btrfs check gone wrong takes the data and every snapshot of it in one motion. They defend against exactly one class of accident — the write that should not have happened — and they are excellent at it: rollback is local, instant, and needs no network.

Real protection means an independent copy: btrfs send walks a snapshot and emits a stream that btrfs receive applies to another filesystem, with -p sending only the delta from a parent snapshot. Send that stream to a different machine or a different pool. The distinction: send is the replication mechanism, and the snapshot is just the object it replicates.

When it breaks

The classic failure is the delete that frees nothing: someone removes a 40 GB directory, df is unmoved, and they delete more to compensate. Run btrfs filesystem du -s across the subvolume and its snapshots and look for one whose Exclusive value accounts for the gap. If the snapshots are read-only, the fix is deleting snapshots, not more files.

The second failure is a full filesystem, which on btrfs is worse than on ext4. Metadata needs space of its own, and a copy-on-write filesystem with no free space cannot allocate the extent a write requires, so one at 100 percent can fail writes that would succeed elsewhere and can go read-only. Keep headroom — 5 percent is not paranoia — bound snapshot retention, and remember that a snapshot of a nearly full filesystem also commits you to space for its divergence.

Check it quickly

Confirm the snapshot shares rather than copies, and find what is holding the space:

btrfs subvolume snapshot -r /data /snap/data-$(date +%F)
btrfs filesystem du -s /data /snap/data-2026-09-16
compsize /snap/data-2026-09-16
btrfs filesystem usage /data

du prints a Total, Exclusive and Set shared triple per path: a fresh snapshot’s Exclusive should be a few MiB of metadata at most while Set shared tracks the origin’s Total. If Exclusive already equals Total, the snapshot is a full copy. btrfs filesystem usage then shows how much divergence the device can still absorb.

Sources

btrfs subvolume documentation, subvolume and snapshot semantics including read-only snapshots, checked 2026-09-16.
btrfs filesystem documentation, defines the du Exclusive and Set shared columns used above, checked 2026-09-16.
btrfs qgroup documentation, exclusive and referenced byte accounting per qgroup, checked 2026-09-16.
btrfs send documentation, the replication stream that moves a snapshot off the machine, checked 2026-09-16.

Inotify reports changes, but it is not a complete history

Wed, 13 Oct 2027 00:00:00 +0000

Inotify delivers Linux filesystem notifications through an instance with registered watches and a readable event queue. Each record identifies a watch, event mask, and possibly a name and move cookie. This can avoid repeatedly checking unchanged paths, but the stream is not an unlimited audit log. Correct consumers interpret notifications as evidence that relevant state may need to be inspected again.

When it breaks

Watching a directory does not recursively watch its descendants. It reports relevant activity on immediate entries, so a new subdirectory needs its own watch if deeper changes matter. A tree watcher must handle initial enumeration and directories arriving later. Assuming one root watch covers the entire tree silently misses changes below that root.

A file watch follows the watched object, not a pathname forever. An editor can finish writing a temporary file and rename it over the target. The parent directory reports the replacement, while the old file watch does not transfer to the new inode. Waiting for a later close-write event on the destination name can fail because the replacement was already closed before publication.

Identical unread events can coalesce, and a full queue drops events while reporting IN_Q_OVERFLOW. Event counts therefore do not reliably count writes. After overflow, a consumer cannot reconstruct every missing operation from the remaining records. It must reconcile the current watched state instead of continuing with an assumption that its event-derived model is complete.

Check it quickly

Save the script as inotify-events.py and run python3 inotify-events.py in a disposable Linux environment. The recorded run used Linux 6.12.76, Python 3.11.2, and ext4. It creates ASCII-named temporary entries and intentionally fills its own queue. The configured queue limit must be at most 32768; the script reads that limit without changing any sysctl:

from pathlib import Path
from tempfile import TemporaryDirectory
import ctypes,os,struct

MODIFY=0x2;ATTRIB=0x4;CLOSE_WRITE=0x8;MOVED_FROM=0x40;MOVED_TO=0x80
CREATE=0x100;DELETE=0x200;DELETE_SELF=0x400;MOVE_SELF=0x800;OVERFLOW=0x4000;IGNORED=0x8000
libc=ctypes.CDLL(None,use_errno=True)
libc.inotify_init1.argtypes=[ctypes.c_int];libc.inotify_init1.restype=ctypes.c_int
libc.inotify_add_watch.argtypes=[ctypes.c_int,ctypes.c_char_p,ctypes.c_uint32];libc.inotify_add_watch.restype=ctypes.c_int
header=struct.Struct('iIII')
with TemporaryDirectory(prefix='xbcatm-inotify-') as directory:
 root=Path(directory);fd=libc.inotify_init1(os.O_NONBLOCK|os.O_CLOEXEC);assert fd>=0,os.strerror(ctypes.get_errno())
 def watch(path,mask):
 wd=libc.inotify_add_watch(fd,os.fsencode(path),mask);assert wd>=0,os.strerror(ctypes.get_errno());return wd
 def drain():
 events=[]
 while True:
 try:data=os.read(fd,65536)
 except BlockingIOError:return events
 offset=0
 while offset<len(data):
 wd,mask,cookie,length=header.unpack_from(data,offset);offset+=header.size
 name=data[offset:offset+length].split(b'\0',1)[0].decode();offset+=length
 events.append((wd,mask,cookie,name))
 try:
 parent=watch(root,MODIFY|CLOSE_WRITE|CREATE|DELETE|MOVED_FROM|MOVED_TO)
 sub=root/'sub';sub.mkdir();drain();(sub/'unwatched').touch();assert drain()==[]
 nested=watch(sub,CREATE);(sub/'watched').touch();assert any(wd==nested and mask&CREATE and name=='watched' for wd,mask,_,name in drain())
 print('a directory watch is not recursive; adding a child-directory watch observes its new entries')
 counter=root/'counter';stream=os.open(counter,os.O_CREAT|os.O_WRONLY,0o600);drain()
 try:
 for _ in range(3):os.write(stream,b'x')
 finally:os.close(stream)
 events=drain();assert sum(wd==parent and bool(mask&MODIFY) and name=='counter' for wd,mask,_,name in events)==1
 moved=root/'moved';counter.rename(moved);events=drain()
 before=next(e for e in events if e[1]&MOVED_FROM and e[3]=='counter');after=next(e for e in events if e[1]&MOVED_TO and e[3]=='moved')
 assert before[2]==after[2]!=0
 print('three unread identical modifications coalesce; rename FROM/TO events share a nonzero cookie')
 target=root/'target';target.write_bytes(b'old');file_watch=watch(target,MODIFY|ATTRIB|DELETE_SELF|MOVE_SELF);drain()
 replacement=root/'replacement';replacement.write_bytes(b'new');drain();replacement.replace(target);events=drain()
 assert any(wd==file_watch and mask&IGNORED for wd,mask,_,_ in events)
 assert any(wd==parent and mask&MOVED_TO and name=='target' for wd,mask,_,name in events)
 assert not any(wd==parent and mask&CLOSE_WRITE and name=='target' for wd,mask,_,name in events)
 target.write_bytes(b'newer');events=drain()
 assert any(wd==parent and mask&MODIFY and name=='target' for wd,mask,_,name in events)
 assert all(wd!=file_watch for wd,_,_,_ in events)
 print('replacement invalidates the old file watch; the parent-directory watch continues to observe the pathname')
 burst=root/'burst';burst.mkdir();drain();watch(burst,CREATE)
 limit=int(Path('/proc/sys/fs/inotify/max_queued_events').read_text());assert 0<limit<=32768,'use a disposable environment with a bounded queue limit'
 for number in range(limit+8):(burst/str(number)).touch()
 events=drain();assert any(wd==-1 and mask&OVERFLOW for wd,mask,_,_ in events)
 current=len(list(burst.iterdir()));assert current==limit+8
 print('actual queue overflow:',len(events),'events retained for',current,'creates; rescan sees the current entries')
 finally:os.close(fd)

The first case watches the parent directory, creates a subdirectory, and drains the parent event. Creating a file inside that subdirectory produces no event for the parent watch. Adding an explicit subdirectory watch makes the next creation visible. This distinguishes direct-entry coverage from recursive monitoring using the same inotify instance.

Three writes to one file occur before the queue is read. The fixture observes one matching modify event, demonstrating coalescing rather than one notification per write. Renaming the file then produces FROM and TO events with the same nonzero cookie. The cookie associates these two records; it is not a durable identifier for the file’s entire future lifetime.

Next, the script watches a file and replaces its pathname with a previously written and closed file. The old watch receives an ignored event when its object is deleted, while the parent watch reports MOVED_TO for the target name. No destination close-write event accompanies that publication. A subsequent write to the replacement is visible through the parent watch, not the removed file watch.

For overflow, the fixture creates 16392 distinct entries with a configured limit of 16384 queued events. It receives 16385 records, including the overflow notification with watch descriptor minus one. A directory rescan finds all 16392 current entries. Those results show both missing event history and a concrete way to recover present-state knowledge.

All watches share one owned instance, which closes before temporary-tree cleanup. The experiment does not induce a system-wide watch shortage or modify other applications’ queues. Its deliberately bounded burst validates overflow behavior without requiring an application to remain permanently overloaded or relying on an assumed loss signal that was never observed.

Reconcile names and object lifetimes

Pair move cookies when both sides are visible, but allow for moves into or out of the observed set and records separated by other events. Recheck the relevant paths before acting. A name in an event describes the reported change; it does not guarantee that the same object will still occupy that name when the consumer later opens it.

For trees, establish watches and reconcile directory contents while accounting for activity during enumeration. On overflow, rebuild the affected state from the filesystem and restore the necessary watch coverage. An overflow record does not identify one missing filename, so handling it as an ordinary per-watch event is insufficient.

Use notifications to trigger work that tolerates repeated or combined changes, such as reloading a complete configuration after validating its current contents. If every transition must be preserved, use an application journal or another interface with the required history contract. Inotify is useful precisely when its coverage and loss rules match the consumer’s actual needs.

Sources

inotify(7), event formats, coalescing, overflow, renames, and nonrecursive watches, checked 2026-09-12.
inotify_add_watch(2), watch registration and object identity, checked 2026-09-12.
Linux 6.12 inotify UAPI, event structure and flag values used by the executable fixture, checked 2026-09-12.

A sparse file separates logical length from allocated blocks

Mon, 11 Oct 2027 00:00:00 +0000

A sparse regular file can have a large logical length while allocating storage for only part of its address range. Reads from holes within that length return zero bytes. File length, byte content, and filesystem allocation are therefore different measurements. Understanding their relationship matters when budgeting virtual disks, copying backups, or deciding whether a zero-filled region actually reserves space for future writes.

When it breaks

Seeking beyond the end of a file changes the descriptor’s offset, not the file’s length. A later write or an explicit extension can create the intervening zero-reading range. Calling seek alone and reporting that a large sparse file now exists is incorrect. Likewise, a large logical size does not mean the filesystem has reserved enough blocks to fill that file later.

A copy that reads every byte and writes it normally can turn holes into allocated zero-filled ranges. The resulting checksum can match perfectly while its allocation is much larger. Transfer and restore workflows that budget only the source’s current allocated space can therefore run out of capacity despite preserving the logical content correctly.

Hole punching is an intentional data operation. Punching an allocated range makes the corresponding bytes read as zeros while preserving length when requested with KEEP_SIZE. It is not a harmless command to tidy away arbitrary file regions. Preallocation solves a different problem: it reserves filesystem allocation, even though an unwritten allocated range can also read as zeros.

Check it quickly

Save the script as sparse.py and run python3 sparse.py using Python 3.11 or later on a disposable Linux system with an ext4 temporary directory. The recorded run used Linux 6.12.76 and Python 3.11.2. It creates three temporary files and uses about 32 MiB of fully allocated data plus the small sparse allocation:

from pathlib import Path
from tempfile import TemporaryDirectory
import ctypes,hashlib,os

size=16*1024*1024;first=4*1024*1024;second=12*1024*1024
libc=ctypes.CDLL(None,use_errno=True);libc.fallocate.argtypes=[ctypes.c_int,ctypes.c_int,ctypes.c_longlong,ctypes.c_longlong];libc.fallocate.restype=ctypes.c_int
def digest(path):
 with path.open('rb') as handle:return hashlib.file_digest(handle,'sha256').hexdigest()
with TemporaryDirectory(prefix='xbcatm-sparse-') as directory:
 root=Path(directory);path=root/'sparse';fd=os.open(path,os.O_RDWR|os.O_CREAT|os.O_EXCL,0o600)
 try:
 os.lseek(fd,size,os.SEEK_SET);assert os.fstat(fd).st_size==0
 os.ftruncate(fd,size);assert os.fstat(fd).st_size==size and os.fstat(fd).st_blocks==0
 assert os.pread(fd,16,0)==b'\0'*16
 os.pwrite(fd,b'A',first);os.pwrite(fd,b'B',second);os.fsync(fd)
 allocated=os.fstat(fd).st_blocks*512;assert allocated<size
 data_start=os.lseek(fd,0,os.SEEK_DATA);hole_start=os.lseek(fd,data_start,os.SEEK_HOLE)
 assert data_start==first and first<hole_start<=second
 print('seek alone leaves size zero; truncate creates zero-reading holes; sparse size',size,'allocated',allocated)
 print('first SEEK_DATA',data_start,'following SEEK_HOLE',hole_start)
 dense=root/'dense'
 with path.open('rb') as source,dense.open('wb') as target:
 while chunk:=source.read(65536):target.write(chunk)
 target.flush();os.fsync(target.fileno())
 assert digest(path)==digest(dense)
 assert dense.stat().st_blocks*512>=size
 print('explicitly writing all copied bytes preserves content but allocates the zero-filled ranges')
 result=libc.fallocate(fd,0x01|0x02,first,4096) # KEEP_SIZE | PUNCH_HOLE
 assert result==0,os.strerror(ctypes.get_errno())
 os.fsync(fd)
 assert os.fstat(fd).st_size==size and os.pread(fd,1,first)==b'\0'
 assert os.pread(fd,1,second)==b'B' and os.fstat(fd).st_blocks*512<allocated
 print('punching the aligned data range releases its blocks and makes it read zero without changing file length')
 reserved=root/'reserved'
 with reserved.open('w+b') as handle:
 os.posix_fallocate(handle.fileno(),0,size);os.fsync(handle.fileno())
 info=os.fstat(handle.fileno());assert info.st_size==size and info.st_blocks*512>=size
 assert handle.read(16)==b'\0'*16
 print('preallocation reserves blocks even while the newly allocated range reads as zeros')
 finally:os.close(fd)

The fixture first seeks to sixteen MiB in an empty file and verifies that its size remains zero. Truncating to that length then creates a file with sixteen MiB of logical content and no allocated data blocks in this ext4 case. Reading its beginning returns zeros, establishing the difference between moving an offset and extending the file.

Writing one byte at four MiB and another at twelve MiB allocates two small regions. After synchronization, the recorded file length is 16777216 bytes and its reported allocation is 8192 bytes. Linux’s st_blocks count is converted using 512-byte units; treating it as a count of the filesystem’s larger block size would miscalculate the result.

The first SEEK_DATA result is 4194304, and the following SEEK_HOLE result is 4198400 in the recorded run. These describe the ext4 region around the first written byte, not only that byte’s nonzero position. A data extent may contain zeros too, so extent reporting is not a search for nonzero content.

The script explicitly copies all logical bytes through ordinary writes into a second file. Its digest matches the sparse source, but its allocated space covers the full logical length. This controlled dense copy demonstrates the expansion risk without claiming that every modern copy tool or filesystem necessarily materializes holes in the same way.

Next, an aligned punch removes the first allocated data region. The original A byte now reads as zero, the distant B byte remains, length stays unchanged, and allocated blocks decrease. Finally, preallocating the third file reserves space across sixteen MiB while its initial bytes read as zeros. Zero content alone cannot distinguish a hole from reserved allocation.

Budget and preserve the intended representation

Check both logical size and allocated blocks after the actual archive, transfer, and restore path. A sparse-aware option may preserve holes, while compression, reflinks, and filesystem-specific copying can produce other storage relationships. Per-file allocation also should not automatically be interpreted as exclusive physical-device usage when the filesystem shares or compresses extents.

Use the target filesystem’s documented preallocation contract when future allocation must be reserved. A merely truncated virtual image can grow its storage consumption later and fail when the shared pool is full. Capacity planning needs the expected populated size and competing allocations, not just an attractive small allocation measured at creation time.

Treat SEEK_DATA and SEEK_HOLE as filesystem-provided extent information. Implementations are allowed to report holes conservatively, including representing a broader range as data. Verify support and behavior before depending on exact offsets. When punching or copying ranges, preserve the application’s intended bytes first; space savings are useful only when the resulting data and allocation match the operation’s actual purpose.

Sources

lseek(2), seek position, SEEK_DATA/SEEK_HOLE, and filesystem reporting limits, checked 2026-09-12.
truncate(2), extending logical length with zero-reading regions, checked 2026-09-12.
fallocate(2), preallocation and hole punching without changing logical size, checked 2026-09-12.
Linux 6.12 fallocate UAPI, KEEP_SIZE and PUNCH_HOLE flags used by the fixture, checked 2026-09-12.

Hard links share an inode until a name is replaced

Sun, 10 Oct 2027 00:00:00 +0000

A hard link adds another directory entry referring to an existing file. Neither name is intrinsically the original: both identify the same inode within its filesystem. File content, ownership, permissions, and link count belong to that shared object. A filename is a way to reach it, so changes to directory entries must be distinguished from changes to the file itself.

When it breaks

Treating a hard link as an independent backup creates a false recovery boundary. Writing or truncating the shared inode through either name changes what the other name reads. Changing its mode likewise changes the same object’s permissions. A second pathname can survive removal of the first, but it does not preserve an independent historical version against ordinary in-place modifications.

Many editors and deployment tools update differently: they create a replacement file and rename it over one pathname. That directory entry now reaches a new inode, while another hard link still reaches the old one. Whether both names show the update depends on the update method. The mere fact that they were linked yesterday does not establish their identity today.

Hard links also stay within the applicable filesystem mount boundary. link cannot join arbitrary files from different mounts, even when directories appear next to each other in one pathname tree. Ordinary directory hard links are prohibited on Linux. These restrictions prevent using hard links as a general replacement for symbolic links across a directory hierarchy.

Check it quickly

Save the script as hardlinks.py and run python3 hardlinks.py as root in a disposable Linux container. It requires writable /tmp and /run on different filesystems and checks that condition. The recorded environment used separate tmpfs mounts, Linux 6.12.76, and Python 3.11.2. All created entries are confined to temporary directories:

from pathlib import Path
from tempfile import TemporaryDirectory
import errno,os

with TemporaryDirectory(prefix='xbcatm-hardlinks-',dir='/tmp') as directory, TemporaryDirectory(prefix='xbcatm-crosslink-',dir='/run') as other:
 root=Path(directory);source=root/'source';alias=root/'alias'
 source.write_bytes(b'original');os.link(source,alias)
 before=source.stat();linked=alias.stat()
 assert (before.st_dev,before.st_ino)==(linked.st_dev,linked.st_ino)
 assert before.st_nlink==linked.st_nlink==2 and before.st_size==linked.st_size==8
 alias.write_bytes(b'changed')
 assert source.read_bytes()==b'changed'
 print('two names share device/inode, size, and link count; in-place write changes both views')
 replacement=root/'replacement';replacement.write_bytes(b'new file');replacement.replace(source)
 assert source.stat().st_ino!=alias.stat().st_ino
 assert source.read_bytes()==b'new file' and alias.read_bytes()==b'changed'
 assert source.stat().st_nlink==alias.stat().st_nlink==1
 print('atomic replacement changes one name to a new inode; the other name retains the old file')
 with alias.open('rb') as opened:
 alias.unlink()
 assert os.fstat(opened.fileno()).st_nlink==0 and opened.read()==b'changed'
 print('last unlink removes the final name but an existing open descriptor still reads the old data')
 assert root.stat().st_dev!=Path(other).stat().st_dev,'exercise requires two filesystem mounts'
 try:os.link(source,Path(other)/'cross-filesystem')
 except OSError as error:assert error.errno==errno.EXDEV,error
 else:raise AssertionError('cross-filesystem hard link unexpectedly succeeded')
 child=root/'directory';child.mkdir()
 try:os.link(child,root/'directory-link')
 except OSError as error:assert error.errno==errno.EPERM,error
 else:raise AssertionError('directory hard link unexpectedly succeeded')
 print('cross-filesystem link fails with EXDEV; directory hard link fails with EPERM on Linux')

The fixture creates source, writes eight bytes, and adds alias with os.link. Both names report identical device/inode pairs, size eight, and link count two. Comparing the device as well as the inode matters because inode numbers are not globally unique across filesystems. Matching file contents alone would also be insufficient evidence of shared identity.

Writing changed through alias updates the existing inode, so reading source returns the same changed bytes. The operation uses an ordinary truncating write, not replacement. This establishes why the second name cannot serve as a separate copy protecting the prior contents from a writer with access to the shared file.

Next, a separate file containing new file replaces source by rename. The two names now have different inode numbers and each has link count one. source reads the replacement, while alias retains changed. Replacement affects the selected directory entry; it neither searches for other names nor rewrites every hard link to the old inode.

The fixture opens alias and removes its final directory entry. File status through the open descriptor reports link count zero, yet that descriptor still reads changed. The open reference preserves access until it is closed. A zero link count means no hard-link names remain, not that every process has immediately lost its existing reference.

Finally, linking into the second filesystem fails with EXDEV, and attempting to hard-link a directory fails with EPERM. The script checks the specific errors. If its mount precondition is absent, it fails explicitly rather than reporting that a cross-filesystem operation was tested when both directories actually used the same filesystem.

Inspect identity and update behavior

Before relying on shared storage, inspect the device/inode pair and link count of the actual paths. Link count tells how many hard-link directory entries refer to the inode; it does not list where those names are. Traversing directories to find matching inodes is a separate operation, affected by permissions and changes happening during the search.

For snapshot-like directory trees built with hard links, ensure updates replace files instead of modifying shared inodes, and control all writers that can reach them. Filesystem snapshots and independent copies have different contracts. A space-saving tree of links should be described by its actual update and retention behavior rather than simply being called a backup.

Unlinking and renaming concern namespace visibility. This experiment verifies identity and open-reference behavior, not persistence across power loss. If durable publication matters, separately verify the filesystem’s synchronization requirements and the application’s recovery procedure. Keeping another name or observing an atomic rename does not by itself prove the intended data will survive a crash.

Sources

link(2), shared inode identity, cross-filesystem restrictions, and directory-link errors, checked 2026-09-12.
unlink(2), link-count reduction and lifetime of an unlinked open file, checked 2026-09-12.
rename(2), replacement of one pathname without changing other hard links, checked 2026-09-12.

A symbolic link stores a path, not a permanent target identity

Tue, 03 Aug 2027 00:00:00 +0000

A symbolic link is a filesystem object containing a target pathname. Following the link resolves that pathname at the time of the operation; it does not bind the link permanently to the target’s inode. This distinction explains why links can cross filesystem boundaries, become dangling, or start reaching a different object even though their stored target text has not changed.

When it breaks

A relative target is interpreted from the directory containing the symbolic link, not from the calling process’s working directory. Moving the link can therefore change its meaning. Moving or renaming the target can instead leave the link dangling. A deployment that rearranges directories without accounting for these bases can silently change which file a familiar link reaches.

Existence checks answer different questions. An operation that follows the link can report a missing target while an operation inspecting the directory entry still finds a symbolic link. Likewise, a permissive pathname-resolution function may produce an absolute string even when its final target is missing. A normalized string is not proof that a usable file currently exists.

Checking a path and later opening it creates a time-of-check/time-of-use gap when another actor can change the relevant directory entries. Quoting the pathname protects shell argument parsing, but does not freeze filesystem resolution. O_NOFOLLOW on ordinary open rejects a symbolic link in the final component; it does not automatically prohibit links in earlier directory components.

Check it quickly

Save this script as symlinks.py and run python3 symlinks.py in a disposable Linux environment. The recorded run used Linux 6.12.76 and Python 3.11.2. It creates harmless temporary files and changes its own links deliberately, so the observations are deterministic and require no concurrent attacker or access to unrelated paths:

from pathlib import Path
from tempfile import TemporaryDirectory
import errno,os

def read_no_follow(path):
 fd=os.open(path,os.O_RDONLY|os.O_NOFOLLOW)
 try:return os.read(fd,100)
 finally:os.close(fd)
with TemporaryDirectory(prefix='xbcatm-symlinks-') as directory:
 root=Path(directory);first=root/'first';second=root/'second';first.mkdir();second.mkdir()
 (first/'data').write_text('first');(second/'data').write_text('second')
 link=first/'link';link.symlink_to('data')
 assert os.readlink(link)=='data' and link.read_text()=='first'
 assert link.lstat().st_ino!=link.stat().st_ino
 moved=second/'link';link.rename(moved)
 assert os.readlink(moved)=='data' and moved.read_text()=='second'
 print('moving a relative symlink changes its target because resolution starts at its containing directory')
 (second/'data').rename(second/'renamed')
 assert moved.is_symlink() and os.path.lexists(moved) and not moved.exists()
 assert moved.resolve(strict=False)==second/'data'
 try:moved.resolve(strict=True)
 except FileNotFoundError:pass
 else:raise AssertionError('strict resolution unexpectedly found missing target')
 moved.unlink();assert (second/'renamed').read_text()=='second'
 print('dangling link exists as a link; permissive resolution can name a missing target; unlink removes only the link')
 safe=root/'safe';outside=root/'outside';safe.write_bytes(b'approved');outside.write_bytes(b'replaced')
 current=root/'current';current.symlink_to('safe');checked=current.resolve(strict=True)
 assert checked==safe
 opened=os.open(current,os.O_RDONLY)
 try:
 replacement=root/'next';replacement.symlink_to('outside');replacement.replace(current)
 assert current.read_bytes()==b'replaced' and os.read(opened,100)==b'approved'
 finally:os.close(opened)
 print('a checked path string can resolve differently later; an already opened descriptor keeps the prior object')
 try:read_no_follow(current)
 except OSError as error:assert error.errno==errno.ELOOP,error
 else:raise AssertionError('O_NOFOLLOW accepted a final-component symlink')
 alias=root/'parent-alias';alias.symlink_to(first.name,target_is_directory=True)
 assert read_no_follow(alias/'data')==b'first'
 print('O_NOFOLLOW rejects the final symlink but still allows symlinks in earlier path components')

The first directory contains a file named data and a link whose stored text is simply data. Reading through the link returns first. lstat inspects the link itself, while stat follows it and reports the target; the fixture confirms these are different inode identities.

After moving the link into the second directory, readlink still returns the same text, but reading through it returns second. The target is now the second directory’s data file. This result demonstrates the resolution base directly, without relying on a changed process working directory to explain the different bytes.

Renaming that target leaves a dangling link. is_symlink and lexists remain true, whereas exists, which follows the link, becomes false. Python’s permissive resolve(strict=False) returns the expected missing target path; strict resolution raises FileNotFoundError. Removing the dangling link leaves the renamed target intact, because unlinking the link does not unlink the target it once referenced.

Next, the script resolves a link to a harmless approved file and opens it. It then replaces the link with one pointing at another fixture file. A new pathname read returns replaced, while the already opened descriptor still returns approved. The earlier check did not constrain the later lookup; the existing descriptor retained the earlier object.

Opening the final link with O_NOFOLLOW fails with ELOOP. However, using that same flag to open parent-alias/data succeeds when parent-alias is a directory symlink. Both cases matter: testing only the final-component rejection could otherwise encourage an incorrect claim that this flag protects the entire path from symbolic-link traversal.

Choose the object and the traversal policy

Use readlink when the question concerns stored link text, lstat when it concerns the link object, and following operations when it concerns the current target. Record which identity a diagnostic reports. A log showing an absolute resolved pathname is helpful context, but is still an observation at a particular time.

For operations within a trusted directory boundary, design traversal around directory descriptors and restrictions appropriate to the threat model. Linux openat2 offers resolution policies such as rejecting symbolic links or constraining traversal beneath a directory. Their exact behavior and kernel support must be checked; merely joining a trusted prefix with untrusted text does not establish containment.

An opened descriptor stabilizes the referenced object against later pathname replacement, but it does not make the object’s contents immutable or establish that opening it was authorized. Validate the selected object and perform the intended operation through the appropriate reference. Keep path-selection guarantees, access permissions, and content consistency as separate properties of the design.

Sources

symlink(7), stored target strings, relative resolution, and dangling links, checked 2026-09-12.
open(2), O_NOFOLLOW and descriptor references after pathname changes, checked 2026-09-12.
path_resolution(7), component-by-component traversal and directory permissions, checked 2026-09-12.
openat2(2), kernel-enforced resolution restrictions for controlled traversal, checked 2026-09-12.

NFS shares a remote directory over the network

Wed, 23 Jun 2027 00:00:00 +0000

An NFS export is a directory on the server plus a list of clients allowed to reach it; the file that describes it is not what serves it. /etc/exports is read by exportfs, which writes the master export table to /var/lib/nfs/etab, which rpc.mountd consults on a client’s MOUNT request. Three files: one that people edit, two that decide what happens.

That indirection produces the most common NFS incident: the export list looks right and the client is refused, because the file changed and the table did not. The rest of the configuration comes down to two questions — which identity the server thinks a request carries, and whether a reply that said “written” survives a crash.

The file and the table

An /etc/exports line is <directory> <client>[(<options>)] ..., so /srv/data 10.0.0.0/24(rw,sync,no_subtree_check,root_squash) exports one directory to one network with four options, and another client group can follow on the same line with its own options. Clients are hostnames, wildcards, CIDR networks or netgroups; comments run to end of line, and a trailing backslash continues a line. Nothing in the file is live until exportfs -ra rebuilds the table from it, and exportfs -v prints the table that is actually loaded.

showmount -e shows that same table from the client side: it asks the mount daemon, not the file, so it answers “what does the server think it is exporting”, not “what did I just write”. The fixture reproduces the trap: after appending /srv/newshare without re-exporting, the file has four entries and the table three, so a client asking for it gets NOENT.

What the options actually decide

ro and rw set the write direction, and read-only is the default: without rw, any request that changes the filesystem is disallowed. sync and async decide when the server may answer — with sync the reply follows the commit to stable storage, with async it may precede the commit, which the documentation says allows a crash to lose acknowledged writes. async was the default up to nfs-utils 1.0.0; sync has been the default since.

The squash options are the identity policy. root_squash maps a remote uid 0 to the anonymous uid, which is what you want almost everywhere; no_root_squash lets a client’s root be root on the server, defensible only for a host you trust; all_squash with anonuid/anongid maps every client id to one anonymous identity, the right shape for a public share. subtree_check versus no_subtree_check is a reliability knob, not a security one: subtree checking verifies that a requested file is still inside the exported subtree, which can break access to files renamed while open; since nfs-utils 1.1.0 the default is no_subtree_check.

Mounting, and the id mapping trap

mount -t nfs -o vers=4.2 server:/srv/data /mnt/data attaches the export; with NFSv4 you export once with fsid=0 and mount subtrees by path. Prefer hard — the default — because a soft mount gives up after retrans retransmissions and returns EIO, which the man page warns can corrupt data silently; the price is that a dead server turns an ls into a process stuck in D state. intr no longer helps: it has been ignored since kernel 2.6.25.

Permissions are decided by numeric uid and gid against the server’s accounts, not by names, so a client whose id has no counterpart there lands on the anonymous uid. id on both sides plus ls -ln on the mount is the check: a file you own showing as 65534 means the mapping is wrong, not the mode.

When it breaks

The classic failure is the stale export table, in both directions. A share added and never exported is invisible; a share removed and never re-exported is still served, to clients you believe you locked out. Both are fixed by exportfs -ra, and both are visible by diffing /etc/exports against exportfs -v.

The second failure is the hung client. A hard mount against an unreachable server blocks in the kernel indefinitely, so every process touching the mount is unkillable and the machine accumulates D-state processes; the fixture’s showmount -e 127.0.0.1 took 35 seconds to fail, the same RPC timeout that makes a bad mount look like a frozen ls. The tempting fix, soft, trades the hang for silent write corruption.

Check it quickly

Compare the loaded table against the file, then check the ids:

exportfs -v
showmount -e localhost
exportfs -ra && exportfs -v
mount | grep 'type nfs'
id; ls -ln /mnt/data | head

exportfs -v must list exactly the directories and options in /etc/exports; anything missing means the table is stale and clients get NOENT while the file looks correct. showmount -e should print the same directories to a remote caller. On the client, id and ls -ln must agree on the numeric owner — 65534 or nobody means the server has no matching account.

Sources

exports(5), export syntax, squash options, sync/async and the subtree_check defaults, checked 2026-09-16.
exportfs(8), the master export table in /var/lib/nfs/etab and how exportfs maintains it, checked 2026-09-16.
nfs(5), mount options including hard/soft, timeo and the obsolescence of intr, checked 2026-09-16.
showmount(8), querying the server’s export list from a client, checked 2026-09-16.

A read-only root restricts filesystem mutation within its scope

Wed, 23 Dec 2026 00:00:00 +0000

A read-only root filesystem prevents ordinary writes to the filesystem layer containing a service’s packaged code and configuration. Explicit mounts can still provide writable locations for runtime data. This reduces accidental modification and some persistence opportunities, but its boundary is filesystem mutation. It does not by itself remove process privileges, prevent reading available data, or block communication with other services.

When it breaks

Applications often write outside their main data directory: temporary files, caches, lock files, generated configuration, or runtime state can appear during startup and uncommon operations. Making the root read-only can expose those assumptions as EROFS failures. Testing only a health endpoint may miss the path that generates a report, refreshes a cache, or performs shutdown cleanup.

A writable mount remains writable even when the container root layer is read-only. It can contain state that a compromised or faulty process can corrupt. Broadly mounting a writable directory over packaged paths can also defeat the intended protection for those paths. Inspect the effective mount layout rather than inferring the entire pathname tree’s behavior from one root-level option.

Read-only and confidential are different properties. A process may still read packaged files, execute existing programs, and send data through allowed network paths. The setting also does not guarantee protection after a container escape or against authority capable of changing the mount arrangement. Its contribution must be understood alongside credentials, capabilities, isolation, and application behavior.

Check it quickly

Run this Python script as root inside a disposable Linux container started with a read-only root, a writable tmpfs at /tmp, and a separate writable volume at /work. Python 3 must already be in the image. The recorded environment used Linux 6.12.76 and Python 3.11.2, with no host bind mounts, no privileged mode, and no external network:

from pathlib import Path
from tempfile import TemporaryDirectory
import errno,os,socket,subprocess,sys,uuid

assert os.geteuid()==0,'run inside a disposable read-only-root container as root'
assert os.statvfs('/').f_flag&os.ST_RDONLY
name='xbcatm-readonly-'+uuid.uuid4().hex
for base in [Path('/'),Path('/usr'),Path('/etc')]:
 path=base/name
 try:fd=os.open(path,os.O_WRONLY|os.O_CREAT|os.O_EXCL,0o600)
 except OSError as error:assert error.errno==errno.EROFS,error
 else:
 os.close(fd);path.unlink();raise AssertionError('root filesystem accepted a new file')
print('even this root process receives EROFS on paths in the read-only root, usr and etc tree')
for base in ['/tmp','/work']:
 assert not os.statvfs(base).f_flag&os.ST_RDONLY
 with TemporaryDirectory(prefix='xbcatm-write-',dir=base) as directory:
 path=Path(directory)/'data';path.write_bytes(b'allowed');assert path.read_bytes()==b'allowed'
 path.write_bytes(b'changed');assert path.read_bytes()==b'changed'
print('explicit tmpfs and data-volume mounts remain writable, including overwriting their contents')
assert Path('/etc/os-release').read_text()
result=subprocess.run([sys.executable,'-c','print("executed")'],capture_output=True,text=True,check=True)
assert result.stdout.strip()=='executed'
with socket.socket() as listener,socket.socket() as client:
 listener.bind(('127.0.0.1',0));listener.listen(1);client.connect(listener.getsockname())
 connection,_=listener.accept()
 with connection:
 client.sendall(b'local');assert connection.recv(5)==b'local'
print('reading packaged files, executing an existing program, and loopback network I/O still work')

The script first checks the root filesystem’s reported read-only flag. It then attempts exclusive creation of uniquely named files under the root, /usr, and /etc. Each fails with EROFS, even for this root identity. Exclusive creation and unique names ensure the probes cannot overwrite a preexisting configuration file.

The fixture next checks that /tmp and /work do not report that flag. Temporary files can be created, read, and overwritten in both mounts. This confirms that writable exceptions remain operational and also establishes their exposure: the root setting does not protect the contents of those deliberately writable locations against the same process.

It reads /etc/os-release and executes an existing Python interpreter successfully. Those results distinguish read-only storage from a policy that blocks reads or execution. Disallowing modification of packaged files can be useful, but it is not a general restriction on what existing code may do with its remaining authority.

Finally, a listener and client exchange a harmless payload over loopback. The container has no external networking, but local socket operations still work. The result shows that the filesystem option does not itself disable networking. It does not establish an external connectivity policy or test communication with any production service.

Every writable probe lives in a temporary directory that is removed afterward. The example verifies selected runtime operations in the actual mount arrangement. It does not claim that every application path has been exercised merely because the root probes fail, nor that arbitrary future writable mounts would inherit the same restriction.

Separate packaged files, scratch space, and persistent state

Choose writable paths from observed application needs and configure the application to use them. Keep the exceptions narrow enough to explain their purpose. A failure should lead to identifying the required path and lifetime, rather than reopening the whole root layer to accommodate one unexpected cache or temporary file.

Use tmpfs for suitable temporary data and account for its memory use and lifecycle. Use a separately managed volume when state needs a lifetime independent of the container’s writable layer, with appropriate backup and recovery. A writable volume supplies a storage location; it does not automatically establish crash durability, confidentiality, or a tested restore procedure.

For systemd-managed services, mechanisms such as ProtectSystem and designated state or runtime directories express related filesystem restrictions with their own documented semantics. Validate the service’s actual view and identity rather than translating an option name mechanically between platforms. Exercise startup, ordinary work, exceptional operations, and cleanup, and confirm both prohibited writes and required writes behave as intended.

Sources

Docker container run reference, read-only root filesystems and separately mounted writable paths, checked 2026-09-12.
Docker tmpfs mounts, temporary mount storage and lifecycle, checked 2026-09-12.
Docker volumes, volume lifetime independent of a container root layer, checked 2026-09-12.
systemd 252 execution environment, ProtectSystem and explicitly writable service directories, checked 2026-09-12.

Give durable state and scratch space different lifecycles

Thu, 03 Dec 2026 00:00:00 +0000

Durable application state and disposable scratch files need different retention, capacity, and recovery policies. Separating their paths makes those policies easier to express, but directories alone do not create persistence or isolate disk capacity. The storage backing each path determines what survives process restart, container replacement, host failure, or cleanup. A reliable layout connects each class of data to an explicit lifecycle.

When it breaks

A cleanup job that treats one mixed directory as disposable can remove committed records alongside temporary downloads. The reverse problem is backing up large rebuildable caches while missing the small database that actually matters. File extensions and age are weak substitutes for knowing which application owns a path and whether its contents can be reconstructed from an authoritative source.

Container restart and replacement also differ. Stopping and starting the same container normally retains its writable filesystem layer. Removing that container discards the layer, while an explicitly retained named volume can survive. A tmpfs mount has another lifecycle and loses its contents when the container stops. Calling all three locations temporary hides distinctions that affect real recovery procedures.

Two separate directories on the same full filesystem still compete for that filesystem’s free space. Moving scratch from /data/tmp to /scratch without changing storage policy does not automatically protect database writes. Capacity isolation requires an appropriate mount, quota, or other enforceable budget, together with monitoring that distinguishes scratch growth from durable-state growth.

Check it quickly

Run this script using host Python and Docker after obtaining the pinned image. The recorded environment used Docker 29.3.1 and Linux arm64 Python 3.11.16 with its SQLite module. It creates one private named volume and an eight-mebibyte tmpfs scratch mount. Only its own containers and volume are removed; the small scratch exhaustion stays inside the experiment:

import subprocess,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],text=True,capture_output=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def execute(container,code):return docker('exec',container,'python','-c',code).stdout.strip()

name='xbcatm-state-'+uuid.uuid4().hex[:10];volume=name+'-data';containers=[name+'-one',name+'-replacement']
try:
 docker('volume','create',volume)
 def start(container):
 docker('run','-d','--name',container,'--network','none','--mount','type=volume,src='+volume+',dst=/state','--tmpfs','/scratch:rw,size=8m',BASE,'python','-c','import time;time.sleep(180)')
 start(containers[0])
 execute(containers[0],'from pathlib import Path;import sqlite3;c=sqlite3.connect("/state/data.sqlite");c.execute("create table records(value text)");c.execute("insert into records values (?)",("committed",));c.commit();c.close();Path("/ephemeral").write_text("container-layer");Path("/scratch/crumb").write_text("scratch")')
 docker('stop','--time','1',containers[0]);docker('start',containers[0])
 execute(containers[0],'from pathlib import Path;import sqlite3;assert Path("/ephemeral").exists();assert not Path("/scratch/crumb").exists();c=sqlite3.connect("/state/data.sqlite");assert c.execute("select value from records").fetchall()==[("committed",)];c.close()')
 print('stop/start retains the container writable layer and volume data, while the tmpfs scratch mount loses its file')
 docker('rm','-f',containers[0]);start(containers[1])
 execute(containers[1],'from pathlib import Path;import sqlite3;assert not Path("/ephemeral").exists();c=sqlite3.connect("/state/data.sqlite");assert c.execute("select value from records").fetchall()==[("committed",)];c.close()')
 print('replacement loses the old writable layer but retains the named-volume database')
 execute(containers[1],'''import errno,os,sqlite3
fd=os.open('/scratch/fill',os.O_CREAT|os.O_WRONLY,0o600)
try:
 try:
 while True:os.write(fd,b'x'*1048576)
 except OSError as error:assert error.errno==errno.ENOSPC
finally:os.close(fd)
source=sqlite3.connect('/state/data.sqlite');source.execute('insert into records values (?)',('after-scratch-full',));source.commit()
backup=sqlite3.connect('/state/check-copy.sqlite');source.backup(backup);backup.close();source.close()
restored=sqlite3.connect('/state/check-copy.sqlite');assert restored.execute('select value from records order by rowid').fetchall()==[('committed',),('after-scratch-full',)];restored.close()
''')
 print('bounded scratch reaches ENOSPC while state still commits; a SQLite backup copy reopens with both committed records')
finally:
 docker('rm','-f',*containers,check=False);docker('volume','rm',volume,check=False)

The first container creates a SQLite database under /state, commits one record, writes /ephemeral in its ordinary writable layer, and puts a crumb under /scratch. The three writes have distinct storage locations even though the process accesses all of them through ordinary filesystem paths. SQLite commit is used for the durable application record rather than relying on an unclosed output buffer.

After stopping and starting that same container, the database record and /ephemeral still exist, but the scratch crumb is absent. The container command does not recreate these files during startup, so the observation is about storage lifetime rather than initialization hiding the loss. This distinguishes a process restart within a retained container from destruction of its writable layer.

The original container is then removed, and a replacement mounts the same named volume. Its database still contains committed, while /ephemeral is absent. The replacement gets a new writable layer but reconnects to the retained volume. This demonstrates the volume’s independence from one container’s lifecycle, without claiming protection against losing the Docker host or deleting the volume itself.

Next, the replacement fills its bounded scratch mount until a write reports ENOSPC. It can still insert and commit after-scratch-full into the database on the volume. The successful commit demonstrates the particular capacity boundary configured here. It does not imply unlimited durable storage or protection from exhausting some other shared resource, such as total host memory.

Finally, SQLite’s backup API creates a database copy that is closed, reopened, and checked for both committed records in order. This uses the database’s copying interface instead of assuming that copying one live database filename captures every required file consistently. The copy remains on the same experimental volume, so it validates copy readability and content, not an independent disaster-recovery backup.

Make policy follow the data contract

Use a stable state directory for authoritative application data and a separately managed scratch location for reconstructable work. The Filesystem Hierarchy Standard distinguishes application state under /var/lib; container paths can differ while preserving that semantic distinction. Name the directories in application configuration and document which component owns creation, permissions, cleanup, and migration.

Set scratch budgets deliberately and handle exhaustion without deleting unrelated state. A cache is disposable only if its source and reconstruction procedure are actually available. Some apparently temporary files are required for an in-progress transaction or recovery, so cleanup needs the application’s contract rather than an indiscriminate age rule.

Keep independent backups with retention appropriate to the data, and test restoring into a fresh environment. A volume surviving replacement is persistence evidence, not backup evidence; a same-volume copy cannot survive loss of that volume. Monitor the actual backing stores, verify restore contents, and make destructive volume removal an explicit lifecycle decision.

Sources

Docker volumes, data lifecycle independent of container writable layers, checked 2026-09-12.
Docker tmpfs mounts, temporary in-memory mounts and size limits, checked 2026-09-12.
Filesystem Hierarchy Standard: /var/lib, variable state information retained for applications, checked 2026-09-12.
SQLite online backup API, consistent database copying through the database API, checked 2026-09-12.

Filesystem · XB Field Notes

Extended attributes belong to the inode and to a namespace

When it breaks

Check it quickly

Preserve meaning across the workflow

Sources

Btrfs snapshots are copy-on-write, not copies

Shared extents, not copied blocks

Where the space actually goes

A restore point is not a replica

When it breaks

Check it quickly

Sources

Inotify reports changes, but it is not a complete history

When it breaks

Check it quickly

Reconcile names and object lifetimes

Sources

A sparse file separates logical length from allocated blocks

When it breaks

Check it quickly

Budget and preserve the intended representation

Sources

Hard links share an inode until a name is replaced

When it breaks

Check it quickly

Inspect identity and update behavior

Sources

A symbolic link stores a path, not a permanent target identity

When it breaks

Check it quickly

Choose the object and the traversal policy

Sources

NFS shares a remote directory over the network

The file and the table

What the options actually decide

Mounting, and the id mapping trap

When it breaks

Check it quickly

Sources

A read-only root restricts filesystem mutation within its scope

When it breaks

Check it quickly

Separate packaged files, scratch space, and persistent state

Further reading

Sources

Give durable state and scratch space different lifecycles

When it breaks

Check it quickly

Make policy follow the data contract

Further reading

Sources