Filesystems · XB Field Notes

File locks protect a shared protocol and a specific object

Sat, 05 Dec 2026 00:00:00 +0000

A file lock coordinates participants that agree on what to lock, when to acquire it, and which operations belong inside the critical section. Linux offers several lock interfaces with different ownership and release rules. Choosing a filename and calling something named lock is insufficient: the protocol must identify the underlying object, the lock family, and the lifetime of every reference involved.

When it breaks

On the local Linux filesystem used here, flock provides advisory exclusion. Another process that requests an incompatible lock waits or fails, while an ordinary write that never requests the lock can proceed. The lock therefore does not grant content privacy or automatically constrain every program capable of opening the file.

Locking a data file becomes subtle when updates replace its pathname with another inode. The existing lock remains associated with the old open file description. A later opener reaches the replacement and can acquire a different lock. Both writers may believe they have exclusive access to the same logical document while actually locking different objects.

Traditional POSIX record locks have another surprising lifetime rule: they belong to the process, and closing any descriptor in that process referring to the file releases its record locks on that file. A library that independently opens and closes the same inode can therefore invalidate a lock held elsewhere in the process. This differs from flock ownership through an open file description.

Check it quickly

Save this as locking.py and run python3 locking.py in a disposable Linux environment. The recorded run used Linux 6.12.76, Python 3.11.2, and a local tmpfs. Each competing lock request runs in a separate process and uses nonblocking acquisition, so an expected conflict produces a result instead of hanging:

from pathlib import Path
from tempfile import TemporaryDirectory
import fcntl,os,subprocess,sys

probe_code=r'''
import fcntl,sys
with open(sys.argv[1],'r+') as handle:
 try:
 if sys.argv[2]=='flock':fcntl.flock(handle,fcntl.LOCK_EX|fcntl.LOCK_NB)
 else:fcntl.lockf(handle,fcntl.LOCK_EX|fcntl.LOCK_NB,1,int(sys.argv[3]),0)
 except BlockingIOError:print('blocked')
 else:print('acquired')
'''
def probe(path,kind='flock',start=0):
 result=subprocess.run([sys.executable,'-c',probe_code,str(path),kind,str(start)],capture_output=True,text=True,timeout=5)
 assert result.returncode==0,result.stderr
 return result.stdout.strip()
with TemporaryDirectory(prefix='xbcatm-locks-') as directory:
 root=Path(directory);target=root/'data';target.write_text('original')
 handle=target.open('r+');fcntl.flock(handle,fcntl.LOCK_EX)
 assert probe(target)=='blocked'
 subprocess.run([sys.executable,'-c',"from pathlib import Path;import sys;Path(sys.argv[1]).write_text('uncooperative')",str(target)],capture_output=True,text=True,check=True)
 handle.seek(0);assert handle.read()=='uncooperative'
 duplicate=os.dup(handle.fileno());handle.close()
 try:assert probe(target)=='blocked'
 finally:os.close(duplicate)
 assert probe(target)=='acquired'
 print('flock blocks cooperating contenders but not ordinary writes; dup retains the same lock')
 with target.open('r+') as handle:
 fcntl.flock(handle,fcntl.LOCK_EX)
 replacement=root/'replacement';replacement.write_text('new inode');replacement.replace(target)
 assert os.fstat(handle.fileno()).st_ino!=target.stat().st_ino
 assert probe(target)=='acquired'
 lockfile=root/'stable.lock';lockfile.touch()
 with lockfile.open('r+') as handle:
 fcntl.flock(handle,fcntl.LOCK_EX)
 replacement.write_text('next inode');replacement.replace(target)
 assert probe(lockfile)=='blocked'
 assert probe(lockfile)=='acquired'
 print('replacing locked data changes the inode; a separate stable lock file protects that replacement protocol')
 first=os.open(target,os.O_RDWR);second=os.open(target,os.O_RDWR)
 try:
 fcntl.lockf(first,fcntl.LOCK_EX|fcntl.LOCK_NB,1,0,0)
 assert probe(target,'record',0)=='blocked' and probe(target,'record',1)=='acquired'
 os.close(second);second=None
 assert probe(target,'record',0)=='acquired'
 print('POSIX record locks permit disjoint bytes; closing another descriptor to the inode releases process-owned locks')
 finally:
 if second is not None:os.close(second)
 os.close(first)

First, the parent holds an exclusive flock, and a cooperating child reports blocked. Another child writes without requesting any lock, and the parent reads the changed content. These two outcomes establish both the useful exclusion and its advisory boundary using the same file and permissions.

The parent then duplicates its descriptor and closes the original wrapper. A contender remains blocked because the duplicate still references the locked open file description. Closing that final duplicate releases the lock, and the next contender succeeds. Counting wrapper objects or relying on the first close would describe the lifetime incorrectly.

Next, the parent locks the data file and atomically replaces its pathname with a newly created file. The descriptor’s inode differs from the inode now found by pathname. A child opens that pathname and acquires its own lock. Atomic replacement keeps the name available, but it does not move the old lock onto the replacement inode.

The following case uses a separate stable.lock file. Writers agree to lock that stable object before replacing the data pathname. The contender remains blocked across replacement and succeeds after release. This works only while participants keep the lock file’s identity stable; deleting and recreating that file can reproduce the original split-object failure.

Finally, lockf exercises traditional POSIX record locking. Locking byte zero blocks a child requesting that byte but allows a child requesting byte one. Closing the parent’s second, independently opened descriptor releases the byte-zero lock, even though the descriptor used to acquire it remains open. The successful final probe makes this release rule observable.

Make the protocol explicit

Use the same lock family and object across all participants. On local Linux filesystems, traditional record locks and flock are normally separate mechanisms, so mixing them does not establish mutual exclusion. NFS and SMB can have different emulation and interaction rules; verify the deployed filesystem and mount behavior before applying local results there.

Choose a range that covers the entire invariant being updated. Separate record ranges only permit useful concurrency when operations really are independent; a shared header or index can require broader coordination. Keep unrelated slow work outside the critical section, and use a consistent acquisition order when an operation needs multiple locks.

Define cleanup for normal return, exceptions, and descriptor transfer. Process termination releases kernel-held locks according to their ownership rules, but duplicated or inherited references can extend some lifetimes. Locks also do not make writes durable or repair partially completed updates. Pair exclusion with an update and recovery protocol appropriate to the file’s data format.

Sources

flock(2), advisory locks, open file descriptions, and remote-filesystem caveats, checked 2026-09-12.
fcntl_locking(2), POSIX record ranges and process-owned lock release rules, checked 2026-09-12.
rename(2), pathname replacement and the persistence of open references, checked 2026-09-12.

Fsync completes a storage request within an update protocol

Fri, 27 Nov 2026 00:00:00 +0000

A successful buffered write makes bytes available through the kernel’s file state, but does not by itself establish crash durability. fsync requests synchronization of a file’s data and associated metadata and waits for completion as reported by the storage stack. It is one operation in an update protocol. Correct ordering, checked errors, directory entries, and the storage system’s actual guarantees still matter.

When it breaks

User-space buffering creates an earlier boundary. Calling fsync on a descriptor cannot synchronize bytes that a language-level buffer has not yet submitted to the kernel. Flush that buffer first, and check errors from both stages. Ordinary writes can also complete partially, and delayed storage errors may surface during synchronization instead of at the original write.

Synchronizing a newly created file does not necessarily synchronize the directory entry that names it. On Linux, a durable creation or replacement protocol also needs the appropriate directory synchronization. Atomic rename provides a namespace visibility property while the system is running; it does not independently establish that the directory change survived a crash.

fdatasync does not mean no metadata. It synchronizes data and metadata needed for subsequent data retrieval, including a changed file length. It can omit other metadata updates that are not needed for that purpose. Treating it as either identical to fsync in every respect or completely indifferent to metadata obscures the reason for choosing between them.

Check it quickly

Save this as syncfile.py and run python3 syncfile.py on a disposable Linux system with Python and strace. The temporary directory must support regular-file and directory synchronization. The recorded run used Linux 6.12.76, Python 3.11.2, strace 6.1, and an ext4 volume. The example checks real calls and runtime results; it does not cut power to the storage:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess,sys

program=r'''
from pathlib import Path
import errno,os,sys
root=Path(sys.argv[1]);current=root/'current';candidate=root/'candidate'
current.write_bytes(b'old');old=current.open('rb')
try:
 with candidate.open('wb') as handle:
 handle.write(b'new');handle.flush();os.fsync(handle.fileno())
 os.replace(candidate,current)
 directory=os.open(root,os.O_RDONLY|os.O_DIRECTORY)
 try:os.fsync(directory)
 finally:os.close(directory)
 assert current.read_bytes()==b'new' and old.read()==b'old'
 with current.open('ab') as handle:
 handle.write(b'-extended');handle.flush();os.fdatasync(handle.fileno())
 assert current.read_bytes()==b'new-extended' and current.stat().st_size==12
 read_end,write_end=os.pipe()
 try:
 try:os.fsync(write_end)
 except OSError as error:assert error.errno==errno.EINVAL
 else:raise AssertionError('fsync on a pipe unexpectedly succeeded')
 finally:os.close(read_end);os.close(write_end)
finally:old.close()
print('replacement content, old open reference, appended length, and unsupported-pipe error verified')
'''
with TemporaryDirectory(prefix='xbcatm-fsync-') as directory:
 root=Path(directory);trace=root/'trace'
 result=subprocess.run(['strace','-yy','-s','200','-e','trace=%file,write,fsync,fdatasync','-o',str(trace),'--',sys.executable,'-c',program,str(root)],capture_output=True,text=True,timeout=10)
 assert result.returncode==0,result.stderr
 lines=trace.read_text().splitlines()
 file_sync=next(i for i,line in enumerate(lines) if 'fsync(' in line and str(root/'candidate') in line and '= 0' in line)
 rename=next(i for i,line in enumerate(lines) if line.startswith('rename') and str(root/'current') in line and '= 0' in line)
 directory_sync=next(i for i,line in enumerate(lines) if 'fsync(' in line and '<'+str(root)+'>' in line and '= 0' in line)
 assert file_sync<rename<directory_sync
 assert any('fdatasync(' in line and str(root/'current') in line and '= 0' in line for line in lines)
 print(result.stdout,end='')
 for line in lines:
 if line.startswith(('fsync(','fdatasync(','rename')):print(line)
 print('syscall order verified: flush data, fsync candidate, replace name, fsync parent directory')

The child creates a current file containing old and keeps an open read reference. It writes new to a separate candidate, flushes the Python buffer, and calls fsync on that candidate. Keeping the candidate in the same directory also keeps the later rename within the same filesystem for this exercise.

It then replaces the current pathname and synchronizes an open descriptor for the containing directory. The trace asserts the sequence: candidate fsync succeeds, rename succeeds, then parent-directory fsync succeeds. These are observed system calls with checked results, rather than a source-code comment claiming that the intended synchronization order was followed.

Reading the current path returns new, while the reference opened earlier still reads old. This distinguishes atomic name replacement from changing every existing open reference. The old reference remains attached to its earlier file object. Readers that need a new version must select it according to the application’s publication protocol.

The child next appends -extended, flushes its buffer, and calls fdatasync. The resulting content is new-extended with length twelve. This exercises successful data synchronization after a size-changing operation, consistent with the contract that metadata needed to retrieve the extended data cannot simply be ignored.

Finally, fsync on a pipe fails with EINVAL. The fixture checks that failure rather than treating every descriptor as a synchronizable regular file. All handles close during cleanup. The trace proves that the local kernel accepted the relevant file and directory requests; it does not independently prove that a particular physical device honors flushes during a power failure.

Define the acknowledgment boundary

Specify which update must survive and what the reader may observe. For a single-file replacement, synchronize the candidate before publication and the affected directory afterward. A move involving different directories can require considering both directory updates. Cross-filesystem movement has different mechanics, so a convenient rename-based protocol cannot be assumed to apply unchanged.

If synchronization fails, do not report the durable update as committed. Earlier writes or a rename may already have taken effect, so failure is not automatic rollback. Retain enough transaction or version information to determine the actual state and recover according to the application’s contract. Fsync alone does not make a sequence of changes to multiple files one atomic transaction.

Batching several operations behind a shared commit boundary can reduce synchronization cost when the application contract permits it. Acknowledge only after the required boundary succeeds. Validate crash recovery on the real storage stack, including its filesystem and device-cache behavior, because a virtualized laboratory’s successful syscall trace establishes API behavior without proving every lower layer’s failure guarantees.

Sources

fsync(2), file and directory synchronization, fdatasync metadata, and errors, checked 2026-09-12.
rename(2), atomic pathname replacement and existing open references, checked 2026-09-12.
write(2), buffered write completion, partial writes, and delayed errors, checked 2026-09-12.

Writeback separates buffered visibility from storage completion

Thu, 26 Nov 2026 00:00:00 +0000

Buffered file I/O can pass through a language buffer, the kernel page cache, and the storage stack at different times. Modified cache pages need writeback so their data reaches the backing filesystem and device. This separation can combine work and improve throughput, but it means that another reader seeing new bytes is not, by itself, evidence that a crash-safe storage boundary has completed.

When it breaks

Buffered writes are not guaranteed to return instantly. Allocation, memory pressure, dirty-page throttling, or filesystem activity can make the writer wait. When the producer consistently outruns storage, dirty data cannot grow without limit; backpressure eventually affects the workload. Describing every write as a pure memory copy with all I/O deferred gives the wrong model for latency spikes under sustained load.

Language-level flush and filesystem synchronization have different meanings. Flush submits buffered application data to the next layer. Fsync requests a file synchronization boundary. Closing a normal file or reading the same bytes through another descriptor does not replace that explicit contract. A benchmark measuring only successful writes can therefore measure acceptance into cache instead of sustained durable throughput.

Writeback settings also are not per-file durability deadlines. Background thresholds encourage flusher activity, higher dirty thresholds affect generating processes, and periodic settings govern wakeups or eligibility of older dirty data. Device progress, filesystem behavior, and competing activity still matter. A chosen interval alone does not prove that every acknowledged update is safe after that many seconds.

Check it quickly

Save this as writeback.py and run python3 writeback.py on Linux with Python and strace, using an ext4 temporary directory. The recorded environment used Linux 6.12.76, Python 3.11.2, and strace 6.1. The fixture writes approximately four MiB, reads accounting information, and leaves global dirty-memory settings and cache controls unchanged:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess,sys

program=r'''
from pathlib import Path
import json,os,subprocess,sys
root=Path(sys.argv[1]);path=root/'data'
def io():return {key:int(value) for key,value in (line.split(':',1) for line in Path('/proc/self/io').read_text().splitlines())}
def dirty():return int(next(line.split()[1] for line in Path('/proc/meminfo').read_text().splitlines() if line.startswith('Dirty:')))
with path.open('wb',buffering=65536) as handle:
 handle.write(b'buffered')
 assert path.stat().st_size==0 and path.read_bytes()==b''
 before=io();handle.flush();after=io()
 assert after['wchar']-before['wchar']>=8
 reader='from pathlib import Path;import sys;assert Path(sys.argv[1]).read_bytes()==b"buffered"'
 subprocess.run([sys.executable,'-c',reader,str(path)],check=True)
 print('visible before explicit fsync',flush=True)
 before=io();chunk=b'x'*(4*1024*1024);handle.write(chunk);handle.flush();after=io()
 assert after['wchar']-before['wchar']>=len(chunk)
 assert after['write_bytes']-before['write_bytes']>=len(chunk)
 dirty_before=dirty();os.fsync(handle.fileno());dirty_after=dirty()
 assert path.stat().st_size==len(chunk)+8
 print(json.dumps({'wchar_delta':after['wchar']-before['wchar'],'write_bytes_delta':after['write_bytes']-before['write_bytes'],'system_dirty_kib_before':dirty_before,'system_dirty_kib_after':dirty_after}))
'''
with TemporaryDirectory(prefix='xbcatm-writeback-') as directory:
 root=Path(directory);trace=root/'trace'
 result=subprocess.run(['strace','-f','-yy','-s','200','-e','trace=write,fsync,fdatasync','-o',str(trace),'--',sys.executable,'-c',program,str(root)],capture_output=True,text=True,timeout=10)
 assert result.returncode==0,result.stderr
 lines=trace.read_text().splitlines()
 visible=next(i for i,line in enumerate(lines) if 'visible before explicit fsync' in line)
 syncs=[i for i,line in enumerate(lines) if 'fsync(' in line and str(root/'data') in line]
 assert len(syncs)==1 and visible<syncs[0] and '= 0' in lines[syncs[0]]
 print(result.stdout,end='');print(lines[syncs[0]])
 print('another process read flushed bytes before the only explicit file synchronization; global Dirty is observational, not an isolated-device counter')

The child opens a file with a 65536-byte Python buffer and writes eight bytes. Before flushing, the pathname reports length zero and a separate open reads no content. The bytes are still in the application buffer. This establishes the earliest boundary directly instead of assuming every language-level write has already issued a kernel write.

After flush, the process’s successful-write byte count increases, and a separate child process reads the expected bytes. The writer prints a marker before its only explicit fsync. The strace output confirms that this visibility marker precedes synchronization. The result proves that another process can observe new data before the application requests that storage boundary.

The writer then submits four MiB more and checks increases in both wchar and write_bytes. These counters answer different accounting questions: successful write-family bytes versus storage-write activity attributed to the process. They are not interchangeable with business payload counts, device-completion timestamps, or a certificate that the corresponding update will survive a power failure.

The fixture records the system Dirty value around the successful fsync, then checks the final file length. In the recorded run, the values were 4148 and 56 KiB. Those numbers are observations of an aggregate kernel view, not isolated measurements of this file’s dirty pages. Other activity and normal writeback can change them, so the test does not assert a universal decrease or exact result.

All generated files are temporary, and the trace checks that the one explicit file synchronization succeeded. This experiment establishes buffering, visibility, and API ordering. It does not test a power cut, predict a device’s maximum write latency, or prove that one aggregate counter can explain every workload’s storage behavior.

Measure the boundary the workload needs

Track write latency, synchronization latency, dirty and writeback state, and the relevant device or cgroup activity together. A fast initial burst followed by stalls can indicate cache absorption followed by constrained drainage. A test should run long enough to exercise the sustained path relevant to the workload rather than reporting only the cache’s initial acceptance speed.

Interpret the configured dirty thresholds carefully. The ratio settings use eligible available memory, not simply all installed RAM, and byte-based alternatives express a different kind of limit. Read the effective settings before proposing changes. Reducing accumulation may improve one latency pattern while increasing write activity; increasing it can postpone pressure without increasing device throughput.

Let the application define when data may be acknowledged as committed. A database can group commits behind an appropriate log synchronization, while a replaceable cache may tolerate loss. Background writeback improves the ordinary buffered path, but it does not invent that business contract. Check write and synchronization errors, and validate recovery according to the promised persistence boundary.

Sources

write(2), buffered write visibility, completion, and delayed storage errors, checked 2026-09-12.
Linux virtual-memory sysctl documentation, dirty thresholds, background writeback, and periodic timing, checked 2026-09-12.
proc_pid_io(5), process I/O accounting and distinctions between byte counters, checked 2026-09-12.
proc_meminfo(5), Dirty and Writeback memory observations, checked 2026-09-12.
fsync(2), explicit file synchronization and failure reporting, checked 2026-09-12.