Storage · XB Field Notes

LVM snapshots capture a block device in time

Mon, 10 Apr 2028 00:00:00 +0000

An LVM snapshot is neither a copy nor a reflink: it is a fixed-size copy-on-write exception store attached to an origin logical volume. lvcreate -s -L 5G -n snap vg/data allocates 5 GiB from the volume group, and the device-mapper snapshot target records the original contents of each origin chunk the first time it is written. Creation is instant whatever the origin’s size, because nothing is copied.

That size is the whole safety margin, and it is the parameter people get wrong. A store sized for a quiet week is exhausted in an hour by a database checkpoint or an rsync, and when it fills the snapshot does not degrade — it becomes invalid and returns I/O errors. The recovery point disappears exactly when you needed it, which is why the store’s free space belongs on a dashboard, not in a runbook.

An exception store, not a copy

A classic LVM snapshot uses two device-mapper targets: the origin is re-mapped as snapshot-origin, and a new snapshot device combines it with a COW device. Reads of unchanged chunks go straight to the origin; the first write to a chunk copies the old chunk into the COW device, then writes the new data. So consumption tracks the number of distinct chunks that changed, not the number of writes: the simulation writes a million times to sixteen hot chunks and consumes fifteen exceptions, about 60 KiB.

Watch it with lvs -a -o lv_name,lv_attr,origin,data_percent,lv_size. A classic snapshot’s lv_attr reads swi-a-s--- — s for snapshot, w writable, and a state character that becomes I when the store is exhausted. data_percent is the store’s fill, and that is the number to alert on, at 70 or 80 percent; 100 percent is not a warning, it is an obituary. lvextend grows the COW device while the snapshot is still valid; lvchange --monitor y lets dmeventd do it automatically.

Exhaustion is invalidation

The kernel documentation is blunt: the COW device will often be smaller than the origin, and if it fills up the snapshot becomes useless and is disabled, returning errors. LVM’s guidance is to plan for expected writes — often 20 percent of the origin — and extend before it fills. A small slice of the store holds the COW block map, so usable data capacity is a little under the -L value.

The blast radius is narrow but the timing is cruel: the origin keeps running normally, so nothing looks wrong until the backup that reads the snapshot fails or the rollback is unavailable. There is no partial mode: at 100 percent, the snapshot never existed.

Merging, and why thin changes the failure

lvconvert --merge vg/snap rolls the snapshot’s changed chunks back into the origin, and it runs in the background while the origin stays usable — writes to chunks not yet merged are deferred. The constraint is on the origin: it must be inactive when the merge is set up, so unmount it, or schedule the merge and activate the origin afterwards. Merging consumes the snapshot, so copy out anything you need first.

Thin snapshots change the failure mode rather than removing it. A thin snapshot is created when the origin is a thin LV and no size is given; it shares blocks in the thin pool instead of owning a store, so its own divergence cannot invalidate it. The pool is the shared resource now: when it runs out, the damage is not one dead snapshot but every thin volume in the pool, which LVM warns may be damaged and hard to repair. thin_pool_autoextend_threshold and dmeventd exist precisely because pool exhaustion is a pool-wide outage.

When it breaks

The classic failure is a backup that silently stops being a backup. The job mounts the snapshot, copies for longer than the store can cover, and the store reaches 100 percent; the snapshot goes invalid and the copy fails — or worse, monitoring watches only the exit code while the job reads a stale mount. Check data_percent on every snapshot LV, alert at 70 percent, and size for the busiest hour in the retention window.

The second failure is the merge that cannot happen when you need it. lvconvert --merge requires the origin closed, and teams discover that during the incident. Rehearse the merge on a disposable volume, keep a real backup independent of the snapshot, and remember that a snapshot whose store is full cannot be merged.

Check it quickly

Look at the store’s fill before trusting the snapshot:

lvs -a -o lv_name,vg_name,lv_attr,origin,data_percent,lv_size
lvs -a -o lv_name,data_percent,metadata_percent vg/thinpool
dmsetup status vg-snap
lvextend -L +2G vg/snap

data_percent is the COW store’s fill and should sit well under 100; a state character of I in lv_attr means the snapshot is already invalid. For a thin snapshot, data_percent and metadata_percent on the pool are the numbers that matter; dmsetup status shows allocated sectors against the total so you can watch a merge drain.

Sources

lvcreate(8), documents -s/--snapshot, COW versus thin snapshots and the sizing guidance, checked 2026-09-16.
lvs(8), defines the lv_attr state characters including invalid snapshot, checked 2026-09-16.
Device-mapper snapshot support, the exception store, invalidation and lvconvert --merge behaviour, checked 2026-09-16.
lvmthin(7), thin snapshots, pool exhaustion and automatic extension, checked 2026-09-16.

ZFS snapshots give the filesystem a rewind button

Tue, 21 Mar 2028 00:00:00 +0000

A ZFS snapshot is a name attached to a dataset’s block tree at one transaction group. Nothing is copied, nothing is quiesced, and the cost is a few metadata blocks: zfs snapshot tank/data@before-upgrade returns before the shell prompt redraws. Because ZFS never overwrites a live block in place, the old tree stays valid on disk until the last reference to it goes away, and the snapshot is that reference.

That is what makes a nightly wall of history practical, and it is also where this article’s title gets dangerous. Snapshots give you many versions of the data; they do not give you safety from losing the pool. Every snapshot of tank/data lives in tank, so the failure that takes the pool takes the dataset and its history in the same instant. Multiversion and multiversion-safety are different properties, and only replication supplies the second.

A name for one consistent tree

Snapshots are created at a transaction group boundary, so the tree they capture is internally consistent — a database’s files and its metadata freeze at the same instant, not microseconds apart. -r snapshots a dataset and all its descendants, which is what you want for anything holding child datasets; without it, a recursive layout gets snapshots taken at different times at different levels, and incremental replication between them becomes awkward.

Names are the retention policy’s only handle: tank/data@daily-2026-09-16 is prunable by pattern, while an autogenerated name needs bookkeeping. zfs list -t snapshot -o name,used,refer,creation -s creation shows the set with the space each one pins.

Where the space is, in zfs list -o space

ZFS reports space in a column set that answers the only question that matters: whose data is it? zfs list -o space tank/data gives USED, USEDSNAP, USEDDS, USEDREFRESERV, USEDCHILD, REFER and AVAIL. REFER is what the dataset alone needs, USEDSNAP is what its snapshots pin, and USED is the sum. A rising USEDSNAP with a flat REFER means the snapshots, not the data, are eating the pool.

The fixture builds a wall of thirty daily snapshots over a 64 MiB dataset with 2 MiB of churn per day. Creation costs 0 KiB every single time — the cost lands on the write that follows, 2048 KiB for 2 MiB rewritten — and the whole wall costs 60 MiB against 1920 MiB for thirty full copies. That ratio is the entire argument for snapshots, and why nobody notices the pool filling until it is nearly full.

Holds, diff, and comparing versions

A hold is a named reference that makes destruction fail with EBUSY: zfs hold keep tank/data@daily-2026-09-16 protects a snapshot, zfs holds lists the holders, and zfs release keep clears it. Backup software holds the snapshot it is sending so retention cannot delete it; the cost is that pruning then fails until the hold is released, and a stuck hold can pin a snapshot for months.

zfs diff tank/data@daily-2026-09-15 tank/data@daily-2026-09-16 reports per-path changes — + created, - removed, M modified, R renamed — and both snapshots must still exist, which is why diffing against yesterday is easy and against last month may not be. Read a snapshot’s contents through the hidden .zfs/snapshot/<name> directory; there is no separate mount for a snapshot, and snapdir=visible makes it listable.

When it breaks

The failure that matters is the one the title invites: a pool loss. A failed controller, a backplane fault, or pool-wide corruption removes the dataset and every snapshot of it atomically; there is no partial survival. The only defence is a copy outside that pool: zfs send -i tank/data@yesterday tank/data@today | zfs receive backup/data writes an independent, checksummed stream into a second pool, and only that stream survives the first pool’s death. A snapshot wall with no replica is a single point of failure with a nicer interface.

The second failure is silent space creep plus retention that cannot run. USEDSNAP grows while REFER stays flat, the pool passes 80 percent, and ZFS starts refusing writes with no space left even though the datasets look reasonable — slop and reservations are not visible from inside the filesystem. If a snapshot is also held for a clone, destroying it fails, and rolling back past a clone’s origin forces a recursive destroy. Long-lived holds turn retention into a dependency graph, so keep hold lifetimes short and audited.

Check it quickly

See what the snapshots cost and whether they can be removed:

zfs list -t snapshot -o name,used,refer,creation -s creation tank/data
zfs list -o space tank/data
zfs holds tank/data@daily-2026-09-16
zfs diff tank/data@daily-2026-09-15 tank/data@daily-2026-09-16
zfs send -i tank/data@daily-2026-09-15 tank/data@daily-2026-09-16 | zfs receive backup/data

used per snapshot is the space that snapshot alone pins, and a fresh snapshot’s value is near zero; the space columns then show USEDSNAP against REFER, and a rising USEDSNAP with a flat REFER is the signal to prune. If zfs destroy fails with a busy snapshot, zfs holds names the tag to release. The last line is the only one that protects against losing the pool; run it on a schedule.

Sources

zfs-snapshot(8), snapshot creation semantics and recursive snapshots, checked 2026-09-16.
zfs-list(8), the space column set including USEDSNAP, USEDDS and REFER, checked 2026-09-16.
zfs-hold(8), holds, zfs holds, and the EBUSY returned by destroy, checked 2026-09-16.
zfs-send(8), incremental replication streams that leave the pool, checked 2026-09-16.

High-cardinality labels increase stored series before anyone queries them

Mon, 10 Jan 2028 00:00:00 +0000

A Prometheus series is identified by a metric name and its complete label set. Adding customer or request identity to a counter can create many series even when every dashboard immediately sums those identities away. Query aggregation changes the returned result; it does not remove the raw series that were ingested. Storage and ingestion costs arise before a user opens the dashboard.

Count observed combinations, not isolated values

One new label value is not automatically one new series across the entire database. It creates a new identity wherever the metric name and other labels form a distinct combination. Multiplying possible value counts gives a Cartesian upper bound for a given metric and target scope; it equals the observed count only when every combination is present. Dependencies between dimensions can make the bound much larger than reality. Actual series count also depends on target labels and metric families.

A route label based on a small set of route templates can be bounded. A full URL containing identifiers and query strings often is not. Customer identifiers, request IDs, arbitrary error messages and timestamps also need deliberate scrutiny. A histogram with multiple bucket series amplifies the impact of each label combination further. Budget from the actual exposition and expected workload rather than one dimension in isolation.

Querying fewer rows does not store fewer rows

An expression summing requests by route can return three results after reading twelve customer-specific series. Those twelve input series still exist. If route totals are the required metric contract, instrumentation can accumulate a route-level counter directly and export three series. That reduces the identities emitted at the source instead of relying on every consumer to hide the extra dimension.

Removing a label during ingestion is not equivalent to this accumulation. Distinct samples can acquire the same identity, and relabeling will not decide how to sum them. Prefer a deliberate instrumentation change or an explicitly defined recording query, understanding that recording adds derived series while retaining raw ones under the storage policy. Preserve detailed correlation in logs or traces when needed, with their own access, retention and cost limits.

Diagnose growth with the right inventory

The TSDB status API exposes head statistics, including series counts by metric name. Those values help identify large families but are not a complete cost model. Sample interval, label length, churn, chunk handling, retention and storage behavior also matter. Do not translate a small fixture’s series reduction into an identical percentage reduction in production memory or disk.

Instant query counts and head-series statistics can differ when some head series no longer have recent selectable samples. Historical blocks introduce another scope again. Record the measurement time and population when comparing reports. A release that changes label values may create churn while older identities remain in storage, so a graph returning fewer current results does not prove that all earlier cost has vanished.

When it breaks

A newly added unbounded label can increase ingestion and memory pressure until scrapes or queries suffer. Start by identifying the metric family and instrumentation change, then stop unnecessary new identities at their source or through a carefully reviewed sample filter. Immediately deleting historical data is not a universal recovery requirement, and changing retention does not remove every resource cost instantly.

A subtler failure is a successful aggregation that makes a dashboard look small while the raw metric continues growing. Review both exported identities and query output. Give new dimensions an expected value range, useful consumer and owner. For counters, preserve reset handling when creating production rate queries; a static sum used to compare fixture totals is not a replacement for rate-before-aggregation semantics.

Check it quickly

Save this as cardinality.py and run python3 cardinality.py. It requires Docker and Prometheus 3.5.1, downloading the image if needed. A local text endpoint exports three routes with four synthetic customer identities each, plus independently accumulated route totals. The real server scrapes those samples and exposes both query and TSDB statistics.

import http.client
import io
import json
import subprocess
import time
import uuid
from urllib.parse import quote

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
 def __init__(self, data):
 self.data = data

 def makefile(self, mode):
 return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
 body = body.encode()
 headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
 f"Connection: close\r\nContent-Type: {content_type}\r\n"
 f"Content-Length: {len(body)}\r\n\r\n").encode()
 result = subprocess.run(
 ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
 input=headers + body, capture_output=True, timeout=5,
 )
 if result.returncode:
 raise RuntimeError(result.stderr.decode())
 response = http.client.HTTPResponse(ResponseSocket(result.stdout))
 response.begin()
 return response.status, response.read().decode()


def get_json(path):
 status, body = request("GET", path)
 assert status == 200, (status, body)
 return json.loads(body)


def eventually(check):
 for attempt in range(150):
 try:
 value = check()
 if value:
 return value
 except (RuntimeError, AssertionError, http.client.HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Expected local server state did not appear")


def query(expression):
 response = get_json("/api/v1/query?query=" + quote(expression, safe=""))
 assert response["status"] == "success"
 return response["data"]["result"]


def value(expression):
 result = query(expression)
 assert len(result) == 1, (expression, result)
 return float(result[0]["value"][1])

port = 9090
# Requires Docker and the Prometheus 3.5.1 image.
script = r"""set -eu
mkdir -p /tmp/www /tmp/targets
cat > /tmp/prom.yml <<'PROM_CONFIG'
global:
 scrape_interval: 1s
 scrape_timeout: 500ms
scrape_configs:
 - job_name: cardinality
 metrics_path: /metrics.txt
 static_configs:
 - targets: ['127.0.0.1:8080']

PROM_CONFIG
cat > /tmp/www/metrics.txt <<'METRIC_DATA'
# TYPE raw_requests_total counter
# TYPE bounded_requests_total counter
raw_requests_total{route="/orders",customer_id="synthetic-1"} 1
raw_requests_total{route="/orders",customer_id="synthetic-2"} 2
raw_requests_total{route="/orders",customer_id="synthetic-3"} 3
raw_requests_total{route="/orders",customer_id="synthetic-4"} 4
bounded_requests_total{route="/orders"} 10
raw_requests_total{route="/cart",customer_id="synthetic-1"} 1
raw_requests_total{route="/cart",customer_id="synthetic-2"} 2
raw_requests_total{route="/cart",customer_id="synthetic-3"} 3
raw_requests_total{route="/cart",customer_id="synthetic-4"} 4
bounded_requests_total{route="/cart"} 10
raw_requests_total{route="/search",customer_id="synthetic-1"} 1
raw_requests_total{route="/search",customer_id="synthetic-2"} 2
raw_requests_total{route="/search",customer_id="synthetic-3"} 3
raw_requests_total{route="/search",customer_id="synthetic-4"} 4
bounded_requests_total{route="/search"} 10

METRIC_DATA
/bin/httpd -f -p 127.0.0.1:8080 -h /tmp/www > /tmp/httpd.log 2>&1 &
exec /bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data --web.listen-address=127.0.0.1:9090
"""
try:
 subprocess.run(
 ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-c", script],
 check=True, capture_output=True, text=True, timeout=20,
 )
 eventually(lambda: len(query('raw_requests_total')) == 12)
 assert value('count(raw_requests_total)') == 12
 assert value('count(bounded_requests_total)') == 3
 assert value('sum(raw_requests_total)') == value('sum(bounded_requests_total)') == 30
 raw_by_route = {x["metric"]["route"]: float(x["value"][1])
 for x in query('sum by (route) (raw_requests_total)')}
 bounded_by_route = {x["metric"]["route"]: float(x["value"][1])
 for x in query('bounded_requests_total')}
 assert raw_by_route == bounded_by_route
 statistics = get_json('/api/v1/status/tsdb')["data"]
 counts = {x["name"]: x["value"] for x in statistics["seriesCountByMetricName"]}
 assert counts["raw_requests_total"] == 12 and counts["bounded_requests_total"] == 3
 print(json.dumps({"raw_series": 12, "bounded_series": 3, "snapshot_total": 30,
 "per_route": raw_by_route, "tsdb_counts_confirmed": True}, indent=2))
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The assertions confirm twelve raw series, three bounded series, and a total of thirty in both representations. Each route totals ten. After query aggregation, the TSDB still reports twelve raw series, demonstrating that a smaller result has not erased its inputs. The isolated container uses no external network, host mounts or published ports and is removed on exit. This validates identities and arithmetic, not a production capacity benchmark.

Sources

Prometheus: data model, series identity is the metric name plus its label set, checked 2026-09-13.
Prometheus: metric and label naming, bounded dimensions and high-cardinality label guidance, checked 2026-09-13.
Prometheus: storage, head data, blocks and retention costs, checked 2026-09-13.
Prometheus: HTTP API, head-series statistics by metric name, checked 2026-09-13.

A checkpoint flushes dirty buffers to disk

Thu, 06 Jan 2028 00:00:00 +0000

A checkpoint establishes a starting point for PostgreSQL crash recovery. It synchronizes the data-file changes required for that checkpoint and records the corresponding recovery information. WAL must reach durable storage before dependent data pages are written. This ordering allows committed work to survive without forcing every modified table page to disk at each commit, while recovery can replay later changes when necessary.

When it breaks

A checkpoint is not a transaction commit. Data pages can contain versions written by transactions that remain uncommitted; visibility and transaction status still determine which rows readers can see. Checkpoint completion does not make those writes visible or turn a raw copy of changing database files into a valid base backup. Concurrent activity can also dirty pages again, so completion does not mean the buffer pool stays completely clean.

WAL retention has other constraints. max_wal_size is a soft target associated with automatic checkpoint scheduling, not a hard disk-usage limit. Archive failure and replication-slot retention can keep WAL beyond it. min_wal_size concerns recycling a reserve of WAL files, not a second threshold that forces a checkpoint. Misreading those settings can hide a growing archive backlog behind apparently reasonable checkpoint intervals.

Check it quickly

Use two fresh psql -X -v ON_ERROR_STOP=1 sessions, A and B, on one quiet, disposable PostgreSQL 18 instance. Use an administrative role for the explicit checkpoint and control-file information. Keep the normal fsync and synchronous-commit settings enabled. In B, create a counter and save the current checkpoint location:

SHOW fsync;
SHOW synchronous_commit;
CREATE SCHEMA checkpoint_demo;
CREATE TABLE checkpoint_demo.items (id integer PRIMARY KEY,n integer);
INSERT INTO checkpoint_demo.items VALUES (1,10);
CREATE TABLE checkpoint_demo.baseline AS
SELECT checkpoint_lsn FROM pg_control_checkpoint();

Both displayed settings should be on. The initial committed counter is ten. In A, change it inside a transaction that remains open:

BEGIN;
UPDATE checkpoint_demo.items SET n=99 WHERE id=1 RETURNING n;

A receives ninety-nine from its own update. Do not commit it. In B, request a checkpoint and inspect what an ordinary reader can see:

CHECKPOINT;
SELECT c.checkpoint_lsn>b.checkpoint_lsn AS checkpoint_advanced,
 (SELECT n FROM checkpoint_demo.items WHERE id=1) AS visible_value
FROM checkpoint_demo.baseline AS b CROSS JOIN pg_control_checkpoint() AS c;

The checkpoint location advances, but B still reads ten. The checkpoint has completed while A’s transaction remains undecided. This separates maintenance progress from business commit status. In A, verify its own version is still visible, then discard it:

SELECT n FROM checkpoint_demo.items WHERE id=1;
ROLLBACK;

A reads ninety-nine before rollback. B can now confirm the original ten and perform a normal committed update, with checkpoint locations captured immediately around that operation:

SELECT n FROM checkpoint_demo.items WHERE id=1;
UPDATE checkpoint_demo.baseline
SET checkpoint_lsn=(SELECT checkpoint_lsn FROM pg_control_checkpoint());
UPDATE checkpoint_demo.items SET n=11 WHERE id=1;
SELECT c.checkpoint_lsn=b.checkpoint_lsn AS no_new_checkpoint,
 (SELECT n FROM checkpoint_demo.items WHERE id=1) AS committed_value
FROM checkpoint_demo.baseline AS b CROSS JOIN pg_control_checkpoint() AS c;
DO $$ BEGIN
 IF (SELECT n FROM checkpoint_demo.items WHERE id=1) IS DISTINCT FROM 11 THEN
 RAISE EXCEPTION 'checkpoint/transaction boundary result mismatch';
 END IF;
END $$;
DROP SCHEMA checkpoint_demo CASCADE;

The results are ten, followed by a true unchanged-checkpoint comparison and committed value eleven in the tested short schedule. The commit succeeded without requesting another checkpoint. An independently scheduled checkpoint can change that comparison in a busy instance; it does not change the meaning of commit. The assertion checks the final row value, and the fixture is removed afterward.

This drill verifies transaction visibility and checkpoint boundaries. It does not measure recovery duration or simulate a storage device losing acknowledged writes. The durability contract still depends on enabled durability settings and storage that honors synchronization requests. Use separate crash-recovery rehearsals to validate operational recovery time against the workload you actually retain.

Tune against recovery and foreground work

Longer checkpoint intervals can reduce repeated flushing and, with full-page writes enabled, avoid some repeated full-page-image logging. They can also leave more work to replay after a crash. Frequent checkpoints can increase write traffic and latency. Fast local storage alone is not evidence that a larger interval meets a particular recovery objective.

checkpoint_completion_target spreads ordinary checkpoint work across a target fraction of the interval; its usual 0.9 value is a pacing target, not a guarantee against every latency spike. Measure checkpoint duration, requested versus timed checkpoints, WAL growth, and foreground latency together. In PostgreSQL 18, checkpoint counters and write/sync times are in pg_stat_checkpointer; the background-writer view describes a different component. Investigate repeated WAL-driven requests and retention causes before changing limits, and compare recovery tests before crediting a tuning change as successful.

Sources

PostgreSQL 18 WAL introduction, write-ahead ordering and commit durability without immediate data-file flushing, checked 2026-09-11.
PostgreSQL 18 CHECKPOINT, explicit checkpoint behavior and required privileges, checked 2026-09-10.
PostgreSQL 18 WAL configuration, checkpoint scheduling, pacing, and WAL size semantics, checked 2026-09-10.
PostgreSQL 18 cumulative statistics, the dedicated checkpointer statistics view, checked 2026-09-10.

Partitioning splits one table into many pieces that look like one

Thu, 23 Dec 2027 00:00:00 +0000

Declarative partitioning lets PostgreSQL expose one logical table while storing its rows in separate partitions. The parent routes writes and provides a common query target; the physical partitions hold the data. Its strongest benefits often concern lifecycle management and selective access, but the choice also affects keys, query shape, and operational locking.

A logical parent with physical children

A partitioned parent has no row storage of its own. Each attached partition holds rows permitted by its bounds and can have its own storage parameters, indexes, and maintenance activity. A partitioned index similarly represents a collection of indexes on the physical partitions. Splitting a table does not copy every row into both a parent store and a child store.

Range, list, and hash partitioning divide data according to different rules. Choose the key from real access and retention requirements: monthly event data may fit range partitioning, while other workloads need a different distribution. Declarative partitioning does not by itself distribute those partitions across independent database servers. Local partitions still share the resources and transactional environment of their database.

Keeping the parent name stable can simplify application access, but it does not prove that application code needs no changes. Queries must expose useful partition-key restrictions to benefit from pruning. Keys and foreign-key references may need redesign, and operations that previously addressed a globally unique identifier must retain their intended identity semantics.

The key affects more than routing

A unique or primary-key constraint on a partitioned table must include all partition-key columns, and the partition key must not contain expressions for these constraints. PostgreSQL’s individual partition indexes cannot independently prove arbitrary uniqueness across the other partitions. A composite key on day and id therefore does not promise that id alone is unique across all days.

That distinction matters during a migration from a single table with a global id key. Merely adding the date to the primary key can weaken the original identity contract. Review callers, references, and duplicate-prevention requirements before changing it. The partition key is not just a physical tuning option that can be changed without examining the public data model.

Updating a partition key can move a row to another partition. The destination must exist and accept the row, and indexes and constraints still apply. Such updates are real data changes, not free metadata edits. If retention has already removed the destination range, a late correction may fail even though the logical parent table still exists.

When it breaks

DETACH removes a table from the partition hierarchy while retaining that table and its rows. Queries through the parent stop seeing it, but the detached relation remains available for separate inspection or export. DROP removes the partition’s data as well; detaching is not deletion, and neither operation automatically creates a verified archive or backup.

Partition maintenance takes locks. A plain detach needs a strong lock on the parent, so it can wait behind long-running work and affect concurrent access. PostgreSQL provides a concurrent detach form with different locking and additional restrictions, including that it cannot run inside an explicit transaction block. Choose the operation using its actual contract rather than assuming that “metadata operation” means “no blocking.”

Too many partitions can also increase planning and memory costs. There is no universal row-count threshold that makes partitioning worthwhile. Measure representative reads and maintenance operations, create future partitions before writes need them, and define how late-arriving data interacts with retention. A small well-indexed table may remain simpler than a large collection of tiny partitions.

Check it quickly

Save this as partition-lifecycle.sql and run psql -X -v ON_ERROR_STOP=1 -f partition-lifecycle.sql in a disposable PostgreSQL 18 database. The experiment uses temporary partitions and ordinary transactional detach; final rollback removes the entire fixture.

\set ON_ERROR_STOP on
BEGIN;
CREATE TEMP TABLE events (day date,id integer,PRIMARY KEY(day,id)) PARTITION BY RANGE(day);
CREATE TEMP TABLE events_sep PARTITION OF events FOR VALUES FROM ('2026-09-01') TO ('2026-10-01');
CREATE TEMP TABLE events_oct PARTITION OF events FOR VALUES FROM ('2026-10-01') TO ('2026-11-01');
INSERT INTO events VALUES ('2026-09-15',1),('2026-09-30',2),('2026-10-01',3);
SELECT id,day,tableoid::regclass AS physical_table FROM events ORDER BY id;
DO $$ BEGIN
 CREATE UNIQUE INDEX events_id_only ON events(id);
 RAISE EXCEPTION 'global id uniqueness accepted without partition key';
EXCEPTION WHEN feature_not_supported THEN RAISE NOTICE 'unique key must include partition key';
END $$;
SAVEPOINT id_scope;
INSERT INTO events VALUES ('2026-10-03',1);
SELECT id,day FROM events WHERE id=1 ORDER BY day;
ROLLBACK TO id_scope;
RELEASE id_scope;
UPDATE events SET day='2026-10-02' WHERE id=2;
SELECT id,day,tableoid::regclass AS physical_table FROM events ORDER BY id;
ALTER TABLE events DETACH PARTITION events_sep;
SELECT (SELECT count(*) FROM events) AS attached_rows,(SELECT count(*) FROM events_sep) AS detached_rows;
DO $$ BEGIN
 IF (SELECT array_agg(id ORDER BY id) FROM events) IS DISTINCT FROM ARRAY[2,3] OR
 (SELECT array_agg(id ORDER BY id) FROM events_sep) IS DISTINCT FROM ARRAY[1] THEN
 RAISE EXCEPTION 'detach changed stored rows';
 END IF;
END $$;
ROLLBACK;

PostgreSQL 18.6 routed ids one and two to September and id three to October. It rejected a parent unique index on id alone. Within a savepoint, a second id one on another day was accepted under the composite primary key; rolling back that savepoint restores the original fixture.

Updating id two’s day moved it to October, as tableoid showed. Detaching September then left two rows visible through the parent and one row in the detached table. Assertions verify the exact identifiers on both sides, proving that detach changed membership without discarding stored data. This validates the lifecycle and key semantics, not production lock duration or a complete archival procedure.

Sources

PostgreSQL table partitioning, physical storage, keys, routing, and partition management, checked 2026-09-10.
PostgreSQL ALTER TABLE, attach and detach behavior and locking restrictions, checked 2026-09-10.

Redis append-only log trades speed for durability

Wed, 17 Nov 2027 00:00:00 +0000

An append-only file records Redis mutations for recovery, but choosing AOF is not the same as promising that no acknowledged write can ever disappear. Durability depends on the fsync policy, effective configuration, storage behavior, and which failure you mean. Start with a recovery objective rather than the reassuring word “log.”

Execution, logging, and acknowledgment

Redis executes commands against its in-memory dataset and then propagates their effects to AOF. This is not a rule that each command must reach disk before memory changes. With appendfsync always, Redis synchronizes appended data before sending replies; commands processed together may share a write and fsync. A lost connection still leaves the client uncertain about an operation whose reply never arrived.

The file is a recovery representation, not a permanent audit trail of every original request. Redis can transform commands for persistence, and rewriting retains the current dataset rather than its entire history. A deleted value is not guaranteed to remain recoverable from a rewritten log.

Choose the synchronization contract

appendfsync everysec schedules synchronization approximately once per second. Its usual tradeoff is roughly one second of exposed writes, not a universal loss bound under arbitrary storage stalls. appendfsync no leaves flushing to the operating system; it means Redis does not request fsync, not that data never reaches disk. Neither policy makes every successful reply evidence of durable storage.

appendfsync always offers a stronger local acknowledgment contract at a latency and throughput cost, but it cannot protect against losing the disk, a faulty storage stack, or deleting the wrong data. Also inspect no-appendfsync-on-rewrite: enabling it suspends normal fsync during background persistence work, weakening durability even when the main policy says always.

Rewriting creates a recoverable file set

Since Redis 7, AOF uses a manifest plus a base file and incremental files. With aof-use-rdb-preamble enabled, the base uses RDB format; later mutations live in incremental AOF files. That base belongs to AOF and is not the independently scheduled dump.rdb snapshot.

BGREWRITEAOF starts or schedules background work; its successful reply does not certify completion. Inspect INFO persistence for the completed rewrite and its status. Copying one conveniently named file, or racing a rewrite while copying the directory, can produce an unusable backup. Preserve a consistent manifest and its referenced active files.

When it breaks

Disk errors, exhausted space, and rewrite failures require monitoring, not merely an appendonly yes setting. Check aof_last_write_status and aof_last_bgrewrite_status alongside disk capacity, rewrite state, and application errors. Background work still consumes I/O and copy-on-write memory; “background” does not mean zero latency impact.

Truncated tails and corruption in the middle have different recovery semantics. Preserve the original artifact before inspection or repair: a repair can discard subsequent commands. AOF also replays application mistakes, so maintain independently retained backups. Do not turn a process-restart demonstration into a claim about power-loss safety, replication acknowledgment, or exactly-once retries.

Check it quickly

Use a POSIX machine with a C compiler, make, curl, tar, and Python 3.10 or newer. This exercise ran on macOS with Redis 8.10.1, redis-py 8.1.0, and Python 3.14.4. The setup builds Redis core with libc and installs the client in a temporary virtual environment.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It creates a private Unix-socket server, without TCP, scheduled snapshots, or automatic AOF rewrites. AOF uses always and keeps fsync enabled during rewriting. Only the subprocess it creates is killed. Persistence files survive that process restart inside the temporary directory, which is removed after the checks.

from contextlib import contextmanager
import os
from pathlib import Path
import signal
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance(directory, aof):
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--dir", directory, "--dbfilename", "dump.rdb", "--save", "",
 "--appendonly", "yes" if aof else "no", "--appendfsync", "always",
 "--appenddirname", "appendonlydir", "--appendfilename", "appendonly.aof",
 "--aof-use-rdb-preamble", "yes", "--no-appendfsync-on-rewrite", "no",
 "--auto-aof-rewrite-percentage", "0",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client, process
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode in (0, -signal.SIGKILL), errors.decode()


def crash(process):
 assert process.poll() is None
 process.kill()
 assert process.wait(timeout=5) == -signal.SIGKILL


with TemporaryDirectory(prefix="redis-aof-", dir="/tmp") as directory:
 with instance(directory, aof=True) as (client, process):
 client.set("counter", 0)
 for expected in range(1, 5):
 assert client.incr("counter") == expected
 assert client.hset("profile", mapping={"name": "Ada", "level": "2"}) == 2
 client.set("deleted", "not retained")
 assert client.delete("deleted") == 1
 rewrites = client.info("persistence")["aof_rewrites"]
 assert client.bgrewriteaof()
 deadline = time.monotonic() + 5
 while True:
 state = client.info("persistence")
 if (state["aof_rewrites"] > rewrites
 and not state["aof_rewrite_in_progress"]
 and not state["aof_rewrite_scheduled"]):
 assert state["aof_last_bgrewrite_status"] == "ok"
 break
 assert time.monotonic() < deadline, "AOF rewrite did not finish"
 time.sleep(0.02)
 assert client.incr("counter") == 5
 assert client.info("persistence")["aof_last_write_status"] == "ok"
 assert not (Path(directory) / "dump.rdb").exists()
 print("rewrite completed; a later INCR returned 5; no standalone RDB")
 crash(process)

 aof_dir = Path(directory) / "appendonlydir"
 entries = []
 for line in (aof_dir / "appendonly.aof.manifest").read_text().splitlines():
 parts = line.split()
 entries.append(dict(zip(parts[::2], parts[1::2], strict=True)))
 active = [entry for entry in entries if entry["type"] in {"b", "i"}]
 assert {entry["type"] for entry in active} == {"b", "i"}
 for entry in active:
 assert (aof_dir / entry["file"]).is_file()
 base = next(aof_dir / entry["file"] for entry in active if entry["type"] == "b")
 assert base.read_bytes().startswith(b"REDIS")
 print("manifest references an RDB-format AOF base and an incremental AOF")

 with instance(directory, aof=True) as (client, process):
 assert client.get("counter") == b"5"
 assert client.hgetall("profile") == {b"name": b"Ada", b"level": b"2"}
 assert client.get("deleted") is None
 assert client.dbsize() == 2
 print("after SIGKILL and restart: counter=5, hash restored, deletion retained")

The test waits for an observed successful rewrite, then adds one more increment. After SIGKILL, the manifest references an RDB-format AOF base and incremental data, while no standalone dump.rdb exists. Restarting restores counter five, both hash fields, and the absence of the deleted key. Checking values and types is stronger than counting keys alone.

The manifest inspection is scoped to this fixture’s simple filenames; it is not a general backup parser. SIGKILL skips graceful shutdown but leaves the operating system running, including its filesystem cache. The result proves this local AOF recovery path, not a measured loss window for everysec or no, nor survival of hardware failure.

Sources

Redis persistence, synchronization, multipart AOF, recovery, and backup boundaries, checked 2026-09-08.
Redis BGREWRITEAOF, scheduled work versus completed rewriting, checked 2026-09-08.
Redis INFO, persistence status fields, checked 2026-09-08.
Redis 8.10.1 configuration, fsync policies, rewrite overrides, and AOF file layout, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only build and allocator selection, checked 2026-09-08.

Btrfs snapshots are copy-on-write, not copies

Sun, 14 Nov 2027 00:00:00 +0000

A btrfs snapshot is not a copy of anything. btrfs subvolume snapshot creates a new subvolume whose root points at the same extent tree as the source, so every extent the two share is referenced twice and stored once. Creation is a metadata operation measured in milliseconds whether the subvolume holds a gigabyte or ten terabytes.

That makes snapshots the cheapest rollback primitive on the box, and it is also why they are the most commonly over-trusted. Space grows with divergence, not with the size of the data; the cost arrives later, on somebody else’s write, on the same device that holds the original. A snapshot you never prune is a slow-motion disk-full incident, and a snapshot on the same disk is not a backup no matter how many of them you keep.

Shared extents, not copied blocks

Snapshots are subvolumes, and a subvolume is a subtree with its own root, so the snapshot machinery is the reference-counting machinery that makes reflinks work. Take a read-only snapshot with btrfs subvolume snapshot -r /data /snap/data-2026-09-16; the -r matters, because a writable snapshot starts diverging the moment anything writes to it, and a backup target should not be writable in the first place.

The first write after a snapshot is what costs. When either side overwrites a shared extent, the filesystem allocates a new extent for the writer and leaves the old one to the other side, so the price of a snapshot is paid by the next write, in units of the extents that write touches — not by the snapshot command. APFS clonefiles share extents under the same rule, and the fixture measures the shape: a 64 MiB clone costs 0 KiB, a plain 64 MiB copy costs 65536 KiB, and overwriting the first 8 MiB of the clone costs 8192 KiB. Rewriting that same 8 MiB a second time costs 0 KiB, because those extents already diverged.

Where the space actually goes

btrfs filesystem du is the honest accounting: it prints Total, Exclusive and Set shared for each argument, where Set shared is the space it shares with the others named on the command line. Right after a snapshot, Exclusive should be near zero and Set shared close to the origin’s Total. compsize gives the same split per file, and btrfs qgroup show -reF tracks exclusive and referenced bytes per qgroup when quotas are enabled.

Deleting the origin does not free the shared extents — they become the snapshot’s exclusive data, and the space changes owner. This is the most common support question about btrfs: a file is deleted and df does not move. Only when the last reference goes away does the space return, which is why retention policy and snapshot cadence must be designed together.

A restore point is not a replica

Snapshots live in the same filesystem on the same device as their source, so a controller failure, a dead disk, or a btrfs check gone wrong takes the data and every snapshot of it in one motion. They defend against exactly one class of accident — the write that should not have happened — and they are excellent at it: rollback is local, instant, and needs no network.

Real protection means an independent copy: btrfs send walks a snapshot and emits a stream that btrfs receive applies to another filesystem, with -p sending only the delta from a parent snapshot. Send that stream to a different machine or a different pool. The distinction: send is the replication mechanism, and the snapshot is just the object it replicates.

When it breaks

The classic failure is the delete that frees nothing: someone removes a 40 GB directory, df is unmoved, and they delete more to compensate. Run btrfs filesystem du -s across the subvolume and its snapshots and look for one whose Exclusive value accounts for the gap. If the snapshots are read-only, the fix is deleting snapshots, not more files.

The second failure is a full filesystem, which on btrfs is worse than on ext4. Metadata needs space of its own, and a copy-on-write filesystem with no free space cannot allocate the extent a write requires, so one at 100 percent can fail writes that would succeed elsewhere and can go read-only. Keep headroom — 5 percent is not paranoia — bound snapshot retention, and remember that a snapshot of a nearly full filesystem also commits you to space for its divergence.

Check it quickly

Confirm the snapshot shares rather than copies, and find what is holding the space:

btrfs subvolume snapshot -r /data /snap/data-$(date +%F)
btrfs filesystem du -s /data /snap/data-2026-09-16
compsize /snap/data-2026-09-16
btrfs filesystem usage /data

du prints a Total, Exclusive and Set shared triple per path: a fresh snapshot’s Exclusive should be a few MiB of metadata at most while Set shared tracks the origin’s Total. If Exclusive already equals Total, the snapshot is a full copy. btrfs filesystem usage then shows how much divergence the device can still absorb.

Sources

btrfs subvolume documentation, subvolume and snapshot semantics including read-only snapshots, checked 2026-09-16.
btrfs filesystem documentation, defines the du Exclusive and Set shared columns used above, checked 2026-09-16.
btrfs qgroup documentation, exclusive and referenced byte accounting per qgroup, checked 2026-09-16.
btrfs send documentation, the replication stream that moves a snapshot off the machine, checked 2026-09-16.

An LRU cache evicts the least recently used entry

Fri, 12 Nov 2027 00:00:00 +0000

LRU chooses a victim by recency, not by popularity, age, or business importance. When a bounded cache needs space, an exact least-recently-used policy removes the entry whose last qualifying access occurred longest ago. That rule can exploit temporal locality, but it neither guarantees useful hits nor keeps cached data fresh.

Follow the access order

Imagine a three-entry cache, listed from oldest to newest. Loading A, B, and C produces A–B–C. Reading A moves it to the newest position, producing B–C–A. Loading D then evicts B, leaving C–A–D. A plain insertion-ordered dictionary does not automatically move an entry when it is read.

An implementation must define which operations count as touches. A common exact design combines a lookup table with a doubly linked recency order, supporting efficient lookup, promotion, and removal. Maintaining that order also consumes metadata and turns cache hits into state updates. Concurrent implementations must coordinate those changes.

Exact and sampled policies differ

Redis uses approximate LRU: it samples candidates and maintains a pool of good eviction candidates instead of keeping an exact order of every key. The configurable sample count trades additional CPU work for a closer approximation. It does not promise the exact victim sequence of the small model below or universally constant eviction latency.

Candidate eligibility matters too. Redis allkeys-lru can evict any key, while volatile-lru considers keys with expiration metadata. A TTL is not protection against eviction before its deadline. If no eligible expiring keys exist, a volatile policy behaves like noeviction for memory-pressure decisions.

Capacity is not freshness

A limit of three entries is not a limit of three kilobytes. Values, keys, and bookkeeping have different sizes; a count-bounded cache can still retain large objects. Python’s lru_cache retains references to arguments and results until eviction or clearing. Choose a capacity unit that matches the resource you need to control.

Recency also says nothing about whether a source value changed. Frequently reading an obsolete cached answer can keep it resident indefinitely. Expiration, explicit invalidation, and versioned keys address freshness separately. An eviction rule is neither a consistency protocol nor a backup policy.

When it breaks

A scan through one-time entries can displace a useful working set. Cycling through four distinct keys with capacity three can miss on every access, even though those keys are repeatedly requested. Historical popularity does not reserve space under LRU. LFU, admission filtering, or a different capacity may help particular workloads, but no policy follows automatically from “read-heavy.”

Measure hits, misses, evictions, retained bytes, and miss cost over representative intervals. A high hit ratio can hide expensive misses; an eviction counter alone does not reveal whether victims were useful. Reconstruct the access trace before changing policies.

A cache hit must also be safe to reuse. Avoid memoizing side effects or results that require a fresh mutable object on each call. Python’s lru_cache keeps its internal state coherent across threads, but simultaneous misses may still execute the wrapped function more than once. Thread safety is not duplicate-work suppression.

Check it quickly

Save the block as lru_check.py and run python3 lru_check.py with Python 3.10 or newer. It ran with Python 3.14.4. Only the standard library is used: an OrderedDict models exact recency, and the real functools decorator separately verifies memoization. No Redis server, network requests, or filesystem writes are involved.

from collections import OrderedDict
from functools import lru_cache


def access(cache, key, capacity):
 assert capacity > 0
 hit = key in cache
 evicted = None
 if hit:
 cache.move_to_end(key)
 else:
 cache[key] = key.lower()
 if len(cache) > capacity:
 evicted, _ = cache.popitem(last=False)
 return hit, evicted, tuple(cache)


cache = OrderedDict()
observed = [access(cache, key, 3) for key in "ABCAD"]
assert observed == [
 (False, None, ("A",)),
 (False, None, ("A", "B")),
 (False, None, ("A", "B", "C")),
 (True, None, ("B", "C", "A")),
 (False, "B", ("C", "A", "D")),
]
print("ABCAD: touching A makes B the victim; final oldest -> newest:", tuple(cache))

for key in "XYZ":
 access(cache, key, 3)
assert tuple(cache) == ("X", "Y", "Z")
assert access(cache, "A", 3)[0] is False
print("a one-time XYZ scan displaces the old working set")

cache.clear()
hits = sum(access(cache, key, 3)[0] for key in "ABCD" * 3)
assert hits == 0
print("capacity 3, repeated ABCD: 0 hits in 12 accesses")

origin = {"A": 1, "B": 2, "C": 3}
loads = []


@lru_cache(maxsize=2)
def read(key):
 loads.append(key)
 return origin[key]


assert [read(key) for key in "ABACB"] == [1, 2, 1, 3, 2]
assert loads == ["A", "B", "C", "B"]
assert tuple(read.cache_info()) == (1, 4, 2, 2)
print("functools trace:", read.cache_info(), "origin reads:", loads)
origin["B"] = 20
assert read("B") == 2
read.cache_clear()
assert read("B") == 20
assert tuple(read.cache_info()) == (0, 1, 2, 1)
print("a recent cached B stays stale until explicit invalidation")

The first assertion checks every intermediate order, not just the final size: touching A makes B the victim. A subsequent XYZ scan displaces the former working set. The repeated ABCD trace then produces zero hits in twelve accesses with capacity three.

The decorated function reads an in-memory origin mapping. Its ABACB trace records one hit, four misses, and two retained entries; the recorded origin reads identify which calls actually recomputed. Changing the origin’s B from two to twenty leaves the cached result at two until cache_clear runs. Clearing also resets the statistics.

These are deterministic correctness checks, not latency or memory benchmarks. They do not exercise Redis sampling, TTL processing, concurrent misses, byte-based capacity, or a production access distribution. Use the measured trace to understand the policy, then validate the actual cache implementation under its own workload.

Sources

Python functools, LRU memoization, cache statistics, references, invalidation, and concurrent-call limits, checked 2026-09-08.
Python collections, OrderedDict reordering and endpoint removal, checked 2026-09-08.
Redis key eviction, approximate LRU, sampling, and candidate eligibility, checked 2026-09-08.

LSM trees swallow random writes

Tue, 02 Nov 2027 00:00:00 +0000

A B-tree updates a page where the row lives, which means a random-key write pattern turns into a random-page write pattern. An LSM tree refuses that trade: it appends, and rewrites in bulk later. Everything else about the design — the memtable, the bloom filters, the compaction policy — follows from that one refusal.

The measurement that motivates it is easy to reproduce. Applying the same 200,000 in-place updates to the same 200,000-row sqlite table, with a 2 MB page cache and a durable commit, takes 0.17 s when the keys arrive in sorted order and 2.60 s when they arrive shuffled: 15.25x, on identical rows and identical page counts. The extra time is scattered page rewrites and the cache evictions they cause.

Buffer, flush, merge

Writes go to a write-ahead log for durability and into an in-memory table for speed. When the memtable fills it is flushed as one immutable sorted run; in RocksDB that run lands in level 0, and every level below it holds exactly one sorted run whose size target grows exponentially. A workload that is random at the key level therefore becomes entirely sequential at the disk level, and the write path is a large ordered append no matter what the application is doing.

The bill for that is paid on the read path. A point lookup has to consider each level, so the engine keeps a per-run bloom filter to answer “not here” cheaply. In a 400,000-write simulation over 60,000 distinct keys, 20,000 point lookups probed a mean of 8.9 runs but actually read only 0.6 of them: bloom filters rejected 166,622 of 177,937 probes, 93.6%. Absent-key lookups still probed 11.0 runs each before returning empty — a bloom filter makes a miss cheap, and does nothing at all for a hit.

Compaction is the tax

The write path is cheap because nothing is ever updated in place; the price is that every logical value gets written more than once. In the same simulation, 56,000,000 logical bytes became 149,031,040 physical bytes across 54 flushes and 4 compactions — a write amplification of 2.66x, and that is with a deliberately generous compaction budget.

RocksDB names the dial explicitly. Level-style compaction merges one file in Ln with all its overlapping files in Ln+1, which minimises the files involved and therefore optimises space amplification. Universal-style compaction merges many files and levels at once, which lowers write amplification but raises space and read amplification. There is no setting that lowers all three; you are choosing which one your workload can afford.

When it breaks

The first failure is compaction falling behind the write rate. Compaction throughput is a hard floor: with a 64 MiB/s ingest and a 2.66x write amplification, the background has to move 170 MiB/s just to stand still. Run it at 145 MiB/s and the backlog is 25.5 MiB/s, which accumulates 91,973 memtable-sized runs per hour and fills a 1 TiB disk in 11.4 hours. Measured against 2,000 level-0 runs, a point lookup probes 775 runs on average instead of 8.9. Reads degrade, then disk fills, and both symptoms trace to the same number.

The second failure is the tombstone-heavy workload. A delete is not a removal, it is a write of a tombstone that must outlive the value it hides. Inserting 40,000 keys and then deleting all of them wrote 12,160,000 bytes for 5,720,000 bytes of live data, and left 40,000 tombstones holding 5,600,000 bytes of dead payload — space amplification of 0.98x the live data size, on data that no longer exists. Lookups for deleted keys still probed runs to discover the absence. Only a compaction that reaches the tombstone reclaims the space, so a workload that deletes as fast as it writes keeps paying for both.

Check it quickly

Measure the B-tree cost you are avoiding, then read the compaction counters:

sqlite3 kv.db "PRAGMA cache_size=-2000; PRAGMA synchronous=FULL; PRAGMA journal_mode=WAL;"
sqlite3 kv.db "SELECT count(*) FROM kv; PRAGMA page_count;"
ldb --db=./db stats | sed -n '/Compaction Stats/,/^$/p'
ldb --db=./db stats | grep -i stall
python3 -c "import rocksdb; d=rocksdb.DB('./db'); print(d.get_property(b'rocksdb.num-files-at-level0'))"

Insert the same keys twice, once sorted and once shuffled, and the first two lines give you the baseline gap: same row count, same page count, very different wall time. The Compaction Stats block reports W-Amp and R-Amp per level — write amplification that climbs as the dataset grows means the fanout is too small, and read amplification that climbs means compaction is behind. Any stall count above zero is the debt signal, and level-0 files sitting near the compaction trigger means the background is only just keeping up.

Sources

RocksDB wiki: RocksDB Overview, memtables, level structure, and the write/read/space amplification trade, checked 2026-09-16.
RocksDB wiki: Leveled Compaction, level-0 flushes, exponentially growing level targets, and per-level lookup cost, checked 2026-09-16.

A sparse file separates logical length from allocated blocks

Mon, 11 Oct 2027 00:00:00 +0000

A sparse regular file can have a large logical length while allocating storage for only part of its address range. Reads from holes within that length return zero bytes. File length, byte content, and filesystem allocation are therefore different measurements. Understanding their relationship matters when budgeting virtual disks, copying backups, or deciding whether a zero-filled region actually reserves space for future writes.

When it breaks

Seeking beyond the end of a file changes the descriptor’s offset, not the file’s length. A later write or an explicit extension can create the intervening zero-reading range. Calling seek alone and reporting that a large sparse file now exists is incorrect. Likewise, a large logical size does not mean the filesystem has reserved enough blocks to fill that file later.

A copy that reads every byte and writes it normally can turn holes into allocated zero-filled ranges. The resulting checksum can match perfectly while its allocation is much larger. Transfer and restore workflows that budget only the source’s current allocated space can therefore run out of capacity despite preserving the logical content correctly.

Hole punching is an intentional data operation. Punching an allocated range makes the corresponding bytes read as zeros while preserving length when requested with KEEP_SIZE. It is not a harmless command to tidy away arbitrary file regions. Preallocation solves a different problem: it reserves filesystem allocation, even though an unwritten allocated range can also read as zeros.

Check it quickly

Save the script as sparse.py and run python3 sparse.py using Python 3.11 or later on a disposable Linux system with an ext4 temporary directory. The recorded run used Linux 6.12.76 and Python 3.11.2. It creates three temporary files and uses about 32 MiB of fully allocated data plus the small sparse allocation:

from pathlib import Path
from tempfile import TemporaryDirectory
import ctypes,hashlib,os

size=16*1024*1024;first=4*1024*1024;second=12*1024*1024
libc=ctypes.CDLL(None,use_errno=True);libc.fallocate.argtypes=[ctypes.c_int,ctypes.c_int,ctypes.c_longlong,ctypes.c_longlong];libc.fallocate.restype=ctypes.c_int
def digest(path):
 with path.open('rb') as handle:return hashlib.file_digest(handle,'sha256').hexdigest()
with TemporaryDirectory(prefix='xbcatm-sparse-') as directory:
 root=Path(directory);path=root/'sparse';fd=os.open(path,os.O_RDWR|os.O_CREAT|os.O_EXCL,0o600)
 try:
 os.lseek(fd,size,os.SEEK_SET);assert os.fstat(fd).st_size==0
 os.ftruncate(fd,size);assert os.fstat(fd).st_size==size and os.fstat(fd).st_blocks==0
 assert os.pread(fd,16,0)==b'\0'*16
 os.pwrite(fd,b'A',first);os.pwrite(fd,b'B',second);os.fsync(fd)
 allocated=os.fstat(fd).st_blocks*512;assert allocated<size
 data_start=os.lseek(fd,0,os.SEEK_DATA);hole_start=os.lseek(fd,data_start,os.SEEK_HOLE)
 assert data_start==first and first<hole_start<=second
 print('seek alone leaves size zero; truncate creates zero-reading holes; sparse size',size,'allocated',allocated)
 print('first SEEK_DATA',data_start,'following SEEK_HOLE',hole_start)
 dense=root/'dense'
 with path.open('rb') as source,dense.open('wb') as target:
 while chunk:=source.read(65536):target.write(chunk)
 target.flush();os.fsync(target.fileno())
 assert digest(path)==digest(dense)
 assert dense.stat().st_blocks*512>=size
 print('explicitly writing all copied bytes preserves content but allocates the zero-filled ranges')
 result=libc.fallocate(fd,0x01|0x02,first,4096) # KEEP_SIZE | PUNCH_HOLE
 assert result==0,os.strerror(ctypes.get_errno())
 os.fsync(fd)
 assert os.fstat(fd).st_size==size and os.pread(fd,1,first)==b'\0'
 assert os.pread(fd,1,second)==b'B' and os.fstat(fd).st_blocks*512<allocated
 print('punching the aligned data range releases its blocks and makes it read zero without changing file length')
 reserved=root/'reserved'
 with reserved.open('w+b') as handle:
 os.posix_fallocate(handle.fileno(),0,size);os.fsync(handle.fileno())
 info=os.fstat(handle.fileno());assert info.st_size==size and info.st_blocks*512>=size
 assert handle.read(16)==b'\0'*16
 print('preallocation reserves blocks even while the newly allocated range reads as zeros')
 finally:os.close(fd)

The fixture first seeks to sixteen MiB in an empty file and verifies that its size remains zero. Truncating to that length then creates a file with sixteen MiB of logical content and no allocated data blocks in this ext4 case. Reading its beginning returns zeros, establishing the difference between moving an offset and extending the file.

Writing one byte at four MiB and another at twelve MiB allocates two small regions. After synchronization, the recorded file length is 16777216 bytes and its reported allocation is 8192 bytes. Linux’s st_blocks count is converted using 512-byte units; treating it as a count of the filesystem’s larger block size would miscalculate the result.

The first SEEK_DATA result is 4194304, and the following SEEK_HOLE result is 4198400 in the recorded run. These describe the ext4 region around the first written byte, not only that byte’s nonzero position. A data extent may contain zeros too, so extent reporting is not a search for nonzero content.

The script explicitly copies all logical bytes through ordinary writes into a second file. Its digest matches the sparse source, but its allocated space covers the full logical length. This controlled dense copy demonstrates the expansion risk without claiming that every modern copy tool or filesystem necessarily materializes holes in the same way.

Next, an aligned punch removes the first allocated data region. The original A byte now reads as zero, the distant B byte remains, length stays unchanged, and allocated blocks decrease. Finally, preallocating the third file reserves space across sixteen MiB while its initial bytes read as zeros. Zero content alone cannot distinguish a hole from reserved allocation.

Budget and preserve the intended representation

Check both logical size and allocated blocks after the actual archive, transfer, and restore path. A sparse-aware option may preserve holes, while compression, reflinks, and filesystem-specific copying can produce other storage relationships. Per-file allocation also should not automatically be interpreted as exclusive physical-device usage when the filesystem shares or compresses extents.

Use the target filesystem’s documented preallocation contract when future allocation must be reserved. A merely truncated virtual image can grow its storage consumption later and fail when the shared pool is full. Capacity planning needs the expected populated size and competing allocations, not just an attractive small allocation measured at creation time.

Treat SEEK_DATA and SEEK_HOLE as filesystem-provided extent information. Implementations are allowed to report holes conservatively, including representing a broader range as data. Verify support and behavior before depending on exact offsets. When punching or copying ranges, preserve the application’s intended bytes first; space savings are useful only when the resulting data and allocation match the operation’s actual purpose.

Sources

lseek(2), seek position, SEEK_DATA/SEEK_HOLE, and filesystem reporting limits, checked 2026-09-12.
truncate(2), extending logical length with zero-reading regions, checked 2026-09-12.
fallocate(2), preallocation and hole punching without changing logical size, checked 2026-09-12.
Linux 6.12 fallocate UAPI, KEEP_SIZE and PUNCH_HOLE flags used by the fixture, checked 2026-09-12.

A PVC gives a Pod durable storage beyond the container

Fri, 13 Aug 2027 00:00:00 +0000

A container is ephemeral by design, so a PersistentVolumeClaim is the handle that pins real storage to it even when the Pod is recreated. The split is deliberate: a PersistentVolume declares capacity that exists, a PersistentVolumeClaim requests capacity that is needed, and a controller matches one to the other. With a StorageClass the claim can create its own volume through a provisioner, so nobody has to pre-build the storage.

The part that bites is that claim and volume have separate lifetimes governed by separate rules. A finalizer decides when the claim is allowed to disappear, and a reclaim policy decides what happens to the volume afterwards. Get the deletion order wrong and you get a claim stuck in Terminating; get the policy wrong and you get either data you meant to keep or a bill for storage nobody is using.

The claim, the volume, and the binding

Binding is a conjunction, not a best match. A volume is a candidate only if its capacity is at least the request, its access modes include the requested mode, its storageClassName matches, and no other claim owns it. A claim that satisfies nothing stays Pending with an event naming the requirement that failed, which is more useful than a generic failure; the usual causes are an access mode the class cannot deliver and a class name that does not exist.

accessModes has no default; you must choose one, and ReadWriteOnce is what most single-node block provisioners actually deliver. It is also single-node, so a StatefulSet with replicas on different nodes cannot share one claim. When a class supports dynamic provisioning, volumeBindingMode sets the timing: Immediate provisions the volume the moment the claim appears, which can place a topology-constrained volume in a zone with no node to use it, while WaitForFirstConsumer keeps the claim Pending until a pod is scheduled and then provisions in that pod’s zone.

The deletion order and the finalizer

The order is pod, then claim, then volume, and the protection is implemented with finalizers. Deleting a claim while a pod still mounts it does not delete anything: the claim gets a deletion timestamp, flips to Terminating, and stops there because kubernetes.io/pvc-protection is still in its finalizer list. The controller removes that finalizer only once no pod references the claim, so the “Terminating forever” PVC is not a stuck controller but a claim something still uses.

The same protection exists in the other direction. A PersistentVolume carries kubernetes.io/pv-protection, so deleting a volume that is bound to a claim is postponed until it is no longer bound. That ordering makes an accidental kubectl delete pv recoverable, and explains why a volume can seem to ignore a delete for as long as its claim exists.

Reclaim policy: Delete or Retain

When the claim is finally gone the volume is released, and the reclaim policy takes over. Delete removes the PV object and the external storage asset with it. Retain keeps the PV object, marks it Released, and deliberately does not make it available for a new claim, because the previous claimant’s data is still on it — reclamation becomes a manual sequence: delete the PV, clean the storage asset, delete the asset.

Dynamically provisioned volumes inherit the policy from their StorageClass, whose default is Delete. That default is right for scratch and wrong for a database, and it is invisible: nothing in the PVC or pod spec mentions the policy, so the decision lives in a cluster-scoped object nobody reviewing the workload ever sees.

When it breaks

The classic failure is the claim that will not die. Someone deletes a PVC, sees Terminating, and waits — while a pod somewhere still mounts it, usually a StatefulSet that was never scaled down or a pod stuck on an unreachable node. Nothing errors, nothing times out, and the PVC blocks the storage behind it. Find the pod rather than deleting the finalizer by hand; removing it while a pod still writes is how you detach a volume under a running process.

The second failure is the Retain orphan. The claim is gone, the pods are gone, the namespace may be gone, and a Released PV plus its storage asset sit there indefinitely. Nothing is in use and nothing is in error, so no alert fires and the volume does not appear in any workload’s list. The inverse mistake is just as quiet: a class whose default Delete policy was never overridden, discovered only when someone tries to recover data that the reclaim already removed.

Check it quickly

Look at both sides of the binding, and at what holds the claim:

kubectl get pvc -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,STATUS:.status.phase,VOLUME:.spec.volumeName,CLASS:.spec.storageClassName
kubectl get pv -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,POLICY:.spec.persistentVolumeReclaimPolicy,CLAIM:.spec.claimRef.name
kubectl get pvc data -o jsonpath='{.metadata.finalizers}{"\n"}'
kubectl get pods -A -o json | jq -r '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName=="data") | "\(.metadata.namespace)/\(.metadata.name)"'

A claim in Terminating whose finalizers still contain kubernetes.io/pvc-protection is being held by a pod, and the fourth command names it. On the volume side, a PV stuck in Released with a Retain policy is an orphan: nothing references it, nothing will reuse it, and only manual reclamation clears it.

Sources

Persistent Volumes, binding, access modes, reclaim policies, Released state and the pvc/pv protection finalizers, checked 2026-09-16.
Storage Classes, dynamic provisioning, reclaimPolicy, and volumeBindingMode, checked 2026-09-16.

NFS shares a remote directory over the network

Wed, 23 Jun 2027 00:00:00 +0000

An NFS export is a directory on the server plus a list of clients allowed to reach it; the file that describes it is not what serves it. /etc/exports is read by exportfs, which writes the master export table to /var/lib/nfs/etab, which rpc.mountd consults on a client’s MOUNT request. Three files: one that people edit, two that decide what happens.

That indirection produces the most common NFS incident: the export list looks right and the client is refused, because the file changed and the table did not. The rest of the configuration comes down to two questions — which identity the server thinks a request carries, and whether a reply that said “written” survives a crash.

The file and the table

An /etc/exports line is <directory> <client>[(<options>)] ..., so /srv/data 10.0.0.0/24(rw,sync,no_subtree_check,root_squash) exports one directory to one network with four options, and another client group can follow on the same line with its own options. Clients are hostnames, wildcards, CIDR networks or netgroups; comments run to end of line, and a trailing backslash continues a line. Nothing in the file is live until exportfs -ra rebuilds the table from it, and exportfs -v prints the table that is actually loaded.

showmount -e shows that same table from the client side: it asks the mount daemon, not the file, so it answers “what does the server think it is exporting”, not “what did I just write”. The fixture reproduces the trap: after appending /srv/newshare without re-exporting, the file has four entries and the table three, so a client asking for it gets NOENT.

What the options actually decide

ro and rw set the write direction, and read-only is the default: without rw, any request that changes the filesystem is disallowed. sync and async decide when the server may answer — with sync the reply follows the commit to stable storage, with async it may precede the commit, which the documentation says allows a crash to lose acknowledged writes. async was the default up to nfs-utils 1.0.0; sync has been the default since.

The squash options are the identity policy. root_squash maps a remote uid 0 to the anonymous uid, which is what you want almost everywhere; no_root_squash lets a client’s root be root on the server, defensible only for a host you trust; all_squash with anonuid/anongid maps every client id to one anonymous identity, the right shape for a public share. subtree_check versus no_subtree_check is a reliability knob, not a security one: subtree checking verifies that a requested file is still inside the exported subtree, which can break access to files renamed while open; since nfs-utils 1.1.0 the default is no_subtree_check.

Mounting, and the id mapping trap

mount -t nfs -o vers=4.2 server:/srv/data /mnt/data attaches the export; with NFSv4 you export once with fsid=0 and mount subtrees by path. Prefer hard — the default — because a soft mount gives up after retrans retransmissions and returns EIO, which the man page warns can corrupt data silently; the price is that a dead server turns an ls into a process stuck in D state. intr no longer helps: it has been ignored since kernel 2.6.25.

Permissions are decided by numeric uid and gid against the server’s accounts, not by names, so a client whose id has no counterpart there lands on the anonymous uid. id on both sides plus ls -ln on the mount is the check: a file you own showing as 65534 means the mapping is wrong, not the mode.

When it breaks

The classic failure is the stale export table, in both directions. A share added and never exported is invisible; a share removed and never re-exported is still served, to clients you believe you locked out. Both are fixed by exportfs -ra, and both are visible by diffing /etc/exports against exportfs -v.

The second failure is the hung client. A hard mount against an unreachable server blocks in the kernel indefinitely, so every process touching the mount is unkillable and the machine accumulates D-state processes; the fixture’s showmount -e 127.0.0.1 took 35 seconds to fail, the same RPC timeout that makes a bad mount look like a frozen ls. The tempting fix, soft, trades the hang for silent write corruption.

Check it quickly

Compare the loaded table against the file, then check the ids:

exportfs -v
showmount -e localhost
exportfs -ra && exportfs -v
mount | grep 'type nfs'
id; ls -ln /mnt/data | head

exportfs -v must list exactly the directories and options in /etc/exports; anything missing means the table is stale and clients get NOENT while the file looks correct. showmount -e should print the same directories to a remote caller. On the client, id and ls -ln must agree on the numeric owner — 65534 or nobody means the server has no matching account.

Sources

exports(5), export syntax, squash options, sync/async and the subtree_check defaults, checked 2026-09-16.
exportfs(8), the master export table in /var/lib/nfs/etab and how exportfs maintains it, checked 2026-09-16.
nfs(5), mount options including hard/soft, timeo and the obsolescence of intr, checked 2026-09-16.
showmount(8), querying the server’s export list from a client, checked 2026-09-16.

Two ways Redis keeps data safe

Tue, 15 Jun 2027 00:00:00 +0000

RDB and AOF preserve different representations of Redis state. Choosing between them starts with two questions: how much recent data may be lost, and how long recovery may take? A persistence setting alone answers neither. You need a recoverable artifact, an understood acknowledgment policy, and a restore procedure tested against actual values.

A snapshot captures a point, not an interval guarantee

An RDB file is a compact point-in-time dataset. BGSAVE forks a child that writes a temporary file and replaces the previous snapshot when complete. The parent continues serving requests, but forking, copy-on-write memory, and disk work still have costs. SAVE instead writes synchronously and blocks other clients; reserve it here for the tiny isolated exercise.

A rule such as save 60 1000 requires both elapsed time and a change threshold. It does not promise a new snapshot every sixty seconds regardless of activity, nor successful completion on schedule. Recovery exposure follows the age of a usable snapshot, including missed or failed saves. Monitor successful saves and their results, not just the configured interval.

AOF changes the recovery representation

AOF records mutations and synchronizes them according to appendfsync. everysec normally exposes roughly the most recent second, while no delegates flushing to the operating system. always synchronizes before replies under its effective configuration, at additional cost. These policies do not protect against losing the storage device or guarantee that a promoted replica contains every locally persisted write.

Modern AOF uses a manifest, a base, and incremental files. Its base can be RDB-formatted, so “AOF always replays a huge textual history” is misleading. Rewriting removes unnecessary historical operations. File size, startup time, rewrite overhead, and latency must be measured for your dataset rather than inferred from the format’s name.

Running both is not an automatic merge

With ordinary startup settings and AOF enabled, Redis loads AOF rather than independently loading dump.rdb and replaying a separate log on top. The RDB-formatted base inside AOF is part of that file set. An independently retained RDB remains useful for backup and deliberate recovery, but it does not magically fill holes in incomplete AOF artifacts.

Do not switch an existing RDB deployment by merely editing appendonly and restarting. Follow Redis’s documented live conversion procedure, wait for successful initial AOF creation, and persist the configuration. The two cases below start with separate empty directories; they are not a migration recipe for an existing database.

When it breaks

A successful BGSAVE request is not a successful completed snapshot. Failed storage writes, unavailable fork resources, or inadequate disk space can leave the usable recovery point old. Inspect INFO persistence, error logs, and resource headroom. Background persistence can also worsen tail latency even while Redis remains responsive.

Neither file format is an independent backup if its only copy shares the server’s failure domain. Retain consistent copies outside that machine and test their restore. AOF can faithfully persist an accidental deletion; snapshots can already contain the same mistake. Retention history and application validation matter as much as a boot that accepts connections.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Redis 8.10.1, redis-py 8.1.0, and Python 3.14.4. Build only Redis core with libc and install the client in the temporary virtual environment.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block using "$redis_lab/venv/bin/python" in the same shell. Each case creates its own Unix-socket server, with no TCP listener or scheduled snapshots. Both explicitly save a tiny snapshot; only the second enables AOF with always and no fsync suppression during rewriting. SIGKILL targets only the created subprocess. Its files stay available for restart, then the temporary directory is removed.

from contextlib import contextmanager
import os
from pathlib import Path
import signal
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance(directory, aof):
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--dir", directory, "--dbfilename", "dump.rdb", "--save", "",
 "--appendonly", "yes" if aof else "no", "--appendfsync", "always",
 "--appenddirname", "appendonlydir", "--appendfilename", "appendonly.aof",
 "--aof-use-rdb-preamble", "yes", "--no-appendfsync-on-rewrite", "no",
 "--auto-aof-rewrite-percentage", "0",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client, process
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode in (0, -signal.SIGKILL), errors.decode()


def crash(process):
 assert process.poll() is None
 process.kill()
 assert process.wait(timeout=5) == -signal.SIGKILL


for aof in (False, True):
 with TemporaryDirectory(prefix="redis-restore-", dir="/tmp") as directory:
 with instance(directory, aof) as (client, process):
 assert client.set("version", "snapshot")
 assert client.incr("counter") == 1
 assert client.save()
 snapshot = (Path(directory) / "dump.rdb").read_bytes()
 assert snapshot.startswith(b"REDIS")
 assert client.set("version", "after-snapshot")
 assert client.set("later", "present")
 assert client.incr("counter") == 2
 if aof:
 assert client.info("persistence")["aof_last_write_status"] == "ok"
 crash(process)

 assert (Path(directory) / "dump.rdb").read_bytes() == snapshot
 with instance(directory, aof) as (client, process):
 expected = ([b"after-snapshot", b"present", b"2"] if aof
 else [b"snapshot", None, b"1"])
 assert client.mget("version", "later", "counter") == expected
 assert client.dbsize() == (3 if aof else 2)
 assert (Path(directory) / "dump.rdb").read_bytes() == snapshot
 print("RDB + AOF" if aof else "RDB only", expected)
print("an unchanged standalone RDB does not override enabled AOF recovery")

RDB-only recovers version snapshot, no later key, and counter one. The AOF case recovers version after-snapshot, the later key, and counter two. In both cases, the standalone RDB bytes remain unchanged across the crash and checked restart. The counter is deliberately incremented without an initial SET: replaying both increments over the snapshot would incorrectly produce three.

These assertions compare recovered state, not just whether Redis starts. SIGKILL bypasses graceful shutdown but does not crash the operating system or discard its cache. The experiment establishes snapshot boundaries and ordinary AOF startup selection, not power-loss survival, backup transport integrity, replica failover behavior, or production recovery time. Define those separate acceptance tests before making a stronger promise.

Sources

Redis persistence, snapshots, AOF startup selection, live conversion, and backups, checked 2026-09-08.
Redis SAVE, synchronous snapshot behavior and blocking cost, checked 2026-09-08.
Redis BGSAVE, background saving and request acceptance, checked 2026-09-08.
Redis INFO, persistence progress and result fields, checked 2026-09-08.
Redis 8.10.1 configuration, snapshot triggers and AOF policy, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only build and allocator selection, checked 2026-09-08.

UNLOGGED tables skip the write-ahead log

Mon, 24 May 2027 00:00:00 +0000

An unlogged PostgreSQL table keeps normal SQL transaction behavior while giving up crash recovery for its contents. It is useful for reproducible staging data, but a committed row is not guaranteed to survive a server crash. The distinction is between clean operation and recovery after an unclean shutdown, not between ordinary queries and every restart.

What bypasses WAL

The table’s data changes and those of its indexes are not written to the write-ahead log. Catalog changes still need WAL, and the database may produce WAL for other work in the same transaction. Calling an unlogged workload “zero WAL” therefore overstates the feature. Its pages also occupy storage and require normal buffer management; UNLOGGED does not mean an in-memory table.

Transactions, snapshots, constraints, and rollback still apply while the server runs. A rejected statement or a disconnected application does not automatically erase previously committed rows. A crashed backend can, however, cause the server to restart other processes and perform recovery. Avoid classifying incidents solely as “application” or “infrastructure”: determine whether PostgreSQL actually entered crash recovery.

Restart and replication boundaries

A clean shutdown preserves unlogged contents for the next start. PostgreSQL’s fast shutdown rolls back active transactions and shuts down cleanly; its immediate shutdown interrupts processes without that cleanup. The latter requires recovery, during which unlogged relations are reset. Their definitions survive, but previously committed rows do not. Normal maintenance must not be described as inevitably deleting them, nor should one successful clean restart be offered as evidence of crash safety.

Physical standbys do not receive the unlogged contents. While recovery is active, querying such a relation raises an error; it does not successfully return an empty result. After promotion, the relation can be read but has no replicated rows to recover. A failover runbook must repopulate it before consumers rely on it. Sharing a schema definition across nodes is different from replicating the table’s data.

A logged table cannot have a foreign key referencing an unlogged table. PostgreSQL rejects that definition because durable references cannot safely depend on rows that may disappear during recovery. The experiment checks this restriction instead of suggesting that ordinary durable child rows survive with silently broken references.

When it breaks

An ingestion pipeline may acknowledge a file once its rows reach an unlogged staging table, then delete the only source copy. A crash makes the acknowledgment irreversible data loss. Keep the source until the durable destination commits, and make the rebuild process identify exactly which input version it needs. Deduplication and checkpoints must belong to the durable part of that contract.

A reporting service can also fail when routed to a standby or started immediately after promotion. Empty-table detection alone misses the standby error, while blindly retrying cannot manufacture missing contents. Handle rebuilding as an explicit service state with admission checks. Test rebuild duration, upstream availability, and validation of the completed dataset; a script that merely creates an empty table is not recovery.

Check it quickly

Save the Bash block as unlogged.sh. It uses PostgreSQL 18 command-line tools as a non-root user. This container invocation supplies those tools without publishing ports or mounting host data:

docker run --rm --network none --user postgres --entrypoint bash -i \
 postgres:18-alpine < unlogged.sh

#!/usr/bin/env bash
set -euo pipefail
pg_lab=$(mktemp -d /tmp/unlogged-lab.XXXXXX)
cleanup() {
 for cluster in standby primary; do
 if [ -f "$pg_lab/$cluster/postmaster.pid" ]; then
 pg_ctl -D "$pg_lab/$cluster" -m immediate -w stop >/dev/null
 fi
 done
 rm -rf -- "$pg_lab"
}
trap cleanup EXIT
primary() { psql -X -h "$pg_lab" -p 55432 -d postgres -v ON_ERROR_STOP=1 "$@"; }
standby() { psql -X -h "$pg_lab" -p 55433 -d postgres -v ON_ERROR_STOP=1 "$@"; }
start_primary() {
 pg_ctl -D "$pg_lab/primary" -l "$pg_lab/primary.log" \
 -o "-k $pg_lab -p 55432 -c listen_addresses=''" -w start >/dev/null
}
initdb -D "$pg_lab/primary" --no-locale --encoding=UTF8 --auth=trust >/dev/null
start_primary
primary <<'SQL'
CREATE TABLE durable (id integer PRIMARY KEY);
CREATE UNLOGGED TABLE scratch (id integer PRIMARY KEY);
INSERT INTO durable VALUES (1);
INSERT INTO scratch VALUES (1);
BEGIN;
INSERT INTO scratch VALUES (2);
ROLLBACK;
DO $$
BEGIN
 EXECUTE 'CREATE TABLE durable_ref (id integer REFERENCES scratch(id))';
 RAISE EXCEPTION 'a logged table unexpectedly referenced an unlogged table';
EXCEPTION WHEN invalid_table_definition THEN
 RAISE NOTICE 'logged-to-unlogged foreign key rejected';
END $$;
SQL
counts='SELECT (SELECT count(*) FROM durable), (SELECT count(*) FROM scratch)'
test "$(primary -Atc "$counts")" = '1|1'
pg_ctl -D "$pg_lab/primary" -m fast -w stop >/dev/null
start_primary
test "$(primary -Atc "$counts")" = '1|1'
printf 'clean restart: 1|1\n'
pg_basebackup -h "$pg_lab" -p 55432 -D "$pg_lab/standby" -R -X stream -c fast
pg_ctl -D "$pg_lab/standby" -l "$pg_lab/standby.log" \
 -o "-k $pg_lab -p 55433 -c listen_addresses=''" -w start >/dev/null
test "$(standby -Atc 'SELECT pg_is_in_recovery()')" = t
test "$(standby -Atc 'SELECT count(*) FROM durable')" = 1
if standby -Atc 'SELECT count(*) FROM scratch' 2>"$pg_lab/read-error"; then
 echo 'unlogged read unexpectedly succeeded on standby' >&2
 exit 1
fi
grep -F 'cannot access temporary or unlogged relations during recovery' "$pg_lab/read-error"
pg_ctl -D "$pg_lab/standby" -w promote >/dev/null
test "$(standby -Atc "$counts")" = '1|0'
printf 'promoted standby: 1|0\n'
pg_ctl -D "$pg_lab/primary" -m immediate -w stop >/dev/null
start_primary
test "$(primary -Atc "$counts")" = '1|0'
printf 'crash recovery: 1|0\n'
primary -Atc 'SHOW server_version'
echo 'PASS: rollback, foreign key restriction, clean restart, standby, promotion, crash'

The primary and standby use separate Unix sockets inside one temporary directory. The trap stops only these experimental clusters and removes their files. The deliberate immediate shutdown affects the disposable primary, never an existing database. On PostgreSQL 18.6, rollback and clean restart leave counts 1|1; the recovery-mode standby rejects the scratch-table query. Promotion and primary crash recovery each produce 1|0. The exact error text is checked with the cluster’s C locale.

These observations establish lifecycle and constraint behavior, not a throughput improvement or a power-loss guarantee for every filesystem. Benchmark the real workload separately, including rebuild cost. Keep WAL and durability settings enabled for the durable destination; disabling them would invalidate the comparison.

Sources

PostgreSQL CREATE TABLE, unlogged contents and associated indexes, checked 2026-09-10.
PostgreSQL WAL introduction, recovery and durability roles, checked 2026-09-10.
PostgreSQL pg_ctl, clean and immediate shutdown modes, checked 2026-09-10.

Bucket versioning and lifecycles manage cost

Sat, 08 May 2027 00:00:00 +0000

Versioning turns an overwrite and a delete into appends. Instead of replacing bytes, a versioned bucket mints a new version ID and keeps the old one, so a mistaken PUT or a mistaken DELETE is reversible. That is genuine cheap disaster recovery for anything you manage through the bucket rather than through a database.

The same property is the problem. Every version you keep is storage you pay for, and the failure mode is not a dramatic one — it is a bucket whose bill climbs with write volume while the object count you can see stays flat.

Version IDs and the null version

With versioning on, every PUT produces a new version ID and the newest version becomes current. A GET without a version ID returns the current version; a GET with one returns exactly that version. So “the object” really means “the current version of the object”, and any tool that assumes one object per key is quietly wrong.

Objects written before versioning was enabled carry the version ID null, and enabling versioning does not retroactively version them. Replaying that sequence: a PUT before versioning stores null; enable versioning, then two overwrites, and the key now has three versions totalling 80 bytes. The null version is still there and is still a single slot that a non-versioned write could land on.

A delete marker is a tombstone, not a delete

A DELETE with no version ID on a versioned bucket writes a delete marker, and the marker becomes the current version. GET then returns 404 — but nothing was freed. In the replay the bucket holds 80 bytes before the delete and 80 bytes after it. The marker is a zero-byte entry that shadows everything underneath it.

The correct undo is to delete the delete marker, which makes the previous version current again and restores the 200. The wrong undo is to delete the newest object version while the marker is still on top: the key stays 404, and the newest recoverable copy is gone. That is the mistake a “clean up the bucket” script makes, and it is unrecoverable. MFA delete exists for exactly this class of action — permanently deleting a version, or turning versioning off, requires the MFA token.

Lifecycle is the only thing that bounds the bill

Two separate rules, and only one of them stops the growth. Expiration acts on the current version. NoncurrentVersionExpiration acts on everything underneath, and without it a bucket accumulates one live version per overwrite forever. Simulating 365 daily overwrites of one key: with no noncurrent rule the bucket ends the year holding 365 versions and 14,742 bytes; with a 30-day noncurrent rule it holds 29 versions and 1,186 bytes, a 12.4-fold difference. The current-version rule was set in both cases and did nothing to prevent it.

You can also transition noncurrent versions to a colder class instead of expiring them, which keeps the recoverable window cheap rather than free. That is usually the right trade for anything you might genuinely need to restore, but it costs a transition request per version, and on a bucket with millions of small objects the requests can exceed the storage saving.

When it breaks

The first failure is a bucket filling up because nothing expires noncurrent versions. It presents as a storage bill rising linearly with write volume while the visible object count is flat, and it is easy to misread as a leak. Diagnose by comparing the object count from a plain listing against the version count from a versioned listing: a hundred-to-one ratio is the signature, and the fix is one NoncurrentVersionExpiration rule, not a bigger bucket.

The second failure is an undelete that restores a delete marker instead of the object. A restore script that copies “the latest version” back into place copies the marker, because the marker is the latest version. The key stays invisible and the operator concludes the data is gone when it is sitting one entry down. Restore by explicit version ID, never by “latest”, and rehearse the restore before you need it.

Check it quickly

Read the version stack for one key, then the bucket’s rules:

aws s3api list-object-versions --bucket B --prefix reports/q3.pdf \
--query 'sort_by(Versions,&LastModified)[-3:].[VersionId,IsLatest,Size]' --output table
aws s3api list-object-versions --bucket B --prefix reports/q3.pdf --query 'DeleteMarkers'
aws s3api head-object --bucket B --key reports/q3.pdf --version-id <vid>
aws s3api get-bucket-lifecycle-configuration --bucket B

The first shows the newest three versions with sizes, and null appearing there means pre-versioning data you cannot expire by version. The second names any delete marker that is currently shadowing the key. The third proves a specific version is readable, which is the only form of restore worth testing, and the fourth tells you whether a noncurrent-version rule exists at all — if it does not, the bucket has no ceiling.

Sources

AWS S3 versioning, how versioning and the null version work, checked 2026-09-16.
AWS S3 delete markers, what a delete marker does and how to undo it, checked 2026-09-16.
AWS S3 lifecycle management, current and noncurrent expiration and transition rules, checked 2026-09-16.

Redis bitmaps flip bits instead of storing numbers

Thu, 29 Apr 2027 00:00:00 +0000

A Redis bitmap is a string interpreted as individual bits, not a separate key type. It is useful for dense yes/no membership such as daily activity. Ten million contiguous flags need 1,250,000 payload bytes, but that estimate excludes keys, object metadata, allocation overhead, and any mapping from business identifiers to bit positions.

Define the offset contract

SETBIT writes zero or one at a zero-based offset and returns the previous bit. Setting an already-set bit again does not increase membership. Offset zero is the most significant bit of the first byte; offset eight starts the second byte. Reading the same key with GET returns its underlying bytes, which need not be valid text.

GETBIT returns zero for a missing key or an offset beyond the current string without expanding it. Zero therefore does not distinguish an explicitly inactive user from an unrecorded one. Decide whether those states may share a representation. SETBIT and GETBIT require a string value and reject a hash rather than converting its fields into bits; BITOP instead writes a new string at its destination.

Memory follows the highest written offset, not the number of ones. Writing offset n requires at least floor(n / 8) + 1 payload bytes, even when the requested value is zero. Clearing that bit later does not shorten the string. Sparse external identifiers can therefore create a huge allocation for almost no membership. Validate the offset range and use a stable, suitably dense mapping.

Count and combine deliberately

BITCOUNT scans all bytes by default. Its optional start and end arguments are inclusive byte positions unless BIT is specified; that modifier is available from Redis 7.0. Changing units changes the selected interval, not the representation stored at the key. The command returns a population count, not the highest occupied identifier.

BITOP OR combines membership, while AND computes intersection. Shorter inputs and missing keys behave as zero-padded inputs up to the longest input length. The destination is overwritten with a new string; an existing destination expiration is removed. Derived results need their own deliberate lifetime policy, and recomputing them does not preserve a previously attached deadline.

When it breaks

BITCOUNT, BITPOS, and BITOP can scan substantial data. Optimized bit operations do not make full-bitmap work constant-time, and extending a string must allocate the intervening space. Evaluate offset span, query frequency, result lifetimes, and peak allocation together. A tiny number of active users says little about the resource cost of a sparse bitmap.

BITPOS returns an absolute bit offset even when searching a subrange. Searching for zero without an explicit end can return the first position beyond an all-one string, treating the right side as zero-padded. With both bounds specified, no matching bit in that interval returns -1. That difference matters when the bitmap represents a fixed-capacity resource pool.

Finding a zero is not reserving it. Two callers can read the same free position before either changes it. A separate BITPOS and SETBIT sequence is not one atomic claim, and ignoring SETBIT’s old-value result can double-assign a slot. Likewise, OR-ing two day keys is only meaningful when both use the same identifier mapping.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. Build only Redis core with libc and install the client in a temporary virtual environment; no optional modules or system-wide installation are required.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It uses its own Redis Unix socket under /tmp, disables TCP and persistence, and removes only its runtime directory after stopping its process. Downloading dependencies requires network access. The largest bitmap in this experiment has one MiB of payload; it does not allocate the enormous example offsets discussed above.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 assert client.getbit("missing", 100) == 0
 assert client.exists("missing") == 0
 assert client.setbit("active", 0, 1) == 0
 assert client.setbit("active", 2, 1) == 0
 assert client.setbit("active", 2, 1) == 1
 assert client.type("active") == b"string"
 assert client.get("active") == b"\xa0"
 assert client.bitcount("active") == 2
 print("offset zero is the high bit; repeated SETBIT returns old 1 without adding membership")

 highest = 8 * 1024 * 1024 - 1
 assert client.setbit("sparse", highest, 0) == 0
 assert client.strlen("sparse") == 1024 * 1024
 assert client.bitcount("sparse") == 0
 assert client.setbit("sparse", highest, 1) == 0
 assert client.setbit("sparse", highest, 0) == 1
 assert client.strlen("sparse") == 1024 * 1024
 print("writing a distant zero allocates 1048576 payload bytes; clearing does not shrink")

 client.set("range", b"\x80\x01")
 assert client.bitcount("range", 0, 1) == 2
 assert client.bitcount("range", 0, 1, mode="BIT") == 1
 assert client.bitpos("range", 1, 1, 1) == 15
 client.set("full", b"\xff")
 assert client.bitpos("full", 0) == 8
 assert client.bitpos("full", 0, 0, 0) == -1
 print("ranges default to bytes; BIT changes units; BITPOS returns absolute bit offsets")

 client.set("other", b"\x50\x01")
 client.set("union", b"obsolete", px=60000)
 assert client.bitop("OR", "union", "active", "other") == 2
 assert client.get("union") == b"\xf0\x01"
 assert client.bitcount("union") == 5
 assert client.ttl("union") == -1
 assert client.bitop("AND", "intersection", "active", "other") == 2
 assert client.get("intersection") == b"\x00\x00"
 print("BITOP zero-pads shorter inputs; OR has 5 flags; destination TTL is removed")

 client.hset("hash", "field", b"value")
 try:
 client.setbit("hash", 0, 1)
 except redis.ResponseError as error:
 assert "WRONGTYPE" in str(error)
 else:
 raise AssertionError("a hash is not converted into a bitmap")
 assert client.hget("hash", "field") == b"value"
 print("bitmap commands reject a hash without converting or replacing it")

Bits zero and two produce byte 0xa0 and count two. Writing a distant zero still creates 1,048,576 payload bytes; toggling and clearing it leaves that length unchanged. Byte-range counting and bit-range counting return two and one respectively, while the bounded BITPOS result remains an absolute offset.

An all-one byte yields position eight for an unbounded zero search but -1 for a bounded search. OR produces five flags across two bytes and removes the destination TTL; AND produces two zero bytes. Wrong-type access leaves the hash intact. These results verify representation and command semantics, not total process memory, concurrent allocation correctness, or production throughput.

Sources

Redis SETBIT, previous-bit replies and string growth, checked 2026-09-08.
Redis GETBIT, missing-key and out-of-range reads, checked 2026-09-08.
Redis BITCOUNT, population counts and byte/bit ranges, checked 2026-09-08.
Redis BITPOS, absolute offsets and bounded zero searches, checked 2026-09-08.
Redis BITOP, output strings, zero-padding, and operation cost, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

Object storage versus block storage

Wed, 17 Mar 2027 00:00:00 +0000

All storage is not the same. Block storage presents a disk you can mount and read at any byte; object storage presents a keyed, addressable store of objects with metadata. Choosing correctly is about the access pattern, not the buzzword.

Block storage is a byte-addressable disk

Block storage is a raw, byte-addressable disk: a volume is handed to a host or container as a device with a filesystem, and the guest owns low-level blocks, page cache, and filesystem. Because it behaves like a local disk, block storage suits databases, filesystems, and anything that needs low latency, byte-level writes, and file locking. Its weakness is elasticity and sharing: a block volume typically binds to one writer at a time, is not a natural fit for ‘millions of files read by many clients’, and generous sizes cost the same whether you use them or not. If your workload is a database or a VM, you are almost always asking for block.

Object storage is web-scale key-value blobs

Object storage is a web-scale key-value store for blobs: you address content by a flat, global key (a bucket name plus an object key), tens of thousands of clients hit it concurrently, and the API (GET/PUT/DELETE) makes it a natural fit for media, backups, and data lakes. Its latency is higher and its model is ‘write whole object, read whole object’, so it is the wrong shape for byte-for-byte OLTP mutations. The cool thing is that it is designed to be shared across unlimited readers, so the cost model and access pattern both favor storing the durable, big, rarely-mutated files here.

Match storage to the access pattern

Match storage to the access pattern, not to brand loyalty. If you need to append bytes, lock rows, run SQL, or mount a filesystem — block. If you need to store an archive, a log snapshot, or user-uploaded media that is written once and read many times, and you want it across many machines — object. The two are complements, not rivals: the common architecture runs block for hot, mutable data and object for cold, durable assets, and moves data between them intentionally. Choosing by how you read and write, not by what you already have, is what keeps the bill and the latency honest.

When it breaks

Treating object storage like a block device fails first. A database file or journal placed directly on a keyed object cannot take random byte writes or file locks: every small update becomes a full-object read-modify-write, concurrent writers silently replace each other, and a crash between download and re-upload leaves a truncated replacement instead of a recoverable page. Latency also betrays the mismatch, because a one-byte change still pays the cost of shipping the whole object.

The second failure is a million tiny objects. Each PUT, GET, and especially LIST is a billed API call with per-request latency, and a flat namespace with no real directories means enumerating one prefix can page through thousands of keys. A workload that writes one object per event and then lists the bucket to find work turns metadata into the bottleneck. The third failure is subtler: read-after-write visibility. A reader can fetch the previous version inside the consistency window, so two processes disagree about which object is current until the new version propagates. The fourth is lifecycle rules: one overly broad prefix or expiration policy quietly deletes backups, and without versioning that delete is final.

Check it quickly

Save the following as storage_semantics.py and run python3 storage_semantics.py. It uses only the standard library, touches no network or disk, and keeps all state in two in-memory objects. The block fixture supports in-place overwrites; the object fixture stores whole immutable values in a dict with a monotonically increasing version.

"""Block in-place overwrite vs object whole-object replace + versioning."""

class BlockVolume:
 def __init__(self, data: bytes):
 self._disk = bytearray(data)

 def write(self, offset: int, chunk: bytes) -> None:
 self._disk[offset:offset + len(chunk)] = chunk

 def read(self) -> bytes:
 return bytes(self._disk)


class ObjectStore:
 """Minimal object semantics: whole-value PUT, version per key."""

 def __init__(self):
 self._objects: dict[str, tuple[bytes, int]] = {}

 def put(self, key: str, data: bytes) -> int:
 version = self._objects[key][1] + 1 if key in self._objects else 1
 self._objects[key] = (bytes(data), version)
 return version

 def get(self, key: str) -> tuple[bytes, int]:
 return self._objects[key]


def main() -> None:
 # 1. Block: in-place overwrite keeps neighbours.
 block = BlockVolume(b"ABCDEF")
 block.write(2, b"XX")
 assert block.read() == b"ABXXEF", block.read()
 print(f"block: {block.read()!r} preserved neighbours")

 # 2. Object: PUT replaces the whole value and bumps the version.
 store = ObjectStore()
 v1 = store.put("report", b"ABCDEF")
 old_data, old_version = store.get("report")
 v2 = store.put("report", b"XYZ")
 new_data, new_version = store.get("report")
 assert (v1, v2) == (1, 2)
 assert len(old_data) == 6 and len(new_data) == 3
 assert old_version == 1 and new_version == 2
 assert new_data == b"XYZ", "object replace must not merge with prior bytes"
 print(f"object: v{v1} -> v{v2} len {len(old_data)} -> {len(new_data)}")

 # 3. Consistency window: a stale reader still holds v1.
 stale = (old_data, old_version)
 current = store.get("report")
 assert stale[1] != current[1], "versions must expose staleness"
 print(f"stale reader saw v{stale[1]}, current is v{current[1]}")

 # 4. No partial update: object API has no offset write.
 assert not hasattr(store, "write"), "object store must not offer block writes"
 print("PASS: block mutates in place, objects replace by version")


if __name__ == "__main__":
 main()

The assertions isolate the semantic gap. The block write changes two bytes while neighbours stay identical, which proves in-place mutation. The object replacement changes the length and bumps the version from one to two, which proves whole-object replacement with explicit history. The stale-version check reproduces the consistency window: a reader holding version one disagrees with version two. There is no partial object update without a read-modify-write cycle.

Expected output:

block: b'ABXXEF' preserved neighbours
object: v1 -> v2 len 6 -> 3
stale reader saw v1, current is v2
PASS: block mutates in place, objects replace by version

The run passed with Python 3.12 on macOS. It models semantics only, not latency, durability, pricing, or real consistency behavior. Before choosing storage, repeat the access-pattern question against production-sized objects and a real client library.

Sources

AWS: Amazon S3, object storage for scale, checked 2027-03-14.
Google Cloud: Cloud Storage, object storage for durable workloads, checked 2027-03-14.
Backblaze: B2 object storage, object storage for durable data, checked 2027-03-14.

Disk schedulers order I/O to move the head efficiently

Tue, 09 Mar 2027 00:00:00 +0000

blk-mq gives the block layer a software staging queue per CPU and a hardware dispatch queue per device queue, with the I/O scheduler between them. A request takes the shortest path — straight to the hardware queue — unless a scheduler is attached or it can be merged, in which case it is staged in software first. That scheduler is the only place in the stack where the kernel decides the order your requests reach the device.

Ordering is worth a great deal on a rotating disk and almost nothing on flash, but the scheduler has a second job that survives the move to NVMe: deciding who waits. Those jobs pull in different directions — reordering for head efficiency is indifferent to fairness, and fairness costs seeks — which is why the four schedulers are matched to a device and a workload rather than ranked.

The head is the budget

On a 7200 RPM disk, rotation costs 4.17 ms per request on average and transferring a 4 KiB block costs 0.026 ms, so almost all service time is seek and rotation. The fixture replays 2000 random 4 KiB reads across 100,000 tracks: in arrival order the head travels 67,907,411 tracks and the run takes 21,954 ms, or 91 IOPS; in ascending track order it travels 99,977 tracks and takes 8,645 ms, or 231 IOPS. Same requests and device: 679 times less head travel, 2.5 times the throughput.

Seek is 62 percent of service time in the arrival-order run and 3 percent in the sorted run, while rotation and transfer stay fixed at 4.19 ms either way. On NVMe the same requests take about 40 ms with no seek at all, so sorting buys nothing.

What each scheduler is for

none is first-come-first-served: right for a deep internal queue with no contention, wrong on a rotator, where the seek component stays untouched. mq-deadline sorts by sector and adds per-direction expiry — reads expire long before writes, each batch capped by fifo_batch — which bounds the tail none leaves unbounded. kyber targets latencies directly, throttling a queue whose requests miss the read or write target (read_lat_nsec, write_lat_nsec), so a bulk writer cannot push interactive reads past the target alone.

bfq is the fairness answer: a proportional-share scheduler with cgroup support that distributes device throughput — not just device time — among processes and groups in proportion to their weights. It privileges latency over throughput by default and idles the device, which helps on rotational media but can cost throughput on flash. Read the active scheduler with cat /sys/block/<dev>/queue/scheduler, which prints the list with the current one in brackets, and change it with echo bfq > /sys/block/sda/queue/scheduler; make that survive reboot with a udev rule, since the attribute is recreated per device.

Fairness is a different question from order

With a 90/10 issue mix between a bulk hog and a light interactive client, arrival order gives the light client 0.099 of the service and an average wait position of 1901 requests — not starved, drowned. Under per-group fair queueing it gets 0.5 of the service regardless of the mix, at the hog’s expense. Only the arbitration changed.

So on flash the choice is usually bfq for isolation or mq-deadline for bounded tails, and none only when one workload owns the device. On a rotator the sorted order is the larger win. In cgroup v2 terms, io.weight gives a group a share and io.max a ceiling; both only matter if the scheduler enforces the order.

When it breaks

The classic failure is a rotator running [none], usually because a host was cloned from an NVMe image or a udev rule matched sd* while the device is nvme0n1. The signature is %util near 100 percent with r/s low and await in the tens of milliseconds — the device is busy, not saturated, and it is busy seeking. Confirm with cat /sys/block/<dev>/queue/rotational: 1 with no reordering scheduler is the bug.

The second failure is a latency collapse with healthy averages. A backup job or a dd on the same device starves interactive reads, so p99 explodes while the mean barely moves. none and mq-deadline will not protect the interactive client; bfq or a cgroup io.max will. On flash, watch slice_idle: idling to preserve sequentiality is a throughput loss when there is no head to preserve.

Check it quickly

Read what the device is and which scheduler it is running:

for d in /sys/block/sd* /sys/block/nvme*n*; do echo "$d rot=$(cat $d/queue/rotational) sched=$(cat $d/queue/scheduler)"; done
cat /sys/block/sda/queue/iosched/read_expire /sys/block/sda/queue/iosched/fifo_batch
iostat -x 2 5
echo mq-deadline > /sys/block/sda/queue/scheduler

The first line pairs each device’s rotational flag with its active scheduler in brackets; rot=1 next to [none] is the failure signature, rot=0 next to [none] is normal. The second shows mq-deadline’s expiry and batch tunables. iostat -x distinguishes saturated from merely busy: high %util with low r/s and high await means queueing and seeking.

Sources

Switching Scheduler, the scheduler names, sysfs paths and how to change them, checked 2026-09-16.
Multi-Queue Block IO Queueing Mechanism (blk-mq), software staging queues versus hardware dispatch queues, checked 2026-09-16.
BFQ (Budget Fair Queueing), proportional-share behaviour, cgroup support and the idling trade-off, checked 2026-09-16.
Kyber I/O scheduler tunables, the read and write latency targets Kyber throttles against, checked 2026-09-16.

A stale replication slot holds WAL back

Mon, 19 Oct 2026 00:00:00 +0000

A PostgreSQL replication slot records retention requirements for a consumer. Once the slot has reserved a WAL position, losing the consumer’s connection does not automatically release that position. The primary may keep generating log while the slot’s requirement remains unchanged. The operational risk is retained storage and an increasingly expensive catch-up or rebuild decision. Slot usability depends on retained WAL availability and the configured invalidation rules.

When it breaks

Not every newly created physical slot immediately holds WAL. Without immediate reservation, restart_lsn can remain null until the slot is first used. An inactive flag alone therefore does not quantify retention. Inspect the slot type, reserved position, availability status, and consumer ownership together. For logical slots, confirmed_flush_lsn describes acknowledged consumer progress; it is not a replacement for understanding the older restart_lsn retention requirement.

Dropping a slot removes that slot’s claim, not necessarily the files at that instant. Checkpoint cleanup and other retention requirements still matter. Other slots, archive backlog, or a WAL reserve can keep files present, and recycling may rename them without returning all allocated bytes to the filesystem. Adding every slot’s lag also overcounts overlapping WAL ranges shared by several consumers.

Retention limits exchange one risk for another. max_slot_wal_keep_size is enforced around checkpoints, not as an instantaneous hard disk quota. A sufficiently lagging slot can lose required WAL and become unusable. That protects against indefinite retention at the cost of abandoning that consumer’s existing catch-up path. Age alone is not authority to discard a live replication dependency.

Check it quickly

Run this script as a non-root account with PostgreSQL 18 server and client binaries in a Linux shell environment. It was executed on PostgreSQL 18.6. The private temporary cluster has no TCP listener and contains only test slots. Small logged inserts and forced segment switches make retention visible quickly; this is not a benchmark of business WAL volume.

#!/bin/sh
set -eu
umask 077
lab=$(mktemp -d /tmp/xbcatm-slots.XXXXXX)
mkdir "$lab/socket"
cluster=$lab/data
cleanup() {
 if test -f "$cluster/postmaster.pid"; then
 pg_ctl -D "$cluster" -m fast -w stop >/dev/null
 fi
}
trap cleanup EXIT
initdb -D "$cluster" --no-locale --encoding=UTF8 --auth-local=trust --auth-host=reject >/dev/null
cat >> "$cluster/postgresql.conf" <<CONF
listen_addresses = ''
unix_socket_directories = '$lab/socket'
port = 56432
wal_level = replica
max_replication_slots = 4
max_slot_wal_keep_size = -1
min_wal_size = '32MB'
max_wal_size = '64MB'
checkpoint_timeout = '1h'
CONF
pg_ctl -D "$cluster" -l "$lab/server.log" -w -t 15 start >/dev/null
export PGHOST=$lab/socket PGPORT=56432 PGDATABASE=postgres PGUSER=$(id -un)
export PGCONNECT_TIMEOUT=2
psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
SELECT * FROM pg_create_physical_replication_slot('unused_slot');
SELECT active,restart_lsn IS NULL FROM pg_replication_slots WHERE slot_name='unused_slot';
DO $$ BEGIN
 IF (SELECT restart_lsn IS NULL FROM pg_replication_slots WHERE slot_name='unused_slot') IS NOT TRUE THEN
 RAISE EXCEPTION 'unexpected immediate reservation';
 END IF;
END $$;
SELECT pg_drop_replication_slot('unused_slot');
CREATE TABLE churn (value text);
SELECT * FROM pg_create_physical_replication_slot('held_slot',true);
SQL
segment=$(psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT pg_walfile_name(restart_lsn) FROM pg_replication_slots WHERE slot_name='held_slot'")
psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
DO $$ BEGIN
 FOR n IN 1..8 LOOP
 INSERT INTO churn SELECT md5(g::text) FROM generate_series(1,5000) g;
 PERFORM pg_switch_wal();
 END LOOP;
END $$;
CHECKPOINT;
SELECT active,wal_status,pg_wal_lsn_diff(pg_current_wal_lsn(),restart_lsn)>64*1024*1024
FROM pg_replication_slots WHERE slot_name='held_slot';
DO $$ BEGIN
 IF (SELECT NOT active AND wal_status='extended'
 AND pg_wal_lsn_diff(pg_current_wal_lsn(),restart_lsn)>64*1024*1024
 FROM pg_replication_slots WHERE slot_name='held_slot') IS NOT TRUE THEN
 RAISE EXCEPTION 'inactive slot did not retain the expected WAL range';
 END IF;
END $$;
SQL
test -f "$cluster/pg_wal/$segment"
echo inactive_slot_retains_old_segment
psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT pg_drop_replication_slot('held_slot')" >/dev/null
test -f "$cluster/pg_wal/$segment"
echo drop_did_not_immediately_remove_segment
psql -X -v ON_ERROR_STOP=1 -qAt -c CHECKPOINT
test ! -f "$cluster/pg_wal/$segment"
echo checkpoint_released_old_segment_name
psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
ALTER SYSTEM SET max_slot_wal_keep_size='32MB';
SELECT pg_reload_conf();
SQL
attempt=0
while test "$(psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT current_setting('max_slot_wal_keep_size')='32MB'")" != t; do
 attempt=$((attempt+1))
 test "$attempt" -lt 100 || exit 1
 sleep 0.1
done
psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
SELECT * FROM pg_create_physical_replication_slot('limited_slot',true);
DO $$ BEGIN
 FOR n IN 1..8 LOOP
 INSERT INTO churn SELECT md5(g::text) FROM generate_series(1,5000) g;
 PERFORM pg_switch_wal();
 END LOOP;
END $$;
CHECKPOINT;
SELECT wal_status,invalidation_reason FROM pg_replication_slots WHERE slot_name='limited_slot';
DO $$ BEGIN
 IF (SELECT wal_status='lost' AND invalidation_reason='wal_removed'
 FROM pg_replication_slots WHERE slot_name='limited_slot') IS NOT TRUE THEN
 RAISE EXCEPTION 'slot limit did not invalidate the test slot';
 END IF;
END $$;
SELECT pg_drop_replication_slot('limited_slot');
SQL
printf 'lab_directory %s\n' "$lab"

The first physical slot is created without immediate reservation. Its f|t output means inactive with a null restart position. The script asserts that condition and removes this unused fixture. The next slot explicitly reserves WAL and never connects a consumer. After writes, segment switches, and a checkpoint, the observed state is f|extended|t: inactive, retaining beyond the configured maximum WAL target, and more than sixty-four mebibytes behind by LSN distance.

The script checks the original required segment still exists, drops that one test slot, and checks the file remains immediately afterward. A subsequent checkpoint removes the old segment name. Those observations distinguish releasing a claim from immediate filesystem cleanup. They do not claim the complete WAL directory becomes empty or that recycled storage necessarily disappears.

The final phase sets a thirty-two-mebibyte slot retention limit, confirms the reload took effect, and reserves a new slot. More logged work and segment switches push it beyond the limit. After checkpointing, the required result is lost|wal_removed; an assertion checks both availability state and invalidation reason. This is an observed loss of the slot’s recovery path, not successful catch-up by a replica.

Only fixture slots are dropped. The temporary server stops on exit, and the printed directory retains its logs and data for review. No real consumer is disconnected or rebuilt.

Decide from ownership and recoverability

Monitor restart_lsn, wal_status, safe_wal_size, inactivity, actual retained bytes, and free space. A null safety margin can mean unlimited retention or an already lost slot, so interpret it with the other fields. PostgreSQL 18 also exposes an invalidation reason, which distinguishes removed WAL from other causes instead of guessing from elapsed silence.

Before retiring a slot, identify its owner and whether that consumer is expected to resume. Decide explicitly when rebuilding is preferable to retaining more history. If a required segment is already gone, increasing the limit does not recreate it. Verify the replacement consumer’s consistent starting point and ongoing progress before considering the incident resolved. A smaller disk graph alone does not prove replication service has recovered.

Sources

PostgreSQL 18 replication slots view, reserved positions, availability states, safety margins, and invalidation reasons, checked 2026-09-12.
PostgreSQL 18 replication configuration, checkpoint-enforced slot retention limits, checked 2026-09-12.
PostgreSQL 18 administrative functions, physical slot creation, immediate reservation, and dropping slots, checked 2026-09-10.

S3 storage classes trade cost for access

Wed, 30 Sep 2026 00:00:00 +0000

Choosing a storage class is a bet on when you will read the bytes again. S3 Standard costs $0.023 per GB-month in us-east-1; Glacier Deep Archive costs $0.00099. That is a 23-fold difference for identical durability, and the whole decision is which number your access pattern deserves.

The headline rate is the least informative number in the table. Retrieval charges, minimum storage durations, per-object metadata overhead and transition requests scale with object count rather than total bytes, so the cheap-looking class can be the expensive one for your workload.

What the real prices say

The published us-east-1 rates: Standard $0.023, Standard-IA $0.0125, One Zone-IA $0.010, Glacier Instant Retrieval $0.004, Glacier Flexible Retrieval $0.0036 and Deep Archive $0.00099 per GB-month. Retrieval is where the classes diverge more sharply: Standard-IA and One Zone-IA charge $0.010 per GB read, and Glacier Instant Retrieval charges $0.030.

The non-obvious part: in us-east-1 the current price list charges nothing per GB for standard-tier retrieval from Glacier Flexible Retrieval and Deep Archive. You pay per restore request instead — $0.00005 — and you wait hours. So “archive classes have retrieval fees” is only reliably true for the instant-retrieval classes. Assume otherwise and the arithmetic inverts: if Deep Archive charged the often-quoted $0.02 per GB, reading a 1 GB object 52 times a year would cost $1.05 instead of $0.015 — a 70-fold swing from one regional detail.

Minimum durations and minimum billable sizes

Minimum storage duration is 30 days for Standard-IA and One Zone-IA, 90 for Glacier Instant Retrieval, 90 for Glacier Flexible Retrieval, 180 for Deep Archive. Delete or transition out earlier and you are billed for the remainder — a 1 GB object deleted after 10 days from a 180-day class still costs $0.00586.

Minimum billable object size is the other trap. Standard-IA, One Zone-IA and Glacier Instant Retrieval bill a floor of 128 KB, so a 10 KB object stored in Glacier Instant Retrieval is charged as 128 KB — a 12.8-fold inflation. Glacier Flexible Retrieval and Deep Archive have no floor but add 40 KB of metadata per archived object, 32 KB at the class rate, 8 KB at the Standard rate. And the transition itself costs a request per object: $0.00001 into Standard-IA, $0.00002 into Glacier Instant Retrieval, $0.00003 into Glacier Flexible Retrieval, $0.00005 into Deep Archive. For a 10 KB object in Glacier Instant Retrieval that single request is 65% of the object’s whole annual cost. Since September 2024 the default lifecycle behaviour refuses to transition objects under 128 KB at all, for precisely that reason.

Break-even, not vibes

For a 1 GB object held a year, Standard-IA stops being cheaper than Standard after 13 reads, and Glacier Instant Retrieval after 8. Both land near one read a month, which is what the documented design points — “once a month”, “once a quarter” — describe. The break-even is computable, so compute it instead of assuming it.

The weekly reader is where this goes wrong. A 1 GB object read 52 times a year and transitioned to Glacier Instant Retrieval costs $1.61 against $0.276 on Standard, 5.8 times more, with $1.56 of retrieval fees accounting for the entire difference. A 30-day transition rule is right for a log archive and wrong for a weekly report, and the rule cannot tell them apart unless you scope it by prefix or tag.

When it breaks

The first failure is a lifecycle rule cooling objects that are read every week. It presents as a storage line that went down and a total bill that went up, with the difference sitting in retrieval charges elsewhere on the invoice. Pull per-class retrieval spend rather than looking at the storage line, and check whether any rule’s transition age is shorter than the interval between reads.

The second failure is objects stranded in an archive class with no tested restore path. Glacier Flexible Retrieval restores take minutes on the expedited tier and 5 to 12 hours in bulk; Deep Archive takes 12 to 48. A restore also produces a temporary copy with its own expiry, so a restore that worked last quarter may not work today. Worse, a 100 KB object in Standard-IA is already more expensive than Standard at zero reads, because the transition request exceeds the entire year’s storage saving — the object paid to become slower.

Check it quickly

See where the bytes actually are and what will happen to them:

aws s3api list-objects-v2 --bucket B --query 'Contents[].StorageClass' --output text | tr '\t' '\n' | sort | uniq -c
aws s3api get-bucket-lifecycle-configuration --bucket B --query 'Rules[].[ID,Status,Transitions[0].Days,Transitions[0].StorageClass]' --output table
aws s3api head-object --bucket B --key K --query 'StorageClass'

The first is the distribution across classes — a large GLACIER count on a bucket whose consumers read weekly is the whole problem. The second shows the transition ages; any age shorter than your read interval is a rule that will bill you for retrieval. The third confirms one object’s actual class, which is how you catch a lifecycle rule that silently skipped objects under 128 KB.

Sources

AWS S3 storage classes, what each class is designed for, checked 2026-09-16.
AWS S3 storage class guide, minimum storage durations and minimum billable object sizes, checked 2026-09-16.
AWS S3 lifecycle transition rules, transition constraints and the 128 KB default, checked 2026-09-16.
AWS S3 pricing, per-GB storage, retrieval and request charges, checked 2026-09-16.

Storage · XB Field Notes

LVM snapshots capture a block device in time

An exception store, not a copy

Exhaustion is invalidation

Merging, and why thin changes the failure

When it breaks

Check it quickly

Sources

ZFS snapshots give the filesystem a rewind button

A name for one consistent tree

Where the space is, in zfs list -o space

Holds, diff, and comparing versions

When it breaks

Check it quickly

Sources

High-cardinality labels increase stored series before anyone queries them

Count observed combinations, not isolated values

Querying fewer rows does not store fewer rows

Diagnose growth with the right inventory

When it breaks

Check it quickly

Sources

A checkpoint flushes dirty buffers to disk

When it breaks

Check it quickly

Tune against recovery and foreground work

Sources

Partitioning splits one table into many pieces that look like one

A logical parent with physical children

The key affects more than routing

When it breaks

Check it quickly

Sources

Redis append-only log trades speed for durability

Execution, logging, and acknowledgment

Choose the synchronization contract

Rewriting creates a recoverable file set

When it breaks

Check it quickly

Sources

Btrfs snapshots are copy-on-write, not copies

Shared extents, not copied blocks

Where the space actually goes

A restore point is not a replica

When it breaks

Check it quickly

Sources

An LRU cache evicts the least recently used entry

Follow the access order

Exact and sampled policies differ

Capacity is not freshness

When it breaks

Check it quickly

Further reading

Sources

LSM trees swallow random writes

Buffer, flush, merge

Compaction is the tax

When it breaks

Check it quickly

Sources

A sparse file separates logical length from allocated blocks

When it breaks

Check it quickly

Budget and preserve the intended representation

Sources

A PVC gives a Pod durable storage beyond the container

The claim, the volume, and the binding

The deletion order and the finalizer

Reclaim policy: Delete or Retain

When it breaks

Check it quickly

Sources

NFS shares a remote directory over the network

The file and the table

What the options actually decide

Mounting, and the id mapping trap

When it breaks

Check it quickly

Sources