Backup · XB Field Notes

pg_dump captures a consistent snapshot

Fri, 14 Jan 2028 00:00:00 +0000

pg_dump exports one PostgreSQL database using a consistent view of transactional data, even while other sessions commit changes. This prevents the ordinary table-by-table copy problem in which related tables reflect different committed states. The snapshot is only part of the backup contract: the archive must also be complete for its intended scope, readable by compatible tools, and successfully restored into a usable database.

Know what the snapshot covers

Rows visible to the dump’s snapshot remain its view throughout the export. Later commits do not gradually enter the dump just because one table is read after another. Parallel dumps synchronize workers’ snapshots rather than independently observing each table. They require directory format; custom format supports parallel restoration, but not parallel dumping.

A database snapshot does not make every business workflow atomic. If an application commits a batch header and its detail rows in separate transactions, a consistent snapshot can legitimately fall between those commits. Sequence state is also not ordinary MVCC row data, and external files or services are outside the database snapshot. Define the recovery invariant around actual transaction boundaries and all required data stores.

Account for load and dependencies

Dumping normally allows ordinary readers and writers to continue, but it still takes ACCESS SHARE locks and consumes CPU, I/O, connections, and snapshot lifetime. Conflicting schema changes can wait or make the dump fail. Long snapshots can prevent cleanup of old row versions. Parallel workers increase load; they do not make backup overhead disappear.

A standby can offload some reads, but recovery conflicts and lag need consideration, and unlogged table contents are excluded when dumping from a standby. One pg_dump does not include cluster-wide roles and tablespaces. Extensions, ownership, permissions, collation behavior, and target configuration must be included in the restore plan through their appropriate mechanisms. Splitting an export into unrelated table dumps can lose the common snapshot and dependency closure.

When it breaks

An archive listing can look healthy while restoration fails on a missing role or extension. Restore into an empty, isolated target with explicit error handling, then validate application invariants and representative queries. pg_restore normally continues after SQL errors and reports an error count; checking only the last printed query is inadequate. A single-transaction restore can provide atomicity for a small rehearsal, but cannot be combined with parallel jobs.

Tool versions are another boundary. A pg_dump client refuses a server newer than its own major version. Output is intended for the same or newer target versions; loading it into an older major version is not guaranteed. Verify the actual client, source, target, extensions, and restore options rather than relying on the generic claim that “Postgres backups are portable.”

Logical exports can meet an explicitly measured recovery objective, but they do not provide arbitrary point-in-time replay between dumps. Physical base backups plus a complete retained WAL chain serve that requirement. The acceptable interval follows the required data-loss window and restore time, not a universal rule that anything shorter than a day requires one specific schedule.

Check it quickly

Save the Bash block as dump.sh. It creates its own PostgreSQL cluster and removes it on exit. Run it with PostgreSQL 18 tools as a non-root user, or use this isolated container with no published port or host-data mount:

docker run --rm --network none --user postgres --entrypoint bash -i \
 postgres:18-alpine < dump.sh

#!/usr/bin/env bash
set -euo pipefail
dump_lab=$(mktemp -d /tmp/pgdump-lab.XXXXXX)
cleanup() {
 if [ -f "$dump_lab/data/postmaster.pid" ]; then
 pg_ctl -D "$dump_lab/data" -m fast -w stop >/dev/null
 fi
 rm -rf -- "$dump_lab"
}
trap cleanup EXIT
initdb -D "$dump_lab/data" --no-locale --encoding=UTF8 --auth=trust >/dev/null
pg_ctl -D "$dump_lab/data" -l "$dump_lab/server.log" \
 -o "-k $dump_lab -p 55432 -c listen_addresses=''" -w start >/dev/null
export PGHOST="$dump_lab" PGPORT=55432 PGDATABASE=source_db
export DUMP_FILE="$dump_lab/source.dump" DUMP_OK="$dump_lab/dump-ok"
createdb source_db
createdb restored_db
psql -X -v ON_ERROR_STOP=1 <<'SQL'
CREATE TABLE customers (id integer PRIMARY KEY, name text);
CREATE TABLE orders (id integer PRIMARY KEY, customer_id integer REFERENCES customers(id));
INSERT INTO customers VALUES (1,'before');
INSERT INTO orders VALUES (10,1);
BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY;
SELECT pg_export_snapshot() AS snapshot_id \gset
\setenv DUMP_SNAPSHOT :snapshot_id
\! psql -X -v ON_ERROR_STOP=1 -c "BEGIN; INSERT INTO customers VALUES (2,'after'); INSERT INTO orders VALUES (20,2); COMMIT;" && pg_dump --format=custom --snapshot="$DUMP_SNAPSHOT" --file="$DUMP_FILE" && touch "$DUMP_OK"
COMMIT;
SQL
test -f "$DUMP_OK"
pg_restore --list "$DUMP_FILE" > "$dump_lab/archive-list"
pg_restore --exit-on-error --single-transaction --dbname=restored_db "$DUMP_FILE"
counts='SELECT (SELECT count(*) FROM customers), (SELECT count(*) FROM orders)'
test "$(psql -X -Atc "$counts")" = '2|2'
test "$(psql -X -d restored_db -Atc "$counts")" = '1|1'
test "$(psql -X -d restored_db -Atc 'SELECT count(*) FROM orders o LEFT JOIN customers c ON c.id=o.customer_id WHERE c.id IS NULL')" = 0
psql -X -d restored_db -v ON_ERROR_STOP=1 -c \
 'SELECT c.name,o.id FROM customers c JOIN orders o ON o.customer_id=c.id'
pg_dump --version
printf 'PASS: source=2|2 restored=1|1 orphan_orders=0\n'

The first session exports a snapshot containing one customer and one order. While that transaction remains open, a second session commits another related pair. The dump explicitly imports the earlier snapshot; restoration therefore produces 1|1, while the live source reports 2|2. The join returns customer “before” with order ten, and an orphan check returns zero.

On PostgreSQL 18.6, the complete restore and all assertions pass. The success marker ensures a failed nested shell dump cannot masquerade as success merely because psql continued. This demonstrates snapshot selection and restoration for the fixture, not a measured recovery time for the real application. Keep actual restore rehearsals, artifact checksums, retention, and dependency records with the production backup procedure.

Sources

PostgreSQL pg_dump, snapshots, formats, scope and version compatibility, checked 2026-09-10.
PostgreSQL pg_restore, restore error handling and transaction options, checked 2026-09-10.

Postgres WAL archiving reaches point-in-time recovery

Fri, 25 Jun 2027 00:00:00 +0000

Point-in-time recovery starts with a valid physical base backup and replays the WAL required to reach a chosen recovery target. The target must fall within the recoverable history of that backup and its available timelines. Archiving does not create the ability to recover to arbitrary earlier dates, nor does it repair a gap between the backup and the desired endpoint.

When it breaks

A copied data directory is not automatically a valid online backup. Use a supported backup method, preserve the required WAL and timeline history, and verify the backup manifest. pg_verifybackup finds many integrity problems, but its success does not demonstrate that later archived WAL exists or that a business recovery target produces the intended state. That requires starting recovery and checking the result.

Choosing a target also requires precision. Time, transaction, LSN, and named targets express different boundaries. For applicable target types, recovery_target_inclusive controls whether the specified boundary is included. A named restore point avoids clock ambiguity in this drill, but it only exists because an administrator deliberately wrote that marker. Reusing its name can stop recovery at the first matching point instead of the one an operator intended.

Check it quickly

Run this script as a non-root account in a Linux environment with PostgreSQL 18 server and client binaries, cp -a, mktemp, hard links, and sync FILE available. It was executed on PostgreSQL 18.6. Private temporary directories and Unix sockets isolate the clusters; TCP listening is disabled. The local archive exercises recovery mechanics without claiming protection against loss of the host.

#!/bin/sh
set -eu
umask 077
lab=$(mktemp -d /tmp/xbcatm-pitr.XXXXXX)
mkdir "$lab/socket" "$lab/archive"
primary=$lab/primary
restored=$lab/restored
missing=$lab/missing
cleanup() {
 for cluster in "$missing" "$restored" "$primary"; do
 if test -f "$cluster/postmaster.pid"; then
 pg_ctl -D "$cluster" -m fast -w stop >/dev/null
 fi
 done
}
trap cleanup EXIT
initdb -D "$primary" --no-locale --encoding=UTF8 --auth-local=trust --auth-host=reject >/dev/null
cat >> "$primary/postgresql.conf" <<CONF
listen_addresses = ''
unix_socket_directories = '$lab/socket'
port = 56432
wal_level = replica
archive_mode = on
archive_command = '$lab/archive-copy.sh "%p" "%f" "$lab/archive"'
CONF
cat > "$lab/archive-copy.sh" <<'ARCHIVE_HELPER'
#!/bin/sh
set -eu
wal_source=$1
wal_name=$2
archive_directory=$3
archive_target=$archive_directory/$wal_name
if test -e "$archive_target"; then
 cmp -s "$wal_source" "$archive_target"
 sync "$archive_target" "$archive_directory"
 exit 0
fi
archive_temporary=$(mktemp "$archive_directory/.incoming.XXXXXX")
trap 'rm -f "$archive_temporary"' EXIT HUP INT TERM
cp "$wal_source" "$archive_temporary"
sync "$archive_temporary"
ln "$archive_temporary" "$archive_target" || cmp -s "$wal_source" "$archive_target"
sync "$archive_target" "$archive_directory"
ARCHIVE_HELPER
chmod 700 "$lab/archive-copy.sh"
pg_ctl -D "$primary" -l "$lab/primary.log" -w -t 15 start >/dev/null
export PGHOST=$lab/socket PGPORT=56432 PGDATABASE=postgres PGUSER=$(id -un)
export PGCONNECT_TIMEOUT=2
psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
CREATE TABLE marker (id integer PRIMARY KEY,value text);
INSERT INTO marker VALUES (1,'base');
SQL
pg_basebackup -D "$lab/base" -X stream -c fast -w
pg_verifybackup -q "$lab/base"
psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT pg_switch_wal()' >/dev/null
psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
INSERT INTO marker VALUES (2,'keep');
SELECT pg_create_restore_point('before_bad_change');
UPDATE marker SET value='bad' WHERE id=1;
DELETE FROM marker WHERE id=2;
SQL
segment=$(psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT pg_walfile_name(pg_current_wal_lsn())')
psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT pg_switch_wal()' >/dev/null
attempt=0
while ! test -f "$lab/archive/$segment"; do
 attempt=$((attempt+1))
 test "$attempt" -lt 100 || { echo 'archive segment did not arrive' >&2; exit 1; }
 sleep 0.1
done
printf 'primary_now '
psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT string_agg(id||':'||value,',' ORDER BY id) FROM marker"
mkdir "$restored"
cp -a "$lab/base/." "$restored/"
cat >> "$restored/postgresql.conf" <<CONF
port = 56433
archive_mode = off
restore_command = 'cp "$lab/archive/%f" "%p"'
recovery_target_name = 'before_bad_change'
recovery_target_action = 'pause'
CONF
touch "$restored/recovery.signal"
pg_ctl -D "$restored" -l "$lab/restored.log" -w -t 15 start >/dev/null
attempt=0
while test "$(PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT pg_get_wal_replay_pause_state()='paused'")" != t; do
 attempt=$((attempt+1))
 test "$attempt" -lt 100 || { echo 'restore target was not reached' >&2; exit 1; }
 sleep 0.1
done
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
SELECT pg_is_in_recovery(),pg_get_wal_replay_pause_state();
SELECT id,value FROM marker ORDER BY id;
DO $$ BEGIN
 IF (SELECT string_agg(id||':'||value,',' ORDER BY id) FROM marker)
 IS DISTINCT FROM '1:base,2:keep' THEN
 RAISE EXCEPTION 'point-in-time result mismatch';
 END IF;
END $$;
SQL
test ! -e "$lab/base/pg_wal/$segment"
mkdir "$lab/archive-missing" "$missing"
for archived_file in "$lab/archive/"*; do
 if test "${archived_file##*/}" != "$segment"; then
 cp "$archived_file" "$lab/archive-missing/"
 fi
done
cp -a "$lab/base/." "$missing/"
cat >> "$missing/postgresql.conf" <<CONF
port = 56434
archive_mode = off
restore_command = 'cp "$lab/archive-missing/%f" "%p"'
recovery_target_name = 'before_bad_change'
recovery_target_action = 'pause'
CONF
touch "$missing/recovery.signal"
if pg_ctl -D "$missing" -l "$lab/missing.log" -w -t 15 start >/dev/null 2>&1; then
 echo 'missing-segment server was briefly available'
fi
attempt=0
while pg_ctl -D "$missing" status >/dev/null 2>&1; do
 attempt=$((attempt+1))
 test "$attempt" -lt 100 || { echo 'missing-segment server did not stop' >&2; exit 1; }
 sleep 0.1
done
grep -F 'recovery ended before configured recovery target was reached' "$lab/missing.log"
test -f "$lab/archive/$segment"
echo missing_segment_target_unreachable
printf 'lab_directory %s\nrequired_segment %s\n' "$lab" "$segment"

The primary first contains row one with value base. After taking and verifying the base backup, the script switches WAL segments, inserts row two with value keep, and creates before_bad_change. Only afterward does it change row one to bad and delete row two. primary_now 1:bad confirms the unwanted changes really reached the source database.

Recovery starts from a separate copy of the base backup. recovery.signal requests archive recovery; restore_command supplies archived files; the named target and pause action stop replay for inspection. The script waits for pg_get_wal_replay_pause_state() to report paused. Merely receiving true from pg_is_wal_replay_paused() would show that a pause was requested, not necessarily that replay has stopped.

Expected recovered output is t|paused, followed by 1|base and 2|keep. A SQL assertion checks that exact combined result. This demonstrates preservation of the earlier good transaction and exclusion of the later bad changes. It is stronger evidence than a server accepting connections, which can happen while recovery is still progressing.

The negative control makes another archive copy while excluding the required post-backup segment. It preserves the original archive and confirms that the base backup did not already contain that segment. Starting recovery against the incomplete copy must end with the fatal message that recovery ended before the configured target was reached. The script checks the server has stopped and inspects the log, rather than interpreting only the initial pg_ctl return value. missing_segment_target_unreachable records this expected failure.

All experimental clusters stop on exit. Their printed directory retains logs and data for inspection. No recovered copy is promoted or exposed to applications by this drill.

Verify before accepting recovery

A paused target provides an opportunity to compare business invariants before completing recovery. Check more than a row count: transaction boundaries, related records, and the last acceptable operation matter. Resuming a target pause completes recovery; it is not a request to keep applying arbitrary later changes. To pursue a later target, follow the documented shutdown and target-change procedure.

Record the base backup identity, available timeline, chosen target, recovered facts, and elapsed recovery time. Keep configuration and required external software alongside the recovery plan, since WAL does not restore every external dependency. Protect the original backup and archive while experimenting. A tested target establishes one recovery result; recurring drills across the retention interval establish whether the promised recovery window remains usable as backups rotate.

Sources

PostgreSQL 18 continuous archiving, base backups, archive recovery, and timeline requirements, checked 2026-09-10.
PostgreSQL 18 recovery targets, target boundaries, pause behavior, and fatal failure before an unreachable target, checked 2026-09-10.
PostgreSQL 18 administrative functions, named restore points and actual replay-pause state, checked 2026-09-10.
PostgreSQL 18 pg_verifybackup, manifest verification and why restore testing remains necessary, checked 2026-09-10.

Backups are untested hope without proof

Wed, 02 Jun 2027 00:00:00 +0000

A completed backup job establishes that the backup process produced an artifact. A restore rehearsal establishes whether that artifact, the available tooling, and the required environment can produce a usable system. These are different claims. Recovery evidence should identify the backup used, the state recovered, the dependencies supplied, and the time needed to make the result useful to the application.

When it breaks

A valid database archive can depend on objects outside its scope. PostgreSQL pg_dump exports one database, while roles are cluster-wide objects. Restoring a dump with grants into an independent cluster can fail because the referenced role does not exist. Rehearsing only in the original cluster can hide that dependency because the missing recovery material is already present there.

Restore errors also need deliberate handling. By default, pg_restore can continue after SQL errors and report their count at the end. A half-restored schema should not be accepted merely because some tables can be queried. For this small fixture, a single transaction makes success atomic and makes an expected failure leave no partial restored tables.

The correct recovery point is another requirement. A perfectly restored backup may intentionally lack changes made afterward. A checksum proves agreement with the recorded artifact, not that the artifact includes the newest business event. Likewise, a row count alone can match while amounts, relationships, or privileges are wrong. Choose checks that express the application’s meaning.

Check it quickly

Run this script as a non-root Linux account with PostgreSQL 18 server and client programs and sha256sum available. It was executed on PostgreSQL 18.6. It creates separate temporary source and target clusters with private Unix sockets and no TCP listeners. The fixture role cannot log in; its grants are tested through controlled role switching rather than through real credentials.

#!/bin/sh
set -eu
umask 077
lab=$(mktemp -d /tmp/xbcatm-restore.XXXXXX)
mkdir "$lab/socket"
source=$lab/source
target=$lab/target
cleanup() {
 for cluster in "$target" "$source"; do
 if test -f "$cluster/postmaster.pid"; then
 pg_ctl -D "$cluster" -m fast -w stop >/dev/null
 fi
 done
}
trap cleanup EXIT
for cluster in "$source" "$target"; do
 initdb -D "$cluster" --no-locale --encoding=UTF8 --auth-local=trust --auth-host=reject >/dev/null
 cat >> "$cluster/postgresql.conf" <<CONF
listen_addresses = ''
unix_socket_directories = '$lab/socket'
CONF
done
printf 'port = 56432\n' >> "$source/postgresql.conf"
printf 'port = 56433\n' >> "$target/postgresql.conf"
pg_ctl -D "$source" -l "$lab/source.log" -w -t 15 start >/dev/null
pg_ctl -D "$target" -l "$lab/target.log" -w -t 15 start >/dev/null
export PGHOST=$lab/socket PGPORT=56432 PGDATABASE=postgres PGUSER=$(id -un)
export PGCONNECT_TIMEOUT=2
psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
CREATE ROLE backup_reader NOLOGIN;
CREATE TABLE customers (id integer PRIMARY KEY,name text NOT NULL);
CREATE TABLE orders (id integer PRIMARY KEY,customer_id integer REFERENCES customers,amount_cents integer CHECK (amount_cents>0));
INSERT INTO customers VALUES (1,'Ada');
INSERT INTO orders VALUES (10,1,4210),(11,1,1865);
CREATE VIEW order_totals AS SELECT c.name,sum(o.amount_cents) AS total_cents FROM customers c JOIN orders o ON o.customer_id=c.id GROUP BY c.id,c.name;
GRANT SELECT ON customers,orders,order_totals TO backup_reader;
SQL
pg_dump -Fc -w -f "$lab/snapshot.dump" postgres
sha256sum "$lab/snapshot.dump" > "$lab/snapshot.sha256"
psql -X -v ON_ERROR_STOP=1 -qAt -c 'INSERT INTO orders VALUES (12,1,1000)'
printf 'source_after_backup_orders '
psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT count(*) FROM orders'
sha256sum -c "$lab/snapshot.sha256"
if PGPORT=56433 pg_restore --single-transaction --exit-on-error -w -d postgres "$lab/snapshot.dump" >"$lab/missing-role.log" 2>&1; then
 echo 'restore unexpectedly ignored a missing role' >&2
 exit 1
fi
grep -F 'role "backup_reader" does not exist' "$lab/missing-role.log"
test "$(PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT to_regclass('public.orders') IS NULL")" = t
echo failed_restore_rolled_back
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c 'CREATE ROLE backup_reader NOLOGIN'
PGPORT=56433 pg_restore --single-transaction --exit-on-error -w -d postgres "$lab/snapshot.dump"
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
SELECT count(*) FROM orders;
SELECT name,total_cents FROM order_totals;
SELECT has_table_privilege('backup_reader','orders','SELECT'),has_table_privilege('backup_reader','orders','INSERT');
DO $$ BEGIN
 IF (SELECT count(*) FROM orders)<>2 OR
 (SELECT string_agg(id||':'||amount_cents,',' ORDER BY id) FROM orders) IS DISTINCT FROM '10:4210,11:1865' OR
 (SELECT total_cents FROM order_totals WHERE name='Ada') IS DISTINCT FROM 6075 OR
 (SELECT count(*) FROM orders o LEFT JOIN customers c ON c.id=o.customer_id WHERE c.id IS NULL)<>0 THEN
 RAISE EXCEPTION 'restored business data mismatch';
 END IF;
END $$;
DO $$ BEGIN
 BEGIN
 INSERT INTO orders VALUES (98,999,100);
 RAISE EXCEPTION 'restored foreign key unexpectedly accepted an orphan';
 EXCEPTION WHEN foreign_key_violation THEN
 RAISE NOTICE 'restored foreign key rejected orphan';
 END;
END $$;
SET ROLE backup_reader;
SELECT id,amount_cents FROM orders ORDER BY id;
DO $$ BEGIN
 BEGIN
 INSERT INTO orders VALUES (99,1,100);
 RAISE EXCEPTION 'read-only role unexpectedly wrote data';
 EXCEPTION WHEN insufficient_privilege THEN
 RAISE NOTICE 'read-only role rejected write';
 END;
END $$;
SQL
printf 'lab_directory %s\n' "$lab"

The source contains one customer, two orders, a totals view, a foreign key, and a read-only role grant. A custom-format dump captures that state. The source then receives a third order. Its printed count of three confirms that later activity exists outside the chosen backup point, rather than allowing a test to confuse the current source with the saved snapshot.

The checksum check succeeds, but the first restore must still fail because backup_reader is absent on the target. The script checks that specific error and verifies the orders table was rolled back. This is the dependency failure that an artifact-only check would miss. It then creates the documented fixture role and repeats the restore with the same archive.

The recovered target must contain two orders with the original identifiers and amounts, and the view must report Ada|6075. The third order is correctly absent. An attempted orphan order must fail through the restored foreign key. Permission checks report allowed selection and denied insertion; switching to the reader role confirms that reads work and a real write attempt is rejected.

These assertions cover data values, an application query, a relationship constraint, and effective privileges. All experimental servers stop on exit, and the printed directory retains the dump and logs. The drill does not test external authentication, encryption-key recovery, or application startup, which must be added when those are part of the actual recovery dependency set.

Record a recovery result people can use

A production rehearsal should retrieve the artifact from its intended recovery source, rebuild the required environment, restore data, and exercise representative application behavior. Provisioning compatible software is a normal part of recovery; include its elapsed time and prerequisites instead of assuming a prepared host will always exist. Keep recovery access available when production systems are unavailable.

Measure recovery duration against the recovery-time objective, and compare the recovered endpoint with the recovery-point objective. Include retrieval, validation, and the application becoming usable, not just the database command’s runtime. For larger restores, choose transaction and parallelism options appropriate to the workload, with an explicit partial-failure policy.

Repeat the rehearsal after changes to backup tooling, schema, roles, retention, or infrastructure. Preserve the successful checks and any corrected failure as evidence. A repeatable restore procedure with known inputs and limits supports an operational decision; a green backup scheduler alone does not answer whether recovery is ready.

Sources

PostgreSQL 18 pg_dump, database-level scope and excluded cluster-wide objects, checked 2026-09-12.
PostgreSQL 18 pg_restore, error handling, single-transaction restoration, grants, and parallelism constraints, checked 2026-09-12.
CISA StopRansomware Guide, testing backup availability and integrity with recovery dependencies, checked 2026-09-12.

WAL archiving preserves a recoverable history

Thu, 22 Apr 2027 00:00:00 +0000

Continuous WAL archiving preserves completed log files outside PostgreSQL’s active WAL directory. Together with a suitable base backup and an uninterrupted recovery chain, those files let recovery replay logged database changes to a covered target. Archiving is separate from ordinary commit acknowledgment: a transaction can commit while its WAL segment is still open or while the archive destination is unavailable.

When it breaks

PostgreSQL modifies buffered data pages during execution. Before a dependent page reaches durable storage, the necessary WAL must be durable first. This ordering does not mean that data files are never written, or that every commit waits for archiving. Temporary and unlogged table contents also do not gain ordinary WAL recovery guarantees just because archiving is enabled.

The archive command’s return code is a data-integrity contract. Zero means the file has been preserved successfully; PostgreSQL can eventually recycle the source when other retention requirements permit. A command that reports success after skipping a failed copy silently breaks that contract. Nonzero results cause retries, and repeated failure retains unarchived WAL, potentially filling the primary’s filesystem. A checkpoint does not repair the destination or discard that obligation.

Existing names need careful treatment. Repeated archival of identical, fully persisted content should succeed. Different bytes under the same name must fail without overwriting the original. Returning success merely because a destination exists can conceal corruption or archives accidentally mixed from different clusters.

Check it quickly

Run the following script as a non-root account with PostgreSQL 18 server and client binaries in PATH. It requires a Linux shell environment supporting cp -a, mktemp, hard links, and sync FILE. The drill was executed on PostgreSQL 18.6. It creates its own private temporary cluster and Unix socket, disables TCP listening, and uses only test data. The local archive illustrates the command contract; it is not an independent disaster-recovery copy.

#!/bin/sh
set -eu
umask 077
lab=$(mktemp -d /tmp/xbcatm-archive.XXXXXX)
mkdir "$lab/socket" "$lab/archive"
primary=$lab/primary
cleanup() {
 if test -f "$primary/postmaster.pid"; then
 pg_ctl -D "$primary" -m fast -w stop >/dev/null
 fi
}
trap cleanup EXIT
initdb -D "$primary" --no-locale --encoding=UTF8 --auth-local=trust --auth-host=reject >/dev/null
cat > "$lab/archive-copy.sh" <<'ARCHIVE_HELPER'
#!/bin/sh
set -eu
wal_source=$1
wal_name=$2
archive_directory=$3
archive_target=$archive_directory/$wal_name
if test -e "$archive_target"; then
 cmp -s "$wal_source" "$archive_target"
 sync "$archive_target" "$archive_directory"
 exit 0
fi
archive_temporary=$(mktemp "$archive_directory/.incoming.XXXXXX")
trap 'rm -f "$archive_temporary"' EXIT HUP INT TERM
cp "$wal_source" "$archive_temporary"
sync "$archive_temporary"
ln "$archive_temporary" "$archive_target" || cmp -s "$wal_source" "$archive_target"
sync "$archive_target" "$archive_directory"
ARCHIVE_HELPER
chmod 700 "$lab/archive-copy.sh"
cat >> "$primary/postgresql.conf" <<CONF
listen_addresses = ''
unix_socket_directories = '$lab/socket'
port = 56432
wal_level = replica
archive_mode = on
archive_command = '$lab/archive-copy.sh "%p" "%f" "$lab/archive"'
CONF
pg_ctl -D "$primary" -l "$lab/primary.log" -w -t 15 start >/dev/null
export PGHOST=$lab/socket PGPORT=56432 PGDATABASE=postgres PGUSER=$(id -un)
export PGCONNECT_TIMEOUT=2
psql -X -v ON_ERROR_STOP=1 -qAt -c 'CREATE TABLE archive_marker (id integer PRIMARY KEY)'
pg_basebackup -D "$lab/base" -X stream -c fast -w
pg_verifybackup -q "$lab/base"
echo base_backup_verified
failures_before=$(psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT failed_count FROM pg_stat_archiver')
psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
ALTER SYSTEM SET archive_command='/bin/false';
SELECT pg_reload_conf();
SQL
attempt=0
while test "$(psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT current_setting('archive_command')='/bin/false'")" != t; do
 attempt=$((attempt+1))
 test "$attempt" -lt 100 || exit 1
 sleep 0.1
done
psql -X -v ON_ERROR_STOP=1 -qAt -c 'INSERT INTO archive_marker VALUES (1)' -c 'SELECT pg_switch_wal()' >/dev/null
attempt=0
while test "$(psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT failed_count>$failures_before FROM pg_stat_archiver")" != t; do
 attempt=$((attempt+1))
 test "$attempt" -lt 700 || { echo 'archive failure was not observed' >&2; exit 1; }
 sleep 0.1
done
failed_segment=$(psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT last_failed_wal FROM pg_stat_archiver')
test ! -f "$lab/archive/$failed_segment"
echo archive_failure_observed
psql -X -v ON_ERROR_STOP=1 -qAt -c CHECKPOINT
test -f "$primary/pg_wal/$failed_segment"
echo unarchived_wal_retained_after_checkpoint
archived_before=$(psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT archived_count FROM pg_stat_archiver')
psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
ALTER SYSTEM RESET archive_command;
SELECT pg_reload_conf();
SQL
attempt=0
while ! test -f "$lab/archive/$failed_segment" || test "$(psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT archived_count>$archived_before FROM pg_stat_archiver")" != t; do
 attempt=$((attempt+1))
 test "$attempt" -lt 700 || { echo 'archive repair was not observed' >&2; exit 1; }
 sleep 0.1
done
cmp -s "$primary/pg_wal/$failed_segment" "$lab/archive/$failed_segment"
echo archive_repaired
"$lab/archive-copy.sh" "$primary/pg_wal/$failed_segment" "$failed_segment" "$lab/archive"
echo identical_repeat_accepted
printf 'different bytes\n' > "$lab/conflicting-input"
if "$lab/archive-copy.sh" "$lab/conflicting-input" "$failed_segment" "$lab/archive"; then
 echo 'conflicting archive input was accepted' >&2
 exit 1
fi
cmp -s "$primary/pg_wal/$failed_segment" "$lab/archive/$failed_segment"
echo conflicting_repeat_rejected
printf 'lab_directory %s\nfailed_segment %s\n' "$lab" "$failed_segment"

The script prints base_backup_verified, then temporarily replaces the archive command with a real failing executable. It waits for an observed archiver failure, checks that the failed file is absent from the archive, requests a checkpoint, and confirms the original WAL remains. Restoring the command produces archive_repaired only after archival progress and the expected destination file are observed; byte comparison checks the copy itself.

The final controls print identical_repeat_accepted and conflicting_repeat_rejected. The latter supplies deliberately different bytes under the existing name and verifies the genuine archive remains unchanged. The helper stages a copy, synchronizes it, and links the final name without overwriting another file. It returns success for an existing name only after comparing and synchronizing that content. These checks exercise the helper’s behavior, rather than assuming a successful shell invocation proves integrity.

All experimental servers stop on exit, while the printed directory retains logs and test files for inspection. A failed wait or comparison stops the drill. The deliberate archiver failure is an expected observation, not a reason to suppress errors from unrelated steps.

Preserve a recoverable interval

Monitor archive progress alongside backlog and available space. pg_stat_archiver.failed_count captures ordinary command failures such as this test; some shell failures or archiver crashes restart the process without incrementing that counter. Read server logs as well. A growing archived count alone does not prove that every file required by a particular backup and target is present.

WAL normally becomes archivable when a segment completes. On quiet databases, archive_timeout can bound waiting for segment switches, at the cost of more archive storage; streaming replication has different delivery timing. Retention must preserve the base backup, all required WAL, and relevant timeline history for the promised interval. Protect configuration and external dependencies separately. Manifest verification checks backup integrity but does not replace a restore rehearsal. Record the recovered business state and covered endpoint before treating an archive as evidence that a recovery objective is met.

Sources

PostgreSQL 18 continuous archiving, archive return codes, duplicate content, WAL retention, and recovery chains, checked 2026-09-10.
PostgreSQL 18 pg_verifybackup, manifest verification and its limits, checked 2026-09-10.
PostgreSQL 18 cumulative statistics, archiver progress and failure counters, checked 2026-09-10.

Full versus incremental backups

Sun, 27 Dec 2026 00:00:00 +0000

A full backup provides a baseline without depending on an earlier backup from the same method. An incremental backup stores changes relative to a specified reference and depends on the material needed to reconstruct the omitted content. That distinction describes recovery dependencies, not a universal promise about speed or bytes transferred. A large changed working set can make an incremental backup substantial too.

When it breaks

A damaged full backup can invalidate every incremental that depends on it. Likewise, removing a required middle increment can break later recovery even when the newest directory and its manifest remain intact. Retention must follow dependencies rather than independently deleting directories whose timestamps look old. A newer file is not automatically a replacement for all older recovery material.

The reference matters. A backup relative to the previous increment forms a chain; one relative to a full baseline has a different dependency shape, often described as differential. A synthetic full reconstructs a new baseline from existing backup material. These terms do not guarantee identical implementations across tools. Check the exact product’s manifest and restore rules before designing rotation around the names.

Physical database backups also need the WAL required to make their captured files consistent. A full backup is independent of earlier backups, not independent of all recovery logs, software, configuration, or external dependencies. The experiment includes the required WAL through streaming and keeps the database version and checksum configuration consistent across the chain.

Check it quickly

Run the following as a non-root Linux account with PostgreSQL 18 server and client binaries, including pg_combinebackup, in PATH. It was executed on PostgreSQL 18.6. The script creates private temporary primary and recovery clusters with no TCP listeners. WAL summarization is enabled before taking backups so PostgreSQL can identify changed blocks for the incremental requests.

#!/bin/sh
set -eu
umask 077
lab=$(mktemp -d /tmp/xbcatm-incremental.XXXXXX)
mkdir "$lab/socket"
primary=$lab/primary
restored=$lab/restored
cleanup() {
 for cluster in "$restored" "$primary"; do
 if test -f "$cluster/postmaster.pid"; then
 pg_ctl -D "$cluster" -m fast -w stop >/dev/null
 fi
 done
}
trap cleanup EXIT
initdb -D "$primary" --no-locale --encoding=UTF8 --auth-local=trust --auth-host=reject >/dev/null
cat >> "$primary/postgresql.conf" <<CONF
listen_addresses = ''
unix_socket_directories = '$lab/socket'
port = 56432
summarize_wal = on
CONF
pg_ctl -D "$primary" -l "$lab/primary.log" -w -t 15 start >/dev/null
export PGHOST=$lab/socket PGPORT=56432 PGDATABASE=postgres PGUSER=$(id -un)
export PGCONNECT_TIMEOUT=2
psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
CREATE TABLE marker (id integer PRIMARY KEY,value text);
INSERT INTO marker VALUES (1,'full');
SQL
pg_basebackup -D "$lab/base" -X stream -c fast -w
psql -X -v ON_ERROR_STOP=1 -qAt -c "INSERT INTO marker VALUES (2,'first_increment')"
pg_basebackup -D "$lab/inc1" --incremental="$lab/base/backup_manifest" -X stream -c fast -w
psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
UPDATE marker SET value='final' WHERE id=1;
DELETE FROM marker WHERE id=2;
INSERT INTO marker VALUES (3,'second_increment');
SQL
pg_basebackup -D "$lab/inc2" --incremental="$lab/inc1/backup_manifest" -X stream -c fast -w
for backup in "$lab/base" "$lab/inc1" "$lab/inc2"; do
 pg_verifybackup -q "$backup"
done
echo input_backups_verified
if pg_combinebackup -o "$lab/missing-middle" "$lab/base" "$lab/inc2" >"$lab/missing-middle.log" 2>&1; then
 echo 'incomplete chain unexpectedly combined' >&2
 exit 1
fi
grep -E 'starts at LSN .*but expected' "$lab/missing-middle.log"
echo missing_middle_rejected
pg_combinebackup -o "$lab/combined" "$lab/base" "$lab/inc1" "$lab/inc2"
pg_verifybackup -q "$lab/combined"
echo combined_backup_verified
mkdir "$restored"
cp -a "$lab/combined/." "$restored/"
cat >> "$restored/postgresql.conf" <<CONF
port = 56433
summarize_wal = off
CONF
pg_ctl -D "$restored" -l "$lab/restored.log" -w -t 15 start >/dev/null
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
SELECT pg_is_in_recovery();
SELECT id,value FROM marker ORDER BY id;
DO $$ BEGIN
 IF (SELECT string_agg(id||':'||value,',' ORDER BY id) FROM marker)
 IS DISTINCT FROM '1:final,3:second_increment' THEN
 RAISE EXCEPTION 'incremental restore result mismatch';
 END IF;
END $$;
SQL
printf 'lab_directory %s\n' "$lab"

The full backup contains row one with value full. The first increment, referenced to that full backup’s manifest, adds row two. The second increment references the first increment’s manifest, changes row one to final, deletes row two, and adds row three. This tests updates and deletion as well as the simple appearance of new rows.

Each input backup is verified before reconstruction. The first combine attempt intentionally supplies only the full backup and second increment. It must fail with a start-LSN mismatch, and the script checks that reason before printing missing_middle_rejected. No original backup is deleted to create the negative control. The failure demonstrates the dependency that a directory-count check would miss.

The complete combine command supplies the full backup, first increment, and second increment in order. Its synthetic full output is verified again, copied to a separate recovery directory, and started. Expected rows are 1|final and 3|second_increment; the SQL assertion checks that exact combined result. The instance has finished recovery, and the deleted row must not reappear.

pg_combinebackup validates relationships between inputs but does not replace each input’s integrity verification. The separate verification steps and real startup therefore establish different facts. The script uses ordinary copying, preserving the input backups while the recovered instance runs. All experimental servers stop on exit, and the printed directory retains logs and artifacts for inspection.

Budget the restore, not just the backup

Incrementals can reduce repeated copying of unchanged blocks, but restoration may require fetching, validating, and combining several artifacts before startup. Measure that complete path, including recovery and application checks. A faster daily backup does not establish that the incident recovery deadline is achievable.

Keep a dependency inventory alongside retention policy. PostgreSQL permits a synthetic full to become a later combine input, which can change the chain an operator needs to retrieve. Verify the new baseline before retiring its predecessors under the chosen policy. Changing checksum state or engine requirements can also require a fresh baseline. Rehearse a recent endpoint and older promised endpoints after rotation so storage savings do not silently shrink the recoverable window.

Sources

PostgreSQL 18 pg_basebackup, incremental references, WAL summaries, and required WAL collection, checked 2026-09-10.
PostgreSQL 18 pg_combinebackup, dependency order, synthetic full backups, and integrity-verification limits, checked 2026-09-12.
PostgreSQL 18 pg_verifybackup, checking backup manifests before actual restore testing, checked 2026-09-10.

Backups follow the 3-2-1 rule

Sun, 30 Aug 2026 00:00:00 +0000

The classic 3-2-1 backup rule calls for three copies of data, on two storage media, with one copy off site. The working data counts as one copy, leaving two backup copies. The rule is a starting point for separating failure exposure. It does not promise that every set of three filenames survives hardware failure, accidental deletion, credential compromise, or a mistake propagated to all copies.

When it breaks

Copies can share more than their contents. Two directories may occupy one disk; two virtual volumes may depend on one storage array; several remote backups may all be deletable by the compromised production administrator. Device, location, and access boundaries need separate examination. An off-site location reduces exposure to a local disaster, but does not automatically make its data offline or immutable.

A hard link is an especially direct counterexample to counting names. It adds another directory entry for the same underlying file. Modifying that file in place changes what both names expose. A separate copied file preserves earlier bytes against that particular mutation, but it can still share the same device and administrative failure risks.

Synchronization is another trap. A mirror that promptly propagates deletion may remove the only version needed for recovery. Retention and version selection matter as much as destination count. Checksums detect disagreement with a trusted expected value, but a checksum stored under the same compromised authority is not by itself proof against malicious replacement.

Check it quickly

Save this as copies_lab.py and run python3 copies_lab.py on a filesystem supporting hard links, with assertions enabled. It uses only temporary files and Python’s standard library. The two directory names simulate locations inside one local filesystem. This is a mutation and restore drill, not evidence that the machine actually has geographically separated backup media.

import hashlib
import json
import os
import shutil
import tempfile
from pathlib import Path

def digest(path):
 return hashlib.sha256(path.read_bytes()).hexdigest()

with tempfile.TemporaryDirectory(prefix="xbcatm-copies-") as directory:
 root = Path(directory)
 primary_site = root / "site-a"
 other_site = root / "site-b-simulation"
 primary_site.mkdir()
 other_site.mkdir()
 primary = primary_site / "account.json"
 primary.write_text('{"account":"example","balance_cents":2500}\n')
 expected = digest(primary)
 alias = primary_site / "hardlink.json"
 os.link(primary, alias)
 independent = other_site / "snapshot.json"
 shutil.copy2(primary, independent)
 assert os.path.samefile(primary, alias)
 assert not os.path.samefile(primary, independent)
 primary.write_text('{"account":"example","balance_cents":0}\n')
 assert digest(alias) != expected
 assert digest(independent) == expected
 print("hardlink_changed_with_primary")
 print("independent_copy_kept_original_bytes")
 primary_site.rename(root / "site-a-unavailable")
 assert not primary.exists()
 restored = root / "restored.json"
 shutil.copy2(independent, restored)
 assert digest(restored) == expected
 assert json.loads(restored.read_text()) == {"account": "example", "balance_cents": 2500}
 print("restored_without_original_path", 2500)
 assert independent.stat().st_dev == restored.stat().st_dev
 print("same_filesystem_lab_not_real_offsite_protection")

The fixture starts with an account containing twenty-five hundred cents. It creates a hard-linked name and a separately copied snapshot, recording the expected digest before changing the original. The identity checks establish which path is the same file. An in-place write changes both the original and hard link to zero, while the independent copy retains the expected bytes.

The first two output markers confirm that contrast. This proves why adding a second filename did not provide another recoverable version against the tested mutation. It does not claim that a filesystem snapshot behaves like a hard link; snapshot implementations have their own copy-on-write and retention semantics.

The script then renames the simulated primary-site directory so the original path is unavailable. Restoration reads only the separate snapshot, writes a new file, checks its digest, and parses the business value. The recovered amount must be twenty-five hundred cents. The final marker explicitly confirms the same-filesystem limitation. All fixture files are removed afterward; no real backup retention policy is changed.

Turn the rule into a recovery design

Inventory the working copy and both backups by storage device or service, physical location, deletion authority, retained versions, and restore dependency. Two different media historically helped avoid common device and format failures. Modern deployments still need meaningful independence; merely renaming cloud resources or folders does not establish it. Document which single failure each separation is intended to survive.

Keep an appropriate copy offline or protected against alteration where the threat model requires it. Protect confidentiality with encryption, and arrange recovery access to the required keys without depending entirely on the failed production environment. Verify that an operator can retrieve the chosen version using the credentials and tooling available during the actual incident.

Rehearse restores from each intended recovery source. Record the recovered business state, its age, and the time needed to make it usable. A recent local copy can provide fast recovery while an older remote copy covers a wider failure, but their recovery-point promises differ. The useful outcome of 3-2-1 is a set of independently usable recovery paths with known limits, supported by exercises instead of a filename count.

Sources

Backblaze 3-2-1 backup strategy, the classic copy/media/location rule and modern device interpretation, checked 2026-09-12.
CISA StopRansomware Guide, offline encrypted backups, recovery dependencies, and regular restoration tests, checked 2026-09-12.

Backup · XB Field Notes

pg_dump captures a consistent snapshot

Know what the snapshot covers

Account for load and dependencies

When it breaks

Check it quickly

Sources

Postgres WAL archiving reaches point-in-time recovery

When it breaks

Check it quickly

Verify before accepting recovery

Further reading

Sources

Backups are untested hope without proof

When it breaks

Check it quickly

Record a recovery result people can use

Sources

WAL archiving preserves a recoverable history

When it breaks

Check it quickly

Preserve a recoverable interval

Further reading

Sources

Full versus incremental backups

When it breaks

Check it quickly

Budget the restore, not just the backup

Further reading

Sources

Backups follow the 3-2-1 rule

When it breaks

Check it quickly

Turn the rule into a recovery design

Further reading

Sources