Physical streaming replication sends PostgreSQL WAL to a standby, which replays the records to maintain another copy of the cluster. It can send records without waiting for a whole segment to finish. This is distinct from logical replication of selected table changes. A running receiver and a reachable standby are useful health signals, but neither establishes that a particular committed row is already visible there.

When it breaks

Streaming replication is asynchronous by default. The primary can acknowledge a commit before the standby has received or applied it. Losing the primary at that point can lose recently acknowledged work during failover. A connected standby can also receive WAL while replay is paused, leaving reads stale even though network delivery is making progress.

Keep the progress markers separate. On the standby, pg_last_wal_receive_lsn() reports the location received and synchronized to disk through streaming. pg_last_wal_replay_lsn() reports the applied location. Their difference can expose replay backlog. A timestamp for the last replayed transaction ages on an idle system too; elapsed wall-clock time alone is not a reliable measure of missing work.

Check it quickly

Use a non-root Linux account with PostgreSQL 18 server and client binaries in PATH; this script was executed on PostgreSQL 18.6. It creates private temporary clusters and a Unix socket directory with no TCP listener. Local trust authentication is confined to these disposable fixtures. The small WAL reserve supports this bounded exercise; the script does not configure a permanent replication slot.

#!/bin/sh
set -eu
umask 077
lab=$(mktemp -d /tmp/xbcatm-streaming.XXXXXX)
mkdir "$lab/socket"
primary=$lab/primary
standby=$lab/standby
cleanup() {
    for cluster in "$standby" "$primary"; do
        if test -f "$cluster/postmaster.pid"; then
            pg_ctl -D "$cluster" -m fast -w stop >/dev/null
        fi
    done
}
trap cleanup EXIT
initdb -D "$primary" --no-locale --encoding=UTF8 --auth-local=trust --auth-host=reject >/dev/null
cat >> "$primary/postgresql.conf" <<CONF
listen_addresses = ''
unix_socket_directories = '$lab/socket'
port = 56432
wal_level = replica
max_wal_senders = 4
wal_keep_size = '64MB'
CONF
pg_ctl -D "$primary" -l "$lab/primary.log" -w -t 15 start >/dev/null
export PGHOST=$lab/socket PGPORT=56432 PGDATABASE=postgres PGUSER=$(id -un)
export PGCONNECT_TIMEOUT=2
psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
CREATE TABLE marker (id integer PRIMARY KEY,value text);
INSERT INTO marker VALUES (1,'base');
SQL
pg_basebackup -D "$standby" -R -X stream -c fast -w
cat >> "$standby/postgresql.conf" <<CONF
port = 56433
hot_standby = on
CONF
pg_ctl -D "$standby" -l "$lab/standby.log" -w -t 15 start >/dev/null
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT pg_wal_replay_pause()' >/dev/null
attempt=0
while test "$(PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT pg_get_wal_replay_pause_state()='paused'")" != t; do
    attempt=$((attempt+1))
    test "$attempt" -lt 100 || { echo 'replay did not pause' >&2; exit 1; }
    sleep 0.1
done
psql -X -v ON_ERROR_STOP=1 -qAt -c "INSERT INTO marker VALUES (2,'later')"
target=$(psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT pg_current_wal_lsn()')
attempt=0
while test "$(PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT pg_last_wal_receive_lsn()>='$target'::pg_lsn")" != t; do
    attempt=$((attempt+1))
    test "$attempt" -lt 150 || { echo 'receiver did not catch up' >&2; exit 1; }
    sleep 0.1
done
psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT state,sync_state FROM pg_stat_replication'
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt <<SQL
SELECT pg_last_wal_receive_lsn()>='$target'::pg_lsn,
       pg_last_wal_replay_lsn()<'$target'::pg_lsn,
       (SELECT count(*) FROM marker);
DO \$\$ BEGIN
  IF (pg_last_wal_receive_lsn()>='$target'::pg_lsn
      AND pg_last_wal_replay_lsn()<'$target'::pg_lsn
      AND (SELECT string_agg(id||':'||value,',' ORDER BY id) FROM marker)='1:base')
      IS NOT TRUE THEN
    RAISE EXCEPTION 'paused replay result mismatch';
  END IF;
END \$\$;
SQL
echo stale_read_while_receiving
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c 'SELECT pg_wal_replay_resume()' >/dev/null
attempt=0
while test "$(PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt -c "SELECT pg_last_wal_replay_lsn()>='$target'::pg_lsn")" != t; do
    attempt=$((attempt+1))
    test "$attempt" -lt 150 || { echo 'replay did not catch up' >&2; exit 1; }
    sleep 0.1
done
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
SELECT id,value FROM marker ORDER BY id;
DO $$ BEGIN
  IF (SELECT string_agg(id||':'||value,',' ORDER BY id) FROM marker)
     IS DISTINCT FROM '1:base,2:later' THEN
    RAISE EXCEPTION 'resumed replay result mismatch';
  END IF;
END $$;
SQL
pg_ctl -D "$primary" -m fast -w stop >/dev/null
echo old_primary_stopped
pg_ctl -D "$standby" -w -t 15 promote >/dev/null
PGPORT=56433 psql -X -v ON_ERROR_STOP=1 -qAt <<'SQL'
SELECT pg_is_in_recovery();
INSERT INTO marker VALUES (3,'promoted');
SELECT id,value FROM marker ORDER BY id;
DO $$ BEGIN
  IF pg_is_in_recovery() OR (SELECT count(*) FROM marker)<>3 THEN
    RAISE EXCEPTION 'promotion result mismatch';
  END IF;
END $$;
SQL
printf 'lab_directory %s\n' "$lab"

The script takes a physical base backup with -R, which writes standby connection settings and creates standby.signal. It starts that copy on a different socket port, requests a replay pause, and waits until the actual state is paused. Then the primary commits row two. A captured post-commit LSN provides a concrete target against which to compare receipt and replay.

Expected primary status is streaming|async. On the standby, t|t|1 means receipt has reached the target, replay has not, and the table still contains only the original row. The assertion also checks that row’s value. stale_read_while_receiving therefore records an actual stale query result while log delivery continues, not a simulated delay in an application cache.

After replay resumes, the script waits for the replay LSN to reach the target and verifies 1|base plus 2|later. Each observation uses a new connection, so a previously opened transaction snapshot cannot explain the earlier stale result. Existing long-lived snapshots in applications may still need to end before those applications see freshly replayed changes.

For the takeover step, the script stops the old primary first, then promotes the standby and waits for completion. pg_is_in_recovery() becomes false, and inserting row three produces 3|promoted. Assertions verify that the new primary accepts the write. All experimental servers stop on exit, while the printed directory retains their logs. This tests an orderly, isolated promotion; it does not test automatic failure detection, network partition handling, or client endpoint migration.

Choose the acknowledgment and retention contract

Synchronous replication needs both a configured selection of synchronous standbys and the relevant synchronous_commit setting. Setting that transaction option to on without selected synchronous standbys does not invent a remote acknowledgment requirement. With synchronous standbys selected, on waits for their durable WAL receipt; remote_write waits for writing into their operating systems; remote_apply waits for replay. These choices change latency and availability as well as durability or read visibility.

Retention is another separate obligation. A replication slot can retain WAL needed by a disconnected receiver, but a stalled consumer can exhaust storage unless limits and monitoring are deliberate. Without sufficient retained or archived WAL, an old standby can require rebuilding. Monitor generation, receive and replay progress, retained bytes, and free space together.

Promotion does not fence a competing primary by itself. A real failover procedure must prevent the old writer from continuing, select an acceptable recovery position, redirect clients, and rejoin the former primary through a supported rebuild or rewind process. A healthy streaming connection proves only one part of that procedure.

Sources