Memory · XB Field Notes

Postgres uses two cache layers

Thu, 27 Jan 2028 00:00:00 +0000

For ordinary buffered relation I/O, PostgreSQL’s shared buffers sit above the operating system’s file cache. A shared-buffer hit avoids another read through that lower layer; a shared-buffer miss can still be served from memory by the operating system. The same file data can occupy both caches, so this is not a promise that PostgreSQL and the kernel share one physical copy.

When it breaks

An EXPLAIN buffer count labelled shared read means the block was brought into PostgreSQL’s buffer pool. It does not establish that a storage device performed a physical read. Treating every shared read as a disk miss can make a warm operating-system cache look cold and lead to the wrong memory-sizing decision. Likewise, a faster second query does not identify which cache changed by itself.

Cache residency is temporary. Other queries can evict pages, and a scan larger than the available cache cannot keep every page warm indefinitely. A relation may use a scan strategy that changes buffer reuse. Test with a known working set, inspect the access path, and keep timing claims separate from the cache-layer facts actually observed.

Check it quickly

Use a disposable PostgreSQL 18 instance as a superuser, with the supplied pg_buffercache and pg_prewarm extensions available but not already installed. The tested instance used 128 MB of shared buffers. The relation below is small enough to fit comfortably. Run blocks in one fresh psql -X -v ON_ERROR_STOP=1 session. The eviction function is for developer testing; restrict it to this isolated fixture.

CREATE SCHEMA cache_layer_demo;
CREATE EXTENSION pg_buffercache WITH SCHEMA cache_layer_demo;
CREATE EXTENSION pg_prewarm WITH SCHEMA cache_layer_demo;
CREATE TABLE cache_layer_demo.items (id integer PRIMARY KEY,payload text)
 WITH (autovacuum_enabled=false);
INSERT INTO cache_layer_demo.items
SELECT g,repeat(md5(g::text),10) FROM generate_series(1,12000) AS g;
VACUUM (ANALYZE) cache_layer_demo.items;
CREATE VIEW cache_layer_demo.residency AS
SELECT count(*) AS heap_buffers FROM cache_layer_demo.pg_buffercache
WHERE reldatabase=(SELECT oid FROM pg_database WHERE datname=current_database())
 AND relfilenode=pg_relation_filenode('cache_layer_demo.items')
 AND relforknumber=0;
SET max_parallel_workers_per_gather=0;
SELECT pg_relation_size('cache_layer_demo.items')/current_setting('block_size')::int AS heap_blocks;
SELECT heap_buffers>0 AS initially_buffered FROM cache_layer_demo.residency;

The heap contains 546 blocks in the tested build, and the initial residency comparison is true. The view filters by database, current file identifier, and main fork, so it does not count another relation’s buffers or the fixture’s auxiliary forks. Next, evict only this relation’s shared buffers:

SELECT * FROM cache_layer_demo.pg_buffercache_evict_relation('cache_layer_demo.items'::regclass);
SELECT heap_buffers=0 AS no_shared_heap_buffers FROM cache_layer_demo.residency;

The function reports evicted, flushed, and skipped buffers. This run returned 550, 550, and zero; the total includes auxiliary forks, explaining why it exceeds the heap-only count. The following comparison confirms no heap buffers remain. It says nothing about whether the operating system still caches those file pages.

Read the heap through the operating system without loading PostgreSQL shared buffers:

SELECT cache_layer_demo.pg_prewarm('cache_layer_demo.items','read','main');
SELECT heap_buffers=0 AS still_no_shared_heap_buffers FROM cache_layer_demo.residency;

The prewarm call processes 546 blocks, and the shared-buffer count remains zero. Its read mode performs synchronous reads; buffer mode would instead load the database buffer pool. Now execute a query that needs every payload:

EXPLAIN (ANALYZE,BUFFERS,COSTS OFF,TIMING OFF,SUMMARY OFF,FORMAT JSON)
SELECT sum(length(payload)) FROM cache_layer_demo.items;

The sequential scan reports 546 shared reads and zero shared hits. Those reads are compatible with an already warm operating-system cache; they are not evidence of 546 device operations. Repeat the same query:

EXPLAIN (ANALYZE,BUFFERS,COSTS OFF,TIMING OFF,SUMMARY OFF,FORMAT JSON)
SELECT sum(length(payload)) FROM cache_layer_demo.items;

The second scan reports zero shared reads and 546 shared hits. Both scans process twelve thousand rows, so the change in buffer accounting is not a reduction in the logical input. Finally, confirm residency and the aggregate result, then remove the fixture and its extensions:

SELECT heap_buffers>0 AS now_shared FROM cache_layer_demo.residency;
SELECT sum(length(payload)) FROM cache_layer_demo.items;
RESET max_parallel_workers_per_gather;
DROP SCHEMA cache_layer_demo CASCADE;

The result is 3,840,000 payload characters and a true shared-residency comparison. Exact block counts depend on row layout and build parameters. The meaningful observations are the empty shared pool after read-mode prewarming, the subsequent shared reads, and the repeated scan’s hits. They do not measure physical-device latency or guarantee a particular speedup.

Budget memory for the complete server

The PostgreSQL resource guide suggests roughly a quarter of system memory as an initial shared-buffer setting for a dedicated server with sufficient RAM. That is a starting point, not a universal optimum. Connection memory, concurrent sorts and hashes, maintenance, and the operating-system cache all consume the same host budget.

Measure representative working sets and concurrent workloads before changing the allocation. Warm-cache microbenchmarks can hide I/O pressure, while cold-cache tests can exaggerate a workload that normally reuses pages. Keep cache-layer counters, actual device observations, and query results together so a configuration change is credited for the effect it actually produced.

Sources

PostgreSQL 18 resource consumption, shared buffers and operating-system cache budgeting, checked 2026-09-10.
PostgreSQL 18 pg_prewarm, read versus buffer prewarming modes, checked 2026-09-10.
PostgreSQL 18 pg_buffercache, buffer identity and development-only relation eviction, checked 2026-09-10.

Redis expiry is a deadline, eviction is a capacity policy

Sun, 26 Dec 2027 00:00:00 +0000

Expiration and eviction can both turn a successful lookup into a miss, but they answer different questions. Expiration follows a key’s deadline. Eviction removes eligible keys to respond to memory pressure. Extending a TTL cannot prevent capacity-driven eviction, and increasing memory cannot correct a deadline that the application set incorrectly.

Time and capacity are separate axes

A volatile key has an expiration attached to it. Redis checks expired keys during access and through active expiration work. This does not promise that the key survives until its deadline: explicit deletion, replacement, and an eligible eviction policy can change the outcome earlier. A cache client must handle misses regardless of how far away the original expiration appeared.

Maxmemory selects an accounting limit used by Redis’s memory-management decisions. It is not an operating-system RSS ceiling. Replication and append-only buffers can include memory excluded from eviction accounting; process overhead, allocator behavior, and persistence work also need capacity headroom. A large command can temporarily exceed the configured limit. Size the process environment as well as the cache budget.

Select candidates before debating ranking

Allkeys policies consider keys regardless of whether they have TTLs. Volatile policies restrict candidates to keys with expiration. Within those sets, LRU favors keys that have gone longer without access, LFU uses frequency information, and random policies select randomly. Redis approximates LRU and LFU rather than promising one globally exact victim order. Volatile-ttl favors shorter remaining lifetimes, not keys that have necessarily already expired.

Noeviction declines memory-growing commands when the limit cannot be accommodated instead of discarding keys for capacity. Reads can still succeed. It is not a durability guarantee: ordinary expiration, explicit deletion, process failure, and persistence configuration remain separate. Inspect the actual configured policy instead of assuming a universal deployment or cluster default.

A volatile policy without expiring candidates behaves like noeviction for this purpose. Merely choosing volatile-lru does not cause permanent keys to become disposable. Conversely, putting session or authoritative records beside rebuildable cache entries under an allkeys policy makes those records eligible too. A key prefix is a naming convention, not an eviction boundary.

When it breaks

A sudden miss spike can reflect expiration, eviction, invalidation, or changed key construction. Compare expired_keys and evicted_keys deltas with the request workload and command outcomes. These are cumulative counters, so compare a bounded interval and account for restarts or resets. A rising eviction counter alone does not reveal which application feature lost data.

Watch refused writes as well as reads. A warm cache can keep returning plausible values while its writers receive memory errors. Ignoring those errors turns stale data into an application consistency defect. Increasing TTL may then make existing values older without fixing the inability to write replacements. Retry policy must not turn memory pressure into an unlimited request storm.

Separating authoritative state from a disposable cache is an architectural decision, not a command to migrate data during an incident. Verify recoverability, persistence, and ownership before changing an existing instance’s policy. The exercise below changes configuration only inside processes it creates and tests three candidate policies without changing any production keyspace.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. Build core Redis with libc and install the pinned client locally. The source and package downloads need the network; no system installation or optional bundled-module build is performed.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Execute the Python block with "$redis_lab/venv/bin/python". Each policy gets its own temporary Unix socket, empty dataset, and process with TCP and persistence disabled. The memory limit is the measured initial allocation plus two mebibytes. Bounded writes use 64-KiB values; TTLs are one hour where enabled. Every process and its runtime directory are cleaned up afterward.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance(policy="noeviction"):
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", policy,
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=True,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


for policy, expiring in [("allkeys-lru", True), ("noeviction", True),
 ("volatile-lru", False)]:
 with instance(policy) as client:
 baseline = client.info("memory")["used_memory"]
 client.config_set("maxmemory", baseline + 2 * 1024 * 1024)
 payload = "x" * 65536
 written, rejected = 0, False
 for index in range(128):
 try:
 client.set(f"item:{index}", payload, ex=3600 if expiring else None)
 except redis.exceptions.OutOfMemoryError:
 rejected = True
 break
 written += 1
 stats = client.info("stats")
 remaining = client.dbsize()
 assert stats["expired_keys"] == 0
 if policy == "allkeys-lru":
 assert written == 128 and not rejected
 assert stats["evicted_keys"] > 0 and remaining < written
 assert client.ttl("item:127") > 0
 else:
 assert rejected and 0 < written < 128
 assert stats["evicted_keys"] == 0 and remaining == written
 assert client.get("item:0") == payload
 if not expiring:
 assert client.ttl("item:0") == -1
 print(policy, {"written": written, "remaining": remaining,
 "evicted": stats["evicted_keys"],
 "expired": stats["expired_keys"], "rejected": rejected})

In this run, allkeys-lru accepted 128 writes, retained 23 keys, and reported 105 evictions with zero expirations. The newest surviving key still had a positive TTL. This establishes capacity-driven loss before expiration without claiming an exact LRU victim sequence.

Noeviction accepted 23 writes before rejecting the next allocation; existing reads still worked and no keys were evicted. Volatile-lru with no expiring keys produced the same refusal behavior. Exact counts depend on allocation overhead and the build, so assertions check relationships rather than those particular counts. This is not a throughput benchmark or evidence about replication, failover, persistence peaks, or an operating-system memory limit.

Sources

Redis key eviction, maxmemory accounting, candidate policies, and approximate ranking, checked 2026-09-08.
Redis EXPIRE, per-key deadlines and expiration behavior, checked 2026-09-08.
Redis INFO, memory accounting and eviction/expiration counters, checked 2026-09-08.
Redis 8.10.1 build instructions, isolated core build and allocator selection, checked 2026-09-08.

Huge pages cut TLB misses at the cost of flexibility

Tue, 21 Dec 2027 00:00:00 +0000

Databases and high-throughput servers often win, but they pay by needing contiguous memory and a pool sized in advance, so the trade is architectural, not free.

Why 4 KB hurts

On most architectures a standard page is 4 KiB, and the TLB can hold only a limited number of translations. A database or high-throughput server that touches gigabytes typically walks a working set of millions of pages, so it keeps missing the TLB, the hardware then has to walk the page table to find each mapping. The cost is not the page itself but the constant page-table walks, and at million-page working sets the misses outrun how fast the TLB can be refilled. Bigger pages mean each entry covers more memory, so the same working set needs far fewer entries, which is exactly the relief a memory-heavy workload feels.

hugetlb vs transparent

There are two ways to get bigger pages. hugetlb gives you a pool of explicitly sized pages, 2 MiB or more, that you allocate from a reserved area and manage yourself, which is deterministic and predictable; transparent huge pages (THP) try to promote mappings automatically in the background, which is convenient but can add latency spikes, fragmentation, and copy-on-write surprises on some workloads. For a database or a latency-sensitive service, hugetlb lets you own the pool and the trade, whereas THP leaves those decisions to a general policy you do not fully control. The choice is about who owns the sizing decision, you or a generic heuristics.

Budget the pool

The win of a large page depends on it being fully used and resident in memory. A 2 MiB page that is mostly unused still occupies a whole huge entry, so the memory it locks is effectively wasted, which makes pool sizing the whole skill. Reserve a pool only for the memory the application genuinely and predictably uses, rather than sizing every large allocation by feel, and account for the extra memory that a sparse page set costs. When the pages are fully resident and the working set trims into a small number of entries, huge pages deliver their whole benefit; when they are drafted for code that rarely fills them, they just burn RAM.

When it breaks

An under-sized pool fails loudly at startup. A database configured for more shared buffers than the reserved pages allow exits with Cannot allocate memory, even though free RAM looks plentiful, because hugetlb mappings cannot fall back to 4 KiB pages. Size the pool from the documented requirement plus margin, then confirm HugePages_Total and HugePages_Rsvd in /proc/meminfo after a restart.

A pool that worked at boot can fail later through fragmentation. Huge pages need contiguous 2 MiB ranges, so a long-running machine with free memory may still refuse a new mapping when no contiguous range remains. The symptom is a runtime error with HugePages_Free above zero plus a kernel message about a failed huge-page allocation; early reservation or boot parameters fix it, not retrying the same mmap.

Placement and automation cause quieter damage. Pages reserved on the wrong NUMA node force remote access on every hit, trading TLB wins for latency, so check per-node counts before pinning a database. Transparent huge pages add another risk: background compaction and copy-on-write splitting can turn a steady p99 into spikes, visible as thp_collapse_alloc stalls and khugepaged CPU. When latency matters, set THP to madvise and opt in only where measured.

Check it quickly

Save the following as hugepage_check.py and run python3 hugepage_check.py. It only reads /proc/meminfo and os.sysconf, allocates nothing, and needs no privileges, so it is safe in an unprivileged container. On a host without huge-page support or without /proc, it prints SKIP and shows the page-table arithmetic instead of raising.

#!/usr/bin/env python3
"""Report hugepage coverage and estimated TLB entries saved (read-only)."""
import os
import re
from pathlib import Path

WORKING_SET_BYTES = 8 * 1024**3 # 8 GiB example working set

def parse_meminfo(text):
 vals = {}
 for key in ("HugePages_Total", "HugePages_Free", "HugePages_Rsvd",
 "HugePages_Surp", "Hugepagesize"):
 m = re.search(r"^" + key + r":\s+(\d+)", text, re.M)
 if m:
 vals[key] = int(m.group(1))
 return vals

def main():
 vals = {}
 try:
 text = Path("/proc/meminfo").read_text(encoding="utf-8")
 except OSError as exc:
 print(f"SKIP: /proc/meminfo unavailable ({exc}); showing page-table math only")
 text = ""
 if text:
 try:
 vals = parse_meminfo(text)
 if "HugePages_Total" not in vals or "Hugepagesize" not in vals:
 print("SKIP: no HugePages fields in /proc/meminfo (hugepages unsupported or disabled)")
 else:
 size_kb = vals["Hugepagesize"]
 total = vals["HugePages_Total"]
 free = vals.get("HugePages_Free", 0)
 rsvd = vals.get("HugePages_Rsvd", 0)
 surp = vals.get("HugePages_Surp", 0)
 cov_mb = total * size_kb / 1024
 print(f"Hugepagesize: {size_kb} kB")
 print(f"HugePages total/free/rsvd/surp: {total}/{free}/{rsvd}/{surp}")
 print(f"Reserved coverage: {cov_mb:.1f} MiB ({cov_mb/1024:.2f} GiB)")
 except Exception as exc:
 print(f"SKIP: failed to parse /proc/meminfo ({exc})")
 try:
 base = os.sysconf("SC_PAGE_SIZE")
 except (AttributeError, ValueError, OSError):
 base = 4096
 huge = vals.get("Hugepagesize", 2048) * 1024
 base_entries = WORKING_SET_BYTES // base
 huge_entries = WORKING_SET_BYTES // huge
 print(f"Base page size (sysconf): {base} bytes")
 print(f"8 GiB needs ~{base_entries} base-page entries vs ~{huge_entries} huge-page entries")
 print(f"Estimated TLB entries saved: ~{base_entries - huge_entries}")

if __name__ == "__main__":
 main()

On Linux it names the huge-page size, the total, free, reserved, and surplus counters, and the reserved coverage in MiB and GiB, plus the estimated entries for an 8 GiB working set with base versus huge pages. On macOS or a minimal container you see SKIP plus the entry math with the local SC_PAGE_SIZE. It passed on macOS without /proc; it measures reservation and potential savings, not TLB misses, fragmentation, or NUMA distance, so confirm those with dmesg, per-node counts, and latency percentiles before resizing a pool.

Sources

Kernel hugepages documentation, defines hugetlb settings and behavior, checked 2026-06-01.
Kernel transparent hugepages documentation, defines THP modes and compaction behavior, checked 2026-06-01.

Linux overcommit is a policy the kernel bet on

Sat, 18 Dec 2027 00:00:00 +0000

Tuning vm.overcommit_memory shifts the failure point from a blind heuristic back to your own budget, which changes exactly who eats an OOM kill.

Why malloc rarely fails

vm.overcommit_memory defaults to 0, the heuristic mode that says yes to almost every mmap and malloc while it can plausibly honor them, and only refuses when the system is genuinely short on memory. Committing virtual memory costs almost nothing up front because pages are handed out lazily, so most allocations succeed and most normal applications stay alive at near-zero cost. The price of the generosity is that the kernel can over-commit far beyond RAM, betting that you will not touch every page; when many bytes actually get written, someone has to pay with an OOM kill.

The knob

vm.overcommit_memory lets you choose who holds the budget. The default 0 is heuristic overcommit: the kernel refuses only obviously impossible requests and otherwise says yes, trusting most processes never touch every byte they reserve. Setting it to 1 means always overcommit: almost any allocation is granted, which suits huge sparse mappings that stay mostly untouched, but every failure is pushed downstream to reclaim or the OOM killer. Setting it to 2 means strict accounting: any allocation beyond CommitLimit, roughly swap plus overcommit_ratio percent of RAM (overcommit_kbytes takes over when nonzero), fails with ENOMEM at request time. Neither extreme is free: always-overcommit hides pressure until the machine stalls, and strict mode breaks programs that reserve generously but touch little.

Heuristic versus strict

The real choice is who pays when the reckoning arrives. In heuristic mode the kernel keeps over-committing and, under pressure, picks a victim to kill; in strict mode the boundary is drawn in advance, so the failure surfaces at malloc time and the operator, not the scheduler, decides which process eats an OOM kill. Containers and VMs typically set a controlled value because a host over-committing on behalf of many guests can collapse when their aggregate demand spikes. A laptop can be generous; a production box that must not silently lose a worker should be explicit about its budget.

When it breaks

Under the default heuristic mode, the most confusing failure is a large malloc or mmap that returns success and then dies later. No pages were touched yet, so the allocation looked fine; once the process writes into them and memory runs short, the OOM killer picks a victim, often not the over-allocator but the largest nearby process. The log shows Out of memory: Killed process with no failed allocation in sight, so teams chase the wrong binary while the real over-reservation stays hidden.

Mode 1, always overcommit, fails slower. Absurd reservations are granted without protest, pressure builds silently, and a machine with swap slides into a swap storm where everything is alive but nothing progresses. Latency collapses before any kill frees memory, sessions stall, and the eventual OOM kill looks random because too many processes hold promises the system could never keep together.

Mode 2, strict accounting, fails at request time instead. A JVM reserving a large heap up front, or a fork-heavy service briefly needing twice its resident set accounted, gets ENOMEM at startup even though it would never touch every page. Operators who copy a strict overcommit_ratio from a swapless laptop onto a database host watch healthy programs refuse to start; size the ratio against the gap between CommitLimit and Committed_AS in /proc/meminfo instead of flipping the mode blindly.

Check it quickly

Save the following as overcommit_probe.py and run python3 overcommit_probe.py. It uses only the standard library, opens the overcommit_* knobs and /proc/meminfo read-only, changes nothing, and needs no privileges. Without those proc files it prints SKIP instead of failing.

"""Read-only overcommit audit: report mode, budget, and pressure."""

from pathlib import Path


def read_value(path):
 try:
 return Path(path).read_text(encoding="utf-8").strip()
 except FileNotFoundError:
 return None
 except OSError as exc:
 return "UNREADABLE: %s" % exc


def meminfo_kb(text):
 fields = {}
 for line in text.splitlines():
 name, _, rest = line.partition(":")
 digits = "".join(ch for ch in rest if ch.isdigit())
 if name and digits:
 fields[name.strip()] = int(digits)
 return fields


def main():
 mode = read_value("/proc/sys/vm/overcommit_memory")
 ratio = read_value("/proc/sys/vm/overcommit_ratio")
 kbytes = read_value("/proc/sys/vm/overcommit_kbytes")
 meminfo = read_value("/proc/meminfo")
 if mode is None or meminfo is None:
 print("SKIP: /proc/sys/vm/overcommit_memory or /proc/meminfo missing (non-Linux?)")
 return
 info = meminfo_kb(meminfo)
 print("overcommit_memory=%s overcommit_ratio=%s overcommit_kbytes=%s" % (mode, ratio, kbytes))
 print("MemTotal=%s kB SwapTotal=%s kB CommitLimit=%s kB Committed_AS=%s kB" % (
 info.get("MemTotal", "?"), info.get("SwapTotal", "?"),
 info.get("CommitLimit", "?"), info.get("Committed_AS", "?"),
 ))
 names = {"0": "heuristic", "1": "always overcommit", "2": "strict accounting"}
 print("mode means: %s" % names.get(mode, "unknown value"))
 limit = info.get("CommitLimit")
 committed = info.get("Committed_AS")
 if mode == "0":
 print("suggestion: heuristic default; watch Committed_AS growth, prefer mode 2 for workers that must not be OOM-killed")
 elif mode == "1":
 print("suggestion: always overcommit; expect swap storms under pressure, prefer 0 or 2 unless sparse mappings need it")
 elif mode == "2" and limit and committed:
 headroom = limit - committed
 print("commit headroom=%s kB; suggestion: enlarge ratio/kbytes if headroom is small, shrink them if headroom is huge" % headroom)
 else:
 print("suggestion: unexpected mode value; compare Committed_AS against CommitLimit before changing anything")


if __name__ == "__main__":
 main()

On a default workstation the script reports overcommit_memory=0 and names it heuristic. Under mode 2, compare the headroom: Committed_AS near CommitLimit explains startup ENOMEM, while huge headroom means the ratio is merely conservative. Under mode 1 the suggestion warns about swap storms, matching the slow-then-random failure above.

Sources

Kernel vm sysctl documentation, defines the overcommit_memory modes and defaults, checked 2026-06-01.
proc manual page, documents /proc/meminfo CommitLimit and Committed_AS accounting, checked 2026-06-01.

A page fault is not always a disk read

Sat, 13 Nov 2027 00:00:00 +0000

Treating all faults as slow conflates the cheap and the expensive, which is why the split between minor and major matters when you tune memory-heavy services.

Minor faults map a page

A minor page fault happens when the page is already resident in RAM but the process does not yet have it mapped. The kernel only has to wire up a page-table entry — pointing the virtual address at physical memory that is already there. Common cases: first touching a freshly allocated zeroed region, mapping a shared library that another process already loaded, or a page that was swapped in once and sits in memory. Minor faults are cheap: a few hundred nanoseconds, no disk, no IO. They happen constantly in normal-running processes and are not, by themselves, a performance problem.

Major faults read the disk

A major fault means the data has to come from disk (or a page-in from swap) before the mapping can complete, which is orders of magnitude more expensive — commonly interpreted as reading a few hundred microseconds to several milliseconds. Majors are the ones that dominate cold-start costs: warming a process, reading a big binary for the first time, or evicting from swap all surface as a burst of major faults. The distinction that matters is that a major fault proves the working set did not locate your reference, so it also scores as what your prefetch and/or cache should have made unnecessary.

Count them separately

Tools split the two for a reason, because telling them apart changes the diagnosis. ps, sar, and /proc/self/stat (minor/major) report them separately; a healthy server has a constant background of minors and few majors. Several things give away the real story: a workload with many majors is suffering from cache misses and slow startup, while a workload with drastic minor spikes but few majors is usually bad page-table churn, not disk pressure at all. Look at them separately, and when majors rise, the fix is about cache/pread, not the kernel.

When it breaks

A major-fault storm looks like latency but behaves like cold cache. After a deploy, a container restart, or a cold node, every worker touches its binary, libraries, and data for the first time at once. perf stat -e minor-faults,major-faults shows majors jumping from near zero to thousands per second while CPU stays oddly idle, and p99 stretches into hundreds of milliseconds. The fix is warming, not more CPU: pre-touch the working set or stagger restarts so processes do not fault the same pages at once.

Swap makes the same signal resonate. When the working set no longer fits in RAM, reclaimed pages fault back from disk on next touch. Majors rise with swap-in, load climbs, and every request pays disk time though the code did not change. Adding threads only grows the working set and deepens the thrash. Fit the hot set in RAM or move cold data off the request path.

A third failure is misreading mmap. Scanning a large file-backed mapping still faults each page in on first touch, so a dashboard full of majors does not mean random I/O is broken. Check residency with mincore before tuning readahead or blaming storage.

Check it quickly

Save the following as fault_probe.py and run python3 fault_probe.py. It uses only the Python standard library, needs no privileges, and runs on Linux or macOS. It snapshots resource.getrusage minor and major counters around two passes over a fresh anonymous mapping, timing each pass with time.perf_counter.

import mmap
import resource
import time

PAGE = 4096
PAGES = 20000 # about 80 MiB: large enough to see, small enough for CI


def counters():
 usage = resource.getrusage(resource.RUSAGE_SELF)
 return usage.ru_minflt, usage.ru_majflt


def touch(mapping):
 for offset in range(0, len(mapping), PAGE):
 mapping[offset] = 1


def main():
 mapping = mmap.mmap(-1, PAGES * PAGE)
 base_min, base_maj = counters()
 start = time.perf_counter()
 touch(mapping)
 first_time = time.perf_counter() - start
 mid_min, mid_maj = counters()

 start = time.perf_counter()
 touch(mapping) # pages are already mapped here
 second_time = time.perf_counter() - start
 end_min, end_maj = counters()
 mapping.close()

 first_min = mid_min - base_min
 first_maj = mid_maj - base_maj
 second_min = end_min - mid_min
 second_maj = end_maj - mid_maj
 print(f"first touch: minflt={first_min} majflt={first_maj} time={first_time:.4f}s")
 print(f"second touch: minflt={second_min} majflt={second_maj} time={second_time:.4f}s")
 assert first_min > 1000, f"expected thousands of minor faults, got {first_min}"
 assert first_maj == 0, f"anonymous first touch should not cause majors, got {first_maj}"
 assert second_min < first_min // 10, f"mapped pages should barely fault: {second_min} vs {first_min}"
 assert second_maj == 0, f"unexpected major faults on second pass: {second_maj}"
 print("PASS: first touch faults as minors without disk reads; second touch is almost fault-free")


if __name__ == "__main__":
 main()

The first pass touches every page once, so expect a large minflt jump of several thousand, almost no majflt change, and a slower elapsed time. The second pass revisits mapped pages, so expect near-zero new faults and a shorter time. That contrast is the point: anonymous first touch costs minor faults, not disk reads.

The assertions passed with Python 3.12 on macOS and Linux. The probe covers anonymous memory only, so it cannot locate a file-backed major, nor replace perf stat -e minor-faults,major-faults on production. Enlarge the mapping if the time gap is too small to see.

Sources

Linux man page proc(5), per-process minor and major fault counters, checked 2026-06-25.
Linux man page mmap(2), demand paging with MAP_PRIVATE and MAP_ANONYMOUS, checked 2026-09-12.
Brendan Gregg: perf examples, counting minor-faults and major-faults with perf stat, checked 2026-09-12.
Kernel documentation: concepts overview, demand paging, page cache, and reclaim, checked 2026-09-12.

Redis memory fills with fragmentation over time

Thu, 03 Jun 2027 00:00:00 +0000

A Redis process can keep a substantial resident footprint after application keys are deleted. That observation deserves explanation, but it does not by itself prove a leak or allocator fragmentation. The useful question is which bytes remain allocated, which pages remain resident, and whether that footprint threatens the actual capacity budget.

Separate the measurements

INFO memory exposes used_memory for Redis-accounted allocations and used_memory_rss for the resident process footprint observed by the operating system. Neither is simply the sum of application payload lengths. Keys, objects, metadata, clients, and other overhead consume memory too. MEMORY USAGE provides per-key allocation estimates, including overhead; it does not measure that key’s independent contribution to process RSS.

Mem_fragmentation_ratio broadly compares resident memory with Redis-accounted allocations. Code, shared libraries, stacks, and allocator retention also contribute to the difference. The ratio can rise when the denominator falls after deletion, even if RSS barely changes. Inspect absolute bytes alongside ratios: a small empty instance can have a high ratio without consuming a dangerous amount of RAM.

Used_memory_peak helps explain a footprint left after an earlier workload peak. Used_memory_dataset is another accounting view, not a promise that an empty keyspace leaves zero process allocations. Measurements are not all sampled at exactly the instant INFO runs. Do not require an exact equality between a rounded ratio and independently observed values during active writes.

Check the allocator and version

Allocator-specific metrics can help distinguish fragmentation from retained pages and non-heap overhead, but their meaning depends on the build. Redis 8.10.1’s source computes allocator fragmentation from small-bin counters with a Lua adjustment; blindly reconstructing every version’s metric as active bytes divided by allocated bytes is unsafe. Check the versioned implementation when a general documentation formula disagrees.

This exercise deliberately builds with libc. That build lacks jemalloc’s detailed accounting; Redis substitutes some allocator fields from other measurements, and the fragmentation counter remains zero. An allocator_frag_ratio of one is therefore not proof that this process has no fragmentation. Always record mem_allocator before interpreting the more specialized fields or choosing an intervention.

When it breaks

A rising RSS curve alongside a growing dataset calls for workload and capacity analysis, not an automatic fragmentation diagnosis. A falling dataset with stable RSS suggests retained footprint, but still does not identify every retained page. Likewise, one snapshot cannot rule out a leak. Compare the same workload over time, value-size distribution, peak usage, and relevant process limits.

MEMORY PURGE attempts allocator reclamation where implemented. Redis documents it as a benign no-op for allocators other than jemalloc, so a successful reply is not evidence of recovered memory. It also does not remove application keys. Active defragmentation is a different mechanism and requires supported builds; enabling it can add CPU and latency costs that need measurement.

Restarting merely to lower a ratio can introduce downtime, cold-cache load, or data loss when persistence and recovery are inadequate. Establish a measured capacity problem and a rehearsed recovery path before treating restart as remediation. Maxmemory is not an RSS ceiling, and reducing key count does not guarantee immediate page return to the operating system.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1 using libc, and redis-py 8.1.0. The commands build Redis core and install the client in a temporary virtual environment. Downloads require the network; nothing is installed system-wide and optional bundled modules are not built.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python". It creates a dedicated Redis process and private Unix socket under /tmp, disables TCP and persistence, and writes only its own 320 sample keys. Alternating values contain 16 or 64 KiB. Short pauses permit metric sampling, without assuming exact refresh timing. The process and temporary runtime directory are removed afterward.

from contextlib import contextmanager
import json
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance(policy="noeviction"):
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", policy,
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=True,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 def snapshot(stage):
 time.sleep(0.2)
 info = client.info("memory")
 fields = ["used_memory", "used_memory_dataset", "used_memory_rss",
 "used_memory_peak", "mem_fragmentation_ratio", "mem_allocator",
 "allocator_frag_ratio", "allocator_frag_bytes"]
 print(stage, json.dumps({"keys": client.dbsize(),
 **{name: info[name] for name in fields}}, sort_keys=True))
 return info

 empty = snapshot("empty")
 for index in range(320):
 client.set(f"sample:{index}", "x" * (16384 if index % 2 == 0 else 65536))
 assert client.dbsize() == 320
 assert client.memory_usage("sample:0") >= 16384
 loaded = snapshot("loaded")
 assert loaded["used_memory"] > empty["used_memory"] + 10 * 1024 * 1024
 for index in range(1, 320, 2):
 assert client.delete(f"sample:{index}") == 1
 assert client.dbsize() == 160
 half = snapshot("large values deleted")
 assert half["used_memory"] < loaded["used_memory"] - 8 * 1024 * 1024
 for index in range(0, 320, 2):
 assert client.delete(f"sample:{index}") == 1
 assert client.dbsize() == 0
 assert client.memory_usage("sample:0") is None
 cleared = snapshot("all values deleted")
 assert cleared["used_memory"] < half["used_memory"]
 assert cleared["used_memory_peak"] >= loaded["used_memory"]
 assert client.memory_purge()
 snapshot("after MEMORY PURGE")
 print("PASS: logical allocations fell; no RSS or purge-effect threshold is assumed")

One run reported about 16.7 MiB of used_memory after loading. Deleting all keys reduced that to about 1.04 MiB, while RSS remained around 20.8 MiB. The overall ratio consequently rose to roughly twenty. MEMORY USAGE confirmed the named key no longer existed; logical allocations fell, but resident memory did not track them proportionally.

The allocator-specific fragmentation ratio stayed at one throughout this libc run, and MEMORY PURGE returned success without a meaningful RSS drop. Counts and logical-allocation relationships are asserted; exact RSS values are intentionally observations, not pass thresholds. This experiment demonstrates accounting differences and the no-op limitation, not jemalloc defragmentation effectiveness, Linux memory-pressure behavior, or a leak diagnosis.

Sources

Redis INFO, process, allocation, peak, and allocator memory fields, checked 2026-09-08.
Redis MEMORY USAGE, per-key allocation estimates and overhead, checked 2026-09-08.
Redis MEMORY PURGE, jemalloc support and other allocators’ no-op behavior, checked 2026-09-08.
Redis 8.10.1 object accounting, version-specific fragmentation calculation, checked 2026-09-08.
Redis 8.10.1 memory sampling, cached measurements and substituted allocator fields, checked 2026-09-08.
Redis 8.10.1 build instructions, core build and allocator selection, checked 2026-09-08.

Swap gives memory a slow extra layer

Wed, 25 Nov 2026 00:00:00 +0000

Swap is not extra memory. It is a bet: the kernel picks pages it predicts will not be needed soon, writes them to a device, and frees the RAM. When the prediction holds, a spike passes without killing a process. When it does not, every wrong guess costs a device round trip, three to five orders of magnitude above reading RAM.

The trade only works as a valve. Sizing swap to cover a chronically under-provisioned workload buys a machine that is slow in a way that looks like a CPU problem; the first question is how much of the working set does not fit.

Reclaim picks victims by prediction

Anonymous pages, the heap, the stack and anything malloc handed out, have no file behind them, so they are the only thing swap holds. File-backed pages do not need swap: the kernel can drop a clean page-cache entry and read it back later. vm.swappiness is documented as the rough relative IO cost of swapping versus filesystem paging, on a scale from 0 to 200 with a default of 60. At 0 the kernel will not initiate swap until free plus file-backed pages fall below the zone’s high watermark, so 0 means “only under duress”, not “no swap”.

Internally the dial becomes a scan weight: swappiness for the anonymous list, 200 - swappiness for the file list. Values above 100 are not a mistake: the documentation suggests 133 when random IO against the swap device is about twice as fast as filesystem IO, and recommends beyond-100 values for in-memory swap such as zram or zswap.

The price is a device round trip

Latencies tell the story: roughly 100 ns for a resident DRAM page, single-digit microseconds for compressed swap in RAM, 100 us on NVMe, 250 us on SATA SSD, 8 ms on a spinning disk. A swap-in is a thousand times a RAM hit on NVMe and eighty thousand times on a disk, and the bill is charged per page needed again.

Model 100,000 page-sized accesses per second over an 8 GiB working set: fitting in RAM, the average is 100 ns with no major faults; at 2% over, 2.1 us, twenty-one times worse; at 20% over, 16.8 us, 167 times worse, with 16,667 major faults per second and 67 MB/s through si and so. The cliff begins at the first page that does not fit.

Thrashing is a device saturation problem

The same 20% overcommit is unremarkable on compressed swap, absorbable on NVMe, and fatal on a spinning disk. A 7,200 rpm disk serves on the order of 150 random IOPS, so 16,667 faults/s cannot be met; the shortfall is never served and the wait queue grows until the OOM killer. This is thrashing: a queueing failure, not a slow-memory failure.

zram and zswap change the cost model, not the principle. zram serves a fault from compressed RAM in microseconds instead of a device round trip, which is why the documentation suggests swappiness above 100. It expects roughly a 2:1 compression ratio, so making a device larger than twice memory buys little. The two sit in different places: zram is the swap device itself, with compressed pages staying in RAM; zswap is a compressed cache in front of a real swap device, evicting to it on an LRU basis when its pool hits its size limit. zram moves swap into memory; zswap only defers the write. Both make /tmp a natural use case, where swap and temp directories stop being separate problems: both are RAM with a cap.

When it breaks

Thrashing presents as a machine busy and idle at once. The load average climbs while CPU utilisation stays low, si and so climb, and major faults dominate. Every process progresses at a fraction of its normal rate while the box is not CPU-bound, which is why this gets misdiagnosed as an application problem. Confirm with vmstat 1: sustained si/so in the thousands of KiB/s alongside a rising run queue and idle CPU is the signature.

The second failure is swap that was never attached. swapon(8) documents that the kernel writes to the swap file directly, without the filesystem’s help, so a file with holes is rejected; truncate and cp both produce one. The same page documents -e/--ifexists and nofail, which silently skip a device that is not there, so a host can boot with no swap, look healthy until the first spike, and OOM-kill something important. Read /proc/swaps rather than trusting /etc/fstab.

Check it quickly

swapon --show
free -h
vmstat 1 5
grep -E 'pswpin|pswpout|pgmajfault' /proc/vmstat

swapon --show should list a device with a nonzero size; empty output means nothing is attached. In vmstat, si and so should sit near zero; sustained thousands of KiB/s with us plus sy low is thrashing rather than load. A climbing pgmajfault means pages are coming back from the device; a flat counter means the working set is resident.

Sources

swapon(8), swap priority, discard policies, and why a sparse swap file is rejected, checked 2026-09-16.
Kernel sysctl vm documentation, swappiness as a 0-200 relative IO cost dial with the zram guidance, checked 2026-09-16.
Kernel zram documentation, compressed RAM block devices, the expected 2:1 ratio, and the /tmp use case, checked 2026-09-16.
Kernel zswap documentation, the compressed swap cache in front of a backing device and its LRU eviction, checked 2026-09-16.

zram squeezes swap into RAM

Tue, 24 Nov 2026 00:00:00 +0000

When memory tightens, cold pages would go to a slow disk. zram keeps them in compressed RAM instead, trading a little CPU for much more headroom.

A compressed block device in memory

zram is a block device that lives in RAM and stores compressed data instead of talking to a real disk. When the kernel swaps a page into it, the page is compressed and kept in memory, so the memory gain comes from shrinking the pages rather than moving them to slow storage. Because the device is memory-backed, it answers reads and writes far faster than a disk-backed swap, but the space it provides is limited by how compressible the pages are and by how much memory you are willing to trade for the swap.

Compression is the space win

Compression is the space win, and it is the reason a small box can appear to hold more memory than it physically has. Whether a page pays off depends on its content: text, zeroed pages and structured data compress well, while already-compressed blobs such as media or archives barely shrink. The honest engineering view is that zram turns spare CPU cycles into apparent memory, so the trade is worthwhile when the workload leans compressible and the extra latency of a real disk swap is exactly what you want to avoid.

It suits gove boxes and phones

The fit shines on hosts with little memory and modest I/O, such as a small VPS, a container host, or a compact device like a phone or an embedded box. It is less obviously right for a server that already has plenty of RAM or that swaps into genuinely cold storage anyway, because a real disk bought with the same spend can hold far more. The practical check is to watch the compression ratio and swap usage, decide whether swapping beats killing the process, and pick zram when resident data stays small and compressible enough to make the CPU cost a fair trade.

When it breaks

A ratio that never arrives is the commonest failure. Photos, video, encrypted buffers, and archives barely shrink, so mm_stat shows orig_data_size just above compr_data_size while swap still fills. The pool and its metadata consume RAM yet add almost no headroom, so the OOM killer fires nearly as if zram were absent.

The wrong compressor fails through CPU, not capacity. A heavy algorithm on a small VPS or old handset makes every swap-out burn cycles, so kswapd spins, system CPU climbs with pressure, and tail latency collapses before swap fills. Prefer fast choices such as lzo-rle, lz4, or low-level zstd on interactive hosts, and change only after measuring under load.

Running zswap with zram compresses twice for no gain. zswap holds its own compressed pool, then hands already-shrunk pages to zram, which finds little left to save. CPU burns on both paths while swap looks used with small effective capacity. Use one layer: zram as swap, or zswap before disk, never both.

A hard-coded disksize goes stale after any resize. A unit that pins 2 GiB still offers 2 GiB after the instance grows to 32 GiB, wasting headroom; copied to a tiny edge box, the same file reserves half of RAM and hastens OOM. The size is fixed at mkswap time, so express it as a fraction of RAM at boot.

Check it quickly

Save the following as zram_ratio.py and run python3 zram_ratio.py. It uses only the standard library, opens /sys/block/zram*/mm_stat read-only, and needs no privileges. Without those files it prints SKIP and demonstrates the same ratio math on a small example instead.

#!/usr/bin/env python3
"""Read-only zram compression-ratio check (stdlib only)."""
import glob
from pathlib import Path


def parse_mm_stat(text):
 parts = text.split()
 if len(parts) < 2:
 raise ValueError("mm_stat needs at least orig and compr fields")
 orig = int(parts[0])
 compr = int(parts[1])
 rest = [int(x) for x in parts[2:]]
 return orig, compr, rest


def report(device, orig, compr):
 if compr <= 0:
 print("%s: orig=%d compr=%d ratio=n/a (empty)" % (device, orig, compr))
 return
 ratio = orig / compr
 saved = 100.0 * (1.0 - compr / orig) if orig > 0 else 0.0
 print("%s: orig=%d compr=%d ratio=%.2f saved=%.1f%%" % (device, orig, compr, ratio, saved))


def main():
 paths = sorted(glob.glob("/sys/block/zram*/mm_stat"))
 if not paths:
 print("SKIP: no /sys/block/zram*/mm_stat found (zram absent? non-Linux?)")
 orig, compr = 300 * 1024 * 1024, 100 * 1024 * 1024
 print("worked example with the same formula:")
 report("example/zram0", orig, compr)
 print("PASS: ratio math works; on a zram host the live lines above replace the example")
 return
 for path in paths:
 try:
 text = Path(path).read_text(encoding="utf-8")
 except OSError as exc:
 print("SKIP: unreadable %s (%s)" % (path, exc))
 continue
 try:
 orig, compr, _ = parse_mm_stat(text)
 except (ValueError, IndexError) as exc:
 print("SKIP: unparsable %s (%s)" % (path, exc))
 continue
 device = path.split("/mm_stat")[0]
 report(device, orig, compr)
 print("PASS: live mm_stat accounting shown; falling ratio plus full pool means resize or recompress")


if __name__ == "__main__":
 main()

On a zram host it prints one line per device with original bytes, compressed bytes, the compression ratio, and memory used, so falling ratios and a full pool are visible before OOM. On macOS or a container without zram* it prints SKIP plus the worked example. The script passed on macOS without /sys. It checks accounting and arithmetic only, not which algorithm or disksize is correct, so confirm those in the unit files before tuning.

Sources

kernel.org: zram block device, the authoritative zram documentation, checked 2026-11-23.
Wikipedia: zram, what zram is and how it works, checked 2026-11-23.
ArchWiki: Zram, practical setup and tuning, checked 2026-11-23.

The OOM killer makes the final memory call

Mon, 26 Oct 2026 00:00:00 +0000

When memory is exhausted, someone has to give. The kernel picks a victim by a policy you can read and influence, turning a crash into an order you can reason about.

Exhaustion forces a choice

Physical memory runs out, the allocator’s reclaim path has already been walked, and the kernel still cannot satisfy a grant — that is when the out-of-memory killer acts. It is not random; it computes which process to kill using a numeric score and terminates the least valuable candidate under pressure. The score weighs resident memory against a process’s share of the host, so a big, idle service that contributes little can be picked off while a small but essential one is spared. Understanding that the OOM killer runs on a score, not on who arrived first, is the whole basis for predicting its next move.

The score picks the victim

The score is steerable. oom_score reflects how costly the footprint is; oom_score_adj shifts a process up or down the league, and a negative value deliberately protects a critical daemon so it is not the victim. Setting a oom_score_adj of -500 on the process that must survive (your container controller, your agent) and leaving the bulk of the memory hog neutral is how you make the killer’s ’least important’ align with your actual priorities. Do this as part of the container/supervisor config, because by the time memory is exhausted it is far too late to decide the victim. Overcommit decides the grant; this score decides who dies when granted memory runs out.

Steer the policy before the crash

The reliable way to make OOM a non-event is to never let the system reach the threshold: cap the workload (ulimit, per-container limits), watch RSS and swap so you see the trend, and provision headroom you can afford. When you do hit it, treat it as a signal in the alerting, not a mystery condition. The OOM killer deserves to be boring — a last-resort breaker, explicitly configured, with a clear accounting of who it took and why, rather than the plan itself.

When it breaks

The classic surprise is that the most important process dies first. A database holding a large buffer pool carries the biggest resident set, so the score picks it while the swarm of small workers that created the pressure survives. The outage looks backwards, but the Out of memory: Killed process line shows the scoring did exactly what it was configured to do.

Misconfigured adjustments fail in the opposite direction. An oom_score_adj of -1000 on every service leaves the killer no acceptable victim, so it reaches past the protected set and takes the supervisor, the SSH session, or the monitoring agent instead. The sign is another trap: positive means more likely to be killed, so copying +500 from a priority example onto a critical daemon paints a target on it. Adjustments set on an interactive shell instead of the supervised unit vanish with the session.

Container scope adds a third confusion. A cgroup that hits its own memory.max triggers a memcg OOM inside that group while the host still shows free memory, so node dashboards report a shortage that never happened. The mirror image is an uncapped container that grows until a host-wide OOM takes down a neighbour. Finally, a slow leak mimics a capacity problem: the same process name holds more memory before each kill, and adding RAM only stretches the interval between restarts.

Check it quickly

Save the following as oom_drill.py in a scratch directory and run python3 oom_drill.py. It uses only the standard library, reads /proc/[pid]/oom_score, oom_score_adj, and comm read-only, and needs no privileges. Without those proc files it prints SKIP and ranks a small simulated table with the same sorting rule instead.

"""Read-only OOM victim ranking drill: live /proc scan with simulated fallback."""

from pathlib import Path


def read_text(path):
 try:
 return Path(path).read_text(encoding="utf-8").strip()
 except (FileNotFoundError, NotADirectoryError):
 return None
 except OSError as exc:
 return "UNREADABLE: %s" % exc


def live_rows():
 rows = []
 try:
 entries = list(Path("/proc").iterdir())
 except OSError:
 return None
 for entry in entries:
 if not entry.name.isdigit():
 continue
 score_text = read_text(entry / "oom_score")
 adj_text = read_text(entry / "oom_score_adj")
 if score_text is None or adj_text is None:
 continue
 if score_text.startswith("UNREADABLE") or adj_text.startswith("UNREADABLE"):
 continue
 try:
 score = int(score_text.split()[0])
 adj = int(adj_text.split()[0])
 except (ValueError, IndexError):
 continue
 name = read_text(entry / "comm") or "?"
 if name.startswith("UNREADABLE"):
 name = "?"
 short = name.split()[0] if name.split() else "?"
 rows.append((entry.name, short, score, adj))
 return rows


def report(title, rows):
 print(title)
 for pid, name, score, adj in sorted(rows, key=lambda row: (-row[2], row[0]))[:5]:
 print("pid=%s name=%s oom_score=%d oom_score_adj=%d" % (pid, name, score, adj))


def main():
 rows = live_rows()
 if rows is None:
 print("SKIP: /proc unavailable (non-Linux?) -- ranking simulated table instead")
 elif rows:
 report("live top candidates by oom_score:", rows)
 print("PASS: live ranking uses kernel scores; higher oom_score is killed first")
 return
 else:
 print("SKIP: no readable /proc/[pid]/oom_score entries -- ranking simulated table instead")
 simulated = [
 ("4242", "batch-worker", 860, 0),
 ("1001", "postgres", 640, -500),
 ("9001", "cache", 500, 0),
 ("7", "agent", 120, -800),
 ]
 report("simulated ranking (same sort rule):", simulated)
 print("PASS: unprotected worker outranks protected postgres and agent; adj sign matches kernel rule")


if __name__ == "__main__":
 main()

On Linux it prints the five highest live scores with names and adjustments, so you can confirm the ordering before pressure arrives. On macOS or containers with hidden PIDs it prints SKIP and then the simulated ranking, where the large unprotected worker outscores the protected controller. What is under test is the ranking logic: re-run it after changing a unit’s OOMScoreAdjust to confirm the intended victim moved.

Sources

Kernel: OOM killer concepts, when the kernel invokes the OOM killer and what it sacrifices, checked 2026-10-25.
man7: proc_pid_oom_score, how the kernel scores each process for OOM selection, checked 2026-10-25.
man7: proc_pid_oom_score_adj, the -1000 to +1000 adjustment range and its meaning, checked 2026-10-25.

Memory · XB Field Notes

Postgres uses two cache layers

When it breaks

Check it quickly

Budget memory for the complete server

Sources

Redis expiry is a deadline, eviction is a capacity policy

Time and capacity are separate axes

Select candidates before debating ranking

When it breaks

Check it quickly

Further reading

Sources

Huge pages cut TLB misses at the cost of flexibility

Why 4 KB hurts

hugetlb vs transparent

Budget the pool

When it breaks

Check it quickly

Further reading

Sources

Linux overcommit is a policy the kernel bet on

Why malloc rarely fails

The knob

Heuristic versus strict

When it breaks

Check it quickly

Further reading

Sources

A page fault is not always a disk read

Minor faults map a page

Major faults read the disk

Count them separately

When it breaks

Check it quickly

Further reading

Sources

Redis memory fills with fragmentation over time

Separate the measurements

Check the allocator and version

When it breaks

Check it quickly

Further reading

Sources

Swap gives memory a slow extra layer

Reclaim picks victims by prediction

The price is a device round trip

Thrashing is a device saturation problem

When it breaks

Check it quickly

Sources

zram squeezes swap into RAM

A compressed block device in memory

Compression is the space win

It suits gove boxes and phones

When it breaks

Check it quickly

Further reading

Sources

The OOM killer makes the final memory call

Exhaustion forces a choice

The score picks the victim

Steer the policy before the crash

When it breaks

Check it quickly

Further reading

Sources