When memory is exhausted, someone has to give. The kernel picks a victim by a policy you can read and influence, turning a crash into an order you can reason about.

Exhaustion forces a choice

Physical memory runs out, the allocator’s reclaim path has already been walked, and the kernel still cannot satisfy a grant — that is when the out-of-memory killer acts. It is not random; it computes which process to kill using a numeric score and terminates the least valuable candidate under pressure. The score weighs resident memory against a process’s share of the host, so a big, idle service that contributes little can be picked off while a small but essential one is spared. Understanding that the OOM killer runs on a score, not on who arrived first, is the whole basis for predicting its next move.

The score picks the victim

The score is steerable. oom_score reflects how costly the footprint is; oom_score_adj shifts a process up or down the league, and a negative value deliberately protects a critical daemon so it is not the victim. Setting a oom_score_adj of -500 on the process that must survive (your container controller, your agent) and leaving the bulk of the memory hog neutral is how you make the killer’s ’least important’ align with your actual priorities. Do this as part of the container/supervisor config, because by the time memory is exhausted it is far too late to decide the victim. Overcommit decides the grant; this score decides who dies when granted memory runs out.

Steer the policy before the crash

The reliable way to make OOM a non-event is to never let the system reach the threshold: cap the workload (ulimit, per-container limits), watch RSS and swap so you see the trend, and provision headroom you can afford. When you do hit it, treat it as a signal in the alerting, not a mystery condition. The OOM killer deserves to be boring — a last-resort breaker, explicitly configured, with a clear accounting of who it took and why, rather than the plan itself.

When it breaks

The classic surprise is that the most important process dies first. A database holding a large buffer pool carries the biggest resident set, so the score picks it while the swarm of small workers that created the pressure survives. The outage looks backwards, but the Out of memory: Killed process line shows the scoring did exactly what it was configured to do.

Misconfigured adjustments fail in the opposite direction. An oom_score_adj of -1000 on every service leaves the killer no acceptable victim, so it reaches past the protected set and takes the supervisor, the SSH session, or the monitoring agent instead. The sign is another trap: positive means more likely to be killed, so copying +500 from a priority example onto a critical daemon paints a target on it. Adjustments set on an interactive shell instead of the supervised unit vanish with the session.

Container scope adds a third confusion. A cgroup that hits its own memory.max triggers a memcg OOM inside that group while the host still shows free memory, so node dashboards report a shortage that never happened. The mirror image is an uncapped container that grows until a host-wide OOM takes down a neighbour. Finally, a slow leak mimics a capacity problem: the same process name holds more memory before each kill, and adding RAM only stretches the interval between restarts.

Check it quickly

Save the following as oom_drill.py in a scratch directory and run python3 oom_drill.py. It uses only the standard library, reads /proc/[pid]/oom_score, oom_score_adj, and comm read-only, and needs no privileges. Without those proc files it prints SKIP and ranks a small simulated table with the same sorting rule instead.

"""Read-only OOM victim ranking drill: live /proc scan with simulated fallback."""

from pathlib import Path


def read_text(path):
    try:
        return Path(path).read_text(encoding="utf-8").strip()
    except (FileNotFoundError, NotADirectoryError):
        return None
    except OSError as exc:
        return "UNREADABLE: %s" % exc


def live_rows():
    rows = []
    try:
        entries = list(Path("/proc").iterdir())
    except OSError:
        return None
    for entry in entries:
        if not entry.name.isdigit():
            continue
        score_text = read_text(entry / "oom_score")
        adj_text = read_text(entry / "oom_score_adj")
        if score_text is None or adj_text is None:
            continue
        if score_text.startswith("UNREADABLE") or adj_text.startswith("UNREADABLE"):
            continue
        try:
            score = int(score_text.split()[0])
            adj = int(adj_text.split()[0])
        except (ValueError, IndexError):
            continue
        name = read_text(entry / "comm") or "?"
        if name.startswith("UNREADABLE"):
            name = "?"
        short = name.split()[0] if name.split() else "?"
        rows.append((entry.name, short, score, adj))
    return rows


def report(title, rows):
    print(title)
    for pid, name, score, adj in sorted(rows, key=lambda row: (-row[2], row[0]))[:5]:
        print("pid=%s name=%s oom_score=%d oom_score_adj=%d" % (pid, name, score, adj))


def main():
    rows = live_rows()
    if rows is None:
        print("SKIP: /proc unavailable (non-Linux?) -- ranking simulated table instead")
    elif rows:
        report("live top candidates by oom_score:", rows)
        print("PASS: live ranking uses kernel scores; higher oom_score is killed first")
        return
    else:
        print("SKIP: no readable /proc/[pid]/oom_score entries -- ranking simulated table instead")
    simulated = [
        ("4242", "batch-worker", 860, 0),
        ("1001", "postgres", 640, -500),
        ("9001", "cache", 500, 0),
        ("7", "agent", 120, -800),
    ]
    report("simulated ranking (same sort rule):", simulated)
    print("PASS: unprotected worker outranks protected postgres and agent; adj sign matches kernel rule")


if __name__ == "__main__":
    main()

On Linux it prints the five highest live scores with names and adjustments, so you can confirm the ordering before pressure arrives. On macOS or containers with hidden PIDs it prints SKIP and then the simulated ranking, where the large unprotected worker outscores the protected controller. What is under test is the ranking logic: re-run it after changing a unit’s OOMScoreAdjust to confirm the intended victim moved.

Further reading

Sources