The systemd journal stores messages with structured metadata in journal files. That makes service output searchable without first inventing a parser for every application’s text format. Keeping those messages for a useful period is a separate requirement. Storage location, size policy, rotation, and forwarding determine which evidence remains available when an incident is investigated later.

When it breaks

The default namespace normally uses Storage=auto: persistent storage is selected when /var/log/journal exists, otherwise runtime storage is used. Storage=volatile writes beneath /run/log/journal; this is not simply the kernel’s text ring buffer. Those runtime files normally disappear across a machine reboot, but restarting the journald service is a different event and need not erase them.

Storage=persistent prefers /var/log/journal, with runtime storage used during early boot or when persistent storage cannot be written. A flush operation transfers runtime data under the documented conditions. Inspect actual files and effective configuration instead of inferring location solely from the presence of an option in one configuration fragment.

journald has default size limits and rotation behavior. It does not grow without bounds merely because no administrator wrote a rotation job. However, SystemMaxUse is not an exact hard quota that guarantees the filesystem can never fill. Runtime storage has separate settings, free-space constraints also apply, other programs consume space, and only archived journal files can be deleted during vacuuming.

Check it quickly

Run this Python script as an administrator in a disposable Linux system managed by systemd. The recorded environment was Debian 12 with systemd 252.39. It creates a unique journal namespace, gives it an explicit persistent-storage configuration, emits two harmless messages through transient services, and removes only that namespace’s data afterward. The vacuum operations deliberately delete the experiment’s old message:

from pathlib import Path
import json,shutil,subprocess,uuid

name='xbcatm-journal-'+uuid.uuid4().hex[:8]
config=Path('/etc/systemd')/('journald@'+name+'.conf')
directory=Path('/var/log/journal')/(Path('/etc/machine-id').read_text().strip()+'.'+name)
assert not config.exists() and not directory.exists()
def run(*args):
    result=subprocess.run(args,text=True,capture_output=True,timeout=15)
    assert result.returncode==0,(args,result.stderr)
    return result.stdout
def journal(*args):return run('journalctl','--namespace='+name,*args)
def emit(message):
    run('systemd-run','--quiet','--wait','--collect','--unit='+name+'-'+message,
        '--property=LogNamespace='+name,'--property=StandardOutput=journal',
        '--property=SyslogIdentifier='+name,'/usr/bin/printf',message+'\n')
    journal('--sync')
def messages():
    rows=journal('--no-pager','-o','json','SYSLOG_IDENTIFIER='+name)
    return [json.loads(line)['MESSAGE'] for line in rows.splitlines()]
config.write_text('[Journal]\nStorage=persistent\nSystemMaxUse=16M\nRuntimeMaxUse=8M\n')
try:
    emit('old');journal('--flush');journal('--sync')
    assert messages()==['old'] and list(directory.glob('*.journal'))
    run('systemctl','restart','systemd-journald@'+name+'.service')
    assert messages()==['old']
    print('persistent namespace: message survives journald service restart')
    journal('--vacuum-size=1')
    assert messages()==['old']
    print('vacuum cannot remove the active journal file to satisfy a one-byte target')
    journal('--rotate');emit('new')
    assert messages()==['old','new']
    assert list(directory.glob('*@*.journal'))
    journal('--vacuum-size=1')
    assert messages()==['new'] and not list(directory.glob('*@*.journal'))
    print('after rotation: archived old message removed; active new message remains')
    print('persistent files observed under /var/log/journal; no reboot durability claim')
finally:
    run('systemctl','stop','systemd-journald@'+name+'.service',
        'systemd-journald@'+name+'.socket','systemd-journald-varlink@'+name+'.socket')
    config.unlink(missing_ok=True)
    for path in (directory,Path('/run/log/journal')/directory.name,Path('/run/systemd')/('journal.'+name)):
        if path.exists():shutil.rmtree(path)

The first message is present in journal files beneath the namespace’s /var/log/journal directory. After restarting that namespace’s journald service, the message is still queryable. This verifies the selected storage path and survival across a daemon restart. It does not simulate a power failure, prove storage-device durability, or test a complete machine reboot.

Vacuuming to a one-byte target initially leaves the message visible because its file is still active. The requested size is too small, but vacuum is not permitted to delete that active file just to meet the number. This provides a concrete counterexample to treating a vacuum target as a guaranteed final disk-usage figure.

The script then rotates the journal and writes a new message. Rotation closes the earlier file as an archive and opens a replacement. Both messages are readable before the second vacuum. Afterward, the archived old message is gone while the new message in the active file remains. Rotation and deletion are separate operations with different effects on evidence.

All journal commands select the unique namespace, so they do not vacuum the system’s default journal. Namespace service and socket instances are stopped before their test directories are removed. A normal query without the namespace selector would look in the default namespace and could miss these records entirely, even though they were successfully stored.

Size the evidence window

Choose retention from the incident-response window and observed message volume. A size limit cannot promise a fixed number of days when traffic changes. A retention-age limit likewise does not reserve enough bytes to preserve every message until that age. Monitor actual disk use and the earliest available records, especially after an unusually noisy failure.

Log collection can also lose messages before retention becomes relevant. Rate limits, application output routing, permissions, and unavailable storage affect what reaches the journal. Preserve the distinction between “no matching stored record” and “the event never occurred.” Avoid putting credentials into messages merely because local journal access is restricted.

If records must survive loss of the machine, provide and verify an independent collection path. Forwarding configuration alone is not evidence that a remote destination received or retained the message. Check delivery, access control, and the remote retention window using an identifiable harmless event. Local searchable logs and durable external evidence answer related but different operational needs.

Further reading

Sources