Skip to content
statance
MenuClose
← All troubleshooting guides

systemd journal logs are filling the disk

Linux

Check journal size with journalctl --disk-usage, shrink it with sudo journalctl --vacuum-size=500M, and set SystemMaxUse=500M in /etc/systemd/journald.conf so it stays capped. If the journal grows fast, find the noisy unit — a service logging in a tight loop is usually a bug worth fixing too.

Symptoms

  • /var/log/journal is many gigabytes
  • Disk fills steadily even with logrotate configured
  • journalctl is slow to open

How to diagnose it

  1. Check journal size

    Shows total size of archived and active journal files.

    journalctl --disk-usage
  2. Find the noisiest units

    Counts log lines per unit in the last hour.

    journalctl --since "1 hour ago" -o json --output-fields=_SYSTEMD_UNIT | grep -o '"_SYSTEMD_UNIT":"[^"]*"' | sort | uniq -c | sort -rn | head
  3. Read what it is logging

    A repeating error is the real problem to fix.

    journalctl -u SERVICE -n 50 --no-pager

Common causes and fixes

No size limit configured

Set a limit and restart journald.

sudo sed -i 's/^#\?SystemMaxUse=.*/SystemMaxUse=500M/' /etc/systemd/journald.conf sudo systemctl restart systemd-journald

Old logs never cleaned

Vacuum by size or age. This removes only archived journal files.

sudo journalctl --vacuum-size=500M sudo journalctl --vacuum-time=14d

A service logging in a loop

Fix the crash loop or error, or lower its log level. Rate limits (RateLimitIntervalSec, RateLimitBurst) can contain a noisy unit.

Prevent it happening again

  • Set SystemMaxUse in journald.conf on every server image.
  • Alert when DISK_used is above 85%.
  • Alert on SERVICES_failed — crash-looping services are a common source of log floods.

Set it up: Linux monitoring · Ubuntu monitoring.

FAQ

Is it safe to delete /var/log/journal?

Use journalctl --vacuum-size instead. It removes archived files cleanly; deleting files by hand while journald runs can leave space held.

Where is the journald config?

/etc/systemd/journald.conf, or a drop-in file in /etc/systemd/journald.conf.d/.

Related guides

Get alerted before this becomes an outage.

Install the agent in one line. Statance watches CPU, memory, disk and services, and its AI assistant helps find the cause.