When /var/log fills up
I’ve run a handful of services on my homelab box—Docker containers, an Nginx reverse proxy, a Samba share, and a few cron jobs that ping my home router. It’s a quiet machine most of the time, but every so often the disk starts choking on logs. When /var/log hits the filesystem limit, new log entries stop, services can silently die, and journalctl starts spitting out “no space left on device”. The fix is usually a quick vacuum of the journal and a tidy‑up of the plain‑text logs. Below is a step‑by‑step recipe that works on Debian, Ubuntu, Arch, and other systemd‑based distros.
Why systemd‑journald is a separate beast
journalctl stores logs in binary blobs under /var/log/journal/<machine-id>/. The default Storage=auto setting writes to that directory if a persistent storage directory exists; otherwise it falls back to /run/log/journal, which is wiped on reboot. Because the journal is binary, traditional logrotate cannot rotate it. Instead, you use journalctl’s built‑in vacuum commands.
The key configuration knobs live in /etc/systemd/journald.conf:
| Option | Meaning | Typical value |
|---|---|---|
SystemMaxUse |
Max disk space for persistent logs | 1G |
SystemKeepFree |
Space to keep free | 100M |
SystemMaxFileSize |
Max size per journal file | 100M |
RuntimeMaxUse |
Max space for volatile logs | 200M |
Adjusting these values gives you a predictable cap on how much space the journal can consume. For a small home lab, SystemMaxUse=1G is usually enough.
Checking what’s eating the space
Before you start deleting, see how much the journal actually uses:
$ sudo journalctl --disk-usage
Disk usage for /var/log/journal/…: 1.3 GB
If the number is close to your filesystem quota, you’re ready to vacuum. You can also inspect the oldest entries:
$ sudo journalctl --since "2026-01-01" --until "2026-01-02" | head
This lets you confirm that the logs you’re about to discard are not recent or critical.
Using journalctl --vacuum-time
The most common way to free space is to keep only the last N days of journal data. For example, to drop everything older than 30 days:
$ sudo journalctl --vacuum-time=30d
If you prefer to limit the total size instead of a time window, you can use --vacuum-size. For instance, to keep the journal under 500 MiB:
$ sudo journalctl --vacuum-size=500M
You can combine these approaches. I usually start with a time‑based vacuum, then run a size‑based one if the disk is still tight.
Cleaning up plain‑text logs
journalctl won’t touch the classic /var/log/*.log files. Those are still managed by logrotate. On most distros you’ll find a default /etc/logrotate.d/ directory with entries for syslog, kern.log, auth.log, etc. If you’re running Docker or Nginx, they’ll have their own logrotate configs in /etc/logrotate.d/docker-containerd or /etc/logrotate.d/nginx. Make sure those configs keep a sensible number of rotated files and a reasonable size limit.
A quick sanity check:
$ sudo du -sh /var/log/*
If you see a file that’s suddenly huge, dig into its rotation policy. The default config usually looks like this:
/var/log/nginx/*.log {
daily
missingok
rotate 14
compress
delaycompress
notifempty
create 0640 www-data adm
}
Feel free to tweak the rotate count or add size thresholds if you notice logs ballooning.
A word on persistence
If you’re running a system where /var/log/journal is on a separate partition, remember that the journal will keep growing until you vacuum it. A common pitfall is to forget that journalctl --vacuum-* only affects the persistent store; logs in /run/log/journal are volatile and disappear on reboot, but they still count toward the runtime limits.
Bottom line
- Check the current usage with
journalctl --disk-usage. - Vacuum with
--vacuum-timeor--vacuum-sizeuntil the disk is breathing again. - Inspect and tweak
/etc/systemd/journald.conffor a sane default cap. - Verify that your plain‑text logs are rotated properly.
That’s all you need to keep your homelab from turning into a log‑filled disaster zone.
Tags: systemd, journald, logs, linux, homelab
See also
- Why I keep a weekly backup copy of my home directory on an external SSD using rsync and how I test the restore
- Why ssh keeps spawning new processes on every connect and how ControlPersist solves it
- Repairing a broken initramfs that locked my Raspberry Pi in emergency mode after a kernel update
- How I added ionice ‑c3 to my nightly rsync backup and finally stopped my laptop from overheating
- Running a nightly backup script after logout with systemd user units – no nohup needed