I was running a little home‑lab server on a 2 GB SSD. The board was an Orange Pi Zero‑like single‑board computer, and it was juggling a personal web server, a GitLab instance, and a handful of containerised apps. After a month of steady traffic, the SSD hit 100 % on /var. The culprit? systemd‑journald had quietly gobbled up the whole drive.
journald is the default logger on most modern distros. It keeps logs in binary form under /var/log/journal. By default it will keep everything until the filesystem is full, then start deleting the oldest entries. On a tiny SSD that can happen faster than you can say “disk full”.
I needed a quick, single‑line fix that would stop the journal from eating the SSD while still keeping recent logs handy for troubleshooting. Below is how I did it, along with trade‑offs, alternatives, and a few security notes.
Why journald grew so fast
| Service | Typical log volume | Notes |
|---|---|---|
sshd |
10–20 MB/month | Verbose auth logs |
docker / podman |
30–50 MB/month | Container start/stop logs |
gitlab |
50–80 MB/month | CI/CD job logs |
systemd itself |
5–10 MB/month | Boot, shutdown, unit status |
| Misc. | 10–20 MB/month | journalctl queries, cron jobs |
Even a modest log volume can fill a 2 GB SSD in a few weeks. The defaults in /etc/systemd/journald.conf look like this:
#Storage=persistent
#SystemMaxUse=0
#SystemKeepFree=0
SystemMaxUse=0 means “no limit”. That’s fine for a machine with a large drive, but it’s a recipe for disaster on a 2 GB SSD.
One‑line cap: SystemMaxUse=200M
The simplest way to constrain the journal is to set SystemMaxUse in journald.conf. I just opened the file and added the line:
sudo nano /etc/systemd/journald.conf
SystemMaxUse=200M
That tells journald to keep at most 200 MB of journal data. Anything beyond that will be purged automatically. Reload the daemon:
sudo systemctl restart systemd-journald
Check the new limit:
journalctl --disk-usage
You should see something like:
Archived: 0 bytes
Current: 180.3 MB
If the journal was already larger than 200 MB, journald will trim it immediately when you restart. Subsequent logs will respect the new cap.
Trade‑offs to consider
| Benefit | Drawback |
|---|---|
| Predictable disk usage | Loss of older logs – you’ll never see logs older than the most recent 200 MB. |
| Simplicity | Potential debugging gaps – if a problem occurs and you need older logs, you’ll be out of luck. |
| No extra tooling | No compression – the journal files are already compressed, but you lose the ability to archive them elsewhere. |
If you’re running services that churn a lot of logs (think CI/CD), you might want to keep a separate archive. A two‑tier approach works well: keep a small local journal and ship logs to a remote syslog server or a log‑aggregation service (e.g., Loki, Graylog). That way you preserve history without bloating the SSD.
Alternative approaches
1. SystemKeepFree
Instead of limiting the total size, you can tell journald to keep a certain amount of free space:
SystemKeepFree=500M
On a 2 GB SSD this leaves 1.5 GB for everything else. The journal will delete entries until that free space is restored. Handy if you want to guarantee that other services never starve for disk.
2. SystemMaxFileSize + SystemMaxFiles
You can set a per‑file size limit and a maximum number of files:
SystemMaxFileSize=50M
SystemMaxFiles=4
That keeps the journal in four 50 MB files, totaling 200 MB. It gives you more granular control, but you have to tweak two settings instead of one.
3. Remote logging
journald can forward logs to a remote syslog server via ForwardToSyslog=yes. On the remote side you can store logs on a larger disk or a dedicated log server. This decouples log storage from the local machine entirely.
4. Log rotation with logrotate
journald handles its own rotation, but you can still use logrotate to compress or delete older binary journal files. For example, create /etc/logrotate.d/journal:
/var/log/journal/*/*.journal {
daily
rotate 7
compress
missingok
}
This keeps seven days of compressed journal files. It’s an extra step but gives you more control over retention.
Security considerations
journald stores logs in /var/log/journal. By default the directory and files are owned by root:systemd-journal with permissions 0700. That means only root and the systemd-journal group can read them. If you add other users to that group, they’ll see all logs, which can expose sensitive data (e.g., debug output containing passwords). Keep the defaults unless you have a compelling reason to change them.
Capping the journal reduces the amount of data that could be exfiltrated if an attacker gains read access. But it also cuts down on forensic evidence after a compromise. Balance the two based on your threat model.
Troubleshooting tips
| Symptom | Check | Fix |
|---|---|---|
journalctl --disk-usage shows >200 MB |
journald.conf |
Make sure SystemMaxUse=200M is set and not overridden by a drop‑in file (/etc/systemd/journald.conf.d/*.conf). |
systemd-journald not restarting |
systemctl status systemd-journald |
Look for errors in the unit file. Restart with sudo systemctl restart systemd-journald. |
| Journal still grows | journalctl --disk-usage |
Verify that the journal is persistent (Storage=persistent). If Storage=volatile, logs are stored in RAM and not on disk. |
| Logs missing after cap | journalctl -b |
Confirm that the limit is enforced by checking the timestamp of the oldest entry. |
If you’re still unsure, run:
journalctl --disk-usage --no-pager
It will give you a concise summary of current usage.
Real‑world usage: a 2 GB SSD home server
I ran this setup on an Orange Pi Zero‑like board. With the SystemMaxUse=200M cap in place, the SSD stayed healthy and the services kept running without surprises. No more “disk full” panics, and I still had enough recent logs to debug issues.
See also
- Why my Borg backup kept failing after upgrading to Ubuntu 24.04 and how I fixed it
- Why My Home Server Keeps Crashing After Reboot: Fixing the Broken systemd Service for My Docker Compose Deployment
- Freeing a Full `/var/log` on a Home Lab Server with `journalctl --vacuum-time` and a One‑Line Bash Cleanup
- Why I keep a weekly backup copy of my home directory on an external SSD using rsync and how I test the restore
- Why ssh keeps spawning new processes on every connect and how ControlPersist solves it