Open Source

On a small home‑lab server or a single‑user laptop, the root filesystem is often the first place that runs out of space. I ran into this on a 2025‑era Raspberry Pi running Ubuntu 24.04 LTS: the systemd journal kept growing until the root partition hit 95 % and the machine refused to boot. The fix was surprisingly simple once you understand how systemd‑journald works and how to steer its output away from the root filesystem.

Raspberry Pi’s are perfect for low‑power tinkering, but a 16‑GB microSD can start biting back if something writes too much data. After a year of unattended cron jobs, my Pi’s /var/log ballooned to 10 GB and the boot process stalled. Below is the exact walk‑through I used to bring the system back online and prevent a repeat.

Why the logs ate the SD card

The default Raspberry Pi OS ships with systemd‑journald for kernel and service logs and rsyslog for the classic syslog files. Both dump their output into /var/log. If you run cron jobs that spit out text and you don’t redirect or limit that output, each run just appends. Over a year, even a few megabytes per job can turn into gigabytes. The SD card’s limited write endurance means a bloated log file can accelerate wear‑out.

When /boot runs out of space and the machine drops into emergency mode

A few minutes after a kernel update, you power on the server and the console shows:

System boot failed.
Emergency mode - type "journalctl -xb" to view logs.

The root filesystem is mounted read‑only, the initramfs cannot be loaded, and the machine refuses to start. The culprit is almost always a full /boot partition. In this post I walk through the exact steps to diagnose, free space, and reboot safely. I’ll cover the most common distributions, show how to keep the system lean, and touch on security best‑practices that come with kernel housekeeping.

The mystery of the reboot crash

I had a single‑board server running Debian 12 in my living‑room. Every time I rebooted it, the Docker Compose stack that powers my home‑lab services would fail to start, and the machine would hang in a boot loop. The logs were a jumble of “failed to start” messages, but the root cause turned out to be a mis‑configured systemd unit. Below is the exact process I followed to diagnose and fix the issue, with a few trade‑offs and security notes that apply to any Docker‑based deployment on a small server.

When /var/log fills up

I’ve run a handful of services on my homelab box—Docker containers, an Nginx reverse proxy, a Samba share, and a few cron jobs that ping my home router. It’s a quiet machine most of the time, but every so often the disk starts choking on logs. When /var/log hits the filesystem limit, new log entries stop, services can silently die, and journalctl starts spitting out “no space left on device”. The fix is usually a quick vacuum of the journal and a tidy‑up of the plain‑text logs. Below is a step‑by‑step recipe that works on Debian, Ubuntu, Arch, and other systemd‑based distros.

A Raspberry Pi stuck in emergency mode after a kernel update?

I hit this one last week while upgrading to the 6.6‑raspi kernel. The Pi rebooted, the splash screen faded, and I was greeted with the dreaded “Emergency mode” prompt. I had no idea what was wrong until I dug into the initramfs. Below is a step‑by‑step guide I used to recover, plus a few lessons that apply to any Debian‑based system.

When a Borg backup stops halfway

You’re in the middle of a nightly backup of your homelab server, the progress bar hits 50 % and then the process dies. Maybe a sudden power loss, a network hiccup, or a disk error. The archive on the remote storage looks like a half‑filled container. You’re left with a repository that has a partially written archive. How do you recover the data you already captured, and how do you finish the backup without starting from scratch?

Stale IPs on a Home Network? Let dnsmasq Clean Up the Mess

If you’ve ever pinged a device that had moved to a new address and the reply came from an old IP, you’ve probably been frustrated by the same stale DNS entries that linger in the resolver cache. On a typical Ubuntu desktop or server, systemd‑resolved is the default DNS stub. It’s great for most use cases, but it can keep outdated records for a long time, especially when your router’s DHCP server changes leases frequently. The result? Your own machine keeps trying to resolve a dead address until the cache finally expires.

Nightly rsync vanished after moving to systemd timers

I had a simple nightly backup script that ran under cron for years. One evening I decided to modernise the scheduler: replace the crontab entry with a systemd timer. The idea was clean, the logs would be in journalctl, and I could keep the backup logic in a single unit file. A few days later the backup disappeared from the destination directory. After a quick audit I discovered that the timer had never fired because the system had been powered off at the scheduled time. The culprit was the missing Persistent=true setting.

Install Podman and Enable User Namespaces

sudo apt update
sudo apt install -y podman

On Debian 12, Podman ships with userns-remap turned off.
I always enable it right away by editing /etc/containers/containers.conf:

[engine]
userns = "keep-id"

keep-id keeps the container’s UID/GID the same as the host user’s, which is the simplest approach for a single‑user homelab. If you have multiple users, you might want auto or a custom mapping.

After that tweak, restart the daemon: