Open Source

When /boot runs out of space and the machine drops into emergency mode

A few minutes after a kernel update, you power on the server and the console shows:

System boot failed.
Emergency mode - type "journalctl -xb" to view logs.

The root filesystem is mounted read‑only, the initramfs cannot be loaded, and the machine refuses to start. The culprit is almost always a full /boot partition. In this post I walk through the exact steps to diagnose, free space, and reboot safely. I’ll cover the most common distributions, show how to keep the system lean, and touch on security best‑practices that come with kernel housekeeping.

The Flicker Problem

I bought a 2022 Dell XPS 13 a few months ago and, like most people, I wanted to use an external 4K monitor via the single USB‑C port. The laptop ships with a Thunderbolt 4 controller that should handle 60 Hz 4K output without issue. On my Fedora 38 machine the monitor flickers when I connect it, and the flicker is worse on Wayland than on Xorg. The screen stutters every 1–2 seconds because the refresh rate drops in a periodic fashion. I spent a week trying to pin down the cause before finding a reliable fix.

USB plug‑in spam is a classic annoyance on modern Linux boxes. Every time you drop a keyboard, a mouse, or even a USB‑SD card in, the kernel goes into a burst‑mode and writes a wall of messages to journalctl. On a busy server or a home‑lab machine that logs everything, those lines can quickly fill the journal, slow down log‑rotation, and make it hard to spot real problems.

Below is a hands‑on guide to quiet those messages without losing the useful diagnostics you need. The techniques work on recent systemd‑based distributions (Debian 12, Ubuntu 24.04, Arch 2026, etc.) and are safe for a security‑aware setup because they only suppress noise, not real alerts.

The mystery of the reboot crash

I had a single‑board server running Debian 12 in my living‑room. Every time I rebooted it, the Docker Compose stack that powers my home‑lab services would fail to start, and the machine would hang in a boot loop. The logs were a jumble of “failed to start” messages, but the root cause turned out to be a mis‑configured systemd unit. Below is the exact process I followed to diagnose and fix the issue, with a few trade‑offs and security notes that apply to any Docker‑based deployment on a small server.

When /var/log fills up

I’ve run a handful of services on my homelab box—Docker containers, an Nginx reverse proxy, a Samba share, and a few cron jobs that ping my home router. It’s a quiet machine most of the time, but every so often the disk starts choking on logs. When /var/log hits the filesystem limit, new log entries stop, services can silently die, and journalctl starts spitting out “no space left on device”. The fix is usually a quick vacuum of the journal and a tidy‑up of the plain‑text logs. Below is a step‑by‑step recipe that works on Debian, Ubuntu, Arch, and other systemd‑based distros.

Why I keep a weekly backup copy of my home directory on an external SSD using rsync

I’m not a “data‑hoarder” in the sense of storing every file I own. I keep a single, up‑to‑date snapshot of my ~/ on a 2 TB SSD that I plug into my workstation once a week. The copy is made with rsync, stored on a separate partition, and I run a quick restore test every month. The routine feels like a small ritual that gives me confidence without eating bandwidth or storage.

A Raspberry Pi stuck in emergency mode after a kernel update?

I hit this one last week while upgrading to the 6.6‑raspi kernel. The Pi rebooted, the splash screen faded, and I was greeted with the dreaded “Emergency mode” prompt. I had no idea what was wrong until I dug into the initramfs. Below is a step‑by‑step guide I used to recover, plus a few lessons that apply to any Debian‑based system.

When a Borg backup stops halfway

You’re in the middle of a nightly backup of your homelab server, the progress bar hits 50 % and then the process dies. Maybe a sudden power loss, a network hiccup, or a disk error. The archive on the remote storage looks like a half‑filled container. You’re left with a repository that has a partially written archive. How do you recover the data you already captured, and how do you finish the backup without starting from scratch?

Keeping a Home Server Alive During Let’s Encrypt Renewals

Let’s Encrypt certificates are only good for 90 days. On a little home box that runs a handful of services—web, mail, VPN, media—any hiccup during renewal can bring the whole stack down. I’ve seen this happen once: the renewal script fired, nginx restarted, and the firewall dropped all traffic. The fix I settled on is a single systemd.timer that guarantees the renewal runs in a safe window, with a fallback that keeps services up if something goes wrong.

The journal can drown you in Wi‑Fi disconnect chatter

Ever notice your laptop or Raspberry Pi stuck in a loop of “connected” / “disconnected” messages? The kernel, NetworkManager, wpa_supplicant – they all go on about it. On a busy box the journal gets clogged with the same line over and over, and your console turns into a flood of warnings that look like fresh problems every few seconds. If you’re a homelab operator, a self‑hoster, or just a sysadmin who likes a tidy log, you’ll want a clean way to silence the noise without losing the diagnostics that help you spot real outages.