Introduction to systemd Recovery
I’ve seen this go wrong when a systemd update fails - it’s a pretty stressful experience, especially when systemd is the core init system for your Linux distribution. Recently, I had a similar issue on my small home server. The update process seemed fine at first, but after a reboot, I was greeted with an error message indicating that systemd had failed to start.
Identifying the Problem
The real trick is to stay calm and use the right tools to diagnose the issue. I booted into rescue mode using a live USB stick, which gave me access to the filesystem. From there, I inspected the systemd logs using the following command:
journalctl -u systemd
This showed me the error messages related to systemd, helping me narrow down the problem. Don’t bother with trying to fix it without checking the logs - you’ll just be guessing.
Reverting the Update
To recover from the failed update, I needed to revert the changes made by the update process. This is where people usually get burned - using the wrong version number can cause more problems. I used the following command to downgrade the systemd package:
apt install systemd=247.3-1
Note that the version number may vary depending on your distribution and the version of systemd you were running before the update. In practice, it’s essential to use the correct version number.
Verifying the Fix
After downgrading systemd, I rebooted the system to verify that the fix worked. Luckily, the system booted up normally, and systemd was functioning correctly again. I then used the following command to re-enable the systemd update:
apt install systemd
This ensured that systemd was updated to the latest version available in the repository.
Preventing Future Issues
I usually start with regular backups using tools like rsync and btrfs to prevent data loss. I’ve also set up a systemd timer to automatically run a script that checks for and applies updates, keeping my system up-to-date without requiring manual intervention.
See also
- Recovering from a Failed Boot after Accidentally Removing Initramfs with a Rootless Podman Container
- Taming Background Tasks with nohup, disown, and systemd's linger Option
- Taming Recursive Greps with find and xargs in My Homelab Log Cleanup Scripts
- Taming Terminal Output with tmux and awk for Easier Log Analysis
- Taming Dependency Hell with apt-mark and Pinning on Debian-Based Systems