Troubleshooting

Auditing Stale SSH Keys with ssh-keygen -l and Protecting with fail2ban

SSH keys that sit unused for months can become a silent backdoor if a private key ever gets compromised. On a production server the safest way to keep the key list lean is to audit it regularly and block any brute‑force attempts that surface while you’re cleaning up.

1. List all authorized keys with fingerprints

The authorized_keys file lives in each user’s ~/.ssh/. To see which keys are actually present, run:

Automating Let’s Encrypt renewal on a single‑node web server without downtime

Let’s Encrypt certificates expire every 90 days. On a single‑node web server the usual approach is to run certbot renew from cron and reload the web service. The trick is to avoid any traffic loss while the new cert is in place. Below is a minimal, production‑ready recipe that works with Apache, Nginx, or any systemd‑managed web server.

1. Install Certbot and the web‑root plugin

# Debian/Ubuntu
sudo apt-get install certbot python3-certbot-nginx   # or python3-certbot-apache

# RHEL/CentOS
sudo dnf install certbot python3-certbot-nginx

The nginx or apache plugin automatically configures the web‑root challenge and reloads the server after a successful renewal.

Disk exhaustion is the most common cause of a sudden “500 Internal Server Error” on a small VPS.
When the filesystem that holds /var/log/apache2 is full, Apache can’t rotate or write new logs, and the web server silently stops accepting requests.
The fix is usually a quick cleanup, but doing it manually every time is error‑prone and can leave you with the same problem again.
Below is a practical, repeatable script that:

Disk‑full crashes are the most common reason a VPS goes from “alive” to “dead” overnight.
A log file or a database that blows past the allotted space will make the kernel refuse any further writes, and every process that needs to touch the disk will start failing. If you’re running a website, mail server, or database, that’s a service outage you can’t afford.

The silver lining? A handful of LVM tricks and a tiny monitoring script can get you back up in minutes and keep future incidents at bay.
Below I walk through a real‑world scenario on a Debian‑based VPS, but the ideas carry over to any distro that uses LVM and systemd.

ProxyJump timeouts when a VPN drops your IP address

When you hop through a bastion with ssh -J, the jump host keeps a TCP tunnel open to the target.
If you’re on a VPN that can change your public IP or even tear the tunnel down, that tunnel can stall or die.
The result? A hanging SSH that eventually spits out “Connection timed out” or “Connection reset by peer”.
For homelab users who rely on VPNs for remote access, this is the kind of annoyance that keeps you awake at night.

Keep a VPN alive when the home router flips between DHCP and static

I’ve been on a laptop that plugs into a home router that sometimes hands out a DHCP lease and other times uses a static IP you set in the admin UI. The VPN client (OpenVPN 3‑CLI on Ubuntu 24.04) would drop the tunnel as soon as the interface lost its address, forcing me to reconnect manually. Below is a minimal, reproducible setup that keeps the tunnel up regardless of the router’s mode.

I was in the middle of setting up a tiny homelab VM that bridges straight to the host. The guest could ping the gateway just fine, but ip addr show br0 was staring back at me with no inet line. The bridge was up, state UP, yet no IPv4. What’s going on?

1. Distinguish link‑state from IP‑state

ip link only tells you about the link layer. An interface can be physically up and still have no IP if the DHCP client never got a lease or a static config never ran.

Headphone jack gone silent after the 2026‑04 kernel update

I was mid‑streaming a podcast when the sound just… vanished. My 3.5 mm jack had been a reliable work‑horse for years, but after pulling the 5.15.12‑arch kernel I got nothing. The driver was still loaded, aplay -l still listed the jack, but the audio was dead. Turns out the culprit was a subtle tweak in ALSA’s power‑management for the HDA codec. A single udev rule and the jack was back in seconds.

When Clonezilla says root=UUID not found

Cloning a bootable SSD with Clonezilla is usually painless, but the new machine can stall at the kernel prompt with:

root=UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx not found

The kernel can’t locate the root filesystem because the UUIDs in the boot configuration still point to the original disk. The fix is a handful of targeted edits and rebuilds.

1. Verify the new UUIDs

Boot from a live USB, open a terminal and list the block devices:

I hit a wall yesterday when I accidentally set /home/user to 777 and lost SSH access.
The server was a 2026‑era Debian‑based homebox running systemd‑managed services.
SSH refuses to log in if any file under the user’s home directory is writable by others, so the key‑based login failed immediately.
The only way to fix it was to boot from a live USB, mount the root filesystem, and restore the correct ownership and permissions.