How a Forgotten SSH Key Stopped My Nightly Off‑Site Borg Backup and the Quick Fix I Implemented

The night I lost an off‑site Borg backup

I run a 24/7 home‑lab server that backs up my home directories to a remote NAS with Borg. The backup is triggered by a systemd timer that fires every night at 02:00 UTC. For months it worked flawlessly, but one morning the timer ran, the job logged “ssh: connect to host …: Permission denied”, and the backup vanished into the night. The culprit? A forgotten SSH key.

How the key disappeared

My backup script uses a dedicated user borgbackup that authenticates to the NAS via an SSH key stored in /home/borgbackup/.ssh/id_rsa. The key was generated in 2024 and added to the NAS’s authorized_keys with ssh-copy-id. In 2025 I rotated the key on the NAS for routine hygiene, but I forgot to replace the local copy. The old key was still on the NAS, but the local id_rsa was deleted during a clean‑up script that removed unused keys.

Because the timer runs as borgbackup, it had no way to prompt for a passphrase or fall back to another key. The failure was silent until the next run, which is why I only noticed it when the backup directory on the NAS was empty.

Quick fix: re‑add the key and make the timer resilient

Generate a new key pair (or reuse the old one if you still have it).
```
sudo -u borgbackup -i
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N ""
```
Using Ed25519 gives better security and faster authentication.

Copy the public key to the NAS.

ssh-copy-id -i ~/.ssh/id_ed25519.pub [email protected]

Verify the key works manually:

ssh -i ~/.ssh/id_ed25519 [email protected] 'echo OK'

Update the systemd service to use the new key automatically.
The service file (/etc/systemd/system/borgbackup.service) already contains:

[Service]
User=borgbackup
ExecStart=/usr/bin/borg create --stats --compression lz4 \
  /mnt/backup::%Y-%m-%d-%H-%M-%S /home/borgbackup
Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket

Add a small wrapper that starts ssh-agent if it’s not running:

# /usr/local/bin/ssh-agent-wrapper
#!/usr/bin/env bash
set -euo pipefail

# Start ssh-agent only if the socket is missing
if [ ! -S "$SSH_AUTH_SOCK" ]; then
  eval "$(ssh-agent -a $XDG_RUNTIME_DIR/ssh-agent.socket)"
  ssh-add ~/.ssh/id_ed25519
fi

exec "$@"

Make it executable:

sudo chmod +x /usr/local/bin/ssh-agent-wrapper

Then change the service to call the wrapper:

ExecStart=/usr/local/bin/ssh-agent-wrapper /usr/bin/borg create ...

Reload systemd and restart the timer:

sudo systemctl daemon-reload
sudo systemctl restart borgbackup.timer

Verify the next run. The timer should now create a new archive on the NAS without prompting.

Why this matters for security and reliability

Key rotation: Rotating keys on the NAS is good practice, but the local copy must stay in sync. Automate key sync with a small script that runs on key rotation events.
Agent usage: ssh-agent keeps the private key in memory and avoids storing it on disk in plain text. The wrapper ensures the agent starts only once per session, reducing the attack surface.
Timer resilience: By wrapping the command, the service becomes idempotent. If the agent dies, the wrapper restarts it automatically.
Audit trail: The systemd journal now records the agent startup, making it easier to debug future failures.

Quick checklist for future backups

Step	Command	Why
Generate key	`ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N ""`	Strong, fast key
Copy key	`ssh-copy-id -i ~/.ssh/id_ed25519.pub user@host`	Sync public key
Agent wrapper	`ssh-agent -a $XDG_RUNTIME_DIR/ssh-agent.socket`	Persistent agent
Systemd service	`Environment=SSH_AUTH_SOCK=$XDG_RUNTIME_DIR/ssh-agent.socket`	Pass socket to service
Timer	`systemctl list-timers`	Confirm schedule

Takeaway

A forgotten key can bring a nightly backup to a halt in seconds. The fix is simple: regenerate, copy, and wrap the command in a small agent starter. The added resilience protects against future key rotations or accidental deletions. If you’re running Borg or any other remote‑host backup, make sure your SSH key lifecycle is automated and your systemd service is robust.