Keeping a Home Server Alive During Let’s Encrypt Renewals
Let’s Encrypt certificates are only good for 90 days. On a little home box that runs a handful of services—web, mail, VPN, media—any hiccup during renewal can bring the whole stack down. I’ve seen this happen once: the renewal script fired, nginx restarted, and the firewall dropped all traffic. The fix I settled on is a single systemd.timer that guarantees the renewal runs in a safe window, with a fallback that keeps services up if something goes wrong.
Below is the one‑liner timer I added, plus the supporting unit files, and a discussion of why this approach beats the usual “run‑once‑at‑boot” or cron‑based methods.
Why the “one‑liner” matters
The renewal process typically involves:
- Stopping the service that holds the port (e.g.,
nginx). - Running
certbot renew(or another ACME client). - Restarting the service.
If the service is stopped and the renewal fails, the service stays down. If the service is restarted before the renewal completes, the new certificate never gets applied. A timer that runs the renewal after a graceful stop and before a restart, with a clear timeout, eliminates the race condition.
The timer definition
Create /etc/systemd/system/letsencrypt-renew.timer:
[Unit]
Description=Run Let’s Encrypt renewal after a safe window
Requires=letsencrypt-renew.service
[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
AccuracySec=1min
[Install]
WantedBy=timers.target
- OnCalendar – runs daily at 02:00 UTC. Pick a low‑traffic hour; 02:00 is usually quiet on a home network.
- Persistent=true – if the server was down at the scheduled time, the timer fires immediately on boot.
- AccuracySec – lets systemd batch timers to reduce power spikes; 1 min is fine for a single renewal job.
The timer pulls in letsencrypt-renew.service which does the heavy lifting.
The service unit
Create /etc/systemd/system/letsencrypt-renew.service:
[Unit]
Description=Renew Let’s Encrypt certificates safely
After=network-online.target
Before=nginx.service
Requires=network-online.target
[Service]
Type=oneshot
ExecStartPre=/usr/bin/systemctl stop nginx
ExecStart=/usr/bin/certbot renew --quiet --deploy-hook "systemctl reload nginx"
ExecStartPost=/usr/bin/systemctl start nginx
TimeoutSec=1800
Restart=on-failure
RestartSec=5min
What each line does
| Directive | Purpose |
|---|---|
After=network-online.target |
Ensures the network is up before attempting renewal. |
Before=nginx.service |
Guarantees the timer runs before nginx starts. |
ExecStartPre |
Stops nginx cleanly. |
ExecStart |
Runs certbot with --quiet to suppress output, and a --deploy-hook that reloads nginx only if renewal succeeded. |
ExecStartPost |
Starts nginx if the renewal succeeded. |
TimeoutSec=1800 |
Gives the renewal 30 minutes; if it hangs, systemd kills it. |
Restart=on-failure |
If the renewal fails, systemd will retry after RestartSec. |
Why not use a cron job? Cron runs in a minimal environment and has no built‑in dependency handling. If the network is down, the job silently fails. systemd timers integrate with the init system, respect network state, and provide reliable restart semantics.
Security considerations
- Least privilege – The service runs as root (required by certbot to write to
/etc/letsencrypt). If you’re using a non‑root ACME client, adjust theUser=directive accordingly. - Audit logs – systemd logs the timer and service to
journalctl. Monitor/var/log/journalfor unexpected restarts. - Deploy‑hook safety – The hook reloads nginx only after a successful renewal. If the hook fails, nginx stays stopped, and the timer will retry. This prevents a broken certificate from being served.
- Network isolation – The timer requires
network-online.target. If you run the server in a container or VM, ensure the container’s network namespace is reachable.
Monitoring and alerts
A single timer is great, but you still want to know if a renewal fails. Add a simple systemd notification script:
#!/usr/bin/env bash
if systemctl is-failed letsencrypt-renew.service; then
# Send an email or push notification
echo "Let’s Encrypt renewal failed on $(hostname)" | mail -s "Renewal alert" [email protected]
fi
Schedule it with a daily timer or trigger it from the ExecStartPost of the renewal service. For more robust monitoring, integrate with Prometheus node exporter or a lightweight log‑watcher like logwatch.
Troubleshooting common pitfalls
| Symptom | Likely cause | Fix |
|---|---|---|
| nginx never starts after timer fires | certbot failed to obtain a new cert |
Check /var/log/letsencrypt/letsencrypt.log and run certbot renew --dry-run manually. |
| Timer never runs | systemctl enable letsencrypt-renew.timer not executed |
Run systemctl enable --now letsencrypt-renew.timer and verify with systemctl list-timers. |
| Service restarts too often | Restart=on-failure triggers on non‑fatal exit codes |
Adjust Restart= or add SuccessExitStatus=0 1 to treat specific codes as success. |
| Network not ready | network-online.target not satisfied |
Ensure systemd-networkd-wait-online.service is active, or replace with After=network.target. |
Trade‑offs and alternatives
| Approach | Pros | Cons |
|---|---|---|
| systemd timer + service (this article) | Reliable, integrated, no external cron, clear dependencies | Slightly more configuration, requires root for certbot |
| cron + certbot | Simple, widely known | No dependency handling, silent failures |
| certbot‑renew‑hook | Minimal config | Still needs a trigger; less control over timing |
| ACME‑client with built‑in timer (e.g., acme.sh) | One binary, no systemd | May not support all hooks; less community support |
If you prefer containers, you can run certbot in a transient container and mount /etc/letsencrypt. The timer would then trigger the container. This isolates the ACME client from the host, but adds complexity.
Practical example: adding a fallback
Sometimes the renewal succeeds but the deploy hook fails (e.g., nginx config syntax error). To avoid a silent failure, add a ExecStartPost that checks
See also
- The Quick Fix to Stop a Flatpak App From Updating on My Wayland Desktop
- Stopping the Journal from Flooding me with Repeated Wi‑Fi Disconnection Notices
- The day my Downloads folder vanished after a simple umask change: my recovery steps
- Replacing Ubuntu’s Default DNS with Dnsmasq: Fixing Stale IPs in My Home Network
- Why my nightly rsync backup disappeared after switching to a systemd timer – and how I fixed it with 'Persistent=true