The mystery of the reboot crash
I had a single‑board server running Debian 12 in my living‑room. Every time I rebooted it, the Docker Compose stack that powers my home‑lab services would fail to start, and the machine would hang in a boot loop. The logs were a jumble of “failed to start” messages, but the root cause turned out to be a mis‑configured systemd unit. Below is the exact process I followed to diagnose and fix the issue, with a few trade‑offs and security notes that apply to any Docker‑based deployment on a small server.
1. Reproduce the failure
The first step is to confirm the problem under controlled conditions. After a clean reboot, run:
systemctl status myapp.service
You’ll see something like:
● myapp.service - Docker Compose stack for MyApp
Loaded: loaded (/etc/systemd/system/myapp.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2026‑09‑25 10:12:43 UTC; 1s ago
Process: 1234 ExecStart=/usr/local/bin/docker compose -f /srv/myapp/docker-compose.yml up (code=exited, status=1/FAILURE)
The key is the ExecStart line. If the command exits with a non‑zero status, systemd marks the unit as failed. The next step is to dig into the logs.
2. Inspect journal logs
Systemd’s journal stores the output of the service’s ExecStart command. Pull the relevant entries with:
journalctl -u myapp.service --since "5 minutes ago"
Typical output for a broken Compose deployment looks like:
docker compose -f /srv/myapp/docker-compose.yml up
[+] Running 3/3
ERROR: Service 'db' failed to start: OCI runtime create failed: container_linux.go:348: starting container process caused: exec: "postgres": executable file not found in $PATH
The error tells us that the docker compose command ran, but the container image failed to start because the binary inside the image wasn’t found. That usually means the image wasn’t pulled or the Compose file references a wrong image name.
3. Verify the Compose file
Open /srv/myapp/docker-compose.yml and check the image: fields. A common typo is postgres instead of postgres:15. If the image tag is omitted, Docker will pull the latest tag, which might change unexpectedly. Pin the tag:
services:
db:
image: postgres:15
restart: unless-stopped
environment:
POSTGRES_PASSWORD: example
After correcting the Compose file, test it manually:
cd /srv/myapp
docker compose up --no-start
If the command exits cleanly, the Compose file is syntactically correct.
4. Re‑write the systemd unit
A broken unit often contains a stale ExecStart line or missing Restart= directive. Here’s a minimal, robust unit:
[Unit]
Description=Docker Compose stack for MyApp
Requires=docker.service
After=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/srv/myapp
ExecStart=/usr/local/bin/docker compose -f docker-compose.yml up -d
ExecStop=/usr/local/bin/docker compose -f docker-compose.yml down
Restart=on-failure
RestartSec=5
User=myapp
Group=myapp
EnvironmentFile=/srv/myapp/.env
# Security hardening
PrivateTmp=yes
ProtectSystem=full
ProtectHome=yes
ReadOnlyDirectories=/srv/myapp
NoNewPrivileges=yes
[Install]
WantedBy=multi-user.target
Why these options?
| Option | Reason |
|---|---|
Type=oneshot + RemainAfterExit=yes |
The service starts once and stays “active” while containers run. |
Restart=on-failure |
If Docker exits with a non‑zero code, systemd will retry. |
PrivateTmp=yes |
Gives the service its own /tmp, preventing accidental leakage. |
ProtectSystem=full |
Mounts the filesystem as read‑only except for /var. |
ProtectHome=yes |
Prevents the service from accessing other users’ home directories. |
ReadOnlyDirectories=/srv/myapp |
The Compose files are immutable; Docker can still write to /srv/myapp/volumes. |
NoNewPrivileges=yes |
Disallows the service from gaining extra privileges via setuid. |
If you’re running on a system with SELinux or AppArmor, you’ll need to adjust the policy to allow Docker to read the Compose file. On Debian, the default Docker policy already permits this, but on Fedora you may need to add a custom rule.
5. Reload and enable the unit
After editing the unit, reload systemd and enable the service:
sudo systemctl daemon-reload
sudo systemctl enable myapp.service
sudo systemctl start myapp.service
Check the status again:
systemctl status myapp.service
You should see “active (exited)” for the unit and “running” for each container in docker ps.
6. Test the reboot loop
Reboot the machine:
sudo reboot
Once back online, confirm that the containers are up:
docker compose -f /srv/myapp/docker-compose.yml ps
If everything is running, the crash loop is fixed.
7. Common pitfalls and trade‑offs
| Pitfall | Fix | Trade‑off |
|---|---|---|
Using ExecStart=/usr/bin/docker-compose |
Switch to the newer docker compose CLI bundled with Docker Engine. |
docker-compose |
See also
- Freeing a Full `/var/log` on a Home Lab Server with `journalctl --vacuum-time` and a One‑Line Bash Cleanup
- Why I keep a weekly backup copy of my home directory on an external SSD using rsync and how I test the restore
- Why ssh keeps spawning new processes on every connect and how ControlPersist solves it
- Repairing a broken initramfs that locked my Raspberry Pi in emergency mode after a kernel update
- How I added ionice ‑c3 to my nightly rsync backup and finally stopped my laptop from overheating