I was running a hobbyist homelab on a Raspberry Pi 4 with 32 GB of eMMC storage. The Pi was a low‑cost, low‑power server that hosted a few services: a personal web server, a GitLab runner, and a Docker build farm for CI pipelines. The build farm was a simple script that pulled the latest code, built a Docker image, and pushed it to a registry. Everything worked until the disk filled up during a large multi‑stage build.
Below is a step‑by‑step walk‑through of what happened, how I identified the root cause, and the practical fixes I applied. The goal is to give you a concrete playbook for avoiding and recovering from disk‑full scenarios on small devices.
Why the disk filled up
Docker stores layers in /var/lib/docker. On a Raspberry Pi, the default storage driver is overlay2, which keeps each layer as a separate directory. A typical multi‑stage build can generate dozens of layers, each a few megabytes, but when you add temporary build artifacts, cache directories, and the base image layers, the cumulative size can exceed the available space quickly.
In my case, the build script had a --cache-from flag that pulled a large base image from the registry. The base image itself was 1.2 GB. The build added a node_modules directory that grew to 600 MB, and the final image was 1.5 GB. The overlay filesystem kept all intermediate layers, so the total consumption hit 4 GB, which is a lot when you only have 32 GB and the OS, swap, and other services already occupy ~10 GB.
Spotting the problem early
The first sign was the df -h output:
Filesystem Size Used Avail Use% Mounted on
/dev/mmcblk0p1 32G 28G 4G 88% /
The root partition was 88 % full. The Docker daemon was still running, but any new layer creation failed with no space left on device. Docker logs showed:
Error response from daemon: failed to create overlay mount: no space left on device
A quick docker system df revealed that the image layers were the largest contributor:
Images space usage: 4.2 GB
Containers space usage: 0.0 GB
Local volumes space usage: 0.0 GB
Build cache space usage: 0.0 GB
The docker system df command is a handy diagnostic tool that shows where Docker is eating space.
Immediate recovery steps
-
Stop the Docker daemon to prevent further writes.
sudo systemctl stop docker -
Clean up unused images and containers. Even though the build was failing, there were old images lingering from previous runs.
sudo docker image prune -a -f sudo docker container prune -f -
Clear the build cache. Docker keeps a cache of intermediate layers that can be purged safely if you’re not using them for incremental builds.
sudo docker builder prune -f -
Check the overlay filesystem for orphaned directories. Sometimes Docker leaves dangling directories if a build is interrupted.
sudo find /var/lib/docker/overlay2 -type d -empty -delete -
Free up space in
/var/lib/dockerby moving the directory to a larger partition (if available) or by mounting a separate SD card for Docker storage.sudo systemctl stop docker sudo mv /var/lib/docker /mnt/extra/docker sudo ln -s /mnt/extra/docker /var/lib/docker sudo systemctl start dockerThis approach is useful if you have a second SD card or an external USB drive. The Pi’s SD card is limited, but a USB 3.0 drive can provide several terabytes of free space.
Long‑term fixes
1. Use a dedicated Docker storage location
Move Docker’s root directory to a separate partition or external drive. In /etc/docker/daemon.json add:
{
"data-root": "/mnt/docker"
}
Then restart Docker. This keeps the OS partition cleaner and allows you to allocate more space to Docker.
2. Limit the size of the build cache
Docker’s build cache can grow unbounded. Set a maximum size with the --max-size flag when you run docker buildx:
docker buildx build --max-size 500m -t myapp:latest .
Alternatively, configure the builder:
docker buildx create --name mybuilder --driver docker-container --use
docker buildx inspect --bootstrap
Then add a --build-arg BUILDKIT_CACHE_MAX_SIZE=500m to your build script.
3. Use multi‑stage builds wisely
Multi‑stage builds help keep the final image small, but they still create intermediate layers. If you’re building a large language runtime, consider pulling the runtime as a base image instead of building it from source. For example:
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
FROM node:20-alpine
WORKDIR /app
COPY --from=builder /app/dist ./dist
CMD ["node", "dist/index.js"]
The builder stage is discarded after the final image is built, but the layers still consume space during the build. Using a smaller base image for the builder can reduce the temporary footprint.
4. Prune Docker automatically
Set up a cron job that runs docker system prune -af every night. Combine it with a check that the disk usage is above a threshold:
#!/usr/bin/env bash
USAGE=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
if
See also
- Using a systemd path unit to clean my home server’s cache when disk space hits 90%
- What a half‑open TCP connection in “ss” really means (and how to close it)
- How I Stopped /var/log from Flooding the Root Filesystem with Systemd Journal Rotations
- Fixing the “root=UUID not found” error after cloning an SSD with Clonezilla
- Pulling all failed SSH login attempts from auth.log in under a minute with awk