When a VPS Disk Fills Mid‑Night: Quick Recovery with LVM Snapshots and Automated Alerts

Disk‑full crashes are the most common reason a VPS goes from “alive” to “dead” overnight.
A log file or a database that blows past the allotted space will make the kernel refuse any further writes, and every process that needs to touch the disk will start failing. If you’re running a website, mail server, or database, that’s a service outage you can’t afford.

The silver lining? A handful of LVM tricks and a tiny monitoring script can get you back up in minutes and keep future incidents at bay.
Below I walk through a real‑world scenario on a Debian‑based VPS, but the ideas carry over to any distro that uses LVM and systemd.

Why LVM?
LVM lets you snap the current state of a volume group in seconds.
A snapshot is a read‑only copy of the data at that exact moment.
If the disk fills, you can roll back to the snapshot and restore a known‑good state without touching the live data.
It’s faster and safer than pulling a backup and restoring it, especially when a runaway process has already corrupted the filesystem.

1. The Problem in Action

Imagine a 20 GB VPS running a small web app.
The app writes to /var/log/app.log.
A recent feature change turned the log into a 1 GB per hour behemoth.
At 2 a.m. the disk hits 100 % and the web server stops responding.
You SSH in at 3 a.m. and see:

$ df -h
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/vg0/lv_root  20480000 20480000          0 100% /

You can read files, but you can’t write anything.
The classic “delete the log file” trick won’t work until you free space.

2. Quick Recovery with an LVM Snapshot

2.1. Inspect the LVM layout

$ sudo vgdisplay vg0
  --- Volume group ---
  VG Name               vg0
  System ID             12345678-90ab-cdef-1234-567890abcdef
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  Max LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               20.00 GiB
  PE Size               4.00 MiB
  Total PE              5120
  Alloc PE / Size       5120 / 20.00 GiB
  Free  PE / Size       0 / 0

The root logical volume is /dev/vg0/lv_root. We’ll snapshot it.

2.2. Create a snapshot

$ sudo lvcreate --size 1G --snapshot --name lv_root_snap /dev/vg0/lv_root
  Logical volume lv_root_snap created.

A 1 GB snapshot is enough to hold the current state.
It’s read‑only, so you can mount it safely to inspect the data.

2.3. Mount the snapshot

$ sudo mkdir /mnt/snap
$ sudo mount /dev/vg0/lv_root_snap /mnt/snap

Now copy the log out of the snapshot to free space on the live system:

$ sudo cp /mnt/snap/var/log/app.log /tmp/app.log.2026-11-02
$ sudo rm /var/log/app.log
$ sudo sync

Because the snapshot keeps the old data intact, the rm now succeeds and the live filesystem is writable again.

2.4. Verify and unmount

$ df -h
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/vg0/lv_root  20480000  20400000   8000000  80% /

The disk is no longer full. Restart the web server:

$ sudo systemctl restart nginx

If the service still fails, check the logs.
Often the app is still writing to the old log file path; you may need to rotate logs or tweak the config.

2.5. Clean up the snapshot

$ sudo umount /mnt/snap
$ sudo lvremove /dev/vg0/lv_root_snap

Snapshots consume space, so delete them once you’re done.

3. Preventing the Next Incident

A snapshot fix is fast, but you don’t want the same thing to happen again.
Two complementary strategies help:

1. Log rotation – keep log files from growing unchecked.
2. Automated alerts – get a heads‑up before the disk reaches 100 %.

3.1. Log Rotation with logrotate

Debian ships logrotate by default.
A typical config for your app might look like:

# /etc/logrotate.d/app
/var/log/app.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    create 0640 www-data adm
    postrotate
        systemctl reload app.service > /dev/null 2>&1 || true
    endscript
}

• daily – rotate every day.
• rotate 7 – keep seven old copies.
• compress – gzip old logs.
• postrotate – reload the app so it starts writing to a fresh file.

Test the config:

$ sudo logrotate -d /etc/logrotate.d/app

The -d flag shows what would happen without actually rotating.
Once satisfied, run:

$ sudo logrotate -f /etc/logrotate.d/app

3.2. Disk‑Space Alerts with systemd‑path and systemd‑timer

A lightweight way to get notified is to use systemd units that run a small script every hour.
The script checks disk usage and sends an email if the threshold is exceeded.

1. Create the script

# /usr/local/bin/check_disk.sh
#!/usr/bin/env bash
THRESHOLD=90
USAGE=$(df / | awk 'NR==2{print $5}' | tr -d '%')
if [ "$USAGE" -ge "$THRESHOLD" ]; then
    echo "Disk usage on / is $USAGE% – please check!" | mail -s "Disk alert" root
fi

Make it executable:

$ sudo chmod +x /usr/local/bin/check_disk.sh

2. Create a systemd service

# /etc/systemd/system/check-disk.service
[Unit]
Description=Check disk usage

[Service]
Type=oneshot
ExecStart=/usr/local/bin/check_disk.sh

3. Create a timer

# /etc/systemd/system/check-disk.timer
[Unit]
Description=Run disk check hourly

[Timer]
OnCalendar=*-*-* 00:00:00
OnCalendar=*-*-* 01:00:00
OnCalendar=*-*-* 02:00:00
OnCalendar=*-*-* 03:00:00
OnCalendar=*-*-* 04:00:00
OnCalendar=*-*-* 05:00:00
OnCalendar=*-*-* 06:00:00
OnCalendar=*-*-* 07:00:00
OnCalendar=*-*-* 08:00:00
OnCalendar=*-*-* 09:00:00
OnCalendar=*-*-* 10:00:00
OnCalendar=*-*-* 11:00:00
OnCalendar=*-*-* 12:00:00
OnCalendar=*-*-* 13:00:00
OnCalendar=*-*-* 14:00:00
OnCalendar=*-*-* 15:00:00
OnCalendar=*-*-* 16:00:00
OnCalendar=*-*-* 17:00:00
OnCalendar=*-*-* 18:00:00
OnCalendar=*-*-* 19:00:00
OnCalendar=*-*-* 20:00:00
OnCalendar=*-*-* 21:00:00
OnCalendar=*-*-* 22:00:00
OnCalendar=*-*-* 23:00:00

[Install]
WantedBy=timers.target

The timer runs the service every hour. Enable it:

$ sudo systemctl enable --now check-disk.timer

Now you’ll get an email if the root filesystem exceeds 90 % usage. Adjust the threshold as needed.

4. Advanced: Using LVM Snapshots for Full System Rollback

If the disk‑full event was caused by a database corruption or a misbehaving app that left the filesystem in an inconsistent state, you might want to roll back the entire system to a known‑good snapshot.

4.1. Take a pre‑deployment snapshot

Before deploying a new version, create a snapshot:

$ sudo lvcreate --size 2G --snapshot --name predeploy /dev/vg0/lv_root

Keep it for a few days. If something goes wrong, you can revert:

$ sudo lvconvert --merge /dev/vg0/lv_root_snap

The merge operation replaces the live volume with the snapshot.
It must be done while the system is offline, so you’ll need to reboot into rescue mode or use a live USB.

4.2. Automate snapshot creation with cron

Add a cron job that creates a snapshot nightly:

# crontab -e
0 3 * * * /usr/local/bin/backup_snapshot.sh

backup_snapshot.sh:

#!/usr/bin/env bash
SNAP="/dev/vg0/lv_root_snap"
SIZE="2G"
NAME="root_snap_$(date +%Y%m%d)"
if lvdisplay "$SNAP" >/dev/null 2>&1; then
    lvremove -y "$SNAP"
fi
lvcreate --size "$SIZE" --snapshot --name "$NAME" /dev/vg0/lv_root

This keeps a clean snapshot history without manual intervention.

5. Caveats and Troubleshooting

6. Security Considerations

• Snapshot permissions – Snapshots inherit the same permissions as the original volume. Make sure only trusted users can create or delete them.
• Email alerts – Use a local MTA that requires authentication to prevent spamming.
• Log files – Keep them on a separate partition if possible. This isolates them from the root filesystem and prevents a log flood from filling the entire system.

7. Bottom Line

1. Snap the root volume with LVM.
2. Mount the snapshot, copy out the offending file(s), delete the snapshot.
3. Set up logrotate to keep logs from blowing up.
4. Deploy a lightweight disk‑usage alert with systemd.
5. (Optional) Automate nightly snapshots for a full rollback safety net.

With these steps you can recover from a midnight disk‑full event in minutes and dramatically reduce the chance of a repeat. LVM snapshots and simple monitoring give you a powerful, low‑maintenance safety net—exactly what a small‑scale VPS or homelab server needs.

See also

• Rootless Podman volumes: how to keep data safe across upgrades without root
• Fixing ProxyJump timeouts when a VPN drops your IP address.
• How to Keep a Linux Laptop Connected to a VPN When the Router Switches Between DHCP and Static IPs
• Testing Restic Incremental Backups on an Active MySQL Server Without Downtime
• Fixing the 2‑second DNS lookup lag on my Pi by tweaking resolvectl and systemd‑resolved cache