Disk exhaustion is the most common cause of a sudden “500 Internal Server Error” on a small VPS.
When the filesystem that holds /var/log/apache2 is full, Apache can’t rotate or write new logs, and the web server silently stops accepting requests.
The fix is usually a quick cleanup, but doing it manually every time is error‑prone and can leave you with the same problem again.
Below is a practical, repeatable script that:
- Detects when the root filesystem is below a configurable threshold.
- Safely removes old log files while keeping the most recent ones.
- Rotates Apache logs if they are still writable.
- Sends a notification to a Slack webhook (or any other webhook) so you’re aware before the disk fills completely.
The script is written for Ubuntu 24.04, but the concepts apply to any Debian‑based system.
It uses only tools that ship with a minimal Ubuntu install (find, du, gzip, systemctl, curl) and a handful of optional utilities (logrotate, jq).
Feel free to adapt the paths or thresholds to your own setup.
1. Why the “Out of Disk” Problem Is Worse Than It Looks
When /var runs out of space, Apache can’t write to its error or access logs.
Because the logs are a required part of the HTTP transaction, Apache aborts the request and the client sees a 500 error.
The web server keeps running, but the logs are corrupted, and the next request that tries to write a log entry will fail again.
If you’re running a site that relies on log‑based analytics or security monitoring, you’ll lose data for days.
The root cause is usually one of:
| Cause | Typical Symptoms | Quick Fix |
|---|---|---|
| Unbounded log growth | /var/log/apache2/*.log > 1 GB |
Rotate or truncate |
| Large static file uploads | tmp/ or /var/tmp full |
Clean old uploads |
| Mis‑configured backup scripts | ~/backup/ growing |
Delete old backups |
| Malware or bot traffic | Sudden spike in log size | Block offending IPs |
The script below focuses on the most common case: log files that keep growing because log rotation is mis‑configured or missing.
2. Prerequisites
| Item | Why it matters | How to check |
|---|---|---|
logrotate |
Handles daily/weekly rotation of logs | `dpkg -l |
jq |
Optional, for parsing JSON in webhook payload | `dpkg -l |
curl |
To send notifications | curl --version |
systemd |
For checking Apache service status | systemctl status apache2 |
If any of these are missing, install them:
sudo apt update
sudo apt install -y logrotate jq curl
3. The Cleanup Script
Create a file /usr/local/bin/cleanup‑apache‑logs.sh and paste the following code.
Make it executable and add it to cron or systemd as described later.
#!/usr/bin/env bash
#
# cleanup-apache-logs.sh
#
# Detects low disk space on the root filesystem,
# rotates Apache logs if possible, and removes old
# log files that exceed a retention period.
#
# Author: mrtomlinux
# Date: 2026-11-03
#
# Usage: sudo /usr/local/bin/cleanup-apache-logs.sh
#
# Optional: set environment variables:
# LOG_RETENTION_DAYS=30
# DISK_THRESHOLD_PCT=10
# SLACK_WEBHOOK_URL="https://hooks.slack.com/services/..."
#
set -euo pipefail
# Configuration
LOG_RETENTION_DAYS=${LOG_RETENTION_DAYS:-30}
DISK_THRESHOLD_PCT=${DISK_THRESHOLD_PCT:-10}
LOG_DIR="/var/log/apache2"
SLACK_WEBHOOK_URL=${SLACK_WEBHOOK_URL:-""}
# Helper: send a Slack message
send_slack() {
local msg="$1"
if [[ -n "$SLACK_WEBHOOK_URL" ]]; then
curl -X POST -H 'Content-type: application/json' \
--data "{\"text\":\"$msg\"}" "$SLACK_WEBHOOK_URL" >/dev/null 2>&1 || true
fi
}
# 1. Check disk usage
USAGE_PCT=$(df --output=pcent "$LOG_DIR" | tail -n1 | tr -d '%')
if (( USAGE_PCT >= DISK_THRESHOLD_PCT )); then
echo "Disk usage is ${USAGE_PCT}%, below threshold ${DISK_THRESHOLD_PCT}% – nothing to do."
exit 0
fi
echo "Disk usage is ${USAGE_PCT}%, below threshold ${DISK_THRESHOLD_PCT}% – starting cleanup."
# 2. Ensure logrotate can run
if ! systemctl is-active --quiet apache2; then
echo "Apache is not running. Skipping log rotation."
else
echo "Running logrotate for Apache."
logrotate -f /etc/logrotate.d/apache2
fi
# 3. Delete old logs beyond retention
find "$LOG_DIR" -type f -name '*.log' -mtime +"$LOG_RETENTION_DAYS" -print -delete | while read -r file; do
echo "Deleted old log: $file"
done
# 4. Compress any remaining logs older than 7 days
find "$LOG_DIR" -type f -name '*.log' -mtime +7 ! -name '*.gz' -print -exec gzip {} \;
# 5. Verify that Apache can still write logs
if ! touch "$LOG_DIR/access.log" 2>/dev/null; then
echo "ERROR: Apache cannot write to $LOG_DIR after cleanup."
send_slack "🚨 Disk cleanup failed: Apache cannot write logs on $(hostname)."
exit 1
fi
echo "Cleanup completed successfully."
send_slack "✅ Disk cleanup finished on $(hostname). Disk usage now ${USAGE_PCT}%."
exit 0
What the script does
- Disk‑usage guard – If the root filesystem is still above the threshold, the script exits early.
This prevents accidental deletion when the disk is healthy. - Forced log rotation –
logrotate -fforces a rotation even if the log file is still open.
Apache will close the old file and open a new one automatically. - Retention purge – Deletes any
.logfile older thanLOG_RETENTION_DAYS.
Adjust the variable to keep more or fewer days. - Compression – Gzips logs older than 7 days to save space while keeping them accessible.
- Safety check – Attempts to touch the access log; if it fails, it alerts you.
This is a quick sanity check that Apache can still write logs.
4. Scheduling the Script
Option A: cron
Add a cron job that runs every day at 02:00 AM:
sudo crontab -e
Insert:
0 2 * * * /usr/local/bin/cleanup-apache-logs.sh >> /var/log/cleanup-apache.log 2>&1
This keeps a log of each run, which is useful for troubleshooting.
Option B: systemd timer
Create a unit file /etc/systemd/system/cleanup-apache-logs.service:
[Unit]
Description=Cleanup Apache logs when disk space is low
After=network.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/cleanup-apache-logs.sh
Then create a timer /etc/systemd/system/cleanup-apache-logs.timer:
[Unit]
Description=Run cleanup-apache-logs.service daily
[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
[Install]
WantedBy=timers.target
Enable it:
sudo systemctl daemon-reload
sudo systemctl enable --now cleanup-apache-logs.timer
The timer will trigger the service every day at 2 AM.
5. Tweaking the Threshold
The script uses a percentage threshold (DISK_THRESHOLD_PCT).
If you want to trigger cleanup earlier, set it to a lower value, e.g., 5.
If you prefer to act on absolute size, modify the script to compare df -h output against a fixed number of GB.
6. Common Pitfalls and How to Avoid Them
| Issue | Why it happens | Fix |
|---|---|---|
| Apache keeps failing after cleanup | Logrotate didn’t close the file because Apache had it open for a long time. | Run systemctl restart apache2 after rotation or add copytruncate to /etc/logrotate.d/apache2. |
| Script deletes logs that are still needed for compliance | Retention period too short. | Increase LOG_RETENTION_DAYS or move critical logs to a separate directory. |
| Slack notifications never arrive | Webhook URL missing or network blocked. | Verify the URL, test with curl -X POST ..., and check firewall rules. |
| Disk usage still high after cleanup | Other directories (e.g., /home, /var/tmp) are the real culprits. |
Extend the script to scan other mount points or use ncdu for interactive cleanup. |
| Script fails with “Permission denied” | Running as a non‑root user. | Ensure the script is owned by root and has sudo privileges or run it via sudo. |
7. Extending the Cleanup
If you host multiple services that write logs to /var/log, you can generalize the script:
- Add a list of log directories in a variable (
LOG_DIRS="/var/log/apache2 /var/log/nginx"). - Loop over each directory performing the same rotation and purge logic.
- Add a
--dry-runflag to preview deletions before actually removing files.
8. Security Considerations
- Least privilege – The script should run as root only because it touches system logs.
If you’re worried about accidental deletion, run it with a dedicated user andsudopermissions limited to the log directories. - Audit trail – Keep the script in version control (e.g., a local Git repo).
This lets you roll back changes if you accidentally delete something important. - Log integrity – Compressing old logs with
gzippreserves them while saving space.
If you need to audit older logs, you can decompress them on demand. - Webhook security – Use a secret token in the Slack webhook URL and keep it out of the script’s source code.
Store it in/etc/cleanup-apache-logs.envand source it at the top of the script.
9. Real‑World Example
A friend of mine runs a small blog on a 2 GB VPS.
Every month the log directory swells to 1.2 GB, and the server starts returning 500 errors.
After adding the script and scheduling it via cron, the disk usage never drops below 15 %.
The Slack notification keeps him informed, and the site stays online.
10. Quick Reference
| Command | Purpose |
|---|---|
df -h |
Check disk usage |
logrotate -d /etc/logrotate.d/apache2 |
Dry‑run log rotation |
find /var/log/apache2 -type f -mtime +30 -delete |
Delete logs older than 30 days |
gzip /var/log/apache2/*.log |
Compress logs |
systemctl status apache2 |
Verify Apache is running |
See also
- When a VPS Disk Fills Mid‑Night: Quick Recovery with LVM Snapshots and Automated Alerts
- Rootless Podman volumes: how to keep data safe across upgrades without root
- Fixing ProxyJump timeouts when a VPN drops your IP address.
- How to Keep a Linux Laptop Connected to a VPN When the Router Switches Between DHCP and Static IPs
- Testing Restic Incremental Backups on an Active MySQL Server Without Downtime