Pull the top 10 HTTP status codes from an Nginx access log with awk

When you’re running a small site or a homelab server, the access log is the first place you look for clues about what’s happening. A quick glance at the most frequent status codes can tell you whether a recent change broke something, if a bot is hammering your API, or if a mis‑configured rewrite is returning 404s for legitimate pages.
Below is a practical, one‑liner‑heavy walk‑through that shows how to extract the top ten status codes from a standard Nginx log using only awk. It covers common pitfalls, performance tricks, and a few security‑related notes that fit naturally into a daily‑use workflow.

Why the top 10 matters

Health check – A sudden spike in 5xx codes usually means a backend failure.
Bot detection – A long list of 403s can indicate a crawler that’s being blocked.
Performance tuning – 304s and 200s give you a rough idea of cache hit ratios.
Audit trail – Knowing which status codes are most common helps you spot anomalies.

If you’re already using a log‑analysis stack (ELK, Loki, etc.), this awk trick is a lightweight sanity check that can run in a cron job or a one‑off script before you dive into a full‑blown dashboard.

The default Nginx log format

The default access_log format in /etc/nginx/nginx.conf is:

log_format main '$remote_addr - $remote_user [$time_local] '
                '"$request" $status $body_bytes_sent '
                '"$http_referer" "$http_user_agent"';

The status code is the 9th field when you split the line on spaces, but the $request field contains spaces itself, so you have to be careful. A robust awk solution uses the -F option to split on spaces and quotes, or simply relies on the fact that the status code is the 9th field in the default format.

One‑liner with awk

awk '{
    # $9 is the status code in the default format
    status[$9]++
}
END {
    # Print status code and count, sorted by count descending
    for (s in status)
        print status[s], s
}' /var/log/nginx/access.log | sort -nr | head -10

Explanation

status[$9]++ builds an associative array keyed by the status code.
In the END block, we iterate over the array and print count status.
The pipe to sort -nr orders by count (numeric, reverse).
head -10 limits the output to the top ten.

Running this against a typical log file yields something like:

Customizing field positions

If you changed the log format, the status code may not be $9. For example, with the combined format (which adds $http_cookie), the status code becomes $10. You can adjust the script:

awk '{ status[$10]++ } END { for (s in status) print status[s], s }' ...

Alternatively, use a more robust field separator that splits on quotes:

awk -F'\"' '
{
    # The status code is the 5th field after splitting on quotes
    split($3, a, " ")
    status[a[2]]++
}
END { for (s in status) print status[s], s }' ...

This method is resilient to changes in the request string.

Handling large logs

Streaming with awk

awk processes the file line by line, so it can handle gigabytes of data without loading the entire file into memory. However, the associative array grows with the number of distinct status codes, which is usually small (less than 20), so memory usage stays low.

Compressed logs

If your logs are rotated and compressed (access.log.1.gz, etc.), you can pipe them through zcat:

zcat /var/log/nginx/access.log.*.gz | awk '...'

This keeps disk I/O minimal and avoids manual decompression.

Security considerations

Log integrity – If you suspect tampering, compare the output of this awk script with a hash of the log file (sha256sum). A sudden change in the top status codes without a corresponding change in the hash may indicate malicious activity.
Access control – Restrict read permissions on /var/log/nginx/access.log to root or a dedicated monitoring user. chmod 640 /var/log/nginx/access.log and chown root:monitor /var/log/nginx/access.log are common patterns.
Log rotation – Ensure logrotate is configured to rotate logs before they grow too large. A typical entry in /etc/logrotate.d/nginx keeps the current log open and compresses the old ones.

Trade‑offs: awk vs other tools

awk – Fast, built‑in, no external dependencies. Good for quick checks or cron jobs.
grep + sort + uniq – Simpler syntax but less efficient for large files because uniq needs sorted input.
Python – More readable for complex parsing but adds an interpreter overhead.
Go or Rust – Excellent for production pipelines but overkill for a one‑liner.

For most homelab users, awk strikes the right balance.

Troubleshooting common pitfalls

Symptom	Likely cause	Fix
No output	Log file empty or wrong path	Verify `/var/log/nginx/access.log` exists and has data.
Wrong status code	Custom log format	Adjust `$9` to the correct field or use the quote‑based split.
Out‑of‑memory error	Extremely high number of unique status codes	Unlikely; check that the script isn’t mis‑parsing fields.
Slow performance on a 10 GB log	Disk I/O bottleneck	Run on a SSD or use `zcat` to avoid decompressing on the fly.

Putting it together in a cron job

# /etc/cron.daily/nginx_status_top10
0 2 * * * root /usr/local/bin/nginx_top10.sh

nginx_top10.sh:

#!/usr/bin/env bash
set -euo pipefail

LOG=/var/log/nginx/access.log
OUT=/var/log/nginx/top10_status.log

awk '{
    status[$9]++
}
END {
    for (s in status)
        print status[s], s
}' "$LOG" | sort -nr | head -10 > "$OUT"

Now you have a daily snapshot you can email or push to a monitoring dashboard.

Wrap‑up

A single awk command can give you a quick pulse on your web server’s health. It’s lightweight, portable, and fits neatly into a self‑hosted or homelab environment. By tweaking the field index or the separator, you can adapt it to any log format you choose. And because the script is pure shell, you can audit it for security or embed it in larger automation without pulling in heavy dependencies.

Pulling the top 10 HTTP status codes from an Nginx access log with awk