Project: Log File Analyzer

Build the mental model

This project shows how several small, single-purpose Unix tools compose into something more powerful than any one of them alone — the core skill this entire tutorial has been building toward. It combines four ideas from earlier chapters. First, functions: count_level() and top_error_messages() wrap logic that would otherwise be repeated or inlined, each taking its own local parameters so they can be reused for any level or any N without rewriting the body. Second, arrays: LEVELS holds the three log levels as a single list, and looping over it with a for loop means adding a fourth level later is a one-line change, not a copy-pasted block. Third, grep -c "$level" "$LOG_FILE" counts matching lines without printing them — cheaper and simpler than piping grep's output into wc -l. Fourth, and most important, the classic sort | uniq -c | sort -nr pipeline: sed strips the timestamp and level prefix so only the message text remains, the first sort groups identical messages together (uniq only collapses lines that are already adjacent, so this step is not optional), uniq -c counts each group, and the second sort -nr orders those counts numerically from highest to lowest so the most frequent failure surfaces first. head -n "$n" then trims that ranked list down to the top N. None of these four pieces is new on its own; what is new here is chaining them into a single script that turns raw, unstructured text into a structured, ranked report — the same pattern used constantly in real ops and data work.

Connect it to a real scenario

Ops and backend engineers spend a surprising amount of time staring at log files trying to answer two simple questions after something goes wrong: how bad was it, and what broke the most? Walk through what this script actually does: it first writes a small sample.log with a heredoc so the example is self-contained and reproducible, then defines LOG_FILE, TOP_N, and the LEVELS array as configuration up top. total_lines comes from wc -l < "$LOG_FILE" — using the redirect form rather than wc -l "$LOG_FILE" avoids the filename appearing in the output, so total_lines holds a clean number. The for loop then calls count_level for each level in turn and prints an aligned column with printf. Finally top_error_messages filters to ERROR lines only, strips everything before the message text with sed, and runs the sort/uniq/sort/head pipeline to surface the two most common failures — 'Database connection failed' (4 times) and 'Disk write failed' (2 times) in the sample data. In production, this is the difference between an on-call engineer scrolling through thousands of lines by hand during an incident — or worse, guessing — and running one command that immediately answers 'what is actually breaking'. The same script, pointed at a real log and scheduled via cron, becomes a daily or hourly summary report instead of a one-off investigation tool.

grep -c counts matching lines without printing them
sort | uniq -c | sort -nr is the standard pattern for ranking frequency in text
Looping over an array keeps repeated per-item logic in one place

Try the working example

bash

#!/usr/bin/env bash
set -euo pipefail

# Create a small sample log file so this example is fully self-contained
cat > sample.log <<EOF
2026-01-01 10:00:01 INFO Server started
2026-01-01 10:00:05 INFO User login: alice
2026-01-01 10:01:12 WARN High memory usage
2026-01-01 10:02:33 ERROR Database connection failed
2026-01-01 10:02:34 ERROR Database connection failed
2026-01-01 10:03:10 INFO User login: bob
2026-01-01 10:04:00 ERROR Disk write failed
2026-01-01 10:05:22 WARN High memory usage
2026-01-01 10:06:45 ERROR Database connection failed
2026-01-01 10:07:00 INFO User logout: alice
2026-01-01 10:08:15 ERROR Disk write failed
2026-01-01 10:09:59 ERROR Database connection failed
EOF

LOG_FILE="sample.log"
TOP_N=3
LEVELS=("INFO" "WARN" "ERROR")

count_level() {
  local level="$1"
  grep -c "$level" "$LOG_FILE"
}

top_error_messages() {
  local n="$1"
  grep "ERROR" "$LOG_FILE" | sed -E 's/^[0-9-]+ [0-9:]+ ERROR //' | sort | uniq -c | sort -nr | head -n "$n"
}

total_lines=$(wc -l < "$LOG_FILE")

echo "Log Analysis Report for $LOG_FILE"
echo "=================================="
echo "Total lines: $total_lines"

for level in "${LEVELS[@]}"; do
  printf "%-5s entries: %s\n" "$level" "$(count_level "$level")"
done

echo ""
echo "Top $TOP_N most frequent error messages:"
top_error_messages "$TOP_N"

You should see

Log Analysis Report for sample.log
==================================
Total lines: 12
INFO  entries: 4
WARN  entries: 2
ERROR entries: 6

Top 3 most frequent error messages:
      4 Database connection failed
      2 Disk write failed

5-minute try-it

Extend the script to also print the single most common WARN message using the same sort/uniq pattern, and add a --level flag that lets the caller filter the whole report to just one log level.

One important caution

Using grep -c "ERROR" without anchoring can also match lines where the word appears inside another word or a message, inflating counts — a more precise field-based match (e.g. with awk) avoids false positives.

Forgetting that uniq -c only collapses adjacent duplicate lines — the input must be sorted first, or identical messages that are not next to each other will be counted as separate groups.

GNU Coreutils Manual — uniq — Bash / Shell Scripting