Project: System Health-Check Script

Build the mental model

This project is less about any single check and more about the framework that holds many checks together, and it introduces a pattern this tutorial hasn't used yet: a global variable, worst_status, that accumulates state across multiple function calls instead of each function being self-contained. The report() helper does two jobs at once — it prints a consistently formatted line with printf, and it updates worst_status using a 'high-water mark' rule: a case statement bumps worst_status to 1 for WARNING or 2 for CRITICAL, but only if the new value is higher than what's already recorded, via (( worst_status < 1 )) and (( worst_status < 2 )). That guard is what lets one CRITICAL result from check_port permanently override an earlier OK from check_process without a later OK ever downgrading it back. The checks array holds function names as plain strings, and "$check" in the for loop invokes each one by name — calling a function stored in a variable is what makes adding a fourth check later as simple as writing the function and appending its name to the array, with zero changes to the loop itself. Finally, exit "$worst_status" turns the script's internal state into a process exit code, the universal interface every monitoring tool, cron wrapper, and shell pipeline already knows how to read. Each check here is deliberately hardcoded rather than calling real df, pgrep, or ss commands, so the framework's control flow — how one bad result survives to the end — is what's being verified, not the OS state of any particular machine.

Connect it to a real scenario

Monitoring systems like cron-triggered alerting, Nagios, or a simple uptime dashboard all need the same basic contract from a health-check script: run some checks, print something human-readable, and exit nonzero the moment anything is seriously wrong. Follow the actual execution: checks=(check_disk_usage check_process check_port) lists three function names, and the for loop calls each one in turn. check_disk_usage hardcodes used_percent=92, which is above its own 90 threshold, so it calls report "disk" "CRITICAL" "..." — report prints the line and, since 2 > 0, raises worst_status from 0 to 2. check_process hardcodes running="yes" and reports OK — worst_status stays at 2 because OK never lowers it. check_port hardcodes listening="no" and reports CRITICAL again — worst_status is already 2, so the (( worst_status < 2 )) guard leaves it unchanged, but the individual CRITICAL line still prints. After the loop, the final case statement reads worst_status=2 and prints 'Overall: CRITICAL', and exit "$worst_status" sends exit code 2 back to whatever invoked the script. This is exactly the Nagios plugin convention (0/1/2 for OK/WARNING/CRITICAL), which is why cron or a monitoring dashboard watching this script's exit code can page someone automatically without parsing any text at all. Without this framework, a naive script that just prints pass/fail lines and always exits 0 would look identical in a terminal but would never trigger an alert — the disk filling up or nginx going down would only be noticed by someone manually reading logs, often long after the outage already started.

• A shared report() helper keeps status formatting and the worst-status tracking in one place
• Exit codes (0/1/2 style) let a health-check script plug directly into cron or monitoring tools
• Looping over an array of check functions makes adding a new check a one-line change

Try the working example

#!/usr/bin/env bash
set -euo pipefail

# Overall worst status seen so far: 0=OK, 1=WARNING, 2=CRITICAL
worst_status=0

report() {
  local name="$1"
  local status="$2"   # OK | WARNING | CRITICAL
  local detail="$3"
  printf "[%-8s] %-16s %s\n" "$status" "$name" "$detail"

  case "$status" in
    WARNING)
      if (( worst_status < 1 )); then worst_status=1; fi
      ;;
    CRITICAL)
      if (( worst_status < 2 )); then worst_status=2; fi
      ;;
  esac
}

# --- Individual checks ---
# In a real script, each of these would run an actual command
# (df, pgrep, a port check, etc). Here they return hardcoded fake
# values so the framework logic below is deterministic to demonstrate.

check_disk_usage() {
  # In a real script: read the used% from `df -h /` and compare to thresholds
  local used_percent=92
  if (( used_percent >= 90 )); then
    report "disk" "CRITICAL" "root partition at ${used_percent}% used"
  elif (( used_percent >= 75 )); then
    report "disk" "WARNING" "root partition at ${used_percent}% used"
  else
    report "disk" "OK" "root partition at ${used_percent}% used"
  fi
}

check_process() {
  # In a real script: running=$(pgrep -x nginx > /dev/null && echo yes || echo no)
  local running="yes"
  if [[ "$running" == "yes" ]]; then
    report "nginx" "OK" "process is running"
  else
    report "nginx" "CRITICAL" "process is not running"
  fi
}

check_port() {
  # In a real script: check with `ss -ltn` or a TCP probe against the port
  local listening="no"
  if [[ "$listening" == "yes" ]]; then
    report "port:5432" "OK" "port is accepting connections"
  else
    report "port:5432" "CRITICAL" "port is not listening"
  fi
}

echo "System Health Check"
echo "===================="

checks=(check_disk_usage check_process check_port)
for check in "${checks[@]}"; do
  "$check"
done

echo "===================="
case "$worst_status" in
  0) echo "Overall: OK" ;;
  1) echo "Overall: WARNING" ;;
  2) echo "Overall: CRITICAL" ;;
esac

exit "$worst_status"

System Health Check
====================
[CRITICAL] disk             root partition at 92% used
[OK      ] nginx            process is running
[CRITICAL] port:5432        port is not listening
====================
Overall: CRITICAL

Nagios Plugin Development Guidelines — Exit Codes — Bash / Shell Scripting