Monitoring and Troubleshooting

Build the mental model

Lesson 3 introduced `_cluster/health` as a readiness check (a dashboard warning light), but production monitoring needs a deeper vocabulary than just glancing at a status color — you need to be able to diagnose root causes. Unassigned shards (one of the most common causes of `status: red`) happen when a node failure, disk space exhaustion, or allocation setting misconfiguration prevents a primary or replica shard from being assigned to any node, and the `GET /_cluster/allocation/explain` API explains the specific reason assignment failed. To diagnose slow queries, you can enable Elasticsearch's "slow log" feature (logging queries that cross a threshold), and adding `profile: true` to a `_search` request shows an execution time breakdown by phase (query, fetch, aggregation) — Lesson 17's non-cacheable `must` clause tradeoff, Lesson 18's leading wildcard cost, and Lesson 12's deep pagination cost can all show up here as an actual production symptom (a slow query). Out-of-memory issues mostly come from not configuring the JVM heap size large enough for your data volume and aggregation complexity, or from letting field data cache (an in-memory structure needed for aggregations and sorting) run uncontrolled on an index with too many fields — Lesson 18's mapping explosion problem directly connects to memory pressure too. Think of monitoring as leveling up from a car's dashboard (readiness) to a mechanic's diagnostic toolkit (root-cause investigation) — beyond glancing at the warning light, you need a toolset to actually open the hood.

Connect it to a real scenario

When an alert fires because the Tutorial Platform's production cluster `_cluster/health` status is `red`, first run `GET /_cluster/allocation/explain` to find which shard failed to assign and why — if disk space is exhausted, expand that node's storage right away. If search API response times are becoming inconsistent, enable the production slow log to capture which query patterns (deep pagination, leading wildcard) are responsible, and drill further with `profile: true` on any slow query found. Monitor the JVM heap usage metric on a dashboard and alert when it crosses a threshold (say 85%) — a field-count metric can also connect back to investigating whether an index has developed mapping explosion.

Unassigned shards are a common cause of red status; find the root cause with allocation/explain
Diagnose slow queries phase-by-phase with the slow log and profile: true
Monitor JVM heap and field data cache to prevent OOM issues proactively

Try the working example

http

GET /_cluster/allocation/explain
{
  "index": "tutorials",
  "shard": 0,
  "primary": false
}

GET /tutorials/_search
{
  "profile": true,
  "query": { "match": { "body": "redis" } }
}

GET /_cat/thread_pool/search?v

You should see

You get a response explaining the specific reason for an unassigned replica shard, and an execution-time breakdown for each query phase.

5-minute try-it

Suppose you receive an alert that `_cluster/health` status is `red` — if the `allocation/explain` result shows the reason "disk watermark exceeded", list the troubleshooting steps you would take next.

One important caution

Assuming a `_cluster/health` status of `yellow` or `red` resolved itself just because it flashed by, without running `allocation/explain` to find the root cause — the underlying issue (disk space, node failure) can still recur.

Guessing what's slow and changing code without first checking the slow log or profile data — optimizing without knowing the real root cause fails to fix the actual problem.

Elasticsearch Guide — Troubleshooting — Elastic