How an Elasticsearch cluster actually fails

Key points

• Yellow means all primaries are assigned but some replicas are not - the data is complete and redundancy is reduced. Red means a primary is missing, so some data cannot be read at all.
• A single-node cluster is permanently yellow if any index has a replica, because a replica cannot be allocated to the same node as its primary. That is expected rather than a fault.
• The disk watermarks catch people out. At 85% Elasticsearch stops allocating new shards to a node; at 90% it moves shards away; at 95% it sets every index on that node to read-only.
• That flood-stage flag is not cleared automatically when disk is freed. Writes keep failing until you explicitly reset index.blocks.read_only_allow_delete - a very common surprise.
• Split brain was solved in 7.x: the cluster now uses a proper quorum and voting configuration rather than a hand-set minimum master count, so a partition cannot elect two masters.
• The master node handles cluster state, and state size grows with shards, indices and fields. Most cluster instability traces back to too much state rather than too much data.

Example

# What is wrong, and specifically which index?
GET /_cluster/health?level=indices
GET /_cat/indices?v&health=red

# The one API that explains an unassigned shard - it gives a reason.
GET /_cluster/allocation/explain
{ "index": "orders-000007", "shard": 0, "primary": true }
//   "explanation": "cannot allocate because all found copies are
//                   either stale or corrupt"

---
# DISK WATERMARKS - and the trap in the third one.
PUT /_cluster/settings
{ "persistent": {
  "cluster.routing.allocation.disk.watermark.low":         "85%",
  "cluster.routing.allocation.disk.watermark.high":        "90%",
  "cluster.routing.allocation.disk.watermark.flood_stage": "95%"
} }

#   85%  no NEW shards allocated here
#   90%  existing shards moved AWAY
#   95%  every index on the node -> READ-ONLY

# And the flag does NOT clear itself when disk is freed:
PUT /*/_settings
{ "index.blocks.read_only_allow_delete": null }
#   Symptom: disk was cleaned up hours ago and writes still fail with
#   "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]"

---
# Master stability: state size, not data size.
GET /_cluster/stats?filter_path=indices.shards.total,indices.mappings
GET /_cat/master?v
GET /_nodes/stats/jvm?filter_path=**.mem.heap_used_percent

#   Sustained master heap above ~75% means cluster state is too large.
#   The fix is fewer shards and fewer fields, not a bigger master.

---
# Dedicated roles for anything beyond a small cluster:
#   node.roles: [ master ]           3 of them, small, no data
#   node.roles: [ data_hot ]         fast disks, recent data
#   node.roles: [ data_warm ]        cheaper disks, older data
#   node.roles: [ ingest ]           pipeline processing
#
# Mixing master and data roles is what turns a heavy query into a
# cluster-wide outage, because the master stops responding.

Red means a missing primary, and the 95% flood-stage read-only flag never clears itself - you must reset it after freeing disk.