Shard sizing and oversharding

Key points

• A shard is a complete Lucene index with its own memory, file handles and merge threads. Shards are not free, and the cost is per shard rather than per document.
• Primary shard count is fixed at index creation. Changing it means reindexing, which is why this is the one number worth thinking about up front.
• Oversharding is the common failure. Hundreds of tiny shards consume heap on the master for cluster state, multiply merge work, and make every search fan out to more places than necessary.
• A workable target is 10 to 50GB per shard, and roughly 20 shards or fewer per GB of heap on a node. Below a few GB, a shard is usually costing more than it earns.
• For time-series data use rollover rather than guessing: one index per period or size, with an alias in front, so shard count grows with data instead of being predicted.
• A search hits every shard of the target indices, so a query across 500 daily indices is 500 shard searches. Index lifecycle management exists to keep that number bounded.

Example

# Fixed at creation. Choose it deliberately.
PUT /orders-000001
{
  "settings": {
    "number_of_shards": 3,          "index.number_of_replicas": 1,
    "refresh_interval": "30s"       // 1s default: raise it for write-heavy
  }
}

# What is actually there? Sort by size and look for tiny shards.
GET /_cat/shards?v&h=index,shard,prirep,state,docs,store,node&s=store:desc

# The health check that matters:
GET /_cat/nodes?v&h=name,heap.percent,ram.percent,cpu,load_1m,disk.used_percent
GET /_cluster/health?level=indices

---
# SIZING RULES OF THUMB:
#
#   shard size          10-50 GB          smaller wastes overhead
#   shards per node     <= 20 per GB heap  30GB heap -> ~600 shards MAX
#   heap                <= 31 GB           above that compressed oops
#                                          is lost and pointers grow
#
# OVERSHARDING, concretely:
#   500 daily indices x 5 shards x 2 (replica) = 5,000 shards
#   for maybe 50 GB of data - about 10 MB per shard.
#   Cluster state alone will destabilise the master node.

---
# Time-series: roll over instead of predicting.
PUT /_ilm/policy/orders-policy
{
  "policy": { "phases": {
    "hot":    { "actions": { "rollover": {
                  "max_primary_shard_size": "30gb", "max_age": "7d" } } },
    "warm":   { "min_age": "7d",  "actions": {
                  "shrink": { "number_of_shards": 1 },
                  "forcemerge": { "max_num_segments": 1 } } },
    "delete": { "min_age": "90d", "actions": { "delete": {} } }
  } }
}

# Write to an alias, never to a concrete index - rollover repoints it.
POST /orders/_doc     # alias -> orders-000007, and moves on rollover

Primary shard count cannot be changed without reindexing, and hundreds of tiny shards will destabilise a cluster faster than too few large ones.