Rollover and index lifecycle management

Key points

• Logs and metrics go into a series of indices, never into one index that is expected to grow forever.
• A data stream hides that series behind one name and always writes to the newest backing index.
• Rollover starts a fresh backing index once a condition such as max_primary_shard_size is met.
• ILM then walks the index through hot, warm, cold, frozen and delete phases based on its age.
• Only the hot phase can roll over, and only the delete phase actually gives the disk space back.
• ILM re-evaluates policies every 10 minutes by default via indices.lifecycle.poll_interval.

Example

PUT _ilm/policy/logs-policy
{
  "policy": { "phases": {
    "hot":    { "actions": { "rollover": {
                  "max_primary_shard_size": "50gb", "max_age": "7d" } } },
    "warm":   { "min_age": "7d",  "actions": { "forcemerge": { "max_num_segments": 1 } } },
    "cold":   { "min_age": "30d", "actions": { "searchable_snapshot":
                  { "snapshot_repository": "backups" } } },
    "delete": { "min_age": "90d", "actions": { "delete": {} } }
  } }
}

Roll over on primary shard size, and remember only the delete phase reclaims disk.