Aggregations, cardinality and circuit breakers

Key points

• Aggregations need doc values - a column-oriented structure built at index time. On a text field there are none, which is why aggregating text requires fielddata and why fielddata is disabled by default.
• Enabling fielddata on an analysed field loads every term into heap. It is the single most reliable way to OOM a data node, and the error message rarely makes the cause obvious.
• A terms aggregation with a large size is expensive: every shard returns its top N, the coordinator merges them, and memory grows with size multiplied by shard count.
• It is also approximate. Each shard reports only its own top terms, so a term that is ninth everywhere can be missed entirely - doc_count_error_upper_bound quantifies that.
• Circuit breakers stop a request before it kills the node. A CircuitBreakingException is the system working correctly; raising the limit to make it go away removes the protection.
• For high-cardinality work, use composite aggregation with paging rather than one huge terms bucket, and cardinality - which is HyperLogLog and approximate - instead of counting distinct values exactly.

Example

// Cheap: keyword field, doc values, bounded size.
{ "size": 0, "aggs": {
    "by_status": { "terms": { "field": "status", "size": 10 } } } }

// EXPENSIVE and approximate - 10,000 buckets from every shard.
{ "size": 0, "aggs": {
    "by_user": { "terms": { "field": "user_id", "size": 10000 } } } }
//   Response includes:
//     "doc_count_error_upper_bound": 1423
//   -> counts may be wrong by up to 1,423. This is not a bug.

// CORRECT for high cardinality - composite, paged, exact, bounded memory.
{ "size": 0, "aggs": { "users": { "composite": {
    "size": 1000,
    "sources": [ { "user": { "terms": { "field": "user_id" } } } ],
    "after": { "user": "u-12345" }        // cursor from the previous page
} } } }

// Approximate distinct count - HyperLogLog, ~1% error, tiny memory.
{ "size": 0, "aggs": {
    "unique_users": { "cardinality": {
        "field": "user_id", "precision_threshold": 3000 } } } }

---

// THE OOM. Never do this on a text field:
{ "properties": { "description": {
    "type": "text",
    "fielddata": true           // loads EVERY term into heap
} } }

// Correct: a keyword sub-field for aggregating and sorting.
{ "properties": { "description": {
    "type": "text",
    "fields": { "raw": { "type": "keyword", "ignore_above": 256 } }
} } }
//   search on  description
//   aggregate  description.raw

---

// Circuit breakers - working as intended when they fire.
GET /_nodes/stats/breaker
//   "request":   { "limit_size": "6gb", "tripped": 0 }
//   "fielddata": { "limit_size": "8gb", "tripped": 12 }   <- investigate

//   CircuitBreakingException: [request] Data too large,
//     data for [<agg>] would be [6.1gb], which is larger than [6gb]
//
//   The fix is a cheaper query, not a bigger limit.

Aggregate on keyword fields, never enable fielddata on text, and page high-cardinality work with composite instead of a huge terms size.