Mapping explosion and field count

Key points

• Dynamic mapping adds a field the first time it appears. If document keys are caller-defined - a per-tenant attribute bag, or metrics keyed by name - the mapping grows without limit.
• Every field costs cluster state on the master and memory per shard. Thousands of fields make the mapping itself a performance problem, quite separate from the data.
• The mapping is additive and permanent. A field created by one bad document cannot be removed without reindexing, and its type cannot be changed either.
• The correct shape for unpredictable keys is the nested key-value pattern: one field for the name and one for the value, so a thousand different attributes become two mapped fields.
• flattened is the lighter alternative: the whole object is indexed as one field, so the mapping never grows. The trade is that everything inside is a keyword - no numeric ranges, no analysis.
• Set dynamic: strict on anything important. It rejects unknown fields outright, which is much better than silently accepting a typo and mapping it forever.

Example

// THE PROBLEM - caller-defined keys with dynamic mapping.
POST /events/_doc
{ "user": "u1", "attrs": { "campaign_a1b2": "x", "session_9f8e": "y" } }
//   Every new key creates a new mapped field, permanently.
//   After a month: 40,000 fields, and the master is struggling.

// Guard rails - these should be on every index.
PUT /events
{ "settings": {
    "index.mapping.total_fields.limit": 1000,   // default is 1000; do not raise
    "index.mapping.depth.limit": 5,
    "index.mapping.nested_fields.limit": 50
  },
  "mappings": { "dynamic": "strict" }           // reject unknown fields
}
//   dynamic: strict   -> 400 on an unknown field. Loud, and correct.
//   dynamic: false    -> stored but not indexed, not searchable
//   dynamic: true     -> the default, and the cause of the problem

---

// FIX 1 - nested key/value. 1,000 attributes, TWO mapped fields.
{ "mappings": { "properties": {
    "attrs": { "type": "nested", "properties": {
        "key":   { "type": "keyword" },
        "value": { "type": "keyword" }
} } } } }

// Document:
{ "user": "u1", "attrs": [
    { "key": "campaign_a1b2", "value": "x" },
    { "key": "session_9f8e",  "value": "y" } ] }

// Query - nested, so key and value must match on the SAME element.
{ "query": { "nested": { "path": "attrs", "query": { "bool": { "filter": [
    { "term": { "attrs.key":   "campaign_a1b2" } },
    { "term": { "attrs.value": "x" } } ] } } } } }

---

// FIX 2 - flattened. Mapping never grows; everything inside is a keyword.
{ "mappings": { "properties": {
    "attrs": { "type": "flattened" }
} } }
{ "query": { "term": { "attrs.campaign_a1b2": "x" } } }
//   Cheapest option. No ranges, no analysis, no per-field mapping.

// How many fields do you have right now?
GET /events/_mapping?filter_path=**.properties
GET /_cluster/stats?filter_path=indices.mappings

Dynamic mapping plus caller-defined keys grows the mapping forever - use nested key/value or flattened, and set dynamic strict.