Where the memory actually goes

Key points

• Redis stores small collections in compact encodings - a listpack or intset - which are arrays scanned linearly. Above a configured threshold it converts to a hash table or skiplist, and memory use jumps.
• The conversion is one-way. Removing elements afterwards does not convert it back, so a hash that briefly exceeded the threshold keeps the expensive encoding forever.
• Every key carries overhead beyond its value - the key string, the dictionary entry, an expiry entry if it has a TTL. Around 50 to 100 bytes per key, which dominates when values are small.
• That is why one million keys holding an 8-byte integer costs far more than 8MB. Grouping related fields into a single hash amortises the overhead across many values.
• Memory is not returned to the operating system promptly. The allocator keeps freed pages, so used_memory falling while RSS stays high is normal rather than a leak - watch the fragmentation ratio.
• Use MEMORY USAGE on real keys and --bigkeys to find outliers. Estimating from value size alone is consistently wrong by an order of magnitude.

Example

# What encoding is a key actually using?
> RPUSH mylist a b c
> OBJECT ENCODING mylist
"listpack"                 # compact, array-like

> RPUSH mylist <200 more items>
> OBJECT ENCODING mylist
"quicklist"                # converted - and it will NOT convert back

# The thresholds that trigger conversion:
hash-max-listpack-entries 128
hash-max-listpack-value   64
list-max-listpack-size    128
set-max-intset-entries    512
zset-max-listpack-entries 128

# PER-KEY OVERHEAD - measured, not estimated:
> MEMORY USAGE user:1:visits
(integer) 64               # for an 8-byte integer value

#   1,000,000 separate keys x ~64 bytes  = ~64 MB
#   the same 1,000,000 values in hashes  = ~16 MB
#
#   BAD:   SET user:1:name  / SET user:1:email / SET user:1:visits
#   GOOD:  HSET user:1 name ... email ... visits ...

# Find the outliers rather than guessing:
$ redis-cli --bigkeys
$ redis-cli --memkeys

# Fragmentation: RSS high while used_memory drops is EXPECTED.
> INFO memory
#   used_memory_human:1.20G          what Redis thinks it holds
#   used_memory_rss_human:1.80G      what the OS has given it
#   mem_fragmentation_ratio:1.50     >1.5 sustained is worth acting on

# Let the allocator hand pages back, gradually:
activedefrag yes
active-defrag-ignore-bytes 100mb
active-defrag-threshold-lower 10

Per-key overhead dwarfs small values, and an encoding conversion never reverses - group related fields into one hash.