Where Redis latency comes from

Key points

• Command execution is single-threaded. Everything queues behind the command currently running, so a single O(n) command over a large collection blocks every other client.
• The dangerous commands are the ones whose cost scales with data size: KEYS, SMEMBERS, HGETALL and LRANGE 0 -1 on a large key. SCAN and its cursor variants exist to replace them.
• Deleting a large key is also O(n). UNLINK frees it on a background thread instead, which turns a multi-second stall into an immediate return.
• Forking for BGSAVE or AOF rewrite pauses the process while page tables are copied. On a large instance that pause is measured in hundreds of milliseconds, and it is invisible in command-level metrics.
• Expiry is sampled, not scheduled. Redis checks a random subset of keys with TTLs, so a large number expiring at once produces a burst of work - stagger TTLs with jitter.
• SLOWLOG and LATENCY DOCTOR attribute this properly. Client-side timing cannot distinguish a slow command from time spent queued behind someone else.

Example

# What has been slow - the first thing to check.
> SLOWLOG GET 10
1) 1) (integer) 14                 # entry id
   2) (integer) 1754251200         # timestamp
   3) (integer) 82134              # MICROSECONDS - 82ms, single-threaded
   4) 1) "KEYS" 2) "session:*"     # blocked everyone for 82ms

> CONFIG SET slowlog-log-slower-than 5000     # log anything over 5ms

# Built-in analysis:
> LATENCY DOCTOR
> LATENCY HISTORY command
$ redis-cli --latency-history -i 5
$ redis-cli --intrinsic-latency 60      # what the MACHINE can do at best

---
# Replace the O(n) commands:
#
#   KEYS pattern          ->  SCAN 0 MATCH pattern COUNT 100
#   HGETALL bighash       ->  HSCAN, or HMGET of the fields you need
#   SMEMBERS bigset       ->  SSCAN
#   LRANGE key 0 -1       ->  LRANGE with a real page window
#   DEL bigkey            ->  UNLINK bigkey        (frees in background)
#   FLUSHALL              ->  FLUSHALL ASYNC

# SCAN is cursor-based and non-blocking:
> SCAN 0 MATCH "session:*" COUNT 100
1) "17"                            # next cursor; 0 means done
2) 1) "session:abc" 2) "session:def"
#   NOTE: SCAN may return duplicates and does not snapshot. It guarantees
#   only that keys present throughout the whole iteration are returned.

---
# TTL stampede: 100k keys expiring in the same second is a work burst.
#
#   BAD:   SETEX key 3600 value          every key, same TTL
#   GOOD:  SETEX key (3600 + rand(300)) value
#
# The same jitter also prevents a cache stampede on the origin when a
# whole cohort of keys expires together.

# Fork pause is invisible in command latency. Watch for it:
> INFO stats
#   latest_fork_usec:284000        # 284ms of stall, on every BGSAVE

One thread means one slow command stalls everyone - replace the O(n) commands, UNLINK large keys, and jitter TTLs.