Failure domains and blast radius

Key points

• A failure domain is the set of things that fail together: a process, a host, a rack, a zone, a region, or a shared dependency such as a configuration service.
• Redundancy inside one domain is not redundancy. Three replicas on three hosts in one rack survive a host failure and not a rack failure - the count looks reassuring and the domain is the same.
• Cell architecture bounds the blast radius by partitioning users into independent cells, each with its own full stack. A failure takes out one cell rather than the service.
• Shared dependencies are the hidden domain. Every service being independently deployable does not help if all of them read the same configuration service at startup.
• Shuffle sharding is a striking trick: assign each customer a random subset of workers, and two customers are unlikely to share the same full set. One bad tenant then affects a small, bounded fraction of the others.
• Deployment is a failure domain too. Rolling to everything at once means a bad release is a total outage; staged rollout by cell or region makes it a partial one you can stop.

Example

/*
 * DOMAINS, smallest to largest - and what each one takes with it:
 *
 *   process      one instance          restart
 *   host         all its containers    reschedule
 *   rack         shared power/network  zone-level redundancy needed
 *   zone         a datacentre          multi-zone needed
 *   region       correlated disaster   multi-region needed
 *   global       config, DNS, deploy   <- the one people forget
 *
 * The last row is why "we are multi-region" is often not true: a single
 * global config service or a single deploy pipeline is a global domain.
 */

# Redundancy that is not redundancy:
#   3 replicas, 3 hosts, ONE rack   -> survives a host, not a rack
# Spread across the real domain:
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule

/*
 * SHUFFLE SHARDING - the counter-intuitive one.
 *
 * 8 workers, each customer assigned 2 at random:
 *   distinct pairs = C(8,2) = 28
 *   P(two customers share BOTH workers) = 1/28 = 3.6%
 *
 * So one abusive customer degrades ~4% of the others, not 100%.
 * With 100 workers and 5 each, the number of combinations is enormous
 * and effective isolation approaches per-customer - without per-customer
 * infrastructure.
 */

/*
 * CELLS - each is a complete, independent stack:
 *
 *   cell-1  [ lb | app | db | cache ]   customers A-F
 *   cell-2  [ lb | app | db | cache ]   customers G-M
 *   cell-3  [ lb | app | db | cache ]   customers N-Z
 *
 * A cell failure affects a third of customers, and the router is the
 * only shared component - so it must be trivially simple.
 *
 * Deploy cell by cell: a bad release is caught in cell-1 and never
 * reaches the others.
 */

Count redundancy by failure domain, not by instance - and remember config, DNS and the deploy pipeline are global domains.