Rollouts, disruption budgets and draining safely

Key points

• Termination is not ordered the way people assume. When a pod is deleted, the SIGTERM and the endpoint removal happen concurrently - so traffic can still arrive after the application has begun shutting down.
• That race is the source of most "errors during deploy". The fix is a preStop sleep: the container stays up and healthy for a few seconds while endpoint removal propagates to every node.
• After the grace period expires the container gets SIGKILL, so terminationGracePeriodSeconds must exceed the preStop sleep plus the longest in-flight request.
• maxSurge and maxUnavailable control the rollout shape. maxUnavailable 0 with maxSurge 1 never drops below full capacity, at the cost of needing room for one extra pod.
• A PodDisruptionBudget governs voluntary disruption only - node drains and cluster upgrades. It does not protect against a node crashing, which is what people assume it does.
• Without a PDB, draining nodes for an upgrade can evict every replica of a Deployment at once, so the cluster upgrade becomes an application outage.

Example

# The race, in order of what actually happens:
#
#   t=0    pod marked Terminating
#          |-- SIGTERM sent to the container      (these two are
#          |-- endpoint removal begins             CONCURRENT)
#   t=0-2  kube-proxy on each node updates its rules... eventually
#          -> traffic still arriving at a pod that is shutting down
#   t=30   SIGKILL if still running

spec:
  terminationGracePeriodSeconds: 45      # > preStop + longest request
  containers:
    - name: app
      lifecycle:
        preStop:
          exec:
            # Stay healthy while endpoint removal propagates. This single
            # line removes most deploy-time 502s.
            command: ["sh", "-c", "sleep 8"]

---
# Never drop below full capacity during a rollout.
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

---
# VOLUNTARY disruption only - drains and upgrades, not crashes.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: orders }
spec:
  minAvailable: 2          # or maxUnavailable: 1
  selector:
    matchLabels: { app: orders }

# Without this, `kubectl drain` during a cluster upgrade can evict all
# replicas at once. WITH it, the drain blocks until it can proceed safely.

# Watch a rollout, and roll back on evidence rather than hope:
#   kubectl rollout status deploy/orders --timeout=5m
#   kubectl rollout undo   deploy/orders

SIGTERM and endpoint removal race each other - a preStop sleep is what stops a rollout from returning errors.