Scaling out behind a load balancer

Key points

• Vertical scaling (a bigger box) is the cheapest fix and the right first move - until you hit the biggest box, which can still die.
• Horizontal scaling means many identical instances behind a load balancer. It needs your app to be stateless.
• Stateless means no session or cache in local memory that another instance lacks. Push state to the database, Redis, or the client token.
• The balancer picks a healthy instance: round robin, least connections, or a hash of a key when you want the same user on the same box.
• Health checks are the whole trick. Without a readiness check the balancer keeps sending traffic to a broken instance.
• L4 balances TCP connections and is fast; L7 reads HTTP so it can route by path or host, do TLS, and retry.

Example

# Kubernetes: a Service load-balances across every ready Pod
apiVersion: v1
kind: Service
metadata:
  name: orders
spec:
  selector:
    app: orders          # any Pod with this label, if it is ready
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders
spec:
  replicas: 4            # scale out: same image, more copies
  selector:
    matchLabels:
      app: orders
  template:
    metadata:
      labels:
        app: orders
    spec:
      containers:
        - name: app
          image: registry.example.com/orders:1.4.2
          ports:
            - containerPort: 8080
          readinessProbe:              # fails -> Pod is pulled out of the Service
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            periodSeconds: 5
          livenessProbe:               # fails -> Pod is restarted
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            periodSeconds: 10

Make the app stateless first; only then does adding instances actually add capacity.