HPA behaviour and why scaling lags

Key points

• The HPA formula is simple: desired replicas equals current replicas multiplied by current metric over target metric, rounded up. Everything difficult is in the delays around it.
• There are several: metrics scrape interval, the metrics server window, the HPA sync period (15s by default), then pod start and warm-up. Total reaction time is commonly a minute or more.
• For a JVM the warm-up matters as much as the scheduling. A new pod is interpreted and cold, so it is slow exactly when it is being sent its share of a traffic spike.
• CPU is frequently the wrong signal. A service that is slow because a downstream is slow shows low CPU, so a CPU-based HPA will not scale - and if it did, it would add load to the struggling downstream.
• Queue depth, request concurrency or consumer lag are usually better signals because they track work waiting rather than work being done.
• Scale-down is deliberately conservative - a five-minute stabilisation window by default - to avoid flapping. Scale-up can be aggressive; the two directions should not share a policy.

Example

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: orders }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: orders }
  minReplicas: 3
  maxReplicas: 30
  metrics:
    # Concurrency tracks work WAITING, which CPU does not.
    - type: Pods
      pods:
        metric: { name: http_inflight_requests }
        target: { type: AverageValue, averageValue: "30" }

  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0      # react immediately
      policies:
        - type: Percent
          value: 100                     # may double
          periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300    # 5 min - avoid flapping
      policies:
        - type: Percent
          value: 10                      # shed slowly
          periodSeconds: 60

---
# desiredReplicas = ceil(currentReplicas x currentMetric / targetMetric)
#
#   4 pods, 60 in-flight each, target 30  ->  ceil(4 x 60/30) = 8
#
# The delays that make this feel slow:
#   scrape interval        ~15s
#   HPA sync period        ~15s
#   pod schedule + pull    ~10-60s
#   JVM start + warm-up    ~20-60s
#   ------------------------------
#   realistic total        ~1-3 minutes before new capacity is USEFUL
#
# Which is why minReplicas must absorb a spike on its own. Autoscaling
# handles a trend; it does not handle a step change.

# Give a cold JVM time before it counts as ready:
#   startupProbe:
#     httpGet: { path: /actuator/health/readiness, port: 8080 }
#     failureThreshold: 30
#     periodSeconds: 5        # up to 150s to start, without a slow
#                             # livenessProbe killing it mid-warm-up

Autoscaling reacts in minutes, not seconds - size minReplicas for the spike and scale on queued work rather than CPU.