What the scheduler is actually deciding

Key points

• Scheduling is two phases. Filtering discards nodes that cannot run the pod at all; scoring ranks the survivors and the highest wins. A Pending pod means filtering removed every node.
• The events tell you which predicate failed, and the message is specific: insufficient cpu, node had taint, didn not match node selector, or no available persistent volume.
• Scheduling uses requests, never actual usage. A node running at 10% CPU can still be unschedulable because the pods on it have reserved everything - which is why cluster utilisation and schedulability are different numbers.
• Topology spread constraints are how you get replicas across zones. Without them the scheduler may legitimately place every replica on one node, and a single node failure takes the whole Deployment down.
• Taints and tolerations repel; affinity attracts. They are complementary: a taint keeps everything off a node unless it tolerates, while affinity expresses a preference to be near or away from other pods.
• Preemption means a higher-priority pod can evict lower-priority ones to fit. Without PriorityClasses, a critical workload has no way to displace batch work during a capacity crunch.

Example

# Why is it Pending? The events name the failing predicate.
$ kubectl describe pod orders-abc | tail -5
#   0/12 nodes are available:
#     3 Insufficient cpu,
#     4 node(s) had untolerated taint {dedicated: gpu},
#     5 node(s) didn't match pod topology spread constraints

# Reserved vs used - these differ, and scheduling only sees the first:
$ kubectl describe node node-1 | grep -A5 "Allocated resources"
#   cpu     3800m (95%)     <- RESERVED, what the scheduler sees
$ kubectl top node node-1
#   cpu     420m  (10%)     <- actually USED
#
# 95% reserved, 10% used: unschedulable and idle at the same time.
# The cause is over-large requests, not a shortage of hardware.

---
# Spread replicas across zones. Without this, all 3 can land on one node.
spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule    # ScheduleAnyway = best effort
      labelSelector:
        matchLabels: { app: orders }

  # Keep replicas off the same node as each other.
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            topologyKey: kubernetes.io/hostname
            labelSelector:
              matchLabels: { app: orders }

---
# Priority: let critical work displace batch work under pressure.
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: critical }
value: 1000000
preemptionPolicy: PreemptLowerPriority

Scheduling counts requests, not usage - a node can be 95% reserved and 10% busy, and the fix is right-sizing requests.